Expand description
BP-2: document conversion for the read and web tools — the “returns it in a form the model can actually use” half of three catalog rows:
pdf_text— PDF → extracted text pages (catalog:27 “Read tool returns images/PDFs/ipynb as model-visible content”). A small in-crate extractor over the PDF content streams (flate2for the/FlateDecodefilter every real-world writer uses), not a full PDF renderer: it recovers the text layer, and says so honestly when a document has none (scanned/image-only or encrypted).notebook_markdown—.ipynb→ cells rendered WITH their outputs (same row).html_to_markdown— HTML → markdown (catalog:44 “Fetch a URL, convert to markdown, return to model”).
Same “small parser over a crate” precedent as config::glob_match,
tools::url_host and builtins::base64_encode.
Structs§
- PdfDocument
- What
pdf_textrecovered from a PDF’s bytes. - PdfPage
- One extracted PDF page (content stream, in file order).
- Search
Result - One result from an HTML search-results page.
Functions§
- decode_
entities - Decode the named entities a text-extraction path actually meets, plus
numeric (
{/{) references. - html_
to_ markdown - Convert an HTML document to markdown: headings, links, list items,
emphasis, code and block structure survive;
<script>/<style>/ comments and every other tag are dropped, entities are decoded. - is_
notebook_ path - Whether
pathhas an.ipynbextension (case-insensitive). - is_
pdf_ path - Whether
pathhas a.pdfextension (case-insensitive). - notebook_
markdown - Render a
.ipynbfor the model: every cell with its type, execution count and its OUTPUTS (stdout/stderr streams, text results, errors) — the half a raw JSON decode buries. - parse_
html_ search_ results - BP-2 (catalog:45 “Provider/server-backed search”): pull results out of a
DuckDuckGo-style HTML results page — the
result__aanchors and theirresult__snippetsiblings, in page order. - pdf_
markdown - Render a PDF for the model: extracted pages, or an honest structured summary when the document carries no recoverable text layer.
- pdf_
text - Extract the text layer of a PDF.
- percent_
decode - Percent-decode a URL component (
%20,+as space). - render_
search_ results - Render extracted results for the model: a numbered list of
title — urlwith each engine snippet under it.