Skip to main content

Module convert

Module convert 

Source
Expand description

BP-2: document conversion for the read and web tools — the “returns it in a form the model can actually use” half of three catalog rows:

  • pdf_text — PDF → extracted text pages (catalog:27 “Read tool returns images/PDFs/ipynb as model-visible content”). A small in-crate extractor over the PDF content streams (flate2 for the /FlateDecode filter every real-world writer uses), not a full PDF renderer: it recovers the text layer, and says so honestly when a document has none (scanned/image-only or encrypted).
  • notebook_markdown.ipynb → cells rendered WITH their outputs (same row).
  • html_to_markdown — HTML → markdown (catalog:44 “Fetch a URL, convert to markdown, return to model”).

Same “small parser over a crate” precedent as config::glob_match, tools::url_host and builtins::base64_encode.

Structs§

PdfDocument
What pdf_text recovered from a PDF’s bytes.
PdfPage
One extracted PDF page (content stream, in file order).
SearchResult
One result from an HTML search-results page.

Functions§

decode_entities
Decode the named entities a text-extraction path actually meets, plus numeric ({ / {) references.
html_to_markdown
Convert an HTML document to markdown: headings, links, list items, emphasis, code and block structure survive; <script>/<style>/ comments and every other tag are dropped, entities are decoded.
is_notebook_path
Whether path has an .ipynb extension (case-insensitive).
is_pdf_path
Whether path has a .pdf extension (case-insensitive).
notebook_markdown
Render a .ipynb for the model: every cell with its type, execution count and its OUTPUTS (stdout/stderr streams, text results, errors) — the half a raw JSON decode buries.
parse_html_search_results
BP-2 (catalog:45 “Provider/server-backed search”): pull results out of a DuckDuckGo-style HTML results page — the result__a anchors and their result__snippet siblings, in page order.
pdf_markdown
Render a PDF for the model: extracted pages, or an honest structured summary when the document carries no recoverable text layer.
pdf_text
Extract the text layer of a PDF.
percent_decode
Percent-decode a URL component (%20, + as space).
render_search_results
Render extracted results for the model: a numbered list of title — url with each engine snippet under it.