crawlkit-parser-0.3.1
- crawlkit-parser 0.3.1
- Permalink
- Docs.rs crate page
- MIT
- 14 July 2026
- Links
- Repository
- crates.io
- Source
- Owners
- cyberdoors
- Dependencies
- crawlkit-core ^0.3.0 normal
  dom_smoothie ^0.18.0 normal
  lazy_static ^1 normal
  regex ^1 normal
  scraper ^0.22 normal
  serde ^1 normal
  serde_json ^1 normal
  skyscraper ^0.6 normal
  thiserror ^2 normal
  tracing ^0.1 normal
  url ^2 normal
- Versions
- 64.13% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

extract_robots

crawlkit_parser0.3.1

In crawlkit_parser::metadata

crawlkit_parser::metadata

Function extract_robots

pub fn extract_robots(document: &Html) -> RobotsMeta

Expand description

提取 robots meta 标签，解析指令