essence-engine-0.2.0
- essence-engine 0.2.0
- Permalink
- Docs.rs crate page
- MIT
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- ruchit-p
- Dependencies
- anyhow ^1 normal
  async-trait ^0.1 normal
  axum ^0.8 normal
  backoff ^0.4 normal
  base64 ^0.22 normal
  blake3 ^1.5 normal
  chromiumoxide ^0.7 normal
  chrono ^0.4 normal
  clap ^4 normal
  dotenvy ^0.15 normal
  encoding_rs ^0.8 normal
  futures ^0.3 normal
  glob-match ^0.2 normal
  governor ^0.8 normal
  hickory-resolver ^0.24 normal
  html2md ^0.2 normal
  html5ever ^0.29 normal
  image ^0.25 normal
  lru ^0.12 normal
  moka ^0.12 normal
  num_cpus ^1.16 normal
  percent-encoding ^2 normal
  psl ^2 normal
  rand ^0.8 normal
  readability ^0.3 normal
  regex ^1 normal
  reqwest ^0.12 normal
  rmcp ^0.15 normal
  robotstxt ^0.3 normal
  schemars ^1.0 normal
  scraper ^0.22 normal
  serde ^1 normal
  serde_json ^1 normal
  thiserror ^2 normal
  tokio ^1 normal
  tokio-retry ^0.3 normal
  tokio-stream ^0.1 normal
  tower ^0.5 normal
  tower-http ^0.6 normal
  tracing ^0.1 normal
  tracing-subscriber ^0.3 normal
  url ^2 normal
  uuid ^1 normal
  whatlang ^0.16 normal
  criterion ^0.5 dev
  mockito ^1 dev
  rusqlite ^0.32 dev
  serial_test ^3 dev
  sha2 ^0.10 dev
  tokio-test ^0.4 dev
  wiremock ^0.6 dev
- Versions
- 59.23% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

Module robots

essence0.2.0

Module robots

Module Items

Functions

In essence::utils

Module robots

Functions§

is_allowed: Check if a URL is allowed by robots.txt
is_allowed_default: Check robots.txt with default user agent