Crate dyer

dyer is designed for reliable, flexible and fast Request-Response based service, including data processing, web-crawling and so on, providing some friendly, interoperable, comprehensive features without compromising speed.

dyer provides some high-level features:

• asynchronous, lock-free, concurrent streaming and I/O, make the best of thread pool, network, and system resource.
• Event-driven, once you set the initials and recursive generator, dyer will handle the rest of it interoperably.
• User-friendly and flexible, dyer offers high-level, easy to use wrappers and APIs what does a lot for you.

Feature Flag

To reduce code redundancy and speed up compilation, dyer use feature flag to mark the necessary modules/functions, Currently here are some supported Features:

• xpath: Enable parse the html response with xpath
• compression: Enable HTTP Compression: br, deflate, gzip
• proxy: Enable use proxies
• full: Enable all features

Get started by installing dyer-cli and looking over the examples.

Modules

affix

This module contains structs that adjusts Task before consume it

body

Contains some data structure that related to the content of HTTP

client

the Client that asynchronously executes Request,

component

Instructions of components including client, affix, request, response, task, utils.

couple

A joints of a Task and Affix in practice, it is used to reproduce a Request

dyer_macros

attribute macro of dyer

engine

An instruction of App and its configuration ArgApp

info

A structure that carries basic meta-data, including its origin, stime stamp, privilege, encoding and so on.

log

A lightweight logging facade.

parsed

The result of parser when parsing Response

plugin

Instructions of plugins including middleware, pipeline and their usage.

proxyproxy

A proxy that will re-route the request to Note that it currently supports HTTP

request

This module contains structs related to HTTP requests, notably the Request type itself as well as a builder to create requests. Typically you’ll import the http::Request type rather than reaching into this module itself.

response

HTTP response types.

task

This module contains structs related to HTTP requests,

utils

some utilities that useful and convenience for dealing with data flow.

xpathxpath

implement the xpath for the response more to see Response::xpath

Structs

Affix

generally, it provide extra infomation , meta data required by server, including User-Agent, Accept-Encoding and so on.

App

An abstraction and collection of data flow

ArgAffix

To control the workflow of engine in dealing with Affix including using affix or not, the amount to use/generate

ArgApp

Arguments that control the App at runtime, including using history or not,
Task Affix Request Response entities consuming and generating There shall be an introduction to every member(maybe coming soon).

ArgRate

some infomation about dyer at rumtime where speed and error-handler based on

AuthBasicproxy

Basic authentication

AuthBearerproxy

Bearer authentication

AuthCustomproxy

Custom authentication

Body

series of Bytes that receives from network or to be sent

Bytes

A cheaply cloneable and sliceable chunk of contiguous memory.

Client

Client that take Request and execute, return Response

Couple

Group a Task and a optional Affix to make this it is the raw stuff before turned into a Request

Extensions

A type map of protocol extensions.

Info

basic meta data related 3 ranging

MetaRequest

Meta Data of the Request

MetaResponse

Meta Data of the Response

MetaTask

Meta Data of the Task

MiddleWare

Represents a medium that handles the dataflow of App

MiddleWareBuilder

Serve as an medium to create an instance of MiddleWare

Parsed

the parsed result returned by parser.

PipeLine

Represents a medium that manipulates the collected data structure of App

PipeLineBuilder

Serve as an medium to create an instance of PipeLine

Proxyproxy

data type that represent proxy

Request

An Wrapper of http::Request

Response

An Wrapper of http::Response

Task

An Task consists of a head and a potentially optional body. The body component is generic, enabling arbitrary types to represent the HTTP body. For example, the body could be Vec, a Stream of byte chunks, or a value that has been deserialized.

Enums

Authproxy

proxy authentication it support 3 types:

ClientType

represent an client that invoke make requests

Statics

CLIENTPOOL

the Pool contains constructed Client

FNMAP

A vector of tuple which contians function name(&str) and function pointer (*const ()),

Traits

Actor

Although it starts a project, the work to do here is not very complicated it is as simple as setting up initial condition and other basic things.

Affixor

Trait to adjust Task before make a request

Buf

Read bytes from a buffer.

Type Definitions

BoxFuture

LocalBoxFuture

Attribute Macros

actor

affix

async_trait

entity

middleware

parser

pipeline