Expand description
dyer is designed for reliable, flexible and fast Request-Response based service, including data processing, web-crawling and so on, providing some friendly, interoperable, comprehensive features without compromising speed.
dyer provides some high-level features:
- asynchronous, lock-free, concurrent streaming and I/O, make the best of thread pool, network, and system resource.
- Event-driven, once you set the initials and recursive generator,
dyerwill handle the rest of it interoperably. - User-friendly and flexible,
dyeroffers high-level, easy to use wrappers and APIs what does a lot for you.
§Feature Flag
To reduce code redundancy and speed up compilation, dyer use feature flag to mark the necessary modules/functions, Currently here are some supported Features:
xpath: Enable parse the html response with xpathcompression: Enable HTTP Compression:br,deflate,gzipproxy: Enable use proxiesfull: Enable all features
Get started by installing dyer-cli and looking over the examples.
Modules§
- affix
- This module contains structs that adjusts Task before consume it
- body
- Contains some data structure that related to the content of HTTP
- client
- the Client that asynchronously executes Request,
- component
- Instructions of components including client, affix, request, response, task, utils.
- couple
- A joints of a Task and Affix in practice, it is used to reproduce a Request
- dyer_
macros - attribute macro of dyer
- engine
- An instruction of App and its configuration ArgApp
- info
- A structure that carries basic meta-data, including its origin, stime stamp, privilege, encoding and so on.
- log
- A lightweight logging facade.
- parsed
- The result of parser when parsing Response
- plugin
- Instructions of plugins including middleware, pipeline and their usage.
- proxy
proxy - A proxy that will re-route the request to
Note that it currently supports
HTTP - request
- This module contains structs related to HTTP requests, notably the Request type itself as well as a builder to create requests. Typically you’ll import the http::Request type rather than reaching into this module itself.
- response
- HTTP response types.
- task
- This module contains structs related to HTTP requests,
- utils
- some utilities that useful and convenience for dealing with data flow.
- xpath
xpath - implement the xpath for the response more to see Response::xpath
Structs§
- Affix
- generally, it provide extra infomation , meta data required by server,
including
User-Agent,Accept-Encodingand so on. - App
- An abstraction and collection of data flow
- ArgAffix
- To control the workflow of engine in dealing with Affix including using affix or not, the amount to use/generate
- ArgApp
- Arguments that control the App at runtime, including using history or not,
Task Affix Request Response entities consuming and generating There shall be an introduction to every member(maybe coming soon). - ArgRate
- some infomation about
dyerat rumtime where speed and error-handler based on - Auth
Basic proxy - Basic authentication
- Auth
Bearer proxy - Bearer authentication
- Auth
Custom proxy - Custom authentication
- Body
- series of Bytes that receives from network or to be sent
- Bytes
- A cheaply cloneable and sliceable chunk of contiguous memory.
- Client
- Client that take Request and execute, return Response
- Couple
- Group a Task and a optional Affix to make this it is the raw stuff before turned into a Request
- Extensions
- A type map of protocol extensions.
- Info
- basic meta data related 3 ranging
- Meta
Request - Meta Data of the Request
- Meta
Response - Meta Data of the Response
- Meta
Task - Meta Data of the Task
- Middle
Ware - Represents a medium that handles the dataflow of App
- Middle
Ware Builder - Serve as an medium to create an instance of MiddleWare
- Parsed
- the parsed result returned by
parser. - Pipe
Line - Represents a medium that manipulates the collected data structure of App
- Pipe
Line Builder - Serve as an medium to create an instance of PipeLine
- Proxy
proxy - data type that represent proxy
- Request
- An Wrapper of http::Request
- Response
- An Wrapper of http::Response
- Task
- An
Taskconsists of a head and a potentially optional body. The body component is generic, enabling arbitrary types to represent the HTTP body. For example, the body could be Vec, a Stream of byte chunks, or a value that has been deserialized.
Enums§
- Auth
proxy - proxy authentication it support 3 types:
- Client
Type - represent an client that invoke make requests
Statics§
- CLIENTPOOL
- the Pool contains constructed Client
- FNMAP
- A vector of tuple which contians function name(&str) and function pointer (*const ()),
Traits§
- Actor
- Although it starts a project, the work to do here is not very complicated it is as simple as setting up initial condition and other basic things.
- Affixor
- Trait to adjust Task before make a request
- Buf
- Read bytes from a buffer.