Requests2

A Rust library the crate help you Write a function similar to that of Python's request repository ([Python] BS4 library).

Each new requests initializes a cache instance which stores the parsed data in key value pairs
When you get an instance of connect, you can call the parser method to parse the data in the form of closure

Example

let data = Cache::new();
let client = Requests::new(&data);
let mut rq = client.connect("https://www.qq.com/", Headers::Default);

rq.parser(|p| {
    p.find_all("a", |x| {
        x.attr("href").map_or(false, |v| v.starts_with("http"))
    }, "href")
}, "href"); //
data.print()

Use data.print you can view the value stored as the [href] key. It is a value enumeration type that contains most data types.

output

Key -- "href" Value -- LIST(["https://qzone.qq.com", "https://qzone.qq.com", "https://mail.qq.com", "https://mail.qq.com/cgi-bin/loginpage", "http://news.qq.com/", "https://v.qq.com/?isoldly=1", "http://gongyi.qq.com/",...]

Headers

Headers defines three types of request headers. The default Header::default has only one user agent, or it can be without any Headers::None.also use JSON string to make a request header containing useragent and host, this code:

let headers = r#"{"user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36", "host": "www.qq.com"}"#;
let store = Cache::new();
let client = Requests::new(&store);
let mut p = client.connect("https://www.qq.com", Headers::JSON(headers));

If you need more request header fields, you need to use add corresponding fields in headers.rs

Parser

When you use the connect method to connect to a URL, you can use the parser method to write the work of parsing HTML, the current parser has find and find_ all method

the parser must return Value type

If you use find find_all the code:

rq.parser(|p| {
    p.find_all("a", |x| {
        x.attr("href").map_or(false, |v| v.starts_with("http"))
    }, "href")
}, "href")

The first parameter of parser is obtained by closure automatically saves the value to value:: list

In general, you may need to handle the parsing manually and customize the returned Value, please use p.select function , this example code:

let data = Cache::new();
    let client = Requests::new(&data);
    let mut parser = client.connect("https://www.qq.com", Headers::Default);
    parser.parser(|p| {
        let mut result = HashMap::new();

        let navs = p.select("li.nav-item", |nodes| {
            let navs = nodes.into_iter().map(|n| {
                let mut item = HashMap::new();
                n.find(Name("a")).next().map_or(HashMap::from([("".to_string(), Value::NULL)]), |a| {
                    let nav_name = a.text();
                    let nav_href = a.attr("href").map_or(String::from(""), |x| x.to_string());
                    item.insert("nav_name".to_string(), Value::STR(nav_name));
                    item.insert("nav_href".to_string(), Value::STR(nav_href));
                    item
                })
            }).collect::<Vec<HashMap<String, Value>>>();

            Value::VECMAP(navs)
        });

        let news = p.select("ul.yw-list", |nodes| {
            let mut news = Vec::new();
            
            for node in nodes {
                for n in node.find(Class("news-top")) {
                    for a in n.find(Name("a")) {
                        let title = a.text();
                        news.push(title);
                    }
                }
            }

            Value::LIST(news)
        });

        result.insert("titles".to_owned(), news);
        result.insert("nav".to_owned(), navs);

        Value::MAP(result)
    }, "index");

    data.print();

Value

pub enum Value {
    /// 字符串类型
    STR(String),
    /// 字符串列表
    LIST(Vec<String>),
    INT(i32),
    /// 空数据
    NULL,
    /// bool
    BOOL(bool),

    /// map类型的列表
    VECMAP(Vec<HashMap<String, Value>>),

    /// map类型
    MAP(HashMap<String, Value>)
}

Add the data type you need in the Value.rs

Concurrency support

use rayon library test the concurrency, this have a simple code: