Requests2

A Rust library the crate help you quickly request, parse and store data ([Python] BS4 library).

• Each new requests initializes a cache instance which stores the parsed data in key value pairs

• When you get an instance of connect, you can call the parser method to parse the data in the form of closure

• use dbnote macro, write data to a database table

• set connect database str in config file

• find find_all select select_all method supports CSS selectors to parse DOM document

config
    postgres=<host=localhost user=your password=test dbname=postgres>
    sqlite=<dbname=sqlite_db>

auto add sqlite_db file to project dir, support sqlite

store data to csv and postgres database, you can use this example code:

let data = Cache::new();
    let client = Requests::new(&data);
    let rq = client.connect("https://www.qq.com/", Headers::Default);

    #[derive(DBfile, Debug)]
    #[dbnote(table_name = "test_link", driver = "postgres", primary_key="href")]
    struct Link<'a> {
        href: &'a str,
        link_name: String,
        title: &'a str,
    }

    rq.free_parse(|p| {
        let title = p.select("title").text();

        let links = p
            .select_all("li.nav-item a")
            .iter()
            .map(|x| Link {
                title: "",
                href: x.attr("href").unwrap_or_default(),
                link_name: x.text(),
            })
            .collect::<Vec<Link>>();


        // create a table
        links[0].create_table();

        for (idx, mut link) in links.into_iter().enumerate() {
            if idx == 0 {
                link.title = &title;
                link.write_csv_head();
            }
            link.to_csv("a");
            link.to_db();
        }
    });

Add #[dbnote(table_name = "test_link", driver = "postgres", primary_key="href")] to struct, you can use write_csv_head() and to_csv put data to a file, table_name as file name. Use createa_table() you can creata table in postgres , but you must add config file to project. to_db() put data to the table.

find() find_all() select() select_all() method unified use css selector as first argument.

See the test folder for more details