Crate rama_proxy

Expand description

rama proxy types and utilities

Proxy protocols are implemented in their relevant crates:

HaProxy: rama-haproxy
HttpProxy: rama-http-backend

See the ProxyFilter for more information on how to select a proxy, and the ProxyDB trait for how to implement a proxy database.

If you wish to support proxy filters directly from the username, you can use the ProxyFilterUsernameParser to extract the proxy filter so it will be added to the input Extensions.

The ProxyDB is used by Connection Pools to connect via a proxy, in case a ProxyFilter is present in the input Extensions.

§DB Live Reloads

ProxyDB implementations like the MemoryProxyDB feel static in nature, and they are. The goal is really to load it once and read it often as fast as possible.

In fact, given that we access everything through shared references, there is also no cheap way to mutate it all the time.

As such the normal way to update data such as your proxy list is by performing a rolling update of your actual rama-driven proxy workloads.

That said. By using crates such as left-right you can relatively affordable perform live reloads by having the writer on its own tokio task and wrap the reader in a ProxyDB implementation. This way you can live reload based upon a signal, or more realistically, every x minutes.

§ProxyDB layer

ProxyDB layer support to select a proxy based on the given input Extensions.

This layer expects a ProxyFilter to be available in the input Extensions, which can be added by using the HeaderConfigLayer (rama-http) when operating on the HTTP layer and/or by parsing it via the TCP proxy username labels (e.g. john-country-us-residential), in case you support that as part of your transport-layer authentication. And of course you can combine the two approaches.

You can also give a single Proxy as “proxy db”.

By default the layer publishes at most five matching proxies as an ordered ProxyRoutes plan. The limit is configurable, including an unbounded mode. Each route carries its Proxy and ProxyID as route-specific extensions. ProxyRoutesConnector installs only the selected route’s metadata on its isolated connection attempt and returned input.

ProxyDBLayer::with_single_proxy opts into the legacy behaviour of using the database’s singular selection and inserting one route. MemoryProxyDB keeps its random selection for this mode. A database miss remains an error in either mode; no direct route is appended implicitly. Existing singular routes are preserved by default in both modes. ProxyDBLayer::with_overwrite_proxy explicitly lets the database selection replace them by publishing the newer decision. ProxyRoutesLayer resolves singular and plural decisions by insertion order before route-aware middleware runs.

§Example

use rama_http_types::{Body, Version, Request};
use rama_proxy::{
     Proxy,
     ProxyDBLayer, ProxyFilterMode,
     ProxyFilter,
};
use rama_core::{
   service::service_fn,
   extensions::{ExtensionsRef},
   Service, Layer,
};
use rama_net::client::ProxyRoute;
use rama_utils::str::non_empty_str;
use std::{convert::Infallible, sync::Arc};

#[tokio::main]
async fn main() {
    let db = MemoryProxyDB::try_from_iter([
        Proxy {
            id: non_empty_str!("42"),
            address: "12.34.12.34:8080".try_into().unwrap(),
            tcp: true,
            udp: true,
            http: true,
            https: false,
            socks5: true,
            socks5h: false,
            datacenter: false,
            residential: true,
            mobile: true,
            pool_id: None,
            continent: Some("*".into()),
            country: Some("*".into()),
            state: Some("*".into()),
            city: Some("*".into()),
            carrier: Some("*".into()),
            asn: None,
        },
        Proxy {
            id: non_empty_str!("100"),
            address: "123.123.123.123:8080".try_into().unwrap(),
            tcp: true,
            udp: false,
            http: true,
            https: false,
            socks5: false,
            socks5h: false,
            datacenter: true,
            residential: false,
            mobile: false,
            pool_id: None,
            continent: None,
            country: Some("US".into()),
            state: None,
            city: None,
            carrier: None,
            asn: None,
        },
    ])
    .unwrap();

    // Singular mode keeps this small example independent of a connector.
    // The normal client stack uses the default plural mode together with
    // ProxyRoutesConnector.
    let service = ProxyDBLayer::new(Arc::new(db))
        .with_filter_mode(ProxyFilterMode::Default)
        .with_single_proxy(true)
        .into_layer(service_fn(async |req: Request| {
            Ok::<_, Infallible>(req)
        }));

    let req = Request::builder()
        .version(Version::HTTP_3)
        .method("GET")
        .uri("https://example.com")
        .body(Body::empty())
        .unwrap();

    req.extensions().insert(ProxyFilter {
        country: Some(vec!["BE".into()]),
        mobile: Some(true),
        residential: Some(true),
        ..Default::default()
    });

    let output = service.serve(req).await.unwrap();
    let proxy_address = output
        .extensions()
        .get_ref::<ProxyRoute>()
        .and_then(ProxyRoute::proxy_address)
        .unwrap();
    assert_eq!(proxy_address.address.to_string(), "12.34.12.34:8080");
    assert_eq!(output.extensions().get_ref::<Proxy>().unwrap().id, "42");
}

§Single Proxy Router

Another example is a single proxy through which one can connect with config for further downstream proxies passed by username labels.

Note that the username formatter is available for any proxy db, it is not specific to the usage of a single proxy.

use rama_http_types::{Body, Version, Request};
use rama_proxy::{
   Proxy,
   ProxyDBLayer, ProxyFilterMode,
   ProxyFilter,
};
use rama_core::{
   service::service_fn,
   extensions::{ExtensionsRef},
   Service, Layer,
};
use rama_net::client::ProxyRoute;
use rama_utils::str::non_empty_str;
use std::{convert::Infallible, sync::Arc};

#[tokio::main]
async fn main() {
    let proxy = Proxy {
        id: non_empty_str!("1"),
        address: "john:secret@proxy.example.com:60000".try_into().unwrap(),
        tcp: true,
        udp: true,
        http: true,
        https: false,
        socks5: true,
        socks5h: false,
        datacenter: false,
        residential: true,
        mobile: false,
        pool_id: None,
        continent: Some("*".into()),
        country: Some("*".into()),
        state: Some("*".into()),
        city: Some("*".into()),
        carrier: Some("*".into()),
        asn: None,
    };

    let service = ProxyDBLayer::new(Arc::new(proxy))
        .with_filter_mode(ProxyFilterMode::Default)
        .with_single_proxy(true)
        .with_username_formatter(|_proxy: &Proxy, filter: &ProxyFilter, username: &str| {
            use std::fmt::Write;

            let mut output = String::new();

            if let Some(countries) =
                filter.country.as_ref().filter(|t| !t.is_empty())
            {
                _ = write!(output, "country-{}", countries[0]);
            }
            if let Some(states) =
                filter.state.as_ref().filter(|t| !t.is_empty())
            {
                _ = write!(output, "state-{}", states[0]);
            }

            (!output.is_empty()).then(|| format!("{username}-{output}"))
        })
        .into_layer(service_fn(async |req: Request| {
            Ok::<_, Infallible>(req)
        }));

    let req = Request::builder()
        .version(Version::HTTP_3)
        .method("GET")
        .uri("https://example.com")
        .body(Body::empty())
        .unwrap();
    req.extensions().insert(ProxyFilter {
        country: Some(vec!["BE".into()]),
        residential: Some(true),
        ..Default::default()
    });
    let output = service.serve(req).await.unwrap();
    let proxy_address = output
        .extensions()
        .get_ref::<ProxyRoute>()
        .and_then(ProxyRoute::proxy_address)
        .unwrap();
    assert_eq!(
        "socks5://john-country-be:secret@proxy.example.com:60000",
        proxy_address.to_string()
    );
}

Structs§

LiveUpdateProxyDBlive-update: A wrapper around a T ProxyDB which can be updated through the only linked writer LiveUpdateProxyDBSetter.
LiveUpdateProxyDBSetterlive-update: Writer to set a new ProxyDB in the linked LiveUpdateProxyDB.
MemoryProxyDBmemory-db: A fast in-memory ProxyDatabase that is the default choice for Rama.
MemoryProxyDBInsertErrormemory-db: The error type that can be returned by MemoryProxyDB when some of the proxies could not be inserted due to a proxy that had a duplicate key or was invalid for some other reason.
MemoryProxyDBQueryErrormemory-db: The error type that can be returned by MemoryProxyDB when no proxy could be returned.
Proxy: A proxy record returned by a ProxyDB.
ProxyContext: The context as relevant to the proxy layer.
ProxyCsvRowReadercsv: A CSV Reader that can be used to create a Proxy database from a CSV file or raw data.
ProxyCsvRowReaderErrorcsv: An error that can occur when reading a Proxy CSV row.
ProxyDBLayer: A Layer which wraps an inner Service to resolve proxy candidates based on the input extensions and publish route-specific proxy metadata.
ProxyDBService: A Service which resolves proxy candidates from the given input Extensions.
ProxyFilter: Filter to select a specific kind of proxy.
ProxyFilterUsernameParser: A parser which parses ProxyFilters from username labels and adds it to the input Extensions.
ProxyID: ID of a proxy selected for a connection attempt.
StringFilter: A string filter that normalizes the string prior to consumption.

Enums§

MemoryProxyDBInsertErrorKindmemory-db: The kind of error that MemoryProxyDBInsertError represents.
MemoryProxyDBQueryErrorKindmemory-db: The kind of error that MemoryProxyDBQueryError represents.
ProxyCsvRowReaderErrorKindcsv: The kind of error that can occur when reading a Proxy CSV row.
ProxyFilterMode: The modus operandi to decide how to deal with a missing ProxyFilter in the input Extensions when selecting a Proxy from the ProxyDB.

Constants§

DEFAULT_PROXY_DB_MAX_PROXIES: Default maximum number of proxy candidates published for one query.

Traits§

ProxyDB: The trait to implement to provide a proxy database to other facilities, such as connection pools, to provide proxy candidates based on the given ProxyContext and ProxyFilter.
ProxyQueryPredicate: Trait that is used by the ProxyDB for providing an optional filter predicate to rule out returned results.
UsernameFormatter: Formats Basic-auth proxy usernames, for example to add routing labels used by an upstream proxy router.

Functions§

proxy_db_updaterlive-update: Create a new ProxyDB updater which allows you to have a (typically in-memory) ProxyDB which you can update live.

Crate rama_proxy

Crate rama_proxy Copy item path

§DB Live Reloads

§ProxyDB layer

§Example

§Single Proxy Router

Structs§

Enums§

Constants§

Traits§

Functions§

Crate rama_proxy