Struct Source

Source

pub struct Source {
    pub url: String,
    pub user_agent: String,
    pub regex_pattern: String,
    pub compiled_regex: Option<SerializableRegex>,
    pub last_used_at: Option<DateTime<Utc>>,
    pub use_count: usize,
    pub failure_count: usize,
    pub last_failure_reason: Option<String>,
    pub last_failure_code: Option<u16>,
    pub parameters: HashMap<String, String>,
    pub proxies_found: usize,
}

Expand description

Represents a source of proxy servers.

A source defines where and how to obtain proxy server information, including the URL to fetch from, the user agent to use in requests, and the regex pattern for extracting proxy information from the response.

The struct also tracks usage statistics such as success rates and failure counts to help evaluate source reliability over time.

§Examples

use gooty_proxy::definitions::source::Source;

let source = Source::new(
    "https://example.com/proxy-list".to_string(),
    "Mozilla/5.0 (compatible; Gooty/1.0)".to_string(),
    r"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{2,5})".to_string(),
).unwrap();

assert_eq!(source.url, "https://example.com/proxy-list");
assert_eq!(source.success_rate(), 0.0); // New source with no usage yet

Fields§

§url: String

The URL of the proxy source.

§user_agent: String

The User-Agent string to use when making requests to the source.

§regex_pattern: String

The regex pattern to use for extracting proxy information from the source.

§compiled_regex: Option<SerializableRegex>

Compiled regex object for performance

§last_used_at: Option<DateTime<Utc>>

When the source was last used

§use_count: usize

Number of times the source has been used

§failure_count: usize

Number of times the source has failed

§last_failure_reason: Option<String>

Last failure reason

§last_failure_code: Option<u16>

Last failure HTTP status code if applicable

§parameters: HashMap<String, String>

Additional parameters for the source

§proxies_found: usize

Number of proxies found from this source

Implementations§

Source §

impl Source

Source

pub fn new( url: String, user_agent: String, regex_pattern: String, ) -> Result<Self, SourceError>

Creates a new proxy source with the required fields.

This constructor validates both the URL and regex pattern to ensure they’re well-formed before creating the source instance.

§Arguments

url - The URL where proxy information can be obtained
user_agent - The User-Agent string to use in HTTP requests
regex_pattern - A regular expression pattern that extracts proxy data from responses

§Returns

A new Source instance if validation succeeds

§Errors

This function will return an error if:

The URL is malformed or invalid
The regex pattern is invalid or cannot be compiled

Source

pub fn add_parameter(&mut self, key: String, value: String)

Adds a parameter to the source configuration.

Parameters will be appended to the source URL as query parameters when making HTTP requests.

§Arguments

key - The parameter name
value - The parameter value

§Examples

source.add_parameter("country".to_string(), "US".to_string());
source.add_parameter("type".to_string(), "https".to_string());

let url = source.get_full_url();
assert!(url.contains("country=US"));
assert!(url.contains("type=https"));

Source

pub fn remove_parameter(&mut self, key: &str) -> Option<String>

Removes a parameter from the source configuration.

§Arguments

key - The name of the parameter to remove

§Returns

The previous value of the parameter if it was set, or None if it wasn’t present

§Examples

source.add_parameter("country".to_string(), "US".to_string());

let value = source.remove_parameter("country");
assert_eq!(value, Some("US".to_string()));

Source

pub fn record_use(&mut self)

Records a successful use of the source.

This method updates usage statistics by incrementing the use count and recording the current time as the last used timestamp.

Source

pub fn record_failure(&mut self, reason: String, status_code: Option<u16>)

Records a failure when using the source.

This method updates failure statistics and records the reason and optional status code for the failure.

§Arguments

reason - A description of why the source failed
status_code - Optional HTTP status code if the failure was related to an HTTP response

Source

pub fn success_rate(&self) -> usize

Returns the success rate of using this source.

The success rate is calculated as the ratio of successful uses to total uses. If the source has never been used, returns 0.0.

§Returns

A float between 0.0 and 1.0 representing the success rate where 1.0 means 100% success.

Source

pub fn update_regex_pattern( &mut self, new_pattern: String, ) -> Result<(), SourceError>

Updates the regex pattern and recompiles it.

This is useful when the pattern needs to be adjusted based on changes to the source format.

§Arguments

new_pattern - The new regex pattern to use

§Returns

Ok(()) if the pattern was valid and updated successfully

§Errors

Returns an error if the new regex pattern is invalid

Source

pub fn validate(&self) -> Result<(), SourceError>

Validates the source configuration.

This method checks that the URL is well-formed and the regex pattern is valid.

§Returns

Ok(()) if the source is valid

§Errors

Returns an error if:

The URL is invalid
The regex pattern is invalid

Source

pub fn get_full_url(&self) -> String

Returns a constructed URL with parameters.

This method takes the base URL and appends any parameters that have been added to the source as query parameters.

§Returns

The complete URL including query parameters

§Examples

source.add_parameter("token".to_string(), "abc123".to_string());

let url = source.get_full_url();
assert_eq!(url, "https://example.com/api?token=abc123");

Source