Struct Dataset 

pub struct Dataset {
    pub data: Vec<Vec<f32>>,
    pub labels: Vec<u8>,
    pub features: Vec<Bytes>,
    pub ftype: FileType,
}

A dataset contains data for training or inference, training requires labels

Fields

data: Vec<Vec<f32>>

Data used for training a model or calculating predictions

labels: Vec<u8>

Data labels, can be empty if only used for inference

features: Vec<Bytes>

N-gram byte features

ftype: FileType

The type of file represented

Implementations

impl Dataset

pub fn load<P: AsRef<Path>>(path: P) -> Result<Dataset>

Load a file

Errors

An error results if the file type can’t be determined, is incorrectly determined, or if the file isn’t a supported format.

pub fn from_csv_file<P: AsRef<Path>>( path: P, data_length: usize, ) -> Result<Self>

Create a dataset struct from a CSV file

Errors

Returns an error if:

• The file can’t be read
• The data contained isn’t numeric
• Feature data is missing
• The expected amount of data isn’t encountered

pub fn from_csv_file_assume_data_length<P: AsRef<Path>>(path: P) -> Result<Self>

Create a dataset struct from a CSV file

Errors

Returns an error if:

• The file can’t be read
• The data contained isn’t a float
• Feature data is missing
• The amount of columns can’t be determined

pub fn from_csv_string(contents: &str, data_length: usize) -> Result<Self>

Create a dataset struct from a CSV string

Errors

Returns an error if:

• The data contained isn’t numeric
• Feature data is missing
• The expected amount of data isn’t encountered

pub fn from_arff_file<P: AsRef<Path>>(path: P) -> Result<Self>

Create a dataset struct from an ARFF string

Errors

Returns an error if:

• The file can’t be read
• The data contained isn’t numeric
• Feature data is missing
• The expected amount of data isn’t encountered

pub fn from_arff_string(contents: &str) -> Result<Self>

Create a dataset struct from an ARFF string

Errors

Returns an error if:

• The data contained isn’t numeric
• Feature data is missing
• The expected amount of data isn’t encountered

pub fn from_libsvm_file<P: AsRef<Path>>(path: P) -> Result<Self>

Create a dataset struct from a libsvm file

Errors

Returns an error if:

• The file can’t be read
• Feature data is missing
• The data isn’t in the expected format
• The expected amount of data isn’t encountered

pub fn from_libsvm_string(contents: &str) -> Result<Self>

Create a dataset from a libsvm string

Errors

Returns an error if the file doesn’t contain the expected format or is missing features

pub fn create_save_from_benign_malicious_files_and_ngrams<P: AsRef<Path>>( malicious_dir: P, benign_dir: P, ngrams_file: P, output_file: P, ) -> Result<()>

Given paths to malicious files, benign files, and n-grams (features), get a Dataset object.

Errors

This will fail if:

• The directories for benign or malicious files don’t exist or are empty.
• The n-gram feature file doesn’t exist, is empty, or doesn’t have hexidecimal-encoded features

pub fn save_csv<P: AsRef<Path>>(&self, path: P) -> Result<()>

Save a dataset as a CSV

Errors

An error will result if the file can’t be opened for writing

pub fn save_arff<P: AsRef<Path>>(&self, path: P) -> Result<()>

Save a dataset as an ARFF file

Errors

An error will result if the file can’t be opened for writing

pub fn save_libsvm<P: AsRef<Path>>(&self, path: P) -> Result<()>

Save a dataset as a libsvm file

Errors

An error will result if the file can’t be opened for writing

pub fn save<P: AsRef<Path>>(&self, path: P) -> Result<()>

Save the dataset using the file extension to determine data format

Errors

There’s an error if the file can’t be written or if the format can’t be determined

pub fn len(&self) -> usize

Return dataset size

pub fn is_empty(&self) -> bool

Indicate if the dataset is empty

pub fn validate(&self) -> bool

Ensure the dataset is valid

• Same size data columns
• If present, the amount of data rows equals the amount of labels

pub fn shuffle(&mut self)

Shuffle the data, using roughly 10 X log10(size). So 10 records = 10 iterations, 1,000 records gets 30 iterations

pub fn shuffle_iterations(&mut self, iterations: u32)

Shuffle the data with a specified amount of iterations, ensures that the labels are swapped with the data, if present

pub fn split(&mut self, ratio: f32) -> Self

Split the dataset, ideally into train/test datasets. The ratio indicates how much data is kept, the remaining size is shed and returned.

pub fn reduce(&mut self, model: &LogisticRegression) -> Result<Vec<usize>>

The model training allows for the algorithm to not only train a model but determine the features most useful for determining benign vs. malicious. This action removes the features deemed unneeded.

Errors

If the model would remove all features, an error is returned as an empty dataset isn’t useful, and it’s instead likely the modal and dataset weren’t for the same data collection.

pub fn column_iter(&self, index: usize) -> Option<ColumnIterator<'_>>

Returns an iterator over a column

Trait Implementations

impl Clone for Dataset

fn clone(&self) -> Dataset

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Dataset

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for Dataset

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl PartialEq for Dataset

fn eq(&self, other: &Self) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for Dataset

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.