Struct TextData

pub struct TextData { /* private fields */ }

A structure representing compiled text/matching data.

This is the key structure used to compare two texts against one another. It handles pre-processing the text to n-grams, scoring, and optimizing the result to try to identify specific details about a match.

Examples

Basic scoring of two texts:

use scancode_rust::askalono::TextData;

let license = TextData::from("My First License");
let sample = TextData::from("copyright 20xx me irl\n\n //  my   first license");
assert_eq!(sample.match_score(&license), 1.0);

The above example is a perfect match, as identifiable copyright statements are stripped out during pre-processing.

Building on that, TextData is able to tell you where in the text a license is located:

let sample = TextData::from("copyright 20xx me irl\n// My First License\nfn hello() {\n ...");
let (optimized, score) = sample.optimize_bounds(&license);
assert_eq!((1, 2), optimized.lines_view());
assert!(score > 0.99f32, "license within text matches");

Implementations

impl TextData

pub fn new(text: &str) -> TextData

Create a new TextData structure from a string.

The given text will be normalized, then smashed down into n-grams for matching. By default, the normalized text is stored inside the structure for future diagnostics. This is necessary for optimizing a match and for diffing against other texts. If you don’t want this extra data, you can call without_text throw it out. Generally, as a user of this library you want to keep the text data, but askalono will throw it away in its own Store as it’s not needed.

pub fn without_text(self) -> Self

Consume this TextData, returning one without normalized/processed text stored.

Unless you know you don’t want the text, you probably don’t want to use this. Other methods on TextData require that text is present.

pub fn lines_view(&self) -> (usize, usize)

Get the bounds of the active line view.

This represents the “active” region of lines that matches are generated from. The bounds are a 0-indexed (start, end) tuple, with inclusive start and exclusive end indicies. See optimize_bounds.

This is largely for informational purposes; other methods in TextView, such as lines and match_score, will already account for the line range. However, it’s useful to call it after running optimize_bounds to discover where the input text was discovered.

pub fn with_view(&self, start: usize, end: usize) -> Self

Clone this TextView, creating a copy with the given view.

This will re-generate match data for the given view. It’s used in optimize_bounds to shrink/expand the view of the text to discover bounds.

Other methods on TextView respect this boundary, so it’s not needed outside this struct.

pub fn white_out(&self) -> Self

“Erase” the current lines in view and restore the view to its original bounds.

For example, consider a file with two licenses in it. One was identified (and located) with optimize_bounds. Now you want to find the other: white-out the matched lines, and re-run the overall search to find a new high score.

pub fn lines(&self) -> &[String]

Get a slice of the normalized lines in this TextData.

pub fn match_score(&self, other: &TextData) -> f32

Compare this TextData with another, returning a similarity score.

This is what’s used during analysis to rank licenses.

pub fn optimize_bounds(&self, other: &TextData) -> (Self, f32)

Attempt to optimize a known match to locate possible line ranges.

Returns a new TextData struct and a score. The returned struct is a clone of self, with its view set to the best match against other.

This will respect any views set on the TextData (an optimized result won’t go outside the original view).

Note that this won’t be 100% optimal if there are blank lines surrounding the actual match, since successive blank lines in a range will likely have the same score.

You should check the value of lines_view on the returned struct to find the line ranges.

Trait Implementations

impl Clone for TextData

fn clone(&self) -> TextData

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TextData

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for TextData

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl<'a> From<&'a str> for TextData

fn from(text: &'a str) -> Self

Converts to this type from the input type.

impl From<String> for TextData

fn from(text: String) -> Self

Converts to this type from the input type.

impl Serialize for TextData

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.