Abbreviation Extractor

Abbreviation Extractor is a high-performance Rust library with Python bindings for extracting abbreviation-definition pairs from text, particularly focused on biomedical text. It implements an improved version of the Schwartz-Hearst algorithm, offering enhanced accuracy and speed. It's based the original python implementation.

Speed Comparison With Other Abbreviation Extraction Libraries

Extraction Accuracy Comparison With Other Abbreviation Extraction Libraries

Features

• Fast and accurate extraction of abbreviation-definition pairs with tokenization.
• Support for both single-threaded and parallel processing
• Python bindings for easy integration with Python projects
• Customizable extraction parameters like selecting the most common or first definition for each abbreviation

Installation

Rust

Add this to your Cargo.toml:

abbreviation-extractor = "0.1.0"

Python

pip install abbreviation-extractor-rs

Usage

Rust

use abbreviation_extractor::{extract_abbreviation_definition_pairs, AbbreviationOptions};

let text = "The World Health Organization (WHO) is a specialized agency.";
let options = AbbreviationOptions::default();
let result = extract_abbreviation_definition_pairs(text, options);

for pair in result {
    println!("Abbreviation: {}, Definition: {}", pair.abbreviation, pair.definition);
}

Python

from abbreviation_extractor import extract_abbreviation_definition_pairs

text = "The World Health Organization (WHO) is a specialized agency."
result = extract_abbreviation_definition_pairs(text)

for pair in result:
    print(f"Abbreviation: {pair.abbreviation}, Definition: {pair.definition}")

Customizing Extraction

Python

from abbreviation_extractor import extract_abbreviation_definition_pairs

text = "The World Health Organization (WHO) is a specialized agency. The World Heritage Organization (WHO) is different."

# Get only the most common definition for each abbreviation
result = extract_abbreviation_definition_pairs(text, most_common_definition=True)

# Get only the first definition for each abbreviation
result = extract_abbreviation_definition_pairs(text, first_definition=True)

# Disable tokenization (if the input is already tokenized)
result = extract_abbreviation_definition_pairs(text, tokenize=False)

# Combine options
result = extract_abbreviation_definition_pairs(text, most_common_definition=True, tokenize=True)

for pair in result:
    print(f"Abbreviation: {pair.abbreviation}, Definition: {pair.definition}")

Rust

use abbreviation_extractor::{extract_abbreviation_definition_pairs, AbbreviationOptions};

let text = "The World Health Organization (WHO) is a specialized agency. The World Heritage Organization (WHO) is different.";

// Get only the most common definition for each abbreviation
let options = AbbreviationOptions::new(true, false, true);
let result = extract_abbreviation_definition_pairs(text, options);

// Get only the first definition for each abbreviation
let options = AbbreviationOptions::new(false, true, true);
let result = extract_abbreviation_definition_pairs(text, options);

// Disable tokenization (if the input is already tokenized)
let options = AbbreviationOptions::new(false, false, false);
let result = extract_abbreviation_definition_pairs(text, options);

for pair in result {
    println!("Abbreviation: {}, Definition: {}", pair.abbreviation, pair.definition);
}

Benchmark

Below is a comparison of how the abbreviation extractor performs in comparison to other libraries, namely Schwartz-Hearst and ScispaCy in terms of accuracy and speed.

Performance Comparison of Abbreviation Extractor Against Other Libraries

Speed Comparison with Other Abbreviation Extraction Libraries

API Reference

For detailed API documentation, please refer to the Rust docs or the Python module docstrings.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the MIT License.

Acknowledgements

This library is based on the Schwartz-Hearst algorithm:

A. Schwartz and M. Hearst (2003) A Simple Algorithm for Identifying Abbreviations Definitions in Biomedical Text. Biocomputing, 451-462.

The implementation is inspired by the original Python variant by Phil Gooch: abbreviation-extractor