Text span utilities for Rust and Python
- Rust doc: https://docs.rs/textspan
Usage (Python)
Install: pip install pytextspan
align_spans
...
Converts the spans defined in text
to those defined in original_text
.
This is useful, for example, when you want to get the spans in the original text of spans obtained in the normalized text.
>>>
>>> = ;
>>> = ;
>>> = ;
>>>
align_spans_by_mapping
...
Converts the spans by the given mapping
.
Generally speaking, the character correspondence between two texts is not
necessarily surjective, not injective, not even a methematical map -
some character in textA
may not have a correspondence in textB
,
or may have multiple correspondences in textB
. Thus, you should
provide mapping
as List[List[Tuple[int,int]]]
.
>>>
>>> =
>>> =
>>>
get_original_spans
...
Returns the span indices of original_text
from the tokens based on the shortest edit script (SES).
This is useful, for example, when you want to get the spans in the original text of tokens obtained in the normalized text.
>>>
>>> =
>>>
lift_span_index
...
Examples:
import textspan spans = [(0, 3), (3, 4), (4, 9), (9, 12)] assert textspan.lift_spans_index((2, 10), spans) == (0, 4)
lift_spans_index
...
remove_span_overlaps
...
Remove overlapping spans from given spans
.
First, longest spans are remained - if the two spans are overlapped, the first span will be remained. If the two spans are overlapped and their start positions are same, the longer span will be remained.
>>>
>>> =
>>> assert ==
remove_span_overlaps_idx
...
Remove overlapping spans from given spans
, and returns remained span indices.
First, longest spans are remained - if the two spans are overlapped, the first span will be remained. If the two spans are overlapped and their start positions are same, the longer span will be remained.
>>>
>>> =
>>> assert ==