Module textsim 

Deciding whether two pieces of prose make the same point.

Everything spar deduplicates used to compare strings exactly. Two agents describing one defect, or two runs a week apart describing it again, never phrase it identically, so exact matching let duplicates straight through: a follow-up filed twice as two issues, and an issue closed with two comments saying the same thing in different words.

This is deliberately shallow. No stemming, no embeddings, nothing that needs a model. It compares the significant words two texts share, which is enough to catch a rewording and cheap enough to run on every finding.

The thresholds lean toward calling things the same, because the two errors are not symmetric. Treating one defect as two files a duplicate, which is the complaint. Treating two defects as one still records the second, as a comment on the first issue rather than an issue of its own, so nothing is lost and a person can split them.

Functions

adds_information

Whether candidate says anything existing does not.

containment

How much of the smaller text’s vocabulary the larger one already contains, from 0.0 to 1.0.

dedupe

Collapse texts that make the same point, keeping the fullest wording of each.

dedupe_by

Collapse texts under a caller supplied notion of sameness.

references

Issue and pull request numbers a text cites.

same_point

Whether two texts make the same point.

same_reason

Whether two reviewers gave the same reason for declining an issue.

same_subject

shared

Words the two texts share. Used to insist on real overlap rather than one lucky word.

strip_provenance

Strip the provenance line spar stamps onto every follow-up it files.

tokens

Significant words, lowercased. Punctuation goes, short words go, and words that appear in every bug report go.