Module fts 

v7.12.1 — full-text search lexer / stemmer.

Powers to_tsvector, plainto_tsquery, to_tsquery, and friends. Two configs are supported in v7.12:

• simple — lowercase + tokenise; no stopwords, no stemming.
• english — lowercase + tokenise + drop PG-standard english stopwords + Porter v1 stem.

Other configs (spanish, german, russian, …) error with EvalError::TypeMismatch carrying the unsupported-config name so callers see the same shape as ::regtype rejection.

Porter stemmer implementation follows the original 1980 Algorithm; corner-case behaviour matches Snowball english v1 (the variant PG also uses).

Structs

Token

v7.39 (round 651) — a typed token, PG-shaped.

Enums

TokenType

Tokenise on Unicode word boundaries — anything that is not an alphanumeric scalar value (or _) splits the token. Lowercases each emitted token. v7.39 (round 651) — PG’s token types, as ts_token_type('default') publishes them. Only the ones SPG’s parser actually produces are here; the numbering is PG’s so pg_ts_config_map.maptokentype and ts_debug.alias agree with it.

TsConfig

v7.12.1 — supported tokeniser configs.

TsDict

The two dictionaries SPG has (round 650).

Constants

DEFAULT_RANK_WEIGHTS

PG default weights: D=0.1, C=0.2, B=0.4, A=1.0.

Functions

apply_rank_norm

v7.38 (read01, T12.1) — apply PG’s ranking normalization bitmask to a raw rank. Flags are applied in PG’s order over the tsvector’s total position count (len) and distinct-lexeme count (uniq): 1 → /log2(len+1) · 2 → /len · 8 → /uniq · 16 → /log2(uniq+1) · 32 → r/(r+1) (flag 4, the cover-extent distance, is cover-density only and handled by the caller.) Verified against live PG 18.4.

is_english_stopword

PG’s standard english stopword list (tsearch_data/english.stop). Subset of the 127 words in PG 17’s distribution — verbatim.

phraseto_tsquery

v7.12.1 — phraseto_tsquery(config, text): same tokenise + stem, but preserve order — fold into nested phrase nodes whose <N> distance is the position gap between surviving lexemes. Dropped stopwords still advance the position counter (as in to_tsvector), so 'cats and dogs' yields 'cat' <2> 'dog', matching PG.

plainto_tsquery

v7.12.1 — plainto_tsquery(config, text): tokenise + stem, fold the surviving lexemes into an AND tree. Returns EvalError::TypeMismatch only for an unsupported config — an all-stopwords input becomes an empty Term("") so the caller can detect it.

porter_stem

to_tsquery

v7.12.1 — to_tsquery(config, text): explicit operator syntax over already-stemmed terms. Reuses the v7.12.0 external-form parser, then walks each leaf through porter_stem (when the config is english). Returns TypeMismatch on malformed input.

to_tsvector

v7.12.1 — tokenise + (optionally) stem text into a sorted + deduped lexeme set with merged positions. Each token’s position is 1-based and clamped at 16383 (PG18-measured, round 753: a 20k-word document’s positions top out at 16383 — the MAXENTRYPOS - 1 clamp — while every lexeme is still recorded).

tokenize

The old tokenizer, kept for callers that want bare words.

tokenize_typed

v7.39 (round 651) — split text the way PG’s default parser does.

ts_query_matches

v7.12.2 — evaluate tsvector @@ tsquery. Walks the query AST treating each leaf as “does the vector contain this lexeme”. Phrase semantics: the v7.12.2 implementation honours the <N> distance — both operand terms must appear with their positions exactly N apart in the vector. Higher-arity phrase chains nest as Phrase(Phrase(a,b,1), c, 1), so the match recursion folds position sets across the AND of the chain (a fully general n-gram match in a single pass).

ts_rank

v7.12.2 — ts_rank(vec, q) basic form. Score is the sum of per-matched-lexeme weight factors divided by 1 + log(unique terms in query). Matches PG’s ts_rank with default normalisation flag 0.

ts_rank_cd

v7.12.2 — ts_rank_cd(vec, q) cover-density variant. Higher score when matched lexemes cluster closer together; defaults to a per-lexeme contribution divided by the average gap between matched positions. Returns 0 when no terms match.

websearch_to_tsquery

v7.12.1 — websearch_to_tsquery(config, text): Google-style syntax. Quoted phrases → phrase node; OR (case-insensitive) → OR; leading - → NOT; otherwise AND.

Type Aliases

RankWeights

v7.38 (read01, T12.1) — a ts_rank weight array in PG order [D, C, B, A].