representation.numeric.integer_number:
title: "Integer Number"
description: >
Whole number without fractional part. String representation of an integer.
Transforms to BIGINT or INTEGER depending on value range.
designation: universal
locales: [UNIVERSAL]
broad_type: BIGINT
frictionless:
type: integer
format_string: null
transform: "CAST({col} AS BIGINT)"
transform_ext: null
decompose: null
validation:
type: string
pattern: "^-?[0-9]+$"
tier: [BIGINT, numeric]
release_priority: 5
aliases: [integer, whole_number]
samples:
- "42"
- "-17"
- "0"
- "999999999"
references: null
notes: >
v1 migration: Was numeric.integer_number. Converts to BIGINT.
Includes negative numbers. Pattern is simple integer format.
representation.numeric.decimal_number:
title: "Decimal Number"
description: >
Number with decimal point (e.g., 3.14, -0.5, 1000.001), optionally with
scientific-notation suffix (e.g., 6e-04, 1.5E+10). String representation
of a floating-point number. Transforms to DOUBLE.
designation: universal
locales: [UNIVERSAL]
broad_type: DOUBLE
frictionless:
type: number
format_string: null
transform: "CAST({col} AS DOUBLE)"
transform_ext: null
decompose:
integer_part: "CAST(FLOOR(CAST({col} AS DOUBLE)) AS BIGINT)"
fractional_part: "CAST((CAST({col} AS DOUBLE) - FLOOR(CAST({col} AS DOUBLE))) * 1000000 AS BIGINT)"
validation:
type: string
pattern: "^-?[0-9]+(\\.[0-9]+)?([eE][+-]?[0-9]+)?$"
tier: [DOUBLE, numeric]
release_priority: 5
aliases: [decimal, float, floating_point]
samples:
- "3.14"
- "-0.5"
- "1000.001"
- "0.0001"
- "6e-04"
- "1.5E+10"
references: null
notes: >
v1 migration: Was numeric.decimal_number. Transforms to DOUBLE.
Decompose extracts integer and fractional parts.
v0.6.20: Validator widened to accept scientific-notation suffix
(precision corpus iter 1, ac-10). The dedicated
representation.numeric.scientific_notation type still exists and
remains the preferred classification when a whole column is in
scientific form; this widening only ensures a mostly-decimal column
with some scientific-notation values doesn't get misclassified out of
decimal_number on a single failing row.
representation.numeric.decimal_number_comma:
title: "Decimal Number (Comma Separator)"
description: >
Number with comma as decimal separator and period as thousands separator
(e.g., 1.234,56, 999,99, -42,5). The inverse of US format where period
is decimal and comma is thousands. Silent corruption risk: "1.234" is
either 1234 (comma-decimal thousands) or 1.234 (US decimal).
Transforms to DOUBLE after swapping separators.
designation: universal
locales: [UNIVERSAL]
broad_type: DOUBLE
frictionless:
type: number
format_string: null
transform: "CAST(REPLACE(REPLACE({col}, '.', ''), ',', '.') AS DOUBLE)"
transform_ext: null
decompose:
integer_part: "CAST(FLOOR(CAST(REPLACE(REPLACE({col}, '.', ''), ',', '.') AS DOUBLE)) AS BIGINT)"
fractional_part: "CAST((CAST(REPLACE(REPLACE({col}, '.', ''), ',', '.') AS DOUBLE) - FLOOR(CAST(REPLACE(REPLACE({col}, '.', ''), ',', '.') AS DOUBLE))) * 1000000 AS BIGINT)"
validation:
type: string
pattern: "^-?[0-9]{1,3}(\\.[0-9]{3})*(,[0-9]+)?$|^-?[0-9]+,[0-9]+$"
tier: [DOUBLE, numeric]
release_priority: 3
aliases: [eu_decimal, european_number, decimal_eu]
samples:
- "1.234,56"
- "999,99"
- "1.000.000,00"
- "-42,5"
- "0,75"
- "12.345,678"
- "100,00"
- "-1.234,00"
references: null
notes: >
New in v0.5.1. European decimal format uses period for thousands
and comma for decimal — the exact inverse of US format. Transform removes
period thousands separators, replaces comma with period, then casts to DOUBLE.
Column-level detection is important since single values like "1.234" are
ambiguous between EU (=1234) and US (=1.234). The Sense model's locale
awareness may help disambiguation.
representation.numeric.scientific_notation:
title: "Scientific Notation"
description: >
Number in scientific notation (e.g., 1.23e-4, 5E+6, 3.14e10).
Transforms to DOUBLE.
designation: universal
locales: [UNIVERSAL]
broad_type: DOUBLE
frictionless:
type: number
format_string: null
transform: "CAST({col} AS DOUBLE)"
transform_ext: null
decompose:
mantissa: "CAST(REGEXP_EXTRACT({col}, '^([0-9.+-]+)e') AS DOUBLE)"
exponent: "CAST(REGEXP_EXTRACT({col}, 'e([+-]?[0-9]+)$') AS SMALLINT)"
validation:
type: string
pattern: "^-?[0-9]+(\\.[0-9]+)?[eE][+-]?[0-9]+$"
tier: [DOUBLE, numeric]
release_priority: 3
aliases: [scientific, exponential]
samples:
- "1.23e-4"
- "5E+6"
- "3.14e10"
- "-2.5e-3"
references: null
notes: >
v1 migration: New for v2. Covers scientific notation like 1.23e-4.
Decompose extracts mantissa and exponent.
representation.numeric.percentage:
title: "Percentage"
description: >
Number as a percentage (e.g., 50%, 99.99%, 0.5%).
May include % symbol or decimal representation. Transforms to DOUBLE (0-100).
designation: universal
locales: [UNIVERSAL]
broad_type: DOUBLE
frictionless:
type: number
format_string: null
transform: "CAST(REGEXP_EXTRACT({col}, '^(-?[0-9]+(\\.[0-9]+)?)') AS DOUBLE)"
transform_ext: null
decompose:
percent_value: "CAST(REGEXP_EXTRACT({col}, '^(-?[0-9]+(\\.[0-9]+)?)') AS DOUBLE)"
has_symbol: "CASE WHEN {col} LIKE '%\"%' THEN TRUE ELSE FALSE END"
validation:
type: string
pattern: "^-?[0-9]+(\\.[0-9]+)?%?$"
tier: [DOUBLE, numeric]
release_priority: 3
aliases: [percent]
samples:
- "50%"
- "99.99%"
- "0.5%"
- "100"
references: null
notes: >
v1 migration: New for v2. Percentage as string representation.
Transform extracts numeric value (0-100 range expected).
representation.numeric.si_number:
title: "SI-Prefix Number"
description: >
Human-readable number with SI/business notation suffix indicating
magnitude. Common in dashboards, reports, and financial summaries.
K=thousands, M=millions, B=billions, T=trillions. May include
optional currency prefix ($, €, £) or sign (+/-).
designation: universal
locales: [UNIVERSAL]
broad_type: DOUBLE
frictionless:
type: number
format_string: null
transform: >
CASE
WHEN regexp_extract(regexp_replace({col}, '^[\\$€£+-]', ''), '[KkMmBbTt]$') IN ('K','k')
THEN CAST(regexp_replace(regexp_replace({col}, '^[\\$€£+-]', ''), '[Kk]$', '') AS DOUBLE) * 1000
WHEN regexp_extract(regexp_replace({col}, '^[\\$€£+-]', ''), '[KkMmBbTt]$') IN ('M','m')
THEN CAST(regexp_replace(regexp_replace({col}, '^[\\$€£+-]', ''), '[Mm]$', '') AS DOUBLE) * 1000000
WHEN regexp_extract(regexp_replace({col}, '^[\\$€£+-]', ''), '[KkMmBbTt]$') IN ('B','b')
THEN CAST(regexp_replace(regexp_replace({col}, '^[\\$€£+-]', ''), '[Bb]$', '') AS DOUBLE) * 1000000000
WHEN regexp_extract(regexp_replace({col}, '^[\\$€£+-]', ''), '[KkMmBbTt]$') IN ('T','t')
THEN CAST(regexp_replace(regexp_replace({col}, '^[\\$€£+-]', ''), '[Tt]$', '') AS DOUBLE) * 1000000000000
END
transform_ext: null
decompose: null
validation:
type: string
pattern: "^[\\$€£+-]?\\d+\\.?\\d*[KkMmBbTt]$"
tier: [DOUBLE, numeric]
release_priority: 3
aliases: [business_notation, human_readable_number, abbreviated_number]
samples:
- "12.2K"
- "1.5M"
- "2.3B"
- "$500K"
- "-1.2M"
references: null
notes: >
Inspired by python-ballpark library "business notation." Extremely common
in web dashboards, financial reports, and social media metrics (e.g.,
"1.2M followers"). The pattern requires a trailing suffix letter to
distinguish from plain numbers.
representation.text.plain_text:
title: "Plain Text"
description: >
Unstructured text content (sentences, paragraphs, descriptions).
Broad categorization.
designation: broad_characters
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "CAST({col} AS VARCHAR)"
transform_ext: null
decompose:
word_count: "CAST(REGEXP_COUNT({col}, '\\s+') + 1 AS SMALLINT)"
char_count: "LENGTH({col})"
validation:
type: string
minLength: 1
maxLength: 65536
tier: [VARCHAR, text]
release_priority: 0
aliases: [text, description, content]
samples:
- "This is a plain text description"
- "A longer paragraph with multiple sentences"
- "Single word"
references: null
notes: >
v1 migration: Was text.text marked broad_characters. Very broad category
with low release priority. Distinguishing from other text types (sentence,
word, etc.) requires statistical analysis.
representation.text.word:
title: "Single Word"
description: >
Single word without spaces (alphanumeric or with common separators like hyphen).
designation: universal
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "LOWER(CAST({col} AS VARCHAR))"
transform_ext: null
decompose: null
validation:
type: string
pattern: "^[a-zA-Z0-9]([a-zA-Z0-9\\-_]*[a-zA-Z0-9])?$"
minLength: 1
maxLength: 100
tier: [VARCHAR, text]
release_priority: 4
aliases: null
samples:
- "hello"
- "world"
- "hello-world"
- "test_123"
references: null
notes: >
v1 migration: Was text.word. Single word without spaces.
Allows hyphens and underscores in middle. Transform lowercases.
representation.format.color_hex:
title: "Color (Hexadecimal)"
description: >
Color as hexadecimal value (e.g., #FF0000, #f00, FF0000).
May include # prefix. Short form (3 chars) or long form (6 chars).
designation: universal
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "UPPER(CAST({col} AS VARCHAR))"
transform_ext: null
decompose:
red: "CAST(CONV(REGEXP_EXTRACT(UPPER({col}), '[A-F0-9]{6}|[A-F0-9]{3}'), 16, 10) AS INTEGER)"
green: "CAST(CONV(REGEXP_EXTRACT(UPPER({col}), '[A-F0-9]{6}|[A-F0-9]{3}'), 16, 10) AS INTEGER)"
blue: "CAST(CONV(REGEXP_EXTRACT(UPPER({col}), '[A-F0-9]{6}|[A-F0-9]{3}'), 16, 10) AS INTEGER)"
validation:
type: string
pattern: "^#?[0-9a-fA-F]{3}([0-9a-fA-F]{3})?$"
tier: [VARCHAR, text]
release_priority: 4
aliases: [hex_color]
samples:
- "#FF0000"
- "#f00"
- "0000FF"
- "#00FF00"
references: null
notes: >
v1 migration: Was text.hex_color. Transform uppercases.
Accepts # prefix or without. 3-digit (short) or 6-digit (long) hex.
representation.format.color_rgb:
title: "Color (RGB)"
description: >
Color as RGB tuple (0-255 for each channel).
Format: "rgb(255,0,0)", "255,0,0", "(255, 0, 0)", or similar.
designation: universal
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "CAST({col} AS VARCHAR)"
transform_ext: null
decompose:
red: "CAST(REGEXP_EXTRACT({col}, '^(?:rgb)?\\(?([0-9]{1,3})') AS SMALLINT)"
green: "CAST(REGEXP_EXTRACT({col}, ',\\s*([0-9]{1,3})') AS SMALLINT)"
blue: "CAST(REGEXP_EXTRACT({col}, ',\\s*([0-9]{1,3})\\)?$') AS SMALLINT)"
validation:
type: string
pattern: "^(?:rgb)?\\(?([0-9]{1,3}),\\s*([0-9]{1,3}),\\s*([0-9]{1,3})\\)?$"
tier: [VARCHAR, text]
release_priority: 3
aliases: [rgb]
samples:
- "rgb(255, 0, 0)"
- "255, 0, 0"
- "(128, 128, 128)"
- "rgb(0,255,0)"
references: null
notes: >
v1 migration: Was text.rgb_color. Format flexible to handle
"rgb(...)", "(r,g,b)", or "r,g,b" variants.
representation.format.color_hsl:
title: "Color (HSL)"
description: >
Color in HSL (Hue, Saturation, Lightness) or HSLA (with alpha) CSS notation.
Hue is 0-360 degrees, saturation and lightness are percentages (0-100%),
optional alpha is 0-1 decimal. CSS3/CSS4 standard colour representation.
designation: universal
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "CAST({col} AS VARCHAR)"
transform_ext: null
decompose:
hue: "CAST(REGEXP_EXTRACT({col}, 'hsla?\\(\\s*(\\d+)') AS SMALLINT)"
saturation: "CAST(REGEXP_EXTRACT({col}, ',\\s*(\\d+)%') AS SMALLINT)"
lightness: "CAST(REGEXP_EXTRACT({col}, ',\\s*\\d+%\\s*,\\s*(\\d+)%|%\\s*,\\s*(\\d+)%') AS SMALLINT)"
validation:
type: string
pattern: "^hsla?\\(\\s*\\d{1,3}\\s*,\\s*\\d{1,3}%\\s*,\\s*\\d{1,3}%\\s*(,\\s*[\\d.]+\\s*)?\\)$"
tier: [VARCHAR, text]
release_priority: 3
aliases: [hsl_color, hsla_color]
samples:
- "hsl(120, 100%, 50%)"
- "hsl(240, 100%, 50%)"
- "hsl(0, 100%, 50%)"
- "hsla(240, 100%, 50%, 0.5)"
- "hsl(60, 75%, 60%)"
- "hsla(300, 50%, 50%, 0.8)"
references: null
notes: >
New in taxonomy expansion. HSL completes the CSS colour
representation family alongside hex and RGB. HSL is arguably more intuitive
for designers (hue = colour wheel position, saturation = intensity,
lightness = brightness). HSLA adds an alpha transparency channel.
representation.text.emoji:
title: "Emoji"
description: >
Single emoji character or emoji sequence.
designation: universal
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "CAST({col} AS VARCHAR)"
transform_ext: null
decompose: null
validation:
type: string
pattern: "^[\\p{So}\\p{Sk}\\p{Mn}\\p{Cf}]+$"
tier: [VARCHAR, text]
release_priority: 3
aliases: null
samples:
- "😀"
- "🎉"
- "❤️"
- "🚀"
references: null
notes: >
v1 migration: Was text.emoji. Single emoji or emoji sequence.
Pattern matches Unicode symbol categories. Emoji width variable.
representation.text.entity_name:
title: "Entity Name"
description: >
Non-person named entity: organisation names, product names, venue names,
song/movie/book titles, brand names. Distinguished from full_name by
containing business suffixes (Inc, Corp, Ltd), numbers, ampersands, or
other non-personal-name patterns.
designation: broad_words
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "CAST({col} AS VARCHAR)"
transform_ext: null
decompose: null
validation:
type: string
minLength: 2
maxLength: 200
tier: [VARCHAR, text]
release_priority: 3
aliases: [company_name, brand, product_name, venue_name, title]
samples:
- "The Olive Garden"
- "iPhone 15 Pro"
- "Game of Thrones"
- "Spotify"
- "McKinsey & Company"
- "Toyota Motor Corp"
- "Harvard University"
- "Super Bowl LVIII"
references: null
notes: >
Addresses full_name overcall: 3,086 SOTAB columns misclassified
as person names are actually entity names (songs, restaurants, products,
venues). CharCNN can learn to distinguish entity_name from full_name via
business suffixes, numbers, and mixed-case patterns. broad_words designation
means column-level disambiguation can override with header hints.
representation.file.extension:
title: "File Extension"
description: >
File name extension or suffix (txt, pdf, docx, jpg, etc.).
May include or exclude leading dot.
designation: broad_words
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "LOWER(REGEXP_REPLACE({col}, '^\\.*', ''))"
transform_ext: null
decompose:
category: "CASE WHEN {col} IN ('txt', 'doc', 'docx', 'pdf', 'rtf') THEN 'document' WHEN {col} IN ('jpg', 'jpeg', 'png', 'gif', 'bmp', 'svg') THEN 'image' WHEN {col} IN ('mp4', 'avi', 'mov', 'mkv', 'webm') THEN 'video' WHEN {col} IN ('mp3', 'wav', 'flac', 'aac', 'm4a') THEN 'audio' WHEN {col} IN ('zip', 'rar', '7z', 'tar', 'gz') THEN 'archive' ELSE 'other' END"
validation:
type: string
pattern: "^\\.?[a-zA-Z0-9]{1,10}$"
tier: [VARCHAR, file]
release_priority: 3
aliases: [file_type]
samples:
- "txt"
- ".pdf"
- "docx"
- "jpg"
- "xlsx"
references: null
notes: >
v1 migration: Was file.extension. Transform lowercases and removes
leading dot for canonical form. Decompose infers file category (document,
image, video, audio, archive).
representation.file.mime_type:
title: "MIME Type"
description: >
MIME type (media type) for files and HTTP content
(e.g., text/plain, application/json, image/png).
designation: broad_characters
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "LOWER(CAST({col} AS VARCHAR))"
transform_ext: null
decompose:
type: "REGEXP_EXTRACT({col}, '^([^/]+)')"
subtype: "REGEXP_EXTRACT({col}, '/([^;]+)')"
charset: "REGEXP_EXTRACT({col}, 'charset=([^;]+)')"
validation:
type: string
pattern: "^[a-zA-Z]+/[a-zA-Z0-9.+\\-]+(;.*)?$"
tier: [VARCHAR, file]
release_priority: 3
aliases: [content_type, media_type]
samples:
- "text/plain"
- "application/json"
- "image/png"
- "text/html; charset=utf-8"
- "application/vnd.ms-excel"
references: null
notes: >
v1 migration: Was file.mime_type marked broad_characters.
Standard format: type/subtype[+suffix];param=value.
Decompose extracts type, subtype, charset.
representation.file.file_size:
title: "File Size"
description: >
File size in bytes, kilobytes, megabytes, etc. (e.g., 1024, 1.5MB, 2 GB).
Transforms to BIGINT (bytes).
designation: universal
locales: [UNIVERSAL]
broad_type: BIGINT
frictionless:
type: integer
format_string: null
transform: >
CASE
WHEN UPPER({col}) LIKE '%GB%' THEN CAST(REGEXP_EXTRACT({col}, '^([0-9.]+)') AS DOUBLE) * 1024 * 1024 * 1024
WHEN UPPER({col}) LIKE '%MB%' THEN CAST(REGEXP_EXTRACT({col}, '^([0-9.]+)') AS DOUBLE) * 1024 * 1024
WHEN UPPER({col}) LIKE '%KB%' THEN CAST(REGEXP_EXTRACT({col}, '^([0-9.]+)') AS DOUBLE) * 1024
WHEN UPPER({col}) LIKE '%B%' THEN CAST(REGEXP_EXTRACT({col}, '^([0-9.]+)') AS DOUBLE)
ELSE CAST({col} AS BIGINT)
END
transform_ext: null
decompose:
size: "CAST(REGEXP_EXTRACT({col}, '^([0-9.]+)') AS DOUBLE)"
unit: "UPPER(COALESCE(REGEXP_EXTRACT({col}, '([A-Z]+)$'), 'B'))"
validation:
type: string
pattern: "^[0-9]+(\\.[0-9]+)?\\s*(B|KB|MB|GB|TB)?$"
tier: [BIGINT, file]
release_priority: 3
aliases: null
samples:
- "1024"
- "1.5 MB"
- "2 GB"
- "512 KB"
references: null
notes: >
v1 migration: Was file.size. Transform converts to bytes.
Handles B, KB, MB, GB, TB. Decompose extracts size value and unit.
representation.file.excel_format:
title: "Excel Number Format String"
description: >
Spreadsheet custom number format code that defines how values are
displayed. Found in metadata, headers, or as literal strings in
spreadsheet data exports. Uses format tokens: # (optional digit),
0 (required digit), comma grouping, percent, date/time codes
(yyyy, mm, dd, h, m, s), and optional multi-section syntax
(positive;negative;zero;text).
designation: universal
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "CAST({col} AS VARCHAR)"
transform_ext: null
decompose: null
validation:
type: string
pattern: "^[#0.,;\\[\\]$€£¥%EeAaPpMm/dDyYhHsS ?:\"\\-+()<>=@*!\\w]*$"
minLength: 1
maxLength: 100
tier: [VARCHAR, file]
release_priority: 3
aliases: [number_format, cell_format, format_code]
samples:
- "#,##0.00"
- "$#,##0.00"
- "0.00%"
- "mm/dd/yyyy"
- "h:mm:ss AM/PM"
- "0.00E+00"
references: null
notes: >
Common in GitTables data where format strings appear as column metadata.
Not a data value itself but a format specification. VARCHAR passthrough
since the format string should be preserved as-is.
representation.scientific.dna_sequence:
title: "DNA Sequence"
description: >
Deoxyribonucleic acid sequence in IUPAC notation
(letters: A, T, G, C, and ambiguity codes).
designation: universal
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "UPPER(CAST({col} AS VARCHAR))"
transform_ext: null
decompose:
length: "LENGTH({col})"
gc_content: "CAST(REGEXP_COUNT({col}, '[GC]') AS DOUBLE) / LENGTH({col})"
validation:
type: string
pattern: "^[ATGCRYSWKMBDHVN]+$"
tier: [VARCHAR, scientific]
release_priority: 3
aliases: [dna]
samples:
- "ATGCAGC"
- "GCTAGCTAGCTAG"
- "ATGATGATG"
references: null
notes: >
v1 migration: Was science.dna_sequence. IUPAC notation:
A=Adenine, T=Thymine, G=Guanine, C=Cytosine
Plus ambiguity codes: R, Y, S, W, K, M, B, D, H, V, N.
Transform uppercases. Decompose calculates GC content.
representation.scientific.rna_sequence:
title: "RNA Sequence"
description: >
Ribonucleic acid sequence in IUPAC notation
(letters: A, U, G, C, and ambiguity codes).
designation: universal
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "UPPER(CAST({col} AS VARCHAR))"
transform_ext: null
decompose:
length: "LENGTH({col})"
gc_content: "CAST(REGEXP_COUNT({col}, '[GC]') AS DOUBLE) / LENGTH({col})"
validation:
type: string
pattern: "^[AUGCRYSWKMBDHVN]+$"
tier: [VARCHAR, scientific]
release_priority: 3
aliases: [rna]
samples:
- "AUGCAGC"
- "GCUAGCUAGCUAG"
- "AUGAUGAUG"
references: null
notes: >
v1 migration: Was science.rna_sequence. Like DNA but uses U instead of T.
IUPAC notation: A=Adenine, U=Uracil, G=Guanine, C=Cytosine
Plus ambiguity codes. Transform uppercases. Decompose calculates GC content.
representation.scientific.protein_sequence:
title: "Protein Sequence"
description: >
Amino acid sequence in single-letter IUPAC notation
(20 standard amino acids plus ambiguity codes).
designation: universal
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "UPPER(CAST({col} AS VARCHAR))"
transform_ext: null
decompose:
length: "LENGTH({col})"
molecular_weight_estimate: "LENGTH({col}) * 110"
validation:
type: string
pattern: "^[ACDEFGHIKLMNPQRSTVWXY*]+$"
tier: [VARCHAR, scientific]
release_priority: 3
aliases: [protein, peptide, amino_acid_sequence]
samples:
- "MKVLLIVGS"
- "ACDEFGHIKLMNPQRSTVWYFL"
- "MPKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQAPILSRVG"
references: null
notes: >
v1 migration: New for v2. Single-letter amino acid notation.
Standard: A, C, D, E, F, G, H, I, K, L, M, N, P, Q, R, S, T, V, W, Y
Ambiguity: B (D or N), Z (E or Q), X (any), * (stop codon)
representation.scientific.measurement_unit:
title: "Measurement Unit"
description: >
Unit of measurement (meter, kilogram, second, Joule, etc.).
May be written as symbol (m, kg, s) or full name (meter, kilogram).
Broad categorization.
designation: broad_words
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "LOWER(CAST({col} AS VARCHAR))"
transform_ext: null
decompose: null
validation:
type: string
enum:
- meter
- kilogram
- second
- ampere
- kelvin
- mole
- candela
- hertz
- newton
- joule
- watt
- pascal
- degree_celsius
- liter
- gram
- m
- kg
- s
- A
- K
- mol
- cd
- Hz
- N
- J
- W
- Pa
- "°C"
- L
- g
tier: [VARCHAR, scientific]
release_priority: 3
aliases: [unit]
samples:
- "meter"
- "kilogram"
- "second"
- "m"
- "kg"
- "Joule"
- "J"
references: null
notes: >
v1 migration: Was science.measure_unit marked broad_words.
SI base units and common derived units. Supports both symbols
and full names.
representation.scientific.cas_number:
title: "CAS Registry Number"
description: >
Chemical Abstracts Service registry number. Unique numeric identifier for
chemical substances. Format: 2-7 digits, hyphen, 2 digits, hyphen, 1 check
digit. The check digit is the weighted sum of all preceding digits mod 10,
where the rightmost digit before the check has weight 1, next has weight 2, etc.
designation: universal
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "CAST({col} AS VARCHAR)"
transform_ext: null
decompose:
part1: "REGEXP_EXTRACT({col}, '^(\\d+)-')"
part2: "REGEXP_EXTRACT({col}, '-(\\d{2})-')"
check_digit: "REGEXP_EXTRACT({col}, '-(\\d)$')"
validation:
type: string
pattern: "^\\d{2,7}-\\d{2}-\\d$"
tier: [VARCHAR, scientific]
release_priority: 3
aliases: [cas_rn, cas_id, chemical_id]
samples:
- "7732-18-5"
- "64-17-5"
- "50-78-2"
- "7647-14-5"
- "67-56-1"
- "71-43-2"
references: null
notes: >
New in taxonomy expansion. CAS numbers are the global standard
for chemical substance identification. Over 200 million substances registered.
Check digit algorithm: number all digits right-to-left (excluding check),
multiply each by its position, sum, mod 10. Water=7732-18-5, Ethanol=64-17-5.
representation.scientific.inchi:
title: "InChI Identifier"
description: >
IUPAC International Chemical Identifier. Machine-readable string that
encodes molecular structure. Always starts with "InChI=1S/" (standard)
or "InChI=1/" (non-standard). Layers encode formula, connections,
hydrogen, charge, stereochemistry.
designation: universal
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "CAST({col} AS VARCHAR)"
transform_ext: null
decompose:
version: "REGEXP_EXTRACT({col}, 'InChI=(1S?)')"
formula: "REGEXP_EXTRACT({col}, 'InChI=1S?/([^/]+)')"
validation:
type: string
pattern: "^InChI=1S?/.+$"
tier: [VARCHAR, scientific]
release_priority: 3
aliases: [international_chemical_identifier]
samples:
- "InChI=1S/H2O/h1H2"
- "InChI=1S/C2H6O/c1-2-3/h3H,2H2,1H3"
- "InChI=1S/CH4/h1H4"
- "InChI=1S/C6H6/c1-2-4-6-5-3-1/h1-6H"
- "InChI=1S/CO2/c2-1-3"
references: null
notes: >
New in taxonomy expansion. InChI is the IUPAC standard for
machine-readable chemical identifiers. The "1S" prefix means standard InChI
version 1. The formula layer (first after version) gives the molecular formula.
Widely used in cheminformatics databases (PubChem, ChEBI, ChemSpider).
representation.scientific.smiles:
title: "SMILES Notation"
description: >
Simplified Molecular-Input Line-Entry System. Compact linear notation for
describing chemical structures using ASCII characters. Atoms represented by
element symbols, bonds by special characters. Widely used in cheminformatics
and drug discovery databases.
designation: broad_characters
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "CAST({col} AS VARCHAR)"
transform_ext: null
decompose: null
validation:
type: string
pattern: "^[A-Za-z0-9@+\\-\\[\\]\\(\\)\\\\/#=%.:]+$"
minLength: 1
maxLength: 1000
tier: [VARCHAR, scientific]
release_priority: 3
aliases: [smiles_string, molecular_notation]
samples:
- "O"
- "CCO"
- "CC(=O)Oc1ccccc1C(=O)O"
- "c1ccccc1"
- "CC(=O)O"
- "C(=O)(N)N"
references: null
notes: >
New in taxonomy expansion. SMILES is extremely common in
chemical and pharmaceutical datasets. The broad_characters designation
reflects that single values like "O" or "CC" are indistinguishable from
plain text — column-level disambiguation (header hints) is essential.
Common molecules: O=water, CCO=ethanol, c1ccccc1=benzene, CC(=O)O=acetic acid.
representation.discrete.ordinal:
title: "Ordinal Value"
description: >
Discrete value from an ordered set with ranking semantics.
Examples: grades ("A"/"B"/"C"/"D"/"F"), priority levels ("low"/"medium"/"high"),
star ratings (1-5), class rankings (1st/2nd/3rd).
Distinguished from categorical by implicit ordering.
Transforms to VARCHAR (ordering is semantic, not numeric).
designation: universal
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "CAST({col} AS VARCHAR)"
transform_ext: null
decompose: null
validation:
type: string
minLength: 1
maxLength: 30
tier: [VARCHAR, discrete]
release_priority: 3
aliases: [rank, grade, rating, tier]
samples:
- "high"
- "A"
- "1st"
- "★★★"
- "3"
- "excellent"
references: null
notes: >
Ordinal values have natural ordering but may not be
numeric. Common in survey data, grading systems, and priority fields.
At the single-value level, hard to distinguish from categorical or integer.
Column-level analysis needed for reliable detection.
representation.boolean.binary:
title: "Binary Boolean (0/1)"
description: >
Boolean value represented as a binary digit: "0" for false, "1" for true.
Common in CSV exports, database dumps, and flag columns.
Transforms to DuckDB BOOLEAN type.
designation: universal
locales: [UNIVERSAL]
broad_type: BOOLEAN
frictionless:
type: boolean
format_string: null
transform: "CAST({col} AS BOOLEAN)"
transform_ext: null
decompose: null
validation:
type: string
enum: ["0", "1"]
tier: [BOOLEAN, binary]
release_priority: 5
aliases: [binary_bool, bit]
samples:
- "0"
- "1"
references: null
notes: >
The simplest boolean format — just 0 and 1. The model must
distinguish this from single-digit integers in count/ordinal columns
(handled by column-level disambiguation rules).
representation.boolean.initials:
title: "Boolean Initials (T/F, Y/N)"
description: >
Boolean value represented as a single initial character:
T/F (true/false) or Y/N (yes/no), in any case.
Transforms to DuckDB BOOLEAN type.
designation: universal
locales: [UNIVERSAL]
broad_type: BOOLEAN
frictionless:
type: boolean
format_string: null
transform: "CAST({col} AS BOOLEAN)"
transform_ext: null
decompose: null
validation:
type: string
enum: ["T", "F", "t", "f", "Y", "N", "y", "n"]
tier: [BOOLEAN, initials]
release_priority: 5
aliases: [initial_bool, char_bool]
samples:
- "T"
- "F"
- "Y"
- "N"
- "t"
- "f"
- "y"
- "n"
references: null
notes: >
Single-character boolean encoding. The model must distinguish
these from categorical single-char columns (handled by boolean_set check
in column classifier).
representation.boolean.terms:
title: "Boolean Terms (True/False, Yes/No)"
description: >
Boolean value represented as a full word or phrase:
true/false, yes/no, on/off, enabled/disabled, active/inactive.
Any casing (TRUE, True, true). Transforms to DuckDB BOOLEAN type.
designation: universal
locales: [UNIVERSAL]
broad_type: BOOLEAN
frictionless:
type: boolean
format_string: null
transform: "CAST({col} AS BOOLEAN)"
transform_ext: null
decompose: null
validation:
type: string
enum: ["true", "false", "yes", "no", "on", "off", "True", "False", "Yes", "No", "On", "Off",
"TRUE", "FALSE", "YES", "NO", "ON", "OFF",
"enabled", "disabled", "Enabled", "Disabled", "ENABLED", "DISABLED",
"active", "inactive", "Active", "Inactive", "ACTIVE", "INACTIVE"]
tier: [BOOLEAN, terms]
release_priority: 5
aliases: [word_bool, text_bool]
samples:
- "true"
- "false"
- "yes"
- "no"
- "on"
- "off"
- "True"
- "False"
references: null
notes: >
Full-word boolean encoding. The most common format in
application data and user-facing systems. DuckDB natively accepts
true/false, yes/no, on/off (case-insensitive) for CAST to BOOLEAN.
representation.identifier.uuid:
title: "UUID"
description: >
Universally Unique Identifier (RFC 4122). Standard formats include
UUID v1 (timestamp-based), v4 (random), and v5 (name-based).
Canonical format: 8-4-4-4-12 hex groups with hyphens.
designation: universal
locales: [UNIVERSAL]
broad_type: UUID
frictionless:
type: string
format: "uuid"
format_string: null
transform: "CAST({col} AS UUID)"
transform_ext: null
decompose: null
validation:
type: string
pattern: "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$"
tier: [UUID, identifier]
release_priority: 5
aliases: [guid]
samples:
- "550e8400-e29b-41d4-a716-446655440000"
- "6ba7b810-9dad-11d1-80b4-00c04fd430c8"
- "f47ac10b-58cc-4372-a567-0e02b2c3d479"
references: null
notes: >
Moved from technology.cryptographic.uuid in v0.5.1.
UUID is a universal identifier used in database design far beyond
technology contexts. DuckDB UUID type accepts standard 8-4-4-4-12
format. Also accepts without hyphens via UPPER() or normalization.
representation.identifier.alphanumeric_id:
title: "Alphanumeric Identifier"
description: >
Mixed letter-and-digit identifier string that doesn't match a known
specific format. Examples: product codes ("SKU-12345"), cabin numbers
("C85"), ticket IDs ("A/5 21171"), lot numbers ("LOT-2024-0042"),
license plates ("ABC 1234").
Format-detectable: contains both alphabetic and numeric characters.
designation: universal
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "CAST({col} AS VARCHAR)"
transform_ext: null
decompose:
alpha_part: "REGEXP_EXTRACT({col}, '[A-Za-z]+')"
numeric_part: "REGEXP_EXTRACT({col}, '[0-9]+')"
validation:
type: string
pattern: "^[A-Za-z][A-Za-z0-9 /_.#-]*[0-9][A-Za-z0-9 /_.#-]*$|^[0-9][A-Za-z0-9 /_.#-]*[A-Za-z][A-Za-z0-9 /_.#-]*$"
tier: [VARCHAR, identifier]
release_priority: 4
aliases: [product_code, reference_code, ticket_id, lot_number]
samples:
- "SKU-12345"
- "C85"
- "A/5 21171"
- "LOT-2024-0042"
- "ABC 1234"
- "REF-00742"
- "B28"
- "FL-2087"
references: null
notes: >
Moved from representation.code.alphanumeric_id in v0.5.1.
Catches mixed letter+digit identifiers that fall through more specific
types (UUID, ISIN, IATA, etc.). The validation pattern requires at
least one letter AND one digit. Format-detectable.
representation.identifier.increment:
title: "Increment / Counter"
description: >
Counter value that increases by fixed steps (often 1). May appear as
row numbers, sequence IDs, or generation numbers.
Broad categorization.
designation: broad_numbers
locales: [UNIVERSAL]
broad_type: BIGINT
frictionless:
type: integer
format_string: null
transform: "CAST({col} AS BIGINT)"
transform_ext: null
decompose: null
validation:
type: string
pattern: "^[0-9]+$"
tier: [BIGINT, identifier]
release_priority: 3
aliases: [counter, sequence, id]
samples:
- "1"
- "2"
- "3"
references: null
notes: >
Moved from representation.numeric.increment in v0.5.1.
Often appears as auto-incrementing primary keys or row numbers.
Difficult to distinguish from regular integers without column context.
representation.identifier.numeric_code:
title: "Numeric Code"
description: >
All-digit string where the digit pattern is meaningful and leading zeros
must be preserved. Covers ISO country numeric codes (840, 036), NAICS/SIC
industry codes, FIPS codes, product category codes, and similar
classification systems. The critical property is that these values MUST
remain as VARCHAR — casting to integer strips leading zeros and destroys
the code's identity. Defers to geography.address.postal_code and
identity.person.phone_number when those more specific types apply.
designation: universal
locales: [UNIVERSAL]
broad_type: VARCHAR
frictionless:
type: string
format_string: null
transform: "CAST({col} AS VARCHAR)"
transform_ext: null
decompose: null
validation:
type: string
pattern: "^[0-9]+$"
tier: [VARCHAR, identifier]
release_priority: 3
aliases: [code, classification_code, category_code]
samples:
- "840"
- "036"
- "5112"
- "06075"
- "0001"
- "99"
references: null
notes: >
Added in v0.5.2. Addresses the #2 analyst frustration:
leading zeros stripped when numeric-looking codes are auto-inferred as
integers. Column-level disambiguation (header hints, consistent length,
leading-zero presence) differentiates from plain integer_number.