Docs

Lexicon format

The lexicon is version-controlled as source CSVs and compiled to JavaScript maps at build time. Four columns describe each word; a three-digit encoding ties them together.

The source CSV

Column	Type	Notes
`Word`	string	Source spelling; proper nouns may be mixed-case
`PoS`	string	Pipe-separated CLAWS7 tags, e.g. `NN2\|VVZ`
`Encoding`	string	Three-digit code; `000` = unchanged
`euspelling`	string	Pipe-separated new spellings; `[]` = unchanged

data/euspell_lexicon.csv

Word,PoS,Encoding,euspelling
night,NN1|NNT1|VV0,101,niht
aahs,NN2|VVZ,012,aahs|aahz
does,NN2|VDZ,202,does|duz

Abbreviations, contractions, and phrases share the same four-column shape in their own files. Contractions add a wrinkle: the PoS field encodes a sequence of grammatical words (spaces separate positions, pipes separate alternative analyses).

All four are in the repository, and they are the only lexicon files edited by hand: euspell_lexicon.csv, abbreviations, contractions, and phrases.

The compiled form

build/compile-lexicon.js turns each CSV into an auto-generated JS module — a Map keyed by lowercase word:

dist/lexicon.js — generated, do not edit

export const data = new Map([
  ['aahs', { pos: ['NN2', 'VVZ'], encoding: 12, spellings: ['aahs', 'aahz'] }],
  // …~205000 entries
]);

Lookup contract

Lookups are always lowercase (lexicon.get(word.toLowerCase())). The encoding is stored as an integer and compared as one (entry.encoding >= 200). matchCase()restores the original word's ALL-CAPS, Title Case, or lowercase afterwards.

CLAWS7 tags — the relevant subset

Part-of-speech tags come from the CLAWS7 tagset. The ones that drive disambiguation:

Tag	Meaning	Example
`VV0`	Base-form verb	read, run, go
`VVZ`	3rd-singular present	reads, runs, goes
`VVD`	Past tense	read, ran, went
`VVN`	Past participle	read, run, gone
`VVG`	Present participle	reading, running
`NN1`	Singular common noun	bass, lead, wind
`NN2`	Plural common noun	bows, tears, winds
`JJ`	Adjective	beloved, blessed

Full reference: ucrel.lancs.ac.uk/claws7tags.html

After editing a CSV

Nothing reads the CSVs at runtime. Every tool loads the compiled dist/ modules, so an edit to a CSV changes nothing until it is recompiled — which is what these commands are for, in a checkout of the repository:

npm run build:lexicon  # data/*.csv → dist/*.js — after any lexicon edit
npm run build          # the above, plus the Rollup content bundle
npm run gen:svm        # only if the NN2|VVZ training data changed

dist/lexicon.js is deliberately absent from the repository: it is a generated artifact, rebuilt on demand rather than committed. The retrained model, by contrast, is committed, at src/disambig/vvz-svm.js, so that a build needs no Python.

Keep reading

Encoding rules

What the three digits mean

Disambiguation

How tags resolve homographs

SSML & POS lexicons

Derived lexicons for TTS & checkers