Skip to content

Check the vocabulary word order in the build (check 10) - #767

Merged
derek73 merged 4 commits into
masterfrom
feat/vocab-word-order
Oct 20, 2026
Merged

derek73 merged 4 commits into
masterfrom
feat/vocab-word-order

Conversation

@derek73

@derek73 derek73 commented Oct 20, 2026 •

Copy link
Copy Markdown
Owner

Part of #747. #750 (the titles) waits for this, so its words land in a checked order.

What

tools/vocab/build.py gains check 10: every language file except und.toml lists its words in one canonical order, and the build refuses a file out of order, naming the entry to move and where it belongs (en.toml: 'snr' (generation) belongs after 'sr'. …). The entry it names is the first one outside the longest run already in order, and its place is given against that run, so following the messages takes the fewest moves and a word appended at the end of a file is the one named. It checks and never rewrites a file, because a comment above an entry belongs to it and must move with it, which no TOML reader can see. Check 10 runs last, so an exclusion's reason (check 6) or a same_as naming no word (check 7) is reported first.

The order:

  1. By kind, in CATEGORY's order. It already listed each category's kinds together and the README's kind table already followed it; an import-time assert now keeps the kinds of a category together, and the README test is now sensitive to the table's order. A word with several readings sorts under its first kind. An entry that only records a refusal (senior, fils) sorts under the kind it refuses; a name reading places an entry only when it has nothing else.
  2. Then by its same_as chain, starting from the word at the root, with the head first: jr, jnr, junior, then sr, snr. Because the whole chain is used, el- keeps its own spellings after it inside the al- group in ar.toml, where sorting by the direct target alone would have split them. Only the file's own entries are followed, so another file's chain never decides this file's order.
  3. Then by text, in codepoint order of the normalized text. That order is defined for every script and is no one language's collation. The costs are that within a script an accented letter comes after z (aḍ- after az-), and that traditional and simplified spellings with no same_as between them end up apart (課長, 课长).

und.toml is exempt: its vetting comments point along the order of the Python lists it came from ("the bare-κ trap above"). A same_as chain that loops back on itself (e.g. jr → jnr → jr) is refused in every file, und.toml included. The check looks at every same_as written, across files and kinds, because nothing holds a word to a single head.

Data

The first commit sorts the 41 of 71 language files that were out of order on cc030b9. Whole entries move and nothing inside an entry changes. The second commit adds the check, so each commit builds.

Docs

  • nameparser/vocab/README.md: a new "Word order" section, and the "Adding a word" steps now point to it.
  • decisions/vocabulary-model: a dated bullet recording why the order is checked rather than rewritten, why it uses the chain rather than the direct target, why codepoint order, and why und.toml is exempt.

Verification

  • uv run python tools/vocab/build.py --check: _compiled.py is current (byte-identical).

  • tools/differential/snapshot.py: matches the snapshot.

  • Full suite: 21,909 passed, 123 skipped, 3 xfailed. mypy and ruff are clean.

  • New rows in tests/v2/test_vocab.py cover a swapped pair, kinds and categories out of order, a variant before its head, a variant away from its head, refusals placed outside their kind, and loops in a language file and in und.toml. The accepted shapes cover a head before its spellings, a refusal beside a name reading, a comment moving with its entry, the ar.toml chain, two scripts in one file, and und.toml out of order passing.

  • Review rounds. /pr-review-toolkit:review-pr ran five reviewers, and dd7b335 fixes what they found:

    • the check ran ahead of check 6;
    • the message named an untouched neighbour;
    • some same_as loops passed;
    • another file's chain decided this file's order;
    • a comment quoted from und.toml pointed the wrong way.

    A second-round review of dd7b335 found that the given place could be beside an entry that was out of place itself (up to 22 moves where 10 would do), plus a hang when check_order is called on its own and a RecursionError on a very long chain. 0a223d8 fixes all three. Each new row fails against the commit before its fix: some raise nothing, the rest name the wrong entry or the wrong loop. test_following_the_order_messages_takes_the_fewest_moves checks the move count over every order of six words.

  • Negative controls, measured: with check_order disabled, every order row compiles, so the check is what refuses each one. Under a plain-text key, the two variant rows pass, so only the head-first grouping fails them. Under a direct-same_as key, the ar.toml chain shape fails.

🤖 Generated with Claude Code

derek73 and others added 2 commits October 19, 2026 19:34
41 of the 71 language files, every one but und.toml: by kind in
CATEGORY's order, then by the same_as path from the root head, then by
text. Whole entries move; nothing inside one changes, so the compiled
module is byte-identical. The next commit adds the check that holds it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tools/vocab/build.py refuses a language file whose words are out of
order, naming the entry and the word it belongs after; it never
rewrites a file, since a comment above an entry must move with it.
und.toml is exempt: its vetting comments point along the order of the
Python lists it came from. A same_as chain returning to its start is
refused too, the key following the chains. The README's kind table is
now held to CATEGORY's order, and nameparser/vocab/README.md states the
rule; decisions/vocabulary-model records why.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73 derek73 added tooling vocabulary Per-word vocabulary model and language packs labels Oct 20, 2026
@derek73 derek73 self-assigned this Oct 20, 2026
@codecov

codecov Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.05%. Comparing base (cc030b9) to head (0a223d8).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #767   +/-   ##
=======================================
  Coverage   99.05%   99.05%           
=======================================
  Files          47       47           
  Lines        4642     4642           
=======================================
  Hits         4598     4598           
  Misses         44       44           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

derek73 and others added 2 commits October 19, 2026 19:52
… loop (#747)

From the #767 review:
- the order check runs last in check_pool, so an exclusion's reason
  (check 6) and a same_as typo (check 7) are seen before it; the
  en.toml exclusion test is back on en.toml
- the message names the first entry outside a longest run already in
  order, so a word appended at the end is named, not its neighbours
- a file's same_as path follows that file's entries only, so another
  file's chain no longer decides its order
- the loop refusal walks every same_as written, across files and kinds,
  and names the loop rather than the chain into it
- the und.toml example quoted the wrong way; und.toml:168 said "above"
  of a note 2,900 lines below

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…747)

- the named place is against the run already in order, so following
  the messages takes exactly n less the longest in-order run moves
  (the first version gave a neighbour that was out of place itself:
  up to 22 moves where 10 would do); held over every order of six words
- _heads leaves out a same_as naming its own word, so check_order
  called alone ends; the loop search is iterative, so a chain longer
  than the recursion limit is no RecursionError

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@derek73
derek73 merged commit 20b595b into master Oct 20, 2026
12 checks passed
@derek73
derek73 deleted the feat/vocab-word-order branch October 20, 2026 03:05
derek73 added a commit that referenced this pull request Oct 20, 2026
…#750) (#769)

* Add title kinds; classify the salutations and given-addressing titles (#750)

Ten title kinds join the generic `title`: salutation, style, academic,
religious, nobility, military, office, occupation, kinship and modifier,
defined in nameparser/vocab/README.md. The salutations of every language
and the 49 titles that address by the given name leave und.toml for the
files of the languages whose names they occur in, six of them new (ak,
et, id, ro, sw, vi); the module comments become `expands`, `note`,
`issue`, `same_as` and, on a marked word only, `gender`.

The title keep-outs that decisions/ prose held become `excluded`
entries the build enforces, with their counts as name readings where
counts were taken (malam, bay, shehu). The per-word counts of
decisions/salutation-titles move onto each word's note, and the prose
points at the query.

Tests pin every check on the title category in a fine kind as well as
the generic one, each row failing when its check is keyed on the kind.

No compiled set moves: build --check is current and the snapshot matches.

Part of #750.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address the docs review of #750 part 1

shehu's argument against a graf-class reading of its counts is back in
vocabulary-exclusions, with malam's contrast and both CLDR greps; the
single-letter trap (bare د, κ) is stated apart from the transliteration
rule; lord, lady and knight are named as not yet nobility; the
forty-six refusal-only entries are dated to the tree they were counted
on; und.toml's positional comments are described without a quote that
moved; paní's note keeps that the unaccented pani is not shipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address the second docs review of #750 part 1

The single-letter trap is decided per word, and the Bengali ড is named
as the decided exception; the Arabic refusals no longer claim their
article forms ship; shehu's bullet drops the digits its entry holds;
the in-place correction to the #767 entry says what was wrong; the
graf-class pointers say which words carry counts and which file holds
herra's note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address the PR review of #769

und.toml's Hebrew, Devanagari civil and Bengali comments no longer point
at blocks this PR moved out, and the #606 header names its noble ranks;
the Hebrew comment gives both boundary tests the mid-word quote fails;
גב''s note gives the mechanism that applies to a word-final quote; bà's
reason says it was refused as ông's pair without its own evidence;
الدكتور's note says it speaks for the group. The README calls one kind
per file a convention, not a check, and says its examples include words
the later parts of #750 classify.

Tests: an addresses exclusion's reason across title kinds (check 6) and
a same_as across title kinds (check 7), each failing under its check
keyed on the kind; and a title-kind exclusion leaving another category
free, which fails if the exclusion over-widens to every reading of the
word. The per-kind compile test's comment no longer overstates how
cased is read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Record the und.toml stale-pointer check in the vocab README (#750)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tooling vocabulary Per-word vocabulary model and language packs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant