Skip to content

Commit 96e8f1b

Browse files
derek73claude
andcommitted
fix(derek73#585): keep a whole first code point as written
Second docs review: the fallback added in a21a4e6 also fired for a first character Unicode excludes from composition, which NFC writes as base letter + mark: U+095B 'ज़' initialled 'ज.' and U+FB2A 'שׁ' initialled 'ש.', where every release kept text[0]. The fallback now fires only for a start longer than one code point. Measured over every code point as a first character: none moves against 53c1c86, and every one gives text[0]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent a21a4e6 commit 96e8f1b

3 files changed

Lines changed: 14 additions & 6 deletions

File tree

‎docs/design/decisions.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1373,7 +1373,7 @@ R3's earlier history is under `decisions.md#R2`, which this entry does not repea
13731373
THE FRAME BUDGET WENT DOWN: `tools/perf/call_count.py` reads `py3.11 parse=406.00 facade=443.00` where the parent read `412.00 / 449.00`. The two marks now come off ONE walk over the roles instead of two generator expressions, and on 3.11 a genexp costs a frame for its creation and one per resume, so the reference name's two non-empty parts pay exactly six. `_CALL_BASELINE[(3,11)]` is `{"parse": 410, "facade": 447}` with `_BAND = 0.02`, so the bands are [401.8, 418.2] and [438.06, 455.94] and both figures land inside: NO baseline edit and no `decisions.md#parse-cost` decision is owed. PEP 709 inlined comprehensions and not generator expressions, so 3.12 through 3.15 should drop by the same six.
13741374
THE NO-PARSE PATHS KEEP THE OLDER READING, the same degradation the 2026-09-13 #528 bullet above accepted and for the same reason: a token the parse never saw carries no mark, so the view falls back to the vocabulary. Measured 2026-09-20, `HumanName("JUAN GARCIA Y LOPEZ").initials()` is "J. G. L." live and "J. G. Y. L." after `pickle`, `copy.copy` or `copy.deepcopy`; a name built from keyword fields carries no tags either, and so does one whose `*_list` property a subclass overrides. The remedy is the v1 one — assign `full_name` again — since the facade has no `revise`.
13751375
HOW FAR IT REACHES, over a STATED population rather than an example, because this bundle MULTIPLIES it and the earlier wording did not say by how much: over the deduped `tools/differential/corpus*.jsonl` glob plus `tests/v2/cases.py`'s texts — 1,568 non-empty names — the v1 `initials()` of a pickled, copied or deep-copied name differs from the live parse's on 34 names here against 6 at the parent 46651750, measured 2026-09-20 in two processes each asserting its own `nameparser.__file__`. #461's lone-connective half is what multiplies it: every part whose only word is the connective initials live and drops after the round trip, so `HumanName("Juan de y").initials()` is "J. y." live and "J." after `copy.deepcopy`. RECOMPUTE by building that population and comparing `initials()` before and after each of the three state paths. A larger figure measured over a generated grid says nothing this does not — the population is what the number is about.
1376-
- 2026-10-02 #585 — DONE: an initial is the shortest start of its word that composes to the composed word's first character, written in the form the writer used. `initials()` on both surfaces had taken a token's first CODE POINT (`t.text[0]` in `_render.initials`, `word[0]` and `w[0]` in `_facade`), which for a name typed decomposed (NFD) is a base letter without the mark that completes it: `émile zola` initialled `e. z.` against `é. z.`, and — the reach the issue did not name, found by the corpus walk's control — decomposed katakana lost its dakuten, `マイケル ジャクソン` initialling `マ. シ.` against `マ. ジ.`. `_render._first_letter` is the one helper both surfaces call. One shape has no such start: a mark composing with nothing that canonical order puts between the letter and a mark that does (`o` + U+0331 + U+0301, whose composed spelling is `ó` + U+0331), where the initial is the composed character decomposed, `o` + U+0301 — found by the docs review, 2026-10-02. WHY NOT "THE FIRST LETTER AND THE COMBINING MARKS AFTER IT", the shape #542 used for case repair and the issue's own option 1, measured before writing it: decomposed hangul jamo are LETTERS (category Lo), not marks, so it would initial a decomposed `김` as `ᄀ` where the composed name gives `김`; and a Bengali vowel sign IS a mark (Mc), so it would move a COMPOSED `কিরণ` from `ক.` to `কি.`, which is a question about Indic syllables nobody has decided, and a change the gate would see on every composed Indic name. Composing the prefix answers both: a composed word gives `text[0]` exactly as before, so no composed name's initials move. DECLINED: composing the initial to NFC (the issue's option 2, and what outside PR #586 did, on the core surface only). It fixes the dropped mark but hands back a composed initial for a decomposed name, where #542 settled that the render views keep the form they were given (decisions.md#R4, 2026-10-02). VERIFICATION, which the gate DOES see, unlike #542's: `_initials` is a compared pseudo-field (#484), and the two NFD example lines this change adds to rules.md#R3 enter the rules corpus and diff at every baseline, classified by a `fix(#585)` `_initials` rule in each of the five ledgers; the CJK rule's corpus claim grows 138 → 139 at 1.4.0 and 2.0.0 with the katakana row, which lies in its script span. `tests/v2/test_properties.py::test_a_decomposed_name_renders_as_its_composed_twin` joins #542's walk rather than opening its own and records 44 disagreements — 22 texts, each on both surfaces — over the 139 texts compared with `text[0]` restored (this change's corpus, its two R3 rows included; measured 2026-10-02), with a live control beside it.
1376+
- 2026-10-02 #585 — DONE: an initial is the shortest start of its word that composes to the composed word's first character, written in the form the writer used. `initials()` on both surfaces had taken a token's first CODE POINT (`t.text[0]` in `_render.initials`, `word[0]` and `w[0]` in `_facade`), which for a name typed decomposed (NFD) is a base letter without the mark that completes it: `émile zola` initialled `e. z.` against `é. z.`, and — the reach the issue did not name, found by the corpus walk's control — decomposed katakana lost its dakuten, `マイケル ジャクソン` initialling `マ. シ.` against `マ. ジ.`. `_render._first_letter` is the one helper both surfaces call. One shape has no such start: a mark composing with nothing that canonical order puts between the letter and a mark that does (`o` + U+0331 + U+0301, whose composed spelling is `ó` + U+0331), where the initial is the composed character decomposed, `o` + U+0301 — found by the docs review, 2026-10-02. That fallback is for a start LONGER than one code point: a first code point that is a whole character Unicode excludes from composition (`क़`, U+0958, which NFC writes `क` + a nukta) is kept as written, its nukta with it, as before this change; the second review found the first draft of the fallback stripping it (`ज़ाकिर` initialling `ज.`). And the fallback writes the character wholly decomposed even where the writer composed part of it (`ẹ` + U+0331 + U+0302 gives `e` + U+0323 + U+0302) — the twin holds, the written form does not, accepted as contrived. WHY NOT "THE FIRST LETTER AND THE COMBINING MARKS AFTER IT", the shape #542 used for case repair and the issue's own option 1, measured before writing it: decomposed hangul jamo are LETTERS (category Lo), not marks, so it would initial a decomposed `김` as `ᄀ` where the composed name gives `김`; and a Bengali vowel sign IS a mark (Mc), so it would move a COMPOSED `কিরণ` from `ক.` to `কি.`, which is a question about Indic syllables nobody has decided, and a change the gate would see on every composed Indic name. Composing the prefix answers both: a composed word gives `text[0]` exactly as before, so no composed name's initials move. DECLINED: composing the initial to NFC (the issue's option 2, and what outside PR #586 did, on the core surface only). It fixes the dropped mark but hands back a composed initial for a decomposed name, where #542 settled that the render views keep the form they were given (decisions.md#R4, 2026-10-02). VERIFICATION, which the gate DOES see, unlike #542's: `_initials` is a compared pseudo-field (#484), and the two NFD example lines this change adds to rules.md#R3 enter the rules corpus and diff at every baseline, classified by a `fix(#585)` `_initials` rule in each of the five ledgers; the CJK rule's corpus claim grows 138 → 139 at 1.4.0 and 2.0.0 with the katakana row, which lies in its script span. `tests/v2/test_properties.py::test_a_decomposed_name_renders_as_its_composed_twin` joins #542's walk rather than opening its own and records 44 disagreements — 22 texts, each on both surfaces — over the 139 texts compared with `text[0]` restored (this change's corpus, its two R3 rows included; measured 2026-10-02), with a live control beside it.
13771377

13781378
### R4 — case repair reads the unjoined mark
13791379

‎nameparser/_render.py‎

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -184,18 +184,23 @@ def _first_letter(text: str) -> str:
184184
text's first character -- 'e' + U+0301 for a decomposed 'é', and
185185
the two or three jamo of a decomposed hangul syllable, which are
186186
letters rather than marks -- or, where a mark composing with
187-
nothing stands inside that start, the character decomposed. A
188-
composed `text` gives text[0], as it always did, so a mark that
189-
composes with nothing (a Bengali vowel sign) stays out of the
190-
initial in either form. Raises IndexError on an empty `text`, as
187+
nothing stands inside that start, the character decomposed (even
188+
where the writer composed part of it). A text whose first code
189+
point holds a whole character gives text[0], as it always did --
190+
a composed text, and one opening with a character Unicode excludes
191+
from composition -- so a mark that composes with nothing (a
192+
Bengali vowel sign) stays out of the initial in either form. Raises IndexError on an empty `text`, as
191193
text[0] does."""
192194
if text.isascii():
193195
return text[0]
194196
first = unicodedata.normalize("NFC", text)[0]
195197
end = 1
196198
while not unicodedata.normalize("NFC", text[:end]).startswith(first):
197199
end += 1
198-
if unicodedata.normalize("NFC", text[:end]) == first:
200+
# end == 1: one code point already holds the first letter, kept
201+
# as written even where NFC would split it (U+0958, excluded from
202+
# composition, is 'क' + a nukta composed; it keeps its nukta here)
203+
if end == 1 or unicodedata.normalize("NFC", text[:end]) == first:
199204
return text[:end]
200205
# A mark that composes with nothing stands, in canonical order,
201206
# between the letter and a mark that does ('o' + U+0331 + U+0301,

‎tests/v2/test_render.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -999,6 +999,9 @@ def test_a_composed_initial_is_the_first_character_as_before() -> None:
999999
rewrite to its unified twin and which is kept as written."""
10001000
from nameparser._render import _first_letter
10011001
assert _first_letter("\uf900x") == "\uf900"
1002+
# excluded from composition: NFC splits it into 'क' + nukta, and
1003+
# the code point is kept whole, nukta and all
1004+
assert _first_letter("\u0958x") == "\u0958"
10021005
assert _first_letter("Émile") == "É"
10031006
with pytest.raises(IndexError):
10041007
_first_letter("")

0 commit comments

Comments
 (0)