Vendor `unicode_names2` by ntBre · Pull Request #28784 · astral-sh/ruff

ntBre · 2026-09-22T16:32:19Z

Summary

This PR adds ruff_unicode_names2 and ruff_unicode_names2_generator crates, vendored from the
upstream unicode_names2 crate with the changes described below. To help with review, I organized
this as 4 commits:

vendor the upstream crate from its master branch

Initially this diff command passed, until I started adapting the vendored crate to our workspace by updating the crate metadata, copying over some symlinked licenses, and configuring clippy and rustfmt to avoid changing anything.

❯ diff -r crates/ruff_unicode_names2 ../unicode_names2
Only in ../unicode_names2: .git
Only in ../unicode_names2: .github
Only in ../unicode_names2: .gitignore
Only in ../unicode_names2: Cargo.lock
Only in ../unicode_names2: CHANGELOG.md
Only in ../unicode_names2: generator
Only in ../unicode_names2: Makefile
Only in ../unicode_names2/src: ngrams.py
Only in ../unicode_names2: target
Only in ../unicode_names2: unicode_names2_macros

❯ diff -r crates/ruff_unicode_names2_generator ../unicode_names2/generator
diff: crates/ruff_unicode_names2_generator/LICENSE-APACHE: No such file or directory
diff: crates/ruff_unicode_names2_generator/LICENSE-MIT: No such file or directory

revert my bug fix PR for an issue introduced by the loose matching PR

Equivalent to git revert --no-commit 63eda5de2bd0dd1e6243c9797efe5a84919a756b in the upstream
repo. I verified these by running the revert commands and repeating the diff above.
revert the loose matching PR, restoring the strict matching present in v1.3.0, which we
currently use

Equivalent to git revert --no-commit d6b2923654a8c65a0155419d206e24b839ccd315 in the upstream
repo. There was one merge conflict in some docs, resolved by deleting both sides.
integrate the vendored crates into ruff

This updates our parser to use the vendored version instead of upstream, adds a few tests to
demonstrate Unicode 17 support and the lack of loose matching, and adds docs for future Unicode
version updates.

The motivations for vendoring are that 1) we don't want loose matching because CPython doesn't do loose matching for its Unicode character name escapes (e.g. "\N{snowman}"), and 2) we do want Unicode 17 support. The former meant we couldn't upgrade from v1.3.0 to v2.0.0 (or v3 or v4), while the latter meant that we needed to upgrade. With the reverted commits in our vendored copy, we're closer to a fork of v1.3.0, as the loose matching is the largest change since then, but our version also includes the performance improvements from progval/unicode_names2#54.

See progval/unicode_names2@v1.3.0...v4.0.0 for the full list of changes.

Test Plan

Vendored tests, plus a few new parser tests that exercise strict vs loose matching.

astral-sh-bot · 2026-09-22T16:34:27Z

Typing conformance results

No changes detected ✅

Current numbers

The percentage of diagnostics emitted that were expected errors held steady at 98.15%. The percentage of expected errors that received a diagnostic held steady at 98.07%. The number of fully passing files held steady at 133/146.

astral-sh-bot · 2026-09-22T16:37:46Z

Memory usage report

Memory usage unchanged ✅

astral-sh-bot · 2026-09-22T16:38:20Z

`ecosystem-analyzer` results

No diagnostic changes detected ✅

Flaky changes detected. This PR summary excludes flaky changes; see the HTML report for details.

Full report with detailed diff (timing results)

astral-sh-bot · 2026-09-22T16:43:37Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

ntBre · 2026-09-22T18:05:39Z

+//! ```rust,ignore
 //! #![feature(proc_macro_hygiene)]
 //!
 //! #[macro_use]
 //! extern crate unicode_names2_macros;
 //!


I didn't vendor this crate since we don't use it, so we had to skip compiling this example.

MichaReiser

I assume the reason why you chose to stick to the alias based implementation is to reduce divergence from upstream (and make it easier to move onto upstream in the future?)

Given that the upstream change is uncertain and there might also not be that much benefit of us to move back to it, I'm inclined to change our crate to always do strict matching, which allows us to remove the somewhat hard to understand matching in Ruff's parser.

MichaReiser · 2026-09-23T06:51:59Z

Thanks for writing this. I'm a bit worried that we'll forge tto bump the unicode version (I also don't know how to do that). Should we add a Update Python Version section to Contributing.md, and mention that we need to update the unicode standard and how to do this?

MichaReiser · 2026-09-23T07:03:38Z

Shouldn't commit 2 contain the new added alias method? At least, that's what your commit history says and commit 2 already uses the method.

I'd be inclined to only change ruff_unicode_names2 in this commit and to make the parser changes in a separate commit (makes it much easier to see what changes we made).

Given that this is a single PR, I think it could also make sense to rebase merge this PR, so that we retain this history

Yeah, sorry. I noticed that they were out of order and thought about switching them but didn't think it mattered since I wasn't expecting to rebase-merge. I can tidy this up some more.

Although from your top-level comment, it sounds like I should just revert the loose-matching PR, and then I think we won't need any changes on our side. We'll just keep using the crate as before. But yes, I kept my upstream PR's approach with the hope that it would land and we would un-vendor. That's also why I didn't think much about future Unicode updates, but I'll write something about that too.

codspeed · 2026-09-23T19:52:26Z

Merging this PR will improve performance by 7.25%

⚡ 1 improved benchmark
✅ 210 untouched benchmarks
⏩ 2 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	Simulation	`linter/default-rules[large/dataset.py]`	20.3 ms	18.9 ms	+7.25%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing brent/unicode-names-vendor (1a9d89e) with brent/0.17.0 (3bac628)²}

2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
No successful run was found on brent/0.17.0 (7739e10) during the generation of this report, so b09692b was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

ntBre · 2026-09-23T20:53:03Z

I think this is ready for another look. I switched to the strict-matching-only approach, restructured the commits, and updated the description. I also documented the Unicode update process, as you suggested.

MichaReiser

Nice, thank you

MichaReiser · 2026-09-24T06:15:42Z


+## Updating the Python version
+
+When adding support for a new Python version, update Ruff's Unicode data to match


Technically not in the scope of this PR, so I'm fine deferring, but we could mention the PythonVersion changes here.

I temporarily pushed this as a separate commit: 19b6f9a

Now I'll start some rebasing to see if we can rebase-merge the PR.

Copy the upstream runtime and generator before the alias-lookup API changes. Source: https://github.com/progval/unicode_names2/tree/63eda5de2bd0dd1e6243c9797efe5a84919a756b Adapt vendored Unicode crates to the Ruff workspace Wire the crates into the workspace, disable nightly-only targets, and configure lint and formatting checks to preserve the vendored source. Replace the relocated generator's broken license symlinks with copies of the licenses.

Restore registered alias spellings as lookup keys before removing loose matching. This reverts progval/unicode_names2#61. Reproduced in an upstream checkout at 63eda5de2bd0dd1e6243c9797efe5a84919a756b: git revert --no-commit 63eda5de2bd0dd1e6243c9797efe5a84919a756b Copy generator/src/lib.rs and src/lib.rs back into the corresponding vendored crates. The revert applies without conflicts.

Restore case-insensitive lookup with exact separator matching by reverting progval/unicode_names2#46. Keep Unicode 17 data and the later name-decoding performance improvements. After reverting alias normalization in the upstream checkout: git revert --no-commit d6b2923654a8c65a0155419d206e24b839ccd315 Resolve the sole conflict in src/lib.rs by deleting both the loose-matching documentation and the obsolete Cargo-enabled section, which upstream had already removed. Copy src/lib.rs, generator/src/lib.rs, and generator/src/phf.rs back into the corresponding vendored crates.

Use the vendored runtime and generator instead of the registry dependency, providing Unicode 17 names with Python-compatible matching. Update workspace dependencies and tool configuration, add parser regressions, and document how to update Unicode data when supporting a new Python version.

Added in astral-sh/ruff#28784

ntBre added the dependencies Issues relating to dependencies, or PRs that update a dependency file label Sep 22, 2026

ntBre force-pushed the brent/unicode-names-vendor branch 2 times, most recently from d9c1bd6 to 1321f68 Compare September 22, 2026 17:42

ntBre mentioned this pull request Sep 22, 2026

Define cfg(hawk) during analysis astral-sh/hawk#169

Closed

ntBre commented Sep 22, 2026

View reviewed changes

ntBre added this to the v0.17 milestone Sep 22, 2026

ntBre marked this pull request as ready for review September 22, 2026 18:28

ntBre requested review from MichaReiser and dhruvmanila as code owners September 22, 2026 18:28

ntBre mentioned this pull request Sep 22, 2026

Ruff 0.17 #28694

Merged

2 tasks done

MichaReiser reviewed Sep 23, 2026

View reviewed changes

ntBre force-pushed the brent/unicode-names-vendor branch from b4fa551 to b2f7b7d Compare September 23, 2026 19:40

ntBre force-pushed the brent/unicode-names-vendor branch from b2f7b7d to 31e1e8b Compare September 23, 2026 19:58

ntBre requested a review from MichaReiser September 23, 2026 20:53

MichaReiser approved these changes Sep 24, 2026

View reviewed changes

ntBre mentioned this pull request Sep 24, 2026

Update Rust crate unicode_names2 to v3 #26077

Closed

1 task

ntBre added 4 commits September 24, 2026 13:22

ntBre force-pushed the brent/unicode-names-vendor branch from 19b6f9a to 1a9d89e Compare September 24, 2026 20:18

ntBre merged commit 24c1344 into brent/0.17.0 Sep 25, 2026
67 of 72 checks passed

ntBre deleted the brent/unicode-names-vendor branch September 25, 2026 13:39

ntBre mentioned this pull request Sep 25, 2026

Python 3.15 Support #28206

Closed

27 of 32 tasks

ntBre mentioned this pull request Sep 30, 2026

Add ruff_unicode_names2 and ruff_unicode_names2_generator astral-sh/crates-policies#33

Merged

zanieb pushed a commit to astral-sh/crates-policies that referenced this pull request Sep 30, 2026

Add ruff_unicode_names2 and ruff_unicode_names2_generator (#33)

612c73a

Added in astral-sh/ruff#28784

BrewTestBot mentioned this pull request Oct 9, 2026

ruff 0.17.0 Homebrew/homebrew-core#316849

Merged


		## Updating the Python version

		When adding support for a new Python version, update Ruff's Unicode data to match

Conversation

ntBre commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Uh oh!

astral-sh-bot Bot commented Sep 22, 2026

Typing conformance results

No changes detected ✅

Uh oh!

astral-sh-bot Bot commented Sep 22, 2026

Memory usage report

Uh oh!

astral-sh-bot Bot commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ecosystem-analyzer results

Uh oh!

astral-sh-bot Bot commented Sep 22, 2026

ruff-ecosystem results

Linter (stable)

Linter (preview)

Formatter (stable)

Formatter (preview)

Uh oh!

Uh oh!

ntBre Sep 22, 2026

Choose a reason for hiding this comment

Uh oh!

MichaReiser left a comment

Choose a reason for hiding this comment

Uh oh!

MichaReiser Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

MichaReiser Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

ntBre Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

codspeed Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by 7.25%

Performance Changes

Footnotes

Uh oh!

ntBre commented Sep 23, 2026

Uh oh!

MichaReiser left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MichaReiser Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

ntBre Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ntBre commented Sep 22, 2026 •

edited

Loading

astral-sh-bot Bot commented Sep 22, 2026 •

edited

Loading

`ecosystem-analyzer` results

`ruff-ecosystem` results

codspeed Bot commented Sep 23, 2026 •

edited

Loading