Add a snippet-based release note approach

[mdboom]

This moves us to snippet-based release note management. This is a prerequisite for proper use of maintenance branches, since release note pages for patch releases can be automatically created by backporting the release note snippet.

For a good description of this approach in general see reno's doc page. Note that CPython also uses a very similar approach to great success.

My initial implementation of this used reno as a dependency, but I ultimately pulled back from that because reno's behavior varied in important ways from what we need. We spent more lines of code "working around" reno's behavior that it took to just write our own implementation here. The conventions used still closely hew to what reno does to avoid re-solving problems they have already solved.

The differences from reno are:

• Known issues had to be sticky. Reno attributes a note to the release that first contains it, so a known issue would have appeared on one page only. cuda-python's long-standing policy is that known issues are listed on every release until resolved.

• Notes had to stay editable after release. Reno reads each note as of the commit that added it, so editing an already-released note changed no page. Since we publish docs from main and want corrections to old release notes to show up, each note's current text is fetched from main.

Review recommendation

I recommend starting with CONTRIBUTING.md to understand how this is intended to work, and then working out from there.

What the change does

Release notes move from one hand-written release/<version>-notes.rst per release to one small YAML
file per change. The design is inspired by reno but is a custom
Sphinx extension, not reno itself.

• Authoring: python toolshed/add_note.py <package> <short-description> creates
  <package>/releasenotes/<short-description>-<16 hex>.yaml from a template. A note maps section keys
  (features, fixes, issues, prelude, ...) to lists of reStructuredText entries. Merged notes
  are never renamed.
• Page generation: the Sphinx extension cuda_python/docs/exts/release_notes.py builds
  release/<version>-notes.rst for every release tag at or after the package's first note-based
  version, plus an "In development" page (unreleased-notes.rst). Which notes go on which page comes
  from git tag ranges (release_ranges.py), not from reno-style version labels.
  • Minor release: notes added since the previous minor release.
  • Patch release: notes added since the previous release of the same X.Y line.
  • Unreleased: notes added since the newest release in HEAD's history.
  • Backports appear on both the patch page and the next minor page (intentional).
• Known issues: issues entries are listed on every page whose tag still contains them, with the
  text they had at that tag. Resolving one means deleting the entry.
• PR links: added automatically from the (#N) suffix of the commit that added the note file, or
  from explicit (#N) markers in the entry.
• Generated pages start with a marker comment, are git-ignored (release/.gitignore), and the
  extension refuses to overwrite a file without the marker.
• Enforcement:
  • Pre-commit release-notes-lint (ci/tools/lint_release_notes.py): file name, YAML shape, known
    keys written plainly (fixes: at line start), no leftover TODO.
  • pr-metadata-check.yml runs ci/tools/check_pr_release_notes.py: a PR that touches a package's
    source_paths needs a note added or modified, unless it has the skip-release-note label.
  • Release workflows run ci/tools/check_release_notes.py: hand-written page before the first
    note-based version, otherwise at least one note added in the release's range.
• Migration: the hand-written 13.5.0 (cuda-bindings) and 1.3.0 (cuda-core) pages are deleted
  and replaced by note files. Earlier pages stay hand-written.

[leofang]

btw IIRC @carterbox moved nvmath-python to use reno. My impression (Daniel can correct me if I am mistaken) is that there's mixed feeling. The time goes from "making sure we don't miss anything" to "fighting against the tooling to get the last mile."

[github-actions]

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-3034/
https://nvidia.github.io/cuda-python/pr-preview/pr-3034/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-3034/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-3034/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

[carterbox]

Yes, nvmath currently uses reno. Yes, our sentiment is mixed, but not because reno is a bad choice per se. The main functionality which is release note aggregation works well, but this functionality is not unique to reno.

I was really excited to have fully autogenerated release notes in our documentation, but when trying to get our documentation out the door we had unexpected difficulty getting the documentation rendered correctly.

The root of problem is that our development workflow doesn't match OpenStack's strict branching/tagging model, so the git-awareness wasn't the convenient feature that I wanted it to be.

There are two issues specifically that we had/have:

1. Our release pipeline checks out the repository by hash (not tag), so the final tag did not exist on the documentation builder runner at release time. This meant that the reno sphinx plugin thought we were still building release candidates and did not render the release notes for a final release. Our WAR was to use reno to generate the release notes and hardcode them onto the docs page instead of having them dynamically generated by the plugin.

2. There was also a mismatched expectation about how backports are handled. I think that OpenStack is very strict when they cut a release branch; any backports must be added in a patch release. nvmath is a bit more flexible; we allow backports for a few weeks between when the release branch is cut (v1.0.0r0) and the final release (v1.0.0). Our workflow results in duplicate release notes in the v1.0.0 and v1.1.0 branches unless you manually prune them because reno expects backports to not be shipped until v1.0.1 (in this case you would want duplicate release notes).

I still have a month to decide whether we're going to try again with reno or if we will switch to towncrier, which is not git-aware.

[leofang]

Thanks, Daniel! Yeah git-awareness is no longer a useful feature to me. For example our CI design was forced to change because setuptools-scm requires it, and I really don't like it especially for useless CI re-runs that it must trigger at tagging time (to generate the correct version). I like automation tools but I really don't want them to be more opinionated than I am (and I am already super opinionated) 🙂

@mdboom is this something we want to push through quickly? I see the PR Review Together label was removed.

[mdboom]

btw IIRC @carterbox moved nvmath-python to use reno.

To be clear, this PR doesn't use reno -- it uses a reno-like approach. The behavior we need is different enough that it was fewer lines of more straightforward code to implement from scratch than it was to workaround reno's differences.

Thanks, Daniel! Yeah git-awareness is no longer a useful feature to me.

This is all in service of being able to do proper patch release management (bugfixes are backported to maintenance branches, and patch releases are made from there to avoid patch releases containing new, half-baked work). That's all very git/branch aware, and having the release notes managed automatically makes backporting much lighter weight and easier to manage.

For example our CI design was forced to change because setuptools-scm requires it

Yes, but the old approach was full of bugs and pitfalls, IMHO. It was possible to release something under any name we wanted, with no way to reverse engineer the tag that created it.

@mdboom is this something we want to push through quickly? I see the PR Review Together label was removed.

No need to rush -- I thought that label was auto-assigned so I removed it. But I'm happy to discuss in the meeting.