[Feature]: /speckit.analyze should flag closed vocabularies that are enumerated differently in different artifacts

Correction, 14 Aug 2026: the paragraph beginning "One of the four is this class" misdescribes the delivered artefact. In the delivered contract both enumerations are of six and agree with each other; the enumeration of two existed earlier in the run and should not have been presented as the shipped contract. Errata: https://doi.org/10.5281/zenodo.21926137 — see my comment below for what this does and does not cost the request. The original text is left intact.

Disclosure, per the Spec Kit AI contributions policy: this issue was drafted with AI assistance. Extent: I directed the investigation and chose the framing; an AI assistant read the source material and wrote the draft; I reviewed every claim against the sources cited and I am responsible for the content. Replies from me are my own.

Problem statement

When a spec declares a closed set of values — statuses, categories, admissible kinds — and more than one part of the spec refers to that set, the set gets restated in each place. There is usually nowhere to declare it once. So it multiplies, and any error in it multiplies with it.

This comes from a measured case rather than a hunch.

I ran an experiment where a 40-module system was generated entirely from a contract — no module written or edited by hand, 44 model calls — and verified against reference trajectories published beforehand in an independent formalism.

Four defects survived complete automated verification. None was in the 391 generated statements. All four were in the 30 hand-written contract signatures, and all four were the same error: mis-transcribing a definition that was available in the project. The model wrote correctly what the contract declared, including what the contract declared wrongly.

One of the four is this class. Three parts of the contract declared which invariant classes were admissible. A correction reached one of the three; the other two kept the original wording. The delivered system carried two incompatible vocabularies for the same closed set — one enumerating two values, the others six — and all three enumerations were wrong, in three different ways.

Two verifications could have caught it and neither did. Structural verification compares the contract against the code: it checks which module decides what, not which vocabulary each signature declares in prose. The trajectory oracle validates the running dynamics against published reference traces: it never reached it, because the reference instances do not exercise that part of the formalism. Trace validation validates what the trace traverses.

The contradiction was real, present in the contract, and inert over the validation corpus. It would have surfaced the first time someone ran an instance outside the two validated ones.

The gap, stated precisely: the verification covered the structure of the artifacts and the dynamics of execution. The vocabulary declared in prose inside the specification was covered by neither. I had the analogous check for numeric constants — a threshold declared twice with two values — and it caught those. The same defect over a set of names went straight through.

Proposed solution

In /speckit.analyze, which already does cross-artifact consistency and coverage, add a check for closed vocabularies: flag when the same named set of values is enumerated in more than one artifact or section and the enumerations are not identical, and report the locations and the difference between them.

It needs no domain understanding — only to notice that a named set is enumerated in more than one place and that the enumerations differ. And it is invisible to every other stage: it is not a coverage gap, since every requirement is covered, and not a structural inconsistency between spec, plan and tasks, since the artifacts agree about what exists. It is two statements of the same closed set that disagree about its members, sitting inert until an execution path finally traverses it.

A warning rather than a gate seems right. Legitimate divergence exists — one section sometimes intends a subset of another — and the point is that today the divergence is invisible either way.

A second signal in the same family, if it helps: the same multiplication happened to a procedure rather than a vocabulary. Six signatures repeated the identical construction recipe for one object, because the contract language had no place to declare how a structure is built. Fields had somewhere to go; the construction procedure did not, so it migrated into the signatures and multiplied. Wherever a spec language lacks a home for something, that thing ends up copied, and the copies drift.

Reference, if the case is of interest: When the Defect Is in the Specification: Trajectory-Level Validation of Generated Code, https://doi.org/10.5281/zenodo.21879238 — §8.3 is this defect with its three transcriptions, §10.1 states the check as pending work. The report declares its own limits plainly: one formalism, one model, one operator, and the author of the oracle is the operator of the experiment.

Can you first create this as a preset so you can see how it works out?

Yes, that makes sense. I'll build it as a preset.

Two things I want to get right first.

A preset command override replaces the whole file, so the preset would carry a copy of templates/commands/analyze.md with the check added. Is that the expected shape, or is there a lighter way to extend a core command that I have missed? I ask because a full copy means the preset has to track upstream changes to analyze.md.

And what would count as working out, for you? What I can show is a run on a project where the same closed set is enumerated in two places with different members — I have that case: three sections declaring the admissible invariant classes of a formalism, with three different enumerations, all three wrong, and the contradiction inert until an execution path finally reached it. I would show the analyze output before and after the override. If you would rather see a different kind of evidence, say so and I will produce that instead.

Disclosure, correcting my own wording in the issue above: this reply was drafted with AI assistance and reviewed by me before posting, the same as the issue itself. The line "replies from me are my own" was too narrow — the accurate statement is that the drafting is assisted and the responsibility is mine.

Presets can append, prepend, wrap and by default replace

Disclosure: drafted with AI assistance, reviewed by me, responsibility mine — same as the issue.

Correction to my own issue, before the preset.

The paragraph describing the motivating defect is wrong. I have published an errata rather than quietly editing it: https://doi.org/10.5281/zenodo.21926137

What the issue says: the delivered contract carried two incompatible vocabularies for the same closed set, one enumerating two values and the others six. That does not describe the artefact that shipped. In the delivered contract both enumerations are of six and they agree with each other. The enumeration of two existed during the run, before an earlier correction, and I presented that intermediate state as if it were the contract that shipped.

What that costs the request is worth stating plainly. Where the enumerations agree, a check that compares enumerations across declarations passes without an observation. So the check I asked for would not have caught the case I used to motivate it. What survived there is a transcription that is internally consistent and wrong against its source, and no comparison internal to the spec can reach it.

The request still stands, on a real case rather than that one. I wrote a read-only checker and ran it over the six contracts on my device, 553 declarations inspected. It found one divergence that no verification of mine had reported:

struct  State.invariants
        "clase en separador|regulador|cobertura|canonico"        4 members

sig     invariant_taxonomy.clase_de
        "separador, regulador, cobertura, canonico,
         redundante o incompatible"                              6 members

That one is exactly what this check catches: two declarations, same closed set, different members, same artefact. And it is not documentary. The implementation declares the four and raises on anything else, so a state carrying one of the other two is legal according to the signature and fails at runtime.

One thing I would build in from the start, because it cost me a false negative. My first extractor reported zero divergences across every contract, including the one above — it did not recognise "en" as a set introducer. Report coverage next to the result: how many declarations inspected, how many yielded a statement the extractor recognised. Mine is three to four per cent. A check that inspects little and reports zero looks identical to one that works.

On the preset: understood on append — that removes the tracking problem, so it will be an append rather than a copy of analyze.md.

Duplicate — posted twice by mistake. See the identical comment directly above.

Solid — appreciate the correction, and the 4-vs-6 case is squarely what this is for. Looking forward to the preset.

Disclosure: drafted with AI assistance, reviewed by me, responsibility mine — same as the issue.

Following up on both things you asked for — the preset form, and what would count as working it out.

It's a preset now, append strategy, no copy of your analyze.md. Installed and verified on a clean project:

BEFORE   speckit.analyze → core (bundled), no presets installed

AFTER    Composition chain:
           1. [base]   core (bundled)           → templates/commands/analyze.md
           2. [append] closed-vocabulary v1.0.0 → .specify/presets/…

         383 lines materialised
         line 161  ### 5. Severity Assignment          ← core, untouched
         line 269  ## Additional pass: closed-set…     ← the added pass

My earlier version inserted the step in the middle of the file, before Severity Assignment. That's gone — it's a step after the core analysis now, so a change on your side propagates without touching this.

On the working-out. Three fixtures ship with it. One that must fire, and two negative controls, because a check that only demonstrates it bites hasn't demonstrated it stays quiet, and staying quiet is the harder half:

001-divergent         order status enumerated twice, one member differs   fires
002-agreeing          same members, different order and separator         quiet
003-declared-subset   "shipment status is a subset of the order status"   quiet

Each negative control exercises a different rule — formatting variance in one, declared relation in the other.

And the part actually worth your time. I ran the check against its own fixtures three times. All three runs reported that the fixtures gave the expected result. Two of the three runs were wrong:

run 1   passed — and compared zero pairs. Both negative controls stayed
        quiet because the grouping never matched their declarations,
        not because the check declined to report.
run 2   grouping on the full noun phrase — lost the true positive.
        "order status" and "status" became two separate groups.
run 3   grouping on the head noun. Correct, verified group by group.

The difference between run 1 and run 3 is invisible in the result line. It shows only in the coverage figure and the per-group detail.

That's why the pass reports coverage on every run, including empty ones. It isn't tidiness — it's the one thing here I'd defend hardest: a finding without declared coverage is not a finding about the system. It is a claim about what the instrument looked at, wearing the costume of a claim about the system.

The case that motivated all of it: an extractor of mine reported zero divergences over a corpus that contained one in plain sight — it didn't recognise in as an introducer or | as a separator, and of 112 declarations it parsed 4. Without the coverage line, that zero reads as the corpus is clean.

Field measurement, with its own limitation attached. I ran the detection rules over 11 real contracts from my own ecosystem:

5 systems     0 declarations inspected — the check does not apply:
              they declare no signatures or structures
6 systems     303 inspected, 44 recognised (15%), 0 divergences

And I know there is a real divergence in that corpus — one artifact enumerates four members of a set the source defines as two disjoint sets of four and three. This check does not find it, for two reasons, and both are honest ones: that divergence is against the source, not between declarations, and the declarations in question agree with each other. Plus 15% recognition leaves most of those signatures out — they're dense prose, not set enunciations.

The zero is only honest because those two numbers sit beside it.

Severity is WARNING, never blocking. Two declarations that enumerate differently may be a defect, or may be two legitimately different sets sharing a name. The check can't tell those apart, and halting on something that needs judgement costs more than it saves. It reports the difference in both directions — they disagree isn't actionable; which member each side omits is.

Calibration is part of the deliverable. An earlier version of a check in this family reported 72 findings where 7 were real, all triggered by a word appearing anywhere in the surrounding text. Hence: a name is required in front of the introducer, and grouping never happens on content similarity. A check that shouts is worse than no check — it teaches people to ignore it, and the real finding goes out with the noise.

Declared limit, stated inside the pass itself. It compares declarations against each other, inside the spec. It cannot tell whether a set that every declaration agrees on is the right set. A spec that is internally consistent and uniformly mis-transcribed from its source is indistinguishable, to this pass, from a correct one. That's not a defect of the check — it's outside what any comparison internal to the spec can reach, and it's stated so a clean result isn't read as more than it is. The field measurement above is that limit, observed.

Three things I have not done, so none of them are implied.

I have not run /speckit-analyze end-to-end against the fixtures. That needs an agent session executing the skill. What's verified here is that the pass arrives intact and in the right position.

The rules were implemented and measured as a standalone instrument, mechanically. The pass shipped in the preset is prose instructing a model to apply those same rules — whether a given model applies them faithfully is not established by these runs, and it's the test I'd most want next.

The speckit_version: ">=0.8.0" floor is not verified against your history. strategy is confirmed present in 0.16.4.dev0, which is what I have installed.

No rate is claimed. Three fixtures, two corpora, one device.

Happy to change any of it, including dropping the coverage requirement if you think it's noise.

Looks like solid progress. Have you rolled it into a preset we can include in the community catalog? Did I miss that?

Not yet. Three files missing, and two fixes surfaced while building the mechanical verification instrument.

Where it stands. The preset works, installed with specify preset add --dev on a scratch project. Composition verified on a clean project:

BEFORE   speckit.analyze -> core (bundled), no presets installed

AFTER    Composition chain:
           1. [base]   core (bundled)           -> templates/commands/analyze.md
           2. [append] closed-vocabulary v1.0.0 -> .specify/presets/...

         383 lines materialised
         line 161  ### 5. Severity Assignment          <- core, untouched
         line 269  ## Additional pass: closed-set...   <- the added pass

Detection verified mechanically, not end-to-end. The rules in commands/speckit.analyze.md were implemented as a standalone, read-only instrument and run against the three fixtures shipped in the repo — mechanically, not by a model reading the composed prose:

001-divergent
  COVERAGE  inspected 26 - recognised 2

  WARNING - closed-set divergence
  set: order status
    spec.md:6   {cancelled, delivered, paid, pending, shipped}
    plan.md:5   {delivered, paid, pending, refunded, shipped}
    in A not in B: cancelled
    in B not in A: refunded

002-agreeing          grouped, compared, members identical -> nothing reported
003-declared-subset   grouped, "is a subset of" -> excluded by declared relation

All three matched. Declared limit: this is not an agentic end-to-end invocation of /speckit.analyze — whether a model reading the composed prose applies the same rules identically is not established by this.

The run surfaced two problems with the pass itself. Both need fixing before this goes anywhere.

declarations inspected has no defined unit. The pass requires the number and never says what is being counted - lines, sentences, candidate fragments? I counted 26 lines because that is what my implementation did. Another agent could reasonably count sentences, or only fragments carrying an introducer, and report 8 or 12. Three defensible numbers for the metric the pass calls its most important part. That gap is invisible to a mechanical implementation, which just picks one convention silently. It would only surface once a model reads the prose and has to pick its own — that comparison has not been run.
Ambiguous name in fixture 003, stable result anyway. In "The shipment status is a subset of the order status: pending, shipped, delivered", what precedes the introducer is order status but the subject of the sentence is shipment status. Two readings, two groupings - and it does not matter, because the declared-relation rule fires first. That is robustness, and it should be stated rather than left to luck.

Missing on my side:

valid preset, tested with --dev    done
semantic versioning 1.0.0          done
mechanical rule verification       done
agentic end-to-end run             not done
public git repo                    MISSING
LICENSE                            MISSING
README with install command        MISSING

Plan: own repo tagged v1.0.0, then a fork adding one alphabetically-sorted entry to presets/catalog.community.json with download_url pointing at the tag archive. I have read presets/PUBLISHING.md, and I understand preset-submission is applied during triage.

On #4160. Not mine, and I was not involved. Two of the findings above bear on it.

The coverage point is the one the review already raised. Why it matters, precisely: a mismatch count of zero and a pass that recognised nothing produce identical output. That is the failure mode this issue started from - a count with no identity behind it. inspected N - recognised M is what makes the zero readable.
The unit problem applies to any prose implementation of this pass. If declarations inspected is not pinned to a unit, two agents running the same command on the same spec report different coverage and both are right.

I wrote the pass, so I am not an independent reader of it. And to be exact about what has been checked so far: this is mechanical rule verification, not a model reading the composed prose — that run has not been done yet.

Fixing the two spec issues and putting up the repo next. I will be back with the link.

Nice progress!

Back with the link, as promised. The check shipped as a preset and is in the community catalog now, merged via #4201. Repo: https://github.com/yunusdim/spec-kit-preset-closed-vocabulary, installable with specify preset add from the release tag.

I see #4160 is taking the same check into the core analyze pass. I have left the two findings from building the preset on that PR, since that is where they matter: report coverage on every run (declarations inspected vs recognised, so an empty result is not read as clean), and pin the unit of declarations inspected, which is undefined today, so two agents reading the same prose report different coverage and both are right.

@yunusdim

Thanks @yunusdim — the preset is now catalogued, which completes the preset-first step we discussed. Thank you also for bringing the coverage and counting-unit findings over to [github/spec-kit#4160](#4160).

For now, let’s keep this available as an opt-in preset. Inclusion in core is a separate decision: we would need actual before-and-after agent runs showing what the existing analyze command misses, what the additional pass catches, and where it correctly stays quiet. Catalog inclusion alone does not establish that.

Drafted for @mnriem by GitHub Copilot (model: GPT-6 Astra).

Navigation Menu

Repository navigation

[Feature]: /speckit.analyze should flag closed vocabularies that are enumerated differently in different artifacts #4106

Problem statement

Proposed solution

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Feature]: /speckit.analyze should flag closed vocabularies that are enumerated differently in different artifacts #4106

Description

Problem statement

Proposed solution

Activity

mnriem commented on Aug 13, 2026

yunusdim commented on Aug 13, 2026

mnriem commented on Aug 13, 2026

yunusdim commented on Aug 14, 2026

yunusdim commented on Aug 14, 2026

mnriem commented on Aug 14, 2026

yunusdim commented on Aug 14, 2026

mnriem commented on Aug 17, 2026

yunusdim commented on Aug 17, 2026

mnriem commented on Aug 17, 2026

yunusdim commented on Sep 16, 2026

mnriem commented on Sep 17, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions