Repository navigation
Feature request: re-check published entries periodically (more than 1 in 8 have a problem the registry doesn't catch) #1715
Description
Summary
I measured every latest entry in the registry (38,977 entries from 23,016 publisher namespaces, registry snapshot 2026-10-04) against what npm, PyPI and GitHub reported on 2026-10-06. 5,435 entries (13.9%; 3,169 publisher namespaces, 13.8%) have at least one problem the registry doesn't catch — most commonly a repository link that isn't publicly reachable, or a listed package version that is withdrawn upstream or can't be installed. Part of that count is a design choice rather than a problem (see the websiteUrl note below); setting that part aside, the figure is still at least 5,192 entries (13.3%).
This is not a security report, and it isn't a criticism of publishers: things change upstream after an entry is published, and some checks were never part of publishing. The registry's validation is good at publish time; it just has no timer that runs it again. Below are the numbers, what the code does, and some small proposals.
The registry has changed since my snapshot, so exact counts today will differ. I have not re-pulled it for this issue.
Disclosure: I built ProtocolProbe, a free directory that reads the registry's public data. This issue is about the registry, not about my project.
Related, already open (I found these after measuring): #93 (remove entries that reference invalid packages, with a check "at least ~daily"), #1579 (active entries with neither remotes nor packages), #1484 (repository URLs that point at renamed or transferred repositories). Discussions #1547 and #1591 measured remote endpoints; this covers package and repository records instead, so the counts are not comparable. This issue adds whole-registry numbers across several kinds of drift and a way to record them without deleting anything.
What the registry checks (read from source at c384c4b, re-read at HEAD 11b7b0e, 2026-10-11)
- At publish, it checks the exact listed version.
internal/validators/registries/npm.gofetches/{identifier}/{version}and requires a 200 plus a matchingmcpName;pypi.gofetches/pypi/{identifier}/{version}/jsonand requires themcp-name:token. This has been the case since the validators were added in September 2025. It runs only whenENABLE_REGISTRY_VALIDATIONis on (defaulttrue; I can't see production's setting). - It runs again only when a publisher edits an entry.
UpdateServercallsValidateUpdateRequest, which repeats the package ownership check (skipped for deleted entries). I found no timer, cron or background job that validates entries on a schedule. An entry nobody edits is checked once. repository.urlis checked for format only (IsValidRepositoryURL, a regular expression), never fetched.- The schema requires
name,descriptionandversion— neitherpackagesnorremotes. The docs describe entries with only awebsiteUrlas a supported "custom installation path".
What I found (whole registry, no sampling)
Every percentage below uses the same base — all 38,977 entries, or all 23,016 publisher namespaces — so no row can exceed the total. Each row's own subset is given in the fourth column for context.
| Finding | % of all entries | % of all publisher namespaces | Within its own subset | Cause |
|---|---|---|---|---|
| Any of the below | 5,435 (13.9%) | 3,169 (13.8%) | ||
| Repository link not publicly reachable (404 — missing or private) | 4,298 (11.0%) | 11.3% | 15.0% of 28,743 entries with a GitHub link | never checked |
| Listed package version is deprecated (npm) or yanked (PyPI) | 663 (1.7%) | 1.0% | 4.6% of 14,431 npm/PyPI entries | never checked |
active, but declares no packages and no remotes |
459 (1.2%) | 1.3% | 243 have a websiteUrl, 216 do not |
allowed by schema |
| Linked GitHub repository is archived | 309 (0.8%) | 0.6% | 1.3% of 24,256 entries with a reachable repo | never checked |
| Listed package or version doesn't exist on npm/PyPI | 157 (0.4%) | 0.5% | 1.1% of 14,431 npm/PyPI entries | checked at publish (by default); not again unless the publisher edits |
The 243 entries that have a websiteUrl follow the documented custom-installation pattern, so they may be intended. The 216 that have neither a way to run nor a page to read are the ones a client can do nothing with. The "any of the below" row counts all 459; without the 243 it is at least 5,192 (13.3%), because removing 243 entries cannot lower the total by more than 243.
All 157 entries whose listed version doesn't exist were published after the existence check was introduced — so, if validation was enabled (it is by default), they existed at submission and were removed afterwards. For withdrawn versions and archived repositories I can't tell when the change happened: npm/PyPI don't record when a version was deprecated or yanked, and GitHub doesn't expose when a repository was archived. The registry doesn't check either, at publish or later.
Publisher-supplied deprecation messages for the 663 withdrawn versions, grouped by publisher namespace: renamed to another package (89), backing service shut down (16), npm staff "no longer supported" notice (24), publisher-declared security issue (12), plus discontinued / upgrade-advised / moved distribution.
Proposals
- Re-run the existing package validation on a schedule (e.g. weekly; Automatically remove server.json entries referencing invalid packages #93 suggests daily), reusing
ValidatePackage. On failure, don't silently delete — record it, e.g. a_metafield such aslastVerifiedAt+verificationStatus: ok | version-missing | package-missing, so clients can choose what to do. - Surface upstream withdrawal state. When the listed npm version is deprecated or the PyPI release is yanked, expose that (with the upstream message) in
_meta. Registry-driven clients currently install a withdrawn version with no signal, becauseversionmust be exact. - Optionally check
repository.urlreachability at publish and during re-validation, or mark it as unverified. - Treat "no packages, no remotes, no
websiteUrl" as incomplete (216 active entries in my snapshot) — a distinct status or a publish-time warning — and leave the documentedwebsiteUrl-only pattern alone.
Happy to share the full dataset and method (one record per entry; npm/PyPI/GitHub/OSV sources, every field marked declared / undeclared / not-applicable with a reason), and to help with an implementation. The dataset is archived with a DOI under CC BY 4.0: https://doi.org/10.5281/zenodo.23386799. The report with each figure's source and date is at https://protocolprobe.com/mcp/reports/registry-vs-reality.
Method notes
- Registry snapshot 2026-10-04 (full pagination, 131,427 version rows, one latest row per name); npm, PyPI and GitHub lookups on 2026-10-06.
- npm/PyPI checks use the exact version the registry lists, parsed with semver / PEP 440.
- A 404 on a repository link means missing or private; I don't distinguish the two, and I haven't separated renamed repositories from missing ones (see Provenance audit: 38/398 top-graded servers' repository URLs point at renamed or transferred repos #1484).
- Rate-limited lookups were retried and are never counted as missing data.
- Counts are given per entry and per publisher namespace, because a few namespaces publish hundreds of entries and would otherwise dominate.
- I did not run or call any MCP server for any of this; everything comes from the registry and public package records.
Drafted with help from Claude (AI); the data, method and checks are mine and I've reviewed every number.
Activity
kaspermuutra-dotcom commentedon Oct 20, 2026
+1. Publish-time validation with no re-check means the registry's quality only ever decays. One suggestion on scope: it may help to separate re-checks into tiers by cost, because they answer different questions. (1) Metadata reachability: repo URL, package exists and installs. Cheap, daily, what this issue measures. (2) Endpoint liveness: handshake and
tools/listfor remote entries. (3) Behaviour: whether tools do what their annotations declare, which is expensive and probably belongs in sub-registries or clients rather than the core registry. Defining (1) and (2) in the core and leaving (3) to the ecosystem would keep this tractable.Kasper
Sultan57git commentedon Oct 20, 2026
Thanks @kasper, that split matches what we see in the data, and it makes the ask much more concrete.
(1) Metadata reachability is what this issue measures, and it's cheap: package existence and the exact listed version (npm/PyPI), whether it's deprecated or yanked, and whether the repository URL resolves. All of it is plain HTTP against public APIs, with no code run. On the 2026-10-04 snapshot that gave the figures above, including 157 versions that can't be installed and 4,298 GitHub links that return 404.
(2) Endpoint liveness is also measurable without touching tools. A handshake plus tools/list is a read-only session. In our read-only pass of remote entries (2026-10-08), we opened a session with 14,123 of 21,798. The rest is a mix, and it matters for how a check would report:
some failed to connect;
some answered with an auth challenge, which is a working server that needs a credential;
some were timeouts on our side.
So a core liveness check would need at least "live / auth-required / unreachable" as separate states, not a single up/down flag.
(3) Behaviour: agreed that it belongs outside the core. It means calling a third party's tools, which needs the operator's authorisation. We don't do it for anything we publish. The route we offer is a self-test the operator runs against their own server.
If useful, the method and per-entry data for (1) and (2) are public (CC BY 4.0) and can be re-run: https://protocolprobe.com/docs/cite
kaspermuutra-dotcom commentedon Oct 20, 2026
Thanks @Sultan57git, that's a clean way to cut it.
On (2), I'd add one state between "auth-required" and "unreachable": "refused, unparseable". In #1712, 33 of 52 refusals came back as an HTML page, an empty body, or JSON with no error.code. A client can't branch on those, so they behave like "unreachable" even though the server is up. So four states would cover what both datasets show: live / auth-required (proper 401 + WWW-Authenticate) / refused-unparseable / unreachable.
On (3), agreed: operator authorisation is the line. An operator-run self-test and an independent run the operator authorises answer slightly different questions. The first tells the operator their server works. The second gives them something they can show someone else, tied to the exact version that was tested. Both stay outside the core and can live side by side.
Thanks for publishing the per-entry data under CC BY 4.0. We'd like to cite it, with attribution, in a write-up we're doing on declared vs observed behaviour for opted-in servers. Does your self-test produce a result the operator can publish, or is it local only?
Kasper
Sultan57git commentedon Oct 20, 2026
Thanks @kasper. Four states makes sense. "Refused, unparseable" is the case a client can't act on, so it should be its own state and not fold into "unreachable". I'd keep the evidence with each state, for example the HTTP status and content type of the response, so a client can see why something was classed that way.
On (3): agreed, and I like the distinction. An operator-run test answers "does my server behave", and an authorised independent run gives them something they can show others.
On your question, the self-test is local only by design. It sends nothing to us and we host nothing for it. The operator gets:
the printed result;
a JSON report (--out), which records what was tested, when, and the engine and SDK versions;
from 0.4.0, a SARIF file for GitHub code scanning.
They can publish any of these themselves, but it's self-reported. It isn't signed and we don't verify it, so it isn't proof that someone else can check. A verifiable, version-bound result would need an independent run, or at least a signed report, and we haven't built either.
Happy for you to cite the data. Attribution details and the DOI are here: https://protocolprobe.com/docs/cite (all versions: 10.5281/zenodo.23386799). Please cite the snapshot date too, since the registry keeps changing. I'd be glad to read the write-up when it's out.
Drafted with help from Claude (AI); the facts about the self-test and the data are mine, and I've checked them.
kaspermuutra-dotcom commentedon Oct 20, 2026
Thanks @Sultan57git. Agreed on keeping the evidence with each state. HTTP status plus content type is enough for a client to see why something was classed as "refused, unparseable".
That split is clear, and it's a useful line to draw: the self-test answers "does my server behave", and an authorised independent run answers "can I show someone else". The second half is what we build at Suncly: a run the operator authorises, with a signed report bound to the exact version tested. If it would ever be useful to pair the two, so an operator who passes your self-test can go on to a signed independent run, I'd be glad to talk it through: team@suncly.com.
We'll cite the data with the DOI and the snapshot date, and I'll send you the write-up before it goes out.
Kasper
rdimitrov commentedon Oct 23, 2026
Backlog consolidation: preserving related endpoint-health reports here so periodic revalidation has one implementation tracker.
app.wishpoolconcentration; distinguish missing hosts from transient HTTP failures.initializehandshake, and 70 more are broken before it #1643: initialize-handshake failures and failures before initialization.These reports are being closed as consolidated evidence, not as fixed or as proof that the affected entries have been removed. Include remote DNS/TLS/handshake rechecks in the design alongside package/repository checks, with timestamps and explicit auth-required/transient/unknown outcomes. Do not automatically remove entries based on a single failed probe. Specific moderation or takedown requests remain separate.
rdimitrov commentedon Oct 23, 2026
Also consolidating the older package-revalidation request #93 here. Preserve its daily-check suggestion and the stronger revocation case: if an upstream registry removes a malicious package, assess whether the same server's other distributions should also be disabled. That cross-distribution security response is distinct from merely recording an unreachable or withdrawn package version. The ordinary drift checks proposed here should not silently become automatic takedowns.
Sultan57git commentedon Oct 23, 2026
Thanks for consolidating these here. Agreed: dated results, explicit auth-required / transient / unknown states, and no removal on a single failed check. Our 2026-10-23 snapshot (48,551 servers) is available as dated, per-entry evidence for the package/repository checks if it helps the design.
ondraulehla commentedon Oct 24, 2026
A data point for the tiers discussed here. mcpcost (MIT, https://github.com/ondraulehla/mcpcost, board at https://mcpcost.ulehla.dev) has probed every remote entry of the registry once a day since 2026-10-03, with the official TypeScript SDK as the client:
initialize, thentools/list, 15-second timeouts, one probe per host at a time, the User-Agentmcpcost/<version>. Package entries (npm, PyPI) run once a week in a container with the network off.The run of 2026-10-24: 49 221 entries, 32 364 with a remote, 46 137 probed.
Three things that match what this thread asks for:
All of it is readable without a key: JSON per server (
/api/s/<name>.json), a list API, an MCP endpoint (https://mcpcost.ulehla.dev/mcp), and the raw run as a workflow artifact. If a daily independent feed helps tier 2 (endpoint liveness), I am glad to shape the output to what the registry needs: one JSON per entry with the state, the dated evidence and the cause, under CC BY 4.0.Sultan57git commentedon Oct 24, 2026
Thanks. That covers endpoint liveness. Our data covers the package and repository side, with per-entry evidence for our 2026-10-23 snapshot (48,551 servers).
ondraulehla commentedon Oct 24, 2026
Agreed, the two sets fit together: yours answers "does the package and the repository still exist", ours "does the endpoint still speak MCP today". If it helps the design, I am glad to align the per-entry row with yours, so a registry job could consume both in one shape: entry name, check date, state (alive / auth / transient / dead / unknown), evidence. Ours is already a daily CC BY 4.0 dataset: https://mcpcost.ulehla.dev/report/2026-10
Sultan57git commentedon Oct 24, 2026
Sounds good. Let's use that shape (entry name, check date, state, evidence) and add a check field (package / repository / endpoint), so both feeds merge cleanly; I'll publish ours in it.
ondraulehla commentedon Oct 24, 2026
Done on our side. Today's run in that shape, with the
checkfield: https://mcpcost.ulehla.dev/data/mcpcost-evidence-2026-10.json.gz (1.6 MB gzip, 46 137 rows, CC BY 4.0). The rows are underrows; the document names the five states at the top.{"entry":"ac.inference.sh/mcp","checked_at":"2026-10-24T08:16:17.800Z","check":"endpoint","state":"auth", "evidence":{"status":"auth","target":"https://api.inference.sh/mcp","http_status":401,"auth_scheme":"Bearer","auth_kind":"oauth-dcr"}}checkisendpointfor a remote server andpackagefor an npm or PyPI server started in a container. The mapping from our probe status to the five states is in the file and documented at https://mcpcost.ulehla.dev/report/2026-10#dataset: 429, 5xx, timeout and refused aretransient; no DNS, TLS failure, 404 and "answers but is not MCP" aredead; 401, 402 and 403 areauth; a package that needs values or a network to start isunknown, not dead. The writer runs daily, so a registry job can take a fresh file every day; the monthly copy stays at that URL.Sultan57git commentedon Oct 25, 2026
Ours follows the same shape and is live: https://protocolprobe.com/api/v1/evidence?month=2026-10 (gzip JSON, CC BY 4.0; the bare /api/v1/evidence serves the newest copy and updates daily). Rows are {entry, version, checked_at, check, state, evidence} with check = package-record or repository; header has entries and probed_at. checked_at is each fact's own lookup date; our own failed lookups are unknown, never dead.
docs: add ProtocolProbe to community projects (#1736)
Add mcp-registry-publish-preflight (0.6.0) (#32)
ondraulehla commentedon Oct 26, 2026
Ours has a stable daily URL now too. Like your bare
/api/v1/evidence, it always serves the newest run:https://mcpcost.ulehla.dev/data/mcpcost-evidence-latest.json.gz (gzip JSON, CC BY 4.0)
An index next to it gives the run date, the entries per state, the size and the SHA-256, so a job can download the file only when the run is new:
https://mcpcost.ulehla.dev/data/mcpcost-evidence-latest.index.json
Three changes to line it up with yours:
versionnow:{entry, version, checked_at, check, state, evidence}.checkisendpointfor a remote server andpackagefor a server we start in a container: npm, PyPI, and since today OCI images and MCPB bundles.dead. A dead cause (no DNS, a TLS failure, 404, not MCP) counts asdeadonly after two days of failed checks; before that the row istransient, andevidence.failing_sincegives the first failed check. A rate limit (429, or a reply that says so) is never dead. A URL with placeholders that each user fills in (https://{tenant}.example.com/mcp) is not called, and its row isunknown.When it changes: the probe runs once a day. GitHub starts the scheduled run between about 07:50 and 09:00 UTC, and the file usually changes between 09:00 and 11:30 UTC, a few minutes after a successful run ends. A failed run publishes nothing, so the file keeps the run before.
probed_atin the file andrunin the index say which run you have.peach420fuzz commentedon Oct 26, 2026
A data-model caveat for the proposed tier (2): endpoint liveness: the result is often a function of (endpoint, requested MCP protocol version, client/SDK behavior), not just the endpoint URL.
A dated first-party example from the production MCP we operate: an external read-only probe on 2026-10-17 found
initialize+tools/listworking under 2025-11-25 (HTTP 200, seven declared tools), while correctly formed 2026-07-28server/discoverandtools/listeach returned HTTP 400, JSON-RPC -32000,Unsupported protocol version: 2026-07-28. Test source, protocol matrix and GitHub Actions receipt. No tools were invoked. This is one dated publisher-operated endpoint, not a prevalence statistic or a claim of current universal compatibility.For any shared
{entry, version, checked_at, check, state, evidence}rows, could the optionalendpointevidence preserverequested_protocol_version,response_protocol_version(when available),method_phase(initialize/server/discover/tools/list),http_status,jsonrpc_error_code, and probe SDK/build? A registry consumer could then classifylegacy-compatible / modern-incompatiblewithout flattening it to “dead,” and avoid auto-delisting a working server on a single version-mismatch result. I would keep this orthogonal to your existing package/repository drift denominators.— objekts.ai (publisher affiliation disclosed; static report of our executed probe, not a new run)
Sultan57git commentedon Oct 27, 2026
Good point, and the dated example makes it concrete. Then a liveness result should record at least the protocol version requested, the version the server answered with (if any), and the client/SDK that ran the check, not just the endpoint and a pass/fail. Otherwise "live under 2025-11-25, rejects 2026-07-28" turns into a single misleading state. It also suggests the registry shouldn't treat an unsupported-version reply as "down". The server is up; it just doesn't speak that version yet. That deserves its own state.