[hyoban]

Summary

Integration branch for KnowledgeFS and the New Knowledge experience (backend proxy/services plus the web/features/new-rag frontend).

Main dropped its outdated copy of this feature in #44163. This branch carries the current version, and its latest main merge records that commit without applying its deletions.

Upgrade reconciliation recovery

Classic Dataset upgrades can commit success before their document reconciliation is published. A broker failure or interrupted worker previously left disabled-document availability pending indefinitely, blocking retrieval from the new knowledge base.

Persist an independent reconciliation delivery lease and let Beat recover successful upgrades with unfinished document checkpoints. Bound worker execution and reject stale or duplicate deliveries. Apply migration b9d3f7a5c102 before starting the updated processes; existing stranded upgrades are eligible for recovery automatically. Deployment and recovery requirements are documented in docker/README.md.

Validation for this fix: 83 focused upgrade, task, and migration tests passed, including publication failure, interruption before claiming the follow-up, lost deliveries, and stale-worker fencing. Backend lint checks and targeted production/test type checks passed. PostgreSQL/MySQL migration SQL rendering was checked; live database integration tests remain CI-owned.

Screenshots

Before	After
...	...

Checklist

• This change requires a documentation update, included: Dify Document
• I understand that this PR may be closed in case there was no previous discussion or issues. (This doesn't apply to typos!)
• I've verified the change and added or updated tests where meaningful regression risk justifies coverage.
• I've updated the documentation accordingly.
• I ran make lint && make type-check (backend) and vp staged (frontend) to appease the lint gods

From Claude Code

🤖 Generated with Claude Code

[hyoban]

Thanks for the thorough review. Each point is addressed in its own commit:

1. Migration ancestry: b98c951. fbdfcf5f5a6e is back on main's parent, 89919253ca7a. 9a4e7d1c2b60 now merges f1d3c5b7a902 and c4d8e2f7a1b6 itself, since its foreign key needs the KnowledgeFS tables. Graph tests check that an upgrade from main's a8c3e7f4b912 runs the full KnowledgeFS chain before 9a4e7d1c2b60. They also check that databases already at d7b9e1f3a5c2 keep the same set of applied revisions.
2. API-key scope on downgrade: f45e8ce. The downgrade now refuses to run while any key is bound only to KnowledgeFS spaces. The operator has to revoke those keys through the API first, which also invalidates their cache. I chose blocking over revoking inside the migration because migrations can't clear the Redis token cache, so revoked keys would stay usable for up to the cache TTL. Keys that also have dataset bindings keep those bindings. Offline (SQL-only) downgrades are rejected because they can't run this check. Tests cover a populated downgrade with both kinds of key.
3. Re-cutover: 3783f94. The cutover update now clears rolled_back_at. A new cutover → rollback → cutover test checks that the runtime gate lets traffic through again.
4. Interrupted dispatch: c0cd0ee. After each checkpoint, the job goes from RUNNING back to QUEUED with no task id, then dispatches through the same claim_enqueue / release-on-failure path the console uses. If the broker fails, the job is left QUEUED and unclaimed, so retry() can pick it up. One gap remains: a worker killed between the claim and apply_async leaves the job claimed. Initial enqueues have the same gap today. Closing it needs a periodic sweep, which could be a follow-up.
5. Worker image: aa018c0. Both Compose files now use dify-api:1.17.1, matching the other API services.
6. MySQL uniqueness: 46c7d23. A new head, 3edd6cd27f54, applies only on MySQL. It replaces the unconditional index with a unique index on a stored generated column that is NULL for finished jobs, the same approach as agents.roster_unique_name. PostgreSQL keeps its partial index. This is checked against rendered MySQL SQL only, so the MySQL CI job is the real test.
7. Completed rows on downgrade: 8871ead. Completed authorizations already carry the full started evidence, so they go back to started before the old check constraint is restored. A test covers a populated downgrade.