Repository navigation
[epic] Build authorized document retrieval and member search #125
Description
Area
bot, service, docs-system
Outcome
The helper bot needs retrievable document content, but fetching text alone does not provide search. This epic connects content acquisition (#82) to the helper experience (#81) and delivers a usable /doc search command independently of conversational answers.
Definition of done
- [task]: Decide the initial chunking and retrieval strategy #208 records chunk-size/overlap and hybrid-versus-vector recommendations using real catalogued Google content.
- [task]: Add authenticated batch embeddings to the LLM service #173 exposes authenticated batch embeddings with model/dimension metadata, and [task]: Store document-chunk embeddings in pgvector #174 stores document-linked vectors in pgvector.
- [task]: Split document text into deterministic chunks with source offsets #218 implements deterministic text chunking with source offsets independently of the embedding service and database.
- [task]: Index document chunks and maintain their lifecycle #175 indexes catalog entities on ingest/content replacement and via an authenticated admin operation, with idempotent replacement and deletion.
- [task]: Re-embed indexed chunks after an embedding-model change #209 detects incompatible model/dimension metadata and supports deliberate, resumable re-embedding with measured cost/duration documented.
- [task] Build the authorized retrieval engine #176 implements the real retrieval engine against seeded chunks, with mandatory person/channel authorization before ranking, source citations, configurable k and a relevance floor.
- [task] Verify indexing to authorized retrieval integration #277 verifies newly ingested and updated documents through the real indexing-to-retrieval path, including deletion, revocation and failures. This is a required October 23 release gate; component completion alone is insufficient.
- [task]: Measure retrieval quality against a fixed question set #178 reports retrieval hit rate at k against a small fixed question/source set.
- [task]: Add member-facing document search to the bot #210 exposes ranked, authorized document results through
/doc search, including no-match and unavailable-service responses.
Ownership
services/llm owns embedding generation (#173). services/documentation-system owns chunking, vector storage, indexing lifecycle, retrieval, and model-change recovery. discord-bot owns member search (#210). #178 owns the evaluation harness; its repository location should be chosen with the existing tooling patterns. Source fetching (#82), conversational answers (#81), and durable meetings (#211) remain separate.
Parallel work
#175 joins the completed chunker (#218), embeddings (#173) and vector storage (#174). #176 can run alongside #175 using seeded chunks and the completed #174/#238 contracts; it no longer waits for indexing. Angela first publishes retrieval request/response examples and a shared fixture for William (#71) and Jessica (#210). #277 joins #175 and #176 for integrated lifecycle and authorization verification. #209 still follows #174/#175, while #178 and #210 retain #176 as their completion prerequisite. Native Relationships record hard blockers; #76 supplies source-permission reconciliation for end-to-end authorization.
Task breakdown
| Task | Area | Effort |
|---|---|---|
| #208 — Decide the initial chunking and retrieval strategy | service, docs-system | S |
| #218 — Split document text into deterministic chunks with source offsets | service, docs-system | M |
| #173 — Add authenticated batch embeddings to the LLM service | service | M |
| #174 — Store document-chunk embeddings in pgvector | service, docs-system | S |
| #175 — Index document chunks and maintain their lifecycle | service, docs-system | L |
| #176 — Build the authorized retrieval engine | service, docs-system | M |
| #277 — Verify indexing to authorized retrieval integration | service, docs-system, integration | TBD |
| #209 — Re-embed indexed chunks after an embedding-model change | service, docs-system | L |
| #178 — Measure retrieval quality against a fixed question set | tooling, docs-system | S |
| #210 — Add member-facing document search to the bot | bot, docs-system | S |
Earlier decisions and completed history
Closed children: #177. Their existing issue states and discussions retain the completion, cancellation, or supersession details.
Scope and implementation notes
Build order / children: #208 → #218; #173 → #174; #218 + #173 + #174 → #175; #174 + #238 → #176; #175 + #176 → #277; #174 + #175 → #209; #176 → #178 and #210. #277 is a release gate, with automated integration complete by October 21 and release acceptance before October 23. GitHub's native Relationships are authoritative for blockers.
Authorization decision: Filtering is part of retrieval, before ranking—not a later bot or LLM filter. Extend #74's doc_visible() / doc_grants rules for authenticated person and channel contexts. Direct document search remains person-scoped; channel answers use group entitlement without the asker's private grants. #177 was merged into #176 and closed as superseded; do not recreate a separate authorization task or unfiltered endpoint.
Indexing decision: Use the existing ingest/content-replacement path plus an authenticated admin re-index operation. No scheduler, queue, or job runner inside the catalog; see docs/ARCHITECTURE.md. Key indexing on catalog entities, not the URL-fetch path, so future meeting records (#211) are not excluded.
Available foundations: #83/#84 deliver stored Google text; #74 supplies grants; #42/#41 supply the LLM provider/auth patterns; catalog CI already exercises Postgres migrations. Google content is sufficient to start. #86 can widen the corpus later; #85 (Notion) is canceled.
Split recorded: #218 owns pure chunking; #175 consumes it and keeps its full indexing/lifecycle scope. #176 owns authorized query implementation against seeded chunks; #277 owns the indexing-to-retrieval acceptance formerly implied by their serial dependency. Authorization stays mandatory in #176. #208 records strategy in services/documentation-system/docs/ARCHITECTURE.md.
Open questions: Absolute versus relative relevance floor; whether search is deliberately narrower than listing while never exceeding the existing visibility rules. Embedding-model selection remains in #173.
Out of scope: Source fetching (#82), conversational answers (#81), tool use (#70/#71), persisting meeting records (#211), and a general scheduled-jobs system. A future scheduler can call the admin operation.
Working conventions
Filed tasks are direct sub-issues of this epic. GitHub Relationships record the existing hard prerequisites; the task table summarizes the work and existing project Size estimates. Keep each task independently reviewable and normally within one CODEOWNERS zone; explain deliberate cross-zone changes. Misty issues may carry several area labels; zones apply to PRs. Follow AGENTS.md and the affected service's CONTRIBUTING.md for implementation, tests, and documentation.
Activity
[-]RAG retrieval layer: /embed + pgvector store + actor-scoped similarity search[/-][+]Epic: RAG retrieval layer — /embed + pgvector store + actor-scoped similarity search[/+]1 remaining item
[-]Epic: RAG retrieval layer — /embed + pgvector store + actor-scoped similarity search[/-][+]Build actor-scoped document retrieval and member search[/+][-]Build actor-scoped document retrieval and member search[/-][+][epic] Build actor-scoped document retrieval and member search[/+][-][epic] Build actor-scoped document retrieval and member search[/-][+][epic] Build authorized document retrieval and member search[/+]