192 lines
7.5 KiB
Markdown
192 lines
7.5 KiB
Markdown
# AGENTS — LegacyHUB
|
|
|
|
Operating instructions for AI agents working inside this repository.
|
|
|
|
## What this project is
|
|
|
|
LegacyHUB ingests legacy PDF archives at scale (~70k docs), runs OCR
|
|
(OCRmyPDF/Tesseract), extracts structured content with Docling, indexes chunks
|
|
into PostgreSQL + OpenSearch (BM25) + Qdrant (BGE-M3 dense), and serves a
|
|
hybrid lexical + semantic search API (FastAPI) reranked by BGE.
|
|
|
|
It is one module of the TeamHUB Suite.
|
|
|
|
## TeamHUB Platform standards (source of truth)
|
|
|
|
For correcting existing code and for all future development, treat the
|
|
TeamHUB platform standards in
|
|
`C:\Users\manag\TeamHUB\TeamHUB-Platform` as the target source of truth.
|
|
|
|
Before behavior-changing work, read the current working-tree version of these
|
|
documents:
|
|
|
|
- `docs/ARCHITECTURE.md`
|
|
- `docs/CONVENTIONS.md`
|
|
- `docs/INTER_MODULE_CONTRACT.md`
|
|
- `docs/submodules/README.md`
|
|
- `docs/submodules/02_MODULE_CONTRACT.md`
|
|
- `docs/submodules/03_INTEGRATION_STANDARDS.md`
|
|
- `docs/submodules/08_DECISIONS.md`
|
|
- `docs/submodules/09_ROLLOUT_AND_SSO.md`
|
|
- `docs/submodules/10_ASSET_AND_DOCUMENT_STANDARD.md`
|
|
- `docs/submodules/plans/LegacyHUB.md`
|
|
|
|
If local code, README, or runbook content conflicts with those platform
|
|
standards, do not resolve the conflict silently. Surface the contradiction,
|
|
then move LegacyHUB toward the platform standard with small verified changes.
|
|
Current local behavior remains evidence; platform docs define the intended
|
|
contract.
|
|
|
|
Immediate platform implications for this repo:
|
|
|
|
- LegacyHUB is the TeamHUB pattern-B reference service.
|
|
- New TeamHUB document integrations use `POST /api/v1/knowledge-ingest` with an
|
|
approved `AssetManifest`; `POST /api/v1/ingest/folder` is only compatibility
|
|
for local workflows.
|
|
- LegacyHUB builds projections/indexes. It must not become the owner of source
|
|
files owned by other modules.
|
|
- Inter-module events go through dispatch as a participant; Celery+Redis remain
|
|
internal processing only.
|
|
- User-facing access targets gateway/SSO trusted `X-TeamHub-*` headers, with
|
|
API-key checks as service/defense-in-depth auth.
|
|
- OCR/Markdown/Search reuse should converge on a stateless shared core rather
|
|
than duplicating or hard-coding LegacyHUB-specific names into other modules.
|
|
|
|
## Stack (canonical)
|
|
|
|
| Layer | Tech |
|
|
|----------|-----------------------------------------------|
|
|
| API | FastAPI, Pydantic v2, SQLAlchemy 2, Alembic |
|
|
| Workers | Celery + Redis |
|
|
| OCR | OCRmyPDF + Tesseract (rus+eng) |
|
|
| Extract | Docling |
|
|
| Store | PostgreSQL 16, MinIO, OpenSearch 2.x, Qdrant |
|
|
| ML | BAAI/bge-m3 (dense, 1024), bge-reranker-v2-m3 |
|
|
| Frontend | React 18, TS 5, Vite 5, Tailwind, shadcn, TanStack Query, Zustand, Framer Motion, Recharts |
|
|
| Tests | pytest |
|
|
| CI | GitHub Actions |
|
|
|
|
## Entry points
|
|
|
|
- **Backend API** — `app/main.py` (`uvicorn app.main:app`)
|
|
- **Celery worker** — `celery -A app.workers.celery_app worker`
|
|
- **CLI scripts** — `scripts/init_db.py`, `scripts/init_opensearch.py`,
|
|
`scripts/init_qdrant.py`, `scripts/ingest_folder.py`,
|
|
`scripts/reindex_document.py`, `scripts/smoke_test.py`
|
|
- **Frontend dev** — `cd frontend && npm run dev` (port 5273)
|
|
- **Docker** — `docker compose up -d --build` (dev), `docker compose -f
|
|
docker-compose.yml -f docker-compose.prod.yml ...` (prod)
|
|
|
|
## Inventory
|
|
|
|
```text
|
|
legacy-knowledge-indexer/
|
|
app/
|
|
api/ routers + Pydantic schemas
|
|
db/ SQLAlchemy models + Alembic migrations
|
|
indexing/ OpenSearch + Qdrant clients, embeddings, reranker, hybrid
|
|
ingestion/ scanner, OCR, Docling, chunker, table/figure processors,
|
|
quality, pipeline
|
|
storage/ MinIO client + key conventions + ensure_artifact helper
|
|
utils/ hashing, text cleaning, language detection, pdf helpers
|
|
workers/ Celery app + tasks
|
|
scripts/ init / ingest / reindex / smoke CLIs
|
|
tests/ pytest suite
|
|
docker/Dockerfile API + worker image (OCRmyPDF + tesseract-rus+eng)
|
|
docker-compose.yml dev orchestration
|
|
docker-compose.prod.yml production overlay
|
|
frontend/ React app — see frontend/README.md
|
|
.github/workflows CI gate (ruff + pytest + tsc + vite build + compose config)
|
|
```
|
|
|
|
## Code discovery order
|
|
|
|
Bounded discovery order for this repo. Use the first available that returns a
|
|
usable answer; mark the rest "not available" for the task.
|
|
|
|
1. **Zoekt / code-index** — primary local indexed search for broad source-code
|
|
discovery, symbols, filenames, and routes. Run a bounded smoke check before
|
|
relying on results.
|
|
2. **ast-grep / sg** — syntax-aware structural search.
|
|
3. **Serena / LSP** — semantic navigation and references when available.
|
|
4. **rg / rg --files** — reliable fallback for strings, configs, docs, scripts,
|
|
route paths, hashes, generated files, or after recording an indexed-tool
|
|
blocker.
|
|
5. **Docling / extracted Markdown in MinIO** — for content questions about
|
|
ingested documents, not source code.
|
|
|
|
Index check and refresh:
|
|
|
|
```powershell
|
|
codex-session-start -RepoRoot .
|
|
repo-index-build -RepoRoot . -Rebuild
|
|
```
|
|
|
|
If any indexer times out or returns stale results, capture the error and fall
|
|
through. Do not retry the same failing indexer.
|
|
|
|
## Module contracts (high level)
|
|
|
|
- `app/ingestion/pipeline.py::process_document_id(document_id, run_id)` — single
|
|
document end-to-end. Idempotent. Returns `{status, chunks, error?}`.
|
|
- `app/indexing/hybrid_search.py::run_search(SearchRequest) -> SearchResponse` —
|
|
the only public search entry. Lexical + semantic + reranker.
|
|
- `app/storage/artifacts.py::ensure_artifact(...)` — single source of truth for
|
|
`document_artifacts` upsert. Used by scanner, pipeline, table_processor,
|
|
figure_processor.
|
|
- `app/storage/minio_client.py::MinioStorage` — bucket bootstrap + retryable
|
|
put/get. Never bypass for object IO.
|
|
- `app/indexing/opensearch_client.py::ensure_index() / index_chunks()` — chunk
|
|
index lifecycle.
|
|
- `app/indexing/qdrant_client.py::ensure_collection() / upsert_chunks()` —
|
|
vector index lifecycle.
|
|
|
|
## Runtime vs legacy scope
|
|
|
|
Everything under `app/` is runtime. `scripts/` are operational tools. `tests/`
|
|
are non-runtime. There is no archived/legacy code yet.
|
|
|
|
## Baseline checks
|
|
|
|
```bash
|
|
# Backend
|
|
python -m pip check
|
|
python -m compileall -q app scripts tests
|
|
python -m pytest tests/ -q
|
|
|
|
# Frontend
|
|
cd frontend
|
|
npx tsc --noEmit
|
|
npm run lint
|
|
npm run build
|
|
|
|
# Docker
|
|
docker compose config --quiet
|
|
```
|
|
|
|
## Operating rules for agents
|
|
|
|
- Inspect before changing. `git status` first.
|
|
- Small reviewable commits. One ownership boundary per commit.
|
|
- Do not delete files, routes, migrations, or env vars without evidence (see
|
|
`software-project-delivery-governance` skill).
|
|
- Do not invent secret values. Use `.env.example` placeholders.
|
|
- Use `ensure_artifact` instead of re-implementing artifact upsert.
|
|
- Use existing UI primitives in `frontend/src/components/ui/*` before adding new
|
|
ones.
|
|
- Never commit `node_modules/`, `dist/`, `.env`, `data/input/*`, `data/work/*`.
|
|
- Failures must be logged via `processing_events` (backend) or `sonner` toast
|
|
(frontend) — not silenced.
|
|
|
|
## Ownership
|
|
|
|
- Backend, ingestion, search — Vadim Malanov.
|
|
- Frontend, design system — Vadim Malanov.
|
|
|
|
## Where to update what
|
|
|
|
- New behavior — update `README.md`.
|
|
- New repeated agent rule — update this file.
|
|
- New deployment / recovery step — update `RUNBOOK.md`.
|
|
- Cleanup findings — `docs/cleanup-report.md` (create on demand).
|