LegacyHUB/AGENTS.md

# AGENTS — LegacyHUB

Operating instructions for AI agents working inside this repository.

## What this project is

LegacyHUB ingests legacy PDF archives at scale (~70k docs), runs OCR
(OCRmyPDF/Tesseract), extracts structured content with Docling, indexes chunks
into PostgreSQL + OpenSearch (BM25) + Qdrant (BGE-M3 dense), and serves a
hybrid lexical + semantic search API (FastAPI) reranked by BGE.

It is one module of the TeamHUB Suite.

## TeamHUB Platform standards (source of truth)

For correcting existing code and for all future development, treat the
TeamHUB platform standards in
`C:\Users\manag\TeamHUB\TeamHUB-Platform` as the target source of truth.

Before behavior-changing work, read the current working-tree version of these
documents:

- `docs/ARCHITECTURE.md`
- `docs/CONVENTIONS.md`
- `docs/INTER_MODULE_CONTRACT.md`
- `docs/submodules/README.md`
- `docs/submodules/02_MODULE_CONTRACT.md`
- `docs/submodules/03_INTEGRATION_STANDARDS.md`
- `docs/submodules/08_DECISIONS.md`
- `docs/submodules/09_ROLLOUT_AND_SSO.md`
- `docs/submodules/10_ASSET_AND_DOCUMENT_STANDARD.md`
- `docs/submodules/18_REPO_LAYOUT_STANDARD.md`
- `docs/submodules/plans/LegacyHUB.md`

If local code, README, or runbook content conflicts with those platform
standards, do not resolve the conflict silently. Surface the contradiction,
then move LegacyHUB toward the platform standard with small verified changes.
Current local behavior remains evidence; platform docs define the intended
contract.

Immediate platform implications for this repo:

- LegacyHUB is the TeamHUB pattern-B reference service.
- New TeamHUB document integrations use `POST /api/v1/knowledge-ingest` with an
  approved `AssetManifest`; `POST /api/v1/ingest/folder` is only compatibility
  for local workflows.
- LegacyHUB builds projections/indexes. It must not become the owner of source
  files owned by other modules.
- Inter-module events go through dispatch as a participant; Celery+Redis remain
  internal processing only.
- User-facing access targets gateway/SSO trusted `X-TeamHub-*` headers, with
  API-key checks as service/defense-in-depth auth.
- OCR/Markdown/Search reuse should converge on a stateless shared core rather
  than duplicating or hard-coding LegacyHUB-specific names into other modules.
- File-structure work follows `docs/structure-map.md`: current `app/`,
  `ingestion/`, `indexing` and `storage` layers are mature compatibility shapes
  and should not be renamed without a scoped migration and focused checks.

## Stack (canonical)

| Layer    | Tech                                          |
|----------|-----------------------------------------------|
| API      | FastAPI, Pydantic v2, SQLAlchemy 2, Alembic   |
| Workers  | Celery + Redis                                |
| OCR      | OCRmyPDF + Tesseract (rus+eng)                |
| Extract  | Docling                                       |
| Store    | PostgreSQL 16, MinIO, OpenSearch 2.x, Qdrant  |
| ML       | BAAI/bge-m3 (dense, 1024), bge-reranker-v2-m3 |
| Frontend | React 18, TS 5, Vite 5, Tailwind, shadcn, TanStack Query, Zustand, Framer Motion, Recharts |
| Tests    | pytest                                        |
| CI       | GitHub Actions                                |

## Entry points

- **Backend API** — `app/main.py` (`uvicorn app.main:app`)
- **Celery worker** — `celery -A app.workers.celery_app worker`
- **CLI scripts** — `scripts/init_db.py`, `scripts/init_opensearch.py`,
  `scripts/init_qdrant.py`, `scripts/ingest_folder.py`,
  `scripts/reindex_document.py`, `scripts/smoke_test.py`
- **Frontend dev** — `cd frontend && npm run dev` (port 5273)
- **Docker** — `docker compose up -d --build` (dev), `docker compose -f
  docker-compose.yml -f docker-compose.prod.yml ...` (prod)

## Inventory

```text
legacy-knowledge-indexer/
  app/
    api/             routers + Pydantic schemas
    db/              SQLAlchemy models + Alembic migrations
    indexing/        OpenSearch + Qdrant clients, embeddings, reranker, hybrid
    ingestion/       scanner, OCR, Docling, chunker, table/figure processors,
                     quality, pipeline
    storage/         MinIO client + key conventions + ensure_artifact helper
    utils/           hashing, text cleaning, language detection, pdf helpers
    workers/         Celery app + tasks
  scripts/           init / ingest / reindex / smoke CLIs
  tests/             pytest suite
  docker/Dockerfile  API + worker image (OCRmyPDF + tesseract-rus+eng)
  docker-compose.yml dev orchestration
  docker-compose.prod.yml  production overlay
  frontend/          React app — see frontend/README.md
  .github/workflows  CI gate (ruff + pytest + tsc + vite build + compose config)
```

## Code discovery order

Bounded discovery order for this repo. Use the first available that returns a
usable answer; mark the rest "not available" for the task.

1. **Zoekt / code-index** — primary local indexed search for broad source-code
   discovery, symbols, filenames, and routes. Run a bounded smoke check before
   relying on results.
2. **ast-grep / sg** — syntax-aware structural search.
3. **Serena / LSP** — semantic navigation and references when available.
4. **rg / rg --files** — reliable fallback for strings, configs, docs, scripts,
   route paths, hashes, generated files, or after recording an indexed-tool
   blocker.
5. **Docling / extracted Markdown in MinIO** — for content questions about
   ingested documents, not source code.

Index check and refresh:

```powershell
codex-session-start -RepoRoot .
repo-index-build -RepoRoot . -Rebuild
```

If any indexer times out or returns stale results, capture the error and fall
through. Do not retry the same failing indexer.

## Module contracts (high level)

- `app/ingestion/pipeline.py::process_document_id(document_id, run_id)` — single
  document end-to-end. Idempotent. Returns `{status, chunks, error?}`.
- `app/indexing/hybrid_search.py::run_search(SearchRequest) -> SearchResponse` —
  the only public search entry. Lexical + semantic + reranker.
- `app/storage/artifacts.py::ensure_artifact(...)` — single source of truth for
  `document_artifacts` upsert. Used by scanner, pipeline, table_processor,
  figure_processor.
- `app/storage/minio_client.py::MinioStorage` — bucket bootstrap + retryable
  put/get. Never bypass for object IO.
- `app/indexing/opensearch_client.py::ensure_index() / index_chunks()` — chunk
  index lifecycle.
- `app/indexing/qdrant_client.py::ensure_collection() / upsert_chunks()` —
  vector index lifecycle.

## Runtime vs legacy scope

Everything under `app/` is runtime. `scripts/` are operational tools. `tests/`
are non-runtime. There is no archived/legacy code yet.

## Baseline checks

```bash
# Backend
python -m pip check
python -m compileall -q app scripts tests
python -m pytest tests/ -q
python scripts/report_module_sizes.py   # report-only size guardrail

# Frontend
cd frontend
npx tsc --noEmit
npm run lint
npm run build

# Docker
docker compose config --quiet
```

## Operating rules for agents

- Inspect before changing. `git status` first.
- Small reviewable commits. One ownership boundary per commit.
- Do not delete files, routes, migrations, or env vars without evidence (see
  `software-project-delivery-governance` skill).
- Do not invent secret values. Use `.env.example` placeholders.
- Use `ensure_artifact` instead of re-implementing artifact upsert.
- Use existing UI primitives in `frontend/src/components/ui/*` before adding new
  ones.
- Never commit `node_modules/`, `dist/`, `.env`, `data/input/*`, `data/work/*`.
- Failures must be logged via `processing_events` (backend) or `sonner` toast
  (frontend) — not silenced.

## Ownership

- Backend, ingestion, search — Vadim Malanov.
- Frontend, design system — Vadim Malanov.

## Where to update what

- New behavior — update `README.md`.
- New repeated agent rule — update this file.
- New deployment / recovery step — update `RUNBOOK.md`.
- Cleanup findings — `docs/cleanup-report.md` (create on demand).

## Внешняя связь (агенты)

- Nextcloud Talk: `~/.usms/talk-send.ps1 -File <utf8-файл>` (room `q4toq6w4`).
- Telegram: `~/.usms/tg-send.ps1 -File <utf8-файл> [-Channel it]` — общий
  движок `teamhub-telegram-bridge` (`TeamHUB\Engines`); env с токеном/chat id —
  `~/.usms-codex/telegram-<channel>-bot.env`; значения секретов не печатать.
- Каждое внешнее сообщение подписывать: `— AI-агент TeamHUB (Claude)`.
- Созвоны и встречи от имени агента не предлагать: агент в них не участвует.