LegacyHUB/README.md

# LegacyHUB - Knowledge Indexing & Hybrid Search for Legacy PDF Archives

LegacyHUB is a production-oriented, fully open-source backend for ingesting,
OCR-ing, structurally extracting, and hybrid-searching large legacy PDF
archives (designed for ~70,000 documents).

It is part of the **TeamHUB** suite.

```
PDFs ──▶ Scanner ──▶ MinIO (originals)
                  └▶ OCRmyPDF (Tesseract) ──▶ MinIO (ocr_pdf)
                                          └▶ Docling ──▶ MD + JSON ──▶ MinIO
                                                       └▶ blocks/tables/figures
                                                                ├▶ PostgreSQL
                                                                ├▶ OpenSearch (BM25)
                                                                └▶ Qdrant (BGE-M3 dense)
                                                                          │
FastAPI /search ◀── BGE Reranker ◀── RRF merge ◀───────────────────────────┘
```

## Stack

| Component        | Tech                                     |
|------------------|------------------------------------------|
| OCR              | `teamhub-document-recognition-engine` over OCRmyPDF + Tesseract (rus + eng) |
| Extraction       | `teamhub-document-recognition-engine` over Docling (layout, tables, figures) |
| Object storage   | MinIO (S3-compatible)                    |
| Relational store | PostgreSQL 16                            |
| Lexical search   | OpenSearch 2.x (BM25 + ru/en analyzers)  |
| Vector search    | Qdrant 1.x (named dense vector)          |
| Embeddings       | BAAI/bge-m3 (dense, 1024d)               |
| Reranker         | BAAI/bge-reranker-v2-m3                  |
| API              | FastAPI + Uvicorn                        |
| Workers          | Celery + Redis                           |
| Logging          | structlog (JSON)                         |

## Quick start

```bash
cp .env.example .env
[ "$TEAMHUB_ALLOW_LOCAL" = "1" ] || { echo "Local docker deploy disabled: TeamHUB runs on server 10.37.142.35. Set TEAMHUB_ALLOW_LOCAL=1 for a local bring-up (debug only)." >&2; exit 1; }
docker compose up -d --build
docker compose exec api python scripts/init_db.py
docker compose exec api python scripts/init_opensearch.py
docker compose exec api python scripts/init_qdrant.py
docker compose exec api python scripts/smoke_test.py
```

Health check:

```bash
curl http://localhost:8050/api/v1/health | jq .
```

Open the interactive Swagger docs at <http://localhost:8050/docs>.

## Ingest TeamHUB assets

TeamHUB modules should call `POST /api/v1/knowledge-ingest` with an
`AssetManifest`. The original file stays in the owner module's MinIO/S3 bucket;
LegacyHUB reads it by `bucket` + `object_key`, verifies `size_bytes` and
`sha256`, records an idempotent ingest job by
`asset_id + sha256 + manifest_version`, then builds OCR/search projections.

```bash
curl -X POST http://localhost:8050/api/v1/knowledge-ingest \
  -H "Content-Type: application/json" \
  -H "X-API-Key: ${LEGACYHUB_API_KEY}" \
  -H "Idempotency-Key: evt_01J..." \
  -d '{
        "manifest": {
          "manifest_version": "1.0",
          "asset": {
            "asset_id": "asset_01J...",
            "owner_module": "mailhub",
            "owner_record_type": "document",
            "owner_record_id": "doc_01J...",
            "asset_kind": "document",
            "title": "incoming-letter.pdf",
            "original_filename": "incoming-letter.pdf",
            "content_type": "application/pdf",
            "size_bytes": 184220,
            "sha256": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
            "created_at": "2026-06-14T12:00:00Z",
            "created_by": "staff:123"
          },
          "storage": {
            "provider": "s3",
            "bucket": "teamhub-mailhub-originals",
            "object_key": "mailhub/2026/06/14/asset_01J/original/incoming-letter.pdf",
            "version_id": null
          },
          "security": {
            "gate_status": "approved",
            "classification": "internal",
            "av_status": "clean",
            "content_scan_status": "clean",
            "pii_status": "reviewed",
            "approved_at": "2026-06-14T12:03:00Z",
            "approved_by": "staff:123",
            "quarantine_reason": null
          },
          "retention": {
            "policy_id": "mailhub-default",
            "retain_until": null,
            "legal_hold": false,
            "legal_hold_reason": null
          },
          "derivatives": [],
          "links": []
        },
        "ingest_options": {
          "generate_derivatives": true,
          "index_full_text": true,
          "index_vectors": true,
          "emit_events": true
        }
      }'
```

Manifests with `security.gate_status != "approved"` are rejected without reading
object storage. Container-local paths such as `/data/input/...`, `C:\...`, or
`file://...` are invalid in the manifest contract.

## Ingest local folders (deprecated)

Mount a folder into the container at `/data/input` (the compose file already
mounts `./data/input` for you), drop PDFs into it, and call the compatibility
endpoint:

```bash
curl -X POST http://localhost:8050/api/v1/ingest/folder \
  -H "Content-Type: application/json" \
  -d '{"path":"/data/input","recursive":true,"force":false}'
```

Or run inline (no Celery, useful for ad-hoc tests):

```bash
docker compose exec api python scripts/ingest_folder.py \
  --path /data/input --recursive --mode inline
```

To re-process a single document by ID:

```bash
docker compose exec api python scripts/reindex_document.py \
  --document-id <uuid>
```

## Search

```bash
curl -X POST http://localhost:8050/api/v1/search \
  -H "Content-Type: application/json" \
  -d '{
        "query": "ГОСТ 21.501-93 рабочие чертежи",
        "limit": 10,
        "search_mode": "hybrid",
        "filters": {"min_ocr_confidence": 0.5}
      }' | jq .
```

`search_mode` can be `lexical`, `semantic`, or `hybrid`. Hybrid mode does:

1. BM25 top-K from OpenSearch
2. Dense top-K from Qdrant (BGE-M3)
3. Reciprocal Rank Fusion merge
4. Top 30-50 candidates re-scored by the BGE reranker (if available)
5. Final top-N returned with citation metadata

Each hit includes the document name, page, block id, table/figure id where
applicable, and quality flags - so AI consumers can produce verifiable answers
with citations.

## Inspect the system

| Service       | URL                                  | Credentials                |
|---------------|--------------------------------------|----------------------------|
| API docs      | <http://localhost:8050/docs>         | -                          |
| MinIO console | <http://localhost:9001>              | use `.env` or `.env.example` placeholders; do not reuse literal secrets from docs |
| OpenSearch    | <http://localhost:9200>              | -                          |
| Qdrant UI     | <http://localhost:6333/dashboard>    | -                          |
| Postgres      | `localhost:5440`                     | use local runtime credentials from `.env`; docs intentionally avoid literal reusable values |

```bash
# Count docs in OpenSearch
curl 'http://localhost:9200/legacy_chunks/_count'
# Inspect Qdrant collection
curl 'http://localhost:6333/collections/legacy_chunks'
# Browse Postgres
docker compose exec postgres psql -U legacyhub -d legacyhub \
  -c "SELECT id, original_file_name, status FROM documents LIMIT 20;"
```

## Environment variables

See [`.env.example`](.env.example) for the full list. Key ones:

- `OCR_LANGUAGES` - Tesseract language packs passed to the shared recognition engine (default `rus+eng`).
- `OCR_ENABLED` - set `false` to skip OCR completely.
- `DOCLING_OCR_ENABLED` - prefer OCRmyPDF; only enable if you do not run OCRmyPDF.
- `EMBEDDING_DEVICE` / `RERANKER_DEVICE` - `cpu`, `cuda`, or `mps`.
- `MAX_DOCUMENT_TIMEOUT_SECONDS` - per-document soft timeout for extraction.

## Handling poor OCR

- The pipeline computes per-chunk `quality_flags`:
  - `low_ocr_confidence`, `very_short_text`, `possible_garbled_text`
  - `table_detected`, `figure_detected`, `handwriting_detected`
  - `needs_manual_review` (any of the above except table/figure detection)
- Garbled chunks are still indexed - so they remain searchable - but the flags
  let you filter them out at query time via `filters.min_ocr_confidence`.
- Original text is always preserved verbatim (no destructive cleaning); the
  `normalized_text` field is a derived form used purely for recall.
- We deliberately preserve technical / legal identifiers (ГОСТ, document
  numbers, dates, serials, slashes, dashes, dots, brackets) during normalization.

## Handling handwriting

- We do not attempt to recognize handwriting reliably. Suspected handwritten
  fragments are flagged with `block_type=handwriting` and
  `quality_flags.handwriting_detected=true` plus `needs_manual_review=true`.
- The API does not present handwriting recognition output as authoritative.

## Idempotency

- Document identity = SHA256 of the original PDF. Re-ingesting the same PDF
  reuses the existing `documents` row.
- The pipeline deletes existing chunks for the document and re-creates them
  before re-indexing; OpenSearch and Qdrant entries are deleted-by-document
  before re-upsert. So re-running ingestion does not duplicate data.

## Failure handling

- Each pipeline stage records a row in `processing_events` with `level` and
  `data` JSON.
- A document that fails OCR is marked `OCR_FAILED` and the pipeline moves on.
- A document that fails Docling is marked `EXTRACTION_FAILED`.
- Indexing failures bring the document to `FAILED`; re-running
  `scripts/reindex_document.py` resumes processing.

## Scaling notes (~70k PDFs)

- The Celery `worker` service is horizontally scalable: `docker compose up -d
  --scale worker=8` (or run several Compose stacks pointing at the same
  Postgres / MinIO / OpenSearch / Qdrant).
- The embedding step is the biggest cost. Set `EMBEDDING_DEVICE=cuda` and a
  GPU-aware worker image if available.
- OpenSearch defaults to 1 shard / 0 replicas - increase for production
  (`PUT /legacy_chunks/_settings`).
- Qdrant is single-node by default; for very large corpora use the cluster
  build of Qdrant or shard by document hash.
- For 70k PDFs at ~50 chunks each, expect ~3.5M vectors. BGE-M3 dense at 1024d
  is ~14 GB on disk; budget memory accordingly.

## Tests

```bash
pip install -e ".[dev]"
pytest -q
```

The unit suite covers hashing, chunking, quality flags, hybrid result merging,
and duplicate detection. Integration tests run against the live Compose stack
via `scripts/smoke_test.py`.

## Repository layout

```
legacy-knowledge-indexer/
  app/
    api/            # FastAPI routes & schemas
    db/             # SQLAlchemy models + Alembic migrations
    indexing/       # OpenSearch, Qdrant, embeddings, reranker, hybrid search
    ingestion/      # scanner, OCR, Docling, chunking, quality, pipeline
    storage/        # MinIO client + key conventions
    utils/          # hashing, text cleaning, language detection, PDF helpers
    workers/        # Celery app + tasks
  scripts/          # init / ingest / reindex / smoke
  tests/            # unit tests
  docker/Dockerfile # API + worker image
  docker-compose.yml
  .env.example
  pyproject.toml
  alembic.ini
```

## Known limitations

- Docling's exact JSON shape varies between versions. The extractor uses
  defensive lookups and falls back to `paragraph` when a label is unknown.
- We do not currently ship a sparse vector path (BGE-M3 supports it). Hybrid
  recall is achieved via OpenSearch BM25 + Qdrant dense, merged with RRF -
  which has been observed to outperform sparse-only or dense-only setups on
  noisy OCR.
- Figure description does not invoke a VLM; captions plus a placeholder are
  used. Plug a VLM into `figure_processor.persist_figures` if needed.
- Auth is layered: an optional shared-secret `X-API-Key` (defence in depth,
  mandatory on ingest endpoints when set) plus gateway/SSO trusted `X-TeamHub-*`
  headers when `AUTH_REQUIRE_IDENTITY=true`. The module performs no own login and
  must be reachable only behind the gateway (see RUNBOOK "Network model").

## License

Apache-2.0.