Ruff: declare adapter re-exports via __all__ in docling_extractor
(F401) and sort the engine import block in the adapter test (I001).
Pytest: engine-dependent tests now importorskip the pinned engine —
CI installs light deps only and the engine repo is private, so they
skip visibly there and run in Docker/dev envs with the engine.
Verified: ruff clean; pytest 74 passed + 2 skipped without engine,
80 passed with the pinned engine v0.1.0 installed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three defects surfaced during the first live ingest:
1. OpenSearch index mapping is dynamic:"strict", so the first chunk
carrying metadata.section_heading was rejected with
strict_dynamic_mapping_exception and zero chunks were indexed. The
metadata and quality_flags sub-objects now opt into
dynamic:true so chunker-specific keys insert without a fixed schema,
while the top-level mapping stays strict.
2. hybrid_search.run_search called get_reranker() unconditionally, so
every search - even search_mode=lexical with RERANKER_ENABLED=false
- triggered a multi-GB FlagReranker download and hung the request.
The loader is now lazy behind the reranker_enabled flag.
3. docker-compose x-common-env never forwarded RERANKER_ENABLED, so the
container always defaulted to true regardless of .env. Added
RERANKER_ENABLED: ${RERANKER_ENABLED:-true} to common-env.
After these fixes POST /search (lexical) returns HTTP 200 in ~50ms with
a ranked hit and citation; OpenSearch _count reports the indexed chunk.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds defence-in-depth shared-secret auth that activates when API_KEY
is set. Behaviour:
- empty API_KEY (dev default): every request allowed, middleware is
not even installed;
- non-empty API_KEY: every request under APP_API_PREFIX except
/health must carry X-API-Key: <value> or
Authorization: Bearer <value>. /, /docs, /redoc, /openapi.json and
CORS preflight stay open. hmac.compare_digest is used for the
constant-time comparison.
The middleware resolves settings lazily so test fixtures can reload
app.config and have the new API_KEY take effect on the next install.
Tests (tests/test_api_security.py, 5 cases):
- /health remains open;
- protected route rejects missing key (401);
- protected route accepts X-API-Key header;
- protected route accepts Authorization: Bearer header;
- protected route rejects a wrong key.
Frontend:
- VITE_API_KEY env reads the key and Axios injects it on every
request, falling back to no header when empty so SSO/reverse-proxy
deployments stay unchanged.
- vite-env.d.ts adds the new env entry.
Docs/ops:
- .env.example documents the dev-default empty key;
- .env.prod.example marks API_KEY as a required rotation point;
- docker-compose.yml forwards API_KEY (defaults to empty);
- docker-compose.prod.yml fails the stack with ?:required when API_KEY
is missing;
- RUNBOOK gains an API authentication section with header examples
and the reverse-proxy + key layering recommendation.
pytest -q: 33 passed (5 new security + 28 prior).
npx tsc --noEmit: clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- scripts/benchmark_reranker.py exercises the configured reranker
with synthetic queries or live OpenSearch samples and prints
p50/p95/p99 latency, mean latency, and pairs/sec throughput.
Supports --warmup, --candidates, --passage-length, --source, and a
--json-only mode for CI.
- app/indexing/reranker.py clips passages to 2048 characters before
scoring so a runaway chunk cannot starve the cross-encoder beyond
bge-reranker-v2-m3's training window.
- RUNBOOK.md gains a Reranker benchmark section with CPU/GPU SLO
targets and a remediation ladder (lower top-K, raise batch size,
switch device, disable reranker) when measured p95 exceeds budget.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- app/indexing/qdrant_client.py: remove the identity-only _qid()
helper and pass chunk_id straight to PointStruct (Qdrant accepts
the UUID string directly).
- services/types.ts: SearchHit gets an explicit, optional
ocr_confidence field so consumers can type the value instead of
casting through metadata.
- widgets/SearchResultCard.tsx: replaces the
(hit.metadata as { ocr_confidence? }) cast with the new field. No
behavior change when the backend omits it.
tsc --noEmit: clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The artifact-upsert helper was duplicated four times (scanner.py,
table_processor.py, figure_processor.py, pipeline.py) with slightly
different signatures. Consolidates into a single keyword-only function
keyed on (document_id, storage_key) - the identity the schema already
enforces - so re-running the pipeline never creates duplicate rows.
scanner / table_processor / figure_processor now import the shared
helper directly. pipeline.py keeps a thin local wrapper to preserve
the positional call sites at three artifact upsert points (OCR_PDF,
MARKDOWN, DOCLING_JSON).
Tests: 24 passed (5 health + 19 original).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CORS:
- New setting CORS_ALLOWED_ORIGINS (comma separated). Defaults cover
the three local Vite ports (5173, 5273, 4173); production overlay
expects the real origin in .env.prod.
- main.py wires CORSMiddleware from settings.cors_origins. No * in
production - see RUNBOOK and .env.prod.example.
- docker-compose.yml forwards the variable to both api and worker.
Tests:
- tests/test_api_health.py uses FastAPI TestClient and monkeypatches
the five probe functions (postgres/minio/opensearch/qdrant/redis).
Verifies the all-ok, any-error, and degraded paths, that the root
endpoint reports the configured api prefix, and that the CORS
preflight echoes the allowed origin.
- pytest tests/test_api_health.py -q: 5 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
