Add scripts/report_module_sizes.py - report-only LOC bands per layer
from the TeamHUB module-size guardrail (CONVENTIONS.md, module
contract p.6). Always exits 0: split-review stays a human decision.
Wire it into CI as a non-blocking step and into the AGENTS.md
baseline checks.
Document data/input + data/work as the deliberate compatibility
exception to the var/ rule of 18_REPO_LAYOUT_STANDARD.md in
docs/structure-map.md (compose mounts and APP_*_DIR depend on the
paths) and ignore var/ for new local runtime output.
Verified: script runs clean on the repo (159 files in scope, zero
flagged); ruff and compileall pass; adversarial review fixed two
classification bugs (frontend shell under frontend/src/app/,
app/api/security.py as infra helper).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The folder scanner commits each document's row inside a per-document
session_scope that only finalizes when the generator advances past its
yield. In inline mode scripts/ingest_folder.py invoked the pipeline
(which opens a fresh session) before that commit landed, so
process_document_id saw "document_missing" and the pipeline aborted at
STORED_ORIGINAL.
Draining the generator with list() forces every scanner commit before
any pipeline run. Verified end to end: a generated PDF now progresses
DISCOVERED -> ... -> CHUNKING_COMPLETED and is searchable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/generate_synthetic_pdfs.py builds real PDF/1.4 documents with
a hand-written xref so we can generate tens of thousands of ~2 KB
PDFs locally. Helvetica only covers latin-1, which is fine for a
load generator (throughput, not retrieval relevance); the docstring
calls this out so no one mistakes the output for a quality corpus.
scripts/load_ingest.py drives POST /ingest/folder, then polls a
hypothetical /documents/stats endpoint every poll-interval seconds
to track terminal-state progression. Writes a JSON history report so
results can be diffed between runs.
scripts/locustfile_search.py defines a SearchUser profile mixing
hybrid / lexical / semantic queries against POST /search plus a
health-check sampler. Asserts non-empty results so a "200 with
zero hits" regression surfaces as a failure rather than a green
percentile graph.
RUNBOOK gains a Load testing section with CPU/GPU SLO tables for
both axes (sustained docs/min, search latency p50/p95/p99).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- scripts/benchmark_reranker.py exercises the configured reranker
with synthetic queries or live OpenSearch samples and prints
p50/p95/p99 latency, mean latency, and pairs/sec throughput.
Supports --warmup, --candidates, --passage-length, --source, and a
--json-only mode for CI.
- app/indexing/reranker.py clips passages to 2048 characters before
scoring so a runaway chunk cannot starve the cross-encoder beyond
bge-reranker-v2-m3's training window.
- RUNBOOK.md gains a Reranker benchmark section with CPU/GPU SLO
targets and a remediation ladder (lower top-K, raise batch size,
switch device, disable reranker) when measured p95 exceeds budget.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
