fix(indexing): unblock chunk indexing and gate reranker model load
Three defects surfaced during the first live ingest:
1. OpenSearch index mapping is dynamic:"strict", so the first chunk
carrying metadata.section_heading was rejected with
strict_dynamic_mapping_exception and zero chunks were indexed. The
metadata and quality_flags sub-objects now opt into
dynamic:true so chunker-specific keys insert without a fixed schema,
while the top-level mapping stays strict.
2. hybrid_search.run_search called get_reranker() unconditionally, so
every search - even search_mode=lexical with RERANKER_ENABLED=false
- triggered a multi-GB FlagReranker download and hung the request.
The loader is now lazy behind the reranker_enabled flag.
3. docker-compose x-common-env never forwarded RERANKER_ENABLED, so the
container always defaulted to true regardless of .env. Added
RERANKER_ENABLED: ${RERANKER_ENABLED:-true} to common-env.
After these fixes POST /search (lexical) returns HTTP 200 in ~50ms with
a ranked hit and citation; OpenSearch _count reports the indexed chunk.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -29,6 +29,7 @@ x-common-env: &common-env
|
||||
EMBEDDING_DEVICE: ${EMBEDDING_DEVICE:-cpu}
|
||||
RERANKER_MODEL: ${RERANKER_MODEL:-BAAI/bge-reranker-v2-m3}
|
||||
RERANKER_DEVICE: ${RERANKER_DEVICE:-cpu}
|
||||
RERANKER_ENABLED: ${RERANKER_ENABLED:-true}
|
||||
APP_LOG_LEVEL: ${APP_LOG_LEVEL:-INFO}
|
||||
APP_INPUT_DIR: /data/input
|
||||
APP_WORK_DIR: /data/work
|
||||
@@ -44,7 +45,7 @@ services:
|
||||
POSTGRES_USER: ${POSTGRES_USER:-legacyhub}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-legacyhub}
|
||||
ports:
|
||||
- "5432:5432"
|
||||
- "5440:5432"
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
|
||||
Reference in New Issue
Block a user