Vadim_Malanov/LegacyHUB
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(ingestion): materialize scanner generator before inline processing

Browse Source 
The folder scanner commits each document's row inside a per-document
session_scope that only finalizes when the generator advances past its
yield. In inline mode scripts/ingest_folder.py invoked the pipeline
(which opens a fresh session) before that commit landed, so
process_document_id saw "document_missing" and the pipeline aborted at
STORED_ORIGINAL.

Draining the generator with list() forces every scanner commit before
any pipeline run. Verified end to end: a generated PDF now progresses
DISCOVERED -> ... -> CHUNKING_COMPLETED and is searchable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Vadim Malanov
2026-06-14 13:41:53 +03:00
parent 0a407f1f09
commit 2806aaf441
1 changed files with 8 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
scripts/ingest_folder.py
View File

@@ -28,7 +28,14 @@ def main(path: Path, recursive: bool, force: bool, mode: str) -> None:
run_id = uuid.uuid4() run_id = uuid.uuid4()
discovered = queued = dups = invalid = 0 discovered = queued = dups = invalid = 0
for record in discover_documents(path, recursive=recursive, force=force): # Materialize generator first: scanner's per-document session_scope only
# commits when the generator advances past the yield. Inline mode invokes
# the pipeline (which opens a fresh session) before that commit happens,
# so the document row is invisible. List() drains the generator end to end,
# forcing all commits before any processing.
records = list(discover_documents(path, recursive=recursive, force=force))
for record in records:
discovered += 1 discovered += 1
if record.duplicate and not force: if record.duplicate and not force:
dups += 1 dups += 1