cid 9ee806d1ab35ead86758f492b08cd822437edf5e2c8b50aeb3395f283099964e · original · open access · via openalex
Title: Extracting AI agent-accessible data from biodiversity literature with corpus Authors: Samuel H. Church; Maciej Karol Mańko; Felipe Zapata; Casey W. Dunn Venue: Zenodo (CERN European Organization for Nuclear Research) Year: 2026 Canonical: doi:10.5281/zenodo.19964909 URL: https://doi.org/10.5281/zenodo.19964909 Collector: openalex OA: true ABSTRACT A hardening + iteration release. v0.2 closes out v0.1's deferred items (vision pass at corpus scale, expanded lexicon + taxonomy support, Bouchet batch-script fixes), adds a real update lifecycle (per-stage resume, input fingerprints, update_corpus.py orchestrator), and lays in a robustness + observability layer (structured stage_failures[], silent-failure quality gates, corpus_status rollup, network circuit breakers). Three substantive bugs surfaced during release validation and ship in this version: a SLURM-array slicing race that corrupted per-doc state files (#55), an input_fingerprint gap that silently skipped lexicon refreshes on --resume (#56), and a broken tesseract-data-* block in environment.yaml that broke fresh installs (#52). The deploy stack also moved from single-EC2 + Let's Encrypt to a shared ALB + ACM pattern that supports multiple organism corpuscles on one hostname-routed load balancer. Geographic extraction (#13) and trait extraction (#14) are pushed to later releases. Added Update lifecycle update_corpus.py orchestrator (#32) — one command runs the pipeline + every post-pipeline script in dependency order with --resume. Forwards the full Stage 1 flag surface so callers don't lose pipeline knobs. Makes "add papers and update everything" a one-liner. Granular per-stage resume (#28) — replaces the all-or-nothing summary.json completion marker with per-stage status tracked in pipeline_state.json. Stages whose artifact is already present and whose inputs haven't drifted are skipped on --resume. Input fingerprints on annotation artifacts (#29) — lexicon_sha256 and taxonomy_snapshot_date are stamped into chunks.json / taxa.json / anatomy.json so staleness is detectable without re-reading the source files. --re-annotate-stale flag (#33) — re-runs only the lexicon categories whose fingerprint changed. Subsumed by per-stage resume + per-category fingerprints. Idempotency contracts on post-pipeline scripts (#30) — build_biblio_authority, build_taxon_mentions, backfill_intext_citations, and reconcile_corpus_to_biblio audited + tested for re-run safety. --audit-orphans (#31) — read-only listing of documents// directories (and LanceDB rows) whose source PDF is no longer in the input set. Deletion stays manual. Robustness + observability Structured stage_failures[] + per-stage timing (#34) — reason codes (timeout, crash, external_unavailable, unsupported_format, corrupted, quality_gate) replace v0.1's free-text errors[]. Every downstream tool reads from the new schema. Silent-failure quality gates (#36) — flag empty text layers, gibberish OCR output, all-black figures, zero-reference papers, and collapsed-extraction chunks. Surfaces what v0.1 was happily writing without complaint. corpus_status (#40) — single command rolls up stage completion, failure breakdown by reason, quality flags, and staleness across the whole build. Dashboard for everything else in this section and the update lifecycle. Huge-document hard cap (#35) — page-count gate with a structured too_large flag for monographs that would blow past reasonable wallclock. Haeckel 1888 Challenger Siphonophorae is the canary. External-service flakiness layer (#37) — shared retry + backoff + circuit breaker + cache for Grobid, BHL, CrossRef, OpenAlex via pipeline/external.py. STRICT_NETWORK env var fails-fast for release builds. Standardized --dry-run (#41) — across all pipeline + post-pipeline CLIs, replacing the prior 4-of-10 inconsistent state. Vision + figures SLURM array support for the vision pass (#27) — batch_pass3b.sh now parallelizes the same way batch_pipeline.sh does. Prerequisite for running Pass 3b (Qwen2.5-VL-7B) at corpus scale (#11). Archaic plate prefixes + Roman-numeral support in figure extraction (#16) — recovers figure numbers from 19th-c. and early-20th-c. caption conventions (Tab. III., Pl. XII.) the v0.1 heuristics didn't match. Indices + features BibTeX round-trip curation (#26) — bib_export serializes biblio_authority.sqlite to BibTeX; bib_import brings hand edits back into the authority database. Closes the loop on user curation of corpus bibliography. Multi-category lexicons (#24) — --lexicon CATEGORY:PATH accepts any number of YAML files (anatomy, biogeography, …); top-level keys define categories. Annotations carry a category field so per-category resume works. Plant-source taxonomy support (#23) — ingest_taxonomy.py accepts non-WoRMS Darwin Core archives, including taxa.tsv Taxon-core layouts (#22). Expanded default Tesseract language pack coverage (#46) — the OCR fallback union covers more 19th-c. European scripts. MCP server Corpuscle-aware server name (#17) — the deployed MCP server identifies itself as corpus: rather than the bare corpus, with __version__ from pipeline/version.py surfaced via bundle_info. Inline figure bytes from get_figure_image — returns PNG content directly so MCP clients don't need filesystem access to the bundle. Two-layer client-side instructions. The InitializeResult.instructions blob now joins a packaged default (mcpsrv/default_instructions.md) with an optional per-corpuscle override prepended on top. Default guidance (defer to corpus taxonomy + bibliography, preserve historical terminology) ships with the server and reaches every client with no operator action; corpus-specific nudges land first via /instructions.md when present. templates/instructions.md is the starting scaffold for the corpuscle layer. /healthz liveness probe on the SSE transport. Returns 200 ok without requiring the bearer token — mounted ahead of the auth middleware in mcpsrv.transport._HealthzASGI so uptime monitors and reverse-proxy readiness checks don't need the shared secret. Exposed through deploy/nginx.conf as location = /healthz. Internal pipeline/ package (#42, #44, #45) — the ~1700-line process_corpus.py is split into focused submodules (pipeline.annotate, pipeline.chunking, pipeline.metadata, pipeline.figure_passes, pipeline.scan, pipeline.extract, pipeline.runner, pipeline.main). process_corpus.py is now a thin CLI shim. No behavior change. mcpsrv/ package (#15) — the ~2050-line mcp_server.py is split per concern (papers / taxonomy / bibliography / figures / transports). No behavior change. bib/ package (#43) — bundles bib_metadata, bib_export, and bib_import into a single namespace. Single-source __version__ in pipeline/version.py, stamped into bundle manifests + MCP bundle_info. main / dev / vN branching model adopted; dev carries a PEP 440 pre-release suffix (0.2.0.dev0 → 0.2.0). schema_version field on persistent artifacts so future schema changes can be detected without parsing. pngquant added to the build environment (#25) so bundled PNGs ship pre-compressed. Changed Repo layout cleanup. Library modules figures.py, taxa.py, grobid_client.py, embeddings.py, vision.py, external.py, and version.py moved from the repo root into the pipeline/ package — they were never run directly, only imported. Import sites: from figures import … → from pipeline.figures import … (and likewise for the other six). One-off utilities (dedup_ghost_works.py, unify_doi_corpus_key.py) moved to tools/. PLAN.md moved to dev_docs/. The repo root now holds only user-facing CLI entry points. Anatomy-only naming scrubbed throughout. The lexicon system is no longer hard-coded to anatomy; field and file names use lexicon / category instead. Documentation refreshed in lock step. CLAUDE.md → AGENTS.md — generic agent-guidance filename for compatibility with non-Claude assistants. MCP hot paths trimmed for token cost — list_papers row shape reduced; previously unbounded MCP list returns are now bounded. Per-PDF subprocess gated by platform (#18) — the docling subprocess wrapper, originally added to contain a Linux segfault, no longer runs on macOS where it was pure overhead. Deploy architecture: ALB + ACM, multi-organism support. Was single-EC2 + nginx + Let's Encrypt; now each EC2 sits behind a shared Application Load Balancer (siphonophores-mcp-alb) that terminates TLS with an ACM wildcard cert. nginx on the instance drops to plain :80 (no certbot, no TLS). Adds a default /health endpoint nginx serves directly for ALB target-group health checks (separate from the /healthz MCP probe under bearer auth). One listener rule per organism (host-header match → target group), so adding cnidaria.siphonophores.org etc. is a target-group + listener-rule + DNS CNAME, no new TLS infra. CloudFormation BucketName parameter now optional via a new CreateBucket flag so re-deploys can attach to an existing bundle bucket. Full runbook rewritten: dev_docs/DEPLOY.md. INSTALL.md promoted to repo root. Moved from dev_docs/ since the content (jbig2enc, OCR language packs, Grobid, pip fallback) is user-facing and belongs alongside README, CONTRIBUTING, and CHANGELOG. dev_docs/ keeps its maintainer-only docs (PLAN, DEPLOY, BOUCHET, MCP_TOOLS, TESTING, OVERVIEW). All cross-references updated. Fixed Bouchet batch scripts (#20, #21) — native-library load problems, CUDA / torch wheel selection, and a fail-loud preflight so configuration drift surfaces before the array launches instead of after each task fails individually. deploy/stack.yaml (#6) — removed stale default-VPC assumptions that broke deploys on accounts without a default VPC. Docling extraction fails loudly instead of writing placeholder text when the parser produces nothing usable. Missing PyMuPDF surfaces as a stage failure instead of a silent skip. --skip-pipeline rejected with --re-annotate-stale — previously these silently combined into a no-op. rapidfuzz required at import in unify_doi_corpus_key and dedup_ghost_works — failure surfaces at startup, not partway through a run. ingest_to_vector_db import wired into pipeline.main — was silently dropped during the package extraction. package_for_serve discovers lexicon outputs dynamically — no more hardcoded list that drifted from the multi-category lex