LATEST BENCHMARK · NOVEMBER 1, 2026

CYO Camp scraping benchmark. Inspect Trident, Firecrawl and Crawl4AI outputs beside the source values, with run settings, timings and token costs.

CAMP-BENCH · SEPTEMBER 10, 2026

Evaluation plan

Four evaluations for the School’s Out pipeline and the gaps to defensible claims. Source: docs/eval-plan.md.

Prepared September 10, 2026. All comparison tables in sections 1–4 are hypothetical examples, not measured School's Out results. Section 5 contains findings from a read-only inspection of current local source, locally available worker refs, and recorded Camp-Bench reports. No production database was queried, no benchmarks were executed, and no application code was changed.

For this plan, discovery means finding relevant providers; scraping means turning provider websites into structured records; ingestion means mapping, matching, and persisting supplied records into the canonical database. Trident uses “ingestion” more broadly and spans multiple stages. These are evaluation boundaries, not assertions about team ownership or tool names.

1. Discovery: more verified new providers for the same budget

Task: Discover providers serving a specified geography, season, and camp category. Each system starts from the same catalog snapshot and has the same information access, maximum elapsed time, and processing budget.

Useful discovery: a unique eligible provider, supported by dated evidence of relevant camp activity, with an official or otherwise adequate source for continued ingestion. Count a provider once; extra pages and URL aliases do not create additional providers. Freeze the rules for brands, branches, locations, and franchises before evaluation.

Hypothetical example, not a measured result
Hypothetical outcome
Our tool
Comparison tool
Submitted provider candidates
100
140
Verified unique relevant providers
80
75
Verified providers new to the shared catalog
60
50
Candidate yield after relevance and uniqueness checks
80%
53.6%
Processing cost
$10
$10

Example claim: “On the specified discovery task, our tool found 60 verified new providers versus 50 at the same $10 budget: 20% more.” The budget refers to an agreed processing-cost scope; adjudication cost is reported separately.

If the new-provider sets share 40 providers, ours adds 20 that the competitor missed and theirs adds 10 that ours missed. Inspect those exclusive finds to identify strengths by location, provider size, and source type.

Metrics: verified new-provider yield; relevant unique-provider yield; irrelevant and duplicate candidates; overlap and exclusive finds; usable source/evidence rate; time and cost per verified new provider; failure rate and variation across runs. Reviewed-pool coverage can be reported, but it does not establish recall of all camps in the market.

Evaluation plan:

  1. Start with a varied development set of geographic discovery tasks, then freeze a separate test set covering large/small markets, camp categories, and seasons. An initial 10–20 tasks is a practical pilot, not a statistical guarantee.
  2. Pin the exact discovery implementation, model, prompts, search access, starting catalog, budget rules, and stopping conditions. Run each task at least three times initially, resetting agent state and recording failures.
  3. Pool every tool's candidates, normalize identity, and have reviewers assess candidates without tool attribution. Review all candidates for exact verified counts; if sampling is necessary, report estimated yield and uncertainty instead.
  4. Review valid discoveries missing from the initial reference; version adjudications and rescore all tools consistently. Reference data and test answers are unavailable to entrants.
  5. Report per-task and aggregate comparisons, including losses and uncertainty at the geographic-task level. A large easy market must not hide failures elsewhere.

Pooling is established in retrieval evaluation: NIST TREC relevance judgments ↗. Our frozen-catalog novelty rule and candidate acceptance criteria are proposed Camp-Bench choices.

2. Ingestion: more correct canonical records from the same input

Task: Give each ingestion implementation the same 1,000 messy input records and identical starting database. Independent labeling establishes that the input represents 800 expected entities. The labeler specifies correct normalization, identity, supported fields, parent relationships, and intended create/update/reject/review outcomes using information available to the system.

Hypothetical example, not a measured result
Hypothetical outcome
Our tool
Baseline
Fully correct expected canonical entities
720 / 800 (90%)
600 / 800 (75%)
Incorrect merges
4
16
Surplus duplicate canonical records
20
60
Processing cost, including retries
$20
$20

Example claim: “On this ingestion test set, our tool produced 20% more fully correct canonical records at the same processing budget, with four incorrect merges versus sixteen.” Error rows are separate diagnostics, not an additive partition of failed entities.

“Fully correct” means the expected identity, predeclared required supported fields, and parent links are correct, without surplus duplicate representations of that entity. Legitimately absent or ambiguous source information follows a declared policy; the tool is not expected to recover facts it was never given.

Metrics: fully correct entity yield; supported field precision and recovery; correct parent links; false merges; missed matches and surplus duplicates; preservation of existing correct facts; repeated-import consistency; throughput, processing cost, queue failures, and human review time. Report automatic results before review separately from assisted results afterward.

Evaluation plan:

  1. Assemble source-shaped cases covering exact and near duplicates, distinct providers with similar names, shared domains, reused IDs, conflicting observations, missing/ambiguous values, malformed values, and children arriving before parents. Split by provider/source family.
  2. Independently label the expected entity clusters and final canonical graph, plus acceptable update/rejection/review behavior. Compare the current pipeline with its preceding version or a deterministic importer. A competitor is appropriate only if it performs this same stage.
  3. Restore a disposable database snapshot for every run. Invoke the real ingestion pipeline, let relevant jobs settle, and score the persisted state and decisions. A completed job is not a correct record.
  4. Replay the input, change its order, supply corrections, and test interruption/restart and reversal. Canonical facts and identity should follow the declared contract; additional raw audit-history rows are allowed.
  5. Measure candidate retrieval separately from matching decisions. Audit confident automatic results as well as flagged reviews. Preserve per-call usage, retries, elapsed time, and review effort.

3. Scraping: more correct, supported facts and complete sessions

Task: Start each stack at the same provider homepages and require the same session schema. In this hypothetical test, 20 websites contain 200 reviewed sessions with eight source-available required fact groups each: 1,600 opportunities. The actual field profile must be frozen before evaluation and should include price meaning, eligibility, and correct session association.

Hypothetical example, not a measured result
Hypothetical outcome
Our tool
Comparison tool
Correct supported facts recovered
1,440 / 1,600 (90%)
1,200 / 1,600 (75%)
Correct facts among all emitted field claims
1,440 / 1,500 (96%)
1,200 / 1,400 (85.7%)
Complete unique sessions
160 / 200 (80%)
130 / 200 (65%)
Wrong or unsupported field claims
60
200
Processing cost
$20
$20

Example claim: “Across the specified websites, our scraper recovered 20% more correct facts at the same processing budget; 96% of its emitted field claims were correct and supported.” This example also demonstrates why high precision alone is insufficient: an empty result must not win by making no false claims.

A fact is correct only when attached to the right provider/program/session, with the right meaning and dated supporting evidence. A correct price for the wrong camp is an error. Equivalent formatting is allowed; missing facts are not repaired by benchmark code. Duplicate records cannot be stitched into a complete session.

Metrics: supported-fact recovery and precision; complete unique sessions; inventory coverage; critical-field errors; duplicates and wrong associations; unsupported assertions; success/failure rate; elapsed time, processing cost, and cost per verified complete session. Report fields as correct, wrong, unsupported, or missing, and distinguish source-unpublished information from extraction misses.

Evaluation plan:

  1. Populate the already proposed five-development/twenty-held-out-provider corpus. Include ordinary HTML, PDFs, JavaScript calendars, external registration, many similar sessions, multiple venues, ambiguous prices, historical seasons, and providers with no eligible camps.
  2. Review source evidence and the complete in-scope reference inventory, with timestamps. Run the same task contract from provider homepages, with three initial attempts per stack, comparable limits, and disclosed setup/cache/repair behavior.
  3. Preserve found URLs, readable captures, filtering decisions, intermediate and final output, field evidence, timings, and available usage. Locate losses at discovery, capture, filtering, extraction, validation, repair, or export.
  4. Add a diagnostic test that gives extractors identical captured material. This isolates interpretation from website access; it cannot by itself establish which crawler is better.
  5. Keep live-site operating tests distinct from fixed-evidence regression tests. Adjudicate source changes. Separate first-time onboarding from refresh that reuses an existing extractor.

4. Database/data quality: better verified stored information

Task: Audit a defined database population against independent provider evidence. To evaluate an improvement, compare the old snapshot and candidate cleanup/refresh of an isolated copy on the same sample and as-of evidence. Keep tracking of original IDs and duplicate clusters through merges/deletions so difficult records cannot disappear from the denominator.

This hypothetical matched audit covers 500 session records and 4,000 source-available critical facts. It illustrates the direction of a before/after claim; it is not a forecast.

Hypothetical example, not a measured result
Hypothetical outcome
Before improvement
After improvement
Correct, supported critical facts
3,600 / 4,000 (90%)
3,880 / 4,000 (97%)
Records meeting the full required-data standard
400 / 500 (80%)
470 / 500 (94%)
Sample records identified as surplus duplicates
25 / 500 (5%)
5 / 500 (1%)
Records whose required facts had genuine source checks within 30 days
200 / 500 (40%)
450 / 500 (90%)

Example claim: “In a matched audit of 500 session records, critical-fact accuracy increased from 90% to 97%, and the share meeting our required-data standard rose from 80% to 94%.” These are increases of seven and fourteen percentage points. A “recent check” needs a dated observation of the relevant facts; an unrelated database write does not qualify. Audit observations themselves are not silently counted as prior operational verification.

Metrics: source-verified fact accuracy; source-available fact completeness; full-record usability under a declared standard; wrong critical facts; duplicate prevalence; parent-link correctness; actual observation age; provenance availability; external market coverage. Keep blank, unknown, unsupported, stale, and incorrect separate.

Evaluation plan:

  1. Freeze a read-only export with stable IDs, hierarchy, canonical values, source receipts, observation timestamps, score/version, and cohort counts. Run whole-snapshot structural checks for broken references and invalid values.
  2. Draw a random sample stratified by geography, category, source, age, and score band. Start with roughly 300–500 records, then size subsequent audits for the precision and subgroup claims needed. Oversampling risky strata requires population weighting for database-wide estimates.
  3. Independently verify facts against dated authoritative sources. Double-review a subset, resolve disagreements, and record unsupported/unavailable evidence. Keep a separate targeted defect sample; do not mix it into a prevalence estimate as if it were random.
  4. Report denominators and uncertainty respecting provider clustering. Compare observed error rates across current score bands to test whether the score predicts data quality. A heuristic score range is not an empirical confidence interval; proportion intervals also need an appropriate sampling design. See NIST proportion confidence intervals ↗.
  5. Audit missing-market coverage from outside the database using an independently assembled provider/session sample. Sampling only stored records cannot reveal providers absent from the catalog.
  6. Recheck a fixed panel over time to measure actual change detection and update delay. For a competitor-database claim, use a common externally selected population, count missing records, and disclose each source's observation date; comparing only overlapping records hides coverage differences.

5. What we have and what prevents the claims today

Discovery

The reviewed worker has URL inventory, crawl filters, page-role classification, and known-site onboarding. Its input is a provider URL. This establishes site discovery infrastructure, not the geography-to-new-providers interface needed for section 1. The actual open-web discovery runner was not identified in the inspected scope; this is not a claim that no teammate has built one elsewhere.

Needed: a named, pinned discovery runner with callable task/output contracts; frozen catalog baseline; geographic task set; independent candidate review and deduplication; per-run budget/usage evidence; a discovery scorecard. Camp-Bench has no recorded comparative provider-discovery result in the evidence reviewed.

Ingestion

The API already preserves uploaded raw data, jobs, mapping and matching decisions, accepted fields, entity IDs, transactions, and reversal history. Disposable local database fixtures and restart/reversal/relationship tests exist. We do not need to recreate these foundations.

Needed: an independently labeled input/expected-database corpus; a Camp-Bench integration that restores and scores database state; representative auditing including high-confidence unflagged decisions; a complete processing-cost ledger. Existing processed/merged/failed counts are operational outcomes rather than measured correctness.

Specific behaviors to evaluate: single-candidate matches can settle automatically; candidate lookup is bounded before later ranking; the upload duplicate counter measures raw repeated content rather than final canonical duplicate prevalence. These are test priorities, not demonstrated production errors. The inspected pipeline model-response type does not retain token usage.

Scraping

This is the strongest existing evaluation. Great Work Montessori has a human-approved five-week reference and nine recorded attempts, three each for Trident, Firecrawl Agent, and Crawl4AI with Codex. Camp-Bench retains frozen configuration, scoring, outputs, evidence, timing, and usage limitations. There is also the separate Little Yellow House Art onboarding/refresh case.

Actual recorded outcome: Trident recovered 20 of 120 core field opportunities, while each comparison stack recovered 120 of 120. Trident delivered zero complete unique weeks across the three attempts. Both comparison stacks still had a failed full-task attempt because core-field recovery does not account for every supplemental assertion. These are results on one development provider, not general performance rates. The recorded corpus contained the required camp content; some losses happened after capture.

Needed: the larger independently reviewed held-out corpus, full cost accounting, and repeatable integration for the exact version being claimed. The current report evaluates complete configured stacks, not crawler engines in isolation. Its reference review and subsequent output adjudication are distinct; output audits were assistant-led. Add independent output review for strong external claims.

The recorded Trident evaluation pins commit 6daccf9; the locally available worker branch inspected for this plan is 4330142. No claim transfers automatically between those versions. That newer source still explicitly excludes PDFs from selected URL evidence, caps URLs per family, and permits only one selected external origin. Test these coverage limits on appropriate sites. The native status endpoint and final result do not expose all grading artifacts or per-field source mappings. Camp-Bench's isolated recorder supplies some evidence that native interfaces lack.

Crucially: better measurement alone cannot make a superiority claim true. The tested Trident version underperformed in this case. Tool improvements must be rerun and measured before a stronger claim is available.

Database/data quality

Intelligence provides exports, linked entity inspection, scoring inputs, and source/merge ledgers. The Go API computes completeness, freshness, consistency, and source-derived confidence signals.

Needed: a current snapshot and unbiased evidence-backed audit; a population/sampling plan; independent-source identity; genuine field observation timestamps; duplicate/relationship adjudication; external coverage checks; and empirical validation of score bands.

Current limitations from source: completeness means populated fields; format/range validation does not establish truth; an ingestion batch can act as a source ID even when multiple batches share an underlying website; some derived freshness uses aggregate updated_at; duplicate counts reflect flagged candidates rather than all duplicates. Market-supply code leaves parentAuditMissRate unset. Current score intervals are designed heuristics rather than statistically estimated confidence intervals.

Production was not queried in this review, so these findings do not establish the present database's accuracy, duplicate prevalence, or freshness.

6. The smallest useful path to claim-ready results

  1. Pin the named tool and boundary for each scorecard, including the exact current discovery runner.
  2. Reuse Camp-Bench's runner/report/evidence foundations and the API's isolated database fixtures. Add missing stage adapters and independent references rather than build a new general framework first.
  3. Run a small development evaluation for each stage, inspect failures, then freeze held-out tasks and independent review rules. Keep estimates and hypothetical tables visually separate from recorded results.
  4. Record full cost scope, latency, retries, failures, version identity, source dates, and human assistance. Where dollars are unavailable, publish usage and an explicit limitation instead of “cheaper.”
  5. Publish narrow claims with counts, workload, version, observation date, uncertainty, and supporting examples. Broaden claims only after representative held-out results support them. Reports are currently local by the user's choice; online publication is a delivery step, not a prerequisite for truthful measurement.

Finally, add one end-to-end evaluation: from a geographic request to net-new, independently verified useful records actually persisted in the database at a fixed total budget. Stage scorecards explain losses; the end-to-end test demonstrates product value. The stage percentages cannot simply be multiplied, because they have different denominators and correlated failures.

Source audit notes

Line references describe local files inspected on September 10, 2026, except worker references explicitly tied to a Git ref. They establish implementation or recorded-report facts, not deployment state. The applicable score-any-camp skill was consulted; its TypeScript backend descriptions are partly stale, so current scoring conclusions use the Go implementation.

Evidence
Location
Existing reports, recorded field results, corpus still pending, unimplemented benchmark areas
camp-bench/README.md, lines 11, 26, 67, 99, 155
Empty broader pilot task list
camp-bench/data/pilot.manifest.json
Pinned one-provider comparison, scoring denominator, cost limits, narrow claim scope
camp-bench/docs/provider-comparison.md, lines 24, 60, 98, 109, 117, 130
Recorded system results and explicit limitations
camp-bench/data/comparison.json, systems and methodology
Captured relevant content, evidence/export and cost gaps in tested Trident
camp-bench/docs/trident-gaps.md, lines 24, 42, 91, 132, 154, 175, 270
Known-site-only API, snapshot evidence, filtering and crawl coverage in newer worker
schools-out-workers, ref origin/caidanw/explore-ingestion-adapter at 4330142: internal/api/ingestion.go:63, internal/ingestion/capture.go:43, internal/ingestion/url_discovery.go:126,171,226, internal/ingestion/crawler.go:70,345,499
Worker stage metrics and coarse exported result
Same worker ref: internal/ingestion/trace.go:9, internal/ingestion/openai.go:44, internal/ingestion/records.go:85
Ingestion routes, raw input/job preservation and canonical merge transaction
apis/internal/ingestion/service.go:28; internal/pipeline/administration.go:110; internal/pipeline/canonical_entities.go:1600
Existing disposable DB fixtures
apis/internal/testdb/database.go:17; internal/pipeline/restart_test.go:27
Candidate limits, automatic decisions, omitted usage
apis/internal/pipeline/store.go:847; runner.go:823; openai.go:505
Raw duplicates vs stored duplicate prevalence; UI operational counts and review selection
apis/internal/ingestion/upload.go:117; data-explorer/www-schools-out-intelligence/library/ingestion.ts:34,267
Completeness and freshness formulas
apis/internal/scoring/fields.go:140,180
Validation signals and source-kind mapping
apis/internal/scoring/input.go:128,354,409
Batch-based source identities, derived timestamps and flagged duplicates
apis/internal/scoring/store.go:118,271,336,383
Heuristic confidence and interval
apis/internal/scoring/engine.go:521
Missing independent coverage measurement
data-explorer/www-schools-out-intelligence/library/market-supply-source.ts:45,97; library/market-supply.ts:522