CYO Camp benchmark
Test setup
Trident and Crawl4AI ran on November 1, 2026; local Firecrawl ran on November 3. Each started at cyo.camp and extracted the provider, programs and separately bookable sessions. Each ran three times with fresh state, one at a time, with no overall time limit.
The November 3 follow-up calls self-hosted Firecrawl /scrape, using the same breadth-first link discovery and Codex extractor as the Crawl4AI baseline. Its three sequential attempts are separate from the November 1 run.
Local Firecrawl setup
Firecrawl v2.11.162 supplies pages through /scrape. The benchmark follows links breadth-first; hosted Agent and Firecrawl’s /crawl endpoint are not used. The Codex prompt, schema, extractor code and model settings match November 1 Crawl4AI. Each attempt starts fresh, with no overall or model time limit.
The hosted Firecrawl Agent also ran three times on November 1. It used Spark 2 at high effort with agent-directed traversal on Firecrawl’s hosted service. Its outputs and timings remain available.
Two setup checks preceded the benchmark attempts. The first stopped at an incorrect smoke-test assertion before any model call; the repaired check completed one synthetic extraction. Both cleaned up.
Source captures and scoring
Public pages were saved before and after the November 1 run. Their extracted text matched across 78 HTML pages and six PDFs. CYO was selected from an existing test corpus before inspecting the outputs.
FAQ: The FAQ removed its 75th-anniversary promotion. Target quotations are unchanged.
Women’s Camp: The adult Women’s Camp page changed from June 12–14, 2026 to June 11–13, 2027. It remains outside the child-session comparison.
The inspected source text stayed unchanged during the local run.
The earlier automated scores below use Jev and Claude judgments against the November 1 captures. They count repeated claims, exclude held records and keep unresolved claims in the denominator. Jev also participates in Trident’s extraction.
The output download preserves 1,112 candidates Trident excluded.
Crawl4AI and local Firecrawl instructions
Starting only from the supplied provider homepage, discover the provider's published organization, child/youth activities and enrollable sessions. Preserve organization/activity/session parent relationships. Include legitimate non-camp programs and published historical or full sessions. Do not infer a year from today's date or silently exclude published historical inventory. Follow provider pages, redirects, PDFs, public interactive calendars and official registration/booking sources linked by the provider. Use only provider-owned evidence or those official linked public sources; do not use unrelated directories, search-result snippets, reviews or other organizations as factual evidence. Do not sign in, create accounts, contact anyone, submit forms, reserve places or make bookings. Do not access account-only information.
Return the requested schema using verbatim source quotations and exact page URLs. Keep identity, dates, hours, weekdays, ages/grades, money/currency/basis, mandatory fees, registration, explicit capacity/status and location on their stated owner. Preserve distinct same-name sessions and ordered collection items. Do not merge records, invent bounds, convert grades into ages, infer missing years, or repair contradictory published facts. Use structured values only when supported by the source and schema; retain unparseable supported text as text_only and unpublished facts as absent. Text fields use quotations without a value. Returned records are final records; held records are explicit rejected or unresolved candidates. Return only the native JSON object, with no commentary.
Compare the output
Targets were checked by the assistant against November 1 captures on November 3. They have no human review. The earlier automated scores below use their original rubric.
Loading saved outputs…
Selected records
Trident · Nov 1
No record selected
Hosted Firecrawl Agent · Nov 1
No record selected
Crawl4AI + Codex · Nov 1
No record selected
Time per attempt
Trident · Nov 1
Hosted Firecrawl Agent · Nov 1
Crawl4AI + Codex · Nov 1
Local Firecrawl + Codex · Nov 3
Elapsed time includes setup, crawling, extraction and result collection.
Tokens and cost
Observed tokens at API rates
Observed usage only; 64 failed Trident processes reported no tokens. $0 new cash; subscription allocation and Jev debit are unknown.
Each total covers three attempts. Hosted Firecrawl Agent reports no model tokens.
Cost calculation
Extraction tokens are priced at the November 1, 2026 API rates. Cached input has its own rate. Review calls are excluded. The recorded new cash spend was $0. Subscription allocation, exact Jev prepaid debit and compute cost are unavailable.
Missing process usage and cache-write premiums are absent from the estimate. Unreported cached input is priced as uncached input. The bars show complete sets of three attempts.
Tokens for each attempt
Cached input and reasoning output are subsets of input and output. Unreported means no counter was supplied.
Tokens by model
Cached input is included in input. Reasoning output is included in output.
Local setup tokens
The synthetic setup check used $0.0799 at API rates. These tokens are excluded from the benchmark totals above.
Earlier automated scores
Both models checked all 4,170 returned claims. A supported claim required agreement and a source excerpt from each model. These scores pool the three attempts.
Wrong means both models found contradicting evidence. Unresolved includes disagreement, mixed claims and insufficient evidence.
Registration matches
The check required a returned session, matching parent program and the exact key for one of five captured registration entries.
Trident held two possible matches per attempt. Crawl4AI returned two possible matches in attempts 2 and 3 without registration keys.
Scores for each attempt
Crawl4AI’s first attempt marked every field absent and contributes no claims.
Review coverage
Jev answered 12,924 review questions and Claude answered 12,910. Claude declined 14 questions covering held records, citations and exclusions. All 4,170 returned factual claims received both reviews.
The five-entry registration list covers an audited subset of the site. One entry had an ambiguous start date. The stricter full-hierarchy rule had no eligible denominator because the reviewers disagreed on provider admission.
Run log
Local Firecrawl + Codex · November 3
Local output checks
A parsed value requires a supporting quote, not only a quote marked as context. These checks concern output rules, not factual correctness; all saved values remain inspectable.
Local crawl coverage
Failed pages
Diagnostics
Download ZIPTrident runtime checks
Chrome failures were reproduced under the earlier worker’s 256-task limit. The November 1 rerun used the service’s 4,096-task and 65,536-file limits with disk-backed storage. Browser replays passed before launch.
The three live workers peaked at 223, 271 and 255 tasks, with zero task-limit denials or out-of-memory events. Output hashes, cleanup and release of sleep protection were verified.
Packet contents and checksum
The ZIP includes saved output fields, source targets, provider tokens and the local Firecrawl follow-up. The original nine-attempt diagnostic packet is included. Extract it and run python3 verify.py.
SHA-256 35c4623e5ad9b2b4698e0454d19956e0975a1baf4082b42a448a06201916bc3f