CYO Camp benchmark

Test setup

Trident and Crawl4AI ran on November 1, 2026; local Firecrawl ran on November 3. Each started at cyo.camp and extracted the provider, programs and separately bookable sessions. Each ran three times with fresh state, one at a time, with no overall time limit.

Setting
Trident
Local Firecrawl + Codex
Crawl4AI + Codex
Runtime
Linux + Chromium
Local Docker
macOS + Chromium
Extraction
Sol, Luna and Jev
Codex Sol
Codex Sol
Discovery
Native Trident traversal
Shared breadth-first traversal, up to 80 pages
Up to 80 pages

The November 3 follow-up calls self-hosted Firecrawl /scrape, using the same breadth-first link discovery and Codex extractor as the Crawl4AI baseline. Its three sequential attempts are separate from the November 1 run.

Local Firecrawl setup

Firecrawl v2.11.162 supplies pages through /scrape. The benchmark follows links breadth-first; hosted Agent and Firecrawl’s /crawl endpoint are not used. The Codex prompt, schema, extractor code and model settings match November 1 Crawl4AI. Each attempt starts fresh, with no overall or model time limit.

The hosted Firecrawl Agent also ran three times on November 1. It used Spark 2 at high effort with agent-directed traversal on Firecrawl’s hosted service. Its outputs and timings remain available.

Extractor
gpt-6.1-sol, xhigh reasoning
Crawl allowance
80 pages, depth 4
Native per-page timeout
60 seconds
PDF download ceiling
Firecrawl 50 MiB; Crawl4AI 20 MiB
Funding
Codex subscription. No model API keys or hosted Firecrawl credits.

Two setup checks preceded the benchmark attempts. The first stopped at an incorrect smoke-test assertion before any model call; the repaired check completed one synthetic extraction. Both cleaned up.

Setup check 1
failed
0
0
Setup check 2
completed
1
0
Source captures and scoring

Public pages were saved before and after the November 1 run. Their extracted text matched across 78 HTML pages and six PDFs. CYO was selected from an existing test corpus before inspecting the outputs.

Before local run
27
177 / 177
25 / 27
After local run
27
177 / 177
25 / 27

FAQ: The FAQ removed its 75th-anniversary promotion. Target quotations are unchanged.

Women’s Camp: The adult Women’s Camp page changed from June 12–14, 2026 to June 11–13, 2027. It remains outside the child-session comparison.

The inspected source text stayed unchanged during the local run.

The earlier automated scores below use Jev and Claude judgments against the November 1 captures. They count repeated claims, exclude held records and keep unresolved claims in the denominator. Jev also participates in Trident’s extraction.

The output download preserves 1,112 candidates Trident excluded.

Crawl4AI and local Firecrawl instructions

Starting only from the supplied provider homepage, discover the provider's published organization, child/youth activities and enrollable sessions. Preserve organization/activity/session parent relationships. Include legitimate non-camp programs and published historical or full sessions. Do not infer a year from today's date or silently exclude published historical inventory. Follow provider pages, redirects, PDFs, public interactive calendars and official registration/booking sources linked by the provider. Use only provider-owned evidence or those official linked public sources; do not use unrelated directories, search-result snippets, reviews or other organizations as factual evidence. Do not sign in, create accounts, contact anyone, submit forms, reserve places or make bookings. Do not access account-only information.

Return the requested schema using verbatim source quotations and exact page URLs. Keep identity, dates, hours, weekdays, ages/grades, money/currency/basis, mandatory fees, registration, explicit capacity/status and location on their stated owner. Preserve distinct same-name sessions and ordered collection items. Do not merge records, invent bounds, convert grades into ages, infer missing years, or repair contradictory published facts. Use structured values only when supported by the source and schema; retain unparseable supported text as text_only and unpublished facts as absent. Text fields use quotations without a value. Returned records are final records; held records are explicit rejected or unresolved candidates. Return only the native JSON object, with no commentary.

Compare the output

Targets were checked by the assistant against November 1 captures on November 3. They have no human review. The earlier automated scores below use their original rubric.

Loading saved outputs…

Selected records

Trident · Nov 1

No record selected

Hosted Firecrawl Agent · Nov 1

No record selected

Crawl4AI + Codex · Nov 1

No record selected

Time per attempt

Trident · Nov 1

Attempt 184.4 min
Attempt 2101.2 min
Attempt 385.1 min

Hosted Firecrawl Agent · Nov 1

Attempt 16.9 min
Attempt 25.0 min
Attempt 36.7 min

Crawl4AI + Codex · Nov 1

Attempt 19.0 min
Attempt 234.6 min
Attempt 315.9 min

Local Firecrawl + Codex · Nov 3

Attempt 117.8 min
Attempt 217.2 min
Attempt 316.1 min

Elapsed time includes setup, crawling, extraction and result collection.

Tokens and cost

Observed tokens at API rates

Codex subscriptionJev prepaid
Trident$14.8088
Crawl4AI + Codex$2.2009
Local Firecrawl + Codex · Nov 3$2.8816

Observed usage only; 64 failed Trident processes reported no tokens. $0 new cash; subscription allocation and Jev debit are unknown.

Trident
Codex subscription
9,538,641
7,444,224 known
1,052,921
$14.8026
Trident
Jev prepaid
149,117
Unreported
2,440
$0.0063
Crawl4AI + Codex
Codex subscription
514,510
159,744
147,537
$2.2009
Local Firecrawl + Codex · Nov 3
Codex subscription
654,546
217,984
198,663
$2.8816

Each total covers three attempts. Hosted Firecrawl Agent reports no model tokens.

Cost calculation

Extraction tokens are priced at the November 1, 2026 API rates. Cached input has its own rate. Review calls are excluded. The recorded new cash spend was $0. Subscription allocation, exact Jev prepaid debit and compute cost are unavailable.

Missing process usage and cache-write premiums are absent from the estimate. Unreported cached input is priced as uncached input. The bars show complete sets of three attempts.

$2.0000
$0.1000
$10.0000
$0.1000
$0.0100
$0.5000
$0.0420
Unavailable
$0.0000
Tokens for each attempt

Cached input and reasoning output are subsets of input and output. Unreported means no counter was supplied.

Trident
1
Codex subscription
3,361,887
2,525,440
377,842
172,788
20
$5.4821
Trident
1
Jev prepaid
55,587
Unreported
904
Unreported
Unreported
$0.0023
Crawl4AI + Codex
1
Codex subscription
159,909
0
9,606
8,804
0
$0.4159
Crawl4AI + Codex
2
Codex subscription
171,815
159,744
73,320
0
0
$0.7733
Trident
2
Codex subscription
3,132,135
2,439,424 known
326,818
152,023
24
$4.6560
Trident
2
Jev prepaid
46,654
Unreported
768
Unreported
Unreported
$0.0020
Crawl4AI + Codex
3
Codex subscription
182,786
0
64,611
7,118
0
$1.0117
Trident
3
Codex subscription
3,044,619
2,479,360 known
348,261
170,592
20
$4.6644
Trident
3
Jev prepaid
46,876
Unreported
768
Unreported
Unreported
$0.0020
Local Firecrawl + Codex · Nov 3
1
Codex subscription
218,205
0
74,696
6,201
0
$1.1834
Local Firecrawl + Codex · Nov 3
2
Codex subscription
218,195
217,984
57,701
8,235
0
$0.5992
Local Firecrawl + Codex · Nov 3
3
Codex subscription
218,146
0
66,266
9,038
0
$1.0990
Tokens by model

Cached input is included in input. Reasoning output is included in output.

Trident
gpt-6.1-sol
9,203,761
7,431,168
1,047,945
493,919
192 counted, 0 missing
Trident
gpt-6-luna
334,880
13,056 known
4,976
1,484
29 counted, 64 missing
Trident
jev-1.13.0
149,117
Unreported
2,440
Unreported
67 reported requests
Crawl4AI + Codex
gpt-6.1-sol
514,510
159,744
147,537
15,922
3 counted, 0 missing
Local Firecrawl + Codex · Nov 3
gpt-6.1-sol
654,546
217,984
198,663
23,474
3 counted, 0 missing
Local setup tokens

The synthetic setup check used $0.0799 at API rates. These tokens are excluded from the benchmark totals above.

Local Firecrawl setup
gpt-6.1-sol
20,336
0
3,918
2,070
1 counted, 0 missing
Earlier automated scores

Both models checked all 4,170 returned claims. A supported claim required agreement and a source excerpt from each model. These scores pool the three attempts.

Trident
1,160 / 1,379
84.1%
1
0
218
Hosted Firecrawl Agent
1,402 / 1,652
84.9%
11
0
239
Crawl4AI + Codex
971 / 1,139
85.3%
4
0
164

Wrong means both models found contradicting evidence. Unresolved includes disagreement, mixed claims and insufficient evidence.

Registration matches

The check required a returned session, matching parent program and the exact key for one of five captured registration entries.

Trident
0 / 5
0 / 5
0 / 5
Hosted Firecrawl Agent
5 / 5
5 / 5
5 / 5
Crawl4AI + Codex
0 / 5
0 / 5
0 / 5

Trident held two possible matches per attempt. Crawl4AI returned two possible matches in attempts 2 and 3 without registration keys.

Scores for each attempt
1
Trident
1
395 / 466
84.8%
70
0 / 5
2
Hosted Firecrawl Agent
1
443 / 521
85.0%
75
5 / 5
3
Crawl4AI + Codex
1
0 / 0
No claims
0
0 / 5
4
Hosted Firecrawl Agent
2
586 / 713
82.2%
120
5 / 5
5
Crawl4AI + Codex
2
570 / 663
86.0%
92
0 / 5
6
Trident
2
381 / 442
86.2%
61
0 / 5
7
Crawl4AI + Codex
3
401 / 476
84.2%
72
0 / 5
8
Trident
3
384 / 471
81.5%
87
0 / 5
9
Hosted Firecrawl Agent
3
373 / 418
89.2%
44
5 / 5

Crawl4AI’s first attempt marked every field absent and contributes no claims.

Review coverage

Jev answered 12,924 review questions and Claude answered 12,910. Claude declined 14 questions covering held records, citations and exclusions. All 4,170 returned factual claims received both reviews.

The five-entry registration list covers an audited subset of the site. One entry had an ambiguous start date. The stricter full-hierarchy rule had no eligible denominator because the reviewers disagreed on provider admission.

Run log

1
Trident
1
84.4 min
Passed
2
Hosted Firecrawl Agent
1
6.9 min
Passed
3
Crawl4AI + Codex
1
9.0 min
Passed
4
Hosted Firecrawl Agent
2
5.0 min
An uppercase email was rejected by the harness’s lowercase-only rule.
5
Crawl4AI + Codex
2
34.6 min
Passed
6
Trident
2
101.2 min
Passed
7
Crawl4AI + Codex
3
15.9 min
Passed
8
Trident
3
85.1 min
Passed
9
Hosted Firecrawl Agent
3
6.7 min
Passed

Local Firecrawl + Codex · November 3

1
completed
Rejected: context-only quote
17.8 min
4.1 min
13.3 min
51
2
completed
Passed
17.2 min
4.5 min
12.3 min
45
3
completed
Rejected: context-only quote
16.1 min
3.9 min
11.6 min
44
Local output checks
1
Passed
Rejected: context-only quote
$.activities[0].fields.overnight requires a quote that is not context
2
Passed
Passed
None
3
Passed
Rejected: context-only quote
$.activities[9].sessions[2].locations[0].fields.name requires a quote that is not context

A parsed value requires a supporting quote, not only a quote marked as context. These checks concern output rules, not factual correctness; all saved values remain inspectable.

Local crawl coverage
1
80
74
6
46
2
80
74
6
46
3
80
74
6
46
Failed pages
1
robots_disallowed
Unreported
1
robots_disallowed
Unreported
1
robots_disallowed
Unreported
1
All scraping engines failed to retrieve content from this URL. Engines tried: [pdf]. This usually happens when: (1) The URL is invalid or the page doesn't exist (404), (2) The website is blocking automated access, (3) The website is down or unreachable, (4) The page requires authentication. Double check the URL is correct and accessible in a browser. Check your server logs for more detailed error information from each engine.
500
1
All scraping engines failed to retrieve content from this URL. Engines tried: [pdf]. This usually happens when: (1) The URL is invalid or the page doesn't exist (404), (2) The website is blocking automated access, (3) The website is down or unreachable, (4) The page requires authentication. Double check the URL is correct and accessible in a browser. Check your server logs for more detailed error information from each engine.
500
1
robots_disallowed
Unreported
2
robots_disallowed
Unreported
2
robots_disallowed
Unreported
2
robots_disallowed
Unreported
2
All scraping engines failed to retrieve content from this URL. Engines tried: [pdf]. This usually happens when: (1) The URL is invalid or the page doesn't exist (404), (2) The website is blocking automated access, (3) The website is down or unreachable, (4) The page requires authentication. Double check the URL is correct and accessible in a browser. Check your server logs for more detailed error information from each engine.
500
2
All scraping engines failed to retrieve content from this URL. Engines tried: [pdf]. This usually happens when: (1) The URL is invalid or the page doesn't exist (404), (2) The website is blocking automated access, (3) The website is down or unreachable, (4) The page requires authentication. Double check the URL is correct and accessible in a browser. Check your server logs for more detailed error information from each engine.
500
2
robots_disallowed
Unreported
3
robots_disallowed
Unreported
3
robots_disallowed
Unreported
3
robots_disallowed
Unreported
3
All scraping engines failed to retrieve content from this URL. Engines tried: [pdf]. This usually happens when: (1) The URL is invalid or the page doesn't exist (404), (2) The website is blocking automated access, (3) The website is down or unreachable, (4) The page requires authentication. Double check the URL is correct and accessible in a browser. Check your server logs for more detailed error information from each engine.
500
3
All scraping engines failed to retrieve content from this URL. Engines tried: [pdf]. This usually happens when: (1) The URL is invalid or the page doesn't exist (404), (2) The website is blocking automated access, (3) The website is down or unreachable, (4) The page requires authentication. Double check the URL is correct and accessible in a browser. Check your server logs for more detailed error information from each engine.
500
3
robots_disallowed
Unreported

Diagnostics

Download ZIP
Trident runtime checks

Chrome failures were reproduced under the earlier worker’s 256-task limit. The November 1 rerun used the service’s 4,096-task and 65,536-file limits with disk-backed storage. Browser replays passed before launch.

The three live workers peaked at 223, 271 and 255 tasks, with zero task-limit denials or out-of-memory events. Output hashes, cleanup and release of sleep protection were verified.

Packet contents and checksum

The ZIP includes saved output fields, source targets, provider tokens and the local Firecrawl follow-up. The original nine-attempt diagnostic packet is included. Extract it and run python3 verify.py.

SHA-256 35c4623e5ad9b2b4698e0454d19956e0975a1baf4082b42a448a06201916bc3f