Ko-WideSearch

The one-line takeaway

What the benchmark asks

Enumerate the whole set, then fill every cell

Each task names a set-parent entity — a TV season, a dynasty, a league, an administrative region, an election — and asks for its full membership plus a per-item attribute table. Difficulty is set by two structural knobs dialed independently: table width (number of attribute columns) and a 2-D composite key (membership becomes a cross-product, e.g. team × season). From Easy to Hard, cross-product membership climbs from 0% to 100%.

What we found

Six findings that hold across twelve agents

Leaderboard

The Item-vs-Row gap is universal

Full pool, n = 228, pass@1. Bold = best per column. Parse = fraction of runs that emit a scorable table (low parse rates reflect models that search heavily then return prose instead of a table).

Why it's hard — the diagnostics

Knobs degrade accuracy; effort doesn't recover it

We probe the Item-vs-Row gap from four angles — which cell types fail, whether big sets are harder, whether more search helps, and what the surviving errors actually are. Every chart points the same way: the bottleneck is finding and normalizing each value, not finding the set or formatting the table.

A failure, cell by cell

The set is right; the rows are empty

The headline gap is easiest to see in a single table. Below, DeepSeek-V4-Pro is asked for Korea's eight metropolitan cities with each city's population and mayor. It recovers all eight cities (Item-F1 100) — but leaves the population column blank for every row and gets the mayor wrong or missing for six, so not one row is fully correct (Row-F1 0). High Item-F1 and zero Row-F1 in the same task is the pattern, not the exception.

Does strict scoring under-state the gap?

Our deterministic scorer is deliberately conservative, so we test it with a semantic second pass: for every cell it marks wrong on a soft-typed column (name, location, free text), an LLM judge (GPT-5.4-mini) decides whether the prediction still denotes the same answer — crediting transliteration variants and administrative granularity (Gangwon Chuncheon ≡ Chuncheon, Korea), but not a different entity, value, or fabricated specificity. What looks like GPT-5.5 cell mis-fill — the has-children enum written as a free-text count (있음 ≡ 자녀 3명), a residence coarsened (서울, 대한민국 ≡ 서울) — is mostly this kind of surface variant, not a wrong value.

Re-scoring this way lifts Row-F1 by only 0.8–4.9 points, and it lifts the stronger models more: among a model's judge-confirmed-wrong cells, the share the judge rescues rises with strength (GPT-5.5 35% vs K-EXAONE 9%). A strong model's residual errors are surface variants; a weak model's are substantive — a different entity, region, value, or category, as the residual taxonomy above shows. We keep the deterministic scorer as the reproducible default and report the judged numbers as a measurement-validity check, not a new ranking.

Construction & release

Built by synthesize-and-verify, released leakage-aware

Citation

BibTeX

@inproceedings{jeong2026kowidesearch,
  title     = {Ko-WideSearch: A Korean Breadth-Search Benchmark
               for Exhaustive Set Enumeration by Web Agents},
  author    = {Jeong, Minbyul},
  year      = {2026},
  note      = {Upstage AI},
  url       = {https://github.com/minstar/Ko-widesearch}
}