Join the Snorkel AI Reading Group, a recurring forum to explore the latest frontier developments in AI while building meaningful connections within the community.

In this session, lead researcher Henry Ehrenberg will present Senior SWE-Bench, Snorkel's new open-source benchmark for evaluating coding agents on the work we actually give them.

Agenda:

4 pm - doors open

4:30 pm - talk begins

Boba tea & other refreshments will be provided !

Among other things, you'll learn:

Why most coding benchmarks treat agents like junior engineers (over-specified requirements, graded mainly on whether the code runs) when most of us already treat agents like senior engineers who work from a Slack message, not a spec.

How Senior SWE-Bench's validation agent breaks the trade-off between realistic instructions & reliable grading: it writes behavioral tests adapted to each agent's actual solution, using expert-designed recipes the solving agent never sees.

Why the benchmark's bug & performance tasks are sourced from real PRs with evidence of significant runtime investigation (logs, profiling data, reproduction steps) - not tasks solvable by pattern-matching hints in the instructions.

How the taste judge scores tasteful solves on minimality, approach quality, hygiene, fluency, & craftsmanship relative to the reference implementation, so agents are rewarded for shipping the right code, not just code that passes tests.

Why even frontier models fall short: Claude Opus 4.8 leads the leaderboard at just 24.0% tasteful solve rate, meaning top models fail senior-level correctness & taste bars on more than three-quarters of tasks.

How the 100-task benchmark (50 public, 50 held out to guard against contamination) draws from 12 real repos spanning libraries to multi-service applications - from Postgres sync engines to self-hosted Git forges - authored by engineers with deep commit history in each codebase.

Senior SWE-Bench is open-source & Harbor-compatible, developed by Snorkel AI in collaboration with Princeton & UW-Madison. Explore the leaderboard Dataset on GitHub