Skip to content

Add Coder Eval to Benchmark - #535

Open
uipreliga wants to merge 1 commit into
InftyAI:mainfrom
uipreliga:add-coder-eval
Open

uipreliga wants to merge 1 commit into
InftyAI:mainfrom
uipreliga:add-coder-eval

Conversation

@uipreliga

Copy link
Copy Markdown

Adds Coder Eval to Benchmark, alphabetically between ASQI Engineer and LiveBench.

Disclosure: I maintain this project. Happy to reword or move it — Tool would also be reasonable if you read it as tooling rather than a benchmark.

Why it belongs next to these

The section already carries lm-evaluation-harness, OpenCompass, opik, terminal-bench and skill-optimizer. Coder Eval is closest to that last pair: rather than scoring a model on a fixed dataset, it holds the task suite fixed and varies the harness and the skills, then reports how each configuration did.

Its specific measurement is skill activation — whether an agent's skill actually fires on the prompts it claims to handle, and stays quiet on the rest — expressed as suite-level precision / recall / F1 over a labelled dataset, with thresholds that can fail a CI run when they regress.

I matched the section's entry format exactly (bold name, colon, description, then the stars / contributors / last-commit badges) and kept alphabetical order. One line, no other changes.

Signed-off-by: uipreliga <tomasz.religa@uipath.com>

Co-Authored-By: Claude <noreply@anthropic.com>
@InftyAI-Agent InftyAI-Agent added needs-triage Indicates an issue or PR lacks a label and requires one. needs-priority Indicates a PR lacks a label and requires one. do-not-merge/needs-kind Indicates a PR lacks a label and requires one. labels Aug 28, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/needs-kind Indicates a PR lacks a label and requires one. needs-priority Indicates a PR lacks a label and requires one. needs-triage Indicates an issue or PR lacks a label and requires one.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants