evalbar

A command line evaluation for LLM systems. Point it at your own cases and it says pass, fail, or that the sample cannot tell, with the interval and the cost of a passed case. Everything on this page runs evalbar's own code in your browser.


    

Ask it about your own numbers


        

The exact (Clopper-Pearson) interval for one run; McNemar's mid-p and Tango's interval for two, paired case by case. PASS only when the whole interval clears the bar, and a difference only when its interval leaves zero.

What it said about real systems

Greek, graded right

A log line, scrubbed before it becomes a test


        

Greek tax numbers, AMKA and IBAN are taken only when their check digits hold. Names are the weak part: a capitalised pair is taken as a name, so a name typed in lower case gets through and a product with a capital letter does not.

Run it on your own cases

git clone https://github.com/Broikos-Nikos/evalbar && cd evalbar
npm ci
npx evalbar run your-cases.jsonl --base-url http://localhost:8080/v1 --model your-model --bar 0.9

A case is one JSON line: {"input": "...", "expected": "...", "tags": ["el"]}. Any OpenAI-compatible endpoint, cloud or local, or a program of your own. Exit codes are 0 for PASS, 1 for FAIL and 3 for CANNOT TELL, so a CI step can stop on the first two. Node 24, no runtime dependencies.

What this does not show