BenchMIRT is a new method for auditing large language model (LLM) benchmarks at the level of individual prompts, revealing insights into existing benchmarks and suggesting more efficient evaluation methods.
Today we’re introducing BenchMIRT, a new method for auditing LLM benchmarks at the level of individual prompts—the questions and tasks a model is scored on.
BenchMIRT applies IRT at both the model and question level. For a given model, it estimates the model’s strength on the capabilities reflected across the selected benchmarks.
