Open-Source Generative AI

With LLMs training on web-scale data, test-set contamination is a pervasive concern in LLM evaluation that can quickly render benchmarks obsolete. To mitigate this, many recent benchmarks crowdsource new prompts and evaluations from human or LLM judges; however, these can introduce significant biases, and they are not reliable for hard questions. For example, LLM judges make mistakes up to 40% of the time on challenging math and reasoning tasks.

To resolve this issue, we developed LiveBench, a contamination-limited benchmark that evaluates large language models on a variety of general intelligence capabilities including reasoning, coding, language understanding, data analysis, instruction following, and mathematics. By frequently releasing updated question sets and developing new tasks over time, we ensure that our results remain an accurate assessment of LLM capabilities as new models are released. Questions are constructed from a variety of recent sources such as research papers and news articles so they can be easily refreshed over time. LiveBench also uses only objective ground-truth judgment to ensure unbiased results.

Our leaderboard at livebench.ai presents results for all major model providers, both open source and proprietary. Since its release, LiveBench has remained one of the most popular LLM benchmarks and has been a featured result in major model release reports.