EleutherAI lm-evaluation-harness Overview

Framework Scope

The project provides a unified framework to test generative language models on a large number of different evaluation tasks. It includes over 60 standard academic benchmarks for LLMs, with hundreds of subtasks and variants implemented. This documentation describes the available evaluation scope without implying local execution or specific hardware performance metrics for any particular machine.

Supported Interfaces

The harness supports models loaded via transformers, GPT-NeoX, and Megatron-DeepSpeed, featuring a flexible tokenization-agnostic interface. It also supports fast and memory-efficient inference with vLLM, as well as commercial APIs including OpenAI and TextSynth. These capabilities are documented features, not verified local installations or performance guarantees for specific hardware configurations.