OpenAI’s Chief Scientist is calling for the AI community to rally around perplexity as a standard for evaluating advances in model capabilities. This move aims to establish shared benchmarks and improve transparency across the fast-evolving field.
What Happened
On November 22, 2026, OpenAI’s Chief Scientist Jakub Pachocki explained why perplexity—a statistical measure indicating how well a model predicts data—should serve as a primary metric for tracking progress in AI models. Pachocki revealed internal OpenAI measurements charting perplexity reductions from GPT-2 to recent models. He disclosed that OpenAI now achieves log-perplexity below 0.8 on its main internal evaluation dataset, a threshold linked to new, transformative AI capabilities. Although sharing these metrics could offer competitors insights into OpenAI’s development, the company believes it is a necessary step to promote common safety standards and transparent progress tracking across labs and developers. OpenAI proposes that model developers across the industry should openly report perplexity results, adopt benchmarking norms, and even consider pacing development based on agreed perplexity limits to manage emerging risks.
Key Takeaways
- OpenAI internally measures model progress using log-perplexity on a held-out dataset.
- GPT-3.5 achieved about 0.99 log-perplexity; GPT-4 reached approximately 0.91.
- Recent OpenAI models now report losses below 0.8, corresponding to new capabilities in cybersecurity and mathematics.
- Perplexity is not standardized across labs due to differing datasets and data mixtures, making direct comparisons challenging.
- OpenAI calls for industry-wide transparency and suggests creating shared norms for evaluating and pacing AI development based on perplexity.
The Details
OpenAI describes AI model development as training large ‘compressors’ on unstructured data, with progress tracked by measuring perplexity on unseen evaluation datasets. Perplexity quantifies how well a model predicts the next token: the lower the number, the better the model performs. OpenAI notes that transitions through key perplexity thresholds have closely correlated with emergent behaviors. For example, GPT-2, at a then state-of-the-art perplexity, could form coherent paragraphs; GPT-3 introduced in-context learning; GPT-3.5 underpinned conversational fluency in ChatGPT; and GPT-4 unlocked advanced reasoning. OpenAI now observes that dropping below 0.8 in log-perplexity has enabled the models to excel in complex and previously unsolved maths and cybersecurity tasks. The company uses an internal evaluation set based on its codebase to measure these figures, although it acknowledges the limits of such data for general-purpose benchmark comparisons.
OpenAI concedes there are challenges associated with perplexity as a shared metric: different labs use distinct data for evaluation; perplexity does not always map to real-world usability; and it may not account for post-training advances, such as reasoning capabilities from techniques like chain-of-thought prompting. Furthermore, it’s unclear what the irreducible minimum (‘entropy’) of perplexity might be. Progress toward that floor could slow, but small gains might yield disproportionate jumps in model potential. OpenAI considers perplexity a robust and time-tested measurement, and urges the industry to set agreed-upon limits and conditions tied to perplexity before crossing new capability thresholds.
Background and Context
Monitoring perplexity has long been standard in NLP model training, but has typically been an internal metric. Earlier models like GPT-2 and GPT-3 were landmarks for OpenAI in terms of capability leaps tied to improved perplexity scores. Recently, the field has seen much quicker advances, partly attributed to increased compute and automation, and partly to AI contributing design ideas to its own development. OpenAI has maintained that unpredictable emergent capabilities often arise as models pass certain perplexity thresholds. Today, shifting toward shared and transparent reporting could lay the groundwork for industry-wide benchmarks and safer AI development practices.
BytesWall Analysis
While OpenAI’s call for perplexity-based reporting promises greater transparency and comparability in AI progress, key risks and trade-offs come into focus. One significant trade-off is that perplexity is heavily dataset-dependent; with each lab using different evaluation corpora, results may not be directly comparable. This could create a false sense of alignment or progress across vendors. Another risk is that focusing on a single metric like perplexity might overlook advances gained through other methods, such as instruction tuning or prompt engineering, which do not always lower perplexity but can vastly improve model usefulness. Furthermore, as models help automate their own training, the accelerating timeline for breakthroughs could outpace the industry’s ability to set and enforce safety norms. The open question is whether a consensus can be reached on standardized datasets or reporting protocols, given competitive and proprietary concerns. For practitioners, the practical step is to begin publishing at least partial perplexity scores on representative datasets, and to engage with partners and consortia in defining what datasets and thresholds should count as common benchmarks. This approach will foster transparency and may position organizations to adapt more quickly to emerging industry standards as they coalesce.
What to Watch Next
- Will key industry players agree on a standard evaluation dataset or protocol for perplexity?
- How will labs adapt their safety and alignment monitoring as models reach lower perplexity and potentially unpredictable new capabilities?
- What mechanisms will be put in place to enforce agreed development limits based on perplexity or similar metrics?
Read more in our AI News Hub
Source: read the original announcement. BytesWall adds summary and analysis.