Jodie Burchell 🇦🇺🇩🇪 @t_redactyl@fosstodon.org
Public

Have you ever looked at the impressive results that LLMs get on benchmarks or difficult exams and wondered if these results are everything they seem? Some of these are due to something called data leakage, where assessments leak into the training data and the models memorise the answers.

If you'd like to learn more about how this classic pitfall of machine learning calls many of the results we see on LLM performance into question, check out my latest blog post.

t-redactyl.io/posts/2025-12-30

Data leakage is a major issue when measuring LLM performance

Why data leakage and benchmark contamination distort LLM performance claims, from coding puzzles to the LM Arena and training data exhaustion.

t-redactyl.io

0
0
0

If you have a fediverse account, you can quote this note from your own instance. Search https://fosstodon.org/users/t_redactyl/statuses/115888501770326823 on your instance and quote it. (Note that quoting is not supported in Mastodon.)