A PERSONAL BLOG

Notes along
the way.

Notes, ideas, and things I’m learning along the way.

Get Connected.

shiyid@cs.cmu.edu

Undergraduate & Graduate Students: Happy to chat about research, careers, or life in AI/ML, and explore collaborations.

Fellow Researchers: Happy to exchange ideas and collaborate.

Industry & Startups: Happy to discuss research questions and explore collaborations.

The latest.

03 ENTRIES

How Popular AI Benchmarks Are Built

Nine benchmark case studies, plus a general design question: should model scores determine task selection, and how do difficulty, quality, and representativeness interact?

34 min read

The AI says “done.” How do we check?

Verifiable and unverifiable benchmarks, explained through acceptance checks, fallible graders, and the gap between a highlight reel and reliable work.

7 min read