Can a Stochastic Parrot Learn new Tricks? Introducing Tutuca-Bench▾
Ask coding agents to use a new framework not in the training set* to solve 3 problems:
- Todo-MVC
- Temperature converter
- 7GUIs CRUD
Results:
🥇️ Claude Code
🥈️ Gemini CLI (best style)
🥉️ OpenCode + Big Pickle
Codex required identity verification so it was disqualified
* Using skills and a framework CLI to guide them