Read May 2026

ProgramBench: Can Language Models Rebuild Programs From Scratch?

My takeaway

Problem

It's a benchmarking dataset for models to challenge not just its coding a single functionality of or a single module, instead it tests models on creating a whole project codebase on its own without HITL.

Solution

Now that the llms write code in any programming language it has been trained on, it can only write a single functionality or maybe a simple module without human interpretation. So this dataset extracts around 200+ code based mostly written on a standalone compiling language - c, c++, rust. From these project repos a task, requirement is generated using LLM. The evaluation models are prompted with this task and requirement files and the llm must generate the whole code base with any programming language on its own , the one which is good at. Results are surprising only claude opus has 95% completion on 3% of 200+ tasks. The success rate is defined by “input-output behaviour”. This tells us the llms have not yet reached the level where it takes architectural decisions on its own to code a large scale project. But this is actually challenging as I saw a project with max size is 2 lakh lines of code and 5000 project files.

benchmark
dataset