Can AI Replace Developers — New MIT Study

The Future of AI in Software Engineering

The authors emphasize that despite impressive progress, we are still far from fully realizing the promises of automation. They note that software engineering extends far beyond code generation and requires a deep understanding of context, systems thinking, and continuous interaction with existing infrastructure.

The work of programmers includes:
— creating new features
— improving architecture
— migrating millions of lines of legacy code
— regular testing
— documentation
— vulnerability analysis.

Thus, the profession of a programmer is much broader and more complex than simply implementing algorithms or solving interview-level problems.

Challenges in Measuring and Evaluating AI Progress for Programming

Optimizing code at an industrial scale, such as tuning GPU kernels or implementing complex multi-level improvements to the V8 engine in Chrome, remains one of the most difficult tasks for objective evaluation. Current performance metrics were designed for short, isolated scenarios, and while multiple-choice tests are still common in natural language processing research, they have never been an adequate measure for AI in programming.

The primary existing benchmark, SWE-Bench, evaluates a model's ability to fix bugs in GitHub repositories. This approach is useful but reflects only part of reality:
— tasks cover only a few hundred lines of code
— there is a risk of data leakage from open-source repositories
— it does not account for a wide range of practical scenarios, such as refactoring, human-AI collaborative programming, or critical system rewrites involving millions of lines of code.

Until such benchmarks learn to encompass larger and more comprehensive contexts, measuring real AI progress in software engineering will remain an unresolved challenge.

Challenges of Communication and Context in AI Code Work

If measuring progress remains one of the main problems, then communication between humans and machines is no less serious an obstacle. Current interaction between programmers and AI resembles a fragile and unreliable communication channel. When generating code, models often produce large, unstructured files and superficial unit tests that do not ensure actual correctness verification. This gap is also evident in the limited ability of AI to utilize a wide range of software engineering tools—from debuggers to static analyzers—which are necessary for deep understanding and code quality control.

The lack of transparency in model operations leads developers to often trust code that compiles but may prove erroneous when deployed in production systems. To overcome this barrier, AI must be able to convey its level of confidence in its decisions and timely seek clarification from the user.

Problems intensify with scale. Large codebases with millions of lines of code pose a significant challenge:
— corporate repositories have unique conventions and standards
— AI may generate code that appears correct but violates internal rules or fails CI/CD checks
— models make errors when searching for code fragments, relying on syntax rather than functional logic, which reduces the accuracy and reliability of suggestions.

A Call for Collaborative Efforts and the Significance of Research

The authors of the study emphasize that there is no universal solution to existing problems. They call on the scientific and engineering communities to collaborate: to create richer datasets that reflect the actual process of writing code—which fragments developers keep, which they delete, and how code evolves over time. Researchers also propose developing common test suites that would allow measuring progress not only in generation accuracy but also in refactoring quality, fix robustness, and migration correctness.

Special attention is given to developing transparent tools capable of showing the model's confidence level and involving humans in the decision-making process. This approach should transform AI from a simple autocomplete mechanism into a full-fledged engineering partner.

This is especially important because software underpins all key sectors—from finance and transportation to healthcare. Human resources are becoming a bottleneck, and it is AI, capable of safely taking over routine tasks, that will be able to free up specialists for creative and strategic decision-making.