RL Environments: A New Trend in Silicon Valley and Billions in Investments

For many years, Big Tech leaders have painted pictures of a future where AI agents can independently use software to perform tasks on behalf of humans. But when trying out current consumer agents—whether it’s OpenAI’s ChatGPT Agent or Perplexity’s Comet—it becomes clear that the technology is still heavily limited. To make agents more reliable, new training methods are needed, which the industry is only beginning to master.

One such method is creating simulated workspaces where agents practice performing multi-step tasks. These systems are called reinforcement learning environments, RL. Just as labeled datasets became the foundation of the previous wave of AI, RL environments are gradually becoming a critical element in the development of agents.

Leading AI labs are increasingly seeking RL environments, and there is an abundance of startups ready to supply them.

Jennifer Lee, General Partner at Andreessen Horowitz
All major AI labs are building RL environments internally. But, as you understand, creating such datasets is difficult, so labs are also considering third-party developers who can produce high-quality environments and evaluations. Everyone is watching this direction.

The growing interest has spawned a new wave of well-funded startups, such as Mechanize and Prime Intellect, which aim to become leaders in the segment. Meanwhile, large data labeling companies like Mercor and Surge are also investing in RL environments to keep pace with the industry’s shift from static datasets to interactive simulations. Anthropic is considering spending over $1 billion on developing RL environments over the next year.

Investors and founders hope that one of these companies will become the “Scale AI for environments”—an analogue of the $29 billion data labeling giant that played a key role in the chatbot era.

What is an RL environment

It is a training ground where an agent learns to act in a simulation of real software.

For example, an environment might simulate the Chrome browser and assign a task to the agent—to buy socks on Amazon. The agent receives a score for execution and a reward signal if it successfully completes the task.

Although the task sounds simple, there are many potential errors: the agent might get confused by dropdown menus or order too many socks. Developers cannot predict all failures, so the environment itself must be flexible enough to capture unpredictable behavior while providing useful feedback. This makes them far more complex than static datasets.

Some environments are complex: they allow the use of tools, the internet, or different applications. Others are highly specialized, such as those for training work with corporate software.

The methodology is not new: as early as 2016, OpenAI created RL gyms, and DeepMind trained AlphaGo, which defeated the world champion in Go, using RL. The difference in today’s approach is that environments are now used to train universal transformer-based agents, rather than narrow specialized models.

Competition in the market

Data labeling companies—Scale AI, Surge, and Mercor—are striving to fit into the new trend. They have more resources than most startups and established connections with AI labs.

Surge CEO Edwin Chen sees “substantial growth in demand” for RL environments. Surge, which earned $1.2 billion last year through collaborations with OpenAI, Google, Anthropic, and Meta, has created a new division dedicated exclusively to RL environments.

Next to Surge is Mercor—a $10 billion startup that also works with OpenAI, Meta, and Anthropic. The company positions itself as a provider of RL environments for specialized fields such as programming, healthcare, and law.

Scale AI, which previously dominated data labeling, lost ground after Meta invested $14 billion and poached its CEO. Google and OpenAI stopped working with Scale, but the company is also betting on environments.

Among the newcomers, Mechanize stands out, founded just six months ago with the ambitious goal to “automate all professions.” For now, they are focused on RL environments for coding agents, offering engineers salaries up to $500,000 and already collaborating with Anthropic.

Another player is Prime Intellect, backed by Andrej Karpathy and funds Founders Fund and Menlo Ventures. The company positions itself as the “Hugging Face for RL environments,” creating an open hub and offering developers computing power.

Prospects and risks

Training agents in RL environments is much more expensive than in previous approaches and opens the door to abuse. Agents sometimes find a way to “cheat the reward system” without honestly completing the task.

Some experts are skeptical:

Ross Taylor, former head of research at Meta, now co-founder of General Reasoning
Even the best available environments require significant refinement.

Andrej Karpathy, investor at Prime Intellect
I believe in environments and agent interactions, but I am skeptical specifically about RL.

The question remains open: will RL environments become the next driver of AI progress?

Source: TechCrunch