Two years ago, "training an agent" mostly meant writing a better system prompt. Today it means reinforcement learning: you let the model act in an environment, score whether it completed the task, and push the weights toward the behavior that worked. RL is now the standard post-training step for tool-using, multi-turn agents — because on a real task there is no single correct trajectory to imitate, only outcomes to optimize, and the agent has to learn to recover from its own mistakes.
The open-source tooling has finally caught up. But the map is confusing, because the projects sit on two different sides of the same problem, and most roundups list them as if they were interchangeable. They aren't.
The split that actually matters#
Every RL framework has to do two jobs: run the trainer (generate rollouts, compute the policy-gradient update, spread it across GPUs) and define the environment and reward (what the agent acts in, and how you score what it did). The trainers have largely converged — they all ship GRPO, they all lean on vLLM for fast rollouts, and the performance deltas between them are an infrastructure problem, not a research one.
The environment and the reward have not converged, because they can't: they're specific to your task. And they are where nearly all the difficulty — and all the reward-hacking risk — actually lives.
The trainer is table stakes. The environment is the moat.
So the right way to read this list is by asking how much a project does for the side that's hard.
The trainers#
verl is the default heavy trainer. If your problem is "I have the environment and the reward, I need production-grade RL infrastructure that scales," this is the one the field standardized on. It expects you to bring your own task.
trl is the on-ramp. It isn't agent-specific, but it's the cleanest place to understand GRPO before you commit to a heavier stack, and for single-turn reasoning tasks it may be all you need.
OpenRLHF and SkyRL are the scalable alternatives to verl — reach for them when its particular abstractions or infra assumptions don't fit yours. All three are trainers; none of them will help you write a good reward.
The environment-and-reward side#
This is the half that decides whether your agent learns the task or learns to game your metric.
verifiers is the clearest expression of the "environment is the product" thesis. It treats the environment and its reward verifier as the reusable, shareable artifact — which is exactly why Prime Intellect built a hub around it. If you're going to invest anywhere on this list, invest in the environment, and this is where you do it.
ART is the most agent-native pick, and RULER is the reason. Hand-authoring a reward function for a multi-step task is the step where most RL-for-agents projects quietly die; RULER lets an LLM judge grade the trajectory instead, which gets you to a first training run in an afternoon rather than a fortnight. The trade — an LLM-judge reward is itself gameable and has to be validated — is real, but it's the right default for getting started.
prime-rl is the trainer that assumes you bought the environment thesis: it's designed to run verifiers environments at scale, decentralized across compute you don't own in one datacenter.
How to choose#
If you have a working agent and want RL without a rewrite, start with ART — GRPO around your loop, RULER for the reward. If you're building environments and evals as the durable asset, start with verifiers. If you need maximum-scale trainer infrastructure and will write your own reward, verl is the default, with OpenRLHF and SkyRL as the alternatives when its abstractions don't fit.
But notice what that decision is not about. It isn't about whose GRPO is 8% faster. Every framework here has commoditized the algorithm; the thing none of them can commoditize for you is the environment and the reward. Pick for the moat, not the trainer.
