PPO vs GRPO — Post-Training Qwen2.5-0.5B-Instruct on GSM8K with veRL

This post compares PPO and GRPO for post-training Qwen2.5-0.5B-Instruct on the GSM8K mathematical reasoning benchmark using veRL.

On Qwen2.5-0.5B-Instruct trained on GSM8K with a rule-based verifiable correctness reward, PPO and GRPO reach very similar test accuracy, while GRPO trains faster without a critic network.

Concretely:

• The base model scores 45.19% on GSM8K under the final evaluation pipeline.
• PPO improves it to 58.83%.
• GRPO improves it to 57.47%.
• GRPO trains ~23% faster and processes ~33% more tokens per second than PPO training.

code repo and docker image for the experiment: gong208/hands-on-modern-rl:ppo-vs-grpo-verl0.5-vllm0.10.1-ray2.55-20260623

---

Why Compare PPO and GRPO?

PPO is a standard RL algorithm used in many RLHF pipelines. In the LLM setting, PPO usually trains an actor model and a critic model together:

• the actor generates responses;
• the critic predicts the expected future return;
• the critic is used to estimate the advantage for policy optimization.

This critic is useful because it provides a learned baseline \(V(s)\). However, it also adds cost: it is another model to store, train, and update.

GRPO removes this critic. Instead of learning a value baseline, it samples multiple responses for the same prompt and compares each response against the group’s average reward. The advantage is computed from relative performance inside the group.

In other words:

方法	Advantage 估计	Advantage 估计中的 baseline
PPO	(\hat{A}_t = \hat{V}t - V\phi(s_t))，通常用 GAE 计算	学习得到的 value function / critic
GRPO	(\hat{A}_i = (R_i - \operatorname{mean}(R)) / \operatorname{std}(R))，在采样组内计算	采样组内的平均 reward

This makes GRPO attractive for LLM reasoning tasks, especially when the reward is easy to verify. GSM8K is exactly such a setting: the final answer can be extracted and compared with the ground-truth answer.

Additionally with the verifiable reward, this experiment does not require a learned reward model. Both PPO and GRPO optimize the same rule-based correctness reward on correctness.

---

Training and Evaluation Setup

Experiment Setup

Both PPO and GRPO were trained inside the same veRL pipeline.

The controlled variables were:

Setting	Value
Model	Qwen2.5-0.5B-Instruct
Dataset	GSM8K train / test
Train set size	7,473 problems
Test set size	1,319 problems
Framework	veRL 0.5.0 + vLLM 0.10
Hardware	Single NVIDIA L40S, 48 GB
Epochs	20
Optimizer steps	1,160
Train batch size	128 prompts
PPO mini-batch size	64
Samples per prompt	4
Max prompt length	512 tokens
Max response length	256 tokens
Actor learning rate	1e-6
KL regularization	actor-side KL loss against reference model
KL coefficient	0.001
KL type	low_var_kl
Clip ratio	0.2
Entropy coefficient	0
Reward	correctness only

The optimized reward is binary correctness:

score = 1 if the extracted answer matches the ground truth
score = 0 otherwise

A format flag is also computed and logged, but it is not added to the optimized score. Therefore, the training reward does not explicitly reward length, style, or answer formatting beyond whatever is needed for the answer extractor to identify the final answer.

The only intended algorithmic difference between the two runs is the advantage estimator:

	PPO	GRPO
Advantage estimator	GAE	group-relative baseline
Critic network	yes	no
Critic learning rate	1e-5	N/A
Advantage baseline	learned value function	mean reward over 4 responses for the same prompt

The equal sampling budget matters. PPO is often run with one response per prompt, while GRPO naturally requires multiple responses per prompt. Here, both methods use rollout.n = 4, so the comparison focuses on the advantage estimator rather than simply giving GRPO more samples.

---

Evaluation Protocol

The final comparison uses the same evaluation harness for all three models:

Setting	Value
Models evaluated	base, PPO, GRPO
Dataset	GSM8K test set
Number of problems	1,319
Prompting	0-shot
Decoding	greedy
Max response length	512 tokens
Precision	bf16
Scoring	same answer matcher as the training reward

The base model is evaluated through the same pipeline as PPO and GRPO, except without any RL training. This makes the three-way comparison directly comparable.

---

Results

Accuracy Results

Model	GSM8K test accuracy	Correct / total	Improvement over base
Qwen2.5-0.5B-Instruct	45.19%	596 / 1319	—
PPO	58.83%	776 / 1319	+13.64 pts
GRPO	57.47%	758 / 1319	+12.28 pts

Both PPO and GRPO substantially improve over the base model.

PPO is the highest single run, but the gap between PPO and GRPO is small:

PPO - GRPO = 58.83% - 57.47% = 1.36 percentage points

This corresponds to 18 more correct answers out of 1,319 test problems.

I would not over-interpret this gap. This is a single-run comparison on one model size and one dataset. Without multiple random seeds or a paired significance test, the result is better read as:

PPO and GRPO have very similar final accuracy in this setting.

Best in-training validation accuracy was also close:

Method	Best validation accuracy
PPO	59.14%
GRPO	58.76%

So the training dynamics and final test results tell the same story: both algorithms improve the model substantially, and their final quality is very close.

---

Training Cost Results

The main difference is training cost.

Metric	PPO	GRPO	GRPO vs PPO
Wall-clock time	8.47 h	6.49 h	−23%
GPU-hours	8.47	6.49	−23%
Estimated cost at $1.50 / GPU-h	$12.71	$9.74	−23%
Median throughput	6,138 tok/s	8,192 tok/s	+33%

GRPO is faster because it does not train a critic. PPO spends part of each training step updating the value model, including critic forward passes, backward passes, and optimizer steps. GRPO replaces this learned value baseline with a simple group-level reward baseline.

In this experiment, PPO’s critic did not produce enough measurable accuracy gain to justify that additional training cost.

A careful version of the conclusion is:

On Qwen2.5-0.5B-Instruct trained on GSM8K with a verifiable correctness reward, GRPO is the more cost-effective choice. It reaches nearly the same accuracy as PPO while avoiding the critic update.

---

Memory Results

The memory picture is slightly subtle.

Metric	PPO	GRPO	Difference
Peak GPU memory fromnvidia-smi	47.36 GB	47.31 GB	~0 GB
Torch max allocated	50.71 GB	45.19 GB	−5.52 GB

The physical peak reported by nvidia-smi looks almost identical. This is likely because the vLLM rollout engine reserves a large fixed memory region, so both runs appear to fill most of the GPU.

The PyTorch max-allocated metric shows a clearer difference: GRPO uses about 5.5 GB less working memory. This is consistent with the algorithmic difference: PPO has to maintain and update a critic model, while GRPO does not.

At the 0.5B scale, this memory saving is real but not decisive because both methods fit on a single L40S. At larger model sizes, dropping the critic may become more important.

---

Inference Profile

After training, I also profiled the final models on the GSM8K test set using batched vLLM greedy decoding with the same prompts.

Model	Avg output tokens	Total generation time	Decode throughput
Base	296.8	17.0 s	23,047 tok/s
PPO	158.7	10.4 s	20,127 tok/s
GRPO	152.8	9.7 s	20,846 tok/s

The most interesting difference is output length. Both RL-trained models generate much shorter answers than the base model:

Base: ~297 tokens
PPO:  ~159 tokens
GRPO: ~153 tokens

So PPO and GRPO roughly halve the response length.

This conciseness was not explicitly rewarded. The reward only checks correctness. However, the training response budget is limited to 256 tokens. A verbose solution is more likely to be truncated before reaching the final answer, which receives zero reward. As a result, the model is indirectly pushed toward shorter and more decisive reasoning.

This is an emergent side effect of the reward and length constraint, not a direct length penalty.

---

Qualitative Result

The quantitative results show that RL improves accuracy. A qualitative example helps show what kind of behavior changes.

The following example comes from an earlier sanity-check evaluation comparing the base model and a PPO-trained model. Both models received the same 4-shot chain-of-thought prompt followed by this GSM8K question:

Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers’ market daily for 2 dollars per fresh duck egg. How much in dollars does she make every day at the farmers’ market?

The correct solution is:

16 eggs total
3 eggs eaten for breakfast
4 eggs used for muffins
16 - 3 - 4 = 9 eggs sold
9 * 2 = 18 dollars

So the correct answer is 18.

Base model output

To determine how much Janet makes every day at the farmers' market, we need to follow these steps:

1. Calculate the number of eggs laid by the ducks each day.
2. Determine the total revenue from selling the eggs.

**Step 1: Calculate the number of eggs laid by the ducks each day.**

Janet's ducks lay 16 eggs per day.

**Step 2: Determine the total revenue from selling the eggs.**

Janet sells the eggs at $2 per fresh duck egg. Therefore, the total revenue is calculated as follows:

Total Revenue = Number of Eggs Laid Each Day × Price per Egg

Substituting the values:

Total Revenue = 16 × 2 = 32

Thus, Janet makes \boxed{32} dollars every day at the farmers' market.

The base model gives the wrong answer, 32. It identifies the number of eggs laid and the price per egg, but skips the subtraction step. It ignores the eggs eaten for breakfast and the eggs used for muffins.

This is a typical reasoning shortcut: the response looks structured, but the actual computation is incomplete.

PPO model output

First, we need to calculate the total number of eggs Janet lays in a day. She lays 16 eggs per day.

Next, we calculate the total number of eggs she eats in a day. She eats 3 eggs for breakfast and 4 eggs baking muffins,
so she eats 3 + 4 = 7 eggs per day.

Subtracting the number of eggs she eats from the total number of eggs she lays gives us the number of eggs she sells:
16 - 7 = 9 eggs per day.

She sells each egg for $2, so the total revenue she makes every day is 9 × 2 = 18 dollars.

Therefore, the amount she makes every day at the farmers' market is \boxed{18}.

The PPO model gives the correct answer, 18. It accounts for all consumed eggs before calculating revenue.

Aspect	Base model	PPO model
Final answer	wrong	correct
Main error	skips subtraction	accounts for consumed eggs
Reasoning style	verbose but incomplete	concise and complete
Format	includes final boxed answer	includes final boxed answer

This example should not be treated as proof by itself, but it illustrates the same trend as the aggregate results: RL training reduces some reasoning shortcuts and encourages the model to complete the necessary arithmetic steps.

---

Discussion

Why This Setting Favors GRPO

GSM8K is a verifiable-reward task. The final answer can be extracted and compared against the ground-truth answer. Therefore, neither PPO nor GRPO needs a learned reward model in this experiment.

This matters because the reward is cheap and direct:

answer matches ground truth → reward 1
answer does not match → reward 0

In this setting, GRPO has a natural advantage. It can sample several responses for the same prompt and compare them using the rule-based reward. If one response is correct and another is wrong, the group-relative advantage provides a useful learning signal without needing a critic.

PPO also uses the same rule-based reward. However, PPO still trains a critic to estimate expected future return. The critic is not the reward model. It does not decide whether the answer is correct. Instead, it tries to predict the expected return from a partial generation state.

So the comparison here is not:

learned reward model vs rule-based reward

Both methods use the same rule-based reward.

The comparison is:

learned value baseline vs group-relative reward baseline

In this experiment, the group-relative baseline is sufficient.

However, this conclusion may not transfer to harder settings where:

• the reward is noisy;
• the reward comes from a learned preference model;
• correctness is not directly verifiable;
• the task requires long-horizon credit assignment;
• there are many partially correct answers;
• the model scale is much larger;
• training stability matters more than final single-run accuracy.

In those cases, PPO’s critic may still provide useful variance reduction or stability.

---

Limitations

This experiment has several important limitations.

First, it uses only one model size: Qwen2.5-0.5B-Instruct. The result may change for larger models. A critic may become more or less useful depending on model scale, batch size, reward quality, and training stability.

Second, it uses only one dataset: GSM8K. GSM8K has short math word problems with verifiable final answers. This is very different from open-ended helpfulness, instruction following, dialogue, coding style, safety, or preference optimization.

Third, the reward is binary and rule-based. This makes the experiment clean, but it also removes many complications of RLHF. In real preference-based RLHF, the reward model can be imperfect, and the critic may behave differently.

Fourth, this is a single-run comparison. PPO is higher than GRPO by 18 problems on the test set, but without multiple seeds or a paired significance test, I would not claim that PPO is meaningfully better.

Finally, both PPO and GRPO use rollout.n = 4. This is useful for fairness because both methods receive the same sampling budget, but it is not necessarily the most common or most efficient PPO configuration.

---

Takeaways

1. Both PPO and GRPO substantially improve the base model.
   On GSM8K, the base model scores 45.19%, while PPO reaches 58.83% and GRPO reaches 57.47%.
2. PPO and GRPO have very similar final accuracy in this setting.
   PPO is higher by 1.36 percentage points, or 18 problems out of 1,319. This is a small single-run gap, not strong evidence that PPO is fundamentally better here.
3. GRPO is cheaper.
   GRPO trains about 23% faster and reaches 33% higher median throughput because it removes the critic update.
4. The critic does not buy much measurable accuracy here.
   In this specific small-model, verifiable-reward GSM8K setup, PPO’s learned value function does not provide enough accuracy gain to justify its extra cost.
5. This does not mean critics are generally unnecessary.
   The conclusion is specific to Qwen2.5-0.5B-Instruct, GSM8K, rule-based correctness reward, and equal sampling budget. For noisy, preference-based, non-verifiable, or long-horizon tasks, PPO’s critic may still be useful.
6. Both RL methods make the model more concise.
   The base model averages about 297 output tokens, while PPO and GRPO average around 159 and 153 tokens. This conciseness emerges even though the reward only scores correctness, likely because the bounded response length implicitly penalizes rambling answers that fail to reach a final answer.

Overall, this experiment suggests that for small LLMs on verifiable math reasoning tasks, GRPO is a strong low-cost alternative to PPO. It removes the critic, reduces training cost, and still obtains almost the same accuracy improvement over the base model.