Building and debugging a terminal-agent RL pipeline

The result and the setup

I wanted a model to work in a terminal, have its files graded, and learn from the attempts. Prime RL handled training and asynchronous rollouts; Verifiers ran the agent loop; Harbor graded the files in a separate sandbox. Modal supplied the GPUs and environments.

I used Qwen3.5-9B with thinking enabled, Bash, and an edit tool. My work was wiring those pieces together, checking that their tokens and rewards agreed, and measuring where time and money went.

Adapter

LoRA rank 16, alpha 32, learning rate 1e-5.

Precision

FP8 inference; BF16 training with FP32 optimization and reduction.

Sampling

Temperature 0.6, top-p 0.95, top-k 20; 8,192 output tokens per request.

Context

50,000 tokens, compaction at 30,000, maximum 50 turns.

Choosing tasks the model sometimes solves

If all eight attempts on a task get the same reward, there is no reward difference for a group-relative update to learn from. I needed tasks that produced both passes and failures.

I adapted TermGrade’s measured-difficulty approach: screen the base model in the harness used for training. TermGrade separates its public Terminus-2 grading from its Bash training-selection measurements, then narrows the initial 25–75% band toward 50%. I kept the wider band.

Across 216 tasks with eight screening attempts each, 103 had two to six passes. I froze 80 for training and 23 for evaluation. These held-out tasks had already been screened and examined; they were excluded from updates, not a blind benchmark.

Before using GPUs, I pinned Git commits and file hashes and checked that a no-op failed while the reference solution passed. Twelve retained Dockerfiles had blocked native Harbor environment upload. Removing those active files restored setup without changing the instructions or tests.

Making terminal rollouts reliable

The generated tokens had to be trainable

My first setup generated answers I could not reliably train on. I moved to Prime’s integrated Bash path for tool execution, compaction, and token capture. I checked a full tool-call exchange for matching token prefixes, finite log-probabilities, and sampled tokens inside their recorded support.

Two migration mistakes mattered: older dependencies could not decode the packed log-probabilities and masks, and a system-only prefix failed with “No user query found in messages.” Restoring compatible versions and using Prime’s Qwen3.5 renderer fixed those paths. I had no evidence that Bash itself solved more tasks.

Commands that kept waiting

Some commands and sandboxes had stayed stuck for hours. The Bash executor allowed 3,600 seconds per command; a shorter HTTP timeout would still leave the process running inside the sandbox.

My patch closes stdin, starts a process group, and kills the group after 60 seconds. It returns partial output with a bounded follow-up read, so a child holding a pipe open cannot keep the call waiting. A Linux check stopped at 60.02 seconds, passed six process tests, and let the next command run.

That deadline also stops legitimate long builds. File edits remain, and detached processes still need sandbox cleanup. I could not attribute every earlier stall to this one cause.

Paying less while the model thinks

Sandboxes often waited while inference generated tokens. I reduced their requests from one CPU and 2 GiB to 0.125 CPU and 0.125 GiB. These were uncapped requests: an earlier sandbox had reached 9.32 GiB despite requesting 2 GiB.

Old floor

1 core + 2 GiB: $0.1899 per sandbox-hour.

New floor

0.125 core + 0.125 GiB: $0.02074 per sandbox-hour.

Modal bills the higher of requested or actual usage. The request floor fell about 89%; busy tasks still paid for extra usage. These historical estimates use sandbox rates of $0.00003942/core-second and $0.00000667/GiB-second, checked October 31, 2026. They exclude GPU and other service costs.

Keeping the task through a long attempt

Compaction could drop the exact task when rebuilding the prompt. I changed Verifiers to keep the original task plus the latest summary; repeated compaction replaces the old summary. CPU checks verified retention, but a bad summary can still carry a mistake forward.

Some responses used all 8,192 tokens thinking without calling a tool. I kept thinking enabled and added one corrective request per episode, based on mini-SWE-agent’s output-limit feedback. The unfinished response stays in the trace but leaves the next prompt. Recovery scored 9/16 with it and 9/16 without, using 62% more output tokens. It did not establish a pass-rate gain.

Throughput and what reached the trainer

Why I chose 64 per inference GPU

I benchmarked 16, 32, 64, 98, and 128 concurrent requests on one H100. Concurrency 128 had the best synthetic long-output throughput, but longer request latency and 25 observed KV-cache preemptions. The separate 98 run was slower than 64 despite no observed preemptions.

I chose 64 per inference GPU—128 globally—for the terminal rollouts. The benchmark helped choose a candidate; it did not prove that 64 produces usable training batches fastest.

Prime keeps collecting attempts while training updates and broadcasts weights. An attempt can start before an update and finish after it. I allowed one off-policy step, grouped eight attempts on each task, and filled each update with 64 accepted attempts.

The stream returned 1,394 episodes; ten batches accepted 640, including 365 passes and 275 failures. All 80 training tasks appeared, but only 55 contributed accepted trajectories. Prime dropped 306 completed stale traces; other extra or cancelled work had different causes.

Across roughly 3 hours 23 minutes of training, the trainer waited 141.5 minutes for batches and spent 52.5 minutes on forward/backward passes. Rollout collection kept it waiting far longer than the updates themselves. Gradients and entropy showed no obvious numerical collapse, but batch reward could not tell me whether unseen-task performance improved.

What the held-out traces showed

Both policies received eight attempts on each of 23 held-out tasks. All 368 were graded, with no infrastructure errors or timeouts. Eleven tasks declined, six improved, and six tied.

The adapter used more reasoning tokens and reached limits more often, while tool-result counts barely changed. Zero-tool attempts rose from four to ten, even though empty Bash calls fell from 32 to 18. The parser alone did not explain that pattern.

The traces also contained ordinary code bugs. One log-parser edit destroyed message whitespace:

" ".join("retry  after   timeout".split())
# 'retry after timeout'

Another attempt saw an empty CSV return no quantiles instead of three zeros, then declared completion without fixing it.

Passing the tests did not mean stopping

Of 640 accepted training trajectories, 452 were truncated, including 253 of the 365 passes. The grader checks files, so an attempt can earn reward even after continuing until a limit. Two final-evaluation attempts spent two full responses without any tool action despite recovery.

I used GRPO advantages with Prime’s IPO loss, token aggregation, no GRPO length penalty, and no separate completion reward. I suspect that gave the model too little incentive to stop once its files were correct. The traces support investigating that hypothesis; they do not prove it caused the score decline.

This was one small run. The paired-task bootstrap 95% interval for the score difference was approximately −13.0 to +4.35 percentage points. Base evaluation actually ran on H200 and final evaluation on H100 despite matching requests. I cannot claim a general improvement or a definitive cause for the lower score.

What the reward could establish

I used TermGrade correctness tests through Harbor. All 103 selected tasks used separate grading and cleared stale reward files; the grader received selected files, not the solver’s running processes. The shared image still defaulted to root without unprivileged Rewardkit workers, leaving a tampering gap identified through Harbor’s guidance. I found no evidence that it caused this regression.

The audit also needed debugging

My checkpoint check rejected trainer step 5 beside orchestrator step 6. Prime’s interval save records the orchestrator’s next step; both were at 10 in the final save. I fixed the check and verified metadata, model/optimizer state, scheduler progress, shard extents, and 256 finite adapter tensors. A full training resume was not tested.

A later audit ran outside Prime’s environment, missed zstandard, and searched the wrong trace directory. Using the native environment and indexed-trace reader let me keep the completed 184 base attempts and run only the missing adapter evaluation.