Putting embodied reinforcement learning into practice requires the training framework and the compute platform to work together. RLinf and Biren combined RLinf’s training and scheduling capabilities with Biren GPUs and the BIREN SUPA™ software stack. The collaboration covered platform integration, model migration, numerical alignment, and system performance.

Online embodied RL is a demanding workload. A single multi-GPU node has to run actor training, rollout inference, and physics simulation together. Vision-language-action (VLA) models add multi-step flow matching denoising, shared attention across two streams, and mixture-of-experts (MoE) action experts. The compute stack needs to support these operations, while the framework has to coordinate resources and data flow and keep training stable.

Infinigence AI and Tsinghua University launched RLinf with Zhongguancun Academy and other institutions. In this collaboration, Infinigence AI and Biren integrated RLinf with SUPA, ran Group Relative Policy Optimization (GRPO) training for LingBot-VLA 1.0 and 2.0, and checked numerical agreement across platforms using matching configurations. At the time of the original publication, SUPA support had landed on RLinf’s main branch (commit 7b3d874), the LingBot-VLA-V2 SFT and GRPO integration was under community review (PR #1652), and the team planned to upstream a set of validated, platform-independent system optimizations.

Biren and the RLinf community co-authored this account of the four stages.

01

1.1

Change Submission Status at publication
SUPA accelerator support (BirenSUPAManager) PR #1581, 5 files and 248 added lines Merged into RLinf main (commit 7b3d874)

The PR stayed small by implementing platform-specific behavior at the framework’s existing extension points.

Overview of the integration between RLinf and the Biren SUPA software stack, in Chinese

1.2

RLinf’s Accelerator Manager gives heterogeneous accelerators common interfaces for device discovery, visibility, communication backends, and profiling configuration. PR #1581 adds BirenSUPAManager at rlinf/scheduler/hardware/accelerators/biren_supa.py using those interfaces.

  • Device discovery. torch_supa exposes the torch.supa namespace. get_num_devices() calls torch.supa.device_count(), allowing Ray to allocate workers to GPUs.
  • Device visibility. The integration uses SUPA_VISIBLE_DEVICES and sets RAY_EXPERIMENTAL_NOSET_SUPA_VISIBLE_DEVICES=1 to keep Ray from overwriting device selection, a problem encountered during other accelerator integrations.
  • Collective communication. The manager returns Biren’s BCCL backend, which serves the role of NCCL, and maps the socket interface variable to BCCL_SOCKET_IFNAME.
  • Device type aliases. torch_supa aliases cuda to supa, so tensor.device.type is "supa". Code paths that previously tested for "cuda" rely on get_device_type() returning the actual type.

1.3

The runtime image includes the SUPA stack. Because torch_supa is unavailable on PyPI, the virtual environment reuses the image’s installation with --system-site-packages. Biren and the community generalized earlier vendor-specific installation logic into three functions: configure_vendor_torch_platform, seed_vendor_torch_metadata, and install_vendor_torch_extras. A new platform can use a single declaration to opt in.

seed_vendor_torch_metadata copies the vendor package’s METADATA, WHEEL, and INSTALLER into the virtual environment and leaves RECORD empty. This lets uv recognize the image’s PyTorch as installed and avoid shadowing it with a CUDA build. The empty RECORD also keeps uninstall operations from deleting the image’s original files.

02

2.1

Change Submission Status at publication
LingBot-VLA-V2 SFT and GRPO integration PR #1652 Under community review

2.2

LingBot-VLA-V2 is a 6.37B-parameter VLA model with a Qwen3-VL-4B backbone, a Qwen2 Action Expert, and a flow matching action head. It concatenates Q, K, and V from its perception and action streams along the token dimension and applies attention jointly. Its action expert includes a 32-expert token MoE. The model uses a padded 55-dimensional canonical action and state space, while RoboTwin executes 14-dimensional actions. The upstream framework had yet to support online RL for this model. Biren ported it to RLinf and validated the port as a baseline for cross-platform alignment.

2.3

Platform differences affect installation, scheduling, data paths, and the backward graph. The port organizes these changes as patches.

  • The upstream RLinf repository stays clean and pinned to a baseline commit.
  • A separate repository mirrors the original module paths, for example rlinf_br/patches/rlinf/scheduler/....
  • Each corresponding file defines only the changed functions or classes, keeping the diff easy to review.
  • A patch manager injects the changes at runtime. sitecustomize.py loads them once per interpreter, and RLINF_BR_AUTO_PATCH=0 disables the entire set.
  • The team can toggle, review, and validate the Biren port by module. Upstream upgrades only need to reconcile the functions that differ.

2.4

The team addressed attention layout compatibility, alternatives to upstream fused kernels, limits on certain backward paths, and operators with equivalent semantics but different intermediate representations. They established a stable baseline with equivalent fallbacks: explicit contiguous conversion and matrix multiplication, eager MoE, moving very small tensors to the CPU, and limiting the backward graph with rl_trainable_scope. That baseline supports longer training checks before individual fallbacks are replaced with native high-performance kernels. Each fallback has unit tests and per-step audits based on the comparison criteria in the next section.

03

Online RL makes step-by-step reproduction difficult. A plausible loss curve gives little information about whether the data, model structure, and algorithm agree after switching accelerators.

Biren extended its numerical alignment tool, Biren Training Tools (BRTT), while adapting RLinf and LingBot-VLA-V2 GRPO to SUPA. BRTT dumps tensors at defined points in the pipeline, compares them layer by layer, and identifies the first divergence. The authors report that this turned investigations that previously took weeks into a reproducible workflow that takes hours.

3.1

BRTT’s LLM reinforcement learning workflow already grouped checks into four stages: rollout sampling, advantage normalization, actor forward and backward passes, and gradient and weight synchronization. The RLinf integration adds a stage for simulator interactions. Hooks capture observations, actions, and rewards at the simulator subprocess boundary so a divergence can be traced to a specific stage and worker.

BRTT has a framework-independent engine and uses hooks to bridge into the framework without modifying the training script.

BRTT engine, bridge, and workflow layers, with the staged tensor comparison and first-divergence procedure, in Chinese

  • Engine. Forward and backward nn.Module hooks capture submodule inputs and outputs. Dump adapters support CUDA, SUPA, and DTensor. The comparison engine grades results as PASS, OK, WARN, or FAIL using cosine similarity and maximum error. It applies architecture-specific tolerances for low-precision types such as bf16.
  • Bridge. The lightweight rlinf_align package propagates across the Ray cluster through RAY_RUNTIME_ENV_HOOK and maps training stages to standard comparison groups. Each dump carries the step and worker identifiers.
  • Workflow. A single YAML configuration controls dumps from both platforms, comparison, and HTML reporting. It also sets the fixed seed and zero sampling temperature and checks SHA-256 hashes for weights and data.

BRTT dumps, compares, and grades each group in pipeline order. At the first divergence it stops and identifies the corresponding worker and tensors, so downstream amplification does not obscure the cause. After a fix, the team reruns the checks through weight synchronization. A new framework needs a registry of instrumentation points, and new hardware needs a dump adapter.

The checks cover the full embodied RL data lifecycle. BRTT stops subsequent dumps when any layer fails. Each record identifies the worker, allowing the team to trace the first divergence to a step, a worker type, and a tensor.

  • Simulator, dataloader, and initial weights. BRTT compares hashes for the full initial parameter manifest. It checks simulator observations at the EnvWorker.env_interact_step boundary, before they enter rollout or training. This catches differences in simulator versions, rendering configuration, and camera parameters.
  • Input and output shapes. BRTT audits more than 30 fields across the input, output, and loss for every actor microbatch. Shape checks are the cheapest and fastest first pass.
  • Forward and backward numerical checks. Module hooks dump forward outputs, log probabilities, losses, and gradients. Cosine similarity grades the results and helps identify the first divergent layer.
  • Worker unit tests. Consistency reruns and single-worker replay for actor, rollout, and environment workers help isolate cluster orchestration differences from numerical differences.

The following example shows how BRTT reports kernel differences by worker.

Kernel Detection Worker Example error Grade
FlashAttention forward kernel in the vision tower’s attention layer Layerwise comparison with forward hooks, reporting the first layer with cosine similarity below 0.999 Actor / rollout (shared forward pass) Cosine 0.9987, MaxAbsErr 8.1e−3 WARN
bf16 reduction kernel in the all-reduce gradient accumulation path Backward gradient comparison and rank-level attribution Actor (gradient synchronization group) Cosine 0.995, drift of 1 ulp WARN
RMSNorm forward pass Module forward hook Actor / rollout Cosine 0.9992 OK

The report finds the first kernel outside tolerance along the data path and checks whether subsequent operations amplify or attenuate the difference. This helps avoid filing amplified downstream differences as separate bugs.

For LingBot-VLA-V2 rollout, the team audited the MoE expert computation in three steps.

  • Module tree. Structural signatures matched for all 1,466 shared modules in the rollout worker’s nn.Module tree. The Biren implementation added 36 SUPA fused operator wrappers.
  • MoE calls. The (T, D, E, I, K, calls) contract matched on each forward pass, supporting agreement on the mathematical path.
  • Routing workload. Both implementations routed 110,160 token rows across the MoE operators, with every row accounted for exactly once.

The report classified the implementations as LOGICALLY ALIGNED / PHYSICALLY DIFFERENT. The reference rollout uses native Triton grouped kernels, and Biren uses fused SUPA operators. The audit verified the shared mathematical path while identifying the expected implementation differences.

3.2

To interpret differences across platforms, the team first runs each platform twice to check whether it can reproduce its own results. Embodied RL has several sources of variability.

  • Decoding temperature, sampling seeds, and dropout can change the action sequence.
  • Asynchronous rollout arrivals can change how the actor assembles microbatches.
  • Simulator reset seeds, initial task states, and rendering parameters affect environment behavior.
  • Operator-level nondeterminism can affect floating-point reductions, including those using atomicAdd.

The static checks compare initial weights, dataset and seed hashes, and the YAML configuration. The dynamic checks compare unordered rollout data as multisets. For each step, all 14 lang_tokens entries matched as a multiset across platforms, with differences in order. All 42 deterministic records matched in order. Cross-platform checks therefore join on stable sample IDs or compare distributions. Pairing microbatches by index can mistake arrival order for a numerical difference.

Alignment mode fixes the global seed, sets the sampling temperature to zero, verifies weight and data SHA-256 hashes on both platforms, pins BLAS and OpenMP to one thread, and disables nondeterministic operator paths. Production training restores exploratory sampling and asynchronous arrival. Its comparisons focus on training dynamics such as loss, KL divergence, and reward curves.

One shape audit initially failed on all 42 microbatch records.

  • Biren produced main_images with shape [1, 240, 320, 3], while the reference produced [1, 64, 64, 3].
  • The launcher set ROBOTWIN_CAMERA_WIDTH and ROBOTWIN_CAMERA_HEIGHT. The synthetic image code read ROBOTWIN_SYNTHETIC_IMAGE_WIDTH and ROBOTWIN_SYNTHETIC_IMAGE_HEIGHT.
  • Every differing field came from the environment worker’s observation output. Actor and rollout fields matched.
  • The smoke test had missed the configuration drift. After the fix, all 42 records passed the shape audit.

3.3

When LingBot-VLA-V2 GRPO starts from supervised fine-tuning (SFT) weights, a run can silently fail to converge. If rewards remain zero, all advantages are zero and both policy loss and grad_norm stay at zero. There is no exception. Loss stays flat and the success rate does not improve.

The diagnostic follows the data path backward through task reward, advantage, actor loss, gradients, and optimizer. Each group has records from both platforms, so the investigation can stop at the first issue.

  • In one click_bell run, the model’s gripper channel stayed between 0.75 and 0.86. The success predicate required an opening below 0.2, so the reward and gradients stayed at zero.
  • In offline open-loop replay, the team fed expert observations directly into the model. The RLinf and official stacks both produced a prediction MAE of 0.396. This narrowed the investigation to the checkpoint or normalization parameters shared by both stacks.

BRTT can alert on silent zero gradients using thresholds on loss traces. Open-loop replay makes it possible to compare forward passes without amplifying small differences through closed-loop simulation. Evidence from each comparison group helps attribute an error to the hardware, framework, or model weights.

3.4

Biren validated the staged approach of dumping tensors, comparing them layer by layer, and locating the first divergence on RLinf with LingBot-VLA-V2. The team plans to standardize dump contracts, thresholds, and diagnostic procedures as reusable skills and extend them to frameworks such as verl. It also plans to run cross-platform numerical checks during release regression testing so it can assess driver, operator library, and compiler changes before merging them.

Biren plans to open-source BRTT and work with the RLinf community on cross-platform numerical alignment and regression testing. The intended extension points are a registry of instrumentation points for each new framework and a dump adapter for each new hardware platform.

04

4.1

Change Submission Status at publication
Platform-independent system optimizations Batches of changes Planned for upstream contribution

Biren developed inference, training, and simulation optimizations for online LingBot-VLA-V2 GRPO and validated their migration to RLinf v0.4 with RoboTwin 2.0’s click_bell task, GPU rendering, and multiple GPUs on one node. Rollout accounted for about 62% of baseline step time, actor training 35%, and weight synchronization 3%. VLA prediction alone took about 30% of step time, roughly half of rollout.

The team captured the whole denoising loop in a graph, combined pinned mirrors with address-preserving offload so the graph could survive offload, and pipelined groups across stages so inference and simulation could alternate. On the training side, it simplified FSDP wrapping. On the environment side, it preserved caches, coalesced intermediate rendering updates, and reused passive force compensation within physics steps. Weight synchronization moved into the initialization window. These changes reduced repeated scheduling and transfers and overlapped work that previously ran in sequence.

With the same existing upstream options, placement, hyperparameters, and training workload, the authors measured 89% higher throughput for V2 (1.89×, with step time down 47%) from the additional features below.

Optimization Implementation Effect
CUDA Graph for the entire denoising loop Capture the complete loop and remove in-loop host synchronization. Generate random noise outside the graph using the original logic. Reduces scheduling and host synchronization in the denoising loop.
Pinned rollout mirror for offload Keep a page-locked mirror in host memory. Free GPU memory on offload, reload through pinned host-to-device transfers, and refresh the mirror during weight synchronization. Reduces repeated GPU memory allocation and synchronous transfers.
Address-preserving VMM offload Keep parameter virtual addresses stable and release physical pages during offload. Restore the backing memory without invalidating graph references. Allows CUDA Graph and offload to coexist without recapturing the graph.
Weight synchronization during bootstrap Synchronize after actor updates while environment bootstrap runs. Consumers wait for synchronization before use, with offload deferred. Overlaps synchronization with bootstrap while completing all synchronization work.
GRPO group pipelining across stages Assign members of each group to different stages with a shared initial seed. Reassemble the full group before computing advantages. Interleaves inference and simulation to reduce serial waits.
FSDP and rendering cache simplification Flatten nested units, bypass frozen modules, use a single root wrapper, and reduce empty_cache calls. Preserve rendering caches and remove unused cameras. Reduces FSDP scheduling and cache rebuilding.
Coalesced rendering updates within action chunks Preserve all physics substeps and reward checks. Refresh before taking images. Use the original path for randomized lighting, display, and recording. Reduces intermediate updates. State, rewards, and images match bit-for-bit in fixed-action replay.
Reuse of passive force compensation within a physics step Reuse compensation while the same entity’s state is unchanged and invalidate it after every physics step. Keep control updates and success checks intact. Reduces redundant compensation calculations. Fixed-action click_bell replay matches bit-for-bit. Other tasks require separate review.

The same optimizations applied to LingBot-VLA-V1 GRPO. With matching baseline options, placement, hyperparameters, and workload, the two V1 runs used identical model code. The authors measured 83% higher throughput (1.83×, with step time down 45%), using the same methodology as V2. The gains came from the added framework and environment features.

The V2 baseline was RLinf v0.4 plus V2 integration. Both runs enabled the existing upstream bootstrap and actor overlap and disabled actor torch.compile and environment offload. The 1.89× figure measures the end-to-end gain from denoising graph and offload coordination, overlapping weight synchronization, regrouping across stages, and the FSDP, cache, rendering, and compensation changes. Both runs kept the same trajectory count, group size, global and microbatch sizes, number of updates, control-step cap, and denoising steps. They kept all zero-advantage groups and training updates. The optimized run removed only unused static cameras and kept the head and both wrist cameras. The CUDA/VMM implementation is separate from the general scheduling mechanism.

The measurements use steady-state windows that cover reset cleanup cycles and include cleanup spikes in the mean. The algorithm review found no approximations or skipped training steps. The graphed and eager denoising paths agreed bit-for-bit on identical inputs. The authors limit the bitwise claim to that check. In a separate earlier numerical experiment, both versions achieved 100% success on the same evaluation scenarios after identical short training runs.

05

  • Platform interfaces and patch boundaries affect maintainability. Generalizing the vendor-image PyTorch setup and consolidating device differences in the Accelerator Manager kept the SUPA PR small. Mirroring module paths in the patch repository made model migration independently reviewable, testable, and easy to toggle.
  • Numerical alignment needs evidence at each stage. The checks cover environment interactions, initial weights, per-step tensor schemas, layerwise forward and backward computation, and three types of worker unit tests. Worker attribution helps identify configuration differences that a file comparison can miss. Loss-trace alerts, backward tracing along the data path, and open-loop replay help diagnose silent failures. In this migration, environment or data evidence accounted for nearly all cases of zero rewards, misaligned loss_mask, or zero gradients and broken backward paths.
  • Performance claims depend on measurement conditions. Steady-state versus cold-start timing, control runs for attention modes, trainable scope, and observation resolution each need separate ablations. Report positive and negative results. Once simulation moved to the GPU, bottlenecks shifted to inference launches, FSDP scheduling, and the weight handoff between training and inference. Eliminating overhead and overlapping stages offered more room for gains than individual kernel tuning in this workload. The team plans to upstream these system optimizations to RLinf.

The team plans to expand the trainable parameter scope and recheck memory use and convergence, run longer training with periodic weight synchronization, compare task success convergence using real camera observations and the official denoising step count, and gradually replace equivalent fallbacks with native Biren kernels.

Biren is also developing reusable numerical alignment skills for layerwise and per-step collection, cross-platform tensor audits, numerical fingerprint comparison, and drift detection between training and inference. The intended workflow defines the comparison criteria, captures and compares tensors, locates the first regression, and helps verify the fix. Biren and the RLinf community plan to continue this work on embodied RL training and reproducible validation for domestic GPU platforms.