RL with OpenSora World Model
======================================

.. |huggingface| image:: /_static/svg/hf-logo.svg
   :width: 16px
   :height: 16px
   :class: inline-icon

.. figure:: https://raw.githubusercontent.com/hpcaitech/Open-Sora-Demo/main/readme/icon.png
   :align: center
   :width: 30%

   OpenSora as an action-conditioned world model.

Train a VLA policy **closed-loop without real robots or a physics simulator** by
using the **action-conditioned OpenSora world model** as the environment backend.
OpenSora generates future video frames from the current observation and an action
sequence, so the policy can be optimized on "imagined" rollouts with RL (GRPO/PPO).

Overview
--------

Train OpenVLA-OFT with GRPO on LIBERO suites simulated by the OpenSora world model.

.. grid:: 2 4 4 4
   :gutter: 2

   .. grid-item-card:: Environments
      :text-align: center

      LIBERO

   .. grid-item-card:: Algorithms
      :text-align: center

      GRPO

   .. grid-item-card:: Tasks
      :text-align: center

      Spatial · Object

   .. grid-item-card:: Hardware
      :text-align: center

      1 node · GPUs

| **You'll do:** install → download the VLA model → download the OpenSora world-model weights + init data → launch ``run_embodiment.sh`` → watch ``env/success_once``.
| **Prerequisites:** :doc:`Installation </rst_source/start/installation>` · an OpenVLA-OFT SFT checkpoint · OpenSora world-model weights and init dataset (steps below).

Tasks
~~~~~

As a world model, OpenSora can in principle fit any task behind a consistent interface. RLinf currently ships weights and init data for two LIBERO suites:

.. list-table::
   :header-rows: 1
   :widths: 22 24 30 24

   * - Environment
     - Task / Suite
     - Config / Weights
     - Focus
   * - OpenSora
     - LIBERO-Spatial
     - ``RLinf/RLinf-OpenSora-LIBERO-Spatial``
     - Use OpenSora as a learned simulator for LIBERO spatial tasks.
   * - OpenSora
     - LIBERO-Object
     - ``RLinf/RLinf-OpenSora-LIBERO-Object``
     - Roll out object manipulation dynamics in the video world model.

Observation and Action
~~~~~~~~~~~~~~~~~~~~~~

.. list-table::
   :header-rows: 1
   :widths: 24 38

   * - Field
     - Description
   * - Observation
     - RGB frames generated by the world model, ``[B, 256, 256, 3]``, seeded from initialization frames.
   * - Action
     - 7-D continuous actions normalized and tokenized to condition generation.
   * - Reward
     - World-model reward classifier output in ``[0, 1]``.
   * - Prompt
     - Natural-language task description used to condition the video world model.

Unlike a traditional simulator, OpenSora has no ``reset()``: it requires initialization frames and a task description, so you download an initialization dataset and point the config at it.

Installation
------------

.. include:: _setup_common.rst

**Option 1: Docker image** — image tag ``agentic-rlinf0.3-opensora``:

.. code:: bash

   docker run -it --rm --gpus all \
      --shm-size 20g \
      --network host \
      --name rlinf \
      -v .:/workspace/RLinf \
      rlinf/rlinf:agentic-rlinf0.3-opensora
      # Mainland China mirror: docker.1ms.run/rlinf/rlinf:agentic-rlinf0.3-opensora

   # Inside the container, switch to the OpenVLA-OFT virtual environment:
   source switch_env openvla-oft

**Option 2: Custom environment** — install bundle ``--env opensora``:

.. code:: bash

   # Add --use-mirror for faster downloads in mainland China.
   bash requirements/install.sh embodied --model openvla-oft --env opensora
   source .venv/bin/activate

Download the VLA Model
----------------------

Download the OpenVLA-OFT SFT checkpoints:

.. code:: bash

   # Method 1: git clone
   git lfs install
   git clone https://huggingface.co/Haozhan72/Openvla-oft-SFT-libero-spatial-traj1
   git clone https://huggingface.co/Haozhan72/Openvla-oft-SFT-libero-object-traj1
   git clone https://huggingface.co/Haozhan72/Openvla-oft-SFT-libero-goal-traj1
   git clone https://huggingface.co/Haozhan72/Openvla-oft-SFT-libero10-traj1

   # Method 2: huggingface-hub (set HF_ENDPOINT=https://hf-mirror.com in mainland China)
   pip install huggingface-hub
   hf download Haozhan72/Openvla-oft-SFT-libero-spatial-traj1 --local-dir Openvla-oft-SFT-libero-spatial-traj1
   hf download Haozhan72/Openvla-oft-SFT-libero-object-traj1 --local-dir Openvla-oft-SFT-libero-object-traj1
   hf download Haozhan72/Openvla-oft-SFT-libero-goal-traj1 --local-dir Openvla-oft-SFT-libero-goal-traj1
   hf download Haozhan72/Openvla-oft-SFT-libero10-traj1 --local-dir Openvla-oft-SFT-libero10-traj1

After downloading, set the model path and ``unnorm_key`` in the config:

.. code:: yaml

   rollout:
      model:
         model_path: Pathto/RLinf/RLinf-OpenVLAOFT-LIBERO-90-Base-Lora
   actor:
      model:
         model_path: Pathto/RLinf/RLinf-OpenVLAOFT-LIBERO-90-Base-Lora
         unnorm_key: libero_90_no_noops_trajall # or libero_130_no_noops_trajall for the RLinf-OpenVLAOFT-LIBERO-130-Base-Lora model

Download the World Model
------------------------

Download the OpenSora weights and the simulation-initialization dataset. RLinf currently
provides weights and data for libero-spatial and libero-object:

.. code:: bash

   # Method 1: git clone
   git lfs install
   git clone https://huggingface.co/RLinf/RLinf-OpenSora-LIBERO-Spatial
   git clone https://huggingface.co/RLinf/RLinf-OpenSora-LIBERO-Object

   # Method 2: huggingface-hub
   pip install huggingface-hub
   hf download RLinf/RLinf-OpenSora-LIBERO-Spatial --local-dir RLinf-OpenSora-LIBERO-Spatial
   hf download RLinf/RLinf-OpenSora-LIBERO-Object --local-dir RLinf-OpenSora-LIBERO-Object

The directory structure of ``RLinf-OpenSora-LIBERO-Spatial`` is:

.. code-block:: text

    RLinf-OpenSora-LIBERO-Spatial/
        ├── dataset_statistics.json             # Dataset normalization statistics
        ├── dataset/                            # Simulation initialization dataset
        │   ├── traj0.npy
        │   ├── traj1.npy
        │   ├── ...
        │   └── trajN.npy
        ├── model-00001.safetensors              # World model weight files
        ├── model.safetensors.index.json
        ├── config.json
        ├── resnet_rm.pth                        # Reward model weight file
        └── vae/                                 # VAE model weight files

After downloading, set the world-model path in the config:

.. code:: yaml

    env:
        train:
            opensora_wm_hf_ckpt_path: /Pathto/model/RLinf-OpenSora-LIBERO-Spatial/

Run It
------

**1. Model parameters**

Configure ``actor.model`` (OpenVLA-OFT example):

.. code-block:: yaml

   actor:
     model:
       model_path: "/path/to/model/Openvla-oft-SFT-libero-spatial-traj1/"    # SFT model path
       model_type: "openvla_oft"                                             # Model type set to openvla_oft
       use_proprio: False                                                    # Whether to use proprioceptive inputs
       num_images_in_input: 1                                                # Number of input images
       num_action_chunks: 8                                                  # Number of action chunks
       unnorm_key: "libero_spatial_no_noops"                                 # Action normalization key (match SFT)

Because the world model does not provide proprioception, does not generate a wrist view,
and uses a fixed chunk length, ``use_proprio`` defaults to ``False``, ``num_images_in_input``
to ``1``, and ``num_action_chunks`` to ``8``.

**2. Environment configuration**

.. code-block:: yaml

   # Recommend opensora_libero_spatial for training and libero_spatial for evaluation
   env/train: opensora_libero_spatial
   env/eval: libero_spatial
   env:
      train:
         opensora_wm_hf_ckpt_path: /Pathto/model/RLinf-OpenSora-LIBERO-Spatial/

   # In env/train/opensora_libero_spatial.yaml:
   env_type: opensora_wm
   wm_env_type: libero
   # Initial image path for world model initialization
   initial_image_path: ${env.train.opensora_wm_hf_ckpt_path}/dataset_for_rlinf_world_model_init/base_policy_rollout_buffer
   # It is not recommended to modify any parameters in world_model_cfg
   world_model_cfg:
      stats_path: /Pathto/model/RLinf-OpenSora-LIBERO-Spatial/best_wm_ckpt/base_policy/dataset_statistics.json
      chunk: 8                     # Align with training and VLA inference length; default 8
      condition_frame_length: 4    # Context memory length; default 4
      model:
         from_pretrained: /Pathto/model/RLinf-OpenSora-LIBERO-Spatial/best_wm_ckpt/base_policy/model

**3. Launch**

OpenVLA-OFT + GRPO uses ``examples/embodiment/config/opensora_libero_spatial_grpo_openvlaoft.yaml``:

.. code-block:: bash

   bash examples/embodiment/run_embodiment.sh opensora_libero_spatial_grpo_openvlaoft

Visualization and Results
-------------------------

Watch **``env/success_once``** for the unnormalized episodic success rate. For every logged
metric, see :doc:`Training metrics <../../reference/metrics>`. Enable
generated-rollout videos with:

.. code-block:: yaml

   env:
      eval:
         video_cfg:
            save_video: True
            video_base_dir: ${runner.logger.log_path}/video/eval

We evaluate every ``task_id`` × ``trial_id`` combination — 500 environments (10 tasks × 50
trials) per suite. SFT (LoRA-base) models use ``do_sample = False``; RL-trained models use
``do_sample = True`` and ``temperature_train = 1.6`` in ``rollout.sampling_params``, with
``env.train.rollout_epoch=2``.

.. note::

    The motivation for choosing OpenSora as a world-model simulator comes from
    `WMPO <https://arxiv.org/abs/2511.09515>`_; we also referred to
    `OpenSora <https://github.com/RLinf/opensora>`_ when training the world model.

.. list-table:: **Evaluation results on LIBERO task groups using the OpenSora simulator**
    :header-rows: 1
    :widths: 50 25 25

    * - Model
      - Object
      - Spatial
    * - |huggingface| `OpenVLA-OFT (LoRA-base) <https://huggingface.co/RLinf/RLinf-OpenVLAOFT-LIBERO-130-Base-Lora>`_
      - 50.20%
      - 51.61%
    * - OpenVLA-OFT (RLinf-GRPO with OpenSora as world model simulator)
      - 75.5%
      - 64.5%
    * - **Improvement**
      - **+25.3%**
      - **+12.9%**
