Quick Start
===========

This quick-start walks you through training the Visual-Language-Action model, including
`OpenVLA <https://github.com/openvla/openvla>`_ and `OpenVLA-OFT <https://github.com/moojink/openvla-oft>`_ on the
`ManiSkill3 <https://github.com/haosulab/ManiSkill>`_ environment with **RLinf**.

Environment Introduction
--------------------------

ManiSkill3 is a GPU-accelerated simulation platform for robotics research,  
focusing on complex contact manipulation and embodied intelligence tasks.  
The benchmark covers multiple domains, including robotic arms, mobile manipulators, humanoid robots, and dexterous hands,  
supporting various tasks such as grasping, assembling, drawing, and locomotion.

We have also implemented system-level optimizations for the GPU simulator (see :doc:`../concepts/execution_modes`).

Launch Training
--------------------------

**Step 1: Download pre-trained models**

If using the **OpenVLA** model, run the following command:

.. code-block:: bash

   # Download OpenVLA pre-trained model
   hf download gen-robot/openvla-7b-rlvla-warmup \
   --local-dir /path/to/model/openvla-7b-rlvla-warmup/

This model is cited in the paper: `paper <https://arxiv.org/abs/2505.19789>`_

If using the **OpenVLA-OFT** model, run the following command:

.. code-block:: bash

   # Download OpenVLA-OFT pre-trained model
   hf download RLinf/Openvla-oft-SFT-libero10-trajall \
   --local-dir /path/to/model/Openvla-oft-SFT-libero10-trajall/
   
   # Download LoRA fine-tuned checkpoint on maniskill
   hf download RLinf/RLinf-OpenVLAOFT-ManiSkill-Base-Lora \
   --local-dir /path/to/model/oft-sft/lora_004000

**Step 2: Download ManiSkill assets**

This step is required for both **OpenVLA** and **OpenVLA-OFT** on ManiSkill3.

.. code-block:: bash

   cd <path_to_RLinf>/rlinf/envs/maniskill
   # For mainland China users, you can use the following for better download speed:
   # export HF_ENDPOINT=https://hf-mirror.com
   hf download --repo-type dataset RLinf/maniskill_assets --local-dir ./assets


**Step 3: Modify the configuration file**

Before running the script, you need to modify the ``./examples/embodiment/config/maniskill_ppo_openvla_quickstart.yaml`` file according to the download paths of the model and dataset. Specifically, update the following configurations to the path where the `gen-robot/openvla-7b-rlvla-warmup` checkpoint is located.

- ``rollout.model.model_path``  
- ``actor.model.model_path``  



For **OpenVLA-OFT**, modify the ``maniskill_ppo_openvlaoft_quickstart.yaml`` file. Set the following model configuration items to the path where the `RLinf/Openvla-oft-SFT-libero10-trajall` checkpoint is located. At the same time, set the LoRA path to the path where the `RLinf/RLinf-OpenVLAOFT-ManiSkill-Base-Lora` checkpoint is located.

- ``rollout.model.model_path``  
- ``actor.model.model_path``  
- ``actor.model.lora_path``
- ``actor.model.is_lora: True``

**Step 4: Launch training**

After completing the above configuration file modifications, run the following command to launch training:

.. code-block:: bash

   bash examples/embodiment/run_embodiment.sh maniskill_ppo_openvla_quickstart

.. note::
   If you installed **RLinf** via Docker image (see :doc:`./installation`), please ensure you have switched to the Python environment corresponding to the target model.
   The default environment is ``openvla``.
   If using OpenVLA-OFT or openpi, please use the built-in script `switch_env` to switch environments:
   ``source switch_env openvla-oft`` or ``source switch_env openpi``.

   If you installed **RLinf** in a custom environment, please ensure you have installed the dependencies for the corresponding model, see :doc:`./installation` for details.

For **OpenVLA-OFT**:

.. code-block:: bash

   source switch_env openvla-oft
   bash examples/embodiment/run_embodiment.sh maniskill_ppo_openvlaoft_quickstart

Training Pipeline Mode
--------------------------

For embodied FSDP training, ``runner.use_training_pipeline`` enables a pipeline
execution path between environment rollout and actor training. When it is set to
``True``, rollout trajectories are processed on the environment worker, converted
into packed actor micro-batches, and streamed to the actor through the channel.
The actor can then train on ready-to-use micro-batches while rollout generation is
still progressing.

This mode is useful when rollout payloads contain nested observations or large
tensors. Sending packed micro-batches makes the channel payload more friendly to
the tensor fast path and reduces extra reconstruction work on the actor side. It
is especially helpful when environment workers and actor workers are placed on
different nodes and the inter-node connection crosses a wide-area network, where
smaller packed tensor payloads reduce transfer overhead.

Example:

.. code-block:: yaml

   runner:
     use_training_pipeline: True

   algorithm:
     adv_type: gae
     normalize_advantages: True

Notes and limitations:

- ``algorithm.normalize_advantages`` is supported. The pipeline path computes raw
  advantages on the environment worker, aggregates masked advantage statistics
  across the environment workers that feed each actor rank, and normalizes before
  streaming actor micro-batches.
- ``algorithm.adv_type`` currently supports only ``gae`` in this mode.
- This mode is intended for embodied FSDP actor training with PPO/GRPO-style
  actor losses. It is not supported for ``embodied_sac``, ``embodied_dagger``,
  or ``embodied_nft``.


View Training Results
--------------------------

- Final model and metrics save path: ``./logs``  
- Launch as follows:

  .. code-block:: bash

     tensorboard --host 0.0.0.0 --logdir path/to/tensorboard/

After opening TensorBoard, you will see an interface similar to the figure below.  
It is recommended to focus on the following metrics:

- ``rollout/env_info/return``  
- ``rollout/env_info/success_once``  

.. raw:: html

   <img src="https://github.com/RLinf/misc/raw/main/pic/embody-quickstart-metric.jpg" width="800"/>

.. note::
   If you want to specify GPU usage,
   you can modify the parameter  
   ``cluster.component_placement`` in the configuration file.

   Set this item to **0-3** or **0-7** to use 4/8 GPUs according to your actual resources.
   See :doc:`../guides/basic_config` for more detailed instructions on Placement configuration.

   .. code-block:: yaml

      cluster:
      num_nodes: 1
      component_placement:
         actor,env,rollout: 0-3
