Skip to content

feat(training): upgrade Isaac Lab runtime to 3.0 #1689

Description

Component

Training Workflows

Problem Statement

RL training and SIL evaluation run on the digest-pinned nvcr.io/nvidia/isaac-lab:2.3.2 image. That image runs Python 3.11, so training/rl pins Python 3.11 and, for Isaac Sim ABI compatibility, NumPy 1.26.4. As a result:

Proposed Solution

Move the shared Isaac Lab runtime to the digest-pinned nvcr.io/nvidia/isaac-lab:3.0.0-beta2-post1 image:

  • Point DEFAULT_ISAAC_LAB_IMAGE, the Azure ML environment pins, and the direct OSMO workflow fallbacks at the new image.
  • Move training/rl to Python 3.12 and NumPy 2.5.1, regenerate training/rl/uv.lock, and update the Dependabot NumPy guard and the setup-dev Isaac Lab commit pin.
  • Keep the image-provided Torch and CUDA packages when installing locked RL dependencies. Install into a writable overlay when the interpreter's site-packages are read-only, and report runtime provenance from that overlay.

Alternatives Considered

  • Stay on Isaac Lab 2.3.2 until a stable 3.0 release ships. This keeps training/rl on Python 3.11 and NumPy 1.x and leaves its NumPy updates blocked.
  • Offer 3.0 only through a DEFAULT_ISAAC_LAB_IMAGE override. The RL lock resolves for a single Python version and NumPy ABI, so the image default and training/rl/uv.lock must move together.

Additional Context

Consideration Impact
Breaking change Isaac Lab-based RL training and SIL evaluation jobs default to Isaac Lab 3.0 with Python 3.12, and training/rl moves to NumPy 2.5.1. Custom tasks or extensions built for Isaac Lab 2.3.2 may need updates.
Validation The target is a beta tag and hosted CI cannot run Isaac Sim, so the change needs an RL training smoke run on a GPU node.
Azure services None added
Network modes Unaffected
Personas ML engineers and robotics developers running RL training or SIL evaluation

Related: #114 tracks a container upgrade that removes the simulation_shutdown.py workaround. This issue keeps that workaround in place.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

enhancementNew feature or improvement requestneeds-triageNeeds initial review and categorization

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions