Skip to content

[Common] Remove CuTeDSL first launch protection - #3620

Open
kainzhong wants to merge 1 commit into
NVIDIA:mainfrom
kainzhong:remove_cutedsl_first_launch_protection
Open

kainzhong wants to merge 1 commit into
NVIDIA:mainfrom
kainzhong:remove_cutedsl_first_launch_protection

Conversation

@kainzhong

@kainzhong kainzhong commented Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator

Description

nvidia-cutlass-dsl 4.8.0 has fixed the deadlock bug NVIDIA/cutlass#3706 so we don't need this temporary fix now

When we enable CuTeDSL in TE by default we'll need to require nvidia-cutlass-dsl >= 4.8.0. Currently we don't require nvidia-cutlass-dsl at all

Fixes # (issue)

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

Please list the changes introduced in this PR:

  • Remove the first time CuTeDSL function launch protection

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@greptile-apps

greptile-apps Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[High risk] Removes serialization wrapper around GPU kernel first launch.

The PR appears safe to merge based on this review.

Findings

  1. P1 Older CuTeDSL versions can deadlock ▶

Summary

The PR removes CuTeDSL first-launch serialization and stores TVM-FFI function handles directly in the MXFP8 kernel cache.

  • No changes were made since the previous review.
  • The previous deadlock concern was conceded in its thread given that CuTeDSL is currently opt-in.

Reviews (2) · Last reviewed commit: "remove first launch protection"

// identifier among all configs of this type.
template <typename Config>
std::optional<TVMFFIKernel> get_or_load(const Config &cfg) {
std::optional<tvm::ffi::Function> get_or_load(const Config &cfg) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Older CuTeDSL versions can deadlock

If the CuTeDSL backend is enabled with nvidia-cutlass-dsl older than 4.8.0, concurrent first MXFP8 launches on multiple GPUs now call the TVM-FFI function without the protection that prevented a deadlock. Transformer Engine does not require 4.8.0 or reject older installations, so those users can encounter the deadlock again. Require the fixed version before removing the guard, or keep the guard for older versions.

@kainzhong kainzhong Oct 2, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When NVTE_WITH_CUTEDSL becomes on by default we will require nvidia-cutlass-dsl >= 4.8.0. Currently it's just in but not enabled by default (and as you can see the current codebase doesn't require nvidia-cutlass-dsl at all

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That addresses the default-installation path: users who do not explicitly enable NVTE_WITH_CUTEDSL will not hit this code, and the dependency can be raised when the backend becomes default.

The remaining risk is limited to users who opt in today while still having nvidia-cutlass-dsl < 4.8.0; for those users, removing the guard can reintroduce the documented first-launch deadlock. If supporting that opt-in combination is intentional, the guard should remain until the dependency requirement is enforced. Otherwise, this is reasonable as a forward-looking migration assumption.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That’s fair. Since NVTE_WITH_CUTEDSL is currently opt-in and Transformer Engine does not declare or enforce a specific nvidia-cutlass-dsl dependency yet, the deadlock concern only affects users who manually enable CuTeDSL with an older DSL version. Once the backend is enabled by default, enforcing nvidia-cutlass-dsl >= 4.8.0 will be necessary; for the current opt-in state, this change is reasonable and I’m resolving the comment.

Signed-off-by: Kaining Zhong <kainingz@nvidia.com>
@kainzhong
kainzhong force-pushed the remove_cutedsl_first_launch_protection branch from ce6dc2c to a2f6f08 Compare October 3, 2026 00:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant