Conversation
|
| // identifier among all configs of this type. | ||
| template <typename Config> | ||
| std::optional<TVMFFIKernel> get_or_load(const Config &cfg) { | ||
| std::optional<tvm::ffi::Function> get_or_load(const Config &cfg) { |
There was a problem hiding this comment.
Older CuTeDSL versions can deadlock
If the CuTeDSL backend is enabled with nvidia-cutlass-dsl older than 4.8.0, concurrent first MXFP8 launches on multiple GPUs now call the TVM-FFI function without the protection that prevented a deadlock. Transformer Engine does not require 4.8.0 or reject older installations, so those users can encounter the deadlock again. Require the fixed version before removing the guard, or keep the guard for older versions.
There was a problem hiding this comment.
When NVTE_WITH_CUTEDSL becomes on by default we will require nvidia-cutlass-dsl >= 4.8.0. Currently it's just in but not enabled by default (and as you can see the current codebase doesn't require nvidia-cutlass-dsl at all
There was a problem hiding this comment.
That addresses the default-installation path: users who do not explicitly enable NVTE_WITH_CUTEDSL will not hit this code, and the dependency can be raised when the backend becomes default.
The remaining risk is limited to users who opt in today while still having nvidia-cutlass-dsl < 4.8.0; for those users, removing the guard can reintroduce the documented first-launch deadlock. If supporting that opt-in combination is intentional, the guard should remain until the dependency requirement is enforced. Otherwise, this is reasonable as a forward-looking migration assumption.
There was a problem hiding this comment.
That’s fair. Since NVTE_WITH_CUTEDSL is currently opt-in and Transformer Engine does not declare or enforce a specific nvidia-cutlass-dsl dependency yet, the deadlock concern only affects users who manually enable CuTeDSL with an older DSL version. Once the backend is enabled by default, enforcing nvidia-cutlass-dsl >= 4.8.0 will be necessary; for the current opt-in state, this change is reasonable and I’m resolving the comment.
Signed-off-by: Kaining Zhong <kainingz@nvidia.com>
ce6dc2c to
a2f6f08
Compare
Description
nvidia-cutlass-dsl4.8.0 has fixed the deadlock bug NVIDIA/cutlass#3706 so we don't need this temporary fix nowWhen we enable CuTeDSL in TE by default we'll need to require nvidia-cutlass-dsl >= 4.8.0. Currently we don't require nvidia-cutlass-dsl at all
Fixes # (issue)
Type of change
Changes
Please list the changes introduced in this PR:
Checklist: