Repository navigation
feat(cloud): Kaggle GPU training path and a guard against Modal GPU on production - #82
Merged
Merged
Conversation
…n production Production Modal (runner/compiler/recorder for app.openadapt.ai) was disabled on 2026-10-16 when GPU training exhausted the workspace spend limit. - kaggle_cloud: stage dataset + GPU script kernel (kernel-metadata.json with enable_gpu/internet, private), push via kaggle datasets/kernels, status, pull results; checkpoints re-upload as dataset checkpoint/ for resume. - kaggle_train_kernel: refuses to run without GPU and internet (exit 3). - modal_gpu_guard: Modal GPU train/serve refuse unless OPENADAPT_ALLOW_MODAL_GPU=1, a non-prod MODAL_ENVIRONMENT, and a non-production Modal profile. - docs: Kaggle usage and setting a Modal workspace budget/spend limit. Signed-off-by: Richard Abrich <richard.abrich@mldsai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On 2026-10-16, GPU training exhausted the spend limit on the production Modal workspace (
abrichr), and Modal then disabled the hosted runner, compiler and recorder behind app.openadapt.ai. The decision: Modal keeps the hosted services only, and GPU training moves to Kaggle.openadapt_ml.cloud.kaggle_cloud:stagebuilds a private dataset (bundle, config, optionalcheckpoint/) and a script kernel whosekernel-metadata.jsonturns on GPU and internet, on T4 or P100.pushrunskaggle datasets create/versionand thenkaggle kernels push.statusandpullusekaggle kernels output. To resume, stage again with the pulled results passed as--checkpoint.kaggle_train_kernel.py: exits 3 when the session has no GPU or no internet, so a run that falls back to CPU can't count as a GPU run. (This happened with earlier Kaggle kernels on an unverified account.)modal_gpu_guard:modal_cloudtrain and serve, and both app builders, refuse unlessOPENADAPT_ALLOW_MODAL_GPU=1,MODAL_ENVIRONMENTis not main/prod/production, and the active Modal profile is not a production workspace (OPENADAPT_MODAL_PROD_WORKSPACES, defaultabrichr).Tests: ruff check and format pass;
pytest tests/gives 534 passed, 2 skipped. The new guard and staging tests need no network. Nothing was pushed to Kaggle or Modal.Opened by an agent session for Richard Abrich.