feat(volume): real modal.CloudBucketMount -> S3 mount support (calque#91 Workstream A) - #185
Merged
Merged
Conversation
…#91 Workstream A) Replaces CloudBucketMount's leak-only treatment with a real mountpoint-s3 mount against the script's own S3 bucket: pyast.py extracts bucket_name/ key_prefix/read_only from an inline CloudBucketMount(...) volumes= value, parse.go decodes them into ir.Function/Class.CloudBucketMounts, internal/plan/cloudbucketmount.go resolves+renders the mount-s3 shell lines, bootstrap.go splices them in before @Enter runs, and RealRunPolicy's new extraBuckets param grants the instance role access to the script's own bucket (separate from calque's --bucket staging area). secret=/bucket_endpoint_url=/requester_pays=/force_path_style= are each leaked distinctly as unhonored. Dict/Queue/NetworkFileSystem/ App.include remain leak-only; NetworkFileSystem is a separate, later workstream.
scttfrdmn
added a commit
that referenced
this pull request
Aug 15, 2026
…91 Workstream B) (#186) A real NetworkFileSystem.from_name(name) used as a network_file_systems= value (a separate decorator kwarg from volumes=, never nested inside it) now resolves to a real EFS-over-NFS mount, bring-your-own only — calque never auto-creates an EFS filesystem (create_if_missing=True is a distinct leak, not a blocker), discovering the pre-provisioned filesystem via a calque:nfs-name=<name> tag convention. pyast.py structurally tracks the from_name(...) binding (mirroring Volume's own zero-leak-on-binding posture) and the network_file_systems= decorator kwarg; parse.go threads it through ir.Function/Class.NetworkFileSystems; internal/plan/efs.go resolves the real EFS filesystem/mount-target coverage/security group; internal/exec/bootstrap.go splices the mount lines; cmd/calque/realrun.go and fleetrun.go narrow the AZ sweep to AZs with live mount-target coverage (a hard error, not a leak, if that narrows to zero) and attach a self-referential NFS/2049-ingress security group via spawnaws.LaunchConfig.SecurityGroupIDs. IAM (ClientMount/ClientWrite) is explicitly out of scope for this pass. This closes out both real-mapping workstreams calque#91 was tracking (Workstream A, CloudBucketMount -> S3, shipped in #185/v0.5.1). modal.Dict/Queue/App.include remain deliberately leak-only, out of scope.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements calque#91 Workstream A: replaces
modal.CloudBucketMount'sleak-only treatment with a real S3 mount via
mountpoint-s3, when it'sused inline as a
volumes=value (the real Modal idiom):@app.function(volumes={"/data": modal.CloudBucketMount("my-bucket", key_prefix="foo/", read_only=True)})tools/pyast/pyast.py:_volumes_map/_cloud_bucket_mountextractbucket_name(positional or kwarg),key_prefix,read_onlyfrom aninline
CloudBucketMount(...)call into a{"__cloud_bucket_mount__": {...}}wire shape, distinguishable froman ordinary Volume mount's plain-string shape. Falls back to the
pre-existing "recognized but not modeled" leak only when
bucket_nameisn't a string literal.
secret=andbucket_endpoint_url=/requester_pays=/force_path_style=are eachleaked separately as unhonored.
internal/ir/ir.go: newir.CloudBucketMounttype;Function/Classget a
CloudBucketMounts map[string]CloudBucketMountfield alongsidethe existing
Volumesfield.internal/parse/parse.go:readConfigKwargswidened to decode boththe plain-Volume and CloudBucketMount shapes out of one
volumes=dict, threaded through
buildFn/buildClass.internal/plan/cloudbucketmount.go(new):ResolveCloudBucketMounts(dedupe-by-mount-path, conflict-leak, mirrors
ResolveVolumes) andMountCommands(idempotentmount-s3install check,mkdir -p, thenmount-s3 <bucket> <mountpoint> [--prefix <prefix>] [--read-only]).internal/exec/bootstrap.go: newBootstrapConfig.CloudBucketMountLinesfield, spliced in after the artifact sync and before either the
HostMode or docker-mode run invocation.
internal/plan/iam.go:RealRunPolicygains anextraBuckets []stringparam — each distinct bucket gets its own read/write/list grant,
separate from calque's own
--bucketstaging area.cmd/calque/realrun.go/cmd/calque/fleetrun.go: newcloudBucketMountSpecsForAppwired into both the single-instance andfleet (dedicated-fallback-instance) paths.
docs/modal-compatibility-matrix.md:modal.CloudBucketMountrowupdated from ⬜ to ✅.
Dict/Queue/NetworkFileSystem/
App.includeremain leak-only/out ofscope for this PR.
modal.NetworkFileSystemis a separate, largerworkstream planned for later — not attempted here.
Test plan
go build ./... && go vet ./... && gofmt -l . && go test -count=1 ./...golangci-lint run ./...ruff check .testdata/scripts/cloud_bucket_mount.py(positive/modeled case)internal/parse/parse_test.go:TestParseCloudBucketMountResolves(new);TestParseRareConstructsAreTaggedNotSilentupdated (CloudBucketMount no longer unconditionally leaked)internal/plan/cloudbucketmount_test.go(new): dedupe/conflict-leak + shell-line outputinternal/exec/bootstrap_demo_test.go:CloudBucketMountLinessplice tests (docker mode + host mode + unchanged-default)internal/plan/iam_test.go:extraBucketscoverage