Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,5 @@ bootstrap.build/
.remember/
.mock-runtime-api-port
_site/
.clj-kondo/
.lsp/
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ All notable changes to this project are documented here.
- Initial public release preparation.
- x86_64 builds: `LAMBDA_ARCH=x86_64 jolt image` (default stays `arm64`); `jolt deploy` reads the architecture from `dist/bootstrap`'s ELF header and passes it on both create and update. Thanks to [@codegod100](https://github.com/codegod100) for this contribution ([#2](https://github.com/b12n-oss/lambda-mvp-jlt/pull/2)), verified end to end on x86_64 Linux and, separately, against the project's own arm64 default on Apple Silicon.
- `jolt demo`: `image` + `deploy` + `invoke` in one command, with every prerequisite (tools, AWS credentials and region, Docker access, emulation for `LAMBDA_ARCH`) checked before the build starts. Also from [#2](https://github.com/b12n-oss/lambda-mvp-jlt/pull/2).
- LocalStack support: `jolt localstack:demo` runs deploy + invoke against a local LocalStack container — no AWS account, credentials, or cost. One switch, `LAMBDA_ENDPOINT_URL`, redirects every aws CLI call this repo makes to the emulator (dummy credentials included); unset, everything behaves exactly as before. See `docs/guide/localstack.md`.

## [0.1.0]

Expand Down
23 changes: 22 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,11 +73,12 @@ See `docs/guide/runtime-api-loop.md` for the loop's design notes and

```sh
jolt probe # offline e2e: mock Runtime API + demo handler (no AWS, no Docker)
jolt test # run script/bench.clj's unit tests (no AWS, no Docker)
jolt test # run the unit tests (no AWS, no Docker)
jolt demo # image + deploy + invoke in one go, prerequisites checked first
jolt image # AL2023 Docker build -> dist/bootstrap + dist/lambda.zip (arm64 default)
jolt deploy # idempotent: create/update the IAM role + Lambda function
jolt invoke # single ad-hoc invoke, prints the response body + REPORT line
jolt localstack:demo # LocalStack variant: start it, deploy, invoke -- no AWS account
jolt bench # cold/warm boot-time comparison across memory tiers
jolt teardown # delete the function + role when you're done
```
Expand All @@ -91,6 +92,26 @@ binary. `jolt deploy`'s IAM role (`lambda-mvp-jlt-role`) and function
`BENCH_MEMORY_TIERS` (default `2048,3008`) and `BENCH_WARM_SAMPLES`
(default `5`).

## Running locally against LocalStack

Every AWS-touching task here also runs against
[LocalStack](https://localstack.cloud) instead of a real account:

```sh
jolt image # as usual: build dist/lambda.zip first
jolt localstack:demo # start LocalStack + deploy + invoke, no AWS account
jolt localstack:status # is the container up, is the edge answering?
jolt localstack:stop # stop + remove the container
```

The switch is one env var: `LAMBDA_ENDPOINT_URL` (set to
`http://localhost:4566` by the wrapper). Set, it redirects every `aws` call
this repo's scripts make to that endpoint and skips the real-AWS credential
preflight; unset, behavior is exactly as before. See
[docs/guide/localstack.md](docs/guide/localstack.md) for details — including
why you shouldn't trust LocalStack numbers for the cold/warm boot question
`jolt bench` exists to answer.

## Cold vs. warm boot time

See [`docs/guide/cold-warm-boot.md`](docs/guide/cold-warm-boot.md) for what
Expand Down
56 changes: 37 additions & 19 deletions bb.edn
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,9 @@
(println " jolt image AL2023 Docker build -> dist/bootstrap + dist/lambda.zip")
(println " jolt deploy idempotent: create/update the IAM role + Lambda function")
(println " jolt invoke single ad-hoc invoke, prints response + REPORT line")
(println " jolt bench cold/warm boot-time comparison across memory tiers")
(println " jolt test run script/bench.clj's unit tests")
(println " jolt bench cold/warm boot-time comparison across memory tiers")
(println " jolt localstack:demo LocalStack variant: start it, deploy, invoke -- no AWS account")
(println " jolt test run the unit tests (bench + localstack helpers)")
(println " jolt teardown delete the function + role")
(println " jolt clean remove build artifacts")
(println " jolt site:build build the docs site into _site/ (needs a docs-engine checkout)")
Expand All @@ -63,21 +64,25 @@
(println "No task hardcodes an AWS profile, account, or region -- see README.md"))}

probe {:doc "offline e2e: mock Runtime API on an OS-assigned port + the demo handler"
:task (let [port-file ".mock-runtime-api-port"]
(when (.exists (java.io.File. port-file)) (.delete (java.io.File. port-file)))
(let [srv (p/process ["python3" "tools/mock_runtime_api.py"]
{:inherit true})]
(loop [n 0]
(when (and (not (.exists (java.io.File. port-file))) (< n 50))
(Thread/sleep 100)
(recur (inc n))))
(if-not (.exists (java.io.File. port-file))
(do (println "probe: mock server did not publish its port -- aborting")
(System/exit 1))
(let [port (.trim (slurp port-file))]
(p/shell {:extra-env {"AWS_LAMBDA_RUNTIME_API" (str "127.0.0.1:" port)}}
"joltc run -m net.b12n.lambda-mvp.main")
(System/exit (:exit (deref srv)))))))}
:task (do (when-not (fs/which "joltc")
(println "probe: joltc is not on PATH -- install jolt with the compiler")
(println " (brew install jolt-lang/jolt/jolt) and retry.")
(System/exit 1))
(let [port-file ".mock-runtime-api-port"]
(when (.exists (java.io.File. port-file)) (.delete (java.io.File. port-file)))
(let [srv (p/process ["python3" "tools/mock_runtime_api.py"]
{:inherit true})]
(loop [n 0]
(when (and (not (.exists (java.io.File. port-file))) (< n 50))
(Thread/sleep 100)
(recur (inc n))))
(if-not (.exists (java.io.File. port-file))
(do (println "probe: mock server did not publish its port -- aborting")
(System/exit 1))
(let [port (.trim (slurp port-file))]
(p/shell {:extra-env {"AWS_LAMBDA_RUNTIME_API" (str "127.0.0.1:" port)}}
"joltc run -m net.b12n.lambda-mvp.main")
(System/exit (:exit (deref srv))))))))}

image {:doc "build bootstrap + lambda.zip in an amazonlinux:2023 container (LAMBDA_ARCH, default arm64)"
:task (let [jolt-version (System/getenv "JOLT_VERSION")
Expand Down Expand Up @@ -114,8 +119,21 @@
bench {:doc "cold/warm boot-time comparison across memory tiers"
:task (p/shell "bb" "script/bench_run.clj")}

test {:doc "run script/bench.clj's unit tests"
:task (p/shell "bb" "test/bench_test.clj")}
localstack:start {:doc "start a LocalStack container (edge on localhost:4566) for local, no-AWS deploys"
:task (p/shell "bb" "script/localstack_run.clj" "start")}

localstack:demo {:doc "start LocalStack (if needed), deploy the built zip to it, invoke -- no AWS account touched"
:task (p/shell "bb" "script/localstack_run.clj" "demo")}

localstack:stop {:doc "stop + remove the LocalStack container"
:task (p/shell "bb" "script/localstack_run.clj" "stop")}

localstack:status {:doc "is the LocalStack container up, and is its edge answering?"
:task (p/shell "bb" "script/localstack_run.clj" "status")}

test {:doc "run the unit tests (bench + localstack helpers)"
:task (do (p/shell "bb" "test/localstack_test.clj")
(p/shell "bb" "test/bench_test.clj"))}

site:build {:doc "build the docs site into _site/ (needs a docs-engine checkout)"
:task (site-task "build")}
Expand Down
2 changes: 2 additions & 0 deletions docs/guide/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ Every AWS-touching command in this repo relies entirely on the caller's own `aws
| [The Runtime API loop](runtime-api-loop.md) | The Lambda custom-runtime contract and this project's implementation |
| [Building on Amazon Linux 2023](al2023-build.md) | The glibc constraint, the from-source build recipe, arm64 |
| [Cold vs. warm boot](cold-warm-boot.md) | What `jolt bench` measures and how to reproduce the comparison |
| [Running against LocalStack](localstack.md) | Deploy/invoke/teardown with no AWS account, via `LAMBDA_ENDPOINT_URL` |
| [Contributing](contributing.md) | Build, test, and PR conventions |

## Find your scenario
Expand All @@ -34,6 +35,7 @@ Every AWS-touching command in this repo relies entirely on the caller's own `aws
| "I want to try this out" | Getting started |
| "I want to understand how it works" | Architecture, then The Runtime API loop and Building on Amazon Linux 2023 |
| "I want to measure cold/warm boot time on my own account" | Getting started, then Cold vs. warm boot |
| "I want the full lifecycle without an AWS account" | Running against LocalStack |
| "I want to contribute a change" | Contributing |

## See also
Expand Down
92 changes: 92 additions & 0 deletions docs/guide/localstack.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Running against LocalStack

Everything in this repo that touches AWS (`jolt deploy`, `jolt invoke`,
`jolt teardown`, `jolt bench`, `jolt demo`) can run against
[LocalStack](https://localstack.cloud) instead of a real account — useful for
trying the full deploy/invoke lifecycle with zero AWS credentials, zero cost,
and zero risk to a real account. The Runtime API loop, the handler, and the
zip are exactly the same artifacts real AWS gets; only the control plane
(`lambda create-function`, `invoke`, IAM) is emulated.

## Quickstart

```sh
jolt image # unchanged: build dist/lambda.zip first
jolt localstack:demo # start LocalStack + deploy + invoke, no AWS account
jolt localstack:stop # stop + remove the container when done
```

`jolt localstack:demo` starts a detached `localstack/localstack` container
(`lambda-mvp-jlt-localstack`, edge on `http://localhost:4566`, host docker
socket mounted), waits for the edge to answer, then runs this repo's own
`deploy` and `invoke` tasks unchanged. `jolt localstack:status` reports
whether the container is up and the edge is answering.

The Lambda function containers LocalStack runs are created through the host's
docker daemon (that's why the socket is mounted), so the function executes
real `bootstrap` + `lib/` from your `dist/lambda.zip`.

## The switch: `LAMBDA_ENDPOINT_URL`

Nothing about the default behavior changes until you set
`LAMBDA_ENDPOINT_URL`. Unset (or blank) means real AWS, exactly as before.
Set, it does two things to every `aws` CLI call the scripts make:

- `--endpoint-url <value>` is injected as a global option, and
- LocalStack's documented dummy credentials (`test`/`test`, region
`us-east-1`) are exported — unless you've set `AWS_ACCESS_KEY_ID` /
`AWS_SECRET_ACCESS_KEY` / `AWS_DEFAULT_REGION` yourself, which then win.

The real-AWS credential/region preflight in `demo`/`deploy`/`invoke`/`bench`
is skipped when the endpoint is set, so no aws CLI configuration is needed at
all for the LocalStack path.

You can drive it by hand, without the wrapper tasks:

```sh
export LAMBDA_ENDPOINT_URL=http://localhost:4566
export AWS_ACCESS_KEY_ID=test AWS_SECRET_ACCESS_KEY=test AWS_DEFAULT_REGION=us-east-1
aws --endpoint-url $LAMBDA_ENDPOINT_URL lambda list-functions # anything, really
bb script/aws_lifecycle.clj deploy # or jolt deploy -- same code path
bb script/aws_lifecycle.clj invoke
bb script/aws_lifecycle.clj teardown
```

(Or start your own LocalStack: `docker run -d -p 4566:4566
-v /var/run/docker.sock:/var/run/docker.sock localstack/localstack`.)

Note on versions: LocalStack's March 2026 licensing change means
`localstack/localstack:latest` (and anything newer) refuses to start without
`LOCALSTACK_AUTH_TOKEN`, even for community features. This repo therefore
pins `localstack/localstack:4.13.1`, the last tag that runs without an
account. If you have a token (the free tier works), export
`LOCALSTACK_AUTH_TOKEN` and optionally `LOCALSTACK_IMAGE` (e.g.
`localstack/localstack:latest`) and the wrapper tasks will pass both to the
container.

## What LocalStack is good for here, and what it isn't

Good:

- End-to-end confidence in the deploy/invoke/teardown lifecycle (packaging,
`provided.al2023`, handler wiring, response/log plumbing) without an
account.
- Fast iteration on `aws_lifecycle.clj` itself.
- CI smoke tests.

Not good:

- **Cold/warm boot numbers.** `jolt bench` runs against LocalStack if the
endpoint is set, but LocalStack executes your function in a plain docker
container on your machine — no Firecracker microVM, no Lambda memory-tier
CPU credits, and on Apple Silicon possibly under emulation. The numbers are
meaningless for the cold-vs-warm question; use a real account for that
(that's what `jolt bench` exists for).
- IAM fidelity. LocalStack's IAM is permissive; a trust policy that works
here may still be rejected by real AWS.

## Requirements

- Docker (running), for both the LocalStack container and the function
containers it spawns.
- The `aws` CLI on PATH (version 2), as for the real-AWS path.
31 changes: 25 additions & 6 deletions script/aws_lifecycle.clj
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,30 @@
[cheshire.core :as json]
[clojure.string :as str]))

;; bb script/x.clj does not put the project root on the classpath, so the
;; helper ns is loaded via load-file and referenced fully qualified (same
;; pattern as script/bench_run.clj loading script/bench.clj).
(load-file "script/localstack.clj")

(def function-name (or (System/getenv "LAMBDA_MVP_FUNCTION_NAME") "lambda-mvp-jlt"))

(def localstack-endpoint
"Non-nil when LAMBDA_ENDPOINT_URL is set: every aws call below is redirected
there (normally a LocalStack on localhost) with dummy credentials, so the
same lifecycle runs against a local emulator instead of a real account."
(script.localstack/endpoint (System/getenv)))

(def role-name (str function-name "-role"))
(def zip-path "dist/lambda.zip")
(def bootstrap-path "dist/bootstrap")
(def policy-arn "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole")

(defn- sh [& args]
(let [{:keys [exit out err]} (apply p/shell {:out :string :err :string :continue true} args)]
(let [{:keys [exit out err]}
(apply p/shell {:out :string :err :string :continue true
:extra-env (script.localstack/credentials-env
(System/getenv) localstack-endpoint)}
(script.localstack/with-endpoint args localstack-endpoint))]
{:exit exit :out out :err err}))

(defn- die! [& msg]
Expand All @@ -44,11 +60,14 @@
"Fail fast with a clear message if the aws CLI has no usable
credentials/region, rather than letting a later call fail obscurely."
[]
(let [{:keys [exit err]} (sh "aws" "sts" "get-caller-identity" "--output" "json")]
(when-not (zero? exit)
(die! "aws CLI has no usable credentials/region."
"Set AWS_PROFILE/AWS_REGION or run `aws configure`, then retry.\n"
(str/trim (or err ""))))))
(if localstack-endpoint
(println "lambda-mvp-jlt: LAMBDA_ENDPOINT_URL set -- using" localstack-endpoint
"; skipping the real-AWS identity check")
(let [{:keys [exit err]} (sh "aws" "sts" "get-caller-identity" "--output" "json")]
(when-not (zero? exit)
(die! "aws CLI has no usable credentials/region."
"Set AWS_PROFILE/AWS_REGION or run `aws configure`, then retry.\n"
(str/trim (or err "")))))))

(def ^:private trust-policy
(json/generate-string
Expand Down
19 changes: 13 additions & 6 deletions script/bench_run.clj
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,11 @@
(Integer/parseInt (or (System/getenv "BENCH_WARM_SAMPLES") "5")))

(defn- sh [& args]
(let [{:keys [exit out err]} (apply p/shell {:out :string :err :string :continue true} args)]
(let [{:keys [exit out err]}
(apply p/shell {:out :string :err :string :continue true
:extra-env (script.localstack/credentials-env
(System/getenv) localstack-endpoint)}
(script.localstack/with-endpoint args localstack-endpoint))]
{:exit exit :out out :err err}))

(defn- die! [& msg]
Expand All @@ -31,11 +35,14 @@
"Fail fast with a clear message if the aws CLI has no usable
credentials/region, rather than letting a later call fail obscurely."
[]
(let [{:keys [exit err]} (sh "aws" "sts" "get-caller-identity" "--output" "json")]
(when-not (zero? exit)
(die! "aws CLI has no usable credentials/region."
"Set AWS_PROFILE/AWS_REGION or run `aws configure`, then retry.\n"
(str/trim (or err ""))))))
(if localstack-endpoint
(println "lambda-mvp-jlt: LAMBDA_ENDPOINT_URL set -- using" localstack-endpoint
"; skipping the real-AWS identity check")
(let [{:keys [exit err]} (sh "aws" "sts" "get-caller-identity" "--output" "json")]
(when-not (zero? exit)
(die! "aws CLI has no usable credentials/region."
"Set AWS_PROFILE/AWS_REGION or run `aws configure`, then retry.\n"
(str/trim (or err "")))))))

(defn- set-memory! [tier]
(let [{:keys [exit err]} (sh "aws" "lambda" "update-function-configuration"
Expand Down
12 changes: 11 additions & 1 deletion script/demo.clj
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,11 @@
(def base-image (or (System/getenv "BASE_IMAGE") "public.ecr.aws/amazonlinux/amazonlinux:2023"))

(defn- sh [& args]
(let [{:keys [exit out err]} (apply p/shell {:out :string :err :string :continue true} args)]
(let [{:keys [exit out err]}
(apply p/shell {:out :string :err :string :continue true
:extra-env (script.localstack/credentials-env
(System/getenv) localstack-endpoint)}
(script.localstack/with-endpoint args localstack-endpoint))]
{:exit exit :out out :err err}))

(defn- die! [& msg]
Expand All @@ -27,6 +31,12 @@
"-- see Prerequisites in docs/guide/getting-started.md."))))

(defn- require-aws! []
(if localstack-endpoint
(println "lambda-mvp-jlt: LAMBDA_ENDPOINT_URL set -- using" localstack-endpoint
"; skipping the real-AWS credential/region checks")
(require-real-aws!)))

(defn- require-real-aws! []
(let [region (some not-empty [(System/getenv "AWS_REGION")
(System/getenv "AWS_DEFAULT_REGION")
(str/trim (:out (sh "aws" "configure" "get" "region")))])]
Expand Down
Loading