Search for a command to run...
Our EKS runner cluster cost $270/month — 95% of it paying for idle. I rebuilt it on Lambda MicroVMs: every CI job gets its own Firecracker VM restored from a snapshot with Docker already warm, cutting job pickup from 2 minutes to ~4 seconds at ~$0.01 per job.
No comments yet. Be the first to comment.
Lambda MicroVM images never touch a registry, so no scanner ever re-checks them for new CVEs. This post builds the "scan-twin" pattern that restores continuous visibility, then automates everything from finding to fix — AI-analyzed issues, a self-opening merge request, and Inspector-verified closure — for about $3/month, with the human merge as the only manual step.
How I turned a quarterly, spreadsheet-driven compliance chore into an event-driven serverless pipeline that tests segmentation within minutes of every relevant network change — and how, along the way,
How we reduced our CI/CD infrastructure costs by switching to EKS Auto Mode with Spot instances, and why Infrastructure as Code was essential for production reliability
A few days ago, I received an email I'd been hoping for: I've been accepted into the AWS Community Builders program under the Dev Tools category. In this post, I want to share what the program is, why
This is Part 1 of a 2-part series. Part 1 (this post): replacing an idle EKS runner cluster with per-job Firecracker VMs — architecture, measured numbers, and the launch-week gotchas. Part 2: MicroVM images are invisible to your vulnerability scanner — closing the CVE loop with Inspector, Step Functions, and AWS DevOps Agent. Companion repo: everything in this series is reproducible from
gitlab.com/sawalanipawan/lambda-microvm-gitlab-runners— CDK stacks, runner image, pipelines, and a zero-to-green runbook.
TL;DR: Our GitLab runners lived on an EKS cluster costing ~$270/month —and 95% of that paid for nothing happening between pipelines. I rebuilt the whole thing on AWS Lambda MicroVMs (GA June 2026). Now every pending CI job gets its own fresh Firecracker VM, restored from a snapshot in which the Docker daemon is already running and common images are already pulled. The VM runs exactly one job and vanishes.
Results: job pickup 60–120 s → ~4 s, cost \(270/mo fixed → ~\)0.01 per job, and every job now gets its own kernel instead of a privileged pod on a shared node.
GitLab marks a job pending and fires a webhook.
A tiny receiver Lambda validates it and drops it on an SQS FIFO queue.
A dispatcher Lambda mints a one-time GitLab runner via the API and calls RunMicrovm.
The MicroVM wakes from its snapshot (~5 s), with dockerd already warm, picks up exactly that one job, and runs it.
The job ends → the VM terminates itself. A sweeper Lambda cleans up anything that slipped through. Nothing is left running.
That's the entire platform: three small Lambdas, one queue, one Dockerfile.
(Context: gitlab.com SaaS, ARM64 images for Fargate. Runner fleet: EKS Auto Mode + kubernetes executor.)
Three things bugged me about our EKS runner setup:
The cost curve is upside down. Cost Explorer said ~\(270/mo for the runner platform. The actual Spot compute doing CI work? About \)14. The other 95% was an idle tax — control plane, manager node, NAT gateway, sitting there through nights and weekends.
privileged: true. Docker builds forced privileged job pods on shared nodes. Any job could root the node it ran on.
The upgrade treadmill. Our infra repo carried three generations of runner stack (EKS 1.31 + Karpenter, 1.34 + newer Karpenter, then Auto Mode). None of that work ever delivered a feature.
If you've only met Lambda as "functions", recalibrate. Lambda MicroVMs (GA 22 June 2026) are full Firecracker micro-VMs with lifecycle control, running on the same virtualization layer as regular Lambda. Three things make them different from a function:
A real OS you control. Amazon Linux 2023, with full OS capabilities on request. That means you can run dockerd inside the VM, mount filesystems, create network namespaces.
A lifetime measured in hours, not seconds. Up to 8 hours running or suspended; terminated on your API call.
VM-level isolation. Each MicroVM is its own kernel. AWS's stated use cases: AI code sandboxes, dev environments, security scanning — and multi-tenant CI executors, which is exactly what a runner fleet is.
The lifecycle, end to end:
You upload a Dockerfile bundle to S3. No docker build on your side, no registry. Lambda builds the image itself, on a managed AL2023 base.
Lambda boots it and waits for your /ready hook. You return 200 only when everything you want "pre-baked" is initialized.
Lambda snapshots the entire VM — memory and disk. This is the magic. The snapshot is not a container image; it's a running machine frozen mid-flight. Warm daemons, filled caches, open files: all captured.
RunMicrovm resumes a fresh copy of that snapshot in seconds. Each VM gets a private HTTPS endpoint and a /run hook that delivers a per-VM payload — your slot for per-job configuration.
Networking is explicit. You attach "connectors": internet or VPC egress, and ingress that can be disabled entirely (NO_INGRESS) or reduced to an IAM-gated shell channel.
Terminate when done — all charges stop. (Suspend/resume exists too: ~1 s each way, storage-only billing while suspended.)
Sizing follows the snapshot. Memory is fixed per image (the snapshot is a RAM image), vCPU is proportional (2 GB = 1 vCPU), and the VM automatically bursts to 4× baseline under load. Billing is per second while running.
The superpower: you initialize an environment once, snapshot it, and every VM launch resumes from that exact state in seconds.
The trick most writeups miss: what you snapshot doesn't have to be your app. It can be infrastructure state. My image's entrypoint:
starts dockerd
pre-pulls maven, node, docker:cli, alpine (all linux/arm64)
answers the /ready hook → Lambda takes the snapshot
So every CI job starts with a hot Docker daemon and a primed layer cache — the two slowest parts of containerized CI, already done, forever.
Design choices worth defending:
The runner runs inside the VM as gitlab-runner run-single --max-builds 1. Every GitLab feature — caches, artifacts, services, timeouts, OIDC id_tokens — just works, because it's a completely normal runner that happens to live for one job. I didn't write a custom executor (that API is in maintenance mode), and GitLab's autoscaler plugins want SSH access that MicroVMs deliberately don't offer.
NO_INGRESS. A CI runner only needs outbound connections. The job VM cannot accept an inbound connection from anywhere. Try that with a node pool.
One-time runners, minted per job. The dispatcher creates an ephemeral runner via POST /user/runners and delivers its token through the VM's /run payload. No long-lived runner token exists anywhere — which also killed the plaintext glrt- token we'd been carrying in git.
No human ever touches a credential. GitLab tokens live in Secrets Manager as managed external secrets: AWS itself rotates them against GitLab's token-rotate API every 14 days, and the bootstrap value is revoked minutes after creation. The Lambdas re-read the secret every 5 minutes, so rotation is invisible.
Idempotent end to end. SQS FIFO dedupes on job id; RunMicrovm takes a per-job clientToken. A redelivered webhook cannot double-launch a VM. A lost webhook is healed by the sweeper, which re-queues any job GitLab still reports as pending.
Your Dockerfile becomes a container inside the VM, layered on a managed AL2023 base. Start FROM public.ecr.aws/lambda/microvms:al2023-minimal; running dockerd requires additionalOsCapabilities: ["ALL"].
Memory is fixed at image build time. The snapshot is a RAM image, so RunMicrovm has no size parameter — different sizes mean different images. Also check your applied "Max allocated memory" quota: the default is 400 GB, but my new account was silently clamped to 8 GB, and the Service Quotas API refuses requests below the default. That's a support-case conversation.
Outbound UDP is blocked, so DNS dies — unless you point resolv.conf at Lambda's stub resolver, 169.254.169.253. Symptom: docker pull times out inside the VM.
Job containers can't reach that resolver over the docker bridge either. Run them with --docker-network-mode host. With one job per VM there's nothing to isolate anyway.
Lifecycle hooks arrive on port 9000, and /run must answer within 60 s — spawn your workload and return immediately.
Debug image builds outside CloudFormation. A failed build surfaces as NotStabilized and rollback deletes the evidence. Iterate with the CLI (aws lambda-microvms create-microvm-image; shorthand keys are camelCase) and read build logs at /aws/lambda-microvms/<name> — then codify in CDK. My own failure was dnf5 pedantry: install_weak_deps=False is an "invalid boolean"; use =0.
Runtime logs are double opt-in. Enable them per RunMicrovm call and give the execution role logs:* permissions — miss either and you get silence.
Grant lambda:PassNetworkConnector to whoever calls RunMicrovm — a PassRole-style guard that only errors at dispatch time.
Bundle the AWS SDK into your Lambdas (externalModules: [] in CDK): the runtime's built-in SDK predates the service. Related: the IAM policy simulator doesn't know the new actions and returns misleading implicitDeny — for brand-new services, trust the actual error messages.
Three more, earned during a second rollout of this design on a real production codebase:
With host networking, docker doesn't put the container's hostname in /etc/hosts — so InetAddress.getLocalHost() fails. Symptom: Quartz dies with "Cannot run without an instance id" and takes the whole Spring test context with it (2,585 cascaded test errors from one lookup). Fix: a fixed hostname via --docker-hostname + --docker-extra-hosts.
run-single caps job logs at 4 MB by default — large test suites get truncated mid-run. Set --output-limit (KB) to whatever your current fleet uses.
CloudFormation's AWS::Lambda::MicrovmImage wants Key/Value pairs in EnvironmentVariables, not Name/Value — you'll only find out on first real use, since an empty list validates fine.
Measured on the demo (July 2026, eu-west-1, 2 GB/1 vCPU baseline VMs):
| Metric | EKS runner cluster | MicroVM runners |
|---|---|---|
| Fixed monthly cost | ~$256 | ~$0 |
| Cost per average job | opaque (shared) | ~$0.01 |
| Push → job running (cold) | 60–120 s | ~25 s total; 4 s queue time |
| RunMicrovm → VM RUNNING | — | 189 ms API call, ~5 s to RUNNING |
| docker build with warm layers | minutes (pull first) | 14.7 s |
| Isolation | privileged pods, shared kernel | 1 job = 1 kernel |
Break-even honesty: per vCPU-hour, MicroVM compute costs ~5–7× x86 Spot. The win is structural — deleting the idle floor, not beating Spot on rate. If your CI volume is ~5× ours, a packed cluster wins on raw compute. Measure your own idle ratio first.
ARM64/Graviton only. Fine for us (we build ARM64 images anyway); a hard stop if your toolchain is x86-native.
The service is weeks old. Expect API/CLI sharp edges (see above).
8 h max VM lifetime. Fine for CI; a hard cap for anything else.
Everything is CDK TypeScript + three small Lambdas + one Dockerfile, published as a public companion repo: gitlab.com/sawalanipawan/lambda-microvm-gitlab-runners — parameterized for any AWS account and gitlab.com namespace (GitLab Free works; the security scanning uses open-source Trivy, no Ultimate needed). The runbook takes you from zero to a green pipeline in ~30 minutes.
This architecture forces an awkward security question: your runner image is now built inside Lambda and never lands in a registry — so Inspector, ECR scanning, and every registry-based scanner you own is blind to it. Part 2 covers the scan-twin pattern that restores continuous CVE visibility, and the Step Functions + AWS DevOps Agent workflow that turns findings into agent-analyzed GitLab issues, a ready-for-review remediation MR, and Inspector-verified closure — with exactly one human step in the loop: the merge. (Spoiler: it took a live image from 23 HIGH CVEs to 0, and caught an Amazon Linux 2023 quirk that silently turns dnf upgrade into a no-op.)
If you run GitLab runners on a cluster today, do one thing before your next platform sprint: open Cost Explorer and compute your own idle ratio. Mine was 95% — and that number, not any of the architecture above, is what made this migration inevitable.
