# Serverless GitLab Runners on AWS Lambda MicroVMs: killing our idle CI cluster > **This is Part 1 of a 2-part series.** **Part 1 (this post):** replacing an idle EKS runner cluster with per-job Firecracker VMs — architecture, measured numbers, and the launch-week gotchas. **Part 2:** [MicroVM images are invisible to your vulnerability scanner — closing the CVE loop with Inspector, Step Functions, and AWS DevOps Agent](https://thepawan.dev/your-lambda-microvm-ci-images-have-cves-nobody-can-see-closing-the-loop-with-inspector-step-functions-and-aws-devops-agent). **Companion repo:** everything in this series is reproducible from [`gitlab.com/sawalanipawan/lambda-microvm-gitlab-runners`](https://gitlab.com/sawalanipawan/lambda-microvm-gitlab-runners) — CDK stacks, runner image, pipelines, and a zero-to-green runbook. * * * **TL;DR:** Our GitLab runners lived on an EKS cluster costing ~$270/month —and 95% of that paid for *nothing happening* between pipelines. I rebuilt the whole thing on **AWS Lambda MicroVMs** (GA June 2026). Now every pending CI job gets its own fresh Firecracker VM, restored from a snapshot in which the Docker daemon is **already running** and common images are **already pulled**. The VM runs exactly one job and vanishes. Results: job pickup **60–120 s → ~4 s**, cost **$270/mo fixed → ~$0.01 per job**, and every job now gets its own kernel instead of a `privileged` pod on a shared node. ## The whole thing in five steps 1. GitLab marks a job **pending** and fires a webhook. 2. A tiny **receiver Lambda** validates it and drops it on an SQS FIFO queue. 3. A **dispatcher Lambda** mints a one-time GitLab runner via the API and calls `RunMicrovm`. 4. The **MicroVM** wakes from its snapshot (~5 s), with dockerd already warm, picks up exactly that one job, and runs it. 5. The job ends → the VM **terminates itself**. A sweeper Lambda cleans up anything that slipped through. Nothing is left running. That's the entire platform: three small Lambdas, one queue, one Dockerfile. ## The itch *(Context: gitlab.com SaaS, ARM64 images for Fargate. Runner fleet: EKS Auto Mode + kubernetes executor.)* Three things bugged me about our EKS runner setup: 1. **The cost curve is upside down.** Cost Explorer said ~$270/mo for the runner platform. The actual Spot compute doing CI work? About $14. The other 95% was an idle tax — control plane, manager node, NAT gateway, sitting there through nights and weekends. 2. `privileged: true`**.** Docker builds forced privileged job pods on shared nodes. Any job could root the node it ran on. 3. **The upgrade treadmill.** Our infra repo carried three generations of runner stack (EKS 1.31 + Karpenter, 1.34 + newer Karpenter, then Auto Mode). None of that work ever delivered a feature. ## First: what exactly is a Lambda MicroVM? If you've only met Lambda as "functions", recalibrate. [Lambda MicroVMs](https://docs.aws.amazon.com/lambda/latest/dg/lambda-microvms-guide.html) (GA 22 June 2026) are **full Firecracker micro-VMs with lifecycle control**, running on the same virtualization layer as regular Lambda. Three things make them different from a function: * **A real OS you control.** Amazon Linux 2023, with full OS capabilities on request. That means you can run *dockerd inside the VM*, mount filesystems, create network namespaces. * **A lifetime measured in hours, not seconds.** Up to 8 hours running or suspended; terminated on your API call. * **VM-level isolation.** Each MicroVM is its own kernel. AWS's stated use cases: AI code sandboxes, dev environments, security scanning — and multi-tenant CI executors, which is exactly what a runner fleet is. **The lifecycle, end to end:** 1. **You upload a** `Dockerfile` **bundle to S3.** No docker build on your side, no registry. Lambda builds the image itself, on a managed AL2023 base. 2. **Lambda boots it and waits for your** `/ready` **hook.** You return 200 only when everything you want "pre-baked" is initialized. 3. **Lambda snapshots the entire VM — memory and disk.** This is the magic. The snapshot is not a container image; it's a *running machine frozen mid-flight*. Warm daemons, filled caches, open files: all captured. 4. `RunMicrovm` **resumes a fresh copy of that snapshot in seconds.** Each VM gets a private HTTPS endpoint and a `/run` hook that delivers a per-VM payload — your slot for per-job configuration. 5. **Networking is explicit.** You attach "connectors": internet or VPC egress, and ingress that can be disabled entirely (`NO_INGRESS`) or reduced to an IAM-gated shell channel. 6. **Terminate when done** — all charges stop. (Suspend/resume exists too: ~1 s each way, storage-only billing while suspended.) **Sizing follows the snapshot.** Memory is fixed per image (the snapshot *is* a RAM image), vCPU is proportional (2 GB = 1 vCPU), and the VM automatically bursts to 4× baseline under load. Billing is per second while running. ## Why that fits CI like a glove The superpower: **you initialize an environment once, snapshot it, and every VM launch resumes from that exact state in seconds.** The trick most writeups miss: what you snapshot doesn't have to be *your app*. It can be *infrastructure state*. My image's entrypoint: 1. starts `dockerd` 2. pre-pulls `maven`, `node`, `docker:cli`, `alpine` (all linux/arm64) 3. answers the `/ready` hook → Lambda takes the snapshot So every CI job starts with a **hot Docker daemon and a primed layer cache** — the two slowest parts of containerized CI, already done, forever. ## Architecture ![Runner architecture](https://gitlab.com/sawalanipawan/lambda-microvm-gitlab-runners/-/raw/main/docs/diagrams/part1-runner-architecture.drawio.png align="center") Design choices worth defending: * **The runner runs *inside* the VM** as `gitlab-runner run-single --max-builds 1`. Every GitLab feature — caches, artifacts, services, timeouts, OIDC `id_tokens` — just works, because it's a completely normal runner that happens to live for one job. I didn't write a custom executor (that API is in maintenance mode), and GitLab's autoscaler plugins want SSH access that MicroVMs deliberately don't offer. * `NO_INGRESS`**.** A CI runner only needs outbound connections. The job VM *cannot accept* an inbound connection from anywhere. Try that with a node pool. * **One-time runners, minted per job.** The dispatcher creates an ephemeral runner via `POST /user/runners` and delivers its token through the VM's `/run` payload. No long-lived runner token exists anywhere — which also killed the plaintext `glrt-` token we'd been carrying in git. * **No human ever touches a credential.** GitLab tokens live in Secrets Manager as **managed external secrets**: AWS itself rotates them against GitLab's token-rotate API every 14 days, and the bootstrap value is revoked minutes after creation. The Lambdas re-read the secret every 5 minutes, so rotation is invisible. * **Idempotent end to end.** SQS FIFO dedupes on job id; `RunMicrovm` takes a per-job `clientToken`. A redelivered webhook cannot double-launch a VM. A *lost* webhook is healed by the sweeper, which re-queues any job GitLab still reports as pending. ## What I learned the hard way (save yourself a day) 1. **Your Dockerfile becomes a container *inside* the VM**, layered on a managed AL2023 base. Start `FROM public.ecr.aws/lambda/microvms:al2023-minimal`; running dockerd requires `additionalOsCapabilities: ["ALL"]`. 2. **Memory is fixed at image build time.** The snapshot is a RAM image, so `RunMicrovm` has no size parameter — different sizes mean different images. Also check your **applied** "Max allocated memory" quota: the default is 400 GB, but my new account was silently clamped to 8 GB, and the Service Quotas API refuses requests below the default. That's a support-case conversation. 3. **Outbound UDP is blocked, so DNS dies** — unless you point `resolv.conf` at Lambda's stub resolver, `169.254.169.253`. Symptom: `docker pull` times out inside the VM. 4. **Job containers can't reach that resolver over the docker bridge** either. Run them with `--docker-network-mode host`. With one job per VM there's nothing to isolate anyway. 5. **Lifecycle hooks arrive on port 9000**, and `/run` must answer within 60 s — spawn your workload and return immediately. 6. **Debug image builds outside CloudFormation.** A failed build surfaces as `NotStabilized` and rollback deletes the evidence. Iterate with the CLI (`aws lambda-microvms create-microvm-image`; shorthand keys are camelCase) and read build logs at `/aws/lambda-microvms/` — then codify in CDK. My own failure was dnf5 pedantry: `install_weak_deps=False` is an "invalid boolean"; use `=0`. 7. **Runtime logs are double opt-in.** Enable them per `RunMicrovm` call *and* give the execution role `logs:*` permissions — miss either and you get silence. 8. **Grant** `lambda:PassNetworkConnector` to whoever calls `RunMicrovm` — a PassRole-style guard that only errors at dispatch time. 9. **Bundle the AWS SDK into your Lambdas** (`externalModules: []` in CDK): the runtime's built-in SDK predates the service. Related: the IAM policy simulator doesn't know the new actions and returns misleading `implicitDeny` — for brand-new services, trust the actual error messages. Three more, earned during a second rollout of this design on a real production codebase: 10. **With host networking, docker doesn't put the container's hostname in** `/etc/hosts` — so `InetAddress.getLocalHost()` fails. Symptom: Quartz dies with "Cannot run without an instance id" and takes the whole Spring test context with it (2,585 cascaded test errors from one lookup). Fix: a fixed hostname via `--docker-hostname` + `--docker-extra-hosts`. 11. `run-single` **caps job logs at 4 MB by default** — large test suites get truncated mid-run. Set `--output-limit` (KB) to whatever your current fleet uses. 12. **CloudFormation's** `AWS::Lambda::MicrovmImage` **wants** `Key`**/**`Value` **pairs in** `EnvironmentVariables`, not `Name`/`Value` — you'll only find out on first real use, since an empty list validates fine. ## The numbers Measured on the demo (July 2026, eu-west-1, 2 GB/1 vCPU baseline VMs): | Metric | EKS runner cluster | MicroVM runners | | --- | --- | --- | | Fixed monthly cost | ~$256 | ~$0 | | Cost per average job | opaque (shared) | ~$0.01 | | Push → job running (cold) | 60–120 s | ~25 s total; **4 s queue time** | | RunMicrovm → VM RUNNING | — | 189 ms API call, ~5 s to RUNNING | | docker build with warm layers | minutes (pull first) | **14.7 s** | | Isolation | privileged pods, shared kernel | 1 job = 1 kernel | Break-even honesty: per vCPU-hour, MicroVM compute costs ~5–7× x86 Spot. The win is structural — deleting the idle floor, not beating Spot on rate. If your CI volume is ~5× ours, a packed cluster wins on raw compute. Measure your own idle ratio first. ## What's still rough * **ARM64/Graviton only.** Fine for us (we build ARM64 images anyway); a hard stop if your toolchain is x86-native. * **The service is weeks old.** Expect API/CLI sharp edges (see above). * **8 h max VM lifetime.** Fine for CI; a hard cap for anything else. ## Steal this Everything is CDK TypeScript + three small Lambdas + one Dockerfile, published as a public companion repo: `gitlab.com/sawalanipawan/lambda-microvm-gitlab-runners` — parameterized for any AWS account and gitlab.com namespace (GitLab Free works; the security scanning uses open-source Trivy, no Ultimate needed). The runbook takes you from zero to a green pipeline in ~30 minutes. ## Coming in Part 2 This architecture forces an awkward security question: your runner image is now built *inside Lambda* and never lands in a registry — so Inspector, ECR scanning, and every registry-based scanner you own is blind to it. Part 2 covers the **scan-twin** pattern that restores continuous CVE visibility, and the Step Functions + AWS DevOps Agent workflow that turns findings into agent-analyzed GitLab issues, a ready-for-review remediation MR, and Inspector-verified closure — with exactly one human step in the loop: the merge. (Spoiler: it took a live image from 23 HIGH CVEs to 0, and caught an Amazon Linux 2023 quirk that silently turns `dnf upgrade` into a no-op.) If you run GitLab runners on a cluster today, do one thing before your next platform sprint: open Cost Explorer and compute your own idle ratio. Mine was 95% — and that number, not any of the architecture above, is what made this migration inevitable.