Search for a command to run...
Lambda MicroVM images never touch a registry, so no scanner ever re-checks them for new CVEs. This post builds the "scan-twin" pattern that restores continuous visibility, then automates everything from finding to fix — AI-analyzed issues, a self-opening merge request, and Inspector-verified closure — for about $3/month, with the human merge as the only manual step.
No comments yet. Be the first to comment.
Our EKS runner cluster cost $270/month — 95% of it paying for idle. I rebuilt it on Lambda MicroVMs: every CI job gets its own Firecracker VM restored from a snapshot with Docker already warm, cutting job pickup from 2 minutes to ~4 seconds at ~$0.01 per job.
How I turned a quarterly, spreadsheet-driven compliance chore into an event-driven serverless pipeline that tests segmentation within minutes of every relevant network change — and how, along the way,
How we reduced our CI/CD infrastructure costs by switching to EKS Auto Mode with Spot instances, and why Infrastructure as Code was essential for production reliability
A few days ago, I received an email I'd been hoping for: I've been accepted into the AWS Community Builders program under the Dev Tools category. In this post, I want to share what the program is, why
This is Part 2 of a 2-part series. Part 1: Serverless GitLab Runners on AWS Lambda MicroVMs — killing our idle CI cluster (architecture, ~4 s job pickup, ~$0.01/job). Part 2 (this post): the security consequence — MicroVM images get zero post-build CVE visibility — and the fully automated find → analyze → fix → verify loop that solves it. Companion repo: gitlab.com/sawalanipawan/lambda-microvm-gitlab-runners
da-microvm-gitlab-runners— the full setup from both parts, parameterized for your account, with open-source Trivy as the build gate so it works on GitLab Free.
TL;DR: Lambda MicroVM images never touch a container registry — Lambda builds them internally from a Dockerfile — so Amazon Inspector and every registry-based scanner is blind to them. I closed that gap with a scan-twin: the pipeline builds the identical Dockerfile into ECR, where Inspector continuously re-scans it as new CVEs are published. From there, everything is automated: each HIGH/CRITICAL finding becomes a GitLab issue with provenance worked out, AWS DevOps Agent attaches an exploitability analysis, and a single ready-for-review merge request carries the fix. A human merge is the only manual step — and no issue closes until Inspector confirms the CVE is actually gone from the next image.
In the live run, one merged MR took the image from 23 HIGH CVEs to 0. The whole loop costs about $3/month to keep running. And along the way I hit an Amazon Linux 2023 gotcha that turns dnf upgrade into a silent no-op — if your patch story is "we rebuild regularly", check yours actually patches anything.
Part 1 ended with a fleet that looks great in a security review: every CI job gets a fresh Firecracker VM, per-job kernels, no long-lived runners, no inbound network.
Then I asked a boring compliance question: who scans the runner image?
Answer: nobody — and nobody can. MicroVM images are built by Lambda itself. You upload a Dockerfile bundle; Lambda builds and stores the result internally. There is no registry artifact for anything to scan. For a payments-adjacent platform, "the CI environment is unscannable" is not a sentence you want to say in an audit.
Here's precisely what you lose compared to a normal ECR image:
| Image in ECR | Lambda MicroVM image | |
|---|---|---|
| Where it's built | your pipeline, pushed to a registry | inside Lambda, from a Dockerfile bundle |
| Scan at build time | your pipeline gate | you can only scan the inputs, never Lambda's output |
| Scan after release | Inspector re-scans on every new CVE disclosure | nothing — no artifact exists to scan |
| CVE published 3 months after you built | finding appears automatically, event fires | you find out never |
| Inventory / SBOM | registry + Inspector give you both | opaque |
The third row is the one that matters. Build-time scanning answers "was this image clean when I built it?" Continuous scanning answers "is it clean today?" For a long-lived runner image, only the second question keeps you honest. The rest of this post is about winning that property back — and then automating everything downstream of it.
Build twice. The pipeline builds the MicroVM image and pushes the identical Dockerfile to ECR — the "scan-twin", tagged to match the MicroVM version exactly.
Inspector watches the twin continuously, re-scanning whenever a new CVE is published.
Every HIGH/CRITICAL finding flows through EventBridge into a Step Functions workflow: classify where the CVE came from, file a GitLab issue, get an AWS DevOps Agent investigation, post the analysis on the issue.
A remediation MR opens itself — one shared, ready-for-review MR that fixes every patchable CVE at once.
A human reviews and merges. That's the only manual step.
Inspector referees the outcome. A verifier closes each issue only when the new image scans clean for that CVE. Nothing self-certifies.
We own the Dockerfile, so the fix is almost embarrassingly simple: build the same Dockerfile twice. Two ECR repositories do all the work:
scan-mirror — the twin itself. Its tags match the MicroVM image version exactly (4.0, 5.0, …), so an Inspector finding on scan-mirror:4.0 is a finding on MicroVM image 4.0. Version parity is the whole trick — without it you're scanning "something similar", which convinces nobody.
base-mirror — the unmodified AL2023 base image, pinned by digest. I call it the provenance oracle: if a CVE appears here too, it's inherited from the base; if it only appears in the twin, our layers introduced it. One API call ends every "is this the base's fault?" debate.
ECR continuous scanning does the rest: a Tuesday CVE disclosure lights up an image you built in March, automatically.
Housekeeping that keeps the twins honest: only the latest and previous version exist on both sides (ECR lifecycle policy on one side, a pipeline job pruning MicroVM versions on the other — there's no native lifecycle for those). Pre-release builds are the pipeline gate's job — fail on any Critical; released images are Inspector's. At work the gate is GitLab Ultimate's Container Scanning; the companion repo ships plain open-source Trivy instead — same engine (Ultimate wraps Trivy internally), same verdicts, works on GitLab Free.
Inspector findings arrive as EventBridge events and drive a Standard Step Functions workflow. Five steps, one Lambda handler:
enrich — parse the finding; skip anything that isn't a released mirror image; ask the provenance oracle whose CVE this is; check for a newer upstream base.
fileIssue — an idempotent GitLab issue per CVE + image version, with a facts table: severity, affected packages, fix availability, provenance. If the same CVE was already tracked on an older version, both threads get cross-linked with a loud "the rebuild did NOT clear this" warning.
agentTask — hand the case to AWS DevOps Agent as an investigation.
wait / checkAgent — poll every 2 minutes (investigations take ~4–5).
harvest — pull the agent's analysis from its journal and post it onto the issue as a collapsible section.
Why Step Functions and not one Lambda? The agent investigation is a minutes-long async job — exactly what Standard workflows exist for. Bonus: the console graph doubles as live documentation. You can literally watch a CVE travel detect → issue → agent → analysis → remediation.
I ran the same findings through the agent in three configurations, plus a direct Bedrock call as a control. The differences were not subtle.
Bundle-only (no capabilities attached — my first setup): the agent reasons over the context bundle you hand it. Useful — for our sqlite CVE it correctly weighed exploitability in a CI-runner context and spotted the backported fix — but it's an essay over your facts, and a direct Bedrock call (46 s, ~$0.04) produces a comparable essay 8× faster.
With a GitLab capability provider (register in the console, then associate-service — the association grants space-wide repo access, no per-task wiring): the analysis changed class. It cited exact lines ("line 26 already tracks latest", "--max-builds 1, line 82"), read our docs and echoed our own design rules back, and found a repo-grounded exploitability argument nothing else produced — our runner uses gRPC as a client only, no --session-server, so the vulnerable listener path never activates. The Bedrock control gave the same shape of advice but invented a plausible-looking Dockerfile pattern that doesn't exist in the repo.
Plus an AWS cloud source (console → agent space → Capabilities → add primary source; the auto-created role's policy is effectively read-only — I audited all 880 actions): the agent stopped trusting my bundle entirely. It queried Inspector itself (found the CVE ACTIVE on all four current mirror images, and understood that the 11 CLOSED findings were candidates our keep-2 retention had reaped), checked ECR's current tags — catching state newer than my pre-computed context — and opened its report with a table of which sources it accessed and which it couldn't.
What the agent still does not do is write code or open MRs — it advises; the pipeline stays the author and the human stays the approver. And two preview gaps to know about: it has no GitLab issue/MR-API tool (repo files, branches, diffs and git log only), and no Lambda MicroVMs API in its toolset — it told me both, unprompted.
Practical notes if you wire this up yourself:
The service's IAM prefix is aidevops, not the CLI's devops-agent namespace. You'll discover this via AccessDenied, like I did.
The agent emits rich Markdown with its own code fences. Render it natively — wrap it in a code block and its first internal fence breaks your formatting in half.
Judge the agent on its tooled configuration. Bundle-only, it's an expensive prompt; with repo + cloud access, it earns the extra minutes.
A remediate step runs after harvest and turns facts into a change. The key design choice: the patch is generated from Inspector's facts, not parsed from the agent's prose. fixedInVersion says deterministically whether a fix exists; the provenance oracle says where it lives. Facts drive automation; prose informs humans.
Nine CVEs were open when the step first ran. It opened one merge request, not nine — every remediable CVE appends itself to a shared, ready-for-review MR. The change is a single managed Dockerfile block:
# --- security refresh (managed by the finding workflow) ---
ARG SEC_REFRESH=2026-07-22
RUN echo "security refresh ${SEC_REFRESH}" \
&& echo latest > /etc/dnf/vars/releasever \
&& dnf -y upgrade --setopt=install_weak_deps=0 \
&& dnf clean all
Bumping SEC_REFRESH invalidates every layer below it, so one rebuild refreshes all OS packages and re-fetches the gitlab-runner and AWS CLI binaries. CVEs with no published fix don't generate MR noise — they get a "waiting on upstream" note on their issue instead.
Three properties I'd defend in review:
The MR proves itself. Its branch pipeline builds the actual refreshed image and runs the scan gate on it — the reviewer sees scan results for the fix before merging.
A human merge is the only manual step. Nothing self-approves.
The MR closes nothing. Issues close exclusively via a resolution verifier that asks Inspector whether the CVE is still present on the newest image. The agent advises, the pipeline proposes, the human merges — and Inspector referees. A fix that doesn't actually fix gets caught and cross-linked, not silently celebrated.
First run of the remediation build: gate green… and 8 of 9 CVEs still present. The dnf upgrade output said it plainly:
Downloading metadata...
Nothing to do.
Complete.
Amazon Linux 2023 pins every image to a point-release snapshot via /etc/dnf/vars/releasever. dnf upgrade faithfully upgrades against that frozen snapshot — which by definition contains no newer packages. Worse, the later dnf install layers resolve against the same snapshot, so they install the exact vulnerable versions Inspector just flagged. If your container patch story is "we rebuild the image regularly", AL2023 quietly made it a no-op.
The fix is one line — echo latest > /etc/dnf/vars/releasever before the upgrade — which un-freezes that upgrade and every later install. The re-run took the image from 23 distinct HIGH CVEs to 0.
Two lessons from this failure:
Build-time gates can't catch it. A gate checks a policy ("no Criticals"), not an intent ("this build was supposed to remove CVE-X"). Only the Inspector-refereed verification noticed the remediation was a placebo.
The one CVE that did clear proved the point. A Go stdlib CVE vanished because the refresh re-downloaded the gitlab-runner binary — one path worked while the other silently didn't. Partial success is the most dangerous kind. Verify per-CVE, not per-build.
The merge landed and the loop ran end to end with no hands on it: new MicroVM image version cut, mirror retagged, Inspector re-scanned, and the verifier closed all nine issues with per-CVE "verified resolved by Inspector" receipts.
Then it did something better. It caught two findings on the new image that the build-time gate had waved through — a Go stdlib CVE and a gRPC module advisory, both embedded in pre-built binaries we download. Trivy and Inspector genuinely disagree here; Inspector's deeper Go-binary analysis found them. The workflow filed both, attached agent investigations, and — after I taught it the difference — correctly declined to open an MR: dnf upgrade cannot patch a Go module inside someone else's binary. Both issues now sit open with a "waiting on an upstream rebuild" note, and the verifier will close them the moment a refreshed binary scans clean.
Two closing lessons: scanner disagreement is a feature if your loop treats one scanner as the referee instead of averaging opinions — and a remediation bot must know what it cannot fix, or it becomes a generator of confident, empty merge requests.
Real numbers, not free-tier hand-waving. Sources: my account's Cost Explorer (the demo has been running the loop live) plus eu-west-1 list prices for the services still inside a trial or free tier on my account.
| Component | Price basis (eu-west-1) | Monthly at demo scale |
|---|---|---|
| Inspector continuous ECR scanning | $0.09 per image per month | ~4 scanned images (2 versions + cache tag + base) ≈ $0.36 |
| ECR storage | $0.10 per GB-month | 2.5 GB across both mirrors ≈ $0.25 |
| MicroVM image snapshot storage | measured (bills as Lambda MicroVM-Snapshot-Storage-GB-Hour) |
~$1.47 for the keep-2 versions |
| Secrets Manager | $0.40 per secret | 2 secrets = $0.80 |
| Step Functions (Standard) | $0.025 per 1k transitions | ~12 transitions per finding → $0.03 per 100 findings |
| Workflow Lambdas + 30-min verifier | per-invocation | < $0.01 |
| EventBridge (AWS events) + SNS email | free at this usage | $0 |
| Trivy gate | open source | $0 |
| GitLab SaaS ARM64 minutes (pipeline) | $10 per 1,000 min | ~9 min per push/merge ≈ $0.09 per run |
| AWS DevOps Agent | preview — no charges billed on my account yet | TBD at GA; watch this line |
Steady state: ≈ $3/month, dominated by MicroVM snapshot storage and Inspector's per-image fee. The marginal cost of processing a finding — issue, agent hand-off, MR update, verification — is under one cent. Continuous CVE coverage for an unscannable image class, for less than a coffee a month.
(Two honesty notes: Inspector was inside its 15-day trial on my account, so its line is list price rather than a bill I've paid; and DevOps Agent is in preview — nothing has hit Cost Explorer for it yet, so treat that row as open until GA pricing lands.)
| Finding → issue + agent analysis | ~4–5 min bundle-only; ~6–7 min with live repo + AWS verification |
| Same analysis via direct Bedrock (control) | 46 s, ~$0.04 — fast, but ungrounded in the repo |
| HIGH CVEs before → after one merged MR | 23 → 0 |
| MRs opened for 9 CVEs | 1 (shared, ready-for-review) |
| Manual steps in the loop | 1 (the merge) |
| Issues closed without Inspector's confirmation | 0, by design |
| Scan-twin build, cold vs layer-cached | 214 s → 121 s (all layers CACHED) |
| Running cost | ≈ $3/month + <1¢ per finding (see cost table) |
Scan-twin: if any of your images are built where scanners can't see them (MicroVMs, on-prem bakeries, appliance firmware), build the identical Dockerfile into ECR with version-parity tags and let continuous scanning do its job. The pattern costs one pipeline job.
Provenance oracle: mirror your unmodified base image and diff findings against it. One API call ends every "is this the base's fault?" debate.
Facts drive automation, prose informs humans: generate patches from fixedInVersion and digests; attach the agent's analysis for the reviewer.
Verify per-CVE with the scanner as referee: never let the thing that proposed a fix also declare it successful.
Give your agent tools before judging it: a capability-provider association turned essays into file:line-grounded, live-verified analyses. The same model with and without tooling is two different products.
Check your AL2023 images today: if plain dnf upgrade is your patch mechanism, grep your build logs for "Nothing to do". You might be rebuilding vulnerabilities with a green pipeline.
Repo layout, state machine definition, and the full finding-workflow Lambda are in the companion repo from Part 1.