# Your Lambda MicroVM CI Images Have CVEs Nobody Can See — Closing the Loop with Inspector, Step Functions, and AWS DevOps Agent

> **This is Part 2 of a 2-part series.** **Part 1:** [Serverless GitLab Runners on AWS Lambda MicroVMs — killing our idle CI cluster](https://thepawan.dev/serverless-gitlab-runners-on-aws-lambda-microvms-killing-our-idle-ci-cluster) (architecture, ~4 s job pickup, ~$0.01/job). **Part 2 (this post):** the security consequence — MicroVM images get zero post-build CVE visibility — and the fully automated find → analyze → fix → verify loop that solves it. **Companion repo:** [gitlab.com/sawalanipawan/lambda-microvm-gitlab-runners](https://gitlab.com/sawalanipawan/lambda-microvm-gitlab-runners)`da-microvm-gitlab-runners` — the full setup from both parts, parameterized for your account, with open-source Trivy as the build gate so it works on GitLab Free.

* * *

**TL;DR:** Lambda MicroVM images never touch a container registry — Lambda builds them internally from a Dockerfile — so Amazon Inspector and every registry-based scanner is blind to them. I closed that gap with a **scan-twin**: the pipeline builds the *identical* Dockerfile into ECR, where Inspector continuously re-scans it as new CVEs are published. From there, everything is automated: each HIGH/CRITICAL finding becomes a GitLab issue with provenance worked out, **AWS DevOps Agent** attaches an exploitability analysis, and a single **ready-for-review merge request** carries the fix. A human merge is the only manual step — and no issue closes until Inspector confirms the CVE is actually gone from the next image.

In the live run, one merged MR took the image from **23 HIGH CVEs to 0**. The whole loop costs **about $3/month** to keep running. And along the way I hit an Amazon Linux 2023 gotcha that turns `dnf upgrade` into a *silent no-op* — if your patch story is "we rebuild regularly", check yours actually patches anything.

## The blind spot

Part 1 ended with a fleet that looks great in a security review: every CI job gets a fresh Firecracker VM, per-job kernels, no long-lived runners, no inbound network.

Then I asked a boring compliance question: *who scans the runner image?*

Answer: nobody — and nobody *can*. MicroVM images are built by Lambda itself. You upload a Dockerfile bundle; Lambda builds and stores the result internally. There is no registry artifact for anything to scan. For a payments-adjacent platform, "the CI environment is unscannable" is not a sentence you want to say in an audit.

Here's precisely what you lose compared to a normal ECR image:

|  | Image in ECR | Lambda MicroVM image |
| --- | --- | --- |
| Where it's built | your pipeline, pushed to a registry | **inside Lambda**, from a Dockerfile bundle |
| Scan at build time | your pipeline gate | you can only scan the *inputs*, never Lambda's output |
| Scan after release | **Inspector re-scans on every new CVE disclosure** | **nothing — no artifact exists to scan** |
| CVE published 3 months after you built | finding appears automatically, event fires | you find out never |
| Inventory / SBOM | registry + Inspector give you both | opaque |

The third row is the one that matters. Build-time scanning answers *"was this image clean when I built it?"* Continuous scanning answers *"is it clean* ***today****?"* For a long-lived runner image, only the second question keeps you honest. The rest of this post is about winning that property back — and then automating everything downstream of it.

## The loop in 60 seconds

![The CVE loop](https://gitlab.com/sawalanipawan/lambda-microvm-gitlab-runners/-/raw/main/docs/diagrams/part2-security-loop.drawio.png align="center")

1.  **Build twice.** The pipeline builds the MicroVM image *and* pushes the identical Dockerfile to ECR — the "scan-twin", tagged to match the MicroVM version exactly.
    
2.  **Inspector watches the twin** continuously, re-scanning whenever a new CVE is published.
    
3.  **Every HIGH/CRITICAL finding** flows through EventBridge into a Step Functions workflow: classify where the CVE came from, file a GitLab issue, get an **AWS DevOps Agent** investigation, post the analysis on the issue.
    
4.  **A remediation MR opens itself** — one shared, ready-for-review MR that fixes every patchable CVE at once.
    
5.  **A human reviews and merges.** That's the only manual step.
    
6.  **Inspector referees the outcome.** A verifier closes each issue *only* when the new image scans clean for that CVE. Nothing self-certifies.
    

## The scan-twin pattern

We own the Dockerfile, so the fix is almost embarrassingly simple: **build the same Dockerfile twice.** Two ECR repositories do all the work:

*   `scan-mirror` — the twin itself. Its tags **match the MicroVM image version exactly** (`4.0`, `5.0`, …), so an Inspector finding on `scan-mirror:4.0` *is* a finding on MicroVM image 4.0. Version parity is the whole trick — without it you're scanning "something similar", which convinces nobody.
    
*   `base-mirror` — the unmodified AL2023 base image, pinned by digest. I call it the **provenance oracle**: if a CVE appears here too, it's inherited from the base; if it only appears in the twin, our layers introduced it. One API call ends every "is this the base's fault?" debate.
    

ECR continuous scanning does the rest: a Tuesday CVE disclosure lights up an image you built in March, automatically.

Housekeeping that keeps the twins honest: only the latest and previous version exist **on both sides** (ECR lifecycle policy on one side, a pipeline job pruning MicroVM versions on the other — there's no native lifecycle for those). Pre-release builds are the pipeline gate's job — fail on any Critical; released images are Inspector's. At work the gate is GitLab Ultimate's Container Scanning; the companion repo ships plain **open-source Trivy** instead — same engine (Ultimate wraps Trivy internally), same verdicts, works on GitLab Free.

## From finding to issue: the Step Functions workflow

Inspector findings arrive as EventBridge events and drive a Standard Step Functions workflow. Five steps, one Lambda handler:

1.  **enrich** — parse the finding; skip anything that isn't a released mirror image; ask the provenance oracle whose CVE this is; check for a newer upstream base.
    
2.  **fileIssue** — an idempotent GitLab issue per CVE + image version, with a facts table: severity, affected packages, fix availability, provenance. If the same CVE was already tracked on an *older* version, both threads get cross-linked with a loud "the rebuild did NOT clear this" warning.
    
3.  **agentTask** — hand the case to AWS DevOps Agent as an investigation.
    
4.  **wait / checkAgent** — poll every 2 minutes (investigations take ~4–5).
    
5.  **harvest** — pull the agent's analysis from its journal and post it onto the issue as a collapsible section.
    

Why Step Functions and not one Lambda? The agent investigation is a minutes-long async job — exactly what Standard workflows exist for. Bonus: the console graph doubles as live documentation. You can literally watch a CVE travel detect → issue → agent → analysis → remediation.

### What the agent adds — and how much that depends on what you give it

I ran the same findings through the agent in three configurations, plus a direct Bedrock call as a control. The differences were not subtle.

**Bundle-only** (no capabilities attached — my first setup): the agent reasons over the context bundle you hand it. Useful — for our sqlite CVE it correctly weighed exploitability *in a CI-runner context* and spotted the backported fix — but it's an essay over your facts, and a direct Bedrock call (46 s, ~$0.04) produces a comparable essay 8× faster.

**With a GitLab capability provider** (register in the console, then `associate-service` — the association grants *space-wide* repo access, no per-task wiring): the analysis changed class. It cited exact lines ("line 26 already tracks `latest`", "`--max-builds 1`, line 82"), read our docs and echoed our own design rules back, and found a repo-grounded exploitability argument nothing else produced — our runner uses gRPC as a *client only*, no `--session-server`, so the vulnerable listener path never activates. The Bedrock control gave the same *shape* of advice but invented a plausible-looking Dockerfile pattern that doesn't exist in the repo.

**Plus an AWS cloud source** (console → agent space → Capabilities → add primary source; the auto-created role's policy is effectively read-only — I audited all 880 actions): the agent stopped trusting my bundle entirely. It queried Inspector itself (found the CVE ACTIVE on all four current mirror images, and understood that the 11 CLOSED findings were candidates our keep-2 retention had reaped), checked ECR's *current* tags — catching state newer than my pre-computed context — and opened its report with a table of which sources it accessed and which it couldn't.

What the agent still does **not** do is write code or open MRs — it advises; the pipeline stays the author and the human stays the approver. And two preview gaps to know about: it has no GitLab issue/MR-API tool (repo files, branches, diffs and git log only), and no Lambda MicroVMs API in its toolset — it told me both, unprompted.

Practical notes if you wire this up yourself:

*   The service's IAM prefix is `aidevops`, not the CLI's `devops-agent` namespace. You'll discover this via AccessDenied, like I did.
    
*   The agent emits rich Markdown **with its own code fences**. Render it natively — wrap it in a code block and its first internal fence breaks your formatting in half.
    
*   Judge the agent on its *tooled* configuration. Bundle-only, it's an expensive prompt; with repo + cloud access, it earns the extra minutes.
    

## The remediation MR: one bump to fix them all

A `remediate` step runs after harvest and turns facts into a change. The key design choice: **the patch is generated from Inspector's facts, not parsed from the agent's prose.** `fixedInVersion` says deterministically whether a fix exists; the provenance oracle says where it lives. Facts drive automation; prose informs humans.

Nine CVEs were open when the step first ran. It opened **one** merge request, not nine — every remediable CVE appends itself to a shared, ready-for-review MR. The change is a single managed Dockerfile block:

```dockerfile
# --- security refresh (managed by the finding workflow) ---
ARG SEC_REFRESH=2026-07-22
RUN echo "security refresh ${SEC_REFRESH}" \
    && echo latest > /etc/dnf/vars/releasever \
    && dnf -y upgrade --setopt=install_weak_deps=0 \
    && dnf clean all
```

Bumping `SEC_REFRESH` invalidates every layer below it, so one rebuild refreshes all OS packages *and* re-fetches the gitlab-runner and AWS CLI binaries. CVEs with no published fix don't generate MR noise — they get a "waiting on upstream" note on their issue instead.

Three properties I'd defend in review:

*   **The MR proves itself.** Its branch pipeline builds the actual refreshed image and runs the scan gate on it — the reviewer sees scan results for the fix *before* merging.
    
*   **A human merge is the only manual step.** Nothing self-approves.
    
*   **The MR closes nothing.** Issues close exclusively via a *resolution verifier* that asks Inspector whether the CVE is still present on the newest image. The agent advises, the pipeline proposes, the human merges — and **Inspector referees**. A fix that doesn't actually fix gets caught and cross-linked, not silently celebrated.
    

## The gotcha that justified the whole verifier: AL2023's frozen repos

First run of the remediation build: gate green… and **8 of 9 CVEs still present**. The `dnf upgrade` output said it plainly:

```plaintext
Downloading metadata...
Nothing to do.
Complete.
```

Amazon Linux 2023 pins every image to a **point-release snapshot** via `/etc/dnf/vars/releasever`. `dnf upgrade` faithfully upgrades against that frozen snapshot — which by definition contains no newer packages. Worse, the later `dnf install` layers resolve against the same snapshot, so they *install* the exact vulnerable versions Inspector just flagged. If your container patch story is "we rebuild the image regularly", AL2023 quietly made it a no-op.

The fix is one line — `echo latest > /etc/dnf/vars/releasever` before the upgrade — which un-freezes that upgrade *and* every later install. The re-run took the image from **23 distinct HIGH CVEs to 0**.

Two lessons from this failure:

1.  **Build-time gates can't catch it.** A gate checks a policy ("no Criticals"), not an intent ("this build was supposed to remove CVE-X"). Only the Inspector-refereed verification noticed the remediation was a placebo.
    
2.  **The one CVE that *did* clear proved the point.** A Go stdlib CVE vanished because the refresh re-downloaded the gitlab-runner binary — one path worked while the other silently didn't. Partial success is the most dangerous kind. Verify per-CVE, not per-build.
    

## Epilogue: what happened when the human merged

The merge landed and the loop ran end to end with no hands on it: new MicroVM image version cut, mirror retagged, Inspector re-scanned, and the verifier closed **all nine issues** with per-CVE "verified resolved by Inspector" receipts.

Then it did something better. It caught **two findings on the new image that the build-time gate had waved through** — a Go stdlib CVE and a gRPC module advisory, both embedded in pre-built binaries we download. Trivy and Inspector genuinely disagree here; Inspector's deeper Go-binary analysis found them. The workflow filed both, attached agent investigations, and — after I taught it the difference — correctly *declined* to open an MR: `dnf upgrade` cannot patch a Go module inside someone else's binary. Both issues now sit open with a "waiting on an upstream rebuild" note, and the verifier will close them the moment a refreshed binary scans clean.

Two closing lessons: **scanner disagreement is a feature** if your loop treats one scanner as the referee instead of averaging opinions — and a remediation bot must know what it *cannot* fix, or it becomes a generator of confident, empty merge requests.

## What it actually costs to run

Real numbers, not free-tier hand-waving. Sources: my account's Cost Explorer (the demo has been running the loop live) plus eu-west-1 list prices for the services still inside a trial or free tier on my account.

| Component | Price basis (eu-west-1) | Monthly at demo scale |
| --- | --- | --- |
| Inspector continuous ECR scanning | $0.09 per image per month | ~4 scanned images (2 versions + cache tag + base) ≈ **$0.36** |
| ECR storage | $0.10 per GB-month | 2.5 GB across both mirrors ≈ **$0.25** |
| MicroVM image snapshot storage | measured (bills as Lambda `MicroVM-Snapshot-Storage-GB-Hour`) | **~$1.47** for the keep-2 versions |
| Secrets Manager | $0.40 per secret | 2 secrets = **$0.80** |
| Step Functions (Standard) | $0.025 per 1k transitions | ~12 transitions per finding → **$0.03 per 100 findings** |
| Workflow Lambdas + 30-min verifier | per-invocation | **< $0.01** |
| EventBridge (AWS events) + SNS email | free at this usage | $0 |
| Trivy gate | open source | $0 |
| GitLab SaaS ARM64 minutes (pipeline) | $10 per 1,000 min | ~9 min per push/merge ≈ **$0.09 per run** |
| AWS DevOps Agent | preview — no charges billed on my account yet | TBD at GA; watch this line |

**Steady state: ≈ $3/month**, dominated by MicroVM snapshot storage and Inspector's per-image fee. The marginal cost of processing a finding — issue, agent hand-off, MR update, verification — is **under one cent**. Continuous CVE coverage for an unscannable image class, for less than a coffee a month.

(Two honesty notes: Inspector was inside its 15-day trial on my account, so its line is list price rather than a bill I've paid; and DevOps Agent is in preview — nothing has hit Cost Explorer for it yet, so treat that row as open until GA pricing lands.)

## Numbers

|  |  |
| --- | --- |
| Finding → issue + agent analysis | ~4–5 min bundle-only; ~6–7 min with live repo + AWS verification |
| Same analysis via direct Bedrock (control) | 46 s, ~$0.04 — fast, but ungrounded in the repo |
| HIGH CVEs before → after one merged MR | **23 → 0** |
| MRs opened for 9 CVEs | 1 (shared, ready-for-review) |
| Manual steps in the loop | 1 (the merge) |
| Issues closed without Inspector's confirmation | 0, by design |
| Scan-twin build, cold vs layer-cached | 214 s → 121 s (all layers CACHED) |
| Running cost | ≈ $3/month + <1¢ per finding (see cost table) |

## Steal this

*   **Scan-twin**: if any of your images are built where scanners can't see them (MicroVMs, on-prem bakeries, appliance firmware), build the identical Dockerfile into ECR with version-parity tags and let continuous scanning do its job. The pattern costs one pipeline job.
    
*   **Provenance oracle**: mirror your unmodified base image and diff findings against it. One API call ends every "is this the base's fault?" debate.
    
*   **Facts drive automation, prose informs humans**: generate patches from `fixedInVersion` and digests; attach the agent's analysis for the reviewer.
    
*   **Verify per-CVE with the scanner as referee**: never let the thing that proposed a fix also declare it successful.
    
*   **Give your agent tools before judging it**: a capability-provider association turned essays into file:line-grounded, live-verified analyses. The same model with and without tooling is two different products.
    
*   **Check your AL2023 images today**: if plain `dnf upgrade` is your patch mechanism, grep your build logs for "Nothing to do". You might be rebuilding vulnerabilities with a green pipeline.
    

*Repo layout, state machine definition, and the full finding-workflow Lambda are in the companion repo from Part 1.*