Continuous PCI DSS Segmentation Testing on AWS — with VPC Reachability Analyzer and a Bedrock-Powered QSA Assistant

How I turned a quarterly, spreadsheet-driven compliance chore into an event-driven serverless pipeline that tests segmentation within minutes of every relevant network change — and how, along the way, I discovered my passing tests were lying to me.

---

If you've ever been responsible for PCI DSS compliance in a cloud environment, you know the segmentation testing ritual. Every six months (or quarter, if you're cautious), someone has to prove that the systems outside your Cardholder Data Environment (CDE) genuinely cannot reach the systems inside it. Traditionally that means scheduling engineers, running nmap scans from jump hosts, screenshotting terminal output, and assembling a document your QSA will accept.

It's slow, it's manual, and worst of all, it's a point-in-time snapshot of a network that changes every week.

PCI DSS v4.0.1 Requirement 11.4.5 requires penetration testing to validate segmentation controls, and 11.4.6 tightens the cadence for service providers to at least every six months and after any changes to segmentation controls or methods. That last clause is the interesting one: your network changes far more often than twice a year. A security group rule added during an incident, a route table tweak, a new peering attachment — any of these can silently break your segmentation, and you won't know until the next scheduled test.

So I built an automated segmentation testing pipeline on AWS — first quarterly, then fully continuous. This post walks through the architecture, the design decisions, and two hard-won lessons: why cross-account event plumbing is trickier than it looks, and why hardcoded resource IDs will eventually make your compliance evidence worse than useless.

Scope note: automated reachability analysis validates segmentation controls and produces excellent recurring evidence, but it complements rather than replaces the human-led segmentation penetration test your QSA expects. Where it shines is the "after any changes" clause and everything between formal tests.

Why VPC Reachability Analyzer?

There are broadly three ways to validate network segmentation on AWS:

1. Active probing — run scanners (nmap, custom probes) from instances in non-CDE networks against CDE targets. Authentic, but it requires agents or instances in every source network, credentials to run them, and it generates real traffic against production databases.

2. Config review — manually (or with scripts) review security groups, NACLs, and route tables. Cheap, but it's easy to miss the composition of controls. A security group might allow traffic that a route table makes impossible, or vice versa. Reviewing controls in isolation doesn't tell you whether a path exists.

3. Formal reachability analysis — VPC Reachability Analyzer builds a model of your network configuration (security groups, NACLs, route tables, gateways, peering, Transit Gateway, load balancers) and mathematically determines whether a path exists between a source and destination. No packets are sent. No agents are installed. It analyzes the entire composition of controls in one shot.

I chose option 3, and I'd argue it's the best fit for recurring segmentation validation:

• No test infrastructure in the CDE. You're not installing scanners inside your most sensitive network segment just to test it.

• Deterministic and explainable. When a path is (or isn't) found, the analyzer tells you exactly which component allows or blocks the traffic — down to the specific security group rule. That's remediation guidance built into the test result.

• Cross-account, org-wide. With AWS Organizations trusted access and a delegated administrator account, you can analyze paths that span accounts — essential when your CDE lives in a dedicated production account and your test/UAT workloads live elsewhere.

• It tests the config plane, which is where cloud segmentation actually lives. In a VPC there is no switch misconfiguration or rogue cable; if the configuration says there's no path, there is no path.

The trade-off is honest to state: Reachability Analyzer validates network-layer configuration. It won't catch an application-layer proxy that happily forwards traffic into the CDE, and it doesn't exercise host-based firewalls. That's why it complements the annual human pentest rather than replacing it.

The architecture

The whole pipeline is serverless, deployed with AWS CDK, and runs in a dedicated security account that is the Reachability Analyzer delegated administrator.

Test cases are data, not code. Each test is a JSON document stored in S3 and version-controlled in Git. Adding a new segmentation boundary to the suite is a merge request, not a change to a pentest statement of work — and the MR history itself becomes compliance evidence.

Four test categories. cross_account_isolation (non-CDE accounts can't reach CDE resources), inbound_isolation (the internet can't reach CDE databases or SSH/RDP), outbound_isolation (the CDE can't initiate connections outward into non-CDE networks), and positive_validation — traffic that should work, like the payment app reaching its processor over 443.

That last category is easy to overlook and absolutely essential. A suite made entirely of "expect NOT_REACHABLE" assertions will happily pass while your analyzer is misconfigured or your entire environment is broken. Positive tests prove the harness itself works. Keep reading for the story of how even that wasn't enough.

Everything is evidence. Raw analyzer output goes to versioned S3 (organized by quarter and trigger type) with lifecycle policies down to Glacier. Per-test results go to DynamoDB for quarter-over-quarter trending. When the auditor asks "show me your Q2 segmentation test," the answer is a timestamped S3 URL, not an email thread.

The AI layer: a Bedrock powered QSA assistant

Raw Reachability Analyzer output is verbose JSON: path components, explanation codes, ARNs. Perfectly precise, and nobody outside the network team wants to read it. Historically, a human turned this into the report leadership and auditors actually consume. That translation step is exactly the kind of structured, factual summarization LLMs are good at.

So the pipeline sends aggregated results — plus the previous quarter's — to Claude on Amazon Bedrock, with a system prompt framing it as a PCI DSS QSA specializing in cloud segmentation. It returns structured JSON: an executive summary, per-finding risk and remediation, quarter-over-quarter trend analysis, and an assessment of whether the suite covers all segmentation boundaries. For change-triggered runs, the prompt also includes what changed — the CloudTrail events that caused the run — so a failure report can connect the regression to the change that introduced it.

Three guardrails made this trustworthy enough for a compliance workflow:

1. Low temperature and a strict JSON schema. This is analysis of structured facts, not creative writing.

2. An explicit "do NOT invent or assume information not present in the test results" instruction, with graceful fallback to a plain statistical summary if the model returns anything unparseable.

3. A standing rule, stated in the report itself: AI-generated analysis supplements but never replaces qualified human review. The model drafts; a human signs.

Going continuous: the part nobody warns you about

Quarterly already exceeds the PCI cadence. The reason to go further is that the interesting requirement is not the cadence — it's the "after any changes" clause. Your segmentation posture doesn't degrade on a schedule; it degrades the moment someone adds a security group rule at 4pm on a Tuesday.

Three lessons from actually building it:

Lesson 1: CloudTrail events don't leave their account

My first design had a single EventBridge rule in the security account matching AuthorizeSecurityGroupIngress and friends. It would have never fired. CloudTrail management events surface on the event bus of the account and region where the API call happened — a rule in your security account is deaf to changes made in production.

The fix: a small forwarder rule in every in-scope account that sends matched events to a central event bus in the security account. If you run Landing Zone Accelerator (or StackSets), that's one CloudFormation template in your customizations config, deployed org-wide in a single pipeline run. The central bus policy allows events:PutEvents from the member accounts, and the forwarders carry an IAM role scoped to exactly that.

Lesson 2: Scope and debounce, or drown

Triggering a test run on every SG change in every account buys you noise, not assurance. Two scoping decisions matter:

• Security group / NACL / route changes only trigger from the CDE account. If your accounts have no peering or Transit Gateway between them, a change in a dev account cannot open a path into the CDE by itself — the CDE-side controls are doing the work.

• Bridging events trigger from anywhere. CreateVpcPeeringConnection and Transit Gateway attachment events are what could connect isolated accounts, so those are never out of scope.

Then debounce: deployments change several rules in seconds. Matched events land in an SQS queue with a 5-minute delivery delay, and the trigger Lambda consumes them with a 5-minute batching window — so one deployment burst produces one test run carrying the full list of changes, not fifteen runs carrying one each.

And control the noise on the way out: a Choice state routes change-triggered runs that pass straight to a silent success — evidence stored in S3 and DynamoDB, no AI report, no email. Failures and errors always get the full report and notification. Alert fatigue kills continuous controls faster than cost does.

In the live fire drill, the end-to-end latency from AuthorizeSecurityGroupEgress in the CDE account to a completed 11-test run was about twelve minutes — CloudTrail delivery, the two debounce windows, and the analyses themselves. The run stored evidence recording exactly which principal changed which security group at which timestamp, and sent nothing to anyone, because everything passed. Compare that to a ninety-day blind spot.

One data-model prerequisite: my results table was keyed on (quarter, test_id) — perfect for quarterly, silently overwriting for anything more frequent. Change-triggered runs now write under run-scoped keys, so every run in a quarter is retained. Check yours before you flip the switch.

Lesson 3: My passing tests were lying to me

This is the one I want every security engineer to take home.

While verifying the continuous setup, I ran a read-only check on the resources my test cases pointed at. Two of the hardcoded ENI IDs — an RDS network interface and an EKS worker node interface — no longer existed. The RDS ENI had churned through host replacement; the EKS cluster had been migrated to Auto Mode, which removed the standalone worker nodes entirely.

Here's the ugly part: those tests were still passing. Reachability Analyzer correctly reported "no path found" to a destination that didn't exist, my pipeline scored NOT_REACHABLE as expected, and the compliance report was green. The tests weren't failing — they were asserting nothing, which is far worse. A false ERROR gets investigated; a false PASS gets filed as evidence.

The fix is to stop pinning resources by ID and resolve them at run time. Test case endpoints now carry a resolve block instead of an ARN:

"destination": {
  "type": "network-interface",
  "resolve": {
    "account": "111111111111",
    "region": "eu-west-1",
    "filters": [
      { "Name": "description", "Values": ["RDSNetworkInterface"] },
      { "Name": "group-name", "Values": ["prod-db-security-group*"] },
      { "Name": "availability-zone", "Values": ["eu-west-1b"] }
    ]
  },
  "label": "Prod CDE RDS ENI (eu-west-1b)"
}

The filters pass straight through to the EC2 Describe APIs, so anything the API supports works — which matters for resources you can't tag, like RDS ENIs (identifiable by description) or EKS Auto Mode cluster interfaces. Instances resolve by tags that already exist for operational reasons, always picking the newest running match so autoscaling churn is invisible. Pinning the availability zone keeps "test both RDS ENIs" pairs deterministic. Cross-account lookups use a read-only role deployed alongside the event forwarders.

And the crucial failure mode is inverted: if a filter matches nothing, the test surfaces as a loud ERROR — which, on a change-triggered run, means a notification — instead of a quiet green checkmark against a ghost.

Continuous testing made this non-negotiable. Quarterly, a rotted ID is an annoyance someone eventually notices. Continuously, it's either a constant stream of false errors (if you're lucky) or a permanent false pass (if you're not).

Gotchas, condensed

• Cross-account setup is a one-time dance. Reachability Analyzer's organization support requires enabling trusted access and registering a delegated administrator from the management account, before anything else works.

• AdditionalAccounts rejects the source and destination accounts. Parse the ARNs and filter them out before starting a cross-account analysis.

• Validate reused analysis paths. Reusing Network Insights Paths across runs saves time, but once resources resolve dynamically, a path's source/destination can silently diverge from the test definition. Compare the path's actual source, destination, protocol, and port against the test case; delete and recreate on mismatch.

• Watch the small print in cross-account EventBridge. The bus policy's StatementId and the statement's Sid must be identical strings — a mismatch fails the deployment with a less-than-obvious error.

• Cost is a non-issue if scoped. An analysis costs ~\(0.10, so a passing change-triggered run over an 11-test suite is about a dollar (the AI step is skipped). The full quarterly run with the Bedrock report lands around \)4. The debounce bounds the volume to roughly one run per change burst.

Where this goes next

Enumerated source→destination pairs will always undertest — you can only assert paths you thought to write down. AWS Network Access Analyzer (Reachability Analyzer's scope-based sibling) inverts the model: define a scope like "nothing outside the CDE may reach anything inside it" and let AWS find any violating path, including ones between resources you never enumerated. Pairing point-to-point regression tests with scope-based discovery is, I think, the end state for cloud segmentation assurance.

Closing thoughts

Segmentation testing earned its painful reputation in data centers, where proving a negative genuinely required sending packets. In the cloud, your network is configuration, and configuration can be formally analyzed — continuously, for pocket change, with an evidence trail that timestamps every relevant change and the test that followed it.

The pipeline described here turns a twice-a-year scramble into: a merge request to add a test, a debounced event to trigger one, an S3 URL to evidence one, and a notification only when something is actually wrong. Your QSA still does the annual pentest. But between visits, you're no longer hoping the segmentation holds — you're watching it.

And check your test targets. If any of them point at resources by ID, some of your green checkmarks may already be ghosts.

If you're building something similar or want to compare notes on continuous compliance on AWS, I'd love to hear from you.

---