home / writing / infrastructure cost audit

Infrastructure cost audit: why your AWS bill is 40% higher than it should be

Cloud waste is never one big mistake. It is thirty small ones that each looked reasonable on the day, compounding monthly. Here is the audit I run, in the order I run it, with the commands, so you can do it yourself before you pay anyone to do it for you.

Nobody wakes up and decides to overspend on infrastructure. What happens is that an instance gets sized generously "to be safe" during a launch, a staging environment gets cloned from production, an engineer leaves, a project ends, a NAT Gateway goes into each availability zone because the tutorial said so, and none of it is ever revisited, because the bill arrives as one number and nobody owns it.

On accounts that have never been audited, I typically find 25–45%. The 40% in the title is not a marketing number, it is roughly the median of what comes out when nobody has looked. The work is not clever; it is systematic. This is the system.

Before you touch anything: you need read access and Cost Explorer enabled. Every command below is read-only. Do not delete anything in the first pass; build the list first, then decide, then act with someone who knows what the resource was for. The fastest way to turn a cost audit into an incident is to delete a volume that looked orphaned.

Step 0: make the bill legible

You cannot audit a number. Break it into services first: in almost every account, three services are 80% of the bill, and that tells you where to spend your attention.

aws ce get-cost-and-usage \
  --time-period Start=2026-05-01,End=2026-08-01 \
  --granularity MONTHLY \
  --metrics UnblendedCost \
  --group-by Type=DIMENSION,Key=SERVICE \
  --output table

Then break the top service down by usage type, which is where the actual mechanism hides: "EC2-Other" is not a service, it is a drawer full of data transfer, EBS and NAT charges:

aws ce get-cost-and-usage \
  --time-period Start=2026-07-01,End=2026-08-01 \
  --granularity MONTHLY \
  --metrics UnblendedCost \
  --group-by Type=DIMENSION,Key=USAGE_TYPE \
  --filter '{"Dimensions":{"Key":"SERVICE",
             "Values":["EC2 - Other"]}}' \
  --output table

Two things to note while you are here. First, look at the trend, not the month: a line that grows 8% every month with flat traffic is a leak, and leaks are more valuable to find than one-off waste. Second, if your resources are not tagged, add tagging to the end of this project; you cannot attribute cost to teams or products without it, and unattributed cost never gets owned.

Step 1: idle and oversized compute

This is usually the single largest finding. Instances are sized on the day of launch by guesswork, and guesswork is conservative.

Start with AWS's own analysis, since Compute Optimizer does the statistical work for you and is free:

aws compute-optimizer get-ec2-instance-recommendations \
  --query 'instanceRecommendations[?finding!=`Optimized`].
           [instanceArn,currentInstanceType,finding,
            recommendationOptions[0].instanceType]' \
  --output table

Then verify with the actual metric, because you know your workload and it does not:

aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --start-time 2026-07-01T00:00:00Z \
  --end-time   2026-08-01T00:00:00Z \
  --period 86400 --statistics Average Maximum \
  --output table

Read it like this. Sustained average CPU under 10% with a maximum under 40% means you are two sizes too big. Under 5% with no meaningful peak usually means the instance is doing nothing at all; find out what it was for before you kill it, but be suspicious.

Two caveats that separate a useful audit from a reckless one. CPU is not the only constraint: check memory (which needs the CloudWatch agent, and if you do not have it, install it before right-sizing memory-bound workloads) and check network and disk on I/O-heavy services. And burstable instances need credit analysis: a t-family instance sitting at 8% CPU may be perfectly sized, or may be silently exhausting CPU credits under load. Check CPUCreditBalance before touching those.

While you are here, check for old-generation instance families. Each generation is typically cheaper per unit of performance than the last, and moving from an older family to the current equivalent is frequently 10–20% for the price of a restart.

Step 2: environments running 24/7 for no reason

Development and staging environments are used, at most, during working hours. Running them 168 hours a week to serve 50 hours of use is the highest-ratio waste in cloud, and it is trivially fixable with a scheduler.

hours_used_per_week      = 50   (9am-7pm, Sun-Thu)
hours_billed_per_week    = 168
theoretical_saving       = 70%

Find them by tag, or by the naming convention your team actually uses:

aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running" \
  --query 'Reservations[].Instances[].
           [InstanceId,InstanceType,
            Tags[?Key==`Name`].Value|[0],
            Tags[?Key==`Environment`].Value|[0]]' \
  --output table

Implement with EventBridge Scheduler plus a small Lambda, or AWS Instance Scheduler if you want it packaged. Do the same for non-production RDS instances, which are usually a bigger line than the EC2 next to them. Two rules from experience: put an override tag on anything that must stay up, and announce it before the first shutdown, or you will spend the saving on the incident.

And look for whole environments belonging to finished projects. In almost every audit there is at least one, still running, still paying, months after the last commit.

Step 3: storage nobody deleted

Individually small, collectively significant, and completely safe to find.

Unattached EBS volumes

aws ec2 describe-volumes \
  --filters Name=status,Values=available \
  --query 'Volumes[].[VolumeId,Size,VolumeType,CreateTime]' \
  --output table

These are volumes detached from terminated instances that nobody deleted. You are paying full price for storage attached to nothing. Snapshot anything you are unsure about, then delete.

gp2 volumes that should be gp3

aws ec2 describe-volumes \
  --filters Name=volume-type,Values=gp2 \
  --query 'Volumes[].[VolumeId,Size]' --output table

gp3 is roughly 20% cheaper per gigabyte than gp2 and lets you provision IOPS independently of size. The migration is a live modification with no downtime. This is the closest thing to free money in the whole audit.

Ancient snapshots

aws ec2 describe-snapshots --owner-ids self \
  --query 'sort_by(Snapshots,&StartTime)[:40].
           [SnapshotId,VolumeSize,StartTime,Description]' \
  --output table

Manual snapshots taken "just before that risky deploy" in 2023 are still billing. Put a lifecycle policy on backups (Data Lifecycle Manager) so this never accumulates again.

S3 without lifecycle rules

Two specific wins. First, incomplete multipart uploads: failed uploads leave parts that are invisible in the console object list and billed forever. Add an AbortIncompleteMultipartUpload rule at 7 days to every bucket, today. Second, storage class: logs and backups older than 30 days rarely need Standard. Intelligent-Tiering is the low-effort option if access patterns are unpredictable.

aws s3api get-bucket-lifecycle-configuration \
  --bucket my-bucket 2>/dev/null \
  || echo "NO LIFECYCLE RULES"

Unassociated Elastic IPs

aws ec2 describe-addresses \
  --query 'Addresses[?AssociationId==null].
           [PublicIp,AllocationId]' --output table

Small per-IP, but they are pure waste and take a minute to clear.

Step 4: the data transfer bill

This is where the surprises are, because data transfer is charged by mechanism rather than by anything visible in your architecture diagram.

NAT Gateway

Each NAT Gateway costs roughly $32–35 per month in hourly charges depending on region, plus a per-gigabyte processing charge on everything that flows through it. Deploy one per availability zone across three AZs in two environments and you are at six gateways before a single byte moves.

aws ec2 describe-nat-gateways \
  --filter Name=state,Values=available \
  --query 'NatGateways[].[NatGatewayId,VpcId,SubnetId]' \
  --output table

The three fixes, in order of value:

  1. Add an S3 gateway VPC endpoint. It is free, and it takes all your private-subnet S3 traffic off the NAT Gateway. If you write logs or backups to S3 from private subnets, this alone can be a large fraction of your NAT processing charges. Same for DynamoDB.
  2. Consider interface endpoints for other AWS services you call heavily from private subnets. These are not free (they carry an hourly and per-GB charge), so compare against what the NAT is processing rather than adding them reflexively.
  3. Right-size your AZ redundancy in non-production. Staging rarely needs a NAT Gateway per AZ. Production usually does; that is a reliability decision, not a cost one.

Cross-AZ traffic

Traffic between availability zones is billed in both directions, typically a cent per gigabyte each way. It is invisible until you look, and chatty services (an application in one AZ talking constantly to a cache or database in another) generate a lot of it. Check where your instances and their dependencies actually sit, and keep hot paths in-AZ where your availability requirements allow.

Egress to the internet

If you serve images, video or large downloads directly from EC2 or S3, put CloudFront in front. Cached delivery is cheaper per gigabyte than origin egress and faster for users. Two wins for one change.

Step 5: databases

RDS is usually the second-biggest line after EC2, and it is where over-provisioning is most defensible and therefore most persistent: nobody wants to be the person who shrank the database.

Check the same way, with data:

aws rds describe-db-instances \
  --query 'DBInstances[].[DBInstanceIdentifier,DBInstanceClass,
           Engine,MultiAZ,AllocatedStorage,StorageType]' \
  --output table

Then work through:

  • Multi-AZ in non-production. It roughly doubles instance cost for a standby you do not need in staging. Production keeps it.
  • Over-allocated storage. RDS storage cannot be shrunk in place, so it accumulates. If you allocated 1 TB and use 80 GB, the fix is a migration, worth it above a certain size, painful below it.
  • Provisioned IOPS you do not use. io1/io2 volumes provisioned for a load test two years ago are a common find. Check actual IOPS against provisioned.
  • Snapshots of deleted instances. Final snapshots persist after the instance is gone, indefinitely, billing.
  • Idle instances entirely. Zero connections for thirty days means somebody's abandoned project. Confirm, snapshot, delete.

And before you scale the instance up because the database is slow: check the queries. A missing index is a free fix that people routinely pay for with a larger instance class, every month, forever. I have seen a two-size upgrade reverted by adding one index.

Step 6: the small stuff that adds up

Thing Why it costs Fix
CloudWatch log groups with no retention Default is never expire, so logs accumulate forever Set retention on every group; 30–90 days for most
Load balancers with no healthy targets Billed hourly regardless of traffic Delete, or consolidate with host-based routing
Idle Elastic IPs, unused Route 53 health checks Small per-unit charges nobody notices Sweep quarterly
Over-provisioned Lambda memory Priced on memory × duration Tune with Compute Optimizer; sometimes more memory is cheaper
ECR repositories with no lifecycle policy Every CI build pushes an image and none are deleted Lifecycle rule: keep last N images
Test and sandbox accounts Invisible in a single-account view Audit every account in the organisation, not just the main one

Find the log retention offenders in one command:

aws logs describe-log-groups \
  --query 'logGroups[?retentionInDays==null].
           [logGroupName,storedBytes]' --output table

Step 7: only now, commitment discounts

Savings Plans and Reserved Instances give meaningful discounts for committing to a level of spend over one or three years. They are also the reason a lot of cost work goes wrong, because they are the easiest thing to do and people do them first.

Buying a commitment before right-sizing locks you into paying for the capacity you were about to delete. Right-size, run at the new baseline for two to four weeks, then commit to that.

When you do commit:

  • Commit to your floor, not your average. Uncommitted usage is on-demand; over-committed usage is money burned.
  • Prefer Compute Savings Plans over EC2 Instance Savings Plans unless your instance families are genuinely fixed; the flexibility is usually worth the smaller discount.
  • Start with a one-year term. Three-year commitments assume you can predict your architecture in 2029.
  • Re-evaluate quarterly, and stagger expiries so you are not renegotiating everything in one month.
aws ce get-savings-plans-purchase-recommendation \
  --savings-plans-type COMPUTE_SP \
  --term-in-years ONE_YEAR \
  --payment-option NO_UPFRONT \
  --lookback-period-in-days SIXTY_DAYS

Guardrails so it does not come back

An audit is a one-off. Without guardrails you will run it again in eighteen months and find the same 40%.

  1. Tagging policy, enforced. Owner, environment, project: required on creation, enforced through Service Control Policies or IaC review. Untagged resources are how waste hides.
  2. Budgets with alerts per account and per team, with a threshold people actually receive.
  3. Cost Anomaly Detection switched on. It catches the step change on the day it happens rather than at month end.
  4. A monthly review with an owner. Fifteen minutes, a named person, top movers only. This is the control that actually works; the tooling just supports it.
  5. Cost in the pull request. Infrastructure changes should state their cost impact the way they state their security impact. Once teams see the number, the behaviour changes without anyone policing it.

What to do in what order

Order Action Effort Typical impact
1 Delete orphaned storage, unattached volumes, idle IPs Hours 2–5%
2 gp2 → gp3, set log retention, S3 lifecycle rules Hours 2–6%
3 Schedule non-production environments 1–2 days 5–15%
4 Right-size compute and databases Days 10–25%
5 S3 gateway endpoint, NAT and cross-AZ review Days 3–10%
6 Commitment discounts on the new baseline Hours 10–20% of remaining
7 Guardrails: tagging, budgets, anomaly detection Days Prevents recurrence

One closing warning, because cost work has a failure mode. It is possible to optimise yourself into an outage: removing the standby, shrinking below your real peak, collapsing redundancy that existed for a reason. Every change in this list should be evaluated against what it does to your availability, and the honest answer is sometimes "this costs more and it stays". A cheaper bill you cannot operate is not a win.

Want someone to run this on your account?

I do infrastructure cost and reliability reviews as a short fixed engagement: a prioritised findings list with estimated savings and the risk of each change. Email contact@kerolosxgad.com.

Notes

Commands are AWS CLI v2 and read-only. Prices referenced are indicative and vary by region; check current AWS pricing for yours. Savings percentages reflect what audits typically surface on accounts without prior cost discipline; your mileage depends entirely on how long nobody has been looking.

← all writing · next: security requirements for Egyptian fintech →