The hidden causes of a high AWS bill

A high AWS bill usually has several owners, even when one line item appears to explain the increase. The visible spike may be EC2, but the cause can sit behind it: a deployment doubled cross-zone traffic, a failed upload left parts in S3, or a log change multiplied ingestion. Treat the invoice as a ledger of engineering decisions, not a single number to negotiate.

I start with the Cost and Usage Report (CUR), or its current Data Exports equivalent, delivered to S3 and queried through Athena. Enable resource IDs, hourly granularity when the investigation needs it, and cost-allocation tags. The AWS Data Exports line-item documentation matters here: line_item_unblended_cost records the cost on usage line items, while credits, discounts, refunds, taxes, and commitment effects have their own line-item types. Mixing them can make a resource look cheaper or more expensive than its actual consumption.

Use a closed billing period first, then the current month. Replace the table and dates below. This query builds the service and usage-type ledger that every later finding must reconcile with.

SELECT
  line_item_product_code AS service,
  line_item_usage_type AS usage_type,
  line_item_operation AS operation,
  SUM(line_item_usage_amount) AS usage,
  SUM(line_item_unblended_cost) AS cost
FROM cur_database.cur_table
WHERE line_item_usage_start_date >= TIMESTAMP '2026-07-01'
  AND line_item_usage_start_date <  TIMESTAMP '2026-08-01'
  AND line_item_line_item_type = 'Usage'
GROUP BY 1, 2, 3
HAVING SUM(line_item_unblended_cost) > 0
ORDER BY cost DESC;

Export the result and keep it beside the bill. If the sum does not reconcile with the bill's usage subtotal, fix the scope, payer account, currency, or line-item filter before hunting resources. A bad denominator can waste a week. Run every query across every linked account and active Region; the console's default Region is a frequent source of false relief.

1. Idle and oversized EC2 instances keep charging normally

An EC2 instance does not become cheap because nobody remembers it. Development hosts left running overnight, replacement nodes that survived a migration, and production instances sized for an old peak all produce legitimate usage, so billing has no reason to flag them. Cost Explorer shows the spend, but CloudWatch and Compute Optimizer supply the utilization evidence needed before a resize or shutdown.

First rank instance resources by cost. Resource IDs appear only when the export includes resource-level detail, and some shared charges will still lack an ID.

SELECT
  line_item_resource_id AS instance_id,
  product_instance_type AS instance_type,
  line_item_availability_zone AS az,
  SUM(line_item_usage_amount) AS hours,
  SUM(line_item_unblended_cost) AS cost
FROM cur_database.cur_table
WHERE line_item_usage_start_date >= TIMESTAMP '2026-07-01'
  AND line_item_usage_start_date <  TIMESTAMP '2026-08-01'
  AND line_item_product_code = 'AmazonEC2'
  AND line_item_usage_type LIKE '%BoxUsage%'
  AND line_item_line_item_type = 'Usage'
GROUP BY 1, 2, 3
ORDER BY cost DESC;

Take the top instances into CloudWatch and compare at least two weeks of CPU, network, disk, and memory. EC2 does not publish guest memory without the CloudWatch agent, so a low CPU chart alone cannot prove an instance is idle. A database cache, license server, or memory-heavy worker can have quiet CPU and still need its shape. Compute Optimizer recommendations are useful evidence, but review the workload window and expected peak before accepting one.

Fix obvious nonproduction schedules before resizing production. Stop instances that can tolerate a cold start, remove abandoned Auto Scaling capacity, then rightsize sustained workloads. Do not terminate anything until an owner confirms the data path, replacement, and rollback. The expensive mistake is deleting an instance while its separate EBS volume, snapshot chain, IP address, and monitoring remain behind.

2. Unattached EBS volumes survive the instances they served

EBS bills for provisioned storage while a volume is in the available state. That state means unattached, not free, empty, or safe to delete. I have seen migration volumes sit for months because a cautious engineer preserved them without an expiry date.

The CUR exposes cost by volume ID and usage type. Grouping by usage type separates gp2, gp3, provisioned IOPS, throughput, and other billed dimensions.

SELECT
  line_item_resource_id AS volume_id,
  line_item_usage_type AS usage_type,
  SUM(line_item_usage_amount) AS quantity,
  SUM(line_item_unblended_cost) AS cost
FROM cur_database.cur_table
WHERE line_item_usage_start_date >= TIMESTAMP '2026-07-01'
  AND line_item_usage_start_date <  TIMESTAMP '2026-08-01'
  AND line_item_product_code = 'AmazonEC2'
  AND (line_item_usage_type LIKE '%EBS:VolumeUsage%'
       OR line_item_usage_type LIKE '%EBS:VolumeP-IOPS%'
       OR line_item_usage_type LIKE '%EBS:VolumeP-Throughput%')
  AND line_item_line_item_type = 'Usage'
GROUP BY 1, 2
ORDER BY cost DESC;

Now compare those IDs with EC2 Volumes inventory filtered to available in every Region. For each candidate, record its creation time, tags, last attachment from CloudTrail if retained, filesystem purpose, and owner. A volume named old is not evidence. Create a final snapshot when recovery requirements justify it, test that the snapshot is usable for important data, then delete the volume.

Attached storage also wastes money. A gp3 volume can carry separately provisioned IOPS and throughput far above observed demand; a gp2 volume can stay oversized because capacity once bought performance. Review VolumeReadOps, VolumeWriteOps, queue length, throughput, and burst behavior. Change one dimension at a time, especially on latency-sensitive databases, and keep the rollback value written down.

3. Snapshot retention grows without an obvious resource to blame

EBS snapshots accumulate quietly because the bill shows storage consumption, while operators think in terms of snapshot count. Incremental billing also causes a dangerous misconception: deleting one old snapshot does not necessarily free the size shown in the console, because blocks referenced by later snapshots remain. AWS manages those references when a snapshot is deleted; nobody should manually infer a deletion order from apparent full sizes.

Use CUR to rank snapshot-related usage, then correlate the spend with snapshot inventory, AMIs, Recycle Bin rules, and AWS Backup recovery points. Resource IDs are not equally populated for every snapshot charge, so the first query intentionally keeps usage type and operation.

SELECT
  line_item_usage_type AS usage_type,
  line_item_operation AS operation,
  line_item_resource_id AS resource_id,
  SUM(line_item_usage_amount) AS gb_month,
  SUM(line_item_unblended_cost) AS cost
FROM cur_database.cur_table
WHERE line_item_usage_start_date >= TIMESTAMP '2026-07-01'
  AND line_item_usage_start_date <  TIMESTAMP '2026-08-01'
  AND line_item_product_code = 'AmazonEC2'
  AND line_item_usage_type LIKE '%EBS:SnapshotUsage%'
  AND line_item_line_item_type = 'Usage'
GROUP BY 1, 2, 3
ORDER BY cost DESC;

The usual failure starts with two retention systems. A backup policy keeps daily recovery points, while a deployment job creates AMIs and never deregisters them. An engineer deletes old AMIs but leaves their backing snapshots, or removes a backup plan without expiring existing recovery points. Each system behaves correctly; together they keep far more history than the recovery objective requires.

Write retention as a recovery contract: how far back the business must restore, how quickly, and which copies satisfy regulatory or customer obligations. Then assign one system as the authority for each dataset. Delete only after checking AMI references, sharing, legal holds, and cross-account copies. Lifecycle policies prevent the next pile, but they do not resolve ambiguous ownership in the current one.

4. RDS waste hides in capacity, standby copies, and retained storage

RDS waste is rarely just an oversized instance. Multi-AZ topology, storage type, provisioned IOPS, backup retention, manual snapshots, read replicas, Extended Support, and data transfer can each persist after the workload changes. A database with low average CPU may still need memory, connections, I/O, or failover capacity, so resizing from one chart is careless.

Start with a billing decomposition that keeps deployment option and usage type visible.

SELECT
  line_item_resource_id AS database_resource,
  product_database_engine AS engine,
  product_deployment_option AS deployment,
  line_item_usage_type AS usage_type,
  SUM(line_item_usage_amount) AS usage,
  SUM(line_item_unblended_cost) AS cost
FROM cur_database.cur_table
WHERE line_item_usage_start_date >= TIMESTAMP '2026-07-01'
  AND line_item_usage_start_date <  TIMESTAMP '2026-08-01'
  AND line_item_product_code = 'AmazonRDS'
  AND line_item_line_item_type = 'Usage'
GROUP BY 1, 2, 3, 4
ORDER BY cost DESC;

For expensive instances, inspect CPU, freeable memory, database connections, read and write latency, IOPS, throughput, queue depth, replica lag, and failover requirements over a representative business cycle. Performance Insights can show whether SQL or waits cause pressure that a larger class merely masks. A rightsize that pushes the database into storage latency or swap is not a saving.

Also list stopped instances nearing automatic restart, unused replicas, clusters retained after cutovers, manual snapshots outside policy, and allocated storage far above actual data. Check backup storage against the free allocation rules that apply to the database and retention design rather than assuming every backup gigabyte is billed. Remove unused replicas and obsolete snapshots first. Change instance class or topology later, inside a maintenance and rollback plan.

5. NAT gateways turn internal design choices into metered traffic

NAT gateway charges have two parts that teams often blur: gateway-hours and processed bytes. Deleting one idle gateway attacks hours; routing high-volume AWS service traffic through NAT attacks bytes. The invoice can rise sharply while EC2 stays flat because a container pull, artifact download, telemetry export, or cross-zone route changed.

This query splits hourly and data-processing usage. AWS usage-type names include regional prefixes, so match the stable suffixes.

SELECT
  line_item_usage_type AS usage_type,
  line_item_resource_id AS nat_resource,
  line_item_availability_zone AS az,
  SUM(line_item_usage_amount) AS usage,
  SUM(line_item_unblended_cost) AS cost
FROM cur_database.cur_table
WHERE line_item_usage_start_date >= TIMESTAMP '2026-07-01'
  AND line_item_usage_start_date <  TIMESTAMP '2026-08-01'
  AND line_item_product_code = 'AmazonEC2'
  AND (line_item_usage_type LIKE '%NatGateway-Hours'
       OR line_item_usage_type LIKE '%NatGateway-Bytes')
  AND line_item_line_item_type = 'Usage'
GROUP BY 1, 2, 3
ORDER BY cost DESC;

CloudWatch's AWS/NATGateway namespace exposes BytesInFromSource, BytesOutToDestination, BytesInFromDestination, and BytesOutToSource; AWS says Sum is the useful statistic for these byte metrics. Use the spike window to select a gateway, then query VPC Flow Logs for the largest talkers. A Logs Insights query can rank accepted flows, although the precise fields depend on the flow-log format you configured.

fields srcAddr, dstAddr, bytes, action
| filter action = "ACCEPT"
| stats sum(bytes) as transferred by srcAddr, dstAddr
| sort transferred desc
| limit 50

Do not centralize NAT across Availability Zones just to reduce gateway-hours without pricing the transfer and failure tradeoff. For heavy traffic to supported AWS services, evaluate gateway or interface VPC endpoints against NAT processing, endpoint hours, endpoint data, DNS behavior, and operational complexity. The cheaper design depends on volume and topology. Measure it from flow data rather than repeating a rule from somebody else's architecture.

6. Cross-AZ and cross-Region transfer can exceed the compute it connects

Data transfer waste appears when architecture crosses a billing boundary more often than its designers realize. A load balancer distributes to targets in another Availability Zone, an application talks to a database across zones, a Kubernetes service takes an extra hop, or replication crosses Regions at an unnecessary frequency. The compute resources can look perfectly sized while the paths between them dominate the change.

CUR usage types contain terms such as DataTransfer, DataXfer, Regional-Bytes, In-Bytes, and Out-Bytes, depending on service and route. Do not assume one spelling covers the account. Discover the billed vocabulary first.

SELECT
  line_item_product_code AS service,
  line_item_usage_type AS usage_type,
  line_item_operation AS operation,
  product_from_location AS from_location,
  product_to_location AS to_location,
  SUM(line_item_usage_amount) AS gb,
  SUM(line_item_unblended_cost) AS cost
FROM cur_database.cur_table
WHERE line_item_usage_start_date >= TIMESTAMP '2026-07-01'
  AND line_item_usage_start_date <  TIMESTAMP '2026-08-01'
  AND (lower(line_item_usage_type) LIKE '%transfer%'
       OR lower(line_item_usage_type) LIKE '%dataxfer%'
       OR lower(line_item_usage_type) LIKE '%regional-bytes%'
       OR lower(line_item_usage_type) LIKE '%out-bytes%')
  AND line_item_line_item_type = 'Usage'
GROUP BY 1, 2, 3, 4, 5
ORDER BY cost DESC;

This is where tags often fail because the charge does not map neatly to one tagged resource. Join billing evidence with load balancer metrics, database connections, application telemetry, and VPC Flow Logs during the same hour. Compare the day before and after a deployment. If bytes moved but requests did not, inspect retries, health checks, replication, compression, and chatty protocols.

Keep resilience constraints explicit. Co-locating every component in one zone may cut transfer but weaken failure tolerance. A better correction can be zone-aware routing, one cache per zone, balanced target capacity, smaller payloads, or fewer accidental hops. Price the corrected path and document why the remaining cross-zone traffic earns its cost.

7. S3 bills contain old versions, tiny requests, and unfinished uploads

S3 waste hides behind a cheap per-gigabyte reputation. Versioned buckets retain noncurrent objects, lifecycle transitions create minimum-duration or request effects, millions of small objects amplify request and metadata cost, and incomplete multipart uploads keep their uploaded parts until the upload completes or is aborted. AWS's S3 multipart documentation explicitly says those stored parts remain billable, which is why cleanup belongs in lifecycle policy rather than a quarterly reminder.

Break S3 cost down by usage type and operation before changing storage classes.

SELECT
  line_item_resource_id AS resource_id,
  line_item_usage_type AS usage_type,
  line_item_operation AS operation,
  SUM(line_item_usage_amount) AS usage,
  SUM(line_item_unblended_cost) AS cost
FROM cur_database.cur_table
WHERE line_item_usage_start_date >= TIMESTAMP '2026-07-01'
  AND line_item_usage_start_date <  TIMESTAMP '2026-08-01'
  AND line_item_product_code = 'AmazonS3'
  AND line_item_line_item_type = 'Usage'
GROUP BY 1, 2, 3
ORDER BY cost DESC;

CUR may not attribute every S3 charge cleanly to a bucket. Use S3 Storage Lens to find large buckets, noncurrent version bytes, delete markers, cold data, and incomplete uploads. Its IncompleteMultipartUploadStorageBytes metric measures parts in scope, and its older-than-seven-days metric makes abandoned uploads easier to separate from active work. Storage Lens metrics are daily, so do not expect them to explain an hourly request burst by themselves.

Set lifecycle rules from access and recovery facts. Expire noncurrent versions only when rollback and compliance allow it. Abort incomplete multipart uploads after a period longer than legitimate uploads need. Model transitions with object size, access frequency, retrieval cost, request cost, and minimum storage duration. Blindly pushing every object into the coldest class can exchange a predictable storage bill for retrieval charges and operational delay.

8. CloudWatch logs charge for both verbosity and neglect

CloudWatch Logs waste begins at ingestion and continues in retained storage. A debug deployment can multiply events overnight; infinite retention then preserves the mistake. Custom metrics, high-cardinality dimensions, verbose Container Insights, and duplicate log routes can add separate charges, so the service total needs to be split by usage type.

SELECT
  line_item_usage_type AS usage_type,
  line_item_operation AS operation,
  line_item_resource_id AS resource_id,
  SUM(line_item_usage_amount) AS usage,
  SUM(line_item_unblended_cost) AS cost
FROM cur_database.cur_table
WHERE line_item_usage_start_date >= TIMESTAMP '2026-07-01'
  AND line_item_usage_start_date <  TIMESTAMP '2026-08-01'
  AND line_item_product_code = 'AmazonCloudWatch'
  AND line_item_line_item_type = 'Usage'
GROUP BY 1, 2, 3
ORDER BY cost DESC;

In CloudWatch Logs, rank log groups by stored bytes and inspect retention settings. Then use incoming-byte metrics around the bill spike to identify the producers. Search a sample for repeated stack traces, health-check access lines, oversized structured fields, or one exception copied by several layers. Sampling must preserve the evidence used for incidents, security, and audits; deleting all debug output after a cost scare merely transfers cost into longer outages.

Set explicit retention per log class. Short retention fits noisy application diagnostics when another system keeps the required record; security and audit logs may need much longer. Filter at the producer when possible, cap exception payloads, avoid logging full request bodies, and prevent retries from logging the same failure at every layer. Retention fixes storage tomorrow. Reducing unnecessary ingestion fixes the larger leak at its source.

9. Idle load balancers and public IPv4 addresses look too small to chase

Small hourly charges become material when environments multiply. Test stacks leave load balancers with no useful targets, old network interfaces retain public IPv4 addresses, and duplicated ingress paths survive migrations. Since AWS charges for public IPv4 addresses, including addresses attached to running resources, an address can be waste even when it is technically associated.

Use the ledger to group load-balancing and public-address costs. Product codes and usage-type strings can differ across export versions, so retain the discovered values rather than hard-coding a monthly report around one label.

SELECT
  line_item_product_code AS service,
  line_item_usage_type AS usage_type,
  line_item_resource_id AS resource_id,
  SUM(line_item_usage_amount) AS usage,
  SUM(line_item_unblended_cost) AS cost
FROM cur_database.cur_table
WHERE line_item_usage_start_date >= TIMESTAMP '2026-07-01'
  AND line_item_usage_start_date <  TIMESTAMP '2026-08-01'
  AND (line_item_product_code = 'AWSELB'
       OR lower(line_item_usage_type) LIKE '%publicipv4%')
  AND line_item_line_item_type = 'Usage'
GROUP BY 1, 2, 3
ORDER BY cost DESC;

For each load balancer, inspect processed bytes, new and active connections, request count where applicable, target health, listeners, DNS dependencies, and traffic over at least one business cycle. Zero healthy targets does not automatically mean unused; it can mean broken production. For each public address, trace the attached network interface and the system that still resolves to it. IP Address Manager Public IP Insights can help inventory public IPv4 use across Regions.

Remove abandoned stacks through their infrastructure definition so they stay gone. Where a public endpoint has a real consumer, decide whether shared ingress, private connectivity, or IPv6 fits that consumer before removing the address. The individual line may look trivial, but this category often reveals an environment lifecycle that leaks larger resources elsewhere.

10. Containers and serverless move waste into reservations and concurrency

Managed compute does not remove idle capacity; it changes where idle capacity appears. ECS tasks can request more CPU and memory than they use, EKS nodes can remain underfilled because requests, daemon sets, topology constraints, or autoscaler settings block packing, and Lambda can retain provisioned concurrency that traffic no longer needs. Fargate tasks that run after a queue drains still bill for requested resources.

The first query separates the compute families and billed dimensions. Adapt product codes to what the baseline ledger actually contains.

SELECT
  line_item_product_code AS service,
  line_item_usage_type AS usage_type,
  line_item_operation AS operation,
  SUM(line_item_usage_amount) AS usage,
  SUM(line_item_unblended_cost) AS cost
FROM cur_database.cur_table
WHERE line_item_usage_start_date >= TIMESTAMP '2026-07-01'
  AND line_item_usage_start_date <  TIMESTAMP '2026-08-01'
  AND line_item_product_code IN ('AmazonECS', 'AmazonEKS', 'AWSLambda')
  AND line_item_line_item_type = 'Usage'
GROUP BY 1, 2, 3
ORDER BY cost DESC;

For ECS and Fargate, compare task reservations with observed CPU and memory, desired count with actual demand, and service schedules with queue depth or request volume. For EKS, compare node allocatable capacity with pod requests, not just live utilization. A cluster can show low CPU while the scheduler cannot place another pod because memory requests or topology rules reserve the remaining space. Account for the control plane, load balancers, volumes, NAT traffic, and logs around the cluster; they may exceed the node adjustment you planned.

For Lambda, split requests, duration by architecture and memory tier, provisioned concurrency, and ephemeral storage. Look at duration percentiles, throttles, errors, cold-start tolerance, and event-source backlog before changing memory or concurrency. More memory can shorten duration enough to cost less, so reducing memory without testing is not optimization. Remove stale functions, versions, event sources, and concurrency reservations only after tracing invocations and aliases.

Fix variable waste before negotiating committed spend

The repair order should follow reversibility and evidence, not the size of the loudest dashboard tile. First stop creation of new waste: revert runaway logging, failed retry loops, accidental replication, or a deployment that changed network paths. Second delete resources with confirmed zero demand, such as abandoned uploads, unattached volumes after recovery review, expired snapshots, and dead test stacks. Third schedule nonproduction capacity and rightsize compute, databases, containers, and provisioned concurrency using representative metrics. Fourth change architecture, including endpoints, zone-aware routing, storage lifecycle, and shared ingress. Only then model Savings Plans or Reservations against the stable baseline.

Buying a commitment at the start is popular because the console can show an immediate discount estimate. It is the wrong first move when waste distorts the baseline. A commitment can discount usage you should eliminate, reduce flexibility during a migration, and make the invoice look improved while consumption stays careless. Use net amortized cost for commitment decisions, but use unblended usage cost and quantities to understand the machinery creating the bill. Those are different questions.

Prove the reduction with a paired window

A cost action is complete only when the billing export shows the expected quantity change and the service still meets its operating target. Compare equal calendar windows that share the same weekday mix. For a shutdown schedule, compare instance-hours. For NAT work, compare processed gigabytes. For log filtering, compare ingested bytes. A dollar-only comparison can lie when exchange rates, tiered pricing, credits, taxes, or commitment allocation changed between windows.

This query compares two seven-day periods by service and usage type. It reports both cost and usage, which makes an apparent saving easier to challenge. Change the periods around the deployment or cleanup date, and investigate rows where cost fell but usage did not.

WITH measured AS (
  SELECT
    CASE
      WHEN line_item_usage_start_date >= TIMESTAMP '2026-07-01'
       AND line_item_usage_start_date <  TIMESTAMP '2026-07-08' THEN 'before'
      WHEN line_item_usage_start_date >= TIMESTAMP '2026-07-15'
       AND line_item_usage_start_date <  TIMESTAMP '2026-07-22' THEN 'after'
    END AS period,
    line_item_product_code AS service,
    line_item_usage_type AS usage_type,
    line_item_usage_amount AS usage,
    line_item_unblended_cost AS cost
  FROM cur_database.cur_table
  WHERE line_item_usage_start_date >= TIMESTAMP '2026-07-01'
    AND line_item_usage_start_date <  TIMESTAMP '2026-07-22'
    AND line_item_line_item_type = 'Usage'
)
SELECT
  service,
  usage_type,
  SUM(CASE WHEN period = 'before' THEN usage ELSE 0 END) AS usage_before,
  SUM(CASE WHEN period = 'after' THEN usage ELSE 0 END) AS usage_after,
  SUM(CASE WHEN period = 'before' THEN cost ELSE 0 END) AS cost_before,
  SUM(CASE WHEN period = 'after' THEN cost ELSE 0 END) AS cost_after
FROM measured
WHERE period IS NOT NULL
GROUP BY 1, 2
HAVING SUM(cost) > 0
ORDER BY cost_before - cost_after DESC;

Tag coverage affects how far attribution can go. Cost-allocation tags must be activated for billing, and new tags do not repair old rows retroactively. Untagged shared resources need an allocation rule that finance and engineering both understand, such as requests, bytes, task-hours, or a fixed platform share. Do not force false precision. If the network charge cannot be assigned below a shared platform, leave it with that platform and assign an engineer to change the architecture or metering.

Budgets and anomaly detection should follow the cleanup, but neither replaces the ledger. Set alerts on both the payer total and services with a history of sudden variable usage. Route an alert to somebody who can inspect a deployment, and include account, Region, service, usage type, expected baseline, and a runbook query. A message that only says the bill increased turns cost control into a finance escalation instead of an engineering signal.

Consolidated billing deserves one more check. A payer-level service increase can come from a new linked account, a resource moved between accounts, or a discount allocation shift rather than more total usage. Re-run the winning query with line_item_usage_account_id in the grouping, then inspect Regions inside that account. Keep sandbox and acquired-company accounts in scope until their owners prove they are closed. An empty dashboard in the main production account says nothing about resources running under a forgotten login.

Keep a finding register with owner, account, Region, resource, evidence window, monthly run rate, dependency, action, rollback, and verified result. Mark estimates as estimates. After each change, compare both usage quantity and cost against the same-length baseline; price changes, credits, and commitment allocation can move dollars without proving that engineering demand fell.

A Team & AI Audit from oleg.is can include this cost trail when infrastructure expense is tangled with team structure and delivery habits. The useful outcome is still the same whether you hire help or do it internally: every large line item has an owner, every exception has an expiry date, and the next invoice can be explained from resource changes before finance has to ask.