AWS RDS gp3 IOPS Burst Depletion: Root Cause & Disk Latency Recovery

1. AWS EBS gp3 Storage Architecture vs Legacy gp2

When AWS introduced gp3 for Amazon RDS, it offered a 20% cost reduction per gigabyte and eliminated the need to over-provision disk size simply to obtain higher IOPS. However, this introduced a critical architectural vulnerability for high-concurrency systems:

Under gp2, an engineering team running a 2TB database automatically received 6,000 baseline IOPS and up to 250 MB/s throughput. When migrating the same 2TB database to gp3, the volume defaults to 3,000 IOPS and 125 MB/s throughput unless explicitly configured with provisioned IOPS and throughput add-ons.

2. Critical Amazon CloudWatch Telemetry to Inspect

During an active database latency spike, evaluate these four CloudWatch metrics simultaneously in the AWS Console:

• DiskQueueDepth: The number of outstanding read and write requests waiting to access physical disk. In a healthy production database, DiskQueueDepth should equal 1 to (IOPS / 1000). A value above 10 indicates severe storage starvation.
• ReadThroughput + WriteThroughput: Sum these two metrics. If their total consistently reaches 125,000,000 bytes/sec (125 MB/s), your volume is being aggressively throttled by the AWS storage hypervisor.
• ReadLatency & WriteLatency: Healthy NVMe EBS volumes exhibit latencies under 5ms. Throttled gp3 volumes will experience latency spikes exceeding 100ms–1,200ms.
• EBSIOBalance% (Legacy gp2 only): If running on legacy gp2, an IO Balance of 0% indicates complete burst credit exhaustion.

3. Live Volume Modification via AWS CLI (Zero Downtime)

To immediately relieve EBS throttling during an incident, provision additional IOPS and throughput using the aws rds modify-db-instance command. This operation takes effect in real-time without restarting the database:

# Scale RDS gp3 volume to 10,000 IOPS and 500 MB/s Throughput
aws rds modify-db-instance \
    --db-instance-identifier production-postgres-cluster \
    --iops 10000 \
    --allocated-storage 500 \
    --storage-type gp3 \
    --apply-immediately

# Verify the modification progress
aws rds describe-db-instances \
    --db-instance-identifier production-postgres-cluster \
    --query "DBInstances[0].PendingModifiedValues"

4. Resolving Temp File Spilling in PostgreSQL Engine

While provisioning additional IOPS resolves the immediate infrastructure symptom, the root cause is frequently database queries spilling temporary sort files to disk because work_mem is set too low.

Query pg_stat_database to verify the volume of temporary bytes written to disk:

-- Check total temporary bytes written since last server restart
SELECT datname, 
       temp_files AS temp_files_created,
       pg_size_pretty(temp_bytes) AS temp_data_written
FROM pg_stat_database 
WHERE datname = current_database();

-- Identify active queries currently consuming storage I/O
SELECT pid, 
       now() - query_start AS duration, 
       query, 
       state 
FROM pg_stat_activity 
WHERE state != 'idle' 
ORDER BY duration DESC 
LIMIT 5;

In your custom RDS Parameter Group, increase work_mem from the default 4MB to 32MB or 64MB for analytics-heavy databases, and ensure composite indexes cover all ORDER BY and GROUP BY clauses.

5. EBS Optimized Instances & Max Bandwidth Thresholds

Even if you provision 12,000 IOPS and 500 MB/s throughput on your gp3 volume, your database can still throttle if the underlying AWS EC2 instance type lacks sufficient dedicated EBS bandwidth. For example, a db.t4g.medium instance is physically capped at 2,780 Maximum Dedicated EBS Throughput (Mbps) with burst allowances. Once burst credits deplete, the EC2 hypervisor throttles traffic between the compute instance and the EBS storage network.

For production workloads processing continuous batch writes, migrate to memory-optimized db.r6g.xlarge or db.r7g.xlarge instances which provide non-burstable, dedicated 10,000 Mbps EBS bandwidth channels.

6. Ext4 Filesystem Mount Options for Cloud Databases

Under heavy write traffic, Linux filesystem journal commits create additional synchronous write I/O. For high-throughput PostgreSQL installations on AWS EBS, verify that your underlying volumes are formatted with noatime and nodiratime mount options to eliminate unnecessary metadata access timestamp writes on every read.

7. Terraform CloudWatch Alarms for Disk Starvation

Deploy deterministic Terraform CloudWatch alarms to notify your pager before EBS throttling triggers application timeouts:

resource "aws_cloudwatch_metric_alarm" "rds_disk_queue_depth" {
  alarm_name          = "rds-production-disk-queue-critical"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 2
  metric_name         = "DiskQueueDepth"
  namespace           = "AWS/RDS"
  period              = 60
  statistic           = "Average"
  threshold           = 10
  alarm_description   = "RDS Disk Queue Depth > 10 indicating EBS storage throughput exhaustion"
  alarm_actions       = [aws_sns_topic.sre_pager.arn]

  dimensions = {
    DBInstanceIdentifier = "production-postgres-cluster"
  }
}

8. SRE Incident Triage Checklist

1. Check CloudWatch DiskQueueDepth (> 10) and total Throughput (> 125 MB/s).
2. Run aws rds modify-db-instance with --iops 10000 --apply-immediately for zero-downtime scaling.
3. Inspect pg_stat_activity for long-running sequential scans and terminate rogue queries.
4. Audit work_mem in your RDS Parameter Group to eliminate temporary disk file generation.
5. Verify CloudWatch alarm thresholds in Terraform to ensure automated pager alerts at 80% throughput capacity.

Aftab M.

Founder of Digixfly.com & Editor-in-Chief of XSeedInfo

Aftab M. is the Founder of Digixfly.com, an AI-powered technical marketing and digital growth agency. He directs technical investigations, infrastructure benchmarks, and unit economics teardowns for technology leaders across North America.

About the Author • Editorial Standards • Contact Desk