~/sph.sh
aws
fargate
debugging

AWS Fargate 103: Production War Stories That'll Save You Hours

Production incidents from running Fargate at scale. Memory leaks, ENI limits, subnet failures, and debugging techniques that work.

There's something humbling about thinking your Fargate setup is solid, seeing all green dashboards, and then discovering there were blind spots you hadn't considered. Over the past couple of years running Fargate workloads at various scales, I've encountered several interesting challenges that taught me valuable lessons.

This concludes our Fargate series (101, 102). I want to share some of the more instructive issues I've encountered, along with the debugging approaches and solutions that worked for us.

The ENI Limit Discovery#

Context: We were preparing for Black Friday traffic on an e-commerce platform, expecting roughly 10x normal load. Auto-scaling was configured, load testing had passed, and our confidence was high.

The Issue: During the evening before Black Friday, our tasks started failing with an error we hadn't seen before:

Text
ResourcesNotReady: The ENI allocation could not be completed

Fargate tasks were stuck in PENDING state. New deployments wouldn't complete, and auto-scaling couldn't provision additional capacity. Most dashboards showed normal operation, but we discovered that available ENIs in our VPC had reached the limit.

What We Learned#

Each Fargate task requires its own ENI, and AWS enforces limits per VPC. We had been running around 200 tasks across 3 availability zones, but hadn't accounted for ENI consumption from other services. The limits we discovered:

  • Default ENI limit: 5,000 per VPC
  • Each Fargate task: 1 ENI
  • Each RDS instance: 1 ENI
  • Each Lambda in VPC: Shares ENI pool
  • Each ELB: Multiple ENIs

We checked our limits:

Bash
aws ec2 describe-account-attributes \
  --attribute-names max-instances

# But the real limit is here:
aws service-quotas get-service-quota \
  --service-code ec2 \
  --quota-code L-0263D0A3  # ENIs per VPC

We discovered we were at 4,847 ENIs. Our load testing had focused on application performance but hadn't considered the cumulative ENI consumption across all services in the VPC.

Resolution Approach#

Immediate steps:

Bash
# Scale down non-critical services in development
aws ecs update-service \
  --cluster development \
  --service api \
  --desired-count 0

# Request quota increase through AWS Support
aws support create-case \
  --subject "ENI quota increase needed - production capacity planning" \
  --service-code "service-quota-increase"

Longer-term improvements:

  1. Multiple VPCs: Split workloads across dev, staging, and prod VPCs
  2. ENI monitoring: CloudWatch custom metric tracking ENI usage
  3. Right-sizing: Reduced over-provisioned tasks
  4. Lambda optimization: Moved Lambdas out of VPC where possible
TypeScript
// ENI monitoring Lambda
export const monitorENIs = async () => {
  const ec2 = new AWS.EC2();
  const cloudWatch = new AWS.CloudWatch();
  
  const enis = await ec2.describeNetworkInterfaces().promise();
  const inUse = enis.NetworkInterfaces?.length || 0;
  
  await cloudWatch.putMetricData({
    Namespace: 'Custom/VPC',
    MetricData: [{
      MetricName: 'ENIsInUse',
      Value: inUse,
      Unit: 'Count'
    }]
  }).promise();
};

Lessons learned:

  • Load testing should include all infrastructure components, not just your application
  • ENI limits are per VPC, not per service
  • AWS Support is surprisingly responsive during critical incidents

The Subnet That Went Rogue#

The Setup: Multi-AZ Fargate deployment across three private subnets. Everything running smoothly for months.

What Happened: Tuesday morning, 40% of our tasks started showing intermittent connectivity issues. Some HTTP requests succeeded, others timed out after 30 seconds.

The weird part? Only tasks in one specific subnet (us-east-1a) were affected.

The Investigation Journey#

First, the obvious checks:

Bash
# Check task health
aws ecs list-tasks --cluster production --service-name api
aws ecs describe-tasks --cluster production --tasks task-abc123

# Check network interfaces
aws ec2 describe-network-interfaces \
  --filters "Name=subnet-id,Values=subnet-12345" \
  --query 'NetworkInterfaces[*].[NetworkInterfaceId,Status,PrivateIpAddress]'

Tasks looked healthy. Network interfaces were attached and active. But something was wrong.

The breakthrough: We enabled VPC Flow Logs and found the smoking gun:

Bash
# Enable VPC Flow Logs for the problem subnet
aws ec2 create-flow-logs \
  --resource-type Subnet \
  --resource-ids subnet-12345 \
  --traffic-type ALL \
  --log-destination-type cloud-watch-logs \
  --log-group-name /aws/vpc/flowlogs

Flow logs showed that packets were leaving our subnet but never reaching their destination. The return packets were getting dropped somewhere.

The Culprit#

Turns out, our network team had modified the route table for that subnet earlier that morning. They changed the NAT gateway route from 0.0.0.0/0 → nat-gateway-123 to 0.0.0.0/0 → nat-gateway-456 without realizing Fargate tasks were running there.

The new NAT gateway was in a different AZ and had different security group rules. Classic.

The fix:

Bash
# Check which route table is associated with the subnet
aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=subnet-12345"

# Verify the routes
aws ec2 describe-route-tables --route-table-ids rtb-abc123 \
  --query 'RouteTables[*].Routes[*].[DestinationCidrBlock,GatewayId,State]'

# Fix the route (revert to original NAT gateway)
aws ec2 replace-route \
  --route-table-id rtb-abc123 \
  --destination-cidr-block 0.0.0.0/0 \
  --nat-gateway-id nat-gateway-123

Lessons learned:

  • Always test routing changes in non-production first
  • VPC Flow Logs are your friend for network debugging
  • Document which route tables serve which services
  • Set up monitoring for routing table changes

The Memory Leak Mystery (No SSH Edition)#

The Setup: Node.js API running on Fargate, memory limit set to 2GB per task. Worked fine for weeks.

What Happened: Memory usage slowly climbing over 3-4 hours, then tasks getting OOM killed. Memory usage graphs looked like ski slopes.

But here's the kicker: no way to SSH into the container to debug.

The Investigation Arsenal#

Since we can't SSH, we need to get creative:

1. ECS Exec (our savior):

Bash
# First, enable it on the service
aws ecs update-service \
  --cluster production \
  --service api \
  --enable-execute-command

# Then connect to a running task
aws ecs execute-command \
  --cluster production \
  --task task-abc123 \
  --container api \
  --interactive \
  --command "/bin/bash"

# Inside the container, check memory usage
> ps aux --sort=-%mem | head -20
> cat /proc/meminfo
> pmap -x 1  # Memory map of PID 1

2. Application-level monitoring:

JavaScript
// Add to your Node.js app
const express = require('express');
const app = express();

// Memory monitoring endpoint
app.get('/debug/memory', (req, res) => {
  const used = process.memoryUsage();
  const stats = {
    rss: Math.round(used.rss / 1024 / 1024 * 100) / 100,      // MB
    heapTotal: Math.round(used.heapTotal / 1024 / 1024 * 100) / 100,
    heapUsed: Math.round(used.heapUsed / 1024 / 1024 * 100) / 100,
    external: Math.round(used.external / 1024 / 1024 * 100) / 100,
    arrayBuffers: Math.round(used.arrayBuffers / 1024 / 1024 * 100) / 100
  };
  
  res.json(stats);
});

// Heap snapshot endpoint (for extreme debugging)
app.get('/debug/heapdump', (req, res) => {
  const heapdump = require('heapdump');
  const filename = `/tmp/heapdump-${Date.now()}.heapsnapshot`;
  
  heapdump.writeSnapshot(filename, (err, filename) => {
    if (err) {
      res.status(500).send(err.message);
    } else {
      res.download(filename);
    }
  });
});

3. The detective work:

We used ECS Exec to install debugging tools and found that our HTTP client wasn't properly closing connections:

Bash
# Inside the container
> npm install -g clinic
> clinic doctor --on-port 8080 -- node index.js &
> curl http://localhost:8080/debug/memory

# Check open file descriptors
> ls -la /proc/1/fd | wc -l
> lsof -p 1 | grep TCP

Bingo! Thousands of TCP connections in CLOSE_WAIT state.

The Root Cause#

Our Node.js HTTP client code looked innocent enough:

JavaScript
// The problematic code
const axios = require('axios');

async function callExternalAPI() {
  const response = await axios.get('https://api.example.com/data');
  return response.data;
}

But we weren't configuring connection pooling or timeouts properly. Each request created new connections that weren't being cleaned up.

The fix:

JavaScript
// Fixed version with proper configuration
const axios = require('axios');
const https = require('https');
const http = require('http');

// Configure connection pooling
const httpAgent = new http.Agent({
  keepAlive: true,
  maxSockets: 50,
  timeout: 5000,
});

const httpsAgent = new https.Agent({
  keepAlive: true,
  maxSockets: 50,
  timeout: 5000,
});

const axiosInstance = axios.create({
  httpAgent,
  httpsAgent,
  timeout: 10000, // 10 seconds
});

// Graceful shutdown
process.on('SIGTERM', () => {
  httpAgent.destroy();
  httpsAgent.destroy();
});

async function callExternalAPI() {
  const response = await axiosInstance.get('https://api.example.com/data');
  return response.data;
}

Lessons learned:

  • ECS Exec is invaluable for containerized debugging
  • Always configure HTTP clients properly in production
  • Monitor file descriptors, not just memory
  • Connection pools matter, even for "simple" HTTP clients

The 30-Second Connection Timeout Phantom#

The Setup: Internal service-to-service communication between two Fargate services. Worked fine 99% of the time.

What Happened: Randomly, about 1% of requests would hang for exactly 30 seconds, then fail with a connection timeout.

The pattern was completely random. No correlation with load, time of day, or deployment history.

The Debugging Odyssey#

Network layer investigation:

Bash
# VPC Flow Logs analysis
aws logs filter-log-events \
  --log-group-name /aws/vpc/flowlogs \
  --start-time 1645564800000 \
  --filter-pattern "REJECT"

# Security group rules audit
aws ec2 describe-security-groups \
  --group-ids sg-12345 \
  --query 'SecurityGroups[*].{GroupId:GroupId,IpPermissions:IpPermissions}'

Security groups looked fine. Flow logs showed packets flowing normally.

Application layer investigation:

JavaScript
// Added detailed connection tracking
const net = require('net');
const original_connect = net.Socket.prototype.connect;

net.Socket.prototype.connect = function(...args) {
  const startTime = Date.now();
  console.log(`[${new Date().toISOString()}] Starting connection to ${args[0]?.host || args[0]?.path}`);
  
  const result = original_connect.apply(this, args);
  
  this.on('connect', () => {
    const duration = Date.now() - startTime;
    console.log(`[${new Date().toISOString()}] Connected after ${duration}ms`);
  });
  
  this.on('error', (err) => {
    const duration = Date.now() - startTime;
    console.log(`[${new Date().toISOString()}] Connection error after ${duration}ms:`, err.message);
  });
  
  return result;
};

The Breakthrough#

The logs showed something interesting: successful connections were taking 2-5ms, but the hanging ones were taking exactly 30,000ms. That's not random - that's a timeout.

Then we noticed the pattern: it only happened when both services were in the same availability zone and the connection was going through the load balancer.

Loading diagram...

The issue: AWS ALB has a known quirk where connections from the same AZ can occasionally loop back through the load balancer infrastructure, causing delays.

The fix (multiple strategies):

  1. Direct service communication for same-AZ:
JavaScript
// Service discovery with AZ awareness
const AWS = require('aws-sdk');
const ecs = new AWS.ECS();

async function getServiceEndpoints() {
  const tasks = await ecs.listTasks({
    cluster: 'production',
    serviceName: 'target-service'
  }).promise();
  
  const taskDetails = await ecs.describeTasks({
    cluster: 'production',
    tasks: tasks.taskArns
  }).promise();
  
  return taskDetails.tasks.map(task => ({
    ip: task.attachments[0].details.find(d => d.name === 'privateIPv4Address').value,
    az: task.availabilityZone,
    port: 8080
  }));
}

// Smart routing
async function callService(endpoint, data) {
  const currentAZ = process.env.AWS_AVAILABILITY_ZONE;
  const endpoints = await getServiceEndpoints();
  
  // Try same-AZ direct connection first
  const sameAZEndpoint = endpoints.find(e => e.az === currentAZ);
  if (sameAZEndpoint) {
    try {
      return await axios.post(`http://${sameAZEndpoint.ip}:${sameAZEndpoint.port}${endpoint}`, data);
    } catch (error) {
      // Fall back to load balancer
      return await axios.post(`https://internal-service.example.com${endpoint}`, data);
    }
  }
  
  // Use load balancer for cross-AZ
  return await axios.post(`https://internal-service.example.com${endpoint}`, data);
}
  1. Connection timeout tuning:
JavaScript
const axiosInstance = axios.create({
  timeout: 5000,  // Fail fast instead of waiting 30s
  httpsAgent: new https.Agent({
    timeout: 2000,  // Connection timeout
    keepAlive: true,
  })
});

Lessons learned:

  • ALBs can introduce unexpected latency for same-AZ communication
  • Service discovery enables direct communication patterns
  • Always implement connection timeouts shorter than your SLA
  • Load balancers aren't always the fastest path

The Deployment That Wouldn't Deploy#

The Setup: Standard blue-green deployment using CodeDeploy. Worked hundreds of times before.

What Happened: New deployment stuck at 50% for hours. Half the tasks were running the new version, half the old. CodeDeploy dashboard showed "In Progress" with no error messages.

Auto-rollback wasn't triggering because technically, nothing was "failing."

The Investigation#

CodeDeploy logs were unhelpful:

Bash
aws deploy get-deployment --deployment-id d-XXXXXXXXX
# Status: InProgress, no error information

aws logs filter-log-events \
  --log-group-name /aws/codedeploy-agent \
  --start-time $(date -d '1 hour ago' +%s)000

ECS service events revealed the issue:

Bash
aws ecs describe-services \
  --cluster production \
  --services api \
  --query 'services[0].events[0:10]'

The events showed:

Text
"(service api) failed to launch a task with (error ECS was unable to assume role...)"

The Root Cause#

Our task execution role had been modified by another team for an unrelated service, and they accidentally removed the trust relationship that allows ECS to assume the role.

The role policy looked like this:

JSON
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"  // WRONG!
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

It should have been:

JSON
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"  // CORRECT
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

The fix:

Bash
# Check the role trust policy
aws iam get-role --role-name fargate-task-execution-role \
  --query 'Role.AssumeRolePolicyDocument'

# Update it
aws iam update-assume-role-policy \
  --role-name fargate-task-execution-role \
  --policy-document file://trust-policy.json

Prevention strategy:

TypeScript
// Automated role validation
export const validateTaskRoles = async () => {
  const iam = new AWS.IAM();
  
  const role = await iam.getRole({
    RoleName: 'fargate-task-execution-role'
  }).promise();
  
  const trustPolicy = JSON.parse(decodeURIComponent(role.Role.AssumeRolePolicyDocument));
  const ecsService = trustPolicy.Statement.some(statement =>
    statement.Principal?.Service === 'ecs-tasks.amazonaws.com'
  );
  
  if (!ecsService) {
    await sendAlert('Task execution role missing ECS trust relationship!');
    return false;
  }
  
  return true;
};

Lessons learned:

  • ECS service events are more detailed than CodeDeploy logs
  • Role trust policies are fragile and need monitoring
  • Blue-green deployments can get stuck in limbo
  • Always check IAM when things mysteriously stop working

The Debug Toolbox That Works#

After all these incidents, here's the debugging toolkit that's saved us countless hours:

1. The Ultimate Fargate Debug Container#

Dockerfile
FROM node:18-alpine
RUN apk add --no-cache \
    curl \
    wget \
    netcat-openbsd \
    bind-tools \
    tcpdump \
    strace \
    htop \
    iotop \
    lsof \
    procps \
    net-tools

# Add your app
COPY . /app
WORKDIR /app

# Debug endpoints
RUN npm install express heapdump clinic

2. Monitoring Stack#

TypeScript
// Health check endpoint with detailed diagnostics
app.get('/health/detailed', async (req, res) => {
  const health = {
    timestamp: new Date().toISOString(),
    uptime: process.uptime(),
    memory: process.memoryUsage(),
    cpu: process.cpuUsage(),
    connections: {
      active: await getActiveConnections(),
      waiting: await getWaitingConnections()
    },
    environment: {
      nodeVersion: process.version,
      availabilityZone: process.env.AWS_AVAILABILITY_ZONE || 'unknown',
      region: process.env.AWS_REGION || 'unknown'
    }
  };
  
  res.json(health);
});

async function getActiveConnections() {
  return new Promise((resolve) => {
    require('child_process').exec('netstat -an | grep ESTABLISHED | wc -l', 
      (error, stdout) => {
        resolve(parseInt(stdout.trim()) || 0);
      }
    );
  });
}

3. Automated Incident Response#

YAML
# CloudWatch alarms that help
ENIUtilizationAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmName: High-ENI-Utilization
    MetricName: ENIsInUse
    Namespace: Custom/VPC
    Statistic: Maximum
    Period: 300
    EvaluationPeriods: 2
    Threshold: 4500  # 90% of 5000 limit
    ComparisonOperator: GreaterThanThreshold
    AlarmActions:
      - !Ref SNSTopic

MemoryUtilizationAlarm:
  Type: AWS::CloudWatch::Alarm  
  Properties:
    AlarmName: Fargate-Memory-High
    MetricName: MemoryUtilized
    Namespace: ECS/ContainerInsights
    Statistic: Average
    Period: 300
    EvaluationPeriods: 3
    Threshold: 80  # 80% memory usage
    ComparisonOperator: GreaterThanThreshold

The Universal Laws of Fargate Debugging#

After all these adventures, here are the patterns that hold true:

  1. When tasks won't start: Check ENI limits, security groups, and IAM roles (in that order)

  2. When tasks are slow: It's usually the network (route tables, NAT gateways, DNS)

  3. When memory keeps climbing: It's always connection pooling or event listeners

  4. When deployments hang: Check service events, not deployment logs

  5. When 1% of requests fail: Look for load balancer quirks or cross-AZ issues

  6. When nothing makes sense: Enable VPC Flow Logs and ECS Exec

The lesson? Fargate removes a lot of infrastructure complexity, but when things go wrong, you need to understand the underlying AWS networking and compute primitives. The abstraction is leaky, and production always finds the leaks.

Keep these debugging techniques handy. Trust me, you'll need them during critical incidents, and when that happens, you'll be grateful for every monitoring endpoint and diagnostic tool you set up beforehand.

Next time someone tells you serverless containers are "set it and forget it," show them this series. Production has other plans, but now you're ready for them.

AWS Fargate Deep Dive Series

Complete guide to AWS Fargate from basics to production. Learn serverless containers, cost optimization, debugging techniques, and Infrastructure-as-Code deployment patterns through real-world experience.

Progress3/4 posts completed
Previous
Advanced Patterns: Cost Optimization and Production Techniques
Next
IaC Deep Dive: Deploying with CDK, Terraform, and SAM
View series →
Loading...

Comments (0)

Join the conversation

Sign in to share your thoughts and engage with the community

No comments yet

Be the first to share your thoughts on this post!

Related Posts