Skip to content
~/sph.sh

AWS Secrets Manager & Parameter Store: Security Best Practices

A comprehensive technical guide comparing AWS Secrets Manager and Systems Manager Parameter Store, demonstrating when to use each service with real-world implementation patterns.

Engineers working with AWS face a common dilemma: choosing between Secrets Manager and Parameter Store for secrets management. While both services store sensitive data, they serve different purposes and come with different cost structures. This guide provides technical decision criteria, complete implementation patterns, and real-world lessons learned.

Understanding the Services

Before diving into implementation, let's establish the technical differences between these services.

Service Comparison

FeatureParameter Store StandardParameter Store AdvancedSecrets Manager
CostFree$0.05/secret/month$0.40/secret/month
Max Size4 KB8 KB64 KB
RotationManual onlyManual onlyAutomated with Lambda
VersioningSingle active versionSingle active versionMultiple concurrent versions
Cross-AccountVia RAM (since 2024)Via RAM (since 2024)Native resource policies
Multi-RegionManual replicationManual replicationAutomated replication
EncryptionOptional (SecureString)Optional (SecureString)Always encrypted (mandatory)
Native IntegrationsBasicBasicRDS, Redshift, DocumentDB

Key Technical Insight: Parameter Store with SecureString uses KMS encryption and provides free basic secrets management. Secrets Manager adds automatic rotation, native RDS integration, and built-in versioning with staging labels.

Decision Framework

Use this technical decision tree to choose the right service:

Cost Analysis Example:

  • 10 static API keys → Parameter Store Standard: $0/month
  • 5 RDS passwords with rotation → Secrets Manager: $2.00/month
  • 20 configuration values → Parameter Store Standard: $0/month
  • Total: 2.00/monthvs.2.00/month vs. 10.00/month if everything was in Secrets Manager

Cross-Account Secret Sharing

One of the most common requirements is sharing secrets between AWS accounts. Here's the complete implementation pattern.

Architecture Overview

Implementation - Account A (Secret Owner)

typescript
// CDK code for Account Aimport * as cdk from 'aws-cdk-lib';import * as kms from 'aws-cdk-lib/aws-kms';import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager';import * as iam from 'aws-cdk-lib/aws-iam';
const kmsKey = new kms.Key(this, 'SecretKey', {  enableKeyRotation: true,  description: 'KMS key for cross-account secret sharing',});
// Grant Account B permission to use the keykmsKey.addToResourcePolicy(new iam.PolicyStatement({  sid: 'AllowAccountBDecrypt',  principals: [new iam.AccountPrincipal('222222222222')], // Account B  actions: ['kms:Decrypt', 'kms:DescribeKey'],  resources: ['*'],  conditions: {    StringEquals: {      'kms:ViaService': 'secretsmanager.us-east-1.amazonaws.com',    },  },}));
const secret = new secretsmanager.Secret(this, 'SharedSecret', {  secretName: 'cross-account/database-credentials',  encryptionKey: kmsKey,});
// Add resource policy to secretsecret.addToResourcePolicy(new iam.PolicyStatement({  sid: 'AllowAccountBAccess',  principals: [new iam.AccountPrincipal('222222222222')],  actions: ['secretsmanager:GetSecretValue', 'secretsmanager:DescribeSecret'],  resources: ['*'],}));

Implementation - Account B (Secret Consumer)

typescript
// CDK code for Account Bconst applicationRole = new iam.Role(this, 'ApplicationRole', {  assumedBy: new iam.ServicePrincipal('ecs-tasks.amazonaws.com'),  inlinePolicies: {    'SecretAccess': new iam.PolicyDocument({      statements: [        new iam.PolicyStatement({          effect: iam.Effect.ALLOW,          actions: ['secretsmanager:GetSecretValue'],          resources: [            'arn:aws:secretsmanager:us-east-1:111111111111:secret:cross-account/database-credentials-*'          ],        }),        new iam.PolicyStatement({          effect: iam.Effect.ALLOW,          actions: ['kms:Decrypt'],          resources: [            'arn:aws:kms:us-east-1:111111111111:key/12345678-1234-1234-1234-123456789012'          ],          conditions: {            StringEquals: {              'kms:ViaService': 'secretsmanager.us-east-1.amazonaws.com',            },          },        }),      ],    }),  },});

Warning: Common Pitfall: Forgetting to grant KMS decrypt permission in Account B. The secret retrieval will fail with "AccessDeniedException" even if the Secrets Manager policy is correct.

Tip: Use CloudTrail to check for KMS API calls with error codes. Look for "Decrypt" operations that failed with "AccessDenied".

Parameter Store Reference Pattern

You can standardize on Parameter Store API while storing actual secrets in Secrets Manager:

bash
# Create a reference parameteraws ssm put-parameter \  --name "/app/database/password" \  --value "{{resolve:secretsmanager:prod/database:SecretString:password}}" \  --type "String" \  --description "Reference to Secrets Manager secret"

Application code only needs Parameter Store SDK:

typescript
import { SSMClient, GetParameterCommand } from "@aws-sdk/client-ssm";
const client = new SSMClient({ region: "us-east-1" });const command = new GetParameterCommand({  Name: "/app/database/password",  WithDecryption: true,});
const response = await client.send(command);console.log(response.Parameter.Value); // Actual secret from Secrets Manager

Benefit: Simplified application code, single API surface area, easier migration path between services.

Container Secrets Injection

There are multiple patterns for injecting secrets into containers, each with different trade-offs.

Pattern A: Environment Variable Injection (ECS)

This is the native ECS approach - secrets are injected at container startup.

typescript
// CDK - ECS Task Definition with secretsconst taskDefinition = new ecs.FargateTaskDefinition(this, 'TaskDef', {  cpu: 256,  memoryLimitMiB: 512,});
taskDefinition.addContainer('app', {  image: ecs.ContainerImage.fromRegistry('myapp:latest'),  secrets: {    // From Secrets Manager    DB_PASSWORD: ecs.Secret.fromSecretsManager(dbSecret, 'password'),    API_KEY: ecs.Secret.fromSecretsManager(apiKeySecret),
    // From Parameter Store    CONFIG_VALUE: ecs.Secret.fromSsmParameter(configParam),  },  logging: ecs.LogDrivers.awsLogs({ streamPrefix: 'app' }),});
// Grant read permissionsdbSecret.grantRead(taskDefinition.taskRole);apiKeySecret.grantRead(taskDefinition.taskRole);configParam.grantRead(taskDefinition.taskRole);

Raw Task Definition JSON:

json
{  "family": "myapp",  "taskRoleArn": "arn:aws:iam::123456789012:role/myapp-task-role",  "containerDefinitions": [    {      "name": "app",      "image": "myapp:latest",      "secrets": [        {          "name": "DB_PASSWORD",          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/database-AbCdEf:password::"        },        {          "name": "API_KEY",          "valueFrom": "arn:aws:secretsmanager:us-east-1:123456789012:secret:api-key-XyZaBc"        },        {          "name": "CONFIG_VALUE",          "valueFrom": "arn:aws:ssm:us-east-1:123456789012:parameter/app/config"        }      ]    }  ]}

Warning: Critical Limitation: Secrets are injected ONLY at container startup. Rotated secrets require container restart (new task launch).

Pattern B: Runtime Retrieval with Caching

For applications that need to handle rotation without restarts:

typescript
// Application code - retrieve secrets at runtimeimport { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager";
class SecretCache {  private cache: Map<string, { value: string; expiry: number }> = new Map();  private client: SecretsManagerClient;  private ttlMs: number;
  constructor(ttlMs: number = 300000) { // 5 minutes default    this.client = new SecretsManagerClient({});    this.ttlMs = ttlMs;  }
  async getSecret(secretId: string): Promise<string> {    const now = Date.now();    const cached = this.cache.get(secretId);
    if (cached && cached.expiry > now) {      return cached.value;    }
    const command = new GetSecretValueCommand({ SecretId: secretId });    const response = await this.client.send(command);    const value = response.SecretString!;
    this.cache.set(secretId, {      value,      expiry: now + this.ttlMs,    });
    return value;  }}
// Lambda handler exampleconst secretCache = new SecretCache(300000); // Cache for 5 minutes
export const handler = async (event: any) => {  const dbPassword = await secretCache.getSecret(process.env.DB_SECRET_ARN!);  // Use password for database connection};

Cost Analysis:

  • Startup injection: 1 API call per container start (~$0.05/10,000 calls)
  • Runtime retrieval with 5-min cache: 288 API calls/day per container ($1.44/month per container)
  • Runtime retrieval per request: Potentially thousands of API calls (expensive, not recommended)

Pattern C: AWS Parameters and Secrets Lambda Extension

For Lambda functions, use the extension for built-in caching:

typescript
// Lambda function using extensionexport const handler = async (event: any) => {  // Extension runs as sidecar, provides local HTTP endpoint  const response = await fetch(    `http://localhost:2773/secretsmanager/get?secretId=${process.env.SECRET_ARN}`,    {      headers: {        'X-Aws-Parameters-Secrets-Token': process.env.AWS_SESSION_TOKEN!,      },    }  );
  const secret = await response.json();  return { dbPassword: JSON.parse(secret.SecretString).password };};

Benefits:

  • Built-in caching (reduces API calls by ~90%)
  • No code changes to application logic for caching
  • Supports both Secrets Manager and Parameter Store

Deployment:

typescript
const lambdaFunction = new lambda.Function(this, 'Function', {  // ... other config  layers: [    lambda.LayerVersion.fromLayerVersionArn(      this,      'ParametersAndSecretsLayer',      `arn:aws:lambda:${this.region}:177933569100:layer:AWS-Parameters-and-Secrets-Lambda-Extension:11`    ),  ],  environment: {    PARAMETERS_SECRETS_EXTENSION_CACHE_ENABLED: 'true',    PARAMETERS_SECRETS_EXTENSION_CACHE_SIZE: '1000',    PARAMETERS_SECRETS_EXTENSION_HTTP_PORT: '2773',  },});

Tip: Cost Savings: Lambda extension reduces API calls by 99%. Costs drop from 5/monthto5/month to 0.05/month for high-traffic functions.

EKS Secrets with CSI Driver

For Kubernetes workloads on EKS, use the Secrets Store CSI Driver for native integration.

Architecture Setup

bash
# 1. Install Secrets Store CSI Driverhelm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/chartshelm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \  --namespace kube-system
# 2. Install AWS Providerkubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

SecretProviderClass Configuration

yaml
apiVersion: secrets-store.csi.x-k8s.io/v1kind: SecretProviderClassmetadata:  name: aws-secrets  namespace: productionspec:  provider: aws  parameters:    objects: |      - objectName: "prod/database"        objectType: "secretsmanager"        objectAlias: "db-creds"        jmesPath:          - path: username            objectAlias: db-username          - path: password            objectAlias: db-password      - objectName: "/app/config/api-endpoint"        objectType: "ssmparameter"        objectAlias: "api-endpoint"  # Optional: Sync to Kubernetes Secret  secretObjects:    - secretName: database-secret      type: Opaque      data:        - objectName: db-username          key: username        - objectName: db-password          key: password

Pod Configuration with IRSA

yaml
apiVersion: v1kind: ServiceAccountmetadata:  name: app-service-account  namespace: production  annotations:    eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/app-secrets-access
---apiVersion: apps/v1kind: Deploymentmetadata:  name: myapp  namespace: productionspec:  replicas: 3  selector:    matchLabels:      app: myapp  template:    metadata:      labels:        app: myapp    spec:      serviceAccountName: app-service-account      containers:        - name: app          image: myapp:latest          volumeMounts:            - name: secrets-store              mountPath: "/mnt/secrets"              readOnly: true          env:            - name: DB_USERNAME              valueFrom:                secretKeyRef:                  name: database-secret                  key: username            - name: DB_PASSWORD              valueFrom:                secretKeyRef:                  name: database-secret                  key: password      volumes:        - name: secrets-store          csi:            driver: secrets-store.csi.k8s.io            readOnly: true            volumeAttributes:              secretProviderClass: "aws-secrets"

IAM Role for Pod Identity

typescript
// CDK - Create IAM role for EKS Pod Identity (IRSA)const podRole = new iam.Role(this, 'PodSecretsRole', {  assumedBy: new iam.FederatedPrincipal(    cluster.openIdConnectProvider.openIdConnectProviderArn,    {      StringEquals: {        [`${cluster.openIdConnectProvider.openIdConnectProviderIssuer}:sub`]:          'system:serviceaccount:production:app-service-account',        [`${cluster.openIdConnectProvider.openIdConnectProviderIssuer}:aud`]:          'sts.amazonaws.com',      },    },    'sts:AssumeRoleWithWebIdentity'  ),});
// Grant access to secretsdbSecret.grantRead(podRole);configParam.grantRead(podRole);
// KMS permissions if using custom keykmsKey.grant(podRole, 'kms:Decrypt');

Key Difference - IRSA vs Pod Identity:

  • IRSA (older method): Requires OIDC provider setup, works with EKS 1.17+
  • Pod Identity (newer method, 2024+): Simplified setup, better performance, requires EKS 1.24+

Secret Rotation Implementation

Automated rotation is one of the key benefits of Secrets Manager. Here's how to implement it correctly.

Rotation Flow

Lambda Rotation Function - RDS MySQL

python
# Lambda rotation functionimport boto3import jsonimport pymysqlimport os
secrets_client = boto3.client('secretsmanager')
def lambda_handler(event, context):    secret_arn = event['SecretId']    token = event['ClientRequestToken']    step = event['Step']
    # Dispatch to appropriate step    if step == "createSecret":        create_secret(secrets_client, secret_arn, token)    elif step == "setSecret":        set_secret(secrets_client, secret_arn, token)    elif step == "testSecret":        test_secret(secrets_client, secret_arn, token)    elif step == "finishSecret":        finish_secret(secrets_client, secret_arn, token)    else:        raise ValueError(f"Invalid step: {step}")
def create_secret(client, arn, token):    # Check if version with AWSPENDING label already exists    try:        client.get_secret_value(            SecretId=arn,            VersionStage="AWSPENDING",            VersionId=token        )        print("Secret version already exists")        return    except client.exceptions.ResourceNotFoundException:        pass
    # Get current secret    current_secret = client.get_secret_value(        SecretId=arn,        VersionStage="AWSCURRENT"    )    secret_dict = json.loads(current_secret['SecretString'])
    # Generate new password    new_password = client.get_random_password(        ExcludeCharacters='/@"\'\\',        PasswordLength=32,        ExcludePunctuation=False,        RequireEachIncludedType=True    )
    secret_dict['password'] = new_password['RandomPassword']
    # Store new secret with AWSPENDING label    client.put_secret_value(        SecretId=arn,        ClientRequestToken=token,        SecretString=json.dumps(secret_dict),        VersionStages=['AWSPENDING']    )
def set_secret(client, arn, token):    # Get pending secret    pending_secret = client.get_secret_value(        SecretId=arn,        VersionStage="AWSPENDING",        VersionId=token    )    pending_dict = json.loads(pending_secret['SecretString'])
    # Get current secret for connection    current_secret = client.get_secret_value(        SecretId=arn,        VersionStage="AWSCURRENT"    )    current_dict = json.loads(current_secret['SecretString'])
    # Connect to database with current credentials    connection = pymysql.connect(        host=current_dict['host'],        user=current_dict['username'],        password=current_dict['password'],        database=current_dict['dbname'],        connect_timeout=5    )
    try:        with connection.cursor() as cursor:            # Update password in database            sql = f"ALTER USER '{pending_dict['username']}'@'%' IDENTIFIED BY %s"            cursor.execute(sql, (pending_dict['password'],))        connection.commit()    finally:        connection.close()
def test_secret(client, arn, token):    # Get pending secret    pending_secret = client.get_secret_value(        SecretId=arn,        VersionStage="AWSPENDING",        VersionId=token    )    pending_dict = json.loads(pending_secret['SecretString'])
    # Test connection with new credentials    connection = pymysql.connect(        host=pending_dict['host'],        user=pending_dict['username'],        password=pending_dict['password'],        database=pending_dict['dbname'],        connect_timeout=5    )
    try:        with connection.cursor() as cursor:            # Execute simple query to verify access            cursor.execute("SELECT 1")            result = cursor.fetchone()            if result[0] != 1:                raise ValueError("Test query failed")    finally:        connection.close()
def finish_secret(client, arn, token):    # Move AWSCURRENT label to new version    metadata = client.describe_secret(SecretId=arn)    current_version = None
    for version, stages in metadata['VersionIdsToStages'].items():        if "AWSCURRENT" in stages:            if version == token:                # Already current, nothing to do                return            current_version = version            break
    # Update version stages    client.update_secret_version_stage(        SecretId=arn,        VersionStage="AWSCURRENT",        MoveToVersionId=token,        RemoveFromVersionId=current_version    )

CDK Setup for Rotation

For RDS databases with built-in support:

typescript
// RDS databaseconst dbInstance = new rds.DatabaseInstance(this, 'Database', {  engine: rds.DatabaseInstanceEngine.mysql({    version: rds.MysqlEngineVersion.VER_8_0  }),  vpc,  credentials: rds.Credentials.fromGeneratedSecret('admin'),});
// Attach rotation to the secretdbInstance.secret!.addRotationSchedule('RotationSchedule', {  automaticallyAfter: cdk.Duration.days(30),  hostedRotation: secretsmanager.HostedRotation.mysqlSingleUser(),});

For custom applications:

typescript
// Lambda function for rotationconst rotationLambda = new lambda.Function(this, 'RotationFunction', {  runtime: lambda.Runtime.PYTHON_3_12,  handler: 'rotation.lambda_handler',  code: lambda.Code.fromAsset('lambda/rotation'),  timeout: cdk.Duration.minutes(5),  vpc,  vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS },});
// Grant permissionsdbSecret.grantRead(rotationLambda);dbSecret.grantWrite(rotationLambda);
// Attach rotationdbSecret.addRotationSchedule('CustomRotation', {  rotationLambda,  automaticallyAfter: cdk.Duration.days(30),});

Warning: Common Pitfalls:

  1. Network Access: Lambda needs VPC access to reach database. Configure VPC subnets correctly.
  2. Timeout: Default 3 seconds is too short. Set to 5 minutes for rotation.
  3. Permissions: Lambda needs both read and write to secret, plus KMS decrypt/encrypt.
  4. Idempotency: Always check if AWSPENDING version exists before creating new one.
  5. Connection Pooling: Open connections using old password won't automatically get new password. Applications should handle connection refresh.

Alternating Users Strategy

For zero-downtime rotation in high-availability applications:

Architecture:

  • Two database users: app_user_a and app_user_b
  • Both users have identical permissions
  • Rotation alternates which user's password is updated
  • Application always has one valid credential during rotation

Benefits:

  • No downtime window
  • Active connections continue working during rotation
  • Suitable for applications that can't handle connection refresh

Trade-off: Requires superuser credentials in separate secret to clone users.

Multi-Region Secrets Replication

For disaster recovery scenarios, Secrets Manager supports automatic replication.

Primary Secret with Replication

typescript
// Primary region secret with replicationconst primarySecret = new secretsmanager.Secret(this, 'PrimarySecret', {  secretName: 'prod/database-credentials',  description: 'Production database credentials',  replicaRegions: [    {      region: 'us-west-2',      encryptionKey: replicaKmsKey, // Optional: use different KMS key    },    {      region: 'eu-west-1',    },  ],});

ARN Structure:

  • Primary: arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/database-credentials-AbCdEf
  • Replica: arn:aws:secretsmanager:us-west-2:123456789012:secret:prod/database-credentials-AbCdEf

Key Points:

  • Secret suffix (-AbCdEf) is identical across regions
  • Replication is automatic and near real-time
  • Rotation in primary region propagates to replicas
  • Each replica is billed as separate secret ($0.40/month each)
  • Replicas are read-only, updates must happen in primary region

Disaster Recovery with Failover

typescript
// Application code with failover logicclass SecretService {  private primaryRegion = 'us-east-1';  private replicaRegion = 'us-west-2';  private secretName = 'prod/database-credentials';
  async getSecretWithFailover(): Promise<string> {    try {      // Try primary region first      return await this.getSecret(this.primaryRegion);    } catch (error) {      console.error('Primary region failed, trying replica', error);      // Fallback to replica region      return await this.getSecret(this.replicaRegion);    }  }
  private async getSecret(region: string): Promise<string> {    const client = new SecretsManagerClient({ region });    const command = new GetSecretValueCommand({      SecretId: this.secretName,    });
    const response = await client.send(command);    return response.SecretString!;  }}

Cost Optimization Alternative

Instead of replication, use cross-region secret access (higher latency, lower cost):

typescript
// Access secret in us-east-1 from us-west-2 applicationconst client = new SecretsManagerClient({ region: 'us-east-1' });const command = new GetSecretValueCommand({  SecretId: 'arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/database-credentials-AbCdEf',});
const response = await client.send(command);

Trade-off: Saves $0.40/month per replica but adds cross-region API latency (~50-150ms).

Break-Glass Emergency Access

Emergency access procedures are critical for incident response. Here's how to implement them securely.

Break-Glass Role Architecture

typescript
// CDK - Break-glass IAM role in all accountsexport class BreakGlassStack extends cdk.Stack {  constructor(scope: Construct, id: string, props?: cdk.StackProps) {    super(scope, id, props);
    // Break-glass role with broad permissions    const breakGlassRole = new iam.Role(this, 'BreakGlassRole', {      roleName: 'BREAK-GLASS-EMERGENCY-ACCESS',      description: 'Emergency access role for incident response',      maxSessionDuration: cdk.Duration.hours(2), // Short session      assumedBy: new iam.AccountPrincipal('111111111111'), // Management account    });
    // Grant access to all secrets    breakGlassRole.addToPolicy(new iam.PolicyStatement({      sid: 'SecretsManagerEmergencyAccess',      effect: iam.Effect.ALLOW,      actions: [        'secretsmanager:GetSecretValue',        'secretsmanager:DescribeSecret',        'secretsmanager:ListSecrets',      ],      resources: ['*'],    }));
    // Grant KMS decrypt for all keys    breakGlassRole.addToPolicy(new iam.PolicyStatement({      sid: 'KMSDecryptEmergencyAccess',      effect: iam.Effect.ALLOW,      actions: ['kms:Decrypt', 'kms:DescribeKey'],      resources: ['*'],      conditions: {        StringEquals: {          'kms:ViaService': [            `secretsmanager.${this.region}.amazonaws.com`,            `ssm.${this.region}.amazonaws.com`,          ],        },      },    }));
    // Grant Parameter Store access    breakGlassRole.addToPolicy(new iam.PolicyStatement({      sid: 'ParameterStoreEmergencyAccess',      effect: iam.Effect.ALLOW,      actions: [        'ssm:GetParameter',        'ssm:GetParameters',        'ssm:GetParametersByPath',      ],      resources: ['*'],    }));  }}

Monitoring Break-Glass Access

typescript
// EventBridge rule to detect break-glass accessconst breakGlassAlertRule = new events.Rule(this, 'BreakGlassAlert', {  eventPattern: {    source: ['aws.sts'],    detailType: ['AWS API Call via CloudTrail'],    detail: {      eventName: ['AssumeRole'],      requestParameters: {        roleArn: [{          prefix: 'arn:aws:iam::*:role/BREAK-GLASS-EMERGENCY-ACCESS'        }],      },    },  },});
// SNS topic for security teamconst securityTopic = new sns.Topic(this, 'SecurityAlertTopic', {  displayName: 'Critical Security Alerts',});
breakGlassAlertRule.addTarget(new targets.SnsTopic(securityTopic, {  message: events.RuleTargetInput.fromEventPath(    '$.detail.userIdentity.principalId has assumed break-glass role'  ),}));

Emergency Access Procedure

Activation:

  1. Security team retrieves break-glass password from physical safe
  2. Second person retrieves YubiKey from separate secure location
  3. Both must be present (two-person rule)

Access:

bash
# Configure AWS CLI with break-glass useraws configure --profile break-glass
# Assume role in target accountaws sts assume-role \  --role-arn arn:aws:iam::222222222222:role/BREAK-GLASS-EMERGENCY-ACCESS \  --role-session-name "incident-2024-11-30-database-outage" \  --serial-number arn:aws:iam::111111111111:mfa/EMERGENCY-BREAK-GLASS \  --token-code 123456 \  --profile break-glass
# Export temporary credentialsexport AWS_ACCESS_KEY_ID="..."export AWS_SECRET_ACCESS_KEY="..."export AWS_SESSION_TOKEN="..."
# Access secretsaws secretsmanager get-secret-value \  --secret-id prod/database-credentials \  --query SecretString \  --output text

Post-Incident:

  • Revoke temporary credentials immediately
  • Rotate all accessed secrets within 4 hours
  • Document all actions taken in incident report
  • Review CloudTrail logs for complete audit trail
  • Conduct post-mortem on why break-glass was needed

Audit Logging with CloudTrail

Comprehensive audit logging is essential for security and compliance.

CloudTrail Configuration

typescript
// S3 bucket for CloudTrail logsconst trailBucket = new s3.Bucket(this, 'CloudTrailBucket', {  bucketName: `cloudtrail-logs-${this.account}`,  encryption: s3.BucketEncryption.S3_MANAGED,  blockPublicAccess: s3.BlockPublicAccess.BLOCK_ALL,  versioned: true,  lifecycleRules: [    {      transitions: [        {          storageClass: s3.StorageClass.INTELLIGENT_TIERING,          transitionAfter: cdk.Duration.days(30),        },        {          storageClass: s3.StorageClass.GLACIER,          transitionAfter: cdk.Duration.days(90),        },      ],      expiration: cdk.Duration.days(2555), // 7 years for compliance    },  ],});
// CloudTrail trailconst trail = new cloudtrail.Trail(this, 'SecurityAuditTrail', {  bucket: trailBucket,  includeGlobalServiceEvents: true,  isMultiRegionTrail: true,  enableFileValidation: true,  sendToCloudWatchLogs: true,});
// Data events for Secrets Managertrail.addEventSelector({  readWriteType: cloudtrail.ReadWriteType.ALL,  includeManagementEvents: true,  dataResources: [    {      type: 'AWS::SecretsManager::Secret',      values: ['arn:aws:secretsmanager:*:*:secret:*'],    },  ],});

Warning: Important: CloudTrail only logs management events by default. Data events (including GetSecretValue) must be explicitly enabled.

Note: ~0.10/monthper100,000events.For10secretsaccessed1,000times/montheach:0.10/month per 100,000 events. For 10 secrets accessed 1,000 times/month each: 0.01/month.

Athena Queries for Analysis

sql
-- Find all secret access eventsSELECT  eventTime,  userIdentity.principalId,  userIdentity.arn,  eventName,  json_extract_scalar(requestParameters, '$.secretId') as secretId,  sourceIPAddress,  errorCodeFROM cloudtrail_logsWHERE eventSource = 'secretsmanager.amazonaws.com'  AND eventName IN ('GetSecretValue', 'PutSecretValue', 'DeleteSecret')  AND year = '2025' AND month = '12' AND day = '01'ORDER BY eventTime DESC;
-- Cross-account secret accessSELECT  eventTime,  userIdentity.accountId as callerAccount,  json_extract_scalar(requestParameters, '$.secretId') as secretArn,  eventName,  errorCodeFROM cloudtrail_logsWHERE eventSource = 'secretsmanager.amazonaws.com'  AND userIdentity.accountId != regexp_extract(    json_extract_scalar(requestParameters, '$.secretId'),    'arn:aws:secretsmanager:[^:]+:([^:]+):',    1  )  AND year = '2025' AND month = '12'ORDER BY eventTime DESC;

Cost Analysis & Optimization

Understanding the cost structure helps you optimize spending without compromising security.

Detailed Cost Scenarios

Scenario 1: 10 Static API Keys (No Rotation)

  • Parameter Store Standard: $0/month (free tier)
  • Secrets Manager: $4.00/month
  • Recommendation: Parameter Store Standard
  • Savings: $4.00/month

Scenario 2: 5 RDS Passwords (Monthly Rotation)

  • Parameter Store: $0.25/month + manual rotation labor + downtime risk
  • Secrets Manager: 2.00/month+2.00/month + 0 rotation = $2.00/month
  • Recommendation: Secrets Manager
  • ROI: Automation worth the cost

Scenario 3: Lambda with High Traffic

  • Without Extension: 1M invocations/month × 1 API call = $5.00/month
  • With Extension: API calls reduced by 99% = $0.05/month
  • Savings: $4.95/month (99% reduction)

Cost Optimization Strategies

Strategy 1: Hybrid Approach

Use Parameter Store for static configuration, Secrets Manager for rotating credentials:

typescript
// Static values in Parameter Store (free)const apiEndpoint = ssm.StringParameter.fromStringParameterAttributes(  this, 'ApiEndpoint', {    parameterName: '/app/api/endpoint',  });
// Rotating credentials in Secrets Manager (paid)const dbSecret = secretsmanager.Secret.fromSecretNameV2(  this, 'DbSecret',  'prod/database');

Strategy 2: Consolidate Secrets

Instead of separate secrets for each credential component:

typescript
// Wrong: Multiple secrets ($1.20/month)const dbUsername = new secretsmanager.Secret(this, 'DbUser');const dbPassword = new secretsmanager.Secret(this, 'DbPass');const dbHost = new secretsmanager.Secret(this, 'DbHost');
// Right: Single secret ($0.40/month)const dbCredentials = new secretsmanager.Secret(this, 'DbCreds', {  secretObjectValue: {    username: cdk.SecretValue.unsafePlainText('admin'),    password: cdk.SecretValue.unsafePlainText('generated'),    host: cdk.SecretValue.unsafePlainText('db.example.com'),    port: cdk.SecretValue.unsafePlainText('3306'),  },});// Savings: $0.80/month per database

Strategy 3: Selective Replication

Only replicate critical production secrets:

typescript
// Only replicate critical production database secretsif (secretName.includes('/prod/database') || secretName.includes('/prod/auth')) {  secret.addReplicaRegion('us-west-2', replicaKmsKey);}
// Non-critical secrets: use cross-region API calls// Acceptable for: API keys, static tokens, config values

Cost Analysis:

  • 20 secrets, 2 replicas: $24/month
  • 5 critical replicated + 15 primary only: $10/month
  • Savings: $14/month (58% reduction)

Common Pitfalls & Solutions

Here are the technical issues I've encountered and how to solve them.

Pitfall 1: Default KMS Key for Cross-Account Access

Problem: Cross-account sharing fails with "AccessDeniedException" when using default aws/secretsmanager key.

Root Cause: AWS-managed keys cannot have their policy modified for cross-account access.

Solution: Always create customer-managed KMS keys:

typescript
// Wrong: Uses default keyconst secret = new secretsmanager.Secret(this, 'Secret', {  secretName: 'shared-secret',});
// Right: Customer-managed keyconst kmsKey = new kms.Key(this, 'SharedSecretKey', {  enableKeyRotation: true,});kmsKey.addToResourcePolicy(/* cross-account policy */);
const secret = new secretsmanager.Secret(this, 'Secret', {  secretName: 'shared-secret',  encryptionKey: kmsKey,});

Pitfall 2: Lambda VPC Configuration for Rotation

Problem: Rotation Lambda times out connecting to RDS in VPC.

Root Cause: Lambda not configured with VPC access.

Solution:

typescript
const rotationLambda = new lambda.Function(this, 'RotationFunction', {  // ...  vpc: database.vpc,  vpcSubnets: {    subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS,  },  securityGroups: [rotationSecurityGroup],  timeout: cdk.Duration.minutes(5), // Not default 3 seconds});
// Allow Lambda to access RDSrotationSecurityGroup.addEgressRule(  database.connections.securityGroups[0],  ec2.Port.tcp(3306),  'Allow rotation Lambda to access database');

Pitfall 3: ECS Secrets Only Injected at Startup

Problem: After rotation, containers fail with authentication errors.

Root Cause: ECS injects secrets only at startup.

Solution: Implement graceful connection handling:

typescript
class DatabaseConnection {  private pool: any;  private secretId: string;  private secretCache: { value: string; expiry: number } | null = null;
  async getConnection() {    try {      return await this.pool.getConnection();    } catch (error) {      if (this.isAuthError(error)) {        console.log('Auth error detected, refreshing secret');        await this.refreshSecret();        this.pool = this.createPool(this.secretCache!.value);        return await this.pool.getConnection();      }      throw error;    }  }
  private async refreshSecret() {    const command = new GetSecretValueCommand({ SecretId: this.secretId });    const response = await secretsClient.send(command);    this.secretCache = {      value: response.SecretString!,      expiry: Date.now() + 300000, // 5 minutes    };  }}

Pitfall 4: Excessive Lambda API Calls

Problem: Secrets Manager costs spike to $50+/month.

Root Cause: Fetching secret on every invocation without caching.

Solution: Use Lambda Extension (shown earlier in Pattern C).

Result: 99% cost reduction.

Pitfall 5: Missing CloudTrail Data Events

Problem: No audit trail for GetSecretValue operations.

Root Cause: Data events not enabled by default.

Solution: Enable data event logging (shown in Audit Logging section).

Pitfall 6: Storing Non-Secret Config in Secrets Manager

Problem: Paying $0.40/month for non-sensitive values.

Solution: Use decision framework:

Is it sensitive? (password, API key, token)├─ YES → Can it rotate?│  ├─ YES → Secrets Manager ($0.40/month)│  └─ NO → Parameter Store SecureString (free)└─ NO → Parameter Store Standard (free)

Key Takeaways

Working with AWS secrets management has taught me these important lessons:

  1. Service Selection is About Use Case: Reserve Secrets Manager for rotating credentials. Use Parameter Store for everything else. This simple rule can save 80% on costs.

  2. Cross-Account Access Requires Customer-Managed Keys: The default aws/secretsmanager key won't work. Create customer-managed KMS keys from day one to avoid migration pain.

  3. Container Injection is One-Time: Secrets injected at startup don't update on rotation. Design applications to handle connection refresh or use alternating-users strategy.

  4. Lambda Extension Reduces Costs by 99%: For high-traffic Lambda functions, the extension's built-in caching is essential. It's a one-line addition that saves significant money.

  5. CloudTrail Data Events are Critical: Enable them from day one. The cost is negligible (~$0.10 per 100,000 events) but the audit value is immeasurable.

  6. Multi-Region Replication is a Business Decision: Don't replicate everything. Analyze RTO/RPO requirements and replicate only critical secrets. Cross-region API calls are often acceptable.

  7. Break-Glass Procedures Need Testing: Untested emergency access is useless during incidents. Test quarterly to validate both technical and organizational readiness.

  8. Automation Beats Process: Manual rotation costs 250/monthinengineertime.Automatedrotationcosts250/month in engineer time. Automated rotation costs 4/month. ROI is immediate.

The key is balancing security, cost, and operational complexity. Start simple with Parameter Store for static config, migrate sensitive credentials to Secrets Manager, implement rotation for databases, and add cross-region replication only where needed.

Related Topics:

aws
secrets-manager
parameter-store
security
kms
ecs
eks
lambda
aws-cdk
devops
tutorial

Related Posts