Mozilla SOPS: GitOps-Native Secret Encryption That Actually Works
A comprehensive guide to Mozilla SOPS for managing encrypted secrets in Git repositories. Learn age encryption, AWS CDK patterns, AWS Lambda integration, and production-ready security strategies for serverless workflows.
Abstract
Mozilla SOPS (Secrets OPerationS) solves a fundamental challenge in GitOps: how to safely commit secrets to version control while maintaining developer productivity. Unlike cloud-native secret stores, SOPS encrypts files directly in Git repositories, preserving YAML/JSON structure while protecting sensitive values. This guide covers practical implementation patterns including age encryption, AWS Lambda integration, AWS CDK workflows, AWS SAM patterns, and CI/CD automation for serverless deployments across GitHub Actions, GitLab CI, and Jenkins.
The GitOps Secret Management Challenge
Working with Infrastructure as Code creates an immediate problem. You need to version-control your serverless configuration files, Terraform variables, and environment configs. But those files contain database passwords, API keys, and service credentials. The moment you commit secrets to Git, you've created a security vulnerability.
Traditional solutions create friction. HashiCorp Vault requires running infrastructure and API calls at deployment time. AWS Secrets Manager costs $0.40 per secret per month and adds runtime API calls to your Lambda functions. AWS Systems Manager Parameter Store is free but still requires runtime fetching. Each approach pulls secrets out of your GitOps workflow and into external systems.
SOPS takes a different approach. It encrypts files directly in your Git repository, keeping secrets versioned alongside your code. When you change an API key and update your Lambda function, both changes go in the same commit. When you roll back, both roll back together. Your Git history becomes your audit trail.
Understanding SOPS Architecture
SOPS uses envelope encryption. When you encrypt a file, SOPS generates a random 256-bit data key and encrypts your file content with AES256-GCM. Then it encrypts that data key with one or more master keys (AWS KMS, age, PGP, GCP KMS, or Azure Key Vault) and stores the encrypted data key in the file's metadata. Age encrypts the SOPS data key using X25519 + ChaCha20-Poly1305.
For structured formats like YAML and JSON, SOPS only encrypts values, not keys. This keeps your file structure visible for code reviews and allows tools to parse the schema even when values are encrypted.
Here's what an encrypted YAML file looks like:
You can still see the database configuration structure. You know there's a host, port, and password field. But the actual values are encrypted. Git diff shows which fields changed, not just that "the encrypted blob changed."
Installation and Setup
Installation varies by platform but takes less than five minutes:
For container environments, use the official image:
Age: The Modern Encryption Choice
SOPS supports multiple key management systems. For team environments, age (pronounced like "h-age") has become the recommended choice over PGP.
Age public keys are 62 characters, private keys are 74 characters. PGP keys are 4096 characters. You can copy and paste an age public key in Slack. PGP keys break across lines. Age uses modern cryptography (X25519 + ChaCha20-Poly1305). PGP comes with decades of complexity from the GPG keyring system.
Generate an age key pair:
The private key is stored in ~/.config/sops/age/keys.txt. The public key is what you share with team members and configure in SOPS.
To encrypt a file with age:
For team distribution, each person generates their own age key and shares their public key. You configure SOPS to encrypt with all team members' public keys. Anyone with their private key can decrypt.
The .sops.yaml Configuration File
Creating a .sops.yaml file in your repository root eliminates manual key management. SOPS reads this file to determine which keys to use based on file paths.
Here's a production-ready configuration:
Now encryption becomes automatic:
The path-based rules eliminate human error. Developers don't need to remember which keys to use for which environment.
AWS KMS Integration
For production environments, AWS KMS provides centralized key management with IAM-based access control and audit logging through CloudTrail.
Create a KMS key:
Configure SOPS to use KMS:
For CI/CD environments, use IAM roles instead of storing credentials:
The IAM role needs KMS decrypt permissions:
Multi-Account AWS Setup
Enterprise environments rarely operate within a single AWS account. Development, staging, and production environments run in separate accounts for security isolation and blast radius containment. SOPS supports this architecture through environment-specific KMS keys and cross-account IAM permissions.
Architecture Overview
A typical multi-account setup separates environments into distinct AWS accounts, each with dedicated KMS keys. This prevents developers from accidentally accessing production secrets and provides clear security boundaries.
This architecture enforces several security principles. Developers need explicit cross-account role assumption to access each environment. KMS keys are account-local, so compromising one environment doesn't expose others. CloudTrail logs in each account provide independent audit trails.
KMS Keys per Environment Account
Each AWS account maintains its own KMS key. The .sops.yaml configuration maps file paths to account-specific KMS keys.
Directory structure mirrors the account separation:
When you encrypt a file in secrets/prod/, SOPS automatically uses the production account KMS key. No manual key selection required.
Cross-Account KMS Access
For CI/CD pipelines to decrypt secrets across accounts, KMS key policies must allow cross-account access. This requires configuration in both the KMS key policy and IAM role permissions.
KMS key policy in the production account (333333333333):
The condition restricts KMS usage to specific AWS services, preventing direct key access outside of legitimate deployment contexts.
IAM role in the CI/CD account (444444444444):
This IAM policy allows the CI/CD role to both decrypt with KMS keys and assume deployment roles in target accounts.
CI/CD with Role Assumption
GitHub Actions workflows assume different roles for different environments. The AWS credentials action supports role chaining for multi-account deployments.
The workflow separates deployment jobs by environment. Each job assumes the appropriate account role before decrypting secrets. The needs dependency ensures dev deploys before production. The environment: production adds manual approval gates.
Developer Workflow with AWS Profiles
Developers working locally need AWS profile configurations for each account. The ~/.aws/config file defines role assumption chains.
Production access requires MFA. When a developer runs SOPS commands for production secrets, AWS prompts for an MFA token.
Encrypting secrets for different environments:
Decrypting for local testing:
The profile selection happens through the AWS_PROFILE environment variable. SOPS automatically uses the correct KMS key based on file path and assumes the appropriate role based on the active profile.
Security Considerations
Multi-account SOPS deployments introduce several security requirements that must be enforced through IAM policies and organizational controls.
Principle of Least Privilege: Developers should access only the environments they actively work with. A junior developer working on development environments shouldn't have production KMS decrypt permissions. Role policies should reflect this segregation.
The explicit deny ensures even if higher-level policies grant access, this developer cannot decrypt production secrets.
Audit Logging with CloudTrail: Each AWS account should have CloudTrail enabled with logs shipped to a centralized security account. This creates an immutable audit trail of all KMS operations.
CloudTrail logs show who accessed which KMS key at what time. This enables detection of unauthorized access attempts or compliance audits.
Key Policies vs IAM Policies: Use both for defense in depth. KMS key policies define who can use the key at the resource level. IAM policies define what the identity can do. Both must allow the operation for it to succeed.
A production KMS key should have a restrictive key policy that only allows specific deployment roles, even if broader IAM policies exist elsewhere. This prevents privilege escalation through IAM policy changes alone.
Break-Glass Procedures: Even with MFA and restrictive policies, emergencies require rapid production access. Maintain an emergency access role with time-limited credentials and automatic alerting when used.
This policy allows emergency access but only within a time window. When the role is assumed, CloudWatch Events trigger alerts to security teams and management.
AWS Lambda Integration with SOPS
Lambda functions need secrets at runtime, but you want to version-control those secrets alongside your function code. SOPS enables this by decrypting secrets during deployment, not at runtime.
AWS CDK Integration
AWS CDK provides the most elegant integration with SOPS. CDK can decrypt secrets at synthesis time and inject them directly into Lambda environment variables, SSM parameters, or Secrets Manager. This approach keeps your infrastructure code clean while maintaining GitOps practices.
Create encrypted secrets:
Encrypt the file:
Pattern 1: Direct Environment Variable Injection
The simplest pattern decrypts SOPS at synthesis time and injects values as Lambda environment variables:
Pattern 2: SSM Parameter Store Population
A more flexible pattern uses SOPS to populate SSM Parameter Store, then references parameters in Lambda:
Your Lambda code fetches parameters at runtime:
Pattern 3: Using cdk-sops-secrets Construct
The cdk-sops-secrets npm package provides a higher-level construct:
Pattern 4: Multi-Stack Secret Sharing
For complex applications, share SOPS secrets across multiple stacks:
Synthesize and deploy:
AWS SAM Integration (Brief)
For teams using AWS SAM, decrypt SOPS files during deployment and populate Secrets Manager:
Local Development Workflow
For local testing with CDK, decrypt secrets temporarily:
Or use SOPS exec mode to run commands with decrypted environment:
For CDK synthesis locally, SOPS decrypts automatically:
SSM Parameter Store vs SOPS Comparison
Use SOPS when:
- Secrets change with code deployments
- You want Git-based audit trails
- Secrets are static (API keys, OAuth credentials)
- Team collaboration on secrets is important
- Cost optimization is priority
Use SSM Parameter Store when:
- Secrets rotate independently of deployments
- Multiple services share the same secrets
- You need AWS-native secret rotation
- Runtime secret updates without redeployment
- Cross-region secret replication needed
Hybrid approach:
Terraform Integration
The Terraform SOPS provider enables reading encrypted variable files while keeping your state file clean.
Configure the provider:
Create an encrypted variables file:
Encrypt it:
Reference in Terraform:
The password never appears in your Terraform state file in plaintext because we use ignore_changes. For initial creation, SOPS decrypts the value. For subsequent applies, Terraform ignores password changes.
A better pattern is using SOPS to populate AWS Secrets Manager, then referencing the secret ARN:
Now your application runtime fetches secrets from Secrets Manager, but the initial secret values are version-controlled with SOPS.
CI/CD Integration Patterns for AWS CDK
GitHub Actions with CDK
The recommended pattern for CDK deployments with SOPS. CDK automatically decrypts during synthesis:
CDK code decrypts SOPS during synthesis, so no explicit decrypt step needed in CI/CD.
The IAM role needs CDK deployment permissions plus KMS decrypt:
Multi-Environment CDK Deployment
Deploy to different environments with environment-specific secrets:
GitHub Actions with AWS SAM (Brief)
For teams using AWS SAM:
GitLab CI
GitLab CI uses similar patterns:
The decrypted secrets artifact is available to subsequent stages but expires after 10 minutes.
Developer Experience and IDE Integration
Editing encrypted files manually would be painful. SOPS provides an edit mode that handles encryption transparently.
Set your editor:
Edit an encrypted file:
SOPS decrypts the file, opens it in your editor, waits for you to save and close, then re-encrypts with updated values. You never see the encrypted content during editing.
For VS Code, install the SOPS extension:
The extension automatically decrypts files when you open them in VS Code and re-encrypts on save.
For meaningful Git diffs, configure a custom differ:
Now git diff secrets.enc.yaml shows the actual value changes, not encrypted blob differences.
Pre-commit Hooks and Validation
Prevent committing decrypted secrets with pre-commit hooks:
This hook verifies that files matching the pattern are encrypted before allowing the commit.
Add custom validation:
The hook prevents both accidentally committing plaintext secrets and committing files that claim to be encrypted but aren't.
Key Rotation Strategies
Age keys should rotate every 90 days. Automating this process prevents it from being forgotten.
Generate a new age key:
Add the new key to all encrypted files:
Rotate the data keys (generates new random keys):
Remove the old key:
Update .sops.yaml:
Commit and distribute the new private key to your team through a secure channel (password manager, encrypted email, secure messaging).
For KMS keys, the process is similar but involves creating a new KMS key, adding it to files, rotating, and removing the old KMS key ARN.
Multi-Team Access with Key Groups
Production environments often require multiple teams to access secrets. SOPS supports this through key groups and Shamir's Secret Sharing.
With shamir_threshold: 2, decryption requires keys from 2 of the 3 groups. This implements separation of duties. A platform engineer can't decrypt production secrets alone. Neither can the security team. But any two groups together can decrypt.
The data key is split into fragments using Shamir's Secret Sharing. Fragment 1 is encrypted for the platform team, fragment 2 for the security team, and fragment 3 for backup. Any 2 fragments can reconstruct the complete data key.
Cost Comparison and Trade-offs
For a scenario with 100 secrets:
SOPS with AWS KMS:
- KMS keys: 3 × 3
- API calls: ~1,000 decryptions/month = $0.03
- Git storage: $0 (existing repository)
- Total: $3.03/month
AWS Secrets Manager:
- Secrets: 100 × 40
- API calls: 10,000/month × 0.05
- Total: $40.05/month
Savings: $37/month (92% reduction)
But cost isn't the only consideration. Secrets Manager provides automated rotation, while SOPS requires scripting. Secrets Manager has built-in audit logs through CloudTrail. SOPS relies on Git history.
SOPS wins for static secrets (API keys, OAuth credentials, database connection strings that change infrequently). Secrets Manager wins for dynamic secrets (database passwords that rotate weekly, service credentials with automated renewal).
A hybrid approach works well:
- Development and staging: SOPS with age keys
- Production static secrets: SOPS with KMS
- Production dynamic secrets: AWS Secrets Manager
- Database root passwords: Secrets Manager
- Third-party API keys: SOPS
Common Pitfalls and Solutions
Pitfall: Committing Decrypted Files
Add to .gitignore:
Pitfall: Losing Age Private Keys
Store backups in multiple locations:
- Password manager (1Password, LastPass)
- Encrypted USB drive in physical safe
- Emergency recovery key stored offline
Create an emergency key and add it to all production secrets:
Pitfall: KMS Permission Issues
The error "AccessDeniedException" when decrypting usually means IAM permissions are wrong. Verify:
Ensure your IAM role has both kms:Decrypt and kms:DescribeKey permissions for the KMS key.
Pitfall: Git Merge Conflicts
When two developers edit the same encrypted file simultaneously, Git creates a merge conflict with encrypted blobs.
Note: For normal editing,
sops secrets.enc.yamldecrypts the file, opens it in your editor, and automatically re-encrypts when you save and exit. But for merge conflicts, you need to compare two different versions, so the manual decrypt → merge → re-encrypt workflow is required.
Resolving requires:
Better: communicate when editing shared secrets, or split large files into smaller domain-specific files to reduce collision probability.
Key Takeaways
SOPS enables GitOps workflows for serverless without external secret dependencies. Secrets are versioned with Lambda code, deployed together, and rolled back together. Your Git history becomes your audit trail.
Age encryption provides a modern, simple alternative to PGP. The keys are short enough to share in chat. The tooling is minimal. The onboarding time is under 30 minutes.
The .sops.yaml configuration file eliminates manual key management. Path-based rules automatically select the right keys for each environment. Developers encrypt files without knowing KMS ARNs or remembering which keys to use.
For Lambda deployments, SOPS decrypts at build/deploy time, not at runtime. This eliminates cold start overhead from fetching secrets. Environment variables are baked into function configuration during deployment.
AWS CDK provides the most elegant SOPS integration. CDK decrypts secrets during synthesis and supports multiple patterns: direct environment variable injection, SSM Parameter Store population, the cdk-sops-secrets construct, and multi-stack secret sharing. This keeps infrastructure code clean while maintaining GitOps practices.
AWS SAM also integrates well for teams preferring CloudFormation templates. SAM deployments can populate Secrets Manager from SOPS files, combining version-controlled secrets with AWS-native rotation.
For production environments, combining KMS with age provides both centralized management and emergency recovery. KMS handles the primary encryption with IAM-based access control. Age provides a backup decryption path if KMS is unavailable.
The cost savings compared to AWS Secrets Manager are significant for static secrets. SOPS works best for configuration that changes with CDK deploys (API keys, OAuth credentials). Use SSM Parameter Store or Secrets Manager for dynamic secrets that rotate independently (database passwords).
A hybrid approach maximizes value: SOPS for static secrets, SSM for dynamic secrets. This balances cost optimization with operational flexibility.
Pre-commit hooks and validation are mandatory, not optional. Without automated checks, someone will eventually commit a decrypted file. Set up guardrails before your team starts using SOPS in production.