Approaches for Real-time Updates of AWS Secrets Manager Secrets in Applications

1. Introduction

AWS Secrets Manager is a centralized service for storing and managing sensitive information like database credentials, API keys, and OAuth tokens.

For an introduction to this service, see our guide on How to Use AWS Secret Manager.

Secrets Manager helps organizations avoid hardcoding sensitive data in application source code, reducing the risk of accidental exposure and simplifying credential rotation.

Key features include:

• Encryption at rest using AWS Key Management Service (KMS)
• Access control through AWS Identity and Access Management (IAM) policies
• Automatic secret rotation
• Versioning support
• Audit logging via AWS CloudTrail

The Challenge: Applications often need to work with the current value of a secret. When a secret changes, manually updating applications is error-prone and inefficient.

The Solution: Automated mechanisms can propagate secret changes to applications, maintaining security and operational efficiency.

This report covers AWS services and application-level strategies for updating secrets dynamically.

2. Automatic Secret Rotation and its Implications

Automatic Secret Rotation Overview

Secrets Manager can rotate secrets automatically, which helps maintain security by periodically updating credentials.

Native Support for AWS Services: For Amazon RDS, Aurora, Redshift, and DocumentDB, Secrets Manager provides managed rotation with minimal configuration.

The rotation process involves:

• Selecting a rotation strategy:
  • Single-user approach
  • Alternating-users method
• Configuring rotation through:
  • AWS Management Console
  • AWS Command Line Interface (CLI)
• Behind the scenes:
  • Secrets Manager creates and manages an AWS Lambda function
  • The Lambda function performs the rotation steps
  • The rotation schedule uses cron or rate expressions

Custom Rotation for Other Services: For secret types beyond those natively supported:

• You can develop custom Lambda functions for rotation logic
• The Lambda function must update both the secret in Secrets Manager and the credential in the target service
• The secret invokes this Lambda function on a defined schedule

For a complete rotation setup with custom Lambda functions, see Secrets Manager Auto-Rotation.

The Gap Between Rotation and Application Update:

Automatic rotation updates the secret in Secrets Manager, but it does not automatically update the applications using that secret.

The default behavior has limitations:

• Applications using the AWS SDK will get the new value on their next request to Secrets Manager
• This is a pull-based mechanism and does not push updates to applications
• Many applications cache secrets to reduce calls to Secrets Manager
• With caching, an application might continue using an outdated secret until the cache expires

Key Challenge: Automatic rotation is important for security, but additional mechanisms help applications receive rotated secret values promptly.

Understanding Staging Labels in the Rotation Process

During rotation, Secrets Manager uses staging labels to manage different versions:

How promotion works:

1. When a new version is created during rotation, it’s initially labeled AWSPENDING
2. After successful testing, it gets promoted to AWSCURRENT
3. The previous AWSCURRENT version becomes AWSPREVIOUS

Application consideration: Applications could be designed to understand staging labels, but this adds complexity to secret retrieval logic and is not recommended for most use cases.

3. Leveraging AWS EventBridge for Real-time Secret Change Detection

What is AWS EventBridge? EventBridge is a serverless event bus service that connects applications using events from AWS services, SaaS applications, and custom applications.

For more on event-driven architectures, see our article on S3 Events with Event Bus Routing and Filtering.

Key benefits:

• Supports decoupled architecture
• Services can react to events without knowing event sources
• EventBridge matches incoming events against defined rules
• Events route to specified targets

Integration with Secrets Manager: EventBridge can capture Secrets Manager events. Secrets Manager logs API calls and rotation events to CloudTrail, which records AWS API calls.

EventBridge listens for CloudTrail entries related to Secrets Manager. Several events are relevant for triggering application updates:

• PutSecretValue: Generated when a new secret value is added, whether through manual update or automatic rotation.
• UpdateSecret: Occurs when secret metadata, such as description or tags, is updated.
• RotateSecret: Signals the start of a secret rotation process.
• RotationSucceeded: Indicates that a secret rotation completed successfully.

Configuring EventBridge Rules:

EventBridge rules filter for secrets-related events based on:

• The source service (aws.secretsmanager)
• The specific event name

Example configurations:

• A rule matches all changes to a specific secret by specifying its ARN
• A rule triggers specifically when a secret value rotates

For configuring EventBridge rules, see our tutorial on Auto Scaling Lifecycle Hooks which covers similar event-driven patterns.

EventBridge Limitations:

• Events from API actions beginning with List, Get, or Describe are generally not processed (with some exceptions for AWS Security Token Service actions)
• A direct call to GetSecretValue by an application does not generate an EventBridge event
• However, calls that modify the secret value or state, such as PutSecretValue or RotateSecret, are captured

For EventBridge to capture Secrets Manager events, CloudTrail must be enabled in the AWS account and configured to log management events. The source field in the EventBridge rule pattern should be set to aws.secretsmanager.

Using EventBridge this way establishes a real-time mechanism to detect secret changes and trigger applications to retrieve the latest values.

4. Methods for Triggering Application Updates via EventBridge

Once EventBridge detects changes in Secrets Manager secrets, several methods can trigger updates in the application:

4.1. Invoking an AWS Lambda function

A common approach is configuring the EventBridge rule to invoke a Lambda function as its target. When a matching event occurs, EventBridge triggers the Lambda function.

The Lambda function can:

• Retrieve the updated secret value from Secrets Manager using the AWS SDK
• Update the application’s configuration. This might involve:
  • Writing the new secret to a configuration file accessible to the application
  • Updating an in-memory configuration cache
  • Sending a signal to the application process to reload its secrets
  • For containerized applications, triggering a new deployment or rolling restart

The Lambda function needs appropriate permissions. Its IAM role must allow secretsmanager:GetSecretValue on the secret that triggered the event.

Depending on how application updates are triggered, the Lambda function might need permissions to interact with ECS, EKS, or S3.

Consider these factors when using Lambda:

• Cold starts might delay the secret update process
• Managing Lambda deployment and updates requires attention
• Lambda invocation costs based on requests and compute duration matter for overall cost

4.2. Sending notifications via Amazon SNS

Another approach configures the EventBridge rule to publish a message to an SNS topic. The application subscribes to this SNS topic to receive notifications when the secret updates.

The SNS message contains information about the changed secret, such as its ARN. Upon receiving notification, the application retrieves the latest secret value from Secrets Manager and updates its configuration.

Applications can subscribe to SNS topics through:

• Amazon Simple Queue Service (SQS) queues
• Lambda functions
• HTTP/HTTPS endpoints
• Email addresses
• SMS messages

This method requires:

• Setting up and managing the SNS topic and subscriptions
• Designing the application to receive and process SNS notifications

SNS provides a highly available and scalable notification service, though message delivery might have delays.

4.3. Placing a message on an Amazon SQS queue

Similar to SNS, an EventBridge rule can send a message to an SQS queue when a Secrets Manager event occurs. Processing events with SQS? See SQS Lambda Event Source Mapping for error handling patterns.

The application or a worker process polls the SQS queue for new messages. When a message indicating a secret update arrives, the application retrieves the latest secret value and updates its configuration.

The SQS message contains metadata about the changed secret. This approach requires:

• Setting up and managing SQS queues
• Implementing polling logic, which introduces delay proportional to the polling frequency

SQS guarantees at-least-once delivery. Applications should handle secret updates idempotently to avoid side effects from processing the same update multiple times.

4.4. Using EventBridge API Destinations

A more direct method uses EventBridge API Destinations. This feature allows an EventBridge rule to directly invoke an HTTPS endpoint of the application when a matching event occurs.

Configuration requirements:

• A connection must be configured in EventBridge, including authorization details and credentials to access the application’s endpoint
• The credentials for this connection can be stored in Secrets Manager with the “events!” prefix
• The HTTP request body can be customized using EventBridge Input Transformers to include relevant information like the updated secret’s ARN

Application requirements:

• The application must expose a publicly accessible or privately reachable HTTPS endpoint
• The endpoint must receive these calls and trigger a secret refresh
• Securing this endpoint is critical to prevent unauthorized access

Limitations:

• EventBridge API Destinations have a maximum client execution timeout of 5 seconds
• EventBridge retries failed requests based on HTTP error codes and the Retry-After header

5. Application-Side Strategies for Dynamic Secret Retrieval

Applications can also implement their own strategies to ensure they use the latest secrets.

5.1. Implementing logic to periodically fetch the secret

One approach is to periodically call the GetSecretValue API using the AWS SDK.

The frequency balances near real-time updates with the cost of frequent API calls.

Characteristics of periodic fetching:

• Reactive by nature: the application only discovers secret changes at the next polling interval
• Straightforward to implement
• Can lead to unnecessary API calls if secrets change infrequently
• May not suit applications requiring immediate updates when secrets rotate
• Works for applications where slight delay in secret updates is acceptable

5.2. Using AWS SDKs with built-in caching mechanisms

AWS provides client-side caching libraries as part of its SDKs for .NET, Java, Python, Go, and Rust to optimize performance and reduce costs.

How SDK caching works:

• Cache components store retrieved secrets in memory for a configurable duration (TTL)
• Subsequent requests within the TTL period are served from the local cache
• This avoids repeated calls to Secrets Manager
• When TTL expires, the cache automatically refreshes the secret on the next request

Advanced cache management:

• Some SDKs (like Java) offer methods such as refreshNow for programmatic cache refresh
• This is useful when combined with EventBridge notifications
• Upon receiving a notification of a secret change, the application can trigger an immediate cache refresh

SDK-specific considerations:

• The .NET and Go caching components do not have explicit cache invalidation mechanisms
• Integrating these libraries requires adding dependencies and modifying secret retrieval logic
• Understanding the default TTL behavior is important

Understanding the default TTL and cache behavior is crucial to avoid using stale secrets for extended periods.

5.3. Utilizing the AWS Secrets Manager Agent

The AWS Secrets Manager Agent is a client-side HTTP service that standardizes how applications consume secrets from Secrets Manager across Lambda, ECS, EKS, and EC2.

How the Agent works:

• The Agent retrieves secrets from Secrets Manager and caches them in memory
• Applications fetch secrets from the Agent’s local HTTP endpoint instead of calling the Secrets Manager API directly
• The Agent refreshes its cache based on a configurable TTL (default: 300 seconds)
• The Agent provides an API endpoint to force immediate refresh of specific secrets

Integration with event-driven architecture: This capability works with EventBridge notifications. When a secret updates in Secrets Manager, the application can promptly retrieve the new value from the local Agent.

Trade-offs:

• Benefits: Simplifies secret retrieval and provides efficient caching
• Challenges: Adds a component to manage in the application deployment
• Best practice: Understanding the Agent’s TTL and refresh mechanisms ensures the application operates with up-to-date secrets

5.4. For containerized applications (ECS, EKS)

For applications in ECS and EKS, specific integration mechanisms with Secrets Manager are available.

ECS Integration:

• Secrets can be injected into container definitions as environment variables or log configurations
• Existing containers do not automatically receive updated secret values
• A new task deployment or force new deployment of the service propagates the new secret
• This can be triggered via the ECS API in response to an EventBridge event

EKS Integration:

• The AWS Secrets and Configuration Provider (ASCP) for Kubernetes synchronizes secrets from Secrets Manager to Kubernetes Secrets
• Applications in Kubernetes pods consume secrets using standard Kubernetes mechanisms
• The External Secrets Operator also achieves this synchronization
• These solutions can automatically update the Kubernetes Secret when the Secrets Manager secret changes
• Depending on the Kubernetes deployment strategy, this might trigger a rolling update or restart of pods

Using these integrations requires:

• Configuring the container orchestration service to communicate with Secrets Manager
• Ensuring IAM roles for ECS tasks or EKS pods have permissions to access the required secrets

These methods do not provide instantaneous updates to running containers, but they ensure new deployments use the latest secret values.

6. Considerations and Best Practices

When implementing approaches for automatically retrieving updated secrets, several considerations apply.

6.1. Security Implications

Security should be a primary concern:

• Follow the principle of least privilege when granting Lambda functions or other AWS services access to Secrets Manager
• If using EventBridge API Destinations, secure the application’s endpoint appropriately
• Audit access to secrets through CloudTrail and Amazon CloudWatch

For security strategies with sensitive information, see our article on Protect Passwords in Cloud Full Project Review.

6.2. Handling Latency and Ensuring Eventual Consistency

Event-driven architectures provide near real-time updates but might introduce latency between secret changes and application updates.

Key considerations:

• Determine acceptable staleness levels based on the application’s requirements
• Implement retry mechanisms to handle transient errors
• Design applications with eventual consistency in mind for secret updates

6.3. Error Handling and Retry Mechanisms

Implement robust error handling in Lambda functions and application logic responsible for retrieving and updating secrets.

Recommended practices:

• Configure dead-letter queues (DLQs) for failed events that can be analyzed and reprocessed
• Implement retry logic with exponential backoff for transient failures
• Set up monitoring and alerting for persistent failures

6.4. Cost Implications

Evaluate the cost of the chosen approach.

Cost considerations include:

• EventBridge events and rule executions
• Lambda function invocations
• SNS notifications and deliveries
• SQS message storage and processing
• Secrets Manager API calls

Cost optimization strategies:

• Use client-side caching or the Secrets Manager Agent to reduce API calls to Secrets Manager
• If using periodic fetching, balance polling intervals between responsiveness and cost
• Implement batching where appropriate

6.5. Best Practices for Managing Secrets

Follow best practices for managing secrets in Secrets Manager:

• Store all sensitive information in Secrets Manager
• Enable automatic secret rotation wherever feasible
• Use client-side caching or the Secrets Manager Agent to improve performance and reduce costs
• Avoid hardcoding secrets in application code or storing them as environment variables
• Make applications environment-agnostic and manage secret declarations as code using CloudFormation or Terraform

For more security strategies, see our article on Five Ways to Enhancement of Your Content Security.

7. Conclusion

This report covered several approaches for ensuring applications automatically retrieve the latest values of Secrets Manager secrets when they change:

• Using EventBridge to trigger Lambda functions, SNS topics, SQS queues, and API Destinations
• Implementing application-side strategies like periodic fetching, SDK caching, and the Secrets Manager Agent
• For containerized environments like ECS and EKS, using specific integration mechanisms

The optimal approach depends on specific requirements:

• The need for real-time updates
• Application architecture complexity
• Cost considerations
• Operational overhead tolerance

For applications requiring immediate updates, EventBridge-based solutions provide a reactive mechanism. The target choice (Lambda, SNS, SQS, or API Destination) should align with the application’s existing infrastructure.

For simpler applications where slight delays are acceptable, periodic fetching or SDK caching with a reasonable TTL might suffice. Combining client-side caching with EventBridge notifications to trigger cache refresh balances performance, cost, and timely updates.

Containerized applications benefit from ECS and EKS integration features.

If you want to go further with EventBridge beyond Secrets Manager notifications, the natural next step is combining it with Step Functions for full workflow orchestration. I wrote about EventBridge + Step Functions patterns for event-driven architecture — covering fan-out, saga patterns, error handling, and how to decide when Step Functions adds value versus when SQS or Lambda alone is enough.

The key is balancing responsiveness to secret changes, security, costs, and solution complexity.

With careful consideration of these factors and appropriate strategies, applications can operate with current, secure credentials managed by Secrets Manager. For more AWS security best practices, see our article on The Ultimate Guide to Ensure Cybersecurity for Small Businesses.

Table 1: Comparison of Approaches for Reacting to Secrets Manager Changes

Table 2: EventBridge Rule Pattern Examples