Approaches for Real-time Updates of AWS Secrets Manager Secrets in Applications

1. Introduction

AWS Secrets Manager serves as a centralized and secure service for managing sensitive information, commonly referred to as secrets. These secrets can include:

• Database credentials
• API keys
• OAuth tokens
• Other critical application credentials

For a comprehensive introduction to this service, check out our guide on How to Use AWS Secret Manager.

By utilizing Secrets Manager, organizations can significantly enhance their security posture by:

• Eliminating the need to hardcode sensitive data directly within application source code
• Reducing the risk of inadvertent exposure
• Simplifying the management and rotation of these critical credentials

Key features of AWS Secrets Manager include:

• Robust encryption at rest using AWS Key Management Service (KMS)
• Granular access control through AWS Identity and Access Management (IAM) policies
• The capability for automatic secret rotation
• Support for versioning secrets
• Comprehensive audit logging via AWS CloudTrail

The Challenge: A significant challenge arises in ensuring that applications consistently utilize the most current value of a secret stored in Secrets Manager, particularly when that secret undergoes a change. Relying on manual updates within applications following a secret modification is an inefficient and potentially error-prone process.

The Solution Approach: There is a clear need for automated and reactive mechanisms to propagate secret changes to applications in order to maintain:

• Security integrity
• Operational efficiency
• Consistent credential management

This report will explore various approaches that enable applications to automatically retrieve updated AWS Secrets Manager secrets whenever a change occurs, ensuring they are always operating with the latest and most secure credentials.

The structure of this report will cover different AWS services and application-level strategies that facilitate this dynamic update process.

2. Automatic Secret Rotation and its Implications

Automatic Secret Rotation Overview

AWS Secrets Manager offers a valuable feature for automatically rotating secrets, which is crucial for maintaining a strong security stance by periodically updating credentials.

Native Support for AWS Services: For AWS services such as Amazon RDS, Aurora, Redshift, and DocumentDB, Secrets Manager provides managed rotation capabilities with minimal configuration.

The rotation process involves:

• Selecting a suitable rotation strategy:
  • Single-user approach
  • Alternating-users method
• Configuring automatic rotation through:
  • AWS Management Console
  • AWS Command Line Interface (CLI)
• Behind the scenes implementation:
  • Secrets Manager creates and manages an AWS Lambda function
  • This Lambda function performs the actual rotation steps
  • The rotation schedule uses cron or rate expressions for flexible timing

Custom Rotation for Other Services: For secret types beyond those natively supported:

• Users can develop custom AWS Lambda functions to implement rotation logic
• The Lambda function must update both:
  • The secret value in Secrets Manager
  • The corresponding credential in the target service or application
• The secret is then configured to invoke this custom Lambda function according to a defined schedule

The Gap Between Rotation and Application Update:

While the automatic secret rotation feature ensures that the secret value stored within Secrets Manager is regularly updated, it does not inherently trigger an immediate update within the applications consuming that secret.

The default behavior has limitations:

• Applications that actively retrieve the secret using the AWS SDK will eventually obtain the new value upon their next request to Secrets Manager
• This is a pull-based mechanism and does not proactively push the updated secret to the application
• Many applications employ client-side caching to reduce the frequency of calls to Secrets Manager
• With caching enabled, the application might continue to use an outdated secret value until the cache entry expires or is explicitly refreshed

Key Challenge: While automatic rotation is a critical security practice, additional mechanisms are often needed to ensure applications are promptly aware of and retrieve the newly rotated secret value.

Understanding Staging Labels in the Rotation Process

During rotation, AWS Secrets Manager uses staging labels to manage different versions of the secret:

How the promotion process works:

1. When a new version is created during rotation, it’s initially labeled AWSPENDING
2. After successful testing, it gets promoted to AWSCURRENT
3. The previous AWSCURRENT version becomes AWSPREVIOUS

Application design consideration: While applications could be designed to understand and utilize these staging labels to manage transitions during rotation, this approach introduces additional complexity to the application’s secret retrieval logic and is generally not recommended for most use cases.

3. Leveraging AWS EventBridge for Real-time Secret Change Detection

What is AWS EventBridge? AWS EventBridge is a serverless event bus service that facilitates the building of event-driven applications by enabling the connection of applications with data from a wide array of sources, including:

• Other AWS services
• Software-as-a-Service (SaaS) applications
• Custom applications

For a deeper understanding of event-driven architectures with AWS, see our article on S3 Events with Event Bus Routing and Filtering.

Key benefits:

• Promotes a decoupled architecture
• Services can react to events without needing explicit knowledge of the event sources
• EventBridge operates based on rules that match incoming events against defined patterns
• Events are routed to specified targets for processing

Integration with Secrets Manager: To enable applications to react to changes in AWS Secrets Manager secrets in real time, EventBridge can be configured to capture relevant events. Secrets Manager logs various API calls and rotation-related events to AWS CloudTrail, a service that records AWS API calls for an account.

EventBridge can then be set up to listen for and match specific CloudTrail log entries pertaining to Secrets Manager. Several key events within Secrets Manager are particularly relevant for triggering application updates:

• PutSecretValue: This event is generated whenever a new secret value is added to a secret, whether through a manual update or as part of an automatic rotation process.
• UpdateSecret: This event occurs when the metadata of a secret, such as its description or tags, is updated.
• RotateSecret: This event signals the initiation of a secret rotation process for a given secret.
• RotationSucceeded: This event indicates that a secret rotation process has completed successfully.

Configuring EventBridge Rules:

EventBridge rules can be defined with specific patterns to filter for secrets-related events based on criteria such as:

• The source service (aws.secretsmanager)
• The specific event name

Example rule configurations:

• A rule can be created to match all changes to a specific secret by specifying its Amazon Resource Name (ARN) in the event pattern
• A rule can be designed to trigger specifically when a secret value is rotated

For a practical example of configuring EventBridge rules, see our tutorial on Auto Scaling Lifecycle Hooks which demonstrates similar event-driven patterns.

Important EventBridge Limitations:

• Events originating from API actions that begin with List, Get, or Describe are generally not processed by EventBridge (with a few exceptions for certain AWS Security Token Service actions)
• This means a direct call to GetSecretValue by an application will not generate an EventBridge event
• However, API calls that modify the secret’s value or its state, such as PutSecretValue or RotateSecret, will be captured

For EventBridge to effectively capture these Secrets Manager-related events, it is crucial that AWS CloudTrail is enabled in the AWS account and configured to log management events. The source field in the EventBridge rule pattern should be set to aws.secretsmanager to specifically target events originating from this service.

By leveraging EventBridge in this manner, organizations can establish a real-time mechanism to detect when secrets are changed in Secrets Manager and subsequently trigger actions to ensure their applications retrieve and utilize the latest values.

4. Methods for Triggering Application Updates via EventBridge

Once AWS EventBridge is configured to detect changes in Secrets Manager secrets, several methods can be employed to trigger an update within the application:

4.1. Invoking an AWS Lambda function

One common approach is to configure the EventBridge rule to invoke an AWS Lambda function as its target. When an event matching the rule’s pattern occurs (e.g., a secret value is updated), EventBridge will automatically trigger the specified Lambda function.

The logic within this Lambda function can be designed to:

• Retrieve the updated secret value from Secrets Manager using the AWS SDK.
• Take various actions to update the application’s configuration or trigger a refresh. This might involve:
  • Writing the new secret to a configuration file stored in a location accessible to the application.
  • Updating an in-memory configuration cache.
  • Sending a signal to the application process to initiate a reload of its secrets.
  • In the case of containerized applications, potentially triggering a new deployment or a rolling restart of the application containers.

To ensure the Lambda function can successfully perform these actions, it is essential to configure the appropriate permissions. The function’s IAM role must have permission to execute the secretsmanager:GetSecretValue action on the specific secret that triggered the event.

Additionally, depending on how the application update is being triggered, the Lambda function might require permissions to interact with other AWS services, such as the Amazon Elastic Container Service (ECS) or Amazon Elastic Kubernetes Service (EKS) APIs for restarting containers, or permissions to write to Amazon Simple Storage Service (S3) if the application reads its configuration from there.

Several factors should be considered when using Lambda functions in this manner:

• Cold starts, which occur when a Lambda function is invoked after a period of inactivity, might introduce a slight delay in the secret update process.
• Managing the deployment and updates of the Lambda function itself is an operational consideration.
• The cost of Lambda invocations, which is based on the number of requests and the compute duration, should be factored into the overall solution.

4.2. Sending notifications via Amazon SNS

Another approach is to configure the EventBridge rule to publish a message to an Amazon Simple Notification Service (SNS) topic. The application, or a component thereof, can then subscribe to this SNS topic to receive notifications whenever the secret is updated.

The SNS message itself can contain relevant information about the secret that was changed, such as its ARN. Upon receiving an SNS notification, the application can then use the AWS SDK to retrieve the latest secret value from Secrets Manager and update its internal configuration accordingly.

Applications can subscribe to SNS topics through various endpoints, including:

• Amazon Simple Queue Service (SQS) queues
• Lambda functions
• HTTP/HTTPS endpoints
• Email addresses
• SMS messages

Implementing this method requires:

• Setting up and managing the SNS topic and the corresponding subscriptions.
• Designing the application with the logic to receive and process SNS notifications.

While SNS provides a highly available and scalable notification service, there might be potential message delivery delays, which should be considered based on the application’s requirements.

4.3. Placing a message on an Amazon SQS queue

Similar to using SNS, an EventBridge rule can be configured to send a message to an Amazon SQS queue when a Secrets Manager event occurs.

The application, or a dedicated worker process, can then periodically poll the SQS queue to check for new messages. When a message indicating a secret update is received, the application can retrieve the latest secret value from Secrets Manager and update its configuration.

The SQS message can contain metadata about the secret that was changed, such as its identifier. This approach necessitates:

• Setting up and managing the SQS queues.
• Implementing the logic for polling the queue, which introduces a delay proportional to the polling frequency.

While SQS guarantees at-least-once delivery of messages, applications should be designed to be idempotent in handling secret updates to avoid any unintended side effects from processing the same update message multiple times.

4.4. Using EventBridge API Destinations

A more direct method for notifying an application of a secret change is by using EventBridge API Destinations. This feature allows an EventBridge rule to directly invoke an HTTPS endpoint of the application when a matching event occurs.

Configuration requirements:

• A connection must be configured within EventBridge, including:
  • Authorization method details
  • Credentials required to access the application’s endpoint
• The credentials for this connection can themselves be stored in Secrets Manager
  • These secrets are typically managed by EventBridge
  • They have a specific naming convention (prefixed with “events!”)
• The HTTP request body can be customized using EventBridge Input Transformers
  • This allows inclusion of relevant information like the ARN of the updated secret

Application requirements:

• The application must expose a publicly accessible or privately reachable HTTPS endpoint
• This endpoint must be able to receive these calls and trigger a secret refresh
• Security of this endpoint is critical to prevent unauthorized access

Important limitations:

• EventBridge API Destinations have a maximum client execution timeout of 5 seconds
• EventBridge will automatically retry failed requests based on:
  • Specific HTTP error codes
  • The Retry-After header provided by the application’s endpoint

5. Application-Side Strategies for Dynamic Secret Retrieval

Beyond relying on EventBridge to trigger updates, applications can also implement strategies on their own to ensure they are using the latest secrets from Secrets Manager.

5.1. Implementing logic to periodically fetch the secret

One of the simplest approaches is to implement logic within the application to periodically call the GetSecretValue API of Secrets Manager using the AWS SDK.

The frequency of these calls needs to be carefully determined, balancing the need for near real-time updates with the potential cost implications of making frequent API calls to Secrets Manager.

Characteristics of the periodic fetching approach:

• Inherently reactive - the application only discovers secret changes at the next polling interval
• Relatively straightforward to implement
• Can lead to unnecessary API calls if secrets don’t change frequently
• May not be suitable for applications requiring immediate updates when secrets are rotated
• Works well for applications where a slight delay in secret updates is acceptable

5.2. Using AWS SDKs with built-in caching mechanisms

To optimize performance and reduce the cost of repeatedly fetching secrets, AWS provides client-side caching libraries as part of its SDKs for various programming languages, including .NET, Java, Python, Go, and Rust.

How SDK Caching Works:

• Caching components store retrieved secrets in memory for a configurable duration (Time-to-Live or TTL)
• Subsequent requests for the same secret within the TTL period are served from the local cache
• This avoids the need to make repeated calls to the Secrets Manager API
• When the TTL expires, the cache automatically refreshes the secret from Secrets Manager on the next request

Advanced Cache Management:

• Some SDKs (like Java) offer methods such as refreshNow that allow programmatically forcing a cache refresh
• This is particularly useful when combined with EventBridge notifications
• When a notification of a secret change is received, the application can trigger an immediate cache refresh

SDK-Specific Considerations:

• The .NET and Go caching components do not offer explicit cache invalidation mechanisms
• Integrating these caching libraries requires:
  • Adding the appropriate dependencies
  • Modifying the secret retrieval logic to use the cache
  • Understanding the default TTL behavior

Important: Understanding the default TTL and behavior of the cache is crucial to ensure the application does not inadvertently use stale secrets for an extended period.

5.3. Utilizing the AWS Secrets Manager Agent

What is the Secrets Manager Agent? The AWS Secrets Manager Agent is a client-side HTTP service that can be deployed alongside an application to standardize the way secrets are consumed from Secrets Manager across various environments, including Lambda, ECS, EKS, and EC2.

How the Agent works:

• The Agent retrieves secrets from Secrets Manager and caches them in memory
• Applications fetch required secrets from the Agent’s local HTTP endpoint instead of directly calling the Secrets Manager API
• The Agent periodically refreshes its cache based on a configurable TTL (default: 300 seconds)
• The Agent provides an API endpoint that can be used to force an immediate refresh of specific secret values

Integration with event-driven architecture: This capability can be leveraged in conjunction with EventBridge notifications to ensure that when a secret is updated in Secrets Manager, the application can promptly retrieve the new value from the local Agent.

Trade-offs to consider:

• Benefits: Simplifies secret retrieval and provides efficient caching
• Challenges: Introduces an additional component to the application’s deployment that needs to be managed
• Best practice: Understanding the Agent’s TTL and refresh mechanisms is essential for ensuring the application operates with up-to-date secrets

5.4. For containerized applications (ECS, EKS)

For applications running within container orchestration services like ECS and EKS, specific integration mechanisms with AWS Secrets Manager are available.

ECS Integration:

• Secrets can be injected directly into container definitions as environment variables or into log configurations.
• Existing containers do not automatically receive updated secret values when the secret is changed in Secrets Manager.
• To propagate the new secret, a new task deployment or a “force new deployment” of the service is typically required.
• This action can be triggered programmatically via the ECS API in response to an EventBridge event indicating a secret update.

EKS Integration:

• The AWS Secrets and Configuration Provider (ASCP) for Kubernetes can synchronize secrets from Secrets Manager to Kubernetes Secrets.
• This allows applications running within Kubernetes pods to consume secrets using standard Kubernetes mechanisms.
• The External Secrets Operator is another tool that can achieve this synchronization.
• These solutions can be configured to automatically update the Kubernetes Secret when the corresponding secret in Secrets Manager is changed.
• This potentially triggers a rolling update or restart of the pods that consume the secret, depending on the deployment strategy configured in Kubernetes.

Utilizing these integration methods requires:

• Configuring the container orchestration service to communicate with Secrets Manager
• Ensuring that the IAM roles associated with the ECS tasks or EKS pods have the necessary permissions to access the required secrets

While these methods might not offer instantaneous updates to running containers, they provide robust ways to ensure that new deployments and updates utilize the latest secret values.

6. Considerations and Best Practices

When implementing any approach for automatically retrieving updated secrets from AWS Secrets Manager, several critical considerations and best practices should be taken into account.

6.1. Security Implications

Security should be a primary concern:

• It is crucial to adhere to the principle of least privilege by granting only the necessary permissions to Lambda functions or other AWS services that need to access Secrets Manager.
• If using EventBridge API Destinations, the application’s endpoint must be secured appropriately to prevent unauthorized access.
• Auditing access to secrets through AWS CloudTrail and Amazon CloudWatch is also essential for monitoring and security analysis.

For more comprehensive security strategies when dealing with sensitive information, see our article on Protect Passwords in Cloud Full Project Review.

6.2. Handling Latency and Ensuring Eventual Consistency

Event-driven architectures, while providing near real-time updates, might still introduce some latency between the secret change and its reflection in the application.

Key considerations for handling latency:

• The acceptable level of staleness for the secret should be determined based on the application’s specific requirements.
• Implementing retry mechanisms in the secret retrieval and application update processes is a good practice to handle transient errors.
• Applications should be designed with eventual consistency in mind for secret updates.

6.3. Error Handling and Retry Mechanisms

Robust error handling should be implemented in any Lambda functions or application logic responsible for retrieving and updating secrets.

Recommended error handling practices:

• For EventBridge targets like Lambda or SQS, configure dead-letter queues (DLQs) to store failed events for later analysis and potential reprocessing.
• Implement appropriate retry logic with exponential backoff to handle transient failures.
• Set up monitoring and alerting for persistent failures that could indicate more serious issues.

6.4. Cost Implications

The cost of the chosen approach should be carefully evaluated.

Cost considerations include:

• EventBridge events and rule executions
• Lambda function invocations
• SNS notifications and deliveries
• SQS message storage and processing
• Secrets Manager API calls

Cost optimization strategies:

• Utilize client-side caching or the Secrets Manager Agent to significantly reduce the number of API calls to Secrets Manager
• If using periodic fetching, choose polling intervals judiciously to balance responsiveness with cost
• Consider implementing batching for operations where appropriate

6.5. Best Practices for Managing Secrets

Adhering to general best practices for managing secrets in AWS Secrets Manager is crucial:

• All sensitive information should be stored in Secrets Manager.
• Automatic secret rotation should be enabled wherever feasible to reduce the risk of compromised long-term credentials.
• Client-side caching or the Secrets Manager Agent should be used to improve performance and reduce costs associated with frequent secret retrieval.
• Hardcoding secrets in application code or storing them directly as environment variables should be avoided.
• Making applications environment-agnostic and managing secret declarations as code using tools like AWS CloudFormation or HashiCorp Terraform are also recommended practices.

For additional security enhancement strategies, check out our article on Five Ways to Enhancement of Your Content Security.

7. Conclusion

This report has explored several approaches for ensuring that applications automatically retrieve the latest values of AWS Secrets Manager secrets when they are changed. These methods range from:

• Leveraging AWS EventBridge to trigger various targets like Lambda functions, SNS topics, SQS queues, and API Destinations
• Implementing application-side strategies such as periodic fetching, SDK caching, and utilizing the AWS Secrets Manager Agent
• For containerized environments like ECS and EKS, utilizing specific integration mechanisms

The optimal approach depends on the specific requirements of the application, including:

• The need for real-time updates
• The complexity of the application architecture
• Cost considerations
• The desired level of operational overhead

For applications where immediate updates are critical, EventBridge-based solutions offer a reactive mechanism to trigger updates. The choice of target for the EventBridge rule (Lambda, SNS, SQS, or API Destination) should be based on the application’s existing infrastructure and its ability to consume these types of events.

For simpler applications or those where slight delays are acceptable, periodic fetching or SDK caching with a reasonable TTL might be sufficient. Combining client-side caching with an EventBridge notification mechanism to trigger a cache refresh can provide a balance between performance, cost-effectiveness, and timely updates.

Applications running in containerized environments can benefit from the specific integration features offered by ECS and EKS.

Ultimately, the key is to strike a balance between:

• Responsiveness to secret changes
• The security of the secrets and the update process
• The associated costs
• The overall complexity of the solution

By carefully considering these factors and implementing the appropriate strategies and best practices, organizations can ensure that their applications always operate with the most up-to-date and secure credentials managed by AWS Secrets Manager. For more AWS security best practices, see our article on The Ultimate Guide to Ensure Cybersecurity for Small Businesses.

Table 1: Comparison of Approaches for Reacting to Secrets Manager Changes

Table 2: EventBridge Rule Pattern Examples