Hey, Bits Lovers! Let’s continue with our articles about AWS. And today will talk about how to use AWS Secret Manager. First, we’re going to examine what Secrets Manager is. Then we will see what we can store inside Secrets Manager. We’ll then look at the automatic rotation.
This is a full AWS Secret Manager Tutorial, and I hope you enjoy the reading.
Why AWS Created the Secret Manager
The story begins with customers’ feedback that AWS heard that managing Secret is vital but, at the same time, pretty hard.
Also, the IAM Roles are excellent because they give us momentary AWS credentials automatically.
For example: if we attach an IAM Role to any instance as the application owner, you can concentrate on creating an outstanding application without concerning about how the application will communicate with all the other AWS resources. The role will automatically provide short-term AWS credentials, offer them to your EC2 instance, and then rotate these credentials numerous times every day.
What is AWS Secret Manager
So AWS decided to build the Secrets Manager, which will allow us to manage the lifecycle of any secrets, like database credentials, SSH keys, or tokens. With any of these credentials, we should be able to utilize secrets manager to store, distribute and rotate automatically.
How to use AWS Secret Manager
Why use AWS Secrets Manager – Capabilities and Advantages
The underlying storage fabric for Secrets Manager offers availability as a layer of 99.999%. Consequently, this means you can be assured that your secrets will be consistently available. Also, the Secrets Manager encrypts your secrets by default. So, for instance, you’ll never end up in a problem where a secret was left unencrypted accidentally. These encryption keys live in your account.
Manage Secret
You have complete control over them. Once encrypted, you can retrieve these secrets through a publicly accessible VPC endpoint specific to your organization.
Cache Mechanism and Compliance
Secrets Manager also delivers client-side caching libraries to cache your secrets locally and securely, so the application is not impacted by transient and network issues. Also, the Secret Manager complies with HIPAA, PCI, and ISO.
Once we store our secrets on Secrets Manager, you don’t have to be concerned about your developers viewing managing secrets. You have the tools to guarantee that your secrets are invisible to your developers.
How can we use IAM Policies with Secret Manager
Access Control – Secrets Manager
The Secrets Manager offers a variety of alternatives that enable us to control who can do what with our secrets tightly. In addition, the IAM policies have excellent integration with Secret Manager and are fine-grained.
The other exciting piece around access control is resource-based.
Resource-Based Policies
Resource-based policies act as another mechanism to control access to your secrets. A great example would be cross-account access. So, for example, let’s consider this scenario where a business partner needs SSH access to an EC2 instance to help troubleshoot. We could email them the private key, but we wouldn’t control it. On the other hand, we could store that key in the Secrets Manager and attach a resource-based policy that allows a role in the business partner account to retrieve the Secret, and then when the task is over, we can rotate our SSH key.
Tag-Based Access
The Secrets Managers support the ability to tag secrets, making it easy to organize and use our secrets. For example, let’s suppose we have a team called Dev. This team requires a variety of secrets, like SSH keys, API tokens, or database credentials. We can store all of these in Secret Manager, and for each Secret, we can add the tag team equals Dev. After that, we can create one IAM policy that grants access to all secrets tagged Dev. So all the team members can access all the secrets the Dev team requires.
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"PermissionByDepartment",
"Effect":"Allow",
"Principal":{
"AWS":"*"
},
"Action":"secretsmanager:GetSecretValue",
"Resource":"*",
"Condition":{
"StringLike":{
"SecretManager:RequestTag/Team":"”Dev”"
}
}
}
]
}
Also, if the team adds more resources like databases in the future, we can create new credentials and store them in the Secrets Manager. So we only need to define the appropriate tag and not worry about updating IAM policies anymore because the policies are tag-based and have already been configured.
The same happens for a new employee who joins the team. We merely attach this one to the policy and are good to go. They have access to all the secrets they need to perform their job without spending much time figuring out the access control mechanisms. These tools make it more manageable to prevent employees from using secrets they shouldn’t be using and reduce secret sprawl.
Secret Manage Policy Example
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"secretsmanager:DescribeSecret",
"secretsmanager:List*",
"secretsmanager:GetSecretValue"
],
"Resource": "arn:aws:secretsmanager:Region:AccountId:secret:a_secret_name-here"
}
]
}
Above, it’s a sample IAM policy where we grant the ability to read the action statement and retrieve the value of the Secret, like the SSH key or the actual username and password that we stored in the Secrets Manager. Also, we added permissions to Describe Secret. That is robust because we can use Describe Secret to read the metadata of the Secret, such as what’s the description, and check if the rotation is enabled. Also, if there are any tags associated with the Secret, for example, if we have tags, we can describe the Secret to understand what it is for and how we should use it.
Also, if you noticed in the policy, the resource statement restricts these permissions to a particular secret. It’s helpful, for example, if we don’t want employees to have access to all secrets, so we can configure fine-grained policies and define which secrets they should be able to use.
Rotate Secrets
The secrets manager has built-in integration for Rotating Credentials for databases hosted on RDS or EC2. Still, if you have other kinds of secrets, you can store and rotate them.
How the Rotate Secret Works
On Secret Manager in the console, you can add the secret and write a custom Lambda function to meet your rotation’s needs. Later we will cover the best practices in this article.
In a typical scenario, if an application needs to retrieve data from a database, the application reaches out to the Secrets Manager. First, it retrieves the database credentials and then uses them to retrieve the data. Then, if you choose to configure rotation, it will create a lambda function in the same VPC as your target database.
Enable the automatic rotation for a Secret:
Enabling automatic rotation will transform a long-term Secret into a short-term Secret. So, for example, a database credential that has been living for years now becomes a credential that’s rotated every 15 days or every 30 days. The Secret Manager makes all of this work by using lambda functions.
It’s cool. But, one second. How does AWS guarantee the application doesn’t fail when the rotation happens? For example, applications must connect to the database and suddenly change the password. Therefore, exists a concept called versions of secrets.
Version of Secret
This rotation function will trigger a rotation based on your defined rotation frequency. So it means that it will create a new credential and give this credential pair a label called AWSPENDING. Also it will then store this credential pair in the database and run a combination of inspections to ensure that this credential pair works the way it should function.
Throughout this process, if the application requires access to your Secret or your database, the label that’s the version of the Secret labeled AWSCURRENT is still functional, so your application is not impacted. So your application continues to function the way it should work. Now the secrets manager ensures that the new credential is behaving appropriately. It updates the version on this new credential to AWSCURRENT, and the previous version is labeled AWSPREVIOUS.
From this point forward, if the application wants to retrieve data from the database, it reaches out to the Secrets Manager. Again, it gets the newer credential, so the previous version, which is labeled AWSPREVIOUS, will be invalidated immediately or at the next rotation. It depends on how you’ve chosen to rotate your Secret.
JDBC Support
AWS provided a library for supporting client-side cache for Java and JDBC drivers. We can use these caching libraries to cache your secrets locally so that you make fewer API calls to the Secrets Manager, so that’s another mechanism to control the number of API calls that are metered and built by the Secrets Manager.
You can check the Library for JDBC on GitHub.
AWS Secret Manager Best Practices
Let’s talk about some topics that you should consider using Secret Manager. Also, recommend best practices that will help you set up for the Secret Manager at scale, so there are six things that you need to keep in mind as you get started.
- Rotate Frequently
- Retrieve Secret programmatically
- Remove plain-text passwords
- Lockdown permissions using IAM Roles
- Do not Reuse the same Secret for different resources.
First, based on your security and compliance requirements, define how frequently you need to rotate your secrets third once you’ve stored your secrets centrally.
Second, how you retrieve and how frequently they depend on your network configuration and application setup.
Third, how will you identify where your secrets are being used today, and what’s your strategy for migrating them to the secrets manager?
Fourth, decide how to manage permissions to store, retrieve and rotate secrets.
Fifth, decide how you will audit the use of secrets and the type of activity you want to monitor.
Finally, plan to use unique secrets in every region and environment.
Let’s see AWS Secret Manager Best Practices with more details.
How to remove plain-text secrets
So we saw about your benefits of removing plain text secrets, so let’s focus on how to get started. The first big decision is your account strategy. For example, do you store all secrets in one account, or do you store secrets in multiple accounts? Later, it would be best to work with your application owners and administrators to identify where your secrets are being used today.
How to Move your Secrets to Secret Manager
Here are some things to help you move your secrets to operate Secrets Manager in each AWS account. So don’t decide to consolidate in a single AWS account; instead, manage the secrets in the same AWS account for each application. There are a couple of benefits to this approach.
First, placing secrets in the same account as the resource helps logically group everything you need to make an application work.
Second, accounts act as natural boundaries, which reduces the risk that minimizes the blast radius should a secret be compromised.
And finally, this makes it easy to manage secrets at scale because now, what you can do is you can give the application owner complete access to the secrets manager within their AWS account. So that they can quickly create and handle secrets. At the same time, you have the proper monitoring and enforcement tools that guarantee that these secrets are managed following your security and compliance policies.
How to name Secrets
Always use meaningful, hierarchical names and use descriptions generously in the description. Add information such as who owns the Secret, what the secret is for who is responsible for managing the Secret. In the long run, getting the description right and setting in place good naming practices will make it easy to manage secrets at scale.
So, for example, a developer can now peek at the description of the secrets to recognize which Secret they need to make the application work. They can also identify who they should reach out to request access to that Secret. Because of that using the correct description can go a long way in building clarity and enabling management at scale.
Always use tags
Tags are super valuable, especially when you want to manage secrets at scale. For example, if you have ten secrets, sure, you can work without tags, but as soon as you get to a few hundred secrets, tags will be priceless, and here is how you enforce them:
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"PermissionByEnvironment",
"Effect":"Allow",
"Principal":{
"AWS":"*"
},
"Action":"secretsmanager:GetSecretValue",
"Resource":"*",
"Condition":{
"StringLike":{
"SecretManager:RequestTag/ENV":"”PROD”"
}
}
}
]
}
We can add this statement to our IAM policy. This statement says if you’re creating a secret, you must provide a tag, and the tag should provide the team identification so you can use various versions of this condition to meet your unique tagging requirements.
KMS and Secret Manager
Secrets Manager utilizes the KMS key associated with a secret to generate a data key for individual secret values.
Also, remember that if you do not specify your KMS key when you create the secret, you will automatically use the AWS-managed key for Secrets Manager (aws/secretsmanager) created by the region on your account.
Secrets Manager uses the KMS key to decrypt that data key when it requires decrypting the encrypted secret value.
One necessary action for you to take before starting to use the KMS Key its grantee is that your Role, for example, for an EC2, contains the proper permissions.
I already made that mistake, and if you see one message like the one below:
com.amazonaws.services.secretsmanager.model.AWSSecretsManagerException: Access to KMS is not allowed (Service: AWSSecretsManager; Status Code: 400; Error Code: AccessDeniedException; Request ID: 19320236-09dc-454e-abeb-a037j60d4755; Proxy: null) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1862)
There are two actions that Secrets Manager performs to request and use a KMS key:
GenerateDataKey – The Secret Manager calls the KMS:GenerateDatakey for those actions: CreateSecret, PutSecretValue, UpdateSecret.
Decrypt – Of course, to decrypt the secret value, the Secret Manager needs to call the KMS Decrypt action.
So, based on that, your policy should look like the example below:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "KMSKeyForSM",
"Effect": "Allow",
"Action": [
"kms:Decrypt",
"kms:GenerateDataKey"
],
"Resource": "*"
}
]
}
How encryption works for Secret Manager
Is it correct to use the default KMS Key?
It depends. Let me explain why:
The default key is great because you don’t have to manage additional AWS KMS permissions. Also, the Secrets Manager creates a unique encryption key for every account and every region, so if you have secrets across accounts and regions, they are all encrypted using their encryption key.
Also, you may have scenarios where you are required or may have compliance that demands that you have CMKs, which are also valuable for cross-account access.
So if you have secrets that need to be retrieved across accounts, you must use CMKs.
Finally, CMKs act as another set of access control, so you can put policies on the CMK to control who can use the CMK to decrypt the secret, so it’s another mechanism for controlling access.
Rotate Frequently
The next big thing you must consider is how you rotate your secrets and when you do that. Again, as we saw earlier, this will depend on your security and compliance requirements, but here are a few things to get started for existing applications.
First, map all codes that contain plain-text hardcoded passwords. And store those passwords on Secret Manager. For last, you take care of the Rotation process. For newer applications, store the secret and set up rotation right from the get-go.
Rotation Strategy
There are a few shortcuts to help you figure out your rotation strategy. The default frequency is 30 days on the console. It should work for most people but check your compliance and security requirements to determine this number.
Remember that we only pay for the rotation API when you rotate a secret—for example, the lambda. You’re not paying for a newer secret, so rotate your secrets actively. You don’t have to worry about it because of the cost.
Create Lambda Function for Rotation
To execute rotation, Secrets Manager creates lambda functions. These Lambda Functions require the ability to communicate with the target resource, like a database and the secrets manager, so make sure you’ve set up the appropriate gateways.
You can make that more accessible by using VPC endpoints. Also, VPC endpoints are recommended because your application and the Lambda Function can talk to the Secrets Manager without going through the internet. This rotation lambda is also responsible for creating the new credential bear. It does this by calling the getter and a password API. By default, Secrets Manager will make the most complex password possible.
Still, you may have scenarios where you want to use simpler passwords:
For example, a legacy application may not support special characters in the password field. Well, in those scenarios, you can update the Lambda Function to supply a parameter to exclude special characters when they get a random password API is called:
Generating a random password using Python and Boto3:
# Get exclude characters from environment variable
exclude_characters = os.environ['EXCLUDE_CHARACTERS'] if 'EXCLUDE_CHARACTERS' in os.environ else ':/@"\'\\'
# Generate a random password
service_client = boto3.client('secretsmanager', region_name='us-east-1')
passwd = service_client.get_random_password(ExcludeCharacters=exclude_characters)
Do we need one lambda function for each Secret?
It is a trick question. However, depending on the use case in general, it’s easier to reuse existing lambdas, and here’s why:
When creating a rotation lambda, we must also make an IAM Role associated with the lambda. Therefore, rotating requires specific IAM permissions and should be very stingy with your IAM permissions. Hence, one path forward is that a tiny number of employees can create these rotation lambdas and the IAM roles. In contrast, a larger group of employees gets permission to store secrets and call these existing lambdas to rotate secrets.
Does that make sense too? It’s just easier to manage a smaller number of lambda functions.
So consider your downstream applications and keep their requirements in mind as you define your lambda functions.
Finally, instead of creating a unique lambda for rotating every secret, try to reuse lambdas that are just more scalable.
Creating IAM Roles for a Secret
So to get started, create IAM roles for your applications and give these roles appropriate permissions. In most scenarios retrieving a Secret once an hour is fine. Or you can also use the client-side caching libraries to offload all of this management. If you use these liabilities, these libraries will retrieve tickets for you. In addition, they’ll refresh the cache every hour, and if there are authentication failures, they will reach out to the Secrets Manager again and refresh the cache.
Retrieving secrets becomes more manageable if we employ one of these client-side caches in libraries. Although the AWS launched client sites caching support for Java and JDBC drivers, future releases will consider supporting client-side caching for Ruby, Go, and Python.
For serverless, retrieving the secrets outside the lambda handler is essential. This allows us to cache the Secret within the lambda container, reducing the number of API calls we must make to the Secrets Manager.
So if you’re using serverless, retrieve the secret outside the handler.
Delete Secrets
Finally, if you do not need a secret anymore, I recommend scheduling it for deletion instead of deleting it immediately. The explanation here is if it turns out that someone did need access to the secret, you can quickly go back and restore the secret and get the application or employee working again quickly.
Lock Down Permissions
Start by identifying who needs what access.
Very few people should have full access to the Secret Manager. So it should be super strict with that. But you can be significantly more liberal with the actions to list secrets and describe them. Because as we saw before, if you grant these abilities to your employees, it makes it more effortless for them to find the right secret and then ask for access to it.
It makes it more uncomplicated for your employees to be agile and operate quickly. Also, remember to configure rotation requires IAM permissions, so be meticulous with who to configure rotation. One way to mitigate or address this would be to split duties—a section of employees’ permissions to store and retrieve a secret. In contrast, it gives fewer employees the ability to configure rotation.
Also, consider using tags, as we learned in this article because tagging is super helpful in managing a more significant number of Secrets.
And finally, cross-account access requires CMKs, which helps you decide if you need to grant extra permissions to create and decrypt using CMKs.
Use Unique Secrets
Use unique secrets within AWS. We require our teams to create and use unique secrets in every environment in every AZ and every region in every account. The benefits of minimizing the blast radius are massive. It also makes it easier to recover. Also, it reduces the overhead of synchronizing secrets across multiple regions.
Always use CloudFormation, Terraform, or another tooling to provision secrets. The more you rely on automation, the more manageable it will be to follow consistent practices across accounts, best practices like creating unique secrets, and then once you’ve made unique secrets. But, of course, that approach requires your applications to reach out to the secrets manager from whichever regional endpoint the application uses.
Audit and Monitor the use of Secrets
If you want a glance at who’s using the secrets manager, you can use IAM access advisor if you wish, more detailed records of who went in and who did what its secrets manager is, you can use CloudTrail.
The CloudTrail integrates with the entire AWS ecosystem and allows you to see how your secret was used once it was retrieved from the secrets manager. So you can build an entire story. For example, you can say Mike logged in and retrieved the database credential from the secrets manager, and then he went to the RDS instance and used this credential. You can also monitor the use of your secrets using CloudWatch events.
What is worth monitoring?
I’ll encourage you to monitor attempts to retrieve secrets scheduled for deletion. Why?
It’s crucial because if someone accidentally deletes a secret, an application still calls that secret or attempts to call it, which means you have a broken application. So once you detect this, you can quickly restore the secret and get the application working again.
You should also monitor access to high-value secrets from your production instances, such as production database credentials or SSH keys. Also, the CloudTrail records all secrets manager API, so expect an increase in the size of your trails once you start using Secret Manager.
AWS Secret Manager Cost
AWS Secret Manager Pricing Summary
- $0.05 per 10K API Calls - No Annual license or upfront cost - $0.40 per Secret/Month. - 30 day free trial for testing
In the payment model with the Secrets Manager, there are no long-term contract licensing fees, operational expenses, etc. It’s all about paying for what you use. As far as policing is concerned, there are two key attributes that I’d like to call out right here.
First is, when you rotate the Secret, you pay for calling the rotation API. Also, you don’t pay to create a new secret because, as I showed you previously, you haven’t made a new one. You just created a new version, so you are not paying an extra 40 cents for this new Secret.
Exam Tips
One important note for the exam is that if we enable rotations, the Secrets Manager instantly rotates the secrets to ensure the configuration works correctly. So we might see a scenario-based question where it’s talking around this. For instance, we desire to enable rotation, but our application still has some passwords hard coded. So basically, we have to guarantee and change our application to get the credentials from Secret Manager using the SDK and not from the files or environment variables. Remember that environment variables are not secure because if we compromise the server, all credentials stored on the environment variables can be easily read by anyone.
Secret Manager Example
If you are looking for an example of CloudFormation to deploy Secret Manager. You can use this as an example.
Conclusion
The AWS Secret Manage contains four fundamental capabilities: First, the Secrets Manager guarantees that your secrets are secured. Second, it enables us to manage who can do it tightly and what to do with our secrets. Third, it allows you to rotate your secrets without affecting the applications or developers. Finally, this is a pay-as-you-go model for cloud services, meaning no long-term contract licensing fees or operational costs exist.
