How to use AWS Secret Manager

How to use AWS Secret Manager

Hey Bits Lovers! Let’s continue with our articles about AWS. And today will talk about How to use AWS Secret Manager. First, we will look at storing your secrets in Secrets Manager. So the foremost we’re going to examine is what Secrets Manager is. Then we will see what we can store inside Secrets Manager. We’ll then look at the automatic rotation.

This is a full AWS Secret Manager Tutorial, I hope that you enjoy the reading.

Why AWS Created the Secret Manager

The story begins with customers’ feedback that AWS heard that managing Secret is vital but, at the same time, pretty hard.

Also, the IAM Roles are excellent because they give us momentary AWS credentials automatically.

For example: if we attach an IAM Role to any instance as the application owner, you can concentrate on creating an outstanding application without concerning about how the application will communicate with all the other AWS resources the role will automatically provide short-term AWS credentials offer it to your EC2 instance and then rotate these credentials numerous times every day.

What is AWS Secret Manager

So AWS decided to build the Secrets Manager, which will allow us to manage the lifecycle of all other secrets, like database credentials, SSH keys, or tokens. With any of these credentials, we should be able to utilize secrets manager to store, distribute and rotate automatically.

How to use AWS Secret Manager

Why use AWS Secrets Manager – Capabilities and Advantages

The underlying storage fabric for Secrets Manager offers availability as a layer of 99.999%. Consequently, what this means is you can be assured that your secrets will consistently be available. Also, the Secrets Manager encrypts your secrets by default. So, for instance, you’ll never end up in a problem where a secret was left unencrypted accidentally. These encryption keys live in your account. 

Manage Secret

You have complete control over them. Once encrypted, you can retrieve these secrets either through a publicly accessible endpoint or a VPC endpoint specific to your organization.

Cache Mechanism and Compliance

Secrets Manager also delivers client-side caching libraries to cache your secrets locally and securely, so the application is not impacted by transient and network issues. Also, the Secret Manager is compliant with HIPAAPCI, and ISO.

Once we store your secrets and Secrets Manager, you don’t have to be concerned about your developers viewing on managing secrets. You have the tools to guarantee that your secrets are not visible to your developers. 

How we can use IAM Policies with Secret Manager

Access Control Secrets Manager

The Secrets Manager offers a variety of alternatives that enable us to control who can do what with our secrets tightly. In addition, the IAM policies have excellent integration with Secret Manager and are super fine-grained.

The other exciting piece around access control is resource-based.

Resource-Based Policies

Resource-based policies are similar to keeping all the season KMS, or as the bucket policies in S3, they act as another mechanism to control access to your secrets. The prototypical use case here would be cross-account access. So, for example, let’s consider this scenario where a business partner needs SSH access to an EC2 instance to help troubleshoot. We could email them the private key, but we wouldn’t control it. On the other hand, we could store that key in the Secrets Manager and attach a resource-based policy that allows a role in the business partner account to retrieve the Secret, and then when the task is over, we can rotate our SSH key.

Tag-Based Access

The Secrets Managers support the ability to tag secrets, making it easy to organize and use our secrets. For example, let’s suppose we have a team called Dev. This team requires a variety of secrets, like SSH keys, API tokens, or database credentials. We can store all of these in Secret Manager, and for each Secret, we can add the tag team equals Dev. After that, we can create one IAM policy that grants access to all secrets tagged Dev. So all the team members can get access to all the secrets required by the Dev team. 

Also, in the future, if the team adds more resources like databases, we can create new credentials and store them in the Secrets Manager. So we only need to define the appropriate tag and not worry about updating IAM policies anymore because the policies are tag-based and have already been configured.

 The same happens for a new employee who joins the team, we merely attach this one to the policy, and we are good to go. They have access to all the secrets that they need to perform their job without spending a lot of time figuring out the access control mechanisms. These tools make it more manageable to prevent your employees from using secrets that they shouldn’t be using and reduce secret sprawl.

How we can use IAM Policies with Secret Manager

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "SecretARN"
    }
  ]
}

Above it’s a sample IAM policy where we grant the ability to read in the action statement and retrieve the value of the Secret like the SSH key or the actual username and password that we stored in the Secrets Manager. Also, we added permissions to Describe Secret. That is robust because we can use Describe Secret to read the metadata of the Secret, such as what’s the description, and check if the rotation is enabled. Also, if there are any tags associated with the Secret, for example, if we have tags, we can describe the Secret to understand what it is for and how we should use it.

Also, if you noticed in the policy, the resource statement restricts these permissions to a particular secret. It’s helpful, for example, if we don’t want employees to have access to all secrets, so we can configure fine-grained policies and define which secrets they should be able to use.

Rotate Secrets

The secrets manager has built-in integration for Rotating Credentials for databases hosted on RDS or EC2. Still, if you have other kinds of secrets, you can store and rotate them.

How the Rotate Secret Works

 On Secret Manager in the console, you can add the secret and write a custom Lambda function to meet your rotation’s needs. Later we will cover the best practices in this article.

In a typical scenario, if an application needs to retrieve data from a database, the application reaches out to the Secrets Manager. First, it retrieves the database credentials and then uses them to retrieve the data. Then, if you choose to configure rotation, it will create a lambda function in the same VPC as your target database.

Enable the automatic rotation for a Secret:

Enabling automatic rotation basically will transform a long-term Secret into a short-term Secret. So, for example, a database credential that was living for years now becomes a credential that’s rotated every 15 days or every 30 days, and the Secret Manager makes all of this work by using lambda functions.

It’s cool. But, one second. How does AWS guarantee that the application doesn’t fail when the rotation happens? For example, applications need to connect to the database and suddenly change the password. Therefore, exists a concept is called versions of secrets.

Version of Secret

This rotation function will trigger a rotation based on your defined rotation frequency. So it means that it will create a new credential and give this credential pair a label called AWSPENDING. Alsoit will then store this credential pair in the database and run a combination of inspections to ensure that this credential pair works the way it should function.

AWS Secret Manager Version: AWSPREVIOUS, AWSCURRENT, and AWSPENDING
AWS Secret Manager Version: AWSPREVIOUS, AWSCURRENT, and AWSPENDING

 Throughout this whole process, if the application requires access to your Secret or access to your database, the label that’s the version of the Secret labeled AWSCURRENT is still functional, so your application is not impacted. So your application continues to function the way it should work. Now one secrets manager ensures that the new credential is behaving appropriately. It updates the version on this new credential to AWSCURRENT, and the previous version of the credential is labeled AWSPREVIOUS.

From this point forward, if the application wants to retrieve data from the database, it reaches out to the Secrets Manager. Again, it gets the newer credential, so the previous version, the one labeled AWSPREVIOUS, will become invalidated immediately or at the next rotation. It depends on how you’ve chosen to rotate your Secret.

AWS Secret Manager Workflow for Versioning: AWSPREVIOUS, AWSCURRENT, and AWSPENDING
AWS Secret Manager Workflow for Versioning: AWSPREVIOUS, AWSCURRENT, and AWSPENDING

JDBC Support

AWS provided a library for supporting client-side cache for Java and JDBC drivers. We can use these caching libraries to cache your secrets locally so that you make fewer API calls to the Secrets Manager, so that’s another mechanism to control the number of API calls that are metered and built by the Secrets Manager.

You can check the Library for JDBC on GitHub.

AWS Secret Manager Best Practices

Let’s talk about some topics that you should consider using Secret Manager. Also, recommend best practices that will help you set up for the Secret Manager at scale, so there are six things that you need to keep in mind as you get started.

  • Rotate Frequently
  • Retrieve Secret programmatically
  • Remove plain-text passwords
  • Lockdown permissions using IAM Roles
  • Do not Reuse the same Secret for different resources.

First, based on your security and compliance requirements, define how frequently you need to rotate your secrets third once you’ve stored your secrets centrally.

Second, how you retrieve them and how frequently they will depend on your network configuration and application setup.

Third, how will you identify where your secrets are being used today and what’s your strategy for migrating them to secrets manager?

Fourth, decide how to manage permissions to store, retrieve and rotate secrets.

Fifth, decide how you will audit the use of secrets and the type of activity you want to monitor.

Finally, plan to use unique secrets in every region and environment.

Let’s see AWS Secret Manager Best Practices with more details.

How to remove plain-text secrets

So we saw about the benefits of you of removing plain text secrets so let’s focus on how to get started. The first big decision is your account strategy. For example, do you store all secrets in one account, or do you store secrets in multiple accounts? Later, it would be best to work with your application owners and administrators to identify where your secrets are being used today. 

How to Move your Secrets to Secret Manager

Here are some things to help you get started with moving your secrets over to operate Secrets Manager in each AWS account. So don’t decide to consolidate in a single AWS account; instead, manage the secrets in the same AWS account for each application. There are a couple of benefits to this approach. 

First, placing secrets in the same account as the resource helps logically group everything you need to make an application work.

Second, accounts act as natural boundaries, which reduces the risk that minimizes the blast radius should a secret be compromised. 

And finally, this makes it easy to manage secrets at scale because now, what you can do is you can give the application owner complete access to the secrets manager within their AWS account. So that they can quickly create and handle secrets. At the same time, you have the proper monitoring and enforcement tools that guarantee that these secrets are managed following your security policies and your compliance policies.

How to name Secrets

Always use meaningful names that are hierarchical and use descriptions generously in the description. Add information such as who owns the Secret, what is the Secret for who is responsible for managing the Secret. In the long run, getting the description right and setting in place good naming practices will make it easy to manage secrets at scale. 

So, for example, a developer can now peek at the description of the secrets to recognize which Secret they need to make the application work. They can also identify who they should reach out to request access to that Secret. Because of that using the correct description can go a long way in building clarity and enabling management at scale.

Always use tags

Tags are super valuable, especially when you want to manage secrets at scale. For example, if you have ten secrets, sure, you can work without tags, but as soon as you get to a few hundred secrets, tags will be priceless, and here is how you enforce them:

{
"Version" : "2012-10-17",
"Statement" : [ {
"Sid" : "PermissionByDepartment",
"Effect" : "Allow",
"Principal" : {
"AWS" : "*"
},
"Action" : "secretsmanager:GetSecretValue",
"Resource" : "*",
"Condition" : {
"StringLike : {
"SecretManager:RequestTag/Team":”Dev”
}
}
} ]
}

We can add this statement to our IAM policy. This statement says if you’re creating a secret, you must provide a tag, and the tag should provide the team so you can use various versions of this condition to meet your unique tagging requirements.

How encryption works for Secret Manager

Is it correct to use the default KMS Key?

It depends. Let me explain why:

The default key is great because you don’t have to manage additional AWS KMS permissions. Also, the Secrets Manager creates a unique encryption key for every account and every region, so if you have secrets across accounts and regions, they are all encrypted using their encryption key. 

Still, you may have scenarios where you are required to, may have compliance needs that demand that you have CMKs are also valuable for cross-account access.

So if you have secrets of that that have to be retrieved across accounts, you have to use CMKs.

Finally, CMKs act as another set of access control so you can put policies on the CMK to control who can use the CMK to decrypt the secret, so it’s another mechanism for controlling access.

Rotate Frequently

The next big thing you need to think about is how you rotate your secrets and when you do that. Again, as we saw earlier, this will depend on your security and compliance requirements, but here are a few things to get started for existing applications. Move their secrets over first and worry about rotations second but if you’re building newer applications, store the secret and set up rotation right from the get-go.

Rotation Strategy

There are a few shortcuts to help you figure out your rotation strategy. The default frequency is 30 days on the console. It should work for most people but check your compliance and security requirements to determine this number. 

Remember that we’re only paying for the rotation API when you rotate a secret—for example, the lambda. You’re not paying for a newer secret, so rotate your secrets actively. You don’t have to worry about it because of the cost.

Create Lambda Function for Secret Manager Rotation

To execute rotation, Secrets Manager creates lambda functions. These Lambda Functions require the ability to communicate with the target resource to the database and the secrets manager, so make sure you’ve set up the appropriate gateways.

You can make that more accessible by using VPC endpoints. Also, VPC endpoints are the recommended practice because now your application and the Lambda Function can talk to the Secrets Manager without going through the publicly accessible internet. This rotation lambda is also responsible for creating the new credential bear. It does this by calling the getter and a password API. By default, Secrets Manager will make the most complex password possible.

Still, you may have scenarios where you want to use simpler passwords:

For example, you may have a legacy application that does not support special characters in the password field. Well, in those scenarios, you can update the Lambda Function to supply a parameter to exclude special characters when they get a random password API is called:

Generating an random password using Python and Boto3:

# Get exclude characters from environment variable
exclude_characters = os.environ['EXCLUDE_CHARACTERS'] if 'EXCLUDE_CHARACTERS' in os.environ else ':/@"\'\\'
# Generate a random password
service_client = boto3.client('secretsmanager', region_name='us-east-1')
passwd = service_client.get_random_password(ExcludeCharacters=exclude_characters)

Do we need one lambda function for each Secret?

It is a trick question. However, depending on the use case in general, it’s easier to reuse existing lambdas, and here’s why:

 When creating a rotation lambda, we also have to make an IAM Role associated with the lambda. Therefore, rotating requires specific IAM permissions also, in general, should be very stingy with your IAM permissions. Hence, one path forward is that a tiny number of employees can create these rotation lambdas and the IAM roles. In contrast, a larger group of employees gets permission to store secrets and call these existing lambdas to rotate secrets. Does that make sense too? It’s just easier to manage a smaller number of lambda functions.

So think about your downstream applications and keep their requirements in mind as you define your lambda functions. 

Finally, instead of creating a unique lambda for rotating every secret, try to reuse lambdas that are just more scalable.

Creating IAM Roles for Secret Manager

So to get started, create IAM roles for your applications and give these roles appropriate permissions. In most scenarios retrieving a Secret once an hour is fine. Or you can also use the client-side caching libraries to offload all of this management. If you use these liabilities, these libraries will retrieve tickets for you. In addition, they’ll refresh the cache every hour, and if there are authentication failures, it will reach out to the Secrets Manager again and refresh the cache. 

The approach to retrieving secrets becomes more manageable if we employ one of these client-side caches in libraries. Although the AWS launched client sites caching support for Java and JDBC drivers, future releases will consider supporting client-side caching for Ruby, Go, and Python.

For serverless, this is important we retrieve the secrets outside the lambda handler. This allows us to cache the Secret within the lambda container, reducing the number of API calls that we have to make to the Secrets Manager.

So if you’re using serverless, retrieve the secret outside the handler. 

Delete Secrets

Finally, if you decide you do not need a secret anymore, I recommend scheduling it for deletion instead of deleting it immediately. The explanation here is if it turns out that someone did need access to the secret, you can quickly go back and restore the secret and get the application or employee working again quickly.

Lock Down Permissions

Start by identifying who needs what access.

Very few people should have full access to the Secret Manager. So it should be super strict with that. But you can be significantly more liberal with the actions to list secrets and describe them. Because, as we saw before if you grant these abilities to your employees, it makes it more effortless for them to find the right secret and then ask for access to that secret. 

It makes it more uncomplicated for your employees to be agile and operate quickly. Also, remember to configure rotation requires IAM permissions, so be meticulous with who to configure rotation. One way to mitigate or address this would be to split duties—a section of employees’ permissions to store and retrieve a secret. In contrast, it gives a smaller selection of employees the ability to configure rotation.

Also, consider using tags, as we learned in this article, because tagging is super helpful in managing a more significant number of Secrets.

And finally, cross-account access requires CMKs, which helps you decide if you need to grant extra permissions to create and decrypt using CMKs.

Use Unique Secrets

Use unique secrets within AWS. We require our teams to create and use unique secrets in every environment in every AZ and every region in every account. The benefits of minimizing the blast radius are massive. It also makes it easier to recover. Also, it reduces the overhead of synchronizing secrets across multiple regions.

Always use CloudFormation, Terraform, or another tooling to provision secrets. The more you rely on automation, the more manageable it will be to follow consistent practices across accounts and to follow best practices. Like, creating unique secrets, and then once you’ve made unique secrets. But, of course, that approach requires your applications to reach out to the secrets manager from whichever regional endpoint the application is using.

Audit and Monitor the use of Secrets

If you want a quick glance at who’s using the secrets manager, you can simply use IAM access advisor if you wish more detailed records of who went in and who did what its secrets manager, you can use CloudTrail. 

The CloudTrail integrates with the entire AWS ecosystem, and it also gives you the ability to see how your secret was used once it was retrieved from the secrets manager. So you can build an entire story. For example, you can say Mike logged in and retrieved the database credential from the secrets manager, and then he went to the RDS instance and used this credential. You can also monitor the use of your secrets using CloudWatch events.

What is worth monitoring?

I’ll encourage you to monitor attempts to retrieve secrets scheduled for deletion. Why?

It’s crucial because if someone accidentally deleted a secret, that an application is still calling that secret or attempting to call that secret, that means you have a broken application. So once you detect this, you can quickly restore the secret and get the application working again.

 You should also monitor access to high-value secrets, such as production database credentials or SSH keys from your production instances. Also, the CloudTrail records all secrets manager API, so expect an increase in the size of your trails once you start using Secret Manager.

AWS Secret Manager Cost

AWS Secret Manager Pricing Summary

- $0.05 per 10K API Calls
- No Annual license or upfront cost
- $0.40 per Secret/Month. 
- 30 day free trial for testing

In the payment model with the Secrets Manager, there are no long-term contact licensing fees, operational expenses, etc. It’s all about paying for what you use. As far as policing is concerned, there are two key attributes that I’d like to call out right here.

First is, when you rotate the Secret, you pay for calling the rotation API. Also, you don’t pay for creating a new secret because, as I showed you previously, you haven’t made a new secret. You just created a new version, so you are not paying an extra 40 cents for this new Secret.

Exam Tips

One important note for the exam is that if we enable rotations, the Secrets Manager instantly rotates the secrets to ensure that the configuration is working correctly. So we might see a scenario-based question where it’s talking around this. For instance, we desire to enable rotation, but our application still has some passwords hardcoded. So basically, we have to guarantee and change our application to get the credentials from Secret Manager using the SDK and not from the files or environment variables. Remember that environment variables are not secure if we compromise one server, all credentials stored on the environment variables, can be easily read.

Secret Manager Example

If you are looking for an example of CloudFormation to deploy Secret Manager. You can use this as an example.

Conclusion

The AWS Secret Manage contains four fundamental capabilities: First, the Secrets Manager guarantees that your secrets are secured. Second, it also enables us to manage who can do it tightly and what to do with our secrets. Third, it allows you to rotate your secrets without affecting the applications or developers. Finally, this is a pay-as-you-go model for cloud services, which means there is no long-term contract licensing fees or operational costs.

Leave a Comment

Your email address will not be published.