Terraform State

Terraform State [Complete Guide]

Terraform must hold state regarding your infrastructure and configuration. This state is operated by Terraform to map real resources in the cloud (already deployed) to your configuration, also keep track of metadata, and enhance performance for extensive infrastructures. Also, check our article about Terraform Workspace.

Terraform State File

Where is Terraform State Stored?

The Terraform state is saved by default in a local file called “terraform.tfstate“, at the root of your project. However can also be stored remotely (like S3 Bucket), which performs excellently with a team of developers and engineers.

Terraform utilizes this local state to make plans and modify our infrastructure. Then, Terraform refreshes prior to any procedure to refresh the state with the whole infrastructure.

The immediate goal of Terraform state is to hold links between entities in a remote system and resource instances defined in our configuration. When Terraform makes a remote entity in reaction to a configuration modification, it will register the essence of that remote entity against a specific resource instance and then potentially change or remove that entity in response to coming configuration modifications.

Terraform Update State Manually

While the structure of the state files is simply JSON, changing the state file directly is not recommended. Rather, terraform supplies the terraform state command to execute fundamental state changes utilizing the command line.

The command line usage and output are cordial for Unix tools like grep awk. Further, the command line protects us from any format modifications within the state itself. The Terraform project will hold the command working while the state structure may shift.

Terraform’s goal is to map a one-to-one mapping between configured resources and remote entities. Generally, that is ensured by Terraform standing as the one to build each entity and record its uniqueness in the state or eliminate an entity and then delete the link for it.

Imagine adding or removing links in the state by additional means, like importing externally-created entities with terraform import or requesting Terraform to “overlook” an existent entity with terraform state rm command. In that circumstance, we’ll then require to guarantee that this one-to-one rule is obeyed, like manually removing an entity we requested Terraform to “overlook” or re-importing it to attach it to some different resource.

Terraform State File Format

State snapshots are held in the JSON structure, and new terraform versions are typically backward compatible with state snapshots created by earlier versions. Regardless, the state structure is subject to modification in new Terraform versions. If we make an application that parses or changes it directly, we should expect to execute constant maintenance of that application as the state structure grows in new versions.

Also, several integration points deliver JSON result that is particularly planned for consumption by external application:

The terraform output command includes a -json argument for getting the complete root module output set or a specific named output value from the latest state snapshot.

The terraform show command includes a -json argument for reviewing the latest state snapshot in full and corresponding saved plan files, including a replica of the initial state when the plan was created.

A standard practice to utilize these in cases where Terraform is operating in automation is to execute them instantly after a successful terraform apply to acquire a picture of the latest state snapshot and then store that result as an artifact associated with the automated run so that additional applications can potentially use it without requiring to execute Terraform itself.

Terraform State File Example

The Terraform State file usually is a big one, but it looks like this one below:

{                                                              
  "version": 4,                                                
  "terraform_version": "0.15.1",                              
  "serial": 3,                                                 
  "lineage": "364d8449-e325-c78f-132a-c1c5791fec40",                                                                           
  "outputs": {},                                               
  "resources": [                                               
    {                                                          
      "module": "module.webserver",                               
      "mode": "data",                                          
      "type": "archive_file",                                  
      "name": "data-source",                                    
      "provider": "provider.archive",                                                                                          
      "instances": [                                           
        {                                                      
          "schema_version": 0,                                 
          "attributes": {                                      
            "excludes": null,                                  
            "id": "1d0f71c7661280bb00571c024a501161e96b4bce",                                                                  
            "output_base64sha256": "8r+/wgwvAMsDn4iimiRzY9z9GZwsaTh7gNlLSwa/E60=",                                                                                                                                                                             
            "output_md5": "fb7feb1d7731375bb0f17d18a42e111f",                                                                  
            "output_path": "data-source.zip",                                                                                   
            "output_sha": "1d0f71c7664180bb00571c024a501169e96b4bce",                                                                                                                                                                                          
            "output_size": 4796,                               
            "source": [],                                      
            "source_content": null,                                                                                            
            "source_content_filename": null,                                                                                   
            "source_dir": "modules/webserver/data-source/",                                                                        
            "source_file": null,                               
            "type": "zip"                                      
          }                                                    
        }                                                      
      ]                                                        
    },                                                         
    {                                                          
      "mode": "data",                                          
      "type": "aws_vpc",                                       
      "name": "myvpc",                                         
      "provider": "provider.aws",
      ...
}

Terraform State File Best Practices

1. Always backup your Terraform State file “terraform.tfstate”. If you lose it, you can’t manage the existing resources on your cloud provider. 

2. Don’t commit your file. Backup your file doesn’t mean that we need to save it to our repository. Also, the Terraform cannot operate without the latest version of the state file, so many people consider committing it to version control. However, committing the state file implies several risks… Firstly, we could reveal passwords from our application configuration, like passwords and database connection strings. And secondly, we risk running Terraform against a stale or old state that we forgot to pull the latest version from our repository. 

3. So, if we can’t commit, how do we store the Terraform State file and guarantee that we don’t lose it? Simple: Always consider using terraform backends. Today, Terraform supports the backend on Azurerm, AWS S3, GCS, Consul, etc. We will show how to do it below in this article, using the AWS S3 Bucket. 

4. Create one Terraform State for each environment. Mistakes happen. We must avoid mixing different environments in the same state because the risk of making a mistake is high, and the significant errors with the state are unrepairable.

5. Never change the Terraform Stage File Manually! As the state file represents our infrastructure in the real world, occasionally, circumstances require us to change our terraform state. For example, if we want to rename a resource block, we must re-assign our state to the new resource name. Unfortunately, many are seduced to dive into the state file itself and start hacking around when they require moving or renaming. But beware, there’s a much safer way! The Terraform CLI provides us with commands that let us remove or move. See below in this article for how to do it.

Terraform State Backup

The commands that change the terraform state always will create a backup before effectively applying the changes. We can control the path where we need to store the backup file using the argument -backup.

It’s worth remembering that the command that just read the terraform state does not create the backup file.

Also, we can’t disable the backups for the terraform state change. Therefore, we need to remove these files manually if we don’t desire to store them.

Terraform State Remote

Remote State

All Terraform commands use the Terraform State, regardless of whether we use a remote or local state.

Modifications in the remote state may take longer because the file and data will be transferred through your network.

Also, backups are created automatically to disk, and the way we use the command is identical to if it were a local state.

Terraform State Backend using S3

As we discussed before, it’s a good practice to store the state using a backend. So let’s see how we do it on our terraform code.

Terraform State on S3

terraform {
  backend "s3" {
    bucket = "bucket-name"
    key    = "path/to/terraform-state/file"
    region = "us-west-1"
  }
}

Also, on a great approach: We must enable Bucket Versioning on the S3 bucket to permit state rescue in unexpected deletions and human mistakes.

Terraform State Command 

The terraform state command is utilized to manage the state. As our configuration becomes more complex, there are some circumstances where we may need to change the Terraform state. Instead of modifying the state directly, the terraform state commands can be used in many possibilities.

This command is a nested subcommand, meaning that it has further subcommands. These subcommands are listed to the left.

Usage

Usage: terraform state <subcommand> [options] [args]

Terraform State Management

How to Manage Terraform State? First, let’s see some commands that we can use to manage our terraform state and situations where we need to use them.

Terraform State Lock

What is Terraform state locking? 

Some backends support a feature to lock our state. It means that it will lock our state for any operations that could change it.

This feature prevents others from obtaining the lock and potentially corrupting our state. 

How to lock Terraform State File

State locking occurs automatically for all operations that could change the state. So we won’t notice any message that it is occurring.

Terraform State Force Unlock

Now that we learned what the lock is. Let’s know how to unlock the Terraform State.

But, you may be thinking, why force unlock if we just learned that the lock protects us? 

Let me explain, giving you one problem that could happen to you:

Suppose that you run a “terraform plan,” and a few seconds later, you recognize that you forget something to do before. Immediately, you hit Control+C to cancel the command process. And here is the problem. 

Next time that you try to run the terraform plan again, you got some issues like this one:

╷
│ Error: Error acquiring the state lock
│ 
│ Error message: ConditionalCheckFailedException: The conditional request failed
│ Lock Info:
│   ID:        c1024f1b-b615-05cf-f512-e49ed2852284
│   Path:      <...>
│   Operation: OperationTypePlan
│   Who:       <...>
│   Version:   1.0.3
│   Created:   2022-02-28 12:14:31.112341 +0000 UTC
│   Info:      
│ 
│ 
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users simultaneously. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false."
│ flag, but this is not recommended.

Using Terraform force-unlock:

terraform force-unlock -force <ID>

So, according to the error message above, our ID is c1024f1b-b615-05cf-f512-e49ed2852284. So, in my case, I need to run:

terraform force-unlock -force c1024f1b-b615-05cf-f512-e49ed2852284

And if not work?

You can simply wait for a couple of minutes or half an hour and give it a try again. Also, you alternatively try to kill the terraform process running on your computer.

Terraform State Refresh

How to update the local state or remote file to match with changes and existing (real) resources in the cloud? Simple!  

We can quickly run:

terraform refresh

Move Terraform State

command:

terraform state mv

This command aims to move an object matched by the address given to the destination address. Also can move to a destination address in a totally different state file.                                                            

This can be utilized to easily rename resources or move objects to and from a module, move a complete module, and more additional options. And because this “terraform state mv” can also                                                           

 transfer data to an entirely new state, it can also be applied for refactoring one configuration into numerous individually managed Terraform configurations.                                                           

Also, a backup copy of the state before saving any modifications.

Removing Objects from Terraform State

command:

terraform state rm

Remove one or more objects from the Terraform state, forcing Terraform to “omit” those objects without destroying them in the cloud.

 This command removes objects from the Terraform state based on the addresses provided. You can check all available objects with the “terraform state list.”

We can’t combine the “terraform state rm” with a wildcard. Instead, we can use simple commands with pipe like xargs and grep.

For example:

terraform state list | grep "module.webserver" | xargs terraform state rm

The command above allows us to use “terraform state rm” to delete multiple resources.

But, if you want to delete just one object, your just need to run:

terraform state rm module.webserver.aws_instance.app

Show Resource attributes from Terraform State

command:

terraform state show

This command is handy when double-checking if our resources were created with our configuration. 

The command only works to show attributes from just one resource at once. So, when we specify the object’s address, it should point to just one resource.

example:

terraform state show module.webserver.aws_instance.app

The object above it’s an EC2 instance, and the output shows us the instance id, IPs, tags, and all configurations from that instance.

How to update a remote state using a local file?

command:

terraform state push

This command “pushes” a local state file and overwrites a remote state. Also, the command will protect us against changing an older serial or a distinct state file lineage unless we set the “-force” flag.

How to use:

terraform state push [options] PATH

Check Current State

command:

terraform state pull

This command pulls the state from its existing location, upgrades our local copy to the latest state file version agreeing with locally-installed Terraform, and outputs the raw format to stdout.

This helps read values out of state. The file is big, so you may use the jq command to help read it. 

Conclusion

In my opinion, Terraform state is the most important topic if you have just started your journey to learn Terraform. The state is the core of the Terraform, and all operations we do are against that state. So, we must know what we are doing with the state and avoid big mistakes.

We also learned that several subcommands inside the Terraform CLI help us manage our state without manually changing it.

Boost your Terraform skills:

Why to use the Terraform Workspace

How to use Terraform Modules

How to create complex expressions using Terraform Template.

What is the difference between Locals and Terraform Variables?

How to use the Terraform Data on your modules.

What are the advantages of Terraform Output?

Learn how to create multiple copies from the same resource on Terraform.

How to create a Pipeline on Gitlab to execute a Terraform code?

Leave a Comment

Your email address will not be published. Required fields are marked *

Free PDF with a useful Mind Map that illustrates everything you should know about AWS VPC in a single view.