Amazon S3 Inventory, in our last post, you learn that we can use it with S3 Batch Operations. Also, it can help you unlock the power of Amazon S3 and simplify your storage management. With this tool, you can audit and report your objects’ replication and encryption status for business, compliance, and regulatory needs. With Amazon S3 Inventory’s scheduled approach, you can accelerate business workflows and large-scale data jobs more quickly than using the synchronous List API operation.
What is Amazon S3 Inventory?
Amazon S3 Inventory is an automated inventory solution that provides comma-separated values (CSV) and Apache-optimized row columnar (ORC) object manifest files, which list your objects and their corresponding metadata on a scheduled basis. These files are stored in an Amazon S3 bucket. They can be used to audit your objects’ replication and encryption status and automate processes such as data archiving, analytics, and compliance.
Apache-optimized row columnar (ORC) is an optimized storage format for large data sets that can be used with Amazon S3 Inventory. ORC files are stored in the same Amazon S3 bucket as the CSV and JSON file outputs of Amazon S3 Inventory, providing a faster way to read, process, and query your objects than CSV files. This means that your data can be easily accessed for analysis or other tasks.
Benefits of Amazon S3 Inventory
Using Amazon S3 Inventory, you can audit and report on your objects’ replication and encryption status for business, compliance, and regulatory needs; speed up business workflows and big data jobs; and quickly access your data for analysis or other tasks. These benefits are made possible through Amazon S3 Inventory’s scheduled alternative to the Amazon S3 synchronous List API operation, as well as its comma-separated values (CSV) and Apache-optimized row columnar (ORC) object manifest files.
How it works
Amazon S3 Inventory is triggered daily, weekly, or monthly depending on your settings. It produces manifest files that contain a list of all your objects and their associated metadata in either CSV or ORC format. These files are stored in a specified bucket, so you can easily download them and use them to audit replication and encryption status. Additionally, Amazon S3 Inventory does not affect the request rate of your bucket, so it will not interfere with other operations.
To get started, you need to configure your source and destination buckets. After that, Amazon S3 Inventory will create manifest files to help you audit and report on the objects in your bucket.
Don’t forget to download the AWS Learning Kit to help you boost your AWS Skills and push your career to the next level!
Inventory List Metadata
Amazon S3 Inventory creates a comprehensive inventory of the objects in your bucket – including each object’s metadata: key (the object’s name), version ID, last modified date, ETag (a hash of the object content), size, storage class, replication status, encryption status, and owner information. For example, the key could be “example.jpg”, the version ID could be “1234ABCD”, and the size could be 1024 bytes. The encryption status is a boolean value (true/false) that indicates whether or not the object is encrypted with server-side encryption. Finally, the owner information contains details about who owns the thing in the bucket, such as the AWS account ID.
These manifest files are stored in an Amazon S3 bucket, allowing you to easily access them to audit and report on your objects’ replication and encryption status, automate data archiving, analytics, and compliance processes, and speed up workflows and big data jobs. With Amazon S3 Inventory, you can quickly access your data for analysis or other tasks without having to use the synchronous List API operation. All this makes it easier than ever to gain insights from your data and ensure that your organization’s compliance needs are met. By using Amazon S3 Inventory, you can optimize your storage costs and maximize the value of your data.
Inventory consistency
Inventory is consistent with all objects in the bucket and is listed at the time of the inventory run. In addition, Amazon S3 guarantees that all related versions and delete markers will be included in your inventory list. This means you can trust that your reports accurately reflect the status of your data over time for business, compliance, and regulatory needs.
To ensure data integrity, Amazon S3 Inventory also verifies and optionally encrypts manifest files before storing them in the destination bucket. Encryption ensures that all information is secure, so you can trust that your reports will not be compromised during transit or storage.
Querying Amazon S3 Inventory with Amazon Athena
Amazon Athena makes it easy to query your Amazon S3 Inventory data. Just point Athena at the destination bucket and specify either CSV or ORC format, and you can quickly run SQL queries against the inventory files. This allows you to gain insights from your data in a fraction of the time it would take with other solutions. With Amazon Athena, you can quickly find the objects in your bucket that meet specific criteria—like encryption status or storage class—or identify any anomalies in your data.
Locating your Inventory List
When an inventory list is published, the manifest files are created to the following location in the destination bucket: <bucket_name>/<prefix>/<date>/<hour of day>. To find your inventory list, navigate to this location, and you will find all of your manifests. You can also access them programmatically using the AWS SDKs or CLI.
Where you can use your Inventory List
The inventory list generated by Amazon S3 Inventory can be used with many other AWS services, such as Amazon Athena, Amazon EMR, AWS Glue, Amazon Redshift, Amazon S3 Analytics, and the AWS Lambda service. Additionally, you can use the inventory list to build custom applications and automated workflows that use the data for reporting, analytics, auditing, archiving, and compliance purposes.
Also, you can use S3 Batch Operations to copy, move, or change the storage class of your objects based on the information generated in the inventory list. With S3 Batch Operations, you can quickly process thousands of objects in minutes, allowing you to save time and money on data processing tasks.
Fixing Empty version ID strings in Inventory reports to null strings
In Inventory reports, empty version IDs are represented as strings that consist of a pair of double quotation marks (“”). Sometimes, this can cause problems for downstream applications that may interpret the empty strings as valid values. To avoid this issue, you can use Athena to convert empty strings to null values in the inventory report. This will help ensure that your applications can adequately interpret the data from the Amazon S3 Inventory report.
To guarantee the success of your S3 Batch Operations job, a simple procedure can be done to replace empty strings in the version ID field with null strings. This should be completed beforehand for an all-versions S3 Inventory report as its manifests – excluding this step could lead to failed tasks on these objects!
Conclusion
Amazon S3 Inventory can help you simplify and speed up your business workflows and big data jobs. With this tool, you can easily audit and report your objects’ replication and encryption status for business, compliance, and regulatory needs. It also does not affect the request rate of your bucket, so you can rest assured that your other operations won’t be affected.
Turbocharge your AWS skillset and jumpstart your career with the help of the AWS Learning Kit – don’t miss out on this invaluable resource!
