S3 Inventory configuration helps to manage the storage on AWS. It generates audit reports on the replication and encryption status of S3 objects for business, compliance, and regulatory needs. S3 Inventory configuration generates comma-separated values (CSV), Apache optimized row columnar (ORC) or Apache Parquet (Parquet) output files.
It comes with some cost, to know about its pricing click here.
In this article, we will create an inventory configuration to store object metadata in CSV format. We can store reports in the same bucket as the source bucket, but we will use a different bucket to store this than the bucket for which we will be creating this inventory configuration. It is to be noted that the destination bucket in which reports will be stored must be in the same AWS Region as the source bucket.
To know more about Inventory Configuration in detail, visit the AWS official documentation here.
Pre-requisites
- AWS Account (Create if you don’t have one).
- 2 S3 Buckets in the same region(Click here to learn to create an S3 bucket).
What will we do?
- Login to AWS.
- Create an Inventory configuration
Login to AWS
Click here to go to AWS Login Page. Enter your user credentials to login into your AWS account.
See the main AWS management console as follows when you successfully login into your AWS account.
Create an Inventory Configuration
Click on services in the top left of the screen and search for S3.
This is the main dashboard of the S3 bucket. Here you can see all the buckets from your account. Click on the bucket for which you want to create an inventory configuration.
If you do not have any bucket created under your account create 2 buckets before proceeding. Click here to create S3 buckets if you don’t have any in your account before you proceed. You can also have one bucket and use it for both source and destination fields.
On the bucket overview page, you can see multiple tabs. Click on management tabs.
Scroll down at the end of the page and you will see the inventory configuration section. Here click on the “Create Inventory Configuration” button to create your first inventory configuration.
Give a name to the configuration and select the destination Bucket where you want this report to be saved. Make a note that the destination bucket must belong to the same region as that of the source bucket for which you are creating this inventory configuration.
You can even limit your scope by adding a prefix, this is an optional field.
The destination bucket where the reports will be stored can belong to the same account or a different account. Here the bucket I have chosen belongs to the same account and is in the same region as the source bucket.
Choose the frequency and output format. You can also enable or disable the configuration as per your need. To publish the inventory report change the status to “Enable”.
Select all metadata that should be included for each listed object in the report and click on the “Create” button.
Once the inventory configuration is successfully created you can see it under the management tab. Note that it may take up to 48 hours to deliver the first report.
Once the report is generated you can check your destination bucket and you will find the reports under it. It takes around 48 hours to generate the first report.
You can download the report once it is generated and use the following command, on Linux/Mac, to extract the CSV file.
gunzip 20bd9db9-fddd-4cb0-a630-ecfce7bd3c10.csv.gz
Since we had specified output format as CSV the report has been generated in CSV format.
In the above screenshot, you can see that the CSV contains only a few additional fields, i.e. It contains size, last modified and the storage type as additional metadata fields. If you are not happy with these fields and want a few more you can choose those fields from the available list while creating an inventory configuration.
Conclusion
In this article, we created an inventory configuration that stores metadata in CSV format in a different S3 bucket. We used different buckets for source and destination and selected only a few fields to be stored in the report. You can choose the metadata fields from the available list to be stored in the report as per your requirement.
