9 AWS S3 Commands with Examples to Manage Bucket and Data

Control and management of data can be a strenuous task. These AWS S3 commands will help you quickly and efficiently manage your AWS S3 buckets and Data.

AWS S3 is the object storage service provided by AWS. It is the most widely used storage service from AWS that can virtually hold an infinite amount of data. It is highly available, durable, and easy to integrate with several other AWS Services.  

AWS S3 can be used by people with any requirements like mobile/web application storage, big data storage, machine learning data storage, hosting static websites, and many more.

If you have been using S3 in your project, you would know that given the vast amount of storage capacity, the management of 100s of buckets and terabytes of data in these buckets can be a demanding job. We have a list of AWS S3 commands with examples that you can use to manage your AWS S3 buckets and data efficiently.

AWS CLI Setup

After you have successfully downloaded and installed the AWS CLI, you need to configure AWS Credentials to be able to access your AWS Account and services. Let us quickly run through how you can configure AWS CLI.

The first step is to create a user with programmatic access to AWS Account. Remember to check this box when you create a user for AWS CLI.

Give the permissions and create a user. At the final screen after you successfully created this user copy the Access key ID and Secret access key for this user. We will use these credentials to log in via the AWS CLI.

Now go to the terminal of your choice and run the following command.

aws configure 

Enter the Access key ID and Secret access key when prompted. Select any AWS region of your choice and the command output format. I personally prefer using the JSON format. This is no big deal you can always change these values later.

You can now run any AWS CLI Command in the console. Let us now go through the AWS S3 Commands.

cp

The cp command simply copies the data to and from S3 buckets. It can be used to copy files from local to S3, from S3 to local, and between two S3 buckets. There are a lot of other parameters that you can supply with the commands.

For example, -dryrun parameter to test the command, –storage-class parameter to specify the storage class of your data in S3, other parameters to set encryption, and much more. The cp command gives you complete control over how you configure your data security in S3.

Usage

aws s3 cp   [--options]

Examples

Copy data from local to S3

aws s3 cp file_name.txt s3://bucket_name/file_name_2.txt

Copy data from S3 to local

aws s3 cp s3://bucket_name/file_name_2.txt file_name.txt

Copy data between S3 buckets

aws s3 cp s3://bucket_name/file_name.txt s3://bucket_name_2/file_name_2.txt

Copy data from local to S3 – IA

aws s3 cp file_name.txt s3://bucket_name/file_name_2.txt --storage-class STANDARD_IA 

Copy all the data from a local folder to S3

aws s3 cp ./local_folder s3://bucket_name --recursive

ls

The ls command is used to list the buckets or the contents of the buckets. So, if you simply want to view information about your buckets or the data in these buckets you can use the ls command.

Usage:

aws s3 ls NONE or  [--options]

Examples

List all buckets in the account

aws s3 ls

Output:
2022-02-02 18:20:14 BUCKET_NAME_1
2022-03-20 13:12:43 BUCKET_NAME_2
2022-03-29 10:52:33 BUCKET_NAME_3

This command list all the buckets in your account with the bucket creation date.

List all top-level objects in a bucket

aws s3 ls BUCKET_NAME_1 or s3://BUCKET_NAME_1 

Output:
                           PRE samplePrefix/
2021-12-09 12:23:20       8754 file_1.png
2021-12-09 12:23:21       1290 file_2.json
2021-12-09 12:23:21       3088 file_3.html

This command list all the top-level objects in an S3 bucket. Note here that the objects with the prefix samplePrefix/ are not shown here only the top-level objects.

List all the objects in a bucket

aws s3 ls BUCKET_NAME_1 or s3://BUCKET_NAME_1 --recursive

Output:
2021-12-09 12:23:20       8754 file_1.png
2021-12-09 12:23:21       1290 file_2.json
2021-12-09 12:23:21       3088 file_3.html
2021-12-09 12:23:20      16328 samplePrefix/file_1.txt
2021-12-09 12:23:20      29325 samplePrefix/sampleSubPrefix/file_1.css

This command list all the objects in an S3 bucket. Note here that the objects with the prefix samplePrefix/ and all the sub prefixes are also displayed.

mb

The mb command is simply used to create new S3 buckets. This is a fairly simple command but to create new buckets, the name of the new bucket should be unique across all the S3 buckets.

Usage

aws s3 mb 

Example

Create a new bucket in a specific region

aws s3 mb myUniqueBucketName --region eu-west-1

mv

The mv command simply moves the data to and from S3 buckets. Just like the cp command, mv command is used to move data from local to S3, S3 to local, or between two S3 buckets.

The only difference between the mv and the cp command is that when using the mv command the file is deleted from the source. AWS moves this file to the destination. There are a lot of options that you can specify with the command.

Usage

aws s3 mv   [--options]

Examples

Move data from local to S3

aws s3 mv file_name.txt s3://bucket_name/file_name_2.txt

Move data from S3 to local

aws s3 mv s3://bucket_name/file_name_2.txt file_name.txt

Move data between S3 buckets

aws s3 mv s3://bucket_name/file_name.txt s3://bucket_name_2/file_name_2.txt

Move data from local to S3 – IA

aws s3 mv file_name.txt s3://bucket_name/file_name_2.txt --storage-class STANDARD_IA 

Move all the data from a prefix in S3 to a local folder.

aws s3 mv s3://bucket_name/somePrefix ./localFolder --recursive

presign

The presign command generates a pre-signed URL for a key in the S3 bucket. You can use this command to generate URLs that can be used by others to access a file in the specified S3 bucket key.

Usage

aws s3 presign