s5cmd
Very fast parallel uploads and downloads with s5cmd.
s5cmd is a very fast command-line tool for S3. It copies thousands of files in parallel, which makes it a good choice for large folders and migrations. It has no setup wizard: you give it the keys in environment variables and the Zata endpoint on each command.
Before you start
- A bucket. See Create a bucket.
- An access key and secret key from the same region as the bucket. See Access keys.
Connection settings
| Setting | Value |
|---|---|
--endpoint-url (or S3_ENDPOINT_URL) | https://idr01.zata.ai for Indore, https://bom01.zata.ai for Mumbai |
AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY | from Access Keys in the Zata console. The key must be from the same region as the bucket. |
The examples below use Indore. For a bucket in Mumbai, use https://bom01.zata.ai.
Install s5cmd
Download the file for your system from the s5cmd releases page. On Linux (64-bit):
curl -sL -o s5cmd.tar.gz https://github.com/peak/s5cmd/releases/download/v2.3.0/s5cmd_2.3.0_Linux-64bit.tar.gz
tar xzf s5cmd.tar.gz s5cmd
sudo mv s5cmd /usr/local/bin/
s5cmd version
Give s5cmd your keys
In the terminal you will use:
export AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY
export AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY
To avoid typing the endpoint on every command, also set:
export S3_ENDPOINT_URL=https://idr01.zata.ai
To keep the keys in a file instead, put them in ~/.aws/credentials and add --profile zata to each command:
[zata]
aws_access_key_id = YOUR_ACCESS_KEY
aws_secret_access_key = YOUR_SECRET_KEY
Test it
Replace my-bucket with your bucket name:
s5cmd --endpoint-url https://idr01.zata.ai ls s3://my-bucket/
Copy many files
Put the source in single quotes, so that s5cmd (not the shell) expands the *. Sub-folders are kept.
Upload a folder:
s5cmd --endpoint-url https://idr01.zata.ai cp '/path/to/folder/*' s3://my-bucket/folder/
Upload only new or changed files (nothing is deleted in the bucket):
s5cmd --endpoint-url https://idr01.zata.ai sync '/path/to/folder/*' s3://my-bucket/folder/
Download a folder:
s5cmd --endpoint-url https://idr01.zata.ai cp 's3://my-bucket/folder/*' ./folder/
Large files are uploaded in parts automatically.
Speed settings
| Option | Where | What it does |
|---|---|---|
--numworkers 64 | before the command (s5cmd --numworkers 64 cp ...) | How many files are copied at the same time (default 256) |
-c 5 / --concurrency 5 | after cp | How many parts of one big file upload at the same time (default 5) |
-p 50 / --part-size 50 | after cp | Size of each part of a big file, in MiB (default 50) |
--stat | before the command | Prints a summary at the end |
Troubleshooting
| Problem | Fix |
|---|---|
AccessDenied or InvalidAccessKeyId | Check the two export lines (no extra spaces), and that the key can use this bucket and is from the bucket's region. |
NoSuchBucket | Use the endpoint of the region the bucket is in (idr01 or bom01). |
ls s3:// shows no buckets | A sub-user's key cannot list buckets. Use the bucket name directly. |
no match found | The local path or * pattern matches no file. Check the path. |
For other errors, see Access denied and Wrong endpoint or region.