AWS CLI
Set up a Zata profile and copy files with the AWS CLI.
The AWS CLI is Amazon's command-line tool for S3. With a separate zata profile, you can list, upload, download and share files in your Zata buckets, and keep your AWS settings untouched.
Before you start
- A bucket. See Create a bucket.
- An access key and secret key from the same region as the bucket. See Access keys.
- AWS CLI version 2.13 or newer. Check with
aws --version.
Connection settings
| Setting | Value |
|---|---|
Endpoint (endpoint_url) | https://idr01.zata.ai for Indore, https://bom01.zata.ai for Mumbai |
| Region | us-east-1 |
| Access key / Secret key | from Access Keys in the Zata console. The key must be from the same region as the bucket. |
| Addressing style | path |
| Checksums | when_required (both settings below) |
Install the AWS CLI
- Linux (x86_64)
- Windows, macOS, ARM Linux
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o awscliv2.zip
unzip -q awscliv2.zip
sudo ./aws/install
aws --version
Download the installer for your system from the AWS CLI install guide, then check the version:
aws --version
Set up a Zata profile
aws configure --profile zata
Answer the questions:
| When it asks | Type |
|---|---|
| AWS Access Key ID | your access key |
| AWS Secret Access Key | your secret key |
| Default region name | us-east-1 |
| Default output format | json |
For a bucket in Mumbai, use https://bom01.zata.ai in the first line.
aws configure set endpoint_url https://idr01.zata.ai --profile zata
aws configure set s3.addressing_style path --profile zata
aws configure set request_checksum_calculation when_required --profile zata
aws configure set response_checksum_validation when_required --profile zata
The two checksum lines are needed: without them, large uploads fail with XAmzContentSHA256Mismatch.
These commands work the same in a Linux, macOS or Windows terminal. They save the settings in ~/.aws/config and ~/.aws/credentials (on Windows, in %USERPROFILE%\.aws\).
On AWS CLI versions older than 2.13, the profile cannot store the endpoint. Add --endpoint-url https://idr01.zata.ai to every command instead.
Test it
Replace my-bucket with your bucket name:
aws s3 ls s3://my-bucket --profile zata
aws s3 cp myfile.txt s3://my-bucket/ --profile zata
aws s3 ls --profile zata lists your buckets. A sub-user's key shows no buckets there; use the bucket name directly.
Everyday commands
# Download a file
aws s3 cp s3://my-bucket/myfile.txt ./myfile.txt --profile zata
# Upload a folder
aws s3 cp ./my-folder s3://my-bucket/my-folder/ --recursive --profile zata
# Upload only new and changed files
aws s3 sync ./my-folder s3://my-bucket/my-folder/ --profile zata
# Delete a file
aws s3 rm s3://my-bucket/myfile.txt --profile zata
# Make a download link that works for 1 hour (3600 seconds)
aws s3 presign s3://my-bucket/myfile.txt --expires-in 3600 --profile zata
# Check that a bucket exists in this region
aws s3api head-bucket --bucket my-bucket --profile zata
aws s3 cp uploads files larger than 8 MB in parts (multipart) by itself. aws s3 presign makes download links only; for upload links, use an SDK such as Python (boto3).
Test a multipart upload step by step
aws s3 cp does all of this by itself. Doing it by hand shows which step fails when another app cannot upload large files. Every part except the last must be at least 5 MB.
On Linux or macOS:
head -c 12M /dev/urandom > bigfile.bin
split -b 5M -d bigfile.bin part-
This makes part-00 and part-01 (5 MB each) and part-02 (2 MB).
The command returns an upload ID:
UPLOAD_ID=$(aws s3api create-multipart-upload --bucket my-bucket --key test/bigfile.bin --query UploadId --output text --profile zata)
echo "$UPLOAD_ID"
Parts can be uploaded in any order:
aws s3api upload-part --bucket my-bucket --key test/bigfile.bin --part-number 1 --body part-00 --upload-id "$UPLOAD_ID" --profile zata
aws s3api upload-part --bucket my-bucket --key test/bigfile.bin --part-number 2 --body part-01 --upload-id "$UPLOAD_ID" --profile zata
aws s3api upload-part --bucket my-bucket --key test/bigfile.bin --part-number 3 --body part-02 --upload-id "$UPLOAD_ID" --profile zata
Save the list for the last step:
aws s3api list-parts --bucket my-bucket --key test/bigfile.bin --upload-id "$UPLOAD_ID" --query '{Parts: Parts[].{PartNumber: PartNumber, ETag: ETag}}' --output json --profile zata > parts.json
cat parts.json
Zata joins the parts into one file:
aws s3api complete-multipart-upload --bucket my-bucket --key test/bigfile.bin --upload-id "$UPLOAD_ID" --multipart-upload file://parts.json --profile zata
To cancel instead (this deletes the parts uploaded so far):
aws s3api abort-multipart-upload --bucket my-bucket --key test/bigfile.bin --upload-id "$UPLOAD_ID" --profile zata
If the complete step fails, Zata removes that upload: start again with a new upload ID.
Unfinished uploads keep using storage until they are completed or cancelled. To find them:
aws s3api list-multipart-uploads --bucket my-bucket --profile zata
A sub-user's key may get AccessDenied on this list; the bucket owner's key works.
| Step | Error | Meaning and fix |
|---|---|---|
| Upload part | XAmzContentSHA256Mismatch | The default checksums. Set the two when_required lines from Set up a Zata profile. |
| Complete | EntityTooSmall | A part other than the last is under 5 MB. Use bigger parts. |
| Complete | InvalidPart | An ETag or part number in parts.json does not match what was uploaded. Make the file again with list-parts. |
| Any step | NoSuchUpload or NoSuchKey | The upload ID is wrong, or the upload was already completed or cancelled. Start a new one. |
| Any step | AccessDenied | The key is not allowed this action on the bucket. |
If these steps work with the AWS CLI but another app still fails on large files, that app most likely sends the default checksums. See Tools built on the AWS SDK.
Troubleshooting
| Problem | Fix |
|---|---|
argument of type 'NoneType' is not a container or iterable | The AWS CLI could not show Zata's error. Run the same command again with --debug at the end and look for <Code>…</Code>: that is the real error, for example AccessDenied or XAmzContentSHA256Mismatch. |
Large uploads fail with XAmzContentSHA256Mismatch; small files work | Set the two checksum lines from Set up a Zata profile. |
InvalidAccessKeyId | The endpoint is missing, so requests go to AWS; or the key is from the other region. Check with aws configure get endpoint_url --profile zata. |
NoSuchBucket for a bucket you can see in the console | The bucket is in the other region. Use that region's endpoint and a key from that region. |
AccessDenied when you create a bucket | Sub-user keys cannot create buckets. Create it in the console or with the owner's key. |
aws configure list --profile zata shows which keys and settings the profile really uses. For other errors, see Signature errors, Access denied and Wrong endpoint or region.
Never use --no-verify-ssl. Zata's certificates are valid, so a certificate error points to a proxy or a wrong clock. See Network and certificates.