How to solve the 400 error in uploading a file to S3 with calculated checksum

Summary:

In Feb, 2022, AWS announced S3 now supports Additional Checksum Algorithms for Amazon S3. With the input of checksum, users could ensure the uploaded file is correct or intact after uploading. In this article, we would show you how to do that with aws cli and related tools.

Let’s do it.

At first, you may check the Video for the introduction.

[embed]

However, when you try to calculated the checksum value, you may notice the value you calculate is different from the value of the object stored in s3.

This is expected, since S3 use base64 to encode the HEX string.

Please allow me to illustrate them by the below examples:

Part I: Examples about using Checksum in S3

Example 1: Check the CRC32Checksum

• Step 1: Upload a file to s3 with s3api and specify using CRC32 as the checksum algorithm:

$ aws s3api put-object --bucket <YOUR_BUCKET_NAME> --key 001.jpg  --checksum-algorithm CRC32   --body 001.jpg
{
    "ETag": "\"27bcea6815913dcb800078899d1a85a3\"",
    "ChecksumCRC32": "tT3VRA=="
}

You would have it retuned with message like below, and find the ChecksumCRC32 is presented.

• Step 2: Calculate the CRC32 checksum locally by using the command:

$ crc32 001.jpg
  b53dd544

Now you have the value “b53dd544”, which is obviously different from the value in Step 1(“tT3VRA==”).

• Step 3: Let’s check if they are really different, by using xxd and base64

$ crc32 001.jpg | xxd -r -p | base64
  tT3VRA==

Yes, the new value matches the one stored in S3.

Example 2: Check the SHA1Checksum Value

• Step 1: Upload a file to s3 with s3api and specify using SHA-1 as the checksum algorithm:

$ aws s3api put-object --bucket <YOUR_BUCKET_NAME> --key 002.jpg  --checksum-algorithm SHA1 --body 001.jpg
{
    "ETag": "\"27bcea6815913dcb800078899d1a85a3\"",
    "ChecksumSHA1": "xETSDwikrnRlBQX+L0wERl6WHXs="
 }

You would find the ChecksumSHA1 is presented.

• Step 2: Calculate the SHA1 checksum locally by using the command:

$ shasum -a 1 001.jpg
  c444d20f08a4ae74650505fe2f4c04465e961d7b  001.jpg

• Step 3: Let’s check if they are really different, by using xxd and base64

$ shasum -a 1 001.jpg | awk '{print $1}' | xxd -r -p | base64
  xETSDwikrnRlBQX+L0wERl6WHXs=

Yes, once again, we have the expected result that matches the value stored in S3.

Example 2: Check the SHA256 Checksum Value

For SHA256, I would just quick listed below

• Step 1:

$ aws s3api put-object --bucket <YOUR_BUCKET_NAME> --key 003.jpg  --checksum-algorithm SHA256 --body 001.jpg
  {
    "ETag": "\"27bcea6815913dcb800078899d1a85a3\"",
    "ChecksumSHA256": "HvSjjIjF1ySRc7ffEMcbLFHKEJlZO7b1U5G5BIYAx5k="
  }

• Step 2:

$ shasum -a 256 001.jpg | awk '{print $1}' | xxd -r -p | base64
  HvSjjIjF1ySRc7ffEMcbLFHKEJlZO7b1U5G5BIYAx5k=

For CRC-32C: (I have not found ways to calculate CRC-32C checksum yet. If you have found it, highly appreciated if you could leave in the comments.)

Part II: The commands we use

There are some command we used above.

**aws s3api put-object:**

• you can ask S3 to calculate the Checksum by using ‘ — checksum-algorithm <algorithm>’

--checksum-algorithm SHA1
  --checksum-algorithm SHA256
  --checksum-algorithm CRC32
  --checksum-algorithm CRC32C

• you can ask S3 to verify if the upload is intact by using options like ‘ — checksum-crc32 <value>’

--checksum-sha1
  --checksum-sha256
  --checksum-crc32
  --checksum-crc32-c

if the file does not match the checksum value, you would have error messages like below:

An error occurred (InvalidRequest) when calling the PutObject operation: Value for x-amz-checksum-crc32 header is invalid.

crc32

• to calculate the CRC32 checksum

$ crc32 001.jpg
  b53dd544

shasum

• to calculate the SHA checksum, and you can specify the algorith by ‘-a’, valide values for the case are 1 (default) & 256

$ shasum -a 1 001.jpg
  c444d20f08a4ae74650505fe2f4c04465e961d7b  001.jpg
  $ shasum -a 256 001.jpg
  1ef4a38c88c5d7249173b7df10c71b2c51ca1099593bb6f55391b9048600c799  001.jpg

xxd:

• make a hexdump or do the reverse. ‘-r’ means ‘to revert’ and ‘-p’ make it shown in plain, so that could be encoded by pipelined to base64.

$ xxd -r -p
  b53dd544
  �=�D

base64

• encode and decode using Base64 representation

$ echo -n "hello world"| base64
  aGVsbG8gd29ybGQ=

awk

• a very powerful text process program, and I only use it to parse the first column of the shasum result

$ echo -n "hello world" | awk '{print $1}'
  hello

Part III:

One more thing, about the ‘=’ padding.

This is expected as it is described in RFC 3548.

(1) the final quantum of encoding input is an integral multiple of 24
   bits; here, the final unit of encoded output will be an integral
   multiple of 4 characters with no "=" padding,
   (2) the final quantum of encoding input is exactly 8 bits; here, the
   final unit of encoded output will be two characters followed by two
   "=" padding characters, or
   (3) the final quantum of encoding input is exactly 16 bits; here, the
   final unit of encoded output will be three characters followed by one
   "=" padding character.

— — — — — — — — — — — — —

Refernce Data:

• https://github.com/minio/minio/issues/14885
• Online Base64 encode page: https://emn178.github.io/online-tools/base64_encode.html
• RFC 3548: The Base16, Base32, and Base64 Data Encodings — https://datatracker.ietf.org/doc/html/rfc3548#page-4
• https://aws.amazon.com/blogs/aws/new-additional-checksum-algorithms-for-amazon-s3/
• AWS CLI: s3api — https://awscli.amazonaws.com/v2/documentation/api/latest/reference/s3api/put-object.html

=== End of the document ===