Copying Objects in S3 From One Bucket to Another Automatically AWS Lambda

There are times when you need to copy objects from one S3 bucket to another. Amazon Web Services makes this simple with a little Lambda magic. The following solution copies objects from a source bucket to a destination bucket, and is triggered by successful PUT requests made to the source bucket. You might find that your solution requires some modification of what I’ve created here. For instance, you might need to run this on a schedule or trigger it for other object creation requests in addition to PUT.

S3 Object Copy

Let’s get started.

Creating an IAM policy for access permissions:

  1. Navigate to IAM in your management console.
  2. Select “Policies” in the sidebar.
  3. Click “Create Policy”.
  4. Select “Create Your Own Policy”.
  5. Enter an appropriate policy name and description.
  6. Paste the following JSON into the policy document:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "s3:GetObject"
                ],
                "Resource": [
                    "Your Source Bucket ARN/*"
                ]
            },
            {
                "Effect": "Allow",
                "Action": [
                    "s3:PutObject"
                ],
                "Resource": [
                    "Your Destination Bucket ARN/*"
                ]
            }
        ]
    }
  7. Substitute “Your Source Bucket ARN” with the ARN for the S3 bucket that you want to copy objects from. Make sure you add “/*” after the bucket ARN. For instance, if your bucket ARN was “arn:aws:s3:::theemperorprotects”, you would use “arn:aws:s3:::theemperorprotects/*”.
  8. Substitute “Your Destination Bucket ARN” with the ARN for the S3 bucket that you want to copy objects to. Make sure you add “/*” after the bucket ARN. For instance, if your bucket ARN was “arn:aws:s3:::theemperorprotects”, you would use “arn:aws:s3:::theemperorprotects/*”.
  9. Click “Create Policy”.

Creating the IAM role for the Lambda function:

  1. Select “Roles” in the sidebar.
  2. Click “Create New Role”.
  3. Enter an appropriate role name and click “Next Step”.
  4. Select “AWS Lambda” within the AWS Service Roles.
  5. Change the filter to “Customer Managed”, check the box of the policy you just created, and click “Next Step”.
  6. Click “Create Role”.

Creating the Lambda function:

  1. Navigate to Lambda in your management console.
  2. Click “Create a Lambda function”.
  3. Select the “Blank Function” blueprint.
  4. Under “Configure triggers”, click the grey box and select “S3”.
  5. Select the source bucket you want to copy objects from for the Bucket.
  6. Select the appropriate Event type. For this example, I’m using “Put”.
  7. Enter a Prefix and/or Suffix if you want. For this example, I left these blank.
  8. Check the box to “Enable trigger” and click “Next”.
  9. Click “Next”.
  10. Enter an appropriate function name and description. Select Node.js for the runtime.
  11. Under “Lambda function code”, select “Edit code inline” for the Code entry type and paste the following code in the box:
  12. 1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    var AWS = require("aws-sdk");
    exports.handler = (event, context, callback) => {
        var s3 = new AWS.S3();
        var sourceBucket = "Source Bucket Name";
        var destinationBucket = "Destination Bucket Name";
        var objectKey = event.Records[0].s3.object.key;
        var copySource = encodeURI(sourceBucket + "/" + objectKey);
        var copyParams = { Bucket: destinationBucket, CopySource: copySource, Key: objectKey };
        s3.copyObject(copyParams, function(err, data) {
            if (err) {
                console.log(err, err.stack);
            } else {
                console.log("S3 object copy successful.");
            }
        });
    };
  13. Substitute “Source Bucket Name” with the name of the bucket you want to copy objects from.
  14. Substitute “Destination Bucket Name” with the name of the bucket you want to copy objects to.
  15. Leave Handler as “index.handler”.
  16. Choose to use an existing role and select the IAM role you created earlier.
  17. Leave the other default values and click “Next”.
  18. Click “Create function”.

Everything should be in place now. You can test the Lambda function by uploading any object to your source bucket and checking to make sure the same object appears in your destination bucket. Feel free to modify this for your specific needs.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s