Scheduling Automated AMI Backups of Your EC2 Instances

When considering disaster recovery options for systems or applications running on Amazon Web Services, a frequent solution is to use AMIs to restore instances to a known acceptable state in the event of failure or catastrophe. If you’re team has decided on this approach, you’ll want to automate the creation and maintenance of these AMIs to prevent mistakes or somebody “forgetting” to do the task. In this post, I’ll walk through how to set this up in AWS within a matter of minutes using Amazon’s serverless compute offering, Lambda.

Automated AMI Backups

In departure from many of my other posts involving Lambda, the following steps will make use of the AWS CLI so I will assume that you’ve already installed and configured it on your machine. This is to grant longevity to these posts and to protect their relevance from slipping due to the browser-based console’s rate of change.

I have also added some flexibility in the maintenance of these backups I encourage the reader to configure to their liking. These include a variable number of backups the reader would like to maintain for each EC2 instance they wish to have AMIs taken, a customizable tag the reader can assign to EC2 instances they’d like to backup, and the option to also delete snapshots of the AMIs being de-registered once they exit the backup window. I have included default values, but I still encourage you to read the options before implementing this solution.

Let’s get started.

Creating an IAM policy for access permissions:

  1. Create a file named iam-policy.json with the following contents and save it in your working directory:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "Stmt1499061014000",
                "Effect": "Allow",
                "Action": [
                    "ec2:CreateImage",
                    "ec2:CreateTags",
                    "ec2:DeleteSnapshot",
                    "ec2:DeregisterImage",
                    "ec2:DescribeImages",
                    "ec2:DescribeInstances"
                ],
                "Resource": [
                    "*"
                ]
            }
        ]
    }
  2. In your command prompt or terminal window, invoke the following command:
    1
    aws iam create-policy --policy-name ami-backup-policy --policy-document file://iam-policy.json
  3. You’ll receive output with details of the policy you’ve just created. Write down the ARN value as you will need it later.

Creating the IAM role for the Lambda function:

  1. Create a file named role-trust-policy.json with the following contents and save it in your working directory:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "lambda.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
  2. In your command prompt or terminal window, invoke the following command:
    1
    aws iam create-role --role-name ami-backup-role --assume-role-policy-document file://role-trust-policy.json
  3. You’ll receive output with details of the role you’ve just created. Be sure to write down the role ARN value provided. You’ll need it later.
  4. Run the following command to attach the policy to the role. You must substitute ARN for the policy ARN you wrote down from the prior step:
    1
    aws iam attach-role-policy --policy-arn ARN --role-name ami-backup-role

Creating the Lambda function:

  1.  Create a file named index.js with the following contents and save it in your working directory:
    Note: The following file is the code managing your AMI backups. There are a number of configurable options to be aware of and I have commented descriptions of each in the code. 

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    119
    120
    121
    122
    123
    124
    125
    126
    127
    128
    129
    130
    131
    132
    133
    134
    135
    136
    137
    138
    139
    140
    141
    142
    143
    144
    145
    146
    147
    148
    149
    150
    151
    152
    153
    154
    155
    156
    157
    158
    159
    160
    161
    162
    163
    164
    165
    166
    167
    168
    169
    170
    171
    172
    173
    174
    175
    176
    177
    178
    var AWS = require("aws-sdk");
    var ec2 = new AWS.EC2();
    var numBackupsToRetain = 2; // The Number Of AMI Backups You Wish To Retain For Each EC2 Instance.
    var instancesToBackupTagName = "BackupAMI"; // Tag Key Attached To Instances You Want AMI Backups Of. Tag Value Should Be Set To "Yes".
    var imageBackupTagName = "ScheduledAMIBackup"; // Tag Key Attached To AMIs Created By This Process. This Process Will Set Tag Value To "True".
    var imageBackupInstanceIdentifierTagName = "ScheduledAMIInstanceId"; // Tag Key Attached To AMIs Created By This Process. This Process Will Set Tag Value To The Instance ID.
    var deleteSnaphots = true; // True if you want to delete snapshots during cleanup. False if you want to only delete AMI, and leave snapshots intact.
    exports.handler = function(event, context) {
        var describeInstancesParams = {
            DryRun: false,
            Filters: [{
                Name: "tag:" + instancesToBackupTagName,
                Values: ["Yes"]
            }]
        };
        ec2.describeInstances(describeInstancesParams, function(err, data) {
            if (err) {
                console.log("Failure retrieving instances.");
                console.log(err, err.stack);
            }
            else {
                for (var i = 0; i < data.Reservations.length; i++) {
                    for (var j = 0; j < data.Reservations[i].Instances.length; j++) {
                        var instanceId = data.Reservations[i].Instances[j].InstanceId;
                        createImage(instanceId);
                    }
                }
            }
        });
        cleanupOldBackups();
    };
    var createImage = function(instanceId) {
        console.log("Found Instance: " + instanceId);
        var createImageParams = {
            InstanceId: instanceId,
            Name: "AMI Scheduled Backup I(" + instanceId + ") T(" + new Date().getTime() + ")",
            Description: "AMI Scheduled Backup for Instance (" + instanceId + ")",
            NoReboot: true,
            DryRun: false
        };
        ec2.createImage(createImageParams, function(err, data) {
            if (err) {
                console.log("Failure creating image request for Instance: " + instanceId);
                console.log(err, err.stack);
            }
            else {
                var imageId = data.ImageId;
                console.log("Success creating image request for Instance: " + instanceId + ". Image: " + imageId);
                var createTagsParams = {
                    Resources: [imageId],
                    Tags: [{
                        Key: "Name",
                        Value: "AMI Backup I(" + instanceId + ")"
                    },
                    {
                        Key: imageBackupTagName,
                        Value: "True"
                    },
                    {
                        Key: imageBackupInstanceIdentifierTagName,
                        Value: instanceId
                    }]
                };
                ec2.createTags(createTagsParams, function(err, data) {
                    if (err) {
                        console.log("Failure tagging Image: " + imageId);
                        console.log(err, err.stack);
                    }
                    else {
                        console.log("Success tagging Image: " + imageId);
                    }
                });
            }
        });
    };
    var cleanupOldBackups = function() {
        var describeImagesParams = {
            DryRun: false,
            Filters: [{
                Name: "tag:" + imageBackupTagName,
                Values: ["True"]
            }]
        };
        ec2.describeImages(describeImagesParams, function(err, data) {
            if (err) {
                console.log("Failure retrieving images for deletion.");
                console.log(err, err.stack);
            }
            else {
                var images = data.Images;
                var instanceDictionary = {};
                var instances = [];
                for (var i = 0; i < images.length; i++) {
                    var currentImage = images[i];
                    for (var j = 0; j < currentImage.Tags.length; j++) {
                        var currentTag = currentImage.Tags[j];
                        if (currentTag.Key === imageBackupInstanceIdentifierTagName) {
                            var instanceId = currentTag.Value;
                            if (instanceDictionary[instanceId] === null || instanceDictionary[instanceId] === undefined) {
                                instanceDictionary[instanceId] = [];
                                instances.push(instanceId);
                            }
                            instanceDictionary[instanceId].push({
                                ImageId: currentImage.ImageId,
                                CreationDate: currentImage.CreationDate,
                                BlockDeviceMappings: currentImage.BlockDeviceMappings
                            });
                            break;
                        }
                    }
                }
                for (var t = 0; t < instances.length; t++) {
                    var imageInstanceId = instances[t];
                    var instanceImages = instanceDictionary[imageInstanceId];
                    if (instanceImages.length > numBackupsToRetain) {
                        instanceImages.sort(function (a, b) {
                           return new Date(b.CreationDate) - new Date(a.CreationDate);
                        });
                        for (var k = numBackupsToRetain; k < instanceImages.length; k++) {
                            var imageId = instanceImages[k].ImageId;
                            var creationDate = instanceImages[k].CreationDate;
                            var blockDeviceMappings = instanceImages[k].BlockDeviceMappings;
                            deregisterImage(imageId, creationDate, blockDeviceMappings);
                        }  
                    }
                    else {
                        console.log("AMI Backup Cleanup not required for Instance: " + imageInstanceId + ". Not enough backups in window yet.");
                    }
                }
            }
        });
    };
    var deregisterImage = function(imageId, creationDate, blockDeviceMappings) {
        console.log("Found Image: " + imageId + ". Creation Date: " + creationDate);
        var deregisterImageParams = {
            DryRun: false,
            ImageId: imageId
        };
        console.log("Deregistering Image: " + imageId + ". Creation Date: " + creationDate);
        ec2.deregisterImage(deregisterImageParams, function(err, data) {
           if (err) {
               console.log("Failure deregistering image.");
               console.log(err, err.stack);
           }
           else {
               console.log("Success deregistering image.");
               if (deleteSnaphots) {
                    for (var p = 0; p < blockDeviceMappings.length; p++) {
                       var snapshotId = blockDeviceMappings[p].Ebs.SnapshotId;
                       if (snapshotId) {
                           deleteSnapshot(snapshotId);
                       }
                   }   
               }
           }
        });
    };
    var deleteSnapshot = function(snapshotId) {
        var deleteSnapshotParams = {
            DryRun: false,
            SnapshotId: snapshotId
        };
        ec2.deleteSnapshot(deleteSnapshotParams, function(err, data) {
            if (err) {
                console.log("Failure deleting snapshot. Snapshot: " + snapshotId + ".");
                console.log(err, err.stack);
            }
            else {
                console.log("Success deleting snapshot. Snapshot: " + snapshotId + ".");
            }
        })
    };
  2. Zip this file to a zip called index.zip.
  3. In your command prompt or terminal window, invoke the following command. You must substitute ARN for the role ARN you wrote down from the prior step:
    1
    aws lambda create-function --function-name ami-backup-function --runtime nodejs6.10 --handler index.handler --role ARN --zip-file fileb://index.zip --timeout 30
  4. You’ll receive output details about the Lambda function you’ve just created. Write down the Function ARN value for later use.

Scheduling the Lambda function:

  1. In your command prompt or terminal window, invoke the following command:
    Note: Feel free to adjust the schedule expression for your own use.

    1
    aws events put-rule --name -ami-backup-event-rule --schedule-expression "rate(1 day)"
  2. You’ll get the Rule ARN value back as output. Write this down for later.
  3. Run the following command. Substitute ARN for the Rule ARN you just wrote down:
    1
    aws lambda add-permission --function-name ami-backup-function --statement-id LambdaPermission --action "lambda:InvokeFunction" --principal events.amazonaws.com --source-arn ARN
  4. Run the following command. Substitute ARN for the Function ARN of the Lambda function you wrote down:
    1
    aws events put-targets --rule ami-backup-event-rule --targets "Id"="1","Arn"="ARN"
Advertisements

Published by momo

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. ( Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s