DynamoDB Logs Ingestion

DynamoDB Streams are a technology built into DynamoDB that lets you subscribe to changes made to your DynamoDB table. This technology is a great way to add messages to an SQS queue when a change is made to your DynamoDB table. You can then have a processor on the SQS to further handle the DynamoDB change (e.g., send an email, clear a cache, etc).

DynamoDB Streams are the primary way we will work with SQS in this guide. However, we will also use the AWS SDK directly to add/modify/delete data from our DynamoDB table based on SQS messages.

Prerequisites

• A DynamoDB and AWS S3 bucket.

Export AWS DynamoDB data to S3 on a recurring basis using Lambda

In this article, we will be walking you through the setup to automate the export process by using lambda as an orchestrator to invoke the API and EventBridge as a trigger to determine the frequency of export.

Enable Point-In-Time Recovery (PITR) for your DynamoDB.

To enable PITR,

• Go to DynamoDB → Select the table → Backup section → Edit PITR → Enable PITR

Create an Amazon S3 Bucket

First, you need to create an Amazon S3 bucket where you will store your objects.

1. Sign into the preview version of the AWS Management Console.
2. Under Storage & Content Delivery, choose S3 to open the Amazon S3 console.
   If you are using the Show All Services view, your screen looks like this:

If you are using the Show Categories view, your screen looks like this with Storage & Content Delivery expanded:

3. From the Amazon S3 console dashboard, choose Create Bucket.

4. In Create a Bucket, type a bucket name in Bucket Name.
The bucket name you choose must be globally unique across all existing bucket names in Amazon S3 (that is, across all AWS customers).

5. In Region, choose Oregon.

6. Choose Create.
When Amazon S3 successfully creates your bucket, the console displays your empty bucket in the Buckets pane.

Create a Lambda Function

We will be creating the lambda function (Nodejs 16.x) to invoke the AWS SDK API. In this example, we will be creating a new IAM Role with the default permission required for lambda.

Copy the following code and paste it in the lambda function and replace the bucket_name and table_arn

1. The code will extract today date and format it in this format (yyyy/mm/dd/).
   Each s3 data will be stored prefix with the formatted date
2. The code invokes the dynamoDB api to trigger the export exportTableToPointInTime

3. For the code change to take effect, we will have to redeploy the lambda function.

4. Lastly, for the code to work we will need to update the IAM Role that we created earlier with the necessary permission to allow PutObject to S3 and ExportTableToPointInTime from DynamoDB.

5. Go to IAM → Roles → Search for the role name (export-to-s3-role) → Click the Edit → Add the JSON snippet below into the existing role → Click the Review Policy → Save Changes.

6. Edit the Policy to include the permission below

{
    "Sid": "DynamoDBPermission",
    "Effect": "Allow",
    "Action": [
      "dynamodb:ExportTableToPointInTime"
    ],
    "Resource": "*"
},
{
    "Sid": "S3Permission",
    "Effect": "Allow",
    "Action": [
      "s3:PutObject"
    ],
    "Resource": "*"
}

Now our lambda function is ready, all we need is a trigger to invoke our lambda function.

Create an EventBridge to trigger the Lambda

• Go to Amazon Eventbridge → Rules → Create Rule → Configure schedule → Select the lambda function created earlier to invoke.

Conclusion

We will be able to see our lambda function being updated with eventbridge as a trigger. Let’s verify our implementation to see if the DynamoDB export is being triggered and the data is being uploaded to S3 successfully. The export process after the lambda trigger the api will take up to 30 minutes depending on the size of your DB.

You can see the object in the following directory yyyy/mm/dd/AWSDynamoDB/{ExportID}/data/. The object is zip in gz format you can download it and unzip it — gunzip {filename} — you will see the data in JSON format.

Configure a feed in Chronicle 

Complete the following steps to configure a feed in Chronicle to ingest the DynamoDB logs:

1. Go to Chronicle settings and click Feeds.    
2. Click Add New.    

3. Select Amazon S3 for Source Type.

4. Select AWS DynamoDB for Log Type.