Amazon S3 logs via CloudWatch

This article covers how to create an Amazon S3 bucket and export AWS logs using the CloudWatch console to Chronicle. 

In the following example, you use the Amazon CloudWatch console to export all data from an Amazon CloudWatch Logs log group named my-log-group to an Amazon S3 bucket named my-exported-logs.

Exporting log data to Amazon S3 buckets that are encrypted by AWS KMS is not supported.

Step 1: Create an Amazon S3 bucket

We recommend that you use a bucket that was created specifically for CloudWatch Logs. However, if you want to use an existing bucket, you can skip to step 2.

Note

The Amazon S3 bucket must reside in the same Region as the log data to export. CloudWatch Logs doesn't support exporting data to Amazon S3 buckets in a different Region.

To create an Amazon S3 bucket

1. Open the Amazon S3 console at https://console.aws.amazon.com/s3/.
2. If necessary, change the Region. From the navigation bar, choose the Region where your CloudWatch Logs reside.
3. Choose Create Bucket.
4. For Bucket Name, enter a name for the bucket.
5. For Region, select the Region where your CloudWatch Logs data resides.
6. Choose Create.

Step 2: Create an IAM user with full access to Amazon S3 and CloudWatch Logs

In the following steps, you create the IAM user with necessary permissions.

To create the necessary IAM user

1. Open the IAM console at https://console.aws.amazon.com/iam/.
2. Choose Users, Add user.
3. Enter a user name, such as CWLExportUser.
4. Select both Programmatic access and AWS Management Console access.
5. Choose either Autogenerated password or Custom password.
6. Choose Next: Permissions.
7. Choose Attach existing policies directly, and attach the AmazonS3FullAccess and CloudWatchLogsFullAccess policies to the user. You can use the search box to find the policies.

8. Choose Next: Tags, Next: Review, and then Create user.

Step 3: Set permissions on an Amazon S3 bucket

By default, all Amazon S3 buckets and objects are private. Only the resource owner, the AWS account that created the bucket, can access the bucket and any objects that it contains. However, the resource owner can choose to grant access permissions to other resources and users by writing an access policy.

When you set the policy, we recommend that you include a randomly generated string as the prefix for the bucket, so that only intended log streams are exported to the bucket.

To set permissions on an Amazon S3 bucket

1. In the Amazon S3 console, choose the bucket that you created in step 1.
2. Choose Permissions, Bucket policy.
3. In the Bucket Policy Editor, add one of the following policies. Change my-exported-logs to the name of your S3 bucket and random-string to a randomly generated string of characters. Be sure to specify the correct Region endpoint for Principal.
   • If the bucket is in your account, add the following policy.
     

     {             
                         "Version": "2012-10-17",
                         "Statement": [
                           {
                               "Action": "s3:GetBucketAcl",
                               "Effect": "Allow",
                               "Resource": "arn:aws:s3:::my-exported-logs",
                               "Principal": { "Service": "logs.us-west-2.amazonaws.com" }
                           },
                           {
                               "Action": "s3:PutObject" ,
                               "Effect": "Allow",
                               "Resource": "arn:aws:s3:::my-exported-logs/random-string/*",
                               "Condition": { "StringEquals": { "s3:x-amz-acl": "bucket-owner-full-control" } },
                               "Principal": { "Service": "logs.us-west-2.amazonaws.com" }
     
                           }
     
                         ]
     
     }

4. Choose Save to set the policy that you just added as the access policy on your bucket. This policy enables CloudWatch Logs to export log data to your Amazon S3 bucket. The bucket owner has full permissions on all of the exported objects.

Warning

If the existing bucket already has one or more policies attached to it, add the statements for CloudWatch Logs access to that policy or policies. We recommend that you evaluate the resulting set of permissions to be sure that they're appropriate for the users who will access the bucket.

Step 4: Create an export task

In this step, you create the export task for exporting logs from a log group.

To export data to Amazon S3 using the CloudWatch console

1. Sign in as the IAM user that you created in Step 2: Create an IAM user with full access to Amazon S3 and CloudWatch Logs.
2. Open the CloudWatch console at https://console.aws.amazon.com/cloudwatch/.
3. In the navigation pane, choose Log groups.
4. On the Log Groups screen, choose the name of the log group.
5. Choose Actions, Export data to Amazon S3.
6. On the Export data to Amazon S3 screen, under Define data export, set the time range for the data to export using From and To.
7. If your log group has multiple log streams, you can provide a log stream prefix to limit the log group data to a specific stream. Choose Advanced, and then for Stream prefix, enter the log stream prefix.
8. Under Choose S3 bucket, choose the account associated with the Amazon S3 bucket.
9. For S3 bucket name, choose an Amazon S3 bucket.
10. For S3 Bucket prefix, enter the randomly generated string that you specified in the bucket policy.
11. Choose Export to export your log data to Amazon S3.
12. To view the status of the log data that you exported to Amazon S3, choose Actions and then View all exports to Amazon S3.

Step 5: Configure Feed in Chronicle to Ingest AWS Logs

1. Go to Chronicle settings, and click Feeds.

2. Click Add New.

3. Select Amazon S3 for Source Type.

4. Select AWS CloudWatch (or other AWS service) for Log Type.

5. Click Next.

6. Select region and provide S3 URI of the Amazon S3 bucket you copied earlier. Further, you could append the S3 URI with:

 {{datetime("yyyy/MM/dd")}}

7. As in the following example, so that Chronicle would scan logs each time only for a particular day:

 s3://aws-cloudtrail-logs-XXX-1234567/AWSLogs/1234567890/CloudTrail/us-east-1/{{datetime("yyyy/MM/dd")}}/

8. Under URI IS A select Directories including subdirectories. Select an appropriate option under Source Deletion Option, this should match with the permissions of the IAM User account you created earlier.

9. Provide Access Key ID and Secret Access Key of the IAM User account you created earlier.

10. Click Next and Finish.