AWS DynamoDB to Redshift using Kinesis Firehose

Demonstrate how to execute an ETL operation from AWS DynamoDB stream to Amazon Redshift using Amazon Kinesis Firehose and AWS Lambda. This guide will cover creating, configuring, and connecting the necessary components to perform the ETL.

---

Steps:

1. Create a DynamoDB Table

   • Go to the DynamoDB service in AWS.

   • Create a new table with the desired primary key.

2. Create a Redshift Cluster

   • Go to the Redshift service in AWS.

   • Launch a new cluster and configure it as needed.

   • Ensure the cluster is publicly accessible.

3. Create a Table in Redshift

   • Connect to the Redshift cluster using a SQL client or the Redshift Query Editor.

   • Create a table with the desired columns using SQL CREATE TABLE statement.

4. Create an IAM Role for Lambda

   • Go to the IAM service in AWS.

   • Create a new role for Lambda and attach the following policies:

     • AmazonDynamoDBFullAccess

     • AmazonKinesisFirehoseFullAccess

     • AWSLambda_FullAccess

     • AWSLambdaBasicExecutionRole

   • Increase the timeout in the Lambda function configuration to handle long-running processes.

5. Enable DynamoDB Streams

   • In the DynamoDB table, go to Export and streams.

   • Enable DynamoDB stream details and choose New images.

6. Create a Lambda Function

   • Go to the Lambda service in AWS.

   • Create a new Lambda function and attach the IAM role created.

   • Increase the timeout setting in the configuration.

   

   https://github.com/Cuchuflim/DynamoDB-to-Redshift/tree/main

7. Create a Trigger for the Lambda Function

   • In the DynamoDB table, go to Triggers.

   • Create a new trigger and select the Lambda function created.

8. Create an S3 Bucket

   • Go to the S3 service in AWS.

   • Create a new bucket to be used as an intermediate storage for the ETL process.

9. Create a Kinesis Firehose Stream

   • Go to the Kinesis service in AWS.

   • Create a new Firehose delivery stream.

   • For the source, choose Direct PUT or other sources.

   • For the destination, select Amazon Redshift.

   • Configure the cluster, authentication, database, and table details.

   • For the intermediate S3 destination, select the bucket created.

   • In the COPY command, specify CSV as the data format.

10. Review VPC Connections

    • Ensure that the VPC settings are correct.

    • Check that the inbound rules of the security group allow the necessary traffic.

    • Confirm that Redshift is publicly accessible if needed.

11. Test the ETL Process

    • Create an item in the DynamoDB table to trigger the process.

    • Monitor the execution of the Lambda function in CloudWatch to ensure everything is working correctly.

Final Notes:

• Ensure that all AWS services and configurations are within the same region.

• Use CloudWatch for monitoring and troubleshooting any issues that arise during the ETL process.