Data Ingestion Methods with: Amazon Kinesis Data Firehose, Lambda functions, Amazon DynamoDB, Amazon Glue, Amazon Athena

Written by Reginald Bratton

Reginald Bratton INC has a new client in the real estate business. Not only are they in the business of selling properties they are also brokerage firm that can complete your closing when you purchase. They are full service! With the explosion of the low interest rates there is an abundance of buyers. With the low interest rates there are more buyers and with more buyers there is more data surrounding the sale of the property and its creating a data explosion of pre-approvals and recent comparable sales then when you purchase! They are going to need management of all this data, storage of the data(using S3 and Amazon DynamoDB) and cost optimization of our resources in the cloud. With all of the data they are going to need something to query the data as well so we can use Amazon Athena and to help with organization of the data we are going to use Amazon Glue.

We have our work cut out for us today!

We are going to use several AWS resources to get this party started!

real-time data? we have a real time solution

EC2 would be good but Amazon Kinesis Analytics would be even better, Kinesis will transform and analyze streaming data in real time!

More growth=more data=more data stored in S3 buckets=more data to be analyzed with Amazon Athena in our DynamoDB table.

Also Kinesis will scale automatically to match the volume (client expects exponential growth) coming and matches the volume and throughout of your incoming data.

First we are going to create an Amazon Kinesis Data Firehose delivery system:

AWS Kinesis data FireHose easily captures, transforms, and loads streaming data. Create a delivery stream, select your destination, and start streaming real-time data with just a few clicks. To make our client happy they are looking to maintain a cost effective and cost friendly business solution. Lets start with heading over to:

scroll down and save after selecting S3 for your desination

Next up we are going to ingest and store clickstream data in an Amazon S3 bucket. In short we need somewhere to store our data…why not our S3 bucket! With the durability of our S3 bucket this makes the best place to store object storage within our cloud. Our last step we step up how the data gets their through Kinesis so this is our next step

Transform using a Lambda function

Final destination of the clickestream data results will go to the S3 bucket that you choose here

Data is headed to this S3 bucket.

Now we need to create a real-time query by using AWS Glue and Amazon Athena.

AWS Glue is a serverless, scalable data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning, and application development. In our case lets revisit our clients request that they are so many streams of data coming in they need to help managing them.

AWS will integrate the data that is coming in(clickstream) in this case and with AWS Athena you can query the data there in our S3 bucket. Athena provides a simplified, flexible way to analyze petabytes of that data where it lives. Analyze data or build applications from an Amazon Simple Storage Service (S3).

Start with:

Pretty cool feature about AWS Glue Data Catalog that its a central repo to store structural and operational metadata for all your data assets. If you have a dataset and you store its table definition and physical location, add business-relevant attributes and track how the data has changed over time. For example you can see how customers buying patterns have changed over time so now you can market them a new service or product.

We have to name and create our database first

After adding your table definitions to the Data Catalog the tables are available for extract, transform and load (ETL) also they are available for querying in Amazon Athena, EMR (managed cluster platform that lets you move data in and out of data stores like a S3 bucket) or RedShift (Large Data-Warehouse).

For our DIY Section we have a couple things to accomplish. We are going to change our buffer hints to 60 seconds. The default value is 300 seconds. We are going to basically shorten the timeout period from 5 minutes to 1 minute.

Next up we are going to send the data that we previously stored in a S3 bucket we are sending that data now to an Amazon DynamoDB table.

DynamoDB is a fast, fully-managed NoSQL database service that makes it simple and cost effective to store and retrieve any amount of data, and serve any level of request traffic. DynamoDB helps offload the administrative burden of operating and scaling a highly-available distributed database cluster. This storage alternative meets the latency and throughput requirements of highly demanding applications by providing single-digit millisecond latency and predictable performance with seamless throughput and storage scalability. This helps our clients with the cost effectiveness and the best part is that you do not need to hire another person to admin the table=more cost savings.

Savings+efficienctly=satisfied client!

First step:

To edit the Kinesis Data Firehose delivery stream configuration to send real time analytics to Amazon DynamoDB table: go back to Kinesis

scroll down

press edit

Press Browse

Use AnalyticsdestinationFunction and press save.

Head back to save to your S3 bucket to edit your buffer hints.

We need to change the buffer interval to 60 seconds from 300 seconds (default) now press save(not shown). Head back over to the DynamoDB table we created now enter in OutputTable and Voila!

That would complete our mission today!

Thanks for following along!

Follow me: Reginald Bratton