CertiPass LogoCertiPass

Building a Data Streaming Pipeline with Amazon Kinesis and AWS Lambda

INTERMEDIATE
100 minutes
5 tasks

In this lab, you will build a data streaming pipeline using Amazon Kinesis Data Streams and Amazon Kinesis Data Firehose. You'll configure AWS Lambda to process streaming data and persist it into Amazon S3 for further analysis. This lab will help you understand streaming data architectures and the integration of these services to automate and facilitate data processing pipelines.

Scenario

A new e-commerce company, FastCommerce, wants to implement a real-time recommendation system that processes user interaction data on its platform. The company needs a scalable solution to ingest, process, and store data with minimal latency. They require a system that leverages serverless architectures to maximize cost efficiency and scalability.

Learning Objectives

  • Understand how to set up and configure Amazon Kinesis Data Streams and Data Firehose.
  • Learn to process streaming data with AWS Lambda functions.
  • Store processed data in Amazon S3 for future analysis.
  • Review serverless architectures for scalability and cost-efficiency.

tasks (5)

task 1: Create a Kinesis Data Stream

15 min

task 2: Set up an AWS Lambda function to process data

25 min

task 3: Configure Kinesis Data Firehose to delivery to S3

20 min

task 4: Verify and monitor data flow to S3

20 min

task 5: Optimize and review security settings

20 min

Prerequisites

  • •Familiarity with AWS management console and basic navigation.
  • •Basic understanding of data streaming concepts.
  • •Know how to create and manage IAM roles and policies.
  • •Understanding of serverless architecture principles.

Skills Tested

Configuring and managing Kinesis Data Streams and Data Firehose.Creating and deploying AWS Lambda functions for data processing.Utilizing S3 for data persistence and ensuring security through encryption and IAM policies.Monitoring data flows with CloudWatch.

References

    Building a Data Streaming Pipeline with Amazon Kinesis and AWS Lambda - Hands-On Lab - CertiPass