Integration between S3 Events - Lambda Function - Batch Process

Overview

This project utilizes events generated by Amazon S3 during object creation to process them using the Batch service. As there is no direct connection between these two services, a lambda function is employed as middleware to capture the S3 event and initiate the batch processing. The architecture is depicted in AWS.

Architecture Overview

The project leverages the event-driven architecture to process objects uploaded to an S3 bucket. The workflow is as follows:

1. When an object is created in the S3 bucket, an event is triggered.
2. The event is captured by a Lambda function acting as middleware.
3. The Lambda function receives the event and initializes the batch processing job.
4. The batch processing job is executed by the Batch service.
5. Upon completion, the results of the batch processing job are stored or forwarded to the desired destination.

By utilizing this architecture, the project ensures efficient and scalable processing of objects in an automated manner, enabling seamless integration between S3 and Batch services.

AWS Services Used

1. Amazon S3: Stores the objects and triggers events upon object creation.
2. AWS Lambda: Serves as middleware, capturing the S3 event and initializing the batch processing job.
3. AWS Batch: Executes the batch processing job based on the event received from Lambda.