Posty

Wyświetlanie postów z sierpień, 2020

Cloudformation chalange #1 - conditional resources dependency

Obraz
Backstory This is based on true story of one of my clients. The requirements were simple: Send notification each time: File is uploaded to S3 bucket File is deleted from S3 bucket Event is caught by CloudWatch Rule Notifications must be send to provided SQS Topic and/or Lambda function (on the same or different AWS Account) If no ARN is provided, notifications resources shouldn't be created. Solution must be generic and easy to automate Design SNS is perfect service for this scenario. It is push notification service and it supports multiple targets. Of course it doesn't count without diagram, so I drew one: Cloudformation template: Solution should be generic and reusable, so I decided to prepare CloudFormation template (I'm a big fan of CloudFormation) with all required logic to fulfill the requirements. Version 1 First version was easy, define all required resources and put parameters everywhere in resource names. You can check v1.template...
Prześlij komentarz
Czytaj więcej

Popularne posty z tego bloga

Cloudformation chalange #1 - conditional resources dependency

Obraz
Backstory This is based on true story of one of my clients. The requirements were simple: Send notification each time: File is uploaded to S3 bucket File is deleted from S3 bucket Event is caught by CloudWatch Rule Notifications must be send to provided SQS Topic and/or Lambda function (on the same or different AWS Account) If no ARN is provided, notifications resources shouldn't be created. Solution must be generic and easy to automate Design SNS is perfect service for this scenario. It is push notification service and it supports multiple targets. Of course it doesn't count without diagram, so I drew one: Cloudformation template: Solution should be generic and reusable, so I decided to prepare CloudFormation template (I'm a big fan of CloudFormation) with all required logic to fulfill the requirements. Version 1 First version was easy, define all required resources and put parameters everywhere in resource names. You can check v1.template...
Czytaj więcej

Lambda trigger on file upload to specified prefix

Obraz
  Backstory Another story based on assigned task. There are h undreds of source systems publishing files to S3 bucket. Each one to it's own prefix. Objects keys are prepared for Athena query optimization - data is partitioned. SNS topic is set as target for  S3 Event notification for " data/ " prefix. Third party system requires notification each time file is uploaded by any source system. The task is to invoke Lambda function whenever file is uploaded by one particular source system, let's call it SystemA. Other source systems shouldn't invoke any additional actions. S3 data structure looks like this: Idea 1 - not good not terrible As  S3 Event notification for every file uploaded to "data/" prefix is already set, I can subscribe Lambda function to SNS topic and verify object Key each time. Sounds  good and is simple to implement, but is not efficient. On 100 uploaded files, only 1 is published by SystemA and invoking Lambda for other 99 is just waste o...
Czytaj więcej