Cloudformation chalange #1 - conditional resources dependency

Backstory

This is based on true story of one of my clients. The requirements were simple:

  1. Send notification each time:
    • File is uploaded to S3 bucket
    • File is deleted from S3 bucket
    • Event is caught by CloudWatch Rule
  2. Notifications must be send to provided SQS Topic and/or Lambda function (on the same or different AWS Account)
  3. If no ARN is provided, notifications resources shouldn't be created.
  4. Solution must be generic and easy to automate

Design

SNS is perfect service for this scenario. It is push notification service and it supports multiple targets.
Of course it doesn't count without diagram, so I drew one:

Cloudformation template:

Solution should be generic and reusable, so I decided to prepare CloudFormation template (I'm a big fan of CloudFormation) with all required logic to fulfill the requirements.

Version 1

First version was easy, define all required resources and put parameters everywhere in resource names.
You can check v1.template in repository
AWSTemplateFormatVersion: 2010-09-09
Description: |
  SNS Notifications for S3 events and CloudWatch Rules
Parameters:
  stage:
    Default: dev
    Type: String
    Description: Stage Name
  
  bucketName:
    Default: data
    Type: String
    Description: Consumable bucket name prefix
  
  projectName:
    Default: s3-notification-example
    Type: String
    Description: Project Name SSM Parameter

Resources:
  s3Bucket:
    Type:  AWS::S3::Bucket
    Properties:
      BucketName: !Sub ${projectName}-${bucketName}-${stage}
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      NotificationConfiguration:
        TopicConfigurations:
          -  
            Event: s3:ObjectCreated:*
            Filter:
              S3Key: 
                Rules:
                  - Name: suffix
                    Value: .json
            Topic: !Ref notificationTopic
          -  
            Event: s3:ObjectRemoved:*
            Filter:
              S3Key: 
                Rules:
                  - Name: suffix
                    Value: .json
            Topic: !Ref notificationTopic
      Tags: 
        - Key: Project
          Value: !Ref projectName
        - Key: Stage
          Value: !Ref stage

  notificationTopic:
    Type: AWS::SNS::Topic
    Properties: 
      DisplayName: !Sub ${projectName}-notifications-${stage}
      Tags: 
        - Key: Project
          Value: !Ref projectName
        - Key: Stage
          Value: !Ref stage
      TopicName: !Sub ${projectName}-notifications-${stage}
  
  notificationTopicPolicy:
    Type: AWS::SNS::TopicPolicy
    Properties: 
      PolicyDocument:
        Version: '2012-10-17'
        Id: Policy
        Statement:
        - Sid: Default
          Effect: Allow
          Principal:
            AWS: "*"
          Action:
            - SNS:GetTopicAttributes
            - SNS:SetTopicAttributes
            - SNS:AddPermission
            - SNS:RemovePermission
            - SNS:DeleteTopic
            - SNS:Subscribe
            - SNS:ListSubscriptionsByTopic
            - SNS:Publish
            - SNS:Receive
          Resource: !Ref notificationTopic
          Condition:
            StringEquals:
              AWS:SourceOwner: !Ref AWS::AccountId
        - Sid: CloudWatchEvents
          Effect: Allow
          Principal:
            Service: events.amazonaws.com
          Action: sns:Publish
          Resource: !Ref notificationTopic
        - Sid: S3
          Effect: Allow
          Principal:
            AWS: "*"
          Action:
            - sns:Publish
          Resource: !Ref notificationTopic
          Condition:
            ArnLike: 
              aws:SourceArn: !GetAtt s3Bucket.Arn
            StringEquals:
              aws:SourceAccount: !Ref AWS::AccountId
      Topics: 
        - !Ref notificationTopic

When stack is creating, cloudformation tries to detect dependencies between defined resources.
In v1.template three resources are defined:
  1. s3Bucket with notifications configured to 
  2. notificationTopic with
  3. notificationTopicPolicy which allows S3 bucket to Publish messages to notificationTopic
Problem:
In order to set S3 events notifications, SNS topic policy must be created before S3 bucket, otherwise deployment fails with following error:

Unable to validate the following destination configurations (Service: Amazon S3; Status Code: 400; Error Code: InvalidArgument


Cloudformation detected dependency between s3Bucket and notificationTopic but not notificationTopicPolicy. Without notificationTopicPolicy which allows S3 to publish to notificationTopic, S3 events can't be set.

Solution:
Use "DependsOn" attribute, documentation says. This attribute helps cloudformation to understand resources dependencies. Sounds simple, so lets try with v2.

Version 2

Lets add "DependsOn" to s3Bucket definition:
  s3Bucket:
    Type:  AWS::S3::Bucket
    DependsOn:
      - notificationTopicPolicy
    Properties:
    (...)
Problem:
Circular Dependencies for resource s3Bucket. Circular dependency with [notificationTopicPolicy]

There is reference to s3Bucket in notificationTopicPolicy definition, and "DependsOn" attribute forces cloudformation to create notificationTopicPolicy before S3 bucket.

 Solution:
Solution is simple, just replace "!GetAtt s3Bucket.Arn" with generated ARN of S3 bucket. You can do it cause SNS doesn't check if S3 bucket already exists.

  notificationTopicPolicy:
    (...)
          Condition:
            ArnLike: 
              aws:SourceArn: !Sub arn:aws:s3:*:*:${projectName}-${bucketName}-${stage}
    (...)

Problem solved. Resources are created in right order, S3 events for bucket are enabled...
but topic and topic policy are created each time even if it's not required.

Version 3 - conditions

We have to provide subscriber ARN and this can be done with template parameters. We also can use conditions to determine if subscribers ARNs are provided.
Parameters:
  (...)
  sqsNotificationsArns:
    Default: None,None
    Type: CommaDelimitedList
    Description: Comma delimited list of SQS ARNs to subscribe to SNS Topic (up to 2)

  lambdaNotificationsArns:
    Default: None,None
    Type: CommaDelimitedList
    Description: Comma delimited list of SQS ARNs to subscribe to SNS Topic (up to 2)

Conditions:
  sqs1NotificationsArnsProvided:
    !Not [!Equals [ !Select [ "0", !Ref sqsNotificationsArns], "None"]] 
  sqs2NotificationsArnsProvided:
    !Not [!Equals [ !Select [ "1", !Ref sqsNotificationsArns], "None"]]
  
  lambda1NotificationsArnsProvided:
    !Not [!Equals [ !Select [ "0", !Ref lambdaNotificationsArns], "None"]]
  lambda2NotificationsArnsProvided:
    !Not [!Equals [ !Select [ "1", !Ref lambdaNotificationsArns], "None"]]
  
  enableNotifications:
    !Or 
      - Condition: sqs1NotificationsArnsProvided 
      - Condition: sqs2NotificationsArnsProvided
      - Condition: lambda1NotificationsArnsProvided
      - Condition: lambda2NotificationsArnsProvided
How does it work?
If no sqsNotificationsArns or lambdaNotificationsArns are provided, the defaults are used ("None","None"). Conditions check if any element of sqsNotificationsArns or lambdaNotificationsArns is different then default value (none). If it's different, then condition returns "True", otherwise "False"

Now we can provide up to two SQS ARNs and up to two Lambda ARNs to template and use "enableNotifications" condition to deploy (or not) notificationTopic and notificationTopicPolicy.
Parameters:
  notificationTopic:
    Type: AWS::SNS::Topic
    Condition: enableNotifications
    Properties: 
    (...)
  
  notificationTopicPolicy:
    Type: AWS::SNS::TopicPolicy
    Condition: enableNotifications
    (...)
Resources:
  s3Bucket:
    Type:  AWS::S3::Bucket
    DependsOn:
      - notificationTopicPolicy
    Properties:
      NotificationConfiguration:
        !If
          - enableNotifications
          - TopicConfigurations:
              -  
                Event: s3:ObjectCreated:*
                Filter:
                  S3Key: 
                    Rules:
                      - Name: suffix
                        Value: .json
                Topic: !Ref notificationTopic
              -  
                Event: s3:ObjectRemoved:*
                Filter:
                  S3Key: 
                    Rules:
                      - Name: suffix
                        Value: .json
                Topic: !Ref notificationTopic
          - !Ref AWS::NoValue

Problem:
When sqsNotificationsArns or lambdaNotificationsArns are provided conditions work as expected. notificationTopicPolicy, notificationTopic and s3Bucket are created in right order, but
when no subscribers ARNs are provided, we encounter following error:
 
An error occurred (ValidationError) when calling the CreateStack operation: Template format error: Unresolved resource dependencies [notificationTopic, notificationTopicPolicy] in the Resources block of the template

Solution:
There is no simple solution. Cloudformation doesn't support conditions in "DependsOn" attribute. We need to bond s3Bucket with notificationTopicPolicy but with enableNotifications condition. 

Version 4 - final

I've spent a while figuring out working solution, and finally did it in v4.template.
Quick look at AWS::SNS::TopicPolicy documentation and no return values, but I tried to use "!Ref notificationTopicPolicy" in  s3Bucket Policy TAG. This should provide required dependency between notificationTopicPolicy and s3Bucket and conditions can be used in TAG value.
In fact Policy returns some kind of ID (i.e. notifications-notificationTopicPolicy-ANKFGG0LEMPM)

So lets remove "DependsOn" from s3Bucket definition
    DependsOn:
      - notificationTopicPolicy
and add TAG with value related to notificationTopicPolicy or just "No Policy" if subscribers ARNs wasn't provided.
    Tags: 
        - Key: Project
          Value: !Ref projectName
        - Key: Stage
          Value: !Ref stage
        - Key: Policy
          Value: 
            !If
              - enableNotifications
              - !Ref notificationTopicPolicy
              - 'No Policy'

This way all requirements are fulfilled. If subscribers ARNs are provided, notificationTopic and notificationTopicPolicy are created and S3 events are enabled. If no, nothing related to notifications is created.
Also stack updates are supported. So you can create S3 bucket first and add/remove notifications later.
Of course AWS::SNS::Subscription is missing but I'm sure you can handle it

Final template:
AWSTemplateFormatVersion: 2010-09-09
Description: |
  SNS Notifications for S3 events and CloudWatch Rules
Parameters:
  stage:
    Default: dev
    Type: String
    Description: Stage Name
  
  bucketName:
    Default: data
    Type: String
    Description: Consumable bucket name prefix
  
  projectName:
    Default: s3-notification-example
    Type: String
    Description: Project Name SSM Parameter
  
  sqsNotificationsArns:
    Default: None,None
    Type: CommaDelimitedList
    Description: Comma delimited list of SQS ARNs to subscribe to SNS Topic (up to 2)

  lambdaNotificationsArns:
    Default: None,None
    Type: CommaDelimitedList
    Description: Comma delimited list of SQS ARNs to subscribe to SNS Topic (up to 2)

Conditions:
  sqs1NotificationsArnsProvided:
    !Not [!Equals [ !Select [ "0", !Ref sqsNotificationsArns], "None"]] 
  sqs2NotificationsArnsProvided:
    !Not [!Equals [ !Select [ "1", !Ref sqsNotificationsArns], "None"]]
  
  lambda1NotificationsArnsProvided:
    !Not [!Equals [ !Select [ "0", !Ref lambdaNotificationsArns], "None"]]
  lambda2NotificationsArnsProvided:
    !Not [!Equals [ !Select [ "1", !Ref lambdaNotificationsArns], "None"]]
  
  enableNotifications:
    !Or 
      - Condition: sqs1NotificationsArnsProvided 
      - Condition: sqs2NotificationsArnsProvided
      - Condition: lambda1NotificationsArnsProvided
      - Condition: lambda2NotificationsArnsProvided

Resources:
  s3Bucket:
    Type:  AWS::S3::Bucket
    Properties:
      BucketName: !Sub ${projectName}-${bucketName}-${stage}
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      NotificationConfiguration:
        !If
          - enableNotifications
          - TopicConfigurations:
              -  
                Event: s3:ObjectCreated:*
                Filter:
                  S3Key: 
                    Rules:
                      - Name: suffix
                        Value: .json
                Topic: !Ref notificationTopic
              -  
                Event: s3:ObjectRemoved:*
                Filter:
                  S3Key: 
                    Rules:
                      - Name: suffix
                        Value: .json
                Topic: !Ref notificationTopic
          - !Ref AWS::NoValue
      Tags: 
        - Key: Project
          Value: !Ref projectName
        - Key: Stage
          Value: !Ref stage
        - Key: Policy
          Value: 
            !If
              - enableNotifications
              - !Ref notificationTopicPolicy
              - 'No Policy'

  notificationTopic:
    Type: AWS::SNS::Topic
    Condition: enableNotifications
    Properties: 
      DisplayName: !Sub ${projectName}-notifications-${stage}
      Tags: 
        - Key: Project
          Value: !Ref projectName
        - Key: Stage
          Value: !Ref stage
      TopicName: !Sub ${projectName}-notifications-${stage}
  
  notificationTopicPolicy:
    Type: AWS::SNS::TopicPolicy
    Condition: enableNotifications
    Properties: 
      PolicyDocument:
        Version: '2012-10-17'
        Id: Policy
        Statement:
        - Sid: Default
          Effect: Allow
          Principal:
            AWS: "*"
          Action:
            - SNS:GetTopicAttributes
            - SNS:SetTopicAttributes
            - SNS:AddPermission
            - SNS:RemovePermission
            - SNS:DeleteTopic
            - SNS:Subscribe
            - SNS:ListSubscriptionsByTopic
            - SNS:Publish
            - SNS:Receive
          Resource: !Ref notificationTopic
          Condition:
            StringEquals:
              AWS:SourceOwner: !Ref AWS::AccountId
        - Sid: CloudWatchEvents
          Effect: Allow
          Principal:
            Service: events.amazonaws.com
          Action: sns:Publish
          Resource: !Ref notificationTopic
        - Sid: S3
          Effect: Allow
          Principal:
            AWS: "*"
          Action:
            - sns:Publish
          Resource: !Ref notificationTopic
          Condition:
            ArnLike: 
              aws:SourceArn: !Sub arn:aws:s3:*:*:${projectName}-${bucketName}-${stage}
            StringEquals:
              aws:SourceAccount: !Ref AWS::AccountId
      Topics: 
        - !Ref notificationTopic

Example deployment with update:

Komentarze

Prześlij komentarz

Popularne posty z tego bloga

Lambda trigger on file upload to specified prefix

Obraz
  Backstory Another story based on assigned task. There are h undreds of source systems publishing files to S3 bucket. Each one to it's own prefix. Objects keys are prepared for Athena query optimization - data is partitioned. SNS topic is set as target for  S3 Event notification for " data/ " prefix. Third party system requires notification each time file is uploaded by any source system. The task is to invoke Lambda function whenever file is uploaded by one particular source system, let's call it SystemA. Other source systems shouldn't invoke any additional actions. S3 data structure looks like this: Idea 1 - not good not terrible As  S3 Event notification for every file uploaded to "data/" prefix is already set, I can subscribe Lambda function to SNS topic and verify object Key each time. Sounds  good and is simple to implement, but is not efficient. On 100 uploaded files, only 1 is published by SystemA and invoking Lambda for other 99 is just waste o...
Czytaj więcej