Open in app

Sign in

Write

Sign in

AWS S3 Bucket Notification to SQS queue via SNS topic Walk Trough

Windix Feng
4 min readDec 2, 2018

--

Here is a working example via aws-cli and jq.

Preparation

Create S3 bucket:

aws s3 mb s3://wei20181202s3

make_bucket: wei20181202s3

Create SNS topic:

aws sns create-topic --name wei20181202sns

{
    "TopicArn": "arn:aws:sns:ap-southeast-2:946527283516:wei20181202sns"
}

Create SQS Queue:

aws sqs create-queue --queue-name wei20181202sqs

{
    "QueueUrl": "https://ap-southeast-2.queue.amazonaws.com/946527283516/wei20181202sqs"
}

Find out SQS ARN:

aws sqs get-queue-attributes \
--queue-url "https://ap-southeast-2.queue.amazonaws.com/946527283516/wei20181202sqs" \
--attribute-names All

{
    "Attributes": {
        "QueueArn": "arn:aws:sqs:ap-southeast-2:946527283516:wei20181202sqs",
        "ApproximateNumberOfMessages": "0",
        "ApproximateNumberOfMessagesNotVisible": "0",
        "ApproximateNumberOfMessagesDelayed": "0",
        "CreatedTimestamp": "1543744958",
        "LastModifiedTimestamp": "1543744958",
        "VisibilityTimeout": "30",
        "MaximumMessageSize": "262144",
        "MessageRetentionPeriod": "345600",
        "DelaySeconds": "0",
        "ReceiveMessageWaitTimeSeconds": "0"
    }
}

Now we have the following information:

S3:

  • bucket name: s3://wei20181202s3
  • ARN: arn:aws:s3:::wei20181202s3

SNS:

  • topic name: wei20181202sns
  • ARN: arn:aws:sns:ap-southeast-2:946527283516:wei20181202sns

SQS:

Hook up SQS to SNS

Fist, I want to subscribe SQS to SNS:

Permission: Allow SNS to send message to SQS queue

cat <<EOT > sqs-permission.json
{
  "Version": "2012-10-17",
  "Id": "allow-sns-sending-message-to-sqs",
  "Statement": [
    {
      "Sid": "wei20181202-sqs-sid",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "SQS:SendMessage",
      "Resource": "arn:aws:sqs:ap-southeast-2:946527283516:wei20181202sqs",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:sns:ap-southeast-2:946527283516:wei20181202sns"
        }
      }
    }
  ]
}
EOT

Generate the JSON file required by “aws sqs set-queue-attributes” and attach the permission to the SQS queue:

jq -c '. | { Policy: @text }' sqs-permission.json > sqs.json

aws sqs set-queue-attributes \
--queue-url "https://ap-southeast-2.queue.amazonaws.com/946527283516/wei20181202sqs" \
--attributes file://sqs.json

Subscribe SQS to SNS topic:

aws sns subscribe \
--topic-arn "arn:aws:sns:ap-southeast-2:946527283516:wei20181202sns" \
--protocol "sqs" \
--notification-endpoint "arn:aws:sqs:ap-southeast-2:946527283516:wei20181202sqs"

{
    "SubscriptionArn": "arn:aws:sns:ap-southeast-2:946527283516:wei20181202sns:d635c372-b15d-4e9c-87cc-e4b137e220ea"
}

Hook up SNS to S3

Next, monitor any new file in s3 /upload directory and send the event to SNS topic

Permission: Allow S3 to publish to SNS

cat <<EOT > sns-permission.json
{
  "Version": "2012-10-17",
  "Id": "allow-s3-publishing-to-sns",
  "Statement": [
    {
      "Sid": "wei20181202-s3-sid",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "SNS:Publish",
      "Resource": "arn:aws:sns:ap-southeast-2:946527283516:wei20181202sns",
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:s3:::wei20181202s3"
        }
      }
    }
  ]
}
EOT

aws sns set-topic-attributes \
--topic-arn "arn:aws:sns:ap-southeast-2:946527283516:wei20181202sns" \
--attribute-name Policy \
--attribute-value file://sns-permission.json

Add a notification to SNS for S3 on creating new object under /upload

cat <<EOT > s3-notification.json 
{
  "TopicConfigurations": [
    {
      "Id": "s3-upload-notification-to-sns",
      "TopicArn": "arn:aws:sns:ap-southeast-2:946527283516:wei20181202sns",
      "Events": [
        "s3:ObjectCreated:*"
      ],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "upload/"
            }
          ]
        }
      }
    }
  ]
}
EOT

aws s3api put-bucket-notification-configuration \
--bucket wei20181202s3 \
--notification-configuration file://s3-notification.json

All done! Now there should be one S3 test event message (Event: s3:TestEvent) in SQS.

aws sqs receive-message \
--queue-url "https://ap-southeast-2.queue.amazonaws.com/946527283516/wei20181202sqs" \
--max-number-of-messages 10

{
    "Messages": [
        {
            "MessageId": "fea5b0d2-2272-4569-9e0d-9c2d9867ed8a",
            "ReceiptHandle": "AQEBsNMFpoeEHbX/8XEZC+5nqebhyv49m46wMarrPLaVVZjRr/LRKk1MvJFfHNBQCMBaLwK381ewoy+Dyno7hHbB6HfLdQSizXExqo8b73PNiqSEw7wEGNSTYRL6zLDYOoaHGvH5hIh8hSH+r449ne1bC2HYCht31ks/YCeV+F8Rt5WLximoDLfFYZVhUm++8mFEPLEZ0ynHuOWbnEvJ11XnjTNN35ZutIcK4Izq688PEeJEWCxV1T+GON6m4AavfcwWdsZfWRbKqjznfifD9eJbRDd3XWq11fOswO6UoKgsPZJ6SKtRtzvzpudrmxhsIQ8idVgg+XCwibj6zF3U8lWi3kqlNwgAN+du6ASEaOsq6fWQHJzYUjdVzXcr5Esf1o1NId6pxtQvQeDAIYSoK3YTFQ==",
            "MD5OfBody": "f26e95c87ce020e197d871ef190ba1a1",
            "Body": "{\n  \"Type\" : \"Notification\",\n  \"MessageId\" : \"c74af7f1-896e-50d4-8ac6-a38c3e585853\",\n  \"TopicArn\" : \"arn:aws:sns:ap-southeast-2:946527283516:wei20181202sns\",\n  \"Subject\" : \"Amazon S3 Notification\",\n  \"Message\" : \"{\\\"Service\\\":\\\"Amazon S3\\\",\\\"Event\\\":\\\"s3:TestEvent\\\",\\\"Time\\\":\\\"2018-12-02T10:33:26.548Z\\\",\\\"Bucket\\\":\\\"wei20181202s3\\\",\\\"RequestId\\\":\\\"E2373D643CA9BB30\\\",\\\"HostId\\\":\\\"9CIlxds7BfgHfK8Fl8bOcXa5M51t/kJkU0k1bTE9LFNvihcAIytER/mQc0HAkzrgGsRWXQttVeM=\\\"}\",\n  \"Timestamp\" : \"2018-12-02T10:33:26.629Z\",\n  \"SignatureVersion\" : \"1\",\n  \"Signature\" : \"szJAthp5BQLT9BiZ+o1b4WZM4qZ+YicK4N7G/gytPUEM7uLPpksdiEHB95GM4RngGqR8mCptq7/8eodu+6kYHR35SVSmZSO1aSlIg5lG4gJj6qE2ZwV6inaKMoy7WcHwqKavZZORJt90zh0FOg2cE3xViDh/Q5JSriBRumcP8IcIFDg2H247skmiAXP1vzdtZmPKGJteim5RgQhGtb5k/m0O8o/PZ8Iw0wyUEaniLW4KYW7P/u3FnCq9A37+So423/RaSZiKNq76B7yLDlMAuDmwHH0nB++GuT4fuvaq0UWyuHx0EoMTtvxAi3oAq3DX+S1Eln4XrHac4BUImwQD+A==\",\n  \"SigningCertURL\" : \"https://sns.ap-southeast-2.amazonaws.com/SimpleNotificationService-ac565b8b1a6c5d002d285f9598aa1d9b.pem\",\n  \"UnsubscribeURL\" : \"https://sns.ap-southeast-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:ap-southeast-2:946527283516:wei20181202sns:d635c372-b15d-4e9c-87cc-e4b137e220ea\"\n}"
        }
    ]
}

Remove this message from queue:

aws sqs delete-message \
--queue-url "https://ap-southeast-2.queue.amazonaws.com/946527283516/wei20181202sqs" \
--receipt-handle "AQEBsNMFpoeEHbX/8XEZC+5nqebhyv49m46wMarrPLaVVZjRr/LRKk1MvJFfHNBQCMBaLwK381ewoy+Dyno7hHbB6HfLdQSizXExqo8b73PNiqSEw7wEGNSTYRL6zLDYOoaHGvH5hIh8hSH+r449ne1bC2HYCht31ks/YCeV+F8Rt5WLximoDLfFYZVhUm++8mFEPLEZ0ynHuOWbnEvJ11XnjTNN35ZutIcK4Izq688PEeJEWCxV1T+GON6m4AavfcwWdsZfWRbKqjznfifD9eJbRDd3XWq11fOswO6UoKgsPZJ6SKtRtzvzpudrmxhsIQ8idVgg+XCwibj6zF3U8lWi3kqlNwgAN+du6ASEaOsq6fWQHJzYUjdVzXcr5Esf1o1NId6pxtQvQeDAIYSoK3YTFQ=="

Test Notification

Now let’s test an upload notification:

cat <<EOT > hello.txt
Hello world!
EOT

aws s3 cp hello.txt s3://wei20181202s3/upload/

upload: ./hello.txt to s3://wei20181202s3/upload/hello.txtaws sqs receive-message \
--queue-url "https://ap-southeast-2.queue.amazonaws.com/946527283516/wei20181202sqs" \
--max-number-of-messages 10

{
    "Messages": [
        {
            "MessageId": "9eff3912-74bd-4dc3-a2b3-8b48e4e81556",
            "ReceiptHandle": "AQEB4hp8sJvjtGwBeEHF5lc6p4R905dccWPcckNOev9HV0vuWg8JT7IoYZszzgox94q+L4BnKXq1c0HkmmVX/oCVVa4AlTkS6Tu8+B0jjqkzzvIiZ/rXCyKroB2pJam90Pr/QcbJ5SmQP9IGWqf9Nm19Uz3iRnenUxJ56LkrPS8ZoZpb1zdPZ9Btq8gTGEzYSnER2yFWWC5PjEog6Kuh59C3c70JTpTFNinrS5XRnFJOJK8gRypGAMrFHo5a42Qe7UHaO8PjPS0pNR3+jDTdD9+Z2XyGBi4JnPXrAkiYMVXGS3TcKQCYt1w9b4WCrqdKgbLNV/kKhrPWzRmcISv4xGBS8X0Zjqu/Fc/FjRUjdnjiW1SWhdciqc4v1sak1Wq+Eg2lzoYemWuDMneb4qNLH0/WjQ==",
            "MD5OfBody": "5d4b91a5eacd5ced69e47e50718794d7",
            "Body": "{\n  \"Type\" : \"Notification\",\n  \"MessageId\" : \"a7c5c19a-4530-5c4f-913d-95e7fc3d0ae7\",\n  \"TopicArn\" : \"arn:aws:sns:ap-southeast-2:946527283516:wei20181202sns\",\n  \"Subject\" : \"Amazon S3 Notification\",\n  \"Message\" : \"{\\\"Records\\\":[{\\\"eventVersion\\\":\\\"2.1\\\",\\\"eventSource\\\":\\\"aws:s3\\\",\\\"awsRegion\\\":\\\"ap-southeast-2\\\",\\\"eventTime\\\":\\\"2018-12-02T10:41:16.967Z\\\",\\\"eventName\\\":\\\"ObjectCreated:Put\\\",\\\"userIdentity\\\":{\\\"principalId\\\":\\\"AWS:AIDAJVAUQJEA4B5QICDUW\\\"},\\\"requestParameters\\\":{\\\"sourceIPAddress\\\":\\\"1.2.3.4\\\"},\\\"responseElements\\\":{\\\"x-amz-request-id\\\":\\\"38DEBC62CAE3461F\\\",\\\"x-amz-id-2\\\":\\\"QT30m0nlCeWw/tk/9d2suSHqls6MpaVmczksOIvfF3Frx6Wn9H6rj58pMC0lSNh9RHLt3FkNSdE=\\\"},\\\"s3\\\":{\\\"s3SchemaVersion\\\":\\\"1.0\\\",\\\"configurationId\\\":\\\"wei-20181202-s3\\\",\\\"bucket\\\":{\\\"name\\\":\\\"wei20181202s3\\\",\\\"ownerIdentity\\\":{\\\"principalId\\\":\\\"A3UQGCHDU2X4TL\\\"},\\\"arn\\\":\\\"arn:aws:s3:::wei20181202s3\\\"},\\\"object\\\":{\\\"key\\\":\\\"upload/hello.txt\\\",\\\"size\\\":13,\\\"eTag\\\":\\\"59ca0efa9f5633cb0371bbc0355478d8\\\",\\\"sequencer\\\":\\\"005C03B6CCDF8ED9CF\\\"}}}]}\",\n  \"Timestamp\" : \"2018-12-02T10:41:17.027Z\",\n  \"SignatureVersion\" : \"1\",\n  \"Signature\" : \"J4/2o65SOh20LYMpi6vfpITLzOKSAYfkq099fptNXr7/9vvOPTuOe1kTESSGod80Yr6RZN3HwxNIeEpkw9cYYxOQxIlbZVRgQw9N6QoxCWFryvMspK4RkO8zyhTIPym+yYtEeXCUasZnNeIiHwxPpUlfJLHadgrjpKRq7ML3yfTGPKneyd04xHV8hyLz1sDs1RYDMgMRSOZan7fC09+8IkN58qS5iOmZJdpYg9AwWsnlcXSEWnZ1PD8URlMUFbZJDsL/N5asBuNz32orlpOu+851BmMFbyN3vd62vMaQ/xzkNufJhockdOeW68n52jzW/9xC+skO93rfhWpc/uCWYw==\",\n  \"SigningCertURL\" : \"https://sns.ap-southeast-2.amazonaws.com/SimpleNotificationService-ac565b8b1a6c5d002d285f9598aa1d9b.pem\",\n  \"UnsubscribeURL\" : \"https://sns.ap-southeast-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:ap-southeast-2:946527283516:wei20181202sns:d635c372-b15d-4e9c-87cc-e4b137e220ea\"\n}"
        }
    ]
}

You can see the S3 event and SNS message has been wrapped in the SQS message.

A closer look to the SNS message (the result from previous query has been save to result.json):

jq '.Messages[0].Body | fromjson' result.json

{
  "Type": "Notification",
  "MessageId": "a7c5c19a-4530-5c4f-913d-95e7fc3d0ae7",
  "TopicArn": "arn:aws:sns:ap-southeast-2:946527283516:wei20181202sns",
  "Subject": "Amazon S3 Notification",
  "Message": "{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"ap-southeast-2\",\"eventTime\":\"2018-12-02T10:41:16.967Z\",\"eventName\":\"ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"AWS:AIDAJVAUQJEA4B5QICDUW\"},\"requestParameters\":{\"sourceIPAddress\":\"1.2.3.4\"},\"responseElements\":{\"x-amz-request-id\":\"38DEBC62CAE3461F\",\"x-amz-id-2\":\"QT30m0nlCeWw/tk/9d2suSHqls6MpaVmczksOIvfF3Frx6Wn9H6rj58pMC0lSNh9RHLt3FkNSdE=\"},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"wei-20181202-s3\",\"bucket\":{\"name\":\"wei20181202s3\",\"ownerIdentity\":{\"principalId\":\"A3UQGCHDU2X4TL\"},\"arn\":\"arn:aws:s3:::wei20181202s3\"},\"object\":{\"key\":\"upload/hello.txt\",\"size\":13,\"eTag\":\"59ca0efa9f5633cb0371bbc0355478d8\",\"sequencer\":\"005C03B6CCDF8ED9CF\"}}}]}",
  "Timestamp": "2018-12-02T10:41:17.027Z",
  "SignatureVersion": "1",
  "Signature": "J4/2o65SOh20LYMpi6vfpITLzOKSAYfkq099fptNXr7/9vvOPTuOe1kTESSGod80Yr6RZN3HwxNIeEpkw9cYYxOQxIlbZVRgQw9N6QoxCWFryvMspK4RkO8zyhTIPym+yYtEeXCUasZnNeIiHwxPpUlfJLHadgrjpKRq7ML3yfTGPKneyd04xHV8hyLz1sDs1RYDMgMRSOZan7fC09+8IkN58qS5iOmZJdpYg9AwWsnlcXSEWnZ1PD8URlMUFbZJDsL/N5asBuNz32orlpOu+851BmMFbyN3vd62vMaQ/xzkNufJhockdOeW68n52jzW/9xC+skO93rfhWpc/uCWYw==",
  "SigningCertURL": "https://sns.ap-southeast-2.amazonaws.com/SimpleNotificationService-ac565b8b1a6c5d002d285f9598aa1d9b.pem",
  "UnsubscribeURL": "https://sns.ap-southeast-2.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:ap-southeast-2:946527283516:wei20181202sns:d635c372-b15d-4e9c-87cc-e4b137e220ea"
}

An even closer look to the original S3 event:

jq '.Messages[0].Body | fromjson | .Message | fromjson' result.json

{
  "Records": [
    {
      "eventVersion": "2.1",
      "eventSource": "aws:s3",
      "awsRegion": "ap-southeast-2",
      "eventTime": "2018-12-02T10:41:16.967Z",
      "eventName": "ObjectCreated:Put",
      "userIdentity": {
        "principalId": "AWS:AIDAJVAUQJEA4B5QICDUW"
      },
      "requestParameters": {
        "sourceIPAddress": "1.2.3.4"
      },
      "responseElements": {
        "x-amz-request-id": "38DEBC62CAE3461F",
        "x-amz-id-2": "QT30m0nlCeWw/tk/9d2suSHqls6MpaVmczksOIvfF3Frx6Wn9H6rj58pMC0lSNh9RHLt3FkNSdE="
      },
      "s3": {
        "s3SchemaVersion": "1.0",
        "configurationId": "wei-20181202-s3",
        "bucket": {
          "name": "wei20181202s3",
          "ownerIdentity": {
            "principalId": "A3UQGCHDU2X4TL"
          },
          "arn": "arn:aws:s3:::wei20181202s3"
        },
        "object": {
          "key": "upload/hello.txt",
          "size": 13,
          "eTag": "59ca0efa9f5633cb0371bbc0355478d8",
          "sequencer": "005C03B6CCDF8ED9CF"
        }
      }
    }
  ]
}

Clean up

aws sqs delete-queue --queue-url "https://ap-southeast-2.queue.amazonaws.com/946527283516/wei20181202sqs"
aws sns delete-topic --topic-arn "arn:aws:sns:ap-southeast-2:946527283516:wei20181202sns"
aws s3 rb s3://wei20181202s3 --force

Notes

It is amazing to see localstack is actually able to mock the entire flow above! (Don’t forget to use awslocal if you want to try).

Reference

https://alestic.com/2014/12/s3-bucket-notification-to-sqssns-on-object-creation/

AWS
Aws Cli
S3
Sns
Sqs

--

--

Windix Feng
Windix Feng

Written by Windix Feng

18 Followers
5 Following

Responses (1)

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams