Fan one message out from an SNS topic to two SQS queues

short · 30 min · Task statement 3.8

Before you start — what this lab needs, and what it can cost

This lab runs in your own AWS account. Work in AWS CloudShell, signed in as a non-root administrative identity: an IAM Identity Center user, or an IAM user with administrator permissions. Never use the root user for this lab. It uses the AWS Free Tier where possible, but charges are possible: SNS charges for publishes and deliveries and SQS for requests, at rates on their pricing pages. With a handful of messages the amount is tiny, and the budget alert you set in lesson 1's lab stays on and emails you if anything starts to cost money.

The ARNs and queue URLs below name US East (N. Virginia), us-east-1. Open CloudShell in that Region, or replace us-east-1 throughout. Replace 111122223333 with your own account ID everywhere it appears. The order in the message is invented.

This lab ends with a clean-up. Do it the same day: anything left running in your account is billed to you.

Task

Build the fan-out pattern from the lesson: publish one order message, once, to an Amazon SNS topic, and see a copy land in each of two Amazon SQS queues, one for billing and one for shipping. At the end you can show both copies, and you can explain from what you saw why SNS pushes each message to every subscriber at once while SQS holds each message until a consumer collects it and deletes it.

Steps

Create the SNS topic that the order system will publish to:

aws sns create-topic --name clf-lab21-orders

It prints the topic's ARN, arn:aws:sns:us-east-1:111122223333:clf-lab21-orders. 2. An SQS queue only accepts messages from SNS if its access policy allows it. Write a policy that lets this one topic, and nothing else, send to the lab's queues, then wrap it in the attributes file that create-queue reads (jq is already installed in CloudShell):

echo '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"sns.amazonaws.com"},"Action":"sqs:SendMessage","Resource":"arn:aws:sqs:us-east-1:111122223333:clf-lab21-*","Condition":{"ArnEquals":{"aws:SourceArn":"arn:aws:sns:us-east-1:111122223333:clf-lab21-orders"}}}]}' > clf-lab21-policy.json
jq -n --arg p "$(cat clf-lab21-policy.json)" '{Policy: $p}' > clf-lab21-attrs.json

The principal is the SNS service, and the condition pins it to your topic. 3. Create the two queues, one per system, each with that policy:

aws sqs create-queue --queue-name clf-lab21-billing --attributes file://clf-lab21-attrs.json
aws sqs create-queue --queue-name clf-lab21-shipping --attributes file://clf-lab21-attrs.json

Each prints its queue URL, such as https://sqs.us-east-1.amazonaws.com/111122223333/clf-lab21-billing. 4. Subscribe both queues to the topic. Raw message delivery passes the message through exactly as published, which makes it easier to read:

aws sns subscribe --topic-arn arn:aws:sns:us-east-1:111122223333:clf-lab21-orders --protocol sqs --notification-endpoint arn:aws:sqs:us-east-1:111122223333:clf-lab21-billing --attributes RawMessageDelivery=true
aws sns subscribe --topic-arn arn:aws:sns:us-east-1:111122223333:clf-lab21-orders --protocol sqs --notification-endpoint arn:aws:sqs:us-east-1:111122223333:clf-lab21-shipping --attributes RawMessageDelivery=true

A queue in your own account is confirmed straight away; an email subscriber would have to click a link first. 5. Publish one message, once, to the topic:

aws sns publish --topic-arn arn:aws:sns:us-east-1:111122223333:clf-lab21-orders --message 'Order 1001: one kettle'

The publisher knows nothing about billing or shipping. It only knows the topic. 6. Collect the message from each queue, as each system's worker would:

aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/111122223333/clf-lab21-billing --wait-time-seconds 5 --query 'Messages[0].Body' --output text
aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/111122223333/clf-lab21-shipping --wait-time-seconds 5 --query 'Messages[0].Body' --output text

Both print Order 1001: one kettle: one publish, two copies. Run the first command again at once and it prints None, because a received message is hidden from other consumers for the queue's visibility timeout (30 seconds by default). After that it comes back, because nobody deleted it. That is the difference from SNS, which pushed the message on and kept nothing.

Verify

Both queues are subscribed to the topic:

aws sns list-subscriptions-by-topic --topic-arn arn:aws:sns:us-east-1:111122223333:clf-lab21-orders --query 'length(Subscriptions)'

Expected output: 2.

Wait at least 30 seconds after step 6, then read the shipping queue again. The message is still held there, because no consumer deleted it:

aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/111122223333/clf-lab21-shipping --wait-time-seconds 5 --query 'Messages[0].Body' --output text

Expected output: Order 1001: one kettle.

Cost and clean-up

SNS and SQS charge per request, and the AWS Free Tier covers a monthly number of them where your account is eligible. Nothing here charges by the hour, but an idle queue still exists until you delete it, so delete everything below now.

Created by this lab: the SNS topic clf-lab21-orders and its two subscriptions; the SQS queues clf-lab21-billing and clf-lab21-shipping; the CloudShell files clf-lab21-policy.json and clf-lab21-attrs.json.

Delete the topic. Deleting a topic also deletes its subscriptions:

aws sns delete-topic --topic-arn arn:aws:sns:us-east-1:111122223333:clf-lab21-orders

Delete both queues, with any messages still in them:

aws sqs delete-queue --queue-url https://sqs.us-east-1.amazonaws.com/111122223333/clf-lab21-billing
aws sqs delete-queue --queue-url https://sqs.us-east-1.amazonaws.com/111122223333/clf-lab21-shipping

Remove the local files:

rm -f clf-lab21-policy.json clf-lab21-attrs.json

Check it is gone

A deleted queue can take up to 60 seconds to leave the list, so wait a minute first.

aws sns list-topics --query "length(Topics[?ends_with(TopicArn, ':clf-lab21-orders')])"

Expected output: 0.

aws sns list-subscriptions --query "length(Subscriptions[?contains(TopicArn, 'clf-lab21')])"

Expected output: 0.

aws sqs list-queues --queue-name-prefix clf-lab21 --query 'length(not_null(QueueUrls, `[]`))'

Expected output: 0.

Notes

Check yourself against the lesson's choices, using what you built. A CloudWatch alarm that must text and email the whole operations team at once: SNS on its own. Orders that arrive faster than one worker can process them: one SQS queue. Each of several systems needs its own copy: SNS plus SQS, as here. Run a job every night, or react to an AWS service's events by their content: Amazon EventBridge. Send designed order-confirmation emails to customers: Amazon SES, not SNS.

This is an independent study companion for the AWS Certified Cloud Practitioner (CLF-C02) exam and is not produced by or endorsed by Amazon Web Services. AWS, Amazon Web Services and AWS Certified Cloud Practitioner are trademarks of Amazon.com, Inc. or its affiliates.

This is an independent study companion for AWS Certified Cloud Practitioner CLF-C02 and is not produced by or endorsed by CompTIA.