# Speed of Pushing SQS Messages In March 2022, I had a task to send nearly 70 million messages to an AWS SQS queue. Each message was simply a number that I needed to read from a CSV file. ## Local processing My initial thought was to send the messages, one-at-a-time, from my local computer to the SQS queue. Each message was simply the number from a row in the CSV file. Testing the speed of sending the messages from my local computer showed that it would take months to publish all 70 million messages. Initially my local Python program was sending about 20 messages per second. I tried making the local Python program into a multi-threaded program. There was improvement all the way up to 80 messages per second. However, that would still take months of time to publish the messages. ## AWS processing Given the ease of spinning up EC2 instances on AWS, I thought to try running the same multi-threaded program from an EC2 instance. It should be faster to send messages from inside AWS systems versus from my local computer. And, being cheap, I tried out a t2.micro first. AWS indicates that this instance type is for "low to moderate" networking. Sure enough, we jumped up to about 473 messages per second in a multi-threaded application. Next, it seemed like it might be helpful to choose a different EC2 instance type so to get improved networking. I experimented with different instance types and found that a c5.large, which is described at "up to 10 gigabit", provided the best performance. ## Results ``` On local MacBook Pro computer: - 80 msgs/sec on my computer. On t2.micro (Low to Moderate): - 473 msgs/sec using 1,000 threads. - 483 msgs/sec using 5,000 threads. On t3.micro (Up to 5 Gigabit): - 574 msgs/sec using 1,000 threads. - 576 msgs/sec using 5,000 threads. On c5.large (Up to 10 Gigabit): - 697 msgs/sec using 1,000 threads. - 704 msgs/sec using 5,000 threads. On c5d.18xlarge (25 Gigabit): - 687 msgs/sec using 1,000 threads. - 697 msgs/sec using 5,000 threads. On c5n.18xlarge (100 Gigabit): - 689 msgs/sec using 1,000 threads. - 689 msgs/sec using 5,000 threads. ``` --- Created: 2022-12-09 08:39