Member-only story
Auto-scaling AWS ECS Fargate Task using SQS and Lambda
Understanding the use case is crucial in this article because there may be several different designs and solutions. It relies on your current infrastructure and business logic whether you can implement one of them.
So without any introduction, let’s understand the use case right off.
Use case
- Suppose we want to run compute on a set of events that do not need to be processed as and when they arrive.
- Compute can take more than 15 minutes on average per event.
- Triggering of the events in this article depends on your use case and your current architecture.
Based on this knowledge, a simple solution could be to trigger a lambda for each event.
However, this has some flaws.
First, this is near real-time computing and can take more than 15 minutes (max lambda timeout reached) on average per event which does not satisfy our use case. Second, lambdas are by default throttled if the number of events generated is significantly more than 1000. Third, cost, which can be extremely high depending on the amount of processing power needed and the volume of events produced.
Current solution
We can then instead use an ECS cluster to run our computing on. However, there are a few questions that need to be addressed first.
- How do we run our compute on ECS -> Service or Tasks?
- How do they process incoming events?
The answer to #1 lies in our use case again. As there is no need to have the events processed just after they arrive, there is no requirement to have at least one of our compute engines running. So unlike ECS Service, we can opt for ECS Tasks.
To answer #2, there are two steps. We need to add an SQS resource to our architecture where all events will go in. Then, the code that will be run inside the ECS Task should loop over available events in this SQS and process them iteratively.
