Too Long; Didn’t Read
Cron runs every 10 minutes, looks for new stuff to do, and sends it to the queue. Workers take tasks from the queue and process them. Save everything to the DB. This is a very naive solution that was running on multiple EC2 (AWS) and Redis.