Was wondering what the best way to handle an expensive web task?
I have an email scheduler that sends a large number of emails at any given hour. Currently the system is getting a list of emails, constructing them, and deploying them. However, this is starting to get very expensive as the list grows.. Is there a way where I can treat this like a producer-consumer multithreaded paradigm?
I was thinking I could gather all the emails and curl each one the second part of the web task — constructing and deploying the email by passing the address with the curl. However, I do believe that the system must wait for the curl’s response before continuing? So this isn’t asynchronous..
What would be the best way to do this? right now I have a cronjob checking hourly for emails to be delivered, that’s fine, but the emails are all constructed and delivered at once, so it can be slow and possibly overlap into the next hour slot — users have a specified time to retrieve so i’d like it for it to be delivered on time.
Should I create a C++ program to handle these with threads? Or can I get away with an easier method.. thanks!
Keep in mind that just by creating a second thread, you don’t magically get more cycles out of your box. If your box is being taxed, the best you can do is force your email “thread” to a lower priority. It will run in the background with degraded performance. Also, moving to C++ may or may not really solve anything. Is the bottleneck the code or is it just the fact that something is blocking trying to send an email?
Ultimately if this is a beefy, time-consuming task, you might want to look at offloading all the email processing to a separate box. Your code is already putting the emails into a database. Have a box dedicated to emailing the world check for emails on the db and send them when needed.
I wasn’t sure what you meant here. But do you know whether curl’ing up to somewhere would be any faster for that server then just emailing? Might it be slower?