I’ve been doing some research and I’m trying to figure out the best method to run one of our sites. In short, there will be about 100 subdomains and each subdomain has 21 cron scripts that need to be run. I have shortened it down to 11 by using some fopen functions where it can be used. Each subdomain uses the same APIs more less to extract data and store it in the local mysql DB. All subdomains have their own DB and it is not shared and information will never overlap between these databases as they are all unique pieces of data.
We intend to host it on VPS hosting, which I’m not sure would be the best idea, but it should do the job from what we are anticipating. Intervals for these 11 scripts are one of the following:
5 min, 8 min, 10 min, 15 min,30 min, 45 min, 1 hour, 3 hours, 8 hours, 12 hours, and 24 hours
The ones that run on under 10 minutes intervals are generally very quick to load (under 20 seconds/script)
My question is, what is the best way/best practices to opening this many scripts without…
a.) crashing the server
b.) slowing the server to something ridiculous
c.) Not timing out our cron scripts.
My thoughts for a method:
1) Create 11 cron scripts in total and run fopen functions with possible a sleep() function inbetween each script. The problem is there may be 100 files that need to be opened per script.
2) Create 1100 cron jobs to run at set times.
3) Create maybe about 110 fopen cron scripts that will run 10 scripts per file.
If anyone can help out with tips or tricks, that would be greatly appreciated!!
Also, for server usage, dedicated or VPS be necessary?
If I understand you correctly, you would probably want to setup the 11 cron scripts, and each of those cron scripts would call the other needed scripts in sequence. You probably would not need to sleep() inbetween because they would run in sequence, not in parallel. To reduce server load you can also make the scripts run in low priority by using the “nice” command.
Dedicated or VPS is completely dependent on how much load you are putting on the server. Performance of a VPS can also vary heavily from provider to provider. It really cannot be gauged from the information given. If you are trying to save money, then I would say just try it on the VPS. You can always move to a more expensive dedicated server later.