I’m making a currency system (not a biggie, just a simple one) where users will be able to convert currencies to another, this is not a big project just a little project with few people. However, I’m using PHP to scrape Google currency calculator which scrapes once a day and stores it into memcached, when the item expires in memcached, it will re-scrape to get the updated currency values.
When this problem arises when scraping Google (currently gets 24 currencies and get 24 currencies for each currency so the script scrapes 48 different pages, when this is running I often get 504 Gateway timeout since the script runs for about 200 seconds.
Since the error is thrown, I’m not sure if you can temporarily increase the nginx timeout via PHP when the script is scraping, then revert back once done.
I was thinking about doing it via cron job, would this solve the problem better instead of doing it when the user visits the site?
You should use a cron job… anything that could cause a timeout on the browser should be avoided at all cost.
Just my 2 cents..