Home/ Questions/Q 132507
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-11T06:16:25+00:00

Basically I need to get around max execution time. I need to scrape pages

  • 0

Basically I need to get around max execution time.

I need to scrape pages for info at varying intervals, which means calling the bot at those intervals, to load a link form the database and scrap the page the link points to.

The problem is, loading the bot. If I load it with javascript (like an Ajax call) the browser will throw up an error saying that the page is taking too long to respond yadda yadda yadda, plus I will have to keep the page open.

If I do it from within PHP I could probably extend the execution time to however long is needed but then if it does throw an error I don’t have the access to kill the process, and nothing is displayed in the browser until the PHP execute is completed right?

I was wondering if anyone had any tricks to get around this? The scraper executing by itself at various intervals without me needing to watch it the whole time.

Cheers 🙂

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. "nothing is displayed in the browser until the PHP execute is completed"

    You can use flush() to work around this:

    flush()

    (PHP 4, PHP 5)

    Flushes the output buffers of PHP and whatever backend PHP is using (CGI, a web server, etc). This effectively tries to push all the output so far to the user’s browser.

    • 0