Home/ Questions/Q 6686037
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-26T05:07:33+00:00

I’ve looked all over for any answer on this and haven’t been able to

  • 0

I’ve looked all over for any answer on this and haven’t been able to find one, hopefully someone can point me in the right direction, I think I’m close

I have two hosts let’s call them host1.mydomain.com and host2.mydomain.com (to get around the 2 concurrent connections per host/per browser issue), so they both point to the same content one is just an alias of the other

User goes to host1.mydomain.com, enters some information to register, clicks Go, which loads an iframe on the same page pointing to a page on host2.mydomain.com which calls a php script via exec(“curl”) sending the request to the background to start a website scraper, the process ID is then stored in the database for the user. After the iframe has successfully loaded (only takes 1 second since it’s creating a background process) I have an AJAX request set on an interval to check the status periodically of the cURL process (by it’s process ID in the database) so that I can display the current step of the scraper (there are 6 steps in total). All good so far.

The problem is that the AJAX requests are timing out after step 4 of the scraper (browser default timeout is 115/120 seconds) even though it shouldn’t be because I’m working with two different hosts…meaning to say that it’s almost as if I’m clogging both connections on host1.mydomain.com when I’m not because I initiated the scraper from host2

The iframe loads this URL: http://host2.mydomain.com/page.php
The contents of the PHP script calls:

exec("curl -o /dev/null 'http://host2.mydomain.com/page.php?method=process' > /dev/null & echo $!", $op);

Then my ajax request is polling http://host1.mydomain.com/status.php?pid=x which looks up in the database to check the status by the process ID

and once the scraper gets to step 4, my ajax requests are timing out

I think I confused myself explaining this, but hopefully someone can help me

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Turns out I was successfully getting around the 2 connections per server/browser limitation…however in doing some research I found the reason why my ajax request was hanging is because I was trying to access and write to the session data from both of the requests. Digging a little deeper I found a session_write_close() which closes the session for reading/writing, I basically have to call this after each page request of the scraper and then reinitialize the session, this allows my ajax requests to go through and stops the blocking of the request.

    Hopefully someone else finds this useful if you stumble across the same issue

    Cheers!
    Jeff

    • 0