Home/ Questions/Q 7737805
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-01T08:05:58+00:00

I have a daemon that does the following retrieves site members from a mysql

  • 0

I have a daemon that does the following

  • retrieves site members from a mysql database (I used LIMIT 1000 to retrieve 1000 rows at a time)
  • send information about these members to a third party server
  • flag each member as having been processed
  • Sleep for 2 seconds
  • Retrieve the next batch of 1000 “unprocessed” members and send to third party server.
    and so on.

I am wondering whether a php daemon (I am using the system Daemon library), is the best way to accomplish this task delineated above.

I am worried of wasting too much memory (as PHP is known for that)

I am also worried about sending multiple requests to third party server, because on a high traffic day, there can be a lot of nonreceipts.

Is there a tool other than daemon I can use to accomplish this task? What methods can I implement to make this efficient considering there is a possibility of having to process over 100K rows in the mysql table, and the task is time sensitive. Also, at what point should I consider adding more servers?

Thanks!

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    A cron should be a very good option for doing a sync job with a third party server.
    Consider the following ‘improvments’:

    1) A lock file to prevent multiple jobs from starting in parallel and taking extra resources from other processes you have running. And also to avoid duplicate processing of data.

    2) If you don’t have already implement an ‘information update’ and ‘sync time’ check on your side. For example if user A hasn’t suffered any changes since he was sync you don’t sync him again.

    3) Consider how often you need data to be sync and if it doesn’t have to be real time factor that into the selection query. Combined with user/time distribution and other factors you migth end up having periods of time when your script doesn’t sync that many accounts.

    4) Do your own memory cleanup unsetting variables, unlinking files and even reusing the same variables so you don’t have garbage variables that are a 1 time use only inside the scripts. Carefull with this as it might lead to obfuscating the code.

    Also consider using smaller datasets when you send them to php for processing. Databases love big datasets, php doesn’t.

    • 0