Home/ Questions/Q 6799861
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-26T18:53:57+00:00

I built an app in php where a feature analyzes about 10000 text files

  • 0

I built an app in php where a feature analyzes about 10000 text files and extracts stuff from them and puts it into a mysql database. The code itself is just a for loop where every file is loaded through file_get_contents() and after the end of that iteration, its unset() from memory. The file analysis is a cron job and a single php file does all this processing.

The problem however is that the app was built (initially) entirely on a shared server everything worked seamlessly really well. I didn’t notice any delays or major lags neither did users however in order for it to be able to handle more of a load, I moved everything to an EC2 server (the micro instance).

The problem I am having now is that every time I run the cronjob (process the files on hourly basis) it slows the entire server down so much that a normal page takes about 5-8 seconds to load, which sort of defeats the purpose of moving it to EC2.

The cron itself is a very long process. Here are some tests results of the script process (every hr)

SQL Insertion Time: 23.138303995132 seconds
Memory Used: 10.05 MB
Execution: 411.00507092476 seconds

But on the top of every hour the server slows down so much for 7 minutes despite of having more dedicated hardware acceleration compared to a shared server (I think at least). The graphs from EC2 dashboard show that the CPU usage is close to 100% but I don’t understand how it gets to that level.

Can anyone help me determine the reason as to why this could be happening? I have noticed not even the slightest lag when the cron runs on the shared server but the case is completely different for EC2.

Please feel free to ask me anything I missed mentioning.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Micro instances are pretty slow. If you use a larger instance, it’ll run a lot faster.

    We use EC2 for all of our production boxes. I can’t say enough good things about that platform. I’ll never go back to another host.

    Also, if you want to write your code in C++, it’ll run A LOT faster. I wrote a simple mysql insert with this code here. It’s multi-threaded, so you can asyncronously run mysql updates or inserts.

    Please let me know if you need any help with it, but I’m sure you’ll be able to just use a micro instance still and get great speeds.

    Hope that helps…

    PS. I’d be willing to help you write a C++ version for your uses… just because it’s fun! 🙂

    • 0