Question

I have a php file with around 100mb which is full of arrays (only arrays).

I’ve made a script that includes this file (for processing), first it exhausted the default Xampp 128mb memory limit, i’ve raised it to 1024mb but it just takes forever and doesn’t do anything.

I’m sure the problem is being created by the sheer size of the file because i’ve tried removing all lines of code and just leaving the include and an echo for me to know when it finishes executing, and it does the same thing (which is taking forever), i’ve also tried to run the 100mb file in separate and same thing again.

A 10mb file is taking forever as well but a similar 1mb file is almost instantly read and executed so the problem must be more than just the file size.

I was avoiding using c++ for a simple project as this and would rather not to as php is easier for me and the task that will be executed doesn’t need to benefit from the added speed that it would have if it had been done in c++ but if i have no luck in solving this problem i guess i’ll have to.

---

EDIT

Reasons for not using a database:
1-Whoever made it didn’t used a database and it will be pretty hard to store this in an organized database if i’m not able to do something with it first, like just reading it, copying parts from it or putting in memory or something.

2-I don’t have experience working with databases as pretty much all stuff i’ve ever done in php didn’t needed large amounts of stored data, 50kb at best, if i was thinking about a big project or huge chunks of data as this one, i definitely would, but i didn’t made this mess to start with and now i have to undo it.

3-The logic for having to store a small portion of data like 10mb in hard drive when now every computer has pretty much enough ram to fit the whole OS in it is pretty much incomprehensible unless someone gives a good explanation about it, if i had to access a lot of said files simultaneously i would understand but like i said, this is a simple project, this is the only file that will be accessed at a given time this isn’t even to make some kind of website, it’s to run a few times and be done with it.

Answer 1

Step 1: throw out the web server. There should be a php executable somewhere, just run your script with that: /path/to/php script.php

That way you will at least see what’s happening directly.

Step 2: make sure you actually have enough free RAM. If not… get a better machine. PHP arrays are a very complex structure and will swap in and out every iteration if you don’t.

Step 3: make sure the problem is with the include. Add some echo right after it, see if it executes.

If everything else fails, try parsing the file line by line. Depending on the format, it might be easier to eval() the code or parse it with a regex, it’s up to you.

Also:

3-The logic for having to store a small portion of data like 10mb in
hard drive

Any decent DB will keep a copy of the data in RAM anyway. The difference is that DBMS are optimized to store and access it efficiently. Of course, there are a few tasks that are done more efficiently by getting the whole dataset into the program, but you don’t want to use a web page for that. (see step 1 above)