Update:
I wrote a working script that finishes this job in a reasonable length of time, and seems to be quite reliable. It’s coded entirely in PHP and is built around the array_diff() idea suggested by saccharine (so, thanks saccharine!).
You can access the source code here: http://pastebin.com/ddeiiEET
I have a MySQL database that is an index of mp3 files in a certain directory, together with their attributes (ie. title/artist/album).
New files are often being added to the music directory. At the moment it contains about 25,000 MP3 files, but I need to create a cron job that goes through it each day or so, adding any files that it doesn’t find in the database.
The problem is that I don’t know what is the best / least taxing way of doing this. I’m assuming a MySQL query would have to be run for each file on each cron run (to check if it’s already indexed), so the script would unavoidably take a little while to run (which is okay; it’s an automated process). However, because of this, my usual language of choice (PHP) would probably not suffice, as it is not designed to run long-running scripts like this (or is it…?).
It would obviously be nice, but I’m not fussed about deleting index entries for deleted files (if files actually get deleted, it’s always manual cleaning up, and I don’t mind just going into the database by hand to fix the index).
By the way, it would be recursive; the files are mostly situated in an Artist/Album/Title.mp3 structure, however they aren’t religiously ordered like this and the script would certainly have to be able to fetch ID3 tags for new files. In fact, ideally, I would like the script to fetch ID3 tags for each file on every run, and either add a new row to the database or update the existing one if it had changed.
Anyway, I’m starting from the ground up with this, so the most basic advice first I guess (such as which programming language to use – I’m willing to learn a new one if necessary). Thanks a lot!
First a dumb question, would it not be possible to simply order the files by date added and only run the iterations through the files added in the last day? I’m not very familiar working with files, but it seems like it should be possible.
If all you want to do is improve the speed of your current code, I would recommend that you check that your data is properly indexed. It makes queries a lot faster if you search through a table’s index. If you’re searching through columns that aren’t the key, you might want to change your setup. You should also avoid using “SELECT *” and instead use “SELECT COUNT” as mysql will then be returning ints instead of objects.
You can also do everything in a few mysql queries but will increase the complexity of your php code. Call the array with information about all the files $files. Select the data from the db where the files in the db match the a file in $files. Something like this.
Read the returned array and label it $db_files. Then find all files in $files array that don’t appear in $db_files array using array_diff(). Label the missing files $missing_files. Then insert the files in $missing_files into the db.