Question

I have a database which holds URL’s in a table (along with other many details about the URL). I have another table which stores strings that I’m going to use to perform searches on each and every link. My database will be big, I’m expecting at least 5 million entries in the links table.

The application which communicates with the user is written in PHP. I need some suggestions about how I can search over all the links with all the patterns (n X m searches) and in the same time not to cause a high load on the server and also not to lose speed. I want it to operate at high speed and low resources. If you have any hints, suggestions in pseudo-code, they are all welcomed.

Right now I don’t know whether to use SQL commands to perform these searches and have some help from PHP also or completely do it in PHP.

Answer 1

First I’d suggest that you rethink the layout. It seems a little unnecessary to run this query for every user, try instead to create a result table, in which you just insert the results from that query that runs ones and everytime the patterns change.

Otherwise, make sure you have indexes (full text) set on the fields you need. For the query itself you could join the tables:

SELECT
    yourFieldsHere
FROM
    theUrlTable AS tu
JOIN
    thePatternTable AS tp ON tu.link LIKE CONCAT('%', tp.pattern, '%');