Question

I have a table used for lookups which stores the human-readable value in one column and a the same text stripped of special characters and spaces in another. e.g., the value “Children’s Shows” would appear in the lookup column as “childrens-shows”.

Unfortunately the corresponding main table isn’t quite that simple – for historical reasons I didn’t create myself and now would be difficult to undo, the lookup value is actually stored with surrounding asterisks, e.g. ‘*childrens-shows*’.

So, while trying to join the lookup table sans-asterisks with the main table that has asterisks, I figured CONCAT would help me add them on-the-fly, e.g.;

    SELECT * 
      FROM main_table m 
INNER JOIN lookup_table l 
        ON l.value = CONCAT('*',m.value,'*')

… and then the table was toast. Not sure if I created an infinite loop or really screwed the data, but it required an ISP backup to get the table responding again. I suspect it’s because the ‘*’ symbol is probably reserved, like a wildcard, and I’ve asked the database to do the equivalent of licking its own elbow. Either way, I’m hesitant to ‘experiment’ to find the answer given the spectacular way it managed to kill the database.

Thanks in advance to anyone who can (a) tell me what the above actually did to the database, and (b) how I should actually join the tables?

Answer 1

When using CONCAT, mysql won’t use the index. Use EXPLAIN to check this, but a recent problem I had was that on a large table, the indexed column was there, but the key was not used. This should not bork the whole table however, just make it slow. Possibly it ran out of memory, started to swap and then crashed halfway, but you’d need to check the logs to find out.

However, the root cause is clearly bad table design and that’s where the solution lies. Any answer you get that allows you to work around this can only be temporary at best.

Best solution is to move this data into a separate table. ‘Childrens shows’ sounds like a category and therefore repeated data in many rows. This should really be an id for a ‘categories’ table, which would prevent the DB from having to run CONCAT on every single row in the table, as you could do this:

    SELECT * 
      FROM main_table m 
INNER JOIN lookup_table l 
        ON l.value = m.value
   /* and optionally */
INNER JOIN categories cat
        ON l.value = cat.id
     WHERE cat.name = 'whatever'

I know this is not something you may be able to do given the information you supplied in your question, but really the reason for not being able to make such a change to a badly normalised DB is more important than the code here. Without either the resources or political backing to do things the right way, you will end up with even more headaches like this, which will end up costing more in the long term. Time for a word with the boss perhaps 🙂