Question

First off, my excuse for posting here, normally I feel I should be able to find an answer to a situation by trying and searching long enough … so far its been about 5 hours searching and testing, and I can not explain the results I’m getting. I’m kind of at wits end at the moment. If any of you could help me, that would be highly appreciated.

The situation

The following is all existing code, but im busy optimizing a behavior.

I work with three tables;

• the first table contains items,
• the second table contains property fields for all items with among other fields a name and a default value
• the third table contains field values for the property fields, linked to the item by id and linked to the field by field id

The idea is to get all fields for each item with a value. if the row for the field and item value does not exist in the value table, the default value should be used.

this all has to happen in one query.

The guy before me “fixed” this issue by making sure that each time a field was added, a default value field was inserted to the value table for all items. which is of course the wrong way to do it when you can have over 10.000 items and over 10 fields in your database tables

My test case

Working on this system for over 2 years, I have finally gotten time to fix this issue from the powers that be. Normal tests on the working system always gave inconsistent returns to what I expect. this is the current state of an in-house test system:

• 277 item rows
• 3 field rows
• 824 value rows

this is after I cleaned values for items and fields that did not exist anymore on the system (yes this part is bug riddled) by executing a one time only query for cleanup:

DELETE FROM values WHERE item_id NOT IN (SELECT id FROM items) OR field_id NOT IN (SELECT id FROM fields);

I also made a dummy system with the bare minimum of requirements, as the original tables contain more fields:

-- table a represents items
CREATE TABLE IF NOT EXISTS `a` (
  `id` int(11) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

INSERT INTO `a` (`id`) VALUES (1),(2),(3);

-- table b represents fields
CREATE TABLE IF NOT EXISTS `b` (
  `id` int(11) NOT NULL,
  `default` int(11) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

INSERT INTO `b` (`id`, `default`) VALUES
(1, 4),
(2, 5),
(3, 6),
(4, 11),
(5, 12);

-- table c represents values
CREATE TABLE IF NOT EXISTS `c` (
  `id` int(11) NOT NULL,
  `a_id` int(11) NOT NULL,
  `b_id` int(11) NOT NULL,
  `value` int(11) NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

INSERT INTO `c` (`id`, `a_id`, `b_id`, `value`) VALUES
(1, 1, 1, 7),
(2, 1, 2, 8),
(3, 2, 3, 9),
(4, 2, 1, 7),
(5, 3, 2, 8),
(6, 3, 3, 9),
(7, 1, 5, 13);

The expected result should be 831 rows (277 items * 3 fields), where the item / field combinations that are not available in the values table, should be filled with the field default value instead of the value from the values table.

The successful test case SQL I have tried on the dummy system I made to check my findings, returns what I expected:

SELECT  a.id,
    b.id, 
    IF(c.value IS NOT NULL, c.value, b.default) as t_value
FROM a
join b
LEFT JOIN c on c.a_id = a.id AND c.b_id = b.id

This returns 15 rows (3 a (items) x 5 b (fields)), with all expected data

when changing the query for the in-house test system, it should have worked. This is the SQL as I sent it:

SELECT items.id AS item_id, fields.id AS field_id, IF(values.value IS NULL, fields.default_value, value.value) AS field_value
FROM        items
JOIN        fields
LEFT JOIN   values ON values.item_id = item.id AND values.field_id = fields.id

… but it returns 1104 rows instead of the expected 831. The tables have been cleaned of inaccurate data and the extra fields im not taking into account are not used in the SQL at all, plus the abstract test has proven the query concept. Only the practical situation keeps failing.

If anyone can point out my mistake, that would be highly appreciated. The table names in here have been renamed, but by request I can also post a partial dump of the in-house test system tables in question. the examples above should be accurate though.

Answer 1

I didn’t see any mention that you checked for uniqueness. Your “extra rows” may be duplicates.

SELECT a.id FROM a GROUP BY a.id HAVING COUNT(1) > 1 ;

SELECT b.id FROM b GROUP BY b.id HAVING COUNT(1) > 1 ;

SELECT c.a_id, c.b_id FROM c GROUP BY c.a_id, c.b_id HAVING COUNT(1) > 1 ;

SELECT s.a_id, s.b_id
  FROM
       (
        your query here
       ) s
 GROUP BY s.a_id, s.b_id 
HAVING COUNT(1) > 1 ;

We’d expect the id columns in a and b to be unique and non-null.

We’d also expect that (a_id,b_id) in c is unique and non-null. The database can enforce this constraint, if you create a unique index or declare a UNIQUE KEY constraint:

ALTER TABLE c ADD CONSTRAINT c_ux UNIQUE KEY (a_id,b_id);

-or-

CREATE UNIQUE INDEX c_ux ON c (a_id, b_id); 

Absent uniqueness guarantees, there’s a potential that your query will return “duplicate” a.id and b.id pairs.

A “GROUP BY” clause added to your query could be used to eliminate duplicates, but that really seems more of kludge. (Given a particular item and a particular field, how many different values are you going to allow to be stored? And when you pull the values back, which of them do you actually want to return?)

---

Normally, we’d expect the id columns in each of those tables can be defined as a PRIMARY KEY:

ALTER TABLE a ADD PRIMARY KEY (id);
ALTER TABLE b ADD PRIMARY KEY (id);
ALTER TABLE c ADD PRIMARY KEY (id);

And we’d also expect (with InnnoDB tables) foreign keys to be defined:

ALTER TABLE c ADD CONSTRAINT FK_c_a (a_id) REFERENCES a (id) ;
ALTER TABLE c ADD CONSTRAINT FK_c_b (b_id) REFERENCES b (id) ;

---

My preference would be to include the CROSS keyword on the join between a and b, although that doesn’t have any influence on what the query does. It just serves as documentation that the absence of an ON clause is intentional, and that I intended a Cartesian product.

---

UPDATE:

Sometimes, an EAV model like this is designed to hold previous values, as well as current values. When that’s the case, there’s usually an “effective_date” and/or “superseded date” and/or a simple “active” flag that can be used to figure out the current value. So the duplicates may be junk, from a bad implemenetation, or the “duplicates” might be intentional.

In that case, the unique key would likely be something like (a_id, b_id, effective_date).

Queries to deal with historical (“temporal”) values in a EAV model can be fairly involved, and are not for the feint of heart; but it’s possible.

Another possibility is that some of the fields are intended to be “multi-valued”. That is, a repeating attribute of the entity. For example, an item might have multiple ‘keyword’ or ‘tag’ values. ‘The Hobbit’ might have ‘tag’ values of ‘high frame rate’, ‘fantasy’, ‘excruciating’. But, without understanding the database design, we can’t really tell from where we’re sitting.

But, if the database is designed hold only a single, “current” value for a field, (with no ability to see historical values), then I concur that the duplicate (a_id,b_id) rows are potentially (probably ) junk. I would probably save off a copy of all the rows in the table (into a separate “save” table) before I started deleting anything.