Question

I have a static database of ~60,000 rows. There is a certain column for which there are ~30,000 unique entries. Given that ratio (60,000 rows/30,000 unique entries in a certain column), is it worth creating a new table with those entries in it, and linking to it from the main table? Or is that going to be more trouble than it’s worth?

To put the question in a more concrete way: Will I gain a lot more efficiency by separating out this field into it’s own table?

** UPDATE **

We’re talking about a VARCHAR(100) field, but in reality, I doubt any of the entries use that much space — I could most likely trim it down to VARCHAR(50). Example entries: “The Gas Patch and Little Canada” and “Kora Temple Masonic Bldg. George Coombs”

Answer 1

If the field is a VARCHAR(255) that normally contains about 30 characters, and the alternative is to store a 4-byte integer in the main table and use a second table with a 4-byte integer and the VARCHAR(255), then you’re looking at some space saving.

Old scheme:

T1: 30 bytes * 60 K entries = 1800 KiB.

New scheme:

T1:  4       bytes * 60 K entries =  240 KiB
T2: (4 + 30) bytes * 30 K entries = 1020 KiB

So, that’s crudely 1800 – 1260 = 540 KiB space saving. If, as would be necessary, you build an index on the integer column in T2, you lose some more space. If the average length of the data is larger than 30 bytes, the space saving increases. If the ratio of repeated rows ever increases, the saving increases.

Whether the space saving is significant depends on your context. If you need half a megabyte more memory, you just got it — and you could squeeze more if you’re sure you won’t need to go above 65535 distinct entries by using 2-byte integers instead of 4 byte integers (120 + 960 KiB = 1080 KiB; saving 720 KiB). On the other hand, if you really won’t notice the half megabyte in the multi-gigabyte storage that’s available, then it becomes a more pragmatic problem. Maintaining two tables is harder work, but guarantees that the name is the same each time it is used. Maintaining one table means that you have to make sure that the pairs of names are handled correctly — or, more likely, you ignore the possibility and you end up without pairs where you should have pairs, or you end up with triplets where you should have doubletons.

Clearly, if the type that’s repeated is a 4 byte integer, using two tables will save nothing; it will cost you space.

A lot, therefore, depends on what you’ve not told us. The type is one key issue. The other is the semantics behind the repetition.