Question

I have a class called Stream that has many records in the database, and many more are constantly being added to it. A field in the streams table called rating can be -1, 0, 1, or be null. I will say “rated” as having rating -1, 0, or 1, and “unrated” as having a null rating. As of right now, there are 24M records in the streams table. 20M are rated, 4M are unrated. The streams table needs to be queried often for the unrated records. The problem is, it takes time to query such a large table.

I thought of breaking up the streams table into 3 separate tables. There could be a rated streams table, an unrated streams table, and an old unrated streams table. The way it would work would be, a new stream is created. New records are unrated, so it would be placed in the unrated streams table. If it is rated, it would be moved to the rated streams table. If it is unrated for more than a week, it would be moved to the old unrated streams table. I would still need to be able to use methods and associations from the Stream class seamlessly, like:

buzz.rated_streams.first
interest.unrated_streams.first
user.old_streams.first
RatedStream.find(1).buzz
UnratedStream.find(2).interest
OldStream.find(3).user

The unrated streams table could be queried much faster. The rated streams table would be useful for analysis. The old unrated streams table would be a dump for old data.

I have some questions. Would breaking up the table be a good idea? If so, how would I do it to keep everything from breaking, and use the same methods and associations from the Stream class? If breaking up the table is a bad idea, what should I do instead?

Answer 1

I think this is a baaaaad idea. A lot of custom code (which is not guaranteed to work).

You should create an index on rating instead. This will likely solve all your problems with queries (you might want to make it covered also).