Home/ Questions/Q 9115419
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-17T04:25:40+00:00

I have a large MySql table with over 11 million rows. This is just

  • 0

I have a large MySql table with over 11 million rows. This is just a huge data set and my task is to be able to analyze the dataset based on certain rules.

Each row belongs to a certain category. There are 2 million different categories. I want to get all rows for a category and perform operations on that.

So currently, I do the following:

  1. Select distinct categories from the table.
  2. for each category : Select fields from table WHERE category=category

Even though my category column is indexed, it takes a really long time to execute Step 2. This is mainly because of the huge data set.

Alternatively, I can use GROUP BY clause, however I am not sure if it will be as fast since GROUP BY on such a huge dataset may be expensive, especially when considering that I will be running my analysis several times on parts of the dataset. A way to permanently ensure a sorted table would be useful.

Therefore as an alternative, I can speed up my queries if only my table is pre-sorted by category. Now I can just read the table row by row and perform the same operations in a much faster time, as all rows of one category will be fetched consecutively.

As the dataset (MySql table) is fixed and no update, delete, insert operations will be performed on it. I want to be able to ensure a way to maintain a default sort order by category. Can anyone suggest a trick to ensure the default sort order of the rows.

Maybe read all rows and rewrite them to a new table or add a new primary key which ensures this order?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Even though my category column is indexed

    Indexed by a secondary index? If so, you can encounter the following performance problems:

    • InnoDB tables are always clustered and the secondary index in clustered table can require a double-lookup (see the “Disadvantages of clustering” in this article).
    • Indexed rows can be scattered all over the place (index can have bad clustering factor – the link is for Oracle but the principle is the same). If so, an index range scan (such as WHERE category = whatever) can end-up loading many table pages, even though the index is actually used and only a small subset of rows is actually selected. This can destroy the range scan performance.

    In alternative to the secondary index, consider using a natural primary key, which in InnoDB tables also acts as a clustering key. The primary/clustering key such as {category, no} will keep the rows of the same category physically close together, making both of your queries (and especially the second one) maximally efficient.

    OTOH, if you want to keep the secondary index, consider covering all the fields that you query, so the primary B-Tree doesn’t have to be touched at all.

    • 0