Home/ Questions/Q 8316965
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-08T21:28:49+00:00

I am working with mysql querying a table that has 12 millions registers that

  • 0

I am working with mysql querying a table that has 12 millions registers that are a year of the said data.
The query has to select certain kind of data (coin, enterprise, type, etc..) and then provide a daily average for certain fields of that data, so we can graph it afterwards.
The dream its to be able to do this in real time, so with a response time less than 10 secs, however at the moment its not looking bright at all as its taking between 4 to 6 minutes.
For example, one of the where querys come up with 150k registers, split about 500 per day, and then we average three fields (which are not on the where clause) using a AVG() and GroupBy.

Now, to the raw data, the query is

SELECT 
`Valorizacion`.`fecha`, AVG(tir) AS `tir`, AVG(tirBase) AS `tirBase`, AVG(precioPorcentajeValorPar) AS `precioPorcentajeValorPar` 
FROM `Valorizacion` USE INDEX (ix_mercado2)
WHERE
(Valorizacion.fecha >= '2011-07-17' ) AND
(Valorizacion.fecha <= '2012-07-18' ) AND
(Valorizacion.plazoResidual >= 365 ) AND
(Valorizacion.plazoResidual <= 3650000 ) AND
(Valorizacion.idMoneda_cache IN ('UF')) AND
(Valorizacion.idEmisorFusionado_cache IN ('ABN AMRO','WATTS', ...)) AND
(Valorizacion.idTipoRA_cache IN ('BB', 'BE', 'BS', 'BU'))
GROUP BY `Valorizacion`.`fecha` ORDER BY `Valorizacion`.`fecha` asc;

248 rows in set (4 min 28.82 sec)

The index is made over all the where clause fields in the order

(fecha, idTipoRA_cache, idMoneda_cache, idEmisorFusionado_cache, plazoResidual)

Selecting the “where” registers, without using group by or AVG

149670 rows in set (58.77 sec)

And selecting the registers, grouping and just doing a count(*) istead of average takes 

248 rows in set (35.15 sec)

Which probably its because it doesnt need to go to the disk to search for the data but its obtained directly from the index queries.

So as far as it goes im of the idea of telling my boss “Im sorry but it cant be done”, but before doing so i come to you guys asking if you think there is something i could do to improve this. I think i could improve the search by index time moving the index with the biggest cardinality to the front and so on, but even after that the time that takes to access the disk for each record and do the AVG seems too much.

Any ideas?

— EDIT, the table structure

CREATE TABLE `Valorizacion` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `idInstrumento` int(11) NOT NULL,
  `fecha` date NOT NULL,
  `tir` decimal(10,4) DEFAULT NULL,
  `tirBase` decimal(10,4) DEFAULT NULL,
  `plazoResidual` double NOT NULL,
  `duracionMacaulay` double DEFAULT NULL,
  `duracionModACT365` double DEFAULT NULL,
  `precioPorcentajeValorPar` decimal(20,15) DEFAULT NULL,
  `valorPar` decimal(20,15) DEFAULT NULL,
  `convexidad` decimal(20,15) DEFAULT NULL,
  `volatilidad` decimal(20,15) DEFAULT NULL,
  `montoCLP` double DEFAULT NULL,
  `tirACT365` decimal(10,4) DEFAULT NULL,
  `tipoVal` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
  `idEmisorFusionado_cache` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
  `idMoneda_cache` varchar(20) COLLATE utf8_unicode_ci DEFAULT NULL,
  `idClasificacionRA_cache` int(11) DEFAULT NULL,
  `idTipoRA_cache` varchar(20) COLLATE utf8_unicode_ci NOT NULL,
  `fechaPrepagable_cache` date DEFAULT NULL,
  `tasaEmision_cache` decimal(10,4) DEFAULT NULL,
  PRIMARY KEY (`id`,`fecha`),
  KEY `ix_FechaNemo` (`fecha`,`idInstrumento`) USING BTREE,
  KEY `ix_mercado_stackover` (`idMoneda_cache`,`idTipoRA_cache`,`idEmisorFusionado_cache`,`plazoResidual`)
) ENGINE=InnoDB AUTO_INCREMENT=12933194 DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Selecting 150K records out of 12M records and performing aggregate functions on them will not be fast no matter what you try to do.

    You are probably dealing with primarily historical data as your sample query is for a year of data. A better approach may be to pre-calculate your daily averages and put them into separate tables. Then you may query those tables for reporting, graphs, etc. You will need to decide when and how to run such calculations so that you don’t need to re-run them again on the same data.

    When your requirement is to do analysis and reporting on millions of historical records you need to consider a data warehouse approach http://en.wikipedia.org/wiki/Data_warehouse rather than a simple database approach.

    • 0