We are building a warehouse using SSIS.
I am doing an aggregation and I received the following message
The Aggregate transformation has encountered 2000037 key combinations. It has to re-hash data because the number of key combinations is more than expected. The component can be configured to avoid data re-hash by adjusting the Keys, KeyScale, and AutoExtendFactor properties
It did not write the output, and most disturbingly, it did not throw any exceptions.
What is going on here? How do I fix this?
Key combinations are the number of groups in the data flow. Set the KeysScale (advanced property of task) value to a number > 2000037. Try to guess as close as possible so that the transformation can allocate enough memory to perform the aggregation.
I would avoid using this transformation when at all possible. Not only is it memory intensive, but it holds up the entire data flow task until it completes.
Where is the data coming from? If a relational database, move the group by, order by, and any aggregates to the database query.