Home/ Questions/Q 4572130
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-21T19:36:09+00:00

I’m building a DB2 Infosphere data warehouse and am expecting to have 8-16 nodes

  • 0

I’m building a DB2 “Infosphere” data warehouse and am expecting to have 8-16 nodes or partitions.

Since I’ll be loading from 130-300 million rows a day, and my load process is also my recovery process – I want the loads to be as fast as possible. I’m not surprised to find this tip in the IBM “infocenter” documentation:

“Better performance can be expected if the database partitions participating in the distribution process are different from the loading database partitions, since there is less contention for CPU cycles.”

I’d prefer not to dedicate an expensive DB2 node just to splitting load files by hashkey – since my ETL servers are so cheap (we use python, not a licensed commercial product). Plus, since I rely on archived loads for recovery – I may have to convert them in case we add nodes to the database. I’d like that also done on an ETL server. Note – I believe DataStage also performs this task on the ETL server rather than through DB2.

Can anyone suggest how our python ETL process can efficiently use the same hashing algorithm and mapping tables that DB2 will use? And other tips?

Thanks

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    First of all:

    You do not need to pre-split the data inside your ETL process. The LOAD utility will handle splitting the data for you. Your python process can either write the data to load to a flat file or write directly to a pipe (that the LOAD utility reads from). In almost every case, it is easier to let the database handle partitioning the data for you.

    The InfoCenter comment about the splitters taking up CPU cycles is probably not something you need to worry about. This generally applies only in extreme situations, where there are many more database partitions (i.e., when you need to have multiple processes splitting the data) and when CPU utilization on the database nodes is very high.

    From a LOAD perspective, the amount of time you’ll save by having pre-split data is negligible. The limiting factor when loading data is writing the data out to disk – not partitioning it. If reloading data is your primary method of recovery, then I wouldn’t worry too much about this.

    If all of this does not convince you and you really want to go down the path of having your ETL process split the data, DB2 does provide an API (in C) that applications can call to handle this: db2GetDistMap() and db2GetRowPartNum(). You may be able to write a native python module to handle this.

    These are most useful in cases where an application is using SQL to INSERT rows into the table (as opposed to using the LOAD utility), and spawns multiple threads to write data to each partition independently (i.e., each thread is doing the transformation and loading in parallel). If you can’t parallelize the transformation portion, then don’t bother with this.

    Obviously, there are a lot of variables, so YMMV.

    • 0