Question

I have 2 databases old & new, old db detail needs to be filtered/manipulated and stored into new.

OLD DB

1. I have around 10000 configurations (DB rows)

2. and 10000 BLOBS (xml file size is 4MB on an average) matching config id’s from above

NEW DB

1. 1 new table that is going to contain the filtered data from old, but this time no BLOB data, instead absolute paths

2. and per configuration some recommendations

Here I wrote a program (using Groovy & MyBatis for DB) which gets all the configuration records available in OLD DB and stores in a List of class and the DB Connection is closed

In order to fetch the BLOBS too for each config id, a new connection is established & is kept open for a long time

List<String> projectid
List<CSMConfigInfo> oldConfigs
List<ConfigInfo> newConfigs 
Map<String,CSMConfigInfo> oldConfigMap

SqlSession session = DatabaseConnectivity.getOldCSMDBSessionFactory().openSession()
/*  trying to batch execute based on project id */
    projectid.each {pid->
        logger.info "Initiating conversion for all configuration under $pid project id"
        oldConfigMap.each {k,v->
/*  Here I am keeping a DB connection open for a long time */           
            if(pid.equals(v)){                  
                createFromBlob(k,session)                   
            }
        }                       
        logger.info "Completed for $pid project id\n"           
    }

session.close()

After fetching the BLOB 1 by 1, I create a temp xml file which is parsed to apply the filter for inserting into NEW DB. Below code you can see that based on whether xml is convertible and parsible a new connection is opened for NEW DB. Is this good practice or do I need to keep the NEW DB connection open for all 10000 records?

/* XML is converted to new format and is parsable */
def createFromBlob(CSMConfigInfo cfg,SqlSession oldCSMSession){
    .
    .

    if(xmlConverted&&xmlParsed){
        //DB Entries
        try{
            /* So now here I am opening a new connection for every old config record, which can be 10000 times too, depending on the filter */
            SqlSession sess = DatabaseConnectivity.getNewCSMSessionFactory().openSession()

            //New CSM Config
            makeDatabaseEntriesForConfiguration(newConfig,sess)
            //Fire Rules
            fireRules(newConfig,sess,newCSMRoot)

            sess.close()

        }
        catch(IOException e){
            logger.info "Exception with ${newConfig.getCfgId().toString()} while making DB entries for CONFIG_INFO"
        }
        logger.info "Config id: "+cfg.getCfgId().toString()+" completed successfully, took "+getElapsedTime(startTime)+ " time. $newabspath"
    }
    else{
        def errormsg = null
        if(!xmlConverted&&!xmlParsed)
            errormsg = "Error while CONVERSION & PARSING of config id "+cfg.getCfgId().toString()+", took "+getElapsedTime(startTime)+ " time."
        else if(!xmlConverted)
            errormsg = "Error while CONVERSION of config id "+cfg.getCfgId().toString()+", took "+getElapsedTime(startTime)+ " time."
        else if(!xmlParsed)
            errormsg = "Error while PARSING  of config id "+cfg.getCfgId().toString()+", took "+getElapsedTime(startTime)+ " time."
        logger.info errormsg
    }

    makeDatabaseEntriesForConvertStatus(csmConfigConvertStatus,oldCSMSession)
}

This currently works for 20 records, but I am not sure how will it react for all 10000 records. Please help

Update

I takes about 3-6 secs for each config

Answer 1

It will always be more efficient to use a pool of database connections, let a container manage establishing and closing that connection when required. Creating a connection, executing your statement and then closing that connection can be avoided with a pool which will pre-connect before giving you the connection to use and in most cases (particulars for what you described) the connection will not need to be made given it will likely be connected already.

So therefore yes it is more efficient to keep the connection open and even better to pool your connections…