I got a system which queries LDAP, pulls 50,000 records (say) and formats that information into a request XML (say 300 users per XML, 50000/300 = 167 requests back to back) sends it to system B, gets back the response from system B.
I am saving the 167 request and response XML’s as CLOB’s in oracle DB currently. And I need to do this everyday.
The problem is the table space close to 10GB gets used up very fast. So…. how to adopt a better design for this requirement?
Right now the system does what it needs to do. No issues around it. I am want to put in a decent design. Experts… Please help…
If each chunk of XML is large enough to benefit, you might try compressing them using java.util.zip.GZIPOutputStream before writing them to the table. Of course if you need to query the plain-text of the XML, then that won’t help you.
You might also redesign and store the XML data on a larger filesystem and then store URIs to the files in the database along with any pertinent lookup data.