I’m working on desktop application (Extract Transform Load type of app) that requires uploading of large CSV files (about 6 GB) to a database using Java as front end and Oracle as a backend but I’m open to other databases and tools as well (i.e. SQLite, H2, HSQLDB, MySQL).
I tried using OpenSource Java libraries for parsing and uploading CSV files such as:
The problem with these libraries is efficiency. They require too much time for uploading. For example, It’ll take 6 hours to upload 4 Gb of CSV file.
Having this experience, I decided to try database utilities for uploading CSV files. One of them is Oracle SQL*Loader. It’s faster and offers desirable result. It can remove whitespaces, load CSV files to various tables, use oracle functions suchs as decode, etc.
However, I have the following problems:
-
I couldn’t find a way to get SQL*Loader’s errors/logs in Java and display them to the user.
-
Since, SQL*Loader is proprietary, there’s no way to embed it to my application installation. For example, the user needs to download and install Oracle Client separately before installing my application. Another thing is I need to check if Oracle Client is installed before installation process begins.
I looked for other probable solutions and found headless databases like SQLite and H2.
On SQLite, I use the .import command to upload CSV files. But there’s no way I could call it in Java.
I turn to H2 database but I have hard time uploading since it does not support escaping the header row.
Please let me know your suggestions.
Thanks you.
It sounds like you could use CloverETL for this. It is a Java based data integration and ETL tool. Along with its commercial editions it has a free, opensource ETL engine. It should be quite easy to embed to your application as a Java library. You also should not have any performance issues you mention above, unless the bottleneck is on the DB side. CloverETL is said to be fast.
If you need to know other technical details, you can either ask me or use CloverETL forum.