Question

When trying to do data processing on large delimited text files I decided to use a database system to do the work, and increase the speed. I needed my program that I develop to do 6 crucial processes in order to complete the desired goal.

1. Create a local database on the fly
2. Create a table with an unknown amount of columns or types
3. Import the delimited file into a table (repeat 2 and 3 as needed)
4. Preform SQL query to get the records desired
5. Export the result into another delimited text file

I decided to go with Apache’s Derby Database system due to the ability to do 1 and the promise of the ability to do the other 4.

I create the database:

    String connectionName = "jdbc:derby:" + databaseName;

    if (createDatabase) {
        connectionName += ";create=true";
    }
    Class.forName("org.apache.derby.jdbc.ClientDriver").newInstance();
    connectionToDatabase = DriverManager.getConnection(connectionName);

This works then I get the first record from the file and get the amount of fields in order to know how many columns needed:

    String statement = "CREATE TABLE " + tableName + " (";
    for (int i = 1; i <= tableAmount; i++) {
        statement += (char) (64 + i) + " VARCHAR(100)";
        if (i != tableAmount) {
            statement += ",";
        }
    }
    PreparedStatement pstmt = connectionToDatabase.prepareStatement(statement + ")");
    pstmt.executeUpdate();
    connectionToDatabase.commit();

This creates a table by the name I chose (I experimented with simply creating a table named “HELLO”) then I try to import the file into the table I made:

    try {

        String schemaName = "APP";
        String tableName = "HELLO";
        String fileName = "C:\\Hello.txt";
        String columnDelimiter = fieldDelimiter;
        String characterDelimiter = "";
        String codeset = "UTF-8";
        short replace = 0;

        Import.importTable(connectionToDatabase, schemaName, tableName, fileName,
                columnDelimiter, characterDelimiter, codeset,
                replace, false);
    } catch (SQLException e) {
        do {
            System.out.println("SQLState:" + e.getSQLState());
            System.out.println("Error Code:" + e.getErrorCode());
            System.out.println("Message:" + e.getMessage());
            Throwable t = e.getCause();
            while (t != null) {
                System.out.println("Cause:" + t);
                t = t.getCause();
            }
            e = e.getNextException();
            StackTraceElement st[] = e.getStackTrace();

            for (int i = 0; i < st.length; i++) {
                System.out.println("Stack Trace " + i + ":" + st[i]);
            }
        } while (e != null);
    }catch (Exception e) {
        System.err.println("Exception: " + e.getMessage());

    }

But when I run this all I get is:

Exception: Syntax error: DERBY-PROPERTIES

What did I do wrong, and if possible, what should I do to make this work?

Edit: It errors out on the line where Import.importTable is called. After revising my output on error (above) I now get the following as output:

SQLState:42X01
Error Code:30000
Message:Syntax error: DERBY-PROPERTIES.
Cause:java.sql.SQLException: Syntax error: DERBY-PROPERTIES.
Cause:ERROR 42X01: Syntax error: DERBY-PROPERTIES.
Stack Trace 0:org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source)
Stack Trace 1:org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
Stack Trace 2:org.apache.derby.impl.jdbc.TransactionResourceImpl.wrapInSQLException(Unknown Source)
Stack Trace 3:org.apache.derby.impl.jdbc.TransactionResourceImpl.handleException(Unknown Source)
Stack Trace 4:org.apache.derby.impl.jdbc.EmbedConnection.handleException(Unknown Source)
Stack Trace 5:org.apache.derby.impl.jdbc.ConnectionChild.handleException(Unknown Source)
Stack Trace 6:org.apache.derby.impl.jdbc.EmbedPreparedStatement.<init>(Unknown Source)
Stack Trace 7:org.apache.derby.impl.jdbc.EmbedPreparedStatement20.<init>(Unknown Source)
Stack Trace 8:org.apache.derby.impl.jdbc.EmbedPreparedStatement30.<init>(Unknown Source)
Stack Trace 9:org.apache.derby.impl.jdbc.EmbedPreparedStatement40.<init>(Unknown Source)
Stack Trace 10:org.apache.derby.jdbc.Driver40.newEmbedPreparedStatement(Unknown Source)
Stack Trace 11:org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
Stack Trace 12:org.apache.derby.impl.jdbc.EmbedConnection.prepareStatement(Unknown Source)
Stack Trace 13:org.apache.derby.impl.load.Import.performImport(Unknown Source)
Stack Trace 14:org.apache.derby.impl.load.Import.importTable(Unknown Source)
Stack Trace 15:root.DatabaseManager.<init>(DatabaseManager.java:46)
Stack Trace 16:root.Startup.main(Startup.java:21)

The Following is the output from my derby.log

Wed Nov 14 20:04:23 CST 2012:
Booting Derby version The Apache Software Foundation - Apache Derby - 10.9.1.0 - (1344872): instance a816c00e-013b-01cf-55d8-000000c58148 on database directory -omitted-  with class loader sun.misc.Launcher$AppClassLoader@1ba34f2 Loaded from file:/-omitted-/lib/derby.jar java.vendor=Sun Microsystems Inc. java.runtime.version=1.6.0_33-b05 user.dir=-omitted-
derby.system.home=null
Database Class Loader started - derby.database.classpath=''

Figured out the problems:

• I was using the wrong driver: I was using client driver instead of embedded driver

• I was using an improper function. It would have work with some editing but the correct one was

  Statement s = connectionToDatabase.createStatement();
  s.execute("CALL SYSCS_UTIL.SYSCS_IMPORT_TABLE(null,'" + tableName
          + "','" + filePath + "','" + columnDelimiter
          + "',null,null,1)");
  connectionToDatabase.commit();
  

• Finally, the last problem was that the text file i was using did not have a next line on the last line.

Answer 1

Firstly, you can get more information from the exception; here’s how: http://wiki.apache.org/db-derby/UnwindExceptionChain

Secondly, you should find your ‘derby.log’ file on disk; it will have lots more information about what went wrong.

Thirdly, once you have figured out exactly what statement you issued to the database, and where in your code you issued that statement, update your question with that information and we can provide more help.