Question

I need some guidance around which approach to use to load binary files from a folder into a MySQL Database using Camel. Basically I want to store voice logs from our PBX system into a database. The directory with the voice logs will be a remote directory

I have designed a prototype but I am not sure if this is really efficient, it works but I am not happy with the design. Let me explain what I am doing. Camel route as follows:

    <camelContext xmlns="http://camel.apache.org/schema/spring">
    <package>com.hia.camelone</package>
      <route>
            <from uri="file://c:/CTest/Inbox?noop=true&amp;recursive=true&amp;delay=3000"/>
            <to uri="bean://fileToSQL"/>
            <to uri="jdbc://timlogdb"/>

       </route>

</camelContext>

<bean id="timlogdb" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
    <property name="driverClassName" value=" com.mysql.jdbc.Driver"/>
    <property name="url" value="jdbc:mysql://127.0.0.1:3306/TimLog" />
    <property name="username" value="root" />
    <property name="password" value="blahblah" />
</bean>
<bean id="fileToSQL" class="com.hia.camelone.fileToSQL"/>

And the code to fileToSQL bean is:

public class fileToSQL {

public String toString(@Headers Map<String,Object> header, @Body Object body){
    StringBuilder sb = new StringBuilder();
    String filename =(String)header.get("CamelFileNameOnly");
    String escapedFileName = StringEscapeUtils.escapeJava(filename).replace("\'", "");
    String filePath = StringEscapeUtils.escapeJava((String)header.get("CamelFilePath"));

    sb.append("insert into FileLog ");
    sb.append("(FileName,FileData) values (");
    sb.append("'").append(escapedFileName).append("',").append("LOAD_FILE(\"").append(filePath).append("\")");
    sb.append(")");
    System.out.println(sb.toString());
    System.out.println(body);
    System.out.println(header.toString());
    return sb.toString();
}
}

Ok short explanation I get the file component to consume the files then I build a SQL string using the MySQL LOAD_FILE() function to load the file.

My thoughts around this:

The LOAD_FILE function only works on the local machine and thus this route will only with the files being on the local machine. I could use a file producer to copy the files from some remote directory to a local directory and then use the route. My route would be something like this then:

<route>
            <from uri="file://c:/CTest/Inbox?noop=true&amp;recursive=true&amp;delay=3000"/>
            <to uri="file://c:/outbox"/>
            <to uri="bean://fileToSQL"/>
            <to uri="jdbc://timlogdb"/>

</route>

However since I have access to the files content in the message from the files consumer I should be able to theoretically be able to access the body/content of the string and build a SQL command that does NOT use the LOAD_FILE() function.

The only way I know how to build such a string is by using the prepared statement of JDBC. This would be first prize if I could somehow build a insert statement with the content from the file consumer.

Can I create a prepared statement in my fileToSQL bean and pass it to my jdbc component?
Or how do I build a INSERT statement without the LOAD_FILE() function?

Since I have to use the LOAD_FILE() function I would now have to cater for both unix and windows filepaths. While this should not be difficult I just dont like the idea of putting OS specific code into my applications(feels like a work around).

Anybody here ever uploaded binary files to a MySQL database using Camel who can give me some guidance on the points above. While I could work around the problems I just want to make sure I dont miss a obvious way of doing things.

I had a look around here and only found people working with mostly text files. Guys please don’t even go down the route of me storing the file on the files system and linking it to the database. We have some very specific disaster recovery requirements and legal requirements that enforce the need for me to store it in a database.

Answer 1

Right so I managed to find a way and it was not that difficult. What I essentially did was get rid of the JDBC Camel Component in the route. I then injected the data source bean into my fileToSQL bean. I then used a simple prepared statement to insert the file and its name into MySQL.

As always code is much more explicit than my english.

 <camelContext xmlns="http://camel.apache.org/schema/spring">
    <package>com.hia.camelone</package>

      <route>
            <from uri="file://c:/CTest/Inbox?noop=true&amp;recursive=true&amp;delay=3000"/>
            <to uri="bean://fileToSQL"/>
            <!--<to uri="jdbc://timlogdb"/>-->

       </route>

</camelContext>

<bean id="timlogdb" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
    <property name="driverClassName" value=" com.mysql.jdbc.Driver"/>
    <property name="url" value="jdbc:mysql://127.0.0.1:3306/TimLog" />
    <property name="username" value="root" />
    <property name="password" value="lalala" />
</bean>
<bean id="fileToSQL" class="com.hia.camelone.fileToSQL">
    <property name="dataSource" ref="timlogdb"/>
</bean>

As you can see I inject my timlogdb bean into my fileToSQL bean. Spring ROCKS!

So here is my fileToSQL bean.

public class fileToSQL {
private DriverManagerDataSource dataSource;
private static final String SQL_INSERT="insert into FileLog(FileName,FileData)values(?,?)";
@Handler
public void toString(@Headers Map<String,Object> header,Exchange exchange){
    Connection conn = null;
    PreparedStatement stmt=null;
    String filename =StringEscapeUtils.escapeJava(((String)header.get("CamelFileNameOnly")).replace("\'", ""));

    try {
        conn= dataSource.getConnection();
        stmt =conn.prepareStatement(SQL_INSERT);
        stmt.setString(1, filename);
        byte[] filedata = exchange.getIn().getBody(byte[].class);
        stmt.setBytes(2,filedata );
        int s = stmt.executeUpdate();

    }
    catch (Exception e)
    {
        System.out.println(e.getMessage());
    }
    finally{
        try
        {
                if (stmt!=null)
                {
                    stmt.close();
                }
                if (conn!=null)
                {
                    conn.close();
                }
        }
        catch(SQLException e)
        {
            System.out.println(e.getMessage());
        }
    }


}

/**
 * @param dataSource the dataSource to set
 */
public void setDataSource(DriverManagerDataSource dataSource) {
    this.dataSource = dataSource;
}
}

The guys from Camel did a great job. Camel is truly flexible especially when you combine it with Spring.

What a ride!