Question

We have a Proxy that is taking messages from a JMS queue and sends them to an FTP folder. We discovered now, that the sending to the FTP is very slow when the target directory on the FTP already contains a lot of files. (i.e. when I have around 2000 files in a directory, it already takes several seconds)

Here the code of our Proxy (get messages (plain-text) from a JMS and writes them to FTP):

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse" name="myProxy" statistics="disable" trace="disable" transports="jms">
<parameter name="transport.jms.Destination">myQueue</parameter>
<parameter name="transport.jms.ConnectionFactory">myQueueConnectionFactory</parameter>
<parameter name="transport.jms.DestinationType">queue</parameter>
<parameter name="transport.jms.ContentType">
    <rules>
        <jmsProperty>contentType</jmsProperty>
        <default>text/plain</default>
    </rules>
</parameter>
<target faultSequence="rollbackSequence">
    <inSequence>
        <log level="custom">
            <property name="STATUS" value="myProxy called"/>
        </log>
        <property name="ClientApiNonBlocking" scope="axis2" action="remove"/>
        <property name="OUT_ONLY" value="true"/>
        <property name="transport.vfs.ReplyFileName" expression="fn:concat(get-property('SYSTEM_DATE','yyyyMMddHHmmss_SSS'), '_result.txt')" scope="transport"/>

        <send>
            <endpoint key="myFTPendpoint"/>
        </send>
    </inSequence>
</target>

And the FTPEndpoint lookes like this:

<?xml version="1.0" encoding="UTF-8"?>
<endpoint xmlns="http://ws.apache.org/ns/synapse" name="myFTPendpoint">
    <address uri="vfs:ftp://USER:[e-mail]/path/toSomewhere?vfs.passive=true"/>
</endpoint>

My analysis for now:

1. It is only slow when using FTP with VFS. When using the local file system – it is fast.
2. The files are tiny – so it’s not the upload time
3. The network is fast
4. !Speed depends on the number of files already in the directory on the FTP!

Possible solutions?

• Fix the problem of the speed. Disable the directory listing?
• Workaraound: Create new folders at the output (that not one folder gets filled too much)

Does someone also discovered the same issue? And how can the FTP speed to big directories be improved?
Thanks for any help

Answer 1

I believe regardless of whether you do an explicit Directory listing there will be always an inferred Directory listing to determine whether the file write operation will be an overwrite or a create.

This leaves you with the other workaround.

You should create new folders at the output. Implement a hashing scheme to aid in the folder naming so that you know that the folders will not get filled too much. For example, instead of file1234.ext consider file/1/2/3/4.ext.