Question

I have the following configuration in the file dataloader.properties

filepath = /xx
exchange = M00,M01,MF2,MF3

I need the MultiResourcePartitioner to process the files from all these folders such as

/xx/M00/*
/xx/M01/*
/xx/MF2/*
/xx/MF3/*

There can be other folders under /xx , but it should process only the files from folder M00,M01,MF2,MF3

<bean id="filepartitioner" class="org.springframework.batch.core.partition.support.MultiResourcePartitioner">
    <property name="resources" value="classpath:./${filepath}" />
</bean>

kindly let me know , how to do this in spring batch .I had a look at a filesystemresource and ResourcesItemReader api , but no idea how to inject it to the MultiResourceParitioner

Answer 1

simple solution – if possible at all
if the pattern stays stable you could try it with

<bean id="filepartitioner" class="org.springframework.batch.core.partition.support.MultiResourcePartitioner">
    <property name="resources" value="classpath:./${filepath}/M*/*" />
</bean>

customized MultiResourcePartitioner – filters folders

public class CustomM... extends MultiResourcePartitioner {
    // constructor with filePath and exchange argument
    // convert exchange argument to list of folder patterns,
    // or let it convert by spring magic
    // use one of the "list all files.." methods from below
    // call setResources(...)
}

<bean id="filepartitioner" class="org.springframework.batch.core.partition.support.CustomMultiResourcePartitioner">
    <constructor-arg type="java.lang.String" value="${filePath}" />
    <!-- spring can convert comma separated values to array and list
         just look into the spring documentation -->
    <constructor-arg type="java.lang.String" value="${exchange}" />
</bean>

a working example is available on my github repo at https://github.com/langmi/spring-batch-examples-playground/blob/4e733dce09daffca1c10d4907f410ac5bead6887/src/main/resources/spring/batch/job/file-multiresourcepartitioner-filter-folders-factory-job.xml, check file-multiresourcepartitioner-filter-folders-job.xml

more pluggable solution: Factory which creates the Resources for the MultiResourcePartitioner

public class FilterFactory {
  public static Resource[] getInstance(final String filePath, final List<String> acceptedFolders) {
    final List<FileSystemResource> files = new ArrayList<FileSystemResource>();
    yourListAllFilesButFilterFoldersMethod(files, filePath, acceptedFolders)
    return files.toArray(new Resource[0]);
  }
}

<bean id="resources" class="de.langmi.spring.batch.examples.playground.resource.FiltersFoldersResourceFactory" factory-method="getInstance">
    <constructor-arg name="filePath" type="java.lang.String" value="${filePath}" />
    <constructor-arg name="acceptedFolders" type="java.util.List" value="${acceptedFolders}" />
</bean>

a working example is available on my github repo at https://github.com/langmi/spring-batch-examples/tree/master/playground/src/main/resources/spring/batch/job, check file-multiresourcepartitioner-filter-folders-factory-job.xml

you can pick the method to set the Resources array from List all files from a directory recursively with Java and similar solutions across the net, only thing left is the filter, there is one solution in the link provided