Question

I need to execute ETL operation on large data package using spring integration.
For example, I have huge xml file like:

<school>
    <cource name="A">
        <class>
            <name>A1</name>
            <students>20</students>
        </class>
        ...
        <class>
            <name>A35</name>
            <students>19</students>
        </class>
    </cource>
    <cource>
    ...
    </cource>
</school>

The result should be 2 cvs files:

First:

A1;20
...
A35;35

Second:

A; 754
..
C; 232

That is simple convertion operation and some aggregation function on container. Count of “class” and “cource” records are really huge and I can’t parse input data in memory (so i need to iterate by elements),
but i still wan’t to use integration patterns for easy modification ETL flow.

Is there any way how can i do that?

Answer 1

You can create a step with ItemReader and ItemWriter associated with it. Large files are streamed, not read as a whole. So, this should do the magic.

More reading here. Example here.