Question

I have a data stream with the following structure

user_id (integer)
user_name (string)

The user_id is anything between 100 and 65536. I want to add a target_user_id (integer) field according to the following logic:

• If user_id is in range 1000..9999, then let the target_user_id field be equal to the user_id
• If not, then fill target_user_id with something in range 1000..9999 without causing a conflict. Preferably the lowest possible.

The length of the stream is under 9000. The user_id field is unique in the original stream.

Answer 1

I am not sure what Kettle environment you are using but a general procedure could be as follows:

1. Create a temporary database table (Maybe an in-memory database table)
2. Initialise it with records with user_id 1000..9999 and user_name=null (use TableOutput)
3. Open the input stream and process records with user_id 1000..9999 by updating the respective database record with the user_name. (use Update) Ignore all other records.
4. Close and reopen the input stream

5. Process each input stream record with user_id not in 1000..9999 by:

   • get the lowest unused user_id by executing a SQL query (DBLookup)

     SELECT MIN(user_id) FROM temporary_table WHERE user_name IS NULL;
     

   • Update this record with the current user_name (use Update)

6. Read each record in the temporary database table with a non null user_name (use TableInput) and write to output stream
7. Delete temporary database table

Hope this helps