Home/ Questions/Q 6755571
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-26T13:25:10+00:00

Now I am using Hadoop to process the data that will finally be loaded

  • 0

Now I am using Hadoop to process the data that will finally be loaded into the same table. I need to a shared sequential number generator to generate id for each row. Now I am using the following approach to generate the unique number:

1) Create a text file, e.g., test.seq, in HDFS for saving the current sequential number.

2) I use a lock file “.lock” to control concurrency. Suppose we have two tasks to processing the data in parallel. If task1 wants to get the number, it will check if the lock file exists. If yes, it means that task2 is accessing the number from the test.seq, then task1 has to wait. When task2 has acquired the number, it overwrites the old number by increasing 1 when it returns, and deletes the lock file “.lock”. When task1 sees the .lock disappear, task1 will firstly create a “.lock” file, then does the same way to get the sequential number.

However, I am not sure if this approach is practical. Because I keep the .lock and test.seq files in the HDFS, even the content of test.seq was changed by task 1, it might not immediately be aware by the task2. As the other tasks get the information about the data in the HDFS is through by namenode. So, the datanode will first notify the change to the namenode, then the other tasks are notified the changes. Is it correct?

Another idea is to create torjan program running on the Master. so, the task get the sequential number is through RPC the Torjan program. But how to run the Torjan program on the master program?

Could anybody give me some advice? thanks!

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    The main problem is that you choose hadoop because of the horizontal scalability properties.
    All forms of horizontal scalability suffer greatly when you include something that needs to be coordinated from a central point.

    So you have two options:

    1. You accept the scaling limitations and go for the solutions proposed by others. (like the zookeeper option)
    2. You choose a solution that does not require a form of central coordination. At the expense of some properties of the key.

    I would try to see if the latter would be enough for your purposes.
    One such solution could be that you take the id of the current tracker instance and append a local counter value.
    This way the value is unique and sequentially per tracker and over multiple runs of the same job, but not within the job.

    • 0