Home/ Questions/Q 6222495
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-24T08:19:26+00:00

2 basic questions that trouble me: How can I be sure that each of

  • 0

2 basic questions that trouble me:

  • How can I be sure that each of the 32 files hive uses to store my tables sits on its unique machine?
  • If that happens, how can I be sure that if hive creates 32 mappers, each of them will work on its local data? Does hadoop/hdfs guarantees this magic, or does hive as a smart application makes sure that it will happen?

Background:
I have a hive cluster of 32 machines, and:

  • All my tables are created with "CLUSTERED BY(MY_KEY) INTO 32 BUCKETS"
  • I use hive.enforce.bucketing = true;
  • I verified and indeed every table is stored as 32 files in the user/hive/warehouse
  • I’m using HDFS replication factor of 2

Thanks!

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team
    1. The data placement is determined by HDFS. It will try to balance bytes over machines. Due to replicate each file will be on two machines, which means you have two candidate machines for reading the data locally.
    2. HDFS knows where each files is stored, and Hadoop uses this information to place mappers on the same hosts as the data is stored. You can look at the counters for your job to see “data local” and “rack local” map task counts. This is a feature of Hadoop that you don’t need to worry about.
    • 0