Question

I have multiple different keys generated in the following format:

“71 1 2”, “69 2 3”, “68 5 6”, etc.

But, I find that most of these pairs go to same reducers.

Even if I implement a custom partitioner, the getNumPartitioner method, in which we use, hash_val % numReducers, mostly returns values, which group to few reducers loading them, whereas, other reducers remain free.
,
According to my understanding, we can use WritableComparator to sort the keys but cannot control keys to go to different reducers.

Is there a way to improve load balancing? Pls help.

I am attaching some code below to make my explanation clear:

String a = "71 1 2";
String b = "72 1 1";
String c = "70 1 3";

int hash_a = a.hashCode();
int hash_b = b.hashCode();
int hash_c = c.hashCode();

int part_a = hash_a % 10;
int part_b = hash_b % 10;
int part_c = hash_c % 10;

System.out.println("hash a: "+hash_a+" part_a: "+part_a);
System.out.println("hash b: "+hash_b+" part_b: "+part_b);
System.out.println("hash c: "+hash_c+" part_c: "+part_c);

Output:

hash a: 1620857277 part_a: 7
hash b: 1621780797 part_b: 7
hash c: 1619933757 part_c: 7

As we see different keys tend to map to same reducer.

Please help! Thanks!

Answer 1

First of all, you cannot simply take the java modulus operation because sometimes hashcode might be negative and surely there isn’t something called a negative partition. So you might probably take an absolute value.

Second here is a strong hash function which I found on the internet. Instead of the normal 32 bit int, it generates a 64bit long. Again this suffers from the problem of negative partitions too, but you can correct that for yourself.

private static long[] byteTable;
private static final long HSTART = 0xBB40E64DA205B064L;
private static final long HMULT = 7664345821815920749L;

private static long[] createLookupTable() {
byteTable = new long[256];
long h = 0x544B2FBACAAF1684L;
for (int i = 0; i < 256; i++) {
  for (int j = 0; j < 31; j++) {
    h = (h >>> 7) ^ h;
    h = (h << 11) ^ h;
    h = (h >>> 10) ^ h;
  }
  byteTable[i] = h;
}
return byteTable;
}
public static long hash(String s) {
byte[] data = s.getBytes();
long h = HSTART;
final long hmult = HMULT;
final long[] ht = createLookupTable();
for (int len = data.length, i = 0; i < len; i++) {
  h = (h * hmult) ^ ht[data[i] & 0xff];
}
return h;
} 

public static void main(String[] args) {

String a = "71 1 2";
String b = "72 1 1";
String c = "70 1 3";

long hash_a = hash(a);
long hash_b = hash(b);
long hash_c = hash(c);

long part_a = hash_a % 10;
long part_b = hash_b % 10;
long part_c = hash_c % 10;

System.out.println("hash a: "+hash_a+" part_a: "+part_a);
System.out.println("hash b: "+hash_b+" part_b: "+part_b);
System.out.println("hash c: "+hash_c+" part_c: "+part_c);
}