I have hex rgb color and black-white mask. It’s two integer arrays:
mColors = new int[] {
0xFFFF0000, 0xFFFF00FF, 0xFF0000FF, 0xFF00FFFF, 0xFF00FF00,
0xFFFFFF00, 0xFFFF0000
};
mColorsMask = new int[] {
0xFFFFFFFF, 0xFF000000, 0xFFFFFFFF, 0xFF000000, 0xFFFFFFFF,
0xFFFFFFFF, 0xFF000000
};
I need to convert my color to black value depending on contrast. Contrast is integer value in a range from 0 to 255:

With white all is fine, I make byte addition:
int newHexColor = (contrast << 16) | (contrast << 8) | contrast | mColors[i];
newColorsArray[i] = mode;
How to convert it to black?
You might look into using the HSB color space. It seems much more suited to what you’re trying to do. In particular, you see those angles that end up black in your “what i want” image? Those correspond to “hues” at 60, 180, and 300 degrees (1.0/6, 3.0/6, and 5.0/6 in Java). The white corresponds to 0, 120, and 240 degrees (0, 1.0/3, and 2.0/3 in Java) — and not coincidentally, the colors at those angles are primary colors (that is, two of the three RGB components are zero).
What you’d do is find the difference between your color’s hue and the nearest primary color. (Should be less than 1/6.) Scale it up (multiplying by 6 should do it), to give you a value between 0 and 1.0. That will give you an “impurity” value, which is basically the deviation from the nearest primary color. Of course, that number subtracted from 1.0 gives you the “purity”, or the closeness to a primary color.
You can create a greyscale color based on the impurity or purity by using the respective value as the R, G, and B, with an alpha of 1.0f.
You could either use a color component of the returned color as the “contrast” value, or change the function so that it returns the “purity” or “impurity” as needed.
Note, the math gets wonky with greyscale colors. (The way Java calculates HSB, pure greys are just reds (hue=0) with no tint (saturation=0). The only component that changes is the brightness.) But since your color wheel doesn’t have greyscale colors…