Question

How is it better to encode a complex number into RGB pixel and vice versa?

Probably (logarithm of) an absolute value goes to brightness and an argument goes to hue.

Desaturated pixes should receive randomized argument in reverse transformation.

Something like:
0 -> (0,0,0)
1 -> (255,0,0)
-1 -> (0,255,255)
0.5 -> (128,0,0)
i -> (255,255,0)
-i -> (255,0,255)

(0,0,0) -> 0
(255,255,255) -> e^(i * random)
(128,128,128) -> 0.5 * e^(i *random)
(0,128,128) -> -0.5

Are there ready-made formulas for that?

Edit: Looks like I just need to convert RGB to HSB and back.

Edit 2: Existing RGB -> HSV converter fragment:

if (hsv.sat == 0) {
    hsv.hue = 0; // !
    return hsv;
}

I don’t want 0. I want random. And not just if hsv.sat==0, but if it is lower that it should be (“should be” means maximum saturation, saturation that is after transformation from complex number).

Answer 1

You will have many problems:

|RGB|=2^24 and |C|=?

Supposing your complex numbers set is finite and discrete, i think the best method for RGB System is like this:
Cartesian Axis:

Polar Axis:

_{(source: sitmo.com)}

i think it will be unique and a little semantic: something like counter grids and coloring of geographical maps and medical images analysis.

you can use HSL system and polar axis instead:

what you want to do with this encoding?