I only found three types of dialog in gVim:
:let n = inputdialog("value for shiftwidth", &sw)
:call confirm("Save changes?", "&Yes\n&No\n&Cancel")
:browse e
Can I write a GUI Dialog, such as a ColorPick-Dialog to choose RGB value?
Does gVim support custom GUI widget?
No, it is not possible.
However you can make up your own DLL and
libcall(). If your library must return a string (pointer to a zero-terminated array of characters) it will leak.