Question

What are the installer options that lets you create an highly customized installers for Windows? I have tried Visual studio and Advanced Installer, they aren’t user friendly and not completely customizable.

By User friendly I mean easy to add files and make registry changes as needed.
By customization, I mean easy to edit installation dialogs etc.

Answer 1

NSIS is a very good installer but may not help if you are targeting an MSI package.

Because you tried Visual Studio and Advanced Installer, I feel you are trying to create a Basic MSI package. So I will answer keeping that in mind.

Installshield has many types of projects:

Basic MSI

Installscript(Non-MSI)

Installscript MSI

With Basic MSI you are basically stuck with the Windows Installer engine. Your UI is as customizable as Windows Installer allows. And that means not much.

Installscript projects are quite flexible and you can do a lot of stuff in it. You can create dialogs or import it from external resource dlls etc… But this is a non-MSI type project.

I would not try Installscript MSI.

If you want real flexibility, create your own UI using any of the many solutions available. VC++, .NET C# Winforms, WPF, Python etc…

The custom UI solution can be made as flexible as you want. This will work well if you keep one thing in mind:

UI sequence is meant for gathering user-input ONLY. NO system changes
are to be made.

So, you can use your custom fancy UI solution to gather all info needed, store the info in Public Properties and invoke the MSI package silently passing all the Public Properties to it. You are done.

Your two goals:

User-friendliness

Customization

can easily be met, then.