I would like to build a custom setup, with an aero form, but I don’t know how to start it.
Is there any way to use DWM API with inno setup?
function dwm(Wnd: HWnd; cxLeftWidth, cxRightWidth, cyTopHeight, cyBottomHeight: integer ): Longint; external 'DwmExtendFrameIntoClientArea@dwmapi.dll stdcall';
DWM API is a native API so you can access it using the DLL Import method.
Then you can call API functions in your script code.
But I would recommend against doing this. Since the DWM only works on Vista or later and it can be disabled by stopping the NT Service. It could prevent your installation from working on a machine that it was designed to work on.
Now that you posted your code…
Original declaration of the API you posted.
My best guess is that it should look like this instead.