Question

When building a Visual Studio 2010 Setup project with a CustomAction on x64 systems, Visual Studio includes the wrong version of InstallUtilLib.dll: It installs the 32bit shim, which will not work for CustomActions compiled as 64-bit (a requirement in my case, since it depends on 64-bit native dlls).

Installing such a .msi results in the System.BadImageFormat exception.

According to this post (64-bit Managed Custom Actions with Visual Studio), the solution is to open the resulting .msi in orca.exe and replace the binary "InstallUtil".

I’d like to automate this. Any ideas?

EDIT: based on the answer provided by mohlsen, I added following script to the solution (not the setup project itself, as files added to the setup project go into the msi…):

Option Explicit
rem -----------------------------------------------------------
rem Setup_PostBuildEvent_x64.vbs
rem 
rem Patch an msi with the 64bit version of InstallUtilLib.dll 
rem to allow x64 built managed CustomActions.
rem -----------------------------------------------------------    

Const msiOpenDatabaseModeTransact = 1
Const msiViewModifyAssign         = 3

rem path to the 64bit version of InstallUtilLib.dll
Const INSTALL_UTIL_LIB_PATH = "C:\Windows\Microsoft.NET\Framework64\v2.0.50727\InstallUtilLib.dll"

Dim installer : Set installer = Wscript.CreateObject("WindowsInstaller.Installer")

Dim sqlQuery : sqlQuery = "SELECT `Name`, `Data` FROM Binary"

Dim database
Set database = installer.OpenDatabase(Wscript.Arguments(0), msiOpenDatabaseModeTransact)
Dim view : Set view = database.OpenView(sqlQuery)

Dim record : Set record = installer.CreateRecord(2)
record.StringData(1) = "InstallUtil"
view.Execute record

record.SetStream 2, INSTALL_UTIL_LIB_PATH

view.Modify msiViewModifyAssign, record
database.Commit

Set view = Nothing
Set database = Nothing

Next, I edited the Setup projects properties: I set the PostBuildEvent property to:

wscript.exe "$(ProjectDir)\..\Setup_PostBuildEvent_x64.vbs" $(BuiltOuputPath)

Note: Right-clicking the setup project in solution explorer and then selecting "Properties" opens up the wrong dialog ("Property Pages"). You want the "Properties Window" (CTRL+W, P).

Answer 1

Not sure how you want to automate this, through script, code, etc. But in any case, this functionality is all available through the Windows Installer SDK, which I believe is part of the Windows SDK now (used to be the Platform SDK).

Regardless, here is a VBScript I have used in the past to manually add a file to an MSI. It has been a while, but I just ran it on a MSI to test, and verified with Orca and the assembly was added to the binary table. This should point you in the right direction.

Option Explicit

Const msiOpenDatabaseModeTransact     = 1
Const msiViewModifyAssign         = 3

Dim installer : Set installer = Nothing
Set installer = Wscript.CreateObject("WindowsInstaller.Installer")

Dim sqlQuery : sqlQuery = "SELECT `Name`,`Data` FROM Binary"

Dim database : Set database = installer.OpenDatabase("YourInstallerFile.msi", msiOpenDatabaseModeTransact)
Dim view     : Set view = database.OpenView(sqlQuery)
Dim record

Set record = installer.CreateRecord(2)
record.StringData(1) = "InstallUtil"
view.Execute record

record.SetStream 2, "InstallUtilLib.dll"

view.Modify msiViewModifyAssign, record 
database.Commit 
Set view = Nothing
Set database = Nothing

Hope this helps!