Question

I’m trying to do speed up our build (csharp, msbuild, .net 3.5). Replace copy with fsutil hardlink create.

Previously, I almost got it with running a script over sln files and making the dll references private = false, then having a post build event create the hardlinks. Problem is transitive dependencies are not included. So, I think I need to reference the ResolveAssemblyReference task in msbuild to get the transitive dependencies I need to hardlink.

Any ideas?

This person tried the same thing, but did not post an final solution.

To be clear: What I want is to keep separate bin directories, but instead of copying file from one to another to create a hard link from a source (of a reference or a dependency) to the destination (the current project’s bin). Which is much faster, and gives approximately the same effect as copying.

Answer 1

This is supported in VS 2010. But not 2008. See the UseHardLinksIfPossible option to Copy in _CopyFilesMarkedCopyLocal.

See also http://social.msdn.microsoft.com/Forums/en/tfsbuild/thread/9382a3d8-4632-4826-ad15-d5e845080981, http://msdn.microsoft.com/en-us/library/ms171466(v=VS.90).aspx for context.

Override the _CopyFilesMarkedCopyLocal target. We add something like this to the csproj files at the bottom: <Import Project="..\..\..\..\..\\CommonBuild\TW.Override.Microsoft.Common.targets" /> (This is auto added to the file with a nant task on every full nant build, so every project benefits).

Our new targets file is:

<Project ToolsVersion="3.5" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
     <UsingTask TaskName="CopyWithHardlinkOption" AssemblyFile="..\lib\TWBuildOptimization\TW.Hardlinker.dll" />
  <!--
    ============================================================
                                        _CopyFilesMarkedCopyLocal
    Overridden in order to allow hardlinking with our custom Copy Task.                                         

    Hardlinking is a major performance improvement. Sometimes 50% of the time compared to copying.

    Copy references that are marked as "CopyLocal" and their dependencies, including .pdbs, .xmls and satellites.
    ============================================================
  -->
    <Target
        Name="_CopyFilesMarkedCopyLocal">
        <CopyWithHardlinkOption
            SourceFiles="@(ReferenceCopyLocalPaths)"
            DestinationFiles="@(ReferenceCopyLocalPaths->'$(OutDir)%(DestinationSubDirectory)%(Filename)%(Extension)')"
            SkipUnchangedFiles="true"
              UseHardlinksIfPossible="true"
            OverwriteReadOnlyFiles="$(OverwriteReadOnlyFiles)">
            <Output TaskParameter="DestinationFiles" ItemName="FileWritesShareable"/>
        </CopyWithHardlinkOption>
    </Target>
</Project>

See the msbuild 4 Copy UseHardlinksIfPossible option. I backported that to 3.5 through decompiling and reimplementing. The relevant logic in CopyFileWithLogging was:

      // The port from 4.0's task that allows hardlinking
        bool hardlinkSucceeded = false;
        if (UseHardlinksIfPossible)
        {
            if (File.Exists(destinationFile))
            {
                FileUtilities.DeleteNoThrow(destinationFile);
            }
            if (!TwNativeMethods.CreateHardLink(destinationFile, sourceFile, IntPtr.Zero))
            {
                var win32Exception = new Win32Exception(Marshal.GetLastWin32Error());
                Log.LogMessage(MessageImportance.High, "Hardlinking had a problem {0}, will retry copying. {1}", new object[] {win32Exception.Message, win32Exception});
            }
            hardlinkSucceeded = true;
        }
        if (!hardlinkSucceeded)
        {
            Log.LogMessageFromResources(MessageImportance.Normal, "Copy.FileComment", new object[] { sourceFile, destinationFile });
            Log.LogMessageFromResources(MessageImportance.Low, "Shared.ExecCommand", new object[0]);
            Log.LogCommandLine(MessageImportance.Low, "copy /y \"" + sourceFile + "\" \"" + destinationFile + "\"");
            File.Copy(sourceFile, destinationFile, true);
        }
        // end port

Also had to add these:

// decompiled from 4.0
internal static class TwNativeMethods
{
    [DllImport("kernel32.dll", CharSet=CharSet.Unicode, SetLastError=true)]
    internal static extern bool CreateHardLink(string newFileName, string exitingFileName, IntPtr securityAttributes);
}

// decompiled from 4.0
internal static class FileUtilities
{
    internal static void DeleteNoThrow(string path)
    {
        try
        {
            File.Delete(path);
        }
        catch (Exception exception)
        {
            if (ExceptionHandling.NotExpectedException(exception))
            {
                throw;
            }
        }
    }
}

// decompiled from 4.0
internal static class ExceptionHandling
{
    // Methods
    internal static bool IsCriticalException(Exception e)
    {
        return (((e is StackOverflowException) || (e is OutOfMemoryException)) || ((e is ExecutionEngineException) || (e is AccessViolationException)));
    }

    internal static bool NotExpectedException(Exception e)
    {
        return (((!(e is UnauthorizedAccessException) && !(e is ArgumentNullException)) && (!(e is PathTooLongException) && !(e is DirectoryNotFoundException))) && ((!(e is NotSupportedException) && !(e is ArgumentException)) && (!(e is SecurityException) && !(e is IOException))));
    }

    internal static bool NotExpectedReflectionException(Exception e)
    {
        return ((((!(e is TypeLoadException) && !(e is MethodAccessException)) && (!(e is MissingMethodException) && !(e is MemberAccessException))) && ((!(e is BadImageFormatException) && !(e is ReflectionTypeLoadException)) && (!(e is CustomAttributeFormatException) && !(e is TargetParameterCountException)))) && (((!(e is InvalidCastException) && !(e is AmbiguousMatchException)) && (!(e is InvalidFilterCriteriaException) && !(e is TargetException))) && (!(e is MissingFieldException) && NotExpectedException(e))));
    }
}