Question

I am building a multiple binaries in C++ & CUDA with a couple of files in Fortran. I found this question and I’m having a similar problem. A user recently asked me to re-build a three year old version of the repository (before we had performed a massive migration and renaming) and I was shocked to see how quickly it built. It would be impossible / incredibly time consuming to determine exactly which of changes between that version and now caused the build to take so friggin’ long.

However, I noticed in an answer’s comment to the aforementioned question:

In particular, remember to use := instead of =, as := does the expansion
immediately, which saves time. – Jack Kelly Mar 23 at 22:38

Are there other suggestions that I should be aware of?

Note:

• I use the ‘include’ paradigm where each directory I want to build has a module.mk file which is directly included into the one and only Makefile.
• I do use several functions like:

(markdown..)

#
# CUDA Compilation Rules
#

define cuda-compile-rule
  $1: $(call generated-source,$2) \
    $(call source-dir-to-build-dir, $(subst .cu,.cubin, $2)) \
    $(call source-dir-to-build-dir, $(subst .cu,.ptx, $2))
    $(NVCC) $(CUBIN_ARCH_FLAG) $(NVCCFLAGS) $(INCFLAGS) $(DEFINES) -o $$@ -c $$<

  $(call source-dir-to-build-dir, $(subst .cu,.cubin, $2)): $(call generated-source,$2)
    $(NVCC) -cubin -Xptxas -v $(CUBIN_ARCH_FLAG) $(NVCCFLAGS) $(INCFLAGS) $(DEFINES) $(SMVERSIONFLAGS) -o $$@ $$<

  $(call source-dir-to-build-dir, $(subst .cu,.ptx, $2)): $(call generated-source,$2)
    $(NVCC) -ptx $(CUBIN_ARCH_FLAG) $(NVCCFLAGS) $(INCFLAGS) $(DEFINES) $(SMVERSIONFLAGS) -o $$@ $$<

  $(subst .o,.d,$1): $(call generated-source,$2)
    $(NVCC) $(CUBIN_ARCH_FLAG) $(NVCCFLAGS) $3 $(TARGET_ARCH) $(INCFLAGS) $(DEFINES) -M $$< | \
    $(SED) 's,\($$(notdir $$*)\.o\) *:,$$(dir $$@)\1 $$@: ,' > $$@.tmp
    $(MV) $$@.tmp $$@
endef

• But most of these functions I used in the older version…

Lastly: How can I determine if it’s the compilation time or the make time which is really slowing things down?

I didn’t want to append the entire Makefile. It’s 914 lines, but I’d be happy to update the question with snippets if it would help.

Update: Here is my dependency generation rule & compile rule:

#
# Dependency Generation Rules
#

define dependency-rules

  $(subst .o,.d,$1): $2
    $(CC) $(CFLAGS) $(DEFINES) $(INCFLAGS) $3 $(TARGET_ARCH) -M $$< | \
    $(SED) 's,\($$(notdir $$*)\.o\) *:,$$(dir $$@)\1 $$@: ,' > $$@.tmp
    $(MV) $$@.tmp $$@

endef

%.d: %.cpp
    $(CC) $(CFLAGS) $(CPPFLAGS) $(TARGET_ARCH) -M $< | \
    $(SED) 's,\($(notdir $*)\.o\) *:,$(dir $@)\1 $@: ,' > $@.tmp
    $(MV) $@.tmp $@

Update 2: Using @Beta’s suggestion, I was able to parcel out the dependency generation and Makefile time was roughly 14.2% of the overall compiling time. So I’m going focus on minimizing header inclusion in my C++ code first. Thanks to both of you for your suggestions!!

Answer 1

1. It shouldn’t be all that difficult to determine which changes slowed everything down. You have all the versions over the past three years (I hope), and you say the difference is dramatic. So try the version from two years ago. If it’s taking too long, do a binary search. You could even automate this process, have it run overnight and give you a graph in the morning, build time sampled each month for the past 36 months.
2. If you’re using GNUMake (as I hope), `make -n` will print out the commands it would execute, without actually executing them. This will give you all of the Make time with no compilation time.
3. One of the biggest sources of unnecessary build time (even bigger than recursion, which you aren’t using) is unnecessary rebuilding, recompiling/relinking/whatever when you don’t really need to. This can be because your makefile doesn’t handle dependencies correctly, or because your C++ files `#include` headers recklessly, or something about CUDA or FORTRAN that I wouldn’t know. Run Make twice in a row, and see if it does anything on the second pass. Look over the makefile for suspiciously huge prerequisite lists. Have a skillful programmer take a look at a few of the source files, especially the newer ones, and check for unnecessary dependencies.