Question

From what I have read being able to use pump mode with distcc requires that you encapsulate make in the pump script. However, I do not have it in my path and I can not find not find it as a package or included in the distcc package for Cygwin.

However, when I compile with distcc and use distccmon-text to monitor which hosts are contacted and their phase I clearly see that some of them, sometimes, are in the Preprocess phase. I thought all preprocessing was done on the client executing the make script when not using pump mode. And that the whole idea of pump mode was preprocessing on the remote hosts (and thus requireing the same include files).

This has left me confused. My main question is: Exactly what does the phases: Startup, Blocked, Connected, Preprocess, Conect, Send, Receive and Done of distcc mean?

And as a sub-question: How can I use pump mode with distcc in Cygwin?

Answer 1

Exactly what does the phases of distcc
mean?

Ok, this is embarrassing, but I just wasted four hours on the web trying to answer that question. Next time I’ll just pull the source and look at it. But, you raise a good point: It’s amazing this isn’t readily available information.

THESE ARE MY GUESSES (because I don’t want to admit I wasted four hours not answering the question!):

• Startup – could otherwise be called
  “initialization/loading”, not yet
  ready for first task
• Blocked – is awaiting access to local
  file or local processor, I stumbled
  across recent bug fixes that set “timeout” to one second while it waited for the
  processor to become available, and I’m aware that it uses zero-length “flock” files to block at times
• Connected – process initiated contact
  with a client, is now reserved for a
  job (??), or is compiling a job (??)
• Preprocess – is performing preprocess
  operation
• Conect – is hand-shaking with client for atomic operation, maybe to become reserved
  (??)
• Send – is sending compiled object
  file back to client
• Receive – is receiving source to be
  compiled, or receiving zipped headers
  (if using pump-mode)
• Done – could otherwise be called
  “idle/available”

NOTE: Because of Google’s “pump-mode” algorithm, there’s actually quite a bit of “hand-shaking” that goes on between the client (running distcc) and volunteer (running distccd). First, in pump-mode, all the headers that are expected to be “needed” are bundled, zipped, and pushed to the volunteer (where it is unzipped into a local mirror like that on the client machine). However, it appears that some further communication between the volunteer-and-client is possible to incrementally transfer other headers as-needed, so that would explain the “more-rich” communications phases/states listed above.

Am I already using pump mode?

I very much doubt it, as you did not configure it by wrapping the compile option through make or scons (necessary to run the Google algorithm to predict header usage for bundling-and-transport to the volunteer), nor can you find the “pump” script. But, I cannot explain your seeing the “Preprocess” states on your volunteers. (You’re not referring to the “Preprocess” state on your clients, right? That would be entirely understandable, as by default preprocessing is on the client.)

Rather, I suppose the implementation makes it possible that the hard-state-machine would move through ALL the states, INCLUDING “preprocessing”, even when there is no preprocessing to do, before it advances to the next state. For example, even if it did no preprocessing on the volunteer side, distccd would receive the source file, write it to disk, and then launch the compiler. If you’re on Cywin, those are not instantaneous, especially if it is a large source file (especially after all the headers included in it). So, you might see the “preprocess” phase until it manually initiates the next phase for the compile operation itself.

HEY … I don’t see an obvious “compile” phase, so it is POSSIBLE that the “preprocess” phase embodies “compile” or “preprocess-and-compile” (since those phases are often combined in many compilers anyway).

Sorry — I’m just guessing.

And how do I use pump mode in Cygwin?

I haven’t tried it, but it is supposed to be possible. Apparently the most common problem with Cygwin is that some Windows compilers cannot handle the default TMPDIR setting when distcc is run under Cygwin. The fix is to put something like export TMPDIR=c:/temp in /etc/profile

The FAQ may be able to help more: http://distcc.googlecode.com/svn/trunk/doc/web/faq.html