my team is currently using buildbot to automate overnight and continuous-integration builds and regression tests. For builds and unit tests, the builder just invokes a script which syncs the sources from p4 and then runs xcodebuild. The regression tests are also launched by a shell script, and are themselves combinations of shell scripts and AppleScripts. The builds are all performed on one system but then buildbot triggers tests to happen on multiple Macs, with different architecture and OS combinations. The things I like about buildbot are the automatic triggers (so the tests run only if and when the build succeeds), and the reporting including the waterfall view to see the overall status.
However, I see a number of problems too. The “master” process which coordinates the builds is either leaky or just has a huge working set which means that it consumes a couple of gigabytes of memory. Occasionally network problems mean that a slave will be lost; rather than retrying later it will just fail the build. In fact if the slave is supposed to be triggered for a dependent build, it will fail the first build after it has otherwise successfully completed.
So what are the rest of you using to automate your Xcode builds and unit tests? How do those solutions work for you? Anything you would recommend?
I use TeamCity in a windows environment but I believe it will work on Mac and has xcode build agents.
Another similar question here.