Here is the graph for one of the projects:
The build initially took about 30 minutes, then I had a period where I jumped to about 70 minutes. This lag was entirely my unit-test and after some refactorings I took is down again to about 30 minutes, which is decent.
Now take a look at another project:
The difference is that the project took all long about 40 minutes, and now I have spikes of 3-4 hours !
What's the catch: well, the first project is taking so long during unit tests since I have a lot of GUI tests, where the code has to sleep and give the interface time to repaint, etc. So, although the time is 30 minutes, it's mostly waiting for the GUI (inside an Xvnc instance) to paint.
The second project though does a massive build where I just produce JARs and don't run any unit tests. It's massively IO-bound.
So, the way I see it, in the past 2 months or so the machine I'm running my VPS on, has been getting slower doing IO request.
IO has always been a problem with my limited VPS experience. First, I got rid of this by moving from the 256mb slice to the 512mb slice since apparently I was just trashing the swap file.
But now, I'm not so certain it's a RAM issue anymore. The 2nd compilation just needs to touch the disk so it doesn't matter how much RAM do I add, after the minimal amount needed for ant and javac.
I'm starting to think I should move the build server onto an EC2 instance. This way I could use a smaller slice just to run Hudson, but do the actual building onto a bigger EC2 instance. I'm not certain it will be cheaper though.
Later edit: The discussion here is continued with my post about using EC2 instances as build servers.
