I have been fighting a nasty memory problem on my subversion server. The server would run fine, then Ruby mongrel processes or httpd processes would crash at nearly random times. It took a few days to narrow it down. In my case it I am ruining RedMin, mod_dav and httpd but the issue can be reproduced without RedMine.
Occasionally httpd processes begin to consume significant memory, eventually crashing. Often occurs a few minutes to an hour after 3ish AM.
The root cause appears to be the Subversion mod_dav httpd module not properly handling an httpd reload (USR1 signal). This signal is sent during httpd logrotate which fires during cron.daily operations (approx 3 am). The problem will not manifest itself until subversion operations are performed against the server, with the ‘log’ operation with ‘xml’ output causing the crash most often. Other subversion operations, checkout, commit, etc. seem OK. This delay between the reload operation and the log operation can make it hard to pinpoint the cause.
Note: There may be other subversion commands that trigger this. Normal PHP, Ruby, HTML, etc. operations within httpd are fine. I found checkout, commit, etc. all worked fine but ‘svn log -xml’ would, on pass 1-3, consume all memory (384 MB) causing processes to be killed.
How to reproduce
Check memory consumption of your HTTPD tasks.
top usually works well. I run it in a separate session so I can easily see changes.
Issue a reload operation to Apache. This should cause HTTPD to release RAM and everything should be fine.
Then issue a log operation to you SVN server. You will need to change the URL and other settings to match your environment. The key is
log --xml. I made use of the subversion settings I configured for my web-server jobs. After this command you should see memory usage increase significantly. Each run will cause more memory to be consumed until processes begin to die. If this does not occur then you may not have this particular problem.
svn --config-dir /home/APACHE-JOB/.subversion log --xml https://REPO.EXAMPLE.com/svn/PROJECT
I need this working now so I used a restart in the logrotate instead of a reload. THis is not a production level fix, but that may await a fix in mod_dav.
You could of course send a signal directly but I choose to use a normal init.d operation:
File (per CentOS 5): /etc/logrotate.d/httpd
/bin/kill -USR1 `cat /var/run/httpd.pid 2>/dev/null` 2> /dev/null || true
/etc/init.d/httpd restart 2> /dev/null || TRUE
This issue could easily be caused by my choice of repositories and mix-n-match I had to do to get the capabilities I desired. Building each of these components myself is probably a good idea, but for now I lack the time and this solution is perfectly adequate for my personal subversion server. Plus, if I am going to build everything myself I will probably evaluate GIT which has features I would fine useful.
RedMine is not required to reproduce this issue.