Monit restarting Mongrel Cluster - “Execution Failed”
Following in the footsteps of numerous other hapless people seeing this error - I had a frustrating number of hours, trying every combination of stuff that *should* work according to various recipes.
The following is what I had to do to get monit playing nice with mongrel cluster (and thereby avoiding the lovely error “execution failed” when trying to get mongrel restarted via monit)
This is on a CentOS 5 Enterprise x86_64 server. For some reason calling the cluster::start option on mongrel_rails would not work, I had to pass in all the flags manually to mongrel_rails, along with absolute paths for everything. But hey it works … you can manually kill a mongrel process and watch monit bring it back up for you, right as rain !!!
(note next step I’d change the monitrc to actually test a response from the mongrel server in question, rather than rely on the pid file)
Setup:
- mongrel user, in mongrel group.
- mongrel owns /var/www/apps/myappname/
- monit is run as root user
- mongrel user doesn’t have sudo privileges
in /etc/monitrc
check process mongrel_8000
with pidfile /var/www/apps/myappname/shared/pids/mongrel.8000.pid
start program = “/usr/local/bin/ruby /usr/local/bin/mongrel_rails start -d -e production -c /var/www/apps/myappname/current/ –user mongrel –group mongrel -p 8000 -P /var/www/apps/myappname/shared/pids/mongrel.8000.pid -l /var/www/apps/myappname/shared/log/mongrel.8000.log”
[stop program, rules etc following]
[Rinse and repeat for each mongrel you're running in the cluster]
Note that this will not work…
start program = “/usr/local/bin/mongrel_rails cluster::start -C /var/www/apps/myappname/current/config/mongrel_cluster.yml –clean –only 8000″

11 Comments, Comment or Ping
Marcus Derencius
I had the same problem with monit and mongrel_cluster, but I gave up and went with god.
Feb 7th, 2008
Nate Carlson
Sigh, odd - your commands above don’t work for me either.
Feb 13th, 2008
admin
Nate - once your process is running that you want to monitor, do a ps wwaux grep | processname, it should list out everything that was called to start the process, copy & paste the entire lot, along with changing everything to absolute paths, and use that for your start program command - hopefully this helps. I also found out that monit uses a very limited environment path setting (something like /usr/bin) so you may also want to prepend your path in if you need to. I believe all of the problems stem from this.
Feb 13th, 2008
Frank
The only way that I’ve been able to get monit and mongrel_rails to play nice together is by adding the following line to /usr/local/bin/mongrel_rails on my system:
ENV['PATH'] = “#{ENV['PATH']}:/usr/local/bin”;
This isn’t an ideal solution because it will break when the mongrel_cluster gem will be upgraded.
Prior to this solution I tried specifying full paths in the monit start/stop program sections and also using /usr/bin/env to set the correct PATH environment variable directly in the monit start/stop section but neither solution worked for some reason.
Mar 14th, 2008
Alex T
I managed to make it work with the following command:
start program = “/usr/bin/env PATH=/opt/local/bin:$PATH mongrel_rails cluster::start -C /path/to/mongrel.conf –clean –only ”
Obviously change /opt/local/bin for whatever your system uses.
Apr 16th, 2008
Alex T
Grrr… Forum ate my port number… That should have been:
start program = “/usr/bin/env PATH=/opt/local/bin:$PATH mongrel_rails cluster::start -C /path/to/mongrel.conf –clean –only 8110”
Apr 17th, 2008
Aaron C
Thanks, Alex. Setting path is this manner did just the trick.
Apr 19th, 2008
Amit Solanki
Exactly the post I was looking for, though I had to change the paths similar to:
start program = “/bin/env PATH=/usr/local/bin:$PATH mongrel_rails cluster::start -C /var/www/myproject/current/config/mongrel_cluster.yml –clean –only 8000″
stop program = “/bin/env PATH=/usr/local/bin:$PATH mongrel_rails cluster::stop -C /var/www/myproject/current/config/mongrel_cluster.yml -f –clean –only 8000″
The only problem with above is whenever any of the port gets down, it get automatically gets started by monit, but in cases where it requires to be restarted, the stop command fails. I could not figure out why the stop command fails, despite the fact that the paths are same for start/stop programs and those commands are working fine on terminal. The log file also do not provide sufficient information for the cause of failure. Can anyone help me with this?
Apr 30th, 2008
Fabrizio Vecchi
Thanks for the guide.
I tried all the above, but unfortunately my mongrels still don’t restart.
When I restart monit, it detects that one mongrel is down, but once it runs the ’start program’ command, here is what I get:
—
May 12 17:08:32 webserver[11034]: ‘mongrel_8000′ start: /usr/bin/mongrel_rails
—
and then I get the usual failed start command…
Any ideas on how to fix this?
Thanks a lot,
Fabrizio.
May 12th, 2008
james borenstein
I’m battling with Monit on Debian Etch and need some help. I’ve
included the mongrel.monitrc file I’m using. If you issue the start
program and stop program commands shown in that file from the command
line, they work fine, regardless of what directory you issue them
from. Interestingly, calling “monit status” does show that the
test3_8001 process is being monitored and shows the correct statistics
for uptime, etc. However, when you run “monit restart test3_8001″
there is a failure. “monit status” eventually shows “Execution
failed” as a status for test3_8001, but does continue to show all the
other correct stats as before. Checking the monit_errors.log shows
only the following:
488 [PDT Jun 6 20:56:22] info : ‘mongrel_test3_8001′ trying to
restart
489 [PDT Jun 6 20:56:22] info : ‘mongrel_test3_8001′ start: /usr/
bin/env
490 [PDT Jun 6 20:56:23] error : ‘mongrel_test3_8001′ failed to
stop
Other notes:
- Paths
—> I tried just having the /usr/local/lib/site_ruby/gems/bin/
mongrel_rails cluster::start -C /home/jay/test/testing/config/
mongrel_cluster.yml –clean –only 8001″ as a path
—> I also tried adding the path to the ruby executable which is /usr/
local/bin/ruby on my machine prior to the above.
the mongrel.monitrc file:
check process test3_8001
with pidfile /home/jay/test/testing/tmp/pids/mongrel.8001.pid
start program = “/usr/bin/env PATH=$PATH:/usr/local/lib/site_ruby/
gems/bin /usr/local/lib/site_ruby/gems/bin/mongrel_rails
cluster::start -C /home/jay/test/testing/config/mongrel_cluster.yml –
clean –only 8001″
stop program = “//usr/bin/env PATH=$PATH:/usr/local/lib/site_ruby/
gems/bin /usr/local/lib/site_ruby/gems/bin/mongrel_rails cluster::stop
-C /home/jay/test/testing/config/mongrel_cluster.yml –only 8001″
# if cpu > 60% for 2 cycles then alert
if cpu > 80% for 4 cycles then restart
if totalmem > 110.0 MB for 4 cycles then restart
# if children > 250 then restart
# if loadavg(5min) greater than 10 for 8 cycles then stop
# if failed host http://www.tildeslash.com port 80 protocol http
# and request “/monit/doc/next.php”
# then restart
# if failed port 443 type tcpssl protocol http
# with timeout 15 seconds
# then restart
if 20 restarts within 20 cycles then timeout
# depends on apache_bin
group test_mongrels
Thanks for any thoughts - I’m at a loss…
Jun 7th, 2008
james borenstein
The answer for me at long last was to set GEM_HOME in the mongrel.monitrc file. So
start program = “env GEM_HOME= /usr/local/bin/mongrel_rails cluster::start -C …
Jun 7th, 2008
Reply to “Monit restarting Mongrel Cluster - “Execution Failed””