Monit restarting Mongrel Cluster - “Execution Failed”

Following in the footsteps of numerous other hapless people seeing this error - I had a frustrating number of hours, trying every combination of stuff that *should* work according to various recipes.

The following is what I had to do to get monit playing nice with mongrel cluster (and thereby avoiding the lovely error “execution failed” when trying to get mongrel restarted via monit)

This is on a CentOS 5 Enterprise x86_64 server. For some reason calling the cluster::start option on mongrel_rails would not work, I had to pass in all the flags manually to mongrel_rails, along with absolute paths for everything. But hey it works … you can manually kill a mongrel process and watch monit bring it back up for you, right as rain !!!

(note next step I’d change the monitrc to actually test a response from the mongrel server in question, rather than rely on the pid file)

Setup:
- mongrel user, in mongrel group.
- mongrel owns /var/www/apps/myappname/
- monit is run as root user
- mongrel user doesn’t have sudo privileges

in /etc/monitrc

check process mongrel_8000
with pidfile /var/www/apps/myappname/shared/pids/mongrel.8000.pid
start program = “/usr/local/bin/ruby /usr/local/bin/mongrel_rails start -d -e production -c /var/www/apps/myappname/current/ –user mongrel –group mongrel -p 8000 -P /var/www/apps/myappname/shared/pids/mongrel.8000.pid -l /var/www/apps/myappname/shared/log/mongrel.8000.log”
[stop program, rules etc following]

[Rinse and repeat for each mongrel you're running in the cluster]

Note that this will not work…
start program = “/usr/local/bin/mongrel_rails cluster::start -C /var/www/apps/myappname/current/config/mongrel_cluster.yml –clean –only 8000″

11 Comments, Comment or Ping

  1. I had the same problem with monit and mongrel_cluster, but I gave up and went with god.

  2. Sigh, odd - your commands above don’t work for me either. ;)

  3. admin

    Nate - once your process is running that you want to monitor, do a ps wwaux grep | processname, it should list out everything that was called to start the process, copy & paste the entire lot, along with changing everything to absolute paths, and use that for your start program command - hopefully this helps. I also found out that monit uses a very limited environment path setting (something like /usr/bin) so you may also want to prepend your path in if you need to. I believe all of the problems stem from this.

  4. Frank

    The only way that I’ve been able to get monit and mongrel_rails to play nice together is by adding the following line to /usr/local/bin/mongrel_rails on my system:

    ENV['PATH'] = “#{ENV['PATH']}:/usr/local/bin”;

    This isn’t an ideal solution because it will break when the mongrel_cluster gem will be upgraded.

    Prior to this solution I tried specifying full paths in the monit start/stop program sections and also using /usr/bin/env to set the correct PATH environment variable directly in the monit start/stop section but neither solution worked for some reason.

  5. I managed to make it work with the following command:

    start program = “/usr/bin/env PATH=/opt/local/bin:$PATH mongrel_rails cluster::start -C /path/to/mongrel.conf –clean –only ”

    Obviously change /opt/local/bin for whatever your system uses.

  6. Grrr… Forum ate my port number… That should have been:

    start program = “/usr/bin/env PATH=/opt/local/bin:$PATH mongrel_rails cluster::start -C /path/to/mongrel.conf –clean –only 8110”

  7. Aaron C

    Thanks, Alex. Setting path is this manner did just the trick.

  8. Exactly the post I was looking for, though I had to change the paths similar to:

    start program = “/bin/env PATH=/usr/local/bin:$PATH mongrel_rails cluster::start -C /var/www/myproject/current/config/mongrel_cluster.yml –clean –only 8000″

    stop program = “/bin/env PATH=/usr/local/bin:$PATH mongrel_rails cluster::stop -C /var/www/myproject/current/config/mongrel_cluster.yml -f –clean –only 8000″

    The only problem with above is whenever any of the port gets down, it get automatically gets started by monit, but in cases where it requires to be restarted, the stop command fails. I could not figure out why the stop command fails, despite the fact that the paths are same for start/stop programs and those commands are working fine on terminal. The log file also do not provide sufficient information for the cause of failure. Can anyone help me with this?

  9. Fabrizio Vecchi

    Thanks for the guide.
    I tried all the above, but unfortunately my mongrels still don’t restart.
    When I restart monit, it detects that one mongrel is down, but once it runs the ’start program’ command, here is what I get:

    May 12 17:08:32 webserver[11034]: ‘mongrel_8000′ start: /usr/bin/mongrel_rails

    and then I get the usual failed start command…

    Any ideas on how to fix this?

    Thanks a lot,
    Fabrizio.

  10. james borenstein

    I’m battling with Monit on Debian Etch and need some help. I’ve
    included the mongrel.monitrc file I’m using. If you issue the start
    program and stop program commands shown in that file from the command
    line, they work fine, regardless of what directory you issue them
    from. Interestingly, calling “monit status” does show that the
    test3_8001 process is being monitored and shows the correct statistics
    for uptime, etc. However, when you run “monit restart test3_8001″
    there is a failure. “monit status” eventually shows “Execution
    failed” as a status for test3_8001, but does continue to show all the
    other correct stats as before. Checking the monit_errors.log shows
    only the following:
    488 [PDT Jun 6 20:56:22] info : ‘mongrel_test3_8001′ trying to
    restart
    489 [PDT Jun 6 20:56:22] info : ‘mongrel_test3_8001′ start: /usr/
    bin/env
    490 [PDT Jun 6 20:56:23] error : ‘mongrel_test3_8001′ failed to
    stop

    Other notes:
    - Paths
    —> I tried just having the /usr/local/lib/site_ruby/gems/bin/
    mongrel_rails cluster::start -C /home/jay/test/testing/config/
    mongrel_cluster.yml –clean –only 8001″ as a path
    —> I also tried adding the path to the ruby executable which is /usr/
    local/bin/ruby on my machine prior to the above.

    the mongrel.monitrc file:

    check process test3_8001
    with pidfile /home/jay/test/testing/tmp/pids/mongrel.8001.pid
    start program = “/usr/bin/env PATH=$PATH:/usr/local/lib/site_ruby/
    gems/bin /usr/local/lib/site_ruby/gems/bin/mongrel_rails
    cluster::start -C /home/jay/test/testing/config/mongrel_cluster.yml –
    clean –only 8001″
    stop program = “//usr/bin/env PATH=$PATH:/usr/local/lib/site_ruby/
    gems/bin /usr/local/lib/site_ruby/gems/bin/mongrel_rails cluster::stop
    -C /home/jay/test/testing/config/mongrel_cluster.yml –only 8001″
    # if cpu > 60% for 2 cycles then alert
    if cpu > 80% for 4 cycles then restart
    if totalmem > 110.0 MB for 4 cycles then restart
    # if children > 250 then restart
    # if loadavg(5min) greater than 10 for 8 cycles then stop
    # if failed host http://www.tildeslash.com port 80 protocol http
    # and request “/monit/doc/next.php”
    # then restart
    # if failed port 443 type tcpssl protocol http
    # with timeout 15 seconds
    # then restart
    if 20 restarts within 20 cycles then timeout
    # depends on apache_bin
    group test_mongrels

    Thanks for any thoughts - I’m at a loss…

  11. james borenstein

    The answer for me at long last was to set GEM_HOME in the mongrel.monitrc file. So
    start program = “env GEM_HOME= /usr/local/bin/mongrel_rails cluster::start -C …

Reply to “Monit restarting Mongrel Cluster - “Execution Failed””

About

Rowan is a Product Development Manager, specialising in architecting, developing and putting web applications into production - in particular Ruby on Rails based apps. He lives in Toronto, Canada but speaks in a funny accent as he's originally from New Zealand. He's been working in the software and web business for over a decade. This blog covers Web Application development and deployment in the real world, dealing with topics from business fundamentals to Ruby on Rails, Merb, PHP, Flex, MySQL, Apache and more.

Read more ...

 

 

View Rowan Hick's profile on LinkedIn

 

Subscribe to my RSS feed