Ticket #224 (new defect)

Opened 2 years ago

Last modified 5 months ago

apache freeze under high load (deadlock?)

Reported by: maklaut@mail.ru Assigned to: somebody
Priority: critical Milestone:
Component: eAccelerator Version: 0.9.5
Keywords: Cc:

Description

I'm was stress-testing php application. After 4 hour of test apache stop answering on any query without errors in logs. Size of php application ~1k php files. apache num: 30, queries per second: degrade from 50 to 30 queries per second while testing (i think it was not eAceelerator problem), system load average: from 20 to 10 while testing.

eAccelerator 0.9.5 [shm:mmap_anon sem:spinlock] PHP 4.4.4-8 [ZE 1.3.0] Using apache on Linux debian 2.6.8-3-686-smp #1 SMP Thu Sep 7 04:39:15 UTC 2006 i686

eaccelerator config:

eaccelerator.shm_size="128"
eaccelerator.cache_dir="/tmp/eaccelerator"
eaccelerator.enable="1"
eaccelerator.optimizer="0"
eaccelerator.check_mtime="1"
eaccelerator.debug="0"
eaccelerator.shm_max="0"
eaccelerator.shm_ttl="900"
eaccelerator.shm_prune_period="900"
eaccelerator.shm_only="1"
eaccelerator.compress="0"
eaccelerator.compress_level="1"
eaccelerator.log_file="/var/log/apache/eaccelerator.log"
eaccelerator.allowed_admin_path=/var
eaccelerator.sessions="shm_only"
session.save_handler=eaccelerator

backtrace log and core files download here http://89.108.89.92/apache.log.tar.bz2 dump script

#!/bin/bash

apache_children=`ps fax | grep "/usr/sbin/apache" | grep -v grep | awk '{print $1;}'`

cat > /tmp/bt.gdb <<EOF
bt full
generate-core-file
q
EOF

for each in $apache_children; do
    sudo gdb -p $each -x /tmp/bt.gdb | tee "apache.$each.log";
done

it was deadlock in _mm_lock function?

Attachments

apache.log.tar.bz2 (11.8 kB) - added by maklaut@mail.ru on 12/09/06 11:38:42.
backtrace logs
spinlock_debug.diff (2.2 kB) - added by growler on 08/13/07 15:22:55.
Spinlock debugging

Change History

12/09/06 11:38:42 changed by maklaut@mail.ru

  • attachment apache.log.tar.bz2 added.

backtrace logs

12/09/06 11:40:09 changed by Maklaut

  • version set to 0.9.5.
  • component changed from Control panel to eAccelerator.

08/13/07 15:22:55 changed by growler

  • attachment spinlock_debug.diff added.

Spinlock debugging

08/13/07 15:26:59 changed by growler

Please apply the attached patch (attachment:spinlock_debug.diff). It will print warnings to stderr (should go to your PHP error log) on several flavours of breakage. It also wont block on a process trying to take a lock it already owns. NOTE: This makes it most definitely not thread safe. Do NOT use this patch if you're using mpm_worker (which you shouldn't be, anyway.)

Please keep an eye out in your logs for any of these messages, you might find that it no longer ends up deadlocked in which case I still want to know what warnings you saw.

Be warned that MAXSPINS and MAXYIELDS are guesstimates, and might cause interesting breakage.

08/13/07 15:40:58 changed by growler

If it deadlocks again you should get a load of "eA: WARNING I've been spinning for a while trying to get a lock. My pid x, lock held by y." Could you get a core from the PID that it says is holding the lock.

10/09/07 14:26:31 changed by smo

I think I've gotten hit by this a couple times with SVN revision 338.

04/09/08 14:33:35 changed by bladder

okay seems to work better. Charisma