apache freeze under high load (deadlock?)

[maklaut@…]

I'm was stress-testing php application. After 4 hour of test apache stop answering on any query without errors in logs. Size of php application ~1k php files. apache num: 30, queries per second: degrade from 50 to 30 queries per second while testing (i think it was not eAceelerator problem), system load average: from 20 to 10 while testing.

eAccelerator 0.9.5 [shm:mmap_anon sem:spinlock] PHP 4.4.4-8 [ZE 1.3.0] Using apache on Linux debian 2.6.8-3-686-smp #1 SMP Thu Sep 7 04:39:15 UTC 2006 i686

eaccelerator config:

eaccelerator.shm_size="128"
eaccelerator.cache_dir="/tmp/eaccelerator"
eaccelerator.enable="1"
eaccelerator.optimizer="0"
eaccelerator.check_mtime="1"
eaccelerator.debug="0"
eaccelerator.shm_max="0"
eaccelerator.shm_ttl="900"
eaccelerator.shm_prune_period="900"
eaccelerator.shm_only="1"
eaccelerator.compress="0"
eaccelerator.compress_level="1"
eaccelerator.log_file="/var/log/apache/eaccelerator.log"
eaccelerator.allowed_admin_path=/var
eaccelerator.sessions="shm_only"
session.save_handler=eaccelerator

backtrace log and core files download here  http://89.108.89.92/apache.log.tar.bz2 dump script

#!/bin/bash

apache_children=`ps fax | grep "/usr/sbin/apache" | grep -v grep | awk '{print $1;}'`

cat > /tmp/bt.gdb <<EOF
bt full
generate-core-file
q
EOF

for each in $apache_children; do
    sudo gdb -p $each -x /tmp/bt.gdb | tee "apache.$each.log";
done

it was deadlock in _mm_lock function?

[maklaut@…]

backtrace logs

[growler]

Spinlock debugging

[growler]

Please apply the attached patch (attachment:spinlock_debug.diff ). It will print warnings to stderr (should go to your PHP error log) on several flavours of breakage. It also wont block on a process trying to take a lock it already owns. NOTE: This makes it most definitely not thread safe. Do NOT use this patch if you're using mpm_worker (which you shouldn't be, anyway.)

Please keep an eye out in your logs for any of these messages, you might find that it no longer ends up deadlocked in which case I still want to know what warnings you saw.

Be warned that MAXSPINS and MAXYIELDS are guesstimates, and might cause interesting breakage.

[growler]

If it deadlocks again you should get a load of "eA: WARNING I've been spinning for a while trying to get a lock. My pid x, lock held by y." Could you get a core from the PID that it says is holding the lock.

[smo]

I think I've gotten hit by this a couple times with SVN revision 338.

[hans]

This is most likely fixed in rev @342. Also see ticket #232

[sim]

 decoration Changed 1 year ago by admin

 bathtub Changed 1 year ago by admin

 solar system Changed 1 year ago by admin

 stair parts Changed 1 year ago by admin

 solar supply Changed 1 year ago by admin

[bascorp]

 pop art pictures