Ticket #224 (closed defect: fixed)
apache freeze under high load (deadlock?)
| Reported by: | maklaut@… | Owned by: | somebody |
|---|---|---|---|
| Priority: | critical | Milestone: | |
| Component: | eAccelerator | Version: | 0.9.5 |
| Keywords: | Cc: |
Description
I'm was stress-testing php application. After 4 hour of test apache stop answering on any query without errors in logs. Size of php application ~1k php files. apache num: 30, queries per second: degrade from 50 to 30 queries per second while testing (i think it was not eAceelerator problem), system load average: from 20 to 10 while testing.
eAccelerator 0.9.5 [shm:mmap_anon sem:spinlock] PHP 4.4.4-8 [ZE 1.3.0] Using apache on Linux debian 2.6.8-3-686-smp #1 SMP Thu Sep 7 04:39:15 UTC 2006 i686
eaccelerator config:
eaccelerator.shm_size="128" eaccelerator.cache_dir="/tmp/eaccelerator" eaccelerator.enable="1" eaccelerator.optimizer="0" eaccelerator.check_mtime="1" eaccelerator.debug="0" eaccelerator.shm_max="0" eaccelerator.shm_ttl="900" eaccelerator.shm_prune_period="900" eaccelerator.shm_only="1" eaccelerator.compress="0" eaccelerator.compress_level="1" eaccelerator.log_file="/var/log/apache/eaccelerator.log" eaccelerator.allowed_admin_path=/var eaccelerator.sessions="shm_only" session.save_handler=eaccelerator
backtrace log and core files download here http://89.108.89.92/apache.log.tar.bz2 dump script
#!/bin/bash
apache_children=`ps fax | grep "/usr/sbin/apache" | grep -v grep | awk '{print $1;}'`
cat > /tmp/bt.gdb <<EOF
bt full
generate-core-file
q
EOF
for each in $apache_children; do
sudo gdb -p $each -x /tmp/bt.gdb | tee "apache.$each.log";
done
it was deadlock in _mm_lock function?
Attachments
Change History
Changed 4 years ago by maklaut@…
-
attachment
apache.log.tar.bz2
added
comment:1 Changed 4 years ago by Maklaut
- Version set to 0.9.5
- Component changed from Control panel to eAccelerator
comment:2 Changed 3 years ago by growler
Please apply the attached patch (attachment:spinlock_debug.diff). It will print warnings to stderr (should go to your PHP error log) on several flavours of breakage. It also wont block on a process trying to take a lock it already owns. NOTE: This makes it most definitely not thread safe. Do NOT use this patch if you're using mpm_worker (which you shouldn't be, anyway.)
Please keep an eye out in your logs for any of these messages, you might find that it no longer ends up deadlocked in which case I still want to know what warnings you saw.
Be warned that MAXSPINS and MAXYIELDS are guesstimates, and might cause interesting breakage.
comment:3 Changed 3 years ago by growler
If it deadlocks again you should get a load of "eA: WARNING I've been spinning for a while trying to get a lock. My pid x, lock held by y." Could you get a core from the PID that it says is holding the lock.
comment:4 Changed 3 years ago by smo
I think I've gotten hit by this a couple times with SVN revision 338.
comment:5 Changed 10 months ago by hans
- Status changed from new to closed
- Resolution set to fixed
This is most likely fixed in rev @342. Also see ticket #232
comment:6 Changed 6 months ago by sim
decoration Changed 1 year ago by admin
bathtub Changed 1 year ago by admin
solar system Changed 1 year ago by admin
stair parts Changed 1 year ago by admin
solar supply Changed 1 year ago by admin
backtrace logs