|
故障现象 使用TinkPHP3.2.x框架，页面偶尔会出现一访问nginx就报502 bad gateway，并不是等一段时间后nginx才报502，打开页面的一瞬间就502了。 php-fpm日志
[28-Sep-2015 23:25:07] WARNING: [pool www] child 2965 exited on signal 7 (SIGBUS) after 319.686109 seconds from start [28-Sep-2015 23:25:07] NOTICE: [pool www] child 3223 started [28-Sep-2015 23:25:49] WARNING: [pool www] child 1251 exited on signal 7 (SIGBUS) after 1005.893950 seconds from start [28-Sep-2015 23:25:49] NOTICE: [pool www] child 3242 started [28-Sep-2015 23:26:46] WARNING: [pool www] child 1467 exited on signal 7 (SIGBUS) after 1057.791921 seconds from start [28-Sep-2015 23:26:46] NOTICE: [pool www] child 3284 started [28-Sep-2015 23:30:33] WARNING: [pool www] child 1178 exited on signal 7 (SIGBUS) after 1289.872260 seconds from start [28-Sep-2015 23:30:33] NOTICE: [pool www] child 3450 started [28-Sep-2015 23:30:35] WARNING: [pool www] child 1115 exited on signal 7 (SIGBUS) after 1292.014337 seconds from start [28-Sep-2015 23:30:35] NOTICE: [pool www] child 3451 started [28-Sep-2015 23:30:35] WARNING: [pool www] child 1487 exited on signal 7 (SIGBUS) after 1285.395573 seconds from start [28-Sep-2015 23:30:35] NOTICE: [pool www] child 3452 started [28-Sep-2015 23:30:36] WARNING: [pool www] child 1958 exited on signal 7 (SIGBUS) after 1181.206336 seconds from start [28-Sep-2015 23:30:36] NOTICE: [pool www] child 3453 started 对php-fpm进行调试 既然看不出来头绪就需要进行调试了，这里需要把php-fpm退出的时候导出coredump php-fpm coredump方法 核心文件，也称核心转储，是操作系统在进程收到某些信号而终止运行时，将此时进程地址空间的内容以及有关进程状态的其他信息写出的一个磁盘文件。这种信息往往用于调试。 核心文件一词来源于磁芯内存。 php需要打开debug参数，如果编译的时候没有打开，需要重新编译，以下是我这里的 --enable-debug echo "/tmp/coredump/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern ulimit -c unlimited ulimit -c 0 gdb /usr/local/php/sbin/php-fpm core.php-fpm.4555.web1.iamle.com.1443458079 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-83.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/local/php/sbin/php-fpm...done. [New Thread 4555] Missing separate debuginfo for /usr/lib/libmcrypt.so.4 Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/c5/699ec0c783a6055a7f983f9eb64146371eb2e6 Missing separate debuginfo for /usr/lib/libpng12.so.0 Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9e/65badd636c1f2e81682a8c498af098b5bbd063 Missing separate debuginfo for /usr/lib/libjpeg.so.62 Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/05/6b2c6a8cb0b42dda3d57a1b5d3670ab99c83bd Missing separate debuginfo for /usr/local/lib/libiconv.so.2 Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/df/7400febafa74aeef81e48b638c9fde24cb2beb Missing separate debuginfo for /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/redis.so Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/64/1aa8440943921a4d553e380282d15828f18377 Missing separate debuginfo for Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/fa/36da47e774d9bc5a739905887a5d25790090f7 Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /usr/lib/libmcrypt.so.4...done. Loaded symbols for /usr/lib/libmcrypt.so.4 Reading symbols from /usr/lib/libpng12.so.0...Missing separate debuginfo for /usr/lib/libpng12.so.0 Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/9e/65badd636c1f2e81682a8c498af098b5bbd063.debug (no debugging symbols found)...done. Loaded symbols for /usr/lib/libpng12.so.0 Reading symbols from /usr/lib/libjpeg.so.62...Missing separate debuginfo for /usr/lib/libjpeg.so.62 Try: yum --enablerepo='*-debug*' install /usr/lib/debug/.build-id/05/6b2c6a8cb0b42dda3d57a1b5d3670ab99c83bd.debug (no debugging symbols found)...done. Loaded symbols for /usr/lib/libjpeg.so.62 Reading symbols from /usr/lib64/libcurl.so.4...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libcurl.so.4 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libnsl.so.1 Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libxml2.so.2 Reading symbols from /usr/lib64/libssl.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssl.so.10 Reading symbols from /usr/lib64/libcrypto.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libcrypto.so.10 Reading symbols from /usr/lib64/libfreetype.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libfreetype.so.6 Reading symbols from /usr/local/lib/libiconv.so.2...done. Loaded symbols for /usr/local/lib/libiconv.so.2 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /usr/lib64/libfreebl3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libfreebl3.so Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libidn.so.11...(no debugging symbols found)...done. Loaded symbols for /lib64/libidn.so.11 Reading symbols from /lib64/libldap-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libldap-2.4.so.2 Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libgssapi_krb5.so.2 Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5.so.3 Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /lib64/libk5crypto.so.3 Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssl3.so Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libsmime3.so Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libnss3.so Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libnssutil3.so Reading symbols from /lib64/libplds4.so...(no debugging symbols found)...done. Loaded symbols for /lib64/libplds4.so Reading symbols from /lib64/libplc4.so...(no debugging symbols found)...done. Loaded symbols for /lib64/libplc4.so Reading symbols from /lib64/libnspr4.so...(no debugging symbols found)...done. Loaded symbols for /lib64/libnspr4.so Reading symbols from /usr/lib64/libssh2.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libssh2.so.1 Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/liblber-2.4.so.2 Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libsasl2.so.2 Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/redis.so...done. Loaded symbols for /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/redis.so Reading symbols from /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/networkbench.so...done. Loaded symbols for /usr/local/php/lib/php/extensions/no-debug-non-zts-20100525/networkbench.so Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 Core was generated by `php-fpm: pool www '. Program terminated with signal 7, Bus error. #0 lex_scan (zendlval=0x7fff8083fae8) at Zend/zend_language_scanner.c:2265 2265 switch (yych) { Missing separate debuginfos, use: debuginfo-install cyrus-sasl-lib-2.1.23-15.el6.x86_64 freetype-2.3.11-14.el6_3.1.x86_64 glibc-2.12-1.149.el6_6.5.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-33.el6.x86_64 libcom_err-1.41.12-21.el6.x86_64 libcurl-7.19.7-40.el6_6.1.x86_64 libidn-1.18-2.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libssh2-1.4.2-1.el6.x86_64 libxml2-2.7.6-17.el6_6.1.x86_64 nspr-4.10.6-1.el6_5.x86_64 nss-3.16.1-14.el6.x86_64 nss-softokn-freebl-3.14.3-17.el6.x86_64 nss-util-3.16.1-3.el6.x86_64 openldap-2.4.39-8.el6.x86_64 openssl-1.0.1e-30.el6_6.4.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 lex_scan (zendlval=0x7fff8083fae8) at Zend/zend_language_scanner.c:1088 #1 0x00000000007ad730 in zendlex (zendlval=0x7fff8083fae0) at /root/oneshell-master/php-5.4.45/Zend/zend_compile.c:6545 #2 0x0000000000795aca in zendparse () at /root/oneshell-master/php-5.4.45/Zend/zend_language_parser.c:3471 #3 0x00000000007a2990 in compile_file (file_handle=0x7fff8083fe30, type=2) at Zend/zend_language_scanner.l:585 #4 0x000000000063255a in phar_compile_file (file_handle=0x7fff8083fe30, type=2) at /root/oneshell-master/php-5.4.45/ext/phar/phar.c:3414 #5 0x00000000007a20bb in compile_filename (type=2, filename=0x299d018) at Zend/zend_language_scanner.l:628 #6 0x00000000008156de in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER (execute_data=0x7f46209c6158) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:27443 #7 0x0000000000832c80 in execute (op_array=0x27b4158) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410 #8 0x00007f4614b27db6 in nb_execute (op_array=0x27b4158) at /tingyun/nbprof/nbprof.c:318 #9 0x00000000007c0e14 in zend_call_function (fci=0x7fff808401b0, fci_cache=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_execute_API.c:956 #10 0x00000000006d8ae5 in zif_call_user_func_array (ht=<value optimized out>, return_value=0x29733b0, return_value_ptr=<value optimized out>, this_ptr=<value optimized out>, return_value_used=<value optimized out>) at /root/oneshell-master/php-5.4.45/ext/standard/basic_functions.c:4754 #11 0x00007f4614b27cbe in nb_execute_internal (current_execute_data=0x7f46209c5f78, return_value_used=1) at /tingyun/nbprof/nbprof.c:403 #12 0x000000000083e643 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:645 #13 0x0000000000832c80 in execute (op_array=0x7f46209f8a70) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410 #14 0x00007f4614b27db6 in nb_execute (op_array=0x7f46209f8a70) at /tingyun/nbprof/nbprof.c:318 #15 0x00000000007c0e14 in zend_call_function (fci=0x7fff80840540, fci_cache=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_execute_API.c:956 #16 0x00000000007e41bf in zend_call_method (object_pp=0x0, obj_ce=<value optimized out>, fn_proxy=0x7f46209f6618, function_name=0xb02423 "__callstatic", function_name_len=<value optimized out>, retval_ptr_ptr=0x7fff80840668, param_count=2, arg1=0x29730a8, arg2=0x29735c0) at /root/oneshell-master/php-5.4.45/Zend/zend_interfaces.c:97 #17 0x00000000007efb39 in zend_std_callstatic_user_call (ht=<value optimized out>, return_value=0x2972f58, return_value_ptr=<value optimized out>, this_ptr=<value optimized out>, return_value_used=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_object_handlers.c:1110 #18 0x00007f4614b27cbe in nb_execute_internal (current_execute_data=0x7f46209c5d88, return_value_used=0) at /tingyun/nbprof/nbprof.c:403 #19 0x000000000083e643 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:645 #20 0x0000000000832c80 in execute (op_array=0x304d188) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410 #21 0x00007f4614b27db6 in nb_execute (op_array=0x304d188) at /tingyun/nbprof/nbprof.c:318 #22 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673 #23 0x0000000000832c80 in execute (op_array=0x2831020) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410 #24 0x00007f4614b27db6 in nb_execute (op_array=0x2831020) at /tingyun/nbprof/nbprof.c:318 #25 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673 #26 0x0000000000832c80 in execute (op_array=0x27a48a0) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410 #27 0x00007f4614b27db6 in nb_execute (op_array=0x27a48a0) at /tingyun/nbprof/nbprof.c:318 #28 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673 #29 0x0000000000832c80 in execute (op_array=0x27a25b8) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410 #30 0x00007f4614b27db6 in nb_execute (op_array=0x27a25b8) at /tingyun/nbprof/nbprof.c:318 #31 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673 #32 0x0000000000832c80 in execute (op_array=0x2832410) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410 #33 0x00007f4614b27db6 in nb_execute (op_array=0x2832410) at /tingyun/nbprof/nbprof.c:318 ---Type <return> to continue, or q <return> to quit--- #34 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673 #35 0x0000000000832c80 in execute (op_array=0x282c1e0) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410 #36 0x00007f4614b27db6 in nb_execute (op_array=0x282c1e0) at /tingyun/nbprof/nbprof.c:318 #37 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673 #38 0x0000000000832c80 in execute (op_array=0x2896bb8) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410 #39 0x00007f4614b27db6 in nb_execute (op_array=0x2896bb8) at /tingyun/nbprof/nbprof.c:318 #40 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673 #41 0x0000000000832c80 in execute (op_array=0x286ced8) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410 #42 0x00007f4614b27db6 in nb_execute (op_array=0x286ced8) at /tingyun/nbprof/nbprof.c:318 #43 0x00000000007c0e14 in zend_call_function (fci=0x7fff80841170, fci_cache=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_execute_API.c:956 #44 0x000000000064ad3f in zim_reflection_method_invoke (ht=<value optimized out>, return_value=0x288e4e8, return_value_ptr=<value optimized out>, this_ptr=0x2859330, return_value_used=<value optimized out>) at /root/oneshell-master/php-5.4.45/ext/reflection/php_reflection.c:2898 #45 0x00007f4614b27cbe in nb_execute_internal (current_execute_data=0x7f46209c0530, return_value_used=0) at /tingyun/nbprof/nbprof.c:403 #46 0x000000000083e643 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:645 #47 0x0000000000832c80 in execute (op_array=0x27a5a58) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410 #48 0x00007f4614b27db6 in nb_execute (op_array=0x27a5a58) at /tingyun/nbprof/nbprof.c:318 #49 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673 #50 0x0000000000832c80 in execute (op_array=0x27a5b58) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410 #51 0x00007f4614b27db6 in nb_execute (op_array=0x27a5b58) at /tingyun/nbprof/nbprof.c:318 #52 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673 #53 0x0000000000832c80 in execute (op_array=0x27b66a0) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410 #54 0x00007f4614b27db6 in nb_execute (op_array=0x27b66a0) at /tingyun/nbprof/nbprof.c:318 #55 0x000000000083ed94 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:673 #56 0x0000000000832c80 in execute (op_array=0x7f46209f4108) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410 #57 0x00007f4614b27e84 in nb_execute (op_array=0x7f46209f4108) at /tingyun/nbprof/nbprof.c:318 #58 0x0000000000816a9c in ZEND_INCLUDE_OR_EVAL_SPEC_CONST_HANDLER (execute_data=0x7f46209bd318) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:2650 #59 0x0000000000832c80 in execute (op_array=0x7f46209f35b8) at /root/oneshell-master/php-5.4.45/Zend/zend_vm_execute.h:410 #60 0x00007f4614b27dff in nb_execute (op_array=0x7f46209f35b8) at /tingyun/nbprof/nbprof.c:318 #61 0x00000000007cc5ae in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /root/oneshell-master/php-5.4.45/Zend/zend.c:1329 #62 0x000000000077221e in php_execute_script (primary_file=0x7fff80845fc0) at /root/oneshell-master/php-5.4.45/main/main.c:2502 #63 0x000000000087b34e in main (argc=<value optimized out>, argv=<value optimized out>) at /root/oneshell-master/php-5.4.45/sapi/fpm/fpm/fpm_main.c:1938 (gdb) 发现 php的lex_scan执行的时候崩溃了，通过搜索明白是某php正在被词法分析的时候又被修改了。 最后确定是TinkPHP开启调试模式后，模板是动态编译的，这样当有并发访问的时候就发生了php文件又在被解析，又在被写入的情况导致lex_scan的时候php-fpm就挂掉了。 当有并发的时候关闭调试模式即可解决 也算是见识了php程序的问题可以让php-fpm进程都挂掉 |