Ytdyklly

I'm having some issues with just a single site that keeps segfaulting php-fpm7.

Can't seem to pinpoint the exact moment when this occurs, i browse around on the site and wait and after a while when something is closed the core dump is produced (This happens very frequently).

[Fri Aug 11 14:31:37 2017] php-fpm[15202]: segfault at 7f68cdd238c5 ip 0000562a8248055f sp 00007ffc22d8ea70 error 4 in php-fpm7.0[562a82214000+45e000]

[Fri Aug 11 14:36:50 2017] php-fpm[18507]: segfault at 7f68cd5d0345 ip 0000562a8248055f sp 00007ffc22d8ea70 error 4 in php-fpm7.0[562a82214000+45e000]

[Fri Aug 11 14:37:17 2017] php-fpm[18658]: segfault at 7f68cd5b6345 ip 0000562a8248055f sp 00007ffc22d8ea70 error 4 in php-fpm7.0[562a82214000+45e000]

[Fri Aug 11 14:38:06 2017] php-fpm[18884]: segfault at 7f68cd5286e5 ip 0000562a8248055f sp 00007ffc22d8ea70 error 4 in php-fpm7.0[562a82214000+45e000]

[Fri Aug 11 14:38:21 2017] php-fpm[18944]: segfault at 7f68cd5fc525 ip 0000562a8248055f sp 00007ffc22d8ea70 error 4 in php-fpm7.0[562a82214000+45e000]

[Fri Aug 11 14:39:26 2017] php-fpm[19447]: segfault at 7f68cd30c915 ip 0000562a8248055f sp 00007ffc22d8ea70 error 4 in php-fpm7.0[562a82214000+45e000]

[Fri Aug 11 14:42:34 2017] php-fpm[21669]: segfault at 7f68cd5575c5 ip 0000562a8248055f sp 00007ffc22d8ea70 error 4 in php-fpm7.0[562a82214000+45e000]

[Fri Aug 11 14:47:29 2017] php-fpm[24499]: segfault at 7f68cd536685 ip 0000562a8248055f sp 00007ffc22d8ea70 error 4 in php-fpm7.0[562a82214000+45e000]

[Fri Aug 11 14:48:30 2017] php-fpm[25072]: segfault at 7f68cd554725 ip 0000562a8248055f sp 00007ffc22d8ea70 error 4 in php-fpm7.0[562a82214000+45e000]

[Fri Aug 11 14:48:54 2017] php-fpm[24750]: segfault at 7f68ccd5a5c5 ip 0000562a8248055f sp 00007ffc22d8ea70 error 4 in php-fpm7.0[562a82214000+45e000]

[Fri Aug 11 14:49:49 2017] php-fpm[25857]: segfault at 7f68cd5f6ba5 ip 0000562a8248055f sp 00007ffc22d8ea70 error 4 in php-fpm7.0[562a82214000+45e000]

From the stacktrace it looks like some soap request is destroyed and that's where the crash occurs but i can't even find a soap request in the site source.

First thought would be to install php7.0-dev but i already have that installed from the error in the stacktrace: /usr/src/builddir/Zend/zend_string.h: No such file or directory

Php-FPM version

PHP 7.0.20-1~dotdeb+8.2 (fpm-fcgi)

Copyright (c) 1997-2017 The PHP Group

Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies

    with Zend OPcache v7.0.20-1~dotdeb+8.2, Copyright (c) 1999-2017, by Zend Technologies

PHP-FPM Pool configuration

[mysite]

user = mysite

group = mysite

prefix = /data/sites/mysite

listen = /var/run/php-fpm-mysite.sock

chdir = /data/sites/mysite/public_html

php_admin_value[disable_functions] = exec,passthru,shell_exec,system

listen.owner = www-data

listen.group = www-data

pm = ondemand

pm.process_idle_timeout = 10s

pm.max_children = 4

pm.max_requests = 200

pm.start_servers = 2

pm.min_spare_servers = 1

pm.max_spare_servers = 2

php_value[session.save_path] = /tmp/php/sessions

php_value[session_save_path] = /tmp/php/sessions

PHP-Modules

ii  dh-php                         0.21-1~dotdeb+8.1                all          debhelper add-on to handle PHP PECL extensions

ii  php-common                     1:42-1~dotdeb+8.1                all          Common files for PHP packages

ii  php7.0-cli                     7.0.20-1~dotdeb+8.2              amd64        command-line interpreter for the PHP scripting language

ii  php7.0-common                  7.0.20-1~dotdeb+8.2              amd64        documentation, examples and common module for PHP

ii  php7.0-curl                    7.0.20-1~dotdeb+8.2              amd64        CURL module for PHP

ii  php7.0-dev                     7.0.20-1~dotdeb+8.2              amd64        Files for PHP7.0 module development

ii  php7.0-fpm                     7.0.20-1~dotdeb+8.2              amd64        server-side, HTML-embedded scripting language (FPM-CGI binary)

ii  php7.0-gd                      7.0.20-1~dotdeb+8.2              amd64        GD module for PHP

ii  php7.0-imagick                 3.4.3-1~dotdeb+8.1               amd64        Provides a wrapper to the ImageMagick library

ii  php7.0-imap                    7.0.20-1~dotdeb+8.2              amd64        IMAP module for PHP

ii  php7.0-intl                    7.0.20-1~dotdeb+8.2              amd64        Internationalisation module for PHP

ii  php7.0-json                    7.0.20-1~dotdeb+8.2              amd64        JSON module for PHP

ii  php7.0-mbstring                7.0.20-1~dotdeb+8.2              amd64        MBSTRING module for PHP

ii  php7.0-mcrypt                  7.0.20-1~dotdeb+8.2              amd64        libmcrypt module for PHP

ii  php7.0-mysql                   7.0.20-1~dotdeb+8.2              amd64        MySQL module for PHP

ii  php7.0-opcache                 7.0.20-1~dotdeb+8.2              amd64        Zend OpCache module for PHP

ii  php7.0-readline                7.0.20-1~dotdeb+8.2              amd64        readline module for PHP

ii  php7.0-soap                    7.0.20-1~dotdeb+8.2              amd64        SOAP module for PHP

ii  php7.0-xml                     7.0.20-1~dotdeb+8.2              amd64        DOM, SimpleXML, WDDX, XML, and XSL module for PHP

ii  php7.0-zip                     7.0.20-1~dotdeb+8.2              amd64        Zip module for PHP

Platform

Kubernetes/Docker container inside Google Container Engine

NGINX version & build

nginx version: nginx/1.10.3

built with OpenSSL 1.0.1t  3 May 2016

TLS SNI support enabled

configure arguments: --with-cc-opt='-g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --modules-path=/usr/lib/nginx/modules --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_v2_module --with-http_dav_module --with-file-aio --with-threads --with-http_addition_module --with-http_geoip_module=dynamic --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module=dynamic --with-http_secure_link_module --with-http_sub_module --with-http_xslt_module=dynamic --with-stream=dynamic --with-stream_ssl_module --with-mail=dynamic --with-mail_ssl_module --add-dynamic-module=/usr/src/builddir/debian/modules/nginx-auth-pam --add-module=/usr/src/builddir/debian/modules/nginx-dav-ext-module --add-module=/usr/src/builddir/debian/modules/nginx-echo --add-module=/usr/src/builddir/debian/modules/nginx-upstream-fair --add-module=/usr/src/builddir/debian/modules/ngx_http_substitutions_filter_module --add-module=/usr/src/builddir/debian/modules/nginx-cache-purge --add-module=/usr/src/builddir/debian/modules/nginx-x-rid-header --with-ld-opt=-lossp-uuid

Stacktrace

Program terminated with signal SIGSEGV, Segmentation fault.

#0  zend_string_release (s=0x7fd20d50c4c0) at /usr/src/builddir/Zend/zend_string.h:269

269     /usr/src/builddir/Zend/zend_string.h: No such file or directory.

(gdb) bt full

#0  zend_string_release (s=0x7fd20d50c4c0) at /usr/src/builddir/Zend/zend_string.h:269

No locals.

#1  zend_hash_destroy (ht=0x55d590f01c30) at /usr/src/builddir/Zend/zend_hash.c:1277

        p = 0x55d590f01c90

        end = 0x55d590f01d10

#2  0x00007fd215897c21 in delete_type_persistent (zv=<optimized out>) at /usr/src/builddir/ext/soap/php_schema.c:2459

        type = 0x55d590f01b10

#3  0x000055d58e956e55 in zend_hash_destroy (ht=0x55d590ef8380) at /usr/src/builddir/Zend/zend_hash.c:1264

        p = 0x55d590f003c8

        end = 0x55d590f00aa8

#4  0x00007fd21589f121 in delete_psdl_int (p=<optimized out>) at /usr/src/builddir/ext/soap/php_sdl.c:3130

        tmp = 0x55d590ef8270

#5  0x00007fd21589f18f in delete_psdl (zv=0x55d590f211a0) at /usr/src/builddir/ext/soap/php_sdl.c:3150

No locals.

#6  0x000055d58e956ef6 in zend_hash_destroy (ht=0x55d590efcf40) at /usr/src/builddir/Zend/zend_hash.c:1275

        p = 0x55d590f211a0

        end = 0x55d590f21200

#7  0x00007fd21586f71b in zm_shutdown_soap (type=<optimized out>, module_number=40) at /usr/src/builddir/ext/soap/soap.c:598

No locals.

#8  0x000055d58e94c577 in module_destructor (module=module@entry=0x55d590c19f20) at /usr/src/builddir/Zend/zend_API.c:2505

No locals.

#9  0x000055d58e94460c in module_destructor_zval (zv=<optimized out>) at /usr/src/builddir/Zend/zend.c:620

        module = 0x55d590c19f20

#10 0x000055d58e9579c9 in _zend_hash_del_el_ex (prev=<optimized out>, p=<optimized out>, idx=<optimized out>, ht=<optimized out>) at /usr/src/builddir/Zend/zend_hash.c:1025

        tmp = {value = {lval = 94375745003296, dval = 4.6627813406802235e-310, counted = 0x55d590c19f20, str = 0x55d590c19f20, arr = 0x55d590c19f20, obj = 0x55d590c19f20, 

            res = 0x55d590c19f20, ref = 0x55d590c19f20, ast = 0x55d590c19f20, zv = 0x55d590c19f20, ptr = 0x55d590c19f20, ce = 0x55d590c19f20, func = 0x55d590c19f20, ww = {w1 = 2428608288, 

              w2 = 21973}}, u1 = {v = {type = 17 '21', type_flags = 0 '00', const_flags = 0 '00', reserved = 0 '00'}, type_info = 17}, u2 = {var_flags = 21973, next = 21973, 

            cache_slot = 21973, lineno = 21973, num_args = 21973, fe_pos = 21973, fe_iter_idx = 21973}}

#11 _zend_hash_del_el (p=0x55d590c0ed00, idx=39, ht=0x55d58edf0d60 <module_registry>) at /usr/src/builddir/Zend/zend_hash.c:1049

        prev = <optimized out>

#12 zend_hash_graceful_reverse_destroy (ht=ht@entry=0x55d58edf0d60 <module_registry>) at /usr/src/builddir/Zend/zend_hash.c:1505

        idx = 39

        p = 0x55d590c0ed00

#13 0x000055d58e94a95c in zend_destroy_modules () at /usr/src/builddir/Zend/zend_API.c:1982

No locals.

#14 0x000055d58e9456a5 in zend_shutdown () at /usr/src/builddir/Zend/zend.c:856

No locals.

#15 0x000055d58e8e4c0b in php_module_shutdown () at /usr/src/builddir/main/main.c:2360

No locals.

#16 0x000055d58e7ca015 in main (argc=-1865301124, argv=0x55d590d1c2dc) at /usr/src/builddir/sapi/fpm/fpm/fpm_main.c:2021

        c = -1865319872

        file_handle = {handle = {fd = 591867904, fp = 0x7fd223473000, stream = {handle = 0x7fd223473000, isatty = 0, mmap = {len = 3799, pos = 0, map = 0x0, 

                buf = 0x7fd2260ed000 <error: Cannot access memory at address 0x7fd2260ed000>, old_handle = 0x0, old_closer = 0x0}, reader = 0x55d58e8fc140 <_php_stream_read>, 

              fsizer = 0x55d58e8e2990 <php_zend_stream_fsizer>, closer = 0x55d58e8e2970 <php_zend_stream_mmap_closer>}}, filename = 0x7fd223402000 "( @#322177", opened_path = 0x0, 

          type = ZEND_HANDLE_FILENAME, free_filename = 0 '00'}

        max_requests = 200

        fcgi_fd = -1897990464

        __func__ = "main"

Any insight would really be appreciated, i can't seem to figure out what is causing this.

I've seem to have identified the source of the crash.

The Wordpress plugin: Resurs Bank Payment Gateway for WooCommerce seems to be causing this crash, when deactivated the segfaults stop.

Somehow their use of the SoapClient must be causing this, how i do not know.

Edit:

The plugin was an indirect cause of the problem, the plugin seemed to have triggered a bug in PHP 7.

https://bugs.php.net/bug.php?id=71662

Supposed to be fixed in 7.0.4.

Tried updating PHP version to 7.1 and no more segfaults.