Random Error 503 Service Unavailable on Varnish; its log shows no fetcherror

Randomly our production site encounters a 503 error; each time after a varnish service restart, the requested site comes up normally. I looked at the varnish log and all I see is "Service Unavailable" but nothing else. I'm posting a portion of the varnishlog, the default.vcl and the varnish.params > * > 857875 - Begin bereq 857874 fetch - Timestamp Start: 1553896092.649343 0.000000 0.000000 - BereqMethod GET - BereqURL /huf_en_us/search/ajax/suggest/?_=1553896063503&q=pot+leaf+sock - BereqProtocol HTTP/1.0 - BereqHeader X-Real-IP: 172.16.204.112 - BereqHeader X-Forwarded-Proto: https - BereqHeader SSL-OFFLOADED: https - BereqHeader X-Forwarded-Port: 443 - BereqHeader Host: www.hufworldwide.com - BereqHeader CF-IPCountry: US - BereqHeader CF-RAY: 4bf50bf2cb163852-LAX - BereqHeader CF-Visitor: {"scheme":"https"} - BereqHeader accept: application/json, text/javascript, */*; q=0.01 - BereqHeader x-requested-with: XMLHttpRequest - BereqHeader accept-language: en-us - BereqHeader listrak-listening: 1 - BereqHeader referer: https://www.hufworldwide.com/huf_en_us/triple-triangle-pullover-hoodie-pf00100?color=tropical-green - BereqHeader dnt: 1 - BereqHeader user-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1 - BereqHeader x-newrelic-id: VQcDV1JVABAEUVJVDgkOXw== - BereqHeader cookie: store=huf_en_us; _vuid=5759a66b-8175-43cc-a614-51a32c0d7a08; PHPSESSID=los3vfdipo3ako096qoo830m05; form_key=QY53qr4aLFMypLqi; private_content_version=927393d221b9f95d0d23c147b7a38d6b; section_data_ids=%7B%22directory-data%22%3A1553894394%2C%22cus - BereqHeader CF-Connecting-IP: 2606:a000:1216:8689:a5a4:dde1:69bd:10ac - BereqHeader CF-Pseudo-IPv4: 252.241.164.167 - BereqHeader True-Client-IP: 2606:a000:1216:8689:a5a4:dde1:69bd:10ac - BereqHeader CDN-Loop: cloudflare - BereqHeader X-Forwarded-For: 172.16.204.112, 127.0.0.1 - BereqHeader Accept-Encoding: gzip - BereqProtocol HTTP/1.1 - BereqHeader X-Varnish: 857875 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 54 web02(172.16.204.102,,80) 172.16.204.104 60836 - Backend 54 d web02(172.16.204.102,,80) - Timestamp Bereq: 1553896092.649819 0.000476 0.000476 - Timestamp Beresp: 1553896093.563128 0.913785 0.913309 - BerespProtocol HTTP/1.1 - BerespStatus 503 - BerespReason Service Unavailable - BerespHeader Date: Fri, 29 Mar 2019 21:48:12 GMT - BerespHeader Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 PHP/7.0.30 - BerespHeader X-Powered-By: PHP/7.0.30 - BerespHeader X-Content-Type-Options: nosniff - BerespHeader X-XSS-Protection: 1; mode=block - BerespHeader X-Frame-Options: SAMEORIGIN - BerespHeader Expires: Thu, 19 Nov 1981 08:52:00 GMT - BerespHeader Cache-Control: no-store, no-cache, must-revalidate - BerespHeader Pragma: no-cache - BerespHeader Set-Cookie: PHPSESSID=los3vfdipo3ako096qoo830m05; expires=Sun, 28-Apr-2019 21:48:13 GMT; Max-Age=2592000; path=/; domain=www.hufworldwide.com; secure; HttpOnly - BerespHeader X-NewRelic-App-Data: PxQGUFJVDAUIR1VXBAYOXl0IFB9AMQYAZBBZDEtZV0ZaClc9HiBQFg1ZWT1JJUpcXhAiDVlFRQkIXVNBPkkuA1cHVhZXZGh0QQRUBEYOQQk4anYRFj9kdUILDxZ0XlkSFl5aXwcUPz55DEwHSltRQkcKS0MdUR1SVAYHUUpTFgoAXVRQGxwGSkYCAwZbUltQB1oMWw0MBQVWRxUHUA1ABzk= - BerespHeader Connection: close - BerespHeader Transfer-Encoding: chunked - BerespHeader Content-Type: text/html; charset=UTF-8 - TTL RFC -1 -1 -1 1553896094 1553896094 1553896092 375007920 0 - VCL_call BACKEND_RESPONSE - TTL VCL 0 10 0 1553896094 - VCL_return deliver - BerespHeader Content-Encoding: gzip - Storage malloc Transient - ObjProtocol HTTP/1.1 - ObjStatus 503 - ObjReason Service Unavailable - ObjHeader Date: Fri, 29 Mar 2019 21:48:12 GMT - ObjHeader Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips mod_fcgid/2.3.9 PHP/7.0.30 - ObjHeader X-Powered-By: PHP/7.0.30 - ObjHeader X-Content-Type-Options: nosniff - ObjHeader X-XSS-Protection: 1; mode=block - ObjHeader X-Frame-Options: SAMEORIGIN - ObjHeader Expires: Thu, 19 Nov 1981 08:52:00 GMT - ObjHeader Cache-Control: no-store, no-cache, must-revalidate - ObjHeader Pragma: no-cache - ObjHeader Set-Cookie: PHPSESSID=los3vfdipo3ako096qoo830m05; expires=Sun, 28-Apr-2019 21:48:13 GMT; Max-Age=2592000; path=/; domain=www.hufworldwide.com; secure; HttpOnly - ObjHeader X-NewRelic-App-Data: PxQGUFJVDAUIR1VXBAYOXl0IFB9AMQYAZBBZDEtZV0ZaClc9HiBQFg1ZWT1JJUpcXhAiDVlFRQkIXVNBPkkuA1cHVhZXZGh0QQRUBEYOQQk4anYRFj9kdUILDxZ0XlkSFl5aXwcUPz55DEwHSltRQkcKS0MdUR1SVAYHUUpTFgoAXVRQGxwGSkYCAwZbUltQB1oMWw0MBQVWRxUHUA1ABzk= - ObjHeader Content-Type: text/html; charset=UTF-8 - ObjHeader Content-Encoding: gzip - Fetch_Body 2 chunked - - Gzip G F E 16777 6581 80 52568 52578 - Timestamp BerespBody: 1553896093.572716 0.923373 0.009588 - BackendClose 54 web02(172.16.204.102,,80) - Length 6581 - BereqAcct 3058 0 3058 869 16789 17658 - End default.vcl > vcl 4.0; > > import std; import directors; > > REM The minimal Varnish version is 4.0 For SSL offloading, pass the following header in your proxy server or load balancer: > 'SSL-OFFLOADED: https' > > backend web01 { > .host = "172.16.204.101"; > .port = "80"; > .connect_timeout = 1600s; > .first_byte_timeout = 1600s; > .between_bytes_timeout = 1600s; > .max_connections = 20000; } > > backend web02 { > .host = "172.16.204.102"; > .port = "80"; > .connect_timeout = 1600s; > .first_byte_timeout = 1600s; > .between_bytes_timeout = 1600s; > .max_connections = 20000; } > > backend web03 { > .host = "172.16.204.120"; > .port = "80"; > .connect_timeout = 1600s; > .first_byte_timeout = 1600s; > .between_bytes_timeout = 1600s; > .max_connections = 20000; } > > backend web04 { > .host = "172.16.204.121"; > .port = "80"; > .connect_timeout = 1600s; > .first_byte_timeout = 1600s; > .between_bytes_timeout = 1600s; > .max_connections = 20000; } varnish.params > REM Varnish environment configuration description. This was derived from > REM the old style sysconfig/defaults settings > > REM Set this to 1 to make systemd reload try to switch VCL without restart. RELOAD_VCL=1 > > REM Set WARMUP_TIME to force a delay in reload-vcl between vcl.load and vcl.use > REM This is useful when backend probe definitions need some time before declaring > REM configured backends healthy, to avoid routing traffic to a non-healthy backend. > REM WARMUP_TIME=0 > > REM Main configuration file. You probably want to change it. VARNISH_VCL_CONF=/etc/varnish/default.vcl > > REM Default address and port to bind to. Blank address means all IPv4 > REM and IPv6 interfaces, otherwise specify a host name, an IPv4 dotted > REM quad, or an IPv6 address in brackets. > REM VARNISH_LISTEN_ADDRESS=192.168.1.5 VARNISH_LISTEN_PORT=8080 > > REM Admin interface listen address and port VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1 VARNISH_ADMIN_LISTEN_PORT=6082 > > REM Shared secret file for admin interface VARNISH_SECRET_FILE=/etc/varnish/secret > > REM Backend storage specification, see Storage Types in the varnishd(5) > REM man page for details. VARNISH_STORAGE="malloc,24G" > > REM User and group for the varnishd worker processes VARNISH_USER=builder VARNISH_GROUP=builder > > REM Other options, see the man page varnishd(1) DAEMON_OPTS="-p thread_pool_min=200 -p thread_pool_max=2000 -p thread_pool_timeout=600 > -p http_resp_hdr_len=90000 -p http_max_hdr=200"