We've run into what appears to be a race with apache trying to reuse a
pooled connection to a backend when that pool connection is closing.
This leads to errors like:
[Fri Dec 07 21:44:10.752362 2018] [proxy_http:error] [pid 19073:tid 139654393218816] (20014)Internal error (specific information not available): [client 104.130.127.213:45408] AH01102: error reading status line from remote server 127.0.0.1:60999
[Fri Dec 07 21:44:10.752405 2018] [proxy:error] [pid 19073:tid 139654393218816] [client 104.130.127.213:45408] AH00898: Error reading from remote server returned by /image/v2/images/ec31a4fd-e22b-4e97-8c6c-1ef330823fc1/file
According to the internets this can be addressed (at the cost of some
performance) by setting the proxy-initial-not-pooled env var for mod
proxy. From the mod_proxy docs:
If this variable is set, no pooled connection will be reused if the client
request is the initial request on the frontend connection. This avoids the
"proxy: error reading status line from remote server" error message caused
by the race condition that the backend server closed the pooled connection
after the connection check by the proxy and before data sent by the proxy
reached the backend. It has to be kept in mind that setting this variable
downgrades performance, especially with HTTP/1.0 clients.
Closes-Bug: #1807518
Change-Id: I374deddefaa033de858b7bc15f893bf731ad7ff2
e344c97c0e