In brief: Using forensics to keep tabs on Apache

Today I have found an elegant solution to a recurring problem in practice, which I would like to share with you:

When setting up geolocation in a TYPO3 instance I had the problem that I suspected that TYPO3 does not see the correct client IP in the incoming requests. For this you have to know that the instance in question is running under Kubernetes. Here incoming requests are - put simply - routed to the application via a proxy (so called Ingress). For this to work, the application must be prepared to evaluate standard proxy headers such as X-Forwarded-For and X-Forwarded-Host.

Normally this is not a major problem in TYPO3. Inside $GLOBALS['TYPO3_CONF_VARS']['SYS'] one can find the configuration options reverseProxyHeaderMultiValue, reverseProxyIP, reverseProxySSL and trustedHostsPattern. Set correctly, TYPO3 evaluates the headers correctly and understands not the IP address of the requesting proxy, but that of the client behind to be the interesting one.

In this case, a Google load balancer on GKE (Google Kubernetes Engine) was used as the Ingress provider. This has the peculiarity of passing only the client's IP in the X-Forwarded-For header. Unfortunately, the standard-compliant variant from RFC 7239 (Forwarded header) has not yet become ubiquitous. Therefore, Google isn't even to blame here. Where there is no standard, I cannot violate one.

Now how did I notice that this is the case? In principle, I can of course look into the TYPO3 backend and check "Admin Tools -> Environment -> PHP Info" to learn about the HTTP headers PHP sees. But there are two problems with this approach:

1. Of course, this only works in the backend. Frontend problems can't be debugged this way. Especially not if front- and backend do not take the same path in the webserver (e.g. different vHosts).
2. I have to get into the backend first. For example, if I want to debug a problem with the referrer configuration, I can't even log into the backend to check the headers without setting [SYS][features][security.backend.enforceReferrer] = false. Not so good.

It would be nice to be able to see directly which headers the web server (in this case, Apache) has received from the client or proxy. In principle, it's just like watching a Varnish instance via varnishlog. And this is where the Apache module mod_log_forensic comes into play!

In the simplest configuration, you set something like the following in the vHost after enabling the module via a2enmod log_forensic:

<IfModule log_forensic_module>
  ForensicLog /var/log/apache2/forensic_log
</IfModule>

Apache then writes the following into the file /var/log/apache2/forensic_log:

+28:5fb2fb02:1d|GET / HTTP/1.1|Host:www.pfmmedical.de.feature-french-layer.pfm-corporate.review.web-factory.de|X-Request-ID:f038a5c9ef22f01276e4aafd790b346d|X-Real-IP:83.135.142.198|X-Forwarded-For:83.135.142.198|X-Forwarded-Proto:https|X-Forwarded-Host:www.pfmmedical.de.feature-french-layer.pfm-corporate.review.web-factory.de|X-Forwarded-Port:443|X-Scheme:https|accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8|accept-encoding:gzip, deflate, br|user-agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15|accept-language:de-de
-28:5fb2fb02:1d
+25:5fb2fb0a:1f|GET /robots.txt HTTP/1.1|Host:feature-french-layer.pfm-corporate.review.web-factory.de|User-Agent:kube-probe/1.17+|Accept-Encoding:gzip|Connection:close
-25:5fb2fb0a:1f

You can see very nicely that two requests came in here, one from a Mac and one from kube-probe (in this case the readiness probe of the application). The practical thing is that the complete request headers also appear, so that it is now much easier to isolate (supposed) proxy problems.

Because, in the end it turned out that the original culprit was not the TYPO3 configuration at all, but a stupid bug in the geolocation logic. Well, that's how it sometimes works.