AWS CloudFront log analysis in Matomo

For most of our clients, we use AWSCloudFront as a CDN and reverse proxy in front of the web server. This speeds up the delivery of pages and assets and this is also reducing the load on the backend. With correctly configured TYPO3 caches and proper cache headers, we’re measuring CDN hit rates of up to 91%. That’s a huge performance gain and a clear reduction in server load.

Running behind a reverse proxy comes with one downside: traditional log file analysis becomes useless. Since CloudFront handles most of the requests, only a few of them ever reach the actual web server. Client-side tracking (e.g. with Matomo or Google Analytics) would be an option, but we decided against that for performance and privacy reasons (and because it’s simply not always necessary).

So, how can we analyze CloudFront log files in Matomo?

At the moment, AWS CloudFront provides three options for accessing request logs:

1. Standard logs written to an S3 bucket
2. Realtime logs, processed via Amazon Kinesis or similar tools
3. Lambda@Edge logs, streamed to CloudWatch

We went with the classic S3-based logging approach.

CloudFront writes one log file per Point of Presence (PoP). What’s slightly opaque: the duration each log file covers is not defined, and Amazon doesn’t specify when exactly these files will appear in the S3 bucket. Reading the documentation: It may take up to 48 hours for logs to become available.

Matomo doesn’t require logs to be in any specific order, nor does it care about when they’re processed. Even better – Matomo supports CloudFront log formats out of the box.

What was still missing was a small script to retrieve the log files one by one from the S3 bucket, process them, and move them into an archive. Deletion after 30 days is then handled by the S3 retention policy.

That’s it.

You can find the script on GitHub
Pull-requests and issues welcome.