GOV.UK Web log data
Context
Web log data is collected by anyone visiting www.gov.uk. The data is not constrained by consent, so can give a full view of the reach of www.gov.uk as well as putting consented data into context. This means that it can help us understand the consented web analytics data we collect using GA4 - for example, understanding cookie opt in rate.
Building the data pipeline
The fastly data is piped straight to BigQuery from Fastly. Each morning the previous day’s raw web log data arrives via this pipeline and is housed in BigQuery in the ‘govuk-production’ project.
As the raw web log data is not consented, access to the data in this project is restricted to only those who require it. The data exists in a partitioned table, with a partition expiry of 7 days. This ensures that data older than 7 days is deleted, and we don’t retain any more non-consented data that we have to.