Use the Knowledge Graph data
The Knowledge Graph contains BigQuery tables describing GOV.UK content, documents, editions and relationships.
Read the GOV.UK Knowledge Graph data-source page for information about access, table locations and schemas.
Get access to Knowledge Graph data
The Knowledge Graph data is available in BigQuery
To get access to the Knowledge Graph data, please post in the #data-engineering Slack channel.
You also need a Google Cloud project with billing enabled to run queries. If you are a member of GDS staff, ask the #data-engineering community for a Google Cloud billing project ID.
If you work for another public sector organisation, your organisation will need to provide a suitable billing project.
You can also access Knowledge Graph data via the GovGraph tool.
Choosing the right table
I need a general list of current GOV.UK pages
Start with: govuk-knowledge-graph.search.page
This table is designed for page-level analysis and is used by GovSearch. It excludes redirected pages, gone pages and content without its own base_path. Chapters of multipart content can appear as separate pages.
I need current Publishing API content
Use: govuk-knowledge-graph.public.publishing_api_editions_current
This table contains the most recent non-draft, publicly available edition of each document. Use it when you need lower-level Publishing API fields, content that can be embedded in other pages, or information about redirect and gone items.
I need previous versions of content
Use tables in: govuk-knowledge-graph.publishing_api
For example, publishing_api.editions contains edition records from the Publishing API source data.
I need relationships between content items
Use: govuk-knowledge-graph.public.publishing_api_links_current
This can be used to analyse relationships involving current content, such as publishing organisations and taxonomy links.
I need unpublishing information
Use: govuk-knowledge-graph.public.publishing_api_unpublishings_current
Use the underlying publishing_api.unpublishings table if you require broader historical unpublishing data.
Understand the data model
The Publishing API data model distinguishes between content items, documents and editions.
Content item
└── Document in a locale
└── Edition of that document
content_ididentifies a content itemcontent_idandlocaletogether identify a document- a document can have multiple editions
- an edition represents a version of a document
Do not assume that content_id is unique in a table containing documents in multiple locales.
Understand what a page means
A page, content item, document, edition and URL are not interchangeable.
For example:
- a content item may have separate documents for English and Welsh
- a multipart guide can be displayed across several URLs
- some content has no URL of its own but is displayed within another page
- redirects and ‘gone’ content are included in some tables and excluded from others
Query the data
You can query GovGraph directly in BigQuery or use it through compatible reporting and data visualisation tools.
The following example returns selected fields from current, publicly available Publishing API documents:
SELECT
content_id,
locale,
base_path,
title,
document_type
FROM `govuk-knowledge-graph.public.publishing_api_editions_current`
WHERE base_path IS NOT NULL
Query efficiently
To limit BigQuery processing:
- select only the columns you need
- apply filters as early as possible
- avoid selecting the entire
detailsJSON field when you only need one value
Sample queries
Some sample queries can be found in the project’s GitHub documentation.
Further information
Read: