Datahub components
Metadata Service (GMS)
The core of Datahub is its Metadata Service.
This contains a GraphQL API and REST API for querying metadata, and is developed to be agnostic to the metadata model in use.
Kafka consumer jobs
We are currently running the Kafka consumer jobs as part of the GMS. They can be split out by passing the global.datahub_standalone_consumers_enabled value to the helm chart.
The Metadata change event consumer consumes from the MetadataChangeProposal_v1 Kafka topic.
It processes requests to change metadata (Metadata change proposals, or MCPs.)
The Metadata audit event consumer consumes from the MetadataChangeLog_Versioned_v1 and MetadataChangeLog_Timeseries_v1 Kafka topics.
It responds to changes in the metadata graph (e.g. by updating the search index.)
React frontend
The React frontend serves Datahub’s own UI.
Entity registry
The Entity Registry configures the entities, aspects, and relationships that make up the metadata model. We use the metadata model that is built into DataHub.