Secrets management
1password
We use 1password to share secrets within the team and setup local development environments.
There is one vault for Find MoJ data, and one vault for Datahub.
There are a number of 1password utilities available to manage credentials from cli and desktop environments.
- Install the 1Password desktop app - https://support.1password.com/get-the-apps/
- Install the 1Password CLI app - https://developer.1password.com/docs/cli/get-started/
- Follow the steps to turn on and test the 1password desktop app integration
Github Actions secrets
Our deployment pipelines and CI rely on secrets in Github Actions.
These secrets can be updated by anyone with maintainer access to the repo. Once set, they cannot be viewed again.
Application secrets
At runtime, applications read secrets from Kubernetes.
Secrets in a namespace can be listed with
kubectl -n data-platform-datahub-catalogue-dev get secret
or viewed with
kubectl -n data-platform-datahub-catalogue-dev get secret $secretname -o yaml
Managing Datahub database variables and secrets
Datahub provides helm templates and appropriate variables to connect to a backend database of your choice. We currently utilise Postgres via Cloud Platforms RDS integration.
Variables & Examples:
- host:
"cloud-platform-xyz.abcd.eu-west-2.rds.amazonaws.com:5432" - hostForpostgresqlClient:
"cloud-platform-xyz.abcd.eu-west-2.rds.amazonaws.com" - port:
"5432" - url:
"jdbc:postgres://cloud-platform-xyz.abcd.eu-west-2.rds.amazonaws.com:5432/my-db-name" - driver:
"org.postgresql.Driver" - username:
"" - password:
"" - DATAHUB_DB_NAME:
"my-db-name"
When utilising cloud platforms RDS integration, a secret store containing the connection variables is automatically created for you.
kubectl get secret -n data-platform-datahub-catalogue-dev rds-postgresql-instance-output -o json | jq '.data | map_values(@base64d)'
{
"database_name": "my-db-name",
"database_password": "",
"database_username": "",
"rds_instance_address": "cloud-platform-xyz.abcd.eu-west-2.rds.amazonaws.com",
"rds_instance_endpoint": "cloud-platform-xyz.abcd.eu-west-2.rds.amazonaws.com:5432",
"rds_url": "jdbc:postgres://cloud-platform-xyz.abcd.eu-west-2.rds.amazonaws.com:5432/my-db-name"
}
Unfortunately the datahub helm templates only support obtaining values from a namespace secret store for the username, password and DATAHUB_DB_NAME variables. Therefore it is necessary to pass in values for variables host, hostForpostgresqlClient and url via the workflow, having first obtained them from github secrets.
This issue provides more detailed information
Example Sql configuration in values-base.yaml:
sql:
datasource:
host: ""
hostForpostgresqlClient: ""
port: "5432"
url: ""
driver: "org.postgresql.Driver"
username:
secretRef: rds-postgresql-instance-output
secretKey: database_username
password:
secretRef: rds-postgresql-instance-output
secretKey: database_password
extraEnvs:
- name: "DATAHUB_DB_NAME"
valueFrom:
secretKeyRef:
name: rds-postgresql-instance-output
key: database_name
Values for host, hostForpostgresqlClient and url are populated in the workflow deploy-workflow.yml:
run: |
helm upgrade \
--install ${RELEASE_NAME} datahub/datahub \
--version ${CHART_VERSION} \
--atomic --debug --timeout 10m0s \
--values helm_deploy/values-base.yaml \
--namespace ${{ secrets.kube_namespace }} \
...
--set global.sql.datasource.host=${POSTGRES_HOST} \
--set global.sql.datasource.hostForpostgresqlClient=${POSTGRES_CLIENT_HOST} \
--set global.sql.datasource.url=${POSTGRES_URL}
For convenience the k8s-rds-secrets-to-github-secrets.sh script can be used from a local device to populate this repo’s environment secrets with the database secrets for the datahub environments secret store.
It can be called as
bash k8s-rds-secrets-to-github-secrets <namespace root> <CP environment name>
for example:
bash k8s-rds-secrets-to-github-secrets.sh data-platform-datahub-catalogue dev
See Managing Secrets using kubectl.
We are not using AWS Secrets Manager as an external secretstore due to limitations of AWS console access in Cloud Platform.