Update DataHub version
There is no prescribed cadence for updates to DataHub, but new minor versions bumps are typically released every couple of months. Due to the rapid pace of feature development within the platform we aim to be close to the most recent release (without unnecessary exposure to unintended breaking changes).
The upgrade process for DataHub is relatively straightforward as it is handled by a dedicated upgrade job:
- Check the DataHub ‘Updating DataHub’ page for breaking changes
- Check the DataHub helm releases to check for the latest release that’s compatible with the destination version
- Check the DataHub helm release changes from current deployed version to future version for any changes that need to be propagated to the helm chart values files (make sure to set the ‘base’ and ‘compare’ to the correct release values in this view)
- Submit the upgrade release PR and send an ‘alert’ message to the #data-catalogue-dev channel to let others know that DataHub may be disrupted
- Deploy to dev and test out the DataHub interface,
ensure that Find MoJ Data dev
continues to function (run a test deployment)
- Make sure to test out multiple pages of Find MoJ Data, as we’ve found that previously described changes not considered to be breaking have broken our front end.
- If there are issues with the upgrade and compatibility with Find MoJ Data, work with the team to determine the best course of action for upgrade, which may involve a temporary rollback.
- Once there are no issues with the upgrade, merge the PR and roll out the upgrade to higher environments.
Troubleshooting
Another operation in progress
If helm fails with Error: UPGRADE FAILED: another operation (install/upgrade/rollback) is in progress,
you can run helm locally to rollback Datahub to an earlier revision, and then rerun the job.
To see the revision numbers:
helm history datahub --namespace data-platform-datahub-catalogue-dev
To rollback to a specific revision:
helm rollback datahub $REVISION --namespace data-platform-datahub-catalogue-dev
Pre-upgrade hooks failed
If you get an error like this
Error: UPGRADE FAILED: release datahub failed, and has been rolled back due to atomic being set: pre-upgrade hooks failed: 1 error occurred:
* timed out waiting for the condition
It means one of the jobs that run via pre-upgrade hooks failed. You should be able to see which one was running last from the github actions output. You can then use k9s to check the logs for that job.
This error can be caused by a misconfigured (or missing) docker image tag.