I also had such a situation. Kubernetes alone cannot satisfy your requirement, I was also not able to find any tool that allows to coordinate multiple deployments together (although Flagger looks promising).
So the only way I found was by using CI/CD: Jenkins in my case. I don't have the code, but the idea is the following:
Deploy all application deployments using single Helm chart. Every Helm release name and corresponding Kubernetes labels must be based off of some sequential number, e.g. Jenkins $BUILD_NUMBER
. Helm release can be named like example-app-${BUILD_NUMBER}
and all deployments must have label version: $BUILD_NUMBER
. Important part here is that your Services
should not be a part of your Helm chart because they will be handled by Jenkins.
Start your build with detecting the current version of the app (using bash script or you can store it in ConfigMap
).
Start helm install example-app-{$BUILD_NUMBER}
with --atomic
flag set. Atomic flag will make sure that the release is properly removed on failure. And don't delete previous version of the app yet.
Wait for Helm to complete and in case of success run kubectl set selector service/example-app version=$BUILD_NUMBER
. That will instantly switch Kubernetes Service
from one version to another. If you have multiple services you can issue multiple set selector
commands (each command executes immediately).
Delete previous Helm release and optionally update ConfigMap
with new app version.
Depending on your app you may want to run tests on non user facing Services
as a part of step 4 (after Helm release succeeds).
Another good idea is to have preStop
hooks on your worker pods so that they can finish their jobs before being deleted.