How to handle complex job dependency?

[ZillaYT]

I have a folder with 4 jobs, namely,

- build-and-test = a multi-branch pipeline job that builds and tests
code and, if successful, pushes a docker image to docker repo. So I can
have several versions of the docker image, say 1.0.0, 1.0.4, 1.0.11
- deploy-to-sandbox = a pipeline job that pulls above docker image from
docker repo, with desired version from above build, and deploys it to a
sandbox environment
- deploy-to-staging = similarly, = a pipeline job that pulls above
docker image from docker repo, with desired version from above build, and
deploys it to a staging environment, ONLY IF desired version has been
deployed to sandbox already
- deploy-to-production = a pipeline job that pulls above docker image
from docker repo, with desired version from above build, and deploys it to
a production environment, ONLY IF desired version has been deployed to
staging already

So I should ONLY be able to run deploy-to-production(1.0.4) if I've already
ran deploy-to-staging(1.0.4), and hence only if I've already ran
deploy-to-sandbox(1.0.4). One job does NOT automatically trigger any
downstream job.

Also, I don't need to have the same version in all environments. For
example, I can have the scenario where...

- deploy-to-sandbox(1.0.11)
- deploy-to-staging(1.0.4)
- deploy-to-production(1.0.0)

...again as long as what I'm deploying in production has already been
deployed to staging, and as long as that version has already been deployed
to sandbox.

Any pointers on how to do this?

Thanks!
Chris

[Martin d'Anjou]

I do not know of an automated way of doing that.

If I understand correctly, the release number is assigned by the
build-and-test phase, and it published the docker repo.
I assume that this release number is known one way or another by the users,
since it has been published to the docker repo.
This release number is an input to the other jobs. So I suggest you make
the release number an input parameter to the other jobs.

When each job runs, it needs to check that the released artifact has
reached the expected "quality" level for the job. If not the jobs would
fail with some meaningful error message.
The quality levels would be 1) built and tested, 2) sandbox, 3) staging,
and 4) production.
In other words, deploy-to-sandbox(1.0.11) needs to check that the artifacts
1.0.11 exists in the docker repo before it attempts to deploy to sandbox.
You also need to store the quality level somewhere persistent, maybe as a
property of the artifacts in the docker repo (if that is possible?).
Artifactory supports properties, or maybe you need a database.

Hope this helps at least from a conceptual point of view.

Martin

Post by ZillaYT
I have a folder with 4 jobs, namely,
- build-and-test = a multi-branch pipeline job that builds and tests
code and, if successful, pushes a docker image to docker repo. So I can
have several versions of the docker image, say 1.0.0, 1.0.4, 1.0.11
- deploy-to-sandbox = a pipeline job that pulls above docker image
from docker repo, with desired version from above build, and deploys it to
a sandbox environment
- deploy-to-staging = similarly, = a pipeline job that pulls above
docker image from docker repo, with desired version from above build, and
deploys it to a staging environment, ONLY IF desired version has been
deployed to sandbox already
- deploy-to-production = a pipeline job that pulls above docker image
from docker repo, with desired version from above build, and deploys it to
a production environment, ONLY IF desired version has been deployed to
staging already
So I should ONLY be able to run deploy-to-production(1.0.4) if I've
already ran deploy-to-staging(1.0.4), and hence only if I've already ran
deploy-to-sandbox(1.0.4). One job does NOT automatically trigger any
downstream job.
Also, I don't need to have the same version in all environments. For
example, I can have the scenario where...
- deploy-to-sandbox(1.0.11)
- deploy-to-staging(1.0.4)
- deploy-to-production(1.0.0)
...again as long as what I'm deploying in production has already been
deployed to staging, and as long as that version has already been deployed
to sandbox.
Any pointers on how to do this?
Thanks!
Chris

[ZillaYT]

Thanks Martin, though you just reworded my post.

But yes one approach to consider is being able to store which, for example,
versions have been run by deploy-to-sandox. IOW if deploy-to-sandbox has
run with versions 1.0.0 and 1.0.4, I want to store those versions in its
property file, say sandboxed_versons.properties. How will I then tell
deploy-to-staging to read /.../deploy-to-sandbox/sandboxed.properties file?

Chris

Post by Martin d'Anjou
I do not know of an automated way of doing that.
If I understand correctly, the release number is assigned by the
build-and-test phase, and it published the docker repo.
I assume that this release number is known one way or another by the
users, since it has been published to the docker repo.
This release number is an input to the other jobs. So I suggest you make
the release number an input parameter to the other jobs.
When each job runs, it needs to check that the released artifact has
reached the expected "quality" level for the job. If not the jobs would
fail with some meaningful error message.
The quality levels would be 1) built and tested, 2) sandbox, 3) staging,
and 4) production.
In other words, deploy-to-sandbox(1.0.11) needs to check that the
artifacts 1.0.11 exists in the docker repo before it attempts to deploy to
sandbox.
You also need to store the quality level somewhere persistent, maybe as a
property of the artifacts in the docker repo (if that is possible?).
Artifactory supports properties, or maybe you need a database.
Hope this helps at least from a conceptual point of view.
Martin

[Martin d'Anjou]

I think you could use the Copy Artifacts
<https://wiki.jenkins.io/display/JENKINS/Copy+Artifact+Plugin> plugin to
share a file between jobs. But managing the list of releases in a file
becomes hairy IMO. I do not know your specific case, but I guess it will
grow over time, not to millions of records, but possibly to hundreds or
maybe thousands. There is also the question of rollbacks in case of
uncontrollable mistakes (how to erase an entry from the list).

You could use a shared workspace for the purpose of storing that file,
using the external workspace manager plugin
<https://github.com/jenkinsci/external-workspace-manager-plugin/blob/master/README.md>
.

I would consider an external database of some kind and the httpRequest
<https://jenkins.io/doc/pipeline/steps/http_request/> plugin to access it.

If you push your releases to a binary repository, there could be a way to
store that info there too.

Martin

Post by ZillaYT
Thanks Martin, though you just reworded my post.
But yes one approach to consider is being able to store which, for
example, versions have been run by deploy-to-sandox. IOW if
deploy-to-sandbox has run with versions 1.0.0 and 1.0.4, I want to store
those versions in its property file, say sandboxed_versons.properties. How
will I then tell deploy-to-staging to read
/.../deploy-to-sandbox/sandboxed.properties file?
Chris

Post by Martin d'Anjou
I do not know of an automated way of doing that.
If I understand correctly, the release number is assigned by the
build-and-test phase, and it published the docker repo.
I assume that this release number is known one way or another by the
users, since it has been published to the docker repo.
This release number is an input to the other jobs. So I suggest you make
the release number an input parameter to the other jobs.
When each job runs, it needs to check that the released artifact has
reached the expected "quality" level for the job. If not the jobs would
fail with some meaningful error message.
The quality levels would be 1) built and tested, 2) sandbox, 3) staging,
and 4) production.
In other words, deploy-to-sandbox(1.0.11) needs to check that the
artifacts 1.0.11 exists in the docker repo before it attempts to deploy to
sandbox.
You also need to store the quality level somewhere persistent, maybe as a
property of the artifacts in the docker repo (if that is possible?).
Artifactory supports properties, or maybe you need a database.
Hope this helps at least from a conceptual point of view.
Martin

[ZillaYT]

Thanks Martin. I will explore tagging my docker images as we deploy them.
So if I deploy dockerUrl/atrifact:1.0.0 to sandbox, I'll give it a
dockerUrl/artifact:1.0.0-sandboxed tag. My deploy-to-staging job will then
have to "look" for the "sandboxed" part in the tag.

Chris

Post by Martin d'Anjou
I think you could use the Copy Artifacts
<https://wiki.jenkins.io/display/JENKINS/Copy+Artifact+Plugin> plugin to
share a file between jobs. But managing the list of releases in a file
becomes hairy IMO. I do not know your specific case, but I guess it will
grow over time, not to millions of records, but possibly to hundreds or
maybe thousands. There is also the question of rollbacks in case of
uncontrollable mistakes (how to erase an entry from the list).
You could use a shared workspace for the purpose of storing that file,
using the external workspace manager plugin
<https://github.com/jenkinsci/external-workspace-manager-plugin/blob/master/README.md>
.
I would consider an external database of some kind and the httpRequest
<https://jenkins.io/doc/pipeline/steps/http_request/> plugin to access it.
If you push your releases to a binary repository, there could be a way to
store that info there too.
Martin

Post by ZillaYT
Thanks Martin, though you just reworded my post.
But yes one approach to consider is being able to store which, for
example, versions have been run by deploy-to-sandox. IOW if
deploy-to-sandbox has run with versions 1.0.0 and 1.0.4, I want to store
those versions in its property file, say sandboxed_versons.properties. How
will I then tell deploy-to-staging to read
/.../deploy-to-sandbox/sandboxed.properties file?
Chris

Post by Martin d'Anjou
I do not know of an automated way of doing that.
If I understand correctly, the release number is assigned by the
build-and-test phase, and it published the docker repo.
I assume that this release number is known one way or another by the
users, since it has been published to the docker repo.
This release number is an input to the other jobs. So I suggest you make
the release number an input parameter to the other jobs.
When each job runs, it needs to check that the released artifact has
reached the expected "quality" level for the job. If not the jobs would
fail with some meaningful error message.
The quality levels would be 1) built and tested, 2) sandbox, 3) staging,
and 4) production.
In other words, deploy-to-sandbox(1.0.11) needs to check that the
artifacts 1.0.11 exists in the docker repo before it attempts to deploy to
sandbox.
You also need to store the quality level somewhere persistent, maybe as
a property of the artifacts in the docker repo (if that is possible?).
Artifactory supports properties, or maybe you need a database.
Hope this helps at least from a conceptual point of view.
Martin

[Matt Hicks]

It doesn't have pipeline support, but the promoted builds plugin
<https://wiki.jenkins.io/display/JENKINS/Promoted+Builds+Plugin> can do
this for freestyle jobs.

[ZillaYT]

thanks, this doesn't help me.

Post by Matt Hicks
It doesn't have pipeline support, but the promoted builds plugin
<https://wiki.jenkins.io/display/JENKINS/Promoted+Builds+Plugin> can do
this for freestyle jobs.

[ZillaYT]

I ended up following Martin's advice and tagged my docker images in AWS ECR
accordingly by following the steps outlined Retagging an Image with the AWS
CLI
<https://docs.aws.amazon.com/AmazonECR/latest/userguide/retag-aws-cli.html>.
So when I deploy 1.0.0 to sandbox successfully, the deploy-to-sandbox job
retags it as 1.0.0-staging, and my deploy-to-staging job will use that tag
for deployment. Obviously it it's not there, that is, has not been deployed
to sandbox, the tag will not be there.

Chris

Post by ZillaYT
thanks, this doesn't help me.

Post by Matt Hicks
It doesn't have pipeline support, but the promoted builds plugin
<https://wiki.jenkins.io/display/JENKINS/Promoted+Builds+Plugin> can do
this for freestyle jobs.