[nightly] Create Nightly Pipeline, make docker-nightly-publish.yml & integration.yml more modular #2628
Conversation
|
Pushing and Pulling the built container to transfer it between hosts in itself takes >10 mins. I would like to avoid this if possible. I added a persistent FSX cache volume to all the self hosted runners. Can you try to use the cache to persist data between runners instead of using ECR ? Maybe this can save us some time. I assume we would run into the same issue with PR sanity tests. |
HappyAmazonian
changed the title
| ./gradlew --refresh-dependencies :serving:dockerDeb -Psnapshot | ||
| - name: Build and push nightly docker image | ||
| if: ${{ inputs.mode == '' || inputs.mode == 'nightly' }} | ||
| - name: Build release docker image |
There was a problem hiding this comment.
minor: Let's call this "Build release candidate docker image"
| - name: Build serving package for nightly | ||
| if: ${{ inputs.mode == '' || inputs.mode == 'nightly' }} | ||
| run: | | ||
| ./gradlew --refresh-dependencies :serving:dockerDeb -Psnapshot |
There was a problem hiding this comment.
we can move this as a command under Build temp docker image? I don't think we need a separate step for this only
| tempRunIdTag="${{ env.AWS_ECR_REPO }}:${{ matrix.arch }}-${{ inputs.mode }}-${GITHUB_RUN_ID}" | ||
| tempCommitTag="${{ env.AWS_ECR_REPO }}:${{ matrix.arch }}-${{ inputs.mode }}-${GITHUB_SHA}" |
There was a problem hiding this comment.
can we make a slight change here where if inputs.mode == 'release, we use the DJL_VERSION value instead of release?
or maybe for both release/nightly/tmp we always just add the djl version number to the tag
HappyAmazonian
changed the title
siddvenk
dismissed
Lokiiiiii’s stale review
changes were addressed as requested
Description
This PR includes the following changes
docker-nightly-publish.ymlnow pushes image to ECR onlydocker-nightly-publish.ymladd arch argintegration.ymladd workflow_call for reusing in futureintegration.ymladd tag-suffix & envs to allow the tests to fetch the image from ECRtests/integration/tests.pynow it uses ECR image to perform the testsdocker_publish.ymlsyncs images in the temp ECR to the staging ECR repo and dockerhubnightly.ymlintroduce a pipeline of build,integtest,publish nightly imagesdocker-nightly-publish now only pushes to ECR.
nightly.yml will handles build, integ test, and push to the staging ECR/dockerhub.
you can use
docker-nightly-publishto build the image and the built image will be in the temp ECR repo.Type of change
Feature/Issue validation/testing
I have created one run here
integ test: https://github.com/deepjavalibrary/djl-serving/actions/runs/12360671952/job/34496566447
nightly build: https://github.com/deepjavalibrary/djl-serving/actions/runs/12401201241/job/34620194153
docker publish: https://github.com/deepjavalibrary/djl-serving/actions/runs/12400140451
nightly pipeline run: https://github.com/deepjavalibrary/djl-serving/actions/runs/12399112958/job/34622823684