[Dependency Cache] Dependency Cache on CI per Project [without GRADLE_RO_DEP_CACHE] #135
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 task
Generated by 🚫 Danger |
ParaskP7
force-pushed
the
gradle/isolate-dependency-cache-per-project-v2
branch
9 times, most recently
from
5fef742 to
84a1cb6
Compare
This is done in order to avoid having each client project adding its own project prefix to distinguish the Gradle dependency cache key. Test Only PRs: - WCAndroid: https://github.com/woocommerce/woocommerce-android/ pull/13122 - JP/WPAndroid: https://github.com/wordpress-mobile/WordPress-Android/ pull/21540 For example, for WCAndroid, this would mean that instead of using this 'WOOCOMMERCE' value via this 'GRADLE_DEPENDENCY_CACHE_PROJECT_PREFIX' environment variable, which ultimately is creating the below 'WOOCOMMERCE_GRADLE_DEPENDENCY_CACHE' key, instead, with this change the 'woocommerce-android_GRADLE_DEPENDENCY_CACHE' key will be automatically created, taking the 'woocommerce-android' value from the standard 'BUILDKITE_PIPELINE_SLUG' environment variable. For more info see: p1734350857095859-slack-C020Q0Z579V
This is done in order for us to be able to output the uncompressed cache size during cache restoration, just as it is being done during the cache saving process, for consistency purposes. But, given the fact that are also other clients of this 'restore_cache' script, keeping this parameter optional was the preferred approach in order to not affect, or need to update, any of those other clients of this specific script.
1. On saving, the script will create a new folder, within the 'GRADLE_HOME' space, named 'dependency-cache', where all the contents from 'caches/modules-2' will be copied after the build finishes (minus '*.lock' and 'gc.properties' files, as per the documentation). 2. On restoring, the script will create the 'caches/modules-2' folder, within the 'GRADLE_HOME' space, and extract the saved contents into it and before the build starts. 3. On restoring, the script will also check if 'modules-2' directory exists prior to placing cache. This check is necessary because during 'restore_cache' it might happen that the dependency cache based on this key is not available for download. Thus, no 'dependency-cache/module-2' folder will exist to be copied into the 'caches/modules-2' folder, which will break the script.
This is done in order to differentiate between the 'GRADLE_RO_DEP_CACHE' and the 'GRADLE_DEP_CACHE' caches, and thus be able to test both caching strategies in-parallel, without one overwriting the other during save, or one reusing the other's cache during restore. Test Only PRs: - WCAndroid: https://github.com/woocommerce/woocommerce-android/ pull/13170 - JP/WPAndroid: https://github.com/wordpress-mobile/WordPress-Android/ pull/21550
ParaskP7
force-pushed
the
gradle/isolate-dependency-cache-per-project-v2
branch
from
1d7a5b2 to
94d9f7a
Compare
This was referenced Jan 10, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Project Thread: paaHJt-7CE-p2
Related: #134 + buildkite-ci#570
(Softly) Depends On:
Description
This change:
BUILDKITE_PIPELINE_SLUG) to the existing gradle dependency cache key to enable dependency cache per project. (3b051ba)$GRADLE_HOME/read-only-dependency-cachedirectory), with the more basic GRADLE_DEP_CACHE dependency cache mechanism (using$GRADLE_HOME/dependency-cachedirectory), for save/restore. (61b1d9d)V2) to the existing gradle dependency cache key to differentiate between the two, the previousGRADLE_RO_DEP_CACHEand the newGRADLE_DEP_CACHEdependency cache mechanisms. This was purely done for testing purposes, but I think it might be wise for us to keep this versioning system in place, just in case we update the mechanism again in the future and want to effectively test, then utilize that new change. PS: I am very open to feedback on that. (94d9f7a)Note
Enhancing the save_cache and restore_cache scripts (2.) will positively affect all clients of these scripts, mainly improve things by logging some more information. But, do think if there is going to be a case when these changes might negatively effect some or all those other clients (`I couldn't).
Testing Instructions
This
GRADLE_DEP_CACHEversion ofa8c-ci-toolkitwas thoroughly tested via the below test-only PRs, via the help of theandroid-stagingagents (see buildkite-ci#570):To help you assessing that this change is safe use to use and will not cause any trouble for any project, try and follow the below:
local pre-command hookhas run).assembletask starting, restore cache gets triggered:assembletasks starts, verify that there is NO mention of aread-only dependency cachecache.local pre-command hookhas run).assembletask starting, restore cache gets triggered:assembletasks starts, verify that there is NO mention of aread-only dependency cachecache.Note
The
GRADLE_RO_DEP_CACHEversion ofa8c-ci-toolkitwas also tested via the below test-only PRs:Merge Instructions
restore cache hook removed)restore cache hook removed)save cache trigger removed)do not mergelabel.Note
These WCAndroid#13288 + FluxC#3125 + ADC#39 PRs are not necessarily needed prior to merging this change, but it is better to clean things up and avoid any confusion with relation to having save/restore working on client/libraries projects, while it is actually not (and shouldn't be as of yet).
CHANGELOG.mdif necessary.