How to reduce "app:patch DexDebug" task execution time

[lcm92]

The project I'm working on takes a lot of time with each build.
Each time Dexpatcher will rewrite the dex in the directory '..\app\build\intermediates\dexpatcher\patched-dex\debug' and it took me almost 5 minutes for each such build.
Do you have any solution?

I am using:

• Android Gradle plugin 3.4.3
• Dexpatcher 2.1.0

[Lanchon]

hi,

i don't know what dexpatcher you are using, start by telling me that. yes, i know you are using dxp-gradle 2.1.0 (which you should not because it is prerelease, but anyway). the version of dxp-tool is specified in your gradle build files.

things you can do:

• paste your gradle.build files, redacting all sensitive info first.
• or at least inform version of dxp-tool used.
• round-trip with apktool (no need to sign the apk, better if you skip resource processing) and inform decode and build times.
• paste the build log using --quiet --stats on dxp-tool (dxp-gradle has options verbosity = QUIET and logStats = true for that)
• are you using bytecode transforms? (if you do not know what they are, you are not)

we can start debugging with that info.

thanks!

[lcm92]

project build.gradle

buildscript {
    repositories {
        google()
        jcenter()
    }
    dependencies {
//        classpath 'com.android.tools.build:gradle:4.1.3'
        classpath 'com.android.tools.build:gradle:3.4.2'
    }
}

buildscript {
    repositories {
        gradlePluginPortal()
        flatDir dirs: rootProject.file('dexpatcher-gradle/libs')
    }
    dependencies {
//        classpath 'com.github.lanchon.dexpatcher:dexpatcher-gradle:2.1.3'
        classpath 'com.github.lanchon.dexpatcher:dexpatcher-gradle:2.1.0'
    }
}

allprojects {
    repositories {
        google()
        jcenter()
    }
}

allprojects {
    repositories {
        maven { url 'https://dexpatcher.github.io/dexpatcher-repo/m2' }
    }
    dependencies {
        configurations.all {
            switch (name) {
                case 'apktool':
                    add name, 'dexpatcher-repo.ibotpeaches.apktool:apktool:2.5.0'
                    break
                case 'dex2jar':
                    add name, 'dexpatcher-repo.dexpatcher.dex2jar:dex-tools:2.1-20190905-lanchon@zip'
                    break
                case 'dexpatcher':
                    add name, 'dexpatcher-repo.dexpatcher.dexpatcher-tool:dexpatcher:1.8.0-beta1'
                    break
                case 'dexpatcherAnnotation':
                    break
                case 'aapt2':
                    break
            }
        }
    }
}

task clean(type: Delete) {
    delete rootProject.buildDir
}

app build.gradle

//apply plugin: 'com.github.lanchon.dexpatcher.patched-application'
apply plugin: 'io.github.dexpatcher.patched-application'

dexpatcherConfig {
    patchedApplication {
        printAppInfo = true
        importSymbols = true
        disableResourceValidation = false
        useAapt2BundledWithApktool = true
    }

    dexpatcher {
        verbosity = QUIET
        logStats = true
        extraArgs.addAll([
                '--verbose',
                '--map-source', '--unmap-output', '--map', 'mapping.txt',
                '--deanon-source', '--reanon-output',
                '--main-plan', 'Anon[_Level]',
                '--alt-plan', '[_]_',
                '--deanon-patches-alt',
                '--no-reanon-errors',
                '--decode-patches', '--no-decode-errors',
        ])
    }

    apktool {
        frameworkInDir = layout.projectDirectory.dir('framework')
        keepBrokenResources = true
    }

    dex2jar {
        handleExceptions = true
        translateCode = true
    }
}

android {
    compileSdkVersion 30
    buildToolsVersion "30.0.3"
    defaultConfig {
        minSdkVersion 23
        targetSdkVersion 29
        applicationId 'xxx.xxx.xxx.xxx'
        versionCode 100
        versionName '1.0.0'
        multiDexEnabled true
    }

    compileOptions {
        sourceCompatibility JavaVersion.VERSION_1_8
        targetCompatibility JavaVersion.VERSION_1_8
    }

    buildTypes {
        debug {
            debuggable true
            minifyEnabled false
            useProguard false
            proguardFiles getDefaultProguardFile('proguard-android.txt')
            signingConfig signingConfigs.debug
            jniDebuggable false
        }
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
        }
    }
}

dependencies {
    sourceApk fileTree(dir: 'apkSrc', include: ['*.apk', '*.jar', '*.zip'])
    implementation fileTree(dir: 'libs', include: ['*.jar'])
//    implementation 'androidx.constraintlayout:constraintlayout:2.0.4'
//    implementation 'androidx.appcompat:appcompat:1.3.0-rc01'
//    implementation 'androidx.cardview:cardview:1.0.0'
//    implementation 'com.google.android.material:material:1.3.0'
}

build log

> Task :app:patchDexDebug
DexPatcher version 1.8.0-beta1 by Lanchon (https://dexpatcher.github.io/)
info: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\decoded-app'
stats: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\decoded-app': 43801 types, 169 ms, 4 us/type
info: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patch-dex\debug'
stats: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patch-dex\debug': 75 types, 1 ms, 13 us/type
stats: patch process: 43876 types, 336 ms, 8 us/type
stats: transform output: 43827 types, 16924 ms, 386 us/type
info: write 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patched-dex\debug'
stats: write 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patched-dex\debug': 43827 types, 279698 ms, 6382 us/type
stats: total process: 43876 types, 297159 ms, 6773 us/type
0 error(s), 0 warning(s)

And the original apk has 11 dex.

[Lanchon]

thanks

remove the '--verbose' item, it is handled by verbosity = ...

im worried by your use of '--no-reanon-errors' and '--no-decode-errors', why are they necessary?

i see you got how to using the advanced features in the latest beta from reading the very sparse docs, congrats.

i haven't used the new transform features (no time) on any real hacks, that's why it is all still in beta, i haven't tested. there could be issues of course. but i aim for correctness first, then deal with optimization if needed, so i haven't got to study performance issues other than at the design phase.

0.34 secs are used to patch the code.

17 s are wasted on the 'transform output' step. you can recover most of that by setting --pre-transform dry (so output code will be pretransformed only of not written -ie, if a dxp error happens). otherwise transform and write back will happen simultaneously. caveats: transform logs will be intermixed with write logs. dxp will produce an (invalid) output if errors happen during output transforms! (given that write starts before transforms are executed) this causes no issues with dxp-gradle though.

280 s are used to write back the dex, there is not much that can be done here. this is an issue with dexlib2 (smali): it is not written for performance.

but there is a way to multi-thread the write back phase: if you have a quad core machine you can have a significant speed gains in this phase. take a look here: https://github.com/DexPatcher/dexpatcher-tool/releases/tag/v1.2.0-beta2

note that multi-dex multi-threading is on by default when using dxp-gradle under certain conditions on debug builds!

how to check if you are using MT mdex? set verbosity = VERBOSE and re run build with logStats = true.

then only paste here the end of the build log like this:

info: write 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patched-dex\debug'
stats: write 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patched-dex\debug': 43827 types, 279698 ms, 6382 us/type
stats: total process: 43876 types, 297159 ms, 6773 us/type
0 error(s), 0 warning(s)

but this time you should see info for each dex file of the output multidex.

try it please, thanks.

[Lanchon]

oh, and please tell me what is the CPU model you use. thanks.

[lcm92]

AMD Ryzen 5 1600

[Lanchon]

ok. after you check what i asked, for that CPU id recommend you use multiDexJobs = n, with n ranging from 6 to 12. currently dxp-tool limits itself to 4 jobs due to java programs not being able to distinguish between CPU cores and hyperthreads. this of course requieres MT mdex being used.

[lcm92]

Thanks for the help, I'll give it a try

[lcm92]

 patchedApplication {
        printAppInfo = true
        importSymbols = true
        multiDexThreadedForMultiDexDebugBuilds = true
        multiDexThreadedForAllDebugBuilds = true
        disableResourceValidation = false
        useAapt2BundledWithApktool = true
    }

    dexpatcher {
        verbosity = VERBOSE
        logStats = true
        multiDex = true
        multiDexThreaded = true
        multiDexJobs = 12
        extraArgs.addAll([
                '--verbose',
                '--pre-transform', 'dry',
                '--map-source', '--unmap-output', '--map', 'mapping.txt',
                '--deanon-source', '--reanon-output',
                '--main-plan', 'Anon[_Level]',
                '--alt-plan', '[_]_',
                '--deanon-patches-alt',
                '--decode-patches',
        ])
    }

DexPatcher version 1.8.0-beta1 by Lanchon (https://dexpatcher.github.io/)
info: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\decoded-app'
stats: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\decoded-app': 43801 types, 178 ms, 4 us/type
info: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patch-dex\debug'
stats: read 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patch-dex\debug': 75 types, 1 ms, 12 us/type
stats: patch process: 43876 types, 328 ms, 7 us/type
info: write 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patched-dex\debug'
stats: write 'E:\PATCH\XXX\app\build\intermediates\dexpatcher\patched-dex\debug': 43827 types, 254646 ms, 5810 us/type
stats: total process: 43876 types, 255184 ms, 5816 us/type
0 error(s), 0 warning(s)

Does it work?

[Lanchon]

hey im sorry! its been a year for me since i touched any of this

i think you need more logging. is there a verbosity = DEBUG? try that instead. sorry :(

[lcm92]

I think you might want to check out my project.

[Lanchon]

dexpatcher {
        verbosity = VERBOSE
        logStats = true
        multiDex = true // THIS SHOULD BE AUTO TRUE IF THE APP MINSDK>=ANDROID 5 I THINK
        multiDexThreaded = true // THIS SHOULD BE AUTO TRUE FOR DEBUG BUILDS ONLY (RECOMMENDED) IF ABOVE CONDITION IS MET
        multiDexJobs = 12 // IT IS PROBABLY BETTER FOR THIS SETTING TO BE SET AT 6 FOR YOUR CPU
        extraArgs.addAll([
                '--verbose', // REMOVE THIS, THIS IS SET BY DXP-GRADLE BASED ON THE verbosity SETTING ABOVE
                '--pre-transform', 'dry', // THIS MADE THE TRANSFORM OUTPUT PHASE GO AWAY, WITH CAVEATS SAID EARLIER
                '--map-source', '--unmap-output', '--map', 'mapping.txt',
                '--deanon-source', '--reanon-output',
                '--main-plan', 'Anon[_Level]',
                '--alt-plan', '[_]_',
                '--deanon-patches-alt',
                '--decode-patches',
        ])
    }

[Lanchon]

from your latest times:

• transform output phase went away due to --pre-transform dry saving 17 s, but producing worse quality logs.
• i guess you are using MT mdex already. you can only increase jobs from default 4 to 6 then.
• you could pre transfrom the source to save some processing, but it's tricky with current dxp-gradle.
• instead of '--map-source', '--unmap-output', '--map', 'mapping.txt', you can inverse-map the patch; it will run faster and produce the same output. CAVEAT: all build logs will refer to the cryptic unmapped items!!!

you can try this:

• round-trip with apktool (no need to sign the apk, better if you skip resource processing) and inform decode and build times.

probably you'll find that smali/dexlib2 is just slow with your big app.

[lcm92]

Right now I cannot swallow such a mass of knowledge :D. I think I need to do more research. Thank you very much, your DexPatcher project is very good.

[Lanchon]

i just tried an apktool roundtrip on your apk (without resource decoding). it took 1m 24s, so there is ample space to increase performance. i suppose transforms are to blame here. i'll check options later when i can.

[lcm92]

I've done many projects with DexPatcher and this is the first one where I had such a problem. Thanks for the support.

[Lanchon]

I've done many projects with DexPatcher

i never find out, due to the nature of most work done with this toolset. i hate that! :)

i committed a change to your repo. it adds build options:

// Build Performance Options
// When set they increase build speed but decrease the quality of the build log.
def lazyTransforms = true   // check transforms while output dex is being written
def fastAnon = true         // apply patch in the anonymized namespace
def fastMap = true          // apply patch in the unmapped namespace

in your PC, when all options are set, i gather your rebuild time should fall from 5 minutes to around 45 seconds.

[Lanchon]

this is the first 'stress test' i've seen the 1.8.0 beta being put though, as i never had time to finish the tool let alone play with or benchmark it.

FYI:
as this is a WIP, the focus is on correctness rather than speed. there is room to improve performance of early transform checks using multi-threading. transforms themselves can probably not be sped up by any significant factor in dxp-tool, but dxp-gradle can be smarter than us here and cache a pre-transformed source dex. this is a double-edge sword: if the mapping file is altered, the pre-transformed dex needs to be rebuilt and thus 2 dex sets need to be rewritten while building; and it turns out dexlib2 is way too slow writing. so enabling this will decrease performance if the mapping file is changed frequently.

the question whether a pre-deanonymized dex can be built and then mapped to a pre-transformed dex (or live during patching) depends on the order of transforms i picked while implementing dxp-tool, which was the result of a very lengthy planning stage of all features before any coding. and if such ordering can or should be altered remains to be seen, but should happen while in beta.

but in case of frequent mapping changes, pre-transforming is clearly not the way out. one possibility remains: dxp-tool could build a complete intermediate mapped source dex representation in memory. this would consume EFFING LOTS of heap memory for big projects, but would probably speed things significantly. it could be away out for those graced with an obscene amount of RAM. it's not hard to implement.

EDIT:
the sequence of transforms in the source dex endpoint and up to the patching stage is:

• map source
• deanonymize source
• encode source
• decode source
• reanonymize source
• unmap source

(note that each mapping transform can apply the composition of two sets of maps.)

this means deanonymize and encode cannot be pre-applied to source so that mapping changes are built faster. the idea was to give the user first deeds on the code, so that the mapping file would not depend on, say, the chosen identifier encoding settings.

this is more than just convenience: it ensures compatibility with mappings of other tools. this was a very important factor in the decision.

i do not think this decision should be reverted in favor of build performance. but should it be done, this would affect the transform orders of all endpoints: not just source, but patches and output too. a better alternative could be the in-memory cache of the transformed code as described earlier.

[lcm92]

Hi, it worked. Thank you very much.

Do you have plans to further develop Dxp in the future?

[Lanchon]

i'll keep this open so others are more likely to find it.

the problem is, i got little time, and no one seems to use the toolset anyway. i would like to at least finish a proper release of -tool and -gradle with the new features.

[Lanchon]

accepted solution:

// Build Performance Options
// When set they increase build speed but decrease the quality of the build log.
def lazyTransforms = true   // check transforms while output dex is being written
def fastAnon = true         // apply patch in the anonymized namespace
def fastMap = true          // apply patch in the unmapped namespace

dexpatcherConfig {
    dexpatcher {
        verbosity = VERBOSE
        logStats = true
        // i need to update dxp-tool with better defaults for newer PCs
        // with more cores so that the following line is no longer needed:
        multiDexJobs = (0.75 * Runtime.getRuntime().availableProcessors() + 0.5) as int
        extraArgs.addAll([
                '--map', 'mapping.txt',
                '--main-plan', 'Anon[_Level]',
                '--alt-plan', '[_]_',
                '--deanon-patches-alt',
                '--decode-patches',
        ])
        if (fastMap) extraArgs.addAll(['--unmap-patches'])
        else extraArgs.addAll(['--map-source', '--unmap-output'])
        if (fastAnon) extraArgs.addAll(['--reanon-patches'])
        else extraArgs.addAll(['--deanon-source', '--reanon-output'])
        if (lazyTransforms) extraArgs.addAll(['--pre-transform', 'dry'])
    }
}