Adapt performance degrades when using several consecuitive times

[mmorsy1981]

I defined the following DANN models. The DANN model starts performing well, then the accuracy drops (under 0.1 for some cases) for both unadapted and adapted models. Can you explain this behavior and how to handle it. @antoinedemathelin @GRichard513 @AlejandrodelaConcha @BastienZim @atiqm

`def get_encoder():
inp = Input(shape=np.expand_dims(XA_env,-1).shape[1:], name="Signal_Stack")
x = BatchNormalization()(inp)
x = Dropout(0.2)(x)
x = Conv1D(H.shape[1], H.shape[-1], use_bias=False, padding='same', name='Conv1D_L0')(x)
x = Activation('tanh')(x)
x = GlobalMaxPooling1D()(x)
x = Dense(x.shape[-1], activation='relu')(x)
model = Model(inputs=[inp], outputs=[x])
return model

enc_out_shape = get_encoder().output_shape

def get_task():
inp = Input(shape= enc_out_shape[-1], name="Signal_Stack")
x = Dense(inp.shape[-1], activation='relu')(inp)
x = Dropout(0.2)(x)
x = Dense(num_classes, activation='softmax', name = 'OutputLayer')(x)
model = Model(inputs=[inp], outputs=[x])
return model

def get_discriminator():
inp = Input(shape= enc_out_shape[-1], name="Signal_Stack")
x = Dense(inp.shape[-1], activation='relu')(inp)
x = Dropout(0.2)(x)
x = Dense(1, activation='sigmoid')(x)
model = Model(inputs=[inp], outputs=[x])
return model

for i in range(4):
for j in range(4):
DANN_model = DANN(encoder = get_encoder(), discriminator = get_discriminator(), task = get_task(), lambda_=0.5)
DANN_model.compile(loss='categorical_crossentropy', optimizer=Adam(0.001), metrics=["acc"])
DANN_model.fit(X = X[i], y = y[i], Xt = X[j], batch_size=32, epochs=100, shuffle=True)
#X and y denote a partitioned dataset with domain shift between various partitions
print(i ,j, DANN_model.score(X[i], y[i], DANN_model.score(X[j], y[j]) `

[antoinedemathelin]

Hi @mmorsy1981,
Thank you for your interest in the Adapt library.
Are you sure this behavior comes from Adapt? Maybe the last pairs of shift are harder than the first.
You can try to change the order of the paired shift you want to test and see if you get the same accuracy for the same pair?
If the order in the loop impacts the scores it may come from Adapt.
Best

[mmorsy1981]

Thanks for your prompt response.

Validating the Adapt model training using the target dataset showed remarkably high variability in the validation results. Due to this, the testing accuracy degrades significantly.

Any suggestions to deal with this issue?

[antoinedemathelin]

Hi @mmorsy1981,
I am not sure to understand your problem exactly but I can give you some insights to improve DANN training:

• First, DANN is known to be unstable, you can pass some validation data with validation_data in the fit method if you want to monitor the performances across epochs
• Second, I suggest you use SGD instead of Adam. From my experience SGD works better than Adam for adversarial training
• Third, you can use a smaller lambda (I usually consider lambda_=0.1)
• Finally, be sure that the distribution of your y[i] are the same. If there is a strong class imbalance between your domains, DANN won't work

A minor suggestion is to remove BatchNorm in the encoder (or replace it by a frozen scaling layer). The current implementation of DANN may encounter strange behavior with BatchNorm as we have not yet corrected the issue #92