Can not reproduce

[terarachang]

Hi,

Running predictions/DRCAML_mimic3_50/train_new_model.sh, I got such result:

evaluating on test
file for evaluation: ../../mimicdata/mimic3/test_50.csv

[MACRO] accuracy: 0.363, precision: 0.557, recall: 0.465, f-measure: 0.501, AUC: 0.855
[MICRO] accuracy: 0.389, precision: 0.619, recall: 0.511, f-measure: 0.560, AUC: 0.881

prec_at_5: 0.553
rec_at_5: 0.523

The performance of DR-CAML I got above is much worse than the one in Table5.
I cannot reproduce the result of CAML as well, while CNN works quite well as in Table5.
Could you release the parameters or new scripts that can reproduce the performance in the paper ?

Thank you very much !

[najmehs]

Hi,

Thanks for sharing your code!
I have a similar problem.
Running predictions/CAML_mimic3_full/train_new_model.sh, I get the following results:

file for evaluation: ../../mimicdata/mimic3/test_full.csv
3372it [00:28, 117.89it/s]

[MACRO] accuracy, precision, recall, f-measure, AUC
0.0468, 0.0758, 0.0626, 0.0686, 0.8875
[MICRO] accuracy, precision, recall, f-measure, AUC
0.3511, 0.6305, 0.4420, 0.5197, 0.9844
rec_at_8: 0.3640
prec_at_8: 0.6943
rec_at_15: 0.5115
prec_at_15: 0.5481

These results are much worse than the results in table 4 of the paper.
I appreciate it if you release the parameters for the model you trained or an updated version of the scripts you used for that so that I can reproduce your results.

Kind regards,
N

[vijay-raghav]

Hi,

I have the same issue as well. I'm able to reproduce the results with the pre-trained models you've provided.

When I attempt to train the model from scratch, it doesn't give the same performance. I think the problem might be the number of epochs. Early stop doesn't occur within 200 epochs, and the final performance is the result after 200 epochs.

Best,
Vijay

[najmehs]

For me, the early stopping occurs at 89th iteration!
Kind regards,
N

[acadTags]

Check evaluate_model.sh for exact reproduction of results of CAML. The model parameters have been published, see model.pth There is also one script in the DR-CAML folder.

Best wishes,
Acadtags

[Jinyc-20]

Hi,

Thanks for sharing your code!
I have a similar problem.
Running predictions/CAML_mimic3_full/train_new_model.sh, I get the following results:

file for evaluation: ../../mimicdata/mimic3/test_full.csv
3372it [00:28, 117.89it/s]

[MACRO] accuracy, precision, recall, f-measure, AUC
0.0468, 0.0758, 0.0626, 0.0686, 0.8875
[MICRO] accuracy, precision, recall, f-measure, AUC
0.3511, 0.6305, 0.4420, 0.5197, 0.9844
rec_at_8: 0.3640
prec_at_8: 0.6943
rec_at_15: 0.5115
prec_at_15: 0.5481

These results are much worse than the results in table 4 of the paper.
I appreciate it if you release the parameters for the model you trained or an updated version of the scripts you used for that so that I can reproduce your results.

Kind regards,
N

Hi,
I have the same problem. Any solution to train a new model to reproduce the results?
Best,
Yucheng