Show the number of not classified instances for multi-label models (#3964)

[piri-p]

Fix #3964 - Show the number of not classified instances for multi-label models @suhaibmujahid

Done by

• Passing extra parameter y_test to calculate the total number of not-classified instances (y_test - classified)
• Adding table header

Run log attached.

issue-3964-run-log.txt

[suhaibmujahid]

Thank you, @piri-p! Please see my comments. Also, please fix the linter (you may want to consider installing pre-commit¹).

Footnotes

1. https://github.com/mozilla/bugbug#auto-formatting ↩

[suhaibmujahid]

The number of "Not classified" was originally part of the table but dropped earlier:

    # Don't show the Not classified row in the table output
    if "__NOT_CLASSIFIED__" in labels and not is_multilabel:
        confusion_matrix_table.pop(labels.index("__NOT_CLASSIFIED__"))

It may be cleaner to avoid dropping it in the first place.

[suhaibmujahid]

I think this comment is helpful in the context of reviewing this PR. However, without this context, the comment feels out of place (after merging the PR).

[suhaibmujahid]

The same case as the comment above.

[piri-p]

Hi @suhaibmujahid, thanks you for feedback!

Also, please fix the linter (you may want to consider installing pre-commit

Done :)

The number of "Not classified" was originally part of the table but dropped earlier:

    # Don't show the Not classified row in the table output
    if "__NOT_CLASSIFIED__" in labels and not is_multilabel:
        confusion_matrix_table.pop(labels.index("__NOT_CLASSIFIED__"))

It may be cleaner to avoid dropping it in the first place.

I just run the command in issue-3964-run-log.txt again and checked the confusion matrix of "Not classified" class, it gives [[6, 0], [1, 1]] for Confidence threshold > 0.6.

Now I am not sure how to interpret this. This confusion matrix doesn't seem to correspond to a specific class; but we would like to see the number of 'Not classified' instances in each specific class' visualizer.

In my current implementation, I calculate the number of "Not classified" class-wise, i.e., any non-classified samples from the test set are counted. Here, the test set is of size 200, so any non-classified samples among these 200 are counted. Also, this way, the total sum of all elements in each matrix is precisely the test size, 200.

Says instead of this using current implementation

Current implementation - any non-classified samples from the test set are counted (here: 192, 190)
label: crash - 
     total sum = 1 + 0 + 192 + 0 + 7 + 0 = 200; 
     192 = instances whose ground truth label isn't 'crash', and the model classifies them to neither 'crash' nor 'not crash', i.e. the predicted probability is somewhat ambiguous, e.g. close to 0.5.
      0 = instances whose ground truth label is 'crash', and the model classifies them to neither 'crash' nor 'not crash'
╒════════════╤═════════════════╤═════════════════╤══════════════════╕
│            │   0 (Predicted) │   1 (Predicted) │   Not classified │
╞════════════╪═════════════════╪═════════════════╪══════════════════╡
│ 0 (Actual) │               1 │               0 │              192 │
├────────────┼─────────────────┼─────────────────┼──────────────────┤
│ 1 (Actual) │               0 │               7 │                0 │
╘════════════╧═════════════════╧═════════════════╧══════════════════╛

label: memory - total sum = 7 + 0 + 190 + 0 + 1 + 3 = 200
╒════════════╤═════════════════╤═════════════════╤══════════════════╕
│            │   0 (Predicted) │   1 (Predicted) │   Not classified │
╞════════════╪═════════════════╪═════════════════╪══════════════════╡
│ 0 (Actual) │               7 │               0 │              190 │
├────────────┼─────────────────┼─────────────────┼──────────────────┤
│ 1 (Actual) │               0 │               1 │                2 │
╘════════════╧═════════════════╧═════════════════╧══════════════════╛

We would have this using confusion matrix

Implementation using   __NOT_CLASSIFIED__ in confusion_matrix_table (here: A, B)
label: crash
╒════════════╤═════════════════╤═════════════════╤══════════════════╕
│            │   0 (Predicted) │   1 (Predicted) │   Not classified │
╞════════════╪═════════════════╪═════════════════╪══════════════════╡
│ 0 (Actual) │               1 │               0 │              A   │
├────────────┼─────────────────┼─────────────────┼──────────────────┤
│ 1 (Actual) │               0 │               7 │                B │
╘════════════╧═════════════════╧═════════════════╧══════════════════╛

label: memory
╒════════════╤═════════════════╤═════════════════╤══════════════════╕
│            │   0 (Predicted) │   1 (Predicted) │   Not classified │
╞════════════╪═════════════════╪═════════════════╪══════════════════╡
│ 0 (Actual) │               7 │               0 │              A   │
├────────────┼─────────────────┼─────────────────┼──────────────────┤
│ 1 (Actual) │               0 │               1 │                B │
╘════════════╧═════════════════╧═════════════════╧══════════════════╛

I think this comment is helpful in the context of reviewing this PR. However, without this context, the comment feels out of place (after merging the PR).

Should I remove these comments or how would you prefer it?