Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance skore.cross_validate: specify which metric is actually used behind test_score #578

Open
sylvaincom opened this issue Oct 23, 2024 · 7 comments
Assignees

Comments

@sylvaincom
Copy link
Contributor

sylvaincom commented Oct 23, 2024

Problem

Part of epic #544

When using skore.cross_validate with linear regression for example, I can see the following scores in the legend: test_score and r2_score, while the test_score is actually the r2_score! This can be confusing for the user.

Suggested solution

It is ok the keep test_score and r2_score, but in any case, tell the user which score is actually used by default by scikit-learn for the test_score (it depends on each estimator).

@sylvaincom sylvaincom changed the title For linear regression, it is redundant to say test_r2 and test_score. for skore.cross_validate, specify which metric is actually used behind test_score Oct 23, 2024
@sylvaincom sylvaincom changed the title for skore.cross_validate, specify which metric is actually used behind test_score Enhance skore.cross_validate: specify which metric is actually used behind test_score Oct 23, 2024
@tuscland tuscland added the epic This issue represents major product increments label Oct 23, 2024
@augustebaum
Copy link
Contributor

So would we show this information in the plot? Or does it also have to be in the output of cross_validate itself (given the constraint that we should try to stick to scikit-learn's output format)?

@augustebaum
Copy link
Contributor

How exactly should we give this information? In the title of the plot?

@sylvaincom
Copy link
Contributor Author

Hmm subtitle of the plot?

@probabl-ai probabl-ai deleted a comment from sylvaincom Nov 21, 2024
@tuscland tuscland removed the epic This issue represents major product increments label Nov 25, 2024
@sylvaincom sylvaincom self-assigned this Jan 6, 2025
@sylvaincom
Copy link
Contributor Author

We need to address this, mentioned today in #1047

@sylvaincom
Copy link
Contributor Author

In the case of the regression task, let's just display R2 (⭐️) and RMSE where the star indicates that it's scikit-learn's default

WDYT? Cc @augustebaum @MarieS-WiMLDS

@MarieS-WiMLDS
Copy link
Contributor

I like the star idea because it's pretty, but is it technically possible to have an emoji in the legend? Plus, it is explicit? The other less pretty option but explicit and easy would be to print sth like R2 (sklearn default). You can do tests with Auguste and see how it seems :) !

@sylvaincom
Copy link
Contributor Author

sylvaincom commented Jan 9, 2025

(We could use unicode emoji / characters that should work in matplotlib, etc, as done in #997)

After discussing with @glemaitre, this issue is on hold, waiting on #997 and #1014 as they will change the way a cross-validation report is displayed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants