Define the scope of web-platform-tests #215
Conversation
gsnedders
changed the title
| Additionally, other features can be added as | ||
| [tentative](https://web-platform-tests.org/writing-tests/file-names.html#:~:text=.tentative,-%3A%20) | ||
| tests: | ||
|
|
||
| * Web browser behavior currently being explored via an | ||
| [explainer](https://tag.w3.org/explainers/), | ||
| but without a specification yet written. | ||
| (XXX: require prototyping to have started?) | ||
|
|
||
| * Historic but unspecified features in web browsers, | ||
| especially where major browsers are interoperable, | ||
| where there exists a long-term intention to specify them. | ||
| (Note: this excludes features which have been deliberately removed from specifications. | ||
| These are explicitly out of scope.) |
There was a problem hiding this comment.
This arguably doesn't actually match up with what the documentation currently says:
Indicates that a test makes assertions not yet required by any specification, or in contradiction to some specification. This is useful when implementation experience is needed to inform the specification. It should be apparent in context why the test is tentative and what needs to be resolved to make it non-tentative.
The "in contradiction to some specification" is kinda troubling — this implies that if some spec change is made, with no expectation of implementations changing behaviour any time soon, you could change the existing tests to be tentative and add new non-tentative tests for the new behaviour. IMO, that should be against policy — anything that is tentative should be on a path to becoming non-tentative. We should probably be explicit in the scope of what can be tentative that everything which is tentative should be on a path to becoming non-tentative.
I don't think the two things listed here are a complete list of cases; we also have cases:
- where there are WG resolutions not yet applied to the spec making tests tentative (e.g.,
/css/css-overflow/line-clamp/line-clamp-005.tentative.htmlis an example of this, with [css-overflow] Is continue: discard working in the fragment tree useful? w3c/csswg-drafts#7708 (comment)), - where there is a proposal to change the spec to some other behaviour (e.g.,
svg-aam/role/role-img.tentative.html, with SVG image role should align with HTML on missing versus empty alt w3c/svg-aam#32), - where there is a proposed addition to the spec (e.g.,
/html/dom/elements/global-attributes/the-anchor-attribute-001.tentative.html, with Add anchor attribute whatwg/html#9144), - where the spec is unclear (e.g,
webdriver/tests/bidi/input/release_actions/sequence_tentative.py, with Should a navigation clean-up thebrowsing context input state map? w3c/webdriver#1859).
The other thing that stands out is:
% rg --glob '*tentative*' 'https?://((bugs\.webkit\.org|bugs\.chromium\.org|crbug\.com|issues\.chromium\.org)/|github\.com/.*/(issue|pull))' --files-without-match | wc -l
2078
Which shows it's very often not obvious "why the test is tentative and what needs to be resolved to make it non-tentative", though that is really a separate issue. (And looking at a few random tests, the commit history often doesn't make it obvious either.)
There was a problem hiding this comment.
I appreciate this analysis and I wonder if we should separate adding more clarity on tentative tests from the rest of this RFC. I.e. let's not attempt to boil the ocean :-)
|
See also #30 for an earlier issue about this, which got closed without anything ever happening. |
| implementation requirements. | ||
|
|
||
| A specification does not need to have cross-vendor support for its tests to be included in web-platform-tests. | ||
| (XXX: Should we set a bar of one-vendor support or are we okay with zero-vendor support?) |
There was a problem hiding this comment.
In theory I would be OK with zero-vendor support, since incubation can start outside browser vendors, but I worry that the necessary cleanup for unsuccessful incubations won't be done if nobody is footing the bill.
There was a problem hiding this comment.
Then again, the discussion around tentative tests below indicates that this wouldn't be a new problem.
| * Web browser behavior currently being explored via an | ||
| [explainer](https://tag.w3.org/explainers/), | ||
| but without a specification yet written. | ||
| (XXX: require prototyping to have started?) |
There was a problem hiding this comment.
I think by definition prototyping has started when there are tests, but I'm not sure what adding that requirement would mean in practice. Maybe pointers to bugs on file or intents to prototype?
| Additionally, other features can be added as | ||
| [tentative](https://web-platform-tests.org/writing-tests/file-names.html#:~:text=.tentative,-%3A%20) | ||
| tests: | ||
|
|
||
| * Web browser behavior currently being explored via an | ||
| [explainer](https://tag.w3.org/explainers/), | ||
| but without a specification yet written. | ||
| (XXX: require prototyping to have started?) | ||
|
|
||
| * Historic but unspecified features in web browsers, | ||
| especially where major browsers are interoperable, | ||
| where there exists a long-term intention to specify them. | ||
| (Note: this excludes features which have been deliberately removed from specifications. | ||
| These are explicitly out of scope.) |
There was a problem hiding this comment.
I appreciate this analysis and I wonder if we should separate adding more clarity on tentative tests from the rest of this RFC. I.e. let's not attempt to boil the ocean :-)
|
|
||
| We over-constrain what is allowed in web-platform-tests, | ||
| potentially raising the bar to change what is allowed to "submit a new RFC", | ||
| leading browser vendors to reduce what they submit to the project. |
There was a problem hiding this comment.
I think that there is no bar to submitting an RFC, any and all change suggestions should be welcome. I suspect you meant that there is a risk of raising the bar to adding more tests without sending an RFC first, because adding said tests would require us to relax some of these rules?
No description provided.