[victor-ab]

Motivation

A predictor's bins keep each categorical feature's last code, and fill_with restored that code's exceptions to the defaults and wrote the new code's for every row or group, even when the code was the same. For a model with many categorical splits a popular category has thousands of exceptions, so fills take much of a request's time: restoring the held code's exceptions one by one, even to write the same code again. The stack's first PR measured this on a large private model, with generated requests whose categories are skewed toward the most frequent.

Design

layout: fill a categorical feature only when its code changes, in one copy when dense

• A feature whose code is unchanged is left as it is. A predictor's bins start as the defaults, holding the no-category code, which has no exceptions, and fill_with is their only writer, so an unchanged code's slots already hold its words.
• When the held code's exceptions exceed an eighth of the feature's slots (RESTORE_DENSITY), the feature's slots are restored by one copy from the defaults rather than one by one; a feature's exceptions all lie in its own slots.

Evidence

predict_fixed per request on the generated requests, one thread, Apple M4 Pro. p50 change from the PR below, median of four paired rounds (A B B A, 1 s per cell), with the rounds' range; every round is faster:

	1 candidate	32	512
this PR vs #65	−18.8% (−19.2 to −17.9)	−16.0% (−23.7 to −5.5)	−14.3% (−22.4 to −12.4)

Against 3.2.2 at 1 candidate (three interleaved rounds, within ±3%): #65 101%, this PR 82%.

The skip's gain depends on how often consecutive groups repeat a feature's category, and the requests are generated, so the two parts are split on four patterns of the same requests' categorical codes: as generated (a feature's category repeats between consecutive candidates 11.2% of the time), uniform over the model's categories (7.3%), never repeating, and always repeating. On AVX-512 (GCP c4-standard-8, Intel Xeon Platinum 8581C, SMT off, all-core turbo, one pinned core; six paired rounds), p50 change:

Requests		1 candidate	32	512
as generated	skip only vs #65	−2.1% (−2.3 to −0.9)	within noise (−4.7 to +3.2)	within noise (−2.4 to +1.1)
	the copy on top	−9.8% (−10.3 to −9.7)	−8.7% (−10.5 to −7.7)	−8.4% (−9.7 to −7.8)
uniform	skip only vs #65	−1.3% (−1.7 to −0.6)	within noise (−5.5 to +2.0)	within noise (−2.0 to +1.6)
	the copy on top	−8.9% (−9.1 to −8.8)	−8.7% (−10.5 to −5.6)	−8.0% (−8.5 to −6.2)
never repeating	skip only vs #65	−1.9% (−2.8 to −1.1)	within noise (−1.6 to +1.7)	within noise (−2.5 to +1.1)
	the copy on top	−9.7% (−10.0 to −9.5)	−8.9% (−11.3 to −8.5)	−8.6% (−10.3 to −6.5)
always repeating	skip only vs #65	−34.4% (−34.8 to −34.0)	−29.6% (−29.7 to −29.4)	−29.5% (−30.1 to −29.3)
	the copy on top	within noise (−0.7 to +0.6)	within noise (−0.2 to +0.9)	within noise (−0.4 to +0.9)

On the M4 at 1 candidate (four rounds, agreeing in sign but for the last cell), in the same order: skip only −8.7%, −8.5%, −7.7% and −38.7%; the copy on top −10.5%, −10.6%, −11.2% and within noise. At 32 and 512 candidates the M4's rounds were too noisy to split.

So the copy carries the gain wherever codes change. The skip pays when categories repeat, by about a third when every one does; at the generated and uniform rates it measures close to when none repeats, so most of the skip's small gain there (2% on AVX-512, 8% on the M4) is code layout, and the skip's own gain scales with how often categories repeat.

• Outputs are bitwise equal to 3.2.2's on synthetic rows and on the generated requests.
• New test: on a feature with a threshold and categories, fills that reuse a predictor's bins, each code twice, with codes on either side of the restore threshold and crossing it both ways, leave them as fresh bins filled once. The existing bin checks compare reused fills with fresh ones over the fixtures, float32 and varying features included.
• cargo test --release --features research passes; clippy (workspace, research, library alone) and fmt.

[ukaratay]

Approving. The skip is safe because fill_with is the only writer of a predictor's bins (kernel.rs:179), and the build asserts that the no-category code has no exceptions.

Two asks:

• How much of the −18.8% is the skip, and how much the dense copy? If the copy is within noise, drop it and RESTORE_DENSITY with it.
• Where does the generated requests' skew come from? If we have logged requests, replay them or set the skew from them. As it stands the headline number depends on a generator parameter.

[victor-ab]

Both measured on the private model; the description has the tables.

1. The split is measured on four patterns of the requests' categorical codes: as generated (a feature's category repeats between consecutive candidates 11.2% of the time), uniform over the model's categories (7.3%), never repeating, and always repeating. On AVX-512 (GCP c4, six paired rounds), the copy on top of the skip is −8.0% to −9.8% at 1, 32 and 512 candidates on each pattern where codes change, and within noise when none does. On the M4 at 1 candidate it's −10.5% to −11.2%. It isn't within noise, so it stays, and RESTORE_DENSITY with it.

   The skip alone is −34.4% at 1 candidate when every category repeats. At the generated and uniform rates, though, it measures as it does when none repeats: −1.3% to −2.1% on AVX-512 and −7.7% to −8.7% on the M4, and within noise at 32 and 512. Since no code repeats in the third pattern, most of that small gain is code layout; the skip's own gain scales with how often categories repeat.

2. The skew is the generator's: each candidate's category rank is ⌊count^u⌋ − 1, with u uniform over the training counts. There are no logged requests to replay, so the four patterns bracket it instead. The copy's gain holds on each pattern where codes change, and the skip's scales with how often they repeat.