[victor-ab]

Motivation

LightGBM models load only after an export through Treelite's Python package into a checkpoint, an extra conversion step and toolchain for every model. This reads LightGBM's own text format (model.txt) directly, into the decoders' shared representation, with Treelite 4.7.2's semantics, so either route gives the same decoded model and bitwise equal predictions.

Design

load: read native LightGBM text models

• ModelFormat::LightGbmText, for from_bytes and from_reader; Forest::load recognizes .txt paths.
• The normalization is Treelite 4.7.2's LightGBM frontend: float64 comparisons and leaves; counts as training weights; Treelite's right-first node numbering before the shared category encoding and heavy-first order; the objective names LightGBM saves and float32 sigmoid alpha; average_output selects average aggregation.
• Missing values follow Treelite, not LightGBM's evaluator: at None and Zero missing-type splits NaN compares as zero, ignoring the default bit, where LightGBM routes Zero-type NaN and zeros by that bit; categorical NaN always goes right. docs/treelite-loading.md spells out where predictions can differ from native LightGBM.
• Rejected as unsupported: linear trees, more than one tree per iteration, and objectives outside identity/sigmoid (multiclass, Poisson, gamma, Tweedie, the sqrt transform). Rejected as malformed where Treelite is more lenient: duplicate keys, out-of-order trees, out-of-range children, cycles and unreachable nodes, trees without decision_type, leaf_count or internal_count (which LightGBM always writes), num_leaves=0 (which Treelite's builder mishandles), and a categorical split index shared by several splits, which LightGBM never writes and which would let a small file rescan the same bitset words.
• Caps: the text is bounded by max_json_bytes, as the JSON decoder's document is (LoadOptions is not non_exhaustive, so a new field would break struct literals), each decoded array by max_binary_array_bytes, and the model's category entries, counted over every tree before their bitset words expand, by half of max_json_bytes, as a JSON document's are at two bytes per category. A tree's node reservation needs leaf_value text to back it, and arrays are checked against their text before the cap, so a corrupt count fails as malformed rather than as a limit.
• The validation, topology checks and bounded document read are shared with the other decoders.

Evidence

• Fixtures from a locked generator (LightGBM 4.7.0, Treelite 4.7.2): native text and Treelite checkpoint pairs for trained models (binary, cross entropy, ranking, random-forest averaging, categorical, an absent objective, single leaves) and hand-written trees for every decision bit, missing type and category words beyond 32 and 64. Every pair's decoded models compare equal bit for bit; one fixture predicts through path, bytes and reader dispatch.
• Malformed, unsupported and capped inputs are rejected with their error kinds, including a model cut at every line before its first leaf values. A header claiming 134 million leaves with one leaf value fails with under 64 KiB allocated, even with the array cap raised. A 60 MiB text of full category words, each tree's list one bit off, stops at a 328 MiB heap peak, the text included, where it reached 913 MiB before the model-wide count.
• Checked locally against a small real LightGBM model (one tree, no categorical splits): its native load and Treelite 4.7.2's checkpoint give equal forests and bitwise equal predictions on 128,000 rows, constant and varying.
• Treelite-route loading is unchanged: outputs on the private model are bitwise equal to 3.2.2's.
• cargo test --release --features research passes; clippy (workspace, research, library alone) and fmt.

[ukaratay]

Native text is a good addition, and requiring leaf text to back the node count is the right way to bound the reservation. One thing is wrong, inline.

On Zero-type missing values: keep Treelite's handling as you have it, documented. The Treelite route already behaves this way, and rejecting those splits would drop models users load today.

[ukaratay]

category_entries resets per tree, so the bitset expansion has no model-wide bound. An 11-byte cat_threshold word expands to 32 categories, 128 bytes, and lists that differ by one bit per tree defeat the intern. I measured it: a 63 MB file, under the default 64 MiB text cap, peaked at 822 MB of heap before max_rank_table_bytes rejected it. Binary holds 4 bytes in the file per category, and JSON at least 2. Count entries across the whole model against one cap.

[victor-ab]

Fixed: entries are now counted over the whole model, against max_json_bytes / 2. That's the bound a JSON document under the same cap has, at two bytes per category. I didn't use max_binary_array_bytes / 4 model-wide: a private model with many categorical splits has 127% of that in total, though under 1% of it in any one tree. The Treelite route loads that model under default options, and the native one would have refused it.

A file of your shape (60 MiB, 114 trees of 50,000 full words, each list one bit off) peaked at 913 MiB on the previous head before max_rank_table_bytes rejected it. It now stops at tree 20, at a 328 MiB peak including the text. A test loads four trees of full words at exactly the bound and rejects them with the bound one entry lower, though each tree is within it on its own.

Zero-type missing values: kept as they are.

[ukaratay]

The model-wide count against max_json_bytes / 2 is the right bound, and the boundary test pins it.