ml_meta_perf.data

  1"""Loading and shaping of the meta-dataset.
  2
  3The meta-dataset has one row per (dataset, model) pair. Two groups of columns matter
  4and they behave very differently, which is why they are named separately here:
  5
  6* dataset features are constant across every row of a given dataset, so on their own
  7  they can only ever predict a per-dataset constant;
  8* model features vary with the model, and one of them -- ``Processing Units Number`` --
  9  also varies with the dataset, since it is a function of the dataset's shape. The
 10  meta-dataset pipeline is the source of this schema; this module loads the generated
 11  corpus and keeps the package-level column contract explicit.
 12
 13That asymmetry is the whole point of the two-equation comparison, so the split is
 14part of the public API rather than something each caller re-derives.
 15
 16**Why there are eighteen features when the equation uses fewer.**
 17
 18The two numbers answer different questions, asked at different stages, and reading the
 19second as a criticism of the first is the most natural mistake to make about this study.
 20
 21*Designing the corpus* comes first, before any equation exists and before anyone knows
 22which terms will be worth having. The requirement there is **identification**: the
 23features must name every dataset and every learner the corpus contains, because two rows
 24sharing a feature vector are two rows no equation over those features can ever tell
 25apart, and a difference between them is then unexplainable rather than merely
 26unexplained. The right move at that stage is a wide feature set chosen for coverage. This
 27corpus meets the requirement exactly -- the twelve dataset features give twenty distinct
 28vectors for twenty datasets, and the six model features separate all twenty-five learners
 29on every dataset, with zero ambiguous rows of 484. ``tests/test_model_features.py`` pins
 30both.
 31
 32Two caveats belong with that claim rather than after it. The model side is **joint**, not
 33standalone: five of the six are constant per model and separate only 19 of the 25 on their
 34own -- ``FT-Transformer``/``TabNet``/``TabTransformer``, ``LightGBM_RF``/``XGBoost``,
 35``DNN``/``MLP``, ``TabICL``/``TabPFN`` and ``BernoulliNB``/``GaussianNB`` collide -- and it
 36is ``Processing Units Number``, which varies with the dataset, that breaks those ties. So
 37the six identify a learner *on a given dataset*. And identification was bought with
 38redundancy: descriptor columns can be strongly correlated, and
 39``log(inst_to_attr) + log(nr_attr)`` **is** ``log(nr_inst)`` to 2e-15, because
 40``inst_to_attr`` is defined as their ratio.
 41
 42*Fitting the equation* comes second, and its criterion is not coverage but **compression**.
 43An equation is a statement about families of datasets and families of learners, not about
 44individuals, so it is expected to need fewer features as it gets better -- and the
 45redundancy above is part of what it is compressing away. E3 searches all eighteen and keeps
 46fewer (``tests/test_model_features.py`` pins that it does). That is the mechanism working,
 47not a shortfall in the corpus, and an equation that used all eighteen would be one that had
 48failed to generalise.
 49
 50So a column may earn its place at either stage: a descriptor that adds little to a fit can
 51still be what keeps two learners distinguishable. **Do not read absence from the equation as
 52evidence against a feature**, and do not read presence as robust either -- which features an
 53equation keeps is a property of one configuration and one length. The search, not the
 54configuration, decides which features an equation uses: every equation is offered every
 55feature it is allowed.
 56
 57**``nr_inst`` describes the source dataset, not the training set.** Every model was
 58trained on a stratified sample capped at 100,000 rows, and ten of the twenty datasets are
 59larger than that -- up to seven million. Nothing in the CSV records the sampled size,
 60because it is the same cap for every dataset above it. So ``nr_inst`` and
 61``inst_to_attr`` are properties of the corpus a dataset was drawn from, and no statement
 62about "more training data" can be tested against them.
 63
 64Study chapter: [1. The dataset][study-chapter] -- the rationale, in
 65prose, with the figures.
 66
 67[study-chapter]: https://github.com/mariolpantunes/ml-meta-perf/blob/main/assets/docs/01-dataset.md
 68"""
 69
 70from __future__ import annotations
 71
 72from pathlib import Path
 73
 74import numpy as np
 75import polars as pl
 76
 77DATASET_COLUMN = "Dataset"
 78MODEL_COLUMN = "Model"
 79TARGET_COLUMN = "MCC"
 80
 81DATASET_FEATURES: tuple[str, ...] = (
 82    "class_ent",
 83    "eq_num_attr",
 84    "gravity",
 85    "inst_to_attr",
 86    "nr_attr",
 87    "nr_bin",
 88    "nr_class",
 89    "nr_cor_attr",
 90    "nr_inst",
 91    "nr_norm",
 92    "nr_outliers",
 93    "ns_ratio",
 94)
 95
 96#: The six generated columns an equation may use to describe a *learner*.
 97#:
 98#: This is the package-level schema expected in ``meta_dataset.csv``. The generation logic
 99#: lives in ``meta_dataset_pipeline``; the package keeps only the validated feature contract
100#: used by loading, fitting, plotting and reporting.
101MODEL_FEATURES: tuple[str, ...] = (
102    "Processing Units Number",
103    "Model Capability",
104    "Solution Stochasticity",
105    "Loss Margin Behaviour",
106    "Input Distribution Modelling",
107    "Fitting Regime",
108)
109
110
111#: Study-specific ordinal assignment of the ten learner families.
112#:
113#: This is the **provenance** of the ``Model Capability`` column, which the CSV carries like
114#: any other feature. It is kept here, and checked against the CSV by ``test_data``, so the
115#: column can be regenerated and so a reader can see where the numbers came from rather than
116#: finding twenty-five unexplained integers in a data file.
117#:
118#: Published tabular-ML comparisons motivate family distinctions; the exact numerical
119#: ordering is a heuristic study assumption, not calibrated capacity or expected MCC.
120#: Encoding sensitivity controls assess the consequences for the retained form.
121#: The assigned values below remain fixed across datasets.
122MODEL_CAPABILITY: dict[str, int] = {
123    "naive bayes": 1,
124    "discriminant": 2,
125    "linear": 3,
126    "instance": 4,
127    "single tree": 5,
128    "generic NN": 6,
129    "tabular NN": 7,
130    "bagged trees": 8,
131    "boosted trees": 9,
132    "tabular foundation": 10,
133}
134
135#: The four asserted mechanism ordinals, by learner.
136#:
137#: These are the **provenance** of four columns the CSV carries like any other feature. They
138#: live here, and are checked against the CSV by ``test_data``, so the columns can be
139#: regenerated and a reader can see where the numbers came from.
140#:
141#: Each is defined for all twenty-five learners, so none carries a "this learner has no such
142#: thing" sentinel, and every rung is occupied and positive, so the whole grammar is defined
143#: on all of them. That is what distinguishes them from the sixty-one hyperparameter
144#: descriptors measured and rejected during corpus design: a hyperparameter a learner does not have
145#: has no value, and encoding that absence as zero collapses applicability into magnitude.
146#:
147#: **They are asserted, not measured**, and carry chapter 4's caveat in full: a term over one
148#: of them is evidence about the ordering claimed here, not about a quantity anyone observed.
149
150#: How deep randomisation reaches into the fitted solution, low to high.
151#:
152#: 1 deterministic given the data | 2 stochastic optimisation (random init or shuffling) |
153#: 3 randomised over training examples (bootstrap) | 4 randomised over features (subspace or
154#: column subsampling) | 5 randomised over the split or parameter values themselves.
155#:
156#: Asserted from published descriptions of the learners: Breiman (1996) on bagging, Ho (1998)
157#: on the random subspace method, Breiman (2001) on random forests, and Geurts, Ernst &
158#: Wehenkel (2006) on extremely randomized trees. The top two rungs are what separate ``DT``
159#: from ``ExtraTree`` and ``LightGBM_RF`` from ``LightGBM_ExtraTrees`` -- pairs that no
160#: measured descriptor in this corpus, and none of sixty-one proposed ones, can tell apart.
161SOLUTION_STOCHASTICITY: dict[str, int] = {
162    "BernoulliNB": 1,
163    "DT": 1,
164    "GaussianNB": 1,
165    "KNN": 1,
166    "LDA": 1,
167    "LR": 1,
168    "LinearSVC": 1,
169    "QDA": 1,
170    "Ridge": 1,
171    "DNN": 2,
172    "FT-Transformer": 2,
173    "MLP": 2,
174    "PassiveAggressive": 2,
175    "Perceptron": 2,
176    "SGD": 2,
177    "TabICL": 2,
178    "TabNet": 2,
179    "TabPFN": 2,
180    "TabTransformer": 2,
181    "AdaBoost": 3,
182    "Bagging": 3,
183    "LightGBM_RF": 4,
184    "XGBoost": 4,
185    "ExtraTree": 5,
186    "LightGBM_ExtraTrees": 5,
187}
188
189
190#: How hard the objective penalises points far from the decision boundary.
191#:
192#: 1 squared error or an impurity criterion | 2 logistic / cross-entropy | 3 exponential |
193#: 4 hinge | 5 the perceptron criterion.
194#:
195#: The standard robustness ordering over losses. This is the weakest claim of the four --
196#: placing the perceptron criterion above hinge is a choice rather than a consensus -- and
197#: chapter 4's caveat about ``Model Capability`` applies here with more force.
198LOSS_MARGIN_BEHAVIOUR: dict[str, int] = {
199    "Bagging": 1,
200    "DT": 1,
201    "ExtraTree": 1,
202    "KNN": 1,
203    "LDA": 1,
204    "QDA": 1,
205    "Ridge": 1,
206    "BernoulliNB": 2,
207    "DNN": 2,
208    "FT-Transformer": 2,
209    "GaussianNB": 2,
210    "LR": 2,
211    "LightGBM_ExtraTrees": 2,
212    "LightGBM_RF": 2,
213    "MLP": 2,
214    "TabICL": 2,
215    "TabNet": 2,
216    "TabPFN": 2,
217    "TabTransformer": 2,
218    "XGBoost": 2,
219    "AdaBoost": 3,
220    "LinearSVC": 4,
221    "PassiveAggressive": 4,
222    "SGD": 4,
223    "Perceptron": 5,
224}
225
226
227#: How much of P(x) the learner commits to modelling.
228#:
229#: 1 discriminative, models P(y|x) only | 2 instance-based, retains the sample and models no
230#: density | 3 generative assuming conditional independence | 4 generative with a shared
231#: covariance | 5 generative with a per-class covariance.
232#:
233#: Ng & Jordan (2001) on discriminative versus generative classifiers, and the classical
234#: LDA/QDA covariance hierarchy. **Lopsided**: the discriminative reference contains most rows; the other categories
235#: hold 20-40 each, so three of its five rungs sit below the ten-percent-of-rows floor the
236#: z-score cap imposes elsewhere. Every value is a real reading -- there is no
237#: not-applicable sentinel -- but a term over this column speaks mostly about one rung.
238INPUT_DISTRIBUTION_MODELLING: dict[str, int] = {
239    "AdaBoost": 1,
240    "Bagging": 1,
241    "DNN": 1,
242    "DT": 1,
243    "ExtraTree": 1,
244    "FT-Transformer": 1,
245    "LR": 1,
246    "LightGBM_ExtraTrees": 1,
247    "LightGBM_RF": 1,
248    "LinearSVC": 1,
249    "MLP": 1,
250    "PassiveAggressive": 1,
251    "Perceptron": 1,
252    "Ridge": 1,
253    "SGD": 1,
254    "TabICL": 1,
255    "TabNet": 1,
256    "TabPFN": 1,
257    "TabTransformer": 1,
258    "XGBoost": 1,
259    "KNN": 2,
260    "BernoulliNB": 3,
261    "GaussianNB": 3,
262    "LDA": 4,
263    "QDA": 5,
264}
265
266
267#: How the parameters are reached.
268#:
269#: 1 closed form | 2 batch iterative | 3 mini-batch stochastic | 4 per-sample online |
270#: 5 amortised, fitted in-context at prediction time.
271FITTING_REGIME: dict[str, int] = {
272    "BernoulliNB": 1,
273    "GaussianNB": 1,
274    "KNN": 1,
275    "LDA": 1,
276    "QDA": 1,
277    "Ridge": 1,
278    "AdaBoost": 2,
279    "Bagging": 2,
280    "DT": 2,
281    "ExtraTree": 2,
282    "LR": 2,
283    "LightGBM_ExtraTrees": 2,
284    "LightGBM_RF": 2,
285    "LinearSVC": 2,
286    "XGBoost": 2,
287    "DNN": 3,
288    "FT-Transformer": 3,
289    "MLP": 3,
290    "SGD": 3,
291    "TabNet": 3,
292    "TabTransformer": 3,
293    "PassiveAggressive": 4,
294    "Perceptron": 4,
295    "TabICL": 5,
296    "TabPFN": 5,
297}
298
299#: The four ordinals by name, for regeneration and for the test that checks them.
300MODEL_ORDINALS: dict[str, dict[str, int]] = {
301    "Solution Stochasticity": SOLUTION_STOCHASTICITY,
302    "Loss Margin Behaviour": LOSS_MARGIN_BEHAVIOUR,
303    "Input Distribution Modelling": INPUT_DISTRIBUTION_MODELLING,
304    "Fitting Regime": FITTING_REGIME,
305}
306
307ALL_FEATURES: tuple[str, ...] = DATASET_FEATURES + MODEL_FEATURES
308
309#: Learner family for each model in the meta-dataset. Model *features* describe capacity
310#: and cost; they do not say what kind of learner a row refers to, and the tabular-ML
311#: literature states its guidance in exactly those terms ("prefer tree ensembles"), so
312#: checking that guidance against this corpus needs the taxonomy written down.
313#:
314#: The split between ``tabular NN`` and ``generic NN`` is the one that matters: an
315#: architecture designed for tabular data and a plain multilayer perceptron are both
316#: "deep learning" and behave nothing alike here.
317MODEL_FAMILY: dict[str, str] = {
318    "TabICL": "tabular foundation",
319    "TabPFN": "tabular foundation",
320    "Bagging": "bagged trees",
321    "DT": "single tree",
322    "ExtraTree": "single tree",
323    "XGBoost": "boosted trees",
324    "LightGBM_RF": "boosted trees",
325    "LightGBM_ExtraTrees": "boosted trees",
326    "AdaBoost": "boosted trees",
327    "TabNet": "tabular NN",
328    "FT-Transformer": "tabular NN",
329    "TabTransformer": "tabular NN",
330    "DNN": "generic NN",
331    "MLP": "generic NN",
332    "KNN": "instance",
333    "Ridge": "linear",
334    "LR": "linear",
335    "LinearSVC": "linear",
336    "PassiveAggressive": "linear",
337    "SGD": "linear",
338    "Perceptron": "linear",
339    "QDA": "discriminant",
340    "LDA": "discriminant",
341    "BernoulliNB": "naive bayes",
342    "GaussianNB": "naive bayes",
343}
344
345#: Families the tabular-ML literature groups together as "tree-based".
346TREE_FAMILIES = ("bagged trees", "single tree", "boosted trees")
347
348#: Families that are neural networks, split by whether the architecture targets tabular
349#: data specifically.
350NEURAL_FAMILIES = ("tabular NN", "generic NN")
351
352#: Plain-language readings of each feature, used wherever a fitted equation is turned into
353#: written guidance. Without these a "best practice" degenerates into restating a column
354#: name, which is not advice anyone can act on.
355FEATURE_GLOSSARY: dict[str, str] = {
356    "class_ent": "class entropy (how evenly the labels are spread)",
357    "eq_num_attr": "equivalent number of attributes (effective feature count)",
358    "gravity": "gravity (separation between the majority and minority class centres)",
359    "inst_to_attr": "instances per attribute",
360    "nr_attr": "number of attributes",
361    "nr_bin": "number of binary attributes",
362    "nr_class": "number of classes",
363    "nr_cor_attr": "proportion of correlated attribute pairs",
364    "nr_inst": "number of instances in the source dataset (before sampling)",
365    "nr_norm": "number of normally distributed attributes",
366    "nr_outliers": "number of attributes containing outliers",
367    "ns_ratio": "noise-to-signal ratio",
368    "Processing Units Number": "family-specific structural-size proxy",
369    "Model Capability": "heuristic study-specific learner-family code (1-10)",
370    "Solution Stochasticity": "assigned stochasticity reference category (1-5)",
371    "Loss Margin Behaviour": "assigned reference loss or decision-criterion category (1-5)",
372    "Input Distribution Modelling": "assigned distribution-modelling category (1-5)",
373    "Fitting Regime": "assigned fitting-regime category (1-5)",
374}
375
376#: The corpus, relative to the repository root. It lives in ``dataset/`` beside the pipeline that
377#: regenerates it, not inside the package: it is the study's data, versioned on its own.
378DATASET_RELATIVE_PATH = Path("dataset") / "meta_dataset.csv"
379
380
381def _default_path() -> Path:
382    """The corpus in the source checkout this module runs from, else under the working directory.
383
384    The first covers an editable install and ``PYTHONPATH=src``. The second covers a regular
385    install, whose module sits in `site-packages` with no repository around it -- which is why
386    a path derived from this file alone failed there -- run from a clone, as CI does. Anywhere
387    else, pass ``--data``. When neither exists the checkout path is returned, so the error
388    `load` raises names where the file was expected.
389    """
390    checkout = Path(__file__).resolve().parents[2] / DATASET_RELATIVE_PATH
391    working = Path.cwd() / DATASET_RELATIVE_PATH
392    return next((path for path in (checkout, working) if path.is_file()), checkout)
393
394
395DEFAULT_PATH = _default_path()
396
397
398class SchemaError(ValueError):
399    """The CSV does not carry the columns the meta-model needs."""
400
401
402def load(path: str | Path | None = None) -> pl.DataFrame:
403    """Read the meta-dataset and check it has the expected shape.
404
405    Every feature is cast to Float64. The integer-valued columns (``nr_attr``,
406    ``nr_class``, ...) are still continuous as far as the equations are concerned,
407    and carrying two dtypes through the term library buys nothing.
408    """
409    resolved = Path(path) if path is not None else DEFAULT_PATH
410    if not resolved.is_file():
411        raise SchemaError(f"meta-dataset not found: {resolved}")
412
413    frame = pl.read_csv(resolved)
414    expected = (DATASET_COLUMN, MODEL_COLUMN, *ALL_FEATURES, TARGET_COLUMN)
415    missing = [column for column in expected if column not in frame.columns]
416    if missing:
417        raise SchemaError(f"missing columns: {missing}")
418
419    frame = frame.select(expected).with_columns(
420        [pl.col(column).cast(pl.Float64) for column in (*ALL_FEATURES, TARGET_COLUMN)]
421    )
422
423    nulls = sum(frame.null_count().row(0))
424    if nulls:
425        raise SchemaError(f"meta-dataset contains {nulls} null values")
426    return frame
427
428
429def aggregate_by_dataset(frame: pl.DataFrame) -> pl.DataFrame:
430    """Collapse to one row per dataset: the dataset features plus the mean MCC.
431
432    **E1 is no longer fitted against this**, and the reason is worth recording. Least
433    squares on a predictor that is constant within a group lands on that group's mean
434    either way, so aggregating first looked free. It was not: it put E1's R2 on a
435    20-point denominator, which cannot be compared with E3's 484-row one, and the 0.506
436    it produced read as *better* transfer than E3's 0.466 when on the common scale it is
437    0.217. All three equations are now fitted on all 484 rows.
438
439    Kept because the aggregated view is still the right one for describing the corpus --
440    20 datasets, one row each -- and because chapter 6 quotes the comparison.
441    """
442    return (
443        frame.group_by(DATASET_COLUMN)
444        .agg(
445            [pl.col(column).first() for column in DATASET_FEATURES]
446            + [
447                pl.col(TARGET_COLUMN).mean().alias(TARGET_COLUMN),
448                pl.len().alias("n_models"),
449            ]
450        )
451        .sort(DATASET_COLUMN)
452    )
453
454
455def columns_as_arrays(frame: pl.DataFrame, features: tuple[str, ...]) -> dict[str, np.ndarray]:
456    """Extract the named columns as a mapping of float arrays, the form terms consume."""
457    return {name: frame[name].to_numpy().astype(np.float64) for name in features}
458
459
460def target(frame: pl.DataFrame) -> np.ndarray:
461    """The MCC column as a float array."""
462    return frame[TARGET_COLUMN].to_numpy().astype(np.float64)
463
464
465def groups(frame: pl.DataFrame, column: str) -> np.ndarray:
466    """The grouping labels used by the leave-one-out splitters."""
467    return frame[column].to_numpy()
468
469
470def corpus_summary(frame: pl.DataFrame) -> pl.DataFrame:
471    """The shape of the corpus: how many rows, groups, features, and how many cells absent.
472
473    Every fact a chapter would otherwise state about the size of the meta-dataset, computed
474    from the file rather than transcribed from it. Counts only -- the distribution of the
475    target is `target_summary`, kept separate so each table can print at its own precision
476    and neither has to render a count as ``484.0000``.
477    """
478    datasets = int(frame[DATASET_COLUMN].n_unique())
479    models = int(frame[MODEL_COLUMN].n_unique())
480    return pl.DataFrame(
481        [
482            {"quantity": "rows", "count": frame.height},
483            {"quantity": "datasets", "count": datasets},
484            {"quantity": "models", "count": models},
485            {"quantity": "cells absent from the dataset-by-model grid", "count": datasets * models - frame.height},
486            {"quantity": "dataset features", "count": len(DATASET_FEATURES)},
487            {"quantity": "model features", "count": len(MODEL_FEATURES)},
488        ],
489        schema={"quantity": pl.String, "count": pl.Int64},
490    )
491
492
493def target_summary(frame: pl.DataFrame) -> pl.DataFrame:
494    """How MCC is distributed across the corpus, including how much of it is pinned.
495
496    The counts at exactly 0 and exactly 1 are here because they are the reason MAE and not
497    SMAPE is the reported error, and a reader checking that argument should be able to see
498    the counts it rests on. Their frequencies depend on the corpus; the negative
499    row is reported separately.
500
501    Not included, because it cannot be: the variation across the five seeds from which each
502    row's maximum was selected. That lives upstream, in the corpus builder, and chapter 1
503    cites it as an external audit rather than pretending this file can recompute it.
504    """
505    values = target(frame)
506    pinned = [("at exactly 1", values == 1.0), ("at exactly 0", values == 0.0), ("below 0", values < 0.0)]
507    rows: list[dict[str, object]] = [
508        {"quantity": "mean", "MCC": float(values.mean()), "rows": frame.height},
509        {"quantity": "standard deviation", "MCC": float(values.std()), "rows": frame.height},
510        {"quantity": "minimum", "MCC": float(values.min()), "rows": int((values == values.min()).sum())},
511        {"quantity": "maximum", "MCC": float(values.max()), "rows": int((values == values.max()).sum())},
512    ]
513    rows += [{"quantity": label, "MCC": float("nan"), "rows": int(mask.sum())} for label, mask in pinned]
514    return pl.DataFrame(rows, schema={"quantity": pl.String, "MCC": pl.Float64, "rows": pl.Int64})
515
516
517def missing_cells(frame: pl.DataFrame) -> pl.DataFrame:
518    """Which datasets are short of models, and how large those datasets are.
519
520    The 24 absent cells are **not** missing at random, and the table says so on its face
521    rather than in a sentence beside it: the datasets with models missing are the smallest
522    ones in the corpus. Reported per dataset with its instance count, so a reader can see the
523    pattern instead of being told about it.
524    """
525    models = int(frame[MODEL_COLUMN].n_unique())
526    counts = (
527        frame.group_by(DATASET_COLUMN)
528        .agg(pl.len().alias("models_present"), pl.col("nr_inst").first().alias("nr_inst"))
529        .filter(pl.col("models_present") < models)
530        .sort("nr_inst")
531    )
532    return counts.with_columns((models - pl.col("models_present")).alias("models_absent")).select(
533        pl.col(DATASET_COLUMN).alias("dataset"), "nr_inst", "models_absent"
534    )