TL;DR: Feature importance tells you what a trading model relies on, but only some methods tell you what actually predicts out of sample. Mean decrease accuracy (MDA) and single feature importance (SFI), computed inside purged cross-validation, measure out-of-sample value; SHAP explains how the fitted model uses its inputs. Read them together, and treat any single dominant feature as a possible leak before treating it as an edge.
Why does feature importance matter for trading models?
A model that scores well out of sample is only half an answer. You also want to know why, for three reasons.
- Leak detection. If one feature carries almost all of the model's skill, it is often a column that accidentally contains future information, or one that is mechanically tied to the label.
- Research direction. Knowing that funding-rate features matter more than momentum on a given asset tells you where to look next.
- Simplification. Features that add nothing still add noise and extra ways to overfit. Removing them usually makes a model more stable.
López de Prado devotes chapter 8 of Advances in Financial Machine Learning (Wiley, 2018) to this, and his central warning is that feature importance should be studied before, and instead of, endless backtesting. Importance analysis does not use the backtest's P&L, so it is less exposed to selection bias.
Why is MDI the wrong default?
Tree libraries give you mean decrease impurity (MDI) for free: every time a tree splits on a feature, the reduction in impurity is credited to it. It is fast and it is everywhere, but it has two problems for trading data.
- It is in-sample. MDI measures how useful a feature was for fitting the training data. An overfit model will credit noise features generously.
- It favors some features. Impurity-based importance is known to be biased toward continuous features and features with many distinct values, which can be pure noise.
MDI is fine as a quick look at what a model did. It is not evidence that a feature predicts anything.
How does mean decrease accuracy (MDA) work?
MDA, also known as permutation importance, asks a direct question: if I scramble this feature in held-out data, how much worse does the model get?
- Fit the model on a training fold.
- Score it on the test fold.
- Randomly shuffle one feature's column in the test fold and score again.
- The drop in score is that feature's importance. Repeat for every feature and average across folds.
Because the scoring happens out of sample, MDA reflects predictive value rather than fitting convenience. A feature can even get a negative MDA if scrambling it improves the score, which is a strong hint that the model was using it to fit noise.
For trading data, the folds must be purged and embargoed. Permutation importance computed on shuffled k-fold splits inherits the same leakage as the model itself. See purged walk-forward validation for why.
The substitution effect
MDA has one well-known weakness. If two features carry the same information, such as a 14-bar RSI and a 21-bar RSI, permuting one barely hurts the model because it can lean on the other. Both look unimportant, even if the information they share is the most valuable thing in the model. scikit-learn's documentation on permutation feature importance describes this problem with correlated features and suggests clustering correlated features and keeping one per cluster.
How does single feature importance (SFI) work?
SFI sidesteps substitution by removing the competition. For each feature, train a model on that feature alone and measure its out-of-sample score.
- Strength: no substitution effect. A feature's score depends only on itself.
- Weakness: no interactions. A funding-rate feature that only matters when open interest is rising will look weak on its own.
That makes MDA and SFI complementary. A feature that is strong in SFI but weak in MDA probably has a correlated twin. A feature that is weak in SFI but strong in MDA probably works through interactions.
What does SHAP add?
SHAP, from Scott Lundberg and Su-In Lee's 2017 paper A Unified Approach to Interpreting Model Predictions, assigns each feature a contribution to each individual prediction, based on Shapley values from cooperative game theory. For tree models, the TreeExplainer in the SHAP library computes these efficiently. Averaging the absolute contributions over many predictions gives a global importance ranking.
SHAP is excellent for understanding a model: which features push a prediction up or down, and by how much, for a particular bar. What it does not do is validate the model. SHAP explains the fitted function, so if the model has memorized noise, SHAP will faithfully explain the memorized noise. Correlated features also share credit under SHAP, in ways that depend on the model's internal choices.
How do MDI, MDA, SFI and SHAP compare?
| Method | What it measures | In or out of sample | Substitution effect | Captures interactions | Best use |
|---|---|---|---|---|---|
| MDI | Impurity reduction during fitting | In-sample | Yes | Partly | Quick look at a tree model |
| MDA | Score drop when a feature is permuted | Out of sample | Yes, correlated features dilute each other | Yes | Main evidence of predictive value |
| SFI | Score of a model trained on one feature | Out of sample | No | No | Checking features in isolation |
| SHAP | Each feature's contribution to predictions | Depends on the data explained | Credit is shared | Yes | Explaining how the model behaves |
A practical reading order: MDA first for "what predicts", SFI to check for substitution, SHAP to understand direction and shape.
What feature importance does Trigr report?
Every ML optimization run in Trigr returns MDA and SFI, and SHAP whenever the SHAP step succeeds, as documented on the ML optimization docs page:
- MDA inside purged folds. Permutation importance is computed on the out-of-sample fold of each purged walk-forward split, reusing that fold's model, and scored as the drop in accuracy. It covers every feature.
- SFI inside purged folds. A fresh model is trained on each feature alone within the same splits. To keep runs affordable, SFI is computed for the top features by MDA; the MDA ranking is unaffected.
- SHAP on the final fit. SHAP values come from a tree explainer on a model fit over the full training window, so they describe the model as saved, not its out-of-sample validity.
- MDI deliberately left out. Because it is in-sample and biased, it is not reported.
The features themselves are built point-in-time from your strategy's own inputs, such as indicators, funding rate and open interest or macro series. Importance is computed for the model family the run selected on purged walk-forward results, using that family's fixed parameters.
Feature importance also feeds the leak verdict. If a single feature accounts for more than 60% of the total MDA, and its own MDA is clearly above noise level, the run is flagged. In a strategy built from ordinary indicators, one column carrying nearly all the skill is far more often a leak or a label-correlated input than a discovery. ML optimization is also available asynchronously over MCP and the API; the agents API docs cover ML runs.
How should you act on feature importance?
- Drop consistent losers. Features with near-zero or negative MDA and weak SFI across runs are candidates for removal.
- Group correlated features. If several variants of the same indicator split the importance, keep one or combine them.
- Investigate a dominant feature. Ask how it is computed and when its data becomes available before celebrating it.
- Do not re-run until it looks good. Removing features, re-running and keeping the best version is another search. Each round makes the out-of-sample numbers less trustworthy, and separate optimizer runs are not yet counted together in Trigr's Deflated Sharpe Ratio.
Backtests are not guarantees; perps are leveraged and can lose more than expected. Importance tells you what a model relies on; it does not promise that relationship will hold.
Next steps
For the full picture of labels, validation and deflated metrics around feature importance, read machine learning for trading strategies without overfitting, then see how labels are built in the triple-barrier method and meta-labeling.