Advantanges

• Feature complexity
• Prediction
• Relative importance
• (Multi)collinearity

Diss-advantanges

• over-fitting (over learning)

Classification

• Categorical response

Regression

• Continuous response

CART

• split (partition) data up into major chunks

• split (partition) data up into major chunks
  • maximizing change in explained deviance
  • when Gaussian error,
    • maximizing between group SS
    • minimizing SSerror

- split (**partition**) data up into __major__ chunks

• split (partition) data up into major chunks

• split (partition) data up into major chunks

• split these subsets

• split these subsets

• split these subsets

• split these subsets

• recursively partition (split)
• decision tree
• simple trees tend to overfit.
  • error is fitted along with the model

Pruning

• reduce overfitting
  • deviance at each terminal node (leaf)

Predictions

• partial plots

R packages

• simple CART

library(tree)

• an extension that facilitates (some) non-gaussian errors

library(rpart)

Limitations

• crude overfitting protection
• low resolution
• limited error distributions
• little scope for random effects

Boosting

• machine learning meets predictive modelling
  
• ensemble models
  • sequence of simple Trees (10,000+ trees)
  • built to predict residuals of previous tree
  • shrinkage
  • produce excellent fit

Over fitting

• over vs under fitting
• residual error vs precision
• minimizing square error loss

minimizing square error loss

• test (validation) data
  • 75% train, 25% test
• out of bag
  • 50% in, 50% out
• cross validation
  • 3 folds

Over fitting

Predictions

Variable importance

##    var  rel.inf
## x1  x1 55.06919
## x2  x2 44.93081

Pseudo \(R^2\)

\[ 1-\left(\frac{\sum (y_i - E(y))^2}{\sum(y_i - \bar{y})}\right) \]

## [1] 0.5650695