Machine Learning Basics: Models, Metrics & Overfitting

Supervised: learn from labeled examples (inputs with known answers). Unsupervised: find structure in unlabeled data (clusters, lower dimensions). Reinforcement: an agent learns by trial and error from rewards.

Classification predicts a category (spam or not spam). Regression predicts a continuous number (house price).

Training: fit the model's parameters. Validation: tune hyperparameters and choose between models. Test: a final, untouched estimate of performance on new data.

Overfitting: the model memorizes the training data, including noise; low training error but high test error. Underfitting: the model is too simple to capture the pattern; high error on both.

Bias is error from overly simple assumptions (underfitting). Variance is error from sensitivity to the particular training set (overfitting). Making a model more complex usually lowers bias and raises variance.

More training data, regularization, a simpler model, cross-validation, early stopping, dropout (neural nets), and pruning or limiting depth (trees).

Both add a penalty on weight size to the loss. L1 adds \lambda\sum|w_i| and can push weights to exactly zero (feature selection). L2 adds \lambda\sum w_i^2 and shrinks all weights smoothly.

Split the data into k folds; train on k - 1 folds and validate on the remaining one, rotating k times, then average the scores. It gives a more reliable estimate than a single split.

TP: predicted positive, actually positive. FP: predicted positive, actually negative (Type I error). TN: predicted negative, actually negative. FN: predicted negative, actually positive (Type II error).

Precision = \dfrac{TP}{TP + FP} (of the predicted positives, how many were right) Recall = \dfrac{TP}{TP + FN} (of the actual positives, how many were found)

0.8

F_1 = 2 \cdot \dfrac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} , the harmonic mean of precision and recall

On imbalanced data. If 99% of transactions are legitimate, a model that always predicts 'legitimate' is 99% accurate but catches no fraud. Use precision, recall, F1 or ROC AUC.

The ROC curve plots true positive rate against false positive rate across all thresholds. AUC is the area under it: 1.0 is perfect, 0.5 is random guessing.

\hat{y} = w_0 + w_1 x_1 + \cdots + w_n x_n , usually fit by minimizing mean squared error \text{MSE} = \dfrac{1}{n}\sum (y_i - \hat{y}_i)^2

A classifier that passes a linear combination through the sigmoid \sigma(z) = \dfrac{1}{1 + e^{-z}} to get a probability between 0 and 1, trained with log loss (cross-entropy).

\theta \leftarrow \theta - \alpha \nabla_\theta J(\theta) , stepping against the gradient of the loss with learning rate \alpha

Predicts using the k closest training points: majority vote for classification, average for regression. No training phase, but slow on large data and needs scaled features.

Unsupervised: pick k centroids, assign each point to the nearest one, move each centroid to the mean of its points, and repeat until assignments stop changing.

Decision tree: splits features into if/else rules; easy to read, prone to overfitting. Random forest: many trees on bootstrap samples, averaged (bagging); lowers variance. Gradient boosting: trees built one after another, each fixing the previous errors (XGBoost, LightGBM).

Standardization: z = \dfrac{x - \mu}{\sigma} (mean 0, SD 1). Min-max: x' = \dfrac{x - \min}{\max - \min} (range 0 to 1).

Parameters are learned from data (weights). Hyperparameters are set before training (learning rate, k in k-NN, tree depth, number of layers).

When information from outside the training data, such as the test set or the future, leaks into training, giving unrealistically good scores. Example: scaling with statistics from the full dataset before splitting.

Principal component analysis reduces dimensionality by projecting data onto the directions (eigenvectors of the covariance matrix) that capture the most variance.

It adds nonlinearity; without it, stacked layers collapse into one linear model. ReLU is f(x) = \max(0, x) .