Quiz · Hard
Multi-step problems on confusion-matrix metrics, base rates, gradient descent, cross-validation, leakage, ROC vs PR and diagnosing bias vs variance.
A classifier is evaluated on 1,000 examples: TP = 45, FP = 15, FN = 5, TN = 935. What are its precision and recall?
Precision 0.75, recall 0.90
Precision 0.75, recall 0.75
Precision 0.98, recall 0.90
Precision 0.90, recall 0.75
Precision = 45 / (45 + 15) = 0.75. Recall = 45 / (45 + 5) = 0.90. Swapping the two is the classic slip: precision uses FP in the denominator, recall uses FN.
Using the same counts (TP = 45, FP = 15, FN = 5, TN = 935), what is the F1 score?
0.75
0.90
0.83
0.82
F1 = 2TP / (2TP + FP + FN) = 90 / 110 = 0.818, which rounds to 0.82. If options differ only in the last digit, compute exactly: 2(0.75)(0.90)/1.65 = 0.818.
A screening test: disease prevalence 2%, sensitivity 95%, specificity 90%. A random person tests positive. About what is the probability they have the disease?
About 50%
95%
2%
About 16%
Per 1000 people: 20 sick, of whom 19 test positive; 980 healthy, of whom 98 test positive. P = 19 / (19 + 98) = 0.162. The base rate dominates, which is why most positives are false alarms.
Gradient descent on starts at with learning rate 1.1. What happens?
It converges to 0 quickly
It oscillates with growing magnitude and diverges
It converges to 0 slowly
It stays at 4
Each step gives . The sign flips and the magnitude grows by 20% per step: 4, -4.8, 5.76... Convergence needs , that is .
In 5-fold cross-validation the fold accuracies are 0.82, 0.78, 0.85, 0.80 and 0.75. What is the mean estimate?
0.80
0.78
0.82
0.85
(0.82 + 0.78 + 0.85 + 0.80 + 0.75) / 5 = 4.00 / 5 = 0.80. The median 0.80 happens to match here, but the reported cross-validation score is the mean.
Training accuracy is 91% and validation accuracy is 90% on a task where human accuracy is 99%. What is the best next step?
Add dropout or stronger regularization
Increase model capacity or train longer
Collect more validation data
Stop: the gap is small so the model is fine
Both errors are high and close together, so the problem is bias (underfitting). A larger model, better features or longer training helps. Regularization would make underfitting worse.
Which decision on a binary classifier moves a point along its ROC curve rather than changing the curve itself?
Changing the classification threshold
Adding a new feature
Switching from logistic regression to a random forest
Training on more data
Each threshold gives one (FPR, TPR) pair, tracing out the curve. Changing the model, features or data changes the scores and therefore the curve itself.
Why can a random forest's out-of-bag score be used as a validation estimate?
The forest is trained on a held-out set
Each tree is evaluated on rows it did not see in its bootstrap sample
It uses the test set for pruning
Out-of-bag rows are duplicated into the training data
About 36.8% of rows are left out of each bootstrap sample, so each row can be scored by the trees that never trained on it. That gives an honest error estimate without a separate validation set.
A feature has 200 distinct category values (e.g. zip code). Which encoding is most likely to blow up the dimensionality for a linear model?
One-hot encoding
Target (mean) encoding with proper cross-fitting
Hashing to 20 buckets
Frequency encoding
One-hot creates one column per category (200 here, thousands for real zip codes). The others give one or few columns. Target encoding needs care to avoid leaking the label.
Which situation calls for precision-recall AUC rather than ROC AUC?
Choosing k in k-means
Regression on house prices
Detecting fraud where 0.1% of transactions are fraudulent
A balanced data set of cats and dogs
With extreme imbalance, ROC AUC can stay high while precision is poor, because FPR is computed over a huge negative class. PR curves expose that.
You fit PCA on the full data, keep 10 components, then split into train and test and train a classifier. What is wrong?
PCA should keep exactly 2 components
Nothing, PCA is unsupervised so it cannot leak
PCA cannot be used before a classifier
The test data influenced the components (leakage); PCA should be fit on training data only
PCA is unsupervised but it still learns directions from the data it is shown. Fitting it on test rows lets test information shape the features and makes validation optimistic.
A logistic regression model has intercept -3 and a single feature coefficient 1.5. What is the predicted probability when x = 2?
0.27
0.95
0.5
0.73
z = -3 + 1.5(2) = 0, so p = σ(0) = 0.5. This x is exactly the decision boundary for a 0.5 threshold.
Elastic net combines which two ideas?
Bias and variance weighting
Bagging and boosting
k-means and PCA
L1 and L2 penalties
Elastic net adds both lasso and ridge penalties, giving sparsity plus stable handling of correlated features, where lasso alone tends to pick one of a correlated group arbitrarily.
A k-NN classifier with k = 1 is evaluated on its own training data. What is its training accuracy (assuming no duplicate points with different labels)?
50%
100%
It depends on feature scaling only
About the test accuracy
Each training point is its own nearest neighbor, so it is always predicted correctly. That is why training accuracy is meaningless for k = 1, and why k must be chosen with validation data.
A neural network's training loss keeps decreasing while validation loss starts to increase after epoch 12. What should you do?
Use early stopping near epoch 12 or add regularization
Increase the learning rate
Keep training: loss will recover
Remove the validation set
Rising validation loss with falling training loss is overfitting. Stop near the validation minimum, or add dropout, weight decay or more data.
Which statement about k-means inertia is true?
It is always minimized at k = 1
It increases as k increases
It never increases as k increases (for the best clustering), so it cannot choose k alone
It measures between-cluster separation
Within-cluster sum of squares keeps shrinking as k grows, down to 0 at k = n. Hence the elbow method or the silhouette score is used to pick k. Inertia is largest at k = 1.