Quiz · Medium
Activation functions, backpropagation, training problems, CNNs, RNNs and LSTMs, attention and transformers, autoencoders and GANs, with figures to read.
Why are nonlinear activation functions essential in a neural network?
They make training faster on CPUs
Without them, any stack of layers is equivalent to a single linear transformation
They remove the need for a loss function
They keep the weights between 0 and 1
Composing linear maps gives another linear map, so depth adds no expressive power. Nonlinearity lets the network model curved decision boundaries.
A hidden layer should use which activation by default in a modern feed-forward or convolutional network?
Step function
Softmax
ReLU
Sigmoid
ReLU is cheap, does not saturate for positive inputs and mitigates vanishing gradients. Softmax is for the output layer of multiclass classifiers; sigmoid and step functions saturate or have zero gradients.
A network outputs one probability per class for exactly one correct class. Which output activation and loss go together?
Linear and binary cross-entropy
Softmax and categorical cross-entropy
Sigmoid and MSE
ReLU and hinge loss
Softmax makes the outputs a probability distribution; categorical cross-entropy compares it with the one-hot label.
In the figure, the sigmoid's derivative (dashed) peaks at about what value?

0.1
0.5
1.0
0.25
is largest at x = 0 where it equals 0.5 x 0.5 = 0.25. Chaining many such factors (each at most 0.25) explains vanishing gradients.
What does backpropagation compute?
The best learning rate
The validation accuracy
The gradient of the loss with respect to every parameter, using the chain rule
The new weights after each update
Backprop only produces gradients. The optimizer (SGD, Adam) uses them to update the weights.
Why are LSTMs better than simple RNNs at long sequences?
They process tokens in parallel
The additive cell-state path lets gradients flow over long spans
They have fewer parameters
They do not need training
Simple RNNs multiply many Jacobians and the gradient vanishes. The LSTM's gated, mostly additive cell-state update mitigates this. LSTMs are still sequential, not parallel.
Training loss keeps falling while validation loss rises. What should you do?
Remove the validation set
Train for many more epochs
Stop near the validation minimum (early stopping) or add regularization
Increase the learning rate
That pattern is overfitting. Early stopping at the lowest validation loss, plus dropout or weight decay, helps.
During training the loss grows larger and larger and finally becomes NaN. The most likely cause is:
A learning rate that is too large (or exploding gradients)
Using a softmax output
Too much data
A learning rate that is too small
Overshooting steps make the loss diverge. Lower the learning rate and use gradient clipping. A tiny learning rate gives slow progress, not NaNs.
What happens to a dropout layer at test (inference) time?
Units are still dropped randomly
The layer is replaced by batch norm
Half of the weights are set to zero permanently
All units are used (no dropping)
Dropout is applied only during training. With inverted dropout the activations were already scaled during training, so nothing else is needed at inference.
A dense layer maps 100 inputs to 50 outputs (with biases). How many trainable parameters does it have?
5,000
5,050
5,100
150
100 x 50 = 5,000 weights plus 50 biases = 5,050. Forgetting the biases gives 5,000.
A 3x3 convolution with padding 1 and stride 1 is applied to a 64x64 image. What is the output size?
32x32
62x62
66x66
64x64
. Padding 1 with a 3x3 kernel preserves the size; with no padding the output would be 62x62.
What does the max-pooling operation in this figure produce from the 4x4 input?

A 2x2 map of block maxima: 6, 8 / 3, 4
A 2x2 map of block averages
A 4x4 map with negatives set to zero
A 2x2 map of block minima
Each 2x2 block is replaced by its largest value: 6, 8, 3, 4. Average pooling would give different numbers. Image: Aphex34 (derivative work: Olexa Riznyk), CC BY-SA 4.0, via Wikimedia Commons.
Why do CNNs need far fewer parameters than fully connected networks for images?
They skip the backward pass
They share the same small filters across all positions
They use smaller images
They have no bias terms
Weight sharing and local connectivity: one 3x3 filter is reused at every location instead of having a separate weight for each pixel pair.
What problem do residual (skip) connections mainly address?

Difficulty training very deep networks (degradation, vanishing gradients)
Overfitting to noisy labels
Too small a batch size
Slow data loading
y = F(x) + x gives gradients a direct path and lets blocks learn small corrections. They are not a regularizer.
In the LSTM figure, the gate that decides what part of the old cell state to keep is the:

Forget gate
Input gate
Reset gate
Output gate
The forget gate multiplies the previous cell state c(t-1) by a value between 0 and 1. The input gate controls new information and the output gate controls what is exposed as h(t). Image: fdeloche, CC BY-SA 4.0, via Wikimedia Commons.
In scaled dot-product attention, why divide by the square root of d_k?
To reduce the number of heads
To keep the dot products from growing large and saturating the softmax
To mask future tokens
To make the weights sum to 1
Dot products have variance that grows with dimension, pushing the softmax toward one-hot outputs with tiny gradients. Summing to 1 is the softmax's job.
Which part of the transformer in this figure stops a position from attending to later tokens?

Add & Norm
The input embedding
Positional encoding
Masked multi-head attention in the decoder
The causal mask in the decoder's first attention block blocks future positions so generation matches training. Image: Yuening Jia, CC BY-SA 3.0, via Wikimedia Commons.
Which model family is trained to predict the next token and is used for most chat language models?
Decoder-only (GPT-style)
GAN
Autoencoder
Encoder-only (BERT-style)
Decoder-only transformers do causal (next-token) language modeling. BERT-style encoders use masked language modeling and are used for understanding tasks.
What is the role of the bottleneck 'code' in this autoencoder?

It forces the network to learn a compact representation of the input
It adds noise to the output
It stores the labels
It is discarded before training
Because the code is narrower than the input, the encoder must keep only the most informative features to reconstruct the input. Image: Michela Massi, CC BY-SA 4.0, via Wikimedia Commons.
In a GAN, what is the generator's goal?

Predict the next token
Classify real vs fake images
Minimize reconstruction error
Produce samples the discriminator labels as real
The generator is trained against the discriminator: it improves by fooling it. The discriminator is the one that classifies real vs fake. Image: Mtanti, CC BY-SA 4.0, via Wikimedia Commons.
True or False: With plain SGD, L2 regularization and weight decay are equivalent.
True
False
For plain SGD adding an L2 penalty to the loss is the same as decaying the weights each step. With adaptive optimizers like Adam they are not equivalent, hence AdamW.
You have only 500 labeled images. Which strategy is usually most effective?
Drop the validation set to get more training data
Fine-tune a network pre-trained on a large image data set
Use a fully connected network on raw pixels
Train a very deep CNN from scratch
Transfer learning reuses general visual features learned from millions of images. Training a deep model from scratch on 500 images would overfit.