RNN | Test Your Understanding

Question 1

What fundamental limitation of feedforward networks do RNNs address?

Accepted Answer

They have no mechanism to carry information across sequential inputs

Question 2

What does the hidden state h_t represent in a vanilla RNN?

Accepted Answer

A compressed, task-dependent summary of all inputs seen up to time t

Question 3

Why are weights shared across all time steps in an RNN?

Accepted Answer

So the same transformation applies at every position, enabling variable-length sequences and keeping parameter count constant

Question 4

What does 'unfolding' an RNN mean?

Accepted Answer

Laying the recurrence flat across time to create an equivalent deep feedforward network with shared weights

Question 5

What is the initial hidden state h_0 typically set to?

Accepted Answer

The zero vector

Question 6

In Elman's (1990) architecture, what feeds back to the hidden layer?

Accepted Answer

The hidden layer's own previous activations (via context units)

Question 7

Which equation correctly describes the vanilla RNN hidden state update?

Accepted Answer

h_t = tanh(W_xh × x_t + W_hh × h_{t-1} + b_h)

Question 8

In BPTT, why does the gradient of h_t receive contributions from BOTH the output at time t AND from h_{t+1}?

Accepted Answer

Because h_t affects the loss both directly (through o_t) and indirectly (through all future hidden states h_{t+1}...h_T)

Question 9

What condition on the spectral radius of W_hh causes gradients to vanish?

Accepted Answer

ρ(W_hh) < 1

Question 10

What is truncated BPTT, and what trade-off does it make?

Accepted Answer

It backpropagates only within fixed-length chunks while carrying hidden state forward across chunks, trading gradient accuracy for computational feasibility

Question 11

How does gradient clipping work?

Accepted Answer

It computes the global gradient norm and rescales the entire gradient vector if it exceeds a threshold, preserving direction

Question 12

What is the Jacobian product in the context of BPTT, and why is it important?

Accepted Answer

The product of per-step Jacobian matrices (W_hh^T × diag(1-h²)) accumulated backward through time, which determines whether gradients vanish or explode

Question 13

In our trained RNN, prediction error spikes at word onsets (the first character after a space). Why does this happen?

Accepted Answer

The model has learned to predict within-word character sequences well (e.g., 't' → 'h' → 'e') but faces genuine uncertainty about which word comes next after a space, because multiple words are valid continuations

Question 14

The Zog experiment substitutes a never-seen character pattern 'zog' for the word 'man' in the corpus and runs it through the trained model WITHOUT further training. Why does 'zog' cluster with NOUN_HUMAN words?

Accepted Answer

The model assigns representations based on distributional context: 'zog' appears in the same templates as 'man' (after "the", before verbs), so the hidden states at 'zog' positions resemble those at 'man' positions

Question 15

Which pattern would you expect to see in neuron activations at space characters in our trained RNN?

Accepted Answer

One or more neurons show consistent spikes or valleys at spaces, acting as emergent "word boundary detectors"

Question 16

Why is W_hh initialized with an orthogonal matrix instead of Xavier uniform?

Accepted Answer

Orthogonal matrices have all eigenvalues with |λ| = 1, placing the spectral radius at exactly 1.0 for optimal gradient flow through time

Question 17

What is teacher forcing, and what problem does it create?

Accepted Answer

Using ground-truth previous tokens as input during training instead of model predictions; creates 'exposure bias' where the model never learns to recover from its own errors

Question 18

Why might a character-level RNN discover 'word boundaries' without being explicitly told about words?

Accepted Answer

Because prediction error is systematically higher at the first character of each word (uncertain which word comes next) and lower mid-word (remaining characters are predictable from the prefix)

Question 19

How would you empirically measure the effective memory length of a trained RNN?

Accepted Answer

Construct test sequences where the correct prediction requires remembering information from N characters back, then measure accuracy as a function of N

Question 20

What is the 'curse of memory,' and how does it differ from the vanishing gradient problem?

Accepted Answer

Vanishing gradients mean gradients shrink over time; the curse of memory means that as memory capacity increases, the network becomes exponentially sensitive to parameter changes, making training unreliable even without gradient explosion

Question 21

What architectural limitation prevents better optimizers (Adam, learning rate schedules) from solving the vanishing gradient problem in vanilla RNNs?

Accepted Answer

The Jacobian product structure ∏ W_hh^T · diag(1-h²) is baked into the vanilla RNN architecture; gradients MUST pass through this product at every step, causing exponential decay when ρ(W_hh) < 1, regardless of optimizer

Question 22

A CharRNN has vocabulary size V=27, hidden size H=64. What is the total number of trainable parameters (W_xh + W_hh + W_hy + biases)?

Accepted Answer

V×H + H×H + H×V + H + V = 27×64 + 64×64 + 64×27 + 64 + 27 = 7,643

Question 23

At training step 1000, W_hh has spectral radius ρ = 0.95. Approximately how many steps back can the model maintain an effective gradient above 10% of the original signal?

Accepted Answer

About 44 steps (0.95^44 ≈ 0.10, just above 10%)

Question 24

If the global gradient norm is 8.5 and the clip threshold is 5.0, by what factor are all gradients rescaled?

Accepted Answer

5.0 / 8.5 ≈ 0.588 (scale down to threshold)

Question 25

In BPTT, the Jacobian product ∂h_T/∂h_0 involves T matrix multiplications. For T=25 and spectral radius 0.9, approximately what is |∂h_T/∂h_0|?

Accepted Answer

Approximately 0.9^25 ≈ 0.072 (exponential decay)

Question 26

A TBPTT training run processes a corpus of 18,247 characters with chunk length 25 for 30,000 steps. Approximately how many times does the training loop cycle through the corpus?

Accepted Answer

About 42 times (30,000 × 25 / 18,247 ≈ 41)

Question 27

The tanh derivative is (1 - h²). If a hidden unit is highly saturated with h = 0.98, what fraction of the gradient passes through this unit?

Accepted Answer

1 - 0.98² = 0.0396, about 4% of the gradient passes through

Question 28

Word2Vec learns distributed representations from co-occurrence statistics. Elman's 1990 RNN learned category representations from sequential prediction. What is the key structural difference?

Accepted Answer

Word2Vec captures static, context-free embeddings from word co-occurrence; Elman RNN captures dynamic, context-dependent representations from sequential prediction, including temporal structure within words

Question 29

An RNN is tested on the formal language a^n b^n (n a's followed by n b's). Why does this test vanilla RNNs so severely?

Accepted Answer

Because the network must maintain an exact count of n in its hidden state, but finite-precision floats and the curse of memory make exact counting unreliable for large n

Question 30

LSTM was introduced to address vanishing gradients. What specific mechanism enables gradients to flow unchanged over many time steps?

Accepted Answer

Forget, input, and output gates that create additive (not multiplicative) information flow via a cell state, allowing unimpeded gradient paths

Question 31

Transformers largely replaced RNNs for long-sequence tasks. What fundamental property of self-attention makes it better for long-range dependencies?

Accepted Answer

Self-attention creates direct O(1) information paths between any two positions, bypassing the exponential decay in gradient flow that limits RNNs to ~10-50 steps of effective memory

Question 32

A character-level RNN trained on a synthetic grammar spontaneously clusters hidden states by semantic category (animals, verbs, food). Why does this emerge without explicit category labels?

Accepted Answer

Words in the same category appear in identical syntactic positions (after the same preceding words), so the RNN learns similar hidden state representations to predict the same distributions

Question 33

The 'curse of memory' (Wang & Li, 2024) identifies a fundamental trade-off in vanilla RNNs. What is that trade-off?

Accepted Answer

Longer effective memory requires spectral radius near 1, but this makes the loss landscape exponentially jagged; tiny parameter changes cause massive output changes, making gradient descent unreliable regardless of optimizer

Question 34

Given a 27-character vocabulary and hidden size H=64 with tanh activation, roughly how many bits of information can a single hidden state vector encode? Which is the best estimate?

Accepted Answer

~2,048 bits (H × 32 bits float32 precision per dimension), but practically much less due to tanh correlations and training dynamics

Question 35

You suspect your trained RNN detects word boundaries (spaces) but NOT sentence boundaries. Which experiment would BEST test this hypothesis?

Accepted Answer

Measure prediction error at word-internal positions vs word-initial positions vs sentence-initial positions. If word-internal < word-initial < sentence-initial, the network detects word boundaries better than sentence boundaries.

Question 36

If you doubled the corpus size (from 600 to 1,200 sentences) but halved the hidden size (from H=64 to H=32), what would MOST LIKELY happen to category emergence quality (silhouette score)?

Accepted Answer

Degrade: the reduced hidden state capacity is the bottleneck; more data can't compensate for fewer dimensions to cluster in