GloVe | Test Your Understanding

Question 1

What does a co-occurrence matrix X record?

Accepted Answer

How often word j appears within a context window of word i, aggregated across the entire corpus

Question 2

GloVe uses 1/d distance weighting in the context window. What does this mean?

Accepted Answer

Closer words within the window contribute fractionally more to the co-occurrence count

Question 3

What is the key insight behind GloVe's use of probability ratios P(k|i)/P(k|j)?

Accepted Answer

Ratios between co-occurrence probabilities discriminate relevant words from irrelevant ones better than raw probabilities

Question 4

If P(k|ice)/P(k|steam) >> 1, what does this tell us about the probe word k?

Accepted Answer

k is strongly associated with ice (e.g., k = 'solid' or 'freeze')

Question 5

The derivation from probability ratios to GloVe's objective requires that F(w_i - w_j, w̃_k) = P(k|i)/P(k|j). What key constraint makes this work?

Accepted Answer

The function must satisfy a homomorphism property between addition in vector space and multiplication in ratio space

Question 6

Why does GloVe have two sets of embeddings, W and W̃?

Accepted Answer

They represent word and context roles symmetrically; the final embedding is typically W + W̃

Question 7

What is the purpose of the weighting function f(X_ij) in GloVe's objective?

Accepted Answer

To down-weight extremely frequent co-occurrences that would otherwise dominate the loss

Question 8

GloVe's objective function J = Σ f(X_ij) · (w_i · w̃_j + b_i + b̃_j − log(X_ij))² is best described as:

Accepted Answer

A weighted least squares regression of the model's prediction against log co-occurrence

Question 9

What role do the bias terms b_i and b̃_j play in GloVe?

Accepted Answer

They absorb word frequency information, freeing the vectors to encode semantic meaning

Question 10

Why does GloVe use AdaGrad instead of standard SGD?

Accepted Answer

AdaGrad adapts per-parameter learning rates so frequent words get smaller updates and rare words get larger ones

Question 11

The gradient ∂J/∂w_i for a single (i,j) pair equals:

Accepted Answer

f(X_ij) · (w_i · w̃_j + b_i + b̃_j − log(X_ij)) · w̃_j

Question 12

Why does vector arithmetic (king − man + woman ≈ queen) work in GloVe's framework?

Accepted Answer

In the log-bilinear model, differences of word vectors correspond to log ratios of co-occurrence probabilities, encoding consistent semantic relationships

Question 13

In GloVe's implementation, the AdaGrad accumulators are initialized to:

Accepted Answer

1.0 (following the Stanford C implementation to prevent division issues)

Question 14

Gradient clipping in GloVe clips gradients to the range:

Accepted Answer

[−100, 100] to prevent exploding gradients

Question 15

The GloVe implementation in train.py uses:

Accepted Answer

Pure NumPy with manually computed analytical gradients

Question 16

Levy and Goldberg (2014) showed that Word2Vec's skip-gram implicitly factorizes a shifted PMI matrix. How does this relate to GloVe?

Accepted Answer

GloVe explicitly factorizes log(X_ij), and its biases absorb the marginal terms that distinguish log(X) from PMI, so both methods converge to similar solutions

Question 17

Why are W and W̃ symmetric in GloVe's objective?

Accepted Answer

Swapping W and W̃ (along with their biases) gives the same loss value; the only asymmetry comes from random initialization

Question 18

The training state recording captures every epoch for the first 10, then every 5th. Why?

Accepted Answer

Early training shows the most dramatic embedding changes; later epochs are more gradual, so less frequent recording suffices