Global Vectors for Word Representation
Build a co-occurrence matrix from the entire corpus, discover that probability ratios encode meaning, then derive an elegant objective function that learns word vectors by factorizing the co-occurrence matrix.
GloVe (Global Vectors for Word Representation) learns word vectors by factorizing a co-occurrence matrix. First, count how often every word pair appears near each other in a corpus (the co-occurrence matrix X). Then notice that probability ratios P(k|i) / P(k|j) encode meaning better than raw probabilities. Derive an elegant log-bilinear model: wᵢᵀw̃ₖ + bᵢ + b̃ₖ = log(Xᵢₖ). Train via weighted least squares with AdaGrad, using a power-law weighting function to balance rare and frequent co-occurrences. The result: word vectors where king − man + woman ≈ queen, and the factorization quality reveals which aspects of meaning the model has captured.
Before GloVe trains a single parameter, it does something Word2Vec never does: it reads the entire corpus and builds a complete census of word neighborhoods. For every pair of words in the vocabulary, it asks a simple question: how often does word j appear near word i? The answer is stored in a giant table called the co-occurrence matrix X, where each entry
Not all co-occurrences are counted equally. A word one position away from the target contributes a full weight of 1.0, while a word 5 positions away contributes only 1/5 = 0.2. This 1/d distance weighting reflects a natural intuition: words standing shoulder-to-shoulder with the target are more likely to share a meaningful relationship than words at the far edge of the window. The paper puts it plainly: “very distant word pairs are expected to contain less relevant information about the words’ relationship.”
The resulting matrix is symmetric when using a symmetric window (looking both left and right of the target): if “ice” appears in the context of “cold,” then “cold” also appears in the context of “ice,” so
Perhaps the most striking property of X is its sparsity: 91% of entries in our toy matrix are zero. Most word pairs in a finite corpus simply never co-occur within any window. This isn't a problem; it's a feature. GloVe trains only on non-zero entries, making its complexity proportional to the number of observed co-occurrences rather than the square of the vocabulary size. The non-zero entries themselves span a wide range (in our toy corpus from 0.4 to ~51, and on production corpora from 0.2 to thousands), and this dynamic range is why GloVe's weighting function and AdaGrad optimizer matter so much.
Important:
Toy vs Production: Our toy corpus has 577 sentences, 1,832 tokens, 144 vocabulary words, and 1,930 non-zero entries (91% sparse). The 6B GloVe used 400K vocabulary and billions of non-zero entries (>98% sparse).
“We've built the matrix. But staring at raw numbers won't reveal meaning. We need to look at these counts more carefully, and ask the right question...”
Raw co-occurrence counts are a starting point, but they're noisy. Convert them to conditional probabilities (
What if we compare the two distributions? Subtraction (
“Solid” gives a ratio of about 10.5, strongly associated with ice over steam. “Gas” gives 0.11, about 9× more associated with steam. Words related to both (like “water,” ratio ≈ 1 because both P values are high) or neither (like “cat,” ratio = 1 because both P values are zero) produce ratios near 1, for different reasons, but both uninformative. Four categories emerge naturally from a single operation: large ratio (word1-associated), small ratio (word2-associated), near-1 with high P (shared), near-1 with low P (unrelated). The ratio cleanly separates discriminative signal from noise without requiring any thresholds or hyperparameters. This mirrors Table 1 from the GloVe paper (which uses a 6B corpus with probe words solid, gas, water, and fashion; our toy corpus shows the same pattern with different magnitudes).
Here is the conceptual leap that births GloVe: if ratios encode meaning so cleanly, can we find vectors
“Ratios encode meaning. But they're just numbers in a table. To make them useful for computation, we need to find vectors whose mathematical operations naturally produce these ratios. This is where GloVe's derivation begins, and it's one of the most elegant pieces of mathematics in all of NLP.”
The GloVe derivation is a sequence of five principled steps, each motivated by a specific mathematical or linguistic constraint. It begins with the question from Chapter 2 (find
The key moves: analogies live in differences (so F should operate on
Walk through each step in the interactive derivation below. Pay special attention to Step 3, the homomorphism argument, where the exponential function is not chosen but forced by mathematical necessity. And remember the prediction from Step 4: the bias
“We have a beautiful equation:
The model equation says the prediction should equal
Second, non-zero co-occurrences span orders of magnitude. A pair that co-occurred once carries noisy signal; a pair that co-occurred 50 times is reliable; a pair at 5,000 is very reliable but shouldn't dominate the entire loss. GloVe introduces a weighting function
The α = 3/4 exponent is sublinear: doubling
The full objective is:
Skip-gram minimizes, over the whole corpus, the negative log probability of context words:
where
This is a weighted cross-entropy, with weights
The result is GloVe's objective: a weighted least-squares problem that is cheaper per term, avoids normalization, and explicitly works with the global co-occurrence statistics rather than sampling individual windows.
“We have the objective: minimize the weighted squared error between our model's predictions and
The gradient for a single pair (i, j) is elegant: let
The paper trains GloVe with AdaGrad, and the choice is well-motivated. Co-occurrence values span orders of magnitude. Frequent word pairs like ice-solid are updated many times per epoch; rare pairs are updated once. With vanilla SGD and a single learning rate, tuning for rare pairs makes frequent pairs diverge, and tuning for frequent pairs starves rare pairs. AdaGrad solves this by accumulating squared gradients per parameter:
GloVe maintains two sets of vectors: W (word vectors) and W̃ (context vectors), each with their own bias vectors and AdaGrad accumulators. When X is symmetric, W and W̃ are mathematically interchangeable; they differ only by random initialization. The paper uses their sum W + W̃ as the final embedding, which averages out initialization-dependent noise and gives a small but consistent performance boost. The paper explains: “for certain types of neural networks, training multiple instances and combining the results can help reduce overfitting and noise.”
Unlike Word2Vec's continuous stream of random samples, GloVe has a natural epoch structure: one pass through all non-zero
Why AdaGrad specifically? It accumulates squared gradients per parameter, building a per-element history
One subtle detail: when
“Training is done. But what did GloVe actually learn? Let's crack open the model and inspect everything: the vectors, the biases, the quality of the factorization, and how the two sets of vectors relate to each other.”
With training complete, we can inspect the model from angles unique to GloVe. Start with nearest neighbors: using
The model equation for word i and context word k is:
Write this for “king” and “man” with the same context word k, then subtract:
The difference vector
To find the word that is “royalty minus male plus female,” we solve:
The paper finds the word d whose vector is closest to
Try it yourself: Pick two words from the same semantic group in the explorer below. Compute their vector difference. Which context words show the largest magnitude in that difference direction? Do they match your intuition about what distinguishes those words?
Toy corpus note: Our corpus has only 144 words and 8 dimensions. Analogies that work flawlessly on a 6B-token corpus may not succeed here; the vocabulary is too small for many standard analogy sets, and the embeddings lack the capacity to encode fine-grained relationships. The explorer below shows accuracy improving with training, but don't expect production-level results.
Next, inspect factorization quality: how well does
Now the payoff from Chapter 3's foreshadowing: what do the biases encode? Scatter-plot
Finally, the PMI connection ties everything together. Since biases absorb the marginal log-frequencies, the dot product alone approximates
“We've inspected what GloVe learned: vectors that factorize the co-occurrence matrix, biases that encode frequency, and PMI structure in the dot products. But where does GloVe fit in the bigger picture? How does it connect to SVD, LSA, and the methods that came before and after?”
GloVe sits at the culmination of a long lineage of count-based methods. LSA (1990) showed that SVD of a term-document matrix captures meaning through dimensionality reduction, the first proof that statistical co-occurrence encodes semantics. HAL (1996) moved to word-word co-occurrence but used raw counts, which are dominated by frequency effects. COALS (Rohde et al., 2006) added correlation-based normalization before SVD. PPMI + SVD (Levy & Goldberg, 2014) computed Positive PMI and applied SVD, a strong baseline competitive with Word2Vec. Each method asked: “What transformation of the co-occurrence matrix gives the best word representations?” GloVe's answer: a weighted factorization of the log, guided by the probability ratio insight.
The paper's Table 2 reveals this progression quantitatively. SVD of raw X scores 7.3% on analogy tasks; frequency effects destroy the signal. SVD of
The Levy & Goldberg result (2014) connects the two sides of the embedding world. They showed that skip-gram with negative sampling implicitly factorizes a shifted PMI matrix:
Pre-trained GloVe vectors from Stanford (6B, 42B, and 840B token corpora) remain widely available and useful for initialization and lightweight tasks. But GloVe produces static embeddings; the same vector for “bank” whether it means riverbank or financial institution. This limitation drove the field toward contextual models: ELMo (2018), BERT (2019), GPT. These produce different vectors for the same word in different sentences. Yet GloVe's core insights (that co-occurrence statistics encode meaning, that ratios are more informative than raw probabilities, and that a principled log-bilinear model can be derived from first principles) remain foundational to understanding how any embedding method works.
