Word2Vec | Test Your Understanding

Question 1

Why is one-hot encoding a poor representation for words?

Accepted Answer

Every word is equally distant from every other word, with no similarity signal

Question 2

The distributional hypothesis states that:

Accepted Answer

Words appearing in similar contexts tend to have similar meanings

Question 3

What does a sliding context window produce from a sentence?

Accepted Answer

(center word, context word) training pairs

Question 4

What does the dot product measure between two word vectors?

Accepted Answer

How aligned their directions are (a measure of similarity)

Question 5

What does the softmax function do in the skip-gram model?

Accepted Answer

Converts raw dot-product scores into a probability distribution that sums to 1

Question 6

What does a single row of the embedding matrix W represent?

Accepted Answer

The dense embedding vector for that word (its "fingerprint")

Question 7

What is the difference between W (embedding matrix) and W' (context matrix)?

Accepted Answer

W represents words when they are the center word; W' represents words when they are context words

Question 8

The cross-entropy loss L = -log P(context | center) is high when:

Accepted Answer

The model assigns low probability to the true context word (it was "surprised")

Question 9

The gradient of the Word2Vec loss decomposes into two forces. The attractive force:

Accepted Answer

Pulls the center word's vector toward the true context word's vector

Question 10

Why does the vector arithmetic king - man + woman ≈ queen work?

Accepted Answer

The training process encodes consistent semantic directions (like gender) into the geometry of the embedding space

Question 11

A skip-gram model with vocabulary size V = 96 and embedding dimension d = 8 has how many total trainable parameters?

Accepted Answer

96 × 8 × 2 = 1,536 (W and W' matrices)

Question 12

Negative sampling replaces the full softmax by:

Accepted Answer

Only updating the true context word (positive) and a few random non-context words (negatives) at each step

Question 13

What is the correct order of operations in one training step?

Accepted Answer

Forward → Loss → Backward → Update

Question 14

The training script records model state at every step for the first 200 steps, then every 10th step after that. Why?

Accepted Answer

Early training shows the most dramatic changes; later training is more gradual

Question 15

What does PCA do to the 8-dimensional embeddings in the visualization pipeline?

Accepted Answer

Projects them down to 2 dimensions while preserving the most variance, so they can be plotted on a scatter chart