Word2Vec — Test Your Understanding

Question 1

Why is one-hot encoding a poor representation for words?

Accepted Answer

Every word is equally distant from every other word — no similarity signal

Question 2

The distributional hypothesis states that:

Accepted Answer

Words appearing in similar contexts tend to have similar meanings

Question 3

What does a sliding context window produce from a sentence?

Accepted Answer

(center word, context word) training pairs

Question 4

What does the dot product measure between two word vectors?

Accepted Answer

How aligned their directions are — a measure of similarity

Question 5

What does the softmax function do in the skip-gram model?

Accepted Answer

Converts raw dot-product scores into a probability distribution that sums to 1

Question 6

What does a single row of the embedding matrix W represent?

Accepted Answer

The dense embedding vector for that word (its "fingerprint")

Question 7

What is the difference between W (embedding matrix) and W' (context matrix)?

Accepted Answer

W represents words when they are the center word; W' represents words when they are context words

Question 8

The cross-entropy loss L = -log P(context | center) is high when:

Accepted Answer

The model assigns low probability to the true context word (it was "surprised")

Question 9

The gradient of the Word2Vec loss decomposes into two forces. The attractive force:

Accepted Answer

Pulls the center word's vector toward the true context word's vector

Question 10

Why does the vector arithmetic king - man + woman ≈ queen work?

Accepted Answer

The training process encodes consistent semantic directions (like gender) into the geometry of the embedding space

Question 11

A skip-gram model with V=96 and d=8 has how many total trainable parameters?

Accepted Answer

96 × 8 × 2 = 1,536 (W and W' matrices)

Question 12

Negative sampling replaces the full softmax by:

Accepted Answer

Only updating the true context word (positive) and a few random non-context words (negatives) at each step

Question 13

What is the correct order of operations in one training step?

Accepted Answer

Forward → Loss → Backward → Update

Question 14

Why does the training script record every step for the first 200, then every 10th?

Accepted Answer

Early training shows the most dramatic changes; later training is more gradual

Question 15

What does PCA do to the 8-dimensional embeddings?

Accepted Answer

Projects them down to 2 dimensions while preserving the most variance, so they can be plotted on a scatter chart