RNN

Orthogonal converges fastest. At step 0, ρ=1.0 means all eigenvalues are on the unit circle; gradients neither vanish nor explode as they flow back through time. Xavier produces ρ≈0.85 (sub-unity, leading to vanishing gradients), while random can produce ρ>1 (unstable early dynamics).

Random initialization produces fragmented, non-word character sequences ('thh e c aa t') even after 30K steps. Its final loss is 2.1 vs 1.12 for orthogonal; it never fully recovers from the chaotic early dynamics caused by ρ>1 initialization.

The effective memory depends on how well gradients survive the backward pass. With ρ≈1 (orthogonal), the product of recurrent Jacobians stays near 1, allowing gradients to flow back 18 chars on average. With random init and ρ>1 early on, the network learns a distorted W_hh that reduces effective gradient reach.

Categories still weakly emerge (the model can distinguish human nouns from verbs), but the silhouette score is much lower. The memory probe shows accuracy drops sharply beyond 6 characters back; H=16 lacks the capacity to maintain distinct context representations.

No. Memory horizon grows from 18 to 23 chars (28% increase, not 100%). The limiting factor shifts from representational capacity to the vanishing gradient problem: BPTT can only propagate meaningful gradients ~20-25 chars back regardless of hidden size, given fixed seq_len=25 and clip=5.

H=128 achieves slightly lower loss (1.05 vs 1.12) and better category clustering, but at 3.6× the parameters (57K vs 16K). For this toy grammar, the marginal gain is small; H=64 already captures the essential structure. H=128 would matter more for longer texts or richer grammars.

The memory probe shows almost no accuracy beyond 4 characters back; the model physically cannot learn dependencies longer than the chunk. Even within the chunk, the decay curve shows the gradient drops to ~0.1 by position 5.

No. The memory horizon goes from 18 to 22 chars (22% increase). The gradient decay curve for seq_len=50 shows the gradient falling below 0.05 by position 25, even though the chunk extends to 50, vanishing gradients make positions 26–50 nearly unreachable.

Yes, around seq_len=25–35. Beyond this, the gradient decay from tanh saturations and W_hh multiplication effectively limits the useful horizon. Longer chunks add computation cost without proportional memory benefit. This is why architectures like LSTMs use gates to maintain gradients instead of just longer chunks.

Without a nonlinearity, the RNN is just a linear dynamical system: h_t = W_hh × h_{t-1} + W_xh × x_t. If any eigenvalue of W_hh exceeds 1, h_t grows exponentially. More fundamentally, without nonlinearity, the model cannot represent the complex pattern separations needed for category emergence.

ReLU does preserve gradients better for active neurons (gradient=1, not <1 like tanh). But two problems emerge: (1) 'dying ReLU' (units that become inactive never recover); (2) unbounded hidden states require aggressive gradient clipping (clip=1.0 here) to prevent instability. The hidden std of 0.71 vs tanh's 0.62 reflects this.

sigmoid maps to [0,1] with derivative ≤ 0.25, while tanh maps to [-1,1] with derivative ≤ 1.0. The 4× larger gradient for tanh means faster gradient flow and faster convergence. Additionally, sigmoid's [0,1] output creates asymmetric representations; positive-only hidden states make it harder to encode contrast (e.g., 'this is NOT a verb').

On 92% of steps, the model receives a rescaled gradient with norm exactly 0.5. The actual gradient information (direction) is preserved, but the magnitude is systematically reduced; learning is glacially slow. After 30K steps, loss is 2.15, worse than clip=5.0's 1.12.

Rarely, very large gradient spikes slip through unchecked. These spikes cause sudden destabilizing updates that set the model back. With clip=5.0, the 28% of clipped steps are precisely the dangerous spikes; the model learns efficiently without catastrophic updates.

Clipping is most frequent in the early training phase (first 5K steps) when the model is in the chaotic high-loss region and gradients are large. All curves show decreasing clip frequency over time as the model enters a smoother loss landscape. clip=5.0's frequency drops from ~45% to ~5% by step 30K.

At lr=0.001, each gradient step is 10× smaller than the baseline. The model makes safe, conservative updates, but after 30K steps it hasn't converged. With 300K steps it would likely reach similar performance to lr=0.01. The tradeoff is time vs safety.

For the first 3K steps, the model is in the high-loss region where gradients are relatively small (the gradient landscape is smooth). Once it reaches the 'transition zone', where loss starts to decrease rapidly - gradients become large, and lr=0.1 amplifies them to destabilizing sizes.

Larger learning rates take larger steps and can cross loss valleys quickly (fast early convergence) but also overshoot the minimum. lr=0.01 takes more careful steps, eventually finding a tighter minimum. This is the classic speed-precision tradeoff in optimization.

tanh, consistent with Experiment 4. ReLU is 0.15 loss units worse on average across configurations, and sigmoid is 0.4 worse. The ranking is stable across hidden sizes and seq_lengths.

H=64 and H=128 are nearly tied for tanh + seq_len=25. H=64 provides about 95% of H=128's performance at 28% of the parameters. For this grammar, H=64 is the practical sweet spot.

Small hidden sizes (H=16, H=32) have limited representational capacity; they benefit from longer sequence context to compensate. Large hidden sizes (H=128) already capture sufficient structure from shorter sequences. Capacity and context length are partially substitutable.