Creative Prediction with Neural Networks

We can now use the following equations to compute \(\hat{y}_t\), by computing \(h\) for the previous steps:

\[h_t = \text{tanh}\big(Ux_t + Vh_{t-1} + b \big)\]

\[\hat{y}_t = \text{softmax}(c + Wh_t)\]

We use the categorical cross-entropy function for loss:

\[\begin{align*} h_t &= \text{tanh}\big( {b} + {Vh}_{t-1} + {Ux}_t \big) \\ \hat{y}_t &= \text{softmax}(c + Wh_t) \\ L_t &= -y_t \cdot \text{log}(\hat{y}_t) \\ \text{Loss} &= \sum_t L_t \\ \end{align*}\]

Propagates error correction backwards through the network graph, adjusting all parameters (U, V, W) to minimise loss.

• Target: A probability distribution with \(P(n) = 1\)
• Output: A probability distribution over all next letters.
• E.g.: “My cat is named Simon” would lead to X: “My cat is named Simo” and y: “n”

• S: Sampling function, sample a letter using the output probability distribution.
• The generated letter is reinserted at as the next input.
• We don’t want to always draw the most likely character. The would give frequent repetition and “copying” from the training text. Need a sampling strategy.

• RNN as a sequence generator
• Input is current symbol, output is next predicted symbol.
• Connect output to input and continue!
• CharRNN simply applies this to a (subset) of ASCII characters.
• Train and generate on any text corpus: Fun!

See: Karpathy, A. (2015). The unreasonable effectiveness of recurrent neural networks.

Shakespeare (Karpathy, 2015):

Second Senator: They are away this miseries, produced upon my soul, Breaking and strongly should be buried, when I perish The earth and thoughts of many states.

DUKE VINCENTIO: Well, your wit is in the care of side and that.

Latex Algebraic Geometry:

N.B. “Proof. Omitted.” Lol.

• Useful for tasks where the whole sequence is available.
• Each output unit (\(\hat{y}\)) depends on both past and future - but most sensitive to closer times.
• Popular in speech recognition, translation etc.

• Does adding deeper layers to an RNN make it work better?
• Several options for architecture.
• Simply stacking RNN layers is very popular; shown to work better by Graves et al. (2013)
• Intuitively: layers might learn some hierarchical knowledge automatically.
• Typical setup: up to three recurrent layers.

• Big mathematical challenge!
• Gradients propagated through the same weights tend to vanish (mostly) or explode (rarely)
• E.g., consider an RNN with no nonlinear activation function or input.
• Each time step multiplies h(0) by W.
• This corresponds to raising power of eigenvalues in \(\Lambda\).
• Eventually, components of h(0) not aligned with the largest eigenvector will be discarded.

\[\begin{align*} h_t &= Wh_{t-1}\\ h_t &= (W^t)h_0 \end{align*}\]

(supposing W admits eigendecomposition with orthogonal matrix Q)

\[\begin{align*} W &= Q\Lambda Q^{\top}\\ h_t &= Q\Lambda ^t Qh_0 \end{align*}\]

• Provide gates that can change the hidden state a little bit at each step.
• The gates are controlled by learnable weights as well!
• Hidden state weights that may change at each time step.
• Create paths through time with derivatives that do not vanish/explode.
• Gates choose information to accumulate or forget at each step.

• Self-loop containing internal state (c).
• Three extra gating units:
  • Forget gate: controls how much memory is preserved.
  • Input gate: control how much of current input is stored.
  • Output gate: control how much of state is shown to output.
• Each gate has own weights and biases, so this uses lots more parameters.

Source: (Olah, C. 2015.)

• Are three gates necessary?
• Other gating units are simpler, e.g., Gated Recurrent Unit (GRU)
• For the moment, LSTMs are winning in practical use.
• Alternative unit design: project idea?

Teaching Recurrent Neural Networks about Monet

New Paint Colours Invented by Neural Network

• iPad music transcribed as sequence of numbers for each performer.
• Trick: encode multiple ints as one (preserving ordering).
• Video