Neural Networks

These are exam preparation notes, subpar in quality and certainly not of divine quality.

See the index with all articles in this series

Connectionist Neurons

A neural network generally has a number of inputs $x_{1} . . . x_{N}$ which are aggregated into $\underset{―}{x}$ . At each node there is a transfer function $y_{i}$ which turns the inputs according to weights $\underset{―}{w}$ into its own output.

A typical function would look like this. The part in the brackets is later referred to as $h$

y_{i} = f (\sum_{j} w_{i j} x_{j} - θ_{i})

Some of the functions being used for $f$ :

Logistic Function:

f (h) = \frac{1}{1 + e x p (- β h}

Hyperbolic tangent:

f (h) = t a n h (β h)

Linear neuron:

f (h) = β h

Binary neuron:

f (h) = s i g n (h)

The $β$ is a slope parameter.

TODO:Transformation

\frac{1}{2} (t a n h \frac{1}{2} + 1)

The first weight or input is a bias node which is always 1. It is not always included in equations as $w$ or $x$ . The bias node could be seen as the length of w(???)

Reasons for nonlinear transfer functions:

multiple layers could beexpressed as one in linear transfer functions (main reason)
sign function for classification problems (0,1)
logistic sigmoidal for probabilities (0..1)

Important variables:

$θ$ threshold
$v and v^{'}$ layer

Types of Neural Networks

Recurrent Neural Networks
There can be loops in the graph
Feedforward Neural Networks (DAG)
no loops
Radial Basis Function Networks

Typical Usecase: Prediction of attributes. MLPs are universal approximators

Always from $R^{N}$ to anything, really.

NN in Regression

Hessian matrix is second derivative of $R^{N} \mapsto R$

Jacobian matrix is first derivative of $R^{N} \mapsto R^{M}$ .

Hessian is often too computationally expensive to compute and therefore backpropagation is often used instead of Newton’s Method.

Generalization Error

ERM

Test Error

The test error is reduced using gradient descend.

w_{i j}^{v^{'} v} (t + 1) = w_{i j}^{v^{'} v} (t) - \hat{η} \frac{\partial E}{\partial w} w_{i j}^{v^{'} v}

where

\frac{\partial E}{\partial w} w_{i j}^{v^{'} v} = \frac{1}{p} \sum_{α = 1}^{p} \frac{\partial e_{[\underset{―}{w}]}^{(α)}}{\partial w_{i j}^{v^{'} v}}

The error is usually quadratic error

e (y_{T}, \underset{―}{x}) = \frac{1}{2} (y_{T} - y (\underset{―}{x}))^{2}

The derivative is trivially:

\frac{\partial e^{(α)}}{\partial y_{(} {\underset{―}{x}}^{(α)}, \underset{―}{w})} = y_{({\underset{―}{x}}^{(α)}} - y_{T}

and is later used in backpropagation.

Backpropagation

In backpropagation the weights of the neural network are adjusted so that the test error is reduced. This is achieved by

Calculating the prediction
Calculating the test error
Going back layer by layer and calculating the delta each time *

It would be possible to do backpropagation by applying the chain rule. But that is a lot more computationally expensive than Backpropagation.

Regularization in Deep Learning

Dropout randomly ignores neurons

Architectures

Convolutional layer

Layer that is only connected to selected previous neurons. For example this can be used in image recognition, having neurons only be connected to some adjacent previous pixels ( a tensor).

Spatial/Feature pooling

Trying to detect features in an image even though the image may be rotated, translated, etc. There are then e.g. three different detection units for a specific pattern that is then aggregated by a neuron with a max() function, to recognize the correctly oriented feature.

Auto-Encoders

Unfortunately excluded from the exam, therefore neglected here

Basically you take an image of what you want to recognize and push it through your network. What you get is a “compressed” version of the image (there is a lot less information in the final layers). In the beginning of your training this will be just noise / randomness. You then have another neural network (the same???) reconstruct the original image.

What is then possible is to compare the reconstruction to the original image and generate error values from it.

You thereby can train two neural networks to meaningfully abstract from images without having to have labelled images.

Time Series

In a time series it is often assumed that y depends on a short time window. Therefore there are convolutions, where some neurons can look “back” in time.

Recurrent NN

Neural Network is “shifted” through time. All previous inputs are summarized as a vector with a weight vector $W$ containing th mapping on itself.

n number of timesteps

Cost function with:

E^{T} = \frac{1}{p} \sum_{α = 1}^{p} (\frac{1}{n_{α}} \sum_{t = 1}^{n} e^{(α, t)})

There is a vector $\underset{―}{W}$ which contains the weights that measure how much of the previous input should be considered in the next timestep.

TODO: Are the weights $\underset{―}{W}$ different for each timestep?

Backpropagation through time

Works just like regular backpropagation.

Assume all ${\underset{―}{W}}^{(t)}$ are independent
compute gradients with backpropagation
All computed gradients are averaged for weight update.

Δ \underset{―}{W} = - η \frac{\partial E^{T}}{\partial \underset{―}{W}}

Exploding / Vanishing gradient

\underset{―}{W} = \underset{―}{U} \underset{―}{Λ} {\underset{―}{U}}^{⊺}

One problem of RNNs is that activity is often either vanishing or exploding over time, when $| y_{i} | \neq 0$ .

Echo State Networks

Echo state networks set W and U so that their

y_i

is almost equal to r. (TODO why is r in range 1.3<->3 ?)

Leaky Units:

There are units that specialize in long or short term memory. This depends on a factor $α$

LSTM

Delay update of hidden layer
Special transfer function (only retrieve state in certain cases)

Radial Basis Function Networks

Also see Wikipedia.

3-layered Radial basis function network

A radial basis function is a function that is only dependent on the distance from the center(Usually Eucleadian distance).

ϕ_{i} (\underset{―}{x}) = \overset{}{ϕ_{i}} (D [\underset{―}{x}, {\underset{―}{t}}_{i}])

Gaussian function often used:

ϕ_{i} (\underset{―}{x}) = e x p (- \frac{| | \underset{―}{x} - {\underset{―}{t}}_{i} | |^{2}}{2 σ_{i}^{2}}))

Learning with RBFs

Three different parameters:

${\underset{―}{t}}_{i}$ centroid (center of basis function)
$σ_{i}$ range of influence
$w_{i}$ weights of the output layer

2-Step Learning procedure is an alternative to normal learning of parameters.

Find centroids and variances $σ_{i}$
Determine output weights $u n d e r l i n e w_{i}$

Find centroids and variances

Use k-means clustering to find centroids

Choose $σ_{i}$ so that it is double the distance of the closest two centroids.

σ_{i} = λ \underset{j \neq i}{m i n} | | {\underset{―}{t}}_{i} - {\underset{―}{t}}_{j} | |, λ \approx 2

Determine output weights

Output weights are found reducing quadratic error with M := number of RBFs:

E^{T} = \frac{1}{2 p} \sum_{α = 1}^{p} (y_{t}^{(α)} - \sum_{i = 1}^{M} (w_{i} ϕ_{i ({\underset{―}{x}}^{(α)})}))^{2}

Pseudo-inverse

({\underset{―}{Φ}}^{T} \underset{―}{Φ}) \underset{―}{w} = {\underset{―}{Φ}}^{T} {\underset{―}{y}}_{T} ⟹ \underset{―}{w} = ({\underset{―}{Φ}}^{T} \underset{―}{Φ})^{- 1} {\underset{―}{Φ}}^{T} {\underset{―}{y}}_{T}

TODO: Do we now use Gradient Descent or invertible matrix?

MLP vs RBF

RBFs have fast convergence, as few parameters needs to be changed per training point, as they have negligible influence on far away points.

RBFs fall under curse of dimensionality, need $n^{d}$ basis functions. (n number of data points along one dimension, d number of dimensions)

RBFs are kernel functions that make it possible to map non-linear data into linearity and then do regression on them.

RBFs are useful for low-dimensional data.