Unsupervised Learning Methods Exam Preparation

Following the principle “You only understood something thoroughly if you can explain it” - here come the prepping notes for Machine Intelligence II. If no sources are indicated, it comes from the lecture slides.

Note This was foremostly written for my own understanding, so it might contain incomplete explanations

Chapters

General Terms and tools
PCA
- PCA
- Hebbian Learning
- Kernel-PCA
Source Separation
- ICA
- Infomax ICA
- Second Order Source Separation
- FastICA
Stochastic Optimization
Clustering
- k-means Clustering
- Pairwise Clustering
- Self-Organising Maps
- Locally Linear Embedding
Estimation Theory
- Density Estimation
  - Kernel Density Estimation
  - Parametric Density Estimation
- Mixture Models - Estimation Models

General Terms and tools

A lot of the different methods rely on some general methodology that will be reused. Need a refresher in Matrix multiplication? Oh, and dot product is the same as a scalar product.

Centered Data

Centering data means making the center of mass 0. This means every dimension is averaged and the average is then subtracted from all data points for each dimension.

X = X - \frac{1}{p} \sum_{α}^{p} x^{(α)}

also called first moment

or with numpy:

# x is our data
x_centered = x - np.mean(x, axis=0)

Covariance matrix

Assuming $p$ centered data points $x^{(α)}$ :

Whitened Data

Whitening turns your data matrix into a matrix with a covariance matrix which is an identity matrix. The data is then uncorrelated (but might be dependent). This is useful to find e.g. outliers.

\begin{aligned} C_{i j} & = \frac{1}{p} \sum_{α = 1}^{p} x_{i}^{(α)} x_{j}^{(α)} \\ o r \\ C & = \frac{1}{p} {\underset{―}{x}}^{T} \underset{―}{x} \end{aligned}

Kullback-Leibler-Divergence

Kullback-Leibler-Divergence measures the difference / distance between two probability distributions - in this example $P$ and $\hat{P}$

D_{K L} [P (\underset{―}{x}), \hat{P} (\underset{―}{x}; \underset{―}{w})] = \int d \underset{―}{x} P (\underset{―}{x}) l n \frac{P (\underset{―}{x})}{\hat{P} (\underset{―}{x}; \underset{―}{w})} = \underset{(\underset{―}{w})}{m i n}

Jacobian Matrix

For a function $f : I R^{n} \to I R^{m}$ the Jacobian matrix is filled with the derivatives

[\begin{matrix} \frac{\partial f_{1}}{\partial x_{1}} & \dots & \frac{\partial f_{1}}{\partial x_{n}} \\ ⋮ & ⋱ & ⋮ \\ \frac{\partial f_{1}}{\partial x_{n}} & \dots & \frac{\partial f_{m}}{\partial x_{n}} \end{matrix}]

Mercers theorem

From the slides:

Every positive semidefinite definite kernel k corresponds to a scalar product in some metric feature space

Markov Process

A markov process is only dependent on the most recent state. E.g. Its probabilities into which state it will go next are independent of any older states.

Variance

σ^{2} = E [(x - μ)^{2}]

Discrete

σ^{2} = \frac{1}{p} \sum_{α = 1}^{p} (x_{α} - μ)^{2}