# More than two classes: Logistic Regression
In the following, we are interested in the case of $p$ classes with
$p>2$. After the previous discussion, it seems natural for the output to
take the integer values $y = 1, \dots, p$. However, it turns out to be
helpful to use a different, so-called *one-hot encoding*. In this
encoding, the output $y$ is instead represented by the $p$-dimensional
unit vector in $y$ direction $\mathbf{e}^{(y)}$,
```{math}
:label: eqn:One-Hot-Encoding
y \longrightarrow \mathbf{e}^{(y)} =
\begin{bmatrix}
e^{(y)}_1 \\
\vdots \\
e^{(y)}_y \\
\vdots \\
e^{(y)}_{p}
\end{bmatrix}
=
\begin{bmatrix}
0 \\
\vdots \\
1 \\
\vdots \\
0
\end{bmatrix},
```
where $e^{(y)}_l = 1$ if $l = y$ and zero for all other $l=1,\ldots, p$. A main advantage of this encoding is that we are
not forced to choose a potentially biasing ordering of the classes as we
would when arranging them along the ray of integers.
A linear approach to this problem then again mirrors the case for linear
regression. We fit a multi-variate linear model,
Eq. [](eqn:Multivariate-Linear-Model), to the one-hot encoded
dataset $\lbrace(\mathbf{x}_{1}, \mathbf{e}^{(y_1)}), \dots, (\mathbf{x}_{m}, \mathbf{e}^{(y_m)})\rbrace$.
By minimising the RSS, Eq. [](eqn:RSS), we obtain the solution
```{math}
\hat{\beta} = (\widetilde{X}^{T}\widetilde{X})^{-1} \widetilde{X}^{T} Y,
```
where $Y$ is the $m$ by $p$ output matrix. The prediction given an input
$\mathbf{x}$ is then a $p$-dimensional vector
$\mathbf{f}(\mathbf{x}|\hat{\beta}) = \tilde{\mathbf{x}}^{T} \hat{\beta}$. On a
generic input $\mathbf{x}$, it is obvious that the components of this
prediction vector would be real valued, rather than being one of the
one-hot basis vectors. To obtain a class prediction
$F(\mathbf{x}|\hat{\beta}) = 1, \dots, p$, we simply take the index of the
largest component of that vector, i.e.,
```{math}
F(\mathbf{x}|\hat{\beta}) = \textrm{argmax}_{k} f_{k}(\mathbf{x}|\hat{\beta}).
```
The $\textrm{argmax}$ function is a non-linear function and is a first
example of what is referred to as *activation function*.
For numerical minimization, it is better to use a smooth activation
function. Such an activation function is given by the *softmax* function
```{math}
F_k(\mathbf{x}|\hat{\beta})= \frac{e^{-f_k(\mathbf{x}|\hat{\beta})}}{\sum_{k'=1}^pe^{-f_{k'}(\mathbf{x}|\hat{\beta})}}.
```
Importantly, the output of the softmax function is a probability
$P(y = k|\mathbf{x})$, since $\sum_k F_k(\mathbf{x}|\hat{\beta}) = 1$. This
extended linear model is referred to as *logistic regression* [^3].
The current linear approach based on classification of one-hot encoded
data generally works poorly when there are more than two classes. We
will see in the next chapter that relatively straightforward non-linear
extensions of this approach can lead to much better results.
[^3]: Note that the softmax function for two classes is the logistic
function.