Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revisionLast revisionBoth sides next revision | ||
| data_mining:neural_network:neurons [2017/02/19 14:29] – [Stochastic binary neurons] phreazer | data_mining:neural_network:neurons [2017/08/19 17:42] – [Rectified Linear Neurons] phreazer | ||
|---|---|---|---|
| Line 70: | Line 70: | ||
| ===== Rectified Linear Neurons ===== | ===== Rectified Linear Neurons ===== | ||
| + | |||
| + | Aka ReLU (Rectified Linear Unit) | ||
| $z=b+\sum_{i} x_{i} w_{i}$ | $z=b+\sum_{i} x_{i} w_{i}$ | ||
| - | $y = \begin{cases} z, & \text{if } z > 0 \\ 0, & \text{otherwhise}\end{cases}$ | + | |
| + | $y = \begin{cases} z, & \text{if } z > 0 \\ 0, & \text{otherwhise}\end{cases} | ||
| Above 0, it is linear, at 0 it is 0 | Above 0, it is linear, at 0 it is 0 | ||
| + | |||
| + | Faster computation, | ||
| Line 90: | Line 95: | ||
| $\text{lim}_{(z-> | $\text{lim}_{(z-> | ||
| - | ==== Softmax group ==== | + | Switch from Sigmoid to ReLU lead to performance improvement (Slope of Sigmoid gradually shrinks to zero). |
| + | |||
| + | ===== tanh ===== | ||
| + | Works better than Sigmoid function. | ||
| + | |||
| + | $y = \frac{e^{z}-e^{-z}}{e^{z}+e^{-z}}$ | ||
| + | |||
| + | Centering of data to 0. | ||
| + | |||
| + | Exception: Output layer, since output should be in [0,1]. | ||
| + | ===== Softmax group ===== | ||
| + | |||
| + | Logistic function output is used for the classification between two target classes 0/1. Softmax function is generalized type of logistic function that can output a **multiclass** categorical **probability distribution**. | ||
| Derivates: | Derivates: | ||
| Line 135: | Line 153: | ||
| Also possible for rectified linear units: Output is treated as the poisson rate for spikes. | Also possible for rectified linear units: Output is treated as the poisson rate for spikes. | ||
| + | |||
| + | |||