data_mining:neural_network:neurons

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
data_mining:neural_network:neurons [2017/02/16 23:07] – [Sigmoid Neuron] phreazerdata_mining:neural_network:neurons [2017/08/19 17:43] (current) – [Rectified Linear Neurons] phreazer
Line 2: Line 2:
  
 Dendritic Tree Dendritic Tree
-- Collects input from other neurons+  * Collects input from other neurons
  
 Axon Axon
-- Branches +  * Branches 
-- Contact dendritic trees at synapses+  * Contact dendritic trees at synapses
  
 Spike generation Spike generation
-- Axon hillock that generates outgoing spikes whenever enough charge has flowed in at synapses to depolarize the cell members+  * Axon hillock that generates outgoing spikes whenever enough charge has flowed in at synapses to depolarize the cell members
  
 Synapses Synapses
-- When Spike of activity travels along an axon and arrives at a synapse : Vesicles of transmitter chemicals to be released (several kinds of transmitter (positive and negative weights)+  * When Spike of activity travels along an axon and arrives at a synapse : Vesicles of transmitter chemicals to be released (several kinds of transmitter (positive and negative weights)
  
 Transmitter molecules diffuse accross the synaptic cleft and bind to receprot molecules in the membrane of the post-synaptic neuron (changing their shape). This opens holes that allo specific ions in or out -> changes depolerization. Transmitter molecules diffuse accross the synaptic cleft and bind to receprot molecules in the membrane of the post-synaptic neuron (changing their shape). This opens holes that allo specific ions in or out -> changes depolerization.
  
 Effectivenes of synapses can be changed:  Effectivenes of synapses can be changed: 
-- Different number of vesicles of transmitter +  * Different number of vesicles of transmitter 
-- Different number of receptor molecules+  * Different number of receptor molecules
  
 Synapses very small and very low power Synapses very small and very low power
 Adapt using locally available signals Adapt using locally available signals
  
-10^11 Neurons with 10^4 weights (high bandwith)+**10^11 Neurons** with **10^4 weights** (high bandwith)
  
 Different bits of the cortex do different things. Different bits of the cortex do different things.
Line 29: Line 29:
  
 Cortex is made of general purpose stuff that has the ability to turn into special purpose hardware in response to experience Cortex is made of general purpose stuff that has the ability to turn into special purpose hardware in response to experience
-- Early brain damages makes functions relocate.+  * Early brain damages makes functions relocate.
  
 ====== Artificial neurons ====== ====== Artificial neurons ======
Line 57: Line 57:
 McCulloch - Pitts (1943) McCulloch - Pitts (1943)
  
-  - 1. Compute weighted sum of inputs +  - Compute weighted sum of inputs 
-  - 2. Send fixed size spike of activity if the weitghtes sum exceeds a threshold. Spike is like the bool value of a proposition and each neuron combines bool values to compute bool value of another proposition. +  - Send fixed size spike of activity if the weitghtes sum exceeds a threshold. Spike is like the bool value of a proposition and each neuron combines bool values to compute bool value of another proposition. 
-  - 3. Output 0 or 1+  - Output 0 or 1
  
 $z=b+\sum_{i} x_{i} w_{i}$ $z=b+\sum_{i} x_{i} w_{i}$
Line 70: Line 70:
  
 ===== Rectified Linear Neurons ===== ===== Rectified Linear Neurons =====
 +
 +Aka ReLU (Rectified Linear Unit)
  
 $z=b+\sum_{i} x_{i} w_{i}$ $z=b+\sum_{i} x_{i} w_{i}$
-$y = \begin{cases} z, & \text{if } z > 0 \\ 0, & \text{otherwhise}\end{cases}$+ 
 +$y = \begin{cases} z, & \text{if } z > 0 \\ 0, & \text{otherwhise}\end{cases} = \max(0,z)$
  
 Above 0, it is linear, at 0 it is 0 Above 0, it is linear, at 0 it is 0
 +
 +Faster computation, since slope doesn't get very small/large.
 +
 +Leaky ReLU:
 +
 +$y =\max(0.01 z,z)$
  
  
Line 89: Line 98:
  
 $\text{lim}_{(z->∞)} \frac{1}{1+e^{-z}} = 1$ $\text{lim}_{(z->∞)} \frac{1}{1+e^{-z}} = 1$
 +
 +Switch from Sigmoid to ReLU lead to performance improvement (Slope of Sigmoid gradually shrinks to zero).
 +
 +===== tanh =====
 +Works better than Sigmoid function.
 +
 +$y = \frac{e^{z}-e^{-z}}{e^{z}+e^{-z}}$
 +
 +Centering of data to 0.
 +
 +Exception: Output layer, since output should be in [0,1].
 +===== Softmax group =====
 +
 +Logistic function output is used for the classification between two target classes 0/1. Softmax function is generalized type of logistic function that can output a **multiclass** categorical **probability distribution**.
  
  
Line 121: Line 144:
  
 $y_i=\frac{e^{z_i}}{\Sigma_{j \in group} e^{z_j}}$ $y_i=\frac{e^{z_i}}{\Sigma_{j \in group} e^{z_j}}$
 +
 +$\frac{\partial y_i}{\partial z_i}=y_i (1- y_i)$
 ===== Stochastic binary neurons =====  ===== Stochastic binary neurons ===== 
  
Line 131: Line 156:
 Output 0 or 1. Output 0 or 1.
  
-Also possible for rectified linear units: Output is trated as the poisson rate for spikes.+Also possible for rectified linear units: Output is treated as the poisson rate for spikes. 
 + 
  • data_mining/neural_network/neurons.1487282879.txt.gz
  • Last modified: 2017/02/16 23:07
  • by phreazer