Differences

This shows you the differences between two versions of the page.

--- data_mining:neural_network:model_combination [2017/04/01 13:34] – [Dropout] phreazer
+++ data_mining:neural_network:model_combination [2017/08/19 20:12] (current) – [Approximating full Bayesian learning in a NN] phreazer
@@ Line 85: / Line 85: @@
 More complicated and effective methods than MCMC method: Don't need to wander the space long.
-If we compute gradient of cost function on a **random mini-batch**, we will get an ubiased estimate with sampling noise.
+If we compute gradient of cost function on a **random mini-batch**, we will get an unbiased estimate with sampling noise.
 ====== Dropout ======
-Ways to combine output of multiple models:
+See [[data_mining:neural_network:regularization|Regularization]]
-  * MIXTURE: Combine models by averaging their output probabilities.
-  * PRODUCT: by geometric mean (typically less than one) $\sqrt{x*y}/ \sum$
-NN with one hidden layer.
-Randomly omit each hidden unit with probability 0.5, for each training sample.
-Randomly sampling from 2^H architextures.
-Sampling form 2^H models, and each model only gets one training example (extreme bagging)
-Sharing of the weights means that every model is very strongly regularized.
-What to do at test time?
-Use all hidden units, but halve their outgoing weights. This exactly computes the geometric mean of the predictions of all 2^H models.
-What if we have more hidden Layers?
-* Use dropout of 0.5 in every layer.
-* At test time, use mean net, that has all outgoing weights halved. Not the same, as averaging all separate dropped out models , but approximation.