Differences

This shows you the differences between two versions of the page.

--- data_mining:neural_network:overfitting [2018/05/10 15:50] – [Weight penalites] phreazer
+++ data_mining:neural_network:overfitting [2018/05/10 16:03] (current) – [Inverted dropout] phreazer
@@ Line 86: / Line 86: @@
 Cost function
-$J(\dots) = 1/m \sum_{i=1}^n L(\hat{y}^{i}, y^{i}) + \frac{\lambda}{2m} \sum_{l=1}^L ||W^{[l]}||_F^2$
+$J(W^{[l]},b^{[l]})= \frac{1}{m} \sum_{i=1}^m J(\hat{y}^{(i)}, y^{(i)}) + \frac{\lambda}{2m} \sum_{l=1}^L || W^{[l]} ||_F^2$
 Frobenius Norm: $||W^{[l]}||_F^2$
@@ Line 97: / Line 97: @@
 For large $\lambda$, $W^{[l]} => 0$
-$J(W^{[l]},b^{[l]})= \frac{1}{m} \sum_{i=1}^m J(\hat{y}^{(i)}, y^{(i)}) + \frac{\lambda}{2m} \sum_{l=1}^L || W^{[l]} ||^2$
 This results in a **simpler** network / each hidden unit has **smaller effect**.
@@ Line 141: / Line 139: @@
 ==== Inverted dropout ====
-Layer $l=3$.
+<code>
+Layer l=3
-$keep.prob = 0.8$
-$d3 = np.random.rand(a3.shape[0], a3.shape[i]) < keep.prob$
+keep.prob = 0.8 // probability that unit will be kept
-$a3 = np.multiply(a3,d3)$
+d3 = np.random.rand(a3.shape[0], a3.shape[i]) < keep.prob // dropout vector
-$a3 /= keep.prob$ // e.g. 50 units => 10 units shut off
+a3 = np.multiply(a3,d3) // activations in layer 3 a3 *= d3
-$Z = Wa+b$ // reduced by 20% => standardize with 0.8 => expected value stays the same
+a3 /= keep.prob // e.g. 50 units => 10 units shut off
+Z = Wa+b // reduced by 20% => standardize with 0.8 => expected value stays the same
+</code>
 Making predictions at test time: No drop out