data_mining:neural_network:initialization

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
Last revisionBoth sides next revision
data_mining:neural_network:initialization [2017/08/19 22:34] – phreazerdata_mining:neural_network:initialization [2017/08/19 22:37] – [Random initialization] phreazer
Line 4: Line 4:
 Weights need to be randomly initialized. For bias zero is ok. Weights need to be randomly initialized. For bias zero is ok.
  
-If weights are zero => $dz_{1,2}$ are the same. Hidden units would compute same function.+If weights are zero: in backprop => $dz_{1,2}$ are the same. Hidden units would compute same function (= are symmetric). 
 + 
 +Solution: $W^{[i]}=np.random.randn((2,2)) * 0.01$ 
 + 
 +$0.01$ because else we would end up at ends of activation function values (and slopes would be small).
  • data_mining/neural_network/initialization.txt
  • Last modified: 2017/08/19 22:38
  • by phreazer