Differences

This shows you the differences between two versions of the page.

--- data_mining:neural_network:word_embeddings [2018/06/09 18:04] – [Word2Vec] phreazer
+++ data_mining:neural_network:word_embeddings [2018/06/09 18:40] (current) – [Debiasing word embeddings] phreazer
@@ Line 66: / Line 66: @@
 Solution: Hierarchical softmax: Tree of classifiers $log |v|$. Common words on top, not a balanced tree.
+=== How to sample context c? ===
+When uniformly random: often frequent words like "the, of, a, ..."
+Heuristics are used for sampling
+==== Negative Sampling ====
+Generate data set
+  * Pick 1 positive example
+  * Pick k negative examples
+    * Choose random word from dicitionary which are not associated with context word: target = 0
+    * Heuristic between uniform and observed distribution
+binary classification problems
+==== GloVe word vectors ====
+Global vectors for word representation
+$x_{ij}$: Number of times i appears in context of j
+Minimize $\sum_{i=1}^{10000} \sum_{j=1}^{10000} f(x_{ij}) (\Theta_i^{T} e_j + b_i - b'_j  - log x_{ij})^2$
+Weighting term $f(x_{ij})$: Weight for frequent, infrequent words
+$e^{final}_w = \frac{e_w + \Theta_w}{2}$
+===== Application =====
+==== Sentiment classification ====
+=== Simple model ===
+  * Extract embedding vector for each word
+  * Sum or Avg those vectors
+  * Pass to softmax to gain output (1-5 stars)
+Problem: Doesn't include order/sequence of words
+=== RNN for sentiment classification ===
+  * Extract embedding vector for each word
+  * Feed into RNN with softmax output
+===== Debiasing word embeddings =====
+Bias in text
+Addressing bias in word embessing:
+  - Identify bias direction (e.g. gender)
+    * $e_{he} - e_{she}$, average them
+  -  Neutralize: For every word that is not definitial (legitimate gender component), project
+  - Equalize pairs: Only difference should be gender (e.g. grandfather vs. grandmother); equidistant