After Tomas Mikolov et al. released the word2vec tool, there was a boom of articles about word vector representations. One of the best of these articles is Stanford’s GloVe: Global Vectors for Word Representation, which did explained why such algorithms work and refolmulating word2vec optimizations as a special kind of factoriazation for word co-occurence matrices.
Here I will briefly introduce the GloVe algorithm and show how to use its text2vec implementation.
GloVe algorithm consists of following steps:
Collect word cooccurence statistics in a form of word coocurence matrix \(X\). Each element \(X_{ij}\) of such matrix represents measure of how often word i appears in context of word j. Usually we scan our corpus in the following manner: for each term we look for context terms withing some area, a window_size before and a window_size after. Also we give less weight for more distant words, usually using the formula \[decay = 1/offset\].
Define a cost function \[J = \sum_{i=1}^V \sum_{j=1}^V \; f(X_{ij}) ( w_i^T w_j + b_i + b_j - \log X_{ij})^2\] Here \(f\) is a weighting function which help us to prevent learning only from extremely common word pairs. The GloVe authors choose the following fucntion:
\[ f(X_{ij}) = \begin{cases} (\frac{X_{ij}}{x_{max}})^\alpha & \text{if } X_{ij} < XMAX \\ 1 & \text{otherwise} \end{cases} \]
Now let’s examine how GloVe embeddings works. As commonly known, word2vec word vectors capture many linguistic regularities. To give the canonical example, if we take word vectors for the words “paris,” “france,” and “italy” and perform the following operation:
\[vector('paris') - vector('france') + vector('italy')\]
the resulting vector will be close to the vector for “rome.”
Let’s download the same Wikipedia data used as a demo by word2vec:
library(text2vec)
library(readr)
temp <- tempfile()
download.file('http://mattmahoney.net/dc/text8.zip', temp)
wiki <- read_lines(unz(temp, "text8"))
unlink(temp)In the next step we will create a vocabulary, a set of words for which we want to learn word vectors. Note, that all of text2vec’s functions which operate on raw text data (create_vocabulary, create_corpus, create_dtm, create_tcm) have a streaming API and you should iterate over tokens as the first argument for these functions.
# Create iterator over tokens
tokens <- strsplit(wiki, split = " ", fixed = T)
# Create vocabulary. Terms will be unigrams (simple words).
vocab <- create_vocabulary(itoken(tokens))These words should not be too rare. Fot example we cannot calculate a meaningful word vector for a word which we saw only once in the entire corpus. Here we will take only words which appear at least 5 times. text2vec provides more options to filter vocabulary (see ?prune_vocabulary).
vocab <- prune_vocabulary(vocab, term_count_min = 5L)Now we have 71,290 terms in the vocabulary and are ready to construct term-co-occurence matrix (TCM).
# We provide an iterator to create_vocab_corpus function
it <- itoken(tokens)
# Use our filtered vocabulary
vectorizer <- vocab_vectorizer(vocab,
# don't vectorize input
grow_dtm = FALSE,
# use window of 5 for context words
skip_grams_window = 5L)
tcm <- create_tcm(it, vectorizer)Now we have a TCM matrix and can factorize it via the GloVe algorithm.
text2vec uses a parallel stochastic gradient descend algorithm. By default it use all cores on your machine, but you can specify the number of cores if you wish. For example, to use 4 threads, call RcppParallel::setThreadOptions(numThreads = 4).
Let’s fit our model. (It can take several of minutes to fit!)
fit <- glove(tcm = tcm,
word_vectors_size = 50,
x_max = 10, learning_rate = 0.2,
num_iters = 15)2016-01-10 14:12:37 - epoch 1, expected cost 0.0662
2016-01-10 14:12:51 - epoch 2, expected cost 0.0472
2016-01-10 14:13:06 - epoch 3, expected cost 0.0429
2016-01-10 14:13:21 - epoch 4, expected cost 0.0406
2016-01-10 14:13:36 - epoch 5, expected cost 0.0391
2016-01-10 14:13:50 - epoch 6, expected cost 0.0381
2016-01-10 14:14:05 - epoch 7, expected cost 0.0373
2016-01-10 14:14:19 - epoch 8, expected cost 0.0366
2016-01-10 14:14:33 - epoch 9, expected cost 0.0362
2016-01-10 14:14:47 - epoch 10, expected cost 0.0358
2016-01-10 14:15:01 - epoch 11, expected cost 0.0355
2016-01-10 14:15:16 - epoch 12, expected cost 0.0351
2016-01-10 14:15:30 - epoch 13, expected cost 0.0349
2016-01-10 14:15:44 - epoch 14, expected cost 0.0347
2016-01-10 14:15:59 - epoch 15, expected cost 0.0345
And now we get the word vectors:
word_vectors <- fit$word_vectors[[1]] + fit$word_vectors[[2]]
rownames(word_vectors) <- rownames(tcm)We can find the closest word vectors for our paris - france + italy example:
word_vectors_norm <- sqrt(rowSums(word_vectors ^ 2))
rome <- word_vectors['paris', , drop = FALSE] -
word_vectors['france', , drop = FALSE] +
word_vectors['italy', , drop = FALSE]
cos_dist <- text2vec:::cosine(rome,
word_vectors,
word_vectors_norm)
head(sort(cos_dist[1,], decreasing = T), 10)
## paris venice genoa rome florence
## 0.7811252 0.7763088 0.7048109 0.6696540 0.6580989You can achieve much better results by experimenting with skip_grams_window and the parameters of the glove() function (including word vectors size and the number of iterations). For more details and large-scale experiments on wikipedia data see this post on my blog.