Summarizing Word2Vec for NLP beginners

This write up attempts to summarize the Google Research Paper:

*Efficient Estimation of Word Representations in Vector Space.*

The original simple NLP models such as N-gram are robust and perform reasonably well on speech and text processing tasks. However, these models treat words as atomic units, this means there is no notion of similarity between words, as these are represented as simple indices in a vocabulary.

The simple NLP techniques take a “bag of words (BOW)” approach to convert text to a machine friendly binary representation.

Photo by Glen Carrie on Unsplash

Let us take a sentence: “Amazingly few discotheques provide jukeboxes.”

In order to create a BOW representation of this sentence, first it is broken into words or tokens (tokenization). Each token is then converted to binary vector by a vectorizer based on a tokenization logic . We use “space” in our example.

|n|- total number of words in the sentence=5

|v|- vocabulary size or number of unique words=5

Each word is a binary vector representation of dimension |v|.

Hence “Amazingly” is represented as [1 0 0 0 0], “few” as [ 0 1 0 0 0 ] and so on and so forth. These are also referred to as embeddings.

The bag of words representation of the complete sentence will be a 2D embedding matrix of size |n| x |v|=5 x 5, shown below.

[[1 0 0 0 0 ], [0 10 0 0 ], [0 01 0 0 ], [0 0 0 1 0 ], [0 0 0 0 1 ]]

The sparsity of this matrix will be (|v|-1)*|n| and degree of sparsity can be given by 1–(1/|v|) or 4/5.

There are 2 issues with this BOW approach-

• Matrix sparsity grows with the size and any implementation is computationally slow.
• There is no context or similarity captured in the matrix coefficients.

Due to the reasons listed above, these simple NLP models are not very scalable and we lean on advanced techniques with a focussed on learning distributed representations of words that try to minimize computational complexity.

In this paper the researchers introduce techniques that can be used for learning high-quality word vectors from a corpora of billions of words.These new techniques measure the quality of the vector representations, with the expectation that not only will similar words tend to be close to each other, but that words can have multiple degrees of similarity .

Simple algebraic operations can be performed on the word vectors to show that the vector representation of “Queen” is given by-

vector(”King”) — vector(”Man”) + vector(”Woman”).

The study proposes two novel NLP model architectures to generate the continuous vector representation of words with high accuracy and at the same time, preserving the linear regularities among words.

Model 1: Continuous Bag-of-Words (CBOW)

CBOW, unlike the standard bag-of-words model, uses continuous distributed representation of the context. It tries to predict the current word based on the context around it.

For example in the sentence “Amazingly few____provide jukeboxes.”, the CBOW model will look at the surrounding words and try to fill in the blank with the best word-“discotheques”

Model 2: Continuous Skip-gram

The skip gram model works in the reverse manner and tries to maximize the classification of a word based on another word in the same sentence. For example for the word “discotheques”, the skip gram model tries to predict the surrounding words- “Amazingly”, “jukeboxes”, “few”, “provide”.

New model architectures by Mikolov et al. (https://arxiv.org/pdf/1301.3781.pdf)

Conclusion

The paper evaluates the quality of vector representations of words generated by feed forward neural network language models (NNLM), Recurrent Neural Net Language Model (RNNLM), CBOW and Skip-gram on a collection of syntactic and semantic language tasks. The study concludes that by using simple model architectures with lower computational complexity, it is possible to compute very accurate high dimensional word vectors from a much larger corpora.

References

Efficient Estimation of Word Representations in Vector Space

Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean