0° Means Besties, 90° Means Strangers: The Math Behind Cosine Similarity

Photo by Alexander Grey on Unsplash

The cosine similarity is used to measure the angle between two vectors in a multi dimensional space where similar vectors point in the same direction. If the angle is smaller which means the vectors are close to each other and if the angle is more denoting that there are least similarities between the vectors.

Before moving on to the details lets refresh our knowledge on some the basics which will help in understanding the concept better

Vector

A vector in general is a mathematical object which represents both magnitude and direction. It’s often represented as array of numbers enclosed in a bracket, where each of the number represents the component along the axis of the space.

for example, vector representation of (2,4) in 2dimensional space means a directed arrow starting from (0,0) to (2,4) in X,Y axis. Each values are represented considered as a scalar value.

Vectors can be mathematically operated to add, subtract, and multiplied by a scalar value to create a new vector.

Dot Product

Its also called as scalar product, denoted by A.B and the components of the vectors are multiplied individually and added together to get the result.

for example, if A is represented by (a1,a2,a3) and B is represented by (b1,b2,b3) then

Cross Product

Mathematically it’s represented as * which is to multiply A and B.

In vectors, its denoted as A x B and the the output of the cross product is a right angled vector which is perpendicular to both A and B

Euclidean Distance

It’s the square root of the sum of the square of the individual values. For the above example vector the distance would be

The value will shows only the magnitude and it ranges from 0 to infinity, where 0 means identical.

Example

Lets assume that we have two vectors in 3 dimensional space represented as A and B. The vectors can be anything like an object, string, cars, fruits, documents, images etc. For the simplicity of this example lets assume that A and B are two words “Dog” and “Cat” that can be found in 3 different documents and we wanted to find the similarity of these documents based on these two words represented by ||A|| and ||B||

Occurrence of these words in the documents are as follows

+---------------+--------------+----------------+
|               |     Dog      |       Cat      |
+---------------+--------------+----------------+
|  Document 1   |      9       |       7        |
|  Document 2   |      4       |       6        |
|  Document 3   |      5       |       2        |
+---------------+--------------+----------------+

• Dog (vector A, represented as ||A||) = [9, 4, 5]
• Cat (vector B, represented as ||B||) = [7, 6, 2]

If we go by the number of highest occurrences of these words in the documents then it will give an impression that the top two documents are similar but thats not the case always. If we plot we would see that the

If we project these occurrence in the 3D space with x,y and z axis representing the 3 documents then we can plot the words correspondingly. The angle between the words in each of the document will identify the similarities in the documents. To find the cosine angle between the words in the documents we use the the formula for Cosine similarity as

• A.B is the dot(.) product of the two vectors A and B
• *||A||||B||* is the cross product of the vectors

In summary if the distance is closer to 0 then the two vectors are similar and when the values are 0 meaning the vectors are identical. In this case as per the value of Dogs and Cats they are closely related.

If we compute the Euclidean distance between these two words we will see that the distance which is farther apart but the angle are still closer and the cosine distance is smaller. This shows that even though the values are farther apart in magnitude the angle between them shows the closeness between the documents.

Range of values

The value ranges from -1 to 1 which shows the dissimilarity and similarities. 0 means identical and is perpendicular/orthogonal.

Why are we not using Sine function?

This is the common question and the answer is Sine is a trignometric function and the value cannot be directly attributed to similarities.

Also cosine is mainly used for determining the angle and direction making it an ideal option which is not the case with any other function.

Python example

I have used a sklearn library to show how it works. We are just going to change the doc_dogand doc_cat in the following snippet to show how the similarity score changes

# Define the documents
doc_dog = "Cat is good. Dog is also good."
doc_cat = "Dog is good. Cat is also good."

documents = [doc_dog, doc_cat]

from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd

count_vectorizer = CountVectorizer()
sparse_matrix = count_vectorizer.fit_transform(documents)

doc_term_matrix = sparse_matrix.todense()
df = pd.DataFrame(doc_term_matrix, 
                  columns=count_vectorizer.get_feature_names_out(), 
                  index=['doc_dog', 'doc_cat'])
print(df)

# Compute Cosine Similarity
from sklearn.metrics.pairwise import cosine_similarity
print(cosine_similarity(df, df))

Case 1 — Same document content, different order.

doc_dog = "Cat is good. Dog is also good."
doc_cat = "Dog is good. Cat is also good."

Case 2 — Different document with some similarities

# Define the documents
doc_dog = "Dog and cats are friendly animal. But dog do not like cat. Cat may seem silly some time."
doc_cat = "Cat may be cunning at some time. but Dog is so close to human. Dogs are really nice to be with"

Case 3 — No common words

# Define the documents
doc_dog = "Snake"
doc_cat = "Cat"

Advantages

• Cosine similarity works well in higher dimensions with lesser word similarity.
• Its not affected by the value or the meaning but focusing only on the directions of the vector
• It has lot applications in text analysis, recommender system and image analysis.

Limitations

• Even though this algorithm has lot of advantages its not used for identifying similarities based on semantic meaning even though there is no match between the words. For example Fish and Food should be considered similar or Error and Failure should be treated the same.

There is another algorithm called Soft Cosine Similarity which will overcome this problem by converting words in word vectors and then finding similarities.

References

• https://weaviate.io/blog/distance-metrics-in-vector-search
• https://www.machinelearningplus.com/nlp/cosine-similarity/