Bag of Words: Convert text into vectors
In this blog, we will study about the model that represents and converts text to numbers i.e. the Bag of Words (BOW). The bag-of-words model has seen great success in solving problems which includes language modeling and document classification as it is simple to understand and implement.
After completing this particular blog, you all will have an overview of: What does the bag-of-words model mean by and why is its importance in representing text. How we can develop a bag-of-words model for a collection of documents. How to use the bag of words to prepare a vocabulary and deploy in a model using programming language.
The problem and its solution…
The biggest problem with modeling text is that it is unorganised, and most of the statistical algorithms, i.e., the machine learning and deep learning techniques prefer well defined numeric data. They cannot work with raw text directly, therefore we have to convert text into numbers.
Word embeddings are commonly used in many Natural Language Processing (NLP) tasks because they are found to be useful representations of words and often lead to better performance in the various tasks performed. A huge number of approaches exist in this regard, among which some of the most widely used are Bag of Words, Fasttext, TF-IDF, Glove and word2vec. For easy user implementation, several libraries exist, such as Scikit-Learn and NLTK, which can implement these techniques in one line of code. But it is important to understand the working principle behind these word embedding techniques. As already said before, in this blog, we see how to implement Bag of words and the best way to do so is to implement these techniques from scratch in Python . Before we start with coding, let’s try to understand the theory behind the model approach.
Theory Behind Bag of Words Approach
In simple words, Bag of words can be defined as a Natural Language Processing technique used for text modelling or we can say that it is a method of feature extraction with text data from documents. It involves mainly two things firstly, a vocabulary of known words and, then a measure of the presence of known words.
The process of converting NLP text into numbers is called vectorization in machine learning language.A lot of different ways are available in converting text into vectors which are:
Counting the number of times each word appears in a document, and Calculating the frequency that each word appears in a document out of all the words in the document.
Understanding using an example
To understand the bag of words approach, let’s see how this technique converts text into vectors with the help of an example. Suppose we have a corpus with three sentences:
- “I like to eat mangoes”
- “Did you like to eat jellies?”
- “I don’t like to eat jellies”
Step 1: Firstly, we go through all the words in the above three sentences and make a list of all of the words present in our model vocabulary.
- I
- like
- to
- eat
- mangoes
- Did
- you
- like
- to
- eat
- Jellies
- I
- don’t
- like
- to
- eat
- jellies
Step 2: Let’s find out the frequency of each word without preprocessing our text.
But is this not the best way to perform a bag of words. In the above example, the words Jellies and jellies are considered twice no doubt they hold the same meaning. So, let us make some changes and see how we can use ‘bag of words’ by preprocessing our text in a more effective way.
Step 3: Let’s find out the frequency of each word with preprocessing our text. Preprocessing is so very important because it brings our text into such a form that is easily understandable, predictable and analyzable for our task.
Firstly, we need to convert the above sentences into lowercase characters as case does not hold any information. Then it is very important to remove any special characters or punctuations if present in our document, or else it makes the conversion more messy.
From the above explanation, we can say the major advantage of Bag of Words is that it is very easy to understand and quite simple to implement in our datasets. But this approach has some disadvantages too such as:
- Bag of words leads to a high dimensional feature vector due to the large size of word vocabulary.
- Bag of words assumes all words are independent of each other ie’, it doesn’t leverage co-occurrence statistics between words.
- It leads to a highly sparse vector as there is nonzero value in dimensions corresponding to words that occur in the sentence.
Bag of Words Model in Python Programming
The first thing that we need to create is a proper dataset for implementing our Bag of Words model. In the above sections, we have manually created a bag of words model with three sentences. However, now we shall find a random corpus on Wikipedia such as ‘https://en.wikipedia.org/wiki/Bag-of-words_model‘.
Step 1: The very first step is to import the required libraries: nltk, numpy, random, string, bs4, urllib.request and re.
Step 2: Once we are done with importing the libraries, now we will be using the Beautifulsoup4 library to parse the data from Wikipedia.Along with that we shall be using Python’s regex library, re, for preprocessing tasks of our document. So, we will scrape the Wikipedia article on Bag of Words.
Step 3: As we can observe, in the above code snippet we have imported the raw HTML for the Wikipedia article from which we have filtered the text within the paragraph text and, finally,have created a complete corpus by merging up all the paragraphs.
Step 4: The very next step is to split the corpus into individual sentences by using the sent_tokenize function from the NLTK library.
Step 5: Our text contains a number of punctuations which are unnecessary for our word frequency dictionary. In the below code snippet, we will see how to convert our text into lower case and then remove all the punctuations from our text, which will result in multiple empty spaces which can be again removed using regex.
Step 6: Once the preprocessing is done, let’s find out the number of sentences present in our corpus and then, print one sentence from our corpus to see how it looks.
Step 7: We can observe that the text doesn’t contain any special character or multiple empty spaces, and so our own corpus is ready. The next step is to tokenize each sentence in the corpus and create a dictionary containing each word and their corresponding frequencies.
As you can see above, we have created a dictionary called wordfreq. Next, we iterate through each word in the sentence and check if it exists in the wordfreq dictionary. On its existence,we will add the word as the key and set the value of the word as 1.
Step 8: Our corpus has more than 500 words in total and so we shall filter down to the 200 most frequently occurring words by using Python’s heap library.
Step 9: Now, comes the final step of converting the sentences in our corpus into their corresponding vector representation. Let’s check the below code snippet to understand it. Our model is in the form of a list of lists which can be easily converted matrix form using this script:
matrix. You first split each token into 
dimensional, 8 vectors in total, as I colored in the figure below. In other words, the input matrix is divided into 8 colored chunks, which are all 
matrices, but each colored matrix expresses the same sentence. And you calculate self-attentions of the input sentence independently in the 8 heads, and you reweight the “values” according to the attentions/weights. After this, you stack the sum of the reweighted “values” in each colored head, and you concatenate the stacked tokens of each colored head. The size of each colored chunk does not change even after reweighting the tokens. According to Ashish Vaswani, who invented Transformer model, each head compare “queries” and “keys” on each standard. If the a Transformer model has 4 layers with 8-head multi-head attention , at least its encoder has 
heads, so the encoder learn the relations of tokens of the input on 32 different standards.
![[ \cdots ]](https://data-science-blog.com/en/wp-content/ql-cache/quicklatex.com-a935a6ae352397cdde28cd5115cc275a_l3.png "Rendered by QuickLaTeX.com")
denotes a token, which is usually an embedding vector in practice.
*I have been repeating the phrase “reweighting ‘values’ with attentions,” but you in practice calculate the sum of those reweighted “values.”
. Let’s take an example of calculating a scaled dot-product in the blue head.
, which are “queries”, “keys”, and “values” respectively.
by 
in the formula. According to the original paper, it is known that re-scaling 
is calculated like in the figure below. The softmax function regularize each row of the re-scaled product 
, and the resulting 
matrix is a kind a heat map of self-attentions.
and regularizing them with a softmax function, you stack those vectors, and the stacked vectors is the heat map of attentions.
. This also should be easy to understand if you know basics of linear algebra.
tokens, basically you have to wait for 
the RNN cell retains the information at the time step 
only via recurrent connections. In this way you cannot attend to tokens in the earlier time steps, and this is obviously far from how we compare tokens in a sentence. You can bring information backward by bidirectional connection s in RNN models, but that all the more deteriorate parallelization of the model. And possessing information via recurrent connections, like a telephone game, potentially has risks of vanishing gradient problems. Gated RNN, such as LSTM or GRU mitigate the problems by a lot of nonlinear functions, but that adds to computational costs. If you understand multi-head attention mechanism, I think you can see that Transformer solves those problems.
, but if you naively give the term ![[0, 1]](https://data-science-blog.com/en/wp-content/ql-cache/quicklatex.com-caffaae885a1287e3dfc31bfb1cd0694_l3.png "Rendered by QuickLaTeX.com")
. With this approach, however, the resolution of encodings can vary depending on the length of the input sequence data. Thus these naive approaches do not meet the requirements above, and I guess even conventional RNN-based models were not so successful in these points.
, 
, where 
. 
is the dimension of word embedding. The heat map below is the most typical type of visualization of positional encoding you would see everywhere, and in this case 
, and 
is discrete number which varies from 
to 
, thus the heat map blow is equal to a 
matrix, whose elements are from 
to 
. Each row of the graph corresponds to one token, and you can see that lower dimensional part is constantly changing like waves. Also it is quite easy to encode an input with this positional encoding: assume that you have a matrix of an input sentence composed of 50 tokens, each of which is a 256 dimensional vector, then all you have to do is just adding the heat map below to the matrix.
.
pairs of circles rather than 
, the index of the depth of each encoding, you can extract a 2 dimensional vector 
. If you constantly change the value 
rotates clockwise on the unit circle in the figure below.
, 
can be represented as a linear function of 
.” For each circle at any depth, I mean for any 
, where 
. Also the shift from “Bay” to “great” has the same rotation.
', fontsize=30)
plt.colorbar()
plt.title("Positional encoding of 50 256-d tokens", fontsize=40)
plt.savefig("positional_encoding_heat_map.png")
plt.show()
', fontsize=40)
ax.set_zlabel(r'
', fontsize=40)
ax.set_xticks(np.arange(0, d_model//2, 10))
plt.subplots_adjust(left=0, right=1, bottom=0, top=1)
#plt.savefig('./positional_encoding_gif/{}.png'.format(j+1))
plt.show()
, and it includes words from “a”, “ablation”, “actually” to “zombie”, “?”, “!”
, where only the No. i element is 
dimensional vector, whose dimension is fewer than the vocabulary size 
. When the inputs are “You’ve got the touch” or “You’ve got the power” , you put the one-hot vector corresponding to “You’ve”, “got”, “the”, “touch” or “You’ve”, “got”, “the”, “power” sequentially every time step 
.
, 
.
, a list of vectors. The vectors are usually embedding vectors, and the 
, where 
denote “You’ve”, “got”, “the”, “power”, “.” respectively. In this case 
.
usually includes two tokens 
and 
at the beginning and the end of the sentence. They mean “Beginning Of Sentence” and “End Of Sentence” respectively. Thus in many cases 
. 
and 
are also both vectors, at least in the Tensorflow tutorial.
is the probability of incidence of the sentence. But it is easy to imagine that it would be very hard to directly calculate how likely the sentence 
as a product of the probability of incidence or a certain word, given all the words so far. When you’ve got the words 
so far, the probability of the incidence of 
, given the context is 
. 
is a probability of the the sentence 
, and the probability of 
can be decomposed this way: 
.
.
be 
be ![P(\boldsymbol{x}^{(t+1)}|\boldsymbol{X}_{[0, t]})](https://data-science-blog.com/en/wp-content/ql-cache/quicklatex.com-9b85b225f0635a7627a99481018f6166_l3.png "Rendered by QuickLaTeX.com")
be 
, then ![P(\boldsymbol{X}) = P(\boldsymbol{x}^{(0)})\prod_{t=0}^{\tau}{P(\boldsymbol{x}^{(t+1)}|\boldsymbol{X}_{[0, t]})}](https://data-science-blog.com/en/wp-content/ql-cache/quicklatex.com-59e2d215f0fea11ec20386dfef220a30_l3.png "Rendered by QuickLaTeX.com")
. Language models calculate which words to come sequentially in this way.
![P(\boldsymbol{x}^{(0)})\prod_{t=0}^{4}{P(\boldsymbol{x}^{(t+1)}|\boldsymbol{X}_{[0, t]})}](https://data-science-blog.com/en/wp-content/ql-cache/quicklatex.com-4d263b8554a322d3ab8a8841552bc75b_l3.png "Rendered by QuickLaTeX.com")
. Given a context 
is 
. In the figure below, the distribution at the left side is less confident because probabilities do not spread widely, on the other hand the one at the right side is more confident that next word is “got” because the distribution concentrates on “got”.
is 
.
gets higher. Thus 
gets lower, where usually 
or 
.
, which is composed of 
sentences in total. Each sentence 
![(\boldsymbol{x}^{(0)})\prod_{t=0}^{\tau ^{(n)}}{P(\boldsymbol{x}_{n}^{(t+1)}|\boldsymbol{X}_{n, [0, t]})}](https://data-science-blog.com/en/wp-content/ql-cache/quicklatex.com-ccc74ac2aee4f90a6b035dcfd2632927_l3.png "Rendered by QuickLaTeX.com")
has 
tokens in total excluding 
. And let 
, where ![z = \frac{-1}{|\mathcal{V}|}\sum_{n=1}^{|\mathcal{D}|}{\sum_{t=0}^{\tau ^{(n)}}{log_{b}P(\boldsymbol{x}_{n}^{(t+1)}|\boldsymbol{X}_{n, [0, t]})}](https://data-science-blog.com/en/wp-content/ql-cache/quicklatex.com-c1ccb76fcad455ad635fc1ec8b2f6f81_l3.png "Rendered by QuickLaTeX.com")
. The 
is usually 
or 
.
is vocabulary {“the”, “You’ve”, “Walberg”, “touch”, “power”, “Nights”, “got”, “Mark”, “Boogie”}. Also assume that the evaluation data set for perplexity of a language model is 
, where 
. In this case 
. I have already showed you how to calculate the perplexity of the sentence “You’ve got the touch.” above. You just need to do a similar thing on another sentence “You’ve got the power”, and then you can get the perplexity of the language model.
be a vector space and let 
be a mapping of 
into itself, defined as 
, where 
is a 
matrix and 
is 
is called an eigen vector if there exists a number 
such that 
and 
. In this case 
, belonging to 
. If 
is basis of the vector space 
matrices 
, whose column vectors are eigen vectors 
, where 
.
Most textbooks keep explaining these type of stuff, but I have to say they lack efforts to make it understandable to readers with low mathematical literacy like me. Especially if you have to apply the idea to data science field, I believe you need more visual understanding of diagonalization. Therefore instead of just explaining the definitions and theorems, I would like to take a different approach. But in order to understand them in more intuitive ways, we first have to rethink waht linear transformation 
means in more visible ways.
is a vector transformed by 
*I am not going to use the term “linear transformation” in a precise way in the context of linear algebra. In this article or in the context of data science or machine learning, “linear transformation” for the most part means products of matrices or vectors. 
Let’s calculate the displacement vector with more vectors 
, and I prepared several grid vectors 
in purple.
square matrices 
, and I plotted displace vectors made by the matrices respectively in the figure below.
, the matrix does not have any real eigan values.
is classified to, and I am going to explain positive semidefinite matrices in the fourth section.
be the number of pairs of grown up rabbits in the 
generation. One pair of grown up rabbits produce one pair of young rabbit The concrete values of 
are 
, 
, 
, 
, 
, 
, 
, 
. Assume that 
and that 
, then you can calculate the number of the pairs of grown up rabbits in the next generation with the following recurrence relation. 
.Let 
be 
, then the recurrence relation can be written as 
, and the transition of 
are eigen values of 
are eigen vectors belonging to them respectively. Also let 
scalars such that 
. According to the definition of eigen values and eigen vectors belonging to them, the following two equations hold: 
. If you calculate 
is, using eigen vectors of 
. In the same way, 
, and 
. These equations show that in coordinate system made by eigen vectors of 
for all values of the vector 
, the 
for all the eigen values 
.
for all values of the vector 
square positive semidefinite matrix 
, whose linear transformation I visualized the second section, is also positive semidefinite.
. I visualized the displacement vectors made by the 
.
, there exist orthonormal matrices 
such that 
, where 
.
, where 
. In other words column vectors 
form an orthonormal coordinate system.
. Combining this fact with what I have told you so far, you we can reach one conclusion that you can orthogonalize a real symmetric matrix 
. This is known as spectral decomposition or singular value decomposition.
is also orthonormal. In other words, assume 
, 
also forms a orthonormal coordinate system.
expands or contracts vectors along each axis. I am going to explain that more precisely in the upcoming articles.
', va='center',ha='right', fontsize=fontsize + 10)
plt.text(eigen_vectors.T[1][0] - small_shift, eigen_vectors.T[1][1],r'
', va='center',ha='right', fontsize=fontsize + 10)
for i in range(total_step):
plt.arrow(0, 0, v_1_list[i+1][0], v_1_list[i+1][1], head_width=0.05, color='darkviolet', length_includes_head=True)
plt.arrow(0, 0, v_2_list[i+1][0], v_2_list[i+1][1], head_width=0.05, color='darkviolet', length_includes_head=True)
plt.text(v_1_list[i+1][0] + 2*small_shift , v_1_list[i+1][1]-2*small_shift,r'
'.format(1,i+1, 1),va='center',ha='right', fontsize=fontsize)
plt.text(v_2_list[i+1][0]-0.1, v_2_list[i+1][1],r'
'.format(2, i+1, 2),va='center',ha='right', fontsize=fontsize)
plt.arrow(v_1_list[i+1][0], v_1_list[i+1][1], v_2_list[i+1][0], v_2_list[i+1][1], head_width=0, color='black', linestyle='--', length_includes_head=True)
plt.arrow(v_2_list[i+1][0], v_2_list[i+1][1], v_1_list[i+1][0], v_1_list[i+1][1], head_width=0, color='black', linestyle='--', length_includes_head=True)
orange_patch = mpatches.Patch(color='orange', label='Eigen vectors')
purple_patch = mpatches.Patch(color='darkviolet', label='Scalar multiples of the eigen vectors')
plt.legend(handles=[orange_patch, purple_patch], fontsize=25, loc='lower right')
for i in range(total_step):
plt.text(x[i][0]+0.1, x[i][1]-0.05, r'n={0}'.format(i), fontsize=20)
plt.grid(True)
plt.ylabel("
", fontsize=20)
plt.xlabel("
", fontsize=20)
plt.title("Fibonacci sequence and its eigen space", fontsize=30)
#plt.savefig("Fibonacci_eigen_space.png")
plt.show()
has two identical real unit eigen vectors.", r"
', fontsize=20)
ax.text(eigen_vectors.T[1][0]*8 , eigen_vectors.T[1][1]*8, eigen_vectors.T[1][2]*8, r'
', fontsize=20)
ax.text(eigen_vectors.T[2][0]*8 , eigen_vectors.T[2][1]*8, eigen_vectors.T[2][2]*8, r'
', fontsize=20)
grey_patch = mpatches.Patch(color='grey', label='Grids')
orange_patch = mpatches.Patch(color='orange', label='Orthogonal eigen vectors of A')
purple_patch = mpatches.Patch(color='darkviolet', label='Displacement vectors made by A')
plt.legend(handles=[grey_patch, orange_patch, purple_patch], fontsize=20, loc='lower right')
ax.set_xlabel(r'
', fontsize=25)
ax.set_ylabel(r'
', fontsize=25)
ax.set_zlabel(r'
', fontsize=25)
#plt.savefig("symmetric_positive_definite_visualizaiton.png")
plt.show()
