How Sentiment Analysis Is Used To Analyze Movie Reviews?

Sentiment analysis is one of the most widely used applications of natural language processing and text analytics, with a variety of sites, publications, and courses dedicated to the subject. Sentiment analysis appears to be the best to operate on subjective material, in which people express their thoughts, feelings, and mood. Sentiment analysis is commonly utilized in the real world to assess corporate investigations, feedback survey data, social media posts, and reviews for movies, places, etc.


1. Text Corpus

A Text Corpus consists of several text documents that is as simple as a single sentence, or complicated as a multi-paragraph document.

2. Sentiment Analysis

Sentiment analysis is also known as opinion analysis or opinion harvesting. The basic idea is of using various tricks from text analytics, NLP (Natural Language Processing), Machine Learning, and languages to scrape necessary data from unformatted text.

3. Sentiment Polarity

Sentiment Polarity is the numeric result that is allocated to both the positive and negative factors of the text document that is dependent on subjective factors like particular words and phrases that will express feelings and emotions. Natural sentiment generally consists of 0 polarities as it does not show the particular sentiment. The positive sentiment that will consist >0, and negative <0.

Here, we will show a step-by-step process that will make a model performing Sentiment Analysis on a huge Movie database. The information was gathered from the Internet Movie Database (IMDB).

We concentrate on analyzing a large corpus of movie reviews and gather opinions. We are attempting to extract sentiment from a big corpus of movie reviews. We’ll use both an unsupervised lexicon-based model and a typical supervised machine learning model to conduct our research.

The main goal is to estimate how people will react to various movie reviews from the Internet Movie Database (IMDb).


Based on the content of the reviews, this data consists of 50,000 movie reviews that were pre-labeled with “good” and “negative” sentiment different classifiers. Hence, our task will be to expect the opinion of 15,000 labeled movie reviews and the use of other 35,000 reviews to training our supervised models.

Setting up Dependencies

We will use various Python tools and frameworks dedicated to text analytics, Natural Language Processing, and machine learning. We must first install Pandas, NumPy, SciPy, scikit-learn, seaborn, matplotlib, and NLP libraries before initiating the project.

After installing nltk with pip or conda, we need to type the following code from a Python or ipython terminal.

To install the library and receive the English model dependencies for spacy, write the following code as administrator in a Unix shell/windows command line.

The important step before going into the phase of feature engineering and modeling involves cleaning, pre-processing, and normalizing information to get textual elements such as phrases and words to a customized standard.

This allows document corpus standardization, which aids in the development of significant features and the reduction of noise supplied by a variety of elements such as irrelevant symbols, special characters, XML and HTML tags, and so on. Use the text utility to perform the following tasks.

Important Libraries

Cleaning Text:

Unnecessary material such as HTML elements, frequently appears in our text and adds little value when analyzing sentiment. As a result, we must ensure that they are removed before extracting features. This is done using the strip html tags(…) method.

Deleting Accented Characters.

In our database, we deal with reviews of the English language hence we assure that characters of any format, especially accented characters are transformed and standardized into ASCII characters.

Expanding Contractions

Contractions are abbreviated versions of words or phrases in the English language. By eliminating key letters and sounds, these abbreviated versions of preexisting words and phrases are generated. For instance, expand don’t to do not and I’d to I Would.

Removing Special Characters

This can be accomplished with the help of simple regexes. If you don’t want numbers in your normalized corpus, you can keep them or remove them. Remove special characters(…) is a function that assists us in removing special characters.

Removing Stop Words

Words with little or no significance especially while developing meaningful features from the text are known as stop words. Words, such as a, an, the, and others are considered to be phrases. The function remove_stopwords(…) will help remove stopwards and retain words with the most significance and context in a corpus.


Lemmatization is similar to stemming in which it removes language affixes to get at a word’s base form. To keep lexicographically correct words, we’ll exclusively use Lemmatization in our normalization pipeline. This is where the function lemmatize text(…) comes in handy.

Building Text-Normalizer

We will utilize all the components and hold them all together in the following function known as normalize_corpus(…), which is used to take a document corpus as input and return the similar corpus with cleaned and normal text documents.

Load Dataset and Normalize the Data

Load the IMDb dataset after importing the appropriate libraries. The dataset will then be divided into training and testing datasets. The normalized corpus(…) method will be used to standardize the information.

1st Approach — Undergoing Sentiment Analysis through Unsupervised Lexicon-Based Models

Unsupervised sentiment opinion models depend on good knowledge bases, taxonomies, lexicons, and datasets that includes precise information on words or phrases such as emotion, mood, partiality, and impartiality. A lexicon model that uses lexicon and also known as dictionary or vocabulary of words that specifically match sentiment classification.

These lexicons often include a set of words related to positive and negative emotion, polarity (the magnitude of a negative or positive score), mood, parts of speech (POS) tags, subjectivity classifiers (strong, weak, neutral), and modality, among other things. There are various other lexicon models, used for opinion analysis which include AFINN Lexicon, SentiWordNet Lexicon, and VADER Lexicon.

Model Training, Forecasting, and Progress Analysis

1. Sentiment Analysis

One of the most basic and often used lexicons for sentiment analysis is the AFINN lexicon. There are over 3300+ words in total, each with a polarity score. A built-in function for this lexicon exists in Python

2. Sentiment Analysis with SentiWordNet

SentiWordNet is a sentiment lexicon derived from the WordNet database where every term is related with numeric scores which indicates positive and negative sentiment data.

3. Sentiment Analysis with VADER

VADER is Valence Aware Dictionary for Sentiment Reasoning model is used for the text opinion analysis which is sensitive for both polarity (positive/negative) and intensity (strength) of emotions.

Which Unsupervised Model Works Best?

It is found that AFINN lexical model has better accuracy of 71.18% and works best among all three lexical models. It is also observed that the performance of the other models is near to that of the AFINN model on the given data.

2nd Approach: Sentiment Classification with Supervised Learning

Using supervised Machine Learning to develop a model to interpret the textual data and forecast the sentiment of text-based evaluations is another option. To be more explicit, classification models will be used to solve this challenge.

Feature Engineering

Word embedding is a method that uses vectors to represent the text. The following are the most popular word embedding:

1. BoW: Bag of words

The Bag of Words (BoW) model is the most basic type of numerical text representation. A phrase can be represented as a bag of words vector, just like the term itself (a string of numbers).

2. TF-IDF Term Frequency-Inverse Document frequency

The term frequency-inverse document frequency statistic is a quantitative expression of how essential a phrase is to a page in a catalog or corpus.

Model Training, Prediction, and Performance Assessment

1. To determine the likelihood of a binary event occurring and to deal with classification concerns, logistic regression is utilized.

Logistic Regression with BoW features

Logistic Regression using TF-IDF features

2. Stochastic Gradient Descent (SGD)

SGD (Stochastic Gradient Descent) is a straightforward but effective optimization approach for determining the values of the parameters of functions that reduce an objective function. In other words, it’s utilized to learn discriminative binding strengths like SVM and Logistic regression.

SVM using BoW features

SVM using TF-IDF features

Which Model Performs Best?

We have discovered that the Logistic Regression model on Bag of Word characteristics performs the best, with an accuracy of 90.65%. Other models’ performance is likewise extremely similar to this one, as can be shown.


By utilizing Unsupervised Lexicon-based models, it is observed that effective Lexicon model as AFFIN that has an accuracy of 71%. Where on using traditional supervised ML models there is a precision of 89–90%. With an accuracy of 90.65%, a logistic regression model trained with bag-of-words features outperformed all other models and an F1 score of about 0.9. It can be concluded that typical supervised models outperform lexical models by comparing the top models from both supervised and unsupervised learning.

For any queries, contact X-Byte Enterprise Crawling today!

Founder of “X-Byte Enterprise Crawling”, a well-diversified corporation providing Enterprise grade Web Crawling service & solution, leveraging Cloud DaaS model