Suppose, We're in a situation to create a Sentiment analysis model, we have the dataset available but the problem here is machine will not going to understand the sentences of any languages, we have to clean those dataset by using stopwords, deleting punctuation and many more irrelevant things inside the data and We have to make it upto that level where we can feed those data to our machine or deep learning algorithms from that we can get some output with it.

We are assuming you have a knowledge of Python, and if not, nothing to worry, we will give you some overview of it.

1.Python String

A String is like a sequence of Characters.

A character is just a symbol.Like, the English language has 26 characters.

Computers don't deal with characters,they deal with binary(numbers) only. Even though you seen characters but internally it is stored and manipulated with the combination of 0's and 1's. The conversion of character to a number is known as encoding, and the reverse is decoding.

String literally surrounded by a single or double quotations like:

LiveAdmins

Strings are arrays

Like in other popular programming languages, strings in Python are arrays of bytes represents unicode characters.

However, Python doesn't have the character as the datatype, single character is just simply a string with length of 1.

We used square bracket to access elements from the string.

#Get the character at position 4(Here, indexing starts from 0)

Slicing

You can get the output upto certain range of characters by using the slicing index.

String Methods

Python have set of built-in methods that you can use on strings.

#strip() will remove whitespace in the string from begining to the end

#lower() will lowercase the words which are upper in the sentences.

#upper() will transform lowercase into upper.

liveadmins LIVEADMINS

#replace() will work like replace one string with another string.

#split() will split the strings into substrings if it finds any instan

String Concatenation

To concatenate or combine two strings by use of + operator.

LiveAdmins Data

Import text

In NLP your dataset would be in .txt, .csv type of format, you need to import it and try to clean the irrelevant datas from the dataset.Here we'll going to understand Python file handling:create,open,read,append,write.

Create a text file

With the use of python you can create the text files by using code,we have demostrated here how you can do this.

ModuleNotFoundError Traceback (most recent call last)

Here we declared file as a variable to open a file named LiveAdmins.txt.Open takes two arguemnets, first one is for the file we want to open and second one represents some kind of permissions or operation we want to do into that file.

Here we taken "w" letter as an arguemnt, which indicates write and will create a file if it is not exist in the library.

That "+" signs indicate both read and write.

The other option beside "w" are, "r" for read, and "a" for append.

#Step2

We have a loop that runs over a range of 5 numbers.

Here using the Write function for entering data into the file.

The Output we want to iterate in the file is "Line number is", which we already declared with write function and then percent d(display integers)

#Step3

NameError Traceback (most recent call last)

This will close the instance of the file LiveAdmins.txt stored.

Append data to a file

You can also append a new text inside the existing file or in a new file.

Once again "+" sign is in the code which means if .txt file are not available, this plus sign will create a new file but here is not any requirement to create a new file.

This will write data into a file in append mode.

We seen earlier this close function will close the instance of the file LiveAdmins.txt stored.

Read the files

Not only you create .txt files in Python but you can also call .txt file in a "read mode"(r).

#Step1. Open the file in read mode

FileNotFoundError Traceback (most recent call last)

FileNotFoundError: [Errno 2] No such file or directory: 'LiveAdmins.t xt'

#Step2. Here we'll check, is our file is open or not, if yes we procee

#Step3. Printing the file

Line number is 1 Line number is 2 Line number is 3 Line number is 4 Line number is 5

Appending Line number 1 Appending Line number 2 Appending Line number 3

Here is the output!

Files mode in Python

Web Scraping with Python

Suppose you have to pull large amount of data from websites and you want to fetch it as quickly as possible. How would you do it? Manually going to the website and collect those datas.It will be tedious work. So, "web scrapping" will help you out in this situation. Web scrapping just makes this job easier and faster.

Here, we will do Web Scrapping with Python, starts with

1. Why we do web scrapping? Web scrapping will be used to collect large amount of data from Websites.But why does someone have to collect such large amount of data from websites? To know about this, let's have to look at the applications of web scrapping:

   
   

   Price comaparison: Parsehub is providing such services to useweb scraping to collect data from some online shopping websites and use to comapre price of products from another.

   Gathering Emails: There are lots of companies that use emails as a medium for marketing, they use web scrapping to collect email id's and send bulk emails.

   Social media scrapping: Web scrapping is used to collect data from Social Media websites such as Twitter to find out what's trending in twitter.

   Research and Development: For reasearch purposes people do web scrapping to collect a large set of data(Statistics, General information, temperature,etc.) from

   websites, which are analyzed and used to carry out surveys or for R&D.

   
   

2. What is Web scrapping and is it legal or not? Web scrapping is an automated to extract large amount of data from websites.And the websites data are unstructured most of the time.Web scrapping will help you out to collect those unstructured data and stored it in a structured form.There are different ways to scrape websites such as online

   services,APIs, or by writing your own code. Here, we'll see how to implementing the web scraping with python.

   
   

   Coming to the question, is scrapping legal or not? Some websites allow web scrapping and some not.To know whether website allows you to scrape it or not by website's "robots.txt" file. You can find this file just append "/robots.txt" to the URL that you want to scrape.Here, we're scrapping from Flipkart website.So, to see the "robots.txt" file, URL is

   www.flipkart.com/robots.txt ( http://www.flipkart.com/robots.txt).

   
   

3. How does web scrapping work?

   
   

   When we run the code for web scraping, a request is sent to the URL that you have mentioned in the code. As a response to the request, the server send the data and allows you to read the HTML or XML page. Then our code will parses the HTML or XML page, find the data and extract it.

   
   

   To extract datas using web scraping with python, you need to follow these basic steps:

   
   

   1.Find that URL that you mentioned in the code and want to scrape it. 2.Inspect the Page for scraping. 3.Find those data you want to extract. 4.Write the code for scrapping. 5.Run the code and extract the data. 6.Store the data in the required format.

   
   

   Now lets see how to extract data from the flipkart website using Python.

   
   

4. Libraries used for Web scrapping

   
   

   We already know, that python used for various applications and there are different libraries for different purposes.In this, we're using the following libraries:

   
   

   Selenium: Selenium library is used for web testing. We will use to automate browser activities.

   BeautifulSoup4: It is generally used for parsing HTML and XML documents.It creates a parse trees that is helpful to extract the datas easily.

   Pandas: It is a Python library used for data manipulation and analysis.Pandas is used to extract data and stored it in the desired format.

   
   

5. For Demo Purpose : Scrapping a Flipkart Website Pre-requisites:

Python 3.x with Selenium, Beautifulsoup4, Pandas library installed. Google Chrome Browser

You can go through this link ( https://github.com/iNeuronai/webscrappper_text.git) for more

In [ ]:

Text Preprocessing

Supose we have textual data available, we need to apply many of pre-processing steps to the data to transform those words into numerical features that work with machine learning algorithms.

The pre-processing steps for the problem depend mainly on the domain and the problem itself.We don't need to apply all the steps for every problem.

Text lowercase

We do lowercase the text to reduce the size of the vocabulary of our text data.

Remove numbers

We should either remove the numbers or convert those numbers into textual representations. We use regular expressions(re) to remove the numbers.

# For Removing numbers

As we mentioned above,you can also convert the numbers into words. This could be done by using the inflect library.

Remove Punctuation

We remove punctuations because of that we don't have different form of the same word. If we don't remove punctuations, then been, been, and been! will be treated separately.

The str.maketrans() method in Python is used to create a translation table that can be used with the str.translate() method to perform specific character replacements in a string. This is often used for tasks like removing or replacing certain characters in a string.

Here's what each argument in str.maketrans() means:

1. The first argument is the set of characters you want to replace. In this case, '' (empty string) means that you don't want to replace any characters.

2. The second argument is the set of characters you want to remove. In this case, string.punctuation is a string constant that contains all punctuation characters (like periods, commas, exclamation marks, etc.).

Putting it together, str.maketrans('', '', string.punctuation) creates a translation table that essentially says "don't replace any characters, but remove all punctuation characters."

You can then use this translation table with the str.translate() method to apply these replacements to a string. For example:

Output:

Hello world How are you

In this example, the cleaned_text variable will contain the original string with all punctuation removed.

In [ ]:

Remove default stopwords:

Stopwords are words that do not contribute to the meaning of the sentence. Hence, they can be safely removed without causing any change in the meaning of a sentence. The

NLTK(Natural Language Toolkit) library has the set of stopwords and we can use these to remove stopwords from our text and return a list of word tokens.

Stemming

From Stemming we will process of getting the root form of a word. Root or Stem is the part to which inflextional affixes(like -ed, -ize, etc) are added. We would create the stem words by removing the prefix of suffix of a word. So, stemming a word may not result in actual words.

If our sentences are not in tokens, then we need to convert it into tokens. After we converted strings of text into tokens, then we can convert those word tokens into their root form. These are the Porter stemmer, the snowball stemmer, and the Lancaster Stemmer. We usually use Porter stemmer among them.

Lemmatization

As stemming, lemmatization do the same but the only difference is that lemmatization ensures that root word belongs to the language. Because of the use of lemmatization we will get the valid words. In NLTK(Natural language Toolkit), we use WordLemmatizer to get the

lemmas of words. We also need to provide a context for the lemmatization.So, we added pos(parts-of-speech) as a parameter.

Parts of Speech (POS) Tagging

The pos(parts of speech) explain you how a word is used in a sentence. In the sentence, a word have different contexts and semantic meanings. The basic natural language processing(NLP) models like bag-of-words(bow) fails to identify these relation between the words. For that we use pos tagging to mark a word to its pos tag based on its context in the data. Pos is also used to extract rlationship between the words.

In the above example NNP stands for Proper noun, PRP stands for personal noun, IN as Preposition. We can get all the details pos tags using the Penn Treebank tagset.

Chunking

Chunking is the process of extracting phrases from the Unstructured text and give them more structure to it. We also called them shallow parsing.We can do it on top of pos tagging. It groups words into chunks mainly for noun phrases. chunking we do by using regular expression.

In the above example, we defined the grammar by using the regular expression rule. This rule tells you that NP(noun phrase) chunk should be formed whenever the chunker find the optional determiner(DJ) followed by any no. of adjectives and then a NN(noun).

Image after running above code.

Libraries like Spacy and TextBlob are best for chunking.

Named Entity Recognition

It is used to extract information from unstructured text. It is used to classy the entities which is present in the text into categories like a person, organization, event, places, etc. This will give you a detail knowledge about the text and the relationship between the different entities.