TextBlob: Simplified Text Processing

TextBlob is a Python (2 and 3) library for processing textual data. It provides a simple API for diving into common natural language processing (NLP) tasks such as part-of-speech tagging, noun phrase extraction, sentiment analysis, classification, translation,and more.

TextBlob stands on the giant shoulders of NLTK and pattern, and plays nicely with both.

Features

Noun phrase extraction Part-of-speech tagging Sentiment analysis

Classification (Naive Bayes, Decision Tree)

Language translation and detection powered by Google Translate Tokenization (splitting text into words and sentences)

Word and phrase frequencies Parsing

n-grams

Word inflection (pluralization and singularization) and lemmatization Spelling correction

Add new models or languages through extensions WordNet integration

Installation

Installing/Uploading from the PyPi

This will install TextBlob and download the necessary NLTK corpora. If you need to change the default download directory set the NLTK_DATA environment variable.

Downloading the minimum corpora:

If you only intend to use TextBlob’s default models (no model overrides), you can pass the lite argument. This downloads only those corpora needed for basic functionality.

Installing with Conda

Note: Conda builds are currently available for Mac OSX only.

TextBlob is also available as a conda package. To install with conda, run

conda install -c https://conda.anaconda.org/sloria textblob python -m textblob.download_corpora

Python

TextBlob supports Python >=2.7 or >=3.4.

Dependencies

TextBlob depends on NLTK3. NLTK will be installed automatically when you run

TextBlob aims to provide access to common text-processing operations through a familiar interface. You can treat TextBlob objects as if they were Python strings that learned how to do Natural Language Processing.

Create a TextBlob

First, the import

Part-of-speech(POS) Tagging

Parts-of-speech tags can be accessed through the tags property.

NameError Traceback (most recent call last)

NameError: name 'wiki' is not defined

Noun Phrase Extraction

Similarly, noun phrases are accessed through the noun_phrases property.

NameError Traceback (most recent call last)

NameError: name 'wiki' is not defined

Sentiment Analysis

The sentiment property returns a named tuple of the form Sentiment(polarity, subjectivity). The polarity score is a float within the range [-1.0, 1.0]. The subjectivity is a float within the range [0.0, 1.0] where 0.0 is very objective and 1.0 is very subjective.

Tokenization

Data is a new fuel.

Explicit is better than implicit. Simple is better than complex.

Word Inflection and lemmatization

Each word in the TextBlob.words or Sentence.words is a Word object(a subclass of unicode) with useful methods, e.g. for word inflection.

Words can be lemmatized just by calling the lemmatize method.

WordNet Integeration

You can access the synets for a Word via the synsets property or the get_synsets method optionally passing in a parts-of-speech.

WordNet

WordNet is a lexical database that is dictionary for the English language, it is specifically for the natural language processing.

Synset

It is a special kind of a simple interface that is present in the NLTK for look up words in WordNet. Synset instances are the groupings of synonymous that express the same type of concept. Some words have only one synset and some have several.

You can access the definitions for each synset via the definitions property or the define()

method, which can also take an optional part-of-speech(pos) argument.

You can also create synsets directly.

WordLists

A wordlist is just the Python list with additional methods.

WordLists will find it out the words which are in the sentence and ignore the spaces in between them.

Spelling Correction

For correcting the words you can use correct() method to attempt spelling correction.

An you pronounce czechoslovakia?

Word objects have a spellcheck() Word.spellcheck() , this method that returns a list of (word, confidence) tuples with spelling suggestions.

This spelling correction is based on the Peter Norvig's "How to Write a Spelling Corrector", as implemented in the pattern library. It is about 70% accurate.

Get Word and Noun Phrase Frequencies

There are two ways to get the frequency of a word or noun phrase in the TextBlob

The first one is through the word_counts dictionary.

If you access the frequencies this way, the search will not be case sensitive, and words that are not found will have a frequency of 0.

The second way is to use the count() method.

You can specify whether or not the search should be case-sensitive (default is False).

In the above example we have given 'Sea' and ofcourse 'Sea' is not available in the sentence, 'Sea' is available in sentence but in lowercase beacause of that it given 0 as a result.

Each of these methods can also be used with noun phrases.

Translation and Language Detection

TextBlobs can be translated between languages.

If no source language is specified, TextBlob will attempt to detect the language. You can specify the source language explicitly, like so. Raises TranslatorError if the TextBlob cannot be translated into the requested language or NotTranslated if the translated result is the same as the input string.

You can also attempt to detect a TextBlob’s language using TextBlob.detect_language().

NameError: name 'TextBlob' is not defined

Parsing

Use the parse() method to parse the text.

And/CC/O/O now/RB/B-ADVP/O for/IN/B-PP/B-PNP something/NN/B-NP/I-PNP completely/RB/B-ADJP/O different/JJ/I-ADJP/O ././O/O

By default, TextBlob uses pattern’s parser

TextBlobs Are Like Python Strings!

You can use Python’s substring syntax.

We can use it as common string method.

You can make comparisons between TextBlobs and strings.

You can concatenate and interpolate TextBlobs and strings.

n-grams

The TextBlob.ngrams() method returns a list of tuples of n successive words.

Get Start and End Indices of Sentences

Use sentence.start and sentence.end to get the indices where a sentence starts and ends within a TextBlob.

Data is a new fuel.

---- Starts at index 0, Ends at index 19 Explicit is better than implicit.

---- Starts at index 20, Ends at index 53 Simple is better than complex.

---- Starts at index 54, Ends at index 84

In [0]:

Let's start building the Text Classification system

The textblob.classifiers module makes it simple to create custom classifiers. As an example, let’s create a custom sentiment analyzer.

Loading Data and Creating a Classifier

First we’ll create some training and test data.

Now we’ll create a Naive Bayes classifier, passing the training data into the constructor.

Loading Data from Files

You can also load data from common file formats including CSV, JSON, and TSV. CSV files should be formatted like so:

I love this sandwich.,pos This is an amazing place!,pos

I do not like this restaurant,neg

JSON files should be formatted like so:

Classifying Text

Call the classify(text) method to use the classifier.

You can get the label probability distribution with the prob_classify(text) method.

Classifying TextBlobs

Another way to classify text is to pass a classifier into the constructor of TextBlob and call its

classify() method.

The advantage of this approach is that you can classify sentences within a TextBlob.

Alcohol is good. pos

But the hangover is horrible. pos

Evaluating Classifiers

To compute the accuracy on our test set, use the accuracy(test_data) method.

Use the show_informative_features() method to display a listing of the most informative features.

Updating Classifiers with New Data

Use the update(new_data) method to update a classifier with new training data.

Feature Extractors

By default, the NaiveBayesClassifier uses a simple feature extractor that indicates which words in the training set are contained in a document.

For example, the sentence “I love” might have the features contains(love): True or contains(hate): False.

You can override this feature extractor by writing your own. A feature extractor is simply a function with document (the text to extract features from) as the first argument. The function may include a second argument, train_set (the training dataset), if necessary.

The function should return a dictionary of features for document.

For example, let’s create a feature extractor that just uses the first and last words of a document as its features.

We can then use the feature extractor in a classifier by passing it as the second argument of the constructor.