Skip to content

[lab-nlp] Jose Costa #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jcosta92 wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[lab-nlp] Jose Costa #51

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
260 changes: 260 additions & 0 deletions challenge-1.ipynb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,260 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Challenge 1: Prepare Textual Data for Analysis\n",
"\n",
"In this challenge, we will walk you through how to prepare raw text data for NLP analysis. Due to time limitation, we will cover **text cleaning, tokenization, stemming, lemmatization, and stop words removal** but skip POS tags, named entity recognition, and trunking. The latter 3 steps are more advanced and not required for our next challenge on sentiment analysis. \n",
"\n",
"## Objectives\n",
"\n",
"* Learn how to prepare text data for NLP analysis in Python\n",
"* Write the functions you will use in Challenge 3 for cleaning, tokenizing, stemming, and lemmatizing data."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Text Cleaning\n",
"\n",
"Text cleaning is also called text cleansing. The goal is to clean up the messy real-world textual data in order to improve the text analysis accuracy at later steps. For generic textual data sources, we usually need to fix the following problems:\n",
"\n",
"* Missing values\n",
"* Special characters\n",
"* Numbers\n",
"\n",
"For web data, we need to additinally fix:\n",
"\n",
"* HTML tags\n",
"* JavaScripts\n",
"* CSS\n",
"* URLs\n",
"\n",
"Case by case, there may also be special problems we need to fix for certain types of data. For instance, for Twitter tweets data we need to fix hashtags and the Twitter handler including a *@* sign and Twitter usernames.\n",
"\n",
"In addition, we also need to convert the texts to lower cases so that when we anaylize the words later, NLTK will not think *Ironhack* and *ironhack* mean different things.\n",
"\n",
"Note that the above are the general steps to clean up data for NLP analysis. In specific cases, not all those steps apply. For example, if you are analyzing textual data on history, you probably don't want to remove numbers because numbers (such as years and dates) are important in history. Besides, if you are doing something like network analysis on web data, you may want to retain hyperlinks so that you will be able to extract the outbounding links in the next steps. Sometimes you may also need to do some cleaning first, then extract some features, then do more cleaning, then extract more features. You'll have to make these judgments by yourself case by case. \n",
"\n",
"In this challenge we are keeping things relatively simple so **you only need to clean up special characters, numbers, and URLs**. Let's say you have the following messy string to clean up:\n",
"\n",
"```\n",
"@Ironhack's-#Q website 776-is http://ironhack.com [(2018)]\")\n",
"```\n",
"\n",
"You will write a function, which will be part of you NLP analysis pipeline in the next challenge, to clean up strings like above and output:\n",
"\n",
"```\n",
"ironhack s q website is\n",
"```\n",
"\n",
"**In the cell below, write a function called `clean_up`**. Test your function with the above string and make sure you receive the expected output.\n",
"\n",
"*Notes:*\n",
"\n",
"* Use regular expressions to identify URL patterns and remove URLs.\n",
"\n",
"* You don't want to replace special characters/numbers with an empty string. Because that will join words that shouldn't be joined. For instance, if you replace the `'` in `you're`, you will get `youre` which is undesirable. So instead, replace special characters and numbers with a whitespace.\n",
"\n",
"* The order matters in terms of what to clean before others. For example, if you clean special characters before URLs, it will be difficult to identify the URLs patterns.\n",
"\n",
"* Don't worry about single letters and multiple whitespaces in your returned string. In our next steps those issues will be fixed."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# Libraries\n",
"\n",
"import re\n",
"import nltk\n",
"from nltk.tokenize import word_tokenize\n",
"from nltk.stem import PorterStemmer, WordNetLemmatizer\n",
"from nltk.corpus import stopwords"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"def clean_up(s):\n",
" \"\"\"\n",
" Cleans up numbers, URLs, and special characters from a string.\n",
"\n",
" Args:\n",
" s: The string to be cleaned up.\n",
"\n",
" Returns:\n",
" A string that has been cleaned up.\n",
" \n",
" \"\"\"\n",
" \n",
" s = re.sub(r\"http\\S+|www\\S+|https\\S+\", \"\", s)\n",
"\n",
" s = re.sub(r\"[^a-zA-Z]\", \" \", s)\n",
"\n",
" s = s.lower()\n",
" \n",
" return s"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Tokenization\n",
"\n",
"We have actually discussed the concept of tokenization in the Bag of Words lab before. In that lab, we did both tokenization and calculated the [matrix of document-term frequency](https://en.wikipedia.org/wiki/Document-term_matrix). In this lab, we only need tokenization.\n",
"\n",
"In the cell below, write a function called **`tokenize`** to convert a string to a list of words. We'll use the string we received in the previous step *`ironhack s q website is`* to test your function. Your function shoud return:\n",
"\n",
"```python\n",
"['ironhack', 's', 'q', 'website', 'is']\n",
"```\n",
"\n",
"*Hint: use the `word_tokenize` function in NLTK.*"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"def tokenize(s):\n",
" \"\"\"\n",
" Tokenize a string.\n",
"\n",
" Args:\n",
" s: String to be tokenized.\n",
"\n",
" Returns:\n",
" A list of words as the result of tokenization.\n",
" \"\"\"\n",
" return word_tokenize(s)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Stemming and Lemmatization\n",
"\n",
"We will do stemming and lemmatization in the same step because otherwise we'll have to loop each token lists twice. You have learned in the previous challenge that stemming and lemmatization are similar but have different purposes for text normalization:\n",
"\n",
"**Stemming reduces words to their root forms (stems) even if the stem itself is not a valid word**. For instance, *token*, *tokenize*, and *tokenization* will be reduced to the same stem - *token*. And *change*, *changed*, *changing* will be reduced to *chang*.\n",
"\n",
"In NLTK, there are three stemming libraries: [*Porter*](https://www.nltk.org/_modules/nltk/stem/porter.html), [*Snowball*](https://www.nltk.org/_modules/nltk/stem/snowball.html), and [*Lancaster*](https://www.nltk.org/_modules/nltk/stem/lancaster.html). The difference among the three is the agressiveness with which they perform stemming. Porter is the most gentle stemmer that preserves the word's original form if it has doubts. In contrast, Lancaster is the most aggressive one that sometimes produces wrong outputs. And Snowball is in between. **In most cases you will use either Porter or Snowball**.\n",
"\n",
"**Lemmatization differs from stemming in that lemmatization cares about whether the reduced form belongs to the target language and it often requires the context (i.e. POS or parts-of-speech) in order to perform the correct transformation**. For example, the [*Word Net lemmatizer* in NLTK](https://www.nltk.org/_modules/nltk/stem/wordnet.html) yields different results with and without being told that *was* is a verb:\n",
"\n",
"```python\n",
">>> from nltk.stem import WordNetLemmatizer\n",
">>> lemmatizer = WordNetLemmatizer()\n",
">>> lemmatizer.lemmatize('was')\n",
"'wa'\n",
">>> lemmatizer.lemmatize('runs', pos='v')\n",
"'be'\n",
"```\n",
"\n",
"In the cell below, import the necessary libraries and define a function called `stem_and_lemmatize` that performs both stemming and lemmatization on a list of words. Don't worry about the POS part of lemmatization for now."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def stem_and_lemmatize(l):\n",
" \"\"\"\n",
" Perform stemming and lemmatization on a list of words.\n",
"\n",
" Args:\n",
" l: A list of strings.\n",
"\n",
" Returns:\n",
" A list of strings after being stemmed and lemmatized.\n",
" \"\"\"\n",
" \n",
" stemmer = PorterStemmer()\n",
" lemmatizer = WordNetLemmatizer()\n",
" \n",
" return [stemmer.stem(lemmatizer.lemmatize(word)) for word in l]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Stop Words Removal\n",
"\n",
"Stop Words are the most commonly used words in a language that don't contribute to the main meaning of the texts. Examples of English stop words are `i`, `me`, `is`, `and`, `the`, `but`, and `here`. We want to remove stop words from analysis because otherwise stop words will take the overwhelming portion in our tokenized word list and the NLP algorithms will have problems in identifying the truely important words.\n",
"\n",
"NLTK has a `stopwords` package that allows us to import the most common stop words in over a dozen langauges including English, Spanish, French, German, Dutch, Portuguese, Italian, etc. These are the bare minimum stop words (100-150 words in each language) that can get beginners started. Some other NLP packages such as [*stop-words*](https://pypi.org/project/stop-words/) and [*wordcloud*](https://amueller.github.io/word_cloud/generated/wordcloud.WordCloud.html) provide bigger lists of stop words.\n",
"\n",
"Now in the cell below, create a function called `remove_stopwords` that loop through a list of words that have been stemmed and lemmatized to check and remove stop words. Return a new list where stop words have been removed."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"def remove_stopwords(l):\n",
" \"\"\"\n",
" Remove English stopwords from a list of strings.\n",
"\n",
" Args:\n",
" l: A list of strings.\n",
"\n",
" Returns:\n",
" A list of strings after stop words are removed.\n",
" \"\"\"\n",
" \n",
" stop_words = set(stopwords.words('english'))\n",
" \n",
" return [word for word in l if word not in stop_words]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Summary\n",
"\n",
"In this challenge you have learned several text preparation techniques in more depths including text cleaning, tokenization, stemming, lemmatization, and stopwords removal. You have also written the functions you will be using in the next challenge to prepare texts for NLP analysis. Now we are ready to move on to the next challenge - Sentiment Analysis."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.12"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Loading