This repository contains a collection of notebooks and code samples exploring various concepts and packages in Natural Language Processing (NLP). The project utilizes popular libraries such as spaCy, NLTK, stemming, lemmatization, stop words, tokenization, bag of words, bag of n-grams, TF-IDF, fastText, spaCy's word embedding, and Word2Vec from the gensim library.
This notebook showcases the importance of text preprocessing in NLP and explores techniques such as tokenization, sentence splitting, and part-of-speech tagging using spaCy and NLTK. It also covers stemming, lemmatization, and stop words removal.
In this notebook, we dive into feature extraction techniques such as the bag of words approach. It demonstrates how to represent text as numerical feature vectors by counting word occurrences. Additionally, it extends the concept to the bag of n-grams, which captures sequences of n consecutive words.
Here, we also explore the TF-IDF (Term Frequency-Inverse Document Frequency) technique, which calculates the importance of words in a document relative to a collection of documents. The notebook illustrates how to compute TF-IDF scores and utilize them for feature representation.
This notebook introduces fastText, a library for word embeddings that captures semantic and syntactic relationships between words. It explores fastText's capabilities for generating word vector representations and demonstrates their application in NLP tasks.
We also explore spaCy's word embeddings and Word2Vec from the gensim library. It provides an overview of how these pre-trained word embeddings can be used to obtain high-dimensional word vector representations and leverage their semantic meaning.