Updating week 4 content

Reference files/Week_4_content.txt

@@ -0,0 +1,630 @@
+
+In this course, you'll learn the complete NLP workflow by exploring a fascinating real-world question: Does review length and language relate to reviewer ratings and decisions in academic peer review? If so, how?
+Using data from the International Conference on Learning Representations (ICLR), you'll develop practical NLP skills while investigating how reviewers express their opinions. Each module builds upon the previous one, creating a coherent analytical pipeline from raw data to insight.
+Learning Path
+Data Loading and Initial Exploration: Setting up your environment and understanding your dataset
+Text Preprocessing and Normalization: Cleaning and standardizing text data
+Feature Extraction and Measurement: Calculating metrics from text
+Visualization and Pattern Recognition: Creating insightful visualizations
+Drawing Conclusions from Text Analysis: Synthesizing findings into actionable insights
+Let's begin our exploration of how NLP can provide insights into academic peer review!
+
+Module 1: Initial Exploration
+The Challenge
+Before we can analyze how review length relates to paper evaluations, we need to understand our dataset. In this module, we'll set up our Python environment and explore the ICLR conference data.
+1.1: Set up and get to your data
+The first step in any NLP project is loading and understanding your data. Let's set up our environment and examine what we're working with:
+python
+# Import necessary libraries
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+import seaborn as sns
+import string
+from nltk.corpus import stopwords
+from nltk.tokenize import word_tokenize, sent_tokenize
+from wordcloud import WordCloud
+
+# Load the datasets
+df_reviews = pd.read_csv('../data/reviews.csv')
+df_submissions = pd.read_csv('../data/Submissions.csv')
+df_dec = pd.read_csv('../data/decision.csv')
+df_keyword = pd.read_csv('../data/submission_keyword.csv')
+Let's look at the first few rows of each dataset to understand what information we have:
+python
+# View the first few rows of the submissions dataset
+df_submissions.head()
+# View the first few rows of the reviews dataset
+df_reviews.head()
+# View all columns and rows in the reviews dataset
+df_reviews
+# View the first few rows of the keywords dataset
+df_keyword.head()
+1.2: Looking at Review Content
+Let's examine an actual review to understand the text we'll be analyzing:
+python
+# Display a sample review
+df_reviews['review'][1]
+Think about: What kinds of information do you see in this review? What language patterns do you notice?
+1.3: Calculating Basic Metrics
+Let's calculate our first simple metric - the average review score for each paper:
+python
+# Get the average review score for each paper
+df_average_review_score = df_reviews.groupby('forum')['rating_int'].mean().reset_index()
+df_average_review_score
+Key Insight: Each paper (identified by 'forum') receives multiple reviews with different scores. The average score gives us an overall assessment of each paper.
+Module 2: Data Integration
+In this module, we'll merge datasets for later analysis.
+2.1 Understanding the Need for Data Integration
+In many NLP projects, the data we need is spread across multiple files or tables. In our case:
+The df_reviews dataset contains the review text and ratings
+The df_dec dataset contains the final decisions for each paper
+To analyze how review text relates to paper decisions, we need to merge these datasets.
+2.2 Performing a Dataset Merge
+Let's combine our review data with the decision data:
+python
+# Step 1 - Merge the reviews dataframe with the decisions dataframe
+df_rev_dec = pd.merge(
+    df_reviews,              # First dataframe (reviews) 
+    df_dec,                  # Second dataframe (decisions)
+    left_on='forum',         # Join key in the first dataframe
+    right_on='forum',        # Join key in the second dataframe
+    how='inner'              # Keep only matching rows
+)[['review','decision','conf_name_y','rating_int','forum']]  # Select only these columns
+# Display the first few rows of the merged dataframe
+df_rev_dec.head()
+2.3 Understanding Merge Concepts
+Join Key: The 'forum' column identifies the paper and connects our datasets
+Inner Join: Only keeps papers that appear in both datasets
+Column Selection: We keep only relevant columns for our analysis
+How to Verify: Always check the shape of your merged dataset to ensure you haven't lost data unexpectedly
+Try it yourself: How many rows does the merged dataframe have compared to the original review dataframe? What might explain any differences?
+
+Module 3: Basic Text Preprocessing
+In this module, you'll learn essential data preprocessing techniques for NLP projects. We'll standardize text through case folding, clean up categorical variables, and prepare our review text for analysis.
+3.1 Case Folding (Lowercase Conversion)
+A fundamental text preprocessing step is converting all text to lowercase to ensure consistency:
+python
+# Convert all review text to lowercase (case folding)
+df_rev_dec['review'] = df_rev_dec['review'].str.lower()
+# Display the updated dataframe
+df_rev_dec
+Why Case Folding Matters
+Consistency: "Novel" and "novel" will be treated as the same word
+Reduced Dimensionality: Fewer unique tokens to process
+Improved Pattern Recognition: Easier to identify word frequencies and patterns
+Note: While case folding is generally helpful, it can sometimes remove meaningful distinctions (e.g., "US" vs. "us"). For our academic review analysis, lowercase conversion is appropriate.
+
+3.2 Examining Categorical Values
+Let's first check what unique decision categories exist in our dataset:
+python
+# Display the unique decision categories
+df_rev_dec['decision'].unique()
+3.3 Standardizing Decision Categories
+We can see that there are multiple "Accept" categories with different presentation formats. Let's standardize these:
+python
+# Define a function to clean up and standardize decision categories
+def clean_up_decision(text):
+    if text in ['Accept (Poster)','Accept (Spotlight)', 'Accept (Oral)','Accept (Talk)']:
+        return 'Accept'
+    else:
+        return text
+# Apply the function to create a new standardized decision column
+df_rev_dec['decision_clean'] = df_rev_dec['decision'].apply(clean_up_decision) 
+# Check our new standardized decision categories
+df_rev_dec['decision_clean'].unique()
+Why Standardization Matters
+Simplified Analysis: Reduces the number of categories to analyze
+Clearer Patterns: Makes it easier to identify trends by decision outcome
+Better Visualization: Creates more meaningful and readable plots
+Consistent Terminology: Aligns with how conferences typically report accept/reject decisions
+Try it yourself: What other ways could you group or standardize these decision categories? What information might be lost in our current approach?
+
+Module 4: Text Tokenization
+4.1 Introduction to Tokenization
+Tokenization is the process of breaking text into smaller units like sentences or words. Let's examine a review:
+python
+# Display a sample review
+df_reviews['review'][1]
+4.2 Sentence Tokenization
+Let's break this review into sentences using NLTK's sentence tokenizer:
+python
+# Import the necessary library if not already imported
+from nltk.tokenize import sent_tokenize
+# Tokenize the review into sentences
+sent_tokenize(df_reviews['review'][1])
+4.3 Counting Sentences
+Now let's count the number of sentences in the review:
+python
+# Count the number of sentences
+len(sent_tokenize(df_reviews['review'][1]))
+4.4 Creating a Reusable Function
+Let's create a function to count sentences in any text:
+python
+# Define a function to count sentences in a text
+def sentence_count(text):
+    return len(sent_tokenize(text))
+4.5 Applying Our Function to All Reviews
+Now we'll apply our function to all reviews to get sentence counts:
+python
+# Add a new column with the sentence count for each review
+df_rev_dec['sent_count'] = df_rev_dec['review'].apply(sentence_count)
+# Display the updated dataframe
+df_rev_dec.head()
+Key Insight: Sentence count is a simple yet effective way to quantify review length. The number of sentences can indicate how thoroughly a reviewer has evaluated a paper.
+
+Module 5: Visualization of Text Metrics
+5.1 Creating a 2D Histogram
+Let's visualize the relationship between review length (in sentences), rating, and decision outcome:
+python
+# Create a 2D histogram with sentence count, rating, and decision
+ax = sns.histplot(data=df_rev_dec, x='sent_count',
+                  y='rating_int',
+                  hue='decision_clean',
+                  kde=True, 
+                  log_scale=(True,False),
+                  legend=True)
+5.2 Enhancing Our Visualization
+Let's improve our visualization with better labels and formatting:
+python
+# Set axis labels
+ax.set(xlabel='Review Length (# Sentences)', ylabel='Review Rating')
+# Move the legend outside the plot for better visibility
+sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))
+# Ensure the layout is properly configured
+plt.tight_layout()
+# Display the plot
+plt.show()
+5.3 Interpreting the Visualization
+This visualization reveals several interesting patterns:
+Length-Rating Relationship: Is there a pattern that entails how length of review is correlated with rating?
+Decision Patterns: Are there visible clusters for accepted vs. rejected papers?
+Density Distribution: Where are most reviews concentrated in terms of length and rating?
+Outliers: Are there unusually long or short reviews at certain rating levels?
+Discussion Question: Based on this visualization, do reviewers tend to write longer reviews when they're more positive or more critical? What might explain this pattern?
+
+Module 6: Additional Text Processing - Tokenization
+Tokenization is the process of breaking text into smaller units (tokens) that serve as the building blocks for natural language processing. In this lesson, we'll explore how to tokenize text, remove stopwords and punctuation, and analyze the results.
+6.1 Text Cleaning
+Before tokenization, we often clean the text to remove unwanted characters. Let's start by removing punctuation:
+python
+# Removing punctuation
+df_rev_dec['clean_review_word'] = df_rev_dec['review'].str.translate(str.maketrans('', '', string.punctuation))
+What's happening here?
+string.punctuation contains all punctuation characters (.,!?;:'"()[]{}-_)
+str.maketrans('', '', string.punctuation) creates a translation table to remove these characters
+df_rev_dec['review'].str.translate() applies this translation to all review texts
+6.2 Word Tokenization
+After cleaning, we can tokenize the text into individual words:
+python
+# Tokenizing the text
+df_rev_dec['tokens'] = df_rev_dec['clean_review_word'].apply(word_tokenize)
+
+# Example: Look at tokens for the 6th review
+df_rev_dec['tokens'][5]
+What's happening here?
+word_tokenize() is an NLTK function that splits text into a list of words
+We apply this function to each review using pandas' apply() method
+The result is a new column containing lists of words for each review
+6.3 Removing Stopwords
+Stopwords are common words like "the," "and," "is" that often don't add meaningful information for analysis:
+python
+# Getting the list of English stopwords
+stop_words = set(stopwords.words('english'))
+
+# Removing stopwords from our tokens
+df_rev_dec['tokens'] = df_rev_dec['tokens'].apply(lambda x: [word for word in x if word not in stop_words])
+What's happening here?
+stopwords.words('english') returns a list of common English stopwords
+We convert it to a set for faster lookup
+The lambda function filters each token list, keeping only words that aren't stopwords
+This creates more meaningful token lists focused on content words
+6.4 Counting Tokens
+Now that we have our cleaned and filtered tokens, let's count them to measure review length:
+python
+# Count tokens for each review
+df_rev_dec['tokens_counts'] = df_rev_dec['tokens'].apply(len)
+
+# View the token counts
+df_rev_dec['tokens_counts']
+What's happening here?
+We use apply(len) to count the number of tokens in each review
+This gives us a quantitative measure of review length after removing stopwords
+The difference between this and raw word count shows the prevalence of stopwords
+6.5 Visualizing Token Counts vs. Ratings
+Let's visualize the relationship between token count, rating, and decision:
+python
+# Create a 2D histogram with token count, rating, and decision
+ax = sns.histplot(data=df_rev_dec, x='tokens_counts',
+                  y='rating_int',
+                  hue='decision_clean',
+                  kde=True, 
+                  log_scale=(True,False),
+                  legend=True)
+
+# Set axis labels
+ax.set(xlabel='Review Length (# Tokens)', ylabel='Review Rating')
+
+# Move the legend outside the plot
+sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))
+
+plt.tight_layout()
+plt.show()
+What's happening here?
+We create a 2D histogram showing the distribution of token counts and ratings
+Colors distinguish between accepted and rejected papers
+Log scale on the x-axis helps visualize the wide range of token counts
+Kernel density estimation (KDE) shows the concentration of reviews
+Module 7: Aggregating Data by Paper
+7.1 Understanding Data Aggregation
+So far, we've been analyzing individual reviews. However, each paper (identified by 'forum') may have multiple reviews. To understand paper-level patterns, we need to aggregate our data.
+7.2 Calculating Paper-Level Metrics
+Let's aggregate our review metrics to the paper level by calculating means:
+python
+# Aggregate reviews to paper level (mean of metrics for each paper)
+df_rev_dec_ave = df_rev_dec.groupby(['forum','decision_clean'])[['rating_int','tokens_counts','sent_count']].mean().reset_index()
+What's happening here?
+We're grouping reviews by both 'forum' (paper ID) and 'decision_clean' (accept/reject)
+For each group, we calculate the mean of 'rating_int', 'tokens_counts', and 'sent_count'
+The reset_index() turns the result back into a regular DataFrame
+The result is a paper-level dataset with average metrics for each paper
+Try it yourself: How many papers do we have in our dataset compared to reviews? What does this tell us about the review process?
+Module 8: Visualizing Token Count vs. Rating
+8.1 Creating an Advanced Visualization
+Now let's visualize the relationship between token count and rating at the paper level:
+python
+# Create a 2D histogram with token count, rating, and decision
+ax = sns.histplot(data=df_rev_dec_ave, x='tokens_counts',
+                  y='rating_int',
+                  hue='decision_clean',
+                  kde=True, 
+                  log_scale=(True,False),
+                  legend=True)
+
+# Set axis labels
+ax.set(xlabel='Review Length (# Tokens)', ylabel='Review Rating')
+
+# Move the legend outside the plot
+sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))
+
+plt.tight_layout()
+plt.show()
+8.2 Interpreting the Visualization
+This visualization reveals important patterns in our data:
+Decision Boundaries: Notice where the color changes from one decision to another
+Length-Rating Relationship: Is there a correlation between review length and rating?
+Clustering: Are there natural clusters in the data?
+Outliers: What papers received unusually long or short reviews?
+Key Insight: At the paper level, we can see if the average review length for a paper relates to its likelihood of acceptance.
+Module 9: Comparing Token Count and Sentence Count
+9.1 Visualizing Sentence Count vs. Rating
+Let's create a similar visualization using sentence count instead of token count:
+python
+# Create a 2D histogram with sentence count, rating, and decision
+ax = sns.histplot(data=df_rev_dec_ave, x='sent_count',
+                  y='rating_int',
+                  hue='decision_clean',
+                  kde=True, 
+                  log_scale=(True,False),
+                  legend=True)
+
+# Set axis labels
+ax.set(xlabel='Review Length (# Sentences)', ylabel='Review Rating')
+
+# Move the legend outside the plot
+sns.move_legend(ax, "upper left", bbox_to_anchor=(1, 1))
+
+plt.tight_layout()
+plt.show()
+9.2 Comparing Token vs. Sentence Metrics
+By comparing these two visualizations, we can understand:
+Which Metric is More Informative: Do token counts or sentence counts better differentiate accepted vs. rejected papers?
+Different Patterns: Do some papers have many short sentences while others have fewer long ones?
+Consistency: Are the patterns consistent across both metrics?
+Discussion Question: Which metric—tokens or sentences—seems to be a better predictor of paper acceptance? Why might that be?
+Module 10: Word Cloud Visualizations
+10.1 Creating a Word Cloud from Review Text
+Word clouds are a powerful way to visualize the most frequent words in a text corpus:
+python
+# Concatenate all review text
+text = ' '.join(df_rev_dec['clean_review_word'])
+
+# Generate word cloud
+wordcloud = WordCloud().generate(text)
+
+# Display word cloud
+plt.figure(figsize=(8, 6))
+plt.imshow(wordcloud, interpolation='bilinear')
+plt.axis('off')
+plt.show()
+10.2 Visualizing Paper Keywords
+Now let's visualize the primary keywords associated with the papers:
+python
+# Concatenate all primary keywords
+text = ' '.join(df_keyword['primary_keyword'])
+
+# Generate word cloud
+wordcloud = WordCloud().generate(text)
+
+# Display word cloud
+plt.figure(figsize=(8, 6))
+plt.imshow(wordcloud, interpolation='bilinear')
+plt.axis('off')
+plt.show()
+10.3 Visualizing Paper Abstracts
+Finally, let's create a word cloud from paper abstracts:
+python
+# Concatenate all abstracts
+text = ' '.join(df_submissions['abstract'])
+
+# Generate word cloud
+wordcloud = WordCloud().generate(text)
+
+# Display word cloud
+plt.figure(figsize=(8, 6))
+plt.imshow(wordcloud, interpolation='bilinear')
+plt.axis('off')
+plt.show()
+Interpreting Word Clouds
+Word clouds provide insights about:
+Dominant Themes: The most frequent words appear largest
+Vocabulary Differences: Compare terms across different sources (reviews vs. abstracts)
+Field-Specific Terminology: Technical terms reveal the focus of the conference
+Sentiment Indicators: Evaluative words in reviews reveal assessment patterns
+Try it yourself: What differences do you notice between the word clouds from reviews, keywords, and abstracts? What do these differences tell you about academic communication?
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+V1.1 Week 4 - Intro to NLP 
+Course Overview
+In this course, you'll learn fundamental Natural Language Processing (NLP) concepts by exploring a fascinating real-world question: What is the effect of releasing a preprint of a paper before it is submitted for peer review?
+Using the ICLR (International Conference on Learning Representations) database - which contains submissions, reviews, and author profiles from 2017-2022 - you'll develop practical NLP skills while investigating potential biases and patterns in academic publishing.
+Learning Path
+Understanding Text as Data: How computers represent and work with text
+Text Processing Fundamentals: Basic cleaning and normalization
+Quantitative Text Analysis: Measuring and comparing text features
+Tokenization Approaches: Breaking text into meaningful units
+Text Visualization Techniques: Creating insightful visual representations
+From Analysis to Insights: Drawing evidence-based conclusions
+Let's dive in!
+…
+Step 4: Text Cleaning and Normalization for Academic Content
+Academic papers contain specialized vocabulary, citations, equations, and other elements that require careful normalization.
+Key Concept: Scientific text normalization preserves meaningful technical content while standardizing format.
+Stop Words Removal
+Definition: Stop words are extremely common words that appear frequently in text but typically carry little meaningful information for analysis purposes. In English, these include articles (the, a, an), conjunctions (and, but, or), prepositions (in, on, at), and certain pronouns (I, you, it).
+Stop words removal is the process of filtering these words out before analysis to:
+Reduce noise in the data
+Decrease the dimensionality of the text representation
+Focus analysis on the content-bearing words
+In academic text, we often extend standard stop word lists to include domain-specific terms that are ubiquitous but not analytically useful (e.g., "paper," "method," "result").
+python
+# Load standard English stop words
+from nltk.corpus import stopwords
+standard_stop_words = set(stopwords.words('english'))
+
+# Add academic-specific stop words
+academic_stop_words = ['et', 'al', 'fig', 'table', 'paper', 'using', 'used', 
+                      'method', 'result', 'show', 'propose', 'use']
+all_stop_words = standard_stop_words.union(academic_stop_words)
+
+# Apply stop word removal
+def remove_stop_words(text):
+    words = text.split()
+    filtered_words = [word for word in words if word.lower() not in all_stop_words]
+    return ' '.join(filtered_words)
+
+# Compare before and after
+example = "We propose a novel method that shows impressive results on the benchmark dataset."
+filtered = remove_stop_words(example)
+
+print("Original:", example)
+print("After stop word removal:", filtered)
+# Output: "propose novel method shows impressive results benchmark dataset."
+Stemming and Lemmatization
+Definition: Stemming and lemmatization are text normalization techniques that reduce words to their root or base forms, allowing different inflections or derivations of the same word to be treated as equivalent.
+Stemming is a simpler, rule-based approach that works by truncating words to their stems, often by removing suffixes. For example:
+"running," "runs," and "runner" might all be reduced to "run"
+"connection," "connected," and "connecting" might all become "connect"
+Stemming is faster but can sometimes produce non-words or incorrect reductions.
+Lemmatization is a more sophisticated approach that uses vocabulary and morphological analysis to return the dictionary base form (lemma) of a word. For example:
+"better" becomes "good"
+"was" and "were" become "be"
+"studying" becomes "study"
+Lemmatization generally produces more accurate results but requires more computational resources.
+python
+from nltk.stem import PorterStemmer, WordNetLemmatizer
+import nltk
+nltk.download('wordnet')
+
+# Initialize stemmer and lemmatizer
+stemmer = PorterStemmer()
+lemmatizer = WordNetLemmatizer()
+
+# Example words
+academic_terms = ["algorithms", "computing", "learning", "trained", 
+                  "networks", "better", "studies", "analyzed"]
+
+# Compare stemming and lemmatization
+for term in academic_terms:
+    print(f"Original: {term}")
+    print(f"Stemmed: {stemmer.stem(term)}")
+    print(f"Lemmatized: {lemmatizer.lemmatize(term)}")
+    print()
+
+# Demonstration in context
+academic_sentence = "The training algorithms performed better than expected when analyzing multiple neural networks."
+
+# Apply stemming
+stemmed_words = [stemmer.stem(word) for word in academic_sentence.lower().split()]
+stemmed_sentence = ' '.join(stemmed_words)
+
+# Apply lemmatization
+lemmatized_words = [lemmatizer.lemmatize(word) for word in academic_sentence.lower().split()]
+lemmatized_sentence = ' '.join(lemmatized_words)
+
+print("Original:", academic_sentence)
+print("Stemmed:", stemmed_sentence)
+print("Lemmatized:", lemmatized_sentence)
+When to use which approach:
+For academic text analysis:
+Stemming is useful when processing speed is important and approximate matching is sufficient
+Lemmatization is preferred when precision is crucial, especially for technical terms where preserving meaning is essential
+In our ICLR paper analysis, lemmatization would likely be more appropriate since technical terminology often carries specific meanings that should be preserved accurately.
+Challenge Question: How might stemming versus lemmatization affect our analysis of technical innovation in ICLR papers? Can you think of specific machine learning terms where these approaches would yield different results?
+
+
+V1.0 Week 4 - Intro to NLP 
+The Real-World Problem
+Imagine you're part of a small business team that has just launched a new product. You've received hundreds of customer reviews across various platforms, and your manager has asked you to make sense of this feedback. Looking at the mountain of text data, you realize you need a systematic way to understand what customers are saying without reading each review individually.
+Your challenge: How can you efficiently analyze customer feedback to identify common themes, sentiments, and specific product issues?
+Our Approach
+In this module, we'll learn how to transform unstructured text feedback into structured insights using Natural Language Processing. Here's our journey:
+Understanding text as data
+Basic processing of text information
+Measuring text properties
+Cleaning and normalizing customer feedback
+Visualizing patterns in the feedback
+Analyzing words vs. tokens
+Let's begin!
+Step 1: Text as Data - A New Perspective
+When we look at customer reviews like:
+"Love this product! So easy to use and the battery lasts forever."
+"Terrible design. Buttons stopped working after two weeks."
+We naturally understand the meaning and sentiment. But how can a computer understand this?
+Key Concept: Text can be treated as data that we can analyze quantitatively.
+Unlike numerical data (age, price, temperature) that has inherent mathematical properties, text data needs to be transformed before we can analyze it.
+Interactive Exercise: Look at these two reviews. As a human, what information can you extract? Now think about how a computer might "see" this text without any processing.
+Challenge Question: What types of information might we want to extract from customer reviews? List at least three analytical goals.
+Step 2: Basic Text Processing - Breaking Down Language
+Before we can analyze text, we need to break it down into meaningful units.
+Key Concept: Tokenization is the process of splitting text into smaller pieces (tokens) such as words, phrases, or characters.
+For example, the review "Love this product!" can be tokenized into ["Love", "this", "product", "!"] or ["Love", "this", "product", "!"] depending on our approach.
+Interactive Example: Let's tokenize these customer reviews:
+python
+# Simple word tokenization
+review = "Battery life is amazing but the app crashes frequently."
+tokens = review.split()  # Results in ["Battery", "life", "is", "amazing", "but", "the", "app", "crashes", "frequently."]
+Notice how "frequently." includes the period. Basic tokenization has limitations!
+Challenge Question: How might we handle contractions like "doesn't" or hyphenated words like "user-friendly" when tokenizing?
+Step 3: Measuring Text - Quantifying Feedback
+Now that we've broken text into pieces, we can start measuring properties of our customer feedback.
+Key Concept: Text metrics help us quantify and compare text data.
+Common metrics include:
+Length (words, characters)
+Complexity (average word length, unique words ratio)
+Sentiment scores (positive/negative)
+Interactive Example: Let's calculate basic metrics for customer reviews:
+python
+# Word count
+review = "The interface is intuitive and responsive."
+word_count = len(review.split())  # 6 words
+
+# Character count (including spaces)
+char_count = len(review)  # 41 characters
+
+# Unique words ratio
+unique_words = len(set(review.lower().split()))
+unique_ratio = unique_words / word_count  # 1.0 (all words are unique)
+Challenge Question: Why might longer reviews not necessarily contain more information than shorter ones? What other metrics beyond length might better capture information content?
+Step 4: Text Cleaning and Normalization
+Customer feedback often contains inconsistencies: spelling variations, punctuation, capitalization, etc.
+Key Concept: Text normalization creates a standardized format for analysis.
+Common normalization steps:
+Converting to lowercase
+Removing punctuation
+Correcting spelling
+Removing stop words (common words like "the", "is")
+Stemming or lemmatizing (reducing words to their base form)
+Interactive Example: Let's normalize a review:
+python
+# Original review
+review = "The battery LIFE is amazing!!! Works for days."
+
+# Lowercase
+review = review.lower()  # "the battery life is amazing!!! works for days."
+
+# Remove punctuation and extra spaces
+import re
+review = re.sub(r'[^\w\s]', '', review)  # "the battery life is amazing works for days"
+
+# Remove stop words
+stop_words = ["the", "is", "for"]
+words = review.split()
+filtered_words = [word for word in words if word not in stop_words]
+# Result: ["battery", "life", "amazing", "works", "days"]
+Challenge Question: How might normalization affect sentiment analysis? Could removing punctuation or stop words change the perceived sentiment of a review?
+Step 5: Text Visualization - Seeing Patterns
+Visual representations help us identify patterns across many reviews.
+Key Concept: Text visualization techniques reveal insights that are difficult to see in raw text.
+Common visualization methods:
+Word clouds
+Frequency distributions
+Sentiment over time
+Topic clusters
+Interactive Example: Creating a simple word frequency chart:
+python
+from collections import Counter
+
+# Combined reviews
+reviews = ["Battery life is amazing", "Battery drains too quickly", 
+           "Great battery performance", "Screen is too small"]
+
+# Count word frequencies
+all_words = " ".join(reviews).lower().split()
+word_counts = Counter(all_words)
+# Result: {'battery': 3, 'life': 1, 'is': 2, 'amazing': 1, 'drains': 1, 'too': 2, 'quickly': 1, 'great': 1, 'performance': 1, 'screen': 1, 'small': 1}
+
+# We could visualize this as a bar chart
+# Most frequent: 'battery' (3), 'is' (2), 'too' (2)
+Challenge Question: Why might a word cloud be misleading for understanding customer sentiment? What additional information would make the visualization more informative?
+Step 6: Words vs. Tokens - Making Choices
+As we advance in NLP, we face an important decision: should we analyze whole words or more sophisticated tokens?
+Key Concept: Different tokenization approaches have distinct advantages and limitations.
+Word-based analysis:
+Intuitive and interpretable
+Misses connections between related words (run/running/ran)
+Struggles with compound words and new terms
+Token-based analysis:
+Can capture subword information
+Handles unknown words better
+May lose some human interpretability
+Interactive Example: Comparing approaches:
+python
+# Word-based
+review = "The touchscreen is unresponsive"
+words = review.lower().split()  # ['the', 'touchscreen', 'is', 'unresponsive']
+
+# Subword tokenization (simplified example)
+subwords = ['the', 'touch', 'screen', 'is', 'un', 'responsive']
+Challenge Question: For our customer feedback analysis, which approach would be better: analyzing whole words or subword tokens? What factors would influence this decision?
+Putting It All Together: Solving Our Problem
+Now that we've learned these fundamental NLP concepts, let's return to our original challenge: analyzing customer feedback at scale.
+Here's how we'd approach it:
+Collect and tokenize all customer reviews
+Clean and normalize the text
+Calculate key metrics (length, sentiment scores)
+Visualize common terms and topics
+Identify positive and negative feedback themes
+Generate an automated summary for the product team
+By applying these NLP fundamentals, we've transformed an overwhelming mass of text into actionable insights that can drive product improvements!
+Final Challenge: How could we extend this analysis to track customer sentiment over time as we release product updates? What additional NLP techniques might be helpful?
+

app/main.py

@@ -16,7 +16,7 @@ from app.components.login import login
 from app.pages import week_1
 from app.pages import week_2
 from app.pages import week_3
-
+from app.pages import week_4
 # Page configuration
 st.set_page_config(
     page_title="Data Science Course App",
@@ -139,6 +139,8 @@ def show_week_content():
         week_2.show()
     elif st.session_state.current_week == 3:
         week_3.show()
+    elif st.session_state.current_week == 4:
+        week_4.show()
     else:
         st.warning("Content for this week is not yet available.")
 
@@ -151,7 +153,7 @@ def main():
         return
     
     # User is logged in, show course content
-    if st.session_state.current_week in [1, 2, 3]:
+    if st.session_state.current_week in [1, 2, 3, 4]:
         show_week_content()
     else:
         st.title("Data Science Research Paper Course")

app/pages/week_4.py

@@ -0,0 +1,217 @@
+import streamlit as st
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+import seaborn as sns
+import nltk
+from nltk.corpus import stopwords
+from nltk.tokenize import word_tokenize, sent_tokenize
+nltk.download('punkt_tab')
+nltk.download('stopwords')
+from nltk.stem import PorterStemmer, WordNetLemmatizer
+from wordcloud import WordCloud
+import string
+import io
+from contextlib import redirect_stdout
+
+# Initialize session state for notebook-like cells
+if 'cells' not in st.session_state:
+    st.session_state.cells = []
+if 'df' not in st.session_state:
+    st.session_state.df = None
+
+def capture_output(code, df=None):
+    """Helper function to capture print output"""
+    f = io.StringIO()
+    with redirect_stdout(f):
+        try:
+            # Create a dictionary of variables to use in exec
+            variables = {'pd': pd, 'np': np, 'plt': plt, 'sns': sns, 'nltk': nltk}
+            if df is not None:
+                variables['df'] = df
+            exec(code, variables)
+        except Exception as e:
+            return f"Error: {str(e)}"
+    return f.getvalue()
+
+def show():
+    st.title("Week 4: Introduction to Natural Language Processing")
+    
+    # Introduction Section
+    st.header("Course Overview")
+    st.write("""
+    In this course, you'll learn fundamental Natural Language Processing (NLP) concepts by exploring a fascinating real-world question: 
+    What is the effect of releasing a preprint of a paper before it is submitted for peer review?
+    
+    Using the ICLR (International Conference on Learning Representations) database - which contains submissions, reviews, and author profiles 
+    from 2017-2022 - you'll develop practical NLP skills while investigating potential biases and patterns in academic publishing.
+    """)
+    
+    # Learning Path
+    st.subheader("Learning Path")
+    st.write("""
+    1. Understanding Text as Data: How computers represent and work with text
+    2. Text Processing Fundamentals: Basic cleaning and normalization
+    3. Quantitative Text Analysis: Measuring and comparing text features
+    4. Tokenization Approaches: Breaking text into meaningful units
+    5. Text Visualization Techniques: Creating insightful visual representations
+    6. From Analysis to Insights: Drawing evidence-based conclusions
+    """)
+
+    # Module 1: Text as Data
+    st.header("Module 1: Text as Data")
+    st.write("""
+    When we look at text like customer reviews or academic papers, we naturally understand the meaning. 
+    But how can a computer understand this?
+    
+    Key Concept: Text can be treated as data that we can analyze quantitatively.
+    Unlike numerical data (age, price, temperature) that has inherent mathematical properties, 
+    text data needs to be transformed before we can analyze it.
+    """)
+
+    # Interactive Example
+    st.subheader("Interactive Example: Text Tokenization")
+    st.write("Let's try tokenizing some text:")
+    
+    example_text = st.text_area(
+        "Enter some text to tokenize:",
+        "The quick brown fox jumps over the lazy dog."
+    )
+    
+    if st.button("Tokenize Text"):
+        tokens = word_tokenize(example_text)
+        st.write("Tokens:", tokens)
+        st.write("Number of tokens:", len(tokens))
+
+    # Module 2: Text Processing
+    st.header("Module 2: Text Processing")
+    st.write("""
+    Before we can analyze text, we need to clean and normalize it. This includes:
+    - Converting to lowercase
+    - Removing punctuation
+    - Removing stop words
+    - Stemming or lemmatization
+    """)
+
+    # Interactive Text Processing
+    st.subheader("Try Text Processing")
+    st.write("""
+    Let's process some text using different techniques:
+    """)
+    
+    process_text = st.text_area(
+        "Enter text to process:",
+        "The quick brown fox jumps over the lazy dog.",
+        key="process_text"
+    )
+    
+    col1, col2 = st.columns(2)
+    
+    with col1:
+        if st.button("Remove Stop Words"):
+            stop_words = set(stopwords.words('english'))
+            words = word_tokenize(process_text.lower())
+            filtered_words = [word for word in words if word not in stop_words]
+            st.write("After removing stop words:", filtered_words)
+    
+    with col2:
+        if st.button("Remove Punctuation"):
+            no_punct = process_text.translate(str.maketrans('', '', string.punctuation))
+            st.write("After removing punctuation:", no_punct)
+
+    # Module 3: Text Visualization
+    st.header("Module 3: Text Visualization")
+    st.write("""
+    Visual representations help us identify patterns across text data.
+    Common visualization methods include:
+    - Word clouds
+    - Frequency distributions
+    - Sentiment over time
+    - Topic clusters
+    """)
+
+    # Interactive Word Cloud
+    st.subheader("Create a Word Cloud")
+    st.write("""
+    Let's create a word cloud from some text:
+    """)
+    
+    wordcloud_text = st.text_area(
+        "Enter text for word cloud:",
+        "The quick brown fox jumps over the lazy dog. The fox is quick and brown. The dog is lazy.",
+        key="wordcloud_text"
+    )
+    
+    if st.button("Generate Word Cloud"):
+        # Create and generate a word cloud image
+        wordcloud = WordCloud().generate(wordcloud_text)
+        
+        # Display the word cloud
+        plt.figure(figsize=(10, 6))
+        plt.imshow(wordcloud, interpolation='bilinear')
+        plt.axis('off')
+        st.pyplot(plt)
+
+    # Practice Exercises
+    st.header("Practice Exercises")
+    
+    with st.expander("Exercise 1: Text Processing"):
+        st.write("""
+        1. Load a sample text
+        2. Remove stop words and punctuation
+        3. Create a word cloud
+        4. Analyze word frequencies
+        """)
+        
+        st.code("""
+        # Solution
+        import nltk
+        from nltk.corpus import stopwords
+        from wordcloud import WordCloud
+        import string
+        
+        # Sample text
+        text = "Your text here"
+        
+        # Remove punctuation
+        text = text.translate(str.maketrans('', '', string.punctuation))
+        
+        # Remove stop words
+        stop_words = set(stopwords.words('english'))
+        words = text.split()
+        filtered_words = [word for word in words if word.lower() not in stop_words]
+        
+        # Create word cloud
+        wordcloud = WordCloud().generate(' '.join(filtered_words))
+        plt.imshow(wordcloud)
+        plt.axis('off')
+        plt.show()
+        """)
+    
+    with st.expander("Exercise 2: Text Analysis"):
+        st.write("""
+        1. Calculate basic text metrics (word count, unique words)
+        2. Perform stemming and lemmatization
+        3. Compare the results
+        4. Visualize the differences
+        """)
+        
+        st.code("""
+        # Solution
+        from nltk.stem import PorterStemmer, WordNetLemmatizer
+        
+        # Initialize stemmer and lemmatizer
+        stemmer = PorterStemmer()
+        lemmatizer = WordNetLemmatizer()
+        
+        # Sample words
+        words = ["running", "runs", "ran", "better", "good"]
+        
+        # Apply stemming and lemmatization
+        stemmed = [stemmer.stem(word) for word in words]
+        lemmatized = [lemmatizer.lemmatize(word) for word in words]
+        
+        # Compare results
+        for word, stem, lemma in zip(words, stemmed, lemmatized):
+            print(f"Original: {word}, Stemmed: {stem}, Lemmatized: {lemma}")
+        """)

requirements.txt

@@ -4,4 +4,6 @@ numpy==1.26.4
 scikit-learn==1.4.0
 matplotlib==3.8.3
 seaborn==0.13.2
-plotly==5.18.0 
+plotly==5.18.0
+nltk==3.8.1
+wordcloud==1.9.3