1: Data

Material based on Chapter 1 Introduction to Modern Statistics

Professor Elizabeth Stanny

Learning Objectives

• Understanding what data looks like
• Learning to ask good questions about data
• Exploring relationships between variables
• Setting up our R environment with tidyverse

Instructor: Welcome everyone! Today we’re starting our journey into statistics and data science. Don’t worry if you’re new to programming - we’ll take it step by step. The goal today is to get comfortable with what data looks like and how we can start asking interesting questions about it.

Remember, statistics is really about using data to answer questions about the world around us. Let’s dive in!

Setting Up Our Tools

# Load the tidyverse package
library(tidyverse)
library(scales)

# Set a clean theme for plots
theme_set(theme_minimal())

Tip

The tidyverse is a collection of R packages designed for data science!

Instructor: First, let’s load our main tool - the tidyverse. Think of it as your Swiss Army knife for data analysis. It includes packages for reading data, manipulating it, and creating visualizations.

Don’t worry about memorizing everything - we’ll practice these commands many times throughout the course.

Case Study: Can Stents Prevent Strokes?

The Big Question: Do stents reduce the risk of stroke?

The Experiment:

• 451 at-risk patients
• Randomly assigned to two groups:
  • Treatment: Stent + medical care
  • Control: Medical care only

# Create the results data
stent_data <- tibble(
  group = c("Control", "Treatment"),
  patients = c(227, 224),
  strokes_365_days = c(28, 45)
)

stent_data

# A tibble: 2 × 3
  group     patients strokes_365_days
  <chr>        <dbl>            <dbl>
1 Control        227               28
2 Treatment      224               45

Instructor: - Let’s start with a real medical study. - Researchers wanted to know if stents - tiny mesh tubes placed in arteries - could prevent strokes. - This is a perfect example of how we use data to answer important questions.

Notice how they set up two groups - one gets the treatment, one doesn’t. This comparison is key to understanding if the treatment works.

Calculating Proportions

# Calculate stroke rates for each group
results <- stent_data |>
  mutate(
    stroke_rate = strokes_365_days / patients,
    percentage = round(stroke_rate * 100, 1)
  )

results

# A tibble: 2 × 5
  group     patients strokes_365_days stroke_rate percentage
  <chr>        <dbl>            <dbl>       <dbl>      <dbl>
1 Control        227               28       0.123       12.3
2 Treatment      224               45       0.201       20.1

Surprising finding: The treatment group had MORE strokes! (20% vs 12.3%)

Instructor: - Here’s where data analysis gets interesting! - The doctors expected stents to REDUCE strokes, but the data shows the opposite. - The treatment group had an 8% HIGHER stroke rate.

• This is why we do experiments - sometimes our intuitions are wrong!
• This study actually led to changes in medical practice.

What is Data?

Data are observations collected from a study or experiment.

# Example: Let's create a small dataset
students <- tibble(
  name = c("Alice", "Bob", "Charlie", "Diana"),
  height_cm = c(165, 178, 172, 168),
  siblings = c(0, 2, 1, 3),
  stats_before = c("No", "Yes", "No", "No")
)

students

# A tibble: 4 × 4
  name    height_cm siblings stats_before
  <chr>       <dbl>    <dbl> <chr>       
1 Alice         165        0 No          
2 Bob           178        2 Yes         
3 Charlie       172        1 No          
4 Diana         168        3 No          

Note

Each row is an observation (also called a case)
Each column is a variable

Instructor:

• Data is simply information we collect about things we’re interested in.

• Here’s a simple example with student data.

• Notice the structure: each student gets their own row, and each thing we measure about them gets its own column.

• This is called “tidy data” and it’s the format we’ll always aim for.

Types of Variables

Instructor: - Variables come in different flavors. - Numerical variables are things we can do math with - like calculating an average height. - Categorical variables are groups or categories.

• The distinction matters because different types of variables need different analysis methods.
• You can’t take the average of eye colors, right?

Load Data

# Load a dataset about loans
loans <- read_csv("https://www.openintro.org/data/csv/loan50.csv")

# Look at the first few values for each variable 
glimpse(loans)

Rows: 50
Columns: 18
$ state                   <chr> "NJ", "CA", "SC", "CA", "OH", "IN", "NY", "MO"…
$ emp_length              <dbl> 3, 10, NA, 0, 4, 6, 2, 10, 6, 3, 8, 10, 10, 2,…
$ term                    <dbl> 60, 36, 36, 36, 60, 36, 36, 36, 60, 60, 36, 36…
$ homeownership           <chr> "rent", "rent", "mortgage", "rent", "mortgage"…
$ annual_income           <dbl> 59000, 60000, 75000, 75000, 254000, 67000, 288…
$ verified_income         <chr> "Not Verified", "Not Verified", "Verified", "N…
$ debt_to_income          <dbl> 0.55752542, 1.30568333, 1.05628000, 0.57434667…
$ total_credit_limit      <dbl> 95131, 51929, 301373, 59890, 422619, 349825, 1…
$ total_credit_utilized   <dbl> 32894, 78341, 79221, 43076, 60490, 72162, 2872…
$ num_cc_carrying_balance <dbl> 8, 2, 14, 10, 2, 4, 1, 3, 10, 4, 3, 4, 3, 2, 3…
$ loan_purpose            <chr> "debt_consolidation", "credit_card", "debt_con…
$ loan_amount             <dbl> 22000, 6000, 25000, 6000, 25000, 6400, 3000, 1…
$ grade                   <chr> "B", "B", "E", "B", "B", "B", "D", "A", "A", "…
$ interest_rate           <dbl> 10.90, 9.92, 26.30, 9.92, 9.43, 9.92, 17.09, 6…
$ public_record_bankrupt  <dbl> 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0…
$ loan_status             <chr> "Current", "Current", "Current", "Current", "C…
$ has_second_income       <lgl> FALSE, FALSE, FALSE, FALSE, FALSE, FALSE, FALS…
$ total_income            <dbl> 59000, 60000, 75000, 75000, 254000, 67000, 288…

Instructor: - This dataset contains information about 50 loans. - The glimpse() function gives us a quick overview - we can see the variable names, types, and some example values.

Notice how R automatically identifies which variables are numbers and which are text (characters).

Exploring Our Data

# Basic summary statistics
loans |>
  summarize(
    avg_loan = mean(loan_amount),
    avg_interest = mean(interest_rate),
    min_loan = min(loan_amount),
    max_loan = max(loan_amount)
  )

# A tibble: 1 × 4
  avg_loan avg_interest min_loan max_loan
     <dbl>        <dbl>    <dbl>    <dbl>
1    17083         11.6     3000    40000

Tip

The |> symbol is called a “pipe” - it passes data from one step to the next!

Instructor: The pipe operator |> is like saying “then”. Take the loans data, THEN summarize it. This makes our code readable - it follows the logical flow of our analysis.

Look at these summary statistics - the average loan is about $17,000 with interest rates averaging 11.6%. - But notice the range - loans go from $3,000 to $40,000!

Relationships Between Variables

Key Question: Do variables relate to each other?

Instructor: Here’s where data gets really interesting - looking at relationships! - This scatterplot shows each loan as a dot. The red line shows the general trend.

• What do you notice? There’s a slight upward trend - people with higher incomes tend to borrow a bit more, but it’s not a strong relationship. See all that scatter around the line?

Association vs. Independence

Associated Variables

• Show a pattern or relationship
• Knowing one helps predict the other
• Can be positive or negative

Independent Variables

• No clear pattern
• Knowing one doesn’t help predict the other

Clear association: Lower grades (riskier loans) have higher interest rates!

Instructor: This boxplot shows a clear relationship - as loan grades get worse (from A to G), interest rates go up. - This makes sense - riskier loans charge higher interest.

This is what we mean by association. If I tell you a loan has grade E, you can make a good guess about its interest rate.

Explanatory vs. Response Variables

When we think one variable influences another:

• Explanatory variable (X): The potential cause
• Response variable (Y): The potential effect

# Example: Does loan term affect interest rate?
loans |>
  group_by(term) |>
  summarize(
    avg_interest = mean(interest_rate),
    n_loans = n()
  )

# A tibble: 2 × 3
   term avg_interest n_loans
  <dbl>        <dbl>   <int>
1    36         10.8      36
2    60         13.6      14

Here, term is explanatory, interest_rate is response

Instructor: - Sometimes we’re not just interested in whether variables are related, but whether one CAUSES changes in the other.

• Look at this example - 60-month loans have higher interest rates than 36-month loans.
• The term length might EXPLAIN the interest rate difference.
• But remember - just because variables are related doesn’t prove causation!

Observational Studies vs. Experiments

Observational Study

• Observe without interfering
• Can show associations
• Cannot prove causation
• Example: Survey existing loan data

Experiment

• Actively assign treatments
• Random assignment is key
• CAN establish causation
• Example: The stent study

Remember This!

Association ≠ Causation

Only well-designed experiments can establish causal relationships!

Instructor: This is a crucial distinction! Our loans data is observational - we’re just observing existing loans. We can see patterns but can’t prove what causes what.

The stent study was an experiment - they randomly assigned treatments. That’s why they could conclude that stents CAUSED more strokes. Random assignment is the secret sauce for proving causation.

Your Turn: Practice with Data!

# Load a fun dataset about penguins
penguins <- read_csv("https://raw.githubusercontent.com/allisonhorst/palmerpenguins/master/inst/extdata/penguins.csv")

# Your task: Explore this data!
glimpse(penguins)

Rows: 344
Columns: 8
$ species           <chr> "Adelie", "Adelie", "Adelie", "Adelie", "Adelie", "A…
$ island            <chr> "Torgersen", "Torgersen", "Torgersen", "Torgersen", …
$ bill_length_mm    <dbl> 39.1, 39.5, 40.3, NA, 36.7, 39.3, 38.9, 39.2, 34.1, …
$ bill_depth_mm     <dbl> 18.7, 17.4, 18.0, NA, 19.3, 20.6, 17.8, 19.6, 18.1, …
$ flipper_length_mm <dbl> 181, 186, 195, NA, 193, 190, 181, 195, 193, 190, 186…
$ body_mass_g       <dbl> 3750, 3800, 3250, NA, 3450, 3650, 3625, 4675, 3475, …
$ sex               <chr> "male", "female", "female", NA, "female", "male", "f…
$ year              <dbl> 2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007, 2007…

Questions

1. How many penguins are in the dataset?
2. What variables do we have?
3. Which are numerical? Which are categorical?

Instructor: Now it’s your turn! Let’s explore this penguin dataset together. Take a moment to run this code and think about these questions.

[Give students 2-3 minutes to explore]

• Who can tell me how many observations we have?
• What about the variables - which ones are measurements and which are categories?

Creating Your First Visualization

# Remove missing values and create a plot
penguins |>
  drop_na(flipper_length_mm, body_mass_g) |>
  ggplot(aes(x = flipper_length_mm, y = body_mass_g, 
             color = species)) +
  geom_point(size = 3, alpha = 0.7) +
  labs(title = "Penguin Size by Species",
       x = "Flipper Length (mm)",
       y = "Body Mass (g)",
       color = "Species") 

What do you notice about the relationship?

Instructor: Look at this beautiful plot! Each dot is a penguin, colored by species.

• What patterns do you see?
• There’s a clear positive association - bigger flippers, heavier penguin.
• But also notice how the species separate into clusters. Gentoo penguins are clearly the largest!

Summary Statistics by Group

# Calculate summary statistics by species
penguins |>
  drop_na(body_mass_g) |>
  group_by(species) |>
  summarize(
    count = n(),
    avg_mass = mean(body_mass_g),
    sd_mass = sd(body_mass_g),
    min_mass = min(body_mass_g),
    max_mass = max(body_mass_g)
  ) |>
  arrange(desc(avg_mass))

# A tibble: 3 × 6
  species   count avg_mass sd_mass min_mass max_mass
  <chr>     <int>    <dbl>   <dbl>    <dbl>    <dbl>
1 Gentoo      123    5076.    504.     3950     6300
2 Chinstrap    68    3733.    384.     2700     4800
3 Adelie      151    3701.    459.     2850     4775

Note

group_by() lets us calculate statistics for each group separately!

Instructor: - Here’s the power of tidyverse - we can easily calculate statistics by group. - Look at the average mass - Gentoo penguins are about 5kg, while Adelie and Chinstrap are around 3.7kg.

• The group_by() function is incredibly powerful - it’s like saying “do this analysis separately for each species.”

Key Concepts to Remember

1. Data Structure: Rows are observations, columns are variables
2. Variable Types: Numerical (continuous/discrete) vs. Categorical (nominal/ordinal)
3. Relationships: Variables can be associated or independent
4. Causation: Only experiments with random assignment can prove causation
5. R/Tidyverse: Our toolkit for exploring and visualizing data

Instructor: Let’s recap the big ideas from today. These five concepts are the foundation for everything we’ll do this semester.

Most importantly, remember that statistics is about using data to answer questions. Every dataset tells a story - our job is to uncover it!

Practice Problems

Try these with the penguins data:

1. Create a plot showing bill length vs. bill depth
2. Calculate the average flipper length for each species
3. Make a boxplot of body mass by island
4. Find which species has the most variation in bill length

# Starter code for problem 1
penguins |>
  ggplot(aes(x = ___, y = ___)) +
  geom_point()

Instructor: Here are some practice problems to solidify today’s concepts. Work on these with a partner if you’d like. I’ll walk around to help.

[Give students 5-10 minutes to work]

Remember - making mistakes is part of learning! Try different things and see what happens.

Solutions to Practice

# Problem 1: Bill length vs depth
penguins |>
  drop_na(bill_length_mm, bill_depth_mm) |>
  ggplot(aes(x = bill_length_mm, y = bill_depth_mm, 
             color = species)) +
  geom_point(alpha = 0.6) +
  labs(title = "Bill Dimensions by Species")

# Problem 2: Average flipper length
penguins |>
  drop_na(flipper_length_mm) |>
  group_by(species) |>
  summarize(avg_flipper = mean(flipper_length_mm))

# A tibble: 3 × 2
  species   avg_flipper
  <chr>           <dbl>
1 Adelie           190.
2 Chinstrap        196.
3 Gentoo           217.

Instructor: Great work! Notice in the bill plot how the species form distinct clusters? This is called Simpson’s Paradox - the overall relationship looks negative, but within each species it’s positive!

This is why visualization is so important - patterns aren’t always what they seem at first glance.

Resources

• R for Data Science: Free online book
• Tidyverse documentation
• OpenIntro Statistics
• Office hours: [Your office hours here]

Remember

Learning to code is like learning a new language - it takes practice! Be patient with yourself and don’t hesitate to ask for help.

Instructor: Here are some great resources for extra practice and help. Remember, everyone learns at their own pace. The key is consistent practice.

I’m here to help - please come to office hours if you’re stuck or just want to chat about data analytics

Thanks for a great first class! See you next time!