---
title: "Packages, Functions, and the Tidyverse"
author: "Rian DeFaccio, for BH Analytics LLC"
date: "`r Sys.Date()`"
output: 
  html_document:
    toc: true
    toc_depth: 3
---

```{r setup, include=FALSE}
knitr::opts_knit$set(warning = FALSE, message = FALSE)
knitr::opts_chunk$set(warning = FALSE, message = FALSE)
```

```{r}
# load libraries and adjust settings
library(dplyr)
```

## Introduction

------------------------------------------------------------------------

In this section we will discuss R packages, get an introduction to functions, learn about control flow, and get acquainted with the most popular collection of packages for R -- the Tidyverse.

## What are R Packages?

------------------------------------------------------------------------

Base R is great, but to add additional functionality we can install and load additional R packages. These packages can contain data and custom functions for doing specialized tasks.

To install a new package to your computer, use the command `install.packages("package_name")`. You only have to do this once for any given package.

Once a package has been installed, you can load it into your current R session by using the `library()` function.

In the next example we will load the Hmisc package and use it to look at some data. We will learn more about the functions and data structures used here later.

```{r}
library(Hmisc)

# let's examine the R built-in dataset cars
describe(cars)
```

Most of the more popular packages also have vignettes, which are tutorials that describe how to use the package. To access the list of vignette's from a package use the `vignette()` function.

```{r eval=FALSE}
vignette(package = "dplyr")
```

The previous command should have opened a new window with a list of all the vignettes for the given package. To view a vignette, add the topic in the call to `vignette()`.

```{r eval=F}
vignette(package = "dplyr", topic = "grouping")
```

The vignette should have opened in the RStudio help pane.

An alternative method for viewing vignettes is to search the internet :)

There is one more consideration with loading packages. Lets say you just wanted a single function from a particular package and did not want to load the entire package. In this case, you can access this function with the following syntax: `package_name::function()`. We use two colons to access functions from a package without loading the entire package into your R session.

In the next example we use the `select()` function from the dplyr package, but without loading the entire dplyr package into memory.

```{r eval=FALSE}
## this won't work!
select(mtcars, mpg, cyl)
```

```{r}
## this will work
dplyr::select(mtcars, mpg, cyl)
```

There is one other reason to use this syntax when calling a function.

What happens when you load two packages, but both of them have a function called `select()`???????

The `select()` function that gets loaded into our environment is the one from the package that was loaded most recently. We have *masked* the `select()`function that was previously loaded.

### 🔍 Explore! R Packages on CRAN

Go the the Comprehensive R Archive Network (CRAN) Task Views Page and find a couple of R packages that could be relevant to your work. Share your findings in the ☞ **Zoom Chat**.

🌐 <https://cran.r-project.org/web/views/>

## R is Functional

------------------------------------------------------------------------

Let's now take a closer look at the object that is most commonly loaded from any R package: functions!

To begin, we can pose the following question:

> What kind of language is R?

Primarily we can think of it as a **functional** language. We are probably most familiar with functions that look like the following.

```{r}
mean(1:10)
```

However, lets consider the following example, some basic arithmetic.

```{r}
3 + 4
```

Without even realizing it, we just used a function!

We can use the addition function in this next way, which may look more like the way we are used to using functions.

```{r}
`+`(3,4)
```

Next is another function that we may not always think of as being a function. The following is the extraction function.

```{r}
mtcars[1, 1]
```

Which could also be rewritten as follows.

```{r}
`[`(mtcars, 1, 1)
```

## Components of a Function

There are 3 components to a function:

-   `formals()`: more commonly known as *arguments* or *parameters*
-   `body()`: the R code that makes up the function
-   `environment()`: which environment does the function belong to?

Let's consider a simple function that adds `1` to a number.

Observe the syntax below to see how we define a function.

```{r}
myfunc <- function(x=10){
  output <- (x + 1) * 10
  return(output)
}
```

```{r}
formals(myfunc)
```

```{r}
body(myfunc)
```

```{r}
environment(myfunc)
```

And here is how we use our new function!

```{r}
# testing out my function
myfunc(5)
myfunc(99)
myfunc(-99)
myfunc(Inf)
myfunc(3/7)
myfunc(data.frame(x=1:10))
myfunc(1:10)
```

### 🛠️Coding Practice: Your First Function

Write a function to calculate body mass index (bmi) from weight and height:

-   Use the formula: `bmi = weight (kg) / height^2 (m^2)`
-   Give `height` a default value of 1.75m
-   Name your function `bmi()`
-   Finally, test your function for someone with a weight of 87 kg and 1.9 m

```{r}
# define your function here


# test it out here

```

**END OF EXERCISE**

------------------------------------------------------------------------

You can create objects inside a function and then recombine them for your final result. Consider the next example:

```{r, echo=TRUE}
fMeanSd <- function(x) { 
  # check that x should be a vector of numbers
  print("calculating the mean...")
  mean1  <- mean(x) 
  print("calculating the sd...")
  sd1    <- sd(x) 
  # make a named vector to hold the output
  output <- round(c("mean" = mean1, "sd" = sd1), 2)
  print("This is a Side Effect")
  return(output)
} 
```

```{r}
x <- fMeanSd(1:10)
x
fMeanSd(c('the','the','the'))
```

### ✍️ Your Turn! Why Make Functions?

With so many existing functions available, why should you create your own?

Put some answers in the ☞ **Zoom Chat**

**Also, record your thoughts here:**

...

...

...

...

### 🛠️Coding Practice: More Complex Functions

Write a function named `fMyStats()` that takes a vector of numbers and returns a named vector of values including:

-   the mean
-   the number of observations (use the `length()` function)
-   the standard deviation
-   the median
-   the minimum
-   the maximum

```{r}
# build your function here


# and test it on the `hp` column from the `mtcars` data set


```

Now, alter your previous function so that it outputs a data frame instead of a named vector. Your should have a column for each statistic, and a single row.

```{r}
# build your function here


# and test it on the `hp` column from the `mtcars` data set


```

**END OF EXERCISE**

## Control Flow in R

------------------------------------------------------------------------

Let's begin by considering two ways that you can control how a program runs. Essentially we can control the order in which commands get executed other than just from one line to the next line.

### If-Else Statements

Let's begin by considering an if-else statement. A couple notes on the syntax:

-   The **condition** goes inside of parenthesis. This must evaluate to `TRUE` or `False`.
-   The **R expression** that then gets evaluated if the condition was true goes inside of curly brackets.
-   There can be multiple commands/lines for each R expression.

```{r}
x = 10

# simple if-else statement
if (x <= 10) {
  print(paste0("You gave ", x, ", a small number!"))
} else {
  print(paste0("You gave ", x, ", a large number!"))
}
```

We can extend the if-else statement to consider as many different conditions as we'd like by stringing together more `else` conditions. The final `else` doesn't have a condition at all and is the catch-all for if none of the previous conditions evaluated to `TRUE`.

💡**Main Idea:** There is only one R expression that is evaluated in an if-else statement.

```{r}
x = -10

# longer statement
if (x <= 10 & x > 0) {
  print(paste0("You gave ", x, ", a small number!"))
} else if (x < 0){
  print(paste0("You gave ", x, ", a negative number!"))
} else {
  print(paste0("You gave ", x, ", a large number!"))
}
```

### 🛠️Coding Practice: A Simple Function

Take the previous set of if/else/else statements and turn it into a function:

-   Call this function `tell_me_about_my_number()`.
-   It should take one argument: `X`, which should be a number.
-   It should print the appropriate response based on the number that was input. This function doesn't have to return anything.
-   **BONUS:** Add an argument called `exl=` that specifies whether or not there should be explanation points at the end of each sentence. It's default value should be `TRUE`. If `FALSE` then each sentence should end in a period instead of an exclamation mark.

```{r}
# BUILD YOUR FUNCTION HERE


# TEST YOUR FUNCTION HERE
# tell_me_about_my_number(-50)
# tell_me_about_my_number(500)
# tell_me_about_my_number(-50, exl = FALSE)
# tell_me_about_my_number(50, exl = FALSE)
```

**END OF EXERCISE**

------------------------------------------------------------------------

One more pattern we want to be familiar with is the for loop.

```{r}
for (i in 1:5) {
  i_squared <- i^2
  print(paste(i, "squared equals", i_squared))
}
```

For the next for loop to make sense, we need to see a new way of extracting columns from data frames.

-   We use the double square brackets to extract a column name by string reference or integer location.
-   This extracts the column as a vector.

```{r}
patient[['State']][1:10]
```

You can also use the `[[]]` to create new columns.

```{r}
patient[['dummy_var']] <- 1
head(patient)
```

Now let's see how we can use a for loop to create several new factor variables in the patient data frame.

```{r}
factor_vars <- c('State', 'Gender', 'Race')

for (col_ in factor_vars) {
  new_col_name <- paste0(col_, "_fct")
  patient[[new_col_name]] <- factor(patient[[col_]])
  print(paste0("Just created [", new_col_name, "] from [", col_, "]."))
}
```

### 🛠️Coding Practice: For Loop Practice

Build a for loop that takes all the character columns in the patient data frame and converts them to uppercase. You can have your loop just write over the previous versions of the columns.

**BONUS:** Also have your code determine which columns need to converted. The `typeof()` function might be useful here. You might want to use and if-else statement inside of your for loop.

```{r}
# ANSWER HERE

```

**END OF EXERCISE**

------------------------------------------------------------------------

## Introduction to the Tidyverse

------------------------------------------------------------------------

Now that we understand what R packages are, what functions are, and how to use them both, let's consider the single most popular collection of R packages out there...

The tidyverse is a collection of R packages designed to make data science and statistical analysis in R more accessible, efficient, and consistent. It was developed by Hadley Wickham and his collaborators. The tidyverse packages share a common philosophy and syntax, making it easier for users to learn and use them together.

Key packages within the tidyverse include:

-   **ggplot2:** A powerful and flexible package for creating static graphics.
-   **dplyr:** A package for data manipulation, providing functions for filtering, selecting, arranging, and summarizing data.
-   **tidyr:** Focuses on tools for tidying data, including functions for reshaping and cleaning data.
-   **readr:** Provides functions for reading rectangular data (like CSV files) into R.
-   **purrr:** Offers a consistent set of tools for working with functions and vectors.
-   **tibble:** Provides a modern version of data frames with additional features.
-   **stringr:** Contains functions for working with strings.
-   **forcats:** Focuses on working with factors, including tools for manipulating and transforming factor levels.

By using the tidyverse packages together, users can create a consistent and efficient workflow for data analysis in R. The philosophy behind the tidyverse is centered on the principles of tidy data, which emphasizes organizing data in a way that makes it easy to work with and analyze. The tidyverse is widely used in the R community and has become a popular choice for data scientists and statisticians working with R.

## The Primary Data Verbs: dplyr

------------------------------------------------------------------------

We will begin our journey through the Tidyverse one package that does much of the heavy-lifting in terms of manipulating our data -- dplyr.

The dplyr package is built around just a few primary functions that conceptually map as data "verbs". Each function can be thought of as a fundamental data-management task. These functions are scoped locally to the data frame, so we can often just type in the variable names directly into the arguments.

-   **filter** - subset by row conditions
-   **select** - subset by column names
-   **mutate** - create new variable, add to data frame, and return data frame
-   **summarize** - summarize data frame and return the summary
-   **arrange** - sort the data frame

Let's see a few of these verbs in action! We can directly reference these functions because we loaded the dplyr package at the beginning of the notebook.

```{r}
# let's use the mtcars data set
head(mtcars)
```

```{r}
head(filter(mtcars, mpg < 15))

# multiple filters??? use & and |
# filter(mtcars, mpg < 15 & cyl == 8)
```

```{r}
head(select(mtcars, mpg, hp, disp, am))

# also try the following:
head(select(mtcars, mpg:hp))
```

```{r}
head(
  mutate(mtcars, 
         mpg,
         kilometers_per_liter = mpg * 2.35215,
         .keep = 'none')
)
```

```{r}
head(
  mutate(mtcars,
         kilometers_per_liter = mpg * 2.35215)
)
```

```{r}
summarise(mtcars,
          mean_mpg   = mean(mpg),
          median_mpg = median(mpg),
          std_mpg    = sd(mpg),
          mean_disp  = mean(disp))
```

### ✍️ Your Turn! Looking at dplyr Pipes

Did you observe a trend in the output of each primary dplyr function? What type of data structure was each output?

**Your Answer Here:**

Did you observe a trend in the arguments to each function? What was the first argument in each of the above examples?

**Your Answer Here:**

### 🛠️Coding Practice: Practicing dplyr Verbs

Use the appropriate dplyr verb to accomplish each of the following tasks below and print the results:

-   Sort the mtcars data by number of cylinders and then also by miles per gallon.

```{r}
# ANSWER HERE

```

-   Get the average number of cylinders (`cyl`) across all the cars, as well as the average horsepower (`hp`) and displacement (`disp`).

```{r}
# ANSWER HERE

```

-   Keep only cars that had `mpg` greater than 20 and `cyl` less than or equal to 6.

```{r}
# ANSWER HERE

```

-   Create a new column that represents the value of miles-per-gallons-squared and return this new column along with all the original columns.

```{r}
# ANSWER HERE

```

**END OF EXERCISE**

### Remove a Variable

To remove a variable, we use the following syntax with the `select()` function.

```{r}
select(mtcars, -hp, -mpg) 
```

------------------------------------------------------------------------

## The Pipe Operator

------------------------------------------------------------------------

We generally pass the data frame into the functions using a chaining operator. All this operator does is take the output of the prior expression, then pass it into the next function as the first argument. There are now two chaining operators we can choose from:

-   `%>%` : this is the original chain operator, from the `magrittr` package, and is part of the tidyverse collection of packages
-   `|>` : this is a new chain operator implemented in base R, a recent addition to the base installation
-   You can use either chain operator, but we will generally use `%>%` in this class.

First consider the following example of using a pipe on a string.

```{r}
# first notice how paste() works
paste(4.14, "inches, there was a lot of rain yesterday!")
```

```{r}
# now using a pipe
4.1467345 %>%
  paste(., "inches, there was a lot of rain yesterday!")
```

```{r}
# if the piped object is the first argument, you can omit the period
4.1467345 %>%
  paste("inches, there was a lot of rain yesterday!")
```

```{r}
# otherwise use the period to determine where the incoming variable should go
4.1467345 %>%
  paste("There was", ., "inches, there was a lot of rain yesterday!")
```

```{r}
# adding another step
4.1467345 %>%
  round(2) %>%
  paste("There was", ., "inches, there was a lot of rain yesterday!")
```

```{r}
# how to the previous, without using a pipe?
paste("There was", 
      round(4.1467345, 2), 
      "inches, there was a lot more rain yesterday!")
```

Now we will use a pipe operator to chain together data verbs from dplyr.

```{r}
# some new data to practice with - patient traits
df_patient <- read.csv('../data/patient_traits.csv')
head(df_patient)
```

```{r}
## original syntax
head(
  filter(df_patient, Age > 120)
)

## do this with chaining operator instead:
df_patient %>%
  filter(Age > 120) %>%
  head()
```

Use of the chaining operator allows you to apply multiple data management steps in a sequence. For example, you might be interested in doing the following:

1.  subset to only patients whose age is greater than 120
2.  sort by patient ID
3.  select only the patient ID column

```{r}
# return patient ID's where age > 120, ordered by age!
df_patient %>%
  filter(Age > 120) %>%
  arrange(p_id) %>%
  select(p_id) %>%
  head()
```

This is simply one big expression, so we can save the result like this:

```{r}
# return patient ID's where age > 120, in order!
funny_ids <- df_patient %>%
  filter(Age > 120) %>%
  arrange(p_id) %>%
  select(p_id) %>%
  head()
```

Also notice how if you use `select()` to extract a single column, it still gives you a data frame. If you wanted to extract a single column to a vector, to dplyr way to do that would be to use `pull()`.

We also use `slice()` to extract certain rows by index.

```{r}
# get back a vector this time
df_patient |>
  filter(Age > 120) |>
  arrange(p_id) |>
  slice(1:10) |>
  pull(p_id)
```

```{r}
# Mutate
df_patient %>%
  mutate(log_age = log(Age)) %>%
  head()
# Mutate but only keep certain columns
df_patient %>%
  mutate(p_id, 
         log_age = log(Age), 
         Age, 
         .keep='none') %>%
  head()
```

Notice the behavior below, the minimum age is calculated and then recycled to fill a whole new column.

```{r}
# mutate and return min age
df_patient %>%
  mutate(p_id, 
         min_age = min(Age, na.rm = T), 
         Age,
         .keep = 'none') %>%
  slice(1:5)
```

If we instead wanted aggregate our data, we can use `summarize()` as in the following example.

```{r}
## summarise and return min age
df_patient %>%
  filter(Age < 45) %>%
  summarise(min_age = min(Age),
            max_age = max(Age))
```

## Dplyr Helper Verbs

There are also many additional "helper" functions from `dplyr` that can be used inside of the main verbs, as in the next example:

-   used with **select():** `starts_with()`, `ends_with()`, `contains()`
-   used with **arrange():** `desc()`
-   used with **summarise():** `n()`, `n_distinct()`, `first()`, `last()`

```{r}
df_patient %>%
  select(p_id, Clust_id, Age, contains("lab_")) %>%
  arrange(desc(lab_value_1)) %>%
  summarise(min_age = min(Age, na.rm = T),
            n_total = n(),
            n_distinct_sites = n_distinct(Clust_id),
            first_id = first(p_id),
            last_id = last(p_id))
```

### 🛠️Coding Practice: Build Your Own Data Pipe

Build a pipe (series of data commands) that does the following manipulations to the patient data set and saves the result to a new data frame called `df_small`:

1.  Filter to only keep patients that had a college degree or graduate degree.
2.  Create a new column called `IncomeK` which is the income column divided by 1,000. Also round the result to the nearest integer. Return only the columns `Age` and `IncomeK`.
3.  Sort the values by descending values of age.

```{r}
# ANSWER HERE


# HINT: you should get back 2 columns and 3,675 rows :)
```

```{r}
# BONUS: plot the relationship between IncomeK and Age

```

**END OF EXERCISE**

------------------------------------------------------------------------

## Coding Style

------------------------------------------------------------------------

WhyCan'tIWriteAllofMySentencesLikeThis? ItSeemsInefficientToUseSpaces!

### R Syntax vs. Good Style

The R language expects very specific syntax in order to run. You will be learning these rules throughout the course but some examples include:

-   where white space matters
-   how to use special symbols like colons, brackets, etc.
-   proper variable names

Deviation from this syntax will give errors.

There is another level of consideration however, which is how easy it is for someone else to read your code and make sense of everything. And by "someone" else we also mean yourself at some point in the future :)

This is where coding style becomes important. We use style guides to help us write readable code. That is easy to troubleshoot.

A popular style guide comes from the makers of the Tidyverse collection of packages which we will learn about later. You can review this style guide at the link below:

> 🌐 <https://style.tidyverse.org/>

A variant of this style is also used at Google.

> 🌐 <https://google.github.io/styleguide/Rguide.html>

### Variable Names

There is one style aspect we will highlight here: how to properly name variables.

Properly named variables should describe their contents in as succinct a way as possible. Whether you err on the side of clarity or brevity ultimately depends on personal preference (or team preference!). Below are some examples for a collection of rainfall measurements.

```{r eval=FALSE}
## bad, uninformative
x <- c(2, 1.5, 1.25, 0.5)

## bad, too long
rainfall_in_my_city_in_Jan <- c(2, 1.5, 1.25, 0.5)

## bad syntax, R doesn't let variable names start with numbers
01_rain <- c(2, 1.5, 1.25, 0.5)

## good?
rainfall_jan <- c(2, 1.5, 1.25, 0.5)

## also good?
rain_01 <- c(2, 1.5, 1.25, 0.5)
```

There are a few common naming conventions:

-   **CamelCase:** This is common in SQL programming and is sometimes called PascalCase.
-   **snake_case:** This is the choice in the tidyverse style guide.
-   **UPPER_SNAKE_CASE:** I guess you could use this...

Sometimes a team will also add some other conventions which help highlight what type of object a variable is. See the following example:

```{r eval=FALSE}
a_num1 <- 1 # a for atomic values
a_num2 <- 2
v_nums <- c(1, 2, 3) # v for vector
v_str  <- c("the", "cat", "in", "the", "hat")
df_strings <- data.frame(v_nums, v_str) # df for data frame
```

## Answers to Exercises

### 🛠️Coding Practice: Your First Function

```{r}
bmi <- function(w, h = 1.75){
  return(w / h^2)
}

# example use
bmi(w = 87, h = 1.9)
```

### 🛠️Coding Practice: More Complex Functions

```{r}
# build your function here
my_stats <- function(x) {
    m    <- mean(x)
    std   <- sd(x)
    mn    <- min(x)
    mx    <- max(x)
    med   <- median(x)
    n     <- length(x)
    final <- round(c(n = n, mean = m, sd = std,
                     min = mn, median = med,max = mx), 2)
    return(final)
}

# and test it on the `hp` column from the `mtcars` data set
my_stats(mtcars$hp)
```

```{r}
# build your function here
# this function returns a data frame
my_stats_df <- function(x) {
    m    <- mean(x)
    std   <- sd(x)
    mn    <- min(x)
    mx    <- max(x)
    med   <- median(x)
    n     <- length(x)
    final <- data.frame(n = n, mean = m, sd = std,
                        min = mn, median = med, max = mx)
    return(final)
}

# and test it on the `hp` column from the `mtcars` data set
my_stats_df(mtcars$hp)
```

### 🛠️Coding Practice: Practicing dplyr Verbs

```{r}
# ANSWERS GO HERE

# SORT
arrange(mtcars, cyl, mpg)

# SUMMARISE
summarise(mtcars, 
          mean_cyl  = mean(cyl),
          mean_hp   = mean(hp),
          mean_disp = mean(disp))

# FILTER
filter(mtcars, mpg > 20 & cyl <= 6)

# MUTATE
mutate(mtcars, 
       mpg_sq = mpg^2)
```

### 🛠️Coding Practice: Build Your Own Data Pipe

```{r}
# ANSWER HERE
df_small <- df_patient %>%
  filter(Education == "College" | Education == "Graduate Degree") %>%
  # alt method for the filter
  # filter(Education %in% c("College", "Graduate Degree"))
  mutate(
    Age, 
    IncomeK = round(Income / 1000, 0),
    .keep = 'none'
  ) %>%
  arrange(desc(Age))

# HINT: you should get back 2 columns and 3,675 rows :)
df_small
```

```{r}
# BONUS: plot the relationship between IncomeK and Age
plot(df_small$Age, df_small$IncomeK)
```
