---
title: "User Defined Functions and Control Flow"
author: "Rian DeFaccio , for BH Analytics LLC"
date: "`r Sys.Date()`"
output: 
  html_document:
    toc: true
    toc_depth: 3
---

```{r setup, include=FALSE}
knitr::opts_knit$set(warning = FALSE, message = FALSE)
knitr::opts_chunk$set(warning = FALSE, message = FALSE)
```

```{r message=FALSE}
# load libraries and adjust settings
library(tidyverse)
```

## Introduction

------------------------------------------------------------------------

In this lesson we will dig a little deeper into programming with R. We will learn about control flow, and then how to build functions with dplyr to allow us to extend our abilities to explore our data beyond out-of-the-box capabilities. We'll discuss:

-   If-Else statements and For Loops
-   Functional Programming with dplyr
-   Using the `across()` helper function
-   Look at an exploratory analysis example

## Patient Data

------------------------------------------------------------------------

In this lesson we will use some simulated patient data.

```{r message=FALSE}
patient <- read_csv('../data/patient.csv')
head(patient)
```

## Control Flow in R

------------------------------------------------------------------------

Let's begin by considering two ways that you can control how a program runs. Essentially we can control the order in which commands get executed other than just from one line to the next line.

### If-Else Statements

Let's begin by considering an if-else statement. A couple notes on the syntax:

-   The **condition** goes inside of parenthesis. This must evaluate to `TRUE` or `False`.
-   The **R expression** that then gets evaluated if the condition was true goes inside of curly brackets.
-   There can be multiple commands/lines for each R expression.

```{r}
x = 10

# simple if-else statement
if (x <= 10) {
  print(paste0("You gave ", x, ", a small number!"))
} else {
  print(paste0("You gave ", x, ", a large number!"))
}
```

We can extend the if-else statement to consider as many different conditions as we'd like by stringing together more `else` conditions. The final `else` doesn't have a condition at all and is the catch-all for if none of the previous conditions evaluated to `TRUE`.

💡**Main Idea:** There is only one R expression that is evaluated in an if-else statement.

```{r}
x = -10

# longer statement
if (x <= 10 & x > 0) {
  print(paste0("You gave ", x, ", a small number!"))
} else if (x < 0){
  print(paste0("You gave ", x, ", a negative number!"))
} else {
  print(paste0("You gave ", x, ", a large number!"))
}
```

### 🛠️Coding Practice: A Simple Function

Take the previous set of if/else/else statements and turn it into a function:

-   Call this function `tell_me_about_my_number()`.
-   It should take one argument: `X`, which should be a number.
-   It should print the appropriate response based on the number that was input. This function doesn't have to return anything.
-   **BONUS:** Add an argument called `exl=` that specifies whether or not there should be explanation points at the end of each sentence. It's default value should be `TRUE`. If `FALSE` then each sentence should end in a period instead of an exclamation mark.

```{r}
# BUILD YOUR FUNCTION HERE


# TEST YOUR FUNCTION HERE
# tell_me_about_my_number(-50)
# tell_me_about_my_number(500)
# tell_me_about_my_number(-50, exl = FALSE)
# tell_me_about_my_number(50, exl = FALSE)
```

**END OF EXERCISE**

------------------------------------------------------------------------

One more pattern we want to be familiar with is the for loop.

```{r}
for (i in 1:5) {
  i_squared <- i^2
  print(paste(i, "squared equals", i_squared))
}
```

For the next for loop to make sense, we need to see a new way of extracting columns from data frames.

-   We use the double square brackets to extract a column name by string reference or integer location.
-   This extracts the column as a vector.

```{r}
patient[['State']][1:10]
```

You can also use the `[[]]` to create new columns.

```{r}
patient[['dummy_var']] <- 1
head(patient)
```

Now let's see how we can use a for loop to create several new factor variables in the patient data frame.

```{r}
factor_vars <- c('State', 'Gender', 'Race')

for (col_ in factor_vars) {
  new_col_name <- paste0(col_, "_fct")
  patient[[new_col_name]] <- factor(patient[[col_]])
  print(paste0("Just created [", new_col_name, "] from [", col_, "]."))
}
```

### 🛠️Coding Practice: For Loop Practice

Build a for loop that takes all the character columns in the patient data frame and converts them to uppercase. You can have your loop just write over the previous versions of the columns.

**BONUS:** Also have your code determine which columns need to converted. The `typeof()` function might be useful here. You might want to use and if-else statement inside of your for loop.

```{r}
# ANSWER HERE

```

**END OF EXERCISE**

------------------------------------------------------------------------

## Functional Programming with dplyr

------------------------------------------------------------------------

If you've learned dplyr from the ground up, chances are you started with a core collection of 'single-table' verbs and put them together into chains for complex data management and analysis techniques.

You might have started with:

-   `filter()`
-   `select()`
-   `mutate()`
-   `rename()`
-   `arrange()`
-   `summarise()`
-   `group_by()`

### 🛠️Coding Practice: Review Basic dplyr

Before we get too far, try out the following task. Take the patient data set and:

1.  Filter for only patients from California.
2.  Get the mean and median income grouped by gender.
3.  Sort the output by ascending mean income.

```{r}
# ANSWER HERE

```

**END OF EXERCISE**

------------------------------------------------------------------------

You can do a lot of data management and analytic work with only these tools, but why stop there? DPLYR has many more verbs, helpers, predicates, and other vocabulary that can accelerate your data management, exploration and analysis.

The first thing we will try to do is generalize a dplyr pipeline into a function. First let's consider our task which is to only keep patients from the 5 most frequently occurring states. The first step is to just develop the necessary code to do this specific task.

```{r}
N=5

states_to_keep <- patient %>%
  count(State, sort=TRUE) %>%
  slice(1:N) %>%
  pull(State)

result <- patient %>% filter(State %in% states_to_keep)
result
```

Now we would like to make a function to generalize this task:

-   The function can count up any column.
-   The user can define N, how many levels to keep of the counted variable.

```{r eval=FALSE}
# GIVES AN ERROR!

# a function that returns a data frame
# containing the observations from the N most frequent levels
# of the count_var
keep_top_N <- function(data, count_var, N) {
  levels_to_keep <- data %>%
    count(count_var, sort=TRUE) %>%
    slice(1:N) %>%
    pull(count_var)
  result <- data %>% filter(count_var %in% levels_to_keep)
  return(result)
}
keep_top_N(patient, State, 5)
```

The problem here is that the dplyr functions were looking for a column called `group_var`, but there was no such column!

What we were hoping would happen is that the argument `group_var` would get evaluated to something which could then get used by dplyr functions like `group_by()` or `count()`.

Without getting into the weeds too much, we'll just say that you can use a **pronoun**, in this case `.data`, which is a reference to the variable that was passed in as the first argument to the dplyr function.

Since the variable that is *almost* always passed into a dplyr function is a data frame, we can use the double square brackets to subset a column from that data frame.

This is a simplified explanation of a more complex situation, if you want to learn more, read the **Programming with dplyr** vignette.

🌐 <https://dplyr.tidyverse.org/articles/programming.html>

Here's how we've fixed up our previous attempt:

```{r}
# a function that returns a data frame
# containing the observations from the N most frequent levels
# of the count_var
keep_top_N <- function(data, count_var, N) {
  levels_to_keep <- data %>%
    count(.data[[count_var]], sort = TRUE) %>%
    slice(1:N) %>%
    pull(.data[[count_var]])
  result <- data %>% 
    filter(.data[[count_var]] %in% levels_to_keep) %>%
    arrange(.data[[count_var]])
  return(result)
}
keep_top_N(patient, 'State', 5)
keep_top_N(patient, 'FirstName', 5)
```

## Using `across()`

What if want to convert several variables in our data set to factors? We did this earlier with a for loop, but there is a way to do this with dplyr.

We could mutate these three variables like the next example:

```{r}
patient %>% 
  mutate(Gender = factor(Gender),
         Race   = factor(Race),
         State  = factor(State))
```

But this can be onerous on more than a few columns. Perhaps a better way would be to select several columns and then transform them.

This is where helper verbs come in! And how do we choose which columns we want? We use `across()`! Here are the arguments that `across()` takes:

-   Specify the variables to mutate.
-   Specify the function to use to mutate each column.

```{r}
patient %>% 
  mutate(across(c(State, Gender, Race), toupper))
```

There are many ways we can use `across()` in creative ways to specify which columns we'd like to manipulate.

```{r}
# useful pattern if you had several ID columns all imported as numeric
patient %>% 
  mutate(across(contains("ID"), as.character)) %>%
  head()
```

```{r}
patient %>% 
  mutate(across(everything(), as.character)) %>%
  head()
```

See the dplyr documentation for more examples of how to use `across()`.

🌐 <https://dplyr.tidyverse.org/reference/across.html>

### 🛠️Coding Practice: Build a Pivot Table Function

Build a function that performs a pivot table in the style of other familiar analytic software. Your function should take the following arguments:

-   `data`:
-   `rows`:
-   `columns`:
-   `values`:
-   `agg_func`:

🗝️**HINT:** Begin by just writing the code that performs a pivot table on some specific data, then try to convert it all into a function.

```{r}
# ANSWER HERE

```

**END OF EXERCISE**

------------------------------------------------------------------------

## 📈 EDA: Berkeley Admissions Data

------------------------------------------------------------------------

Before moving onto our next topic we need to introduce a new data set. The Berkeley admissions data contains information about all the students who applied to UC-Berkeley's graduate programs in Fall 1973.

First we will import and examine this data:

```{r message=FALSE}
berkeley <- read_csv("../data/berkeleydf.csv")
head(berkeley)
```

Another example of pairwise data analysis is to examine if a categorical variable is related to another categorical variable. This is traditionally done by looking at frequency tables.

We've seen this in the uni variate case, and will now extend it to pairs of variables.

In this case, we are primarily concerned with the `Admit` variable, which is a binary variable indicating whether or not a applicant was admitted to the school. We want to see whether admissions rates are associated with either gender or department.

🎯 **QUICK CHECK:** What is the summary statistic that describes a binary variable? There are a couple of names we could use here.

**Answer here:**

```{r}
## overall summary 
table(berkeley$Admit)
table(berkeley$Admit) |> prop.table()
```

```{r}
## using dplyr
berkeley |>
  summarise(total_admit = sum(Admit == 'Admitted'),
            admit_prop = sum(Admit == 'Admitted') / n())
```

### Contingency Tables

Let's first examine if admissions are related to gender.

The appropriate tool here is a contingency table. We can use a group-by analysis to get there.

```{r}
## using dplyr - calculate admission rate by gender
berkeley |>
  group_by(Gender) |>
  summarise(total_admit = sum(Admit == 'Admitted'),
            admit_prop = sum(Admit == 'Admitted') / n()) |>
  ungroup()
```

Does admission appear to be related to gender? It sure looks like it so far! More males were admitted overall, and at a higher rate than females.

Are admissions related to department?

```{r}
## using dplyr - calculate admission rate by department
berkeley |>
  group_by(Dept) |>
  summarise(total_admit = sum(Admit == 'Admitted'),
            admit_prop = sum(Admit == 'Admitted') / n())
```

It also looks like admissions are definitely related to department. Some departments must be much more competitive than others. It also appears that there were more applicants to the departments that were easier to get into.

A fantastic way to learn about the steps of a complicated-looking dplyr pipeline is to successively highlight and run more rows. Start with just the first couple of lines, and then add another, and so on, to see all the intermediate products.

```{r}
berkeley |>
  group_by(Gender, Dept) |>
  summarise(admit_total = sum(Admit == "Admitted"),
            admit_rate = sum(Admit == "Admitted") / n(),
            .groups = 'drop') |>
  pivot_wider(names_from = Gender, values_from = c(admit_rate, admit_total)) |>
  select(Dept, contains("Female"), contains("Male")) |>
  knitr::kable()
```

### ✍️ Your Turn! Interpret the Berkeley Data

Interpret the results of this last table. Do you still think U.C. Berkeley was accepting females at a lower rate than males?

**Answer here:**

## Conclusion

------------------------------------------------------------------------

This concludes our presentation on control flow in R and programming with dplyr. We hope it will be helpful!

## Answers to Questions

------------------------------------------------------------------------

### 🛠️Coding Practice: A Simple Function

```{r}
# BUILD YOUR FUNCTION HERE
tell_me_about_my_number <- function(x, exl=TRUE) {
  
  
  ending <- ifelse(exl, "!", ".")
  
  if (x <= 10 & x > 0) {
    print(paste0("You gave ", x, ", a small number", ending))
  } else if (x < 0){
    print(paste0("You gave ", x, ", a negative number", ending))
  } else {
    print(paste0("You gave ", x, ", a large number", ending))
  }
}

# TEST YOUR FUNCTION HERE
tell_me_about_my_number(-50)
tell_me_about_my_number(500)
tell_me_about_my_number(-50, exl = FALSE)
tell_me_about_my_number(50, exl = FALSE)
```

### 🛠️Coding Practice: For Loop Practice

```{r}
# simple approach
char_cols <- c('FirstName','LastName','State','Gender','Race')
for (col_ in char_cols) {
  current_data <- patient[[col_]]
  patient[[col_]] <- toupper(current_data)
}
 
# BONUS approach 
for (col_ in colnames(patient)) {
  current_data <- patient[[col_]]
  if (typeof(current_data) == 'character') {
    patient[[col_]] <- toupper(current_data)
    paste0('Converted [', col_, '] to uppercase.') %>% print()
  } else {
    paste0('------ [', col_, '] was not a character column.') %>% print()
  }
}

# let's take a look
patient
```

### 🛠️Coding Practice: Review Basic dplyr

```{r}
patient %>% 
  filter(State=="CA") %>% 
  group_by(Gender) %>%
  summarise(avg_income    = mean(Income, na.rm=T),
            median_income = median(Income, na.rm=T)) %>%
  arrange(avg_income)
```

### 🛠️Coding Practice: Build a Pivot Table Function

```{r}
pivot_table <- function(data, rows, columns, values, agg_func, fill_val=0) {
  df <- data %>%
    group_by(.data[[rows]], .data[[columns]]) %>%
    summarise(agg_values = agg_func(.data[[values]], na.rm=TRUE),
              .groups = 'drop') %>%
    pivot_wider(names_from = columns, 
                values_from = agg_values,
                values_fill = fill_val)
  return(df)
}
pivot_table(patient, 
            rows = "Gender", columns = "Race",
            values = "Income", agg_func = mean)
```

### ✍️ Your Turn! Interpret the Berkeley Data

This is an example of Simpson's Paradox. In this example, males were applying mostly to the easy-to-get-into departments and females were applying to hard-to-get-into departments. This made it appear in the aggregate that females were generally being accepted at a lower rate. When we stratified by gender and department, we see that acceptance rate in any given department was either roughly equal, or females were actually accepted at a higher rate.

*"Simpson's paradox is a phenomenon in probability and statistics in which a trend appears in several groups of data but disappears or reverses when the groups are combined" - Wikipedia*
