---
title: "Dates, Factors, and More EDA"
author: "Rian DeFaccio, for BH Analytics LLC"
date: "`r Sys.Date()`"
output: 
  html_document:
    toc: true
    toc_depth: 3
---

```{r setup, include=FALSE}
knitr::opts_knit$set(warning = FALSE, message = FALSE)
knitr::opts_chunk$set(warning = FALSE, message = FALSE)
```

```{r}
# load libraries and adjust settings
library(tidyverse)
```

## Introduction

------------------------------------------------------------------------

In this lesson we will explore a couple more packages from the tidyverse that work with specialized data types, as well perform our first multivariate exploratory analysis! We will look at the following topics:

-   Dates and Times
-   Manipulating Factors
-   Exploring Pairs of Continuous Variables

## Patient Traits Data

------------------------------------------------------------------------

We will use the simulated patient traits data set in this lesson.

```{r}
df_patient <- 
  read_csv('../data/patient_traits.csv')
head(df_patient)
```

## Date and Date-Time Variables

------------------------------------------------------------------------

We will first learn about date and date-time variables. These are variables that hold information about dates and times. In this section we will consider two different ways to create and manipulate date and date-time variables in R:

-   Using base R methods.
-   Using the lubridate package from the Tidyverse.

We will sometimes present methods from both, and other times only present one method when we feel one tool or the other is easier to use.

### Creating Dates

R can convert dates from strings. R's native date type is called POSIX. Technically R has two date formats which differ only in terms of how the data is stored internally:

-   **POSIXlt:** dates stored as a list with elements for \[second, minute, hour, day, month, year, ...\]
-   **POSIXct:** dates stored as "seconds since January 1, 1970"

For our purposes, we don't need to worry too much about the difference between these two date formats. We will mostly concern ourselves with the class of these objects, which will present as `Date`.

We will generally be creating date variables from strings. The lubridate package has conveniently named functions like `ymd()` which correspond to the expected form of the incoming string.

```{r}
penguin_awareness_day <- lubridate::ymd("2023-01-20")
class(penguin_awareness_day)
```

What if there is a time component?

```{r}
penguin_party_time <- ymd_hms('2023-01-20 14:30:00')
penguin_party_time
class(penguin_party_time)
```

If you noticed that the previous object actually had two classes, then you are very observant! POSIXct and POSIXlt variables also have an additional class called POSIXt which basically allows us to use the two types of variables interchangeably.

### Get Current Date/Time

We can get the current date and/or time using the following functions:

The lubridate package provides us with the `now( )` and `today( )` functions.

```{r}
todays_date <- today()
todays_date
class(todays_date)
```

```{r}
time_now <- now()
time_now
class(time_now)
```

### Extract Date Components

To extract information from a date or date-time variable such as the year or month we can use the base R function `strftime()`.

### ✍️ Your Turn! Date and Time Elements

Brainstorm as many different date or time elements that you could possible extract from a date-time variable. Put some answers in the **Zoom Chat ☞** and collect them in your notebook if you'd like.

**Answer here:**

We use strftime formatting codes to extract date and time elements. See the **R strptime Documentation** for the full list of codes. Scroll down to the **Details** section.

🌐 <https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/strptime>

```{r}
# get the year
strftime(penguin_awareness_day, format = '%Y')
```

```{r}
# get the month as a padded 2-digit number
strftime(penguin_awareness_day, format = '%M')
```

```{r}
# get the month as a fully spelled name
strftime(penguin_awareness_day, format = '%B')
```

```{r}
# get the hour of the penguin party (24-hour time)
strftime(penguin_party_time, format = '%H')
```

You can also use the `strftime()` function to create more complex outputs.

```{r}
strftime(penguin_party_time, 
         format = "The party is at %I o'clock %p on %A.")
```

To extract date components with the lubridate package we use special functions that correspond to particular date parts. Some functions have additional arguments to control how the component is rendered.

```{r}
# get the year
lubridate::year(penguin_awareness_day)
```

```{r}
# get the hour
lubridate::hour(penguin_party_time)
```

### Sequences of Dates \[OPTIONAL\]

To make sequences of dates, you can use the base R `seq()` function.

```{r}
# sequence of days
seq(ymd("2015-03-15"), ymd("2015-03-20"), by="day")
```

```{r}
# sequence of month-starts
seq(ymd("2015-03-01"), ymd("2015-06-01"), by="month")
```

```{r}
# sequence of fortnights
seq(ymd("2015-03-15"), ymd("2015-05-20"), by="14 days")
```

The object that we get out of the previous commands are vectors of date variables.

```{r}
my_seq <- seq(ymd("2015-03-15"), ymd("2015-05-20"), by="14 days")
class(my_seq)
length(my_seq)
```

All of the previous commands that we used on individual/atomic dates also work on vectors of dates.

```{r}
strftime(my_seq, '%A')
```

### 🛠️Coding Practice: Getting Day of Week with lubridate

Practice using the documentation from the lubridate package and figure out how to extract the days of the week from the variable `my_seq`.

🌐 <https://lubridate.tidyverse.org/reference/index.html>

```{r}
# ANSWER HERE

```

Now see if you can extract the quarter from `my_seq` using an appropriate lubridate function. For a **BONUS**, can you extract the fiscal quarter? The fiscal year begins on October 1.

```{r}
# first, here is my_seq for reference
my_seq

# THEN FIND THE QUARTER(S) HERE

```

**END OF EXERCISE**

------------------------------------------------------------------------

### Time Spans \[OPTIONAL\]

There are three types of time spans introduced with the `lubdridate` package:

-   **durations:** an exact number of seconds
-   **periods:** time represented in a number of units (like weeks)
-   **intervals:** have starting and ending point

#### Durations

We can construct durations using any of the conveniently named functions from lubridate that start with the `d` (for 'duration') prefix.

```{r}
lubridate::dseconds(120)
lubridate::dminutes(2)
```

```{r}
lubridate::dyears(1)
```

We can perform arithmetic on our durations:

```{r}
2*lubridate::dyears(1)
lubridate::dyears(1) + lubridate::dmonths(1) + lubridate::ddays(1)
```

We can also use these durations to get the amount of time before or after a date variable:

```{r}
penguin_awareness_day
penguin_awareness_day - lubridate::dyears(1) #hmm, not as expected...
```

Durations can be a little imprecise at times. Consider as an example the fact that not all years have the same number of seconds (leap-year?). Daylight savings can also make some duration calculations give unexpected results.

#### Periods

A somewhat more flexible object for tracking the span of time is the period. This type of variable tracks a duration of time in some kind of unit, such as "a period of 3 weeks".

There are convenient functions for building these, which look kind of like the duration functions, except without the `d` prefix this time.

For the most part, periods behave a lot like durations.

```{r}
lubridate::minutes(120)
lubridate::hours(2)
```

Arithmetic is similar:

```{r}
2 * lubridate::years(1)
lubridate::years(2) + lubridate::weeks(14)
```

And we can use periods with date variables also.

```{r}
penguin_awareness_day
penguin_awareness_day - lubridate::years(1)
```

We can see here an example of where a duration and a period come out differently.

```{r}
# a period ... "the same date a year later"
penguin_awareness_day - lubridate::years(1)

# a duration ... when a year equals exactly 365.25 days
penguin_awareness_day - lubridate::dyears(1)
```

#### Intervals

The last time span is the interval, defined by two date endpoints. There are two ways to construct an interval, both demonstrated below.

Dates can be tested to see if they fall `%within%` an interval.

> Be careful! The tsibble object (a time series package) also has an `interval()` function, but it does something different. Be careful of name collisions, and pay attention to masking warnings when you load packages. Using the `::` notation is often a good choice with this function.

```{r}
summer2023 <- lubridate::interval(ymd("2023-June-21"), ymd("2023-Sep-22"))
summer2023
lasagne_day <- lubridate::ymd("2023-07-29")
lasagne_day %within% summer2023
```

```{r}
# another syntax to establish an interval
summer2019 <- ymd("2019-June-21") %--% ymd("2019-Sep-22") 
summer2019
xmas <- ymd("2023-12-26")
xmas %within% summer2019
```

There are many useful functions for examining interval variables which generally begin with the prefix `int_` or `time_`.

```{r}
int_length(summer2019) # in seconds
time_length(summer2019, unit="day")
```

### Date Differences

If we take the difference of two dates, we end up with a `difftime` object in base R.

```{r}
penguin_awareness_day
lasagne_day
```

```{r}
lasagne_day - penguin_awareness_day
# the result here is a base R difftime object
class(lasagne_day - penguin_awareness_day)
```

We can convert `difftime` objects into integers if we want to.

```{r}
as.numeric(lasagne_day - penguin_awareness_day)
```

Using the base R `difftime()` function gives us a little more control over the result.

```{r}
difftime(lasagne_day, penguin_awareness_day, units = "weeks")
```

To achieve a difference in years, we can use a lubridate interval.

```{r}
lubridate::interval(penguin_awareness_day, lasagne_day)
lubridate::interval(penguin_awareness_day, lasagne_day) / years(1)
```

### 🛠️Coding Practice: Dates in a Data Frame

Let's do some work on the `Baseline_date` column from the patient traits data frame. It is currently a character column. First, create a version of this variable called `baseline_date_dt` that has a date as its data type. Check that the conversion was successful.

```{r}
# ANSWER HERE

```

Now, create two new variables called `days_since_baseline` and `years_since_baseline` that calculates how many days/years today is from the baseline date for each patient. You don't have to save these variables, but print your results in a way that you can check your work.

```{r}
# ANSWER HERE

```

**END OF EXERCISE**

------------------------------------------------------------------------

### Time Zones \[OPTIONAL\]

You can check the time zone R thinks you are in like this:

```{r}
Sys.timezone()
```

We are going to use the lubridate package for time zone management.

To set the time zone for a variable, use the `tz=` argument.

```{r}
xmas_morning <- ymd_hms('2023-12-26 08:00:00', tz = "America/Los_Angeles")
xmas_morning
```

The main reason for using time zones is to control how a variable is displayed. To keep the underlying "moment in time" the same while changing the time zone, use the `with_tz()` function.

```{r}
# still the same time, but displayed differently
with_tz(xmas_morning, tz = "America/New_York")
```

If you think that a variable has been labelled with the wrong time zone, you can also change the underlying "moment in time" using the `force_tz()` function.

```{r}
# use the old time, but give it a new time zone
force_tz(xmas_morning, tz = "America/New_York")
```

## Factor Conversions

------------------------------------------------------------------------

Let's now consider one more type of column you can have in a Data Frame, a factor. Any discrete variable with "not too many" levels is a candidate to be converted to a factor variable.

-   Let's start with the `outcome` variable. This is a generic bad outcome... it could be re-hospitalization, or loan default... it is just a binary indicator for bad outcome.
-   There are no missing values, and the only current values are 0 and 1. We can convert this directly to a factor, mapping the zeros as 'No' and the ones as 'Yes'.
-   I will create a new variable `outcome_f` so I don't lose the original information.

```{r}
# different ways to explore this variable
unique(df_patient$outcome)
table(df_patient$outcome)
```

```{r}
# Convert to factor, update labels
df_patient$outcome_f <- 
  factor(df_patient$outcome,
         levels = c(0, 1),        # current values
         labels = c('No', 'Yes')) # values I want to end up with
# Now it has levels!
head(df_patient$outcome_f)
```

```{r}
str(df_patient$outcome_f)
```

Notice the structure of the factor variable. We have 2 levels which are actually just the integers 1 and 2. Then we have labels that get mapped in the following way:

-   1 = "No"
-   2 = "Yes"

Its in this way that factors are more memory efficient, they aren't storing all those extra digits!

How about the Location variable? There is an issue here we need to solve before we create the factor. What are the problems?

```{r}
# investigate Location variable
unique(df_patient$Location)
```

```{r}
table(df_patient$Location, useNA = 'ifany')
```

It looks like there are some missing values (NA), and some more values with `999` as the data. These are probably missing, and should be set to missing. One way to implicitly set the `999` values to missing is to simply exclude them from the levels argument in the `factor()` function.

```{r}
# create factor
df_patient$Location_fac <- 
  factor(df_patient$Location,
         levels = c('Urban', 'Rural'))
table(df_patient$Location_fac, useNA = "ifany")
```

### 🛠️Coding Practice: Clean Up the Education Variable

And finally let's convert the education variable.

```{r}
table(df_patient$Education, useNA = 'ifany')
```

Try to convert the education variable to a factor, but notice if there is any other data cleaning you should do first.

🗝️**HINT:** The `tolower()` function might be helpful here.

```{r}
# ANSWER HERE

```

**END OF EXERCISE**

------------------------------------------------------------------------

### Intermediate Factor Manipulation with `forcats`

Working in factors using base R tools is fine, but we will briefly introduce the `forcats` package for easier handling of factors. We will illustrate a couple of the features here that are a little more difficult with base R tools.

The first thing we might want to do is relabel factor levels.

```{r}
# first make sure we have [education_fac] column
df_patient$education_fac <- df_patient$Education %>%
  tolower() %>%
  factor(levels=c('high school', 'college', 'graduate degree'))

df_patient$education_alt <- 
  fct_recode(df_patient$education_fac,
             pre_college = "high school", 
             uni = "college", 
             grad = "graduate degree")
table(df_patient$education_fac)
```

```{r}
table(df_patient$education_alt)
```

There is a function that will reverse the order of a factor.

```{r}
df_patient$education_rev <- fct_rev(df_patient$education_alt)
table(df_patient$education_rev)
```

You can also reorder the levels of a factor based on the frequency count for that factor. Factor ordering is important for a couple of reasons, but the most common use case is for display of results in figures and tables.

```{r}
## build a factored education variable ordered by frequency
df_patient$education_by_freq <- fct_infreq(df_patient$education_alt)

## see how the output differs
table(df_patient$education_by_freq)
```

### ✍️ Your Turn! Table Ordering

Play around with the `table()` function and answer the following questions:

1.  In what order does the `table()` function display outputs of character columns?
2.  In what order does the `table()` function display the outputs of factor columns?

```{r}
# TRY SOME CODE HERE

```

One more thing you might want to do is re-order the levels of one factor based on the values of another variable. We will usually use this in the context of displaying data in plots.

First consider the following simple data frame.

```{r}
df_pets <- data.frame(name   = c("Suzy", "Spike", "Beethoven", "Dog"),
                      weight = c(15, 20, 30, 40))
df_pets
```

```{r}
df_pets$name_fct <- fct_reorder(df_pets$name, df_pets$weight)
ggplot(df_pets, aes(name, weight)) + geom_col()
ggplot(df_pets, aes(name_fct, weight)) + geom_col()
```

We can also make the level ordering of one variable depend on a summary of another variable.

In the next example, we will make the levels of education be ordered by the median value of income within each education group.

```{r}
df_patient$Location_fac2 <- fct_reorder(df_patient$Location, 
                                        df_patient$lab_value_3, 
                                        .fun = median,
                                        .na_rm = TRUE)
ggplot(df_patient, aes(Location, lab_value_3)) + geom_boxplot()
ggplot(df_patient, aes(Location_fac2, lab_value_3)) + geom_boxplot()
```

## 📈 EDA: Continuous by Continuous Variables

------------------------------------------------------------------------

So how can we identify the relationship between two continuous variables?

The primary question is this: *Does the distribution of one variable look different across levels of the other variable?*

So what does that look like when both variables are continuous?

```{r}
# grab the gapminder data
df_gapminder <- read.csv('../data/gapminder.csv')
df_2002 <- df_gapminder %>%
  filter(year == 2002 & continent != 'Oceania')
head(df_2002)
```

### Scatter Plots

The appropriate plot here is a scatter plot. Let's examine the association between life expectancy and GDP per capita.

```{r}
## Is life expectancy related to population?
ggplot(df_2002, aes(x = gdpPercap, y = lifeExp)) + 
  geom_point()
```

There appear to be some outliers that have somewhat higher GDP per capita, but relatively low life expectancy in 2007. We could work on these outliers a little...

```{r}
## find outliers using dplyr and guessing at cut-points
df_2002 |> filter(gdpPercap > 8000 & lifeExp < 60)
```

### Correlation

The appropriate statistical summary for examining the association between two continuous variables is the correlation. Correlation ranges from:

-   **-1:** Perfect negative correlation (as X does down Y goes up)

-   **0:** No correlation (as X changes it has no association with Y)

-   **1:** Perfect positive correlation (as X goes up so does Y)

We can examine different types of correlation using the base R `cor()` function:

-   **Pearson:** This is a measure of the linear association between two variables.
-   **Spearman:** This is a rank-based statistic and may be more appropriate when the association between two variables is non-linear.

```{r}
cor(df_2002$gdpPercap, df_2002$lifeExp) # pearson is default
cor(df_2002$gdpPercap, df_2002$lifeExp, method = "spearman")
```

There is a classic example of how correlation can be misleading. The following image shows Anscombe's Quartet, a collection of four data sets that have basically the same correlation. We can see the actual relationships between the two variables are quite different from each other despite the correlations being similar.

![Anscombe's Quartet: <https://en.wikipedia.org/wiki/Anscombe%27s_quartet>](images/ansombes_quartet.png){width="600"}

💡**MAIN IDEA: Always Use Visualizations in Addition to Summary Statistics!**

Why???

...

...

...

...

...

...

There might be a dinosaur in your data...

The **Data Saurus Dozen** is another collection of data sets with equivalent correlation. You can grab this data from the R package!

🌐 <https://cran.r-project.org/web/packages/datasauRus/vignettes/Datasaurus.html>

## Conclusion

------------------------------------------------------------------------

That wraps up our discussion of working with dates and times, factors, and performing exploratory analysis of pairs of continuous variables!

## Answers to Questions

------------------------------------------------------------------------

### 🛠️Coding Practice: Getting Day of Week with lubridate

Practice using the documentation from the lubridate package and figure out how to extract the days of the week from the variable `my_seq`.

🌐 <https://lubridate.tidyverse.org/reference/index.html>

```{r}
# ANSWER HERE 
week(my_seq)
```

### 🛠️Coding Practice: Dates in a Data Frame

```{r}
df_patient$baseline_date_dt <- mdy(df_patient$Baseline_date)
df_patient %>% select(contains("baseline")) %>% slice(1:5)
```

```{r}
df_patient %>%
  mutate(days_since_baseline   = 
           difftime(today(), baseline_date_dt, units = 'days'),
         years_since_baseline  = days_since_baseline / 365.25,
         years_since_baseline2 = 
           interval(baseline_date_dt, today()) / years(1)) %>%
  select(contains('baseline')) %>%
  head()
```

### 🛠️Coding Practice: Clean Up the Education Variable

```{r}
# Create new clean variable:
# convert to character
# make lower case, then convert to factor with correct order
df_patient$education_fac <- df_patient$Education %>%
  tolower() %>%
  factor(levels=c('high school', 'college', 'graduate degree'))

# check your work
table(df_patient$Education, df_patient$education_fac, useNA = 'ifany')
```
