---
title: "Working with Strings"
author: "Rian DeFaccio | created for BH Analytics"
output: 
  html_document:
    toc: true
    toc_depth: 3
    self_contained: yes
    mode: selfcontained
---

```{r setup, include=FALSE}
knitr::opts_knit$set(warning = FALSE, message = FALSE)
knitr::opts_chunk$set(warning = FALSE, message = FALSE)
```

## Introduction

------------------------------------------------------------------------

In this lesson we will learn about working with text data. We will cover the following topics:

-   How to import text data into R
-   An introduction to the **stringr** package
-   An introduction to regular expressions

We will focus on the **stringr** package, which is very useful for working with text data. We will highlight some of the most useful features here, but it has many more use cases.

## Importing Unstructured Data

------------------------------------------------------------------------

First, we will show you how to import a text file into R. We use the base R `readLines()` function to get the text file into R as a vector. Each line of the text file is an element of the vector. We then convert the vector into a single column data frame.

We will import a data set with information on consults at a hospital.

```{r}
# each row of the .txt file is an element of the vector
dat <- readLines("../data/mtsamples1.txt")
head(dat)
```

```{r}
## convert to data frame
df_str <- as.data.frame(dat)
names(df_str) <- "consults"
df_str
```

## Managing Text Data with `stringr`

------------------------------------------------------------------------

Now that we have some text data imported and converted into a data frame, we can do all sorts of things!

### Counting Characters

First we will count the number of characters for each row in a given string column.

We will use the `str_count()` function from **stringr** to do this.

Notice that you can use this function on either a single string, or on a vector.

```{r}
library(stringr)
str_count("The Cat in the Hat")
```

```{r}
str_count(df_str$consults)[1:6]
```

We could also save that result to a new column in the original data frame. It can be very useful to store information about text in this way.

```{r}
library(dplyr)
df_str |> 
  mutate(nchar = str_count(consults)) |>
  slice(1:5)
```

### Extract By Character Index

We can extract characters based on integer location using `str_sub()`. Lets do that to make these entries a little shorter and easier to view.

```{r}
df_short <- df_str |>
  mutate(consults = str_sub(consults, start = 1, end = 50))
df_short |> 
  slice(1:5)
```

### Detect Pattern

We search for patterns using `str_detect()`.

```{r}
df_short |>
  mutate(contains_header = str_detect(consults, "Consult")) |>
  slice(1:5)
```

### 🛠️Coding Practice: Addresses

Let's practice some simple string manipulations using **stringr** functions.

Begin by importing the 'addresses.txt' file and converting the result into a single column data frame called `df_address`. Rename the single column in the data frame to be `address`. Finish the cell below by printing `df_address` to the screen.

```{r}
# import addresses data


# rename the column


# print to the screen


```

Now, build a new data frame called `df_address2` which has the following columns in addition to the original address column:

-   a column called `WA` which equals `TRUE` if the address is in Washington state
-   a column called `nchar` which counts the number of characters in each address

```{r}
# create the [WA] and [nchar] columns


```

**END OF CODING PRACTICE**

------------------------------------------------------------------------

### Extract Pattern

We can also extract patterns with `str_extract()`. This function accepts regular expressions, which we will learn more about shortly.

In the first example we are extracting the exact pattern "Consult" from the full text.

In the second example we use a very simple regular expression. The pipe operator in this context means "or"... So we are extracting the first instance of either "EKG" or "Alzheimer".

```{r}
df_short |>
  mutate(pattern1 = str_extract(consults, "Consult"),
         pattern2 = str_extract(consults, "EKG|Alzheimer")) |>
  slice(1:6)
```

Another idea related to extracting patterns is that of the "capture group". We use parentheses in this setting to indicate which part of a pattern we would like to extract. Consider the following example where we would like to extract "Abnormal EKG", but only when it is preceded by the text "Consult -".

When we use capture groups, we also have to specify the group using the `group=` argument. In some cases you might have multiple capture groups.

```{r}
df_short |>
  mutate(pattern = str_extract(consults, 'Consult - (Abnormal EKG)', group = 1)) |>
  slice(1:4)
```

### Remove Characters

We can also remove elements from strings. The function `str_remove()` removes the first instance of the pattern. Use `str_remove_all()` to remove all matches within a string.

```{r}
# removing the first instance of our pattern
df_removed <- df_short |>
  mutate(med_word_removed = str_remove(consults, "with|and|Consult - | a |of|The"),
         only_med_words = str_remove_all(consults, "with|and|Consult - | a |of|The")) |>
  slice(1:6)
df_removed
```

### 🛠️Coding Practice: Extracting City and State

In this exercise, try to extract the city and state from the addresses into columns called `city` and `state`. We don't need to save these results, simply perform the operation and print to the screen. You can use `df_address2` as your starting point.

```{r}
# extract city and state


```

**END OF CODING PRACTICE**

------------------------------------------------------------------------

### Padding Strings

We can pad strings up to a certain length with `str_pad()`.

```{r}
df_removed |>
  mutate(words_pad = str_pad(only_med_words, width = 30, side = "left", pad = "_")) |> 
  select(only_med_words, words_pad) |>
  slice(1:6)
```

### Pattern Replacement

Another option is to replace a pattern with another string.

```{r}
df_short |>
  mutate(consults2 = str_replace(consults, "Consult", "Office Visit")) |>
  slice(1:6)
```

### Separate and Unite

One more thing you may want to do is to split a string up into multiple pieces. In the context of a string column in a data frame, this means splitting a single string column into multiple columns.

We will use the `separate()` function from the **tidyr** package to do this.

```{r}
library(tidyr)
df_sep <- df_short |>
  separate(consults, into = c("first", "second"), sep = " - ", remove = FALSE)
df_sep |> 
  select(first, second) |> 
  slice(1:6)
```

The reverse of `separate()` is `unite()`.

```{r}
df_sep |>
  select(first, second) |>
  unite(together, sep = "...", na.rm = TRUE)
```

### 🛠️Coding Practice: More with stringr

Let's practice a little more with the addresses data. We have supplied some code to generate a data frame to get you started. It is your job to add the following columns:

-   a new zip code column called `zipcode_clean` where you add "0"s on the left side, padding up to 5 digits
-   a new column called `address_clean` where you replaces instances of "San Dego" with "San Diego"

Feel free to build these columns and just print the result to the screen.

```{r}
# don't worry, we'll learn about the code below shortly!
#df_address3 <- df_address |>
#  mutate(zipcode = str_extract(address, '[0-9]{3,5}$'))

# here's your starting data set, add new columns and print
#df_address3

```

**END OF CODING PRACTICE**

------------------------------------------------------------------------

That wraps up our introduction to using the **stringr** package (and a little bit of **tidyr**).

Next up, we are going to learn about a tool that will supercharge our ability to manipulate strings.

## Introduction to Regular Expressions

------------------------------------------------------------------------

We can pass regular expressions (REGEX) to many **stringr** functions which will allow us to capture more types of string patterns. REGEX is essentially a mini-language that allows us to express very complex and flexible pattern matches (instead of a simple exact string matches). Regular expressions are implemented in many different computing languages, not just in R. This is a deep topic, but we will introduce you to some regular expression concepts that are quite useful for data science work.

### Anchors

To begin, let's consider if we wanted to match only strings that start with certain characters, or end with certain characters. There are two special symbols that indicate the start and the end of the string:

-   `^`: start of string

-   `$`: end of string

We can use these in our matches to indicate start and end of string matches. Here is an example where we indicate which rows started with the word "Consult".

```{r}
# which rows start with "Consult"?
df_short |> 
  mutate(pattern = str_detect(consults, '^Consult')) |> 
  slice(1,4,7,8)
```

Here is an example of using the `$` to identify which rows end with "Atrial Fibrillation".

```{r}
df_short |> 
  mutate(pattern = str_detect(consults, 'Atrial Fibrillation$')) |> 
  slice(7:10)
```

### Special Word Collections

Sometimes we want to match multiple types of strings, such as any alphanumeric string, or any white space string. There are reserved collections of characters like this in regular expressions. Here are some that we find particularly useful:

-   `\s` any white space

-   `\S` any not white space

-   `\d` any digit

-   `\D` any notdigit

-   `\w` any Unicode string pattern (a-z, 0-9, \_)

Let's find all the strings that don't start with white space!

The regular expression: `^\S` should match:

1.  the start of the string
2.  any non-white space character.

So any line of text that meets this criteria should be retained in the filter.

```{r}
df_short |> 
  filter(str_detect(consults, '^\\S')) |>
  slice(1:6)
```

An alternative approach to matching the non white space would be to match any alphanumeric:

```{r}
df_short |> 
  filter(str_detect(consults, '^\\w')) |>
  slice(1:6)
```

We can add more instances of our set to capture longer patterns flexibly. For instance, here we indicate which rows start with at least two consecutive alphanumeric characters.

```{r}
df_short |> 
  mutate(pattern = str_detect(consults, '^\\w\\w')) |>
  slice(1,24,25)
```

If we wanted to filter to only rows that start with three letter words, we could use a pattern like the following:

```{r}
df_short |> 
  filter(str_detect(consults, '^\\w\\w\\w '))
```

### The Wildcard

In the prior line of code, we wanted to match any line of text that started with three word characters (`\w`).

Another very useful special character in regular expressions is the period `.`

This will match anything (but only once, since there is only one period).

For example, here we capture ages through the pattern "..-year-old". We will return to using the full `mtsamples` data frame here, not the abbreviated version.

```{r}
df_str |> 
  mutate(pattern = str_extract(consults, '..-year-old')) |>
  select(pattern) |>
  filter(!is.na(pattern))
```

We can combine the previous REGEX with the idea of capture groups to extract just the age.

```{r}
df_str |> 
  mutate(pattern = str_extract(consults, '(..)-year-old', group=1)) |>
  select(pattern) |>
  filter(!is.na(pattern))
# some funny ages here
# we'll do a better extraction for this shortly
```

### Custom Sets

We can actually define our own sets like `\d`, but with whatever we want in it. We can define a set by using `'[]'`, then putting whatever we want to match inside the brackets. For example, if we wanted to match any line that starts with an 'A' or 'B' or 'P', we could do the following:

```{r}
df_short |> 
  filter(str_detect(consults, '^[ABP]'))
```

All characters are indexed in REGEX, so we can specify ranges of values like in the following example:

```{r}
df_str |> 
  mutate(pattern = str_extract(consults, '([0-9][0-9])-year-old', group=1)) |>
  select(pattern) |>
  filter(!is.na(pattern))
```

If we want to define a set by saying 'everything but these characters...' we can do so by putting a caret at the start of the set (just after the square bracket). This is a bit confusing, since you already learned the caret `^` means 'starts with' when it is used outside of the square brackets. Here are some examples:

-   `[^a]` - this would match anything that is not 'a'

-   `[^ab]` - this would match anything that is not 'a' or 'b'

-   `^[^ab]` - this would match if the string starts with something other than 'a' or 'b'

In the next example we keep rows that start with characters other than "C" and "P".

```{r}
df_short |> 
  filter(str_detect(consults, '^[^CP]'))
```

### Frequency Indicators

We often wish to match some pattern multiple times. We have a few options for doing this. What if we wanted to filter for rows that start with 3 letter words? We could put in three `\w` indicators followed by a space like the following:

```{r}
df_short |> 
  filter(str_detect(consults, '^\\w\\w\\w '))
```

There is a more efficient way to do this! We can use the curly brackets to indicate how many times to match a particular pattern.

```{r}
df_short |> 
  filter(str_detect(consults, '^\\w{3} '))
```

But what if we wanted to match either 1 or 3 letters? This is where defining enumeration with the curly braces `{}` becomes really useful. If we put in curly braces with one integer, we are telling R to match the preceding pattern that many times. If we add a comma and a second number like in the example below, we tell R to find strings starting with a `\w`, which must be repeated either **1, 2 or 3 times**:

```{r}
df_short |> 
  filter(str_detect(consults, '^\\w{1,3} '))
```

Do any lines start with 8 or 9 letter words?

```{r}
df_short |> 
  filter(str_detect(consults, '^\\w{8,9} '))
```

There are two other modifiers we should mention now that allow multiple matches of the preceding argument.

-   `*`: the prior character can be matched 0 or more times.
-   `+`: the prior character can be matched 1 or more times.

We take a number of REGEX ideas to do a better job of extracting age now:

-   We use the set `[0-9]` to look for numbers.
-   We use the `+` to indicate we'd like to match numbers 1 or more times.
-   The overall pattern to match is those numbers followed by "-year-old".
-   We use a capture group the finally extract only the numbers.

```{r}
df_str |> 
  mutate(pattern = str_extract(consults, '([0-9]+)-year-old', group=1)) |>
  select(pattern) |>
  filter(!is.na(pattern))
```

### 🛠️Coding Practice: Addresses

Let's practice our new REGEX skills on the addresses data.

Begin by extracting the house address into a new column called `house_address`. The house address should be the first sequence of numbers in an address. Print the house address alongside the full address.

```{r}
# extract the street address


```

Can you extract the zip code using an appropriate anchor?

```{r}
# extract the zip code using a REGEX anchor


```

Can you extract the state without explicitly typing in the state abbreviations?

```{r}
# extract the state


```

**END OF CODING PRACTICE**

------------------------------------------------------------------------

## Conclusion

------------------------------------------------------------------------

That wraps up our introduction to using the **stringr** package and REGEX!

## Answers to Questions

------------------------------------------------------------------------

### 🛠️Coding Practice: Addresses

```{r}
# import addresses data
df_address <- readLines('../data/addresses.txt') |> 
  as.data.frame() 

# rename the column
names(df_address) <- 'address' 

# print to the screen
df_address
```

```{r}
# create the [WA] and [nchar] columns
df_address2 <- df_address |>  
  mutate(WA = str_detect(address, 'WA'),          
         nchar = str_count(address)) 
df_address2
```

### 🛠️Coding Practice: Extracting City and State

```{r}
# extract city and state
df_address2 |>   
  mutate(city = str_extract(address, 'Seattle|Portland|San Dego|Great Falls'),        
         state = str_extract(address, 'WA|CA|OR|WY'))
```

### 🛠️Coding Practice: More with stringr

```{r}
# don't worry, we'll learn about the code below shortly! 
df_address3 <- df_address |>   
  mutate(zipcode = str_extract(address, '[0-9]{3,5}$'))  

# here's your starting data set, add new columns and print 
df_address3 |>   
  mutate(zipcode_clean = str_pad(zipcode, side='left', pad='0', width=5),
         address_clean = str_replace(address, 'San Dego', 'San Diego')) |>   
  select(zipcode, zipcode_clean, address_clean)
```

### 🛠️Coding Practice: Addresses

```{r}
# extract the street address
df_address |>  
  mutate(house_address = str_extract(address, '\\d{4,5}'))
```

```{r}
# extract the zip code using a REGEX anchor
df_address |>   
  mutate(zipcode = str_extract(address, '\\d{3,5}$'))
```

```{r}
# extract the state
df_address |>   
  mutate(state = str_extract(address, ', ([A-Z]{2}) \\d{3,5}$', group=1))
```
