---
title: "Exercise: Strings and Regular Expressions"
author: "Rian DeFaccio, for BH Analytics LLC"
date: "`r Sys.Date()`"
output: html_document
---

## Notebook Setup

------------------------------------------------------------------------

Before we jump into the questions, check your working directory and load any packages you think you'll need.

```{r}
getwd()
library(tidyverse)
```

## MTSamples Data

------------------------------------------------------------------------

We will now practice using R and tidyverse packages to convert some unstructured data into structured data. The image below shows what we are trying to do. If you think you can do it on your own, great! Go to the bottom of the notebook, add some code cells, and try it out. Otherwise, we will guide you through this process in the following cells.

![](images/mtsamples.png)

Begin by importing the text file "mtsamples2.txt" and converting the result into a data frame called `df_raw`. Also rename the single column to `text`. Print `df_raw` to the screen after you've created it.

```{r}
df_raw <- readLines('../data/mtsamples2.txt') |>
  as.data.frame()
colnames(df_raw) <- "text"
df_raw
```

Begin by removing rows that only contain empty strings. Feel free to overwrite `df_raw` in this step.

You can achieve this using one **stringr** function along with a REGEX.

```{r}
df_raw <- df_raw |>
  filter(str_detect(text, '^\\w'))
df_raw
```

We have a nice pattern here:

-   odd rows have the visit type and date
-   even rows have the description

Start by keeping only the odd rows in a new data frame called `df_visits`. To get a sequence of only odd numbers, you can use the `seq()` function with the following arguments: `seq(from, to, by)`.

```{r}
# practice generating a sequence of odd numbers 
odd_rows <- seq(1, nrow(df_raw), by=2)
odd_rows
```

```{r}
# now filter to keep only odd rows in the data frame
df_visits <- df_raw |> slice(odd_rows)
df_visits
```

Now split up `df_visits` so that we have the visit type and date in separate columns. Save this result into a data frame called `df_visits_final`. Discard any data/columns you don't want/need.

You should end up with two columns called `visit_type` and `date`.

```{r}
df_visits_final <- df_visits |>
  tidyr::separate(col=text, 
                  into=c('trash', 'visit_type', 'date'),
                  sep = ' - ') |>
  select(-trash)
df_visits_final
```

Finally, find a way to get the descriptions added to `df_visits1`.

```{r}
even_rows <- seq(2, nrow(df_raw), by=2)
even_rows
```

```{r}
df_visits_final$description <- df_raw[even_rows, 'text']
df_visits_final
```

**BONUS:** If you have the time, try to extract some or all of the following elements and save them as columns in `df_visits_final`.

-   name of the doctor for each visit
-   age of the patient
-   sex of the patient: This one is also a little tricky. You should scan for various ways male and female might be defined. Then, you could create a `male` column, a `female` column, and use the `case_when()` function from **dplyr** to combine the results into a single column `sex`.
-   `heart_related`: This one might be challenging. It should return `TRUE` if there is mention of a heart related issue. You'll need to look around for various types of patterns that might indicate the visit is related to a heart problem. You can check your work by seeing if the following rows are listed in your result: [1, 3, 4, 9-13]

These element are not present in all visits.

```{r}
df_visits_final <- df_visits_final |>
  mutate(doctor = str_extract(description, 'Dr. (\\w+)', group=1),
         patient_age = str_extract(description, '(\\d+)-year-old', group=1),
         male = str_detect(description, ' male|man'),
         female = str_detect(description, 'female|woman|Woman'),
         sex = case_when(
           female==TRUE ~ 'female',
           male==TRUE   ~ 'male'),
         heart_related = str_detect(description, 'heart|Atrial|chest pain|Coronary|coronary|CHF')) |>
  select(-male, -female)
# not a perfect algorithm for detecting heart-related issues, but it'll do!

df_visits_final |>
  select(visit_type, date, doctor, sex, patient_age, heart_related)
```

Finally, save your work as a CSV file in the data folder called *mtsamples_rjd.csv*. (Insert your own initials)

Don't include row numbers in your CSV file.

Go and examine the file you made to make sure it wrote properly.

```{r}
write.csv(df_visits_final, '../data/mtsamples_rjd.csv', row.names=FALSE)
```
