New features in dplyr 1.1.0, plus an intro to ivs

Slide 1

Slide 1 text

dplyr 1.1.0 Davis Vaughan January 2023

Slide 2

Slide 2 text

dplyr 1.1.0 ivs

Slide 3

Slide 3 text

Per-operation grouping Joins New vector functions

Slide 4

Slide 4 text

🔗 dplyr.tidyverse.org # A tibble: 6 × 3 company year revenue 1 A 2019 20 2 A 2019 50 3 A 2020 4 4 B 2021 10 5 B 2023 12 6 B 2023 18

Slide 5

Slide 5 text

🔗 dplyr.tidyverse.org transactions # A tibble: 6 × 3 company year revenue 1 A 2019 20 2 A 2019 50 3 A 2020 4 4 B 2021 10 5 B 2023 12 6 B 2023 18

Slide 6

Slide 6 text

🔗 dplyr.tidyverse.org Total yearly revenue of each company? transactions # A tibble: 6 × 3 company year revenue 1 A 2019 20 2 A 2019 50 3 A 2020 4 4 B 2021 10 5 B 2023 12 6 B 2023 18

Slide 7

Slide 7 text

🔗 dplyr.tidyverse.org transactions %>% group_by(company, year) %>% summarise(revenue = sum(revenue)) #> `summarise()` has grouped output by 'company'. #> You can override using the `.groups` argument. #> # A tibble: 4 × 3 #> # Groups: company [2] #> company year revenue #> #> 1 A 2019 70 #> 2 A 2020 4 #> 3 B 2021 10 #> 4 B 2023 30

Slide 8

Slide 8 text

🔗 dplyr.tidyverse.org transactions[, .(revenue = sum(revenue)), by = .(company, year)]

Slide 9

Slide 9 text

🔗 dplyr.tidyverse.org transactions %>% group_by(company, year) %>% summarise(revenue = sum(revenue)) transactions %>% summarise(revenue = sum(revenue), .by = c(company, year))

Slide 10

Slide 10 text

🔗 dplyr.tidyverse.org transactions %>% summarise(revenue = sum(revenue), .by = c(company, year)) #> # A tibble: 4 × 3 #> company year revenue #> #> 1 A 2019 70 #> 2 A 2020 4 #> 3 B 2021 10 #> 4 B 2023 30

Slide 11

Slide 11 text

.by group_by() Per-operation grouping Persistent grouping Selects variables using tidy-select Computes expressions using data-masking (i.e., like mutate())

Slide 12

Slide 12 text

🔗 dplyr.tidyverse.org transactions %>% slice_max(revenue, n = 2, by = company) #> # A tibble: 4 × 3 #> company year revenue #> #> 1 A 2019 50 #> 2 A 2019 20 #> 3 B 2023 18 #> 4 B 2023 12 Top two transactions per company?

Slide 13

Slide 13 text

mutate() summarise() filter() slice_*()

Slide 14

Slide 14 text

So what about group_by()?

Slide 15

Slide 15 text

Joins

Slide 16

Slide 16 text

🔗 dplyr.tidyverse.org # A tibble: 2 × 2 id name 1 A Apple 2 B Boeing companies transactions # A tibble: 6 × 3 company year revenue 1 A 2019 20 2 A 2019 50 3 A 2020 4 4 B 2021 10 5 B 2023 12 6 B 2023 18

Slide 17

Slide 17 text

🔗 dplyr.tidyverse.org inner_join(transactions, companies, by = c(company = "id")) # A tibble: 2 × 2 id name 1 A Apple 2 B Boeing companies transactions # A tibble: 6 × 3 company year revenue 1 A 2019 20 2 A 2019 50 3 A 2020 4 4 B 2021 10 5 B 2023 12 6 B 2023 18

Slide 18

Slide 18 text

🔗 dplyr.tidyverse.org inner_join(transactions, companies, join_by(company == id)) # A tibble: 2 × 2 id name 1 A Apple 2 B Boeing companies transactions # A tibble: 6 × 3 company year revenue 1 A 2019 20 2 A 2019 50 3 A 2020 4 4 B 2021 10 5 B 2023 12 6 B 2023 18

Slide 19

Slide 19 text

🔗 dplyr.tidyverse.org # A tibble: 3 × 3 id since name 1 A 1976 Apple 2 B 1916 Boeing 3 B 2022 Amazon companies transactions # A tibble: 6 × 3 company year revenue 1 A 2019 20 2 A 2019 50 3 A 2020 4 4 B 2021 10 5 B 2023 12 6 B 2023 18

Slide 20

Slide 20 text

Slide 21

Slide 21 text

🔗 dplyr.tidyverse.org inner_join(transactions, companies, join_by(company == id), keep = TRUE) #> # A tibble: 9 × 6 #> company year revenue id since name #> #> 1 A 2019 20 A 1976 Apple #> 2 A 2019 50 A 1976 Apple #> 3 A 2020 4 A 1976 Apple #> 4 B 2021 10 B 1916 Boeing #> 5 B 2021 10 B 2022 Amazon #> 6 B 2023 12 B 1916 Boeing #> 7 B 2023 12 B 2022 Amazon #> 8 B 2023 18 B 1916 Boeing #> 9 B 2023 18 B 2022 Amazon #> Warning: #> Each row in `x` is expected to match at most 1 row in `y`. #> ℹ Row 4 of `x` matches multiple rows. #> ℹ If multiple matches are expected, set `multiple = "all"` #> to silence this warning.

Slide 22

Slide 22 text

Slide 23

Slide 23 text

Slide 24

Slide 24 text

🔗 dplyr.tidyverse.org transactions # A tibble: 6 × 3 company year revenue 1 A 2019 20 2 A 2019 50 3 A 2020 4 4 B 2021 10 5 B 2023 12 6 B 2023 18 # A tibble: 3 × 3 id since name 1 A 1976 Apple 2 B 1916 Boeing 3 B 2022 Amazon companies

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Slide 27

Slide 27 text

Slide 28

Slide 28 text

Slide 29

Slide 29 text

🔗 dplyr.tidyverse.org inner_join(transactions, companies, join_by(company == id, year >= since)) #> # A tibble: 8 × 5 #> company year revenue since name #> #> 1 A 2019 20 1976 Apple #> 2 A 2019 50 1976 Apple #> 3 A 2020 4 1976 Apple #> 4 B 2021 10 1916 Boeing #> 5 B 2023 12 1916 Boeing #> 6 B 2023 12 2022 Amazon #> 7 B 2023 18 1916 Boeing #> 8 B 2023 18 2022 Amazon

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Slide 32

Slide 32 text

🔗 dplyr.tidyverse.org 2023 >= 1916 is TRUE 2023 >= 2022 is TRUE transactions # A tibble: 6 × 3 company year revenue 1 A 2019 20 2 A 2019 50 3 A 2020 4 4 B 2021 10 5 B 2023 12 6 B 2023 18 # A tibble: 3 × 3 id since name 1 A 1976 Apple 2 B 1916 Boeing 3 B 2022 Amazon companies

Slide 33

Slide 33 text

Slide 34

Slide 34 text

🔗 dplyr.tidyverse.org 2023 >= 1916 is TRUE 2023 >= 2022 is TRUE

Slide 35

Slide 35 text

🔗 dplyr.tidyverse.org 2023 >= 1916 is TRUE 2023 >= 2022 is TRUE 2022 is closer to 2023 than 1916 (107 years) (1 year)

Slide 36

Slide 36 text

🔗 dplyr.tidyverse.org inner_join(transactions, companies, join_by(company == id, closest(year >= since))) #> # A tibble: 6 × 5 #> company year revenue since name #> #> 1 A 2019 20 1976 Apple #> 2 A 2019 50 1976 Apple #> 3 A 2020 4 1976 Apple #> 4 B 2021 10 1916 Boeing #> 5 B 2023 12 2022 Amazon #> 6 B 2023 18 2022 Amazon

Slide 37

Slide 37 text

Slide 38

Slide 38 text

🔗 dplyr.tidyverse.org •Equality Joins •join_by(id, a == b) •Inequality Joins •join_by(id, date >= since) •Rolling Joins •join_by(id, closest(date >= since)) •Overlap Joins •join_by(id, between(date, y_lower, y_upper)) •join_by(id, overlaps(x_lower, x_upper, y_lower, y_upper)) •join_by(id, within(x_lower, x_upper, y_lower, y_upper))

Slide 39

Slide 39 text

Slide 40

Slide 40 text

consecutive_id() case_match()

Slide 41

Slide 41 text

🔗 dplyr.tidyverse.org # A tibble: 8 × 2 name text 1 Hadley I'll never learn Python. 2 Davis But aren't you speaking at PyCon? 3 Hadley So? 4 Hadley That doesn't influence my decision. 5 Hadley I'm not budging! 6 Mara Typical, Hadley. Stubborn as always. 7 Davis Fair enough! 8 Davis Let's move on. transcript

Slide 42

Slide 42 text

Slide 43

Slide 43 text

Slide 44

Slide 44 text

🔗 dplyr.tidyverse.org transcript %>% mutate(id = consecutive_id(name)) #> # A tibble: 8 × 3 #> name text id #> #> 1 Hadley I'll never learn Python. 1 #> 2 Davis But aren't you speaking at PyCon? 2 #> 3 Hadley So? 3 #> 4 Hadley That doesn't influence my decision. 3 #> 5 Hadley I'm not budging! 3 #> 6 Mara Typical, Hadley. Stubborn as always. 4 #> 7 Davis Fair enough! 5 #> 8 Davis Let's move on. 5

Slide 45

Slide 45 text

🔗 dplyr.tidyverse.org transcript %>% mutate(id = consecutive_id(name)) %>% summarise( text = stringr::str_flatten(text, collapse = " "), .by = c(id, name) ) #> # A tibble: 5 × 3 #> id name text #> #> 1 1 Hadley I'll never learn Python. #> 2 2 Davis But aren't you speaking at PyCon? #> 3 3 Hadley So? That doesn't influence my decision. I'm not budging! #> 4 4 Mara Typical, Hadley. Stubborn as always. #> 5 5 Davis Fair enough! Let's move on.

Slide 46

Slide 46 text

🔗 dplyr.tidyverse.org x <- c(1, 12, -5, 6, -2, NA, 0) case_when( x >= 10 ~ "large", x >= 0 ~ "small", x < 0 ~ NA )

Slide 47

Slide 47 text

🔗 dplyr.tidyverse.org x <- c(1, 12, -5, 6, -2, NA, 0) case_when( x >= 10 ~ "large", x >= 0 ~ "small", x < 0 ~ NA ) #> Error: `NA` must be , not . 😭 NA_character_

Slide 48

Slide 48 text

🔗 dplyr.tidyverse.org x <- c(1, 12, -5, 6, -2, NA, 0) case_when( x >= 10 ~ "large", x >= 0 ~ "small", x < 0 ~ NA ) #> [1] "small" "large" NA "small" NA NA "small"

Slide 49

Slide 49 text

🔗 dplyr.tidyverse.org x <- c(1, 12, -5, 6, -2, NA, 0) case_when( x >= 10 ~ "large", x >= 0 ~ "small", is.na(x) ~ "missing", TRUE ~ "other" ) #> [1] "small" "large" "other" "small" "other" "missing" "small"

Slide 50

Slide 50 text

🔗 dplyr.tidyverse.org x <- c(1, 12, -5, 6, -2, NA, 0) case_when( x >= 10 ~ "large", x >= 0 ~ "small", is.na(x) ~ "missing", .default = "other" ) #> [1] "small" "large" "other" "small" "other" "missing" "small"

Slide 51

Slide 51 text

🔗 dplyr.tidyverse.org x <- c("USA", "Canada", "France", "UK", "China", NA, "Mexico", "Russia") case_when( x %in% c("USA", "Canada", "Mexico") ~ "North America", x %in% c("France", "UK") ~ "Europe", x %in% "China" ~ "Asia" ) #> [1] "North America" "North America" "Europe" "Europe" "Asia" NA #> [7] "North America" NA

Slide 52

Slide 52 text

Slide 53

Slide 53 text

🔗 dplyr.tidyverse.org x <- c("USA", "Canada", "France", "UK", "China", NA, "Mexico", "Russia") case_match( x, c("USA", "Canada", "Mexico") ~ "North America", c("France", "UK") ~ "Europe", "China" ~ "Asia" ) #> [1] "North America" "North America" "Europe" "Europe" "Asia" NA #> [7] "North America" NA

Slide 54

Slide 54 text

🔗 dplyr.tidyverse.org x <- c("USA", "Canada", "France", "UK", "China", NA, "Mexico", "Russia") case_match( x, c("USA", "Canada", "Mexico") ~ "North America", c("France", "UK") ~ "Europe", "China" ~ "Asia", NA ~ "missing", .default = "unknown" ) #> [1] "North America" "North America" "Europe" "Europe" "Asia" "missing" #> [7] "North America" "unknown"

Slide 55

Slide 55 text

ivs

Slide 56

Slide 56 text

🔗 davisvaughan.github.io/ivs # A tibble: 7 × 3 name start end 1 Davis 2014 2016 2 Greg 2015 2017 3 Greg 2019 2021 4 Ben 2019 2020 5 Robby 2016 2017 6 Robby 2019 2021 7 Robby 2022 2023 apartment

Slide 57

Slide 57 text

🔗 davisvaughan.github.io/ivs # A tibble: 7 × 3 name start end 1 Davis 2014 2016 2 Greg 2015 2017 3 Greg 2019 2021 4 Ben 2019 2020 5 Robby 2016 2017 6 Robby 2019 2021 7 Robby 2022 2023 apartment # A tibble: 7 × 2 name range > 1 Davis [2014, 2016) 2 Greg [2015, 2017) 3 Greg [2019, 2021) 4 Ben [2019, 2020) 5 Robby [2016, 2017) 6 Robby [2019, 2021) 7 Robby [2022, 2023) apartment iv(start, end)

Slide 58

Slide 58 text

🔗 davisvaughan.github.io/ivs

Slide 59

Slide 59 text

🔗 davisvaughan.github.io/ivs Number of people in the apartment per year?

Slide 60

Slide 60 text

🔗 davisvaughan.github.io/ivs 1 2 2 0 3 0 2 0 1 Number of people in the apartment per year?

Slide 61

Slide 61 text

🔗 davisvaughan.github.io/ivs rolling %>% mutate(count = iv_count_between(year, apartment$range)) #> # A tibble: 9 × 2 #> year count #> #> 1 2014 1 #> 2 2015 2 #> 3 2016 2 #> 4 2017 0 #> 5 2018 0 #> 6 2019 3 #> 7 2020 2 #> 8 2021 0 #> 9 2022 1 # A tibble: 7 × 2 name range > 1 Davis [2014, 2016) 2 Greg [2015, 2017) 3 Greg [2019, 2021) 4 Ben [2019, 2020) 5 Robby [2016, 2017) 6 Robby [2019, 2021) 7 Robby [2022, 2023) apartment

Slide 62

Slide 62 text

🔗 davisvaughan.github.io/ivs When was the apartment occupied?

Slide 63

Slide 63 text

🔗 davisvaughan.github.io/ivs When was the apartment occupied?

Slide 64

Slide 64 text

🔗 davisvaughan.github.io/ivs apartment %>% reframe(range = iv_groups(range)) #> # A tibble: 3 × 1 #> range #> > #> 1 [2014, 2017) #> 2 [2019, 2021) #> 3 [2022, 2023) # A tibble: 7 × 2 name range > 1 Davis [2014, 2016) 2 Greg [2015, 2017) 3 Greg [2019, 2021) 4 Ben [2019, 2020) 5 Robby [2016, 2017) 6 Robby [2019, 2021) 7 Robby [2022, 2023) apartment

Slide 65

Slide 65 text

🔗 davisvaughan.github.io/ivs When was the apartment empty?

Slide 66

Slide 66 text

🔗 davisvaughan.github.io/ivs When was the apartment empty?

Slide 67

Slide 67 text

🔗 davisvaughan.github.io/ivs apartment %>% reframe(range = iv_set_complement(range)) #> # A tibble: 2 × 1 #> range #> > #> 1 [2017, 2019) #> 2 [2021, 2022) # A tibble: 7 × 2 name range > 1 Davis [2014, 2016) 2 Greg [2015, 2017) 3 Greg [2019, 2021) 4 Ben [2019, 2020) 5 Robby [2016, 2017) 6 Robby [2019, 2021) 7 Robby [2022, 2023) apartment

Slide 68

Slide 68 text

dplyr.tidyverse.org davisvaughan.github.io/ivs Thank you!