TokyoR#103_DataProcessing

Slide 1

Slide 1 text

#103 @kilometer00 2023.01.21 BeginneR Session Data processing & visualization

Slide 2

Slide 2 text

Who！？ Who?

Slide 3

Slide 3 text

Who！？名前：三村 @kilometer 職業：ポスドク (こうがくはくし) 専⾨：⾏動神経科学(霊⻑類) 脳イメージング医療システム⼯学 R歴： ~ 10年ぐらい流⾏: ペーパークラフト

Slide 4

Slide 4 text

宣伝!!（書籍の翻訳に参加しました。）絶賛販売中！

Slide 5

Slide 5 text

BeginneR Session

Slide 6

Slide 6 text

BeginneR

Slide 7

Slide 7 text

Before After BeginneR Session BeginneR BeginneR

Slide 8

Slide 8 text

BeginneR Advanced Hoxo_m If I have seen further it is by standing on the shoulders of Giants. -- Sir Isaac Newton, 1676

Slide 9

Slide 9 text

#103 @kilometer00 2023.01.21 BeginneR Session Data processing & visualization

Slide 10

Slide 10 text

import Tidy Transform Visualize Model Communicate Modified from “R for Data Science”, H. Wickham, 2017 Data Science ① ②

Slide 11

Slide 11 text

import Tidy Transform Visualize Model Communicate Modiﬁed from “R for Data Science”, H. Wickham, 2017 preprocessing Data processing Data science Data Science

Slide 12

Slide 12 text

import Tidy Transform Visualize Model Communicate Modified from “R for Data Science”, H. Wickham, 2017 preprocessing Data science Data Observa?on Hypothesis feedback Data processing Data Science

Slide 13

Slide 13 text

import Tidy Transform Visualize Model Communicate Modiﬁed from “R for Data Science”, H. Wickham, 2017 preprocessing Data science Data Observation Hypothesis feedback Data processing Narra/ve of data

Slide 14

Slide 14 text

import Tidy Transform Visualize Model Communicate Modiﬁed from “R for Data Science”, H. Wickham, 2017 preprocessing Data science Data Observa?on Hypothesis Narra/ve of data feedback Data processing

Slide 15

Slide 15 text

import Tidy Transform Visualize Model Communicate Modiﬁed from “R for Data Science”, H. Wickham, 2017 Data processing Data Science

Slide 16

Slide 16 text

raed_csv() write_csv() Table Data Wide form Long form pivot_longer() Nested form pivot_wider() Plot group_nest() unnest() {ggplot2} {patchwork} Image Files ggsave() Data Processing

Slide 17

Slide 17 text

data.frame tibble raed_csv() write_csv() Table Data Wide form Long form pivot_longer() Nested form pivot_wider() Plot group_nest() unnest() {ggplot2} {patchwork} Image Files ggsave() Data Processing

Slide 18

Slide 18 text

data.frame

Slide 19

Slide 19 text

vector in Excel

Slide 20

Slide 20 text

vector in R in Excel pre <- c(1, 2, 3, 4, 5) post <- pre * 5 > pre [1] 1 2 3 4 5 > post [1] 5 10 15 20 25

Slide 21

Slide 21 text

vector vec1 <- c(1, 2, 3, 4, 5) vec2 <- 1:5 vec3 <- seq(from = 1, to = 5, by = 1) > vec1 [1] 1 2 3 4 5 > vec2 [1] 1 2 3 4 5 > vec3 [1] 1 2 3 4 5

Slide 22

Slide 22 text

vector vec1 <- seq(from = 1, to = 5, by = 1) vec2 <- seq(1, 5, 1) > vec1 [1] 1 2 3 4 5 > vec2 [1] 1 2 3 4 5

Slide 23

Slide 23 text

> ?seq vector seq{base} Sequence Generation Description Generate regular sequences. seq is a standard generic with a default method. … Usage seq(...) ## Default S3 method: seq(from = 1, to = 1, by = ((to - from)/(length.out - 1)), length.out = NULL, along.with = NULL, ...)

Slide 24

Slide 24 text

vector vec1 <- rep(1:3, times = 2) vec2 <- rep(1:3, each = 2) vec3 <- rep(1:3, times = 2, each = 2) > vec1 [1] 1 2 3 1 2 3 > vec2 [1] 1 1 2 2 3 3 > vec3 [1] 1 1 2 2 3 3 1 1 2 2 3 3

Slide 25

Slide 25 text

vector vec1 <- 11:15 > vec1 [1] 11 12 13 14 15 > vec1[1] [1] 11 > vec1[3:5] [1] 13 14 15 > vec1[c(1:2, 5)] [1] 11 12 15

Slide 26

Slide 26 text

list list1 <- list(1:6, 11:15, c("a", "b", "c")) > list1 [[1]] [1] 1 2 3 4 5 6 [[2]] [1] 11 12 13 14 15 [[3]] [1] "a" "b" "c"

Slide 27

Slide 27 text

list list1 <- list(1:6, 11:15, c("a", "b", "c")) > list1[[1]] [1] 1 2 3 4 5 6 > list1[[3]][2:3] [1] "b" "c" > list1[[2]] * 3 [1] 33 36 39 42 45

Slide 28

Slide 28 text

named list list2 <- list(A = 1:6, B = 11:15, C = c("a", "b", "c")) > list2 $A [1] 1 2 3 4 5 6 $B [1] 11 12 13 14 15 $C [1] "a" "b" "c"

Slide 29

Slide 29 text

> list2$A [1] 1 2 3 4 5 6 > list2$C[2:3] [1] "b" "c" > list2$B * 3 [1] 33 36 39 42 45 named list list2 <- list(A = 1:6, B = 11:15, C = c("a", "b", "c"))

Slide 30

Slide 30 text

list1 <- list(1:6, 11:15, c("a", "b", "c")) > class(list1) [1] "list" > names(list1) NULL list2 <- list(A = 1:6, B = 11:15, C = c("a", "b", "c")) > class(list2) [1] "list" > names(list2) [1] "A" "B" "C" named list list

Slide 31

Slide 31 text

list3 <- list(A = 1:3, B = 11:13) > class(list3) [1] "list" > names(list3) [1] "A" "B" df1 <- data.frame(A = 1:3, B = 11:13) > class(df1) [1] "data.frame" > names(df1) [1] "A" "B" named list & data.frame

Slide 32

Slide 32 text

> str(list3) List of 2 $ A: int [1:3] 1 2 3 $ B: int [1:3] 11 12 13 > str(df1) 'data.frame': 3 obs. of 2 variables: $ A: int 1 2 3 $ B: int 11 12 13 list3 <- list(A = 1:3, B = 11:13) df1 <- data.frame(A = 1:3, B = 11:13) named list & data.frame

Slide 33

Slide 33 text

> list3 $A [1] 1 2 3 $B [1] 11 12 13 > df1 A B 1 1 11 2 2 12 3 3 13 named list & data.frame

Slide 34

Slide 34 text

data.frame vs. matrix A B 1 1 11 2 2 12 3 3 13 [,1] [,2] [1,] 1 11 [2,] 2 12 [3,] 3 13 df1 <- data.frame(A = 1:3, B = 11:13) > str(mat1) int [1:3, 1:2] 1 2 3 11 12 13 > str(df1) 'data.frame': 3 obs. of 2 vars.: $ A: int 1 2 3 $ B: int 11 12 13 mat1 <- matrix(c(1:3, 11:13), nrow = 3, ncol = 2)

Slide 35

Slide 35 text

data.frame variables observa*on

Slide 36

Slide 36 text

data.frame tibble raed_csv() write_csv() Table Data Wide form Long form pivot_longer() Nested form pivot_wider() Plot group_nest() unnest() {ggplot2} {patchwork} Image Files ggsave() Data Processing

Slide 37

Slide 37 text

> anscombe x1 x2 x3 x4 y1 y2 y3 y4 1 10 10 10 8 8.04 9.14 7.46 6.58 2 8 8 8 8 6.95 8.14 6.77 5.76 3 13 13 13 8 7.58 8.74 12.74 7.71 4 9 9 9 8 8.81 8.77 7.11 8.84 5 11 11 11 8 8.33 9.26 7.81 8.47 6 14 14 14 8 9.96 8.10 8.84 7.04 7 6 6 6 8 7.24 6.13 6.08 5.25 8 4 4 4 19 4.26 3.10 5.39 12.50 9 12 12 12 8 10.84 9.13 8.15 5.56 10 7 7 7 8 4.82 7.26 6.42 7.91 11 5 5 5 8 5.68 4.74 5.73 6.89 Wide form data

Slide 38

Slide 38 text

> df tag x1 x2 x3 x4 y1 y2 y3 y4 1 1 10 10 10 8 8.04 9.14 7.46 6.58 2 2 8 8 8 8 6.95 8.14 6.77 5.76 3 3 13 13 13 8 7.58 8.74 12.74 7.71 4 4 9 9 9 8 8.81 8.77 7.11 8.84 5 5 11 11 11 8 8.33 9.26 7.81 8.47 6 6 14 14 14 8 9.96 8.10 8.84 7.04 Wide form data df <- rownames_to_column( anscombe, var = "tag" )

Slide 39

Slide 39 text

Wide form → Long form data df_long_1 <- pivot_longer( data = df, cols = !tag ) df_long_2 <- pivot_longer( data = df, cols = !tag, names_to = c(".value", "key"), names_pattern = c("(.)(.)")

Slide 40

Slide 40 text

Long form → Wide form data pivot_wider( data = df_long_1, values_from = value, names_from = name ) pivot_wider( data = df_long_2, values_from = c(x, y), names_from = name )

Slide 41

Slide 41 text

data.frame / tibble raed_csv() write_csv() Table Data Wide form Long form pivot_longer() pivot_wider() Plot {ggplot2} Image Files ggsave() Data Processing

Slide 42

Slide 42 text

raed_csv() write_csv() Table Data Wide form Long form pivot_longer() pivot_wider() Plot {ggplot2} Image Files ggsave() Data Processing Long form Long form Long form Long form Long form Long form Long form Long form data.frame / -bble

Slide 43

Slide 43 text

vignette("dplyr")

Slide 44

Slide 44 text

It (dplyr) provides simple “verbs” to help you translate your thoughts into code. func>ons that correspond to the most common data manipula>on tasks Introduc6on to dplyr h"ps://cran.r-project.org/web/packages/dplyr/vigne"es/dplyr.html WFSCT {dplyr}

Slide 45

Slide 45 text

dplyrは、あなたの考えをコードに翻訳するための【動詞】を提供する。データ操作における基本のキを、シンプルに実⾏できる関数 (群) Introduction to dplyr https://cran.r-project.org/web/packages/dplyr/vignettes/dplyr.html WFSCT {dplyr} ※ かなり意訳

Slide 46

Slide 46 text

1. mutate() 2. filter() 3. select() 4. group_by() 5. summarize() 6. left_join() 7. arrange() Data.frame manipula?on

Slide 47

Slide 47 text

1. mutate() 2. filter() 3. select() 4. group_by() 5. summarize() 6. left_join() 7. arrange() Data.frame manipula?on 0. %>%

Slide 48

Slide 48 text

1JQFBMHFCSB X %>% f X %>% f(y) X %>% f %>% g X %>% f(y, .) f(X) f(X, y) g(f(X)) f(y, X) %>% {magri8r} 「dplyr再⼊⾨（基本編）」yutanihilaCon h"ps://speakerdeck.com/yutannihila6on/dplyrzai-ru-men-ji-ben-bian

Slide 49

Slide 49 text

① ② ③ ④ lift take pour put Bring milk from the kitchen!

Slide 50

Slide 50 text

① lift Bring milk from the kitchen! lift(Robot, glass, table) -> Robot' take ② take(Robot', fridge, milk) -> Robot''

Slide 51

Slide 51 text

Bring milk from the kitchen! Robot' <- lift(Robot, glass, table) Robot'' <- take(Robot', fridge, milk) Robot''' <- pour(Robot'', milk, glass) result <- put(Robot''', glass, table) result <- Robot %>% lift(glass, table) %>% take(fridge, milk) %>% pour(milk, glass) %>% put(glass, table) by using pipe, # ① # ② # ③ # ④ # ① # ② # ③ # ④

Slide 52

Slide 52 text

The Pdyverse style guides h"ps://style.;dyverse.org/syntax.html#object-names "There are only two hard things in Computer Science: cache invalida:on and naming things"

Slide 53

Slide 53 text

Slide 54

Slide 54 text

Robot' <- lift(Robot, glass, table) Robot'' <- take(Robot', fridge, milk) Robot''' <- pour(Robot'', milk, glass) result <- put(Robot''', glass, table) result <- Robot %>% lift(glass, table) %>% take(fridge, milk) %>% pour(milk, glass) %>% put(glass, table) by using pipe, # ① # ② # ③ # ④ # ① # ② # ③ # ④ Thinking Reading Bring milk from the kitchen!

Slide 55

Slide 55 text

Programing Write Run Read Think Write Run Read Think Communicate Share

Slide 56

Slide 56 text

1JQFBMHFCSB X %>% f X %>% f(y) X %>% f %>% g X %>% f(y, .) f(X) f(X, y) g(f(X)) f(y, X) %>% {magri8r} 「dplyr再⼊⾨（基本編）」yutanihilation https://speakerdeck.com/yutannihilation/dplyrzai-ru-men-ji-ben-bian

Slide 57

Slide 57 text

1. mutate() 2. filter() 3. select() 4. group_by() 5. summarize() 6. left_join() 7. arrange() Data.frame manipula?on 0. %>% ✔

Slide 58

Slide 58 text

WFSCT {dplyr} mutate # カラムの追加 + mutate(dat, C = fun(A, B))

Slide 59

Slide 59 text

WFSCT {dplyr} mutate # カラムの追加 + dat %>% mutate(C = fun(A, B))

Slide 60

Slide 60 text

WFSCT {dplyr} filter # 行の絞り込み dat %>% filter(tag %in% c(1, 3, 5))

Slide 61

Slide 61 text

ブール演算⼦ Boolean Algebra A == B A != B George Boole 1815 - 1864 A | B A & B A %in% B # equal to # not equal to # or # and # is A in B? wikipedia

Slide 62

Slide 62 text

"a" != "b" # is A in B? ブール演算⼦ Boolean Algebra [1] TRUE 1 %in% 10:100 # is A in B? [1] FALSE

Slide 63

Slide 63 text

George Boole 1815 - 1864 A Class-Room Introduc;on to Logic h"ps://niyamaklogic.wordpress.com/c ategory/laws-of-thoughts/ Mathematician Philosopher &

Slide 64

Slide 64 text

WFSCT {dplyr} select # カラムの選択 dat %>% select(tag, B)

Slide 65

Slide 65 text

WFSCT {dplyr} select # カラムの選択 dat %>% select("tag", "B")

Slide 66

Slide 66 text

WFSCT {dplyr} select # カラムの選択 dat %>% select("tag", "B") dat %>% select(tag, B)

Slide 67

Slide 67 text

WFSCT {dplyr} # Select help func>ons starts_with("s") ends_with("s") contains("se") matches("^.e") one_of(c(”tag", ”B")) everything() hEps://kazutan.github.io/blog/2017/04/dplyr-select-memo/ 「dplyr::selectの活⽤例メモ」kazutan

Slide 68

Slide 68 text

1. mutate() 2. filter() 3. select() 4. group_by() 5. summarize() 6. left_join() 7. arrange() Data.frame manipula?on 0. %>% ✔ ✔ ✔ ✔

Slide 69

Slide 69 text

(SBNNBSPGEBUBNBOJQVMBUJPO By constraining your options, it helps you think about your data manipulation challenges. Introduc6on to dplyr hEps://cran.r-project.org/web/packages/dplyr/vigneEes/dplyr.html

Slide 70

Slide 70 text

選択肢を制限することで、データ解析のステップをシンプルに考えられますヨ。（めっちゃ意訳） Introduc6on to dplyr hEps://cran.r-project.org/web/packages/dplyr/vigneEes/dplyr.html ※ まさに意訳 (SBNNBSPGEBUBNBOJQVMBUJPO

Slide 71

Slide 71 text

より多くの制約を課す事で、魂の⾜枷から、より⾃由になる。 Igor Stravinsky И8горь Ф Страви́нский The more constraints one imposes, the more one frees one's self of the chains that shackle the spirit. 1882 - 1971 ※ 割と意訳

Slide 72

Slide 72 text

import Tidy Transform Visualize Model Communicate Modiﬁed from “R for Data Science”, H. Wickham, 2017 Data Science ① ②

Slide 73

Slide 73 text

Programing Write Run Read Think Write Run Read Think Communicate Share

Slide 74

Slide 74 text

Text Image Information Intention Data decode encode Data analysis feedback ≠

Slide 75

Slide 75 text

Text Image First, A. Next, B. Then C. Finally D. >me Intention encode "Frozen" structure A B C D Nme value α β

Slide 76

Slide 76 text

Encode Apple （Real） Apple （Information） Decode

Slide 77

Slide 77 text

Divergence Real Info. Data Apple = 1 Encoding

Slide 78

Slide 78 text

Apple Encode Fruit Red 1 (image) Real Information

Slide 79

Slide 79 text

Apple Encode Fruit Red 1 (image) Real Information channel

Slide 80

Slide 80 text

写像 (mapping) 𝑓: 𝑋 → 𝑌 𝑋 𝑌 ある情報の集合の要素を、別の情報の集合のただ1つの要素に対応づけるプロセス

Slide 81

Slide 81 text

写像 (mapping) 𝑚𝑎𝑝𝑝𝑖𝑛𝑔 𝐷𝑎𝑡𝑎 𝐼𝑚𝑎𝑔𝑒

Slide 82

Slide 82 text

𝑋 𝑌 𝑦! 𝑥! 𝑦" 𝑥" 𝑋 𝑌 𝑥! 𝑥" 𝑦! 𝑦" 可視化 ⊂ 写像 mapping

Slide 83

Slide 83 text

𝑋 𝑌 𝑦! 𝑥! 𝑦" 𝑥" 𝑋 𝑌 𝑥! 𝑥" 𝑦! 𝑦" 可視化 ⊂ 写像 mapping x axis, y axis, color, ﬁll, shape, linetype, alpha… aesthetic channels

Slide 84

Slide 84 text

Apple Encode Fruit Red 1 (image) Real Information channel

Slide 85

Slide 85 text

𝑋 𝑌 𝑦! 𝑥! 𝑦" 𝑥" 𝑋 𝑌 𝑥! 𝑥" 𝑦! 𝑦" mapping x axis, y axis, color, ﬁll, shape, linetype, alpha… aesthetic channels data ggplot2 package

Slide 86

Slide 86 text

ggplot2 # install.packages("tidyverse") library(tidyverse) dat <- data.frame(a = 1:3, b = 8:10) Attach package Simple example > dat a b 1 1 8 2 2 9 3 3 10

Slide 87

Slide 87 text

dat <- data.frame(a = 1:3, b = 8:10) ggplot(data = dat) + geom_point(mapping = aes(x = a, y = b)) ggplot2

Slide 88

Slide 88 text

dat <- data.frame(a = 1:3, b = 8:10) ggplot(data = dat) + geom_point(mapping = aes(x = a, y = b)) 𝑋 𝑌 𝑦! 𝑥! 𝑦" 𝑥" 𝑋 𝑌 𝑥! 𝑥" 𝑦! 𝑦" mapping x axis, y axis, color, ﬁll, shape, linetype, alpha… aesthetic channels data ggplot2

Slide 89

Slide 89 text

dat <- data.frame(a = 1:3, b = 8:10) ggplot(data = dat) + geom_point(mapping = aes(x = a, y = b)) + geom_path(mapping = aes(x = a, y = b))

Slide 90

Slide 90 text

dat <- data.frame(a = 1:3, b = 8:10) ggplot(data = dat, mapping = aes(x = a, y = b)) + geom_point() + geom_path() inheritance

Slide 91

Slide 91 text

dat <- data.frame(a = 1:3, b = 8:10) ggplot(data = dat) + aes(x = a, y = b) + geom_point() + geom_path()

Slide 92

Slide 92 text

dat <- data.frame(a = 1:3, b = 8:10) ggplot(data = dat) + aes(x = a, y = b) + geom_point() + geom_path()

Slide 93

Slide 93 text

dat <- data.frame(a = 1:3, b = 8:10) g <- ggplot(data = dat) + aes(x = a, y = b)) + geom_point() g + geom_path()

Slide 94

Slide 94 text

Anscombe's quartet > anscombe x1 x2 x3 x4 y1 y2 y3 y4 1 10 10 10 8 8.04 9.14 7.46 6.58 2 8 8 8 8 6.95 8.14 6.77 5.76 3 13 13 13 8 7.58 8.74 12.74 7.71 4 9 9 9 8 8.81 8.77 7.11 8.84 5 11 11 11 8 8.33 9.26 7.81 8.47 6 14 14 14 8 9.96 8.10 8.84 7.04 7 6 6 6 8 7.24 6.13 6.08 5.25 8 4 4 4 19 4.26 3.10 5.39 12.50 9 12 12 12 8 10.84 9.13 8.15 5.56 10 7 7 7 8 4.82 7.26 6.42 7.91 11 5 5 5 8 5.68 4.74 5.73 6.89

Slide 95

Slide 95 text

ggplot(data = anscombe) + aes(x = x1, y = y1) + geom_point() Anscombe's quartet

Slide 96

Slide 96 text

Slide 97

Slide 97 text

Slide 98

Slide 98 text

> anscombe x1 x2 x3 x4 y1 y2 y3 y4 1 10 10 10 8 8.04 9.14 7.46 6.58 2 8 8 8 8 6.95 8.14 6.77 5.76 3 13 13 13 8 7.58 8.74 12.74 7.71 anscombe_long <- pivot_longer(data = anscombe, cols = everything(), names_pattern = "(.)(.)", names_to = c(".value", "key")) Wide -> Long form

Slide 99

Slide 99 text

anscombe_long <- pivot_longer(data = anscombe, cols = everything(), names_pattern = "(.)(.)", names_to = c(".value", "key")) > anscombe_long # A tibble: 44 x 3 key x y 1 1 10 8.04 2 2 10 9.14 3 3 10 7.46 4 4 8 6.58 5 1 8 6.95 6 2 8 8.14 7 3 8 6.77 8 4 8 5.76 Anscombe's quartet

Slide 100

Slide 100 text

Slide 101

Slide 101 text

anscombe_long <- pivot_longer(data = anscombe, cols = everything(), names_pattern = "(.)(.)", names_to = c(".value", "key")) g_anscomb <- ggplot(data = anscombe_long)+ aes(x = x, y = y, color = key)+ geom_point() > anscombe_long # A tibble: 44 x 3 key x y 1 1 10 8.04 2 2 10 9.14 3 3 10 7.46 4 4 8 6.58 5 1 8 6.95 6 2 8 8.14 7 3 8 6.77 8 4 8 5.76 Anscombe's quartet

Slide 102

Slide 102 text

g_anscomb + facet_wrap( ~ key) Anscombe's quartet

Slide 103

Slide 103 text

g_anscomb + facet_wrap( ~ key) + theme(legend.position = "none") Anscombe's quartet

Slide 104

Slide 104 text

まとめ

Slide 105

Slide 105 text

import Tidy Transform Visualize Model Communicate Modiﬁed from “R for Data Science”, H. Wickham, 2017 Data Science ① ②

Slide 106

Slide 106 text

import Tidy Transform Visualize Model Communicate Modiﬁed from “R for Data Science”, H. Wickham, 2017 preprocessing Data science Data ObservaPon Hypothesis Narra/ve of data feedback Data processing

Slide 107

Slide 107 text

data.frame / -bble raed_csv() write_csv() Table Data Wide form Long form pivot_longer() pivot_wider() Plot {ggplot2} Image Files ggsave() Data Processing

Slide 108

Slide 108 text

Slide 109

Slide 109 text

Slide 110

Slide 110 text

1. mutate() 2. filter() 3. select() 4. group_by() 5. summarize() 6. left_join() 7. arrange() Data.frame manipula:on

Slide 111

Slide 111 text

import Tidy Transform Visualize Model Communicate Modiﬁed from “R for Data Science”, H. Wickham, 2017 Data Science ① ②

Slide 112

Slide 112 text

𝑋 𝑌 𝑦! 𝑥! 𝑦" 𝑥" 𝑋 𝑌 𝑥! 𝑥" 𝑦! 𝑦" mapping x axis, y axis, color, ﬁll, shape, linetype, alpha… aesthetic channels data ggplot2 package

Slide 113

Slide 113 text

dat <- data.frame(a = 1:3, b = 8:10) ggplot(data = dat) + aes(x = a, y = b) + geom_point() + geom_path()