2022年12月17日に開催された「R研究集会2022」の発表資料です。
当日の投影内容から、メモリの消費量の算出方法に誤りがあったため一部内容を変更しています。 リポジトリ: https://github.com/uribo/talk_221217_rjpusers
ӝੜਅʢಙౡେֶσβΠϯܕ"*ڭҭݚڀηϯλʔʣʹΑΔେنσʔλͷॲཧ౷ܭཧݚڀॴڞಉར༻ݚڀूձʮσʔλղੳڥ3ͷඋͱར༻ʯV@SJCPIUUQTHJUIVCDPNVSJCPUBML@@SKQVTFST
View Slide
Ұԡ͠ϙΠϯτେنσʔληοτͷޮతͳॲཧͱసૹͷͨΊʹઃܭ"QBDIF"SSPX\BSSPX^ύοέʔδΠϯϝϞϦͷΧϥϜܕσʔλΛऔΓѻ͏ʢγϦΞϥΠζෆཁʣଟݴޠπʔϧϘοΫεͱͯ͠ɺ͞·͟·ͳϓϩάϥϛϯάݴޠͰ࣮$ 3 1ZUIPO 3VTU +VMJB %VDL%#ͳͲEQMZSόοΫΤϯυΛఏڙɻύΠϓԋࢉࢠʹΑΔॲཧΛద༻ՄೳQBSRVFUʹରԠɻύʔςΟγϣχϯάʹΑΔཧMVCSJEBUFɺSMBOHɺTUSJOHSɺUJEZTFMFDUͳͲͷؔʹରԠະରԠͳؔΛ\EVDLEC^ύοέʔδ͕Χόʔ͍ খ͍͞ಡΈࠐΈɾूܭՃॲཧ͕ ϑΝΠϧɾϝϞϦαΠζ͕ޮతDTWϑΝΠϧ͔ΒͷEQMZSEBUBUBCMFͰͷॲཧͱൺֱͯ͠ɺ͔ͳΓߴσʔλͷૢ࡞ɾཧ͕
ࣗݾհӝੜਅ͏ΓΎ͏͠Μཧۭؒσʔλղੳ݄ʙಙౡ౷ܭɺσʔλαΠΤϯεԾ$ ҊΧόʔɿϓϩηε $༻ $(ɿετοΫϑΥτͷ 3' ը૾ʢ ԁʣσʔλՄࢹԽ 3ݴޠaདྷ݄ग़൛
େ͖͍σʔλͷॲཧɺͲ͏͍ͯ͠·͔͢ʁ͜͜Ͱʮେ͖͍σʔλʯΛʮݱ࣮తʹѻ͑ͦ͏ͳʯͱ͠·͢୯ҰͷςΩετɾόΠφϦϑΝΠϧҰͭͷϑΝΠϧ͕࠷େͰ(#ɺશମͰ(#Ҏఔσʔλج൫ʢ%#ɺ)BEPPQɺ"NB[PO"UIFOBɺ#JH2VFSZʣΛར༻͠ͳ͍BXLͳͲͰલॲཧΛ͠ͳ͍ఔͷେ͖͞ϩʔΧϧɾΫϥυͰॲཧʜͳΔ҆͘ՁɾੑೳͷڥͰࡁ·͍ͤͨฒྻԽϝϞϦΛͨ͘͞ΜੵΜͩίϯϐϡʔλʔͰॲཧͤ͞Δ
େ͖͍σʔλͷॲཧɺͲ͏͍ͯ͠·͔͢ʁ3ɺ34UVEJP͋Δ͋ΔॲཧͷաఔͰϝϞϦΛ͍Δ34UVEJP͕രൃ͢Δ
"QBDIF"SSPX
"QBDIF"SSPXɹɹɹɹɹɹͷσʔλੳπʔϧͷϓϥοτϑΥʔϜڞ௨ͷ࣮ΛݴޠΛ͑ͯڞ༗Ͱ͖ΔϝϞϦͷσʔλΛޮతʹߏԽͯ͠ॲཧʢγϦΞϥΠζෆཁɺߴʣσʔλަ࣌ͷੑೳվળʹΑΔߴͳॲཧΛ࣮ݱ"QBDIF"SSPXܗࣜʜBSSPXQBSRVFUྻࢦͷσʔλʢΧϥϜܕσʔλʣΛѻ͏͜ͱʹಛԽෳݴޠରԠΠϯϝϞϦͰͷॲཧΛ࣮ݱ
ΧϥϜܕσʔλσʔλੳʹಛԽIUUQTBSSPXBQBDIFPSHPWFSWJFX৯ྨྶྨྶྨϨοαʔύϯμνϯύϯδʔϚϯτώώ৯ྨௗྨϥΠΦϯϑϯϘϧτϖϯΪϯߦ୯ҐͰσʔλΛอ࣋ɾऔΓग़͠$47.Z42-ͳͲͷσʔλϕʔεߦࢦಛఆྻͷͷૢ࡞ˠશ෦ͷߦɾྻͷΞΫηεΛ͏ྻ୯ҐͰσʔλΛอ࣋ɾऔΓग़͠ྻࢦ ৯ྨྶྨྶྨϨοαʔύϯμνϯύϯδʔϚϯτώώ৯ྨௗྨϥΠΦϯϑϯϘϧτϖϯΪϯˠࢦఆ͞ΕͨྻͷΈಡΈࠐΈେྔͷߦʹର͢Δूॲཧ͕ޮత
\BSSPX^ύοέʔδ
\BSSPX^ύοέʔδͷಛ"SSPX$ϥΠϒϥϦͷΠϯλʔϑΣʔεΛఏڙ"SSPX"SSPX͕ѻ͏BSSPX QBSRVFUܗࣜͷσʔλ\EVDLEC^ύοέʔδɺ1ZUIPOͱͷ࿈ܞʢϝϞϦʹΓΒͳ͍نͷʣෳͷϑΝΠϧΛಡΈࠐΈɺॲཧ\EQMZS^Λ͡Ίͱͯ͠\MVCSJEBUF^ɺ\TUSJOHS^ɺ\UJEZTFMFDU^ͳͲͷؔʹରԠˠ\EQMZS^ʹΑΔύΠϓॲཧ3ͷؔͰେنσʔλͷૢ࡞͕࣮ݱσʔλͷίϐʔΛ͢ΔաఔͰΫϥυڥΛؚΊͨɺ\BSSPX^ͰରԠ͠ͳ͍ॲཧΛ࣮ߦ͢ΔͨΊɺผͷύοέʔδͱݴޠͰσʔλΛڞ༗ෳϑΝΠϧͷߴͳಡΈॻ͖ΠϯϝϞϦͰॲཧ
\BSSPX^ΛͬͨॲཧͷྲྀΕσʔλͷಡΈࠐΈ QBSRVFUGFBUIFSUYUDTWKTPOUTW📄σʔλϑϨʔϜͱͯ͠ฦΓΛಘΔ\EQMZS^ͳͲͷؔʹΑΔॲཧd <- read_parquet("data/zoo.parquet.parquet",as_data_frame = FALSE)result <- d |>select(name, taxon, body_length_cm) |>filter(taxon %in% c("霊長類", "齧歯類", "鳥類")) |>group_by(taxon) |>summarise(body_length_mean = mean(body_length_cm))result |>collect()ू͞ΕͨσʔλΛੳɾՄࢹԽσʔλΛϝϞϦ্ʹಡΈࠐΉԆධՁ
\EVDLEC^ύοέʔδ\BSSPX^ଆͰ\EVDLEC^ͱͷ࿈ܞͷͨΊʹto_duckdb() Λఏڙಈ࡞͢ΔίϯϐϡʔλʔͷϝϞϦ্Ͱಈ࡞͢ΔܰྔσʔλϕʔεγεςϜΦϥϯμͷࠃཱݚڀॴͷϝϯόʔ͕த৺ʹͳͬͯ։ൃ$Ͱॻ͔Εͨ"1*Λଟݴޠʹϥοϓͯ͠ఏڙ\EVDLEC^ͦͷ͏ͪͷ3ݴޠ͚ϥούʔύοέʔδˠ\BSSPX^ଆͰରԠ͠ͳ͍Ұ෦ͷؔΛ࣮ߦͰ͖Δ\%#*^ʹΑͬͯσʔλϑϨʔϜΛૢ࡞
\BSSPX^͕ରԠ͠ͳ͍ؔʁlist_compute_functions()"SSPXRVFSZFOHJOFલड़ͷͱ͓Γɺ\BSSPX^Ͱ"SSPX$ͷΠϯλʔϑΣΠεΛఏڙ\EQMZS^ͷؔར༻Մೳ?aceroˠͯ͢ͷ3ͷ͕ؔ\BSSPX^Ͱ࣮ߦՄೳͳΘ͚Ͱͳ͍d <-read_parquet(here::here("data/typeB/36_tokushima/year=2022/month=10/part-0.parquet"),as_data_frame = FALSE)࣍ͷεϥΠυͰྫΛࣔͨ͢ΊʹBSSPXΦϒδΣΫτΛ༻ҙEBUBGSBNFʹมͯ͠ॲཧPSΤϥʔ
EBUBGSBNFʹมͯ͠ॲཧd |>select(datetime) |>mutate(is_jholiday = zipangu::is_jholiday(datetime))#> Warning: Expression#> zipangu::is_jholiday(datetime) not supported in Arrow;#> pulling data into R#> datetime is_jholiday#> 1 2022-10-01 FALSE#> 2 2022-10-01 FALSE#> 3 2022-10-01 FALSE#> 4 2022-10-01 FALSE#> 5 2022-10-01 FALSE#> 6 2022-10-01 FALSEେ͖͍σʔλͷ··ͩͱॲཧʹ͕͔͔࣌ؒΔ͔
ΤϥʔͰॲཧΛఀࢭˠ\EVDLEC^ͷมͰରԠd |>group_by(location_no) |>slice_min(order_by = traffic, n = 1) |>collect()#> Error: Slicing grouped data not supported in Arrowd |>group_by(location_no) |>to_duckdb() |>slice_min(order_by = traffic, n = 1) |>collect()#> # A tibble: 25,237 × 10#> # Groups: location_no [274]#> datetime source_…¹ locat…² locat…³ meshc…⁴ link_…⁵ link_no traffic#> #> 1 2022-10-29 17:50:00 3028 54 西ノ丸… 513404 2 662 2#> 2 2022-10-29 19:50:00 3028 54 西ノ丸… 513404 2 662 2
\BSSPX^ύοέʔδͰͷ1BSRVFUϑΝΠϧͷѻ͍
"QBDIF1BSRVFUେنσʔλΛѻ͏ͨΊʹ͘ΘΕ͍ͯΔඪ४తͳܗࣜΛఏڙྻࢦσʔλΛѹॖ͞ΕͨόΠφϦܗࣜͰอଘɺ$47ͳͲͷඇѹॖςΩετܗࣜͱൺֱͯ͠ϑΝΠϧαΠζ͕খ͘͞ͳΔIUUQTQBSRVFUBQBDIFPSHޮతͳྻσʔλදݱͷརΛར༻Ͱ͖ΔΑ͏ʹઃܭݩ)BEPPQΤίγεςϜͷதͰػೳ͢ΔΑ͏ʹ։ൃ
QBSRVSUͷಡΈࠐΈ୯ҰϑΝΠϧ"844ɺ(PPHMF$MPVE4UPSBHF (4$͔ΒͷಡΈࠐΈ0,read_parquet("data/typeB/36_tokushima/year=2022/month=10/part-0.parquet")σʔλϑϨʔϜͱͯ͠ಡΈࠐΈʢσϑΥϧτʣ"SSPX5BCMFͱͯ͠ಡΈࠐΈˠΠϯϝϞϦͰॲཧՄೳread_parquet("s3://[access_key:secret_key@]bucket/path[?region=]")read_parquet("gs://anonymous@bucket/path")IUUQTBSSPXBQBDIFPSHEPDTSBSUJDMFTGTIUNMread_parquet("data/typeB/36_tokushima/year=2022/month=10/part-0.parquet",as_data_frame = FALSE,# tidyselectでの列選択col_select = c(datetime, ends_with("no")))
QBSRVSUͷಡΈࠐΈෳϑΝΠϧڞ௨ͷྻྻΛͭෳͷQBSRVFUϑΝΠϧΛಡΈࠐΉʢ͜ͷஈ֊ͰεΩʔϚͷΈʣopen_dataset(here::here("data/typeB/13_tokyo/year=2021/"))#> FileSystemDataset with 12 Parquet files#> datetime: timestamp[us, tz=Asia/Tokyo]#> source_code: string#> location_no: int32#> location_name: string#> meshcode10km: string#> link_type: int32#> link_no: int32#> traffic: int32#> to_link_end_10m: string#> link_ver: int32#> month: int32#>#> See $metadata for additional Schema metadata݄ຖʹ͔ΕͨݸͷϑΝΠϧDTWͰ0,open_dataset(..., format = "csv")
ύʔςΟγϣχϯάҰͭͷେ͖ͳσʔληοτΛෳͷࡉ͔ͳϑΝΠϧʹׂ͢Δઓུ)JWFܗࣜʜLFZWBMVFඞཁͳσʔλʹૉૣ͘ΞΫηεɺॲཧͷෛ୲͕ܰݮׂΛత֬ʹߦ͏͜ͱ͕ॏཁ ྫ͑ʜɺ݄ʢʣɺಓݝʢʣd |>write_dataset(path = "data",partitioning = c("year", "month"))σʔληοτΛ ɺ݄ʹ͚ͯอଘEBUBZFBSNPOUIQBSUQBSRVFUNPOUIQBSUQBSRVFUNPOUIQBSUQBSRVFUʜEBUBZFBSNPOUIQBSUQBSRVFUNPOUIQBSUQBSRVFUNPOUIQBSUQBSRVFU
εΩʔϚʢܕͷࢦఆʣྻࢦͷσʔλܗࣜʜྻ͝ͱʹܾ·ͬͨܕʢ·ͱ·ͬͨஔʣ͕͋Δ๛ͳσʔλλΠϓͰྻྻͷΛఆٛ͢ΔϏοτʢJOUʣͱϏοτ JOU۠ผ͞ΕΔopen_dataset("data",schema = schema(datetime = timestamp(unit = "ms",timezone = "Asia/Tokyo"),source_code = utf8(),location_no = int32(),…))#> FileSystemDataset with 12 Parquet files#> datetime: timestamp[ms, tz=Asia/Tokyo]#> source_code: string#> location_no: int32#> location_name: string#> ...IUUQTBSSPXBQBDIFPSHEPDTSSFGFSFODFEBUBUZQFIUNM
\BSSPX^ύοέʔδʹΑΔେنσʔλͷॲཧ
Ұൠಓ࿏ͷஅ໘ަ௨ྔใσʔλެӹࡒஂ๏ਓຊಓ࿏ަ௨ใηϯλʔ͕Φʔϓϯσʔλͱͯ͠ެ։֤ಓݝܯ͕ं྆ײثͳͲͷܭଌػثͰऩूͨ͠அ໘ަ௨ྔʹؔ͢ΔɹɹใΛܯிʹ͓͍ͯͱΓ·ͱΊͨͷΫϦΤΠςΟϒίϞϯζදࣔࠃࡍ $$#:IUUQTXXXKBSUJDPSKQ˞શࠃͯ͢ͷ࣍ϝογϡΛɹཏ͢ΔΘ͚Ͱͳ͍
Ұൠಓ࿏ͷஅ໘ަ௨ྔใσʔλ[JQϑΝΠϧల։ޙͷDTWΛEBUBUBCMFͱͯ͠ಡΈࠐΉؔΛ༻ҙIUUQTHJUIVCDPNVSJCPKBSUJDSremotes::install_github("uribo/jarticr")dplyr::glimpse(jarticr::read_jartic_traffic("data-raw/typeB_tokushima_2022_10/徳島県警_202210.csv"))#> Rows: 2,384,773#> Columns: 10#> $ datetime 2022-10-01, 2022-10-01, 2022-10-01, 2022-10-01, 2022-…#> $ source_code "3028", "3028", "3028", "3028", "3028", "3028", "3028"…#> $ location_no 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16,…#> $ location_name "徳島本町→県庁前", "県庁前→徳島本町", "新助任橋北詰→徳…#> $ meshcode10km "513404", "513404", "513404", "513404", "513404", "513…#> $ link_type 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, …#> $ link_no 710, 705, 681, 676, 679, 678, 11, 375, 6, 5, 6, 5, 842…#> $ traffic 21, 24, 25, 27, 25, 28, 30, 16, 42, 42, 27, 25, 16, 26…#> $ to_link_end_10m "6", "6", "11", "27", "21", "29", "26", "8", "7", "92"…#> $ link_ver 202100, 202100, 202100, 202100, 202100, 202100, 202100…
ݕূσʔλ αΠζͷҟͳΔσʔλΛछ༻ҙ-BSHF4NBMMશࠃ.FEJVN)VHF શࠃ ؒؒ౦ژ ؒ౦ژ ϲ݄ؒߦ ྻσʔλαΠζQBSRVFUҬ ظؒϑΝΠϧαΠζDTWສ ԯສ(# .# (# (#ԯສ (#ԯສ (#(#(#-BSHFɺ)VHFͷDTWϑΝΠϧαΠζ[JQѹॖ࣌ͷେ͖͞ʢల։ޙ͞Βʹେ͖͍ʣ
ݕূڥ"QQMF..BD3ϝϞϦ(#ʢݱߦͷ.#1Ͱ࠷େʣlibrary(arrow) # 10.0.1library(readr) # 2.1.3library(data.table) # 1.14.6library(dplyr) # 1.0.10library(dtplyr) # 1.2.2library(duckdb) # 0.6.1͍ͣΕͷύοέʔδ$3"/࠷৽൛
ݕূͭͷॲཧΛൺֱάϧʔϓԽͱूܭಡΈࠐΈ݁߹DTWϑΝΠϧͷಡΈࠐΈreadr::read_csv() data.table::fread()arrow::read_csv_arrow()\BSSPX^\EQMZS^\EBUBUBCMF^\EVDLEC^ʜBSSPXΦϒδΣΫτ͔Βม\EUQMZS^ʜEBUBUBCMFΦϒδΣΫτΛEQMZSͷؔͰॲཧόοΫΤϯυͰ\EBUBUBCMF^͕ػೳ
݁ՌϑΝΠϧͷಡΈࠐΈ4NBMMEBUBUBCMF🏅
݁ՌϑΝΠϧͷಡΈࠐΈ.FEJVNEBUBUBCMF🏅
݁ՌϑΝΠϧͷಡΈࠐΈ-BSHF )VHFͷDTW༻ҙͰ͖ͣಉنͷQBSRVSUϑΝΠϧΛ ͰಡΈࠐΉʜ0,arrow::open_dataset()# 全国1年分(2021年)traffic_large <-open_dataset(here::here("data/typeB/"),schema = jartic_typeB_schema) |>filter(year == 2021)dim(traffic_large)#> [1] 3923767686 12lobstr::obj_size(traffic_large)#> 487.10 kB# 全国4年分traffic_huge <-open_dataset(here::here("data/typeB/"),schema = jartic_typeB_schema)dim(traffic_huge)#> [1] 15677276121 12lobstr::obj_size(traffic_huge)#> 261.59 kB-BSHF )VHFॲཧͷ༰ʹΑͬͯ݁ՌΛಘΒΕΔঢ়ଶ
݁ՌάϧʔϓԽͱूܭBSSPX🏅4NBMM
݁ՌάϧʔϓԽͱूܭBSSPX🏅.FEJVN
݁ՌάϧʔϓԽͱूܭछͷσʔλαΠζͷ݁ՌΛൺֱσʔλαΠζʹԠ࣮ͯ͡ߦ࣌ؒ૿Ճ
݁Ռ݁߹EQMZS🏅4NBMM
݁Ռ݁߹BSSPX🏅.FEJVN
݁Ռ݁߹-BSHF )VHFͱʹࣦഊfiMUFSEJTUJODUΛͤͣʹ݁߹ͨͨ͠Ίʁtictoc::tic();traffic_large |>distinct(location_no, meshcode10km) |>left_join(arrow_mesh10km,by = c("meshcode10km" = "meshcode10km")) |>collect();tictoc::toc()#> 16.746 sec elapsedtictoc::tic();traffic_huge |>distinct(location_no, meshcode10km) |>left_join(arrow_mesh10km,by = c("meshcode10km" = "meshcode10km")) |>collect();tictoc::toc()#> 67.593 sec elapsedEJTUJODUΛڬΊ0,-BSHF)VHF
ॴݟύʔςΟγϣχϯάͷج४ͷܾΊํ͕ॏཁ\BSSPX^ͰେنͳσʔλΛॲཧͰ͖Δʢগͳ͘ͱΠϯϝϞϦͷঢ়ଶʹʣେنͳϑΝΠϧQBSRVFUͰอଘ͓ͯ͘͠ͷ͕ྑͦ͞͏ͲΜͳͱ͖ʢαΠζɾॲཧ༰ʣͰ\BSSPX^͕ϕετͳબࢶͰͳ͍αϙʔτ͠ͳ͍ؔˠ\EVDLEC^͢\EBUBUBCMF^ɺ\EUQMZS^ؤு͍ͬͯΔ
ँࣙ͜ͷΛआΓͯײँਃ্͛͠·͢ɻ͜ͷൃද(JU)VC4QPOTPSTͰࢧԉͯͩ͘͠͞ΔํʑͷఏڙͰ͓ૹΓ͠·ͨ͠ɻkatsurakobsiero5335ito4303niszetNishina-NIESteramonagitakehikoihayashiytknzwIUUQTHJUIVCDPNTQPOTPSTVSJCP
ࢀߟจݙɾ63-ͦΖͦΖ3Ϣʔβʔ"QBDIF"SSPXͰ1BSRVFUΛͬͯΈ·ͤΜ͔ʁ 3GPS%BUB4DJFODF OEFEJUJPO "QBDIF"SSPXϑΥʔϚοτͳ͍ͥͷ͔ ʲ3ʳ"QBDIF"SSPXͱEVDLECΛࢼͯ͠ΈΔ2JJUB "QBDIF"SSPXَ͑͑ʂ͜ͷ··$47શ෦1BSRVFUʹม͍ͯ͜͠͏ͥʂ "QBDIF"SSPX3$PPLCPPL -BSHFS5IBO.FNPSZ%BUB8PSLflPXTXJUI"QBDIF"SSPX IUUQTOPUDIBJOFEIBUFOBCMPHDPNFOUSZIUUQTSETIBEMFZO[BSSPXIUNMIUUQTTMJEFSBCCJUTIPDLFSPSHBVUIPSTLPVECUFDITIPXDBTFPOMJOFIUUQTFJUTVQJHJUIVCJPUPLZPSTMJEFUPLZPS@IUUQTBSSPXVTFSOFUMJGZBQQIUUQTBSSPXBQBDIFPSHDPPLCPPLSJOEFYIUNMIUUQTRJJUBDPNFJUTVQJJUFNTDFFCGCFFEF
s_uryu
wadayusuke
nozaki
miyabigouji
gmvalentino8
murabayashi
hschwentner
skanehira
tapompom
kiyoshiro
hirodragon112
yatasunshine
subomi
lara
geeforr
dougneiner
bcantrill
searls
kevinliebholz
jeffersonlam
chriscoyier
phodgson
sstephenson
aki_iinuma
malarkey