Tweet
Tweet

Slide 1

Slide 1 text

େ༰ྔSSDͱOpenStreetMap Taro Matsuzawa(@smellman) OpenStreetMap Foundation Japan 1

Slide 2

Slide 2 text

ࣗݾ঺հ ஍ཧۭؒܥΤϯδχΞ ೔ຊUNIXϢʔβձձ௕/Ұൠࣾஂ๏ਓOSGeo೔ຊࢧ෦ཧࣄ/Ұൠࣾஂ๏ਓΦʔ ϓϯετϦʔτϚοϓϑΝ΢ϯσʔγϣϯδϟύϯϝϯόʔ ౦ژిػେֶOB/౦ژిػେֶCySecݱ໾ੜ breakcoreΫϥελ ૊௕ͱݺ͹Εͯ·͢

Slide 3

Slide 3 text

૊௕ʢ୤ୀࡁΈʣ

Slide 4

Slide 4 text

OSSͷ׆ಈ React NativeܥϥΠϒϥϦͷվम react-native-static-server (iOSରԠ), redux-persist (ϥΠϒϥϦΛ·Δ͝ͱ TypeScriptʹॻ͖׵͑Δ) ͳͲ OpenMapTilesϓϩδΣΫτ΁ͷߩݙɺ஍ཧؔ܎։ൃ΍຋༁ ckanext-spatial (Python3ରԠ) ͳͲ ͦͷଞɺؾ͕޲͘··ίʔυ΍υΩϡϝϯτΛॻ͍ͯ·͢

Slide 5

Slide 5 text

ࣥචྺ ͦͷଞC Magazineͷಛू΍ɺSoftware Designͷ࿈ࡌͳͲ ݩʑ͸Firefox(Mozilla)Λத৺ʹ׆ಈɺݱࡏ͸஍ཧ΍εϚϗΞϓϦ͕ϝΠϯ

Slide 6

Slide 6 text

એ఻

Slide 7

Slide 7 text

FOSS4G 2021 Japan Online 12/4-5 OSSͳ஍ཧͷΠϕϯτͰ͢ ίΞσΠͱνϡʔτϦΞϧσΠͷೋ೔ؒ ։࠵ ίΞσΠ͕ൃදத৺ Youtube LiveͰͲͳͨͰ΋ݟΔࣄ͕Ͱ͖· ͢ https://www.osgeo.jp/events/foss4g-2021/foss4g-2021-japan-online

Slide 8

Slide 8 text

GEOSPATIAL Hackers Program ૯຿লओ࠵ͷ஍ཧۭؒܥΤϯδχΞͷҭ੒ϓϩάϥϜ ࠓ݄຤͙Β͍͔Βืू͕։࢝͞ΕΔͱࢥ͏ͷͰཁνΣοΫ ͳ͓ɺ๻͸νϡʔλʔΛ΍Γ·͢ https://ghp.connpass.com/

Slide 9

Slide 9 text

ຊ୊

Slide 10

Slide 10 text

େ༰ྔSSDͱOpenStreetMap Powered by Kioxia

Slide 11

Slide 11 text

͜ͱͷ͸͡·Γ ʮ૊௕ɺSSDି͢ͷͰϕϯνϚʔΫهࣄΛಉਓࢽʹॻ͍ͯΈͳ͍ʁʯ

Slide 12

Slide 12 text

ಉਓࢽ Kioxia(چ౦ࣳϝϞϦ)͕Maker Fair Tokyo޲ ͚ʹ࡞੒ͨ͠ಉਓࢽ ΠϕϯτͷΦϯϥΠϯԽʹ൐͍PDFͰ ແঈͰެ։ SSDͷೱ͍࿩͕ͨ͘͞Μ ͳ͔ͥͦ͜ʹࠞͬͨ͟ https://about.kioxia.com/ja-jp/news/2021/20210930-1.html

Slide 13

Slide 13 text

ࠓճ͸ಉਓࢽͷݩωλΛ࿩͢ͱ͍͏ײ͡Ͱ͢ɻ ৽࡞ϕϯνϚʔΫ͸ͪΐͬͱ͚ͩ͋Δɻ

Slide 14

Slide 14 text

ςʔϚ OpenStreetMapʹ͍ͭͯ ି͠ग़͠ػࡐͱ࣮ݧ؀ڥʹ͍ͭͯ ࣮ࢪͨ͠ϕϯνϚʔΫͷ঺հ ࠓޙͷల๬ʹ͍ͭͯ

Slide 15

Slide 15 text

OpenStreetMapʹ͍ͭͯ

Slide 16

Slide 16 text

OpenStreetMapͱ͸(1) ୭Ͱ΋ࣗ༝ʹ஍ਤΛॻ͘͜ͱ͕Ͱ͖Δϓ ϥοτϑΥʔϜ ཁϢʔβొ࿥ ੈքதͷ஍ਤσʔλΛ୯Ұͷσʔλϕʔ εͰಈ͔͍ͯ͠Δ

Slide 17

Slide 17 text

OpenStreetMapͱ͸(2) σʔλ͸ node, way, relation ͷ૊Έ߹ΘͤͰ੒Γཱͭσʔλϕʔε σʔλͷදݱܗࣜͱͯ͠XMLΛ࠾༻͍ͯ͠Δ σʔλ͸ ODbL ͱ͍͏ϥΠηϯεͰެ։͞Ε͍ͯΔ ঎༻ར༻΋OK ͪͳΈʹosm.orgͰϗεςΟϯά͞ΕͯΔʮը૾ʯ͸CC BY-SA

Slide 18

Slide 18 text

OpenStreetMapͱ͸(3) ͼ͗Ͷͬͱ͞ΜΛྫʹ͢Δͱ node͕ձࣾͷPoI(Point of Interest) ดͨ͡wayͰձࣾͷܗ(Polygon)Λදݱ ඒ஛௨Γ(Polyline)͸wayͰදݱ

Slide 19

Slide 19 text

OpenStreetMapͱ͸(4) ੈքதͷσʔλ(Planet)ΛXMLͰදݱ͢Δ ͱ1.5TB ѹॖͯ͠഑෍͢Δͷ͕ී௨ Bzip2: 109GB Protocol Buffer: 60GB

Slide 20

Slide 20 text

Protocol Bufferʹ͍ͭͯ XMLʹ࠷దԽ͞Εͨσʔλަ׵ܗࣜ ͔ͳΓσʔλѹॖ͕Մೳ 1.5TB XML -> 60GB Protocol Buffer ࠷ۙͷVector Tile΋಺෦͸Protocol Bufferܗ͕ࣜத৺ ϥΠϒϥϦͳͲ΋ॆ࣮͍ͯ͠ΔͷͰOSMͷσʔλΛ࢖͏ͳΒσʔλ༰ྔʹ ߹ΘͤͯProtocol BufferͱXMLΛ࢖͍෼͚Δͱྑ͍

Slide 21

Slide 21 text

ି͠ग़͠ػࡐͱ࣮ݧ؀ڥʹ͍ͭͯ

Slide 22

Slide 22 text

ࠓճ͸ػࡐ(PC)Λ·Δ͝ͱିͯ͠΋Β͍·ͨ͠ɻ ͳ͓ɺࠓ΋ିͯ͠΋Β͍ͬͯ·͢ɻ

Slide 23

Slide 23 text

εϖοΫ ROG STRIX Z390-F GAMING Intel Core i7-9700F 3.00GHz (8core/8thread) DDR4 PC4-19200 CL15 8GBx2 SSD KXD5YLN13T84 (3,840GB NVMe)

Slide 24

Slide 24 text

εϖοΫ(͓·͚) AMD/ATI Vega 20 Ͳ͏ߟ͑ͯ΋ήʔϛϯάϚγϯͰ͢

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

SSD KXD5YLN13T84 Kioxiaͷσʔληϯλʔ޲͚SSD Ұൠ༻ʹ͸ʮചΒΕ͍ͯͳ͍ʯ ͍͓ͪ͏յͯ͠΋ౖΒΕͳ͍ͱ͍͏࿩ʹͳͬͯΔ .oO(͍͘Β͢ΜͩΖɺාͯ͘ฉ͚ͳ͍) https://business.kioxia.com/ja-jp/ssd/data-center-ssd/xd5.html

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

جຊతͳ؀ڥ Debian/Bullseyes IPΞυϨεΛݻఆԽ ͋ͱ͸SSHͰΑ͠ͳʹ جຊίϯιʔϧͰ࡞ۀ

Slide 30

Slide 30 text

جຊߏ੒

Slide 31

Slide 31 text

࣮ݧ؀ڥ ࣮ՈͷRaspberry Pi͔Βࣗ୐ʹVPNͰৗ࣌઀ଓ ࣗ୐͔Β࣮ՈͷRaspberry PiʹϩάΠϯ Raspberry PiΛ౿Έ୆ʹ࣮ͯ͠ݧػʹϩάΠϯͯ͠ͻͨ͢Β࡞ۀ ࣮ՈͷRaspberry PiʹPrometheusΛಈ͔ͯ͠I/OΛܭଌ

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

࣮ࢪͨ͠ϕϯνϚʔΫʹ͍ͭͯ

Slide 34

Slide 34 text

Overpass APIͷߏங OpenStreetMapͰҰൠతʹ࢖ΘΕ͍ͯΔAPI XMLͰΫΤϦΛॻ͍ͯɺXMLͰฦ౴Λड͚Δ ؆ུԽͨ͠ܗࣜͰΫΤϦΛॻ͘Overpass Turboͱ͍͏ͷ΋͋Δ ͋͘·Ͱଟ͘Πϯελϯε͕͋Δͷ͕Overpass API

Slide 35

Slide 35 text

Πϯετʔϧ https://wiki.openstreetmap.org/wiki/Overpass_API/Installation جຊతʹ͜͜ʹ͋Δํ๏ͰOK init_osm3s.sh ͷ࣮ߦ࣌ؒΛܭଌ

Slide 36

Slide 36 text

݁Ռ bin/init_osm3s.sh ../data/planet-210816.osm.bz2 $DB_DIR $EXEC_DIR 107634.58s user 12608.34s system 91% cpu 36:32:35.51 total 1.5೔΄ͲͰΠϯϙʔτ׬ྃ

Slide 37

Slide 37 text

ߟ࡯ ೖྗܗ͕ࣜbzip2ܗࣜ ѹॖΛղ͘ͷʹίετ͕͔͔ͬͯͳ͍͔ʁ εΫϦϓτΛݺΜͰbunzip2Λ࣮ߦͯ͠Δͱ͜ΖΛಛఆͯ͠ɺ1.5TBͷXML ΛಡΈࠐ·ͤͯΈͨ

Slide 38

Slide 38 text

݁Ռ มԽͳ͠orz ୯ҰϓϩηεͰಈ࡞͢ΔͨΊɺΠϯϙʔτϓϩηεࣗମ͕ͦ΋ͦ΋Ϙτ ϧωοΫʹͳ͍ͬͯͨ ಉ͘͡500MB/sec͙Β͍ͷεϐʔυ͕ग़͍ͯͨ ·ͩ·ͩߦ͚Δ͸ͣ

Slide 39

Slide 39 text

imposm3ʹΑΔPostgreSQL΁ͷσʔλ౤ೖ Imposm3ͱ͍͏ෳ਺ͷϓϩηεͰΠϯϙʔτΛߦ͏πʔϧ͕͋Δ Πϯϙʔτઌ͸PostgreSQL+PostGIS+hstore PostgreSQL͸ PGTune Λར༻ͯ͠νϡʔχϯά ೖྗϑΝΠϧ͸Protocol Bufferܗࣜ https://imposm.org/docs/imposm3/latest/

Slide 40

Slide 40 text

݁Ռ ./imposm import -mapping mapping.json -connection -read -write 118021.70s user 28366.90s system 387% cpu 10:29:22.71 total 10࣌ؒ30෼΄ͲͰΠϯϙʔτ׬ྃ

Slide 41

Slide 41 text

ߟ࡯ ೔ຊͷΠϯϙʔτͱੈքͷΠϯϙʔτΛൺֱ ੈքͷํ͕ύϑΥʔϚϯεʹҧ͍͕ग़ͨʂ

Slide 42

Slide 42 text

ੈքͷΠϯϙʔτ ೔ຊͷΠϯϙʔτ͸࠷େͰread͕400MB/ sec͕ͩͬͨɺੈքͷΠϯϙʔτͰ͸࠷ େ1600MB/sec·ͰύϑΥʔϚϯε͕ग़ͨ ཧ࿦஋ͷ3/4͙Β͍·Ͱग़͍ͯΔʂ

Slide 43

Slide 43 text

osm2pgroutingͷϕϯνϚʔΫ pgRoutingͱ͍͏ܦ࿏୳ࡧ༻ͷϓϩάϥϜΛௐࠪ ͳ͓ɺओͳpgRouting͸ओʹฐ͕ࣾϝϯςφϯε͍ͯ͠·͢ ͦ΋ͦ΋IPAͷະ౿ϓϩδΣΫτൃ https://pgrouting.org/

Slide 44

Slide 44 text

τϥϒϧ osm2pgrouting͕ϝϞϦϦʔΫ͕ܹ͘͠ɺσϑΥϧτͷઃఆͰ͸͙͢ʹམ ͪͯ͠·͏ SwapϑΝΠϧΛ512GB༻ҙͯ͠ରԠ େ༰ྔSSDͳΒͰ͸ͷ࢖͍ํ(·ͯ

Slide 45

Slide 45 text

݁Ռ ೔ຊશҬΛΠϯϙʔτͨ͠ͱ͜Ζɺܦ࿏͕ਖ਼͘͠ΠϯϙʔτͰ͖ͳ͔ͬ ͨorz ୅ΘΓʹؔ౦ͷΤϦΞΛΠϯϙʔτ ./osm2pgrouting -d pgrouting_kanto -U osm -W osm -c -f 603.93s user 216.84s system 22% cpu 1:00:40.31 total 1࣌ؒ΄ͲͰΠϯϙʔτ׬ྃ

Slide 46

Slide 46 text

ܦ࿏୳ࡧٴͼ݁Ռ ࣗసं޲͚ϧʔτΛܭࢉ Πϯϙʔτ΋γϯάϧϓϩηεͳͷͰɺ readੑೳͰ200MB/secఔ౓ ϚϧνϓϩηεԽͨ͠Β΋ͬͱྑ͘ ͳΓͦ͏͕ͩɺ݁ߏ೉ͦ͠͏ʼʻ

Slide 47

Slide 47 text

ύϑΥʔϚϯε·ͱΊ ϚϧνϓϩηεͰ͔ͭೖྗ͕େ͖͍ϑΝΠϧͷಡΈࠐΈͷ৔߹ʹύ ϑΥʔϚϯε͕ߴ͘ग़Δͱ͍͏݁Ռʹͳͬͨ େ༰ྔSSDͳΒੈքதͷσʔλΛͿΜճͤΔʂ

Slide 48

Slide 48 text

ࠓޙͷల๬ʹ͍ͭͯ

Slide 49

Slide 49 text

openmaptilesͷςετ ೔ຊͷλΠϧαʔόߏஙʹ࢖͍ͬͯΔϓϩάϥϜΛςετ͍ͯ͠Δ ಺෦తʹimposm3Λར༻ ݩʑDesk mini a300ͱ͍͏ϚγϯͰ΍͍ͬͯͨ࡞ۀ Desk mini a300ʹ٧ΜͰΔͷ͸Ryzen 3400G (4core/8thread)ͳͷͰൺֱର ৅ͱͯͪ͠ΐͬͱऑ͍ https://openmaptiles.org/

Slide 50

Slide 50 text

ੑೳൺֱ Asia ͷൣғͷΠϯϙʔτͷൺֱ Desk mini a300: 20೔ఔ౓ Kioxiaͷݕূظ: 16೔ఔ౓ ୯७ʹCPUͷੑೳࠩͱࢥΘΕΔ

Slide 51

Slide 51 text

όʔδϣϯΞοϓ OpenmaptilesϓϩδΣΫτͷDockerϑΝΠϧ͕࠷ۙόʔδϣϯΞοϓ ૣ଎ൺֱͯ͠Έͨ

Slide 52

Slide 52 text

݁Ռ 2021-10-4 (kioxia) Japan: 131345 seconds (36.5࣌ؒ) 2021-10-19 (kioxia) Japan: 100535 seconds (28࣌ؒ) ࠩ෼: 30810ඵ=8.5࣌ؒͷੑೳ޲্ ϨϯλϦϯάͷੑೳ͕େ෯ʹ্͕ͬͨͱࢥΘΕΔ

Slide 53

Slide 53 text

࠷ۙͷٞ࿦ JsonbܗࣜͰࠃࡍԽΛ͍ͯ͠ΔͷΛ͹Βͨ͠ΒύϑΥʔϚϯε্͕͕Δ ͷͰ͸ʁ SQLϕʔεͰ΍͍ͬͯΔ࡞ۀΛPythonʹஔ͖׵͑ͨΒͲ͏ͳΔ͔ʁ νϡʔχϯάͷͨΊͷݕূ͕ٞ࿦͞Ε͍ͯΔ

Slide 54

Slide 54 text

ͦͷଞͷϓϩάϥϜ https://tilemaker.org/ OpenStreetMap Foundation (ຊՈ)Ͱݕ౼͞Ε͍ͯΔϓϩάϥϜ OpenMapTilesͷ੒ՌΛ࢖͍ͭͭɺ৽͍͠Πϯϙʔτͷ࢓૊ΈΛ࡞Ζ͏ ͱ͍ͯ͠Δ ͕࣌ؒऔΕͨΒͬͪ͜ͷϕϯνϚʔΫΛऔΖ͏͔ͱࢥ͍·͢

Slide 55

Slide 55 text

·ͱΊ

Slide 56

Slide 56 text

େ༰ྔSSDͱڊେͳσʔλϕʔε͸૬ੑ͕ྑ͍ ͨͩ͠ɺϚϧνϓϩηεͳ΋ͷͰͳ͍ͱSSDͷੑೳ͕ੜ͔ͤͳ͍ େ༰ྔSSD͸ເ͕͋Δ 1TBͰ΍Γ͘Γ͢Δͷ͸ͦΖͦΖෆໟ

Slide 57

Slide 57 text

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠