大容量SSDとOpenStreetMap

OpenSourceConference 2021 Online/Fallの資料です。

Transcript

1. େ༰ྔSSDͱOpenStreetMap Taro Matsuzawa(@smellman) OpenStreetMap Foundation Japan 1
2. ࣗݾ঺հ ஍ཧۭؒܥΤϯδχΞ ೔ຊUNIXϢʔβձձ௕/Ұൠࣾஂ๏ਓOSGeo೔ຊࢧ෦ཧࣄ/Ұൠࣾஂ๏ਓΦʔ ϓϯετϦʔτϚοϓϑΝ΢ϯσʔγϣϯδϟύϯϝϯόʔ ౦ژిػେֶOB/౦ژిػେֶCySecݱ໾ੜ breakcoreΫϥελ ૊௕ͱݺ͹Εͯ·͢
3. ૊௕ʢ୤ୀࡁΈʣ
4. OSSͷ׆ಈ React NativeܥϥΠϒϥϦͷվम react-native-static-server (iOSରԠ), redux-persist (ϥΠϒϥϦΛ·Δ͝ͱ TypeScriptʹॻ͖׵͑Δ) ͳͲ OpenMapTilesϓϩδΣΫτ΁ͷߩݙɺ஍ཧؔ܎։ൃ΍຋༁ ckanext-spatial (Python3ରԠ) ͳͲ ͦͷଞɺؾ͕޲͘··ίʔυ΍υΩϡϝϯτΛॻ͍ͯ·͢
5. ࣥචྺ ͦͷଞC Magazineͷಛू΍ɺSoftware Designͷ࿈ࡌͳͲ ݩʑ͸Firefox(Mozilla)Λத৺ʹ׆ಈɺݱࡏ͸஍ཧ΍εϚϗΞϓϦ͕ϝΠϯ
6. એ఻
7. FOSS4G 2021 Japan Online 12/4-5 OSSͳ஍ཧͷΠϕϯτͰ͢ ίΞσΠͱνϡʔτϦΞϧσΠͷೋ೔ؒ ։࠵ ίΞσΠ͕ൃදத৺ Youtube LiveͰͲͳͨͰ΋ݟΔࣄ͕Ͱ͖· ͢ https://www.osgeo.jp/events/foss4g-2021/foss4g-2021-japan-online
8. GEOSPATIAL Hackers Program ૯຿লओ࠵ͷ஍ཧۭؒܥΤϯδχΞͷҭ੒ϓϩάϥϜ ࠓ݄຤͙Β͍͔Βืू͕։࢝͞ΕΔͱࢥ͏ͷͰཁνΣοΫ ͳ͓ɺ๻͸νϡʔλʔΛ΍Γ·͢ https://ghp.connpass.com/
9. ຊ୊
10. େ༰ྔSSDͱOpenStreetMap Powered by Kioxia
11. ͜ͱͷ͸͡·Γ ʮ૊௕ɺSSDି͢ͷͰϕϯνϚʔΫهࣄΛಉਓࢽʹॻ͍ͯΈͳ͍ʁʯ
12. ಉਓࢽ Kioxia(چ౦ࣳϝϞϦ)͕Maker Fair Tokyo޲ ͚ʹ࡞੒ͨ͠ಉਓࢽ ΠϕϯτͷΦϯϥΠϯԽʹ൐͍PDFͰ ແঈͰެ։ SSDͷೱ͍࿩͕ͨ͘͞Μ ͳ͔ͥͦ͜ʹࠞͬͨ͟ https://about.kioxia.com/ja-jp/news/2021/20210930-1.html
13. ࠓճ͸ಉਓࢽͷݩωλΛ࿩͢ͱ͍͏ײ͡Ͱ͢ɻ ৽࡞ϕϯνϚʔΫ͸ͪΐͬͱ͚ͩ͋Δɻ
14. ςʔϚ OpenStreetMapʹ͍ͭͯ ି͠ग़͠ػࡐͱ࣮ݧ؀ڥʹ͍ͭͯ ࣮ࢪͨ͠ϕϯνϚʔΫͷ঺հ ࠓޙͷల๬ʹ͍ͭͯ
15. OpenStreetMapʹ͍ͭͯ
16. OpenStreetMapͱ͸(1) ୭Ͱ΋ࣗ༝ʹ஍ਤΛॻ͘͜ͱ͕Ͱ͖Δϓ ϥοτϑΥʔϜ ཁϢʔβొ࿥ ੈքதͷ஍ਤσʔλΛ୯Ұͷσʔλϕʔ εͰಈ͔͍ͯ͠Δ
17. OpenStreetMapͱ͸(2) σʔλ͸ node, way, relation ͷ૊Έ߹ΘͤͰ੒Γཱͭσʔλϕʔε σʔλͷදݱܗࣜͱͯ͠XMLΛ࠾༻͍ͯ͠Δ σʔλ͸ ODbL ͱ͍͏ϥΠηϯεͰެ։͞Ε͍ͯΔ ঎༻ར༻΋OK ͪͳΈʹosm.orgͰϗεςΟϯά͞ΕͯΔʮը૾ʯ͸CC BY-SA
18. OpenStreetMapͱ͸(3) ͼ͗Ͷͬͱ͞ΜΛྫʹ͢Δͱ node͕ձࣾͷPoI(Point of Interest) ดͨ͡wayͰձࣾͷܗ(Polygon)Λදݱ ඒ஛௨Γ(Polyline)͸wayͰදݱ
19. OpenStreetMapͱ͸(4) ੈքதͷσʔλ(Planet)ΛXMLͰදݱ͢Δ ͱ1.5TB ѹॖͯ͠഑෍͢Δͷ͕ී௨ Bzip2: 109GB Protocol Buffer: 60GB
20. Protocol Bufferʹ͍ͭͯ XMLʹ࠷దԽ͞Εͨσʔλަ׵ܗࣜ ͔ͳΓσʔλѹॖ͕Մೳ 1.5TB XML -> 60GB Protocol Buffer ࠷ۙͷVector Tile΋಺෦͸Protocol Bufferܗ͕ࣜத৺ ϥΠϒϥϦͳͲ΋ॆ࣮͍ͯ͠ΔͷͰOSMͷσʔλΛ࢖͏ͳΒσʔλ༰ྔʹ ߹ΘͤͯProtocol BufferͱXMLΛ࢖͍෼͚Δͱྑ͍
21. ି͠ग़͠ػࡐͱ࣮ݧ؀ڥʹ͍ͭͯ
22. ࠓճ͸ػࡐ(PC)Λ·Δ͝ͱିͯ͠΋Β͍·ͨ͠ɻ ͳ͓ɺࠓ΋ିͯ͠΋Β͍ͬͯ·͢ɻ
23. εϖοΫ ROG STRIX Z390-F GAMING Intel Core i7-9700F 3.00GHz (8core/8thread) DDR4 PC4-19200 CL15 8GBx2 SSD KXD5YLN13T84 (3,840GB NVMe)
24. εϖοΫ(͓·͚) AMD/ATI Vega 20 Ͳ͏ߟ͑ͯ΋ήʔϛϯάϚγϯͰ͢
26. SSD KXD5YLN13T84 Kioxiaͷσʔληϯλʔ޲͚SSD Ұൠ༻ʹ͸ʮചΒΕ͍ͯͳ͍ʯ ͍͓ͪ͏յͯ͠΋ౖΒΕͳ͍ͱ͍͏࿩ʹͳͬͯΔ .oO(͍͘Β͢ΜͩΖɺාͯ͘ฉ͚ͳ͍) https://business.kioxia.com/ja-jp/ssd/data-center-ssd/xd5.html
29. جຊతͳ؀ڥ Debian/Bullseyes IPΞυϨεΛݻఆԽ ͋ͱ͸SSHͰΑ͠ͳʹ جຊίϯιʔϧͰ࡞ۀ
30. جຊߏ੒
31. ࣮ݧ؀ڥ ࣮ՈͷRaspberry Pi͔Βࣗ୐ʹVPNͰৗ࣌઀ଓ ࣗ୐͔Β࣮ՈͷRaspberry PiʹϩάΠϯ Raspberry PiΛ౿Έ୆ʹ࣮ͯ͠ݧػʹϩάΠϯͯ͠ͻͨ͢Β࡞ۀ ࣮ՈͷRaspberry PiʹPrometheusΛಈ͔ͯ͠I/OΛܭଌ
33. ࣮ࢪͨ͠ϕϯνϚʔΫʹ͍ͭͯ
34. Overpass APIͷߏங OpenStreetMapͰҰൠతʹ࢖ΘΕ͍ͯΔAPI XMLͰΫΤϦΛॻ͍ͯɺXMLͰฦ౴Λड͚Δ ؆ུԽͨ͠ܗࣜͰΫΤϦΛॻ͘Overpass Turboͱ͍͏ͷ΋͋Δ ͋͘·Ͱଟ͘Πϯελϯε͕͋Δͷ͕Overpass API
35. Πϯετʔϧ https://wiki.openstreetmap.org/wiki/Overpass_API/Installation جຊతʹ͜͜ʹ͋Δํ๏ͰOK init_osm3s.sh ͷ࣮ߦ࣌ؒΛܭଌ
36. ݁Ռ bin/init_osm3s.sh ../data/planet-210816.osm.bz2 $DB_DIR $EXEC_DIR 107634.58s user 12608.34s system 91% cpu 36:32:35.51 total 1.5೔΄ͲͰΠϯϙʔτ׬ྃ
37. ߟ࡯ ೖྗܗ͕ࣜbzip2ܗࣜ ѹॖΛղ͘ͷʹίετ͕͔͔ͬͯͳ͍͔ʁ εΫϦϓτΛݺΜͰbunzip2Λ࣮ߦͯ͠Δͱ͜ΖΛಛఆͯ͠ɺ1.5TBͷXML ΛಡΈࠐ·ͤͯΈͨ
38. ݁Ռ มԽͳ͠orz ୯ҰϓϩηεͰಈ࡞͢ΔͨΊɺΠϯϙʔτϓϩηεࣗମ͕ͦ΋ͦ΋Ϙτ ϧωοΫʹͳ͍ͬͯͨ ಉ͘͡500MB/sec͙Β͍ͷεϐʔυ͕ग़͍ͯͨ ·ͩ·ͩߦ͚Δ͸ͣ
39. imposm3ʹΑΔPostgreSQL΁ͷσʔλ౤ೖ Imposm3ͱ͍͏ෳ਺ͷϓϩηεͰΠϯϙʔτΛߦ͏πʔϧ͕͋Δ Πϯϙʔτઌ͸PostgreSQL+PostGIS+hstore PostgreSQL͸ PGTune Λར༻ͯ͠νϡʔχϯά ೖྗϑΝΠϧ͸Protocol Bufferܗࣜ https://imposm.org/docs/imposm3/latest/
40. ݁Ռ ./imposm import -mapping mapping.json -connection -read -write 118021.70s user 28366.90s system 387% cpu 10:29:22.71 total 10࣌ؒ30෼΄ͲͰΠϯϙʔτ׬ྃ
41. ߟ࡯ ೔ຊͷΠϯϙʔτͱੈքͷΠϯϙʔτΛൺֱ ੈքͷํ͕ύϑΥʔϚϯεʹҧ͍͕ग़ͨʂ
42. ੈքͷΠϯϙʔτ ೔ຊͷΠϯϙʔτ͸࠷େͰread͕400MB/ sec͕ͩͬͨɺੈքͷΠϯϙʔτͰ͸࠷ େ1600MB/sec·ͰύϑΥʔϚϯε͕ग़ͨ ཧ࿦஋ͷ3/4͙Β͍·Ͱग़͍ͯΔʂ
43. osm2pgroutingͷϕϯνϚʔΫ pgRoutingͱ͍͏ܦ࿏୳ࡧ༻ͷϓϩάϥϜΛௐࠪ ͳ͓ɺओͳpgRouting͸ओʹฐ͕ࣾϝϯςφϯε͍ͯ͠·͢ ͦ΋ͦ΋IPAͷະ౿ϓϩδΣΫτൃ https://pgrouting.org/
44. τϥϒϧ osm2pgrouting͕ϝϞϦϦʔΫ͕ܹ͘͠ɺσϑΥϧτͷઃఆͰ͸͙͢ʹམ ͪͯ͠·͏ SwapϑΝΠϧΛ512GB༻ҙͯ͠ରԠ େ༰ྔSSDͳΒͰ͸ͷ࢖͍ํ(·ͯ
45. ݁Ռ ೔ຊશҬΛΠϯϙʔτͨ͠ͱ͜Ζɺܦ࿏͕ਖ਼͘͠ΠϯϙʔτͰ͖ͳ͔ͬ ͨorz ୅ΘΓʹؔ౦ͷΤϦΞΛΠϯϙʔτ ./osm2pgrouting -d pgrouting_kanto -U osm -W osm -c -f 603.93s user 216.84s system 22% cpu 1:00:40.31 total 1࣌ؒ΄ͲͰΠϯϙʔτ׬ྃ
46. ܦ࿏୳ࡧٴͼ݁Ռ ࣗసं޲͚ϧʔτΛܭࢉ Πϯϙʔτ΋γϯάϧϓϩηεͳͷͰɺ readੑೳͰ200MB/secఔ౓ ϚϧνϓϩηεԽͨ͠Β΋ͬͱྑ͘ ͳΓͦ͏͕ͩɺ݁ߏ೉ͦ͠͏ʼʻ
47. ύϑΥʔϚϯε·ͱΊ ϚϧνϓϩηεͰ͔ͭೖྗ͕େ͖͍ϑΝΠϧͷಡΈࠐΈͷ৔߹ʹύ ϑΥʔϚϯε͕ߴ͘ग़Δͱ͍͏݁Ռʹͳͬͨ େ༰ྔSSDͳΒੈքதͷσʔλΛͿΜճͤΔʂ
48. ࠓޙͷల๬ʹ͍ͭͯ
49. openmaptilesͷςετ ೔ຊͷλΠϧαʔόߏஙʹ࢖͍ͬͯΔϓϩάϥϜΛςετ͍ͯ͠Δ ಺෦తʹimposm3Λར༻ ݩʑDesk mini a300ͱ͍͏ϚγϯͰ΍͍ͬͯͨ࡞ۀ Desk mini a300ʹ٧ΜͰΔͷ͸Ryzen 3400G (4core/8thread)ͳͷͰൺֱର ৅ͱͯͪ͠ΐͬͱऑ͍ https://openmaptiles.org/
50. ੑೳൺֱ Asia ͷൣғͷΠϯϙʔτͷൺֱ Desk mini a300: 20೔ఔ౓ Kioxiaͷݕূظ: 16೔ఔ౓ ୯७ʹCPUͷੑೳࠩͱࢥΘΕΔ
51. όʔδϣϯΞοϓ OpenmaptilesϓϩδΣΫτͷDockerϑΝΠϧ͕࠷ۙόʔδϣϯΞοϓ ૣ଎ൺֱͯ͠Έͨ
52. ݁Ռ 2021-10-4 (kioxia) Japan: 131345 seconds (36.5࣌ؒ) 2021-10-19 (kioxia) Japan: 100535 seconds (28࣌ؒ) ࠩ෼: 30810ඵ=8.5࣌ؒͷੑೳ޲্ ϨϯλϦϯάͷੑೳ͕େ෯ʹ্͕ͬͨͱࢥΘΕΔ
53. ࠷ۙͷٞ࿦ JsonbܗࣜͰࠃࡍԽΛ͍ͯ͠ΔͷΛ͹Βͨ͠ΒύϑΥʔϚϯε্͕͕Δ ͷͰ͸ʁ SQLϕʔεͰ΍͍ͬͯΔ࡞ۀΛPythonʹஔ͖׵͑ͨΒͲ͏ͳΔ͔ʁ νϡʔχϯάͷͨΊͷݕূ͕ٞ࿦͞Ε͍ͯΔ
54. ͦͷଞͷϓϩάϥϜ https://tilemaker.org/ OpenStreetMap Foundation (ຊՈ)Ͱݕ౼͞Ε͍ͯΔϓϩάϥϜ OpenMapTilesͷ੒ՌΛ࢖͍ͭͭɺ৽͍͠Πϯϙʔτͷ࢓૊ΈΛ࡞Ζ͏ ͱ͍ͯ͠Δ ͕࣌ؒऔΕͨΒͬͪ͜ͷϕϯνϚʔΫΛऔΖ͏͔ͱࢥ͍·͢
55. ·ͱΊ
56. େ༰ྔSSDͱڊେͳσʔλϕʔε͸૬ੑ͕ྑ͍ ͨͩ͠ɺϚϧνϓϩηεͳ΋ͷͰͳ͍ͱSSDͷੑೳ͕ੜ͔ͤͳ͍ େ༰ྔSSD͸ເ͕͋Δ 1TBͰ΍Γ͘Γ͢Δͷ͸ͦΖͦΖෆໟ
57. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠