大容量SSDとOpenStreetMap

Transcript

1. େ༰ྔSSDͱOpenStreetMap Taro Matsuzawa(@smellman) OpenStreetMap Foundation Japan 1

2. ࣗݾ঺հ ஍ཧۭؒܥΤϯδχΞ ೔ຊUNIXϢʔβձձ௕/Ұൠࣾஂ๏ਓOSGeo೔ຊࢧ෦ཧࣄ/Ұൠࣾஂ๏ਓΦʔ ϓϯετϦʔτϚοϓϑΝ΢ϯσʔγϣϯδϟύϯϝϯόʔ ౦ژిػେֶOB/౦ژిػେֶCySecݱ໾ੜ breakcoreΫϥελ ૊௕ͱݺ͹Εͯ·͢

3. ૊௕ʢ୤ୀࡁΈʣ

4. OSSͷ׆ಈ React NativeܥϥΠϒϥϦͷվम react-native-static-server (iOSରԠ), redux-persist (ϥΠϒϥϦΛ·Δ͝ͱ TypeScriptʹॻ͖׵͑Δ) ͳͲ OpenMapTilesϓϩδΣΫτ΁ͷߩݙɺ஍ཧؔ܎։ൃ΍຋༁

   ckanext-spatial (Python3ରԠ) ͳͲ ͦͷଞɺؾ͕޲͘··ίʔυ΍υΩϡϝϯτΛॻ͍ͯ·͢

5. ࣥචྺ ͦͷଞC Magazineͷಛू΍ɺSoftware Designͷ࿈ࡌͳͲ ݩʑ͸Firefox(Mozilla)Λத৺ʹ׆ಈɺݱࡏ͸஍ཧ΍εϚϗΞϓϦ͕ϝΠϯ

6. એ఻

7. FOSS4G 2021 Japan Online 12/4-5 OSSͳ஍ཧͷΠϕϯτͰ͢ ίΞσΠͱνϡʔτϦΞϧσΠͷೋ೔ؒ ։࠵ ίΞσΠ͕ൃදத৺ Youtube

   LiveͰͲͳͨͰ΋ݟΔࣄ͕Ͱ͖· ͢ https://www.osgeo.jp/events/foss4g-2021/foss4g-2021-japan-online

8. GEOSPATIAL Hackers Program ૯຿লओ࠵ͷ஍ཧۭؒܥΤϯδχΞͷҭ੒ϓϩάϥϜ ࠓ݄຤͙Β͍͔Βืू͕։࢝͞ΕΔͱࢥ͏ͷͰཁνΣοΫ ͳ͓ɺ๻͸νϡʔλʔΛ΍Γ·͢ https://ghp.connpass.com/

9. ຊ୊

10. େ༰ྔSSDͱOpenStreetMap Powered by Kioxia

11. ͜ͱͷ͸͡·Γ ʮ૊௕ɺSSDି͢ͷͰϕϯνϚʔΫهࣄΛಉਓࢽʹॻ͍ͯΈͳ͍ʁʯ

12. ಉਓࢽ Kioxia(چ౦ࣳϝϞϦ)͕Maker Fair Tokyo޲ ͚ʹ࡞੒ͨ͠ಉਓࢽ ΠϕϯτͷΦϯϥΠϯԽʹ൐͍PDFͰ ແঈͰެ։ SSDͷೱ͍࿩͕ͨ͘͞Μ ͳ͔ͥͦ͜ʹࠞͬͨ͟ https://about.kioxia.com/ja-jp/news/2021/20210930-1.html

13. ࠓճ͸ಉਓࢽͷݩωλΛ࿩͢ͱ͍͏ײ͡Ͱ͢ɻ ৽࡞ϕϯνϚʔΫ͸ͪΐͬͱ͚ͩ͋Δɻ

14. ςʔϚ OpenStreetMapʹ͍ͭͯ ି͠ग़͠ػࡐͱ࣮ݧ؀ڥʹ͍ͭͯ ࣮ࢪͨ͠ϕϯνϚʔΫͷ঺հ ࠓޙͷల๬ʹ͍ͭͯ

15. OpenStreetMapʹ͍ͭͯ

16. OpenStreetMapͱ͸(1) ୭Ͱ΋ࣗ༝ʹ஍ਤΛॻ͘͜ͱ͕Ͱ͖Δϓ ϥοτϑΥʔϜ ཁϢʔβొ࿥ ੈքதͷ஍ਤσʔλΛ୯Ұͷσʔλϕʔ εͰಈ͔͍ͯ͠Δ

17. OpenStreetMapͱ͸(2) σʔλ͸ node, way, relation ͷ૊Έ߹ΘͤͰ੒Γཱͭσʔλϕʔε σʔλͷදݱܗࣜͱͯ͠XMLΛ࠾༻͍ͯ͠Δ σʔλ͸ ODbL ͱ͍͏ϥΠηϯεͰެ։͞Ε͍ͯΔ

    ঎༻ར༻΋OK ͪͳΈʹosm.orgͰϗεςΟϯά͞ΕͯΔʮը૾ʯ͸CC BY-SA

18. OpenStreetMapͱ͸(3) ͼ͗Ͷͬͱ͞ΜΛྫʹ͢Δͱ node͕ձࣾͷPoI(Point of Interest) ดͨ͡wayͰձࣾͷܗ(Polygon)Λදݱ ඒ஛௨Γ(Polyline)͸wayͰදݱ

19. OpenStreetMapͱ͸(4) ੈքதͷσʔλ(Planet)ΛXMLͰදݱ͢Δ ͱ1.5TB ѹॖͯ͠഑෍͢Δͷ͕ී௨ Bzip2: 109GB Protocol Buffer: 60GB

20. Protocol Bufferʹ͍ͭͯ XMLʹ࠷దԽ͞Εͨσʔλަ׵ܗࣜ ͔ͳΓσʔλѹॖ͕Մೳ 1.5TB XML -> 60GB Protocol Buffer

    ࠷ۙͷVector Tile΋಺෦͸Protocol Bufferܗ͕ࣜத৺ ϥΠϒϥϦͳͲ΋ॆ࣮͍ͯ͠ΔͷͰOSMͷσʔλΛ࢖͏ͳΒσʔλ༰ྔʹ ߹ΘͤͯProtocol BufferͱXMLΛ࢖͍෼͚Δͱྑ͍

21. ି͠ग़͠ػࡐͱ࣮ݧ؀ڥʹ͍ͭͯ

22. ࠓճ͸ػࡐ(PC)Λ·Δ͝ͱିͯ͠΋Β͍·ͨ͠ɻ ͳ͓ɺࠓ΋ିͯ͠΋Β͍ͬͯ·͢ɻ

23. εϖοΫ ROG STRIX Z390-F GAMING Intel Core i7-9700F 3.00GHz (8core/8thread)

    DDR4 PC4-19200 CL15 8GBx2 SSD KXD5YLN13T84 (3,840GB NVMe)

24. εϖοΫ(͓·͚) AMD/ATI Vega 20 Ͳ͏ߟ͑ͯ΋ήʔϛϯάϚγϯͰ͢

25. None

26. SSD KXD5YLN13T84 Kioxiaͷσʔληϯλʔ޲͚SSD Ұൠ༻ʹ͸ʮചΒΕ͍ͯͳ͍ʯ ͍͓ͪ͏յͯ͠΋ౖΒΕͳ͍ͱ͍͏࿩ʹͳͬͯΔ .oO(͍͘Β͢ΜͩΖɺාͯ͘ฉ͚ͳ͍) https://business.kioxia.com/ja-jp/ssd/data-center-ssd/xd5.html

27. None
28. None

29. جຊతͳ؀ڥ Debian/Bullseyes IPΞυϨεΛݻఆԽ ͋ͱ͸SSHͰΑ͠ͳʹ جຊίϯιʔϧͰ࡞ۀ

30. جຊߏ੒

31. ࣮ݧ؀ڥ ࣮ՈͷRaspberry Pi͔Βࣗ୐ʹVPNͰৗ࣌઀ଓ ࣗ୐͔Β࣮ՈͷRaspberry PiʹϩάΠϯ Raspberry PiΛ౿Έ୆ʹ࣮ͯ͠ݧػʹϩάΠϯͯ͠ͻͨ͢Β࡞ۀ ࣮ՈͷRaspberry PiʹPrometheusΛಈ͔ͯ͠I/OΛܭଌ

32. None

33. ࣮ࢪͨ͠ϕϯνϚʔΫʹ͍ͭͯ

34. Overpass APIͷߏங OpenStreetMapͰҰൠతʹ࢖ΘΕ͍ͯΔAPI XMLͰΫΤϦΛॻ͍ͯɺXMLͰฦ౴Λड͚Δ ؆ུԽͨ͠ܗࣜͰΫΤϦΛॻ͘Overpass Turboͱ͍͏ͷ΋͋Δ ͋͘·Ͱଟ͘Πϯελϯε͕͋Δͷ͕Overpass API

35. Πϯετʔϧ https://wiki.openstreetmap.org/wiki/Overpass_API/Installation جຊతʹ͜͜ʹ͋Δํ๏ͰOK init_osm3s.sh ͷ࣮ߦ࣌ؒΛܭଌ

36. ݁Ռ bin/init_osm3s.sh ../data/planet-210816.osm.bz2 $DB_DIR $EXEC_DIR 107634.58s user 12608.34s system 91%

    cpu 36:32:35.51 total 1.5೔΄ͲͰΠϯϙʔτ׬ྃ

37. ߟ࡯ ೖྗܗ͕ࣜbzip2ܗࣜ ѹॖΛղ͘ͷʹίετ͕͔͔ͬͯͳ͍͔ʁ εΫϦϓτΛݺΜͰbunzip2Λ࣮ߦͯ͠Δͱ͜ΖΛಛఆͯ͠ɺ1.5TBͷXML ΛಡΈࠐ·ͤͯΈͨ

38. ݁Ռ มԽͳ͠orz ୯ҰϓϩηεͰಈ࡞͢ΔͨΊɺΠϯϙʔτϓϩηεࣗମ͕ͦ΋ͦ΋Ϙτ ϧωοΫʹͳ͍ͬͯͨ ಉ͘͡500MB/sec͙Β͍ͷεϐʔυ͕ग़͍ͯͨ ·ͩ·ͩߦ͚Δ͸ͣ

39. imposm3ʹΑΔPostgreSQL΁ͷσʔλ౤ೖ Imposm3ͱ͍͏ෳ਺ͷϓϩηεͰΠϯϙʔτΛߦ͏πʔϧ͕͋Δ Πϯϙʔτઌ͸PostgreSQL+PostGIS+hstore PostgreSQL͸ PGTune Λར༻ͯ͠νϡʔχϯά ೖྗϑΝΠϧ͸Protocol Bufferܗࣜ https://imposm.org/docs/imposm3/latest/

40. ݁Ռ ./imposm import -mapping mapping.json -connection -read -write 118021.70s user

    28366.90s system 387% cpu 10:29:22.71 total 10࣌ؒ30෼΄ͲͰΠϯϙʔτ׬ྃ

41. ߟ࡯ ೔ຊͷΠϯϙʔτͱੈքͷΠϯϙʔτΛൺֱ ੈքͷํ͕ύϑΥʔϚϯεʹҧ͍͕ग़ͨʂ

42. ੈքͷΠϯϙʔτ ೔ຊͷΠϯϙʔτ͸࠷େͰread͕400MB/ sec͕ͩͬͨɺੈքͷΠϯϙʔτͰ͸࠷ େ1600MB/sec·ͰύϑΥʔϚϯε͕ग़ͨ ཧ࿦஋ͷ3/4͙Β͍·Ͱग़͍ͯΔʂ

43. osm2pgroutingͷϕϯνϚʔΫ pgRoutingͱ͍͏ܦ࿏୳ࡧ༻ͷϓϩάϥϜΛௐࠪ ͳ͓ɺओͳpgRouting͸ओʹฐ͕ࣾϝϯςφϯε͍ͯ͠·͢ ͦ΋ͦ΋IPAͷະ౿ϓϩδΣΫτൃ https://pgrouting.org/

44. τϥϒϧ osm2pgrouting͕ϝϞϦϦʔΫ͕ܹ͘͠ɺσϑΥϧτͷઃఆͰ͸͙͢ʹམ ͪͯ͠·͏ SwapϑΝΠϧΛ512GB༻ҙͯ͠ରԠ େ༰ྔSSDͳΒͰ͸ͷ࢖͍ํ(·ͯ

45. ݁Ռ ೔ຊશҬΛΠϯϙʔτͨ͠ͱ͜Ζɺܦ࿏͕ਖ਼͘͠ΠϯϙʔτͰ͖ͳ͔ͬ ͨorz ୅ΘΓʹؔ౦ͷΤϦΞΛΠϯϙʔτ ./osm2pgrouting -d pgrouting_kanto -U osm -W

    osm -c -f 603.93s user 216.84s system 22% cpu 1:00:40.31 total 1࣌ؒ΄ͲͰΠϯϙʔτ׬ྃ

46. ܦ࿏୳ࡧٴͼ݁Ռ ࣗసं޲͚ϧʔτΛܭࢉ Πϯϙʔτ΋γϯάϧϓϩηεͳͷͰɺ readੑೳͰ200MB/secఔ౓ ϚϧνϓϩηεԽͨ͠Β΋ͬͱྑ͘ ͳΓͦ͏͕ͩɺ݁ߏ೉ͦ͠͏ʼʻ

47. ύϑΥʔϚϯε·ͱΊ ϚϧνϓϩηεͰ͔ͭೖྗ͕େ͖͍ϑΝΠϧͷಡΈࠐΈͷ৔߹ʹύ ϑΥʔϚϯε͕ߴ͘ग़Δͱ͍͏݁Ռʹͳͬͨ େ༰ྔSSDͳΒੈքதͷσʔλΛͿΜճͤΔʂ

48. ࠓޙͷల๬ʹ͍ͭͯ

49. openmaptilesͷςετ ೔ຊͷλΠϧαʔόߏஙʹ࢖͍ͬͯΔϓϩάϥϜΛςετ͍ͯ͠Δ ಺෦తʹimposm3Λར༻ ݩʑDesk mini a300ͱ͍͏ϚγϯͰ΍͍ͬͯͨ࡞ۀ Desk mini a300ʹ٧ΜͰΔͷ͸Ryzen 3400G

    (4core/8thread)ͳͷͰൺֱର ৅ͱͯͪ͠ΐͬͱऑ͍ https://openmaptiles.org/

50. ੑೳൺֱ Asia ͷൣғͷΠϯϙʔτͷൺֱ Desk mini a300: 20೔ఔ౓ Kioxiaͷݕূظ: 16೔ఔ౓ ୯७ʹCPUͷੑೳࠩͱࢥΘΕΔ

51. όʔδϣϯΞοϓ OpenmaptilesϓϩδΣΫτͷDockerϑΝΠϧ͕࠷ۙόʔδϣϯΞοϓ ૣ଎ൺֱͯ͠Έͨ

52. ݁Ռ 2021-10-4 (kioxia) Japan: 131345 seconds (36.5࣌ؒ) 2021-10-19 (kioxia) Japan:

    100535 seconds (28࣌ؒ) ࠩ෼: 30810ඵ=8.5࣌ؒͷੑೳ޲্ ϨϯλϦϯάͷੑೳ͕େ෯ʹ্͕ͬͨͱࢥΘΕΔ

53. ࠷ۙͷٞ࿦ JsonbܗࣜͰࠃࡍԽΛ͍ͯ͠ΔͷΛ͹Βͨ͠ΒύϑΥʔϚϯε্͕͕Δ ͷͰ͸ʁ SQLϕʔεͰ΍͍ͬͯΔ࡞ۀΛPythonʹஔ͖׵͑ͨΒͲ͏ͳΔ͔ʁ νϡʔχϯάͷͨΊͷݕূ͕ٞ࿦͞Ε͍ͯΔ

54. ͦͷଞͷϓϩάϥϜ https://tilemaker.org/ OpenStreetMap Foundation (ຊՈ)Ͱݕ౼͞Ε͍ͯΔϓϩάϥϜ OpenMapTilesͷ੒ՌΛ࢖͍ͭͭɺ৽͍͠Πϯϙʔτͷ࢓૊ΈΛ࡞Ζ͏ ͱ͍ͯ͠Δ ͕࣌ؒऔΕͨΒͬͪ͜ͷϕϯνϚʔΫΛऔΖ͏͔ͱࢥ͍·͢

55. ·ͱΊ

56. େ༰ྔSSDͱڊେͳσʔλϕʔε͸૬ੑ͕ྑ͍ ͨͩ͠ɺϚϧνϓϩηεͳ΋ͷͰͳ͍ͱSSDͷੑೳ͕ੜ͔ͤͳ͍ େ༰ྔSSD͸ເ͕͋Δ 1TBͰ΍Γ͘Γ͢Δͷ͸ͦΖͦΖෆໟ

57. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠