2024_1_17_ローカルLLMに向き合う会_LT会発表資料

Transcript

1. hikettei ਂ૚ֶशίϯύΠϥೖ໳ ϩʔΧϧLLMʹ޲͖߹͏ձ (2024/01/17༻)
2. ࣗݾ঺հ hikettei @ichndm - ਂ૚ֶशͷόοΫΤϯυΛ͍ͬͯ͡ΔਓؒͰ͢ @hikettei - ڵຯ͋Δ͜ͱ: ਂ૚ֶशίϯύΠϥ/ྔࢠԽ/ߦྻԋࢉ - Common Lisp͕޷͖ͰझຯͷOSSΛΑ͘΍ͬͯ·͢ - ·ͨLLMؔ࿈ͷ͜ͱ΋΍Γ͍ͨͰ͢
3. ʙ ͓඼ॻ͖ ʙ - 1. ਂ૚ֶश”ϑϨʔϜϫʔΫ”ͷ੒ཱ - 2. தؒදݱ౷ҰԽͷಈ͖ - 3. ਂ૚ֶश”ίϯύΠϥ"ͷ੒ཱ - 4. ࠷దԽख๏Λ೷͍ͯΈΔ (Polyhedral Compiler) - 5. ࠷ۙͷ໘നͦ͏ͳDLίϯύΠϥͱ͔ (Tinygrad) - 6. ࢀߟจݙ (΋ͬͱ஌Γ͍ͨਓ޲͚) ਂ૚ֶशͷόοΫΤϯυ΍Δਓؒͬͯ݁ߏك༗Β͍͠ ͔ͤͬ͘ͳͷͰֶशͨ͠ϞσϧΛͲ͏ӡ༻͢Δ͔ͷ࿩Λ͠·͢ ʙ໨࣍ʙ
4. 1. ਂ૚ֶश”ϑϨʔϜϫʔΫ”ͷ੒ཱ ਂ૚ֶशϞσϧͷܭࢉաఔΛ಺෦ͰͲ͏දݱ͢Δ͔ʁ (Ҿ༻: https://onnx.ai/onnx/intro/concepts.html) - ӈͷΑ͏ͳܭࢉάϥϑΛ૊Έ߹Θͤͯදݱ͢Δ ྫ: y=ax+b ςϯιϧ͕leaf, ԋࢉ͕ϊʔυͷந৅ߏจ໦(AST) - ͜ͷදݱΛ౷Ұ͠ɼମܥతͳϞσϧͷֶशɼਪ࿦ɼݕূΛࢧԉ͢Δ ͷ͕ਂ૚ֶशϑϨʔϜϫʔΫ PyTorch΍Tensor fl ow, JAXͳͲ ଟ͘͸૊ΈࠐΈDSL (Embedded DSL)ͱ࣮ͯ͠ݱ ͜ΕΒϑϨʔϜϫʔΫ͸ѻ͑Δάϥϑͷදݱྗ͕ߴ͍ͷͰɼදݱͰ͖ΔϞσϧ΋෯޿͍ C++/Python/Julia͕ओྲྀ PythonͷAPIΛհͯ͠άϥϑߏ଄Λ࡞੒ͨ͠Γ͢Δɻ - ͜ͷάϥϑߏ଄Λ΋ͱʹɼࣗಈඍ෼ͨ͠Γॱ఻ൖΛܭࢉͨ͠Γ͢Δɻ
5. 2. தؒදݱ౷ҰԽͷಈ͖ - ONNX (ΦχΩε) 2017೥9݄7೔ʹFacebookɼMicrosoft͕ࣾத৺ͱͳΓఏএ - ͦΕ·ͰҟͳΔϑϨʔϜϫʔΫؒͷܭࢉάϥϑ͸ޓ׵ੑ͕ͳ͔ͬͨ ONNXΛதؒදݱͱ֤͠ϑϨʔϜϫʔΫ͕αϙʔτ͢Δ͜ͱͰɼ૬ޓӡ༻ੑΛ࣮ݱ - ֶश؀ڥͱਪ࿦؀ڥΛ෼཭͢Δಈ͖ ҟͳΔϑϨʔϜϫʔΫؒͰಉ݁͡ՌΛ࠶ݱ͠΍͘͢ͳͬͨ e.g.:) PyTorchͰֶशͨ͠ϞσϧΛONNXʹอଘͯ͠ɼਪ࿦ʹಛԽͨ͠ϑϨʔϜϫʔΫͱ૬ޓӡ༻͢Δɻ ਪ࿦ʹಛԽͨ͠ίϯύΠϥΛ։ൃ͢Δ౔୆͕Ͱ͖ͨ ֶशϞσϧΛϓϩμΫτ౳ʹԠ༻͢Δࡍʹɼ
6. 3. ਂ૚ֶश”ίϯύΠϥ”ͷ੒ཱ ܭࢉάϥϑΛ࠷దԽ͢ΔίϯύΠϥ 1. ֶशࡁΈਂ૚ֶशϞσϧΛ (ONNXͳͲͷ) தؒදݱͱͯ͠ड͚औΓ (ೖྗ) 2. (಺෦) άϥϑͷ਺ֶతੑ࣭΍σʔλͷґଘؔ܎Λݩʹɼ࠷దԽΛ͠ (ߴϨϕϧදݱ) 3. ໨తͷϋʔυ΢ΣΞ(CPU/GPU)ͷੑ࣭ʹಛԽͨ͠࠷దԽΛ͠ (௿Ϩϕϧදݱ) 4. ϋʔυ΢ΣΞͰ௚઀࣮ߦͳܗࣜ(C++/CUDAͳͲ)ͰίʔυΛੜ੒͢Δ (ग़ྗ) ͜Ε͕ਂ૚ֶशίϯύΠϥͷఆٛ
7. 4. ਂ૚ֶशίϯύΠϥͷ௕ॴ ϓϩμΫτ։ൃʹ͓͚Δ௕ॴ ֶशࡁΈϞσϧ ֶश؀ڥ ਪ࿦؀ڥ(ίϯύΠϥ) “ίϯύΠϥ”͸”ϑϨʔϜϫʔΫ”ΑΓ΋୲౰͢Δकඋൣғ͕ڱ͍ (ਪ࿦͚ͩΛߟ͑ΔͳΒ) - ਪ࿦͢Δ؀ڥʹಛԽ͕͠΍͍͢ i.e.:) GPU؀ڥͰֶशͨ͠ϞσϧΛCPU΍NPUʹҠ২͢Δͷָ͕ ྫ: GPU͋ΓͷPyTorchͰֶशͨ͠ϞσϧΛ࠷దԽͯ͠ϥζύΠʹಈ͔࣌͢ͳͲ - ϑϨʔϜϫʔΫͷϨϕϧͰ͸ࢪͤͳ͍࠷దԽ͕ՄೳʹͳΔ (࠷ۙͷPyTorch͸JITඪ४౥ࡌͰ͕͢ɻɻɻ) PyTorchؙ͝ͱҠ২͸ࠎ͕ંΕΔ͚ͲɼίϯύΠϥ͍ͬͯ͡Cͷίʔυు͔ͤͯҠ২͢Δํ਑ͳΒݱ࣮తͩ͠࠷దԽ͠΍͍͢ (ਤ: ֶश~ਪ࿦ͷྲྀΕ͸͜ͷΑ͏ͳ෼ۀߏ଄ʹͳͬͯΔ)
8. 5. ࣮ࡍͷ࠷దԽख๏ͷྫ - ࠷ۙษڧͨ͠ख๏ʹ͍ͭͯ࿩ͨͯ͘͠ɼࠓ͔Β͍ͩͿ৔ҧ͍ͰϚΠφʔͳ࿩Λ͠·͢ɻ Θ͔Γ΍͘͢఻͑ΕΔࣗ৴͕ͳ͍ͷͰɼ΋ͬͱ໢ཏతͰΘ͔Γ΍͍͢ࢿྉΛష͓͖ͬͯ·͢ɻ https://xsig.ipsj.or.jp/wp-content/uploads/sites/6/2019/06/imai_20190529_xSiG_public_s.pdf - ਂ૚ֶशίϯύΠϥͷ֓ཁͱ࠷ۙͷಈ޲ - ࠓ͔Β࿩͢ͷ͸ɼߦྻੵ(gemm)΍৞ΈࠐΈ(Conv, im2col)ͳͲ ϝϞϦͷಡΈํ͕ෳࡶͳϞσϧΛࣗಈͰฒྻԽ/࠷దԽ͢Δͱ͍͏ٕज़Ͱ͢ɻ - ͜ͷੈքʹ͸௚ײతͳख๏͔Βͳ͔ͥߦྻ͕ͨ͘͞Μग़ͯ͘Δख๏͕͋Γ·͢ɻ
9. 5. ࣮ࡍͷ࠷దԽख๏ͷྫ ଟॏforจͷ࠷దԽ άϥϑͷߏ଄͔ΒҎԼͷίʔυ͕ੜ੒Ͱ͖Δͱ͢Δ
10. 5. ࣮ࡍͷ࠷దԽख๏ͷྫ ଟॏforจͷ࠷దԽ άϥϑͷߏ଄͔ΒҎԼͷίʔυ͕ੜ੒Ͱ͖Δͱ͢Δ ← kͱjΛೖΕସ͑ͯ΋ຊ࣭తͳҙຯ͸ಉ͡
11. 5. ࣮ࡍͷ࠷దԽख๏ͷྫ ଟॏforจͷ࠷దԽ άϥϑͷߏ଄͔ΒҎԼͷίʔυ͕ੜ੒Ͱ͖Δͱ͢Δ ← kͱjΛೖΕସ͑ͯ΋ຊ࣭తͳҙຯ͸ಉ͡ - i, j, kͷϧʔϓͷॱ൪͸ద౰ʹฒͼସ͑ͯ΋ɼίʔυ͸ਖ਼͘͠ಈ࡞͢Δɻ(ґଘੑ͕ͳ͍͔Β) - Ͳͷॱ൪ͰϧʔϓΛճ͔͢Ͱɼίʔυͷ଎౓͸େ͖͘มΘΔɻ - ద౰ʹมܗ͢ΔͱฒྻԽͨ࣌͠ʹϝϞϦΛಡΉॱ൪͕͓͔͘͠ͳͬͯɼίʔυ͕յΕΔ - ϝϞϦ্Ͱ͍ͭ͘ஔ͖ʹ഑ྻͷཁૉΛಡΈࠐΜͰ͍Δ͔ʁ(CPUͳΒ1ͭஔ͖ͩͱخ͔ͬͨ͠Γ) Q. ͜ͷෳࡶͳґଘؔ܎ΛࣗಈͰ࠷దԽ͢ΔϞσϧΛ࡞Εͳ͍͔ʁ
12. 5. ࣮ࡍͷ࠷దԽख๏ͷྫ Polyhedral Compiler - kͷ஋͸0, 1, 2, … 32ͷॱ൪Ͱಡ·ͳ͍ͱ͍͚ͳ͍ - i, jΛಡΉॱ൪͸೚ҙͰ͍͍ ੍໿ (࣮ࡍ͸΋ͬͱటष͍) ໨తؔ਺ - jΛ࠷ޙʹ࣋ͬͯ͘ΔͱϝϞϦੑೳ͕ߴ͍ 1. Z[i, k]͸X[i, j]ʹܾఆ͞ΕΔ 2. Z[i, k]͸Y[j, k]ʹܾఆ͞ΕΔ (einsum(ij, jk -> ik)ͷσʔλґଘؔ܎, Credit: https://github.com/ftynse/clint) → ͜ΕΛάϥϑʹॻ͘ͱ… (Polyhedra) - ໼ҹ͕σʔλͷґଘؔ܎Λද͢ ͜ΕΛյ͞ͳ͚Ε͹ίʔυมܗͯ͠OK - ੔਺ઢܗܭը໰୊ͱͯ͠࠷దԽͰ͖Δ LaTeXྗͷѹ౗తͳܽ೗Ͱ៉ྷʹ਺ࣜΛॻ͚·ͤΜͰͨ͠ɻ͝༰͍ࣻͩ͘͞ Կ͕خ͍͔͠ʁ → ྫ: ߦྻԋࢉͷ଎౓͓͍ͯॏཁͳฒྻԽΛࣗಈͰνϡʔχϯάͰ͖Δ
13. 5. ࣮ࡍͷ࠷దԽख๏ͷྫ (༨ஊ) ࣮૷ͯ͠Έͨ (https://github.com/hikettei/cl-polyhedral) (OpenBLASͰ9G fl opsग़ΔCPUͱߦྻͷαΠζͩͱࣗ࡞ͷ΍͕ͭ3GFlops͘Β͍ɼଟ෼AVX512໋ྩ࢖͑͹΋͏গ͕ࠩ͘͢͠ͳΔͳΔ͸ͣ) → ਂ૚ֶशͰ࢖͏ΦϖϨʔλʔΛɼࣗલͰߴ଎ʹੜ੒͢Δ͜ͱ͕Ͱ͖ͨ
14. 6. ࠷ۙͷDLίϯύΠϥ Tinygrad (https://github.com/tinygrad/tinygrad) - 2022೥11݄͔Β։ൃ͕࢝·ͬͨൺֱత৽͍͠DLίϯύΠϥ (ΊͬͪΌ໘ന͍) - ಛ௃: 0͔Β100·Ͱࣗಈίʔυੜ੒ʹཔΔ ਂ૚ֶशͰ࢖͏ΦϖϨʔλʔͷ࣮૷Λશͯࣗಈੜ੒ʹґଘ͢Δͱ͍͏ۃ୺ͳํ਑ ௕ॴ ୹ॴ - ϚΠφʔ؀ڥʹ΋1000ߦ͘Β͍ͷ֦ுΛॻ͚͹ϑϨʔϜϫʔΫؙ͝ͱҠ২Ͱ͖Δ e.g.:) ϥζύΠʹಛԽͨ͠ਂ૚ֶशίϯύΠϥΛ࡞Ζ͏ͱࢥͬͯ΋धཁ͕ͳͯ͘։ൃͷܖػ͕ͳ͔ͬͨ - શͯίʔυੜ੒ʹཔΔ <-> ґଘϥΠϒϥϦ͕΄΅0 e.g.:) CUDAͱ͔PyTorchͷόʔδϣϯ؅ཧͰϓϩδΣΫτ͕ࢮ͵͜ͱ͸͋Μ·Γͳ͍ - ߴ଎Խ - ߴ଎Խ͕೉͍͠ ΍ͬͺΓਆͷΑ͏ʹݡ͍ίϯύΠϥ͕ඞཁ + खಈͰνϡʔχϯά͢Δ͜ͱͱͷࠩ͸େ͖͍ e.g.:) ܭࢉʹ༻͍Δάϥϑͷ୯Ґ͕ࡉ͔͍ -> ΑΓଟ͘ͷ໋ྩΛ༥߹ͨ͠ΓͰ͖Δ - ΍ͬͯΔ͜ͱͷIQ͕ߴ͗͢Δ άϥϑͷ୯Ґ͕େ͖͍ vs খ͍͞͸ɼ࣮૷ͷ͠΍͢͞ vs ೉͠͞ͷτϨʔυΦϑɻͦͷ఺Tinygrad͸Ұ൪೉͍͠ͱ͜ΖΛߦͬͯΔ ίϯύΠϥຊମ΋5000ߦ͘Β͍ͷPythonίʔυͰهड़͞Ε͍ͯΔ
15. ਂ۷Γ͍ͨ͠ਓ΁ - The Deep Learning Compiler: A Comprehensive Survey https://arxiv.org/pdf/2002.03794.pdf ͜ΕΛಡΊ͹શମ૾͕Θ͔ΔɼDLίϯύΠϥͷSurvey࿦จ - Introduction to Polyhedral Compiler https://www.slideshare.net/ahayashi10/introduction-to-polyhedral-compilation Polyhedral Compilerͷ΋͏ͪΐͬͱݫີͳεϥΠυ ͜ͷಈը΋ྑ͔ͬͨ: https://www.youtube.com/watch?v=rBhvUoGYhXo - ͦͷଞ໘ന͍ϦϙδτϦ … TVM, ONNX, Tinygrad, Petalisp, etc… ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ :) SpeakerDeck: (ະఆ)