Toxic Oversight

Quick privacy review among major LLM services, along with search for possible remedies. (OWASP Saitama MTG #25, talk #1)

Transcript

1. TOXIC OVERSIGHT OWASP SAITAMA MTG #25 TALK #1 Image by greenkozi on flickr, CC-BY-NC-ND 2.0
2. TEXT SESSION FLAGS ▸ ࿥ըɾ࿥Իɾެ։: OK Image by Nico Kaiser on flickr, CC-BY 2.0
3. TEXT BACKGROUND ▸ ChatGPT, DALL-E, Gemini, Copilot, Stable Diffusion, Midjourney.. ▸ ͳΜͰ΋஌͍ͬͯΔ৺ڧ͍૬ஊ૬ख ▸ ͳΜͰ΋࡞ΕΔ΋ͷ͍͢͝ΞʔςΟετ ▸ ײ৘΋ײੑ΋͋Δ ▸ 2023/11/30ͷOpenAIʹΑΔChatGPTͷൃද͕ ൽ੾Γ… ΋͏2ࡀ Image by Xi on flickr, CC-BY-NC-ND 2.0
4. TEXT BACKGROUND ▸ Large Language Models (LLM; େن໛ݴޠϞσϧ) ▸ ೖྗ͔ΒʮͦΕͬΆ͍ʯ͜ͱΛੜ੒ ▸ σδλϧԽ͞ΕͨύϥϨϧϫʔϧυͷॅਓ ▸ 64-bit௨ΓͷseedͱͦΕ·Ͱͷݴ༿Ͱܾఆ࿦ తʹܾ·Δੈքઢ ▸ ୯ͳΔʮิ׬Ϟσϧʯ Image by Xi on flickr, CC-BY-NC-ND 2.0
5. TEXT BACKGROUND ▸ ௨ৗɺ੡࡞ऀͷαΠτܦ༝͔APIͰ࢖͏ ▸ ChatGPT ▸ Claude ▸ Gemini ... Image by Xi on flickr, CC-BY-NC-ND 2.0
6. TEXT WHO I AM ▸ Takahiro Yoshimura (@alterakey) https://keybase.io/alterakey ▸ Monolith Works Inc. Co-founder, CTO Security researcher ▸ ໌࣏େֶαΠόʔηΩϡϦςΟݚڀॴ ٬һݚڀһ (ʙ3݄)
7. TEXT WHAT I DO ▸ Security research and development ▸ iOS/Android Apps →Financial, Games, IoT related, etc. (>200) →trueseeing: Non-decompiling Android Application Vulnerability Scanner [2017] ▸ Windows/Mac/Web/HTML5 Apps →POS, RAD tools etc. ▸ Network/Web penetration testing →PCI-DSS etc. ▸ Search engine reconnaissance (aka. Google Hacking) ▸ Whitebox testing ▸ Forensic analysis
8. TEXT WHAT I DO ▸ CTF ▸ Enemy10, Sutegoma2 ▸ METI CTFCJ 2012 Qual.: Won ▸ METI CTFCJ 2012: 3rd ▸ DEF CON 21 CTF: 6th ▸ DEF CON 22 OpenCTF: 4th ▸ ൃදɾߨԋͳͲ DEF CON 25 Demo Labs (2017) DEF CON 27 AI Village (2019) CODE BLUE (2017, 2019) CYDEF (2020) etc. Image by Wiyre Media on flickr, CC-BY 2.0
9. TEXT WOULD YOU LIKE TO HAVE A STRAWBERRY? ▸ 2024೥9݄ͷOpenAIͷGPT-4 o1ͷެ։લ໷ ▸ strawberryͷrͷ਺Λ਺͍͑ͤͯͨ͞ΞΧ΢ϯ τ͕ݢฒΈౚ݁͞Εͨ… ͳͥʁ ▸ GPT-4 o1͸಺෦తʹ"strawberry"ͱݺ͹Ε͍ͯ ͨ… "r"Λےಓཱ͖ͯͯͪΜͱ਺͑ΒΕΔΑ͏ ʹ͢Δͧͱ͍͏ҙؾࠐΈ ※LLM͸͜ͷྨͷ៛ີͳਪ࿦͕େͷۤख ▸ ͭ·Γձ࿩಺༰Λݟ͍ͯΔ Image by TimAlosi.com on flickr, CC-BY-NC-ND 2.0
10. TEXT ... ARE YOU OK? ▸ ʮChatGPTར༻ऀͷϝϯλϧ่յΛݒ೦ʯ →͍·Ͱ΋ܧଓతʹݟ͍ͯΔ ▸ ύεϙʔτِ଄͕Ͱ͖ͨͱ͍͏ใࠂޙɺ2࣌ؒఔ ౓Ͱີ͔ʹϓϩϯϓτ͕෧࠯͞Εͨࣄ݅ →΄΅ϦΞϧλΠϜʹݟ͍ͯΔ ▸ DALL-Eʹ͓͍ͯ΋ϓϩϯϓτ܏޲෼ੳ →ؙདͰ͢Αɺͱ Image by TimAlosi.com on flickr, CC-BY-NC-ND 2.0
11. TEXT WE ARE WATCHING YOU, SIR ▸ ձ࿩಺༰͔Βࣗಈతʹ௨ใ͞ΕΔ͜ͱ΋͋Δ ▸ Meta AIͷγεςϜϓϩϯϓτʹʮෆ๏ߦҝ͕ ٙΘΕΔ৔߹ʹ͸ฦ౴ͤͣଈ࠲ʹ௨ใͤΑʯ ͱ͍͏هࡌ͕͋Δ ▸ ͲΕ͚ͩޮ͍͍ͯΔͷ͔͸෼͔Βͳ͍͕… llama͕πʔϧ΋࢖͑Δ͜ͱΛߟྀ͢Δͱͦ͜ ͦ͜ػೳ͍ͯ͠ΔՄೳੑ Image by telmo32 on flickr, CC-BY-ND 2.0
12. TEXT YES, LOOKING AT YOU, SIR ▸ ؂ࢹ΍৘ใར༻͕Ճ଎͖ͯͨ͠ ▸ ྙཧنఆ΍ڝ૪༏Ґੑ͔Βͷཁ੥ ▸ ChatGPTग़ݱॳ಄ͷ৘ใྲྀग़ٙ࿭ͱ͸ผ෺ →opt-outՄೳͩͬͨ →API͸ͦ΋ͦ΋ର৅Ͱ͸ͳ͔ͬͨ ▸ ͜Μͳ΋ͷ৴པͰ͖Δ͔ʁ →ʮѱ༻๷ࢭʯʮCSAMʯͱ͸ຐ๏ͷݴ༿ (·ͨOpenAI͸๷Ӵ࢈ۀೖΓ͍ͯ͠Δ) Image by loren chipman on flickr, CC-BY-NC 2.0
13. TEXT TRUMP MADNESS ▸ ୈೋ࣍τϥϯϓ੓ݖ ▸ ࣗࠃ༏ઌओٛͱอ਎ ▸ શͯ͸ຽओౘͱҠຽͷ͍ͤ ▸ ۙࢹ؟తํࡦ ▸ ൓DEI ▸ Ҡຽഉআ੓ࡦ →·͞ʹϑΝγζϜతͳ͜ͱ͕ى͖͍ͯΔ Image by IoSonoUnaFotoCamera on flickr, CC-BY-SA 2.0
14. TEXT AT YOUR SERVICE, PRESIDENT ▸ ͍ΘΏΔΦϦΨϧώͷελϯυϓϨʔ͕ͻͲ͍ ▸ Өͷ੓෎ͱԽ͍ͯ͠Δײ৮ →ಛʹΠʔϩϯͱDOGE ▸ ൓DEI΁ͷศ৐͔ΒҰؾʹӈ܏Խ →ϩϏʔ׆ಈͷҰ؀ͱͯ͠ͷศ৐ʁ Image by dmoberhaus on flickr, CC-BY 2.0
15. TEXT OUTFLOWING MADNESS ▸ ٸܹͳن੍؇࿨ํ޲ͷಈ͖ ▸ ੓࣏తѹྗΛ͔͚Δಓ۩ʹͳ͍ͬͯΔ ▸ ৘ใऩू͓Αͼར༻͸ଟ෼ଓ͘ ▸ Ϟϥϧϋβʔυ͸݈ࡏ ▸ ࣃࢭΊ͸֎Ε͖͍ͯͯΔ (≒΍Γ͍ͨ์୊) Image by shioshvili on flickr, CC-BY-SA 2.0
16. TEXT YES THEY ARE WATCHING US ▸ ϓϥΠόγʔ͕͔ͭͯͳ͍΄Ͳةݥʹ ▸ ͜Ε·Ͱ΋Ϋϥ΢υʹ͍ͭͯݴΘΕ͍͕ͯͨ →CSAMޡೝ໰୊ɺใ෮తใࠂ໰୊ ▸ ੜ੒AI͸ѱ༻๷ࢭͳͲͷҙຯ߹͍ͰαʔϏε؂ ࢹ͞Ε͍ͯΔ →Ͳ͜Ͱ΋ੜ੒AI…ʁ ▸ Google: ܉ࣄ΍؂ࢹʹར༻͠ͳ͍ͱ͍͏දه Λີ͔ʹམͱ͍ͯ͠ΔʢGemini…ʁʣ Image by ashley rose, on flickr, CC-BY-NC-ND 2.0
17. TEXT YES THEY ARE WATCHING US ▸ ϓϥΠόγʔ͕͔ͭͯͳ͍΄Ͳةݥʹ ▸ xGrok 3ʹมͳ͜ͱΛฉ͍͍ͯΔਓͨͪ… ▸ ࠃ಺ͷੜ੒AIαʔϏεʹมͳϓϩϯϓτΛ༩ ͍͑ͯΔਓͨͪ… ▸ ਎ͷةݥ͕ഭ͍ͬͯΔͱݴͬͯ΋ࢧোͳ͍ ▸ ࠓ͙͢ʹ΍Ίͨ΄͏͕ྑ͍ ʢ೔ຊͰ΋ཱͪೖΓௐࠪ͞ΕΔՄೳੑʣ Image by ashley rose, on flickr, CC-BY-NC-ND 2.0
18. TEXT PRIVACY PLEASE, SIR ▸ ւ֎Ͱ͸໰୊ఏى͕ͳ͞Ε͖ͯͨ ▸ 9݄16೔ͷnatureͷهࣄ "Forget ChatGPT: why researchers now run small AIs on their laptops" ▸ ݚڀऀͨͪ͸Open-weightͷLLMΛࣗ਎ͷϚ γϯͰಈ͔͍ͯ͠Δͱ͍͏࿩ (Local LLM) ▸ techܥ͚ͩͰͳ͘ඇtechܥ΋ Image by calebjcook on flickr, CC-BY-NC-ND 2.0
19. TEXT BECAUSE MY THOUGHTS ARE .. ▸ ೳྗ͸ྼΔͷʹͳͥʁ ▸ ࣗ਎ͷϚγϯͰ͋Ε͹׬શʹϓϥΠϕʔτ ▸ ୭ʹ΋௨ใ͞Εͳ͍ ▸ ৘ใΛར༻͞Εͳ͍ ▸ ࣗ༝ͳࢥࡧɾਪ࿦ɾ஌త࡞ۀ͕Ͱ͖Δ Image by Frankenstein on flickr, CC-BY-NC 2.0
20. TEXT .. NONE OF YOUR BUSINESS ▸ ͞Βʹ͸ ▸ ֤ྖҬͷ஌ࣝਓ΍ݚڀऀ͕ɺӦརڝ૪ʹ૸Γ͸ ͡Ί֤ͨࣾʹةػײΛ͍࣋ͬͯΔ͜ͱͷදΘΕ Image by Frankenstein on flickr, CC-BY-NC 2.0
21. TEXT .. AM I WRONG? ▸ ೔ຊͰ͸Ͳ͏͔ ▸ NICT΍PFNΛ࢝Ίͱͯ͠Ϟσϧ͕࡞ΒΕ͍ͯΔ ͕…ͧͬͯ͜Closedɺग़དྷ͸…………… ▸ ؂ࢹ͢ΔΤϯςΟςΟ͕੓෎΍௚׋ػؔʹ →͜Ε͸͜ΕͰ͍ͩͿେ͖ͳ໰୊ ▸ ࢖͏ͷͰ͋Ε͹Ϋϥ΢υ্ʹdeploy͢Δ͔ →Ͱ͖ͳ͚Ε͹ϋʔυ΢ΣΞ͝ͱങ͏͜ͱʹ →େมʹ࢖͍ʹ͍͘ Image by Toni Blay on flickr, CC-BY-NC-ND 2.0
22. TEXT CLOSED MODELS ARE NOT YOUR FRIENDS ▸ Openͳͷ͸Swallow͘Β͍ ▸ ೔ຊޠରԠͷ೉͠͞͸ΞυόϯςʔδͰ͸͋Δ ͷ͕ͩɺͦ͜ʹด͡ࠐ΋͍ͬͯΔΑ͏ʹݟ͑Δ ▸ ϘϥϯςΟΞҙࣝͷࠩͳͷ͔ɺͦΕͱ΋୯ͳΔ ৘ॹతن੍Λ༏ઌ͢ΔͨΊͩΖ͏͔ ▸ ೔ຊͷݚڀऀ͕೔ຊޠͰࣗ༝ʹߟ͑Δʹ͸͓ۚ Λࢧ෷Θͳ͚Ε͹ͳΒͳ͍…ͱ͸ɺ͏ʔΜ Image by WilsonB on flickr, CC-BY-SA 2.0
23. TEXT ESCAPE PLAN A ▸ Local LLM؀ڥ: ࣗ਎ͷϚγϯͰinfer͢Δ ▸ ௕ॴ ▸ ׬શʹϩʔΧϧʹ༓ดͰ͖Δ ▸ ԿΛಈ͔ͯ͠΋໰୊ͳ͍ ▸ ୹ॴ: ▸ ܭࢉೳྗͱϝϞϦ͕ඞཁ ʢe.g. 8B:q8_0 -> 8.5GB, 70B:q4_K_M -> 43GBʣ ▸ Open-weightܥϞσϧͷΈ࢖༻Ͱ͖Δ
24. TEXT ESCAPE PLAN A ▸ Local LLM؀ڥྫ ▸ ollama API ollama + Open-WebUI ▸ OpenAI-compatible API llama-cpp / vLLM + Open-WebUI llama-cpp / vLLM + LM Studio ... ※LM Studio͸proprietaryͰ͋Δ͜ͱʹ஫ҙ ▸ খن໛ͳΒllama.cppɺେن໛ͳΒvLLM͕༏ल (Flash Attention 2 vs. Paged Attention, ฒྻੑೳ)
25. TEXT ESCAPE PLAN B ▸ ४Local LLM؀ڥ: ࣗ਎͕ॴ༗͢ΔΫϥ΢υܭࢉ؀ڥ ▸ ௕ॴ (※AWS) ▸ ASICͰͷinferͳͷͰߴ଎ ▸ ServerlessͰ࢖༻Ͱ͖ΔϞσϧ͕ଟ͍ (e.g. Claude, Deepseek R1, LLaMA 3.1 405B etc.) ▸ ୹ॴ ▸ ίετ͕ൃੜ͢Δ ▸ ϩʔΧϧʹ͸ด͡ΒΕͳ͍ Image by auxesis on flickr, CC-BY-SA 2.0
26. TEXT ESCAPE PLAN B ▸ Ϋϥ΢υܭࢉ؀ڥͱ͸… ▸ ୯ͳΔߴ଎ͳܭࢉ؀ڥ →޷͖ͳϞσϧΛ࢖༻Ͱ͖Δ΋ͷ ▸ APIαʔόͰ͸ͳ͍͜ͱʹ஫ҙ (cf. Azure vs. AWS) Image by auxesis on flickr, CC-BY-SA 2.0
27. TEXT ESCAPE PLAN B ▸ ४Local LLM؀ڥྫ (※AWS) ▸ BedrockͰOpenAI-compatible APIΛग़͠… …ΫϥΠΞϯτ͔Βར༻ ▸ ͕ͩBedrock୯ମͰ͸OpenAI-compatible API͕ ग़ͤͳ͍ Image by auxesis on flickr, CC-BY-SA 2.0
28. TEXT ESCAPE PLAN B ▸ bedrock-access-gatewayʂ ▸ AWS Sampleͱ্͕͍ͯͬͯ͠Δ΋ͷ https://github.com/aws-samples/bedrock-access- gateway/ ▸ BedrockΛ࢖༻͢ΔServerlessͳOpenAI- compatible APIαʔό࣮૷ɺͳͷ͕ͩ ▸ CloudFormationͰVPC͔ΒALB΍Βͳʹ΍Βશͯ ࡞ͬͯ͘ΕΔ… ͔͠΋1-clickϘλϯͰىಈͱ͔ ▸ େมʹ࢖͍ʹ͍͘୅෺
29. TEXT ESCAPE PLAN B ▸ haversackʂ ▸ https://github.com/alterakey/haversack ▸ SAMͰॻ͖௚͠ɺ͔ͭ༨ܭͳϦιʔε࡞੒Λ ഉͨ͠΋ͷ ▸ ECS༻ͷΠϝʔδΛϕʔεʹ͢Δ͜ͱͰ ▸ ALB΋API Gateway΋ෆཁʹʂ ʢ୯ҰͷLambda Function URLͰఏڙʣ Image by avlxyz on flickr, CC-BY-SA 2.0
30. TEXT ESCAPE PLAN B ▸ haversackʂ ▸ bedrock-access-gatewayͷECR΁ϩάΠϯޙ sam build, sam deploy -gͰߏஙdone ▸ ͪͳΈʹhaversackɺͱ͸ AD&Dʹొ৔͢Δຐྗ͕͔͔ͬͨbagͷҰछ Image by avlxyz on flickr, CC-BY-SA 2.0
31. TEXT ESCAPE PLAN B ▸ ४Local LLM؀ڥྫ (※AWS) ▸ Bedrock + haversack + Open-WebUI ▸ Bedrock + haversack + LM Studio ▸ Bedrock + haversack + etc.. ▸ શͯserverlessʂ
32. TEXT CAVEATS ▸ Ϟσϧͷઃఆ஋ʹ͍ͭͯ (e.g. System Prompt, context size, temp. etc.) ▸ ެࣜAPIʹ͓͚Δ૝ఆͱɺOpen-WebUI΍LM StudioͳͲͷҰൠతΫϥΠΞϯτ͕૝ఆͯ͠ ͍Δ஋ͱ͸͔ͳΓҧ͏ ▸ ࠷େݶʹ׆༻͢Δʹ͸୳ࡧ͢ΔΑΖ͠ Image by Aleksey Gureiev on flickr, CC-BY-NC-ND 2.0
33. TEXT TAKEAWAYS ▸ ੜ੒AI͸୯ͳΔʮิ׬Ϟσϧʯ →ʮͦΕͬΆ͍ʯ͜ͱΛੜ੒͢ΔϞσϧ →seedͱձ࿩Ͱܾ·Δੈքઢͷॅਓ ▸ ྙཧنఆΛ६ʹͨ͠؂ࢹ΍৘ใऩू͕ಛʹܹԽ →લͷΑ͏ʹopt-out͸΋͏Ͱ͖ͳ͍ →ࣗ෼ͷϚγϯͰLLMΛಈ͔͢ͷ͕ւ֎Ͱ͸ओྲྀ ▸ ೳྗ΁ͷڪා΍ݒ೦͕ग़͍ͯΔ →ਖ਼͍͠ϦεΫධՁ͕େࣄ Image by caste_aka_adrem on flickr, CC-BY-NC 2.0
34. TEXT TAKEAWAYS ▸ ୈೋ࣍τϥϯϓ੓ݖʹΑΔ๫੓ →ͦΕʹ܈͕ΔΦϦΨϧώͷӨͱࡦ๳ ▸ ϓϥΠόγʔ͕͔ͭͯͳ͍΄Ͳةݥʹ →ʮͲ͜Ͱ΋ੜ੒AIʯ͸ʮͲ͜Ͱ΋౵ൈ͚ʯ Image by IoSonoUnaFotoCamera on flickr, CC-BY-SA 2.0
35. TEXT TAKEAWAYS ▸ ೔ຊʹ͓͚ΔLocal LLMͷར༻ʹ͸ →Open-weightͳϞσϧ͕଍Γͳ͍… →ਅͷҙຯͰࣗ༝ͳࢥࡧ͸Ͱ͖ͳ͍ ▸ ४LocalͰ͋Ε͹·ͩͳΜͱ͔ →ͨͩίετ͸͔͔Δ ▸ OpenAI-compatible API͔ollama APIΛग़ͤ͹ ༷ʑͳΫϥΠΞϯτ͔Βར༻ՄೳʹͳΔ Image by Chris Randall on flickr, CC-BY-NC 2.0
36. ONE MORE THING.. Image by Chris Pirillo on flickr, CC-BY-NC-ND 2.0
37. TEXT ONE MORE THING ▸ Deepseekࣄ݅ ▸ ৘ใྲྀग़͕ͱ͔ར༻ن੍Λͱ͔ͳΜͱ͔૽͕Ε ͍ͯΔ͕ɺ͋͘·Ͱ΋ΞϓϦ΍ެࣜαΠτͷ࿩ ▸ ୯ʹinfer͢Δ͚ͩͰ͋Ε͹ԿΒӨڹͳ͍ →४Local؀ڥͰ͸҆͝৺Λ ▸ Open-WebUI͸Webϕʔε… ͱ͍͏͜ͱ͸ϞόΠϧ୺຤͔ΒͰ΋ར༻Մೳʂ →ΞϓϦྨ͸Ұ੾ෆཁ Image by dakohuang on flickr, CC-BY 2.0
38. STAY SAFE! Image by KaCey97078 on flickr, CC-BY-NC 2.0
39. FIN. 4.22.2025 TAKAHIRO YOSHIMURA (@ALTERAKEY) Image by neilmoralee on flickr, CC-BY-NC-ND 2.0