Share
Embed
Toxic Oversight
Quick privacy review among major LLM services, along with search for possible remedies. (OWASP Saitama MTG #25, talk #1)
Takahiro Yoshimura
April 22, 2025
Transcript
39 slides
-
TOXIC OVERSIGHT OWASP SAITAMA MTG #25 TALK #1 Image by
greenkozi on flickr, CC-BY-NC-ND 2.0 -
TEXT SESSION FLAGS ▸ ըɾԻɾެ։: OK Image by Nico Kaiser
on flickr, CC-BY 2.0 -
TEXT BACKGROUND ▸ ChatGPT, DALL-E, Gemini, Copilot, Stable Diffusion, Midjourney..
▸ ͳΜͰ͍ͬͯΔ৺ڧ͍૬ஊ૬ख ▸ ͳΜͰ࡞ΕΔͷ͍͢͝ΞʔςΟετ ▸ ײײੑ͋Δ ▸ 2023/11/30ͷOpenAIʹΑΔChatGPTͷൃද͕ ൽΓ… ͏2ࡀ Image by Xi on flickr, CC-BY-NC-ND 2.0 -
TEXT BACKGROUND ▸ Large Language Models (LLM; େنݴޠϞσϧ) ▸ ೖྗ͔ΒʮͦΕͬΆ͍ʯ͜ͱΛੜ
▸ σδλϧԽ͞ΕͨύϥϨϧϫʔϧυͷॅਓ ▸ 64-bit௨ΓͷseedͱͦΕ·Ͱͷݴ༿Ͱܾఆ తʹܾ·Δੈքઢ ▸ ୯ͳΔʮิϞσϧʯ Image by Xi on flickr, CC-BY-NC-ND 2.0 -
TEXT BACKGROUND ▸ ௨ৗɺ࡞ऀͷαΠτܦ༝͔APIͰ͏ ▸ ChatGPT ▸ Claude ▸ Gemini
... Image by Xi on flickr, CC-BY-NC-ND 2.0 -
TEXT WHO I AM ▸ Takahiro Yoshimura (@alterakey) https://keybase.io/alterakey ▸
Monolith Works Inc. Co-founder, CTO Security researcher ▸ ໌࣏େֶαΠόʔηΩϡϦςΟݚڀॴ ٬һݚڀһ (ʙ3݄) -
TEXT WHAT I DO ▸ Security research and development ▸
iOS/Android Apps →Financial, Games, IoT related, etc. (>200) →trueseeing: Non-decompiling Android Application Vulnerability Scanner [2017] ▸ Windows/Mac/Web/HTML5 Apps →POS, RAD tools etc. ▸ Network/Web penetration testing →PCI-DSS etc. ▸ Search engine reconnaissance (aka. Google Hacking) ▸ Whitebox testing ▸ Forensic analysis -
TEXT WHAT I DO ▸ CTF ▸ Enemy10, Sutegoma2 ▸
METI CTFCJ 2012 Qual.: Won ▸ METI CTFCJ 2012: 3rd ▸ DEF CON 21 CTF: 6th ▸ DEF CON 22 OpenCTF: 4th ▸ ൃදɾߨԋͳͲ DEF CON 25 Demo Labs (2017) DEF CON 27 AI Village (2019) CODE BLUE (2017, 2019) CYDEF (2020) etc. Image by Wiyre Media on flickr, CC-BY 2.0 -
TEXT WOULD YOU LIKE TO HAVE A STRAWBERRY? ▸ 20249݄ͷOpenAIͷGPT-4
o1ͷެ։લ ▸ strawberryͷrͷΛ͍͑ͤͯͨ͞ΞΧϯ τ͕ݢฒΈౚ݁͞Εͨ… ͳͥʁ ▸ GPT-4 o1෦తʹ"strawberry"ͱݺΕ͍ͯ ͨ… "r"Λےಓཱ͖ͯͯͪΜͱ͑ΒΕΔΑ͏ ʹ͢Δͧͱ͍͏ҙؾࠐΈ ※LLM͜ͷྨͷ៛ີͳਪ͕େͷۤख ▸ ͭ·Γձ༰Λݟ͍ͯΔ Image by TimAlosi.com on flickr, CC-BY-NC-ND 2.0 -
TEXT ... ARE YOU OK? ▸ ʮChatGPTར༻ऀͷϝϯλϧ่յΛݒ೦ʯ →͍·Ͱܧଓతʹݟ͍ͯΔ ▸ ύεϙʔτِ͕Ͱ͖ͨͱ͍͏ใࠂޙɺ2࣌ؒఔ
Ͱີ͔ʹϓϩϯϓτ͕෧͞Εͨࣄ݅ →΄΅ϦΞϧλΠϜʹݟ͍ͯΔ ▸ DALL-Eʹ͓͍ͯϓϩϯϓτੳ →ؙདͰ͢Αɺͱ Image by TimAlosi.com on flickr, CC-BY-NC-ND 2.0 -
TEXT WE ARE WATCHING YOU, SIR ▸ ձ༰͔Βࣗಈతʹ௨ใ͞ΕΔ͜ͱ͋Δ ▸ Meta
AIͷγεςϜϓϩϯϓτʹʮෆ๏ߦҝ͕ ٙΘΕΔ߹ʹฦͤͣଈ࠲ʹ௨ใͤΑʯ ͱ͍͏هࡌ͕͋Δ ▸ ͲΕ͚ͩޮ͍͍ͯΔͷ͔͔Βͳ͍͕… llama͕πʔϧ͑Δ͜ͱΛߟྀ͢Δͱͦ͜ ͦ͜ػೳ͍ͯ͠ΔՄೳੑ Image by telmo32 on flickr, CC-BY-ND 2.0 -
TEXT YES, LOOKING AT YOU, SIR ▸ ࢹใར༻͕Ճ͖ͯͨ͠ ▸ ྙཧنఆڝ૪༏Ґੑ͔Βͷཁ
▸ ChatGPTग़ݱॳ಄ͷใྲྀग़ٙͱผ →opt-outՄೳͩͬͨ →APIͦͦରͰͳ͔ͬͨ ▸ ͜Μͳͷ৴པͰ͖Δ͔ʁ →ʮѱ༻ࢭʯʮCSAMʯͱຐ๏ͷݴ༿ (·ͨOpenAIӴ࢈ۀೖΓ͍ͯ͠Δ) Image by loren chipman on flickr, CC-BY-NC 2.0 -
TEXT TRUMP MADNESS ▸ ୈೋ࣍τϥϯϓݖ ▸ ࣗࠃ༏ઌओٛͱอ ▸ શͯຽओౘͱҠຽͷ͍ͤ ▸
ۙࢹ؟తํࡦ ▸ DEI ▸ Ҡຽഉআࡦ →·͞ʹϑΝγζϜతͳ͜ͱ͕ى͖͍ͯΔ Image by IoSonoUnaFotoCamera on flickr, CC-BY-SA 2.0 -
TEXT AT YOUR SERVICE, PRESIDENT ▸ ͍ΘΏΔΦϦΨϧώͷελϯυϓϨʔ͕ͻͲ͍ ▸ ӨͷͱԽ͍ͯ͠Δײ৮ →ಛʹΠʔϩϯͱDOGE
▸ DEIͷศ͔ΒҰؾʹӈԽ →ϩϏʔ׆ಈͷҰͱͯ͠ͷศʁ Image by dmoberhaus on flickr, CC-BY 2.0 -
TEXT OUTFLOWING MADNESS ▸ ٸܹͳن੍؇ํͷಈ͖ ▸ ࣏తѹྗΛ͔͚Δಓ۩ʹͳ͍ͬͯΔ ▸ ใऩू͓Αͼར༻ଟଓ͘ ▸
Ϟϥϧϋβʔυ݈ࡏ ▸ ࣃࢭΊ֎Ε͖͍ͯͯΔ (≒Γ͍ͨ์) Image by shioshvili on flickr, CC-BY-SA 2.0 -
TEXT YES THEY ARE WATCHING US ▸ ϓϥΠόγʔ͕͔ͭͯͳ͍΄Ͳةݥʹ ▸ ͜Ε·ͰΫϥυʹ͍ͭͯݴΘΕ͍͕ͯͨ
→CSAMޡೝɺใ෮తใࠂ ▸ ੜAIѱ༻ࢭͳͲͷҙຯ߹͍ͰαʔϏε ࢹ͞Ε͍ͯΔ →Ͳ͜ͰੜAI…ʁ ▸ Google: ܉ࣄࢹʹར༻͠ͳ͍ͱ͍͏දه Λີ͔ʹམͱ͍ͯ͠ΔʢGemini…ʁʣ Image by ashley rose, on flickr, CC-BY-NC-ND 2.0 -
TEXT YES THEY ARE WATCHING US ▸ ϓϥΠόγʔ͕͔ͭͯͳ͍΄Ͳةݥʹ ▸ xGrok
3ʹมͳ͜ͱΛฉ͍͍ͯΔਓͨͪ… ▸ ࠃͷੜAIαʔϏεʹมͳϓϩϯϓτΛ༩ ͍͑ͯΔਓͨͪ… ▸ ͷةݥ͕ഭ͍ͬͯΔͱݴͬͯࢧোͳ͍ ▸ ࠓ͙͢ʹΊͨ΄͏͕ྑ͍ ʢຊͰཱͪೖΓௐࠪ͞ΕΔՄೳੑʣ Image by ashley rose, on flickr, CC-BY-NC-ND 2.0 -
TEXT PRIVACY PLEASE, SIR ▸ ւ֎Ͱఏى͕ͳ͞Ε͖ͯͨ ▸ 9݄16ͷnatureͷهࣄ "Forget ChatGPT:
why researchers now run small AIs on their laptops" ▸ ݚڀऀͨͪOpen-weightͷLLMΛࣗͷϚ γϯͰಈ͔͍ͯ͠Δͱ͍͏ (Local LLM) ▸ techܥ͚ͩͰͳ͘ඇtechܥ Image by calebjcook on flickr, CC-BY-NC-ND 2.0 -
TEXT BECAUSE MY THOUGHTS ARE .. ▸ ೳྗྼΔͷʹͳͥʁ ▸ ࣗͷϚγϯͰ͋ΕશʹϓϥΠϕʔτ
▸ ୭ʹ௨ใ͞Εͳ͍ ▸ ใΛར༻͞Εͳ͍ ▸ ࣗ༝ͳࢥࡧɾਪɾత࡞ۀ͕Ͱ͖Δ Image by Frankenstein on flickr, CC-BY-NC 2.0 -
TEXT .. NONE OF YOUR BUSINESS ▸ ͞Βʹ ▸ ֤ྖҬͷࣝਓݚڀऀ͕ɺӦརڝ૪ʹΓ
͡Ί֤ͨࣾʹةػײΛ͍࣋ͬͯΔ͜ͱͷදΘΕ Image by Frankenstein on flickr, CC-BY-NC 2.0 -
TEXT .. AM I WRONG? ▸ ຊͰͲ͏͔ ▸ NICTPFNΛ࢝Ίͱͯ͠Ϟσϧ͕࡞ΒΕ͍ͯΔ ͕…ͧͬͯ͜Closedɺग़དྷ……………
▸ ࢹ͢ΔΤϯςΟςΟ͕ػؔʹ →͜Ε͜ΕͰ͍ͩͿେ͖ͳ ▸ ͏ͷͰ͋ΕΫϥυ্ʹdeploy͢Δ͔ →Ͱ͖ͳ͚ΕϋʔυΣΞ͝ͱങ͏͜ͱʹ →େมʹ͍ʹ͍͘ Image by Toni Blay on flickr, CC-BY-NC-ND 2.0 -
TEXT CLOSED MODELS ARE NOT YOUR FRIENDS ▸ OpenͳͷSwallow͘Β͍ ▸
ຊޠରԠͷ͠͞ΞυόϯςʔδͰ͋Δ ͷ͕ͩɺͦ͜ʹด͡ࠐ͍ͬͯΔΑ͏ʹݟ͑Δ ▸ ϘϥϯςΟΞҙࣝͷࠩͳͷ͔ɺͦΕͱ୯ͳΔ ॹతن੍Λ༏ઌ͢ΔͨΊͩΖ͏͔ ▸ ຊͷݚڀऀ͕ຊޠͰࣗ༝ʹߟ͑Δʹ͓ۚ ΛࢧΘͳ͚ΕͳΒͳ͍…ͱɺ͏ʔΜ Image by WilsonB on flickr, CC-BY-SA 2.0 -
TEXT ESCAPE PLAN A ▸ Local LLMڥ: ࣗͷϚγϯͰinfer͢Δ ▸ ॴ
▸ શʹϩʔΧϧʹ༓ดͰ͖Δ ▸ ԿΛಈ͔ͯ͠ͳ͍ ▸ ॴ: ▸ ܭࢉೳྗͱϝϞϦ͕ඞཁ ʢe.g. 8B:q8_0 -> 8.5GB, 70B:q4_K_M -> 43GBʣ ▸ Open-weightܥϞσϧͷΈ༻Ͱ͖Δ -
TEXT ESCAPE PLAN A ▸ Local LLMڥྫ ▸ ollama API
ollama + Open-WebUI ▸ OpenAI-compatible API llama-cpp / vLLM + Open-WebUI llama-cpp / vLLM + LM Studio ... ※LM StudioproprietaryͰ͋Δ͜ͱʹҙ ▸ খنͳΒllama.cppɺେنͳΒvLLM͕༏ल (Flash Attention 2 vs. Paged Attention, ฒྻੑೳ) -
TEXT ESCAPE PLAN B ▸ ४Local LLMڥ: ͕ࣗॴ༗͢ΔΫϥυܭࢉڥ ▸ ॴ
(※AWS) ▸ ASICͰͷinferͳͷͰߴ ▸ ServerlessͰ༻Ͱ͖ΔϞσϧ͕ଟ͍ (e.g. Claude, Deepseek R1, LLaMA 3.1 405B etc.) ▸ ॴ ▸ ίετ͕ൃੜ͢Δ ▸ ϩʔΧϧʹด͡ΒΕͳ͍ Image by auxesis on flickr, CC-BY-SA 2.0 -
TEXT ESCAPE PLAN B ▸ Ϋϥυܭࢉڥͱ… ▸ ୯ͳΔߴͳܭࢉڥ →͖ͳϞσϧΛ༻Ͱ͖Δͷ ▸
APIαʔόͰͳ͍͜ͱʹҙ (cf. Azure vs. AWS) Image by auxesis on flickr, CC-BY-SA 2.0 -
TEXT ESCAPE PLAN B ▸ ४Local LLMڥྫ (※AWS) ▸ BedrockͰOpenAI-compatible
APIΛग़͠… …ΫϥΠΞϯτ͔Βར༻ ▸ ͕ͩBedrock୯ମͰOpenAI-compatible API͕ ग़ͤͳ͍ Image by auxesis on flickr, CC-BY-SA 2.0 -
TEXT ESCAPE PLAN B ▸ bedrock-access-gatewayʂ ▸ AWS Sampleͱ্͕͍ͯͬͯ͠Δͷ https://github.com/aws-samples/bedrock-access-
gateway/ ▸ BedrockΛ༻͢ΔServerlessͳOpenAI- compatible APIαʔό࣮ɺͳͷ͕ͩ ▸ CloudFormationͰVPC͔ΒALBΒͳʹΒશͯ ࡞ͬͯ͘ΕΔ… ͔͠1-clickϘλϯͰىಈͱ͔ ▸ େมʹ͍ʹ͍͘ -
TEXT ESCAPE PLAN B ▸ haversackʂ ▸ https://github.com/alterakey/haversack ▸ SAMͰॻ͖͠ɺ͔ͭ༨ܭͳϦιʔε࡞Λ
ഉͨ͠ͷ ▸ ECS༻ͷΠϝʔδΛϕʔεʹ͢Δ͜ͱͰ ▸ ALBAPI Gatewayෆཁʹʂ ʢ୯ҰͷLambda Function URLͰఏڙʣ Image by avlxyz on flickr, CC-BY-SA 2.0 -
TEXT ESCAPE PLAN B ▸ haversackʂ ▸ bedrock-access-gatewayͷECRϩάΠϯޙ sam build,
sam deploy -gͰߏஙdone ▸ ͪͳΈʹhaversackɺͱ AD&Dʹొ͢Δຐྗ͕͔͔ͬͨbagͷҰछ Image by avlxyz on flickr, CC-BY-SA 2.0 -
TEXT ESCAPE PLAN B ▸ ४Local LLMڥྫ (※AWS) ▸ Bedrock
+ haversack + Open-WebUI ▸ Bedrock + haversack + LM Studio ▸ Bedrock + haversack + etc.. ▸ શͯserverlessʂ -
TEXT CAVEATS ▸ Ϟσϧͷઃఆʹ͍ͭͯ (e.g. System Prompt, context size, temp.
etc.) ▸ ެࣜAPIʹ͓͚ΔఆͱɺOpen-WebUILM StudioͳͲͷҰൠతΫϥΠΞϯτ͕ఆͯ͠ ͍Δͱ͔ͳΓҧ͏ ▸ ࠷େݶʹ׆༻͢Δʹ୳ࡧ͢ΔΑΖ͠ Image by Aleksey Gureiev on flickr, CC-BY-NC-ND 2.0 -
TEXT TAKEAWAYS ▸ ੜAI୯ͳΔʮิϞσϧʯ →ʮͦΕͬΆ͍ʯ͜ͱΛੜ͢ΔϞσϧ →seedͱձͰܾ·Δੈքઢͷॅਓ ▸ ྙཧنఆΛ६ʹͨ͠ࢹใऩू͕ಛʹܹԽ →લͷΑ͏ʹopt-out͏Ͱ͖ͳ͍ →ࣗͷϚγϯͰLLMΛಈ͔͢ͷ͕ւ֎Ͱओྲྀ
▸ ೳྗͷڪාݒ೦͕ग़͍ͯΔ →ਖ਼͍͠ϦεΫධՁ͕େࣄ Image by caste_aka_adrem on flickr, CC-BY-NC 2.0 -
TEXT TAKEAWAYS ▸ ୈೋ࣍τϥϯϓݖʹΑΔ →ͦΕʹ܈͕ΔΦϦΨϧώͷӨͱࡦ ▸ ϓϥΠόγʔ͕͔ͭͯͳ͍΄Ͳةݥʹ →ʮͲ͜ͰੜAIʯʮͲ͜Ͱൈ͚ʯ Image by
IoSonoUnaFotoCamera on flickr, CC-BY-SA 2.0 -
TEXT TAKEAWAYS ▸ ຊʹ͓͚ΔLocal LLMͷར༻ʹ →Open-weightͳϞσϧ͕Γͳ͍… →ਅͷҙຯͰࣗ༝ͳࢥࡧͰ͖ͳ͍ ▸ ४LocalͰ͋Ε·ͩͳΜͱ͔ →ͨͩίετ͔͔Δ
▸ OpenAI-compatible API͔ollama APIΛग़ͤ ༷ʑͳΫϥΠΞϯτ͔Βར༻ՄೳʹͳΔ Image by Chris Randall on flickr, CC-BY-NC 2.0 -
TEXT ONE MORE THING ▸ Deepseekࣄ݅ ▸ ใྲྀग़͕ͱ͔ར༻ن੍Λͱ͔ͳΜͱ͔૽͕Ε ͍ͯΔ͕ɺ͋͘·ͰΞϓϦެࣜαΠτͷ ▸
୯ʹinfer͢Δ͚ͩͰ͋ΕԿΒӨڹͳ͍ →४LocalڥͰ҆͝৺Λ ▸ Open-WebUIWebϕʔε… ͱ͍͏͜ͱϞόΠϧ͔ΒͰར༻Մೳʂ →ΞϓϦྨҰෆཁ Image by dakohuang on flickr, CC-BY 2.0