GPT-4o with iOS

「Mobile勉強会 Wantedly × チームラボ × Sansan #14」での発表資料です。
https://wantedly.connpass.com/event/316051/

記事：
[GPT-4oのマルチモーダル入力をiOSから試す](https://zenn.dev/shu223/articles/gpt4o-multimodal)

Transcript

1. GPT-4o with iOS ϚϧνʔϞʔμϧAPIʹඋ͑Δ అ मҰ ʢ@shu223ʣ
2. ࣗݾ঺հ • అ मҰ • @shu223 (GitHub, Zenn, Qiita, note, Docswell, 𝕏, YouTube, Podcast, etc...) • ॻ੶ʢ঎ۀग़൛4࡭ɺݸਓग़൛ଟ਺ @BOOTHʣ:
3. 2024.5.13 GPT-4oൃද Hello GPT-4o | OpenAI 1ߦ໨ɿ We’re announcing GPT-4o, our new flagship model that can reason across audio, vision, and text in real time. ʮϚϧνϞʔμϧʯ͕؊
4. GPT-4oͷϞμϦςΟ GPT-4o (“o” for “omni”) is a step towards much more natural human-computer interaction—it accepts as input any combination of text, audio, image, and video and generates any combination of text, audio, and image outputs. • ೖྗɿ ςΩετɾԻ੠ɾը૾ɾಈըͷ͋ΒΏΔ૊Έ߹Θͤ • ग़ྗɿ ςΩετɾԻ੠ɾը૾ͷ͋ΒΏΔ૊Έ߹Θͤ
5. End-to-end With GPT-4o, we trained a single new model end-to- end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network. Ի੠ → [Whisper] → ςΩετ → [GPT] → ςΩετ → [TTS] → Ի੠ ⇓ Ի੠ → [GPT] → Ի੠
6. ʮϚϧνϞʔμϧίϛϡχέʔγϣ ϯʯͷओઓ৔ͱ͍͑͹ɾɾɾ
7. εϚϗ • ͍ͭͰ΋Ͳ͜Ͱ΋࢖͑Δ • ΈΜͳ࣋ͬͯΔ • ϚΠΫɾΧϝϥ׬උ • ʢςΩετೖྗΊΜͲ͍͘͞ʣ
8. ຊൃදͷςʔϚɿ ϞόΠϧΞϓϦ ⇔ GPT ͷϚϧνϞʔμϧίϛϡχέʔγϣϯʹඋ͑Α͏
9. ΋͏API͸ରԠ͍ͯ͠Δͷ͔ʁ
10. Chat Completion APIରԠঢ়گ ϞσϧʹGPT-4oΛࢦఆՄೳ add gpt-4o model (#1417) · openai/openai-python
11. ֤छϞμϦςΟͷAPIରԠঢ়گ ϞμϦςΟ ೖྗ ग़ྗ ςΩετ ◯ ◯ ը૾ ◯ ✕ ಈը △ʢޙड़ʣ - Ի੠ ✕ ✕
12. Chat Completion APIʹը૾ɾಈըΛ౤͛Δ API Reference - OpenAI API • ը૾ɿ "image_url" ʹը૾σʔλ΍URLΛೖΕΔ • ಈըɿ ϑϨʔϜը૾Λෳ਺ೖΕΔ "content": [ { "type": "image_url", "image_url": ... } ]
13. ಈըཧղ͸Մೳ͕ͩɺಈըͱ͍͏ϞμϦςΟʹ ׬શʹରԠ͍ͯ͠Δͱ͸ݴ͍೉͍ To-Be: ಈը → [GPT] → ग़ྗ As-Is: ಈը → ෳ਺ͷը૾ → [GPT] → ग़ྗ
14. ʢิ଍ʣಈըΛෳ਺ը૾ʹ෼ղͯ͠ૹΔํ๏ͷ໰୊఺ • End-to-endͰֶश͞ΕͨGPT-4oͷϞμϦςΟΛ׆͔͍ͤͯͳ ͍ • ΫϥΠΞϯταΠυͰಈըͷσίʔυॲཧͷΦʔόʔϔου • ಈըϑΥʔϚοτΛσίʔυͯ͠ૹΔͨΊσʔλྔ΋૿͑Δ • Ի੠৘ใ΋࢖༻͞Εͳ͍
15. ʢิ଍ʣOpenAI APIͷVisionػೳʹ͍ͭͯ • GPT-4 Turbo with VisionϞσϧ͕ϦϦʔε͞Εͨ2023೥ 11݄ʹAPI͕௥Ճ • ಉ݄ɺOpenAIͷAPIυΩϡϝϯτʹVisionػೳͷ࢖͍ํ͕௥ Ճ • ಈըΛݸʑͷϑϨʔϜը૾ʹ෼ׂͯ͠Ϟσϧʹೖྗ͢Δ͜ͱ Ͱಈըͷ಺༰ཧղ͕Ͱ͖Δͱઆ໌͞Ε͍ͯΔ → ͭ·Γ͜ͷ΁Μ͸GPT-4oͷ৽ػೳͰ͸ͳ͍
16. GPT-4oͷϚϧνϞʔμϧೖྗΛiOS͔ Βࢼ͢
17. σϞ - iOS×GPT-4oͰϦΞϧλΠϜ ಈըཧղ
18. iOSͰ࢖͑ΔOpenAI APIΫϥΠΞϯτ ͍Ζ͍Ζ͋Δ ͕ɺݱஈ֊Ͱ͸ MacPaw/OpenAI ͰΑͦ͞͏ • ελʔ਺͕΋ͬͱ΋ଟ͍ʢ㲈ར༻࣮੷ʣ • ௚ۙͰ΋ϝϯς͞Ε͍ͯΔ • ґଘϥΠϒϥϦͳ͠ • ετϦʔϛϯάAPIɺը૾ੜ੒ɺTTS౳΋αϙʔτ
19. MacPaw/OpenAI ͸ Vision API Λαϙʔτ ͍ͯ͠Δͷ͔ʁ READMEʹ͸Կ΋ॻ͍ͯͳ͍͕ɺιʔείʔυΛಡΉͱ࣮͸αϙʔ τ͍ͯ͠Δ 1 1 ௐࠪͷৄࡉɿ MacPaw/OpenAI ͸ϚϧνϞʔμϧೖྗʹରԠ͍ͯ͠Δͷ͔
20. iOS×GPT-4oͰը૾ཧղ • ϓϩϯϓτɿ "What's in this image?" • ʢίʔυ͸ެ։༧ఆʣ
21. iOS×GPT-4oͰಈըͷཁ໿ • ϓϩϯϓτɿ"ಈըͷཁ໿Λఏڙͯ͘͠ ͍ͩ͞ɻ" • 60sͷಈը͔Β2ඵ͓͖ʹϑϨʔϜநग़ ʗ௕ล768pxʹͳΔΑ͏ϦαΠζ 2 • ʢίʔυ͸ެ։༧ఆʣ 2 ؔ࿈هࣄɿ GPTʹΑΔಈըཧղͷOpenAIެࣜΨΠυΛಡΉ
22. iOS×GPT-4oͰϦΞϧλΠϜ ಈըཧղ • "detail" ͸ "low" Λࢦఆ • 512x512ʹϦαΠζ&Ϋϩοϓ • લϑϨʔϜͷઆ໌ςΩετ΋ಉࠝ • ฒྻԽ͸·ͩ • ʢίʔυ͸ެ։༧ఆʣ
23. ετϦʔϛϯάೖग़ྗʹඋ͑Δ ϞμϦςΟ ೖྗ ग़ྗ ςΩετ ◯ ◯ ը૾ ◯ ✕ ಈը ͜͜ - Ի੠ ͜͜ ͜͜
24. ͜ͷ΁ΜΛώϯτʹඋ͓͑ͯ͘ͱΑͦ͞͏ʁ • Whisper APIɾɾɾԻ੠ೖྗAPI • TTS APIɾɾɾԻ੠ग़ྗAPI
25. Whisper APIʢԻ੠ೖྗʣ • ετϦʔϛϯάೖྗ͸αϙʔτͯ͠ͳ͍ • ʮԻ੠ΛνϟϯΫ͝ͱʹૹΔʯํࣜͰετϦʔϛϯάΛ࣮ݱ͠ ͍ͯΔαʔυύʔςΟʔ࣮૷͸͍Ζ͍Ζ͋Δ
26. TTS APIʢԻ੠ग़ྗʣ • ετϦʔϛϯάग़ྗΛαϙʔτ͍ͯ͠Δ • υΩϡϝϯτɿ Text to speech - OpenAI API • Python࣮૷ • ࢒೦ͳ͕Β MacPaw/OpenAI Ͱ͸ετϦʔϜग़ྗ͸αϙʔτ ͍ͯ͠ͳ͔ͬͨʢιʔεಡΜͰ֬ೝʣ • ʢContributionνϟϯεʂʣ
27. ৽APIͷϦϦʔεʹؾ෇͘ʹ͸ ެࣜPythonϥΠϒϥϦͷreleaseΛwatch͢Δ https://github.com/openai/openai-python/releases
28. ·ͱΊ • GPT-4oͷʮϚϧνϞʔμϧʯ͸εϚϗͰେ׆༂ͦ͠͏ • OpenAI APIͷݱࡏͷରԠঢ়گ • iOSͰͷʮGPT-4oΛ༻͍ͨը૾ཧղɾಈըཧղʯͷ࣮૷ • ετϦʔϛϯάೖग़ྗରԠʹඋ͑Α͏
29. ͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ