生成AIとPythonで ラズパイカーを動かそう！〜GradioとLangChainで作るプロトタイプ〜

ΈΜͳͷ1ZUIPOษڧձ ੜ੒"*ͱ1ZUIPOͰ ϥζύΠΧʔΛಈ͔ͦ͏ʂ ʙ(SBEJPͱ-BOH$IBJOͰ࡞ΔϓϩτλΠϓʙ 5PNPLP'VSVLJ !LPNP@GS

9!LPNP@GS w 5PNPLP'VSVLJ w 9ʢ5XJUUFSʣ!LPNP@GS w ॴଐגࣜձࣾϏʔϓϥ΢υ w ීஈ͸1ZUIPOΛ࢖ͬͨσʔλαΠΤϯε෼໺ͷٕज़ࢧԉ΍ɺ γεςϜ։ൃͳͲʹैࣄ
w ࠷ۙͷϒʔϜ--.Λ࢖ͬͨখ͞ͳझຯϓϩάϥϜΛ࡞ͬͨΓ͢Δ͜ͱ "CPVUNF

9!LPNP@GS w --.1JDP$BS w ੜ੒"*Λར༻ͨ͠ɺԻ੠΍ը૾Ͱಈ͓͘΋ͪΌͷं w ͲΜͳϥΠϒϥϦ΍ٕज़Λ࢖ͬͨͷ͔ w ࡞Δ͏͑Ͱ޻෉ͨ͠఺ ࠓ೔࿩͢͜ͱ

9!LPNP@GS w ͭͭͷٕज़͸ͱͬͯ΋؆୯ʂʢνϡʔτϦΞϧϨϕϧʂʣ w (SBEJPػցֶशͷσϞΞϓϦͳͲΛ؆୯ʹ࡞ΕΔ8FCΞϓϦͷϑϨʔϜ ϫʔΫɻԻ੠ೖྗ΍ը૾ೖྗ΋؆୯ʹ࡞ΕΔ w -BOH$IBJO--.ΞϓϦ։ൃͷϑϨʔϜϫʔΫ w .JDSPEPU3BTQCFSSZ1J1JDP্Ͱ΋ಈܰ͘ྔͷ8FCϑϨʔϜϫʔΫ
👉ੜ੒"*ͱ*P5Λ૊Έ߹ΘͤΔΞΠσΞͷώϯτʹͳΕ͹͏Ε͍͠Ͱ͢ ࠓ೔ͷϙΠϯτ

9!LPNP@GS wԿΛ࡞ͬͨͷ͔ wͲ͏΍ͬͯ࡞ͬͨͷ͔ w4UFQ8FCΞϓϦ͔ΒԻ੠Λೖྗ w4UFQԻ੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵ w4UFQίϚϯυΛड৴ ࣮ߦ w݁Ռ wԠ༻ը૾ʹΑΔࢦࣔɺNJDSPCJUͱͷ࿈ܞ
w·ͱΊ Ξ΢τϥΠϯ

ԿΛ࡞ͬͨͷ͔

9!LPNP@GS w ੠΍ը૾ʹΑΔࢦࣔͰಈ͓͘΋ͪΌͷं w ʮ੠ͰখܕΧʔʹࢦࣔΛग़ͤͨΒɺϖοτΆͯ͘ՄѪ͍ͷͰ͸ʁʯͱ͍͏ ΞΠσΞ͔Βੜ·Εͨ ·͙ͬ͢ߦͬͯ ӈʂ གྷͬͯΈͯʂ --.1JDP$BS
ԿΛ࡞ͬͨͷ͔

9!LPNP@GS w ੜ੒"*Λ࢖͑͹ɺࣗવݴޠˠߏ଄Խ͞Εͨࢦࣔσʔλʹ؆୯ʹม׵Ͱ͖ͦ͏ w ϓϩϯϓτͷΠϝʔδ ʮࣗવݴޠʹΑΔࢦࣔΛɺࣙॻͷϦετܗࣜͰίϚϯυʹม׵͠ͳ͍͞ʯ ·͙ͬ͢ߦͬͯ ӈʹۂ͕ͬͯʂ [ {
"action": { "type": "forward", "distance_cm": 20,}}, { "action": { "type": "turn", "direction": "right", "angle": 90,}}, ...] ࣗવݴޠ ߏ଄Խ͞ΕͨࢦࣔσʔλʢࣙॻͷϦετʣ લʹDNਐΉ ӈʹ౓ճస ੜ੒"* ΞΠσΞ ԿΛ࡞ͬͨͷ͔

9!LPNP@GS w ʮ·͙ͬ͢ߦͬͯɺӈʹͪΐͬͱਐΜͰʯ w ʮགྷͬͯΈͯʯ %&.0--.1JDP$BS ԿΛ࡞ͬͨͷ͔ IUUQTXXXZPVUVCFDPNXBUDI WHM&C&;TTX3"

Ͳ͏࡞ͬͨͷ͔

9!LPNP@GS w ,JUSPOJL3BTQCFSSZ1J1JDP༻ࣗ૸ϩϘοτϓϥοτϑΥʔϜʢόΪʔλΠϓʣ w IUUQTLJUSPOJLDPVLQSPEVDUTBVUPOPNPVTSPCPUJDTQMBUGPSNGPSQJDP ࠓճ͸ະ࢖༻͚ͲҎԼ΋͍ͭͯΔ w ڑ཭ηϯα w αʔϘίωΫλ
w ϥΠϯτϨʔε༻ηϯα ंྠ෇͖Ϟʔλʢݸʣ -&% ϒβʔ -&% ࠓճ࢖ͬͨػମʢ1JDP$BSʣ Ͳ͏࡞ͬͨͷ͔ 3BTQCFSSZ1J1JDP

9!LPNP@GS w ిࢠ޻࡞ͷܦݧ͕͋·Γͳ͍ͷͰɺࢢൢͷΩοτΛ࢖͍͍ͨ w Ϧν΢ϜΠΦϯి஑͸؅ཧ͕େมͦ͏ͳͷͰɺסి஑Ͱಈ͔͍ͨ͠ w 3BTQCFSSZ1J❌סి஑͚ͩͰಈ͔͢ͷ͸ݫ͍͠ w 3BTPCFSSZ1J1JDP✅סి஑͚ͩͰಈ͘ʂ 👉3BTQCFSZ1J1JDP
סి஑Ͱಈ͘ຊΩοτʹͨ͠ બఆͷཧ༝ Ͳ͏࡞ͬͨͷ͔

9!LPNP@GS w ిࢠ޻࡞ͷܦݧ͕͋·Γͳ͍ͷͰɺࢢൢͷΩοτΛ࢖͍͍ͨ w Ϧν΢ϜΠΦϯి஑͸؅ཧ͕େมͦ͏ͳͷͰɺסి஑Ͱಈ͔͍ͨ͠ w 3BTQCFSSZ1J❌סి஑͚ͩͰಈ͔͢ͷ͸ݫ͍͠ w 3BTPCFSSZ1J1JDP✅סి஑͚ͩͰಈ͘ʂ 👉3BTQCFSZ1J1JDP
סి஑Ͱಈ͘ຊΩοτʹͨ͠ બఆͷཧ༝ Ͳ͏࡞ͬͨͷ͔ 3BTQCFSSZ1J1JDP8) XJUI8J'JBOEIFBEFSQJOT

9!LPNP@GS 8FCΞϓϦ͔Β Ի੠Λೖྗ ίϚϯυΛड৴ ࣮ߦ Ի੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵ Ի੠σʔλ
ߏ଄Խ͞Εͨ ࢦࣔσʔλ 4UFQ 4UFQ 4UFQ શମͷߏ੒ Ͳ͏࡞ͬͨͷ͔ εϚϗͳͲͷ 8FCϒϥ΢β 1JDP$POUSPMMFS4FSWFS ʢ.BDʣ 1JDP$BS4FSWFS ʢ3BTQCFSSZ1J1*DP$BSʣ

9!LPNP@GS w 1JDPΧʔ୯ମͩͱԻ੠ೝࣝ΍ੜ੒"*Λ࢖͏ͷ͕೉͍͠ ʢॲཧ͕ॏ͍ɺ.JDSP1ZUIPOͩͱҰ෦ϥΠϒϥϦΛ࢖͑ͳ͍ʣ 👉Ի੠ೝࣝ΍ੜ੒"*Λ࢖͏ॲཧ͸ɺ.BD্Ͱಈ͘8FCαʔόʹ೚ͤͨ 1JDPΧʔͰ͸Ϟʔλ΍-&%ͷ੍ޚ͚ͩΛߦ͏ w ʮ1$ʹ࿩͔͚͠Δʯܗͩͱʮ1JDPΧʔʹ࿩͔͚͍ͯ͠Δʯײ͕ബ͘ͳΔ 👉8FCϒϥ΢βܦ༝ͷ6*ʹͯ͠ɺεϚϗ͔Β΋ૢ࡞Ͱ͖ΔΑ͏ʹͨ͠ ʢτϥϯγʔόʔతͳΠϝʔδͰʣ
ߏ੒ͷཧ༝ Ͳ͏࡞ͬͨͷ͔

9!LPNP@GS εϚϗͳͲͷ 8FCϒϥ΢β 1JDP$POUSPMMFS4FSWFS ʢ.BDʣ 1JDP$BS4FSWFS ʢ3BTQCFSSZ1J1*DP$BSʣ 8FCΞϓϦ͔Β Ի੠Λೖྗ ίϚϯυΛड৴
࣮ߦ Ի੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵ Ի੠σʔλ ߏ଄Խ͞Εͨ ࢦࣔσʔλ 4UFQ 4UFQ 4UFQ 4UFQ8FCΞϓϦ͔ΒԻ੠Λೖྗ

9!LPNP@GS w (SBEJPͱ͸ w ػցֶशͷσϞΞϓϦͳͲΛ؆୯ʹ࡞ΕΔ8FCΞϓϦϑϨʔϜϫʔΫ w Ի੠ೖྗ΍ը૾ೖྗͳͲͷ෦඼͕ɺ͋Β͔͡Ί༻ҙ͞Ε͍ͯΔ 👉ϑϩϯτΤϯυͷ஌͕ࣝͳͯ͘΋؆୯ʹը໘Λ࡞ΕΔ (SBEJPͰ8FCΞϓϦΛ࡞੒ 4UFQ8FCΞϓϦ͔ΒԻ੠ೖྗ

9!LPNP@GS Ի੠ೖྗίϯϙʔωϯτ ૹ৴ϘλϯϦηοτϘλϯ จࣈى͜͠ͷ݁ՌΛදࣔ͢Δ ςΩετϘοΫε ݱࡏͷεςʔλεΛදࣔ͢Δ ςΩετϘοΫε ߏ଄Խ͞Εͨࢦࣔσʔλ ʢࣙॻͷϦετʣΛ දࣔ͢ΔΤϦΞ
ਐḿϥϯϓ ࠓճ࡞ͬͨ6* 4UFQ8FCΞϓϦ͔ΒԻ੠ೖྗ

ਐḿϥϯϓ ࠓճ࡞ͬͨ6* 4UFQ8FCΞϓϦ͔ΒԻ੠ೖྗ ೖྗΤϦΞ

ਐḿϥϯϓ ࠓճ࡞ͬͨ6* 4UFQ8FCΞϓϦ͔ΒԻ੠ೖྗ ग़ྗΤϦΞ

9!LPNP@GS import gradio as gr ... with gr.Blocks() as demo:
... with gr.Tab("Audio Instructions"): # ίϯϙʔωϯτΛ഑ஔ audio_input = gr.Audio( sources=["microphone"], type="filepath", format="wav") with gr.Row(): audio_send_button = gr.Button("Submit", variant="primary") audio_reset_button = gr.Button("Reset", variant="secondary") ... # ΠϕϯτΛઃఆ audio_send_button.click(handler_audio_input, inputs=audio_input, outputs=[audio_transcript_output, audio_commands_output, audio_status_output, progress_lamps,],) ... # αʔόΛىಈ demo.launch(server_name="0.0.0.0", share=True) HS"VEJP HS#VUUPO HS#VUUPO HS5FYUCPY HS5FYUCPY HS+40/ TUSVDUVSFEBDUJPOT ίʔυ(SBEJP#MPDLT HS)5.- 4UFQ8FCΞϓϦ͔ΒԻ੠ೖྗ

... with gr.Tab("Voice Input"): # ίϯϙʔωϯτΛ഑ஔ audio_input = gr.Audio( sources=["microphone"], type="filepath", format="wav") with gr.Row(): audio_send_button = gr.Button("Submit", variant="primary") audio_reset_button = gr.Button("Reset", variant="secondary") audio_transcript_output = gr.Textbox(label="Transcription Result") audio_status_output = gr.Textbox(label="Processing Status") audio_commands_output = gr.JSON(label=“Generated Commandas") # ΠϕϯτΛઃఆ ... # αʔόΛىಈ demo.launch(server_name="0.0.0.0", share=True) ίʔυը໘ϨΠΞ΢τ HS"VEJP HS#VUUPO HS#VUUPO HS5FYUCPY HS5FYUCPY HS+40/ TUSVDUVSFEBDUJPOT HS)5.- 4UFQ8FCΞϓϦ͔ΒԻ੠ೖྗ Ի੠ೖྗ༻ ίϯϙʔωϯτ Ϙλϯ ݁Ռग़ྗ༻ͷ ίϯϙʔωϯτ

... with gr.Tab("Voice Input"): # ίϯϙʔωϯτΛ഑ஔ … # ΠϕϯτΛઃఆ audio_send_button.click(handler_audio_input, inputs=audio_input, outputs=[audio_transcript_output, audio_commands_output, audio_status_output, progress_lamps,],) audio_reset_button.click(...) audio_input.start_recording(...) audio_input.stop_recording(...) # αʔόΛىಈ demo.launch(server_name="0.0.0.0", share=True) ίʔυΠϕϯτϋϯυϥ $MJDL 4UFQ8FCΞϓϦ͔ΒԻ੠ೖྗ

... with gr.Tab("Voice Input"): # ίϯϙʔωϯτΛ഑ஔ ... # ΠϕϯτΛઃఆ audio_send_button.click(handler_audio_input, inputs=audio_input, outputs=[audio_transcript_output, audio_commands_output, audio_status_output, progress_lamps,],) audio_reset_button.click(...) audio_input.start_recording(...) audio_input.stop_recording(...) # αʔόΛىಈ demo.launch(server_name="0.0.0.0", share=True) ίʔυΠϕϯτϋϯυϥ ೖྗ ग़ྗ ΫϦοΫ 4UFQ8FCΞϓϦ͔ΒԻ੠ೖྗ

... with gr.Tab("Voice Input"): # ίϯϙʔωϯτΛ഑ஔ ... # Πϕϯτͷઃఆ audio_send_button.click(handler_audio_input, inputs=audio_input, outputs=[audio_transcript_output, audio_commands_output, audio_status_output, progress_lamps,],) audio_reset_button.click(...) audio_input.start_recording(...) audio_input.stop_recording(...) # αʔόΛىಈ demo.launch(server_name="0.0.0.0", share=True) ίʔυΠϕϯτϋϯυϥ .click(handler, inputs=..., outputs=...) audio_transcript_output ΫϦοΫ࣌ʹ࣮ߦ͞ΕΔؔ਺ w Ի੠ࢦࣔͷจࣈى͜͠ w ੜ੒"*ʹΑΔίϚϯυม׵ॲཧ w 1JDPΧʔʹίϚϯυૹ৴ w ʜ audio_status_output audio_commands_output audio_input 4UFQ8FCΞϓϦ͔ΒԻ੠ೖྗ

... with gr.Tab("Voice Input"): # ίϯϙʔωϯτΛ഑ஔ ... # ΠϕϯτΛઃఆ audio_send_button.click(handler_audio_input, inputs=audio_input, outputs=[audio_transcript_output, audio_commands_output, audio_status_output, progress_lamps,],) audio_reset_button.click(...) audio_input.start_recording(...) audio_input.stop_recording(...) # αʔόΛىಈ demo.launch(server_name="0.0.0.0", share=True) ίʔυαʔόͷىಈ 4UFQ8FCΞϓϦ͔ΒԻ੠ೖྗ

9!LPNP@GS w จࣈى͜͠΍ੜ੒"*ͷॲཧ͸͕͔͔࣌ؒΔ w ͦͷͨΊʮ͢΂ͯͷॲཧ͕ऴΘ͔ͬͯΒ݁ՌΛදࣔ͢ΔʯͷͰ͸ͳ͘ ʮ్தͷεςοϓ͝ͱʹɺஈ֊తʹ݁ՌΛදࣔ͢ΔʯΑ͏ʹͨ͠ ᶃจࣈى͜͠׬ྃ ੜ੒"*ॲཧ׬ྃ 1JDPΧʔͷॲཧ׬ྃ ଴ͪ࣌ؒΛײͤ͡͞ͳ͍޻෉
4UFQ8FCΞϓϦ͔ΒԻ੠ೖྗ

9!LPNP@GS audio_send_button.click(handler_audio_input, inputs=audio_input, outputs=[audio_transcript_output, audio_commands_output, audio_status_output, progress_lamps]) def handler_audio_input(audio_file): success,
text = transcribe(audio_file) if success: yield text, None, "จࣈى͜͠׬ྃ", render_lamps(["s", "u", "u"]) commands = text_to_command(text) commands = convert_commands(commands) yield text, commands, "ίϚϯυੜ੒׬ྃ", render_lamps(["s", "s", "u"]) response = send_commands(commands) status = response.get("status") yield text, commands, status, render_lamps(["s"] * 3) else: yield text, None, "จࣈى͜͠ʹࣦഊ͠·ͨ͠ɻऴྃ͠·͢", render_lamps(["e", "u", "u"]) w (SBEJPͰ͸ʮஈ֊తʹ݁ՌΛฦ͢ʯ6*΋؆୯ʹ࡞ΕΔ w ΠϕϯτϋϯυϥͷதͰyieldΛ࢖ͬͯ݁ՌΛฦ͢ ᶈ͢΂ͯͷ݁ՌΛฦ͢ Πϕϯτϋϯυϥ ग़ྗઌͷίϯϙʔωϯτ ᶃจࣈى͜͠Λ࣮ߦ ᶅੜ੒"*Λ࢖ͬͯɺจࣈྻΛߏ଄Խ͞ΕͨίϚϯυʹม׵͢Δ ᶇ1JDPΧʔʹίϚϯυΛૹ৴ ZJMFEΛ࢖ͬͯஈ֊తʹ݁ՌΛฦ͢ ᶆੜ੒"*͕ม׵ͨ͠ίϚϯυΛฦ͢ ᶄจࣈى͜͠ͷ݁ՌΛฦ͢ 4UFQ8FCΞϓϦ͔ΒԻ੠ೖྗ

9!LPNP@GS 1JDP$BS4FSWFS ʢ3BTQCFSSZ1J1*DP$BSʣ εϚϗͳͲͷ 8FCϒϥ΢β 8FCΞϓϦ͔Β Ի੠Λೖྗ ίϚϯυΛड৴ ࣮ߦ
Ի੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵ 4UFQ 4UFQ 4UFQ ߏ଄Խ͞Εͨ ࢦࣔσʔλ Ի੠σʔλ 4UFQԻ੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵ 1JDP$POUSPMMFS4FSWFS ʢ.BDʣ

9!LPNP@GS w Ի੠ͷ··ੜ੒"*ʹೖྗ͢Δͱ"1*ྉۚͷίετ͕͔͔Γͦ͏ 👉ϩʔΧϧͰGBTUFSXIJTQFSΛ࢖ͬͯจࣈى͜͠Λ͢Δ from faster_whisper import WhisperModel model_size =
"tiny" model = WhisperModel(model_size, device="cpu", compute_type="float32") …ུ… segments, _ = model.transcribe(tmp_audio_path, language="ja") text = "".join([segment.text for segment in segments]) ᶃϞσϧͷಡΈࠐΈ ᶄϞσϧͷ࣮ߦʢจࣈى͜͠ʣ Ի੠ˠςΩετʢจࣈى͜͠ʣ 4UFQԻ੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵

9!LPNP@GS w -BOH$IBJO w ೥ʹൃද͞Εͨɺ--.ΞϓϦ։ൃͷͨΊͷ1ZUIPOϑϨʔϜϫʔΫ w -BOH$IBJOܦ༝Ͱ0QFO"*ͷ"1*ʢHQUʣΛݺͼग़͠ɺ ࣗવݴޠͷࢦࣔΛɺߏ଄Խ͞ΕͨࢦࣔσʔλʢࣙॻͷϦετʣʹม׵͢Δ ·͙ͬ͢ߦͬͯ ӈʹۂ͕ͬͯʂ
[{ "action": { "type": "forward", "distance_cm": 20,}}, { "action": { "type": "turn", "direction": "right", "angle": 90,}}, ...] ࣗવݴޠ ߏ଄Խ͞ΕͨࢦࣔσʔλʢࣙॻͷϦετʣ લʹDNਐΉ ӈʹ౓ճస ੜ੒"* ςΩετˠߏ଄Խ͞Εͨࢦࣔσʔλ 4UFQԻ੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵

9!LPNP@GS from langchain.chat_models import init_chat_model from langchain_core.prompts import ChatPromptTemplate ...
prompt = ChatPromptTemplate( [ ("system", TEXT_TO_COMMAND_PROMPT), ("human", "ࢦࣔ: {text}"), ] ) llm = init_chat_model(model="gpt-4.1", temperature=0) chain = prompt | llm result = chain.invoke({"text": text}) instructions = result.content # Add RGB color info based on the LED color name (CSS color name) in the generated data instructions = _add_led_rgb(instructions) instructions = json.loads(instructions) ᶃϓϩϯϓτςϯϓϨʔτͷߏங ᶄϞσϧͷॳظԽ ᶅϓϩϯϓτͱϞσϧΛܨ͙ ᶆJOWPLFϝιουͰ࣮ߦ ίʔυ-BOH$IBJOͷجຊ 4UFQԻ੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵

prompt = ChatPromptTemplate( [ ("system", TEXT_TO_COMMAND_PROMPT), ("human", "ࢦࣔ: {text}"), ] ) llm = init_chat_model(model="gpt-4.1", temperature=0) chain = prompt | llm result = chain.invoke({"text": text}) instructions = result.content # CSS৭໊ΛRGBʹม׵ͯ͠௥Ճ instructions = _add_led_rgb(instructions) instructions = json.loads(instructions) ᶃϓϩϯϓτςϯϓϨʔτͷߏங ᶄϞσϧͷॳظԽ ᶅϓϩϯϓτͱϞσϧΛܨ͙ ᶆJOWPLFϝιουͰ࣮ߦ ίʔυ-BOH$IBJOͷجຊ 4UFQԻ੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵

prompt = ChatPromptTemplate( [ ("system", TEXT_TO_COMMAND_PROMPT), ("human", "ࢦࣔ: {text}"), ] ) llm = init_chat_model(model="gpt-4.1", temperature=0) chain = prompt | llm result = chain.invoke({"text": text}) instructions = result.content # CSS৭໊ΛRGBʹม׵ͯ͠௥Ճ instructions = _add_led_rgb(instructions) instructions = json.loads(instructions) ༩͑ΒΕͨ೔ຊޠͷࢦࣔจΛɺpicoΧʔ༻ͷJSONܗࣜͷಈ࡞ίϚ ϯυ഑ྻʹม׵͍ͯͩ͘͠͞ɻ ࢦࣔͰ͸ͳ͍ݴ༿ΛݴΘΕͨ৔߹ʢʮ͓͸Α͏ʂʯʮ͹͔ʂʯͳͲʣ ͸ɺݴΘΕͨݴ༿ʹର͢Δײ৘Λද͢ಈ࡞ͷίϚϯυ഑ྻʹม׵͠ ͍ͯͩ͘͞ɻ ίϚϯυ഑ྻ͚ͩΛग़ྗ͠ɺͦΕҎ֎ͷઆ໌จͳͲ͸ؚΊͳ͍Ͱ͘ ͍ͩ͞ɻ ֤ίϚϯυ͸ҎԼͷܗࣜΛ࢖༻͍ͯͩ͘͠͞: … ϓϩϯϓτ 1SPNQU 4UFQԻ੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵

prompt = ChatPromptTemplate( [ ("system", TEXT_TO_COMMAND_PROMPT), ("human", "ࢦࣔ: {text}"), ] ) llm = init_chat_model(model="gpt-4.1", temperature=0) chain = prompt | llm result = chain.invoke({"text": text}) instructions = result.content # CSS৭໊ΛRGBʹม׵ͯ͠௥Ճ instructions = _add_led_rgb(instructions) instructions = json.loads(instructions) ༩͑ΒΕͨ೔ຊޠͷࢦࣔจΛɺpicoΧʔ༻ͷJSONܗࣜͷಈ࡞ίϚ ϯυ഑ྻʹม׵͍ͯͩ͘͠͞ɻ ࢦࣔͰ͸ͳ͍ݴ༿ΛݴΘΕͨ৔߹ʢʮ͓͸Α͏ʂʯʮ͹͔ʂʯͳͲʣ ͸ɺݴΘΕͨݴ༿ʹର͢Δײ৘Λද͢ಈ࡞ͷίϚϯυ഑ྻʹม׵͠ ͍ͯͩ͘͞ɻ ίϚϯυ഑ྻ͚ͩΛग़ྗ͠ɺͦΕҎ֎ͷઆ໌จͳͲ͸ؚΊͳ͍Ͱ͘ ͍ͩ͞ɻ ֤ίϚϯυ͸ҎԼͷܗࣜΛ࢖༻͍ͯͩ͘͠͞: … 1SPNQU ϓϩϯϓτ 4UFQԻ੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵

9!LPNP@GS ͋ͳͨՄѪ͍ΘͶʙ w ᐆດͳݴ༿ΛݴΘΕͨͱ͖͸ɺͦͷݴ༿ʹର͢Δײ৘Λද͢ಈ࡞ɾ৭ʹ ม׵͢ΔΑ͏ϓϩϯϓτͰࢦࣔ 👉ᐆດͳݴ༿ΛݴΘΕͨ࣌΋ɺԿΒ͔ͷʮੜ͖෺Β͍͠൓ԠʯΛͯ͘͠ΕΔ ͹͔͹͔ʂ ஥௚Γ͠Α͏ ϐϯΫͰલਐͯ͠ࠨӈʹ;Γ;Γ ʢتͼʁʣ
੨৭Ͱޙୀʢ൵͠Έʁʣ ྘Ͱલਐʢߠఆʁʣ ϓϩϯϓτͷ޻෉ 4UFQԻ੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵

9!LPNP@GS ͋ͳͨՄѪ͍ΘͶʙ w ᐆດͳݴ༿ΛݴΘΕͨͱ͖͸ɺͦͷݴ༿ʹର͢Δײ৘Λද͢ಈ࡞ɾ৭ʹ ม׵͢ΔΑ͏ϓϩϯϓτͰࢦࣔ 👉ᐆດͳݴ༿ΛݴΘΕͨ࣌΋ɺԿΒ͔ͷʮੜ͖෺Β͍͠൓ԠʯΛͯ͘͠ΕΔ ͹͔͹͔ʂ ஥௚Γ͠Α͏ ϐϯΫͰલਐͯ͠ࠨӈʹ;Γ;Γ ʢتͼʁʣ
੨৭Ͱޙୀʢ൵͠Έʁʣ ྘Ͱલਐʢߠఆʁʣ ϓϩϯϓτͷ޻෉ 4UFQԻ੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵ ϐϯΫͰલਐͯ͠ࠨӈʹ;Γ;Γʢتͼʁʣ ϓϩϯϓτͰ͸ɺʮ൵͠Έ͸੨Ͱද͢ʯ ͱ͍ͬͨࢦࣔ͸༩͍͑ͯͳ͍ ੜ੒"*͕ࣗ෼Ͱࣗવͳ৭ΛબͿͷͰ ෯޿͍ݴ༿ʹॊೈʹ൓ԠͰ͖Δ

9!LPNP@GS [ { "action": {"type": "forward", "distance_cm": 20}, "led": {"color":
"green"}}, { "action": {"type": "turn", "direction": "right", "angle": 90}, "led": {"color": "green"}}, ...] BDUJPOӈճస MFE྘ BDUJPOલਐ DN MFE྘ ͭ໨ͷίϚϯυ ͭ໨ͷίϚϯυ ੜ੒͞ΕΔࢦࣔσʔλ 4UFQԻ੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵

9!LPNP@GS [ { "action": {"type": "forward", "distance_cm": 20}, "led": {"color":
"green", "rgb": (0, 255, 0) }}, { "action": {"type": "turn", "direction": "right", "angle": 90}, "led": {"color": "green", "rgb": (0, 255, 0)}}, ...] $44৭໊Λ3(#஋ʹม׵ w ੜ੒"*͸$44ͷ৭໊͚ͩΛग़ྗɻޙॲཧͰ1ZUIPOͰ3(#஋ʹม׵͢Δ w ৭໊ͷํ͕ײ৘Λදͤͦ͏ ਓ͕ؒݟͯཧղ͠΍͍͢ͷͰɺ "*ʹ͸৭໊Λग़ྗͤͨ͞ w $44৭໊͔Β3(#΁ͷม׵͸ɺγϯϓϧͰ࣮֬ʹ΍ΒͤΔͨΊ1ZUIPOʹ೚ͤͨ ੜ੒͞ΕΔࢦࣔσʔλ 4UFQԻ੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵

9!LPNP@GS 1JDP$POUSPMMFS4FSWFS ʢ.BDʣ εϚϗͳͲͷ 8FCϒϥ΢β 8FCΞϓϦ͔Β Ի੠Λೖྗ ίϚϯυΛड৴ ࣮ߦ
Ի੠Λจࣈى͜͠ ੜ੒"*Ͱࢦࣔσʔλʹม׵ 4UFQ 4UFQ 4UFQ ߏ଄Խ͞Εͨ ࢦࣔσʔλ Ի੠σʔλ 4UFQίϚϯυΛड৴ ࣮ߦ 1JDP$BS4FSWFS ʢ3BTQCFSSZ1J1*DP$BSʣ

9!LPNP@GS w 3BTQCFSSZ1J1JDPͰ͸ඪ४ͷ$1ZUIPO͸ಈ࡞͠ͳ͍ͷͰ .JDSP1ZUIPOΛ࢖͏ w .JDSPEPUΛ࢖ͬͯ8FC"1*αʔόΛ࡞੒ w 'MBTLʹண૝ΛಘͨϛχϚϧͳ8FCϑϨʔϜϫʔΫ w $1ZUIPOͱ.JDSP1ZUIPOΛαϙʔτ͍ͯ͠ΔͷͰɺ1JDP্Ͱ΋ಈ͔ͤΔ
w ࠓճͷ༻్ͷ৔߹ɺmicrodot.pyʢ໿,#ʣΛ1JDP্ʹஔ͚ͩ͘Ͱ ࢖͑Δ 1JDP্Ͱ8FC"1*αʔόΛಈ͔͢ 4UFQίϚϯυΛड৴ ࣮ߦ

9!LPNP@GS from microdot import Microdot from pico_car import Car, RealCar
car = Car() if settings.USE_MOCK else RealCar() app = Microdot() ... @app.route("/command", methods=["POST"]) async def handle_command(request): try: car.sound_freq(1000, 0.1) car.sound_freq(1200, 0.1) command_list = request.json car.run_commands(command_list) return {"status": "ok"} except Exception as e: handle_buggy_error() print_exception(e) return {"status": "error", "message": str(e)}, 500 ... app.run(port=5001) ᶃड৴ԻΛ໐Β͢ ᶄίϚϯυΛ࣮ߦ Τϥʔൃੜ࣌ͷॲཧʢΤϥʔԻΛ໐Β͢ɺϞʔλఀࢭͳͲʣ αʔόΛىಈ ίʔυ1JDP্ͷ.JDSPEPUαʔό ίϚϯυΛड৴࣮ͯ͠ߦ͢Δ"1*ͷఆٛ 4UFQίϚϯυΛड৴ ࣮ߦ ࣗ࡞ͷ1JDPΧʔΫϥεͷΠϯελϯεΛੜ੒ .JDSPEPUΞϓϦΛ࡞੒

car = Car() if settings.USE_MOCK else RealCar() app = Microdot() ... @app.route("/command", methods=["POST"]) async def handle_command(request): try: car.sound_freq(1000, 0.1) car.sound_freq(1200, 0.1) command_list = request.json car.run_commands(command_list) return {"status": "ok"} except Exception as e: handle_buggy_error() print_exception(e) return {"status": "error", "message": str(e)}, 500 ... app.run(port=5001) ᶃड৴ԻΛ໐Β͢ ࣗ࡞ͷ1JDPΧʔΫϥεͷΠϯελϯεΛੜ੒ ᶄίϚϯυΛ࣮ߦ Τϥʔൃੜ࣌ͷॲཧʢΤϥʔԻΛ໐Β͢ɺϞʔλఀࢭͳͲʣ αʔόΛىಈ ίʔυ1JDP্ͷ.JDSPEPUαʔό ίϚϯυΛड৴࣮ͯ͠ߦ͢Δ"1*ͷఆٛ 4UFQίϚϯυΛड৴ ࣮ߦ ಈ࡞֬ೝ͠΍͍͢Α͏ ઃఆͰϞοΫͱ࣮ػΛ੾Γସ͑ .JDSPEPUΞϓϦΛ࡞੒

car = Car() if settings.USE_MOCK else RealCar() app = Microdot() ... @app.route("/command", methods=["POST"]) async def handle_command(request): try: car.sound_freq(1000, 0.1) car.sound_freq(1200, 0.1) command_list = request.json car.run_commands(command_list) return {"status": "ok"} except Exception as e: handle_buggy_error() print_exception(e) return {"status": "error", "message": str(e)}, 500 ... app.run(port=5001) ᶃड৴ԻΛ໐Β͢ ࣗ࡞ͷ1JDPΧʔΫϥεͷΠϯελϯεΛੜ੒ ᶄίϚϯυΛ࣮ߦ Τϥʔൃੜ࣌ͷॲཧʢΤϥʔԻΛ໐Β͢ɺϞʔλఀࢭͳͲʣ αʔόΛىಈ ίʔυ1JDP্ͷ.JDSPEPUαʔό ίϚϯυΛड৴࣮ͯ͠ߦ͢Δ"1*ͷఆٛ .JDSPEPUΞϓϦΛ࡞੒ 4UFQίϚϯυΛड৴ ࣮ߦ ಈ࡞֬ೝ͠΍͍͢Α͏ ઃఆͰϞοΫͱ࣮ػΛ੾Γସ͑

car = Car() if settings.USE_MOCK else RealCar() app = Microdot() ... @app.route("/command", methods=["POST"]) async def handle_command(request): try: car.sound_freq(1000, 0.1) car.sound_freq(1200, 0.1) command_list = request.json car.run_commands(command_list) return {"status": "ok"} except Exception as e: handle_buggy_error() print_exception(e) return {"status": "error", "message": str(e)}, 500 ... app.run(port=5001) ᶃड৴ԻΛ໐Β͢ ࣗ࡞ͷ1JDPΧʔΫϥεͷΠϯελϯεΛੜ੒ ᶄίϚϯυΛ࣮ߦ Τϥʔൃੜ࣌ͷॲཧʢΤϥʔԻΛ໐Β͢ɺϞʔλఀࢭͳͲʣ αʔόΛىಈ ίʔυ1JDP্ͷ.JDSPEPUαʔό ίϚϯυΛड৴࣮ͯ͠ߦ͢Δ"1*ͷఆٛ .JDSPEPUΞϓϦΛ࡞੒ 4UFQίϚϯυΛड৴ ࣮ߦ 'MBTLͬΆ͍ߏจ

car = Car() if settings.USE_MOCK else RealCar() app = Microdot() ... @app.route("/command", methods=["POST"]) async def handle_command(request): try: car.sound_freq(1000, 0.1) car.sound_freq(1200, 0.1) command_list = request.json car.run_commands(command_list) return {"status": "ok"} except Exception as e: handle_buggy_error() print_exception(e) return {"status": "error", "message": str(e)}, 500 ... app.run(port=5001) ᶃड৴ԻΛ໐Β͢ ࣗ࡞ͷ1JDPΧʔΫϥεͷΠϯελϯεΛੜ੒ ᶄίϚϯυΛ࣮ߦ Τϥʔൃੜ࣌ͷॲཧʢΤϥʔԻΛ໐Β͢ɺϞʔλఀࢭͳͲʣ αʔόΛىಈ ίʔυ1JDP্ͷ.JDSPEPUαʔό ίϚϯυΛड৴࣮ͯ͠ߦ͢Δ"1*ͷఆٛ .JDSPEPUΞϓϦΛ࡞੒ 4UFQίϚϯυΛड৴ ࣮ߦ

9!LPNP@GS w ୯७ʹforϧʔϓͰίϚϯυΛॱʹ࣮ߦ w ίϚϯυʹԠͯ͡Ϟʔλ΍-&%Λ੍ޚ͢Δ w Ϟʔλʔ΍-&%ͷ੍ޚʹ͸ɺ1JDPΧʔͷΩοτ༻ͷϥΠϒϥϦΛ࢖༻ w IUUQTHJUIVCDPN,JUSPOJL-UE,JUSPOJL1JDP"VUPOPNPVT3PCPUJDT1MBUGPSN.JDSP1ZUIPO ίϚϯυΛॱ൪ʹॲཧ
4UFQίϚϯυΛड৴ ࣮ߦ from PicoAutonomousRobotics import KitronikPicoRobotBuggy buggy = KitronikPicoRobotBuggy() ... buggy.motorOn("l", "f", SPEED) # ࠨंྠ: લਐ buggy.motorOn("r", "f", SPEED) # ӈंྠ: લਐ buggy.motorOn("l", "f", SPEED) # ࠨंྠ: લਐ buggy.motorOn("r", "r", SPEED) # ӈंྠ: ޙਐ લਐ ӈճస Ϟʔλͷ੍ޚྫ

9!LPNP@GS 4UFQඵؒಈ͔ͨ͠ͱ͖ͷ ڑ཭֯౓Λ࣮ଌ 4UFQݱࡏͷઃఆΛऔಘ 4UFQͷ࣮ଌ஋Λݩʹઃఆ w ૸ΒͤΔ৔ॴͷঢ়ଶʢϑϩʔϦϯάΧʔϖοτͳͲʣʹΑͬͯɺ ಉ͡ίϚϯυͰ΋࣮ࡍʹਐΉڑ཭΍֯౓͕มΘͬͯ͠·͏ w 👉(SBEJPͰઃఆը໘Λ࡞ΓɺʮϞʔλͷ଎͞ʯͳͲΛௐ੔Մೳʹ
Ϟʔλͷ଎౓ͷௐ੔ 4UFQίϚϯυΛड৴ ࣮ߦ

9!LPNP@GS %&.0ʮ͋ͳͨՄѪ͍ΘͶʯ΁ͷ൓Ԡ ݁Ռ IUUQTXXXZPVUVCFDPNXBUDI WHM&C&;TTX3"UT

9!LPNP@GS w ͭͭͷٕज़͸ͱͬͯ΋؆୯ʂʂ w ֤ϥΠϒϥϦͷνϡʔτϦΞϧϨϕϧͷٕज़ͰͰ͖͍ͯΔ w ͦΕͰ΋૊Έ߹ΘͤΔͱ໘ന͍΋ͷ͕࡞ΕΔ ϙΠϯτ ݁Ռ

9!LPNP@GS w ੜ੒"*Y1JDP͸ɺ·ͩ·ͩԠ༻͢Δͱ໘ന͍΋ͷ͕Ͱ͖ͦ͏ w 1JDPΧʔͷΞΫγϣϯΛ૿΍͢ w ྫ-&%ͷޫΓํͷύλʔϯΛ૿΍͢ɺαʔϘϞʔλͰ৲ඌΛ༳Β͢ w ϓϩϯϓτͰ1JDPΧʔͷੑ֨ΛΧελϚΠζ w
ྫʮ͹͔͹͔ʂʯͱݴΘΕͨ࣌ʹ൵͠ΉλΠϓ͔ɺౖΔλΠϓ͔ ʢ-&%͕੨PS੺ʣ ΞΠσΞ ݁Ռ

Ԡ༻ը૾ʹΑΔࢦࣔ

9!LPNP@GS w (SBEJPͰ࡞ͬͨ6*͔Βɺखඳ͖Ͱϧʔτը૾Λೖྗ w 4ελʔτ஍఺ɺ(ΰʔϧ஍఺ w ϓϩϯϓτͱҰॹʹը૾Λੜ੒"*ʹ౉͢ Ωϟϯόε্ʹϧʔτΛඳ͍ͯ ೖྗ ʢgr.ImageEditorʣ
༩͑ΒΕͨը૾ʹϧʔτ͕ඳ͔Ε͍ͯ·͢ɻ S͕ελʔτ஍఺ͰɺG͕ΰʔϧ஍఺Ͱ͢ɻ ͜ͷϧʔτը૾ΛɺpicoΧʔ༻ͷJSONܗࣜͷ ಈ࡞ίϚϯυ഑ྻʹม׵͍ͯͩ͘͠͞ɻ …ུ… ϓϩϯϓτ ϧʔτը૾Λ༩͑ͯ1JDPΧʔΛಈ͔͢ Ԡ༻ը૾ʹΑΔࢦࣔ

9!LPNP@GS %&.0 Ԡ༻ը૾ʹΑΔࢦࣔ IUUQTXXXZPVUVCFDPNXBUDI WHM&C&;TTX3"UT

Ԡ༻NJDSPCJUͱͷ࿈ܞ

9!LPNP@GS w ࠓճ͸3BTQCFSSZ1J1JDPͷΧʔΩοτΛ࢖͕ͬͨɺ ΩοτΛࠩ͠ସ͑Ε͹ଞͷ΋ͷ΋ಈ͔ͤΔ w ಛʹNJDSPCJU͕࢖͑Δͱɺಈ͔ͤΔػମͷ෯͕޿͕Δ ػମΛม͑ͯΈΔ Ԡ༻ྫNJDSPCJUͱͷ࿈ܞ 3BTQCFSSZ1J1JDP NJDSPCJU

9!LPNP@GS w ϓϩάϥϛϯάɾϑΥϩGPSNJDSPCJU εΠονΤσϡέʔγϣϯ w IUUQTTXJUDIFEVDBUJPODPNQSPEVDUT %FNPNJDSPCJU༻ͷͭ଍ϩϘοτ Ԡ༻ྫNJDSPCJUͱͷ࿈ܞ IUUQTZPVUVCF1B:KFO"TU/&

9!LPNP@GS NJDSPCJUΛ࢖͏৔߹ͷ੍໿ Ԡ༻ྫNJDSPCJUͱͷ࿈ܞ Ի੠σʔλ ߏ଄Խσʔλ εϚϗͳͲͷ 8FCϒϥ΢β $POUSPMMFS4FSWFS ʢ.BDʣ $BS4FSWFS
w ͜͜·Ͱઆ໌ͨ͠3BTQCFSSZ1J1JDP༻ͷ࢓૊Έ͸ͦͷ··Ͱ͸࢖͑ͳ͍ w NJDSPCJU୯ମʹ͸8J'J௨৴ͷػೳ͕ͳ͍ w ͦͷͨΊɺNJDSPCJU্ʹ8FC"1*αʔόΛཱͯΔͷ͸ݱ࣮తͰ͸ͳ͍ NJDSPCJU

9!LPNP@GS NJDSPCJU༻ͷߏ੒ Ԡ༻ྫNJDSPCJUͱͷ࿈ܞ Ի੠σʔλ ߏ଄Խ σʔλ εϚϗͳͲͷ 8FCϒϥ΢β $POUSPMMFS4FSWFS w
.BDͱNJDSPCJU͸༗ઢͩͱγϦΞϧ௨৴Ͱ؆୯ʹσʔλަ׵Ͱ͖Δ w NJDSPCJUಉ࢜͸ແઢͰ؆୯ʹ௨৴Ͱ͖Δ w 👉ͦͷͨΊɺ.BDʹதܧ༻ͷNJDSPCJUΛ64#Ͱܨ͗ ͦͷதܧ༻NJDSPCJUͱϩϘοτ্ͷNJDSPCJUΛ௨৴͢Δ $BS4FSWFS .BD NJDSPCJUᶃ NJDSPCJUᶄ จࣈྻσʔλ จࣈྻσʔλ 64# .BD γϦΞϧ௨৴ ແઢ

·ͱΊ

9!LPNP@GS w ੜ੒"*Λར༻ͯ͠ɺԻ੠΍ը૾Ͱಈ͓͘΋ͪΌͷंͷ࡞ΓํΛ঺հ w --.Ͱɺࠓ·Ͱ͸೉͔ͬͨ͠ॲཧΛ؆୯ʹ࣮ݱͰ͖ΔΑ͏ʹͳͬͨ w (SBEJPͩͱɺԻ੠΍ը૾ೖྗΛ؆୯ʹѻ͑Δ w -BOH$IBJOͰɺ؆୯ʹੜ੒"*ΛΞϓϦʹ૊ΈࠐΊΔ w
ੜ੒"*Y1JDP͸ɺ·ͩ·ͩԠ༻͢Δͱ໘ന͍΋ͷ͕Ͱ͖ͦ͏ ·ͱΊ

9!LPNP@GS 5IBOLZPVGPSMJTUFOJOH ࢿྉͱϦϙδτϦΛެ։͍ͯ͠·͢ (JU)VC3FQPTJUPSZ 4MJEFT 4MJEFT IUUQTTQFBLFSEFDLDPNLPNPGSTUBQZMMNQJDPDBS (JU)VCSFQPTJUPSZ IUUQTHJUIVCDPNLPNPGSMMNQJDPDBS %FNP
IUUQTXXXZPVUVCFDPNXBUDI WHM&C&;TTX3" 9 5XJUUFS !LPNP@GS

生成AIとPythonで ラズパイカーを動かそう！ 〜GradioとLangChainで作るプ...

生成AIとPythonで ラズパイカーを動かそう！ 〜GradioとLangChainで作るプロトタイプ〜

More Decks by komo_fr

Featured

Transcript

生成AIとPythonで ラズパイカーを動かそう！〜GradioとLangChainで作るプ...

生成AIとPythonで ラズパイカーを動かそう！〜GradioとLangChainで作るプロトタイプ〜