ポスターセッション: 「まっすぐ行って、右！」って言ってラズパイカーを動かしたい〜生成AI × Raspberry Pi Pico × Gradioの試作メモ〜

Slide 1

Slide 1 text

ੜ੒"* ʮ·͙ͬ͢ߦͬͯӈʂʯͬͯݴͬͯ ϥζύΠΧʔΛಈ͔͍ͨ͠ 1Z$PO+1ϙελʔηογϣϯ º (SBEJP 3BTQCFSSZ1J 1JDP º ͷࢼ࡞ϝϞ 5PNPLP'VSVLJʢ!LPNP@GSʣ

Slide 2

Slide 2 text

ൃදऀ͕͍Δ࣌ؒଳ %BZ %BZ ൃදऀ 5PNPLP'VSVLJʢ!LPNP@GSʣ ҎԼͷ࣌ؒଳͰ͸௚઀આ໌͕Ͱ͖Δ༧ఆͰ͢ʂ ͥͻཱ͓ͪدΓ͍ͩ͘͞ େֶͰ͸஌ೳ৘ใϝσΟΞΛઐ߈͠ɺଔۀޙ͸γεςϜ։ൃ΍ɺ 1ZUIPOΛ࢖ͬͨσʔλ׆༻ͷ1P$։ൃɺٕज़ࢧԉͳͲʹैࣄɻ 1Z$PO+1ɺ1Z$PO,ZVTIVJO0LJOBXBɺ4DJ1Z+BQBOͳͲʹొஃɻ ࣥචɾ຋༁ ɾʮ1ZUIPOΠϯλϥΫςΟϒɾσʔλϏδϡΞϥΠθʔγϣϯೖ໳ʕ1MPUMZ%BTIʹΑΔσʔλՄࢹԽͱ8FCΞϓϦߏஙʕʯ ʢே૔ॻళɺڞஶʣ ɾʮੈքඪ४.*5ڭՊॻɹ1ZUIPOݴޠʹΑΔϓϩάϥϛϯάΠϯτϩμΫγϣϯୈ൛ʯʢۙ୅Պֶࣾɺڞ༁ʣ ݱࡏɺגࣜձࣾϏʔϓϥ΢υॴଐɻ ᶃ ᶄ ᶅ ᶃ ᶄ ड෇ʙ0QFOJOH։࢝લ னͷ௕ΊͷٳΈ࣌ؒ ϙελʔηογϣϯͷ࣌ؒ னͷ௕ΊͷٳΈ࣌ؒ ϙελʔηογϣϯͷ࣌ؒ ϝϯόʔ௨೥ืूதʂ εϙϯαʔϒʔε΋ ΍ͬͯΔΑ 9!LPNP@GS ݄೔ʢ౔ʣͷ 1Z$PONJOJ౦ւͰ΋ -BOH$IBJOͱ-BOH(SBQIʹؔ͢Δ τʔΫΛ͢Δ༧ఆͰ͢ɹͥͻ͖ͯͶ

Slide 5

Slide 5 text

(SBEJPͰ࡞ͬͨ 8FCΞϓϦ͔ΒԻ੠ೖྗ ॲཧ଴ͪதͷ޻෉ 4UFQ 6* ᶅ Ի੠ೝࣝ΍ੜ੒"*ॲཧͷ଴͕ͪ࣌ؒ௕͍ͨΊɺ ʮ͢΂ͯͷॲཧ͕׬͔ྃͯ͠Βදࣔ͢ΔʯͷͰ͸ͳ͘ɺ ʮ֤εςοϓͷॲཧ͕׬ྃ͢Δ͝ͱʹ్த݁Ռ͕දࣔ͞ΕΔʯΑ͏ʹͨ͠ (SBEJPͱ͸ ػցֶशͷσϞΞϓϦͳͲΛ ؆୯ʹ࡞ΕΔ8FCΞϓϦ ϑϨʔϜϫʔΫ બఆཧ༝ w Ի੠ೖྗ΍ը૾ೖྗΛ ؆୯ʹ࡞ΕΔ w ॲཧͷεςοϓ͝ͱʹ ঃʑʹ݁ՌΛฦ͢Α͏ͳ 6*Λ࡞Γ΍͍͢ Ի੠ೖྗ෦෼ ʢHS"VEJPΫϥεʣ ૹ৴ϦηοτϘλϯ ʢHS#VUUPOΫϥεʣ จࣈى͜͠ͷ݁Ռ ʢHS5FYU#PYΫϥεʣ ૹ৴݁ՌʢHS5FYU#PYΫϥεʣ ੜ੒͞ΕͨίϚϯυʢHS+40/Ϋϥεʣ ೖྗྖҬ ग़ྗྖҬ audio_input = gr.Audio(sources=["microphone"], type="filepath", format=“wav") …ུ… audio_input.stop_recording(fn=handler_audio_input, # ࿥Իఀࢭ࣌ʹݺ͹ΕΔؔ਺ inputs=audio_input, # σʔλͷೖྗݩͱͳΔ෦඼ # σʔλͷग़ྗઌͱͳΔ෦඼ outputs=[audio_transcript_output, audio_commands_output, audio_status_output, progress_lamps]) def handler_audio_input(audio_file): success, text = transcribe(audio_file) if success: yield text, None, "จࣈى͜͠׬ྃ", render_lamps(["s", "u", "u"]) commands = text_to_command(text) commands = convert_commands(commands) yield text, commands, "ίϚϯυੜ੒׬ྃ", render_lamps(["s", "s", "u"]) response = send_commands(commands) status = response.get("status") yield text, commands, status, render_lamps(["s"] * 3) else: yield text, None, "จࣈى͜͠ʹࣦഊ͠·ͨ͠ɻऴྃ͠·͢", render_lamps(["e", "u", "u"]) ᶃ(SBEJPͷԻ੠ೖྗͷ෦඼Λ࡞੒ ᶄ࿥Իఀࢭ࣌ʹݺ͹ΕΔؔ਺΍ σʔλͷग़ྗઌΛઃఆ ᶅจࣈى͜͠ͷॲཧΛߦ͏ ᶆจࣈى͜͠ͷ݁ՌΛZJFMEͰฦ͢ ᶈੜ੒"*ͷ݁ՌΛZJFMEͰฦ͢ ᶊQJDPΧʔͷಈ࡞݁ՌΛZJFMEͰฦ͢ ᶇੜ੒"*ͰίϚϯυʹม׵ ᶉQJDPΧʔʹίϚϯυΛૹ৴ ॲཧͷਐḿϥϯϓ ʢHS)5.-Ϋϥεʣ ࿥Իఀࢭϋϯυϥ Ի੠ೖྗͷઃఆ จࣈى͜͠ͷ݁Ռ ม׵ޙͷίϚϯυ ૹ৴݁Ռ ॲཧͷਐḿϥϯϓ QJDP@DPOUSPMMFSBQQQZ

Slide 6

Slide 6 text

-BOH$IBJO౳Ͱ ࣗવݴޠΛίϚϯυʹม׵ ςΩετˠίϚϯυʢࣙॻͷϦετʣ 4UFQ Ի੠ˠςΩετʢจࣈى͜͠ʣ ᶆ Ի੠ͷ··ੜ੒"*ʹೖྗ͢Δͱ"1*ྉۚͷίετ͕͔͔Γͦ͏ !ϩʔΧϧͰGBTUFSXIJTQFSΛ࢖ͬͯจࣈى͜͠Λ͢Δ from faster_whisper import WhisperModel model_size = "tiny" model = WhisperModel(model_size, device="cpu", compute_type="float32") …ུ… segments, _ = model.transcribe(tmp_audio_path, language="ja") text = "".join([segment.text for segment in segments]) -BOH$IBJOܦ༝Ͱ0QFO"*ͷ"1*ʢHQUʣΛ࢖ͬͯίϚϯυΛੜ੒ ᐆດͳݴ༿ΛݴΘΕͨ৔߹͸ɺʮײ৘ʯΛද͢ಈ͖΍৭ʹม׵͢Δ from langchain_openai import ChatOpenAI from langchain_core.prompts import ChatPromptTemplate …ུ… prompt = ChatPromptTemplate.from_messages( [ ("system", TEXT_TO_COMMAND_PROMPT), ("human", "ࢦࣔ: {text}"), ] ) llm = ChatOpenAI(model="gpt-4.1", temperature=0) chain = prompt | llm result = chain.invoke({"text": text}) instructions = result.content instructions = _add_led_rgb(instructions) # ੜ੒͞ΕͨίϚϯυ಺ͷLEDͷ৭ʢCSSͷ৭໊ʣΛݩʹRGB৘ใΛ௥Ճ instructions = json.loads(instructions) ༩͑ΒΕͨ೔ຊޠͷࢦࣔจΛɺpicoΧʔ༻ͷJSON ܗࣜͷಈ࡞ίϚϯυ഑ྻʹม׵͍ͯͩ͘͠͞ɻ ࢦࣔͰ͸ͳ͍ݴ༿ΛݴΘΕͨ৔߹ʢʮ͓͸Α͏ʂʯ ʮ͹͔ʂʯͳͲʣ͸ɺݴΘΕͨݴ༿ʹର͢Δײ৘Λ ද͢ಈ࡞ͷίϚϯυ഑ྻʹม׵͍ͯͩ͘͠͞ɻ ίϚϯυ഑ྻ͚ͩΛग़ྗ͠ɺͦΕҎ֎ͷઆ໌จͳͲ ͸ؚΊͳ͍Ͱ͍ͩ͘͞ɻ ֤ίϚϯυ͸ҎԼͷܗࣜΛ࢖༻͍ͯͩ͘͠͞: …ུ… ᶃϞσϧͷಡΈࠐΈ ᶄϞσϧͷ࣮ߦʢจࣈى͜͠ʣ QJDP@DPOUSPMMFSBVEJP@UP@UFYUQZ QJDP@DPOUSPMMFSUFYU@UP@DPNNBOEQZ [ { "action": {"type": "forward", "distance_cm": 20}, "led": {"color": "green", "rgb": (0, 255, 0)}}, { "action": {"type": "turn", "direction": "right", "angle": 90}, "led": {"color": "green", "rgb": (0, 255, 0)}}, …] ੜ੒͞ΕΔίϚϯυͷྫ -BOH$IBJOͱ͸ ೥ʹൃද͞Εͨɺ--.ΞϓϦέʔγϣϯ։ൃͷͨΊͷ1ZUIPOϑϨʔϜϫʔΫ ͭ໨ͷίϚϯυ BDUJPOલਐ͢ΔʢDNʣ MFE྘৭ ͭ໨ͷίϚϯυ BDUJPOӈճసʢ౓ʣ MFE྘৭ ᶃϓϩϯϓτͷߏங ᶄϞσϧͷݺͼग़͠ ᶅϓϩϯϓτͱϞσϧΛܨ͍Ͱ࣮ߦ ᶆग़ྗ݁Ռʹ৘ใ௥Ճ੔ܗͳͲ

Slide 1

Slide 1 text

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

Slide 9

Slide 9 text

Slide 10

Slide 10 text