Share
Embed
ポスターセッション: 「まっすぐ行って、右!」って言ってラズパイカーを動かしたい 〜生成AI × Raspberry Pi Pico × Gradioの試作メモ〜
PyCon JP 2025のポスターセッションの資料です
発表の概要: https://2025.pycon.jp/ja/timetable/talk/9CD3UW
GitHubリポジトリ: https://github.com/komo-fr/llm-pico-car
デモ動画: https://www.youtube.com/watch?v=glEbEZsswRA
発表者X: @komo_fr (https://x.com/komo_fr)
komo_fr
September 25, 2025
Transcript
10 slides
-
ੜ"* ʮ·͙ͬ͢ߦͬͯӈʂʯͬͯݴͬͯ ϥζύΠΧʔΛಈ͔͍ͨ͠ 1Z$PO+1ϙελʔηογϣϯ º (SBEJP 3BTQCFSSZ1J 1JDP º ͷࢼ࡞ϝϞ
5PNPLP'VSVLJʢ!LPNP@GSʣ -
ൃදऀ͕͍Δ࣌ؒଳ %BZ %BZ ൃදऀ 5PNPLP'VSVLJʢ!LPNP@GSʣ ҎԼͷ࣌ؒଳͰઆ໌͕Ͱ͖Δ༧ఆͰ͢ʂ ͥͻཱ͓ͪدΓ͍ͩ͘͞ େֶͰೳใϝσΟΞΛઐ߈͠ɺଔۀޙγεςϜ։ൃɺ 1ZUIPOΛͬͨσʔλ׆༻ͷ1P$։ൃɺٕज़ࢧԉͳͲʹैࣄɻ 1Z$PO+1ɺ1Z$PO,ZVTIVJO0LJOBXBɺ4DJ1Z+BQBOͳͲʹొஃɻ
ࣥචɾ༁ ɾʮ1ZUIPOΠϯλϥΫςΟϒɾσʔλϏδϡΞϥΠθʔγϣϯೖʕ1MPUMZ%BTIʹΑΔσʔλՄࢹԽͱ8FCΞϓϦߏஙʕʯ ʢேॻళɺڞஶʣ ɾʮੈքඪ४.*5ڭՊॻɹ1ZUIPOݴޠʹΑΔϓϩάϥϛϯάΠϯτϩμΫγϣϯୈ൛ʯʢۙՊֶࣾɺڞ༁ʣ ݱࡏɺגࣜձࣾϏʔϓϥυॴଐɻ ᶃ ᶄ ᶅ ᶃ ᶄ डʙ0QFOJOH։࢝લ னͷΊͷٳΈ࣌ؒ ϙελʔηογϣϯͷ࣌ؒ னͷΊͷٳΈ࣌ؒ ϙελʔηογϣϯͷ࣌ؒ ϝϯόʔ௨ืूதʂ εϙϯαʔϒʔε ͬͯΔΑ 9!LPNP@GS ݄ʢʣͷ 1Z$PONJOJ౦ւͰ -BOH$IBJOͱ-BOH(SBQIʹؔ͢Δ τʔΫΛ͢Δ༧ఆͰ͢ɹͥͻ͖ͯͶ -
খܕΧʔΛࣗવݴޠͰ ಈ͔͍ͨ͠ ࣗવݴޠͰখܕΧʔʹࢦ͕ࣔग़ͤͨΒ ͪΐͬͱϖοτΆ͔ͯ͘Θ͍͍ͷͰʁ Πϝʔδਤ .PUJWBUJPO ΞΠσΞ ੜ"*ͰࣗવݴޠˠίϚϯυʹมͨ͠Β ͍͚Δؾ͕͢Δʂ ·͙ͬ͢ߦͬͯ
ӈʹۂ͕ͬͯʂ ·͙ͬ͢ߦͬͯӈʂ [{ "action": { "type": "forward", "distance_cm": 20,}, { "action": { "type": "turn", "direction": "right", "angle": 90,}, ...] གྷͬͯΈͯʂ ᶃ ࣗવݴޠ ίϚϯυʢࣙॻͷϦετʣ લʹਐΉ ӈʹճస -
શମͷߏ Πϝʔδਤ )PX εϚϗͳͲͷ 8FCϒϥβ 1JDP$BSαʔό ʢ3BTQCFSSZ1J1*DPΧʔʣ 8FCΞϓϦ͔Β ԻΛೖྗ ίϚϯυΛड৴
࣮ߦ ࠓճͬͨػମ ,JUSPOJL3BTQCFSSZ1J1JDP༻ ࣗϩϘοτϓϥοτϑΥʔϜʢόΪʔλΠϓʣ w ిࢠ࡞ͷܦݧ͕͋·Γͳ͍ͷͰɺࢢൢͷΩοτΛ͍͍ͨ w ϦνϜΠΦϯిཧ͕େมͦ͏ͳͷͰɺסిͰಈ͔͍ͨ͠ ! 3BTQCFSZ1J1*DP סిͰಈ͘ຊΩοτʹͨ͠ ᶄ ԻΛจࣈى͜͠ ੜ"*ͰίϚϯυʹม 1JDP$POUSPMMFSαʔό ʢ.BDʣ Ի ίϚϯυ ˞ੜ"*ॲཧ0QFO"*ͷ"1*Λར༻ ࠓճͷߏͷཧ༝ બఆཧ༝ w 1JDPΧʔ୯ମͩͱԻೝࣝੜ"*Λ͏ͷ͕͍͠ʢॲཧ͕ॏ͍ɺ.JDSP1ZUIPOͩͱҰ෦ϥΠϒϥϦΛ͑ͳ͍ʣ ! Իೝࣝੜ"*Λ͏ॲཧɺ.BD্Ͱಈ͘8FCαʔόʹͤͨ 1JDPΧʔͰϞʔλ-&%ͷ੍ޚ͚ͩΛߦ͏ w ʮ1$ʹ͔͚͠Δʯܗͩͱɺʮ1JDPΧʔʹ͔͚͍ͯ͠Δʯײ͕ബ͘ͳͬͯ͠·͏ ! 8FCϒϥβܦ༝ͷ6*ʹͯ͠ɺεϚϗ͔Βૢ࡞Ͱ͖ΔΑ͏ʹͨ͠ʢτϥϯγʔόʔతͳΠϝʔδͰʣ ंྠ͖Ϟʔλʢݸʣ-&%ʢंମͷ۱ʣϒβʔ ࠓճະ༻͕ͩɺڑηϯααʔϘϥΠϯτϨʔε༻ηϯα༗ -
(SBEJPͰ࡞ͬͨ 8FCΞϓϦ͔ΒԻೖྗ ॲཧͪதͷ 4UFQ 6* ᶅ Իೝࣝੜ"*ॲཧͷ͕͍ͪ࣌ؒͨΊɺ ʮͯ͢ͷॲཧ͕͔ྃͯ͠Βදࣔ͢ΔʯͷͰͳ͘ɺ ʮ֤εςοϓͷॲཧ͕ྃ͢Δ͝ͱʹ్த݁Ռ͕දࣔ͞ΕΔʯΑ͏ʹͨ͠ (SBEJPͱ
ػցֶशͷσϞΞϓϦͳͲΛ ؆୯ʹ࡞ΕΔ8FCΞϓϦ ϑϨʔϜϫʔΫ બఆཧ༝ w Իೖྗը૾ೖྗΛ ؆୯ʹ࡞ΕΔ w ॲཧͷεςοϓ͝ͱʹ ঃʑʹ݁ՌΛฦ͢Α͏ͳ 6*Λ࡞Γ͍͢ Իೖྗ෦ ʢHS"VEJPΫϥεʣ ૹ৴ϦηοτϘλϯ ʢHS#VUUPOΫϥεʣ จࣈى͜͠ͷ݁Ռ ʢHS5FYU#PYΫϥεʣ ૹ৴݁ՌʢHS5FYU#PYΫϥεʣ ੜ͞ΕͨίϚϯυʢHS+40/Ϋϥεʣ ೖྗྖҬ ग़ྗྖҬ audio_input = gr.Audio(sources=["microphone"], type="filepath", format=“wav") …ུ… audio_input.stop_recording(fn=handler_audio_input, # Իఀࢭ࣌ʹݺΕΔؔ inputs=audio_input, # σʔλͷೖྗݩͱͳΔ෦ # σʔλͷग़ྗઌͱͳΔ෦ outputs=[audio_transcript_output, audio_commands_output, audio_status_output, progress_lamps]) def handler_audio_input(audio_file): success, text = transcribe(audio_file) if success: yield text, None, "จࣈىྃ͜͠", render_lamps(["s", "u", "u"]) commands = text_to_command(text) commands = convert_commands(commands) yield text, commands, "ίϚϯυੜྃ", render_lamps(["s", "s", "u"]) response = send_commands(commands) status = response.get("status") yield text, commands, status, render_lamps(["s"] * 3) else: yield text, None, "จࣈى͜͠ʹࣦഊ͠·ͨ͠ɻऴྃ͠·͢", render_lamps(["e", "u", "u"]) ᶃ(SBEJPͷԻೖྗͷ෦Λ࡞ ᶄԻఀࢭ࣌ʹݺΕΔؔ σʔλͷग़ྗઌΛઃఆ ᶅจࣈى͜͠ͷॲཧΛߦ͏ ᶆจࣈى͜͠ͷ݁ՌΛZJFMEͰฦ͢ ᶈੜ"*ͷ݁ՌΛZJFMEͰฦ͢ ᶊQJDPΧʔͷಈ࡞݁ՌΛZJFMEͰฦ͢ ᶇੜ"*ͰίϚϯυʹม ᶉQJDPΧʔʹίϚϯυΛૹ৴ ॲཧͷਐḿϥϯϓ ʢHS)5.-Ϋϥεʣ Իఀࢭϋϯυϥ Իೖྗͷઃఆ จࣈى͜͠ͷ݁Ռ มޙͷίϚϯυ ૹ৴݁Ռ ॲཧͷਐḿϥϯϓ QJDP@DPOUSPMMFSBQQQZ -
-BOH$IBJOͰ ࣗવݴޠΛίϚϯυʹม ςΩετˠίϚϯυʢࣙॻͷϦετʣ 4UFQ ԻˠςΩετʢจࣈى͜͠ʣ ᶆ Իͷ··ੜ"*ʹೖྗ͢Δͱ"1*ྉۚͷίετ͕͔͔Γͦ͏ !ϩʔΧϧͰGBTUFSXIJTQFSΛͬͯจࣈى͜͠Λ͢Δ from faster_whisper
import WhisperModel model_size = "tiny" model = WhisperModel(model_size, device="cpu", compute_type="float32") …ུ… segments, _ = model.transcribe(tmp_audio_path, language="ja") text = "".join([segment.text for segment in segments]) -BOH$IBJOܦ༝Ͱ0QFO"*ͷ"1*ʢHQUʣΛͬͯίϚϯυΛੜ ᐆດͳݴ༿ΛݴΘΕͨ߹ɺʮײʯΛද͢ಈ͖৭ʹม͢Δ from langchain_openai import ChatOpenAI from langchain_core.prompts import ChatPromptTemplate …ུ… prompt = ChatPromptTemplate.from_messages( [ ("system", TEXT_TO_COMMAND_PROMPT), ("human", "ࢦࣔ: {text}"), ] ) llm = ChatOpenAI(model="gpt-4.1", temperature=0) chain = prompt | llm result = chain.invoke({"text": text}) instructions = result.content instructions = _add_led_rgb(instructions) # ੜ͞ΕͨίϚϯυͷLEDͷ৭ʢCSSͷ৭໊ʣΛݩʹRGBใΛՃ instructions = json.loads(instructions) ༩͑ΒΕͨຊޠͷࢦࣔจΛɺpicoΧʔ༻ͷJSON ܗࣜͷಈ࡞ίϚϯυྻʹม͍ͯͩ͘͠͞ɻ ࢦࣔͰͳ͍ݴ༿ΛݴΘΕͨ߹ʢʮ͓Α͏ʂʯ ʮ͔ʂʯͳͲʣɺݴΘΕͨݴ༿ʹର͢ΔײΛ ද͢ಈ࡞ͷίϚϯυྻʹม͍ͯͩ͘͠͞ɻ ίϚϯυྻ͚ͩΛग़ྗ͠ɺͦΕҎ֎ͷઆ໌จͳͲ ؚΊͳ͍Ͱ͍ͩ͘͞ɻ ֤ίϚϯυҎԼͷܗࣜΛ༻͍ͯͩ͘͠͞: …ུ… ᶃϞσϧͷಡΈࠐΈ ᶄϞσϧͷ࣮ߦʢจࣈى͜͠ʣ QJDP@DPOUSPMMFSBVEJP@UP@UFYUQZ QJDP@DPOUSPMMFSUFYU@UP@DPNNBOEQZ [ { "action": {"type": "forward", "distance_cm": 20}, "led": {"color": "green", "rgb": (0, 255, 0)}}, { "action": {"type": "turn", "direction": "right", "angle": 90}, "led": {"color": "green", "rgb": (0, 255, 0)}}, …] ੜ͞ΕΔίϚϯυͷྫ -BOH$IBJOͱ ʹൃද͞Εͨɺ--.ΞϓϦέʔγϣϯ։ൃͷͨΊͷ1ZUIPOϑϨʔϜϫʔΫ ͭͷίϚϯυ BDUJPOલਐ͢ΔʢDNʣ MFE৭ ͭͷίϚϯυ BDUJPOӈճసʢʣ MFE৭ ᶃϓϩϯϓτͷߏங ᶄϞσϧͷݺͼग़͠ ᶅϓϩϯϓτͱϞσϧΛܨ͍Ͱ࣮ߦ ᶆग़ྗ݁ՌʹใՃܗͳͲ -
1JDPΧʔ্ͷ αʔόʹίϚϯυΛૹ৴ ίϚϯυΛॱ൪ʹॲཧ 4UFQ ᶇ .JDSPEPUͱ w 'MBTLʹணΛಘͨϛχϚϧͳ8FCϑϨʔϜϫʔΫ w ඪ४ͷ$1ZUIPOʹՃ͑.JDSP1ZUIPOΛαϙʔτ͍ͯ͠ΔͷͰɺ1JDP্Ͱಈ͔ͤΔ
w ࠓճͷ༻్ͷ߹ɺNJDSPEPUQZʢ,#ʣΛ1JDP্ʹஔ͚ͩ͘Ͱ͑Δ from microdot import Microdot from pico_car import Car, RealCar car = Car() if settings.USE_MOCK else RealCar() app = Microdot() …ུ… @app.route("/command", methods=["POST"]) async def handle_command(request): try: car.sound_freq(1000, 0.1) car.sound_freq(1200, 0.1) command_list = request.json car.run_commands(command_list) return {"status": "ok"} except Exception as e: handle_buggy_error() print_exception(e) return {"status": "error", "message": str(e)}, 500 …ུ… app.run(port=5001) ࣗ࡞ͷ1JDPΧʔΫϥεͷ ΠϯελϯεΛੜ .JDSPEPUΞϓϦΛ࡞ ίϚϯυΛ࣮ߦ͢Δ"1*ͷఆٛ ᶃड৴ԻΛ໐Β͢ ᶄίϚϯυΛ࣮ߦ QJDP@DBSNBJOQZ Τϥʔൃੜ࣌ͷॲཧΛ࣮ߦʢΤϥʔԻΛ໐Β͢ɺϞʔλఀࢭʣ αʔόΛىಈ 'MBTLͬΆ͍ॻ͖ํ͕ Ͱ͖Δ ࣮ػͷ1*DPΧʔͰಈ͔͍ͯ͠Δ࣌ ݱࡏԿͷॲཧΛ͍ͬͯΔͷ͔ϩάͳͲͰ ֬ೝ͠ʹ͍͘ͷͰɺԻͰ௨ ୯७ʹGPSจͰίϚϯυΛॱʹ࣮ߦ͍ͯ͘͠ Ϟʔλʔ-&%ͷ੍ޚɺ࣮ػͷΩοτ༻ͷϥΠϒϥϦ Λ༻ IUUQTHJUIVCDPN,JUSPOJL-UE,JUSPOJL1JDP"VUPOPNPVT3PCPUJDT1MBUGPSN.JDSP1ZUIPO ͳͲͷઃఆը໘ ΒͤΔॴͷঢ়ଶ1JDPΧʔͷঢ়ଶʹ ΑΓɺಉ͡ڑʢDNͳͲʣΛࢦఆ͠ ࣮ͯࡍʹਐΉڑճΔ͕֯มΘͬ ͯ͠·͏ ! (SBEJPͰʮϞʔλͷ͞ʯʮඵؒ ͰਐΉڑɾճస͢Δ֯ʯΛௐͰ͖ Δઃఆը໘Λ࡞ 4UFQඵؒಈ͔ͨ͠ͱ͖ͷ ڑ֯Λ࣮ଌ 4UFQݱࡏͷઃఆΛऔಘ 4UFQͷ࣮ଌΛݩʹઃఆ ઃఆʹΑͬͯ ϞοΫ࣮ػΛΓସ͑ ʢಈ࡞֬ೝΛεϜʔζʹʣ 1JDP্Ͱ.JDSPEPUαʔόΛಈ͔͢ -
࣮ߦ݁Ռɾίʔυ ໘ന͍ͱ͜Ζ 0VUQVU ᶈ σϞಈը (JU)VC ςΩετˠίϚϯυมͰੜ"*Λ͍ͬͯΔͨΊ ᐆດͳݴ༿Ͱಈ࡞ʹม͞ΕΔ གྷͬͯΈͯʂ ͋ͳͨՄѪ͍ΘͶʙ
͔͔ʂ ࠓΦʔλχ͞Μ͕ ϗʔϜϥϯଧͬͨΜͩͬͯ Γ͠Α͏ ϐϯΫͰࠨӈʹৼΕͯ ԫ৭Ͱલޙ ੨৭Ͱޙୀ ʢ൵͠Έʁʣ Ͱલਐ ʢߠఆʁʣ ԫ৭Ͱલਐͯ͠ ࠨӈʹৼΕΔʢςϯγϣϯߴʁʣ ϐϯΫͰલਐͯ͠ ࠨӈʹ;Γ;Γ ʢتͼʁʣ IUUQTHJUIVCDPNLPNPGSMMNQJDPDBS IUUQTZPVUVCFHM&C&;TTX3" -
վળɾԠ༻ ॲཧͪ࣌ؒதͷϑΟʔυόοΫ ᶉ ը૾͔Βͷϧʔτೖྗ จࣈى͜͠ੜ"*ॲཧͷ͕͍ͪ࣌ؒ !εςοϓ͝ͱʹ1JDPΧʔʹݱࡏͷεςʔλεΛૹ৴͠ɺԻ-&%ͰԠ จࣈى͜͠։࢝ 1JDP$POUSPMMFSαʔό TUBUVTΛߋ৽ ੜ"*ʹΑΔ
ίϚϯυͷม։࢝ "audio_to_text_started" 1JDP$BSαʔό ϒβʔԻΊ -&%໓ TUBUVTΛߋ৽ “text_to_command_started" ϒβʔԻߴΊ -&%ॱ൪ʹ౮ 1045TUBUVT 1045TUBUVT Իೖྗ։࢝ TUBUVTΛߋ৽ "recording_started" 1045TUBUVT ϒβʔԻͳ͠ -&%ബ͘౮ ίϚϯυੜྃ 1045DPNNBOE ίϚϯυΛॱʹ࣮ߦ Ωϟϯόε্ʹ ϧʔτΛඳ͍ͯೖྗ ʢHS*NBHF&EJUPSʣ 6* (SBEJPͰը૾ೖྗΛ؆୯ʹ࣮ݱ ը૾ˠίϚϯυม -BOH$IBJOͰϓϩϯϓτͱҰॹʹ ը૾Λ0QFO"*ͷ"1*ʹૹ৴ ༩͑ΒΕͨը૾ʹϧʔτ͕ඳ͔Ε͍ͯ·͢ɻ S͕ελʔτͰɺG͕ΰʔϧͰ͢ɻ ͜ͷϧʔτը૾ΛɺpicoΧʔ༻ͷJSONܗࣜͷಈ࡞ ίϚϯυྻʹม͍ͯͩ͘͠͞ɻ …ུ… ੜ͞ΕͨίϚϯυ ʢHS+40/ʣ 1JDPΧʔ͕ಈ͘ʂ σϞಈը গ͠ෳࡶʹͳΔͱؒҧ͏ͷͰ վળͷ༨͋Γ -
·ͱΊɾײ ͜ͷϙελʔࢿྉ ͔͜͜ΒݟΕ·͢ εϙϯαʔϒʔεͰ ͷϒʔεͬͯΔΑ ༡ͼʹ͖ͯͶ (JU)VCσϞಈըͷ ϦϯΫ͋Γ·͢ w --.Ͱɺࠓ·Ͱͩͱ͔ͬͨ͠ॲཧΛ؆୯ʹ࣮ݱͰ͖ΔΑ͏ʹ
ͳͬͨ͜ͱΛ࣮ײͨ͠ ࣗવݴޠˠίϚϯυɺը૾ˠίϚϯυͷม w (SBEJPͩͱɺԻը૾ೖྗΛ؆୯ʹѻ͑ͯศར w ੜ"*º1JDPɺ·ͩ·ͩԠ༻͢Δͱָͦ͠͏ ྫ1JDPΧʔͷΞΫγϣϯΛ૿͢ɻͨͱ͑ɺ-&%ͷޫΓํͷύλʔϯΛ૿͢ɺ ԻͰײΛද͢ɺαʔϘΛ͚ͭͯ৲ඌͷΑ͏ʹ༳Β͢ͳͲ w ϓϩϯϓτͰ1JDPΧʔͷੑ֨ΛΧελϚΠζ͢Δͱ໘നͦ͏ ྫ͔ʂͱݴΘΕͨ࣌ʹ൵͠ΉλΠϓ͔ɺౖΔλΠϓ͔ʢ-&%͕੨PSʣ གྷͬͯΈͯʂͱݴΘΕͨ࣌ͷςϯγϣϯͷߴ͞ʢಈ͖ͷܹ͠͞ʣͷҧ͍ͳͲ w ੜ"*ͷར༻ํ๏ʹΑͬͯɺਫ਼ɾɾྉۚίετ͕Ͳ͏มΘΔ ͔Λɺ͏গ͠ൺֱɾݕ౼͍ͨ͠ ྫɿϞσϧͷҧ͍ɺϩʔΧϧ࣮ߦWT"1*ར༻ɺԻΛͦͷ··͔͢ࣄલʹจࣈى ͜͢͠Δ͔ͳͲ w ॲཧͪ࣌ؒΛͲ͏͢Δ͔·ͩ՝ʜ ྫॲཧ࣌ؒࣗମΛ͘͢Δ͔ɺͪ࣌ؒͷΛ͢Δ͔ w ϓϩάϥϜࣗମ͙͢࡞Ε͚ͨͲɺ1JDPͷѻ͍ʹ׳Εͯͳͯ͘खؒ औͬͨ ྫ1JDPͷࠩ͠ࠐΈ͕͍ͷ͔Ϟʔλʔ͕ಈ͔ͳ͍ɺंྠ͕͙͢֎ΕΔɺ8J'J͕ͳ͔ ͳ͔ܨ͕Βͳ͍ɺͷௐͳͲ ᶊ IUUQTTQFBLFSEFDLDPNLPNPGSQZDPOKQQPTUFS