Upgrade to Pro — share decks privately, control downloads, hide ads and more …

An Integrated Distributed Simulation Environment weaving by Hakoniwa and mROS 2

takasehideki
October 20, 2023
400

An Integrated Distributed Simulation Environment weaving by Hakoniwa and mROS 2

ROSCon 2023
2023/10/20
https://roscon.ros.org/2023/

Vimeo Archive:
https://vimeo.com/879000700/6e149a4a3a

Speakers and Contributors:
Hideki Takase, Shintaro Hosoai (The Univ. of Tokyo), Tatsuya Fukuta (INTEC Inc.), Mitsutaka Takada (Nagoya Univ.), Shin Kuboaki (Change Vision, Inc.) and Takashi Mori (ESM, Inc. & Hakoniwa Lab.)

Abstract:
In the era of IoT/Cloud Robotics, the need for collaborative development among engineers from various specialized fields, such as embedded systems, control, and AI, has become increasingly crucial. However, it is not easy to integrate each engineer's diverse technical fields, development languages, and environments. In this talk, we show a successful implementation of an integrated distributed simulation environment using "Hakoniwa" and "mROS 2". This environment allows engineers to leverage their familiar development setups while working together, enabling seamless collaboration in robotics development.

Demonstration on p.28
https://twitter.com/takasehideki/status/1715464360439554305

takasehideki

October 20, 2023
Tweet

Transcript

  1. An Integrated Distributed
    Simulation Environment
    weaving by Hakoniwa and mROS 2
    Hideki Takase (The Univ. of Tokyo)
    Shintaro Hosoai (The Univ. of Tokyo)
    Tatsuya Fukuta (INTEC Inc.)
    Mitsutaka Takada (Nagoya Univ.)
    Shin Kuboaki (Change Vision, Inc.)
    Takashi Mori (ESM, Inc. & Hakoniwa Lab.)

    View full-size slide

  2. We are TOPPERS/Hakoniwa WG!!
    2
    • Hideki Takase
    • @takasehideki
    • The Univ. of Tokyo
    • EMB/IoT PF, ROS
    • Shintaro Hosoai
    • @s-hosoai
    • The Univ. of Tokyo
    • IDE, Modeling
    • Tatsuya Fukuta
    • @fudekunjp
    • INTEC Inc.
    • Cloud, Robotics
    • Mitsutaka Takada
    • @mitsu48
    • Nagoya Univ.
    • Kernel, Automotive
    • Shin Kuboaki
    • @kuboaki
    • Change Vision, Inc.
    • MBD/MDD, App
    • Takashi Mori
    • @kanetugu2020
    • ESM & Hakoniwa Lab
    • Tech Leader!!
    • About TOPPERS Project (Toyohashi OPen Platform for Embedded Real-time Systems)
    • NPO in Japan to promote embedded systems technology and its industry by developing and
    releasing high-quality open-source software, especially in real-time kernels and components
    • Hakoniwa WG: working group to establish simulation technology for the IoT fields

    View full-size slide

  3. Agenda
    1. What is Hakoniwa?
    2. Our Past Achievements, especially for ROS Robot
    3. Latest Update: hakoniwa-mros2sim
    4. Fusion of Virtual and Real weaving by Hakoniwa and mROS 2
    5. Wrapping Up
    3

    View full-size slide

  4. Agenda
    1. What is Hakoniwa?
    • Whatʼs the issue
    • Concept: HUB to Everything for Simulation!
    • Key Functions: Hakoniwa Core, Conductor and Protocol Data Unit
    2. Our Past Achievements, especially for ROS Robot
    3. Latest Update: hakoniwa-mros2sim
    4. Fusion of Virtual and Real weaving by Hakoniwa and mROS 2
    5. Wrapping Up
    4

    View full-size slide

  5. Issue and Background
    • IoT system spans a variety of technical domains,
    so its development requires the collaboration of
    experts from different fields.
    • Integration test and validation are so challenging
    due to these technologiesʼ diversity.
    • When a problem unfortunately happens,
    investigating the cause and its path
    becomes intricate.
    • Costs for conducting
    empirical experiments
    also increase significantly.
    5
    Legend
    Cloud
    CAN
    In-vehicle PC
    Smartphone
    Web
    Autonomous
    driving control
    ECU Control
    Backend
    Server
    Mechanical
    Electric
    Network
    IT Engineers
    Transportation services
    Embedded Systems Engineer
    Control System Engineer
    ICT Engineer
    Sensor

    View full-size slide

  6. Letʼs bring them into a BOX!!
    • Arranging various technologies according to everyone's preferences
    • Trying out various combinations as often we want
    • Observing them according to everyone's preferences into the same box
    • Bringing together things (e.g., software) from various tech fields to perform
    end-to-end integration, and verification on the desk!
    -> Developing IoT/robot systems in a virtual environment Hakoniwa
    6
    Network
    Engineers
    Cloud
    Engineers
    Web
    Engineers
    Embedded Systems
    Engineers
    Robot Control
    Engineers BTY, why ”Hakoniwa”?
    - Hako(箱) = box
    - Niwa(庭) = garden
    a miniature garden,
    one of the traditional JP
    cultures (like “Bonsai”)

    View full-size slide

  7. Concept
    7
    Hakoniwa
    Core
    (C/C++)
    Hakoniwa Daemon
    (for Remote Access)
    Hakoniwa APIs
    (for prog. Languages)
    Hakoniwa Commands
    (for Test Automation)
    Hakoniwa Library
    (for Physics Engines)
    Gazebo
    Ruby
    PyBullet
    • HUB to Everything for Simulation!
    • not just a simulator,
    but also a framework
    to construct simulators
    • Hakoniwa Assets:
    the components of
    the simulation target

    View full-size slide

  8. Hakoniwa Core
    • Functions for the seamless integration of various assets
    8
    Hakoniwa
    Core
    Visualization Tool
    Microcontroller
    Simulator
    Hakoniwa
    Asset
    Automated Testing
    Tool
    Physics
    Simulator
    Time
    Management
    Synchronization
    & Communication
    Asset
    Management
    Scheduling

    View full-size slide

  9. Hakoniwa Conductor
    • Mediate simulation between assets on Hakoniwa
    • gRPC based communication between the server and clients
    9
    マシン2
    マシン2
    Clients-side
    Server-side
    Hakoniwa Core functions
    Shared Memory
    Hakoniwa API
    Server for
    Hakoniwa Conductor
    Client for
    Hakoniwa
    Conductor
    箱庭アセット
    箱庭アセット
    Hakoniwa
    assets
    • registration of Hakoniwa assets
    • start/stop/reset of each simulation
    箱庭アセット
    箱庭アセット
    Hakoniwa
    assets
    • connect to the server
    as the Hakoniwa assets

    View full-size slide

  10. Protocol Data Unit (PDU)
    • Common data format to exchange in Hakoniwa
    • Function design is separated to asset-dependent and -independent
    • An example between Unity and ROS (topics)
    • Note: Unity and ROS are positioned as one of the "assets" in Hakoniwa
    10
    ≪ROS comm. process≫
    ≪C#≫ PDU-ROS
    data converter
    ≪Hakoniwa robot asset comm. I/F≫
    sensing data to be sent
    ≪C#≫
    Unity-ROS message
    ≪C#≫ Unity-ROS
    comm. API
    ≪comm. data for Hakoniwa robot asset≫
    PDU (Protocol Data Unit)
    ≪Hakoniwa robot asset≫
    robot model
    Unity
    ROS
    generation of
    Unity-ROS
    message
    Hakoniwa
    ≪ROS≫
    ROS msg type
    ≪C#≫
    Unity-ROS
    message
    ≪C#≫
    PDU-ROS
    converted data
    ≪C#≫
    Unity-ROS
    callback func.
    ≪JSON≫
    ROS msg type
    ≪JSON≫
    ROS topic info.
    generation of
    PDU-ROS
    converted data
    generation of
    Unity-ROS
    callback func.
    ROS topic info.
    format conversion
    of ROS msg
    ശ ఉ

    View full-size slide

  11. Distributed Simulation with Hakoniwa
    11
    Computer3
    Computer1 Computer2
    Server-side Hakoniwa Conductor
    Client-side Hakoniwa Conductor
    Hakoniwa Proxy for PDU
    Hakoniwa Asset
    Computer4

    View full-size slide

  12. Agenda
    1. What is Hakoniwa?
    2. Our Past Achievements, especially for ROS Robot
    • 4 Prototypes to Ensure the Hakoniwaʼs Concept
    • hakoniwa-ros2sim: ROS controlled Robot Simulation
    • Actual Use Cases
    3. Latest Update: hakoniwa-mros2sim
    4. Fusion of Virtual and Real weaving by Hakoniwa and mROS 2
    5. Wrapping Up
    12

    View full-size slide

  13. Hakoniwa Prototypes Previously Achieved
    13
    Microcomputer-controlled
    robot simulation
    ROS controlled
    Robot Simulation
    Collaborative simulation of
    multiple robots
    input
    control
    Virtual Learning Env Evaluation values
    RL script
    Q table
    control simulator



    Integration with
    reinforcement learning

    View full-size slide

  14. 14
    Unity
    terminal B terminal A
    docker
    OS
    ROS-TCP-Endpoint
    hakoniwa-ros2sim container
    ros2app.cpp
    ROS comm.
    tb3
    開始
    • Docker for the platform
    • pre-built container image
    for building & executing ros2app
    • able to use on multiple environments
    (Windows/WSL2 as the default)
    • Unity for physics and visualization
    • TurtleBot3 as the reference
    • ROS-TCP-Endpoint for
    assets communication
    ROS controlled Robot Simulation

    View full-size slide

  15. Actual Use Cases
    • Athrill (micro-controller simulator) for ET-robocon https://www.etrobo.jp/
    • Group exercises (PBL) in some Japanese universities
    • Automotive software education for embedded engineers
    https://github.com/ncesnagoya/autosar_os_training
    15

    View full-size slide

  16. Agenda
    1. What is Hakoniwa?
    2. Our Past Achievements, especially for ROS Robot
    3. Latest Update: hakoniwa-mros2sim
    • Introduction of mROS 2
    • [FYI] ROSCon JP 2023 Workshop
    • hakoniwa-mros2sim: Simulator for mROS 2 app by Hakoniwa Tech.
    4. Fusion of Virtual and Real weaving by Hakoniwa and mROS 2
    5. Wrapping Up
    16

    View full-size slide

  17. What is mROS 2?
    17
    autonomous communication
    in accordance with RTPS specification
    real-time kernels for EMB
    contributes perf. and mem. usage
    pub/sub messaging for Topic
    partially compliant with rclcpp
    Embedded devices
    HAL library
    Kernel (ASP3, Mbed, POSIX)
    mROS 2 application
    mROS 2 comm. lib.
    mROS 2 API
    UDP stack (lwIP)
    RTPS (embeddedRTPS)
    CMSIS wrapper lightweight and efficient process
    by C/C++ for <~200MHz / <~1MB
    mROS 2
    on EMB board
    better perf. & mem. usage
    partially compatible with rclcpp
    only for Topic comm.,
    and many unsupported features
    such as QoS, Service,,,
    https://vimeo.com/showcase/9954564/video/767140724

    View full-size slide

  18. Currently Supported
    18
    NEW!!
    watch
    later!!

    View full-size slide

  19. 19
    News!!
    https://roscon.jp/
    2023/#workshop
    contents is public!!
    (only in JP:D

    View full-size slide

  20. hakoniwa-mros2sim :D
    • Target: mros2 app & robot
    20
    Pi:Co Classic3 (MicroMouse)
    https://rt-net.jp/products/pico-classic3/
    https://github.com/mROS-
    base/rcjp2023_mros2

    View full-size slide

  21. Agenda
    1. What is Hakoniwa?
    2. Our Past Achievements, especially for ROS Robot
    3. Latest Update: hakoniwa-mros2sim
    4. Fusion of Virtual and Real weaving by Hakoniwa and mROS 2
    • Hakoniwaʼs Future Vision for Robotic Service
    • Expected System in the Future and its Architecture Design
    • Integration of mROS 2 into Hakoniwa Communication
    • Demonstration!!
    5. Wrapping Up
    21

    View full-size slide

  22. Our Future Vision for Robotic Service
    22
    Robot
    Cloud Service
    AI Analysis
    Space
    Virtual Service
    Gaming
    Space
    Generative AI Agent
    Space
    Robot
    Controller
    Micro
    Controller
    Robot Service
    Networks
    Robot
    Robot
    Robot
    Controller
    Robot
    Controller
    Micro
    Controller
    Micro
    Controller
    Coverage of ROS
    A new trend!!
    Challenges faced in
    the embedded
    systems domain
    Challenges faced in
    the cloud domain
    Coverage of ROS
    How to Integrate?
    How to Integrate?
    What to do?

    View full-size slide

  23. Fusion of Virtual and Real
    • Integration of Virtual and Real in Simulation
    23
    Virtual
    Real
    Robot
    Gaming
    Space
    Generative AI Agent
    Space
    AI Analysis
    Space

    View full-size slide

  24. Expected System in the Future
    24
    Real World
    Virtual World AI Agent
    Virtual Robot
    (Digital Twin)
    Real Robot
    Virtual Signal
    AR Agent
    AR Agent
    Virtual People
    (Digital Avatar)
    People

    View full-size slide

  25. Architecture Design: Overview
    25
    Virtual World
    Real World
    Real Robot
    mROS
    Sensor Actuator
    Wireless Network
    Cloud
    Network
    Cloud Server
    Central
    Server
    (MQTT)
    (ROS)
    Virtual Service
    Virtual Signal
    Virtual Robot Virtual People
    AI Agent
    Virtual/Real Map
    AR
    Device
    Infrastructure
    mROS

    View full-size slide

  26. Architecture Design: Detail
    26
    APIs(C#)
    Hakoniwa Asset
    Hakoniwa Core
    Shared Memory(PDU)
    APIs(C/C++)
    Hakoniwa Asset
    mros2-posix
    Field
    Mac (Apple Silicon)
    Virtual
    Signal
    Virtual
    Robot
    Virtual
    People
    AI
    Agent
    APIs(Python)
    Hakoniwa Asset
    Virtual/Real Map
    (Python)
    Embedded Device
    (Real Robot)
    mros2 ROS
    Embedded Device
    (Infrastructure)
    mros2
    ROS
    Hakoniwa Asset
    AI Agent
    (Python)
    APIs(Python)
    Hakoniwa Asset
    Virtual Agent
    Controllers
    (Python)
    Virtual Agent
    Controllers
    (Python)
    Virtual Agent
    Controllers
    (Python)
    Virtual Agent
    Controllers
    (Python)
    Hakoniwa Asset
    AR Service
    (Python)
    AR Device
    AR
    Application
    AR COM PROTOCOL

    View full-size slide

  27. but Why mROS 2?
    • Integration of mROS 2 communication into Hakoniwa!
    • lightweight but direct communication on RTPS
    • mros2-posix can run on general-purpose OSes (even on macOS)
    -> support mros2 communication as one of Hakoniwa PDUs!!
    27
    Host Computer
    Embedded
    Device
    Unity
    Hakoniwa Core
    箱庭
    依存部
    ROS 2 Proxy
    mros2-
    posix
    PDU shared memory
    Embedded Device
    Hakoniwa
    Conductor
    (Server)
    Mac/Ubuntu
    mros2
    ROS 2 app
    RTOS
    RTPS comm.

    View full-size slide

  28. Demonstration!!
    28
    Hakoniwa/Unity AR application actual robot
    ROS node Hakoniwa Core

    View full-size slide

  29. Agenda
    1. What is Hakoniwa?
    2. Our Past Achievements, especially for ROS Robot
    3. Latest Update: hakoniwa-mros2sim
    4. Fusion of Virtual and Real weaving by Hakoniwa and mROS 2
    5. Wrapping Up
    29

    View full-size slide

  30. Wrapping Up!
    • Introduction of Hakoniwa (箱庭)
    • HUB to Everything for Simulation!
    not just a simulator, but also a framework to construct simulators
    • hakoniwa-ros2sim: our past achievement for ROS robot
    • hakoniwa-mros2sim: new simulator for mros2 robot
    • Fusion of Virtual and Real weaving by Hakoniwa and mROS 2
    • Integration of mROS 2 into Hakoniwa Communication
    • Expected System in the Future and its Architecture Design
    • Whatʼs Next??
    • Open up the future by increasing the adoption in actual development scenes
    • and, enrich documentation, especially in English ;(
    30

    View full-size slide