CEO / CTO ZettaScale Technology Angelo Corsaro, PhD Advanced Zenoh Tutorial Episode 3 

News 

ROS2 Kilted Kaiju This release features Tier-1 Zenoh Support! 

Tier-1 Meaning A Tier 1 middleware supports all of the listed Tier 1 operating systems for that particular ROS release, and we verify that support with a full battery of tests that run daily. Moreover, each Tier 1 middleware supports a full SROS2 security suite that includes access control, authentication, and encryption tools. We also require that all Tier 1 middleware support all of the Tier 1 code quality requirements de fi ned in REP-2004 and be included in the core ROS packages as listed in REP-2005. To learn more about the Eclipse Zenoh RMW please visit the following resources. 

https://github.com/kydos/zsak Zenoh Swiss Army knife (ZSAK) We’ll extensively use ZSAK an experimental command line tool designed to learn and experiment with Zenoh 

Get It. Build It. Run It $ git clone [email protected]:kydos/zsak.git $ cd zsak $ cargo build —release $ ./target/release/zsak -h 

Storage Alignment Followup 

Assumptions Zenoh’s doesn’t try to detect partitions. Instead it focuses on detecting misalignment and repairing those Zenoh also assumes that all events on the system are properly ordered — by default using HLC (Hybrid logical Clock) Storage replicas have to have exactly the same key expression Router Router Router Router Router Client Client Peer Peer Peer Peer Peer Router Client Client Client Client Router Router Client Peer Peer Peer Peer Client 

Time Discretization Zenoh storage alignment algorithm divides time into (con fi gurable) fi xed size intervals starting from a common origin (November 10th 2017 :-) This way intervals can be identi fi ed with an integer w/o any ambiguities and in spite of acceptable (in the sense of HLS) clocks misalignment ... O t 0 1 2 i n n+1 n-1 

Era Classi fi cation Zenoh classi fi es intervals into di ff erent kind of epochs depending on how likely it is to receive “live” updates within those epochs. Speci fi cally: Hot Era. This is at very least the interval preceding the current, in which live updates can still arrive. Warm Era. This is a con fi gurable number of intervals, preceding the Hot Era Cold Era. This is the collection of intervals preceding the Warm Era down to the time origin. ... O t 0 1 2 i n now n-1 Hot Era Warm Era Cold Era. 

Digest To e ffi ciently detect misalignment and identify where it happened, a digest is computed and distributed 

Demo 

Workshop Registration Replica 1 Replica 2 Replica 3 Router 2 Router 3 Router 1 Main Router Suppose we have multiple point of service (POS) for registering attendees to today’s workshop Attendees will check-in when arriving and check-out before leaving Eventually any of the replicas should have the full list of participants in the room, in spite of network failures, partitioning, etc. 

Workshop Registration Replica 1 Replica 2 Replica 3 Router 2 Router 3 Router 1 Main Router To begin with, we’ll assume that the storages on each of the POS are disconnected from the rest of the the Zenoh infrastructure because of networking issues As attendees will be registered but each replica will have a view of just the registrant that used the given POS Then we’ll re-estalish connections and see what happens… 

RUST ABI Compatibility Rust does not guarantee a stable ABI for its native calling convention (extern “Rust") The layout of types, calling conventions, and name mangling may change between compiler versions. Rust does not guarantee that two independently compiled Rust libraries will be binary-compatible, even if they share the same code and version. 

Puzzled face as I had tested this in the morning… But had forgotten about one more compilation :-D What was the Problem? I had recompiled Zenoh in release right before the webinar, but the demo was still pointing to the plugin libraries for the previous compilation I was beaten by Rust ABI incompatibility Could I have avoided this? Actually yes… Let’s see how Advanced Zenoh Tutorial Ep.2 

Zenoh Required Plugins When con fi guring Zenoh plug-ins, you can specify whether they are required or not If required plugins cannot be charged, zenohd won’t simply start and produce an error message Otherwise, if plugins are not required, zenohd will log the fact that it was unable to load, because of unavailability or version/API incompatibility, but will start anyway "plugins": { "rest": { “__required__”: true, "http_port": 9441, "required": true }, "storage_manager": { “__required__”: true, "storages": { "replication-test": { "key_expr": "workshop/**", "volume": "memory", "complete":"true", "replication": { "interval": 3, "sub_intervals": 5, "hot": 6, "warm": 24, "propagation_delay": 50, } } } } } 

Golden Rule Always set as required the plugins that you really need for your application to run correctly. 

Plugin Con fi guration Plugins loading is disabled by default, for security reasons, and needs to be explicitly enabled Additionally, to enable storages, you need to load the storage manager plugins_loading: { enabled: true }, storage_manager: { __required__: true, volumes: { influxdb: { url: "https://myinfluxdb.example" }, private: { username: "user1", password: "pw1", }, }, storages: { demo: "$STORAGE", mem_store: { key_expr: "alpha/**", volume: "memory", complete: "true" }, roksdb_store: { key_expr: "beta/**", volume: { id: “influxdb", db: "example" }, complete: "true" } } } 

Time in Zenoh 

Time and Ordering Zenoh uses Hybrid Logical Clocks as a way to track causality as well as ensure that all events in the system are totally ordered HLC as several nice properties as it builds upon a “su ff i ciently” synchronized physical clock which is combined with a logical clock. Logical Physical Clocks and Consistent Snapshots in Globally Distributed Databases 

Controlling Drift The ULC library we’ve implemented allows to bound the maximum drift from the physical clock (see doc) This parameter can be con fi gured using the UHLC_MAX_DELTA_MS environment variable 

Who Timestamps In the default con fi guration, Zenoh Routers are responsible for time stamping — in other terms if data w/o timestamp is received one is added by the router That said time-stamping can be enabled by con fi guration timestamping: { enabled: { router: true, peer: true, client: false }, 

Setting the Timestamp In some cases you may want to control the timestamp for the data you are producing Zenoh provides API to do that for all operations that may be timestamped. For instance, for a put operation, you can set it as follows: z.put(&kexpr, value).timestamp(ts).await.unwrap(); 

Zenoh Delete 

Zenoh’s delete is a primitive that can be used to delete keys. Deleting Keys z.delete(key_expr).await.unwrap(); If storages are available on the Zenoh network, the associated key will be removed from the database. 

Liveliness Tokens 

The Problem Location transparency is a great feature But sometimes, in distributed systems you need to know, or ensure, that a certain group of applications are available In other terms, you would like to be able to observe the liveliness of a group of elements of your system Router Router Brokered Routed Router Router Router Client Client Mesh Peer Peer Peer Peer Peer Router Client Client Client Client Router Router Client Clique Peer Peer Peer Peer Client 

Group Management In distributed systems, (closed) group management, is the classical solution used to de fi ne a group and as a consequence track the membership (join/leave) as well as the alive members of the group Often group management was tied to the group communication abstractions, which in a way creates an unnecessary and often problematic coupling as not all communicating applications require a closed group management 

Liveliness Tokens A Liveliness Tokens is a Zenoh key-expression with an associated liveliness. The Liveliness Token is bound to a zenoh session, and will be alive as far as the session on which it was declared is alive The token liveliness will be lost if: • the token is undeclared / dropped • the associated session is closed • the application crashes • the application looses connectivity 

Declaring Liveliness Tokens Liveliness tokens are declared as follows: let token = z.liveliness().declare_token(key_expr).await.unwrap(); NB: You have to keep a reference to the token, because dropping it will have it loose its liveliness. Example: let token = z.liveliness().declare_token(“fleet/10/bot/1”).await.unwrap(); 

Subscribing to Liveliness Tokens let sub = z.liveliness().subscribe(key_expr).await.unwrap(); Example: let sub = z.liveliness().subscribe(“fleet/10/bot/*”).await.unwrap(); Liveliness tokens are declared as follows: 

Lorem ipsum dolor sit amet Querying Livelines Token let sub = z.liveliness().get(selector).await.unwrap(); Example: let sub = z.liveliness().get(“fleet/10/bot/*”).await.unwrap(); Liveliness tokens are declared as follows: 

` Noti fi cations with SampleKind::Put represent joins while SampleKind::Delete represent leaves while let Ok(sample) = sub.recv_async().await { match sample.kind() { SampleKind::Put => { println!( "\t{}: {}", sample.key_expr().as_str().bold(), "Joined".bold().green() ); } SampleKind::Delete => { println!( "\t{}: {}", sample.key_expr().as_str().bold(), "Left".bold().red() ); } } 

Connectivity Information 

Connectivity Information A Zenoh session gives the ability to enumerate the peers and routers with which it has active sessions let sinfo = z.info(); let peers = sinfo.peers_zid().await; for p in peers { println!("{:?}", p); } let routers = sinfo.routers_zid().await; for r in routers { println!("{:?}", r); } 

Observing Connectivity Connectivity events can be observed by registering a listener on @//session/** The events that can be observed are: ‣ Session connectivity ‣ Transport connectivity ‣ Link connectivity 

Matching Status 

Am I Matching Something? Sometimes it is handy to be able to know if a Publisher has matching Subscribers or a Querier has matching Queryables Zenoh provide API to retrieve as well as observe the matching status. The API are equivalent for both the Publisher and the Querier, thus we’ll focus on the publisher. 

Current Matching Status The current matching status can be retrieved directly on the publisher, as follows: let status = publisher.matching_status().await.unwrap(); if status.matching() { println!("Hurray, I have matching Subscribers!"); } 

Observing Matching Status The matching status changes can be observed by registering a listener: let status_listener = publisher.matching_listener().await.unwrap(); while let Ok(status) = status_listener.recv_async().await { if status.matching() { println!("Hurray, I have some Publisher matching me!!!"); } else { println!("Nobody is matching me, sigh...") } } 

QoS Remapping 

Why QoS Remapping? Depending on your deployment use case, you may have slightly di ff erent requirements w.r.t. reliability, congestion control, priority, etc. across di ff erent system boundaries For instance, in robotics applications, often you have di ff erent requirements in terms of QoS when communicating inside the robot and when moving out Wouldn’t it be great if one could remap QoS on the fl y? Well you can do that with Zenoh 

QoS Remapping Zenoh provides an extremely fl exible manner to remap QoS at runtime by con fi guration qos: { publication: [ { key_exprs: ["demo/**", "example/key"], config: { congestion_control: "drop", priority: "data_high", express: true, reliability: "best_effort", allowed_destination: "remote", } } ], network: [ { id: "lo0_en0_qos_overwrite", // Optional list of ZIDs on which qos will be overwritten // zids: ["38a4829bce9166ee"], // Optional list of interfaces, all interfaces by default. interfaces: [ "lo0", "en0", ], /// Optional list of link protocols. link_protocols: [ "tcp", "udp", "tls", "quic","serial"], messages: [ "put", // put publications "delete" // delete publications "query", // get queries "reply", // replies to queries ], flows: ["egress", "ingress"], key_exprs: ["test/demo"], overwrite: { priority: "real_time", congestion_control: "block", express: true } } ] } 

Surprise 

Introducing Zetta R2 

Key Highlights Zenoh is one of a kind protocol — it uni fi es data in motion, data at rest and computations It is the only protocol able to run from the microcontroller up to the datacenter It has not topological constraints It is extremely easy to use! 

Patience, persistence and perspiration make an unbeatable combination for success. Thank You