for data science newcomers Oleg Nenashev January 23, 2023 

Agenda ● Quick self-intro ● jq introduction ● Small demo 

> whoami --community @oleg_nenashev oleg-nenashev 

> whoami --academia “I have a PhD in Design… Hardware Design” Oleg @ Open Source Design Meetup Automation and Industrial Control Automation and Computer Systems Components of computing and control systems Thesis - Reengineering of digital hardware, and embedding test modules and interfaces into devices described by multilevel models BSc - 2009 MSc - 2011 PhD - 2015 Applied Computer Science in Economics Secondary MSc - 2011 

> whoami --industry Since 2016, Neuchatel 2008 - 2016, Russia 

Getting started! Source: commitstrip.com/en/2018/08/24/http- headers-ftw/ 

● One of the most popular data formats ● Service REST APIs ● Metadata ● [Un]structured data 

What is jq? “jq is a lightweight and flexible command-line JSON processor” “jq is like sed for JSON data” stedolan.github.io/jq 

“jq is like sed for JSON data” Input JSON Resulting JSON ❖ Filtering ❖ Search ❖ Sorting 

Data Access with jq ● Index slices ● Field slices ● Optional fields support 

Search with jq ● Selector queries ● Search by value ● Functional processing 

Advanced jq features Syntax ● Math ● Comparisons ● Variable binding ● Iterators ● Micro-scripting ● Regular expressions Tool ● Stream processing ● Modules / Plugins ● IO operations 

FAQ: How about YAML? … and other formats 

YAML Engineering is HOT 

Working with YAML ● YAML is [almost] like JSON ● Many tools ● yq - a wrapper for YAML ● github.com/mikefarah/yq ● Other formats are supported 

Getting Started with jq 

jq - Where to get it? ● Linux - [Any?] package manager ○ sudo apt install jq ● Windows - Chocolatey ○ choco install jq ● Docker - official image ○ docker pull stedolan/jq ● Python - jq binding ○ pip install jq ● VSCode plugins 

VSCode Plugins Live editing Queries with preview Linting and previews 

Try out jq online! https://jqplay.org 

Demo time! Data Analysis with jq 

Let’s do Big Data! ETH Library library.ethz.ch/en/archiving-and-digitising/o pen-data/interfaces-and-downloads.html Source: https://www.servicemax.com/uk/fsd/2017/11/03/comic-brake-tho ught-big-data-difficult-manage-meet-ginormous-data/ 

Our data ● Let’s look up for Renku publications ● https://daas.library.ethz.ch/rib/v3/search?q=any,contains,%22Renku%22 Source: en.wikipedia.org/wiki/Japanese_poetry#/media/File:' Kokinshu'-_Anthology_of_Classic_Japanese_Poetry _LACMA_M.91.250.332.jpg 

Query examples cat eth-library-renku.json | jq … ● Selector - '.docs[1]' ● Selector fields - '.docs[].pnx.display.title' ● Entries about data: '.docs[].pnx.display | select(.title[0] | contains("data"))' 

Learn More ● jq documentation ○ https://stedolan.github.io/jq/ ● jq tutorials ○ Szymon Stepniak: youtube.com/playlist?list=PLKaiHc24qCTSOGkkEpeIMupEmnInqHbbV ○ https://www.baeldung.com/linux/jq-command-json ● yq documentation ○ https://mikefarah.gitbook.io/yq/ 

Contacts: E-mail: [email protected] GitHub: oleg-nenashev Twitter: @oleg_nenashev QUESTIONS? 

Credits ● All developers of JSON APIs ● Stephen Dolan, jq author ○ https://github.com/stedolan ● Mike Farah, yq author ○ https://github.com/mikefarah ● OpenMoji project ○ https://openmoji.org