バイナリファイル解析 CLI ツール fq のご紹介/Web通信と配信サービスの裏側公開イベント

Slide 1

Slide 1 text

バイナリファイル解析 CLI ツール fq のご紹介株式会社ソラコムプリンシパルソフトウェアエンジニア小熊崇 (@ogu)

Slide 2

Slide 2 text

Slide 3

Slide 3 text

Slide 4

Slide 4 text

Slide 5

Slide 5 text

© SORACOM Supported formats - Media containers - MP4, MP3, Ogg, FLAC, Matroska, PNG, JPEG, Exif … - Media codecs - AAC, NALU, H.264, MP3 frames … - Network protocols - PCAP, Ethernet, IPv4, IPv6, TCP, UDP, … - TCP reassembly and IPv4 defragmentation - Serialization formats - MsgPack, ASN1 BER, CBOR, ProtoBuf, bplist, … - and more!

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

© SORACOM Supported formats $ fq -h formats aac_frame Advanced Audio Coding frame adts Audio Data Transport Stream adts_frame Audio Data Transport Stream frame amf0 Action Message Format 0 apev2 APEv2 metadata tag ar Unix archive asn1_ber ASN1 BER (basic encoding rules, also CER and DER) av1_ccr AV1 Codec Configuration Record av1_frame AV1 frame av1_obu AV1 Open Bitstream Unit avc_annexb H.264/AVC Annex B avc_au H.264/AVC Access Unit avc_dcr H.264/AVC Decoder Configuration Record avc_nalu H.264/AVC Network Access Layer Unit avc_pps H.264/AVC Picture Parameter Set avc_sei H.264/AVC Supplemental Enhancement Information avc_sps H.264/AVC Sequence Parameter Set avro_ocf Avro object container file bencode BitTorrent bencoding bitcoin_blkdat Bitcoin blk.dat bitcoin_block Bitcoin block bitcoin_script Bitcoin script bitcoin_transaction Bitcoin transaction bsd_loopback_frame BSD loopback frame bson Binary JSON bzip2 bzip2 compression

Slide 9

Slide 9 text

© SORACOM Top level elements $ fq . example.mp4 |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f|0123456789abcdef|.{}: example.mp4 (mp4) 0x000000|00 00 00 20 66 74 79 70 69 73 6f 6d 00 00 02 00|... ftypisom....| boxes[0:4]: * |until 0xaf8b1c.7 (end) (11504413) | | 0x000030|de 02 00 4c 61 76 63 35 38 2e 35 34 2e 31 30 30|...Lavc58.54.100| tracks[0:2]: * |until 0xaf8b1c.7 (end) (11504365) | |

Slide 10

Slide 10 text

© SORACOM Dig into children $ fq .boxes[] example.mp4 |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f|0123456789abcdef|.boxes[0]{}: box 0x00|00 00 00 20 |... | size: 32 0x00| 66 74 79 70 | ftyp | type: "ftyp" (File type and 0x00| 69 73 6f 6d | isom | major_brand: "isom" 0x00| 00 00 02 00| ....| minor_version: 512 0x10|69 73 6f 6d 69 73 6f 32 61 76 63 31 6d 70 34 31|isomiso2avc1mp41| brands[0:4]: |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f|0123456789abcdef|.boxes[1]{}: box 0x20|00 00 00 08 |.... | size: 8 0x20| 66 72 65 65 | free | type: "free" (Free space) | | | data: raw bits |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f|0123456789abcdef|.boxes[2]{}: box 0x000020| 00 af 41 44 | ..AD | size: 11485508 0x000020| 6d 64 61 74| mdat| type: "mdat" (Media data 0x000030|de 02 00 4c 61 76 63 35 38 2e 35 34 2e 31 30 30|...Lavc58.54.100| data: raw bits * |until 0xaf416b.7 (11485500) | | |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f|0123456789abcdef|.boxes[3]{}: box 0xaf4160| 00 00 49 b1| ..I.| size: 18865 0xaf4170|6d 6f 6f 76 |moov | type: "moov" (Container 0xaf4170| 00 00 00 6c 6d 76 68 64 00 00 00 00| ...lmvhd....| boxes[0:4]: 0xaf4180|00 00 00 00 00 00 00 00 00 00 03 e8 00 00 ea d6|................| * |until 0xaf8b1c.7 (end) (18857) | |

Slide 11

Slide 11 text

© SORACOM Convert to JSON values $ fq tovalue example.mp4 { "boxes": [ { "brands": [ "isom", "iso2", "avc1", "mp41" ], "major_brand": "isom", "minor_version": 512, "size": 32, "type": "ftyp" }, { "data": "<0>", "size": 8, "type": "free" }, { "data": "<11485500>3gIATGF2YzU4LjU0LjEwMAAB6GhZKpB2RYXYAHFbnRmdY7u+QB+4mc2yKb1ftL+JGFMaI7AZ2A0wLMG4PyF85+/E

Slide 12

Slide 12 text

Slide 13

Slide 13 text

Slide 14

Slide 14 text

© SORACOM REPL $ fq -i . example.mp4 mp4> . |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f|0123456789abcdef|.{}: example.mp4 (mp4) 0x000000|00 00 00 20 66 74 79 70 69 73 6f 6d 00 00 02 00|... ftypisom....| boxes[0:4]: * |until 0xaf8b1c.7 (end) (11504413) | | 0x000030|de 02 00 4c 61 76 63 35 38 2e 35 34 2e 31 30 30|...Lavc58.54.100| tracks[0:2]: * |until 0xaf8b1c.7 (end) (11504365) | | mp4> .tracks[] |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f|0123456789abcdef|.tracks[0]{}: track 0x000210| 00 00 db b7 65 b8 00 04 00 00 04 1e 4c 50| ....e.......LP| samples[0:1200]: 0x000220|00 37 27 27 27 27 27 11 c7 d2 72 72 72 62 80 61|.7'''''...rrrb.a| * |until 0xaf416b.7 (11485018) | | | | | id: 1 | | | data_foramt: "avc1" |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f|0123456789abcdef|.tracks[1]{}: track 0x000030|de 02 00 4c 61 76 63 35 38 2e 35 34 2e 31 30 30|...Lavc58.54.100| samples[0:470]: * |until 0xaf3521.7 (11482354) | | | | | id: 2 | | | data_foramt: "mp4a" mp4>

Slide 15

Slide 15 text

© SORACOM REPL $ fq -i . example.mp4 mp4> .tracks[0].samples[0] |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f|0123456789abcdef|.tracks[0].samples[0][0:2]: sample (avc_au) 0x00210| 00 00 db b7 65 b8 00 04 00 00 04 1e 4c 50| ....e.......LP| [0]{}: nalu 0x00220|00 37 27 27 27 27 27 11 c7 d2 72 72 72 62 80 61|.7'''''...rrrb.a| * |until 0xddcc.7 (56251) | | 0x0ddc0| 00 00 ba| ...| [1]{}: nalu 0x0ddd0|2a 65 00 08 70 b8 00 04 00 00 04 1e 23 14 00 09|*e..p.......#...| * |until 0x197fa.7 (47662) | | mp4>