Slide 1

Slide 1 text

Data Parsing And Visualization Kirby Wu @ g0v.tw

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

http://comic.sfacg.com/HTML/Naruto/ http://comic.sfacg.com/Utility/4/xxx.js comicCounterID = 4; give link parse  ID parse  list get js url get imgs

Slide 5

Slide 5 text

var comicName = “怠蔣敹!"; var nextVolume=“/HTML/Naruto/476/"; var preVolume=“/HTML/Naruto/474/"; var picCount = 17; var picAy = new Array(); var hosts = ["http://hotpic.sfacg.com","http://hotpic.sfacg. “http://ltpic.sfacg.com"]; picAy[0] = “/Pic/OnlineComic1/Naruto/475/001_1924.png"; picAy[1] = “/Pic/OnlineComic1/Naruto/475/002_1209.png"; picAy[2] = “/Pic/OnlineComic1/Naruto/475/003_17512.png"; picAy[3] = “/Pic/OnlineComic1/Naruto/475/004_13346.png"; picAy[4] = “/Pic/OnlineComic1/Naruto/475/005_6797.png"; picAy[5] = “/Pic/OnlineComic1/Naruto/475/006_16563.png"; picAy[6] = “/Pic/OnlineComic1/Naruto/475/007_4992.png"; picAy[7] = “/Pic/OnlineComic1/Naruto/475/008_5900.png"; picAy[8] = “/Pic/OnlineComic1/Naruto/475/009_30082.png"; picAy[9] = “/Pic/OnlineComic1/Naruto/475/010_18438.png"; picAy[10] = “/Pic/OnlineComic1/Naruto/475/011_19255.png"; picAy[11] = “/Pic/OnlineComic1/Naruto/475/012_17436.png"; picAy[12] = “/Pic/OnlineComic1/Naruto/475/013_14834.png"; picAy[13] = “/Pic/OnlineComic1/Naruto/475/014_16148.png"; . . . . . . (後略)

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

http://campaign-ﬁnance.g0v.ctiml.tw/

Slide 8

Slide 8 text

@g0v hackath8n

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Data Scraping crawling / 砍材

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

http://www.ptt.cc/bbs/Gossiping/index.html index4534.html

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

HTML • html — document with tags

Header

hot!

this is a book …

back

Slide 17

Slide 17 text

CSS Selector • html — document with tags • tag selector: p, div • class selector: .title, .desc • id selector: #content, #dialog • relations: h1 > a, br ~ p, p div • pseudo: a:ﬁrst-letter, h2:ﬁrst-line • attr: a[href=“#”]

Slide 18

Slide 18 text

.r-ent h1 .r-ent .title a .r-ent date .r-ent author document.querySelectorAll(“.r-ent h1”)

Slide 19

Slide 19 text

cheerio jQuery subset in server side cheerio = require(“cheerio”) $ = cheerio.load(YourHtmlSnippet); ! result = $(“.r-ent .hl”).text() npm install cheerio

Slide 20

Slide 20 text

request simpliﬁed HTTP Client request = require(“request”) ptt = “http://ptt.cc/bbs/Gossiping/index.html” ! request(ptt, function(e,r,b) { console.log(b); }); npm install request

Slide 21

Slide 21 text

request + cheerio request = require(“request”) cheerio = require(“cheerio”) ! ptt = “http://ptt.cc/bbs/Gossiping/index.html” ! request(ptt, function(e,r,b) { $ = cheerio.load(b); result = $(“.r-ent .hl”).text(); }); npm install cheerio request

Slide 22

Slide 22 text

request + cheerio result = {}, idx = 1; function download() { request(ptt(idx), function(e,r,b) { if(e) return; $ = cheerio.load(b); result[idx] = $(“.r-ent .hl”).text(); idx++; setTimeout(download, 0); }); }

Slide 23

Slide 23 text

http://zbryikt.github.io/visualize/ptt-user/

Slide 24

Slide 24 text

Slide 25

Slide 25 text

HTTP request response CLIENT SERVER GET,POST,… Referer Cookie Session Source IP Frequency Captcha State

Slide 26

Slide 26 text

https://cris.hpa.gov.tw/ 癌症線上登記系統 curl 'https://cris.hpa.gov.tw/pagepub/Home.aspx?itemNo=cr.q.40' -H 'Cookie: ASP.NET_SessionId=zdmkqgelubtzswj0majv3aiw; _ga=GA1.3.1312976917.1398148250' -H 'Origin: https://cris.hpa.gov.tw' -H 'Accept-Encoding: gzip,deflate,sdch' -H 'Accept-Language: zh-TW,zh;q=0.8,en-US;q=0.6,en;q=0.4,ja;q=0.2,zh-CN;q=0.2' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36' -H 'Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryuuaKlSwaG0BKt1db' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/ *;q=0.8' -H 'Cache-Control: max-age=0' -H 'Referer: https://cris.hpa.gov.tw/ pagepub/Home.aspx?itemNo=cr.q.40' -H 'Connection: keep-alive' --data-binary $'------WebKitFormBoundaryuuaKlSwaG0BKt1db\r\nContent-Disposition: form-data; name="__EVENTTARGET"\r\n\r\n\r\n------WebKitFormBoundaryuuaKlSwaG0BKt1db\r \nContent-Disposition: form-data; name="__EVENTARGUMENT"\r\n\r\n\r\n------ WebKitFormBoundaryuuaKlSwaG0BKt1db\r\nContent-Disposition: form-data; name="__VIEWSTATE_ID"\r\n\r\n871165\r\n------WebKitFormBoundaryuuaKlSwaG0BKt1db \r\nContent-Disposition: form-data; name="__VIEWSTATE"\r\n\r\n\r\n------ WebKitFormBoundaryuuaKlSwaG0BKt1db\r\nContent-Disposition: form-data; name="__EVENTVALIDATION"\r\n\r\nXOBXdAUfZ/ +c9MQLjFuAu4ua2oATUYv7Ky61orP3bnuQBqTBl/he3+CGfCqTF6lo5pzEGdQnpmzPocNVZ +EjiTjjj/0tLHu8PFynQ0i1IVvE3EEyJvHguowAYIThKE4kvfrG7A==\r\n------ WebKitFormBoundaryuuaKlSwaG0BKt1db\r\nContent-Disposition: form-data;

Slide 27

Slide 27 text

phantomjs a headless WebKit scriptable with a JavaScript API page = require(“webpage”).create() page.open(SomeURL, function() { comicID = page.evaluate(function() { return comicCounterID; }); page.render(“snapshot.png”); }); http://github.com/hcchien/doh-cancer/

Slide 28

Slide 28 text

Summary * CSS Selector ! * Cheerio + request ! * HTTP Header *wget / curl ! * PhantomJS ! * Tools like OpenCV

Slide 29

Slide 29 text

Slide 30

Slide 30 text

Search & Replace

Slide 31

Slide 31 text

Search & Replace [email protected] http://4am.tw 0800-000-123 texts with pattern picAy[1] = “ … “

Slide 32

Slide 32 text

123 —> 123 123 —> \d{3} abc —> \w{3} ! B98902123 —> B\d{8}

Slide 33

Slide 33 text

abc abbc abbbc … ab+c ac abc abbc abbbc … ab*c ac abc ab?c

Slide 34

Slide 34 text

aac aBc a:c a-c a1c a2c …. a.c aac abc acc adc aec …. a[a-z]c aac abc acc adc aec …. ^a[a-z]c$ a[^A-Z]c

Slide 35

Slide 35 text

aacabcacc (a[a-z]c)+

Slide 36

Slide 36 text

+ * ? . [ ] ^ $ () \w \W \d \D \s \S \\ \. \+ \? \* …

Slide 37

Slide 37 text

gmail.com tkirby [a-zA-Z0-9]+ [^.]+\.[^.]+ [email protected] [a-zA-Z0-9]+@[^.]+\.[^.]+

Slide 38

Slide 38 text

Regular Expression a sequence of characters that forms a search pattern, mainly for use in pattern matching with strings… 正規表達式

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

09xx-xxx-xxx \d+-\d+-\d+ \d{4}-\d{3}-\d{3} \d{4}-?\d{3}-?\d{3} b98902123 b\d{8} …

Slide 41

Slide 41 text

hl f3”>(\d+)< “date”>([^<]+)< “author”>([^<]+)< title“>([^<]+)<

Slide 42

Slide 42 text

re = { push: ‘hl f3”>(\d+)<‘ title: ‘title“>([^<]+)<‘ date: ‘“date”>([^<]+)<‘ author: ‘“author”>([^<]+)<‘ for(i=0;i

Slide 43

Slide 43 text

grep grep [-rvi] “regexp” [filename] globally search a regular expression and print command:

Slide 44

Slide 44 text

sed sed “s/pattern/string/“ stream editor for ﬁltering and transforming text command:

Slide 45

Slide 45 text

衛⽣生福利部台灣鄉鎮別罹癌統計

Slide 46

Slide 46 text

xlsjs XLS (Excel 97-2004) and XML (2003/2004) parser xls2csv [filename] command: npm install xlsjs

Slide 47

Slide 47 text

xls —> csv —> json doc —> txt —> json pdf —> txt —> json html —> txt —> json shp —> json xml —> json …… Always Search for Existing Tools

Slide 48

Slide 48 text

v/[a-zA-Z0-9]\+@[^.]\+\.[^.]\+/d Vim g/[a-zA-Z0-9]\+@[^.]\+\.[^.]\+/d highly conﬁgurable text editor to enable efﬁcient text editing

Slide 49

Slide 49 text

Vim 指令式⽂文字編輯器 h j k l — 左下上右指令模式 / 編輯模式 a o i — 附加新增⼀一⾏行插入退出編輯模式 c d r x — 改寫刪除覆蓋刪掉⼀一個字數字鍵 — 重複次數 y p — 複製貼上

Slide 50

Slide 50 text

100i2 — 插入 100 個 2 ddp — 與下⼀一⾏行交換 100dd100jp — 下⼀一百⾏行移到兩百⾏行之後 d100w — 刪除⼀一百個字 qad100wq — 將「d100w」命名為「a」 @a — 重複剛剛命名為「a」的動作

Slide 51

Slide 51 text

⾃自動化的⽂文字編輯 q100dd100jp100j 100@a 將0~100, 100~200 ⾏行交換, 200~300, 300~400 ⾏行交換, 400~500, 500~600 ⾏行交換, ……. ⼀一直重複到 10000 ⾏行為⽌止

Slide 52

Slide 52 text

⾃自動化的⽂文字編輯範例 g0v.tw 聊天室記錄 HTML 檔 ( http://logbot.g0v.tw/ ) 台北市親山步道列表 ( http://zbryikt.github.io/visualize/hiking/ )  ( http://www.tcge.taipei.gov.tw/MP_106051.html )

Slide 53

Slide 53 text

No content

Slide 54

Slide 54 text

td:nth-of-type(2n) td:nth-of-type(2n+1) tr td:nth-of-type(1) td:nth-of-type(2n+1) tr td:nth-of-type(3)

Slide 55

Slide 55 text

! 捷運關渡站 1.75K → 約34分雕塑公園 0.4K → 約7分打印台 0.6K → 約16分忠義山打印台 0.8K → 約25分⾏行天宮登山⼜⼝口 ! 捷運忠義站 1.1K → 約15分⾏行天宮登山⼜⼝口 0.8K → 約30分忠義山打印台 1.34K → 約43分中央北路四段30巷

Slide 56

Slide 56 text

Online Tools • Kimonolabs • ScraperWiki • Web Scrape Master • Web Scraper

Slide 57

Slide 57 text

Summary • Regular Expression • grep, sed, sort, … • xlsjs, xls2csv, … • vim, sublime Text, Emacs • 定義規則, 重複操作, 利⽤用前⼈人的成果

Slide 58

Slide 58 text

axe.g0v.tw cris.hpa.gov.tw 斧頭幫的挑戰狀

Slide 59

Slide 59 text

No content

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

Useful Junk? The Effects of Visual Embellishment on Comprehension and Memorability of Charts Scott Bateman, Saskatchewan university

Slide 64

Slide 64 text

No content

Slide 65

Slide 65 text

1 2 3 4 反⿊黑箱⽩白⾊色正義 (⼈人次)

Slide 66

Slide 66 text

1 2 3 4 反⿊黑箱⽩白⾊色正義 2 2 y (⼈人次)

Slide 67

Slide 67 text

50萬反⿊黑箱⽩白⾊色正義 10萬錯誤的⻑⾧長條圖使⽤用⽅方式! 基準點需要從零開始

Slide 68

Slide 68 text

50萬反⿊黑箱⽩白⾊色正義 0 正確的⻑⾧長條圖使⽤用⽅方式! ...好吧這是個失敗的例⼦子

Slide 69

Slide 69 text

1000 反⿊黑箱⽩白⾊色正義 0 換個觀點，陳述的事實也不同

Slide 70

Slide 70 text

1000 反⿊黑箱⽩白⾊色正義 0 萬⼈人響應 ———— 九⼈人到場少數⼈人  不代表主流⺠民意 y: 響應 / 到場

Slide 71

Slide 71 text

No content

Slide 72

Slide 72 text

brand / position / 3D

Slide 73

Slide 73 text

No content

Slide 74

Slide 74 text

No content

Slide 75

Slide 75 text

No content

Slide 76

Slide 76 text

No content

Slide 77

Slide 77 text

No content

Slide 78

Slide 78 text

No content

Slide 79

Slide 79 text

No content

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 2 8 7 6 5 2 2 2 8 2

Slide 82

Slide 82 text

7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 2 8 2 8 7 6 5 2 2 2 8 2

Slide 83

Slide 83 text

7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 5 2 8 7 6 2 8 2 8 6 5 2 2 2 8 2 7

Slide 84

Slide 84 text

No content

Slide 85

Slide 85 text

Visualization in Browser

Slide 86

Slide 86 text

Primitive Shapes in CSS

Slide 87

Slide 87 text

width: 10px  height: 10px border-bottom: 20px solid #000 border-left: 20px solid transparent border-right: 20px solid transparent border-radius: 50%

Slide 88

Slide 88 text

No content

Slide 89

Slide 89 text

http://zbryikt.github.io/visualize/kirby/

Slide 90

Slide 90 text

Scalable Vector Graphics SVG

Slide 91

Slide 91 text

Slide 92

Slide 92 text

! ! ! ! hello world!

Slide 93

Slide 93 text

Slide 94

Slide 94 text

Slide 95

Slide 95 text

Slide 96

Slide 96 text

M (x,y)+ - Move to (+ lineto) Z - Close Path L (x,y)+ - Line to C (x1,y1,x2,y2,x,y)- Curve to ctrl1 ctrl2

Slide 97

Slide 97 text

No content

Slide 98

Slide 98 text

Q (x1,y1,x,y)+ - Quadratic Bezier

Slide 99

Slide 99 text

A (rx,ry,a,b,c,x,y)+ - Quadratic Bezier

Slide 100

Slide 100 text

m (x,y)+ - Move to (+ lineto) z - Close Path l (x,y)+ - Line to c (x1,y1,x2,y2,x,y)+ - Curve to ctrl1 ctrl2 q (x1,y1,x,y)+ - Quadratic Bezier ! a (rx,ry,a,b,c,x,y)+ - Quadratic Bezier

Slide 101

Slide 101 text

M75,50.5l5-1.5c-0.1-2.2-0.4-4.3-0.9-6.3l-5.2-0.1c-0.2-0.6-0.4-1.1-0. 6-1.7l4-3.3c-0.9-1.9-2-3.8-3.2-5.5L69.2,34c-0.4-0.5-0.8-0.9-1.2-1.3l 2.4-4.6c-1.6-1.4-3.3-2.7-5.1-3.8l-3.7,3.6c-0.5-0.3-1.1-0.5-1.6-0.8l0.5 -5.2c-2-0.7-4-1.3-6.2-1.6l-2.1,4.8c-0.6-0.1-1.2-0.1-1.8-0.1l-1.5-5c-2. 2,0.1-4.3,0.4-6.3,0.9l-0.1,5.2c-0.6,0.2-1.1,0.4-1.7,0.6l-3.3-4c-1.9,0.9 -3.8,2-5.5,3.2l1.9,4.9c-0.5,0.4-0.9,0.8-1.3,1.2l-4.6-2.4c-1.4,1.6-2.7,3 .3-3.8,5.1l3.6,3.7c-0.3,0.5-0.5,1.1-0.8,1.6l-5.2-0.5c-0.7,2-1.3,4-1.6,6 .2l4.8,2.1c-0.1,0.6-0.1,1.2-0.1,1.8l-5,1.5c0.1,2.2,0.4,4.3,0.9,6.3l5.2, 0.1c0.2,0.6,0.4,1.1,0.6,1.7l-4,3.3c0.9,1.9,2,3.8,3.2,5.5l4.9-1.9c0.4,0. 5,0.8,0.9,1.2,1.3l-2.4,4.6c1.6,1.4,3.3,2.7,5.1,3.8l3.7-3.6c0.5,0.3,1.1, 0.5,1.6,0.8l-0.5,5.2c2,0.7,4,1.3,6.2,1.6l2.1-4.8c0.6,0.1,1.2,0.1,1.8,0. 1l1.5,5c2.2-0.1,4.3-0.4,6.3-0.9l0.1-5.2c0.6-0.2,1.1-0.4,1.7-0.6l3.3,4c 1.9-0.9,3.8-2,5.5-3.2L66,69.2c0.5-0.4,0.9-0.8,1.3-1.2l4.6,2.4c1.4-1.6 ,2.7-3.3,3.8-5.1l-3.6-3.7c0.3-0.5,0.5-1.1,0.8-1.6l5.2,0.5c0.7-2,1.3-4, 1.6-6.2l-4.8-2.1C74.9,51.7,75,51.1,75,50.5zM50,65c-8.3,0-15-6.7-1 5-15c0-8.3,6.7-15,15-15s15,6.7,15,15 C65,58.3,58.3,65,50,65z d=“ ”

Slide 102

Slide 102 text

http://goo.gl/fPUFgv SVG Editor

Slide 103

Slide 103 text

Slide 104

Slide 104 text

helloworld!

Slide 105

Slide 105 text

No content

Slide 106

Slide 106 text

Coordinate …… preserveAspectRatio=“None” or x{Min,Mid,Max}Y{Min,Mid,Max}

Slide 107

Slide 107 text

SMIL Animation in SVG

Slide 108

Slide 108 text

屬性變化 ! 線性轉換變化 ! 路徑

Slide 109

Slide 109 text

Slide 110

Slide 110 text

Slide 111

Slide 111 text

Slide 112

Slide 112 text

SVG Animation in Practice http://loading.io

Slide 113

Slide 113 text

Work with HTML

Hello World!

Slide 114

Slide 114 text

Tips and Pits * SVG 屬於 XML — 記得 *部分屬性可以寫成 CSS *功能族煩不及備載，包含 *圖⽚片 / Pattern / 漸層 *連結 / 群組

Slide 115

Slide 115 text

D3JS js library for manipulating documents based on data d3.json(“data.json”, function(data) { min = d3.min(data); });

Slide 116

Slide 116 text

Data Driven ! Document Model Powerful  Visualizing Lib d3js d3js d3js d3js d3js d3js d3js d3js d3js d3js

Slide 117

Slide 117 text

data data data data data selection enter exit selection

Slide 118

Slide 118 text

data data data data selection enter exit selection d3.selectAll(“div”) .data(data) .enter().append(“div”) .exit().remove() d3.selectAll(“div”)

Slide 119

Slide 119 text

new  data data new data.onChange enter exit selection

Slide 120

Slide 120 text

No content

Slide 121

Slide 121 text

Pack Layout data = { children: [ {value: 1}, {value: 2}, …… ]} pack = d3.layout.pack(); pack.node(data);

Slide 122

Slide 122 text

pack = d3.layout.pack(); pack.node(data); ! d3.select(“svg”) .selectAll(“circle”) .data(data) .enter() .append(“circle”) .attr({ “cx”: function(it) { return it.x;}, “cy”: function(it) { return it.y;}, “r”: function(it) { return it.r; } });

Slide 123

Slide 123 text

No content

Slide 124

Slide 124 text

No content

Slide 125

Slide 125 text

http://mbostock.github.io/d3/talk/20111018/collision.html zbryikt.github.io/visualize/dorling/

Slide 126

Slide 126 text

http://www.informationisbeautiful.net

Slide 127

Slide 127 text

information is beautiful easel.ly visual.ly piktochart.com stockphoto noun project dafont

Slide 128

Slide 128 text

Summary * Purpose of Visualization * Various Chart — Pits and Falls * CSS and SVG Visualization * D3JS basic * Examples * Resources * Infographics? should be 另⼀一⾨門課了! :p

Slide 129

Slide 129 text

links http://comic.sfacg.com/HTML/Naruto/ http://campaign-ﬁnance.g0v.ctiml.tw/ http://ptt.cc/bbs/Gossiping/ http://zbryikt.github.io/visualize/ptt-user/ https://cris.hpa.gov.tw/ http://g0v.github.io/nsc-projects/v2.html http://g0v.github.io/nsc-projects/index.html  http://zbryikt.github.io/visualize/kirby/ http://mbostock.github.io/d3/talk/20111018/collision.html http://zbryikt.github.io/visualize/dorling/ http://bl.ocks.org/zbryikt/raw/4696905/ http://bl.ocks.org/zbryikt/raw/4248542 http://zbryikt.github.io/visualize/banana/ http://zbryikt.github.io/visualize/mrt/ http://g0v.github.io/cancer/web/ http://zbryikt.github.io/visualize/crossﬁlter/ http://zbryikt.github.io/visualize/jobless/ http://zbryikt.github.io/visualize/nsc-2/