errors 0.8% FTP transfers produce incorrect checksum • Transmission Liu et al, HPDC 2018 Cross-geography scienti c data transferring trends and behavior • Storage • Human errors • OCR fi 4
errors • Transmission • Storage Human data entry errors = 4% 14 laboratory studies of spreadsheet development, 967 participants working alone on a variety of tasks. https://panko.com/ssr/ • Human errors • OCR errors 6
errors • Transmission • Storage • Human errors • OCR errors Human data entry errors = 4% 14 laboratory studies of spreadsheet development, 967 participants working alone on a variety of tasks. https://panko.com/ssr/ OCR errors = 2.5% of the tokens Impact of OCR errors on the use of digital libraries Chiron et al. JCDL 2017 7
by custom validation functions. • One bad eld => the whole record is rejected and cleansed from the pipeline. Invalid data is often silently dropped fi fi 8
that we can use: • {"item": "apple", "price": 3.14} ✓ • {"item": "apple", "price": ***3.14} ✗ • A known empty value ∅ = {} ✓ • Valid partial values -- waypoints fi 10
or waypoints Assumptions: • We have a validator that we can use: • {"item": "apple", "price": 3.14} ✓ • {"item": "apple", "price": ***3.14} ✗ • A valid empty value ∅ = {} ✓ e.g. address Jack, Luo, [email protected] Rahul, Gopinath, [email protected] e.g. date 12/01/2025 10/03/2026 • Valid partial values -- waypoints • Can only work with deletions 12
error feedback. • e.g. {"item": "apple","price": 3.14} Columnar data often come with custom validators (e.g. Excel) that do not provide error feedback. e.g. address Jack, Luo, [email protected] Rahul, Gopinath, [email protected] e.g. date 12/01/2025 10/03/2026 16
• Most records in a column are valid • Use them to infer an approximate grammar • Repair towards it. 2026-03-11 2024-11-24 2018/11/02 2022-02-15 2020-10-07 19
• Most records in a column are valid • Use them to infer an approximate grammar • Repair towards it. 2026-03-11 2024-11-24 2022-02-15 2020-10-07 2018/11/02 20
• Most records in a column are valid • Use them to infer an approximate grammar • Repair towards it. 2026-03-11 2024-11-24 2022-02-15 2020-10-07 Grammar Inference 2018/11/02 21
• Most records in a column are valid • Use them to infer an approximate grammar • Repair towards it. • Needs only an ✓, ̸ oracle and example data. • Repair is optimal with respect to the inferred grammar, so it stays close to existing records. ▷̸ 27
time ^\d{2}:\d{2}:\d{2}$ 8.0 isbn ^(?:\d[- ]?){9}[\dX]$ 13.0 ipv4 ^(\d{1,3}\.){3}\d{1,3}$ 13.6 ipv6 ^([0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}$ 26.4 url ^https?:\/\/(www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256} \.[a-zA-Z0-9()]{1,6}\b([-a-zA-Z0-9()@:%_+.~#?&//=]*)$ 32.8 • 100 valid records per subject: 50 for inference, 50 to corrupt. • 1, 2, 3 random point corruptions: insert, delete, substitute. • Baseline εRepair; ddmax cannot start on xed-width elds. • 300 s timeout per record. RQ1 How many records are repaired? RQ2 How good are the repairs? RQ3 How costly is repair? fi fi 28
recovered (higher is better) Corruption βMax εRepair Corruption βMax εRepair 1 2.12 2.68 1 97% 88% 2 3.49 3.30 2 94% 85% 3 3.94 5.10 3 94% 72% Average 3.18 3.69 Average 95% 82% Even approximating JSON with a regular grammar, βMax recovers more of the original data at every corruption level. 32
rahulgopinath
kobayashi31
corochann
analokmaus
institutdelelevage
recsyslab
tattaka
datascientistsociety
youten622
kinoue_smarthr
kento1109
kentaitakura
bermonpainter
cassininazir
jdejongh
.png){kind=avatar}
helenjbeal
uxyall
aleyda
reverentgeek
baasie
elsirapls
kevinliebholz
yeseniaperezcruz
kneath
