Presentation.pdf

Transcript

1. IMPLEMENTING A NON-BLOCKING STREAMING CODEC FOR TOPOJSON Prisca Chidimma Maduka
2. ABOUT OUTREACHY INTERNSHIP PROGRAM
3. TOPOJSON • Encodes topology ( relationships between the different geographic features). • Stores geometries as arcs rather than as individual points, lines, and polygons. Shared boundary lines are stored only once. GEOJSON • Represents geographic features as a collection of simple data structures (e.g. points, lines, polygons) • Stores all coordinates for each feature, resulting in a larger file. GEOJSON vs TOPOJSON
4. Image credit: Umar Yusuf Geometry types Image credit: Umar Yusuf
5. Image credit: Michael Dorman
6. Geojson { "type": "FeatureCollection", "features": [ { "type": "Feature", "geometry": { "type": "MultiPoint", "coordinates": [[125.1, 40.0], [155.9, 22.5]] }, "properties": {"name": "Dinagat Islands"} }, { "type": "Feature", "geometry": { "type": "MultiLineString", "coordinates": [ [ [170.0, 45.0], [180.0, 45.0] ], [ [-180.0, 45.0], [-170.0, 45.0] ] ] }, "properties": {"name": "Martha", "country": "Jamaica"} } ] } Topojson { "type": "Topology", "bbox": [-180.0, 22.5, 180.0, 45.0], "objects": { "unnamed": { "type": "GeometryCollection", "geometries": [ { "type": "MultiPoint", "coordinates": [[125.1, 40.0], [155.9, 22.5]], "properties": { "name": "Dinagat Islands" } }, { "type": "MultiLineString", "arcs": [[1], [0]], "properties": { "name": "Martha", "country": "Jamaica"} } ] } }, "arcs": [ [ [-180.0, 45.0], [-170.0, 45.0] ], [ [170.0, 45.0], [180.0, 45.0] ] ] }
7. While Topojson is more efficient than Geojson especially when it comes to size and data compression. However, it is relatively slow when it comes to handling very large datasets. To address this issue, a non-blocking streaming codec can be used. Hence the importance of this project. • Data is processed as it is received • Data is processed without blocking the program from executing other operations
8. JSONE • Jsone is a non-blocking streaming codec (library) to decode and encode the JSON data format. It can process JSON text without blocking on IO and without a complete in-memory representation of the data.
9. Functions • decode_single_object • encode_value • map_objects • fold_objects
10. • decode_single_object : Decodes a single JSON object from a decoder stream and returns it as an Ezjsone.value. It works by recursively processing the JSON elements using an internal stack to keep track of nested objects and arrays. • encode_value: Encodes a JSON value to its Topojson equivalent. It uses a stack to keep track of the context while encoding arrays and objects. HELPER FUNCTIONS
11. MAIN FUNCTIONS • map_objects: Maps over objects in a TopoJSON object and applies the provided function f to the object's geometry, returning a new TopoJSON object with the updated geometries. • fold_objects: Identical to map_object except that it takes an initial accumulator instead of a destination and you can accumulate a value as you iterate over the objects. The final accumulated value is returned.
12. let map_object f src dst = let decoder = Jsone.decoder src in let encoder = Jsone.encoder dst in let loc () = Jsone.decoded_range decoder in let enc v = match Jsone.encode encoder v with | `Ok -> () | `Partial -> raise (Abort (`Unexpected "partial encoding")) in let rec loop_through_objects decoder = match Jsone.decode decoder with | `Lexeme `Oe -> ignore (enc (`Lexeme `Oe)); () | `Lexeme (`Name geometry_name) -> ( let geometry_json = decode_single_object decoder in match Topo.Geometry.of_json geometry_json with | Error (`Msg m) -> failwith m | Ok g -> let new_name, new_geometry = f (geometry_name, g) in enc (`Lexeme (`Name new_name)); encode_value encoder (Topo.Geometry.to_json new_geometry); loop_through_objects decoder) | _ -> failwith "Unexpected lexeme" in
13. let rec go () = match Jsone.decode decoder with | `Lexeme (`Name "objects" as t) -> ( enc (`Lexeme t); match Jsone.decode decoder with | `Lexeme `Os -> enc (`Lexeme `Os); loop_through_objects decoder; go () | `Lexeme _ as t -> enc t; go () | `Error e -> raise (Abort (`Error (loc (), e))) | `End -> ignore @@ Jsone.encode encoder `End | `Await -> assert false) | `Lexeme _ as t -> enc t; go () | `Error e -> raise (Abort (`Error (loc (), e))) | `End -> ignore @@ Jsone.encode encoder `End | `Await -> assert false in try Ok (go ()) with Abort e -> Error e
14. let test_fold_object ~fs () = with_src fs "example.json" @@ fun src -> let initial_acc = 0 in let open Topojsone in let f acc (_, _geometry) = acc + 1 in match fold_object f initial_acc src with | Ok final_acc -> print_endline ("Total acc: " ^ string_of_int final_acc); if final_acc > 0 then () else failwith "fold_object test failed" | Error e -> Topojsone.Err.pp Format.err_formatter e; failwith "Internal err" Total acc: 2 TEST
15. Skills learned Hard skills • OCaml programming language • Geospatial concepts • Pp formatter • MDX documentation • Technical Writing • Communication • Critical thinking • Expressing Technical issues in simple terms • Asking questions Soft skills
16. Image credit: BetterTeam • Outreachy Organizers • Patrick Ferris • OCaml community