An Introduction to Apache Avro

An Introduction to Apache Avro, what is it and how is it used ?
Plus some code examples to show JSON etc.

Transcript

1. Apache Avro • What is it ? • Language API's • JSON Example • Schema Evolution • Avro Use Example www.semtech-solutions.co.nz info@semtech-solutions.co.nz
2. Avro – What is it ? • Data serialization system • Uses JSON based schemas • Uses RPC calls to send data • Schema's sent during data exchange • Integrated with many languages • Fast binary data format or encode with JSON www.semtech-solutions.co.nz info@semtech-solutions.co.nz
3. Avro – Language API's Avro API's exist for the following languages • Java • C# • C • C++ • Python • Ruby www.semtech-solutions.co.nz info@semtech-solutions.co.nz
4. Avro – JSON Example This text shows an example JSON schema for a customer record { "namespace": "customer.avro", "type": "record", "name": "customer", "fields": [ {"name": "name", "type": "string"}, {"name": "age", "type": "int"}, {"name": "address", "type": "string"}, {"name": "phone", "type": "string"}, {"name": "email", "type": "string"}, ] } www.semtech-solutions.co.nz info@semtech-solutions.co.nz
5. Avro – Schema Evolution In a fast changing environment it is possible to • Write data to file with a schema • Change the schema – Add extra fields – Delete fields – Rename fields • And still read written file with changed schema www.semtech-solutions.co.nz info@semtech-solutions.co.nz
6. Avro – Use Example // define schema for customer { "namespace": "customer.avro", "type": "record", "name": "Customer", "fields": [ {"name": "name", "type": "string"}, {"name": "age", "type": "int"}, {"name": "address", "type": "string"}, {"name": "phone", "type": "string"}, {"name": "email", "type": "string"}, ] } // compile the schema java -jar /....../avro-tools-1.7.4.jar compile schema customer.avsc . // create a couple of customers Customer cust1 = new Customer("Dave", 33, "12 redmont street", "021234675", "davef@somedom@co.nz" ); Customer cust2 = new Customer("Emily", 27, "13a lake cresent", "022876234", "esuanders@somenz@co.nz" ); // now serialize customers to disk File file = new File("customers.avro"); DatumWriter<Customer> userDatumWriter = new SpecificDatumWriter<Customer>(Customer.class); DataFileWriter<Customer> dataFileWriter = new DataFileWriter<Customer>(customerDatumWriter); dataFileWriter.create(cust1.getSchema(), new File("customers.avro")); dataFileWriter.append(cust1); dataFileWriter.append(cust2); dataFileWriter.close(); www.semtech-solutions.co.nz info@semtech-solutions.co.nz
7. Contact Us • Feel free to contact us at – www.semtech-solutions.co.nz – info@semtech-solutions.co.nz • We offer IT project consultancy • We are happy to hear about your problems • You can just pay for those hours that you need • To solve your problems