Upgrade to Pro — share decks privately, control downloads, hide ads and more …

An Introduction to Apache Avro

An Introduction to Apache Avro

An Introduction to Apache Avro, what is it and how is it used ?
Plus some code examples to show JSON etc.

Mike Frampton

July 19, 2013
Tweet

More Decks by Mike Frampton

Other Decks in Technology

Transcript

  1. Apache Avro • What is it ? • Language API's

    • JSON Example • Schema Evolution • Avro Use Example www.semtech-solutions.co.nz [email protected]
  2. Avro – What is it ? • Data serialization system

    • Uses JSON based schemas • Uses RPC calls to send data • Schema's sent during data exchange • Integrated with many languages • Fast binary data format or encode with JSON www.semtech-solutions.co.nz [email protected]
  3. Avro – Language API's Avro API's exist for the following

    languages • Java • C# • C • C++ • Python • Ruby www.semtech-solutions.co.nz [email protected]
  4. Avro – JSON Example This text shows an example JSON

    schema for a customer record { "namespace": "customer.avro", "type": "record", "name": "customer", "fields": [ {"name": "name", "type": "string"}, {"name": "age", "type": "int"}, {"name": "address", "type": "string"}, {"name": "phone", "type": "string"}, {"name": "email", "type": "string"}, ] } www.semtech-solutions.co.nz [email protected]
  5. Avro – Schema Evolution In a fast changing environment it

    is possible to • Write data to file with a schema • Change the schema – Add extra fields – Delete fields – Rename fields • And still read written file with changed schema www.semtech-solutions.co.nz [email protected]
  6. Avro – Use Example // define schema for customer {

    "namespace": "customer.avro", "type": "record", "name": "Customer", "fields": [ {"name": "name", "type": "string"}, {"name": "age", "type": "int"}, {"name": "address", "type": "string"}, {"name": "phone", "type": "string"}, {"name": "email", "type": "string"}, ] } // compile the schema java -jar /....../avro-tools-1.7.4.jar compile schema customer.avsc . // create a couple of customers Customer cust1 = new Customer("Dave", 33, "12 redmont street", "021234675", "davef@[email protected]" ); Customer cust2 = new Customer("Emily", 27, "13a lake cresent", "022876234", "esuanders@[email protected]" ); // now serialize customers to disk File file = new File("customers.avro"); DatumWriter<Customer> userDatumWriter = new SpecificDatumWriter<Customer>(Customer.class); DataFileWriter<Customer> dataFileWriter = new DataFileWriter<Customer>(customerDatumWriter); dataFileWriter.create(cust1.getSchema(), new File("customers.avro")); dataFileWriter.append(cust1); dataFileWriter.append(cust2); dataFileWriter.close(); www.semtech-solutions.co.nz [email protected]
  7. Contact Us • Feel free to contact us at –

    www.semtech-solutions.co.nz – [email protected] • We offer IT project consultancy • We are happy to hear about your problems • You can just pay for those hours that you need • To solve your problems