spark serialization

spark serialization

Transcript

1. Spark Serialization @wwwy3y3
2. What is serialization? • an object can be represented as a sequence of bytes • write to disk , or transfer through internet • JAVA: ObjectInputStream, ObjectOutputStream
3. when does spark use serialization?
4. data shuffle
5. after computation storage level serialize? not fit in memory? MEMORY_ONLY no recomputed MEMORY_AND_DISK no(memory) store on disk MEMORY_ONLY_SER yes recomputed MEMORY_AND_DISK_S ER yes store on disk DISK_ONLY yes store on disk
6. what serialization can we use?
7. Data Serialization • Java serialization: By default, Spark serializes objects using Java’s ObjectOutputStream framework • Kryo serialization: Spark can also use the Kryo library (version 2) to serialize objects more quickly. conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
8. Kryo • register class • If a class is not registered or no serializer is specified, a serializer is chosen automatically from a list
9. source code where they use serializer
10. BlockManager • 在shuffle之前, 會 將RDD cache在 memory或是disk 中 • replicate到其他 node • 提供diskStore, memoryStore
11. memory store • Stores blocks in memory, either as Arrays of deserialized Java objects or as serialized ByteBuffers. • hashMap to store blocks
12. shuffle
13. others • broadcast • sparkSQL • Partitioner • …
14. Custom serializer
15. Serializer Serializer • def newInstance(): SerializerInstance SerializerInstance • def serialize[T: ClassTag](t: T): ByteBuffer • def deserialize[T: ClassTag](bytes: ByteBuffer): T • def serializeStream(s: OutputStream): SerializationStream • def deserializeStream(s: InputStream): DeserializationStream
16. SerializationStream/ DeserializationStream • def read/writeObject • def read/writeKey • def read/writeValue • def flush() • def close() • def writeAll • Iterator (read)
17. JavaSerialization
18. JavaDeserialization
19. KryoSerializer
20. stream
21. use Avro with Kryo • Kryo tries to use its default serializer for generic Records, which will include a lot of unneeded data in each record. • user registers the schemas, then the schema's fingerprint will be sent
22. Avro • Defining a schema {"namespace": "example.avro", "type": "record", "name": "User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] }
23. Avro • Compiling the schema java -jar /path/to/avro-tools-1.7.7.jar compile schema <schema file> <destination> User user = new User("Ben", 7, "red"); • create user
24. Avro DatumWriter<User> userDatumWriter = new SpecificDatumWriter<User>(User.class); DataFileWriter<User> dataFileWriter = new DataFileWriter<User>(userDatumWriter); dataFileWriter.create(user1.getSchema(), new File("users.avro"));
25. without code gen Schema schema = new Schema.Parser().parse(new File(“user.avsc”)); GenericRecord user = new GenericData.Record(schema); DatumWriter<GenericRecord> datumWriter = new GenericDatumWriter<GenericRecord>(schema); DataFileWriter<GenericRecord> dataFileWriter = new DataFileWriter<GenericRecord>(datumWriter); dataFileWriter.create(schema, file);
26. use with spark kryo val schema : Schema = SchemaBuilder .record("testRecord").fields() .requiredString("data") .endRecord() conf.registerAvroSchemas(schema) val record = new Record(schema) record.put("data", "test data")
27. KryoSerializer.scala kryo.register(classOf[GenericRecord], new GenericAvroSerializer(avroSchemas)) kryo.register(classOf[GenericData.Record], new GenericAvroSerializer(avroSchemas))