Kickstart your Kafka with Faker Data

We all love to play with the shiny toys, but an event stream with no events is a sorry sight. In this session you’ll see how to create your own streaming dataset for Apache Kafka using Python and the Faker library. You’ll learn how to create a random data producer and define the structure and rate of its message delivery. Randomly-generated data is often hilarious in its own right, and it adds just the right amount of fun to any Kafka and its integrations!

Transcript

1. Francesco Tisiot - Developer Advocate @ftisiot Kickstart your Apache Kafka with Faker Data
2. @ftisiot | @aiven_io Kafka is Up!
3. @ftisiot | @aiven_io What Now?
4. @ftisiot | @aiven_io
5. @ftisiot | @aiven_io Choose Your Hero
6. @ftisiot | @aiven_io Kafka Python from kafka import KafkaProducer producer = KafkaProducer( bootstrap_servers= ['broker1:1234'] ) producer.send( 'my-topic', b'raw_bytes' ) producer.flush()
7. @ftisiot | @aiven_io Faker 🦀$ pip install Faker 1. Install from faker import Faker fake = Faker() 2. Use fake.name() # 'Lucy Cechtelar' fake.address() # '426 Jordy Lodge # Cartwrightshire, SC 88120-6700'
8. @ftisiot | @aiven_io Faker fake = Faker('it_IT') #default is en_EN fake.name() #Sig. Severino Zaccagnini fake.address() #Incrocio Milanesi 58 #Livio sardo, 33587 Palermo (BZ) 3. Localize fake.unique.name() 4. Uniqueness & Repeatable Results Faker.seed(4321)
9. @ftisiot | @aiven_io Providers https:/ /faker.readthedocs.io/en/master/providers.html Automotive Bank Geo Python Pizza?
10. @ftisiot | @aiven_io from faker.providers import BaseProvider valid_pizza_names = [ 'Margherita', 'Marinara', 'Diavola', 'Mari & Monti', 'Salami' ] class PizzaProvider(BaseProvider): def pizza_name(self): return random.choice(valid_pizza_names) 5. Extend
11. @ftisiot | @aiven_io fake.add_provider(PizzaProvider) 6. Create message = { 'id': 0, 'shop': fake.shop_name(), 'name': fake.unique.name(), 'phoneNumber': fake.unique.phone_number(), 'address': fake.address(), 'pizzas': pizzas } pizzas = [] for pizza in range(5): pizzas.append(fake.pizza_name())
12. @ftisiot | @aiven_io { "id": 0, "shop": "Luigis Pizza", "name": "Arsenio Pisaroni-Boccaccio", "phoneNumber": "+39 51 0290746", "address": "Viale Donatoni 01\nBianca terme, 85639 Novara (CZ)", "pizzas": [ {"pizzaName": “Diavola", "additionalToppings": [“ham"] }, {"pizzaName": "Diavola", "additionalToppings": ["mozzarella","banana","onion"] }, ] 7. Output
13. @ftisiot | @aiven_io References https:/ /github.com/joke2k/faker https:/ /aiven.io go.aiven.io/pizza_producer_repo go.aiven.io/pizza_producer_blog