Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kickstart your Kafka with Faker Data

A23789f299ed06fe7d9f1c6940440bfa?s=47 FTisiot
March 26, 2021

Kickstart your Kafka with Faker Data

We all love to play with the shiny toys, but an event stream with no events is a sorry sight. In this session you’ll see how to create your own streaming dataset for Apache Kafka using Python and the Faker library. You’ll learn how to create a random data producer and define the structure and rate of its message delivery. Randomly-generated data is often hilarious in its own right, and it adds just the right amount of fun to any Kafka and its integrations!



March 26, 2021


  1. Kickstart your Kafka with Faker Data Francesco Tisiot - Developer

    Advocate @ftisiot
  2. @ftisiot Kafka is Up!

  3. @ftisiot What Now?

  4. @ftisiot

  5. @ftisiot Choose Your Hero

  6. @ftisiot from kafka import KafkaProducer Kafka Python producer = KafkaProducer(

    bootstrap_servers=[‘broker1:1234'] ) producer.send( 'my-topic', b’raw_bytes' ) producer.flush()
  7. @ftisiot 🦀 $ pip install Faker 1. Install from faker

    import Faker fake = Faker() 2. Use Faker fake.name() # 'Lucy Cechtelar' fake.address() # '426 Jordy Lodge # Cartwrightshire, SC 88120-6700'
  8. @ftisiot Faker fake = Faker('it_IT') #default is fake.name() #Sig. Severino

    Zaccagnini fake.address() #Incrocio Milanesi 58 #Livio sardo, 33587 Palermo (BZ) 3. Localize fake.unique.name() 4. Uniqueness & Repeatable Results Faker.seed(4321)
  9. @ftisiot Providers https://faker.readthedocs.io/en/master/providers.html Automotive Bank Geo Python Pizza?

  10. @ftisiot from faker.providers import BaseProvider 5. Extend valid_pizza_names = [

    'Margherita', 'Marinara', 'Diavola', 'Mari & Monti', 'Salami', 'Peperoni' ] class PizzaProvider(BaseProvider): def pizza_name(self): return random.choice(valid_pizza_names)
  11. @ftisiot fake.add_provider(PizzaProvider) 6. Create message = { 'id': 0, 'shop':

    fake.shop_name(), 'name': fake.unique.name(), 'phoneNumber': fake.unique.phone_number(), 'address': fake.address(), 'pizzas': pizzas } pizzas = [] for pizza in range(5): pizzas.append(fake.pizza_name())
  12. @ftisiot { "id": 0, "shop": "Luigis Pizza", "name": "Arsenio Pisaroni-Boccaccio",

    "phoneNumber": "+39 51 0290746", "address": "Viale Donatoni 01\nBianca terme, 85639 Novara (CZ)", "pizzas": [ { "pizzaName": "Diavola", "additionalToppings": ["ham"] }, { "pizzaName": "Diavola", "additionalToppings": ["mozzarella","banana","onion"] }, ] } 7. Output
  13. @ftisiot Resources https://github.com/joke2k/faker https://aiven.io - ks2021aiven 500$ go.aiven.io/pizza_producer_repo go.aiven.io/pizza_producer_blog