Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kickstart your Kafka with Faker Data

March 26, 2021

Kickstart your Kafka with Faker Data

We all love to play with the shiny toys, but an event stream with no events is a sorry sight. In this session you’ll see how to create your own streaming dataset for Apache Kafka using Python and the Faker library. You’ll learn how to create a random data producer and define the structure and rate of its message delivery. Randomly-generated data is often hilarious in its own right, and it adds just the right amount of fun to any Kafka and its integrations!


March 26, 2021

More Decks by FTisiot

Other Decks in Technology


  1. @ftisiot | @aiven_io Kafka Python from kafka import KafkaProducer producer

    = KafkaProducer( bootstrap_servers= ['broker1:1234'] ) producer.send( 'my-topic', b'raw_bytes' ) producer.flush()
  2. @ftisiot | @aiven_io Faker 🦀$ pip install Faker 1. Install

    from faker import Faker fake = Faker() 2. Use fake.name() # 'Lucy Cechtelar' fake.address() # '426 Jordy Lodge # Cartwrightshire, SC 88120-6700'
  3. @ftisiot | @aiven_io Faker fake = Faker('it_IT') #default is en_EN

    fake.name() #Sig. Severino Zaccagnini fake.address() #Incrocio Milanesi 58 #Livio sardo, 33587 Palermo (BZ) 3. Localize fake.unique.name() 4. Uniqueness & Repeatable Results Faker.seed(4321)
  4. @ftisiot | @aiven_io from faker.providers import BaseProvider valid_pizza_names = [

    'Margherita', 'Marinara', 'Diavola', 'Mari & Monti', 'Salami' ] class PizzaProvider(BaseProvider): def pizza_name(self): return random.choice(valid_pizza_names) 5. Extend
  5. @ftisiot | @aiven_io fake.add_provider(PizzaProvider) 6. Create message = { 'id':

    0, 'shop': fake.shop_name(), 'name': fake.unique.name(), 'phoneNumber': fake.unique.phone_number(), 'address': fake.address(), 'pizzas': pizzas } pizzas = [] for pizza in range(5): pizzas.append(fake.pizza_name())
  6. @ftisiot | @aiven_io { "id": 0, "shop": "Luigis Pizza", "name":

    "Arsenio Pisaroni-Boccaccio", "phoneNumber": "+39 51 0290746", "address": "Viale Donatoni 01\nBianca terme, 85639 Novara (CZ)", "pizzas": [ {"pizzaName": “Diavola", "additionalToppings": [“ham"] }, {"pizzaName": "Diavola", "additionalToppings": ["mozzarella","banana","onion"] }, ] 7. Output