Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kickstart your Kafka with Faker Data

A23789f299ed06fe7d9f1c6940440bfa?s=47 FTisiot
March 26, 2021

Kickstart your Kafka with Faker Data

We all love to play with the shiny toys, but an event stream with no events is a sorry sight. In this session you’ll see how to create your own streaming dataset for Apache Kafka using Python and the Faker library. You’ll learn how to create a random data producer and define the structure and rate of its message delivery. Randomly-generated data is often hilarious in its own right, and it adds just the right amount of fun to any Kafka and its integrations!

A23789f299ed06fe7d9f1c6940440bfa?s=128

FTisiot

March 26, 2021
Tweet

Transcript

  1. Kickstart your Kafka with Faker Data Francesco Tisiot - Developer

    Advocate @ftisiot
  2. @ftisiot Kafka is Up!

  3. @ftisiot What Now?

  4. @ftisiot

  5. @ftisiot Choose Your Hero

  6. @ftisiot from kafka import KafkaProducer Kafka Python producer = KafkaProducer(

    bootstrap_servers=[‘broker1:1234'] ) producer.send( 'my-topic', b’raw_bytes' ) producer.flush()
  7. @ftisiot 🦀 $ pip install Faker 1. Install from faker

    import Faker fake = Faker() 2. Use Faker fake.name() # 'Lucy Cechtelar' fake.address() # '426 Jordy Lodge # Cartwrightshire, SC 88120-6700'
  8. @ftisiot Faker fake = Faker('it_IT') #default is fake.name() #Sig. Severino

    Zaccagnini fake.address() #Incrocio Milanesi 58 #Livio sardo, 33587 Palermo (BZ) 3. Localize fake.unique.name() 4. Uniqueness & Repeatable Results Faker.seed(4321)
  9. @ftisiot Providers https://faker.readthedocs.io/en/master/providers.html Automotive Bank Geo Python Pizza?

  10. @ftisiot from faker.providers import BaseProvider 5. Extend valid_pizza_names = [

    'Margherita', 'Marinara', 'Diavola', 'Mari & Monti', 'Salami', 'Peperoni' ] class PizzaProvider(BaseProvider): def pizza_name(self): return random.choice(valid_pizza_names)
  11. @ftisiot fake.add_provider(PizzaProvider) 6. Create message = { 'id': 0, 'shop':

    fake.shop_name(), 'name': fake.unique.name(), 'phoneNumber': fake.unique.phone_number(), 'address': fake.address(), 'pizzas': pizzas } pizzas = [] for pizza in range(5): pizzas.append(fake.pizza_name())
  12. @ftisiot { "id": 0, "shop": "Luigis Pizza", "name": "Arsenio Pisaroni-Boccaccio",

    "phoneNumber": "+39 51 0290746", "address": "Viale Donatoni 01\nBianca terme, 85639 Novara (CZ)", "pizzas": [ { "pizzaName": "Diavola", "additionalToppings": ["ham"] }, { "pizzaName": "Diavola", "additionalToppings": ["mozzarella","banana","onion"] }, ] } 7. Output
  13. @ftisiot Resources https://github.com/joke2k/faker https://aiven.io - ks2021aiven 500$ go.aiven.io/pizza_producer_repo go.aiven.io/pizza_producer_blog