Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kickstart your Kafka with Faker Data

A23789f299ed06fe7d9f1c6940440bfa?s=47 FTisiot
March 26, 2021

Kickstart your Kafka with Faker Data

We all love to play with the shiny toys, but an event stream with no events is a sorry sight. In this session you’ll see how to create your own streaming dataset for Apache Kafka using Python and the Faker library. You’ll learn how to create a random data producer and define the structure and rate of its message delivery. Randomly-generated data is often hilarious in its own right, and it adds just the right amount of fun to any Kafka and its integrations!

A23789f299ed06fe7d9f1c6940440bfa?s=128

FTisiot

March 26, 2021
Tweet

More Decks by FTisiot

Other Decks in Technology

Transcript

  1. Francesco Tisiot - Developer Advocate @ftisiot Kickstart your Apache Kafka

    with Faker Data
  2. @ftisiot | @aiven_io Kafka is Up!

  3. @ftisiot | @aiven_io What Now?

  4. @ftisiot | @aiven_io

  5. @ftisiot | @aiven_io Choose Your Hero

  6. @ftisiot | @aiven_io Kafka Python from kafka import KafkaProducer producer

    = KafkaProducer( bootstrap_servers= ['broker1:1234'] ) producer.send( 'my-topic', b'raw_bytes' ) producer.flush()
  7. @ftisiot | @aiven_io Faker 🦀$ pip install Faker 1. Install

    from faker import Faker fake = Faker() 2. Use fake.name() # 'Lucy Cechtelar' fake.address() # '426 Jordy Lodge # Cartwrightshire, SC 88120-6700'
  8. @ftisiot | @aiven_io Faker fake = Faker('it_IT') #default is en_EN

    fake.name() #Sig. Severino Zaccagnini fake.address() #Incrocio Milanesi 58 #Livio sardo, 33587 Palermo (BZ) 3. Localize fake.unique.name() 4. Uniqueness & Repeatable Results Faker.seed(4321)
  9. @ftisiot | @aiven_io Providers https:/ /faker.readthedocs.io/en/master/providers.html Automotive Bank Geo Python

    Pizza?
  10. @ftisiot | @aiven_io from faker.providers import BaseProvider valid_pizza_names = [

    'Margherita', 'Marinara', 'Diavola', 'Mari & Monti', 'Salami' ] class PizzaProvider(BaseProvider): def pizza_name(self): return random.choice(valid_pizza_names) 5. Extend
  11. @ftisiot | @aiven_io fake.add_provider(PizzaProvider) 6. Create message = { 'id':

    0, 'shop': fake.shop_name(), 'name': fake.unique.name(), 'phoneNumber': fake.unique.phone_number(), 'address': fake.address(), 'pizzas': pizzas } pizzas = [] for pizza in range(5): pizzas.append(fake.pizza_name())
  12. @ftisiot | @aiven_io { "id": 0, "shop": "Luigis Pizza", "name":

    "Arsenio Pisaroni-Boccaccio", "phoneNumber": "+39 51 0290746", "address": "Viale Donatoni 01\nBianca terme, 85639 Novara (CZ)", "pizzas": [ {"pizzaName": “Diavola", "additionalToppings": [“ham"] }, {"pizzaName": "Diavola", "additionalToppings": ["mozzarella","banana","onion"] }, ] 7. Output
  13. @ftisiot | @aiven_io References https:/ /github.com/joke2k/faker https:/ /aiven.io go.aiven.io/pizza_producer_repo go.aiven.io/pizza_producer_blog