Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kickstart your Kafka with Faker Data

FTisiot
March 26, 2021

Kickstart your Kafka with Faker Data

We all love to play with the shiny toys, but an event stream with no events is a sorry sight. In this session you’ll see how to create your own streaming dataset for Apache Kafka using Python and the Faker library. You’ll learn how to create a random data producer and define the structure and rate of its message delivery. Randomly-generated data is often hilarious in its own right, and it adds just the right amount of fun to any Kafka and its integrations!

FTisiot

March 26, 2021
Tweet

More Decks by FTisiot

Other Decks in Technology

Transcript

  1. Francesco Tisiot - Developer Advocate @ftisiot Kickstart your Apache Kafka

    with Faker Data
  2. @ftisiot | @aiven_io Kafka is Up!

  3. @ftisiot | @aiven_io What Now?

  4. @ftisiot | @aiven_io

  5. @ftisiot | @aiven_io Choose Your Hero

  6. @ftisiot | @aiven_io Kafka Python from kafka import KafkaProducer producer

    = KafkaProducer( bootstrap_servers= ['broker1:1234'] ) producer.send( 'my-topic', b'raw_bytes' ) producer.flush()
  7. @ftisiot | @aiven_io Faker 🦀$ pip install Faker 1. Install

    from faker import Faker fake = Faker() 2. Use fake.name() # 'Lucy Cechtelar' fake.address() # '426 Jordy Lodge # Cartwrightshire, SC 88120-6700'
  8. @ftisiot | @aiven_io Faker fake = Faker('it_IT') #default is en_EN

    fake.name() #Sig. Severino Zaccagnini fake.address() #Incrocio Milanesi 58 #Livio sardo, 33587 Palermo (BZ) 3. Localize fake.unique.name() 4. Uniqueness & Repeatable Results Faker.seed(4321)
  9. @ftisiot | @aiven_io Providers https:/ /faker.readthedocs.io/en/master/providers.html Automotive Bank Geo Python

    Pizza?
  10. @ftisiot | @aiven_io from faker.providers import BaseProvider valid_pizza_names = [

    'Margherita', 'Marinara', 'Diavola', 'Mari & Monti', 'Salami' ] class PizzaProvider(BaseProvider): def pizza_name(self): return random.choice(valid_pizza_names) 5. Extend
  11. @ftisiot | @aiven_io fake.add_provider(PizzaProvider) 6. Create message = { 'id':

    0, 'shop': fake.shop_name(), 'name': fake.unique.name(), 'phoneNumber': fake.unique.phone_number(), 'address': fake.address(), 'pizzas': pizzas } pizzas = [] for pizza in range(5): pizzas.append(fake.pizza_name())
  12. @ftisiot | @aiven_io { "id": 0, "shop": "Luigis Pizza", "name":

    "Arsenio Pisaroni-Boccaccio", "phoneNumber": "+39 51 0290746", "address": "Viale Donatoni 01\nBianca terme, 85639 Novara (CZ)", "pizzas": [ {"pizzaName": “Diavola", "additionalToppings": [“ham"] }, {"pizzaName": "Diavola", "additionalToppings": ["mozzarella","banana","onion"] }, ] 7. Output
  13. @ftisiot | @aiven_io References https:/ /github.com/joke2k/faker https:/ /aiven.io go.aiven.io/pizza_producer_repo go.aiven.io/pizza_producer_blog