SymfonyCon 2022 Keynote: Webhooks

Transcript

1. Don't call us, we will call you
   Fabien Potencier
   

   View Slide

2. Don't call us, we will call you
   Fabien Potencier
   

   View Slide

3. Don't call us, we will call you
   Fabien Potencier
   

   View Slide

4. https://symfony.com/components | https://symfony.com/stats/downloads
   Let's make it 200?
   That's almost
   
   
   15 billion dls
   ~240 dl/second
   Full-stack
   PHP compat
   Frontend
   3rd party
   

   providers
   Most of
   

   the code is here!
   

   View Slide

5. Provide
   fl
   exible low-level abstractions
   

   Build higher-level abstractions Still on-going
   I want more of these!
   

   Productivity boosts
   

   View Slide

6. RateLimiter
   Lock
   Login Throttling
   Cache
   3.3 3.1
   5.2
   5.2
   security:
   
   
   firewalls:
   
   
   default:
   
   
   # default limits to 5 login attempts per minute,
   
   
   # the number can be configured via "max_attempts"
   
   
   login_throttling: ~
   
   
   # define your own RateLimiter
   
   
   login_throttling:
   
   
   limiter: login
   Great high-level features
   
   
   Throttling Login Attempts
   

   View Slide

7. Twig
   Great high-level features
   
   
   Sending Emails
   Serializer
   2.0
   + Dotenv
   3.3
   + Messenger
   4.1
   +
   Mime
   4.3
   +
   HttpClient
   4.3
   + Mailer
   4.3
   =>
   

   View Slide

8. Con
   fi
   guration
   
   
   Event Dispatcher
   
   
   Messenger
   
   
   Twig
   
   
   Dotenv
   
   
   Web Pro
   fi
   ler
   
   
   Speci
   fi
   c PHPUnit constraints
   
   
   ...
   An Ecocsystem with a Shared Experience
   It's also about the processes:
   

   BC promise
   

   Release cycle
   
   
   Security management
   

   Consistency across the board
   
   
   ...
   

   View Slide

9. Incremental Improvements from the Community
   
   
   Mailer Examples
   ✓ More providers
   
   
   ✓ DKIM
   
   
   ✓ Draft Emails
   
   
   ✓ mailer:test command
   
   
   ✓ Tags and Metadata
   Currently supported by
   

   MailChimp, Mailgun, Mailpace*, Postmark,
   

   Sendgrid, Sendinblue, Amazon SES*
   The power of the community
   
   
   Based on real-world needs
   

   View Slide

10. Building flexible high-level abstractions
    

    on top of solid low-level ones
    
    
    What's next?
    

    View Slide

11. Clicks, Bounces, Spam, and more…
    

    View Slide

12. email_webhooks:
    
    
    resource: '@MailerBundle/Resources/config/routing/webhooks.xml'
    
    
    prefix: /_email
    Webhooks: Mount the “universal” controller
    

    View Slide

13. class WebhookFailureHandler implements MessageHandlerInterface
    
    
    {
    
    
    private $logger;
    
    
    public function __construct(LoggerInterface $logger)
    
    
    {
    
    
    $this->logger = $logger;
    
    
    }
    
    
    public function __invoke(FailureWebhook $failure)
    
    
    {
    
    
    $this->logger->error('APP: New email failure',
    
    
    ['type' => $failure->getType(), 'email' => $failure->getEmail()]);
    
    
    }
    
    
    }
    Webhooks: Do something…
    

    View Slide

14. routing:
    
    
    'Symfony\Component\Mailer\Webhook\FailureWebhook': amqp
    Webhooks: … sync or async
    Webhooks were tied
    

    to Mailer
    Very specialized
    

    webhook events
    

    View Slide

15. Time to revisit?
    

    View Slide

16. Does Webhooks handling
    

    deserve to become a
    

    Symfony Component?
    Do I have a need myself?
    Do I feel some pain?
    

    View Slide

17. Our Symfony Corp
    

    "vanity" Slack channel
    It leverages several components
    

    like HTTP Client, Noti
    fi
    er
    
    
    And some custom code
    

    for Stripe Webhooks
    Slack support here was key
    

    to creating the Notifier component
    Second use case for Webhooks...
    

    View Slide

18. Does Webhooks handling deserve
    

    to become a Symfony Component?
    Are there any best practices
    

    that could be abstracted and enforced?
    Is it generic enough to build a
    

    useful and reusable abstraction?
    Could Symfony leverage it
    

    in some components?
    Would it be useful
    

    for many of our users?
    

    View Slide

19. Does Webhooks handling deserve
    

    to become a Symfony Component?
    ✓ Support Mailer and Noti
    fi
    er
    
    
    ✓ Consume and send
    
    
    ✓ Replay payloads in tests
    
    
    ✓ Prevent production replay
    
    
    ✓ Promote dataless noti
    fi
    cations
    
    
    ✓ Check payload signatures
    
    
    ✓ ...
    Abstract and reuse
    Implement
    

    common "best practices"
    Fully featured
    

    View Slide

20. Webhook
    Will land in 6.3
    

    as an experimental component
    

    View Slide

21. A Webhook
    

    is a noti
    fi
    cation
    

    from one system
    

    to another system
    

    of some state change
    
    
    A successful payment in Stripe
    Twilio sent an SMS successfully
    A SaaS provider
    

    or an internal system
    Your application
    It allows to react to changes from external systems
    What are Webhooks?
    

    View Slide

22. A system makes
    

    real-time HTTP requests
    

    to other servers,
    

    sending events as JSON payloads
    Very common, but not universal
    language agnostic
    What are Webhooks?
    Broadcast
    

    View Slide

23. Well understood universal concept
    
    
    with support from many providers
    
    
    But not standardized
    Payload is freeform
    What are Webhooks?
    Security is provider dependent
    

    View Slide

24. Symfony Webhooks
    

    Support in Practice
    

    View Slide

25. A Single HTTP Entry Point
    

    for Consuming Webhooks
    webhook:
    
    
    resource: ../vendor/symfony/webhook/Controller/
    
    
    type: attribute
    
    
    prefix: /webhook
    Prefix of
    

    your choice
    Similar to
    

    HTTP fragments
    

    Web Profiler
    
    
    ...
    

    View Slide

26. framework:
    
    
    webhook:
    
    
    routing:
    
    
    emails:
    
    
    service: mailer.webhook_request_parser.mailgun
    
    
    secret: '%env(MAILGUN_WEBHOOK_SECRET)%'
    Webhooks Routing
    Your provider
    Similar to
    

    Messenger Routing
    https://somwhere.com/webhook/emails
    

    View Slide

27. #[AsRemoteEventConsumer(name: 'emails')]
    
    
    class MailerEventConsumer implements ConsumerInterface
    
    
    {
    
    
    /**
    
    
    * @param AbstractMailerEvent $event
    
    
    */
    
    
    public function consume(Event $event): void
    
    
    {
    
    
    $email = $event->getRecipientEmail();
    
    
    error_log(match ($event->getName()) {
    
    
    MailerDeliveryEvent::BOUNCE => sprintf('Email to %s bounced (%s)', $email, $event->getReason()),
    
    
    MailerEngagementEvent::UNSUBSCRIBE => sprintf('Unsubscribe from %s', $email),
    
    
    default => sprintf('Receive unhandled email event %s', $event->getName()),
    
    
    });
    
    
    }
    
    
    }
    Mailer Webhook Consumer
    Logic independent
    

    from the provider!
    

    View Slide

28. framework:
    
    
    messenger:
    
    
    routing:
    
    
    Symfony\Component\RemoteEvent\Messenger\ConsumeRemoteEventMessage: async
    👌Webhook Best Practice
    

    Fast HTTP Response
    For later...
    

    View Slide

29. protected function getRequestMatcher(): RequestMatcherInterface
    
    
    {
    
    
    return new ChainRequestMatcher([
    
    
    new MethodRequestMatcher('POST'),
    
    
    new IsJsonRequestMatcher(),
    
    
    ]);
    
    
    }
    1 Incoming HTTP Request Validation
    return new ChainRequestMatcher([
    
    
    new MethodRequestMatcher('POST'),
    
    
    // https://postmarkapp.com/support/article/800-ips-for-firewalls#webhooks
    
    
    // localhost is added for testing
    
    
    new IpsRequestMatcher(['3.134.147.250', '50.31.156.6', '50.31.156.77', '18.217.206.57', '127.0.0.1']),
    
    
    new IsJsonRequestMatcher(),
    
    
    ]); Enforce provider security
    Reuses HttpFoundation infrastructure
    
    
    Used by Security
    

    View Slide

30. if (
    
    
    !isset($content['signature']['timestamp'])
    
    
    || ...
    
    
    ) {
    
    
    throw new RejectWebhookException(406, 'Payload is malformed.');
    
    
    }
    
    
    --
    
    
    private function validateSignature(array $signature, string $secret): void
    
    
    {
    
    
    if (!hash_equals($signature['signature'], hash_hmac('sha256', $signature['timestamp'].$signature['token'], $secret))
    
    
    throw new RejectWebhookException(406, 'Signature is wrong.');
    
    
    }
    
    
    }
    2 Signature & Payload Validation
    Enforce provider security
    

    View Slide

31. final class MailgunPayloadConverter implements PayloadConverterInterface
    
    
    {
    
    
    /** @return AbstractMailerEvent */
    
    
    public function convert(array $payload): Event
    
    
    {
    
    
    if (in_array($payload['event'], ['failed', 'accepted', 'rejected', 'delivered'])) {
    
    
    // ...
    
    
    } else {
    
    
    $event = match ($payload['event']) {
    
    
    'clicked' => MailerEngagementEvent::CLICK,
    
    
    // ...
    
    
    default => throw new RejectWebhookException(406, sprintf('Not supported event "%s".', $payload['event']))
    
    
    };
    
    
    $wh = new MailerEngagementEvent($event, $payload['id'], $payload);
    
    
    }
    
    
    if (!$date = \DateTimeImmutable::createFromFormat('U.u', $payload['timestamp'])) {
    
    
    throw new RejectWebhookException(406, sprintf('Invalid date "%s".', $date));
    
    
    }
    
    
    $wh->setRecipientEmail($payload['recipient']);
    
    
    $wh->setTags($payload['tags']);
    
    
    // ...
    
    
    3 Payload Parsing & Conversion
    Where the standardization happens!
    

    View Slide

32. namespace Symfony\Component\RemoteEvent\Event\Mailer;
    
    
    abstract class AbstractMailerEvent extends Event
    
    
    {
    
    
    private \DateTimeImmutable $date;
    
    
    private string $email = '';
    
    
    private array $metadata = [];
    
    
    private array $tags = [];
    
    
    public function getDate(): \DateTimeImmutable
    
    
    {
    
    
    return $this->date;
    
    
    }
    
    
    public function getRecipientEmail(): string
    
    
    {
    
    
    return $this->email;
    
    
    }
    
    
    public function getMetadata(): array
    
    
    {
    
    
    return $this->metadata;
    
    
    }
    
    
    public function getTags(): array
    
    
    {
    
    
    return $this->tags;
    
    
    }
    
    
    }
    Mailer
    

    Event
    Supports common
    

    provider features
    

    View Slide

33. final class MailerDeliveryEvent extends AbstractMailerEvent
    
    
    {
    
    
    public const RECEIVED = 'received';
    
    
    public const DROPPED = 'dropped';
    
    
    public const DELIVERED = 'delivered';
    
    
    public const DEFERRED = 'deferred';
    
    
    public const BOUNCE = 'bounce';
    
    
    private string $reason = '';
    
    
    public function getReason(): string
    
    
    {
    
    
    return $this->reason;
    
    
    }
    
    
    }
    final class MailerEngagementEvent extends AbstractMailerEvent
    
    
    {
    
    
    public const OPEN = 'open';
    
    
    public const CLICK = 'click';
    
    
    public const SPAM = 'spam';
    
    
    public const UNSUBSCRIBE = 'unsubscribe';
    
    
    }
    Mailer
    

    Event
    Fined grained
    

    Provider agnostic
    

    View Slide

34. Webhooks for Noti
    fi
    er?
    Can we do the same
    

    for SMS, Chat, ...?
    

    View Slide

35. namespace Symfony\Component\RemoteEvent\Event\Sms;
    
    
    final class SmsEvent extends Event
    
    
    {
    
    
    public const FAILED = 'failed';
    
    
    public const DELIVERED = 'delivered';
    
    
    private string $sms = '';
    
    
    public function getRecipientSms(): string
    
    
    {
    
    
    return $this->sms;
    
    
    }
    
    
    }
    SMSEvent
    Provider agnostic
    

    View Slide

36. #[AsRemoteEventConsumer(name: 'sms')]
    
    
    class SmsEventConsumer implements ConsumerInterface
    
    
    {
    
    
    public function consume(Event $event): void
    
    
    {
    
    
    $phone = $event->getRecipientSms();
    
    
    error_log(match ($event->getName()) {
    
    
    SmsEvent::DELIVERED => sprintf('SMS delivered to %s', $phone),
    
    
    SmsEvent::FAILED => sprintf('SMS failed for %s', $phone),
    
    
    default => sprintf('Receive unhandled SMS event %s', $event->getName()),
    
    
    });
    
    
    }
    
    
    }
    $sms = new SmsMessage('+1411111111', 'A new login was detected!');
    
    
    $sms->options((new TwilioOptions())->webhookUrl('https://somewhere.com/webhook/sms'));
    
    
    $texter->send($sms);
    SMS Webhook Consumer
    framework:
    
    
    webhook:
    
    
    routing:
    
    
    sms:
    
    
    service: Symfony\...\TwilioRequestParser
    Logic independent
    

    from the provider!
    Routing
    
    
    /webhook/sms
    Plumbing specific
    

    to provider
    

    View Slide

37. 🙋 I need your help
    
    
    ‣ Implement support for more providers
    
    
    ‣ Validate event names and common payload items
    
    
    ‣ Write tests
    
    
    ‣ Write documentation
    Next Steps?
    I will submit the Pull Request
    

    after the 6.2 release
    

    View Slide

38. class Event
    
    
    {
    
    
    public function __construct(
    
    
    private string $name,
    
    
    private string $id,
    
    
    private array $payload,
    
    
    ){
    
    
    }
    
    
    public function getName(): string
    
    
    {
    
    
    return $this->name;
    
    
    }
    
    
    public function getId(): string
    
    
    {
    
    
    return $this->id;
    
    
    }
    
    
    public function getPayload(): array
    
    
    {
    
    
    return $this->payload;
    
    
    }
    
    
    }
    name: user.new
    
    
    id: 1337
    name: invoice.paid
    
    
    id: evt_1M2yqF2Gz5TJuMKBAlWnwTKe
    name: complained
    
    
    id: -Agny091SquKnsrW2NEKUA
    name: bounced
    
    
    id: 00000000-0000-0000-0000-000000000000
    Your own internal
    

    application Events
    Implement your own
    

    Webhooks!
    

    View Slide

39. $subscriber = new Subscriber($userDefinedUrl, $userDefinedSecret);
    
    
    $event = new Event('user.new', '1', [
    
    
    'id' => 1,
    
    
    'name' => 'Fabien Potencier',
    
    
    'email' => 'fabien@symfony.com',
    
    
    ]);
    
    
    $this->bus->dispatch(new SendWebhookMessage($subscriber, $event));
    Sends the Webhook
    
    
    via HTTP async
    What about providing Webhooks
    
    
    to other applications?
    Application defined
    Payload
    

    View Slide

40. > POST /webhook/users HTTP/1.1
    
    
    Host: 127.0.0.1:8000
    
    
    X-Event: user.new
    
    
    X-Id: 1
    
    
    X-Signature: sha256=5331fa2b02cd43616c9d8c44cd...
    
    
    Content-Type: application/json
    
    
    Accept: */*
    
    
    Content-Length: 22
    
    
    User-Agent: Symfony HttpClient/Curl
    
    
    Accept-Encoding: gzip
    
    
    {"id":1, "name":"Fabien Potencier", ...}
    POST HTTP request Payload signature
    JSON payload
    HTTP Transport Abstraction
    Default implementation, fully customizable
    

    View Slide

41. #[AsRemoteEventConsumer(name: 'users')]
    
    
    class SomeEventConsumer implements ConsumerInterface
    
    
    {
    
    
    public function consume(Event $event): void
    
    
    {
    
    
    error_log('Receive an internal and generic event '.var_export($event, true));
    
    
    }
    
    
    }
    Some other
    

    Symfony application(s)
    Consumers can be anything
    
    
    ... including Symfony itself for SOA/micro-services/multi-apps/...
    

    View Slide

42. class Event
    
    
    {
    
    
    public function __construct(
    
    
    private string $name,
    
    
    private string $id,
    
    
    private array $payload,
    
    
    ){
    
    
    }
    
    
    public function getName(): string
    
    
    {
    
    
    return $this->name;
    
    
    }
    
    
    public function getId(): string
    
    
    {
    
    
    return $this->id;
    
    
    }
    
    
    public function getPayload(): array
    
    
    {
    
    
    return $this->payload;
    
    
    }
    
    
    }
    Alternative transports:
    
    
    Queues for internal systems
    
    
    APIs
    
    
    ...
    A Webhook is just
    

    one possible transport
    
    
    for an event - via HTTP
    

    View Slide

43. Let's decouple
    

    Remote Events and Webhooks
    vs local events
    
    
    from EventDispatcher
    Webhook events are Remote Events
    Like Mime & Mailer
    

    View Slide

44. RemoteEvent
    Will land in 6.3
    

    as an experimental component
    

    View Slide

45. namespace Symfony\Component\RemoteEvent;
    
    
    class Event
    
    
    {
    
    
    public function __construct(
    
    
    private string $name,
    
    
    private string $id,
    
    
    private array $payload,
    
    
    ){
    
    
    }
    
    
    public function getName(): string
    
    
    {
    
    
    return $this->name;
    
    
    }
    
    
    public function getId(): string
    
    
    {
    
    
    return $this->id;
    
    
    }
    
    
    public function getPayload(): array
    
    
    {
    
    
    return $this->payload;
    
    
    }
    
    
    }
    RemoteEvent
    

    in Symfony
    

    View Slide

46. Webhooks vs API?
    RemoteEvent Component
    Shared infrastructure
    
    
    ‣ RemoteEvent class structure
    
    
    ‣ Payload converters
    
    
    ‣ Consumer code
    push
    Webhook: A system makes real-time HTTP requests
    

    to other servers, the payload is an event
    
    
    API: A system API endpoint that can be polled on-demand;
    

    It returns all events that have occurred since last request
    pull
    Only useful if the provider supports
    

    Webhooks and an API for events
    

    View Slide

47. Event API
    
    
    Resource waste (polling)
    
    
    Not real-time
    Webhooks vs Event API: which one to use?
    Many providers provide both
    Good providers
    

    mitigate those
    Webhooks
    
    
    Uncertain security
    
    
    Downtime might lose events
    
    
    No ordering guarantees
    
    
    Needs some tunnel to test locally
    Frequent HTTP rq
    

    without new events
    Depends on provider retry strategy
    Expose a web server
    

    Security depends on the provider
    Don't call us
    

    We will call you
    Matters sometimes
    

    View Slide

48. Webhooks vs Event API?
    
    
    Use both?
    Subscribe to real-time noti
    fi
    cations via a webhook
    

    Use the API to retrieve the event
    That's dataless notifications
    

    View Slide

49. Webhooks vs Event API?
    
    
    Depends on your use case...
    Vanity notifications
    

    on Slack on paid
    

    invoices from Stripe
    Stripe invoices
    

    for accounting
    Event API Webhooks
    

    View Slide

50. 🙋 I need your help
    
    
    ‣ Payload replay for tests
    
    
    ‣ Logs / Audit trail
    
    
    ‣ Production Replay prevention
    
    
    ‣ Dataless noti
    fi
    cations
    
    
    ‣ Support for CRM, payment gateways, ...?
    
    
    ‣ ...
    Next Steps?
    The main concepts
    

    and low-level infrastructure
    

    are well-defined
    or wait a few years :)
    

    View Slide

51. https://symfony.com/sponsor
    Sponsor Symfony
    Thank you!
    

    View Slide