Empowering App Dev by Nicely-Crafted High-Level AWS Components

English version of a presentation slide for TreasureData Tech Talk 2022
https://techplay.jp/event/879660

Transcript

1. #TdTechTalk 2022-11-29
   EMPOWERING APP DEV
   BY NICELY-CRAFTED
   HIGH-LEVEL AWS
   COMPONENTS
   TATSUNO “Taz” Yasuhiro, Insights and Data Systems group
   

   View Slide

2. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
   Area I talk today
   2
   DB
   App
   App
   official apps
   Custom App
   Customer-built apps
   App
   TreasureData CDP
   TreasureData-managed
   Platform
   DB
   App Runtime
   API
   API
   API
   API
   Overly simple view of TD platform
   Custom App
   Custom App
   

   View Slide

3. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
   Area I talk today
   3
   DB
   App
   App
   official apps
   Custom App
   Customer-built apps
   App
   TreasureData CDP
   TreasureData-managed
   Platform
   DB
   App Runtime
   API
   API
   API
   Background: Some customers wants TD
   to offer infra for their custom app
   API
   Aoki
   Mikurube
   Solution: Dedicated Engineering team
   provides nicely-crafted components
   Overly simple view of TD platform
   My talk Custom App
   Custom App
   Keisuke &
   okumin
   

   View Slide

4. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
   Me, TATSUNO “Taz” Yasuhiro
   4
   • Joined TreasureData on Jan. 2022. Previously on IT subsidiary
   in automotive enterprise, and two startups
   • An average web app developer. Passionate on AWS. No work
   experience with CDP or any data analysis
   • Loves Scala, but writes Ruby and TypeScript
   • Works remotely from Nagano, Japan
   • Huge progress on Pikmin Bloom thanks to business trip
   exoego
   exoego
   

   View Slide

5. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
   I am (mostly inactive)
   maintainers of some OSS
   5
   Scala Steward Fomantic-UI Vue-i18n
   rspec-openapi
   Supervised Japanese translation
   of Scala book (2021)
   

   View Slide

6. Background story:
   Not a small number of
   Enterprise customers expect
   TD to offer IT infrastructure for
   their custom app development
   

   View Slide

7. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
   TreasureData as Enterprise SaaS
   7
   Enterprise SMB
   Small & Mid-size Businesses
   “Wall Street”, Government, … “Main Street”
   Number of companies Small Many
   Buying power
   High
   - Lots of users
   - Big IT budget
   Low
   We, TreasureData
   

   View Slide

8. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
   Technical service at Enterprise SaaS
   8
   • Asking business and IT issues
   • Response to technical questionnaire
   • Demo, Proof-of-Concept
   • Solution proposal
   • Integration with customer’s system or 3rd party
   • Advanced setup of products
   • Technical training
   • Technical documentation: Manuals, API Doc, etc Technical
   Writer
   Sales Engineering
   Professional Service
   

   View Slide

9. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
   TD’s offering for system integration
   9
   • Integrating TD products with customer system and/or 3rd party
   • Various and mature ways. Such as SDK, Custom Scripts,
   Workflow, Connector, OSS (fluentd, embulk, …) and so on
   • Customer-built custom app integrated with TD
   • Possible by using SDK or other tech offerings
   • However, IT infra for custom app should be set up by
   customer themselves. Such as app server, DB server, etc
   

   View Slide

10. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Customer’s circumstance and expectation on TD
    10
    • Though IT insourcing is trend, employing talented IT pro is hard.
    Many enterprises fill the gaps by outsourcing IT work to vendors.
    Common in Japan and many countries.
    • Many want to leverage cloud technologies, such as AWS, to
    minimize initial IT infra cost and leading time, to reduce
    operational costs, for their small-start custom app. However,
    talented cloud professionals are rare.
    • Not a small number of our customers expect TreasureData to
    provider IT infra and support on their customs app dev.
    

    View Slide

11. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Customer
    - Win in business with
    TD as CDP
    11
    TreasureData
    - Filling the product gap without
    customization
    - Expanding sales opportunities
    through partner vendor
    - Revenue from providing IT infra
    Vendors
    - High-profit project with TD
    - Appealing TD dev experience
    to other customer
    https://imgflip.com/memegenerator/206999279/Epic-Handshake-3-arms
    Win-Win
    for all
    

    View Slide

12. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Motivation to provide IT infrastructure?
    12
    • First of all, strategic partnership
    • Expanding sales opportunities through partner vendors
    • Understanding customer demands through partnership,
    which might lead to up-sell or cross-sell
    • Incubating small app for a specific customer into the
    app that might sell to other customers (if TD owns IP)
    • Backends for new TD products in single-tenant fashion
    

    View Slide

13. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    But… AWS is not easy in reality
    13
    • Security incidents due to misconfiguration are pretty common
    • Citizen data breaches from S3 owned by 80+ US local governments (2021)
    • Customer data breaches from S3 owned by Twilio (2021)
    • Lots of AWS services (Route53, ELB, SG, VPN, RDS, etc) are required
    to build serious app. Normal app developers touch these only at initial
    stage and rarely touch again for months… hard to develop skill
    • TreasureData SE/PS team support customers technically. But their
    primary expertises are about TD products, not AWS.
    • So…
    

    View Slide

14. Our solution:
    Dedicated Engineering Team
    provides nicely-crafted
    AWS components… and API
    

    View Slide

15. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    IT Infra capability problems and our measures
    15
    1. AWS components
    nicely-crafted with IaC
    Arbitrary choice on each app
    projects, leading to unshared
    skills and poor impl
    7. Dedicated team
    Should not assume AWS
    expertise in parner app devs
    or customer, and should
    work OOTB for most cases
    Should not be just
    “subleasing AWS”...instead,
    should be value-added
    solution only from TD
    6. API to operate those
    AWS components
    2. Well-selected configs
    4. Appealing integration
    with TreasureData products
    Problems Our measures
    5. AWS cross-accounts
    3. Purpose-built Web UI
    

    View Slide

16. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    “Provisioner” service overview
    16
    TD Main Account “Provisioner” Account
    AWS CDK
    Other services/Products
    Core services
    Provisioner
    auth, etc
    integrate
    integrate
    provision
    Request provisioning
    through API
    TD Admins
    App Devs
    execute
    Well-crafted blocks, all single-tenant
    Multi-tenant basis
    

    View Slide

17. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Our “AWS+α” components for custom app
    development with TreasureData
    17
    • PostgreSQL server that can be queried from TD console
    • AWS in use: Aurora, VPC, Backup, KMS, CloudWatch
    Logs, IAM, SecretsManager, etc
    • Serverless function to extend ActiveDataLayer, to host simple
    web app, or other purpose.
    • AWS in use: Lambda, ALB, Route53, CloudWatch Logs、
    IAM, S3, SecretsManager, etc
    

    View Slide

18. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Meas.1. AWS components nicely-crafted with IaC
    18
    • Define component with AWS CDK (Cloud-Dev-Kit) in TypeScript
    • Statically-typed (safe and productive), fast try-and-error
    cycle
    • Can code AWS infra in familiar productive language, not in
    dumb language like JSON or YAML
    • Quality assurance with automated tests on every pull requests
    • Unit test to check validation logics on customer, combination
    of configs, etc
    • Snapshot test to check internal changes. Specially, to
    ensure everything is fine on frequent updates on CDK
    • Deployments and system tests in staging to ensure everything
    are configured as expected in real AWS
    

    View Slide

19. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Meas.2. Well-selected configs
    19
    • Though most parts of AWS are carefully configured by dedicated
    team, some configs are still needed to cover various use cases
    • Vanilla AWS is general-purpose so has loooots of config. Versatility
    sometimes confuses app devs and our customers
    • So we carefully handpicked configs for customers
    • Allow-list of IP addresses (commonly requested by enterprise)
    • Preferred maintenance time, etc..
    

    View Slide

20. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Meas.3. Purpose-built Web UI
    20
    • Web UI that can be logged-in with TD user account, no AWS user
    needed
    • Due to contract with AWS, we could not invite customer or
    outside app dev parter to log-in TD-owned AWS account
    • Track records of configuration changes: “who”, “when” and “what”
    • Note) Vanilla AWS offers similar via CloudFormation and
    CloudTrail
    • Log viewer. We collect logs from AWS using Kinesis Stream
    Lambda.
    

    View Slide

21. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Meas.4. Great integration with TD products
    21
    • Our AWS components should not be just “subleasing”
    • Tight integration with TD products, to help our custom app
    developers building useful app with TD
    • TreasureData console
    • ADL
    • Our billing system
    • and more
    

    View Slide

22. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Meas.5. AWS cross-account
    22
    • These AWS components are provisioned in separate
    “Provisioner” AWS account, not in the main AWS account for
    TD backends
    • Cross-account access from “Provisioner” to main or vice-versa
    are strictly limited in via cross-account policy
    • This sounds difficult, but is actually a lot easier and safer
    to control access in single AWS account. (IMHO)
    • This also mitigates resource quota per AWS account
    • E.g.) Number of ENI for Lambda in VPC
    

    View Slide

23. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Meas.6. API to operate these AWS components
    23
    • “Provisioner” also offers internal API. Other TD services can
    create/destroy “nicely-crafted AWS components” only by
    single API call.
    • This allows our products to provision the necessary infra, in
    single-tenant manner, when our customer clicks something.
    • E.g. ) Single-tenant, isolated and stable DB for Treasure
    Insights dashboard (planned)
    

    View Slide

24. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Meas.7. Dedicated team
    24
    • TreasureData SE/PS team support customers technically.
    Their primary expertises are about TD products, not AWS.
    • We need not only AWS infra experts but a feature team to
    build and operate the “Provisioner” system. Since we
    need to implement TD integration to our infra offerings, APIs,
    and Web UIs.
    • So we formed the dedicated team by people from TD platform
    engineering.
    

    View Slide

25. Thoughts after 2 year:
    Did the solution work
    nicely in reality?
    

    View Slide

26. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Nice?
    26
    • Yes if it is DB
    • Since app developers rarely change config once DB is
    initially setup, troubles or complains are rarely observed
    • No if it is application runtime (Lambda)
    • There are many pain points for app developers. They want
    to try-and-error frequently while developing, deployments
    in provisioner right now is very slow. Developer tools for
    Lambda is not supported. Log viewer is not slow and
    search is limited.
    

    View Slide

27. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Hard times for the “Provisioner” team
    27
    • CDK is great tool, but we often need extra cares on coding with CDK to workaround AWS’s enigmatic
    limitation.
    • E.g.) A Lambda function created in the imported VPN can not be deleted straightforwardly… We
    needed to add the hidden resource to the dependency tree of infra.
    • We struggled coordination with customers regarding maintenance, since Provisioner-managed infras are
    all single-tenant.
    • We introduced “Preferred maintenance window” mechanism and asked customer to configure it.
    Also, we implemented tools to streamline maintenance.
    • When we introduced Database Activity Stream feature to DB, it disrupted customer’s production DB… It
    was due to AWS Aurora’s defect. On some Aurora PG versions, if DAS is enabled, and if customer uses
    CURSOR SQL statement, DB dies… AWS disappointed us…
    • To prevent, we now run various SQL statements in system tests.
    

    View Slide

28. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Future work
    28
    • Makes it super-easy maintaining lots of single tenant AWS
    resources… otherwise the team spend much time on
    operations rather than engineering
    • Addresses many features requests from other TD app/platform
    teams more timely. We keep people waiting since the
    “dedicated” team is focusing on other related products
    • Improves developer experience on our serverless function
    which is not great
    

    View Slide

29. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Reverse Q&A (in case no one raise question…)
    29
    • How do you design/implement/operate a closed/internal
    infrastructure for your partner app developers??
    • How do you design/implement/operate provide modern IT infra
    products, such as Vercel and Serverless Cloud, that uses
    AWS Lambda and so on internally??
    

    View Slide

30. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    30
    Thank you
    

    View Slide