$30 off During Our Annual Pro Sale. View Details »

Empowering App Dev by Nicely-Crafted High-Level AWS Components

Empowering App Dev by Nicely-Crafted High-Level AWS Components

English version of a presentation slide for TreasureData Tech Talk 2022
https://techplay.jp/event/879660

TATSUNO Yasuhiro

November 29, 2022
Tweet

More Decks by TATSUNO Yasuhiro

Other Decks in Technology

Transcript

  1. #TdTechTalk 2022-11-29
    EMPOWERING APP DEV
    BY NICELY-CRAFTED
    HIGH-LEVEL AWS
    COMPONENTS
    TATSUNO “Taz” Yasuhiro, Insights and Data Systems group

    View Slide

  2. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Area I talk today
    2
    DB
    App
    App
    official apps
    Custom App
    Customer-built apps
    App
    TreasureData CDP
    TreasureData-managed
    Platform
    DB
    App Runtime
    API
    API
    API
    API
    Overly simple view of TD platform
    Custom App
    Custom App

    View Slide

  3. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Area I talk today
    3
    DB
    App
    App
    official apps
    Custom App
    Customer-built apps
    App
    TreasureData CDP
    TreasureData-managed
    Platform
    DB
    App Runtime
    API
    API
    API
    Background: Some customers wants TD
    to offer infra for their custom app
    API
    Aoki
    Mikurube
    Solution: Dedicated Engineering team
    provides nicely-crafted components
    Overly simple view of TD platform
    My talk Custom App
    Custom App
    Keisuke &
    okumin

    View Slide

  4. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Me, TATSUNO “Taz” Yasuhiro
    4
    • Joined TreasureData on Jan. 2022. Previously on IT subsidiary
    in automotive enterprise, and two startups
    • An average web app developer. Passionate on AWS. No work
    experience with CDP or any data analysis
    • Loves Scala, but writes Ruby and TypeScript
    • Works remotely from Nagano, Japan
    • Huge progress on Pikmin Bloom thanks to business trip
    exoego
    exoego

    View Slide

  5. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    I am (mostly inactive)
    maintainers of some OSS
    5
    Scala Steward Fomantic-UI Vue-i18n
    rspec-openapi
    Supervised Japanese translation
    of Scala book (2021)

    View Slide

  6. Background story:
    Not a small number of
    Enterprise customers expect
    TD to offer IT infrastructure for
    their custom app development

    View Slide

  7. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    TreasureData as Enterprise SaaS
    7
    Enterprise SMB
    Small & Mid-size Businesses
    “Wall Street”, Government, … “Main Street”
    Number of companies Small Many
    Buying power
    High
    - Lots of users
    - Big IT budget
    Low
    We, TreasureData

    View Slide

  8. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Technical service at Enterprise SaaS
    8
    • Asking business and IT issues
    • Response to technical questionnaire
    • Demo, Proof-of-Concept
    • Solution proposal
    • Integration with customer’s system or 3rd party
    • Advanced setup of products
    • Technical training
    • Technical documentation: Manuals, API Doc, etc Technical
    Writer
    Sales Engineering
    Professional Service

    View Slide

  9. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    TD’s offering for system integration
    9
    • Integrating TD products with customer system and/or 3rd party
    • Various and mature ways. Such as SDK, Custom Scripts,
    Workflow, Connector, OSS (fluentd, embulk, …) and so on
    • Customer-built custom app integrated with TD
    • Possible by using SDK or other tech offerings
    • However, IT infra for custom app should be set up by
    customer themselves. Such as app server, DB server, etc

    View Slide

  10. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Customer’s circumstance and expectation on TD
    10
    • Though IT insourcing is trend, employing talented IT pro is hard.
    Many enterprises fill the gaps by outsourcing IT work to vendors.
    Common in Japan and many countries.
    • Many want to leverage cloud technologies, such as AWS, to
    minimize initial IT infra cost and leading time, to reduce
    operational costs, for their small-start custom app. However,
    talented cloud professionals are rare.
    • Not a small number of our customers expect TreasureData to
    provider IT infra and support on their customs app dev.

    View Slide

  11. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Customer
    - Win in business with
    TD as CDP
    11
    TreasureData
    - Filling the product gap without
    customization
    - Expanding sales opportunities
    through partner vendor
    - Revenue from providing IT infra
    Vendors
    - High-profit project with TD
    - Appealing TD dev experience
    to other customer
    https://imgflip.com/memegenerator/206999279/Epic-Handshake-3-arms
    Win-Win
    for all

    View Slide

  12. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Motivation to provide IT infrastructure?
    12
    • First of all, strategic partnership
    • Expanding sales opportunities through partner vendors
    • Understanding customer demands through partnership,
    which might lead to up-sell or cross-sell
    • Incubating small app for a specific customer into the
    app that might sell to other customers (if TD owns IP)
    • Backends for new TD products in single-tenant fashion

    View Slide

  13. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    But… AWS is not easy in reality
    13
    • Security incidents due to misconfiguration are pretty common
    • Citizen data breaches from S3 owned by 80+ US local governments (2021)
    • Customer data breaches from S3 owned by Twilio (2021)
    • Lots of AWS services (Route53, ELB, SG, VPN, RDS, etc) are required
    to build serious app. Normal app developers touch these only at initial
    stage and rarely touch again for months… hard to develop skill
    • TreasureData SE/PS team support customers technically. But their
    primary expertises are about TD products, not AWS.
    • So…

    View Slide

  14. Our solution:
    Dedicated Engineering Team
    provides nicely-crafted
    AWS components… and API

    View Slide

  15. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    IT Infra capability problems and our measures
    15
    1. AWS components
    nicely-crafted with IaC
    Arbitrary choice on each app
    projects, leading to unshared
    skills and poor impl
    7. Dedicated team
    Should not assume AWS
    expertise in parner app devs
    or customer, and should
    work OOTB for most cases
    Should not be just
    “subleasing AWS”...instead,
    should be value-added
    solution only from TD
    6. API to operate those
    AWS components
    2. Well-selected configs
    4. Appealing integration
    with TreasureData products
    Problems Our measures
    5. AWS cross-accounts
    3. Purpose-built Web UI

    View Slide

  16. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    “Provisioner” service overview
    16
    TD Main Account “Provisioner” Account
    AWS CDK
    Other services/Products
    Core services
    Provisioner
    auth, etc
    integrate
    integrate
    provision
    Request provisioning
    through API
    TD Admins
    App Devs
    execute
    Well-crafted blocks, all single-tenant
    Multi-tenant basis

    View Slide

  17. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Our “AWS+α” components for custom app
    development with TreasureData
    17
    • PostgreSQL server that can be queried from TD console
    • AWS in use: Aurora, VPC, Backup, KMS, CloudWatch
    Logs, IAM, SecretsManager, etc
    • Serverless function to extend ActiveDataLayer, to host simple
    web app, or other purpose.
    • AWS in use: Lambda, ALB, Route53, CloudWatch Logs、
    IAM, S3, SecretsManager, etc

    View Slide

  18. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Meas.1. AWS components nicely-crafted with IaC
    18
    • Define component with AWS CDK (Cloud-Dev-Kit) in TypeScript
    • Statically-typed (safe and productive), fast try-and-error
    cycle
    • Can code AWS infra in familiar productive language, not in
    dumb language like JSON or YAML
    • Quality assurance with automated tests on every pull requests
    • Unit test to check validation logics on customer, combination
    of configs, etc
    • Snapshot test to check internal changes. Specially, to
    ensure everything is fine on frequent updates on CDK
    • Deployments and system tests in staging to ensure everything
    are configured as expected in real AWS

    View Slide

  19. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Meas.2. Well-selected configs
    19
    • Though most parts of AWS are carefully configured by dedicated
    team, some configs are still needed to cover various use cases
    • Vanilla AWS is general-purpose so has loooots of config. Versatility
    sometimes confuses app devs and our customers
    • So we carefully handpicked configs for customers
    • Allow-list of IP addresses (commonly requested by enterprise)
    • Preferred maintenance time, etc..

    View Slide

  20. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Meas.3. Purpose-built Web UI
    20
    • Web UI that can be logged-in with TD user account, no AWS user
    needed
    • Due to contract with AWS, we could not invite customer or
    outside app dev parter to log-in TD-owned AWS account
    • Track records of configuration changes: “who”, “when” and “what”
    • Note) Vanilla AWS offers similar via CloudFormation and
    CloudTrail
    • Log viewer. We collect logs from AWS using Kinesis Stream
    Lambda.

    View Slide

  21. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Meas.4. Great integration with TD products
    21
    • Our AWS components should not be just “subleasing”
    • Tight integration with TD products, to help our custom app
    developers building useful app with TD
    • TreasureData console
    • ADL
    • Our billing system
    • and more

    View Slide

  22. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Meas.5. AWS cross-account
    22
    • These AWS components are provisioned in separate
    “Provisioner” AWS account, not in the main AWS account for
    TD backends
    • Cross-account access from “Provisioner” to main or vice-versa
    are strictly limited in via cross-account policy
    • This sounds difficult, but is actually a lot easier and safer
    to control access in single AWS account. (IMHO)
    • This also mitigates resource quota per AWS account
    • E.g.) Number of ENI for Lambda in VPC

    View Slide

  23. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Meas.6. API to operate these AWS components
    23
    • “Provisioner” also offers internal API. Other TD services can
    create/destroy “nicely-crafted AWS components” only by
    single API call.
    • This allows our products to provision the necessary infra, in
    single-tenant manner, when our customer clicks something.
    • E.g. ) Single-tenant, isolated and stable DB for Treasure
    Insights dashboard (planned)

    View Slide

  24. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Meas.7. Dedicated team
    24
    • TreasureData SE/PS team support customers technically.
    Their primary expertises are about TD products, not AWS.
    • We need not only AWS infra experts but a feature team to
    build and operate the “Provisioner” system. Since we
    need to implement TD integration to our infra offerings, APIs,
    and Web UIs.
    • So we formed the dedicated team by people from TD platform
    engineering.

    View Slide

  25. Thoughts after 2 year:
    Did the solution work
    nicely in reality?

    View Slide

  26. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Nice?
    26
    • Yes if it is DB
    • Since app developers rarely change config once DB is
    initially setup, troubles or complains are rarely observed
    • No if it is application runtime (Lambda)
    • There are many pain points for app developers. They want
    to try-and-error frequently while developing, deployments
    in provisioner right now is very slow. Developer tools for
    Lambda is not supported. Log viewer is not slow and
    search is limited.

    View Slide

  27. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Hard times for the “Provisioner” team
    27
    • CDK is great tool, but we often need extra cares on coding with CDK to workaround AWS’s enigmatic
    limitation.
    • E.g.) A Lambda function created in the imported VPN can not be deleted straightforwardly… We
    needed to add the hidden resource to the dependency tree of infra.
    • We struggled coordination with customers regarding maintenance, since Provisioner-managed infras are
    all single-tenant.
    • We introduced “Preferred maintenance window” mechanism and asked customer to configure it.
    Also, we implemented tools to streamline maintenance.
    • When we introduced Database Activity Stream feature to DB, it disrupted customer’s production DB… It
    was due to AWS Aurora’s defect. On some Aurora PG versions, if DAS is enabled, and if customer uses
    CURSOR SQL statement, DB dies… AWS disappointed us…
    • To prevent, we now run various SQL statements in system tests.

    View Slide

  28. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Future work
    28
    • Makes it super-easy maintaining lots of single tenant AWS
    resources… otherwise the team spend much time on
    operations rather than engineering
    • Addresses many features requests from other TD app/platform
    teams more timely. We keep people waiting since the
    “dedicated” team is focusing on other related products
    • Improves developer experience on our serverless function
    which is not great

    View Slide

  29. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    Reverse Q&A (in case no one raise question…)
    29
    • How do you design/implement/operate a closed/internal
    infrastructure for your partner app developers??
    • How do you design/implement/operate provide modern IT infra
    products, such as Vercel and Serverless Cloud, that uses
    AWS Lambda and so on internally??

    View Slide

  30. © 2022 Treasure Data, Inc. Confidential—Internal Use Only
    30
    Thank you

    View Slide