#TdTechTalk 2022-11-29 EMPOWERING APP DEV BY NICELY-CRAFTED HIGH-LEVEL AWS COMPONENTS TATSUNO “Taz” Yasuhiro, Insights and Data Systems group

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Area I talk today 2 DB App App official apps Custom App Customer-built apps App TreasureData CDP TreasureData-managed Platform DB App Runtime API API API API Overly simple view of TD platform Custom App Custom App

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Area I talk today 3 DB App App official apps Custom App Customer-built apps App TreasureData CDP TreasureData-managed Platform DB App Runtime API API API Background: Some customers wants TD to offer infra for their custom app API Aoki Mikurube Solution: Dedicated Engineering team provides nicely-crafted components Overly simple view of TD platform My talk Custom App Custom App Keisuke & okumin

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Me, TATSUNO “Taz” Yasuhiro 4 • Joined TreasureData on Jan. 2022. Previously on IT subsidiary in automotive enterprise, and two startups • An average web app developer. Passionate on AWS. No work experience with CDP or any data analysis • Loves Scala, but writes Ruby and TypeScript • Works remotely from Nagano, Japan • Huge progress on Pikmin Bloom thanks to business trip exoego exoego

© 2022 Treasure Data, Inc. Confidential—Internal Use Only I am (mostly inactive) maintainers of some OSS 5 Scala Steward Fomantic-UI Vue-i18n rspec-openapi Supervised Japanese translation of Scala book (2021)

Background story: Not a small number of Enterprise customers expect TD to offer IT infrastructure for their custom app development

© 2022 Treasure Data, Inc. Confidential—Internal Use Only TreasureData as Enterprise SaaS 7 Enterprise SMB Small & Mid-size Businesses “Wall Street”, Government, … “Main Street” Number of companies Small Many Buying power High - Lots of users - Big IT budget Low We, TreasureData

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Technical service at Enterprise SaaS 8 • Asking business and IT issues • Response to technical questionnaire • Demo, Proof-of-Concept • Solution proposal • Integration with customer’s system or 3rd party • Advanced setup of products • Technical training • Technical documentation: Manuals, API Doc, etc Technical Writer Sales Engineering Professional Service

© 2022 Treasure Data, Inc. Confidential—Internal Use Only TD’s offering for system integration 9 • Integrating TD products with customer system and/or 3rd party • Various and mature ways. Such as SDK, Custom Scripts, Workflow, Connector, OSS (fluentd, embulk, …) and so on • Customer-built custom app integrated with TD • Possible by using SDK or other tech offerings • However, IT infra for custom app should be set up by customer themselves. Such as app server, DB server, etc

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Customer’s circumstance and expectation on TD 10 • Though IT insourcing is trend, employing talented IT pro is hard. Many enterprises fill the gaps by outsourcing IT work to vendors. Common in Japan and many countries. • Many want to leverage cloud technologies, such as AWS, to minimize initial IT infra cost and leading time, to reduce operational costs, for their small-start custom app. However, talented cloud professionals are rare. • Not a small number of our customers expect TreasureData to provider IT infra and support on their customs app dev.

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Customer - Win in business with TD as CDP 11 TreasureData - Filling the product gap without customization - Expanding sales opportunities through partner vendor - Revenue from providing IT infra Vendors - High-profit project with TD - Appealing TD dev experience to other customer https://imgflip.com/memegenerator/206999279/Epic-Handshake-3-arms Win-Win for all

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Motivation to provide IT infrastructure? 12 • First of all, strategic partnership • Expanding sales opportunities through partner vendors • Understanding customer demands through partnership, which might lead to up-sell or cross-sell • Incubating small app for a specific customer into the app that might sell to other customers (if TD owns IP) • Backends for new TD products in single-tenant fashion

© 2022 Treasure Data, Inc. Confidential—Internal Use Only But… AWS is not easy in reality 13 • Security incidents due to misconfiguration are pretty common • Citizen data breaches from S3 owned by 80+ US local governments (2021) • Customer data breaches from S3 owned by Twilio (2021) • Lots of AWS services (Route53, ELB, SG, VPN, RDS, etc) are required to build serious app. Normal app developers touch these only at initial stage and rarely touch again for months… hard to develop skill • TreasureData SE/PS team support customers technically. But their primary expertises are about TD products, not AWS. • So…

Our solution: Dedicated Engineering Team provides nicely-crafted AWS components… and API

© 2022 Treasure Data, Inc. Confidential—Internal Use Only IT Infra capability problems and our measures 15 1. AWS components nicely-crafted with IaC Arbitrary choice on each app projects, leading to unshared skills and poor impl 7. Dedicated team Should not assume AWS expertise in parner app devs or customer, and should work OOTB for most cases Should not be just “subleasing AWS”...instead, should be value-added solution only from TD 6. API to operate those AWS components 2. Well-selected configs 4. Appealing integration with TreasureData products Problems Our measures 5. AWS cross-accounts 3. Purpose-built Web UI

© 2022 Treasure Data, Inc. Confidential—Internal Use Only “Provisioner” service overview 16 TD Main Account “Provisioner” Account AWS CDK Other services/Products Core services Provisioner auth, etc integrate integrate provision Request provisioning through API TD Admins App Devs execute Well-crafted blocks, all single-tenant Multi-tenant basis

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Our “AWS+α” components for custom app development with TreasureData 17 • PostgreSQL server that can be queried from TD console • AWS in use: Aurora, VPC, Backup, KMS, CloudWatch Logs, IAM, SecretsManager, etc • Serverless function to extend ActiveDataLayer, to host simple web app, or other purpose. • AWS in use: Lambda, ALB, Route53, CloudWatch Logs、 IAM, S3, SecretsManager, etc

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Meas.1. AWS components nicely-crafted with IaC 18 • Define component with AWS CDK (Cloud-Dev-Kit) in TypeScript • Statically-typed (safe and productive), fast try-and-error cycle • Can code AWS infra in familiar productive language, not in dumb language like JSON or YAML • Quality assurance with automated tests on every pull requests • Unit test to check validation logics on customer, combination of configs, etc • Snapshot test to check internal changes. Specially, to ensure everything is fine on frequent updates on CDK • Deployments and system tests in staging to ensure everything are configured as expected in real AWS

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Meas.2. Well-selected configs 19 • Though most parts of AWS are carefully configured by dedicated team, some configs are still needed to cover various use cases • Vanilla AWS is general-purpose so has loooots of config. Versatility sometimes confuses app devs and our customers • So we carefully handpicked configs for customers • Allow-list of IP addresses (commonly requested by enterprise) • Preferred maintenance time, etc..

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Meas.3. Purpose-built Web UI 20 • Web UI that can be logged-in with TD user account, no AWS user needed • Due to contract with AWS, we could not invite customer or outside app dev parter to log-in TD-owned AWS account • Track records of configuration changes: “who”, “when” and “what” • Note) Vanilla AWS offers similar via CloudFormation and CloudTrail • Log viewer. We collect logs from AWS using Kinesis Stream Lambda.

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Meas.4. Great integration with TD products 21 • Our AWS components should not be just “subleasing” • Tight integration with TD products, to help our custom app developers building useful app with TD • TreasureData console • ADL • Our billing system • and more

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Meas.5. AWS cross-account 22 • These AWS components are provisioned in separate “Provisioner” AWS account, not in the main AWS account for TD backends • Cross-account access from “Provisioner” to main or vice-versa are strictly limited in via cross-account policy • This sounds difficult, but is actually a lot easier and safer to control access in single AWS account. (IMHO) • This also mitigates resource quota per AWS account • E.g.) Number of ENI for Lambda in VPC

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Meas.6. API to operate these AWS components 23 • “Provisioner” also offers internal API. Other TD services can create/destroy “nicely-crafted AWS components” only by single API call. • This allows our products to provision the necessary infra, in single-tenant manner, when our customer clicks something. • E.g. ) Single-tenant, isolated and stable DB for Treasure Insights dashboard (planned)

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Meas.7. Dedicated team 24 • TreasureData SE/PS team support customers technically. Their primary expertises are about TD products, not AWS. • We need not only AWS infra experts but a feature team to build and operate the “Provisioner” system. Since we need to implement TD integration to our infra offerings, APIs, and Web UIs. • So we formed the dedicated team by people from TD platform engineering.

Thoughts after 2 year: Did the solution work nicely in reality?

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Nice? 26 • Yes if it is DB • Since app developers rarely change config once DB is initially setup, troubles or complains are rarely observed • No if it is application runtime (Lambda) • There are many pain points for app developers. They want to try-and-error frequently while developing, deployments in provisioner right now is very slow. Developer tools for Lambda is not supported. Log viewer is not slow and search is limited.

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Hard times for the “Provisioner” team 27 • CDK is great tool, but we often need extra cares on coding with CDK to workaround AWS’s enigmatic limitation. • E.g.) A Lambda function created in the imported VPN can not be deleted straightforwardly… We needed to add the hidden resource to the dependency tree of infra. • We struggled coordination with customers regarding maintenance, since Provisioner-managed infras are all single-tenant. • We introduced “Preferred maintenance window” mechanism and asked customer to configure it. Also, we implemented tools to streamline maintenance. • When we introduced Database Activity Stream feature to DB, it disrupted customer’s production DB… It was due to AWS Aurora’s defect. On some Aurora PG versions, if DAS is enabled, and if customer uses CURSOR SQL statement, DB dies… AWS disappointed us… • To prevent, we now run various SQL statements in system tests.

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Future work 28 • Makes it super-easy maintaining lots of single tenant AWS resources… otherwise the team spend much time on operations rather than engineering • Addresses many features requests from other TD app/platform teams more timely. We keep people waiting since the “dedicated” team is focusing on other related products • Improves developer experience on our serverless function which is not great

© 2022 Treasure Data, Inc. Confidential—Internal Use Only Reverse Q&A (in case no one raise question…) 29 • How do you design/implement/operate a closed/internal infrastructure for your partner app developers?? • How do you design/implement/operate provide modern IT infra products, such as Vercel and Serverless Cloud, that uses AWS Lambda and so on internally??

© 2022 Treasure Data, Inc. Confidential—Internal Use Only 30 Thank you