[Data เบ็ดเตล็ด x DataHungry] Data & AI Bootcamp 2024

As an instructor, I hosted a 2-day offline hands-on class focusing on Data and AI technologies. Participants will learn modern tools like Docker, Apache Airflow, Google Cloud Storage, BigQuery, Gemini Code Assist, Vertex AI Agent Builder, and LINE Messaging API. The objective of the Bootcamp is to build modern data pipelines, accelerate development with GenAI tools, and build AI agents to interact with your data seamlessly

Transcript

1. DATA & AI BOOTCAMP 2024
2. DATA & AI BOOTCAMP 2024 Add LINE Group
3. DATA & AI BOOTCAMP 2024 JOIN WIFI WIFI Name: Dee Dar Bar Guese “ตอเลย ฟรี ไมมี Password”
4. DATA & AI BOOTCAMP 2024 Course Outline Data Foundation • Modern Data Stack ◦ Docker สําหรับจัดการ Environment ◦ Apache Airflow สําหรับออเคสตรา Data Pipeline • Database System ◦ MongoDB (NoSQL) สําหรับขอมูล Semi-structured ◦ PostgreSQL สําหรับขอมูลแบบ Relational • Google Cloud Platform ◦ Cloud Storage เปน Data Lake ◦ BigQuery เปน Data Warehouse พรอมใชวิเคราะห
5. DATA & AI BOOTCAMP 2024 Interactive-Session / Active Listener ยกมือถาม ตอบโต้ เรียนรู้ไปด้วยกัน TODAY PRIZE🥇 WE ARE HERE TO SUPPORT 󰗞 FINISH TOGETHER 🚀
6. DATA & AI BOOTCAMP 2024 Git Repository + VS Code https://github.com/wuttichai-hung/data-ai-bootcamp ** หมายเหตุ - แนะนําใหใช Github Codespace ในการทํา Lab ทั้งหมด แตสําหรับทานที่ใช Macbook หรือมี VS Code และ Docker ในเครื่องอยูแลวสามารถที่จะใชงานใน Local Enviroment ของทานได 1. กด Fork Repository
7. DATA & AI BOOTCAMP 2024 Git Repository + VS Code 2. กด create Fork Repository เพื่อ นํา code ของ bootcamp มาไวใน Github ของเรา
8. DATA & AI BOOTCAMP 2024 Git Repository + VS Code 3. ปุม Sync Fork ใชเพื่อ check วา code ของเราที่ดึงมาเปน version ลาสุดที่ตรงกับทีมสอน หากเปน version ลาสุดจะแสดงเครื่องหมายถูก
9. DATA & AI BOOTCAMP 2024 Git Repository + VS Code ในกรณีที่ทีมสอนมีการ Update Code เรา สามารถกด Update branch เพื่อใหได code ใหมลาสุดได
10. DATA & AI BOOTCAMP 2024 Start in Codespace 4. กด code > Open in codespace
11. DATA & AI BOOTCAMP 2024 Start in Codespace Welcome to Codespace!! Now You’re Ready to Rock the Bootcamp
12. DATA & AI BOOTCAMP 2024 Understanding Basic Data Foundation
13. DATA & AI BOOTCAMP 2024 Why data is important to Business GDP = C+I+G+(X-M) C = Consumption การบริโภคของบริษัทและ ประชาชนทั่วไป I = Investment การลงทุนจากภาคเอกชนในการทํา กิจกรรมตางๆในระบบเศรษฐกิจ G = Government Spending คาใชจายของรัฐบาล/ การลงทุนภาครัฐ X - M = Export - Import ตัวเลขการสงออกลบดวย การนําเขาถึงจะเห็นอัตราการบริโภคสุดทายที่แทจริง
14. DATA & AI BOOTCAMP 2024 Why data is important to Business
15. DATA & AI BOOTCAMP 2024 Why data is important to Business https://www.deloitte.com/content/dam/assets-shared/legacy/docs/analysis/2022/dttl-analytics-analytics-advantage-report.pdf
16. DATA & AI BOOTCAMP 2024 Who oversees analytics initiative https://www.deloitte.com/content/dam/assets-shared/legacy/docs/analysis/2022/dttl-analytics-analytics-advantage-report.pdf
17. DATA & AI BOOTCAMP 2024 The 4 levels of data maturity how data mature is your business? https://www.edq.com/blog/data-maturity-how-mature-are-you/
18. DATA & AI BOOTCAMP 2024 Main Type of Data Source
19. DATA & AI BOOTCAMP 2024 Main Type of Data Source
20. DATA & AI BOOTCAMP 2024 Data Classification
21. DATA & AI BOOTCAMP 2024 Roles & Responsibility in Data Career • Software Engineer • System Engineer • Data Engineer • Data Analyst • Data Scientist • Analytics Engineer
22. DATA & AI BOOTCAMP 2024 Extract Transform Load (ETL)
23. DATA & AI BOOTCAMP 2024 Basic Python & SQL
24. DATA & AI BOOTCAMP 2024 Docker (Containerize an application)
25. DATA & AI BOOTCAMP 2024 Why Docker? • Isolation • Lightweight • Simplicity • Workflow • Community
26. DATA & AI BOOTCAMP 2024 Why Docker?
27. DATA & AI BOOTCAMP 2024 Docker vs. VM Use Docker if:Use Docker if: • You need lightweight, scalable solutions. • You are working with microservices or cloud-native applications. • Consistency across environments is crucial. Use Virtual Machines if: • You need to run multiple OS environments. • Applications require complete isolation. • Legacy applications are involved that demand dedicated OS resources. https://k21academy.com/docker-kubernetes/docker-vs-virtual-machine/
28. DATA & AI BOOTCAMP 2024 Postgres
29. DATA & AI BOOTCAMP 2024 Mongodb
30. DATA & AI BOOTCAMP 2024 Quiz Time!!
31. DATA & AI BOOTCAMP 2024 Apache Airflow
32. DATA & AI BOOTCAMP 2024 What is Apache Airflow ? Airflow is an open-source platform for developing, scheduling, and monitoring batch-oriented workflows create by Airbnb
33. DATA & AI BOOTCAMP 2024 What is Data Pipeline? Data pipeline is a means of moving data from one place (the source) to a destination (such as a data warehouse). Along the way, data is transformed and optimized, arriving in a state that can be analyzed and used to develop business insights.
34. DATA & AI BOOTCAMP 2024 Traditional Data Pipeline
35. DATA & AI BOOTCAMP 2024 Apache Airflow Features
36. DATA & AI BOOTCAMP 2024 Apache Airflow Features - Pure Python
37. DATA & AI BOOTCAMP 2024 Apache Airflow Features - Robust Integration
38. DATA & AI BOOTCAMP 2024 Apache Airflow Features - Useful UI
39. DATA & AI BOOTCAMP 2024 Apache Airflow Features - Open Source
40. DATA & AI BOOTCAMP 2024 Apache Airflow Features - Easy to Use
41. DATA & AI BOOTCAMP 2024 What is DAG ? D = Direct A = Acyclic G = Graph
42. DATA & AI BOOTCAMP 2024 Pipeline as DAG
43. DATA & AI BOOTCAMP 2024 Example Data pipeline
44. DATA & AI BOOTCAMP 2024 Apache Airflow Core Components
45. DATA & AI BOOTCAMP 2024 Apache Airflow Core Components
46. DATA & AI BOOTCAMP 2024 Airflow Concept to Code • DAG - the graphical representation of your data pipeline • Operator - describes a single task in your data pipeline • Task- an instance of operator task.
47. DATA & AI BOOTCAMP 2024 Dependencies between Tasks In Airflow, we commonly use the binary right shift operator (>>) to define the dependencies
48. DATA & AI BOOTCAMP 2024 Operator in Apache Airflow
49. DATA & AI BOOTCAMP 2024 Operator in Apache Airflow
50. DATA & AI BOOTCAMP 2024 Operator in Apache Airflow
51. DATA & AI BOOTCAMP 2024 Airflow Scheduling
52. DATA & AI BOOTCAMP 2024 Airflow Task State
53. DATA & AI BOOTCAMP 2024 Airflow XCom
54. DATA & AI BOOTCAMP 2024 • Apache Airflow https://airflow.apache.org/ • Apache Airflow Best Practices https://airflow.readthedocs.io/en/stable/best-practices.html • Apache Airflow Guides https://www.astronomer.io/guides/ • Apache Airflow (YouTube Channel) https://www.youtube.com/channel/UCSXwxpWZQ7XZ1WL3wqevChA • Data Council (YouTube Channel) https://www.youtube.com/c/DataCouncil/ • Awesome Apache Airflow https://github.com/jghoman/awesome-apache-airflow Study More
55. DATA & AI BOOTCAMP 2024 Google Cloud Platform
56. DATA & AI BOOTCAMP 2024 Google Cloud Platform https://console.cloud.google.com/home/dashboard?project=dataaibootcamp
57. DATA & AI BOOTCAMP 2024 ขอตกลงรวมกันในการใช GCP Project • สามารถใช Project สวนตัวได หรือ ใชรวมกันก็ได • Project นี้จะถูกลบ หลัง Class จบ • Project นี้ถูกสรางมาเพื่อใหชวยอํานวยความสะดวกใหนักเรียนไมตองเสียเวลาสรางProject เอง • ขอความกรุณา ไมใช Project นี้สําหรับงานอื่นนอก เนื้อหาการสอนดวยคะ • ขอความกรุณา ไมสงตอ Service Account ไฟลใหทานอื่นที่ไมใชนักเรียนนะคะ 󰢚 • Naming for BigQuery DataSet: dataai_NAME_YYYY ◦ เชน dataai_beat_1991 • Naming for GCS Bucket: data-ai-NAME-YYYY • และหากตองสราง service ใดเพิ่มเติม ◦ ให ใช data-ai-NAME-YYYY เปน Prefix
58. DATA & AI BOOTCAMP 2024 Google Cloud Storage
59. DATA & AI BOOTCAMP 2024 Data Lake A Data Lake is a centralized repository that allows organizations to store structured, semi-structured, and unstructured data at any scale. Unlike traditional databases or data warehouses, a data lake can store raw data in its native format until it's needed. This makes it highly flexible for diverse use cases, including data analytics, machine learning, and big data processing.
60. DATA & AI BOOTCAMP 2024 Google Cloud Storage is solution for Data Lake
61. DATA & AI BOOTCAMP 2024 Storage Classes
62. DATA & AI BOOTCAMP 2024 Use case for Google Cloud Storage (GCS) • Big Data Analytics: Process and analyze large datasets. • Machine Learning: Provide raw data for training algorithms. • Data Archiving: Retain historical data for regulatory compliance or future analysis. • Data Integration: Serve as a single source of truth for disparate data sources.
63. DATA & AI BOOTCAMP 2024 Google BigQuery
64. DATA & AI BOOTCAMP 2024 Warehouse A large building where raw materials or manufactured goods may be stored before their export or distribution for sale.
65. DATA & AI BOOTCAMP 2024 Data Warehouse In computing, a data warehouse (DW or DWH), also known as an enterprise data warehouse (EDW), is a system used for reporting and data analysis, and is considered a core component of business intelligence throughout the enterprise. https://en.wikipedia.org/wiki/Data_warehouse
66. DATA & AI BOOTCAMP 2024 Data Warehouse Vs. Data Lake
67. DATA & AI BOOTCAMP 2024 Data Warehouse Vs. Operation Database
68. DATA & AI BOOTCAMP 2024 BigQuery is Google’s Data Warehouse Solution
69. DATA & AI BOOTCAMP 2024 BigQuery BigQuery is a serverless, highly-scalable, and cost-effective cloud data warehouse with an in-memory BI Engine and machine learning built in. ☑ Real-time analytics ☑ Standard SQL ☑ Big data ecosystem integration ☑ Federated query and logical data warehousing ☑ Storage (Colossus) and compute (Dremel) separation ☑ Geospatial data types and functions
70. DATA & AI BOOTCAMP 2024 BigQuery
71. DATA & AI BOOTCAMP 2024 Querying in BigQuery
72. DATA & AI BOOTCAMP 2024 Quiz Time!!
73. DATA & AI BOOTCAMP 2024
74. DATA & AI BOOTCAMP 2024 Course Outline Day 2: AI-Enhanced Data Journey • Integrate with Data User ◦ LINE Integration สําหรับแจงเตือนและติดตาม สถานะ ◦ Chat with Data ผาน Vertex AI Agent Generate Data Transform Data Analyse Data Utilize Data Software Engineer Data Engineer Data Analyst/ Analytic Engineer AI Engineer
75. DATA & AI BOOTCAMP 2024 Agenda 1 Line Official Account 2 Create LINE Official Account 3 Workshop LINE Notify Message 4 Workshop : Create LINE Webhook with Python SDK + Cloud Function 3 Vertex AI Agent Builder - RAG 4 Workshop : Create Chat with Data Agent 5 Vertex AI Search + Gemini Image Understanding 6 Workshop : Create Search Agent
76. DATA & AI BOOTCAMP 2024 LINE Chatbot
77. DATA & AI BOOTCAMP 2024 Why LINE Messaging API
78. DATA & AI BOOTCAMP 2024 LINE Official Account
79. DATA & AI BOOTCAMP 2024 LINE Message Events
80. DATA & AI BOOTCAMP 2024 LINE Message types • Text message • Sticker message • Image message • Video message • Audio message • Location message • Imagemap message • Template message • Flex Message
81. DATA & AI BOOTCAMP 2024 LINE Webhook
82. DATA & AI BOOTCAMP 2024 LINE Webhook
83. DATA & AI BOOTCAMP 2024 Validate LINE Signature
84. DATA & AI BOOTCAMP 2024 LINE Webhook
85. DATA & AI BOOTCAMP 2024 Flex Message https://developers.line.biz/flex-simulator/
86. DATA & AI BOOTCAMP 2024 Flex Message
87. DATA & AI BOOTCAMP 2024 Go to Workshop
88. DATA & AI BOOTCAMP 2024 Vertex AI Agent Builder
89. DATA & AI BOOTCAMP 2024 Deterministic vs. Generative
90. DATA & AI BOOTCAMP 2024 Flow-based Agent
91. DATA & AI BOOTCAMP 2024 Flow-based Agent
92. DATA & AI BOOTCAMP 2024 Generative AI
93. DATA & AI BOOTCAMP 2024 Flow-based (Deterministic) vs. Generative-based Agent
94. DATA & AI BOOTCAMP 2024 Retrieval Augmented Generation
95. DATA & AI BOOTCAMP 2024 Vertex AI Agent Builder
96. DATA & AI BOOTCAMP 2024 Conversation Agent Use-case
97. DATA & AI BOOTCAMP 2024 Vertex AI Agent Builder Step
98. DATA & AI BOOTCAMP 2024 Create Conversation Agent
99. DATA & AI BOOTCAMP 2024 Create Data Store
100. DATA & AI BOOTCAMP 2024 Create Data Store
101. DATA & AI BOOTCAMP 2024 Tools
102. DATA & AI BOOTCAMP 2024 Define Agent Goal
103. DATA & AI BOOTCAMP 2024 Integrate with LINE
104. DATA & AI BOOTCAMP 2024 Integrate with LINE
105. DATA & AI BOOTCAMP 2024 Vertex AI Search
106. DATA & AI BOOTCAMP 2024 Vertex AI Search
107. DATA & AI BOOTCAMP 2024 Vertex AI Search 1. Create Search Agent 2. Create Data Store 3. Import Documents 4. Select Large Language Model (LLM) for Search Result Summarization 5. Search Result in both Widget and API
108. DATA & AI BOOTCAMP 2024 Create Vertex AI Search
109. DATA & AI BOOTCAMP 2024 Vertex AI Search
110. DATA & AI BOOTCAMP 2024 Handle Text Message
111. DATA & AI BOOTCAMP 2024 Vertex AI Search - Integration
112. DATA & AI BOOTCAMP 2024 Vertex AI Search
113. DATA & AI BOOTCAMP 2024 Flex Message for Search Results
114. DATA & AI BOOTCAMP 2024 Gemini Image Understanding
115. DATA & AI BOOTCAMP 2024 Image Search Solution
116. DATA & AI BOOTCAMP 2024 Image Search Result
117. DATA & AI BOOTCAMP 2024 Summary
118. DATA & AI BOOTCAMP 2024 https://binariks.com/blog/how-big-data-and-ai-work-together/ Big Data & AI Integration
119. DATA & AI BOOTCAMP 2024 Quiz Time
120. DATA & AI BOOTCAMP 2024
121. DATA & AI BOOTCAMP 2024 Course Outline Day 2: AI-Enhanced Data Journey • AI-Powered Development ◦ ใช Gemini Code Assist เพิ่มความเร็วในการ สราง Pipeline ◦ BigQuery Data Canvas วิเคราะหขอมูลดวย AI • Integrate with Data User ◦ LINE Integration สําหรับแจงเตือนและติดตาม สถานะ ◦ Chat with Data ผาน Vertex AI Agent
122. DATA & AI BOOTCAMP 2024 Agenda - Github Folder 11-12 1 What is Generative AI 2 How AI code-assisted boost productivity 3 Demo : AI-assisted Dev workflow 4 How Gemini in BigQuery help data analysis 5 Demo : Gemini in BigQuery 6 Summary
123. DATA & AI BOOTCAMP 2024 Gemini Code Assist for Data Engineering Task
124. DATA & AI BOOTCAMP 2024 Discriminative vs. Generative AI
125. DATA & AI BOOTCAMP 2024 Generative AI based on Text Input Data
126. DATA & AI BOOTCAMP 2024 Generative AI & Use-Case Overview
127. DATA & AI BOOTCAMP 2024 AI across the development workflow
128. DATA & AI BOOTCAMP 2024 Example of Today’s Tech Landscape
129. DATA & AI BOOTCAMP 2024 Traditional Bugging Tactics
130. DATA & AI BOOTCAMP 2024 Future Bugging Tactics
131. DATA & AI BOOTCAMP 2024 Hallucination - Challenges in Large Language Model
132. DATA & AI BOOTCAMP 2024 Code Assistants Landscape
133. DATA & AI BOOTCAMP 2024 Why you should consider using an AI coding assistant ?
134. DATA & AI BOOTCAMP 2024 Available for multiple IDEs and developer surfaces
135. DATA & AI BOOTCAMP 2024 Available for multiple IDEs and developer surfaces
136. DATA & AI BOOTCAMP 2024 Gemini Code Assist
137. DATA & AI BOOTCAMP 2024 Gemini Code Assist in Cloud Shell and VS Code Cloud Shell Editor VS Code
138. DATA & AI BOOTCAMP 2024 Enable Gemini for Google Cloud
139. DATA & AI BOOTCAMP 2024 To Start: Add Gemini Code Assist Extension in VSCode
140. DATA & AI BOOTCAMP 2024 Interact with LLM - Prompting Technique
141. DATA & AI BOOTCAMP 2024 Interact with LLM - Prompting Technique
142. DATA & AI BOOTCAMP 2024 Let’s build data pipeline with GenAI https://www.coingecko.com/
143. DATA & AI BOOTCAMP 2024 Gemini in BigQuery
144. DATA & AI BOOTCAMP 2024 Gemini in BigQuery
145. DATA & AI BOOTCAMP 2024 BigQuery Data Canvas
146. DATA & AI BOOTCAMP 2024 Understand data with Gemini in BigQuery Bigquery Public Dataset : https://console.cloud.google.com/marketplace/product/bigquery-public-data/thelook-ecommerce
147. DATA & AI BOOTCAMP 2024 Go to Workshop
148. DATA & AI BOOTCAMP 2024 BigQuery ML
149. DATA & AI BOOTCAMP 2024 BigQuery ML
150. DATA & AI BOOTCAMP 2024 Summary
151. DATA & AI BOOTCAMP 2024 AI-assisted: best practices
152. DATA & AI BOOTCAMP 2024 How Coding Assistance may shape your Dev Workflow ?
153. DATA & AI BOOTCAMP 2024 Balancing Between Generative AI Code Assist Benefit vs.Challenge
154. DATA & AI BOOTCAMP 2024 Human + AI prompting Interaction
155. DATA & AI BOOTCAMP 2024 Gemini’s Utility
156. DATA & AI BOOTCAMP 2024 Quiz Time!!