Excel + Python + Deep Learning = ! Jeongkyu Shin

About me Lablup Inc. : Make AI Accessible Maintainer / Open-source machine learning cluster platform: Backend.AI Google Developers Experts ML / DL GDE (Context retrieval) https://github.com/inureyes/

And online again T_T

Today’s Agenda • Left behind o Why my life is same as before AI wave? o Why Excel + Deep Learning? • Make my Excel supercharged with AI o Excel + TypeScript + Python + TensorFlow o Architecture overview: how I created • “Trainable excel functions” o Simple examples • Closing o Personal example: how it becomes the painkiller for my ‘real’ life. [1] https://openwho.org/channels/covid-19-national-languages

Left Behind Excel + Python + Deep Learning = !

Rise of AI Era • AI Wave: DEEP LEARNING o ImageNet competition (2012) o AlphaGo (2016) o Attention / Transformer (2017 / 2018) o XLNet / T5 (2019) o AlphaStar (2019) o GPT-3 (2020) • Boosted applications o Image segmentation / categorization o NLU o Self-driving o Robotics [1] LaMDA / Google I/O 2021 Keynote

Moonshot • Reaching the stars o Scanning the universe[1] o Filtering gravity lens noise[2] • Combat with COVID-19 o Analyzing coronavirus details[3] o Finding cure candidates[4] • Solving the nature riddles o Protein-folding prediction[5] https://www.flaticon.com/free-icon/shuttle_2285537 https://www.flaticon.com/free-icon/moon-phase_2909605 https://www.flaticon.com/free-icon/explosion_978623 [1] https://iopscience.iop.org/article/10.3847/1538-4357/abd62b [2] https://academic.oup.com/mnras/article-abstract/504/2/1825/6219095 [3] https://link.springer.com/article/10.1007/s10796-021-10131-x [4] https://www.frontiersin.org/articles/10.3389/frai.2020.00065/full [5] https://www.nature.com/articles/s41586-021-03819-2

Shades under the moon • Privacy o Recognizing personal identities[1] o Surveillance to unspecified citizens • War tool o Military programs[2,3] https://www.bbc.com/news/business-56195288 https://www.popularmechanics.com/technology/robots/a29610393/robot-soldier-boston-dynamics/ [1] https://www.nytimes.com/2019/04/14/technology/china-surveillance-artificial-intelligence-racial-profiling.html [2] https://ieeexplore.ieee.org/document/9384519 [3] https://sgp.fas.org/crs/natsec/R45178.pdf

Why my life is same as before AI wave? • Strongness and weakness of Deep Learning models o Work: Categorization / detection o Data: Image / Text data • And my life is full of… o Excel / HWP o Tedious repetitive works o And more • Characteristics of my life works o Based on specific tools o Have super-personalized detail (works / data) https://www.flaticon.com/free-icon/moon-phase_2909605 https://oldpc.tistory.com/71 https://www.my-internet-explorer.com/ie6/ https://tutorialexcel.com/versiones-de-excel/excel-95/

Why Excel + Deep Learning? • Data = Excel for daily work o Half of the most tedious tasks are happening on the Excel sheet o (Another half: Active-X + IE11) • So hard to be automated with ‘numerical functions’ o Many fields are not “numerical” https://www.flaticon.com/free-icon/risk_2145705

Make my Excel Supercharged with AI Excel + Python + Deep Learning = !

Excel + ?: Old ways • Traditional methods o Xlwings[1] (https://www.xlwings.org) o Openpyxl[2] (https://foss.heptapod.net/openpyxl/openpyxl) • Limitations: COM wrapper / depends on VBA o Works only on Windows o Limited real-time binding [1] https://www.xlwings.org [2] https://openpyxl.readthedocs.io/en/stable/ https://www.flaticon.com/free-icon/sweet_2581014

Excel + ?: Modern way • 2015 o Microsoft announces JavaScript API for Office[1] • 2018 o Microsoft announces Python support for script language[2] § Not implemented / Revamping Microsoft Office for Web § Transitioning to Web Standard o JavaScript-based Excel CustomFunction announced[3] • 2020 o JavaScript as an official script language[4] [1] https://developer.microsoft.com/en-us/office/blogs/connect-2015-office-extensibility-news/ [2] https://developers.slashdot.org/story/17/12/15/1133217/microsoft-considers-adding-python-as-an-official-scripting-language-in-excel [3] https://docs.microsoft.com/en-us/office/dev/add-ins/excel/custom-functions-overview [4] https://github.com/OfficeDev/Excel-Custom-Functions https://www.flaticon.com/free-icon/back-to-the-future_86503

Docker Building blocks for today • Microsoft Excel 365 (with 2021~ versions) • TypeScript o To compile Excel CustomFunction with JavaScript SDK • Docker (for windows / macOS) o To run sandboxed deep learning runtime • Backend.AI as glue / runtime framework o Glue between TypeScript and Deep learning container o JavaScript and Python SDK counterpart exists • You may choose one among other glue projects! o TF.js without Python/Docker o Anaconda with your handmade TypeScript SDK https://www.flaticon.com/premium-icon/blocks_1273426 https://www.flaticon.com/free-icon/brick_3813701 Excel

Excel + TypeScript + Python • Excel o Custom Functions (2020)[1] o Full JavaScript / TypeScript support o Limitation: Browser compatibility issues [2] • TypeScript o Binding CustomFunctionsRuntimeAPI [3] o Will use Backend.AI TypeScript SDK [4] to communicate Excel JavaScript fn. with Python container • Python o Local cluster / VM or baremetal o Docker for containerized environment [5] [1] https://docs.microsoft.com/en-us/office/dev/add-ins/excel/custom-functions-overview [2] https://docs.microsoft.com/en-us/office/dev/add-ins/develop/application-specific-api-model [3] https://docs.microsoft.com/en-us/office/dev/add-ins/excel/custom-functions-requirement-sets [4] https://github.com/lablup/backend.ai-client-js [5] https://hub.docker.com/u/lablup

Excel + TypeScript + Python + TensorFlow • TensorFlow o TensorFlow 2.5 Environment runtime[1] o Training § TensorFlow 2.5 (CPU / OneAPI) o Inference § TensorFlow Serving • API connector o GraphQL-based API o Connect with JavaScript SDK § Converted from TypeScript+Node.js 14 to Pure ESM [1] docker pull lablup/python-tensorflow:2.5-py38-cuda11.3

Architecture overview

“Trainable excel functions” Excel + Python + Deep Learning = !

Making custom functions • Two functions for one model o No general model functions: it will not look like ‘Excel function’ o One for training, one for inferencing • EXAMPLETRAIN(data_to_train) o To training & saving EXAMPLE model o e.g. IRIS_MODEL_TRAIN(data) • EXAMPLE(data_to_inference) o To inferencing with the EXAMPLE model o e.g. IRIS_MODEL(column) • Characteristics o Training time o Training data preservation o Keeping checkpoint to additional training https://www.flaticon.com/premium-icon/customization_1066631

So we will make • BTF o Bidirectional TensorFlow workflow example for Excel CustomFunction • Why not BTFECF? o Rhyme; o Not BTS but TensorFlow

Preparing development environment • Dev. Environment preparation o Install yeoman generator as global npm / yarn package § [NPM] npm install -g yo generator-office § [Yarn] yarn global add yo generator-office o Initialize Microsoft JavaScript Add-in repository § yo office § Create “TypeScript” type, “EXAMPLETRAIN” project o Install TypeScript SDK § npm install backend.ai-client o Build / sideload test § [Build] npm run build § [DevServer] npm run dev-server § [Sideloading] npm start https://www.flaticon.com/premium-icon/customization_1799807

Platform dependency • Excel (App) o Windows: EdgeHTML (Legacy) / Blink (Modern) o macOS: WebKit2 • Excel (Web) o Depends on the user browser • Supports o JavaScript ES5 or above o Transpilers help you

Preparing development environment

Preparing development environment npm run dev-server npm start:desktop Tip: CHANGE localhost to 127.0.0.1 in manifest.yml

Preparing development environment You will have task panel (template) on the ribbon.

Creating CustomFunction test logic • Modify manifest.yml o Change CONTOSO to your namespace. e.g. BTF o CHANGE localhost to 127.0.0.1 • Check src/functions/functions.ts o We will add functions here! o Default example functions: add, clock, currentTime, increment… https://www.flaticon.com/free-icon/example_5360938

Excel CustomFunction test /** * Adds two numbers. * @customfunction * @param first First number * @param second Second number * @returns The sum of the two numbers. */ /* global clearInterval, console, setInterval */ export function add(first: number, second: number): number { return first + second; }

Excel CustomFunction test /** * Adds two numbers. * @customfunction * @param first First number * @param second Second number * @returns The sum of the two numbers. */ /* global clearInterval, console, setInterval */ export function add(first: number, second: number): number { return first + second; }

Test: Initializing route to Python • Install JavaScript / node.js SDK • Load test (as ESModule) src/functions/functions.ts import * as ai from "backend.ai-client/backend.ai-client-es6.js"; $ npm i backend.ai-client

Test: Initializing route to Python Office.onReady(() => { const api_key: string = 'AKIAIXXXXXXXXEXAMPLE'; const secret_key: string = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'; const api_endpoint: string = 'http://127.0.0.1:8091/'; const clientConfig : any = new ai.backend.ClientConfig( api_key, secret_key, api_endpoint ); g.bclient = new ai.backend.Client( clientConfig, `Excel Adapter`, ); g.bclient.ready = false; g.client.get_manager_version().then((response) => { console.log("Server Health Ok.") }); }); src/functions/functions.ts Depends on your config. NOTE: You SHOULD setup the certificate for your local environment! • Office generator does it for you. • Reuse it for Backend.AI / Local environment • Otherwise, you will experience CORS[1] issue https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS

/** * Test connection between Excel and Cluster * @customfunction TEST_CONNECTION * @param invocation Custom function handler */ export function testConnection(invocation: CustomFunctions.StreamingInvocation): void { const api_key = "AKIAIXXXXXXXXEXAMPLE"; const secret_key = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"; const api_endpoint = "http://127.0.0.1:8091/"; invocation.setResult("Connecting..."); const clientConfig = new ai.backend.ClientConfig( api_key, secret_key, api_endpoint ); globalThis.bclient = new ai.backend.Client( clientConfig, `Excel Adapter`, ); globalThis.bclient.ready = false; globalThis.bclient.get_manager_version().then((response) => { invocation.setResult("Succeed"); }).catch((e) =>{ invocation.setResult("Failed"); }); invocation.onCanceled = () => { }; } src/functions/functions.ts

Let function be trainable • Details of EXAMPLETRAIN o Create temporary .csv file to train o Call the training function in the model folder o Give temporary .csv file as an argument (file to upload) o Get the result from API and send result to Excel https://www.flaticon.com/premium-icon/csv_172534 https://www.flaticon.com/free-icon/excel_732220 https://www.flaticon.com/premium-icon/folder_2140705 https://www.flaticon.com/free-icon/engine_999298 https://www.flaticon.com/premium-icon/play_2377746 Trainer EXAMPLETRAIN

Let function be trainable • Details of EXAMPLE o Call the TensorFlow serving image with model name / argument o Reuse inference container if already exists § Start session with same name (e.g. EXAMPLE) will automatically reuse the previous one. o Get the result and send to Excel Model EXAMPLE

async function connectToManager() { if ('bclient' in globalThis && globalThis.bclient.ready == true) { return Promise.resolve(true); } const api_key = "AKIAIXXXXXXXXEXAMPLE"; const secret_key = "ABCDEFGHIJKLMNOPQRSTUVWXYZ"; const api_endpoint = "https://127.0.0.1:8091"; const clientConfig = new ai.backend.ClientConfig( api_key, secret_key, api_endpoint ); globalThis.bclient = new ai.backend.Client( clientConfig, `Excel Adapter`, ); globalThis.bclient.ready = false; globalThis.bclient.get_manager_version().then((response) => { globalThis.bclient.ready = true; return Promise.resolve(true); }).catch((e) =>{ return Promise.resolve(false); }); } Connection • Connect to Python Environment o Establish connection o Share it to other customfunctions • globalThis o Global object of JavaScript o Different from the runtime o Undocumented but works in Excel

Demo: Iris dataset / training & inference • Simple classification example • Iris Flower dataset o R. A. Fisher (1936) o Length and the width of the sepals and petals o Species • Famous and simple o One of the “Hello world” example data https://en.wikipedia.org/wiki/Iris_flower_data_set#/media/File:Kosaciec_szczecinkowaty_Iris_setosa.jpg

/** * Train IRIS flower dataset. * @customfunction IRIS_TRAIN * @param {number[][]} values IRIS data * @param invocation Custom function handler */ export function run_iris_training(values, invocation: CustomFunctions.StreamingInvocation): void { try { let data = JSON.stringify(values); invocation.setResult(data); connectToManager().then(() => { invocation.setResult("Creating TensorFlow Environment..."); let resources = {"cpu": 1, "mem": "6g", "mounts": ['code’], ... "maxWaitSeconds": 30}; return globalThis.bclient.createIfNotExists("cr.backend.ai/testing/python-tensorflow:2.5-py38-cuda11.3", "TF_IRIS", resources); }).then(response => { invocation.setResult(`My session is created: ${response.sessionId}`); return globalThis.bclient.execute("TF_IRIS", 1, "query", `import os;os.system("cd /home/work/code;python receive_iris_data.py ${data}")`, {}); }).then(response => { if (response.result.exitCode === 0) { invocation.setResult(`Training...`); return globalThis.bclient.execute("TF_IRIS", 1, "query", `import os;os.system("cd /home/work/code;python train_iris_model.py")`, {}); }).then(response => { if (response.result.exitCode === 0) { invocation.setResult(`Training completed.`); } }} ... Iris flower Example: Excel Part

/** * Train IRIS flower dataset. * @customfunction IRIS_TRAIN * @param {number[][]} values IRIS data * @param invocation Custom function handler */ export function run_iris_training(values, invocation: CustomFunctions.StreamingInvocation): void { try { let data = JSON.stringify(values); invocation.setResult(data); connectToManager().then(() => { invocation.setResult("Creating TensorFlow Environment..."); let resources = {"cpu": 1, "mem": "6g", "mounts": ['code’], ... "maxWaitSeconds": 30}; return globalThis.bclient.createIfNotExists("cr.backend.ai/testing/python-tensorflow:2.5-py38-cuda11.3", "TF_IRIS", resources); }).then(response => { invocation.setResult(`My session is created: ${response.sessionId}`); return globalThis.bclient.execute("TF_IRIS", 1, "query", `import os;os.system("cd /home/work/code;python receive_iris_data.py ${data}")`, {}); }).then(response => { if (response.result.exitCode === 0) { invocation.setResult(`Training...`); return globalThis.bclient.execute("TF_IRIS", 1, "query", `import os;os.system("cd /home/work/code;python train_iris_model.py")`, {}); }).then(response => { if (response.result.exitCode === 0) { invocation.setResult(`Training completed.`); } }} ... Iris flower Example: Excel Part

/** * Train IRIS flower dataset. * @customfunction IRIS_TRAIN * @param {number[][]} values IRIS data * @param invocation Custom function handler */ export function run_iris_training(values, invocation: CustomFunctions.StreamingInvocation): void { try { let data = JSON.stringify(values); invocation.setResult(data); connectToManager().then(() => { invocation.setResult("Creating TensorFlow Environment..."); let resources = {"cpu": 1, "mem": "6g", "mounts": ['code’], ... "maxWaitSeconds": 30}; return globalThis.bclient.createIfNotExists("cr.backend.ai/testing/python-tensorflow:2.5-py38-cuda11.3", "TF_IRIS", resources); }).then(response => { invocation.setResult(`My session is created: ${response.sessionId}`); return globalThis.bclient.execute("TF_IRIS", 1, "query", `import os;os.system("cd /home/work/code;python receive_iris_data.py ${data}")`, {}); }).then(response => { if (response.result.exitCode === 0) { invocation.setResult(`Training...`); return globalThis.bclient.execute("TF_IRIS", 1, "query", `import os;os.system("cd /home/work/code;python train_iris_model.py")`, {}); }).then(response => { if (response.result.exitCode === 0) { invocation.setResult(`Training completed.`); } }} ... Iris flower Example: Excel Part

Iris flower Example: Pandas Part import sys, json import numpy import pandas as pd data = json.loads(sys.argv[1]) pd.DataFrame(data).to_csv(”temp_iris.csv",header=[len(data), 4, 'setosa','versicolor','virginica'], index=None) …Super simple

Iris flower Example: TensorFlow Part import tensorflow as tf import matplotlib.pyplot as plt import os dataFile = "./temp_iris.csv" column_names = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'species'] feature_names = column_names[:-1] label_name = column_names[-1] class_names = ['Iris setosa', 'Iris versicolor', 'Iris virginica'] batch_size = 32 train_dataset = tf.data.experimental.make_csv_dataset( dataFile, batch_size, column_names=column_names, label_name=label_name, num_epochs=1) def pack_features_vector(features, labels): features = tf.stack(list(features.values()), axis=1) return features, labels train_dataset = train_dataset.map(pack_features_vector) features, labels = next(iter(train_dataset)) Data Preparation

Iris flower Example: TensorFlow Part Model Creation def create_model(): model = tf.keras.models.Sequential([ tf.keras.layers.Dense(10, activation='relu', input_shape=(4,)), tf.keras.layers.Dense(10, activation='relu'), tf.keras.layers.Dense(3) ]) model.compile(optimizer='adam', loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=[tf.metrics.SparseCategoricalAccuracy()]) return model model = create_model()

Iris flower Example: TensorFlow Part Training checkpoint_path = "ckpt_iris_model/cp.ckpt" checkpoint_dir = os.path.dirname(checkpoint_path) # Create a callback that saves the model's weights cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path, save_weights_only=True, verbose=0) # Train the model with the new callback model.fit(train_dataset, epochs=200, verbose=0, callbacks=[cp_callback]) model.save('saved_models/iris_model')

Demo: Iris dataset / training & inference • Simple categorization example

No content

Demo: Stock price prediction / training & inference • Time-series data prediction (Yahoo Finance Data / Tesla Stock Price)

No content

Ok. Let’s solve my problem

How can I enjoy the AI wave myself • Back to my life: o Excel / HWP o Tedious repetitive works o And more • Characteristics of my life works o Based on specific tools o Have super-personalized detail (works / data)

How can I enjoy the AI wave myself • Reduce paperworks o Not repetitive o But complex enough one • Credit card payment history o As a start-up lead o Can I automatically guess the purpose of card payment? https://evangelion.fandom.com/wiki/Absolute_Terror_Field

How can I enjoy the AI wave myself • Reduce paperworks o Not repetitive o But complex enough one • Credit card payment history o As a start-up lead o Can I automatically guess the purpose of card payment? https://bbs.ruliweb.com/community/board/300143/read/47700389

8 hours to 5 seconds • How it saves my life o Quarterly credit card payment history must be specified to correct deductions o I watched it all with my own eyes, but as the company grew, the number of items easily exceeded several hundred... • Dogfooding o Data: The deduction details received as an Excel file from HomeTax o Training: based on manual entry data of deductions for the past 5 years o Inference: Use a function that automatically enters the usage (inference result) in the remarks column o A/B & Pipeline: If there is a strange place, fix it manually and then train further

8 hours to 5 seconds • Example 2019-09- 27 비씨카드（ 주） XXXX-XXXX- XXXX-XXXX 104-81-83559 주식회사 티머니 13,800 교통비 Transport fee 2019-09- 24 신한카드 주식회사 XXXX-XXXX- XXXX-XXXX 220-81-39938 （주）엘지유플러스 2,985,400 IDC

8 hours to 5 seconds • Result (in progress…) 가맹점명 합계 공제여부 결정 자동비고 써브웨이 선릉점 8,700 불공제 식사 （주）스타벅스커피 코리아 7,000 공제 커피 나이스페이먼츠 주식회사 (NICE Payments Co., Ltd.) 31,100 불공제 우노 42,000 불공제 식사 홍콩반점0410보라매공원점 13,000 불공제 식사 세광양대창 시청점 116,000 불공제 식사 한국정보통신（주） 113,000 공제 한국맥도날드(유) 강남삼성 DT점 5,100 불공제 식사 (주)비지에프리테일 비지에프사옥점 5,400 불공제 부경에스이에스（주） 14,100 공제 엔에이치엔한국사이버결제 주식회사 15,024 공제 AWS클라우드 나이스페이먼츠 주식회사 (NICE Payments Co., Ltd.) 67,800 불공제 식사 나이스페이먼츠 주식회사 (NICE Payments Co., Ltd.) 26,520 불공제 식사 한국파파존스（주） 92,620 불공제 식사

Closing Excel + Python + Deep Learning = !

Today we run together through… • Left behind o Why my life is same as before AI wave? o Why Excel + Deep Learning? • Make my Excel supercharged with AI o Excel + TypeScript + Python + TensorFlow o Architecture overview: how I created • “Trainable excel functions” o Simple examples o Personal experience https://www.flaticon.com/free-icon/winner_2617899

Technically, • AI-powered Excel example using o Excel CustomFunction o TypeScript § Yeoman generator (Microsoft Office Generator) / Backend.AI TypeScript SDK o Docker § To run virtualized environment o TensorFlow § To training / inferencing

Endless possibility • “Very-personalized local & small deep learning models” o Democratization of the use of deep learning models o Make AI Accessible: Make my PC to personal cloud • To do o Generalized CustomFunction to prevent repetitive function creation o One-click windows installer o Pre-installed VM image o More examples

Thank You J https://github.com/inureyes/PyconKR-2021-BTF (Will be ready soon;)

References • Papers / Articles o https://iopscience.iop.org/article/10.3847/1538-4357/abd62b o https://academic.oup.com/mnras/article-abstract/504/2/1825/6219095 o https://link.springer.com/article/10.1007/s10796-021-10131-x o https://www.frontiersin.org/articles/10.3389/frai.2020.00065/full o https://www.nature.com/articles/s41586-021-03819-2 o https://www.nytimes.com/2019/04/14/technology/china-surveillance-artificial-intelligence- racial-profiling.html o https://ieeexplore.ieee.org/document/9384519 o https://sgp.fas.org/crs/natsec/R45178.pdf

References • Software / Libraries / Technical Docs. o https://docs.microsoft.com/en-us/office/dev/add-ins/excel/custom-functions-overview o https://docs.microsoft.com/en-us/office/dev/add-ins/develop/application-specific-api-model o https://docs.microsoft.com/en-us/office/dev/add-ins/excel/custom-functions-requirement- sets o https://github.com/lablup/backend.ai-client-js o https://hub.docker.com/u/lablup o https://developer.microsoft.com/en-us/office/blogs/connect-2015-office-extensibility-news/ o https://developers.slashdot.org/story/17/12/15/1133217/microsoft-considers-adding- python-as-an-official-scripting-language-in-excel o https://docs.microsoft.com/en-us/office/dev/add-ins/excel/custom-functions-overview o https://github.com/OfficeDev/Excel-Custom-Functions o https://www.xlwings.org o https://openpyxl.readthedocs.io/en/stable/

References • Icon / Clipart / Screenshot o https://www.flaticon.com/free-icon/shuttle_2285537 o https://www.flaticon.com/free-icon/moon-phase_2909605 o https://www.flaticon.com/free-icon/explosion_978623 o https://www.flaticon.com/premium-icon/blocks_1273426 o https://www.flaticon.com/free-icon/brick_3813701 o https://www.flaticon.com/free-icon/sweet_2581014 o https://www.flaticon.com/free-icon/risk_2145705 o https://www.flaticon.com/free-icon/winner_2617899 o https://www.flaticon.com/premium-icon/customization_1799807 o https://www.flaticon.com/premium-icon/customization_1066631 o https://www.flaticon.com/premium-icon/csv_172534 o https://www.flaticon.com/free-icon/excel_732220 o https://www.flaticon.com/premium-icon/folder_2140705 o https://www.flaticon.com/free-icon/engine_999298 o https://www.flaticon.com/premium-icon/play_2377746 o https://oldpc.tistory.com/71 o https://www.my-internet-explorer.com/ie6/ o https://tutorialexcel.com/versiones-de-excel/excel-95