Excel + Python + Deep Learning = ! (PyCon KR 2021)

Excel + Python + Deep Learning
= !
Jeongkyu Shin

View Slide

About me
Lablup Inc. : Make AI Accessible
Maintainer / Open-source machine
learning cluster platform: Backend.AI
Google Developers Experts
ML / DL GDE (Context retrieval)
https://github.com/inureyes/

View Slide

And online again T_T

View Slide

Today’s Agenda
• Left behind
o Why my life is same as before AI wave?
o Why Excel + Deep Learning?
• Make my Excel supercharged with AI
o Excel + TypeScript + Python + TensorFlow
o Architecture overview: how I created
• “Trainable excel functions”
o Simple examples
• Closing
o Personal example: how it becomes the painkiller for my ‘real’ life.
[1] https://openwho.org/channels/covid-19-national-languages

View Slide

Left Behind
Excel + Python + Deep Learning = !

View Slide

Rise of AI Era
• AI Wave: DEEP LEARNING
o ImageNet competition (2012)
o AlphaGo (2016)
o Attention / Transformer (2017 / 2018)
o XLNet / T5 (2019)
o AlphaStar (2019)
o GPT-3 (2020)
• Boosted applications
o Image segmentation / categorization
o NLU
o Self-driving
o Robotics
[1] LaMDA / Google I/O 2021 Keynote

View Slide

Moonshot
• Reaching the stars
o Scanning the universe[1]
o Filtering gravity lens noise[2]
• Combat with COVID-19
o Analyzing coronavirus details[3]
o Finding cure candidates[4]
• Solving the nature riddles
o Protein-folding prediction[5]
https://www.flaticon.com/free-icon/shuttle_2285537
https://www.flaticon.com/free-icon/moon-phase_2909605
https://www.flaticon.com/free-icon/explosion_978623
[1] https://iopscience.iop.org/article/10.3847/1538-4357/abd62b
[2] https://academic.oup.com/mnras/article-abstract/504/2/1825/6219095
[3] https://link.springer.com/article/10.1007/s10796-021-10131-x
[4] https://www.frontiersin.org/articles/10.3389/frai.2020.00065/full
[5] https://www.nature.com/articles/s41586-021-03819-2

View Slide

Shades under the moon
• Privacy
o Recognizing personal identities[1]
o Surveillance to unspecified citizens
• War tool
o Military programs[2,3]
https://www.bbc.com/news/business-56195288
https://www.popularmechanics.com/technology/robots/a29610393/robot-soldier-boston-dynamics/
[1] https://www.nytimes.com/2019/04/14/technology/china-surveillance-artificial-intelligence-racial-profiling.html
[2] https://ieeexplore.ieee.org/document/9384519
[3] https://sgp.fas.org/crs/natsec/R45178.pdf

View Slide

Why my life is same as before AI wave?
• Strongness and weakness of Deep Learning models
o Work: Categorization / detection
o Data: Image / Text data
• And my life is full of…
o Excel / HWP
o Tedious repetitive works
o And more
• Characteristics of my life works
o Based on specific tools
o Have super-personalized detail (works / data)
https://www.flaticon.com/free-icon/moon-phase_2909605
https://oldpc.tistory.com/71
https://www.my-internet-explorer.com/ie6/
https://tutorialexcel.com/versiones-de-excel/excel-95/

View Slide

Why Excel + Deep Learning?
• Data = Excel for daily work
o Half of the most tedious tasks are happening on the Excel sheet
o (Another half: Active-X + IE11)
• So hard to be automated with ‘numerical functions’
o Many fields are not “numerical”
https://www.flaticon.com/free-icon/risk_2145705

View Slide

Make my Excel
Supercharged with AI

View Slide

Excel + ?: Old ways
• Traditional methods
o Xlwings[1] (https://www.xlwings.org)
o Openpyxl[2] (https://foss.heptapod.net/openpyxl/openpyxl)
• Limitations: COM wrapper / depends on VBA
o Works only on Windows
o Limited real-time binding
[1] https://www.xlwings.org
[2] https://openpyxl.readthedocs.io/en/stable/
https://www.flaticon.com/free-icon/sweet_2581014

View Slide

Excel + ?: Modern way
• 2015
o Microsoft announces JavaScript API for Office[1]
• 2018
o Microsoft announces Python support for script language[2]
§ Not implemented / Revamping Microsoft Office for Web
§ Transitioning to Web Standard
o JavaScript-based Excel CustomFunction announced[3]
• 2020
o JavaScript as an official script language[4]
[1] https://developer.microsoft.com/en-us/office/blogs/connect-2015-office-extensibility-news/
[2] https://developers.slashdot.org/story/17/12/15/1133217/microsoft-considers-adding-python-as-an-official-scripting-language-in-excel
[3] https://docs.microsoft.com/en-us/office/dev/add-ins/excel/custom-functions-overview
[4] https://github.com/OfficeDev/Excel-Custom-Functions
https://www.flaticon.com/free-icon/back-to-the-future_86503

View Slide

Docker
Building blocks for today
• Microsoft Excel 365 (with 2021~ versions)
• TypeScript
o To compile Excel CustomFunction with JavaScript SDK
• Docker (for windows / macOS)
o To run sandboxed deep learning runtime
• Backend.AI as glue / runtime framework
o Glue between TypeScript and Deep learning container
o JavaScript and Python SDK counterpart exists
• You may choose one among other glue projects!
o TF.js without Python/Docker
o Anaconda with your handmade TypeScript SDK
https://www.flaticon.com/premium-icon/blocks_1273426
https://www.flaticon.com/free-icon/brick_3813701
Excel

View Slide

Excel + TypeScript + Python
• Excel
o Custom Functions (2020)[1]
o Full JavaScript / TypeScript support
o Limitation: Browser compatibility issues [2]
• TypeScript
o Binding CustomFunctionsRuntimeAPI [3]
o Will use Backend.AI TypeScript SDK [4] to communicate Excel JavaScript fn. with Python
container
• Python
o Local cluster / VM or baremetal
o Docker for containerized environment [5]
[1] https://docs.microsoft.com/en-us/office/dev/add-ins/excel/custom-functions-overview
[2] https://docs.microsoft.com/en-us/office/dev/add-ins/develop/application-specific-api-model
[3] https://docs.microsoft.com/en-us/office/dev/add-ins/excel/custom-functions-requirement-sets
[4] https://github.com/lablup/backend.ai-client-js
[5] https://hub.docker.com/u/lablup

View Slide

Excel + TypeScript + Python + TensorFlow
• TensorFlow
o TensorFlow 2.5 Environment runtime[1]
o Training
§ TensorFlow 2.5 (CPU / OneAPI)
o Inference
§ TensorFlow Serving
• API connector
o GraphQL-based API
o Connect with JavaScript SDK
§ Converted from TypeScript+Node.js 14 to Pure ESM
[1] docker pull lablup/python-tensorflow:2.5-py38-cuda11.3

View Slide

Architecture overview

View Slide

“Trainable excel functions”

View Slide

Making custom functions
• Two functions for one model
o No general model functions: it will not look like ‘Excel function’
o One for training, one for inferencing
• EXAMPLETRAIN(data_to_train)
o To training & saving EXAMPLE model
o e.g. IRIS_MODEL_TRAIN(data)
• EXAMPLE(data_to_inference)
o To inferencing with the EXAMPLE model
o e.g. IRIS_MODEL(column)
• Characteristics
o Training time
o Training data preservation
o Keeping checkpoint to additional training
https://www.flaticon.com/premium-icon/customization_1066631

View Slide

So we will make
• BTF
o Bidirectional TensorFlow workflow example for Excel CustomFunction
• Why not BTFECF?
o Rhyme;
o Not BTS but TensorFlow

View Slide

Preparing development environment
• Dev. Environment preparation
o Install yeoman generator as global npm / yarn package
§ [NPM] npm install -g yo generator-office
§ [Yarn] yarn global add yo generator-office
o Initialize Microsoft JavaScript Add-in repository
§ yo office
§ Create “TypeScript” type, “EXAMPLETRAIN” project
o Install TypeScript SDK
§ npm install backend.ai-client
o Build / sideload test
§ [Build] npm run build
§ [DevServer] npm run dev-server
§ [Sideloading] npm start
https://www.flaticon.com/premium-icon/customization_1799807

View Slide

Platform dependency
• Excel (App)
o Windows: EdgeHTML (Legacy) / Blink
(Modern)
o macOS: WebKit2
• Excel (Web)
o Depends on the user browser
• Supports
o JavaScript ES5 or above
o Transpilers help you

View Slide


View Slide

npm run dev-server npm start:desktop
Tip: CHANGE localhost to 127.0.0.1
in manifest.yml

View Slide

You will have task panel (template)
on the ribbon.

View Slide

Creating CustomFunction test logic
• Modify manifest.yml
o Change CONTOSO to your namespace. e.g. BTF
o CHANGE localhost to 127.0.0.1
• Check src/functions/functions.ts
o We will add functions here!
o Default example functions: add, clock, currentTime, increment…
https://www.flaticon.com/free-icon/example_5360938

View Slide

Excel CustomFunction test
/**
* Adds two numbers.
* @customfunction
* @param first First number
* @param second Second number
* @returns The sum of the two numbers.
*/
/* global clearInterval, console, setInterval */
export function add(first: number, second:
number): number {
return first + second;
}

View Slide

Test: Initializing route to Python
• Install JavaScript / node.js SDK
• Load test (as ESModule)
src/functions/functions.ts
import * as ai from "backend.ai-client/backend.ai-client-es6.js";
$ npm i backend.ai-client

View Slide

Test: Initializing route to Python
Office.onReady(() => {
const api_key: string = 'AKIAIXXXXXXXXEXAMPLE';
const secret_key: string = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ';
const api_endpoint: string = 'http://127.0.0.1:8091/';
const clientConfig : any = new ai.backend.ClientConfig(
api_key,
secret_key,
api_endpoint
);
g.bclient = new ai.backend.Client(
clientConfig,
`Excel Adapter`,
);
g.bclient.ready = false;
g.client.get_manager_version().then((response) => {
console.log("Server Health Ok.")
});
});
Depends on your config.
NOTE: You SHOULD setup the certificate
for your local environment!
• Office generator does it for you.
• Reuse it for Backend.AI / Local environment
• Otherwise, you will experience CORS[1] issue
https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS

View Slide

/**
* Test connection between Excel and Cluster
* @customfunction TEST_CONNECTION
* @param invocation Custom function handler
*/
export function testConnection(invocation: CustomFunctions.StreamingInvocation): void {
const api_key = "AKIAIXXXXXXXXEXAMPLE";
const secret_key = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
const api_endpoint = "http://127.0.0.1:8091/";
invocation.setResult("Connecting...");
const clientConfig = new ai.backend.ClientConfig(
api_key,
secret_key,
api_endpoint
);
globalThis.bclient = new ai.backend.Client(
clientConfig,
`Excel Adapter`,
);
globalThis.bclient.ready = false;
globalThis.bclient.get_manager_version().then((response) => {
invocation.setResult("Succeed");
}).catch((e) =>{
invocation.setResult("Failed");
});
invocation.onCanceled = () => {
};
}

View Slide

Let function be trainable
• Details of EXAMPLETRAIN
o Create temporary .csv file to train
o Call the training function in the model folder
o Give temporary .csv file as an argument (file to upload)
o Get the result from API and send result to Excel
https://www.flaticon.com/premium-icon/csv_172534
https://www.flaticon.com/free-icon/excel_732220
https://www.flaticon.com/premium-icon/folder_2140705
https://www.flaticon.com/free-icon/engine_999298
https://www.flaticon.com/premium-icon/play_2377746
Trainer
EXAMPLETRAIN

View Slide

Let function be trainable
• Details of EXAMPLE
o Call the TensorFlow serving image with model name / argument
o Reuse inference container if already exists
§ Start session with same name (e.g. EXAMPLE) will automatically reuse the previous one.
o Get the result and send to Excel
Model
EXAMPLE

View Slide

async function connectToManager() {
if ('bclient' in globalThis && globalThis.bclient.ready == true) {
return Promise.resolve(true);
}
const api_key = "AKIAIXXXXXXXXEXAMPLE";
const secret_key = "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
const api_endpoint = "https://127.0.0.1:8091";
const clientConfig = new ai.backend.ClientConfig(
api_key,
secret_key,
api_endpoint
);
globalThis.bclient = new ai.backend.Client(
clientConfig,
`Excel Adapter`,
);
globalThis.bclient.ready = false;
globalThis.bclient.get_manager_version().then((response) => {
globalThis.bclient.ready = true;
return Promise.resolve(true);
}).catch((e) =>{
return Promise.resolve(false);
});
}
Connection
• Connect to Python
Environment
o Establish connection
o Share it to other
customfunctions
• globalThis
o Global object of
JavaScript
o Different from the
runtime
o Undocumented but
works in Excel

View Slide

Demo: Iris dataset / training & inference
• Simple classification example
• Iris Flower dataset
o R. A. Fisher (1936)
o Length and the width of the sepals and petals
o Species
• Famous and simple
o One of the “Hello world” example data
https://en.wikipedia.org/wiki/Iris_flower_data_set#/media/File:Kosaciec_szczecinkowaty_Iris_setosa.jpg

View Slide

/**
* Train IRIS flower dataset.
* @customfunction IRIS_TRAIN
* @param {number[][]} values IRIS data
* @param invocation Custom function handler
*/
export function run_iris_training(values, invocation: CustomFunctions.StreamingInvocation): void {
try {
let data = JSON.stringify(values);
invocation.setResult(data);
connectToManager().then(() => {
invocation.setResult("Creating TensorFlow Environment...");
let resources = {"cpu": 1,
"mem": "6g",
"mounts": ['code’],
...
"maxWaitSeconds": 30};
return globalThis.bclient.createIfNotExists("cr.backend.ai/testing/python-tensorflow:2.5-py38-cuda11.3", "TF_IRIS", resources);
}).then(response => {
invocation.setResult(`My session is created: ${response.sessionId}`);
return globalThis.bclient.execute("TF_IRIS", 1, "query",
`import os;os.system("cd /home/work/code;python receive_iris_data.py ${data}")`, {});
if (response.result.exitCode === 0) {
invocation.setResult(`Training...`);
return globalThis.bclient.execute("TF_IRIS", 1, "query",
`import os;os.system("cd /home/work/code;python train_iris_model.py")`, {});
if (response.result.exitCode === 0) {
invocation.setResult(`Training completed.`);
}
}}
...
Iris flower Example: Excel Part

View Slide

Iris flower Example: Pandas Part
import sys, json
import numpy
import pandas as pd
data = json.loads(sys.argv[1])
pd.DataFrame(data).to_csv(”temp_iris.csv",header=[len(data), 4,
'setosa','versicolor','virginica'], index=None)
…Super simple

View Slide

Iris flower Example: TensorFlow Part
import tensorflow as tf
import matplotlib.pyplot as plt
import os
dataFile = "./temp_iris.csv"
column_names = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'species']
feature_names = column_names[:-1]
label_name = column_names[-1]
class_names = ['Iris setosa', 'Iris versicolor', 'Iris virginica']
batch_size = 32
train_dataset = tf.data.experimental.make_csv_dataset(
dataFile,
batch_size,
column_names=column_names,
label_name=label_name,
num_epochs=1)
def pack_features_vector(features, labels):
features = tf.stack(list(features.values()), axis=1)
return features, labels
train_dataset = train_dataset.map(pack_features_vector)
features, labels = next(iter(train_dataset))
Data Preparation

View Slide

Model Creation
def create_model():
model = tf.keras.models.Sequential([
tf.keras.layers.Dense(10, activation='relu', input_shape=(4,)),
tf.keras.layers.Dense(10, activation='relu'),
tf.keras.layers.Dense(3)
])
model.compile(optimizer='adam',
loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=[tf.metrics.SparseCategoricalAccuracy()])
return model
model = create_model()

View Slide

Training
checkpoint_path = "ckpt_iris_model/cp.ckpt"
checkpoint_dir = os.path.dirname(checkpoint_path)
# Create a callback that saves the model's weights
cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,
save_weights_only=True,
verbose=0)
# Train the model with the new callback
model.fit(train_dataset,
epochs=200,
verbose=0,
callbacks=[cp_callback])
model.save('saved_models/iris_model')

View Slide

Demo: Iris dataset / training & inference
• Simple categorization example

View Slide

View Slide

Demo: Stock price prediction / training & inference
• Time-series data prediction (Yahoo Finance Data / Tesla Stock Price)

View Slide

View Slide

Ok. Let’s solve my problem

View Slide

How can I enjoy the AI wave myself
• Back to my life:
o Excel / HWP
o Tedious repetitive works
o And more
• Characteristics of my life works
o Based on specific tools
o Have super-personalized detail (works / data)

View Slide

• Reduce paperworks
o Not repetitive
o But complex enough one
• Credit card payment history
o As a start-up lead
o Can I automatically guess the
purpose of card payment?
https://evangelion.fandom.com/wiki/Absolute_Terror_Field

View Slide

• Reduce paperworks
o Not repetitive
o But complex enough one
• Credit card payment history
o As a start-up lead
o Can I automatically guess the
purpose of card payment?
https://bbs.ruliweb.com/community/board/300143/read/47700389

View Slide

8 hours to 5 seconds
• How it saves my life
o Quarterly credit card payment history must be specified to correct deductions
o I watched it all with my own eyes, but as the company grew, the number of items easily
exceeded several hundred...
• Dogfooding
o Data: The deduction details received as an Excel file from HomeTax
o Training: based on manual entry data of deductions for the past 5 years
o Inference: Use a function that automatically enters the usage (inference result) in the remarks
column
o A/B & Pipeline: If there is a strange place, fix it manually and then train further

View Slide

• Example
2019-09-
27
비씨카드（
주）
XXXX-XXXX-
XXXX-XXXX
104-81-83559 주식회사 티머니 13,800
교통비
Transport fee
2019-09-
24
신한카드
주식회사
XXXX-XXXX-
XXXX-XXXX
220-81-39938 （주）엘지유플러스 2,985,400
IDC

View Slide

• Result (in progress…)
가맹점명 합계
공제여부
결정
자동비고
써브웨이 선릉점 8,700 불공제 식사
（주）스타벅스커피 코리아 7,000 공제 커피
나이스페이먼츠 주식회사 (NICE Payments Co., Ltd.) 31,100 불공제
우노 42,000 불공제 식사
홍콩반점0410보라매공원점 13,000 불공제 식사
세광양대창 시청점 116,000 불공제 식사
한국정보통신（주） 113,000 공제
한국맥도날드(유) 강남삼성 DT점 5,100 불공제 식사
(주)비지에프리테일 비지에프사옥점 5,400 불공제
부경에스이에스（주） 14,100 공제
엔에이치엔한국사이버결제 주식회사 15,024 공제 AWS클라우드
나이스페이먼츠 주식회사 (NICE Payments Co., Ltd.) 67,800 불공제 식사
나이스페이먼츠 주식회사 (NICE Payments Co., Ltd.) 26,520 불공제 식사
한국파파존스（주） 92,620 불공제 식사

View Slide

Closing

View Slide

Today we run together through…
• Left behind
o Why my life is same as before AI wave?
o Why Excel + Deep Learning?
• Make my Excel supercharged with AI
o Excel + TypeScript + Python + TensorFlow
o Architecture overview: how I created
• “Trainable excel functions”
o Simple examples
o Personal experience
https://www.flaticon.com/free-icon/winner_2617899

View Slide

Technically,
• AI-powered Excel example using
o Excel CustomFunction
o TypeScript
§ Yeoman generator (Microsoft Office Generator) / Backend.AI TypeScript SDK
o Docker
§ To run virtualized environment
o TensorFlow
§ To training / inferencing

View Slide

Endless possibility
• “Very-personalized local & small deep learning models”
o Democratization of the use of deep learning models
o Make AI Accessible: Make my PC to personal cloud
• To do
o Generalized CustomFunction to prevent repetitive function creation
o One-click windows installer
o Pre-installed VM image
o More examples

View Slide

Thank You J
https://github.com/inureyes/PyconKR-2021-BTF
(Will be ready soon;)

View Slide

References
• Papers / Articles
o https://iopscience.iop.org/article/10.3847/1538-4357/abd62b
o https://academic.oup.com/mnras/article-abstract/504/2/1825/6219095
o https://link.springer.com/article/10.1007/s10796-021-10131-x
o https://www.frontiersin.org/articles/10.3389/frai.2020.00065/full
o https://www.nature.com/articles/s41586-021-03819-2
o https://www.nytimes.com/2019/04/14/technology/china-surveillance-artificial-intelligence-
racial-profiling.html
o https://ieeexplore.ieee.org/document/9384519
o https://sgp.fas.org/crs/natsec/R45178.pdf

View Slide

References
• Software / Libraries / Technical Docs.
o https://docs.microsoft.com/en-us/office/dev/add-ins/excel/custom-functions-overview
o https://docs.microsoft.com/en-us/office/dev/add-ins/develop/application-specific-api-model
o https://docs.microsoft.com/en-us/office/dev/add-ins/excel/custom-functions-requirement-
sets
o https://github.com/lablup/backend.ai-client-js
o https://hub.docker.com/u/lablup
o https://developer.microsoft.com/en-us/office/blogs/connect-2015-office-extensibility-news/
o https://developers.slashdot.org/story/17/12/15/1133217/microsoft-considers-adding-
python-as-an-official-scripting-language-in-excel
o https://docs.microsoft.com/en-us/office/dev/add-ins/excel/custom-functions-overview
o https://github.com/OfficeDev/Excel-Custom-Functions
o https://www.xlwings.org
o https://openpyxl.readthedocs.io/en/stable/

View Slide

References
• Icon / Clipart / Screenshot
o https://www.flaticon.com/free-icon/shuttle_2285537
o https://www.flaticon.com/free-icon/moon-phase_2909605
o https://www.flaticon.com/free-icon/explosion_978623
o https://www.flaticon.com/premium-icon/blocks_1273426
o https://www.flaticon.com/free-icon/brick_3813701
o https://www.flaticon.com/free-icon/sweet_2581014
o https://www.flaticon.com/free-icon/risk_2145705
o https://www.flaticon.com/free-icon/winner_2617899
o https://www.flaticon.com/premium-icon/customization_1799807
o https://www.flaticon.com/premium-icon/customization_1066631
o https://www.flaticon.com/premium-icon/csv_172534
o https://www.flaticon.com/free-icon/excel_732220
o https://www.flaticon.com/premium-icon/folder_2140705
o https://www.flaticon.com/free-icon/engine_999298
o https://www.flaticon.com/premium-icon/play_2377746
o https://oldpc.tistory.com/71
o https://www.my-internet-explorer.com/ie6/
o https://tutorialexcel.com/versiones-de-excel/excel-95

View Slide

Excel + Python + Deep Learning= ! (PyCon KR 2021)

Excel + Python + Deep Learning = ! (PyCon KR 2021)

Jeongkyu Shin

More Decks by Jeongkyu Shin

Other Decks in Programming

Featured

Transcript