Upgrade to Pro — share decks privately, control downloads, hide ads and more …

카카오뱅크 MLOps

kakao
December 08, 2022

카카오뱅크 MLOps

#ML #mlops #아키텍처 #시스템구성 #MLPipeline #라이브러리 #MLModelServing

데이터과학자가 개발한 ML 모델을 서비스에 빠르게 적용하기 위한 방법을 모색합니다. MLOps를 만들어 가기 위한 툴로 다양한 Open Source가 존재하지만, 각자의 회사에 알맞은 시스템 구성이 필요합니다. 본 세션에서는 MLOps에 대해 고민했던 내용에 대해 말씀드리고, 카카오뱅크만의 MLOps 시스템 구성부터 주요 콤포넌트 그리고 서비스 적용에 대한 경험을 공유합니다.

발표자 : toby.k
카카오뱅크 빅데이터분석팀 토비입니다.

kakao

December 08, 2022
Tweet

More Decks by kakao

Other Decks in Programming

Transcript

  1. ࣗ೐౟ਝযѐߊ੗р੄ࣗాҗഈসਸъઑೞחѐߊޙച %4৬ূ૑פযр੄ࣗాҗഈসਸъઑೞח.-ূ૑פয݂ޙച ѐߊ੗ܳਤೠ੗زച೐۽ࣁझੋ૑ࣘ੸ੋా೤ਸ੄޷  ௏٘୽جޙઁ೧Ѿ ؘ੉ఠ ؘ੉ఠझః݃ ݽ؛ী؀೧పझ౟ೞҊѨૐ ૑ࣘ੸ੋࢲ࠺झઁҕ߂૑ࣘ੸ੋߓನܳ੄޷ ݽ؛೟ण߂৘ஏࢲ࠺झܳ੗زਵ۽ߓನ೧ঠೞחदझమ

    .-दझమҊਬࣘࢿ ݽ؛ਸ੗زਵ۽੤೟णदఃҊࢲ࠺झઁҕ %FW0QT৬.-0QT੄࠺Ү • CT (지속적 학습) • CD DEVOps • 정의 • CI .-0QT • 정의 • CI • CD • CT (지속적 학습) • CD • CI • CI • CD • 정의 DEVOps • 정의 .-0QT ※ DevOps의 CT(Continuous Testing)과 MLOps의 CM(Continuous Monitoring)은 추가적으로 설명하지 않음. ※ DevOps와 MLOps의 차이는 서비스를 운영하는 목적의 차이가 있을 뿐, 상당히 유사하다고 볼 수 있음.
  2. ஠஠য়ߛ௼.-0QTইఃఫ୊ (PPHMF.-0QTࣻળ_ױ҅੄ҳࢿਃࣗ৬ౠ૚ਸળࣻೞৈ ஠஠য়ߛ௼.-0QTইఃఫ୊ܳҳࢿೞ৓਺ %BUB FYUSBDUJPO %BUB QSFQBSBUJPO .PEFM USBJOJOH .PEFM

    FWBMVBUJPO .PEFM WBMJEBUJPO 4PVSDF DPEF %BUB BOBMZTJT %BUB FYUSBDUJPO %BUB WBMJEBUJPO %BUB QSFQBSBUJPO .PEFM USBJOJOH .PEFM FWBMVBUJPO .PEFM WBMJEBUJPO .-NFUBEBUBTUPSF (JUMBC $* .PEFM SFHJTUSZ 'FBUVSF 4UPSF %BUB.BSU &%8 1FSGPSNBODF .POJUPSJOH $% .PEFMTFSWJOH 1SFEJDUJPO TFSWJDF %BUBTPVSDF NFUBTZTUFN 5SJHHFS &DBNT $% "VUPNBUFE1JQFMJOF 0SDIFTUSBUFE&YQFSJNFOU FYQFSJNFOUBUJPOUFTU QSPEVDUJPO #BUDI 'FUDIJOH .-0QT œ஠஠য়ߛ௼.-0QTইఃఫ୊ח_ױ҅۽ա־য૓೯ೞ৓Ҋ ݃૑݄ױ҅੄ইఃఫ୊݅ҕਬ೤פ׮
  3. .-0QTदझమҳࢿب ҃৔ݎ Әਲ਼ѐߊݎ Әਲ਼਍৔ݎ FOE - VTFS .-4FSWJOH4FSWFS .PEFM3FQP %PDLFS

    3FHJTUSZ .-1JQFMJOF 0OMJOF'4 0GGMJOF'4 (JU-BC ߓ஖ࢲࡂ (JU-BC पदрࢲࡂ .PEFM#VJME4FSWFS .FUB%BUB 4FSWJDF 4FSWJDF 4FSWJDF 경영망 금융개발망 금융운영망 .PEFM3FQP (JU-BC .PEFM#VJME4FSWFS (JU-BC %PDLFS 3FHJTUSZ .-1JQFMJOF 0GGMJOF'4 0OMJOF'4 .-4FSWJOH4FSWFS
  4. 'FBUVSF4UPSF 0GGMJOF'4 0OMJOF'4 %BUB4PVSDF $IBOOFM &%8 $PSF#BOLJOH 4USFBNJOH #BUDI LBGLB

    'FBUVSF4UPSF .PEFM 3FBM - UJNF %BUB-BLF .PEFM 4FSWF .PEFM 5SBJOJOH 1SPDFTTJOH ,VEV)%'4 %BUB.BSU $IBOOFM $PSF#BOLJOH &%8 %BUB.BSU $IBOOFM 0OMJOF'4 0GGMJOF'4 .PEFM 4FSWF .PEFM 5SBJOJOH
  5. .PEFM SFHJTUSZ .-ݽ؛౵੉೐ۄੋమ೒݁ .-ݽ؛౵੉೐ۄੋਸమ೒݁ചೞৈ ੗زചػ.-౵੉೐ۄੋਸऔҊࡅܰѱࢲ࠺झী੸ਊೣ %BUB FYUSBDUJPO %BUB WBMJEBUJPO %BUB

    QSFQBSBUJPO .PEFM USBJOJOH .PEFM FWBMVBUJPO .PEFM WBMJEBUJPO 1SFEJDUJPO TFSWJDF झ௏য14* MJGU BDDVSBDZj 1FSGPSNBODF .POJUPSJOH $5 /P :FT ೖ୊14* ৘ஏчࢤࢿ -JGUj 005j .-'SBNFXPSLT ݽ؛ۨನ੷੢ ೖ୊ࢤࢿ ೖ୊੐ߓ٬ ݽ؛೟ण ૑ࣘ੸೟ण 14* 1PQVMBUJPO4UBCJMJUZ*OEFY ݽ૘ױ੄উ੿ࢿਸաఋղח૑಴ -JGU ೱ࢚ب ੹୓߈਽ܫ؀࠺ ೧׼࠙ਤ੄߈਽ܫ࠺Ү 005 0VUPG5JNF ݽ؛ѐߊীࢎਊػؘ੉ఠ৬৮੹൤׮ܲӝр੄ ؘ੉ఠ۽Ѩૐ %BUB1SPDFTTJOH
  6. ؘ੉ఠҗ೟੗৬ূ૑פযр੄ழޭפா੉࣌ਸ׸׼ ੌҙػ௏٘7JFX ؘ੉ఠҗ೟੗оࢎਊೞח௏٘ܳ਍৔ࢲ࠺झী߄۽੸ਊ ࠛ೙ਃೠ௏٘߸ജ .-ݽ؛ࢲࡂਸࠁ׮औҊࡅܰѱ ௏٘੄न܉ࢿ .-0QTۄ੉࠳۞ܻ 역할 주요기능 'FBUVSF.BOBHFNFOU

    %FGJOJUJPO 'JOE 'FBUVSF0QFSBUPST $SFBUF6QEBUF%FMFUF7JFX "VUPNBUJD'FBUVSF"OBMZTJT &%" $PSSFMBUJPO .PEFM.BOBHFNFOU %FGJOJUJPO 'JOE .PEFM0QFSBUPST $SFBUF6QEBUF%FMFUF7JFX 1JQFMJOF"QQMJFS • • • • • • • • • 1JQFMJOF)BOEMFS 7BMJEBUJPO 1BDLJOH 'FBUVSF .PEFM 1JQFMJOF 'FBUVSF .PEFM 1JQFMJOF
  7. .-0QT૑ࣘ੸೟ण $5 $POUJOVPVT5SBJOJOH  ੉޷૑୹୊JHVB[JP ஠஠য়ߛ௼"5.੉ਊѤࣻ .-0QT૑ࣘ੸೟ण « पઁ «

    ৘ஏ ª ஠஠য়ߛ௼"5.੉ਊѤࣻ .-0QT૑ࣘ੸೟ण $PODFQU%SJGUदр੉૑թীٮۄؘ੉ఠ੄ా҅੸ౠࢿ੉߸ೞחѪ
  8. ؀୹פૉझ௏য૑ࣘ੸೟ण $5 ੸ਊѾҗ .PEFM FWBMVBUJPO %BUB FYUSBDUJPO %BUB QSFQBSBUJPO .PEFM

    USBJOJOH .PEFM WBMJEBUJPO .-౵੉೐ۄੋ ೖ୊ࢤࢿ ఋѶчࢤࢿ ׹উ૑ USBJOUFTUؘ੉ఠࣇ ੐߬٬୊ܻ ݽ؛೟ण ݽ؛౵ੌ੷੢ ݽ؛ಣо NFUSJD੿ࠁ੷੢ ݽ؛Ѩૐ  005 0VUPG5JNF ӝઓݽ؛җ$5ݽ؛ࢿמ࠺Ү $5ݽ؛ࢿמݽפఠ݂ ѐਘ œ઱੄ $5ݽ؛੄Ѿҗо೦࢚જ਷Ѫ਷ইש ¬ $5ݽ؛੸ਊ «$5ݽ؛ «ӝઓݽ؛ ӝઓݽ؛җ$5ݽ؛ࢿמ࠺Ү $5ݽ؛ࢿמݽפఠ݂ ѐਘ
  9. .-0QTۄ੉࠳۞ܻ઱ਃӝמ Date Parameters  - DOU୭Ӕѐਘ࢚ಿѐࣻ  - HJKVO@ZNӝળ੉غחਘ .

     - UY@ZN@NPC.ਘ  - UY@ZN@NPC.ਘ 
 4&-&$5"DVTUPNFS@JE \HJKVO@ZN^BTHJKVO@ZN  $06/5 %*45*/$5CQSPEVDU@OBNF BTDOU '30.EFGBVMUPSEFST"-&'5065&3+0*/ EFGBVMUQSPEVDUT# 0/"QSPE@JE#QSPE@JE 8)&3&46#453 "PSEFS@EBUF   \UY@ZN@NPC^s "/%46#453 "PSEFS@EBUF   \UY@ZN@NPC^s "/%"PSEFS@JUFN@DPEFs "/%"PSEFS@TUBUVT ߓ࣠৮ܐ  (3061#:"DVTUPNFS@JE  - DOU୭Ӕѐਘ࢚ಿѐࣻ  - CBTF@ZNӝળ੉غחਘ .  - CG@N@ZN.ਘ  - CG@N@ZN.ਘ 
 4&-&$5"DVTUPNFS@JE r\CBTF@ZN^BTHJKVO@ZN  $06/5 %*45*/$5CQSPEVDU@OBNF BTDOU '30.EFGBVMUPSEFST"-&'5065&3+0*/ EFGBVMUQSPEVDUT# 0/"QSPE@JE#QSPE@JE 8)&3&46#453 "PSEFS@EBUF   r\CG@N@ZN^s "/%46#453 "PSEFS@EBUF   \CG@N@ZN^s "/%"PSEFS@JUFN@DPEFs "/%"PSEFS@TUBUVT ߓ࣠৮ܐ  (3061#:"DVTUPNFS@JE { "base_ym": "202210", "thmm_ym": "202210", "bf_1m_ym": "202209", "bf_2m_ym": "202208", "bf_3m_ym": "202207", "bf_4m_ym": "202206", "bf_5m_ym": "202205", "bf_1m_dt": "20220901", "bf_2m_dt": "20220801", "bf_3m_dt": "20220701", "af_1m_ym": "202211", "af_2m_ym": "202212", "af_3m_ym": "202301” … } .-0QT-JC
  10. .-0QTۄ੉࠳۞ܻ઱ਃӝמ Pandas Dataframe to Impala Table */4&35EFGBVMUVTFSJOGP 7"-6&4 rUPCZs rUPCZLs

    rIPNFUPCZs  VOBNF GVMMOBNF IEJS  BCSU FUDBCSU  BEN BEN WBSBEN  BWBIJ - BVUPJQE CJO WBSMJCBWBIJ  CJO CJO CJO  DMPVEFSB IPNFDMPVEFSB  DMPVEFSBj $MPVEFSB VTSMJCDMPVEFSB j pandas hue pandas hue
  11. def save_pandas_to_table(self, pandas_df: PandasDataFrame, table_name: str, table_creation: Optional[bool] = False)

    -> bool: 
 """ 
 Pandas DataFrame 데이터를 지정한 테이블에 적재합니다. 
 :param PandasDataFrame pandas_df: Pandas DataFrame 
 :param str table_name: 데이터를 적재할 테이블명 
 :param bool table_creation: 테이블 미존재 시 테이블을 생성합니다. (default = False) 
 
 >>> pandas_df 
 base_dt col1 col2 
 0 202206 1 2 
 1 202206 1 2 
 >>> fs.save_pandas_to_table(pandas_df=pandas_df, table_name='tmp.test_table_name') 
 or 
 >>> fs.save_pandas_to_table(pandas_df=pandas_df, table_name='tmp.test_table_name', table_creation=True) 
 
 .. note:: 데이터를 적재할 테이블이 존재할 경우 테이블과 Pandas Dataframe 컬럼명과 동일해야 합니다. 
 """ 
 if isinstance(pandas_df, PandasDataFrame): 
 if not self.meta_data.table_exists(table_name=table_name): 
 self._create_table_if_ture(pandas_df=pandas_df, table_name=table_name, table_creation= table_creation) 
 return self._save_pandas_to_table(pandas_df=pandas_df, table_name=table_name) 
 raise InputTypeException(f"Please input data in pandas dataframe type. type={type(pandas_df)}") .-0QTۄ੉࠳۞ܻ઱ਃӝמ Pandas Dataframe to Impala Table %PDTUSJOH ప੉࠶੗زࢤࢿ 0QUJPOBM
  12. { "responses": [ { "affiNo": "100000001", "affiCode": "CODE01", "score": 0.1182257

    }, { "affiNo": "100000002", "affiCode": "CODE02", "score": 0.61977172 }, … { "affiNo": "100000015", "affiCode": "CODE15", "score": 0.3917292 } ] } { "requests": [ { "customId": "00000000001", "affiNo": "100000001", "affiCode": "CODE01", "feature01": 5.0E7, "feature02": 3.0E7, "feature03": 1.0E3, "feature04": 60, "feature05": 10, "feature06": 2.0, "feature07": 0.0, "feature08": "N", "feature09": "Y", "feature10": "Y" }, … ] } पदр.-ࢲࡂࢲ࠺झѐߊ 추천 알고리즘 ML모델 Inference 과정 3FRVFTU 3FTQPOTF %BUB1SFQBSBUJPO ୭ઙѐ*OQVU ౵ࢤ߸ࣻ୶о ѐ ୭؀ ѐ Request Data Preparation ୭ઙѐ*OQVU ౵ࢤ߸ࣻ୶о Response ↟↟↟
  13. पदр.-ࢲࡂࢲ࠺झѐߊ BentoML 1차 성능 테스트 పझ౟ജ҃ 7.ࢲߡ $16DPSF .FNPSZH ؀

    ߮ష.-੄8PSLFSѐࣻ ௏যѐࣻ $16ࢎਊ۝_ 314 .FBO$OUT ૑োदр UIQDU _NT ࢿמపझ౟חѐౣ݂ (BUMJOH ਵ۽૓೯ೣ 314 3FRVFTUT1FS4FDPOE 1차 성능 테스트 결과
  14. पदр.-ࢲࡂࢲ࠺झѐߊ BentoML 2차 성능 테스트 పझ౟ജ҃ ର৬زੌ 7.ࢲߡ $16DPSF .FNPSZH

    ؀ ߮ష.-੄8PSLFSѐࣻ ௏যѐࣻ $16ࢎਊ۝_ 314 .FBO$OUT ૑োदр UIQDU _NT 2차 성능 테스트 결과 ѐࢶࢎ೦ 1ZUIPO௏٘ࢿמѐࢶ ؀ۚߓ 314 3FRVFTUT1FS4FDPOE
  15. 1..-WFSTJPO )FBEFSDPQZSJHIU,/*.& "QQMJDBUJPOOBNF,/*.&j )FBEFS %BUB'JFMEEBUB5ZQFTUSJOHu OBNF&NQMPZNFOUu PQUZQFDBUFHPSJDBM 7BMVFWBMVF1SJWBUF 7BMVFWBMVF$POTVMUBOU 7BMVFWBMVF4FMG&NQ

    7BMVFWBMVF14-PDBM j 7BMVFWBMVF/" 7BMVFWBMVF7PMVOUFFS %BUB'JFME j %BUB%JDUJPOBSZ j 1..- पदр.-ࢲࡂࢲ࠺झѐߊ PMML (Predictive Model Markup Language) ୹୊+BWB1..-"1* PMML File Sample
  16. पदр.-ࢲࡂࢲ࠺झѐߊ PMML 모델 생성 코드 (데이터 과학자) GSPNMJHIUHCNJNQPSU-(#.$MBTTJGJFS 
 GSPNTLMFBSOQNNMQJQFMJOFJNQPSU1..-1JQFMJOF

    
 GSPNTLMFBSOQNNMQSFQSPDFTTJOHMJHIUHCNJNQPSUNBLF@MJHIUHCN@EBUBGSBNF@NBQQFS 
 GSPNTLMFBSOQNNMJNQPSUTLMFBSOQNNM 
 
 tuu1..-౵੉೐ۄੋࢤࢿtuu 
 NBQQFSNBLF@MJHIUHCN@EBUBGSBNF@NBQQFS 9@UFTUEUZQFT NJTTJOH@WBMVF@BXBSF5SVF 
 DMBTTJGJFS-(#.$MBTTJGJFS QBSBNT GFBUVSF@OBNF9@USBJODPMVNOTUPMJTU j 
 QJQFMJOF1..-1JQFMJOF < rNBQQFSs NBQQFS  DMBTTJGJFS DMBTTJGJFS > 
 
 ݽ؛೟ण 
 QJQFMJOFGJU 9@USBJO :@USBJO DMBTTJGJFS@@FWBM@TFUFWBMT j 
 
 tuuݽ؛੷੢tuu 
 TLMFBSOQNNM QJQFMJOF GNPEFM@GJMF\NPEFM@GJMF@OBNF^
  17. पदр.-ࢲࡂࢲ࠺झѐߊ ML모델 서빙 애플리케이션 코드 (엔지니어) @Bean
 public Evaluator evaluator(ModelProperties

    properties) throws JAXBException, SAXException, IOException {
 File modelPmmlFile = new File(properties.getPmmlLocation());
 return new LoadingModelEvaluatorBuilder().setLocatable(false)
 .setVisitors(new DefaultModelEvaluatorBattery())
 .load(modelPmmlFile)
 .build();
 } private LinkLoanResponse applyModelScore(LinkEmergencyLoanFeature linkEmergencyLoanFeature) {
 Map<FieldName, ?> score = this.modelEvaluator.evaluate(linkEmergencyLoanFeature);
 return new LinkLoanResponse(linkEmergencyLoanFeature.getIntr_ltmn_no(), linkEmergencyLoanFeature.getLkl_inst_cd(), (Double) score.get(this.modelEvaluator.getProbability()));
 } PMML Model Loading PMML Model Evaluate
  18. पदр.-ࢲࡂࢲ࠺झѐߊ PMML 3차 성능 테스트 పझ౟ജ҃ 7.ࢲߡ $16DPSF .FNPSZH ؀

    ੉੹ࢿמపझ౟ח؀ైੑ $16ࢎਊ۝੉ೞ 314  .FBO$OUT ૑োदр UIQDU _NT 3차 성능 테스트 결과 ؀ۚߓ 314 3FRVFTUT1FS4FDPOE
  19. EoD