카카오클라우드 MLOps 활용 방안 소개

#빅데이터 #MLops #AI #Cloud

카카오클라우드 위에서 AI를 위한 데이터 수집부터 모델 개발, 학습, 튜닝, 서빙까지의 파이프라인 구성과 활용 방안에 대해 소개합니다.

발표자 : evan.ejin
카카오클라우드에서 Data 및 Machine Learning 플랫폼 PM을 담당하고 있는 Evan입니다.

Transcript

2. 
   .-0QT ۆ 
   .-0QT PO
   ,BLBPDMPVE 
   ഝਊ ৘ઁ
3. 
   .-0QT ۆ 
   .-0QT PO
   ,BLBPDMPVE 
   ഝਊ ৘ઁ
4. .BDIJOF
   -FBOJOH 0QFSBUJPOT ݠन ۞׬ ѐߊҗ ਍৔ਸ ই਋ܰח दझమ ߂ ೐۽ࣁझ ؘ੉ఠ੄ ন੉ ӝೞәࣻ੸ਵ۽ ૐоೞҊ ੓Ҋ ੉ী ٮܲ ੹୊ܻ ੿ઁ ؘ੉ఠ ߡ੹ ҙܻ ١੄ ੘স ࠂ੟بо ૐоೞҊ ੓਺ ٩۞׬ਸ ನೣೠ ࠂ੟ೠ ݽ؛੉ ݆੉ ഝਊغݶࢲ ള۲ ಣо పझ౟ ӒܻҊ ߓನ җ੿੄ ࠂ੟بо ૐоೞҊ ੓਺ (16
   /16
   516
   ١੄ Ҋࢿמ ஹೊ౴ ੗ਗ੉ ݆੉ ഝਊغݶࢲ ੉ ੗ਗਸ ୭੸ച೧ ࢎਊೡ ࣻ ੓ח ਃҳࢎ೦੉ ૐоೞҊ ੓਺ ؘ੉ఠ ࠂ੟ࢿ ૐо ݽ؛੄ ࠂ੟ࢿ ૐо ஹೊ౴ ੗ਗ ୭੸ച .-0QT ۆ
5. %BUB0QT .BDIJOF
   -FBSOJOH
   'MPX ؘ੉ఠ ࣻ૘ ؘ੉ఠ ੹୊ܻ ؘ੉ఠ ߡ੷׬ ؘ੉ఠ ੿ઁ ؘ੉ఠ ߸ജ %BUB
   -BLF 1JQFMJOF ݽ؛ ѐߊ ݽ؛ ೟ण ೞ੉ಌ౵ۄ޷ఠ ౚ׬ ݽ؛ ಣо ݽ؛ ߓನ 
   ࢲࡂ ࢿמ ݽפఠ݂ সؘ੉౟ Ѿ੿ ݽ؛ ߡ੹ ҙܻ 1JQFMJOF .-0QT ۆ
6. 
   .-0QT ۆ 
   .-0QT PO
   ,BLBPDMPVE 
   ഝਊ ৘ઁ
7. .-0QTܳ *OHFTUJPO
   1SFQBSBUJPO
   "OBMZUJDT
   .- ױ҅۽ ա׀׮ݶ %BUB
   *OHFTUJPO 1SFQBSBUJPO "OBMZUJDT 
   .- .-0QT PO
   ,BLBPDMPVE
8. %BUB
   *OHFTUJPO 1SFQBSBUJPO 데이터수집및 적재 Infrastructure as a Service Management 1VC4VC #FZPOE
   4UPSBHF
   4FSWJDF 데이터 ETL )BEPPQ
   &DP %BUB
   $BUBMPH 데이터분석및 학습 .BOBHFE
   ,BGLB #FZPOE
   $PNQVUJOH
   4FSWJDF #FZPOE
   /FUXPSLJOH
   4FSWJDF "OBMZUJDT 
   .- *". .POJUPSJOH $MPVE
   5SBJM "MFSU
   $FOUFS .POJUPSJOH
   'MPX ,VCFGMPX %BUB
   2VFSZ .-0QTܳ *OHFTUJPO
   1SFQBSBUJPO
   "OBMZUJDT
   .- ױ҅۽ ա׀׮ݶ .-0QT PO
   ,BLBPDMPVE
9. Object Storage Pub/Sub Managed Kafka Hadoop Eco Data Catalog Data Query ؘ੉ఠ ੋࢎ੉౟ 2 
   *OHFTUJPO 
   1SFQBSBUJPO 
   "OBMZUJDT 3 Data 1 )VF 4VQFSTFU ;FQQFMJO ഝਊ ࢲ࠺झ ߂ ো҅ 'MPX %BUB
   *OHFTUJPO
   1SFQBSBUJPO
   "OBMZUJDT
   ױ҅ ૑ਗ 'MPX .-0QT PO
   ,BLBPDMPVE
10. +VQZUFS /PUFCPPL 5SBJOJOH
    0QFSBUPS ,BUJC ,4FSWF .FUBEBUB 1JQFMJOF ,VCFGMPX $MVTUFS (16
    
    $16
    4FSWFS
    /PEFT ܻࣗझ ௪ఠ ҙܻ ࢎਊ੗Ӓܛ ӂ ೠ ҙܻ (16
    .JH
    ӝמ ঘ࣌ ۽Ӓ ҙܻ ੗ਗ ݽפఠ݂ ӝמ ,BLBP$MPVE
    ,VCFGMPX • ਕ௼೒۽਋ ױ҅߹ ਬਊೠ ో ߂ ౵੉೐ۄੋ ઁҕ • ஠஠য়௿ۄ਋٘ ௑ࣛ উীࢲ औҊ ࡅܰѱ ҳࢿ оמ • ੗ਗ ௪ఠ ӂೠ ҙܻ ݽפఠ݂ ١ ݒפ૚ ӝמ ઁҕ ,VCFGMPX ౠ૚ ,BLBP$MPVE ,VCFGMPX੄ ׮নೠ ஹನք౟ٜਸ ഝਊೞৈ .BDIJOF
    -FBSOJOH
    'MPXܳ ૑ਗೣ .-0QT PO
    ,BLBPDMPVE
11. 
    .-0QT ۆ 
    .-0QT PO
    ,BLBPDMPVE 
    ഝਊ ৘ઁ
12. -#
    "DDFTT
    -PH
    ؘ੉ఠܳ ഝਊೠ ౟ې೗ ৘ஏ ݽ؛ ҳഅਸ ా೧ दझమ ੉࢚ ૚റܳ पदрਵ۽ ఐ૑ೞҊ ؀਽ ࣘبܳ ೱ࢚ दఃҊ੗ ೣ ୶૓ ߓ҃
13. ౟ې೗ ৘ஏ ݽ؛ ҳഅ 'MPX -#
    ؘ੉ఠ ਃ୒ ߂ ਽׹ ۽Ӓ ౟ې೗ ੿ࠁ ী۞ ߂ ࢚క ௏٘ ֎౟ਕ௼ ࢿמ ݫ౟ܼ ؘ੉ఠ ੹୊ܻ ݽ؛ ѐߊ ݽ؛ ߓನ %BUB$BUBMPH )BEPPQ&DP /PUFCPPL 5FOTPSCPBSE *OHFTUJPO 1SFQBSBUJPO .BDIJOF
    -FBSOJOH ೞ੉ಌ ౵ۄ޷ఠ ౚ׬ ,4FSWF ,BUJC ഝਊ ৘ઁ
14. *OHFTUJPO 1SFQBSBUJPO .BDIJOF
    -FBSOJOH Load Balancer Object Storage Hadoop Eco Load & ETL Managed Kafka Pub/Sub Hadoop Eco Kubeflow ML CI/CD Test&Analyze Train Deploy Pre-Process Dashboard Data Catalog Hadoop Eco Serving ഝਊ ৘ઁ ݽ؛ ҳഅ 'MPX
    ߹ ഝਊ ࢲ࠺झ
15. ݽ؛ ҳഅ 'MPX
    ߹ ഝਊ ࢲ࠺झ -PBE
    #BMBODFS 0CKFDU
    4UPSBHF )BEPPQ
    &DP -PBE
    
    &5- .BOBHFE
    ,BGLB 1VC4VC )BEPPQ
    &DP ,VCFGMPX .-
    $*$% 5FTU"OBMZ[F 5SBJO %FQMPZ 1SF1SPDFTT Dashboard %BUB
    $BUBMPH )BEPPQ
    &DP 4FSWJOH 2 1 *OHFTUJPO 1SFQBSBUJPO .BDIJOF
    -FBSOJOH ഝਊ ৘ઁ
16. ؘ੉ఠ ࣻ૘ 0CKFDU
    4UPSBHF ؘ੉ఠ ੿ઁ %BUB
    $BUBMPH *OHFTUJPO
    ª 1SFQBSBUJPO
    
17. parser = argparse.ArgumentParser() parser.add_argument("--end_time", type=str, required=True, help="End time in 'yyyy/MM/dd HH:mm:ss' format") args = parser.parse_args() time_format = "%Y-%m-%d %H:%M:%S" end_time = args.end_time start_time = (datetime.strptime(end_time, time_format) - timedelta(minutes=30)).strftime(time_format) spark.sql("USE kclogs") df = spark.sql(""” SELECT FROM_UNIXTIME(UNIX_TIMESTAMP(`time`, 'yyyy/MM/dd HH:mm:ss:SS')) AS `time`, client_port AS source, REGEXP_EXTRACT(client_port, '(.*?)(?=:)') AS source_ip, REGEXP_EXTRACT(client_port, ':(.*)') AS source_port, target_port AS destination, REGEXP_EXTRACT(target_port, '(.*?)(?=:)') AS destination_ip, REGEXP_EXTRACT(target_port, ':(.*)') AS destination_port FROM tb_alb UNION ALL SELECT FROM_UNIXTIME(UNIX_TIMESTAMP(`time`, 'yyyy/MM/dd HH:mm:ss')) AS `time`, client_port AS source, REGEXP_EXTRACT(client_port, '(.*?)(?=:)') AS source_ip, REGEXP_EXTRACT(client_port, ':(.*)') AS source_port, destination_port AS destination, REGEXP_EXTRACT(destination_port, '(.*?)(?=:)') AS destination_ip, REGEXP_EXTRACT(destination_port, ':(.*)') AS destination_port FROM tb_nlb """) filtered_df = df.filter(F.col("time").between(start_time, end_time)) 1SFQBSBUJPO
    ª %BTICPBSE &YUSBDU
    5SBOTGPSN
    -PBE
    4QBSL ؘ੉ఠ ੸੤ %SVJE
18. %BTICPBSE
    4VQFSTFU 1SFQBSBUJPO
    ª %BTICPBSE
19. .BDIJOF
    -FBSOJOH $POUJOVPVT
    5SBJOJOH
    XJUI
    ,'1 5VOOJOH
    XJUI
    ,BUJC
20. .BDIJOF
    -FBSOJOH $POUJOVPVT
    EFQMPZNFOU
    XJUI
    ,TFSWF *OGFSFODF
    XJUI
    ,'1
21. %BTICPBSE
22. /FYU
23. 2"
24. &0%