$30 off During Our Annual Pro Sale. View Details »

코끼리를 GPU에 넣는 법

kakao
December 09, 2022

코끼리를 GPU에 넣는 법

#Large-scale #Engineering

카카오브레인에서 경험한 라지스케일 모델 학습 최적화 사례를 소개합니다.

발표자 : monk.detective
카카오브레인 Large-Scale AI Studio의 Monk 입니다. 라지스케일 AI 를 실용적으로 사용하기 위해 항상 고민 중 입니다.

kakao

December 09, 2022
Tweet

More Decks by kakao

Other Decks in Programming

Transcript

  1. View Slide

  2. View Slide

  3. View Slide

  4. View Slide

  5. View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. View Slide

  12. View Slide

  13. View Slide

  14. View Slide

  15. View Slide

  16. View Slide

  17. View Slide

  18. View Slide

  19. View Slide

  20. View Slide

  21. View Slide

  22. View Slide

  23. View Slide

  24. View Slide

  25. View Slide

  26. # for intra-node FSDP
    for nr in range(args.nn):
    wr_list = [nr * args.np + lr for lr in range(args.np)]
    if nr == args.node_rank:
    sharding_group = dist.new_group(ranks=wr_list, backend=args.dist_backend)
    else: dist.new_group(ranks=wr_list, backend=args.dist_backend)
    # for inter-node PowerSGD
    for lr in range(args.np):
    wr_list = [nr * args.np + lr for nr in range(args.nn)]
    if lr == distenv.local_rank:
    averaging_group = dist.new_group(ranks=wr_list, backend=args.dist_backend)
    else:
    dist.new_group(ranks=wr_list, backend=args.dist_backend)

    View Slide

  27. View Slide

  28. View Slide

  29. View Slide

  30. View Slide

  31. View Slide

  32. View Slide

  33. View Slide

  34. View Slide

  35. View Slide