코끼리를 GPU에 넣는 법

#Large-scale #Engineering

카카오브레인에서 경험한 라지스케일 모델 학습 최적화 사례를 소개합니다.

발표자 : monk.detective
카카오브레인 Large-Scale AI Studio의 Monk 입니다. 라지스케일 AI 를 실용적으로 사용하기 위해 항상 고민 중 입니다.

Transcript

1. View Slide

2. View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. View Slide

18. View Slide

19. View Slide

20. View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. # for intra-node FSDP
    for nr in range(args.nn):
    wr_list = [nr * args.np + lr for lr in range(args.np)]
    if nr == args.node_rank:
    sharding_group = dist.new_group(ranks=wr_list, backend=args.dist_backend)
    else: dist.new_group(ranks=wr_list, backend=args.dist_backend)
    # for inter-node PowerSGD
    for lr in range(args.np):
    wr_list = [nr * args.np + lr for nr in range(args.nn)]
    if lr == distenv.local_rank:
    averaging_group = dist.new_group(ranks=wr_list, backend=args.dist_backend)
    else:
    dist.new_group(ranks=wr_list, backend=args.dist_backend)
    

    View Slide

27. View Slide

28. View Slide

29. View Slide

30. View Slide

31. View Slide

32. View Slide

33. View Slide

34. View Slide

35. View Slide