사진편집기! 이것DOM 적용해볼GL?

#WebGL

새로운 사진편집기에 WebGL을 적용하고, 공용 컴포넌트를 개발하면서 마주했던 선택과 문제 해결과정을 공유합니다.

발표자 : sky.blue
더 멋진 카카오의 에디터를 위해 노력합니다. 카카오 FE플랫폼팀 스카이입니다.

Transcript

1. Copyright 2022. Kakao Corp. All rights reserved. Redistribution or public display is not permitted without written permission from Kakao.
   ! 사진편집기! 이것DOM 적용해볼GL?
   황선준 Sky.blue
   카카오
   if(kakao)2022
   

   View Slide

2. 1. ! 사진편집기에 WebGL 부어보기
   2. " DOM딱 DOM딱 아이템 붙이기
   

   View Slide

3. ! 사진편집기에 WebGL 부어보기
   

   View Slide

4. Who?
   “누구십니까?”
   

   View Slide

5. Brunch
   Daum cafe Kakao story
   

   View Slide

6. View Slide

7. ! 사진편집기!
   - 글쓰기에서 제공하는 사진 편집 기능
   - 자르기 / 크기조절
   - 보정 / 필터
   - 텍스트 / 이미지 추가
   

   View Slide

8. ! 사진편집기!
   

   View Slide

9. ! 사진편집기!
   

   View Slide

10. Re: ! 사진편집기에 WebGL 부어보기
    

    View Slide

11. Why?
    “왜 붓게 되었을까?”
    

    View Slide

12. API
    # 옛날 옛날엔..
    

    View Slide

13. (축) API 종료
    # 옛날 옛날엔..
    $
    

    View Slide

14. 능력자 절찬 모집중
    # 옛날 옛날엔..
    %
    

    View Slide

15. 여기가 맞나요?
    # 옛날 옛날엔..
    

    View Slide

16. &' 새친구후보 WebGL
    - Web Graphics Library
    - OpenGL ES 2.0 기반
    - GPU 사용
    - JS 개발 지원
    - 대부분의 모던 브라우저 지원 ( IE 11 지원 )
    

    View Slide

17. (
    “알아보자”
    

    View Slide

18. How to?
    “어떻게 사용하는 걸까??”
    

    View Slide

19. ❓ WebGL
    F
    Image
    V
    User Canvas WebGL
    create
    input
    create
    create
    input
    render
    T
    Input
    

    View Slide

20. ❓ WebGL
    F
    Image
    V
    User Canvas WebGL
    create
    input
    create
    create
    input
    render
    T
    Input
    

    View Slide

21. ❓ WebGL
    F
    Image
    V
    User Canvas WebGL
    input
    create
    create
    input
    render
    T
    Input
    create
    

    View Slide

22. ❓ WebGL
    F
    Image
    V
    User Canvas WebGL
    input
    create
    create
    input
    render
    T
    Input
    create
    

    View Slide

23. ❓ WebGL
    F
    Image
    V
    User Canvas
    create
    input
    create
    create
    input
    render
    T
    Input
    WebGL
    

    View Slide

24. F
    V
    Vertex Fragment
    T
    Texture
    

    View Slide

25. F
    V
    좌표 Fragment
    T
    Texture
    

    View Slide

26. F
    V
    좌표 색칠
    T
    Texture
    

    View Slide

27. F
    V
    좌표 색칠
    T
    데이터
    

    View Slide

28. ❓ WebGL
    Vertex Shader Fragment Shader
    Original Clipspace
    Vertex Shader
    attribute vec3 a_position;
    uniform mat4 u_matrix;
    void main() {
    gl_Position = u_matrix * a_position
    }
    

    View Slide

29. ❓ WebGL
    Vertex Shader Fragment Shader
    Original Clipspace
    Vertex Shader
    attribute vec3 a_position;
    uniform mat4 u_matrix;
    void main() {
    gl_Position = u_matrix * a_position
    }
    

    View Slide

30. ❓ WebGL
    Vertex Shader Fragment Shader
    Original Clipspace
    Vertex Shader
    attribute vec3 a_position;
    uniform mat4 u_matrix;
    void main() {
    gl_Position = u_matrix * a_position
    }
    

    View Slide

31. ❓ WebGL
    Vertex Shader Fragment Shader
    Original Clipspace
    Vertex Shader
    attribute vec3 a_position;
    uniform mat4 u_matrix;
    void main() {
    gl_Position = u_matrix * a_position
    }
    

    View Slide

32. ❓ WebGL
    Image Filter Result
    

    View Slide

33. ❓ WebGL
    Image Filter Result
    

    View Slide

34. ❓ WebGL
    Image Filter Result
    

    View Slide

35. ⏰ 얼마나 빠르지?
    

    View Slide

36. ⏰ 속도?
    

    View Slide

37. ⏰ 속도?
    정보
    OS macOS Monterey ( 12.5.1 )
    CPU Intel core i9 2.4GHz 8 core
    Graphic ADM Radeon PRO 5300M 4 GB
    Memory 32GB 2667 MHz DDR4
    

    View Slide

38. API ( ms ) WebGL ( ms ) 차이 ( % )
    100 1191.0 1216.7 -2.1
    200 1212.2 1230.5 -1.5
    300 1391.9 1203.1 13.6
    400 1455.4 1267.9 12.9
    500 1826.0 1400.2 23.3
    600 1874.9 1310.6 30.1
    700 1684.9 1580.2 6.2
    800 2189.9 1874.1 22.3
    900 2412.7 1714.4 28.9
    1000 2673.8 1899.4 29.0
    1100 2746.3 2007.3 26.9
    1200 2835.3 2279.5 19.6
    1300 3449.1 2072.0 39.9
    1400 3677.4 2169.7 41.0
    1500 3324.3 2839.0 14.6
    1600 3558.4 2430.2 31.7
    1700 3518.8 2913.2 17.2
    1800 5277.7 2940.4 44.3
    1900 5206.2 3096.6 40.5
    2000 5128.1 3784.9 26.2
    

    View Slide

39. API ( ms ) WebGL ( ms ) 차이 ( % )
    100 1191.0 1216.7 -2.1
    200 1212.2 1230.5 -1.5
    300 1391.9 1203.1 13.6
    400 1455.4 1267.9 12.9
    500 1826.0 1400.2 23.3
    600 1874.9 1310.6 30.1
    700 1684.9 1580.2 6.2
    800 2189.9 1874.1 22.3
    900 2412.7 1714.4 28.9
    1000 2673.8 1899.4 29.0
    1100 2746.3 2007.3 26.9
    1200 2835.3 2279.5 19.6
    1300 3449.1 2072.0 39.9
    1400 3677.4 2169.7 41.0
    1500 3324.3 2839.0 14.6
    1600 3558.4 2430.2 31.7
    1700 3518.8 2913.2 17.2
    1800 5277.7 2940.4 44.3
    1900 5206.2 3096.6 40.5
    2000 5128.1 3784.9 26.2
    

    View Slide

40. 0
    3000
    6000
    100 300 500 700 900 1100 1300 1500 1700 1900
    ⏰ 속도?
    API
    WebGL
    임의 필터에 대한 10회 실행시간 누적합 ( disabled cache, ms )
    개발환경에 따라 결과가 상이할 수 있습니다
    

    View Slide

41. +
    “그의 손에 쥐어지는..”
    

    View Slide

42. What?
    “무엇이 바뀌었을까요??”
    

    View Slide

43. 모자이크
    효과
    

    View Slide

44. , WebGL 활약
    

    View Slide

45. , WebGL 활약
    

    View Slide

46. , WebGL 활약
    

    View Slide

47. , WebGL 활약
    

    View Slide

48. -
    “짝짝짝”
    

    View Slide

49. 1. ! 사진편집기에 WebGL 부어보기
    2. " DOM딱 DOM딱 아이템 붙이기
    

    View Slide

50. .
    “어떤걸 선택하지?”
    

    View Slide

51. Canvas
    

    View Slide

52. Item
    

    View Slide

53. DOM Canvas
    편의성 일관성
    

    View Slide

54. / 선택지
    DOM Canvas
    디버깅 난이도 낮음 높음
    이벤트 처리 난이도 낮음 높음
    CSS 사이드 이펙트 있음 없음
    History - Canvas 동기화 있음 없음
    공간 제약 없음 있음
    

    View Slide

55. 0
    “여정의 시작”
    

    View Slide

56. ὥ
    “희망편”
    

    View Slide

57. Dev Easy
    빠르고 쉬운 개발 편리한 사용 방법
    

    View Slide

58. 1 희노애락
    const Decorator = (props: Props) => {
    // ...states
    // ... events
    // ... react hooks
    return
    
    {children}
    
    
    
    
    
    
    ;
    };
    export default Decorator;
    

    View Slide

59. 1 희노애락
    Components Decorator
    const Decorator = (props: Props) => {
    // ...states
    // ... events
    // ... react hooks
    return
    
    {children}
    
    
    
    
    
    
    ;
    };
    export default Decorator;
    const ItemComponents = () => {
    const [state, setState] = useState();
    return
    Good
    
    }
    

    View Slide

60. 1 희노애락
    Components Decorator
    interface Props {
    moveable: boolean;
    resizable: boolean;
    removeable: boolean;
    rotatable: boolean;
    size?: Size;
    zoom?: number;
    degree?: number;
    position?: Position;
    fixRatio?: boolean;
    hidden?: boolean;
    onChange?: DecoratorOnChangeCallback;
    removeCallback?: DecoratorRemoveCallback;
    }
    const ItemComponents = () => {
    const [state, setState] = useState();
    return moveable={true}
    resizable={false}
    removeable={false}
    rotatable={true}>
    Good
    
    }
    

    View Slide

61. .
    “너무 과하지 않나?”
    

    View Slide

62. history preview
    menu
    1 희노애락
    update
    notify
    update render
    item
    

    View Slide

63. history preview
    menu
    1 희노애락
    update
    notify
    update render
    item
    

    View Slide

64. history preview
    menu
    1 희노애락
    update
    notify
    update render
    item
    

    View Slide

65. ,
    “LGTM”
    

    View Slide

66. 2
    “불편해..”
    

    View Slide

67. 3
    ”코드 중복”
    

    View Slide

68. 1 희노애락
    const Item = (props: Props) => {
    const dispatch = useDispatch();
    const [state, setState] = useState();
    const onMoveCallback = useCallback(() => { /* .. */ }, []);
    const onResizeCallback = useCallback(() => { /* .. */ }, []);
    const onRemoveCallback = useCallback(() => { /* .. */ }, []);
    const onRotateCallback = useCallback(() => { /* .. */ }, []);
    return
    
    
    /* .. пઙ ೩ٜ۞ .. */
    
    ;
    };
    export default Item;
    

    View Slide

69. 1 희노애락
    스티커 텍스트 서명
    const Sticker = (props: Props) => {
    const dispatch = useDispatch();
    const [state, setState] = useState();
    const onMoveCallback
    = useCallback(() => { /* .. */ }, []);
    const onResizeCallback
    = useCallback(() => { /* .. */ }, []);
    const onRemoveCallback
    = useCallback(() => { /* .. */ }, []);
    const onRotateCallback
    = useCallback(() => { /* .. */ }, []);
    return
    
    
    
    
    
    
    
    ;
    };
    export default Sticker;
    const Text = (props: Props) => {
    const dispatch = useDispatch();
    const [state, setState] = useState();
    const onMoveCallback
    = useCallback(() => { /* .. */ }, []);
    const onResizeCallback
    = useCallback(() => { /* .. */ }, []);
    const onRemoveCallback
    = useCallback(() => { /* .. */ }, []);
    const onRotateCallback
    = useCallback(() => { /* .. */ }, []);
    return
    
    
    
    
    
    
    
    ;
    };
    export default Text;
    const Sign = (props: Props) => {
    const dispatch = useDispatch();
    const [state, setState] = useState();
    const onMoveCallback
    = useCallback(() => { /* .. */ }, []);
    const onResizeCallback
    = useCallback(() => { /* .. */ }, []);
    return
    
    
    
    
    
    ;
    };
    export default Sign;
    

    View Slide

70. 1 희노애락
    스티커 텍스트 서명
    const Sticker = (props: Props) => {
    const dispatch = useDispatch();
    const [state, setState] = useState();
    const onMoveCallback
    = useCallback(() => { /* .. */ }, []);
    const onResizeCallback
    = useCallback(() => { /* .. */ }, []);
    const onRemoveCallback
    = useCallback(() => { /* .. */ }, []);
    const onRotateCallback
    = useCallback(() => { /* .. */ }, []);
    return
    
    
    
    
    
    
    
    ;
    };
    export default Sticker;
    const Text = (props: Props) => {
    const dispatch = useDispatch();
    const [state, setState] = useState();
    const onMoveCallback
    = useCallback(() => { /* .. */ }, []);
    const onResizeCallback
    = useCallback(() => { /* .. */ }, []);
    const onRemoveCallback
    = useCallback(() => { /* .. */ }, []);
    const onRotateCallback
    = useCallback(() => { /* .. */ }, []);
    return
    
    
    
    
    
    
    
    ;
    };
    export default Text;
    const Sign = (props: Props) => {
    const dispatch = useDispatch();
    const [state, setState] = useState();
    const onMoveCallback
    = useCallback(() => { /* .. */ }, []);
    const onResizeCallback
    = useCallback(() => { /* .. */ }, []);
    return
    
    
    
    
    
    ;
    };
    export default Sign;
    

    View Slide

71. 4
    “으 냄새”
    

    View Slide

72. 5
    ”관리 중복”
    

    View Slide

73. history preview
    menu
    1 희노애락
    update
    notify
    update render
    item
    

    View Slide

74. preview
    item
    menu
    1 희노애락
    notify
    update render
    update
    history
    

    View Slide

75. 6
    “어라 왜이렇게 작동하지”
    

    View Slide

76. 7
    “᤼ᔥᤍ᪫”
    

    View Slide

77. const Decorator = (props: Props) => {
    const [box, setBox] = useState();
    const onMoveCallback = useCallback(() => { /* .. */ }, []);
    const onResizeCallback = useCallback(() => { /* .. */ }, []);
    const onRemoveCallback = useCallback(() => { /* .. */ }, []);
    const onRotateCallback = useCallback(() => { /* .. */ }, []);
    const onMoveStop = useCallback(() => { onChange(nextPosition) }, []);
    const onResizeStop = useCallback(() => { onChange(nextSize) }, []);
    const onRotateStop = useCallback(() => { onChange(nextAngle) }, []);
    return
    { children }
    
    { props.moveable && }
    { props.resizeable && }
    { props.removable && }
    { props.rotatable && }
    
    ;
    };
    export default Decorator;
    1 희노애락
    

    View Slide

78. const Decorator = (props: Props) => {
    const [box, setBox] = useState();
    const onMoveCallback = useCallback(() => { /* .. */ }, []);
    const onResizeCallback = useCallback(() => { /* .. */ }, []);
    const onRemoveCallback = useCallback(() => { /* .. */ }, []);
    const onRotateCallback = useCallback(() => { /* .. */ }, []);
    const onMoveStop = useCallback(() => { onChange(nextPosition) }, []);
    const onResizeStop = useCallback(() => { onChange(nextSize) }, []);
    const onRotateStop = useCallback(() => { onChange(nextAngle) }, []);
    return
    { children }
    
    { props.moveable && }
    { props.resizeable && }
    { props.removable && }
    { props.rotatable && }
    
    ;
    };
    export default Decorator;
    1 희노애락
    

    View Slide

79. 1 희노애락
    Component Props
    interface Props {
    moveable: boolean;
    resizable: boolean;
    removeable: boolean;
    rotatable: boolean;
    size?: Size;
    zoom?: number;
    degree?: number;
    position?: Position;
    fixRatio?: boolean;
    hidden?: boolean;
    onChange?: DecoratorOnChangeCallback;
    removeCallback?:
    DecoratorRemoveCallback;
    }
    const Decorator = (props: Props) => {
    const [box, setBox] = useState();
    const onMoveCallback = useCallback(() => { /* .. */ }, []);
    const onResizeCallback = useCallback(() => { /* .. */ }, []);
    const onRemoveCallback = useCallback(() => { /* .. */ }, []);
    const onRotateCallback = useCallback(() => { /* .. */ }, []);
    const onMoveStop = useCallback(() => { onChange(nextPosition) },
    []);
    const onResizeStop = useCallback(() => { onChange(nextSize) }, []);
    const onRotateStop = useCallback(() => { onChange(nextAngle) }, []);
    return
    { children }
    
    { props.moveable && }
    { props.resizeable && }
    { props.removable && }
    { props.rotatable && }
    
    ;
    };
    export default Decorator;
    

    View Slide

80. history preview
    item
    menu
    1 희노애락
    notify
    update render
    update
    

    View Slide

81. history preview
    item
    menu
    1 희노애락
    notify
    update render
    update
    decorator
    update
    update
    

    View Slide

82. 1 희노애락
    

    View Slide

83. ၌ᛑ
    “절망편”
    

    View Slide

84. 8
    “장미와 가시”
    

    View Slide

85. / 선택지
    DOM Canvas
    디버깅 난이도 낮음 높음
    이벤트 처리 난이도 낮음 높음
    CSS 사이드 이펙트 있음 없음
    History - Canvas 동기화 있음 없음
    공간 제약 없음 있음
    

    View Slide

86. 9
    “Sync”
    

    View Slide

87. addon
    attacher
    resize/crop
    processing
    history adapter
    text / image
    addon
    image
    :; 희노애락
    canvas
    processor
    

    View Slide

88. addon
    attacher
    resize/crop
    processing
    history adapter
    text / image
    addon
    image
    :; 희노애락
    canvas
    processor
    

    View Slide

89. addon
    attacher
    resize/crop
    processing
    history adapter
    text / image
    addon
    image
    :; 희노애락
    canvas
    processor
    

    View Slide

90. :; 희노애락
    export const calcAbsPosition = (
    position: Position,
    itemSize: Size,
    imgSize: Size
    ): Coordinate => {
    // ӝળ੼੉ হ׮ === left-top
    if (!position.centerPoint) return {x: position.x, y: position.y};
    // ӝળ੼੉ য়ܲଃੋ҃਋ => left۽ ߸҃
    const x = position.centerPoint[0] === 'l' ? position.x : imgSize.width -
    position.x - itemSize.width;
    // ӝળ੼੉ ইېੋ҃਋ => topਵ۽ ߸҃
    const y = position.centerPoint[1] === 't' ? position.y : imgSize.height -
    position.y - itemSize.height;
    return {x, y};
    };
    

    View Slide

91. addon
    attacher
    resize/crop
    processing
    history adapter
    text / image
    addon
    image
    :; 희노애락
    canvas
    processor
    

    View Slide

92. :; 희노애락
    const x = (position?.x || 0) + (flip?.flipV ? width : 0);
    const y = (position?.y || 0) + (flip?.flipH ? height : 0);
    const flipWidth = flip?.flipV ? -width : width;
    const flipHeight = flip?.flipH ? -height : height;
    const font = textStyle?.font || '';
    const fontSize = textStyle?.size || DEFAULT_TEXT_SIZE;
    const fontHeight = fontSize * lineHeight;
    const align = textStyle?.align || 'left';
    const textTop = ((fontHeight - fontSize) >> 1);
    const foreColor = color2String(textStyle?.textColor);
    const backColor = color2String(textStyle?.backgroundColor);
    const lines = content.split('\n');
    

    View Slide

93. ctx.save();
    ctx.translate(x, y);
    ctx.globalAlpha = (transparent || PERCENT) / PERCENT;
    if (position?.rotate) this.rotateCanvas(ctx, position.rotate, flipWidth, flipHeight);
    if (flip?.flipV || flip?.flipH) ctx.scale(flip.flipV ? -1 : 1, flip.flipH ? -1 : 1);
    // Text Background
    ctx.fillStyle = backColor;
    ctx.fillRect(0, 0, width, height);
    // Text
    ctx.textBaseline = 'top';
    ctx.font = toFontString(fontSize, font, textStyle?.bold, textStyle?.italic);
    ctx.fillStyle = foreColor;
    lines.forEach((line, idx) => {
    const lineY = textTop + (idx * fontHeight);
    const lineWidth = ctx.measureText(line).width;
    const alignPadding = align === 'left' ? 0 : (align === 'center' ? (width - lineWidth) >> 1 : (width -
    lineWidth));
    ctx.fillText(line, alignPadding, lineY);
    // Text Underline
    if (textStyle?.underline) {
    const underlineHeight = Math.round(fontSize / FONT_BASE);
    const underlineY = lineY + fontSize - underlineHeight;
    ctx.beginPath();
    ctx.lineWidth = underlineHeight;
    ctx.strokeStyle = foreColor;
    ctx.moveTo(alignPadding, underlineY);
    ctx.lineTo(alignPadding + lineWidth, underlineY);
    ctx.stroke();
    }
    });
    

    View Slide

94. addon
    attacher
    resize/crop
    processing
    history adapter
    text / image
    addon
    image
    :; 희노애락
    canvas
    processor
    

    View Slide

95. <
    “EZ”
    

    View Slide

96. =
    “나비효과”
    

    View Slide

97. :; 희노애락
    

    View Slide

98. :; 희노애락
    const Preview = () => {
    return
    
    
    
    
    
    }
    

    View Slide

99. :; 희노애락
    const Preview = () => {
    return
    <>
    
    
    
    
    
    
    
    
    
    
    
    >
    }
    

    View Slide

100. :; 희노애락
     

     View Slide

101. :; 희노애락
     

     View Slide

102. :; 희노애락
     

     View Slide

103. :; 희노애락
     

     View Slide

104. >
     “?”
     

     View Slide

105. :; 희노애락
     

     View Slide

106. :; 희노애락
     

     View Slide

107. :; 희노애락
     

     View Slide

108. export const pos2CropPos = (item: Rectangle & {rotate?: number}, crop: Crop) => {
     // ੹୊ܻ ߂ nullish ୊ܻ
     const cropCtr = {x: crop.centerX, y: crop.centerY};
     const cropAxis = {x: crop.width >> 1, y: crop.height >> 1};
     const cropBoxLt = cropBoxLtPosition(crop);
     const angle = {item: item.rotate ?? 0, crop: crop.angle * ((crop.flipH + crop.flipV) % 2 ? -1 : 1)};
     const resRotate = (FULL_ANGLE + angle.item - (crop.rotate * QUATER_ANGLE + angle.crop)) %
     FULL_ANGLE;
     // பߡझী ࠁৈ૑ח Cropbox੄ Left-top
     const rtCropBoxLt = reverseRotateDeg(cropBoxLt, crop.angle, cropCtr);
     // pos ӝળ item ઺बઝ಴ ୶୹
     const itemCtr = {x: item.x + (item.width >> 1), y: item.y + (item.height >> 1)};
     const {x, y} = flipOnAxis(itemCtr, cropAxis, {flipV: !!crop.flipV, flipH: !!crop.flipH});
     const r = (crop.angle + crop.rotate * QUATER_ANGLE) / HALF_ANGLE * Math.PI;
     const a = x * Math.cos(r) + y * Math.sin(r);
     const b = x * Math.sin(r) - y * Math.cos(r);
     // pos ઝ಴҅ীࢲ CropPos ઝ಴҅۽ ੉ز
     const cropItemCtr = {x: rtCropBoxLt.x + a, y: rtCropBoxLt.y - b};
     const ltCorr = {x: item.width >> 1, y: item.height >> 1};
     const cropLt = {x: cropItemCtr.x - ltCorr.x, y: cropItemCtr.y - ltCorr.y};
     const result = {x: cropLt.x, y: cropLt.y, rotate: resRotate};
     return result;
     };
     

     View Slide

109. ?
     “이걸 이렇게 오래?”
     

     View Slide

110. :; 희노애락
     export type Item = {
     id: number;
     src: string;
     width: number;
     height: number;
     custom: boolean;
     // for preview and canvas
     position: AnglePosition;
     flip: Flip;
     // for crop menu
     cropPosition: AnglePosition;
     cropFlip: Flip;
     };
     

     View Slide

111. @
     “짠”
     

     View Slide

112. ᚨ
     “즐겨~”
     

     View Slide

113. A
     “많 관 부”
     

     View Slide

114. 참고 자료 & 이미지 출처
     1 ) Web GL ӝୡ https://webglfundamentals.org/webgl/lessons/ko/webgl-fundamentals.html
     P 14, 15) https://commons.wikimedia.org/wiki/File:WebGL_Logo.svg?uselang=ko
     

     View Slide

115. E.O.D
     

     View Slide