Online Dance Lesson Support System Using Flipped Classroom Shuhei Tsuchida†, Daichi Shimizu†, Kanako Shibasaki‡, 
 Tsutomu Terada†, Masahiko Tsukamoto† † Kobe University ‡ Sports Academy Tsuchiura 1 11/28-30/2022 MoMM202

Using video distribution systems or Video conferencing systems Di ff i culty Detailed instruction, communication, physical learning Flexibility Easy to attend (distance, time) Easy to share information
 Di ffi cult to simply apply the previous methods used…? “I miss being able to discuss with other students.” “It would have been helpful to have chats and shared sheets.” “I liked having 
 the materials and reference information readily available.” Student Impressions in University Classes (partially modi fi ed) 2 Education under COVID-19 pandemic

Use of video conferencing system (Zoom, etc.) In the beginning, the previous class method was simply applied. Example fo lesson fl ow Show the reference choreography Repeat Provide detailed guidance on how to move the part of choreography Practice Dance the whole part of choreography Dance education/lessons under COVID-19 pandemic Di ffi culty Di ff i cult to understand the complexity of instruction and choreography Di ff i cult to teach and communicate at the same time

Di ff i cult to teach basic movements Network delay Timing of teacher's movements/sounds and students' movements/sounds are out of sync
 Di ffi cult to communicate each other Di ffi cult to give instructions to the whole group Di ffi cult to follow up with individual students Timing of movement and sound are out of sync. Di ffi cult to ask the instructor about 
 the parts that are hard to understand 4 Problem in Online Lesson

Conventional: focus on real-time lesson Demerit Di ffi cult to understand the complexity of instruction and choreography Di ffi cult to teach and communicate at the same time Merit Easy to access (from anywhere) Easy to review interactions through recording videos Easy to see the interaction between the instructor and other students Instructor Student Zoom etc. Proposed: On-demand AI Dance Scoring Allows each student to learn in detail at their own pace Applying Flipped Classroom Scoring/Video feedback Instructor Student Zoom etc. Real-time Sharing craft language and perspectives 5 Dance lessons suitable for online environment

A teaching style that replaces traditional classroom teaching (learning) 
 with homework (development and practice).ʢLage, Platt & Treglia, 2000ʣ Class ʢBasic learningʣ Homework ʢDevelopment) Class ʢDevelopmentʣ Homework ʢBasic learningʣ Traditional Classroom Flipped Classroom 6 Flipped classroom

Problem in Online Lesson Di ff i cult to teach basic movements Network delay Timing of teacher's movements/sounds and students' movements/sounds are out of sync Di ffi cult to communicate each other Di ffi cult to give instructions to the whole group Di ffi cult to follow up with individual students 7 How can fl ipped classroom work? Solution Flipped Classroom Taking the basic movement practice o ff l ine Avoid timing deviations Allow students to make detailed corrections by themselves by repeatedly watching the video. (although communication is not possible).

Real-time (class) On-demandʢhomeworkʣ Environment VR Zoom Basic steps Combination + + Fundamental support Comprehensive support Movement Goal 8 Flipped classroom

Goal: Provide repetition and detailed support to acquire basic steps Example stepɿNew jack Swing 9 Basic training tool (On-demand)

10 Partial scoring Overall scoring ɹɹ3 ɹɹ1 ɹɹ1 All score 2 Basic training tool (On-demand) Hand Waist Foot Goal: Provide repetition and detailed support to acquire basic steps Score

11 Basic training tool (On-demand) Dancer dances 900 times. Data Collection& Annotation Partial scoring Overall scoring ɹɹ3 ɹɹ1 ɹɹ1 All score 2 Hand Waist Foot Goal: Provide repetition and detailed support to acquire basic steps Score

12 Headset+Controller 3 points 6 points 10 points Tracker Tracker × × 3 7 Device Track part None Head, Left hand, Right hand Head, Left hand, Right hand WaistɾLeft legɾRight leg Head, Left hand, Right hand WaistɾLeft legɾRight leg Left elbow, Right elbow, Left knee, Right knee Tracking by VIVE PRO Headset+Controller Headset+Controller Ease level of obtaining

13 Headset+Controller 3 points 6 points 10 points Tracker Tracker × × 3 7 Device Ease level of obtaining Track part None Head, Left hand, Right hand Head, Left hand, Right hand WaistɾLeft legɾRight leg Head, Left hand, Right hand WaistɾLeft legɾRight leg Left elbow, Right elbow, Left knee, Right knee Tracking by VIVE PRO Headset+Controller Headset+Controller

Data Collection 14 ɾCollecting motion ɾCutting motion ɾScoring

Data Collection 15 ɾCollecting motion ɾCutting motion ɾScoring

16 900 motion collection & scoring ɾCollecting motion ɾCutting motion ɾScoring Data Collection

Motion data Score VIVE PRO PC Model Student Tracker Student Reference Score 17 System Con fi guration Hand: Jump: Waist: Hand: ALL: 2 1 2 2 1

Model 18 SVM Features Score Hand ɹ Jump ɹ Waist ɹ Foot ɹ All Mean/Standard deviation of the motion position di ff erence t 2 2 1 1 2

19 PIFuHD Instructor image Human Robot Reference Avator

20 Demo Video VR space User’s view

Real-time (class) On-demandʢhomeworkʣ Environment VR Zoom Basic steps Combination + + Fundamental support Comprehensive support Movement Goal 21 Flipped classroom

ᶃBasic Demo do basic steps ᶄTutoring Consultation ᶅDo advanced steps ᶆShare comments ᶇPoint out issues Development Detail Instruction Communication 22 Lesson con fi guration (Real-time)

23 Comments Instructor The students have been re fi ning the basic steps, 
 so I was able to implement real time smoothly. 
 Usually I have to teach from scratch, but I was able to focus on the application part. It feels like you are actually with the instructor. 
 And I can repeat the course as many times as I want. The advantage is that the student's anxiety is gone, 
 they are a little more familiar with the instructor, 
 and they can communicate with an open mind. Student The immersion is a lot of fun. 
 It's unusual, so that alone makes me want to do it. I felt like I was in a room by myself and could work on it comfortably and happily.
 I had no experience and thought I would never be able to participate in an environment where so many people were dancing around me, 
 but I was able to participate without feeling that way. In the real-time lessons, they picked up on my weak points right away with comments and such. 
 It was nice that they were fl exible and quick to respond to that, 
 like "let's take it slow.

24 On-demand AI Dance Scoring Scoring/Video feedback Instructor Student Zoom etc. Real-time Sharing craft language and perspectives Proposed: Allows each student to learn in detail at their own pace Applying Flipped Classroom Summary

Conducting lessons for a variety of situations Pro fi ciency in dance Compulsory education Participants over a long period of time Veri fi cation of lesson e ff ectiveness and re fi nement to suit the site Quantitative e ff ectiveness measurement Follow-up surveys over ta long period of time Re fi nement of on-demand lesson system Increase in available basic steps Building more e ff ective video feedback 25 Future Work