TSEiA : The Story Envisioning Intelligent Assistant

The next generation game story using NLP

Transcript

1. TSEiA : The Story Envisioning Intelligent Assistant ZHANG QIANG qiang@couger.co.jp https://akitohisano.wixsite.com/ultraprofile COUGER couger.co.jp
2. Structure • Introduction • Method • Evaluation • Future work • Conclusion
3. Introduction • Conversation Analysis • People converse with each other to share information, emotions knowledge, thoughts, etc. • Five viewpoints(levels) of conversation: • verbal communication: dialogues, languages, etc. • non-verbal communication: body languages, facial expressions, etc. • social discourse: socialized backgrounds, culture backgrounds, etc. • narratives and content flow: structures and information transfer, etc. • cognitive processes: mental processes, intentions, etc.
4. Introduction • Storytelling • Serving as a n inevitable part of conversational and cultural activities. • Essential method of sharing experience and knowledge.
5. Introduction • Graphic Recording • Graphic recording describes the method of using imagery to lead groups and discussions twards a goal. • Use visualization to help people understand difficult contents.
6. Introduction • Visualized Stroytelling • Conduct storytelling while visualizing the contents of the story • Act scenarios and conversations of the story • Digital Visualized Storytelling • Stories are rebuilt in a virtual world in which resources such as characters, animations and objects are used to represent the story contents.
7. Introduction • Interactive Stroytelling • Interactive storytelling is a form of digital environment in which the story line is not predetermined. • User can experience a unique story based on their interactions with the story world.
8. Introduction • Interactive storytelling in games • Open-world • Story branches • Multiple Ending • ...
9. Collaborative Storytelling/Authoring • Story is created by and shared among multiple authors. • Significantly increase the speed of creating story. • Significantly increase the complexity of story space.
10. Problem • Stories can be difficult to understand because of the differences on culture, knowledge and personal background. • Different interpretations add branches into current story(non-linear storytelling) and create the story space. • Researchers have been focusing on using AI to do storytelling and build story space for a long time.
11. Problem • Game developers are not only improving the final performance of the games, but also the methods of creating games. • How can we develop a game more conveniently?
12. Problem • Need tools and methods for quick story visualization • Tools that can help people for collaborative storytelling/authoring • Tools that can quickly create game-like interactive dramas
13. Related Work • Interactive Storytelling • Choose-Your-Own-Adventure [1976] • AESOPWORLD [1999] • Façade [2003] • ...
14. Related Work • Text-to-scene/animation systems • WordsEye [2001](B Coyne, R Sproat) • Carsim [2003] (O Åkerberg, et al.) • CONFUCIUS [2006] • SceenSeer [2017] • TAPE [2018]
15. Related Work • Visual Novel • Doki Doki Literature Club • Nekopara • Cyberpunk Bartender Action • ...
16. Method • Story Envisioning • Entails the method of using interactive drama to build connections between abstract concepts and intuitive objects. • Consider conversation as an update of common ground. • Using interactive drama create visualized story space. • Story Envisioning • Users are interested in avoiding misunderstanding in daily conversation. • Researchers/Engineers are interested in analyzing the story space. • Educators/Trainers are interested in building virtual environments.
17. Method Conversation participants Other potential participants (Researchers/Engineers/Educators /Trainers) • Using story envisioning to create a shared story space.
18. TSEiA • Build AI that can smooth human conversation using story envisioning. • Story Visualization Module - Convert the story written in natural language into interactive drama • Story Editing Module - Manage story space • Collective Storytelling Module - Create shared space for different individuals to create story collaboratively • Usage • Tool to help users understand story better • Tool to create story space for multiple purposes System Architecture System Smple UI
19. TSEiA • Story visualizaiton module • Extract action sequences from the story. • Mapping actions to repostiory using word embedding. • Render the scenarios by implemeting a timeline asset in Unity. smaple action sequences
20. TSEiA • Story Editing Module • provides the function of adding, deleting, comparing branches. • provides the function of traversing (going forward and back) through story branches. • Collective Storytelling Module • create shared space by merging game files and re-destributing them to local machines. • predictive model to give suggestions on creating new branches.
21. TSEiA-Demo • https://www.youtube.com/watch?v=A2SAFO5RhpQ&
22. Evaluation • Effectiveness of visualization • Quality and complexity of story space • Usability of the framework
23. Experiment • Role-playing game platform powered by TSEiA was built. • Participants learn some background knowledge of the story through text. • Participants need to check the dramas generated by TSEiA and extract necessary information. • Participants complete the story by answering some questions using the information they obtained. • TSEiA visualizes the story based on contents generated by participants. Experiment Flow
24. Effectiveness of Visualization Relevance score Effectiveness Analysis Two types of question are designed: • related to knowledge from text (TQ) • related to knowledge from drama (DQ) Effectiveness analysis attributes: • Comprehnsion - understand the certain portion of the story. • Acc - accuracy to retell the story content correctly. • Consistency - able to use the information obtained to pursue the primary goal.
25. Quality of story space Story contents generated by participants: • culture driven - communication/storytelling habit. • personality driven - personal opinions on certain topics. • situation driven - information about situation analysis ability. Contets generated by participants “I may have taken the icecream from his son and threw it on the ground to show him what is mean”
26. Complexity of story space 180+ branches from 19 paticipants
27. Quality and Complexity of Story Space • Summary • TSEiA can be used to collect cultural, personality, situation driven data. • TSEiA can be used to collect rich content with high complexity.
28. Usability of the framework Questionnaire is used to show the usability. Most of the people agreed to the statements.
29. Application • Virtual Reality Graphic Recorder[1] (IEA-AIE 2018, HCII 2018) • Vitrual Reality Conversation Envisioning Framework[2] (EuroCALL 2018) [1]Mirzaei M.S., Zhang Q., Nishida T. (2018) Conversation Envisioning to Train Inter-cultural Interactions. In: Meiselwitz G. (eds) Social Computing and Social Media. Technologies and Analytics. SCSM 2018. Lecture Notes in Computer Science, vol 10914. Springer, Cham [2]Mirzaei, M. S., Zhang, Q., Van der Struijk, S., & Nishida, T. (2018). Language learning through conversation envisioning in virtual reality: a sociocultural approach. In P. Taalas, J. Jalkanen, L. Bradley & S. Thouësny (Eds), Future-proof CALL: language learning as exploration and encounters – short papers from EUROCALL 2018 (pp. 1-7). Research-publishing.net.
30. Future Application • Games to create dialogue systems, cut scenes, CG and so on. • Support actions/reactions of game characters, avatars. • Serveing as one of the services provided by Rachel.
31. Future work • Including speech recognition systems to activate the feature of voice input. • Using the data collected to build the predictive model in order to give suggestions automatically under certain situations.
32. Conclusion • Introduced story envisioning as the method to help people understand stories in daily conversation better and create visualized stroy space. • Built framework TSEiA as platform for story envisioning. • To our knowledge, TSEiA is the first framework that provides direct controlment of Unity Timeline. • Experiment evidences showed that story envisioning could be an effective tool to facilitate stories and tool to create story space.
33. Thank you for your attention !