Making Digital Tangible: The Battle Against the “Pixel Empire”

Slide 1

Slide 1 text

Making Digital Tangible The Battle Against the “Pixel Empire” SIGCHI Lifetime Research Award Lecture CHI 2019 in Glasgow, UK, May 6th, 2019 Hiroshi Ishii MIT Media Lab Tangible Media Photo courtesy of Nobukazu Kuriki @ishii_mit ishii.mit

Slide 2

Slide 2 text

Photo courtesy of Nobukazu Kuriki thanks a million! My sincere appreciation to the Tangible Media Group & MIT Media Lab students over the past 25 years, the CHI, TEI, UIST, CSCW & SIGGRAPH communities over the past 30 years, and the NTT Human Interface Lab colleagues from the early 90s!

Slide 3

Slide 3 text

October 2015 MIT Media Lab 30th Anniversary Tangible Media Group 20th Anniversary

Slide 4

Slide 4 text

CHI 1990 Prof. Bill Buxton & Prof. Marilyn Mantei CHI 1997 Tangible Bits with Brygg Ullmer CHI 1992 ClearBoard with Minoru Kobayashi CHI 1998 Dr. Douglas Engelbart

Slide 5

Slide 5 text

CHI 2006 CHI Academy: SIGCHI Award Banquet CHI 2016 Materiable & Media Lab Party CHI 2017 Prof. Ben Shneiderman, Prof. Jun Rekimoto, … CHI 2018 Prof. Sean Follmer, Prof. Lining Yao & party! CHI 2015 bioLogic & TRANSFORM

Slide 6

Slide 6 text

SIGCHI Award SIGCHI Award SIGCHI Award CHI Community The Human-Computer Interaction (HCI) Pioneers Project by Prof. Ben Shneiderman https://hcipioneers.wordpress.com Prof. Ben Shneiderman

Slide 7

Slide 7 text

My Heroes & Gurus Mark Weiser 1952 - 1999 Douglas Engelbart 1925 - 2013 William Mitchell 1944 - 2010

Slide 8

Slide 8 text

Evolution Photo courtesy of Nobukazu Kuriki 2006 CHI Academy Tangible Bits @ MIT Media Lab (1995-2010) 2000 NTT ICC, Tokyo “Tangible Bits” Exhibition 1997 Tangible Bits Paper at CHI ‘97 2001-04 Ars Electronica, Linz “Getting in Touch” Exhibition 1990 2000 2010 2020 2016-19 Ars Electronica, Linz “Radical Atoms” Exhibition 2019 SIGCHI Lifetime Research Award Radical Atoms @ MIT Media Lab (2005-….) 1995 NTT! MIT Seamless Media @ NTT HI Labs (1990-95) 1992-93 Univ. of Toronto

Slide 9

Slide 9 text

https://foursquare.com/v/nttԣਢլݚڀ։ൃηϯλ/ 4b62542ef964a52097422ae3?openPhotoId=597ec130898bdc5ce5364cd8 1980-1995 NTT Yokosuka ECL & NTT Human Interface Labs NTT envisioned the future of an “Information Superhighway” enabled by B-ISDN (Broadband Integrated Services Digital Network) and ATM (Asynchronous Transfer Mode) Technologies in the 1980’s-90’s. My team at NTT Human Interface Labs focused on the creation of a new bi-directional realtime video-based distributed collaboration service, leading to the inventions of TeamWorkStation (1990) and ClearBoard (1992).

Slide 10

Slide 10 text

TeamWorkStation Hiroshi Ishii & Kazuho Arita CSCW 90, CACM 91, CACM 94 1990 1990

Slide 11

Slide 11 text

TeamWorkStation Translucent Video Overlay of Desktop and Computer Screen for Remote & Realtime Collaboration • Seamlessness (continuity) with existing work practices (desktop & computers) • Seamlessness (smooth transition) between individual and shared workspaces Hiroshi Ishii & Kazuho Arita, et.al. NTT Human Interface Labs CSCW 90, CACM 91, CACM 94

Slide 12

Slide 12 text

1992 ClearBoard-1 NTT Human Interface Labs with Minoru Kobayashi, et.al. CHI 92, CSCW 92

Slide 13

Slide 13 text

ClearBoard • Seamless Integration of Inter-Personal and Shared Work Space • Gaze-awareness Hiroshi Ishii & Minoru Kobayashi, NTT Human Interface Labs CHI ’92, CSCW ’92, TOIS ’93, CACM ’94

Slide 14

Slide 14 text

Sources of Inspiration • VideoDraw by John C. Tang & Scott L. Minneman, Xerox PARC John C. Tang and Scott L. Minneman. 1990. VideoDraw: a video interface for collaborative drawing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '90), Jane Carrasco Chew and John Whiteside (Eds.). ACM, New York, NY, USA, 313-320. • Colab Meeting Room by Mark Steﬁk, et.al. Xerox PARC Mark Steﬁk, Gregg Foster, Daniel G. Bobrow, Kenneth Kahn, Stan Lanning, and Lucy Suchman. 1987. Beyond the chalkboard: computer support for collaboration and problem solving in meetings. Commun. ACM 30, 1 (January 1987), 32-47. • NLS by Douglas Engelbart, Bootstrap Institute D. Engelbart. 1988. The augmented knowledge workshop. In A history of personal workstations, Adele Goldberg (Ed.). ACM, New York, NY, USA 185-248

Slide 15

Slide 15 text

1994 Atlanta Vanguard Conf

Slide 16

Slide 16 text

1994 Vanguard Conference in Atlanta Alan Kay Nicholas Negroponte

Slide 17

Slide 17 text

https://www.lwa-architects.com/project/mit-media-lab/ MIT Media Lab 1995

Slide 18

Slide 18 text

Photo courtesy of Nobukazu Kuriki http://kurikiyama.jp @ishii_mit The reason I chose to come to MIT… It was a mountain so high that the peak was beyond the clouds. There seemed to be no path leading to the top. But… (cont.) Mountain

Slide 19

Slide 19 text

Nicholas Negroponte “REBOOT!” Feb. 10th, 1995 at MIT Media Lab

Slide 20

Slide 20 text

No content

Slide 21

Slide 21 text

October 1995- MIT Media Lab

Slide 22

Slide 22 text

October 1995- MIT Media Lab

Slide 23

Slide 23 text

Strategy https://www.nasa.gov/launchschedule/ main stream GUI • intangible: visual (eyes) • remote control • single user • general purpose new stream TUI • tangible: tactile & kinesthetic (hands & body) • direct manipulation • multi-user, multi-hands • special purpose https://www.art.com/products/p34962790556- sa-i9385929/zeppelin-airship-caught-in- searchlights-during-a-bombing-raid-over- england-1916.htm

Slide 24

Slide 24 text

Tangible Media MIT Media Lab 1995 Photo Credit: Flavia Sparacino

Slide 25

Slide 25 text

Make Digital Tangible Photo Credit: Flavia Sparacino

Slide 26

Slide 26 text

Invent new tangible interactions that inspire and engage people Photo Credit: Flavia Sparacino

Slide 27

Slide 27 text

MIT Media Lab FRAMES 1996 #54 TMG members 1996 Photo Credit: Webb Chappell MIT

Slide 28

Slide 28 text

Tangible Bits graspable media & ambient media foreground (center) & background (periphery) MIT Media Lab FRAMES 1996 “Getting in Touch with the Digital World” Photo Credit: Webb Chappell

Slide 29

Slide 29 text

ABACUS: The Origin of Tangible Bits 1960 2004 2009 PERVASIVE 2004 in Vienna Tokyo, Japan AXIS magazine Vol. 142 ABACUS

Slide 30

Slide 30 text

http://en.wikipedia.org/wiki/File:Grand_orrery_in_Putnam_Gallery,_2009-11-24.jpg

Slide 31

Slide 31 text

Orrery Tangible Representation of Knowledge handle

Slide 32

Slide 32 text

hands — handle

Slide 33

Slide 33 text

A Philosopher Giving a Lecture on the Orrery (sometimes called simply The Orrery) is a painting (oil on canvas, ca. 1766) by Joseph Wright of Derby depicting a public lecture about a model solar system, with a lamp—in place of the sun—illuminating the faces of the audience. http:// en.wikipedia.org/?title=Portal:History_of_science/Previous_pictures#/media/File:Wright_of_Derby,_The_Orrery.jpg collaboration

Slide 34

Slide 34 text

tangible aesthetic interactive

Slide 35

Slide 35 text

tangible - embody aesthetic - inspire interactive - engage

Slide 36

Slide 36 text

Tangible Bits CHI ’97 paper 1997 Photo Credit: Flavia Sparacino

Slide 37

Slide 37 text

Tangible Bits: Towards Seamless Interfaces between People, Bits March 1997 Presented at CHI ‘97 in Atlanta http://dl.acm.org/citation.cfm?id=258715

Slide 38

Slide 38 text

Mark Weiser 1952-1999 “Ubiquitous Computing” "The Computer for the 21st Century" - Scientiﬁc American Special Issue on Communications, Computers, and Networks, September, 1991 "The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.” Mark Weiser

Slide 39

Slide 39 text

Mark Weiser’s message Jan. 26, 1997  Date: Sun, 26 Jan 1997 23:34:10 PST  To: [email protected], [email protected]  From: Mark Weiser   Subject: "Tangible Bits" Dear Hiroshi and Brygg, I recently had a chance to read your CHI 97 paper "Tangible Bits"!   Great work! In my opinion this is the kind of work that will characterize the technological landscape in the twenty-ﬁrst century…. I do have a request. My request is that you help me stop the spread of misunderstanding of ubiquitous computing based simply on its name… "Tangible Bits" is very nice, and maybe could serve as an overall umbrella, but then you might lose it as the name of your research project! …

Slide 40

Slide 40 text

Sources of Inspiration 1 • Ubiquitous Computing (1991) by Mark Weiser Mark Weiser, The Computer for the 21st Century. Scientiﬁc American. September 1991. • DigitalDesk (1991) by Pierre Wellner, Rank Xerox EuroPARC, Cambridge, UK Pierre Wellner. 1993. Interacting with paper on the DigitalDesk. Commun. ACM 36, 7 (July 1993), 87-96. Pierre Wellner. 1991. The DigitalDesk calculator: tangible manipulation on a desk top display. In Proceedings of the 4th annual ACM symposium on User interface software and technology (UIST '91). ACM, New York, NY, USA, 27-33. • Dangling String (1995) by Natalie Jeremijenko, Xerox PARC, CA, U.S.A. Mark Weiser and John Seely Brown, Designing Calm Technology, Xerox PARC December 21, 1995 https://people.csail.mit.edu/rudolph/Teaching/weiser.pdf

Slide 41

Slide 41 text

Sources of Inspiration 2 • Marble Answering Machine (1992) by Durrell Bishop, RCA, London, UK Durrell Bishop. 2009. Visualising and physicalising the intangible product: "What happened to that bloke who designed the marble answer machine?". In Proceedings of the 3rd International Conference on Tangible and Embedded Interaction (TEI '09). ACM, New York, NY, USA, Article 2, 1 pages. • Graspable UI (1995) by George Fitzmaurice, et.al., U. of Toronto, Canada George W. Fitzmaurice, Hiroshi Ishii, and William A. S. Buxton. 1995. Bricks: laying the foundations for graspable user interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '95), Irvin R. Katz, Robert Mack, Linn Marks, Mary Beth Rosson, and Jakob Nielsen (Eds.). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 442-449.

Slide 42

Slide 42 text

Tangible Bits Physical embodiment of   digital information and   computation (CHI ’97) graspable media ambient media

Slide 43

Slide 43 text

Painted Bits (GUI) and Tangible Bits (TUI) Graphical User Interface • Intangible representation (pixels on a screen & sound) • Generic input devices as “remote-controllers” Tangible User Interface • Tangible representation as an interactive control mechanism to manipulate the information directly with hands Urp running on the Sensetable Xerox Star

Slide 44

Slide 44 text

Painted Bits (GUI) and Tangible Bits (TUI) Hiroshi Ishii. 2008. Tangible bits: beyond pixels. In Proceedings of the 2nd international conference on Tangible and embedded interaction (TEI '08). ACM, New York, NY, USA, xv-xxv. DOI=http://dx.doi.org/10.1145/1347390.1347392

Slide 45

Slide 45 text

THE PHYSICAL WORLD THE DIGITAL WORLD GUI TUI RADICAL ATOMS PAINTED BITS TANGIBLE BITS A Graphical User Interfaces only let users see digital information through a screen, as if looking through a surface of the water. We interact with the forms below through remote controls such as a mouse, a keyboard or a touch screen. A Tangible User Interface is like an iceberg: there is a portion of the digital that emerges beyond the surface of the water - into the physical realm - that acts as physical manifestations of computation, allowing us to directly interact with the ‘tip of the iceberg.’ Radical Atoms is our vision for the future of interaction with hypothetical dynamic materials, in which all digital information has physical manifesta- tion so that we can interact directly with it - as if the iceberg had risen from the depths to reveal its sunken mass.

Slide 46

Slide 46 text

Tangible Bits embody digital information to interact with directly with hands GUI TUI 1997 painted bits tangible bits

Slide 47

Slide 47 text

Evolution of Tangible Bits 1 Background - Ambient Media graspable media Center Periphery Foreground Background ambient media ambientROOM (CHI ’97, ’98) pinwheels & water lamp (CoBuild ‘98, CHI ’01) metaDESK (CHI ’97, UIST ’97) Foreground - Graspable Media inTouch (CHI ’97, CSCW ’98) curlybot (SIGGRAPH ’99, CHI ’00) topobo (CHI ’04, ‘08) I/O coincident actuated tangibles topobo topobo (CHI ’04, ‘08) Triangles (DIS ’97, CHI ‘98) constructive assembly mediaBlocks (SIGGRPH’98) HandSCAPE (CHI ’00) TouchCounters (CHI ’99) musicBottles (SIGGRAPH ’99, CHI ’01) LumiTouch ComTouch (CHI ’01) (DIS ’02)

Slide 48

Slide 48 text

ambient media • to be aware of bits at the periphery using ambient display media such as light, sound, airﬂow, and water movement. Center and Periphery Foreground and Background Architectural Spaces as Interfaces • to grasp & manipulate bits in the center of the user's focus by coupling bits with physical objects and surfaces. graspable media Center Periphery Foreground Background ambient media graspable media

Slide 49

Slide 49 text

Foreground metaDESK & Tangible Geospace Ullmer and Ishii, CHI ’97, UIST ‘97 passiveLENS Background ambientROOM Architectural Spaces as Interfaces CHI ’97, ‘98 Ishii, H., Wisneski, C., Dahley, A., Gorbet, M., Brave, S., Ullmer, B., Yarin, P, CHI ’97, ’98, CoBuild ’98 Ullmer, B. and Ishii, H. CHI ’97, UIST ‘97 phicons (physical icons) activeLENS

Slide 50

Slide 50 text

Background (Ambient Media) Pinwheels: wind of bits Ishii, H., Wisneski, C., Dahley, A., Gorbet, M., Brave, S., Ullmer, B., Yarin, P, CHI ’97, ’98, CoBuild ’98 • The architectural space becomes an interface • Ambient information display spinning in a "wind of bits” • NTT ICC 2000 “Tangible Bits” Exhibit in Tokyo

Slide 51

Slide 51 text

musicBottles Ishii, Fletcher, Mazalek, Lee, Choo, Berzowska, Paradiso SIGGRAPH ’99, ’01, CHI ‘01 Jazz Techno Classical Weather • Seamless extension of metaphors and physical affordances into the digital domain • A Principle of Tangible Interface Design: Augmentation of everyday physical objects

Slide 52

Slide 52 text

musicBottles Ishii, Fletcher, Mazalek, Lee, Choo, Berzowska, Paradiso SIGGRAPH ’99, ’01, CHI ‘01 Classical

Slide 53

Slide 53 text

Origin: Weather Bottle present for my mother model: soy sauce bottle in her kitchen

Slide 54

Slide 54 text

ੴҪ ࿨ࢠ Kazuko ISHII 1926 - 1998 Origin: Weather Bottle

Slide 55

Slide 55 text

Bottles: A Transparent Interface as a Tribute to Dr. Mark Weiser IEICE TRANS. INF. & SYST., VOL.E87–D, NO.6 JUNE 2004

Slide 56

Slide 56 text

Coincidence of Input and Output Spaces A Principle of Tangible Interface Design curlybot Frei, P., Su, V., & Ishii, H. CHI ’00, SIGGRAPH ’99 record and playback physical motion topobo Rafﬂe, H. S., Parkes, A. J., and Ishii, H. CHI ’04, ’06, ’08, SIGGRAPH ’07, IDC ‘07 building block with kinetic memory inTouch Brave, S., Dahley, A., Frei, P., & Ishii, H CHI ’97, CSCW ‘98 haptic interpersonal communication

Slide 57

Slide 57 text

Evolution of Tangible Bits 2 Tabletop TUIs Illuminating Light (CHI ’98) Urp (CHI ’99, SIGGRAPH ’99, ISMAR ’02,) metaDESK (CHI ’97, UIST ’97) PingPongPlus (CHI ’99, ACE ‘11) Sensetable (CHI ’01) Audiopad (NIME ’02, ACE ’06) IP NW WB (CHI ’03) Senseboard (CHI ’02) Urban Sim. (ISMAR ’02) Supply Chain Vis (Sys. Dyn. Rev. ’10) Sensetable platform Actuated Workbench (UIST ’02) PSyBench (CSCW ’98) PICO (CHI ’07) actuated tabletop TUIs Illuminating Clay & SandScape (CHI ’02) Phoxel Space (DIS ’04)

Slide 58

Slide 58 text

Seamless Integration of Tangibles and Digital Shadow (Projected Pixels) A Principle of Tangible Interface Design wind light reﬂections digital shadows Urp: Urban Planning Workbench John Underkofﬂer and Hiroshi Ishii, CHI ’98, 99, SIGGRAPH ‘99

Slide 59

Slide 59 text

Luminous Table in Urban Design Studio at MIT SA+P Ben-Joseph, Ishii, Underkofﬂer, Chak, Yeung, Piper CHI ’99, SIGGRAPH ’99, ISMAR ‘02 Urban Planning Workbench used in the Spring 2000 / 2001 MIT SA+P Courses

Slide 60

Slide 60 text

From a Scale Model of the Real World to an Abstract Computational Simulation Model A Principle of Tangible Interface Design Urp CHI ’98, 99, SIGGRAPH ‘99 Sensetable (CHI ’01) Audiopad (NIME ’02, ACE ’06) IP Network Design WB (CHI ’03) Supply Chain Vis (Sys. Dyn. Rev. ’10) Sensetable platform Supply Chain Visualization IP Network Design

Slide 61

Slide 61 text

Audiopad on the Sensetable Platform James Patten and Ben Recht, NIME ’02, ACE ’06 • A new way to perform electronic music. • Designed to combine the expressive power of traditional musical instruments with the modularity of a computer • Based on the Sensetable project.

Slide 62

Slide 62 text

Photo courtesy of Sergi Jordà ReacTable Sergi Jordà, Martin Kaltenbrunner, Günter Geiger, Ross Bencina Proceedings of the ICMC 2005, Barcelona

Slide 63

Slide 63 text

PingPongPlus Ishii, Lee, Wisneski, Orbanes, Chun, Paradiso SIGGRAPH ’98, CHI ’99, NTT ICC ’00, Ars Electronica ‘01 • Interactive Surface • Digital augmentation of ping pong play using a "reactive table" • Ball tracking using a microphone array underneath the table • Shift from competition to collaboration Photo Credit: Webb Chappell

Slide 64

Slide 64 text

Function Application Magnetic forces to move objects on a table in two dimensions. Realtime remote collaboration based on the synchronized distributed “Actuated PICO James Patten and Hiroshi Ishii CHI ‘07 • Simulation of cellular telephone networks • Mechanical constraints, coupled with computer-controlled actuation • Guide the motion of physical objects to inform the computational process Application Actuation of tangibles on table top TUIs Actuated Workbench Dan Maynes-Aminzade, Gian Pangaro & Hiroshi Ishii UIST ‘02

Slide 65

Slide 65 text

SandScape 2003 Illuminating Clay 2002 CHI ’02, TGIS ’04, BT Tech. J. ‘04

Slide 66

Slide 66 text

Hiroshi Ishii, Carlo Ratti, Ben Piper, Yao Wang, and Assaf Biderman Users can alter the form of the landscape model by Users can alter the form of the landscape model by manipulating sand while seeing the resultant effects of the computational manipulating sand while seeing the resultant effects of the computational analysis projected on the surface of the sand in real-time. analysis projected on the surface of the sand in real-time. SandScape Ars Electronic Center 2003 CHI ’02, TGIS ’04, BT Tech. J. ‘04 A Principle of TUI Design: Integration of form giving and computational analysis into one material

Slide 67

Slide 67 text

Lack of Continuity Between Physical and Digital Representation in Design Physical Ease of manipulation Clearer communication Aids spatial understanding Digital Greater precision Easy distribution Quantitative analysis How can we merge these media?

Slide 68

Slide 68 text

Tangible Design Media for Seamless Form Giving & Computational Reflection Physical Digital Upper Stream Lower Stream Rough and rapid form giving Precise and quantitative with hands for ideation computational reflection simultaneous form giving & computational reflection A Principle of Tangible Interface Design

Slide 69

Slide 69 text

Evolution of Tangible Bits 3 I/O Brush (CHI ’04, ’07) Remix+Robo Topobo (IDC ’07, SIGGRAPH ’07) Picture This! (INTERACT ’07, UbiComp ’08) jabberstamp (IDC ’07, SIGGRAPH ’07) Wetpaint (CHI ’09) Piezing (TEI ’09) OnObject (UIST ‘10) Psychohaptics (CHI ’09) Sourcemap (CHI ‘10) CopyCAD (UIST ‘10) deFORM & KidCAD (UIST ’11, CHI ’12) AR-Jig (ISMAR ’07) Beyond (CHI ‘10) g-stalt (TEI ‘10) T(ether) (SUI ’14) Kinected Conf. (CSCW ’11) AnnoScape (SUI ’14) Senspectra (CHI ’07) Glume (CHI ’06) Cord UIs (TEI ’15)

Slide 70

Slide 70 text

I/O Brush Kimiko Ryokai, Stefan Marti & Hiroshi Ishii CHI ’04, ’07, Ars Electronica ‘04 Your environment becomes a color palette to draw with I/O Brush Exhibition Ars Electronica Center Sep. 2004 ~ Aug. 2005

Slide 71

Slide 71 text

I/O Brush: History Mode Kimiko Ryokai, Stefan Marti & Hiroshi Ishii CHI ’04, ‘07 Where does the ink come from? Capturing and weaving the (hi)story for/with every stroke

Slide 72

Slide 72 text

“The World as the Palette” “The World as the Palette” “The World as the Palette” Colors in Barcelona Colors in Barcelona

Slide 73

Slide 73 text

Museums  NTT ICC 2000 Tangible Bits Exhibition Photo Courtesy: NTT ICC

Slide 74

Slide 74 text

NTT ICC 2000 Tangible Bits Exhibition Photo Courtesy: NTT ICC

Slide 75

Slide 75 text

Ars Electronica Center 2001-04 Get in Touch Exhibition

Slide 76

Slide 76 text

Tangible Bits Get in Touch Exhibition 2001-2004 Ars Electronica Center

Slide 77

Slide 77 text

inTouch musicBottles PingPongPlus ClearBoard SandScape I/O Brush Urp PegBlocks curlybot Triangles pinwheels Tangible Bits Projects Exhibited at Ars Electronica 2001-2004

Slide 78

Slide 78 text

tangible bits 1997 radical atoms 2012 Radical Atoms Dynamic, Physical & Computational Materials that • Conform to structural constraints, • Transform structure & behavior, & • Inform new abilities.

Slide 79

Slide 79 text

1. Frozen Atoms: inert, rigid, passive physical materials 2. Intangible Pixels: dynamic, virtual and intangible pixels (bits)   trapped behind a 2D ﬂat screen Two Material Options Exist Today

Slide 80

Slide 80 text

1. Frozen Atoms: inert, rigid, passive physical materials 2. Intangible Pixels: dynamic, virtual and intangible pixels (bits)   trapped behind a 2D ﬂat screen 3. Radical Atoms: dynamic, physical and computational materials that transform and change   properties, driven by digital data and computation Introducing The Third Material Two Material Options Exist Today

Slide 81

Slide 81 text

Sources of Inspiration • Programmable matter (1991) by Toffoli and Margolus Toffoli, Tommaso; Margolus, Norman (1991). "Programmable matter: concepts and realization". Physica D. 47: 263–272. • Reconﬁgurable Robotics: M-Blocks (2015) by Daniela Rus, et.al., MIT CSAIL John W. Romanishin, Kyle Gilpin, Sebastian Claici, and Daniela Rus, 3D M-Blocks: Self-reconﬁguring Robots Capable of Locomotion via Pivoting in Three Dimensions, 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, Washington, May 26-30, 2015 • Shape Display: FEELEX (2001) by Hiroo Iwata, et.al., U. of Tsukuba, Japan Hiroo Iwata, Hiroaki Yano, Fumitaka Nakaizumi, and Ryo Kawamura. 2001. Project FEELEX: adding haptic surface to graphics. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques (SIGGRAPH '01). ACM, New York, NY, USA, 469-476

Slide 82

Slide 82 text

Linz, Austria, September 8 - 12, 2016 RADICAL ATOMS AND THE ALCHEMISTS OF OUR TIME

Slide 83

Slide 83 text

Radical Atoms Exhibition 2016~2019 @ Ars Electronica Center, Linz, Austria RADICAL ATOMS AND THE ALCHEMISTS OF OUR TIME

Slide 84

Slide 84 text

KinetiX Cilllia aeroMorph bioLogic inFORM musicBottles Radical Atoms Exhibition @ Ars Electronica Center Tangible Media Group | MIT Media Lab September 2016 ~ Summer 2019 in Linz, Austria Prof. Hiroshi Ishii

Slide 85

Slide 85 text

Radical Atoms Exhibition Linz Austria September 2016-2019 11 ML Projects musicBottles ZeroN Perfect Red SandScape topobo inFORM bioLogic LineFORM PneUI jamSheets Rovables Active Wood Self-Assembler Lab Lift-Bit Carlo Ratti Associati Inﬁnite Cube ART+COM guests projects SPAXELS Ars Eelectronica Futurelab Media Lab, Responsive Environments Group

Slide 86

Slide 86 text

Tangible Media Group | MIT Media Lab Radical Atoms Exhibition 2016-19 @ Ars Electronica Center

Slide 87

Slide 87 text

ARS ELECTRONICA Linz, Austria Ars Electronica Center 2016-19 Radical Atoms Exhibition

Slide 88

Slide 88 text

Evolution of Radical Atoms 1 Actuated Tangibles inTouch (CHI ’97, CSCW ’98) curlybot (SIGGRAPH ’99, CHI ’00) topobo (CHI ’04, ‘08) I/O coincident actuated tangibles Actuated WB (UIST ’02) PSyBench (CSCW ’98) actuated tabletop TUIs PICO (CHI ’07) Weight/Volume Changing UI (TEI ’14) Amphorm (CogInfoCom ’12) Perfect Red & Radical Atoms (interactions ‘12) Kinetic Sketchup & Bosu (TEI ’09, DIS ’10) ZeroN (UIST ’11) levitating tangibles vision MirrorFugue (NIME ’13, CHI ’16) Art & Science

Slide 89

Slide 89 text

ACM interactions ‘12

Slide 90

Slide 90 text

No content

Slide 91

Slide 91 text

Arts & Sciences

Slide 92

Slide 92 text

MirrorFugue III (2013) Xiao Xiao

Slide 93

Slide 93 text

MirrorFugue III (2013) Xiao Xiao

Slide 94

Slide 94 text

MirrorFugue III Xiao Xiao MirrorFugue III Xiao Xiao

Slide 95

Slide 95 text

Evolution of Radical Atoms 2 Shape Displays Static / Passive Tangible Bits Tangible Bits ! LineFORM & ChainFORM (UIST ’15, ’16 IEEE Perv. Comp. ’17) Illuminating Clay & SandScape (CHI ’02) Recom pose (UIST ‘11) Relief (TEI ’09, UIST ‘11) SUBLI MATE (CHI ‘13) inFORM (UIST ’13, ’14, Fast Comp. Design Award ’14) Tangible CityScape TRANSFORM (CHI ’15. A’Design Award ’15) Kinetic Blocks (UIST ‘15) Materiable (CHI ’16) inFORCE (TEI ‘19) AnimaStage (DIS ‘17) Active / Kinetic Shape Displays Physical Telepresence (UIST ’14) Radical Atoms

Slide 96

Slide 96 text

LineFORM & ChainFORM: Actuated Line/Curve Interfaces Ken Nakagaki, Artem Dementyev, Sean Follmer, Joseph Paradiso, and Hiroshi Ishii. UIST ’15, ‘16 LineFORM ChainFORM

Slide 97

Slide 97 text

Recompose based on the Relief Anthony DeVincenzi, David Lakatos, Matthew Blackshaw, Daniel Leithinger & Hiroshi Ishii Relief: A 2.5D Shape Display Daniel Leithinger, David Lakatos, Anthony DeVincenzi, Matthew Blackshaw, and Hiroshi Ishii. TEI ’10, UIST ‘11 Daniel Leithinger, Jinha Lee, Sean Follmer, Austin Lee, Matthew Chang & Hiroshi Ishii Daniel Leithinger & Hiroshi Ishii

Slide 98

Slide 98 text

Fast Company Innovation by Design Awards: Winner - Experimental Red Dot Award: Best of the Best - Design Concept Laval Virtual 2014 Award - INDUSTRIAL DESIGN & SIMULATION Core 77 Award - Interaction Student Winner IDSA IDEA Award Bronze inFORM Sean Follmer, Daniel Leithinger, Alex Olwal, Akimitsu Hogge, Hiroshi Ishii UIST ’13, ‘14

Slide 99

Slide 99 text

inFORM Sean Follmer, Daniel Leithinger, Alex Olwal, Akimitsu Hogge, Hiroshi Ishii. UIST ’13, ‘14

Slide 100

Slide 100 text

Cooper Hewitt Design Museum inFORM Exhibition Dec. 2014 - May 2015, New York Prof. Daniel Leithinger, Prof. Sean Follmer Philipp Schoessler, Jared Counts, Ken Nakagaki, David Doan, Basheer Tome and Prof. Hiroshi Ishii

Slide 101

Slide 101 text

inFORM ENGINES Designed by Daniel Leithinger & Sean Follmer, and Rendered by Amit Zoran

Slide 102

Slide 102 text

inFORM ENGINES Designed by Daniel Leithinger & Sean Follmer, and Rendered by Amit Zoran

Slide 103

Slide 103 text

The three panels of the triptych were sold separately in the mid-1970s.[9] Bacon was unhappy that the panels had been split up, writing on a photograph of the left-hand panel that it was "meaningless unless it is united with the other two panels." Triptych Francis Bacon

Slide 104

Slide 104 text

TextTextText TextTextText TRANSFORM Tangible Media MIT Media Lab (C) LEXUS DESIGN AMAZING 2014 MILAN

Slide 105

Slide 105 text

TRANSFORM TRANSFORM Tangible Media Tangible Media MIT Media Lab MIT Media Lab

Slide 106

Slide 106 text

Milano Design Week LEXUS DESIGN AMAZING 2014 MILAN April 8-13, 2014 Tangible Media Group MIT Media Lab

Slide 107

Slide 107 text

No content

Slide 108

Slide 108 text

Slide 109

Slide 109 text

https://www.ﬂickr.com/photos/79203622@N02/9734698717/ TRANS-Disciplinary Finding opportunity in conﬂict between disciplines Breaking down old paradigms to create new archetypes

Slide 110

Slide 110 text

“Intriguing Elegance through Careful Juxtaposition of Opposing Elements” Lﬁnesse by LEXUS Design vs Technology Stillness vs Motion Atoms vs Bits

Slide 111

Slide 111 text

Milano Design Week 2014 TRANSFORM Exhibit 04/08-13/14, Milano, Italia Platinum A’DESIGN AWARD 2015 Prof. Hiroshi Ishii Prof. Hiroshi Ishii Prof. Daniel Leithinger Prof. Daniel Leithinger Prof. Sean Follmer Prof. Sean Follmer Prof. Amit Zoran Prof. Amit Zoran Philipp Schoessler Philipp Schoessler Jared Counts Jared Counts Tangible Media Group | MIT Media Lab @ LEXUS DESIGN AMAZING 2014 MILAN

Slide 112

Slide 112 text

Luke Vink, Viirj Kan, Ken Nakagaki, Daniel Leithinger, Sean Follmer, Philipp Schoessler, Amit Zoran, and Hiroshi Ishii. CHI ’15 Tangible Media Group | MIT Media Lab CHI 2015 Golden Mouse Award

Slide 113

Slide 113 text

A measure of a ﬂuid’s resistance to gradual deformation by shear stress or tensile stress WATER HONEY The ability for a material to resist an applied force and to return to its original shape CLAY RUBBER The extent to which a material can be deformed in response to an applied force CONCRETE FOAM FLEXIBILITY ELASTICITY VISCOSITY Ken Nakagaki, Luke Vink, Jared Counts, Daniel Windham, Daniel Leithinger, Sean Follmer, and Hiroshi Ishii. CHI ‘16 Materiable: Rendering Material Properties in Response to Direct Touch with Shape Changing Interfaces

Slide 114

Slide 114 text

SHAPE CHANGE + DIRECT MANIPULATION + MID-AIR GESTURE MATERIAL PROPERTY REPRESENTATION + BI-DIRECTIONAL FORCE CONTROL Leithinger et al. (TEI10, UIST11, CHI11) Follmer, Leithinger, et al. (UIST13, UIST14, CHI15) Nakagaki, Vink, et al. (CHI16, TEI19) PHYSICAL TELEPRESENCE + INTERMATERIAL INTERACTION + DYNAMIC AFFORDANCES Relief + Recompose inForm + TRANSFORM MATERIABLE + inFORCE

Slide 115

Slide 115 text

Evolution of Radical Atoms 3 Programmable Materials Jamming UI (UIST ‘12) PneUI (UIST ‘13) jamSheets (TEI ‘14) optiElastic (UIST ‘14) Pneuduino (TEI ’16) aeroMorph (UIST ’16) Printﬂatables (CHI ’17) Pneumatic Shape Changing UIs bioPrint & xPrint (UIST ’14, CHI ’16) bioLogic (CHI ’15, A’Design Awards ’16, Sci. Adv. ‘17) Transfor- mative Appetite (CHI ’17) Hygromorphic Materials uniMorph (UIST ’15) Programmable Droplets (CHI ’18) HydroMorph (TEI ’16) Organic Primitives (CHI ’17) Cilllia (CHI ’16) SensorKnits (3D Prt. & Add. Manu. ’19) kinetiX (Comp. & Graphics ‘18)

Slide 116

Slide 116 text

PneUI (2012-13) UIST ‘13 jamSheets (2013-14) TEI ‘14 bioLogic (2014-15) CHI ‘15 aeroMoprh (2015-16) UIST ‘16 KinetiX (2014-18) Comp. & Graphics ‘18 Cilllia (2015-19) CHI ‘16 SensorKnit (2018-19) 3D Prt. & Add. Manu. ’19 Create responsive materials at the mesoscale to design novel sensors, actuators and displays uniMorph (2015) UIST ‘15 Programmable Materials

Slide 117

Slide 117 text

Cilllia 3D-Printing Functional Hair-like Structures Cilllia presents a computational method of 3D printing hair structures. It allows us to design and generate hair geometry at 50 micrometer resolution and assign various functionalities to the hair. The ability to fabricate customized hair structures enables us to create super ﬁne surface texture; mechanical adhesion property; new passive actuators and touch sensors on a 3D printed artifact. 14,400 strands of hair on a 4cm by 4cm substrate Figures with detailed surface texture Figures with detailed surface texture Figures with detailed surface texture Figures with detailed surface texture Cilllia Jifei Ou, Gershon Dublon, Chin-Yi Cheng, Liang Zhou, Felix Heibeck and Hiroshi Ishii Golden A’Design Award 2017, CHI ’16

Slide 118

Slide 118 text

surface with mechanical adhesion sensing touch speed & direction with sound analysis Rotary Linear passive actuation with vibration: moving direction control Combined Cilllia Jifei Ou, Gershon Dublon, Chin-Yi Cheng, Liang Zhou, Felix Heibeck and Hiroshi Ishii Golden A’Design Award 2017, CHI ’16

Slide 119

Slide 119 text

Photo by Mattia Balsamini Fur & Feather as a medium for design Cilllia Jifei Ou, Gershon Dublon, Chin-Yi Cheng, Liang Zhou, Felix Heibeck and Hiroshi Ishii Golden A’Design Award 2017, CHI ’16

Slide 120

Slide 120 text

from Dr. Lining Yao’s Ph.D. thesis

Slide 121

Slide 121 text

Lining Yao, Jifei Ou, Chin-Yi Cheng, Helene Steiner, Wen Wang, Guanyun Wang, Hiroshi ishii. bioLogic: Natto Cells as Nanoactuators for Shape Changing Interfaces.In Proc. of CHI 2015. ACM BioLogic: from Build to Grow

Slide 122

Slide 122 text

bioLogic: Lining Yao, Wen Wang, Guanyun Wang, Helene Steiner, Chin-Yi Cheng, Jifei Ou, Oksana Anilionyte, Hiroshi Ishii. 3D Print. & Additive Manuf.’15, CHI ’15, Science Advances ’17, 3 A’Design Awards ’16, Fast Co.Design 2016 Innovation By Design Award, DIA Excellence Award '16

Slide 123

Slide 123 text

Slide 124

Slide 124 text

Hiroshi Ishii MIT Media Lab “Bio is the new Interface” bioLogic Exhibit at MIT Media Lab E14 lobby in October 2015 Beyond Human-Machine Interfaces Towards Human-Living Machine Interfaces

Slide 125

Slide 125 text

Prof. Lining Yao, concept creation, interaction design and fabrication, MIT Media Lab …now CMU professor Dr. Wen Wang, biotechnology and material science, MIT Dept. of Chemical Engineering Guanyun Wang, industrial design and fabrication, MIT Media Lab/Zhejiang University Helene Steiner, interaction design, MIT Media Lab/ Royal College of Art Chin-Yi Cheng, computational design and simulation, MIT Architecture Jifei Ou, concept design and fabrication, MIT Media Lab Oksana Anilionyte, fashion design, MIT Media Lab/ Royal College of Art Prof. Hiroshi Ishii, advising and directing, Tangible Media Group, MIT Media Lab bioLogic Team 3 A’DESIGN AWARDS 2016 Textile Wearable Fashion Platinum Gold Silver

Slide 126

Slide 126 text

bioLogic: Lining Yao, Jifei Ou, Chin-Yi Cheng, Helene Steiner, Wen Wang, Guanyun Wang, Hiroshi ishii CHI ‘15 Grow rather than Build Radical Atoms The Bacterium Bacillus subtilis taken with a Tecnai T-12 TEM. Taken by Allon Weiner, The Weizmann Institute of Science, Rehovot, Israel. 2006.

Slide 127

Slide 127 text

TRANSFORM: Hiroshi Ishii, Daniel Leithinger, Sean Follmer, Amit Zoran, Philipp Schoessler, and Jared Counts. CHI ‘15 bioLogic: Lining Yao, Jifei Ou, Chin-Yi Cheng, Helene Steiner, Wen Wang, Guanyun Wang, Hiroshi ishii CHI ‘15 Programmable Droplets Udayan Umapathi, Patrick Shin, Ken Nakagaki, Daniel Leithinger and Hiroshi Ishii CHI ‘18 Dance Radical Atoms

Slide 128

Slide 128 text

CHI 2018 Golden Mouse Award Programmable Droplets Udayan Umapathi, Patrick Shin, Ken Nakagaki, Daniel Leithinger and Hiroshi Ishii CHI ‘18

Slide 129

Slide 129 text

Ω Levitate Radical Atoms

Slide 130

Slide 130 text

Fmag + mg = 0N ZeroN Jinha Lee, Rehmi Post, Hiroshi Ishii, MIT Media Lab UIST ‘11

Slide 131

Slide 131 text

ZeroN: Tangible Media Group Defy Gravity SPAXELS: Ars Electronica Futurelab

Slide 132

Slide 132 text

https://www.ﬂickr.com/photos/79203622@N02/9734698717/ INVENT & INSPIRE

Slide 133

Slide 133 text

https://www.ﬂickr.com/photos/79203622@N02/9734698717/ Be Artistic & Analytic Be Poetic & Pragmatic

Slide 134

Slide 134 text

Arts & Sciences Arts & Sciences Arts & Sciences

Slide 135

Slide 135 text

ART DESIGN SCIENCE TECHNOLOGY questions the world around us explains the world around us articulates the solution enables the solution

Slide 136

Slide 136 text

TRANSCENDING ART | DESIGN | SCIENCE | TECHNOLOGY Sagrada Familia Cathedral by Gaudi - spiral staircase The Tower of Babel by Pieter Bruegel the Elder (1563)

Slide 137

Slide 137 text

BUILDING SPIRAL TOWER OF THE BABEL TRANS-DISCIPLINARY STUDY THAT TRANSCENDS ART | DESIGN | SCIENCE | TECHNOLOGY BUILDING SPIRAL TOWER OF THE BABEL TRANS-DISCIPLINARY STUDY THAT TRANSCENDS ART | DESIGN | SCIENCE | TECHNOLOGY The Tower of Babel by Pieter Bruegel the Elder (1563) & ART DESIGN SCIENCE TECHNOLOGY Spiral by Hiroshi Ishii

Slide 138

Slide 138 text

Slide 139

Slide 139 text

What’s Next? Body Syntonic …

Slide 140

Slide 140 text

The Future is not to predict, but to invent Alan Kay 1971 This is the century in which you can be proactive about the future; you don't have to be reactive. The whole idea of having scientists and technology is that those things you can envision and describe can actually be built. Photo courtesy of Nobukazu Kuriki

Slide 141

Slide 141 text

vision

Slide 142

Slide 142 text

Vision Needs Technologies > 100 years ~10 years ~1 year Lifespan

Slide 143

Slide 143 text

Photo courtesy of Nobukazu Kuriki Envision vision vision vision vision

Slide 144

Slide 144 text

Envision Embody Inspire Photo courtesy of Nobukazu Kuriki

Slide 145

Slide 145 text

Envision Embody Inspire Photo courtesy of Nobukazu Kuriki Art Design, Science & Technology Art

Slide 146

Slide 146 text

ग़ߌྗ ಓఔྗ ଄ࢁྗ

Slide 147

Slide 147 text

ग़ߌྗ ಓఔྗ ଄ࢁྗ Three Driving Forces 1D: Stick Out 2D: Blaze the Trail 3D: Build the Mountain

Slide 148

Slide 148 text

ग़ߌྗ Stick Out The nail that just pokes out gets hammered down. Stick out far enough that no hammer can reach you. Stand out among the crowd. @ishii_mit

Slide 149

Slide 149 text

ಓఔྗ Blaze the Trail There is no road laid out before me. I charge forward, and a road emerges behind me. Neither spectators nor referees nor stopwatches exist here. http://www.mot-art-museum.jp/exhibition/148/3

Slide 150

Slide 150 text

Photo courtesy of Nobukazu Kuriki http://kurikiyama.jp @ishii_mit The reason I chose to come to MIT…It was a mountain so high that the peak was beyond the clouds. There seemed to be no path leading to the top. ଄ࢁྗ Build the Mountain

Slide 151

Slide 151 text

Photo courtesy of Nobukazu Kuriki http://kurikiyama.jp @ishii_mit But…I soon realized there was no existing mountain to climb. To create the mountain from the ground up, to be the ﬁrst to reach the peak. That’s how you survive in MIT. ଄ࢁྗ Build the Mountain The reason I chose to come to MIT…It was a mountain so high that the peak was beyond the clouds. There seemed to be no path leading to the top.

Slide 152

Slide 152 text

Photo courtesy of Nobukazu Kuriki http://kurikiyama.jp

Slide 153

Slide 153 text

Today today Photo courtesy of Nobukazu Kuriki http://kurikiyama.jp

Slide 154

Slide 154 text

today 2050 2050 Photo courtesy of Nobukazu Kuriki http://kurikiyama.jp

Slide 155

Slide 155 text

today 2050 2100 2100 Photo courtesy of Nobukazu Kuriki http://kurikiyama.jp

Slide 156

Slide 156 text

today 2050 2100 2200 2200 Photo courtesy of Nobukazu Kuriki http://kurikiyama.jp

Slide 157

Slide 157 text

today 2050 2100 2200 Photo courtesy of Nobukazu Kuriki http://kurikiyama.jp What do you want to pass on to those living in 2200? How do you want to be remembered? What do you want your legacy to be?

Slide 158

Slide 158 text

future 2200 Photo courtesy of Nobukazu Kuriki http://kurikiyama.jp

Slide 159

Slide 159 text

2200 future Photo courtesy of Nobukazu Kuriki http://kurikiyama.jp

Slide 160

Slide 160 text

2200 Life has a set end point But the future is never-ending Photo courtesy of Nobukazu Kuriki http://kurikiyama.jp

Slide 161

Slide 161 text

2200 Technology soon becomes obsolete But true vision is ever-lasting “Cradle of Stars” by Scott Cresswell https://www.ﬂickr.com/photos/scott-s_photos/11763686274/

Slide 162

Slide 162 text

MIT Media Lab 2200 “Cradle of Stars” by Scott Cresswell https://www.ﬂickr.com/photos/scott-s_photos/11763686274/

Slide 163

Slide 163 text

Thanks! ੴҪ ༟ Hiroshi Ishii MIT Media Lab @ishii_mit ishii.mit ishii.mit

Slide 164

Slide 164 text

Thanks! ੴҪ ༟ Hiroshi Ishii MIT Media Lab @ishii_mit ishii.mit @ishii_mit ishii.mit

Slide 165

Slide 165 text

Making Digital Tangible The Battle Against the “Pixel Empire” SIGCHI Lifetime Research Award Lecture CHI 2019 in Glasgow, May 6th, 2019 Hiroshi Ishii MIT Media Lab Tangible Media Photo courtesy of Nobukazu Kuriki @ishii_mit ishii.mit