Whereas the three-dimensional output we discussed in  the previous chapter  can create entirely new virtual worlds, or enhanced versions of our own world, it’s equally important that software be able to interface with the physical world. One of the first research papers to recognize this was the seminal work  Tangible Bits: Towards Seamless Interfaces between People, Bits and Atoms . ^{8 8}

Hiroshi Ishii and Brygg Ullmer. (1997). Tangible bits: Towards seamless interfaces between people, bit, atoms. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI).

This basic idea, that the massive rift between our physical and digital worlds should be bridged by a diversity of new physical input and output media, dovetailed Weiser’s vision of ubiquitous computing. ^{19 19}

Mark Weiser (1991). The Computer for the 21st Century. Scientific American 265, 3 (September 1991), 94-104.

The impact of this vision was an explosion of research to create more physical input and output. In this chapter, we’ll discuss this research, and the history of physical computing that it build upon. 

Printing

Most people who’ve interacted with computers have used a printer at some point in their life to turn bits into atoms. The basic principle is simple: take a digital document and create a copy with paper and some kind of marking substance, such as spraying ink (inkjets), burning the paper (laser printers), or one of a variety of other approaches. 

Why was printing so important as computers first became ubiquitous?

For most of the 20th century, paper was the central medium for transmitting information. We used typewriters to create documents. We stored documents in file cabinets. We used photocopiers to duplicate documents. The very notion of a file and folder in graphical user interfaces mirrored the ubiquity of interacting with paper. Printers were a necessary interface between the nascent digital world and the dominant paper-based world of information. 

One of the earliest forms of digital printing was the stock ticker machine, which printed text on a thin strip of paper on which messages are recorded:



While ticker tape was the most ubiquitous before the advent of digital computers, printing devices had long been in the imagination of early computer scientists.  Charles Babbage  was an English mathematician and philosopher who first imagined the concept of a programmable digital computer in 1822. He also imagined, however, a mechanical printing device that could print the results of his imagined differencing machine. Eventually, people began to engineer these printing devices. For example, consider the dot matrix printer shown in the video below, which printed a grid of ink. These printers were ubiquitous in the 1980’s, and a general extension of both ticker tape printers and a deeper integration with digital printing from general purpose computers. This was the beginning of a much greater diversity of printing mediums we use today, which use toner, liquid ink, solid ink, or dye-sublimation. 

Most of these printing technologies have focused on 2-dimensional output because the documents we create on computers are primarily 2-dimensional. However, as the plastics industry evolved, and plastic extruders reshaped manufacturing, interest in democratizing access to 3D fabrication expanded. This led to the first 3D printer, described in a U.S. patent in 1984, which described a process for generating 3D objects by creating cross-sectional patterns of an object. ^{7 7}

Hull, C. (1984). Patent No. US4575330 A. United States of America.

While the basic idea of 3D printing is now well established, and the market for 3D printers is expanding, researchers have gone well beyond the original premise. Much of this exploration has been in exploring materials other than plastic. One example of this is an approach to printing  interactive electromechanical objects with wound in place coils . ^{17 17}

Huaishu Peng, François Guimbretière, James McCann, Scott Hudson (2016). A 3D Printer for Interactive Electromagnetic Devices. ACM Symposium on User Interface Software and Technology (UIST).

Another project explored printing with wool and wool blend yarn to create soft felt objects rather than just rigid plastic objects. ^{6 6}

Scott E. Hudson (2014). Printing teddy bears: a technique for 3D printing of soft interactive objects. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI).

These new forms of printing pose new challenges in authoring. One cannot print arbitrary 3D shapes with 3D printers, and so this requires authors to understand the limitations of printing methods. This lack of understanding can lead to failed prints, which can be frustrating and expensive. Some researchers have explored “patching” 3D objects, calculating additional parts that must be printed and mounted onto an existing failed print to create the desired shape. ^{18 18}

Alexander Teibrich, Stefanie Mueller, François Guimbretière, Robert Kovacs, Stefan Neubert, Patrick Baudisch (2015). Patching Physical Objects. ACM Symposium on User Interface Software and Technology (UIST).

These techniques, while not advancing  how  objects are printed, innovate in how we author objects to print, much like research innovations in word processing software from the 1980’s and 1990’s.

Of course, just as 2D printing imposes gulfs of execution and evaluation — trying to configure a printer to have the right printed margins, debugging why they don’t — 3D printing comes with its own sets of challenges. For example, Kim et al. ^{10 10}

Jeeeun Kim, Anhong Guo, Tom Yeh, Scott E. Hudson, and Jennifer Mankoff (2017). Understanding Uncertainty in Measurement and Accommodating its Impact in 3D Modeling and Printing. ACM Conference on Designing Interactive Systems (DIS).

Morphing

Whereas printing is all about creating physical form to digital things, another critical bridge between the physical and digital world is adapting digital things to our physical world. Screens, for example, are one of the key ways that we present digital things, but their rigidity has a way of defining the physical form of objects, rather than objects defining the physical form of screens. The iPhone is a flat sheet of glass, defined by the screen; a laptop is shape of a screen. Researchers have long pondered whether screens must be flat, envisioning different forms of output that might have new unimagined benefits.

Some research has focused on making screens flexible and bendable. For example, one technique takes paper, plastics, and fabrics, and makes it easy to create programmable shape-changing behaviors with those materials. ^{16 16}

Jifei Ou, Mélina Skouras, Nikolaos Vlavianos, Felix Heibeck, Chin-Yi Cheng, Jannik Peters, Hiroshi Ishii (2016). aeroMorph - Heat-sealing Inflatable Shape-change Materials for Interaction Design. ACM Symposium on User Interface Software and Technology (UIST).

Other projects have created  stretchable  user interfaces with sensing capabilities and visual output, allowing for conventional experiences in unconventional places. ^{20 20}

Michael Wessely, Theophanis Tsandilas, Wendy E. Mackay (2016). Stretchis: Fabricating Highly Stretchable User Interfaces. ACM Symposium on User Interface Software and Technology (UIST).

Some research has even explored foldable interactive objects by using thin-film printed electronics. ^{15 15}

Simon Olberding, Sergio Soto Ortega, Klaus Hildebrandt, Jürgen Steimle (2015). Foldio: digital fabrication of interactive and shape-changing objects with foldable printed electronics. ACM Symposium on User Interface Software and Technology (UIST).

This line of research innovates in both the industrial forms that interfaces take, while offering new possibilities for how they are manipulated physically. It envisions a world in which digital information might be presented and interacted with in the shape most suitable to the data, rather than adapting the data to the shape of a screen. Most of these innovation efforts are not driven by problems with existing interfaces, but opportunities for new experiences that we have not yet envisioned. With many innovations from research such as foldable displays now make it to market, we can see how technology-driven innovation, versus problem-driven innovation, can struggle to demonstrate value in the marketplace. 

Haptics

Whereas morphing interfaces change the  structural  properties of the interface forms, others have focused on offering physical, tangible feedback. Feedback is also critical to achieving the vision of tangible bits, as the more physical our devices become, the more they need to communicate back to us through physical rather than visual form. We call physical feedback  haptic  feedback, because it leverages people’s perception of touch and sense of where their body is in space (known as proprioception).

Kim et al. ^{11 11}

Erin Kim, Oliver Schneider (2020). Defining Haptic Experience: Foundations for Understanding, Communicating, and Evaluating HX. ACM SIGCHI Conference on Human Factors in Computing Systems (CHI).

Numerous works have explored this design space in depth. For instance, some haptic feedback operates at a low level of human performance, such as this idea, which recreates the physical sensation of writing on paper with a pencil, ballpoint pen or marker pen, but with a stylus ^{5 5}

Cho, Y., Bianchi, A., Marquardt, N., & Bianchi-Berthouze, N. (2016). RealPen: Providing realism in handwriting tasks on touch surfaces using auditory-tactile feedback. ACM Symposium on User Interface Software and Technology (UIST).

Other haptic feedback aims to provide feedback about user interface behavior using physical force. For example, one project used electrical muscle stimulation to steer the user’s wrist while plotting charts, filling in forms, and other tasks, to prevent errors. ^{13 13}

Pedro Lopes, Alexandra Ion, Patrick Baudisch (2015). Impacto: Simulating physical impact by combining tactile stimulation with electrical muscle stimulation. ACM Symposium on User Interface Software and Technology (UIST).

Many projects have used haptics to communicate detailed shape information, helping to bring tactile information to visual virtual worlds. Some have used ultrasound to project specific points of feedback onto hands in midair. ^{3 3}

Tom Carter, Sue Ann Seah, Benjamin Long, Bruce Drinkwater, Sriram Subramanian (2013). UltraHaptics: multi-point mid-air haptic feedback for touch surfaces. ACM Symposium on User Interface Software and Technology (UIST).

Some work leverages  visuo-haptic illusions  to successfully trick a user’s mind into feeling something virtual. For example, one work displayed a high resolution visual form with tactile feedback from a low-resolution grid of actuated pins that move up and down, giving a sense of high-resolution tactile feedback. ^{1 1}

Parastoo Abtahi, Sean Follmer (2018). Visuo-haptic illusions for improving the perceived performance of shape displays. CHI.

All of these approaches to haptic feedback bridge the digital and physical worlds by letting information from digital world to reach our tactile senses. In a sense, all haptic feedback is about bridging gulfs of evaluation in physical computing: in a physical device, how can the device communicate that it’s received input and clearly convey its response?

While this exploration of media for bridging bits and atoms has been quite broad, it is not yet deep. Many of these techniques are only just barely feasible, and we still know little about what we might do with these techniques, how useful or valued these applications might be, or what it would take to manufacture and maintain the hardware they require. There are also many potential unintended consequences by giving computers the ability to act in the world, from 3D printing guns to potential injuries.

Nevertheless, it’s clear that the tangible bits that Ishii and Ullmer envisioned are not only possible, but rich, under-explored, and potentially transformative. As the marketplace begins to build some of these innovations into products, we will begin to see just how valuable these innovations are in practice.