Clayton Miller has posted an intriguing concept demonstration for multi-touch interaction as applied to traditional desktop systems. Rather than turning our monitors into touch screens, he suggests adding a large trackpad capable of sensing all ten fingers. This gets around the major problems that he points out as the primary roadblocks to touch interactions at our desks: fatigue from non-ergonomic techniques and occlusion by our own hands.

Clayton’s video artfully explains the problems he is trying to solve, the rationale for his approach, and both the hardware and software that form the solution. As such, it is a much more effective presentation than those recently released by Microsoft.

I can certainly see this concept in use. The track pad on Apple’s Macbooks is moving in that direction. However, I’m of the belief that one of the greatest benefits of a touch interface is direct manipulation. Clayton’s solution is still just as indirect as the mouse—just with nine more cursors. We can learn to type and play the piano without seeing what our fingers are touching due to the static placement of the keys and the tactile feedback they give us. Clayton’s pad would offer neither of these, and I imagine it would take a lot of effort to become proficient in its use.

Finally, I have to wonder about its actual utility. He presents a good case for its usefulness in managing windows, but that’s not what I use a computer for. I don’t sit down at my Mac thinking, “I’m going to move some windows around.” I use my computer to pay bills, prepare to teach my class, specify UI designs, and write blog posts. How will 10/GUI help me do those things more efficiently? I’m not saying it can’t, but I’d like to see it applied to more important tasks. Managing desktop clutter is just a place to start.


Perhaps I’m just jaded, but I can’t get excited over the two videos Gizmodo has put up demonstrating Microsoft’s Courier concept. First of all, Gizmodo is treating it as if it is a unique, unforeseen vision for a tablet computer. What exactly makes it so innovative? Is it the folding, two-page form factor? I don’t think so. Last year saw Negroponte’s announcement of the second-generation OLPC XO: the XO-2.


And there was the Canovo that made the rounds in 2007.


Then, of course, we’ve seen the dual screen eReaders by ASUS (2009) and iRiver (2007).

ASUS Eee Reader

iRiver E-BOOk

Well, then, is it the stylus? Surely not—we’ve had tablets and PDAs for years that have those. The iPhone was revolutionary for doing away with the stylus. Okay, could it be the multi-touch? Obviously not, as we’ve seen that in the iPhone, Surface, and any number of other demonstrations. And the combination of the two is already available in tablets such as Panasonic’s Toughbooks.

I’ve ruled out the form factor and input methods, so it isn’t the device itself or the technology behind it. Is the user interface especially innovative? There are some interesting ideas in it. Using the spine as a place to tuck things that you want to move from one page to another is a clever implementation of cut and paste, but beyond that, I don’t see any interaction patterns that I haven’t seen before. Quite frankly, I don’t find the scenarios to be very compelling. The UI has that “visionary concept” quality to it that suggests it hasn’t been fleshed out much beyond the script. The handwriting recognition is flawless, the screens show only the controls that provide access to the features showcased, and complex actions, such as selecting and copying a graphic and two columns of a 3-column table, are accomplished with a single touch of a finger.

Yes, it’s an interesting concept, and yes, there is value in creating such visionary explorations. However, I’ve seen far too many of these that don’t result in anything other than inspiration (not to say there is anything wrong with that), and Microsoft in particular has a horrible track record of delivering innovative products. Please pardon me for some uncharacteristic pessimism. I’m not going to take a deep breath, let alone hold it.

Interaction 10 Proposal

The deadline for session proposals for the Interaction 10 conference is rapidly approaching. I submitted a demonstration today. All proposals are publicly available on the conference website. Not only can you read them, but make comments that will be taken into account when the committee decides which proposals to except. Not only that, but you can leave comments as to how the potential presenter can improve his or her submission. We have until October 1st to revise our proposals.

Wicked Problems Inspire Creative Solutions: A case study of software design for industry
We are well aware of many great examples of Interaction Design found on the web. Consumer-oriented services and mobile devices like the iPhone capture the public conscious and are the stars of our discipline. Software developed for industry, however, is much less likely to have an interaction designer involved to begin with. The irony is that there are typically much more difficult challenges to tackle in industrial contexts—designers are sorely needed.

I’ll be demonstrating a unique application that was designed to meet an interesting combination of requirements. The tool is used to record information about complex electrical systems in large facilities, such as factories and hospitals. The demonstration will show how a user interface was designed for needs and constraints such as:
  • mobile use
  • harsh environments
  • touch/stylus input
  • relatively small display
  • visualization, navigation, and editing of very large, complex electrical systems
  • quick, non-linear access to individual parts of said systems
  • repetitious data entry
  • validation of entered data

The resulting solution required creative, custom enhancements to existing design patterns. Of course, it also required close work with the technicians that were to benefit from it. The presentation will include examples from the collaborative process, from initial research and a failed first attempt to “a day in the life” shadowing and high-fidelity prototypes.

In summary, the session will present a complete case study of an industry-oriented software design project through demonstration of prototypes and, ultimately, the final, working application.

This isn’t up yet on the conference site, but eventually you will be able to comment on it here.

Cursed Cursors

I’ve done a fair amount of UI design for applications intended for use on tablet PCs. That means that the software is running on Microsoft’s tablet version of Windows. It has a number of additional features, such as a pop-up keyboard and handwriting recognition, that is specific to stylus and finger input. Typical of Microsoft, however, they haven’t really thought through the details. One thing that has always bugged me is the cursor. If I’m using a stylus, I have no need for a cursor whatsoever. The cursor is a stand-in for direct manipulation. That is, it represents your finger when you have to use something like a mouse to interact with the UI indirectly. On a touch screen, you can tap things—there is no click.

The cursor is superfluous, yet Microsoft left it there. Every time you tap, the cursor jumps to that point on the screen. This often leaves it hovering over a button, and what does that end up doing? Well, for one, it partially obscures whatever icon or label is on the button. Even worse, it usually causes a tooltip to pop up, as the cursor just stays there. The tooltip remains, covering whatever portion of the display it happens to cover, until the standard timeout is reached, or until you tap somewhere else, moving the cursor to a new location. Tooltips are not an interaction design pattern intended for touch interfaces. I guarantee you will not find tooltips in the iPhone UI.

I received email today from Apple’s App Store advertising the latest iPhone apps. I got excited to see that LucasArts has released The Secret of Monkey Island Special Edition for the iPhone. The original was the very first game I purchased for the Macintosh—my LCII. It’s one of my all-time favorites, and I was ready to buy it, but I read the reviews first. Every one that I read complained about the UI. Apparently, they didn’t adapt the UI for the iPhone. Rather than just tapping things, you actually have to drag a cursor around the screen!

Epic Fail!

Touch Book

I’ve worked on a number of projects in which the goal was to deliver computer-enhanced capabilities to technicians in the field. We focused on tasks such as data logging, repair procedures, diagnostics, and the like. As part of the solution, we’ve evaluated PDAs, laptops, tablets, and even wearable computers. Most recently, I’ve worked with Panasonic Toughbooks that can pull the Transformer-like move of flipping the screen around to go from laptop to tablet. This makes for a versatile solution, allowing the user to carry it around and enter information with the stylus, but still type when he has a place to sit down. The one issue is that these laptop-tablet hybrids can be pretty hefty.

Always Innovating has announced an interesting product that is the first “netbook” I’ve found at all interesting. The Touch Book is a small laptop that has a detachable keyboard. Unlike the laptops with the flip-over screens, the guts of the computer are inside the display portion, rather than the keyboard portion. This allows the keyboard to be removed, leaving an 8.9” touchscreen that weighs less than two pounds. Rather than a hard drive, it uses a micro SD card for storage, and they are claiming 10 to 15 hours of battery life. They’ve even magnetized the device so that the display can hang on the fridge (or whatever metallic surface happens to be in the vicinity).

While they are marketing this as a consumer device, it seems to me to be the perfect device for a mobile field worker, except for one thing. A ruggedized version would weigh more and be a bit pricier, but would be necessary for many of the environments I’ve designed for.

Another thing I noticed is that the company has given credit to the designer, Fred Bould, on the product page. This is a rarity, and it’s obvious that Always Innovating has put much import on the design of the device. I think it’s going to pay off.

Interaction Design in the National Design Awards

The Cooper-Hewitt’s 10th National Design Awards have been announced for 2009. This year sees Interaction Design as a new category, and a quite welcome one. A true sign of the times, the winner in this category is Jeff Han’s Perceptive Pixel, a firm “dedicated to the research, development and deployment of multi-touch interfaces for the knowledge worker.”

If one can be judged by the organizations she associates with, Lisa Strausfeld, one of two other finalists, can’t do much better. She has master’s degrees from both Harvard and MIT, where she worked in the Media Lab. She is a partner at Pentagram and teaches at Yale. I’ve been an admirer of her work in information visualization, and she is responsible for the OLPC’s Sugar OS.

The other finalist I didn’t know by name, but I had heard of one of their projects—huge touch screen tables designed for the National World War I Museum. “Founded by graduates of the MIT Media Laboratory, Potion is a design and technology firm that specializes in interactive installations.”

In addition to the new category, this year’s Lifetime Achievement award goes to Bill Moggridge, one of the pioneers of Interaction Design, and in fact, the man who coined the term.

Designing for Touch Screens and Interactive Gestures

I attended Dan Saffer and Bill DeRouchey’s workshop at Interaction ’09 titled Designing for Touch Screens and Interactive Gestures. While a lot of good information was communicated (Did you know that the largest portion of the brain is devoted to your mouth, while the second largest controls your hands?), much of it was old hat for me. I’ve been designing UIs for tablet PCs with touch and/or stylus input for about as long as I’ve been working in the field, and most of the considerations are the same. For example, it is better to place controls at the bottom of the screen, rather than the top, because in reaching for the controls at the top, you obscure the screen with your hand and arm.

What I found most valuable was the exercise. I’ve never thought highly of paper prototypes. I know, a lot of designers swear by them, but I find them much too tedious to be of use. I can spend the same amount of time or less creating an interactive prototype in Director that is the same or greater fidelity. In my opinion, paper prototypes work well for very simple interactions. Once you start rigging scroll bars with string, you are taking things to a ridiculous extreme.

The workshop, however, made me realize that the new frontier of large scale, touch sensitive displays is the perfect application for paper prototypes. I stated that paper prototypes work well for simple interactions. I now believe it can be measured as a bell curve. In the middle, there is a large area of medium complexity that is too tedious to model with paper and only moderately time consuming to represent in software. Then, as you push beyond the fat point of the curve, you get to a level of complexity that is quite difficult to model with our current, computer-based tools, and paper comes to the rescue.

Our class had been broken into several groups, and we were tasked with designing a music purchasing experience for a retail space. We were given a size constraint, which was, conveniently enough, approximately the size of the circular dining tables at which we were seated. We used paper, markers, and tape to hastily create a very rough prototype of our solution, which borrowed heavily from Microsoft’s Surface. I was particularly fond of our solution, but I was very impressed with how well paper worked in this instance.

Testing with Paper

It lends itself perfectly to the problem space. You can’t prototype at that scale on a computer screen. You would have to do some pretty fancy stuff to get a projector hung straight down from the ceiling, and then mimic the users motions in near-real-time to make it at all feasible in software. With paper, you can make every element to size, and the test subject can slide pieces of paper around as much as they want.

I don’t know how long it will be before I have the opportunity to design such an interface for real, but I do know that when that time comes, I’ll likely be adding paper prototyping to my toolbelt.

First Touch

I mentioned in a previous post that a feature of Interaction ’09 was the Tangible Interaction Café. Manifest Digital provided a Microsoft Surface for us to fiddle around with. This was my first hands-on experience with the table, and while it was interesting, I was a little underwhelmed. It wasn’t as big as I imagined it would be; it’s the size of a smallish coffee table. The resolution is also quite low. The interactions were very natural, however, and it was fun to spend a few minutes flicking photos around and playing with the water table “screensaver”.

One thing that made it a bit more engaging was that Manifest had developed some software expressly for our conference. They had it pulling in photos and tweets from the conference feeds, which was a nice tie in, but the icing on the cake was the interface with Crowdvine, the social networking site tied into the conference. Everyone who had registered with Crowdvine had a 3-D barcode on the back of their nametag. When I placed my nametag on the Surface, it recognized me and pulled in my photo from Crowdvine.

Microsoft Surface

The red “connect” swash could be dragged onto another person’s photo, which would send them a message through Crowdvine saying that I connected with them on the Surface at Interaction ’09. While this is a rather trivial example, it turned what could have been a short, distraction with a novelty item into a more relevant and engaging experience.

Patently Untouchable?

In yesterday’s post, I suggested that multitouch is going to become an expected feature of touch-based user interfaces. If your product doesn’t have it, you will reap the ire of the customers you manage to attract. What then of the patents Apple is said to have on multitouch? Will this keep other companies from being able to include such interactions? Is Apple stifling product innovation by protecting its own?

In his article The iPhone Multitouch Patent Myth on RoughlyDrafted Magazine, Daniel Dilger explains,

“Anyone who has ever glanced over a patent filing knows that one can’t simply patent a word like multitouch. Patent filings require making specific claims that define an invention and describe why it is new and useful in some original way that hasn’t been done before.”

He goes on to point out that the title of Apple’s patent clearly indicates that it isn’t a patent on multi-touch as a method of input, but a much more specific claim against innovative implementation of touch input behaviors: “Touch screen device, method, and graphical user interface for determining commands by applying heuristics.”

I read part of the patent, but I’m not a lawyer, and I can’t decipher legalese. I get little hints and suggestions about what the claims cover. As John Mahoney states on Gizmodo, “The meat of every patent is a list of claims, and it is the claims and only the claims that spell out exactly what can get you sued and what can’t. Unfortunately for us, but very fortunately for the thousands of patent lawyers hoping to feed their families, claims are written in a language not comprehensible to normal humans. The goal is to be both incredibly vague and legally specific at the same time.”

To know for certain what Apple is protecting—what it will fight over—we’ll have to wait and see. And that may be the point. Again, from Gizmodo, “…it’s impossible to identify a single patent that has a lock on the iPhone’s multitouch magic as we know it. That patent probably does not exist. But here’s the key—patent wars are intrinsically cold wars. They entail both sides jacking up their arsenals with as many patents as possible, with hopes of scaring their adversaries out of even attempting to try something.”

Regardless of the actual coverage of multitouch and the specific gestures used on the iPhone, it may stifle innovation through intimidation.