Coding Horror: Will Mouse Gestures Ever Be Mainstream?

February 20, 2006

Will Mouse Gestures Ever Be Mainstream?

Darwinia is the third game I've played with mouse gesture support:

Bungie's classic 1998 game Myth used gestures in a limited way to indicate squad facing post-movement.
Lionhead's 2001 game Black and White used gestures to invoke various spells.
Introversion Software's 2005 game Darwinia offers gestures as an alternative control scheme.

I've never been comfortable with mouse gestures in any of these games. It seems like a great idea in concept, but it breaks down in execution, at least for me. I distinctly remember the awkwardness of getting my squads to face the right direction in Myth. And although there was a certain element of mysticism in scrawling spells on the screen in Black and White, the game sometimes had difficulty recognizing what I had drawn. It ultimately ended up feeling like extra work when I could have simply clicked an icon or pressed a key to cast the very same spell. Perhaps problems like these are why mouse gesture control mode is not the default control mode in Darwinia.

Games tend to be experimental when it comes to UI. But mouse gestures are slowly-- very slowly-- making their way into mainstream operating systems and applications:

There's a plugin to add gesture support to Firefox
Strokeit offers system-wide gesture support in Windows
The Opera browser natively supports gestures
Sensiva provides system-wide gestures for Tablet PCs
There's a plugin that brings gestures to Internet Explorer
Here's sample code that demonstrates how to add gesture support to a .NET app

Even without any of this software installed, you can use one very familiar mouse gesture: dragging and dropping.

I've experimented with several of the above utilities at various times, and I'm still ambivalent about mouse gestures for a few reasons:

Gestures have the one-button mouse problem. There are zero visual cues that any of these fancy gestures are possible. Additional buttons on a mouse, or toolbar buttons on the screen, can prompt a "I wonder what this does?" reaction from a new user. There is no such discoverability for mouse gestures.
Gestures are extremely mouse-centric. Have you ever tried "writing" your name with the mouse? Very few users have that kind of fine motor control. But the mouse isn't your only option. Left-hand keyboard accelerator keys can be just as effective-- without requiring you to take your other hand off the mouse.
There are only so many gestures you can draw. Once you get beyond a dozen simple strokes in the cardinal directions (up, down, diagonal, etc), you're in trouble. It's a slippery slope to something overwhelming like Graffiti.
Gestures require a dedicated accelerator key. There's no way to automatically detect a mouse gesture; they have to be manually initiated via a button or keypress. In most apps, you hold down the right mouse button to begin your gesture, then release it when you're done. It's one more accelerator to remember. And it's now possible to accidentally trigger a gesture when all you wanted was the right-click menu.

In short, gestures are for extremely advanced users only, perhaps even more so than traditional keyboard shortcuts. But they do have some specific uses that are interesting:

"The motion of performing a gesture is more natural than sliding the mouse over to a button or menu," he said. "And because it works anywhere in the window (not just on the button), it saves a bit of time and effort, especially as screens get bigger and you have to move farther to reach a button."

Rather than mousing over and clicking on the relatively small minimize, maximize, or close buttons, it's faster to invoke a gesture right where your mouse is to perform the same action. This is particularly nice when the gesture has some conceptual mapping to the activity: drawing a diagonal slash to close, down to minimize, up to maximize/restore, etcetera.

A mouse gesture on the Wikipedia page (close window)

For mouse gestures to become mainstream, they need to be simple, discoverable, and most definitely system-wide.

Posted by Jeff Atwood View blog reactions

« Darwinia

C# Snippet Parity »

Comments

I have strokeit installed on my laptop and my work computer - everyone at work has it and it's extrememly annoying when someone won't install it for whatever reason (standard installations are great when you use other people's machines).

The only gestures I use consistently are back and forward, which work in Eclipse to go to the next/previous edit locations. I even have back/forward buttons on my mouse, butholding right-mouse and just flicking back and forward just feels extrememly natural.

I very occasionally use some of the other gestures, but I find it hard to remember what they all are, so it's easier to just click the button usually.

I think having maybe up to five system-wide gestures would be a good start, but more than that and you will start getting massive overload.

Ian Tyrrell on February 20, 2006 10:30 PM

I explained the forward and back mouse gestures in Firefox to my parents like this: think of it as turning the page of a book. Your right finger goes down, you draw it across the surface of the page, and the page turns. And they GOT it. Mouse gestures aren't mainstream, not because they require a sophisticated user, but because developers make them so. Or maybe they're just not being explained very well.

Burton Johnson on February 20, 2006 11:20 PM

Once I got used to Mouse gestures in Firefox, there was just no way back for me.
It so easy to do the common tasks.
Thinking about it, I'm only two to four common gestures that I use all the time. Mostly close tab / restore tab / move left tab / move right tab.

For those, it's perfect.

Ayende Rahien on February 21, 2006 1:07 AM

I just realized that I frequently right-drag ZIP archives to move and unzip them. Which means Strokeit's use of the right mouse button in the Explorer shell is a non-starter for me. I guess I can configure it to use the middle (wheel) button..

Jeff Atwood on February 21, 2006 1:22 AM

It's interesting that two people have already brought up forward and back.

Personally, I think forward and back belong on every single mouse as the thumb buttons. I use forward about 1/100th as often as I use back, so every mouse I use has to have a dedicated back button-- it's non-negotiable.

As far as I'm concerned, forward and back are completely redundant gestures. I'm not against them, but they clearly belong on the mouse.

Jeff Atwood on February 21, 2006 1:47 AM

Bizarrely overprovisioned mice like mine are another good solution, too, since I have buttons to do all that on it. =p Every few days I get tired of mousing around all day and think "Damn, when are my touch-screens going to show up?" But I still can't afford a Cintiq just for that and Photoshop.

( <a href="http://www.wacom.com/lcdtablets/index.cfm">http://www.wacom.com/lcdtablets/index.cfm</a> )

When some developers still can't grasp the idea of standard windows/linux shortcut keys, I have very low hopes about a standard of gestures emerging, though.

Foxyshadis on February 21, 2006 2:00 AM

Just a few thoughts... I don't know if gestures are necessarily for advanced users only. It may simply be a matter of old habits dying hard. Mouse targeting + clicking only seems obvious because its been used for such a long time.

Its not really obvious in daily usage that items should or shouldn't be clicked (or double clicked). Designers have gone as far as to make "buttons" resemble actual buttons to reinforce the behavior. Even if a mouseover cue happens, nothing about that actually says "click me". The website www.dontclick.it shows just how deeply ingrained the learned behavior of pointing & clicking is.

As far as gestures being mouse-centric - well, we are using a mouse after all :) ...but maybe gestural implementations can be far deeper with a stylus rather than a mouse simply because of the form factor.

It's true that there are only so many gestures that can be drawn, but how many are really necessary? Even with standard mouse usage, such as drag n' drop, we regularly invoke "combos" of multiple mouse controls simultaneously.

Gestures do require an accelerator key, but many standard mouse operations do also - we don't expect them to be immediately accessible

Gary Boodhoo on February 21, 2006 2:05 AM

I agree that one of the problems are that mouse gestures for the most part remain hidden. Perhaps one solution could be hinting. If you start to draw a gesture, based on the context of your gesture a suggested hint appears in a pale, translucent colour, along with an indication of what it does.

But still, that requires initializations. Apart from a few good analogies (that's a fantastic one there Burton Johnson) you can't really do enough iconic symbolism like you can with buttons and keyboard shortcuts anyway.

My question would be do novice users really need mouse gestures? The more we move to options of how to do things, the more things become complicated. Keyboard shortcuts are an example similar to gestures, in that they are for the large part hidden and not used by novice users. However, they have to great advantages over gestures. They are more intuitive and easier to remember, and they make things quicker because you don't have to move from keyboard to mouse as you're typing. Mouse gestures don't do that, as you can already navigate to a button or menu.

I have gestures installed in firefox, but that's only because I use the "tap right mouse button then left" feature for back, and the reverse for forward (I love the ergonomics of my 2-button scroll wheel mouse and I'm not willing to give it up lightly for a 20-button behemoth that splays under my non-average hand) For everything else, I can use the pull-down menu's as I don't use them often enough to let them clutter up my display, or remember gestures for them.

[ICR] on February 21, 2006 4:28 AM

This is a slight aside from mouse gestures, but talk of fine moust control reminded me of it: When I visited a chiropractor a few years back she observed that my fine mouse control was damaging my shoulders- the muscles on my right shoulder have developed to put pressure on the nerves so that my right hand has very little shake to it, which does mean my mouse use is very controlled, but also means that my shoulder is very inflexible and the over-developed muscle then starts pulling the muscle on my left shoulder out of place and generally wreaking havoc with my back. In her words, at 26 I had the shoulder of a 60 year old man.

Apparently this is very common among those of us who use mice regularly. Her recommendation was to make sure I moved and shrugged my shoulders from time to time during the working day and ideally to use the mouse in my left hand for a while and then to alternate a few months at a time. I would like to do that but I cannot make myself be ambimoustrous.

Ben Moxon on February 21, 2006 5:49 AM

> I have gestures installed in firefox, but that's only because I use the "tap right mouse button then left" feature for back, and the reverse for forward

If that's the *only* reason you're using gestures, you really have to get over this phobia! The back (thumb) button is absolutely essential on a mouse, IMHO.

I do like mouse gestures quite a bit for min/max/close. It saves me a bunch of mental effort not having to Fitt's Law my way over to the button cluster in the upper right of each window.

Plus on a multi-mon system, the "edges" of the screen don't always stop the mouse pointer.

Jeff Atwood on February 21, 2006 6:36 AM

I just don't see the point in buying a mouse that I don't like for two extra buttons that I'm only ever really going to use for back and forward when I can tap the two I have just as fast, if not faster, than moving my thumb to the right button and clicking it.

[ICR] on February 21, 2006 6:45 AM

i think to popularize mouse gestures (i use them with firefox), various tools or applications must directly support them...at least a set of them. that way it will gain more acceptance....but a seperate button for gestures...hmmm maybe not.

javed on February 21, 2006 8:33 AM

Slightly off topic, but I swap the mouse back and forth frequently. I generally use it on the left at work, and on the right at home.

I don't use guestures... but I do like the thumb buttons for back/forward. Got used to them at a previous job, and don't have them at work now... :(

Eric D. Burdo on February 21, 2006 8:47 AM

I guess I have another point of view. I don't want to become *more* dependent on my mouse, I want to be able to do more things straight from the keyboard. No matter what neat things I can do with gestures, they require me to have a hand on the mouse which means its not on the keyboard. I don't see how a little gesture on the mouse to move back or forward in a browser can be more efficient then Backspace and Shift-Backspace.

I don't just do page scrolling and navigation in a browser, I also hit Alt-D to type in a new aDdress or Alt-G to enter new search terms in the Google toolbar. If I'm constantly moving my hand between the keyboard and mouse just so I can do these little gestures, what have I gained?

David Avraamides on February 21, 2006 8:51 AM

So what does drawing a breast on top of a Wikipedia page do now?

Tom Clancy on February 21, 2006 8:59 AM

A related topic is Pie Menus. These bridge the gap nicely by providing the power of gestures with visual feedback. Check out RadialContext (<a href="http://www.radialthinking.de/radialcontext/)">http://www.radialthinking.de/radialcontext/)</a> and easyGestures (<a href="http://easygestures.mozdev.org/)">http://easygestures.mozdev.org/)</a> for implementations in Firefox.

Eric Hoch on February 21, 2006 10:47 AM

I discovered StrokeIt a few years ago and it's one of the things I can't do without anymore.

Mouse gestures negate Fitt's Law for common tasks. There's no need to "acquire" a target with the mouse, because at any time your mouse is already in the right position. "Back" and "forward" is very common while browsing. I can do this by:

(a) click the back button, but that requires me to acquire (locate and move the cursor to) the button and click it;
(b) use a keyboard shortcut. Possible, but it breaks the convenience of just using the mouse by itself, which I like sometimes;
(b) use a mouse gesture. It's as easy as depressing the RMB, dragging left, and releasing.

Same thing goes for minimizing or closing an application. I can try to hit the little icons in the top bar, or just make a gesture at the point where my mouse happens to be.

Controlling Winamp: I'm using the mouse, and I want to change the song. I can bring up Winamp and click next song (first clicking the tiny tray icon at the bottom of the screen, then going up to the winamp window and clicking again); I can use a shortcut (Ctrl-Alt-PgDn, not very convenient). Or just make a simple gesture that I've defined (right-up = next song, left-up = previous song).

Gestures are only useful for actions you normally perform a lot, but it's amazing what kind of impact they have on my workflow.

> I just realized that I frequently right-drag ZIP
> archives to move and unzip them. Which means
> Strokeit's use of the right mouse button in the
> Explorer shell is a non-starter for me. I guess
> I can configure it to use the middle (wheel)
> button..

Not necessarily, I still do that all the time with StrokeIt running. Mouse gestures have a timeout, after which Windows takes over with normal right-click behaviour.

So just right-drag as usual, wait two seconds, and it will behave as usual.

RiX0R on February 21, 2006 11:32 AM

P.S: Opera's "close" gesture (an "L"... down-right) is much more convenient than StrokeIt's standard "close" gesture (a "C"... or a breast if you prefer :).

I recommend it to everyone.

RiX0R on February 21, 2006 11:37 AM

> I want to be able to do more things straight from the keyboard. No matter what neat things I can do with gestures, they require me to have a hand on the mouse which means its not on the keyboard.

The whole point of shortcuts is to provide several so people can choose the ones that work best for them.

So for "back" in the browser, we can use any of these:

- the Back button on the mouse
- the Backspace key
- shift+mousewheel down
- right-drag left (gesture)
- the toolbar button

Although for efficiency's sake, it does make sense to optimize for either mouse or keyboard depending on what you're doing at the moment. If you're typing a lot (like I am now), you'd want to opt for keyboard shortcuts to keep both hands on the keyboard. If you're just browsing around and not typing anything, maybe sticking with the mouse makes more sense.

Jeff Atwood on February 21, 2006 12:18 PM

Indeed. Shortcuts are needed for both keyboard and mouse to decrease the need for homing.

[ICR] on February 22, 2006 6:37 PM

The game Arx Fatalis also used Gestures to perform magic spell, quite well IIRC/

Russ on February 27, 2006 9:36 AM

Just seconding the relationship between gestures and their similarities to pie menus. You can see them in use in the Sims so the usability of them seems to be fairly simple, with the provided contextual information. Pie Menus also allow movement memory to perform fast actions without having to wait for the visual queues, thus providing a similar experience to mouse gestures in some implementations.

Here's something I wrote about Pie Menus and Set-top boxes a while back <a href="http://www.benmeadowcroft.com/reports/piemenu/">http://www.benmeadowcroft.com/reports/piemenu/</a>

Ben Meadowcroft on April 4, 2006 5:18 AM

I really like the pie menus. I use them in Battlefield 2 all the time.

The only criticism I have is that once you move beyond the 4 cardinal directions and "no movement" (center), it gets hard to remember what direction does what. So the optimum number of commands is limited to 5.

Jeff Atwood on April 4, 2006 10:48 AM

the talk about forward and back gestures... and mouse buttons for forward and back...

On my browser (avant browser), i have a guesture for forward and back that is so simple that I can't live without it.

to go forward, click left and hold, then click right. click multiple times to go forward multiple pages. same thing, only in reverse for going back.

its a lot quicker than draging your mouse back and forth on the page, and it still has the benifit of being usable nearly anywhere on the page. And because of the way I move my mouse, using my thumb to hit buttons is just annoying, because it means i can't move at the same time.

People who move their mouse with their palm, are destine to get carple tunnnle syndrom

apeinago on January 6, 2007 10:30 AM

Mouse gestures are THE BEST THING
EVER EVER EVER EVER EVER EVER EVER EVER!!!!!!!!!!

SIMPLY SPEAKING THEY SHOULD BE USED BY EVERYONE AND EVERYONE!!

PERIOD

ALL HAIL STROKEIT

mouse on February 16, 2007 12:24 PM

Mouse gestures are THE BEST THING
EVER EVER EVER EVER EVER EVER EVER EVER!!!!!!!!!!

SIMPLY SPEAKING THEY SHOULD BE USED BY EVERYONE AND EVERYONE!!

PERIOD

ALL HAIL STROKEIT

mouse on February 16, 2007 1:07 PM

I found anoter program like StrokeIt thats free that looks miles better but needs a key press to start http://silentdevelopment.blogspot.com/2008/04/gesture-magic-10-released.html

Steve on May 4, 2008 3:02 AM

hi, I am currently working on a mouse gesture algorithm which combines the traditional algorithm with computer learning methods. the main idea is algorith will learn (in time) how a specific user draws a specific shapes. everyone draw a letter in a different way so I am trying to make a user oriented mouse gesture algorithm.

actually I am designing a RPG video game in which a player will have nearly 100% freedom of sword movement. I will share my mouse gesture engine with anybody who is interested.

artecleas on October 8, 2008 12:50 AM