The Apple Motion Sensor As A Human Interface Device

© Amit Singh. All Rights Reserved. Written in March 2005

Background

I first described the working of Apple's Sudden Motion Sensor (SMS, also known as AMS — in this chapter, we will use "SMS" and "AMS" as interchangeable terms and prefixes), initially a feature of the PowerBook line of computers, in "The Sudden Motion Sensor". My original reason for investigating the sensor was to fabricate some interesting programming examples for my book on Mac OS X internals. The discussion of the SMS generated a substantial amount of interest — perhaps more than I had expected. It is quite interesting to observe how one's fantasy is tickled by abnormal ways of doing the normal.

In the following discussion, I assume familiarity with the concepts and terminology introduced in "The Sudden Motion Sensor".

Early Sudden Motion Sensor Tools

I initially provided the following Mac OS X applications that use the orientation data retrieved from the SMS:

AMSTracker proved to be the most interesting application. Whereas the others are software "toys", AMSTracker can be used as a fundamental building block to create arbitrary applications that do something interesting based on the computer's orientation.

Still More Interesting Uses

I had also mentioned other ways in which the SMS could be employed, such as for scrolling in applications and for input in video games. The latter was constantly brought up in the feedback I received from "the Internet". Apparently many owners of SMS-equipped PowerBooks wish to experience physical motion-based gaming.

The games most suitable for being played using the SMS would be those that wholly or primarily require directional input. The x and y components of the SMS orientation data can be mapped to the motion of a mouse in a 2D plane, or to specific keys (say, the arrow keys) on a keyboard. Thus, it is conceptually straightforward, both for the programmer and the end-user, to use the SMS as a human interface device (HID).

Using the SMS as a Human Interface Device

While one could modify existing applications, including games, to accept SMS orientation data as input, it would be unnecessary and sub-optimal to do so in most cases. A surprisingly large number of people seemed to think that applications and games must be specially written to be "tilt-sensitive". While such an approach may be justifiable in certain special cases (such as when you need to process the orientation data for a purpose other than human input), doing so would typically be an impulsive reinvention of the proverbial wheel. If the application of interest already receives mouse or keyboard input, it is the SMS that should masquerade as a mouse or a keyboard. After all, one does not create games from scratch for specific models of keyboards, joysticks, or mice. In most cases, the transparent approach would allow the SMS to be used with target applications without any modifications to the latter. This is especially desirable when you do not have access to the source of a target application.

An SMS-based HID could be implemented in two broad ways, as described below.

The Ideal Solution

A particularly seamless solution would be to create a HID driver for a pseudo-device whose input "signals" are the motion sensor's orientation data. Such a software mouse or joystick would then be available to applications simply as yet another input device.

While an SMS-based HID driver is feasible, it is somewhat easier to adopt a relatively "Quicker and Dirtier" approach.

The Pragmatic Solution

Given my utter lack of free time, I had promised myself (and some other people) that I would not experiment with motion sensors beyond what I had already done. There are several justifiable reasons for such a vow:

Now, I had played a game (on real hardware) that was modified (hardwired, for experimentation) to use the motion sensor's data, and it was a very entertaining and unique experience. Moreover, I had received numerous requests for an SMS-based game. In light of all this, I decided to create a user-space "mapper" of SMS data to a HID. While not as clean a solution as a HID driver, it is more flexible and easier to use in many cases. These properties make it particularly suited for experimentation by the end-user.

SMSServer: SMS Over the Network

As alluded to earlier, I still needed a way to simulate the behavior of a user-controlled motion sensor without involving the mouse or the keyboard on the "local" machine. A simple solution was to extend the existing local simulation across the network.

SMSServer is a program that makes the data from a real or fake (mouse-based) SMS available over the network. My SMS API can thereafter use an additional type of SMS: network-based. A client of this API is oblivious of the source of the orientation data, which can now be one of the following:

It would be interesting to come up with creative uses of across-the-network orientation data. Specifically, what could you do with a computer's physical orientation being transmitted to someplace else in near real time?

Let us now look at the program itself, which I call AMS2HID.

AMS2HID

AMS2HID Main View
Figure 1. AMS2HID

AMS2HID is a prototype application that translates raw SMS orientation data into input data from a keyboard or a mouse. It presents the translated data to other applications through synthesized keyboard and mouse events. Thus, you can use an SMS-equipped PowerBook as an orientation-driven mouse or keyboard. Perhaps the most interesting and entertaining demonstration of AMS2HID's capabilities is its use for playing a variety of existing games.

AMS2HID performs the SMS-to-HID channeling for a single application at a time. Whenever its target application is active (that is, it is the "front" application), AMS2HID activates its event synthesis. Conversely, when the active application is not AMS2HID's target application, AMS2HID is dormant.

In a pinch, the use of AMS2HID could be described as the following (refer to Figure 1): you select a HID personality (mouse or keyboard), and a target application from a list of currently running applications. The target application will thereafter receive HID events based on the PowerBook's orientation and movement.

Note that unlike AMSTracker, AMSVisualizer, and StableWindow, the version of AMS2HID available on this page requires an SMS-equipped PowerBook: it will not simulate the SMS. I may make SMSServer available in the future. An important benefit of SMSServer is that AMS2HID can reside on any Mac OS X computer, and may be remotely driven by another computer. The driving computer may or may not be SMS-equipped. The latter case is akin to a "remote mouse".

The AMS2HID User Interface

Let us look at the tunable parameters available in AMS2HID for mouse-event synthesis, with reference to Figure 2. Corresponding parameters and their semantics for keyboard-event synthesis are identical except when otherwise noted.

AMS2HID Main View
Figure 2. AMS2HID (Mouse)
  1. Select a HID personality (such as "Keyboard" or "Mouse") using this pop-up button. SMS data will be translated to events corresponding to the selected device.
  2. Select a target application. HID events will only be synthesized for this application, and only when it is the "front" application.
  3. Select whether to run in "Normal" or "Test" mode. Although unlikely in practice, it is possible under certain circumstances for the user-interface to become unresponsive to events from the physical keyboard or mouse. This may happen if the combination of AMS2HID settings, the PowerBook's orientation, and available resources is such that more events than what can be handled are being generated. In such a case, you may not be able to interact with AMS2HID, its target application, or perhaps even the rest of the user interface. If you have a sane "cold zone" defined (see item 11), keeping the PowerBook level and steady will quiesce AMS2HID. Alternatively, logging in from another computer, say via SSH, should allow you to terminate AMS2HID from the command line. Finally, Test mode allows you to use AMS2HID with a 60-second auto-reset timer. Regardless of what state AMS2HID and its target application, if any, are in, AMS2HID's state will reset to no HID personality being selected. It is recommended that you test AMS2HID in each new scenario before running it normally.
  4. When AMS2HID is running in Test mode, a countdown timer appears in this area.
  5. This area shows the raw orientation data retrieved from the SMS.
  6. Select this button to cause the mouse pointer to visually (in addition to logically) move on the screen in response to SMS-caused events. De-selecting the button causes the pointer to remain visually stationary. The corresponding setting for the keyboard personality is "repeat keys", which simulates a rapid sequence of key presses and depresses, as opposed to a key being held continuously.
  7. This specifies the frequency at which the SMS is sampled, and thus represents the maximum number of events that can be generated in a given time interval. 60 Hz is an appropriate frequency for a mouse.
  8. This setting is used to reverse the logical direction of the x-axis. It can be used when AMS2HID's sense of direction along the x-axis is the opposite of that of the SMS.
  9. This slider, initially at zero, can be moved left or right to correct the "zero-error" of the SMS, if it has any. Zero error can be determined by keeping the PowerBook on an approximately perfectly level surface, and noting the raw orientation values. If the x and y values are not zero, then correction values with the same magnitudes but opposite signs can be specified. However, even if you do have a minor zero-error, you should try AMS2HID with its default settings before correcting the error.
  10. The sensitivity value represents the minimum "delta" for which SMS2HID would generate an event. The maximum sensitivity value corresponds to the minimum delta.
  11. A "cold zone" along an axis is a line segment that defines a range of SMS values for which AMS2HID will not generate any events. It corresponds to the center of the display screen in case of the mouse, and a no-keypress zone in case of the keyboard. For example, an x-axis cold zone specification of 10 means that unless the SMS x value is more than 10, or less than -10, no events will be generated for that x value. Thus, the combined x and y cold zone values define a rectangle. This feature is useful since it is normal for the SMS values to fluctuate by minor amounts even when the PowerBook is perceptibly still.
  12. Settings 8 through 11 are y-axis settings corresponding to those discussed above for the x-axis.
  13. Since the ranges of possible SMS x and y values are different from typical screen dimensions, transforms must be applied to scale SMS values to arrive at the locations of the synthesized events. If you have multiple displays, only the resolution of the primary display (the one with the menu bar) is used in these calculations. This area shows the resolution in use.
  14. This area shows the currently active application.

Experimenting With AMS2HID

Let us briefly discuss two uses of AMS2HID: first, to play video games, and second, to scroll in an application.

Playing Games
AMS2HID Movie
Figure 3. Neverball Like Never Before

AMS2HID allows you to play games using the PowerBook itself as a controller — the computer's motion in physical space provides input to games. As noted earlier, any game whose primary controls are directional should work well with SMS2HID. This includes many car-racing or driving games, where "left"/"right", "up", and "down" can be used for steering, acceleration and braking, respectively. Flight simulators and maze-based puzzle games are other likely candidates.

A game extremely well-suited to AMS2HID is Neverball, in which you "tilt the floor to roll the ball through an obstacle course before time runs out". In this case, the physical tilting of the computer closely matches the virtual tilting required in the game to result in a very immersive and enjoyable gaming experience.

Neverball is available for Mac OS X, among other platforms. Its source code is available under the terms of the GNU General Public License (GPL).

The following is a rough sequence of steps needed to play Neverball through AMS2HID:

You should also try Neverball with the mouse personality. It is possible that you find that the mouse offers finer-grained control in the game.

Movies

This movie (QuickTime format, no audio, 2.96 MB) depicts the use of AMS2HID for playing Neverball.

This movie (QuickTime format, with audio, 5.44 MB) depicts the use of AMS2HID for playing several games, such as a flight simulator, car-racing games, and so on.

Scrolling In Safari

You may use the following steps to scroll horizontally or vertically in an application, say Safari, by tilting the PowerBook appropriately:

Movie

This movie (QuickTime format, no audio, 5.03 MB) depicts the use of AMS2HID for scrolling the content of the Google Maps web site in Safari.

Tips and Miscellaneous Notes

The out-of-the-box experience while using AMS2HID will differ based on various factors such as the calibration and sensitivity of the specific SMS unit, the nature and behavior of the target application, and individual user skills and preferences. This section contains information that may be helpful while experimenting with AMS2HID.

Some Limitations

Download

Download AMS2HID.

Acknowledgments

I would like to thank Ted Bonkenburg for his invaluable feedback on AMS2HID. Ted, a longtime friend, is one of the finest minds I know in Computer Science. I have never had a fruitless discussion on a technology-related matter with him, and this time was no different.