GestureFlow: Streaming Multi-Touch Gestures to an Audience
New mobile devices with large multi-touch displays, such as the iPad, have brought revolutionary changes to ways users interact with computers. Instead of traditional input devices such as keyboards, touchpads and mice, multi-touch gestures are used as the primary means of interacting with mobile devices. Surprisingly, body-motion gestures are evolving to become a new, natural, and effective way for game players to interact with game consoles in a very similar fashion: in Kinect for Xbox 360, a controller-free gaming experience is made possible by using body-motion gestures to play games.
The use of gestures to interact with computers is, of course, not limited to playing action-intensive games. Applications that allow users to create artistic or musical content in an interactive and collaborative fashion may also be gesture-intensive: gestures are frequently needed to create and manipulate artistic strokes or musical notes. These content creation applications share a common feature with interactive games: it is desirable to support collaboration (or competition) among multiple participating users or players. As an example, it would certainly be exciting if music composition hobbyists may collaborate in real time to work on a musical piece.
To support such collaboration among multiple users in real time, we propose that gestures are streamed in a broadcast fashion from one user to all participating users, in a broadcast streaming session. Streaming gestures themselves, rather than application-specific data, makes it possible to optimize the design and implementation of a gesture broadcast protocol that can be reused by any gesture-intensive application that needs to support multi-party collaboration. We believe this is a more elegant and reusable solution to serve the needs of an entire category of gesture-intensive applications. Once received, a gesture stream can be rendered in real time by a live instance of the same application on a receiver. To take such broadcast of gestures a step further, we believe that multiple gesture broadcast sessions need to be supported concurrently, so that any participating user can be the source of a gesture stream.
In this work, we have designed GestureFlow, a new gesture broadcast protocol specifically designed for multiple concurrent broadcast sessions of user gestures. We point out that gesture streams typically incur low yet bursty bit rates, unlike traditional media streams. They do pose unique challenges, as gesture streams need to be received with the lowest possible delay, and packet losses are not tolerable. In our design of GestureFlow, we motivate the use of network coding, and present a detailed design that takes advantage of inter-session network coding to support low latencies across multiple broadcast sessions.
We believe that the most convincing way to validate our design is to use a real-world implementation of GestureFlow, combined with a gesture-intensive application that presents a need for multi-party collaborative creation of content. Towards this objective, we have implemented both GestureFlow and MusicScore — a multi-touch music composition application — using Objective C from scratch on the iPad. MusicScore takes full advantage of our implementation of the GestureFlow framework to allow composers to enjoy a live collaborative session.
During extensive experiments presented in our INFOCOM 2011 paper, we have discovered new challenges in the use of network coding within the GestureFlow implementation. It turns out that coded blocks are linearly dependent with one another with an alarmingly high probability, leading to a much higher overhead than what we originally anticipated. We have discovered that it is due to the fact that the coding window size is typically very small, which is required to satisfy stringent delay requirements. We have proposed to use systematic Reed-Solomon codes on the source to mitigate the overhead due to such linear dependence.
