Skin Distortion Tracking
This is a tricky problem. Bend and stretch sensors are generally uni-axial (sensing in one direction). Further, applying them to the skin will alter the movement of the skin they might try to measure. But NYU’s very own Movement Lab may have just the thing in its Nonrigid Motion Acquisition and Modeling work. A dancer or model with a structured pattern of marks applied to the skin surrounding his or her joints could potentially be analyzed using the Movement Lab’s video-based nonrigid motion acquisition and modeling techniques.
Kinect / Optical Motion Capture Accuracy Comparison
The ideal setup for comparing the inexpensive Kinect to a much more capable (and expensive) system such as that produced by Vicon is to run both simultaneously, performing the same motion capture. However, as both use infrared light (in different ways) this seemed highly problematic. After some digging I learned that the Vicon cameras have multiple lighting modes that should work fine with a Kinect. And, in fact, this flying robot that relies on both a Kinect and Vicon demonstrates that the two can play nice together.
All this being said, further searching revealed that several recent projects have already done work comparing the accuracy of the Kinect against a traditional optical motion capture system. One looks at passive in-home measurement of stride-to-stride gait variability. Another compares the two systems in terms of human perception of their resultant data streams. And a third project conducted a fairly extensive biomedical validation of upper and lower body joint movements between the two technologies. In terms of a class project, this work could be validated and/or extended to motions not already specifically tested.
Biomechanics of Expressivity
My eventual dissertation work in Human Computer Interaction is likely to involve work in free-space gesture. I am particularly interested in the idea of sensing the gesturing between humans (i.e. dyadic non-verbal communication) as an input to a system in favor of gesturing directly to a system (e.g. Kinect-based games). A component of working with such human-to-human interaction is the expressivity of the individuals interacting with one another. Of course, expressivity can be a highly subjective term incorporating senses of artistry, subtlety, liveliness, etc. in different contexts.
The combination of expressivity and biomechanics has attracted little work in academic literature (such references do appear fairly often in discussions of dance and theater — though not usually in a strict engineering sense). Most often expressive biomechanics appears in the context of computer animation or in human mimicry in robots. For my purposes, I’d like to begin working with expressivity by objectively measuring its “energy” content in an individual and/or between individuals. As such, quantifying expressivity — especially between people — could be used to automatically code video of interactions, act as an input to novel games, or serve as an input to environmental control.
A simple way to do this might be to simply perform analysis of frame-to-frame change in 2D video or volume-to-volume change in 3D depth sensing. However, a far more nuanced approach might be to use knowledge of the range of motion of human joints to yield a measure of expressivity. Calculating the kinematics of the human body using a technology like the Kinect could produce a reasonably objective measure of body language expressivity.
I propose tracking joint orientation and limb acceleration and velocity up the kinetic chain of the body while possibly incorporating an estimated model of human mass to properly weight (in the mathematical sense) a total expressivity. For example, free-space motions at the shoulder require greater force than motions at the wrist. It so happens that motions at the shoulder also tend to yield larger motions in volumetric space than motions at the wrist. Thus, the total measured forces and subjective evaluation of expressivity are likely well correlated. Clearly, a precise definition of expressivity is needed, but it is my hope that this sketch of the idea communicates the idea well enough. A further extension might be to sum all gestural vectors to arrive at a single “affective vector.” For example, a downward affective vector might map to boredom or embarrassment.
Body language is most fully expressed between two people. While the Kinect can track two skeletons, it generally works best facing the individuals it is tracking. So measuring expressivity between two individuals facing one another may likely require an arrangement of two Kinects. The line of sight of these two Kinects might cause interference with one another and their angle of sensing could also be less than ideal. I believe these problems can be solved with two different approaches: vibrating the cameras to eliminate interference and using an SDK other than the default able to sense the human form at even oblique orientations to the Kinect device itself.