General Tau theory

For homework this week, I attempted to build an algorithm which implements Dr. David Lee's General Tau theory, as described in a recent paper, "How Movement is Guided".

A few relevant paraphrased quotes:
Principles of Animal Movement:
1) Movement requires prospective control.
2) The perceptual information guiding movement must extrapolate the movement into the future and must be readily available.
3) Movement requires constant intrinsic-cum-perceptual guidance. Intrinsic guidance is necessary because animals have to fashion movements to their purpose.
4) Movement guidance must be simple and reliable.
5) There are simple universal principles of movement guidance in animals.

Rather than use multiple information about the size, velocity, and deceleration of the motion-gap, we simply use the tau of the motion-gap. Tau is a measure of how a motion-gap is changing. It is the time-to-closure of the motion-gap at the current rate of closure, or equivalently, the first-order time to closure of the motion-gap.

Tau was first formulated in an optic variable that specifies time-to-collision if the closing velocity is maintained. Note that tau is not in general the actual time-to-closure of a motion-gap, because the velocity of a closure of a motion-gap may not be constant. The Tau of a motion gap is numerically equal to the ratio of the current size, x, of the motion-gap to its current rate of closure, i.e. T(x) = x / x'.

If the tau of two motion gaps remain in constant ratio to each other, they are said to be tau-coupled, and this is basically how we use perception in action to move about a space. For example, a bat landing on a perch needs to control simultaneously the closure of two extrinsic motion-gaps: the distance motion-gap, X, between itself and the perch and the angular motion-gap, A, between the direction line to the perch and the direction that the line should assume during the final approach to the perch. Bats tau-couple A and X (T(A) = kT(X) for a constant K throughout the maneuver.

Raising food to the mouth is tau-coupled, as is the motion-gap between the hand and the bat with the motion-gap between the ball and the bat in baseball.

Motion-gaps are not necessarily movement of objects. They can also be the change in other dimensions. Tau coupling may work in the following power law relationships:
Guide by sound including dolphins and bats.
Guide by smell including microbes.
Guide by infra-red radiation in rattlesnakes.
Guide by electrical fields in fish, sharks, platipi, and bacteria.

Tau-coupling has been studied in trombone playing, where the movements of the trombone slide, the lips, and the resulting acoustic pitch-slide are tau-coupled with a similar K value. It has also been tested in the neurological activity of monkeys:

"The hypothesis was tested by analyzing the neural power data collected from monkey motor cortex and parietal cortex area 5 during a reaching experiment. In each cortex a neural power motion-gap was found whose "tau melody" ( the temporal pattern of tau ) was proportional to the tauG melody and to the tau melody of the motion gap between the monkey's hand and the target as it reached. In the motor cortex, the neural tau melody preceded the hand movement tau melody by about 40 ms, indicating that it was prescribing the movement. In the parietal cortex area 5, the neural tau melody followed the movement tau melody by about 95ms, indicating that it was monitoring the movement."

In short, there's a lot of evidence which suggests that tau is a measure by which we understand intrinsic and extrinsic motion, and the constant between tau-coupled motion should be useful for creating gestures between joints.

I've started to code a framework in which I calculate the relative position of the joints from the torso in spherical coordinates and use those to obtain a tau for the radius, theta, and phi between all of the joints in relation to each other. Using the OpenNI skeleton, there are something like 1,307,674,368,000 (15!) possible generic combinations of movements to be made if you were ONLY using distance and not the angles, give or take ( somewhat less because you can't have a gesture with a limb in reference to itself, I think, but also quite more because that's not saying what *direction* they're moving in, or even thinking about tau-coupling distance with one or both of the angles, which basically puts you into an infinite number ). Most of those gestures are nonsense.

Anyway, if the ratio between two limbs' closure of a motion-gap remains at a constant ratio K, and that K is above a certain threshold, AND (I've decided) both of the limbs are moving, then that is a valid gesture, and it can be cataloged. Figuring out the thresholds is a bit tricky, bit it's also more tricky to figure out what the closure of the motion-gap is when you don't know the end point. I tried to write some code that used the distance and angles between the joints and the torso to define the motion gap, but found that small movements create huge taus (given that I'm calculating the first order derivative as the difference between last frame's position and this one - which might very well be wrong ).

I think I'm going to go back and rework the solution to be defined by the *opening* of a motion gap, since I have historical data. In this way, as long as a K between joints is constant, the gesture is still in motion. When the K changes and stabilizes, a new gesture is being signified. Unfortunately, I've only worked on this problem today, so I'm not quite sure what the results of that will be.