It has been quiet for quit some time, so I’ll try to get you all up to date. After getting my toe-detection sort of working, I rushed to get some nice results for my milestone presentation at the end of January. One of those results was analyzing the roll off pattern of a paw. In humans, there are five distinctive moments which occur in nearly every normal foot roll off:
From: de Cock – de Clerq – Willems – Witvrouw – Temporal characteristics of foot roll-over during barefoot jogging reference data for young adults. Gait and Posture Vol 21/4 pp 432-439
- The heel strikes the ground
- The forefoot starts to make contact
- The forefoot reaches its maximal contact surface
- The heel comes of the ground
- The forefoot leaves the ground
These intervals are very reliable and have clear connection with the kinematics of walking, which makes them very useful for describing gait. So my first instinct was to start looking for something similar in dogs. But how?
I decided to first look at the pressure over time:
Clearly the start time of the toe regions is of no interest: dogs seem to place their paws entirely flat, so there goes every hope of a initial forefoot contact… Then there’s two other easy detectable characteristics: when is the pressure the highest and when is there no pressure under a toe?
I calculated it for every dog and threw them up a bunch and made a histogram: let the data tell me which event is the most reliable. Turned out the data was all over the place! Why? Because I forgot to normalize my results before comparing… Especially normalizing the time axis to go from 0-100% for each trial.
Now clearly that’s some of a difference, but what was more important, the graphs of all the dogs looked a lot more alike. So when I calculated the peak and end times again, I got the following table (pardon me for the Dutch, have fun with Google Translate if you can’t guess it):
Note that I calculate the toe the same way for left and right, so while Toe 1 is always the central heel pad, the other toes are mirrored. If you then look at the table again, we spot a very high similarity between left and right. We can also see that the percentages are still all over the place…
I was almost tempted to just cut the roll off into four equal parts, because at least that would very repeatable and give us a clear overview of progression during the roll off. However, since that lacks any kinematical links, I decided to simply try several variations. But every combination resulted in some trials having only three frames in certain intervals. Then I looked more closely at the central heel pad:
- Start – Max = 0 to 20/30%
- Max – Heel off = 20/30% to 60/70%
- Heel off – Toe off = 60-70% to 100%
All the other intervals would be only a couple of frames of these moments and they have a very clear and understandable kinematical meaning!<hr/>
After seeing all my results in one PowerPoint, you can imagine the clinic was very satisfied with the results so far! Even so satisfied that they gave me an external hard drive with another 100 dogs… Clearly somebody has not been reading how much work it is to process the data!0 Comments
As my experimentations showed yesterday, when calculating the peak detection on the average of all paws (left/right all bunched together) you seem to get 5 toes in most dogs. I’ve started to exclude any dog that didn’t pass test, so that brings down the amount of dogs to about 16.
I decided to see how well it would perform if I only averaged over one measurement (typically 8 or more impacts), rather than having to load multiple measurements.
As you can see here, to toe locations of the ‘average’ paw matched right paws very well, but actually needed to be mirrored for the left ones.
So my next step became to test whether an impact preferred a left or right alignment of the toes. I did this by taking a slice around the average’s toes and compare the sums of the pressure within these slices (each slice has the size of the white box in the figures above).
This worked surprisingly well for identifying what kind of toe layout an impact preferred. Here we have four alternation columns comparing the ‘new’ toe detection based on the average paw and next to it, the vanilla toe detection on a standardized paw. As you can see, it found 5 toes in all trials and they even look like a paw!
I then wanted to see how this worked on different dogs.
As you can see, it was still not perfect. For some reason the background filtering doesn’t work great on my standardized paws, which might be caused by the trimming. The standardizing cuts out the paw from anything under 10% of the maximal pressure, so there aren’t that many zero cells to subtract left.I kept tweaking things to try and improve things, like averaging over all paws and then using these toes for all measurements. But alas, nothing seemed to get rid of these unexplainable problems.
After getting fed up with all these ‘ghost’ toes popping up out of nowhere and the quite long processing time (because I had to average everything first before I could start comparing each paw again) I decided a radically different approach.
First I figured that I didn’t need to slice around the ‘average’ toe, because simply using the toe’s coordinate should work good enough. Furthermore, I decided to take a neutral positioning of the toes as a default. From there I would calculate the distance to each toe that was being found for that standardized paw and take the one that was the closest to the toe.
When I ran this over all the measurements, it not only ran pretty quickly, it turned out to find 5 toes in at least 80% of the trials. There were only 4 dogs where it performed sub-optimal at best with a 50% success rate and I haven’t looked into what was going wrong here.
Some other small quirks I have to solve:
- Not allow the same toe to be picked twice
- See how the standardizing can be improved to reduce the ‘noise’ that cause the toes on the edges to pop up
- When picking between two peaks with a similar distance, take the one with the highest value, rather than the first.
Even without it being perfect, I’m very pleased with my current implementation and look forward to recalculate the pressure over time for each toe! As usual, you’ll hear back from me when I have new results and if you have any questions or suggestions: leave a comment!0 Comments
After my initial problems with my toe detection, based on the problems I was seeing I wanted to try two things:
- Increasing the binary structure that’s being used for looking for peaks;
- Resizing the paws to better match the ‘ideal’ size
So I made a loop that would try different binary structures (array with True values) to see what the result was. But the results didn’t really seem to improve, whether I used a 2×2, 3×4, 3×5 or even a 5×5 structure.
To see why this wasn’t doing what I expected, I dug a bit deeper into the toe detection function.
local_max = maximum_filter(image, footprint=neighborhood)== image
Here image is the top left figure, neighborhood is the binary structure and maximum_filter thus returns the maximal values within areas of the size footprint in image. I was under the impression that the size of neighborhood was a unique area and wouldn’t overlap with other areas. Here’s a plot of
Turns out, as long as the maximum values don’t overlap, the areas can overlap as much as they want! So clearly, once again, I didn’t really know what to do with this to get the wanted behavior from it.
Therefore I got out plan B: resizing. I started out with a combination of both methods:
- the default sized paw with the standard toe detection,
- the resized paw with the standard toe detection and
- the resized paw with an altered version of the toe detection.
I started out with a small dog and looked to see if it helped interpolating the image, so I had more pixels. You see, looking for 5 maxima within a 3×3 area in a 8×8 sized array isn’t going to work out. So I tried several sizes, scaling them to 10, 12, 14 and even 20 pixels to see whether it would pick more toes.
But after several iterations and tweaking the binary structures in combination with reshaping, it was clear that most of the small dogs measurements were beyond salvation. The only paws that would ever get 5 peaks were probably all front paws, meaning I would never be able to compare the values among the four paws. So I hereby declare all the small dog trials trash! Note that I can still divide them into quarters and compare those, but that’s beyond the scope of this project
Next up were the large dogs. While I don’t have any figures to prove it, it turns out that it would vary per paw whether resizing helped or not! So sometimes having a different binary structure or size would get me 5 toes, but it would get me 6 toes on another and vice versa. Note to self: did you check whether this was for front or hind paws? Perhaps there’s a pattern
More frustratingly is that the resizing settings that worked for a large dog didn’t work so well for my ‘ideal’ sized paws! I decided that just messing around wasn’t getting me any further, so I first did a basic filtering:
Calculate an average for all the paws of the same dog and resize them to the same size for all dogs. Then use the standard toe detection to see what it gives. If the results yield anything less than 4 or more than 6, they’re out.
As you can see, it actually performs very well on most ‘average’ resized paws. And interestingly enough it directly tells me which all the small dogs are as their data is in nearly all cases just a gigantic blob.
Seeing that in all the other cases it seems that it’s perfectly capable of finding the toes, it seems this should work, even though trial based inspection gives a different impression. Assuming that on average the toes are more or less in the same place, I could create a ‘mask’ where I manually tell which areas belong to which toe. Once I have several of these masks, I could run them against a measurement to see which combination of toe locations gives the largest sum overall.
Anyway, at least I’ll know what I have to do tomorrow!
But before I go, I just wanted to share you something I’m particularly proud of, because I wrote it without any help!
I call it: the manual paw annotator v0.1!
Basically what it does is, it plots a measurement, shows the annotations from Joe’s sorting algorithm and displays the chronological ordering (the white line), then it starts to loop through the list of paws and highlights them one by one and prompts the user for input.
Current annotation for this impact is : RH What would you like to change it into? 7 = Front Left —————- 9 = Front Right ——- 5 = Keep current annotation ———– 1 = Hind Left —————— 3 = Hind Right Enter your choice: 3 You chose LH
I mapped the paws to the keyboards numpad, so that it feels slightly natural to annotate the paw with a similarly spatially located key! All I need to figure out now is:
- how to overwrite the original object in Joe’s data.hdf5 file and
- how to let a measurement know I manually checked it, so don’t dare to ask me again unless I say so!
With this function, I can easily sort through several trials for each dog and built up a more curated training set for the automated sorting. Especially for trials where the dogs didn’t follow the trapezoidal pattern, as the algorithm seemed to perform particularly poor here.
That’s it for today and once again if you want to know anything or have suggestions: leave a comment!0 Comments
Not letting myself be stopped by the insufferable toe detection, I decided to check the paw size for each dog. Because as you could see in large dogs, the toe’s are larger than the area it’s trying to allocate a toe in, thus it recognizes some toes twice (especially the heel).
As you can see, the nail of the toe near 10,2 is outside the 2×2 area of the toe, just as the oval shape of the heel stretches from 2:5,2:10, clearly larger than 2×2 I’d say!
These are the graphs of almost every dog, where the paw size (on the y-axis in cm^2) is plotted over time (seconds). There is a large difference in paw size, as some reach maximums of around 40 cm^2, others barely reach 15. That’s less than half the size! Granted, there are 5 weight categories included in this study, ranging from below 5kg up to 50 kg, so large differences are to be expected.
But what it does mean is that choosing hardcoded, arbitrary values like 2×2 are very, very bad! Note to self: next time, try to gather this kind of info before you start to solve these problems.
Anyway, so I decided to tweak the peak detection code to try and change the size of the areas it looks for toes in.
neighborhood = generate_binary_structure(2,2)
Gives as a results:
Funny, I thought 2,2 meant I an array that was sized 2×2. However, as I learned the other day, this probably has to do with Python indexing, so never mind. Ok let’s try:
neighborhood = generate_binary_structure(3,3)
Hmmm, ok! I learn something new every day it seems! Fed up with not understanding what I was doing, I went to Scipy.org and look up what the function actually does. Turns out there’s a nice little note explaining what I was doing wrong:
generatebinarystructure can only create structuring elements with dimensions equal to 3, i.e. minimal dimensions. For larger structuring elements, that are useful e.g. for eroding large objects, one may either use iterate_structure, or create directly custom arrays with numpy functions such as numpy.ones.
So basically, if I wanted any other shape than I was currently using, I might as well make one myself! Well, then I guess it’s back to the drawing board and figure out what to put in here. I’ll keep you posted and if anyone has any suggestions: leave a comment!0 Comments
Here I was testing my new results on all measurements, just to bug-test it and see if it would work. Checking some of the results, just to see if everything was working properly. Until I notices something funny, my code would crash on some dogs measurements and I had no idea why.
So I look for the dog that’s causing the problem, start printing some of the results into the console, to figure out what’s causing the problem. Hmmm, that’s strange the lower left dog only has 18 lines, versus over 80 on the top two. So what’s causing this?
Turns out, the peak detection algorithm I got from my first SO question doesn’t work too well on anything but a ‘medium’ sized paw!
Looking good eh?
Looking not so good huh?
Turns out that with my large dogs, the rear toe area is so large or wide, that it recognizes two peaks that don’t overlap in a 3×3 area with each other. I guess I could tweak the toe-sorting loop, to have it find the two most rear toes and locate an area around them or even just use both.
But guess what?!? I also have dogs that have 7 and 8 toes, so good luck merging those peaks into toes. And newflash! Those cute tiny dogs quite often only have 4! So clearly, this is not looking good for several of these measurements until I have a better way of detecting toes.
Thank god I even checked for the amount of toes in the first place, else I would have been looking a lot longer (though it would have crash my toe-sorting loop, as it expects only 5…)
My incomplete step checker (see below) actually checks for three things:
def incomplete_step(data, x, y,maxtime): incomplete = 0 if touches_edges(data, x, y,maxtime): incomplete = 1 elif sumovertime(data)[-1] > (0.1* max(sumovertime(data))): incomplete = 1 elif not fivetoescheck(detect_peaks(data)): incomplete = 1 return incomplete
It first looks to see if the impact touches the edge of the plate or if it was active at the last frame of the measurement. In those cases I can’t guarantee the impact is complete, so they’re out. This filters out a relatively small amount of impacts and in practice shouldn’t occur too much, because you would simply repeat the measurement if the dog missed the plate too much.
Then it checks whether the summed up pressure(sumovertime) in the last frame is larger than 10% of the maximal pressure during the entire stance phase. If so, bye bye measurement! Though come to think of it, this should actually overlap with the paw being active at the last frame, which means the measurement stopped before the paw pushed of the plate. Note to self: check if this is true!
And the last one is the evil: check for 5 toes test! The upside to all this is: it surely cleans up, none of the incomplete steps are left! The downside is: it’s a bit too clean for my liking!
Accepted vs Rejected counts:
- (73, 44)
- (31, 52)
- (28, 76)
- (0, 214)
- (13, 168)
- (52, 67)
- *(5, 200) *
- (31, 68)
- (28, 81)
- (82, 54)
- (1, 209)
- (72, 80)
- (45, 63)
- (2, 221)
- (88, 43)
- (31, 54)
- (73, 31)
- (96, 23)
- (77, 28)
- (2, 189)
- (7, 155)
- (0, 153)
- (12, 73)
- (16, 196)
- (86, 26)
- (2, 212)
This is a list of all the dogs, where I counted how many impacts got filtered out. As you can see, A LOT of impacts are being filtered. In 6 out of 26 dogs, nearly every measurement is gone! Based on the number of impacts, I guess these are all the little dogs, which means the toe detection couldn’t find enough toes. But that doesn’t mean it liked the big dogs any better, because there a significant portion also get’s filtered out.
Obviously, I need a better way of detecting these toes!
Because I didn’t want to have my entire day ruined, I turned the filtering off and only created an average and the standard deviation of the summed pressure over time for every trial for every dog.
Just for the kick of it, here’s the comparison before and after the filtering:
The order is a bit scrambled, but as you can see for several dogs, we end up with a much larger variation. This is partially caused by the hind paws not making it through the filtering, but at least it clearly shows the need for a filtering!
For now, I guess I’ll have to get back to the drawing board on that SO-question…0 Comments