Hi
I'm currently working on a thing where I need to determine what kind of clothing a person is wearing from a video feed. The lighting conditions are unknown. The person is masked and I also got a skeletal representation to my disposal. I got fairly good colour reproduction/constancy already. So, essentially the remaining part comes down to skin/non-skin detection.

At the moment I have tried two methods: In the first one I train a two pdf's, with skin texels from the hands and non-skin texels from the torso. Then I use these(Bayes maximum likelihood) to calculate the percentage of skin on the limbs to determine long sleeve/t-shirt, trousers/shorts etc. To make the system more stable a ROC-curve is used to control the threshold. The problem with the method is that there is always a risk that a certain peace of clothing will fall withing the cluster of the skin distribution.

In the other method I traverse the limbs from the hands and inward. When a discontinuous is reached the sleeve is considered found, a kind of edge detection. Concerns are made to coop with noise and wrist wares. The problem here is that rapid transitions in illumination and shadows frequently gets detected as sleeves. The threshold is also problematic.

In some cases the first method works best and in some the second. My idea is that I might be able to combine the two. Something like: I traverse the limb from a "secure" skin region(the hands) and builds up a distribution; to become resistant for certain features such as wrinkles in the hands etc. Then when the sleeve is reached, the continuity should be detected even if the "cloth and skin clusters" happened to intersect.

I'm not entirely sure how to solve this problem. If anybody got some suggestion, have faced a similar problem, know of any papers, please let me know :)

Quote: Original post by Jason Z
Sounds like a very interesting application - hopefully you can show some results once you get it working!

This type of a problem can be approached with a classification technique, such as a Support Vector Machine. If you aren't familiar with them, it basically has you define an n-dimensional space where each dimension corresponds to some important property of your input data set. In your case, I would imagine something like texture, distance to a discontinuity, location on the skeleton, color (or area within the color distribution as you mentioned).

Once you have a good set of dimensions, you train the SVM with known examples - typically hand selected. This will produce a hyperplane in your n-dimensional space that divides between skin and not skin. Then, to classify new unknown points at runtime you would more or less measure the Euclidean distance from the input point to the hyperplane - depending on which side of the plane you are on will make the determination about skin or not.

There are more complicated ways to treat it as well, including transforming the input vectors with non-linear transforms to better fit the distributions of the properties. In my experience the standard SVM works quite well on its own for problems that the eye can easily discern. A few google searches and wikipedia should have some free implementations on your hands too, so it should be a possible way to go. Good luck!

Hi Jason
Thanks for the reply and apologies for late reply. That SVM sounds interesting, I skimmed the net a little. So if I understood it correctly it's possible to transform the original space (rgb in my case) to an artificial hyper space. The hyper space is generated based on the two distributions. Then the plane is picked so that the overlap between the distributions is minimal?

At the moment I based the solution of the problem on pdf's, one pdf for the skin and one for clothing. I travel along the limb and evaluate the likelihood that a sample belongs either to one distribution or the other. This is done in several colour spaces (rgb, pca, etc). Then custom rules can be used to evaluate the most suitable space (e.g. the amount of noise).

Anyway at the moment there are some bugs in the system, caused by other people. After they’re solved I can determine if my current solution is ok or if I need to investigate SVM or similar.

Thanks ++