Datasets Used

The code was first tested with the Triesch Dataset, after which we created our own dataset to suit our testing conditions (unideal) as well as expand our letter-base

Triesch Dataset

Triesch dataset contains letters a,b,c and v against 3 different backgrounds as shown below.

User Dataset

The created dataset contains images of letters a,b,c,d,g,i,l,v and y, recorded at different angles and at different distances with respect to the camera.
This variation is introduced in order to introduce tolerances for orientation and scale.

Data Augmentation

Left-Right Hands -> We generate a set of rotated images from the existing images to help train our model to make it more robust.
Tilt/Zoom -> Generated set of images with different levels of tilt and zoom to increase the variety of the training dataset.

Approach

Hand Segmentation

The Otsu algorithm segments the image into background and foreground. The input image is the grayscaled gaussain blurred image. This method classifies the pixels based on their grayscale pixel values and cluster them into 2 regions. We used this method for hand segmentation due to its invariance to dynamic light conditions

Feature Extraction

Various features extracted for an image include the hull points, contour area, convexity defect coordinates, HoG features and bounding box. We particularly selected the HoG features due to clear distinction obtained for individual characters.

The Contour in the image contains a set of all connected points which form the outline of the hand region.
The Hull contains the area included by connecting the extremities (bulging points) of the contour in the detected hand region.
The Convexity Defects are convex regions formed by the area between the fingers in the contour of the hand region. These are called defects because the area between the fingers forms a convex curve in an otherwise concave hull.
The Bounding Box is the minimum rectangle large enough to crop the entire hand region. Computing the ratio of contour area with respect to bounding box area makes the detection invariant to scale.

HoG Features have been computed on the contours of each letter in the database. HOG features compute the occurrences of gradient orientations in portions of an image, so we can describe characteristic edges regarding the shape of the hands. The figure below shows the HoG features for each letter in the dataset.

(Click to expand)

Results

The below video demonstrates how correct words are predicted from sequence of 4 letters. We provided a series of characters 'b','a','y','d' and it displayed "BAD" as the corrected output.

1. HoG + SVM works really well on the Treisch dataset as it is nicely centered. But, when the model trained on this dataset is used in a live video, the predictions fail.
2. HoG + SVM has slightly worse accuracy for in-set when trained using our dataset. But the model trained using this dataset works a lot better with live video.

Dataset	In-Set Accuracy	Live-Video Accuracy
Triesch	87%	40%
4-Letter Dataset	80%	65%
9-Letter Dataset	78%	61%

Initially we started training out models with only 4 different letters. However this was not enough as when the hands were changed(right -> left), the accuracy of the classifier fell.
We also included height, width and the ratio of the bounding box in training the classifier. These features were given more significance as they were better spread out as seen from the PCA plot.

Another problem we faced was to find letter boundaries, this was done using edge filtering. Since edge filtering was used we were not able to form words where a certain letter is consecutively repeated.
To remove this drawback and also to include a word auto-correct feature, we implemented a word speller(using autocorrect library in python). The combination of letter edge filtering along with word prediction enabled us to create words using fingerspelling gestures. Additionally, it also increased our overall datasets spread as more letters could be incorporated into words due to the auto-correct feature.

The below video demonstrates how correct words are predicted from sequence of 4 letters. We provided a series of characters 'b','a','y','d' and it displayed "BAD" as the corrected output.