Character Recognition and American Sign Language

ABSTRACT

In this project, a character’s image will be taken and sent to Matlab. It then proceeds to analyze the image and identify the character. The result is sent to a microcontroller – an Arduino – which will control the fingers to sign the American sign Language of the specific character.

METHODS AND MATERIALS

I. Materials:

a. 3D Printed Prosthetic arm

b. 5 DC servos

c. 5 LEDs

d. 5 330Ohms resistor

e. 4 1.5V AA batteries

f. Computer with Matlab installed (Image Processing toolbox)

II. Methods:

1. Step 1: We started by setting up the Prosthetic Arm by plugging each wire corresponds with each servo into the breadboard. The external power (4 1.5V AA batteries) is also needed to apply in order for the 5 servos to function. Connect the Arduino to computer so that Matlab can communicate with it.

2. Step 2: Next, we will need to connect the webcam to the computer. The purpose of it is to take an image of the character.

3. Step 3: At this moment, everything is set for the project to run.

a. The “create_templates” function is where I put all the standard images of 20 characters. Since there are more than one way to write each character and different person writes differently, for some characters there are more than one standard images. As a result, there are total of 32 standard images. All of them then are saved in a cell called “templates” with each character separated from one another. All of the standard images are 2-dimensional binary Black&White image. Dimension of each is 24x42. The character is white and the background is black.

b. An image will then be taken by a webcam. Since this taken image is a 3-dimensional RGB image which is not compatible with the standard images. Therefore, we first need to convert this image to 2-dimensional image using a built-in function – rgb2gray(). This function will convert the truecolor image RGB to the gray scale image.

c. For now, the gray image still has pixel value between 0 and 255 and not yet Black&White. Another built-in function is used to convert it to a binary Black&White – im2bw(image, threshold). A “~” sign is used so that the character will be white and the background will be black. The threshold is calculated using graythresh(image) function which calculates the threshold of a gray scale image. The threshold helps to minimize the intraclass variance of the thresholded black and white pixels.

d. For all 32 standard images, they look like this picture below which contain the character only and the black background. Therefore, the taken image should be somehow manipulated to have same property (More discussion in the Challenges section). The “crop_image” function will crop the unnecessary part of the taken image so that the cropped image only contains the character only. As a result, the taken image now will look similar to the standard ones.

e. The taken image now is converted to have the same format as the standard images. However, the size of the taken image is still not the same size as the standard images (Matlab requires 2 images to have identical size in order to compare). The imresize(image, dimensions) function is used to resize the image to the size of the standard images.

f. At the moment, the original taken image is converted to a binary Black&White image with the character is white and the background is black. It also has the same size as the standard images. The “read_character” will then compare the cropped image to all the standard images using the corr2(standard images, taken image) function. It will compare the cropped image individually with each standard image and print out a single value (positive or negative) which tells how similar between the two images. The result is a matrix with 32 components called comp. There will be the maximum component in this matrix, and the find command will find the position of the max value in the matrix. Depends on that position, I use If statement to give the result.

4. Step 4: The result is a character. This result will then be sent to the Arduino. The Arduino is already programmed so that it can move the servos to control the fingers to sign the character in American Sign Language.

5. Step 5: There are 5 LEDs with each represents each finger. As soon as a finger is curled down, an LED associated with it will light up.

CHALLENGES

1. The first challenge that I encountered is that I did not know I have to crop the taken image for the corr2 function, which calculates the similarity between two images, for Matlab to be able to recognize the character. An example will be shown below.

• Without cropping the taken image:

Using the corr2 function to calculate the similarity between uncropped binary BW image and the two standard images of B I got -0.1184 and -0.0910. However, when calculated with standard image of A I got 0.1077. Therefore, the result is incorrect.

• Cropping the taken image:

The corr2 function now gives me 0.2811 and -0.0069 when compared the cropped image to the two standard images of B. It gives me the value of -0.0252 when compared with the standard A this time.

2. The image taken from the webcam is not always perfect. As a result, the output result will sometimes be incorrect.

The important part of the code is the corr2 function. It calculates the how similar in pixels intensity between two images. And afterwards, the read_character function will pick the one that has the max value. Therefore, the image needs to be taken perfectly so that when it is converted to binary BW image, the character will be white and the background is black for the code to work. This is a weak point of this program.

3. The ability of this code to identify a handwritten character. An example is shown below.

One more time, the code has to depend on the corr2 function. It depends on the results from this function. This time, it gives the value of 0.0265 and 0.1166 when compared with the standard images of B and 0.1457 and 0.2515 when compared with two standard images of number 3. Clearly, the value got from comparing with number 3 is higher. Therefore, the result will be printed out as 3. Another weak point of the program.

FUTURE WORK

What I really want to keep improve on my arm design so that it will have more flexibility! At the moment, clearly there are a number of limitations. What I also want to do is to design the whole arm which will increase the number of characters the hand can do.

This part is more important part which deals with the code. I need to figure out how to take a perfect image so that I don’t have to use the set-up I’m using right now (lamp, stand, clamp etc.). “Is it the camera problem or this will happen with other camera as well?”

Next part is how to recognize an image instead of comparing like I’m doing now. This method I’m using has quite low probability for Matlab to recognize a handwritten character since there is only certain amount of way to write a character. I know Matlab has a toolbox called Neural Networks. It is a computing model whose layered structure resembles the networked structure of neurons in the brain, with layers of connected nodes. A neural network can learn from data—so it can be trained to recognize patterns, classify data, and forecast future events. I saw people have used this toolbox to do a lot of cool things – handwritten character recognition. This might be a solution to my program’s problem.

CONCLUSION

Matlab is able to analyze many things such as images. For my purpose, Matlab is used to identify a character in the image then send result to the microcontroller - an Arduino - to control the arm to sign American Sign Language. There are both good and bad side of the code. It can recognize a character which is similar to the corresponding standard image. However, as we know, handwriting of each of us is not the same. The probability that the code can understand and identify a character that it is not “trained” to identify is quite low. As a result, future work will be required.

REFERENCES

[1] Website, What is a Neural Networks?

https://www.mathworks.com/discovery/neural-network.html