Developing an Offline Persian Automatic Lip Reader as a New Human–Mobile Interaction Method in Android Smart Phones

Abstract

In this paper, a new method to communicate with mobile phones is introduced. A camera phone can track the user’s lip motions using the lip reading algorithms. They recognize the words and sentences to run the appropriate application automatically. However, two of the challenges when implementing these algorithms in mobile phones are the changing ambient’s light and the limited resources such as memory and CPU. In this research, two methods are introduced to solve these problems. An offline lip reading application is implemented for Android phones which tracks the user’s lips in the Persian language. The application recognizes Persian words using visual features extraction from the user’s lips. The significance of this paper lies in the fact that it reports one of the first systems ever developed for offline lip reading. The whole process takes place completely on the device without the support of a dedicated server for the execution of the algorithms. In addition, this research is one of the first studies conducted on Persian mobile lip reading systems. Finally, the application was tested by disabled people who suffer from hand disabilities. The application satisfied them more than other applications which use sound, touch or text as an interface.