Video Database of Irish Sign Language performed by native signer

Hosting University

Dublin City University

Overview

The following project will be part of a larger team-effort, whose aim is to research and develop a computer-based Sign language translation system which can recognise Irish Sign Language (ISL) and translate it to English text. Our system will be developed and implemented on a standard personal computer connected to a colour video camera. Hand, face and body gestures will be recognised using image processing techniques based on Principal Component Analysis (PCA) and multi-scale hierarchical search. The variability in gestures will be modelled using Hidden Markov Models. The system will have a vocabulary of over 1000 gestures and will incorporate an ISL grammar and syntax. The system will be trained and tested by native ISL signers.

In order to train our system we need a good database containing examples of gestures from ISL. We propose to employ an intern student to help create this database. We hope that this intern student will be a native ISL signer, in order that his/her signs will be authentic and fluent. The database will consist of both video sequences and dataglove data and these will be manually segmented and labelled by the experienced signer. If the signer performs the same gesture many times, we can build up a distribution of hand-shapes within that gesture. If the signer performs a number of sentences in which the same gesture occurs preceded and followed by other gestures we can build a model of how one gesture modifies another. We can also use datagloves to get a full 3D description of how the hand changes in between gestures and this will allow us to construct a physiological model.

Facial expressions are also an important part of ISL and we will build a model of which facial expressions occur and how they modify the meaning of the gesture.

This system will form part of a tutorial system, which will aid people trying to learn ISL. It would be very useful for parents of Deaf children or school-teachers or university lecturers, who have to teach Deaf students. find the closest match to the current image.

We intend to compare multi-scale techniques for shape-matching. Work has already been done using Fourier Descriptors and Moments. We intend to extend this by convolving the data using Gaussian kernels of varying widths and applying an eigenvalue decomposition. This should express the shape data as a tree where successive levels represent different resolutions. This should speed up the matching process and make the algorithm more robust.

We will train and test the system on hand images derived from videos of native ISL signers but the technique can be applied to shape data from many other sources. The technique could be useful for other projects within the CDVP such as object recognition for video retrieval.

Relevance to Host Laboratory

Our group already contains a Marie Curie Fellow, and a post-doc funded by an SFI Research Frontiers grant, an IRCSET funded postgraduate and two School-funded postgraduates. All are working on various aspects of Sign Language recognition. The work in this project will provide accurate and authentic data about ISL, which is necessary to train and test our system.

Supervisor

Dr. Alistair Sutherland     

Students who have worked on this project: