A regional public library system came to us with an inclusion goal they could not staff their way to. Hearing-impaired patrons were relying on pen and paper, or waiting for an interpreter, to handle everyday tasks like finding a book or settling a fine. We built ASL Vision, a kiosk that translates American Sign Language into on-screen text and spoken English in real time. A computer-vision pipeline tracks hand landmarks from a standard camera, a trained model classifies each sign, and a speech engine voices the assembled sentence, so a patron and a staff member who does not sign can hold a conversation directly.
View Impact
The library needed something patrons could walk up to and use without training, and that staff could support without knowing sign language. We delivered real-time recognition of the ASL alphabet and a library of everyday phrases, with the recognised character, the assembled sentence, and word suggestions all on screen, plus a speak control that voices the result.
A public library is a hard environment for computer vision: cluttered backgrounds, changing daylight, and patrons of every hand size and skin tone standing at slightly different distances. Recognition also had to be fast enough to feel like conversation rather than dictation, and run without sending video anywhere.
Rather than classify raw pixels, we fed MediaPipe hand landmarks into a custom-trained CNN, which made the model far more robust to background and lighting. We curated a dataset of over 25,000 hand-sign images augmented across hand sizes and skin tones, added a temporal smoothing layer to separate motion blur from intentional signs, and kept all inference on the device.
The kiosks are now permanent fixtures across five library branches, handling more than 100 patron interactions a week at 96.5 percent recognition accuracy in real public conditions. Patrons report a marked increase in independence, and because nothing leaves the device, the library met its privacy obligations without a separate review.
View Impact
The library needed something patrons could walk up to and use without training, and that staff could support without knowing sign language. We delivered real-time recognition of the ASL alphabet and a library of everyday phrases, with the recognised character, the assembled sentence, and word suggestions all on screen, plus a speak control that voices the result.
A public library is a hard environment for computer vision: cluttered backgrounds, changing daylight, and patrons of every hand size and skin tone standing at slightly different distances. Recognition also had to be fast enough to feel like conversation rather than dictation, and run without sending video anywhere.
Rather than classify raw pixels, we fed MediaPipe hand landmarks into a custom-trained CNN, which made the model far more robust to background and lighting. We curated a dataset of over 25,000 hand-sign images augmented across hand sizes and skin tones, added a temporal smoothing layer to separate motion blur from intentional signs, and kept all inference on the device.
The kiosks are now permanent fixtures across five library branches, handling more than 100 patron interactions a week at 96.5 percent recognition accuracy in real public conditions. Patrons report a marked increase in independence, and because nothing leaves the device, the library met its privacy obligations without a separate review.
Check these Also
Forging Digital Solutions with precision and passion. From freelance roots to global impact.
Trusted & Verified
