AI and Machine Learning
SignVerse
A prototype of the pipeline behind text-to-sign products: English is reduced to gloss, each word is looked up in a pose lexicon built from WLASL video, the clips are stitched together and played in the browser. It exists to find out what is hard before spending money recording a language that has no dataset.
89% of the words in 20 everyday sentences have a real sign, up from 42% at the start, and the remaining 7 are proper nouns that get fingerspelled.
01 / 03A sentence signed, with each word marked as a real sign or fingerspelled
Overview
The eventual target is Ethiopian Sign Language, which has no usable public dataset. Recording one is expensive, so this prototype was built on ASL first, where WLASL exists, to find out which parts of the problem are actually hard before anyone pays for video.
The pipeline is deliberately plain: lemmatise the English and drop the words ASL does not sign, look up one pose clip per word, trim each clip to where the signing actually happens, blend the joins and play the result with a pose viewer. A word with no entry is fingerspelled rather than skipped, and the page marks which words came from the lexicon, so the demo never pretends to more coverage than it has.
The measurements are the point. Coverage over 20 everyday sentences went from 42% to 89% through two independent changes, and the write-up in the repository records what each one was worth.
Features
- English to ASL gloss: articles and the copula dropped, time words fronted
- A lexicon of 800 signs built from WLASL video through MediaPipe pose extraction
- Clips trimmed to the active signing span, then blended at the joins
- Words with no sign are fingerspelled and shown as such
- The same pose retargeted onto a 3D figure
- A coverage script that scores the lexicon against test sentences
Challenges
- Picking words alphabetically wastes the lexicon. WLASL's own ordering is arbitrary, so choosing the 800 most frequent English words it covers, rather than the first 800, took coverage from 71% to 89% on its own.
- About a third of WLASL's video URLs are dead, so the fetcher tries several sources per word and still recovers 693 of 700.
- Signing is not word-for-word English. A gloss layer that drops articles and the copula and fronts time words lifted coverage from 42% to 53% before any extra vocabulary was added.
- This is stitched dictionary lookup, not fluent ASL. Word order is not ASL grammar, no deaf signer has reviewed the output, and the page says so.