BhashaSetu is an open-source translator for Indian Sign Language. Camera in, Hindi or English out, and the reverse through a signing avatar. We're a small team, it's early, and we're building the whole thing in the open with Deaf collaborators. We could use more hands.
Where is the nearest hospital?
नज़दीकी अस्पताल कहाँ है?
Nobody agrees on the exact count. The 2011 Census recorded about 5 million deaf Indians; broader estimates of significant hearing loss run to 63 million. The other side of the ratio is not in dispute. The official directory lists fewer than 300 certified ISL interpreters for the whole country. A classroom, a hospital ward, a courtroom, a job interview: most of the time, no interpreter is coming.
BhashaSetu is our attempt to shrink that gap with software. A model anyone can run on a phone or a laptop, translating both directions in real time. It won't replace human interpreters, and it shouldn't. It's for the rooms they can never reach.
Numbers that size stop meaning anything after a while. So here is where the interpreter shortage actually lands: at school, at the doctor, and at work.
of Deaf children aged 6 to 13 are out of school entirely. Many schools that do enrol them teach through lip-reading and speech drills instead of ISL. And a teacher who wants to sign has no formal curriculum to learn it from.
DHH out-of-school study · 360info, 2014of disabled Indians live in rural areas, which is exactly where those 300-odd interpreters are not. Explaining chest pain to a doctor through gestures and a relative's guesswork is the current reality. Translation in a clinic is not a convenience feature. It is informed consent.
Census of India · 2011of working-age Deaf adults are in marginal or informal work. Miss school, struggle with written Hindi or English as a result, then walk into interviews where nobody signs. Each step makes the next one worse, and the cost gets passed down through families.
Census of India · 2011It's a full language with its own grammar. Most sign-translation demos fail in India because they treat signing as word-by-word substitution, which produces nonsense in both directions. Four things a model has to get right:
ISL fingerspells with both hands. ASL and most Western sign languages use one. Models pretrained on Western data transfer poorly because the hand shapes themselves are different.
"I rice eat," not "I eat rice." A word-for-word translation produces gibberish. ISL → text requires a real grammatical reordering step, not a lookup table.
Raised eyebrows mark yes/no questions. A head-shake makes a clause negative. Mouth-shape carries intensity. Ignore non-manuals and you lose half the meaning.
The grammar is shared across India but vocabulary shifts by region. A signer from Mumbai and one from Kolkata agree on sentence structure, not always on the sign for a word. Regional fine-tunes are part of the plan, not an afterthought.
The commercial sign-language tools that exist today are built for ASL, closed, and priced for Western institutions, not for a government school in Bihar or a district hospital in Telangana. The model is the moat. That business does not fit a country with one shared sign language, dozens of regional variants, and a Deaf community that has spent decades being talked at instead of asked.
And the data could hardly be more personal. It is video of people's faces and hands, often recorded at home. We think that footage should never have to leave the device, and the only way to make that promise believable is to open everything: the weights, the code, the consented recordings, and the eval suite, under permissive licenses anyone can audit, fork, and retrain.
A note on framing.The disability-rights principle "nothing about us, without us" applies to software too. Deaf signers, ISL educators, and accessibility researchers review what we ship, and that review started before the first line of code.
One model, both directions: continuous signing into text and speech, and text back into signing through a 3D avatar, all running on the device itself. To be clear, these are design goals we're building toward, not shipped features.
Most published ISL models classify one isolated sign at a time. That wins benchmarks and loses conversations. We're working on continuous-sentence recognition that keeps the grammar, the fingerspelling, and the non-manual markers (eyebrows, mouth shape, head tilt), since those carry half the meaning in ISL.
Type or speak in Hindi or English and a rigged 3D avatar signs it back in ISL, with smooth transitions between signs. That closes the loop: a hearing person and a Deaf signer holding an actual conversation instead of passing notes.
The model targets browser inference through WebGPU and Android through NNAPI. Your camera feed stays on your device. Nothing uploads, nobody makes you sign in, and it keeps working where the network doesn't.
Public ISL datasets are tiny. INCLUDE covers 263 signs, CISLR lists 4,765 words with a handful of clips each. That is nowhere near a language. We start there anyway, and grow the corpus with consented recordings from real signers.
Captions everywhere, haptic cues, high-contrast theming, large hit targets, full keyboard navigation. We treat WCAG 2.2 AA as the minimum to ship, and our Deaf collaborators catch what the checklist misses.
Permissive licenses across the board. Fine-tune for your school, your clinic, your state. Commercial use is fine. No CLAs.
Camera frames go in. Translated speech and a signing avatar come out. Every stage is swappable and benchmarked.
Hand, body and face landmarks per frame using MediaPipe Holistic: 21 hand + 33 body + 468 face points, batched on GPU.
mediapipe · 30 fpsAn ST-GCN learns sign morphology over a sliding window, fusing manual + non-manual features (eyebrows, mouth).
st-gcnA transformer-CTC head emits gloss sequences; a small LM reorders ISL's SOV grammar into natural Hindi / English / regional.
transformer-ctcBack-translation pairs gloss with multilingual text using a seq2seq fine-tuned on parallel ISL↔text data we're curating.
mT5 · smallGloss sequences drive a rigged 3D signer with smoothed inverse kinematics and non-manual blending.
three.js · webgpuNone of this is installable yet, and we'd rather say so plainly than fake an npm badge. The plan is a thin JS SDK for the browser, a Python package for notebooks and servers, and open weights on HuggingFace. All three land with v0.1.
Drop-in browser SDK with WebGPU runtime, a web component for the signing avatar, and an event-driven translation stream.
For notebooks, servers, and batch processing. Same API as the JS SDK, ONNX runtime under the hood, CPU and CUDA backends.
Weights, model card, eval results, and a Spaces demo. Apache-2.0 licensed, fully fine-tunable, no gating.
We froze the API surface early so contributors have something stable to build against. This is what calling it will look like.
Available via npm for web, and pip for Python notebooks & servers.
The SDK requests webcam permission once. Frames are processed entirely on-device.
Listen for onTranslation events to get gloss, text, and confidence per utterance.
Drop in the <setu-avatar> web component for two-way conversations.
// install: npm i @bhashasetu/web (coming soon)
import { Setu } from "@bhashasetu/web";
const setu = await Setu.load({
model: "setu-isl-base",
target: "hi", // output language
backend: "webgpu",
});
await setu.start(document.getElementById("cam"));
setu.onTranslation(({ text, gloss, conf }) => {
console.log(gloss, "→", text, `(${conf.toFixed(2)})`);
});
// the other direction: text → signing avatar
const avatar = setu.avatar("#stage");
await avatar.say("नमस्ते, आप कैसे हैं?");A translation model can fail in ways a hearing developer will never notice. So Deaf signers, ISL educators, and accessibility groups review our datasets, the UI, the error states, and the model card itself. When they object, the release waits.
There's nothing to download yet, and the waitlist is how this project finds its people. Tell us who you are and we'll email you when something actually happens. A handful of emails a year, each one earned.
Data first, model second, polish last. That order is deliberate, and it means the flashy parts come at the end.
Survey existing ISL datasets (INCLUDE, CISLR, ISLRTC dict), define gloss vocabulary, draft model card & consent protocols.
Isolated sign recognition on existing public data. Browser SDK with the API above. Honest about what doesn't work yet.
Transformer-CTC head, sliding-window inference, real-time streaming translations.
The other direction. Rigged 3D signer driven by gloss sequences from a multilingual seq2seq.
Fine-tunes for state-level ISL variations, mobile-first runtime, classroom & clinic deployment kits.
We're looking for ML engineers, ISL signers, accessibility researchers, and anyone who wants to help. Repo and community channels go public with v0.1.