The moment: ASL to text, right in your keyboard
It finally happened: sign language is a first‑class text input on phones. Google DeepMind’s new SL2T (sign‑language‑to‑text) model now powers sign‑to‑text dictation in Gboard and Live Transcribe on Pixel 11 devices, starting with American Sign Language (ASL) to English. For the world’s estimated 70 million Deaf and hard‑of‑hearing signers, this brings a long‑missing parity with voice dictation. (deepmind.google)
What’s under the hood (and why it matters for testing)
SL2T treats sign languages as languages—not “English on the hands.” It uses on‑device pose tracking (MediaPipe Holistic) to map face, hands, and upper‑body landmarks; only these 2D coordinates go to secure servers for translation and the original video is discarded, which is important for user trust when you run timed tests with cameras on. The launch starts with ASL→English on Pixel 11, with more devices and languages to follow. (deepmind.google)
Performance‑wise, the team reports training across 100,000+ hours spanning 50+ sign languages and a zero‑shot 70 BLEURT score on the FLEURS‑ASL benchmark—paired with practical work on streaming latency, hallucination prevention when nobody is signing, better one‑handed signing (phone in the other hand), and fairness for the ~10% of left‑handed signers. Those last points are essential for leaderboard design. (deepmind.google)
Quick setup on Pixel 11 (Gboard + Live Transcribe)
- Prerequisites: Pixel 11, latest Gboard, and keyboard language set to en‑US or en‑CA. Grant camera permission the first time you use sign‑to‑text. (support.google.com)
- Gboard steps: In any app with Gboard, tap the Sign‑to‑text tool in the toolbar (add it via Edit if needed), frame your face, upper body, and hands, then start signing. Text streams into the field; move your hands out of the frame to finish and switch back to the keyboard at any time. Privacy note: pose coordinates are processed on secure servers; videos aren’t stored. (support.google.com)
- Live Transcribe steps: Open Live Transcribe, tap the Sign‑to‑text icon, and sign your message (one‑handed, two‑handed, or with fingerspelling). Tap the checkmark to send your signed reply as text. On foldables, Dual Screen mode can mirror text for both participants. (support.google.com)
- Pixel blog highlights: One‑handed and two‑handed signing are supported from day one, with translations appearing in real time on Pixel 11. (blog.google)
Why typing tests must evolve
Traditional typing tests assume a QWERTY keyboard and English spelling. SL2T is more like live machine translation from a visual language to written English with streaming output. That shifts what’s “fair.” To welcome signers into drills and leaderboards, we need metrics that reflect signing speed, system delay, and post‑edit effort—without penalizing left‑handed or one‑handed use.
Below is a practical, implement‑today metric suite your site can adopt.
Metric 1: Signs‑per‑Minute (SPM), defined fairly
Challenge: Gboard/Live Transcribe output English text, not a count of signed units. Two workable approaches:
- Prompt‑anchored SPM (recommended): Publish a library of ASL prompts with reference “sign counts” vetted by Deaf linguists (e.g., based on gloss‑level annotations or established teaching materials). During a test, show the English target sentence, the user signs the ASL equivalent, and you compute SPM = reference_sign_count ÷ elapsed_minutes. This avoids conflating English word count with ASL structure.
- Output‑proxy SPM (quick start): Use streamed English output as a proxy. SPM_est = WPM_output × r, where r is a prompt‑specific sign:word ratio you pre‑compute (e.g., 0.8–1.2) from small Deaf‑reviewed samples. Display it as “SPM (est.)” to remain transparent.
Implementation tips:
- Start timer on the first character arrival (not on “Start” button press) to avoid punishing network wake‑ups.
- End timer when no new characters arrive for, say, 1.0s and the user confirms “Done” (or Live Transcribe checkmark). This aligns with the modality’s pause‑based turn‑taking. (support.google.com)
Metric 2: Translation latency (because speed isn’t just on the user)
Measure the system, not the signer. Log these per test:
- First‑token lag: time from “Begin” click to first character inserted by SL2T (captures camera init + server round‑trip).
- Stream lag: median delta between observed signing bursts (approximate with user’s “segment” button or silence gaps) and text for that segment appearing.
- Finalization lag: time from user’s “done” gesture (hands out of frame or Live Transcribe checkmark) to last character output.
Why it matters: DeepMind explicitly targeted low streaming latency; your logs can confirm experience quality across lighting, networks, and devices. Publish anonymized latency dashboards so users can compare conditions (home Wi‑Fi vs. 5G, portrait vs. landscape, etc.). (deepmind.google)
Metric 3: Post‑edit cost after auto‑transcription
SL2T is strong, but errors remain in rapid fingerspelling, rare signs, some classifiers, and tense without context. Track how much cleanup users need after the model’s first pass:
- Compute character‑level or word‑level Levenshtein distance between the raw SL2T output and the final text the user submits after edits.
- Normalize as “edits per 100 characters” (E100) and “edits per minute” (EPM) so people can optimize speed vs. accuracy.
- Offer a toggle for “ASL‑faithful acceptance” that scores semantic equivalence instead of literal word order, acknowledging ASL’s distinct grammar.
Show users where edits cluster (e.g., fingerspelling spans) and recommend drills. DeepMind notes those error classes in examples. (deepmind.google)
Metric 4: Fairness for left‑ vs right‑handed and one‑handed use
DeepMind built for fairness across handedness (~10% of signers are left‑handed) and improved one‑handed signing. Your site should reflect that:
- Collect self‑reported handedness and whether the user is signing one‑ or two‑handed.
- Segment leaderboards accordingly, or normalize scores using per‑cohort medians (e.g., a FairScore = user_score ÷ cohort_median × global_median).
- Publish cohort‑level accuracy deltas (e.g., median WER difference) and set a parity goal such as “≤2 percentage‑point gap.”
- Add a mirrored‑UI rehearsal screen so left‑handed signers can verify framing before timed runs.
These design choices align with SL2T’s own fairness focus and make competitive play inclusive. (deepmind.google)
Test hygiene: Practical tips you can display before each run
- Framing & lighting: Use the front camera; keep face, upper body, and both hands visible; avoid backlighting and busy backgrounds. (support.google.com)
- Network: SL2T translates coordinates on secure servers; reliable connectivity improves latency. (support.google.com)
- Permissions & health: If Android shows a system‑health warning, close background apps. (support.google.com)
- Foldables: For in‑person drills, Live Transcribe’s Dual Screen mode can show text to both parties. (support.google.com)
Device caveats and what’s next
At launch (August 2026), SL2T sign‑to‑text is available on Pixel 11 via Gboard and Live Transcribe, starting with ASL→English. Google’s product post and DeepMind’s blog both state that more devices and additional sign languages are planned, and one‑handed use is supported from day one. For now, require a Pixel 11 in your test instructions and note the keyboard language prerequisite (en‑US or en‑CA). (blog.google)
Bringing signers onto the leaderboard—today
Start by adding a “Sign‑to‑Type (ASL)” tab next to QWERTY and voice tests. Run 60‑second drills with prompt‑anchored SPM, surface latency stats, and show post‑edit cost. Split results by handedness and one‑ vs two‑handed use, and celebrate parity milestones publicly. With SL2T in consumers’ hands, there’s no reason Deaf users should be excluded from speed tests or typing games anymore. (deepmind.google)
