Abstract:Simultaneous speech-to-speech translation (S2ST) holds the promise of breaking down communication barriers and enabling fluid conversations across languages. However, achieving accurate, real-time translation through mobile devices remains a major challenge. We introduce SimulTron, a novel S2ST architecture designed to tackle this task. SimulTron is a lightweight direct S2ST model that uses the strengths of the Translatotron framework while incorporating key modifications for streaming operation, and an adjustable fixed delay. Our experiments show that SimulTron surpasses Translatotron 2 in offline evaluations. Furthermore, real-time evaluations reveal that SimulTron improves upon the performance achieved by Translatotron 1. Additionally, SimulTron achieves superior BLEU scores and latency compared to previous real-time S2ST method on the MuST-C dataset. Significantly, we have successfully deployed SimulTron on a Pixel 7 Pro device, show its potential for simultaneous S2ST on-device.
| Source (Spanish) | Reference (English) | Predicted (English) |
|---|---|---|
| ver toda la filmografía de Atsushi Matsuda. | See the complete filmography of Atsushi Matsuda. | See the complete filmography of Atsushi Matsuda. |
| Papá, mira, solo quiero saber de qué estás hablando. | Dad look, I just want to know what you're talking about. | Dad look, I just want to know what you're talking about. |
| Bueno, eso era solo para la práctica, así que nadie está fuera del juego. | Well, that was just for practice, so nobody is out of the game. | Well, that was just for practice, so nobody is out of the game. |
| Y la policía francesa es igual de mala, peor, creo. | And the French Police are just as bad - worse, I think. | And the French Police as just as bad - worse, I think. |
| Pero no hablaron acerca de su matrimonio o de su marido. | But they did not talk about her marriage or her husband. | But they did not talk about his marriage or her husband. |
| Source (Spanish) | Reference (English) | Predicted (English) |
|---|---|---|
| ver toda la filmografía de Atsushi Matsuda. | See the complete filmography of Atsushi Matsuda. | See the complete filmography of Atsushi Matsuda. |
| Papá, mira, solo quiero saber de qué estás hablando. | Dad look, I just want to know what you're talking about. | Dad look, I just want to know what you're talking about. |
| Bueno, eso era solo para la práctica, así que nadie está fuera del juego. | Well, that was just for practice, so nobody is out of the game. | Well, that was just for practice, so nobody is out of the game. |
| De todos modos, creo que son reales, y si lo son, son espectaculares. | Anyway, I think they're real, and if they are, they are spectacular. | Anyway, I think they're real, and if they are, they spectacular. |
| Sí, es un gran disco de baile, pero escucha este disco con cuidado. | Yeah, it's a great dance record, but listen to this record carefully. | Yeah, that's a big dance record, but listen to that record carefully. |