David Versus Goliath is story about an underdog who defeats a mighty opponent against overwhelming odds. What does it teach translators, teams and agencies about machine translation?
Despite six (6) decades of research and a proclamation of human parity, we continue to experience machine translations (MT) blunders and reality. A bigger is better mindset has prevailed through three (3) major MT technology generations. Coincidence?
First Generation MT
In MT’s humble beginnings, expert linguists painstakingly crafted rules for two languages. Software engineers encoded rules-based MT (RbMT) engines like SYSTRAN’s Desktop Translator. RbMT with its big teams of experts and Goliath-sized resources established the mindset that bigger is better when building MT engines.
The bigger is better mindset, that experts with Goliath-sized resources build MT engines, is RbMT’s lasting contribution to MT.
The Goliath-sized costs and low quality stopped RbMT’s growth. At the end of its life cycle, RbMT never achieved broad market acceptance.
Second Generation MT
In the early 2000s, artificial intelligence (AI), machine learning (ML), “big data” training corpora and data scientists replaced the expert linguists who crafted rules. Statistical machine translation (SMT) engines moved to powerful online servers because PCs couldn’t run ML and AI.
The bigger is better mindset persisted as SMT experts proclaimed only experts with Goliath-sized resources can build SMT engines.
Online SMT engines were designed as jacks of all trades, masters of none that serviced millions of requests across virtually every possible topic. Customized SMT engines delivered better quality but they were even more expensive.
Third Generation MT
By late 2016, newer ML & AI and “Goliath data” became the expert linguists. Neural machine translation (NMT) engines moved to Goliath-sized super-computers and displaced online SMT.
The bigger is better mindset reaches new heights as NMT experts secure ever bigger Goliath-sized resources to build NMT engines.
Bigger Is Better, Really?
Let’s see how the bigger is better transition from SMT to NMT improved the translator’s experience.
In mid 2016, Memsource published this study of Google and Microsoft SMT titled Machine and Professional Human Translations Identical. It shows translators accepted 5% of the MT suggestions as publishable without edits.
Two years later and after Google had changed to NMT, thirty-one (31) Slate Rocks customers translated their works with Google’s NMT. We learned 5% of Google’s NMT exactly matched their own translations.
Translators experienced practically no change in the NMT that matched the translators’ work but according to the bigger is better mindset the number should have increased.
Why do so many translators report that NMT is better? I suspect there’s a strong placebo effect. Experts say NMT is better. Translators know they’re working with NMT. Therefore, NMT is better. This is just like a patient feels better after a doctor administers a placebo and tells the patient he’ll feel better.
Unfortunately, most translators don’t monitor their work and NMT vendors are biased towards their products. We will never really know the truth until language researchers use genuine scientific studies with double-blind processes.
Underestimated Small Packages
The same thirty-one (31) Slate Rocks customers translated their works with their personal, customized Slate Desktop engine. We learned 34% of the customized SMT exactly matched their own translations.
A smaller SMT engine created a seven (7) fold increase in segments that matched the translators’ works! Why do translators report that SMT is not as good as NMT?
This Slate Rocks study was the first research to rely on objective criteria. Traditional language technology research uses subjective criteria and processes that are vulnerable to placebo effect. Researchers desperately need transformation to double-blind studies.
Through sixty (60) years of MT research and development, the bigger is better mindset has perpetuated two myths.
MT itself does not require Goliath-sized resources. The use case determines the size of the required resources.
Deploy an MT engine to a Goliath-sized use case and Goliath-sized resources are required. Example: MT engine deployed to online servers servicing millions of demands across virtually every possible topic requires a training corpus of tens of millions of segments from thousands of different translators.
Deploy an MT engine to a David-sized use cases require David-sized resources. Example: An MT engine for a translation team of 3 or 4 colleagues working on insurance claims requires a training corpus with 3 or 4 years of those translators’ segment pairs.
David Versus Goliath can be interpreted as the underdog defeating a mighty opponent against overwhelming odds.
It also teaches how a small competitor with confident, like a translator, team or small agency, wins against the self-absorbed giant stuck in outdated beliefs. It teaches bigger isn’t always better. It demonstrates good things come in small packages.