Your time is valuable. You’re probably considering Slate™ Desktop as a productivity tool. Traditional human and automatic MT evaluations are time consuming and confusing. Slate™ Desktop has a unique Engine Summary Report that shows an engine’s “vital signs” that are fast and easy indicators of the engine’s health.
Human vital signs (body temperature, pulse rate, respiration rate and blood pressure) do not indicate a person is healthy. Likewise, engine vital signs do not indicate an engine will produce quality translations. Rather, vital signs both indicate absence or presence of serious problems.
If the engine vital signs indicate serious problems, you can quickly delete the engine with confidence because human evaluation would waste your time.
In the absence of problems, the vital signs serve to estimate the engine’s capability of creating a measurable number of quality translations and you can start human evaluation.
Quality quotient (QQ) is the percent of exact matches between Slate™ Desktop output and human translations you should look for when using your engine. I.e. the percent of edit-distance zero segments. Higher scores indicate translators are likely to benefit more from the engine. Personalized Slate™ Desktop engines typically score 25% to 35%. Google and Microsoft NMT typically score 3% to 9%.
BLEU Score (all) reports the closeness between all Slate™ Desktop segments and human references, including exact matches. Google & Microsoft typically score 25 to 35. Slate™ Desktop engines typically score 65 to 75.
BLEU Score (filtered) reports the closeness between Slate™ Desktop segments and human references that are not exact matches. The difference between the “filtered” and “all” scores indicates how much work expect when fixing the errors. An engine with a high QQ and high BLEU filtered score is very likely to perform well for the translator.
Edit Distance Per Line is the average edit distance (Levenshtein) score for lines that are not exact matches. Lower scored indicate fewer changes are necessary to change the Slate™ Desktop output to the final translation.