Slate Toolkit™

Slate Toolkit™ is our open source distribution of the machine translation tools for phrase-based SMT models that are in our commercial Slate Desktop™ products, packaged as a stand-alone developer toolkit.

To our knowledge, Slate Toolkit™ is the world’s only complete collection of native Windows 64-bit utilities for phrase-based SMT. Others have referred to this as the Moses Toolkit. It is not. Some of the packaged utilities come from the Moses Toolkit, some are not part of Moses and countless Moses utilities are not included. Please read the Unsupported Utilities & Features section carefully.

Getting Started

Lengthy command lines from the various open source utilities can be error-prone. Therefore, this toolkit includes Windows .cmd and Bash .sh shell scripts, plus a very small sample training corpus. These do not constitute a production-ready environment. Rather, they demonstrate the open source command lines for essential steps that prepare corpora, train & tune models and translate text.

The demo-all script is the best place to start. Just run it in-place.

Outputs from upstream scripts become the inputs to downstream scripts. Therefore, the script names are numbered in the order to follow when you run them individually. The order is also referenced in the demo-all script.

Supported SMT Utilities

Support for open source utilities maintains cross-platform functionality of phrase-based “mode” of statistical machine translation. Factored phrase-based, hierarchical and other SMT modes might work, but are not tested or supported.

Supported SMT Features

Support for phrase dictionary, lexical reordering, language modeling and other advanced features is maintained through the supported utilities.

Unsupported Utilities & Features

Open source utilities and features that are not used in our commercial products are either missing or untested, and therefore unsupported. This collection distributes all utilities under their respective open source licenses without warranty. The README file has more details.

These features and functions come with Slate Toolkit™.

Languages

Support for 28 languages in any combination. With a bilingual corpus, possibly converted from your translation memories, you can create translation models between 756 language pairs.

Sample Translation Memories

Sample translation memories and other files to help you practice and learn.

Software Development Tools

Slate Toolkit™ is the kernel that provides phrase-based statistical machine translation functionality in our proprietary applications. A skilled software engineer can use it to create their own programs with features like these.

  • Organize Translation Memories
  • Build Customized Engines
  • Evaluate Engines
  • Deploy Engines
  • Pre-Translate Files
  • Plugins Connect to CAT Tools
  • Forced Terminology
  • Terminology On-The-Fly
  • Weighted Updates
  • Backup & Restore

You need to provide the following before working with Slate Toolkit™.

Hardware System Requirements
  • Intel Core i3 (i7 recommended) or AMD Athlon 64 CPU (4-core x86-64, 2.4 GHz or faster)
  • 4 GB of RAM (8 GB recommended)
  • 2 GB of free hard drive space for installation
  • 250 GB (or more) free space on a high-performance drive is ideal after installation
Windows Operating System (Option)
  • Windows 7 64-bit Edition with Service Pack 1
  • Windows 8 or 8.1 64-bit Edition
  • Windows 10 64-bit Edition
  • 32-bit Editions not supported
Linux Operating System (Option)
  • Linux, x86_64 kernel version 3.2+
  • Ubuntu 16.04 or newer (other Debian-based on request)
  • CentOS/RHEL-based (other RPM-based on request)
MacOS Operating System (Option)
  • To be determined, currently unsupported
Translation Memories (corpus)

Personalized engines

  • 70,000 to 150,000 sentence segments
  • One full-time translator’s work for 3 to 4 years

Customized engines

  • 200,000 to 500,000 sentence segments
  • Support a team of translators
  • No upper limit number of segments
  • Too many segments risks degrading the engine
File Types

SlateToolkit can use these file types:

  • Text files with UTF-8 character encoding, Linux or Windows new line separators
  • Tab-delimited files are specialized Text file (as above) with one tab per line. Text left of the tab is the source language. Text right of the tab is target language.

The installer installs and manages the following required dependencies.

GNU Utilities

The GNU Utilities are essential open source utilities that the Slate Toolkit™ needs to create models. The MS Windows package installs and updates them but we do not maintain them.

Perl Scripting Runtime

Perl 64-bit version 5.28 or newer is a free open source scripting runtime environment.

Python Scripting Runtime

Python 64-bit version 3.72 or newer is a free open source scripting runtime environment. Required dependency libraries include: pip, pywin32, six, numpy, nltk, lxml, regex, polib, jieba, PyArabic, tinysegmenter3, hazm, wxPython

License and support included with your purchase of Slate Toolkit™.

Open Source Licenses

We distribute Slate Toolkit™ with many open Source software components, from many different projects, written by many authors and contributors, under their respective open source licenses. In turn, they can have sub-components from different sources that could be licensed differently.

Multiple Platforms

Install and activate on any supported operating systems. Today’s support includes MS Windows and Linux. MacOS is planned.

Free Open Source

You may redistribute this package and its utilities freely under their respective open source licenses. The price is a packaging and distribution fee.

Slate Toolkit™ distributes these components under their respective open source licenses.

Slate Demo

Shell scripts (Windows .cmd files and Bash .sh files) and a sample training corpus to demonstrate native Moses and MGIZA++ utilities.

MGIZA++

MGIZA++ is an essential open source utility that is the Slate Toolkit™ uses to create statistical machine translation engines.

Moses Decoder

The decoder is the utility that converts source to target text using statistical machine translation engines created by the toolkit.

Moses Toolkit

The toolkit is a collection of open source utilities, derived from the original Moses open source project, that create SMT models.

GNU Utilities

The GNU Utilities are essential open source utilities that the Slate Toolkit™ needs to create models. The MS Windows package installs and updates them but we do not maintain them.

Slate Toolkit™ is our open source distribution of the machine translation tools for phrase-based SMT models that are in our commercial Slate Desktop™ products, packaged as a stand-alone developer toolkit.

To our knowledge, Slate Toolkit™ is the world’s only complete collection of native Windows 64-bit utilities for phrase-based SMT. Others have referred to this as the Moses Toolkit. It is not. Some of the packaged utilities come from the Moses Toolkit, some are not part of Moses and countless Moses utilities are not included. Please read the Unsupported Utilities & Features section carefully.

Supported SMT Utilities

Support for open source utilities maintains cross-platform functionality of phrase-based “mode” of statistical machine translation. Factored phrase-based, hierarchical and other SMT modes might work, but are not tested or supported.

From MGIZA++:

  • mgiza(.exe)
  • mkcls(.exe)
  • snt2cooc(.exe)
  • merge_alignment.py

From Moses Toolkit:

  • build_binary(.exe)
  • consolidate(.exe)
  • evaluator(.exe)
  • extract(.exe)
  • extractor(.exe)
  • lexical-reordering-score(.exe)
  • lmplz(.exe)
  • mert(.exe)
  • moses(.exe)
  • processLexicalTable(.exe)
  • processPhraseTable(.exe)
  • query(.exe)
  • score(.exe)
  • symal(.exe)
  • extract-parallel.perl
  • filter-model-given-input.pl
  • filter-rule-table.py
  • flexibility_score.py
  • giza2bal.pl
  • LexicalTranslationModel.pm
  • mert-moses.pl
  • moses_sim_pe.py
  • reduce_combine.pl
  • score-parallel.perl
  • train-model.perl
Unsupported Moses Utilities

Slate Toolkit™ does not include all of the utilities in the original Moses and MGIZA++ projects. We do not support utilities that are not listed above. For example, these utilities are neither included nor supported:

  • BerkeleyAligner
  • IRSTLM
  • RandLM
  • SRILM

Some utilities that we package with in Slate Toolkit™ may or may not work. We have not tested them because they are not needed by our commercial Slate products and therefore, we do not support them. These utilities includes but are not limited to the following:

  • clean-corpus-n.perl
  • detokenizer.perl
  • lowercase.perl
  • snt2cooc.pl
  • tokenizer.perl

The above lists may change. Contact us if you have questions.

Supported SMT Features

Support for phrase dictionary, lexical reordering, language modeling and other advanced features is maintained through the supported utilities.

  • PhraseDictionaryMemory
  • PhraseDictionaryBinary
  • LexicalReordering (memory)
  • LexicalReordering (binary)
  • KenLM (all modes)
  • max-kenlm-order=12
  • with-xmlrpc-c (support for -xml-input)
  • cmph support
Unsupported Moses Features

We do not support feature that are not listed above. Those include, but are not limited to these features.

  • BerkeleyAligner
  • PhraseDictionaryOnDisk
  • PhraseDictionaryCompact
  • LexicalReordering (compact)
  • IRSTLM
  • RandLM
  • SRILM
  • hierarchical models
  • suffix arrays
  • bilingual language models

If you need cross-platform (e.g. on Windows) support for a particular utility or feature from one of the original open source packages that is not listed here, please let us know. We may be able to add it to Slate Toolkit™.

Getting Started

Lengthy command lines from the various open source utilities can be error-prone. Therefore, this toolkit includes Windows .cmd and Bash .sh shell scripts, plus a very small sample training corpus. These do not constitute a production-ready environment. Rather, they demonstrate the open source command lines for essential steps that prepare corpora, train & tune models and translate text.

The demo-all script is the best place to start. Just run it in-place.

Outputs from upstream scripts become the inputs to downstream scripts. Therefore, the script names are numbered in the order to follow when you run them individually. The order is also referenced in the demo-all script.

File Types

SlateToolkit can use these file types:

  • Text files with UTF-8 character encoding, Linux or Windows new line separators
  • Tab-delimited files are specialized Text file (as above) with one tab per line. Text left of the tab is the source language. Text right of the tab is target language.
Caveats

These open source projects were written for academic use on Unix-like systems. Therefore, there are a few things you can do to protect yourself from problems.

Naming of files and folders

Unix and Windows deal with locations of files and folders in different ways:

  • Windows paths use drive letters and backslashes; Unix paths use slashes to indicate where a file is. Slate Toolkit™ generally supports each system’s native style, but if you run into glitches, please let us know.
  • In Windows, “a.txt” and “A.txt” are the same file; in Unix they are different. Avoid names that can be confused in this way, and make sure you capitalize all names consistently. The software may not always realize that the two are the same name on your system.
  • Unix software often uses whitespace to separate one filename from another. Files or folders are allowed to have whitespace in them, but it often brings out bugs in software. Avoid whitespace in file and folder path names.
  • Handling of non-ASCII characters can differ between individual computers depending on configuration, and likewise often triggers software bugs.
  • Many punctuation marks can have special meanings on different systems, such as colons, quotes and apostrophes, equals signs, dollar signs, percentage signs, asterisks, tildes, and so on. Avoid these marks. Keep it simple! When in doubt, use dashes and/or underscores.

For trouble-free use, we recommend that you use only files and folders with names consisting exclusively of ASCII letters (a-z), digits (0-9), dots, and dashes or underscores (-, _). With your help and patience we hope to improve the user experience over time.

Line endings

On Windows systems, a line of text ends in a fixed sequence of two characters: carriage return and line feed, also written as “”. Unix systems use just the line feed, or “”.

This may confuse some tools when dealing with files that were not written with your system’s native line endings. Windows Notepad may show all contents in a Unix text file as a single, long line; or instead of returning to the starting column for every new line, some software may just start the next line right below where the last one ended. Rare Unix tools may interpret the carriage returns as “jump back to the beginning of the line and erase all text that was previously displayed.”

Slate Toolkit™ accepts input files with either style of line endings. It generally creates output files with platform-specific line endings, but at times it creates Unix-style files on Windows systems. This may not be perfect and with your feedback we hope to improve it over time.

Contact Information
Slate Rocks LLC
Web: https://www.slate.rocks/
Email: [email protected]
System Hardware Requirements
  • Intel Core i3 (i7 recommended) or AMD Athlon 64 CPU (4-core x86-64, 2.4 GHz or faster)
  • 4 GB of RAM (8 GB recommended)
  • 2 GB of free hard drive space for installation
  • 250 GB (or more) free space on a high-performance drive is ideal after installation
Operating System Requirements

Windows Systems

  • Windows 7 64-bit Edition with Service Pack 1
  • Windows 8 or 8.1 64-bit Edition
  • Windows 10 64-bit Edition
  • 32-bit Editions not supported

Linux Systems

  • Linux, x86_64 kernel version 3.2+
  • Ubuntu 16.04 or newer (other Debian-based on request)
  • CentOS/RHEL-based (other RPM-based on request)

MacOS Systems

  • To be determined, currently unsupported
GNU Utilities

The GNU Utilities are essential open source utilities that the Slate Toolkit™ needs to create models. The MS Windows package installs and updates them but we do not maintain them.

  • sort(.exe)
  • split(.exe)
  • libiconv2(.dll) and libintl3(.dll)
  • gzip(.exe), also copied as gunzip(.exe) and bzcat(.exe)
Perl Scripting Runtime

Perl 64-bit version 5.28 or newer is a free open source scripting runtime environment.

  • The Windows package installs Perl from Strawberry Perl and updates the system %PATH%.
  • The Linux package works “out of the box” on Ubuntu 12.04 or newer systems. Other Linux systems may work. We welcome your feedback about your experiences.
Python Scripting Runtime

Python 64-bit version 3.72 or newer is a free open source scripting runtime environment. Required dependency libraries include: pip, pywin32, six, numpy, nltk, lxml, regex, polib, jieba, PyArabic, tinysegmenter3, hazm, wxPython

  • The Windows package installs Python from python.org and updates the system %PATH%.
  • The Linux package works “out of the box” on a standard Ubuntu 12.04 or newer systems. Other Linux systems may work and we welcome your feedback about your experiences.

Secured by PayPal

Social Sharing Discount

A win! win! win! You get 10% off at checkout. Your colleagues learn about Slate. We have lower advertising expenses. Thank you!