Machine Translation (MT) -
Custom Machine Translation Engines

Anyone can create a custom MT engine. But it takes experience and expertise to create a great custom MT engine that is optimized to produce MT output for you specific purpose. Much like in a kitchen, just throwing ingredients in a pot does not make a great meal. A good recipe, a skilled chief, and the right tools will give a much better result.

The key to creating a high-quality machine translation engine is a combination of the data that it is built on and pre-/post-processing that is optimized for your specific purpose. The video in the right-hand bar provides a good overview of what is involved.

With over 14 years of experience creating tens-of-thousands of custom machine translation engines, the Omniscien team have refined a methodology and set of tools that provide unmatched translation quality. Many of the tools and processes are unique to Omniscien and are designed to make it fast and easy to create your own custom machine translation engines. 

Why do you need a custom machine translation engine?

As machine translation becomes more and more accurate, industry specific machine translation engines that take into account the content of the text that is being translated have become common. These take you part of the way to better contextual support within your machine translation engine but machine translated content can produce much higher translation quality when it is processed via a professionally customized machine translation engine .

Like human translators, the highest quality machine translation is always achieved via a specialized translator that has had appropriate domain-specific training. Someone with deep knowledge of football will be able to translate that domain better than someone who specializes in legal, financial or medical.

A professionally customized machine translation engine will not only be adapted to your industry domain but also to your specific vocabulary and writing styles.

To illustrate the point, let’s examine the automotive domain. If you consider two manufacturers, Honda and Toyota, their cars generally target the same market segment and have similar features. As such, their writing style is quite similar. However, if you then compare Rolls Royce and Ferrari, the target market segment is very different and there is a very different writing style and choice of vocabulary in their sales and marketing content. 

In addition to the differences between each company within the automotive domain, there are different use cases. The writing styles for sales and marketing are totally different to legal or technical manuals. You would not want your sales and marketing content to read like a legal or technical document.

With Language Studio, multiple domains and multiple writing styles can be designed and included in a single custom engine, ensuring the appropriate writing style and translation output quality.

The benefits are very simple. Translation quality and accuracy are improved. If you are involved in human post-editing then there is less effort and time required to publish your content. If you are simply translating to communicate or understand information then the meaning of the translation is more accurate and the message conveyed more clearly. When a machine translation engine is professionally customized, it should be difficult to differentiate between machine translated content and human translation. In many use cases, machine translation post editing may not be needed at all.

Can your MT Engine Do This?

One of the many areas that we seek to understand is writing style. Our experienced project managers work with you to understand your requirements. One of the first things to determine is the writing style or styles. Multiple styles can be incorporated directly into a single machine translation engine.

In the example to the right, two writing styles were built into an engine for different purposes. You can see clearly that the output style between “Business” and “Children’s Books” is totally different, although the meaning is the same.

The Upload and Pray Approach

Our competitors have a very different, more simplistic approach. They request that you upload your existing translation memories and terminology if you have any, and then they combine that with their file translation memories to create a custom machine translation engine.

We refer to this approach as “Upload and Pray” because it is a blind approach that does not take into account any of the variables that go to make a great translation and does not consider the suitability of the data being provided for the intended purpose.

Also, the data that any organization usually has for training is typically between 10,000 to 500,000 sentences, greatly limiting the scope of coverage and benefits of MT engine customization.

This is the approach that nearly all of our competitors offer for customized machine translation, including Google Translate with Google AutoML and Microsoft Translator.

The Language Studio Approach

Modern MT engines are built on billions of sentences of data. While uploading your legacy translation memories and terminology will improve translation, it is a minor improvement overall. The reality is that you don’t have sufficient data (no organization does) to customize an engine to your writing style, context, and vocabulary. You have some data that can influence the output, but a true customization is based around understanding your purpose and use and adapting it specifically to meet that purpose. If this sounds familiar, it is because the basic premise of translating with a machine is no different than human translation. If you don’t guide your human translators properly, then they will not have the necessary information for the correct style, context and purpose. The final deliverable is likely to be sub-standard.

The Right Data is Essential to Delivering High Quality Translations

5 years ago, a typical custom machine translation engine was based on between 2-to-4 million generic bilingual sentences and between 100-to-500 thousand domain-specific sentences provided by the user/client. At that time, it was already a challenge for many organizations to gather that many bilingual translation memory sentences. If you did not have any of your own data, finding suitable data to customize a machine translation engine was a significant barrier to building your own custom machine translation engine.

With the advent of Neural Machine Translation, the volume of data required has multiplied many times over. Today’s typical custom machine translation engines are based on between 150 million to 1+ billion bilingual sentences.

This is comprised of approximately:

• 80% Language Pair Foundation Data that acts as a general base underpinning the engine for grammar, vocabulary, and core linguistic knowledge that trains the artificial intelligence algorithms on how to better understand relationships between words and phrases. Certain language pairs will have more training data available than others for training due to the ease of sourcing training data and the prominence of the language. Less prominent language pairs may have fewer resources available to create training data. Omniscien aims to translate directly between languages wherever possible. Already all official EU languages have at least 500 million bilingual sentences available for training in all directions without pivoting between English. Many have well over 1 billion bilingual sentences.
• 15% Industry Domain Data such as life sciences, information technology or automotive that teaches the engine a specific context.
• <1% Client Data makes up the total data volume. This is too small overall to make a significant difference, but does act as a base to manufacturing and synthesizing data. This data usually consists of translation memory files and term base files that can be easily exported from the clients translation management system. 
• 4-5% Manufactured and Synthesized Data makes a huge difference in translation quality. A good sized translation memory exported from a client’s translation management systems is a great help, but modern neural machine translation engines need more data than any one company can provide. Even with many translation projects, the data is unlikely to exceed 500,000 sentences for most organizations. Most companies struggle to come up with training data. 
  The additional training data needed to feed the machine learning algorithms with sufficient data to really make a translation quality difference is created using a set of powerful tools that are able to create specific data for your purpose. Even when no bilingual training data is available from the client, the Language Studio tools can still generate millions of unique sentences.

Language Studio provides powerful tools to create, match, mine and synthesize bilingual sentences that are specific to your context, domain, and purpose. A rapid customization can be performed with your own data with some reasonable translation quality improvement. Comparatively, with the additional data created by Language Studio tools, translation quality is taken to the next level.