My dW is taking on machine translation. We just launched a pilot of IBM Research's Real-Time Translation Solution (RTTS), affectionately called n.Fluent. This smart software translates text between English and 11 other languages.
I've included the n.Fluent widget in the left hand column of my blog. Using n.Fluent is a breeze. Select the language from the pulldown and translate. Magic happens as the page reloads in the selected language. When you hover over any section of the translated page, a pop-up window displays which shows the original English and allows you to enter a better translation.
Now comes the hard part. How can we assess the quality of the translated content ? Our ultimate goal is to provide real-time, "good enough" translation across additional areas of the site. But, what are the questions and what are the quantifiables, and more perplexing, how do you measure "good enough" ?
Here are the initial questions we need to answer, as well as my thoughts on how to narrow in on the answer. I would appreciate any thoughts or suggestions you have. I will be working with our implementation team to structure our pilot's feedback avenues.
1. Is there a need for real-time machine translation on my dW ?
A number of users will invoke translation out of curiosity. We need to know how many users repeatedly use the widget. This will give us a rough idea of how many users are need real-time translation as opposed to how many just want to check it out.
2. Which languages are in greater demand ?
If we measure how many times the widget is invoked per language, we get a general idea of which languages are in greater demand or of greater interest to our user base.
3. How acceptable is n.Fluent's current quality ?
The suggested content improvements will give us an idea of the quality of the raw translation. We should also include some open-ended questions to assess current MT quality via a user survey or feedback form.
4. How willing is our community to crowdsource machine translation results ?
Again, by counting the number of suggested improvements, we get an idea of how active our community is willing to help improve n.Fluent's machine language dictionaries.
5. What are other comments or questions our users having regarding my dW and machine translation ?
We need to maintain an open dialogue with users to determine what is working, what is not working, and their suggestions for how n.Fluent can better be used on my dW.
Thanks for sharing any thoughts or ideas you have !