I presented at Drupal Dev Days Athens 2026, focusing on a RAG-based .po file translation tool that I initially started building for the Japanese Drupal community. While building it, I realised it didn’t have to target only the Japanese community, so I made it multilingual. I then realised it would be useful outside of the Drupal ecosystem so I generalised it further.

The result is an over-engineered proof-of-concept called the RAG-LLM Translator .

Below is a summary of the talk . You can also watch the session on YouTube , and the slide deck can be downloaded from here .

The problem

During the contribution sprint at Drupalcon Nara in 2025, about 10 of us gathered to discuss how to utilise LLMs to ease the burden of translating Drupal strings into Japanese.

Translation is hard - at least I find it so when I translate from English to Japanese, or vice versa. I might be going off on a tangent, but when translating an English sentence to Dutch, it feels more like a conversion than translation. The two languages don’t only belong to the same Indo-European linguistic family, but also under the Germanic branch. As a result, the two languages share the same words, phrases, and also some grammar. On the other hand, Japanese and English don’t share anything by default. If I were to suggest a similarity between them, it would be nothing more than the English loanwords that began entering the language around 150 years ago (and some of the words already carry different meanings, or are used with different nuances).

Consistency of LLM translation

LLMs translate from one language to another incredibly well. The primary challenge with LLM translation, however, is consistency.

The use of terminology

For example, the word ‘publish’ can be translated into several different Japanese words. And in the context of Drupal UI, the word ‘publish’ must always be translated as 「掲載」. However, LLMs almost always translate it as 「公開」 (FYI I’ve tried over 10 different models, and I can say this with confidence). We could include a glossary in the prompt, but the more irrelevant information we include, the higher the chance of degrading the translation quality. Also, you don’t want to have a bloated prompt unless you have lots of tokens to spare.

Some words are also translated differently depending on the context. They appear fine and the sentences flow naturally, but they are often not consistent with the rest of the UI text and the inconsistencies confuse users.

The RAG-LLM Translator as a solution

The RAG-LLM Translator was originally built as a proof-of-concept to be shared with the Japanese translation community and evaluated if the use of RAG could improve the consistent use of terminology by LLMs. But I enjoyed building it so much that I got carried away, and ended up adding lots of features.

Quality

As shown in the presentation, the translator comes with an LLM-as-a-Judge feature, which performs a randomised blind test to compare translations with and without RAG, evaluating at a 95% confidence level with +/-5% margin of error.

Reports by LLM-as-a-Judge show the translations produced by RAG-LLM Translator score consistently higher when compared to LLM-only translations. The results are available in the thread of the project’s issue queue so you can check out the detailed results if you are interested.

Other features

As I mentioned, it’s packed with features:

Customise

Prompt override per language

The translator comes with two default prompts: ja.md and generic.md, and the latter is applied when your target language is not Japanese. You can follow the file naming convention {lang code}.md and add a custom prompt for your target language.

Glossary generator

You have no glossary? No problem. Simply ingest existing translations into the vector DB, and run a command to generate a list of candidates for your glossary. As a bonus, if there are inconsistencies in how certain words are translated, you will find them in the list so you can improve the existing translation.

demo_prep.sh

The script will conveniently download all the files needed to test run the system.

RAG fine-tuning helper

The translator’s RAG is pre-tuned, but it may not be suitable for your application. Also as your glossary grows, it may become less accurate and you will need to fine-tune it. The translator comes with a fine-tuning helper and a solid documentation so you are not stuck with the default thresholds.

Post-processing framework

LLMs often fail to apply string formatting rules consistently. In Japanese, for example, it is recommended that a whitespace is added between a multibyte character and a ‘half-width’ latin characters for improved readability. Here’s an example:「正しい URL を入力してください」(‘Please enter a correct URL’) - notice a whitespace before and after the word URL. I’ve tried making LLMs follow such rules, but it’s been a hit and miss so I added the post-processing framework to take care of such formatting as a post-translation process.

This framework accepts custom plugins, so if your language requires specific formatting, you can add your own rules and configure the plugin to run for the target language of your choice.

Final thoughts

The talk was well received, I think. I organised a BoF and a couple of separate sessions in the following days to help six participants set up the translator on their local environment so they could try it out.

During the process, we found a couple of bugs that led to patches and updates to the documentation. I also received great feature requests, some of which have already been implemented.

Building the translator has been great fun, and it was a privilege to share it with the community. I look forward to improving the tool further, and hope that it will help reduce the burden of OSS translation maintainers.

Lastly, I’d like to thank the Drupal Dev Days Athens volunteers for being so helpful and working so dillgently before, during and after the event. It was definitely one of the best event I’ve attended!