Yasmin Moslem

CRE-Router: Cost-Aware LLM Serving

Wed, 28 Oct 2026 00:00:00 +0000

If you serve open-weight models on your own GPUs, you pay for GPU time, and that time depends on the GPU type, the request load and the response length. CRE-Router (Cluster, Route, Escalate) is a router, which decides which model in a pool answers each query, and it prices each model by the time it takes on the hardware that serves it. On three benchmarks, it comes close to the accuracy of the strong model it escalates to, at a lower cost per token.

Model routing sends each coming query to the one model in a pool that best balances cost and accuracy. Cascading lets a cheaper model answer first and passes the query to a stronger one when its answer is not good enough [1]. Most methods price a model from an API price list, a token count or a recorded run time, not from the hardware that serves it.

Stage 1 assigns each query to a cluster and routes it to the model chosen for that cluster under the cost budget. Stage 2 lets a quality estimator accept the answer or escalate the query to the strong model.

How CRE-Router Works

CRE-Router combines routing and cascading in two stages.

Stage 1 groups the training queries into clusters, using k-means on their embeddings. For each cluster, it scores every model by its error on that cluster and its cost, measured either as the time per output token (TPOT) or the end-to-end latency per request (E2EL). The operator sets a budget on that cost, and CRE-Router selects the most accurate routing within it. At inference, a new query is assigned to its nearest cluster and sent to that cluster’s model, so a query from a hard cluster goes straight to a strong model and does not pay for a weak answer first.

Stage 2 applies where a cluster is routed to a cheaper model. A small classifier, the quality estimator (a fine-tuned ModernBERT-base), reads each answer and either accepts it or escalates the query to the strong model. It catches the errors that routing cannot predict from the query alone.

Both stages are fitted from task-correctness labels, which standard benchmark evaluation already provides, so no extra annotation is needed.

Results

We evaluated CRE-Router on two telecom benchmarks, TeleMath and TeleQnA, and on AIME, with pools of two to nine open-weight models served with vLLM on one and two A100 GPUs [2]. CRE-Router is more accurate than the cheaper models in its pool and close in accuracy to the strong model it escalates to, at a lower cost per token. Given the same cost per token, it is more than 12 points more accurate than FrugalGPT [3] on TeleMath, and significantly more accurate on TeleQnA.

Accuracy against cost on TeleMath on one A100, per output token (left) and per request (right, log scale). Higher and further left is better; the line joins the systems that no other system beats on both. CRE-T20 is CRE-Router fitted under a TPOT budget of 20 ms. CRE-Router lies on the frontier under both cost terms.

Three Findings About Cost

Cost term. The cost term decides which models serve the workload. TPOT does not see how long an answer is, so a TPOT budget admits models that reason at length. On TeleMath, a TPOT budget sends two of the four clusters to Qwen3-30B with reasoning, while an E2EL budget sends them to Gemma4-26B without reasoning, at about the same cost per token and a fraction of the wait per request.

Every path a TeleMath request can take through CRE-Router, with the share of requests that take it and their mean E2EL. Under the TPOT budget, over 70% of requests go to reasoning models and wait more than three minutes on average; under the E2EL budget, no path averages more than 40 seconds.

Per-cluster cost. A model can be cheap on one kind of query and expensive on another. Pricing each model per cluster, rather than with one cost for the whole workload, shortens the time per request by 20 to 22% on TeleMath, at comparable accuracy.

Hardware. A second GPU speeds models up unevenly, so at the same budget, one and two A100s select different routings on all three benchmarks. Refitting needs new cost measurements, but no new labels.

Routing pays when no single model is both accurate enough and cheap enough for the budget. If one model already is, serve that model.

CRE-Router at GitHub

The code, configurations, data and model outputs are on GitHub. The training statistics measured in the paper ship with the repository, so you can fit a routing without a GPU.

Fit a routing from the stored statistics:

git clone https://github.com/ymoslem/CRE-Router.git
cd CRE-Router
pip install -e .
cre fit --stats configs/telemath_stats_1xA100_Sep2026.json --budget 25

It prints the routing that a TPOT budget of 25 ms selects on TeleMath, and the quality estimators Stage 2 needs.

To use the package without cloning the repository, pip install "cre-router[full]" installs everything needed for serving, including vLLM, so it needs a GPU machine. The README covers the full pipeline, from clustering and evaluation to serving with vLLM.

References

Dynamic Model Routing and Cascading for Efficient LLM Inference: A Survey (Moslem & Kelleher, TMLR 2026)
Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving (Moslem et al., arXiv 2026)
FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance (Chen et al., TMLR 2024)

Acknowledgements

This work was funded by Huawei Ireland through a research collaboration with the ADAPT Centre at Trinity College Dublin.

Disaggregated Serving

Fri, 23 Oct 2026 00:00:00 +0000

I recently tried disaggregated serving, a setup some production systems use to serve LLMs faster. Normally, each GPU both processes the prompt (prefill) and generates the answer (decode). In disaggregated serving, prefill and decode run on separate GPUs, so long new prompts do not slow down answers that are already streaming.

In my test, I compared three standard replicas, one per GPU, with a disaggregated setup on the same three GPUs, where two GPUs process the prompts and pass the KV cache to a third GPU that only generates answers. Both setups received the same traffic.

Median time per output token (TPOT) and time to first token (TTFT) as traffic grows on three A100s; dashed lines mark the latency limits.

Results

As traffic grew to 16 requests per second, the time per token (TPOT) with disaggregated serving stayed under 10 ms, while standard replicas slowed to 72 ms. The trade-off is that with disaggregation, the time to first token (TTFT) is a little longer, because each request waits for its KV cache to be passed from one GPU to another. At 12 requests per second, the first token took 0.54 s against 0.24 s for standard replicas.

Counting only the requests served within 1 s to the first token and 20 ms per token (goodput), as in DistServe [1], disaggregated serving handled 10.0 requests per second against 7.6 for three standard replicas.

Chunked Prefill

The vLLM documentation notes that disaggregation does not improve raw throughput, and that tuning chunked prefill, which splits long prompts into smaller pieces, can reach a similar goal, although the right chunk size is hard to find. So I tried that too. With a 512-token chunk budget, standard replicas slowed to 29 ms instead of 72 ms, but still served fewer requests within the latency limits than the disaggregated setup, 8.0 against 10.0 requests per second.

When Disaggregation Helps

Prefill and decode use the GPU differently, as prefill is compute-intensive and decode is memory-intensive [2]. Disaggregation is likely to help most when prompts are long, as in agentic workflows with a growing context, few-shot prompting, or questions over long documents. The DistServe authors [1] make the same argument for long contexts, since the KV cache to transfer grows linearly with the prompt, while prefill computation grows quadratically, so the transfer takes a shrinking share of the time. They also note that with only a few GPUs there is little room to rebalance prefill and decode, yet even on three GPUs, the split improved goodput for these long prompts. With short prompts, prefill takes little GPU time, so it barely slows the answers being generated. Passing the KV cache from one GPU to another then adds latency without much gain.

Setup

The model is Qwen3-4B-Instruct, served with vLLM 0.19.0 on three A100 GPUs. Each prompt is a 25-shot TeleQnA question of about 3.5K tokens, with different examples per request. The disaggregated setup runs two prefill instances and one decode instance with vLLM’s NixlConnector, and the KV cache is sent over NVLink within one node. All setups use 128-token KV blocks with prefix caching off. Chunked prefill is on for the standard replicas (the vLLM default) and off on the prefill instances in disaggregated serving.

Requests arrive at random (Poisson) at 2 to 16 requests per second, with 600 requests per rate and one run per rate. The requests-per-second figures count only the requests that meet both latency limits, at the best of the eight rates. Accuracy is the same in every setup, 67.7 to 68.5%. Real systems also turn on prefix caching, which lowers latency in either setup.

Disaggregation in vLLM

Each GPU runs its own vllm serve instance with the same NixlConnector configuration. In my runs, the prefill instances processed whole prompts, with chunked prefill off, and since all instances share one machine, each needs its own NIXL side-channel port.

KV='{"kv_connector": "NixlConnector",
     "kv_role": "kv_both",
     "kv_buffer_device": "cuda"}'

# Prefill, GPU 0 (GPU 1 is the same, with ports 8101 and 5601)
CUDA_VISIBLE_DEVICES=0 VLLM_NIXL_SIDE_CHANNEL_PORT=5600 \
  vllm serve Qwen/Qwen3-4B-Instruct-2507 --port 8100 \
  --block-size 128 --no-enable-prefix-caching \
  --no-enable-chunked-prefill \
  --kv-transfer-config "$KV"

# Decode, GPU 2
CUDA_VISIBLE_DEVICES=2 VLLM_NIXL_SIDE_CHANNEL_PORT=5602 \
  vllm serve Qwen/Qwen3-4B-Instruct-2507 --port 8102 \
  --block-size 128 --no-enable-prefix-caching \
  --kv-transfer-config "$KV"

The standard replicas run the same vllm serve command on each GPU, without --kv-transfer-config, and the run with 512-token chunks adds --max-num-batched-tokens 512.

vLLM does not route requests between the prefill and decode instances by itself, so a small proxy sits in front of them. It sends each request to a prefill instance with max_tokens=1, then passes the kv_transfer_params that instance returns to the decode instance, which generates the answer. vLLM’s toy proxy server for NixlConnector is a good starting point.

References

DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving (Zhong et al., OSDI 2024)
Splitwise: Efficient Generative LLM Inference Using Phase Splitting (Patel et al., arXiv 2023)

Iterative Layer Pruning for Efficient Inference

Mon, 17 Nov 2025 00:00:00 +0000

Model pruning is a compression technique, that aims to remove redundant components without significantly compromising the model’s performance or accuracy. This process facilitates efficient deployment of complex models by making them smaller and faster.

Pruning is a hardware-agnostic compression approach. Unlike some other compression approaches such as quantisation, models resulting from structured pruning can be deployed on any modern GPU with similar performance gains. Moreover, pruning can be part of a sophisticated compression pipeline that incorporates other techniques such as quantisation and efficient fine-tuning (e.g. LoRA, QLoRA), which can lead to higher compression and efficiency levels.

In this article, we will cover some insights from our two papers about Iterative Layer Pruning at IWSLT 2025 [1] and WMT 2025 [2].

Iterative Layer Pruning

The process of Iterative Layer Pruning involves incrementally identifying and removing layers with minimal contribution to translation or generation quality, one layer at a time. The pruning process is usually followed by fine-tuning the resulting models on relevant training data to restore the translation quality. Moreover, knowledge distillation data from the baseline (teacher) model can be used to help the pruned (student) model to reach the quality of the teacher model.

Layer Importance Evaluation

We conduct layer importance evaluation by measuring translation performance without each layer. The process is as follows:

Remove one layer of the model.
Evaluate the model (chrF++).
Repeat for the rest of the layers.
Prune the least important layer (best chrF++ without it).
Repeat #1 to #4 until reaching the pruning target.

Evaluation Results

For translation from Czech to German (CES-DEU), pruning 8 layers and then fine-tuning the resulting model retains 98% of the translation quality (as measured by COMET), while achieving considerable speedup gains. Interestingly, for translation from English to Egyptian Arabic (ENG-ARZ), the model resulting from pruning up to 16 layers and then fine-tuning outperforms the Aya-Expanse-8B baseline for this language pair. Pruned models achieve up to ~2× speedup.

Knowledge Distillation

Knowledge Distillation aims at transferring knowledge from a larger model (teacher) to a smaller one (student). In “sequence-level” knowledge distillation, the student model is trained to generate sequences that match the teacher’s sequence outputs. Fine-tuning the pruned models on a combination of authentic and synthetic data (from Aya-Expanse-32B) improved the Czech to German (CES-DEU) translation quality, with the 24-layer pruned model nearly matching the performance of the Aya-Expanse-8B baseline.

Further performance gains

It is highly recommended to use an efficient inference engine such as vLLM, which outperforms inference with the Transformers framework. In both cases, pruned models demonstrate up to ~2× speedup.

Moreover, you can quantise the pruned models for further compression. For example, in our IWSLT 2025 paper, we applied QLoRA after pruning the models. However, notice that low-precision quantisation (e.g. 4-bit and 8-bit) requires special hardware (e.g. H100 or H200) to observe performance gains.

Questions and Answers

Q. Why iterative layer pruning instead of random or middle layer pruning?
- A. Iterative layer pruning relies on layer importance analysis. Hence, only the layers with minimal contribution to the output quality can be removed. In our experiments, iterative layer pruning achieves better results than middle layer pruning.
Q. Can we prune other parts of the model other than layers?
- A. There are two types of pruning, structured pruning and unstructured pruning. In structured pruning, you can remove whole layers, attention heads, or other entire computational blocks, while in unstructured pruning, you can remove individual weights from a neural network. While unstructured pruning can achieve higher compression rates, it requires specialised hardware for efficient deployment. On the contrary, models resulting from structure pruning can be deployed on standards GPUs.
Q. Is it better to fine-tune the baseline model before pruning?
- A. If your task, domain, or language is very different from the distribution of the baseline model, it is better to fine-tune the baseline model first. Otherwise, you can prune the baseline directly. On the contrary, fine-tuning after pruning is always required to restore the quality of the baseline model.
Q. Can we fine-tune after each layer pruning step?
- A. We experimented with both fine-tuning after each layer pruning step, and after a number of pruned layers. In both cases, there was no difference than just fine-tuning once after the end of the pruning process. This might be because of overfitting resulted from several fine-tuning passes.
Q. Can the same approach be applied to encoder-decoder models?
- A. Yes, we applied this iterative layer pruning approach to both a decoder-only model, Aya-Expanse, and an encoder-decoder model, Qwen2-Audio. However, for encoder-decoder models, we observed that only pruning the decoder leads to better overall performance.

GitHub Repository

References

Efficient Speech Translation through Model Compression and Knowledge Distillation (Moslem, IWSLT 2025)
Iterative Layer Pruning for Efficient Translation Inference (Moslem et al., WMT 2025)

Adaptive Translation and Terminology with Large Language Models

Wed, 10 Jan 2024 00:00:00 +0000

Translation clients expect consistent terminology and style, but a translation model that has never seen their approved translations cannot adapt in real time. Large language models (LLMs) offer a simple way around this. If the prompt includes a few similar approved translations, or a short glossary, the LLM follows them while translating the new sentence, with no retraining. This is called in-context learning. We studied this idea in several papers. This post collects what we found and what you can do with it.

How It Works

The approach needs a translation memory (TM), which is a collection of approved translations. For each new sentence, you retrieve the most similar sentences from the TM, called fuzzy matches. In our experiments, we compared sentence embeddings to find them. Then you put the matches in the prompt, followed by the new sentence. A prompt with two matches looks like this.

English: <source fuzzy match 2>
Arabic: <target fuzzy match 2>
English: <source fuzzy match 1>
Arabic: <target fuzzy match 1>
English: <new source segment>
Arabic:

The most similar match (1) sits closest to the new sentence, and the model continues the pattern.

Terminology works in a similar way. In our WMT 2023 submission, we gave the LLM an existing translation and asked it to use the required terms and leave everything else the same. This is a form of automatic post-editing, where an LLM revises the output of an MT system.

What the Results Show

Similar examples beat random ones

In the EAMT 2023 paper, we tested GPT-3.5 on English to Arabic, Chinese, French, Kinyarwanda and Spanish, using TICO-19 as the TM. Few-shot translation with similar sentences beat few-shot translation with random sentences, and both beat zero-shot translation. Adding more matches helped (2, then 5, then 10), with smaller gains each time.

An LLM can beat a strong MT system, depending on the language

Without examples, ModernMT translated better than GPT-3.5 for the four language pairs it supports. With 5 or 10 similar examples, GPT-3.5 beat ModernMT for English to Chinese, French and Spanish, even when ModernMT used a TM too. For Arabic it did not, partly because of how the GPT-3.5 tokenizer handles Arabic. Kinyarwanda is a low-resource language. How well an LLM supports your language pair matters a lot.

If the LLM is weaker, add the MT output to the prompt

For English to Arabic, putting the translation from an encoder-decoder system (OPUS) in the prompt, next to the similar examples, beat both the MT system alone and the LLM with examples alone. The same idea helped English to Chinese with Google’s translation. When the LLM was already clearly better, as for French and Spanish, adding MT output did not help.

Terminology adherence can improve, with some care

An LLM extracted term pairs from sentence pairs, and over 95% were accurate in our human evaluation for Arabic, French and Spanish. Giving glossary terms in the prompt made the model use them more often, although forcing terms sometimes hurt grammar, and contradictory terms hurt quality.

For the WMT 2023 terminology task (German to English, English to Czech, Chinese to English), our pipeline had four steps. An LLM generated synthetic sentences from the terminology, we fine-tuned an MT model on a mix of that data and generic data, we translated with it, and finally an LLM post-edited the output to insert missing terms. On the blind test set, the pipeline raised the average share of required terms used in the translations from 36.67% with the baseline model to 72.88%. Translation quality did not suffer, and it even improved according to automatic metrics (BLEU, chrF++ and COMET).

A small open model can be enough

If you cannot send your data to a commercial API, you can fine-tune an open model. We fine-tuned Mistral 7B with QLoRA on 20,000 Spanish to English medical segments, half with plain prompts and half with one similar example. For this language pair and domain, in plain translation, the fine-tuned model matched NLLB 3.3B and beat gpt-3.5-turbo. With one similar example, it matched gpt-3.5-turbo and beat NLLB 3.3B. Training on both kinds of prompts means one model handles regular and adaptive translation.

An LLM is not always the answer

In a 2025 follow-up on medical translation for English to French, English to Portuguese, English to Swahili and Swahili to English, NLLB-200 3.3B beat every LLM we tested in the 7 to 8 billion parameter range in three of the four directions. Fine-tuning improved Mistral and Llama, but fine-tuned NLLB stayed ahead. Before you choose, compare a task-specific MT model with an LLM on your own language pair.

Code and Data

The fine-tuning code, data and evaluation scores are on GitHub. The repository covers fuzzy match retrieval with Sentence-Transformers and FAISS, fine-tuning, inference with CTranslate2 (and the ChatGPT API for comparison), and evaluation. It includes Spanish to English, plus English to French, Portuguese and Swahili, and Swahili to English.

Papers and Citations

Adaptive Machine Translation with Large Language Models

Moslem, Haque, Kelleher and Way. EAMT 2023. First preprint, January 2023. PDF

BibTeX

@inproceedings{moslem-etal-2023-adaptive,
    title = "Adaptive Machine Translation with Large Language Models",
    author = "Moslem, Yasmin  and
      Haque, Rejwanul  and
      Kelleher, John D.  and
      Way, Andy",
    booktitle = "Proceedings of the 24th Annual Conference of the European Association for Machine Translation",
    month = jun,
    year = "2023",
    address = "Tampere, Finland",
    publisher = "European Association for Machine Translation",
    url = "https://aclanthology.org/2023.eamt-1.22/",
    pages = "227--237",
}

Domain Terminology Integration into Machine Translation: Leveraging Large Language Models

Moslem, Romani, Molaei, Haque, Kelleher and Way. WMT 2023. First preprint, October 2023. PDF

BibTeX

@inproceedings{moslem-etal-2023-domain,
    title = "Domain Terminology Integration into Machine Translation: Leveraging Large Language Models",
    author = "Moslem, Yasmin  and
      Romani, Gianfranco  and
      Molaei, Mahdi  and
      Kelleher, John D.  and
      Haque, Rejwanul  and
      Way, Andy",
    booktitle = "Proceedings of the Eighth Conference on Machine Translation",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.wmt-1.82/",
    doi = "10.18653/v1/2023.wmt-1.82",
    pages = "902--911",
}

Fine-tuning Large Language Models for Adaptive Machine Translation

Moslem, Haque and Way. Preprint, December 2023, and a chapter of my thesis. PDF

BibTeX

@article{moslem-etal-2023-fine-tuning-llms,
      title={Fine-tuning Large Language Models for Adaptive Machine Translation}, 
      author={Yasmin Moslem and Rejwanul Haque and Andy Way},
      year={2023},
      eprint={2312.12740},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2312.12740}, 
}

Language Modelling Approaches to Adaptive Machine Translation

Moslem. PhD thesis, Dublin City University, 2024. PDF

BibTeX

@article{moslem-2024-adaptive-mt-llms,
      title={Language Modelling Approaches to Adaptive Machine Translation, {PhD} thesis}, 
      author={Yasmin Moslem},
      year={2024},
      eprint={2401.14559},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2401.14559}, 
}

Domain-Specific Translation with Open-Source Large Language Models: Resource-Oriented Analysis

Wassie, Molaei and Moslem. Preprint, 2025. PDF

BibTeX

@article{Wassie2025-DomainSpecific-Translation-LLM,
  title         = "{Domain-Specific Translation with Open-Source Large Language Models: Resource-Oriented Analysis}",
  author        = "Wassie, Aman Kassahun and Molaei, Mahdi and Moslem, Yasmin",
  month         =  may,
  year          =  2025,
  url           = "https://arxiv.org/abs/2412.05862",
  archivePrefix = "arXiv",
  primaryClass  = "cs.CL",
  eprint        = "2412.05862"
}

Purely Synthetic Bilingual Data for Machine Translation?

Mon, 12 Dec 2022 00:00:00 +0000

Fine-tuning a machine translation (MT) system for a specialised domain needs in-domain bilingual data, which is often scarce or missing. This post shows how a large language model can generate synthetic in-domain text, which you then back-translate and use for mixed fine-tuning. In our Arabic-English experiments, this improved translation quality in both directions, in automatic and human evaluation.

Method

Interestingly, when you feed such large language models with an in-domain sentence, they can generate more synthetic sentences, that simulate the domain and linguistic characteristics of the authentic sentence. In the research “Domain-Specific Text Generation for Machine Translation” (Moslem et al., 2022), we investigated the feasibility of this domain-specific text generation technique, when either no or limited bilingual in-domain dataset is available. We proposed a novel approach to domain adaptation leveraging state-of-the-art pre-trained language models to generate huge amounts of synthetic bilingual in-domain data with the goal of improving translation of in-domain texts. The process can be summarised in three simple steps:

1. Text generation (target)

Generate target-side synthetic sentences using a large pre-trained language model.

When there is a small in-domain translation memory, you can use each target sentence as a prompt to generate text, that simulates the domain characteristics of the authentic in-domain data. If there is no translation memory at all, you can first forward-translate the source text to be translated, or a portion of it, using the baseline MT model.

2. Back-translation (source)

Back-translate the synthetic target-side sentences into source language.

Combining the idea of in-domain text generation with back-translation, you can generate huge amounts of synthetic bilingual in-domain data, for both use cases.

3. Mixed fine-tuning

Fine-tune the baseline model, on a mix of synthetic and authentic data.

Finally, the baseline MT model should be fine-tuned using a combination of the synthetic bilingual in-domain dataset and a randomly sampled section of the original generic dataset.

Target Text Generation

Among autoregressive language models, trained to predict the next word in a sequence, are BLOOM, GPT-3, and GPT-J. This code snippet shows how to load the GPT-J language model. You can use some efficient loading techniques such float16 and low_cpu_mem_usage.

from transformers import GPTJForCausalLM, AutoTokenizer
import torch


tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-j-6B", padding_side="left")
tokenizer.pad_token = tokenizer.eos_token

model =  GPTJForCausalLM.from_pretrained("EleutherAI/gpt-j-6B",
                                         revision="float16",
                                         torch_dtype=torch.float16,
                                         low_cpu_mem_usage=True,
                                         cache_dir = "models_cache/",
                                         pad_token_id=tokenizer.eos_token_id)
model = model.half()
model = model.to("cuda")

Afterwards, you can use each target segment in the authentic in-domain dataset, to generate synthetic in-domain text. We use top-K and top-p sampling to generate diverse text sequences. Here, we set num_return_sequences to generate 5 sequences. Each sequence might include multiple sentences. You can then split these text sequences into several sentences using any sentence splitter.

target_segment = "I am an example sentence that talks about something very specialized!"
input_ids = tokenizer(target_segment, add_special_tokens=True, return_tensors="pt").input_ids.to("cuda")

sample_outputs = model.generate(input_ids,
                                do_sample=True, 
                                max_length=300, 
                                top_k=50, 
                                top_p=0.95, 
                                num_return_sequences=5,
                                early_stopping=True)

generated_text = tokenizer.batch_decode(sample_outputs, skip_special_tokens=True)

The quality of the language model is important. Here, you can see examples of generated text, that is both linguistically correct, and factually accurate.

In March 2020, India ordered the countrywide shut down of all non-essential economic activities due to the spreading COVID-19 pandemic.

While the overall worldwide economic impact of COVID-19 will only be realized through the end of 2020 and the recovery phase in 2021, it is clear that certain parts of the world have been severely impacted.

Sometimes, the generated text can be linguistically correct; however, numbers or names might be inaccurate.

Antiviral drugs are approved for pregnant women and should be considered for children younger than XX years, although some are still being investigated.

Scientists have found some species of unicorn in Amazon rainforests.

If there are only small mistakes, the generated synthetic data can be still used. Obviously, the better the quality of the text generated by the language model, the better the quality we can expect when fine-tuning the baseline MT model on this synthetic data.

Back-Translation

Now, we have the target side of our new in-domain dataset. To generate the source side, use back-translation, into the other language direction. For back-translation, you can either train another MT model yourself, or use pre-trained models such as OPUS models. Optionally, you can convert OPUS models to the CTranslate2 formate with quantisation to enhance efficiency.

Basically, both the source and target sides of our new large in-domain dataset, consist of synthetic data. The target side is generated by a language model, while the source text is generated by back-translation, in the other language direction.

Mixed Fine-Tuning

Now, it’s time to apply mixed fine-tuning to the baseline model.

In other words, continue training our baseline model on a mix of (a) the synthetic bilingual in-domain dataset we obtained from the two previous steps, and (b) a randomly sampled portion of the original generic dataset. In our experiments, we oversampled the synthetic in-domain dataset, by a 9x ratio.

Mixed fine-tuning (Chu et al., 2017)

To apply oversampling, we employed the dataset weights feature in OpenNMT-tf. If you are using OpenNMT-py or OpenNMT-tf, you can find more details in this tutorial on mixed fine-tuning of MT models.

Results

In both scenarios, our proposed method achieves significant improvements, demonstrated by both automatic and human evaluations. As expected, Setup 1 (when there is a tiny bilingual dataset) reveals better results than Setup 2 (where there is no bilingual dataset at all). Still, models resulted from both setups outperform the baseline model.

Evaluation results on the in-domain test set, TICO-19

Conclusion

Previously, synthetic data for machine translation had been created either on the source-side only (forward-translation) or the target side only (back-translation). In some cases, researchers replaced a few words from the source and/or the target with synonyms or similar words. The assumption was that “relevant” monolingual data was available, which was not always the case!

In real-life scenarios, things can get more complex. Usually, there is insufficient human-produced data to train or fine-tune high-quality MT systems. Production-level projects can be so specialised, while mining crawled monolingual datasets is inefficient and not necessarily helpful.

This research work generates brand new synthetic data on both the source and target sides. It employs large language models to put together coherent sentences similar to those to be translated in the current project. Then, the new synthetic data can be used to fine-tune production-level MT systems for domain-specific scenarios. Feel free to check out our paper, Domain-Specific Text Generation for Machine Translation.

Download Scripts

You can download our scripts and configuration files at GitHub. If you have applied the method and/or have questions, please let me know.

Citation

@inproceedings{moslem-etal-2022-domain,
    title = "Domain-Specific Text Generation for Machine Translation",
    author = "Moslem, Yasmin  and
      Haque, Rejwanul  and
      Kelleher, John  and
      Way, Andy",
    booktitle = "Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)",
    month = sep,
    year = "2022",
    address = "Orlando, USA",
    publisher = "Association for Machine Translation in the Americas",
    url = "https://aclanthology.org/2022.amta-research.2",
    pages = "14--30",
    abstract = "Preservation of domain knowledge from the source to target is crucial in any translation workflow. It is common in the translation industry to receive highly specialized projects, where there is hardly any parallel in-domain data. In such scenarios where there is insufficient in-domain data to fine-tune Machine Translation (MT) models, producing translations that are consistent with the relevant context is challenging. In this work, we propose a novel approach to domain adaptation leveraging state-of-the-art pretrained language models (LMs) for domain-specific data augmentation for MT, simulating the domain characteristics of either (a) a small bilingual dataset, or (b) the monolingual source text to be translated. Combining this idea with back-translation, we can generate huge amounts of synthetic bilingual in-domain data for both use cases. For our investigation, we use the state-of-the-art Transformer architecture. We employ mixed fine-tuning to train models that significantly improve translation of in-domain texts. More specifically, in both scenarios, our proposed methods achieve improvements of approximately 5-6 BLEU and 2-3 BLEU, respectively, on the Arabic-to-English and English-to-Arabic language pairs. Furthermore, the outcome of human evaluation corroborates the automatic evaluation results",
}

Translation Auto-suggestions: What do Linguists Think?

Mon, 24 Oct 2022 00:00:00 +0000

Translation auto-suggestion and auto-completion are among the important features that can help translators better utilize Machine Translation (MT) systems. In a Computer-Aided Translation (CAT) environment, a translator can make use of the MT word auto-suggestion feature as follows. By typing a few words, or clicking a word in a proposed MT translation, a list of suggestions is displayed. By selecting one of the word suggestions from the list, the rest of the translation is modified accordingly.

In a user survey we designed and distributed via social media networks, we asked participants whether they thought an MT word-level auto-suggestions feature could be helpful, and provided a simple definition and an illustrative image. If their answer was “Yes”, the respondent was asked to specify a reason. By the time of writing this article, we received 41 responses to our survey. While we do not believe this survey is enough to justify introducing an auto-suggestions feature into every MT system, it can be an indicator as to why some users think such a feature could be helpful.

To answer the question, “Which of the following best describes you?” 46.3% (19) of the respondents chose “Translator/Linguist”, 31.7% (13) selected “NLP Engineer/Researcher”, and the rest 22% (9) were other “MT Users”, not included in the two aforementioned categories.

Among the respondents to the survey, 90.2% (37) answered “Yes” to the question “In general, do you believe that a word-level auto-suggestions feature is helpful?” The figure below shows the breakdown of answers to the question, “Why do you believe that a word-level auto-suggestions feature can be helpful?” taking into consideration those who answered “No” to the previous question.

Out of the 37 persons who believed a word-level auto-suggestions feature can be helpful, 40.5% (15) of the respondents specified that it can give them some inspiration. This answer is specifically interesting as it is not constrained by time-saving benefits; hence, it focusses more on effectiveness rather than efficiency. The respondent that answered with “Other” mentioned that it allows them to look for alternative senses or phrasings, especially when they suspect the initial translation is bad, and referred to this as “human in the loop”.

Respondents were allowed to give extra comments; among the notable comments were:

I think word-level suggestions can be a useful feature, particularly when the target language can have several translations of a single source word.

Word-level suggestions can be helpful, but sometimes you end up spending a lot of time figuring out if the MT suggestion is a valid translation in that context. So, I’m not really sure yet how I feel about it.

It’s useful, as long as it’s seen as a suggestion, and not inserted in the target where the translator is typing.

Among the respondents who answered “For me, it is easier or faster than typing”, comments included:

Though most of the time; the suggestions are lousy.

I don’t think it gives me inspiration as I mostly need it for structures, not single words.

Auto-suggestion does not have to come from machine translation. History is much more useful.

The last comment above might be referring to the fact that in some CAT tools, auto-suggestions can also include glossary terms, and translation memory sub-segments, which encourages further research efforts to investigate methods to enhance leveraging and interaction between various translation resources in human-in-the-loop environments.

We hope this survey will inspire future user studies to look deeper into how diverse users of MT and CAT tools prefer to utilize certain features, such as auto-suggestions, and the value they seek. More aspects should be taken into consideration such as language pairs, translation workflows, and user interfaces. This can help improve these features to better support linguists and other MT users and boost their productivity as well as translation quality.

Citation

@inproceedings{moslem2022-autosuggest,
    title = "Word-Level Auto-Completion: What can we achieve out of the box?",
    author = "Moslem, Yasmin  and
      Haque, Rejwanul  and
      Way, Andy",
    booktitle = "In Proceedings of the Seventh Conference on Machine Translation",
    month = Dec,
    year = "2022",
    address = "Abu Dhabi, UAE",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2210.12802",
}

Machine Translation Robustness

Wed, 28 Sep 2022 00:00:00 +0000

Let’s talk briefly about the concept of “Robustness” of neural machine translation (NMT) systems. While robustness should be emphasized when building any NMT system, even high-resource languages with plenty of data can still face linguistic challenges.

What does NMT “Robustness” mean?

It simply means that a certain NMT engine can handle a specific linguistic feature, found in the input to be translated. This feature might not be naturally occurring in the training data. Among examples of linguistic features we want our NMT model to be robust while translating are: domain terminology, proper names, number formats, text case, misspellings, code switching (between two languages), and untranslatables such as tags, email addresses, etc.

How can we improve NMT robustness?

The first step to machine translation robustness is defining the issues that your model frequently encounters when translating a certain type of text. This step is underestimated, and in my opinion it is a sign of the maturity of production-level operations.

This goes beyond numerical human evaluation, and moves a step further towards defining specific types of issues. In simple words, human evaluators are asked to mention a clear reason why they think a translation should be ranked as 3 out of 5, for example. At the beginning, they might be provided with common lists, but they should also have the option to add more issues, that can be integrated into that list later. Such explanations should not be vague; they should be precise in a way allowing MT engineers to fix these issues. Problematic words should be marked; sometimes the track-changes feature is used. The main question is: What is the most critical issue in this translation?

How can we improve NMT “Robustness”?

In the findings of the WMT2020 Robustness Shared Task, under the “Common Trends” section, Specia et al. (2020) stated: “Participating systems were trained following a standard recipe, i) using big-transformer models, ii) boosting performance with tagged back-translation, iii) continued training with filtered data and in-domain data (where available), iv) ensembling different models to obtain further improvements.”

In this sense, data augmentation techniques can be helpful, and then you integrate this new data into the NMT system, via combining or fine-tuning.

As training a new system frequently might not be feasible, it is common in some companies to temporarily apply on-the-fly find-replace operations on translations, until the next training is possible. Some researchers also suggest making such on-the-fly handling easier by injecting the training data with certain placeholders, to be able to replace later. To apply this, in a portion of the training data, natural tags (html, xml, long numbers, etc.) are replaced with pseudo-tags (e.g. <t0>, <t1>, <t2>, …). These pseudo-tags should be also added as user_defined_symbols to the SentencePiece model (cf. SPM options). At inference time, it is easy to define and replace these tags with untraslatables during pre-processing and post-processing steps. On a relevant note, activating the SentencePiece option split_digits helps with copying longer numbers without intervention, while the option byte_fallback sometimes helps with irregular characters in the training data.

References

MQM - Multidimensional Quality Metrics (Lommel et al., 2013)
Training Neural Machine Translation to Apply Terminology Constraints (Dinu et al., 2019)
Improving Robustness in Real-World Neural Machine Translation Engines (Gupta et al., 2019)
How Should Markup Tags Be Translated? (Hanneman and Dinu, 2020)
Evaluating Robustness to Input Perturbations for Neural Machine Translation (Niu et al., 2020)
Findings of the WMT 2020 Shared Task on Machine Translation Robustness (Specia et al., 2020)
Business Critical Errors: A Framework for Adaptive Quality Feedback (Stewart et al., 2022)
Improve MT for Search with Selected Translation Memory using Search Signals (Zhang 2022)

Preparing MT Data with SentencePiece and PyTorch

Sat, 09 Jul 2022 00:00:00 +0000

This tutorial shows how to prepare parallel data for training a machine translation (MT) model with PyTorch. We filter the data, train SentencePiece subword models, convert the text to IDs, split it into training, validation and test sets, and build padded batches. The code uses only SentencePiece and plain PyTorch.

Updated October 2026. An earlier version of this tutorial used Torchtext. Torchtext development has stopped, and version 0.18 (April 2024) was the last stable release. This version replaces the Torchtext functions with SentencePiece and PyTorch, and the steps stay the same. The code was tested with Python 3.11, PyTorch 2.9 and SentencePiece 0.2.

Table of Contents:

Installation and Importing
Filtering Data
Subword Tokenization
- Training SentencePiece Models
- Subwording Data with the SentencePiece Models
Data Preparation Function
Vocabulary
Build Batches
- Bucketing
Next Steps

Installation and Importing

Let’s first install PyTorch and SentencePiece:

pip3 install torch sentencepiece

Then, import what we need.

import random

import sentencepiece as spm
import torch
from torch.utils.data import DataLoader
from torch.nn.utils.rnn import pad_sequence

Filtering Data

It is highly recommended you filter your data before training an MT model. The purpose is mainly to remove empty rows, duplicates, and too-long source/target rows. You can either use one of Moses scripts (clean-corpus-n.perl), or one of my MT Preparation scripts (filter.py).

Subword Tokenization

Subwording is a type of text tokenization. Dividing words into subword units help decrease the size of vocabulary and hence minimize out-of-vocabulary situations. Among subwording techniques are Byte-Pair Encoding (BPE) (Sennrich et al., 2016) and unigram language model (Kudo, 2018), supported by several tools. SentencePiece is one of the popular tools used for subwording text.

Training SentencePiece Models

Unless you use a shared vocabulary, you have to train one model for the source text and another model for the target text. The following code creates source.spm.model and source.spm.vocab for the source, and the same two files for the target.

The IDs of the special tokens are set explicitly, so we know them later: <unk> is 0, <pad> is 1, <s> (start of a sentence) is 2, and </s> (end of a sentence) is 3.

source_file = "source.txt"
target_file = "target.txt"
vocab_size = 32000

for name, path in [("source", source_file), ("target", target_file)]:
    spm.SentencePieceTrainer.train(
        input=path,
        model_prefix=f"{name}.spm",
        vocab_size=vocab_size,
        model_type="unigram",
        unk_id=0, pad_id=1, bos_id=2, eos_id=3,
    )

print("Done! Training SentencePiece models completed.")

Training prints a long log. You can choose model_type="bpe" instead of "unigram" for Byte-Pair Encoding.

💡Tip: What to do if “Vocabulary size too high”?

If you are experimenting with a very small file, a vocabulary size of 32000 might be too big. SentencePiece stops with an error such as “Vocabulary size too high (32000). Please set it to a value <= 664.” The message tells you the largest size your data supports, so use a value up to that number. Obviously, it would be better to add more data.

💡Tip: What if the corpus is very large?

SentencePiece does not need every sentence to train a good model. For a corpus of tens of millions of sentences, train on a random sample by adding input_sentence_size=10000000 and shuffle_input_sentence=True to the arguments of SentencePieceTrainer.train. For languages with a large character set, such as Chinese or Japanese, keep the default character_coverage=0.9995. For languages with a small character set, you can set it to 1.0.

💡Tip: How to rather build a shared vocabulary?

To train one model on both languages, give SentencePiece both files, separated by a comma. This works well for languages that share a script or many words.

spm.SentencePieceTrainer.train(
    input=f"{source_file},{target_file}",
    model_prefix="shared.spm",
    vocab_size=vocab_size,
    model_type="unigram",
    unk_id=0, pad_id=1, bos_id=2, eos_id=3,
)

Then use the same model for the source and the target in the next steps.

Subwording Data with the SentencePiece Models

Load the trained models.

source_sp = spm.SentencePieceProcessor(model_file="source.spm.model")
target_sp = spm.SentencePieceProcessor(model_file="target.spm.model")

The encode method converts text into subword pieces or into IDs, depending on out_type. Both of the following accept a single sentence or a list of sentences.

print(source_sp.encode("This is an example.", out_type=str))
print(source_sp.encode("This is an example.", out_type=int))

For training, we use the IDs.

Data Preparation Function

This function reads the source and target files, optionally lowercases them, converts the sentences to IDs, and splits the data into training, development and test sets. The pairs are shuffled together before splitting, so the source and target sentences stay aligned.

def prepare(source_file, target_file, dev_size, test_size, source_sp, target_sp, lower=False, seed=1):
    with open(source_file) as source, open(target_file) as target:
        source_lines = [line.strip() for line in source]
        target_lines = [line.strip() for line in target]

    if len(source_lines) != len(target_lines):
        raise ValueError("Length of source and target lines must be the same!")

    if lower:
        source_lines = [line.lower() for line in source_lines]
        target_lines = [line.lower() for line in target_lines]

    # Subword the sentences, i.e. convert them to lists of IDs
    source_ids = source_sp.encode(source_lines, out_type=int)
    target_ids = target_sp.encode(target_lines, out_type=int)

    # Shuffle the pairs together, then split into dev, test, and train
    data = list(zip(source_ids, target_ids))
    random.Random(seed).shuffle(data)

    dev_dataset = data[:dev_size]
    test_dataset = data[dev_size:dev_size + test_size]
    train_dataset = data[dev_size + test_size:]

    return train_dataset, dev_dataset, test_dataset

Now, we can use this function to prepare the data.

dev_size = 2000
test_size = 2000

train_dataset, dev_dataset, test_dataset = prepare(source_file, target_file, dev_size, test_size,
                                                   source_sp, target_sp, lower=False)

Let’s print some data statistics.

print("Training dataset size: %s" % len(train_dataset),
      "Dev dataset size: %s" % len(dev_dataset),
      "Test dataset size: %s" % len(test_dataset),
      sep=" | ")

Now, let’s print a few lines of the test dataset.

print(*test_dataset[:5], sep="\n")

Vocabulary

With SentencePiece, the vocabulary is the model itself, so there is no separate step to build one. Each model already maps every piece to an ID and back.

print("Source vocabulary size:", source_sp.get_piece_size())
print("Target vocabulary size:", target_sp.get_piece_size())

# IDs of the special tokens
print("pad, unk, bos, eos:", target_sp.pad_id(), target_sp.unk_id(), target_sp.bos_id(), target_sp.eos_id())

# Convert IDs back to pieces
print(source_sp.id_to_piece([2, 15, 16, 3]))

Keep the .model files together with your trained model, because you need the same ones at translation time. Unknown pieces map to the ID of <unk> automatically.

At translation time, encode the source with the source model, and decode the IDs that the model generates with the target model.

print(target_sp.decode([2, 15, 16, 3]))

Special tokens such as <s> and </s> are left out of the decoded text.

Build Batches

As training large datasets is time consuming, we build batches from the training data. First, let’s determine the device, the batch size, and the maximum length of the source and target sentences.

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
batch_size = 128  # examples
max_len = 100  # tokens in source or target
pad_idx = target_sp.pad_id()

Very long or empty sentences make training slower and less stable, so we keep only training pairs of a reasonable length.

train_dataset = [(source, target) for source, target in train_dataset
                 if 1 < len(source) < max_len and 1 < len(target) < max_len]

The collate function builds one batch. It adds the start and end tokens, which help the model know when to stop, and converts each sentence to a tensor. Sentences are padded to the longest sentence in the batch, so all of them have the same length.

def collate_batch(batch):
    sources, targets = [], []
    for source, target in batch:
        source = [source_sp.bos_id()] + source + [source_sp.eos_id()]
        target = [target_sp.bos_id()] + target + [target_sp.eos_id()]
        sources.append(torch.tensor(source, dtype=torch.int64))
        targets.append(torch.tensor(target, dtype=torch.int64))

    sources = pad_sequence(sources, padding_value=pad_idx)
    targets = pad_sequence(targets, padding_value=pad_idx)

    return sources, targets

Finally, let’s create the data loaders, which yield batches. Each takes the collate_batch function that we have just created as an argument. We shuffle the training data only.

train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True, collate_fn=collate_batch)
valid_dataloader = DataLoader(dev_dataset, batch_size=batch_size, shuffle=False, collate_fn=collate_batch)
test_dataloader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False, collate_fn=collate_batch)

If you want to test a data loader, you can use the following code. In your training loop, move each batch to the device.

for x, y in test_dataloader:
    x, y = x.to(device), y.to(device)
    print(x.shape, y.shape)

Each shape is (sequence length, batch size), because pad_sequence puts the sequence first. If your model expects the batch first, pass batch_first=True to pad_sequence.

Bucketing [optional]

Sentences in a random batch have very different lengths, so a large part of each batch can be padding. Bucketing groups sentences of similar lengths in the same batch, which reduces the padding and speeds up training. The following function shuffles the data, sorts chunks of it by source length, cuts the chunks into batches, and shuffles the batches.

def bucket_batches(dataset, batch_size, chunk_size=100, seed=1):
    rng = random.Random(seed)
    data = list(dataset)
    rng.shuffle(data)

    batches = []
    for i in range(0, len(data), batch_size * chunk_size):
        chunk = sorted(data[i:i + batch_size * chunk_size], key=lambda pair: len(pair[0]))
        batches += [chunk[j:j + batch_size] for j in range(0, len(chunk), batch_size)]

    rng.shuffle(batches)
    return batches

Since the batches are already formed, set batch_size=None in the data loader. Create the data loader again for each epoch, with a different seed, to get new batches.

train_dataloader = DataLoader(bucket_batches(train_dataset, batch_size),
                              batch_size=None, collate_fn=collate_batch)

Next Steps

You can now feed these batches to a PyTorch model, such as torch.nn.Transformer. If you prefer to train with a toolkit, see How to Build and Deploy Machine Translation Models.

How to Build and Deploy Machine Translation Models

Tue, 15 Mar 2022 00:00:00 +0000

This is a Neural Machine Translation (NMT) tutorial with OpenNMT-py and relevant tools. It covers data preprocessing, model training, evaluation, and deployment. The tutorial was put together as part of a mentorship activity I organised in 2022.

Fundamentals

Data Processing (notebook, repository)
NMT Model Training with OpenNMT-py (notebook)
Translation/Inference with CTranslate2 (code)
MT Evaluation with BLEU (tutorial, repository)
Simple Web UI (tutorial, repository)

Advanced Topics

Running TensorBoard with OpenNMT (tutorial)
Low-Resource Neural Machine Translation (tutorial)
Domain Adaptation with Mixed Fine-tuning (tutorial)
Overview of Domain Adaptation Techniques (tutorial)
Multilingual Machine Translation (tutorial)
Preparing Data for PyTorch Models with SentencePiece (tutorial)
Using Pre-trained NMT models with CTranslate2 (tutorial)

Mixed Fine-Tuning - Domain Adaptation That Works!

Thu, 06 Jan 2022 00:00:00 +0000

Training a robust generic model is an interesting task. However, when you want to customize your Machine Translation model to observe the terminology and style of a certain domain or client, Domain Adaptation comes to life. In previous posts, we discussed several approaches to Domain Adaptation. In this post, we are going to concentrate on a very effective approach called Mixed Fine-Tuning, originally proposed by Chu et al., 2017.

Regular fine-tuning of an NMT model usually consists of two steps:

Building a baseline NMT model, e.g. a generic model.
Continuing training the baseline NMT model on an in-domain dataset.

However, fine-tuning in this way can lead to “catastrophic forgetting”, i.e. the model overfits the in-domain data, starts forgetting the information learned from the baseline data, and loses generalization. So in practice, when you compare its quality to the baseline model, the in-domain model would give a better BLEU score and human evaluation for translation of sentences very similar to the in-domain training dataset, but worse BLEU score for out-of-domain sentences or even new in-domain sentences.

Solution: Unlike plain fine-tuning, in the Mixed Fine-Tuning approach (Chu et al., 2017), you randomly sample a portion from the generic data you used to train the baseline model, and use it during the fine-tuning step along with the in-domain dataset. Over-sampling the in-domain data is the main trick.

The training procedure of the Mixed Fine-tuning approach is as follows:

Train a baseline NMT model on out-of-domain data until convergence.
Continue training the NMT baseline model on a mix of in-domain and out-of-domain data (by oversampling the in-domain data) until convergence.

In NMT tools, such as OpenNMT and MarianMT, dataset weights can be used to replicate over-sampling.

Example:

Dataset Counts:

Generic Dataset: 1,000,000 sentences
In-domain Dataset: 100,000 sentences

Use weights 1:10 so that the training takes 1 sentence from the bigger generic dataset, and 10 sentences from the smaller in-domain dataset.

Generic Dataset: 1
In-domain Dataset: 10

In this example, we sequentially sample 1 example from the “Generic Dataset” and 10 examples from the “In-domain Dataset” and so on. By giving the “In-domain Dataset” a higher weight, the model can learn the style and terminology from the in-domain dataset while still be able to generalize, i.e. output high-quality translations for out-of-domain sentences.

Setting the dataset weights differs from one tool to another. In OpenNMT-py, dataset weights are set as numbers as in the aforementioned example. In OpenNMT-tf, dataset weights are set as ratios.

Further notes on the Mixed Fine-tuning approach (feel free to experiment with something different, though!)

The approach works well for in-domain datasets between 50k and 500k. For very small in-domain datasets, this approach might not work well; for bigger in-domain datasets, you might want to try different weights; and for very big in-domain datasets, you can just use the in-domain dataset only, but enrich it with missing aspects like shorter sentences, if needed.
If your baseline training data is too big, you randomly extract 10 times the size of the in-domain data.
If both the generic and in-domain data are available before training the baseline, we build the vocabulary and SentencePiece models on all datasets, both generic and in-domain datasets.
During fine-tuning, we extract a dev/validation dataset from the in-domain dataset only.
After fine-tuning, we use two test datasets, one that we used for the out-of-domain baseline, and one extracted from the in-domain dataset, to make sure the model works in both cases.
To alleviate “catastrophic forgetting” on generic data, consider averaging the baseline model with the fine-tuned model.

Among the advantages of the Mixed Fine-tuning approach is that this fine-tuned NMT in-domain model still works well on both unseen in-domain data and general/out-of-domain data. Moreover, the approach can be fully automated (e.g. for various clients) once you verify it for your use cases.

It is worth mentioning that we have successfully applied the Mixed Fine-Tuning approach, proposed by Chu et al. (2017), in production-level scenarios in the industry. We also employed it in a number of our Domain Adaptation and Low-Resource NMT papers such as Haque et al. (2020) in combination with other approaches, through which we achieved the first place at ICON 2020 shared task, as well as Moslem et al. (2022) where we used synthetic in-domain data.

References

Axelrod, A., He, X., & Gao, J. (2011). Domain Adaptation via Pseudo In-Domain Data Selection. Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, 355–362. https://aclanthology.org/D11-1033
Chinea-Ríos, M., Peris, Á., & Casacuberta, F. (2017). Adapting Neural Machine Translation with Parallel Synthetic Data. Proceedings of the Second Conference on Machine Translation, 138–147. https://doi.org/10.18653/v1/W17-4714
Chu, C., Dabre, R., & Kurohashi, S. (2017). An Empirical Comparison of Domain Adaptation Methods for Neural Machine Translation. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 385–391. https://doi.org/10.18653/v1/P17-2061
Freitag, M., & Al-Onaizan, Y. (2016). Fast Domain Adaptation for Neural Machine Translation. In arXiv [cs.CL]. arXiv. http://arxiv.org/abs/1612.06897
Haque, R., Moslem, Y., & Way, A. (2020). Terminology-Aware Sentence Mining for NMT Domain Adaptation: ADAPT’s Submission to the Adap-MT 2020 English-to-Hindi AI Translation Shared Task. Proceedings of the 17th International Conference on Natural Language Processing (ICON): Adap-MT 2020 Shared Task, 17–23. https://aclanthology.org/2020.icon-adapmt.4
Kobus, C., Crego, J., & Senellart, J. (2017). Domain Control for Neural Machine Translation. Proceedings of Recent Advances in Natural Language Processing, 372–378. http://arxiv.org/abs/1612.06140
Luong, M.-T., & Manning, C. 2015. Stanford neural machine translation systems for spoken language domains. Proceedings of the 12th International Workshop on Spoken Language Translation: Evaluation Campaign, 76–79. https://aclanthology.org/2015.iwslt-evaluation.11
Moslem, Y., Haque, R., Kelleher, J., & Way, A. (2022). Domain-Specific Text Generation for Machine Translation. Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), 14–30. https://aclanthology.org/2022.amta-research.2
Moslem, Y. (2024). Language Modelling Approaches to Adaptive Machine Translation. In arXiv [cs.CL]. arXiv. http://arxiv.org/abs/2401.14559
Saunders, D. (2022). Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey. Journal of Artificial Intelligence Research, 75, 351–424. https://doi.org/10.1613/jair.1.13566
Sennrich, R., Haddow, B., & Birch, A. (2016a). Controlling Politeness in Neural Machine Translation via Side Constraints. Proceedings of the 2016 Conference of the North AMerican Chapter of the Association for Computational Linguistics: Human Language Technologies, 35–40. https://doi.org/10.18653/v1/N16-1005

Sennrich, R., Haddow, B., & Birch, A. (2016b). Improving Neural Machine Translation Models with Monolingual Data. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 86–96. https://doi.org/10.18653/v1/P16-1009