Building Toward a Production-Ready Bilingual AI Customer Service Platform on Azure

We are currently building a production-oriented Azure AI solution around a UAE retail customer-service scenario. The objective is not simply to connect a language model to a search index. We are treating retrieval, generation, machine learning, deployment, monitoring and rollback as parts of the same technical delivery problem.

This solution is still being implemented and evaluated. It is not a description of a live customer deployment or measured production results. It is the architecture and delivery approach we are building and testing.

Why bilingual customer-service AI is harder than it looks

For an English-only proof of concept, it is easy to test a small group of questions, see reasonable responses and conclude that the system works. That is not enough for a bilingual service.

A UAE customer might ask a question entirely in Arabic, entirely in English, or switch between the two in the same sentence. Product names, model numbers and technical terms are also often kept in English even when the rest of the query is Arabic.

The system therefore has to cope with Arabic sentences containing English product names while still retrieving the correct policy, product information or support document. More importantly, we cannot assume that an improvement in the overall system means an improvement for both languages.

The architecture

The design has two main technical paths. The first is a bilingual RAG assistant: approved knowledge moves through document storage, processing, embeddings and Azure AI Search before a language model produces a grounded response with citations. The second is an Azure Machine Learning path for predicting escalation risk from historical support-ticket information.

These paths solve different problems. RAG helps answer the customer using approved knowledge. The ML model provides a predictive signal that could help determine whether a case needs additional attention. Keeping the responsibilities separate also makes them easier to evaluate and operate.

Building the bilingual RAG layer

One design decision we are evaluating is whether English and Arabic content should use one multilingual search index or separate language-specific indexes. There is no reason to declare one approach universally better before testing.

A common multilingual index can simplify ingestion and operations. Separate indexes can provide more control where analyzers, ranking behaviour, processing or release cycles differ. The correct choice should come from evaluation rather than assumption.

Our test set therefore needs English questions, Arabic questions and mixed-language questions, including realistic cases where an English product or model name appears inside an Arabic sentence.

Retrieval quality and generation quality are different

For retrieval, the first question is whether Azure AI Search returned the information required to answer the query. That can include whether the expected document or chunk appears in the top results, ranking behaviour and relevance.

Generation is evaluated after that: is the response grounded in the retrieved evidence, did it answer the question, do the citations support the answer, and did the system behave safely when the documents did not contain enough information?

Arabic needs its own release criteria

Consider an illustrative result. These figures are examples, not measured project results:

If we looked only at English, the new candidate would appear better. Looking at the bilingual service we actually intend to provide, it is a regression. If Arabic falls below its agreed threshold, that candidate should not be promoted simply because English improved.

This is why the CI/CD design includes AI evaluation gates. Changes to prompts, retrieval settings, indexes or model configuration should be evaluated against a controlled dataset before promotion.

Predicting escalation risk with Azure Machine Learning

The second path uses Azure Machine Learning to build an escalation-risk model from historical support-ticket information. The planned flow covers feature preparation, model training, MLflow experiment tracking, evaluation, model registration and managed deployment.

One issue we are paying particular attention to is point-in-time correctness. If a model is intended to predict escalation at a particular stage of a case, it must only receive information that would have existed at that time.

A field created after escalation could look extremely predictive offline while being useless in a real prediction. Preventing that leakage is therefore a data-design requirement rather than a cleanup task after model training.

Version the complete AI system, not only the model

A RAG application's behaviour can change because the prompt changed, the model changed, the embedding model changed, chunking logic changed, source documents changed, search configuration changed or the index itself changed.

A model version alone is therefore not enough to reproduce a result. The release process needs traceability between the deployed application and the prompt, index, retrieval configuration and model versions it is using.

GitHub Actions provides the CI/CD mechanism in the proposed design, with separate development, test and production stages. Promotion should depend on both conventional software checks and the AI evaluation results that matter for the service.

Monitor two different types of health

Azure Monitor and Application Insights can answer the operational question: is the service available, how long are requests taking, are dependencies failing, are endpoints healthy?

But an HTTP 200 response does not tell us that an AI answer is correct.

AI-quality telemetry therefore needs to answer a second question: is the AI still behaving as expected? Depending on privacy and logging requirements, useful signals can include language, retrieved document identifiers, prompt/model/index version, citation behaviour, token use and evaluation results.

For a bilingual system, those quality signals need to be filterable by language so that an Arabic degradation cannot hide behind healthy infrastructure metrics and acceptable aggregate results.

Safe releases, rollback and retraining

Rollback needs to be designed before it is needed. For the RAG path, sufficient version information must be retained to return to a previous known-good configuration instead of rebuilding it under pressure.

For the ML path, multiple deployments behind an Azure Machine Learning online endpoint can support controlled release patterns such as staged traffic movement or blue/green deployment. Retraining should create a candidate that still has to pass evaluation; the newest model should not automatically become the production model.

Security follows the same production mindset: secrets belong in Key Vault rather than application configuration, while Entra ID, RBAC and managed identities reduce unnecessary credential handling.

Four practical lessons shaping the build

1. Evaluate languages separately. If Arabic matters to the service, Arabic needs its own dataset, metrics and release thresholds.
2. Test retrieval before blaming the language model. Generation evaluation cannot replace retrieval evaluation.
3. Version the complete AI system. Prompts, indexes, embeddings, retrieval settings and models can all change behaviour.
4. Design for a bad release. Evaluation gates, monitoring and rollback are architectural requirements, not final-stage extras.

The interesting engineering challenge is not getting an LLM to produce a convincing customer-service response. The harder problem is detecting when the system becomes worse, understanding which component caused the change, preventing a regression from being promoted and recovering safely when something goes wrong.