Swap the LLM, keep the code: one RAG app on Azure, OpenAI and local Mistral

I swapped the LLM under a running RAG application three times last week — Azure OpenAI to the plain OpenAI API to a local Mistral running in Docker. The application code never changed. The configuration nearly beat me, and the two places where it fought back are exactly what part 2 of my RAG tutorial series is about.

Part 1 built the smallest working RAG loop: ingest documents, retrieve chunks, augment the prompt, generate. One provider (Azure OpenAI), one in-memory store, everything visible. Part 2 asks the question that decides whether an abstraction is real: can you change the expensive decisions — which model, which store — without touching the code that does the work?

The design: profiles select, code injects

The RAG service and the ingestion pipeline inject three Spring AI interfaces: ChatModel, EmbeddingModel, VectorStore. Which implementations arrive is decided at startup by Spring profiles — one axis for the provider (azure, openai, ollama), one for the store (simple in-memory, pgvector on PostgreSQL). Any combination works:

mvn spring-boot:run -Dspring-boot.run.profiles=pgvector,openai
mvn spring-boot:run -Dspring-boot.run.profiles=simple,ollama   # fully offline

The ollama variant is the one I care about most: Mistral for chat, nomic-embed-text for embeddings, both served by a dockerized Ollama. No API key, no cloud, no document leaves the machine. For anyone answering questions over internal documents under Swiss or EU data expectations, "the offline profile exists and is a first-class citizen" is not a demo feature — it's the argument.

Trap one: none means all of them

With three provider starters on the classpath, Spring AI activates every autoconfiguration whose selector property is unset. So you disable them all by default and let each profile re-enable its own:

spring.ai.model:
  chat: none
  embedding: none

That's what I wrote first. It compiles, it looks complete, and it crashes at startup under the ollama profile — with an Azure error: "Endpoint must not be empty."

The reason: chat and embedding are not the only model types. The Azure and OpenAI starters also ship image, audio and moderation autoconfigurations, each with its own selector (spring.ai.model.image, spring.ai.model.audio.transcription, …) — and each defaulting to active when unset. The image autoconfiguration dutifully tried to build an Azure client for a profile that has no Azure credentials. The fix is boring and complete: set every selector to none in the base config. The lesson is less boring: when you disable by default, enumerate the whole surface, not the part you're using. Conditional beans that default to "on" fail in the profile you didn't test.

Trap two: swapping embeddings is a data migration

The chat model is genuinely swappable: same prompt in, answer out, nothing stored. The embedding model is not, for two stacked reasons.

First, dimensionality: OpenAI/Azure text-embedding-3-small produces 1536-dimensional vectors, nomic-embed-text produces 768. pgvector bakes the dimension into its table schema, so the tutorial exposes it as a config knob (rag.vector-store.dimensions) that the ollama profile overrides.

Second — and this one bites people who got the dimensions right — vectors from different models live in different spaces. Even if the sizes matched, comparing a query embedded by model A against chunks embedded by model B produces distances that mean nothing. Retrieval doesn't error; it just quietly returns garbage.

So the honest rule: changing the embedding provider means dropping the vector table and re-ingesting everything. Treat it like a schema migration, not a config change. Practically, that means keeping your sources retrievable — files, URLs, wiki pages — so re-ingestion is a script you run, not an archaeology project. The tutorial's ingestion history exists precisely so you can see what would need to be replayed.

What pgvector buys you

The second profile axis adds PostgreSQL with the pgvector extension via docker-compose. I picked it over dedicated vector databases for a reason worth stating: it's just Postgres. Backups you already know, SQL access to your chunks (select * from vector_store is a debugging superpower), operations your team already runs. A dedicated vector DB earns its place at a scale most internal RAG systems never reach.

The takeaway

An abstraction is only real once you've swapped the implementation under it — and the swap is where you learn. Two things transfer beyond this tutorial: disable-by-default must cover the entire autoconfiguration surface, and embedding models are stateful decisions with migration costs, not interchangeable endpoints.

Part 3 connects the pipeline to where documents and questions actually live: Confluence, Microsoft Teams and Slack — as connectors that are off by default and verify every webhook signature.

Code: github.com/halviclabs/rag-tutorials — part 2 is rag-tutorial-02-llms, self-contained with docker-compose for Postgres and Ollama.