RAG explained by building it: the smallest working app with Spring AI
You don't need LangChain or a vector database to understand RAG. The whole loop fits in two Java classes — and building it that small is the point.
by
RAG is three steps at question time — retrieve the most relevant chunks from a vector store, put them into the prompt, let the model answer grounded in them — plus an ingestion pipeline that runs once per document: extract text, chunk, embed, store. Part 1 of this four-part series builds exactly that and nothing more: Spring Boot with Spring AI's in-memory SimpleVectorStore, Azure OpenAI, Tika and Jsoup for extraction, and an Angular dashboard that shows which chunks every answer was grounded in. The full project is on GitHub, self-contained and runnable.
You don't need LangChain, a vector database, or a five-hour framework course to understand RAG. The whole loop fits in two Java classes. I know, because I just built it that way — deliberately as small as possible — as part 1 of a four-part tutorial series. This post walks through what RAG actually is, using code you can clone and run.
The problem RAG solves
A language model knows what was in its training data. It knows nothing about your wiki, your contracts, yesterday's meeting notes. Ask anyway, and it does the worst possible thing: it answers confidently. RAG — Retrieval-Augmented Generation — fixes this without touching the model. Instead of teaching the model your documents, you hand it the relevant excerpts at question time and tell it to answer from those.
Three steps, every single question:
- Retrieval — find the document chunks most similar to the question.
- Augmentation — put those chunks into the prompt as context.
- Generation — let the model answer, grounded in that context.
For step 1 to work, documents first pass through an ingestion pipeline, once: extract the text (Apache Tika for PDFs and Office files, Jsoup for web pages), split it into chunks, compute an embedding for each chunk — a vector of floats that captures its meaning — and store the vectors. Similar meaning, nearby vectors. That's the entire trick.
The whole loop, in code
Ingestion is a pipeline of four calls:
List<Document> chunks = splitter.apply(List.of(new Document(text, metadata)));
vectorStore.add(chunks); // embeds each chunk and stores it
And answering is three steps you can point at:
// 1. Retrieval
List<Document> chunks = vectorStore.similaritySearch(
SearchRequest.builder().query(question).topK(4).build());
// 2. Augmentation
String context = chunks.stream().map(Document::getText)
.collect(Collectors.joining("\n---\n"));
Prompt prompt = new Prompt(List.of(
new SystemMessage(SYSTEM_TEMPLATE.formatted(context)),
new UserMessage(question)));
// 3. Generation
String answer = chatModel.call(prompt).getResult().getOutput().getText();
That's it. The "augmented prompt" everyone talks about is a string with your chunks pasted in. No magic, no chains, no graph of runnables. The vector store in part 1 is Spring AI's in-memory SimpleVectorStore — gone on restart, and that's fine: while you're learning, persistence is one more thing hiding what actually happens.
What the dashboard teaches you
The Angular frontend is two panels: ingest on the left (file upload + URL), chat on the right. The detail that matters: every answer shows which chunks it was grounded in, with source and similarity score. The first time an answer cites the wrong chunk, you stop believing RAG is magic and start reasoning about retrieval quality — which is precisely the mental model you need before adding anything fancier.
What I deliberately left out
Hybrid search, reranking, metadata filtering, semantic chunking — all real techniques, all improvements you'll eventually want, all absent from part 1. Not because they don't matter, but because every one of them tunes one of the four pipeline stages, and you can't tune a stage you haven't seen in isolation. When a production RAG system gives a bad answer, the bug lives in one of these steps: extraction produced garbage, chunking cut mid-sentence, retrieval fetched the wrong chunks, or the prompt let the model ignore its context. Debugging that requires knowing the naive version cold.
The same goes for the framework question. Spring AI's ChatModel, EmbeddingModel and VectorStore are ordinary Spring beans — if you run JVM services in production, RAG becomes a library, not a new platform. If you're Python-native, take LangChain or LlamaIndex; the concepts in this series transfer one-to-one.
The takeaway
Build the smallest RAG loop before you adopt anyone's RAG stack. It's an afternoon of work, and it converts RAG from a buzzword into four debuggable functions. The complete project — backend, dashboard, walkthrough in English and German — is self-contained and runnable with one Azure OpenAI key.
Part 2 makes the two big infrastructure choices swappable: Azure OpenAI vs. plain OpenAI vs. a fully offline local Mistral, and in-memory vs. persistent pgvector — via Spring profiles, without touching this application code. That swap sounds trivial. It wasn't, and the details are the interesting part.
Code: github.com/halviclabs/rag-tutorials — part 1 is rag-tutorial-01-basics. And when your agent tooling needs to reach systems beyond documents, the same "small, explicit, debuggable" rule applies — see writing your first MCP server.
Frequently asked questions
What is Retrieval-Augmented Generation (RAG)?
RAG lets a language model answer questions about documents it was never trained on. At question time, three steps happen: retrieval (a similarity search finds the most relevant document chunks in a vector store), augmentation (those chunks are placed into the prompt as context), and generation (the model answers grounded in that context instead of guessing). Before any question can be answered, documents pass once through an ingestion pipeline: extract text, split into chunks, embed each chunk, store the vectors.
Do I need a vector database to build a RAG application?
Not to start. A vector store is any structure that answers "which stored chunks are closest to this query vector" — Spring AI's in-memory SimpleVectorStore does that in a few lines and is ideal for learning, because nothing is hidden. You graduate to a persistent store like PostgreSQL with pgvector when you need data to survive restarts, which is exactly what part 2 of the series adds.
What does chunking do in a RAG pipeline?
Chunking splits long documents into pieces small enough to embed meaningfully and to fit several of them into one prompt. Too large and a chunk's embedding averages over too many topics; too small and it loses context. The tutorial uses Spring AI's TokenTextSplitter with defaults — a reasonable starting point you only tune once retrieval quality demands it.
Why build RAG with Spring AI instead of LangChain?
If your team runs on the JVM, Spring AI gives you RAG with the abstractions you already operate: ChatModel, EmbeddingModel and VectorStore are Spring beans, providers are swappable via configuration, and there is no extra runtime. LangChain and LlamaIndex are excellent in Python ecosystems — the concepts in this series transfer one-to-one.