Situation:
Many companies want answers from their private documents, but sending them to a cloud LLM is not always an option. This ongoing personal AI project, started in 2026, tests a provider-independent approach.
Task:
As architect and developer, I designed a RAG application that answers questions from private documents with either a local or a cloud LLM. The goal was grounded answers, with data staying in-house when needed.
Action:
- Designed the indexing flow as single-responsibility classes, with structure-aware Markdown chunking that splits documents along their real sections.
- Implemented cosine-similarity retrieval on PostgreSQL with pgvector, so relevant passages are found by meaning.
- Added a guardrail that returns "I don't know" without calling the LLM when no chunk passes the threshold, which avoids guesses and unnecessary token costs.
- Calibrated the MinScore threshold per embedding model with in-scope and out-of-scope test questions.
- Built grounded prompts with numbered sources, so the Microsoft Agent Framework agent answers only from retrieved context and cites it.
- Enabled switching between local Ollama and Google Gemini with one configuration setting via Microsoft.Extensions.AI.
Result:
- Built a working RAG application that answers from private documents and cites its sources.
- Kept documents on local hardware in local mode, so they never leave it.
- Returned "I don't know" instead of a guess, which keeps answers trustworthy and token costs low.
- Made the choice of LLM provider a configuration setting, so the application is not tied to one vendor.
- Showed in practice how private AI search can work with real control of data and costs. I build client solutions on the same principles.
Technologies Used:
.NET 10, C# 14, Microsoft Agent Framework, Microsoft.Extensions.AI, Ollama, Google Gemini, PostgreSQL, pgvector, RAG, Vector Search, Local LLMs, Prompt Engineering, Markdown Chunking
Source Code:
Other references

How AI workflows, custom AI skills and prompt engineering sped up software delivery at Helmsauer Gruppe. It also includes a concept for a RAG application.

Five years of building automation software for a smart home platform with about 50,000 active users. It shows my long experience in workflow automation, from devices to voice control.

A real-time dashboard for precision machinery, built to handle high-frequency data streams and diagnostic alerts. It shows my experience in industrial automation and live monitoring.