AI without the noise

Provider-Independent RAG Application for Local and Cloud LLMs

A RAG application that answers questions from private documents with either a local or a cloud LLM. It keeps data on local hardware when needed and cites its sources.

Situation:

Many companies want answers from their private documents, but sending them to a cloud LLM is not always an option. This ongoing personal AI project, started in 2026, tests a provider-independent approach.

Task:

As architect and developer, I designed a RAG application that answers questions from private documents with either a local or a cloud LLM. The goal was grounded answers, with data staying in-house when needed.

Action:

  • Designed the indexing flow as single-responsibility classes, with structure-aware Markdown chunking that splits documents along their real sections.
  • Implemented cosine-similarity retrieval on PostgreSQL with pgvector, so relevant passages are found by meaning.
  • Added a guardrail that returns "I don't know" without calling the LLM when no chunk passes the threshold, which avoids guesses and unnecessary token costs.
  • Calibrated the MinScore threshold per embedding model with in-scope and out-of-scope test questions.
  • Built grounded prompts with numbered sources, so the Microsoft Agent Framework agent answers only from retrieved context and cites it.
  • Enabled switching between local Ollama and Google Gemini with one configuration setting via Microsoft.Extensions.AI.

Result:

  • Built a working RAG application that answers from private documents and cites its sources.
  • Kept documents on local hardware in local mode, so they never leave it.
  • Returned "I don't know" instead of a guess, which keeps answers trustworthy and token costs low.
  • Made the choice of LLM provider a configuration setting, so the application is not tied to one vendor.
  • Showed in practice how private AI search can work with real control of data and costs. I build client solutions on the same principles.

Technologies Used:

.NET 10, C# 14, Microsoft Agent Framework, Microsoft.Extensions.AI, Ollama, Google Gemini, PostgreSQL, pgvector, RAG, Vector Search, Local LLMs, Prompt Engineering, Markdown Chunking

Source Code:

View the project on GitHub

Other references

AI-Assisted Development and Workflow Automation
How AI workflows, custom AI skills and prompt engineering sped up software delivery at Helmsauer Gruppe. It also includes a concept for a RAG application.
Smart Home Automation Platform Grown to 50,000+ Users
Five years of building automation software for a smart home platform with about 50,000 active users. It shows my long experience in workflow automation, from devices to voice control.
Real-Time Industrial Dashboard for Precision Machinery
A real-time dashboard for precision machinery, built to handle high-frequency data streams and diagnostic alerts. It shows my experience in industrial automation and live monitoring.