Building a Fully Offline local AI Agent Stack with Ollama, LangChain, and Private Document Embedding Vectors

Keskeiset kohdat

1. Install Ollama. It manages local model downloads and serving with a simple API.
2. Choose models. Smaller models run on CPU; larger ones need GPU. Match model size to hardware.
3. Set up local embeddings. Use a local embedding model to vectorize documents without cloud APIs.
4. Store vectors locally. Qdrant or Chroma in local mode keeps embeddings on disk.
5. Build agent workflows. LangChain orchestrates tool use and reasoning with local models.
6. Restrict network access. Block outbound traffic to ensure no data leaves the machine.
7. Test offline. Disconnect from the network and verify all functionality works.
8. Monitor resource usage. Local models consume CPU, RAM, and GPU; track usage to avoid overload.

Architecture

Ollama serves models locally via an HTTP API. LangChain orchestrates agent workflows, calling Ollama for generation and local embedding models for vectorization. Qdrant stores vectors on disk. All components run on the same machine with no network access.

Production configuration

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b
ollama pull nomic-embed-text

ollama list
ollama run llama3.1:8b 'Hello'

iptables -A OUTPUT -p tcp --dport 443 -j DROP
iptables -A OUTPUT -p tcp --dport 80 -j DROP

from langchain_ollama import OllamaLLM, OllamaEmbeddings
from langchain_qdrant import QdrantVectorStore
from langchain.chains import RetrievalQA

llm = OllamaLLM(model='llama3.1:8b')
embeddings = OllamaEmbeddings(model='nomic-embed-text')
vectorstore = QdrantVectorStore.from_documents(docs, embeddings, path='./qdrant_db')
qa = RetrievalQA.from_chain_type(llm=llm, retriever=vectorstore.as_retriever())
answer = qa.invoke('What is the policy?')

from langchain.agents import create_react_agent, AgentExecutor
from langchain.tools import Tool

def search_docs(query):
    return vectorstore.similarity_search(query, k=3)

tools = [Tool(name='search', func=search_docs, description='Search private docs')]
agent = create_react_agent(llm, tools, prompt)
executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
result = executor.invoke({'input': 'Find the backup procedure'})

Edge cases from production

• Large models need significant RAM; use quantized versions for constrained hardware.
• Embedding speed depends on CPU; GPU acceleration helps but is not required.
• Offline mode means no model updates; plan for manual updates.
• Agent loops can be slow on local models; set iteration limits.

Usein kysytyt kysymykset