Lesson 4.4 — Chat with Your Own Docs (Mini RAG)

Introduction to AI · Module 4: Local LLMs · About 90 min · Windows PC, PowerShell 5, Python, Ollama 0.40.11

Goal: ask questions about your own documents and get answers that name the file they came from, all on your PC.

RAG = Retrieval-Augmented Generation: (1) cut your docs into chunks, (2) turn each chunk into numbers (an "embedding"), (3) for each question, find the closest chunks, (4) give only those to the model to answer from.

Before you start

Python works (Lesson 4.2)
Pick 10–20 of your own text docs (.txt or .md). Strata minutes, notes and how-tos are ideal. For a PDF, open it and copy the text into a .txt
Write 5 questions you already know the answers to:

Step 1 · Get an embedding model (⏱ 0:00–0:05)

ollama pull nomic-embed-text
ollama list

You should see nomic-embed-text:latest, about 270 MB.

Embedding model pulled

Step 2 · Collect your docs (⏱ 0:05–0:15)

mkdir C:\llm-test\docs -Force
explorer C:\llm-test\docs

Copy your 10–20 files into that folder, then check:

(dir C:\llm-test\docs -File).Count

10–20 files in docs

Step 3 · Build the indexer and the asker (⏱ 0:15–0:35)

Paste the whole block. It writes rag.py (about 40 lines, standard Python only):

$code = @'
import json, sys, os, math, urllib.request
OLL = "http://localhost:11434"; CHAT = "qwen3.8:27b"; EMB = "nomic-embed-text"
DOCS = r"C:\llm-test\docs"; INDEX = r"C:\llm-test\index.json"
def post(path, body):
    r = urllib.request.Request(OLL + path, json.dumps(body).encode(), {"Content-Type": "application/json"})
    return json.load(urllib.request.urlopen(r))
def embed(texts): return post("/api/embed", {"model": EMB, "input": texts})["embeddings"]
def chunks(text, size=800, overlap=150):
    i = 0
    while i < len(text):
        yield text[i:i+size]; i += size - overlap
def build():
    items = []
    for f in os.listdir(DOCS):
        t = open(os.path.join(DOCS, f), encoding="utf-8-sig", errors="ignore").read()
        cs = list(chunks(t))
        for c, v in zip(cs, embed(cs)): items.append({"file": f, "text": c, "vec": v})
        print("indexed", f, len(cs), "chunks")
    json.dump(items, open(INDEX, "w")); print("DONE:", len(items), "chunks")
def cos(a, b):
    return sum(x*y for x, y in zip(a, b)) / (math.sqrt(sum(x*x for x in a)) * math.sqrt(sum(y*y for y in b)))
def ask(q, k=4):
    items = json.load(open(INDEX)); qv = embed([q])[0]
    top = sorted(items, key=lambda it: -cos(qv, it["vec"]))[:k]
    ctx = "\n\n".join(f"[{t['file']}]\n{t['text']}" for t in top)
    sysmsg = ("Answer ONLY from the sources below. After the answer write 'Source:' and the file name(s) in brackets. "
              "If the answer is not in the sources, say 'Not found in your docs.'")
    r = post("/api/chat", {"model": CHAT, "stream": False, "messages": [
        {"role": "system", "content": sysmsg}, {"role": "user", "content": f"Sources:\n{ctx}\n\nQuestion: {q}"}]})
    print(r["message"]["content"])
if __name__ == "__main__":
    build() if sys.argv[1] == "build" else ask(" ".join(sys.argv[2:]))
'@
[IO.File]::WriteAllText("C:\llm-test\rag.py", $code)
cd C:\llm-test
python rag.py build

You should see one "indexed ..." line per file, then DONE: N chunks.

Index built

Step 4 · Ask your 5 questions (⏱ 0:35–1:05)

python rag.py ask "When is the next strata AGM?"

(Use your own question.) You should see an answer, then Source: [filename].

#	Question	Correct? (Y/N)	Right source file? (Y/N)
1
2
3
4
5

All 5 asked and scored

Step 5 · Test the "I don't know" (⏱ 1:05–1:15)

Ask something that's NOT in your docs:

python rag.py ask "What is my dog's name?"

You should see Not found in your docs. (or close to it).

Refused correctly

Step 6 · If you scored under 4/5 (⏱ 1:15–1:30)

Try one change at a time, then run python rag.py build again and re-ask:

Smaller chunks: change size=800 to size=500
More context: change k=4 to k=6

✅ Pass check

At least 4/5 answers correct, each naming the source file
The out-of-scope question got "Not found"
Told Skills Coach so 4.4 gets marked done

Troubleshooting

Problem	Fix
`model "nomic-embed-text" not found`	Run Step 1 again.
UnicodeDecodeError or odd text	Re-save the file as UTF-8 in Notepad. The script already skips bad characters.
Answers ignore your docs	Check that `index.json` exists and its date is today. Rebuild after adding files.
Very slow build	Normal for many files: embedding runs once per chunk. 20 docs should take under 2 min.
Out of memory	Close other apps. The embedding model plus Qwen fit in 24GB together.

Stuck? Ask Skills Coach. Send a screenshot or paste the error plus the step number. Don't stay stuck on one step for more than 10 minutes.

Finish this lesson

Back to all modules and lessons Next lesson → 4.5

← Previous (4.3)