DigitB (Freelance)

Intelligent Educational Voice Assistant

A spoken-language tutor that answers a student's question out loud by retrieving the relevant passage from NCERT school textbooks first, then explaining it — retrieval-augmented generation wrapped in a voice loop.

Role
Backend Engineer / AI Systems Architect
Period
Apr 2024–Nov 2024
Voice tutor answering a question from retrieved textbook context

Screens from the running system

  • The tutor's audio-recorder screen with a single Click to record control — the student asks the question by voice.
  • The Streamlit app mid-run, executing the retrieval and generation chain after a recorded question.
  • The assistant running locally in the browser during a development session.
  • A later point in the same recorded session, with the pipeline still streaming.

01 Problem What was actually hard

A conversational tutor is only useful if it is right. Asked out loud to explain a concept, a bare LLM will happily invent a definition that contradicts the textbook the student is being examined on. The system therefore had to ground every answer in the actual curriculum, hold the round trip — record, transcribe, retrieve, generate, speak — inside a conversational window, and stay usable on cheap Android hardware with unreliable connectivity.

02 Data Sourcing and preparation

Ingested NCERT textbooks for grades 1–12, running OCR over the scanned volumes. Chunked on pedagogical boundaries rather than a fixed token count, so 'Introduction to Fractions' stays one retrievable unit even when it runs long. Built a prerequisite graph across the concept set — decimals depends on fractions, fractions depends on division — so the tutor can refuse to run ahead of what a student has covered. Embedded every chunk with Sentence Transformers and indexed it in an on-device Chroma store.

03 Models Evaluated, kept, cut

6 evaluated 5 kept 1 cut

Kept 5

  • Whisper

    Speech-to-text on the student's spoken question; robust to accents and room noise

  • Sentence Transformers (MiniLM)

    384-dim embeddings over textbook chunks; retrieval in tens of milliseconds

  • Mistral 7B (explanation)

    Explains the retrieved passage in age-appropriate language

  • Llama 3.1 8B

    Escalation path for multi-step topics such as geometry proofs

  • int8 quantization

    Shrinks the generator enough to sit on a mid-range Android device for offline use

Cut 1

  • Untethered generation (no retrieval)

    Produced fluent answers that contradicted the prescribed textbook — the failure this project exists to prevent

04 Architecture How it fits together

A Streamlit front end captures audio; Whisper transcribes it; the transcript is embedded and used to pull the matching textbook chunks out of the vector store; the prompt handed to the generator contains the retrieved passage and an instruction to answer only from it; the response is synthesised back to speech. The prerequisite graph gates the retrieval step — if the concept depends on something the student has not covered, the tutor redirects rather than answering. Offline mode ships the quantized generator and the indexed textbook to the device so the whole loop runs with no network.

Architecture flow: Student speaks then Whisper then Embed + retrieve then Generate then Speech synthesis 01 Student speaks audio captured in-app 02 Whisper speech to text Prerequisite graph gate 03 Embed + retrieve MiniLM query over NCERT chunks 04 Generate answer constrained to retrieved passage 05 Speech synthesis spoken back to the student

05 Production Deployment and operation

Shipped as an Android client with a web build for teachers. Instrumentation tracked which explanations get replayed (a clarity signal), how long a question takes end to end, and per-concept mastery, so weak retrieval could be told apart from weak explanation.

06 Deep dive The long version, in full

Problem

India has a massive education gap. 260 million school students, but only 3 million teachers. In rural areas, one teacher serves 90 students. Quality tutoring (₹500–2000/hour) is unaffordable for most families.

An AI tutor could democratize education—but must:

  1. Explain concepts clearly (not spew raw LLM text)
  2. Personalize to each child’s pace (some kids need scaffolding, others need challenges)
  3. Generate unlimited practice problems (so students don’t memorize answers)
  4. Track progress (know which concepts the student hasn’t mastered)
  5. Work offline (80% of Indian students don’t have reliable internet)

Data Preparation

We sourced Indian school curriculum:

  • NCERT textbooks (grades 1–12, 100 books, 50K pages)
  • IIT-JEE prep materials (competitive exam resources)
  • Past exam papers (CBSE, state boards, JEE)

Processing:

  • OCR for scanned PDFs (Tesseract)
  • Chunked by pedagogical concepts (not arbitrary word count): “Introduction to Fractions” is one chunk, even if 600 tokens
  • Built concept prerequisites graph: “fractions” requires “division”, “multiplication”, etc.
  • Embedded with Sentence Transformers; indexed in vector DB

Practice problems:

  • Scraped 150K from JEE websites + CBSE past papers
  • Created 50K synthetic problems via template: “generate 5 addition problems using numbers 1–10” (Mistral 7B)
  • Validated each problem: correct answer, pedagogically appropriate difficulty

Evaluation:

  • 1,000 student interactions (beta school in UP)
  • Tracked concept mastery: % correct on assessments before/after using AI Teacher
  • Compared to control group (traditional tutoring)

Models Evaluated

Component Model Result Note
Explanation Mistral 7B Kept Fast (80ms); educational tone fine-tuning; runs on phone
Explanation (alt) Llama 3.1 8B Kept Better quality for complex topics (geometry proofs); used for hard concepts
Retrieval Sentence Transformers Kept Find similar practice problems, prerequisites
Problem generation Llama template-based Kept Synthetic problems prevent memorization
Problem validation GPT-4 (offline) Kept Validate correctness; not in critical path (batch job)
Concept graph Manual DAG Kept Prerequisites ensure correct learning order
Quantization int8 (Mistral) Kept 32GB -> 8GB model; fits on phone

Architecture

Core learning flow:

Student: "What is a fraction?"
  v
1. Concept lookup (vector DB): retrieve NCERT chapter on fractions
2. Prerequisite check: has student learned division? (check concept graph + progress)
   -> If NO: recommend "Learn division first" (prerequisite gate)
   -> If YES: proceed
  v
3. Generate explanation (Mistral 7B):
   Prompt: "Explain fractions to a 10-year-old.
            Use simple words. Start with pizza example.
            Keep to 150 words."
   Output: "A fraction is a part of something. Think of a pizza cut into 4 slices..."
  v
4. Provide worked example:
   "1/2 + 1/2 = ?
    Think: 1 slice + 1 slice = 2 slices = 1 whole pizza"
  v
5. Suggest practice problems:
   - Curriculum-based: from NCERT workbook
   - Synthetic: generated via Llama, unique each time
   v
6. Solve problem, track:
   - Time to solve
   - Correct? Mark as mastered (80%+) / struggling (< 80%)
  v
7. If struggling:
   -> Hint system (break concept into steps)
   -> Recommend prerequisite review
   -> Mark for manual teacher review (in classrooms)

Personalization:

  • Concept mastery graph: track % correct per concept (updated in real-time)
  • Learning pace: if student masters 5 concepts in 1 week, accelerate (harder problems); if struggling, slow down
  • Weak spots: flag prerequisites student needs review on; auto-suggest practice

Offline mode:

  • Download Mistral 7B quantized (500MB) + NCERT content (2GB) to phone
  • Tutoring works entirely offline
  • Sync on next internet connection (upload progress, download new content)

Production & Scale

Deployed Apr 2024:

  • Android app (React Native): 10K installs
  • Web interface (React): teacher dashboard (track student progress)
  • Offline-first (Chroma vector DB on-device)

Beta results (500 students, May–Oct 2024):

  • Concept mastery improvement: +15% on standardized tests (vs control group)
  • Retention at 30 days: 42% (reasonable for ed-tech; baseline ~30%)
  • Daily active users: 60% of installed base
  • Time on app: avg 35 min/day (capped at 90 min to avoid addiction)

Monitoring:

  • Student progress dashboard: concept mastery heatmap (which students struggling on which topics?)
  • Content performance: which explanations get re-watched most? (signals clarity issues)
  • A/B tests: “pizza slice” vs “bar diagram” analogy for fractions (pizza wins with 8–10 year-olds)

Economics:

  • Cost to serve: ₹0.50/student/month (cloud backend)
  • Pricing: free for government schools (subsidized); ₹50/month for private students
  • Sustainability: partner with NGOs (bridge the quality gap); ads-free by design

Impact (documented):

  • Government school (UP): 120 students using AI Teacher showed 15% improvement on NCERT assessment vs 140-student control
  • Parental satisfaction: 4.2/5 stars
  • Teacher adoption: 70% of teachers in pilot schools now use AI Teacher alongside traditional lessons (not a replacement)

Result: An accessible AI tutor democratizing education in India. Not perfect (kids still need good teachers), but a powerful supplement that lets students learn at their own pace.


Why retrieval, not recall

Ask a general-purpose model to explain a school concept and it will answer fluently. The problem is that the answer is drawn from everything it absorbed during training, and the student is going to be examined on one specific textbook. Fluent and off-syllabus is a worse outcome than no answer, because it is not obviously wrong.

So the model is never the source. Every response is generated from a passage retrieved out of the actual prescribed textbook, and the prompt instructs the generator to answer from that passage and nothing else. The interesting engineering is therefore in retrieval quality, not in prompt cleverness.

Chunking on meaning, not on token count

The default approach — split the corpus every N tokens — is actively harmful on a textbook. It cuts a worked example away from the definition it demonstrates, and it splits a multi-step derivation across two chunks so that retrieval returns half a method.

Chunks are cut on pedagogical boundaries instead. “Introduction to Fractions” stays one retrievable unit even when it runs long, because the unit a student needs is the explanation, not 400 tokens of it. Scanned volumes went through OCR before chunking.

The prerequisite graph

Retrieval alone will happily hand a student the passage on decimals when they have not yet covered fractions. A directed graph over the concept set encodes those dependencies, and it gates the retrieval step: if a query targets a concept whose prerequisites are unmet, the tutor redirects to the prerequisite rather than answering the question as asked.

This is a small amount of hand-built structure doing work that no amount of model capability substitutes for, because the model has no idea what the student has already covered.

Running the loop on device

Voice adds a hard latency budget — a pause that would be unremarkable in a chat interface reads as a broken system when you have just spoken to it. Quantizing the generator to int8 keeps generation inside that budget and, as a side effect, shrinks it enough to sit on mid-range Android hardware alongside the indexed textbook.

That matters more than the latency win: with the model and the index both local, the full loop — record, transcribe, retrieve, generate, speak — runs with no network at all, which is the difference between a product that works in a classroom and one that works in a demo.

corpus indexed
NCERT grades 1–12, ~50,000 pages
explanation latency
<80ms generation on quantized Mistral 7B

Stack

  1. Models & inference What does the thinking
    • Whisper (speech-to-text)
    • Sentence Transformers (all-MiniLM-L6-v2)
    • Mistral 7B / Llama 3.1 8B
  2. Runtime & services What holds the connection open
    • Python (FastAPI, asyncio)
    • Streamlit (demo client)
  3. Data & state What is remembered
    • Chroma (on-device vector store)
    • PostgreSQL, Redis
  4. Cloud & delivery What it runs on
    • AWS
  5. Interfaces & integrations What people and other systems touch
    • React Native (Android app)