Experimental Laboratory

01System index

What is on the bench

Seven rows, six of them implemented, and the column that matters isinference. Whether a model runs on this machine or on someone else’s decides its latency, its cost, its privacy and what happens to it when the network is gone. Most of this bench is local; the part that is not says so, and so does the row with nothing behind it.

02Local intelligence

Inference that never leaves

Two projects run a language model through Ollama on the same machine as the application. No request goes to a hosted API, so there is no key, no per-token cost and no rate limit — and the model is only as good as what fits on the hardware, which is the trade the whole arrangement makes.

The second one is the more interesting decision. NepalSathi asks forqwen2.5:0.5b — half a billion parameters — at temperature 0.3 with an 80-token ceiling, and if Ollama is not running it answers from rules instead. There is no state in which the feature is simply broken. Both halves of that are deliberate and neither is a compromise; see experiment 005.

Jarvis — the full project record

03Computer vision

From a frame to a decision

Every vision system here is the same five steps, and the step that varies is the fourth. A frame is captured, converted, run through a pretrained graph, reduced to something small enough to reason about, and acted on. What differs between projects is what “small enough to reason about” means — a bounding box, twenty-one landmarks, or ten ranked labels.

04Voice

Two recognisers, one microphone

Listening for a wake word and transcribing a command are two problems wearing one name. The first has to be cheap enough to run forever; the second has to be accurate enough to be worth acting on. Jarvis gives each to a different model, and the small one keeps listening while the assistant is speaking so it can be interrupted.

05Gesture and interaction

Twenty-one landmarks, one number

This is the clearest case on the bench of AI, hardware and software being one problem. MediaPipe returns twenty-one hand landmarks per frame; the whole design decision is refusing to classify them. Steering is the x-distance between the wrist and the middle-finger knuckle — one subtraction, already continuous, already signed.

Those last two numbers — an exponential moving average at 0.3 and a threshold at ±0.02 — are the part with no tutorial behind it. They belong to one camera, one hand and one game, and they were found by playing it. A good input device is mostly a filter.

06Automation

A program that can type is a program that can destroy your work

There are no agents here and nothing autonomous. What exists is a recognised command mapped onto a keystroke — and a fence around it, because the failure mode of a voice-driven desktop controller is that it mishears one syllable and closes something you had not saved.

None of those limits are decorative, and none of them were in the first version. They are what the first version taught.

07Model provenance

Nothing here was trained

No model on this bench was trained or fine-tuned. Every one of them is published weights, pulled and applied. There is no dataset, no training run, no accuracy figure and no loss curve, because none of those things exist — and a laboratory page that implied otherwise would be the exact thing this archive is built not to do.

What that leaves is a real discipline, and it is the one this bench actually practises: applying a general model to a narrow problem well. The plant gate, the label map, the smoothing constant, the wake-word split — none of them touch a weight, and all of them decide whether the system works. Training is the next thing to learn, and it has not been started.

08Experiment records

The notebook

Filed in the order the work happened. Each record states the question, the apparatus and the result — and the result field is allowed to be empty. Most of these were never instrumented, so most of them say “not measured” rather than carrying a number that was never taken.

1. Experiment 001succeeded2026-07-23

   Aiming a general classifier at five crops

   Question

   A general-purpose image classifier can be made to identify five specific crops without any crop-specific training, if the mapping from its vocabulary to the target classes is done carefully enough.

   Apparatus

   • MobileNetV2
   • ImageNet weights
   • TensorFlow 2.15
   • Pillow
   • NumPy

   Result

   Works for the five target crops. No accuracy figure was measured — the project has no labelled test set, so there is no number to report.

   The setup

   agrovision/ai_crop.py loads MobileNetV2 with stock ImageNet weights. Nothing is trained, fine-tuned or frozen-and-retrained. The network has never seen a crop dataset. What the project adds is three layers of judgement on top of its output:

   1. The image is resized to 224×224 and classified, keeping the top ten ImageNet predictions rather than the first.
   2. A plant gate decides whether the photograph is of a plant at all: at least one of those ten labels must contain a botanical keyword — plant, leaf, corn, tomato, and fourteen others — at a probability of 0.10 or better. Fail it and the answer is Unknown.
   3. A label map translates ImageNet vocabulary into the five target crops. corn, corncob and ear all accumulate onto maize; mashed_potato onto potato. Probability is summed across every matching label, and the crops are ranked by that total.

   Why it works, and exactly how far

   It works because all five crops have close ImageNet neighbours. It would not extend to a sixth crop that ImageNet has never seen, and no amount of care in the label map would rescue that — the information is not in the network. That is the actual finding: the ceiling here is the vocabulary, not the code.

   The defect I would fix first

   The confidence figure the interface reports is the model’s probability for its top ImageNet label, not its confidence in the crop that was finally chosen. Those are different numbers, and showing the first while labelling it as the second overstates what the system knows. There is also a fallback that suggests rice, maize and wheat when nothing maps — a reasonable default that makes the interface look confident about an image it made no determination on.

   Both are the same mistake in two places: a number is displayed because one was available, not because it answered the question being asked.

   Source for experiment 001↗

2. Experiment 002inconclusive2026-08-09

   Attention as an input

   Question

   An assistant that can tell whether it is being attended to should answer differently when it is not — shorter, or not at all.

   Apparatus

   • MediaPipe Face Detection
   • MediaPipe Face Mesh
   • OpenCV

   Result

   Not measured. The state machine works and the style hint changes; whether either improves the interaction was never tested, so this experiment is recorded as inconclusive rather than as a success.

   What it computes

   vision.py runs two MediaPipe graphs on the same frame. Face Detection gives presence — is anyone there — with a confidence floor of 0.3 and a one-second timeout before the state flips to FACE LOST. Face Mesh gives a landmark set, from which the module derives a crude pitch estimate: the nose tip’s y-position relative to the eye line, normalised by the distance between the eye corners so it does not change when you lean closer to the camera.

   From those two it maintains a state — face present, face lost, looking down, distracted — and emits it as an event.

   What it does with it

   emotion_engine.py consumes that state. It is 599 bytes and its own docstring says “NO ML, NO threads, NO side effects”, which is the right way to describe it: a lookup from vision state to a style hint. DISTRACTED, LOOKING DOWN and FACE LOST return short, and the assistant answers more briefly.

   It is named the emotion engine and it does not detect emotion. I am leaving the name in this record because the file is honest about it in a way the filename is not, and that gap is worth seeing.

   Why this is filed as inconclusive

   The mechanism works. Presence is detected, the pitch estimate tracks, the state changes, the hint changes, the answers get shorter.

   Whether any of that makes the assistant better was never measured. There is no A/B, no timing, no record of whether shorter answers while looking away are actually more useful than full ones. The honest outcome for an experiment whose result was never observed is not “succeeded” — it is this.

   The pitch estimate is also a proxy and should be read as one. Nose-below-eyes is not gaze; it is head pose, and someone can look at a screen with their head down. Calling it attention is a simplification I would want to justify with data before I built anything else on top of it.

   Source for experiment 002↗

3. Experiment 003succeeded2026-08-09

   Hand tilt as a steering axis

   Question

   A racing game can be steered with a bare hand if the landmark stream is reduced to a single continuous axis rather than classified into discrete gestures.

   Apparatus

   • MediaPipe Hands
   • OpenCV
   • pynput
   • 1280×720 capture, 320×240 inference

   Result

   Playable. No frame-rate, latency or false-trigger rate was measured — the thresholds were set by playing the game, not by instrumentation.

   The reduction

   MediaPipe returns 21 landmarks per hand. The temptation is to classify them — train something, or write a decision tree over finger positions, and emit left or right. gesture_control.py does something much smaller.

   Steering is the x-distance between landmark 0 (the wrist) and landmark 9 (the middle-finger knuckle). One subtraction. Tilt the hand and that difference changes sign and magnitude; hold it level and it sits near zero. Twenty-one landmarks reduce to one float, and the float is already continuous, already signed, and already means what you want it to mean.

   Braking and acceleration are counted rather than measured: a fist is three or more fingertips below their knuckles, an open hand is three or more above. Counting is enough because those two states are far apart.

   Where the difficulty actually was

   Not in the detection. In the noise.

   A raw landmark stream jitters, and a jittering steering axis crossing a threshold produces a stream of key presses that makes the car undriveable. The fix is one line — an exponential moving average with SMOOTHING_FACTOR = 0.3, so each frame is 30% new reading and 70% history — and then the thresholds matter: ±0.02, which is deliberately tiny, with detection confidence at 0.7 and tracking at 0.5.

   Those four numbers are the project. They were found by playing the game, and they are the reason the file’s docstring says “ultra-sensitive”: at ±0.02 the system responds to a wrist movement you would not describe as a gesture.

   One more thing that turned out to matter: frames are captured at 1280×720 and downscaled to 320×240 before inference. The landmarks are normalised coordinates, so nothing downstream notices, and the pipeline gets its frame budget back.

   What the filter taught

   That a good input device is mostly a filter. The recognition was library work; the smoothing constant and the two thresholds were the engineering, and no tutorial has them in it, because they belong to this camera, this hand and this game.

   Source for experiment 003↗

4. Experiment 004succeeded2026-08-09

   Two speech recognisers instead of one

   Question

   A single speech recogniser cannot be both cheap enough to run continuously and accurate enough to transcribe a command, so the two jobs should be given to two different models.

   Apparatus

   • Vosk (vosk-model-en-us-0.22)
   • faster-whisper (small)
   • sounddevice
   • 16 kHz mono capture

   Result

   Both recognisers run on one machine and one microphone. Latency and word error rate were not measured — no figure is reported for either.

   The problem

   An always-listening assistant has two speech jobs that pull in opposite directions. The wake word must be recognised continuously, on every frame of audio, forever — so it has to be cheap. The command must be transcribed accurately, once, after the wake word — so it can afford to be expensive.

   One model cannot be both. A model small enough to run all day gets the command wrong; a model good enough for the command cannot run all day.

   What Jarvis does

   voice.py runs both, each on the job it suits.

   Vosk holds the microphone continuously through a KaldiRecognizer at 16 kHz mono. It listens for jarvis, and — this is the part I did not expect to need — it keeps listening while the assistant is speaking, for the cancel phrases in CANCEL_PHRASES. That is barge-in: the speaker loop calls engine.stop() mid-utterance when it hears one. An assistant you cannot interrupt is one you end up talking over.

   faster-whisper is loaded lazily, the first time a command is actually needed, and records a short clip bounded by three limits in config.py rather than by a fixed duration: a 3-second minimum, a 4-second maximum, and a stop after 450 ms of silence below an energy threshold of 400. The thresholds are there because a microphone is a physical device and this one needed tuning.

   What I learned

   That the interesting decision was not which recogniser to use. It was noticing that “listen for the wake word” and “transcribe the command” are two problems wearing one name, and that once they are separated each has an easy answer.

   The barge-in path is the piece I would keep in anything I build next. It costs one extra recogniser running during playback and it is the difference between a demo and something usable.

   Source for experiment 004↗

5. Experiment 005succeeded2026-08-21

   A 0.5B model, on purpose

   Question

   For a narrow question-answering job with a fixed context, a very small local model plus a deterministic fallback beats a large one that might not be running.

   Apparatus

   • Ollama
   • qwen2.5:0.5b
   • Flask

   Result

   Answers in the demo's question set are served either by the model or by the fallback, with no failure path in between. No latency, quality or fallback-rate figure was recorded.

   The decision

   app/services/ai_service.py in NepalSathi points at http://localhost:11434 and asks for qwen2.5:0.5b — half a billion parameters, roughly three orders of magnitude smaller than the models this kind of feature usually reaches for. The generation options are as tight as the model: temperature: 0.3, max_tokens: 80, top_p: 0.9, and a 20-second timeout on the HTTP call.

   That is not a compromise made to save money. It is what the job needs. The questions are about road, river and project status in Nepal; the facts are supplied to the model in a fixed context block; the answer is required to be one or two sentences. A larger model would be slower on a laptop during a hackathon demo and would not know anything the context block did not already tell it.

   The part that matters more than the model

   If Ollama is not running, the request throws and _generate_fallback answers instead — from rules, not from a model. The service is constructed so that there is no state in which the feature is simply broken: either the model answers, or the deterministic path does.

   That is the same shape as the fallback in this website’s own GitHub integration, and I did not notice the parallel until writing this page. Both say the same thing: a dependency you do not control is allowed to be absent, and the system has to have already decided what it does then.

   What I would check next

   Nothing here is measured. I do not know how often the fallback fires, how much faster 0.5B actually is on this hardware, or how the answers compare against a 7B model on the same context. Those are three cheap experiments and none of them has been run — which is why the result field above says so rather than estimating.

   Source for experiment 005↗

What the bench is for

Nothing on this page is a model I made. All of it is a system I assembled, ran on my own hardware, took apart when it misbehaved, and can still explain a year later. That is the skill I have actually been building, and it is a different one from prompting well.

The measurements are missing and I have said so in every place they are missing. Instrumenting these — latency on the local model, false triggers on the gesture threshold, accuracy on the crop classifier against a labelled set — is the obvious next work, and it is more useful than adding a seventh system to the index.