Architecture
How an answer is produced, and what stops a bad one.
The technical view: the path one question takes through the running product, the controls around the paid model, the ontology the model is held to, and the on-site design it grows into.
In the product today
One question, six steps
- 1Stills
Ten stills are cut from the clip in the browser. Faces are masked on the device before anything is sent; the video itself never leaves.
- 2Guard
The server accepts only same-origin requests, ten JPEG stills of one clip under size limits, and sample stills byte-identical to the committed ones.
- 3Instant path
A recorded answer, or the clip's event record, answers without any model call.
- 4Live model
Otherwise one vision-model call, inside a site-wide daily budget, routed to zero-retention endpoints, with a hard timeout and bounded output.
- 5Gate
Deterministic checks decide the verdict: cleared, held, not actionable, abstained or unverified.
- 6Question log
Every answer shown is appended to a hash-chained log with its source and verdict.
Vision model on this deployment: openai/gpt-4o, replaceable per deployment without code changes. Current prompts: answers ask-v3-2026-09-26, event records extract-v1-2026-09-26. Every recording states the prompt that produced it; the answers on the Evidence tab came from the earlier ask-v2.
Controls
What surrounds the paid model
A public page in front of a vision model is a cost and privacy surface. These are the controls, and which of them run today.
- Spend ceilingEvery live call is counted against a durable daily budget per visitor, per product and site-wide; when it is spent, the product answers from its recorded evidence and says so.Running
- Request guardSame-origin only; at most ten JPEG stills of one clip, each under 300 KB and 1,280 px; questions of 300 characters or fewer.Running
- Sample integrityStills claimed to come from a sample clip must match its committed SHA-256 hashes, so labels and duration are the server's, never the client's.Running
- Data routingModel calls are routed to zero-data-retention endpoints first, and never to endpoints that train on or collect the data.Running
- Error hygieneProvider errors are logged on the server only; the page receives a fixed, plain message.Running
- Face maskingAn on-device face detector masks faces before any upload still is sent; if it cannot load, nothing is sent without the visitor's consent.Running
- Prompt scopeThe model is told to ignore instructions inside the question or the frames; answers render as text, never as markup.Running
- Edge processingRaw video stays on site; only masked events and pointers leave.Design
- Role-based access and SSOQuestions and playback scoped by role; works-council viewers see aggregates only.Design
- Anchored audit logThe question log's chain head anchored in write-once storage, so recomputing hashes is detectable.Design
Event ontology
What counts, and what must not
Nine typed events, each with an observable cue and a guard against the confusion vision models make most. The same definitions go into the model's instructions and into the event record the gate checks.
Hard-hat compliance
SafetyHighCounts: A person in an active work area without a rigid-shell protective helmet. A hard-hat is a rigid shell with a brim, usually a solid bright colour.
Does not count: A baseball cap, beanie, hood, or bare head styled like headwear is NOT a hard-hat - the single most common VLM confusion. If the shell/brim cannot be resolved, report 'unsure', never 'yes'.
Hi-vis compliance
SafetyMediumCounts: A person in a vehicle/equipment area without a high-visibility garment (fluorescent yellow/orange vest or jacket, usually with reflective strips).
Does not count: A plain bright-coloured shirt without reflective strips is a weak cue - report 'unsure' rather than counting it as hi-vis.
LOTO breach (lockout/tagout)
SafetyCriticalCounts: A person with body parts inside a machine's guarded envelope or service area while no lockout device (padlock/hasp/tag on the isolation point) is visible.
Does not count: Standing NEAR a stopped machine is not a breach. Energization state usually cannot be read from a still frame - if no isolation point or lock is visible, say the frames cannot support a LOTO judgement rather than inventing one.
Forklift-pedestrian proximity
SafetyCriticalCounts: A powered industrial truck (forklift, reach truck) moving within roughly one vehicle-length of a person on foot in a shared aisle, with no physical separation.
Does not count: A parked or unmanned forklift is NOT a proximity event. A hand pallet jack is not a powered truck. If no forklift is visible in any frame, the grounded answer is 'none' - not a guess.
Restricted-zone entry
SafetyHighCounts: A person crossing visible floor markings, barriers, or gates into a marked exclusion or machine-guarded area.
Does not count: Only count when the marking/barrier itself is visible in frame. Faded or ambiguous markings -> 'unsure'. Do not infer zones that are not visible.
Spill / walkway obstruction
SafetyMediumCounts: Liquid, debris, or stored material blocking a pedestrian walkway or emergency route.
Does not count: Normal work-in-progress material at a workstation is not an obstruction; the cue is material IN a walkway/aisle.
Micro-stoppage
OperationsMediumCounts: A station that is inactive in some frames while adjacent stations remain active - visible as an operator waiting, a machine stopped mid-cycle, or material starved/blocked. Requires evidence across at least TWO frames.
Does not count: A single still frame cannot prove a stoppage (it may be mid-cycle dwell). One-frame evidence -> 'unsure, needs temporal confirmation'.
Idle station / line
OperationsMediumCounts: A workstation or machine unmanned or inactive across consecutive frames while the area is otherwise in an active shift.
Does not count: An empty area with no evidence of an active shift is just an empty area - do not label it idle time.
Cycle complete
OperationsInfoCounts: A unit of work visibly completed - a piece leaves a station, a weld/stitch/assembly step finishes, a package is sealed.
Does not count: Motion alone is not a completed cycle; the cue is a unit visibly finishing or leaving the station.
Production design
Extract events once on site, query them many times
Not running in this web product. The design a plant deployment grows into: perception and masking at the edge, so only events and pointers leave the site.
- 01On site
Ingest
Pull camera streams over RTSP/ONVIF into an on-site edge node.
- 02On site
Mask and perceive
Blur faces and bodies, then turn video into typed, timestamped events under the ontology.
- 03On site
Event record
Store events with pointers to encrypted local clips. Read once; query many times.
- 04Query time
Classify the question
Map a question to typed events, zones, shifts and time ranges.
- 05Query time
Answer
Answer from the event record; call a vision model only when the record cannot answer.
- 06Query time
Gate and log
Grade every answer, hold what fails, log every question and view.
The gate and the question log wrap every answer, whether it came from the event record or a model call.
Requirements
Must-haves, and where each one stands
Events extracted continuously from RTSP/ONVIF streams without footage leaving the site.
Every ontology event judged observed, not observed or cannot judge, each tied to the stills that support it.
Answers cite stills; 'cannot tell' when the footage cannot support an answer; never an invented event.
Every cited timestamp plays the footage at that moment.
Faces masked before perception; no biometric identity stored. Running in the browser; edge masking is design-stage.
Every question logged in a hash chain. Role-based access is design-stage.
Any answer that contradicts a label the question depends on is held, whatever its score.
Pilot targets (hypotheses to test, not results)
- Investigation time down 80%: from re-watching footage to reviewing a cited, gated answer.
- Zero actions taken on held answers: every held answer routed to a person before anything happens.
- Per-event precision published per site: measured on the site's own labelled clips before go-live.
- Works-council and DPO sign-off with no raw-video egress: on every EU deployment.
Deliberately out of scope
- Safety-rated machine control or e-stop: a different reliability class and liability regime.
- Individual worker scoring: off-mission, and high-risk under the EU AI Act.
- Selling cameras or hardware: margins and sales cycles both suffer.
- Line-speed defect inspection: a crowded market with a different latency profile.
Build or buy
Buy perception, build judgement
- Perception modelsBuyImproving fast and commoditising; keep the model layer swappable.
- Event ontology and guardsBuildThe domain judgement that decides what counts.
- Evidence gate and evaluation setBuildThe product: what makes an answer defensible.
- Edge runtime and maskingBuild on standard partsRegulatory necessity; commodity detectors, owned orchestration.
- Event store and queryBuy infra, build glueStandard storage; the question-to-event mapping is ours.
- EHS and MES integrationsBuildWhere answers become corrective actions; drives retention.
Status
What runs today, and what is design
Running in the product today · 6
Live vision-model answers
Custom questions and uploads get one fresh call to the configured vision model inside a daily budget. The answering model, latency, tokens and billed cost come back with every live answer.
Recorded answers
Suggested questions on the sample clips replay unedited answers recorded from the live model: instant, free, and labelled with the model and date.
Masking before sending
Upload stills are masked by an on-device face detector before they leave the browser; the sample stills were masked once, offline, before any answer was recorded.
Read-once event record
An uploaded clip can be read once into a typed record of people, PPE, objects and every ontology event. Later questions about typed events are answered from it with no model call.
Evidence gate
Deterministic checks return one of five verdicts; a single blocking failure holds an answer whatever its score.
Question log
Every answer shown is logged in the visitor's browser in a SHA-256 hash chain. It detects an edited entry; it cannot stop someone who recomputes every hash, which is why production anchors the chain server-side.
Production design, not yet built · 4
Edge appliance and camera ingestion
Production keeps raw video on site. This web product sends masked stills to a third-party model, and says so beside every answer.
Role-based access and SSO
Scoping questions and playback by role, with aggregate-only views for works-council members.
Server-anchored audit chain
The question log's head written to write-once storage, so recomputing the chain is detectable.
Multi-site rollup and EHS integration
Cross-plant questions, and held answers routed into the EHS system's corrective-action workflow.
Sample clips are stock footage, labelled as such, never customer or plant data. Their human labels are published on the Evidence tab.