DEUS ex MACHINA

Project Overview: Deus Ex Machina
DEUS ex MACHINA is an interactive, camera-driven generative art desktop application built for macOS. It bridges generative computation and physical human presence by fusing Conway’s Game of Life cellular automata with real-time computer vision and pose tracking.
The application translates webcam input, hand gestures, facial landmarks, and body poses into live perturbations and cell injections within an infinite cellular automaton grid, styled with a high-contrast monochrome aesthetic.
System Architecture & Tech Stack
Layer │ Technologies │ Role / Description
─────────────────────────────────────
Desktop Shell:
│ Tauri 2, Rust │ Native window management, low-level OS bridging, and high-performance IPC.
Frontend Framework:
│ React 19, TypeScript, Vite 8 │ UI layout, state orchestration, settings modals, and HUD diagnostics.
Styling:
│ Tailwind CSS 4 │ Minimalist dark-mode HUD and control interfaces.
Generative Graphics:
│ p5.js │ 2D simulation canvas, rendering pipeline, and particle/effect drawing.
Native Apple Vision:
│ Swift, Apple Vision, CoreVideo │ Apple Neural Engine (ANE) hardware-accelerated body-pose estimation.
Web ML Pipeline:
│ ml5.js, TensorFlow.js │ Hand pose, face mesh, and COCO-SSD object detection.
2. Key Modules & Implementation Details
_ A. Cellular Automata Engine (Grid)
• Spatial Hashing: Employs integer hash-pairing (hashPair) to store live cells in a sparse Set<number>, bypassing fixed 2D array memory limitations.
• Simulation Throttling & Dirty Flags: While the p5 canvas renders smoothly up to 60 FPS, the Game of Life evaluation is throttled (default 10 Hz). Rendering uses an offscreen graphics buffer that repaints only when the simulation state advances or panning shifts, minimizing CPU/GPU overhead.
• Pattern Presets & Hashlife: Supports RLE (Run-Length Encoded) pattern loading from a library of classic patterns (examples.json), alongside an experimental Hashlife algorithm.
_ B. Dual-Pipeline Computer Vision
1. macOS Native ANE Bridge (VisionBridge.swift):
• Captures BGRA frames from the webcam and passes them directly as raw Uint8Array buffers via Tauri IPC (detect_vision_frame).
• Rust invokes the Swift C-ABI function vision_bridge_detect_bgra, utilizing Apple's VNDetectHumanBodyPoseRequest to process body landmarks on the Apple Neural Engine / GPU with near-zero latency.
2. Client-Side Models (UnifiedVisionWrapper.tsx):
• Handles MediaPipe HandPose (21 landmarks per hand) and MediaPipe FaceMesh (468 landmarks) for fine-grained gesture control.
• Optional COCO-SSD model for general object detection.
_ C. 100% Offline Machine Learning Architecture
As outlined in offline-ml-plan.md, the app requires zero network access at runtime:
• All model definitions and binary weight shards (movenet/, handpose/, facemesh/, cocossd/) are vendored locally under public/models/.
• A standalone script (download-models.mjs) automates updates or re-vendoring of these weights.
_ D. Physical & Gesture Interaction Mechanics
• Hand Gesture Tracking & Pinching: Smooths landmark positions across frames (handTracker) to detect pinches between index finger and thumb.
• Cell Injections: Gestures trigger visual effects (e.g. triangle flashes) and seed clusters of living cells into the automaton grid.
• Spring-Damper Pan & Throw Physics: Panning the grid features an underdamped elastic spring-damper model (ζ = 0.65) during drag, transitioning into an inertia-based throw and exponential velocity friction decay upon release.
_ E. User Interface & HUD
• Startup Terminal Loader: Styled diagnostic console displaying step-by-step subsystem initialization and real-time telemetry (FPS, active cell count, detected entities, population density, interaction rate).
• Settings Modal : Floating configuration drawer allowing users to switch cell sizes (2, 4, 6, 8 px), select pattern seeds, toggle object detection, and modify render frame rates.
• Native Screenshot Utility (screenshot.rs): Saves base64-encoded canvas snapshots directly to the user's ~/Downloads directory via Rust.