Code Refactoring
Evolving a web prototype into an Apple Silicon desktop app using Tauri and Rust. Features a unique hybrid CV engine combining TensorFlow.js and native Apple Vision.
Architectural Evolution: From Web Prototype to an Apple Silicon Powerhouse
The project initially began as a modern web app. We built the foundation using a Vite + React + TypeScript stack, utilizing P5.js for rich, interactive visual rendering within the React ecosystem. For the core feature ‘Computer Vision’ we relied on TensorFlow.js. Running entirely locally in the browser, the app used ML models to handle real-time face, pose, hand, and object recognition. This web-centric architecture was fantastic for rapid prototyping and cross-platform accessibility.
The Turning Point: The Need for Native Power
To achieve the seamless, high-framerate experience, it was time to rethink the architecture. We made the bold decision to pivot from a generic web approach to a highly optimized, native desktop experience specifically tailored for modern Mac hardware.
The Refactoring Process: Building for Apple Silicon
The transformation involved a complete architectural overhaul, separating our concerns into three highly optimised layers:
1. The Frontend: Keeping the Best of the Web
We retained the Vite, React, and TypeScript codebase, but stripped it of its heavy computational duties. It was repurposed to act purely as the presentation layer—handling the UI, state, and P5.js canvas visuals—resulting in an incredibly snappy frontend.
2. The Backend: Transitioning to Rust and Tauri
To turn this web frontend into a lightweight desktop application, we wrapped it in the Tauri framework. Unlike memory-heavy alternatives like Electron, Tauri allowed us to build a robust, standalone desktop app with a tiny footprint. Beneath the React frontend, we implemented a brand-new Rust-based backend. Rust now handles the heavy lifting, system-level operations, and data routing, providing unparalleled performance and memory safety.
3. A Hybrid Computer Vision Engine: TensorFlow + Native Apple Vision
Instead of entirely replacing our original ML stack, we engineered a best-of-both-worlds hybrid pipeline. We retained TensorFlow.js and p5.js within our web layer to efficiently handle the highly intricate tasks of face and hand recognition. Specifically, for facial tracking, we leveraged p5.js to capture 468 tracking points, providing a significantly higher resolution and more detailed facial representation than the native Apple Vision framework, which is limited to 76 landmarks. However, for full-body pose detection, we offloaded the heavy lifting to the native macOS layer. By bridging our Rust backend with Swift, we integrated Apple’s powerful Vision framework. This allows the pose recognition workload to tap directly into the Mac's Neural Engine. The result is a balanced system where TensorFlow.js and native Apple Vision work in tandem to deliver a zero-latency, high-framerate experience.