Skip to content

monkeynut.org

Using wgpu with Electron on macOS

Here is a brief guide to adding a wgpu-rendered underlay to an Electron window on macOS. Doing this isn't particularly difficult once you know how, but I couldn't find this information anywhere else. So here's the guide that I wish had existed before I went down this path!

The overall approach is to let Electron "run the show", invoking the Rust logic as a library, wrapped using napi-rs. In Rust we add a subview, positioned underneath Electron's NSView. From this subview we create a wgpu surface which we then pass to our own rendering thread.

One key challenge is to support low-latency resizes of the wgpu surface so that the web UI and rendered content move in concert. (If we didn't care about latency we could just write a JS resize handler that calls out to Rust on each resize operation). We solve this problem by (i) setting reconfigurable constraints on the subview so that macOS resizes it whenever the main window is resized, and (ii) configuring the subview to directly send resize messages to our rendering thread which can then reconfigure the surface with the new dimensions. (Surface reconfiguration must occur between frame renders, so it's easiest to do this in the rendering thread.)

Caveat emptor: the code samples below are directly taken from an application that I'm working on, with some light editing for generality. There may be some minor errors (e.g. missing imports, or mistranscribed names).

1. Get the root NSView

In our Electron main process, we create the browser window and pass the results of getNativeWindowHandle to our initialization function:

function createWindow() {
  const mainWindow = new BrowserWindow({ /* ... */ });
  myrustlib.init(
    mainWindow.getNativeWindowHandle(),
    // ...
  );

  // Set transparent background so that underlay is visible
  mainWindow.setBackgroundColor('#00000000');
}

In our Rust initialization function, we convert the native window handle Buffer to the root NSView (also know as the "content view"):

use napi::bindgen_prelude::Buffer;
use napi_derive::napi;
use objc2::rc::Retained;
use objc2_app_kit::NSView;

#[napi]
pub fn init(content_view: Buffer) -> Result<(), napi::Error> {
    init_inner(content_view).map_err(|e| napi::Error::from_reason(e.to_string()))
}

fn init_inner(content_view: Buffer) -> Result<(), Box<dyn Error>> {
    let content_view = content_view.to_vec();
    let content_view = u64::from_le_bytes(content_view[..8].try_into()?);
    let content_view = content_view as *mut std::ffi::c_void;
    let content_view: Retained<NSView> = 
        unsafe { Retained::retain(content_view.cast()).unwrap() }
}

2. Define our subview

We now define a subclass of NSView that sends resize notifications over a channel, and otherwise behaves identically to a standard NSView. We'll send these notifications to our rendering thread, which can then reconfigure the surface dimensions.

use std::sync::mpsc;
use std::sync::OnceLock;

use objc2::define_class;
use objc2::msg_send;
use objc2::rc::Allocated;
use objc2::DefinedClass as _;
use objc2_foundation::NSSize;

#[derive(Debug)]
pub enum MyMessage {
    Resized {
        width: u32,
        height: u32,
    },
    // ...
}

define_class!(
    #[unsafe(super(NSView))]
    #[ivars = Ivars]
    #[name = "MyNSView"]
    pub struct MyNSView;

    impl MyNSView {
        #[unsafe(method_id(init))]
        fn _init(this: Allocated<Self>) -> Retained<Self> {
            let this = this.set_ivars(Ivars::default());
            unsafe { msg_send![super(this), init] }
        }

        #[unsafe(method(setFrameSize:))]
        fn set_frame_size(&self, size: NSSize) {
            unsafe { msg_send![super(self), setFrameSize: size] }
            self.ivars().tx.get().unwrap().send(MyMessage::Resized {
                width: size.width as u32,
                height: size.height as u32,
            })
            .unwrap();
        }
    }
);

#[derive(Default)]
pub struct Ivars {
    tx: OnceLock<mpsc::Sender<MyMessage>>,
}

impl MyNSView {
    pub fn init(this: Allocated<Self>, tx: mpsc::Sender<MyMessage>) -> Retained<Self> {
        let this: Retained<Self> = unsafe { msg_send![this, init] };
        this.ivars()
            .tx
            .set(tx)
            .unwrap();
        this
    }
}

3. Create our subview

Next we create the subview and add it to the content view:

use objc2_foundation::MainThreadMarker;

fn init_inner(content_view: Buffer) -> Result<(), Box<dyn Error>> {
    // ...
    let mtm = unsafe { MainThreadMarker::new_unchecked() };
    let my_view: Allocated<MyNSView> = mtm.alloc();
    let (tx, rx) = mpsc::channel::<MyMessage>();
    let my_view = MyNSView::init(my_view, tx.clone());
    
    unsafe {
        content_view.addSubview_positioned_relativeTo(
            &my_view,
            objc2_app_kit::NSWindowOrderingMode::Below,
            None,
        );
    }
}

4. Set constraints on our subview

Next we set constraints on our subview so that macOS resizes it as the content view resizes. You can also expose a Rust function (not shown) to alter the constants from JS.

use objc2_app_kit::NSLayoutAttribute;
use objc2_app_kit::NSLayoutConstraint;
use objc2_app_kit::NSLayoutPriorityDefaultLow;
use objc2_app_kit::NSLayoutRelation;

fn init_inner(content_view: Buffer) -> Result<(), Box<dyn Error>> {
    // ...
    unsafe {
        my_view.setTranslatesAutoresizingMaskIntoConstraints(false);
    }

    let top_constraint = unsafe {
        NSLayoutConstraint::constraintWithItem_attribute_relatedBy_toItem_attribute_multiplier_constant(
            &my_view,
            NSLayoutAttribute::Top,
            NSLayoutRelation::Equal,
            Some(&content_view),
            NSLayoutAttribute::Top,
            1.0,  // Multiplier
            0.0,  // Constant 
        )
    };
    unsafe { top_constraint.setPriority(NSLayoutPriorityDefaultLow) }
    unsafe { content_view.addConstraint(&top_constraint) }
    
    // ...
}

5. Create the wgpu surface

Next we create the wgpu surface from the subview. Note that we must create (and drop) this surface in the main thread.

fn init_inner(content_view: Buffer) -> Result<(), Box<dyn Error>> {
    // ...
    let instance = wgpu::Instance::default();
    let raw_display_handle = raw_window_handle::RawDisplayHandle::AppKit(
        raw_window_handle::AppKitDisplayHandle::new(),
    );
    let raw_window_handle =
        raw_window_handle::RawWindowHandle::AppKit(raw_window_handle::AppKitWindowHandle::new(
            std::ptr::NonNull::new(Retained::as_ptr(&video_view).cast_mut() as *mut _)
                .expect("overlay should be non-null"),
        ));
    let surface_target = wgpu::SurfaceTargetUnsafe::RawHandle {
        raw_display_handle,
        raw_window_handle,
    };
    let my_surface =
        unsafe { instance.create_surface_unsafe(surface_target)? };
}

6. Spawn the rendering thread

Finally we spawn our rendering thread, passing in the receive channel and surface. It's useful to store the constraints, surface and join handle in some context structure (I used a thread-local variable, not shown).

fn init_inner(content_view: Buffer) -> Result<(), Box<dyn Error>> {
    // ...
    let mut renderer = Renderer::new(rx, surface.clone());
    let join_handle = thread::Builder::new()
        .name("renderer".into())
        .spawn(move || renderer.run())
        .unwrap();
    // ...
}