Capture individual windows/surfaces?

[tmccombs]

The screencopy protocol doesn't seem to have a great way to capture individual windows (toplevel surfaces?). It might be able to build something to do this using sway ipc to get the region for a window and tracking its movement (if capturing video and not a single frame), but such a solution would be sway-specific. Would it be feasible to add a way for a client to specify a toplevel surface to capture rather than an output or region?

The biggest obstacle I see is how the client would choose which window to capture. The client may want the user to select the desired window by clicking on the window directly (something like slurp, but get a handle to the surface instead of coordinates) or give the user a menu of windows to select from. The menu might be possible using the foreign_toplevel protocol, but I'm not sure how to accomplish something like clicking on the window directly.

See also: emersion/xdg-desktop-portal-wlr#12 (comment)

[emersion]

The initial plan was to somehow let screencopy and export-dmabuf to accept a foreign toplevel handle.

[dos1]

The initial plan was to somehow let screencopy and export-dmabuf to accept a foreign toplevel handle.

We have implemented that in phoc/phosh as part of the initial window thumbnail implementation. There's a request in the private protocol that takes zwlr_foreign_toplevel_handle_v1 and returns zwlr_screencopy_frame_v1.

Some insights:

• export-dmabuf isn't really viable there - you need to composite the subsurfaces
• since you need to composite the surface separately anyway, it's beneficial to let the compositor scale the capture down for the thumbnail use case, since doing that client-side would be just a waste of resources
• there's no way to specify that a protocol depends on another protocol (AFAIK), so one has to rely on the order of inclusion of generated headers, which somewhat works for C but I guess it may be a major PITA for other bindings
• screencopy's copy_with_damage doesn't gracefully handle changing surface dimensions
• xdg-shell defines a parent-child relationship between surfaces, which may be wanted to take into account in such captures; although the client could check the relationship by themselves using wlr-foreign-toplevel-management, there's no way to retrieve relative positions between such surfaces so they couldn't be accurately drawn on the capture

[dos1]

One more thing to note, although that one is more related to implementation than the protocol: in order to prevent race conditions, the capture/composition part needs to happen already when buffer event is being sent. Doing that on copy will cause troubles if the surface has been resized meanwhile. This means that the compositor won't be able to render straight to dmabuf provided by the client via screencopy protocol but will have to copy it from an internal buffer anyway (which is where the approach from export-dmabuf could play some role)

[emersion]

export-dmabuf isn't really viable there - you need to composite the subsurfaces

Right. If ever supported, it would be an optional optimization, which would only work in very specific cases. Probably not worth the trouble TBH.

since you need to composite the surface separately anyway, it's beneficial to let the compositor scale the capture down for the thumbnail use case, since doing that client-side would be just a waste of resources

Right.

I wonder if a protocol to produce virtual wl_buffers that mirror the contents of a toplevel would be better for the window switcher use-case. Clients could use a subsurface to display these. They would completely avoid any copy/lag, but wouldn't allow for arbitrary effects (3D stuff wouldn't work).

there's no way to specify that a protocol depends on another protocol (AFAIK), so one has to rely on the order of inclusion of generated headers, which somewhat works for C but I guess it may be a major PITA for other bindings

Yeah. That's a more general issue with Wayland protocols, maybe worth opening an issue in the Wayland bug tracker?

screencopy's copy_with_damage doesn't gracefully handle changing surface dimensions

Yeah, at least some of it is tracked in #82 I believe?

xdg-shell defines a parent-child relationship between surfaces, which may be wanted to take into account in such captures; although the client could check the relationship by themselves using wlr-foreign-toplevel-management, there's no way to retrieve relative positions between such surfaces so they couldn't be accurately drawn on the capture

Maybe it would be better if the client could specify that it wants the children toplevels blended on the buffer as well?

in order to prevent race conditions, the capture/composition part needs to happen already when buffer event is being sent

Hrm, that's pretty annoying. Can't we just handle the resizes more gracefully in screencopy, and ask the client to provide a new buffer with the correct size?

[emersion]

wlr-protocols has migrated to gitlab.freedesktop.org. This issue has been moved to:

https://gitlab.freedesktop.org/wlroots/wlr-protocols/-/issues/93