Is there a way to remove the backround from a webRTC Media stream using shaders and WebGL

Hello guys, I’m kind of going rogue with Flutter, trying to build something like this:

The only available library option would be using Flutter webRTC, while Firebase being its signalling service and the overall backend. Now I encountered a slight predicament :face_with_bags_under_eyes:

  • Background removal requires some heavy Webgl/ Fragment shader gymnastics. I also knew that the Opengl API is currently not available in Flutter. [The goal here is to process the frames and select the layer that has the person’s body/ face … ]
  • It seems to be impossible to process the video frames using that library [at least for now, although I have noticed in the plugin’s source code that someone is currently working on it ] :sleepy_face:
    So here is my question: could someone who has worked on a similar niche give me some advice that does not involve going fully native, because I plan on targeting all available platforms starting from desktop, and it will be too complicated for one lazy GUY to take over Rome and die trying.

P.S.: If I miraculously pulled this off, I will be releasing this app to the community Absolutely For Free :kissing_face_with_smiling_eyes: ,

1 Like