Good examples of Flutter and long running C/C++ isolates?

Help
,
1

Hello! I am here developing an an app which calculates a lot of things using a C++ library, then draws those using Flutter CustomPainter.

The problem is in this case, that of course waiting for the C++ library to calculate can make the UI stall. Solving this with isolates is not exactly a trivial problem, but I have not seen much example projects really using a long standing isolate.

Could somebody point me to some example project that does this kind operation ? I could go into more detail about our app, but mainly I would be interested in seeing if somebody knows here anything to take a good look at.

Thanks!

1 Like
2

I made this example project a while ago. Maybe it’s helpful GitHub - escamoteur/flutter_mandel at using_isolates

1 Like
3

Thanks, I will check it out!
Too bad can’t run it directly on my mac, but maybe I can take some example from the code.

My main problem is figuring out how to make all the calls through a single interface that is running in the isolate probably, but to be honest I’ve only spent couple of days looking at it at this point, will probably make a test project that is using a simplified API to make all the calls to the C++ through the isolate, might share it for others to look at later also.

1 Like
4

Why can’t you run it on the Mac?

5

Use integral_isolates | Dart package.

It will create a forever-running isolate that you can call whenever you want.

It also supports stream (useful for event driven).

2 Likes
6

Have you considered simply wrapping your long running computation in Isolate.run call?

If you are concerned about isolate spawning overhead you could reuse the same isolate by setting up a communication channel between the temporary isolate created by Isolate.run and the main isolate. Below you can see an example of such a wrapper. One thing to be aware of is that there is currently a limitation on the number of isolates which can be simultaneously blocked on some synchronous function (be it an FFI call or dart:io sleep), it is documented here

import 'dart:async';
import 'dart:io';
import 'dart:isolate';

enum Event { data, error, done }

Future<void> _runInIsolate<R>(
    Stream<R> Function() computation, SendPort sendPort) {
  return Isolate.run(() async {
    try {
      await for (var v in computation()) {
        sendPort.send([Event.data, v]);
      }
      sendPort.send([Event.done]);
    } catch (e, st) {
      sendPort.send([Event.error, e, st]);
    }
  });
}

Stream<R> stream<R>(Stream<R> Function() computation) {
  late final port = ReceivePort();
  late final StreamController<R> controller;

  controller = StreamController<R>(
    onListen: () {
      _runInIsolate(computation, port.sendPort).catchError((e, st) {
        controller.addError(e, st);
      });

      port.listen((event) {
        switch (event) {
          case [Event.data, final v]:
            controller.add(v);
          case [Event.error, final e, final st]:
            controller.addError(e, st);
          case [Event.done]:
            controller.close();
            port.close();
        }
      });
    },

    // TODO(mraleph): you can also forward pause, resume and cancel to the isolate.
  );

  return controller.stream;
}

void main() async {
  await for (var v in stream(() async* {
    for (var i = 0; i < 10; i++) {
      yield i;
      sleep(const Duration(milliseconds: 100));
    }
  })) {
    print('got: $v');
  }
}
4 Likes
7

We’ve been considering expanding the isolate docs or Dart samples repo to cover some more complex isolate use cases. So I understand correctly - is it the managing of long running isolate that you’re looking for an example of, regardless of what is being executed on the isolate? Or is it specifically challenging because of the C++ interop within the isolate?

1 Like
8

Adding a page to reduce the memory copy overhead would also be great. like leaf calls.

1 Like
9

FWIW, I think there’s a gap in the docs when it comes to Isolates vs FFI in general. It doesn’t matter if the other language is C++, C, Rust or whatever else. The point is that you have an external (non-Dart) library that you want to use throughout the execution of the app, and you need to communicate with this external library continuously (i.e. it’s not a single batch job).

Examples:

  • @MarcoB’s flutter_soloud package, which manages a (C++) audio engine through FFI
  • computer vision libraries (e.g. OpenCV)
  • high-performance simulation libraries (e.g. fluid dynamics, original box2d)
  • video manipulation and streaming (e.g. FFmpeg)
  • high-performance compression (e.g. zlib)
  • high-performance image manipulation (e.g. libpng)
  • neural networks (e.g. Tensorflow Lite)
  • linear algebra (e.g. Eigen)
  • OCR (e.g. Tesseract)
  • custom code written in C/C++/Rust for performance reasons (e.g. some code that prepares Vertices.raw arrays that can then be simply canvas.drawVertices()'d in a custom painter)

The problems that need documenting, imho:

  • external (non-Dart) code doesn’t know about isolates, which makes managing callbacks from FFI-land to Dart really hard (this is a known issue but there are workarounds)

  • assuming we have the consumer (main) isolate, the worker isolate (with FFI code) and the external (C/C++/Rust) code, how should one send data in a way that minimizes copying?
    Assume there’s a non-trivial amount of data going in one or both directions every frame.

    • If the answer is to share data between Dart and external code, where should one allocate and who should own (and free) the memory?

  • more generally, how does one wire all this together? @mraleph’s code above is a great start but it took me quite a while to understand what’s going on with things like

    await for (var v in stream(() async* { ... })) { ... }

    and, to be honest, I’m still having trouble grokking the whole thing.

This is an advanced topic, so I think the docs can assume familiarity with concepts such as TypedData, FFI, threads, mutex, pointers, possibly even Finalizer. Otherwise, you’d basically have to make this docs resource into a full course.

I have a feeling that, despite having a niche (advanced) audience, this resource would have high impact because it unlocks a lot of functionality to Flutter apps. It doesn’t move the needle for the majority of apps, but it might be a game changer for the top 1000 or top 10K apps (and games).

3 Likes
10

I would also suggest an alternative of the golden gift suggested by @mraleph. As like for @filip, I’m still having troubles grokking the await for...

You could use nativeCallable.listener. This facilitate the use of [Send|Receive]port between C and Dart. IIRC it creates a new Isolate and for this reason you must call NativeCallable.close() to free it when it is no more needed.

Dart side:

  typedef dartStreamDataCallback_TFunction = ffi.Void Function(
    ffi.Pointer<ffi.UnsignedChar> data, ffi.Int dataLength);

  ffi.NativeCallable<dartStreamDataCallback_TFunction>? nativeStreamDataCallable;

  /// The function which will be called from native
  void _streamDataCallback(
    ffi.Pointer<ffi.UnsignedChar> data,
    int dataLength,
  ) {
    /// Do whatever you want with the `data` received.
  }

  void setDartEventCallbacks() {
    // Create a NativeCallable for the Dart functions.
    nativeStreamDataCallable =
        ffi.NativeCallable<dartStreamDataCallback_TFunction>.listener(
      _streamDataCallback,
    );

	// Call the native function which will store the pointer to the Dart callback.
    _bindings.setDartEventCallback(
      nativeStreamDataCallable!.nativeFunction,
    );
  }
  [...]
  // somewhere in your code call `nativeStreamDataCallable?.close()`

C side:

typedef void (*dartStreamDataCallback_t)(unsigned char * data, int dataLength);

void streamDataCallback(unsigned char *data, int dataLength)
{
    if (dartStreamDataCallback != nullptr)
        dartStreamDataCallback(data, dataLength);
}

/// Must be called from Dart to set the Dart callback function.
FFI_PLUGIN_EXPORT void setDartEventCallback(
    dartStreamDataCallback_t stream_data_callback)
{
    dartStreamDataCallback = stream_data_callback;
}

Now, in you C code, call void streamDataCallback() which will then call the Dart void _streamDataCallback().

Since here NativeCallable.listener has been used, you can do your call from any native threads.

I didn’t find the issue/PR where NativeCallable has been discussed, but IIRC we need to pay attention because every time we create it, a new Isolate that makes use of SendPort and ReceivePort is created.

11

I’ve filed a ticket on dart-lang/site-www, basically with all the information that @filip outlined. Feel free to comment on the issue if there’s more you’d like to see. Also keep the conversation going here if there’s more to discuss, it’s helpful!

4 Likes