Serialization is a key part of almost all applications. And a key problem to figure out for every programming language and ecosystem. Python has pydantic, Swift has Codable, Kotlin has kotlinx.serialization, Rust has serde. Dart has …?
Currently, the Dart ecosystem around serialization is in a bad state. There is no coherent story around general serialization at all. Over the last months I worked on a new system that brings together all the things I learned about serialization from building dart_mappable and researching solutions from other languages.
This RFC proposes a new serialization protocol for Dart, aiming to be modular, universally usable across different data formats, and highly performant. Key points include:
Separation of encoding/decoding logic from data formats.
Support for various data formats (e.g., JSON, CSV, MessagePack) through extension methods.
Handling of nested objects, collections, custom types, binary formats and more.
Extended protocol for polymorphism and generic type handling.
Performance improvements over current methods (+ Benchmark).
The proposal aims to improve serialization in the Dart ecosystem, making it more efficient and flexible.
Looking forward to your feedback! (But please read the RFC before doing so, since it contains a lot more details on the problem statement and reasoning.)
First of all respect for the amount of work that has gone into this.
However my gut feeling is do we need such a complex approach when our current solutions are good enough.
Json is the dominant format for backends and probably will be for the next time.
Could there be a simpler solution that’s good enough without forcing an additional mental effort to understand it?
This RFC tries to offer a super flexible solution where I ask myself how often will I need this?
Many of its advanced features only make real sense if it would be combined with an api definition framework like open-api.
I guess to make this more approachable a smaller Doc focusing on the reference implementations and how they can be used would be helpful.
If json performance is better than the current solutions this should quickly push this to get the goto solution.
Thats where I hard disagree. Almost all other languages have better solutions one way or another, and Dart is much lacking here. Though I do recognize that many devs don’t experience this day to day. I addressed this in the RFC.
Could there be a simpler solution that’s good enough without forcing an additional mental effort to understand it?
My response to this is twofold:
I don’t think the proposed solution is - at its core - very complex. Its four abstract interfaces (Encodable/Decodable, Encoder/Decoder) with two additional convenience interfaces (Codable, SelfEncodable). And the Encoder/Decoder implementation can be considered a black-box for the developer.
An application developer doesn’t even need to touch these interfaces in real-world use, since they would be part of generated code using macros or builders. Kinda like if json_serializable just that the generated implementation is different.
I guess to make this more approachable a smaller Doc focusing on the reference implementations and how they can be used would be helpful.
Its hard because I didn’t want to make the RFC even longer. But the “how could this be used actually” question would require designing a macros api on top of it. But before we discuss how we generate a serialization implementation, we first need to align on what this implementation should actually look like.
I’m not an expert in this topic, but at first glance I’m wondering how this proposal relates to the current dart:convert APIs.
Everything is (seemingly) about JSON.
I might be misunderstanding something here, but doesn’t the Converter API in dart:convert support multiple encodings? I don’t see a reason to create the Encodable / Decodable interfaces but I might be missing something.
Is there something in the current API that prevents developers from building modular / universally usable / performant encodings?
At first I was trying to understand the benefit of this, given how many serialisation solutions we already have, but then I read the RFC and I realise if everyone follows the same interface, it would be simpler to change from one serialiser to another.
Also a hard disagree to @escamoteur suggestion that things are good enough as is or that this is somehow too complicated.
@schultek This was a really great read and I very much enjoyed the shape of the API you were proposing.
I did however have a similar question to what @johnpryan mentioned which is I didn’t quite get why the dart:convert API didn’t work for you here as it seemed to be a very similar shape at first glance to what you are proposing or at least as I understand it at the moment.
Either way, thanks so much for all the hard work and design that went into this already it’s looking great.
Exceptionally simple toJson()/fromJson() serialization of models (arguably much simpler in practice). At the same time being much more comprehensive than anything possible with existing dart:convert classes/interfaces/methods.
Several times faster than existing dart:convert JSON serialization
Additionally allowing for the possible support of ANY X serialization format with only the addition of an import X statement. (Existing models now extended to include toX()/fromX() methods)
The RFC is not intended to illustrate/sell the simplicity of the end user experience using the Codable serialization protocol. (Possibly some here are missing this).
I can’t help but believe that a closer, thorough, reading (and understanding) of the the entire RFC would elicit anything but excitement from any user of the current dart:convert serialization capabilities. Further more I think there are very good arguments for incorporating the protocol INTOdart:convert, as it is complementary and would provide functionality missing from the core library.
For everyday JSON end users of this protocol there is essentially ZERO additional complexity in the encoding/decoding JSON - with the UPSIDE of being several TIMES faster.
Here’s to the future of Dart serialization Thank you @ schultek for all of this amazing engineering!
I’ll end with benchmarks - I encourage anyone to do the same and also check out the extensive examples, tests and benchmarks. (and the existing code has not been optimized for speed)
git clone https://github.com/schultek/codable.git
cd codable
dart test -P benchmark
I read the entire RFC, thanks @schultek for the read.
I do think Dart’s current solution regarding serialisation can be improved. I hope this will happen in the near future.
I think the end-user API surface you propose is quite minimal (so far, I like it!), and I’m looking forward to playing with it and develop a stronger opinion about it. I would like to see examples of how the API behaves with schema-based serializations, versioning, etc.
I also hope you can get ownership of the codable pub name!
In all, thanks again for this magnificent effort and I heavily encourage moving it forward! Keep us posted!
P.S. I understand the frustration of Map<String, dynamic> toJson being named toJson.
Note they say: “One area we’ll be investing in is better support for data in Dart… In fact, our primary motivation for macros was to provide better data handling, serialization, and deserialization”
I also think implementing serialization directly on the language level will ultimately be the better option that trying to solve it with macros that are intended to sole a multitude of cases.
Using contents of this forum for the purposes of training proprietary AI models is forbidden. Only if your AI model is free & open source, go ahead and scrape. Flutter and the related logo are trademarks of Google LLC. We are not endorsed by or affiliated with Google LLC.
Thank you @ 