Catch up - F# + serialization + Unity

After some digging around, I ended up using MessagePack-CSharp and an FSharp extension. This didn't work out of the box, and I needed to do a decent amount of hacking. I learned some good to know things along the way; like what F# DU's compile to, how dynamic assemblies work and basic IL generation.

I went with MessagePack-CSharp mostly because it advertises Unity/il2cpp support, and I saw there was an F# extension. I figured this is everything I need! What I didn't see was that MessagePack-CSharp had updated to a v2 of its api and the F# extension hadn't been updated to match. So I forked it and updated the IL generators to match the newer API. For most of this process I had no idea what I was doing, but rewriting the IL emit stuff and learning me some opcodes was surprisingly devoid of serious pain. Learning how to do this also proved useful later.

After I got the extension working, I took a look at making sure it works with however MessagePack-CSharp was doing the AOT/il2cpp stuff. Turns out you point it at a C# project file and it'll parse the source, then stick the types into a template so you get generated .cs files out of it. I was beginning to see why they put "CSharp" in the name. Shucks. That wouldn't work for me.

I realized I had just assumed they took a dll as input and worked on the IL classes. Well, that seems like an alright approach to me - so I wrote a utility to do exactly that. It looks at all the types in an input assembly with the MessagePack attribute, and recursively all the types of its fields or any generic type parameters, and calls the MessagePack GetFormatter api for that type. This kicks in the code generation stuff, which usually builds types on a transient dynamic assembly.

I modified the standard MessagePack resolver a tiny bit to make those assemblies savable, along with a couple other hacks to get this working (also learned that one of LoadFile or LoadFrom is evil and should not be used, but no longer remember which). [1]

The resolver generation step also emits a parent resolver type which holds in a dictionary all the generated formatters (this is where learning how to generate a type + method IL came in handy). At serialization time there's very little reflection going on, just grab the type token and look for it in a dictionary. The formatter's (de)serialize methods already have any dependent subtype formatters referenced.

This works out to run pretty fast which helps with server-side time travel. FsPickler used to noticeably freeze the game (though part of this is because it's way more costly the first run as it needs to warm up) but with MessagePack + prebaked formatters it's not noticeable that a save event occurred even when done synchronously on the main thread.

Since I use this output dll in my application (and therefore reference it), I actually needed to separate my types to their own project so I could generate a separate dll and not have a circular reference. Since F# enforces compilation order, these files were already separated - all I had to do was copy and paste the msbuild items into a new .fsproj.

The end result dependencies look like this:

Type definitions project
┣ Generated resolver dlls
┗ ┻ Application project

And the steps are:

1. build types definition project
2. run resolver generator on the output
3. reference that output in application project

If I had to do it again, I think I'd search a little harder for a different route. That said, I'm not convinced there's an easier way to get the same end result. This path gets me fast serialization, works with AOT and the data can be versioned. A reader showed me Odin Serializer which might be good for others looking to do the same thing. It looks like the AOT step is tied to Unity, so I'm not sure if it would've worked for me out of the box either.

To deal with the 7 generic nested types = runtime error issue I plan to eventually count type nest depth with a utility as a post-build step. I already do count them in the resolver generation step, but I was running into the issue with just 4 or 5 deep nests, so I think those types go on to be nested into other generics which trip the limit. I'll need to move this to the last step in the build process, for now I just report any types that hit 5 deep when I generate the resolver. I also just kind of avoid tuples now. :(

Since my last post I've finished adding a basic tutorial system, and now I'm working on getting server-side time travel finished.

Since I don't want to wait until Unity has fully loaded to start the interaction with the server, I decided to add a custom launcher activity for Unity. This meant I had to introduce another (android specific) repo and language (Kotlin). This turned out to be not too horrendous (the Unity documentation was actually pretty good!) but I'll go over exactly how this works and what the development process looks like later.

As for the backend, I decided on serverless with Azure Functions. It's been pretty nice so far, I like the built-in authentication pipeline and one-liner CLI publishing. I added it as an AfterPublish task in msbuild, so I don't think about it at all. There's local hosting/debugging (though I couldn't get authentication working locally), and the unit costs look okay with some back of the napkin math. I'll probably write more on this later as well.

If you stumble on this blog and want to say hi, go for it! I've added my email in the footer (so you don't need to dig through whois 😅). And if you have any questions or suggestions just send them my way.

That's it for this catchup!