FBSerializer serialize types to binary/json

[shadow008]

I've finally gotten my serialization framework to the point where there's enough functionality worthy of a release. I've made the interface as "plug and play" friendly as possible so in the majority of cases you write a simple declaration of a UDT and that UDT can then immediately be serialized to binary, deserialized from binary, and serialized to (minified) json.

Download the latest: FBSerializer (last updated 2/09/24)

See Serializer.bi for the cheat sheet for quick reference.

The interface in most cases will look extremely simple:

Code: Select all

type MyUDT
	dim i as integer
	dim s as string
	dim f as single
end type

CREATE_SERIALIZER(MyUDT, _	
	MEMBER_SIMPLE(i), _
	MEMBER_SIMPLE(s), _
	MEMBER_SIMPLE(f))

type NestedUDT
	'This UDT holds an array of MyUDT's
	dim udt as MyUDT ptr
	'The count of the array
	dim arrayCount as integer
end type

CREATE_SERIALIZER(NestedUDT, _
	MEMBER_DYNAMIC_ARRAY(udt, arrayCount), _
	MEMBER_SIMPLE(arrayCount))

'You're set to go

In all cases you simply call one of the following 3 functions:

Code: Select all

FBSerializer.SerializeToBinary(@variable, stream)
FBSerializer.DeserializeFromBinary(@variable, stream)
dim json as string = FBSerializer.SerializeToJSON(@variable)
dim retError as string = FBSerializer.ValidateJSON(@variable, stream)
dim retError as string = FBSerializer.DeserializeFromJSON(@variable, stream)

Quick note: Always pass in an empty object when deserializing! Delete any allocated memory in the type etc.
Additionally, if the variable is a UDT, the @ may be omitted and will be passed byref

Of course, this framework has quite a few limitations. If you have a UDT that doesn't "work" out of the box, you can write the serialization yourself, and it will still work with other serializers. The interface is a bit less friendly, but servicable:

Code: Select all

type MyUDT
	dim i as integer
	dim s as string
	dim f as single
end type

CREATE_SERIALIZER(MyUDT, _	
	MEMBER_SIMPLE(i), _
	MEMBER_SIMPLE(s), _
	MEMBER_SIMPLE(f))

type NestedUDT
	'This UDT holds a dynamic FB array of MyUDT's
	'Dynamic FB arrays are not supported out of the box
	dim udt(any) as MyUDT
end type

CUSTOM_SERIALIZER_BEGIN(NestedUDT)

	'Note this is declared private
	private sub SERIALIZE_TO_BINARY_SIGNATURE(inUDT, stream)
		dim l as integer<32> = lbound(inUDT->udt)
		dim u as integer<32> = ubound(inUDT->udt)
		FBSerializer.SerializeToBinary(@l, stream)
		FBSerializer.SerializeToBinary(@u, stream)
		for i as integer = l to u
			FBSerializer.SerializeToBinary(inUDT->udt(i), stream)
		next
	end sub
	
	'Also private
	private sub DESERIALIZE_FROM_BINARY_SIGNATURE(inUDT, stream)
		dim l as integer<32>
		dim u as integer<32>
		FBSerializer.DeserializeFromBinary(@l, stream)
		FBSerializer.DeserializeFromBinary(@u, stream)
		redim inUDT->udt(l to u)
		for i as integer = l to u
			FBSerializer.DeserializeFromBinary(inUDT->udt(i), stream)
		next
	end sub

	'Also private
	private function SERIALIZE_TO_JSON_SIGNATURE(inUDT)
		dim retVal as string = "["
		for i as integer = lbound(inUDT->udt) to ubound(inUDT->udt)
			retVal &= FBSerializer.SerializeToJSON(inUDT->udt(i))
			if i < ubound(inUDT->udt) then
				retVal &= ","
			end if
		next
		return retVal & "]"
	end function
	
	'TODO: examples for custom validate and deserialize from json

CUSTOM_SERIALIZER_END()

dim stream as MemoryStreamType
dim test as NestedUDT

redim test.udt(0 to 5)

for i as integer = 0 to 5
	test.udt(i).i = i
	test.udt(i).s = "string: " & i
	test.udt(i).f = i * 1.5f
next

'Call the custom serializer
'Note the interface is exactly the same as the generated one
FBSerializer.SerializeToBinary(test, stream)

dim newTest as NestedUDT

'Call the custom deserializer
FBSerializer.DeserializeFromBinary(newTest, stream)

'Call the custom serializer to JSON
print FBSerializer.SerializeToJSON(newTest)

sleep

There's a slightly more extensive example at the bottom of Serializer.bas

A quick brief of what types are handled and their corresponding macros and example:

#macro MEMBER_SIMPLE(_MEMBER)
- Every built in type including zstring/wstring pointers, but not including dynamic FB arrays or "ptr ptr" etc
Example: See above

#macro MEMBER_STATIC_FB_ARRAY(_MEMBER)
- A static FB array

#macro MEMBER_DYNAMIC_ARRAY(_MEMBER, _COUNT_MEMBER)
- A pointer based array where the array length is held in another member of the same type

#macro MEMBER_STATIC_ARRAY(_MEMBER, _COUNT)
- A pointer based array where the size is determined by a constant value, this value should not ever change

#macro MEMBER_POINTER(_MEMBER)
- A single pointer to another type.
BE VERY CAREFUL WITH THIS!!! Serializing a pointer that is meant to be just a reference will serialize just fine, but will result in resource duplication on deserialization. This is because the framework assumes a "pointer" is owned by the UDT, not just exists as a reference to an object owned elsewhere.

#macro MEMBER_NAMED_UNION(_MEMBER)
- A member that is a type of a named union. I haven't come up with a good way of representing named unions yet so they're currently just treated as blittable data. You may, if you wish, ignore the fact that a member is a named union and instead just serialize the largest member of the union.

When adding members to the serializer array, you can forego adding a member if you don't want it to be [de]serialized. There is no issue with this other than you having to manually "patch up" the results of a deserialization as it didn't cover the whole type. The following is just fine:

Code: Select all

type udt
	dim i as integer
	dim s as string
end type

CREATE_SERIALIZER(udt, _
	MEMBER_SIMPLE(i))

dim test as udt
test.i = 10
test.s = "hello world"

'Serialization/deserialization will not pick up on the string member

As stated before, there are limitations to this library (most situations can be overcome with a custom serializer setup though). Please consider them carefully:
- Dynamic UDT member arrays are not handled (yet, if ever).
- Pointers that are references are not to be deserialized! You will want to either not add that member to the serializer or write a custom one.
- I don't have a good way to handle anonymous unions. You can choose to simply serialize the largest member, or write a custom serializer.
- Fixed length strings (as opposed to fixed length zstrings or wstrings) are jank. They should be removed from the language for being bad and as such, they work here only due to monkey patching. If you use fixed length strings; stop. Get some help. Convert to zstrings.
- Requires use of the provided stream class. I'm not sure of a good way to hook in custom input/output bits outside of the extremely simple stream interface provided.
- Does not support "ptr ptr" or "ptr ptr ptr" et al out of the box. That's a custom situation.
- Any deserialization of a pointer array or pointer member will create new memory via "new" or "new []". Therefore, all memory must be deleted with "delete" and not "deallocate".
- Additionally, any UDT without a default constructor cannot be serialized automatically. I don't have a way of writing custom allocators yet so this CANNOT BE HANDLED AT ALL.
- UDTs with private members cannot have those members serialized through the convenience macros. You would have to write the array directly with the offsets etc. I don't recommend doing this.
- The custom serializer interface is (still) a bit unfriendly. Feedback is very welcome.

Regardless, let me know if you find it useful.
~shadow008

[shadow008]

Added:
- Deserializing from json
- Validating json against a UDT schema
- Pretty print for formatting json