How to embed any binary file into your FB executable

Post your FreeBASIC source, examples, tips and tricks here. Please don’t post code without including an explanation.
Post Reply
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

How to embed any binary file into your FB executable

Post by caseih »

In another thread on this forum, I discussed a method[1] of embedding any binary resource into your FB executable at compile time, allowing for the bundling of things such as audio, images, shader scripts, etc, into a single executable. I figure I should post it here so others can find it. Also @coderJeff created a slick macro-based wrapper[2] for FB 1.08 or newer to make it super easy to work with embedded resources.

Here's what I wrote, followed by @coderJeff's version of the code:

Say you have a file hello.txt, which could be anything from plain text to any binary file such as a bitmap image or audio file. Turn it into a .o file:

Code: Select all

ld -r -b binary -o hello.o hello.txt
The linker will build symbols based on the file name. For example, if you used hello.txt, then it will provide two symbols, _binary_hello_txt_start and _binary_hello_txt_end. These are not variables. They are bytes that actually exist at the location of your file's data. Take their addresses to get pointers to the beginning and end of the data. The start and end addresses are actually an exclusive range, meaning that size = end - start. the end address is actually just past the end of your binary data.

Here's an example of using these symbols to access the data:

Code: Select all

extern hello_txt_start alias "_binary_hello_txt_start" as byte
extern hello_txt_end alias "_binary_hello_txt_end" as byte

dim hello as byte ptr
dim hello_length as integer

hello = @hello_txt_start
hello_length = @hello_txt_end - @hello_txt_start

dim x as integer

for x = 0 to hello_length-1
	print chr(hello[x]);
next

print
When compiling your .bas file, include the hello.o file:

Code: Select all

fbc myfile.bas hello.o
And here's @coderJeff's version which I recommend using because it encapsulates everything in an idiomatic way and deals with differences between 32-bit and 64-bit Windows behavior of the GNU linker:

Code: Select all

type RESOURCE
   name as const zstring const ptr
   data as const ubyte const ptr
   data_end as const ubyte const ptr
   declare const property size() as const uinteger
   declare sub dump()
end type

property RESOURCE.size() as const uinteger
   return this.data_end - this.data
end property

sub RESOURCE.dump()
   print "Name: " & *this.name
   print "Data: " & hex( cint(this.data), sizeof( any ptr) * 2 )
   print "Size: " & this.size
   
   for i as integer = 0 to this.size - 1
      select case this.data[i]
      case 32 to 127
         print chr( this.data[i] );
      case else
         print "<" & hex( this.data[i], 2 ) & "> ";
      end select
   next
end sub

#if defined( __FB_WIN32__ ) and not defined( __FB_64BIT__)
   '' win32 32-bit gcc/gas
   #define BINARYPREFIX binary_
#else
   '' everything else
   #define BINARYPREFIX _binary_
#endif

#macro decl_readonly_resource( symbol, filename )
   extern "c"
      extern as const byte __FB_JOIN__( BINARYPREFIX, filename##_start )
      extern as const byte __FB_JOIN__( BINARYPREFIX, filename##_end   ) 
   end extern
   
   dim shared as RESOURCE symbol = _
      ( _
         @ #symbol, _
         @__FB_JOIN__( BINARYPREFIX, filename##_start ), _
         @__FB_JOIN__( BINARYPREFIX, filename##_end   ) _
      )
#endmacro

decl_readonly_resource( hello, hello_txt )

hello.dump()
It's important to note that none of these examples will work on MacOS versions of FB, as MacOS does not use the GNU compiler tools. This should work, however, on any Linux distribution on Intel or ARM. So it would work on the Pi, for example. The only limit is how much you want to bloat your exe. Everything must be mapped into memory at load time, so this technique may not always be appropriate.

EDIT: In another forum post, @dodicat shows how to wrap the embedded binary object file into a static library archive[3], which means you can link it into your FB program with just and #inclib line of code, rather than fuss about with passing the arbitrary embedded object file to the compiler, which is something an IDE might not make so easily. To make the static library you would do something like this:

Code: Select all

ar rcs -o libhello.a hello.o
And then in your FB code, just add:

Code: Select all

#inclib "hello"
[1] viewtopic.php?p=286041#p286041
[2] viewtopic.php?p=286050#p286050
[3] viewtopic.php?p=286107#p286107
Last edited by caseih on Oct 14, 2021 23:32, edited 1 time in total.
srvaldez
Posts: 3379
Joined: Sep 25, 2005 21:54

Re: How to embed any binary file into your FB executable

Post by srvaldez »

thanks caseih and coderJeff
this is a very useful tip, there's a difference in the output however, caseih's example will print newlines whereas coderJeff's example prints it witout newlines
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: How to embed any binary file into your FB executable

Post by caseih »

Editing this post for clarity. In both cases the printing out of the data and the dump() routine are essentially for debugging purposes to show that the data is in fact there. In a real-world application, you'd send the data pointer off to some function to process the data, such a routine to display an image. It will get the exact bytes as were in the original file.
Last edited by caseih on Oct 13, 2021 20:18, edited 1 time in total.
xbgtc
Posts: 249
Joined: Oct 14, 2007 5:40
Location: Australia

Re: How to embed any binary file into your FB executable

Post by xbgtc »

I was just thinking this should be in tips and tricks and it is haha. Thanks for this! Should come in useful sometime.
srvaldez
Posts: 3379
Joined: Sep 25, 2005 21:54

Re: How to embed any binary file into your FB executable

Post by srvaldez »

caseih, in this example a text file was used so for a text file it's normal to expect newlines to be printed, I don't understand why you took my simple comment the wrong way
next time I will just shut up.
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: How to embed any binary file into your FB executable

Post by caseih »

Sorry! No offense was intended, nor was I reacted to your comment. I just wanted to be clear that both code snippets access the data identically, and that the data can be used verbatim by any function or algorithm. That's all.
srvaldez
Posts: 3379
Joined: Sep 25, 2005 21:54

Re: How to embed any binary file into your FB executable

Post by srvaldez »

thanks caseih for the clarification, looks like it was I that misinterpreted
I respect your insightful posts in this forum :-)
UEZ
Posts: 988
Joined: May 05, 2017 19:59
Location: Germany

Re: How to embed any binary file into your FB executable

Post by UEZ »

@caseih: I'm getting error messages when following your instruction:

Code: Select all

...\FreeBASIC\bin\win32\ld.exe: Example1.o:fake:(.text+0x32): undefined reference to `_binary_hello_txt_start'
...\FreeBASIC\bin\win32\ld.exe: Example1.o:fake:(.text+0x3a): undefined reference to `_binary_hello_txt_end'
...\FreeBASIC\bin\win32\ld.exe: Example1.o:fake:(.text+0x3f): undefined reference to `_binary_hello_txt_start'
Tested on Win10 and FreeBASIC Compiler - Version 1.08.1 (2021-07-05), built for win32 (32bit).

Same ems on dodicat's example.

On the other hand coderJeff's example works properly.
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: How to embed any binary file into your FB executable

Post by caseih »

Yes, that is correct, @UEZ. @coderJeff explained this in his post and how he handles the differences of the Windows 32-bit linker in his macro. As dodicat says below, it's because the 32-bit linker names the symbols differently. See his post. Basically 32-bit leaves off the beginning underscore. You can use the nm.exe utility to list out the symbols in an object file or .a static library file.
Last edited by caseih on Oct 16, 2021 0:18, edited 1 time in total.
dodicat
Posts: 7983
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: How to embed any binary file into your FB executable

Post by dodicat »

The only difference is
32 bit
extern bob_bmp_start alias "binary_bob_bmp_start" as byte
extern bob_bmp_end alias "binary_bob_bmp_end" as byte

...
64 bit
extern bob_bmp_start alias "_binary_bob_bmp_start" as byte
extern bob_bmp_end alias "_binary_bob_bmp_end" as byte

If you work from the bin folders and use;
ld -r -b binary -o bob.o bob.bmp
ar rcs -o libbob.a bob.o
Then you get the 32 bit and 64 bit object/lib files respectively.

Otherwise of course you use your mingw (if you have it)
Caseih's is very easy to handle, no udt's macros e.t.c. needed.
You can easily use #if to separate which alias string to use:

extern bob_bmp_start alias "binary_bob_bmp_start" as byte
extern bob_bmp_end alias "binary_bob_bmp_end" as byte
or the other.
I tested 32 and 64 bits, the only snag is that I have cluttered up my bin folders during my deliberations getting the bitmaps tested for various sizes.
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: How to embed any binary file into your FB executable

Post by caseih »

Good point, @dodicat. Thanks for providing the underlying explanation that I neglected. Hope that helps, @UEZ.
UEZ
Posts: 988
Joined: May 05, 2017 19:59
Location: Germany

Re: How to embed any binary file into your FB executable

Post by UEZ »

Thank you both. I finally got both examples to work properly. My issue was the "extern ..." line which I used incorrectly for the appropriate version. Further I've to add #Inclib to the source otherwise I get the fake ems.

@dodicat: your bmpload function seems not to work properly.

Image

In the middle on the left side there are some white pixels and on the right side some black pixels.
dodicat
Posts: 7983
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: How to embed any binary file into your FB executable

Post by dodicat »

Hi UEZ
Better to use the bsave method maybe.
viewtopic.php?f=3&t=29652
(second from bottom)
I'll try and get the other method pixel perfect because we wouldn't want to speckle your signature picture.
UEZ
Posts: 988
Joined: May 05, 2017 19:59
Location: Germany

Re: How to embed any binary file into your FB executable

Post by UEZ »

dodicat wrote:Hi UEZ
Better to use the bsave method maybe.
viewtopic.php?f=3&t=29652
(second from bottom)
I'll try and get the other method pixel perfect because we wouldn't want to speckle your signature picture.
I like more the method to load from memory directly instead of saving it first to disk. :-)

Here my version using base64 and compression to embed an image to exe and load it directly from mem using your code just for comparison with this method of this topic.

x86 compilation file length:
Dodicat's version: 237 kb
My version: 191 kb

bob.bas can be used from here: https://pastebin.com/QYxGpAmg

Code: Select all

#Include "bob.bas"
#Include Once "crt/String.bi"

'needed for lzmat
Type info_t
  g_inbuf As Ubyte Ptr
  g_outbuf As Ubyte Ptr
  g_inbuf_pos As Long
  g_outbuf_pos As Long
  g_bit_buf As Long
  g_bit_count As Long
End Type

#Define A_BITS_DEF          1                                    '' 1 xx                  	2
#Define B_BITS_DEF          2                                    '' 01 xx               	2
#Define C_BITS_DEF          3                                    '' 001 xx              	2
#Define D_BITS_DEF          5                                    '' 0001 xxx               	3
#Define E_BITS_DEF          7                                    '' 00001 xxxxx            	5
#Define F_BITS_DEF          9                                    '' 00000 xxxxxxxxx         9
#Define A_DEF             	(1 Shl A_BITS_DEF)
#Define B_DEF             	((1 Shl B_BITS_DEF) + A_DEF)
#Define C_DEF             	((1 Shl C_BITS_DEF) + B_DEF)
#Define D_DEF            	((1 Shl D_BITS_DEF) + C_DEF)
#Define E_DEF             	((1 Shl E_BITS_DEF) + D_DEF)
#Define F_DEF             	((1 Shl F_BITS_DEF) + E_DEF)
#Define SLOT_BITS_DEF       4 
#Define NUM_SLOTS_DEF       (1 Shl SLOT_BITS_DEF)
#Define W_BITS_DEF          19
#Macro FILL_OUT_MACRO(xbuf , d1 , d2)
	xbuf[d1] = xbuf[d2]
	d1 += 1
	d2 += 1
#Endmacro

Private Sub init_bits(Byval inbuf As Ubyte Ptr, Byval outbuf As Ubyte Ptr, Byval tinfo As info_t Ptr)
	tinfo->g_bit_count = 0
	tinfo->g_bit_buf = 0
	tinfo->g_inbuf_pos = 0
	tinfo->g_outbuf_pos = 0
	tinfo->g_inbuf = inbuf
	tinfo->g_outbuf = outbuf
End Sub

Private Function get_bits(Byval n As Long, Byval tinfo As info_t Ptr) As Long
	While tinfo->g_bit_count < n
		tinfo->g_bit_buf Or= tinfo->g_inbuf[tinfo->g_inbuf_pos] Shl tinfo->g_bit_count
		tinfo->g_inbuf_pos += 1
		tinfo->g_bit_count += 8
	Wend
	Dim x As Long = tinfo->g_bit_buf And ((1 Shl n) - 1)
	tinfo->g_bit_buf Shr= n
	tinfo->g_bit_count -= n
	Return x
End Function
   
Private Function lzmat_uncompress(Byval outbuf As Ubyte Ptr, Byval outsize As Long, Byval inbuf As Ubyte Ptr) As Long
	If inbuf = NULL Then
		'fprintf(stderr, !"Compressed buffer corrupted!\n")
		Return -1
	End If
	If outsize < 1 Or outbuf = NULL Then
		'fprintf(stderr, !"Decompression buffer corrupted: size=%d\n", outsize)
		Return -2
	End If
	Dim tinfo As info_t
	init_bits(inbuf + 4, NULL, @tinfo)

	Dim p As Long = 0
	While p < outsize
		If get_bits(1, @tinfo) Then
			Dim len0 As Long
			If get_bits(1, @tinfo) Then
				len0 = get_bits(A_BITS_DEF, @tinfo)
			Elseif get_bits(1, @tinfo) Then
				len0 = get_bits(B_BITS_DEF, @tinfo) + A_DEF
			Elseif get_bits(1, @tinfo) Then
				len0 = get_bits(C_BITS_DEF, @tinfo) + B_DEF
			Elseif get_bits(1, @tinfo) Then
				len0 = get_bits(D_BITS_DEF, @tinfo) + C_DEF
			Elseif get_bits(1, @tinfo) Then
				len0 = get_bits(E_BITS_DEF, @tinfo) + D_DEF
			Else
				len0 = get_bits(F_BITS_DEF, @tinfo) + E_DEF
			End If
			Dim log0 As  Long = get_bits(SLOT_BITS_DEF, @tinfo) + (W_BITS_DEF - NUM_SLOTS_DEF)
			Dim s As Long = (Not Iif(log0 > (W_BITS_DEF - NUM_SLOTS_DEF), _
						   get_bits(log0, @tinfo) + (1 Shl log0), _
						   get_bits(W_BITS_DEF - (NUM_SLOTS_DEF - 1), @tinfo))) + p
			If s < 0 Then
			   'fprintf(stderr, !"Compressed buffer corrupted: s=%d\n", s)
			   Return -3
			End If
			FILL_OUT_MACRO(outbuf, p, s)
			FILL_OUT_MACRO(outbuf, p, s)
			FILL_OUT_MACRO(outbuf, p, s)
			While len0 <> 0
				FILL_OUT_MACRO(outbuf, p, s)
				len0 -= 1
			Wend
			Else
			outbuf[p] = get_bits(8, @tinfo)
			p += 1
		End If
	Wend
  Return p
End Function


'original code by D.J.Peters
Function Base64Decode(sString As String, Byref iBase64Len As Uinteger) As Ubyte Ptr
	#Define P0(p) Instr(B64, Chr(sString[n + p])) - 1
	Dim As String*64 B64 = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
	Dim As String sDecoded
	Dim As Long nChars = Len(sString) - 1
	If nChars < 0 Then Return 0
	For n As Long = 0 To nChars Step 4
		Var b = P0(1), c = P0(2), d = P0(3)
		If b >-1 Then
			Var a = P0(0)
			sDecoded += Chr((a Shl 2 + b Shr 4))
		End If
		If c > -1 Then sDecoded += Chr((b Shl 4 + c Shr 2))
		If d > -1 Then sDecoded += Chr((c Shl 6 + d      ))
	Next
	iBase64Len = Len(sDecoded)
    
    'workaround For multiple embedded file other crash will occure
    Static As Ubyte aReturn(0 To iBase64Len - 1)
    Redim aReturn(0 To iBase64Len - 1) As Ubyte
	
	For i As Ulong = 0 To Len(sDecoded) - 1 'convert result String To ascii code values
		aReturn(i) = Asc(sDecoded, i + 1)
	Next
	Return @aReturn(0) 'Return Pointer To the array
End Function


Function reverse(Byref im As Ulong Ptr) As Any Ptr
    #Define ppset(_x,_y,colour)    *Cptr(Ulong Ptr,row+ (_y)*pitch+ (_x) Shl 2)  =(colour)
    #Define ppoint(_x,_y)          *Cptr(Ulong Ptr,row + (_y)*pitch + (_x) Shl 2)
    Dim As Integer pitch
    Dim As Any Ptr row
    Dim As Ulong Ptr pixel
    Dim As Integer dx,dy,sz
    Imageinfo im,dx,dy,,pitch,row,sz
'    ' leopard's way...
    dx -= 1 : dy -= 1
    Dim As Ulong tempclr
    For y As Integer=0 To dy
        For x As Integer=0 To dx \ 2
            tempclr = ppoint(x,y)
            ppset (x,y,ppoint((dx)-x,y))
            ppset ((dx)-x,y,tempclr)
        Next x
    Next y 
    Return im
End Function

Function bmpload(s As Ubyte Ptr, l As Ulong) As Ulong Ptr
	Static As Ulong Ptr u : u=New Ulong[l-1]
	u[0]=7
	u[1]=4
	u[2]=*Cast(Ulong Ptr,@s[18])
	u[3]=*Cast(Ulong Ptr,@s[22])
	u[4]=(u[2]*4)
	u[5]=0
	u[6]=0
	u[7]=0
	Dim As Long ctr=7
	Var k=Iif(u[2] Mod 16=0,1,0)
	For n As Long=8 To l+6 Step 3+k
		ctr+=1
		u[ctr]=*Cast(Ulong Ptr,@s[55-k+n])
	Next
	ctr -= 7
	u=reverse(u)
	For n As Long=8 To ctr\2
		Swap u[n],u[ctr-n]
	Next
	Return u
End Function


Dim As Uinteger iLines, iCompression, iFileSize, iCompressedSize
Dim As String sBaseType, sBase64, aB64(1)

Restore __Label0:
Read iLines
Read iCompression
Read iFileSize
Read iCompressedSize
Read sBaseType

For i As Ushort = 0 To iLines - 1
   Read aB64(0)
   sBase64 &= aB64(0)
Next
Dim As Uinteger l 
Dim As Ubyte Ptr aBinary = Base64Decode(sBase64, iCompressedSize)

Dim As Ubyte Ptr aBinaryC = Allocate(iFileSize) 
lzmat_uncompress(aBinaryC, iFileSize, aBinary)

Screenres 200, 200, 32,, 8 'GFX_NO_FRAME
Put(0, 0), bmpload(aBinaryC, iFileSize), Pset

Sleep
Deallocate(aBinaryC)
aBinary = 0
sBase64 = ""
Btw, a funny effect on my avatar image when commenting out ppset (x,y,ppoint((dx)-x,y)) in the reverse function:

Image
Post Reply