error 51: User Defined Type too big

[erik]

I have 64-bit FreeBASIC. And the code:

Code: Select all

Const Capacity As ULongInt = &h00000000FFFFFFFF
Type Foo
	x As ZString * Capacity
End Type

I get a compiler error:

Code: Select all

test.bas(7) error 51: User Defined Type too big in 'x As ZString * Capacity'

And this code:

Code: Select all

Dim Shared AAA As ZString * Capacity

compiles without errors.

Is this a compiler bug?
Is there any restriction on the size of the structure that is not specified in the help? A 64‐bit system is able to accommodate a structure with the size of FFFFFFFF bytes.

[fxm]

See Most Important Features:

.....
- String: fixed, variable-length or null-terminated (Zstring), up to 2GB long
.....
- Fixed- and variable- length arrays are supported, up to 2 GB in size.
.....

[erik]

In this page https://www.freebasic.net/wiki/TblVarTypes
Zstring
32bit: +2147483647
64bit: +9223372036854775807

Which of these pages is correct?

[erik]

String: fixed, variable-length or null-terminated (Zstring), up to 2GB long

In this code, the variable takes up 4 gigabytes of memory and this does not cause a compilation error:

Code: Select all

Dim Shared AAA As ZString * Capacity

Why does the compilation error appear only when the variable is a member of the structure?

[fxm]

In this page https://www.freebasic.net/wiki/TblVarTypes
Zstring
32bit: +2147483647
64bit: +9223372036854775807

See the associated note:

[**] All runtime library string procedures take and produce Integer values for sizes and positions. The actual maximum size will vary (smaller) with storage location and/or platform.

[fxm]

In this code, the variable takes up 4 gigabytes of memory and this does not cause a compilation error:

Code: Select all

Dim Shared AAA As ZString * Capacity

Why does the compilation error appear only when the variable is a member of the structure?

By compiling with gcc, I get a warning:

this decimal constant is unsigned only in ISO C90

[erik]

I compile in the standard way: "fbc64.exe test.bas", there are no GCC backend warnings here. When you compile with some other backend or in a non‐standard way, then such warnings should not be interpreted as valid.

The 64‐bit platform allows you to address and use large blocks of memory, much more than 2^32 degrees of bytes. Why should FreeBASIC still use 32‐bit addressing on a 64‐bit platform?

The internal data type for the array "FBARRAY" on a 64‐bit platform has int64 type fields. Thus, there are no reasonable limits of 2 gigabytes for the capacity of the array. It's just a bug in the compiler.

How can FreeBASIC declare a structure or array with a capacity of more than 2 gigabytes of memory?

[fxm]

@Jeff,

For fbc 32-bit, maximum allocation size < 2 GB.
For fbc 64-bit, maximum allocation size < virtual memory.
This is true for dynamic string, array, ... in the heap

- But for a UDT, the maximum size is < 2 GB in both cases 32/64-bit (while a UDT instance can also be allocated in the heap).
Why ?

- For 'Dim Shared As Zstring * ...', the results for fbc 64-bit are weird (dependent on the required size and the backend), but in that case, the allocation is done in the .BSS section, then ?

[erik]

When using the "-std=c89" parameter, the backend will not issue warnings if you specify the constant specifier as a "long integer" or a "unsigned long integer":

Code: Select all

static int8 Vector1[4294967295ull];
static int8 Vector2[4294967295ll];

I think FreeBASIC should be fixed so that it can do this.

[coderJeff]

- But for a UDT, the maximum size is < 2 GB in both cases 32/64-bit (while a UDT instance can also be allocated in the heap).
Why ?

UDT structure size has a hard-coded size limit of &h7fffffff. Original fbc code was all 32-bit only so the limit was added in the early days of fbc 32-bit to 64-bit feature additions to prevent overflowing fbc's internal calculations for type sizes and member offsets. Otherwise dealing with offsets and sizes larger than +-2 GiB could have caused many hard to find bugs. Also, at the time it would have been uncommon to see a PC with > 2-3 GiB of RAM. Maybe this limit can be lifted on fbc 64-bit now? fbc's internal math should now be all 64-bit, even when cross compiling from 32-bit host to 64-bit target (no guarantees though).

[coderJeff]

The 64‐bit platform allows you to address and use large blocks of memory, much more than 2^32 degrees of bytes. Why should FreeBASIC still use 32‐bit addressing on a 64‐bit platform?

64-bit fbc uses 64-bit addressing. We should expect run-time memory allocations to allocate and address > 4 giB. (i.e. ALLOCATE, REDIM, STRING). For example:

Code: Select all

type T
	n as ubyte
end type
sub dump( byref n as string, a() as T )
	var p1 = cint(@a(lbound(a)))
	var p2 = cint(@a(ubound(a)))
	print "REDIM " & n & "(" & lbound(a) & " to " & ubound(a) & ") as T"      
	print hex( p1, sizeof(any ptr)*2 ) & " to " & hex( p2, sizeof(any ptr)*2 ); 
	print " = " & hex( (p2 - p1 + 1) \ sizeof( T ) )
end sub
const capacity = &h100000000ull '' 4 GiB
redim a(0 to capacity) as T     '' 4 GiB + 1
redim b(0 to capacity) as T     '' 4 GiB + 1
dump "a", a()
dump "b", b()

'' OUTPUT:
'' REDIM a(0 to 4294967296) as T
'' 000000007FFF0040 to 000000017FFF0040 = 100000001
'' REDIM b(0 to 4294967296) as T
'' 0000000180001040 to 0000000280001040 = 100000001

[coderJeff]

When using the "-std=c89" parameter, the backend will not issue warnings if you specify the constant specifier as a "long integer" or a "unsigned long integer":

Code: Select all

static int8 Vector1[4294967295ull];
static int8 Vector2[4294967295ll];

I think FreeBASIC should be fixed so that it can do this.

OK, I didn't get any warnings, but adding 'ull' or 'll' suffixes is an easy addition.

However, even if fbc gets the calculations correct and emits the correct gcc code, there is the possibility of bugs related to 64-bit in whatever gcc version or other tools (AS, LD) are being used:

For exampl, noticed this weirdness with gcc 9.3 (what last release is based on):

Code: Select all

const capacity = &h100000000ull '' 2^32 = 4 GiB
dim shared x1 as zstring * capacity
dim shared x2 as zstring * capacity
print "sizeof(x1)=" & hex(sizeof(x1))
print "sizeof(x2)=" & hex(sizeof(x2))
print "@x1="; hex( cuint(@x1), sizeof(any ptr)*2 )
print "@x2="; hex( cuint(@x2), sizeof(any ptr)*2 )
''
'' OUTPUT:
'' sizeof(x1)=100000000
'' sizeof(x2)=100000000
'' @x1=000000000040D040 << same addresses?
'' @x2=000000000040D040 << gcc is not handling static allocation correctly.

[coderJeff]

- For 'Dim Shared As Zstring * ...', the results for fbc 64-bit are weird (dependent on the required size and the backend), but in that case, the allocation is done in the .BSS section, then ?

There does appear to be limits as to the size of the .BSS section. I can get various gcc / AS / LD errors depending on how much data is allocated to .BSS.

[erik]

@coderJeff, How do I declare a structure that takes up more than 2 gigabytes?

Code: Select all

Const capacity = &h0FFFFFFF '       0FFF_FFFF
Type Foo
	a1 As ZString * capacity
	a2 As ZString * capacity
	a3 As ZString * capacity
	a4 As ZString * capacity
	a5 As ZString * capacity
	a6 As ZString * capacity
	a7 As ZString * capacity
	a8 As ZString * capacity
	a9 As ZString * capacity '       <------- compiler error
End Type

Compiler error:

Code: Select all

Copyright (C) 2004-2021 The FreeBASIC development team.
standalone
target:       win64, x86-64, 64bit
backend:      gcc
compiling:    test.bas -o test.c (main module)
test.bas(11) error 51: User Defined Type too big in 'a9 As ZString * capacity'

[fxm]

You can workaround this behavior through pointers declared in the Type, and the memory allocation/deallocation executed by the constructor/destructor:

Code: Select all

Const capacity = &h0FFFFFFF '       0FFF_FFFF
Type Foo
    pa1 As ZString Ptr
    pa2 As ZString Ptr
    pa3 As ZString Ptr
    pa4 As ZString Ptr
    pa5 As ZString Ptr
    pa6 As ZString Ptr
    pa7 As ZString Ptr
    pa8 As ZString Ptr
    pa9 As ZString Ptr
    Declare Constructor()
    Declare Destructor
End Type

Constructor Foo()
    This.pa1 = Callocate(capacity, Sizeof(Zstring))
    This.pa2 = Callocate(capacity, Sizeof(Zstring))
    This.pa3 = Callocate(capacity, Sizeof(Zstring))
    This.pa4 = Callocate(capacity, Sizeof(Zstring))
    This.pa5 = Callocate(capacity, Sizeof(Zstring))
    This.pa6 = Callocate(capacity, Sizeof(Zstring))
    This.pa7 = Callocate(capacity, Sizeof(Zstring))
    This.pa8 = Callocate(capacity, Sizeof(Zstring))
    This.pa9 = Callocate(capacity, Sizeof(Zstring))
End Constructor

Destructor Foo()
    Deallocate(This.pa1)
    Deallocate(This.pa2)
    Deallocate(This.pa3)
    Deallocate(This.pa4)
    Deallocate(This.pa5)
    Deallocate(This.pa6)
    Deallocate(This.pa7)
    Deallocate(This.pa8)
    Deallocate(This.pa9)
End Destructor

Dim As Foo f
*f.pa5 = "FreeBASIC"
Print *f.pa5

Sleep

• (for each unsuccessful memory allocation, a null pointer is returned)