error 51: User Defined Type too big

General FreeBASIC programming questions.
caseih
Posts: 2044
Joined: Feb 26, 2007 5:32

Re: error 51: User Defined Type too big

Post by caseih »

What do you needa 2GB UDT for? What problem are you trying to solve? I'm sure there's a better way to do it.
coderJeff
Site Admin
Posts: 4009
Joined: Nov 04, 2005 14:23
Location: Ontario, Canada
Contact:

Re: error 51: User Defined Type too big

Post by coderJeff »

erik wrote: Sep 19, 2022 17:53 @coderJeff, How do I declare a structure that takes up more than 2 gigabytes?
It's not possible unless the UDT size check is removed/changed in the compiler. Even if >2GiB struct was allowed, it is quite unwieldy and couldn't be used without a run time memory allocation anyway (i.e. ALLOCATE or NEW) due to static variable and stack size limitations (platform dependent).

As caseih suggests, there is probably a better way, which depends on what problem you have to solve.
erik
Posts: 35
Joined: Dec 28, 2020 17:27
Location: Krasnoyarsk
Contact:

Re: error 51: User Defined Type too big

Post by erik »

I have a long file with a certain structure. Instead of reading it byte by byte into memory, I decided to mapping it in memory. This way I get a pointer to memory.
This memory is organized in a certain way, in the form of a data structure, so it is logical to cast a raw pointer to a pointer to the structure.

But I can't declare such a structure because the compiler forbids creating structures larger than 2 gigabytes. A structure declaration is not a memory allocation, it is not a static variable creation, it is not compiled into any processor instructions. It is just a declaration.

In C language, large structures can be declared. Just imagine: a C programmer will come to port his program to FreeBASIC. And then the compiler shows him an error. A C programmer will exclaim, "Is it still impossible to create large structures in FreeBASIC? Funny!"

Therefore, I do not see any reasonable reasons for the existence of such this restriction.
fxm
Moderator
Posts: 11413
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: error 51: User Defined Type too big

Post by fxm »

Another workaround:

Code: Select all

Type ZstringPointerMapping
    Dim As Zstring Ptr p(Any)
    Declare Constructor(Byval nbBlock As Integer, Byval sizeBlock As Integer, Byval address1stBlock As Any Ptr)
End Type

Constructor ZstringPointerMapping(Byval nbBlock As Integer, Byval sizeBlock As Integer, Byval address1stBlock As Any Ptr)
    Redim This.p(0 To nbBlock - 1)
    This.p(0) = address1stBlock
    For I As Integer = 1 to nbBlock - 1
        This.p(I) = This.p(I - 1) + sizeBlock
    Next I
End Constructor


Const capacity = &h0FFFFFFF '       0FFF_FFFF

Redim As Ubyte array(0 To 9 * capacity -1)
array(8 * capacity + 0) = Asc("F")
array(8 * capacity + 1) = Asc("r")
array(8 * capacity + 2) = Asc("e")
array(8 * capacity + 3) = Asc("e")
array(8 * capacity + 4) = Asc("B")
array(8 * capacity + 5) = Asc("A")
array(8 * capacity + 6) = Asc("S")
array(8 * capacity + 7) = Asc("I")
array(8 * capacity + 8) = Asc("C")

Dim As ZstringPointerMapping zpm = ZstringPointerMapping(9, capacity, @array(0))
Print zpm.p(8)[0]

Sleep
erik
Posts: 35
Joined: Dec 28, 2020 17:27
Location: Krasnoyarsk
Contact:

Re: error 51: User Defined Type too big

Post by erik »

@fxm, but I can write code much easier if the structures were allowed to be larger than 2 gigabytes:

Code: Select all

Type Database
	Field1 As Long
	Field2 As Long
	...
End Type

Dim p As Database Ptr = MapViewOfFile(...)

p->Field1 = SomeValue1
p->Field2 = SomeValue2
However, I am getting a compilation error.
fxm
Moderator
Posts: 11413
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: error 51: User Defined Type too big

Post by fxm »

What kind of compilation error?
caseih
Posts: 2044
Joined: Feb 26, 2007 5:32

Re: error 51: User Defined Type too big

Post by caseih »

erik wrote: Sep 20, 2022 6:22 I have a long file with a certain structure. Instead of reading it byte by byte into memory, I decided to mapping it in memory. This way I get a pointer to memory.
This memory is organized in a certain way, in the form of a data structure, so it is logical to cast a raw pointer to a pointer to the structure.
Sure that's a good idea. But are you telling me your individual data structures (records) are 2 GB in length? How many elements within each record are you trying to access? Just looking at your UDT that FB doesn't like, I can't think of any good reason to use a UDT for that. Why not just create functions for each of the a1 through a9 members and have that function calculate the pointer offset and return it as a zstring pointer. Or macros. Or just do the pointer arithmetic in code. For example

my_string_ptr = memory_pointer + 0x0fffffff * a_number
In C language, large structures can be declared. Just imagine: a C programmer will come to port his program to FreeBASIC. And then the compiler shows him an error. A C programmer will exclaim, "Is it still impossible to create large structures in FreeBASIC? Funny!"

Therefore, I do not see any reasonable reasons for the existence of such this restriction.
I'm sure C does support structs with element offsets in the GB range. But how often are they used? I'm sure it is a useful feature for some, including you. Is it worth modifying FB to support? Guess that depends on the devs.
dodicat
Posts: 7647
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: error 51: User Defined Type too big

Post by dodicat »

On this machine in 64 bits Win 10, I can set a pointer to the huge zstring, and even If I stay within that memory slot, I can only use a small part of it.
If I try to use too much
say
For n As Long=1 To 11
g+=g
Next
I get a crash.
The whole program is on a knife edge on this box with this chunk of memory in use.

Code: Select all

#cmdline "-exx"

Const Capacity As Ulongint = &h00000000FFFFFFFF

Dim Shared AAA As ZString * Capacity

Type Foo 
      As ubyte Const Ptr p=@AAA[0]
      Declare Operator Cast() As String
      Declare Sub set(As  String)
      Declare Sub copy(As ubyte Ptr,As Long)
End Type

Operator Foo.cast() As String
Return *cast(zstring ptr,p)
End Operator

Sub Foo.copy(src As ubyte Ptr,count As Long)
      For n As Long=0 To count
            p[n]=src[n]
      Next n
End Sub

Sub Foo.set(s As String)
      copy(@s[0],Len(s))
End Sub

Dim  As foo u
print "Adress of field ";u.p

u.set("FreeBASIC ")
Print u

Dim shared byref As zString g=AAA
print "Address of string ";@g[0]
For n As Long=1 To 10
      g+=g
Next

u.set(g)
var tmp=*cast(zstring ptr,u.p)

print left(tmp,50)
print "String length ";len(tmp)


u.set("Press any key to end .  .  .")
Print u

Sleep
 
caseih
Posts: 2044
Joined: Feb 26, 2007 5:32

Re: error 51: User Defined Type too big

Post by caseih »

Fortunately, the original poster's problem does not involve actual memory allocation. So nothing will be on a "knife edge." The addresses he is accessing refer to a memory-mapped file, so there's no memory allocation done. The file is swapped in and out of real memory as needed, as part of the virtual memory system. It's just like a swap file essentially. On a Linux system you would see the VSS memory use in top grow to reflect this but that would have not much to do with actual RAM pressure.

Off topic, but nerdy, file I/O on most operating systems is done using a similar mechanism. Upon opening, the file is mapped into the address space, and then as the memory locations are read, the VM pulls the file in. All of this allows the kernel to do intelligent caching and managing the physical ram. The difference here is when you read from a file, it copies the file's contents from one memory location to another (your target buffer string for example), whereas explicitly mapping a file to memory allows direct access without the copying done by the file i/o routines in the runtime library. Although maybe there're optimizations that just adjust the memory map when a copy is made, eliminating actual copying of memory pages. This sort of thing is done by network drivers.
fxm
Moderator
Posts: 11413
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: error 51: User Defined Type too big

Post by fxm »

Documentation updated:
- TblVarTypes → fxm [added maximum size for UDTs] (in 'Standard Data Type Limits')
- KeyPgType → fxm [added link to 'Standard Data Type Limits']
fxm
Moderator
Posts: 11413
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: error 51: User Defined Type too big

Post by fxm »

dodicat wrote: Sep 20, 2022 20:11 On this machine in 64 bits Win 10, I can set a pointer to the huge zstring, and even If I stay within that memory slot, I can only use a small part of it.

All your problems stem from the fact that you want to allocate a large memory (4GB - 1) in the .BSS section (bad behavior as Jeff and I previously reported: compiler/linker error or even runtime crash).

No problem if we allocate the memory in the heap (your above code just modified consequently):

Code: Select all

#cmdline "-exx"

Const Capacity As Ulongint = &h00000000FFFFFFFF

Dim Shared AAA As ZString Ptr
AAA = Callocate(Capacity)

Type Foo 
      As ubyte Const Ptr Ptr p=@AAA
      Declare Operator Cast() As String
      Declare Sub set(As  String)
      Declare Sub copy(As ubyte Ptr,As Long)
End Type

Operator Foo.cast() As String
Return **cast(zstring ptr ptr,p)
End Operator

Sub Foo.copy(src As ubyte Ptr,count As Long)
      For n As Long=0 To count
            (*p)[n]=src[n]
      Next n
End Sub

Sub Foo.set(s As String)
      copy(@s[0],Len(s))
End Sub

Dim  As foo u
print "Adress of field ";*u.p

u.set("FreeBASIC ")
Print u

Dim byref As zString g=*AAA
print "Address of string ";@g[0]
print g
For n As Long=1 To 28
      g+=g
Next

u.set(g)
var tmp=*cast(zstring ptr,*u.p)

print left(tmp,50)
print "String length ";len(tmp)


u.set("Press any key to end .  .  .")
Print u

Deallocate(AAA)

Sleep
erik
Posts: 35
Joined: Dec 28, 2020 17:27
Location: Krasnoyarsk
Contact:

Re: error 51: User Defined Type too big

Post by erik »

That's not what I need.
I just need to declare a large structure.
I think the right solution would be to remove the structure size check from the compiler.
paul doe
Moderator
Posts: 1661
Joined: Jul 25, 2017 17:22
Location: Argentina

Re: error 51: User Defined Type too big

Post by paul doe »

erik wrote: Sep 20, 2022 6:22 ...
In C language, large structures can be declared. Just imagine: a C programmer will come to port his program to FreeBASIC. And then the compiler shows him an error. A C programmer will exclaim, "Is it still impossible to create large structures in FreeBASIC? Funny!"
...
Not according to this:

https://stackoverflow.com/questions/772 ... structures

Quoting: '...So the maximum size is between 2^30 and (2^31 - 1).'

Which is roughly 2 GB. Like other posters said in this thread, I see no point in having such a large structure. Probably a code smell more than anything else.
caseih
Posts: 2044
Joined: Feb 26, 2007 5:32

Re: error 51: User Defined Type too big

Post by caseih »

erik wrote: Sep 21, 2022 16:47I just need to declare a large structure.
You never answered my questions before. How many members do you need inside this large structure? How big is each member? Are they all the same size? Are you dealing with multiple records?
fxm
Moderator
Posts: 11413
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: error 51: User Defined Type too big

Post by fxm »

Yes, I have the same similar question:
If there was not this 2GB limit for Type size, what would be the exact structure of the Type that you would declare for mapping in the particular case of your application ?
Post Reply