error 51: User Defined Type too big
Re: error 51: User Defined Type too big
What do you needa 2GB UDT for? What problem are you trying to solve? I'm sure there's a better way to do it.
Re: error 51: User Defined Type too big
It's not possible unless the UDT size check is removed/changed in the compiler. Even if >2GiB struct was allowed, it is quite unwieldy and couldn't be used without a run time memory allocation anyway (i.e. ALLOCATE or NEW) due to static variable and stack size limitations (platform dependent).
As caseih suggests, there is probably a better way, which depends on what problem you have to solve.
Re: error 51: User Defined Type too big
I have a long file with a certain structure. Instead of reading it byte by byte into memory, I decided to mapping it in memory. This way I get a pointer to memory.
This memory is organized in a certain way, in the form of a data structure, so it is logical to cast a raw pointer to a pointer to the structure.
But I can't declare such a structure because the compiler forbids creating structures larger than 2 gigabytes. A structure declaration is not a memory allocation, it is not a static variable creation, it is not compiled into any processor instructions. It is just a declaration.
In C language, large structures can be declared. Just imagine: a C programmer will come to port his program to FreeBASIC. And then the compiler shows him an error. A C programmer will exclaim, "Is it still impossible to create large structures in FreeBASIC? Funny!"
Therefore, I do not see any reasonable reasons for the existence of such this restriction.
This memory is organized in a certain way, in the form of a data structure, so it is logical to cast a raw pointer to a pointer to the structure.
But I can't declare such a structure because the compiler forbids creating structures larger than 2 gigabytes. A structure declaration is not a memory allocation, it is not a static variable creation, it is not compiled into any processor instructions. It is just a declaration.
In C language, large structures can be declared. Just imagine: a C programmer will come to port his program to FreeBASIC. And then the compiler shows him an error. A C programmer will exclaim, "Is it still impossible to create large structures in FreeBASIC? Funny!"
Therefore, I do not see any reasonable reasons for the existence of such this restriction.
Re: error 51: User Defined Type too big
Another workaround:
Code: Select all
Type ZstringPointerMapping
Dim As Zstring Ptr p(Any)
Declare Constructor(Byval nbBlock As Integer, Byval sizeBlock As Integer, Byval address1stBlock As Any Ptr)
End Type
Constructor ZstringPointerMapping(Byval nbBlock As Integer, Byval sizeBlock As Integer, Byval address1stBlock As Any Ptr)
Redim This.p(0 To nbBlock - 1)
This.p(0) = address1stBlock
For I As Integer = 1 to nbBlock - 1
This.p(I) = This.p(I - 1) + sizeBlock
Next I
End Constructor
Const capacity = &h0FFFFFFF ' 0FFF_FFFF
Redim As Ubyte array(0 To 9 * capacity -1)
array(8 * capacity + 0) = Asc("F")
array(8 * capacity + 1) = Asc("r")
array(8 * capacity + 2) = Asc("e")
array(8 * capacity + 3) = Asc("e")
array(8 * capacity + 4) = Asc("B")
array(8 * capacity + 5) = Asc("A")
array(8 * capacity + 6) = Asc("S")
array(8 * capacity + 7) = Asc("I")
array(8 * capacity + 8) = Asc("C")
Dim As ZstringPointerMapping zpm = ZstringPointerMapping(9, capacity, @array(0))
Print zpm.p(8)[0]
Sleep
Re: error 51: User Defined Type too big
@fxm, but I can write code much easier if the structures were allowed to be larger than 2 gigabytes:
However, I am getting a compilation error.
Code: Select all
Type Database
Field1 As Long
Field2 As Long
...
End Type
Dim p As Database Ptr = MapViewOfFile(...)
p->Field1 = SomeValue1
p->Field2 = SomeValue2
Re: error 51: User Defined Type too big
What kind of compilation error?
Re: error 51: User Defined Type too big
Sure that's a good idea. But are you telling me your individual data structures (records) are 2 GB in length? How many elements within each record are you trying to access? Just looking at your UDT that FB doesn't like, I can't think of any good reason to use a UDT for that. Why not just create functions for each of the a1 through a9 members and have that function calculate the pointer offset and return it as a zstring pointer. Or macros. Or just do the pointer arithmetic in code. For exampleerik wrote: ↑Sep 20, 2022 6:22 I have a long file with a certain structure. Instead of reading it byte by byte into memory, I decided to mapping it in memory. This way I get a pointer to memory.
This memory is organized in a certain way, in the form of a data structure, so it is logical to cast a raw pointer to a pointer to the structure.
my_string_ptr = memory_pointer + 0x0fffffff * a_number
I'm sure C does support structs with element offsets in the GB range. But how often are they used? I'm sure it is a useful feature for some, including you. Is it worth modifying FB to support? Guess that depends on the devs.In C language, large structures can be declared. Just imagine: a C programmer will come to port his program to FreeBASIC. And then the compiler shows him an error. A C programmer will exclaim, "Is it still impossible to create large structures in FreeBASIC? Funny!"
Therefore, I do not see any reasonable reasons for the existence of such this restriction.
Re: error 51: User Defined Type too big
On this machine in 64 bits Win 10, I can set a pointer to the huge zstring, and even If I stay within that memory slot, I can only use a small part of it.
If I try to use too much
say
For n As Long=1 To 11
g+=g
Next
I get a crash.
The whole program is on a knife edge on this box with this chunk of memory in use.
If I try to use too much
say
For n As Long=1 To 11
g+=g
Next
I get a crash.
The whole program is on a knife edge on this box with this chunk of memory in use.
Code: Select all
#cmdline "-exx"
Const Capacity As Ulongint = &h00000000FFFFFFFF
Dim Shared AAA As ZString * Capacity
Type Foo
As ubyte Const Ptr p=@AAA[0]
Declare Operator Cast() As String
Declare Sub set(As String)
Declare Sub copy(As ubyte Ptr,As Long)
End Type
Operator Foo.cast() As String
Return *cast(zstring ptr,p)
End Operator
Sub Foo.copy(src As ubyte Ptr,count As Long)
For n As Long=0 To count
p[n]=src[n]
Next n
End Sub
Sub Foo.set(s As String)
copy(@s[0],Len(s))
End Sub
Dim As foo u
print "Adress of field ";u.p
u.set("FreeBASIC ")
Print u
Dim shared byref As zString g=AAA
print "Address of string ";@g[0]
For n As Long=1 To 10
g+=g
Next
u.set(g)
var tmp=*cast(zstring ptr,u.p)
print left(tmp,50)
print "String length ";len(tmp)
u.set("Press any key to end . . .")
Print u
Sleep
Re: error 51: User Defined Type too big
Fortunately, the original poster's problem does not involve actual memory allocation. So nothing will be on a "knife edge." The addresses he is accessing refer to a memory-mapped file, so there's no memory allocation done. The file is swapped in and out of real memory as needed, as part of the virtual memory system. It's just like a swap file essentially. On a Linux system you would see the VSS memory use in top grow to reflect this but that would have not much to do with actual RAM pressure.
Off topic, but nerdy, file I/O on most operating systems is done using a similar mechanism. Upon opening, the file is mapped into the address space, and then as the memory locations are read, the VM pulls the file in. All of this allows the kernel to do intelligent caching and managing the physical ram. The difference here is when you read from a file, it copies the file's contents from one memory location to another (your target buffer string for example), whereas explicitly mapping a file to memory allows direct access without the copying done by the file i/o routines in the runtime library. Although maybe there're optimizations that just adjust the memory map when a copy is made, eliminating actual copying of memory pages. This sort of thing is done by network drivers.
Off topic, but nerdy, file I/O on most operating systems is done using a similar mechanism. Upon opening, the file is mapped into the address space, and then as the memory locations are read, the VM pulls the file in. All of this allows the kernel to do intelligent caching and managing the physical ram. The difference here is when you read from a file, it copies the file's contents from one memory location to another (your target buffer string for example), whereas explicitly mapping a file to memory allows direct access without the copying done by the file i/o routines in the runtime library. Although maybe there're optimizations that just adjust the memory map when a copy is made, eliminating actual copying of memory pages. This sort of thing is done by network drivers.
Re: error 51: User Defined Type too big
Documentation updated:
- TblVarTypes → fxm [added maximum size for UDTs] (in 'Standard Data Type Limits')
- KeyPgType → fxm [added link to 'Standard Data Type Limits']
- TblVarTypes → fxm [added maximum size for UDTs] (in 'Standard Data Type Limits')
- KeyPgType → fxm [added link to 'Standard Data Type Limits']
Re: error 51: User Defined Type too big
All your problems stem from the fact that you want to allocate a large memory (4GB - 1) in the .BSS section (bad behavior as Jeff and I previously reported: compiler/linker error or even runtime crash).
No problem if we allocate the memory in the heap (your above code just modified consequently):
Code: Select all
#cmdline "-exx"
Const Capacity As Ulongint = &h00000000FFFFFFFF
Dim Shared AAA As ZString Ptr
AAA = Callocate(Capacity)
Type Foo
As ubyte Const Ptr Ptr p=@AAA
Declare Operator Cast() As String
Declare Sub set(As String)
Declare Sub copy(As ubyte Ptr,As Long)
End Type
Operator Foo.cast() As String
Return **cast(zstring ptr ptr,p)
End Operator
Sub Foo.copy(src As ubyte Ptr,count As Long)
For n As Long=0 To count
(*p)[n]=src[n]
Next n
End Sub
Sub Foo.set(s As String)
copy(@s[0],Len(s))
End Sub
Dim As foo u
print "Adress of field ";*u.p
u.set("FreeBASIC ")
Print u
Dim byref As zString g=*AAA
print "Address of string ";@g[0]
print g
For n As Long=1 To 28
g+=g
Next
u.set(g)
var tmp=*cast(zstring ptr,*u.p)
print left(tmp,50)
print "String length ";len(tmp)
u.set("Press any key to end . . .")
Print u
Deallocate(AAA)
Sleep
Re: error 51: User Defined Type too big
That's not what I need.
I just need to declare a large structure.
I think the right solution would be to remove the structure size check from the compiler.
I just need to declare a large structure.
I think the right solution would be to remove the structure size check from the compiler.
Re: error 51: User Defined Type too big
Not according to this:erik wrote: ↑Sep 20, 2022 6:22 ...
In C language, large structures can be declared. Just imagine: a C programmer will come to port his program to FreeBASIC. And then the compiler shows him an error. A C programmer will exclaim, "Is it still impossible to create large structures in FreeBASIC? Funny!"
...
https://stackoverflow.com/questions/772 ... structures
Quoting: '...So the maximum size is between 2^30 and (2^31 - 1).'
Which is roughly 2 GB. Like other posters said in this thread, I see no point in having such a large structure. Probably a code smell more than anything else.
Re: error 51: User Defined Type too big
Yes, I have the same similar question:
If there was not this 2GB limit for Type size, what would be the exact structure of the Type that you would declare for mapping in the particular case of your application ?
If there was not this 2GB limit for Type size, what would be the exact structure of the Type that you would declare for mapping in the particular case of your application ?