Array size limit

[creek23]

Hi all,

From my understanding, an array size is only limited by the available memory. But somehow my app is is crashing if the array size is 52517 long... while it works fine if its 52516 long only. And since I have 4GB, I was expecting a bigger size than what I'm getting now.

Btw, my array is an array of this UDT:

Code: Select all

Type Foo
   aaa As String
   bbb As Double
   ccc As Integer
   ddd As UInteger
   eee As UInteger
   fff As Integer
   ggg As Integer
   hhh As Integer
   iii As Long
   jjj As Integer
End Type

I guess 52516 is as good as 52516(size)x10(type-members), which would mean I'm technically getting a whooping 525,160. o.O

I'm using fbc 0.23 btw.

~creek23

[VANYA]

I also asked this question before :)

Use global arrays

Code: Select all

Type Foo
   aaa As String
   bbb As Double
   ccc As Integer
   ddd As UInteger
   eee As UInteger
   fff As Integer
   ggg As Integer
   hhh As Integer
   iii As Long
   jjj As Integer
End Type

Dim shared a(60000) As foo

[pestery]

Its the stack size. When you don't use Shared the memory is allocated from the stack. On Windows I think the stack size defaults too about 1 MB (on Linux it's different, but I forget how exactly). You can increase the stack size by using the -t compiler option (here).

To solve your problem either use (Dim Shared array(1 to 52517) As Foo) or use the -t compiler option to increase the stack size.

[marcov]

On Windows I think the stack size defaults too about 1 MB (on Linux it's different, but I forget how exactly).

On Linux, stack sizes are more enforced by the system, though these limits are typically lax and non-enforced on single user systems.

On servers however this is more strict. Type "ulimit -a" to see the limits.

[creek23]

Hi all,

Thank you for the replies.

With regards to declaration, I actually declared it like this:

Code: Select all

Extern Bar() As Foo

since it will be used across BAS files and ReDim'ed it from inside a function. Is that as good as Shared?

~creek23

[petan]

Hi,
I use only "Shared" command for global variables (multidim, UDT...), and after, redimmed from anywhere, as I need.

Pete

[fxm]

With regards to declaration, I actually declared it like this:

Code: Select all

Extern Bar() As Foo

since it will be used across BAS files and ReDim'ed it from inside a function. Is that as good as Shared?

~creek23

The array Bar() is so dynamic and it is not in the stack.

Remark:
Useful memory size of one element 'Bar(I)':
Sizeof(Foo) + Cast(Integer Ptr, @Bar(I).aaa)[2] = 56 + Cast(Integer Ptr, @Bar(I).aaa)[2] bytes
('Cast(Integer Ptr, @Bar(I).aaa)[2]' bytes being the useful memory size allocated for the string data, which is greater or equal to the string length)
('Cast(Integer Ptr, @Bar(I).aaa)[2]' being the third data of the array descriptor which is at '@Bar(I).aaa')
In case of non-nul string, one additional byte is allocated to the string data, corresponding to the non-useful terminal character 'Chr(0)'.

Code: Select all

Type Foo
   aaa As String
   bbb As Double
   ccc As Integer
   ddd As UInteger
   eee As UInteger
   fff As Integer
   ggg As Integer
   hhh As Integer
   iii As Long
   jjj As Integer
End Type

Print Sizeof(Foo)
Print
Print Offsetof(Foo, aaa)
Print Offsetof(Foo, bbb)
Print Offsetof(Foo, ccc)
Print Offsetof(Foo, ddd)
Print Offsetof(Foo, eee)
Print Offsetof(Foo, fff)
Print Offsetof(Foo, ggg)
Print Offsetof(Foo, hhh)
Print Offsetof(Foo, iii)
Print Offsetof(Foo, jjj)

Sleep

[creek23]

Thanks fxm. With your snippet, I just realized that from 52516 array size multiplied by 56 bytes (as returned by SizeOf) I am actually using 2,940,896 bytes. o.O

I guess it's a dead end for me here unless I write a better algorithm for my program. :(

~creek23

[Richard]

There will always be arrays that get too big for memory.
A file on disk does not have the same size limitation.
Let the HDD cache speed up the disk access.

[fxm]

I just realized that from 52516 array size multiplied by 56 bytes (as returned by SizeOf) I am actually using 2,940,896 bytes.

That corresponds to 2.8 MB versus 4 GB of memory!
Why your program crashes when the array is greater than this limit of 52516 elements?

[pestery]

Why your program crashes when the array is greater than this limit of 52516 elements?

Because creek23 was allocating the memory from the stack, not the system memory. Or at least that was the impression I got ;-)

Also, regarding the maximum amount of allocated memory, I think its only 2 GB because the pointers use signed Integers (because FB is a 32bit compiler). Although you can't normally allocate 2 GB in a single big chunk anyway, I've tried :-) Instead you should use 2 * 1 GB chunks, or 4 * 512 MB chunks, and there also needs to be enough memory left over for the operating system and any other programs you have running.

I guess it's a dead end for me here unless I write a better algorithm for my program. :(

Maybe not. You might just need to change how you define the array initially. For example:

Code: Select all

' The memory for the array is allocated from
' the stack, but is too large for the stack
' You could use this if you increase the stack size
Dim As Foo array(1 To 52517)

' The memory is allocated from the system memory (RAM)
' but is also visible from inside subs and functions
Dim Shared As Foo array(1 To 52517)

' The memory is allocated from the system memory (RAM)
' and is not visible from inside subs and functions
' The array is also resizeable
ReDim As Foo array(1 To 52517)

' This causes a syntax error
Common As Foo array(1 To 52517)

' This also causes a syntax error, but only because of
' the String in Foo. Because Constructors/Destructors
' are not allowed in common arrays. If the String is
' removed then this works like the normal ReDim, but
' is accessable from any module that includes it
Common As Foo array()
ReDim As Foo array(1 To 52517) ' Only include in 1 module

' This causes a syntax error
Extern As Foo array()
Dim As Foo array(1 To 52517)

' This works like the ReDim Shared, but is accessable
' from any module that includes the Extern statement
' And unlike Common you can also use Strings,
' Constructors and Destructors
Extern As Foo array()
ReDim As Foo array(1 To 52517) ' Only include in 1 module

[TJF]

Also, regarding the maximum amount of allocated memory, I think its only 2 GB because the pointers use signed Integers (because FB is a 32bit compiler). Although you can't normally allocate 2 GB in a single big chunk anyway, I've tried :-) Instead you should use 2 * 1 GB chunks, or 4 * 512 MB chunks, and there also needs to be enough memory left over for the operating system and any other programs you have running.

The 2 GB limit and the restriction regarding signed INTEGER is relevant for STRING variables. Their length is stored in an signed INTEGER.

Pointers in general are UINTEGERs and you can allocate up to 4 GB in one chunk (some memory must be left for the OS).

[MichaelW]

Under Windows each 32-bit process normally has a user-mode virtual address space of 2 GB, and the other 2 GB of the 32-bit address space is reserved for the OS and not accessible from user mode:

http://technet.microsoft.com/en-us/query/aa366778

A process cannot use more memory than will fit in the address space. Since Windows implements a virtual memory system, backed by a paging file, and assuming that the paging file can accommodate the size, the size of the allocation is not limited to the size of the physical memory. This code allocates and then reads through 1.8 GB of memory, and except for a low virtual-memory warning it runs OK on my systems, both of which have only 512 MB of physical memory.

Code: Select all

''=============================================================================
#include "windows.bi"
''=============================================================================
dim as ubyte ptr p
dim as ubyte r

p = VirtualAlloc( NULL, 1.8*1024*1024*1024, MEM_COMMIT, PAGE_READWRITE)
print hex(p);"h"
if p = 0 then
    sleep
    end
end if
for i as integer = 0 to 1.8*1024*1024*1024-4096
    if i and (i mod 1024*1024 = 0) then
        locate 3,1,0
        print i / (1024*1024)
    end if
    r = p[i]
next
print "done"
sleep

I arrived at 1.8 GB by starting at 2 GB and working back until the allocation succeeded.

[fxm]

Why your program crashes when the array is greater than this limit of 52516 elements?

Because creek23 was allocating the memory from the stack, not the system memory. Or at least that was the impression I got ;-)

With regards to declaration, I actually declared it like this:

Code: Select all

Extern Bar() As Foo

since it will be used across BAS files and ReDim'ed it from inside a function. Is that as good as Shared?

~creek23

The array Bar() is so dynamic and it is not in the stack.

The declaration:
Extern Bar() As Foo
is only compatible with dynamic array!

[fxm]

I guess it's a dead end for me here unless I write a better algorithm for my program. :(

Maybe not. You might just need to change how you define the array initially. For example:

Code: Select all

' The memory for the array is allocated from
' the stack, but is too large for the stack
' You could use this if you increase the stack size
Dim As Foo array(1 To 52517)

' The memory is allocated from the system memory (RAM)
' but is also visible from inside subs and functions
Dim Shared As Foo array(1 To 52517)

' The memory is allocated from the system memory (RAM)
' and is not visible from inside subs and functions
' The array is also resizeable
ReDim As Foo array(1 To 52517)

' This causes a syntax error
Common As Foo array(1 To 52517)

' This also causes a syntax error, but only because of
' the String in Foo. Because Constructors/Destructors
' are not allowed in common arrays. If the String is
' removed then this works like the normal ReDim, but
' is accessable from any module that includes it
Common As Foo array()
ReDim As Foo array(1 To 52517) ' Only include in 1 module

' This causes a syntax error
Extern As Foo array()
Dim As Foo array(1 To 52517)

' This works like the ReDim Shared, but is accessable
' from any module that includes the Extern statement
' And unlike Common you can also use Strings,
' Constructors and Destructors
Extern As Foo array()
ReDim As Foo array(1 To 52517) ' Only include in 1 module

Additional syntax:

Code: Select all

' The memory is allocated from the system memory (RAM)
' and is not visible from inside subs and functions
' The array is not resizeable
Static As Foo array(1 To 52517)

Remark:
Static Shared ...
is equivalent to
Dim Shared ...