Problem writing 8GB array to disk

General FreeBASIC programming questions.
cbruce
Posts: 163
Joined: Sep 12, 2007 19:13
Location: Dallas, Texas

Problem writing 8GB array to disk

Post by cbruce »

.
I can write elements of the 8GB array all over the 8GB file using put #... but I can't put # the array itself... program just hangs.

All of my offsets and sizes look good to me. I can't figure out the issue.

The code comments have all of the details in this example... (compiles).

Code: Select all

' ================================================================
' TEST.bas
' Console program
' Running on Windows 10
' NOTE:
'   - This requires FreeBASIC 64-BIT due to the 8GB array size.
'   - !! == AND SOME RAM == !!
'   - I have 32GB RAM - that's why I'm taking advantage of FB's 
'     large array capability.
'   - I don't know how well it would run if it hit the Windows 
'     page file hard.
' ================================================================
    '
    ' - Make a hits() 4G array of USHORT elements - (8GB array).
    ' - Update the hits() array and then put # the array to disk...
    ' - Program freezes.
    ' - I can put # individual elements all over the 8GB file space 
    '   with no problem.
    ' - It's just when I try to put # the 8GB array that it freezes.
    '
    ' 4294967295 ushort elements = 8GB array
    redim shared hits( 0 to ((2^32)-1) ) as ushort
    '
    print
    print "Bounds of hits()        = "; lbound(hits); " to "; ubound(hits)
    print "Size of hits() elements = "; sizeof(hits(1))
    print
    '
    dim fbin as long
    '
    ' Stick a few values throughout the hits() array...
    hits(1)          = 1
    hits(7)          = 2
    hits(100)        = 3
    hits(1427429324) = 4
    hits(3324216828) = 5
    '
    ' Set initial length of file by writing to MAX ushort position.
    ' This works with no problem.
    ' Note:
    '   - Watch disk activity... I don't know if put # is initializing 
    '     the 8GB file by writing all zeroes to it... or if Windows is
    '     doing it (???). But the disk is getting pounded as the file 
    '     is being created.
    fbin = freefile
    open "C:\beh_TEMP\TEST-8GB-ARRAY.dat" for binary as #fbin
        dim fbin_len as longint = ((2^32)*2)   '8GB file.
        dim fbin_value as ushort = &hffff
        '
        print date; "_"; time;" ... Setting max length of file - start ..."
        put #fbin, fbin_len, fbin_value
        print date; "_"; time;" ... Setting max length of file - done  ..."
        print
    close #fbin
    '
    ' Pause until the 8GB file is done being created (watch Task Manager).
    dim gk as long
    gk = getkey
    '
    ' Write hits() array to file.
    ' Note:
    '   - There is no disk i/o when the put # statement is reached.
    '   - At the put #, the program just sits there. There is no more 
    '     disk i/o... it doesn't get to the next print statement... it 
    '     doesn't abend or blow up... ... it will just sit there forever.
    '   - After I break out of the program and hex view the file... 
    '     that iniitial value I wrote at the end of the file is there, 
    '     but none of the values in hits() are there - (as expected, 
    '     since the program never did a close file).
    fbin = freefile
    open "C:\beh_TEMP\TEST-8GB-ARRAY.dat" for binary as #fbin
        print date; "_"; time;" ... Writing hits() to file - start ..."
        put #fbin, 1, hits()
        print date; "_"; time;" ... Writing hits() to file - done  ..."
        print
    close #fbin
    '
END
Thanks!
Bruce
cbruce
Posts: 163
Joined: Sep 12, 2007 19:13
Location: Dallas, Texas

Re: Problem writing 8GB array to disk

Post by cbruce »

.
Hmmm... Don't know if it's a bug, a limit, (or a feature [smile])...

It looks like I can work with an 8GB array all day long... but the biggest array that I can write to disk, appears to be a 2GB array.

I'm on Windows 10 and writing a binary file... so that seems like a strange limit to have on writing an array to a file. ???
cbruce
Posts: 163
Joined: Sep 12, 2007 19:13
Location: Dallas, Texas

Re: Problem writing 8GB array to disk

Post by cbruce »

.
Tried a UNION, because then I could just write out four 2GB arrays instead of the 8GB array...

Code: Select all

    union hits_union
        dim x8G(0 to (2^32))     as ushort
        dim x2G(1 to 4, 0 to (2^30)) as ushort
    end union
    redim shared hits as hits_union
... but get "error 50: Array too big in 'dim x8G(0 to (2^32)) as ushort" errors with a DIM
... and can't use REDIM because no variable arrays are allowed in UNIONs.
fxm
Moderator
Posts: 12083
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Problem writing 8GB array to disk

Post by fxm »

The data elements of a Union occupy a common space in memory. I suppose that this not that you want.
Use a type instead:

Code: Select all

    type hits_union
        redim x8G(0 to (2^32))     as ushort
        redim x2G(1 to 4, 0 to (2^30)) as ushort
    end type
    dim shared hits as hits_union
fxm
Moderator
Posts: 12083
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Problem writing 8GB array to disk

Post by fxm »

Obviously I suppose you disk is well formatted in NTFS (no in FAT)?
counting_pine
Site Admin
Posts: 6323
Joined: Jul 05, 2005 17:32
Location: Manchester, Lancs

Re: Problem writing 8GB array to disk

Post by counting_pine »

I suspect there may be a (signed?) 32-bit overflow somewhere, if 2GB is not working.

Here's a function that writes a pointer in 1GB chunks.

Code: Select all

function putptr(p as any ptr, size as longint, count as longint, handle as integer, position as longint = 0) as integer
    if count <= 0 or size <= 0 then return 0
    dim as longint length = count * size

    if position > 0 then seek #handle, position

    '' write in 1GB chunks
    const CHUNKSIZE as longint = 1024^3
    while length >= CHUNKSIZE
        if put(#handle, , *cast(ubyte ptr, p), CHUNKSIZE) then return 1
        p += CHUNKSIZE: length -= CHUNKSIZE
    wend

    if length >= 0 then
        if put(#handle, , *cast(ubyte ptr, p), length) then return 1
    end if

    return 0

end function

var handle = freefile()
dim as string s = "Hello world"

open "putptr.txt" for binary as #handle

print putptr(strptr(s), sizeof(s[0]), len(s), handle)

close #handle
I don't have enough RAM, but I've tested it using smaller chunk sizes, and it seems to work fine.
fxm
Moderator
Posts: 12083
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Problem writing 8GB array to disk

Post by fxm »

counting_pine wrote:I suspect there may be a (signed?) 32-bit overflow somewhere
in the compiled code?
cbruce
Posts: 163
Joined: Sep 12, 2007 19:13
Location: Dallas, Texas

Re: Problem writing 8GB array to disk

Post by cbruce »

.
@counting_pine:
Way cool function! That's what I was trying to figure out how to do next... (and a BIG thanks, because I can see that I was headed in the wrong direction!).

And I can write a 2GB array... just not any larger.

@fxm:
TYPE... no. I was trying a UNION so that I could get at the 8GB array in 2GB chunks. I am NTFS and I can read and write TB size files with no problem at all... I just can not write an array bigger than 2GB out to a file.

I appreciate all of the help VERY much!
Bruce
counting_pine
Site Admin
Posts: 6323
Joined: Jul 05, 2005 17:32
Location: Manchester, Lancs

Re: Problem writing 8GB array to disk

Post by counting_pine »

Hmm, looks like it might be a bug not in FB itself, but in Windows fwrite():
https://stackoverflow.com/questions/186 ... 9#18694929
https://brucebcampbell.wordpress.com/20 ... ite-limit/

We use it in https://github.com/freebasic/fbc/blob/2 ... rite.c#L19, we may have to think about "chunking" the output, pretty much as I have above.

Similarly with reading (https://github.com/freebasic/fbc/blob/2 ... read.c#L31). Has anyone tried Get with large arrays?
cbruce
Posts: 163
Joined: Sep 12, 2007 19:13
Location: Dallas, Texas

Re: Problem writing 8GB array to disk

Post by cbruce »

That pegged it @counting_pine. I can write a 4GB array max:

Code: Select all

' 4GB WORKS !!! (just nothing bigger!)
redim shared hits( 0 to ((2^31)-2) ) as ushort

' I had tried with ((2^31)-1) max elements previously - but had forgotten that I needed 
' to account for the fact that my elements were 2 bytes each.
cbruce
Posts: 163
Joined: Sep 12, 2007 19:13
Location: Dallas, Texas

Re: Problem writing 8GB array to disk

Post by cbruce »

.
@counting_pine... Yes... I can GET an 8GB array with no problem.

Bruce
fxm
Moderator
Posts: 12083
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Problem writing 8GB array to disk

Post by fxm »

2 Giga = 2*(2^30) = 2^31
Therefore, 2 Giga elements corresponds to an index from 0 up to (2^31)-1.

redim shared hits( 0 to ((2^31)-1) ) as ushort
declares an array of 2 Giga ushort, so 4 GB.
cbruce
Posts: 163
Joined: Sep 12, 2007 19:13
Location: Dallas, Texas

Re: Problem writing 8GB array to disk

Post by cbruce »

@fxm... That was my original calculation also. But ((2^31)-2) works and ((2^31)-1) does not. So I just figured that my math was wrong.
fxm
Moderator
Posts: 12083
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Problem writing 8GB array to disk

Post by fxm »

About a {For...Next] loop:

Code: Select all

For iterator [ As datatype ] = startvalue To endvalue [ Step stepvalue ]
  [ statement block ]
Next [ iterator ]
The criterion to exit the loop is:
iterator > endvalue
so the final value of the iterator is greater than endvalue.

Example with endvalue = (2^31)-1 (and stepvalue = 1):
(for fbc 64-bit only)

Code: Select all

Dim As Integer n

For n = 0 To (2^31)-1
Next n
Print Hex(n)

Sleep
80000000
cbruce
Posts: 163
Joined: Sep 12, 2007 19:13
Location: Dallas, Texas

Re: Problem writing 8GB array to disk

Post by cbruce »

.
@fxm... I totally agree with you.

But, to quote Han Solo in Star Wars, "It's ... Not ... My ... Fault!" [SMILE]

Either Windows or FreeBASIC will not PUT a 4GB(?) array DIM'd as:

Code: Select all

redim shared hits( 0 to ((2^31)-1) ) as ushort
But I can PUT a 4GB(?) array DIM'd:

Code: Select all

redim shared hits( 0 to ((2^31)-2) ) as ushort
Post Reply