Hey all,
My program works with massive arrays (8GB, etc) and a large part of it needs to be filled with just one integer value (language information, Laplace smoothing). So I was wondering if it's possible to initialize the arrays elements at a certain integer value other than 0 because filling an array of that size does take allot of time.
Thanks
How initialize a data memory area as quickly as possible?
How initialize a data memory area as quickly as possible?
Last edited by Provoni on Sep 04, 2015 10:51, edited 1 time in total.
Re: Is it possible to initialize array elements with a value
Arrays are Arrays (small, medium, large) no difference in handling. See manual:
http://www.freebasic.net/wiki/wikka.php ... roPgArrays
@Admin, could this be moved to Beginners, please
http://www.freebasic.net/wiki/wikka.php ... roPgArrays
@Admin, could this be moved to Beginners, please
Re: Is it possible to initialize array elements with a value
I think your question is more general:
How initialize a data memory area as quickly as possible?
Presently there are keywords ('memset' in C and 'clear' in FB) which allow to initialize a memory block, but working only at the byte level.
In your case, you seem to want to work at the integer level.
Some speed magnitudes on my laptop:
- a 'memset'/'clear' keyword (working at the byte level) => 0.5 s / GB
- a 'for' loop (working at the integer level) => 1 s / GB
How initialize a data memory area as quickly as possible?
Presently there are keywords ('memset' in C and 'clear' in FB) which allow to initialize a memory block, but working only at the byte level.
In your case, you seem to want to work at the integer level.
Some speed magnitudes on my laptop:
- a 'memset'/'clear' keyword (working at the byte level) => 0.5 s / GB
- a 'for' loop (working at the integer level) => 1 s / GB
Re: Is it possible to initialize array elements with a value
Depends on the implementation of memset. Memset gets the number of elements upfront, and a sane implementation might do something like
if bytes>threshold
if bytes>threshold
- fill n bytes to align using the simple approach
- now aligned, fill a sse register with n* the value
- store sse register y/n times several times unrolled.
- do rest non aligned bytes.
- do simple loop always
Re: Is it possible to initialize array elements with a value
It is a many dimensions byte array. Here is an example, I tried to use the Clear keyword.
Doesn't seem to work: error 24: Invalid data types in 'dim as byte ptr p=array(0,0,0,0,0)
It needs to be a 1 dimension array?
Code: Select all
screenres(500,500,32)
redim as byte array(25,25,25,25,25)
dim as byte ptr p=array(0,0,0,0,0)
clear p[1],1,11881376
print array(1,2,3,4,5)
sleep
It needs to be a 1 dimension array?
Re: Is it possible to initialize array elements with a value
Since a Pointer always has "the address" of something ...Provoni wrote:dim as byte ptr p=array(0,0,0,0,0)
Code: Select all
dim as byte ptr p=@array(0,0,0,0,0)Re: Is it possible to initialize array elements with a value
Great, thanks allot!MrSwiss wrote:Since a Pointer always has "the address" of something ...Provoni wrote:dim as byte ptr p=array(0,0,0,0,0)Code: Select all
dim as byte ptr p=@array(0,0,0,0,0)
Re: How initialize a data memory area as quickly as possible
Code: Select all
redim as byte array(25,25,25,25,25)
clear array(0,0,0,0,0), 1, @array(25,25,25,25,25)-@array(0,0,0,0,0)+1
print array(1,2,3,4,5)
sleep
Code: Select all
redim as byte array(25,25,25,25,25)
dim as byte ptr p0 = @array(0,0,0,0,0), p1 = @array(25,25,25,25,25)
clear *p0, 1, p1-p0+1
print array(1,2,3,4,5)
sleep
Re: How initialize a data memory area as quickly as possible
I tested only the CRT memset, rep stosq, and clear. Running on my Windows 8.1, 4GB laptop, after my initial test with a ~3.6GB array the system required ~5 minutes to recover from heavy use of the disk (swap file or whatever it's currently called), so I settled on an array size of ~1.6GB, and the system was apparently able to keep the entire array in physical memory.
I think a desktop system with plenty of memory and a fast processor could do much better.
Running on my 3.00GHz Core i3-G3220, 8GB desktop system under Windows 7-64:
3.868519031325774 GB/s
19.98207180510517 GB/s
17.07506332007391 GB/s
Unfortunately, I didn't think to try running the first test twice.
Code: Select all
#include "crt.bi"
#define DIMSIZE 70
redim as byte array(DIMSIZE-1,DIMSIZE-1,DIMSIZE-1,DIMSIZE-1,DIMSIZE-1)
dim as byte ptr p = @array(0,0,0,0,0)
dim as double t1,t2
dim as longint size
dim as integer i1,i2,i3,i4,i5
size = DIMSIZE^5
print size;" bytes"
print
sleep 5000
t1 = timer
memset( p, 3, DIMSIZE * DIMSIZE * DIMSIZE * DIMSIZE * DIMSIZE )
t2 = timer
print size / 1e9 / (t2-t1); " GB/s"
print
for i as integer = 1 to 100
i1 = int(rnd * DIMSIZE)
i2 = int(rnd * DIMSIZE)
i3 = int(rnd * DIMSIZE)
i4 = int(rnd * DIMSIZE)
i5 = int(rnd * DIMSIZE)
print array(i1,i2,i3,i4,i5),
next
print
sleep
t1 = timer
asm
push rdi
mov rdi, p
mov rax, 0x0505050505050505
mov rcx, DIMSIZE * DIMSIZE * DIMSIZE * DIMSIZE * DIMSIZE / 8
rep stosq
pop rdi
end asm
t2 = timer
print size / 1e9 / (t2-t1); " GB/s"
print
for i as integer = 1 to 100
i1 = int(rnd * DIMSIZE)
i2 = int(rnd * DIMSIZE)
i3 = int(rnd * DIMSIZE)
i4 = int(rnd * DIMSIZE)
i5 = int(rnd * DIMSIZE)
print array(i1,i2,i3,i4,i5),
next
print
sleep
t1 = timer
clear( *p, 7, size )
t2 = timer
print size / 1e9 / (t2-t1); " GB/s"
print
for i as integer = 1 to 100
i1 = int(rnd * DIMSIZE)
i2 = int(rnd * DIMSIZE)
i3 = int(rnd * DIMSIZE)
i4 = int(rnd * DIMSIZE)
i5 = int(rnd * DIMSIZE)
print array(i1,i2,i3,i4,i5),
next
print
sleep
Code: Select all
1680700000 bytes
1.359164907053308 GB/s
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
3 3 3 3 3
2.413042594217742 GB/s
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
5 5 5 5 5
2.483118697953229 GB/s
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
7 7 7 7 7
Running on my 3.00GHz Core i3-G3220, 8GB desktop system under Windows 7-64:
3.868519031325774 GB/s
19.98207180510517 GB/s
17.07506332007391 GB/s
Unfortunately, I didn't think to try running the first test twice.
Last edited by MichaelW on Sep 10, 2015 4:00, edited 2 times in total.
Re: How initialize a data memory area as quickly as possible
On my PC, the first test is always measured the slower, regardless of the order of tests!
Perhaps because of the cache?
Perhaps because of the cache?
Re: How initialize a data memory area as quickly as possible
@MichaelW,
a really nice piece of code. However, you've used what I call "The Disc-Manufacturers Cheat" to calculate Bytes:
1e9
instead of
1024e6
which, should actually be used ... to obtain correct results.
a really nice piece of code. However, you've used what I call "The Disc-Manufacturers Cheat" to calculate Bytes:
1e9
instead of
1024e6
which, should actually be used ... to obtain correct results.
Re: How initialize a data memory area as quickly as possible
Sadly, it depends:
Hard drives are advertised in Gigabytes being 10^9 bytes.
But operating systems measure Gigabytes using 2^30 bytes (no 1024*10^6).
Hard drives are advertised in Gigabytes being 10^9 bytes.
But operating systems measure Gigabytes using 2^30 bytes (no 1024*10^6).
Re: How initialize a data memory area as quickly as possible
@fxm,
the really correct way is:
1024 * 1024 * 1024 = 1'073'741'824 Bytes = 1 GB, and NOT
1'000'000'000 (= Manufacturers Cheat)
the really correct way is:
1024 * 1024 * 1024 = 1'073'741'824 Bytes = 1 GB, and NOT
1'000'000'000 (= Manufacturers Cheat)
Re: How initialize a data memory area as quickly as possible
That's what I said for operating systems:
1024 * 1024 * 1024 = 2^10 * 2^10 * 2^10 = 2^30
1024 * 1024 * 1024 = 2^10 * 2^10 * 2^10 = 2^30
Re: How initialize a data memory area as quickly as possible
MichaelW
I assume that that the assembler instructions are for 64 bit.
I get a pile of errors here on 32 bit windows.
FBIDETEMP.asm: Assembler messages:
C:\Documents and Settings\USER\Desktop\FreeBASIC\FBIde0.4.6r4\FBIDETEMP.asm:222: Error: invalid use of register
C:\Documents and Settings\USER\Desktop\FreeBASIC\FBIde0.4.6r4\FBIDETEMP.asm:223: Error: operand size mismatch for `mov'
C:\Documents and Settings\USER\Desktop\FreeBASIC\FBIde0.4.6r4\FBIDETEMP.asm:224: Error: ambiguous operand size for `mov'
C:\Documents and Settings\USER\Desktop\FreeBASIC\FBIde0.4.6r4\FBIDETEMP.asm:225: Error: invalid instruction suffix for `stos'
Results:
Compilation failed
A stroke of luck that Provoni needs only byte or ubyte arrays, otherwise looping in the values would be only way I reckon!
I assume that that the assembler instructions are for 64 bit.
I get a pile of errors here on 32 bit windows.
FBIDETEMP.asm: Assembler messages:
C:\Documents and Settings\USER\Desktop\FreeBASIC\FBIde0.4.6r4\FBIDETEMP.asm:222: Error: invalid use of register
C:\Documents and Settings\USER\Desktop\FreeBASIC\FBIde0.4.6r4\FBIDETEMP.asm:223: Error: operand size mismatch for `mov'
C:\Documents and Settings\USER\Desktop\FreeBASIC\FBIde0.4.6r4\FBIDETEMP.asm:224: Error: ambiguous operand size for `mov'
C:\Documents and Settings\USER\Desktop\FreeBASIC\FBIde0.4.6r4\FBIDETEMP.asm:225: Error: invalid instruction suffix for `stos'
Results:
Compilation failed
A stroke of luck that Provoni needs only byte or ubyte arrays, otherwise looping in the values would be only way I reckon!