How initialize a data memory area as quickly as possible?

General FreeBASIC programming questions.
Provoni
Posts: 347
Joined: Jan 05, 2014 12:33
Location: Belgium

How initialize a data memory area as quickly as possible?

Postby Provoni » Sep 04, 2015 6:25

Hey all,

My program works with massive arrays (8GB, etc) and a large part of it needs to be filled with just one integer value (language information, Laplace smoothing). So I was wondering if it's possible to initialize the arrays elements at a certain integer value other than 0 because filling an array of that size does take allot of time.

Thanks
Last edited by Provoni on Sep 04, 2015 10:51, edited 1 time in total.
MrSwiss
Posts: 3359
Joined: Jun 02, 2013 9:27
Location: Switzerland

Re: Is it possible to initialize array elements with a value

Postby MrSwiss » Sep 04, 2015 7:57

Arrays are Arrays (small, medium, large) no difference in handling. See manual:
http://www.freebasic.net/wiki/wikka.php?wakka=ProPgArrays

@Admin, could this be moved to Beginners, please
fxm
Posts: 9528
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: Is it possible to initialize array elements with a value

Postby fxm » Sep 04, 2015 8:59

I think your question is more general:
How initialize a data memory area as quickly as possible?

Presently there are keywords ('memset' in C and 'clear' in FB) which allow to initialize a memory block, but working only at the byte level.
In your case, you seem to want to work at the integer level.

Some speed magnitudes on my laptop:
- a 'memset'/'clear' keyword (working at the byte level) => 0.5 s / GB
- a 'for' loop (working at the integer level) => 1 s / GB
marcov
Posts: 2839
Joined: Jun 16, 2005 9:45
Location: Eindhoven, NL
Contact:

Re: Is it possible to initialize array elements with a value

Postby marcov » Sep 04, 2015 9:51

Depends on the implementation of memset. Memset gets the number of elements upfront, and a sane implementation might do something like

if bytes>threshold
  • fill n bytes to align using the simple approach
  • now aligned, fill a sse register with n* the value
  • store sse register y/n times several times unrolled.
  • do rest non aligned bytes.
else
  • do simple loop always
Provoni
Posts: 347
Joined: Jan 05, 2014 12:33
Location: Belgium

Re: Is it possible to initialize array elements with a value

Postby Provoni » Sep 04, 2015 10:25

It is a many dimensions byte array. Here is an example, I tried to use the Clear keyword.

Code: Select all

screenres(500,500,32)

redim as byte array(25,25,25,25,25)

dim as byte ptr p=array(0,0,0,0,0)

clear p[1],1,11881376

print array(1,2,3,4,5)

sleep

Doesn't seem to work: error 24: Invalid data types in 'dim as byte ptr p=array(0,0,0,0,0)

It needs to be a 1 dimension array?
MrSwiss
Posts: 3359
Joined: Jun 02, 2013 9:27
Location: Switzerland

Re: Is it possible to initialize array elements with a value

Postby MrSwiss » Sep 04, 2015 10:36

Provoni wrote:dim as byte ptr p=array(0,0,0,0,0)

Since a Pointer always has "the address" of something ...

Code: Select all

dim as byte ptr p=@array(0,0,0,0,0)
Provoni
Posts: 347
Joined: Jan 05, 2014 12:33
Location: Belgium

Re: Is it possible to initialize array elements with a value

Postby Provoni » Sep 04, 2015 10:53

MrSwiss wrote:
Provoni wrote:dim as byte ptr p=array(0,0,0,0,0)

Since a Pointer always has "the address" of something ...

Code: Select all

dim as byte ptr p=@array(0,0,0,0,0)

Great, thanks allot!
fxm
Posts: 9528
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: How initialize a data memory area as quickly as possible

Postby fxm » Sep 04, 2015 11:53

Code: Select all

redim as byte array(25,25,25,25,25)

clear array(0,0,0,0,0), 1, @array(25,25,25,25,25)-@array(0,0,0,0,0)+1
print array(1,2,3,4,5)

sleep
Or

Code: Select all

redim as byte array(25,25,25,25,25)

dim as byte ptr p0 = @array(0,0,0,0,0), p1 = @array(25,25,25,25,25)
clear *p0, 1, p1-p0+1
print array(1,2,3,4,5)

sleep
MichaelW
Posts: 3500
Joined: May 16, 2006 22:34
Location: USA

Re: How initialize a data memory area as quickly as possible

Postby MichaelW » Sep 06, 2015 7:55

I tested only the CRT memset, rep stosq, and clear. Running on my Windows 8.1, 4GB laptop, after my initial test with a ~3.6GB array the system required ~5 minutes to recover from heavy use of the disk (swap file or whatever it's currently called), so I settled on an array size of ~1.6GB, and the system was apparently able to keep the entire array in physical memory.

Code: Select all

#include "crt.bi"

#define DIMSIZE 70

redim as byte array(DIMSIZE-1,DIMSIZE-1,DIMSIZE-1,DIMSIZE-1,DIMSIZE-1)

dim as byte ptr p = @array(0,0,0,0,0)

dim as double t1,t2
dim as longint size
dim as integer i1,i2,i3,i4,i5

size = DIMSIZE^5
print size;" bytes"
print

sleep 5000

t1 = timer
memset( p, 3, DIMSIZE * DIMSIZE * DIMSIZE * DIMSIZE * DIMSIZE )
t2 = timer
print size / 1e9 / (t2-t1); " GB/s"
print

for i as integer = 1 to 100
    i1 = int(rnd * DIMSIZE)
    i2 = int(rnd * DIMSIZE)
    i3 = int(rnd * DIMSIZE)
    i4 = int(rnd * DIMSIZE)
    i5 = int(rnd * DIMSIZE)
    print array(i1,i2,i3,i4,i5),
next
print

sleep

t1 = timer
asm
    push    rdi
    mov     rdi, p
    mov     rax, 0x0505050505050505
    mov     rcx, DIMSIZE * DIMSIZE * DIMSIZE * DIMSIZE * DIMSIZE / 8
    rep     stosq
    pop     rdi   
end asm
t2 = timer
print size / 1e9 / (t2-t1); " GB/s"
print

for i as integer = 1 to 100
    i1 = int(rnd * DIMSIZE)
    i2 = int(rnd * DIMSIZE)
    i3 = int(rnd * DIMSIZE)
    i4 = int(rnd * DIMSIZE)
    i5 = int(rnd * DIMSIZE)
    print array(i1,i2,i3,i4,i5),
next
print

sleep

t1 = timer
clear( *p, 7, size )
t2 = timer
print size / 1e9 / (t2-t1); " GB/s"
print

for i as integer = 1 to 100
    i1 = int(rnd * DIMSIZE)
    i2 = int(rnd * DIMSIZE)
    i3 = int(rnd * DIMSIZE)
    i4 = int(rnd * DIMSIZE)
    i5 = int(rnd * DIMSIZE)
    print array(i1,i2,i3,i4,i5),
next
print

sleep

Code: Select all

 1680700000 bytes

 1.359164907053308 GB/s

 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3
 3             3             3             3             3

 2.413042594217742 GB/s

 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5
 5             5             5             5             5

 2.483118697953229 GB/s

 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7
 7             7             7             7             7

I think a desktop system with plenty of memory and a fast processor could do much better.

Running on my 3.00GHz Core i3-G3220, 8GB desktop system under Windows 7-64:

3.868519031325774 GB/s

19.98207180510517 GB/s

17.07506332007391 GB/s

Unfortunately, I didn't think to try running the first test twice.
Last edited by MichaelW on Sep 10, 2015 4:00, edited 2 times in total.
fxm
Posts: 9528
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: How initialize a data memory area as quickly as possible

Postby fxm » Sep 06, 2015 8:49

On my PC, the first test is always measured the slower, regardless of the order of tests!
Perhaps because of the cache?
MrSwiss
Posts: 3359
Joined: Jun 02, 2013 9:27
Location: Switzerland

Re: How initialize a data memory area as quickly as possible

Postby MrSwiss » Sep 06, 2015 9:37

@MichaelW,

a really nice piece of code. However, you've used what I call "The Disc-Manufacturers Cheat" to calculate Bytes:
1e9
instead of
1024e6
which, should actually be used ... to obtain correct results.
fxm
Posts: 9528
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: How initialize a data memory area as quickly as possible

Postby fxm » Sep 06, 2015 9:47

Sadly, it depends:
Hard drives are advertised in Gigabytes being 10^9 bytes.
But operating systems measure Gigabytes using 2^30 bytes (no 1024*10^6).
MrSwiss
Posts: 3359
Joined: Jun 02, 2013 9:27
Location: Switzerland

Re: How initialize a data memory area as quickly as possible

Postby MrSwiss » Sep 06, 2015 10:02

@fxm,

the really correct way is:
1024 * 1024 * 1024 = 1'073'741'824 Bytes = 1 GB, and NOT
1'000'000'000 (= Manufacturers Cheat)
fxm
Posts: 9528
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: How initialize a data memory area as quickly as possible

Postby fxm » Sep 06, 2015 10:17

That's what I said for operating systems:
1024 * 1024 * 1024 = 2^10 * 2^10 * 2^10 = 2^30
dodicat
Posts: 6227
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: How initialize a data memory area as quickly as possible

Postby dodicat » Sep 06, 2015 13:54

MichaelW
I assume that that the assembler instructions are for 64 bit.
I get a pile of errors here on 32 bit windows.

FBIDETEMP.asm: Assembler messages:
C:\Documents and Settings\USER\Desktop\FreeBASIC\FBIde0.4.6r4\FBIDETEMP.asm:222: Error: invalid use of register
C:\Documents and Settings\USER\Desktop\FreeBASIC\FBIde0.4.6r4\FBIDETEMP.asm:223: Error: operand size mismatch for `mov'
C:\Documents and Settings\USER\Desktop\FreeBASIC\FBIde0.4.6r4\FBIDETEMP.asm:224: Error: ambiguous operand size for `mov'
C:\Documents and Settings\USER\Desktop\FreeBASIC\FBIde0.4.6r4\FBIDETEMP.asm:225: Error: invalid instruction suffix for `stos'

Results:
Compilation failed

A stroke of luck that Provoni needs only byte or ubyte arrays, otherwise looping in the values would be only way I reckon!

Return to “General”

Who is online

Users browsing this forum: Google [Bot], paul22 and 1 guest