is redim inside threads safe? ( memory leak help :O, ) (SOLVED)

[SamL]

My main loop runs threads to handle files and other data. everything needing a mutex is of course.
I'm using threaddetach() on all threads once a thread is done another thread can be created. total running threads is tracked by a shared array.

my thread has no static variables but a few redim's for single dimension, dynamic string arrays.

When the threads run my program is slowly using more and more memory.
could this be because im using redim?
also do i need to make the strings = "" before ending the thread?

when i convert the code to use a single sub instead of a thread i dont get this issue.

could this be a windows 10 memory management issue?

Thanks for the help, i have been trying forever to isolate this issue and i'm down to if i run the code in a thread it leaks if not in a thread no leak.
last week and 1/2 I have spent retesting and isolating code. easy 8+ hours a day so any tips / help would be great.
-SamL

[Xusinboy Bekchanov]

Are you not releasing something maybe?
Can you show the code?

[fxm]

I'm using threaddetach() on all threads once a thread is done another thread can be created. total running threads is tracked by a shared array.

A little short to fully understand your thread management principle.
(sample code would be welcome)

[fxm]

is redim inside threads safe?

If the critical section of the code accessing the array (Redim, read/write access, access to U/Lbound, ...) in each thread code including the main code (implicit thread) is protected by a locking/unlocking mutex block, this mutex being common to all threads (including main code), no problem IMHO.

[SamL]

OK, id love to provide the code but its large. what I'm going to do is remove all code from the program so I'm only working with the issue. then if its short enough ill post it.

There has to be something I'm over looking or i don't understand.

-SamL

[speedfixer]

Is the data in the redimmed arrays accessed outside the thread?
If the data is ONLY local to the thread function, absolutely - there is a simple code error.

If the data is supposed to be visible to any other thread or the main, that data (static declared and guarded) should be in its own function, with data referenced by a shared pointer.

On each redim, that shared pointer must immediately be reassigned to the array in that function.
If the data in the array is complex, then it could get tricky.

Either way, without some better hint of the actual code, this is about as far as anyone can GUESS how to help you.

An actual leak is interesting: some programmers get them often - some never get them.


daivd

[SamL]

I found it!!!

The issue was using the command DIR while in a thread. MutexLock does not help.

I have been searching for this leak for a long time. I never thought that there would be a command that just does not like threads.
if there is a list of any commands i should never use in a thread i would love to have it!

I replaced the DIR command with an open for input command to check if the file exists and that seems to fix my issue.

Thanks for the help. I was trying to reduce the code so I could post it but started testing random parts and came across the issue.

All the tips helped. i have been having this issue with a leak for a long time and decided to fix it once and for all. it only took me about 2 weeks. :D
I might be a little slow, but i had to write code to emulate and cancel out other code so i could narrow down where the issue was.

This is great!! now i can keep going on the project!! :D
SamL

[fxm]

It does not surprise me that 'Dir' is not thread safe.

To get a list of items in a directory, 'Dir' must be called multiple times (because returning only one item per call) in a section of code (multiple lines of code).
So to avoid any conflict with another thread using 'Dir', the entire section of code must be protected by a single lock/unlock mutex block (and not individually each instruction line using 'Dir').

[SamL]

It does not surprise me that 'Dir' is not thread safe.

here is some example code to show how to make a memory leak with 'Dir' and threads.
It is similar to my code with the issue.( except i don't open files in a do, loop like this example does)

Code: Select all

#include "fbthread.bi"
dim shared as ulong MAX_THREADS 
dim shared as ubyte wait_for_thread_ini
dim shared as string file_location
dim shared as any ptr mutex_thread_number
dim shared as any ptr mutex_dir
mutex_thread_number = mutexcreate 
mutex_dir = mutexcreate 

declare sub check_file_thread(param as any ptr)
declare function FreeThread( reset_number as ulong = 0 ) as ulong
declare function launch_thread() as byte

'NOTE: this will make a bunch of empty files in the exepath folder. no more then MAX_THREADS

'---------------------MAIN--------------------------

MAX_THREADS = 254
file_location = exepath & "\"

do 'main loop
    
    launch_thread()
    
    sleep 1,1
loop until inkey = chr(27) 'press escape key to end
'---------------------------------------------------
mutexdestroy mutex_thread_number 
mutexdestroy mutex_dir 
print "done, press a key"
sleep



sub check_file_thread(param as any ptr)
    
    dim as ulong thread_number = *cast(ulong ptr, param)
    wait_for_thread_ini = 0
    sleep 300,1 ' for testing, this allows more threads to run at the same time.
    dim as string file_ready
    dim as byte do_once = 0
    print "thread #", thread_number
    dim as ubyte ff = freefile
    dim as string file_name = file_location & str(thread_number) & ".txt"
    do
        if do_once = 1 then
            mutexLock mutex_dir
            file_ready = dir(file_name) 'DO NOT USE DIR LIKE THIS!!!! causes memory leaks!!!
            mutexUnLock mutex_dir
            if file_ready <> "" then
                exit do
            end if
        end if
        
        if do_once = 0 then
            do_once = 1
            open file_name for output as #ff
            close #ff
        end if
        sleep 1,1
    loop

    
    FreeThread(thread_number)'release freethread #
end sub

function launch_thread() as byte
    'launch a thread and wait for thread to report back that it has saved thread_number.
    
    dim as ulong thread_number 'get a freethread #
    thread_number = FreeThread() 'get a freethread #
    if thread_number > 0 then
        wait_for_thread_ini = 1
        threaddetach( threadcreate(@check_file_thread, cast(any ptr, @thread_number) ))
        while wait_for_thread_ini = 1
            sleep 1,1
        wend    
    end if
end function

function FreeThread( reset_number as ulong = 0 ) as ulong
    'manage how many threads are running by assigning or unassigning a unique numer.
    'keeps track of a number that is in use until a thread unregisters the number.
    'an unregistered number can be reused.
    'Passing 0 into function registers and returns next free number.
    'Passing a number greater then 0 will unregisters that number.
    mutexLock mutex_thread_number
    static as ulong avalable_Threads(1 to MAX_THREADS)
    select case reset_number
        case is = 0
            for i as ulong = 1 to MAX_THREADS
                if avalable_Threads(i) = 0 then 'NOT_IN_USE
                    avalable_Threads(i) = 1 'set to IN_USE
                    mutexUnLock mutex_thread_number
                    RETURN i
                end if
            next i
        case is > 0
            if reset_number <= ubound(avalable_Threads) then
                avalable_Threads(reset_number) = 0 'set to NOT_IN_USE
                mutexUnLock mutex_thread_number
                RETURN 1
            end if
    end select
    mutexUnLock mutex_thread_number
    RETURN 0
end function

[speedfixer]

This is a good catch. Almost not really intuitive.

fxm, a note in the docs would be helpful, please?

[adeyblue]

You must be using an older version of FB, I fixed Dir (and one of the INPUT statements) leaking last september. Here was my sample code showing the leak
----
Dir is one of the four explicitly thread safe things in FB. That's what caused the leak in the first place, the thread specific data was never freed if you didn't get to the end of the Dir enumeration.

[speedfixer]

Cool!

[fxm]

If 'Dir' is thread-safe (because of using TLS memory), mutex usage is useless whatever the instruction sequence calling 'Dir'.

Dir is one of the four explicitly thread safe things in FB.

Can you expand a bit?

[SamL]

Ya I'm using an older version of FreeBasic, I should have updated before posting.

FreeBASIC Compiler - Version 1.07.1 (2019-09-27), built for win64 (64bit)
Copyright (C) 2004-2019 The FreeBASIC development team.
standalone

Because its fixed in newer version of fbc would that example code be safe to use DIR like in my example? or does DIR need to be used in a special way?

SamL

[SamL]

I'm still getting a leak using DIR after updating my compiler to,

FreeBASIC Compiler - Version 1.07.3 (2020-12-31), built for win64 (64bit)
Copyright (C) 2004-2019 The FreeBASIC development team.
standalone

I changed my test code to fix an issue using freefile. Now the test code uses the thread number as a file # instead. this fixes an issue of using same file # to open files while testing.

If some one can verify this is leaking with new compiler version that would be nice i still consider my self a noob. if your not getting enough threads running then increase the thread sleep time.

For now I'll use "if open( "filename" for input as ff ) = 0 then" instead of DIR in my code to check if a file exists.

Here is the updated test code I'm using, I omitted the use of FreeFile.

Code: Select all

#include "fbthread.bi"
dim shared as ubyte MAX_THREADS 
dim shared as ubyte wait_for_thread_ini
dim shared as string file_location
dim shared as any ptr mutex_thread_number
dim shared as any ptr mutex_dir
dim shared as ulong Thread_sleep

mutex_thread_number = mutexcreate 
mutex_dir = mutexcreate 

declare sub check_file_thread(param as any ptr)
declare function FreeThread( reset_number as ulong = 0 ) as ulong
declare function launch_thread() as byte

'NOTE: this will make a bunch of empty files in the exepath folder. no more then MAX_THREADS

MAX_THREADS = 254 'using ubyte because thread numbers are also being used as for file numbers.
Thread_sleep = 300 'increasing this allows more threads to run at the same time.
file_location = exepath & "\"
'file_location = "b:\"

do 'main loop
    
    launch_thread()
    
    sleep 1,1
loop until inkey <> "" 'press any key to end
'---------------------------------------------------
mutexdestroy mutex_thread_number 
mutexdestroy mutex_dir 
print "done, press a key"
sleep


sub check_file_thread(param as any ptr)
    
    dim as ulong thread_number = *cast(ulong ptr, param)
    wait_for_thread_ini = 0
    sleep Thread_sleep,1 ' for testing, this allows more threads to run at the same time.
    dim as string file_ready
    dim as byte do_once = 0
    print "thread #", thread_number
    dim as string file_name = file_location & str(thread_number) & ".txt"
    
    open file_name for output as #thread_number
    close #thread_number
    
    do
        mutexLock mutex_dir
        file_ready = dir(file_name) 'DO NOT USE DIR LIKE THIS!!!! causes memory leaks!!!
        mutexUnLock mutex_dir
        if file_ready <> "" then
            exit do
        end if
        sleep 1,1
    loop
    FreeThread(thread_number)'release freethread #
end sub

function launch_thread() as byte
    'launch a thread and wait for thread to report back that it has saved thread_number.
    
    dim as ulong thread_number 'get a freethread #
    thread_number = FreeThread() 'get a freethread #
    if thread_number > 0 then
        wait_for_thread_ini = 1
        threaddetach( threadcreate(@check_file_thread, cast(any ptr, @thread_number) ))
        while wait_for_thread_ini = 1
            sleep 1,1
        wend    
    end if
end function

function FreeThread( reset_number as ulong = 0 ) as ulong
    'manage how many threads are running by assigning or unassigning a unique numer.
    'keeps track of a number that is in use until a thread unregisters the number.
    'an unregistered number can be reused.
    'Passing 0 into function registers and returns next free number or 0 if all the numbers are in use.
    'Passing a number greater then 0 will unregisters that number.
    mutexLock mutex_thread_number
    static as ulong avalable_Threads(1 to MAX_THREADS)
    select case reset_number
        case is = 0
            for i as ulong = 1 to MAX_THREADS
                if avalable_Threads(i) = 0 then 'NOT_IN_USE
                    avalable_Threads(i) = 1 'set to IN_USE
                    mutexUnLock mutex_thread_number
                    RETURN i
                end if
            next i
        case is > 0
            if reset_number <= ubound(avalable_Threads) then
                avalable_Threads(reset_number) = 0 'set to NOT_IN_USE
                mutexUnLock mutex_thread_number
                RETURN 1
            end if
    end select
    mutexUnLock mutex_thread_number
    RETURN 0
end function

SamL