multithread detection

[speedfixer]

What are the triggers that cause multithreading detection and library usage?
When is this detection happening?

It seems any FB multithread keyword I use is consumed in pre-compiled object libraries for static linking. My main app code calls these pre-compiled functions of mine so the compiler now misses choosing multithreading.

How can I force multithread compiling from the code without the command line switch?

david

[jj2007]

If you don't remember whether you use multithreading, you might have lost control over your code ;-)

[Provoni]

How can I force multithread compiling from the code without the command line switch?

Is this a thing?

[caseih]

If you don't remember whether you use multithreading, you might have lost control over your code ;-)

Speedfixer certainly remembers! It's the compiler that doesn't seem to be able to tell when working with a library that requires multithreading support and thus is not initializing whatever needs to be initialized. That's the issue. How to tell FBC that it's working with a multi-threaded library that needs threading support initialized. Am I reading him correctly?

[adeyblue]

Code: Select all

Sub ForceFBMTLibrary_WithAReallyLongNameSoYoullKnowIfYouEverManageToAccidentallyCallItWhenYouShouldntEverDoSo()
threadwait(0) '' could also use threadcreate or threadcall
End Sub

[coderJeff]

What are the triggers that cause multithreading detection and library usage?

Needs to be explicit '-mt' on the command line option for at least one executable, object file, library to select the multithreading library. If '-mt' is specified, it gets saved in a special section of the compiled library.

>>When is this detection happening?
At linking time, fbc will read the special sections to determine if any compiled libraries or object files have the '-mt' option saved. If they do, multi-threaded libraries are selected. If no '-mt' was given on the command line but was saved in a linked file, a warning is shown.

>> It seems any FB multithread keyword I use is consumed in pre-compiled object libraries for static linking. My main app code calls these pre-compiled functions of mine so the compiler now misses choosing multithreading.
'-mt' only affects linking, it does not affect code generation unless there is some conditional source code on __FB_MT__

>>How can I force multithread compiling from the code without the command line switch?
Currently not possible. Command line option '-mt' needs to be explicit. Could be addressed with addition of the '#cmdline' directive, what's been on the TODO.txt list for a while.

A little more information here: OBJINFO

[speedfixer]

@caseih: you are correct

I compile each function two times: once with and once without '-mt' to create my object files for later static linking. I have one threaded, one non-threaded object directory. (Actually, several more *pairs* of directories, one pair for each different architecture: four pairs at present.) My compile script header reads the __FB_MT__ (and an architecture flag from the system environment) to determine which directory will be used for linking. All the library source and object files sit on a particular server in the garage. From any of my systems I compile an app for that local system - whatever flavor it is.

Yes: some of these libraries have #conditionals, especially related to threading.

Yes: any app can be compiled with or without the -mt command line flag.

An app will have at the top at least one lib flag file; these followed by the compile script file; then the code for this particular app.

The flags set tell the compile script file which *other* libraries must be included to support the initial included library.

My compile script header will include at least four files from each/any library:
a dim file - basically, declare/dimension the common data required for this library
a declare file - declare the functions callable in this library set
an init file - intilialize objects, structures, functions, declared data that might change at run time - not precompilable
an include file - naming the #inclib files possibly needed
(actually, an added destructor function per library is added after this; sometimes empty for simple basic libs.)

These are included in the above order and properly does not allow inclusions out-of-order.

The mutexes and conditionals are initialized in the init files.

SO ...

basically the main app code has conditionals and mutexes created at the top of the apps (init file) - that must be parsed in the FIRST processing step of compiling - long before we get to including the files that actually contain the thread routines.

Unless I missed something, a mutex only has value in a multithreaded environment.

Why isn't the app flagged as MT at this time?
If I write an small app in one file and I start a thread, I do NOT have to use the -mt switch.

fxm: the wiki is currently incorrect:

KeyPgDdfbmt states

indicates if the the multithreaded option -mt was specified on the command line at the time of compilation

At least in Linux, command line 'fbc test.bas' DOES show multithread compile:

Code: Select all

    #if __FB_MT__
        #print at start: multithreaded compile
    #endif

Sub print_dots(ByRef char As String)
    For i As Integer = 0 To 29
        Print char;
        Sleep CInt(Rnd() * 100), 1
    Next
End Sub

Sub mythread(param As Any Ptr)
    '' Work (other thread)
    print_dots("*")
End Sub

    Randomize(Timer())

    Print " main thread: ."
    Print "other thread: *"

    '' Launch another thread
    Dim As Any Ptr thread = ThreadCreate(@mythread, 0)

    '' Work (main thread)
    print_dots(".")

    '' Wait until other thread has finished, if needed
    ThreadWait(thread)
    Print
    Sleep

    #if __FB_MT__
        #print at end: multithreaded compile
    #endif

If it is too much trouble to add detection at the first step, could we at least have a compile switch, or access to flip the switch in the 'fbctinf' file from the source file?

david

[fxm]

the wiki is currently incorrect:

KeyPgDdfbmt states

indicates if the the multithreaded option -mt was specified on the command line at the time of compilation

At least in Linux, command line 'fbc test.bas' DOES show multithread compile:

Code: Select all

    #if __FB_MT__
        #print at start: multithreaded compile
    #endif

Sub print_dots(ByRef char As String)
    For i As Integer = 0 To 29
        Print char;
        Sleep CInt(Rnd() * 100), 1
    Next
End Sub

Sub mythread(param As Any Ptr)
    '' Work (other thread)
    print_dots("*")
End Sub

    Randomize(Timer())

    Print " main thread: ."
    Print "other thread: *"

    '' Launch another thread
    Dim As Any Ptr thread = ThreadCreate(@mythread, 0)

    '' Work (main thread)
    print_dots(".")

    '' Wait until other thread has finished, if needed
    ThreadWait(thread)
    Print
    Sleep

    #if __FB_MT__
        #print at end: multithreaded compile
    #endif

Perhaps this description update:

__FB_MT__ indicates if the the multithreaded option -mt was specified on the command line at the time of compilation, or whether one of the ThreadtCreate or ThreadCall keywords is used more above in the source code.

[coderJeff]

Perhaps this description update:

__FB_MT__ indicates if the the multithreaded option -mt was specified on the command line at the time of compilation, or whether one of the ThreadtCreate or ThreadCall keywords is used more above in the source code.

Yes, this is correct. I didn't notice this behaviour first time around when looking at fbc sources. As speedfixer's example points out, __FB_MT__ is only automatically set from the point of usage of ThreadCreate or ThreadCall onward.

It looks like like ThreadCreate or ThreadCall need to automatically select '-mt' because they require support that is only available in the mt runtime lib.

I suppose if you were creating threads using some other method, either platform specific or some other threading library, Mutex*, Cond*, etc could still be useful? But you would need to either manage the locks on the calls to the fb runtime yourself, or limit fb runtime calls to a single thread only.

If it is too much trouble to add detection at the first step, could we at least have a compile switch, or access to flip the switch in the 'fbctinf' file from the source file?

Detection at the first step is possible, kind of like how '#lang' is handled. #lang works by aborting the compile and restarting from the top. Gets really tricky though if there are multiple statements that can trigger a restart and there are conditionals in the source before them. '#cmdline' directive also can become a mess too if it is allowed to trigger a restart, as there could be multiple '#cmdline' statements potentially.

Feels like the way forward is to document the behaviour of '-mt' and '__FB_MT__', 'ThreadCreate', and 'ThreadCall'. Then later add the '#cmdline' directive - either no restart, or maybe one restart only allowed.

[fxm]

Feels like the way forward is to document the behaviour of '-mt' and '__FB_MT__', 'ThreadCreate', and 'ThreadCall'.

Done:
- KeyPgThreadWait → fxm [updated __FB_MT__ behavior]
- KeyPgThreadCreate → fxm [updated __FB_MT__ behavior]
- CompilerOptmt → fxm [updated __FB_MT__ behavior]
- KeyPgDdfbmt → fxm [updated __FB_MT__ behavior]

[jj2007]

Just in case the compiler has, at a certain moment, the complete source code somewhere in a buffer: on my trusty old Core i5, checking for the presence of the keyword 'ThreadCreate' in a two megabyte source takes typically less than 0.5 milliseconds (see Check a 2MB source for a keyword)

[speedfixer]

Surely, when restarting from the top, SOMETHING is changed to indicate whatever triggered the change.

If that is a one-flag-only process or anything like that, the deficiencies are obvious. I doubt that is the case.
Same with cmdline: it seems it is quirky in requiring an order of parameters at times.
I think most times FB strives to not be so complex as the big compilers.

How much fun will it be explaining/defending a one-time-restart-process policy and philosophy?

If it can NOW do multiple restarts, that should be the goal: fix that process.

Personally, I am content understanding how things really exist: I know what to expect.
Most of the changes made to FB in my time here have been appreciated. It truly has moved forward.

I know the original effort and intent was for FB be a one-pass-only processor: that guiding light has helped it be the speedy compiler that it is. I think that speed is one of the most attractive features of FB. Hopefully, in everyone's rule set, 2 of the top ten would be: keep it simple, and, if it isn't broke don't fix it. Times have changed; all parts of a computer system have gotten MUCH faster. Maybe it is time to retire that particular goal: one pass. The groundwork has been laid down - I think FB will stay fast until a LOT of things change.


(I have not looked at the doc changes. I am satisfied that any corrections have been made. I won't look until I need to: I get super picky when words get etched onto the boulder.)

david

quite late edit:

jj2007:

Just in case the compiler has, at a certain moment, the complete source code somewhere in a buffer ...

That's part of the problem: I don't make threads anymore: they are wrapped up and managed by my thread manager library. I feed it a sub; it makes the thread with pause/status/terminate controls. Or numbers of subs/threads (I've tested up to 1000 threads.) Each has its own control structure.

All my functions are already precompiled waiting to be statically linked. Same with my key input/mouse routines, screen/color control, gui controls, text mgmt, remote comm, etc. All threaded; all precompiled. Each function in a module. ONLY needed modules linked makes for really small - but big function - executeables.

I don't make mutexes or conditionals: all inside libraries, already. Accessible through the thread control system. Most of these, at least, I don't precompile: they are automatically included and initialized. But they aren't enough to trigger MT detection.

[speedfixer]

Found a way.
I use a simple define to include a tiny 'fake thread' routine.