Multiplatform coroutines in C and also in FreeBasic

General discussion for topics related to the FreeBASIC project or its community.
exagonx
Posts: 314
Joined: Mar 20, 2009 17:03
Location: Italy
Contact:

Re: Multiplatform coroutines in C and also in FreeBasic

Post by exagonx »

SARG wrote: Nov 29, 2022 9:09
exagonx wrote: Nov 29, 2022 6:12 The usefull things are threading in DOS 16 bit
Do you mean that you want the same for 16bit programs ?
I suppose It would possible using only 16bit registers but not sure about the result and I can't test.
Your work is fantastic and believe me I envy your preparation, as a self-taught I have to settle for adapting my knowledge trying to figure out if there is a different way to do something and maybe better.

Thank you but the library written in C and compiled with djgcc works fine for me, you don't need to waste time converting it to freebasic as there is no real performance gain, also in your code many instructions are direct reverence to the register the which makes the code unstable if it has to operate with different processors.


Unfortunately this library does not manage a background thread but subs or functions where it is possible to switch without terminating the same sub or function, which do not operate in the background, instead a real multithread under 16-bit msdos would be useful, while a thread works on main other threads can display current date and time or save and read data etc. etc.
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: Multiplatform coroutines in C and also in FreeBasic

Post by caseih »

Coroutines are a bit hard to wrap one's head around. I never truly understood them or their purpose for many years. And libraries like this one are pretty abstract, so it's hard to figure out how one would use it. I get the feeling @exagonx that you maybe could use a little explanation. I know I could have used one! You seem to expect that coroutines could be a way of doing multitasking in DOS. But that's not really what they do and what they are used for.

Coroutines aren't meant to replace conventional threading. Rather they are tools for dealing with data flows (pipelines if you will). Today asynchronous programmign is all the rage and coroutines form a fundamental part of that. I'm no expert but I do use coroutines fairly regularly (in Python). Maybe some pseudo code might help explain it and how I use them. To me coroutines are most useful with what Don Knuth called the "emit operator" and also an iteration protocol, where data is emitted from one function and consumed by another. Languages that have inbuilt coroutines tend to provide this, such as C+20, C#, and Python. But we can use some kind of queue class instead to provide the data channel.

Here's a bit of generic psuedocode (not specific or even necessarily valid to libco) that hopefully makes sense:

Code: Select all

dim queue as Queue 'shared queue structure used between the coroutines
dim done_flag 'a way to indicate that the coroutine is finished emitting data

function producer(...., byref queue, byref done_flag)
	while some long-running loop
		read in some data
		put it in the queue
		co_switch(consumer_handle)
	wend
	
	done_flag = true
end function

function consumer(..., byref queue, byref done_flag)
	while not all data processed:
		if there's something in the queue then
			remove data from queue and process it
			co_switch(producer_handle)
		end if
	wend
end function
I've seen a lot of coroutine implementations in C that use static variables to maintain state in a function instead of using a library like libco. the problem there is that the functions are not reentrant and cannot be used in more than one place at a time. At least here in the pseudo code, as long as each producer/consumer pair had it's own queue and done flag, you can run as a many of these as you want, even in multiple threads.

To me this method of doing something that produces and consumes data has several advantages to other methods of doing this. First it avoids the hassles and overhead of threads if you were to make the producer it's own thread and the consumer it's own thread. They'd just be spinning waiting for each other. Since running of the two functions is linked via the queue anyway, threads would just be overkill. Second, it allows you to separate the producer code from the consumer code, making it easier to refactor or add new layers of functionality to the consumer without complicating the producer side. Or you could replace the producer with something else entirely, perhaps reading from a network socket instead of a file.

I'm not sure any of that made anything clearer...
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: Multiplatform coroutines in C and also in FreeBasic

Post by caseih »

EDIT: I only partially understand coroutines and only use a subset of them. Would love to see a real-world example of full coroutines.
Last edited by caseih on Nov 30, 2022 15:37, edited 1 time in total.
angros47
Posts: 2321
Joined: Jun 21, 2005 19:04

Re: Multiplatform coroutines in C and also in FreeBasic

Post by angros47 »

exagonx wrote: Nov 29, 2022 13:13 Unfortunately this library does not manage a background thread but subs or functions where it is possible to switch without terminating the same sub or function, which do not operate in the background, instead a real multithread under 16-bit msdos would be useful, while a thread works on main other threads can display current date and time or save and read data etc. etc.
Free basic supports threads under DOS, but only in 32 bits mode. It cannot produce 16 bit code (not even single threaded). If you need 16 bit code you need another compiler (like Power basic for DOS).
If you want to use Free basic for DOS you can create threads, and have then running in the background, exactly like you would do on Windows and Linux.
exagonx
Posts: 314
Joined: Mar 20, 2009 17:03
Location: Italy
Contact:

Re: Multiplatform coroutines in C and also in FreeBasic

Post by exagonx »

angros47 wrote: Nov 30, 2022 11:39
Free basic supports threads under DOS, but only in 32 bits mode. It cannot produce 16 bit code (not even single threaded). If you need 16 bit code you need another compiler (like Power basic for DOS).
If you want to use Free basic for DOS you can create threads, and have then running in the background, exactly like you would do on Windows and Linux.
FreeBASIC for DOS doesnt support Thread:

https://www.freebasic.net/wiki/CatPgThreading
angros47
Posts: 2321
Joined: Jun 21, 2005 19:04

Re: Multiplatform coroutines in C and also in FreeBasic

Post by angros47 »

The documentation is not updated, it does (I know because I personally added the code).

Try compiling a thread example
exagonx
Posts: 314
Joined: Mar 20, 2009 17:03
Location: Italy
Contact:

Re: Multiplatform coroutines in C and also in FreeBasic

Post by exagonx »

angros47 wrote: Dec 01, 2022 1:31 The documentation is not updated, it does (I know because I personally added the code).

Try compiling a thread example
You're right, I trusted the documentation and I wasn't aware that in the latest version it was possible to use the thread function even in 16bit DOS.

I just compiled a small program that uses Threads and it worked great, this allows me to create hadlers in the interfaces and improve my software making them more performing.

I am truly grateful for your intervention.
angros47
Posts: 2321
Joined: Jun 21, 2005 19:04

Re: Multiplatform coroutines in C and also in FreeBasic

Post by angros47 »

You are welcome!
Please, could you stop calling it "16 bit DOS"? It is actually 32 bit DOS, not 16 bit. In fact it requires a DPMI DOS extender to work, and at least a 386 processor.
marcov
Posts: 3455
Joined: Jun 16, 2005 9:45
Location: Netherlands
Contact:

Re: Multiplatform coroutines in C and also in FreeBasic

Post by marcov »

angros47 wrote: Nov 30, 2022 11:39 If you want to use Free basic for DOS you can create threads, and have then running in the background, exactly like you would do on Windows and Linux.
Except that Dos will only use one core and no HT ?
exagonx
Posts: 314
Joined: Mar 20, 2009 17:03
Location: Italy
Contact:

Re: Multiplatform coroutines in C and also in FreeBasic

Post by exagonx »

angros47 wrote: Dec 01, 2022 8:50 You are welcome!
Please, could you stop calling it "16 bit DOS"? It is actually 32 bit DOS, not 16 bit. In fact it requires a DPMI DOS extender to work, and at least a 386 processor.
The version of DOS I'm using is MS DOS 6.22 and it's natively 16bit although it uses EMM386 to support extended memory and CWSDPMI to run protected mode programs that use 32bit instructions in a 16Bit environment but the executables still remain 16bit natively.

Surely there is a misunderstanding due to the fact that you refer to the executable as if it were 32bit while I distinguish between 16Bit Binary and 32Bit Instruction, The Binary file (executable) is 16 bit while the instructions that go to create in memory are 32Bit for this requires a protected mode using a DPMI extender.
If the binary was compiled in 32Bit MSDOS 6.22 would not be able to interpret it much less execute it, as it is not able to execute the 32 Bit executables of Windows NT

This is why I continue to define it as a 16BIT environment even if the instructions loaded in memory are at 32

Anyway I will stop call 16 bit
exagonx
Posts: 314
Joined: Mar 20, 2009 17:03
Location: Italy
Contact:

Re: Multiplatform coroutines in C and also in FreeBasic

Post by exagonx »

marcov wrote: Dec 01, 2022 12:02 Except that Dos will only use one core and no HT ?
Hyper Threading is a feature of the new generation CPUs of Doubling the number of executable threads for each Core this technology is interpreted as a second CPU or Core but in reality it is not.

While for MultiThread in the programming field we mean the possibility of simultaneously executing several routines that operate in complete autonomy, for example:
You can write a routine that prints you the current time in the top right corner every second, a routine that writes saving all data in memory updating the file to disk every minute, and a routine that reads data from keyboard and mouse performing one more routine without ever stopping the others.
marcov
Posts: 3455
Joined: Jun 16, 2005 9:45
Location: Netherlands
Contact:

Re: Multiplatform coroutines in C and also in FreeBasic

Post by marcov »

exagonx wrote: Dec 01, 2022 13:38
marcov wrote: Dec 01, 2022 12:02 Except that Dos will only use one core and no HT ?
Hyper Threading is a feature of the new generation CPUs
New generation since the Pentium 4 HT 2002 ? That is twenty years on x86 alone.
exagonx wrote: Dec 01, 2022 13:38 of Doubling the number of executable threads for each Core this technology is interpreted as a second CPU or Core but in reality it is not.
HT has two effects. One is to use execution units of one core over two threads. Modern cores can execute 14+ instructions per clock, and the main running thread often doesn't use all of that parallelism, and a secondary thread could. Now that cores are getting wider again (started by the Apple M1), this might get more important again, since more CPU real estate will go unused with certain threads.

But the primary effect is simply that a hardware secondary thread allows to immediately continue if the first thread blocks, this cuts down on thread switching overhead.
While for MultiThread in the programming field we mean the possibility of simultaneously executing several routines that operate in complete autonomy, for example:
You can write a routine that prints you the current time in the top right corner every second, a routine that writes saving all data in memory updating the file to disk every minute, and a routine that reads data from keyboard and mouse performing one more routine without ever stopping the others.
Hyperthreading can technically run two threads at the same time per physical core (and some Sparcs 4 to 1 even). How efficient however that depends, and I haven't seen really detail benchmarks of that, only global ones that unfortunately measure also measure the effect that the secondary thread can be promoted to primary thread in the 100 cycles magnitude without OS interference.

I would guess that threads that use very different execution units would mix better (e.g. one thread doing primarily integer, one SSE2). The problem is that the OS doesn't have that information to effectively schedule that.

12 and 13th generation Core architectures feature hardware scheduling feedback that is primarily meant for the little/big core scheme, but could also be expanded/used to improve secondary thread scheduling based on memory access and instruction usage patterns.
marcov
Posts: 3455
Joined: Jun 16, 2005 9:45
Location: Netherlands
Contact:

Re: Multiplatform coroutines in C and also in FreeBasic

Post by marcov »

exagonx wrote: Dec 01, 2022 13:26 Surely there is a misunderstanding due to the fact that you refer to the executable as if it were 32bit while I distinguish between 16Bit Binary and 32Bit Instruction,

The Binary file (executable) is 16 bit while the instructions that go to create in memory are 32Bit for this requires a protected mode using a DPMI extender.
If the binary was compiled in 32Bit MSDOS 6.22 would not be able to interpret it much less execute it, as it is not able to execute the 32 Bit executables of Windows NT
A go32v2 (DJGPP) binary has a 16-bit small stub that initiates protected mode and then launches the 32-bit code. It is a bit like a packed EXE, stub and payload. But that is something else then simply using a few 32-bit instructions with db 66; prefix in a otherwise 16-bit exe, the go32v2 payload is a fully 32-bit protected mode application.
This is why I continue to define it as a 16BIT environment even if the instructions loaded in memory are at 32
Well, Dos 5+ itself contains 386+ code in among others memory management. And 16-bits is also divided into categories like protected vs realmode. It is not as clear cut "pure"16-bit as you make it out to be, specially in later versions
exagonx
Posts: 314
Joined: Mar 20, 2009 17:03
Location: Italy
Contact:

Re: Multiplatform coroutines in C and also in FreeBasic

Post by exagonx »

marcov wrote: Dec 01, 2022 14:29
Well, Dos 5+ itself contains 386+ code in among others memory management. And 16-bits is also divided into categories like protected vs realmode. It is not as clear cut "pure"16-bit as you make it out to be, specially in later versions
Thanks for adding so much detail,

For the rest, when I refer to a 16-bit executable it means that the interpreter executes it in 16-bit real mode, the fact that it then loads 32-bit data into memory in protected mode this in fact does not make it a 32-bit NT executable or Linux
angros47
Posts: 2321
Joined: Jun 21, 2005 19:04

Re: Multiplatform coroutines in C and also in FreeBasic

Post by angros47 »

There is no interpreter, it is compiled code.

Anyway, years ago I worked on another Basic compiler that produced assembly code compiled with NASM. Under DOS it produced a binary file that didn't work, it needed to be load in memory by a specific loader, so it was necessary to run another program called StubIt to add a loader at the beginning. And that loader was the only 16 bit part of the whole program
Post Reply