console/terminal on program exit

Linux specific questions.
dkl
Site Admin
Posts: 3235
Joined: Jul 28, 2005 14:45
Location: Germany

Re: console/terminal on program exit

Post by dkl »

Hi,

I've just looked at shell() again, and the terminal state handling part of it is not thread safe (in the Unix version of FB).

The FB runtime changes some terminal settings (e.g. it disables echoing), and it restores the original state on exit (otherwise the terminal would be messed up). If the terminal ends up messed up, then something went wrong (e.g. the program was just killed/crashed and didn't have a chance to clean up anymore).

The problem with things like shell() and dylibload() in this regard is that they have to temporarily reset the terminal state to the original one, to allow the loaded program to see it, in case it's an FB program that runs longer than the "main" FB runtime instance. Because whichever FB instance terminates last is the one that has to restore the original terminal state. However, shell()'s sequence of "restore state, run command, save & modify state" is not atomic with regards to threads. So there's a race condition there, and I think that could explain the issues you're seeing.

Honestly this is one of the more nasty parts of the FB runtime. I'm afraid that it might not even be safe with regards to running multiple FB processes in parallel... but it seems like it's needed to implement FB's console I/O functions.

I think making shell() thread safe in this regard means guarding all of it with a lock. But that means that shell() commands could never run in parallel anymore, which is probably also an issue.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: console/terminal on program exit

Post by fxm »

dkl wrote:I've just looked at shell() again, and the terminal state handling part of it is not thread safe (in the Unix version of FB).
Does this mean that 'Shell' is thread-safe in the Windows version of FB!!!
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: console/terminal on program exit

Post by caseih »

speedfixer wrote:(That would not explain my seeing the 'Terminated' message after my kill command since I can see no hung process with htop, and the process count doesn't change when that kill happens. I watched that many times to be sure.)
I must confess I'm completely confused by your using shell to send a kill signal. The "Terminated" message you are seeing comes from the shell itself I think. So there's nothing much to explain there. When you send the kill signal to an app, it typically is terminated. But what I don't understand is why you thought to do this is in the first place!
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: console/terminal on program exit

Post by caseih »

dkl wrote:I think making shell() thread safe in this regard means guarding all of it with a lock. But that means that shell() commands could never run in parallel anymore, which is probably also an issue.
EDIT: Nevermind my comment. Definitely shell() from within the thread is probably contributing to his problem. Does open pipe have the same issues with the tty? I wouldn't think it would.
Last edited by caseih on Jan 19, 2016 2:55, edited 1 time in total.
speedfixer
Posts: 606
Joined: Nov 28, 2012 1:27
Location: CA, USA moving to WA, USA
Contact:

Re: console/terminal on program exit

Post by speedfixer »

Why 'sleep 10,1' inside the [For...Next] block to create the threads?
(to workaround a malfunctioning?)
I'm asking the system to do something, and should give it a timeslice to let it do the job. If I hog the cpu, it is an invitation to disaster.

Sleep gets inconsistent below about 8 on the systems that I tested it with. So - 10.


In another test I did with starting and controlling threads, I created 1000 threads that depended upon a different matching thread to finish before each one could begin. So - 2000 threads all waiting for me to say 'go'. (I went up to 20000 - just to see what would happen.)
With timing much below 10, the thread creation loop fails - sooner or later. I think more because the system doesn't have time and FB can't catch a reply (the address of the thread, at least), more so than there is a specific fault in any routine. At this many threads - the most important thing I learned was that the system 'reserves' memory for each thread. Only virtual memory. The threads run - all is ok. BUT - the system has to release all that virtual memory. The threads run quick and complete and terminate - but it takes forever to release the virtual memory. You can watch that reservation count down from over 26 G !! in the main before it can close - long after the threads have all disappeared. At first, I thought the system was hanging up. I like htop.

david
speedfixer
Posts: 606
Joined: Nov 28, 2012 1:27
Location: CA, USA moving to WA, USA
Contact:

Re: console/terminal on program exit

Post by speedfixer »

@fxm: remark out thread use

1 - The results take over twice as long. 29 seconds as opposed to 13 seconds - HOWEVER:
2 - I loose the ping replies. I only get the DNS replies. Everything looks like it is online. (Also means I get a 254 long listing that includes all of the blanks.)

BUT - the screen does return normally.

I fixed the missing parts to allow the program to do what it should: ping, etc. All results correct. 492 seconds - the reason for threads.

Screen is normal.

Threads?

I will try encapsulating the OS queries.
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: console/terminal on program exit

Post by caseih »

Speedfixer, check out this C library:

http://noping.cc/

If you can make a wrapper for it you should be able to do all your pings programmatically inside your threads, which should eliminate the shell() problem. oPing is GPLv2, though, which should be fine for your own purposes, but if you ever intend to sell your product, and want to keep your source code proprietary, you cannot use it. But I'm sure there are other, similar libraries out there.
speedfixer
Posts: 606
Joined: Nov 28, 2012 1:27
Location: CA, USA moving to WA, USA
Contact:

Re: console/terminal on program exit

Post by speedfixer »

@casieh: re: oping

Thank you, I will keep it in mind. What I am writing will never see daylight, though, so licenses aren't an issue.

@dkl

Does that mean that launching something with OPEN PIPE will have the same problems, or does it stand alone??

I do some very complex stuff with OPEN PIPE.

Also, am I hearing that any parallel FB programs could be an issue, or just any that are started from the same, single console/terminal instance?
If I were to use the crt libraries directly, does FB notice and do anything in regards to managing the terminal interfaces?

David
speedfixer
Posts: 606
Joined: Nov 28, 2012 1:27
Location: CA, USA moving to WA, USA
Contact:

Re: console/terminal on program exit

Post by speedfixer »

FYI: for anyone looking at this:

a mutex guard at the OS query line did not change anything
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: console/terminal on program exit

Post by fxm »

speedfixer wrote:FYI: for anyone looking at this:

a mutex guard at the OS query line did not change anything
The mutex is destroyed insde killping() which is called before the creation of thread!!!
You must destroy the mutex at least after the waiting for all thread endings.

Remark: mutex use inside sub rmp_putpingtable() is useless because each thread has its own array element.
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: console/terminal on program exit

Post by caseih »

Speedfixer, anytime you use threads you have to be extremely careful what you call and make sure everything you use is known to be thread safe. As fxm says, parts of the runtime library are not thread safe. In other languages like C one also has to be careful. Many parts of the glibc runtime library are thread safe, some are not.

Threaded programming is very hard to get right, especially when you're unfamiliar with synchronization primitives and how and when to use them. Threads lead to weird bugs like you are seeing. And often such bugs disappear entirely in a debugger, as the debugger changes the timing and hides race conditions and other subtle problems.

Under the hood, open pipe uses popen() from glibc, which from what I've read seems to be thread safe (posix claims it should be anyway). So theoretically it should be okay to use, but I can't speak to whether the FB runtime code that actually calls popen is thread safe. I've looked at the runtime source code and there are some locks FB itself uses with regards to runtime error handling. But as near as I can tell open pipe should be thread safe. And it's a much better thing to use than shell() as it doesn't need to mess with your current TTY session.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: console/terminal on program exit

Post by fxm »

Using mutex is simple enough as long as we use it to its first goal which is only mutual exclusion.
If away from its primary goal, it is diverted to also serve for real synchronization, or if conditional variables are used, this gets more complicated.

See at How Manage a Critical Section of code of a Thread in FB.
speedfixer
Posts: 606
Joined: Nov 28, 2012 1:27
Location: CA, USA moving to WA, USA
Contact:

Re: console/terminal on program exit

Post by speedfixer »

@fxm:

The mutex is killed at the end of the program. What are you talking about?

re: mutex at each element: I see that, you are right.
I have read, a couple of times, all of the material you have referenced about threads. It has been helpful, and has altered how I use them now. But threads are like a hammer, or a screwdriver, or a shovel: each is just a tool. There are better ways to use them, ideal purposes for each, and specialized tools for just that one special task. But they are tools, nonetheless. Sometimes, a hammer is just a heavy weight to whack on something. It works and is the best tool for the job, but not the designed purpose for that hammer. Or I could use another language that has 35 different types of hammers. I can't afford 35 hammers. Takes too long to learn which is which, and I don't expect to live that long.

@caseih: re: debugging

I have about 200 commands grouped into 13 library categories in my personal libraries. One script generates both single and multithreaded versions of each to a static lib for each 'command'. Takes about a minute. An alternate script generates the same libs, but all with debugging. Another script recompiles all my applications based on those libs, keeps my apps small, and compile time WAY down for each new app. Again, with or without debugging. I wrote my own build system to do this. I do regularly test and retest to know that everything works. This console fail was annoying. It was a coincidence that I stumbled upon the kill command as I was writing up the original post in this thread.

I stepped away from programming for about 25 years. The world is a little more complex now than it was then. At least I can still learn. I was NEVER afraid to ask questions.


Anyone:

So the fact that my routine takes 10 to 13 seconds, and without it takes almost 500 seconds, is not a good enough reason to use threads? This was true on my single core systems before I replaced them, also. You all need to rethink what threads are and what you were taught. Kinda like the teaching on the GOTO statement: I agree with Linus - NOT using it is impossible - you just don't know you are using it if you think you aren't.

I'm not always right, but I learned a long time ago that our teachers are not always right either. If you only do what is taught, there will be no progress.
I want the 480 seconds. It works. And I now have a fixed command addition to my personal libraries that terms my programs with the kill statement.

I can accept that FB still has problems with threads. The kill $PPID corrects the terminal status - for me - seemingly without repercussions as far as I can tell. (It is now 'kill -15 $PPID' as I have read more about kill in linux.) This works local, remote, and from windows into linux (I only program in linux.) It is an FB problem, not how I use the language. Avoiding the problem doesn't make it go away. Terminal and console routines in FB are regulary rewritten and adjusted as problems surface. I read DEEP into the forums before I post a question. And I make simple stupid mistakes, too, like anyone else.

This particular bit of code was an old learning exercise. The result of it is just a tiny part of a much, much larger system. I asked questions, we all thought about this a little, we all have learned something. You have helped me. I'm happy. Thank you.


david
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: console/terminal on program exit

Post by fxm »

speedfixer wrote:@fxm:

The mutex is killed at the end of the program. What are you talking about?
Excuse me, I read a little too fast the code, and as there is no right indent when entering the procedures.....
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: console/terminal on program exit

Post by fxm »

I can not help you more because I don't work in Linux!
Post Reply