console/terminal on program exit

Linux specific questions.
speedfixer
Posts: 606
Joined: Nov 28, 2012 1:27
Location: CA, USA moving to WA, USA
Contact:

console/terminal on program exit

Post by speedfixer »

I have several programs - most fairly complex - that leave the console 'unclean' on exit.

Typically, the command line does not do a newline on a simple <enter> keypress. The returning prompt just flows behind the previous.
Often, I will lose all screen output.

The shell is still working, as a command typed, or scroll-back to previous, will work - usually with correct output.
This can happen whether or not I use an fbgfx command or not. Most of what I do does not use graphics.
Simple clear in terminal does a clear, but does not fix broken control.

My system:
Lubuntu 14.04 x86_64
AMD fx 6150
nvidia GTX 560 Ti

FB: 1,05.0 10-15-2015 64bit linux

(I have a LOT of other systems. Some with more of everyting, some with less.)

This has been true for several versions of FB and linux.
I have many more systems here at home, and will be host on them, or putty to them.
Same thing. putty or console. I tried to use 'screen 0' - no difference, or, sometimes it would show up where it had not been before.
(Why does 'end' not call any destructor left over, or whatever it is the comments suggest on the wiki?)

NOW ...
From another forum thread far, far away - for a completely unrelated issue - I caught trying to send 'kill xxx $PPID' from a shell script. Let me try ...

From inside the program, I do:

open pipe "kill $PPID" for input as filnum1


A 'terminated' message appears - and all is well. Console/putty clean !!
Host or remote.

I do use threads sometimes, but I also check that my programs terminate cleanly and leave nothing behind, no leaked mem, no extra process hanging around. A program does not have to have graphics or threads to do this.
Any suggestion or idea what is going on?

I can post code, if someone wants - but it will still be 600 lines after I clean it up, I expect. And a person would have to alter it for their system to have it work. (It queries my personal DNS server, so would have to be adjusted for your system.)

david
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: console/terminal on program exit

Post by caseih »

Can you post a small example program that replicates the issue, say 20-30 lines of code? You're correct that a 600-line example is less than useful.

Another debugging tack is to redirect all output to a file (if it's non-interactive). Theoretically that should prevent the terminal from going weird. Then you can examine the text file, perhaps a bit at a time, and see what in the output is doing this to the terminal.

FB's console routines do use ncurses, so it could be something with that that's messing things up.

Also, I'm sure you're aware that when the terminal goes into a broken state, you can run the reset command (actually a program) that will reset the terminal to a good state. I often find if I cat a non-text file, the non-ascii bytes will trigger various modes in the terminal emulator and mess things up like you describe. Before I was told about the reset command I used to have to just close the terminal and open a new one when that happened!
speedfixer
Posts: 606
Joined: Nov 28, 2012 1:27
Location: CA, USA moving to WA, USA
Contact:

Re: console/terminal on program exit

Post by speedfixer »

Thank you for reading my not-small post.

Actually, my last research suggested that FB only uses the terminfo of ncurses.

Some of my programs do not have console output.
Some of them already have wholly redirected output: I wrote alternate print, clear, color, locate, sleep, etc. routines that I can switch on or off - or reroute to an alternate terminal/net/file/etc. with a single #define.

If I could isolate a smaller than 100 lines of code that causes the problem, I would post it.

As I said - sometimes I am on putty to another system, sometimes I am on the local console - lxterminal, usually. Not always.
I will test the fail from a windows Xming session and see what happens. I seem to recall it also happens from there.

Let me see how small I can get a guaranteed-fail-program. May be a day or two.
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: console/terminal on program exit

Post by caseih »

Yes if you can do a standalone example, I'm willing to try it on my machine.

When you say you can redirect your output to an alternate console using a #define, what do you mean? Usually a program can only output the tty it's attached to, or redirect that output to a pipe or file.
speedfixer
Posts: 606
Joined: Nov 28, 2012 1:27
Location: CA, USA moving to WA, USA
Contact:

Re: console/terminal on program exit

Post by speedfixer »

The program I will post doesn't use my redirection.
However, assume I have:

#define _redir_net_

at the top of whatever I compile.

I can have a command - druck, for example, that should print - - somewhere.
_redir_net_ can cause a change in the library path to link to a library set that sends that command - druck - to a network destination.

or _redir_msg_ can cause the same keyword to use a different library to be used that will send a druck(print) to a file - or console - or whatever.

One define at the top - I currently have 5 destinations for the output. Or maybe no output (_redir_none_).
I have precompiled libraries for 9 keywords - 5 flavors of libraries and destinations. One single define at top. Fast compiles and links.


I will prep my program - strip what I can - verify it still fails and post it.

david
speedfixer
Posts: 606
Joined: Nov 28, 2012 1:27
Location: CA, USA moving to WA, USA
Contact:

Re: console/terminal on program exit - inventory network

Post by speedfixer »

Inventory Network
481 lines - wrote this 2+ years ago as learning tool

Code: Select all

' showsys.bas

' show the current systems up (per ping reply)

type rm_local_environ
    sysname as string           ' system name
    ip as string                ' ip of this system
    sysrole as long
    sysndx as long
end type

#include "file.bi"
#include "string.bi"
#include "datetime.bi"

declare function rmp_bytein(indata as ulong, dinfo as ubyte, bndx as integer) as ulong
declare function rmp_byteout(indata as ulong, bndx as integer) as ulong
declare function rmp_fnqueryip overload (qpaddr as string) as string
declare function rmp_fnqueryip(addrip as integer) as string

declare sub rmp_getsubnet(netip as string)
declare function rmp_pingsub as integer
declare sub rmp_netstatsetup

declare sub rm_homeid
declare sub rmp_scanhome(subnet as string = "")

declare sub rmp_pthread(paddr as any ptr)
declare sub rmp_putpingtable(mark as integer, index as integer)
declare sub killping

dim shared as rm_local_environ myhome       ' loaded from sys.id except on system start
dim shared as rm_local_environ item(0 to 256)
dim shared as string slash
dim shared as string tila, sqte

dim shared as string rmsg_netstat

dim shared as string idfilemc    ' master control ID file
dim shared as string idfileproc  ' running processes: PID, name_of_proc, UID
dim shared as string homepath    ' path of 'home' directory
dim shared as string homeurl     ' home network
dim shared as string localnet
dim shared as string sourceurl

' this only used by net stuff ---------------------------------------------------------------------------------------------------
dim shared as any pointer mtx_putpingtable

mtx_putpingtable = mutexcreate()

dim shared pingtable() as integer
redim pingtable(0 to 256) as integer

rm_homeid

dim as integer filnum1

print : print
print "      Systems that do not answer a ping request"
print "               may not be revealed."
print

rmp_getsubnet(myhome.ip)
print "            subnet scanned: "; localnet
print

dim as integer result

result = rmp_pingsub                     ' ping and build pingtable
rmp_scanhome

print "       scanned from "; myhome.sysname; " = "; myhome.ip

rmp_netstatsetup
print

dim as string siii

for xixi as integer  = 1 to 256

    siii = right("    " + trim(str(xixi)), 4)

    if item(xixi).sysname <> "" and item(xixi).sysrole = 0 then
        print tab(4);
        print siii; " ";
        print tab(20);
        color 12, 0
        print item(xixi).sysname
        color 15, 0
    end if

    if item(xixi).sysrole <> 0 then
        print tab(4);
        print siii; tab(20);
        color 10, 0
        print item(xixi).sysname,
        color 15, 0
        print item(xixi).ip;
        if item(xixi).sysname = myhome.sysname then
            color 14, 0
            print "  -  ME!"
            color 15, 0
        else
            print
        end if
    end if
    if xixi = 256 then
        print
        print "       total up systems: "; item(xixi).sysndx
    end if
next

print
killping

sleep
'========= uncomment these three lines and it works fine ===============
'filnum1 = freefile
'open pipe "kill $PPID" for input as filnum1
'close

' ------------------------- subs only below ----------------------------------
function rmp_pingsub as integer
' make-run ping thread

dim as integer mark
dim as string purl, tstamp

dim as any ptr pad()
redim as any ptr pad(0 to 255)

for i as integer = 1 to 254
    pad(i) = threadcreate(@rmp_pthread, CPtr(Any Ptr, i))
    if pad(i) = 0 Then
        print : print
        print "Error creating thread:"
        print i
        print : sleep : end
    end If
    sleep 10,1
next

for i as integer = 1 to 254
         if pad(i) <> 0 Then        ' pause until all threads complete
            threadwait(pad(i))
         end If
next

for i as integer = 1 to 254                 ' count the answers
    if pingtable(i) = 0 then mark += 1
next
item(256).sysndx = mark     ' total of all answers

return mark
end function                                                    ' rmp_pingsub

sub killping
' stop all pingsub processes, clean up leftovers

mutexdestroy(mtx_putpingtable)
end sub

sub rmp_getsubnet(netip as string)
' extract subnet from full tcpip - return in shared localnet
dim as integer mark1, mark2, mark3
dim as string period
period = chr(46)

mark1 = instr(netip, period)
mark2 = instr(mark1 + 2, netip, period)
mark3 = instr(mark2 + 2, netip, period)
if mark1 + mark2 + mark3 < 12 then
 print :  print
 print "Bad home ip - check system name or dns server - cannot continue."
 print
 print netip
 print : sleep : end
end if

localnet = left(netip,mark3 - 1)

end sub                                                       ' rmp_getsubnet

sub rmp_scanhome(subnet as string = "")
' scan ping results, attempt to mount anyone home - make rm_sysrcv
' check for name, master, etc
' read/verify pingdone(pingflag) - erase pingdone after process so no confusion

dim as string scanurl, newrmtdir, query, newdirresult
dim as integer filnum, result, newdirstat, mylip

mylip = val(right(myhome.ip, (len(myhome.ip) - instrrev(myhome.ip, "."))))

scanurl = subnet
if subnet = "" then
    scanurl = localnet
end if
if scanurl = "" then scanurl = homeurl

for i as integer = 1 to 254     ' exclude 0 and 255
    if i = mylip then  ' don't include yourself
        rmp_putpingtable(28672,i)                                         ' = home   -- sysrole 7
    end if                                                               ' exclude yourself
next

'            newrmtdir = remotedir + "/rmt" + format(i, "000")
for i as integer = 1 to 254
    if pingtable(i) = 0 then                                ' got ping answer
        item(i).sysrole = 1
    end if  'not local
    item(i).sysname = rmp_fnqueryip(i)
next

pingtable(255) += 1 ' add the local

end sub                                                        ' rmp_scanhome

function rm_queryurl(surl as string) as string
' if input is a url, not ip, see if it is in local dns
' input name, output ip string

dim as string query, reply
dim as integer filnum, openerr

if surl = "" then return ""

query = "dig barnes " + surl + " +short"    ' need to add domain to remove
        ' spurious replies from nethost search engine

filnum = freefile
open pipe query for input as #filnum
    openerr = err
    line input #filnum, reply
close #filnum

if openerr <> 0 then reply = "no name"

return reply
end function                                                    ' rm_queryurl

function rmp_fnqueryip(addrip as integer) as string
' ip in, name out
' this = full name - other first name only

dim as integer mark
dim as string query, sname, addxx, ipaddr

addxx = trim(str(addrip))
ipaddr = homeurl + "." + addxx

query = "dig -x " + ipaddr + " +short"

mark = freefile
open pipe query for input as #mark
    line input #mark, sname
close #mark

mark = instr(sname,".")
if mark <> 0 and sname <> "" then
    sname = left(sname, mark - 1)
end if

return sname
end function                                                   ' rmp_fnqueryip

function rmp_fnqueryip(qpaddr as string) as string
' ip in, name out
' this = full name - other first name only

dim as integer mark
dim as string query, sname, ipaddr
ipaddr = qpaddr

if instr(ipaddr,".") = 0 then      ' not full ip
    ipaddr = homeurl + "." + ipaddr
end if

query = "dig -x " + ipaddr + " +short"

mark = freefile
open pipe query for input as #mark
    line input #mark, sname
close #mark

mark = instr(sname,".")
if mark <> 0 and sname <> "" then
    sname = left(sname, mark-1)
end if

return sname
end function                                                   ' rmp_fnqueryip

function rmp_queryip(ipaddr as string) as string
' get the system name - strip something????

dim as string sname
dim as integer mark

sname = rmp_fnqueryip(ipaddr)        ' name returns with 'domain' (.barnes in this case)
mark = instr(sname,".")
if mark <> 0 and sname <> "" then
    sname = left(sname, mark - 1)
end if

return sname
end function                                                     ' rmp_queryip

sub rmp_putpingtable(mark as integer, index as integer)
' put mark into pingtable()

mutexlock mtx_putpingtable
pingtable(index) = mark
mutexunlock mtx_putpingtable

end sub                                                         ' rm_puttable

function rmp_makeurl(term as integer, howlong as integer = 2) as string
' return full ip address from net item
dim as string ppurl, murl

ppurl = localnet + "." + trim(str(term))
murl = "ping " + ppurl + " -c 1 -q -W "

murl = murl + trim(str(howlong))
murl = murl + " > /dev/null"

return murl
end function                                                    ' rmp_makeurl

sub rmp_pthread(paddr as any ptr)
' put result of ping into table
dim as integer stopcount

dim as string ppurl, murl
dim as integer mark, perr, padr

padr = cint(paddr)
murl = rmp_makeurl(padr)
mark = shell (murl)
perr = err

rmp_putpingtable(mark, padr)     ' ------------------------------------ write pingtable---------

end sub                                                          ' rm_pthread

function rmp_urlint(turl as string) as ulong
' return int of url

dim as ubyte url1, url2, url3, url4
dim as string tempdata
dim as ulong fulldata

tempdata = turl
url4 = val(right(tempdata,len(tempdata)-(instrrev(tempdata,"."))))
tempdata = left(tempdata,instrrev(tempdata,".")-1)
url3 = val(right(tempdata,len(tempdata)-(instrrev(tempdata,"."))))
tempdata = left(tempdata,instrrev(tempdata,".")-1)
url2 =  val(right(tempdata,len(tempdata)-(instrrev(tempdata,"."))))
tempdata = left(tempdata,instrrev(tempdata,".")-1)
url1 =  val(tempdata)

fulldata = rmp_bytein(fulldata, url4, 4)
fulldata = rmp_bytein(fulldata, url3, 3)
fulldata = rmp_bytein(fulldata, url2, 2)
fulldata = rmp_bytein(fulldata, url1, 1)

return fulldata
end function                                                     ' rmp_urlint

function rmp_inturl(tdatain as ulong) as string
' return url string
dim as string turl

turl = str(rmp_byteout(tdatain,1))
turl = turl + "." + str(rmp_byteout(tdatain,2))
turl = turl + "." + str(rmp_byteout(tdatain,3))
turl = turl + "." + str(rmp_byteout(tdatain,4))
return turl
end function                                                     ' rmp_inturl

function rmp_bytein(indata as ulong, dinfo as ubyte, bndx as integer) as ulong
' mask indata into dinfo as bndx byte - not sure how I came up with this - replace with shfts
' bytes count from most to least: 1, 2, 3, 4 ... 4x7 = least significant bit
dim as ulong tempdat, tempvar, tempval

if bndx < 1 or bndx > 4 then
    print "fail in bytein: index out of bounds: "
    print bndx
    sleep :  end
end if

tempval = dinfo

if bndx = 4 then tempvar = 4294967040
if bndx = 3 then
    tempvar = 4294902015
    tempval shl= 8
end if

if bndx = 2 then
    tempvar = 4278255615
    tempval shl= 16
end if

if bndx = 1 then
    tempvar = 16777215
    tempval shl= 24
end if

tempdat = indata and tempvar
tempdat = tempdat or tempval

return tempdat

end function                                                      ' rm_bytein

function rmp_byteout(indata as ulong, bndx as integer) as ulong
' parse out desired byte of data
if bndx < 1 or bndx > 4 then
    print "fail in byteout: index out of bounds: "
    print bndx
    print : sleep : end
end if

if bndx = 1 then return hibyte(hiword(indata))
if bndx = 2 then return lobyte(hiword(indata))
if bndx = 3 then return hibyte(loword(indata))
if bndx = 4 then return lobyte(loword(indata))

return 0
end function                                                     ' rm_byteout

sub rmp_netstatsetup
' this will leave a data structure on (virtual) disk to track other systems
' this is a multistep process: pingsub -> scanhome -> here

dim as ubyte iphome

item(0).ip = myhome.ip                  ' keep my ip in (0) ---------- why???
item(255).sysndx = pingtable(255)       ' all rdkl systems: after dirmnt

for i as integer = 1 to 254     ' process pingable values
    if pingtable(i) <> 1 then           ' 1 = down
        if pingtable(i) = 28672 then    ' include me to be complete
            item(i).sysrole = 1
            item(i).sysname = myhome.sysname
            continue for
        end if

        if pingtable(i) = 2 then item(i).sysrole = 0     ' not rdkl
        if pingtable(i) = 3 then item(i).sysrole = 1       ' up - rdkl

        if pingtable(i) > 3 then           ' don't think we can do this here
            close : print
            print "out-of-bounds in ::netstatsetup"
            print
            sleep
            end
        end if
    end if                                  ' not down
next                                        ' all in pingtable

end sub                                                    ' rmp_netstatsetup

sub rm_homeid
' make sys.id - each new master or startup - and only a master - of local system will come here

dim as integer idfile, result
dim as string mark, abcabc

idfile = freefile                               ' get my system name
open pipe "uname -n" for input as #idfile
line input #idfile, myhome.sysname
close #idfile

myhome.ip = rm_queryurl(myhome.sysname)         ' get my ip as string
result = instrrev(myhome.ip, ".")
homeurl = left(myhome.ip, result - 1)

end sub                                                           ' rm_homeid
htop only shows the program being killed when I enable the last lines - one process killed only.
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: console/terminal on program exit

Post by caseih »

I had to hard-code my ip address in there, but the program runs without problems. I'm not sure what you mean about the lines you un-comment. Does the problem happen with or without these lines active? I saw no problems either way.

Killing one's own PID is really weird. I wouldn't recommend doing this in general! I wouldn't be surprised if it did weird things when things go down uncleanly.

I'm not sure why I had to hard code my IP address... my machine's name is definitely in the DNS, both forward and reverse.
speedfixer
Posts: 606
Joined: Nov 28, 2012 1:27
Location: CA, USA moving to WA, USA
Contact:

Re: console/terminal on program exit

Post by speedfixer »

Forgot.

Line 227 - dig command - you may have to adjust that for whatever DNS you use. You may get replies from DNS servers that can't see inside your subnet, and so will give you bad/wrong replies.

If you don't have any other systems on your subnet, then you should only get your own system listed.
If you do have systems, but no local DNS, I think you will still get those that reply to the ping.

By the way, this is an excellent example where and why threads DO save time.
I wrote and first used this on my Pentium 4 systems - at least 5 were single core. (All replaced now.)

Runs in just a few seconds. Then and now.
One core - 254 threads. 2 seconds timeout for each failing ping thread. Does NOT take 500 seconds.
So - listen with a little skepticism when they all tell you threading without multiple cores doesn't help.
99% depends upon the task, not the processor.

david
speedfixer
Posts: 606
Joined: Nov 28, 2012 1:27
Location: CA, USA moving to WA, USA
Contact:

Re: console/terminal on program exit

Post by speedfixer »

For me, the problem happens without the last lines.

Do you get the names of all your systems listed, even the ones not up now?

The hard coding was probably because you did not get a good reply from the DNS server.
marcov
Posts: 3455
Joined: Jun 16, 2005 9:45
Location: Netherlands
Contact:

Re: console/terminal on program exit

Post by marcov »

I'm no FB wizard, but do a tcgetaddr on startup, and a tcsetaddr (with the record gotten from tcgetaddr) in the on halt/stop handler.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: console/terminal on program exit

Post by fxm »

Is your program "normally" ends when you do not use threads?

Code: Select all

for i as integer = 1 to 254
'    pad(i) = threadcreate(@rmp_pthread, CPtr(Any Ptr, i))
    rmp_pthread(CPtr(Any Ptr, i))
'    if pad(i) = 0 Then
'        print : print
'        print "Error creating thread:"
'        print i
'        print : sleep : end
'    end If
    sleep 10,1
next

'for i as integer = 1 to 254
'         if pad(i) <> 0 Then        ' pause until all threads complete
'            threadwait(pad(i))
'         end If
'next
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: console/terminal on program exit

Post by caseih »

speedfixer wrote:For me, the problem happens without the last lines.

Do you get the names of all your systems listed, even the ones not up now?

The hard coding was probably because you did not get a good reply from the DNS server.
Correct. I get both red and green hosts, and it runs fine with or without the kill stuff.
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: console/terminal on program exit

Post by caseih »

In your threads are you printing to the screen from inside of the multiple threads? Or do the threads do the work and then when they are finished you print everything?

As far as threads are concerned, single-threaded, asynchronous I/O would probably be just as fast in a case like this, provided you had an asynchronous mechanism in FB for doing the pings. Anytime an operation is I/O bound, such as most network operations, asynchronous I/O can be simpler than threading and scale far better. But it requires a whole different mindset to use. It's similar to GUI programming in some ways as your code becomes event driven.
speedfixer
Posts: 606
Joined: Nov 28, 2012 1:27
Location: CA, USA moving to WA, USA
Contact:

Re: console/terminal on program exit

Post by speedfixer »

@marcov

tcgetaddr?

I did not know what that is - but now I do. The problem isn't getting any address, changing or using the terminal in some special way, or my use of the kill $PPID. That was simply a lucky find that will clear the console/terminal for me. The question is: why does my screen mess up in the first place? (I use crt/pid/ppid in other programs. Familiar with them.) I don't want to, and should not have to, create special terminal I/O. If I am using vanilla FB commands, FB should not allow my terminal to get messed up. If I create my own control code script or send random characters to the screen - I understand. I am not doing anything special with this routine.

@fxm

I believe it does. It takes so long, that is why I went to threads.
But I will try that later.

@caseih

Nothing goes to the screen from a thread. Single threading: no, not in a case such as this. Each thread is concurrently timing out, not sequentially. It DOES take 2 to 4 minutes if they aren't threaded. Since it is the OS and hardened services that I work with, I don't worry about collisions or corruptions. The OS requests never fail.

Speculation:

Some forum threads indicate that FB has trouble with the multiple communication pipes to talk to the system/console/terminal from FB. Could it be that not carefully catching the OS replies to my requests could cause overflows in these pipes and cause these problems?
There are certainly multiple pipes that FB opens during output.

For example, if I open a graphic screen and do nearly the same thing, SOME - not all - of the OS replies can NOT be trapped by FB. Some will still echo to the text console, even with the graphics screen open. (The error output, perhaps? Maybe I will redirect that as a test.)

(That would not explain my seeing the 'Terminated' message after my kill command since I can see no hung process with htop, and the process count doesn't change when that kill happens. I watched that many times to be sure.)

david
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: console/terminal on program exit

Post by fxm »

speedfixer wrote:@fxm

I believe it does. It takes so long, that is why I went to threads.
But I will try that later.
  • - In fact, i am not sure that 'Shell' is thread-safe.
    A compromise could be to keep the multi-threading, but to modify the [Mutexlock...Mutexunlock] block to encompass also 'Shell':

    Code: Select all

    sub rmp_putpingtable(mark as integer, index as integer)
    ' put mark into pingtable()
    
    'mutexlock mtx_putpingtable
    pingtable(index) = mark
    'mutexunlock mtx_putpingtable
    
    end sub                                                         ' rm_puttable
    

    Code: Select all

    sub rmp_pthread(paddr as any ptr)
    ' put result of ping into table
    dim as integer stopcount
    
    dim as string ppurl, murl
    dim as integer mark, perr, padr
    
    padr = cint(paddr)
    murl = rmp_makeurl(padr)
    mutexlock mtx_putpingtable
    mark = shell (murl)
    perr = err
    
    rmp_putpingtable(mark, padr)     ' ------------------------------------ write pingtable---------
    mutexunlock mtx_putpingtable
    
    end sub                                                          ' rm_pthread
    
    - Why 'sleep 10,1' inside the [For...Next] block to create the threads?
    (to workaround a malfunctioning?)
Post Reply