Feature Request: Stack Trace Capability

[SoruCoder]

Specifically, make new keywords that model the existing error handling functions, maybe with "s" (plural) or "St" (Stack Trace) at the end:

• ErLs or ErLSt:

  Code: Select all

  Declare Function ErLs() As Integer Ptr '  each line in stack trace, with 0 as terminator.

• ErFNs or ErFNSt:

  Code: Select all

  Declare Function ErFNs() As ZString Ptr Ptr ' each function name in stack trace, with 0 pointer as terminator.

• ErMNs or ErMNSt:

  Code: Select all

  Declare Function ErMNs() As ZString Ptr Ptr ' each module name in stack trace, with 0 pointer as terminator.

[adeyblue]

I'm not sure how much appetite there is for this to be completely built in, (to read the debug info that has line and function info you need to cart around a separate program or a dll on Windows) but I've made a start on a lib to translate stack traces to symbols.

I can get the line and file info from dwarf and stabs debug info (the two fbc can produce) on Windows, now I've got to see how fancy the combination of backtrace and dladdr is on Linux and/or backtrace_symbols since it looks like a magic wand.

Then maybe there could be a way to only get a stacktrace for Error built-in, which would be far less hassle

[adeyblue]

As usual, I did all the work for this assuming the basic lynchpin on Windows (CaptureStackBackTrace) would just work without checking it first. And it turns out, FB must do weird things to the stack that it doesn't like. Because it doesn't

This C code compiled with Nuwen GCC 9.2 64-bit

Code: Select all

#include <stdio.h>
#include <windows.h>

int Fun2()
{
    void* array[63] = {0};
    DWORD hash = 0;
    USHORT num = CaptureStackBackTrace(0, 63, array, &hash);
    printf("Caught %hu frames from stacktrace\n", num);
    for(USHORT i = 0; i < num; ++i)
    {
        printf("%hu: %p\n", num - i, array[i]);
    }
    return num;
}

void Fun1(int val)
{
    int numFun = Fun2();
    printf("Fun2 returned %d with val %d\n", numFun, val);
}

int main()
{
    Fun1(76);
}

Produces this output, which is perfect

Caught 7 frames from stacktrace
7: 000000000040159D
6: 000000000040162D
5: 0000000000401663
4: 00000000004013B2
3: 00000000004014FB
2: 000000007719556D
1: 00000000773F372D
Fun2 returned 7 with val 76

This FB code though with the same GCC
%fbc64% -O 0 -gen gcc

Code: Select all

#include "windows.bi"

Private Function Fn2() As Long
    dim frames(0 to 60) As Any Ptr
    dim framesPtr As Any Ptr Ptr = @frames(0)
    dim hash as DWORD
    dim as Long caught = CaptureStackBackTrace(0, 61, framesPtr, @hash)
    Print Using "Caught & frames using stack capture"; caught
    For i As Long = 0 To caught - 1
        Print Using "&) &"; caught - i; Hex(frames(i))
    Next
    Return caught
End Function

Private Sub Fn1(num as ULong)
    dim as Long numFn2 = Fn2()
    Print Using "Fn2 returned & with num = &"; numFn2; num
End Sub

Fn1(87)

Only prints this

Caught 1 frames using stack capture
1) 4016D2
Fn2 returned 1 with num = 87

StackWalk64 is a little better

Code: Select all

Const as HANDLE g_hProc = Cast(HANDLE, -1)

Type ThreadParams
    dim threadId As ULong
    dim pCtx as CONTEXT ptr
End Type

Private Sub CaptureThread(p as Any Ptr)
    dim pParams as ThreadParams ptr = p
    const threadPerms as ULONG = THREAD_GET_CONTEXT Or THREAD_SUSPEND_RESUME
    dim hThread as HANDLE = OpenThread(threadPerms, FALSE, pParams->threadId)
    SuspendThread(hThread)
    GetThreadContext(hThread, pParams->pCtx)
    ResumeThread(hThread)
    CloseHandle(hThread)     
End Sub

Private Sub TryOldStackWalk()
    dim ctx as CONTEXT
    ctx.ContextFlags = CONTEXT_ALL
    dim tp as ThreadParams
    tp.threadId = GetCurrentThreadId()
    tp.pCtx = @ctx
    dim thread as Any Ptr = ThreadCreate(@CaptureThread, @tp)
    ThreadWait(thread)
    ''RtlCaptureContext(@ctx)
    Print "Stack is around " & Hex(ctx.Rbp)
    dim sf as STACKFRAME64
    sf.AddrPC.Mode = AddrModeFlat
    sf.AddrFrame.Mode = AddrModeFlat
    sf.AddrStack.Mode = AddrModeFlat
    sf.AddrPC.Offset = ctx.Rip
    sf.AddrFrame.Offset = ctx.Rbp
    sf.AddrStack.Offset = ctx.Rsp
    dim i as Long = 1
    Print "StackWalk64 produces:"
    While StackWalk64( _
        IMAGE_FILE_MACHINE_AMD64, _
        g_hProc, _
        GetCurrentThread(), _
        @sf, _
        @ctx, _
        NULL, _
        @SymFunctionTableAccess64, _
        @SymGetModuleBase64, _
        NULL _
    )
        Print Using "&) &"; i; Hex(sf.AddrPC.Offset)
        i += 1
    Wend
End Sub

Private Sub Init()
    Const symOpts As DWORD = SYMOPT_DEBUG Or SYMOPT_OMAP_FIND_NEAREST Or SYMOPT_UNDNAME Or SYMOPT_LOAD_LINES
    SymSetOptions(symOpts)
    dim ret as WINBOOL = SymInitialize(g_hProc, NULL, TRUE)
    Print "SymInitialize = " & Str(ret)
End Sub

SymInitialize = 1
Stack is around 22FDD0
StackWalk64 produces:
1) 7795FEFA
2) 22FDD0
3) 410050
4) 40456A
5) 22F868
6) 22FDD0
7) 10
8) 3700000000315F80
Press key to exit

But still, only 1 & 4 out of those are code addresses, when there should be at least 5/6

[adeyblue]

Turns out backtrace is similarly bad for FB :-(

bob@UBUNTUVIRT:~/dumprep$ gcc-9 -O0 -g stack.c
bob@UBUNTUVIRT:~/dumprep$ ./a.out
backtrace returned 6 elems
6) 0x5587121721df
5) 0x558712172283
4) 0x5587121722a6
3) 0x5587121722c5
2) 0x7f06e74010b3
1) 0x5587121720ce
FirstFunc returned 0x6

bob@UBUNTUVIRT:~/dumprep$ $FBC64 -g -gen gcc -O 0 stack.bas
bob@UBUNTUVIRT:~/dumprep$ ./stack
backtrace returned 1 elems
1) 0x402857
FirstFunc returned 0x1

[SARG]

@adeyblue
Maybe I can help you : fbdebugger has its own 'stackwalk' for retrieving at any moment all the called procedures. However you need to use gas64 on 64bit OS.

The principle should also work with gas32 as prologue/epilogue are the same (push [r/e]bp -mov [r/e]bp,[r/e]sp).

[adeyblue]

Thanks, I tried finding it in your debugger code but couldn't.

I found out why they're so broken. It seems these functions lean on the unwind tables quite heavily to get it right, but FBC is hardcoded to turn them off, even for debug builds

Code: Select all

FBC.bas
		'' Avoid gcc exception handling bloat
		ln += "-fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables "

If you compile with
-Wc -funwind-tables
then

SymInitialize = 1
Caught 7 frames using stack
7) 401A27
6) 401ED7
5) 40202A
4) 4013B4
3) 40150B
2) 77A2F33D
1) 77B63281
TryUnwind
1) 401B5F
2) 401ED7
3) 40202A
4) 4013B4
5) 40150B
6) 77A2F33D
7) 77B63281
Unwind found 7 frames
Fn2 returned 7 with num = 87
Stack is around 22F7A0
StackWalk64 produces:
1) 401C44
2) 401F3E
3) 40202A
4) 4013B4
5) 40150B
6) 77A2F33D
7) 77B63281

They work fine. Gas needs special directives in the asm to generate them.

Also my FBC is upto date with master, but this works fine with -gen gcc but crashes after printing "Before context" with -gen gas64

Code: Select all

#include "windows.bi"

Sub TryUnwind()
Print "in tryunwind"
	dim as UNWIND_HISTORY_TABLE unwindTable
	unwindTable.Search = TRUE
	dim as ULONGLONG prevImageBase = 0
	dim as CONTEXT ctx
	ctx.ContextFlags = CONTEXT_CONTROL Or CONTEXT_INTEGER
Print "Before context"
	RtlCaptureContext(@ctx)
Print "After context"
	dim as KNONVOLATILE_CONTEXT_POINTERS NvContext
	dim as PVOID HandlerData = 0
	dim as ULONGLONG EstablisherFrame = 0
	dim as ULong counter = 0
	Print "Before loop"
	While True
		Print "CCRip: " & Hex(ctx.Rip)
		dim as PRUNTIME_FUNCTION pPrevFunc = RtlLookupFunctionEntry(ctx.Rip, @prevImageBase, @unwindTable)
		if(pPrevFunc = 0) Then		
			Print "IN Manual adjust"
			ctx.Rip  = cast(ULONGLONG, *cast(ULONGLONG ptr, ctx.Rsp))
			ctx.Rsp += 8
		else
			RtlVirtualUnwind( _
				UNW_FLAG_NHANDLER, _
				prevImageBase, _
				ctx.Rip, _
				pPrevFunc, _
				@ctx, _
				@HandlerData, _
				@EstablisherFrame, _
				@NvContext _
			)
		End If
		if(ctx.Rip = 0) Then Exit While
		counter += 1
		Print Using "&) &"; counter; Hex(ctx.Rip)
	Wend
	Print Using "Unwind found & frames"; counter
End Sub

Private Function Fn2() As Long
    dim frames(0 to 60) As Any Ptr
    dim framesPtr As Any Ptr Ptr = @frames(0)
    dim hash as DWORD
    dim as Long caught = CaptureStackBackTrace(0, 61, framesPtr, @hash)
    Print Using "Caught & frames using stack capture"; caught
    For i As Long = 0 To caught - 1
        Print Using "&) &"; caught - i; Hex(frames(i))
    Next
    TryUnwind()
    Return caught
End Function

Private Sub Fn1(num as ULong)
    dim as Long numFn2 = Fn2()
    Print Using "Fn2 returned & with num = &"; numFn2; num
End Sub

Fn1(87)

[SARG]

I tried finding it in your debugger code but couldn't.

proc(j).db And proc(j).fn contains first/last addresses of every procedure. They are retrieved in debug (stabs) data.
At the end of this code (after wend) pridx() contains a list of the running procedure indexes. The rest of the procedure (not attached) looks for procedures already running.
Made for each thread.

It's a relatively easy way. If you need more information no problem. Linux and Windows compatible, readprocessmemory is a generic function.

Code: Select all

'=======================================================
'' after stopping run  retrieves all procedures
'=======================================================
private sub proc_runnew()

	#ifdef __fb_win32__
		dim as integer dummy
		Dim vcontext As CONTEXT

		if cast(integer,@vcontext) mod 16 <>0 then
			messbox("PRBM","Context not 16byte aligned")
		EndIf
		vcontext.contextflags=CONTEXT_CONTROL or CONTEXT_INTEGER
	#endif	
	Dim libel As String
	Dim As Integer regbp,regip,regbpnb,regbpp(PROCRMAX),ret(PROCRMAX),retadr
	Dim As ULong j,k,pridx(PROCRMAX)
	Dim tv As integer



	''loading with rbp/ebp and proc index
	For ithd As Integer =0 To threadnb
		if thread(ithd).sv=-1 then continue for
		regbpnb=0
		#ifdef __fb_win32__
				GetThreadContext(thread(ithd).hd,@vcontext)
				regbp=vcontext.regbp
				regip=vcontext.regip 'current proc
			#else
				ptrace(PTRACE_GETREGS, threadcur, NULL, @regs)
				regbp=regs.xbp
				regip=regs.xip
		#endif
		While 1
			For j =1 To procnb
			   If regip>=proc(j).db And regip<=proc(j).fn Then
					regbpnb+=1
					regbpp(regbpnb)=regbp
					ReadProcessMemory(dbghand,Cast(LPCVOID,regbp+SizeOf(integer)),@regip,SizeOf(Integer),0) 'return EIP/RIP
					ret(regbpnb)=regip
					pridx(regbpnb)=j
			   	Exit For
			   EndIf
			Next
			If j>procnb Then Exit While
			ReadProcessMemory(dbghand,Cast(LPCVOID,regbp),@regbp,SizeOf(integer),0) 'previous RBP/EBP
		Wend

[adeyblue]

Thanks. I've written about three different stack walks methods now since they all go about halfway to covering all the circumstances.

Frame chain walking won't go through gen gcc code by default or the system libraries on Lin or Win on X64 because they don't use rbp for that.
While the OS stack walk methods don't need rbp but need tables FB won't generate by default to navigate through FB code.

You can start a walk from an arbitrary context on Windows using RtlVirtualUnwind so half-and-half walking is possible, but you need an accurate rsp and the info is on how big the stack frame is without having to disassemble... is in the tables
On Linux you need libunwind to walk from an arbitrary context, and I don't want that to be required

You can generate those tables at runtime using the debug symbols, but if you do, neither basic OS method will then use them, which seems a bit pointless
And any sort of manual scan has the obvious problems

So I gave up and put it onto the user to decide whether they want to use system method or rbp chain + a manual scan method.
https://github.com/adeyblue/FBStackTrace

It says Linux but I've only used it on Ubuntu 20.04 64-bit. I have no interest in setting up a bunch of different distros to see which others it may or may not work with. It definitely won't build for ARM as-is without a few things being written by someone who knows them, nor on any architecture where the stack grows upward, which the pthread man pages assure me exist.

[SARG]

@adeyblue,
I didn't look at all the files however very impressive.

One thing you should check (if not already done) : when using -gen gas64/-g and several object files (.o) the linker regroups the different .dbgstr sections in one but all the offsets in .dbgdat are not changed.

In fbdebugger a variable stores the max value for the offset and it uses it when a stab code 0 (zero) is found for defining the real new offset value.

[adeyblue]

Thank you, I fixed it. I also made it build with gas64 because...
I've made gas64 output the required asm statements to generate the unwind table.

It's only 20 extra lines.

I put the changed file up here:
https://www.airesoft.co.uk/files/temp/ir-gas64.bas

The lines I added are the
cfi_ asm_code
and
cfi_windows_asm_code
ones

It works fine on Windows with the changes, but I don't have a setup to build fbc on Linux so I don't know if it still requires a linker argument or not.

[jj2007]

The other day I posted this here: Can somebody explain in a few words what a stack walker does, and what we can expect from it, in terms of simplifying the debugging of an application?

It turned out that it was actually not a big deal to write a StackWalker macro in Assembly (sorry, I know this is a FB forum, but I guess it could be interesting also in FB). When a crash happens, a console window opens and displays this kind of info:

Code: Select all

proc            called by   1st arg   count
SetTTP          00404823    00402370  879
SubMenuWin      765F630A    00411AFE  880
DispMenu        00408B56    0040859F  932
SubMenuWin      765F630A    00411AFE  933
BubbleHelp      00408948    0040859F  1104
WndProc         765F630A    00261664  1105
SelectBM        00403A1A    00000000  1177
FindTextP       0040D757    00000005  1178
SetMyTimerP     0040D1C3    FFFFFF8D  1181
WndProc         765F630A    00261664  1182
SetLcBufferP    0040CC81    00000000  1190
gtbP            00401BBB    0000000A  1191
gtbP            00401BFC    00000012  1192
CbTimer         765F630A    00261664  1193
SelectBM        00403A1A    00000000  1246
CbTimer         765F630A    00261664  1247
SelectBM        00403A1A    00000000  1268
FindTextP       0040D757    00000005  1269
SetMyTimerP     0040D1C3    FFFFFF8D  1271
WndProc         765F630A    00261664  1272
SetLcBufferP    0040CC81    00000000  1280
gtbP            00401BBB    0000000A  1281
gtbP            00401BFC    00000012  1282
CbTimer         765F630A    00261664  1283
MyTest          004031D2    00000000  1323    <<<<<<<< this is where the crash (divide by zero) was provoked
WndProc         765F630A    00261664  1324
SetTTP          004035C7    00000000  1327
ClosePopup      004035D3    00000000  1328
DispMenu        00404817    00402370  1329
SetTTP          00404823    00402370  1330
SubMenuWin      765F630A    00411AFE  1331

Is that more or less what "stack tracing" means, or do I miss something?

[adeyblue]

What this does is this


i.e. from any point in your program, you can call a function and it will tell you the sequence of calls the thread took to get there, turning the raw code addresses into function names, source files and line numbers.

The only reason this is relatively so much code is that:
* FB can produce the function/file/line information in three different ways, none of which have built-in ways to use/extract it on any platform,
* FB doesn't by default produce any of the things needed to get the addresses in the first place. And its two 64-bit generators require two different methods of getting them
* and I wanted to make it work for 'all' types of exes wherever they came from, not just those created by FB
i.e. if you have the Microsoft symbols, it'll correctly translate address in Microsoft code to function names too (see the user32 entries have changed from the nearest exported function (GetSystemMetrics), to their actual function names)


If it was Windows only, with code built with Microsoft tools, you'd use the tools' flag to generate a PDB file and then this library would be reduced to nothing more than:

Code: Select all

trace = CaptureStackBackTrace()
SymInitialize()
For each address in trace
symbolInfo = SymFromAddr(address)
print symbolInfo
Next
SymCleanup()

I've since sent changes to the FB compiler to generate the needed info to do it the easy way on both its 64-bit generators. If those are all added in, half of this code could be thrown away in future as the easy way will just work.

[SARG]

It works fine on Windows with the changes, but I don't have a setup to build fbc on Linux so I don't know if it still requires a linker argument or not.

Do you want that I do something ?
Compiling fbc with your changes on Linux and generating some exes (gcc/gas64) with debug data ?
Just upload the modified fbc.bas source as you did for ir-gas64.

Maybe, to avoid useless added data to exes, you could add stackwalker data only when users use -g.

[Munair]

If this is officially added to the FB compiler, will it be thoroughly documented?

[adeyblue]

If this is officially added to the FB compiler, will it be thoroughly documented?

What more do you need to know? It only has four public functions and two of those are just convenience which call the others for you. The readme describes them all in the API section. I don't think you really need half a page each to say 'this function gets a stack trace' or 'this functions takes an address and returns the function name, source file and line number of that address''?

Besides, there's literaly zero chance of adding a thing that'll enlarge your exe by 600K (linux, libdwarf is *big*) or that requires an extenal dll (Windows). The stack tracing bit alone maybe since that has no depedencies, but if all the patches to generate the required data are added, it's not even necessary as the OS functions for traces will just work then.
Parsing stabs data doesn't have dependencies either, but stabs isn't default for gen gcc. Stabs is default for gas64 but without the patch above, getting the traces with gas64 isn't accurate in the presence of non-gas64 compiled code.

Basically through whatever history and means the FBC defaults were arrived at, when it comes to stack traces and symbols, it's a total mess and they're basically the worst of all worlds. It's amazing we have this many code generators, but that they couldn't converge on one debug format to be THE FB debug format (as opposed to three incompatible ones) is technically horrfic.

idk, everytime I want to do something with this language, it seems to be set up in just a way to make doing that thing as hard as it can possibly be...

Maybe, to avoid useless added data to exes, you could add stackwalker data only when users use -g.

...And then everybody is resistant to change because 80k is too much for this scary new thing they've never heard of.

Post mortem debugging is useless? Being able to use OS provided tools to instrument your app is useless? Literally every tool that may use stack traces as part of its job is useless? All of them will use the OS provided means to get one, and that's not rbp walking, which means they are all impossble to use with gas64 in its current state. If anything adding it to non-debug programs is even more important.

I don't claim it's the most important decision to ever be made, but to exclude all those potential extra uses because you're scared of an extra 80k per 2MB seems not a good tradeoff.