fbfrog header translator

[dkl]

https://github.com/dkl/fbfrog

fbfrog is a command line tool that reads *.h files (C API declarations) and generates a corresponding *.bi file (FreeBASIC API declarations). It's intended to automate most of the work needed to create and maintain FB bindings for C libraries.

----------------------------------------
Original message about the first experimental version:

This is a new C-to-FB header translation helper. It's based on rearranging and translating tokens, and at the moment it gets through the libcurl headers pretty well.

I started this project a while back, wanting to somehow automate header translation. Then about two weeks ago I talked about it on IRC and that made me finish it. Now it can translate the important things, like structures, fields, function declarations, function pointers. And it can do some tricks like merging in #includes. But you can read more about that in the readme.

Well, by design it is very limited, but at the same time it sort of preserves the look and feel of the original headers, not perfectly, but as close as I have ever seen. I don't know whether this is the right way to go about making FB bindings, since it's more of a 1:1 translation than a binding, but so far it was a cool experiment. No idea where this could go from here. By the way, I have never used h_2_bi or FBSWIG, so I don't know what the differences are.

fbfrog itself is a command line tool, but there's a separate GUI program for it, take a look:
Linux (GTK+) | Win32

Aaand phew, thank god it's out now, so I can go back to other stuff... Have fun with it, and take a look at the source!

[TJF]

Interesting! It's very similar to h_2_bi.

I compiled the source without problems (on Ubuntu11.10 / fbc 0.23 -- good readme.txt). But I've trouble using it, ie:

• the GUI sends lots of warnings like

  `menu_proxy_module_load': ./fbfrog-gui: undefined symbol: menu_proxy_module_load
  
  (fbfrog-gui:3087): Gtk-WARNING **: Failed to load type module: (null)

  or

  (fbfrog-gui:3087): Gtk-CRITICAL **: gtk_text_layout_real_invalidate: assertion `layout->wrap_loop_count == 0' failed

• Translating GLib headers I got

  translating: ../glib-2.28.0/glib/gdebug.h
  bug: failure at translate_decl(1187): tk_get(x) =TK_SEMI

  (I would like to see something like a character position or line# where the bug occurs.)

Some hints:

• I don't like the TODO messages included in the source. And untranslated C-source results in compile errors. In h_2_bi I append messages at the end of the line and mark C-source as comment.
• Some basics don't get translated (ie int a = 5;)
• It's pretty easy to translate ( ? : ) -> IIF( , , ), its more complicated to identify CASTs.
• Translating MACROs is important for cross-platform libraries

  Code: Select all

  #define MY_EXTERN /'__declspec(dllexport)'/
  #define MY_CALL __attribute__((__stdcall__))/'__stdcall'/
  
  /' some #defines in front, as is pretty common '/
  '' TODO: unknown construct
  MY_EXTERN MY_CALL TT1 f7(void);

  h_2_bi can expand these macros before translation and generates the desired results (that's why the control file is mandatory).
• Named nested unions/structs can be translated as (original h_2_bi translation of unsupported.h file fragment)

  Code: Select all

  ' struct T {
  ' 	union {
  ' 		int a;
  ' 	} foo;
  ' };
  UNION T_foo
  	AS INTEGER a
  END UNION
  
  TYPE T
  	AS T_foo foo
  END TYPE

Can you please explain what is done in the preparsing phase? (I'm just curious and I'm too lazy to study the source.)

[JohnK]

I am very interested in these C -> FB translators. However, now we have 3 (?). It is a little confusing. There was the original by Victor1, then h_2_bi, now this.
Are they all improving?
Does one have a strength the other does not?
Converting C++ to bi is still a "holy grail"?

Thanks for these really cool tools
-JohnK

[maddogg6]

Well, by design it is very limited, but at the same time it sort of preserves the look and feel of the original headers, not perfectly, but as close as I have ever seen. ......

.... By the way, I have never used h_2_bi or FBSWIG, so I don't know what the differences are.

This rather confuses me.

I mean, I realize you are comparing to hand translated headers (right?), but how do you know if it was not a combination (ie. first swig, then fix/hand translate.) that translated something you may assume to be hand translated??

I admit my ignorance, so please don't take this as snark or anything, but it does seem like a claim made in admitted ignorance.
Sorta like saying I am the best ________ in my family... only then later explaining when pressed that I'm also an only child and my parents are dead, but.. ;)

[dkl]

There I tried to describe a happy moment while testing fbfrog, but well, don't take it all too serious. It was just a good feeling to have made this program work.

And I really don't know what the differences are to the other tools, in terms of how they work or how they are used or what exactly they can and cannot do. Sure enough the results must be similar somehow. Although, I am assuming them to work differently, in a way that does not allow them to preserve commentary and whitespace (but instead to translate more C constructs to FB), because I haven't seen any FB headers like that, except for some nice manually translated ones.

I'd say the strengths of fbfrog are the close-to-the-original translation and the TODOs. I think it's cool to have it do 90% of the work, then go through the TODOs to fix up the rest manually and be done that much faster compared to working completely manually.


TJF, during preparsing it collects some information on #includes and #defines. It figures out the dependencies between multiple header files, so it can decide whether to merge them in, or to concatenate them, or leave them alone (and all that merging etc. is optional too), and for #defines, it checks whether they represent a calling convention or similar, and currently this is what it does:

Code: Select all

#define ITS_EMPTY
#define ITS_EXTERN __declspec(dllexport)
#define ITS_STDCALL __stdcall
#define ITS_CDECL __attribute__((cdecl))

int f();
ITS_EMPTY int f();
ITS_EXTERN int f();
ITS_STDCALL int f();
ITS_CDECL int f();
ITS_EXTERN ITS_CDECL int f();

Code: Select all

#define ITS_EMPTY
#define ITS_EXTERN __declspec(dllexport)
#define ITS_STDCALL __stdcall
#define ITS_CDECL __attribute__((cdecl))

declare function f() as integer
declare function f() as integer
declare function f() as integer
declare function f ITS_STDCALL () as integer
declare function f ITS_CDECL () as integer
'' TODO: unknown construct
ITS_EXTERN ITS_CDECL int f();

You can see it's very basic. It accepts a single define in front of function declarations, but not two, and that would need to be added to the parser. But yea, it doesn't do macro expansion at all. I mean, it could also do a preprocessing pass or something, but that was too much work for me, and I think it's better to preserve #defines without expanding them where they're used (to stay close to the original). And yea, since there are endless possibilities of how to hide C constructs behind macros, it's very easy to come across something that fbfrog can't translate. That's definitely a limitation, but it doesn't really bother me, as long as fbfrog can handle the set of common constructs found in most headers.

[maddogg6]

There I tried to describe a happy moment while testing fbfrog, but well, don't take it all too serious. It was just a good feeling to have made this program work.

lol - I see... oh and kudos. Thanks for explanation.

I wish my C was stronger but it just messes with me something fierce.

[TJF]

I am very interested in these C -> FB translators. However, now we have 3 (?). It is a little confusing. There was the original by Victor1, then h_2_bi, now this.
Are they all improving?
Does one have a strength the other does not?
Converting C++ to bi is still a "holy grail"?

Thanks for these really cool tools
-JohnK

You're wellcome. I hope you aren't not only interested in translator tools but use them as well and publish your results.

Here's my rating (in chronologic order)

• FBSWIG (the original by v1ctor) is the dinosaur. It's the first attempt to this topic. It's a real compiler (not lightweight) and it's windows only. I couldn't get it running so I don't realy know about it's features.
• h_2_bi is like a Rolls Royce. It has many features (allthough it's a lightweight solution -- no compiler). It can expand macros (recursiv, VARARG parameters) and resolve naming conflicts. It has more than 100 switches to control file input, translation and file output. It protocols info on the translation process. It can be integrated into Geany as a custom command. But it needs some experience to handle all these features.
• fbfrog is the solution inbetween (leightweight -- no compiler). It has a good set of basic features and it's easy to use. It's best choise for beginners to get started with header translations (when it gets more stabile).

I don't like the TODO messages included in the source. And untranslated C-source results in compile errors.

The next day I think this is a good solution for beginners. They get precise information about the pitfalls in the translation process and where they occur.

Although, I am assuming them to work differently, in a way that does not allow them to preserve commentary and whitespace ...

In the early versions (<0.2) h_2_bi preserved whitespace and it still can preserve commentary (on demand). The commentary (and/or if required the original C-source) is placed in front of the translation.

TJF, during preparsing it collects some information on #includes and #defines. It figures out the dependencies between multiple header files, so it can decide whether to merge them in, or to concatenate them, or leave them alone (and all that merging etc. is optional too), and for #defines, it checks whether they represent a calling convention or similar, and currently this is what it does:

Thanks for explaining that. h_2_bi collects information to support these decisions in each run. It's up to the user to make them before the next run. This is less convenient but more flexible.

[dkl]

Hello!

A new version of fbfrog has long been in the making, and now it's pretty much finished. It has a much more extensive C parser than before, and even a full C preprocessor for macro expansion and #if evaluation. It supports 64bit and automatically adds Extern blocks and fixes symbol identifier conflicts, and lots of other details beyond parsing C.

Doing full C preprocessing is probably the detail with the biggest impact: It means that it absolutely does not matter what deeply nested #ifs or ugly macros are used in the input headers, it's all stripped out. All that remains is the pure C API (and #defines, which are given special treatment). This really helps when translating C headers like libpng's png.h where the function declarations are hidden behind multiple layers of conditionally defined macros...

Another side-effect of the full CPP is that fbfrog has to read the input headers at least 5 times: once for each target system (DOS/DJGPP, Linux/x86, Linux/x86_64, Win32, Win64). And in the end, that produces 5 APIs that will be merged into one binding. It will automatically insert #if __FB_WIN32__ and such where needed. This feature could easily be extended to support further targets (e.g. Linux/ARM or FreeBSD).

And this feature can also be used to combine the APIs of multiple versions of a library into one binding, for example the v1.2.* and v1.6.* series of libpng. The binding's user could then select the desired API version by doing something like #define __LIBPNG_VERSION 12 before #include "png.bi". I think this will be useful for compiling an FB program against a specific version of libpng installed on the system, in case it's different than what png.bi defaults to. libpng is a good example here: Most Linux distros still have libpng 1.2, but the png.bi from FB 0.90.1 only supports libpng 1.5, and the two are incompatible (I think they're ABI-incompatible, and libpng will literally abort the program if it detects a version mismatch like that).

Internally, fbfrog now works a lot like a C compiler, that happens to generate FB code. It's based on an AST now, and even though it still allows preserving commentary to some degree, it's essentially the opposite of the original idea of "just rearranging tokens". But it's also what has allowed for all the new features.

[Amundo]

Haven't been around much, so just saw this. Amazing work, dude, thanks for sharing.

I really must test it by feeding it some headers from the Lotus Notes C API, but because FB isn't case-sensitive, it always trips up on some variable somewhere :-(


Awesome, just awesome...

[frisian]

Converted my first .h file into a .bi file with your FBfrog (1.5), worked very well.

The only thing that wend wrong was it missed a "#endif" at the end of the .bi file.
The .h file had at the end "#endif /* __GMP_H__ */" perhaps the comment messed things up, I removed the comment and tried again but it was still missing.

Working with FBfrog made me realize it would be nice if it includes some things.
It would be nice to start a converted file with "Converted by FBfrog <version nr.>" so everybody knows with what program and version number it was converted.

Most .bi files start with something like

#ifndef __gmp_bi__
#define __gmp_bi__

#inclib "gmp"

the #inclib part is what it's all about, the header file belongs to a certain library file.

Regards

[dkl]

Yea, fbfrog will currently emit #pragma once, instead of the #ifndef/#define/#endif include guards. But it can still emit #if blocks sometimes (to handle target-specific sections of code), so if there was an #endif missing there, that's probably a bug.

It could emit it's name/version number into the .bi file of course, but I've been going with the idea that fbfrog is just the tool to do the initial bulk conversion - but not the full conversion. Because there's almost always some further fixes needed... so it's more like "initially converted by fbfrog, and then hand-edited by ...".

About #inclib, I'm thinking that it should just be added manually. fbfrog had an -inclib option for a while to allowing adding them this way, but it was more complicated than doing it manually.

[caseih]

Found a problem trying to convert a simple header file on Linux that includes limits.h. Inside of the default limits.h it uses a "#include_next" declaration, which fbfrog can't deal with. I'm not totally sure what include_next does, and I'm not sure which of the limits.h files I have (there's about 8 on my hard drive) get pulled in by this include_next statement. How does fbfrog search for include files? just /usr/include, or does it pull in some from the gcc system itself (say /usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed/)?

[dkl]

It only searches #includes relative to the parent file, and doesn't use the gcc include directories. Although, you could specify those via -incdir <path> options, and #include_next support could be added based on that...

Is limits.h really needed though? Maybe it's enough to copy the header file you want to translate out of /usr/include, then fbfrog won't resolve the #include <limits.h> to /usr/include/limits.h anymore (unless you specified -incdir /usr/include). Often headers can be translated even though not all #includes were found.

If the header can't be preprocessed without limits.h then perhaps -fbfroginclude limits.h will do the trick - I added a stripped down, generic version of limits.h (at include/fbfrog/limits.h) which was enough to solve this kind of problem for me in the past. (where headers triggered #error directives if the limits.h #defines were missing)

As a last resort (while #include_next support is missing), you can copy the involved headers out of /usr/include, and expand #include_next's manually... I think the #include_next <limits.h> refers to gcc's private one (/usr/lib/gcc/i686-linux-gnu/4.9/include-fixed/limits.h on my system).

Edit: just implemented #include_next based on -incdir options

[TeeEmCee]

(Forgive me if reviving this old thread isn't appropriate, but it's been done several times before)

Is there any way to get fbfrog to preserve size_t instead of converting it to uinteger? (I'm converting system headers on FreeBSD)

Also, in a separate project I successfully converted a simple C++ header to FB using it. Aside from dropping inline function bodies, the only thing I had to do was manually convert by-reference arguments like "int &foo" to "byref foo as integer". fbfrog can't parse the &. It would be quite useful if it could do that, although I'm sure you have no desire to write a C++ parser.

[dkl]

The size_t handling is currently hard-coded, so the only way to disable it would be to remove it from that internal table. It might not be a 100% precise translation, but I assumed that size_t will always be the same as UInteger, so might aswell use that.

Indeed, adding support for & => byref is quite easy, also the struct parser could be extended to support simple classes. I think C++ default parameters already had to be supported aswell, due to some headers using them.