fbJSON - JSON Parser for FreeBASIC

Headers, Bindings, Libraries for use with FreeBASIC, Please include example of use to help ensure they are tested and usable.
Oz
Posts: 586
Joined: Jul 02, 2005 14:21
Location: Waterloo, Ontario, Canada
Contact:

fbJSON - JSON Parser for FreeBASIC

Post by Oz »

Image
news wrote:I have moved this repo to github, since I rarely use svn these days. Any further updates are going to be tossed in here. I will leave the old google code project up for historical reasons.
-Oz
11-06-2013
I wrote a JSON parser for FreeBASIC. It's a little different, in style and features, than Kristopher Windsor's parser, and mine is a little bulkier.

You can find all the information in the github project page.
To grab a fresh zip of the master branch, you can visit this link.
To see the latest updates, visit the commits section.

Update: To grab a copy from the git
git clone git://github.com/mrozbarry/fbjson.git
In the current download, you get:
  • fbJSON.bi and fbJSON.bas - The header/source file
  • (new) UTF8String and UTF8Char UDT/Classes for handling non-ascii characters
  • test.bas - Some quick tests to make sure there are no bugs
  • test*.json - Sample json files, for testing
  • Internet awesomeness! You may even get a meme named after you!
Here is some sample code showing how easy fbJSON is!
Sample JSON (test1.json)

Code: Select all

{
    "glossary": {
        "title": "example glossary",
		"GlossDiv": {
            "title": "S",
			"GlossList": {
                "GlossEntry": {
                    "ID": "SGML",
					"SortAs": "SGML",
					"GlossTerm": "Standard Generalized Markup Language",
					"Acronym": "SGML",
					"Abbrev": "ISO 8879:1986",
					"GlossDef": {
                        "para": "A \"meta-markup\" language, used to create markup languages such as DocBook.\tTab!\u002e",
						"GlossSeeAlso": ["GML", "XML"]
                    },
					"GlossSee": "markup"
                }
            }
        }
    }
}
Reading and displaying JSON

Code: Select all

dim test as fbJSON ptr = fbJSON_ImportFile( "test1.json" )
if test = NULL then
	? "Unable to load json file/string!"
	end 1
end if

print fbJSON_ExportString( test, 1 )

fbJSON_Delete( test )
Accessing JSON Data

Code: Select all

dim test as fbJSON ptr = fbJSON_ImportFile( "test1.json" )
if test = NULL then
	? "Unable to load json file/string!"
	end 1
end if

' Output 'example glossary'
print test->childByName("glossary")->childByName("title")->toString()

fbJSON_Delete( test )
Modifying JSON

Code: Select all

dim test as fbJSON ptr = fbJSON_ImportFile( "test1.json" )
if test = NULL then
	? "Unable to load json file/string!"
	end 1
end if

' Output 'example glossary'
print test->childByName("glossary")->childByName("title")->asString("changed the title!")
fbJSON_ExportFile( "test1.json", 1 )

fbJSON_Delete( test )
Remember, this is all open source, so please contribute! If you have content you want to submit (tests, patches), please reply on this thread, or contact me via the project page!

Thanks for the support!
-Oz
Last edited by Oz on Jun 12, 2013 2:32, edited 1 time in total.
AGS
Posts: 1284
Joined: Sep 25, 2007 0:26
Location: the Netherlands

Re: fbJSON - JSON Parser for FreeBASIC

Post by AGS »

I tried the following (simply added it to fbjson.bas):

Code: Select all

var tst1 = fbJSON_ImportString( !"{\"name\":[[[1,2,3,4],5,6],7,8]}")
fbJSON_ExportFile( tst1,"o1.json",0)
var s = fbJSON_Print( tst1, 0, 1, 0)
print s
fbJSON_Delete(tst1)
var tst2 = fbJSON_ImportFile("t1.json")
var s2 = fbJSON_Print(tst2,0,1,0)
print s2
fbJSON_ExportFile( tst2,"o2.json",0)
fbJSON_Delete(tst2)
output (same regardless of using exportfile or print)

Code: Select all

{
	"name": [	1,	2,	3,	4	]

}
Where is the rest of the JSON? Content of t1.json is (tried with more whitespace between tokens as well but to no avail)
{ "name":[[[1,2,3,4],5,6],7,8] }
which should match the content of string s1.

I got a nasty crash when trying to delete a root that was already deleted. Might I suggest changing fbJSON_Delete

Code: Select all

sub fbJSON_Delete( byval root as fbJSON ptr )
      if (root = 0) then
        return
      end if
      delete root
end sub
And would it be an idea to use string indexing instead of calling mid when processing the intput a character at a time? It would speed up the code.

Loading a JSON file into memory could be done in one go.
Something like

Code: Select all

function fbJSON_ImportFile( byref path as string, byval utf8 as ubyte ) as fbJSON ptr
	dim as string dstr
	dim as integer fh=freefile()
	open path for binary access read as #fh
        var err_ = err()
        if (err_) then
          print "could not open ";path
          return 0
        end if
        var file_len = LOF(fh)
        if (file_len = 0) then
           print "file ";path;"is empty (nothing to process)"
           return 0
        end if
        dstr = string(file_len,0)
        get #fh, ,dstr
	close #fh
	return fbJSON_ImportString( dstr )
end function
Reading a file in binary format should be good for both UTF8 and non - UTF8. You'd have to check for the BOM and if it's not there you could still detect UTF8 (or you could specify that a BOM at the start is needed when parsing from file).

There is something in the JSON standard on adding restrictions to the input format http://www.ietf.org/rfc/rfc4627.txt

Adding newlines to the input (as was done in the original function) is not needed? (the json format specifies \r\n? as whitespace).

And I was wondering why there is no test case for UTF8? You go through considerable trouble to get UTF8 support going and then you don't seem to test it. All I found was a unicode escape sequence (\u002e) in test1.json.

But that file did not have a UTF8 BOM nor did it contain any characters outside the range 0 - 7F (basic latin). How about some chinese/cyrillic/cherokee/cerman/anything outside the basic Latin script?

Final remarks:
I think you could use a hash table to store objects. And a dynamic array for storing arrays.

The problem with such an approach would be that it is not known in advance how large the individual hash tables/arrays should be. You could guess at the start and then grow the table as parsing continues but that would make things really slow (lots of reallocations/copying of data).

You could postpone creation of data structures until it is known how large the different data structures should be (this would possibly involve creating a more 'elaborate' parser). I do think the way the fbJSON has been written could allow for an approach that postpones creation of data structures.

The advantage of using hash tables is the fast lookup time and if you use a solid hash function ( http://burtleburtle.net/bob/c/lookup3.c ) then the number of collisions should be low lookup will be fast and there will be little need for table resizing/rehashing.
Oz
Posts: 586
Joined: Jul 02, 2005 14:21
Location: Waterloo, Ontario, Canada
Contact:

Re: fbJSON - JSON Parser for FreeBASIC

Post by Oz »

AGS wrote:I tried the following (simply added it to fbjson.bas):

Code: Select all

var tst1 = fbJSON_ImportString( !"{"name":[[[1,2,3,4],5,6],7,8]}")
fbJSON_ExportFile( tst1,"o1.json",0)
var s = fbJSON_Print( tst1, 0, 1, 0)
print s
fbJSON_Delete(tst1)
var tst2 = fbJSON_ImportFile("t1.json")
var s2 = fbJSON_Print(tst2,0,1,0)
print s2
fbJSON_ExportFile( tst2,"o2.json",0)
fbJSON_Delete(tst2)
output (same regardless of using exportfile or print)

Code: Select all

{
	"name": [	1,	2,	3,	4	]

}
Where is the rest of the JSON? Content of t1.json is (tried with more whitespace between tokens as well but to no avail)
{ "name":[[[1,2,3,4],5,6],7,8] }
which should match the content of string s1.

I got a nasty crash when trying to delete a root that was already deleted. Might I suggest changing fbJSON_Delete

Code: Select all

sub fbJSON_Delete( byval root as fbJSON ptr )
      if (root = 0) then
        return
      end if
      delete root
end sub
All fixed in the svn now, thanks for the bug report!
AGS wrote:And would it be an idea to use string indexing instead of calling mid when processing the intput a character at a time? It would speed up the code.

Loading a JSON file into memory could be done in one go.
Something like

Code: Select all

function fbJSON_ImportFile( byref path as string, byval utf8 as ubyte ) as fbJSON ptr
	dim as string dstr
	dim as integer fh=freefile()
	open path for binary access read as #fh
        var err_ = err()
        if (err_) then
          print "could not open ";path
          return 0
        end if
        var file_len = LOF(fh)
        if (file_len = 0) then
           print "file ";path;"is empty (nothing to process)"
           return 0
        end if
        dstr = string(file_len,0)
        get #fh, ,dstr
	close #fh
	return fbJSON_ImportString( dstr )
end function
...

Final remarks:
I think you could use a hash table to store objects. And a dynamic array for storing arrays.

The problem with such an approach would be that it is not known in advance how large the individual hash tables/arrays should be. You could guess at the start and then grow the table as parsing continues but that would make things really slow (lots of reallocations/copying of data).

You could postpone creation of data structures until it is known how large the different data structures should be (this would possibly involve creating a more 'elaborate' parser). I do think the way the fbJSON has been written could allow for an approach that postpones creation of data structures.

The advantage of using hash tables is the fast lookup time and if you use a solid hash function ( http://burtleburtle.net/bob/c/lookup3.c ) then the number of collisions should be low lookup will be fast and there will be little need for table resizing/rehashing.
You're free to write a patches for dynamic loading and the use of a hash table.
AGS wrote:Reading a file in binary format should be good for both UTF8 and non - UTF8. You'd have to check for the BOM and if it's not there you could still detect UTF8 (or you could specify that a BOM at the start is needed when parsing from file).

There is something in the JSON standard on adding restrictions to the input format http://www.ietf.org/rfc/rfc4627.txt

Adding newlines to the input (as was done in the original function) is not needed? (the json format specifies \r\n? as whitespace).

And I was wondering why there is no test case for UTF8? You go through considerable trouble to get UTF8 support going and then you don't seem to test it. All I found was a unicode escape sequence (\u002e) in test1.json.

But that file did not have a UTF8 BOM nor did it contain any characters outside the range 0 - 7F (basic latin). How about some chinese/cyrillic/cherokee/cerman/anything outside the basic Latin script?
Next on the chopping block is reading data as "real" UTF-8 data. It's a work in progress :-)

Thanks again for finding that bug!
-Oz
AGS
Posts: 1284
Joined: Sep 25, 2007 0:26
Location: the Netherlands

Post by AGS »

Found some more bugs.

Input is always given using a file and a call to fbJSON_ImportFile and output is written using fbJSON_ExportFile

input: {"":""}
output: {}


input: {"hello":""}
output: {"hello":"}"}

input: {"":"hello"}
output: {}

I tested some array input as well.

input: [{"hello":"hello"}]
output: ["hello"]

input: [{"hello":1}]
output: [1]

Two more...

input: [[],[]]
output: [[]]

input: [[],[],{"a":"b"}]
output:[[],{"a":"b"}]

Last result is al-most right (object is where it should be). But alas, the [] was removed.

FYI: I entered the JSON grammar into a compiler compiler and the JSON grammar is an SLR(1) grammar http://www.scribd.com/doc/51358638/39/SLR-1-Grammar
AGS
Posts: 1284
Joined: Sep 25, 2007 0:26
Location: the Netherlands

Post by AGS »

I changed my previous post somewhat as one of the JSON strings I posted wasn't JSON at all.

I found another bug:

input: [{"hello":1},{ "foo":"what_foo"}]
output: [1,{"foo":"what_foo"}]

All the bugs I found (the one in this message and the ones in my previous message) were found in the latest version of fbjson(the latest source code I could find at code.google.com).

I use the online json parser at http://json.parser.online.fr/ to check whether the JSON I'm posting is correct (just making sure I don't post syntactically incorrect JSON input (again)).
Oz
Posts: 586
Joined: Jul 02, 2005 14:21
Location: Waterloo, Ontario, Canada
Contact:

Re: fbJSON - JSON Parser for FreeBASIC

Post by Oz »

Just wanted to bump this - I got caught up in a lot of work, and fbJSON took a place in the back burner for a while, but I have recently started getting back into it. I have moved the code to github, and will try and get some more updates in. In the current state of the repo, I have broken the json parsing (segfault), and I will fix this in the very near future. Otherwise, feel free to poke around with the new UTF8String UDT/Class I've written; in it's current state, it is not meant to be used in production code, but with a few tweaks and extra features, I feel it will be an important piece of at least my parser.

AGS, thanks for the help in the past. I'm not sure if I came off as being thankful, but I have appreciated all the parser tests you have written.

-Oz
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Re: fbJSON - JSON Parser for FreeBASIC

Post by TJF »

Do you know this thread?
Parser / generator and utf8 character encoding are ready to use. And: it generates a tree of GObjects from a JSON-stream and vice versa. (Ie transfering an XML file is just a few lines of code.)
RockTheSchock
Posts: 252
Joined: Mar 12, 2006 16:25

Re: fbJSON - JSON Parser for FreeBASIC

Post by RockTheSchock »

What do you think about using lua and json as alternative. Could be fast and flexible.

http://www.kyne.com.au/~mark/software/lua-cjson.php
TJF
Posts: 3809
Joined: Dec 06, 2009 22:27
Location: N47°, E15°
Contact:

Re: fbJSON - JSON Parser for FreeBASIC

Post by TJF »

RockTheSchock wrote:Could be fast and flexible.
Could you send an example, please (in FB code)?
Oz
Posts: 586
Joined: Jul 02, 2005 14:21
Location: Waterloo, Ontario, Canada
Contact:

Re: fbJSON - JSON Parser for FreeBASIC

Post by Oz »

TJF wrote:Do you know this thread?
Parser / generator and utf8 character encoding are ready to use. And: it generates a tree of GObjects from a JSON-stream and vice versa. (Ie transfering an XML file is just a few lines of code.)
Nope, but then again, I haven't been too active in the community here as of late. Looks cool, but I'd personally shy away from it because of it's glib dependency. fbJSON has no external dependencies, can parse UTF8 Strings, and convert between the built-in string type and my UTF8String UDT.
RockTheSchock wrote:What do you think about using lua and json as alternative. Could be fast and flexible.

http://www.kyne.com.au/~mark/software/lua-cjson.php
I don't know why I would use lua to use a c interface through freebasic. Now, if I were doing some lua scripting in a freebasic application, I would definitely consider lua-cjson, but for anything that I'm trying to do with minimal dependencies, I would avoid it.

-Oz
RockTheSchock
Posts: 252
Joined: Mar 12, 2006 16:25

Re: fbJSON - JSON Parser for FreeBASIC

Post by RockTheSchock »

I don't know why I would use lua to use a c interface through freebasic. Now, if I were doing some lua scripting in a freebasic application, I would definitely consider lua-cjson, but for anything that I'm trying to do with minimal dependencies, I would avoid it.
You are absolutly right. I am not an expert with lua but i think, you can use it if you want an easy customizable and updatable part of a programm. This is just a decoder. But it shows how to push and pop nested tables with so it should be nice to post it here.

i have compiled the cjson as dll
http://www.file-upload.net/download-770 ... n.dll.html

Code: Select all

#Include ONCE "Lua/lua.bi"
#INCLUDE ONCE "Lua/lauxlib.bi"
#INCLUDE ONCE "Lua/lualib.bi"
 
Screen 19

Function LoadFile(filename As String) As String
	Dim As String json, lot
	Dim As Integer fhandle
	fhandle = FreeFile
	Open filename For Input As fhandle
	do While not eof(fhandle)
  		Line input #fhandle,lot
  		json=json+lot+Chr(13)+Chr(10)
	Loop	
	Return json
End Function

Sub iterateLuaTable(L As Lua_State Ptr, index As Integer,depth As Integer=0)
	Dim As String key,value
		
   lua_pushvalue(L, index)
   'stack now contains: -1 => table
   lua_pushnil(L)
   'stack now contains: -1 => nil; -2 => table
   Do While lua_next(L, -2)
      'stack now contains: -1 => value; -2 => key; -3 => table
      'copy the key so that lua_tostring does not modify the original
      lua_pushvalue(L, -2)
      'stack now contains: -1 => key; -2 => value; -3 => key; -4 => table
      key = *(lua_tostring(L, -1))
      value = *(lua_tostring(L, -2))
      
      Print Space(2*depth);key;" => "; value
      Sleep

      if lua_istable(L, -2) Then 
    		iterateLuaTable(L,-2,depth+1)
      EndIf
      'pop value + copy of key, leaving original key
      lua_pop(L, 2)
      'stack now contains: -1 => key; -2 => table
   Loop
   'stack now contains: -1 => table (when lua_next returns 0 it pops the key
   'but does not push anything.)
   'Pop table
   lua_pop(L, 1)
   'Stack is now the same as it was on entry to this function
End Sub

DIM Lua AS lua_State PTR
DIM AS INTEGER m = 27, n = 36
Dim As String text,table

text=LoadFile("test.json" )
'Print text

Lua = lua_open()
luaL_openlibs(lua)
luaopen_base(Lua) 


' Skript laden
If luaL_loadfile(Lua, "test.lua") ORELSE lua_pcall(Lua, 0, 0, 0) THEN
  PRINT "Fehler: " & *lua_tostring(Lua, -1)
  Sleep
  END
END IF

lua_getglobal(Lua, "decode")
lua_pushstring(Lua, text)
If lua_pcall(Lua, 1, 1, 0) THEN
  PRINT "Skriptfehler: " & *lua_tostring(Lua, -1)
  sleep
  END
END IF
PRINT "Decode:"
iterateLuaTable lua,-1

Sleep 
test.lua

Code: Select all

-- Module instantiation
local cjson = require "cjson"

function dump(obj,depth)	
   if type(obj) == 'table' then  
   	  s=''      
        for k,v in pairs(obj) do
            if type(k) ~= 'number' then 
            	k = '\n' .. string.rep(" ",depth*2) .. '"' .. k ..'"' 
            end    
            s = s  ..k..' = ' .. dump(v,depth+1) 
        end
        return s
   else
        return tostring(obj)
   end   
end

function encode(value)
   return cjson.encode(value)
end

function decode(text)			
	json = cjson.decode(text)
	return json
 	-- return dump(json,0)
end
test.json

Code: Select all

{
  "Herausgeber": "Xema",
  "Nummer": "1234-5678-9012-3456",
  "Deckung": 2e+6,
  "Waehrung": "EURO",
  "Inhaber": {
    "Name": "Mustermann",
    "Vorname": "Max",
    "maennlich": true,
    "Hobbys": [ "Reiten", "Golfen", "Lesen" ],
    "Alter": 42,
    "Kinder": {
    	 "1981-11-11": "Hans" ,
    	 "1983-10-25": "Karl"     	  
    },
    "Partner": null
  }
}
Last edited by RockTheSchock on Jun 12, 2013 14:37, edited 1 time in total.
Oz
Posts: 586
Joined: Jul 02, 2005 14:21
Location: Waterloo, Ontario, Canada
Contact:

Re: fbJSON - JSON Parser for FreeBASIC

Post by Oz »

RockTheSchock wrote:
I don't know why I would use lua to use a c interface through freebasic. Now, if I were doing some lua scripting in a freebasic application, I would definitely consider lua-cjson, but for anything that I'm trying to do with minimal dependencies, I would avoid it.
You are absolutly right. I am not an expert with lua but i think, you can use it if you want an easy customizable and updatable part of a programm. This is just a decoder. But it shows how to push and pop nested tables with so it should be nice to post it here.
I appreciate the enthusiasm, but this may not be the most appropriate thread to be posting code snippets for converting a lua table to a json object and vice-versa.

-Oz
glennl
Posts: 3
Joined: Jun 28, 2022 17:57

Re: fbJSON - JSON Parser for FreeBASIC

Post by glennl »

I know this is an old post - hopefully you can still help me with it.
I am trying to compile your fbjson-master.zip files and am getting an error.
I unzipped the fbjson-master files into a folder called fbjson-master.
Then I opened a CMD window and ran the following.

...\fbjson-master\fbc test.bas fbJSON.bas -x unit-test.bas

fbJSON.bas(254) warning 35(0): Mixing signed/unsigned operands

Thanks
Glenn
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: fbJSON - JSON Parser for FreeBASIC

Post by fxm »

Are you trying to compile a 64 bit version?
If so, try compiling a 32-bit version.
Oz
Posts: 586
Joined: Jul 02, 2005 14:21
Location: Waterloo, Ontario, Canada
Contact:

Re: fbJSON - JSON Parser for FreeBASIC

Post by Oz »

glennl wrote: Jun 28, 2022 18:37 I know this is an old post - hopefully you can still help me with it.
I am trying to compile your fbjson-master.zip files and am getting an error.
I unzipped the fbjson-master files into a folder called fbjson-master.
Then I opened a CMD window and ran the following.

...\fbjson-master\fbc test.bas fbJSON.bas -x unit-test.bas

fbJSON.bas(254) warning 35(0): Mixing signed/unsigned operands
Hi glennl,

The short answer is I haven't written any FreeBASIC in quite a while, and I'm sure there's been various improvements in the compiler, but I have not touched the code for fbJSON for 9 years, which is understandably a very long time.

The long answer is, it looks like fbc doesn't like that I'm comparing a signed and unsigned integer on line 254. I think you could _probably_ update line 253 to:

Code: Select all

dim p as integer = this.instr( s, start ) ' Notice change from uinteger to integer
That makes more sense to me anyway, since `this.instr` returns an integer, and not a uinteger.

If that works, I'd be more than happy to accept a PR in my github repository.
Post Reply