Programming Languages Benchmark

[StarByte]

@dodicat = thumbs up

here is the source code from B4J (b4x.com):

Code: Select all

'Non-UI application (console / server application)
#Region Project Attributes
    #CommandLineArgs:
    #MergeLibraries: True
#End Region

Sub Process_Globals
	Public fCount As Int: fCount = 1024
	Public fArrayA(fCount,fCount) As Double
	Public fArrayB(fCount,fCount) As Double
	Public fArrayC(fCount,fCount) As Double
	Public BenchmarkTests,i,j,k As Int
	Public tmp As String
	End Sub
	
Sub AppStart (Args() As String)
	Log($"B4J Matrix Multiplication Benchmark started. Please wait..."$)
	Dim start As Long = DateTime.Now
	' Add random values to Arrays A and B (0 to 1)
	For BenchmarkTests = 0 To 4
		For i = 0 To fCount-1
			For j = 0 To fCount-1
				fArrayA(i,j) = Rnd(0,1)
				fArrayB(i,j) = Rnd(0,1)
				fArrayC(i,j) = 0
			Next
		Next
		' Multiply A to B values and add them to C
		For i = 0 To fCount -1
			For j = 0 To fCount -1
				For k = 0 To fCount -1
					fArrayC(i,j)  = (fArrayC(i,j) + fArrayA(i,k)) * fArrayB(k,j)
				Next
			Next
		Next	
	Next
	Log($"B4J Matrix Muliplication Benchmark finished in: ${DateTime.Now - start}ms"$)     
End Sub

The result: about 10% faster than Freebasic.

We are not in a competition here!

Optimized code generation

While FreeBASIC is not an optimizing compiler, it does many kinds of general optimizations to generate the fastest possible code on x86 CPU's, not losing to other BASIC alternatives, including the commercial ones. Completely free All third-party tools are also free. No piece of abandoned or copyrighted software is used (except GoRC on Win32). The assembler, linker, archiver, and other command-line applications come from the GNU binutils programming tools.

[deltarho[1859]]

Also got rid of rnd which will throw off comparison.

Evidence?

Citing Rnd as having any bearing on the comparison is a red herring.

With your code I now get BlitzMax to be 7% faster than FreeBASIC.

Perhaps we can find something to slow down BlitzMax with a lesser effect on FreeBASIC. Eventually we could end up with two codes which run at the same time. We could then publish that and say BlitzMax and FreeBASIC run at the same speed.

[caseih]

Including rnd is indeed valid... if you're benchmarking rnd.

We are not in a competition here!

Haha! Good to hear it.

As far as meaningful benchmarks go, at the moment, for me, on my computer, FreeBASIC is many times faster than Blitzmax ng (horrible name by the way--every time I type it I have to look up the name): I can develop a small program, compile it, run it in a fraction of the time it would take me in Blitzmax ng.

[marcov]

We are not in a competition here!

Optimized code generation

Note that java JDKs currently are 64-bit, so you should compare to the 64-bit FB exe then. Specially for floating point that may matter, as well as that you test on the same CPU.

[dodicat]

Also got rid of rnd which will throw off comparison.

Evidence?

Citing Rnd as having any bearing on the comparison is a red herring.

With your code I now get BlitzMax to be 7% faster than FreeBASIC.

Perhaps we can find something to slow down BlitzMax with a lesser effect on FreeBASIC. Eventually we could end up with two codes which run at the same time. We could then publish that and say BlitzMax and FreeBASIC run at the same speed.

Hi deltarho[]
You have been busy speeding up, testing, speeding up, testing, ... random number generators with an entourage of members for these last few years.
I notice that at no time did you throw in a matrix multiplication at the end, and if it did happen would it have been misleading (red herring)?
I would guess that benchmarking matrix multiplication would only be testing float operations, and any external functions (rnd) should not be included.
The topic starts ... my "matrix multiplication benchmark"... by vinion , after all!
There is much to argue about benchmarking, and how to avoid comparing apples and oranges.
No matter how you test will annoy somebody somewhere.
I got BlitzMax a while back, it was free, I messed about with it a little, and I reckon it an OK compiler.
But I wasn't won over in any way, and here I am, still stuck in this place.

[deltarho[1859]]

@dodicat

My point about Rnd being a red herring was in the context of the codes being compared above. The amount of time consumed by using Rnd is negligible compared with the application session times. Even if we used a Rnd of infinite speed in FreeBASIC it would not make a blind bit of difference when we come to the comparing of the applications via BlitzMax and FreeBASIC.

The speed of a PRNG is only relevant if its use takes up a significant percentage of an application's session time. For the vast majority of applications, that is not true. So for the vast majority of applications, the speed of a PRNG is irrelevant. With the original codes above the Rnd usage, in FreeBASIC, was about three times faster using MSWSII compared with Rnd's default. How did that influence the final comparison? It didn't – not in the slightest. There is the red herring.

The important metric of a PRNG is the quality of its randomness. For a few random number requests, even the quality is of less importance and impossible to determine anyway. However, I take the view that if we use a PRNG with a good quality of randomness, passing PractRand to 1TB or more, then we cannot go wrong. From a randomness perspective, PractRand cannot separate PCG32II and MSWSII. What we have here is a limitation of PractRand, as I reckon MSWSII is probably better than PCG32II since MSWSII is non-linear whereas PCG32II is a permuted linear method. This consideration is off-topic.

[deltarho[1859]]

Erratum regarding my remark about BlitzMax not having 32-bit and 64-bit unsigned integers. I must have looking at an old part of the website which had not been updated. The truth is, 32-bit and 64-bit unsigned integers are covered.

Just for the record the BlitzMax random module in random.zip has a SIMD-oriented Fast Mersenne Twister by Mutsuo Saito, Makoto Matsumoto and Hiroshima (2007-2022) and xoshiro256++ by David Blackman and Sebastiano Vigna (2019). We have a xoshiro256++ implementation.

[dodicat]

It has been introduced to a new one now, algorithm previously tested to 1Tb practrand.
blitzmax

Code: Select all

Type mrnd
Field a:ULong
Field b:ULong
Field c:ULong
Field d:ULong
Field e:ULong
Method rndu:ULong()
   e = a - ((b Shl 7) Or (b Shr (57)))
   a = b ~ ((c Shl 13) Or (c Shr (51)))
   b = c + ((d Shl 37) Or (d Shr (27)))
   c = d + e
   d = e + a
   Return d
End Method

Method seed:Int(n:ULong)
a=n;b=n;c=n;d=n
Local m:ULong
    For m =1 To n
    rndU()
Next
Return 0
End Method

End Type


Local z:mrnd=New mrnd
z.seed(2000)
Local n
For n=1 To 10
Print z.rndu()
Next

 

result

Code: Select all

Building randomstype
[ 23%] Processing:randomstype.bmx
[ 86%] Compiling:randomstype.bmx.console.release.win32.x64.c
[100%] Linking:randomstype.exe
Executing:randomstype.exe
1342416086889183520
14834550578516216992
7696141743378967536
7743552022530922768
12564773170212065888
11780089991564600448
4736501714731261456
14426354289245502928
17678209079631715168
2806349347317571744

Process complete
 

Nice language!
Kinda old style basic.
ulong is the 64 bit unsigned integer.
And it's FREE

[deltarho[1859]]

It has been introduced to a new one now ...

Nice one dodicat.

[jgrillout]

I really cannot understand why the world wide developer's community turned their back to BASIC language and embraced Python, the most horrible of horrible programming languages.

I totally agree. I'm a BASIC programmer. I made a descent living for 35 years usin BASIC running on systems such as VAX , Tandy business computer, Commodore Business Machine (CBM),WANG,and BBx business BASIC. Several well known Accounting/Inventory & Distribution systems are written using BBx and were very popular in the 80s. They are still being used today. BBx has been and still is cross platform.

Through the years, my clients and prospects hired IT consultants that ridiculed the software I sold. The common criticism was don't get a system written in BASIC because it's an interpreted language, You need a system written in C. Well the problem for them was there were few ERP systems written in C at that time. Our biggest competitors system were written in COBOL which was another under rated language that powered thousands fortune 500 companies. I don't know this for a fact but some say there are more ERP systems written with BASIC and COBOL within large corporations & government entities than other languages.

I am amused that two of the reasons python became so popular are compilation was not required and the interpreted nature allows immediate execution from a command prompt. Been there and done that since 1979 !

Now to be fair many languages and operating system wouldn't exist without C.

[caseih]

Indeed interpreted languages have always been very useful and important. My only concern with a software package written in the early 80s in interpreted BASIC would have been maintainability of the code by the vendor. At time most BASIC code bases of significant size were very hard to understand and maintain without the Pascal-like structure that came in the mid 80s. Nothing wrong with GOTO and GOSUB and line numbers but it does make the code much harder to read. I think that was really the only reason people would have balked at an enterprise system written in BASIC at the time. I think it's great that you have built such successful tools in interpreted BASIC. Hopefully FB will be useful to you also.

I am amused that two of the reasons python became so popular are compilation was not required and the interpreted nature allows immediate execution from a command prompt. Been there and done that since 1979 !

Erm, no I don't think either of those attributes contributes that much to Python's popularity. There are dozens of other languages that have those same characteristics. We have had Perl for many decades (35 years now apparently) which had those two attributes. Granted Perl was extremely popular for many years. And it is a very useful language.

Now to be fair many languages and operating system wouldn't exist without C.

Indeed. FB wouldn't be here, nor would python nor most other language in the last 40 years. FBC itself is self hosted, but the runtime library is still in C, but maybe some day the pure FB runtime will be finished.

Merry Christmas, Happy New Year.

[marcov]

Keep in mind that there was a reason for interpreted BASICs in minis. It was not for some language advantage, but simply because the tokenized program would be very compact and the interpreter in ROM, making good utilization of RAM. Also it allowed things like CHAINing.

This might not be what users of e.g. Python primarily want from an interpreted language nowadays.

[pidd]

There is a theoretical flaw in these benchmarks, the calculations achieve nothing, the resulting array is not used, any optimiser worth its salt should realise this and not perform any of the calculations, assignments or loops and only do the timing functions and prints which would be nearly instantaneous.

I was hugely disappointed that php7.4 was a long way out the running at around 8 times slower than fbc, I haven't profiled to find out why but my php is set up for development which would add unnecessary overhead. I suspect php8.2 would perform considerably better but I haven't got php8 loaded up anywhere yet.

fbc -O 2 ran roughly 40% faster than default for my computer, I'm not sure how it could optimise to that extent on such a sequential program which is doing little other than iteratively popping numbers from compulsory calculations in pigeon holes. Many other programs I write similar to this haven't achieved anything like that improvement. Even if all the inner loops were in-lined I would still expect the calculation times to significantly swamp the loop savings.

[dodicat]

I read somewhere that Microsoft included qbasic (interpreted quickbasic) in their earlier versions (DOS, win 95/98/me up to but excluding win 2000) because people would have to purchase quickbasic if they wanted a compiler.
So it was a cash driven idea.

[caseih]

That's a bit cynical. By the time MS started offering qbasic with MS-DOS 6.0, QuickBasic had been out for many years. It made a lot of sense that in a time when many/most BASIC programmers were coding in QuickBasic, that MS would introduce a compatible, but interpreted version of QB and ship it with DOS as a replacement for GWBASIC, which was over 10 years old. The intent was not to bait people into buying QB. At the time QB was nearly the end of its life anyway, version 4.5 which was the last version released.

As a side note, QuickBasic was kind of a hybrid between an interpreter and a compiler, and quite revolutionary at the time. QB compiled source code on the fly to an intermediate form that was still editable but nearly executable directly in a light-weight interpreter, for lack of a better term. See https://retrocomputing.stackexchange.co ... ded-p-code. The p-code thing gave, in my opinion, QB excellent debugging capabilities. And you stop your program and do immediate commands to interrogate the state of your variables, and call functions. This is stuff that only relatively recently you could do in Visual Studio.