Programming Languages Benchmark
Programming Languages Benchmark
A few days ago I wrote a matrix multiplication algorithm in Java and then transfer this algorithm to a few other programming languages.
I also uploaded all code to a new GitHub repository (https://github.com/nsiatras/programming ... -benchmark).
The matrices of my "matrix multiplication benchmark" are of size 1024x1024 (1.073.741.824 multiplication and addition operations) and I populated them with random values between 0.0 and 1.0. Each experiment is run 5 times and the total time it took to complete is displayed as a result.
The results:
FreePascal-------21.0020 sec
Java--------------22.1462 sec
FreeBasic--------25.8051 sec
C------------------28.0330 sec
C# ----------------56.1140 sec
Python----------- 4426.7331 sec
The first programming language I ever write code was Microsoft's QBasic and I probably was 9 years old... BASIC is the most easy and fast to learn language ever. It has a fantastic syntax and a fantastic performance and you can even teach kids to write tiny programs in an afternoon.
In the other hand Python is horrible. It has a horrible syntax and its performance proves that it is faster to perform calculations using a pencil and a piece of paper. For the record my benchmark proves that Python is 210.77 times slower than FreeBasic.
I heard many astrophysicists uses Python to write custom scripts and analyze data received from radio-telescopes and stuff. Perhaps Python's performance has to do with the fact that we have not find any alien civilizations yet... Analyzing data with Python would take years, while with Basic or some other language it would take a few days.
I really cannot understand why the world wide developer's community turned their back to BASIC language and embraced Python, the most horrible of horrible programming languages.
Anyway.. I hope you enjoy my benchmark!
I also uploaded all code to a new GitHub repository (https://github.com/nsiatras/programming ... -benchmark).
The matrices of my "matrix multiplication benchmark" are of size 1024x1024 (1.073.741.824 multiplication and addition operations) and I populated them with random values between 0.0 and 1.0. Each experiment is run 5 times and the total time it took to complete is displayed as a result.
The results:
FreePascal-------21.0020 sec
Java--------------22.1462 sec
FreeBasic--------25.8051 sec
C------------------28.0330 sec
C# ----------------56.1140 sec
Python----------- 4426.7331 sec
The first programming language I ever write code was Microsoft's QBasic and I probably was 9 years old... BASIC is the most easy and fast to learn language ever. It has a fantastic syntax and a fantastic performance and you can even teach kids to write tiny programs in an afternoon.
In the other hand Python is horrible. It has a horrible syntax and its performance proves that it is faster to perform calculations using a pencil and a piece of paper. For the record my benchmark proves that Python is 210.77 times slower than FreeBasic.
I heard many astrophysicists uses Python to write custom scripts and analyze data received from radio-telescopes and stuff. Perhaps Python's performance has to do with the fact that we have not find any alien civilizations yet... Analyzing data with Python would take years, while with Basic or some other language it would take a few days.
I really cannot understand why the world wide developer's community turned their back to BASIC language and embraced Python, the most horrible of horrible programming languages.
Anyway.. I hope you enjoy my benchmark!
Re: Programming Languages Benchmark
fArrayC(i,j) += fArrayA(i,k) * fArrayB(k,j)
which is
fArrayC(i,j)=fArrayC(i,j) + fArrayA(i,k) * fArrayB(k,j)
which means you ask for fArrayC(i,j) every time, so the compiler has to go get that array value each time.
I just use a double to gather up then after the additions get the new fArrayC(i,j).
Also you can optimise the code from the source in fb. (first line)
I hope we don't find any alien civilizations.
Or if they see us coming they give us a really wide body swerve.
So, for that reason alone, keep with the Python you boffins!
Thanks for the benchmarks.
which is
fArrayC(i,j)=fArrayC(i,j) + fArrayA(i,k) * fArrayB(k,j)
which means you ask for fArrayC(i,j) every time, so the compiler has to go get that array value each time.
I just use a double to gather up then after the additions get the new fArrayC(i,j).
Also you can optimise the code from the source in fb. (first line)
Code: Select all
#cmdline "-gen gcc -O 2"
Sub matmult(m1() As Double,m2() As Double,ans() As Double)
Dim rows As Long=Ubound(m1,1)
Dim columns As Long=Ubound(m2,2)
If Ubound(m1,2)<>Ubound(m2,1) Then Print "Can't do matmult":Return
Redim ans(1 To rows,1 To columns)
Dim rxc As Double
For r As Long=1 To rows
For c As Long=1 To columns
rxc=0
For k As Long = 1 To Ubound(m1,2)
rxc=rxc+m1(r,k)*m2(k,c)
Next k
ans(r,c)=rxc
Next c
Next r
End Sub
dim shared as double m1(1 to 1024,1 to 1024)
dim shared as double m2(1 to 1024,1 to 1024)
dim shared as double answer(1 to 1024,1 to 1024)
for r as long =1 to 1024
for c as long =1 to 1024
m1(r,c)=rnd
m2(r,c)=rnd
next
next
var t=timer
matmult(m1(),m2(),answer())
print timer-t; " seconds"
for r as long=1 to 5
for c as long=1 to 5
print answer(r,c);",";
next
print " . . ."
next
print " . . ."
sleep
Or if they see us coming they give us a really wide body swerve.
So, for that reason alone, keep with the Python you boffins!
Thanks for the benchmarks.
Re: Programming Languages Benchmark
Python is not horrible, it is just very slow and uses lots of memory. But for many tasks, that is not a problem. You can probably find fast matrix stuff in the numpy library.
Re: Programming Languages Benchmark
Yes, python is an interpreted, dynamic language, with few optimizations. So when it comes to raw computation, it's not very efficient when used naively. There are some implementations of Python that use JIT to speed up things like this significantly. Your little benchmark might see some speedup with the jit (try pypy). But it will never be as fast as an optimized compiled routine. Turns out that leveraging compiled routines from other languages is super easy to call from Python. Which is the reason Python is so widely used in data science. Just as another data point, before python became popular, the most popular language in astrophysics and data science was Matlab which is also an intepreted language. Compiled languages like C, FB, or even Java, were never widely used in this field. Matlab and Python are both fast and capable languages when used in this capacity.
As I said before, a compiled routine is faster than pure python and Python has access to many such routines and libraries. No one would ever think to naively implement matrix multiplication in pure python. Numpy is the go-to tool for working with matrices and it is fast and heavily optimized.I heard many astrophysicists uses Python to write custom scripts and analyze data received from radio-telescopes and stuff. Perhaps Python's performance has to do with the fact that we have not find any alien civilizations yet... Analyzing data with Python would take years, while with Basic or some other language it would take a few days.
Python itself isn't particularly fast, but it is extremely expressive and very quick to write code in, which is why scientists use it. Using numpy and other tools (see the SciPy project), Python is an incredibly powerful and fast tool. Just as fast as something you could write in C or FB. Just don't expect to write Java code (or even FB code) with it. The idioms are just different. Python is a fantastic glue language for bringing pieces together. Similar to the unix philosophy of tying programs together with pipes to do new and useful things, Python's generator expressions can glue data filters (often external, compiled routines) and processors together quickly and flexibly for processing large data streams. For those who use Python every day its syntax is one of its most appealing features actually. There are some downsides to whitespace syntax, but overall I absolutely love the fact that Python is essentially executable pseudo-code. How you'd code on the back of a napkin is pretty much actual python syntax.
Re: Programming Languages Benchmark
Here's my results:
Free Basic Matrix Multiplication Benchmark finished in: 25.27700090408325 seconds
Python NumPy Matrix Multiplication Benchmark finished in: 0.043869
Off topic but here's the Python code using numpy (which every python math programmer uses):
Anyway, yes Python is slow but I simply demonstrate that when used correctly with fast libraries, Python is a good choice for heavy math and data analysis. Makes it easy to slurp up data and hand it off to the Fortran-based math libraries for the heavy lifting.
I'm sure there are some very fast libraries that could be called from FB as well that would be much faster than the for loops. Does anyone know of a good one that's call-able from FB? I think BLAS can do matrix multiplication (numpy uses it under the hood), but it's very low-level.
Free Basic Matrix Multiplication Benchmark finished in: 25.27700090408325 seconds
Python NumPy Matrix Multiplication Benchmark finished in: 0.043869
Off topic but here's the Python code using numpy (which every python math programmer uses):
Code: Select all
import random
import time
import numpy as np
n = 1024
# Add random values to Arrays A and B
A = np.array([[random.random() for row in range(n)] for col in range(n)])
B = np.array([[random.random() for row in range(n)] for col in range(n)])
#C = [[0 for row in range(n)] for col in range(n)]
print("Python Numpy Matrix Multiplication Benchmark started. Please wait...");
start = time.time()
# Multiply A to B values and add them to C
C = np.matmul(A,B)
print (C)
end = time.time()
print("Python Numpy Matrix Multiplication Benchmark finished in: %0.6f" % (end-start))
I'm sure there are some very fast libraries that could be called from FB as well that would be much faster than the for loops. Does anyone know of a good one that's call-able from FB? I think BLAS can do matrix multiplication (numpy uses it under the hood), but it's very low-level.
Re: Programming Languages Benchmark
my times
FB 64-bit about 11 seconds
caseih python code
Python Numpy Matrix Multiplication Benchmark finished in: 0.248770
Python Numpy Matrix Multiplication Benchmark finished in: 0.009937
Python Numpy Matrix Multiplication Benchmark finished in: 0.010943
the first run is about .25 sec and after about .01 sec
FB 64-bit about 11 seconds
caseih python code
Python Numpy Matrix Multiplication Benchmark finished in: 0.248770
Python Numpy Matrix Multiplication Benchmark finished in: 0.009937
Python Numpy Matrix Multiplication Benchmark finished in: 0.010943
the first run is about .25 sec and after about .01 sec
Re: Programming Languages Benchmark
I was looking into gsl cblas but it's naming convention sucks, can't find matrix multiplication, if anyone cares to have look viewtopic.php?t=28508caseih wrote: ↑Sep 13, 2022 22:50 I'm sure there are some very fast libraries that could be called from FB as well that would be much faster than the for loops. Does anyone know of a good one that's call-able from FB? I think BLAS can do matrix multiplication (numpy uses it under the hood), but it's very low-level.
Re: Programming Languages Benchmark
There is a gsl_linalg_matmult in the .def file for the dll.
Re: Programming Languages Benchmark
By the way, here's another python version that's more "pythonic." It's still slow, but orders of magnitude faster than the naive version. This is using a combination of list comprehensions and a generator expression, which is something that makes python quite fast at some things.
Python Matrix Multiplication Benchmark finished in: 93.499248
Still slow, but only 4 times slower than the naive FB looping version, which is actually pretty amazing to me.
Code: Select all
import random
import time
def matrix_mul(a, b):
return [[sum(i * j for i, j in zip(r, c)) for c in zip(*b)]
for r in a]
n = 1024
# Add random values to Arrays A and B
A = [[random.random() for row in range(n)] for col in range(n)]
B = [[random.random() for row in range(n)] for col in range(n)]
print("Python Matrix Multiplication Benchmark started. Please wait...");
start = time.time()
# Multiply A to B values and add them to C
C = matrix_mul(A,B)
end = time.time()
print("Python Matrix Multiplication Benchmark finished in: %0.6f" % (end-start))
Still slow, but only 4 times slower than the naive FB looping version, which is actually pretty amazing to me.
Last edited by caseih on Sep 14, 2022 0:00, edited 1 time in total.
Re: Programming Languages Benchmark
hi dodicat
if I remember right there's even an example included with the FB distribution
was looking at https://github.com/CNugteren/CLBlast but the naming is just as bad or worst
if I remember right there's even an example included with the FB distribution
was looking at https://github.com/CNugteren/CLBlast but the naming is just as bad or worst
Re: Programming Languages Benchmark
These libraries are pretty powerful and fast but somewhat awkward to use. I think that kind of answers the question the OP had as to why scientists are choosing Python over BASIC. Numpy hits a sweet spot. By the way all of these fast math libraries are also built into Matlab which is why it reigned supreme for years if you wanted to do a lot of linear algebra.
Re: Programming Languages Benchmark
Here's a gsl version in FreeBASIC:
I'm puzzled, though, as to why it's so slow. It's 7.6 seconds on my machine here. And I'm just timing the actual multiply, not the matrix setup. Does anyone know why it might be so slow compared to numpy? EDIT: sadly it appears GSL's multiply implementation is not optimized to use SIMD CPU instructions.
Code: Select all
#include "gsl/gsl_matrix.bi"
#include "gsl/gsl_linalg.bi"
#include "gsl/gsl_blas.bi"
Dim StartTime As Double, EndTime as Double
Dim As gsl_matrix Ptr A = gsl_matrix_alloc(1024, 1024)
Dim As gsl_matrix Ptr B = gsl_matrix_alloc(1024, 1024)
Dim As gsl_matrix Ptr C = gsl_matrix_alloc(1024, 1024)
For i As Integer = 0 To 1023
For j As Integer = 0 To 1023
gsl_matrix_set (A, i, j, rnd)
gsl_matrix_set (B, i, j, rnd)
Next
next
print("FreeBasic Matrix Multiplication Benchmark started. Please wait...")
StartTime = Timer
gsl_linalg_matmult(A,B,C)
EndTime = Timer
print("Free Basic Matrix Multiplication Benchmark finished in: " & (EndTime-StartTime) & " seconds")
Sleep
Re: Programming Languages Benchmark
hi caseih
may try this? https://github.com/CNugteren/CLBlastFree Basic Matrix Multiplication Benchmark finished in: 1.298868600002606 seconds
Re: Programming Languages Benchmark
@caseih
from https://stackoverflow.com/questions/406 ... -using-gsl
your example with a tiny mod
from https://stackoverflow.com/questions/406 ... -using-gsl
your example with a tiny mod
Code: Select all
#include "gsl/gsl_matrix.bi"
#include "gsl/gsl_linalg.bi"
#include "gsl/gsl_blas.bi"
Dim StartTime As Double, EndTime as Double
Dim As gsl_matrix Ptr A = gsl_matrix_alloc(1024, 1024)
Dim As gsl_matrix Ptr B = gsl_matrix_alloc(1024, 1024)
Dim As gsl_matrix Ptr C = gsl_matrix_alloc(1024, 1024)
For i As Integer = 0 To 1023
For j As Integer = 0 To 1023
gsl_matrix_set (A, i, j, rnd)
gsl_matrix_set (B, i, j, rnd)
Next
next
print("FreeBasic Matrix Multiplication Benchmark started. Please wait...")
StartTime = Timer
'gsl_linalg_matmult(A,B,C)
gsl_blas_dgemm(CblasNoTrans, CblasNoTrans, 1.0, B, A, 0.0, C)
EndTime = Timer
print("Free Basic Matrix Multiplication Benchmark finished in: " & (EndTime-StartTime) & " seconds")
Sleep
Free Basic Matrix Multiplication Benchmark finished in: 0.497285199999169 seconds