Programming Languages Benchmark

[Vinion]

A few days ago I wrote a matrix multiplication algorithm in Java and then transfer this algorithm to a few other programming languages.
I also uploaded all code to a new GitHub repository (https://github.com/nsiatras/programming ... -benchmark).

The matrices of my "matrix multiplication benchmark" are of size 1024x1024 (1.073.741.824 multiplication and addition operations) and I populated them with random values between 0.0 and 1.0. Each experiment is run 5 times and the total time it took to complete is displayed as a result.

The results:
FreePascal-------21.0020 sec
Java--------------22.1462 sec
FreeBasic--------25.8051 sec
C------------------28.0330 sec
C# ----------------56.1140 sec
Python----------- 4426.7331 sec

The first programming language I ever write code was Microsoft's QBasic and I probably was 9 years old... BASIC is the most easy and fast to learn language ever. It has a fantastic syntax and a fantastic performance and you can even teach kids to write tiny programs in an afternoon.

In the other hand Python is horrible. It has a horrible syntax and its performance proves that it is faster to perform calculations using a pencil and a piece of paper. For the record my benchmark proves that Python is 210.77 times slower than FreeBasic.

I heard many astrophysicists uses Python to write custom scripts and analyze data received from radio-telescopes and stuff. Perhaps Python's performance has to do with the fact that we have not find any alien civilizations yet... Analyzing data with Python would take years, while with Basic or some other language it would take a few days.

I really cannot understand why the world wide developer's community turned their back to BASIC language and embraced Python, the most horrible of horrible programming languages.

Anyway.. I hope you enjoy my benchmark!

[dodicat]

fArrayC(i,j) += fArrayA(i,k) * fArrayB(k,j)
which is
fArrayC(i,j)=fArrayC(i,j) + fArrayA(i,k) * fArrayB(k,j)
which means you ask for fArrayC(i,j) every time, so the compiler has to go get that array value each time.
I just use a double to gather up then after the additions get the new fArrayC(i,j).
Also you can optimise the code from the source in fb. (first line)

Code: Select all

#cmdline "-gen gcc -O 2"
Sub matmult(m1() As Double,m2() As Double,ans() As Double)
      Dim rows As Long=Ubound(m1,1)
      Dim columns As Long=Ubound(m2,2)
      If Ubound(m1,2)<>Ubound(m2,1) Then Print "Can't do matmult":Return
      Redim ans(1 To rows,1 To columns)
      Dim rxc As Double
      For r As Long=1 To rows
            For c As Long=1 To columns
                  rxc=0
                  For k As Long = 1 To Ubound(m1,2)
                        rxc=rxc+m1(r,k)*m2(k,c)
                  Next k
                  ans(r,c)=rxc
            Next c
      Next r
End Sub

dim shared as double m1(1 to 1024,1 to 1024)
dim shared as double m2(1 to 1024,1 to 1024)
dim shared as double answer(1 to 1024,1 to 1024)
for r as long =1 to 1024
      for c as long =1 to 1024
            m1(r,c)=rnd
            m2(r,c)=rnd
      next
next
var t=timer
matmult(m1(),m2(),answer())
print timer-t; "  seconds"
for r as long=1 to 5
      for c as long=1 to 5
            print answer(r,c);",";
      next
      print " . . ."
next
print "            . . ."
sleep


 

I hope we don't find any alien civilizations.
Or if they see us coming they give us a really wide body swerve.
So, for that reason alone, keep with the Python you boffins!
Thanks for the benchmarks.

[badidea]

Python is not horrible, it is just very slow and uses lots of memory. But for many tasks, that is not a problem. You can probably find fast matrix stuff in the numpy library.

[caseih]

In the other hand Python is horrible. It has a horrible syntax and its performance proves that it is faster to perform calculations using a pencil and a piece of paper. For the record my benchmark proves that Python is 210.77 times slower than FreeBasic.

Yes, python is an interpreted, dynamic language, with few optimizations. So when it comes to raw computation, it's not very efficient when used naively. There are some implementations of Python that use JIT to speed up things like this significantly. Your little benchmark might see some speedup with the jit (try pypy). But it will never be as fast as an optimized compiled routine. Turns out that leveraging compiled routines from other languages is super easy to call from Python. Which is the reason Python is so widely used in data science. Just as another data point, before python became popular, the most popular language in astrophysics and data science was Matlab which is also an intepreted language. Compiled languages like C, FB, or even Java, were never widely used in this field. Matlab and Python are both fast and capable languages when used in this capacity.

I heard many astrophysicists uses Python to write custom scripts and analyze data received from radio-telescopes and stuff. Perhaps Python's performance has to do with the fact that we have not find any alien civilizations yet... Analyzing data with Python would take years, while with Basic or some other language it would take a few days.

As I said before, a compiled routine is faster than pure python and Python has access to many such routines and libraries. No one would ever think to naively implement matrix multiplication in pure python. Numpy is the go-to tool for working with matrices and it is fast and heavily optimized.

Python itself isn't particularly fast, but it is extremely expressive and very quick to write code in, which is why scientists use it. Using numpy and other tools (see the SciPy project), Python is an incredibly powerful and fast tool. Just as fast as something you could write in C or FB. Just don't expect to write Java code (or even FB code) with it. The idioms are just different. Python is a fantastic glue language for bringing pieces together. Similar to the unix philosophy of tying programs together with pipes to do new and useful things, Python's generator expressions can glue data filters (often external, compiled routines) and processors together quickly and flexibly for processing large data streams. For those who use Python every day its syntax is one of its most appealing features actually. There are some downsides to whitespace syntax, but overall I absolutely love the fact that Python is essentially executable pseudo-code. How you'd code on the back of a napkin is pretty much actual python syntax.

[caseih]

Here's my results:
Free Basic Matrix Multiplication Benchmark finished in: 25.27700090408325 seconds
Python NumPy Matrix Multiplication Benchmark finished in: 0.043869

Off topic but here's the Python code using numpy (which every python math programmer uses):

Code: Select all

import random
import time
import numpy as np

n = 1024

# Add random values to Arrays A and B
A = np.array([[random.random() for row in range(n)] for col in range(n)])
B = np.array([[random.random() for row in range(n)] for col in range(n)])
#C = [[0 for row in range(n)] for col in range(n)]

print("Python Numpy Matrix Multiplication Benchmark started. Please wait...");

start = time.time()

# Multiply A to B values and add them to C
C = np.matmul(A,B)
print (C)

end = time.time()
print("Python Numpy Matrix Multiplication Benchmark finished in: %0.6f" % (end-start))

Anyway, yes Python is slow but I simply demonstrate that when used correctly with fast libraries, Python is a good choice for heavy math and data analysis. Makes it easy to slurp up data and hand it off to the Fortran-based math libraries for the heavy lifting.

I'm sure there are some very fast libraries that could be called from FB as well that would be much faster than the for loops. Does anyone know of a good one that's call-able from FB? I think BLAS can do matrix multiplication (numpy uses it under the hood), but it's very low-level.

[srvaldez]

my times
FB 64-bit about 11 seconds
caseih python code
Python Numpy Matrix Multiplication Benchmark finished in: 0.248770
Python Numpy Matrix Multiplication Benchmark finished in: 0.009937
Python Numpy Matrix Multiplication Benchmark finished in: 0.010943
the first run is about .25 sec and after about .01 sec

[srvaldez]

I'm sure there are some very fast libraries that could be called from FB as well that would be much faster than the for loops. Does anyone know of a good one that's call-able from FB? I think BLAS can do matrix multiplication (numpy uses it under the hood), but it's very low-level.

I was looking into gsl cblas but it's naming convention sucks, can't find matrix multiplication, if anyone cares to have look viewtopic.php?t=28508

[dodicat]

There is a gsl_linalg_matmult in the .def file for the dll.

[caseih]

By the way, here's another python version that's more "pythonic." It's still slow, but orders of magnitude faster than the naive version. This is using a combination of list comprehensions and a generator expression, which is something that makes python quite fast at some things.

Code: Select all

import random
import time

def matrix_mul(a, b):
    return [[sum(i * j for i, j in zip(r, c)) for c in zip(*b)]
        for r in a]

n = 1024

# Add random values to Arrays A and B
A = [[random.random() for row in range(n)] for col in range(n)]
B = [[random.random() for row in range(n)] for col in range(n)]

print("Python Matrix Multiplication Benchmark started. Please wait...");
start = time.time()

# Multiply A to B values and add them to C
C = matrix_mul(A,B)

end = time.time()
print("Python Matrix Multiplication Benchmark finished in: %0.6f" % (end-start))

Python Matrix Multiplication Benchmark finished in: 93.499248

Still slow, but only 4 times slower than the naive FB looping version, which is actually pretty amazing to me.

[caseih]

There is a gsl_linalg_matmult in the .def file for the dll.

I suspect gsl_linalg_matmult should be able to do the 1000 matrix nearly instantaneously.

[srvaldez]

hi dodicat
if I remember right there's even an example included with the FB distribution
was looking at https://github.com/CNugteren/CLBlast but the naming is just as bad or worst

[caseih]

These libraries are pretty powerful and fast but somewhat awkward to use. I think that kind of answers the question the OP had as to why scientists are choosing Python over BASIC. Numpy hits a sweet spot. By the way all of these fast math libraries are also built into Matlab which is why it reigned supreme for years if you wanted to do a lot of linear algebra.

[caseih]

Here's a gsl version in FreeBASIC:

Code: Select all

#include "gsl/gsl_matrix.bi"
#include "gsl/gsl_linalg.bi"
#include "gsl/gsl_blas.bi"

Dim StartTime As Double, EndTime as Double

Dim As gsl_matrix Ptr A = gsl_matrix_alloc(1024, 1024)
Dim As gsl_matrix Ptr B = gsl_matrix_alloc(1024, 1024)
Dim As gsl_matrix Ptr C = gsl_matrix_alloc(1024, 1024)

For i As Integer = 0 To 1023
    For j As Integer = 0 To 1023
        gsl_matrix_set (A, i, j, rnd)
        gsl_matrix_set (B, i, j, rnd)
    Next
next

print("FreeBasic Matrix Multiplication Benchmark started. Please wait...")

StartTime = Timer
gsl_linalg_matmult(A,B,C)
EndTime = Timer

print("Free Basic Matrix Multiplication Benchmark finished in: " & (EndTime-StartTime) & " seconds")
Sleep

I'm puzzled, though, as to why it's so slow. It's 7.6 seconds on my machine here. And I'm just timing the actual multiply, not the matrix setup. Does anyone know why it might be so slow compared to numpy? EDIT: sadly it appears GSL's multiply implementation is not optimized to use SIMD CPU instructions.

[srvaldez]

hi caseih

Free Basic Matrix Multiplication Benchmark finished in: 1.298868600002606 seconds

may try this? https://github.com/CNugteren/CLBlast

[srvaldez]

@caseih
from https://stackoverflow.com/questions/406 ... -using-gsl
your example with a tiny mod

Code: Select all

#include "gsl/gsl_matrix.bi"
#include "gsl/gsl_linalg.bi"
#include "gsl/gsl_blas.bi"

Dim StartTime As Double, EndTime as Double

Dim As gsl_matrix Ptr A = gsl_matrix_alloc(1024, 1024)
Dim As gsl_matrix Ptr B = gsl_matrix_alloc(1024, 1024)
Dim As gsl_matrix Ptr C = gsl_matrix_alloc(1024, 1024)

For i As Integer = 0 To 1023
    For j As Integer = 0 To 1023
        gsl_matrix_set (A, i, j, rnd)
        gsl_matrix_set (B, i, j, rnd)
    Next
next

print("FreeBasic Matrix Multiplication Benchmark started. Please wait...")

StartTime = Timer
'gsl_linalg_matmult(A,B,C)
gsl_blas_dgemm(CblasNoTrans, CblasNoTrans, 1.0, B, A, 0.0, C)
EndTime = Timer

print("Free Basic Matrix Multiplication Benchmark finished in: " & (EndTime-StartTime) & " seconds")
Sleep

Free Basic Matrix Multiplication Benchmark finished in: 0.497285199999169 seconds