Neural network - flappy bird

User projects written in or related to FreeBASIC.
Gunslinger
Posts: 103
Joined: Mar 08, 2016 19:10
Location: The Netherlands

Neural network - flappy bird

Post by Gunslinger »

Computer try to learn flappy bird game with a generation learning neural network.
Kinda works but something is wrong.

Code: Select all

#include "fbgfx.bi"
#include "string.bi"

#if __FB_LANG__ = "fb"
Using FB '' Scan code constants are stored in the FB namespace in lang FB
#endif

#If Defined(__FB_WIN32__) 'windows timers resolution update
Declare Function HighPrecisionClock Lib "winmm" Alias "timeBeginPeriod"(As Ulong=1) As Long
HighPrecisionClock
#EndIf

const graph_scr_x = 1920					'screenres
const graph_scr_y = 1080 
const graph_pix_per_vsync = 6				'wrong name for speed of the pipes per frame
const graph_xoffset = 1400					'draw location of neural netwerk
const graph_yoffset = 200
const graph_size = 50						'draw size of neural netwerk
const graph_set_fps = 60
dim as boolean graph_vsync_on

const pipe_with = 100						'with of the moving pipes
const pipe_interval = 100					'next pipe interval frame - pipe_rnd_interval
const pipe_rnd_interval = 40				'decided space betwin pipes
const pipe_max = 19							'max nummer of pipes in screen

const player_count = 199					'nummer of players
const player_size = 20						'display size of player (no collion check for size only center point)
const player_jump_interval = 30				'how many frames pass to flap(jump) again
const pipe_holesize = 0.25					'1 = full screen hole / 0 = no hole

'const gen_threats = 1 						'not use now
const gen_size = player_count				'nummer of players(bird) = nummer of neural networks(gen)
const gen_mutate_perc_top = 0.1		'top 0.1= 10% selection to mutate from
const gen_mutate_perc_weight = 0.1 			'how many weight are been modified
const gen_mutate_streght_weight = 1			'how mach to to modify the weights
const gen_update_fit_selection_frames = 50	'inteval how many times opdate the fittset brains
const gen_mutate_new = 0.2					'random new chance

const nn_node_in = 6
const nn_node_hidden = 5
const nn_layer_hidden = 1 					'nummer of hidden layers (0 = 1)
const nn_node_out = 2
const nn_bias = nn_node_in + (nn_node_hidden * (nn_layer_hidden+1)) + nn_node_out - 1

'pre calculation neural netwerk
const nn_weights_in_hidden    = nn_node_in * nn_node_hidden 		'connection from in to first hidden layer
const nn_weights_hidden_layer = (nn_node_hidden * nn_node_hidden) * nn_layer_hidden 	'hidden layer to layer connecions
const nn_weights_hidden_out   = nn_node_hidden * nn_node_out
const nn_weights = nn_weights_in_hidden + nn_weights_hidden_layer + nn_weights_hidden_out -1

const gen_fit_selection_size = (gen_size + 1) * gen_mutate_perc_top
const gen_fit_mutate_size = (nn_weights + 1) * gen_mutate_perc_weight

declare function round(n as single) as string
declare Function Regulate(Byval MyFps As long,Byref fps As long, reset_ as boolean = false) As long

type Neural_network
	as single bias(nn_bias)
	as single weight(nn_weights)
	as short ID
	as ulong fitness = 0
	as ulong best_fitness = 0
	declare constructor
end type

type nn_gen
	as Neural_network nn_brain(gen_size)
	as ushort nn_fit_selection(gen_fit_selection_size) 'gen pointer array value
	as ushort nn_fit_selection_counter = 0
	declare sub set_all_random()
	declare sub gen_draw_brain(a as ubyte)
	declare sub gen_calc_brain(a as ubyte)
	declare sub select_fittest()
	declare function is_ID_in_selection(s as ushort) as boolean
	declare sub mutate(n as ushort)
end type

type bird
	as single x, y
	as single old_x, old_y
	as single vel_y 'vel_x
	as single next_top_pipe_height
	as single next_bottom_pipe_height
	as single next_pipe_distance
	as long jump_timeout = 0
	as byte state = 1 '0 = ready, -1 = dead, 1 = active
	declare sub jump()
	declare sub go_left()
	declare sub go_right()
	declare sub process()
	declare sub respawn()
	declare constructor
end type

type pipe_hole
	as short x
	as short old_x
	as single top
	'as single old_top
	as longint active_time = 0
	as longint pipe_id
end type

type flappy_game
	as pipe_hole pipes(pipe_max)
	as longint count_pipe = 0
	as double start_time = 0
	declare sub draw_pipes()
	declare sub process_timers()
	declare sub add_pipe()
	declare sub remove_pipe()
	declare function check_player_collision(p as bird) as boolean
end type

type rnd8_16bit
	Union
		as ulong in32_rnd
		type
			as ubyte out8_a
			as ubyte out8_b
			as ushort out16_c
		end type
	end Union
	declare constructor
	declare function give16_rndXrnd() as single '-0.4999 to 0.4999
	declare function give16_rnd() as single     '0 to 0.9999999
end type

screenres graph_scr_x, graph_scr_y, 32, , 1
Dim stats_image As Any Ptr = ImageCreate( graph_scr_x, graph_scr_y, RGB(0, 64, 128) )

randomize timer,2
dim Shared as short pipe_count = 0
dim shared as longint graph_frame_counter = 1
dim as short stats_buffer(player_count)
dim as short stats_counter = 0
dim as single stats_scale = 0, stats_y_pointer = 0
dim as long fps
randomize timer
dim ai as nn_gen
ai.set_all_random()


start:  'note: ai is not reset
dim as string key
dim as bird player(player_count)
dim game as flappy_game
'game.start_time = 0
graph_frame_counter = 1
game.add_pipe()
player(0).state = 1

 graph_vsync_on = true
for tmp as ushort = 0 to 199
	graph_frame_counter += 1
	game.process_timers()
next

do
	dim as short stats_buffer(player_count)
	stats_counter = 0
	for i as ushort = 0 to gen_size
		if player(i).state >= 0 then
			stats_counter += 1
			stats_buffer(ai.nn_brain(i).ID) += 1
		end if
	next i
	if graph_vsync_on or (graph_frame_counter mod 30 ) = 0 then
		if stats_counter > 0 then
			stats_scale = (graph_scr_y - 1) / (stats_counter)
			stats_y_pointer = 0
			line stats_image, (graph_scr_x - 1, 0)-(graph_scr_x - 1, graph_scr_y-1),&h000000
			for i as ushort = 0 to gen_size 
				if stats_buffer(i) > 0 then
					line stats_image, (graph_scr_x - 1, stats_y_pointer)-(graph_scr_x - 1, stats_y_pointer + (stats_buffer(i) * stats_scale) - 2), ai.nn_brain(i).ID*647321
					stats_y_pointer += stats_buffer(i) * stats_scale
					'print i, stats_buffer(i), ai.nn_brain(i).fitness
				end if
			next i
			Put stats_image, (-1, 0), stats_image, PSet
		end if
	end if
	'print
	graph_frame_counter += 1
	key = inkey ' clear keyboard buffer
	game.process_timers()
	if (graph_frame_counter mod gen_update_fit_selection_frames) = 0 then ai.select_fittest()
	'if game.check_player_collision(player(0)) andalso player(0).state > 0 then beep: player(0).state = -1: goto start
	if stats_counter = 0  then ai.select_fittest()
	
	for i as short = 0 to gen_size
		player(i).process()
		'if stats_counter = 0 then 
		if player(i).state = -1 then player(i).respawn: ai.mutate(i)
		if game.check_player_collision(player(i)) then player(i).state = -1
		if player(i).state > 0 then
			ai.nn_brain(i).fitness += 1
			ai.nn_brain(i).bias(0)=player(i).next_top_pipe_height
			ai.nn_brain(i).bias(1)=player(i).next_bottom_pipe_height
			ai.nn_brain(i).bias(2)=-player(i).vel_y*5
			ai.nn_brain(i).bias(3)=player(i).next_pipe_distance / graph_scr_x
			ai.nn_brain(i).bias(4)=(player(i).x-(graph_scr_x/2)) / graph_scr_x
			ai.nn_brain(i).bias(5)=(player(i).y-(graph_scr_y/2)) / graph_scr_y
			ai.gen_calc_brain(i)
			
			if ai.nn_brain(i).bias(nn_weights_hidden_out) > .05 then player(i).jump
			if ai.nn_brain(i).bias(nn_weights_hidden_out+1) > .25 then player(i).go_right
			if ai.nn_brain(i).bias(nn_weights_hidden_out+1) < -.25 then player(i).go_left
		end if
	next i
	'if stats_counter = 0 then sleep
	
	if multikey(SC_UP) then player(0).jump
	if multikey(SC_LEFT) then player(0).go_left
	if multikey(SC_RIGHT) then player(0).go_right
	if multikey(SC_ESCAPE) then exit do
	if key = " " then
		regulate(graph_set_fps, fps, true) 'reset
		if graph_vsync_on then graph_vsync_on = false else graph_vsync_on = true
	end if
	if graph_vsync_on then sleep regulate(graph_set_fps, fps) else var tmp =  regulate(graph_set_fps, fps)
	 locate 1,1
	 
	screenlock
		'cls
		if graph_vsync_on then Put (0, 0), stats_image, PSet
		if graph_vsync_on then
			ai.gen_draw_brain(ai.nn_fit_selection(0))
			print stats_counter
			print ai.nn_fit_selection_counter
			print
			if ai.nn_fit_selection_counter > 0 then
				for i as ushort = 0 to ai.nn_fit_selection_counter - 1
					color ai.nn_brain( ai.nn_fit_selection(i)).id *647321
					print i ,ai.nn_fit_selection(i), ai.nn_brain(ai.nn_fit_selection(i)).best_fitness
				next i
				color RGB(255,255,255)
			end if
		end if
		game.draw_pipes()
		draw string (200,12),"Framerate = " & fps & " ( " & format(fps / graph_set_fps, "#") & "x )"
		draw string (graph_scr_x / 2 - 50, graph_scr_y / 2),"<Press spacebar> to run normal/full speed", &h000000
		for n as short = player_count to 0 step -1
			if player(n).state >= 0 then
					'circle (player(n).old_x, player(n).old_y*graph_scr_y), player_size, &h000000, , , , F
					if graph_vsync_on then if graph_vsync_on then circle (player(n).x, player(n).y*graph_scr_y), player_size, ai.nn_brain(n).ID*647321, , , , F
					'draw string (player(n).x-48, player(n).y*graph_scr_y-4),str(ai.nn_brain(n).ID), ai.nn_brain(n).ID*647321
					if graph_vsync_on then circle (player(n).x, player(n).y*graph_scr_y), player_size-2, &h000000,
					circle (player(n).x, player(n).y*graph_scr_y), player_size, &hFFFFFF,
			end if
		next
		'locate 1,50: print graph_frame_counter
	screenunlock
loop
sleep
end


sub bird.go_left()
	if state = 0 then state = 1
	x -=5
	if x < 0 then x = 0
end sub

sub bird.go_right()
	if state = 0 then state = 1
	x +=5
	if x > graph_scr_x-1 then x = graph_scr_x - 1
end sub

sub bird.jump()
	if state = 0 then state = 1
	if (graph_frame_counter - jump_timeout) > player_jump_interval then
		vel_y /= 2.5
		vel_y -= 0.015
		jump_timeout = graph_frame_counter
	end if
end sub

sub bird.process()
	if state = 1 then
		'Gravity
		old_x = x
		old_y = y
		vel_y += 0.0005
		y += vel_y
	end if
end sub

sub bird.respawn()
	x = 200+rnd*500
	y = rnd*.8 + 0.1
	state = 1
	vel_y = 0
end sub

constructor bird
	x = 200+rnd*500
	y = rnd*.8 + 0.1
end constructor

sub flappy_game.process_timers()
	'print (graph_frame_counter - start_time)
	if (graph_frame_counter - start_time) > pipe_interval then start_time += (graph_frame_counter - start_time)-rnd*  pipe_rnd_interval: add_pipe()
end sub

sub flappy_game.remove_pipe()
	for i as ubyte = 0 to pipe_max-1
		pipes(i) = pipes(i + 1)
	next i
	pipes(pipe_max).active_time = 0
	pipe_count -=1
end sub

sub flappy_game.add_pipe()
	dim as single pipe_rnd 
	pipe_count += 1
	for i as ubyte = 0 to pipe_max
		if pipes(i).active_time = 0 then
			count_pipe +=1
			pipes(i).pipe_id = count_pipe
			pipes(i).active_time = graph_frame_counter
			pipe_rnd = rnd * (1 - pipe_holesize )
			pipes(i).top = pipe_rnd
			pipes(i).x = graph_scr_x - (graph_frame_counter - pipes(i).active_time) * graph_pix_per_vsync
			pipes(i).old_x = pipes(i).x
			exit sub
		end if
	next i
end sub

sub flappy_game.draw_pipes()
	dim as longint t = graph_frame_counter
	for i as ubyte = 0 to pipe_max
		if pipes(i).active_time <> 0 then
			'line (pipes(i).old_x, 0)-(pipes(i).old_x + pipe_with, graph_scr_y*pipes(i).top), &h000000, BF
			'line (pipes(i).old_x, graph_scr_y * (pipes(i).top + pipe_holesize))-(pipes(i).old_x + pipe_with, graph_scr_y-1), &h000000, BF
			'draw string (pipes(i).old_x+ pipe_with/2-4, 8), str(pipes(i).pipe_id), &h000000
			
			pipes(i).old_x = pipes(i).x
			if pipes(i).old_x < -pipe_with then
				remove_pipe(): i -=1
			else
				pipes(i).x = graph_scr_x - (t - pipes(i).active_time) * graph_pix_per_vsync
				line (pipes(i).x, 0)-(pipes(i).x + pipe_with, graph_scr_y*pipes(i).top), &hAAAAAA, BF
				line (pipes(i).x, graph_scr_y * (pipes(i).top + pipe_holesize))-(pipes(i).x + pipe_with, graph_scr_y-1), &h888888, BF
				'draw string (pipes(i).old_x+ pipe_with/2-4, 8), str(pipes(i).pipe_id), &h000000
			end if
		end if
	next i
end sub

function flappy_game.check_player_collision(p as bird) as boolean
	dim as short x1 = any
	dim as short x2 = any
	dim as short y1 = any
	dim as short y2 = any
	dim as short py = any
	dim as short last_x2 = 0
	
	if p.y > 1 then return true
	if p.x <= 0 then return true
	
	for i as ubyte = 0 to pipe_count-1
		x1 = pipes(i).x
		x2 = (pipes(i).x + pipe_with)
		
		if x1 < p.x andalso x2 > p.x then
			y1 = graph_scr_y * pipes(i).top
			y2 = graph_scr_y * (pipes(i).top + pipe_holesize)
			py = p.y * graph_scr_y
			if y1 < py andalso y2 > py then
				'line (x1, y1)-(x2, y2), &h00FF00, bf
			else
				return true
			end if
		end if
		
		if last_x2 < x2 andalso x2 > p.x then
			'look for next colision height
			last_x2 = x2
			
			'y1 = graph_scr_y * pipes(i).top
			'y2 = graph_scr_y * (pipes(i).top + pipe_holesize)
			'line (0, y1+1)-(x2, y1+1), &hff0000
			'line (0, y2-1)-(x2, y2-1), &hff0000
			
			p.next_top_pipe_height = p.y - pipes(i).top
			p.next_bottom_pipe_height = -(p.y - (pipes(i).top + pipe_holesize))
			p.next_pipe_distance = x1
			'draw string (x2, y1-8),"1 = " & p.next_top_pipe_height
			'draw string (x2, y2),"2 = " & p.next_bottom_pipe_height
			
			exit for ' need pipe do not need checks now
		end if
	next i
end function


Function Regulate(Byval MyFps As long,Byref fps As long, reset_ as boolean = false) As long
	Static As Double timervalue, _LastSleepTime, t3, frames
	if reset_  then timervalue = 0: _LastSleepTime = 0: t3 = 0: frames = 0: fps = 0
	frames += 1
	If (Timer-t3) >= 1 Then t3 = Timer: fps = frames: frames = 0
	dim as double sleeptime=_LastSleepTime + ((1 / myfps) - Timer + timervalue) * 1000
	If sleeptime<1 Then sleeptime = 1
	_LastSleepTime = sleeptime
	timervalue = Timer
	Return sleeptime
End Function


'-------------------------------------------------Neural netwerk -------------------------

sub nn_gen.mutate(n as ushort)
	static as ushort loop_counter :loop_counter += 1
	if nn_fit_selection_counter = 0 then exit sub
	dim as ushort loop_select = loop_counter mod nn_fit_selection_counter
	dim as ushort random_select = nn_fit_selection(loop_select)
	
	if rnd < gen_mutate_new then nn_brain(n).constructor: nn_brain(n).ID = n
	
	if is_ID_in_selection(n) = false then
	
		nn_brain(n) = nn_brain(random_select) ' copy is done
		if nn_brain(n).fitness > nn_brain(n).best_fitness then nn_brain(n).best_fitness = nn_brain(n).fitness
		nn_brain(n).fitness = 0
		
		
		dim as ushort b 
		for a as ubyte = 0 to gen_fit_mutate_size -1
			dim as rnd8_16bit pole
			b = nn_weights * pole.give16_rnd
			nn_brain(n).weight(b) += pole.give16_rndXrnd
			'if nn_brain(n).weight(b) > 1 then nn_brain(n).weight(b) = 1
			'if nn_brain(n).weight(b) < -1 then nn_brain(n).weight(b) = -1
		next a
		'print
		'print nn_weights, gen_fit_mutate_size
		'print gen_mutate_streght_weight
		'sleep
		'end
	end if
	if nn_brain(n).fitness > nn_brain(n).best_fitness then nn_brain(n).best_fitness = nn_brain(n).fitness
	nn_brain(n).fitness = 0
end sub


function nn_gen.is_ID_in_selection(s as ushort) as boolean
	if nn_fit_selection_counter > 0 then
		for i as short = 0 to nn_fit_selection_counter-1
			if nn_brain(nn_fit_selection(i)).ID = nn_brain(s).ID then return true : exit function
		next i
	end if
end function


sub nn_gen.select_fittest()
	dim as ushort highest_fitness_found = 0, highest_fitness_found_old
	dim as ubyte found
	dim as byte selection(gen_size)		'pre mask array to nn_selection(gen__selection_size)
	'dim as ushort 'fit_counter = 0
	nn_fit_selection_counter = 0
	
	for a as ushort = 0 to gen_size
		if nn_brain(a).fitness > highest_fitness_found then highest_fitness_found = nn_brain(a).fitness
	next
	
	'print
	'print gen_fit_selection_size, highest_fitness_found
	highest_fitness_found_old = highest_fitness_found
	do
		found = 0
		for a as ushort  = 0 to gen_size
			if selection(a) = 0 andalso nn_brain(a).fitness = highest_fitness_found then
				selection(a) = 1
				if is_ID_in_selection(a) = false then
					nn_fit_selection(nn_fit_selection_counter) = a
					'fit_counter += 1
					nn_fit_selection_counter += 1
					found = 1
				end if
				'print is_ID_in_selection(a)
				exit for
			end if
		next
		if highest_fitness_found = 0 then exit do
		if found = 0 then highest_fitness_found -= 1 ' this is slow!!!!!!!!!!!!!!!!!!!!
		if (highest_fitness_found_old /10) > highest_fitness_found then exit do
	loop until nn_fit_selection_counter >= gen_fit_selection_size ' leave loop when fit selection is full
	
		'call mutate after this
end sub


sub nn_gen.gen_calc_brain(a as ubyte)
	Clear nn_brain(a).bias(nn_node_in), 0, (nn_bias-nn_node_in+1) * SizeOf(single) 'clear als outputs
	dim as single output_offset = nn_node_in
	dim as single input_offset = 0
	dim as ushort count_weight = 0
	
	for c as ubyte = 0 to nn_node_hidden - 1
		for b as ubyte = 0 to nn_node_in - 1
			nn_brain(a).bias(output_offset+c) += nn_brain(a).bias(input_offset+b) * nn_brain(a).weight(count_weight)
			count_weight += 1
		next
		'nn_brain(a).bias(output_offset+c) /= nn_node_in
	next
	
	if nn_layer_hidden > 0 then
		for d as ubyte = 0 to nn_layer_hidden -1
			output_offset = nn_node_in + (nn_node_hidden * (d+1)) 
			input_offset = nn_node_in + (nn_node_hidden * d) 
			'print "mid input_offset"; input_offset
			'print "mid output_offset"; output_offset
			for c as ubyte = 0 to nn_node_hidden - 1
				for b as ubyte = 0 to nn_node_hidden - 1
					nn_brain(a).bias(output_offset+c) += nn_brain(a).bias(input_offset+b) * nn_brain(a).weight(count_weight)
					count_weight += 1
				next
				'nn_brain(a).bias(output_offset+c) /= nn_node_hidden
			next
		next
	end if
	
	
	output_offset = nn_node_in + (nn_node_hidden * (nn_layer_hidden+1)) 
	input_offset = nn_node_in + (nn_node_hidden * (nn_layer_hidden))
	for c as ubyte = 0 to nn_node_out - 1
		for b as ubyte = 0 to nn_node_hidden - 1
			nn_brain(a).bias(output_offset+c) += nn_brain(a).bias(input_offset+b) * nn_brain(a).weight(count_weight)
			count_weight += 1
		next
		'nn_brain(a).bias(output_offset+c) /= nn_node_hidden
	next
end sub

'subs and functions
sub nn_gen.set_all_random()
	for a as ushort = 0 to gen_size
		nn_brain(a).ID = a
	next
end sub

constructor Neural_network
	for c as ushort = 0 to nn_weights
		weight(c) = rnd * 2 -1
	next
	fitness = 0
	best_fitness = 0
end constructor


sub nn_gen.gen_draw_brain(a as ubyte)
	dim as short in_offset = ((nn_node_in - 1) / 2 * graph_size)
	dim as short hidden_offset = ((nn_node_hidden - 1) / 2 * graph_size)
	dim as short out_offset = ((nn_node_out - 1) / 2 * graph_size)
	dim as short x, y, x2, y2
	dim as ushort count_bias = 0
	dim as ushort count_weight = 0
	
	'part weights
	for c as ubyte = 0 to nn_node_hidden - 1
		for b as ubyte = 0 to nn_node_in - 1
			x = graph_xoffset
			y = graph_yoffset + b*graph_size - in_offset
			x2 = graph_xoffset + graph_size*2
			y2 = graph_yoffset + c*graph_size - hidden_offset
			line(x, y)-(x2, y2), &hff00ff
			draw string ((x+x2)/2 - 16,(y+y2)/2-8+b*8),round(nn_brain(a).weight(count_weight)), &hFFFF00
			count_weight += 1
		next
	next
	
	if nn_layer_hidden > 0 then
		for e as ubyte = 0 to nn_layer_hidden-1
			for d as ubyte = 0 to nn_node_hidden - 1
				 for f as ubyte = 0 to nn_node_hidden - 1
					x = graph_xoffset + (4 + (e-1)*2) * graph_size
					y = graph_yoffset + d*graph_size - hidden_offset
					x2 = graph_xoffset + (4 + (e)*2) * graph_size
					y2 = graph_yoffset + f*graph_size - hidden_offset
					line(x, y)-(x2, y2), &hff00ff
					draw string ((x+x2)/2 - 16,(y+y2)/2-8+f*8),round(nn_brain(a).weight(count_weight)), &hFFFF00
					count_weight += 1
				next
			next
		next
	end if
	
	for d as ubyte = 0 to nn_node_hidden - 1
		 for f as ubyte = 0 to nn_node_out - 1
			x = graph_xoffset + (4 + (nn_layer_hidden-1)*2) * graph_size
			y = graph_yoffset + d*graph_size - hidden_offset
			x2 = graph_xoffset + (4 + nn_layer_hidden*2) * graph_size
			y2 = graph_yoffset + f*graph_size - out_offset
			line(x, y)-(x2, y2), &hff00ff
			draw string ((x+x2)/2 - 16,(y+y2)/2-8+f*8),round(nn_brain(a).weight(count_weight)), &hFFFF00
			count_weight += 1
		next
	next
	
	' part bias
	for b as ubyte = 0 to nn_node_in - 1
		x = graph_xoffset
		y = graph_yoffset + b*graph_size - in_offset
		circle (x, y), graph_size/4 , &hFF0000
		draw string (x-12,y-4), round(nn_brain(a).bias(count_bias)), &hFFFFff
		count_bias += 1
	next
	for c as ubyte = 0 to nn_node_hidden - 1
		x = graph_xoffset + graph_size*2
		y = graph_yoffset + c*graph_size - hidden_offset
		circle (x, y), graph_size/4 , &h00FF00
		draw string (x-12,y-4), round(nn_brain(a).bias(count_bias)), &hFFFFff
		count_bias += 1
	next
	if nn_layer_hidden > 0 then
		for e as ubyte = 0 to nn_layer_hidden -1
			for d as ubyte = 0 to nn_node_hidden - 1
				x = graph_xoffset + (4 + e*2) * graph_size
				y = graph_yoffset + d*graph_size - hidden_offset
				circle (x, y), graph_size/4 , &h00FF00
				draw string (x-12,y-4),round(nn_brain(a).bias(count_bias)), &hFFFFff
				count_bias += 1
			next
		next
	end if
	for f as ubyte = 0 to nn_node_out - 1
		x = graph_xoffset + (4 + nn_layer_hidden*2) * graph_size
		y = graph_yoffset + f*graph_size - out_offset
		circle (x, y), graph_size/4 , &h0000FF
		draw string (x-12,y-4),round(nn_brain(a).bias(count_bias)), &hFFFFff
		count_bias += 1
	next
end sub

function round(n as single) as string
	return Format(n, "#.##")
end function

constructor rnd8_16bit
	in32_rnd = rnd * 4294967296
end constructor

function rnd8_16bit.give16_rndXrnd() as single
	return ((out8_a-128) * (out8_b-128)) /16128
end function 

function rnd8_16bit.give16_rnd() as single
	return out16_c / 65535.002
end function 

BasicCoder2
Posts: 3906
Joined: Jan 01, 2009 7:03
Location: Australia

Re: Neural network - flappy bird

Post by BasicCoder2 »

Code too complex for me to figure out what is wrong. Is it in the net or how it is trained or how the inputs to the net are selected or pre-processed. From what I have read using an ANN is a bit of a black art and can involve some tweaking to get working results.
Maybe test your network on other problems known to be solved using an ANN?
Gunslinger
Posts: 103
Joined: Mar 08, 2016 19:10
Location: The Netherlands

Re: Neural network - flappy bird

Post by Gunslinger »

Black art eh.. i think the probleem is with respanning right again.
And spawn insite a pipes to die right again. So even fit one's have no fare chance to get selected again.
Let me try respawn all and reset world.
And make code a bit more organized.

At some point i had very good bird with complex behavings.
But that was only a traind mutaton of the first best without compention from others.
After change the selection it kind of broke.
And this is the simplest test or start game i can think off.
I know it works but never seems to master flapping the wings right all the times.

Give me some time to get back to this.
Thanks for your reply.
Gunslinger
Posts: 103
Joined: Mar 08, 2016 19:10
Location: The Netherlands

Re: Neural network - flappy bird

Post by Gunslinger »

Update to Neural network - flappy bird

Removed a lot of bugs on selection of fittest an others bugs
Training is improved a lot with the add of cross mutate of fittest and also by tunning the settings more.
added a very precise time display of cpu usage, make it clear the selecion of fittest need some more work.
I give it more inputs and output then is needed jump, left and right controlles to play with.

I'm very happy how it performance now.
So strange i only have a basic understanding of general neural networks and got it this point.
As BasicCoder2 pointed out it really seems black art.
Also the cpu time indicater is very usefull tool to use for other projects i think.

compile with -gen gcc -Wc -O3

have fun.

Code: Select all

#include "fbgfx.bi"
#include "string.bi"
#if __FB_LANG__ = "fb"
Using FB '' Scan code constants are stored in the FB namespace in lang FB
#endif
#If Defined(__FB_WIN32__) 'windows timers resolution update
Declare Function HighPrecisionClock Lib "winmm" Alias "timeBeginPeriod"(As Ulong=1) As Long
HighPrecisionClock
#EndIf

'#Define DEBUG

dim shared as boolean graph_vsync_on = true
dim shared as single avg_pipe_global
dim shared as ulong ID_counter = 0
const graph_scr_x = 1920					'screenres
const graph_scr_y = 1080 
const graph_pix_per_vsync = 6				'wrong name for speed of the pipes per frame
const graph_xoffset = 1300					'draw location of neural netwerk
const graph_yoffset = 250
const graph_size = 50						'draw size of neural netwerk
const graph_set_fps = 60

const player_count = 1000					'nummer of players
const player_size = 20						'display size of player (no collion check for size only center point)
const player_jump_interval = 20				'how many frames pass to flap(jump) again

const pipe_with = 100						'with of the moving pipes
const pipe_interval = 100					'next pipe interval frame - pipe_rnd_interval
const pipe_rnd_interval = 40				'decided space betwin pipes
const pipe_max = 9							'max nummer of pipes in screen
const pipe_holesize = 0.25					'1 = full screen hole / 0 = no hole

'const gen_threats = 1 						'not use now
const gen_size = player_count				'nummer of players(bird) = nummer of neural networks(gen)
const gen_mutate_perc_top = 0.025			'top 0.1= 10% selection to mutate from
const gen_mutate_perc_weight = 0.5 			'Max how many weight are been modified on mutate 
const gen_mutate_streght_weight = 10		'Max how mach to to modify the weights
const gen_mutate_prec_dead_weight = 0.0		'Max 0.1 = 10%, 0 weights value connection
const gen_mutate_prec_fitnessToKeep = 0.25	'% fittnes to keep after mutate
const gen_mutate_cross_perc = 0.01 		'chance to cross mutate
'const gen_update_fit_selection_frames = 100	'inteval how many times update the fittset brains

const node_in = 8							'inputs
const node_hidden = 7						'mid layers note count
const layer_hidden = 2 						'nummer of hidden layers (0 = 1)
const node_out = 3							'jump / left or right
const bias = node_in + (node_hidden * (layer_hidden+1)) + node_out - 1 ' do not change

'pre calculation neural netwerk (do not change)
const weights_in_hidden    = node_in * node_hidden 		'connection from in to first hidden layer
const weights_hidden_layer = (node_hidden * node_hidden) * layer_hidden 	'hidden layer to layer connecions
const weights_hidden_out   = node_hidden * node_out
const weights = weights_in_hidden + weights_hidden_layer + weights_hidden_out -1
if weights >= 255 then print "To many neural network weights": end -1

const as ushort gen_fit_selection_size = (gen_size + 1) * gen_mutate_perc_top
const as ushort gen_fit_mutate_size = (weights + 1) * gen_mutate_perc_weight

declare function round(n as single) as string
declare Function Regulate(Byval MyFps As long,Byref fps As long, reset_ as boolean = false) As long

function nummerToRGB(i as longint) as long
	i = i * 647328+125
	return rgb(i mod 266, (i\256) mod 256, (i\65536) mod 256)
end function 

type Neural_network
	Public:
	as single bias(bias)
	as single weight(weights)
	as long ID
	as double mutated = 0
	as ulong fitness = 0
	as ulong best_fitness = 0
	declare constructor
	as single perc_weight = rnd * gen_mutate_perc_weight '0.2 			'how many weight are been modified on mutate 
	as single streght_weight = rnd * gen_mutate_streght_weight '50		'how mach to to modify the weights
	as single dead_weight = rnd * gen_mutate_prec_dead_weight '0.0		'dead weight connection
	as ushort cels_alive = 0
end type

type gen
	Public:
	as Neural_network brain(gen_size)
	as Neural_network fit_selection(gen_fit_selection_size) 'gen pointer array value
	as ushort fit_selection_counter = 0
	declare sub gen_draw_brain(a as ushort)
	declare sub gen_calc_brain(a as ushort)
	declare sub select_fittest()
	declare function is_ID_in_selection(s as ushort) as ushort
	declare sub mutate(n as ushort)
	declare function CrossMutate(a as Neural_network, b as Neural_network) as Neural_network
end type

type bird
	Public:
	as single x, y
	as single old_x, old_y
	as single vel_y 'vel_x
	as single next_top_pipe_height
	as single next_bottom_pipe_height
	as single next_pipe_distance
	as single next2_top_pipe_height
	as single next2_bottom_pipe_height
	as long jump_timeout = 0
	as byte state = 1 '0 = ready, -1 = dead, 1 = alive
	declare sub jump()
	declare sub go_left()
	declare sub go_right()
	declare sub process()
	declare sub respawn()
	declare constructor
end type

type pipe_hole
	Public:
	as short x
	as short old_x = graph_scr_y
	as single top
	as single old_top
	as longint active_time = 0
	as longint pipe_id
end type

type flappy_game
	Public:
	dim as pipe_hole pipes(pipe_max)
	as longint count_pipe = 0
	as double start_time = 0
	declare sub draw_pipes()
	declare sub process_timers()
	declare sub add_pipe()
	declare sub remove_pipe()
	declare function check_player_collision(p as bird) as boolean
end type

type rnd8_16bit
	Union
		as ulong in32_rnd ' = input 32bit
		type
			as ubyte out8_a
			as ubyte out8_b
			as ushort out16_c
		end type
	end Union
	declare constructor
	declare function give16_rndXrnd() as single '-0.99999 to 0.99999
	declare function give16_rnd() as single     '0 to 0.99999
end type

type moving_avg
	as double array(255)
	as ubyte arrayIndex
	as double ReturnSom
	declare function MA(in as double, lengte as ubyte) as long
end type

Enum MyState
	timer_startup = 0
	timer_stats
	timer_neural
	timer_collision
	timer_render
	timer_renderback
	timer_renderPipes
	timer_renderBrain
	timer_renderList
	timer_mutate
	timer_screensync
	timer_selectfit
End Enum

type cpu_time
	as double current_timer = timer
	as moving_avg timers(19)
	as MyState current_index = 0
	declare sub state(s as MyState)
end type

sub cpu_time.state(s as MyState)
	dim as double this_time = timer
	dim as double passed_time = this_time - current_timer
	if s <> current_index then 
		timers(current_index).ma(passed_time, 254)
		current_index = s
		current_timer = this_time
	end if
end sub

dim cputime as cpu_time
cputime.state(timer_startup)

randomize timer, 2
screenres graph_scr_x, graph_scr_y, 32,, 1
Dim stats_image As Any Ptr = ImageCreate( graph_scr_x, graph_scr_y, RGB(0, 64, 128) )
dim shared ai as gen
dim fitness as moving_avg
dim cels as moving_avg
dim shared as short pipe_count = 0
dim shared as longint graph_frame_counter = 0 
dim as short stats_counter = 0
dim as long fps
dim as string key
dim as bird player(player_count)
dim game as flappy_game
dim as ushort OneAliveIndex = 0
dim as double run_time = timer
dim as ushort run_time_count = 0
for tmp as ushort = 0 to 249 ' pre-render some pipes movement
	graph_frame_counter += 1
	game.process_timers()
next

cls
locate 1,1
color &hFFFFFF
print "count_pipe = " & game.count_pipe
'sleep

dim as ulong highest_fitness_alive, old_highest_fitness_alive
do
	cputime.state(timer_stats)
	dim as short stats_buffer(player_count)
	stats_counter = 0
	old_highest_fitness_alive = highest_fitness_alive
	highest_fitness_alive = 0
	for i as ushort = 0 to gen_size ' count the players alive
		if player(i).state >= 0 then
			if ai.brain(i).fitness > highest_fitness_alive then highest_fitness_alive = ai.brain(i).fitness: OneAliveIndex = i
			stats_counter += 1
		end if
	next i
	
	graph_frame_counter += 1 'to next frame
	avg_pipe_global = ((game.pipes(0).top + pipe_holesize) + game.pipes(0).top) / 2
	key = inkey ' clear keyboard buffer
	game.process_timers()
	
	cputime.state(timer_selectfit)
	if stats_counter = 0 then '*respawn all becouse of all dead fit-list first then random fitlist pick with mutation
		game.remove_pipe()
		game.remove_pipe()
		for i as short = 0 to gen_size
			if ai.brain(i).fitness > ai.brain(i).best_fitness then ai.brain(i).best_fitness = ai.brain(i).fitness
		next
		ai.select_fittest()
	end if
	
	cputime.state(timer_mutate)
	if stats_counter = 0 then '*respawn all becouse of all dead fit-list first then random fitlist pick with mutation
		for i as short = 0 to gen_size 'one less for randow neiuwe
			player(i).respawn
			game.check_player_collision(player(i))
			if i < ai.fit_selection_counter then 'first fit to 0 - selection
				ai.brain(i) = ai.fit_selection(i)
				ai.brain(i).fitness *= gen_mutate_prec_fitnessToKeep
			else
				if rnd < gen_mutate_cross_perc then 'cross mutate here
					ai.brain(i) = ai.CrossMutate( ai.fit_selection(rnd * ai.fit_selection_counter), ai.fit_selection(rnd * ai.fit_selection_counter) )
					ai.brain(i).mutated += 1
					ai.brain(i).ID = ID_counter
					ai.brain(i).best_fitness = 0
					ai.brain(i).fitness *= gen_mutate_prec_fitnessToKeep
					ai.brain(i).perc_weight = rnd * gen_mutate_perc_weight '0.2 			'how many weight are been modified on mutate 
					ai.brain(i).streght_weight = rnd * gen_mutate_streght_weight '50		'how mach to to modify the weights
					ai.brain(i).dead_weight = rnd * gen_mutate_prec_dead_weight '0.0		'dead weight connection
					ID_counter += 1
				else 'normal mutate here
					ai.mutate(i)
				end if
			end if
		next i
		
		dim as long avg_top
		for i as short = 0 to (ai.fit_selection_counter \ 5)
			avg_top += ai.fit_selection(i).fitness
		next i
		avg_top /= (ai.fit_selection_counter \ 10)
		
		line stats_image, (graph_scr_x - 1, 0)-(graph_scr_x - 1, graph_scr_y - 1), rgb(0, 0, 0)
		if timer - run_time > 60 then 
			run_time_count += 1
			line stats_image, (graph_scr_x - 1, 0)-(graph_scr_x - 1, graph_scr_y - 1), rgb(127, 127, 0)
			draw string stats_image, (graph_scr_x - 2 -len(str(run_time_count )) * 8, 0), str(run_time_count), rgb(127, 127, 0)
			run_time = timer
		end if
		
		line stats_image, (graph_scr_x - 1, graph_scr_y - 1)-(graph_scr_x - 1, graph_scr_y - 1 - (old_highest_fitness_alive/500)), rgb(0, 0, 127)
		line stats_image, (graph_scr_x - 1, graph_scr_y - 1)-(graph_scr_x - 1, graph_scr_y - 1 - (avg_top/500)), rgb(63, 63, 63)
		'circle stats_image, (graph_scr_x - 1, graph_scr_y - 1 - (cels.ma(ai.fit_selection(0).cels_alive, 20)*4 )),2 , rgb(63, 255, 63),,,,f
		circle stats_image, (graph_scr_x - 1, graph_scr_y - 1 - (fitness.ma(avg_top, 50)/500)),2 , rgb(255, 63, 63),,,,f
		Put stats_image, (-1, 0), stats_image, PSet
	end if
	
	
	cputime.state(timer_collision)
	for i as short = 0 to gen_size
		if player(i).state > 0 then
			if game.check_player_collision(player(i)) then player(i).state = -1 'dieded
		end if
	next i
	
	cputime.state(timer_neural)
	for i as short = 0 to gen_size
		player(i).process()
		if player(i).state > 0 then
			ai.brain(i).fitness += 1
			ai.brain(i).bias(0)=player(i).next_top_pipe_height
			ai.brain(i).bias(1)=player(i).next_bottom_pipe_height
			ai.brain(i).bias(2)=player(i).next_pipe_distance / (graph_scr_x / 2)
			ai.brain(i).bias(3)=-player(i).vel_y*100
			ai.brain(i).bias(4)=csng((player(i).x-(graph_scr_x/2)) / (graph_scr_x / 2))
			ai.brain(i).bias(5)=csng((player(i).y-(graph_scr_y/2)) / (graph_scr_y / 2))
			ai.brain(i).bias(6)=player(i).next2_top_pipe_height
			ai.brain(i).bias(7)=player(i).next2_bottom_pipe_height
			ai.gen_calc_brain(i)
			
			if ai.brain(i).bias(weights_hidden_out) > .0 then player(i).jump
			if ai.brain(i).bias(weights_hidden_out+1) > 0 then player(i).go_right': ai.brain(i).Fitness += 1 'extra fitness for moving forward player
			if ai.brain(i).bias(weights_hidden_out+2) > 0 then player(i).go_left': ai.brain(i).Fitness -= 1
		end if
	next i
	
	cputime.state(timer_screensync)
	if lcase(key) = "s" then sleep
	if multikey(SC_ESCAPE) then exit do
	if key = " " then
		regulate(graph_set_fps, fps, true) 'reset
		if graph_vsync_on then graph_vsync_on = false else graph_vsync_on = true
	end if
	if graph_vsync_on then sleep regulate(graph_set_fps, fps) else var tmp = regulate(graph_set_fps, fps)
	
	locate 1,1 
	screenlock
	cputime.state(timer_renderback)
	if graph_vsync_on or stats_counter = 0 then Put (0, 0), stats_image, PSet
	print "Framerate = " & fps & " ( " & format(fps / graph_set_fps, "#") & "x )"
	'if highest_fitness_alive = 2000000 then graph_vsync_on = true: print "we made it": print "we made it": print "we made it": print "we made it": print "we made it": print "we made it": print "we made it"
	'if highest_fitness_alive = 2000000 then player(OneAliveIndex).state = -1 ' kill the fitest
	
	cputime.state(timer_renderList)
	if ai.fit_selection_counter > 0 and (stats_counter = 0 or graph_vsync_on = true) then
		if graph_vsync_on then print "active players: " & stats_counter
		print
		print "List", "Ai.id to copy",, "top fitness", "best fitness","Versie=Cross.mutation","%Mutate","%Strengt","%deadweight", "Alive cels"
		for i as ushort = 0 to ai.fit_selection_counter - 1
			color nummerToRGB(ai.fit_selection(i).id)
			print i, ai.fit_selection(i).id,,
			print     ai.fit_selection(i).fitness,
			print     ai.fit_selection(i).best_fitness,
			print using "#####.######"; ai.fit_selection(i).mutated;
			#If Defined(DEBUG) 'count cels
				color rgb(0, ai.fit_selection(i).perc_weight*255*20, 0)
				print ,,ai.fit_selection(i).perc_weight,
				color rgb(0, ai.fit_selection(i).streght_weight*127, 0)
				print ai.fit_selection(i).streght_weight,
				color rgb(0, ai.fit_selection(i).dead_weight*255*2, 0)
				print ai.fit_selection(i).dead_weight,
				color rgb(127,127,127)
				print ai.fit_selection(i).cels_alive, ""
			#else
				print
			#endif
		next i
		color RGB(255,255,255)
		print
		print using "timer startup    ###.########"; cputime.timers(timer_startup).ReturnSom
		print using "timer stats      ###.########"; cputime.timers(timer_stats).ReturnSom
		print using "timer neural     ###.########"; cputime.timers(timer_neural).ReturnSom
		print using "timer collision  ###.########"; cputime.timers(timer_collision).ReturnSom
		print using "timer render     ###.########"; cputime.timers(timer_render).ReturnSom
		print using "timer renderBack ###.########"; cputime.timers(timer_renderback).ReturnSom
		print using "timer renderList ###.########"; cputime.timers(timer_renderlist).ReturnSom
		print using "timer renderPipes###.########"; cputime.timers(timer_renderpipes).ReturnSom
		print using "timer renderBrain###.########"; cputime.timers(timer_renderbrain).ReturnSom
		print using "timer selectFit  ###.########"; cputime.timers(timer_selectfit).ReturnSom
		print using "timer mutate     ###.########"; cputime.timers(timer_mutate).ReturnSom
		print using "timer screensync ###.########"; cputime.timers(timer_screensync).ReturnSom
	end if
	
	cputime.state(timer_renderPipes)
	game.draw_pipes() 'and calculate move pipes
	cputime.state(timer_renderBrain)
	if graph_vsync_on then
		ai.gen_draw_brain(OneAliveIndex)
		draw string (graph_scr_x / 2 - 50, graph_scr_y / 2),"<Press spacebar> to switch normal or full speed", &h0FFF00
	end if
	
	cputime.state(timer_render)
	for n as short = player_count to 0 step -1
		if player(n).state >= 0 then
				'circle (player(n).old_x, player(n).old_y*graph_scr_y), player_size, &h000000, , , , F 
				'draw string (player(n).x-48, player(n).y*graph_scr_y-4),str(ai.brain(n).ID), nummerToRGB(ai.brain(n).ID)
				if graph_vsync_on then 
					circle (player(n).x, player(n).y*graph_scr_y), player_size, nummerToRGB(ai.brain(n).ID), , , , F
					circle (player(n).x, player(n).y*graph_scr_y), player_size-2, &h000000,
					draw string (player(n).x-8, player(n).y*graph_scr_y-4),str(int(ai.brain(n).mutated)), rgb(0,0,0)
					circle (player(n).x, player(n).y*graph_scr_y), player_size, &hFFFFFF,
				end if
		end if
	next
	if graph_vsync_on then
		line (player(OneAliveIndex).x-player_size*2, player(OneAliveIndex).y*graph_scr_y)-step(player_size*4, 0), RGB(255,255,255)
		line (player(OneAliveIndex).x, player(OneAliveIndex).y*graph_scr_y-player_size*2)-step(0, player_size*4), RGB(255,255,255)
	end if
	'locate 1,50: print graph_frame_counter
	screenunlock
loop

ImageDestroy stats_image
sleep
end


sub bird.go_left()
	'if state = 0 then state = 1
	x -=5
	if x < 0 then x = 0
end sub

sub bird.go_right()
	'if state = 0 then state = 1
	x +=5
	if x > graph_scr_x-1 then x = graph_scr_x - 1
end sub

sub bird.jump()
	'if state = 0 then state = 1
	if (graph_frame_counter - jump_timeout) > player_jump_interval then
		vel_y /= 2.5
		vel_y -= 0.015
		jump_timeout = graph_frame_counter
	end if
end sub

sub bird.process()
	if state = 1 then
		'Gravity
		old_x = x
		old_y = y
		vel_y += 0.00075
		y += vel_y
	end if
end sub

sub bird.respawn()
	x = 200+rnd*50
	'y = rnd*.8 + 0.1
	y = avg_pipe_global+ ((rnd-.5) / 20)
	state = 1
	vel_y = 0
end sub

constructor bird
	x = 200+rnd*50
	y = rnd*.6 + 0.2
	state = 1
end constructor

sub flappy_game.process_timers()
	'print (graph_frame_counter - start_time)
	if (graph_frame_counter - start_time) > pipe_interval then start_time += (graph_frame_counter - start_time)-rnd*  pipe_rnd_interval: add_pipe()
end sub

sub flappy_game.remove_pipe()
	if pipe_count > 0 then
		'beep
		for i as ubyte = 0 to pipe_count-1
			pipes(i) = pipes(i + 1)
		next i
		'pipes(0).active_time = 0
		'pipes(pipe_count-1).active_time = 0
		pipe_count -=1
	end if
end sub

sub flappy_game.add_pipe()
	if pipe_count < pipe_max then
		dim as single pipe_rnd 
		pipe_count += 1
		for i as ubyte = 0 to pipe_max
			if pipes(i).active_time = 0 then
				count_pipe +=1
				pipes(i).pipe_id = count_pipe
				pipes(i).active_time = graph_frame_counter
				pipe_rnd = rnd * (1 - pipe_holesize )
				pipes(i).top = pipe_rnd
				pipes(i).x = graph_scr_x - (graph_frame_counter - pipes(i).active_time) * graph_pix_per_vsync
				pipes(i).old_x = pipes(i).x
				exit sub
			end if
		next i
	else
		'beep
	end if
end sub

sub flappy_game.draw_pipes()
	dim as longint t = graph_frame_counter
	if pipes(0).old_x < -pipe_with then remove_pipe()
	
	for i as ubyte = 0 to pipe_count-1
		if pipes(i).active_time <> 0 then
			pipes(i).old_x = pipes(i).x
			pipes(i).x = graph_scr_x - (t - pipes(i).active_time) * graph_pix_per_vsync
			if graph_vsync_on then
				line (pipes(i).x, 0)-(pipes(i).x + pipe_with, graph_scr_y*pipes(i).top), &hAAAAAA, BF
				line (pipes(i).x, graph_scr_y * (pipes(i).top + pipe_holesize))-(pipes(i).x + pipe_with, graph_scr_y-1), &h888888, BF
				line (pipes(i).x, 0)-(pipes(i).x + pipe_with, graph_scr_y*pipes(i).top), &h000000, B 'F
				line (pipes(i).x, graph_scr_y * (pipes(i).top + pipe_holesize))-(pipes(i).x + pipe_with, graph_scr_y-1), &h000000, B 'F
				'draw string (pipes(i).old_x+ pipe_with/2-4, 8), str(pipes(i).pipe_id)&"/"&i, &h000000
			end if
		end if
	next i
end sub

function flappy_game.check_player_collision(p as bird) as boolean
	dim as short x1 = any
	dim as short x2 = any
	dim as short y1 = any
	dim as short y2 = any
	dim as short py = any
	dim as short last_x2 = 0
	
	if p.y > 1 or p.y < 0 then return true
	if p.x <= 0 then return true
	if pipe_count <= 0 or pipe_count > pipe_max then beep: screen 0: print pipe_count:end
	
	for i as ubyte = 0 to pipe_count-1
		x1 = pipes(i).x
		x2 = (pipes(i).x + pipe_with)
		
		if x1 < p.x andalso x2 > p.x then
			y1 = graph_scr_y * pipes(i).top
			y2 = graph_scr_y * (pipes(i).top + pipe_holesize)
			py = p.y * graph_scr_y
			if y1 < py andalso y2 > py then
				'line (x1, y1)-(x2, y2), &h00FF00, bf
			else
				return true
			end if
		end if
		
		if last_x2 < x2 andalso x2 > p.x then
			'look for next colision height
			last_x2 = x2
			
			y1 = graph_scr_y * pipes(i).top
			y2 = graph_scr_y * (pipes(i).top + pipe_holesize)
			'line (0, y1+1)-(x2, y1+1), &hff0000
			'line (0, y2-1)-(x2, y2-1), &hff0000
			
			p.next_top_pipe_height = p.y - pipes(i).top
			p.next_bottom_pipe_height = -(p.y - (pipes(i).top + pipe_holesize))
			p.next_pipe_distance = x1
			p.next2_top_pipe_height = p.y - pipes(i+1).top
			p.next2_bottom_pipe_height = -(p.y - (pipes(i+1).top + pipe_holesize))
			'draw string (x2, y1-8),"1 = " & p.next_top_pipe_height
			'draw string (x2, y2),"2 = " & p.next_bottom_pipe_height
			
			exit for ' need pipe do not need checks now
		end if
	next i
end function


Function Regulate(Byval MyFps As long,Byref fps As long, reset_ as boolean = false) As long
	Static As Double timervalue, _LastSleepTime, t3, frames
	if reset_  then timervalue = 0: _LastSleepTime = 0: t3 = 0: frames = 0: fps = 0
	frames += 1
	If (Timer-t3) >= 1 Then t3 = Timer: fps = frames: frames = 0
	dim as double sleeptime=_LastSleepTime + ((1 / myfps) - Timer + timervalue) * 1000
	If sleeptime<1 Then sleeptime = 1
	_LastSleepTime = sleeptime
	timervalue = Timer
	Return sleeptime
End Function


'-------------------------------------------------Neural netwerk ---------------------------------------------------------

function gen.CrossMutate(a as Neural_network, b as Neural_network) as Neural_network
	dim as ushort start_noteA = rnd * weights
	dim as ushort start_noteB = rnd * weights
	dim as ushort lenght_note = 1 + (rnd * (weights -1))
	dim as ubyte node_nrA ,node_nrB
	dim tmp as Neural_network = a
	
	for i as ubyte = 0 to lenght_note
		node_nrA = (start_noteA + i) mod (weights + 1)
		node_nrB = (start_noteB + i) mod (weights + 1)
		tmp.weight(node_nrA) = b.weight(node_nrB)
	next i
	return tmp
end function

sub gen.mutate(n as ushort)
	static as ushort loop_counter :loop_counter += 1
	if fit_selection_counter = 0 then exit sub
	dim as ushort loop_select = loop_counter mod (fit_selection_counter + 1)
	
	brain(n) = fit_selection(loop_select) ' copy from fit list is done
	if brain(n).fitness > brain(n).best_fitness then brain(n).best_fitness = brain(n).fitness
	brain(n).fitness *= gen_mutate_prec_fitnessToKeep
	brain(n).mutated += 0.000001
	
	dim as ushort b 
	for a as ushort = 0 to (weights + 1) * brain(n).perc_weight 'old (gen_fit_mutate_size-1)
		dim as rnd8_16bit pole
		b = weights * pole.give16_rnd	'select a note
		if brain(n).weight(b) <> 0  then 'andalso (brain(n).weight(b) + pole.give16_rndXrnd) <> 0
			brain(n).weight(b) += pole.give16_rndXrnd * brain(n).streght_weight  'gen_mutate_streght_weight
			'if brain(n).weight(b) > 1 then brain(n).weight(b) = 1
			'if brain(n).weight(b) < -1 then brain(n).weight(b) = -1
		end if
	next a
	if brain(n).fitness > brain(n).best_fitness then brain(n).best_fitness = brain(n).fitness
	brain(n).fitness = 0
	
	#If Defined(DEBUG) 'count cels
		brain(n).cels_alive = 0
		for a as ushort = 0 to weights
			if brain(n).weight(a) <> 0 then brain(n).cels_alive += 1
		next
	#EndIf
end sub


function gen.is_ID_in_selection(s as ushort) as ushort
	dim as ushort count = 0 
	if fit_selection_counter > 0 then
		for i as short = 0 to fit_selection_counter -1
			if fit_selection(i).ID = brain(s).ID then count +=1
		next i
	end if
	return count
end function


sub gen.select_fittest()
	dim as long highest_fitness_found = 0, highest_fitness_found_old
	dim as ushort found
	dim as ubyte selection(gen_size)		'clear mask array for selection(gen__selection_size)
	dim as short aIndex
	fit_selection_counter = 0
	
	for a as ushort = 0 to gen_size ' search best fit
		if brain(a).fitness > highest_fitness_found then highest_fitness_found = brain(a).fitness: aIndex = a
	next a
	
	highest_fitness_found_old = highest_fitness_found
	do
		found = 0
		for a as ushort  = 0 to gen_size
			if selection(a) = 0 andalso brain(a).fitness = highest_fitness_found then
				'print is_ID_in_selection(a),
				if is_ID_in_selection(a) = 0 then
					selection(a) = 1
					fit_selection(fit_selection_counter) = brain(a)
					if fit_selection_counter = gen_fit_selection_size then exit do
					fit_selection_counter += 1
					found = 1
					'color &hFF0000: print a,
				end if
			end if
		next a
		if highest_fitness_found < 0 then  exit do 'if bug it wil exit here
		if found = 0 then highest_fitness_found -= 1 ' this is slow!!!!!!!!!!!!!!!!!!!!
		'if (highest_fitness_found_old / 100) > highest_fitness_found then exit do
		'if fit_counter > 100 then exit do
	loop until fit_selection_counter > (gen_fit_selection_size) ' leave loop when fit selection top is full
	if gen_fit_selection_size <> ( fit_selection_counter ) then print "         Error: "; fit_selection_counter, gen_fit_selection_size, highest_fitness_found_old: sleep: end
end sub


sub gen.gen_calc_brain(a as ushort)
	Clear brain(a).bias(node_in), 0, (bias-node_in+1) * SizeOf(single) 'clear als outputs
	dim as single output_offset = node_in
	dim as single input_offset = 0
	dim as ushort count_weight = 0
	
	for c as ubyte = 0 to node_hidden - 1
		for b as ubyte = 0 to node_in - 1
			brain(a).bias(output_offset+c) += brain(a).bias(input_offset+b) * brain(a).weight(count_weight)
			count_weight += 1
		next
		''brain(a).bias(output_offset+c) /= node_in
	next
	
	if layer_hidden > 0 then
		for d as ubyte = 0 to layer_hidden -1
			output_offset = node_in + (node_hidden * (d+1)) 
			input_offset = node_in + (node_hidden * d) 
			'print "mid input_offset"; input_offset
			'print "mid output_offset"; output_offset
			for c as ubyte = 0 to node_hidden - 1
				for b as ubyte = 0 to node_hidden - 1
					brain(a).bias(output_offset+c) += brain(a).bias(input_offset+b) * brain(a).weight(count_weight)
					count_weight += 1
				next
				''brain(a).bias(output_offset+c) /= node_hidden
			next
		next
	end if
	
	
	output_offset = node_in + (node_hidden * (layer_hidden+1)) 
	input_offset = node_in + (node_hidden * (layer_hidden))
	for c as ubyte = 0 to node_out - 1
		for b as ubyte = 0 to node_hidden - 1
			brain(a).bias(output_offset+c) += brain(a).bias(input_offset+b) * brain(a).weight(count_weight)
			count_weight += 1
		next
		''brain(a).bias(output_offset+c) /= node_hidden
	next
end sub

constructor Neural_network
	for c as ushort = 0 to weights
		dim as rnd8_16bit pole
		if pole.give16_rnd > dead_weight then
			'weight(c) = pole.give16_rndXrnd * 1
			weight(c) = pole.give16_rndXrnd * streght_weight
			if weight(c) > 1 then weight(c) = 1
			if weight(c) < -1 then weight(c) = -1
		else
			weight(c) = 0
		end if
	next
	fitness = 0
	'mutated = 0
	best_fitness = 0
	ID = ID_counter
	ID_counter += 1
end constructor


sub gen.gen_draw_brain(a as ushort)
	dim as short in_offset = ((node_in - 1) / 2 * graph_size)
	dim as short hidden_offset = ((node_hidden - 1) / 2 * graph_size)
	dim as short out_offset = ((node_out - 1) / 2 * graph_size)
	dim as short x, y, x2, y2
	dim as ushort count_bias = 0
	dim as ushort count_weight = 0
	
	draw string (graph_xoffset+hidden_offset, graph_yoffset-in_offset), "Nr. " + str(a) + " / ID:" + str(int(brain(a).id)), rgb(0,0,0)
	
	'part weights
	for c as ubyte = 0 to node_hidden - 1
		for b as ubyte = 0 to node_in - 1
			x = graph_xoffset
			y = graph_yoffset + b*graph_size - in_offset
			x2 = graph_xoffset + graph_size*2
			y2 = graph_yoffset + c*graph_size - hidden_offset
			if brain(a).weight(count_weight) > 0 then
				line(x, y)-(x2, y2), rgb(0, brain(c).bias(count_bias)*128, 0)
			elseif brain(a).weight(count_weight) = 0 then
				line(x, y)-(x2, y2), &h000000
			else
				line(x, y)-(x2, y2), rgb(brain(c).bias(count_bias)*128, 0, 0)
			end if
			draw string ((x+x2)/2 - 16,(y+y2)/2-8+b*8),round(brain(a).weight(count_weight)), &hFFFF00
			count_weight += 1
		next
		count_bias += 1
	next
	
	if layer_hidden > 0 then
		for e as ubyte = 0 to layer_hidden-1
			for d as ubyte = 0 to node_hidden - 1
				 for f as ubyte = 0 to node_hidden - 1
					x = graph_xoffset + (4 + (e-1)*2) * graph_size
					y = graph_yoffset + d*graph_size - hidden_offset
					x2 = graph_xoffset + (4 + (e)*2) * graph_size
					y2 = graph_yoffset + f*graph_size - hidden_offset
					if brain(a).weight(count_weight) > 0 then
						line(x, y)-(x2, y2), rgb(0, brain(d).bias(count_bias)*128, 0)
					elseif brain(a).weight(count_weight) = 0 then
						line(x, y)-(x2, y2), &h000000
					else
						line(x, y)-(x2, y2), rgb(brain(d).bias(count_bias)*128, 0, 0)
					end if
					draw string ((x+x2)/2 - 16,(y+y2)/2-8+f*8),round(brain(a).weight(count_weight)), &hFFFF00
					count_weight += 1
				next
			next
			count_bias += 1
		next
	end if
	
	for d as ubyte = 0 to node_hidden - 1
		 for f as ubyte = 0 to node_out - 1
			x = graph_xoffset + (4 + (layer_hidden-1)*2) * graph_size
			y = graph_yoffset + d*graph_size - hidden_offset
			x2 = graph_xoffset + (4 + layer_hidden*2) * graph_size
			y2 = graph_yoffset + f*graph_size - out_offset
			if brain(a).weight(count_weight) > 0 then
				line(x, y)-(x2, y2), rgb(0, brain(d).bias(count_bias)*128, 0)
			elseif brain(a).weight(count_weight) = 0 then
				line(x, y)-(x2, y2), &h000000
			else
				line(x, y)-(x2, y2), rgb(brain(d).bias(count_bias)*128, 0, 0)
			end if
			draw string ((x+x2)/2 - 16,(y+y2)/2-8+f*8),round(brain(a).weight(count_weight)), &hFFFF00
			count_weight += 1
		next
		count_bias += 1
	next
	
	' part bias
	count_bias = 0
	for b as ubyte = 0 to node_in - 1
		x = graph_xoffset
		y = graph_yoffset + b*graph_size - in_offset
		circle (x, y), graph_size/4 , &hFF0000
		draw string (x-12,y-4), round(brain(a).bias(count_bias)), &hFFFFff
		count_bias += 1
	next
	for c as ubyte = 0 to node_hidden - 1
		x = graph_xoffset + graph_size*2
		y = graph_yoffset + c*graph_size - hidden_offset
		circle (x, y), graph_size/4 , &h00FF00
		draw string (x-12,y-4), round(brain(a).bias(count_bias)), &hFFFFff
		count_bias += 1
	next
	if layer_hidden > 0 then
		for e as ubyte = 0 to layer_hidden -1
			for d as ubyte = 0 to node_hidden - 1
				x = graph_xoffset + (4 + e*2) * graph_size
				y = graph_yoffset + d*graph_size - hidden_offset
				circle (x, y), graph_size/4 , &h00FF00
				draw string (x-12,y-4),round(brain(a).bias(count_bias)), &hFFFFff
				count_bias += 1
			next
		next
	end if
	for f as ubyte = 0 to node_out - 1
		x = graph_xoffset + (4 + layer_hidden*2) * graph_size
		y = graph_yoffset + f*graph_size - out_offset
		if brain(a).bias(count_bias) > 0 then
			circle (x, y), graph_size/4 , &h00FF00,,,,f
		else
			circle (x, y), graph_size/4 , &hFF0000,,,,f
		end if
		draw string (x-12,y-4),round(brain(a).bias(count_bias)), &hFFFFff
		count_bias += 1
	next
end sub

function round(n as single) as string
	return Format(n, "###.##")
end function

constructor rnd8_16bit
	in32_rnd = rnd * 4294967296
end constructor

function rnd8_16bit.give16_rndXrnd() as single
	return ((out8_a-127.5) * (out8_b-127.5)) / 16256.2599999999
end function 

function rnd8_16bit.give16_rnd() as single
	return out16_c / 65535.9999999999
end function 

function moving_avg.MA(in as double, lengte as ubyte) as long
	dim as double som
	array(arrayIndex) = in
	arrayIndex += 1 
	if arrayIndex > lengte then arrayIndex = 0
	for i as ubyte = 0 to lengte
		som += array(i)
	next i
	ReturnSom = som / lengte
	return ReturnSom
end function
Edit removed double check "if game.check_player_collision(player(i)) then player(i).state = -1 'dieded"
tinram
Posts: 89
Joined: Nov 30, 2006 13:35
Location: UK

Re: Neural network - flappy bird

Post by tinram »

Pretty cool for 800 lines of code.

(Just need a 'var' added to 'fitness = 0' to get the present version to compile.)

Surely though, wasted on Flappy Bird ... this intriguing code could be solving something far more useful ;)
Gunslinger
Posts: 103
Joined: Mar 08, 2016 19:10
Location: The Netherlands

Re: Neural network - flappy bird

Post by Gunslinger »

tinram wrote: Jan 29, 2022 18:53 Pretty cool for 800 lines of code.

(Just need a 'var' added to 'fitness = 0' to get the present version to compile.)

Surely though, wasted on Flappy Bird ... this intriguing code could be solving something far more useful ;)
Thank for you comment Tinram, i'm not able to find 'fitness = 0' line thats gives you problems with compile.

I also have remove a lot more bugs, big one at wrong output nodes
Added nice graphic plot of fitness development
new version here. (1120 line of code now)
https://github.com/TTIcecube/Neural-net ... appy-bird-
With video of last run at github.

I'm going to split neural network from flappy game now and do some other project with it.
Luxan
Posts: 222
Joined: Feb 18, 2009 12:47
Location: New Zealand

Re: Neural network - flappy bird

Post by Luxan »

Is this compatible with Linux, using Ubuntu 20.04 the code returns an error at line 7 .
srvaldez
Posts: 3373
Joined: Sep 25, 2005 21:54

Re: Neural network - flappy bird

Post by srvaldez »

something is not right, there's a problem in the constructor Neural_network, for reference here's the Neural_network type

Code: Select all

type Neural_network
    Public:
    as single bias(bias)
    as single weight(weights)
    as long ID
    as double mutated = 0
    as ulong fitness = 0
    as ulong best_fitness = 0
    declare constructor
    as single perc_weight = rnd * gen_mutate_perc_weight '0.2           'how many weight are been modified on mutate 
    as single streght_weight = rnd * gen_mutate_streght_weight '50      'how mach to to modify the weights
    as single dead_weight = rnd * gen_mutate_prec_dead_weight '0.0      'dead weight connection
    as ushort cels_alive = 0
end type
and here's the constructor

Code: Select all

constructor Neural_network
    for c as ushort = 0 to weights
        dim as rnd8_16bit pole
        if pole.give16_rnd > dead_weight then
            'weight(c) = pole.give16_rndXrnd * 1
            weight(c) = pole.give16_rndXrnd * streght_weight
            if weight(c) > 1 then weight(c) = 1
            if weight(c) < -1 then weight(c) = -1
        else
            weight(c) = 0
        end if
    next
    fitness = 0         'Neural network - flappy bird.bas(731) error 42: Variable not declared, fitness in 'fitness = 0'
    'mutated = 0
    best_fitness = 0
    ID = ID_counter
    ID_counter += 1
end constructor
if you change fitness = 0 to this.fitness = 0 then all is well but then why doesn't the compiler complain about the rest of the member-variables best_fitness and ID ?
srvaldez
Posts: 3373
Joined: Sep 25, 2005 21:54

Re: Neural network - flappy bird

Post by srvaldez »

I found it, in the main code Gunslinger declares the variable fitness as moving_avg, I guess that confuses the compiler in line 731
Luxan
Posts: 222
Joined: Feb 18, 2009 12:47
Location: New Zealand

Re: Neural network - flappy bird

Post by Luxan »

After setting up FreeBasic for Win10, changing fitness = 0
to this.fitness = 0, within constructor Neural_network; your
code compiles and runs .

Are there more errors to address.

As you have used fbgfx with __FB_WIN32__ I'm not
sure this code will run under Linux; even with the
ammendments mentioned.

Is there supposed to be a long, and intensive, training
epoch prior to an actual trial .
Luxan
Posts: 222
Joined: Feb 18, 2009 12:47
Location: New Zealand

Re: Neural network - flappy bird

Post by Luxan »

After setting up FreeBasic for Ubuntu20.04 , changing fitness = 0
to this.fitness = 0, within constructor Neural_network; your
code compiles and runs .

I'm using Flappy bird frames with neural network.bas, from
https://github.com/TTIcecube/Neural-net ... appy-bird-

Looking at NN-flappy.mp4, how did you train the neural network to
do that, and what is that background graph and how might I display
that.
Luxan
Posts: 222
Joined: Feb 18, 2009 12:47
Location: New Zealand

Re: Neural network - flappy bird

Post by Luxan »

Using the Ubuntu 20.04 version of flappy bird, as mentioned in the last
post, on a 4 core CPU at 3GHz.

I trained the Neural Network on the default settings for a day and
a night, the result wasn't too good; only a few 'birds' survived.



Then I chose the 2 option, avg of last and best, as printed upon the
screen, and let that learn for an hour and a half. The result is an
improvement, with most of the flock making their way through the obstacles.

Irrespective of the training method, perhaps you need to stop and some how
save the Neural Network weights, after a certain number of epochs; otherwise
the network may become over trained.

Here's the link to the readily available movie of the latter results:

https://drive.google.com/file/d/1azvVG3 ... sp=sharing

To view this outside of your web browser, using VLC or similar, download the file
to your computer, open and view it there; select full screen from your menu's
options.
The mp4 file is about 208.7MB in size.
Avata
Posts: 102
Joined: Jan 17, 2021 7:27

Re: Neural network - flappy bird

Post by Avata »

Do we need to start over every time? Can you save the progress and start from where you left off last time instead of restarting each time?
Gunslinger
Posts: 103
Joined: Mar 08, 2016 19:10
Location: The Netherlands

Re: Neural network - flappy bird

Post by Gunslinger »

Hi. Nice to see i got some interest for this project.
In summer time I don't have much time for freebasic.

Running on default setting the github version supose to work fine. In a houw big improvement must be visible. Running at full steed.
I have a i7 at 2.7gh clock windows 10 64bit
You can make it easy by increasing the pipe distance.

I have made a basic save and load on latest version.
It is kind of boring to load in already traind birds. But usefull yes.

I will share a updated version tomorrow.
Still the code is what it is. Work in progress.

Not sure if the load function works a cross different platforms.
Save and load your own procces suit be fijne.

I did learn more about neural networks.
And I think you right, networks get overtraind.
Not really sure how to move on now.
The code it self becomes big and may still have lot of bugs.
I don't like the cross mutates make them all the same over time.

Have fun .
Gunslinger
Posts: 103
Joined: Mar 08, 2016 19:10
Location: The Netherlands

Re: Neural network - flappy bird

Post by Gunslinger »

Post Reply