TSNE problem.
-
- Posts: 30
- Joined: Jul 08, 2014 17:45
TSNE problem.
Or possibly not.
I used the supplied test_client.bas program with TSNE_V3.bi using one of my own websites as the website to get and all worked perfectly.
I am trying to produce a program that will get a .csv file from a website but if I program that site instead of mine I get a "No IPV6 supported" error. I have tested and my connection to the web is IPV4. What I'm having a problem with is that if I go to the target website via a browser (I've tried Edge, Chrome and Firefox) I can download the .csv file with no problem. Therefore, I can't see why there should be an IPV6 problem. This is my first attempt at trying to do something like this over the internet so it's almost certainly something simple but I have no idea what. Can anyone shed any light?
One other thing - the website I'm trying to connect to is https rather than http but I can't see why that would generate an IPV6 error.
I used the supplied test_client.bas program with TSNE_V3.bi using one of my own websites as the website to get and all worked perfectly.
I am trying to produce a program that will get a .csv file from a website but if I program that site instead of mine I get a "No IPV6 supported" error. I have tested and my connection to the web is IPV4. What I'm having a problem with is that if I go to the target website via a browser (I've tried Edge, Chrome and Firefox) I can download the .csv file with no problem. Therefore, I can't see why there should be an IPV6 problem. This is my first attempt at trying to do something like this over the internet so it's almost certainly something simple but I have no idea what. Can anyone shed any light?
One other thing - the website I'm trying to connect to is https rather than http but I can't see why that would generate an IPV6 error.
Re: TSNE problem.
Can you give the URL for our testing, or is it a problem?
-
- Posts: 30
- Joined: Jul 08, 2014 17:45
Re: TSNE problem.
https://promo.betfair.com/betfairsp/pri ... 072020.csv is the file I'm trying to get.
Normally I go to https://promo.betfair.com/betfairsp/prices and click on a file to download - but I'd like to automate the procedure to save some time.
Normally I go to https://promo.betfair.com/betfairsp/prices and click on a file to download - but I'd like to automate the procedure to save some time.
Re: TSNE problem.
Thanks, Phil. I've tested the URL with a WinInet-based program, and it downloads just fine. Same for URLDownloadToFileA...
-
- Posts: 862
- Joined: May 05, 2015 5:35
- Location: Germany
Re: TSNE problem.
AFAIK TSNE can't handle https yet. I would recommend using libcurl for that kind of download.
Code: Select all
#Include Once "curl.bi"
#Include "vbcompat.bi"
Type tCurlInfo
curl As CURL Ptr
ff As Integer
totlen As Double
dllen As Double
timeout As Double = 3.0
process As boolean = FALSE
content As String
End Type
Declare Function download_curl(url As String, tout As Double = 10.0) As String
Declare Function write_callback_curl Cdecl (buffer As Byte Ptr, size As Long, nitems As Long, info As tCurlInfo Ptr) As Integer
Function download_curl(url As String, tout As Double = 10.0) As String
Dim As tCurlInfo curlinfo
Dim As String dlurl, tmp
Dim As Double timeout
curlinfo.curl = curl_easy_init()
If curlinfo.curl = 0 Then
Print "ERROR"
Return ""
End If
'' set url and callback
curl_easy_setopt(curlinfo.curl, CURLOPT_URL, url)
curl_easy_setopt(curlinfo.curl, CURLOPT_WRITEDATA, @curlinfo)
curl_easy_setopt(curlinfo.curl, CURLOPT_WRITEFUNCTION, @write_callback_curl)
curl_easy_setopt(curlinfo.curl, CURLOPT_SSL_VERIFYPEER, 0)
curl_easy_setopt(curlinfo.curl, CURLOPT_FOLLOWLOCATION, 1)
curlinfo.timeout = tout
curl_easy_perform(curlinfo.curl) 'execute..
curl_easy_cleanup(curlinfo.curl) 'shutdown
Close curlinfo.ff
Return curlinfo.content
End Function
Function write_callback_curl Cdecl (buffer As Byte Ptr, size As Long, nitems As Long, info As tCurlInfo Ptr) As Integer
Dim As Integer x, bytes = size * nitems
Dim As Double tot_len, dl_len, timeout = Timer + info->timeout
curl_easy_getinfo(info->curl, CURLINFO_CONTENT_LENGTH_DOWNLOAD, @info->totlen )
curl_easy_getinfo(info->curl, CURLINFO_SIZE_DOWNLOAD, @info->dllen )
If info->ff Then
If info->totlen = -1 Then
Print Format(info->dllen/1024/1024,"#.##");" MB (ESC to terminate)"
Else
Print Int(100 * info->dllen / info->totlen);"% of ";Format(info->totlen/1024/1024,"#.##");" MB (ESC to terminate)"
EndIf
Locate CsrLin - 1,1,0
If info->totlen = info->dllen Then
Print
EndIf
Put #info->ff, ,*buffer,bytes 'write downloaded bytes to file
Else
For x = 0 To bytes - 1
info->content += Chr(buffer[x])
Next
EndIf
If (InKey = Chr(27)) Or (Timer > timeout) Then
Print
Return 0
EndIf
Return bytes
End Function
Dim As String content
content = download_curl("https://promo.betfair.com/betfairsp/prices/dwbfpricesukwin13072020.csv")
Print "done"
Open "C:\downloadtest.csv" For Output As #1
Print #1, content;
Close
Sleep
Re: TSNE problem.
grindstone wrote:AFAIK TSNE can't handle https yet. I would recommend using libcurl for that kind of download.Code: Select all
#Include Once "curl.bi"[/quote]URLDownloadToFile is a one-liner that does the job - if you have no problem with AV warnings, and you are running Windows, I'd go for that.
-
- Posts: 30
- Joined: Jul 08, 2014 17:45
Re: TSNE problem.
This all being new to me I copied your source and tried to run it. I got the following error:grindstone wrote:AFAIK TSNE can't handle https yet. I would recommend using libcurl for that kind of download.
C:\PROGRA~2\FREEBA~1\bin\win32\ld.exe: cannot find -lcurl (I'm using FBIde rather than command line)
I have no idea where -lcurl is supposed to be
I am running Windows but incorporating URLDownloadToFile is something I have no clue how to do. Including other people's libraries is a whole new experience for me :)jj2007 wrote: URLDownloadToFile is a one-liner that does the job - if you have no problem with AV warnings, and you are running Windows, I'd go for that.
Re: TSNE problem.
you can download pre-compiled libcurl from https://curl.haxx.se/windows/
just copy libcurl.a to your FB lib/win32 folder
just copy libcurl.a to your FB lib/win32 folder
Re: TSNE problem.
Sorry, I thought it was a one-liner, but FB is slightly more complicated...
Code: Select all
#include "Windows.bi"
Dim URLDownloadToFileA as function (byval pCaller as any ptr, _
byval szURL as zstring ptr, _
byval szFileName as zstring ptr, byval dwResv as uinteger, _
byval lpfnCB as any ptr) as integer
URLDownloadToFileA = GetProcAddress(dylibload("urlmon"), "URLDownloadToFileA")
Dim DownloadOK As long=URLDownloadToFileA(0, @"https://promo.betfair.com/betfairsp/prices/dwbfpricesukwin13072020.csv", @"tmp.csv", 0, 0)
print "OK if zero: "; DownloadOK
sleep
Re: TSNE problem.
Win 10
One of these methods should work.
One of these methods should work.
Code: Select all
#include "file.bi"
Sub savefile(filename As String,p As String)
Dim As Integer n
n=Freefile
If Open (filename For Binary Access Write As #n)=0 Then
Put #n,,p
Close
Else
Print "Unable to save " + filename
End If
End Sub
Function loadfile(file As String) As String
If Fileexists(file)=0 Then Print file;" not found":Sleep:End
Var f=Freefile
Open file For Binary Access Read As #f
Dim As String text
If Lof(f) > 0 Then
text = String(Lof(f), 0)
Get #f, , text
End If
Close #f
Return text
End Function
Dim As String site="https://promo.betfair.com/betfairsp/prices/dwbfpricesukwin13072020.csv"
Sub download(site As String,newfile As String)
If Instr(site," ") Then site=Chr(34)+site+Chr(34)
savefile( Curdir +"\"+newfile,"") 'create an empty file
shell "bitsadmin /create mydownload"
Shell "bitsadmin /transfer mydownload /download /priority normal " + _
site +" "+ Chr(34) + Curdir +"\"+ newfile +Chr(34)
shell "bitsadmin /complete mydownload"
If Len(loadfile(newfile)) Then
Shell Chr(34) +Curdir + "\"+newfile + Chr(34)
End If
End Sub
sub download2(site As String,newfile As String)
If Instr(site," ") Then site=Chr(34)+site+Chr(34)
shell "powershell Invoke-WebRequest "+site+ "/ -OutFile "+ curdir +"/"+ newfile
If Len(loadfile(newfile)) Then
Shell Chr(34) +Curdir + "\"+newfile + Chr(34)
End If
end sub
sub cleanup destructor '(only for download using bitsadmin)
shell "bitsadmin /complete mydownload"
sleep 1000
end sub
download2(site,"New.csv")
-
- Posts: 30
- Joined: Jul 08, 2014 17:45
Re: TSNE problem.
I got a strange result running this code (probably my fault). Firstly I copied and ran it as is without issue. Trying to modify it so that I could download a number of files I altered it bit by bit, firstly by putting the filenames "https://promo.betfair.com/betfairsp/pri ... 072020.csv" and "tmp.csv" into strings and using those strings. That didn't work until I removed the @ character, but then it did. However, it also produced an error when compiling for the line: URLDownloadToFileA = GetProcAddress(dylibload("urlmon"), "URLDownloadToFileA") which came up as "Suspicious pointer assignment"jj2007 wrote:Sorry, I thought it was a one-liner, but FB is slightly more complicated..
Despite producing that error the program still appeared to work but only for one file. Trying to use multiple file names, one at a time, doesn't work*. I also downloaded the latest FreeBasic software - having discovered that windows.bi didn't exist on my PC (I had windows9.bi !). It was sort of interesting that it did work without, apparently, windows.bi. However, I still get the "Suspicious pointer assignment" error.
Can you offer any further ideas?
* I have a list of file names which I would use weekly e.g. dwbfpricesukwin17072020.csv, dwbfpricesukwin16072020.csv, dwbfpricesukwin15072020.csv,
dwbfpricesukwin14072020.csv, etc. I simply constructed a loop to read those in from a file and created for each the appropriate URL address and a file to store the results.
Re: TSNE problem.
Phil,
That should work fine in a loop. When you print the URLs before your DownloadOK=URLDownloadToFileA(0, szUrl, destFile, 0, 0), do they all look ok? Can you post your code?
Btw the "Suspicious pointer assignment" is just a warning, you can ignore it. FB and WinAPI don't understand each other well ;-)
That should work fine in a loop. When you print the URLs before your DownloadOK=URLDownloadToFileA(0, szUrl, destFile, 0, 0), do they all look ok? Can you post your code?
Btw the "Suspicious pointer assignment" is just a warning, you can ignore it. FB and WinAPI don't understand each other well ;-)
Re: TSNE problem.
This works here (win 10)
Takes a few seconds for powershell to fire up, after that only a few seconds more.
Takes a few seconds for powershell to fire up, after that only a few seconds more.
Code: Select all
#include "file.bi"
sub download(site As String,newfile As String)
If Instr(site," ") Then site=Chr(34)+site+Chr(34)
shell "powershell Invoke-WebRequest "+site+ "/ -OutFile "+ curdir +"/"+ newfile
end sub
dim as string site="https://promo.betfair.com/betfairsp/prices/"
Dim As String file(1 to 5)={"dwbfpricesukwin13072020.csv", _
"dwbfpricesukwin17072020.csv", _
"dwbfpricesukwin16072020.csv", _
"dwbfpricesukwin15072020.csv", _
"dwbfpricesukwin14072020.csv"}
for n as long=1 to ubound(file)
var s=site+file(n)
var g="new"+str(n)+".csv"
download(s,g)
print g;" ";iif(fileexists(g),"OK","Error"); " = ";file(n);" size = ";filelen(g)
next n
print "press a key to delete these files"
sleep
for n as long=1 to ubound(file)
kill "new"+str(n)+".csv"
print iif(fileexists("new"+str(n)+".csv"),"Error deleting","OK")
next n
sleep
-
- Posts: 30
- Joined: Jul 08, 2014 17:45
Re: TSNE problem.
I was able to rework some of this and used it to do what I needed to do. Thank you.dodicat wrote:Win 10
One of these methods should work.
-
- Posts: 30
- Joined: Jul 08, 2014 17:45
Re: TSNE problem.
I would like to thank you all for your assistance. I have managed to create a program that does exactly what I needed derived from the code you supplied. If I ever meet you in a hostelry the drinks will be on me.
Thank you all for stepping in and trying to help. It was much appreciated.
Thank you all for stepping in and trying to help. It was much appreciated.