Introduction
A year and a half ago I was looking into a way to automate the downloading of a file from FTP and if the download was interrupted, I wanted some way of resuming the download where it had left off. This is important in situations where you have low or unreliable internet bandwidth. If you are downloading a large file and you are disconnected from the internet for some reason, you don't want to start your download again from the beginning.
Previously I had been using the AutoIt built-in function InetGet(). It uses Internet Explorer's download feature and that works fine in perfect conditions, but if there's one slight interruption, it stops. Crucially there's no resume feature available for InetGet().
Another major requirement was that the solution be free of any cost, so I only looked for open source and freeware products.
A year and a half ago I was looking into a way to automate the downloading of a file from FTP and if the download was interrupted, I wanted some way of resuming the download where it had left off. This is important in situations where you have low or unreliable internet bandwidth. If you are downloading a large file and you are disconnected from the internet for some reason, you don't want to start your download again from the beginning.
Previously I had been using the AutoIt built-in function InetGet(). It uses Internet Explorer's download feature and that works fine in perfect conditions, but if there's one slight interruption, it stops. Crucially there's no resume feature available for InetGet().
Another major requirement was that the solution be free of any cost, so I only looked for open source and freeware products.
At first I did some research on the web that told me that resuming downloads requires the owner of the FTP site to have switched on the capability to resume. This was not what I wanted to hear, that is something out of my control. But I didn't fully believe it either, dedicated FTP software like WS_FTP will resume downloads. Of course a Windows GUI tool was not useful to me, I wanted a command line tool so that I could create an automated download process.
Wget
I came across Wget, an open source program, originally a tool from the Unix/Linux world, a Windows command line version is available and works extremely well! It has a 'continue' feature, this will continue (resume) downloads if broken.
It's a single exe file of about 400K. You can download it from here:
To use Wget you need to open a command window CMD. Like most command line tools you can get a list of options/parameters at the command line itself, here's what you enter:
wget --help
You can also find a list of command line options/parameters here:
Here's an example of using Wget to download a single file from an FTP site where a user name and password is required:
wget -c --ftp-user=michael --ftp-password=abc123 ftp://192.168.0.3/file.zip
What does it all mean? The command wget is followed by four parameters, each with a space between them. The above should be entered as one continuous line (not two lines as it may appear above). Here's an explanation of the parameters I used:
-c
This stands for 'continue', you could use --continue instead of -c, both do the same thing. This will mean that if the download of this file is interrupted, you can issue exactly the same command again later and it'll continue the download where it left off.
--ftp-user=michael
If the FTP site you are accessing requires authentication then you can specify the user name this way. Here I put the user name "michael", this is just an example.
--ftp-password=abc123
This is how to specify the password, the "abc123" is only an example password.
ftp://192.168.0.3/file.zip
Put the FTP address, path and file name. The above is just an example.
The following shows what it would look like for real if you used the above command line (click them for larger images):
The following is a video tutorial by jimmyr.com (not me!) and it's pretty good at showing you the basics:
Proxy
Wget has a lot of features to help you download. One I wanted and at first I couldn't find much information on, was downloading from an FTP site when your FTP goes through a proxy. If this is the case, before you run your Wget command line, set the command line (DOS) variable ftp_proxy. For example:
set ftp_proxy=10.1.1.1:8080
Of course "10.1.1.1:8080" is just an example, you'd need to put whatever your proxy address and port number is here.
Limit Bandwidth Used
The following is a useful parameter for Wget, especially in low bandwidth scenarios:
--limit-rate=
You can use this to limit the bandwidth used by Wget when downloading. For example:
--limit-rate=6k
--limit-rate=6k
The 6k means 6 kilobytes per second, not kilobits per second as you might expect. Kilobits (Kbps) is the usual way of measuring bandwidth. As there are 8 bits to every byte, you can calculate what this is in kilobits by dividing kilobits by 8 to get the number of kilobytes for the limit rate parameter.
Here are some examples:
4k means 32Kbps6k means 48Kbps
8k means 64Kbps
16k means 128Kbps
32k means 256Kbps
64k means 512Kbps
128k means 1024kbps (1Mbps)
Limiting the bandwidth used doesn't mean Wget will download at that speed exactly, it is the maximum, it won't use any more bandwidth than this limit.
When Wget runs I don't think it hogs all the bandwidth in any case, but this feature really is useful in ensuring it doesn't hog the bandwidth, especially if your internet line has to be used by other services at the same time.
By the way, if you want to check how long it takes for a particular size file to download at a particular speed (kbps), take a look at this website:
http://www.numion.com/calculators/Time.html
Conclusion
Wget is an excellent command line tool. I went on to use it in a number of programs I wrote using AutoIt. Using AutoIt you can create a seamless wrapper for Wget and control what parameters are sent to it depending on the circumstance. Wget has proven reliable, fast and versatile. The above is just one thing it can do, it also supports downloading multiple files and folders, HTTP, HTTPS, as well as FTP.
Currently Wget doesn't support SFTP, although to do this from the command line, take a look at another great free utility for Windows: WinSCP.
Wget works the same on Windows as it does on Linux. This can be useful if you use Linux at any time, for example, Ubuntu has Wget installed as standard. You can use it with the same command line parameters as described above, it works exactly the same. More interesting features can be found in the manual, a link is listed below.
(this is the same as the Wget above but you need some DLLs to be in the same folder for it to run)
Wget manual:
Lots more details about Wget:
WinWget - a Windows GUI interface for Wget:
AutoIt Windows scripting:
http://www.autoitscript.com/
WinSCP (SFTP Windows command line)
http://winscp.net/
Download speed / time / file size calculator:
http://www.numion.com/calculators/Time.html
No comments:
Post a Comment