In the UNIX/Linux environment, you can quickly move to directories using cd command in Terminal of window. If you ever want to get a file from internet and save it in the current directory, it will take a while for you to use the web browser to download the file and select the directory to store that file. With the wget tool available in UNIX/Linux, you can easily download file to current directory quickly.
1. Syntax
Wget is a command-line utility, which is used for downloading files and content on the internet, which can be a website or an FTP site. Wget is very flexible and has many options to use for many different purposes.
The general syntax for wget is:
For example:
Code:
[root@server ~]# wget forumweb.hosting
Code:
--2018-04-18 17:10:46-- http://forumweb.hosting/
Resolving forumweb.hosting... 104.18.49.112, 104.18.48.112, 2400:cb00:2048:1::6812:3170, ...
Connecting to forumweb.hosting|104.18.49.112|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://forumweb.hosting/ [following]
--2018-04-18 17:10:47-- https://forumweb.hosting/
Connecting to forumweb.hosting|104.18.49.112|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: “index.html”
[ <=> ] 161,895 223K/s in 0.7s
2018-04-18 17:10:49 (223 KB/s) - “index.html” saved [161895]
[root@server ~]#
You can specify multiple paths for wget as follows:
Code:
[root@server ~]# wget URL1 URL2 URL3
2. Save file with another name
Typically, downloaded files have the same name as file in URL, and information about the loading process is displayed on the screen.
You can save the downloaded file with a different name than the existing one, using the -O option. If the file with the specified name already exists, then the contents of the downloaded file overwrite the existing one.
Instead of displaying the information of the download process, you can save this information in a file, using the -o option.
Code:
[root@server ~]# wget ftp://example_domain.com/somefile.img -O dloaded_file.img -o log
By using this command, no information is printed on the screen. The log or process will be written to the log file, and the downloaded file will be named dloaded_file.img.
3. Automatically reload when corrupted
During the download, if the connection is unstable, the file download may be interrupted and failed. In these cases, you usually re-execute the load. However, instead of having to manually reload the download, wget provides us with an option to reload the download automatically every time the download is lost.
To do this, you use the -t argument in wget as follows:
Code:
[root@server ~]# wget -t 5 URL
In the above command, 5 is the number of times that wget will attempt to reload the file when the connection is lost during the download, instead of 5 times the number that you want wget to perform.
If we do not want to specify the number of reloads and want wget to repeat the load until the new stop, in this case you do as follows:
[root@server ~]# wget -t URL
4. Limit download speed
When you have a limited internet bandwidth and many applications share this connection, and if you need to load a large file, it will take up the bandwidth of other applications, making these applications inactive.
To limit download speed in wget, use the -limit-rate parameter as follows:
Code:
[root@server ~]# wget --limit-rate 20k http://example.com/file.iso
Explain:
k - Kilobyte (KB)
m - Megabyte (MB)
You can also specify the maximum load rating. The load will stop when the norm is reached.
To specify the load limit, use --quota or -Q as follows:
Code:
[root@server ~]# wget -q 100m http://example.com/file1 http://example.com/file2
5. Restart and continue loading
If the download is interrupted before it is complete, you can resume the download at the interrupt by using the -c option as follows:
Code:
[root@server ~]# wget -c URL
6. Copy whole site
Wget has 1 option to download an entire web page by recruiting all the URL links in web pages and downloading all of them. Therefore, you can download all web pages of a website.
To load web pages, use the -mirror option as follows:
Code:
[root@server ~]# wget --mirror --convert-links example.com
Or use the following command:
Code:
[root@server ~]# wget -r -N -k -l DEPTH URL
Explain:
-l => indicates the depth of the web pages as levels. This means that wget will only go through the number of levels that you specify.
DEPTH => depth of site.
-r (recursive) => recursively, shared with -l.
-N is used to activate lock time for files.
URL is the basic path for a website where the load should be initialized
-k or -convert-links => instructs wget to convert links to other pages in the page loaded to local copies of those pages.
In addition to loading a web page on the machine, you can use the lynx command as follows:
Code:
[root@server ~]# lynx -dump URL> webpage_as_text.txt
For example:
Code:
[root@server ~]# lynx -dump http://google.com> plain_text_page.txt
6. HTTP or FTP authentication
Some sites require authentication for HTTP or FTP links. To perform this authentication, use -user and -password arguments as follows:
Code:
[root@server ~]# wget --user u
Hope it helps!
