Αν θέλετε να κατεβάσετε πολλά αρχεία τα οποία μπορεί να έχετε βρεί σε μια ιστοσελίδα, μπορείτε να το κάνετε εύκολα με το wget. Ανοίξτε ένα αρχείο με το κειμενογράφο σας (gedit, nano, vi κτλ) και κάντε paste όλα τα links των αρχείων. Σώστε το πχ. σαν filenames.txt, από το τερματικό δώστε την εντολή:

wget -i filenames.txt

και το κατέβασμα των αρχείων θα αρχίσει. Μπορείτε επίσης να φτιάξετε κάποιο script, ώστε όταν τελειώσει το κατέβασμα.. πχ να κλείσει ο υπολογιστής.

I was downloading Ubuntu Studio the other day and being 1.4Gb I left it downloading while I went off for the day. However, when I returned I saw the download had stopped at 58%!

Not wanting to waste time, bandwidth and money downloading it again I did a little search and found out about ‘wget‘. It contains a command to resume incomplete downloads. So I only had to open the terminal in the folder where the file is contained and enter the following “wget -c http://cdimage.ubuntu.com/ubuntustudio/releases/9.10/release/ubuntustudio-9.10-alternate-i386.iso“

The ‘ -c ‘ is the command used to continue a download and as long as you open the terminal from the folder containing the file, wget will continue the download from where it originally finished.

FOSS 4ever!

The page I found this on is http://www.cyberciti.biz/tips/wget-resume-broken-download.html

http_proxy="http://login:password@proxy:port/"
Например,
http_proxy="http://login:password@proxy:3128/" wget "http://www.debian.org/"

Aunque Pacman es un gran gestor de paquetes, siempre se pueden ajustar algunos detalles para que funcione más rápido. Por una parte podemos optimizar la base de datos para que el acceso a estos archivos sea más rápido, y por otra parte cambiar el gestor de descargas de pacman por otro para descargar los paquetes más rápidamente aprovechando, por ejemplo, las descargas simultáneas que permiten algunos.

Para optimizar la base de datos existen varios scripts por internet, pero algunos de ellos pueden dañarla por lo que usaremos el oficial, que aunque no optimiza tanto como otros, es completamente seguro.

pacman-optimize && sync

Ahora se nos presentan varias opciones, podemos instalar wget, aria2 o Powerpill. Escoged según vuestras preferencias. Por supuesto, también podéis dejar Pacman como esta. Este cambio es útil además de para ganar velocidad de descarga, para poder usar configuraciones avanzadas de proxy, las cuales pacman no soporta.

Para usar wget abrimos una terminal y escribimos.

pacman -Sy wget

Ahora modificamos el archivo /etc/pacman.conf, añadiendo la siguiente línea al final del apartado options.

XferCommand = /usr/bin/wget -c –passive-ftp -c %u

Suele venir comentada (#) en el archivo, asi que seguramente sólo haya que descomentarla.

Para usar Aria2 en primer lugar lo instalamos.

pacman -Sy aria2

Y ahora añadimos la siguiente al archivo de configuración de pacman.

XferCommand = /usr/bin/aria2c –no-conf -s 2 -m 2 -d / -o %o %u

Powerpill es otra alternativa, pero en realidad no funciona como wget o aria2, funciona parecido a Yaourt. Una vez instalado tendremos que usar el comando powerpill para instalar los programas, con las mismas opciones que pacman. Por ejemplo, para actualizar el sistema, ejecutaríamos powerpill -Syu. Usando este programa si que notaremos mejoras significativas en el tiempo de descargas, ya que descarga varios paquetes de forma simultánea y usando diversos mirrors. Además usando Powerpill sólo se segmentarán los paquetes más grandes ahorrando el ancho de banda que se usa al segmentar paquetes pequeños.

Para instalar powerpill ejecutamos el siguiente comando.

pacman -Sy powerpill

Además de estos pequeños trucos para optimizar Pacman, encontraréis más en la wiki de Arch, eso sí, en ingles

В GUI synaptic есть возможность сгенерировать скрипт загрузки выбранных пакетов с их последующей установкой: “генерировать скрипт закачки пакетов” и “добавить закаченные пакеты”.
В случае с CLI можно использовать опции –print-uris в apt-get и вместо установки будет выведен список адресов для загрузки выбранных пакетов с учетом зависимостей.

apt-get -y –print-uris install список_пакетов

Список индексов:

apt-get –print-uris update

Эти пакеты можно загрузить отдельно и скопировать в /var/cache/apt/archives, например:

apt-get -qq -y –print-uris upgrade | cut -f1 -d’ ‘ | tr -d “‘” > packet_list.txt
cd /var/cache/apt/archives
wget –input-file packet_list.txt

затем повторить команду “apt-get install список_пакетов”, пакеты будут установлены из кэша.
Вместо /var/cache/apt/archives можно указать и другую директорию:

apt-get -o dir::cache::archives=”/download” install список_пакетов

Aprenderemos a bajarnos archivos y webs enteras usando wget en Linux. Todas las siguientes lineas de código las tenemos que poner en la consola de Linux pero sustituir la web por la que queramos.

Su uso básico es para bajarnos archivos así que pondremos:

wget http://sitioweb.com/archivo.zip

De este modo nos bajaríamos el archivo.zip, pero podemos añadirle otros parámetros y usarlos de modo que nos ayuden a perfeccionar nuestra tarea.

Si ahora ponemos:

wget -r http://sitioweb.com

Con el parámetro -r le estaremos diciendo que baje recursivamente todos los archivos del sitio web, esto significa que nos bajará imágenes, archivos html, etc. Pero si lo usan de este modo seguro que el sitio les bloquea la ip debido a que estamos enviando demasiadas peticiones seguidas y descargando todo, para evitar esto podríamos usar algo asi:

wget –random-wait –limit-rate=20k -r http://sitioweb.com

–random-wait Hace que entre un archivo y otro espere un periodo de tiempo aleatorio. De este modo el sitio de donde estamos bajando los archivos no se dará cuenta de nada.

–limit-rate=20k Lo que hace es limitar las descargas de los archivos a una transferencia máxima de 20k. Al igual que el parámetro anterior ayuda a evitar que nos baneen la IP mientras estamos bajando.

Aunque aún le podemos añadir mas parámetros, por ejemplo algo así:

wget –wait=20 –limit-rate=20K -r -p -U Mozilla http://sitioweb.com

–wait=20 Hace una pausa de 20 entre archivo y archivo pero creo que lo mejor seria usar random, –random-wait

-p Indica que descargue los archivos para ser visualizados correctamente como html, con esto me refiero a que se baje las páginas para verlas en tu disco como si estuvieras navegando por internet

-U mozilla Hará creer al sitio web que estamos usando algún navegador de la familia de Mozilla para hacer las descargas. Es decir como si estuviéramos navegando pro la web con Firefox.

También podemos poner niveles para bajar la página como en este ejemplo:

wget -r -l 4 http://sitioweb.com

-l 4 Significa que solo nos bajaremos cuatro niveles des de esa web, debemos tener en cuenta que cuatro niveles puede ser bastante y que a cada nivel se aumenta exponencialmente la información descargada. Podemos sustituir el 4 por otro numero, el numero de niveles que deseemos descargar.

Para descargar solo un tipo de archivo concreto podemos poner este código:

wget -r -A=.jpg,.png http://sitioweb.com

A=.jpg,.png Con este parámetro nos bajaríamos los archivos .jpg y .png. Aunque podemos poner A= #, dónde # equivale a cualquier otro formato, puede ser que dependiendo del formato no nos encuentre ningún archivo. Si queremos poner múltiples formatos debemos separarlos con comas y sin espacios. El formato debe incluir punto.

Espero que les aya servido, para cualquier cosa pueden dejar un comentario.

]]> http://nixtricks.wordpress.com/2009/10/30/cli-find-out-the-public-ip-in-the-command-line/ Fri, 30 Oct 2009 19:33:17 +0000 kousik http://nixtricks.wordpress.com/2009/10/30/cli-find-out-the-public-ip-in-the-command-line/

You may find out your IP using commands like ifconfig, but if your computer is behind a firewall, that is not the IP that the world sees. Here’s a quick how-to to find out what your external IP is.

First, let’s create an alias: (link)

Next time when you need the public IP of your computer, just type getip in the command line and hit enter.

The above command gets the IP from http://whatismyip.com/automation/n09230945.asp (just copy and paste this URL to your location bar and hit enter, and see what it does!). The “-q” is to suppress verbose information (quiet mode) and “-O -” causes the output to be written to STDOUT.

You may use dyndns.org too to find out the IP, but in this case you may need some trimming: substitute the text within the code above by the following:


wget http://checkip.dyndns.org/ -O - -o /dev/null | cut -d: -f 2 | cut -d\< -f 1

A few words about various flags above:
The output of the first command (everything before the first pipe) is

<code><html><head><title>Current IP Check</title></head><body>Current IP Address: xxx.xxx.xx.xx</body></html>

(the actual IP address is masked by x).
The “-o /dev/null“ part redirects the STDERR of wget to /dev/null. The “-d :” option in first “cut” tells it to use colon (:) to be used as the delimiter and “-f 2” causes it to print second of the delimited fields. Similarly, the flags of the second “cut” cause it to use “<” as the delimiter for the piped output from the first “cut” and choose the first of the delimited fields.

UPDATE: Another one using curl [link]. You may again change the quoted text within the alias above by the following:


curl --connect-timeout 3 http://www.whatismyip.org/

WGet utility is the best option to download files from internet. wget can pretty much handle all complex download situations including large file downloads, recursive downloads, non-interactive downloads, multiple file downloads etc., In this article let us review how to use WGet for various download scenarios using awesome wget examples :

1. Download Single File with wget

The following example downloads a single file from internet and stores in the current directory.

$ wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2

While downloading it will show a progress bar with the following information:

• %age of download completion (for e.g. 31% as shown below)
• Total amount of bytes downloaded so far (for e.g. 1,213,592 bytes as shown below)
• Current download speed (for e.g. 68.2K/s as shown below)
• Remaining time to download (for e.g. eta 34 seconds as shown below)

Download in progress:

$ wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
Saving to: `strx25-0.9.2.1.tar.bz2.1'

31% [=================> 1,213,592   68.2K/s  eta 34s

Download completed:

$ wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
Saving to: `strx25-0.9.2.1.tar.bz2'

100%[======================>] 3,852,374   76.8K/s   in 55s    

2009-09-25 11:15:30 (68.7 KB/s) - `strx25-0.9.2.1.tar.bz2' saved [3852374/3852374]

2. Download and Store With a Different File name Using wget -O

By default wget will pick the filename from the last word after last forward slash, which may not be appropriate always.

Wrong: Following example will download and store the file with name: download_script.php?src_id=7701

$ wget http://www.vim.org/scripts/download_script.php?src_id=7701

Even though the downloaded file is in zip format, it will get stored in the file as shown below.

$ ls
download_script.php?src_id=7701

Correct: To correct this issue, we can specify the output file name using the -O option as:

$ wget -O taglist.zip http://www.vim.org/scripts/download_script.php?src_id=7701

3. Specify Download Speed / Download Rate Using wget –limit-rate

While executing the wget, by default it will try to occupy full possible bandwidth. This might not be acceptable when you are downloading huge files on production servers. So, to avoid that we can limit the download speed using the –limit-rate as shown below.

In the following example, the download speed is limited to 200k

$ wget --limit-rate=200k http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2

4. Continue the Incomplete Download Using wget -c

Restart a download which got stopped in the middle using wget -c option as shown below.

$ wget -c http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2

This is very helpful when you have initiated a very big file download which got interrupted in the middle. Instead of starting the whole download again, you can start the download from where it got interrupted using option -c

Note: If a download is stopped in middle, when you restart the download again without the option -c, wget will append .1 to the filename automatically as a file with the previous name already exist. If a file with .1 already exist, it will download the file with .2 at the end.

5. Download in the Background Using wget -b

For a huge download, put the download in background using wget option -b as shown below.

$ wget -b http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
Continuing in background, pid 1984.
Output will be written to `wget-log'.

It will initiate the download and gives back the shell prompt to you. You can always check the status of the download using tail -f as shown below.

$ tail -f wget-log
Saving to: `strx25-0.9.2.1.tar.bz2.4'

     0K .......... .......... .......... .......... ..........  1% 65.5K 57s
    50K .......... .......... .......... .......... ..........  2% 85.9K 49s
   100K .......... .......... .......... .......... ..........  3% 83.3K 47s
   150K .......... .......... .......... .......... ..........  5% 86.6K 45s
   200K .......... .......... .......... .......... ..........  6% 33.9K 56s
   250K .......... .......... .......... .......... ..........  7%  182M 46s
   300K .......... .......... .......... .......... ..........  9% 57.9K 47s

Also, make sure to review our previous multitail article on how to use tail command effectively to view multiple files.

6. Mask User Agent and Display wget like Browser Using wget –user-agent

Some websites can disallow you to download its page by identifying that the user agent is not a browser. So you can mask the user agent by using –user-agent options and show wget like a browser as shown below.

$ wget --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" URL-TO-DOWNLOAD

7. Test Download URL Using wget –spider

When you are going to do scheduled download, you should check whether download will happen fine or not at scheduled time. To do so, copy the line exactly from the schedule, and then add –spider option to check.

$ wget --spider DOWNLOAD-URL

If the URL given is correct, it will say

$ wget --spider download-url
Spider mode enabled. Check if remote file exists.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

This ensures that the downloading will get success at the scheduled time. But when you had give a wrong URL, you will get the following error.

$ wget --spider download-url
Spider mode enabled. Check if remote file exists.
HTTP request sent, awaiting response... 404 Not Found
Remote file does not exist -- broken link!!!

You can use the spider option under following scenarios:

• Check before scheduling a download.
• Monitoring whether a website is available or not at certain intervals.
• Check a list of pages from your bookmark, and find out which pages are still exists.

8. Increase Total Number of Retry Attempts Using wget –tries

If the internet connection has problem, and if the download file is large there is a chance of failures in the download. By default wget retries 20 times to make the download successful.

If needed, you can increase retry attempts using –tries option as shown below.

$ wget --tries=75 DOWNLOAD-URL

9. Download Multiple Files / URLs Using Wget -i

First, store all the download files or URLs in a text file as:

$ cat > download-file-list.txt
URL1
URL2
URL3
URL4

Next, give the download-file-list.txt as argument to wget using -i option as shown below.

$ wget -i download-file-list.txt

10. Download a Full Website Using wget –mirror

Following is the command line which you want to execute when you want to download a full website and made available for local viewing.

$ wget --mirror -p --convert-links -P ./LOCAL-DIR WEBSITE-URL

• –mirror : turn on options suitable for mirroring.
• -p : download all files that are necessary to properly display a given HTML page.
• –convert-links : after the download, convert the links in document for local viewing.
• -P ./LOCAL-DIR : save all the files and directories to the specified directory.

11. Reject Certain File Types while Downloading Using wget –reject

You have found a website which is useful, but don’t want to download the images you can specify the following.

$ wget --reject=gif WEBSITE-TO-BE-DOWNLOADED

12. Log messages to a log file instead of stderr Using wget -o

When you wanted the log to be redirected to a log file instead of the terminal.

$ wget -o download.log DOWNLOAD-URL

13. Quit Downloading When it Exceeds Certain Size Using wget -Q

When you want to stop download when it crosses 5 MB you can use the following wget command line.

$ wget -Q5m -i FILE-WHICH-HAS-URLS

Note: This quota will not get effect when you do a download a single URL. That is irrespective of the quota size everything will get downloaded when you specify a single file. This quota is applicable only for recursive downloads.

14. Download Only Certain File Types Using wget -r -A

You can use this under following situations:

• Download all images from a website
• Download all videos from a website
• Download all PDF files from a website

$ wget -r -A.pdf http://url-to-webpage-with-pdfs/

15. FTP Download With wget

You can use wget to perform FTP download as shown below.

Anonymous FTP download using Wget

$ wget ftp-url

FTP download using wget with username and password authentication.

$ wget --ftp-user=USERNAME --ftp-password=PASSWORD DOWNLOAD-URL

source: here

Ever had a feeling there’s a tool you were searching for a long time sitting right in front of you and you couldn’t see it? Well, that’s how my search for a program that could download complete www pages onto my hard drive ended: wget -r --level=3. And suddenly the current version of learnyouahaskell.com is on my drive. Neat!

Twitter è un servizio fantastico anche se rimane un po’ incompreso dai più. Servizi come Facebook la fanno da padrone e forse anche a ragione. Questo però non vuol dire che Twitter debba essere trascurato o abbandonato. Twitter non ha di certo tutte le capacità che invece ci offre facebook ma proprio per questo forse va apprezzato. Twitter fa solo una cosa e la fa molto bene: permette agli utenti di scrivere piccole frasi (twit). Non c’è altro, solo questo. Altri utenti possono a loro discrezione decidere se seguire i tuoi twit oppure no e tu puoi fare altrettanto con gli altri utenti.

Per venire all’argomento del mio post, quello che mi ha sempre “infastidito” prima che trovassi la soluzione descritta qui, era che per aggiornare il mio status di twitter dovessi accedere via browser al sito oppure utilizzare programmi più o meno belli e più o meno completi. Quello che mi “infastidiva” era il fatto di rendere complessa una operazione che doveva fare una cosa molto semplice: l’aggiornamento dello status. Il mio obiettivo era avere uno strumento che mi permettesse di aggiornare lo status e nient’altro e di farlo nel modo più veloce possibile (per scrivere una frase di poche parole non devo aprire il browser ed effettuare l’accesso al sito altrimenti si perde il “gusto” del twit).

Finalmente grazie a due siti web sono riuscito a raggiungere il mio obiettivo. I due siti sono
http://www.networkworld.com/columnists/2008/052108-gearhead.html

http://phalacee.com/index.php?page=development&section=app&development=19

Il primo mi serve per Ubuntu e il secondo per Windows XP.

Il funzionamento di entrambi è molto semplice: creare un file che, ad esempio twitter, e metterlo in una cartella accessibile (sotto c:\windows o nella cartella dell’utente). All’interno del file ci sono alcune righe di codice. Per usarlo basta digitare nella finestra Esegui il comando: twitter “Nuovo status” e pensa a tutto lui.

Parte tecnica Ubuntu
Apri la finestra del terminale. Nella tua cartella utente crea un file chiamato twitter col comando
pico twitter

All’interno scrivi il seguente codice (con le dovute correzioni)
#!/bin/bash
curl -u nomeutente:password -d status=”$1″ http://twitter.com/statuses/update.xml >/dev/null

Salva il file con CTRL+O ed esci da pico con CTRL+X.
Dai i permessi di esecuzione al file appena creato con
chown u+rx twitter

Aggiornamento: nel caso in cui curl non sia installato di default nella tua distribuzione, installalo con sudo apt-get install curl

Per provare se funziona dai il comando
./twitter “scrivi quello che vuoi”
Funziona anche dalla finestra esegui in modalità grafica. Per aprirla premi ALT+F2.

Parte tecnica Windows XP
Vai sul sito phalacee.com che ho scritto sopra e scarica twitter CLI. Segui le istruzioni scritte all’interno del file zip.
Per scrivere i twit basta aprire la finestra di Esegui di Windows (Windows+R) e scrivere ad esempio twitter “Hello World”.

Introducción

wget es un estupendo programa que permite automatizar procesos de descarga de un servidor HTTP o FTP. Es de línea de comandos y esta pequeña página hace un repaso de las opciones más usuales.

Una forma de generar esas opciones es con el Generador de líneas de comandos de wget escrito en Javascript.
Opciones más comunes

Para ver las opciones del programa wget:

wget –help

Para mandar esas opciones a un fichero wget.txt:

wget –help > wget.txt

Para ver las opciones de pantalla en pantalla:

wget –help | more

Para descargar un fichero de la dirección http://www.playboy.com/virus.zip:

wget http://www.playboy.com/virus.zip

Para descargar un fichero de la dirección http://www.playboy.com/virus.zip, intentándolo un número indefinido de veces (-t 0), retomando la descarga en la próxima ejecución si se corta la conexión o se para el programa (-c), identificándose como un navegador web normal para evitar ser detectado por algunos servidores (-U “Mozilla”) y no identificándose como un robot para tener acceso completo a las páginas (-erobots=off):

wget -t 0 -c -U “Mozilla” -erobots=off http://www.playboy.com/virus.zip

Para descargar completamente una página (-r) y convertir los enlaces para verla localmente (-k):

wget -t 0 -c -U “Mozilla” -erobots=off -r -k http://xxx.unizar.es

Para descargar la lista de enlaces de un fichero urls.txt:

wget -i urls.txt

Para aceptar solamente los ficheros JPG y MPG (-A *.jpg,*.mpg):

wget -t 0 -c -U “Mozilla” -erobots=off -r -k -A *.jpg,*.mpg http://www.fotos_y_videos_guarros.es

Espero que esta lista de opciones algo depravadas dé una idea de las posibilidades.

Controllare la posta in arrivo su GMail da riga di comando
$ curl -u USERNAME --silent "https://mail.google.com/mail/feed/atom" | perl -ne 'print "\t" if //; print "$2\n" if /(.*)/;'
Continua a leggere…

Aggiungere sottotitoli ad un file video
$ mencoder -ovc xvid -xvidencopts fixed_quant=3 -oac copy -sub filesottotitoli.srt -subfont-text-scale 4 -o filevideoconsottotitoli.avi filevideosenzasottotitoli.avi
Continua a leggere…

wget + Google Image Ripper (script Bash)
Un grabber in bash che usa wget e GIR.
Continua a leggere…

How to Set Up a High Performance Cluster (HPC) Using Debian Lenny and Kerrighed
Cluster controller or Head node: Debian Lenny or any other distribution. The single image passed on to the nodes will be Debian Lenny though.
Continua a leggere…

Make the Minimalist, Fading Google Homepage Your Default

javascript:void(document.cookie="PREF=ID=abac7a90f5a3784b:LD=en:NR=10:TM=1254990196:LM=1254990236:S=uB6F4jDnMP_DuxtT;path=/; domain=.google.com");

Continua a leggere…

[^] torna su | post<li> | 

If you are behind a router, ifconfig won’t give you your external IP address. As a possible solution you can (w)get it from http://checkip.dyndns.org/ and grep your external IP from the html file:

 wget http://checkip.dyndns.org/ -qO - | grep -Eo '\<[[:digit:]]{1,3}(\.[[:digit:]]{1,3}){3}\>'

GNU Wget is a popular file download program, being installed by default on many Linux distributions. Recent Mac OS versions don’t ship Wget, though – Apple ships cURL instead.

Fink provides a wget package that installs Wget. It includes SSL (https) support provided by Mac OS built-in OpenSSL.  There’s a problem with that, though: on Mac OS versions earlier than 10.6, Apple’s OpenSSL doesn’t use the trusted root certificates available on the system (the ones listed by Keychain.app), so it is not able to validate SSL certificates on its own. Note that OpenSSL itself (independently of being shipped with Mac OS) isn’t distributed with root certificates by default.

Because of this, on Mac OS versions earlier than 10.6 the command

wget https://fedorahosted.org

won’t work:

ERROR: cannot verify fedorahosted.org's certificate,
issued by `/C=US/O=Equifax/OU=Equifax Secure
Certificate Authority':
Unable to locally verify the issuer's authority.
To connect to fedorahosted.org insecurely, use
`--no-check-certificate'.
Unable to establish SSL connection.

There are a couple of options to circumvent this. As the error message says, it’s possible to use –no-check-certificate, which is insecure. Another option is –ca-certificate=file where file is a bundle of trusted certification authority certificates. Fink provides a package called ca-bundle that installs a convenient file containing a bundle of CA certificates commonly used by open source Web browsers. After running

fink install ca-bundle

you should be able to use /sw/etc/ssl/certs/ca-bundle.crt with Wget:

wget --ca-certificate=/sw/etc/ssl/certs/ca-bundle.crt \
https://fedorahosted.org

Fortunately, you may specify that option in one of Wget’s startup files (e.g. $HOME/.wgetrc or /sw/etc/wgetrc) by adding the following line to your startup file of choice:

ca_certificate = /sw/etc/ssl/certs/ca-bundle.crt

And voilà!, you may use wget as usual:

wget https://fedorahosted.org

This is particularly useful if you’re using Wget as your DownloadMethod and Fink needs to download a source file from an https URL.

wget utility is the best option to download files from internet. wget can pretty much handle all complex download situations including large file downloads, recursive downloads, non-interactive downloads, multiple file downloads etc.,

In this article let us review how to use wgetfor various download scenarios using 15 awesome wget examples.

1. Download Single File with wget

The following example downloads a single file from internet and stores in the current directory.

$ wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2

While downloading it will show a progress bar with the following information:

• %age of download completion (for e.g. 31% as shown below)
• Total amount of bytes downloaded so far (for e.g. 1,213,592 bytes as shown below)
• Current download speed (for e.g. 68.2K/s as shown below)
• Remaining time to download (for e.g. eta 34 seconds as shown below)

Download in progress:

$ wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
Saving to: `strx25-0.9.2.1.tar.bz2.1'

31% [=================> 1,213,592   68.2K/s  eta 34s

Download completed:

$ wget http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
Saving to: `strx25-0.9.2.1.tar.bz2'

100%[======================>] 3,852,374   76.8K/s   in 55s    

2009-09-25 11:15:30 (68.7 KB/s) - `strx25-0.9.2.1.tar.bz2' saved [3852374/3852374]

2. Download and Store With a Different File name Using wget -O

By default wget will pick the filename from the last word after last forward slash, which may not be appropriate always.

Wrong: Following example will download and store the file with name: download_script.php?src_id=7701

$ wget http://www.vim.org/scripts/download_script.php?src_id=7701

Even though the downloaded file is in zip format, it will get stored in the file as shown below.

$ ls
download_script.php?src_id=7701

Correct: To correct this issue, we can specify the output file name using the -O option as:

$ wget -O taglist.zip http://www.vim.org/scripts/download_script.php?src_id=7701

3. Specify Download Speed / Download Rate Using wget –limit-rate

While executing the wget, by default it will try to occupy full possible bandwidth. This might not be acceptable when you are downloading huge files on production servers. So, to avoid that we can limit the download speed using the –limit-rate as shown below.

In the following example, the download speed is limited to 200k

$ wget --limit-rate=200k http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2

4. Continue the Incomplete Download Using wget -c

Restart a download which got stopped in the middle using wget -c option as shown below.

$ wget -c http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2

This is very helpful when you have initiated a very big file download which got interrupted in the middle. Instead of starting the whole download again, you can start the download from where it got interrupted using option -c

Note: If a download is stopped in middle, when you restart the download again without the option -c, wget will append .1 to the filename automatically as a file with the previous name already exist. If a file with .1 already exist, it will download the file with .2 at the end.

5. Download in the Background Using wget -b

For a huge download, put the download in background using wget option -b as shown below.

$ wget -b http://www.openss7.org/repos/tarballs/strx25-0.9.2.1.tar.bz2
Continuing in background, pid 1984.
Output will be written to `wget-log'.

It will initiate the download and gives back the shell prompt to you. You can always check the status of the download using tail -f as shown below.

$ tail -f wget-log
Saving to: `strx25-0.9.2.1.tar.bz2.4'

     0K .......... .......... .......... .......... ..........  1% 65.5K 57s
    50K .......... .......... .......... .......... ..........  2% 85.9K 49s
   100K .......... .......... .......... .......... ..........  3% 83.3K 47s
   150K .......... .......... .......... .......... ..........  5% 86.6K 45s
   200K .......... .......... .......... .......... ..........  6% 33.9K 56s
   250K .......... .......... .......... .......... ..........  7%  182M 46s
   300K .......... .......... .......... .......... ..........  9% 57.9K 47s

Also, make sure to review our previous multitail article on how to use tail command effectively to view multiple files.

6. Mask User Agent and Display wget like Browser Using wget –user-agent

Some websites can disallow you to download its page by identifying that the user agent is not a browser. So you can mask the user agent by using –user-agent options and show wget like a browser as shown below.

$ wget --user-agent="Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.3) Gecko/2008092416 Firefox/3.0.3" URL-TO-DOWNLOAD

7. Test Download URL Using wget –spider

When you are going to do scheduled download, you should check whether download will happen fine or not at scheduled time. To do so, copy the line exactly from the schedule, and then add –spider option to check.

$ wget --spider DOWNLOAD-URL

If the URL given is correct, it will say

$ wget --spider download-url
Spider mode enabled. Check if remote file exists.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Remote file exists and could contain further links,
but recursion is disabled -- not retrieving.

This ensures that the downloading will get success at the scheduled time. But when you had give a wrong URL, you will get the following error.

$ wget --spider download-url
Spider mode enabled. Check if remote file exists.
HTTP request sent, awaiting response... 404 Not Found
Remote file does not exist -- broken link!!!

You can use the spider option under following scenarios:

• Check before scheduling a download.
• Monitoring whether a website is available or not at certain intervals.
• Check a list of pages from your bookmark, and find out which pages are still exists.

8. Increase Total Number of Retry Attempts Using wget –tries

If the internet connection has problem, and if the download file is large there is a chance of failures in the download. By default wget retries 20 times to make the download successful.

If needed, you can increase retry attempts using –tries option as shown below.

$ wget --tries=75 DOWNLOAD-URL

9. Download Multiple Files / URLs Using Wget -i

First, store all the download files or URLs in a text file as:

$ cat > download-file-list.txt
URL1
URL2
URL3
URL4

Next, give the download-file-list.txt as argument to wget using -i option as shown below.

$ wget -i download-file-list.txt

10. Download a Full Website Using wget –mirror

Following is the command line which you want to execute when you want to download a full website and made available for local viewing.

$ wget --mirror -p --convert-links -P ./LOCAL-DIR WEBSITE-URL

• –mirror : turn on options suitable for mirroring.
• -p : download all files that are necessary to properly display a given HTML page.
• –convert-links : after the download, convert the links in document for local viewing.
• -P ./LOCAL-DIR : save all the files and directories to the specified directory.

11. Reject Certain File Types while Downloading Using wget –reject

You have found a website which is useful, but don’t want to download the images you can specify the following.

$ wget --reject=gif WEBSITE-TO-BE-DOWNLOADED

12. Log messages to a log file instead of stderr Using wget -o

When you wanted the log to be redirected to a log file instead of the terminal.

$ wget -o download.log DOWNLOAD-URL

13. Quit Downloading When it Exceeds Certain Size Using wget -Q

When you want to stop download when it crosses 5 MB you can use the following wget command line.

$ wget -Q5m -i FILE-WHICH-HAS-URLS

Note: This quota will not get effect when you do a download a single URL. That is irrespective of the quota size everything will get downloaded when you specify a single file. This quota is applicable only for recursive downloads.

14. Download Only Certain File Types Using wget -r -A

You can use this under following situations:

• Download all images from a website
• Download all videos from a website
• Download all PDF files from a website

$ wget -r -A.pdf http://url-to-webpage-with-pdfs/

15. FTP Download With wget

You can use wget to perform FTP download as shown below.

Anonymous FTP download using Wget

$ wget ftp-url

FTP download using wget with username and password authentication.

$ wget --ftp-user=USERNAME --ftp-password=PASSWORD DOWNLOAD-URL

က်ေနာှက ရုပ္ရွင္ျကိုက္တဲ့သူျဖစ္တာမို့ ရုပ္ရွင္ေတြကို အင္တာနက္ကေန download လုပ္ဖို့အတြက္ အလြယ္ကူဆံုးျဖစ္တဲ့နည္းကို ရွာျပီး သံုးေနခဲ့ပါတယ္။ က်ေနာှက တစ္ခါတစ္ေလ ရုပ္ရွင္ ၁၅ ကား အကား ၂၀ ေလာက္ကို တစ္ခါတည္း download လုပ္တာဆိုေတာ့ ျကည့္ခ်င္တဲ့ရုပ္ရွင္ rapidshare links ေတြကို အရင္စုစည္းျပင္ဆင္ပါတယ္။ လင့္ခ္ေတြအာလံုးပါတဲ့ text file တစ္ဖိုင္ကိုျပင္ဆင္ပါတယ္။ ဒီလိုျပင္ဆင္တဲ့ေနရာမွာလဲ ကိုယ္ download လုပ္ခ်မယ့္ links ေတြက အလုပ္လုပ္ေသးလား ေသေနျပီလားဆိုတာကိုပါ တစ္ခါတည္းစစ္ေဆးျကည့္ရွုျပီးမွ အလုပ္လုပ္တဲ့လင့္ခ္ေတြကိုပဲ တစ္ခါတည္း စုစည္းပါတယ္။ အဲလိုလင့္ခ္ေတြကို text file နဲ့ျပင္ဆင္ျပီးမွ wget နဲ့ တစ္ခါတည္း ဆြဲခ်ေတာ့ပါပဲ။ ဘာမွထပ္လုပ္စရာမလိုေတာ့ပါဘူး။

Requirements:
1) Internet connection
2) Mozilla Firefox
3) Firefox plugins: Greasemonkey, Copy Links
4) Userscript: Rapidshare Links Checker
5) Text Editor: gedit, notepad++ (or whatever)
6) wget
7) Rapidshare Premium account

Environments Setup:
Firefox မွာ အေပာှကလိုအပ္တဲ့ plugins ေတြ install လုပ္ပါ။ Greasemonkey plugin ကို install လုပ္ျပီးတဲ့အခါ Rapidshare Links Checker userscript ကို install လုပ္ပါ။ ကိုယ့္စက္ထဲမွာ text editor နဲ့ wget ကိုရွိေနမယ္လို့ယူဆပါမယ္။ မရွိရင္ install လုပ္ေပးထားမယ္။ ေနာက္ျပီး ကိုယ့္စက္က Internet နဲ့ခ်ိန္ဆက္ထားတယ္လို့လဲ ယူဆပါတယ္။

Downloading:
$ wget --auth-no-challenge --user=RS_USER_LOGIN --password=RS_PREMIUM_PASSWORD -i rs_links.txt

ဒီေနရာမွာ;
RS_USER_LOGIN ဆိုတာက rapidshare premium user login name ပါ။
RS_PREMIUM_PASSWORD ေနရာမွာ rapidshare premium password ကိုအစားသြင္းေပးပါ။
rs_links.txt ဆိုတာက rapidshare links ေတြပဲ စုစည္းထားတဲ့ text file ပါ။

Enjoy with it!

I’ve been having an issue recently (not sure if it’s the Mac) but it has indeed become very apparent to me, that I would download things (applications / videos / ISOs / etc) in Firefox and the download would ‘finish’. It shows it as completed in the downloads window but in fact it is a truncated download. As seen in the screenshots below the SDK finished at 1.6GB when in fact the actual full file size is 2.1GB.

I initially tried using wget to download the file but got a ERROR 403: Forbidden error probably due to some cookie credentials. So not wanting to waste my 1.6GB download that took about 12 hours I resorted to looking into getting wget to work with cookies and was happy to see that the folks who created wget indeed had an option to load the cookies file. Sweet! Very sweet indeed. So next task was to get the cookies out of Firefox which was a bit more of an issue.

Export Cookies was known to work but it was not compatible with Firefox 3.5 so I had to download the Nightly Tester Tools to allow Export Cookies to work. Export the cookies (“Tools > Export cookies”) and in Terminal/Command line and enter the wget command to resume the download as follows:
wget --server-response --continue --load-cookies cookies.txt http://location/of/download

Download wget (Windows | Mac)

Заменяем определенный текст в файле

perl -i -pe 's/text0/text1/;' file_name

Здесь text0 — последовательонсть символов в текстовом файле которую хотим заменить, text1 — последовательность символов, которая и заменит исходный отрывок text0. file_name — имя файла в котором производится замена, можно использовать символы групповых операций (* ? []).

Удаляем теги html из текстового файла

sed 's/<[^>]*>//g' index.html

index.html — файл в котором ищутся и удаляются теги.

Принудительное завершение сеанса через 24 часа

fuser -k `who -u &#124; awk '$6 == "old { print "/dev/"$2'}`</code></pre>
Эта команда откроет заново login shell для любого пользователя, находящегося за компьютером более суток. Можно контролировать детей =).

<strong>Создание iso образа с CD/DVD</strong>
[sourcecode language='bash']readom dev=/dev/scd0 f=/path/to/image.iso

/dev/scd0 — ваш привод для CD/DVD дисков (можно уточнить при помощи комнады ls /dev/sd*). /path/to/image.iso — путь к будущему iso образу. dd подходит хуже для этих целей, т.к. она не делает никакую проверку на ошибки. Использую команду dd вы не можете точно знать скопирована вся информация или нет.

Выключаем машину после загрузки файла (wget’ом)

while [ -n "`pgrep wget`" ]; do sleep 2 ;done; [ -e "/tmp/nosleep"] || echo mem >/sys/power/state

Вы скачиваете файл из сети, но уже наступил вечер и вам хочется пойти спать. Кто выключит ПК за вас? Можно конечно оставить работать компьютер до утра, а можно и воспользоваться приведенной командой.
Если вдруг вы решите, что ПК отключать не надо, а загрузку приостанавливать не хочется, то просто созадйте файл nosleep в директории /tmp/ (touch /tmp/nosleep).

http://www.commandlinefu.com/

Hay muuuchas cosas que es mas facil hacerlas desde la terminal. Una de ellas es agregar la llave de un repositorio desde la terminal.
Por ejemplo, hoy queria instalar este software para la reproduccion de archivos multimedia. Segun las instrucciones de su sitio tenia que ir a un link, copiar el texto que aparecia ahi, pegarlo en un archivo nuevo, luego ir a la configuracion de los repositorios e importar el archivo recien creado.
Pues la verdad se me hizo mas facil hacer lo siguiente:

wget -q “http://keyserver.ubuntu.com:11371/pks/lookup?op=get&search=0×2EEE37419EDAD4D4″ -O – | sudo apt-key add -

El primer comando (wget) descarga (de forma silenciosa, “-q”) el archivo indicado en la URL y lo envia a la salida estandar (“-O -”) despues mediante el uso de una tuberia ( “|” ) lo envia al comando apt-key, el cual la agrega leyendola de la entrada estandar (“-”).
Nota: Hay que poner ojo, entre -O y – hay un espacio, para ver que onda con esto checa la doc mediante “man wget” y “man apt-key”

byte!

Se avete la necessità di utilizzare la vostra distribuzione server dietro un proxy e dovete utilizzare wget eccovi alcuni comandi utili.

Definire una variabile di sistema prima di utilizzare wget, nel seguente modo:
export http_proxy="http://proxy.example.com:8080"
Quindi potete specificare la seguente opzione in wget dalla linea di comandoper attivare il funzionamento dietro un proxy:
–proxy=on
la seguente è per disattivare l’utilizzo dietro proxy server:
–proxy=off
Se nel vostro server proxy sono richiesti nome utente e password potete usare le seguenti opzioni, sempre nella linea di comando di wget:
-proxy-username="user name" –proxy-passwd="password"
Un alternativa è specificarli nella variabile di sistema  http_proxy / ftp_proxy come segue:
export http_proxy="http://username:password@proxy.example.com:8080"