I was asked how to use grep to select everything in a file starting with a pattern, and ending with a different one. The file is our diagnostic log and if this has originated with one of our system testers could be massive (a few hundred thousand lines long). gnu-grep can be used for this. You could do something like:

grep -A 9999999 'some first expression' < wayToBigFile | grep -B 9999999 'some other expression'

Here 9999999 is some number of lines that is guessed big enough to contain all the lines of interest (not known ahead of time), so the command says “give me everything after the expression, and then give me everything before the other expression in that output”

Perl gives you a nicer way of doing this:

#!/usr/bin/perl -n

$foundIt = 1 if ( /some first expression/ ) ;

next if ( !$foundIt ) ;

print ;

exit if ( /some other expression/ ) ;

# done.

The -n flag says to run the whole script as if it is in a ‘while (<>){ … }’ loop.  Until the initial pattern is seen $foundIt is false, and nothing will be printed, and we bail if the second pattern is seen. Note that this relies on perl’s lazy variable initialization, since $foundIt = 0 until modified.

Observe also that this script is actually also it’s own test case.

myPrompt$ chmod 755 ./theScript
myPrompt$ ./theScript < ./theScript
$foundIt = 1 if ( /some first expression/ ) ;

next if ( !$foundIt ) ;

print ;

exit if ( /some other expression/ ) ;

I wrote a utility that can grep through flat files and look for user-defined regular expressions. I included the ability to search the files and adjacent folders recursively, as well as the capability to count the number of lines (per file) that results were found on. There are a few more configurable options to control the search (disable/enable cases sensitivity or search file names). Lines that contain the regular expression are returned right to the results window!

Screenshot (click the image to enlarge it):

Download Link:

Como utilizo muito terminal  Solaris no trabalho e Ubuntu em casa, um pequeno script é muito útil quando fazemos uma busca por arquivos e no resultado queremos saber qual deles tem um determinado texto dentro:

$ for i in `find . -name '*.trc' -ls | grep 'Apr 22' | awk -F" " '{ print $9 }'`; do grep "texto" $i; done

excplicando:

for i in # loop com a iteração em i

`find . -name ‘*.trc’ -exec -ls -l {}; | grep ‘Apr 22′ | aw’k -F” ” ‘{ print  $8}’`; # primeiro procure, a partir do diretório atual todos arquivos com extensão .trc e os mostre utilizando um “ls -l”, na lista filtre os arquivos pela data de 22 de abril e por fim exiba somente a 8 coluna da lista, usando espaço como separador de colunas.

do grep “texto” $i; # a partir do resultado da linha acima faz uma busca do texto em cada arquivo

done # final do for

Um observação importante é que a data e a quantidade de colunas pode variar de acordo com o sistema operacional.

Nothing really needing explaining here, but wanted an example for doing a recrsive grep handy as I can never remember it.

grep -RH –include “*.java” TEXT_I_AM_LOOKING_FOR *

En esta Oportunidad les Traigo una recopilación de lo q yo concidero son los comandos más, utiles, importantes y usados tanto en Ubuntu como en muchas otras distros linux

• Adduser: Se utiliza para añadir un usuario.

Uso: adduser <nombreusuario>

• Alias (este esta bien explicado en un post anterior)
  

Uso: alias nombrequeledamos=’comando’

• apt-get install nombredelprograma =  instala el paquete.

• apt-get remove nombredelprograma = Borra el paquete. Con la opción –purge borra tambien la configuración del paquete instalado.

• apt-get update = Actualiza la lista de paquetes disponibles para instalar.

• apt-get upgrade  = Instala las nuevas versiones de los diferentes paquetes disponibles.

• cd = Cambia el directorio.

Uso: cd nombredirectorio

Continuar leyendo el resto de comandos:

• chmod: Cambia los permisos de los archivos.

r:lectura w:escritura x:ejecución
+: añade permisos -:quita permisos
u:usuario g:grupo del usuario o:otros
Uso: chmod permisos nombrearchivo

• chown: Cambia el propietario de un archivo.

Uso: chown <nombrepropietario> <nombrearchivo>

• cp:Copia archivos en el directorio indicado.

Uso: cp archivoorigen archivodestino

• deluser: Elimina una cuenta de usuario

Uso: deluser <nombreusuario>

• exit: Cierra las ventanas o las conexiones remotas establecidas

• fsck: Sirve para chequear si hay errores en nuestro disco duro.

Uso: fsck -t <fs_typo> <dispositivo>

• ftp: Protocolo de Transferencia de Archivos, permite transferir archivos de y para computadores remotos.

Uso: ftp <maquina_remota>

• grep: Su funcionalidad es la de escribir en salida estándar aquellas líneas que concuerden con un patrón. Busca patrones en archivos.

grep: grep <patronabuscar> <nombrearchivos>

• ifconfig: Obtener información de la configuración de red

• kill: Termina un proceso

Uso: kill numeroPID

• ls:Lista los archivos y directorios dentro del directorio de trabajo.

• make: Crea ejecutables a partir de los archivos fuente.

• man: muestra el manual de un programa o comando si este esta disponible

Uso: man <nombredelcomando>

• mkdir: Crea un nuevo directorio.

Uso: mkdir nombredirectorio

• mount: Monta una unidad.

Uso: mount -t< sistema_de_archivo> <dispositivo> <nombredirectorio>

• mv: Este comando sirve para renombrar un archivo.

Uso: mv <nombrearchivo> <nuevonombrearchivo>

• netstat:Muestra las conexiones y puertos abiertos por los que se establecen las comunicaciones.

• poweroff: Apaga el ordenador.

• ps: Muestra información acerca de los procesos activos.

• rlogin: Conecta un host local con un host remoto.

Uso: rlogin <maquina_remota>

• rm: Remueve o elimina un archivo.

Uso: rm <rutaynombrearchivo>

• rmdir: Elimina el directorio indicado, el cual debe estar vacío.

Uso: rmdir nombredirectorio

• sudo o su: Con este comando accedemos al sistema como root.

• umount: Desmontar unidades montadas anteriormente

Uso: umount <nombredirectorio>

• wc:Cuenta los caráteres, palabras y líneas del archivo de texto.

Uso: wc nom_archivo

• whereis: Devuelve la ubicación del archivo especificado, si existe.

Uso: whereis nomb_archivo

Larga vida a la consola!!

Someone once asked me in an interview: “What is your favorite Unix tool?” Not sure of what to say, I simply tried to avoid appearing ignorant, and replied: “grep.”

Thinking about this problem again, I realized that there is a much better answer. Today, my favorite Unix tool is: the pipe.

modules.pcimap file can be found in the following directory :
/lib/modules/$(uname -r)/

This file is used to find the corresponding module for a PCI device
providing the model id through a 'grep' command :
Here, the model id is 0900 :

grep 0900 /lib/modules/2.6.26-2-686/modules.pcimap |more

The result consists of many matching lines. Only one is the good one.
You have to issue a new 'grep' command with the vendor id :
Here, the vendor id is 1039 :

grep 0900 /lib/modules/2.6.26-2-686/modules.pcimap |grep 1039

And the result is : 

 sis900            0x00001039 0x00000900  0xffffffff 0xffffffff 0x00000000 0x00000000    0x0

If you want to have more infos about this module : 

modinfo sis900

To create or to re-create this modules.pcimap file, issue the command below :

depmod -a -m -F /boot/System.map-2.6.26-2-686 2.6.26-2-686

Looking for a certain functionality in someone else’s code could prove to be a tough challenge. This is increased by the size of the given project. Many systems exist today facilitating easy search in a project. For Java, eclipse is best used for that task, as it contains a strong in java class indexing, classification, and searching.

Some projects provide for you a portal to search their code base. Famous examples of these are LXR for the Linux kernel, and it’s customized version MXR used for Mozilla projects.

When all the above options are not available, the alternative exists natively in most linux systems. This is namely “grep”. Grep is a strong command for regular expression matching. On if it’s features is that it can also match to the content within files. Putting that in mind, the following command could be very useful:

grep -R "search query" .

This looks recursively for all mentions of search query in all files under the current directory. This prints output in the console the file name where this query was found, and the line that included it.

To be able to store this search result in a file, the output could be piped as follows:

grep -R "search query" . > log.txt

I sometime have problem finding which persistent process is not letting me eject a volume (CD/DVD/USB drive). The error message is not very helpful at all:

Try quitting applications

OK, but which one(s)? I often have hard time figuring that out using htop/ top or a combination of ps and grep.

But actually this is what I want:

where Volume_Name is the name of the volume (or an ‘identifying’ part of the name) I want to eject. This gives me the name(s) and process ID(s) of the run-away process(es) as well as the path to the file(s) in use on the volume, Volume_Name.

Now that I know the names I should first try to save the documents and quit the applications involved in the normal way. However, if a particular application does not oblige, I can always kill it (with the risk of potential data loss!) using its process ID (say, 123):

After that, the ejection of the volume should not be difficult at all.

Needless to say, being a Unix utility, lsof (=“list open files”) may be used on other *nix-based systems as well.

Introdução

Certa vez, após instalar o GLPI (sistema de helpdesk), houve a necessidade de customiza-lo para ficar de acordo com as necessidades da empresa. O grande problema, era saber em qual arquivo encontrar o conteúdo que deveria ser alterado.

Segue abaixo um exemplo de como efetuar a busca:

# grep -R manaus /home/smith/ 2>/dev/null | cut -d: -f2 | uniq

Vamos dissecar o comando:

grep -R manaus /home/smith/ – Procura recursivamente pela palavra “manaus”, em todos arquivos a partir do diretório /home/smith.

2>/dev/null – Direciona as linhas de erro para /dev/null.

| (pipe) – Redireciona a saída do comando anterior (grep -R manaus /home/smith/ 2>/dev/null), para a entrada do próximo comando.

cut -d: -f2 – O grep gera um arquivo no seguinte formato: /path/arquivo:conteúdo da linha
Como queremos saber apenas qual arquivo contém a palavra “manaus”, utilizaremos o comando cut para tratar a saída. O cut lê a saída do comando anterior e seleciona apenas o segundo campo (-f2), separado pelo delimitar (-d) “dois pontos” (“:”), ignorando tudo que estiver depois do delimitador.

uniq – Caso haja mais de uma ocorrência da palavra “manaus” em um mesmo arquivo, será exibida uma linha para cada ocorrência. O uniq remove as linhas duplicadas.

:wq!
Andrey Smith

I want to filter log files by matching lines that contains all of  set of matches (I don’t want to alternate them).

For example lines like:

RawURL /1.2/user/authorize?ip=10.254.142.175&method=GET

I want to match lines that contain the terms “RawURL” and “/1.2/user/authorize?” without resorting to piping multiple greps:

$ grep RawURL | grep /1.2/user/authorize? *.txt

This means I want a regexp like:

RawURL.+\/1.2\/user\/authorize

Translating to grep extended regex, this becomes:

$ grep -E RawURL.+/1.2/user/authorize *.txt

Note: There is no need to enforce non-greedy matching.

Regular expressions, character classes and special characters

The expression:

RawURL.+\/1.2\/user\/authorize

Is not equivalent to:

RawURL[.]+\/1.2\/user\/authorize

Because in the second one, the special character ‘.’ is no longer special. Character classes have all special characters turned off.

Recently I got stuck trying to fix a problem in Heroku, a slick cloud-based Ruby hosting service. The problem involved gems that I had added to my app’s private gem repository. I had to check the versions of the gems in the repository, but Heroku’s documentation on gem management didn’t provide any suggestions.

Turns out, the trick is to use Heroku’s console command, and then use the Ruby Gem command. To open a new console session, fire up a terminal and enter:

heroku console

Then list the versions of the gems in your private repo, by enclosing the gem list command in backticks. (On Macs, the backtick key is in the upper left corner of the keyboard. I know, it took me a few seconds to find it as well.)

`gem list`

Since Heroku has many built-in gems, you’ll probably want to pipe through grep to find what you want:

`gem list | grep foo`

And you’re off to the races!

Found a nice little script on mroddball.wordpress.com. It uses curl to upload images to ImageShack:

curl -H Expect: -F fileupload="@example.jpg" -F xml=yes -# "http://www.imageshack.us/index.php"

If you invoke the above command the image will be uploaded and you get an xml file like this:

<?xml version="1.0" encoding="iso-8859-1"?><links>
    <image_link>http://img123.imageshack.us/img123/4567/example.jpg </image_link>
    <thumb_link>http://img123.imageshack.us/img123/4567/example.th.jpg </thumb_link>
    <ad_link>http://img123.imageshack.us/my.php?image=example.jpg </ad_link>
    <thumb_exists>no</thumb_exists>
    <total_raters>0</total_raters>
    <ave_rating>0.0</ave_rating>
    <image_location>img123/4567/example.jpg</image_location>
    <thumb_location>img123/4567/example.th.jpg</thumb_location>
    <server>img123</server>
    <image_name>bricks.jpg</image_name>
    <done_page>http://img123.imageshack.us/content.php?page=done&amp;l=img123/4567/example.jpg</done_page>
    <resolution>64x64</resolution>
    <filesize>11771</filesize>
    <image_class>r</image_class>
</links>

You can use grep to parse the image link from the xml file:

curl -H Expect: -F fileupload="@example.jpg" -F xml=yes -# "http://www.imageshack.us/index.php" | grep image_link | grep -o  http[^\<]*

You could also use find, for etc. to upload as many pictures as you like at the same time. The whole script is located on pastebin. Visit mroddball.wordpress.com for further information. ]]> http://cambrico.info/2009/10/11/el-origen-de-grep-en-unix/ Sun, 11 Oct 2009 15:24:52 +0000 Ruben Antonio Fernández http://cambrico.info/2009/10/11/el-origen-de-grep-en-unix/ http://redbeardtechnologies.wordpress.com/2009/10/09/encode-mov-files-with-ffmpeg-h246/ Fri, 09 Oct 2009 15:24:57 +0000 redbeardtechnologies http://redbeardtechnologies.wordpress.com/2009/10/09/encode-mov-files-with-ffmpeg-h246/

Hello,

I had a project today to mass encode a bunch of videos that are in the .mov format. The requirements for the job specify if the video file is 640×480 resolution just encode the video with the H264 codec. If the video is not 640×480 I must resize the video to 720×480. I wrote a little bash script to help me sort through the files and encode them.

#!/bin/bash
files="$(ls *\.mov)"
for movie_file in $files ; do
    if [ -f "${movie_file}" ]
    then
        video_size="$(ffmpeg -i "${movie_file}" 2>&1 | grep -m 1 Video |  grep -oE "[[:digit:]]{3,4}x[[:digit:]]{3,4}")"
        if [ "${video_size}" = "640x480" ]
        then
            ffmpeg -i ${movie_file} -acodec libfaac -r 30 -ac 2 -vcodec libx264 -vpre hq -crf 22 -sameq -threads 0 "${movie_file}.mp4"
        else
            ffmpeg -i ${movie_file} -acodec libfaac -r 30 -ac 2 -vcodec libx264 -vpre hq -crf 22 -sameq -threads 0 -s 720x480 "${movie_file}.mp4"
        fi
        qt-faststart ${movie_file}.mp4 ${movie_file}.fs.mp4
    fi
done

Let me explain how the program works for those unfamiliar with Bash

files="$(ls *\.mov)"

The command ls gets a directory listing with only the .mov files and saves it into the variable “files”. In this example I am only looking for files with the .mov file extension.

for movie_file in $files ; do
 [stuff]
done

The for loop lets my program iterate through the files available in the directory that have a .mov file extension.

if [ -f "${movie_file}" ]

This just checks that the file is valid and exists.

video_size="$(ffmpeg -i "${movie_file}" 2>&1 | grep -m 1 Video |  grep -oE "[[:digit:]]{3,4}x[[:digit:]]{3,4}")"

This command is essential for identifying the video resolution. I use ffmpeg to get information about the file and then use grep to determine the resolution using a regular expression. It is important to note that the grep command “grep -m 1 Video” only matches the first resolution found. Some times ffmpeg will list multiple video resolutions. If the resolution is listed more than once the bash program fails.

ffmpeg -i ${movie_file} -acodec libfaac -r 30 -ac 2 -vcodec libx264 -vpre hq -crf 22 -sameq -threads 0 "${movie_file}.mp4"

This command converts the video file using the H264 codec. I try to retain the same quality as the original video using the parameters shown above.

qt-faststart ${movie_file}.mp4 ${movie_file}.fs.mp4

qt-faststart puts the meta data contained in the video file in the head of the file. The meta data by default is contained in the tail of the file. The tail meta data messes up flowplayer or the flash player you are using because video must be downloaded before the video player can obtain the meta data about the video. Google search on the term “moov atom” to learn more.

If you are behind a router, ifconfig won’t give you your external IP address. As a possible solution you can (w)get it from http://checkip.dyndns.org/ and grep your external IP from the html file:

 wget http://checkip.dyndns.org/ -qO - | grep -Eo '\<[[:digit:]]{1,3}(\.[[:digit:]]{1,3}){3}\>'

Ende Juli veröffentlichte ich das Skript “findgrep”, das mit Hilfe von “find” und “grep” Verzeichnisse rekursiv nach Textmustern durchsucht.

Lizenz:Creative Commons Attribution ShareAlike 3.0; (click pic for details)

Der dazu erschienene Artikel (samt Skript für die bash) hat sich inzwischen zu einem der meistaufgerufenen Artikel  dieses Blogs gemausert. Auch bei den Suchanfragen, über die dieses Blog gefunden wird, stehen Ausdrücke wie “find und grep rekursiv” oft ganz oben auf der Hitliste. Bevor ich das Skript zusammengeschustert hatte, war ich auch eine gute Weile mit Recherche beschäftigt, ohne auf die Schnelle eine passende Lösung zu finden.

Deshalb habe ich das Skript in den letzten Wochen auf meinem Klapprechner  überarbeitet. Das Resultat, “findgrep Version 0.2″ kann weiter unten heruntergeladen werden.

findgrep v 0.2: Verzeichnisse rekursiv nach Text durchsuchen

Verbesserungen:

• Die gefilterten Ergebnisse werden ja in einer Textdatei (Variable $SAVEFILE) gespeichert. War diese Datei jedoch unterhalb des Suchverzeichnisses angesiedelt, suchte sich findgrep einen Wolf, da es die bereits gefundenen Ergebnisse erneut als Treffer anzeigte. Dieser Bug wurde behoben.
• Bisher musste der Nutzer nach jedem Start des Programms den Suchpfad angeben. Zusätzlich hat der Nutzer nun die Möglichkeit, durch “ENTER” das Standardverzeichnis zu bestätigen (Standard: /var/log)  Zum Ändern einfach die Variable $SEARCHDIR im Skript anpassen.
• Sowohl bei der Auswahl als auch bei der Präsentation der Ergebnisse wurde das Programm grafisch anschaulicher aufpoliert
• findgrep zeigt nun die Anzahl der gefundenen Matches an

findgrep: Suche nach "warning" in /var/log beendet ...

Download findgrep v 0.2: Bitte hier klicken und mit copy und paste holen. Rechtsklick und “Speichern unter …” funktioniert nicht.

Das Skript ist für Ubuntu getestet. Manche Versionen von grep kennen die Option “color” nicht und brechen das Skript mit Fehlermeldung ab. Äh, ach ja: Der nette junge Herr oben bin nicht ich. Ich fand das Bild nur passend zum Thema und habe es auf Wikimedia gefunden.

Alle findgrep Artikel

Changing MAC address of a machine is called spoofing a MAC address or faking a MAC address. In linux, you can change MAC address of your machine. This is how it is done.

First find the physical MAC address of your machine by running the following command :



$ ifconfig -a | grep HWaddr
eth0  Link encap:Ethernet HWaddr 00:1f:f3:cc:c2:f9

The hexadecimal numbers in blue denote my machine’s MAC address. Yours will be different.

Next, login as root in Linux and enter the following commands -

# ifconfig eth0 down
# ifconfig eth0 hw ether 00:11:22:33:44:55
# ifconfig eth0 up
# ifconfig eth0 | grep HWaddr

Note above that I have changed the MAC address to a different number highlighted in blue. 00:11:22:33:44:55 is the new MAC address I have provided for my Linux machine. You can choose any 48 bits hexadecimal address as your MAC address.

We’ve had a marketing driven name change that impacts a lot of internal error strings in our code in a silly way, and I’ve got a lot of stuff like the following output

# grep -n '".* BA' *C
foo.C:1486:                       "Unexpected BA error - panic",
foo.C:1561:                       "Unexpected BA error - panic",
foo.C:1569:                       "Unexpected BA error",
...

All these ‘BA’s have to be changed to ‘BF’.

Ideally no customers would ever see these developer centric messages, but they will be in our logfiles, and potentially visible and confusing.

It’s not too hard to replace these, but there’s a lot of them. I’ve had this kind of task before, and have done it using hacky throw away command line “one-liners” like the following:

for i in `cut -f1,2 -d: grepoutput` ; do
   f=`echo $i | cut -f1 -d:`
   l=`echo $i | cut -f2 -d:`
   vim +$l +:'s/\/BF/g' +wq $i
done

Okay, it’s not a one liner as above since I’ve formatted this with newlines instead of semicolons, but when tossing off throwaway bash/ksh for loop stuff like this I usually do it as a one liner. This bash loop is easy enough to write, but messy and also fairly easy to get wrong. I’m tired of doing this over and over again.

It seemed to me that it was time to code up something that I can tweak for automated tasks like this, and wrote the perl script below that consumes grep -n ouput (ie. file:lineno:stuff output), and makes the desired replacements, whatever they are. I’ve based this strictly on the grep output because the unrestricted replacements could be dangerous and I wanted to visually verify that all the replacement sites were appropriate.

#!/usr/bin/perl

my %lines ;

while (<>)
{
   chomp ;

   /^(.*?):(.*?):/ or die ;

   $lines{$1} .= ",$2" ;
}

foreach (keys %lines)
{
   process_file( $_, split(/,/, $lines{$_} ) ) ;
}

exit ;

sub process_file
{
   my ($filename, @n) = @_ ;

   my %thisLines = map { $_ => 1 } @n ;

   open my $fhIn, "$filename.2" or die ;

   my $lineno = 0 ;
   while ( <$fhIn> )
   {
      $lineno++ ;

      if ( exists($thisLines{$lineno}) )
      {
         # word delimiters to replace BA but not BASH nor ABAB, ...
         s/\bBA\b/BF/g ;
      }

      print $fhOut $_ ;
   }

   close $fhIn ;
   close $fhOut ;
}

This little script, while certainly longer than the one-liner method, is fairly straightforward and easy to modify for other similar ad-hoc replacement tasks later. However, I have to wonder if there’s an easier way?

Print columns 1, 5 and then 2, put a colon, and then print column 10 of a line that matches pattern from the file columns.txt:

.
Protecting the special meaning of the single quotes and curly braces
I came across this situation when I was trying to watch the above command as well as two other commands (say, command1 and command2) at the same time. The correct way to do that is by properly protecting the meaning of the special characters in the above awk invocation:

.
N.B. The “-d”flag highlights the changes, whereas the “-n 20” flag causes the above to watch every 20 seconds.

Just for the heck of it, let me put the full command that I was actually using:

(suggestions for making the above shorter, other than by aliasing, are most welcome!).

Reference: here.

I always get confused about various flavors of grep. Here’s a summary to shine some light: (man grep to know more!)

• egrep or grep -E (in linux only) is extended grep where additional regular expression metacharacters have been added like +, ?, | and ()
• fgrep or grep -F (in linux only) is fixed or fast grep and behaves as grep but does not recognize any regular expression metacharacters as being special.