This morning we have a syndicate bonanza with a very solid IPO pricing and many secondary offerings getting priced.  Fortinet, Inc. (NASDAQ: FTNT) priced 12.5 million shares in IPO at $12.50, above the $9.00 to $11.00 range.   There are also many secondary offerings:  American Water Works (NYSE: AWK) 37.35 million shares priced at $21.63; Boise Inc. (NYSE: BZ) 17 million shares priced at $4.85; Longtop Financial Technologies Limited (NYSE: LFT) 3.7 million shares priced at $31.25; Smith International Inc. (NYSE: SII) priced 28 million shares priced at $26.50.

We have detailed information on the size, use of funds, underwriters, and performance on each below.

At Fortinet (NASDAQ: FTNT) and the priced 12.5 million shares in IPO at $12.50, the deal breakdown is as follows for the $156 million offering: 5,781,683 shares are being sold by the company and 6,718,317 shares are being offering by selling stockholders. Morgan Stanley, J.P. Morgan, and Deutsche Bank are joint book-runners; Robert W. Baird, RBC Capital Markets, ThinkEquity, and JMP Securities, and Signal Hill Capital Group LLC are co-managers. In addition, Fortinet has granted the underwriters a 30-day option to purchase up to an additional 1,875,000 shares of common stock to cover over-allotments.  Fortinet is in the field of network security and unified threat management solutions, and it priced above the $9 to $11 indicated range.

American Water Works Company, Inc. (NYSE: AWK), the largest water utility in the U.S., priced a 37.35 million shares at $21.63 per share after a closing bell price of $21.63 yesterday for a sale of close to $800 million.  This stake is entirely from RWE AG, the former parent and majority/largest holder, so none of the $784 million in gross proceeds go to American Water itself.  BofA Merrill Lynch, Credit Suisse, and Morgan Stanley are joint book-runners.  Shares are up 0.3% at $21.71 after a $21.63 close yesterday.

Boise Inc. (NYSE: BZ) 17 million shares priced at $4.85, and all shares here are also being sold by shareholders and insiders in the $82+ million raise.  Goldman Sachs was the sole book-runner.  Boise Cascade Holdings, L.L.C. (and trusts affiliated with Jason Weiss) will now ‘only’ hold about 25% of Boise’s common stock after the offering.  Shares are down 3.3% at $4.92 after a $5.09 close yesterday.

Longtop Financial Technologies Limited (NYSE: LFT) is actually trading way up this morning after it priced 3.7 million shares (ADS’s) at $31.25 per share, for more than $115 million raised.  Longtop is into software development and solutions which target the financial services industry in China.  The use of the net proceeds of the offering are earmarked for potential acquisitions and for general corporate purposes.  Deutsche Bank and Morgan Stanley were the joint book-runners.

Smith International Inc. (NYSE: SII) priced 28 million shares priced at $26.50 in a $730 million approximate raise.  Oddly enough, Citigroup just raised the rating on Smith International.  The net proceeds from the offering will be used for debt repayment, general corporate purposes, and funding of potential acquisitions or investments.  J.P. Morgan Securities Inc. is the book-running manager for the deal.  Shares are actually up 4% at $27.97.

We are seeing a whole new slate of S-3 secondary filings and still seeing plenty of S-1 IPO filings at the SEC each day.  If the market holds up, there will be many more days where the stock market has to absorb the $1.86 billion absorbed by the market this morning in new shares on the market.

You can join our open email distribution list to hear more news on IPOs, secondary offerings, key analyst calls, top day trader alerts, mergers and acquisitions, Buffett and other investment gurus, private equity, and more.

JON C. OGG
November 18, 2009

In various sciences one is often invested in processing voluminous data which follows a given format, with many examples following (hopefully) the same format. A typical data file might have a line of environment variables, followed by raw data, which repeats in columns and/or rows eg.,

4,23,45,56,78
12,23,45,56,67,78,89,87,…,72
32,34,23,21,23,56,43,23,…,34
34,54,32,45,89,76,54,98,…,58
67,67,88,32,34,21,22,97,…,51

A typical section of C code which would use this might look like:

float config[8];
scanf(“%f,%f,%f,%f,%f”,&config[1],&config[2],&config[3],
&config[4],&config[5]);

printf(“%f,%f,%f,%f,%f\n”,config[1],config[2],config[3],
config[4],config[5]);

int RECORDS = (int) config[1];
int i;
if ( RECORDS < 100){
int SIZE = sizeof(float)*RECORDS;

float *deltaCSMean = malloc(SIZE);
float *deltaCSSigma = malloc(SIZE);
float *CQMean = malloc(SIZE);
float *CQSigma = malloc(SIZE);
float *etaMean = malloc(SIZE);
float *etaSigma = malloc(SIZE);
float *brdF2 = malloc(SIZE);
float *brdF1 = malloc(SIZE);
float *amplitude = malloc(SIZE);

for (i=0; i< RECORDS; i++){
scanf(“%f,%f,%f,%f,%f,%f,%f,%f,%f”,&deltaCSMean[i],
&deltaCSSigma[i],&CQMean[i],&CQSigma[i],
&etaMean[i],&etaSigma[i],
&brdF2[i],&brdF1[i],&amplitude[i]);
printf(“%f,%f,%f,%f,%f,%f,%f,%f,%f\n”,deltaCSMean[i],
deltaCSSigma[i],CQMean[i],CQSigma[i],
etaMean[i],etaSigma[i],brdF2[i],brdF1[i],amplitude[i]);
}
}

Using redirect a simple invocation of executable foo using input foo1.txt and output foo2.txt would of course be:

./foo < foo1.txt > foo2.txt

which is fine until one needs different data from the file. Supposing in this example data repeats along columns, then one can use awk and a pipe to reshape the data input, removing the need to edit and recompile the C source for different data subsets:

gawk ‘BEGIN{FS=”,”; print “4,12,23,44,56″; getline;} {for(i=lowBnd;i<upBnd;i++){printf(“%f,”,$i) }; printf(“%f\n”,$upBnd) }’ foo1.txt | ./foo > foo2.txt

where lowBnd and upBnd correspond to the data column limits in the input file. Note that the environment variables are written during the BEGIN block, allowing the specification of (for instance) a different job for the new data set.

The following script prints all lines of a text file that are followed by a line containing the regular expression xyz.

awk '{line[NR]=$0;if($0~/xyz/){print line[NR-1]}}' input.file

Enjoy.

If you use print in awk you’ll lose the formatting of the input file — most of the times that may be OK, but not always. The solution is simple — replace print by printf. Here’s an example with strings in the input fields:

.

“%…s” means a character string.

20 means a length of 20 characters.

awk defaults to right-alignment (presumably for columns of figures) so you need -20 for left-alignment.

Credit: here.

After issuing a command like apt-get (remove|purge), some configuration files remain whereas the packages to whom they belong have just been removed.
The following command will help you keeping up to date all the useful configuration files :

dpkg –purge `dpkg –get-selections|awk ‘/deinstall/ {print $1}’`

This is an update on our bash command line calculator posted a few days ago — except for the fact that this time we’ll use awk to do the calculation instead of bc. As I mentioned in that post, you may use python or ruby (irb) to do the same thing, but these tricks may be useful if you don’t ruby or python installed (bc and awk, in general, come by default in any Unix or GNU/Linux distro).

First, create (or rewrite if you used our last trick) function “?”  as follows and put it in your ~/.bashrc file:

and make sure to reload your ~/.bashrc file (do the similar thing if you're using any other shell). [NOTE: ZSH does not like ``?'' as a function, so you might consider replacing it with something reasonable, e.g., ``compute'']


Now, if you want to calculate an expression, do it, for example, as

and don’t forget the quotes.

.

The advantage of this over bc is that you can use more arithmetic and trigonometric functions (link):

              atan2(y,x)     Arctan of y/x between -pi and pi.

	      cos(x)	     Cosine function, x in radians.

	      exp(x)	     Exponential function.

	      int(x)	     Returns x truncated towards zero.

	      log(x)	     Natural logarithm.

	      rand()	     Returns a random number between zero and one.

	      sin(x)	     Sine function, x in radians.

	      sqrt(x)	     Returns square root of x.

.
You may include variables as well in the function definition itself:

where we have defined the variable pi and degree (such that tan(pi/4.0) = 1.0 and pi radians is equivalent to 180 degrees) to be used later, e.g.

and I’m sure that you’ll get -1 and 0 (within the machine precision), respectively, as the answer!

(You may find some more interesting calculator related tricks posted in this blog scattered in different pages)

Credit: here and here (via LifeHacker).

UPDATE: I just realized that I can use the calc package (besides bc, awk, python and irb) to do the command line math wizardry more efficiently (and let’s take that as the end to this command line calculator series!). It has a larger set of built-in functions. You may grab the source code from the maintainer’s website and follow my instructions to install it on your system [although binaries are also readily available, e.g. apcalc package for Ubuntu 9.10 (Karmic Koala)].

You know what you have to do… you have to collect all the strings for every field, keep the max length for each column, then at the end spit them back out again.

But… you need to store each field for each row and awk only supports 1 dimensional arrays… suck it up and say the script only supports up to a max of 255 fields:

BEGIN {
	row = 0
	max_fields = 0;
	super_max = 255;
} {
	for ( i = 1 ; i <= NF ; i++ ) {
		idx = row + super_max * i;
		len = length( $i );
		data[ idx ] = $i;
		if ( len > max[ i ] ) {
			max[ i ] = len;
		}
	}
	if ( NF > max_fields ) {
		max_fields= NF;
	}
	row++;
} END {
	for ( r = 0 ; r < row ; r++ ) {
		for ( i = 1 ; i <= max_fields ; i++ ) {
			idx = r + super_max * i;
			fmt = "%-" max[ i ]  "s ";
			printf( fmt , data[ idx ], max[ i ]  );
		}
		printf( "\n" );
	}
}

columnbo.awk

Here is some prettified output:

this.logger            = logger;
this.dataSourceFactory = dataSourceFactory;
this.wsdlUrl           = wsdlUrl;
this.wsdlKey           = wsdlKey;
this.documentDAO       = documentDAO;

I enjoy tinkering with both Java and C# and publish a lot of my source on this blog under a very liberal open-source license.

I have wondered, however, what I would do if I had to protect a commercial program written in either of the two. The obvious choice would be to purchase a professional obfuscation program that defeats most decompilers. Fair enough. Most of these work on the compiled intermediate code.

What, however, if I just wanted to put some mild protection into the compiled product? The mild protection I refer to would still allow the code to be decompiled back to source form, but the person decompiling would have to work hard to make use of the bulk of the code.

My solution was to try a simple shrouding utility.

Shrouding ( also referred to as fogging ) is a kind of source-level obfuscation that was somewhat popular years ago for C programs sold in source form. The programs would generally compile but would be unreadable enough that they would discourage someone from using portions of unlicensed code in other programs.

A lot of these source-level shroud utilities are language-aware and determine what variables, functions, methods, and classes are locally-scoped. The shroud utility then replaces these with generated identifiers.

Most shrouding utilities go a step further and obfuscate literal strings as well as rewriting flow-control constructs.

All I had intended to do was to change the locally-scoped identifier-names to something that my script generates.

My proof-of-concept is an AWK script called foggy.awk. foggy.awk replaces any identifier it finds beginning with two underscore characters with a new identifier containing the prefix “i_” and a sequentially generated number.

This means that in my code, I have to observe a coding convention. I will have to prefix the name of any identifier that should be shrouded with two underscores.

First, here’s the shrouding script:

foggy.awk

# Foggy.awk
# A source-code shrouding utility.
# Replace any "identifier" beginning with __ with a sequentially-
# numbered new identifier
#
# License: MIT / X11
# Copyright (c) 2009 by James K. Lawless
#
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation
# files (the "Software"), to deal in the Software without
# restriction, including without limitation the rights to use,
# copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following
# conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.
#
BEGIN {
   legal="abcdefghijklmnopqrstuvwxyz";
   legal=legal "ABCDEFGHIJKLMNOPQRSTUVWXYZ";
   legal=legal "_0123456789" ;
   id_counter=0;
   id_list["__"]="__";
}
{
   state=0;
   s="";
   for(i=1;i<=length($0);i++) {
      c=substr($0,i,1);
      if(state==0) {
         if(c=="_") {
            state=1;
         }
         else {
            s=s c;
         }
      }
      else {
         if(state==1) {
            if(c=="_") {
               state=2;
               id="__";
            }
            else {
               s=s "_" c;
               state=0;
            }
         }
         else {
            # state == 2
            if(index(legal,c)<1) {
               if(id_list[id]=="") {
                  tmp_id="i_" id_counter;
                  id_counter++;
                  id_list[id]=tmp_id;
               }
               s=s id_list[id];
                  # decrement the index to forget the current
                  # character we're looking at
               i--;
               state=0;
            }
            else {
               id=id c;
            }
         }
      }
   }
   if(state==0) {
      printf("%s\n",s);
   }
   else {
      if(state==1) {
         printf("%s_\n",s);
      }
      else {
         if(state==2) {
            if(id_list[id]=="") {
               tmp_id="i_" id_counter;
               id_counter++;
               id_list[id]=tmp_id;
            }
            printf("%s%s\n",s,id_list[id]);
         }
      }
   }
}

As a side note, I have compiled the above script with the Thompson Automation AWK (TAWK) compiler for Windows into the exe foggy.exe available in the foggy.zip file referenced at the end of this post.

Unfortunately, TAWK is no longer commercially available. I reviewed this compiler years ago in Dr. Dobbs Journal. ( See Examining the TAWK Compiler, DDJ May ‘97 here: http://www.ddj.com/architect/184410193 …)

I also tested with GNU GAWK for Windows.

I would have preferred to use a regular expression with captured groups, but the AWK tools I have do not support those sorts of regex features. This likely would have been a smaller Perl script, but I wanted to try it in AWK. The necessary parser state transitions didn’t seem to be too deep, so I gave it a go.

To use the above script either type:

gawk -f foggy.awk < inputfle > outputfile

...or, if you're using foggy.exe

foggy.awk < inputfle > outputfile

Let's first take a look at a Java program that now contains shroud-ready identifiers. I took the source from MicroHttp1.java from my post here: http://jimlawless.wordpress.com/2009/08/23/tracing-xslt-with-a-tiny-java-web-server/

// MicroHttp1
// A small , specialized web server in Java
//
// License: MIT / X11
// Copyright (c) 2009 by James K. Lawless
// jimbo@radiks.net http://www.radiks.net/~jimbo
// http://www.mailsend-online.com
//
// Permission is hereby granted, free of charge, to any person
// obtaining a copy of this software and associated documentation
// files (the "Software"), to deal in the Software without
// restriction, including without limitation the rights to use,
// copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the
// Software is furnished to do so, subject to the following
// conditions:
//
// The above copyright notice and this permission notice shall be
// included in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
// OTHER DEALINGS IN THE SOFTWARE.

import java.io.*;
import java.net.*;
import java.util.Date;

public class MicroHttp1Obf {
   public static void main(String[] __args)
      throws IOException {
      ServerSocket __serv;
      Socket __s;
      String __str;
      PrintWriter __pw;
      BufferedReader __br;
      __serv=new ServerSocket(80);
      System.out.println("Micro HTTP Server v 0.1");
      System.out.println();
      for(;;){
         __s=__serv.accept();
         __br=new BufferedReader(
            new InputStreamReader(
               __s.getInputStream()));
         for(;;) {
            __str=__br.readLine();
            System.out.println(__str);
            if( ! __br.ready())
               break;
         }
         System.out.println();
         __pw=new PrintWriter(
            __s.getOutputStream(), true);

         __pw.print("HTTP 200 OK\r\n");
         __pw.print("Content-type: text/html\r\n\r\n");
         __pw.print("<html><head /><body>");
         __pw.print("Current date/time " + new Date());
         __pw.print("</body></html>");
         __pw.close();
      }
   }
}

Note that I had to change the class definition to what I would be targetting for my output filename, since the main Java class and source filename prefix must match.

There are lots of underscores now in this short script. Here's the shrouded output:

// MicroHttp1
// A small , specialized web server in Java
//
// License: MIT / X11
// Copyright (c) 2009 by James K. Lawless
// jimbo@radiks.net http://www.radiks.net/~jimbo
// http://www.mailsend-online.com
//
// Permission is hereby granted, free of charge, to any person
// obtaining a copy of this software and associated documentation
// files (the "Software"), to deal in the Software without
// restriction, including without limitation the rights to use,
// copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the
// Software is furnished to do so, subject to the following
// conditions:
//
// The above copyright notice and this permission notice shall be
// included in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
// OTHER DEALINGS IN THE SOFTWARE.

import java.io.*;
import java.net.*;
import java.util.Date;

public class MicroHttp1Obf {
   public static void main(String[] i_0)
      throws IOException {
      ServerSocket i_1;
      Socket i_2;
      String i_3;
      PrintWriter i_4;
      BufferedReader i_5;
      i_1=new ServerSocket(80);
      System.out.println("Micro HTTP Server v 0.1");
      System.out.println();
      for(;;){
         i_2=i_1.accept();
         i_5=new BufferedReader(
            new InputStreamReader(
               i_2.getInputStream()));
         for(;;) {
            i_3=i_5.readLine();
            System.out.println(i_3);
            if( ! i_5.ready())
               break;
         }
         System.out.println();
         i_4=new PrintWriter(
            i_2.getOutputStream(), true);

         i_4.print("HTTP 200 OK\r\n");
         i_4.print("Content-type: text/html\r\n\r\n");
         i_4.print("<html><head /><body>");
         i_4.print("Current date/time " + new Date());
         i_4.print("</body></html>");
         i_4.close();
      }
   }
}

There are really only about five shrouded identifiers in this program. I suppose someone who really wanted to work at it could perform global search-and-replace operations to change them into something meaningful as they analyze the code.

Let's try something a little larger that has more local identifiers. For this next test, I took the C# source code from my command-line Twitter client Twimmando. See: http://jimlawless.wordpress.com/2009/05/24/twimmando-a-command-line-twitter-client/

Here's the shroud-ready source:

// Twimmando - a command-line Twitter client

// License: MIT / X11
// Copyright (c) 2009 by James K. Lawless
// jimbo@radiks.net http://www.radiks.net/~jimbo
// http://www.mailsend-online.com
//
// Permission is hereby granted, free of charge, to any person
// obtaining a copy of this software and associated documentation
// files (the "Software"), to deal in the Software without
// restriction, including without limitation the rights to use,
// copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the
// Software is furnished to do so, subject to the following
// conditions:
//
// The above copyright notice and this permission notice shall be
// included in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
// OTHER DEALINGS IN THE SOFTWARE.

using System;
using System.Net;
using System.Text;
using System.IO;
using System.Runtime.Serialization.Formatters.Binary;
using System.Web;

namespace Twimmando
{

   public enum __RequestMethod
   {
      __Get,
      __Post,
      __Head
   }
   class __Config
   {
      public __RequestMethod __Method;
      public string __Uri;
      public string __UserName;
      public string __Password;
      public string __PostData;
   }

   class __Twimmando
   {
      public static string __Version = "Twimmando v1.01 (obfuscated)";
      public static void Main(string[] __args)
      {
         __Config __config;
         try
         {
            __config=__processCommandLine(__args);
            if(__config.__Uri==null)
               Environment.Exit(1);
            __doHttpRequest(__config);
         }
         catch(Exception e)
         {
            Console.Error.WriteLine(e.ToString());
            Environment.Exit(1);
         }
         Environment.Exit(0);
      }

      public static __Config __processCommandLine(string[] __args)
      {
         int __i;
         __Config __config=new __Config();
         __config.__PostData="";
         __config.__Method=__RequestMethod.__Get;

         for(__i=0;__i<__args.Length;__i++) {
            if(string.Compare(__args[__i],"-head",true)==0) {
               __config.__Method=__RequestMethod.__Head;
            }
            else
            if(string.Compare(__args[__i],"-post",true)==0) {
               __config.__Method=__RequestMethod.__Post;
            }
            else
            if(string.Compare(__args[__i],"-uri",true)==0) {
               __config.__Uri=__args[++__i];
            }
            else
            if(string.Compare(__args[__i],"-u",true)==0) {
               __config.__UserName=__args[++__i];
            }
            else
            if(string.Compare(__args[__i],"-p",true)==0) {
               __config.__Password=__args[++__i];
            }
            else
            if(string.Compare(__args[__i],"-f",true)==0) {
               if(__config.__PostData.Length>0) {
                  __config.__PostData+="&";
               }
               __config.__PostData+=__args[__i+1]+"="+HttpUtility.UrlEncode(__args[__i+2]);
               __i+=2;
            }
         }
         if(__config.__Uri==null) {
            Console.WriteLine(__Twimmando.__Version + "\nby Jim Lawless (@lawlessGuy)");
            Console.WriteLine("Usage:\n\ttwimmando [options]\nWhere options are:");
            Console.WriteLine("   -uri resourceURI   ; such as /statuses/public_timeline.xml");
            Console.WriteLine("   -u username        ; Twitter password");
            Console.WriteLine("   -p password        ; Twitter user name");
            Console.WriteLine("   -head              ; send HTTP HEAD request, default is GET");
            Console.WriteLine("   -post              ; send HTTP POST request, default is GET");
            Console.WriteLine("   -f name value      ; add POST data name/value pair to request");
         }
         return __config;
      }

      public static void __doHttpRequest(__Config __config)
      {

         HttpWebRequest __webRequest;
			HttpWebResponse __webResponse;
         Uri __uri=new Uri("http://twitter.com" + __config.__Uri);
			__webRequest = (HttpWebRequest)HttpWebRequest.Create(__uri);
         if(__config.__Password!=null) {
            __webRequest.Credentials = new NetworkCredential(__config.__UserName, __config.__Password);
         }

	 __webRequest.UserAgent = __Twimmando.__Version;
         if(__config.__Method==__RequestMethod.__Post) {
            __webRequest.Method = "POST";
               // remove Expect header
            __webRequest.ServicePoint.Expect100Continue = false;
            __webRequest.ContentType="application/x-www-form-urlencoded";
            __webRequest.ContentLength = __config.__PostData.Length;
            StreamWriter __sOut=new StreamWriter(__webRequest.GetRequestStream(),System.Text.Encoding.ASCII);
            __sOut.Write(__config.__PostData);
            __sOut.Close();
         }
         else
         if(__config.__Method==__RequestMethod.__Head) {
            __webRequest.Method="HEAD";
         }
	 __webResponse = (HttpWebResponse)__webRequest.GetResponse();
         if(__config.__Method==__RequestMethod.__Head) {
            Console.WriteLine(__webResponse.Headers.ToString());
         }
         else {
            Stream __stream = __webResponse.GetResponseStream();
            StreamReader __streamReader =
               new StreamReader(__stream, Encoding.ASCII);
            Console.WriteLine(__streamReader.ReadToEnd());
         }
         __webResponse.Close();
      }
   }
}

Here's the encoded result:

// Twimmando - a command-line Twitter client

// License: MIT / X11
// Copyright (c) 2009 by James K. Lawless
// jimbo@radiks.net http://www.radiks.net/~jimbo
// http://www.mailsend-online.com
//
// Permission is hereby granted, free of charge, to any person
// obtaining a copy of this software and associated documentation
// files (the "Software"), to deal in the Software without
// restriction, including without limitation the rights to use,
// copy, modify, merge, publish, distribute, sublicense, and/or sell
// copies of the Software, and to permit persons to whom the
// Software is furnished to do so, subject to the following
// conditions:
//
// The above copyright notice and this permission notice shall be
// included in all copies or substantial portions of the Software.
//
// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
// EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
// OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
// NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
// HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
// WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
// FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
// OTHER DEALINGS IN THE SOFTWARE.

using System;
using System.Net;
using System.Text;
using System.IO;
using System.Runtime.Serialization.Formatters.Binary;
using System.Web;

namespace Twimmando
{

   public enum i_0
   {
      i_1,
      i_2,
      i_3
   }
   class i_4
   {
      public i_0 i_5;
      public string i_6;
      public string i_7;
      public string i_8;
      public string i_9;
   }

   class i_10
   {
      public static string i_11 = "Twimmando v1.01 (obfuscated)";
      public static void Main(string[] i_12)
      {
         i_4 i_13;
         try
         {
            i_13=i_14(i_12);
            if(i_13.i_6==null)
               Environment.Exit(1);
            i_15(i_13);
         }
         catch(Exception e)
         {
            Console.Error.WriteLine(e.ToString());
            Environment.Exit(1);
         }
         Environment.Exit(0);
      }

      public static i_4 i_14(string[] i_12)
      {
         int i_16;
         i_4 i_13=new i_4();
         i_13.i_9="";
         i_13.i_5=i_0.i_1;

         for(i_16=0;i_16<i_12.Length;i_16++) {
            if(string.Compare(i_12[i_16],"-head",true)==0) {
               i_13.i_5=i_0.i_3;
            }
            else
            if(string.Compare(i_12[i_16],"-post",true)==0) {
               i_13.i_5=i_0.i_2;
            }
            else
            if(string.Compare(i_12[i_16],"-uri",true)==0) {
               i_13.i_6=i_12[++i_16];
            }
            else
            if(string.Compare(i_12[i_16],"-u",true)==0) {
               i_13.i_7=i_12[++i_16];
            }
            else
            if(string.Compare(i_12[i_16],"-p",true)==0) {
               i_13.i_8=i_12[++i_16];
            }
            else
            if(string.Compare(i_12[i_16],"-f",true)==0) {
               if(i_13.i_9.Length>0) {
                  i_13.i_9+="&";
               }
               i_13.i_9+=i_12[i_16+1]+"="+HttpUtility.UrlEncode(i_12[i_16+2]);
               i_16+=2;
            }
         }
         if(i_13.i_6==null) {
            Console.WriteLine(i_10.i_11 + "\nby Jim Lawless (@lawlessGuy)");
            Console.WriteLine("Usage:\n\ttwimmando [options]\nWhere options are:");
            Console.WriteLine("   -uri resourceURI   ; such as /statuses/public_timeline.xml");
            Console.WriteLine("   -u username        ; Twitter password");
            Console.WriteLine("   -p password        ; Twitter user name");
            Console.WriteLine("   -head              ; send HTTP HEAD request, default is GET");
            Console.WriteLine("   -post              ; send HTTP POST request, default is GET");
            Console.WriteLine("   -f name value      ; add POST data name/value pair to request");
         }
         return i_13;
      }

      public static void i_15(i_4 i_13)
      {

         HttpWebRequest i_17;
			HttpWebResponse i_18;
         Uri i_19=new Uri("http://twitter.com" + i_13.i_6);
			i_17 = (HttpWebRequest)HttpWebRequest.Create(i_19);
         if(i_13.i_8!=null) {
            i_17.Credentials = new NetworkCredential(i_13.i_7, i_13.i_8);
         }

	 i_17.UserAgent = i_10.i_11;
         if(i_13.i_5==i_0.i_2) {
            i_17.Method = "POST";
               // remove Expect header
            i_17.ServicePoint.Expect100Continue = false;
            i_17.ContentType="application/x-www-form-urlencoded";
            i_17.ContentLength = i_13.i_9.Length;
            StreamWriter i_20=new StreamWriter(i_17.GetRequestStream(),System.Text.Encoding.ASCII);
            i_20.Write(i_13.i_9);
            i_20.Close();
         }
         else
         if(i_13.i_5==i_0.i_3) {
            i_17.Method="HEAD";
         }
	 i_18 = (HttpWebResponse)i_17.GetResponse();
         if(i_13.i_5==i_0.i_3) {
            Console.WriteLine(i_18.Headers.ToString());
         }
         else {
            Stream i_21 = i_18.GetResponseStream();
            StreamReader i_22 =
               new StreamReader(i_21, Encoding.ASCII);
            Console.WriteLine(i_22.ReadToEnd());
         }
         i_18.Close();
      }
   }
}

This shrouded program was much more difficult for me to read, as I used some local class definitions and used more local variables and methods than I had used in the sample Java program.

My conclusion?

Well, if I had to employ this method, I think I could grow into learning to deal with the added noise of the extra underscores in the source.

However, I still think that language-aware shrouding utilities would be much less painful to use.

The source and Windows executable file for foggy.awk can be downloaded in a single archive at:
http://www.mailsend-online.com/wp/foggy.zip

Save to del.icio.us
Digg it
Save to Reddit
Share on Facebook
Share on Twitter
More bookmarks

Unless otherwise noted, all code and text entries are Copyright © 2009 by James K. Lawless

The problem: I have  two files, file-1 and file-2, each of which has two columns; and I want to add the second column of the second file as the third column in the first file.
Let’s say the following is the content of file-1:

A   1
B   2
C   3

and that of file-2:

D   4
E   5
F   6

and I want to have something like this in file-3

A   1   4
B   2   5
C   3   6

Of course, the actual data are not as simple as the above!

Solution:

$ awk '{str1=$1; str2=$2; getline < "file-2"; print str1" \t "str2" \t "$2 > "file-3"}' file-1

I inserted the tab characters (\t) just to make file-3 look nice (scientists don’t care about white spaces, do they?)!

Reference: here.

awk ‘BEGIN{count=0;sum=0;min=0;max=0} {n=$1 ; if (0==count||nmax) max=n; count++; sum+=n;} END {printf( “min: %d\nmax: %d\nsum: %d\ncnt: %d\navg: %f\n”, min,max, sum,count, sum/count );}’

Here we are going into yet another earnings season.  We saw Monday how the market has rallied significantly from the March lows and the major indexes are even up in positive territory for the 2009 calendar.  The DJIA is up 51% from its absolute lows of March, and the S&P 500 has rallied more than 61% from its absolute lows in March.  If you look at the December 31, 2008 closing bell levels, the DJIA is now up about 12.75% and the S&P 500 is now up more than 19% year-to-date.

But almost as always, there are still some key very large and/or very active stocks which have not recovered anywhere close to the same amounts with the overall stock markets.  Some of these lagging stocks are Abbott Laboratories (NYSE: ABT), Altria Group Inc. (NYSE: MO), American Water Works Company, Inc. (NYSE: AWK), Burger King Holdings Inc. (NYSE: BKC), Energy Conversion Devices, Inc. (NASDAQ: ENER), Genzyme Corp. (NASDAQ: GENZ), Kroger Co. (NYSE: KR), Orbital Sciences Corp. (NYSE: ORB) and Wal-Mart Stores Inc. (NYSE: WMT).  Two similar situation stocks that are Leap Wireless International Inc. (NASDAQ: LEAP) and MetroPCS Communications Inc. (NYSE: PCS).  We wanted to explore the forward values and relative performance, and the consensus estimates based upon Thomson Reuters data.  Only two of these stocks have market capitalization rates under $1 billion, and almost all are very actively traded and well known in their sectors.

Abbott Laboratories (NYSE: ABT) is on deck to report earnings this week and it was surprising how far it has lagged to some of the large peers.  The good news is that it has started to participate in the rally with a 10% gain over the last month.  Saying it has ‘only’ recovered by 21% from its April-May lows shows how well the market sentiment is and how well other stocks have performed.  But shares are only about 10% higher from the early March inflection point lows.  Abbott trades at about 12.2-times normalized 2010 earnings estimates of $4.12 and the stock carries a 3.2% dividend yield.  Guidance was spotty before and obviously the sector has political risks into 2010 and beyond because it is tied to healthcare.

Altria Group Inc. (NYSE: MO) might not be a shock that it has lagged if you believe that smoking is a dead industry in the future.  But the one recognition that has always been the case is that Big Tobacco has always survived.  Things are even good enough that it recently juiced up its dividend and appears to have enough earnings power for plenty of dividend coverage for the future.  It is hard to bash a stock that has risen 28% from the March lows, but this greatly lags the performance of Philip Morris International, Inc. (NYSE: PM), Reynolds American Inc. (NYSE: RAI) and Lorillard, Inc. (NYSE: LO).  The Big-Mo trades at just under 10-times a normalized estimate of $1.85 EPS for 2010 and its dividend yield is almost 7.5% now that it has been hiked.  This only needs to rally about 14% to hit 52-week highs, so this is just representative more of nearly-dead money rather than anything that looks or feels sinister.

American Water Works Company, Inc. (NYSE: AWK) is in that long-discussed great sector of water, but as a utility.  It may just be more shares sold and the risks of more shares being sold, but this one has effectively been dead money ever since it came public (again).  At almost $20.00, shares are up about 23% from the absolute lows. That looks to be well under all of its larger peers here in the U.S.  This stock and sector does have a premium considering the utility status, but analysts are looking for growth from 2009 to 2010.  This one trades at less than 14-times normalized earnings estimates of $1.46 EPS for 2010.  America’s largest water utility did hike its dividend for August and it now yields about 4.2%.

Burger King Holdings Inc. (NYSE: BKC) is supposed to be in a class of eating-out that wins in recessions via fast food and very casual dining.  It seems like that crazy looking king character in commercials that scared so many kids has also scared investors.  This company has only recovered some 13% from its recent 52-week lows despite only trading at about 12.2-times its Fiscal-2010 (June) earnings estimates.  Its market cap is just under $2.4 billion.  The much larger, and far better-run, McDonald’s Corp. (NYSE: MCD) has not exactly been a performing beast this year, but its shares are up 25% from lows.  The Big Whopper would also have to rally almost 50% before hitting 52-week highs.  Unfortunately, its dividend yield is only about 1.4%.  For this to start performing again, management is going to have to do a better turnaround plan than what it has managed so far.

Energy Conversion Devices, Inc. (NASDAQ: ENER) is one where the figures are actually a tad better than what is reflected because of the solid gains seen on Monday.  Before Monday, this one was not even 10% above its 52-week lows and shares are still under the inflection point lows in the market during March.  Despite this being one of the old top solar stocks for traders, it has failed miserably to bounce when you consider the bounce of the major solar players.  First Solar, Inc. (NASDAQ: FSLR) is up over 75% and Suntech Power Holdings Co. Ltd. (NYSE: STP) is up almost 200% from its absolute lows over the last year.  Unfortunately this is expected to post a loss for all of its Fiscal 2010 (June) and estimates are $0.28 for its following Fiscal 2011 (June).  There is no dividend and this is likely to trade around how well oil prices do and on its own guidance and not its trading based on how the market does.  This is down from a 52-week high north of $48 and down from a peak of $80 back in the Summer-2008 highs.  No dividends are to be found here.

Genzyme Corp. (NASDAQ: GENZ) did recover from lows in August.  This troubled biotech giant is up about 20% from the absolute lows in August and ‘only’ up about 16% from its lowest close of $48.19 in August.  The reason here is simple.  Serious manufacturing issues resulted in contamination issues, and the company is effectively under a red flag warning by the FDA.  But this also trades at a much deeper discount than most of the large biotech stocks.  It would take close to a 50% rally for this to challenge its old 2008 highs.  The question that exists here is not over whether it still has an addressable market, but whether it can get its house entirely back in order before competition comes on in 2010.  Like most biotech companies, it has no dividend.  Earnings estimates are now all over the place, but it trades at roughly 14-times normalized 2010 earnings.  Even if you take a median estimate over an average estimate for 2010, that forward multiple is only around 15-times 2010 earnings.  Of course there are still risks here, but that is why the stock is already down so much on its woes.

Kroger Co. (NYSE: KR) was also up on Monday as an analyst upgraded the stock on valuation.  But the stock is still a laggard compared to rival Safeway (NYSE: SWY).  The company’s latest earnings report kept it from rallying because it was still experiencing difficult pricing and that has kept its shares from rallying.  We have barely seen a 15% gain since March’s lows.  Its 2010 ends in January 2011, but it trades at 10.4-times the $2.14 estimate.  This is representative of a mature grocery sector and the discount to Safeway is small but still there:  Safeway is up 24% from its lows and trades at 11-times the comparable forward earnings.  Analysts are still looking for growth and this would have to rally over 30% before its 52-week highs came back into play.

Orbital Sciences Corp. (NYSE: ORB) may seem the riskiest of all these underperforming stocks as its main operations are in small space and rocket systems, as well as GEO satellites for communications and broadcasting and LEO spacecraft that perform remote sensing and scientific research.  But effectively, the company is a pure-play on space operation and exploration, including deep space.  This stock is still down about one-third from a year ago and is ‘only’ up 22% from its lows in March.  Earnings estimates have come down over the last quarter but there has not been a significant downgrade worth noting from analysts of late.  With the government purse being tight and with the Obama administration reportedly considering privatizing much space exploration, a wild card exists here.  Loral Space & Communications (NASDAQ: LORL) is considered its top comparable stock, and that one has risen exponentially from its lows.

Wal-Mart Stores Inc. (NYSE: WMT) is supposed to be the winner in the recession, right?  Yet don’t go looking for it to be the golden child of stock performance.  And its 2.2% dividend yield might not sound like much.  Wal-Mart is under $50.00, and is only about 8% above its 52-week lows.  It would also take ‘only’ a 20% gain for Wal-Mart to challenge its 52-week highs.  And year-to-date the stock is down 10%.  It seems that its policy of dropping guidance and its same store sales reporting has not been as rewarding as it would hope.  Now the valuation question for the world’s largest retailer.  This trades at about 13.7-times a normalized 2010 earnings (actually Jan. 2011) estimate of $3.92.  Wal-Mart is still growing.  It would also seem as though that as long as there is an army of unemployed and as long as the notion of thrift is semi-secular, then this is lagging and cheap. Target Corp. (NYSE: TGT) trades at about 14.5-times 2010 comparable estimates, but Target has also almost doubled from its absolute lows.  Costco Wholesale Corporation (NASDAQ: COST) has rallied 50% from its lows, despite a much higher forward multiple.

Leap Wireless International Inc. (NASDAQ: LEAP) and MetroPCS Communications Inc. (NYSE: PCS) are two very similar companies that have been gross disappointments.  Despite the overall wireless sector being a tough space and despite fairly low performance even from the leaders, these two have been an outright disappointment by every single measurement besides being great performers for short sellers.  As there once was a merger on the table (which was blocked) and as these are both in the no-contract or pre-paid wireless sector, we are only going to mention the performance metrics.

• Leap Wireless is down close to 40% year to date.  The stock is up about 15% from its 52-week low.
• MetroPCS is down over 45% year to date.  The stock is also not even 10% off of its 52-week lows.

Leap is now expected to lose money in 2010, and if the $0.69 estimate is right on MetroPCS then it trades at about 12-times 2010 estimates.  Unfortunately, that is 20-times a 2009 estimate for MetroPCS.  We won’t go as far as to predict a merger here, but a proposed merger this time around could be out of need and not out of greed.

Again, the estimates are based on Thomson Reuters estimates and share performance is based from the weekend when a static snapshot existed equally among all.  Many of these estimates will change throughout this earnings season.  Despite the rally, there are still some very large or very active stocks which have just not participated.

JON C. OGG
OCTOBER 12, 2009

Ja, ich hab mir auch ‘nen VServer zugelegt. Warum?
Nun, es bietet die Möglichkeit so ganz nebenbei einen eigenen Teamspeak-Server laufen zu lassen, falls man mal mit jemand quatschen will, beim Entwickeln der nächsten OpenSource, beim Spielen von irgendwas, oder auch einfach mal so. Und wenn man mag, kann man auch noch seine Homepage oder so Zeugs drauf packen. Außerdem ist ein VServer im Grunde nicht teurer als ein weiterer Rechner, der zuhause nur rumsteht, Strom frisst, Lärm produziert und Platz wegnimmt.
Nun zeigt sich aber leider die Problematik, daß das gute Stück im Zusammenspiel mit PuTTY/Xming nicht so ganz gerne X11 Anwendungen auf den heimischen Windows-Rechner bringen mag.
Das grausamste Szenario bleibt, nach jedem Login auf dem Unix-Rechner im Internet, mit der Hand die $DISPLAY-Umgebungsvariable setzen zu müssen. Das nervt natürlich gewaltig, zumal Umgebungsvariablen, wie $SSH_CLIENT und $SSH_CONNECTION in der Regel bereits richtig gesetzt sind. Leider hat PuTTY jedoch Probleme die aktuell korrekte dynamische IP-Adresse für den eigenen DSL-Zugang richtig zu erkennen und entsprechend auf dem Zielsystem zu setzen.
Pfui, PuTTY!

Nun, es gibt dennoch einfache Lösungen für diese Problemstellung, die uns das manuelle Setzen der $DISPLAY Umgebungsvariable einfach machen. Hier eine recht simple Variante davon, vorausgesetzt, die $SSH_CLIENT Umgebungsvariable ist bereits korrekt gesetzt, was jedoch in der Regel der Fall ist, da diese das Zielsystem evaluiert wird, nicht durch PuTTY selbst.
Zunächst extrahieren wir aus $SSH_CLIENT unsere korrekte aktuelle IP-Adresse mit einem kleinen Programm, welches daraus eine “DISPLAY=client-ip:0.0″ Zeichenkette formt und diese auf “stdout” zurückliefert. Die Verwendung von putenv() oder setenv() zwecks direktem Setzen der Umgebungsvariable hätte an dieser Stelle nicht die gewünschte Wirkung.

Der zweite Teil der Lösung ist, unseren generierten $DISPLAY-String in die Umgebung zu schubsen. Das machen wir am besten im “.profile“, welches beim Login ausgeführt wird.
Für eine bash-Shell, wäre das z.B. eine Zeile wie, “export $(./evaldisp)“, wobei “evaldisp” unser kleines Programm wäre, welches die $DISPLAY-Zeichenkette liefert. Die Zeile “export $(./evaldisp)” bewirkt, daß zunächst unser kleines Programm “evaldisp” ausgeführt wird und die zurückgelieferte Zeichenkette “DISPLAY=…” als Variable für den “export” in die Umgebung verwendet wird.

Wenn wir nun nach dem Login in unseren gemieteten Unix-VServer ein X11 Programm aufrufen, müssen wir nicht mehr nachsehen, welche IP Adresse wir aktuell haben und diese mit der Hand setzen, sonder ein simples “xterm”, “emacs” oder “xeyes” reicht nun völlig aus, da die Umgebung bereits das $DISPLAY über unsere kleine Hilfestellung korrekt gesetzt hat.

Und nicht vergessen zuhause Xming laufen zu lassen… man möchte ja auch die X11-Anwendung auf dem Desktop sehen können

Hier ein C-Beispiel für unser “evaldisp” Programm, das wir mit “cc evaldisp.c -o evaldisp” compilieren können:

/* evaldisp.c */
#include "stdio.h"
#include "stddef.h"
#include "stdlib.h"
#include "string.h"
int main( int argc, char* argv[] )
{
char buf[512];
char* pc1 = getenv( "SSH_CLIENT" );
char* pc2 = strchr( pc1, ' ' ); /* get end of IP address */
*pc2 = '0'; /* set end of string */
memset( buf, 0, 512 );
sprintf( buf, "DISPLAY=%s:0.0", pc1 ); /* create $DISPLAY string */
*pc2=' '; /* set back to original value*/
printf( "%s\n", buf ); /* return $DISPLAY via stdout */
return EXIT_SUCCESS;
}


Alternativ hier noch eine Variante, bei mittels Python, anstelle von C, die IP-Adresse aus der Umgebungsvariable $SSH_CLIENT filtert:

#evaldisp.py
import os
s1 = os.getenv("SSH_CLIENT")
l1 = s1.split(' ')
print "DISPLAY=%(ip)s:0.0" % { 'ip':l1[0] }

Ensprechend lautet der Aufruf in “.profile” dann, “export $(python evaldisp.py)“.

~~~Nachtrag:

Und der Vollständigkeit halber, ich wurde darauf hingewiesen, man könne unter Verwendung von awk die Problematik auch in einem Einzeiler in .profile formulieren. Danke für den Hinweis und hier die entsprechende Zeile:

export DISPLAY=$(echo $SSH_CLIENT|awk ‘{print $1 ":0.0"}’)

Just picked this up from my favorite site: commandline-fu.

$ awk '{delta = $1 - avg; avg += delta / NR; mean2 += delta * ($1 - avg); } END { print sqrt(mean2 / NR); }'


Let’s test it by finding the standard deviation of 1, 2, 3, 4, and 5 whose standard deviation is sqrt(2): in BASH, pipe in the out put of the following (which just echoes out numbers from 1 through 5) to the above command to find that it is in fact sqrt(2) or 1.42421
$ for n in {1..5}; do echo $n; done


awk 'BEGIN { FS = "\t" } ; { print $1 }'  /path/to/file

gawk manual: 3.5.3 Setting FS from the Command Line

手慣れたスクリプト言語でワンライナーを書いてしまうべきか数秒まよったが、なんとなく awk を使うことに。

Print columns 1, 5 and then 2, put a colon, and then print column 10 of a line that matches pattern from the file columns.txt:

.
Protecting the special meaning of the single quotes and curly braces
I came across this situation when I was trying to watch the above command as well as two other commands (say, command1 and command2) at the same time. The correct way to do that is by properly protecting the meaning of the special characters in the above awk invocation:

.
N.B. The “-d”flag highlights the changes, whereas the “-n 20” flag causes the above to watch every 20 seconds.

Just for the heck of it, let me put the full command that I was actually using:

(suggestions for making the above shorter, other than by aliasing, are most welcome!).

Reference: here.

In many situations, this is a trivial task: create a simple regex script in Perl (or Python), and execute it. However, with linux routers it becomes a bit tricky. Usually by default these routers do not have high-level scripting languages installed due to limited resources. It is possible to add them by extending the diskspace through external means (like USB or SD card), but generally unless the user needs to use scripts extensively it isn’t necessary. There are several means to do substitution here.

Method #1: sed substitution.

Sed substitution is the easiest method to do line substitution, and the most recommended method. However, the cryptic syntax is not very user-friendly, especially those unfamiliar to sed (yet familiar to generic regex). To do a simple  substitution, use the following syntax:

# Syntax to do substitution in the file.
> sed -i 's/text1/text2/' file


This will take all instances of ‘text1′ and replaces it with ‘text2′ in the file. Note that the ‘-i’ means that it will change the file directly instead of printing the changes to stdout. This only does pure substitution, not line substitution. To do that, we need to use the ‘c\’ option instead of the ’s’ option at the beginning of the string. See below:

# Syntax to do line substitution in the file, trying to match text1 with text2.
> sed -i '/text1/ c\ text2' file


Note that the ’s’ in the beginning of the string is no longer there. There are actually several options in this same format. ‘i\’ will insert the line before the matched text1, and ‘a\’ will append the line after the matched text1.

In a practical example, openWRT tends to not setup the firewall (iptables) properly. When using the UI, it tends to comment out the option to include firewall.user. So I used this simple sed statement to uncomment the option whenever the firewall is changed:

# Uncomment out the etc/firewall.user
> sed -i '/option.*\/etc\/firewall.user/ c\ option path /etc/firewall.user' /etc/config/firewall


This will make:

/etc/config/firewall

...
# option path '/etc/firewall.user'
...

look like this:

/etc/config/firewall

...
 option path /etc/firewall.user
...

(A simplier way is to actually go to the source code and fix the issue, but this example is meant to show how this could be useful).

To understand more about how to use sed, see below references.

Method #2: awk substitution.

Another way to do substitution is to do it through awk. If sed seems cryptic, awk is even more so. Though the language is very powerful and there are many methods to accomplish this through awk, here is a simple example that does the line replacement:

# Replace line containing text1 with text2.
> awk '/text1/ {print "text2"; next} /.*/ print ;}' file > tmp.file
> mv tmp.file file


One disadvantage is that you can’t replace the file directly (since it seems to erase the file completely).

Some explanation over the awk syntax: AWK goes through a pattern-action routine. If /pattern/ matches, then {run action}. So if it matches /text1/, then it’ll {print “text2″}. The next means to skip to the next line, or otherwise it’ll continue to try to match against other patterns. The second pattern (/.*/) is there to match all the other lines, and print as normal.

So, using the above openWRT situation, an awk statement will look like this:

# Uncomment out the etc/firewall.user
> awk '/option.*\/\etc\/firewall.user/ {print "option path /etc/firewall.user"; next} /.*/ {print ;}' /etc/config/firewall > tmp
> mv -f tmp /etc/config/firewall


More information about awk can be found in the references below.

Method #3: Perl substitution.

For those who really need to use Perl or just so happen to have it installed, here is how to do line substitution in Perl:

# Replace line containing text1 with text2.
> perl -pi -e 's/^.*test1.*/test2/' file


Like in sed, the -i means to edit directly into the file. The -p allows to read in file as input, and -e executes the commands the quotes. To run the openWRT example above, it would look something like this:

# Uncomment out the etc/firewall.user
> perl -pi -e 's/.*option.*\/etc\/firewall.user.*/option path \/etc\/firewall.user/' /etc/config/firewall


Method #4: Python substitution.

This section will be added later once I have a better understanding of python. Unfortunately, the language is elusive to me, so there's room for improvement to understand how to work with this language.


References:

Sed: An Introduction and Tutorial
AWK: Find and Replace text
AWK: Manpage for awk(1)
Perl: Command-Line Options
Perl: Regular Expressions

Guy Co-Worker: Umm, Lauren, will you help me?
Me: Sure.
GCW: Will you grab these keys out of my pocket? (shifts back pocket toward me)

It is one thing that we had two fairly large IPOs this week.  But the flood of public secondary stock offerings is almost running at alarming levels when you consider that these share sales are after such a large run-up in the markets.  Melco Crown Entertainment Ltd. (NASDAQ: MPEL) $200 million and GSI Commerce Inc. (NASDAQ: GSIC) (and holders) sold about $201 million in secondary offerings this morning.  Other stocks in offerings are scheduled from American Water Works Co. (NYSE: AWK), Great Lakes Dredge & Dock Corp. (NASDAQ: GLDD), and U-Store-It Trust (NYSE: YSI).  Then there is the flood of regional and community banks selling stock.

Details are as follows:

American Water Works Co. (NYSE: AWK) is on deck with German-parent RWE selling another 30 million shares via Bank of America Merrill Lynch, Citigroup and Goldman Sachs.  With shares close to $19.00 today, this generates a $570 million estimates take.

Great Lakes Dredge & Dock Corp. (NASDAQ: GLDD) is on deck with some 12.5 million shares being sold through Barclays Capital and Bank of America Merrill Lynch.  This should come to roughly $75 million being raised based on today’s price.

U-Store-It Trust (NYSE: YSI) is set to sell 22 million shares through  Bank of America Merrill Lynch and Wells Fargo and this deal is expected to raise close to $110 million based on today’s price.

Then there are the regional and community banks that we have seen from Dandler O’Neill.  The Bancorp Bank (NASDAQ: TBBK) is on deck to sell roughly $50 million worth of shares through Sandler O’Neill.  Hampton Roads Bancshares Inc. (NASDAQ: HMPR) is expected to sell some 32 million shares through Sandler O’Neill and should raise roughly $140 million based on today’s price.  Seacoast Banking Corp. of Florida (NASDAQ: SBCF) is expected to sell about 28 million shares also through Sandler O’Neill and will bring in close to $70 million based on today’s prices.

Two more banking deals are also on deck that will either come tonight or perhaps next week.  Cadence Financial Corp. (NASDAQ: CADE) is expected to sell 31.5 million shares through FBR Capital Markets in an effort to raise roughly $55 million based upon today’s price.  Be advised, Cadence is extremely think trading volume.  Umpqua Holdings Corp. (NASDAQ: UMPQ) is expected to sell some $175 million in common stock via JPMorgan and others including KBW.

When we tally all of this up, it comes to more than $1.6 billion in secondary offerings just between today and what may come tomorrow.  That cash has to come from somewhere and generally has to stay in the stock for at least a little bit of time.

You can join our open email distribution list if you like to hear about secondary offerings, IPOs, Warren Buffett and other guru activities, mergers, key daily analyst reports, and other top trading and investing issues.

JON C. OGG
AUGUST 13, 2009

Time again to pop a shell and dig into the deep, geeky Unix internals of OS X with Dig Into Unix. Today we are going to look at two top-shelf power tools for text editing: sed and awk.

Sed is a Stream EDitor, and if you recall our previous Dig Into Unix installment concerning standard streams, you’ll understand that the streams we are talking about are actually just text from one source or another. Sed’s bread and butter is text search and replace, very similary to the “Edit” and “Find…” functions in TextEdit and many other GUI text editors. Unlike those text editors though, sed, by default, will write its output to the screen, or stdout.

As an example, try some basic operations on this string of text:
The quick brown fox jumped over the lazy dog's back.

Save the string of text as a file named test.txt, and type this into the Terminal:
sed s/quick/slow/g test.txt

The fox is now slow on the screen, but not changed in the file itself. To follow the stream, the text came from the file, through sed, and to the screeen. The best set of examples I’ve found for getting right into sed and starting to play with it is the collection of sed one liners hosted at Sourceforge.

Personally, I use sed when I’ve got a large number of configuration files that need to be edited. For example, it might be decided that we do not need our Nagios monitoring system alerting on the a certain statistic. I could go into 100 different files and perform the same action on all of them, or I could rely on a simple shell script and sed to do it for me.


for each in `ls *.cfg`; do
mv $each $each.bak #Safety First!
sed '30,35s/^/\#/g' $each.bak > $each
done


This will plow through all of the config files in a certain directory and add a # sign at the beginning of lines 30 through 35, commenting those lines out. Then I can restart Nagios, and if all goes well, delete all of the .bak files created as backups by the script.

While sed operates on lines and regular expressions (the subject of a future Dig Into Unix article!), awk works with fields. When given a stream of text, either from a text file or piped in from another application, awk can manipulate the text and rearrange the words. By default, awk separates the text fields by a space character, but you can use any other character you’d like.

Like sed, awk also has a great collection of one liners, this collection here is a great resource collected by Eric Pement. In my day to day activities, I call on awk when I want to format text for a report or to be input into another application.

To use our quick brown fox example again, we can print only the fourth, third, and second words, in reverse order, with this command:
awk '{ print $4, $3, $2 }' test

That will print out “fox brown quick”. Not very practical or useful. Something more practical might be to manipulate a list of comma separated values. Using the “-F” flag in awk, you can tell awk to separate it’s fields based on the comma, as in:
awk -F, '{ print $1 }' test

Since there are no commas in the test file, this will print the entire string of text. So, we could run this command to take care of that:
sed s/\ /,/g test > test2

This will use sed to replace all spaces with commas. The backslash is there to escape a special character, so the space is interpreted literally and not as part of a command. Now, you could use awk to manipulate the string of text as needed.
awk -F, '{ print $7, $8, $9, $10, $5, $6, $1, $2, $3, $4 " did" }' test2

This article has just barely skimmed the surface of what sed and awk can do. There are some rather hefty books dedicated to the pair, this one from O’Reilly has been on my desk for years now. Awk is an entire programming language, but the point of this series is not to teach the in-depth details, it’s just to get your feet wet, and maybe, just maybe leave you thirsting for more. The real rub is that everything that sed and awk can do can also be done, at times more efficiently, with the practical extraction and report language…better known as Perl, which is the subject of a future Dig Into Unix article.

Here is something I emailed myself from a previous job, 2 short AWK scripts for parsing Cisco IOS and PIX/ASA syslogs into lists of source/destination IPs and dest Ports. Very handy for trying to figure out just what’s been going through your devices.

The PIX/ASA logs need to be on informational (the level where it shows build and teardown messages, and the IOS logs need to have tcp/udps any any with gt’s in them, so ports are listed. That is to say, if you do IP any any log, it will show the port as (0), the router doesn’t inspect down to that level with IP to return a port number.

So the logging ACL needs to be something like this:

ip access-list extended PATRICK
permit tcp any gt 0 any gt 0 log
permit tcp any gt 0 any gt 0 log
permit icmp any any log
permit ip any any

# these commands AWKs all the build connection lines:

/Built/ {if (($8 ~ /^TCP$/ || $8 ~ /^UDP$/) && $7 ~ /^outbound$/) print "src",substr($16,1+ index($16,"("),-2 + index($16,"/")),"dst",substr($13,1+ index($13,"("),-2 + index($13,"/")),$8,substr($12,1+index($12,"/"),5)}
/Built/ {if (($8 ~ /^TCP$/ || $8 ~ /^UDP$/) && $7 ~ /^inbound$/) print "src",substr($13,1+ index($13,"("),-2 + index($13,"/")),"dst",substr($16,1+ index($16,"("),-2 + index($16,"/")),$8,substr($15,1+index($15,"/"),5)}
/Built/ {if ($8 ~ /^ICMP$/) print "src",substr($14,1+ index($14,"("),-1 + index($14,"/")),"dst",substr($12,0,-1 + index($12,"/")),$8}

# put the above into a file and reference it against the syslog folder you want to capture:

zcat /u01/networks/vpn-fw4.adl.nw.local/*gz | awk -f search_pix3.awk > pix3output

# then sort and count the unique lines:

sort pix3output | uniq -c
script for parsing router syslogs:

/SEC-6-IPACCESSLOG/ {
        if ($12 ~ /^log_hw_ports_inbound$/ && ($14 ~ /^tcp$/ || $14 ~ /^udp$/)) \
        print \
        $12,$14,"src",substr($15,0,-1 + index($15,"\(")), \
        "dst",substr($17,0,-1+index($17,"\(")),substr($17,1+index($17,"("),index($17,")")-index($17,"(")-1);
        #
        if ($12 ~ /^log_hw_ports_outbound$/ && ($14 ~ /^tcp$/ || $14 ~ /^udp$/)) \
        print \
        $12,$14,"src",substr($15,0,-1 + index($15,"\(")), \
        "dst",substr($17,0,-1+index($17,"\(")),substr($17,1+index($17,"("),index($17,")")-index($17,"(")-1);
        #
        if ($14 ~ /^icmp$/) \
        print \
        $12,$14,"src",$15,"dst",$17,$18

if anyone reads this and needs some help implementing it, I can try to explain it, it made sense to me at the time!

Meaning of awk & sed from Wikipedia

“AWK is a language for processing files of text. A file is treated as a sequence of records, and by default each line is a record. Each line is broken up into a sequence of fields, so we can think of the first word in a line as the first field, the second word as the second field, and so on. An AWK program is of a sequence of pattern-action statements. AWK reads the input a line at a time. A line is scanned for each pattern in the program, and for each pattern that matches, the associated action is executed.” – Alfred V. Aho

“SED (stream editor) is a Unix utility that (a) parses text files and (b) implements a programming language which can apply textual transformations to such files. It reads input files line by line (sequentially), applying the operation which has been specified via the command line (or a sed script), and then outputs the line.”

An introductory book:  Dougherty, D. & Robbins, A. (1997). Sed & awk. 2nd. ed. Sebastopol, CA : O’Reilly.