Here, you can use “virtual machine” as in “Java Virtual Machine”, not as in virtualization. We will play with the virtual machine described in the paper Tracing the Meta-Level: PyPy’s Tracing JIT Compiler by C.F. Bolz, A. Cuni, M. Fijalkowski and A. Rigo (it’s a great read by the way).

PyPy is a fascinating project, too complex to describe here. Among other things, PyPy can take any interpreter written in a subset of Python, translate it to C, and automatically generate a JIT compiler for this language. Does it sound too good to be true? Let’s try this.

• grab PyPy source code
• create the interpreter in pypy/translator/goal/target-toy.py with the following code:

import os, sys
import autopath
import py

# these are the opcodes for the interpreted language
JUMP_IF_A  = 1
MOV_A_R    = 2
MOV_R_A    = 3
ADD_R_TO_A = 4
DECR_A     = 5
RETURN_A   = 6

from pypy.rlib.jit import JitDriver
tlrjitdriver = JitDriver(greens = ['pc', 'bytecode'],
                         reds = ['a', 'regs'])

# the main interpreter loop
def interpret(bytecode, a):
   regs = [0] * 256
   pc = 0
   while True:
       tlrjitdriver.jit_merge_point(bytecode=bytecode, pc=pc, a=a, regs=regs)
       opcode = bytecode[pc]
       pc += 1
       if opcode == JUMP_IF_A:
           target = bytecode[pc]
           pc += 1
           if a:
               if target<pc:
                   tlrjitdriver.can_enter_jit(bytecode=bytecode, pc=target, a=a, regs=regs)
               pc = target
       elif opcode == MOV_A_R:
           n = bytecode[pc]
           pc += 1
           regs[n] = a
       elif opcode == MOV_R_A:
           n = bytecode[pc]
           pc += 1
           a = regs[n]
       elif opcode == ADD_R_TO_A:
           n = bytecode[pc]
           pc += 1
           a += regs[n]
       elif opcode == DECR_A:
           a -= 1
       elif opcode == RETURN_A:
           return a

# __________  Entry point  __________
def entry_point(argv):
    # the program we want to interpret
    # it computes the square of its argument
    bytecode = [
        MOV_A_R,    0, # i = a
        MOV_A_R,    1, # copy of ’a’
        # 4:
        MOV_R_A,    0, # i--
        DECR_A,
        MOV_A_R,    0,
        MOV_R_A,    2, # res += a
        ADD_R_TO_A, 1,
        MOV_A_R,    2,
        MOV_R_A,    0, # if i!=0: goto 4
        JUMP_IF_A,  4,
        MOV_R_A,    2,
        RETURN_A
    ]
    result = interpret(bytecode, int(argv[1]))
    print result
    return 0

def jitpolicy(driver):
    from pypy.jit.metainterp.policy import JitPolicy
    return JitPolicy()

# _____ Define and setup target ___
def target(*args):
    return entry_point, None

# main function, if this script is called from the command line
if __name__ == '__main__':
    entry_point(sys.argv)

• the lines in italic are the annotations for the JIT compiler. We need to give PyPy some insight on the interpreted language by declaring what is the instruction pointer (the green variables), the beginning of the dispatch loop and the backward branches (see the paper for full details).
• check that you can execute this script correctly by running python target-toy.py 12, the output should be 144
• PyPy can translate this script in C. For this, first install the dependencies and then run the following command: python translate.py target-toy.py
• this should give you an executable target-toy-c, rename it target-toy-native and check that ./target-toy-native 12 yields 144
• now we can ask PyPy to translate target-toy.py in C and generate a JIT compiler for it. For this, we just run python translate.py --opt=jit target-toy.py
  • note: the 64-bit backend of PyPy is not implemented yet, so if you are on a 64-bit system, you will have to struggle a bit. You will have to use a 32-bit Python interpreter (see my former post), create an alias for gcc -m32 (let’s call it gcc32) and then pass the option –cc=gcc32 to translate.py.
• this should give you another target-toy-c executable, rename it to target-toy-jit and check that ./target-toy-jit 12 yields 144

Ok, everything is working, so let’s now see how all this performs by computing large squares:

~/pypy-trunk/pypy/translator/goal$ time python target-toy.py 1000000
1000000000000
real 0m18.637s

~/pypy-trunk/pypy/translator/goal$ time ./target-toy-native 1000000
-727379968
real 0m0.024s

~/pypy-trunk/pypy/translator/goal$ time ./target-toy-jit 1000000
-727379968
[...]
real 0m0.005s

The first run is the square program interpreted by our program, itself interpreted by the Python interpreter. Double interpretation is slow.

The second run is the square program interpreted by a native version of our interpret function. Interpretation by native code is ok.

The third run is the square program interpreted and JIT’ed on the fly. It’s super awesome

Final note: I must thank everybody from #pypy on freenode, for their help and resilience to stupid questions. Thanks guys!

(disclaimer: the author of Savarin, Matthieu Kaczmarek, is a colleague working in the office next door and a friend of mine)

Savarin is a free online binary classification service (you can think of it as automatic diff’ing against large databases of programs). It is in beta, not fully polished yet, but you can still squeeze some interesting results out of it. Here is your daily shot of binary analysis, freshly brewed.

You will need:

• 2 different malware samples in the same malware family. We are going to use Sasser.A (already in Savarin’s database) and an unpacked Sasser.G (md5 b973853d0863070aca89ce00d4ee0fb9 [offensivecomputing.net])
• IDA with IDAPython for the actual diff’ing (I have IDA 5.5, I don’t know if this works with the free version)

Let’s go:

1. open Savarin
2. in “Classification against custom database”, choose SasserA
3. upload the Sasser.G sample
4. in the results page, click More to see the similarity with other binaries in the Sasser family
5. you can see that the sample is 41.95% similar to a sample with md5 edc66a4031f5a41f9ddf08595a1d4c92

At this point, you have a classification of a sample against a (small) database of programs. You can therefore see the distance between this sample and other samples. If you ask me, it’s a lot better to see that unknownsample.exe is 80% similar to badguy.exe and 90% similar to badguy2.0.exe than just “infected” or “not infected”.

For the actual diff’ing, follow these steps:

1. open the Sasser.G sample in IDA
2. download the IDAPython analysis report on Savarin’s analysis page (this report contains all the data needed to visualize the binary differences in IDA)
3. execute the IDAPython analysis report
4. right now, the situation is pretty anticlimactic since you should see no change apart from a few lines in the console. Wait until next step for the interesting stuff. Yes, you had nothing to do in this step, so what?
5. type SavColor(‘md5.edc66a4031f5a41f9ddf08595a1d4c92′, 0×0088ff) in the IDAPython console (it is the md5 value of the Sasser.A sample)
6. type SavComment(‘md5.edc66a4031f5a41f9ddf08595a1d4c92′) in the IDAPython console
7. this is it, now you can browse the Sasser.G sample, and the common parts with Sasser.A will be colored. Additionally, for two matching instructions you will see the corresponding address in the Sasser.A sample.

The Fine Screenshots:

Thanks to Silvio Cesare and Felix Gröbert, I now have 44 species in my packer zoo. I tested TraceSurfer on these 44 packers, here are the full results and the visualizations.

(quick recap: TraceSurfer uses Pin to trace every instruction in target binaries, extracting a trace file. Then the trace is analysed (or surfed) to detect layers of self-modifying code (or code waves) and code protection patterns)

Highlights:

• only 35 binaries execute correctly on my machine
• Pin v. 31933 works on 30 of these binaries (success rate of 85.71%)
• some packers don’t seem to pack anything since they have only 1 code wave (!Epack Lite 1.4, Pepack, VmProtect). Either the binaries are indeed packed and I fail to detect the self-modifying code, or they’re not self-modifying at all. It’s probably the latter, since the trace sizes are very close to the original (unpacked) binary.
• very few packers seem to use code scrambling (supposedly an advanced anti-dumping technique): Acprotect, Petite and the Yoda family
• integrity checking seems to be more popular, it is used in Acprotect, Themida, Enigma, Pelock, NSPack…
• given the trace size, it seems that anti-emulation loops can be found in Morphine (164M instructions) and !Epack 1.0 (87M instructions)
• nice similarities between Yoda Crypter 1.3 and Yoda Protector 1.4
• the Monster Award is attributed to… Themida 1.8.5.2 (80M instructions, 31 code waves, 142890 decrypted bytes + integrity checking)

I have been annoyed so much this week by a Javascript issue with Internet Explorer.

In IE6, IE7, and IE 8 it seems to be a problem to write code like this:

a={ foo: true, bar: false, };

The code seems to work all fine in Firefox and Safari, however, in Internet Explorer it will cause a Javascript error because of the last comma not followed by another declaration. Removing the last comma will make the code work in IE as well .

The issue is well known and Google has a lot of advice on it (simply search Google for “trailing comma javascript ie”).

I wanted to have something that auto-detects this issue for frequently changing Javascript code in a specific folder and that should be run as part of a test suite.

JSLint seems to be an excellent validation tool, however, it’s written in Javascript and I don’t like the clumsy workarounds that are suggested to execute Javascript on the command line.

There are also full-featured Javascript tools like Rhino and stuff around, however, they are not lightweight and cannot be run when there is no Java around.

So I put something together in PHP that tries to find the trailing commas in a bunch of Javascript files as well. It is small and simple and does not pretend to do anything else like all the other validators around.

The code can be found at http://code.google.com/p/phpcodetools/

Copying and pasting existing code normally leads to having a lot of redundant code around. Not only will the codebase grow in size but furthermore, copying & pasting will likely make future bugfixes and changes more time-consuming, error prone and difficult than actually necessary.

I think that even if it is not desired, it is still quite common to have at least a bit of identical code around: code that worked well in one situation is often used as the starting point (read: copied) for a new solution or script.

I was wondering how much identical code I had but didn’t really have a clue.

So I hacked together a small script that will go through a list of PHP files, read their code and try to find the duplicate codes parts in them and between them.

The duplication detection works as follows:

• Foreach PHP script that is read, all single and multilines comments will be removed (this is done using PHP’s built-in tokenizer)
• The remaining code is processed line by line. Each line is normalized by removing all whitespace and curly brackets (this is done to account for different indenting styles people use)
• For all remaining non-empty lines, a fingerprint is calculated using CRC32 and stored in a buffer
• When the code of all scripts has been read and processed, all possible pairs of files will be checked for identical code
• Code is considered identical if 10 consecutive code lines (there is an option to change this to a different value) from two lines are identical. Note that different comments and indenting are ignored here and just the net code is relevant.

For any duplicate code that is found, a message will be printed showing the identical code parts ranges with the corresponding file names and line number ranges.

The output will be sorted so that the longest matches will be reported on top. There will also be a reported of files with duplicate code parts, again sorted with worst cases on top.

Example output when running the check for Zend Framework 1.9, limited to the files in directory Zend/Controller:

Worst matches:
-    69 matching lines found between Zend/Controller/Router/Abstract.php:70,138 and Zend/Controller/Dispatcher/Abstract.php:282,350
-    69 matching lines found between Zend/Controller/Router/Abstract.php:70,138 and Zend/Controller/Front.php:653,721
-    69 matching lines found between Zend/Controller/Dispatcher/Abstract.php:282,350 and Zend/Controller/Front.php:653,721
-    15 matching lines found between Zend/Controller/Action/Helper/ContextSwitch.php:1139,1153 and Zend/Controller/Action/Helper/ContextSwitch.php:1179,1193
-    12 matching lines found between Zend/Controller/Request/Apache404.php:58,69 and Zend/Controller/Request/Http.php:396,408

Worst files:
-   138: Zend/Controller/Router/Abstract.php
-   138: Zend/Controller/Front.php
-   138: Zend/Controller/Dispatcher/Abstract.php
-    30: Zend/Controller/Action/Helper/ContextSwitch.php
-    12: Zend/Controller/Request/Apache404.php
-    12: Zend/Controller/Request/Http.php

The script is itself is written in PHP and it should work at least with PHP 5.2.x. It is self-contained and should not have any dependencies to any extensions, libraries etc. than the built-in PHP functions. It is designed to be run from the command line so the output will not be formatted nicely in a web browser.

The command line options when invoking the script are:

• –threshold: number of minimum lines that must be identical to be considered a match. Defaults to 10.
• –regex: file inclusion regex. Must be a PCRE enclosed in forward slashes. Defaults to /\.php[345]?$/.

Please note that the script should not be run on an extremely large codebase unless you have a lot of time. This is because the script will compare all lines from all lines against all lines from all other files and this can really get slow with a huge number of files.

The project is hosted at Google code and can be downloaded here: http://code.google.com/p/phpcodetools/

We are currently looking for some tool to help us find weaknesses in our code. There is not lots of options currently on the market. A tool we are looking on is http://www.fortify.com and http://www.armorize.com/, but who does best work on code? Biggest advantage for Fortify is that it supports many different languages, but damn it is really expensive tools.

Microsoft is  planning to release their Security Tools but i think they are currently on early stage.
https://connect.microsoft.com/site/sitehome.aspx?SiteID=734&wa=wsignin1.0

• CAT.NET – the managed code security source code scanning tool
• WACA – Web Application Configuration Analyzer
• WPL – Web Protection Library (formerly Anti-XSS)
• TAM – Threat Modeling and Analysis tool

Only tool i tried to so far is CAT.NET but something is going terribly wrong when i run test on code (see error below), but i think this is a great initiative from Microsoft. What is wrong with my pdb, think is have wrong paths set or something…

Error from my run CAT.NET  

C:\Program Files (x86)\Microsoft Information Security\Microsoft Code Analysis for .NET (CAT.NET) v2.0>CATNetCmd.exe /fil
e:c:\aDLLFILE.dll /configdir:C:\myPathToFolder
Microsoft (r) Code Analysis Tool for .NET (CAT.NET) Tool 2.0.0.0
Copyright (c) Microsoft Corporation 2009.  All rights reserved.

Running in 32-bit mode

2009-11-25 13:17 : Information : Loading analysis rules…done.
2009-11-25 13:17 : Information : Total 40 rules loaded by the engine.
2009-11-25 13:17 : Information : Processing analysis rules…
2009-11-25 13:17 : Information : Initializing configuration analysis engine…done.
2009-11-25 13:17 : Information : Initializing interprocedural data flow analysis engine…The available PDB has been str
ipped so it does not contain the required
information for native or IJW images.
Please find the full PDB for this binary and re-run your scenario.

Unhandled Exception: System.AccessViolationException: Attempted to read or write protected memory. This is often an indi
cation that other memory is corrupt.
   at Phx.Pdb.ReaderImplementation.GetDataStream(String sectionNameString)
   at Phx.PE.ReaderPhase.ReadPEFixups()
   at Phx.PE.ReaderPhase.SeedFixups()
   at Phx.PE.ReaderPhase.CodeDiscovery()
   at Phx.PE.ReaderPhase.Translate()
   at Phx.PEModuleUnit.LoadGlobalSymbols()
   at Microsoft.InformationSecurity.CodeAnalysis.Engines.AnalysisEngine.TaintedAnalysisEngine.Initialize(String assembly
path, String[] phoenixargs, Int32 maxnumofpasses, Rules rules)
   at Microsoft.InformationSecurity.CodeAnalysis.Engines.RulesEngine.RulesEngine.ProcessRules()
   at Microsoft.InformationSecurity.CodeAnalysis.UI.CommandLine.Program.Main(String[] args)

Update

Seems like it is a Windows 7 problem.

http://social.msdn.microsoft.com/Forums/en-US/phoenix/thread/2085fa72-19d8-4a6b-b6e0-8777e4fbfc59?prof=required

When you develop reusable components quality is very important. The higher a reusable asset’s usage, the higher the need for robustness. Unlike monolithic code, defects with reusable assets can rapidly impact several business processes and applications. All the more reason for the criticality of automated testing. In addition to testing, code analysis tools can detect defects, warn regarding unsafe/potentially buggy code, and also recommend various source code style/formatting improvements. All of these contribute to higher quality. I have used findbugs, pmd, and checkstyle in this space and they are very useful to analyze source code. You can also include these in your continuous integration suite.

Many readers have requested that I post sample code and scripts – so here is a sample apache ant build script that you can use with your code. Just be sure to modify the properties file to point to appropriate folders. The script will produce html reports for pmd and checkstyle as well as a find bugs output file (for viewing it with the findbugs client).

Code Analysis Script

Pre-requisites prior to running this script:

Set JAVA_HOME to your JDK 1.5 or above folder

Install apache ant 1.6+, findbugs, pmd, and checkstyle in your environment

Make sure the findbugs, pmd, and checkstyle jar files are in ant’s CLASSPATH.

Let me know if you have issues with using this script. Enjoy!

Like this post? Subscribe to RSS feed or get blog updates via email.

I have been working with my PhD supervisor on a dynamic typing system to detect and visualize the temporal evolution of self-modifying programs (it’s not as complicated as it sounds). The typing system works as follows:

• each memory address has a read, write and execution level (r, w, x)
• initially, every memory address begin with type (0, 0, 0)
• when an address with type (r, w, x) is executed, its type becomes (r, w, w+1)
• when an instruction with type (r1, w1, x1) reads a memory address with type (r2, w2, x2), the target address type becomes (x1, w2, x2)
• when an instruction with type (r1, w1, x1) writes to a memory address with type (r2, w2, x2), the target address type becomes (r2, x1, x2)

With that we can get a trace from a program (with DBI, an emulator, a debugger, whatever) and see what is executed (execution level >= 1). By construction, if we have code with an execution level of 2, it means that it has been written by the program itself before being executed, therefore it is self-modifying code.

Again by construction, if we see code with an execution level k+1, it means that it has been written by code at level k. Hence we can precisely distinguish between different layers of code (in our jargon, different code waves)

Now we can detect some interesting properties based on the type of memory addresses:

• if an address has been read, written and then executed (RWX), we label it decrypted
• if an address has only been written and executed (WX), we label it blind write
• if an address has been executed and then read (XR), we assume there has been an integrity check
• if an address has been executed and then written (XW), we assume the code has been scrambled (supposedly as an anti-memory-dump technique)

Therefore we have a way to trace different layers of code, and some relations between the layers (decryption, blind writes, integrity checking and code scrambling). This gives us the following visualization for some packers:

Note 1: thanks to Silvio Cesare for providing the packed samples

Note 2: we are going to present all this stuff at Malware (Montréal) with Jean-Yves Marion and Wadie Guizani, and at Deepsec (Vienna)

Recently I’ve been wondering, how is malware analysis different from traditional program analysis? The fundamental reason is that programs can generally self-modify themselves. There is a direct consequence: with malware we have to admit that we don’t have static access to the program listing (thus preventing standard program analyses). And since turning self-modifying code into normal code is undecidable, we end up only with technical, partial solutions. This is why virtually every paper on malware analysis will only be a report on how a given technology/implementation is better/faster/stronger than the others.

This has a corollary too: since we have only partial solutions, in some cases they don’t work. And malware authors actively exploit that fact, by implementing techniques to defeat our implementations. This opened a sub-research field: the production of techniques to defeat the analysis-defeating techniques. Yes, there is some irony in this, for instance think about packing -> emulation-based unpacking -> anti-emulation techniques -> other-wonderful-unpacking-techniques…

Now, you might wonder, how did we get into this quagmire? As Schneier pointed it out before me, this is an accident – a historic by-product of the way the IT industry evolved. The x86 architecture allowed self-modifying code, and operating systems did nothing to prevent or regulate that. And bam, a research niche was born.

هي اداة برمجية مجانية تقوم شركة مايكروسوفت بإصدارها ، وهي عبارة عن اضافة على Visual Studio 2005 وما فوق ، تهدف الى البحث في اكواد مشاريعك عن نقاط الضعف في البيانات ، والتي يقصد بها الثغرات المعروفة كـ SQL Injection ،XSS Injection و XPath Injection في اكوادك وتعرضها عليك ، هذه قائمة بهذه الثغرات التي تكتشفها:

Cross Site Scripting
SQL Injection
Process Command Injection
File Canonicalization
Exception Information
LDAP Injection
XPATH Injection
Redirection to User Controlled Site

لتحميل هذه الاداة اتبع الرابط التالي ، علما بانها لاتزال تحت التطوير beta .

What is FxCop ?

From the MSDN, FxCop is an IL code analyzer. It will analyze assemblies and check for rules violations according to the .NET Design Guidelines.

You can download it (v-1.36) from here. It is quite handy, even though not all rules make sense for all projects — especially if you are developing applications rather than libraries.

How to use it ?

Create a project and add compiled libraries for it to analyze. It is better to have it analyze assemblies with debug information so that it can link to the sources.

You can then decide the categories of rules to run and which to exclude. The next step is to analyze your assemblies which will provide a report containing details of all rule violations. This can be a pretty heavy report, especially if you have a large project already advanced in its life cycle.

It is now time to take care of the violations found, either by fixing them, excluding them from the project or excluding them from the source. Sometimes it is not possible or it does not make sense to fix a violation (according to the context) so it is preferable to exclude it so that it does not keep coming back. Excluding them from the project is quick but prone to later problem: conflicts in or loss of the project file, renaming of files/classes/methods. Those problems disappear when using in source exclusion with the SuppressMessageAttribute specifying the rule violation to suppress and possibly a justification for future reference. Here is an example (I always put my constant in ALL CAPS):

   21 [SuppressMessage( "Microsoft.Naming", "CA1709:IdentifiersShouldBeCasedCorrectly",

   22                   MessageId = "WIDTH", Justification="Coding conventions" )]

FxCop is integrated in the team edition of Visual Studio but not in the free or professional edition.

In order to get this attribute to work, the CODE_ANALYSIS symbol must be defined during the compilation of the assembly. Without this, FxCop will ignore the violations exclusions defined in the source code.

For batch run (or MS Build integration), you can use the FxCopCmd utility. You can find here a tutorial on how to integrate FxCopCmd with MS Build.

StyleCop

StyleCop is a closely related tool as it does source code analysis but directly on C# code rather than on disassembled MS IL. It also provides a set of rules and can be run from within Visual Studio.

It can be downloaded (latest version) from here. You can find here a tutorial on how to integrate StyleCop within an MS Build project.

One of the most useful metrics to us in the Softwareschneiderei is “CRAP”. For java, it is calculated by the Crap4J tool and provided as an HTML report. The report gives you a rough idea whats going on in your project, but to really know what’s up, you need to look closer.

A closer look on crap

The Crap4J tool spits out lots of numbers, especially for larger projects. But from these numbers, you can’t easily tell some important questions like:

• Are there regions (packages, classes) with lots more crap than others?
• What are those regions?

So we thought about the problem and found it to be solvable by data visualization.

Enter CrapMap

If you need to use advanced data visualization techniques, there is a very helpful project called prefuse (which has a successor named flare for web applications). It provides an exhaustive API to visualize nearly everything the way you want to. We wanted our crap statistics drawn in a treemap. A treemap is a bunch of boxes, crammed together by a clever layouting strategy, each one representing data, for example by its size or color.

The CrapMap is a treemap where every box represents a method. The size gives you a hint of the method’s complexity, the color indicates its crappyness. Method boxes reside inside their classes’ boxes which reside in package boxes. That way, the treemap represents your code structure.

A picture worth a thousand numbers

This is a screenshot of the CrapMap in action. You see a medium sized project with few crap methods (less than one percent). Each red rectangle is a crappy method, each green one is an acceptable method regarding its complexity.

Adding interaction

You can quickly identify your biggest problem (in terms of complexity) by selecting it with your mouse. All necessary data about this method is shown in the bottom section of the window. The overall data of the project is shown in the top section.

If you want to answer some more obscure questions about your methods, try the search box in the lower right corner. The CrapMap comes with a search engine using your methods’ names.

Using CrapMap on your project

CrapMap is a java swing application, meant for desktop usage. To visualize your own project, you need the report.xml data file of it from Crap4J. Start the CrapMap application and load the report.xml using the “open file” dialog that shows up. That’s all.

In the near future, CrapMap will be hosted on dev.java.net (crapmap.dev.java.net). Right now, it’s only available as a binary executable from our download server (1MB download size). When you unzip the archive, double-click the crapmap.jar to start the application. CrapMap requires Java6 to be installed.

Show your project

We would be pleased to see your CrapMap. Make a screenshot, upload it and leave a comment containing the link to the image.

So, you’ve inspected your Java code in any possible way, using Findbugs, Checkstyle, PMD, Crap4J and many other tools. You know every number by heart and keep a sharp eye on its trend. But what about some simple questions you might ask yourself about your project, like:

• How many instance variables aren’t final?
• Are there any setXYZ()-methods without any parameter?
• Which classes have more than one constructor?

Each of this question isn’t of much relevance to the project, but its answer might be crucial in one specific situation.

Using QDox for throw-away tooling

QDox is a fine little project making steady progress in being a very intuitive Java code structure inspection API. It’s got a footprint of just one JAR (less than 200k) you need to add to your project and one class you need to remember as a starting point. Everything else can be learnt on the fly, using the code completion feature of your favorite IDE.

Let’s answer the first question of our list by printing out all the names of all instance variables that aren’t final. I’m assuming you call this class in your project’s root directory.

public class NonFinalFinder {
    public static void main(String[] args) {
         File sourceFolder = new File(".");
         JavaDocBuilder parser = new JavaDocBuilder();
         builder.addSourceTree(sourceFolder);
         JavaClass[] javaClasses = parser.getClasses();
         for (JavaClass javaClass : javaClasses) {
             JavaField[] fields = javaClass.getFields();
             for (JavaField javaField : fields) {
                 if (!javaField.isFinal()) {
                     System.out.println("Field "
                       + javaField.getName()
                       + " of class "
                       + javaClass.getFullyQualifiedName()
                       + " is not final.");
                }
            }
        }
    }
}

The QDox parser is called JavaDocBuilder for historical reasons. It takes a directory through addSourceTree() and parses all the java files it finds in there recursively. That’s all you need to program to gain access to your code structure.

In our example, we descend into the code hierarchy using the parser.getClasses() method. From the JavaClass objects, we retrieve their JavaFields and ask each one if it’s final, printing out its name otherwise.

Praising QDox

The code needed to answer our example question is seven lines in essence. Once you navigate through your code structure, the QDox API is self-explanatory. You only need to remember the first two lines of code to get started.

The QDox project had a long quiet period in the past while making the jump to the Java 5 language spec. Today, it’s a very active project published under the Apache 2.0 license. The developers add features nearly every day, making it a perfect choice for your next five-minute throw-away tool.

What’s your tool idea?

Tell me about your code specific aspect you always wanted to know. What would an implementation using QDox look like?

Metasm is an LGPL’ed assembly manipulation framework written in Ruby. It is capable of following flow dependences along multiple paths, a feature called backtracking in metasm jargon and close to the notion of program slicing [wikipedia.org].

Definition: a node j is flow-dependent on a node i if (1) a variable x is referenced at j, (2) x is defined at i and (3) there exists a path from i to j without intervening definitions of x. Source [research.ibm.com]

We are going to use the following ruby script to test the backtracker with different source programs, it takes as input a program (sourcecode), a node (labeled ’sliceme’) and a variable (eax) and returns the flow-dependences of eax and in some cases, the actual values that eax will take.

require 'metasm'
include Metasm

def sliceit(asmsource, label, reg)
    # encode the shellcode
    sc = Shellcode.assemble(Ia32.new, asmsource)
    dasm = sc.disassemble(0)

    # get the address of the label
    offset = sc.encoded.export[label]

    reg = reg.to_sym

    # backtrace
    log = []
    dasm.backtrace(reg, offset, :log => log)

    # return the trace
    log
end

sourcecode = <<EOS
; asm program here
EOS

slice = sliceit(sourcecode, 'sliceme', 'eax')
# slice est un tableau qui contient le log du backtrace
slice.each { |ev, *args|
    if ev == :di    # decodedinstruction
        after = args[0]
        before = args[1]
        instr = args[2]
        puts "#{instr} [#{before} -> #{after}]"
    end#
}

Let’s take some simple code samples. If there is a single execution path, metasm can compute the value contained in the variable, for instance:

mov eax, 42h
inc eax
sliceme: jmp eax

-> metasm returns:
5 inc eax [eax -> eax+1]
0 mov eax, 42h [eax+1 -> 43h]

We can see that useless definitions of the variable are not taken into account:

mov eax, 0h
mov eax, 42h
inc eax
sliceme: jmp eax

-> metasm returns:
0ah inc eax [eax -> eax+1]
5 mov eax, 42h [eax+1 -> 43h]

Definitions of other variables are also ignored (if eax does not depend on them):

mov eax, 42h
mov ebx, 0
sliceme: jmp eax

-> metasm returns:
0 mov eax, 42h [eax -> 42h]

But they are included if eax depends on them:

mov ebx, 1h
mov eax, 42h
add eax, ebx
sliceme: jmp eax

-> metasm returns:
0ah add eax, ebx [eax -> eax+ebx]
5 mov eax, 42h [eax+ebx -> ebx+42h]
0 mov ebx, 1 [ebx+42h -> 43h]

If there are multiple acyclic paths, metasm will be able to compute the value of the sliced variable along each path, for instance:

mov eax, 42h
cmp ebx, 0
jnz pouet
dec eax
jmp sliceme
pouet: inc eax
sliceme: jmp eax

-> metasm returns:
0dh inc eax [eax -> eax+1]
0 mov eax, 42h [eax+1 -> 43h]
0ah dec eax [eax -> eax-1]
0 mov eax, 42h [eax-1 -> 41h]

Metasm states that at the end, eax will be either 41h or 43h but it doesn’t know which one. It doesn’t state it explicitly, but the value of eax is control-dependent on ebx.

Finally in the worst case scenario, if there are cyclic paths (such as loops), metasm will go through each path once but will be unable to compute the value after the cycle. Note that in the general case, this is undecidable.

mov eax, 0
entry: cmp ebx, 0
jnz sliceme

dec ebx
inc eax
jmp entry

sliceme: jmp eax

-> metasm returns:
0bh inc eax [eax -> eax+1]
0 mov eax, entrypoint_0 [eax -> 0]

This is mostly a note to myself, but I guess people interested in automating reverse engineering will be interested at some point in IR suitable for low-level abstractions. I consider top-down IR used by optimizing compilers and bottom-up IR used by decompilers and other reversing tools.

Intermediate Representations for Reverse Engineering

REIL. Used in BinNavi, the Reverse Engineering Intermediate Language defines a very simple RISC architecture (17 instructions), with the nice property that each instruction has at most one side-effect. Thomas Dullien and Sebastian Porst recently presented at CanSecWest an abstract interpretation framework for REIL (paper, slides). It is clearly possible to easily write analyses and transformation passes for REIL without getting into the complexity of the whole x86 architecture, given x86 -> REIL and REIL -> x86 translators.

Here are some sample REIL instructions :

1006E4B00: str edi, , edi
1006E4D00: sub esp, 4, esp
1006E4D01: and esp, 4294967295, esp
1006E4D02: stm ebp, , esp

Language Reference

Hex Rays Microcode. Presented at Black Hat USA 2008 by Ilfak Guilfanov (paper, slides), it is an IR used during decompilation. From the paper: “The microcode language is very detailed and precisely represents how each instruction modifies the memory, registers, and processor condition codes. Typically one CPU instruction is converted into 5-15 microinstructions”. According to the REIL paper, REIL and the microcode language are significantly different, for instance the microinstructions can have a variable number of operands and perform multiple side effects.

Sample microcode:

mov esi.4, eoff.4
mov ds.2, seg.2
add eoff.4, #4.4, eoff.4
ldx seg.2, eoff.4, et1.4
mov et1.4, eax.4

I couldn’t find the language reference.

ELIR. Part of the ERESI project, the goal of ELIR is to simplify static analysis by providing a platform independent abstraction. An overview was presented at Ekoparty08 (slides) and some ideas appeared in Phrack 64, but 30s of Googling didn’t get me to the language reference or a code sample, so that’s all I will say about ELIR for the moment.

Pin Inspection API. PIN, Intel’s Dynamic Binary Instrumentation framework provides a very handy instruction inspection API. This is not an IR but provides the same type of information about complex instructions without having to make giant switch statements. For instance, this is the way to log memory writes with PIN given an instruction:

VOID RecordMemWrite(VOID * addr, UINT32 size) {
    fprintf(trace,",%dW%p", size, addr);
}

// this function is called each time an instruction is encountered
VOID Instruction(INS ins, VOID *v) {
    // isn't that a nice API ?
    if (WRITES && INS_IsMemoryWrite(ins)) {
        INS_InsertPredicatedCall(
            ins, IPOINT_BEFORE, (AFUNPTR)RecordMemWrite,
            IARG_MEMORYWRITE_EA,
            IARG_MEMORYWRITE_SIZE,
            IARG_END);
    }
}

API Documentation

Valgrind IR. On my todo list.

FermaT Transformation System. I’ll have to write something about it someday. Oh lucky you, a wikipedia entry and a bunch of papers!

Optimizing Compilers Intermediate Representations

LLVM Bitcode. This language uses low-level RISC-like instructions in SSA form with type information. It is clean and well defined, and is a very suitable target for platform-independent analysis and optimization. It is designed to convey high-level information in lower level operations, so converting machine code to LLVM bitcode probably requires some intensive work.

Here is the hello world example :

; Declare the string constant as a global constant...
@.LC0 = internal constant [13 x i8] c"hello worldA0"          

; External declaration of the puts function
declare i32 @puts(i8 *)                                           

; Definition of main function
define i32 @main() {
        ; Convert [13 x i8]* to i8  *...
        %cast210 = getelementptr [13 x i8]* @.LC0, i64 0, i64 0 ; i8 *

        ; Call puts function to write out the string to stdout...
        call i32 @puts(i8 * %cast210)
        ret i32 0
}

Language reference

Register Transfer Language. One of the IR used in GCC, it is an architecture-neutral assembly language that represents instructions in a LISP-like form (d’oh), like this:

(insn 2 49 3 test.c:3 (set (mem/c/i:SI (plus:DI (reg/f:DI 6 bp)
                (const_int -20 [0xffffffffffffffec])) [0 argc+0 S4 A32])
        (reg:SI 5 di [ argc ])) 47 {*movsi_1} (nil))

It feels a bit old-fashioned and less clean than LLVM bitcode, but this is just a gut feeling. Use gcc -fdump-rtl-all to see what it looks like.

Side note: the idea of dumping RTL to a file, performing transformations on it and giving this back to GCC is quite common, but RMS qualifies it as “not feasible”, even though the creator of RTL says it is not only feasible but quite useful actually.

In SWF Scan 1.0, go to Settings / Checks tab to find what this program actually checks for.

The list comes up, with sortable fields of Enabled, Check Name, and Severity.

I sorted by Severity, to find the worst offenders.  Highest on the list is Critical, which includes such no-no’s as

• Application Source Available
• Possible Credit Card Number Disclosure
• Insecure Security.allowInsecureDomain() usage
• Insecure LocalConnection.allowDomain() usage
• Insecure Security.allowDomain() usage
• Insecure LocalConnection.allowInsecureDomain() usage

These are heavyweight problems to be sitting in source code, and the list then goes on to High where some notable items are Possible Social Security Number, Possible Database Connection String to various db vendors to ENABLEDEBUGGER Tag Detected.

Going down the Severity category list is Medium, Low, Info and BestPractice.

This can be a very useful tool to find potential gotchas that could potentially slip through the cracks, but no one should hardcoding connection strings on a regular basis.  I’d like to say it’s scary, but I guess it happens.

Update: InSideRIA article about SWFScan