Just thought of sharing the code i written to learn linked list implementation the day before my data structures model practical exam.

/*
 *      linkedlist.c
 *
 *      Copyright 2009 Rag Sagar.V <ragsagar@gmail.com>
 *
 *      This program is free software; you can redistribute it and/or modify
 *      it under the terms of the GNU General Public License as published by
 *      the Free Software Foundation; either version 2 of the License, or
 *      (at your option) any later version.
 *
 *      This program is distributed in the hope that it will be useful,
 *      but WITHOUT ANY WARRANTY; without even the implied warranty of
 *      MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 *      GNU General Public License for more details.
 *
 *      You should have received a copy of the GNU General Public License
 *      along with this program; if not, write to the Free Software
 *      Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
 *      MA 02110-1301, USA.
 */

#include <stdio.h>
#include <stdlib.h>

typedef struct list
{
	int data;
	struct list *next;
}LIST;
LIST *ptr,*temp,*start=NULL;
void insert_after(int ,int );
void remove_item(int );
void display(void);
int count=0;
int main()
{
	int item,opt,dat;
	system("clear");
	ptr=NULL;
	/* printf("%d",sizeof(LIST)); */
	do
	{
		printf("\n########## MENU ##########\n");
		printf("1.Insert\n2.Remove\n3.Display\n4.Exit\n");
		printf("Enter your option : ");
		scanf("%d",&opt);
	 	switch(opt)
	 	{
	 		case 1:
	 		printf("Enter the data to insert ");
	 		scanf("%d",&item);
	 		if(count==0)
	 		{
	 			ptr = (LIST *)malloc(sizeof(LIST));
	 			ptr->next = NULL;
	 			ptr->data = item;
	 			start = ptr;
			}
			else
			{
				printf("Enter the item after which you have to insert new element : ");
				scanf("%d",&dat);
				insert_after(dat,item);
			}
			count++;
				break;
	 		case 2:
	 		if(count==0)
	 		{
	 			printf("\nList is empty\n"); break;
			}
	 		printf("Enter the item to remove : ");
	 		scanf("%d",&item);
	 		remove_item(item);
	 		count--;
	 		break;
	 		case 3:
	 		if(count==0)
	 		{
	 			printf("\nList is empty\n");
	 			break;
	 		}
	 		else
	 		{
	 			printf("List elements are \n"); display();
			}
			break;
	 		case 4: break;
		}
	}while(opt!=4);
	return 0;
}
void insert_after(int data, int item)
{
	LIST *tmp;
	temp=(LIST *)malloc(sizeof(LIST));
	temp->data=item;
	ptr=start;

	while(ptr!=NULL)
	{
		if(ptr->data==data)
		{
			tmp=ptr->next;
			ptr->next=temp;
			temp->next=tmp;
			break;
		}
	ptr=ptr->next;
	}

}	

void remove_item(int item)
{
	ptr=start;
	if(ptr->data == item)
	{
		start = ptr->next;
		free(ptr);
	}
	while(ptr->next!=NULL)
	{
		if((ptr->next)->data==item)
		{
			temp=ptr->next;
			ptr->next=(ptr->next)->next;
			free(temp);
			break;
		}
		ptr=ptr->next;
	}
}

void display()
{
	ptr = start;
	while(ptr!=NULL)
	{
		printf("%d -> ",ptr->data);
		ptr=ptr->next;
	}
}			

Got bored, wanted to test my memory (and latent C skills):

#include 

typedef struct node {
        char* data;
        struct node* next;
}* NODE;

int main()
{

        NODE root = NULL;
        NODE tail = NULL;

        root = (NODE) malloc(sizeof(struct node));
        tail = (NODE) malloc(sizeof(struct node));

        root->data = "hello";
        tail->data = "world";

        root->next= tail;
        tail->next = NULL;

        // display the contents of the linked list
        NODE cur = root;
        for(;cur != NULL; cur= cur->next)
        {
                printf("%s ", cur->data);
        }

        free(root);
        free(tail);

        printf("\n");

        return 0;
}

And it worked. First time around. Nice!

But then I got distracted so I’m writing it now instead, and it necessarily will be a very different post than it would’ve been had I gone through with my original plan, because you know, life happens, and by life I mean Wednesday and Thursday and a good bit of Friday too.

So!  First order of business is my health.  I woke up Monday feeling AWESOME BEYOND ALL BELIEF, but in the reality outside my own delusional reality, I didn’t actually feel fully up to par till, um, Thursday-ish.  Ergo, I suspect it was the sunshine and cupcakes and my burning desire to leave my room that truly inspired my AWESOME BEYOND ALL BELIEF feeling Monday morning.

[Fun fact!  The instant I finished typing the word "sunshine," the chapel bells started playing "Here Comes the Sun."]

Turns out I made an excellent call not going to the health center over the weekend, for a number of reasons.  A number of reasons equaling two.  Reason one!  They apparently stopped actually diagnosing people with the swine, and instead have been just deporting everyone who walks in there with a fever and a bad mood.  Not cool, health services people, not cool.

Reason two!  Because of this, the quarantine mansion, AKA Mt. Hope, AKA Mt. DOOM, is full, so people within like driving distance are getting sent home.  Imagine how that return would go: “Hey mom!  Nice to see you for the first time since September!  Now care for me, because I have swine flu!”

[I'm in an all-caps, exclamation point sort of mood.  Could you tell?]

Anyway, this whole hype-enhanced epidemic means that a few of my classes are emptier than usual. This is particularly noticeable in classes like CS, where there are usually only 13 students… four of them girls… three of them absent today… Let’s just say I felt awfully female. At least my prof’s female.

Our CS lab for the last couple weeks before Thanksgiving is funny to me, because it’s basically the infamous GridWorld case study from AP CS, except we’re writing the code and not just understanding it, and– well, there are lots of other technical differences that are unimportant because hey, it’s a bunch of squares and little creatures moving through them and we get to make them battle and that’s FUN.

Speaking of battling, video games have been introduced to the Pratt 3 common room.  Sleeping and study habits have declined proportionally.  Swearing and machismo have increased proportionally.

Also!  Ben apparently has the magic touch with regards to my computer, because he broke it (i.e. incurred the spontaneous apparition of a black box at various places, plus other bizarre bugs) and fixed it– all just by hitting random keys.  And it FREAKS ME OUT.

By the way, this is in fact related to the previous discussion of video games, but unfortunately for you who were not involved in last night and its fairly ridiculous discussions, the relation is not topical but temporal.

Also!  Alex and Ben have a way of shouting my name that prevents me from distinguishing the two of them at all, which kind of messes with my mind.  They also have a knack for bothering me about math homework, although their ways of doing so are more distinguishable.  Last night I had the realization of epiphanic proportions that I’m going to be in linguistics class with both of them next semester.  It will be interesting.

I am not proceeding through my week in an at all efficient (let alone chronological) manner.

Because I should really be wrapping up, but there’s so much stuff I’ve missed.  Like my CS prof’s [AGE REDACTED] birthday party!  And the utter failure of Pratt 3 to produce a birthday party for our man Ian!  And the amazingly hilarious debate between Adams and Garrity about whether the derivative or the integral is superior (somehow mimes and cavemen were involved)!  And how Adams still calls me Emily but at least has apologized, and how I literally ran into Burger after the debate (sorry), and other things relating to this day!  And CHEESE!  And LLAMAS!  And BULLDOZERS!

See, this is why I need to blog more frequently.  You need to remind me to blog more frequently.  Yes, YOU do.

Python is a programming language that I learnt somewhat recently (something like 2, 3 years ago) and that I like very much. It is simple, to the point, and has several functional-like constructs that I am already familiar with. But Python is slow compared to other programming languages. But it was unclear to me just how slow Python was compared to other languages. It just felt slow.

So I have decided to investigate by comparing the implementation of a simple, compute-bound problem, the eight queens puzzle generalized to any board dimensions. This puzzle is most easily solved using, as Dijkstra did, a depth-first backtracking program, using bitmaps to test rapidly whether or not a square is free of attack¹. I implemented the same program in C++, Python, and Bash, and got help from friends for the C# and Java versions². I then compared the resulting speeds.

The eight queens puzzle consist in finding a way to place 8 queens on a 8×8 chessboard so that none of the queen checks another queen. By “checking”, we mean that if the other queen was of the opposing camp, you could capture it if it were your turn to play. So this means that no queen is on the same row, column, or diagonal as another queen. Using a real chessboard and pawns in lieu of queens, you can easily find a solution in a few seconds. Now, to make things more interesting, we might be interested in enumerating all solutions (and, for the time being, neglecting to check if a solution is a rotated or mirrored version of another solution).

The basic algorithm to solve the Eight queens puzzle is a relatively simple recursive algorithm. First, we place a queen somewhere on the first row and we mark the row it occupies as menaced. We also do so with the column and two diagonals. We then try to place a second queen somewhere on the second row, on a square that is menace-free, and mark the row, column, and diagonals of the new queen as menaced. And we proceed in the same fashion for other queens. But suppose that at the th stage, we cannot find a menace-free square, preventing us from placing the th queen. If the situation arises, we give up for the th queen and backtrack to the th queen. We try a new (and never tried before) location for the th queen and we go forth trying to place the th queen. If may happen that we go all the way back to the th queen because there are no other solutions for the th queen, which asks for the th queen to be moved, which can also result in a dead-end; and so forth all the way down to the th queen.

Because the algorithm proceeds depth-first and can rear back quite a bit in looking for new solutions, it is called a backtracking algorithm. Backtracking algorithms are key to many artificial intelligence systems like, well, chess programs.

So I wrote the program for the queens puzzle for a board in C quite sometime ago (circa 1995), then I ported to C++ recently (2006). And just for kicks, I decided to port it to different languages, with help from friends for Java and C# (of which I know about zilch). The C++ versions consist in a generic version that accepts the board size as a command-line argument and in a constant-propagated version where the board size is determined at compile-time. The Python versions, as there are two of them also, differ on how python-esque they are. The first variant is a rather literal translation of the C++ program. The second uses pythonesque idioms such as sets rather than bitmaps and turns out to be quite a bit faster—about 40% faster in fact. The Bash version is necessarily rather bash-esque as Bash does not offer anything much more sophisticated than arrays in terms of data structures.

All implementations were compiled with all optimizations enabled (-O3, inlining, interprocedural optimizations, etc.). For C++, I used g++ 4.2.4, for C#, gmcs 1.2.6.0, for Java, gcj 4.2.4, Python 2.6.2, and, finally, Bash 3.2.39—all latest versions for Ubuntu 8.04 LTS, which doesn’t means that they’re the really latest versions.

So I ran all seven implementations with boards sizes ranging from 1 to 15 on a AMD64 4000+ CPU (the numbers are arbitrary; I wanted to get good data but also limit the CPU time spent as the time increases factorially in board size!). At first, the results are not very informative:

Results, Time, Linear Scale

As expected, all times shoot up quite fast, with, unsurprisingly, BASH shooting up faster, followed by the two Python implementations. At this scale, Java, C#, C++, C++-fixed (the constant-propagated version) all seems to be more or less similar. To remove the factorial growth (as even with a log plot the data remains rather unclear), I scaled all performances relative to the C++ version. The C++ version is therefore 1.0 regardless of actual run-time; for board size , the time needed by Bash, for example, was divided by the time needed by C++ for board size . We now get:

Speed Ratio, Linear Scale

We can use a log-scale for the axis to better separate the similar results:

Speed Ratio, Log Scale

We see three very strange things. The first is Bash shooting up wildly. The second is that C# relative time goes down with the increasing board size. The third is that before , the results are fluctuating. The first is easily explained: Bash is incredibly slow. About 10,000 times as slow as the optimized compiled C++ version. It is so slow that the two last data points are estimated as it would have taken an extra week, or so, to get them. The second anomaly needs a bit more analysis. Looking at the raw timing data, we can see that the C# version seems to be needing an extra 7ms regardless of board size. I do not know where that comes from, as the timings do not time things such as program load and initialization, but only the solving of the puzzle itself. It may well be the JIT that takes a while to figure out that the recursive function is expensive and the run-time compilation takes 7ms? Anyway, would we remove this mysterious extra 7ms, the odd behaviour of the C# implementation would vanish. The third anomaly is easily explained by the granularity of the timer and the operating system; up to puzzle size of 8, the times are much less than 100 μs for most implementations. Suffice to move the mouse and generate a couple of interrupts to throw timing off considerably.

Removing the small board sizes yields:

Speed Ratios, Log Scale, Without Outliers

which shows that the respective implementations are well behaved (except for C# and its most probably JIT-related extra 7ms).

So we see that Bash is about 10,000 times slower than the optimized C++ version. The constant-propagated version is only a bit (~5%) faster than the generic C++ version. The Java version takes about 1.5× the time of the C++ version, which is way better than I expected—don’t forget that this is a native-code version of the Java program, it doesn’t run on the JVM. The C# version is twice as slow as the C++ version, which is somewhat disappointing but not terribly so. The Python versions are 120× (for the more pythonesque version) and 200× slower. That’s unacceptably slow, especially that the Python programs aren’t particularly fancy nor complex. We do see that using pythonesque idioms yields a nice performance improvement—40%—but that’s still nowhere useful.

*
* *

So what does this tell us? For one thing, that Bash is slow. But that Bash is slow even when it doesn’t use any external command should not come as a surprise. From what I understand from Bash, data structures are limited to strings and arrays. Lists and strings are the same. Basically, a list is merely a string with items separated by the IFS character(s), which causes all array-like accesses to lists to be performed in linear time as each time the string is reinterpreted given the current IFS. So even though a construct such as ${x[i]} looks like random-access, it is not. As for explicitly constructed arrays (as opposed to lists), there seems to be a real random-access capability, but it’s still very slow. I do not think that bash uses something like a virtual machine and an internal tokenizer to speed up script interpretation. Maybe that’d be something to put on the to-do list for Bash 5? In any case, I also learnt that Bash is a lot more generic than I thought.

The other thing is that Python is not a programming language for compute-bound problems. This makes me question how far projects such as Pygame (which aims at developing a cromulent gaming framework for Python) can go. While all of the graphics and sound processing can be off-loaded to third party libraries written in C or assembly language and interacting with the platform-specific drivers, the central problem of driving the game itself remains complete. How do you provide strong non-player characters/opponents when everything is 100× slower than the equivalent C or C++ program? What about strategy games? How can you build a MMORPG with a Python engine? Is a MMORPG I/O or compute bound? Could you write a championship-grade chess engine in Python?

My guess is that you just can’t.

And that’s quite sad because I like Python as a programming language. I used it in a number of (I/O-bound) mini-projects and I was each time delighted with the ease of coding compared to C or C++ (for those specific tasks). It pains me that Python is just too slow to be of any use whatsoever in scientific/high-performance computing. I wish for Python 4 to have a new virtual machine and interpreter to bring Python back with Java and C#, performance-wise. Better yet, why not have a true native compiler like gcj for Python?

*
* *

I am fully aware that the queens on a board puzzle is a toy problem of sorts. But its extreme simplicity and its somewhat universal backtracking structure makes it an especially adequate toy problem. If a language can’t handle such a simple problem very well, how can we expect it to be able to scale to much more complex problems like, say, a championship-level chess engine?

*
* *

The raw data (do not forget that the two last timings for Bash are estimated). All times are in seconds.

The More Pythonesque version:

#!/usr/bin/python -O
# -*- coding: utf-8 -*-

import sys
import time

########################################
##
##   (c) 2009 Steven Pigeon (pythonesque version)
##

diag45=set()
diag135=set()
cols=set()
solutions=0

########################################
##
## Marks occupancy
##
def mark(k,j):
    global cols, diag45, diag135
    cols.add(j);
    diag135.add(j+k)
    diag45.add(32+j-k)

########################################
##
## unmarks occupancy
##
def unmark(k,j):
    global cols, diag45, diag135
    cols.remove(j);
    diag135.remove(j+k)
    diag45.remove(32+j-k)

########################################
##
## Tests if a square is menaced
##
def test(k,j):
    global cols, diag45, diag135
    return not((j in cols) or \
        ((j+k) in diag135) or \
        ((32+j-k) in diag45))

########################################
##
## Backtracking solver
##
def solve( niv, dx ):
    global solutions, nodes
    if niv > 0 :
        for i in xrange(0,dx):
            if test(niv,i) == True:
                mark ( niv, i )
                solve( niv-1, dx)
                unmark ( niv, i )
    else:
        for i in xrange(0,dx):
            if (test(0,i) == True):
                solutions += 1

########################################
##
## usage message
##
def usage( progname ):
    print "usage: ", progname, " <size>"
    print "size must be 1..32"

########################################
##
## c/c++-style main function
##
def main():
    if len(sys.argv) < 2:
        usage(sys.argv[0])
    else:
        try:
            size = int(sys.argv[1])
        except:
            usage(sys.argv[0])
            return

        if (size <= 32) & (size>0):

            start = time.time()
            solve(size-1,size)
            elapsed = time.time()-start

            print "%s %0.6f" % (solutions,elapsed)

        else:
            usage(sys.argv[0])
#
if __name__ == "__main__":
    main()

The other versions can be found here in super_reines.zip. The Bash and C++ versions are somewhat Linux-specific.

² Frédéric Marceau translated the program from C++ to C#. François-Denis Gonthier translated from C++ to Java.

*
* *

So I added a third Python version to the archive. I also benchmarked it. It is quite a bit faster than the original one, and even than the ‘pythonesque’ version. For example:

$ super-reines.py 12
14200 15.966019
$ super-reines-2.py 12
14200 9.478235
$ super-reines-3.py 12
14200 6.845071

So a more ‘functional’ version performs about 2.3× faster than a direct translation from C++. For the same problem size, however, the C++ (fixed) version takes 0.069s… Roughly 100× faster than the 3rd version.

Speed Ratios, Log Scale, Without Outliers, with 3rd Python version

RIA Services (new home page) is a collection of tools and libraries for making Rich Internet Applications, especially line of business applications, easier to develop.  Brad Abrams did a great presentation of RIA Services at MIX 2009 that touches on querying, validation, authentication, and how to share logic between the server and client sides.  Brad also has a huge series of articles (26 as I write this) on using Silverlight and RIA Services to build a realistic application.

I love the concept of RIA Services.  Brad and his team have done a fantastic job of identifying the critical issues for LOB systems and have the right idea to simplify those common data access tasks through the whole pipeline from database to UI controls, using libraries, Visual Studio tooling, or whatever it takes to get the job done.

So before I lay down some heavy criticism of RIA Services, take into consideration that it’s still a CTP and that my scenario pushes the boundaries of what was likely conceived of for this product, at least for such an early stage.

Shared Data Model with WPF & Silverlight Clients

The cause of so much of my grief with RIA Services has been my need to share a data model, and access to a shared database, across WPF as well as Silverlight client applications.  Within the constraints of this situation, I keep running into problem after problem while trying to use RIA Services productively.

The intuitive thing to do is: define a single data model project that compiles to a single assembly, and then reference that in my Silverlight and non-Silverlight projects.  This would be a 100% full-fidelity shared data model.  As long as the code I wrote was a subset of both Silverlight and normal .NET Frameworks (an intersection), we could share identical types and write complex validation and model manipulation logic, all without having to constrain ourselves to work within the limitations of a convoluted code generation scheme.  Back when I wrote Compact Framework applications, I did this with great success despite the platform gap, and I didn’t have anything like RIA Services to help.

Incompatible Assemblies

Part of the problem arises because Silverlight assemblies are incompatible with non-Silverlight assemblies.  A lot of what RIA Services is doing is trying to find a way around this limitation: picking up attributes and code files from one project and inserting that code into the Silverlight project with a build action.  This Visual Studio “magic” has been criticized for its weakness in dealing with multiple-solution systems where Visual Studio can’t update the client because it’s not loaded, and I’ve heard there’s work being done to address this, but for my current needs, this magic aspect of it isn’t a problem.  The specifics of how it works, however, are.

Different Data Access APIs

Accessing entities requires a different API in Silverlight via RiaContextBase versus ObjectContext elsewhere.  Complex logic in the model (for validation and other actions against the model) requires access to other entities and therefore access to the current object context, but the context APIs for Silverlight and WPF are very different.  Part of this has to do with Silverlight’s inability to make synchronous calls to the server.

In significantly large systems that I build, I use validation logic such as “this entity is valid if it’s pointing to an entity of a different type that contains a PropertyX value of Y”.  One of my tables stores a tree of data, so I have methods for loading entire subtrees and ensuring that no circular references exist.  For these kinds of tasks, I need access to the data context in basic validation methods.  When I delete nodes from a tree, I need to delete child nodes, so update logic is part of the model that needs to be the same in every client.  I don’t want to define that multiple times for multiple clients.  I like to program very DRY.  In other words, I find myself in need of a shared model.

RIA Services doesn’t provide anything like type equivalence for a shared model, however.  Data model classes in Silverlight inherit from Entity, but EntityObject in WPF.  In the RIA Services domain context, we RaiseDataMemberChanged, but in a normal EF object context, we need to ReportPropertyChanged.  In WPF, I can call MyEntity.Load(MergeOption.PreserveChanges), but in Silverlight there’s no Load method on the entity and no MergeOption enum.  In WPF I can query against context.SomeEntitySet, but in Silverlight you would query against context.GetSomeEntitySetQuery() and then execute the query with another method call.

This chasm of disparity makes all but the simplest shared model logic impractical and frustrating.  The code generation technique, though good in principle, keeps getting in the way.  For example, I have both parameterless and parameterized constructors in my entity classes.  This works great in my WPF client, but when this code is synchronized to my Silverlight client, I get an error because the Silverlight-side entity class is generated in two parts: in the hidden partial class, a parameterless constructor is generated which calls partial method OnCreated; and in the visible partial class, the constructor method I defined on the server is dumped into another file, so I have duplicate constructors.  If I remove the parameterless constructor from the server side, I get an error because my entity class requires a parameterless constructor (and defining a non-default constructor effectively removes the default one from the resulting type unless it’s explicitly defined).  I thought I could define the partial method OnCreated and put my construction logic in there, but the partial method is only defined on the client side.  That means sharing construction logic consists of copying and pasting the OnCreated method across the various clients—far from an ideal solution.

Entity Data Model Required to be in Web Project

Another strategy I attempted was to define the .edmx file and my partial class extensions in a class library, and then reference that from the web project.  I could define the LinqToEntitiesDomainService<MyDataContext>, but sharing entity class code (by generating code in the Silverlight project) isn’t possible unless the .edmx file and partial class extensions are defined in the web project itself.  This would mean that my WPF client would have to reference a web project for data access, which by itself seems wrong.  (Or making a copy of the data model, which is worse.)  It would be better for the WPF client to talk to the same domain service as the Silverlight client, but RIA Services doesn’t give you an option to link that web project to a non-Silverlight project, so again I ran into a brick wall.

So Don’t Do That

The kind of advice I’m getting for this is, “so don’t do that”.  In other words, don’t write complex validation logic in the model or otherwise try to access the data context; don’t write parameterized constructors; don’t aim for 100% type fidelity across all endpoints of a system; don’t try to share data models with Silverlight and non-Silverlight projects, etc.  But I see the potential for RIA Services, so I have to push for these things unless I hear really convincing arguments against them (or compelling alternatives).

Conclusion

The fact that there are different data contexts and data item definitions within those contexts imposes a burden on the developer to use different techniques for each environment, and creates challenges for centralizing data model logic and reusing equivalent logic across different kinds of clients.  My gut feeling is that RIA Services in its current form has some fundamental design flaws that will need to be addressed, taking into consideration systems with a mix of Silverlight, WPF, and other clients, before it becomes a truly robust data access platform.

Somewhat ironically, we underwent some serious technical problems in CS today.  At least more serious than the ones we typically undergo, which involve Jeannie’s computer being unhappy and everyone complaining about the wireless network.  (Purple Air, why do you hate my iPhone?)

Today’s problems involved the projector, which refused to project anything but a blue screen of death.  We looked through the menus and were particularly attracted to one option that depicted a smiling face gazing serenely at a screen.  But alas, it did no good.

See, being in CS classes does NOT imply technical know-how.  The only thing that implies technical know-how is being on the staff in OIT, whom we called for assistance.  Fortunately, with their help everything was running smoothly within 20 minutes, so Jeannie could show us photos of cacti and kings of England as was her plan.

After that was calculus, where our professor is diligently training us to become gamblers.

English, however, would have to be my favorite class of the day.  Because we listened to rap and watched music videos and got to hear our very academic and soft-spoken professor say “shit” (“Yes, I do know that word”).

But moreover, because it took a good portion of the hour to figure out how to operate the projector/screen/computer/magical techiness system in the lecture hall– which admittedly was a little complicated but really shouldn’t have required the full mental capacities of ten college students working on it.

… But okay, mostly because we transitioned from Whitman to Jay-Z in a single hour  I later told Alex about this unusual class and he called me a “closet thug,” which I think is a description that anyone who truly knows me would have to agree with.

In other news, yay it’s the weekend!

This post was made possible by two Milky Way bars and one Snickers.

Programmers aren’t always the very rational beings they please themselves to believe. Very often, we close our eyes and take decisions based on what we think we know, and based on what have been told by more or less reliable sources. Such as, for example, taking red-black trees rather than AVL trees because they are faster, while not being able to quite justify in the details why it must be so. Programming using this kind of decision I call cargo cult programming.

Originally, I wanted to talk about red-black vs. AVL trees and how they compare, but I’ll rather talk about the STL std::map that is implemented using red-black trees with G++ 4.2, and std::unordered_map, a hash-table based container introduced in TR1.

TR1 std::unordered_map is a map that does not maintain any particular order between the keys of the data it contains. It is therefore implemented as a hash table. The std::map is implemented, at least in G++ 4.2, as a red-black tree. Self-balancing trees change their shape with each insertion in order to maintain most leaves at an equal depth, ensuring an access time. The difference between an AVL tree and a red-black tree is the way the tree is rebalanced, and the red-black tree does about half as much operations as an AVL tree on insertion, making somewhat faster—how much faster remains to be quantified.

So I ran a few simple experiments to compare std::unordered_map and std::map. First, I got hold of the Zingarelli word list, containing some 585,000 Italian words (I can’t find a working URL, but I got the list a few years ago on a Scrabble-related page). I ran three tests: insertion, successful search, and failed search. The list is broken into two parts, one containing 90% of the words, to be inserted, and 10% of the words where kept for the unsuccessful search test.

The insertion test consisted into inserting all of Zingarelli’s list into both data structures, in the most simplistic way: scanning the list sequentially and inserting items one by one.

Insertion times are about the same; so we cannot conclude anything special here except maybe that the insertion time is largely dominated by memory allocation and copy. For the successful search (10,000 tries in each case):

Map look-up is immensely faster with std::unordered_map, a ratio of about 8:1! Failed searches exhibit the same behavior. For ~60,000 failed searches:

We see the same kind of differences here again, but the failed searches are much slower than the successful searches.

When we repeat the experiment with integers (with the same kind of numbers; 500,000 integers of which 10% are randomly chosen for the failed searches), we get essentially the same picture but a massive speedup as strings are rather costly to copy. Indeed, for the insertion:

Now, we see the algorithmic difference between the two algorithms as the time spent allocating and copying strings is eliminated. For successful and failed searches, we get (again for 10,000 look-ups):

and:

The std::unordered_map therefore seems to be much faster than std::map. What have we lost in order to gain this speed? First, a lot of memory as hash table must retain a certain sparseness to sport their average constant-time look-ups; and the ordering of keys. Listing the items sorted in a hash map won’t produce an ordered list, but rather a randomized version of the list. std::map on the other hand, allow lexicographic enumeration of its contents.

*
* *

So you’re reading this and are thinking, mmppfh, of course, it’s a hash table, you nitwit. Well, yes. Maybe so.

But here’s what prompted me to do the test. Red-black trees offer access. But that’s assuming (and quite wrongly) that comparing keys can be performed in constant time. This may be true for simple keys, such as machine-size integers (known in C and C++ as int) but not so for more complex data, like strings and other structures. For string, for example, comparison may still be very fast because the cost of comparing two string only depends on the longest common prefix; if two long strings have differences in the first few characters, comparison terminates rapidly and the cost is moderate. If, on the other hand, the string share a very long prefix, then the comparison algorithm must scan both strings until the end is reached or a difference is found; this can be very long. So, lets be the average common prefix length. The expected search time is now which can grow large if is large.

For a hash table look-up, you must first compute the hash key which can be at best done in linear time in the average key length. In our first case, this length is the average string length. Let this average length be . The number of probe is a small constant that depends on the sparseness of the table and the number of items, , it contains. For each of those probes, a string comparison is performed, leading to an expected complexity of .

Now, it is not clear when . Both cancel, which boils down to . So, in some conditions, it may be possible to make the tree appear faster than the hash table.

On the average, however, the hash map should win becase we expect .

*
* *

A cargo cult is described as:

A cargo cult is a type of religious practice that may appear in primitive tribal societies in the wake of interaction with technologically advanced, non-native cultures. The cults are focused on obtaining the material wealth of the advanced culture through magical thinking, religious rituals and practices, believing that the wealth was intended for them by their deities and ancestors.

Except from the part with ancestor spirits but especially when considering magical thinking, cargo culting applies very often to how programmers write code and take decisions about data structures and algorithms. Choosing a red-black tree over an AVL, or over a splay tree, because we think that it is somehow always better—sometimes because Ancestor X (a more or less reliable source or authority, such as a more experience programmer, a teacher, or some Internet dude) said so—is a form of cargo cult where the programmer does not use rationality to its full extent to make a decision.

When choosing data structures, one must be fully aware of the dual cost of data structures. The first cost is the run-time cost. Theoretical complexity and actual implementation-dependant run-times may be quite different. The second cost is memory usage. Memory is large on modern systems but not infinite. A particularly wasteful method that offers constant-time access to the data may use as much as, say, ten times the memory used by a method that gives you access. If for small data set this 10× memory usage pose no particular problem, it may be quite different with a largish data set. It’s all a very delicate balancing act between run-time performance and scalability.

It’s very hard to trust one’s gut feelings about a data structure and the data put in. Combined with the access patterns, the data structure may yield a very counter-intuitive performance. Rather than giving into magic thinking cargo cult programming, you should always take a little time to validate our assumptions and hypotheses about the data, the data structure, and the access patterns, as data structure behavior is clearly not independent from the data and the access patterns.

Consider this very simple example. We have a simple binary tree and a list of strings. Binary trees offer insertion and access times, so it is, a priori, a good choice. Now, the list of string is composed of words, but it so happens it’s already sorted. If we insert the strings in the order they are in the list, we get a degenerate binary-tree that’s in fact a list! Indeed so: insertions are always performed at the far right of the tree, causing the tree to degenerate into something we could call a vine and then all operations degenerate to linear time. If we randomize the list and insert the words into the tree in that randomized order, we get a ragged tree, but about equally deep everywhere, leading to the expected access and insertion time.

The sorted list / binary tree example is an over-simplistic one, I admit, but it gets to the point: the programmer used a data structure cargo-culted to offer access time, but due to his incomplete comprehension of the data (the sorted list) and of the access pattern (inserting items sequentially from the list), the result was disastrous.

But the thing is, we all do that to a certain extent!

Borahan hocanın derste takip ettiği slaytlar (pdf olarak) :

http://ifile.it/wsjzg4r

Dersin kitabı McGraw-Hill – Introduction to Algorithms 2ed Cormen ( Analysis of Algorithms dersinde de kullanılacak ) :

http://ifile.it/023p51u

Eski sınav soruları ve quizler :

http://ifile.it/76yupo0

İlk proje (son gün 16 Ekim) :

http://ifile.it/tokur2p

I’m currently hanging in Paresky till the rain lets up. Goddammit. I had my umbrella with me all day to ensure it wouldn’t rain, and Emily had hers too so the weather would in fact be absolutely lovely.  (Apparently, the effects of umbrellas on the weather are cumulative.)  And so it came to pass.

But it was so nice and sunny when I headed out to practice piano that I figured surely, surely I wouldn’t need an umbrella. Will I ever learn…

Anyway, I’d like to preface the remainder of this post with the disclaimer that not all my days are so thoroughly wonderful as you may believe from reading this blog; I just always feel like writing when I’ve had a thoroughly wonderful day.

So in CS I was telling Katie a little about my program, including my hopes to have it generate compilable code from its own code and thus evolve into sentience. Katie apparently found this worth drawing our professor’s attention to with the following exclamation: “Hey Jeannie, we’ve got a megalomaniac here!”

I protested that it’s almost certainly every programmer’s dream to write a program that could eventually surpass its creator in intelligence and consequently (of course) take over the world. But evidently that’s not the case.  What?  What’s everyone else doing in CS classes then?

Then there was calculus. Multivariable thus far has been a kind of hybrid of precalc, calc, and linear (a little). If I had a professor less fantastic than Adams– or alternatively, if I remembered more than the scantest remnants of precalc, calc, or linear– this would be torture.

Fortunately, I have a professor who can core an apple with his bare hands, AKA the most badass math professor in the history of the profession.  I mean, I knew Adams was friendly and funny and brilliant, but I had no idea he was so fucking badass.  I think he might be superhuman.

The other thing worth mentioning at this point is the puppy chow that Taylor, Annie, and I made for snacks last night.  I hadn’t heard of puppy chow before this, so let me explain. Take Chex cereal, melted chocolate, peanut butter, and powdered sugar. Put them in a garbage bag.  Shake violently.

Bring trash bag to common room and invite entry to reach in and see what mysteries await them. Be amused at their skepticism.  Feel happy when they realize just how awesome the contents are.  Snack on the plentiful leftovers whenever you go into the common room.  Bring a bowl of leftovers to your room to snack on intermittently (or rather constantly). Decide this is either the best or the worst idea you’ve ever had.

Ooh, rain’s stopped.  Briefly.

Because after classes today (more on that in a moment), I walked all the way back to the 1914 Library, in the rain, to get the books I’d bought with the voucher.  And when I got there, not only was the line negligible, but on a whim with no expectation of satisfaction, I asked if there were any multivariable calc texts that just magically materialized during the past couple days.

And I GOT ONE.  It was a miracle of epic proportions.  I walked back to my dorm with a heavy, heavy bag.  A bag heavy with KNOWLEDGE.  Sweet, papery knowledge.  Which sounds like some kind of pastry.

Okay, time to talk about classes.

Intro to Logic and Semantics: My first college class (besides Linear Algebra at Cornell, which I’m going to discount because it kind of sucked) could not have been better.  It was like Steven Pinker in a class form.

I hope the full and incredible import of this fact does not escape you.  Formal logic and linguistics are things that I read about on my own for fun.  The notion of being able to take a class on these subjects and get tests and grades and credits for it is not something that our public school system prepared me for.  Of course in high school we could choose some classes, and of course I honestly loved many classes I took in high school.  But this is totally, completely, and wonderfully different.

I guess what I’m saying is, if you’re not yet in college, get there ASAP because it rocks.

It happens that I know tons of people in my class, where “tons” is equivalent to three (and counting).  Hey, that’s a lot if you’re a frosh.  Professor Sanders is also hilarious and amazing, but I knew that already because I sat in on one of his courses back in September ‘08.  Which reminds me: I need to send a wedding gift for the chair and his intelligence, plus some dandelions for the unicorn in his garden.

Data Structures and Advanced Programming: I was actually fairly scared walking into this class because of my deep dark secret: I have done absolutely no programming since the topological hell of yesteryear.  I know, it’s shameful.  Fortunately we’re starting with some Java review, although what we really started with today was playing Boggle.  Fun times.

Jeannie is a very straight forward and cool professor, about whom I’ve heard only good things.  Which is pretty typical for Williams profs, as the same applies to Sanders and Adams (more about the latter anon).  When she was taking attendance and came to my name, she asked if I was “Antal’s buddy,” which I found highly amusing.  Antal is ubiquitously known in the CS department here, and I’m not surprised.

Multivariable Calculus: Ultimately, I’m really glad I got kicked out of Cornell’s multivariable class, even though it sucked at the time.  Because my professor here is Colin Adams.  And multivariable is, according to him, the funnest class offered at Williams.

I’m going to try really, really, really hard to not be a giddy fangirl, because dude, Adams wrote The Knot Book!  Plus other books!  He spent our first class talking about mathematical beauty and then knots, which is basically what guys should talk about if they want me to fall in love with them.  (Bonus points if they mention Gödel.)  (Or muffins.)

The class is huge (liberal arts college-huge, that is, so like 40-50 people), but I have a couple friends there, and I don’t think I’ll get lost in the crowd too much.  And if I feel at risk of doing so, I’ll just go to all of my professor’s office hours and talk to him about knots, and maybe calculus too because that’s also fun.

Quite frankly, the math department is probably the number one biggest reason I chose Williams.  Needless to say, I am excited about multivariable.

Poetry and the City: I actually could still get kicked out of this class, because it’s a 200-level gateway English class and I’m a first-year who’s probably not going to be an English major.

But my silly AP Lit exam score means I basically can’t take any 100-level English classes (they’re mostly full at this point anyway), and I do think it’s important for me to take a writing class my first semester here.  Who would’ve thought doing well on an AP would cause problems?

The professor here does not radiate enthusiasm and awesomeness like my others do, but I can tell she’s still really good, knows her stuff, and is impressively open to having students talk to her outside of class.  Her office hours actually take place in Paresky snack bar.

One slight downside to this class is that I know absolutely no one in it, but it’s very discussion-based so I’m sure I’ll get to know people.  And even though I’m not a very English-y person, I actually really enjoy talking about poetry, so I think I’m going to like this class.  In addition to all my other ones.

In other news, this is the first time it’s rained since I got to Williams.

Productivity Enhancing Tools

Visual Studio has come a long way since its debut in 2002.  With the imminent release of 2010, we’ll see a desperately-needed overhauling of the archaic COM extensibility mechanisms (to support the Managed Package Framework, as well as MEF, the Managed Extensibility Framework) and a redesign of the user interface in WPF that I’ve been pushing for and predicted as inevitable quite some time ago.

For many alpha geeks, the Visual Studio environment has been extended with excellent third-party, productivity-enhancing tools such as CodeRush and Resharper.  I personally feel that the Visual Studio IDE team has been slacking in this area, providing only very weak support for refactorings, code navigation, and better Intellisense.  While I understand their desire to avoid stepping on partners’ toes, this is one area I think makes sense for them to be deeply invested in.  In fact, I think a new charter for a Developer Productivity Team is warranted (or an expansion of their team if it already exists).

It’s unfortunately a minority of .NET developers who know about and use these third-party tools, and the .NET community as a whole would without a doubt be significantly more productive if these tools were installed in the IDE from day one.  It would also help to overcome resistance from development departments in larger organizations that are wary of third-party plug-ins, due perhaps to the unstable nature of many of them.  Microsoft should consider purchasing one or both of them, or paying a licensing fee to include them in every copy of Visual Studio.  Doing so, in my opinion, would make them heroes in the eyes of the overwhelming majority of .NET developers around the world.

It’s not that I mind paying a few hundred dollars for these tools.  Far from it!  The tools pay for themselves very quickly in time saved.  The point is to make them ubiquitous: to make high-productivity coding a standard of .NET development instead of a nice add-on that is only sometimes accepted.

Consider just from the perspective of watching speakers at conferences coding up samples.  How many of them don’t use such a tool in their demonstration simply because they don’t want to confuse their audience with an unfamiliar development interface?  How many more demonstrations could they be completing in the limited time they have available if they felt more comfortable using these tools in front of the masses?  You know you pay good money to attend these conferences.  Wouldn’t you like to cover significantly more ground while you’re there?  This is only likely to happen when the tool’s delivery vehicle is Visual Studio itself.  Damon Payne makes a similar case for the inclusion of the Managed Extensibility Framework in .NET Framework 4.0: build it into the core and people will accept it.

The Gorillas in the Room

CodeRush and Resharper have both received recent mention in the Hanselminutes podcast (episode 196 with Mark Miller) and in the Deep Fried Bytes podcast (episode 35 with Corey Haines).  If you haven’t heard of CodeRush, I recommend watching these videos on their use.

• CodeRush Express (which is free)
• CodeRush with Refactor!
• DXCore (the library you can use to build your own add-ins, and on which CodeRush and Refactor! are built)

For secondary information on CodeRush, DXCore, and the principles with which they were designed, I recommend these episodes of DotNetRocks:

• Tool Development
• Discoverability
• The Science of Good UI

I don’t mean to be so biased toward CodeRush, but this is the tool I’m personally familiar with, has a broader range of functionality, and it seems to get the majority of press coverage.  However, those who do talk about Resharper do speak highly of it, so I recommend you check out both of them to see which one works best for you.  But above all: go check them out!

Refactor – Rename

Refactoring code is something we should all be doing constantly to avoid the accumulation of technical debt as software projects and the requirements on which they are based evolve.  There are many refactorings in Visual Studio for C#, and many more in third-party tools for several languages, but I’m going to focus here on what I consider to be the most important refactoring of them all: Rename.

Why is Rename so important?  Because it’s so commonly used, and it has such far-reaching effects.  It is frequently the case that we give poor names to identifiers before we clearly understand their role in the “finished” system, and even more frequent that an item’s role changes as the software evolves.  Failure to rename items to accurately reflect their current purpose is a recipe for code rot and greater code maintenance costs, developer confusion, and therefore buggy logic (with its associated support costs).

When I rename an identifier with a refactoring tool, all of the references to that identifier are also updated.  There might be hundreds of references.  In the days before refactoring tools, one would accomplish this with Find-and-Replace, but this is dangerous.  Even with options like “match case” and “match whole word”, it’s easy to rename the wrong identifiers, rename pieces of string literals, and so on; and if you forget to set these options, it’s worse.  You can go through each change individually, but that can take a very long time with hundreds of potential updates and is a far cry from a truly intelligent update.

Ultimately, the intelligence of the Rename refactoring provides safety and confidence for making far-reaching changes, encouraging more aggressive refactoring practices on a more regular basis.

Abolishing Magic Strings

I am intensely passionate about any tool or coding practice that encourages refactoring and better code hygiene.  One example of such a coding practice is the use of lambda expressions to select identifiers instead of using evil “magical strings”.  From my article on dynamically sorting Linq queries, the use of “magic strings” would force me to write something like this to dynamically sort a Linq query:

Customers = Customers.Order("LastName").Order("FirstName", SortDirection.Descending);

The problem here is that “LastName” and “FirstName” are oblivious to the Rename refactoring.  Using the refactoring tool might give me a false sense of security in thinking that all of my references to those two fields have been renamed, leading me to The Pit of Despair.  Instead, I can define a function and use it like the following:

public static IOrderedEnumerable<T> Order<T>(this IEnumerable<T> Source,
    Expression<Func<T, object>> Selector, SortDirection SortDirection)
{
    return Order(Source, (Selector.Body as MemberExpression).Member.Name, SortDirection);
}

Customers = Customers.Order(c => c.LastName).Order(c => c.FirstName, SortDirection.Descending);

This requires a little understanding of the structure of expressions to implement, but the benefit is huge: I can now use the refactoring tool with much greater confidence that I’m not introducing subtle reference bugs into my code.  For such a simple example, the benefit is dubious, but multiply this by hundreds or thousands of magic string references, and the effort involved in refactoring quickly becomes overwhelming.

Coding in this style is most valuable when it’s a solution-wide convention.  So long as you have code that strays from this design philosophy, you’ll find yourself grumbling and reaching for the inefficient and inelegant Find-and-Replace tool.  The only time it really becomes an issue, then, is when accessing libraries that you have no control over, such as the Linq-to-Entities and the Entity Framework, which makes extensive use of magic strings.  In the case of EF, this is mitigated somewhat by your ability to regenerate the code it uses.  In other libraries, it may be possible to write extension methods like the Order method shown above.

It’s my earnest hope that library and framework authors such as the .NET Framework team will seriously consider alternatives to, and an abolition of, “magic strings” and other coding practices that frustrate otherwise-powerful refactoring tools.

Refactoring Across Languages

A tool is only as valuable as it is practical.  The Rename refactoring is more valuable when coding practices don’t frustrate it, as explained above.  Another barrier to the practical use of this tool is the prevalence of multiple languages within and across projects in a Visual Studio solution.  The definition of a project as a single-language container is dubious when you consider that a C# or VB.NET project may also contain HTML, ASP.NET, XAML, or configuration XML markup.  These are all languages with their own parsers and other language services.

So what happens when identifiers are shared across languages and a Rename refactoring is executed?  It depends on the languages involved, unfortunately.

When refactoring a C# class in Visual Studio, the XAML’s x:Class value is also updated.  What we’re seeing here is cross-language refactoring, but unfortunately it only works in one direction.  There is no refactor command to update the x:Class value from the XAML editor, so manually changing it causes my C# class to become sadly out of sync.  Furthermore, this seems to be XAML specific.  If I refactor the name of an .aspx.cs class, the Inherits attribute of the Page directive in the .aspx file doesn’t update.

How frequent do you think it is that someone would want to change a code-behind file for an ASP.NET page, and yet would not want to change the Inherits attribute?  Probably not very common (okay, probably NEVER).  This is a matter of having sensible defaults.  When you change an identifier name in this way, the development environment does not respond in a sensible way by default, forcing the developer to do extra work and waste time.  This is a failure in UI design for the same reason that Intellisense has been such a resounding success: Intellisense anticipates our needs and works with us; the failure to keep identifiers in sync by default is diametrically opposed to this intelligence.  This represents a fragmented and inconsistent design for an IDE to possess, thus my hope that it will be addressed in the near future.

The problem should be recognized as systemic, however, and addressed in a generalized way.  Making individual improvements in the relationships between pairs of languages has been almost adequate, but I think it would behoove us to take a step back and take a look at the future family of languages supported by the IDE, and the circumstances that will quickly be upon us with Microsoft’s Oslo platform, which enables developers to more easily build tool-supported languages (especially DSLs, Domain Specific Languages). 

Even without Oslo, we have seen a proliferation of languages: IronRuby, IronPython, F#, and the list goes on.  A refactoring tool that is hard-coded for specific languages will be unable to keep pace with the growing family of .NET and markup languages, and certainly unable to deal with the demands of every DSL that emerges in the next few years.  If instead we had a way to identify our code identifiers to the refactoring tool, and indicate how they should be bound to identifiers in other languages in other files, or even other projects or solutions, the tools would be able to make some intelligent decisions without understanding each language ahead of time.  Each language’s language service could supply this information.  For more information on Microsoft Oslo and its relationship to a world of many languages, see my article on Why Oslo Is Important.

Without this cross-language identifier binding feature, we’ll remain in refactoring hell.  I offered a feature suggestion to the Oslo team regarding this multi-master synchronization of a model across languages that was rejected, much to my dismay.  I’m not sure if the Oslo team is the right group to address this, or if it’s more appropriate for the Visual Studio IDE team, so I’m not willing to give up on this yet.

A Default of Refactor-Rename

The next idea I’d like to propose here is that the Rename refactoring is, in fact, a sensible default behavior.  In other words, when I edit an identifier in my code, I more often than not want all of the references to that identifier to change as well.  This is based on my experience in invoking the refactoring explicitly countless times, compared to the relatively few times I want to “break away” that identifier from all the code that references.

Think about it: if you have 150 references to variable Foo, and you change Foo to FooBar, you’re going to have 150 broken references.  Are you going to create a new Foo variable to replace them?  That workflow doesn’t make any sense.  Why not just start editing the identifier and have the references update themselves implicitly?  If you want to be aware of the change, it would be trivial for the IDE to indicate the number of references that were updated behind the scenes.  Then, if for some reason you really did want to break the references, you could explicitly launch a refactoring tool to “break references”, allowing you to edit that identifier definition separately.

The challenge that comes to mind with this default behavior concerns code that spans across solutions that aren’t loaded into the IDE at the same time.  In principle, this could be dealt with by logging the refactoring somewhere accessible to all solutions involved, in a location they can all access and which gets checked into source control.  The next time the other solutions are loaded, the log is loaded and the identifiers are renamed as specified.

Language Property Paths

If you’ve done much development with Silverlight or WPF, you’ve probably run into the PropertyPath class when using data binding or animation.  PropertyPath objects represent a traversal path to a property such as “Company.CompanyName.Text”.  The travesty is that they’re always “magic strings”.

My argument is that the property path is such an important construct that it deserves to be an core part of language syntax instead of just a type in some UI-platform-specific library.  I created a data binding library for Windows Forms for which I created my own property path syntax and type, and there are countless non-UI scenarios in which this construct would also be incredibly useful.

The advantage of having a language like C# understand property path syntax is that you avoid a whole class of problems that developers have used “magic strings” to solve.  The compiler can then make intelligent decisions about the correctness of paths, and errors can be identified very early in the cycle.

Imagine being able to pass property paths to methods or return then from functions as first-class citizens.  Instead of writing this:

Binding NameTextBinding = new Binding("Name") { Source = customer1; }

… we could write something like this, have access to the Rename refactoring, and even get Intellisense support when hitting the dot (.) operator:

Binding NameTextBinding = new Binding(@Customer.Name) { Source = customer1; }

In this code example, I use the fictitious @ operator to inform the compiler that I’m specifying a property path and not trying to reference a static property called Name on the Customer class.

With property paths in the language, we could solve our dynamic Linq sort problem cleanly, without using lambda expressions to hack around the problem:

Customers = Customers.Order(@Customer.LastName).Order(@Customer.FirstName, SortDirection.Descending);

That looks and feels right to me.  How about you?

Summary

There are many factors of developer productivity, and I’ve established refactoring as one of them.  In this article I discussed tooling and coding practices that support or frustrate refactoring.  We took a deep look into the most important refactoring we have at our disposal, Rename, and examined how to get the greatest value out of it in terms of personal habits, as well as long-term tooling vision and language innovation.  I proposed including property paths in language syntax due to its general usefulness and its ability to solve a whole class of problems that have traditionally been solved using problematic “magic strings”.

It gives me hope to see the growing popularity of Fluent Interfaces and the use of lambda expressions to provide coding conventions that can be verified by the compiler, and a growing community of bloggers (such as here and here) writing about the abolition of “magic strings” in their code.  We can only hope that Microsoft program managers, architects, and developers on the Visual Studio and .NET Framework teams are listening.