In order to apply paging to collections, I find it useful to use the following extension methods:

   /// <summary>
   /// Apply paging to an IEnumerable.
   /// <typeparam name="TSource">The type that the collection contains.</typeparam>
   /// <param name="source">The data source.</param>
   /// <param name="page">The page index, starting at 0.</param>
   /// <param name="pageSize">The max number of items to return.</param>
   /// <returns>The resulting object collection.</returns>
   public static IEnumerable<TSource> Page<TSource>(this IEnumerable<TSource> source, int? page, int pageSize)
   {
      return source.Skip((page ?? 0) * pageSize).Take(pageSize);
   }

   /// <summary>
   /// Apply paging to an IQueryable.
   /// </summary>
   /// <typeparam name="TSource">The type that the collection contains.</typeparam>
   /// <param name="source">The data source.</param>
   /// <param name="page">The page index, starting at 0.</param>
   /// <param name="pageSize">The max number of items to return.</param>
   /// <returns>The resulting object collection.</returns>
   public static IQueryable<TSource> Page<TSource>(this IQueryable<TSource> source, int? page, int pageSize)
   {
      return source.Skip((page ?? 0) * pageSize).Take(pageSize);
   }

By adding them to an extension class within a certain namespace, all IQueryable and IEnumerable objects automatically receives paging functionality. You will then be able to write (for instance):

      List<string> strings = new List<string> { "a","b", "c","d","e","f","g","h","i","j"};
      strings.Page(2, 2);

And get a list that contains “c” and “d”.

Man, I really must start using a code tool, so that  I do not have to hack the code manually when writing these blogs 

The primary difference is that the extension methods defined for IQueryable<T> take Expression objects instead of Functional objects, meaning the delegate it receives is an expression tree instead of a method to invoke.

IEnumerable<T> is great for working with in-memory collections, but IQueryable<T> allows for a remote data source, like a database or web service.

IEnumerable doesn’t have the concept of moving between items, it is a forward only collection. It’s very minimalistic; something that most any data source can provide. Using only this minimal functionality, LINQ can provide all of these great operators.

IQueryable<T> is a very powerful feature that enables a variety of interesting deferred execution scenarios (like paging and composition based queries).

Hope this helps !!!

Reference Links :

• C# 3.0 and Linq posted by scott
• IQueryable<T> vs. IEnumerable<T> in LINQ to SQL queries by Jon Kruger

Quick tip:  If you are going to composite method calls together in order to make use of multiple method calls to refine the SQL Query to be generated, make sure the calls are made against IQueryable<T> objects and not IEnumerable<T>.

LINQ method calls (Where, Skip, Take, Count, etc) against IQueryable<T> have different extension methods that get called compared to those that get called when called against an IEnumerable<T> argument.

To illustrate this, consider the following design.  I have a model which only does a few things, but to reduce the amount of code duplication I split the query refinements into individual methods.  I have a base method which just selects the data from the table.  Another method which will perform a Where query on data passed in, another which will Page the data passed in, and another which will just count the number of rows in the data passed in.

In other words, I can do this:

IEnumerable<TEntity> GetSearchResults( string search, int page )
{
  IEnumerable<TEntity> results = QueryData();
  results = SearchData( results, search );
  results = PageData( results, page, Constants.ResultsPerPage );

  return results;
}

If these methods take and return IEnumerable<T> bad things start to happen, foremost is that the base query selects everything out of the table.  The Where query is all performed client side rather than being generated into the SQL query, and the same for the paging and counting!

Swap out IEnumerable<T> for IQueryable<T> and things begin to work well though.  The Where query is generating Where clauses in SQL and the same goes for the other methods.  And since IQueryable<T> requires implementers to also implement IEnumerable<T>, at any point in time I can decide to stop chaining calls and revert to IEnumerable<T> so method callers have no idea what the internal implementation of getting that IEnumerable<T> really is.

The only changes I need to make to the code I have above is to change the results variable to IQueryable<T>, as wells as the return type and the parameter type of QueryData, SearchData, and PageData to deal with IQueryable<T>.  The return type of the GetSearchResults method doesn’t need to be changed!

Edit: I tried to use the syntax highlighting feature of WordPress provided by a google code project, but it failed miserably on the generics.

Quite recently, Tim Mallalieu (Program Manager of the LINQ to SQL and Entity Framework Team) made it clear in his blog post that the Entity Framework (LINQ to Entities) will get more focus in the .NET 4.0 build as opposed to “LINQ to SQL” — which I am slowly growing to love by the day (in any case, you don’t want to know what I do by night). Yet, with my disappointment of the possible abandonment of the rather lightweight “LINQ to SQL” elucidation, comes my appreciation for third-party Object/Relational Mapping (ORM) implementations of LINQ.

I was first introduced to SubSonic earlier this year when I worked on the BoomerTowne project. I must admit, I quickly consumed the “Koolaid” with complete admiration. In fact, I feared that SubSonic will die a natural death with the introduction of LINQ to SQL, but my steadfast hope is now renewed.

In a recent discussion with Scott Hanselman (on HanselMinutes), Rob Conery of Subsonic (a popular .NET ORM/code generation tool) mentioned that Subsonic is working on implementing “iQueryable” in the next version of Subsonic. In other words, the next version of SubSonic will be bundled with a LINQ Provider (aka LINQ to SubSonic). NHibernate (another popular ORM tool) already has a LINQ provider (LINQ to NHibernate) which affords developers the flexibility of writing queries using LINQ syntax. I think there are huge opportunities for open-source OR/M tools to step in and create even more lightweight and efficient LINQ providers. If you are already a die-hard SubSonic or NHibernate fan, it makes sense to “stick with the known” while dipping into LINQ.

If you are interested in creating your own LINQ Provider, head over to Kevin Halverson’s post on How to implement IQueryable (Part 1).

I must admit I’ve been a bit disappointed with LINQ to SQL. I went looking for the concept of LINQ batches, that is, the ability to execute more than one query in the same database request. I would have thought that this feature would be highly desirable. Instead of making an SQL request, waiting for all the results, making a second request, waiting for it’s results, then a third request, waiting for it’s results…well, you get my point.

If only you could make a single request and get all the results back in one hit. This is one of the benefits of sql server stored procedures. You can put a number of select statements in the stored procedure, call it from your client, and get all the results back via a DataReader or DataSet. With a DataReader, you get the first result back and then when you’ve finished reading all the rows of that result, you call the NextResult() method and it moves on to the next result and so on. With a DataSet, you can set up a DataAdapter, set it’s select statement to call a stored proc, and then call the Fill method to populate one-to-many DataTables within the DataSet. It’s very efficient, and can return the entire batch very quickly.

I often find implementations of the domain model, where someone has implemented a whole lot of objects in .net that hide the details of the underlying sql calls. Basically, as the developer traverses the object model, extra calls are created to the database. This causes all sorts of unnecessary traffic. In some instances, I’ve seen up to a hundred calls in a Page_Load!

 With web sites, the ultimate goal is to try to only have a maximum of a single request per action. So (caching aside) a Page_Load would only call the database once to retrieve everything it needed to populate the page. After the page is populated, you modify some data and then click an Update button, and that action should perform another single request to the database to update the data and get any further results needed to ensure the state of the page is correct. I often achieve pretty close to this, so all that extra traffic is unnecessary and unwanted.

With LINQ to SQL, you shouldn’t need to call stored procedures just to batch queries – especially for simple ones. I mean, doesn’t that defeat the purpose of getting people to develop in LINQ, if as soon as it gets too hard, they say, nup, go off and use a stored procedure? Well, I’ve come up with a simple solution that does allow you to batch LINQ queries.

Here is an example of the types of queries you can execute using the batching function. Note that this performs a single query to the database, consisting of multiple select statements, and returns three enumerable collections.

  var StocktraderLevelQuery =
   from StocktraderLevel in db.StocktraderLevels
   where StocktraderLevel.ItemID==10
   select Stocktrader;

  var StocktraderFunctionQuery =
   from StocktraderFunction in db.StocktraderFunctions
   select StocktraderFunction ;

  var StocktraderLevelAccessQuery =
   from StocktraderLevelAccess in db.StocktraderLevelAccesses
   select StocktraderLevelAccess;

  var mr = db.GetMultipleResults(StocktraderLevelQuery, StocktraderFunctionQuery, StocktraderLevelAccessQuery);

  StocktraderLevels = mr.GetResult<StocktraderLevel>();
  StocktraderFunctions = mr.GetResult<StocktraderFunction>();
  StocktraderLevelAccesses = mr.GetResult<StocktraderLevelAccess>();

 
To be able to do this, open up your project and add a dbml file. You do this by right-clicking on the project, selecting Add New Item, and then selecting LINQ to SQL classes. Type in your database name (in my case I called it Stocktrader.dbml). Then you use the Server Explorer to add a database connection to that database and drag the tables onto the dbml designer.

Behind the scenes, there is designer generated code, created by the LINQ to SQL custom tool. If you can’t see this file, you need to select “Show all files” from the toolbar. If you open that file, you can see that it’s a partial class. Good, that means we can extend it. So add a new class to the project. I named mine StocktraderExt.cs. Then I added the following code:

using System.Data;
using System.Linq;
using System.Data.Linq;
using System.Data.SqlClient;
using System.Collections.Generic;
using System.Data.Common;

namespace Stocktrader.DBML
{
 public partial class StocktraderDataContext
 {
  private DbParameter CloneParameter(DbParameter src)
  {
   SqlParameter source = (SqlParameter)src;
   SqlParameter destination = new SqlParameter();

   destination.Value = source.Value;
   destination.Direction = source.Direction;
   destination.Size = source.Size;
   destination.Offset = source.Offset;
   destination.SourceColumn = source.SourceColumn;
   destination.SourceVersion = source.SourceVersion;
   destination.SourceColumnNullMapping = source.SourceColumnNullMapping;
   destination.IsNullable = source.IsNullable;

   destination.CompareInfo = source.CompareInfo;
   destination.XmlSchemaCollectionDatabase = source.XmlSchemaCollectionDatabase;
   destination.XmlSchemaCollectionOwningSchema = source.XmlSchemaCollectionOwningSchema;
   destination.XmlSchemaCollectionName = source.XmlSchemaCollectionName;
   destination.UdtTypeName = source.UdtTypeName;
   destination.TypeName = source.TypeName;
   destination.ParameterName = source.ParameterName;
   destination.Precision = source.Precision;
   destination.Scale = source.Scale;

   return destination;
  }

  private SqlCommand CombineCommands(List<DbCommand> commandList)
  {
   SqlCommand batchCommand = new SqlCommand();
   SqlParameterCollection newParamList = batchCommand.Parameters;
   int commandCount = 0;
   foreach (DbCommand cmd in commandList)
   {
    string commandText = cmd.CommandText;
    DbParameterCollection paramList = cmd.Parameters;
    int paramCount = paramList.Count;
    for (int currentParam = paramCount – 1; currentParam >= 0; currentParam–)
    {
     DbParameter param = paramList[currentParam];
     DbParameter newParam = CloneParameter(param);
     string newParamName = param.ParameterName.Replace(“@”, string.Format(“@{0}_”, commandCount));
     commandText = commandText.Replace(param.ParameterName, newParamName);
     newParam.ParameterName = newParamName;
     newParamList.Add(newParam);
    }
    if (batchCommand.CommandText.Length > 0)
    {
     batchCommand.CommandText += “;”;
    }
    batchCommand.CommandText += commandText;
    commandCount++;
   }
   return batchCommand;
  }

  public IMultipleResults GetMultipleResults(params IQueryable[] queryableList)
  {
   List<DbCommand> commandList = new List<DbCommand>();
   foreach (IQueryable query in queryableList)
   {
    DbCommand cmd = this.GetCommand(query);
    commandList.Add(cmd);
   }
   SqlCommand batchCommand = CombineCommands(commandList);
   batchCommand.Connection = (SqlConnection)this.Connection;

   DbConnection cn = this.Connection;
   cn.Open();

   DbDataReader dr = batchCommand.ExecuteReader(CommandBehavior.CloseConnection);
   var mr = this.Translate(dr);
   return mr;
  }

 }
}

What I am doing here is passing a list of IQueryables into the GetMultipleResults method. The LINQ DbContext object exposes a GetCommand method. This enabled me to obtain the DbCommand object that would otherwise be used to execute the LINQ query. So I build up a list of DbCommand objects for all the queries in my parameter list.

Next, I decided to combine all the command objects into a single command (called the batch command). There are three parts to this. Firstly, the CommandText for each command can have parameters specified. In LINQ, they are always the same, starting at @p0 and working their way up, @p1, @p2, @p3 etc.

This isn’t a bad thing, because I can put in a counter that increments for every command that I am adding to the batch command, then substitute the name for a new name that includes my command counter. So @p0 becomes @0_p0 (the @p0 param for the first command). I am also using string.Replace to replace the parameters inside the CommandText, and I am building up a collection of new parameters for my new batch command. Note that this idea is based on the SqlCommandSet class that is hidden inside the SqlDataAdapter class. You could probably use this code as a basis for writing a standard batch command process for executing queries in bulk, if you wanted to.

After combining the DbCommand objects into one enormous command (potentially), I then attach to the DbContext connection, and execute the DbReader with an instruction to close the connection as soon as all data is read. No problem doing that because it’s a batch query – you shouldn’t need the connection afterwards as you’ve done all your queries in the one hit.

Finally, we need an object that has an interface of IMultipleResults. That’s done by another internal method, called Translate. Translate takes a DbDataReader as input, and returns multiple results from the reader.

Note these lines of code in the code section at the top:

  StocktraderLevels = mr.GetResult<StocktraderLevel>();
  StocktraderFunctions = mr.GetResult<StocktraderFunction>();
  StocktraderLevelAccesses = mr.GetResult<StocktraderLevelAccess>();

Basically, these retrieve the results in the order of the LINQ queries you passed to the GetMultipleResults method. And that’s it, you can now perform batch queries with LINQ to SQL.

I have provided the code here. Unfortunately wordpress only supports a few different file types, so here it is as a doc file: linqtosqlbatchqueries

The more I play around with Linq, the more it really appeals to me. It really is amazing how powerful it is, and how flexible it can actually make your applications. “Examples” I hear you cry, “give me examples”. Who am I to deny you when you ask so nicely?

Quite often, when an record is deleted in a database, the application design mandates that the record isn’t physically deleted. Instead, it is given a status of Deleted. This means that retrieval of live records needs to take this status into account. Now, in stored procedures you would add a check for the status in each query that you perform against one of these tables. As you can imagine, this quickly becomes very tedious.

So, how can Linq help to make this situation better? Well one of the tricks that you can employ with Linq is refining queries, so you start with a general query and then refine it until you get to a specific query. So, how does this work in practice? Well, take a look at the following query to get all of the records from a table (cunningly called MyTable in this example):

protected virtual IQueryable<MyTable> GetAll()
{
  IQueryable<MyTable> query = from myTable in context.MyTable // The context is the DataContext.
     select myTable;
  return query;
}

Now, as you can see, this query retrieves all of the records from MyTable regardless of status. However, we’ve already stated that we want a query that retrieves records that have been marked as Deleted (in MyTable, there’s a bit field called IsDeleted that states whether or not the record has been deleted). It turns out that it’s really easy to refine these queries. In the next example, we are going to provide methods that refine the original query further and further. We want a method that gets active products, and another method that gets active customers.

protected virtual IQueryable<MyTable> GetAllLiveItems()
{
  IQueryable<MyTable> query = GetAll();
  query = from myTable in query
          where myTable.IsDeleted == false
          select myTable;
}  public List<MyTable> GetActiveProducts(string name)
{
  IQueryable<MyTable> query = GetAllLiveItems();
  query = from myTable in query
          where myTable.ProductName == name
          select myTable;
  return query.ToList<MyTable>();
}  public List<MyTable> GetActiveCustomers(string name)
{
  IQueryable<MyTable> query = GetAllLiveItems();
  query = from myTable in query
          where myTable.CustomerName == name
          select myTable;
  return query.ToList<MyTable>();
}

I know, the table normalisation sucks big time, but this is intended to highlight how the queries have been refined. As you can see, the GetAllLiveItems method refines the original GetAll query so that it retrieves records that haven’t been deleted. The GetActiveCustomers and GetActiveProducts queries take the query from GetAllLiveItems and refine it further still to return the appropriate list. It’s this ability to refine and extend broad stroke queries that, in my mind, opens Linq up as a real rapid database application development tool.

Sorry for the delay in continuing with the discussions on regular expressions, but I got a little sidetracked with the upgrade to Visual Studio 2008 (.NET 3.5), and all the “goodies” that it brings to you. For the moment I’m going to be putting the discussion on regular expressions to one side. I will get back to them but, for the moment, I want to take you on a journey into .NET 3.5.

Possibly the biggest headline in .NET 3.5 is the introduction of Linq. I have to admit that I was highly skeptical about how useful it will actually be in the “real world”, but having played around with it for a little while I have to admit that I’m actually quite impressed. It’s a well thought out area that actually does do a lot to increase productivity. Now, in this entry I’m not going to delve into the internals of Linq and how to use it. Instead, I’m going to talk about a feature of .NET 3.5 and how it can be used to “extend” the behaviour of Linq. Specifically, we’re going to look at how to use Extension Methods to add the ability to return a DataSet and DataTable from Linq in the same way you’d return a generic list.

The following code shows what you actually need to do to extend the IQueryable object in Linq. For the moment, don’t worry about what IQueryable actually does – once you’ve used Linq for a little while it becomes apparent what this interface is for, and where it fits into the big Linq picture.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Data.Linq;
using System.Data.Linq.Mapping;
using System.Reflection;
using System.Data.Common;
using System.Data;
using System.Data.SqlClient; namespace LinqTesting
{
 /// <summary>
 /// This class contains miscellaneous extension methods for Linq to Sql.
 /// </summary>
 public static class LinqDataExtensions
 {
  /// <summary>
  /// Add the ability to return a DataTable based on a particular query
  /// to a queryable object.
  /// </summary>
  /// <param name="extenderItem">The IQueryable object to extend.</param>
  /// <param name="query">The query to execute.</param>
  /// <param name="tableName">The name of the datatable to be added.</param>
  /// <returns>The populated DataTable.</returns>
  public static DataTable ToDataTable(this IQueryable extenderItem,
    DbCommand query,
    string tableName)
  {
    if (query == null)
    {       throw new ArgumentNullException("query");
    }
    SqlCommand cmd = (SqlCommand)query;
    SqlDataAdapter adapter = new SqlDataAdapter();
    adapter.SelectCommand = cmd;
    DataTable dt = new DataTable(tableName);
    try
    {
    cmd.Connection.Open();
    adapter.Fill(dt);
    }
    finally
    {
    cmd.Connection.Close();     }
    return dt;
  }
  /// <summary>
  /// Add the ability to return a DataSet based on a particular query
  /// to a queryable object.
  /// </summary>
  /// <param name="extenderItem">The IQueryable object to extend.</param>
  /// <param name="query">The query to execute.</param>
  /// <param name="tableName">The name of the DataTable to be added.</param>
  /// <returns>The populated dataset.</returns>
  public static DataSet ToDataSet(this IQueryable extenderItem,
    DbCommand query,
    string tableName)
  {
    if (query == null)
    {
    throw new ArgumentNullException("query");
    }
    return ToDataSet(extenderItem, query, null, tableName);
  }   /// <summary>
  /// Add the ability to return a dataset based on a particular query
  /// to a queryable object.
  /// </summary>
  /// <param name="extenderItem">The IQueryable object to extend.</param>
  /// <param name="query">A generic dictionary containing the
  /// query to execute along with the name of the table to add it to.</param>
  /// <returns>The populated dataset.</returns>
  public static DataSet ToDataSet(this IQueryable extenderItem,
    Dictionary<string, DbCommand> query)
  {
    if (query == null)
    {
    throw new ArgumentNullException("query");
    }
    if (query.Count == 0)
    {
    throw new ArgumentException("query");
    }
    return ToDataSet(extenderItem, query, null);
  }   /// <summary>
  /// Add the ability to return a dataset based on a particular query   /// to a queryable object.
  /// </summary>
  /// <param name="extenderItem">The IQueryable object to extend.</param>
  /// <param name="query">A generic dictionary containing the
  /// query to execute along with the name of the table to add it to.</param>
  /// <param name="dataSet">An optional DataSet. This allows application
  /// to add multiple tables to the dataset.</param>
  /// <returns>The populated dataset.</returns>
  public static DataSet ToDataSet(this IQueryable extenderItem,
    Dictionary<string,
    DbCommand> query,
    DataSet dataSet)
  {
    if (query == null)
    {
    throw new ArgumentNullException("query");
    }
    if (query.Count == 0)
    {
    throw new ArgumentException("query");
    }
    if (dataSet == null)       dataSet = new DataSet();  

    foreach (KeyValuePair<string, DbCommand> kvp in query)
    {
      dataSet = LinqDataExtensions.ToDataSet(extenderItem,
        kvp.Value,
        dataSet,
        kvp.Key);
    }
    return dataSet;
  }   /// <summary>
  /// Add the ability to return a dataset based on a particular
  /// query to a queryable object.
  /// </summary>
  /// <param name="extenderItem">The IQueryable object to extend.</param>
  /// <param name="query">The query to execute.</param>
  /// <param name="dataSet">An optional DataSet. This allows
  /// application to add multiple tables to the dataset.</param>
  /// <param name="tableName">The name of the DataTable to be added.</param>
  /// <returns>The populated dataset.</returns>
  public static DataSet ToDataSet(this IQueryable extenderItem,
    DbCommand query,
    DataSet dataSet,
    string tableName)
  {
    if (query == null)
    {
    throw new ArgumentNullException("query");
    }
    if (dataSet == null)
      dataSet = new DataSet();
    DataTable tbl = LinqDataExtensions.ToDataTable(extenderItem,
      query,
      tableName);
    if (tbl != null)
    {
      if (dataSet.Tables.Contains(tableName))
        dataSet.Tables.Remove(tableName);
      dataSet.Tables.Add(tbl);
    }
    return dataSet;
  }
 }
}

A couple of points to note about the class. Extension methods have to be in static classes, and they have to be static methods. In order to note what class is being extended, you use the “this” keyword before the first parameter. The code itself is fairly self explanatory and doesn’t do anything clever. The clever bit actually occurs in the Linq side. Here’s an example:

public DataSet GetDataSet()
{
    IQueryable q = GetAllQuery();
    return q.ToDataSet(context.GetCommand(GetAllQuery()), null, "MyTable");
} 

private IQueryable<MyTable> GetAllQuery()
{
    IQueryable<MyTable> q = from p in context.MyTables
                        select p;     return q;
}

The context is the DataContext item that you will create when you import the tables using Visual Studio. And that’s it. Using similar techniques you can extend other parts of Linq as you see fit.