LAMP Programming For Professionals

Designed for beginners, intermediate and professional developers, this book provides genuine domain knowledge to learn web based, application development, usingApache as the Web server, PHP as the programming language, MySQL as the data store, all mounted on the Linux OS. It delves deeply into the core of the MySQL and PHP providing a sound understanding of both.

This book also demonstrates a step-by-step guide to developing a few Webapplications [using PHP 6 & MySQL 5.4 under Apache 2.2] which re-enforces all the learning that takes place throughout the book. 

This book uses the learning by doing pattern with a lot of ready-to-use examples. This pattern has proven to be the best approach for learning a new technology. Most [if not all] of the concepts that you will find in this book are demonstrated using well described and easy to understand examples, yet sophisticated enough to demonstrate LAMP Programming in a real-world context.

Additionally, a special chapter covering the virtual installation of Linux is included. This provides an opportunity to create a Virtual LAMP Framework on MS Windows. 

What You’ll Learn? 

Installing and Setting Up LAMP
Working with the MySQL Database 
Working with the PHP programming language
o    Basics – Variables, Operators, Control Structures, Functions
o    Regular Expression [PCRE]
o    Using MySQLnd, the new native driver
o    Using MySQL, MySQLi and PDO extensions
o    Cookies and Sessions
o    Object Oriented Programming
o    Debugging And Error Handling
o    Web Services Using SOAP
Building Web Based Applications
o    Contact Management System
o    Uses jQuery and Ajax
o    Uses SendMail to dispatch reminder emails for birthdays and anniversaries
o    RSS Widget
o    Uses MagpieRSS [RSS parser for PHP]
o    Uses JavaScript to provide Drag and Drop functionality
o    Newsletter Delivery System
o    Uses SendMail to dispatch newsletter
o    An Automatic Email Dispatcher 
o    Uses CronJob to schedule email dispatch
o    Uses SendMail to dispatch emails
o    Using The Model View Controller Architecture
Setting Up Linux On A Virtual Machine Over MS Windows 
Using phpMyAdmin – A Web based MySQL Administration tool

ISBN 10: 81-8404-843-8

ISBN 13: 978-81-8404-843-8
Edition: First
Downloads

Sample Chapter                                                      TOC

THE TOPICS COVERED IN THE BOOK

All About The L.A.M.P. Framework
Introduction to Linux, Apache, MySQL and PHP. How the L.A.M.P. framework works.

Setting Up The L.A.M.P. Environment
Installing Cent OS, Apache, PHP and MySQL. Configuring the Development Environment.

Understanding The MySQL Database
Db Storage Engines, Db Administration, SQL Elements, Table Creation, Working With Table Data, Tables And Its Structure, Operators And Regular Expression, Grouping Data, Joins, Views, Functions, Data Constraints, Subquery, Union, Indexes, Stored Procedures, Transactions, Cursors

Understanding The PHP Language
Mixing PHP And HTML, Variables And Operators, Control Structures, Functions, Regular Expressions, Working With Database using MySQL, MySQLi and PDO, Cookies, Sessions, Object Oriented Programming, Debugging And Error Handling, Web Services Using SOAP

Application Development
Contact Management, RSS Widget, Newsletter Delivery System, Automatic Email Dispatcher, Programming Using The Model View Controller Architecture    

Linux On Virtual Machine
Installing CentOS On Sun VirtualBox

Appendix
phpMyAdmin – A web based MySQL Db Administration tool

CONTENTS OF THE CD-ROM

Setup files for Apache Web Server 2.2.13
MySQL 5.4
PHP 6.0 [Development Snapshot]
ICU 4.2.1
PHP 5.3 [Stable Release]
phpMyAdmin 3.2.2
Video Footage covering LAMP Installation
Source code for the examples covered in this book   

THE TOPICS COVERED IN THE BOOK

All About The L.A.M.P. Framework
Introduction to Linux, Apache, MySQL and PHP. How the L.A.M.P. framework works.

Setting Up The L.A.M.P. Environment
Installing Cent OS, Apache, PHP and MySQL. Configuring the Development Environment.

Understanding The MySQL Database
Db Storage Engines, Db Administration, SQL Elements, Table Creation, Working With Table Data, Tables And Its Structure, Operators And Regular Expression, Grouping Data, Joins, Views, Functions, Data Constraints, Subquery, Union, Indexes, Stored Procedures, Transactions, Cursors

Understanding The PHP Language
Mixing PHP And HTML, Variables And Operators, Control Structures, Functions, Regular Expressions, Working With Database using MySQL, MySQLi and PDO, Cookies, Sessions, Object Oriented Programming, Debugging And Error Handling, Web Services Using SOAP

Application Development
Contact Management, RSS Widget, Newsletter Delivery System, Automatic Email Dispatcher, Programming Using The Model View Controller Architecture    

Linux On Virtual Machine
Installing CentOS On Sun VirtualBox

Appendix
phpMyAdmin – A web based MySQL Db Administration tool

CONTENTS OF THE CD-ROM

Setup files for Apache Web Server 2.2.13
MySQL 5.4
PHP 6.0 [Development Snapshot]
ICU 4.2.1
PHP 5.3 [Stable Release]
phpMyAdmin 3.2.2
Video Footage covering LAMP Installation
Source code for the examples covered in this book

(CFD.net.au – Contract for Difference, Share, Forex, ETFs, Commodities Traders) –

20 SMA has turned strongly bearish, thus bigger time frames show indicators a bit exhausted to the downside, suggesting some upside corrective movement to come in the next hours that could reach 1.4910 without harming current bearish trend.

“Break under 1.4840 will deny upside corrective ve movements and accelerate the fall, with movements likely following local indexes and gold,” said

Valeria Bednarik, collaborator at FXstreet.com

.

Support levels: 1.4840 1.4810 1.4760. Resistance levels: 1.4880 1.4910 1.4945.

Source: EUR/USD Current Price: 1.4849

As text-book as yesterday’s supports held, ditto today for resistance. I watched the open rubbing my hands waiting for a spot to short this rally that I was sure was going to fail mightily. I chose the SPY as my vehicle of choice for an intra day trade, and finally when it set up on the 5 min. chart I was in, right around 9:45. I should’ve waited for one more candle, but I was pretty confident of the move. Then we stalled and began to fall. I thought for sure we would fill that gap and then some, even possibly take out yesterday’s support. Much to my chagrin, we did not. In fact at 10:20 we started moving up, for what I thought was a breath and then took out most of the down move, and my small gains with it. We ended up grinding the rest of the day and I ended up a little below even and still in my “day trade”. Why did I stay in? Well we didn’t hit my stop, at 110.95, and man o man these charts still look pretty bearish to this northern chartist. So I made the decision to hold overnight, so I am short several positions that are grinding me out right now. I need a good solid move one way or the other and I am still leaning to the downside.

The reason I went short was due to the wte initial claims, the UUP the VIX and the eur/usd. As this mkt. rallied, these did not react as they should, in fact, they reacted the exact opposite as I would’ve expected. So I did not think this move up had legs, and I was right. But I waited for a decent entry and had to give up a good portion of the move. Well, we have more news tomorrow, and maybe I should be hoping for good news, then the mkt. will drop and I can get out of my shorts with at least some small profit. Quit being a baby AKOT and let’s look at some charts: Okay, but first speaking of babies……

Let’s start with the eur/usd since I was just chatting about it. As this falls, so does the mkt. The top chart is the DOW the bottom is the eur/usd. You can see that the eur/usd has started dropping, yet the DOW has yet to follow suit.

Currently the eur/usd has stalled out on support I have been showing for a few days, the 1.4717. However, now it has formed a bearish wedge / flag. I will be looking for another big burst down out of this wedge.

The VIX 60 min. chart shows this current up trend. We took out the gap and the rallied right back above it at the close. A strong break of this trend line will be sentiment change signal. However, this thing opened the day down .80 and pretty much rallied all day from then one, with one blip.

Here is a very pretty resistance touch and pullback. Never nudged a line on this chart. It is screaming breakout, but to which side is the question. You know my thoughts.

Not as picture perfect as the DOW, but the SPX gaps up, rallies close to resistance and sells off 50% of the move. Looking at just this chart, I would expect a move / gap down tomorrow and a re-test or even break of the red support line. V has been down three days in a row.

I want to point out this divergence in the NASDAQ. It is not brutally apparent but it is there. Again, another great reversal candle on resistance seems to signal reversal, but reversal candles are a dime a dozen right now.

So, with the previous chart in mind, check out the QQQQ. Divergences all over the place, and the current one is the longest baddest of them all. Taken in context with another tombstone doji, perhaps it is time to start dropping. But if you look at 12/1 we had a similar, not quite as bearish candle, and moved sideways 3 days before dropping. I really don’t want that right now, this sideways action is brutal.

I should’ve put this up there with the eur/usd, but I am too lazy to move it right now. As I type this, I just notice that looks like a bullish flag right there. Interesting, and on support. This looks like another pop up signal to me.

Man I got my charts all out of whack tonight. The RUT has made a second double top or a double top squared as I like to call it. Followed by today’s bearish engulfing candle and the RSI finally curling down, we could surely see some selling here. The 20 ma is moving up into the 50 and that is more bullish than bearish, so watch that.

Here is one of my Chart.ly / twitter charts from a couple days ago. It looks like it is working and still playable for a small short trade. Be cautious at support, but if the RSI is still above the 20 line, It may break support.

I thought LIZ may want to test that 200 ma again, and it appears that it may be headed there. I think it would be better if the RSI had come up a little more to give us more room to the downside, but still playable.

This could be affected by retail numbers tomorrow, but it is bouncing off support and the RSI is oversold, so if the numbers are good or okay, this has technical reasons to rally from here. But bad numbers and we could see a break of support and a move to the $32 area.

Overall in up trend, but a hammer followed by a tombstone doji today, resistance at the 20 ma and trend line and the RSI crashing could lead to a test / break of support.

In closing: More econ news tomorrow, and we are now on the new futures contracts with one week before 3f. I don’t think it will take much to push us over the edge ( like wte initial claims numbers today, yeh that worked real great AKOT!) Well I have to share what I truly think, and believe me I put my $$ where my blog is, and this week, other than TOL, it has been a little painful. I will post more charts on Chart.ly / twitter tonight, it is faster, easier and overall more efficient.

So trade well and prosper my friends. And I will leave you with my favorite line from the Star Trek movie: Old Spock to young Spock: “Since my customary farewell would seem oddly self-serving, I shall simply say good luck.”

Searching used to be hard. Now it’s not. Profound words.

databases used to be a series of shelves full of paper that held bibliographic indexes like PsychInfo.

Within a resource, searching took place in indexes, tables of contents, directories, etc.

Browsing was an important part of searching in physical materials.

Browsing remains a sticking point in the digital environment.

Metadata: traditional – title, author, subject indicator, possibly controlled vocab, maybe not though

You could only search simple metadata and you could only search it by word or first letter.

The technologies of searching before the Internet encouraged browsing. Once you got into a catalog or even the stacks of a library, browsing was very beneficial.

Digital Searching:

searching by metadata, but there is a great deal more and you can search full text

free text/keyword searching

still within structure

expanded notion of what that structure is

document representation may incorporate document itself, plus metadata

• in past document surrogate was a physical card or a very short bit of text in a huge paper bibliographic index
• now document surrogate is the document itself plus metadata

rotated index -  concordance – you search a word and you get four words on either side of the word

precoordinate subject headings in a postcoordinate way

meta-search engines – an engine that searches other search engines

Most people think searching is really easy because of Google.

Web searching

central problem of any search is language ambiguity

synonymy: (faculty? teachers? educators? mentors? guides? professors? instructors?)

Polysemy: (“pitch” “shoe” “china” “bush”) – words that have many disparate meanings

known item searches:

• search most specific pieces you’re most sure of, that will uniquely find that and nothing else, where most likely to find it
• by fragments of author, title, source, etc.
• search the smallest possible database that you can

Kinds of searching:

determine Major concepts/aspects of topic, in order of importance and specificity

search most specific first

use controlled vocabulary and pearl growing

keywords when necessary

For a topic: everything ever written is a high recall search

High precision search – you only want a few things that are exactly what you want

other tricks: search antonyms and smallest database possible

Factors of search effectiveness:

amount and volume of content of the object of retrieval

• low: dictionary entry, library catalog record
• high: google, amazon within-a-book search

amount and level of metadata available for that object:

• low: Google
• high: dictionary entry, library catalog record

Inherent or imposed structure of object available for retrieval:

• low: Amazon WTB
• high: catalog record, dictionary entry

When you want info on a general topic, used metadata and structured objects.

When you want a very specific thing, and you have a search string that it is very specific, you want lots of content.

conceptual/linguistic ambiguity of domain:

• low: anything subject-specific
• high: everything else

Science has lower ambiguity in search terms

Humanities and social science has much more ambiguity in search terms

Joe Janes example for a good search tool: Ebay.

First comes content, then comes metadata along with the structure of the content, then metadata is structured, because metadata must be structured if there is too much of it

You have to have access points and mechanisms for finding something.

Searching has become so commonplace in the last 10 to 15 years.

“Knowledge is power – that is crap. Finding Knowledge is power.” – Joe Janes

“Search changes the way we think” Janes

“We are different because of Google” Janes

Research and scholarship is different.

“You have a more intimate relationship with information” Janes

Imagine a situation where you have to create an index on a table having 1 billion records. It might take approximately 4-5 hours to create the index. But there are ways to fasten this process. Following are some of the tips that will help you to create indexes faster than the normal time. 

Tip#1 – PARALLEL:
While creating index oracle must first collect the symbolic key/ROWID pairs with a full-table scan. When you specify Parallel option to the create index script oracle will perform a full table scan in parallel threads based on the number of CPUs, table partitioning and disk configuration and hence will reduce index creation time.

Following will work fine on a server that has 4 CPUs:

create index indx_t_doc on t_document(doc_id,doc_type) parallel 3;
 

Tip#2 – NOLOGGING:
The NOLOGGING keyword in create index command will restrict the database from generating large redo logs and will create minimal redo log. This will improve the performance of Create index command.

create index indx_t_doc on t_document(doc_id,doc_type) PARALLEL 3 NOLOGGING;
 

Tip#3 – COMPRESS:
The COMPRESS option will enable key compression. It eliminates repeated occurrence of key column values and may substantially reduce storage. Use integer to specify the prefix length (number of prefix columns to compress).
For unique indexes, the valid range of prefix length values is from 1 to the number of key columns minus 1. The default prefix length is the number of key columns minus 1.

For nonunique indexes, the valid range of prefix length values is from 1 to the number of key columns. The default prefix length is the number of key columns.

create index indx_t_doc on t_document(doc_id,doc_type) PARALLEL 3 NOLOGGING COMPRESS;

Follow these simple tips and save your time.

Happy Indexing

I’ve been thinking through and experimenting a bit more with some of the concepts in SQL Azure – specifically I was considering the impact of fragmentation on both the storage (in terms of the storage limit) as well as the maintenance. This is not a new issue, DBA’s face fragmentation regularly and can deal with it in a variety of ways, but with SQL Azure the problem looks magnified by a lack of tools and working space. Whilst looking into this, I then realised that there is an unfortunate consequence of not knowing how much data space your index is actually using.

Each table in SQL Azure has to have a clustered index if data is going to be inserted into it and clustered indexes can suffer from fragmentation if chosen poorly. The combination of SQL Azure and the time-honoured fragmentation provides three consequences about in, fragmentation:

• will occur and you have no way in which to measure it due to the lack of DMV support.
• will create wasted space within your space allocation limit.
• will reduce your performance.

You could work it out if you knew how much space you had actually used vs. what the size of the data held is, but we are unable to measure either of those values. If you have chosen the data compression option on the index then even those values would not give you a fragmentation ratio.

This leaves us with a situation in which you can not know how much you are fragmented, meaning:

• You schedule a regular index rebuild.
• Hope SQL Azure performs index rebuilds for you.

I’m not aware of SQL Azure doing this for you – and you do not have SQL Agent facilities either.

So this seems very wrong, the concept of SQL Azure is to take away a lot of the implementation details and hassle from the subscriber – DR and failover is handled etc. But there looks to be as gap in which certain items such as fragmentation is falling – I have not seen any documentation saying SQL Azure handles it (but there could be some hidden somewhere and I hope there is!) and neither are you given the right tools in which to program and handle it yourself.

What happens when you hit that size limit?

Msg 40544, Level 20, State 5, Line 1 The database has reached its size quota. Partition or delete data, drop indexes, or consult the documentation for possible resolutions. Code: 524289 

That took a lot of time to get to, (SQL Azure is not fast), but was generated using a simple example that would also demonstrate fragmentation.

Create Table fragtest ( id uniqueidentifier primary key clustered,
padding char(3000)
) 

Very simple stuff, deliberately using a clustered key on a GUID to cause a decent level of fragmentation, and also using the padding fixed with character field to ensure 2 rows per page only, maximising the page splits.

insert into fragtest values (newid(), replicate('a',1000))
go 200000

Because of the randomness of the newid() function, the level of fragmentation is not predictable but will certainly occur – in my test I hit the wall on 196,403 records and failed with an out of space message.

Given the 2 rows per page and the number of rows, with ~0% fragmentation the data should be able ~767Mb – that is considerably short of 1 Gb – so there is a significant level of fragmentation in there wasting space, about 23% of it. If you include the 2k per page being wasted by the awkward row size then the actual raw data stored is roughly ~60% of the overall size allowing for row overheads etc.

So there are two important points from this contrived example:

• You can lose significant space from bad design.
• Doing this backs you into a corner that you will not be able to get out of – this is the worst part.

How are you cornered? well, try work out how to get out of the situation and defrag the clustered index / free up the space, you could:

• Attempt an index rebuild.
• Try to rebuild it with SORT_IN_TEMP.
• Drop the index.
• Delete data.

The first three fail, the SORT_IN_TEMP is not supported and would not of rescued the situation either since you have no working space in which to write the newly sorted rows prior to removing the old ones.  So do you really want to delete data? I don’t think we can consider that an option for now.

This all seems like a ‘rock’ and a ‘hard place’; whilst SQL Azure can support these data quantities,  it seems prudent that you never consider actually going close to them at all – and that you equally are going to find it difficult to understand if you are close to them, since there is no way of measuring the fragmentation. The alternative is that you manually rebuild indexes on a regular basis to control fragmentation, but then enough free space is going to have to be left to allow you to rebuild your largest index without running out of space – reducing your data capacity significantly.

The corner is not entirely closed off, the way out of the corner would be to create another SQL Azure database within my account and select the data from database1.fragtest to database2.fragtest and then drop the original table and transfer it back – not ideal but it would work in an emergency.

I think the key is to design to make sure you do not have to face this issue; keep your data quantities very much under the SQL Azure size limits, and watch for the potential of tables being larger than the remaining space and preventing an re-indexing from occurring.

Interested to know your thoughts on this one, and what other consequences of being close to the limit will come out.

Long before I ever thought about writing a book I was fascinated by indexes. I realize now that I have always wanted to read a history of indexes: how themes are chosen, the hidden politics of indexing, unsung heroic indexers etc. Is there one out there?

Of course this realization was spawned by the fact that I just found out that I’m getting an index. I know it makes sense, I just hadn’t thought about it! To me this is much more exciting that the cover finally being chosen.

I mean, check out this snippet:

Reagan, Ronald, 82–83, 96–98, 189–190, 195–196
Reblochon, 92
recombinant bovine growth hormone (rBGH), 26, 74, 84–87, 94
Red Hawk, 134, 180, 184
rednecks, 41–42
Redwood Hill Farm, 44, 55–56, 88, 141, 151
rennet, 5, 93–96

That is a better advertisement for the book than anything I can come up with. My only regret is that I didn’t somehow incorporate The Rezillos into the manuscript. That would have made this little index cross section even better.

There’s no such option! There is – technically – but not quite.

Background:
I wanted to override certain aspects of the apache server using ‘.htaccess’ file, like ExpiredByType – for adding Expiry headers to files that are requested from the server – and SetOutputFilter – to send files gzip comoressed from server.

Here’s I what I wanted:

Add an expiry of one month from the time of access to the following File Types -
gif
javascript
jpeg
ico
css

Enable gzip compression for the following type of docs
html
css
javascript

Putting the above rules in “httpd.conf” worked fine. But when the same was put in “.htaccess”, these rules were ignored.

Reason:

In order for the server to use the settings from “.htaccess” file, the following condition need to be met:
- AllowOverride should be set to some thing other than “None”

To add Expiry header, the following conditions should be met:
- expires_module should be enabled in apache config file
- “ExpiresActive On” rule should be set

Solution:

AllowOverride take 3 types of arguements:
1. All - all directives in .htaccess context are allowed
2. None - .htaccess file is ignored by server
3.
This can be one among the following
a. AuthConfig
b. FileInfo - for Gzip compression
c. Indexes - for stamping expiry header
d. Limit

So the final httpd.conf should have the following lines:

LoadModule expires_module modules/mod_expires.so
AllowOverride FileInfo Indexes

In .htaccess file, the following rules are added:

#Add Expire header to the file types listed above
ExpiresActive On
ExpiresDefault “access plus 1 month”
ExpiresByType image/gif A2592000
ExpiresByType text/javascript A2592000
ExpiresByType image/jpeg A2592000
ExpiresByType image/ico A2592000
ExpiresByType text/css A2592000

#enable gzip compression
SetOutputFilter DEFLATE
AddOutputFilterByType DEFLATE text/html text/css text/javascript text/plain

From GAWKER’s 10 Juicy items in the sneak peaks of Sarah Palin’s historical-fiction memoir (#10) :

No Flipping to the Back. Second-most conspicuous absence: an index, which Halperin says is “subtle revenge on the party’s Washington establishment, whose members tend to flip to the back pages and scan for their own names.” This is possible, but I’m much more inclined to believe that her editors plumb forgot that this peculiar, vapid woman they were working with is an actual politician, who actually interacts with important people, and slipped into Chicken Noodle Soup for the Soul mode by accident.

Andrew Sullivan takes note and links to Samuel P. Jacobs at the Daily Beast , who develops the “vengeance” theme at greater length and offers a more probable explanation:

One answer: time. It takes two to three weeks to put together a good index, says Peter Osnos, the founder of Public Affairs, who has published Bill Clinton, Vernon Jordan, Scott McClellan, and nearly every other Washington macher over the years. Cutting an index can mean the difference between getting a book into stores well before Thanksgiving or missing the holiday sales season altogether. Speed is at an even greater premium now, in the age of e-books and instant downloads on the Kindle. And of course, skipping the index means fewer pages—and fewer dollars spent to bring the book to market. “Every penny counts,” Osnos said.

See my post Negative Reviews and Indexes if you want to see why I relate.

I have mentioned query parameterization before and the process by which SQL extracts literal values from a query and re-writes the query in effect to use parameters to get a query plan cache hit, which negates the need to recompile the plan costing both time and CPU. There are a lot of good articles and book chapters that cover the topic.

What has confused me for a while is witnessing date literals within a query being parameterized on one query and not on another, even though both databases have parameterization set to ‘Simple’ mode. When a date literal is not parameterized the chances of getting a query plan cache hit is obviously very low which has performance impacts. The problem to date has been that I had been unable to ascertain the commonality and requirements that allowed the query to get parameterized and when it just kept the literal. I was paying too much attention to the query and as it turns out not enough to the table.

Well, after a number of hours getting to know the brick wall very well I finally tracked it down, and oddly it had nothing to do with the query I was submitting, but I could reproduce reliably by using an unrelated non-clustered index, which is confusing to say the least and I can not yet think of any reason why, is it a bug or just ’Weird and Odd’.

The following steps reproduce the issue in both SQL Server 2005 and 2008.

Create a database, default settings, nothing specific, then the second step is to create our test table, a simple structure is suffice.

CREATE TABLE [dbo].[paramtest](  [id] [int] IDENTITY(1,1) NOT NULL, [somedate] [datetime] NOT NULL, [somefield] [varchar](50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
 CONSTRAINT [PK_paramtest] PRIMARY KEY CLUSTERED
(
 [id] ASC
)WITH (IGNORE_DUP_KEY = OFF) ON [PRIMARY]
) ON [PRIMARY]

We need some data to work against just to make sure we are selecting results, and we can insert these trivially as follows: 

insert into [paramtest] values (getdate(),'a')
go 10000

So we now have 10k rows within the table, and a clustered primary index on the identity column.

The test starts with freeing up the procedure cache, and then running the select statement, the datetime I used was roughly in the middle of the range of values I had inserted but is not a deciding factor in the query plan results.

dbcc freeproccache select * from paramtest where somedate > '2009-11-12 21:14:50.000'

Using a standard query plan cache extraction query the specific line of the xml plan we are interested in is the SQL Statement.

<StmtSimple StatementText="(@1 varchar(8000))SELECT * FROM [paramtest] WHERE [somedate]>@1" StatementId="1" StatementCompId="1" StatementType="SELECT" StatementSubTreeCost="0.0379857" StatementEstRows="3831.48" StatementOptmLevel="TRIVIAL">

From it you can see the literal varchar value was extracted as @1 with a type of varchar(8000) and the query altered to use this parameter – this is exactly the behaviour we would expect from parameter sniffing.

Next step is to create a non-clustered index on the varchar ’somefield’ – completely unrelated to the date literal being used, and should have no impact on the query at all.

CREATE NONCLUSTERED INDEX [ix_test] ON [dbo].[paramtest] ([somefield] ASC
)WITH (SORT_IN_TEMPDB = OFF, DROP_EXISTING = OFF, IGNORE_DUP_KEY = OFF, ONLINE = OFF) ON [PRIMARY]

Free the procedure cache up again and rerun the query

dbcc freeproccache select * from paramtest where somedate > '2009-11-12 21:14:50.000'

Extract the query plan again from the cache, but this time it is noticable different, the parameterisation has not occurred. The literal has

<StmtSimple StatementText="select * from paramtest  where somedate > '2009-11-12 21:14:50.000'" StatementId="1" StatementCompId="1" StatementType="SELECT" />

 To revert to the old plan, drop the index and clear the cache again, then run the query once more.

DROP INDEX [ix_test] ON [dbo].[paramtest] WITH (ONLINE = OFF)

Then clear the cache again and run the query

select * from paramtest where somedate > '2009-11-12 21:14:50.000'

And we are back to being parameterized.

So the application of a single non-clustered index on a separate field to the one being queried is preventing the simple parameterization mode from parameter sniffing the date literal – this makes absolutely no sense, and you can play around with it a lot more knowing what it causing the effect on the query plan. Even placing the additional non-clustered index on the identity field, which already has a clustered index results in the parameterization failing. If this behaviour is be design, then it makes for an interesting design or limitation on the parameterization.

As soon as the database is in ‘Forced’ parameterization mode, the literal was converted each time, so this looks specific to simple mode, but is not explainable, just demonstratable.

We are expecting some serious volatility moves in the future. No need to post the VIX today, the SP charts speak for themselves.

Send us your email for updates as the trades are placed. ocm2010@gmail.com

Non-clustered indexes normally have a ‘tipping point’, which is the point at which the query engine decides to change strategies from seeking the index with a nested loop operator back to a seek on the underlying table or choosing to just scan the underlying table and ignore the index. Kimberley Tripp wrote a great article about ‘The Tipping Point‘ , and the guidance is at about the 25-33% the query engine will change strategies.

If the non-clustered index is a covering index (it contains all the fields within the query) the query engine does not take the same decision – it makes sense that if any change in strategy occurs, it would have to be at a far higher figure, and as we are about to see, it will not take that decision and tip.

To test what strategy the engine would use I created a test situation of 2 separate tables, with different page counts, due to the padding column forcing the second table to use far more pages (5953 pages vs 9233)

CREATE TABLE [dbo].[tblIxTest1]( [PersonnelID] [int] IDENTITY(1,1) NOT NULL, [FirstName] [char](30) NULL, [LastName] [char](30) NULL,
   [Department] [char](30) NULL, [SomePadding] [char](10) NULL
) ON [PRIMARY]

And,

CREATE TABLE [dbo].[tblIxTest2]( [PersonnelID] [int] IDENTITY(1,1) NOT NULL, [FirstName] [char](30) NULL, [LastName] [char](30) NULL,
   [Department] [char](30) NULL, [SomePadding] [char](1000) NULL
) ON [PRIMARY]

Next step was to insert some data, I needed random data to be able to ensure the index was not unbalanced in some way, so I broke out my useful little random string generation function. I should mention how to create this, a SQL function will not directly support the inclusion of a Rand() call within them, any attempt to do this results in the error:

Msg 443, Level 16, State 1, Procedure test, Line 13
Invalid use of a side-effecting operator 'rand' within a function.

However, there is nothing stopping a view from using this, and the function from using the view to get around the limitation: 

Create View [dbo].[RandomHelper] as Select Rand() as r

And then the function can be generated to use this, it is not necessarily the most efficient random string generation function, but it works nicely.

CREATE FUNCTION [dbo].[RandomString] (@Length int) RETURNS varchar(100)
WITH EXECUTE AS CALLER
AS
BEGIN
  DECLARE @Result Varchar(100)
  SET @Result = ''
  DECLARE @Counter int
  SET @Counter = 0
  WHILE @Counter <= @Length
  BEGIN
     SET @Result = @Result + Char(Ceiling((select R from randomhelper) * 26) + 64)       
     SET @Counter = @Counter + 1   END
  RETURN(@Result)
END

This now allows me to generate random data and insert it into the tables to get a nice data distribution, and this was run for both of the tables.

insert into tblIxTest1 values (dbo.RandomString(20),dbo.RandomString(20),dbo.RandomString(20),'')
go 1000000

Two NC indexes are now needed, one for each table and both are identical and cover just the FirstName and PersonnelID fields within the table.

CREATE NONCLUSTERED INDEX [IX_Test1] ON [dbo].[tblIxTest1] ( [FirstName] ASC, [PersonnelID] ASC
)WITH (STATISTICS_NORECOMPUTE  = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
GO
CREATE NONCLUSTERED INDEX [IX_Test2] ON [dbo].[tblIxTest2] ( [FirstName] ASC, [PersonnelID] ASC
)WITH (STATISTICS_NORECOMPUTE  = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
GO

The setup is complete and it is pretty easy to now show the NC covering index is not going to tip, the most extreme where clause is where I am allowing every record to be returned:

select personnelid , firstname from tblixtest1 where firstname >= 'a' and firstname <= 'zzzzzzzzzzzzzzzzzzzzz'

This still produces a query plan with a seek strategy, regardless of which of my two tables it was executed on:

select personnelid , firstname from tblixtest1  where firstname >= 'a' and firstname <= 'zzzzzzzzzzzzzzzzzzzzz'  
    |--Index Seek(OBJECT:([FilteredIndexTest].[dbo].[tblIxTest1].[IX_Test1]), SEEK:([FilteredIndexTest].[dbo].[tblIxTest1].[FirstName] >= [@1] AND [FilteredIndexTest].[dbo].[tblIxTest1].[FirstName] <= [@2]) ORDERED FORWARD)

If we just select the entire table, unsurprisingly at that point it chooses to perform an index scan.

select personnelid , firstname from tblixtest1

Results in the following plan:

select personnelid , firstname from tblixtest1   |--Index Scan(OBJECT:([FilteredIndexTest].[dbo].[tblIxTest1].[IX_Test1])) 

The row counts on both queries were identical at 1 million. Slightly more interesting is that if I use a Like clause instead of a direct string evaluation, the behaviour alters slightly when selecting all the values:

select personnelid , firstname from tblixtest1 where firstname like '[a-z]%'

Gives the query plan:

select personnelid , firstname from tblixtest1  where firstname like '[a-z]%'  
   |--Index Scan(OBJECT:([FilteredIndexTest].[dbo].[tblIxTest1].[IX_Test1]),  WHERE:([FilteredIndexTest].[dbo].[tblIxTest1].[FirstName] like '[a-z]%'))

So the query engine is potentially making an optimisation that it knows the like clause covers 100% and adopts an automatic scan, but it is not really very clear why it has this optimisation path. If the like clause changes to [a-y] then it reverts back to a seek, so it looks specific to covering all the values within the like statement. If a between statement is used, it remains a seek regardless.

So the result is that a Non-clustered covering index is very unlikely to tip, you either have to not give it a where clause, or use a like statement across all the values available, it will steadfastly refuse to seek and choose the scan.

Why?

Well the I/O cost of the operation remains the same, it has to read every page in the table and it considered the cost of traversing the B-Tree negligible, so the difference between seek and scan is not very great. Running the seek based query and scan based query in the same batch the relative percentages are 48% vs 52% – that is the scan scoring 52% even though they read the same number of rows.

Outputting the IO statistics when they are run side by side shows the same number of pages being read, but the seek is clearly being favoured and is slightly faster as far as SQL is concerned – it is quite weird to consider a seek of an entire index is more efficient than a scan of the index.

(1000000 row(s) affected)
Table 'tblIxTest1'. Scan count 1, logical reads 5995, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.
(1000000 row(s) affected)
Table 'tblIxTest1'. Scan count 1, logical reads 5995, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

So if you come across a covering index in a query plan that is scanning, it would be worth investigating as to whether it is intended. The chances are more likely the index field order is not supporting the predicates being used, than engine has chosen to tip the index like it would for the non-covering non-clustered indexes.

Before I dive into this, here’s my experimental image index that is discussed in this post. You can see some sample pages from the text using Amazon’s “look inside” feature.

Background
Over the summer I took a class on indexing and abstracting. As part of my final project, I indexed some of the images of Greek vases in the book The History of Greek Vases, by John Boardman (London: Thames & Hudson Ltd, 2001). It seems kind of quaint to produce a back-of-the-book image index; maybe that’s just because I’m too steeped in digital stuff. Indexes are crucial for print materials, and I know it would have been useful to my art historical research to have the subjects, vase types etc. indexed. However, this probably isn’t the most practical of endeavors. Any good index to the text would probably index image captions, or the locators would at least get you close enough that you could find that one image you remembered that showed Perseus with a detached Gorgon head in hand. This exercise was more just to see what would happen if indexed multiple attributes but proceeded as if I were just creating a traditional subject index to a text. What would the index look like? What would the cross-references be like, and how would the image index differ from an index to the text?

Method and Meanderings
I didn’t index drawings, maps, or photographs of sections of vases; I focused on images depicting vases in their entirety, and on the descriptive information about them that was included in captions. The locators in my index are for page numbers, although each image was numbered in the text. It just seemed easier to navigate to a page than to an image number.

When deciding what aspects of the images to index, I took inspiration from the access points used by the Beazley Archive at Oxford University. Some of the elements by which one can search the Beazley pottery database are:

• Fabric
• Technique
• Shape name
• Date range
• Inscription type
• Inscription
• Artist name
• Scholar name
• Decoration description
• Collection name
• Publication name

There are even more than that, and it’s pretty impressive. Not all of these categories would be useful for my purposes (a back-of-the-book image index), but several of them (inscription type and vase shape name, for example) are especially useful for various types of art historical research in this field.

I looked at the VRA Core 4.0 metadata schema and noted which of its elements might correspond to those used by the Beazley Archive. The Beazley’s “fabric” element combines VRA’s “cultural context” and “style/period” elements. “Technique” corresponds to VRA’s “material” and “technique”. VRA does have an element for “inscriptions”, but it’s not clear to me if the “type” attribute for the “text” sub-element could be used to indicate the type of inscription (e.g. epoisen or egraphsen signatures). I was mainly using VRA Core as a point of reference, to get an idea of the types of attributes generally deemed important in creating descriptive metadata for images. (at the time, I didn’t know about Cataloging Cultural Objects, but since VRA Core is based on it I don’t see that as a big deal). I also considered the facets of AAT, and what they indicate about elements that can be combined (e.g. style and period).

The attributes I finally chose to index were:

• artist name
• vase shape
• technique
• decoration
• inscription
• subject (both “things” depicted (e.g. warriors) and mythological figures (e.g. Achilles))

I don’t think any of the vases in my sample set ended up having inscriptions. I chose to index only these six elements because I had a limited amount of time to devote to this. If I was indexing images in an online setting I would definitely want to use more access points.

It was difficult to differentiate between imagery that could be both a subject and a decorative element. Many of the vases I was indexing were decorated with rows of animals in a repeating pattern. This is a common motif, so I needed to decide on a policy for how to index it. At first I was making very specific subheadings indicating the type of vase on which the pattern was appearing, but then I realized that this was creating too much work, and it probably wouldn’t be all that useful. It was also starting to conflate the “fabric” and “technique” elements with the “subject”, and I wanted to keep them separate in hopes of having a less chaotic index.

So, for a vase that had a motif of lions in a row, I decided to just give the locator after the heading (“lions, 30-31”). However, if there was a vase with a lion in any other, non-decorative context, I made a more specific subheading (“lions – being hunted, 20”). I worried that it would be confusing to have some locators listed after the heading, and then a subheading with more locators. So I decided that if any subheadings were required, I would instead list the decorative appearances of the subject with the subheading “as decorative element”. Hopefully more examples will make this clearer:

Sphinxes were only used as decorative elements in the vases I indexed. So the entry for sphinxes is:

sphinxes, 20, 33, 41, 45, 46

Lions were used as decorative elements on some of the vases, but sometimes they were part of the narrative scene. So my entry for lions is:

lions
  being hunted, 20, 23, 30-31
  as decorative element, 29, 33

Problems / Discussion
Many of my locators appear multiple times within the same entry for a couple of reasons. First is the fact that one page could contain multiple images of different vases with the same subject matter, technique, or fabric. Second: most vases have multiple sections of varying imagery. I may have been indexing too deeply, but I thought that failing to index all the different aspects of the subjects depicted would be akin to giving the researcher a list of undifferentiated locators. A good example of this is the heading for “warriors”. Since I was creating a heading for “hares – being hunted by warriors” it made sense to have “warriors – hunting hares” instead of just listing all the pages for warriors. Why not give the user as much information as possible? Additionally, my entry for “hares” has duplicate locators because in one section of the same vase (on page 33) the hares are a decorative motif, and in another section they are part of a narrative scene in which warriors are hunting them. (confused yet?!)

As mentioned above, deciding to index both “subject” and “decorative elements” caused problems for me when the two were hard to differentiate. I wanted my imaginary user to be able to use the index to find lion freizes and not have to go through a bunch of locators just to find irrelevant images of lions NOT in freizes. But then again, I always expect too much of my information resources.

In hindsight, I think my index would be more useful if I had created separate indexes for each attribute I was indexing. But there are pros and cons to having the index divided by facets instead of in one long alphabetical sequence. One of the books I examined had one index for mythological figures and one for objects that are commonly attributed to specific gods/goddesses (I thought this was awesome). The book I was indexing had three separate indexes: (1) Artists, Groups, and Wares; (2) Mythological and Divine figures; and (3) General (three indexes in a mere 3 pages!). Though it makes sense to divide the index this way, it could lead to confusion if it spanned more than 3 pages. A user could be looking for Odysseus in the wrong index and not know it unless each page was very clearly marked with a header (I think I remember committing this error as an undergraduate). Another benefit to having one long alphabetical index is that it didn’t force me to always differentiate between the aforementioned troublesome decorative elements and subjects, which overlap so often on the same pot.

In real life, it would probably be best to keep the index as simple as possible, only providing sub-headings when the list of locators for certain topic/shape/artist got excessively long. Nevertheless, if my methods in this exercise could be used on a larger scale, the index would be very useful. Of course that’s unrealistic because of how time-consuming it is, and how “everything is online” now. Back-of-the-book image indexes aren’t unheard of, though. I inspected the indexes in the back of Boardman’s book and another book on Greek art. One of them only listed very general topics (“women” or “fighting scenes”). The other gave some sparse subheadings (and I think this one was the better index overall, though neither of them had any “see” references!).

Conclusion
Indexing a set of images in a book is a good way to become familiar with the basic issues of image description and index construction. It requires decisions that appear simple on the surface, but force you to carefully consider the nature of your subject and the needs of your potential users. You’re forced to make decisions about what to index, how deeply to index, and how to best express what you’re indexing…and those are all just as important in the online environment. So this is a good exercise for a rainy day when you need a rest from computer eye strain.

Also, I think it would be interesting to survey scholarly texts in any image-focused field and try to get an idea of how people deal with images in back-of-the-book indexes. Projects for the future!

– Get all existing indexes, but NOT the primary keys
DECLARE cIX CURSOR FOR
SELECT OBJECT_NAME(SI.Object_ID), SI.Object_ID, SI.Name, SI.Index_ID,
FG.name
FROM sys.indexes SI
left outer join sys.filegroups FG
on SI.data_space_id = FG.data_space_id
WHERE OBJECTPROPERTY(SI.Object_ID, ‘IsUserTable’) = 1
and SI.Name is not null
and SI.is_primary_key = 0 and SI.is_unique_constraint = 0
AND INDEXPROPERTY(SI.Object_ID, SI.Name, ‘IsStatistics’) = 0
ORDER BY OBJECT_NAME(SI.Object_ID), SI.Index_ID

DECLARE @IxTable sysname
DECLARE @IxTableID INT
DECLARE @IxName sysname
DECLARE @FGName sysname
DECLARE @IxID INT

– Loop through all indexes
OPEN cIX
FETCH NEXT FROM cIX INTO @IxTable, @IxTableID, @IxName, @IxID, @FGName
WHILE (@@FETCH_STATUS = 0)
BEGIN
DECLARE @IXSQL NVARCHAR(4000)
SET @IXSQL = ‘CREATE ‘

– Check if the index is unique
IF (INDEXPROPERTY(@IxTableID, @IxName, ‘IsUnique’) = 1)
SET @IXSQL = @IXSQL + ‘UNIQUE ‘
– Check if the index is clustered
IF (INDEXPROPERTY(@IxTableID, @IxName, ‘IsClustered’) = 1)
SET @IXSQL = @IXSQL + ‘CLUSTERED ‘

SET @IXSQL = @IXSQL + ‘INDEX ‘ + @IxName + ‘ ON ‘ + @IxTable + ‘(‘

– Get all columns of the index
DECLARE cIxColumn CURSOR FOR
SELECT SC.Name
FROM sys.index_columns IC
JOIN sys.columns SC ON IC.Object_ID = SC.Object_ID AND IC.Column_ID = SC.Column_ID
WHERE IC.Object_ID = @IxTableID AND Index_ID = @IxID
ORDER BY IC.key_ordinal

DECLARE @IxColumn sysname
DECLARE @IxFirstColumn BIT SET @IxFirstColumn = 1

– Loop throug all columns of the index and append them to the CREATE statement
OPEN cIxColumn
FETCH NEXT FROM cIxColumn INTO @IxColumn
WHILE (@@FETCH_STATUS = 0)
BEGIN
IF (@IxFirstColumn = 1)
SET @IxFirstColumn = 0
ELSE
SET @IXSQL = @IXSQL + ‘, ‘

SET @IXSQL = @IXSQL + @IxColumn

FETCH NEXT FROM cIxColumn INTO @IxColumn
END
CLOSE cIxColumn
DEALLOCATE cIxColumn

SET @IXSQL = @IXSQL + ‘)’ + ‘on ‘+ @FGName
– Print out the CREATE statement for the index
PRINT @IXSQL
PRINT ‘GO’

FETCH NEXT FROM cIX INTO @IxTable, @IxTableID, @IxName, @IxID, @FGName
END

CLOSE cIX
DEALLOCATE cIX

http://www.sqlservercentral.com/scripts/Miscellaneous/31893/

Each year, we review our portfolio of journal subscriptions to make sure we’re covering the best publications in the industry. We currently subscribe to 65 journals, through both electronic and traditional (paper) subscriptions. A complete list of our current subscriptions is available at http://www.naric.com/research/rehab/portfolio.cfm.

In more than 30 years of collection development, we have indexed articles from more than 100 journals. You can browse through REHABDATA by journal title at http://www.naric.com/research/rehab/journal.cfm

SQL Injection Attacks and Defense by Justin Clarke.

Microsoft SQL Server 2008 Reporting Services Unleashed (SAMS Publishing).

Oracle Essbase 9 Implementation Guide (Packt).

Kevvie Fowler’s comprehensive tome, SQL Server Forensic Analysis (Addison Wesley). www.applicationforensics.com.

Fowler’s other Web site is Ring Zero (www.ringzero.ca)

Chapter 1. Introduction to Databases

Running Chapter 1 Sample Scripts

Databases Explained

How Databases Are Used

Databases and COTS Applications

Database Structure

Structured Query Language (SQL)

Database Transactions

The ACID Model

Referential Integrity

Summary

Chapter 2. SQL Server Fundamentals

History of SQL Server

SQL Server Versions and Editions

Architecture

SQL Server Connections

Context Switching

SQL Server Databases

Data Storage

Memory Management

Security

Permissions

Encryption

Dynamic Management and Database Console Commands

Logging

SQL Server Agent

Summary

Chapter 3. SQL Server Forensics

The Road to SQL Server Forensics

SQL Server Forensics

SQL Server Forensic Methodology

Summary

Chapter 4. SQL Server Artifacts

SQL Server Artifacts

Resident SQL Server Artifacts

Nonresident SQL Server Artifacts

Artifact Summary

Summary

Chapter 5. SQL Server Investigation Preparedness

SQL Server Investigation Preparedness Overview

Configuring Your Forensics Workstation for a SQL Server Investigation

Creating a SQL Server Forensics Incident Response Toolkit

Summary

Chapter 6. Incident Verification

Running Chapter 6 Sample Scripts

Incident Verification Explained

What Not to Do When Investigating a Live SQL Server

Responding to an Incident

Identifying the SQL Server Instance Name

Connecting to a Victim System

Disconnecting from the Victim System

Identifying Signs of an Intrusion

Submitting Preliminary Findings

Summary

Chapter 7. Artifact Collection

Focus on Ad Hoc Collection

Running the Sample Scripts

Maintaining the Integrity of Collected Data

Automated Artifact Collection via Windows Forensic Toolchest

Identifying the Victim’s SQL Server Version

Ad Hoc Artifact Collection

Collecting Volatile SQL Server Artifacts

Collecting Nonvolatile SQL Server Artifacts

Summary

Chapter 8. Artifact Analysis I

Working Along with Chapter 8 Examples

Pre-analysis Activities

Authentication and Authorization

Configuration and Versioning

Summary

Chapter 9. Artifact Analysis II

Working Along with Chapter 9 Examples

Pre-analysis Activities

Activity Reconstruction

Data Recovery

Summary

Chapter 10. SQL Server Rootkits

Traditional Rootkits

SQL Server Rootkits: The New Threat

Generations of SQL Server Rootkits

First-Generation SQL Server Rootkits

How Rootkits Can Affect a SQL Server Investigation

Detecting Database Rootkits

When to Check for Database Rootkits

What to Do if You Find a Rootkit

Summary

Chapter 11. SQL Server Forensic Investigation Scenario

Scenario Overview

Importing Sample Artifacts

Investigation Synopsis

Incident Verification

Artifact Collection

Artifact Analysis

Activity Reconstruction

Investigation Summary

Appendix A. Installing SQL Server 2005 Express Edition with Advanced Services on Windows

Appendix B. SQL Server Incident Response Scripts

SSFA_DataCache.sql

SSFA_ClockHands.sql

SSFA_PlanCache.sql

SSFA_RecentStatements.sql

SSFA_Connections.sql

SSFA_Sessions.sql

SSFA_TLOG.sql

SSFA_DBObjects.sql

SSFA_Logins.sql

SSFA_Databases.sql

SSFA_DbUsers.sql

SSFA_Triggers.sql

SSFA_Jobs.sql

SSFA_JobHistory.sql

SSFA_Configurations.sql

SSFA_CLR.sql

SSFA_Schemas.sql

SSFA_EndPoints.sql

SSFA_DbSrvInfo.sql

SSFA_AutoEXEC.sql

SSFA_TimeConfig.sql

dbta.com: New Additions to the DBA Bookshelf.

I just finished my 11th (to be) published non-fiction book, and the good news is that I didn’t have to make an index for it.

The bad news is that I didn’t have to make an index for it.

My Strange Wisconsin index

My dirty little secret: I like making indexes. The old way. With a pencil and index cards. I know there are software programs for this, but I agree with the Chicago Manual of Style when it states that a computer-made index “cannot in any way substitute for a real index prepared with the aid of human intelligence.” (Thankfully, degrees of human intelligence are evidently not an issue.)

The crux of it is that indexing requires a sifting process to decide exactly what is pertinent to the subject at hand, and that process is at least partly subjective . If a book is about strange creatures, “phantom pigs” is probably a pertinent entry, but its exact  location in the obscure Welsh hamlet of Pentrefoelas may not be considered index-worthy. At least I didn’t think so when I prepared the index for Hunting the American Werewolf. A software program set for  proper nouns might have beeped to differ.

Besides, I find the process relaxing. You get the galley in the mail and look at how the pages have shaken out and how the designer has arranged things. That’s always enlightening. Then starting at page one, you write the words you choose on the indexcards, alphabetizing each. And don’t forget the page numbers.  Several packs of cards will be required for most books targeted above kindergarten level. And that Chicago Manual of Style will be invaluable for the picky parts.

Along the way you pick out any lingering typos that can still be fixed without disrupting design flow. A favorite beverage and snack is mandatory. The only tedious part is the data entry after you reach The End, but you could also enter as you go.

I’ve done this for five books — the others provided professional indexers at their cost – and this last one is part of a series with detailed chapter entries up front. But an index is normally a lovely and necessary thing to any researcher (or purchasing librarian — I’ve been one), and in my opinion no NF book should suffer the indignity of an indexless rear end.

I will add that I would never sign a contract for a book that required ME to pony up for a professional indexer. It would be like paying someone else to pet my dog or do my crossword puzzles. Genre NF advances are meager enough as it is.

And in the end, literally, a non-fiction tome without an index is just, er, book-naked.

We are always looking for the next great idea and we think you have it!

A Suggestion Box has been installed in the library and now the only thing it needs is your ideas.

Where?

• Look to your right when you walk in. It is mounted on the wall next to the New Books Board.