In business, you want to stay ahead of the competition. One of the best ways to do that is to know what the competition is up to in the first place.

You need to be aware of what innovations they’ve got in the pipeline. What big deals are in the works? Are there any weaknesses you can exploit?

Ah, but how to find all that out?

You must snoop around.

Corporate espionage is illegal, yet you can still learn the inside story.

Short of coaxing information out of someone who works for a competitor, the best tool for sleuthing is right at your fingertips. Akin to a search engine, you can spider around the web to uncover competitive intelligence.

Here are three ways to go about it:

Stay on alert

People are likely talking and writing about the competition online. But how do you know where and when it’s happening?

Set up alerts. All major search engines enable you to set up free news alerts from their news sites. For an extra edge Google alerts not only sorts through news, but also blogs, video, discussion groups and the web in general. Socialmention.com works in a similar way, though its focus is social media. Another means to monitor blogs is to use a blog search engine, such as Technorati, Blog Search Engine,  Google Blog Search and Bloglines, where you can search for terms and then set up RSS alerts to keep up with online conversations.

There are also for-pay tools for online watchdogging (some also provide analytics) such as The Search Monitor,  PR Newswire eWatch, Trackur, CyberAlert, and Radian6.

Scour your competitor’s web site

Plenty of great information is right there for all to see. Yet you’d be surprised how often this most basic investigative tool is overlooked.

Thoroughly scour a competitor’s web site.

Analyze press releases and news announcements. Sometimes a company’s plans are out in the open, but you also need to read between the lines. Certain key new hires can indicate an intended but as yet unannounced entry into a new area of business. Reductions in workforce can be early sign of trouble, or perhaps the company is looking to get out of one area and move into another.

Check out the careers section. Note what job categories have the most openings. Are they doing a lot of executive hiring? Read the job descriptions, especially if the position is new to the company. Here again you may be able to tell if a company is looking to ramp up existing areas of business or enter new ones.

Review product, service and landing pages. Note what features and benefits are highlighted. What’s the gist of the sale pitch? This is all great fodder for knowing how you can counter with even better promotional efforts.

Visit discussion groups, forums and corporate blogs. These can be goldmines for finding out what’s up with a company beyond its marketing and public relations schemes. Notice what’s being hyped. What are customers complaining about? The latter can help you identify potential weaknesses to pounce on.

Read annual reports and financial statements. If it’s a publicly traded company these documents are available. They’re another great source of information where you can glean insight into what’s really going on deep inside an enterprise. For instance, if you see a sizable rise or drop in R&D that gives you some idea of how much innovation is in the works. Financial breakdowns can provide clues about how specific divisions are performing.

Social media. A company’s online presence is more than its main web site. Be sure to pay attention to what competitor’s are doing, and to what customers are saying, on social media platforms like Twitter, Facebook and MySpace.

Let them tell you what’s new

To get the latest news and special offers straight from a competitor’s mouth sign up for its RSS feeds, email newsletters and other promotions.

Note: Some businesses comb through these lists and weed out email addresses that belong to members of the competition. The workaround here is to go undercover: Set up a free email account with Goggle, Yahoo, or Hotmail.

– Deni Kasrel

What to YOU think of these three ways to spy on the competition? Do you know of other (legal) methods to gather competitive intelligence? Comments welcome.

We’ve just added blogs to the free Moreover RSS feeds pages. The blog search runs over the latest posts from around 250,000 blogs, i.e. those on the UGC Metabase White List.  You can use the sliders to increase the relevancy and quality of the search results, for example slide the sources filter to the right and you’ll just get results from top blogs (using Moreover’s feedRank).

You can also select one of the prebuilt feed categories and search within that – by sticking to a predefined universe of content you can get even more specific with your results.  The full list of the new blog feeds is also on the prebuilt feeds page, just click the green RSS icons,

We hope you like these latest additions! Let us know if you have any comments, we love to hear from you, or if you’ve got a blog you want us to include in the search just drop us a line with the address.

Finally, a breakthrough in building a faster search engine! For anyone else out there trying to do this stuff, save yourself some trouble and read on…

Background

I’m working with MySQL on a very modest setup. At the time of writing, my search engine has to deal with about half a million documents. My search queries were running very slow and putting strain on the server. It was easy to blame the hardware but I wasn’t convinced. Many search engines are dealing with billions of documents and queries, and run very quickly. They know something I don’t.

The Bottleneck

I eliminated anything that could complicate things and found the bottleneck was in simply selecting words from the page word index. This was very disappointing. On further inspection I found the actual query was quite fast, but what was really slow was loading the required index (like a pathway) to gain access to the correct records.

The Solution

The problem here is a bloated page word index. Time to sort it out…

1. Get rid of stop words.
   Any very common words (ie. a, an, the, this) are not really so important when it comes to search. Its better to filter them altogether. This will greatly reduce the amount of word being indexed. These are called “stop words”. There are many stop word lists available on the net. Just have a quick hunt around.
2. Get rid of anything else that’s not really needed.
   My page word index was full of words which were really just gibberish. To identify these words I simply keep a count of how many pages a word appears on. Words with lower counts are probably rubbish words not worth indexing.
3. Break things down.
   I found dividing my page word index into separate tables helped in a big way. Sure, I had more tables, but these tables had smaller indexes and the performance gain was instantly obvious. I split my page words by first letter or number. Perhaps on more popular letters or numbers I could break it down further. So I ended up with tables like: page_words_a, page_words_b, page_words_c and so on.

More Tips & Ideas

There are other ways of speeding things up.

1. Keep commonly used data in smaller separate tables.
   For example, if you have a table full of page data with all the page texts, and you intend to sort pages by date, put the dates in a separate table which refers back to the main page table. This keeps the record size small, which means better speed.
2. Accuracy vs speed.
   Often your search doesn’t need to give perfect and accurate results. If a visitor is only looking at a first page of 10 results, there’s no point fetching and sorting thousands (or millions) of pages. Try to select page words which seem to be good candidates. Then work out those scores and sort. A bit of inaccuracy results in far faster search times and is much kinder on your server, which means you can handle more traffic.
3. Cache common search queries.
   This one is pretty obvious. Any results from common searches can be periodically saved and used when required. No need to cache all results, maybe just the first few pages.

There’s probably plenty of other speedup tricks available. With a bit of creative thinking almost anything is possible.

It’s been a good test getting lots of data into the system. However, making Bluebloomer run quickly has been tough…

Search

Search has been the toughest. I’m testing a variety of different approaches. Again it’s a trade of accuracy vs speed. Although at this stage even simple fetching of indexed single words is taking too long. Hardware and setup may be some of the problem here, though I’m not 100% sure.

Caching

I tested caching topic results. Caching all results was unwieldy and slow. What I’ve now done is limited the amount of results in the cache and also split them into different tables depending on filters and sorting. This new method I haven’t yet tested, but my hopes are high. I’m guessing much of Bluebloomer’s traffic will be visitors viewing topics. If the cache works for these, I should be able to save some strain on the server.

Background Server

I have added another server to Bluebloomer. This will help in a few ways. First it gives me a place to backup the database. But in addition to this it can also handle data from the crawlers, create topics, cache topics and a few other odd jobs. Once again, this takes strain off the main public server.

All of this is taking some time to get sorted, but in my opinion any speed improvement is worth the effort. A slow site is not so much fun to use. Will keep trying.

This is very ironic. Bluebloomer manages to find its own blog. Glad to know it doesn’t suffer the Altavista syndrome…

Bluebloomer finds itself.

Woo hoo! It’s been tough getting meaningful topics to automatically appear across the Bluebloomer website. Yesterday it was a right mess. Who would blog about the topic “00 Pm” or “2 3″? Today, a different a story. I made a minor alteration to the topic selecting formula and wow what a difference! Topics now make sense and are relevant to there parent topics.

I’m now experimenting with the sensitivity of the topic formula. I’m able to adjust how topical phrases need to be. At the moment I think there are too many topics coming up. It’s just trial and error really.

It’s not absolutely perfect, there will need to be some human intervention. For example, it gets confused between, “celebrities” and “celebrations”, or “Tiger Woods” turns up under the “Bush & Forests” section. But generally, its all working well. What a relief.

The next step is getting the hot topic selection formula right. Gonna take a bit of thought.

Topics and stories magically appear on the front page.

Had to ease up the amount of blog crawling. The crawler looks ahead and gathers links, but there’s only so many I can get through in a day. There are many tasks Bluebloomer has to deal with, ie. page finding, crawling and indexing text, crawling images, phrase finding, category creation, caching… All these tasks vary in time needed, it’s a balancing act getting everything to work in harmony.

I’d say its going to take a few weeks measuring and sorting out the best balance. Also as I add new crawlers I’ll need to readjust things to suit. But if the overall ratios are ok, generally things should be fine.

I’m hoping to activate category creation today. This means things will start appearing on the front page. We’ll see how we go. In the meantime gotta keep on crawling.

Yesterday I decided things were running a bit slow. When building search engines there are a few factors which really affect the speed.

1. Word table size (all distinct words occurring in all pages)
2. Page word table size (all words appearing on a page)

Bluebloomer’s tables were getting huge. So the last few days I’ve been optimizing.

One quick way of shrinking both tables is by using a more aggressive stop word filter. This filters out more commonly used words. Easy enough. Cool.

Bluebloomer had been storing word positions. The reason for this was to aid in phrase finding. This resulted in a massive page word table, not good. Yes, the phrase finding was perfectly accurate, but the cost was too high. So instead of being truely accurate, Bluebloomer now guesses which pages contain particular phrases and how important those phrases are to a page. The result is a greatly reduced page word table and faster search speeds.

So far that’s as far as I’ve got. I think the lesson here is that compromises must be made (unless you’re Google). Don’t let perfectionism get in the way of speed and usability.

Warning: this is a bit of techy post.

I’ve been working on finding phrases from thousands (potentially millions) of blogs. This is no easy task but I’ve tried two different approaches and the results are interesting to what performs better…

Method 1 – The rugged way

Crawler Machine:

When blog posts are crawled phrases are extracted from the document. These phrases are indexed as are single words. Indexing of phrases only applies to the blog post being crawled. This ends up with a large word index. A few tricks are used to help intelligently reduce the amount of phrases found.

Master Machine:

Phrases that start appearing across many blog posts are then indexed against all other pages. This means the phrases are now indexed globally. From time to time these phrases need to be reindexed, but it becomes quicker as pages already indexed need not be checked again.

Many phrases will never show any promise. These can be cleaned out if they become too old. This frees up some room on the database and keeps things running quickly.

Method 2 – The elegant way

Crawler Machine:

Blog posts are crawled but only single words are indexed. Crawling is quicker than method 1.

Master Machine:

Words which appear on many blogs are put through a phrase finding function. This takes some time, especially on popular words. It has to do some guessing to reduce time.

As before, phrases are indexed against all pages.

There is no need to clean the database out. The index is smaller and only includes promising phrases. Very tidy, very elegant.

So which method works in the real world?

It may come as a surprise but it appears that method 1 works better. More phrases are found, but this is done on the crawler machines which takes a load off the master machine. The master machine has no trouble putting lots of info into the database, but it has far more trouble phrase finding. The gain method 2 has in crawl speed is well offset by the time to phrase find. The worse bit is that method 2 gets bogged down on the master machine.

When it comes to indexing phrases method 1 tends to be quicker. The reason is that it can do a more accurate job of picking good phrases and simply ignoring the rest. Method 2 needs to use a bit of guessing to find good phrases, this can actually lead to more phrases being indexed.

Method 1 requires a clean up phase. But this only takes a few moments.

Regarding method 1, having a master machine doing less work frees it up to do other tasks ie. being a database server. This is so important for real world applications.

So it looks like Bluebloomer will stay on the rugged method 1 path for now.

Final thought

Imagine indexing all possible phrases. Never throwing away a thing. No indexing would be required on the master machine, but the database would become extremely bloated and slow. The great thing about method 1 is it strikes a balance between accuracy, memory and speed.

Well yesterday seemed all good. Bluebloomer was busy scanning blogs so I took a rest, came back and trouble. The database was locked, my memory usage was through the roof, I had emails from my web host. Headache!

So I tried to find out what was going wrong. Turns out I had been a bit naive about things. This particular blog entry http://anshumanpandey.wordpress.com/2009/05/27/top-ten-war-machines/ was evil. It had a high word count, but that was ok. The evil part was the entire blog entry was duplicated inside a hidden layer. Because Bluebloomer does phrase matching this caused all sorts of trouble.

Really it’s my own fault though. I should have been a bit more careful. Things are now sorted, a new filter is in place to stop rogue blogs getting through and there was a silver lining as I found and fixed a few other bugs along the way.

Hopefully things will be ok now and lots of new blogs will start appearing on the site.

Bluebloomer is still officially in development. There is a website there, but things may change and updates will be inconsistent. Running a development version is great for testing though…

A few hours ago I removed everything from the site. Bluebloomer is now finding new blogs. My guess is it will add anywhere between 200,000 – 300,000 blog entries during this scan. As the day goes on these will begin to appear and subtopics will be automatically created (I hope).

There are more things which need to be sorted after all this and then a massive list of improvements to add at a later date.

I’m really itching to get a solid version up and running. Quite keen to get on with the marketing side of things.

…patience.

There's nothing there!

Welcome to the Bluebloomer blog. It seems fitting that Bluebloomer has it’s own blog when you know what Bluebloomer actually is…

At a glance Bluebloomer may look like another blog directory, but there’s more than meets the eye.

Imagine some big event occurs, like Susan Boyle’s debut album. Lots of people are gonna be blogging about this. Bluebloomer finds these blogs. It then tries to find if there is a common topic, maybe “Susan Boyle’s New Album”. Hmm,  “Susan Boyle’s New Album”, that should go under the “music” category. Makes sense.

Ok, so Susan Boyle, this is big news. Bluebloomer knows it. So “Susan Boyle’s New Album” is featured as a hot topic on the frontpage. If you visit this topic you’ll lots of relevant up to date blogs about the Susan’s new album. Great!

Of course it doesnt just deal with Susan Boyle. Bluebloomer covers a vast range of topics: Arts, Entertainment, Lifestyle, News, Science, Technology… the list goes on. All these topics are broken down into more defined subtopics and so on.

So since Bluebloomer is about blogs, a Bluebloomer blog seems right.

Go check it out for yourself -  www.bluebloomer.com (development version)

Many of my friends, colleagues, and clients have been asking “Ok Scott, I’ve created a blog.  Now how do I get people to visit it?”

There are many ways to drive traffic to your blog.  One is to list your blog, and the RSS feed of your blog, with Blog Search Engines.  Blog Search Engines are just like traditional search engines, except the specialize in blog content.

Here are a few tips when submitting your blog to Blog Search Engines:

1. Only use those that allow free submissions.  There is really no reason to pay for a submission to a blog search engine.
2. Always list the domain name of your blog, not the URL of the 3rd party blog provider.  For example, I list “http://www.scottgordon.com”, not “http://swgordon.wordpress.com”.   Why?  Because I own and control www.scottgordon.com.  I’m want to drive traffic to ScottGordon.com (the domain I own), not wordpress.com – which I don’t own.  And, if I decide to change blog providers (from wordpress to blogspot, for example), these listing will follow me.
3. Whenever possible, add your blog RSS feed as well.  Follow rule #2, and use your domain name as well.  For example, use http://www.scottgordon.com/feed/ instead of http://swgordon.wordpress.com/feed/ .  And always test the URL of the feed you are submitting before you submit it!

So where do you submit your feed?  Here is a list of Blog Search Engine sites:
2RSS
http://www.2rss.com/

Blogdex
http://www.blogdex.net/
http://blogdex.net/add.asp

Blogdigger
http://www.blogdigger.com/
http://www.blogdigger.com/addFeedForm.jsp

Bloghop
http://www.bloghop.com/search.htm
http://www.bloghop.com/addblog.htm

Blogsearchengine
http://www.blogsearchengine.com/
http://www.blogsearchengine.com/add_link.html

Daypop
http://www.daypop.com/
http://www.daypop.com/info/submit.htm

Eatonweb
http://portal.eatonweb.com/
http://portal.eatonweb.com/register/

Globeofblogs
http://www.globeofblogs.com/

IceRocket
http://www.icerocket.com

Read A Blog
http://www.readablog.com/
http://www.readablog.com/AddFeed.aspx

Sindic8
http://www.syndic8.com/
http://www.syndic8.com/suggest_start.php

Technorati
http://www.technorati.com/

Weblogalot
http://www.weblogalot.com/Ping/

We’re pleased to announce that blogs are now available with the Moreover search engine service (see Search Engine Toolkit).  Alongside news sources you can search across the latest blog posts from the Moreover White List of approx. 250,000 blogs. Search across the title, and/or the post’s text, slice by rank or topic, apply relevancy filters, set a time range, and a whole host of other filters and options.

With the toolkit you basically get API access to our news+blogs search engine. Customers can completely white label search engine features and imbed them within their applications and UI. Effectively it outsources news aggregation and search engine development so you can focus on the actual delivery of services to your customers and users. Case in point: BusinessWeek’s BusinessExchange - interweaving Moreover-fed news links across their universe of topics.

For our free RSS feed loving folk – we’ll be adding blogs alongside news there too in the near future, so stay tuned!

“Google for Advertisers” is a brand new site from the search engine giant, and as the title suggests, it

Malayalam search engines in the internet are not that much popular. Google started there malayalam search and malayalam blog search very effectively. We can simple construct a malayalam search engine very easly. search.keralamla.com provide a malayalam typing interface and malayalam search results with the help of google search. You can simply go and check that here

prune juice explosion mommy,  boston market cupcakes,  noggin channel gives me headache,  wondertime magazine preschool valentine crafts

These are the top searches on my blog today.  Apparently, I have a new title: Prune Juice Explosion Mommy (which I’m hoping is just in reference to this post and not something else!)  And I must tell you that many people search my blog about Boston Market Cupcakes.  Are they leery about trying them?  They are quite tasty!  Go ahead and try them.  Eat one while watching Noggin!

The ball is definitely rolling folks! We’re on Thing #3 this week. Did you know there were search engines dedicated to blogs? Here are a few to check out and compare:

• BlogPulse
• Google Blog Search
• Icerocket
• Technorati

Bloggers looking to earn a buck could definitely benefit from having their blogs listed in this type of database. After all, the more exposure they can get, the better their earning potential. Questions to consider: What is the benefit for reference services to utilize these search engines? Would using these specific search engines come in handy at the reference desk? Would it benefit libraries to be listed? Does an average user use these sites or have a need to?

Regator.com, a website co-created by my friend Kimberly Turner, has a snazzy redesign featuring photos and a more user-friendly interface. In my definitely biased opinion, it looks top-notch.

For those unfamiliar with Regator, the site lets you search top-quality blog posts on hundred of topics from a constantly updated directory of hand-picked blogs. If you’re looking for a way to sort through the thousands of blogs on the Internet, it’s definitely worth a look.

Everyone is familiar with the power of word of mouth.  In fact there is an organization dedicated to it – WOMMA.  There are companies like BzzAgent harnessing and trying to spur word of mouth for their clients.  And with blogging, people can voice their opinions faster and more loudly than ever.

While the opinions you find in blogs are an opinion of one, monitoring blogs regularly might help you identify trends that you can take advantage of when positive or market against when negative. 

Think about Mentos and Diet Coke, who would have thought a soda volcano would have boosted the sales, awareness and interest in a candy.  Had you been tracking the web, you might have.

Even if you can afford research, keeping an eye on blogs will keep you closer to your audience.

Here are some blog search engines that you might find helpful:

www.blogsearchengine.com

www.blogsearch.google.com

www.technorati.com

www.blogpulse.com

www.icerocket.com

Hey Everybody,

Whether you are actively tracking a recent campaign or just interested in what the chatter online is today, searching the world of blogs and micro-blogs is becoming increasingly important. But with so many different outlets and so many different searches, what is the best. Well, I don’t know if there is a best. But here are a few that I use, I hope that they are helpful to you as well

1. The nine-hundred lb. gorilla, Google. Google tends to find almost everything, big and small. I use it’s blog search function a lot more than its news search function. Google Blogsearch seems to be the most thorough. It’s downfall is in its sorting and ranking. You can only search by post, not by blog-theme and you can only sort by relevance or date. Not a real ranking system. But overall, if you need a lot of hits, this is the first place to shop.

2. The sorting king, Technorati. Technorati is a blog search engine. It is pretty nice. Sometimes, Google will find a few hits that Technorati misses. It also takes Technorati longer to index then Google and sometimes will be a little slower on the uptake. But, Technorati is my favorite, because it allows you to evaluate blogs by their authority ranking and by who has made this post or this blog their favorite. Technorati also allows you to search blogs (not just posts) search for blogs devoted entirely to a subject.

3. IceRocket, is a pretty nice little blog search engine. It returns pretty solid results including a strong images and video search. It also allows you to search by cloud. Another cool feature is that allows you to enter a Blog’s URL and you can see its usage stats, they call this BlogTracker. Another feature is that you can search MySpace. This sounds pretty great, but as anyone who tries to search MySpace on their own knows, it is pretty unsearchable. This feature suffers the same fate. IceRocket has some great bells and whistles, but it is not a thorough as the two big boys above.

4. BlogCatalog, is a very large social network for bloggers that allows you to search posts, blogs and other info that bloggers offer up (their other social accounts). It is pretty nice, but it is a social network, not a powerhouse of search. It is a lot of fun though and I really enjoy how it aggregates all of your presences. If this were a post about social networks, I would definitely recommend it. It is a lot of fun. But for blog searches, it is only okay.

5. Twitter, this microblogging site is often the pulse of the hot issue of the moment (literally the second). But you can go back and search what has been said by keyword. Do this by visiting www.search.twitter.com. Again these posts are limited to 140 characters, but they often provide links with in depth information. And it is an easy way to get a quick reading on whether the world hates, loves, or doesn’t care about you at the moment.

Two others of note are Blogsearch.com, which offers a good starting point by listing a variety of searches and what they provide, and Yahoo! blog search.

Hope you enjoy and if you have any others that you like, please let me know.

Thanks,

Will Flavell

I see more and more people raising the questing about the need for blog search engine, especially when Google is doing such a great job finding good content from blog as well as from web sites. It seems like that Google itslef is not investing too much in their blog search too. So, in this post I will explain what I think should be the duties of a blog search engine and why I still see a need for one.

Blog search engines (should) serve multiple purposes

• Finding great bloggers, blogs and blog posts
• Recognizing great bloggers, blogs and blog posts – rank.
• Categorizing blogs and bloggers in multiple ways not limited to content type. Categorize blogs by their objectives: personal blogging is not the same as corporate blog or professional bloggers, subject expert, politics, go green, artist or others. It is not just about what that the blogger writes about but also about what the blogger is trying to achieve.
• Monitoring blog and blogger progress – is this blog alive? a shooting star?
• Web-now – see Twitter Search Trending Topics, Twingly’s Hot right now or Technorati’s what’s percolating in blogs now
• Alerts – a list of new blogs in a given category that are doing well
• Community building – increasing cooperation among bloggers (e.g. you should read this blog)

What do we need to know?

• The top bloggers in a category
• The top blogs in a category
• The top blog post in a category

Who needs it?

• The readers – to know what to read, what is going on in real-time
• The blogger
  • To present a case to a sponsor
  • To know whom to look up to
  • To see and share about the blogger progress
• The business
  • To know where to buy ad real-estate or whom to sponsor
  • PR – where to spend my effort effectively

The challenges of blog search engines today.

Using the reaction counting method for ranking, the service needs to distil humans actions from automated (bots) one in order to be accurate. So far this is not working well and adds another questions mark around the validity of blog search engines.

Here are some example for both:

• Human reactions
  • Blog post reacting to another
  • Update on Twitter or Jaiku
  • Digging on Digg
  • Submitting to social bookmarking site
  • Posting on a social network
  • Bloggers community
• Bot reactions
  • Splog (spam blog) – copycats
  • Related articles by Zemanta

The number of sites that offer posting of human blog reactions is growing faster than the crawling capabilities and sometime does not offer access to crawlers.

The service should also remove the “me” links from the count i.e. links from all the social object under the same owner.

A couple of thoughts

Maybe someone could think about another way to rank blogs and bloggers. Measuring traffic is probably a more accurate way (Alexa). The traffic is relative to the category. I assume that a blog about Technology will get more traffic than a blog about biology. The rank should be within a category and not across all (or not just across all blogs).

In my opinion there is a need for blog/blogger search engine but the emphasize of the search capability should be less around finding content (leave that to Google) and more about discovering leading blogs and bloggers. 

It does not need to be a free service at least not for the business. The premium or a sponsor account model could work as well.