1650 PST: Sorry folks the site is experiencing a problem right now – our engineers have been dispatched and are working on it. More as we get it.

1700 PST: It appears that Amazon, a major part of our infrastructure, is experiencing problems. Again, we’ll keep posting more as we get it.

17:50 PST: all clear, sorry for the inconvenience. UltraDNS, one of Amazon’s providers, had an outage which eventually impacted us.

Yesterday’s Blackberry outage was eye-opening.  There are mixed reports all over the web about the extent of this outage and what caused it, but I lost email, PIN messaging, Blackberry messenger, and had intermittent problems across all other apps on my device.

I was traveling with a Blackberry-wielding colleague.  Usually this would make us feel like business Samurai, ready for anything.  Yesterday, this overconfidence caught up with us.

Emails to my travel agent never made it through, and there was no rental car waiting for me in DC.  Even at Hertz, I had one of those “not exactly” moments, and it took an interminable half hour to get a car.  In transit from the airport to meet people for dinner, neither of us could get wireless data connectivity.  No ability to search for the hotel and restaurant by name.  No ability to use Google Maps to get directions.

Also, we kept dropping calls.  Even calling to get directions was not as smooth as usual, but eventually this is the “old fashioned” way we found our destination.

Me:  “I can see the mall on my left, and the XYZ company on my right.  No, I don’t know what street I’m on.”

Friend/colleague on other end:  “I think I know where you are; go 1 mile, then turn left.”  (Not so long ago, this would have seemed miraculously cool – like in the Matrix:  “I need an exit!” – now it seems pretty pedestrian and painful.)

This sort of event brings home how much we depend on technology on a daily basis.  And how quickly we’ve become spoiled by technological capabilities that are relatively new.  It reminds me of a sort of post-apocalyptic Stephen King novel, where the protagonist needs to find their way through the world using their feet and their wits, but they are suddenly bereft of all modern technology and convenience.

Without Google Maps (and frequent United flights!) I’m not sure I’d make it to Denver in The Stand.  I’d probably jog right past.  Would you make it?

Perhaps more importantly, this has me seriously questioning my loyalty to Blackberry.  From a competitive analysis perspective, this is a disaster for RIM. Two major outages in a week.  A general degradation in service over the past year.  Meanwhile, my partner is having great fun with his iPhone, and my wife’s Droid Eris seems pretty darn cool.

I used to have a BlackBerry at one of my jobs.  It had outages then as well.  Anyone suffered with this problem?  Why is this outage stuff still happening?  Isn’t it used by critical services people like Ambulance people and EMTS I think I once read.  Click here to read more.

Research In Motion Ltd. said late Tuesday its technicians are working to resolve e-mail messaging delays on its Blackberry smart phones in North and South America.

Phone calling and texting services appeared to be functional, but users in the Americas have been unable to send or receive e-mail messages. Would-be users tweeted their frustration on social networking site Twitter. Some said they also could not connect to the Internet.

Waterloo, Ontario-based Research in Motion Ltd. said in a statement it apologizes for any inconvenience experienced by customers.

It marks the second time in less than a week that BlackBerry users in North America have had to deal with e-mail outages on their Blackberry devices. Research In Motion said last Thursday that technicians had isolated and resolved the issue and were investigating the cause. RIM didn’t say how many users were affected or how long that outage lasted.

By our nature, web workers are dependent on online tools to do our work. Sometimes those tools become inaccessible, though. For example, recently a hosting outage took down 37signals’ Basecamp, an app that many web-based workers depend on heavily.

These outages can quickly become crises. How do you know what step to take next on a project if the project is entirely managed on a site that is currently down? How do you get in touch with a client if you usually only communicate through that same project management tool? You may not be able to work quite as fast without your tools, but these tips can help you keep the situation from escalating.

1. Have an alternative method of communication for clients and co-workers. It may seem obvious that it’s important to have multiple methods of communication, but it’s worth going beyond just having an email address for the people you work with. Depending on just what tools have gone down, there’s a possibility that an email server has gone down as well. Having a phone number or another form of communication available — even being able to send a message through a social networking tool — may be necessary to be able to get through to the folks you need to talk to.
2. Have a backup of project data. All your project information is in the cloud, right? Even if the service goes down for a little while, you’ll be fine, right? Not necessarily. There are plenty of scenarios that end with you having to recreate your project from scratch (as could happen if the app doesn’t come back up in time, or it goes bust). Having a backup of your information can offer extra peace of mind.
3. Choose a tool to keep working. Many outages are open-ended, because it can be tough to tell exactly what is necessary to bring an application back online.If you’ve got deadlines, waiting around for something to happen is not the best option. Ideally, there may be some aspect of your project that you can work on without logging in at all. If that’s not the case, there are a couple of options: consult your backup documentation or your client to see what would be most practical to work on in the mean time.
4. Discuss a new deadline. If you’re in a situation where you really can’t proceed with a project — maybe the only copies of important files are inaccessible — you may quickly realize that your current deadlines just aren’t going to be doable. If that’s the case, talk to your client or manager as soon as you can and explain the situation. If you aren’t sure how much of an extension you’ll need (a typical situation if you don’t know when your tools will be available again or if you’ll have to rebuild the project from your backup), make that clear so that you’ll have grounds for future discussions, if need be.

Unfortunately, the best methods for dealing with any kind of application outage require some prep work while your most important tools are still up — getting contact information for a client who can’t respond to the only email address you have isn’t going to move things along. In the worst-case scenario, when an outage has caught you entirely by surprise, it may be difficult to come up with a next step that will actually help you with the work. The best option may be just trying to get in touch with your client or co-worker and check what can be done at all. In the event that you can’t reach anyone, you may just have to wait out the outage and deal with the repercussions down the road.

What do you do when your web apps go down?

Image by Flickr user p4inkiller

As I wrote in an earlier post Zenbe Mail which was a promising product seems to continue it fall from a hopeful path. As I reported the free  Zenbe Personal has basically become abandon-ware in that it receives no updates, does not allow new sign-up, and judging from the posts I have been reading no user support either.

In that aforementioned post I down graded Zenbe to a “watch”, however due to some recent events I am down grading it again to my lowest rating of “Just say NO!”.

The reason for this is now on top of all of the neglect of the free product, their server for it has gone down due to a  maintenance issue. This happens, but no one was told the maintenance was taking place in the first place. Further I am seeing comments on the official blog post regarding  it that peoples inboxes now have the entire contents of their trash cans dumped into the inbox making for huge messes. Whats worse is the forum has been completely down for about a month now leaving users with little recourse to seek assistance.

Let me make one thing clear: though I only use the free Zenbe service this suggestion not to use the product applies to all Zenbe products and services. My logic is that if a team can not support and reasonably assure one products viability it is rather likely they lack the resources to maintain any of their products. I know I am not going to pay for something when it appears that the free product and the support forums become more useless on a regular basis. It’s a shame too, I really liked Zenbe.

We’re going into read-only mode as a safety precaution to clear some database storage issues. We hope this will be very quick, and will post here if anything additional occurs.  Sorry for the inconvenience.

09:15 PST – issue cleared, back out of read-only

ALERT! There is a Blackberry outage in North America.  I just spent the last half hour fiddling with my wireless settings, but it turns out it’s not just me.  Verizon, AT&T and TMobile customers have been hit… maybe this is just collateral damage from the vicious AT&T/Verizon 3G AdWar.  Recent commercials from both companies have been throwing it down; clearly, the gloves are off.

One of the original Map commercials by Verizon:

…And one of AT&T’s latest:

A post about Operation Chokehold popped up on (fake) Steve Jobs’ blog this morning. It seems some folks are just plain tired of AT&T giving excuses about their network. The straw that broke the camel’s back came when AT&T floated the idea of instituting bandwidth limitations for data accounts. Now, someone hatched the idea of organizing enough users to bring the whole network down by maxing their bandwidth at the same time.

We’re not quite sure what to think about this. Our friend Google told us that there’s plenty of press already out there regarding Operation Chokehold so it’s not beyond comprehension that this could have an effect on the network. On the other hand, AT&T already knows about it and we’d wager they’re working on a plan to mitigate any outages that might occur.

As for the effectiveness of the message?  We’d have more sympathy for AT&T if they didn’t have exclusivity contracts for their smart phones (most notably the iPhone). And if you’re selling an “Unlimited Plan” it should be just that. What do you think?

[Thanks Bobbers]

[Headlock photo]

Just to let you all know, the Main website will be down for an hour or two while I try and update to DotNetNuke 5.2.

Wish me luck!

- Andrew

Update

09-Dec-17

For the most part the upgrae went well, I did run into a few errors that I needed to search the main DNN website for “Error after install 5.2 Host Settings“, but the site is upgraded! Yeah!

Unfortunately, the upgrade did NOT help in fixing the main issue FOR the Upgrade “Can’t Edit/Update Page Settings fields“, which is drivnig me nuts!

So that the UT TeleCampus can continue to provide a stable learning environment, 2:00-6:00am Central time on Tuesdays is reserved for minor preventive maintenance. This maintenance window is scheduled during the lowest usage time for the system. Blackboard may or may not be available during this time, depending on whether maintenance is necessary. Whenever possible, this time will be utilized to perform all minor maintenance.

Today we got the story about Amazon’s outage of part of its EC2 cloud. In this post, I’ll examine what happened and what you can do to avoid the same fate.

Data center power distribution is even more critical today due to the spread of the data center cloud. Due to the condensing effect of the cloud, failure of a poorly designed power system can wipe out far more customer service than ever before.

The failure of the EC2 section was caused by the failure of one PDU, then the failue of the backup while the first one was being repaired. Two almost simultaneous failures of PDUs makes me wonder if Amazon is buying them from a shady guy in a back alley (hey mister, ya vant to buy a PDU?).

The problem here was that the reliability of the PDUs wasn’t properly characterized. In order to make the decision of n+x (n+1, n+2), you have to determine what the chances of multiple failures leaving you without enough capacity to operate are. You need enough redundancy to be sure you can repair a failure before having a second one. The sad part of this is that this is just another PR black eye for cloud reliabilty (especially when you add in 5 hrs to restore customer workload).

Amazon swings, and they whiff.

Vern, SwiftWater Telecom

You can trust us!  Oops, sorry about that.  Click here to read more.