Dressed the the nines

My good friend Moans (who posts to his blog about as often as I do), recently wrote up his current opinion on high availability here.  His post resonated with me, as I’ve had a fair amount of experience with system availability over the years.  While ruminating on his post, I dug up some excellent work on the subject from HP (one of the biggest proponents of 5nines during the First Internet Gilded Age).  This presentation by William Sawyer is particularly good, as well as this paper by Tzvi Chumash.

I’m particularly proud of the reliability of the system I’m currently a part of.  From a good design, to talented engineers, to very good operational folks, we have  a system that consistently achieves high availability.  One thing that I like about the system is that it isn’t very complex.  And it’s especially simple in its redundancy features.

I have to agree with most of the authors from above.  Designing, building and maintaining high-availability features can be very costly and error-prone in and of themselves.  A lot of ideas look good on paper, and sound good coming from the vendor, but in the end, your own people have to own and maintain them.  And if they can’t, for whatever reason, you’ve just wasted a ton of time and money.

This doesn’t mean that you don’t need higher levels of availability — it just means that you need to make sure you capture all of the costs.  That includes training and drills for the people responsible for the system.

High availability is achievable, but a realistic assessment of how to get and stay there really makes it easier.

(I also found this article interesting)


Doug Burns has written an article here that talks about what he is calling a Development DBA.  Further commentary here by Tom Kyte.  I’ve talked about this role in the past (including a paper on the subject at the 2006 Hotsos conference). 

Doug makes some excellent points in his article, describing the triangular relationship between application developers, production DBAs and the development DBAs.  In my paper, I suggest that one of the main challenges with defining the necessary role is the ambiguity of the title.  Tom often calls the role a development DBA or DBA developer.  Personally I like the term Database Engineer, as I think that developers responsible for the databases during development should exercise engineering-based skills.

Doug observes that many of the more experienced DBAs tend to be Production DBAs — and that’s a shame.  I like to say that the skill sets of Database Engineers and DBAs overlap — but that Production DBAs focus on High Availability, Backup / Recovery and Space / System Management, while Database Engineers focus on Schema Design / Development and Database Coding.  Experts are required in both areas.

About the only thing I disagree with in Doug’s article is the discussion about database ownership and responsibility.  In my opinion, a database engineer should retain ownership and responsibility for the availability and performance of the development database.  That doesn’t mean you prevent developers from accessing it, but it does mean that you have the responsibility to make sure that developers don’t shoot themselves in the foot or prevent other developers from getting their work done.  Doug is right about being helpful — try to understand what the developer is trying to do and work together to figure out a solution — don’t be a “scowling brick wall” :-)  Take pride in the ownership and be part of the solution.

Mean Mr. Modeler

With apologies to the Beatles.  I read a post by Tom Kyte the other day about a data model that had passwords as clear-text, unique keys on a table and even used them as foreign keys!  After reading it, I was reminded of a conversation I had with my wife, Wendy, about how she’s seeing a revival of interest in enterprise data modeling due to the SOA / BPEL fad surge ;-).

Despite some people’s opinion to the contrary, I’m less of a purist than I sound.  I love data modeling, but I recognize that it can be an endless quest for non-achievable perfection.  I once worked on a project to fully model all of the financial reporting attributes for a Fortune 500 company — the project took a full year to model, at which point the model was obsolete :-).

After that debacle, I’ve come to realize that data modeling (and the eventual goal of a data model, which is its representation in the physical implementation world) has a heavy dose of artistry along with the science of analysis and normalization.  And I’m not sure how to teach the art — so much of how to model something includes the requirement to understand how the data will be accessed and processed.

A common problem I encounter is a data “model” produced by programmers who have no experience in the art.  These are usually physical models which mimic flat files or maybe, just maybe an advanced structure like a linked-list or tree.  Most of the time they suffer greatly from poor attribute definition, lack of any keys, and no appreciation for scalability or schema evolution.

In the end, those last 2 items appear to be the biggest problems.  I usually handle them with 3 questions every time I’m presented with a physical model:

  1. How will this handle concurrency?
  2. What will it look like with 1 million entries?  10 million?  1 billion?
  3. How hard will it be to add attributes?  Add dimensions (versions, history, etc.)?