Adding Disk to my Xen VMs

Ok, I added some space to one Xen VM this morning.  Wasn’t too bad, although there are so many ways to do this in Linux.  I wanted to do something simple.  Create a file on the host O/S (grant), attach it to the VM (sherman) and then just add it into the filesystem.  I was surprised at the number of commands I had to run, but they all made sense:

  1. Use dd to create a file on grant that will become a new disk on sherman
  2. Shutdown sherman using xm shutdown
  3. Edit the sherman configuration file under /etc/xen to add the new file as another disk (took me a minute to remember the disk naming conventions)
  4. Re-start (“re-create”) sherman so that it recognizes the disk
  5. Use fdisk to partition the disk on sherman using a primary partition and changing the type to 8e so that the Linux LVM can manage it (Look at all the types!)
  6. Use pvcreate to create a new physical volume on the device partition
  7. Use vgscan to look at the volume groups
  8. Use vgdisplay to display the current volume group
  9. Use vgextend to add the new physical volume to the volume group
  10. Use lvscan to verify the addition
  11. Use lvextend to extend the logical volume onto the newly added space
  12. Use resize2fs to extend the filesystem to the newly extended logical volume space

Whew!

Many thanks to the following:

  1. Re: Adding more space to the system without using LVM in the host
  2. Extend Your MythDora Video Partition using LVM
  3. Extending a logical volume

Back with a Xengence

I’ve recently come into possession of 2 fine Dell 2950 servers and I’ve been trying to figure out how I want to configure them.  I had them sent to me without any O/S on them, so as to make the decision a bit harder :-)

What I’d like to do is split each machine into 2 virtual machine “containers” that I can run a variety of images / configurations in.  I played around with using VMWare ESX Server, but balked a bit at the price of the management server.  After that I started to look into the Xen software.  Basically it started to come down to whether or not I wanted to boot into a thin hypervisor, or run a straight O/S “underneath” the hypervisor.

I decided to run Linux underneath any hypervisor — so at that point it came down to which Linux to run.  I see a lot of raves for Ubuntu and CentOS on the blogs, but in the end I decided to give Oracle’s Unbreakable Linux a whirl.  I downloaded  the 5 ISO’s, “remembered” how to burn ISOs to CDs and then booted the Dell’s using the CDs — worked like a champ.

And, lo and behold, Xen was included in the distro — no need to install it.

So began my fun with Xen.

The first challenge I had was trying to get an O/S installed into the Virtual Machine I created.  There’s a graphical Virtual Machine “manager”, but it’s not much more than a wrapper around the base Xen (xm) commands.  I find this to be a pain with all of the virtualization managers — guest O/S installation.  The installer asks me for a URL or FTP of the ISO images — and since I’ve just configured these 2 machines from scratch, I don’t have that stuff set up.  What I have is the 5 CDs — I want to use them.  (Actually, I would just like it to clone the installation I’ve just done — I don’t want to install the O/S again — just use the same one I already have).

I try using the CDs and have all kinds of fun.

First problem is the graphical console to the VM — the mouse position appears “relative” — I can’t navigate to certain areas of the virtual screen.  This is a “known” issue, so I give up on the graphical installer and use the text mode.  Everything is fine and dandy until it asks me for the 2nd CD.  I push the eject button on the drive and nothing happens.  Here’s where I learn about how the CD drive is managed between the host O/S and the various VMs.  I can understand the challenge here — you need some way to attach the drive to a specific VM and manage conflicting access demands.  I get close using the block-detach force and block-attach commands — I get the 2nd CD mounted, but I run out of luck on the 3rd CD — no idea why.  What I really want is a DVD install disk instead of the 5 CD images.  I’ve got a DVD burner in my laptop — should be a piece of cake — just locate the DVD ISO…..  Not a lot of luck there either.

I finally find this (Create RedHat DVD from CD ISO Images) which gets me 99% of what I need instructions-wise.  Once I get my DVD ISO and burn it — I’m on my way.  One minor thing — I needed to go into the BIOS and enable CPU Virtualization support (why isn’t it on by default? — some other issue?).  Once that’s done, I get my VMs up and running.

One other odd thing — when you shutdown a Xen VM there isn’t an obvious way to restart it — it disappears from the graphical manager.  Actually, you “re-create” it, re-using the configuration file from when you “created” it in the first place.  So, xm create sherman will actually restart the sherman VM if it has a valid config file in /etc/xen.  Seems odd to me as “create” implies “create from scratch”.  Now with sherman and mcpherson running on top of the grant domain0 my next task is to add disk space to them….

I know why it’s always the database

I got to this post via Elisa Gabbert and Jeff Hunter after doing my daily reading which started with Jeremy Schneider’s excellent post on a recent healthcheck.

Jeff sums it up best in wondering why, when an application is a complex combination of system components, is the database always blamed for poor performance?

I’ll tell you why.

People ask for data. And they don’t like the first set of data they get, so they ask again, and then they browse back and forth through the data.

And where is the data? In the database.

People don’t ask the network, or the disk, or the CPU or the memory for their data. And despite the amount of Javascript, PHP and Java code in their browsers and middle-tiers, they don’t ask their browser or application server either.

The database is responsible for serving data quickly. And since you’re the DBA, when the data doesn’t come back quickly enough, you’re asked to “look at the database”.

BTW, this is one of the reasons I’m a huge fan of constant monitoring — I like to see if the database is acting differently from when “everything was ok”. This requires some kind of historical view or “baseline”. Some way to look back in time to yesterday, or last week at this time, or last month during the monthly close, etc. If the profile during the “slow response” is the same as it was during the “fast response”, generally there’s nothing wrong with the database overall — there may be something going on with that particular request, but you know how to fix those.

And, unfortunately, the database is ridiculously well instrumented, and you’ve convinced everybody that you can quickly see everything going on in the database. Oracle’s event model has really no counterpart on the other parts of the application stack.

Check out Jeremy’s post again — look at how “easy” it was to see that AIX was hopelessly mis-configured and was using a ton of swap space.

I’ve never really liked the idea of swap space — Why would you ever want anything at all swapped out of memory on a database server? Do you monitor memory usage on your PC? What happens when you start using swap space? You go to the Kingston web site and purchase more memory, that’s what. I used to get annoyed at H/W O/S vendors when I’d ask why we needed so much swap. My favorite answers:

  1. Just in case
  2. We’ll crash / panic otherwise
  3. We “pre-swap” code text images to speed up swapping
  4. The system becomes more efficient during non-peak usage as it can handle more processes then as long as they’re idle

This is nuts.

Another fun excuse I’ve heard is when there is significant data caching at the middle tier and web tier — “it takes a long time for the database to give us data to fully populate the cache, so queries are slow”.

This is why I advocate SIMPLE architectures for data-intensive applications. Fewer non-database places where things can go wrong. Fewer times that the DBA gets asked about “why isn’t my data coming back fast enough” and has to spend time finding out which part of the application stack isn’t pulling its weight.

Superscript

…back to Change Management.

In my earlier post, I talked about the strong change management ethic at my current company.  It was interesting to see how my predecessor here adapted database schema evolution to that ethic.

I need to add here that I find “ancient” software development organizational ideas like “chief” or “surgeon” programming teams to be fascinating.  In particular because each model describes “super” programmers who end up being responsible for 80-90% of the code anyway due to shear talent and dominant personality.  Almost everywhere I’ve gone has had a local expert who seemed to qualify for that kind of title.  The relevance of ideas from Mills and Brooks is commented on here.

The biggest thing I’ve taken away from those ideas of “super” programmers is that, as a manager, I need to help every member of my staff become as close to “super” as possible.  That means removing as many non-productive, non-programming, non-learning tasks as possible.

Another way to put this is to find out ways to reduce the effort and work required to do non-programming tasks.

This is where the DDL generation scripts came into the picture.

When I arrived, every programmer hand-coded the scripts which were checked into the source code control system.  Everyone had to remember to include the source code control tags, and script names weren’t obvious.  Also, it wasn’t always clear that every changed object and/or procedure was being captured with every release.

To address those issues, I:

  1. Required that all code checked into the source code control system had to be generated via the DDL generation scripts.  We modified the scripts to include the source code control tags automatically.  We also did things like include primary key indexes, grants and initial stats analysis in table creation scripts.  We also injected source code control tags into the stored procedure and package code so that they could be seen when browsing the database.
  2. Modified the scripts to include the object type, schema name and object name in the generated file name.  So, a script that created the SCOTT.EMP table ended up being called mk-tbl-scott-emp.sql  (We used mk for created, rm for drop and ch for alter).
  3. Turned on DDL auditing for all objects in the development database.  This allowed us to double-check and make sure we captured and verified which objects were changed leading up to a release.

A note on DDL auditing — I like to try and have this turned on in every database I work with.  Development, Test, QA, Production, you name it.  I think it’s always useful to have a log of structure changes to the database — and why code up a special trigger when the database will track it automatically?  The only objection I’ve heard to this is space — and the space required is so ridiculously small.  Sometimes people complain about performance — and I say that if you have a system that relies on 1000′s of DDLs per day, you’ve got a system that needs work.  Who does DDL as part of an OLTP transaction?

Our biggest remaining challenge with this approach was how to create the “master” installation “super” scripts.  I’ll cover that in a later post…

(One other concern is the shear number of files this approach can generate.  If Oracle did this for the SYS schema, you’d have over 4,000 individual files.  On the other hand, this works well when you only need to re-create a single object without searching through 4-5 cat scripts to find the object definition.  Other systems use object definition files which are then read by specialized programs to generate installation scripts — that seems like a lot of work for an internal corporate application).

Talkin’ ‘Bout My Generation…

With apologies to The Who.

I’ve written before about my obsession with sql.bsq :-)

The genesis of which began during a stint at a customer site where I simply providing on-call support without any specified tasks.  This site made heavy use of Pro*C, with a common problem being making sure that local variables were declared with the same precision as their database column counterparts.

I couldn’t understand why the developers were declaring their variables by hand — didn’t the data dictionary have all of the information to generate this stuff?  And didn’t SQL*Plus have an ability to spool output to a flat file?  How hard could it be to just ask for the table definition and turn it into a .h file?

And so, my first code generation script:

rem
rem SCCS
rem File Name : %W%
rem Check-in Date :
rem SCCS Version : %I%
rem Path : %F%
rem NAME
rem procvars.sql
rem DESCRIPTION
rem Select script that will create Pro*C variable declarations for
rem a given table. All variables except NUMBER are declared as
rem VARCHARs (NUMBERS are declared as long or double, depending on
rem their data scale).
rem
rem This script spools its output to a file with the name
rem [table_name]_vars.h
rem HISTORY
rem Who When Reason
rem —— —— —————————————–
rem Dom 1/10/94 Creation
rem

The interesting thing about this script (I liked source control, even back then), was that it started me down the path of exploring the data dictionary.  I was fascinated by the data dictionary in Oracle 6 — it had EVERYTHING in it.  I’d worked with other databases before (dBase III & IV, SQL/DS), but none of them had anything as detailed as the Oracle data dictionary.  I loved the fact that it was also self-describing — selecting from dba_views gave the names of every view.  To this day, I don’t understand people who have those huge posters of the data dictionary — why would you put up something which would become obsolete so quickly when you could just query the system for the info?

Anyway, the script referenced all_tab_columns, and I wondered why it was so slow — and that’s when I started digging into tab$, col$ and sql.bsq.  I liked going after the base tables instead of the views — and it sped up things considerably.

After that, I began writing scripts (using the same framework) to generate DDL for all kinds of things — since at the time I wasn’t familiar enough with the Unix strings command to pull DDL from Export dump files.

I still have those scripts on my hard drive — they’ve followed me everywhere I go.  Scripts with ALL CAPS names limited to 8 characters long.

MKINDEX.SQL
MKSEQUEN.SQL
MKSYNONY.SQL
MKTABLE.SQL
MKVIEW.SQL

When I first was confronted with the need to maintain source code controlled database scripts, these old scripts became the basis for how I would attack the problem…

Why code stuff by hand when you could generate it?  And not out of thin air either, but based on the actual definition in the system.