VAXen, my children...
VAXen, my children, just don't belong some places. In my business, I am frequently called by small sites and startups having VAX problems. So when a friend of mine in an Extremely Large Financial Institution (ELFI) called me one day to ask for help, I was intrigued because this outfit is a really major VAX user - they have several large herds of VAXen - and plenty of sharp VAXherds to take care of them.
So I went to see what sort of an ELFI mess they had gotten into. It seems they had shoved a small 750 with two RA60's running a single application, PC style, into a data center with two IBM 3090's and just about all the rest of the disk drives in the world. The computer room was so big it had three street addresses. The operators had only IBM experience and, to quote my friend, they were having "a little trouble adjusting to the VAX", were a bit hostile towards it and probably needed some help with system management. Hmmm, Hostility... Sigh.
Well, I thought it was pretty ridiculous for an outfit with all that VAX muscle elsewhere to isolate a dinky old 750 in their Big Blue Country, and said so bluntly. But my friend patiently explained that although small, it was an "extremely sensitive and confidential application." It seems that the 750 had originally been properly clustered with the rest of a herd and in the care of one of their best VAXherds. But the trouble started when the Chief User went to visit his computer and its VAXherd.
He came away visibly disturbed and immediately complained to the ELFI's Director of Data Processing that, "There are some very strange people in there with the computers." Now since this user person was the Comptroller of this Extremely Large Financial Institution, the 750 had been promptly hustled over to the IBM data center which the Comptroller said, "was a more suitable place." The people there wore shirts and ties and didn't wear head bands or cowboy hats.
So my friend introduced me to the Comptroller, who turned out to be five feet tall, 85 and a former gnome of Zurich. He had a young apprentice gnome who was about 65. The two gnomes interviewed me in whispers for about an hour before they decided my modes of dress and speech were suitable for managing their system and I got the assignment.
There was some confusion, understandably, when I explained that I would immediately establish a procedure for nightly backups. The senior gnome seemed to think I was going to put the computer in reverse, but the apprentice's son had an IBM PC and he quickly whispered that "backup" meant making a copy of a program borrowed from a friend and why was I doing that? Sigh.
I was shortly introduced to the manager of the IBM data center, who greeted me with joy and anything but hostility. And the operators really weren't hostile - it just seemed that way. It's like the driver of a Mack 18 wheeler, with a condo behind the cab, who was doing 75 when he ran over a moped doing it's best to get away at 45. He explained sadly, "I really warn't mad at mopeds but to keep from runnin' over that'n, I'da had to slow down or change lanes!"
Now the only operation they had figured out how to do on the 750 was reboot it. This was their universal cure for any and all problems. After all it works on a PC, why not a VAX? Was there a difference? Sigh.
But I smiled and said, "No sweat, I'll train you. The first command you learn is HELP" and proceeded to type it in on the console terminal. So the data center manager, the shift supervisor and the eight day operators watched the LA100 buzz out the usual introductory text. When it finished they turned to me with expectant faces and I said in an avuncular manner, "This is your most important command!"
The shift supervisor stepped forward and studied the text for about a minute. He then turned with a very puzzled expression on his face and asked, "What do you use it for?" Sigh.
Well, I tried everything. I trained and I put the doc set on shelves by the 750 and I wrote a special 40 page doc set and then a four page doc set. I designed all kinds of command files to make complex operations into simple foreign commands and I taped a list of these simplified commands to the top of the VAX. The most successful move was adding my home phone number.
The cheat sheets taped on the top of the CPU cabinet needed continual maintenance, however. It seems the VAX was in the quietest part of the data center, over behind the scratch tape racks. The operators ate lunch on the CPU cabinet and the sheets quickly became coated with pizza drippings, etc.
But still the most used solution to hangups was a reboot and I gradually got things organized so that during the day when the gnomes were using the system, the operators didn't have to touch it. This smoothed things out a lot.
Meanwhile, the data center was getting new TV security cameras, a halon gas fire extinguisher system and an immortal power source. The data center manager apologized because the VAX had not been foreseen in the plan and so could not be connected to immortal power. The VAX and I felt a little rejected but I made sure that booting on power recovery was working right. At least it would get going again quickly when power came back.
Anyway, as a consolation prize, the data center manager said he would have one of the security cameras adjusted to cover the VAX. I thought to myself, "Great, now we can have 24 hour video tapes of the operators eating Chinese takeout on the CPU." I resolved to get a piece of plastic to cover the cheat sheets.
One day, the apprentice gnome called to whisper that the senior was going to give an extremely important demonstration. Now I must explain that what the 750 was really doing was holding our National Debt. The Reagan administration had decided to privatize it and had quietly put it out for bid. My Extreme Large Financial Institution had won the bid for it and was, as ELFI's are wont to do, making an absolute bundle on the float.
On Monday the Comptroller was going to demonstrate to the board of directors how he could move a trillion dollars from Switzerland to the Bahamas. The apprentice whispered, "Would you please look in on our computer? I'm sure everything will be fine, sir, but we will feel better if you are present. I'm sure you understand?" I did.
Monday morning, I got there about five hours before the scheduled demo to check things over. Everything was cool. I was chatting with the shift supervisor and about to go upstairs to the Comptroller's office. Suddenly there was a power failure.
The emergency lighting came on and the immortal power system took over the load of the IBM 3090's. They continued smoothly, but of course the VAX, still on city power, died. Everyone smiled and the dead 750 was no big deal because it was 7 AM and gnomes don't work before 10 AM. I began worrying about whether I could beg some immortal power from the data center manager in case this was a long outage.
Immortal power in this system comes from storage batteries for the first five minutes of an outage. Promptly at one minute into the outage we hear the gas turbine powered generator in the sub-basement under us automatically start up getting ready to take the load on the fifth minute. We all beam at each other.
At two minutes into the outage we hear the whine of the backup gas turbine generator starting. The 3090's and all those disk drives are doing just fine. Business as usual. The VAX is dead as a door nail but what the hell.
At precisely five minutes into the outage, just as the gas turbine is taking the load, city power comes back on and the immortal power source commits suicide. Actually it was a double murder and suicide because it took both 3090's with it.
So now the whole data center was dead, sort of. The fire alarm system had it's own battery backup and was still alive. The lead acid storage batteries of the immortal power system had been discharging at a furious rate keeping all those big blue boxes running and there was a significant amount of sulfuric acid vapor. Nothing actually caught fire but the smoke detectors were convinced it had.
The fire alarm klaxon went off and the siren warning of imminent halon gas release was screaming. We started to panic but the data center manager shouted over the din, "Don't worry, the halon system failed its acceptance test last week. It's disabled and nothing will happen."
He was half right, the primary halon system indeed failed to discharge. But the secondary halon system observed that the primary had conked and instantly did its duty, which was to deal with Dire Disasters. It had twice the capacity and six times the discharge rate.
Now the ear splitting gas discharge under the raised floor was so massive and fast, it blew about half of the floor tiles up out of their framework. It came up through the floor into a communications rack and blew the cover panels off, decking an operator. Looking out across that vast computer room, we could see the air shimmering as the halon mixed with it.
We stampeded for exits to the dying whine of 175 IBM disks. As I was escaping I glanced back at the VAX, on city power, and noticed the usual flickering of the unit select light on its system disk indicating it was happily rebooting.
Twelve firemen with air tanks and axes invaded. There were frantic phone calls to the local IBM Field Service office because both the live and backup 3090's were down. About twenty minutes later, seventeen IBM CEs arrived with dozens of boxes and, so help me, a barrel. It seems they knew what to expect when an immortal power source commits murder.
In the midst of absolute pandemonium, I crept off to the gnome office and logged on. After extensive checking it was clear that everything was just fine with the VAX and I began to calm down. I called the data center manager's office to tell him the good news. His secretary answered with, "He isn't expected to be available for some time. May I take a message?" I left a slightly smug note to the effect that, unlike some other computers, the VAX was intact and functioning normally.
Several hours later, the gnome was whispering his way into a demonstration of how to flick a trillion dollars from country 2 to country 5. He was just coming to the tricky part, where the money had been withdrawn from Switzerland but not yet deposited in the Bahamas. He was proceeding very slowly and the directors were spellbound. I decided I had better check up on the data center.
Most of the floor tiles were back in place. IBM had resurrected one of the 3090's and was running tests. What looked like a bucket brigade was working on the other one. The communication rack was still naked and a fireman was standing guard over the immortal power corpse. Life was returning to normal, but the Big Blue Country crew was still pretty shaky.
Smiling proudly, I headed back toward the triumphant VAX behind the tape racks where one of the operators was eating a plump jelly bun on the 750 CPU. He saw me coming, turned pale and screamed to the shift supervisor, "Oh my God, we forgot about the VAX!" Then, before I could open my mouth, he rebooted it. It was Monday, 19-Oct-1987. VAXen, my children, just don't belong some places.
-- Mike O'Brien, obrien@aero.org