c.R.E.A.M. (The speed of light Rules Everything Around Me)

No joy in Brooklyn tonight as I wait for the file transfer to trigger the beginning of the end of my time here to complete.  It's robocopying as fast as it can but you can only push electrons so fast. 

On the upside, the transfer is reading as a pretty consistent 98 to 99.5 Mbps on the performance window and I've chosen to interpret that as a sign of good wiring. 

I'm pruning a lot of the general musings out of the documentation I've been writing so far.  As much as I know future generations would appreciate my diary-like writing style I don't think it'd be much help to someone trying to figure out why the next migration isn't working right. It's been suggested that I have a 'holistic' style of writing and I'm really making an effort here to put together something functional. 

The other day someone asked me what my experience with Linux is and I had kind of a hard time answering.  The truth is that I have extensive Linux experience at home and school but I was a having a hard time answering the question without instead talking about the applications I've run on it. Most of what I've done with Linux directly seems so pedestrian: editing text files and managing processes. The answer I gave was to rattle off the various installations I have around my house and what their primary function is but I feel like that was a pretty weak answer.

So in that spirit I'm also giving a thought to how I'm going to describe this whole internship project to future employers.   

Final sprint

Very close now to having everything migrated and working perfectly.  The only thing that isn't great is that a VM needs to be re-copied over and of course it's the one that's like 200 GB. I tried using an external hard drive enclosure, as that would be about four times as fast as going over the local network, but the one we have isn't compatible with Windows Server 2012 R2. If that weren't enough the finicky drive on that server finally clicked it's last which means I only have one copy of this gigantic VM I need copied over so I'd better be quick about it.

So over the 100 Mbps network we go.  In case I ever need to robocopy a bunch of large files over a slow connection again: There's a robocopy flag called /IPG:xx that delays for xx milliseconds every 64k.  This shouldn't be very large (I'm using 10 here) but it breaks up the transfer enough that it doesn't totally overwhelm all other network traffic.

All said I should be able to put this project to bed pretty soon

Gold-ish

So after a long weekend of frustration for much of the team things have been returned to normal functionality here.

I think I've found the issue and developed a new appreciation for how important it is that your applications don't just fail silently.

Christmas!

It's here!  And it's amazing.  Well, mostly amazing.  It takes a very long time to boot up and I'm in that new-OS constantly rebooting period.

I got the RAID set up and Server 2012 installed without issue.  Now it's just a matter of assembling it and I've already got a pretty good road map for that.

Or rather I did, but today our Confluence isn't working and I can't access my notes.  I've got an older version that I exported to hand in as my capstone documentation and that was pretty up to date.

Now...   Well frankly there's a fair amount of waiting.

There was a slight crisis on the old dev server due to it running out of space on the one remaining good storage drive.   Since I retired the drive that was having controller issues there's been a few additional projects spun up and there just isn't room for it all.   I was able to move some stuff off for now and it's very good news that the new hardware is here already.

Going dark

Been quiet lately as I'm trying to nurse out the last of the hours left in my internship.  Tried to do a switchover to the new hardware for the file server and everything worked perfectly.   Tried to do the same for the build server and I hit a snag that required a total, multi hour, return to the old hardware.  

The intranet site was working, the VMs were building and installing, but for some reason the step in the scripts where they:

1. Revert the VM to a clean snapshot
2. Restart the VM
3. Start the build

Wasn't firing properly.  The VM would suspend but wouldn't restart.   If I manually clicked start the reversion would happen and the process would proceed normally.  I'm still not sure what caused it and I've stressed everyone else out enough for now so right now I'm researching and just fixing things here and there when they pop up. 

I'll try again next week, if I can figure out what happened. 

Not quite Christmas

No hardware to play with today.

I'm still trying to make a new version of the current build environment on a more recent operating system than 2003. I had a remote desktop into my new VM with a build going at almost any given time today and I've managed to resolve the issues that prevented the .NET portion of the compilation to succeed and now it's the very beginning of the VB 6 portion of the code that is failing.  Following some advice from a Microsoft forum I tried upgrading to service pack 6 but that didn't fix the issue.

In other news I wrote a Powershell script today that I should have written weeks ago that pulls the environment variables from a remote computer and saves them as a csv.

OMG I can't believe that the hardware will be here next week.

Today has been a huge reminder of the vast universe of things I don't know.  I discovered that the source of some errors I've encountered is due to the way post-Vista operating systems handle sessions and that is almost literally everything I understand about it.

Your listed functionality got in my upgrade path

Made some good progress today getting my potential-upgrade VMs working.   I was able to get the build process going without issue and while I did hit a snag I found the solution pretty easily.   Applying the solution has meant my VM has been tied up by Windows Update for a long time but I'm pretty sure I'll be good to go afterwards. 

I've been reading through the configuration section of the manual for the server that I'll have come Monday and it seems both exciting and awful.  The bad news is that I might not have all of the functionality I expected because of Lenovo's Features On Demand model where the hardware they send you is capable of some function but in order to unlock it you've got to fork over a bit more cash.  I might not have an integrated RAID controller after all, but I'll know for sure either way soon. 

Misc updates

My drive letter swap gambit worked great with one small exception.  The automatic nightly build failed that evening and I haven't figured out why or been able to reproduce what happened.  Things are working just fine now.

I've written a Powershell script to make it easier to remove computers from the domain.   Currently whenever bringing up a new version of a VM that shared a name with an old one there's an annoying process of removing it from the domain, rebooting, re-adding it to the domain, and rebooting one more time. To streamline the process I wrote the following:

1. Import the Active Directory Module
1. If that fails try and activate that feature using DISM
1. If that fails give the user instructions on how to add the feature using Windows dialogues 
2. Tries to import the Active Directory Module again
2. Removes the given computer from the AD Directory

The script is meant to run on Desktop OSs because I haven't figured out how to get Active Directory tools for Powershell installed on a server that isn't a domain controller.  There's a group of features called ADRMS that looks promising but is a little outside my scope for now. 

One of my suspicions about the in-house build tool were confirmed by Bruce in that there's no known example of it running on anything other that Server 2003.  Bruce tackled it and got a version running on Server 2008 successfully but it broke compatibility with 2003 so we've reverted to the old version for now.  

Some of my hardware has arrived so I've got a small stack of drives and RAM at my workstation.

I can't wait for Monday.

It's harder than it looks

Today I've spent a lot of setting things up that need time.  Robocopy VMs.  Zip up other VMs.  Clear out old snapshots. Resource intensive operations on multiple machines at once. I'm like a conductor trying to keep everything going without overloading anything.

At one point Bruce came over and asked if I was busy and, to be honest, it's a complicated question. I'm doing a lot of things but it sure doesn't look like it so I said no.  Then he gave me another project that started with unzipping another VM on a different server and that is going to take 45 minutes.

Once my delayed action items are done I'm going to do a drive letter shuffle on the build server and hopefully it'll work like the old trick where you pull out a tablecloth without the dishes winding up on the floor.

Delivery date for the new hardware: June 27th!   Very exciting!

Mutiny

Hello to my probably accidental Latvian readers!

The retiring build server is on to us and is lashing out, throwing disk controller errors and suddenly dismounting the drive that holds the virtual machines at random intervals.  I understand it's frustration but I wish it were going out with more dignity.

I'm trying to robocopy the VMs onto another drive before the next tantrum to buy us another week or so of functionality.

The new hardware ships on Thursday.

Gingerly treading along

Solved an issue haunting some of the VMs with sysprep.   If VMWare is complaining about duplicate MAC addresses you just might have a UUID conflict instead.  I might have time to generate an answer file that would smooth the process but there's a lot going on as it is. 

Rather than continue to push the current host holding the build VMs with my tinkering I've started instead powering up a VM to use as an upgrade template on my own workstation (since it already had VMWare installed).  It's just an older i3 with 4GB of RAM so I don't know that it'll actually be much faster but it'll at least be isolated from the build server.   

Only one more week till it ships

I'm feeling very possessive about this new hardware, even though it's only going to be in my care for a very short while.

Ping...

Came in to the VM host server undergoing a check disk because of increasing instability.  It found enough bad blocks that the report was cut off in Event Viewer.

Tried to get a build going on my new VM by kickstarting it from the command line but it just wasn't having it.  The build would fail right away after throwing a cryptic error. Spent a while poking through source code trying to figure out the right parameters to pass to the build tool but no luck today.

Thursday I'm going to revisit the overall configuration and make sure I didn't miss an environment variable.  When I started I thought this project was going to be like re-building a house, but now it feels more like re-baking a cake. There's a black-box element to the project that has led to me discovering all new things that I didn't know.

Every now a fan will start spinning in my workstation and it'll be a bit alarming, but not half as bad as when it quickly spins down after it's been going so long I've gotten used to it.    Twice today I had a input mismatch as my ears were telling me that the computer had shut down suddenly but I could clearly see the screen was still on.

Progress for whom?

My 2008 R2 VM is ready for testing. Got the rest of the programs it needed installed today and learned some valuable lessons about subversion along the way. I'm going to add "Successfully installed Visual Studio 6.0 on Windows Server 2008 R2" to my resume.

Testing is stalled a bit, however, since there's something else afoot. Instead of that I got to run a build straight out of Visual Studio 2010 for diagnostic purposes. Whatever is happening is somewhere between a query and a concern and has something to do with the software building itself so that's taken priority.

There was also a VMWare-related MAC Address issue that I think I've successfully diagnosed but this other issue has sucked most of the oxygen out of the room so I'll probably have to wait till tomorrow to check.

I'm very excited to get my hands on the new build environment hardware they've ordered and let the current one retire gracefully.

Rounding out

Got a bunch of little things done today:

• Edited cmd and bat scripts so that they display their name at the start of their run and in the title bar of the command prompt window.  Then got them checked in to Subversion.
• Opened up a LaCie Hard Drive enclosure to see if I could swap out the failed drives inside and discovered four ancient IDE drives that are no good to anyone. 
• Disabled Windows Update on the build VMs by disabling the service (2003) or local policy (2008)
• Cloned from a Master VMWare VM and started getting it set up to potentially replace one of the development VMs.  Midway through an install it BSOD itself and I had to start over.  This time I'm taking snapshots at a pretty brisk pace. This project is going to take a few days but if I get it right should give me a template to upgrade multiple VMs with minimal fiddling. 

It's been a fairly productive day. 

Holding pattern

After a few days out sick I'm back in the saddle and doing some lower priority type projects because even though my new hardware was approved it's not even going to ship for two weeks.  Documented some more scripts, got the necessary Service Pack for SQL Server 2008 to update one of the VMWare servers.

With a floor like this

After some review I've determined that the cost of moving the build environment to the cloud is just too high.  Here's the conclusion of my research:

AWS

Because meeting the specific needs of the build environment would require a dedicated host on AWS the cost becomes prohibitive almost immediately.  The least expensive option (the compute-optimized c4 dedicated host) alone would cost between 37k and 72k over six years before paying for storage or bandwidth.

Azure

Azure is in a similar situation.  Their compute-focused VM instances start at about $100 a month each meaning that even keeping one active over six years would top $3.7k.   Running 8 VMs and a single SQL Server for that period tops $32k before even paying for persistent storage or bandwidth.

Local

Either option proposed for a local alternative is considerably less expensive than the cloud-based options and would represent gigantic performance upgrade over the existing solution.  In addition the process of transitioning would be considerably less complex. 

At this point I recommend keeping the build environment in house at this time. 

Which is sad in the sense that I was excited about the prospect of a cloud deploy but that's just not really feasible with the requirements of this particular location.  

It's good, though, in that I'm still on track to do a migration and that's very exciting. I've requested a new server with:

A 10-core hyper-threaded processor 

32 GB RAM

2x300 GB SAS Drives for an OS (RAID 1)

2x2 TB SAS Drives for VM storage and file sharing

Whether I get all that or not is yet to be determined but I'm excited to get started on the next phase of the project. 

Spent a good part of today documenting CMD scripts.  Many of them are usually called by an ASP application but by reading through and documenting their command line parameters they won't always have to be.

I've got a pretty good sense of the existing hardware and what they'll need moving forward but understanding cloud pricing is it's own magic science.

First day jitters

First day out in the wider world and I'm starting to fell like I might be figuring out what's going on.

The goal is to have everything migrated either on to new hardware or AWS in the very near future so I spent the day poking at two different servers and looking at what will need to be migrated.  One poor machine had five active VMs on just four cores (and more, just waiting to stand up).

Read some CMD files and felt the syntax coming right back like it was yesterday when I was furiously trying to figure out the right commands to just get my freaking video game to work back in Windows 95.  Had a self-indulgent 'kids these days' moment.

I can't wait to see what they've got for me tomorrow.