It got better

We’ve been running automated deployments for about 4 weeks now and they’ve greatly reduced the amount of time we spend pushing code out for testing. Powershell turned out to be a bit more of a pain-in-the-asslearning curve than I expected. Discovering Scott Guthrie’s blog post about app_offline.htm simplified another issue of ours, maintenance pages during the release.

Our Xml Updater utility is done. Our TFSDeployer scripts are a bit more modular now. We decided to just deploy everything rather than cherry-picking changes (it only takes about 4 minutes to deploy anyhow). The only thing we can’t do yet is automatically apply database changes and deploy to web servers with file sharing disabled.

Incorporating the app_offline.htm functionality to take the site down before deploying, we ended up with a basic code block of:

function Deploy-Directory
param([string] $appSourcePath, [string] $appDestinationPath)

# create our destination if it's not already there
new-item -force -path $appDestinationPath -itemtype "directory" | out-null

# create app_offline.htm to take destination offline for maintenance
Offline-App $appDestinationPath

# delete everything from destination, except the offline page.
get-childitem $appDestinationPath -exclude "app_offline.htm" | remove-item -force –recurse

# copy the published website to our destination
get-childitem $appSourcePath | copy-item -force -recurse -destination $appDestinationPath

# remove app_offline.htm
Online-App $appDestinationPath

# create a build indicator
$buildFile = $appDestinationPath + "CurrentBuild.txt"
$TfsDeployerBuildData.BuildNumber + " deployed on " + $(get-date).ToString() > $buildFile

# Creates an app_offline.htm file to let everyone know the websites's down.
function Offline-App
param([string] $appDestinationPath)

$offlineFile = $appDestinationPath + "app_offline.htm"

$output = "<HTML>" +
"<HEAD><TITLE>App down for maintenance</TITLE></HEAD>" +
"<BODY>This website is temporarily offline while Build " + $TfsDeployerBuildData.BuildNumber + " is being deployed. " +
"Please try again in a few minutes. " +
"<!-- " +
" Adding additional hidden content so that IE Friendly Errors don't prevent" +
" this message from displaying (note: it will show a friendly 404" +
" error if the content isn't of a certain size)." +
"-->" +
"</BODY>" +

$output > $offlineFile

Codecamp and Linq

Went to the Orlando Code Camp yesterday and was pleasantly surprised by how enjoyable and informative the sessions were. It’s always a little humbling to see all the technologies you’re not familiar with and realize you haven’t even scratched the surface of being an “expert”. 😉

The most interesting (and most crowded) seminars were definitely the Linq ones. I tried getting my toes wet with it for an my Xml updater, but just couldn’t wrap my head around it and went back to plain old DOM. After seeing LINQ to XML and LINQ to SQL sessions, it seemed a lot cooler, but I still just didn’t “get” why it was so much of a benefit. After being clued in to LinqPad, it began to click for me. LinqPad is one of the more ingenious learning tools I’ve used, especially for its simplicity.

Once I noticed that Linq works not just for Sql or Xml, but for basic collections, arrays, etc, I finally started to see how much more readable and maintainable it can make code. It can eliminate a lot of the nested foreach/if/foreach/if blocks you see scattered around business logic. Code like:

foreach (Order o in currentBatch.Orders)
if (o.ShipDate < '1/1/2008')
foreach (OrderLineItem oli in o.LineItems)
if (oli.Category == "Cookies")
break; //only 1 free milk per order


var query = from o in currentBatch.Orders
where o.ShipDate < '1/1/2008'
o.LineItems.Any(i => i.Category == "Cookies")
select o;

foreach (Order o in query)

While it doesn’t save much as far as number of lines, the bottom example just seems so much cleaner and more concise to me.

XML Namespaces

Martin Honnen is my hero for solving my XML namespace issue in my config file updater utility. We’re getting close to closing the automation loop now. Developers can check their changes into their dev branches and we automatically kick off an integration build for one of those branches. Once the developers merge down into trunk, an integration build will automatically kick off there too. TFS 2008’s CI stuff works painlessly here, even if it’s not that full-featured.

Once the build succeeds, QA can decide they want to deploy it to their test environment by changing the build quality to “Ready for Initial Testing”, at which point TFS Deployer does its magic. I’m now done with the little utility we can call after a deployment to update the web.config-like files to have test, final-test, or production settings depending on where we’re deploying to. I’m also almost done with some wrappers to upload assemblies to our servicehosts and custom task scheduling services we have. These will be a snap to integrate into TFS Deployer’s powershell scripts when they’re done.

The only major piece left is a utility that will interrogate the build’s changesets to determine what files changed and what corresponding assemblies and websites to deploy. Deploying everything wholesale would be a bit too cumbersome with the amount of projects we have.

Really happy to see all of this starting to come together.

Automating deployment

One problem with having lots of n-tier apps is deployment changes. For our environment, it means that a change to a single common assembly might be deployed to 4 different websites, remoting servicehosts, and any other places referenced. Keeping it all straight is a manual, error-prone process. We’ve been testing TFS Deployer as a means to allow automated deployment of Team Builds whenever QA accepts them. So far, it seems to be a very simple concept that works great. PowerShell is incredibly easy to write scripts for and it took about 10 minutes to get it deploying one of our websites to a test directory anytime I updated the build status to “Ready for Initial Testing”.

One thing holding us back from going live with it is a way to automatically update configuration settings for the sites after deploying over them. We need to point the websites to the right databases, override values for testing, etc. Our original plan for this was to just never deploy web.config and xml configuration files that contained environment-specific values, but we ended up losing code changes that were made to them during deployments.

We’ve now decided to forcibly deploy every file, including web.config, and run a program to update all environment-specific values afterward. Since all our configuration files are XML-based, it seems like it should be real simple, but XML namespaces are giving me trouble. Hopefully there’s a simple solution I’m missing, because once my XML problem’s solved, I can get out of the CM business.

Fixed build step deletions

I tried increasing the duration I run my “delete extraneous build steps” job and found that deleting the steps during particularly large project builds (such as web deployment projects) can cause the build to abort. I updated the query to only delete the build steps once the build is finished and that seems to have resolved the issue. Unfortunately, it means I still have to wait for the build to complete before I’m able to see what failed to build.

Stopping excessive build steps in TFS 2008

EDIT: Updated the query below to only delete the steps of finished builds. Deleting steps while the build is in progress can cause the build to abort.

Since upgrading to TFS 2008, we’ve had two extremely annoying sources of pain:

  1. the amount of msbuild output that goes into the BuildLog.txt
  2. the excessive number of build steps generated during a Team Build if you use project references within a solution

The excessive output in BuildLog.txt is easy enough to fix. MSBuild 3.5 defaults to diagnostic logging, so appending “/fileLoggerParameters:verbosity=normal” to the TFSBuild.rsp will cut down on the file size considerably.

The Build Steps are unaffected by the verbosity option though and there appears to be no simple flag to cut down on the chatter generated. A Team Builds of 4 or 5 solutions which generated around 400 steps under TFS 2005 now generates over 3000 steps under TFS 2008.

The bulk of these steps are repetitive and rather useless information messages about building targets GetTargetPath, GetNativeManifest, and GetCopyToOutputDirectory. Rather than writing custom tasks to iterate through the BuildStep node tree, I wrote the query below and scheduled it to run every 2 minutes in TFSBuild. It will delete any successful buildstep nodes that mention one of the three targets above. Running this cut our build steps down to a much more manageable size. While I haven’t had any issues with it, Microsoft doesn’t like you touching their tables directly, so buyer beware.

DECLARE @nodes TABLE (NodeId int)

INSERT @nodes
SELECT bif.NodeId
FROM tbl_BuildInformationField bif INNER JOIN tbl_BuildInformation bi
ON bif.NodeId = bi.NodeId INNER JOIN tbl_Build b
ON bi.BuildId = b.BuildId
bif.FieldValue LIKE '%"GetTargetPath".'
OR bif.FieldValue LIKE '%"GetNativeManifest".'
OR bif.FieldValue LIKE '%"GetCopyToOutputDirectoryItems".'
AND EXISTS( --only delete nodes that succeeded
FROM tbl_BuildInformationField status
WHERE status.NodeId = bif.NodeId
AND status.FieldName = 'Status'
AND status.FieldValue = 'Succeeded'
AND b.FinishTime IS NOT NULL

DELETE tbl_BuildInformationField
WHERE NodeId IN (SELECT NodeId FROM @nodes)

DELETE tbl_BuildInformation
WHERE NodeId IN (SELECT NodeId FROM @nodes)

Team Building shelvesets

With our smaller development team, we decided to try and minimize the number of branches we use to simplify our merge/release process. One drawback we ran into is how to handle pushing out an emergency fix when we’ve already got changes being tested by QA in trunk (our integration branch, for all intents and purposes). We could merge the change into the Release branch (a branch that contains code that’s already been deployed to Production and gone live) and check it in, but the idea of release is to only check in once it’s verified by QA and deployed.

Another solution is to have an interim branch between trunk and Release, a production integration branch of sorts. This adds extra complexity by having another merge to another branch you’re responsible for.

To work around this, we decided to merge to the Release branch but rather than performing a check-in, we shelve our changes. We have set up a teambuild that will get the latest source from Release, unshelve a shelveset, build the newly modifed code, and then undo the shelveset. Can’t remember where I yanked it from, but it simply involved adding the following targets to our build .proj definition:

<Target Name=”AfterGet”>
<Exec WorkingDirectory=”$(SolutionRoot)” Command=””$(TeamBuildRefPath)” unshelve “Shelveset build test”;mcooper” />

<Target Name=”BeforeDropBuild”>
<Exec WorkingDirectory=”$(SolutionRoot)” Command=””$(TeamBuildRefPath)” undo /recursive /noprompt $/” ContinueOnError=”true” />

Now all we have to do is merge to release but don’t check in, shelve the changes, and update the proj file to have the shelveset name and owner before kicking off a build.

Trimming TFS database size

Our TfsVersionControl database was at nearly 20 gigs when I started digging into why some source control actions were so slow. While 20 gigs is relatively small in terms of enterprise databases, it’s huge for our small source tree. Simple tasks like getting latest on a branch had a noticeable 5-30 second spin-up time while they queried the database.

The largest table turned out to be tbl_LocalVersion, which clocked in at 10 gigs. This table tracks the links between versioned items and the local workspaces of the developers. Listing out the workspaces with “tf workspaces /owner:*” showed the problem, we had 500+ workspaces, most of them belonging to developers who were no longer here. Deleting these workspaces also deletes their associated LocalVersion records. A quick script to run “tf workspace /delete “my dev branch;johnsmith”” for all the old workspaces brought us down to about 40 workspaces and a LocalVersion table of less than 3 gb.

To get a rough idea of who’s using what, run this in TfsVersionControl:

SELECT i.DisplayName, COUNT(DISTINCT w.WorkspaceId) AS Workspaces, COUNT(1) AS LocalVersionCount
tbl_Identity i WITH (NOLOCK) INNER JOIN tbl_Workspace w WITH (NOLOCK)
ON i.IdentityId = w.OwnerId INNER JOIN tbl_LocalVersion v WITH (NOLOCK)
ON w.WorkspaceId = v.WorkspaceId
GROUP BY i.DisplayName

Trimming out these workspaces has made the single largest improvement in the perceived performance of TFS out of anything I’ve tried so far.

Error 32000 – tfsdb.exe /showui:590116

Our upgrade from TFS 2005 to TFS 2008 took about 2 weeks from conception to completion. Surprisingly, most of this time was spent trying to get a new VM set up to mirror our existing TFS 2005 environment. VMware’s snapshots are a lifesaver, and made the process so much easier. We simply rolled back over and over until we got it figured out, got the steps documented, and then tested our documented steps. Once we had TFS 2005 up and running on a separate server, upgrading the test server to TFS 2008 was simple and painless. A day of playing with the builds, getting everything compiling, checkins and checkouts, etc. and we could find nothing that didn’t work under TFS 2008, even from a VS2005 client.

So that night we took the plunge and upgraded. 6:30pm rolled around and I confirmed the developers had gone home so I kicked it off. Naturally, what worked over and over in testing did not go so smoothly on Prod. After 20 minutes or so, an error popped up:

Error 32000.The Commandline ‘TfsDb.exe upgrade [blah blah] /showui:590116’ returned non-zero value: 100.

A few googles later, the best I could find was a suggestion to hit “Retry” and pray. After retrying a few times with no success, Profiler revealed that it was failing during the creation of the SQL Server Maintenance Jobs:

— Create the job
EXEC msdb.dbo.sp_add_job
@job_name = @adminJobName,
@category_id = 3, — Database maintenance
@description = ‘Performs maintenance on the Microsoft Team Foundation Version Control database.’,
@job_id = @job_id OUTPUT,
@owner_login_name = ‘sa’ — should set owner login name as sa, incase user who runs setup account expires

Microsoft recommends you rename the “sa” account as an added security measure, but then hardcodes “sa” into the TFS upgrade? Renaming our account back to “sa” temporarily for the upgrade resolved this and the rest of the upgrade went off without a hitch.

Another one

In an effort to keep the remaining friends I have on LJ, I’ve decided to start posting my work experiences in a separate blog. We recently had a pretty nasty shake-up (or lay-offs, or “trimming the fat”, or “right-sizing”) at my company (lost about 60% of IT) and as a result, I found myself unexpectedly thrust into the Configuration Manager/Build Guy role. This was on top of my existing DBA responsibilities.

We’re a very small shop with only 5 developers now so some of our processes no longer made sense. Branching for every release to production, branching for every minor new project, queueing up tickets in our integration branch until the existing release was done so we could dump them onto our QA team all at once and give them 2 weeks to test it and get it out the door… none of this made sense anymore. This is especially true now that we’ve just gone from 1 QA lead to manage everything, 5 QA testers, and 2 dedicated CMs to just 1 QA manager/tester and me as CM/DBA.

All of this went down about a month ago. Since then, I’ve been slammed getting our CM stuff under control. We’re unquestionably a Microsoft shop and we use VS2005/TFS2005. Unfortunately our idea of CM has been looking over a list of changesets we think might need to be released, figuring out which assemblies and/or websites to deploy based on that list (we’re very n-Tier, so 1 assembly change might need to be deployed to 10 different apps), and then copying and pasting files around.

To try and get our heads out from under water, we’ve decided to upgrade to VS2008 and TFS2008. I’m not sure what VS2008 brings to the table for the developers beyond LINQ, but they seem pretty excited about just that aspect of it. TFS2008 though… that’s where the money is for me. 99% of the improvements I’ve run across in TFS2008 are on the build side and they kick ass, so we’ve crammed through an upgrade in the span of about 2 weeks that went live yesterday morning. The developers didn’t notice a thing… flawless.