What are we going to do tomorrow night, Brain?

Work Harder

2008-10-21T15:26:00Z

Work Harder! That is a statement I find myself repeating daily. People often wonder what makes one person better at a certain task than others. There is a truly simple answer, practice.

Natural ability is a myth. With the exception of physical limitations, all ability is gained through practice. No infant is born with the innate ability to walk. No musician picks up an instrument for the first time and performs a flawless number. Everything ‘great' is preceded by countless not-so-great attempts.

The human race is a term that to some is a simple definition of our status as homosapiens. To me, it is a bit more literal. I recognize that there are billions of people in this world, all of whom want the same things I do. What sets certain individuals apart from the pack is their willingness to work harder than the rest. So for me, it's the race of humans. Who wants to work harder? Who wants to spend that extra effort day after day to be the best at what they do. Someone who doesn't work as hard as I do does not deserve the same things I deserve, just as I do not deserve the same benefits of the people who work harder than I am willing to.

I hear complaints from people about the state of their lives, every argument of which never focuses on their lack of effort. It is always the result of some innocuous outside influence. Bunk. People who work hard, achieve more. Good things happen to good people who work hard. Being good isn't enough. There are lots of "good" people, if the definition of "good" can be simplified to "not being bad". Spend a few more hours with your face buried in a book rather than a pint of beer. Unless, of course, your goal is to be the best drinker on earth.

Everything that has gone right in my life happened to me when I went the extra mile. Every real accomplishment didn't come from the 8 to 5 daily grind. Good things happened during the hundred-hour weeks of work. Times when 8 to 5 wouldn't have been enough to get the job done, allowing the opportunity to pass right by.

There have been many times I have explained to people that epiphanies have come to me in my sleep. Though I was not lying, this is not a result of me having some abnormal super power. It is a result of me wrapping my mind so deeply around a problem that I am consumed by it even during my few hours of sleep.

I like to think that everything that has happened to me in my life, I've earned. This, of course, includes all of the bad things. For many years my interests were simply in having fun, so my career took a nose-dive. Then I focused entirely on work which destroyed countless relationships, as well as my physical health. So while balance has become important, I have recognized that every decision comes with consequences; consequences which I hope to never complain about.

People make choices in their lives. If your choice is to pursue a career or path in your life that will never financially stabilize, you have no right to complain about finances. You have chosen to sacrifice financial success for your love or interest in what it is you do. It is not society's responsibility to fill in for your financial woes. If you choose to sacrifice your personal life for professional glory, then do not cry about your lack of intimacy. You chose that path. Just realize that everything requires hard work. Even if an individual's goal in life is to be the best parent their child could have, then they must work for it.

This is not to say that I am better or worse than anyone else in this world. What I am attempting to say is that I take responsibility for what happens in my life. I will not give up, I will not rest until I am the best I can be at what I do, regardless of what this life throws at me. The best I can be at anything will require a lifetime of effort. So I suppose it is a valid assumption to say that I will only achieve this in the moment immediately before my death.

I truly wish more people would take responsibility for their lives.

Thread Safe Dictionary Update

2008-09-29T16:14:00Z

For those of you that have read my post about the Thread Safe Dictionary, I have a few updates.

In recent months, I have done a ridiculous amount of multi-threaded work. This has forced me to expand my threading libraries. One of the primary objects that required changes\fixes was my dictionary.

The dictionary uses ReaderWriterLockSlim locks. These are clean, lightweight locks that MS has provided. Most importantly, they allow upgradeable read locks. I now use simple IDisposable locking wrappers. This allows us to use the "using" syntax to ensure lock releases.

It is important to note that no safe dictionary can provide safe inserts without the caller forcing a lock. This is because by the time you check for the existence of an object and then insert it, it may have been inserted by another thread. The only way to prevent this is by doing a "blind" merge. The reason I call it blind, is that when you do a merge, we check for existence, and do a replace\insert within the scope of a single write lock. This means you cannot control which objects get replaced in the dictionary. This is rarely important in caching situations.

Here's the code:

ThreadSafeDictionary.zip

Thread Safe Dictionary in .NET with ReaderWriterLockSlim

2008-09-26T15:14:00Z

Since MS has decided to not include a thread-safe dictionary in the .NET framework, many of us have been forced to implement our own. What MS has done now with .NET 3.5 is given us a newer locking mechanism which makes this task easier.

Previously, we had 2 lock mechanisms, neither of which worked very well for my uses. The first is the Lock statement. While this method is effective, it causes serious contention issues with simultaneous reads\writes. Since it is generally safe to execute multiple concurrent reads from a dictionary, this was a huge performance flaw as the Lock statement was forcing us to perform all actions against the object in a serial fashion.

So that left ReaderWriterLock, which would allow us to take out Reader & Writer locks independently. This seemed to be the answer. Unfortunately, after further testing and review, MS and the dev communities warned against this lock type as it could have unintended consequences in certain situations. The explanations for these I will not go in to, a simple Google search will answer those questions. For the most part, I don't have the mental capacity to understand, much less explain the failures. I will tell you that I have tested this method, and ran in to some issues, and subsequently abandoned the approach.

With .NET 3.5 comes the brand-spankin-new ReaderWriterLockSlim. ReaderWriterLockSlim is similar to the ReaderWriterLock in functionality, minus some of the drawbacks. Of course any new approach will bring on its own consequences, and if you do a little research, you will find that Slim isn't infallible either.

Many of our applications implement the Factory model. Factory models are great, they are extensible, and they can be extremely performant. With that said, any data driven application that uses the Factory model without a caching model, will most likely perform poorly. So, in our factories, we build in a caching mechanism. A very simple type and ID based cache that allows us to persist commonly used objects to in-memory cache for a specific amount of time. I won't go in to the implementation of this model here as every caching implementation has its own needs and wants.

At the heart of this and likely any caching model is inevitably a Dictionary or Dictionary like structure. MS programmers have spent countless hours fine-tuning the Dictionary class to make it as performant as possible. I challenge anyone to write a managed Dictionary that performs as well in such a wide variety of scenarios. I've tried and failed. Unfortunately, MS has not provided a thread-safe implementation. Generally when you run into the question of thread-safety, most people have no idea what you are talking about. The problem with writing truly thread-safe code that can work in any scenario is that you will sacrifice performance. If you tailor your thread-safe mechanisms to your particular problem, you can generally maintain a higher level of performance.

The reason we need thread-safety is not that we are specifically spawning threads. In IIS, worker processes are inherently multi-threaded. Therefore, using static objects in a web-farm generally requires some level of thread safety, otherwise you will get dreaded "An item with the same key has already been added" errors. This is because between the time you check to see if a key already exists in your dictionary and the time you insert it, some other thread may have inserted this key. This is especially the case in object factories, because you have multiple threads attempting to access/create objects and subsequently cache them.

This is the code I've written. This has not been tested yet, so please use it carefully. I will keep this post updated with any changes and test results. I also have not made this class utilize lock escalation, which ReaderWriterLockSlim supports. As always, please perform your own testing.

ThreadSafeDictionary.cs

[Update]
Through some testing, I found the need to update the ThreadSafe Dictionary. I changed all of the locking to use IDisposables to ensure lock creation and destruction.

Microsoft 2008 Platform Launch - Heroes Happen {Here}

2008-03-01T20:28:00Z

For those of you that live under a rock, the Microsoft 2008 Platform Launch has kicked off. It is the launch event for Sql Server 2008, Windows Server 2008, and Visual Studio 2008.

The company I work for, Big Hammer, has been featured in the event. As well of some of my products.

Check out the case study to get an idea of what I do all day.

SQL Server Constants... The Constant Struggle..

2008-02-13T22:13:00Z

Sql Server does not allow for user defined constants. So we are forced to take other routes like hard-coded values, scalar udf's, settings tables, etc.

Unfortunately, all methods are not equal. You need to carefully consider your use of different methods, and test the results as all of these methods result in different approaches by the query optimizer.

We use an object factory that is tightly coupled to database tables. To simplify data access we have a single view that becomes, in essence, a partitioned table. We'll call it dbo.Object

This view looks something like this:

create view dbo.Object
as
select 1 as ObjectType_ID, ID, Description from dbo.[User]
union all 
select 2 as ObjectType_ID, ID, Description from dbo.object2
union all 
select 3 as ObjectType_ID, ID, Description from dbo.object3
union all 
select 4 as ObjectType_ID, ID, Description from dbo.object4
--etc..

Now to query against that view, you can simply predicate on objecttype. If you check the IO statistics, it is only looking up against the one table, provided you query it properly. Don't bother looking at the Query Plan as it does not truly reflect what the server does to fetch the data.

So the idea is, we have a table which directly related concrete types in code to database object types. This way objects can very simply load themselves from the DB. Poor coding practices aside, for many applications this is a perfectly legitimate model.

The problem comes in when writing queries against these partitioned views. We were looking for a way to not have to hard-code type ID's into sprocs or queries. This is where constants would be a big help. I could potentially define a constant called @@UserType and equate it to it's corresponding database object type.

This way when writing a query to extract all user info, I could simply write:

1	Select * from dbo.Object where ObjectType_ID = @@UserType

Here are the multiple ways we have tried to accomplish this, each has it's own unique performance outcomes:

First, turn IO statistics output on for your connection.

1	set statistics io on

This query works properly, unfortunately we are hardcoding the ID, which is what we are trying not to do. This is the baseline. We want results identical to this.

1	Select * from dbo.Object where ObjectType_ID = 2

So lets try creating a UDF that returns that ObjectType_ID

create function [ObjectType].[User]()
returns int
as
begin
    return(1) 
end

This query unfortunately evaluates each table, but at least it doesn't scan them

declare @objtype int
set @objtype = ObjectType.[User]() -- a udf that returns a value

Select * from dbo.Object where ObjectType_ID = @objtype

This query is closer, but it is creating a working table and is evaluating for every row

1	Select * from dbo.Object where ObjectType_ID = ObjectType.[User]()

I modified the UDF to take a useless param, in hopes that the deterministic decisions were based on the input param to output, but to no avail.
The results were same as above.

1	Select * from dbo.Object where ObjectType_ID = ObjectType.[User](0)

I changed the UDF to take the string Type name as an input, and a simple CASE or IF statement is user to return the correct ID.
This would be optimal, as it would prevent me from having to create a UDF for each Type. I also tried it with looking up the value in a table.
Unfortunately we get the same result.

1	Select * from dbo.Object where ObjectType_ID = ObjectType.GetType('User')

Lets try a subquery.
This is the worst, as it scans each table.

1 2	Select * from dbo.Object where ObjectType_ID = (Select ObjectType.[User]())

Joining on a settings table is an option, unfortunately as we may be doing multiple joins on tables requiring these constants, we would have to do multiple joins on the settings table, which turns out to be very inefficient.

Chuck Heinzelman pointed out an intersting function to call to check if the UDF is being treated as deterministic:

1	select objectpropertyex(object_id('objecttype.user'), 'isdeterministic') -- returns 0

This returns False

So taking Chuck's advice, I schema bound the function:

alter function [ObjectType].[User]()
returns int
with schemabinding
as
begin
return(1) 
end

Now, we try this again:

1	Select objectpropertyex(object_id('objecttype.user), 'isdeterministic') -- returns 1

Excellent, now it returns True.

Unfortunately, we get almost the same results if we re-run the queries.
I did not, however, profile it to see if it is truly evaluating the function for every row. I suspect it is not. So the use of the UDF may be not so bad if the only result is that it is using a worktable. Unfortunately, every place i use the UDF in a query, it creates another worktable.

So until we have a better CONST struct in SQL we will be forced to hardcode values for some of our queries.

Media Center Woes

2008-01-26T19:46:00Z

So about a year ago i began my quest for the ultimate home media experience. I have the A/V components(TV's, Projectors, Speakers, Etc.) but i needed to tie all of my installations together. I have a Projector with a 110+ inch screen in my basement, and a 73" DLP in my great room. I also have flat panel LCD's scattered throughout my home.

The goal was to provide live TV, recorded TV, music, and DVD access to all stations through one component. Whether that component was an xbox 360, a Vista Media Center PC, or whatever, i wanted one at each station. So i sat and chatted with a friend from work, Casey Chesnut who is a former Media Center MVP. We decided to try a few approaches.

So I started with the primary media center pc. This was going to be my focal point, my connector to my digital cable, and the center for any DVR. I did some research and found that the legal jackals at the digital cable companies are ensuring that only qualified (read paid-up) manufacturers can sell media center pc's with digital cable tuner cards. So after some digging, i made a choice and purchased a s1Digital media center with two digital cable tuner cards. This thing was flaky as could be for the first few weeks. After some configuration changes, uninstalling the Intel Viiv garbage, and a firmware update for my the tuner cards, it is finally stable. The only thing that is left is that the machine has a built-in IR receiver that works only half of the time. So i was forced to connect an external IR port. The only problem is that now, half of the time, the machine is registering double button actions from the remote. This sucks. So after an hour of looking around in the case for the cable or port to disconnect the built-in IR sensor, i just decided to cover the front face with tape. Black Gaffers tape is the ultimate visual enhancement for a high-end home A/V setup. The good news is that the machine is pretty slick. Lots of processing power, and the video quality from digital cable is incredible.

Now, on to the second part of this install. I have somewhere around 500 DVD's that i've acquired over the past years. I previously had them most of them in a 400 disc changer that was controlled via a pretty slick Escient media interface. The Escient did all of the cataloging, and only twice did it fail to recognize a DVD. Unfortunately, i found myself shuttling DVD's from my upstairs to my downstairs or to my bedroom to watch them. This is a hassle, and since i am a big lazy goon, i'm generally not too careful with them. So, i decided to rip all of my DVD's to a giant drive array. So i bought 2 NAS devices, one is a ReadyNAS, and the other is a linksys. The ReadyNAS is great, and i haven't used the linksys yet, so i have no verdict. Either way, i have something in the range of 5TB+ of RAID 5 storage. This is great. The only pain is that you can't just copy most DVD's to a file structure, as DRM prevents it. The good news is that Slysoft AnyDVD fixes this problem and strips of all DRM. I could have converted or compressed the DVD's and saved some space, but i'm one of those people who prefers to have an exact copy of my hard media.

Excellent, so now i have my movies all in one place. Scratch that. I will have them in one place once i've finished ripping them all. Anyways, now i have a huge directory structure with many of my dvd's. Unfortunately, coming from the Escient, i want a better and prettier interface than a folder structure. Casey recommended the My Movies plugin for Media Center. I installed it and it works fairly well. It's a little buggy, but acceptable for a free community initiative. This cataloged my DVD's and allowed me to play them on my media center. Sweet.

So my Primary TV is mostly set. Now lets get this lovely media piped to other rooms! My first plan of attack was an xbox 360. This worked really well. I could connect to my media center and watch live and recorded TV through my xbox! This was awesome, such a simple solution. Not only are 360 consoles fairly cheap, but they have great video and sound capabilities and i could play a game anywhere in the house. Excellent, order up 5 xbox consoles right? Negative. The 360 console will not allow you to play a DVD format movie from a network share. They have been recently extended to allow divx though. Unfortunately at 110" divx looks like crap. Also, i'll be damned if i'm going to convert and store every movie i have to divx.

Casey, so intrigued by this debacle, began trying to circumvent this limitation. Custom media streams, on-the-fly conversions, etc. Nothing looked or worked very well. Thanks for the effort though bud.

On to plan B. A media center PC at every TV. Sounds like a winner if i can find a cheap enough PC to warrant using just for this purpose. Enter HP Slimline Refurbs. It turns out that buy.com was selling these very capable, very nice looking refurbs for just over $300. Excellent. Order a couple and give em a shot. The machines themselves are impressive, great stats for a cheap machine, and they come with Vista Home Premium. Excellent. Yes! I can play my dvd's!

But wait... i can't use a Media Center PC as an extender, just like the xbox? Nope! MS has failed me again. The initiave in MS to provide a software based extender has been stalled or flat-out cancelled according to Casey.

So, on these nice Media Center PC's, i cannot watch live TV or recorded TV from my primary media center. And because the cable companies won't allow a home-built user to buy and install his own digital cable tuners, I cannot just use them independently. Even if i play some tricks and share the saved files for recorded TV, the drm prevents me from watching Whose Line at any TV i want.

Damn... So, i'm going to try a few non-MS methods. There are a few independant Media Center clones out there, even a few open-source ones in C#. I'll keep you all posted.

MS DTC Configuration for SQL 2005 in a Distance-Cluster on Windows Server 2003 Datacenter IA64

2008-01-14T18:01:00Z

As does everyone from time-to-time, we had a lovely "learn as you go" moment in the middle of a deployment recently. As it turns out there are some very specific rules for setting up MS DTC in a clustered environment. As always, individual results may very.

First of all, most DTC changes don't require an operating system restart. Unfortunately, it would appear, in a clustered environment this is not true. We did, however, find that if you bring down services in the correct order, you can successfully make changes to DTC without a system restart.

Offline SQL Instances
Offline DTC
Offline Backup Agents(we use SQL Safe from Idera)
Make DTC changes
Online Backup Agents
Online DTC
Online SQL Instances

Now that we have that out of the way, we can get to the settings. For this example, lets assume we have 3 servers, 2 of them are distance-clustered, the 3rd is not a member of the cluster but will be executing distributed transactions to the cluster itself.

First of all, if you don't know where the DTC settings are they exist in the "Component Services" snap-in for MMC(run:mmc; add snap-in:"Component Services"). For me, I RDP into the console on the server itself, so the default "My Computer" node that appears works, otherwise connect to the appropriate server. Then, right-click 'Properties" on the server name and you have a window which gives you a tab labeled "MSDTC".

This tab looks like this:

So this screen tells us some things. First make sure that DTC is started on all the servers involved(unless you brought them offline purposely as described above). If not, nothing is going to work. You will probably want to make sure that service starts automatically. Your cluster group should probably handle this.

Now the fun stuff. Like I said, individual results may vary but these settings worked for us.

Open the Security Configuration dialog:

This dialog shows the configuration as works for our Server 1 & 2(the clustered servers), "Mutual Authentication Required" will NOT work in a clustered environment. Not sure if "Incoming Caller Authentication Required" works, we couldn't make it work properly.

On the non-clustered server, you must also NOT use "Mutual Authentication Required", however "Incoming Caller Authentication Required" authentication seems to be sufficient for the non-clustered servers.

What this means then is that all servers that will be opening or receiving distributed transactions to or from a clustered environment, must have their DTC settings changed. We found this out the hard way, as DTC transactions were working in one direction, and not the other. So it is possible that if your transactions are only being initiated in one direction you may not need to mess with certain servers. Test it, you'll be glad you took the time.

If you are reading this post, you've probably already seen the DTCPing.exe app that is on the Microsoft site. It is important to note that DTCPing happily returned no errors even when our authentication settings were jacked. So, while this utility may help diagnose certain DTC issues, i'm not sure that it helps in this scenario.

Some of this information is outlined very obscurely in this KB article http://support.microsoft.com/kb/899191/

Notes:

I am not sure as to the security implications of these settings in an exposed environment. As our servers are safely sitting in an internal network, I'm not too concerned. You will have to research and evaluate that decision on your own.
I did not have to change any settings on the second machine in the cluster, as cluster manager did it for me when I initiated a test fail-over. Once again, run your own test.

Sql 2005 Datacenter Cluster Issues

2008-01-11T23:34:00Z

I'm sitting at my office desk waiting for our distance cluster to finish failing back over to our local server. For the second time today, the our SQL cluster decided to fail over to our remote location. My suspicion is that due to the ungodly number of SQL instances we have on these servers, we consumed the available 64gb of ram and caused the splitter driver to fail.

For a little background, we utilize monster 64bit Unisys ES7000 Itanium 2 servers at work. These bad boys are wonderful. The only complaint I have ever had is that the distance clustering solution by EMC has done some questionable things, but when it works, it is one of the slickest clustering implementations I've ever seen.

We just moved the remainder of our databases to this cluster on Wednesday, which due to our failure in throttling the memory usage, must have pushed it over the limit.

We work with a million technologies that the majority of people never come across in their work, each of which presents it's own set of challenges. We have a fairly large server farm, 2 of them really. We utilize Windows Auth for pretty much all authentication. This creates all sorts of wonderful issues as well. We do constrained delegation via SQL CLR stored procedures from server-to-server. This is a fun trick. We actually use a SQL Server as a routing engine for database calls, most people would call this questionable architecture, but once you look at the scale of what we accomplish with it, you'll see it is quite a brilliant solution. So setting up SPN's and MS DTC in a cluster is in interesting challenge.

When I get some time, I'll post some specifics about our infrastructure and some of the hurdles we've had to jump to make it all work. Topics i plan to touch on:

MS DTC in a cluster
SPN's and delegation
SQL CLR (cross-server calls & impersonation)
SQL Performance
Multi-Tenant DB's
etc.

Till then...