The Single Founder

When it comes to software, my second biggest pet peeve is software that doesn’t work. By that I mean software that blatantly doesn’t do things that it should fundamentally be able to do. For example, things like… I don’t know… like maybe changing the administrator password of the application to something other than “admin” without crashing the whole damned craplication until you reinstall it.

I don’t blame the developers. Sure, they write the code, but at the end of the day they’re not the ones who sign off on the release. I think it’s pretty rare that some junior developer says “We’re ready. Ship it!”, and only then will the product ship.
(more…)

Last December, Jeff Atwood of Coding Horror wrote that Hardware is cheap, Programmers are expensive. While I certainly agree with the spirit of his premise and eventual conclusion, it is only applicable if you are running Software as a Service. But he doesn’t say this and I wonder if it was an oversight, or if he forgets what it’s like to ship software to other people. There is clearly a case to be made for telling developers to optimize their code in shipping products.

Let’s first take a look at the rules for optimization that Jeff lays out for us.

1. Throw cheap, faster hardware at the performance problem.
2. If the application now meets your performance goals, stop.
3. Benchmark your code to identify specifically where the performance problems are.
4. Analyze and optimize the areas that you identified in the previous step.
5. If the application now meets your performance goals, stop.
6. Go to step 1.

As Jeff points out, you can multiply the performance of an application by throwing hardware at the problem. Take the guys over at Plenty of Fish who upgraded their database server to a HP ProLiant DL785 with 512 GB of RAM and 32 CPU’s. No, that is not a typo, and yes it really means 512 Gigabytes of RAM with eight CPU sockets with quad core processors. This is a classic case of throwing hardware at the problem to avoid a lot of additional engineering time restructuring and optimizing the code to ensure that the server can keep up with the workload. This was probably at least a six figure expense that saved more than a year of engineering time, so a cost-benefit analysis would show that this was well worth it.

The problem lies in the very first step where there are some major underlying assumptions which you can infer from the list.

1. This is your budget we’re talking about.
2. You have the budget and means to upgrade the hardware.
3. The software isn’t a dog out of the box.

First, if you’re running Software as a Service and your installation is the only installation in the world, then it’s easy to justify throwing hardware at the problem. It’s a lot more difficult to tell your customer that he needs to buy new hardware to get your software to run faster. Don’t get me wrong, there are cases where this approach is justifiable. Most applications slow down over time as they outgrow the hardware they are on because the business grows, but the hardware stays constant. Eventually, the system slows down until you either upgrade to a newer (and hopefully faster) version of the software, or you break down and purchase new hardware. I think most of us can agree that this is a reasonable expectation and a customer should be able to budget for this sort of thing over the life-cycle of an application.

The second problem is actually more difficult for most people to understand. It’s easy to forget that the budgets of IT departments in larger companies are far below what they used to be and these companies are completely subservient to their budgets. One person shops and small companies don’t realize this for two reasons. First, they are more profitable on a per-employee basis than a larger company. Second, they tend to look at the total dollar amounts that larger companies are making and ignore the costs associated with actually running the company. Smaller companies simply have a lot more flexibility in making purchases.

In large companies, budgets exist so people can plan ahead and make the best use of their money to move the business forward. The ultimate goal of that budgeting process is to stay in business. If you ignore your budgets, you will spend more money than you make and ultimately, that’s the reason that any company goes under. Upgrading hardware isn’t always in the budget. It might make sense, but it’s not always possible.

The final assumption is worth taking note of and is the real reason for this post, so let me give you an example. Or “the” example, rather. A few weeks ago, I was asked to perform a demo of the final Release Candidate for an Enterprise software package. Minimum requirements for the installation alone were as follows:

• Windows 2003 Server (64 bit recommended)
• 4 GB of RAM (16GB recommended)
• 2 CPU’s (4 cpu’s recommended)
• SQL Server 2005 (64 bit recommended)
• 10GB disk space

Try installing that onto a virtual environment on a moderately high end laptop to do a demo for a customer. Seriously, I’d like you to try it. Survey says! bzzzzzzzzz!

I had two main issues with these hardware requirements. First, was that they were absolute requirements to get past the installer. You couldn’t install the software unless you had 4GB of RAM and 2 CPU’s. I think it’s reasonable at the Enterprise level to require “new-ish” hardware for installing your application onto a server. The fact of the matter is that in many cases, dual socket, quad-core servers are not uncommon as a requirement to adequately run an application that touches thousands of computers in an environment. Hardware requirements like these are implemented for the “general good” of the customer.

But come on guys. If the standard mechanism for selling your software is to have a pre-sales engineer install it on a mid-range laptop to perform an on-site demo for a customer, do everyone a favor and drop the hardware requirements a bit. You can barely find a laptop that’s capable of meeting those requirements, especially if it needs to run other virtual machines at the same time. Resellers of your software shouldn’t have to bring a mini-data center to a customer site just to perform a demo. I’ve seen it done, and it’s not fun. If you’re going to enforce hardware requirements for the sake of “scalability” to keep your customers from doing something stupid, make sure that people who know what they are doing can say “I know what I’m doing, so let me do this anyway”.

That’s Rule #1.

When the hardware requirements outlined above are met and it still takes 5 minutes just to get to a login screen on a fresh installation, then something is seriously wrong. I don’t care if it’s common for the “minimum hardware requirements” to mean that it barely runs and is completely not demo-able. Dual CPU’s and 4GB of RAM in this day and age should not mean barely functional for an unloaded web application.

That’s Rule #2.

I think that ultimately Rule #1 is more important.

I wrote this article the day that Bill Gates announced he was leaving Microsoft to pursue a career as a philanthropist. It sat on the shelf until about two weeks ago when I started helping Rob Walling edit his article Nine Things Developers Want More Than Money. I couldn’t help but be reminded of it, and Rob encouraged me to publish it. If you enjoy this article, you should definitely check out Rob’s article, which pinpoints what developers are really looking for in a job. Enjoy!

Earlier this year, Bill Gates announced that he will be leaving Microsoft within the next two years. In a way, I’m rather surprised that he lasted this long. Let me tell you, if I had $40 billion in the bank, one of the last things I’d be doing is working. Still, it does prove a point. People have a tendency to work extremely hard at the things they love, regardless of the pay. Great pay helps, but it also prolongs the inevitable if you’re just in it for the money.

To be honest, I have worked some places just for the money. It’s not fun. In fact, it downright sucks going to work at a place you simply can’t stand just to see a fine looking paycheck at the end of the week. The common inclination is to think you would feel otherwise in that situation. “Oh, if I were making six figures, I wouldn’t care what I was doing. I’d work there forever.”

No, you wouldn’t. I guarantee it. It’s not about the money. It never is. People always say this, but it isn’t true. It’s about enjoying your career. If you work at a job for long enough that you don’t like, you’re going to eventually start looking around for what else is out there. There are two underlying reasons for this.

First, and more importantly, the job sucks. It’s either boring, not challenging, monotonous, or you sit in meetings all day when you hate going to meetings, etc… You make up excuses like “I’m working from home today because my Chihuahua is sick and I need to take him to the vet.” Be honest. You don’t even own a Chihuahua.

Let me put things a little differently. If you had a 9-5 job where your sole task was to sharpen pencils all day long and it paid 50% more than what you were making now, or $100k/year (whichever is greater), how long do you think you would last? A day? A month? A year? What about two years? And your ten year plan is what? Sharpen enough pencils to get in the Guinness Book of World Records! Score!!! Maybe not.

The second reason is that after a while, you come to realize that you need to move on and find a new job, even if it means taking a pay cut. As you sharpen those pencils every day, you literally feel your brain being dulled as the pencils grow sharper. They are now being sharpened by osmosis. Do that too long, and you will become utterly unemployable, and then where would you be? I’ll give you a hint. It’s one word, starts with ‘unemploy’, and ends with ‘ed’.

When I lived in Rochester, NY, there were some companies that I would have taken a $20k pay cut to work at because the work would have been exciting, incredibly interesting, and my technical talents would have increased tremendously. Personal growth is important to everyone, no matter what career you have. Try as they may, a lot of companies don’t seem to understand this and don’t offer career advancement paths. This is partly why I left the IT department at Wegmans, which has been touted a number of times as one of the best companies to work for. I eventually just got bored.

When I first moved to Massachusetts, I had started looking around at jobs and was considering a mild career change from that of a software developer to more of a software/hardware interfacing developer. I have the hardware training. Hell, if I can reverse engineer and rewire a Furby so you can control it over the internet, I can sure as hell get somewhere in a hardware/software interfacing career.

On the power of my cover letter alone, I was able to land an interview at iRobot. I was told that my cover letter blew them away and was their basis for even calling me. I had basically explained my experience and that I was considering a career change and wanted a chance in the ‘new’ field. They were impressed with the projects I had undertaken outside of work and wanted to bring me in.

I guess in my mind, it wouldn’t have really been all that much of a new career. I mean, after you’ve designed a CPU at the substrate layer, interfacing IC chips to form a product just seems like a natural extension of that.
But the job market is a funny place and you’re not going to even get a foot in the door unless it’s an entry level position, and this was an entry level position.

Unfortunately, I had to decline the iRobot interview as I had accepted a position at Pedestal Software literally the day before. I hadn’t signed anything with Pedestal, but I wasn’t going to go back on my word. I guess that’s just the type of person I am. I thanked iRobot, and that was the end of it. Sort of. The HR rep spent the next five minutes begging me to “just come in for an interview and see what you think”. Three years later, iRobot went public. Oops.

It turns out that I could have done what I loved, and probably made a lot of money at the same time. Pedestal Software did very well for itself and was eventually acquired by Altiris to the tune of $65-$75 million or so. (The actual amount is a bit fuzzy, since it depends on how you count their cash on hand, and who was speaking when the press release was issued.) Me? Well, I really enjoyed working at Pedestal, but I made less than $10k from my stock options, which turns out to be around 0.00014% of the deal. It made me realize something.

A lot of people have delusions of grandeur when working at startups. This product is going to be great, everyone is going to want one, and we’re all going to be rich!

Yea? Let me know how that works out for you.

The most viable exit strategy for most companies these days is selling out to a larger company, which by default means no IPO. The reality is that unless you’re the CEO or one of the founders of a company, you’re not going to make a lot of money when that happens. By a lot, I mean millions of dollars. Even vice presidents of startups don’t make a huge amount of money when the company is sold. They make a few hundred thousand to be sure, so it’s not as if it wasn’t worth their time. But they don’t make out nearly as well as the CEO and the founders.

An IPO is reserved for Google, iRobot and those other rare, one in 100,000 startups. But you don’t have to go public with your company to have a successful career. You don’t have to sell your startup company to make a good living.

One of the things that attracts a lot of people to the Fog Creek way of doing things is how Joel lays everything out. It’s not that he has truly original ideas. But Joel has a unique way with words and puts things so well that it’s very difficult to disagree with him unless you start nitpicking, at which time you’re probably missing the point of his argument anyway.

Who doesn’t want to work at a company where the engineers make the decisions instead of the marketing or sales guys? When is the last time you talked with a sales or marketing guy who really understood the underlying technology? Unless he was a developer turned sales guy, I bet you never have. Part of the reason for this is the barrier to entry for a developer to get into sales. When you make the money you do as a developer, it’s really hard to give all that up and start over at the bottom rung, taking a huge pay cut. Not impossible, but very difficult. In a market like Boston where rent and mortgages are outrageously expensive, it’s even more difficult.

As desirable as these people would be, I would wager that you will likely never even have the opportunity to hire one. Do you know why? Because these are the type of people who start their own companies. They not only see and understand what people want and need, but they have the technical background to build it. They understand all of the angles and realize that they can control their own destiny, and that’s what self funding your own company is all about. Being in complete control of your own destiny.

Now, I’m not saying that everyone out there needs to run out and start their own company to have a happy and healthy career. Far from it. You simply need to work in an environment that is encouraging you and your professional growth, rather than discouraging it.

If you’re anything like me, you’re a bit of a geek, and you enjoy being a geek. Karnaugh maps are cool. At least a little bit. Wiring Furby’s to the internet is even cooler. Geeky, but definitely cool. Gadgets and gizmos, slashdot and thinkgeek. We play with new technology on our own time for one good reason. Because we can and because we like it. Ok, that was two reasons. $echo complaints | /dev/null

When you first get out of college, life is great. You get a job as a developer, and although it’s grunt work, the pay is better than you got working at the Student Union flipping burgers or washing aprons. As time goes on, things change. Despair.com and Dilbert turn into role models as working in corporate America gradually takes its toll. I’ve said it in the past, and I’ll say it again. Dilbert is only funny because its true. We’ve all had our PHB’s and wondered what life would be like without them.

And you know what? It’s really hard to find companies that aren’t like that. That’s why I started Moon River Software. I was tired of looking for one. Convincing Joel to move Fog Creek, or Eric Sink to move SourceGear to Boston simply wasn’t going to happen anytime soon.

So now, I’m doing what I love, there are no PHB’s in sight, and things are going well. As someone who went before me probably said, “I think I’m onto something here”.

Special thanks to Rob Walling for reading drafts of this article.

Furl – del.icio.us – Technorati – reddit!
BlinkList – Digg – Google – My Yahoo

I’m not sure why I hadn’t thought of this sooner, but in a very small software company, using a code generator would not only seem to be a great time saver, but would be an invaluable tool for creating vast quantities of reusable, high quality code. This is assuming of course that you do a good job with the templates to begin with.

One of my first experiences with code generation was working at Clearwire. The web based ordering software I was working on was under the unfortunate constraint of reporting to the sales and marketing team. Reporting to either of these groups alone would be a nightmare, but at Clearwire, doubly so as they were basically the same department. Even more difficult was that I was located in Buffalo, NY while the Sales/Marketing team was  located in Dallas, Texas.

I rather clearly remember one particular nightmare I ended up getting involved in. Let me give you the short story.

Week 1:
VP: “Hey Mike, can you make up some suitable text for this web page?”
Mike: “No problem.”

Later that day:
VP: “Mike, can you change that text you made up from X to Y? I’d appreciate it, and keep up the good work.”
Mike: “Sure, and thanks!”

Week 2:
VP: “Mike, can you change the text from Y to X for me?”
Mike: “Sure, no problem.” (Heh. Should have done it my way to begin with)

Week 3:
VP: “Mike, got a favor to ask. Can you change the text from X to Y? That would be great.”
Mike: “Hmm. Well, I guess so.”

Week 4:
VP: “Mike, the text on that page gives the wrong impression to our customers, can you change it from Y to X?”
Mike: “It just was X and you had me change it.”
VP: “Really? Well, I’m sorry. Can you change it please?”
Mike: “Allright.”

Week 5:
VP: “Hey, you got a second? Can you change the text from X to Y?”
Mike: “It just was Y.”
VP: “Who told you to change it?”
Mike: “ummm. You did.”
VP: “Really? Oh, I’m sorry about that. Can you change it back please?”
Mike: “No.”
VP: “What? Why not?”
Mike: “Because I’ve got too much code to write to respond to trivial requests to change meaningless text that nobody reads anyway just because you woke up on a different side of the bed than you usually do after drinking all night and snorting pixie stix with your buddies!”

Chances are that I probably didn’t say that last sentance, but I was pretty annoyed at that point. Basically, I insisted that I wasn’t going to make any further text changes until I had it in writing from him that it was what he really wanted, because I really had too much to do to waste time on changing text every single week.

Shortly thereafter, I received a bit of an unofficial reprimand from my boss for giving the VP of Sales a hard time. Our little chat involved a discussion about how an entry level engineer should address a VP of the company with respect, no matter how stupid he’s being at the time, after which I can bring my concerns up with my boss who will address it with his boss, who will address it with the VP in a social environment of beer and pixie stix so as not to damage his frail little ego.

Fortunately, my point was well taken, I was not formally reprimanded, and future changes were submitted in writing. Apparently, it was my tone that was ill taken. I took it as a valuable lesson about how to reprimand your superiors for excessive stupidity.

Even so, when I was tasked with rewriting the application quite some time later, I built templates that would regenerate the application UI by tweaking configuration files and then running a script. I always found it an impressive little design I had come up with. Code that writes code. Very spiffy!

Of course, whenever you write any code, you’re writing code that writes code because it eventually compiles into machine code. But that’s really not quite the same thing as ‘code that writes code’. Converting to machine code is merely compiling the source code you’ve written into something the machine can understand.

On the other hand, a true code generator will operate from a set of widgets to generate structured code that is useable in other applications, either directly, or as a library that is compiled after being generated.

There are a lot of subtleties that I’m intentionally glossing over. For example, would software that unrolls the loops in javascript to make the page load and execute the javascript faster considered a code generator? Probably not, and definately not in my book.

Would that really work anyway because the load time of the page would be increased in direct proportion to the number of loops that are unrolled, forcing the javascript parser to take longer to parse, load and exacute said javascript? No, probably not. Don’t laugh, I’ve seen this suggested, and no it didn’t actually work.

The fact is that a good code generator equipped with the right generation templates can save you a ton of time and development costs. How much you ask? Well, this month I’ve been doing a lot of research on code generators. So much research in fact that I’ve been seriously neglecting this blog.

As a one person startup company (soon to be expanding! woohoo!), it’s really hard to do everything, so any edge that you can find to get more done in less time is a welcome addition. This includes everything from outsourcing, automation, and yes, code generation.

I’m the type of person who is big into databases. I’ve been writing applications to interface with databases for nearly a decade. There are literally dozens of different schemes for doing so, and so far not one of them has been proclaimed by developers to be the ‘best’. The reason is simple.

Every data layer schema involves tradeoffs. When you abstract the database to the point that it becomes no different than the code itself, you lose bare metal access. If you lean in the other direction and rely on the database to do some of the work, you lose flexibility to use any database you like because not everything works the same in every database.

Sure, the basics are the same, but do you really feel like writing case statements based on how to get the database date, based on the database you’re connecting to? I didn’t think so. Writing database independent code is hard, and rightfully so.

Database vendors don’t make it easy, nor would they want to. Do you really think that Oracle wants their customers to be able to swap SQL Server, mySQL, PostGres, Firebird, or anything else in as the back end database whenever the customer decides they don’t want to pay $40k/processor per year for licensing fees?

The business justification for Oracle and Microsoft avoiding this sort of database swapping becomes even more relevant as computers move towards multi-cores. Oracle has decided pricing should be based on a per-core basis, while Microsoft prices SQL Server based on a per-socket basis. While obviously performance comes into play, if it were easy to swap one database for the other, pricing becomes a much more serious consideration.

I don’t think that there’s any doubt as to whether a code generator can save time if good templates are already available. The tool I’ve been concentrating on lately is CodeSmith, from CodeSmith Tools. It’s not real expensive, and it certainly has a lot going for it. As someone who has worked with SQL Server and web applications for years though, I find it somewhat lacking in the templates that I really want.

Fortunately, there’s a decent sized user community behind it, and all of the templates can be easily customized or rewritten. So if you’re like me and you don’t like the way a set of data access layer templates are written, you can rewrite them.

But is this saving time?

There are a lot of things going against you when trying to answer this question. You have to find the tools, you need to work with them to evaluate them, then you need to build the templates you want, and hope that it’s saving you time rather than wasting it. It’s possible that given all of that, you could potentially use more time. So, how do you know whether a code generator is right for you?

Simple math, my dear Watson.

CodeSmith has a nifty little ROI calculator, which is likely more for marketing purposes on their part than anything else. The project that I’m undertaking right now has something of a large database behind it, and after generating around 80,000 lines of code, CodeSmith estimated that I had saved 1,600 hours of work and nearly $100k in development costs, assuming 50 lines of code per hour and $60/hour for salary.

That alone is justification for CodeSmith, or any other code generation tool for that matter. Just don’t spend 10 months working with and learning how to use the tool before you start on your project or you’re back where you started. Even if other tools don’t include an ROI calculator, it’s pretty trivial to throw a perl script together to count the lines of code that were generated for you.

The other great thing about code generation is that when you make changes to underlying application structures, you can simply regenerate the code from the templates. Did you add a column to a table in the database? Regenerate it, and all of your collections, enumerations, business objects, and everything else are updated quickly, cleanly, and flawlessly.

Wait, flawlessly? Well, maybe. You should still do testing to make sure the new code works, and you should verify that none of the other source files changed. This is best accomplished with source control and differencing source files. You are using source control, right? Since you already have a tool which writes code for you, why not use it to write unit tests for the various objects you’ve created? Why not indeed.

Code generators can also be used to help write documentation. One of the first tools that I ever used for this was javadoc. While it was meant explicitly for java, it did the job. There are other tools out there like NDoc, Doxygen, and literally dozens more. Need to document the database for your application? Some are free, and some not. The documentation doesn’t need to ship with your product, but as an internal document, it can be invaluable for new hires.

Like all things, code generators do have their downsides. These are mainly measured in template development and learning how to use the tools. But if I could trade 160 hours of work and $500 to get 1600 hours worth of code, you can bet your bottom dollar I’m going do it.

To take that a step further, if your company is planning on rolling out 5 products, and you can reuse your templates, you will have accomplished 4 years of development in just one month. That’s a pretty staggering productivity rate. Not to mention that all of your code will have completely uniform coding standards, making it easy to follow for anyone.

After having said all of this, I do feel the need to attach a bit of a disclaimer. While code generators can save you a lot of time and effort, the temptation is certainly there to continue using this two ton wrecking ball to put a screw into a piece of drywall. Not only is it the wrong tool, but the scale is seriously out of whack.

Code generators are not a one size fits all solution. Nor should they be used in every situation. It’s just not always appropriate to use them. Still, they can provide some useful functionality, and are definately worth a good look. And if it can save you a lot of time, effort, money, and ultimately result in a higher quality product, then you owe it to yourself to explore the possibility, even if you decide not to go that route.

Focus on Quality. It sounds innocent enough. Just make sure you do everything perfectly, or at least as perfectly as you are able.

If only it were as simple as that. I distinctly recall during my younger years as a developer, believing firmly that if software you wrote had bugs in it that you were aware of, you simply should continue working on it until it didn’t. This was about the time that Windows would crash on you every 20 minutes, plus or minus 30 minutes. It was increasingly frustrating to even use a computer, mainly because the system would just die, often for what was apparently no good reason.

“Why didn’t they fix this before it went out the door?!?!”

Well, either they didn’t know about it, or they didn’t care. The correct answer is neither. They simply didn’t have time.

As my career unfolded, I ran into an increasing number of managers who simply didn’t know what they were doing, and of course I knew better. “The software still has this bug in it. Why are we shipping without fixing this bug!”

Eight years later, I finally know the answer to that. If we waited until all of the bugs were fixed, the software would never be finished. There’s always something else to tweak, more functionality to add, a bit more performance that could be squeezed out of this algorithm or that one. Developers tend to be perfectionists. I think you almost have to be if you want to be a good developer. But it can be frustrating to know that your software that is going out the door still has problems, or isn’t as manageable as you’d like it to be.

The main source of these problems is really a lack of time to do the work. Adding more developers may solve short term problems, but it’s not a long term solution. It simply adds to the confusion generated when architecture changes are made, or more features are added. But there are some rather simple things that you can do to cut down on the time that you spend working on trivial stuff that you shouldn’t have to devote extra time for to begin with.

If you take steps to increase the amount of time available to you, then any time savings you create can be used to increase the quality of your product. Interestingly enough, many of these crucial time savings actions actually contribute to making a higher quality product.

1) Use source control for nearly everything, no matter how large or small your team is.

Even when working on projects all by myself, I always use source control of some kind. It’s incredibly stupid not to, but people still do it. These are the people who have never lost their work to a drive failure. Your hard drive can go at literally any time. Most of the time, you at least have a little bit of warning. With one exception, every drive failure I’ve ever had gave an audible indicator that something was going seriously wrong.

The drive would start clicking, or making strange whirring noises. That meant it was time to buy a new drive and copy everything to it ASAP. Some people use multiple computers and store copies on multiple machines. This is rather effective, but if you’re doing this, then you’re missing out on one of the key benefits of source control, which is the ability to go back to a previous version. Check in early, and check in often.

If you’re copying files all over the network, then you’re wasting disk space left and right. Disk space is cheap right? Certainly. And if it’s so cheap, then why not use a source control product that tracks the changes between versions for you? Then you can track every version you’ve ever checked in. Personally, I prefer to use SourceGear’s Vault. If you’re flying solo, then Vault is free for one user. Perforce is free for two users. And if that’s the price for you, then you always have CVS to turn to.

There are certainly other options, but source control is a must. Don’t say that I didn’t warn you.

2) Use a virgin build machine to build all of your test builds.

There will certainly be cases where you need to spin off a special build for QA to test something specific, but for your regular releases, you really need to build your product/application on a virgin build machine. ‘Virgin build machine?’ By that, I mean that your build machine should not install the product, nor should it ever. Developers sometimes install software that adds things to their Global Assemblies, or they assume that code or images they added to their local solutions have been added into source control. If the build breaks because they forgot to check something in, then it’s a sign that the process is working.

In my own environment, I have a Windows XP Professional machine that has all of the latest patches running builds. Better yet, my build machine is actually a VMWare image. So I can back it up and move it around whenever I feel like it. If a developer needs to make some tweaks to the build process, or needs some one shot customizations done, he can copy the image, tweak it, and never even touch the actual build machine. There’s no chance of hosing it. Better yet, is the fact that if anything ever happens to the computer it’s running on, you simply fire up the image on another computer. In 2 minutes, your build machine has been recreated. No hassle, no fuss.

“But, what’s the big idea of having a build machine in the first place? My computer can build the software just fine!”

Perhaps. But my build environment is a little more complicated than simply clicking Build. First, it gets all of the latest versions of everything in source control. Then it builds the applications. Next, it obfuscates the C# code. Next, comes the digital signatures, and finally the installer is fired off.

All of this is done through a single script. How many times do you think that I’ve messed up the build process? Yea, not many is right.

3) Use coding standards and make sure that everyone follows them.

This is a much bigger pill for an established development team to follow than for one that is just being formed. Most larger teams have some sort of coding standards they follow so that the code maintains at least a small level of consistency from one class to the next. It’s bad enough when you didn’t write the code, but to follow up after someone who writes really cryptic code can be an absolute nightmare.

During my career, I’ve been on my fair share of development teams where at least one person could have been labeled as “Does not play well with others.” It doesn’t mean they’re dumb. In fact, it can mean quite the opposite. At Clearwire, one of the developers we worked with was practically driven out of the company because his code simply couldn’t be read by anyone. He really was a very smart guy, and his code had a tendency to be very good. But when bugs were detected in his code, it was very nearly impossible for anyone else to track down the problem. Often, it was interoperability problems due to architecture changes.

But still, the fact remained that his code was simply too cryptic. There are a lot of tools out there which can help with this sort of thing. You can run them as part of your nightly build process. You’re doing those now, right? On your new, fancy VMWare image? Good.

Recently, I started a project to get the source code here at Moon River Software under control using FXCop. By doing so, I can make sure that all naming conventions are followed, and that all of the code at least conforms to a standard. The great thing about standards is that there are so many to choose from. And FXCop appears to allow users to customize the rules to some extent, further defining their own internal development standards. This is probably more time consuming than either of the previous two efforts, but code maintainability is very important.

4) Document everything that you can.
I’m a big fan of documentation, but notice that I said ‘everything that you can’ as opposed to simply saying ‘everything’. The fact is that occasionally, there simply isn’t time. Of course, that’s why you’re reading this. Try to make time to do documentation. This includes application documentation, internal process documentation, network documentation, etc.

Most teams I’ve worked on have had their own intranet and development server. Developers have access to create web pages with links to all sorts of things such as testing documentation, build environment requirements, setup requirements, source code documentation generated via javadoc or other documentation generators, etc. You know what works great for this sort of thing? A Wiki. Most small teams could benefit from setting up a Wiki to help the development team add documentation for various procedures and technical notes. Even full product designs can be done in a Wiki, because it’s a fairly collaborative environment. Chat boards can also work.

5) Create a functional spec for new projects and new versions.

There’s a difference between a functional spec, and a design spec. It’s a pretty basic difference. Functional specs describe how the software will work for the end user. Design specs describe the internal workings of the software and how different components interact with one another. The combination of the two will give you a solid blueprint for your software and how it needs to be put together.

6) Reuse code whenever you can.

No, this isn’t a license to copy-paste code. Quite the opposite. Anytime that code can be reused, you accomplish a number of different things all at the same time. First, you save time because you’re not rewriting the code to begin with. Second, you’re consolidating functionality so if bugs are found in the code, it will need to be fixed in fewer places. Finally, if the code has been in use previously, then theoretically it has already been tested to some degree. So, the quality should theoretically be higher.

7) Automate your testing, not to mention everything else that you possibly can.
I have to admit a little bit of prejudice here. I hate doing testing. It’s not so bad the first time, or even the second. Possibly the third. But by about the tenth time of doing something, it’s pretty much all I can do to keep from putting my head on the railroad tracks and singing as load as I can. Repetitive tasks are the most boring job ever devised. It should be the punishment for delinquint children. But these are the sort of tasks that can be automated. Quite easily in fact. There are probably a few dozen products on the market, all aimed right at allowing you to write fully automated test suites.

Many are so simple that you can click a button and tell it to record what you do for the next few minutes. Then after you click stop it will show you what keys you pressed, when you pressed them, where your mouse moved, and lets you add and delete new actions as if they were a movie sequence. Pretty slick stuff. Imagine writing the tests to test the testing software though. Just the other day I had to modify a script that was perhaps a couple thousand lines long.

I use CodeWright for most of my script editing. I got into it back in 2000, and have used it ever since. Anyway, I discovered that CodeWright has a Macro feature, where it will record keystrokes for you. I recorded holding the shift key, down 6 times, delete, down two more times, hold shift, hit down, delete, down two more times, hold shift, down 6 more times, finally delete again.

It took me perhaps 30 seconds to figure out how to use it and set it up. 200 macro runs later, I was finished. Now, I’m a very well accomplished script writer and could have easily parsed the file with a perl script, sed, awk, or something along those lines. It really wasn’t worth the time and effort. It was too much work to do manually, and too much work to write the script to do the work. But the macro was the perfect solution. The same thing goes for automated test suites. I think it would be hard to go wrong with an automated test suite. It’s amazing the things they can catch, especially if you wire it up to the end of your build process to install your product, load it with test data, use it, and then uninstall. All of this in the middle of the night when you’re not doing anything else with the machine anyway.

I wrote this article several days ago and waited to publish it because I felt that I was on the wrong track. It seemed as if I was concentrating on all kinds of things that a company should be doing that were related, but not exactly focusing on quality. After careful consideration, I realized that I was on the right track and simply didn’t realize it yet.

If you want to focus on quality and you try to do so explicitly, you’re going to fail. You can’t simply tell someone to write better code and think that it has the remote chance of happening. You also can’t decide to design your software better. That doesn’t happen either. At least not until Version 3 hits the shelves.

The simple fact is that you can’t focus directly on quality. You need to focus on everything around it. Focus on the processes surrounding the things that you want to be higher quality. Want higher quality code? Put a process in place for code reviews. Want better documentation for the software? Have your doc writer send the documentation to the developers to review for technical accuracy. Then send it to the marketing folks to see if they understand it. If both groups give their seal of approval, then your documentation is flawless.

Want a better designed product? Wait till version 3. Just kidding. Write functional and design specs. Then read them. Reread them. Make sure everyone else has read them. Read them again. Then execute.

It’s not exactly rocket science. But people have been screwing this up for years now, and the trend is very likely to continue. The fact is, people generally don’t take the time to improve processes because they don’t think they have the time available to do so. In fact, I don’t think that most people realize the benefit of having these processes in place to begin with. It’s amazing how much easier your life can be if you just change your focus a little bit.

By focusing on the processes surrounding the problems you’re encountering, you can modify the process to be better. This will in turn, provide higher quality in whatever it is that the process is controlling.

For the past several weekends, I have been entrenched in the truly terrible task of stripping wallpaper and getting the walls ready for painting. I’m probably about as far from being a violent person as you can be without actually being a pacifist, but were Jean-Michel Papillon still alive today, I’d probably beat him to death for inventing wallpaper.

The painting itself doesn’t bother me. I spent an entire summer of my college years painting the walls in a high school. No, it wasn’t a penitential mandate from the state, and yes it really was a three-quarters time second job that I got paid for.

Back to the point, as we were just finishing up the first coat of primer, my wife asked me if we should bother doing a second coat of primer on the walls. My immediate answer was yes. We weren’t going to skimp on the primer.

This doesn’t date back from my brief brush (no pun intended) with a career as a painter. My answer came from the knowledge that to do a good job required at leat two coats of primer, and anything less and it would just be shoddy work.

This past weekend, I went to get a haircut and on the way back I passed a car dealership. Now, I look at cars whenever I pass a used car dealership. I’m not sure why. I have no intentions of pulling in, but I still like to look as I drive by. It occurred to me while I was looking how happy I was that I traded in my old car about a year and a half ago. I had been driving a 1997 Saab 900S, which after eight years was really on its last limb. I had only owned it for five of those eight years. Now don’t get me wrong. I loved the car when it was running well.

The key of course, is when it was running well. It didn’t always run well. In fact, it seemed like every time I turned around I had to sink a lot more money into it. From the year after I bought it to the time I got rid of it, I spent on average about $2,000 per year in addition to car payments fixing it. The serpentine belt, the alternator, rotors, etc. If it was expensive, it broke on my car at one point or another.

Eventually in December of 2004, I traded it in for a 2002 Honda Accord. One of the key factors in my decision of what to purchase next was how expensive the new car was going to be to fix, and how often I would have to fix it. This entire train of thought came about because of the used car dealership that I saw, and the train ended with “I’m glad I bought a Honda. I haven’t had anything go wrong with it since I bought it.”

This happened as I was drawing near the end of Moon River Milestones v1.1.0 development, and it got me thinking about quality in software products.

Hypothesis: Would people pay a premium for quality?
My immediate reaction to this question was yes. I mean, who pays more for software with a lot of bugs in it? For that matter, who in their right mind pays more for anything with more defects in it? Well, read the latest Consumer Reports for car ratings. The BMW 7 Series of used cars from 1998-2005 is ranked as one of the Bad Bets. So is the Lincoln Navigator, Mercedes-Benz CLK, and a couple of Jaguars, all of which are pretty high on the price list.

I will concede that quality and price are not the only considerations people make when purchasing a new car. But there should at least be some correlation between price and quality right? Who in their right mind doesn’t value quality!

Well, let me beat up on the car industry a bit more. Recently, GM announced a restructuring plan to hopefully save them billions of dollars. Looking at their Wikipedia entry shows the plants that they will be closing. Let’s put this next to the J.D. Power and Associates listing of the Assembly Plant Awards for highest quality. Notice anything?

It would appear that the Oshawa #2 Ontario Canada plant is going to be closed, and the Oshawa #1, Ontario plant is going to be scaled back to eliminate the 3rd shift entirely. Interestingly enough, both of these plants are on the J.D. Power and Associates lists for outstanding quality listed as the number one and number two plants.

Perhaps its just my overly cynical view of public companies and thier quest for increasing their stock prices at any expense, but wouldn’t you want to increase production at these facilities to compensate for the ones that are producing garbage?

Apparently not. Somehow, their marketing and sales forces are going to overcome the fact that they’re selling crap to the consumer by hiding behind a limited warranty.

“Hey, if you want me to take a dump in a box and mark it guaranteed, I will. I got spare time.”

From the movie Tommy Boy

On the other hand, you have companies like Hyundai offering outstanding warranties on their cars that would have seemed insane 20 years ago. They can do this for only one of two reasons. Either their cars are genuinely good and can last until the end of the warranty, or they are made so cheaply that they can fix all of them, and still make a profit. Take a wild stab in the dark at which of those two companies are growing, and which is shrinking? I’m willing to bet that part of it is due to quality. In fact, Hyundai has bet their reputation as a car company on their quality and you know what? They’re beating GM at it. True or not is a different story, but they’ve proclaimed their quality to the world, and the world is listening.

But these examples are cars, not software. The fact of the matter is, that my entire argument relating cars to software is fundamentally flawed. It’s unfortunate that I realize this now, after writing over 1,000 words on the topic. I’m probably lucky in the respect that writing helps me think things through. That’s why I write a lot of things down. In the end, it boils down to much less than I originally wrote. Most of what I write down I end up throwing away because it sucks. But the journey does tend to yield a lot of good results.

The fundamental flaw in my argument is that software and cars are like apples and oranges. You can’t effectively compare them on a general basis. All cars have a common purpose. They get you from point A to point B. There’s a lot of other features as well, but they all basically do the same sort of thing. That’s why the differences are features. That’s not the case with software because software functionality varies widely between applications. It accomplishes something. But beyond that, there’s very little you can generalize about. It solves a problem maybe? I don’t know how well games fit into that category unless you count boredom as a genuine problem. And if that’s the case, then you’re either a games developer or in college and should be studying instead of playing Quake.

There’s plenty of software out there that I think is absolute crap, but still sells like hotcakes. And I don’t mean crap in the sense of the functionality of the software because some of it could be really useful. I mean crap terms of the quality, such as software that is useful, but crashes every third click on the gui, and those sorts of things. It’s positively insane how poor some types of software work, yet it still ends up on lots of computers. Don’t believe me? There’s software that is distributed by Dell which uses some sort of watchdog to detect when the hardware is locked up. When it detects a lockup, it reboots the system.

At the core, it sounds pretty simple. Quite useful in fact, especially for web servers. It stops working, so the hardware reboots the machine to get your website back up and running without any user intervention. In practice, it’s not nearly on par with where it should be. I’ve been told by people who have this software that they disable it on most of their servers because of its flaws. Isn’t that Petarded?  It’s sole function is to detect a hardware lockup and reboot. How complicated can it be? What flaws could it possibly have that are that bad?

It turns out that the flaw is that the hardware watchdog blue screens the machine quite regularly. Ummm…. yea. Just be thankful that this software doesn’t run the power steering in your car yet. They might want to focus on quality, don’t you think?

Quality sells, but not in the way that you think.

If you asked me whether or not I would buy another Saab, I would likely tell you no. I liked my car, but it was the flaws that I remember more than how enjoyable the car was. My Honda has been running without anything more than regular oil changes and minimal maintenance for a year and a half now. Currently, it is 4 years old and running strong. Would I buy another Honda? Most likely. Would I recommend one to my friends? Of course.

Go back and reread that, because there are two very important things in there. The first is, would I buy another. The second, is would I recommend one to my friends. More importantly are my answers to the questions. I think these are key principles that really do apply to software.

If someone is willing to buy more of your product because they like it, then I believe that they are just as likely to recommend it to people they know. Perhaps not actively as an evangelist, but if a friend said they had problem xyz and your product solves that problem, this particular someone would recommend it to his friend because it works to his satisfaction. Even better is when your product is so far above and beyond that of every competitor that these customers become evangelists and do your marketing for you.

When a company spouts off lots of rhetoric about how great their software is, how it will solve all your problems, make you breakfast in bed, etc, we take that with a grain of salt. Marketers lie. All of them. Seth Godin says so, not to mention there’s a website that says this too, so it must be true.

But when you hear how great something is from your colleagues, classmates, friends, drinking buddies or whomever, you take it a little more seriously. It becomes more credible somehow because you trust those people. Maybe not with your life, but in a pinch, who would you trust to save your hide, your best drinking buddy, or the guy hocking the software that in addition to its original purpose, makes breakfast in bed? It’s not a hard decision.

Having current customers buy more of your software is a great concept because you already have a business relationship with them. You are a known entity. Customer evangelists are the single greatest accomplishment a company can achieve because your product is good, and your customers love you for it. Apple has lots of these. It’s a long, hard road though. So, how do you get there?

Focus on Quality.

If you make a conscious effort to focus on the quality of your software, you will probably see less of an immediate impact on your sales than if you implemented new features. I would venture so far as to say that I am sure of that. Quality doesn’t happen overnight. It takes a lot of time, effort, and a commitment to quality. In time, assuming you had a decent product that was desired in the market to begin with, those efforts will likely pay off tenfold.

For a smaller company, focusing on quality can be really hard to do, especially for new products. A version 1.x product typically doesn’t have nearly as many features as the well established competitors do. It doesn’t have the established customer base and steady revenue which would provide the company a position where they can focus on quality. In fact, the smaller the number of developers in the company, the harder it is to focus on quality because of time constraints. Their time must be divided among bug fixes, new features, customer technical problems, one or more products, and a host of other commitments.

It gets worse as time goes on and your business grows. More customers means more support calls, and more technical problems. Unless you focused on quality from day one, as the number of customers rise, so will the support calls. The only way to fight this is to make your code virtually indestructable, and that means focusing on quality.

Focusing on quality will reduce bug counts at the cost of longer development cycles and fewer new features. By talking to your current customers, you can find their pain points in your own software. Perhaps some of the menus are too confusing. When you learn this, fix it! This will allow you to fix the software, reducing the number of support calls from current customers while increasing the number of sales. The net result is the company receives more money for the same number (or fewer) support calls. This frees up your time for other things, such as implementing new features, intensifying the quality of your product, or working on other products.

Conclusion:

While it’s hard to say with absolute certainty whether people would actually pay a premium for software that is guaranteed to be of a higher quality, the fact remains that most people (who don’t run public companies) value quality. Quality isn’t just a word. It’s a commitment. And if you think otherwise as you attempt to address it within your own organization, you’re going to fail.

You can’t pay lip service to quality because it shows. People notice. No amount of lying from your marketers is going to tell someone that your software will rock their world when it crashes twice a day. Polishing an application to look great is related to quality, but not quite the same thing. Buggy software that looks great is still garbage. The code has to work flawlessly every time, and when it doesn’t, you need to know about it. This is where customer feedback forms come in.

When you focus on quality from day one, you set a prescedent for the future, that poor quality is not to be tolerated. Nothing less than perfection is acceptable. This doesn’t mean you can’t make mistakes. It simply means that you don’t let bugs go unchallenged. Anything more complicated than a Hello World! program is likely to have bugs. Heck, I’ve misspelled the word Hello in a Hello World! program before because I was typing too fast. Finding bugs or customer problems and allowing them to persist for too long is when you start running into problems.

In my next article, I’ll be examining many different ways that you can improve the quality of your products.

It’s fairly obvious that whenever you work on a development team, everyone fills different roles. This can sometimes make it difficult to evaluate the individual contributions of particular team members, especially when their roles overlap to a large degree.

These roles vary from team to team, as do specific titles but most are essentially the same. You have the grunt, who does a lot of coding. You have the architect, who does mostly design work. There’s a team leader, who puts together a schedule, assigns tasks, and generally tries to make sure things get done on time. Most people fall somewhere between the grunt and the architect. Some grunts do design work, and some architects write code, so obviously there’s some overlap.

What I’ve noticed is that most teams have skillsets that overlap in so many different areas that it is pretty rare when one person truly owns something. There is one place where this becomes a serious problem: databases.

If your application uses a database, you need a database gatekeeper.

Every team that works with databases needs a gatekeeper for each application they are working on. It doesn’t need to be the same person for each application and arguably, is better if it is not. But you need someone to fill this role nonetheless. The gatekeeper I will focus on is the database gatekeeper. You can easily argue for other gatekeepers. Others that come immediately to mind are gatekeepers for the build, source code check-ins, branches, website applications, code versioning, bug tracking, etc. The fact is that any important part of a project could probably use a gatekeeper to keep things working smoothly.

Even on development teams which rely extremely heavily on databases, it has been my experience that there are perhaps only a small percentage of people on the team who truly know how to use a database effectively. Most developers view a database as an overglorified list of objects for which exists solely so they don’t need to worry about important details like saving the data or maintaining it between application starts.

This is not only inherently wrong, but it’s a dangerous line of thought. The fact is that a database can be an extremely complex part of your software and whomever designs it really needs to know what they are doing. Database design is very important for several different reasons. A poorly designed database is like a poorly designed algorithm. It still works, but not nearly as effectively or as efficiently as it could. Operations can take orders of magnitude longer if they aren’t done correctly.

If not used properly, a database can also end up filled with a lot of useless data. Consider this simple example. You’re building a nifty little forum for your website. You create a table called ‘User’ to store… err… well, the users. And you create a table called ‘Topic’, and ‘Post’ to store the topics and posts to that topic. Seems pretty simple.

But there are some basic concepts that must be followed. For example, what happens if you delete a user? Are all of the topics they created deleted? Are all of their posts deleted?

Yes, they are deleted. If someone comes in and spams your forum, perhaps you do want to get rid of everything they did, especially if it was a script that logged in and posted into every single topic.

The answer is also, No, they are not deleted. Perhaps you do routine maintenance and delete users who have been inactive for a period of time. You then implement a mechanism in the code to state that a post was made by a user who has been deleted.

People who don’t know how to use a database tend to fall into one of these two categories because they simply don’t know any better, and not because they made a conscious design decision. Their expectation is that the code they are writing will take care of every situation and will handle all of the business logic. These are the developers who have lots of extraneous logic in their code to handle referential integrity checks, and use a “select max(id) from table_x”, then increment the value by some constant value to determine what the id of the new element they are inserting should be.

This is not how it’s supposed to work. Every database I have ever used has some form of sequencing to help you determine the next identifier, be it a sequence in Oracle, or an Autonumber in SQL Server. The referential integrity should be enforced by the database with foreign keys. Unique data should be maintained unique with constraints. Every database comes with a set of functions that help you maintain the data.

“The purpose of a database is to maintain and manage your data”

I’m going to avoid getting into how a database really should be used. The fact is, that someone on your team should be assigned to be the Database Gatekeeper. It is his or her job to make sure that the database is designed correctly to begin with. If a developer needs to add a column to a table, it should be cleared by the gatekeeper. Not only should it be cleared by the gatekeeper, it should be him who adds the column to the database scripts.

What good can possibly come of this?

For starters, you get a solid naming convention. People experienced with databases tend to have a naming convention they use to help them figure out what the relationships between tables is by glancing at the database schema. Feel free to try using an established standard such as Hungarian Notation.

“The great thing about standards is that there are so many to choose from.”

Of course, this only gets you so far. What about the use of ’s’ at the end of table names? Some people will name a table ‘User’, and others would name the same table ‘Users’. Mostly, it’s a matter of opinion. My personal preference is to avoid the use of ’s’ at the end of a table name unless the table is used (solely for) relating two tables to one another. Regardless of your mechanism, at the end of the day, don’t you want all of your database tables to follow the same convention?

This is where a gatekeeper comes in handy. Since it’s his responsibility to make the changes, it is also his responsibility to make sure that those changes adhere to the established convention. One day when your database gatekeeper is long gone, the next one can look at the database design and at least have a chance of making changes without making things difficult for everyone else.

So far I’ve only addressed naming conventions. What else can a database gatekeeper can do for your team?

Prevent data redundancy.
Your database gatekeeper can tell you if the data you want to add to the database already exists. Having been the gatekeeper on a team before, I can’t count how many times I’ve prevented a developer from storing redundant information in the database. These redundancies are not only confusing, but waste space in the database.

Hard drive space is generally cheap these days, but what about access speed or the cost of storing that extra data? A pre-existing column might have been indexed, while the new one being created is not. Backup applications need to store that extra data as well. Both of these have hidden costs that a developer who is intent on implementing a feature might not think of.

Enforce database integrity.
Your database should enforce the integrity of your data, not the application code. When a database requires that relationships exist between tables, you can’t insert or delete data without satisfying that relationship. Using a sufficiently complex database merely as a storage center will cause you innumerable headaches trying to root out data inconsistencies. Most developers find these restrictions to be annoying to no end. “I want to delete this data, and the database won’t let me! Stupid database. I hate this thing!” That’s why they don’t use foreign keys and constraints. The fact is, if the database is preventing you from doing something, there’s probably a good reason.

Use the right data types for the job.

In SQL Server, it’s very easy to make a column of the type int. Too easy in fact. Enterprise Manager was/is a great tool, to be sure, but it certainly gives too much power to those who might not know what to do with it. Most people use an int without thinking twice about it, but is it really the right data type for the job? In SQL Server, anything that’s likely to go over 2 billion is probably not a good candidate to be an int. Similarly, any number which isn’t going to exceed 255 isn’t a good choice for an int either. Unicode, non-Unicode data, and variable length data can have serious impacts on the resulting data storage.

Using Unicode where ASCII text is being stored immediately doubles your storage requirements. The difference between 50 and 100 bytes doesn’t sound like a lot, but it can cost your customers money, nonetheless. Imagine using the wrong data type on every single column in a table because a developer simply didn’t know any better. Companies are pretty paranoid these days about backing up their data, and although hard drives are generally cheap, RAID 5 isn’t. Tape drives are terrible to work with, and every byte counts. An extra 100 bytes in a single row can result in 2GB of extra data in a 20 million record table. Multiply that by 100 tables and you run into some serious storage problems. Reloading a database in its entirely presents its own problems.

SQL Server 2000 ships with 27 built-in datatypes. Are you using the right ones?

Index important bits of data for speed.
Your gatekeeper can help speed up certain operations by tuning the indexes on the database. In laymans terms, it means building a cache to help find certain bits of data quicker. If a specific operation in your application is too slow, talk to your database gatekeeper and see what he can do. He might be able to speed it up by a factor of 100 without a single algorithm change in the application code.

Use Stored Procedures for complicated operations.

While Stored procedures are somewhat new to mySQL, they’ve been around for years in SQL Server and Oracle. Using a stored procedure moves some of the business logic out of the application and into the database. This is faster for a number or reasons, first and foremost is that the data gathering is done on the database server instead of over the network. Stored procedures also offer more security in applications by preventing SQL injection attacks.

Use scripted operations for all changes.

When I worked for Wegmans, it was departmental policy to refuse any database changes which were not scripted. While security certainly played a role in this, reproducibility did as well. If table changes were made via a gui, data is loaded, and then the database server crashes, how can you be sure that everything is restored exactly the way that it was before if you used the gui?

The short answer is that you can’t. Most tweaks are small in nature, but even minor tweaks can have huge performance impacts, especially when you have a database that is several terrabytes in size. A database that is entirely scripted is much easier to manage. It is easier to test, and easier to tweak for different releases.

Intelligently choose relationships.

I’ve seen people use descriptions to maintain relationships between database tables. This is bad practice at best, and assinine at worst. Descriptions have a tendancy to change. And if someone is using it to maintain relationships between tables, you will need to change it in every table it occurs. Without a gatekeeper, it will be difficult to track down those occurrences. If your application is using a description field as a table relationship, you probably have more serious problems.

What is the single worst way to choose a gatekeeper?
Use his resume. I recall interviewing someone who seemed to have pretty extensive SQL Server 2000 experience on his resume with the intent that he would be our database gatekeeper. Our team was really lacking in people who had database experience, so I was really looking forward to the interview. I asked him some database questions, and he seemed to know what he was talking about. Then I asked him about triggers, and what he had used them for.

He explained a very elaborate system of triggers that he had implemented to delete data from a very large database that was his current project. To simplify things, recall my example of nifty forum software above, imagining that he was attempting to delete a user and wanted to delete all of the topics the user had created, all of those topic posts, and delete the posts of that specific user. A trigger is a database mechanism for intercepting inserts, updates or deletes to the database and performing additional operations before the actual insert/update/delete takes place. In this case, he would delete the user, then the trigger would intercept the delete.

This trigger would in turn attempt to delete the Topic, thus firing a second trigger, which attempted to delete the Posts that the user had made. The original trigger would also delete all of this specific users’ posts. Traversing down the chain of triggers and then back up again results in the ability for the developer to delete a user and all associated data, while maintaining the database constraints, and referential integrity.

After patiently listening to his explanation of how he had used triggers, I asked him how well the resulting implementation worked. Apparently, not very well. Data inconsistencies were still being found as some of the table hierarchies traversed 10 levels deep. It was extremely difficult to track them down. He had spent the last 3 months trying, and still wasn’t finished. I asked why he didn’t use SQL Server’s built in ‘Cascade Delete’ and he had no idea what I was talking about.

After about five more minutes of questions, it quickly became clear to me that this developer had some serious holes in his knowledge of databases and wasn’t going to cut it as our database gatekeeper. He understood that referential integrity was important, but he lacked sufficient knowledge of the tools he was using to do it properly and efficiently.

How do you choose a good gatekeeper?

This is a lot harder to answer. Even the person I interviewed in the above example would probably not have been a terrible database gatekeeper if he weren’t by himself. I included the example to prove that a resume is not a tell all indicator of suitability for any given task. I think that so long as the person you have chosen has a decent knowledge of more than one type of database, and has a genuine interest in learning more about them, he is likely to be a good candidate. Someone who used to be a DBA would be an excellent choice, but ex-DBA’s turned developer are hard to come by.

Another possible solution is to use a couple of people who have database training to discuss any database issues. Two heads are better than one, and while a lot of their knowledge may overlap, some of it is likely to be distinct. People hate committees, but this can work well on some teams without overloading any one developer or trampling on their egos.

Conclusion:

Every development team using a database could gain some benefit from a full time DBA. Unfortunately, very few of us are going to have that luxury. Your best chance for avoiding some nasty database snags is to assign someone to the role of Database Gatekeeper. It is their responsibility to maintain the database design, make changes, and be the resident database expert for that application. It will save you time, money, and ultimately a ton of headaches.

Recently, I was evaluating obfuscators for Moon River Milestones, and after going through all of the motions, I finally settled on one. Now, being as tight with money as I am these days, not to mention the massive costs I’ve incurred while starting up my business, I decided that I’d wait a little while before I bought it so I could decide on an installer to use as well.

So Thursday afternoon, I pulled out the Moon River Software credit card, and proceeded to spend a few thousand dollars. This software isn’t exactly cheap, which makes the evaluation process that much more important. After all was said and done, I laid out nearly $3,500. Unfortunately, not everyone is up on the times about ordering things on the internet and instant gratification. So, I had to wait until Friday afternoon somewhere around 7pm EST before I finally got all of the software I ordered.

So, back in the bat-cave I’ve spent the last two days trying to get my build environment fully automated. The ultimate goal was to have a single script that I fire off, and when it’s finished, I get a shiny new msi package which I can install and it will work, everyone will be happy, and the Wicked Witch of the West melts into a puddle. Or at least that was the plan.

Let me back up to the beginning so I can explain why all of this was necessary. When I was evaluating the installers, I decided that I liked the Wise for Windows installer the best. It seemed to have great support for web applications, could integrate very easily with Microsoft SQL Server, and it was very easy to learn. The problem that I ran into was that I wanted the user to be able to select the SQL Server database to install Moon River Milestones to. I know that some people have naming conventions they want to follow and I wanted to provide them with the flexibility to do what they wanted. But the installer was giving me the same problems I ran into back when I did my evaluation.

The Wise for Windows (WfW) has the ability to do find and replace on text strings, which seemed to work well enough during my evaluation. I ran into a problem where it stopped working entirely at one point, and after contacting support, and wasting well over 25 hours of time on it, I tried recreating my project. Everything seemed just peachy. I still had to do some work to create the custom dialog boxes which let people type the name of their database, but I decided to leave well enough alone. Support confirmed that there seemed to be some sort of problem, but they weren’t able to duplicate it. Other people had workarounds that also involved recreating their projects. I certainly didn’t want to have to recreate my project every time I wanted to make a minor change, but luckily I have something better. Source control. I have all of my code backed up in Vault, which is an incredibly great source control tool from Source Gear.

So, I had purchased WfW, and now needed to make the aforementioned changes. I had received my license for the obfuscator on Thursday night, and tried installing it on my build machine. Insert barf noises here. My build machine choked on it, and hard. The installation instructions clearly state to uninstall the evaluation version before installing the full version. Every time I tried though, it choked and died on me. Since it was integrated with Visual Studio, it fired up four separate msiexec.exe files. FOUR of them! And it still couldn’t uninstall properly.

Finally, I decided to do a repair, and it seemed to work ok. My build machine is a VMWare image, so taking snapshots isn’t a big deal for me. Speaking of which, VMWare has a great deal going. It’s called a VMWare Technology Network subscription, or VMTN for short. It gives you access to VMWare Workstation, GSX Server, ESX Server, Virtual SMP, and P2V Assistant, all for under $300. Talk about a sweet bargain. The lone caveat is that you can’t use any of the software for production use. It’s all for development use only, and you must have a VMTN license for each developer. You also can’t give demos to clients using any of the software, but since I’m just using it for development, it works out great for me.

So, with my VMWare image loaded with my obfuscator, and my new installer, I started pulling all the final pieces together for my automation. But I ran into my arch-nemesis again. The replacement feature stopped working. I spent most of Saturday trying to figure this out. Most of Sunday as well. I’d previously spent 25 hours on this one problem, so I’ve probably logged 35-40 hours for this one problem. To be specific, there’s a replace function that sets the name of the database the user wants to install to. Having had my share of problems with the obfuscator, and seen the sort of problems that installations can cause, I wanted to make sure that the user got everything right on the first try.

The installer is the first thing the customer is going to run when they download your product. They’ve already paid for it, but you want their experience to be as painless as possible. The very reason they bought your software is because they have a problem they need to solve. So, do everyone a favor and make sure your installer works, and test it as many times as you can, and in as many different situations as possible. Your customers don’t want any surprises, and neither do you.

So, after flipping through countless menus, and attempting numerous times to get the damned SQL code to execute properly, I finally noticed a seemingly innoccuous dialog box under the files I was installing. Notice anything?

I did too. ‘Read Only’. On a hunch, I did some experiments. It turns out that the WfW Installer copies all of the files to the target machine (even the temporary ones) with the same properties you have on your machine. And since I’m using mutually exclusive mode in Vault, my sql files were also being created as Read Only on the target machine. The replace function was failing because the installer does post-processing on the files!!

My guess is that it’s relatively simple to write out the file, and then search through the text for certain strings, as opposed to extracting the file into memory and then searching for strings to replace before writing them out. But I swear if the developer who made that decision was within arms reach when I finally realized that, I’d probably still be choking his lifeless body. So, I made a few changes, and everything seems to work fine now.

I had been hoping to do a full launch of Moon River Milestones as of Monday morning this week, but that’s simply not going to happen now. I’m so close to being completely done that I can taste it. No, I think that’s peanuts and beer. But it’s 5 minutes past midnight, and I’ve got a long day ahead of me tomorrow. Customer demos and such. You understand. So, I’m still working on the final details of the launch, but it is certainly within arms reach. Within the next few days, you’ll be able to see Moon River Milestones in all its glory.

And then the real work begins.