Musings of a Software Development Manager

The Case for Agile Development

2004-07-15T08:08:00.000-05:00

I’ve put this document together as a result of my recent attendance to the Agile Developers Conference. The purpose of this document is to convince my company that the Agile methodologies make a lot of sense, and that we should adopt them. I’ve put it on my blog in the hopes that it may help others.

The paper covers two topics: Project Management and Agile Teams. Both go hand in hand, in that you can’t be successful in one without success in the other. As always, I welcome all comments.

Project Management, the traditional way

The "Traditional," or "Waterfall" method of project management has been in use, in some form, since the term Project Management has been around. The basic steps of this method, in software development are:

1. Plan – Simply planning and securing the resources needed for a project
2. Analyze – Gather the requirements for the project
3. Design – Perform the technical design or model for the project
4. Code – Write the software
5. Test – Test the software with some sort of QA methodology
6. Deploy – Deliver the project to the customer
7. Customer Acceptance – The customer accepts the project as complete

The term "waterfall" is used when viewing the above steps in the following chart:

The application of the waterfall method requires each step of the process to be completed prior to starting the next step. Think of the steps above as a container that can hold water. Once the planning step is full of water, it will then spill over to the next container, which will fill up, and so on.

There are several benefits to this approach of project management:

Once the Design piece is complete, a fairly accurate estimate can be given for the project in both resource time and duration.
Everyone on the team is working from the same analysis and design specifications, so everyone understands the "big picture."
When the project is deployed, the customer has a complete product with which to work.

There are also some disadvantages to this approach of project management:

High risk – Once the design piece is complete, and an estimate is give, there is a risk that the project will not continue. At that point, the time spent, even if funded, is wasted.
High risk – The possibility exists at any point in this cycle for a problem to be found that was not considered in the analysis or design phase. If this occurs, it may be necessary to back up some steps and start from that point again.
Due to resource allocation, there may be a lag from the time one step ends and the next step begins.
The customer must wait until all six development steps are complete before they have access to the product.
The customer is forced to do all of their testing on all features of the product at once, at the end of the development cycle.
The customer must wait until all seven steps are complete and accepted before they have a usable product.

Even with these disadvantages, there are some projects that will work perfectly well following the waterfall method. These tend to be very small projects - since they are short term, the risks, as shown above, tend to be smaller.

Project Management, the agile way

The Agile Manifesto was created in the late 1990s by a group of developers and project managers that realized a need in their community to simply make things better. They focused on massaging the traditional method of project management. Remember, when used on small projects, the traditional method works just fine. The basis of the agile way is "iterative releases," and when you take a new look at the project management chart, with the same steps, you can begin to see the advantages:

The only thing that has changed in the model is that instead of completing each step for the whole project before moving on to the next step, each step is done based on a single functional point. All the players from the waterfall method are still playing, but their playing with a much smaller ball. In fact, each feature should be able to go from the analysis phase to the deployment phase in no longer than two weeks (10 working days). Once of the fundamental reasons this works, is that it’s easier to work with a "thing" than with an "idea."

There are some pretty hefty advantages to this method, when compared to the waterfall method:

The customer now gets to try out each feature as they are completed. This allows them to add their input at the very early stages of the process, instead of at the end.
Resources in each category are not waiting around for all of the previous phase to be completed, which means that programmers are always coding on the project and testers are always testing, etc.
The interaction between the phases is on a much quicker pace – i.e. a feature is tested immediately upon its completion, while it’s still fresh in everyone’s mind.
The customer sees the progress of the project based on the functionality they see, rather than a "progress report."
Most of the risk that was stated as disadvantages from the waterfall method are gone completely, or greatly minimized.

Even with these advantages, there are some disadvantages:

The customer will no longer be able to get the "fairly accurate estimate" up front.
The customer has to accept that they will be testing and evaluating the features throughout the project’s life cycle, and be prepared to give feedback.

Project Management with Feature Driven Development

The third method of project management is called Feature Driven Development (FDD). The FDD method builds off the "Agile" method, but has one major difference: The planning and analysis are done in their entirety before the coding begins.

The two things that come out of the analysis phase of this method are an overall model of the project, and a detailed features list. The model is the 10,000 foot view of the project. The features list is a categorized list of features that are to be delivered. The features in this list are put into the following categories: Must have; Should have; Could have; Won’t have. The features are worked on based on their priority from this list. It’s important to note that once this features list is accepted by the customer, a "hard" need-by date for the project can be scheduled, with the understanding the there may be some of the C and S features missing.

An Agile Development Team

An agile team is a subset of a department of agile developers. It’s also a team where the most talented programmers are still programming, and not forced into a management or supervisory position. A typical agile department is set up like this:

The Manager of the department is responsible for resource allocation, staffing, and supervision of everyone in the department. When a project is required, it is the manager that will assign the project to a Chief Programmer.

The Chief Programmer is the person responsible for getting the technical aspects of the project completed. The CP will model the processes and features, and then pull resources from the Programming Staff, with the approval of the Manager.
During the project, the CP monitors the work being done, does some coding him/herself, and conducts code reviews with the programmers.
At the end of each project, the CP will submit a review of programmers’ performance for that project to the Manager. This provides regular feedback to the manager about a programmer.

The Programmer is a resource used to complete the project, under the guidance of the Chief Programmer. There can be varying levels of programmers (trainee, senior), and different areas of expertise (interface design, business logic, database). Because the Chief Programmer is providing the design for the programmer to work from, and because the programmer is working on individual features, they do not have to be product experts.

This team design does not force the best developers into supervisory positions. It keeps them where they are most useful (and profitable). Some studies have shown that the "A-list" programmers in an organization can have between a 10:1 and 50:1 productivity advantage over average programmers. Based on this, there’s no reason to force someone into a position outside of development. As a CP, they are working on the projects directly and are able to mentor others.

The agile development team will use development practices to obtain the goal of project completion. Some of these practices are:

Test Driven Development – Writing test cases BEFORE the coding for a feature is begun. You have to know what you’re trying to accomplish before you set out to accomplish it.
Pair Programming/Code inspection – Having two sets of eyes on every piece of code. This has been described as the single most effective way to catch bugs in software.
Refactoring – Software evolves as time goes on.
Collective Ownership – Everyone on the team is responsible for the entire project.
Continuous Integration – New features are added constantly. No feature should take more than two weeks from design to deployment.
YAGNI - You Aint Gonna Need It! The art of not writing software is as important as writing software.
Wait – Make final decisions as late in the cycle as possible. I.e. a GUI and business logic can be designed and written independently, and the last step can be how they communicate with one another.

Conclusion
The position of this document is that in order for us to write better, more robust, more bug-free software, we must adopt these practices. All groups within the Product Development department (Business Analysis, Software Development, and Quality Assurance) must change at the same time in order for this to work efficiently. The majority of work must be done in projects, and these projects must have a team of dedicated people working towards the goal of project completion. The Product Managers, the Developers, and the Testers would need to all be working on the project at the same time in order to create the iterative releases that are so vital to these methodologies.

User Groups (and User Group Conferences)

2004-04-21T07:27:00.000-05:00

Each year, around this time, my company sponsors a User Group Conference in different parts of the country (this year, it’s in New Orleans). It is open to representatives from all of our customers, and we routinely get attendees from about 35% of our customers, including all of our top-10. Almost all of those in attendance are mid-level management or higher. My company also sends between 15 and 20 people, also ranging from the mid-level management level or higher.

The whole shebang serves a few purposes, namely 1) gets us in front of our customers; 2) gets our customers in front of each other; 3) allows our customers to gang up on us. I’ll now explain why all three of these work out well.

1) Gets us in front of our customers.
All three days of the conference are spent with my company showing our customers both our latest and greatest software, and expanded features of our existing software. Demonstrations are done and then the floor is open to discussion. There are four sessions per day, two in the morning and two in the afternoon. Keeping in mind that the audience is made up of the executives, and not the individual workers, we’re able to focus on the bells and whistles, while still being able to convey the meat of what the product accomplishes. The biggest benefit to this piece is that by and large, the customers see what they could be doing, if they upgraded. They also leave knowing that we’re busy little beavers, putting their software maintenance fees to good use.

2) Gets our customers in front of each other.
During the open-floor part of the aforementioned sessions, our customers not only shoot questions at us, but they end up helping each other by relaying how they use a particular product. Customer X wants to do something special with their member services department; Customer Y uses this product THIS way to accomplish the same thing. The biggest benefit to this kind of discussion is that now two customers are talking to each other to solve a problem, instead of the customer calling our support line, taking up our resources to solve the problem. We’ don’t worry about the lost revenue for our assistance since we’ve always got plenty of work.

3) Allows our customers to gang up on us.
The “Users Only Roundtable,” is on the last day of the conference. All of the customers, and NONE of my company’s employees attend this session. Part of this meeting is used to elect new officers for the UG, but the bulk of it is used as a gripe session. At least that’s what we assume, since we’re never in the room. The last thing that happens in the conference is that my company’s employees are called back into the room and presented a list of complaints that the group as a whole has, that they’d like addressed. One of the issues two years ago had to do with license fee increases. My company has historically increased our fees every two years. Our customers asked that we increase them each year, by half as much, so that it would make their budgeting job easier. Of course, some of the issues are not so easy to deal with, but they are always constructive.

Of course, all of this only works because of the quality of people that work for my company, and the quality of our customers. Everybody is trying to get the job done, so we’re all (us and the customer) always pulling in the same direction. There are always times when things get tense, but hey – that’s why we get paid the big bucks.

Make it work before you leave

2004-04-13T22:00:00.000-05:00

My new(est) mantra is “make it work before you leave.”

Programmer, Developer, Software Engineer, or whatever other name du jour I’m called, at the end of the day, I get paid to make a computer do what I tell it to do. Let’s focus on the “at the end of the day” piece of that statement for a minute. Also, set aside the fact that I’m a salaried employee, and would get paid the same even if I didn’t get a computer to do what I told it to do. For the longest time, as a programmer, my days would go something like this:

Loop until project “BigHuge” complete {
Turn on lights
Turn on workstation
Design and/or Code
Go home
}

For many years, I was very happy running this little life algorithm, collecting my paycheck, and going home to my family. At least I thought I was happy. Somewhere in the deep recesses of my soul there was something missing – especially on large projects, you know the kind that can span months. I realized, after a while, that the thrill of actually getting something done was missing. There is typically fanfare and a party at the end of a project, but us programmers have to have many highs throughout the project to keep us motivated. Without motivation, you get mediocrity. With mediocrity comes a poorer product, which leads to sagging sales, etceteras, and etceteras. I had to figure out a way to work on large projects and to keep myself out of the doldrums. To combat this battle fatigue I simply said to myself each and every day, “make something work before you leave for the day.”
Every day, before I leave work, I make sure I made at least one thing work. One thing that is getting me down the road to success for the whole project. One thing gets done at a time. One thing is done right, and then I move on. In affect, what I’ve done is made myself a better project manager. At the beginning of each project, I spend a couple of days writing the signatures for each of the classes and methods that I’ll be creating. Then, day by day, I work down the list. On some days, I have small victories, like being able to wrap the text in a label. On others, I have bigger victories, like passing XMLs back and forth with Chicago. I still get the big payoff at the end of the project when everything comes together, but I also get to celebrate my small successes along the way, and that’s the Big Picture Point to the story. After all, it’s the little things that keep us going.

Functional and Technical Specs really work

2004-03-08T09:51:00.000-05:00

A few months ago, I implemented a new way for Software to be developed at my company. The idea I pitched, and which was subsequently adopted, was to extend our existing functional specification process, and add a Technical Specification section. Not really a “eureka” in the annals of Software Development, but for a company that’s been neglecting this piece of the puzzle for 25+ years, it came to some as a shock. First, some history:

The Dark Ages:
In the very early days of my company, programming requests were submitted in a Programming Enhancement Request Form, or PERF. The PERF had exactly enough information in it to bill the customer properly. The worst part (for developers) was that it had a single text area for the Description – what was supposed to be done. My company mythology says that PERFs used to be kept on paper, but nobody really believes that.

Renaissance:
After the PERF, we started on something called the Product Design Document, or PDD. This is what most other companies in the world call Functional Specifications. This document accompanies any “large” project that we’re going to tackle. Its author is typically a member of our Customer Service department, and before it is submitted for a programming estimate, a member of our Product Development staff reviews it. Its various sections include:
Business Requirements – This section includes the “what and why” for the project. In other words, the purpose of the request as well as what benefits will be derived from the work.
Assumptions and Limitations – This lists our scope limits. For example if the request is for a new report, this section may note that the data that’s being reported is assumed to already be on the system.
Sample Outcomes – Sample of report, or they way a particular calculation should work

Just the implementation of the above saved my company hundreds of person-hours per year! It made everyone aware of what we were doing, and what was expected. Instead of just a description in a PERF, now the developer actually knew what was supposed to happen when his/her work was done! Our testing staff now had something to compare the program’s results to, rather that trying to test in a vacuum. Our Documentation Specialists had something to refer to for their process overviews.

For a while it seemed that we had found the utopia where all things worked as well as they could. Knowing that this could not be the case, I started studying what was happening once the PDDs landed on my desk.

As the supervisor, it’s my responsibility to assign the work to a developer. Once I was familiar with the PDD, I then needed to judge who on my staff was able to accomplish the task. I have programmers of several levels of knowledge and competency, as do most organizations, and finding an exact fit wasn’t always easy. I certainly didn’t want my most talented programmer working on something that someone else could do, nor did I want a relative newcomer to be tasked with something that would shatter their feeble-little fresh-out-of-college minds.

Once it was assigned to a programmer, I kept a watchful eye – remember I’m trying to make things better. I found that more often that not, the programmer would read over the PDD, and then schedule the author for a meeting to discuss the specifics. Not a problem – I’d rather them get it right the first time, rather than have to re-code. Then, almost 90% of the time, regardless of the programmer, they would schedule me to discuss how they should accomplish the task. Again, not a problem because I want them to get it right the first time. So I would set them on the path of righteousness, and be done with it.

Then something very interesting happened. A programmer came into my office and told me that we (meaning me) had made an error in the direction the programmer was taking. As it turns out, my directions put him square in the middle of Lebanon, Kansas, and they needed to be in New York City. The problem, of course, was that I and everybody else at my company were “Designing on the Fly.” Lots and lots of problems can occur by doing this, the biggest of which will become evident as I describe the new philosophy of our Specifications.

Enlightenment:
Last year, I pitched an idea to the rest of the Managers in my department, as well as the VP. This idea was to add Technical Specifications to our existing Functional Specifications. The two major sections that were added are a section for Design Overview, and Flowcharts.

The Design Overview section is now something that is completed by a high-level programmer BEFORE the request is even given an hour estimate. This is the 10,000 foot view of what needs to be changed or added. All programs, screens, tables, or parameters are listed here. The basic idea is that someone who has a good understanding of the process is deciding right there what will need to be changed. They are also in a position to give he best estimate since they know the process best.

The Flowcharts section is just that. It’s either a flowchart in the traditional sense, or a UML document, etc. Again, the senior resource for that software does all the design BEFORE it gets coded. He does this from start to finish, and here’s the best part. If he takes a path that turns out to be wrong, NO CODING HAS TO BE RECYCLED. All he has to do is change his specifications. We’ve all been in a situation where we’ve coded a huge method only to find out that we forgot one little piece of information that changes the whole design. Do we then trash the method and start over or do we work with what we have. Come on, let’s be honest, by this time, we’re either married to the code we’ve written, or we consider it one of our offspring. But it’s EASY to change a design document. We’ve got no emotional attachment to a flowchart – that’s crazy talk.

Once the technical design is complete, the project comes to me for assignment. The hope of the technical design is that once it’s done, any programmer, regardless of product knowledge, can accomplish the task.

Now life is worth living.

Resume – to give you background on me

2004-03-03T09:18:00.000-05:00

I am a Software Development Manager for a company that specializes in benefits administration for single and multi-employer benefit plans. I have been a developer for 13 years, and have managed a team for the last 5.

I was educated at Towson State University in Baltimore, Maryland, and double-majored in Applied Mathematics and Computer Science.

I was raised an Army Brat – meaning that until I went to college, I moved every three years of my life. I’ve lived in Okinawa, Japan, Kaiserslautern, Germany, and Alabama, Arizona, Georgia, Virginia, Washington, DC, and Maryland in the US.

I do all of my best work in Java, for which I am a SCJP. My company uses a non-1NF database, and with it a language called UniBasic, so I have expertise in these too, although I try to avoid working with this language.

I plan on using this blog to write about what happens in my department, and company from time to time. I will write about the outcomes of the various decisions that my colleagues and I make.

My hope for this little project is two fold: 1) to make me a better manager; and 2) to make me a better writer

I can always be emailed at mkclark9@yahoo.com. So that I can filter these emails out, please put “Your Blog” in the subject. The only thing I want to keep private is the company where I work – thus the yahoo address.