Technically, the new Intel Macs are the best thing that Apple has done in years. With Parallels, they are just fantastic: there are plenty of Windows apps I need to use, and this nicely solves that problem. I daresay that even Microsoft must be happy with this turn of events: they now get to move Windows license revenues on the small but growing Mac segment. At some point, I'd like to upgrade my wife's machine, though it probably won't be this year. But, why, oh, why is Apple running the worst advertising campaign in this century? I have no idea what on earth they were thinking: the Mac guy looks like more of a loser than the Windows guy. At least the Windows guy looks like he's got a job.
On the other hand, I may just be getting old.
Tuesday, June 27, 2006
Tuesday, June 20, 2006
iTunes flaw
As much as I appreciate what Apple has done with the iPod (by which I mean the complete package including the iPod itself, the iTunes client, and the iTunes Web service), there's a terrible limitation in the whole infrastructure. As a part of the fat client, I lose the ability to pass a URL around. And that just kills what I presume is a tremendous viral sales opportunity.
For example, imagine that I want to tell a friend over IM to buy Wynton Marsalis Septet's The Marciac Suite. Now I can provide you a URL to find out about it. Or I can provide you a URL to buy it from Amazon.com. But I can't send you a UI for iTunes. Not only that, if I google for Marciac Suite, I won't be directed to iTunes. The not-so-subtle point is that abandoning the Web paradigm, iTunes loses out on the scalability of the Web. Its actually an inconvenience for me to use iTunes these days, despite the fact that I like the service and I own two iPods.
Am I missing something here?
PS. the real point of this blog entry is that you should buy the Marciac Suite. I don't care where: it is sublime.
For example, imagine that I want to tell a friend over IM to buy Wynton Marsalis Septet's The Marciac Suite. Now I can provide you a URL to find out about it. Or I can provide you a URL to buy it from Amazon.com. But I can't send you a UI for iTunes. Not only that, if I google for Marciac Suite, I won't be directed to iTunes. The not-so-subtle point is that abandoning the Web paradigm, iTunes loses out on the scalability of the Web. Its actually an inconvenience for me to use iTunes these days, despite the fact that I like the service and I own two iPods.
Am I missing something here?
PS. the real point of this blog entry is that you should buy the Marciac Suite. I don't care where: it is sublime.
Thursday, June 15, 2006
The Best of Times, The Worst of Times
Way back in 2001, I worked with Russ Gold (founder/maintainer of HttpUnit and otherwise great guy) on the HP (formerly Bluestone) Application Server 8.0. A J2EE 1.3 server we collectively built in less than a year. That project was one of my best professional experiences and also a time where I got to know many other first-rate software folks, some of whom I continue to collaborate with to this day. Bruce Kratz is now at PrincetonSoftech, along with Al Smith, Pete Petersen, and Jay Hiremath. I worked very closely with Jon Maron and Mark Little and after we split from HP, we kept together to write a book. Jon and I still work together and Mark is at Red Hat. In looking for some info on agile programming that I need for a forthcoming blog entry, I stumbled on this article from Google cache profiling some of the work Russ and I did on HPAS. I had fun reading it and thought I'd save it from the bit bucket for a few more years...
[edit: after the initial post, I found an article Russ and I had written for the HP middleware newsletter. That article is included at the end of the post as well.]
XP's 12-practice methodology may play an important role for mobile developers struggling to maintain footing in a landscape evolving at warp speeds. It's already working For HP Middleware teams.
By Amy Cowen
Picture of Two Coders at a Computer
Agile or extreme?
While "extreme programming" remains popular in today's coding circles, a newer catchall phrase frequently appears on the scene - agile programming. In documenting their foray into XP, Gold and Pavlik use agile programming as an umbrella of sorts for a group of lightweight software development methodologies which includes XP.
"There was the recognition that there are a number of other methodologies which share XP's general approach of adapt rather than anticipate," explains Gold, addressing a growing shift from "extreme" to "agile" programming.
"Then too, at least part of the shift was probably Microsoft's fault," Gold continues. "By adopting the term XP for their new OS, they muddied the mindscape somewhat. Of course, now that Microsoft has adopted the slogan regarding agile businesses..."
Others attribute the shift from "extreme" to "agile" to be one of image. "Agile" conjures up speed and flexibility - but not risk - a change which could prove less intimidating for managers deciding whether or not to give an alternative lightweight methodology a spin.
Extreme Programming... Just the phrase raises images associated with sports from the X Games, sports that hover on the edge of mainstream legitimacy - dangerous, radical, and rule-breaking. Indeed, XP's name imbues it with a heady aura of defiance, post-modernity, and the spirit of daredevils and renegades.
Reviewing the tenets of XP, it becomes clear that XP is extreme in its level of difference from other traditional software methodologies. The irony of XP, however, is that it breaks the rules first and foremost by initiating a new set of rules.
And playing by the rules is key in XP.
This is not free-for-all programming where anything goes, where radical features are dreamed up and coded by pizza chugging hacksters who thought of a cool tweak while skateboarding through the office cubicles on a midnight break.
XP does encourage turning things up a notch. But it does so by offering a set of rules designed to ensure successful communication between team members, encourage a three musketeers mentality, rev up the production process, produce clearer and more straightforward and stable code, streamline costs, and, ultimately, enable teams to deliver solid products to happy clients.
For developers working in the ever-changing mobile application space, an "extreme" approach that churns out proudcts at light speeds compared to traditional development cycles might give developers an edge and put apps on the market in time to meet consumer demand - in time to nourish a growing user base and win critical marketshare.
XP on the scene
XP was developed and first documented by Kent Beck in his 1996 Extreme Programming Explained. Under Beck's guidance the first XP implementation happened on the now legendary - in XP circles - Chrysler Comprehensive Compensation - or C3 - development team.
Since its automotive beginnings, XP has infiltrated the ranks in all venues of software development, with success stories cropping up at companies large and small.
Foundations to build on
Beck bases the XP methodology on four core values - communication, simplicity, feedback, and courage. These values underwrite the 12 guiding XP practices (see sidebar below) and provide touchstones visible throughout the development cycle.
Traditional development often involves teams of coders holed up in individual cubbies, each working on pieces of a larger project. When finished, the pieces are assembled, and beta testing begins to see what works and what doesn't.
Assuming each coder has her own way of writing the code, her own way of approaching problems, and minimal communication between team members, it is easy to see how problems are likely to appear when the work of such individual contributors gets glued together.
There's not much holding the "whole" together.
For an XP team, however, the "whole" is being considered throughout.
The 12 XP Practices: The Planning Process, Small Releases, Metaphor, Simple Design, Testing, Refactoring, Pair Programming, Collective Ownership, Continuous Integration, 40-Hour Week, On-Site Customer, Coding Standard
From collective ownership of the code to the emphasis on the simplest design and a commitment to refactoring, the 12 practices that form the foundation of the XP methodology formalize the development process, bringing high levels of discipline, communication, and team spirit to an arena typically ruled by solo code jockeys.
HP goes XP
For Bruce Kratz, HP Middleware Lab Director, XP has found firm footing with his teams. Kratz chalks the HP Middleware team's adoption of XP methodologies up to rapidly changing needs.
"In our industry, requirements change every 90 - 120 days," explains Kratz. "XP allows us to release product in short intervals with a high level of confidence that it is a quality product."
The Middleware Division's first XP project was conducted by the HP Application Server (HPAS) team when working on Version 8 of the server. Following the successful use of XP for HPAS, both the Rich Media Technology team and the Mobile Infrastructure team are working with XP methodologies, says Kratz.
Greg Pavlik and Russell Gold, architects in the Middleware Division, worked on rebuilding EJB-related code for HP Application Server 8 and have documented their team's use of XP. While they didn't employ all 12 of XP's practices on the project, both found working with XP a positive and successful experience.
According to Pavlik, XP was the right approach for the project because "we needed a process that was highly resilient to change and evolution in the code base."
"[HP has] been in the application server business" since its acquisition of Bluestone, continues Pavlik. "But the code base today is completely different than what we started with, so the rate of change is phenomenal. Some of the software has been rewritten from scratch, but the EJB 1.1 server had been battle-hardened in some of the container logic. We wanted to preserve the foundations that we knew were solid, but didn't want them to become an albatross either. You have to have a process that supports constant and dynamic refactoring to move forward with new features and architecture changes without turning a project into spaghetti code."
The realities of ongoing and near-constant change are ones developers constantly battle.
XP helps offset the problems - and costs - associated with such change, making it a solid strategy for the HPAS team.
"You'll notice that the first XP book by Kent Beck was subtitled Embrace Change, and in one sense that's what XP is really about - flattening the cost of change away from an exponential growth curve," continues Pavlik.
Simple is as simple does
Simplicity may be a surprising tenet to see in XP discussions, but on an XP project, "simple" is the result of hard, calculated, and relentless work. Striving towards the simplest design, Beck says the XP coach has to ask, "what is the simplest thing that could possibly work?"
The simplest solution is often one that turns a blind eye to the future, and this is important in XP because XP discourages the integration of hooks for future expansion and development. Instead, on an XP project, the goal is to write a program that solves the client's immediate needs.
"Extreme Programming (XP) is a high-discipline, low-ceremony approach to development, meaning that it does not produce a lot of formal documentation or rely on a lot of formal reviews, but does insist that its practitioners be consistent in following the essential practices."
- Pavlik and Gold
"Simplicity is not easy," says Beck. "It is the hardest thing in the world to not look toward the things you'll need to implement tomorrow and next week and next month. But compulsively thinking ahead is listening to the fear of the exponential cost of change curve."
According to Beck, the "right" design is one that:
1. Runs all the tests.
2. Has no duplicated logic.
3. States every intention important to the programmers.
4. Has the fewest possible classes and methods.
Skeptics might view this focus on immediate needs - rather than on the "big picture" of the application over time - as a potentially costly approach since the code isn't necessarily extensible in ways that make it easily upgradable as needs evolve - a reality which could mean having to start over at the beginning when new requirements take center stage.
However, for those working with XP, the benefits to the XP focus on simplicity outweigh the risks.
"I feel over-engineering for the future, in anticipation of features that are not yet needed, winds up costing more," says Kratz.
Show and tell
Key to accelerating the development cycle and time-to-market is XP's focus on frequent releases and ongoing integration. Small 'nuts-and-bolts' releases happen throughout the development. The customer can see the product, evaluate the current feature set, and consider new features he wants to incorporate or add.
The scope of the project evolves and changes with each micro-release.
In part, the theory is that the penultimate product reflects what he wants rather than what he "thought" he wanted at the outset - leading to happier customers and ensuring the coding throughout the project was appropriately targeted at the desired features.
The lack of a fully-determined development specification - and the determination of features on an 'as we go' basis may sound shortsighted - and costly - to XP-skeptics.
Developers who have worked on projects in which the customer wasn't sophisticated enough to know up front what he needed, what was possible, and what would work best, may find the process of shedding reams of documentation, outlines, feature specs, and flowcharts intimidating if not downright foolish. For these developers - bitten by the clueless client in the past - XP may seem to set up a never-ending - rather than faster - project.
But for those who have gone the XP route, such fears seem misplaced.
A traditional development cycle can be so long - and so bogged down in paperwork - that by the end of the project the scope of work is no longer fully adequate to meet current client or market needs.
As Kratz notes, "No matter if we release frequently or [in] longer cycles, customer feedback may launch us in a new direction. By releasing frequently, we have the opportunity to get this feedback sooner and incorporate it into the next release."
"The Application Server Market product space is constantly evolving," Kratz adds. "So we expect to be in a constant development cycle anyway. XP allows us to keep up with technology and measure our success against customers and the market."
Back-seat driver
One of the most talked-about hallmarks of XP is the practice of pair programming. Programmers team up and work together when writing all code. One partner sits at the computer and does the typing. The other member of the pair watches everything that is typed in - looking for problems, mistakes, and potential pitfalls, and helping to talk through the code as it is being written.
While it may sound costly to devote two programmers to the same task, the XP theory is that working in a pair produces cleaner and better code, thus reducing time spent later debugging and cleaning up.
Working with someone else can be a big adjustment since programmers traditionally tend to work solo - disappearing for hours or even days to bang out some code to solve a certain problem.
"There's an important watchdog role to play when you're not the coder in the pair, but it's not what people are accustomed to. The temptation for me [when working solo] is always there to disappear for a few days and come back with a subsystem coded, missing tests, etc. Pairing can help add discipline."
- Greg Pavlik
Working in a pair means having someone else constantly watching and evaluating the code. The pairs are also constantly in communication. Some XP advocates go so far as to say that pairs should be talking at least every 45 seconds or so.
Despite the adjustment, pair programming seems to be a hit among the teams that have used it.
Having worked in an XP programming pair, Gold prefers the duo arrangement to working alone.
"I like working in a pair whenever possible," he says, "since I find it focuses me better and allows someone to catch me before I go down a path that in hindsight was foolish."
Pavlik is more noncommittal about whether he prefers to code solo or in a pair, but he readily admits that pair programming produces "much better code and ensures better practices." The caveat for Pavlik is that pair programming is slower than working alone, so not feasible for all projects.
For both Pavlik and Gold, identifying the hardest part of adjusting to pair programming is easy - "Not writing code all the time," says Pavlik.
Gold concurs, noting that the hardest aspect of working in a pair is "staying engaged when not doing the typing."
This is not necessarily surprising since for a programmer used to having free reign at the keyboard, sitting back and watching can make the fingers itch. Harnessing years of being able to tackle problems in one's own way and watch, instead, as another coder works can be difficult.
But careful watching - in addition to solid communication with the person in the driver's seat - makes pair programming work.
On the flip side, the person driving the code has to also adjust the way he works in order to remain in communication with the watcher.
"An experienced pair programmer helps his partner by thinking out loud while typing and explaining what he is doing at every point," explains Russell.
Clearly, pair programming dramatically alters how a developer approaches writing code regardless of whether or not the coder is sitting in front of the keyboard or watching from a hands-off position.
Pairs are dynamic, too, changing frequently as members of the team group and re-group to work together on aspects of the project.
Every pair will be different. Some pairs consist of two strong coders. Sometimes a pair will consist of a more experienced coder and a newer team member. Pairs might even be made up of designers and architects. Nevertheless, XP theorists maintain there are strengths in all kinds of teams - even a team of two new programmers can produce stronger code than if they were working independently.
Speaking from a managerial perspective, Kratz says he has been surprised at how well pair programming has worked within the Middleware teams.
"It's probably not for everyone," he acknowledges, "but from my seat, things are thought out better, younger team members are becoming stronger quicker, and things are getting done. Hey, they may even be having fun!"
Goodbye paper trail
An interesting offshoot of XP is a reduction in documentation requirements. Team members are in constant communication with each other - and in frequent communication with the customer - so there is no reason to spend valuable time producing documentation when there is coding that can be done instead.
In XP, the focus is on producing code - not producing paper that talks about the code that will someday be produced. The time saved leaves coders fresh to tackle the project and works with the other XP practices to push the project towards early completion.
For those with either a packrat or a cover-our-tail mentality, putting aside the documentation process can feel both risky and maybe even leave the process feeling naked.
But XP veterans have found the reduced documentation to be a liberating aspect of the process.
"Too many design docs mean death for a project," says Pavlik. "When people talk about project failure and 'analysis paralysis,' they're talking about things like a giant static model, reams of documentation, and no working code or functionality."
With a smile, Russell agrees that it wasn't hard to ditch documentation.
"In my experience, most development projects don't actually produce a lot of usable documentation during development anyway," he notes. "Most design documentation built at the start of the process winds up obsolete by the end and is not maintained."
XP recognizes this and emphasizes writing code that is clear enough to serve as its own documentation. XP advocates add that the increased emphasis on person-to-person communication helps replace the need for a paper trail.
"The biggest hurdle is teaching developers to refactor working code," says Russell.
Refactor, refactor, refactor
A central XP practice is relentless refactoring. From the outside, the process of refactoring can seem a bit vague - coders review their code to ensure that they've taken the most straightforward, most streamlined, and simplest approach. All excess is trimmed or tweaked, keeping the code facile, lean, and clean.
In their own account of their experience using XP, Gold and Pavlik write:
"It is very common to develop software which works and fulfills all existing requirements - but which cannot be easily extended to handle new requirements. The most common reaction to encountering such a situation is to sigh, roll up one's sleeves, and rewrite the offending code, taking into consideration the new needs. But doing so discards all of the work that went into debugging the original code."
XP's emphasis on refactoring counters this approach, leaving code in such a streamlined state "that new features can be added easily without changing the behavior of the working code at all," explain Gold and Pavlik.
According to Pavlik, "Refactoring is really the core practice."
Explaining the important role refactoring plays, Gold says, "The essence is to recognize that there are a large number of well understood incremental transforms which you can make to software to improve its organization and design. By applying them, one after another, and verifying that you have not made a mistake by rerunning your unit tests, you can safely restructure your software to minimize coupling, increase cohesion, shrink your methods until they can be easily understood, and give your methods, classes, and variables names which clearly convey their meaning."
Boot camp for extreme developers
Join other extreme coders in the mobile space in Helsinki, Finland, April 9-11 for 60 hours of non-stop coding and development. As you squeeze nearly 2 weeks of work into a few days, you can enable your mobile e-service app with hp mobile technologies.
Sleeping bags, saunas, and plenty of food will be available. So bring your app, your imagination and mobile services vision, and get extreme!
Visit the HP Bazaar Site for more information or to register.
"The most important aspect is to change the system without affecting the behavior," adds Pavlik. "If you are changing the behavior and the structure of system elements, you are not refactoring."
Producing leaner, more malleable code, refactoring addresses what otherwise might prove a shortcoming in the XP approach - XP's refusal to code for the future. By insisting the final code is solid, pared down so that only the mandatory essence remains, XP leaves code in a state where future work can begin with minimal rebuilding.
Indeed, refactoring "is the mechanism by which you are able to affect the cost of change curve," says Pavlik.
(For more information on refactoring, both Gold and Pavlik encourage looking at Refactoring by Martin Fowler. "It is one of those books which every serious developer should own," says Gold.)
"Testing, testing"
Another important XP practice is the use of "tests" as a QA tool. This may not sound unusual. But in XP, the tests are written prior to the commencement of writing the code. The tests are then deployed throughout to ensure quality control and to highlight bugs that might crop up.
While at first glance it seems convoluted to create tests before what's even being built has been concretely defined - keeping in mind that XP takes an "add as you go" approach to determining the scope, extreme coders find the process of creating the tests clarifying.
According to Pavlik and Gold, "testing was one of our clear wins on this project. We got into the habit of writing unit tests for the software, using a combination of ant, JUnit, and HttpUnit to run the tests in suites. We adopted a rule that nobody could commit changes to the baseline unless all tests passed. As a result, the EJB container code was almost never broken by something one the of the EJB developers did, and we had a high degree of confidence that something that worked at one time continued to work through the 8.0 release."
"Creating the tests first help you understand what you are attempting to build," Kratz adds.
A winning combination
For HP's Middleware Division, XP's 12 practices have already added up to measurable success.
However, while XP's tenets have held up well so far, Kratz remains realistic, admitting that XP isn't always the right approach.
"Smart teams will know when to follow the XP approach and when to resort to traditional development methodologies. My thought is that 'one size does not fit all.'"
Gold, too, remains pragmatic.
"XP is not magic. It is a set of 12 well-defined practices which have been found to work together well."
And the second article....
Experiences with Agile Programming Models on HP-AS 8.0
Background and Definitions
Among the constant challenges in the development of software is the problem of how to handle change, especially in system requirements. Most traditional models of development have noted that it is much cheaper to fix a problem during requirements or design than to correct it once the code has already been written. As a result, these models have tended to focus on trying to be as thorough as possible in making sure that the requirements are complete and correct, and that the design addresses all major considerations which might arise before moving on to implementation.
Unfortunately, in most systems of interest to J2EE developers the problems simply cannot be known completely until the system is built and tried. And of course, once this has happened, changing the system is difficult, costly, and error-prone.
An alternative approach for dealing with change is to build a system around what is known for certain initially, and to make the system adaptable throughout the project lifecycle using techniques that reduce the costs and risks associated with modifying working code. The use of such techniques is known as Agile Programming.
What is Agile Programming?
In the last three years or so there has been a lot of discussion among software vendors about agile software development processes. In particular, there is a lot of excitement associated with the practices that Kent Beck popularized in his book Extreme Programming Explained: Embrace Change. Other "agile" processes that are gaining adherents in the developer community include systems of practices described in SCRUM, Feature Driven Software Development, and Pragmatic Programming. Each of these systems shares some common themes: close collaboration between developers and customers, recognition that changing requirements are a reality of software, and that development practices must play to the strengths of how developers work best.
We chose to use practices from Extreme Programming, probably the best known of the Agile Programming processes, to develop HP Application Server EJB container.
What is Extreme Programming?
Extreme Programming (XP) is a high-discipline, low-ceremony approach to development, meaning that it does not produce a lot of formal documentation or rely on a lot of formal reviews, but does insist that its practitioners be consistent in following the essential practices, including:
Onsite Customer The customer makes all functionality decisions and is always available
The Planning Game Customers get detailed continuous control over what work is done
System Metaphor The overall design is described in terms of shared concepts
Simple Design The design addresses current needs, not possible future additions
Collective Ownership All code is the responsibility of multiple developers
Forty-hour Week Developers may not work overtime two weeks in a row
Test-Driven Design Functionality is validated via automated tests written before coding
Pair Programming Code is written in pairs, thus giving continuous reviews
Continuous Integration Code is pushed to the baseline on a daily basis
Refactor Mercilessly Poorly structured code is cleaned up aggressively
Small Releases Customers are given updated working code to use frequently
Coding Conventions The developers agree on a common coding style and follow it
These practices have been shown to work together to produce consistent, reliable and flexible software without running a development team into the ground. Of course, XP is deeper than just a set of practices. It declares that programming should be a humane discipline that doesn't drive talent out of the trade.
And of course, as part of its "embrace change" attitude, XP insists that each team constantly analyze those practices it is following and change them when appropriate. It is more important to emphasis the values of listening, testing, coding, and designing than to adhere to someone else's checklist or rules.
EJB Development
Both of us worked on the EJB container in the latest release of HP Application Server. The project was a major undertaking. Within a six month time frame, the team had the responsibility of rearchitecting and rewriting most of the EJB-related code base. We needed to fit the container into a service framework on which the application server was being built, interact with other newly defined services, support a number of new features like hot redeployment and pluggable transport protocols (ensuring that all J2EE Compatibility Test Suite tests ran at 100%), and bring the container in line with the EJB 2.0 specification, public final draft two.
While we had experimented with XP techniques like automated unit testing in the past, we wanted to learn more about XP and what it had to offer through experience. Another great benefit was that Russ had joined the team at this point, and brought with him experience with XP. We'll try to summarize some of the what we did do, some of what we didn't do, and look at what the implications were for our project.
XP Programming Techniques We Used
Frequent Testing
Testing was one of our clear wins on this project. We got into the habit of writing unit tests for the software, using a combination of ant, JUnit, and HttpUnit to run the tests in suites. We adopted a rule that nobody could commit changes to the baseline unless all tests passed. As a result, the EJB container code was almost never broken by something on the of the EJB developers did, and we had a high degree of confidence that something that worked at one time continued to work through the 8.0 release.
One of the most useful rules the team developed was that bug reports should be accompanied by automated JUnit tests. The test code showed the problem in a repeatable way and the tests were added into the developer suite to ensure that regression didn't occur.
Continuous Integration
We encouraged our developers to check in their code every day, while keeping the baseline working (all regression tests running at 100%). This was tricky since we automatically absorbed all changes from other groups working on other portions of the application server every day as well. At times the upstream tests would miss something that would affect us further downstream. Since we couldn't always wait for the upstream changes to be fixed, this caused us to stumble a few times when we checked in code that broke the same tests. But for the most part, we were able to keep the baseline clean throughout the cycle, and this really helped us maintain the quality of the software and protect against regression even during aggressive overhauls of the container.
Another important benefit we saw from frequent integration was that it helped everyone understand how both refactoring and new development were effecting the evolution of the system. Since change was incremental, it was comprehensible. If we had all waited until some integration phase to bring pieces together within the system, everything that a developer wasn't actively working on would have become like a black box.
Refactor Mercilessly
It is very common to develop software which works and fulfills all existing requirements - but which cannot be easily extended to handle new requirements. The most common reaction to encountering such a situation is to sigh, roll up one's sleeves, and rewrite the offending code, taking into consideration the new needs. But doing so discards all of the work that went into debugging the original code; it's also time consuming, and often leads to code no better structured than the version which was replaced. Refactoring addresses this, providing a way to restructure the original code so that new features can be added easily without changing the behavior of the working code at all. This is one of the most important practices of XP; in some ways, it is the essence of XP. There are two things about refactoring that are critical to understand. First of all, it doesn't work without tests, period. If you don't have the tests, you will almost certainly break the software. Secondly, refactoring can be divided into small and well understood steps, catalogued in such places as Martin Fowler's Refactoring: Improving the Design of Existing Code and on his website, http://www.refactoring.com. It is also greatly simplified by support in automated tools -- something that's now creeping into Java IDEs.
There were a number of times during the development of the EJB container when we started with code left over from the Total-e-Server implementation, and refactored it to allow the addition of new features. We didn't do a lot of it, but what we did was invaluable. For example, we were able to refactor a legacy class from over 2500 lines to 300 lines, gradually, over the course of the release. This made it possible for us to more readily support hot redeployment of EJBs within the container.
Simple Design
Trusting in our ability to refactor as needed, we tried hard to avoid common expensive practices such as "adding hooks" for future growth. This allowed us to avoid unnecessary code, and we did not see any place where it hurt us. It also made the code much easier to understand.
User Stories
XP also promotes the creation of user stories, which are lightweight use cases for pieces of functionality. User stories were something we used consistently, but also something we struggled with doing correctly. The user stories we developed were very much like functional requirements for the system. As a result many of the stories were driven by developers rather than the customer proxy. We did, however, use these as a low-investment strategy for planning. Our group manager played the role of a tracker and tried to make sure the stories within a cycle were consistent with the velocity measurements he had taken for past cycles. This was a great benefit for tracking and planning with agility.
XP Programming Techniques We Didn't Use Consistently
Collective Ownership
With XP, assignments are distributed so that developers don't specialize. In practice, we assigned developers to specific functional areas and for the most part did not overlap, fearing that without specialization we would not be able to develop the necessary expertise in each area. The downside, of course, is that there are many parts of the code known by only one developer and never reviewed or even looked at by the others. It is too soon to tell how this will affect future maintenance.
Pair Programming
Pair programming is a concept that seems alien and we've found that this is an idea that people find hard to digest. For one thing, it's simply very different from what we're used to. It also seems counter-intuitive that a process designed around rapid software deliveries would tie two people to a computer. Turns out we were wrong.
We found that when we paired on user stories, different areas of experience of different programmers really combined to give the pair a deeper understanding of the problem space. As a general rule, pairing led to better code. We extended this outside of our project and attempted to always pair when we worked with other teams interfacing to our container. This helped both sides to get a better understanding of the critical glue code between subsystems.
Another critical point about pairing: it encourages discipline. If one person is behind and tempted to cheat on test coverage, for example, an individual can get away with it. Another benefit of pairing is that it tends to keep people focused. You can't waste time on nonproductive web surfing when someone is working with you.
When push came to shove, we needed to deliver and we fell back on what on the more conventional techniques that we knew. We had to rely on code reviews that were often stressful and grinding to audit each other, and the there were times when class design suffered. Pair programming may seem unnatural, but it has some real benefits that make it worth considering as a basic practice for your team.
Test-Driven Design
For the most part, the team didn't do "test first" programming. By the time we started seriously writing automated tests, most of the code was already in place, and needed updating rather than writing from scratch. In addition, despite reading the XP literature, we did not realize just how powerful this technique can be. Some of us did have the opportunity to write functionality that was brand-new for EJB 2.0, and we did use test-first in those parts. The result seemed to be fewer false starts and simpler designs than in areas built without such tests.
Architecture vs. Metaphor
XP puts forward the notion of a metaphor for a system as a substitute for architecture. We didn't attempt this: we spent time working out a macro-architecture, talking it through face-to-face, and adjusting it as necessary as we progressed. The metaphor idea didn't seem to make sense for something like a framework and the approach we took seemed to work well. So this was an aspect of XP that we didn't utilize, but also didn't feel that we were missing anything.
Limits of Tests
It also became quickly clear that there are limits to what the unit tests provide. Tests aren't a substitute for really digging into the system. Constant and liberal changes can result in subtle changes of untested assumptions. It's easy to say that the problem is there aren't enough tests which, while true, isn't an answer. After all, there will never be enough tests to check every assumption about the system. Software is complicated, and middleware particularly so: it combines the challenges of concurrent, distributed systems programming with a need for an extensive understanding of low-level concepts in security, distributed transaction processing, high availability system engineering, etc.
Performance
One of the questions we had was how XP would effect performance. Refactoring typically leads to more indirection and we specifically avoided writing "optimized" code. Instead, we waited until after all features were complete to systematically profile the system under load with real end-user applications. This paid off hugely: the container was considerably faster than previous iterations and the unit tests helped to ensure that the optimizations we introduced did not break the system.
The most important bottleneck in middleware for distributed systems is resource contention, and we were able to isolate and eliminate the contention issues we identified. The result was not only a faster system, but also a more scaleable system. We had great success with this approach, but would hesitate to recommend it without unit tests in place.
Conclusion
Over the course of the release, there were times when even we doubted our ability to deliver on all of the requirements in the time required. It's our strong conclusion that using the techniques from XP that we used were fundamental to pulling the software together into a well-received, sophisticated application server. We found that the basic practices combine and play off each other to help make better software and that we missed out on some key benefits from practices we weren't able to implement. We've worked on projects where the code regresses over time and we're happy to report that the code base for the EJB container took a different path: it got better and better as the release progressed. There are unique challenges to every project: some teams are very large, some are very big, some distributed geographically.
We would encourage you to experiment with XP and other Agile Programming techniques and adopt them to your environment. They pay off in a big way: well written software that meets real requirements in short time frames.
More information regarding Extreme Programming is available at:
http://www.extremeprogramming.com;
and for more information regarding Agile Programming, check out:
http://www.agilealliance.org.
ABOUT THE AUTHORS:
This article was contributed by Russell Gold and Greg Pavlik architects with Hewlett-Packard's Middleware Division. Russell has been instrumental in applying the eXtreme programming methodologies to the Middleware development process. Russell is also the original author and maintainer of HttpUnit, an open source library for automating tests of web sites. Greg was the lead architect on the HP Application Server's EJB 2.0 and 1.1 implementations and has been a member of the EJB expert group.
[edit: after the initial post, I found an article Russ and I had written for the HP middleware newsletter. That article is included at the end of the post as well.]
XP's 12-practice methodology may play an important role for mobile developers struggling to maintain footing in a landscape evolving at warp speeds. It's already working For HP Middleware teams.
By Amy Cowen
Picture of Two Coders at a Computer
Agile or extreme?
While "extreme programming" remains popular in today's coding circles, a newer catchall phrase frequently appears on the scene - agile programming. In documenting their foray into XP, Gold and Pavlik use agile programming as an umbrella of sorts for a group of lightweight software development methodologies which includes XP.
"There was the recognition that there are a number of other methodologies which share XP's general approach of adapt rather than anticipate," explains Gold, addressing a growing shift from "extreme" to "agile" programming.
"Then too, at least part of the shift was probably Microsoft's fault," Gold continues. "By adopting the term XP for their new OS, they muddied the mindscape somewhat. Of course, now that Microsoft has adopted the slogan regarding agile businesses..."
Others attribute the shift from "extreme" to "agile" to be one of image. "Agile" conjures up speed and flexibility - but not risk - a change which could prove less intimidating for managers deciding whether or not to give an alternative lightweight methodology a spin.
Extreme Programming... Just the phrase raises images associated with sports from the X Games, sports that hover on the edge of mainstream legitimacy - dangerous, radical, and rule-breaking. Indeed, XP's name imbues it with a heady aura of defiance, post-modernity, and the spirit of daredevils and renegades.
Reviewing the tenets of XP, it becomes clear that XP is extreme in its level of difference from other traditional software methodologies. The irony of XP, however, is that it breaks the rules first and foremost by initiating a new set of rules.
And playing by the rules is key in XP.
This is not free-for-all programming where anything goes, where radical features are dreamed up and coded by pizza chugging hacksters who thought of a cool tweak while skateboarding through the office cubicles on a midnight break.
XP does encourage turning things up a notch. But it does so by offering a set of rules designed to ensure successful communication between team members, encourage a three musketeers mentality, rev up the production process, produce clearer and more straightforward and stable code, streamline costs, and, ultimately, enable teams to deliver solid products to happy clients.
For developers working in the ever-changing mobile application space, an "extreme" approach that churns out proudcts at light speeds compared to traditional development cycles might give developers an edge and put apps on the market in time to meet consumer demand - in time to nourish a growing user base and win critical marketshare.
XP on the scene
XP was developed and first documented by Kent Beck in his 1996 Extreme Programming Explained. Under Beck's guidance the first XP implementation happened on the now legendary - in XP circles - Chrysler Comprehensive Compensation - or C3 - development team.
Since its automotive beginnings, XP has infiltrated the ranks in all venues of software development, with success stories cropping up at companies large and small.
Foundations to build on
Beck bases the XP methodology on four core values - communication, simplicity, feedback, and courage. These values underwrite the 12 guiding XP practices (see sidebar below) and provide touchstones visible throughout the development cycle.
Traditional development often involves teams of coders holed up in individual cubbies, each working on pieces of a larger project. When finished, the pieces are assembled, and beta testing begins to see what works and what doesn't.
Assuming each coder has her own way of writing the code, her own way of approaching problems, and minimal communication between team members, it is easy to see how problems are likely to appear when the work of such individual contributors gets glued together.
There's not much holding the "whole" together.
For an XP team, however, the "whole" is being considered throughout.
The 12 XP Practices: The Planning Process, Small Releases, Metaphor, Simple Design, Testing, Refactoring, Pair Programming, Collective Ownership, Continuous Integration, 40-Hour Week, On-Site Customer, Coding Standard
From collective ownership of the code to the emphasis on the simplest design and a commitment to refactoring, the 12 practices that form the foundation of the XP methodology formalize the development process, bringing high levels of discipline, communication, and team spirit to an arena typically ruled by solo code jockeys.
HP goes XP
For Bruce Kratz, HP Middleware Lab Director, XP has found firm footing with his teams. Kratz chalks the HP Middleware team's adoption of XP methodologies up to rapidly changing needs.
"In our industry, requirements change every 90 - 120 days," explains Kratz. "XP allows us to release product in short intervals with a high level of confidence that it is a quality product."
The Middleware Division's first XP project was conducted by the HP Application Server (HPAS) team when working on Version 8 of the server. Following the successful use of XP for HPAS, both the Rich Media Technology team and the Mobile Infrastructure team are working with XP methodologies, says Kratz.
Greg Pavlik and Russell Gold, architects in the Middleware Division, worked on rebuilding EJB-related code for HP Application Server 8 and have documented their team's use of XP. While they didn't employ all 12 of XP's practices on the project, both found working with XP a positive and successful experience.
According to Pavlik, XP was the right approach for the project because "we needed a process that was highly resilient to change and evolution in the code base."
"[HP has] been in the application server business" since its acquisition of Bluestone, continues Pavlik. "But the code base today is completely different than what we started with, so the rate of change is phenomenal. Some of the software has been rewritten from scratch, but the EJB 1.1 server had been battle-hardened in some of the container logic. We wanted to preserve the foundations that we knew were solid, but didn't want them to become an albatross either. You have to have a process that supports constant and dynamic refactoring to move forward with new features and architecture changes without turning a project into spaghetti code."
The realities of ongoing and near-constant change are ones developers constantly battle.
XP helps offset the problems - and costs - associated with such change, making it a solid strategy for the HPAS team.
"You'll notice that the first XP book by Kent Beck was subtitled Embrace Change, and in one sense that's what XP is really about - flattening the cost of change away from an exponential growth curve," continues Pavlik.
Simple is as simple does
Simplicity may be a surprising tenet to see in XP discussions, but on an XP project, "simple" is the result of hard, calculated, and relentless work. Striving towards the simplest design, Beck says the XP coach has to ask, "what is the simplest thing that could possibly work?"
The simplest solution is often one that turns a blind eye to the future, and this is important in XP because XP discourages the integration of hooks for future expansion and development. Instead, on an XP project, the goal is to write a program that solves the client's immediate needs.
"Extreme Programming (XP) is a high-discipline, low-ceremony approach to development, meaning that it does not produce a lot of formal documentation or rely on a lot of formal reviews, but does insist that its practitioners be consistent in following the essential practices."
- Pavlik and Gold
"Simplicity is not easy," says Beck. "It is the hardest thing in the world to not look toward the things you'll need to implement tomorrow and next week and next month. But compulsively thinking ahead is listening to the fear of the exponential cost of change curve."
According to Beck, the "right" design is one that:
1. Runs all the tests.
2. Has no duplicated logic.
3. States every intention important to the programmers.
4. Has the fewest possible classes and methods.
Skeptics might view this focus on immediate needs - rather than on the "big picture" of the application over time - as a potentially costly approach since the code isn't necessarily extensible in ways that make it easily upgradable as needs evolve - a reality which could mean having to start over at the beginning when new requirements take center stage.
However, for those working with XP, the benefits to the XP focus on simplicity outweigh the risks.
"I feel over-engineering for the future, in anticipation of features that are not yet needed, winds up costing more," says Kratz.
Show and tell
Key to accelerating the development cycle and time-to-market is XP's focus on frequent releases and ongoing integration. Small 'nuts-and-bolts' releases happen throughout the development. The customer can see the product, evaluate the current feature set, and consider new features he wants to incorporate or add.
The scope of the project evolves and changes with each micro-release.
In part, the theory is that the penultimate product reflects what he wants rather than what he "thought" he wanted at the outset - leading to happier customers and ensuring the coding throughout the project was appropriately targeted at the desired features.
The lack of a fully-determined development specification - and the determination of features on an 'as we go' basis may sound shortsighted - and costly - to XP-skeptics.
Developers who have worked on projects in which the customer wasn't sophisticated enough to know up front what he needed, what was possible, and what would work best, may find the process of shedding reams of documentation, outlines, feature specs, and flowcharts intimidating if not downright foolish. For these developers - bitten by the clueless client in the past - XP may seem to set up a never-ending - rather than faster - project.
But for those who have gone the XP route, such fears seem misplaced.
A traditional development cycle can be so long - and so bogged down in paperwork - that by the end of the project the scope of work is no longer fully adequate to meet current client or market needs.
As Kratz notes, "No matter if we release frequently or [in] longer cycles, customer feedback may launch us in a new direction. By releasing frequently, we have the opportunity to get this feedback sooner and incorporate it into the next release."
"The Application Server Market product space is constantly evolving," Kratz adds. "So we expect to be in a constant development cycle anyway. XP allows us to keep up with technology and measure our success against customers and the market."
Back-seat driver
One of the most talked-about hallmarks of XP is the practice of pair programming. Programmers team up and work together when writing all code. One partner sits at the computer and does the typing. The other member of the pair watches everything that is typed in - looking for problems, mistakes, and potential pitfalls, and helping to talk through the code as it is being written.
While it may sound costly to devote two programmers to the same task, the XP theory is that working in a pair produces cleaner and better code, thus reducing time spent later debugging and cleaning up.
Working with someone else can be a big adjustment since programmers traditionally tend to work solo - disappearing for hours or even days to bang out some code to solve a certain problem.
"There's an important watchdog role to play when you're not the coder in the pair, but it's not what people are accustomed to. The temptation for me [when working solo] is always there to disappear for a few days and come back with a subsystem coded, missing tests, etc. Pairing can help add discipline."
- Greg Pavlik
Working in a pair means having someone else constantly watching and evaluating the code. The pairs are also constantly in communication. Some XP advocates go so far as to say that pairs should be talking at least every 45 seconds or so.
Despite the adjustment, pair programming seems to be a hit among the teams that have used it.
Having worked in an XP programming pair, Gold prefers the duo arrangement to working alone.
"I like working in a pair whenever possible," he says, "since I find it focuses me better and allows someone to catch me before I go down a path that in hindsight was foolish."
Pavlik is more noncommittal about whether he prefers to code solo or in a pair, but he readily admits that pair programming produces "much better code and ensures better practices." The caveat for Pavlik is that pair programming is slower than working alone, so not feasible for all projects.
For both Pavlik and Gold, identifying the hardest part of adjusting to pair programming is easy - "Not writing code all the time," says Pavlik.
Gold concurs, noting that the hardest aspect of working in a pair is "staying engaged when not doing the typing."
This is not necessarily surprising since for a programmer used to having free reign at the keyboard, sitting back and watching can make the fingers itch. Harnessing years of being able to tackle problems in one's own way and watch, instead, as another coder works can be difficult.
But careful watching - in addition to solid communication with the person in the driver's seat - makes pair programming work.
On the flip side, the person driving the code has to also adjust the way he works in order to remain in communication with the watcher.
"An experienced pair programmer helps his partner by thinking out loud while typing and explaining what he is doing at every point," explains Russell.
Clearly, pair programming dramatically alters how a developer approaches writing code regardless of whether or not the coder is sitting in front of the keyboard or watching from a hands-off position.
Pairs are dynamic, too, changing frequently as members of the team group and re-group to work together on aspects of the project.
Every pair will be different. Some pairs consist of two strong coders. Sometimes a pair will consist of a more experienced coder and a newer team member. Pairs might even be made up of designers and architects. Nevertheless, XP theorists maintain there are strengths in all kinds of teams - even a team of two new programmers can produce stronger code than if they were working independently.
Speaking from a managerial perspective, Kratz says he has been surprised at how well pair programming has worked within the Middleware teams.
"It's probably not for everyone," he acknowledges, "but from my seat, things are thought out better, younger team members are becoming stronger quicker, and things are getting done. Hey, they may even be having fun!"
Goodbye paper trail
An interesting offshoot of XP is a reduction in documentation requirements. Team members are in constant communication with each other - and in frequent communication with the customer - so there is no reason to spend valuable time producing documentation when there is coding that can be done instead.
In XP, the focus is on producing code - not producing paper that talks about the code that will someday be produced. The time saved leaves coders fresh to tackle the project and works with the other XP practices to push the project towards early completion.
For those with either a packrat or a cover-our-tail mentality, putting aside the documentation process can feel both risky and maybe even leave the process feeling naked.
But XP veterans have found the reduced documentation to be a liberating aspect of the process.
"Too many design docs mean death for a project," says Pavlik. "When people talk about project failure and 'analysis paralysis,' they're talking about things like a giant static model, reams of documentation, and no working code or functionality."
With a smile, Russell agrees that it wasn't hard to ditch documentation.
"In my experience, most development projects don't actually produce a lot of usable documentation during development anyway," he notes. "Most design documentation built at the start of the process winds up obsolete by the end and is not maintained."
XP recognizes this and emphasizes writing code that is clear enough to serve as its own documentation. XP advocates add that the increased emphasis on person-to-person communication helps replace the need for a paper trail.
"The biggest hurdle is teaching developers to refactor working code," says Russell.
Refactor, refactor, refactor
A central XP practice is relentless refactoring. From the outside, the process of refactoring can seem a bit vague - coders review their code to ensure that they've taken the most straightforward, most streamlined, and simplest approach. All excess is trimmed or tweaked, keeping the code facile, lean, and clean.
In their own account of their experience using XP, Gold and Pavlik write:
"It is very common to develop software which works and fulfills all existing requirements - but which cannot be easily extended to handle new requirements. The most common reaction to encountering such a situation is to sigh, roll up one's sleeves, and rewrite the offending code, taking into consideration the new needs. But doing so discards all of the work that went into debugging the original code."
XP's emphasis on refactoring counters this approach, leaving code in such a streamlined state "that new features can be added easily without changing the behavior of the working code at all," explain Gold and Pavlik.
According to Pavlik, "Refactoring is really the core practice."
Explaining the important role refactoring plays, Gold says, "The essence is to recognize that there are a large number of well understood incremental transforms which you can make to software to improve its organization and design. By applying them, one after another, and verifying that you have not made a mistake by rerunning your unit tests, you can safely restructure your software to minimize coupling, increase cohesion, shrink your methods until they can be easily understood, and give your methods, classes, and variables names which clearly convey their meaning."
Boot camp for extreme developers
Join other extreme coders in the mobile space in Helsinki, Finland, April 9-11 for 60 hours of non-stop coding and development. As you squeeze nearly 2 weeks of work into a few days, you can enable your mobile e-service app with hp mobile technologies.
Sleeping bags, saunas, and plenty of food will be available. So bring your app, your imagination and mobile services vision, and get extreme!
Visit the HP Bazaar Site for more information or to register.
"The most important aspect is to change the system without affecting the behavior," adds Pavlik. "If you are changing the behavior and the structure of system elements, you are not refactoring."
Producing leaner, more malleable code, refactoring addresses what otherwise might prove a shortcoming in the XP approach - XP's refusal to code for the future. By insisting the final code is solid, pared down so that only the mandatory essence remains, XP leaves code in a state where future work can begin with minimal rebuilding.
Indeed, refactoring "is the mechanism by which you are able to affect the cost of change curve," says Pavlik.
(For more information on refactoring, both Gold and Pavlik encourage looking at Refactoring by Martin Fowler. "It is one of those books which every serious developer should own," says Gold.)
"Testing, testing"
Another important XP practice is the use of "tests" as a QA tool. This may not sound unusual. But in XP, the tests are written prior to the commencement of writing the code. The tests are then deployed throughout to ensure quality control and to highlight bugs that might crop up.
While at first glance it seems convoluted to create tests before what's even being built has been concretely defined - keeping in mind that XP takes an "add as you go" approach to determining the scope, extreme coders find the process of creating the tests clarifying.
According to Pavlik and Gold, "testing was one of our clear wins on this project. We got into the habit of writing unit tests for the software, using a combination of ant, JUnit, and HttpUnit to run the tests in suites. We adopted a rule that nobody could commit changes to the baseline unless all tests passed. As a result, the EJB container code was almost never broken by something one the of the EJB developers did, and we had a high degree of confidence that something that worked at one time continued to work through the 8.0 release."
"Creating the tests first help you understand what you are attempting to build," Kratz adds.
A winning combination
For HP's Middleware Division, XP's 12 practices have already added up to measurable success.
However, while XP's tenets have held up well so far, Kratz remains realistic, admitting that XP isn't always the right approach.
"Smart teams will know when to follow the XP approach and when to resort to traditional development methodologies. My thought is that 'one size does not fit all.'"
Gold, too, remains pragmatic.
"XP is not magic. It is a set of 12 well-defined practices which have been found to work together well."
And the second article....
Experiences with Agile Programming Models on HP-AS 8.0
Background and Definitions
Among the constant challenges in the development of software is the problem of how to handle change, especially in system requirements. Most traditional models of development have noted that it is much cheaper to fix a problem during requirements or design than to correct it once the code has already been written. As a result, these models have tended to focus on trying to be as thorough as possible in making sure that the requirements are complete and correct, and that the design addresses all major considerations which might arise before moving on to implementation.
Unfortunately, in most systems of interest to J2EE developers the problems simply cannot be known completely until the system is built and tried. And of course, once this has happened, changing the system is difficult, costly, and error-prone.
An alternative approach for dealing with change is to build a system around what is known for certain initially, and to make the system adaptable throughout the project lifecycle using techniques that reduce the costs and risks associated with modifying working code. The use of such techniques is known as Agile Programming.
What is Agile Programming?
In the last three years or so there has been a lot of discussion among software vendors about agile software development processes. In particular, there is a lot of excitement associated with the practices that Kent Beck popularized in his book Extreme Programming Explained: Embrace Change. Other "agile" processes that are gaining adherents in the developer community include systems of practices described in SCRUM, Feature Driven Software Development, and Pragmatic Programming. Each of these systems shares some common themes: close collaboration between developers and customers, recognition that changing requirements are a reality of software, and that development practices must play to the strengths of how developers work best.
We chose to use practices from Extreme Programming, probably the best known of the Agile Programming processes, to develop HP Application Server EJB container.
What is Extreme Programming?
Extreme Programming (XP) is a high-discipline, low-ceremony approach to development, meaning that it does not produce a lot of formal documentation or rely on a lot of formal reviews, but does insist that its practitioners be consistent in following the essential practices, including:
Onsite Customer The customer makes all functionality decisions and is always available
The Planning Game Customers get detailed continuous control over what work is done
System Metaphor The overall design is described in terms of shared concepts
Simple Design The design addresses current needs, not possible future additions
Collective Ownership All code is the responsibility of multiple developers
Forty-hour Week Developers may not work overtime two weeks in a row
Test-Driven Design Functionality is validated via automated tests written before coding
Pair Programming Code is written in pairs, thus giving continuous reviews
Continuous Integration Code is pushed to the baseline on a daily basis
Refactor Mercilessly Poorly structured code is cleaned up aggressively
Small Releases Customers are given updated working code to use frequently
Coding Conventions The developers agree on a common coding style and follow it
These practices have been shown to work together to produce consistent, reliable and flexible software without running a development team into the ground. Of course, XP is deeper than just a set of practices. It declares that programming should be a humane discipline that doesn't drive talent out of the trade.
And of course, as part of its "embrace change" attitude, XP insists that each team constantly analyze those practices it is following and change them when appropriate. It is more important to emphasis the values of listening, testing, coding, and designing than to adhere to someone else's checklist or rules.
EJB Development
Both of us worked on the EJB container in the latest release of HP Application Server. The project was a major undertaking. Within a six month time frame, the team had the responsibility of rearchitecting and rewriting most of the EJB-related code base. We needed to fit the container into a service framework on which the application server was being built, interact with other newly defined services, support a number of new features like hot redeployment and pluggable transport protocols (ensuring that all J2EE Compatibility Test Suite tests ran at 100%), and bring the container in line with the EJB 2.0 specification, public final draft two.
While we had experimented with XP techniques like automated unit testing in the past, we wanted to learn more about XP and what it had to offer through experience. Another great benefit was that Russ had joined the team at this point, and brought with him experience with XP. We'll try to summarize some of the what we did do, some of what we didn't do, and look at what the implications were for our project.
XP Programming Techniques We Used
Frequent Testing
Testing was one of our clear wins on this project. We got into the habit of writing unit tests for the software, using a combination of ant, JUnit, and HttpUnit to run the tests in suites. We adopted a rule that nobody could commit changes to the baseline unless all tests passed. As a result, the EJB container code was almost never broken by something on the of the EJB developers did, and we had a high degree of confidence that something that worked at one time continued to work through the 8.0 release.
One of the most useful rules the team developed was that bug reports should be accompanied by automated JUnit tests. The test code showed the problem in a repeatable way and the tests were added into the developer suite to ensure that regression didn't occur.
Continuous Integration
We encouraged our developers to check in their code every day, while keeping the baseline working (all regression tests running at 100%). This was tricky since we automatically absorbed all changes from other groups working on other portions of the application server every day as well. At times the upstream tests would miss something that would affect us further downstream. Since we couldn't always wait for the upstream changes to be fixed, this caused us to stumble a few times when we checked in code that broke the same tests. But for the most part, we were able to keep the baseline clean throughout the cycle, and this really helped us maintain the quality of the software and protect against regression even during aggressive overhauls of the container.
Another important benefit we saw from frequent integration was that it helped everyone understand how both refactoring and new development were effecting the evolution of the system. Since change was incremental, it was comprehensible. If we had all waited until some integration phase to bring pieces together within the system, everything that a developer wasn't actively working on would have become like a black box.
Refactor Mercilessly
It is very common to develop software which works and fulfills all existing requirements - but which cannot be easily extended to handle new requirements. The most common reaction to encountering such a situation is to sigh, roll up one's sleeves, and rewrite the offending code, taking into consideration the new needs. But doing so discards all of the work that went into debugging the original code; it's also time consuming, and often leads to code no better structured than the version which was replaced. Refactoring addresses this, providing a way to restructure the original code so that new features can be added easily without changing the behavior of the working code at all. This is one of the most important practices of XP; in some ways, it is the essence of XP. There are two things about refactoring that are critical to understand. First of all, it doesn't work without tests, period. If you don't have the tests, you will almost certainly break the software. Secondly, refactoring can be divided into small and well understood steps, catalogued in such places as Martin Fowler's Refactoring: Improving the Design of Existing Code and on his website, http://www.refactoring.com. It is also greatly simplified by support in automated tools -- something that's now creeping into Java IDEs.
There were a number of times during the development of the EJB container when we started with code left over from the Total-e-Server implementation, and refactored it to allow the addition of new features. We didn't do a lot of it, but what we did was invaluable. For example, we were able to refactor a legacy class from over 2500 lines to 300 lines, gradually, over the course of the release. This made it possible for us to more readily support hot redeployment of EJBs within the container.
Simple Design
Trusting in our ability to refactor as needed, we tried hard to avoid common expensive practices such as "adding hooks" for future growth. This allowed us to avoid unnecessary code, and we did not see any place where it hurt us. It also made the code much easier to understand.
User Stories
XP also promotes the creation of user stories, which are lightweight use cases for pieces of functionality. User stories were something we used consistently, but also something we struggled with doing correctly. The user stories we developed were very much like functional requirements for the system. As a result many of the stories were driven by developers rather than the customer proxy. We did, however, use these as a low-investment strategy for planning. Our group manager played the role of a tracker and tried to make sure the stories within a cycle were consistent with the velocity measurements he had taken for past cycles. This was a great benefit for tracking and planning with agility.
XP Programming Techniques We Didn't Use Consistently
Collective Ownership
With XP, assignments are distributed so that developers don't specialize. In practice, we assigned developers to specific functional areas and for the most part did not overlap, fearing that without specialization we would not be able to develop the necessary expertise in each area. The downside, of course, is that there are many parts of the code known by only one developer and never reviewed or even looked at by the others. It is too soon to tell how this will affect future maintenance.
Pair Programming
Pair programming is a concept that seems alien and we've found that this is an idea that people find hard to digest. For one thing, it's simply very different from what we're used to. It also seems counter-intuitive that a process designed around rapid software deliveries would tie two people to a computer. Turns out we were wrong.
We found that when we paired on user stories, different areas of experience of different programmers really combined to give the pair a deeper understanding of the problem space. As a general rule, pairing led to better code. We extended this outside of our project and attempted to always pair when we worked with other teams interfacing to our container. This helped both sides to get a better understanding of the critical glue code between subsystems.
Another critical point about pairing: it encourages discipline. If one person is behind and tempted to cheat on test coverage, for example, an individual can get away with it. Another benefit of pairing is that it tends to keep people focused. You can't waste time on nonproductive web surfing when someone is working with you.
When push came to shove, we needed to deliver and we fell back on what on the more conventional techniques that we knew. We had to rely on code reviews that were often stressful and grinding to audit each other, and the there were times when class design suffered. Pair programming may seem unnatural, but it has some real benefits that make it worth considering as a basic practice for your team.
Test-Driven Design
For the most part, the team didn't do "test first" programming. By the time we started seriously writing automated tests, most of the code was already in place, and needed updating rather than writing from scratch. In addition, despite reading the XP literature, we did not realize just how powerful this technique can be. Some of us did have the opportunity to write functionality that was brand-new for EJB 2.0, and we did use test-first in those parts. The result seemed to be fewer false starts and simpler designs than in areas built without such tests.
Architecture vs. Metaphor
XP puts forward the notion of a metaphor for a system as a substitute for architecture. We didn't attempt this: we spent time working out a macro-architecture, talking it through face-to-face, and adjusting it as necessary as we progressed. The metaphor idea didn't seem to make sense for something like a framework and the approach we took seemed to work well. So this was an aspect of XP that we didn't utilize, but also didn't feel that we were missing anything.
Limits of Tests
It also became quickly clear that there are limits to what the unit tests provide. Tests aren't a substitute for really digging into the system. Constant and liberal changes can result in subtle changes of untested assumptions. It's easy to say that the problem is there aren't enough tests which, while true, isn't an answer. After all, there will never be enough tests to check every assumption about the system. Software is complicated, and middleware particularly so: it combines the challenges of concurrent, distributed systems programming with a need for an extensive understanding of low-level concepts in security, distributed transaction processing, high availability system engineering, etc.
Performance
One of the questions we had was how XP would effect performance. Refactoring typically leads to more indirection and we specifically avoided writing "optimized" code. Instead, we waited until after all features were complete to systematically profile the system under load with real end-user applications. This paid off hugely: the container was considerably faster than previous iterations and the unit tests helped to ensure that the optimizations we introduced did not break the system.
The most important bottleneck in middleware for distributed systems is resource contention, and we were able to isolate and eliminate the contention issues we identified. The result was not only a faster system, but also a more scaleable system. We had great success with this approach, but would hesitate to recommend it without unit tests in place.
Conclusion
Over the course of the release, there were times when even we doubted our ability to deliver on all of the requirements in the time required. It's our strong conclusion that using the techniques from XP that we used were fundamental to pulling the software together into a well-received, sophisticated application server. We found that the basic practices combine and play off each other to help make better software and that we missed out on some key benefits from practices we weren't able to implement. We've worked on projects where the code regresses over time and we're happy to report that the code base for the EJB container took a different path: it got better and better as the release progressed. There are unique challenges to every project: some teams are very large, some are very big, some distributed geographically.
We would encourage you to experiment with XP and other Agile Programming techniques and adopt them to your environment. They pay off in a big way: well written software that meets real requirements in short time frames.
More information regarding Extreme Programming is available at:
http://www.extremeprogramming.com;
and for more information regarding Agile Programming, check out:
http://www.agilealliance.org.
ABOUT THE AUTHORS:
This article was contributed by Russell Gold and Greg Pavlik architects with Hewlett-Packard's Middleware Division. Russell has been instrumental in applying the eXtreme programming methodologies to the Middleware development process. Russell is also the original author and maintainer of HttpUnit, an open source library for automating tests of web sites. Greg was the lead architect on the HP Application Server's EJB 2.0 and 1.1 implementations and has been a member of the EJB expert group.
Monday, June 12, 2006
Mothra Attacks
The sleepy little town of Shamong, New Jersey, where I live, is currently under occupation by an army of gypsy moths. My own property is heavily wooded, but I have to admit I haven't noticed any real activity: until I started looking up on the drive into the office. Its somewhat jarring to see whole sections of forest denuded of leaves!
Perspectives on Diversity
I recently had the opportunity to participate in a workshop on diversity for managers. The interesting thing about the exercise was that the focus of the workshop was not on issues of overt discrimination, but how a person's own identity influences and shapes their spheres of inclusion: in discussions, in peer groups and by implication in large organizations. I think it goes without saying that overt discrimination is a problem that needs to be dealt with, but it is also one that any healthy organization is committed to addressing quickly and effectively in the modern and globalizing business environment.
The way we project our identity into work environments unconsciously, however, is a much more interesting issue. On the one hand, our received identities are a tremendous source of personal strength and a way to build bridges with others. On the other hand, they can be a wall. Simple example: a group of Indian and British guys sitting around talking about cricket may be a wall to, say, a Canadian baseball fan. Then again, it's also a chance to build a bridge.
But what I found most interesting of all was reflecting on my own self-identification and how its shifted over the years. When I was growing up, in a lower-middle-to-working class, predominantly white and Catholic neighborhood, I picked up many of the dominant biases of my environment. In general, these were parochial: in retrospect, it was an environment that suffered from a fair amount of myopia about the broader world. My way of looking at the world was also one that focused exclusively on defending legitimacy claims about its own interests without showing the same consideration for others. After all, when you're right, you're right.
When I went off to be educated at an Ivy League school, my impulse was to seek philosophical rationalization for my own biases. Of course, I didn't think these were biases at all: as a kid, I thought I knew how the world worked (though of course I didn't) and my mission was to prove that this received way of looking at the world was correct. In many ways, I have come to see that this is quite common, and it applies to the all kinds of philosphical biases: people often tend to search for reasons to prove they are correct in what they think they know, rather than seeking a balanced perspective. In my case, I read widely, but in retrospect, with an aim to narrowing rather than opening the mind. Looking for justification rather than understanding and systematizing against all the wrong things. I managed to make an ass of myself more than once and I can only imagine what sensible friends thought at the time.
Fast forward through the years (now we're on order of decades!) and I look at my own identity as being very different than it was as a older child/younger man. I tend to err on the side of ambiguity when it comes to philosophy, religion, comparative judgements across cultures. My friends are literally from all over the world, from all kinds of religions, races, and ethnicities. And I find that even my own cultural identification is progressively more difficult to pin down, even in simple, basic ways: I can't imagine not eating Indian food, listening to jazz, trying to struggle with Japanese, coming to terms with the sheer ancientness and depth of Chinese civilization, planning trips to Africa, spending time in Europe. And there is something else very important that I've learned through life. If there's one thing you can count on about a stereotype is that the first thing real people do is prove it wrong.
The 20th century persons I tend to admire now are people like Martin Luther King or Ghandi -- though both were in some sense religious figures, while I struggle with the idea of faith regularly, I've been most attracted to their emphasis on both nonviolence and bridging extraordinary cultural divisions. Unlike the titanic political figures of the 20th century, neither changed the world by brute force; instead, they were closer to what seems to me to be the ultimate expression of the ethic that the Jesus we know about from religious history taught and lived. As humans, we like our heroes to be god-like, seemingly perfect. In fact, there is no such person that has existed. But some people do manage to rise above our imperfect condition and change the world in ways that deserve to be admired.
I've often thought about how my own process of identification and outlook has changed -- what I regard as a maturation process -- and I'm not sure how to explain it with confidence. I like to think that this is a result of reflection and inner-directed growth, but it could also simply be the influences of an environment that I'm now a part of -- and one that I love. My day to day world is both multicultural in the American sense and really quite international. One of the defining facts of the technology industry is that it is global and it will only continue to become more so.
There is a practical point to all of this: its impossible to do global business without a perspective that builds on mutual respect for people of all kinds of backgrounds. I would argue that is also true even in the American context, but that's in some way yet another blog entry. The essential question is how to ensure a broad understanding and acceptance of this fact. The cliche answer that it's a matter of education, I think, falls short. Education can just as easily reinforce biases and enforced education may increase barriers to acceptance all the more. On this I don't have a clear answer, but it is undoubtedly among the most important issues of this young century.
The way we project our identity into work environments unconsciously, however, is a much more interesting issue. On the one hand, our received identities are a tremendous source of personal strength and a way to build bridges with others. On the other hand, they can be a wall. Simple example: a group of Indian and British guys sitting around talking about cricket may be a wall to, say, a Canadian baseball fan. Then again, it's also a chance to build a bridge.
But what I found most interesting of all was reflecting on my own self-identification and how its shifted over the years. When I was growing up, in a lower-middle-to-working class, predominantly white and Catholic neighborhood, I picked up many of the dominant biases of my environment. In general, these were parochial: in retrospect, it was an environment that suffered from a fair amount of myopia about the broader world. My way of looking at the world was also one that focused exclusively on defending legitimacy claims about its own interests without showing the same consideration for others. After all, when you're right, you're right.
When I went off to be educated at an Ivy League school, my impulse was to seek philosophical rationalization for my own biases. Of course, I didn't think these were biases at all: as a kid, I thought I knew how the world worked (though of course I didn't) and my mission was to prove that this received way of looking at the world was correct. In many ways, I have come to see that this is quite common, and it applies to the all kinds of philosphical biases: people often tend to search for reasons to prove they are correct in what they think they know, rather than seeking a balanced perspective. In my case, I read widely, but in retrospect, with an aim to narrowing rather than opening the mind. Looking for justification rather than understanding and systematizing against all the wrong things. I managed to make an ass of myself more than once and I can only imagine what sensible friends thought at the time.
Fast forward through the years (now we're on order of decades!) and I look at my own identity as being very different than it was as a older child/younger man. I tend to err on the side of ambiguity when it comes to philosophy, religion, comparative judgements across cultures. My friends are literally from all over the world, from all kinds of religions, races, and ethnicities. And I find that even my own cultural identification is progressively more difficult to pin down, even in simple, basic ways: I can't imagine not eating Indian food, listening to jazz, trying to struggle with Japanese, coming to terms with the sheer ancientness and depth of Chinese civilization, planning trips to Africa, spending time in Europe. And there is something else very important that I've learned through life. If there's one thing you can count on about a stereotype is that the first thing real people do is prove it wrong.
The 20th century persons I tend to admire now are people like Martin Luther King or Ghandi -- though both were in some sense religious figures, while I struggle with the idea of faith regularly, I've been most attracted to their emphasis on both nonviolence and bridging extraordinary cultural divisions. Unlike the titanic political figures of the 20th century, neither changed the world by brute force; instead, they were closer to what seems to me to be the ultimate expression of the ethic that the Jesus we know about from religious history taught and lived. As humans, we like our heroes to be god-like, seemingly perfect. In fact, there is no such person that has existed. But some people do manage to rise above our imperfect condition and change the world in ways that deserve to be admired.
I've often thought about how my own process of identification and outlook has changed -- what I regard as a maturation process -- and I'm not sure how to explain it with confidence. I like to think that this is a result of reflection and inner-directed growth, but it could also simply be the influences of an environment that I'm now a part of -- and one that I love. My day to day world is both multicultural in the American sense and really quite international. One of the defining facts of the technology industry is that it is global and it will only continue to become more so.
There is a practical point to all of this: its impossible to do global business without a perspective that builds on mutual respect for people of all kinds of backgrounds. I would argue that is also true even in the American context, but that's in some way yet another blog entry. The essential question is how to ensure a broad understanding and acceptance of this fact. The cliche answer that it's a matter of education, I think, falls short. Education can just as easily reinforce biases and enforced education may increase barriers to acceptance all the more. On this I don't have a clear answer, but it is undoubtedly among the most important issues of this young century.
Wednesday, June 07, 2006
Open Source Tivoli?
In an interesting move, Hyperic has open sourced their management infrastructure. Perhaps its not too surprising to see that they have Bob Bickel, who built and sold off the JBoss company to RedHat, as a company advisor. I wonder if this is an area where open source makes sense, but its surely going to get them some attention.
Tuesday, June 06, 2006
Historical Blog Entries
I've gotten several emails asking how to get at content from my now-discontinued Oracle blog. You can find the last entry here; just scroll back to get to previous entries.
The only way to find a particular entry, though, seems to be Google. I can help if that proves difficult.
The only way to find a particular entry, though, seems to be Google. I can help if that proves difficult.
Monday, June 05, 2006
International Conference on Service Oriented Computing 2006
ICSOC is back in the United States this year in Chicago starting on December 4th. I am pleased to have an opportunity to serve on the program committee this year. The Call for Papers may be found here.
Subscribe to:
Posts (Atom)