What do Psychology and Software Development have in common?
Well, as notoriously rational as software developers like to claim they are, they are humans, and subject to all the cognitive biases and errors that plague the rest of humanity. Well, most of them at least.
Three particular issues seem the most common in the industry and infect a lot of developers' thinking.
Status Competition refers to the way humans compete with each other socially. Instead of physically beating up or dominating each other, civilized developers typically seek evidence that they’re “beating” their peers socially. Vying for the corner office, showing off a new company car, and gloating over executive bathrooms are examples of status competition rearing its head within teams.
In the software world, being given the responsibility to design a system, or to lead a team are often high-status achievements.
Near/Far Bias, for these purposes, is the way people tend to think about things differently if they’re happening in the immediate future or physically nearby. Of course, this is relative to the way humans think about things that are farther in the future, or physically or emotionally far away.
For example, thinking about going to the grocery store later today, it’s easy to think about every stoplight, every turn, even every aisle at the store you are going to visit. On the other hand, thinking about that vacation you're going to take in a year-and-a-half doesn’t have the same "turn-by-turn" level of detail.
In the software world, creating a complete system architecture, or a master project plan, is "far-biased" thinking. Writing unit tests for your current feature is "near-biased" thinking. Generally, thinking under a "far bias" involves a lot more optimism, and a lot less detail than a "near bias".
Most people are familiar with the basic concepts of narcissism: the idea that a narcissist feels he or she is more important than others & that his or her opinion matters much more than the opinion of others, too. Furthermore narcissists believe they have unique insights into the world that others lack, and that makes them better/wiser/smarter.
In the software world, narcissism is the tendency to blame everyone else for problems or mistakes. To believe that you're the only one capable of solving a problem, or that everyone else in the group is just a 'resource' in your personal game; a resource that you guide like a wise, enlightened master.
Psychology and Waterfall Development
What's fascinating about these three issues (Near/Far, Status & Narcissism) is how much the classic 'Waterfall' development model enables them. For example:
- Product Owners create a 'vision', and create 'requirements' to implement that vision
- Based on the vision, Project Managers assign 'resources' to 'tasks' and create 'master plans' for the project
- Architects are responsible for 'designing the system', and to 'anticipate every possible contingency'
Already, it is possible to see how the aforementioned psychological issues can influence these projects. 'Vision' is always a far-biased activity, and being responsible for defining that vision is high-status. (Note that almost every project must begin with a 'vision', that's not an unreasonable activity). Assigning 'resources' to 'tasks' is high-status, and involves the fairly narcissistic attitude of thinking of the software developers as 'replaceable parts' of a machine, rather than individual humans. 'Master plan' is far-biased thinking at its best. Being responsible for designing the system? Very high-status. And to attempt to 'anticipate every possible contingency' is an extremely "far-biased" thought process.
What happens after Owners, Managers, and Architects do their part is often (not always) very predictable:
- The customer's needs change, and they weren’t anticipated
- The product owner's vision was too simplistic, and the real implementation is messy
- The architecture missed major requirements, or fails to meet them in complicated ways
- The system is much harder to build than originally planned
- The developers and architects (and sometimes the owners) make estimates based on limited and invalid information, due to a need for estimates months and months in advance.
- Many tasks become stuck at '90%' complete due to logistical or technical issues that can't be anticipated
- People refusing to work on certain problems because it's "not their job"
- Developers and testers can't just be 'moved around', and adding more 'resources' doesn't make the project move faster
- Everyone blames others for the project’s problems
- Lots of weekend work, 'tiger teams' and 'war rooms', and late hours at the office
These kinds of problems don't happen every time, but they happen often. And innate psychological weaknesses are part of the reason why.
Psychology and Agile Development
One of the fascinating things about Agile Development is how it addresses these issues fairly well, without ever discussing or naming them. For example:
- The Product Backlog
- The Product Owner creates a vision for the project (far biased) but must also prioritize and limit what can be done in the next iteration (near biased)
- Defines the goals for the next iteration doesn't mean the developers must do everything in the list. They are able to push back, relieving narcissism and keeping status competition in check.
- Iterative Development
- Instead of thinking 6-18 months in advance for work, the team focuses on the current iteration (near biased)
- If everything committed-to in the iteration doesn't get done, the team may have to work longer to get it completed, but they get to factor that information into the next iteration's estimates (near biased & status-checking)
- Development Team
- Traditionally, there are no 'architects' vs 'developers' in an agile team. Generally everyone does some design and some coding, eliminating status competition
- The team makes commitments together, and are expected to help each other out if things go wrong (anti-narcissism)
- Teams are often responsible for building automated tests for their code, rather than relying on testers to find all possible defects (near-biased, status-checking)
- Pair programming is used in some agile disciplines (anti-narcissism)
- Test-Driven and Test-First development practices force developers to think of themselves as users of their code, not just authors (near-biased, anti-narcissism)
- Regular Demos
- End-of-iteration demos to customers, or customer proxies, provide immediate feedback on what's missing or misunderstood (near-biased, anti-narcissism)
- Developing the system to be demonstrated regularly enforces near-biased thinking as well
- Refactoring forces the developers to rethink their code regularly (anti-status, near-biased) instead of attempting to build a framework in advance to handle every possible usage scenario
- Estimates are primarily made for the current iteration only (near-biased). Also, project estimates are typically not commitments, just well-educated rough ideas.
Psychological Weaknesses of Agile Development
While this is certainly some food for thought, Agile Development is not without its own psychological weaknesses. All of the following issues in Agile projects have been witnessed in the past:
- The development team can become self-absorbed, caring more about writing the software than delivering value to customers
- Agile development doesn't prevent narcissistic attitudes, it merely makes them easier to identify, and hopefully correct
- Sometimes, the nature of the work requires consistency across iterations, and strong-willed agile teams tend to fight against that
So in the end, it's not perfect. Agile Development is not going to solve every problem, and for some types of software development, Agile isn't even the right tool for the job. But in most enterprise environments, for most of your typical software projects, an agile approach will ameliorate many of the psychological weaknesses and mistakes that waterfall projects encounter by design. And given how much those psychological weaknesses undermine the success of the project, why would any team want to handicap themselves?