So you know what you’re going to build and how you’re going to build it. What process should you go about building it? Who’s going to build what? What order should you build it in? How do you make sure everyone is in sync while you’re building it? ^{22 22}

Tim Pettersen (2016). Git Workflows with Tim Pettersen. Software Engineering Daily Podcast.

At the foundation of all of these questions are basic matters of  project management : plan, execute, and monitor. But developers in the 1970’s and on found that traditional project management ideas didn’t seem to work. The earliest process ideas followed a “waterfall” model, in which a project begins by identifying requirements, writing specifications, implementing, testing, and releasing, all under the assumption that every stage could be fully tested and verified. (Recognize this? It’s the order of topics we’re discussing in this class!). Many managers seemed to like the waterfall model because it seemed structured and predictable; however, because most managers were originally software developers, they preferred a structured approach to project management ^{31 31}

Gerald M. Weinberg (1982). Over-structured management of software engineering. ACM/IEEE International Conference on Software Engineering.

In 1988, Barry Boehm proposed an alternative to waterfall called the  Spiral model ^{4 4}

Barry W. Boehm (1988). A spiral model of software development and enhancement. IEEE Computer.

Around the same time, two influential books were published. Fred Brooks wrote  The Mythical Man Month ^{6 6}

Fred P. Brooks (1995). The mythical man month. Pearson Education.

These early ideas in software project management led to a wide variety of other discoveries about process. For example, organizations of all sizes can improve their process if they are very aware of what the people in the organization know, what it’s capable of learning, and if it builds robust processes to actually continually improve process ^{9,10 9}

Tore Dybå (2002). Enabling software process improvement: an investigation of the importance of organizational issues. Empirical Software Engineering.

Beyond process improvement, other factors emerged. For example, researchers discovered that critical to team productivity was  awareness  of teammates’ work ^{16 16}

Amy J. Ko, Rob DeLine, and Gina Venolia (2007). Information needs in collocated software development teams. ACM/IEEE International Conference on Software Engineering.

In addition to awareness,  ownership  is a critical idea in process. This is the idea that for every line of code, someone is responsible for its quality. The owner  might  be the person who originally wrote the code, but it could also shift to new team members. Studies of code ownership on Windows Vista and Windows 7 found that less a component had a clear owner, the more pre-release defects it had and the more post-release failures were reported by users ^{3 3}

Christian Bird, Nachiappan Nagappan, Brendan Murphy, Harald Gall, and Premkumar Devanbu (2011). Don't touch my code! Examining the effects of ownership on software quality. ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE).

Pace  is another factor that affects quality. Clearly, there’s a tradeoff between how fast a team works and the quality of the product it can build. In fact, interview studies of engineers at Google, Facebook, Microsoft, Intel, and other large companies found that the pressure to reduce “time to market” harmed nearly every aspect of teamwork: the availability and discoverability of information, clear communication, planning, integration with others’ work, and code ownership ^{24 24}

Julia Rubin and Martin Rinard (2016). The challenges of staying together while moving fast: an exploratory study. ACM/IEEE International Conference on Software Engineering.

Because of the importance of awareness and communication, the  distance  between teammates is also a critical factor.  This is most visible in companies that hire remote developers, building distributed teams, or when teams are fully distributed (such as when there is a pandemic requiring social distancing). One motivation for doing this is to reduce costs or gain access to engineering talent that is distant from a team’s geographical center, but over time, companies have found that doing so necessitates significant investments in socialization to ensure quality, minimizing geographical, temporal and cultural separation ^{26 26}

Darja Šmite, Claes Wohlin, Tony Gorschek, Robert Feldt (2010). Empirical evidence in global software engineering: a systematic review. Empirical Software Engineering.

A critical part of ensuring all that a team is successful is having someone responsible for managing these factors of distance, pace, ownership, awareness, and overall process. The most obvious person to oversee this is, of course, a project manager ^{5,21 5}

Mike Borozdin (2017). Engineering management with Mike Borozdin. Software Engineering Daily Podcast.

While all of this research has strong implications for practice, industry has largely explored its own ideas about process, devising frameworks that addressed issues of distance, pace, ownership, awareness, and process improvement. Extreme Programming ^{2 2}

Kent Beck (1999). Embracing change with extreme programming. IEEE Computer.

• Be iterative
• Do small releases
• Keep design simple
• Write unit tests
• Refactor to iterate
• Use pair programming
• Integrate continuously
• Everyone owns everything
• Use an open workspace
• Work sane hours

Note that none of these had any empirical evidence to back them. Moreover, Beck described in his original proposal that these ideas were best for “ outsourced or in-house development of small- to medium-sized systems where requirements are vague and likely to change ”, but as industry often does, it began hyping it as a universal solution to software project management woes and adopted all kinds of combinations of these ideas, adapting them to their existing processes. In reality, the value of XP appears to depend on highly project-specific factors ^{20 20}

Matthias M. Müller and Frank Padberg (2003). On the economic evaluation of XP projects. ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE).

At the same time, Beck began also espousing the idea of  “Agile” methods , which celebrated many of the values underlying Extreme Programming, such as focusing on individuals, keeping things simple, collaborating with customers, and being iterative. This idea of begin agile was even more popular and spread widely in industry and research, even though many of the same ideas appeared much earlier in Boehm’s work on the Spiral model. Researchers found that Agile methods can increase developer enthusiasm ^{28 28}

Sharifah Syed-Abdullah, Mike Holcombe & Marian Gheorge (2006). The impact of an agile methodology on the well being of development teams. Empirical Software Engineering.

Ultimately, all of this energy around process ideas in industry is exciting, but disorganized. None of these efforts really get to the core of what makes software projects difficult to manage. One effort in research to get to this core by contributing new theories that explain these difficulties. The first is Herbsleb’s  Socio-Technical Theory of Coordination (STTC) . The idea of the theory is quite simple:  technical dependencies  in engineering decisions (e.g., this function calls this other function, this data type stores this other data type) define the  social constraints  that the organization must solve in a variety of ways to build and maintain software ^{11,12 11}

James D. Herbsleb and Audris Mockus (2003). Formulation and preliminary test of an empirical theory of coordination in software engineering. ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE).

I extended this idea to congruence with beliefs about  product  value ^{17 17}

Amy J. Ko (2017). A Three-Year Participant Observation of Software Startup Software Evolution. ACM/IEEE International Conference on Software Engineering, Software Engineering in Practice.