“Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations.”
According to what is now celebrated as Conway’s Law, “if you have four groups working on a compiler, you’ll get a 4-pass compiler.” In other words, process becomes product. In an illustrative example, Melvin Conway describes a contract research organization with eight people who were to produce a COBOL and an ALGOL compiler. After some initial estimates of difficulty and time, five people were assigned to the COBOL job and three to the ALGOL job. Not surprisingly, the resulting COBOL compiler ran in five phases, and the ALGOL compiler ran in three.
In a similar fashion, the communication structure of an organization is shaped by its administrative structure and thus mirrors it. An example in mini-computer hardware design is offered by Tracy Kidder. Excerpted from his classic book, The Soul of a New Machine, is the following narrative: “Looking into the VAX, West had imagined he saw a diagram of DEC’s corporate organization. He felt that VAX was too complicated. He did not like, for instance, the system by which various parts of the machine communicated with each other, for his taste, there was too much protocol involved. He decided that VAX embodied flaws in DEC’s corporate organization. The machine expressed that phenomenally successful company’s cautious, bureaucratic style.” In other words, organization design becomes product design.
Conway advised that: a design effort should be organized according to the need for communication. We know from experience that the design which occurs first is almost never the best possible. Since the need for communication changes according to how design evolves over time, it is important to keep organizations lean and flexible. Therefore, Conway suggested that one must not naively assume that adding manpower adds to productivity. Instead, with great prescience in 1968, he advised answering basic questions about value of resources and techniques of communication as a first step towards a technology (i.e., a process innovation) of building systems with confidence.
The process of developing a new technology through open discussion is called collective invention. According to Peter Meyer, “it is a process in which improvements or experimental findings about a production process or tool are regularly shared.” Mayer documented five episodes of collective invention from recent historical experience: (1) steam engine (1811-1904); (2) iron blast furnace in Britain’s Cleveland district (1850s-1970s); (3) early steel production in the U.S. (1865-1880); (4) microcomputer clubs (1975-1985); and (5) Linux (1991-present).
In each case, there was no one single user or any single inventor. Instead, there was one central figure that played a key role in coordinating the success of collective invention: (1) Joel Lean, who edited the Lean’s Engine Reporter; (2) Isaac Lowthian and others, who published technical information about blast furnaces in operation; (3) Alexander Holley, who consulted widely and published technical reports for his steel industry clients; (4) Lee Felsenstein, who moderated the Homebrew Computer Club; and (5) Linus Torvald, who started Linux and guided its development. They offered the valuable service of information brokerage from the center of a star-shaped social network of experimenters, thus reducing the cost of search for innovations in the network. This enabled the accumulation of innovations to happen over time through experimentation and sharing across the collective invention network.
There are many explanations as to why individuals or firms would want to participate in sharing. A state of technological uncertainty and opportunity contributes to most of them. Sharing and experimentation are in fact complementary activities. It would be inefficient to do one without the other within the collective search process. After all, without a venue in which to share findings or learn from each other, some experiments might not have been carried out in the first place. Among the many diverse motivations that Robert Allen has identified as drivers of information sharing, two of them bear closer examination: (i) there are advantages in establishing engineering standards by giving away designs or software; and (ii) while firms compete locally against other firms, collectively they compete against other regions, and thus have an incentive to work together to make local production as efficiently as possible and remote regions irrelevant. Taken as a whole and reinterpreted in the modern context, one might say that agreeing to common engineering standards is a first step towards building a shared knowledge infrastructure that enables a proximate cluster of emerging firms to build a new and more efficient value network, in the process displacing the incumbents that are entrenched in the older value network. Economists would recognize this phenomenon and characterize it as follows: experimentation creates productive capital through sharing.
Individuals or firms have diverse resources, opportunities, insights, abilities, interests, skills, and agendas. Each one may have something unique to contribute to the collective search process. Those who want to make progress in the collective search process experiment more and find that it is optimal to share their findings. And those who participate in sharing find it optimal to experiment more. This summarizes the underlying self-reinforcing dynamics that drive a collective invention network. In contrast, a hierarchically organized system suffers from what is commonly known as the “Peter Principle,” where the selection of a candidate for a position is based on the candidate's performance in their current role rather than on abilities relevant to the intended role, and as a result “mangers rise to the level of their incompetence” throughout the system. Hierarchy impedes performance.
The emergence of the Internet and the Web offers an excellent example of large-scale collective invention in action, where the decentralized nature of its communication structure initially took shape as a “network of networks”, and subsequently evolved into what is now a “network of platforms.” This occurred over the past few decades within the collective invention network nurtured by DARPA, NSF, and eventually the open Internet and the Web, and through exposure to economic experimentation and community feedback from usage.
The structure of the Internet itself was unanticipated; its development started at a time when "packet switching" and "network of networks" were budding theoretical concepts, and nobody knew where implementation would lead. Not only was the early Internet a radical technological departure from existing practice, the geographical dispersal of its diverse research participants represented another major departure from DARPA's typical centralized program administration. Even the governance structure of the Internet was unprecedented in its openness and transparency, led by a surprising set of technological leaders, many of whom were graduate students at the time. Was it any coincidence then that the Internet became the underlying structure for decentralized exploration which created massive market value over time by aggregating innovations from its diverse participants as it continued to evolve?
The accumulated knowledge enabled the further creation of value in myriad numbers of applications, e.g., the new, ongoing "Internet of Things", that continue to shape the world around us. Perhaps in the not-too-distant future we might even see a “networked intelligence” arising through this process of collective invention. “It would take off on its own, and re-design itself at an ever increasing rate,” is one such scenario anticipated by Stephen Hawking. In his recent remark about the future of artificial intelligence, Hawking also predicted that: “humans, who are limited by slow biological evolution, couldn't compete, and would be superseded.”
Original Plan: The Mermaid (by John Reinhard Weguelin).
Unanticipated Outcome: The Collective Invention (by Rene Magritte).
Interestingly, collective invention is most valuable when there is uncertainty in a large search space along many dimensions, as there is the possibility of truly major innovations on the horizon. Collective invention may simply be an essential phase of technology improvement at its earliest stage. Early automobiles and airplanes seem to have developed along a collective invention path before industries started to form. When searching for market inefficiencies in the financial universe, we often wonder if sharing across the network – beyond academic publishing – may be a way of searching more efficiently given the accessibility of present cloud-computing infrastructure, and how that might impact the incumbents of today’s financial markets. What do you think?
“Everything we see hides another thing, we always want to see what is hidden by what we see.”
References:
- Conway, Melvin (1968). How do Committees Invent? Datamation (April, 1968). Retrieved from: http://www.melconway.com/Home/Committees_Paper.html and http://www.melconway.com/Home/pdf/committees.pdf
- Kidder, Tracy (1981). The Soul of a New Machine. Little Brown and Company.
- Allen, Robert C. (1983). Collective Invention. Journal of Economic Behavior and Organization, Vol. 4, pp. 1-24. Retrieved from: http://www.nuffield.ox.ac.uk/users/allen/collinvent.pdf
- Cowan, Robin and Jonard, Nicolas (2000). The Dynamics of Collective Invention. Journal of Economic Behavior and Organization, Vol. 52, No. 4, pp. 513-532. Retrieved from: http://arno.unimaas.nl/show.cgi?fid=279
- Meyer, Peter B. (2003). Episodes of Collective Invention. U.S. Bureau of Labor Statistics. Retrieved from: http://www.bls.gov/ore/pdf/ec030050.pdf
- Greenstein, Shane (2009). Nurturing the Accumulation of Innovations: Lessons from the Internet. In: Accelerating Energy Innovation: Insights from Multiple Sectors (2011), Rebecca M. Henderson and Richard G. Newell, editors (pp. 189-223). Retrieved from: http://www.nber.org/chapters/c11755.pdf
- Raymond, Eric S. (1996). A Brief History of Hackerdom. Retrieve from: http://www.catb.org/~esr/writings/cathedral-bazaar/hacker-history/
- Raymond, Eric S. (1999). Revenge of the Hackers. Retrieve from: http://www.catb.org/~esr/writings/cathedral-bazaar/hacker-revenge/
- Kessler, Andy (2005). How We Got Here: A Slightly Irreverent History of Technology and Markets. Harper Collins.
- Isaacson, Walter (2014). The Innovators: How a Group of Hackers, Geniuses, and Geeks Created the Digital Revolution. Simon and Schuster.