The evolution of the Internet: from military experiment to General Purpose Technology

Pre-history: 1956–1966

That the Internet owes its existence to the Cold War is well known. But, as ever, retrospective generalisation glosses over a more complicated story.

The first strand of this concerns the doctrine of ‘mutual assured destruction’ (MAD) which governed the nuclear stand-off between the United States and the Soviet Union. MAD supposedly ensured national security by guaranteeing that if one side launched a nuclear attack, the other would retaliate in (devastating) kind. There was, however, one apparent flaw in the logic, in that the doctrine could give an advantage to the aggressor if his pre-emptive strike was so devastating that it rendered the enemy’s command-and-control system inoperative, thereby making it impossible to retaliate.

There was therefore an urgent need to design a communications system capable of surviving a devastating thermonuclear attack. This challenge was taken up by a researcher in the RAND Corporation, Paul Baran, who came up with a design for a mesh network based on high levels of link redundancy and a digital communications technology called packet switching (Hafner and Lyon 1996, 51).⁶ At a time when communications networks were almost exclusively analogue and based on circuit switching, these were radical ideas. Baran worked out a detailed design for a network based on these principles, but for various bureaucratic and other reasons, a prototype network was never built, and the trail apparently went cold.

It transpired, however, that Baran was not the only person to come up with the packet switching idea. In the UK’s National Physical Laboratory, Donald Davies was seeking to create a new kind of communications network for purely civilian applications. He sought to exploit the advantages of digital switches to enable responsive, interactive time-shared computing over large distances and realised that the circuit switching technology of the analogue telephone system was not adequate for this task (Hafner and Lyon 1996, 66). So Davies independently conceived the idea of packet switching as a means of achieving his goal.

The second strand of the story opens with the Soviet Union’s successful launch of the Sputnik satellite in October 1957, an event that profoundly shocked the US defence establishment and led to the setting up of the Advanced Research Projects Agency (ARPA) within the Department of Defense. Early in its organisational life, ARPA morphed into the agency within the Pentagon that funded advanced, ‘blue-skies’ research which could have military applications. In due course, ARPA found itself funding the purchase, operation, and maintenance of at least a dozen (Abbate 1999, 45) expensive mainframe computers for the various university departments and institutes which held research contracts from the agency. The problem was that these machines were incompatible with one another, and therefore could not function as shared resources for the community of ARPA-funded researchers across the US (Hafner and Lyon 1996, 41). From this came the idea, and the funding, for a network that would enable these valuable resources to be shared. ARPANET (Advanced Research Projects Agency Network) was the result.

The ARPANET: 1967–1972

The idea of a ‘resource-sharing’ network first emerged in ARPA in 1966 (Naughton 1999, 84). Design work, conducted in a collegial style (Abbate 1999, 56) unusual in the defence industry, then proceeded over the next two years. The contract to build the network was awarded in early 1969 to Bolt, Beranek and Newman, a Boston-based consultancy firm with strong links to MIT.

The technological and conceptual challenges that faced the network’s designers have long been obliterated by the omniscience of hindsight, but they were formidable.⁷ Given that the network was supposed to facilitate the sharing of expensive and scarce resources, namely the mainframe computers that ARPA had funded in various research centres across the country, a key obstacle to overcome was the fact that these machines were incompatible with one another. For each of them to participate in a network would require the creation of complex, customised networking software which would enable each machine to communicate with every other machine on the system. In the end, this problem was not so much solved as side-stepped: it was decided to build a ‘sub-net’ of identical minicomputers (which came to be called ‘interface message processors’ or IMPs) each linked to a single mainframe ‘host’. In that way the task of writing networking software for a host was greatly reduced: it would simply have to communicate with a single machine – the IMP assigned to it.

Given the technical challenges implicit in the task, the ARPANET was built with astonishing speed. By 1972, the network was essentially complete; the 15 original sites were all connected and operational and a major public demonstration of the system was held in Washington, DC in the Autumn of that year (Hafner and Lyon 1996, 176–186).

From the perspective of the present, three aspects of the ARPANET project stand out.

The first is that while it was a triumph of project management in the conventional sense, success was achieved in an unusually collegial way. This was sensible for several reasons: the network was intended to link high-profile researchers working in elite institutions, and such individuals are not easily herded, plus it made sense to harness the collective IQ of that community at every level, including that of graduate students. For that reason, for example, design of the central protocols of the network was entrusted to a Network Working Group that largely consisted of students. In this way was established the collaborative ethos that has been an important feature of Internet technical development up to the present day.⁸

Secondly, the ARPANET provides an interesting case study in the extent to which technologies are socially shaped. In this case, the shaping was done by the network’s users, first of all because many of them were actively involved in the design of the network and therefore they were both designers and ‘customers’, that is, users. And in the latter capacity, they sprang some major surprises on ARPA managers. The network was intended to be a facility for resource-sharing, but it transpired that it was not much used for this original purpose. Instead, its users employed it mostly for communicating with one another, sharing files and software, and for sending and receiving email (Abbate 1999, 108). In that sense, the community of users came up with a new conception of what ‘networking’ meant – not so much the sharing of machines as the linking of people. As Abbate puts it, ‘Increasingly people within and outside the ARPA community would come to see the ARPANET not as a computing system but rather as a communications system’ (Abbate 1999, 111). Given the technical sophistication of the network’s users, it is also not surprising that they were vocal in their demands for system modification and innovation. But while many user tweaks were ‘encouraged or at least tolerated’ by ARPA, the agency did not always welcome users’ attempts to steer the evolution of the system (Abbate 1999, 93). Its reluctance in this respect may not have been due to hierarchical reflexes so much as the need to reassure Congress that the network was not a publicly funded experiment in computer science, but an administrative tool useful for military and defence purposes.

Thirdly, although the ARPANET was based on the packet switching technology that characterises the modern Internet, it was a unitary network: the subnet of identical, centrally managed IMPs constituted the core of the system. And the network was owned and administered by a single entity – ARPA. So although the ARPANET was the precursor of what came later, it differed in significant ways from its successors.

Development of the TCP/IP-based ‘internetwork’: 1973–1983

During and after the construction of the ARPANET, other significant developments in networking technology were under way. At the University of Hawaii, researchers had built ALOHA – a packet-switched network that operated, not over leased telephone lines, as ARPANET did, but via radio. Within ARPA, it was decided to build on this work by creating a packet-switched radio network (named PRNET) in the San Francisco area. The motivation for this was obvious: ARPA was part of the US Department of Defense and its planners were interested in the potential of packet switching for command-and-control in battlefield conditions. The agency had also begun to experiment with using the technology in satellite communications, for example, for linking seismic monitoring stations in Scandinavia (established to monitor Soviet nuclear testing) with the US, via a network which was christened SATNET.

By the early to mid-1970s, therefore, ARPA found itself running three separate ‘experimental’ networks – ARPANET, PRNET, and SATNET – all of which used packet switching technology, but in different ways. An obvious next step was to see whether a method for ‘internetworking’ them, so that they functioned as an apparently seamless whole, could be developed.

The ‘internetworking’ project began in late 1973. A key challenge for the designers of the new system was to find a way of transitioning from a unitary network like ARPANET to something that could incorporate a variety of different networks that were owned and operated by independent organisations and entities. From a technical point of view, there were various ways of achieving this goal. One was to allow networks wishing to join the new ‘internetwork’ to retain their existing protocols and simply construct ‘gateway’ computers that would translate those into a common set of conventions. The other was to require that all candidate networks adopted a new set of protocols, which would become the lingua franca of the new overarching network (Abbate 1999, 128).

In the end, the second option was adopted, and a suite of interlocking protocols centred on two new ones – TCP and IP – evolved. In this way TCP/IP became the cornerstone of the new ‘network of networks’. The great advantage of this approach was that implicit in it was the possibility of organic growth: as long as a given network ‘spoke’ TCP/IP (as it were) it was free to join the Internet. And because the system was not owned or controlled by anybody (unlike the ARPANET), there were no gatekeepers to control admission to it.

But the potential for organic growth was not the only affordance implicit in the TCP/IP architecture. The designers also faced the puzzle of how to create a network that would be as future-proof as possible, that is, one that could cope with applications that had not been anticipated by the designers. Their solution was to design a system that was not optimised for any particular application (in contrast to, say, the analogue telephone network, which had been optimised for voice calls but proved inadequate for computer-to-computer communication). The Internet, concluded its designers, should do only one thing: it should take in data packets at one end and do its best to deliver them to their destination. It would be entirely agnostic about the contents or purpose of the packets. In this way, all of the ingenuity would be left to users of the network. If someone had an idea that could be realised using data packets, then the Internet would do it for them with no questions asked.⁹

This philosophy – of leaving innovation to the edges of the network – had profound implications. As Van Schewick (2012) describes it, the TCP/IP design created an architecture for ‘permissionless innovation’ which enabled the explosion of disruptive creativity that is perhaps the most distinctive feature of the Internet.

As the new network took shape, the ARPANET continued to operate alongside it. But because it was no longer an experimental facility, ARPA began to look for a new owner for it. It was first offered to AT&T, the national regulated telephone monopoly, but the company turned it down (Abbate 1999, 135). In 1975, the agency transferred operational responsibility for the network to the Defense Communications Agency (DCA) which provided communications services for the US armed forces. This had predictable effects: it began the process of reorienting the network away from its original research focus and towards military applications; it inevitably led to tensions between the military mindset and the more freewheeling ethos of the research community; and it also led to heightened concerns about computer security and access controls.

While this was happening, the Internet project was gathering pace and the TCP/IP family of protocols was being finalised. The imperative was to extend adoption of the new protocols to the point where network effects came into operation. This turned out to be more difficult than anticipated: many nodes were reluctant to devote the necessary resources to configure their operations around the new protocols. It was at this point that DCA management of the ARPANET was to prove decisive. In March 1981, the Pentagon announced that all ARPANET hosts would be required to adopt TCP/IP by January 1983. Not all sites were able to meet the deadline, but by the middle of 1983, every ARPANET host was running TCP/IP (Abbate 1999, 142), which is why we can say that 1983 marks the beginning of the Internet that we use today.

A few months before that, however, DCA concern about the security of the network had led to a decision to split it into civilian and military domains. From October 1982, one domain – the ARPANET – would continue as a research enterprise; the other – labelled MILNET – would henceforth be entirely devoted to military communications. The switchover was implemented in April 1983.

Transition from a military/research network to a ‘civilian’ one: 1983–1995

The creation of the MILNET domain meant that ARPANET returned to being a research-focused network dominated by universities and research institutions, so the breach was an essential first step towards achieving ARPA’s goal of transferring the network to civilian control. The second step was to take measures to foster the dissemination of TCP/IP technology within the computer industry. To that end, ARPA funded various operators to create TCP implementations for various operating systems (notably Unix¹⁰) and launched a $20m fund to help computer manufacturers implement TCP/IP software on their machines (Abbate 1999, 143). So by 1990, TCP/IP was available for most computers, at least in the US market.

Until the end of the 1970s, access to the developing Internet was restricted to those working in a relatively small number of institutions which held research contracts from ARPA. As computer science became an accepted academic discipline in universities, the exclusiveness of the ARPANET/Internet club was increasingly perceived as irritating and dysfunctional. This led the US National Science Foundation (NSF) to fund the creation of the Computer Science Network (CSNET) in the early 1980s. After an initial hiccup, it was decided CSNET would use the TCP/IP protocols, which meant that a connection between CSNET and ARPANET was feasible and so, at a stroke, the community of networked computer scientists was significantly expanded. While access to ARPANET was only granted to researchers funded by the agency, membership of CSNET was open to computer scientists in any institution willing to pay the annual subscription (although commercial use of the network was prohibited under the NSF’s ‘acceptable use’ policy). The result was that the network began to grow at a faster rate – from 2000 host computers in 1985, to 185,000 in October 1989, and 1,776,000 in July 1993.¹¹

CSNET turned out to be just the first step in the NSF’s involvement in networking. In mid-1984, the foundation began funding the establishment of several new supercomputing centres around the US. To make these available to the widest possible community of researchers, a national network was required. The original idea was for a network – NFSNET – linking the centres that would form the ‘backbone’ of a wider academic network, but in 1998, an agreement was reached to use the ARPANET as the (temporary) backbone of the new network while it was being built. With this decision the Internet became ‘a civilian network in all but name’ (Abbate 1999, 194). It transpired, however, that the ageing ARPANET proved inadequate as a backbone for a rapidly expanding national network, and so in the end a swap was done, with NFSNET becoming the backbone for the ARPANET.

On 28 February 1990, the ARPANET was officially decommissioned; the era of formal military involvement in the operation of the Internet had ended. That still left open the question of how the network should be funded in the coming decades. Should it remain a publicly funded operation, with the NSF using taxpayers’ dollars to pay the costs? Among the arguments against this was that it implied a continuation of the rule that the network could not be used for commercial purposes – which would preclude the exploitation of the economic potential of the technology. In the end, the NSF decided that the only way to allow commercial use of the Net would be to privatise it – to take it off the government’s books.

And this is in fact what happened. In 1994, the NSF implemented a plan to allow Internet service to be taken over by commercial companies known as ‘Internet Service Providers’ (ISPs), each of which would operate its own backbone, enabling the old NSF backbone to be decommissioned. Customers would connect to one of the companies’ backbones, and the ISPs would operate a set of gateways at which a number of ISPs could interconnect their systems, allowing traffic to pass smoothly from one network to another, giving end users the illusion of interacting with a seamless, unitary system. What this also implied, though, was that the network was ‘open for business’.

The process by which the network was privatised was a critical determinant of how the Internet evolved, and it has been largely obscured by the Whig interpretation of the network’s history. But there was nothing preordained about the transition. The eventual outcome of the NSF’s handover of the network to private interests was a product of judgement, foresight, consultation, political astuteness, luck, and timing.

A full account of the process lies beyond the scope of this paper,¹² so a single illustration must suffice. One of the key questions facing the NSF administrators was: what kinds of commercial entities should run the network’s backbone? A decade or so earlier, the answer would almost certainly have been AT&T, the regulated telephone monopoly. In 1991–1992, when the actual decision had to be made, an equally plausible candidate might have been IBM (and indeed IBM saw itself precisely in that frame). The attractions for replacing a unitary public service provider (NSF) with a single commercial provider, were obvious in terms of continuity, stability, and order. But giving a single organisation such a degree of control over the network would violate its fundamental design axioms. It was therefore important to ensure that a number of competing ISPs should run the backbone. But that raised the question of how these organisations would co-operate in such a way that the illusion of a seamless network would be maintained. The ISPs proposed a solution: the Commercial Internet Exchange – essentially a router that took traffic from a number of competing ISPs who had agreed to pass data traffic between their networks (Greenstein 2015, 80). Since each exchange was funded by a flat-rate subscription from its members, data exchange between them was effectively free, rather than being metered by volume. This ‘peering’ system¹³ became the predominant model for interconnection and evolved to become a central component of the network’s infrastructure. But in order to make it feasible in 1992, a small yet significant amendment to the NSF’s Charter had to be made – and this was achieved by a Bill, sponsored by a Virginia Congressman, Rick Boucher. The point of this story is simple: there is nothing preordained about technological development.

The first Internet boom: 1995–2000

In the 1980s, ‘cyberspace’ – the term coined by the novelist William Gibson to describe the virtual world behind the computer screen (Gibson 1984) – was an unusual space. It was essentially a geek preserve, with a social ethos that was communal, libertarian, collaborative, occasionally raucous, anti-establishment and rich in debate and discussion (see Hauben and Hauben 1997). It had no commerce, no hierarchies, no crime and no spam, and in general it was populated by people who either knew one another, or at least knew others’ institutional affiliations. In that sense, cyberspace and the so-called real world existed as parallel universes. Most people outside of the magic circle had no knowledge of the network – and even if they did, they would have found it difficult to gain admission to it.

Two developments eroded the distinction between the two universes and caused them gradually to merge. The first was the commercialisation of the network achieved by the NSF’s decision to hand the backbone over to ISPs. This meant that lay people could now access the network – generally via slow dial-up connections. The second development was the arrival of the network’s second ‘killer application’¹⁴ – the World Wide Web.

The Web was the creation of a single individual – the physicist and computer scientist Tim Berners-Lee, who was employed in the late 1980s and early 1990s at CERN, the multinational particle-research laboratory located just outside Geneva in Switzerland. The underlying idea was to develop a way of publishing, locating, and retrieving documents stored on Internet servers across the world, something that would be useful for a large international laboratory like CERN, which had large numbers of visiting physicists and a perennial problem with document control. Berners-Lee’s idea was to take an established technology called ‘hypertext’ – software which created documents with extensive cross-referencing between related sections of text and associated graphics (Naughton 1999, 220) – and make it work across the Internet. In a remarkable burst of creativity at the end of 1990, Berners-Lee created a working prototype of what he dubbed the ‘WorldWideWeb’, in three months (Berners-Lee 2000, 30–32).

The prototype, however, did not generate much excitement at CERN, save among a small group of enthusiasts led by Berners-Lee’s colleague, Robert Cailliau (Berners-Lee 2000, 48; Gillies and Cailliau 2000, 201). Most people did not see the potential of the Web at the beginning. With hindsight, this seems surprising, but in fact it is perfectly plausible when one remembers the context in which the technology first appeared. As Gillies and Cailliau put it:

From virtually the outset of the project, Berners-Lee had been posting documentation on the project on CERN’s Internet server (info.cern.ch). In March 1991, he released the code to a selected number of CERN colleagues who were using NeXT workstations (Berners-Lee 2000, 50). In August, having obtained agreement from CERN management that the laboratory had no interest in retaining the IP rights to the software, he uploaded to info.cern.ch the program he had written to run the Web on NeXT machines, plus the code for a simple line-mode (i.e. text) browser and for a generic web server, and posted a note drawing attention to this in alt.hypertext, the Internet discussion group for researchers working on hypertext. Shortly afterwards, he opened a public Telnet server¹⁵ on info.cern.ch which would enable anyone to dial-in and use the browser installed on the CERN machine. This had the advantage of allowing people who were not in a position to install the Web software on their own machines to experiment with it.

Despite all this, dissemination of the Web in 1991–1992 was slow and remained so until the spring of 1993, when Marc Andreessen and Eric Bina, then working at the National Center for Supercomputer Applications (NCSA) at the University of Illinois at Urbana-Champaign, released Mosaic, a browser they had written for the Web. Although Mosaic was not the first graphical browser, it was the first one to display graphics inline (i.e. as an integral component of a page, rather than in a separate window).

The launch of Mosaic was a landmark moment in the evolution of both the Web and the Internet. It provided a dramatic illustration of the Web’s potential for both publication and commerce. It opened up the possibility that the Web could be an entertainment medium. And because one needed access to the Internet in order to use the Web, Mosaic triggered a sudden surge in the demand for Internet connections among the general public. One source claims that in 1993, traffic on the nascent World Wide Web increased by over 300,000% (Cassidy 2002, 57).

Andreessen left Illinois in 1994 to join with the technology entrepreneur Jim Clark in order to set up Netscape, the first company founded expressly to exploit the Web commercially. The company’s first product was a browser, Netscape Navigator, which although resembling Mosaic in many respects, had been coded from scratch to avoid intellectual property disputes with the NCSA. Navigator rapidly became the dominant browser in the nascent market, partly because versions of it were available for a range of computer systems, including the IBM PC.

On 9 August 1995, the Netscape corporation filed for an Initial Public Offering (IPO). The stock was priced at $28 a share. In the first day of trading, the price peaked at $75 before eventually closing at just over $58, a price that valued the year-old company at $2.9 billion.

Netscape’s extraordinary IPO triggered intense speculative interest in the Internet and paved the way for the first Internet boom, an extraordinary outbreak of what the economist Robert Schiller dubbed ‘irrational exuberance’ (Schiller 2000), and which was later christened the ‘dot-com bubble’.¹⁶ What fuelled the mania was speculative interest in the stock market valuation of a multitude of Web-based companies (‘dot-coms’) which materialised following Netscape’s extraordinary debut, and which was amplified by the activities of fund managers, stock analysts, journalists, and pundits. As one sceptical observer put it, what really happened was that ‘Wall Street moved West’ (Keen 2015).

The core business model of these fledgling companies was the idea of harnessing the network effects implicit in the rapid growth of consumer interest in the Internet, in order to obtain a dominant market share in a range of sectors.¹⁷ At the height of the frenzy, dot-com companies with few customers, little (sometimes no) revenues and few employees, briefly enjoyed stock market valuations greater than those of long-established companies with significant revenues and thousands of employees.

The boom followed the traditional pattern of speculative manias through the centuries and, eventually, in March 2000, the bubble burst. In just over a month, the total market capitalisation of companies on the NASDAQ exchange fell from $6.71 trillion to $5.78 trillion (Geier 2015). In other words, nearly a trillion dollars in value had been obliterated. And fewer than half of the dot-coms founded in the boom survived the crash (Goldfarb, Kirsch, and Miller 2007).

In the disillusionment that followed, many people drew the conclusion that the Internet phenomenon was overblown. This was understandable, but misconceived. If the first Internet bubble showed anything (other than the perennial gullibility of humans) it was that expectations can sometimes run far ahead of technological, and economic, reality. The truth is that in the last decade of the twentieth century, the Internet was still not a technology mature enough to bear the weight of the (feverish) expectations of dot-com investors.

At that time, for example, only an estimated 413m people – 6.7% of the global population – had access to the Net.¹⁸ Most of them connected to the network via slow, noisy, and low-bandwidth dial-up lines. There was virtually no wireless networking outside of research labs, and very few people had persistent IP addresses. Most mobile phones did not connect to the network, and the few that did, connected via very limited WAP connections. There was very little ‘cloud’ computing. Given these realities, the Internet of the late 1990s was very much an immature technology.

The most paradoxical aspect of the first Internet boom however, is that the bubble created much of the technological infrastructure necessary to hasten the maturation of the network. When the bubble began to inflate, some canny observers quoted the old maxim of the Californian Gold Rush of the 1850s: that the people who made most money in California were not the miners or those who panned for gold, but the merchants who sold them pickaxes and shovels. The modern embodiments of those merchants were the telecommunications companies which, in the 1990s, invested heavily in building large fibre-optic cable networks and server farms to service the ‘new’ economy that was apparently coming into being. When the bubble burst, these companies were left with apparently unwanted assets, and some went bankrupt. But the infrastructure that they had built remained, and indeed turned out to be critical in enabling what came next. As the economist J. Bradford DeLong put it, ‘Investors lost their money. We now get to use all their stuff. What got built wasn’t profitable, but a large chunk of it will be very useful’ (DeLong 2003).

And this is an old story. DeLong points out, for example, that the ‘railway mania’ of the nineteenth century lost investors a lot of money, but the extent of the railway network that was the product of the frenzy enabled completely new industries to be built.

Profits, in other words, are not the same thing as social value. And so it proved with the Internet.

‘Web 2.0’: 2000–2003

The Web was originally conceived as a means of sharing information among particle physicists who were scattered across the world. Since most of that information was in the form of documents, the design was therefore for a system that would make it possible to format these documents in a standardised way, publish them online, and make them easy to access. So the first ‘release’ of the Web (to use a software term) created a worldwide repository of linked, static documents held on servers distributed across the Internet.

Given that it was intended as a system for academic researchers, the original Web design was probably fit for purpose in its first two years. But once the Mosaic browser appeared in 1993 and the commercial possibilities of the technology became obvious to the corporate world, the limitations of the original concept began to grate. The early Web did not make provisions for images, for example. And it was a one-way, read-only medium with no mechanism for enabling people to interact with web pages, which meant that it was unsuitable for e-commerce. There was no way for users to talk back to authors or publishers; no way to change or personalise web pages; no way to find other readers of the same page; and no way to share or collaborate over the Web.

From 1993 onwards therefore, there was a steady accretion of innovative technologies designed to extend Berners-Lee’s creation and to overcome some of its perceived limitations. The main driver behind this was e-commerce, which desperately needed to transform the Web into a medium that facilitated transactions.

In order to make transactions possible, a whole range of problems had to be solved. For example, ways had to be found to allow interactivity between browsers and servers; to facilitate personalisation of web content; and to overcome the problem that the httpprotocol was both insecure (in that communications between browser and server could be intercepted and monitored by third parties) and stateless (i.e. unable to support multistep transactions).

In time, solutions to these problems emerged in the forms of: ‘cookies’; HTTPS (an encrypted version of the basic httpprotocol); the evolution of browsers with capabilities added by specialised ‘plug-ins’ which enabled them to handle audio and video and other kinds of file; and, eventually, JavaScript, which effectively turned web pages into small virtual machines. Many of these technologies had an ad hoc feel to them, which was hardly surprising, given that they had been grafted onto a system rather than being designed into it. But they nevertheless proved extraordinarily powerful in supporting the dramatic expansion of the Web from 1995 onwards.

In pondering the Web 1.0 enterprises that had survived the crash, and the new ones that had arisen afterwards, it became clear that they had several important features in common, an observation which eventually led to them being dubbed ‘Web 2.0’ by one prominent observer of the technology (O’Reilly 2005). One of these features was that they harnessed the collective intelligence available on the Web, either via software such as Google’s PageRank algorithm (which ranks web pages using a kind of automated peer-review) or by exploiting the willingness of users to engage with the enterprise (as, for example, in Amazon’s utilisation of product reviews by customers). Another example of collective intelligence at work was Wikipedia – an enterprise made possible by Ward Cunningham’s invention of the ‘wiki’ – a web page that could be edited by anyone who read it (see Naughton 2012, 95). Cunningham’s software transformed the Web from a one-way, read-only medium, into what Tim Berners-Lee later called the ‘read–write Web’.

A second distinguishing feature of the ‘new’ Web was ‘user-generated content’ or ‘peer production’ – that is, material created and published freely by people who do it for no apparent economic motive (Benkler 2007).

Another distinctive feature of the ‘new’ Web was that many of the emerging services on it were dynamically interconnected by means of software tools like the syndication tool RSS and Application Programming Interfaces (APIs). The latter provide the ‘hooks’ on which other pieces of software can hang. What was distinctive about some of the web services that evolved after 1999 was that they used APIs to specify how entire web services could work together. A typical example is the API published by Google for its Maps service. This made it possible for people to create other services – called ‘mashups’ – which linked Google Maps with other Internet-accessible data sources.

Fourthly, many of the new Web services were distinctive by never being ‘finished’ – by being in what programmers would call a ‘perpetual Beta’ stage. This intrinsic, experimental ethos of the emerging Web was exemplified by the Google search engine which, when it launched, and for a considerable time afterwards, carried the subscript ‘BETA’. What was significant about this was that it signalled its designers’ philosophy of regarding their web-based service as a work in progress – subject to continual and sometimes rapid change – rather than as something fixed and immutable. What made this possible of course, was the fact that it was a cloud-based service, so every user’s version of the software could be upgraded at a stroke, and without any effort on their part, beyond occasionally upgrading their browser software or installing some (free) plug-ins designed to take advantage of whatever new features Google had decided to add.

A final distinguishing characteristic of the post-1999 Web was that the enterprises and services that were becoming dominant were effectively using the Web as a programming platform. So while the Internet was the platform on which Web 1.0 was built, Web 1.0 in turn became the platform on which the iconic services of Web 2.0 were constructed. This was made possible firstly by the fact that the Web provided a common standard, and secondly by the fact that if a service was provided via the http protocol, it could bypass the firewalls used by organisations to prevent unauthorised intrusions (since most firewalls were programmed to allow ‘web pages’ to pass through).

Mobile connectivity, surveillance, cybercrime, corporate power, changing patterns of use and their implications: (2004–present)

The most recent phase in the evolution of the Internet has been characterised by significant changes in the ways that people access and use the network and by the ways in which the infrastructure of the network has evolved to cope with these changes.

A comprehensive survey of these developments is beyond the scope of this paper, so an outline of some of the more significant will have to suffice. Of these, the most prominent are: the rise of mobile connectivity; the rapid expansion of so-called social media; pervasive surveillance by both state and commercial entities; increase of the power and influence wielded by a small number of large technology companies and consolidation of their grip on the network; increases in cybercrime; the possibility of ‘Balkanisation’ of the network; changes in patterns of media consumption; and the emergence of new intermediaries like Uber, Airbnb, and Coursera which use the network as a platform on which to run businesses that are potentially very disruptive to incumbents.