Pardis Noorzad

The evolution of software engineering

2026-02-20T00:00:00+00:00

Interactive. Explore the evolution of software engineering in a visual, scrollable timeline with eras and milestones from the article.

Click to open the interactive timeline.

Introduction
Foundations
Internet and Web
Cloud and infrastructure
AI coding
Discussion
Conclusion

Built by Floyd Jennings in 1946, a rotating mirror projects the outside world through lenses onto a horizontal viewing table. A decade later the building was modified to look as if it were a camera left behind by visitors. More here.

Introduction

My first exposure to AI, save a few books from Scholastic, was through several electives in college. The understanding at the time was that if we were to achieve the promises of AI, we would not merely write ordinary software but rather software that would write other software. Of course, code generation was already underway in many forms. Compilers turned high-level code into machine code. Parser generators like yacc turned a grammar into a parser. But that was all purpose-built and deterministic. AI code generation is different.

The discourse about AI today is rightfully grand. “It’s the largest technological leap we’ve seen.” “It’s bigger than the internet.” “It’s as big as smartphones.” “SaaS is so over.” This article examines seven decades of software engineering evolution, from FORTRAN to LLMs. We draw on this history to analyze where today’s AI tools fit. What changed when new paradigms arrived? What stayed the same? And what can the economic patterns of previous breakthroughs reveal about this one?

The article is organized into four eras, each built from milestones presented in roughly chronological order. We begin in 1957.

Foundations

This era established the core abstractions that programming would build on for decades.

1957. FORTRAN eliminates the need for scientists to understand computer hardware

Problem. In the mid-1950s, scientific computing required programming in assembly language. The IBM 704 had 36-bit words and three 15-bit index registers. A programmer writing code to solve differential equations needed to understand both the mathematical method and the hardware details. These included which registers to use, instruction timing, and minimizing the instruction count in inner loops.

This dual expertise created a bottleneck. Universities employed small numbers of programmers who understood both scientific problems and machine architecture. A physicist at Los Alamos might wait weeks for a programmer to translate equations into code [Met59]. The programmer might not understand the scientific context, causing errors.

In addition, programs were non-portable. Code for the IBM 704 would not run on UNIVAC. Each new machine required complete reimplementation.

Solution. In 1956, John Backus was able to convince IBM executives to fund FORTRAN. He estimated that in 1954, more than half of operating costs were programming costs, despite computers being enormously expensive. He argued that automating translation would reduce programming costs. The first FORTRAN compiler shipped in 1957 [Bac57].

FORTRAN (Formula Translation) let scientists write mathematical expressions in notation close to standard syntax. The statement Y = A * X + B directly expressed the computation without registers or memory addresses. A recurrence like the Fibonacci sequence could be written in a handful of lines:

      INTEGER I, N, F0, F1, TMP
      F0 = 0
      F1 = 1
      READ *, N
      DO 10 I = 1, N
        TMP = F1
        F1 = F0 + F1
        F0 = TMP
   10 CONTINUE
      PRINT *, F1

The compiler performed register allocation, instruction selection, and optimization. Many believed compilers could never match skilled assembly programmers. But Backus demonstrated the compiler often generated faster code than hand-written assembly by performing tedious optimizations systematically.

Within five years, most scientific computing moved from assembly to FORTRAN. Scientists became programmers. Computational fluid dynamics, molecular modeling, weather forecasting, and financial modeling all benefited. FORTRAN did not merely accelerate existing work. It made feasible work that had been almost impossible to undertake.

References

[Bac57] Backus, J. 1957. "The FORTRAN Automatic Coding System." Western Joint Computer Conference. ACM. Available at dl.acm.org

[Met59] Metropolis, N. et al. 1959. "Early Computing at Los Alamos." Annals of the History of Computing 1(1):23-34. Available at ieeexplore.ieee.org

1968. Structured programming makes programs comprehensible by constraining control flow

Problem. By 1968, program size had outpaced human comprehension. FORTRAN, COBOL, and assembly relied heavily on GOTO statements that could transfer control to any labeled statement. A program might contain hundreds of GOTOs jumping to labels scattered throughout thousands of lines.

GOTO statements made local reasoning impossible. Understanding what a program did at any point required tracing all possible execution paths from anywhere. A label on line 500 might be reached by GOTOs from lines 100, 250, 780, and 1200. The number of paths grew combinatorially with program size.

Dijkstra called this “spaghetti code” where control flow wove like tangled strands [Dij68]. By the late 1960s, commercial systems exceeded 50,000 lines and operating systems approached 100,000 lines. NATO convened a conference in 1968 to address “the software crisis” [NR69]. Programs had become too complex to understand.

Solution. Dijkstra’s “Go To Statement Considered Harmful” argued that GOTOs should be eliminated entirely [Dij68]. He proposed restricting control flow to three constructs. These were sequential execution, conditional execution (if-then-else), and iteration (while loops).

goto. Randall Munroe, xkcd.

Böhm and Jacopini proved these three constructs were sufficient to express any algorithm expressible with GOTOs [Böh66]. The restriction did not reduce expressive power. Floyd and Hoare had shown that structured constructs admitted formal reasoning (preconditions, postconditions), whereas arbitrary GOTOs did not [Flo67] [Hoa69]. Niklaus Wirth designed Pascal [Wir71] to enforce structured programming through syntax. The language had no GOTO statement. Programs could be understood by reading top to bottom, following nested control flow.

The practical effect was that software could grow. Before structured programming, systems above a certain size simply could not be understood or maintained. After it, teams could build operating systems, banking platforms, and airline reservation systems. The improvement in maintainability was consequential. Programs hundreds of thousands of lines long became possible.

References

[Böh66] Böhm, C. & Jacopini, G. 1966. "Flow Diagrams, Turing Machines and Languages with Only Two Formation Rules." Communications of the ACM 9(5):366-371. Available at dl.acm.org

[Dij68] Dijkstra, E. W. 1968. "Go To Statement Considered Harmful." Communications of the ACM 11(3):147-148. Available at dl.acm.org

[Flo67] Floyd, R. W. 1967. "Assigning Meanings to Programs." Proceedings of Symposium in Applied Mathematics 19:19-32. Available at berkeley.edu

[Hoa69] Hoare, C. A. R. 1969. "An Axiomatic Basis for Computer Programming." Communications of the ACM 12(10):576-580. Available at dl.acm.org

[NR69] Naur, P. & Randell, B. 1969. Software Engineering: Report on NATO Conference. NATO Science Committee. Available at Newcastle ePrints

[Wir71] Wirth, N. 1971. "The Programming Language Pascal." Acta Informatica 1(1):35-63. Available at link.springer.com

1970. Relational databases separate logical data organization from physical storage implementation

Problem. Through the 1960s, programs stored data in flat files with application-specific formats. Each program defined its own file structure and wrote custom parsing code. This worked for isolated applications but created problems as organizations accumulated data and needed to share it.

The fundamental issue was tight coupling. Every program accessing a customer file needed to understand the exact byte layout, and adding a new field required modifying every program that touched that data, even those not using the new field.

IBM’s Information Management System (IMS), introduced in 1966, provided hierarchical organization. But accessing data required manual navigation. To find all orders for a customer, a program traversed pointers to child records. There was no declarative way to express access patterns. Different applications wrote redundant filtering logic. When business rules changed, organizations faced updating inconsistent code across dozens of programs.

Solution. Codd’s 1970 paper “A Relational Model of Data for Large Shared Data Banks” [Cod70] proposed organizing data as mathematical relations, that is, tables with rows and columns. Each table represented an entity type, each row an instance, each column an attribute.

Rule 0 (Foundation). The system must manage databases entirely through its relational capabilities.

Rule 1 (Information). All information is represented explicitly by values in tables.

Rule 2 (Guaranteed access). Every datum is accessible by table name, primary key value, and column name.

Rule 3 (Nulls). Null values are supported for missing or inapplicable information.

Codd's 12 rules en.wikipedia.org

Codd's 12 rules, often called the "twelve commandments," define what makes a database management system fully relational. en.wikipedia.org

Codd grounded the model in set theory and predicate logic. Relational algebra and calculus were equivalent, so systems could accept declarative queries and automatically generate efficient procedural execution plans.

SQL [Cha74] provided the practical implementation of these ideas. This single statement replaced hundreds of lines of code. The database optimizer analyzed the query and generated an execution plan.

SELECT customer_name, SUM(order_amount)
FROM customers JOIN orders ON customers.customer_id = orders.customer_id
WHERE customers.state = 'CA' AND orders.order_date >= '2026-01-01'
GROUP BY customer_name HAVING SUM(order_amount) > 1000

Later, relational databases made ACID transactions [HR83] (atomicity, consistency, isolation, durability) standard. Applications could focus on business logic rather than implementing concurrency control and crash recovery.

The crucial innovation was separating logical organization from physical storage. Users worked with tables conceptually. The database system decided how to store them, what indexes to maintain, and how to organize bytes. Changing storage layout did not require modifying applications. That data independence made possible a single, shared source of truth. Multiple applications and users could read and update the same data with consistent results. These are the systems we take for granted today, from banking and reservations to inventory and ERP, where many programs depend on the same records.

References

[Cha74] Chamberlin, D. D. & Boyce, R. F. 1974. "SEQUEL: A Structured English Query Language." Proceedings of ACM SIGFIDET Workshop on Data Description, Access and Control, 249-264. Available at dl.acm.org

[Cod70] Codd, E. F. 1970. "A Relational Model of Data for Large Shared Data Banks." Communications of the ACM 13(6):377-387. Available at dl.acm.org

[HR83] Härder, T. & Reuter, A. 1983. "Principles of Transaction-Oriented Database Recovery." ACM Computing Surveys 15(4):287-315. Available at dl.acm.org

1971. Unix establishes the operating system as a portable hardware abstraction layer

Problem. Before Unix, operating systems were tightly coupled to their hardware. Software written for an IBM mainframe could not run on a DEC minicomputer. The problem was not just different instruction sets. The ways programs interacted with the system (file I/O, process creation, inter-process communication), together with device drivers, memory management, and scheduling, were all machine-specific. Moving software to new hardware meant rewriting it for a new operating environment, not just recompiling.

Solution. Thompson and Ritchie built Unix at Bell Labs between 1969 and 1971 [RT74]. Unix exposed a single interface instead of vendor-specific system calls. Programs ran as processes. The same read and write operations applied to files on disk, terminals, and devices. “Everything is a file” meant that one abstraction covered all I/O. The kernel implemented the interface in privileged mode and mediated access to hardware. Programs invoked it through system calls, so the kernel hid the details of any particular device.

That interface had two consequences. Software written to it could run on any machine running Unix, so organizations could change hardware without abandoning software. The abstraction also allowed an interpreted shell. Thompson added one that read command sequences from the terminal or from scripts and executed them. The shell turns command strings into system calls the same as any other process. Programmers could orchestrate tools with a short script instead of compiled software. For example, the following runs two compressions at once. A trailing & runs a command in the background so the next one starts right away. The shell’s wait pauses until all background commands finish.

gzip file1.log & gzip file2.log & wait

Software portability and orchestration efficiency gave organizations incentive to adopt Unix and to port the kernel to new architectures. Linux, BSD, and macOS implemented the same interface and became the principal Unix-like systems. The shell established scripted tool chains as the standard approach to orchestration. Unix-like systems dominate servers, mobile devices, and cloud infrastructure.

References

[RT74] Ritchie, D. M. & Thompson, K. 1974. "The UNIX Time-Sharing System." Communications of the ACM 17(7):365-375. Available at dl.acm.org

1973. C makes systems software like Unix portable across different computer architectures

Problem. Unix had made application software portable across machines that ran Unix. However, systems software such as kernels, device drivers, and system utilities was not portable. It was written in assembly for performance via control over memory layout, interrupts, and registers. No high-level language had shown it could match assembly for that workload. Different machines (IBM, DEC, CDC, and others) came with different instruction sets and architectures. Assembly code for one did not run on another. Porting a Unix kernel or systems stack meant a full rewrite in that machine’s assembly. Thus portability and performance seemed to be in conflict.

Solution. Dennis Ritchie developed C between 1969 and 1973 at Bell Labs to achieve portability without sacrificing performance [Rit93]. C provided pointers and low-level operations while abstracting machine-specific details. Data types such as int and char had no fixed size, so compilers could map them to each architecture. Programmers confined machine-dependent code to a small amount of assembly or conditional code. The rest was portable C. The same source could be compiled for different CPUs with minimal changes.

A program written to the Unix interface could run on any machine that ran Unix. But porting Unix to a new machine meant rewriting the kernel and utilities in that machine’s assembly. C provided a portable language for systems programming. In 1973 Thompson and Ritchie rewrote Unix in C [RT74]. Porting the Unix kernel to a new architecture was as easy as recompiling the C source and adapting a small amount of assembly, not rewriting the entire system.

Unix had made programs portable across machines that ran Unix. C made Unix portable across architectures, and made systems software written in C portable by recompilation. C performance stayed close to assembly. It became and remains the standard language for operating systems, databases, and network stacks.

References

[Rit93] Ritchie, D. M. 1993. "The Development of the C Language." ACM SIGPLAN Conference on History of Programming Languages, 201-208. Available at dl.acm.org

1970s–1980s. Object-oriented programming enforces encapsulation to manage complexity

Problem. By the late 1970s, software systems had grown to hundreds of thousands of lines. Procedural programs organized code as functions operating on global or parameter-passed data. That structure worked for small programs but failed at scale. The central issue was that data structures lived as global variables or parameters, and any function could read or modify their internals. Verifying that an invariant held, such as that account balances never went negative, required checking every function that touched the relevant data. Changing the representation of a type, such as dates, forced updates across every function that used it. With all functions in a single namespace, naming conflicts and unintended coupling were common. Architectural boundaries existed only by convention. Programmers under pressure could bypass them, and large systems tended to degrade.

Solution. Object-oriented languages such as C++ [Str85], Smalltalk, and Java addressed the encapsulation problem by making boundaries enforceable in the type system. Classes bundled data with the operations that could act on that data. Callers could use only the exposed methods. Invariants could be enforced in one place instead of by auditing every function. Inheritance and polymorphism supported reuse and abstraction without breaking encapsulation. Design patterns [GHJV94] codified recurring designs. Before OOP, keeping a large system coherent depended on every programmer respecting boundaries by discipline. After, the language enforced those boundaries. Teams could own classes, change internals without breaking callers, and build systems that could grow to millions of lines without the same collapse into unmaintainability. OOP became and remains the dominant basis for enterprise and systems software.

References

[GHJV94] Gamma, E., Helm, R., Johnson, R., & Vlissides, J. 1994. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley.

[Str85] Stroustrup, B. 1985. The C++ Programming Language. Addison-Wesley.

Internet and Web

This era saw the Internet become a common foundation and the Web the primary way software reached users.

1983. TCP/IP makes the Internet a universal network layer

Problem. Through the 1970s, computer networks had proliferated in isolation. ARPANET used its Network Control Protocol (NCP). The Xerox PARC Ethernet had different conventions. Packet radio networks, satellite networks, and local area networks each had distinct protocols for addressing, routing, and reliability. Interconnecting them required understanding each network’s quirks. A program written for one could not simply talk to another. Programmers building distributed systems had to implement compatibility layers or choose a single network and accept its limitations.

Solution Vint Cerf and Bob Kahn had laid the theoretical foundation in 1974 with “A Protocol for Packet Network Intercommunication” [CK74], which described how to interconnect dissimilar networks through gateways. The protocol split into two layers. IP (Internet Protocol) handled addressing and routing packets across networks, and TCP (Transmission Control Protocol) handled reliable, ordered delivery on top. The design was deliberately minimal. Networks kept their internal structure, and the Internet layer handled only what was necessary to pass packets between them.

On January 1, 1983, ARPANET completed the transition from NCP to TCP/IP [Pos81]. Every host on the network switched to the new protocol. The “flag day” created a single, interoperable network.

The abstraction was profound. Programmers no longer needed to understand packet switching, routing algorithms, or the differences between Ethernet and satellite links. They wrote to sockets, a simple API for sending and receiving byte streams, and the network handled the rest. TCP guaranteed delivery and ordering. IP handled addressing across any connected network. Applications could be built once and run anywhere the Internet reached.

File transfer (FTP/SFTP), email (SMTP), naming (DNS), the Web (HTTP), and every subsequent Internet application built on this foundation. TCP/IP eliminated the need to understand the network layer.

References

[CK74] Cerf, V. G. & Kahn, R. E. 1974. "A Protocol for Packet Network Intercommunication." IEEE Transactions on Communications 22(5):637-648. Available at ieeexplore.ieee.org

[Pos81] Postel, J. 1981. "NCP/TCP Transition Plan." RFC 801. Available at rfc-editor.org

1989–1993. The World Wide Web enables universal software distribution through browsers

Problem. In the 1980s, getting software to people was a logistics problem. Applications like WordPerfect and Lotus 1-2-3 were sold in boxes of floppy disks, each compiled for a specific operating system. A program for MS-DOS would not run on Mac OS or Unix. Updates required mailing new physical media to every user. Business applications accessed by multiple users followed a client-server model that required installing and maintaining software on every client machine independently. Distribution was slow, updates were painful, and every new operating system meant recompiling and repackaging from scratch.

Solution. Tim Berners-Lee proposed the Web at CERN in 1989 [BL89] as a way to share documents and files across the Internet. Its original motivation had nothing to do with software delivery. CERN’s scientific documentation was scattered across hundreds of incompatible computers. Researchers spent significant time just locating information that existed somewhere on the network. Berners-Lee wanted to link documents through hypertext so people could navigate between them without knowing where they were physically stored. By 1990, he had built the first HTTP server, the first browser, and defined HTML and URLs. Hostnames in URLs were resolved by DNS, the same naming layer the rest of the Internet already used. CERN released the protocol royalty-free in 1993 [CERN93].

The first image on the Web was a band photo at CERN in 1992 (Les Horribles Cernettes). en.wikipedia.org

The insight that made the Web transformative was universality. Any computer with a browser could access any server, regardless of operating system. Marc Andreessen and Eric Bina built Mosaic [AB93] at the National Center for Supercomputing Applications, the first graphical browser. As adoption grew, programmers saw the implication. Applications could be hosted on a server rather than shipped on floppy disks. A single deployment to the server made the new version available to every user, eliminating the need to mail updated media to customers. Initially that meant downloading software from the Web. Netscape Navigator was distributed that way, as were Winamp, RealPlayer, and countless early desktop applications. The pattern persists today. Zoom, VS Code, and most desktop and mobile installers are still distributed by download from a website or app store.

Making software run inside the browser, not just be downloaded from it, took two more steps. JavaScript, created by Brendan Eich at Netscape, was the next necessary piece. Static HTML pages couldn’t respond to user input without sending a request back to the server and reloading the entire page. JavaScript ran directly in the browser, so a form could validate input before submission, a button could trigger an action, a page could change without disappearing and reappearing. Web pages started feeling less like documents and more like applications. The shift completed with AJAX. Jesse James Garrett named the pattern [Gar05] that programmers had already begun using. Applications sent requests in the background and updated only the changed parts of the page instead of reloading. Gmail (2004) proved this worked at scale. An entire productivity application ran in the browser, feeling as responsive as desktop software. The Web had evolved from a tool for sharing scientific documents into the primary platform for delivering software to users.

References

[AB93] Andreessen, M. & Bina, E. 1993. "NCSA Mosaic: A Global Hypermedia System." Internet Research 3(1). Available at emerald.com

[BL89] Berners-Lee, T. 1989. "Information Management: A Proposal." CERN. Available at w3.org

[CERN93] CERN. 1993. "CERN Puts Web into Public Domain." Available at cern.ch

[Gar05] Garrett, J. J. 2005. "Ajax: A New Approach to Web Applications." Available at MIT. See also Garrett's 2025 reflection.

1991. Python becomes the default for scripting, automation, and data science

Problem. The dominant assumption at the time was that runtime performance mattered most. C and FORTRAN optimized for execution speed. The opposite insight was that programmer time is more expensive than machine time. Most code runs once or rarely (scripts, glue code, prototypes). The cost of writing, debugging, and maintaining it dwarfs execution time. A language that made the common case fast to write, even if slow to run, would win.

Solution. Guido van Rossum released Python in 1991 [Pyt91]. Python prioritized readability and ease of use over raw performance. It required no compile step, used clear syntax, and would become “batteries included” as its standard library grew. The crucial design choice was extensibility. When a hot path needed speed, programmers could drop into C. NumPy (2006) demonstrated the pattern. Python for glue code and control flow, C (via extensions) for the numerical inner loops. Programmers got productivity for the 95% of code that wasn’t performance-critical, and C-level speed where it mattered. pandas, Django, TensorFlow, and PyTorch followed the same model. Python became the default for data science, ML, and glue code because it optimized for the right variable (programmer time).

Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one obvious way to do it.

PEP 20, Zen of Python peps.python.org

Guiding principles for Python design. peps.python.org

References

[Pyt91] Python Software Foundation. "History of Python." Available at python.org

1994–1998. Standard algorithm libraries make common algorithms and data structures reusable

Problem. Programmers implementing a sorted collection had to build their own balanced tree or settle for a slower linked list. Those needing O(log n) lookup implemented a red-black tree. Those needing to sort implemented quicksort or merge sort. These implementations were subtle. Off-by-one errors, edge cases with empty collections, and incorrect handling of equal elements were common. Every team duplicated the same work, and bugs in algorithm implementations were hard to detect because the logic was buried in application code.

Algorithm theory (Big O notation, complexity analysis) had given programmers a vocabulary for reasoning about performance, but it did not eliminate the need to implement. The gap between theory and practice remained.

Solution. Alexander Stepanov and Meng Lee developed the Standard Template Library (STL) for C++ at Hewlett-Packard in 1994 [Step94]. The design separated containers, iterators, algorithms, and functors. The key insight was generic programming. Algorithms were written once in terms of iterators and worked with any container. std::sort worked on a vector, a queue, or a custom container, as long as it provided random-access iterators. HP released the STL freely. It was incorporated into the C++ standard and shipped with every C++ compiler.

In 1998, Java followed with the Collections Framework in JDK 1.2 [Bloch01]. List, Set, Map, and their implementations (ArrayList, HashMap, TreeMap) became part of the standard library, with interfaces for sorting, searching, and bulk operations. Like the STL, it provided complexity guarantees. Donald Knuth had established the theoretical basis in The Art of Computer Programming, from 1968 onwards [Knuth68]. The STL and Java Collections made those guarantees practical. Programmers chose by need (sorted, O(1) lookup, ordered iteration) and the library supplied a correct implementation. The pattern spread to Python’s list, dict, and set, C#’s System.Collections, and Rust’s std::collections. Algorithms and data structures became part of the standard toolkit. Programmers used them without implementing them.

References

[Step94] Stepanov, A. & Lee, M. 1994. "The Standard Template Library." Hewlett-Packard. Available at stepanovpapers.com

[Bloch01] Oracle. 1998. "Java Collections Framework." JDK 1.2. Design by J. Bloch. See Java Collections Design FAQ.

[Knuth68] Knuth, D. E. 1968. The Art of Computer Programming, Volume 1. Fundamental Algorithms. Addison-Wesley. Subsequent volumes from 1969 onwards.

1995. Garbage collection makes entire categories of memory errors impossible

Problem. Through the early 1990s, most commercial software was written in C and C++ requiring manual memory management. Programmers explicitly allocated memory with malloc() or new and deallocated with free() or delete. Getting the pairing wrong led to memory leaks, use-after-free, double-free, and corruption. Memory leaks consumed all available memory in long-running programs. Use-after-free errors occurred when code freed memory but later accessed it through a dangling pointer, often causing data corruption. Double-free errors corrupted the allocator’s internal structures. These bugs were insidious because they might not manifest during testing but caused failures only after days of production operation.

The consequences were real. BlueKeep (2019), a use-after-free in Windows RDP, let attackers execute arbitrary code with kernel privileges over the network without authentication. The NSA issued a rare advisory. Microsoft patched even end-of-life systems. Microsoft estimated 70% of their 2006–2018 security vulnerabilities were memory-safety issues [MSRC19].

Solution. Java, released in 1995 [Gos96], popularized garbage collection for mainstream commercial development. Java’s innovation was demonstrating that automatic memory management was practical despite performance overhead.

Java eliminated manual deallocation. Programmers allocated objects with new but never freed them. The garbage collector, a background process that ran as part of the Java runtime during program execution, periodically scanned the heap to identify objects that were no longer reachable from any live variable or reference, and reclaimed their memory. Because this happened automatically at runtime, programmers could not introduce use-after-free or double-free bugs. The entire class of dangling pointer errors disappeared by construction.

The idea of automatic memory reclamation was not new. John McCarthy’s Lisp in 1960 [McC60] included the first garbage collector. But early collectors were too slow for commercial systems. The breakthrough was generational collection, developed in the 1980s [Lie83]. Most objects die young, so collecting younger generations frequently and older ones rarely reduced overhead enough that GC became viable for production. That made Java’s approach credible when it launched.

The performance overhead was non-zero but acceptable. Early collectors had pause times of seconds. Improvements reduced pauses to milliseconds for typical applications. An entire category of serious bugs disappeared. Garbage collection also simplified concurrent programming. Threads could share object references without complex deallocation coordination.

Following Java’s success, garbage collection became standard in new languages. C#, Go, JavaScript, Python, and Ruby all adopted it. Rust (2015) took a different approach. Its ownership and borrowing model, a zero-cost abstraction, enforces memory safety at compile time with no runtime overhead or GC pauses [Jun18].

References

[Gos96] Gosling, J., Joy, B., & Steele, G. 1996. The Java Language Specification. Addison-Wesley. Available at docs.oracle.com

[Jun18] Jung, R., et al. 2018. "RustBelt: Securing the Foundations of the Rust Programming Language." Proceedings of POPL, 66:1-66:34. Available at dl.acm.org

[Lie83] Lieberman, H. & Hewitt, C. 1983. "A Real-Time Garbage Collector Based on the Lifetimes of Objects." Communications of the ACM 26(6):419-429. Available at dl.acm.org

[McC60] McCarthy, J. 1960. "Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I." Communications of the ACM 3(4):184-195. Available at dl.acm.org

[MSRC19] Microsoft Security Response Center. 2019. "A Proactive Approach to More Secure Code." Available at msrc-blog.microsoft.com

1995–2010. Package managers make dependency management automatic

Problem. Before package managers, reusing code meant finding it, downloading it, manually placing it in your project, and ensuring it worked with other dependencies. Version conflicts were discovered only when builds failed. There was no central registry, no automated resolution. Dependency management was manual and error-prone.

Solution. CPAN, launched in 1995 for Perl, established the pattern. It provided a central archive, a standard layout for how modules were packaged, and a tool that installed modules and dependencies with a single command. Maven, released in 2004, brought the same approach to Java with declarative pom.xml files. npm, launched in 2010, did the same for Node and became the largest package ecosystem in history. The abstraction was declarative. Programmers specified what they needed, not how to get it. The same pattern spread to Python (pip), Ruby (RubyGems), and virtually every language. Dependency management became part of the standard toolkit.

1998. Open source makes collaborative, publicly developed software the default

Problem. Through the 1990s, most software was proprietary. Companies kept source code secret to protect their competitive advantage. Reusing code meant licensing it or rewriting it. Richard Stallman had founded the free software movement (GNU, GPL) and framed it as “free as in free speech, not free beer.” That established legal and ethical foundations, but “free” carried political baggage that made businesses hesitant. Linux and Apache had proven that open collaboration could produce production-grade software, yet there was no neutral term that invited broad adoption. Programmers who wanted to share code faced a fragmented landscape of licenses and ideologies.

Solution. In 1998, Christine Peterson coined the term “open source,” and Bruce Perens and Eric S. Raymond founded the Open Source Initiative (OSI) [OSI98]. The shift from “free software” to “open source” was deliberate. It emphasized practical benefits (peer review, faster iteration, no vendor lock-in) over the freedom-first stance that Stallman had championed. The OSI defined criteria for open source licenses and certified them. Apache, MIT, and GPL became mainstream choices rather than ideological statements.

The model proved itself. Linux, Apache, MySQL, PHP (LAMP) powered the early web. Firefox challenged Internet Explorer. Android was built on Linux. Companies from Google to Microsoft adopted open source as strategy. GitHub (2008) reduced contribution friction (fork, change, pull request). By the 2010s, open source was the default for infrastructure, frameworks, and tools. Programmers no longer needed to build or buy everything. They could adopt, adapt, and contribute back. The abstraction was organizational. The efficiency gain was communal. The best software in the world was built collaboratively, in public, by anyone who cared to participate.

References

[OSI98] Open Source Initiative. "History of the OSI." Available at opensource.org

2000. REST APIs standardize how web services communicate

Problem. Business-to-business commerce and supply-chain integration required one organization’s applications to talk to another’s over the Internet. Existing distributed computing (CORBA, DCOM, Java RMI) was vendor-specific, complex, and did not work across organizational boundaries. HTTP could carry requests and XML could encode data. What was missing was an agreed way to structure a call. How should one program ask another for a record or submit an update? Without a standard, every service invented its own. SOAP (Microsoft, IBM, W3C 2000) proposed one approach. It was heavyweight (XML envelopes, WSDL, WS-*). No lightweight alternative existed.

Solution. Roy Fielding [Fie00], a co-author of HTTP/1.1, had helped design the Web’s protocol. In his 2000 doctoral dissertation he named and formalized the architectural style already present in the Web. Resources were identified by URLs. The interface was the HTTP methods (GET, POST, PUT, DELETE). Requests were stateless. He called this style REST. He was not inventing a new protocol. He was documenting what had made the Web scale. That gave programmers a clear model for designing application programming interfaces (APIs). To get a customer, use GET on a URL. To create one, use POST. No XML envelope, no WSDL. When JSON replaced XML as the preferred format, REST with JSON was easy to use and to test in a browser. Public APIs that let external developers access a site’s data (Amazon, Google, and others) adopted REST. By the 2010s, REST had become the default way machines talked to each other on the Web.

References

[Fie00] Fielding, R. T. 2000. "Architectural Styles and the Design of Network-based Software Architectures." Doctoral dissertation, University of California, Irvine. Available at ics.uci.edu

2001. IDEs automate the mechanical scaffolding of programming, an early step toward code generation

Problem. As Java and enterprise applications grew into systems of hundreds of thousands of lines, the mechanical overhead of programming became a significant drag on productivity. Adding a new method to a class meant manually hunting through dozens of files to find every call site that needed updating. Renaming a class required running grep across the entire codebase and editing each hit by hand. Finding where a method was actually defined meant navigating through directories of source files. These tasks required no deep thought. They were purely mechanical, but they consumed hours each day.

Solution. IntelliJ IDEA (2001) [Jet01] and Eclipse (2001, first release 2004) [Ecl01] represented a generational leap in development tools. They parsed entire codebases and built an internal model of every class, method, and reference. This let them provide intelligent code completion. As a programmer typed, the IDE suggested valid method names and parameter types. Automated refactoring made operations like renaming a class or extracting a method into a single action that propagated correctly across the entire project. Integrated debugging let programmers step through code without leaving the editor. Visual Studio provided similar capabilities for C# and .NET development. Design patterns (Gamma et al., 1994) [GHJV94] had codified common OOP solutions. Refactoring (1999) [Fow99], JUnit (1997) [Jun97], and test-driven development (TDD) made restructuring and automated testing mainstream practices.

The productivity gains were substantial. Operations that previously required manual searching and editing across a codebase became instantaneous. By the mid-2010s, IDEs had become so essential that programmers who worked without one felt as disadvantaged as those without version control.

References

[Ecl01] Eclipse. 2001. "Eclipse IDE." Available at eclipse.org

[Fow99] Fowler, M., Beck, K., Brant, J., Opdyke, W., & Roberts, D. 1999. Refactoring: Improving the Design of Existing Code. Addison-Wesley. Available at martinfowler.com

[Jet01] JetBrains. 2001. "IntelliJ IDEA." Available at jetbrains.com

[Jun97] Beck, K. & Gamma, E. 1997. JUnit. Available at junit.org

2002. Dependency injection frees enterprise programmers from framework boilerplate

Problem. Java 2 Enterprise Edition (J2EE) and Enterprise JavaBeans (EJBs) were the standard platform for enterprise Java in the early 2000s. J2EE was the platform. EJBs were the component model for server-side business logic, objects that ran in a container and handled transactions and persistence. In practice it required extensive XML, deployment descriptors, and boilerplate just to wire objects together. A simple database service might need dozens of config files and hundreds of lines of scaffolding. Objects created their own dependencies. Testing and swapping implementations required rewriting wiring code throughout. Programmer time went to infrastructure, not features.

Solution. Spring (2003) [Spr03] replaced J2EE’s heavy wiring with dependency injection. Rod Johnson’s 2002 book [Joh02] had argued that Plain Old Java Objects (POJOs) and a lightweight container could replace EJBs, and Spring implemented that idea. Instead of objects creating their own dependencies, a container created and injected them, so a class that needed a database connection could simply declare the dependency and Spring would provide it. Testing became straightforward because tests could inject mocks. Wiring became explicit and centralized rather than scattered throughout the codebase.

J2EE had established the enterprise Java market but created the complexity Spring addressed. Practitioner-built frameworks could outcompete committee-designed standards. Spring Boot (2014) took the next step by providing sensible defaults and auto-configuration, so programmers could start a Spring application with minimal or no config files. By the mid-2010s, dependency injection had become standard across languages and frameworks.

References

[Joh02] Johnson, R. 2002. Expert One-on-One J2EE Design and Development. Wrox. Available at wiley.com

[Spr03] Spring. 2003. "Spring Framework." Available at spring.io

Cloud and infrastructure

This era saw cloud computing, mobile, big data, and the commoditization of previously specialized infrastructure.

2004–2009. MapReduce and Hadoop make processing massive datasets accessible

Problem. By the early 2000s, companies like Google were crawling and indexing billions of web pages. The sheer volume of data dwarfed what any single machine could store or process. Google solved this internally by building the Google File System (GFS) in 2003 [GGL03], a distributed file system that spread data across hundreds or thousands of commodity servers, and MapReduce in 2004 [DG04]. MapReduce was a programming model that let programmers express massively parallel computation in a simple way. A Map function processed individual records and a Reduce function aggregated results. The framework handled distributing work, shuffling data, and recovering from failures. Google published papers but did not release the code.

Solution. Doug Cutting and Mike Cafarella had started Nutch, an open-source web crawler, in 2002. When Google’s GFS and MapReduce papers appeared, they implemented the techniques in Nutch but needed institutional backing. Yahoo hired Cutting in 2006 to build distributed data processing for its search engine. He extracted the distributed file system and MapReduce implementation from Nutch into a new project, Hadoop. Hadoop comprised HDFS (the file system) and MapReduce (the processing framework). The same name, MapReduce, was intentional. It implemented the same model from the Google papers. Yahoo dedicated a large team to developing it. By 2007, Yahoo was running Hadoop on a 1,000-node cluster.

Yahoo open-sourced its Hadoop work in 2009, ran Hadoop at scale, and adoption followed quickly. Facebook ran Hadoop and built Presto for interactive SQL, Twitter built Scalding (a Scala API on Cascading and Hadoop MapReduce), and LinkedIn built Kafka for event streaming [KNR11]. eBay and others adopted the ecosystem, and eventually more than half of the Fortune 500 ran big data pipelines on open-source tools. Spark emerged in 2009 [Zah10] as an alternative to MapReduce that kept intermediate data in memory rather than writing it to disk, improving performance for iterative workloads while requiring more memory. Kafka became the de facto standard for event streaming, Flink (2014) [Car15] offered true stream processing at lower latency than Spark’s micro-batch model, and Tez optimized batch DAGs for Hadoop. Big data processing went from out of reach to something any company could run.

References

[Car15] Carbone, P., et al. 2015. "Apache Flink: Stream and Batch Processing in a Single Engine." IEEE Data Engineering Bulletin 36(4):28-38. Available at arxiv.org

[DG04] Dean, J. & Ghemawat, S. 2004. "MapReduce: Simplified Data Processing on Large Clusters." Proceedings of OSDI, 137-150. Available at usenix.org

[GGL03] Ghemawat, S., Gobioff, H., & Leung, S.-T. 2003. "The Google File System." Proceedings of SOSP, 29-43. Available at research.google

[KNR11] Kreps, J., Narkhede, N., & Rao, J. 2011. "Kafka: A Distributed Messaging System for Log Processing." Proceedings of NetDB. Available at engineering.linkedin.com

[Zah10] Zaharia, M., et al. 2010. "Spark: Cluster Computing with Working Sets." Proceedings of HotCloud. Available at usenix.org

2005. Git enables distributed collaboration at global scale

Problem. For the first decade of Linux kernel development (1991–2002), there was no formal version control at all. Contributors emailed patches to mailing lists, and Linus Torvalds manually applied them to his own source tree before cutting releases. This worked when the project was small, but Linux had grown into the most important open-source project in the world, with thousands of contributors. The manual process became a serious bottleneck.

In 2002, Torvalds adopted BitKeeper, a proprietary distributed system that was far ahead of CVS or Subversion. In early 2005, BitMover revoked the free license and the kernel community lost its version control overnight.

Solution. Torvalds had spent months considering what kernel development required. CVS and Subversion were centralized, which made cheap branching and offline work impossible, and no open-source alternative was mature. He began writing Git on April 3, 2005, and had a working system within roughly 10 days. The design was fully distributed. Every clone contained the complete repository history, which allowed programmers to commit, branch, and merge locally without network access. Branching became a lightweight operation, a pointer to a commit that made it essentially free. The Linux kernel 2.6.12 release in June 2005 was the first managed entirely by Git.

Git-based workflows later enabled continuous integration and deployment. Jenkins (2011) and Travis CI (2011) automated testing and deployment pipelines. Programmers pushed code to Git repositories, triggering automated builds, tests, and deployments. GitHub launched in 2008 [Dab12], adding pull requests and code review workflows that made open-source collaboration frictionless. The model enabled global collaboration at unprecedented scale. Projects like Linux, with thousands of contributors across continents, could coordinate effectively. This DevOps movement reduced the time between writing code and running it in production from weeks to minutes.

References

[Dab12] Dabbish, L., et al. 2012. "Social Coding in GitHub." Proceedings of CSCW, 1277-1286. Available at dl.acm.org

2006. Cloud platforms transform infrastructure into elastic, pay-per-use resources

Problem. Before 2006, running applications meant purchasing servers, networking equipment, and storage, renting rack space and power in a data center, and hiring system administrators to maintain all of it. For a startup launching a web service, the upfront capital was substantial. Ordering, installing, and configuring new hardware took weeks or months. Capacity planning made this worse. Organizations had to forecast future demand and either overprovision and pay for idle capacity or underprovision and risk outages. Spiky workloads, such as retail at holidays or tax software in filing season, made the tradeoff brutal.

Solution. Amazon Web Services launched Elastic Compute Cloud (EC2) in August 2006 [AWS06], providing virtual servers provisionable through an API in minutes with pay-per-hour billing. This transformed infrastructure from capital expenditure (CapEx) to operational expense (OpEx) and from static to elastic. EC2 is the foundational example of what the industry came to call Infrastructure as a Service (IaaS). The cloud provider manages physical hardware, networking, and virtualization, while the customer retains responsibility for operating systems, applications, and data. The customer rents compute, storage, and network capacity rather than purchasing it.

This transformed capacity planning. Organizations could scale elastically to match current demand. The cloud model expanded in layers. PaaS (Heroku, Google App Engine, Elastic Beanstalk) shifted OS and runtime management to the provider, so programmers could deploy applications without configuring servers. SaaS (Salesforce, Gmail, Dropbox) delivered entirely managed applications. Infrastructure as code (Chef, Puppet, Terraform) automated the provisioning of IaaS resources through version-controlled scripts, replacing weeks of manual setup. AWS operated data centers worldwide. Startups could deploy globally with the same API calls. Following AWS’s success, Azure and Google Cloud emerged. Cloud computing became the dominant deployment model.

References

[AWS06] Amazon Web Services. 2006. "Announcing Amazon Elastic Compute Cloud (Amazon EC2)." Available at aws.amazon.com

2007. Mobile platforms turn the phone into a general-purpose computer with app ecosystems

Problem. At its peak, Nokia controlled over 40% of the global mobile phone market. Within six years, that share had collapsed to under 5%. Hardware was not the issue. Nokia’s model treated phones as closed appliances. SDKs were fragmented (J2ME on some devices, proprietary on others), there was no unified channel for programmers to distribute software to users, and Symbian was not built for a phone as a general-purpose computer running third-party software.

Solution. Apple released the iPhone in June 2007. The multitouch screen and full web browser were significant, but the deeper change was conceptual. The iPhone was positioned as a general-purpose computing device that also made calls, with a browser that rendered full web pages rather than a stripped-down mobile experience. The iPhone SDK launched in March 2008 [App08] and the App Store opened in July 2008. For software distribution, Apple provided a single channel. Programmers could submit apps and reach millions of devices without going through carriers or OEMs. Google followed with Android in September 2008 [And08] and the Android Market, with a more permissive review process. Both platforms provided high-level APIs and enabled instant global distribution. By the mid-2010s, mobile had created new categories such as ridesharing, mobile payments, and social photography, and changed how software was discovered, distributed, and monetized.

References

[App08] Apple. 2008. "iPhone SDK Announcement." Available at apple.com

[And08] Android Open Source Project. 2008. "Android Platform Overview." Available at source.android.com

2008–2012. Microservices replace monoliths as the architecture for large-scale applications

Problem. Large web companies in the late 2000s built their platforms as monolithic applications, that is, large codebases deployed as one unit. Early Netflix illustrates the pattern. Its core system was a Java application backed by an Oracle database. In August 2008, a hardware failure took the entire service down for three days. The cause was initially suspected to be database corruption. Every part of the system depended on the same database, so a failure in one place propagated everywhere. Such failures are inherent to monolithic architecture.

Beyond availability, monoliths created organizational bottlenecks. Because the application was a single deployment unit, teams working on different features, such as recommendations, billing, and streaming playback, had to deploy together. A bug in one component could take down the whole process and break unrelated features. Because the codebase had no service boundaries, adding a feature required understanding and testing the entire application. Because all components ran in the same process, scaling meant adding more copies of the entire application. Provisioning more compute for one component required scaling everything, wasting capacity on components that needed none. As the codebase grew, development slowed and onboarding became increasingly difficult.

Solution. Amazon arrived at the architecture first [Vog22]. Its monolithic e-commerce platform became unmanageable as it expanded. Architects required every internal capability to be exposed as an independent service. That restructuring produced the infrastructure that became AWS. Netflix began migrating to that model on AWS in 2009, a seven-year process. The core idea was to break the monolith into small, independently deployable services, each owning its own database. A failure in one service no longer took down the whole platform. Netflix eventually decomposed into over 700 microservices. At that scale, services must find each other, handle failures gracefully, and distribute load across instances. Netflix open-sourced its operational tooling as Eureka (service discovery), Hystrix (circuit breaker), and Ribbon (load balancing) [Net12]. Fowler and Lewis gave the pattern its name and a widely cited reference in 2014 [Mar14]. By the 2020s microservices had become the dominant architecture for large-scale web applications, adopted by most large enterprises.

References

[Mar14] Fowler, M. & Lewis, J. 2014. "Microservices." Available at martinfowler.com

[Net12] Netflix. 2012. "Netflix Shares Cloud Load Balancing And Failover Tool: Eureka!" Available at netflixtechblog.com

[Vog22] Vogels, W. 2022. "The Distributed Computing Manifesto." All Things Distributed. Available at allthingsdistributed.com

2009. NoSQL databases trade consistency for scale and flexibility

Problem. By the late 2000s, web-scale data and traffic exceeded what relational databases could handle. Horizontal scaling required ACID across partitions and two-phase commit, which did not scale. The CAP theorem [Bre00] formalized the tradeoff. Relational databases chose consistency and became unavailable during partitions. Fixed schemas forced sparse tables or many joins for heterogeneous data. Schema changes required migrations that locked tables. Row-oriented storage made analytical scans expensive. Relational full-text search did not scale.

Solution. NoSQL databases relax relational constraints in exchange for scale. Different designs addressed different constraints.

Relational databases had required two-phase commit across partitions and went offline when partitions occurred. Two designs from 2006–2007 showed that abandoning ACID across partitions enabled horizontal scaling. They chose opposite sides of the CAP tradeoff.

BigTable [CDG+06] (Google, 2006) chose consistency. It used a sparse, multi-dimensional sorted map. Data was organized in column families, groups of columns stored together within each row, so each row could have different columns. Heterogeneous data such as web crawl records with varying fields per URL no longer required sparse tables or many joins. It offered strong consistency within a row and suited read-heavy workloads such as Google’s search index and Google Maps. That consistency required a central coordinator, at the cost of availability. BigTable’s design was adopted in the open-source Apache HBase.

Dynamo [DHJ+07] (Amazon, 2007) chose availability. It stayed writable during partitions when relational systems went offline. Its key-value model had no central coordinator. It suited shopping carts and session data, where availability mattered more than immediate consistency. Dynamo’s design was adopted in Amazon’s DynamoDB (commercial) and Riak (open-source).

Document stores addressed sparse data and schema evolution. Fixed schemas had forced migrations that locked tables. MongoDB (2009) [Mon09] and CouchDB stored JSON-like documents. Applications could add fields without migrations. The model suited user profiles with varying attributes, product catalogs with nested specifications, and content with arbitrary metadata. The tradeoff was eventual consistency and no ACID across documents. Cassandra (2008) [LM10] combined BigTable’s column-family model with Dynamo’s decentralized distribution. It handled sparse data, scaled horizontally, and offered tunable consistency. Use cases included activity feeds and time-series data. The cost was eventual consistency by default.

MongoDB is web scale (parody). youtube.com

Columnar stores and search engines addressed analytical scans and full-text search. Row-oriented storage had made broad aggregations slow. Vertica and ClickHouse stored each column separately, so scans could read only the columns needed for aggregations. They suited analytical dashboards, sales reports, and click analytics (OLAP) but were poor for transactional point updates (OLTP). Elasticsearch (2010) [Ban10] and Solr, built on Lucene, provided full-text search over HTTP for product search, log analysis, and site search. NoSQL made web-scale storage reachable without purpose-built hardware or specialist teams.

References

[Bre00] Brewer, E. 2000. "Towards Robust Distributed Systems." Proceedings of ACM PODC. Available at researchgate.net

[CDG+06] Chang, F., Dean, J., Ghemawat, S., et al. 2006. "Bigtable: A Distributed Storage System for Structured Data." Proceedings of OSDI, 205-218. Available at research.google

[DHJ+07] DeCandia, G., Hastorun, D., Jampani, M., et al. 2007. "Dynamo: Amazon's Highly Available Key-value Store." Proceedings of ACM SOSP, 205-220. Available at dl.acm.org

[LM10] Lakshman, A. & Malik, P. 2010. "Cassandra: A Decentralized Structured Storage System." ACM SIGOPS Operating Systems Review 44(2):35-40. Available at dl.acm.org

[Ban10] Banon, S. 2010. "You Know, for Search." Elasticsearch. Available at elastic.co

[Mon09] Chodorow, K. & Dirolf, M. 2010. MongoDB: The Definitive Guide. O'Reilly Media. Available at oreilly.com

2009. Node.js makes JavaScript full-stack and enables the npm ecosystem

Problem. By the late 2000s, web applications had a split identity. The browser ran JavaScript. The server ran Java, PHP, Python, or Ruby. Programmers wrote frontend and backend in different languages, with different runtimes and toolchains. Building a real-time feature meant WebSockets on the server and JavaScript in the browser. Full-stack development meant context-switching between languages, deployment targets, and debugging environments. Programmer time was spent on integration friction, not features. There was no way to share code reliably between client and server.

Solution. Ryan Dahl released Node.js in 2009 [Dah09]. Node.js ran JavaScript on the server using Google’s V8 engine, the same one powering Chrome. The key innovation was non-blocking I/O. Instead of threads, it used an event loop. A single process could handle thousands of concurrent connections. This suited I/O-bound workloads such as APIs, proxies, and real-time applications that had dominated server-side scaling challenges.

Node.js made JavaScript full-stack. Programmers could write client and server in the same language. npm, launched in 2010, became the package registry for Node and the browser. The same require or import worked on both sides. Rails (2004) and Django (2005) had popularized convention-over-configuration, relying on sensible defaults (e.g. a Post model maps to a posts table) instead of explicit configuration for every detail. Node.js and frameworks like Express (2010) brought the same model to JavaScript. The ecosystem expanded rapidly. By the mid-2010s, Node.js powered Netflix, LinkedIn, Uber, and PayPal. JavaScript went from a browser scripting language to the most widely used language for web development. One language and one ecosystem spanned the stack from end to end.

References

[Dah09] Node.js. "About Node.js." Available at nodejs.org

2010–2015. Type safety, component architecture, and safer concurrency reach mainstream development

Problem. Dynamic languages such as JavaScript, Python, and Ruby had become central to web and backend development, but type checking and clear separation of concerns lagged. Dynamic typing deferred errors to runtime that static typing would have caught at compile time. In JavaScript, jQuery-based applications entangled DOM manipulation, business logic, and data fetching with no clear separation.

Concurrent programming in Java, C++, and similar languages faced a separate set of challenges. Mutable shared state and race conditions produced bugs that were difficult to reproduce and debug. Locks offered a remedy but introduced deadlocks and contention, and correctness depended on precise lock ordering that most programmers found impractical to maintain.

Solution. TypeScript (2012) [Mic12] and React (2013) [Fac13] added type checking and component architecture to JavaScript, transforming jQuery spaghetti into structured development.

Functional concepts entered mainstream languages. Scala bridged object-oriented and functional programming on the JVM, making functional ideas accessible to Java programmers. Twitter’s adoption of Scala for high-concurrency systems [Eri12] demonstrated that type-safe functional programming could handle production scale. Java 8 (2014) [Ora14] followed with lambdas and streams, bringing functional patterns to the mainstream. Immutable data and pure functions addressed concurrency without locks.

References

[Eri12] Eriksen, M. et al. 2012. "Effective Scala." Twitter Engineering. Available at twitter.github.io/effectivescala

[Fac13] Facebook. 2013. "React: A JavaScript Library for Building User Interfaces." Available at react.dev

[Mic12] Microsoft. 2012. "Introducing TypeScript." Available at devblogs.microsoft.com

[Ora14] Oracle. 2014. "What's New in Java SE 8." Available at docs.oracle.com

2013–2014. Containers and orchestration make deployment portable and scalable

Problem. Two constraints slowed deployment. The first was environment inconsistency. Applications ran in development but crashed in production due to different library versions, missing dependencies, or configuration drift. Virtual machines provided isolation but were heavyweight. Each VM required a full OS and consumed gigabytes. Manual configuration and documentation were brittle.

The second constraint was orchestration at scale. Distributing workloads across clusters required placement decisions, failure handling, traffic routing, and elastic scaling. Manual coordination did not scale. Cluster managers such as Apache Mesos offered resource scheduling but did not provide declarative desired-state configuration, automated rollouts and rollbacks, integrated service discovery, or self-healing of failed workloads.

Solution. Docker (2013) [Mer14] addressed environment inconsistency. Linux containers packaged applications into images that ran identically anywhere. A Dockerfile replaced manual setup. Containers were lightweight compared to VMs and required no full OS per instance.

Kubernetes (2014) [Bur+16], based on Google’s Borg, addressed orchestration at scale. Programmers declared desired state in YAML, for example replicas: 10 and containerPort: 80, and Kubernetes reconciled the cluster to match. The system restarted crashed instances, scaled in response to load, and exposed infrastructure as declarative YAML stored in version control. Cloud providers offered managed Kubernetes. Within a decade, containers and Kubernetes had become the standard for cloud-native deployment that scales and recovers across many machines.

References

[Mer14] Merkel, D. 2014. "Docker: Lightweight Linux Containers for Consistent Development and Deployment." Linux Journal 2014(239):2. Available at linuxjournal.com

[Bur+16] Burns, B., Grant, B., Oppenheimer, D., Brewer, E., & Wilkes, J. 2016. "Borg, Omega, and Kubernetes." ACM Queue 14(1):70-93. Available at dl.acm.org

2014. Serverless computing shifts the unit of deployment from servers to functions

Problem. Containers and orchestration made deployment portable and scalable, but containers were always-on. A workload handling one request per hour still required a running container and continuous compute cost. For bursty or infrequent workloads, organizations paid for idle capacity, the same inefficiency that cloud computing had aimed to eliminate.

Solution. AWS Lambda, launched in November 2014 [AWS14], introduced Function-as-a-Service, or serverless. The platform invoked functions only when triggered by events such as HTTP requests, file uploads, queue messages, or scheduled jobs. Each execution was ephemeral. The runtime was torn down afterward, so no compute was allocated between runs and pricing was per-invocation and per-duration. Cost aligned with actual usage rather than provisioned capacity, eliminating idle cost for bursty or infrequent workloads. Lambda scaled automatically from zero to thousands of concurrent executions.

The stateless model imposed constraints. Cold starts introduced latency after inactivity, and execution time was capped, so serverless suited event-driven workloads such as APIs, background processing, data pipelines, and scheduled tasks but was not a universal replacement for containers. Google Cloud Functions and Azure Functions followed. Serverless became a standard deployment option alongside IaaS and containers, selected by workload characteristics.

References

[AWS14] Amazon Web Services. 2014. "Announcing AWS Lambda." Available at aws.amazon.com

2015–2016. ML frameworks democratize machine learning without research-level expertise

Problem. Before 2015, applying machine learning meant implementing algorithms from academic papers and using tools like R and Python that required statistical expertise. Deep learning had emerged as a research direction, but implementing backpropagation, designing architectures, and training at scale demanded deep knowledge of linear algebra, optimization, and distributed systems. Google, Facebook, Twitter, and a few labs built their ow internal frameworks. They either had to hire PhDs or stay out. The gap between “research breakthrough” and “programmer can use it” was enormous. The bottleneck was expertise, not compute.

Solution. TensorFlow (2015) [Aba+15] and PyTorch (2016) [Pas+17] made deep learning tractable for programmers without research-level expertise. Both provided backpropagation, GPU acceleration, and a high-level API. Programmers defined computation as a graph or in imperative code. The frameworks handled the math, learning-rate tuning, and distributed training across GPUs. Both integrated with NumPy, pandas, and Jupyter. Transfer learning allowed fine-tuning of pretrained models with minimal data. Scikit-learn had already made classical ML such as regression, classification, and clustering accessible. TensorFlow and PyTorch did the same for deep learning, and the language models behind AI coding assistants such as Copilot and Codex were trained with these frameworks [CKB+21].

References

[Aba+15] Abadi, M., et al. 2016. "TensorFlow: A System for Large-Scale Machine Learning." Proceedings of OSDI, 265-283. (Released 2015.) Available at usenix.org

[Pas+17] Paszke, A., et al. 2019. "PyTorch: An Imperative Style, High-Performance Deep Learning Library." Advances in NeurIPS 32. (Released 2016.) Available at arxiv.org

[CKB+21] Chen, M., Tworek, J., Jun, H., et al. 2021. "Evaluating Large Language Models Trained on Code." arXiv:2107.03374. Available at arxiv.org

AI coding

AI coding assistants depend on language models that map input sequences to output sequences. The following milestones trace the architectural and scaling developments that made those models possible.

2017. Transformers replace recurrence with self-attention

Problem. By 2017, neural networks had replaced rule-based systems, count-based n-grams, phrase tables, and hand-engineered features for tasks like machine translation, language modeling, and question answering. The dominant architecture was the encoder-decoder, implemented with recurrent neural networks (typically LSTMs). It mapped an input sequence to an output sequence, for example one sentence to another or a natural language description to code. The encoder processed the input token by token and produced a fixed-length vector (the final hidden state). The decoder consumed that vector and generated the output token by token, autoregressively. The limitation was recurrence. At each step $t$, the hidden state $h_t$ depended on the previous hidden state $h_{t-1}$ and the current input $x_t$, so computation was strictly sequential.

\[h_t = f(h_{t-1}, x_t)\]

Because $h_t$ depends on $h_{t-1}$, the forward pass required $n$ sequential steps and could not be parallelized. Information from position $t$ to $t+k$ propagated through $k$ steps. During backpropagation, the gradient was multiplied by $\partial h_t / \partial h_{t-1}$ at each step. The product of $k$ Jacobians often had spectral norm below one, so the gradient decayed exponentially and long-range dependencies received negligible signal. An architecture that allowed parallel computation and direct flow between arbitrary positions was needed.

Solution. Vaswani et al. [VSP+17] introduced the Transformer, an encoder-decoder that dispenses with recurrence. Instead of the recurrent update above, each layer uses self-attention. Let $i$ and $j$ denote sequence indices (positions). At each layer, the representation at position $i$ is computed as a weighted sum over all positions $j$,

\[\begin{aligned} h_i &= \sum_j \alpha_{ij} V_j \\\\ \alpha_{ij} &= \mathrm{softmax}\bigl( q_i \cdot k_j \big/ \sqrt{d} \bigr) \end{aligned}\]

The query $q_i$, key $k_j$, and value $V_j$ are learned linear projections of the input representations. Unlike recurrence, $h_i$ depends on all $h_j$ in one step, with no sequential dependency. All positions are updated in parallel, and information between any two positions flows in one layer regardless of their distance in the sequence. This design yields three effects. Training is fully parallelizable over the sequence. Long-range dependencies avoid vanishing gradients because the gradient between any two positions traverses one layer, not many. The architecture scales to very large models and datasets. Transformers underpin BERT (2018), GPT-2 (2019), GPT-3 (2020), Codex (2021), and all subsequent code-capable language models.

References

[VSP+17] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. 2017. "Attention Is All You Need." Advances in Neural Information Processing Systems 30. Available at arxiv.org

2020. Large language models demonstrate in-context learning

Problem. By 2019, Transformer-based language models such as BERT and GPT-2 had been pretrained on large text corpora. The standard way to apply these models to a specific task was supervised fine-tuning. A practitioner took a pretrained model, collected labeled examples for the target task, and trained the model on those examples. Translation required labeled translation pairs. Sentiment analysis required labeled sentences. Code generation required labeled specification-code pairs. Each task demanded its own dataset and its own training run. Deploying a new capability meant fine-tuning, validating, and shipping a new model variant. The cost of data collection and the expertise required for training and deployment limited adoption to organizations with dedicated ML infrastructure.

Solution. Brown et al. [BMR+20] showed that fine-tuning could be dropped and demonstrated it at scale. A 175-billion-parameter model, trained only on next-token prediction over text, performed well across many tasks when given a few in-context examples and no gradient update. Smaller models had not shown the same capability, so scale mattered. The pretraining corpus contained many input–output style subsequences, such as translations, Q&A, and code with comments, so the model had already learned to continue them without task labels. At inference the input was a prompt of a few pairs plus the new query. A translation prompt could look like:

“Hello, world.” → “Bonjour, le monde.”
“Good morning.” → “Bonjour.”
“See you tomorrow.” → ?

The model produced the continuation by computing $P(\text{next token} \mid \text{prefix})$ with the prompt as prefix, with no second phase or parameter update. They called this in-context learning because the task was specified only in the prompt at inference.

References

[BMR+20] Brown, T. B., Mann, B., Ryder, N., et al. 2020. "Language Models are Few-Shot Learners." Advances in Neural Information Processing Systems 33:1877-1901. Available at arxiv.org

2021. Copilot and Codex bring AI code generation to mainstream development

Problem. Software engineering continued to face productivity bottlenecks. Significant time was spent on mechanical tasks, including implementing CRUD endpoints and validation logic, consulting documentation for library and API usage, searching codebases for analogous implementations, translating schemas to types and API specs to stubs, and writing unit tests with conventional arrange-act-assert structure.

Solution. Research had already shown that pretraining on code improved over general-purpose LMs. CodeBERT [Fen+20] and related work demonstrated that joint representations of code and natural language supported search, summarization, and completion. Codex [CKB+21] was a GPT model fine-tuned on publicly available code from GitHub. It used the same next-token, in-context paradigm as Brown et al., but with a code-heavy training distribution, and outperformed general-purpose models on code. GitHub Copilot [Git21] (June 2021) was the first mainstream assistant, with 55% faster task completion [Git22]. The model completed code as programmers typed. The abstraction was autocomplete at the level of functions and blocks. Verification remained necessary. Output was statistically plausible, not formally correct.

References

[CKB+21] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., et al. 2021. "Evaluating Large Language Models Trained on Code." arXiv:2107.03374. Available at arxiv.org

[Fen+20] Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., et al. 2020. "CodeBERT: A Pre-Trained Model for Programming and Natural Languages." Findings of EMNLP. Available at arxiv.org

[Git21] GitHub. 2021. "Introducing GitHub Copilot: Your AI pair programmer." Available at github.blog

[Git22] GitHub. 2022. "Research: Quantifying GitHub Copilot's impact on developer productivity and happiness." Available at github.blog

2022. RLHF aligns code models to programmer intent

Problem. Models optimized for next-token prediction did not reliably follow instructions or match user preference. A programmer asking to “add error handling” might receive technically valid code that didn’t match their error-handling conventions. Early Copilot and Codex produced code that was statistically plausible but often misaligned with intent.

Solution. The fix was to add a second training phase that optimized the policy for human preference, not only for next-token likelihood. Here the policy is the code model that, given a prompt $x$, defines a distribution over completions $y$, written $\pi_\theta(y \mid x)$ with parameters $\theta$. The procedure is reinforcement learning from human feedback (RLHF). It has two components. (1) A reward model and (2) an RL phase.

(1) The reward model is a network that assigns each prompt-completion pair a scalar reward, e.g. $r(x, y) \in \mathbb{R}$ where $x$ is the prompt and $y$ is the completion. It is trained on human pairwise preferences so that preferred completions receive higher reward. That required new human-labeled data, but only preference labels that indicate which of two completions is better, not full target completions. The scale of this preference data is on the order of tens of thousands of comparisons, far less than pretraining data.

(2) The RL phase has one goal. The policy is adjusted so that its completions get high reward from the reward model (i.e. what humans prefer), without drifting so far from the pretrained reference that outputs degenerate into gibberish or reward-hacking (e.g. repeating phrases the reward model likes). The policy is fine-tuned with PPO (proximal policy optimization), an RL algorithm that updates the policy in constrained steps. In plain language, the training objective is to maximize the average reward on completions the policy produces, then subtract a penalty for how far the policy has drifted from the pretrained reference. So the policy is pushed toward high-reward outputs but kept close to the reference so that outputs stay readable, valid code.

Formally, the objective is

\[\mathbb{E}_{y \sim \pi_\theta}[r(y)] - \beta\,\mathrm{KL}(\pi_\theta \| \pi_{\mathrm{ref}})\]

where $\mathbb{E}_{y \sim \pi_\theta}$ is the expectation (average over completions drawn from the policy), $y$ is a completion, and $r(y)$ is its reward. Thus the objective maximizes the first term (expected reward under the policy) and minimizes the second (deviation from the reference policy). The coefficient $\beta$ controls the tradeoff between reward and staying close. KL denotes the Kullback–Leibler divergence,

\[\mathrm{KL}(\pi_\theta \| \pi_{\mathrm{ref}}) = \mathbb{E}_{y \sim \pi_\theta}\left[\log \pi_\theta(y) - \log \pi_{\mathrm{ref}}(y)\right],\]

Without the KL term, the policy can collapse toward high-reward, low-fluency or reward-hacking outputs. The KL term keeps outputs close to the reference distribution so that they stay readable, valid code. The policy is thus optimized for preference, not only for likelihood on a fixed corpus.

InstructGPT (March 2022) [Ouy+22] and ChatGPT (November 2022) established the pipeline. Code assistants adopted it with programmer labelers and code completions.

References

[Ouy+22] Ouyang, L., et al. 2022. "Training language models to follow instructions with human feedback." Advances in NeurIPS 35. Available at arxiv.org

2023. RAG grounds code generation in the codebase

Problem. A language model’s context window is the maximum number of tokens (roughly, words or subwords) it can take as input in one call. Code-capable models of the Codex and Copilot era (2021–2022) had context windows of 2k–8k tokens. That was enough for a short prompt and a few in-context examples, but not for real codebases. Typical limits remained 4k–8k tokens through 2022. A programmer fixing a bug needed relevant files in context, but those limits could not hold them. Even a modest service spanning dozens of files and tens of thousands of lines exceeded the window, so the model never saw most of the code.

Solution. RAG (retrieval-augmented generation) [Lew+20] was introduced for knowledge-intensive NLP in 2020. Code assistants adopted it for the codebase context problem in 2023. The delay reflected two factors. Code-specific retrieval infrastructure (repository indexing, code-aware embeddings) had to be developed. In addition, context limits became a pressing constraint only once coding assistants were widely adopted. RAG sidesteps the context limit by not sending the whole codebase. A retrieval step (e.g. semantic search over embeddings or a code index) selects a subset of files or snippets relevant to the programmer’s request. Only that subset is concatenated into the prompt, so the model’s fixed context window holds the query plus the retrieved material instead of the entire repo. The model’s output is therefore grounded in actual codebase structure rather than generic patterns. Cursor, GitHub Copilot Chat, and others adopted RAG for codebase search. Programmers could point the assistant at a repo and get answers grounded in its structure and contents.

References

[Lew+20] Lewis, P., et al. 2020. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." Advances in NeurIPS 33. Available at arxiv.org

2023–2024. Long-context and agentic interfaces expand scope

Problem. RAG addressed context limits by supplying a retrieved subset of the codebase to the model, but the assistant remained a single-turn completer. It produced output only in response to the current prompt and had no ability to execute tools, query the repository, run tests, or incorporate execution results into the next step. Any task that required multiple steps (for example, fixing failing tests by running the test suite, reading failures, editing code, and re-running until green) therefore had to be orchestrated entirely by the programmer, who ran each step, read the outcome, and re-prompted by hand. The cognitive and manual burden of multi-step tasks stayed with the programmer rather than shifting to the assistant.

Solution. Two developments unfolded over 2023 and 2024.

First, context windows grew. Models with 100k-token context (e.g. Claude 2, GPT-4 Turbo) reached production in 2023, and 200k-token windows became available by 2024. Entire moderate-sized repositories could fit in context, so the model could reason about architectural patterns, cross-file dependencies, and project-wide conventions without retrieval.

Second, agentic interfaces enabled multi-step behaviour. The enabling mechanism is tool use (function calling). The model emits structured tool invocations (e.g. run command, read file, edit file, run tests). The host executes them and appends the results to the model context, so that the model chooses the next action in a repeating plan, act, observe loop. Cursor embedded this pattern in the IDE. Devin (Cognition, March 2024) applied it to autonomous multi-file coding. Claude’s “computer use” capability [Ant24cu] (Anthropic, October 2024) extended it to direct desktop control (cursor, keyboard, screen) in addition to structured tool APIs.

A single request such as “fix the failing tests” could thus trigger a multi-step workflow (run tests, read failures, locate code, generate fixes, rerun tests, iterate) without the programmer re-prompting at each step. Multi-agent systems (e.g. MetaGPT, Devin) went beyond a single model driving tools by deploying several agents that divide the work. Each agent has a distinct role (e.g. planning, coding, reviewing, testing), and they pass outputs to one another so that planning, implementation, and verification are separated and sequenced rather than performed by one monolithic assistant.

References

[Ant24cu] Anthropic. 2024. "Introducing computer use, a new Claude 3.5 Sonnet, and Claude 3.5 Haiku." Available at anthropic.com

2024. Extended reasoning and enterprise fine-tuning complete the AI coding assistant stack

Problem. By 2024, coding assistants combined RLHF, RAG, long-context windows, and agentic tool use. Two gaps remained.

The first gap was the absence of an explicit reasoning phase before the model produced code. Many tasks benefited from weighing options before committing, such as diagnosing a failing test whose cause might lie in several files or choosing among plausible implementations. The autoregressive model used in these assistants did not. It generated one token at a time, at each step computing the distribution over the next token given the prefix and then emitting it.

\[P(x_t \mid x_1, \ldots, x_{t-1})\]

That autoregressive model did not weigh alternatives before committing. When asked to fix a bug, it could output the first line of a patch immediately, without having considered other possible causes. A human might consider several alternatives before writing any code; such a model did not, and on those tasks its outputs were often wrong or suboptimal.

The second gap was a distribution mismatch between model output and each organization’s codebase. RAG and long-context windows both supplied the organization’s code as input in the prompt, so that the model had access to it at inference. The weights, however, had been learned only on public corpora and did not change at inference. The model could reuse names or patterns from the prompt, but when the prompt did not fully determine style, structure, or naming, it fell back on what it had learned in training. Output often looked more like public repos than the organization’s code, so programmers edited heavily or rejected it.

Solution. Each gap had a direct technical fix.

The first gap was the absence of explicit reasoning before committing to code. Extended-reasoning models solved it. Models such as o1 [Ope24] add an internal chain-of-thought phase before the final output. Instead of generating code tokens directly from the user prompt, the model first generates a sequence of reasoning tokens $r_1, \ldots, r_k$ and then the answer tokens $y_1, \ldots, y_n$ (the code). The user sees only the answer. The next-token distribution at each step conditions on the full prefix, including the model’s own reasoning, so the model can explore steps or alternatives before committing to code. Formally, the output distribution is

\[P(y_{1:n} \mid x) = \sum_{r_{1:k}} P(r_{1:k} \mid x)\, P(y_{1:n} \mid x, r_{1:k}).\]

In practice the model is trained to produce $(r_{1:k}, y_{1:n})$ and is given more inference-time compute for the reasoning segment. For complex tasks (e.g. multi-file debugging or choosing among implementations) this yielded substantially better results than direct generation.

The second gap was a distribution mismatch between model output and each organization’s codebase. Enterprise fine-tuning solved it. The model’s parameters are updated on the organization’s code. Let $\theta$ denote the base parameters (trained on public corpora). Let $\mathcal{D}_{\mathrm{org}}$ denote the organization’s dataset (e.g. proprietary code or prompt–completion pairs). Fine-tuning minimizes the negative log-likelihood on $\mathcal{D}_{\mathrm{org}}$,

\[\mathcal{L}(\theta) = -\sum_{(x,y) \in \mathcal{D}_{\mathrm{org}}} \log P_\theta(y \mid x),\]

yielding parameters $\theta_{\mathrm{org}}$ that assign higher probability to continuations consistent with the organization’s style, naming, and structure. The model’s default behaviour at inference then reflects the fine-tuning corpus rather than public code. Copilot Enterprise (2024) [Git24] offered such customization on proprietary repositories.

By 2024 the ecosystem had diversified. Developers could choose among multiple leading models (Claude, GPT-4, Gemini, open code models such as DeepSeek Coder) and AI-native IDEs (Cursor, Windsurf) alongside incumbent tools. The jump from 2021 Copilot to 2025-era assistants came not mainly from larger base models but from adding RAG, long context, tool use, extended reasoning, and enterprise fine-tuning. Those additions changed what the assistant can do and how well it matches an organization’s codebase.

References

[Git24] GitHub. 2024. "Fine-tuned models are now in limited public beta for GitHub Copilot Enterprise." Available at github.blog

[Ope24] OpenAI. 2024. "Introducing OpenAI o1." Available at openai.com

2024. Code evals establish comparable benchmarks and reveal the gap to real-world tasks

Problem. HumanEval was introduced alongside Codex in the 2021 Codex paper [CKB+21] and gave the field its first standardized benchmark for code generation. However, it only measured function-level generation from docstrings. Modifying a large, unfamiliar codebase from an ambiguous bug report was a different kind of work and still had no shared evaluation. The field could not separate algorithmic performance from real-world codebase capability, so capability claims that mixed the two were not distinguishable.

Solution. SWE-bench [JYW+24] in 2024 supplied the missing benchmark for codebase-editing evaluation. SWE-bench Verified is the curated subset with validated, solvable tasks used for the results reported here. Each instance is an actual bug from open-source repos such as Django, Flask, Matplotlib, and Scikit-learn. The model gets the GitHub issue and must produce a patch that passes the project’s test suite. Success depends on locating the relevant code, respecting architecture and invariants, and avoiding regressions.

In June 2024 Claude 3.5 Sonnet reached 93% on HumanEval and 33.5% on SWE-bench Verified [Ant24]. The same model thus showed a wide spread between the two benchmarks. On SWE-bench Verified, GPT-4 reached 1.74% in early 2024, OpenAI o1 reached 48.9% in December 2024 [Ope24], and Claude 4 reached 72.5% in May 2025 [Ant25]. The spread between function-level generation under clear specs and codebase editing under ambiguous, multi-file constraints is therefore substantial. Leaderboards at swebench.com track current results, and any capability claim must state the benchmark and task class.

References

[Ant24] Anthropic. 2024. "Claude 3.5 Sonnet." Available at anthropic.com

[Ant25] Anthropic. 2025. "Introducing Claude 4." Available at anthropic.com

[Ope24] OpenAI. 2024. "Introducing OpenAI o1." Available at openai.com

[JYW+24] Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., & Narasimhan, K. 2024. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" ICLR 2024. Available at arxiv.org

Discussion

The historical framework above equips us with a lens to understand where AI coding stands today and what its impact may be. The following sections use that lens to further assess the impact of AI coding on software engineering.

The internet, cloud, and mobile eras put AI in context

The internet (TCP/IP, 1983) became a common foundation for connecting machines and distributing software. Cloud computing (AWS EC2, 2006) turned infrastructure from capital expenditure into operational expense and enabled elastic scaling. Mobile (iPhone and Android, 2007–2008) made the phone a general-purpose computer and established app stores as a dominant distribution channel. All three changed how software reached users. AI coding operates at a different layer. It alters how code is produced, not how it reaches users. Nevertheless, our framework does not settle the magnitude of AI’s economic impact relative to the internet, cloud, or mobile.

Verification and maintenance costs determine whether AI displaces SaaS

SaaS prevails where the cost of building, operating, and maintaining software in-house has historically exceeded the cost of subscription. Vendors amortize development, maintenance, security, and compliance across many customers. AI may lower the cost of initial construction and can reduce ongoing maintenance, integration, and compliance. In each use case, subscription is displaced only when AI-assisted in-house development costs less in total than subscribing. Bacchelli and Bird [BB13] find that the expertise to verify code matches the expertise to write it, so verification cannot be offloaded yet and remains a large share of in-house cost. Where that holds, total in-house cost may stay above subscription even when AI lowers the cost of producing code.

SaaS has other moats that in-house builds do not easily reproduce. Vendors spread the cost of compliance certifications (e.g. SOC 2, HIPAA), availability and SLAs, ongoing R&D, and data that grows with the customer base. A single organization replicating that must bear the full cost of audits, redundancy, feature development, and acquiring equivalent data.

References

[BB13] Bacchelli, A. & Bird, C. 2013. "Expectations, Outcomes, and Challenges of Modern Code Review." Proceedings of ICSE, 712-721. Available at dl.acm.org

Each past abstraction eliminated the need to acquire entire areas of knowledge

Successive abstractions removed entire domains of knowledge from the programmer’s burden. FORTRAN let programmers write in a high-level language instead of coding machine instructions. Unix removed vendor-specific system calls and device interfaces. Relational databases removed the need to understand the physical storage layout. TCP/IP obviated the knowledge of each network’s internals so any machine could talk to any other on a single global network. In each case the abstraction was sound. Programmers could rely on it without verifying the layer below.

The AI case differs. Coding assistants significantly reduce the effort of producing code, but not the need to verify it. Past abstractions eliminated the need to acquire certain knowledge. AI may accelerate production without removing the expertise required to evaluate and maintain the result. Whether that distinction holds as tools evolve remains an open question.

English is not a programming language

One obstacle persists no matter how capable AI becomes at testing and verification. The artifact that is stored, run, reviewed, and maintained is code, not the natural-language prompts that may have produced it. Meyer puts it directly. Programmers save the source code, not the prompts, because prompts cannot serve as reproducible specification [Mey25]. English is not a programming language because the code is not in English.

Programming languages serve two important purposes. First, they eliminate ambiguity for machines. Natural language is inherently ambiguous. Berry and Kamsties show that ambiguity in requirements is inescapable; different readers take different meanings from the same text [BK04]. “Export recent orders” leaves format, date range, and fields unspecified. “Retry on failure” leaves how many attempts, which exceptions, and whether to back off unspecified. “Delete inactive users” leaves the inactivity threshold and soft-delete versus purge unspecified. The same prompt yields different code from an LLM on different runs.

Second, they force precision in human thinking. Writing code commits to each choice. In Python you might call export_orders(since_days=7, format='csv', fields=['id', 'total', 'created_at']), retry(times=3, on=TimeoutError), or delete_inactive_users(inactive_since_days=90, soft=False). Each argument answers a question the English left open. The language demands answers. The discipline of expressing logic in code makes the logic itself clearer.

Intentional Software and generations of research aimed to let humans specify intent without writing code. The idea was that domain experts would edit in their own notation (e.g. tax rules or business logic in domain vocabulary) and the system would maintain a single representation and generate code, like WYSIWYG for documents but for software [Sim95]. The vision was influential. Intentional Software was acquired by Microsoft in 2017, but the approach never became mainstream. Nevertheless, the dream persists and may be more accessible than ever.

References

[Mey25] Meyer, C. 2025. "English Isn't a Programming Language." Substack. Available at csmeyer.substack.com

[BK04] Berry, D. M. & Kamsties, E. 2004. "Ambiguity in Requirements Specification." In Perspectives on Software Requirements, Springer, 7-44. Available at link.springer.com

[Sim95] Simonyi, C. 1995. "The Death of Computer Languages, The Birth of Intentional Programming." Microsoft Research Technical Report MSR-TR-95-52. Available at microsoft.com

Open source creation and maintenance both benefit from AI

Empirical work on open source suggests that both creation and maintenance benefit from AI. Hoffmann et al. find that maintainers with access to GitHub Copilot increase coding activity and reduce project management load, and that these effects persist for at least two years [HBB24]. Yeverechyahu et al. find that maintenance-related contributions rise more than original contributions [YMO24], so the larger gain appears to be in maintenance.

References

[HBB24] Hoffmann, M., Boysel, S., et al. 2024. "Generative AI and the Nature of Work." SSRN. Available at papers.ssrn.com

[YMO24] Yeverechyahu, D., Mayya, R., & Oestreicher-Singer, G. 2024. "The Impact of Large Language Models on Open-source Innovation." SSRN. Available at papers.ssrn.com

Languages, frameworks, and tools are consolidating, and AI may accelerate the trend

Consolidation around a small number of languages, frameworks, and ecosystems has long been the norm. Language adoption follows a power law, with a few languages accounting for most use [MR13]. Gu et al. run thousands of algorithmic coding tasks and hundreds of framework selection tasks and find that mainstream languages and frameworks achieve significantly higher success rates in AI-generated code than niche ones [Gu25].

Twist et al. give eight LLMs coding tasks with expert-written reference solutions that specify which libraries to use. They find that LLMs import dominant libraries like NumPy even when those libraries do not appear in the reference solution, in up to 48% of cases. For language choice, the same models are given project initialization tasks in domains where Python is suboptimal for performance. Python is still chosen in 58% of cases and Rust zero times [Twist25].

On the frontend, observers note that models default to React because it dominates training data, even when simpler approaches would serve the task [NS25]. The winner-take-all pattern predates AI, however, current models are accelerating it.

References

[MR13] Meyerovich, L. A. & Rabkin, A. S. 2013. "Empirical analysis of programming language adoption." Proceedings of OOPSLA. ACM. Available at dl.acm.org

[Gu25] Gu, F., Liang, Z., Ma, J., & Li, H. 2025. "The Matthew Effect of AI Programming Assistants: A Hidden Bias in Software Evolution." arXiv:2509.23261. Available at arxiv.org

[Twist25] Twist, L., Zhang, J. M., Harman, M., Syme, D., Noppen, J., Yannakoudakis, H., & Nauck, D. 2025. "A Study of LLMs' Preferences for Libraries and Programming Languages." arXiv:2503.17181. Available at arxiv.org

[NS25] Cass, S. 2025. "Web Development in 2025. AI's React Bias vs. Native Web." The New Stack. Available at thenewstack.io

Can AI improve existing abstraction layers?

Research shows AI improving existing abstraction layers in several domains. Learned query optimizers outperform classical optimizers on some workloads, and GenJoin consistently outperforms PostgreSQL on standard benchmarks [Gen24]. In compilers, models trained on LLVM IR and assembly reach a substantial fraction of autotuning search potential [Met24]. In cloud infrastructure, reinforcement learning for dynamic resource allocation has been shown to reduce CPU allocation and improve utilization over rule-based autoscaling [Fet23]. Nevertheless, these results show that AI is already delivering real gains within existing abstraction layers.

The cost framework in this article implies that new abstractions emerge when the cost of the incumbent exceeds the cost of the alternative. Further, past transitions such as relational algebra, garbage collection, and TCP/IP required conceptual shifts. AI may lower the cost of exploring new designs. Whether that yields qualitatively new abstractions, such as new models of concurrency, persistence, or distribution, or meaningfully better cloud, databases, or languages remains to be seen.

References

[Gen24] Sulimov, P., Lehmann, C., & Stockinger, K. 2024. "GenJoin: Conditional Generative Plan-to-Plan Query Optimizer." arXiv:2411.04525. Available at arxiv.org

[Met24] Meta. 2024. "LLM Compiler: Foundation Models of Compiler Optimization." arXiv:2407.02524. Available at arxiv.org

[Fet23] Fettes, Q., Karanth, A., Bunescu, R., Beckwith, B., & Subramoney, S. 2023. "Reclaimer: A Reinforcement Learning Approach to Dynamic Resource Allocation for Cloud Microservices." arXiv:2304.07941. Available at arxiv.org

Conclusion

Seven decades of software engineering followed one dynamic. When the cost of manual work exceeded the cost of automation, the abstraction won. What counted as cost varied. Programmer time, portability, errors, capital, and scale each drove different shifts and produced layer upon layer of abstractions that reduced cost and expanded what was possible.

AI coding fits the same cost logic. Assistants reduce programmer time on mechanical work and accelerate production. Open source evidence suggests both creation and maintenance benefit. Code remains the durable artifact that teams can review, refine, and own. Consolidation around dominant languages and frameworks may deepen, a pattern that has long applied.

Research already shows AI improving existing layers. Query optimizers, compilers, and cloud resource allocation all see gains. Whether AI will yield qualitatively new abstractions remains open. Better tooling and formal methods may narrow the verification gap further. Software engineering has absorbed every prior shift, and there is good reason to expect it to absorb this one.

Revisiting Moneyball

2025-07-24T00:00:00+00:00

Update, July 25, 2025. This post reached the front page of Hacker News. Thanks to @matsonj and @akm for the heads-up.

Introduction
Understanding Moneyball
Critiquing Moneyball
Conclusion

Oakland Coliseum, September 2022.

Introduction

Why are we still talking about Moneyball? Why am I talking about Moneyball?

For one, I’ve been meaning to write this post since 2019. 2018 even.

My friends are the best. From here.

The book was published in 2003, and the movie was released in 2011. It feels silly to rehash, except the whole thing fascinates fans two decades later. Why? On the one hand, it is loved because it tells the classic underdog story while suggesting that nerdy analytics contributed to the wins; on the other, and especially as the A’s leave Oakland, it is blamed for encouraging low payrolls and critiqued for overstating the impact of analytics on the 2002 season and beyond.

From here, here, and here.

In this post, I’ll go over the author’s intentions for writing the book, followed by popular critiques of Moneyball. The goal is to address some of the recurring debates as we cover the main themes and provide historical context for each.

Understanding Moneyball

Let’s begin by discussing the book’s central themes and what the author found intriguing about the A’s in 2002. Moneyball has been so influential that many of these themes have since become memes.

Oakland Coliseum, September 2022.

The 2001 A’s lost significant talent.

After losing MVP Jason Giambi (7-year, $120M), Johnny Damon (4-year, $31M), and closer Jason Isringhausen (4-year, $27M), totaling $31.6 million in annual value ($17.1M + $7.75M + $6.75M), the cash-strapped A’s faced an impossible challenge with just a $41 million payroll. This is what made the season irresistible to Michael Lewis as he watched the A’s replace these stars not with expensive equivalents, but with undervalued players.

The A’s used analytics to unlock transparent decision-making.

The A’s GM, Billy Beane, working with statistician Paul DePodesta, used sabermetrics to challenge traditional scouting methods that relied on more subjective evaluations.

This transparency allowed decisions regarding players to be justified through objective metrics. When the A’s signed Scott Hatteberg or traded for David Justice, they could point to specific evidence, like on-base percentage (OBP) or plate discipline, to justify their decision.

Getting on base was undervalued.

While scouts focused on batting average, home runs, and RBIs, Beane recognized that OBP had a stronger correlation with run production than any of the traditional metrics.

The table below reproduces r² from a Bucknell University paper on MLB run scoring that covers 146 MLB team seasons from 1996 through 2000 [1]. Each row regresses runs per game on the statistic in the first column.


On-base plus slugging (OPS)	.900
On-base percentage (OBP)	.835
Slugging percentage (SLG)	.804
Batting average (BA)	.672
Home runs (HR)	.542
Walks (BB)	.404
Strikeouts (SO)	.078
Stolen bases (SB)	.001

Separate work on Southeastern Conference team seasons from 2014 through 2017 still regresses team runs on team on-base percentage at the college level, which supports treating the link as more than a one-off MLB artifact [2].

The math was simple. Teams that get on base more frequently score more runs, and teams that score more runs win more games. Yet in 2002, players with high OBP were available at below-market prices. Players like Scott Hatteberg (.361 career OBP) and David Justice (.378 career OBP) were affordable because their most valuable skill, i.e., getting on base, wasn’t appreciated sufficiently by the market.

The strategy worked. The 2002 A’s ranked 4th in baseball in OBP (.349) despite having the 3rd lowest payroll in baseball.

In 2002, the A’s achieved 4th highest OBP (.349) with 3rd lowest payroll ($40M) [3][4].

Scott Hatteberg was undervalued.

Scott Hatteberg was available at a below-market price due to perceived limitations. The former Boston Red Sox catcher developed nerve damage in his throwing elbow, making catching nearly impossible and leaving him without a clear defensive position.

Oakland signed Hatteberg to first base for $950K on a one-year contract after Colorado declined salary arbitration. The A’s saw past the injury to identify exceptional plate discipline and on-base skills. His .367 OBP in 2000 and .410 OBP in 1999 demonstrated an ability to get on base.

Catcher-to-first-base conversions are common in baseball, and most are injury-driven. But players like Joe Mauer and Buster Posey had some experience at first base before switching permanently. Hatteberg had zero professional innings at the position. He went from injured catcher to opening day first baseman in one spring training, learning the position from infield coach Ron Washington.

Hatteberg’s 2002 performance validated the analytical approach: .280/.374/.433 slash line, 68 walks versus 56 strikeouts, and 4.15 pitches per plate appearance (3rd in AL). His defining moment came on September 4, 2002, when his walk-off home run against Kansas City secured the A’s 20th consecutive victory and set an AL record.

Hatteberg's historic walk-off home run that gave the A's a 12–11 win and a then-AL record 20-game winning streak.

The streak was historic and remarkable.

The 20-game winning streak (August 13 - September 4, 2002) became the dramatic centerpiece of the Moneyball story.

During the historic run, Billy Koch (acquired December 7, 2001) recorded wins or saves in 12 of the 20 games, while Cory Lidle (acquired January 8, 2001), who was arguably the streak’s MVP, posted a microscopic 0.20 ERA in August with 32 consecutive scoreless innings.

The streak’s climactic finish, Oakland blowing an 11–0 lead to Kansas City before Hatteberg’s walk-off home run, provided Hollywood-worthy drama that helped make the analytical approach culturally compelling.

More importantly, the streak occurred during a season where the A’s won 103 games despite having one of baseball’s lowest payrolls.

The A’s achieved 103 wins (tied for MLB lead) with the 3rd lowest payroll ($40M) [4][5].

You can build a player in aggregate.

The A’s discovered they could construct effective offensive production by combining players with complementary skills rather than seeking complete players. This insight challenged the traditional scouting preference for “five-tool players” who could hit for average, hit for power, run, field, and throw.

Instead of expensive superstars, the A’s assembled a roster where different players contributed specific, undervalued skills:

Scott Hatteberg (signed Jan 2, 2002): Injured catcher converted to first baseman, valued for exceptional plate discipline
David Justice (traded Dec 14, 2001): Aging slugger acquired cheaply after the Yankees gave up on him, provided veteran power and leadership
Jeremy Giambi (traded Feb 18, 2000): High-OBP outfielder whose walk rate offset his defensive limitations
Chad Bradford (traded Dec 7, 2000): Submarine reliever overlooked by scouts, acquired for a minor league catcher
Billy Koch (traded Dec 7, 2001): Proven closer bought low after a rough 2001, rebounded to win AL Reliever of the Year
Cory Lidle (traded Jan 8, 2001): Control pitcher with poor surface stats but strong peripherals, became a reliable #4 starter

This aggregate approach allowed Oakland to compete with teams spending three times their payroll. Rather than paying premium prices for complete players, they constructed a competitive roster through strategic combinations to produce runs and wins.

From here.

The playoffs are a crapshoot.

Billy Beane famously told Michael Lewis, “My shit doesn’t work in the playoffs,” acknowledging that his analytical approach, while dominant over 162 games, couldn’t overcome October’s inherent randomness. It’s a “crapshoot” because the best regular season team frequently loses due to small sample sizes and variance inherent in short series [6].

The statistical evidence supports this theory. Since the Wild Card era began in 1995, the team with the best regular season record has won the World Series only 8 out of 29 times (a 28% success rate).

World Series winners with best regular season record since 1995 and their win-loss records.

The Oakland A’s themselves became the perfect case study for this phenomenon. From 2000 to 2003, they averaged 98 wins per season, yet lost in the Division Series (first round of the playoffs) each of the four years, with each series going the full five games.

Academic analysis by Stanford [6] found no correlation between regular season and postseason performance (p = .6201), while a comprehensive Braves Journal study concluded that playoffs are “90% crapshoot, 10% skill” [7].

The small sample size problem is a fundamental issue in the playoffs. While batting statistics require 200+ plate appearances to stabilize, playoff series provide players with only 15–30 plate appearances, resulting in massive variance. Even Oakland’s signature OBP, which is the cornerstone of their analytical advantage, declined 11% in the playoffs (.305 vs .341 regular season), demonstrating how short series neutralize statistical edges.

This randomness explains why sabermetricians often view regular-season performance as a more reliable indicator of a team’s true quality than its playoff results.

It’s hard not to be romantic about baseball.

The A’s story showed ordinary players achieving extraordinary things when given the opportunity. The analytics enhanced rather than detracted from the game with moments like Bradford’s submarine delivery, Hatteberg’s transformation from injured catcher to first baseman, or the electricity of the 20-game streak.

From here.

Now that we’ve gone through the book’s central themes, let’s discuss how they hold up. In particular, let’s examine them through the lens of criticisms of Moneyball and whether those criticisms are warranted.

Critiquing Moneyball

Moneyball overlooks existing talent on the 2002 roster.

The most substantial criticism of Moneyball, both the book and especially the film, is that it ignored the exceptional talent on the 2002 Oakland roster. The narrative focused so heavily on the undervalued acquisitions that it obscured the presence of conventional superstars.

The A’s were excellent at drafting talent. And their lineup in 2002 was made up of more than just underdogs.

The Big Three. Tim Hudson (6th round, 1997), Mark Mulder (2nd overall, 1998), and Barry Zito (9th overall, 1999) were all top-10 picks. By 2002, this trio anchored Oakland’s rotation with elite performance.

Miguel Tejada (international signing, 1993) and Eric Chavez (10th overall, 1996) provided MVP-caliber offense and Gold Glove defense. Both were premium draft investments that matured into franchise cornerstones.

Miguel Tejada won the 2002 AL MVP award, hitting .308 with 34 home runs and 131 RBIs while providing leadership throughout the season. Barry Zito won the AL Cy Young Award with a 23–5 record and 2.75 ERA. These weren’t marginal players elevated by analytics; they were elite performers by any standard.

Oakland executives Billy Beane, David Forst, and scout Ron Washington later acknowledged that “there’s no way the A’s make the playoffs every year from 2000 through 2003, and no way a best-selling book and Brad Pitt movie ever happen, if not for the efforts of the Big Three” [8]. The Big Three compiled a collective 261–131 record from 1999–2006, providing the foundation that allowed Beane’s analytical approach to flourish.

Moneyball promotes low payrolls in baseball, thus ruining the game.

Oakland won 103 games in 2002 with the third-lowest payroll in baseball. Sports economists have argued that Moneyball became a cultural reference point in arguments about spending restraint [11]. The critique is that the story made low payrolls look like the intelligent default rather than a limitation.

However, the actual lesson of Moneyball for the league was not “cheap is good.” In fact, the most successful application of Moneyball principles came from the Boston Red Sox, who were anything but frugal.

After the 2002 season, Red Sox ownership (led by John W. Henry) tried to hire Billy Beane as a GM with a five-year, $12.5M contract that would have made him the highest paid GM in baseball history. The offer was a testament to how highly Henry and then-CEO Larry Lucchino regarded Beane’s analytical approach.

Although Beane declined, the Red Sox promoted Theo Epstein to GM in November 2002. Under Epstein, they adopted a more data-driven approach, complemented by their substantial financial resources. In 2004, only two years after the attempt to hire Beane, the Red Sox broke their 86-year Curse of the Bambino by winning the World Series, and again in 2007.

When DePodesta left Oakland in 2004 to become the Dodgers’ GM, he was replaced by Farhan Zaidi. Zaidi carried forward the analytical tradition through the Dodgers (2014–2018) before joining the San Francisco Giants, where his 2021 team achieved a franchise-record 107 wins, demonstrating how Moneyball principles scale effectively when combined with greater financial resources.

Fast forward to today, the Dodgers won the 2024 World Series while employing one of baseball’s largest analytics departments (with over 47 personnel, compared to 3 in 1988) and maintaining one of MLB’s highest payrolls. Their championship validated that analytics enables more effective spending rather than reduced spending.

The 2024 season saw a record nine teams exceed the luxury tax threshold, resulting in $311 million in penalties. The Mets, under Steve Cohen, topped the league with a $333 million payroll, while the A’s payroll of $66.5 million represents ownership decisions rather than analytical ones.

It turns out that Moneyball is about money. Billy Beane’s Oakland A’s succeeded not because they rejected the importance of money, but because they maximized every dollar’s impact when competing against teams with three times their budget. The book’s enduring relevance lies in demonstrating how intelligence and efficiency can overcome, though maybe not eliminate, financial disadvantage.

Rather than promoting frugality, Moneyball’s lasting impact has been to make teams more analytical. Every MLB franchise now employs statisticians and data scientists. The result has been more informed decision making at all spending levels, not a reduction in overall spending. Moneyball’s impact, short-term or long-term, has not been frugality.

The criticisms tend to confuse cause and effect. The analytical methods were in response to frugality, not the cause of it. The cost-cutting was already occurring due to economic pressures unrelated to analytics or the impact of Moneyball.

Moneyball promotes the use of analytics in baseball, thus ruining the sport.

The critique is that Moneyball helped normalize a front-office style tied to more strikeouts, walks, and home runs, fewer balls in play, longer games, and shifts that many fans experience as less action on the field. Figures inside that movement, including Theo Epstein and Bill James, have said publicly that analytics-driven optimization has hurt the game’s aesthetic.

It’s important to note that the analytics revolution was already underway before Moneyball. The book brought it into mainstream consciousness rather than creating it. Allan Roth served as baseball’s first team statistician with the Brooklyn Dodgers from 1947–1964, tracking advanced metrics like OBP and developing situational statistics. Earnshaw Cook’s 1964 book “Percentage Baseball” was the first full-length sabermetrics work, while Bill James began publishing Baseball Abstracts in 1977, coining “sabermetrics” in 1980.

To understand baseball’s numerical history, you can go back even further. Henry Chadwick, an English cricket lover enchanted by the new American sport, created the first box score for a Brooklyn Excelsiors game in 1859, establishing the systematic tracking that made all future analytics possible.

From 1876.

Analytics has improved baseball in measurable ways:

Player development. Teams now use personalized training programs based on biomechanical analysis, injury prevention through wearable technology, and real-time feedback systems.

Strategic decision making. Real-time analytics enable managers to optimize defensive positioning, make data-driven pitching changes, and predict injury risks before they occur. Front offices can identify undervalued talent and make smarter roster construction decisions.

However, the use of analytics has not been without downsides:

The “Three True Outcomes” problem. Home runs, walks, and strikeouts now dominate baseball, with 35% of plate appearances ending without involving seven defensive players. This has reduced balls in play by 20% since 1980, resulting in longer games with less action.

Entertainment decline. As Theo Epstein admitted, “executives like me who have spent a lot of time using analytics…have unwittingly hurt the aesthetic value of the game.” Even Bill James acknowledged the game’s aesthetics “went to hell in a dump truck” due to excessive strikeouts and endless pitching changes.

Analytics improved baseball’s efficiency but arguably damaged its entertainment value. This is a tradeoff the MLB addressed with 2023 rule changes, including pitch clocks and shift bans.

Moneyball suggests baseball is unfair, even though it’s not that unfair, comparatively speaking.

The critique is that baseball rewards money in a way fans experience as unfair, with postseason outcomes and playoff gates increasingly correlated with payroll rank in recent years.

However, despite conventional wisdom linking salary caps to competitive balance, the data shows that MLB’s luxury tax system has produced championship diversity equal to hard-cap leagues. Since 1995, counting 30 championship years, 16 different winners each in MLB, NFL, and NHL, with only the NBA lagging at 13 unique champions. This challenges the assumption that financial constraints alone determine competitive outcomes.

Having said that, recent trends suggest MLB’s historical parity may be eroding under financial pressure. From 2015–2024, eight of ten World Series champions ranked in the top 10 for payroll, with only the 2015 Royals and 2017 Astros winning from outside the top half of spending teams [9]. Meanwhile, in 2024, all six highest-spending teams made the playoffs, while the Dodgers’ $327 million payroll advantage over Miami represents the largest spending gap in modern baseball history.

Despite this, playoff access remains remarkably broad as 28 of 30 MLB teams (93.3%) have made the playoffs since 2015, compared to 28 of 32 NFL teams (87.5%).

While the NBA currently enjoys unprecedented parity, seven different champions in seven years (2019–2025), this represents a dramatic shift from its historically dynasty-heavy nature where superstars concentrated championships among elite teams.

The NFL’s “National Parity League” reputation may be an overstatement. Only five AFC teams (Broncos, Ravens, Patriots, Steelers, Colts) represented the conference in 13 consecutive Super Bowls, while the Chiefs have reached more Super Bowls in six years than any MLB team has World Series appearances in the 21st century [10].

This suggests that playoff format, season length, and sport-specific factors matter more than financial structures alone.

Conclusion

Whether the 2002 A’s succeeded primarily through analytics or traditional talent is less important than the broader principle that analytics can reveal value that conventional wisdom misses. They also showed the power of intelligence, creativity, and sheer persistence in overcoming financial disadvantage.

Ultimately, it’s hard to overlook the attention that Moneyball has brought to baseball, as it continues to capture the imagination of fans, bandwagoners, and Jonah Hill admirers.

From here.

Proud to finally meet Brad Pitt back in October 2019. He gave one of the best talks I ever heard.

From here.

References

[1] Bucknell University Baseball Statistics Research (1996–2000). “Runs Scored Correlations.” Available at: https://www.eg.bucknell.edu/~bvollmay/baseball/runs1.html

[2] FanGraphs Community Research (2014–2017). “Relationship Between OBP and Runs Scored in College Baseball.” Available at: https://community.fangraphs.com/relationship-between-obp-and-runs-scored-in-college-baseball/

[3] Baseball Almanac - 2002 AL OBP Leaders https://www.baseball-almanac.com/yearly/top25.php?s=OBP&l=AL&y=2002

[4] The Baseball Cube - 2002 MLB Team Payrolls https://www.thebaseballcube.com/content/payroll_year/2002/

[5] Baseball-Reference - 2002 MLB Standings https://www.baseball-reference.com/leagues/majors/2002-standings.shtml

[6] Stanford Sports Analytics Club - “Examining MLB Postseason Cluster Luck: or, Why the Playoffs Might Be a Crapshoot” https://stanfordsportsanalytics.wordpress.com/2015/03/24/examining-mlb-postseason-cluster-luck-or-why-the-playoffs-might-be-a-crapshoot/

[7] Braves Journal - “The Playoffs are a Crapshoot” https://bravesjournal.com/2019/12/30/the-playoffs-are-a-crapshoot-a-5-part-series-introduction/

[8] Grantland - “Baseball’s Big Three: A Look Back at Tim Hudson, Mark Mulder, and Barry Zito in Oakland” https://grantland.com/the-triangle/mlb-oakland-as-big-three-tim-hudson-barry-zito-mark-mulder-billy-beane-moneyball/

[9] Cronkite News - “How big MLB payrolls affect postseason success” https://cronkitenews.azpbs.org/2024/11/12/big-mlb-payrolls-affect-postseason-success/

[10] The Athletic - “NFL is the parity league? MLB would like a word” https://www.nytimes.com/athletic/6116536/2025/02/06/mlb-nfl-parity-super-bowl-world-series/

[11] Matheson, Victor. Sportico. “Study Table: Blame Moneyball for Major League Labor Strife.” Available at: https://www.sportico.com/leagues/baseball/2022/study-table-blame-moneyball-1234668622/

This post was originally published on Medium and is cross-posted here.

Building an auth flow with AI

2025-07-20T00:00:00+00:00

In the first part of this series, I shared my experience with vibe coding to build a markdown-based data room application for General Folders. In particular, I highlighted best practices for working with AI coding assistants.

Now, in the second part, I’ll focus on one of the most critical aspects of any application: secure authentication. Specifically, I’ll walk through implementing magic link authentication, a passwordless approach that provides a seamless experience for investors accessing confidential documents.

The Data Room App (thedataroom.app) provides a clean, intuitive interface for investors to access confidential documents.

Solving the authentication challenge

At first, I tried to implement magic links with Windsurf. Given that auth flows are not my area of expertise, I needed more opinionated help. I then tried Replit which provided a quick and working solution.

Replit is a browser-based IDE and hosting platform that started in 2016 as a simple code playground but has evolved into a comprehensive development environment. What drew me to it was its ability to handle full-stack applications with built-in authentication, along with serverless deployments.

Replit vs Windsurf: technical comparison

Replit features

Authentication and session management

Replit’s integrated authentication blueprints provide pre-configured Passport.js setups, session store integration with PostgreSQL, and automatic HTTPS for secure cookie handling. Magic link authentication was implemented in 15 minutes compared to hours of OAuth configuration debugging.

Database integration

Replit enables instant PostgreSQL provisioning through Neon. Connection strings, pooling, and SSL certificates are handled automatically. Drizzle ORM integration works seamlessly with database push deployments without migration file management.

Update (July 2025): Replit has since launched separate development and production databases, which makes the platform more suitable for developing real-world applications. This feature enables safer iteration by isolating development changes from live customer data.

Deployment infrastructure

The platform handles load balancing and scaling automatically.

Replit limitations

Git operations lack the intuitiveness of terminal workflows. Even basic branching and commit management feel clunky in the interface. Simple git commands in a terminal turn into vague processes with less feedback and control.

The container-based environment limits access to lower-level system functions and file operations, which can make debugging more challenging when compared to local development.

Windsurf comparison

Windsurf provides control with traditional Git workflows, intuitive file system access, and familiar terminal operations. However, the setup overhead for authentication, databases, and deployment significantly slows initial development velocity compared to Replit’s integrated infrastructure.

Data room platform implementation

Here are the implementation details we narrowed in on with Replit.

Tech stack

React frontend with TypeScript and shadcn/ui components
Express.js backend with PostgreSQL database
Drizzle ORM for type-safe database operations
JWT-based authentication with 7-day token expiration

Architecture decisions

Simple content management: Founders can organize their data room using a simple Markdown file, just like writing documentation. No clunky admin panels.

Changes to sections.md are reflected instantly - no rebuild required.

Flexible document handling: The app supports both direct file uploads (protected by authentication) and links to existing documents in Google Drive or Dropbox.
Passwordless authentication: Investors access the data room through magic links sent to their email.
Role-based access: Different permissions for founders (who pay for the service) and investors (who get free access).
Engagement analytics: Founders can see which documents investors have accessed and when.
Frictionless invitations: Founders can invite investors with a simple email, which in turn generates secure access tokens.

Implementation results

The platform is deployed at thedataroom.app and operates in waitlist mode for market validation. The implementation includes PostgreSQL-backed secure sessions, reliable email delivery for magic links and invitations, and robust file security with strict validation and secure storage for documents up to 50MB.

Development timeline

Here’s a summary of our development timeline.

Week 1: Authentication and data modeling implementation
Week 2: Document management and UI development
Week 3: Email system integration and production deployment

Feedback request

If you have past experience with data rooms, I’d love to hear your thoughts on the platform. If you’re going to be needing a data room soon, I’d love to show you what we’ve built and get your feedback.

Live platform: thedataroom.app

Access: Waitlist signups receive discounted pricing; investor access remains free but you would need an invite.

Best practices in AI coding

2025-06-20T00:00:00+00:00

This is part 1 of a 2-part post. This first is about my experience with vibe coding, or rather, AI-assisted programming. The second part is about how I set up the magic link authentication flow.

It's different.

The problem

To put these cool tools to use, I’ve been working on a couple of side projects. The one we’ll discuss here is a custom data room app for General Folders.

I evaluated several popular data room solutions to understand their strengths and limitations before deciding to work on a new one.

The solution

Given that context, our goal is to build a secure platform for startups to quickly set up data rooms and share links to confidential documents with authenticated investors.

What we’re envisioning is one flavor of a file explorer for the browser. In addition to the file explorer, we can add on a file viewer, like a slide viewer or a doc viewer, and replicate the logic that DocSend or Google Workspace has built. However, apart from enabling granular tracking, it’s hard to justify the effort that goes into building a file viewer, given the affordable price tag on Google Workspace products and the familiarity of the experience.

As a sidenote, this exercise made me realize that GitHub, Dropbox, and Box are all file explorers at the core, which is quite beautiful.

The tools

Now we get to the topic of vibe coding. I believe the tools have finally reached that critical threshold where they’re not just fun demos but genuinely practical for everyday use. And it looks like others feel the same!

Last time I felt as giddy as I do when vibe coding was my first ever visual basic app. Seismic shifts afoot people, seismic.

— Harry Brundage (@harrybrundage) • May 31, 2024

Programming with AI is what you thought programming would be like prior to learning it.

— Martin Casado (@martin_casado) • June 1, 2024

To build this data room app, I used Windsurf, an AI-assisted development platform founded in June 2021 by Varun Mohan and Douglas Chen. Originally launched as Exafunction (focusing on GPU optimization), the company pivoted to developer tools and rebranded as Codeium in 2022, before becoming Windsurf in April 2025.

For a comparison of coding agents, see Yogesh’s head-to-head comparison, C# Corner’s top AI tools, Kingy AI’s agent analysis, and BrightCoding’s technical benchmarks. To understand how they work, see Sourcegraph’s anatomy of coding assistants.

Observations and best practices

Now let’s get to the main point of this post. Below, I share some observations I’ve made while using these tools over the past few months.

Observations

IDE interface. Interacting with LLMs directly inside an IDE is what makes the programming use case for LLMs so successful. Fast feedback loops while iterating on a project is key.

Skill issue. These tools increase the surface area of projects that I would take on. I know I can rely on them where I lack skills. This means I can take on brand new projects with more confidence.

Best practices

Version control. Version control is good in any situation, but specifically when working with agents that have write access to your codebase. This is not disimlar to how collaborating with a colleague is made practical via version control.

The agent sometimes changes already functional files and modules that are not relevant to your prompt. Sometimes it hallucinates and makes updates that are all wrong. Some say to lock a page you’re sure about; but practically, you rarely want to lock a file in an evolving project, so the best bet is to have meaningful commits that you can revert to.

I would go so far to recommend you manage your git workflow manually. That way you can be sure you have meaningful commits and ways to correct big mistakes and hallucinations. Otherwise you end up resetting a full day’s worth of work.

Local vs global solutions. The agent sometimes does a local hacky solution rather than fixing the root cause. For example, instead of fixing a css issue globally, it might fix it locally for a specific part of the website, or separately for every part. On those occasions it requires nudging about best practices, for example, about separation of concerns. Otherwise, you’ll end up with unmaintainable code.

Code review. As I went deep into a couple of projects, spending more and more time on Windsurf, it had me wondering, am I getting good at anything? What skills, if any, am I gaining? When I program, absent an agent, I get better at programming. When I write, I get better at writing. When I vibe code, especially in a domain I’m not familiar with, am I actually learning anything? This is where I’d recommend reviewing every commit. This way, you’ll not only have better command over your codebase but also learn from your robot friend.

Imprecise details. Being imprecise can be the source of a lot of pain. Be as clear as possible. English is code. I wasted a full half day due to imprecisely describing the folder structure in a project only to realize that it was my prompt and not the agent’s dumbness that was leading the project astray.

Technical details. I’ve found that the more accurate I can explain what I want and the more technical context I can provide, the faster and smoother things go.

Planning file (plan.md). Aside from technically detailed prompts, it helps to have a well-defined project definition at the outset. The project goes smoother if there is a clear roadmap. Windsurf’s Planning Mode not only helps keep track of tasks but can also serve as a reference to past prompts instead of the alternative of “as per my tenth to last prompt.”

Cold starts. Starting from a template is a great idea, when available. Starting from scratch can be challenging especially if you don’t know what the project structure should look like. Lack of templates isn’t necessarily a blocker but templates make the experience smoother; they steer the agent towards best practices in domains where you are not as opinionated.

Debugging. Debugging works easier when you understand the codebase. For example, if I can identify the root cause of a bug, it’s easier to write a prompt and fix the problem. Otherwise, it’s just an endless loop of the blind leading the blind; which can still work but can take forever.

Work estimates. One thing I’ve struggled with is trying to estimate how long something takes. Is it easier or harder to provide work estimates when working with an agent? I’d say it’s harder. Imagine working with someone without having visibility into their strengths and weaknesses. In my experience I’ve found the agent is not great at auth flows, but really good at CSS, for example, but I wouldn’t have known that until I got deep into the project.

Pure vibes. If it’s a small, short-term project, it makes sense to let the agent take the wheel. All you need to do is to nudge it in the right direction once in while but little involvement is necessary. However, for a bigger project, or one that’s meant to be revised later, or one that is shared with other collaborators, it’s different. In this case it’s easier if you know the codebase and review every commit, just as you would if you were working with a colleague who might move to another project at any point. Additionally, if your project is one where performance or security matters then you need to be more alert and involved.

Memories and rules. A new chat window doesn’t always remember how we did things before and will find a new way of doing things, that while probably correct, is not consistent with the rest of the codebase. In these settings it makes sense to create a memory or a rule to guide the agent.

Advice from Windsurf itself

I asked Windsurf directly what advice it would give to someone starting with AI-assisted programming. Here are some key insights from the tool itself.

Context windows. Be mindful of the AI’s context window limitations. For large codebases, focus the AI on specific files or components rather than expecting it to understand the entire system at once. This improves response quality and reduces confusion.

Documentation-first approach. Ask the AI to document its implementation strategy before writing code. This forces clarity of thought and gives you a chance to course-correct before any code is written.

Tool complementarity. AI coding tools work best when complemented with traditional development tools like linters, type checkers, and test suites that can catch issues the AI might miss.

Elsewhere on the web

Check out the Windsurf documentation [7] and Windsurf rules at UI Bakery [8] for general Windsurf best practices. Also see David Crawshaw’s blog post [9] for an interesting look at programming with agents. For more on vibe coding, check out 12 Rules to Vibe Code Without Frustration [10].

That’s it for now. I’ll add to this list as I learn more.

Next steps

In the second part I explain the login flow with magic link authentication. I’ve also published a link for you to try out the data room app.

Next up, stay tuned as I share some of my explorations into MCP servers and agents.

If you have any feedback or advice, please let me know via X, Bluesky, or Medium. I’m looking forward to hearing from you.

References

[1] Papermark. (2023). “Best virtual data room providers comparison”.

[1b] FirmRoom. (2023). “11 best data room providers you need in 2024: Comparison”.

[2] Yogesh, B. (2025). “AI coding tools: A developer’s head-to-head comparison”. A technical comparison of Cursor, GitHub Copilot, Replit, Cline, and Claude Code from an engineer’s perspective.

[3] C# Corner. (2025). “Top 7 AI tools for software developers”. Comprehensive analysis of GitHub Copilot, ChatGPT, Replit AI, Windsurf, Tabnine, Cursor AI, and V0 by Vercel.

[4] Kingy AI. (2025). “AI coding agents in 2025: Cursor vs. Windsurf vs. Copilot vs. Claude vs. VS Code AI”. In-depth analysis of strengths and weaknesses across code generation, debugging, refactoring, and large-codebase support.

[5] BrightCoding. (2025). “AI coding assistants compared: technical benchmarks”. Technical analysis showing Windsurf processes 180 tokens/second with Llama 3.1 405B model, Cursor averages 220 tokens/second with GPT-4o, and GitHub Copilot processes 150 tokens/second with standard models. Engineers prefer Cursor (42%) for refactoring, Windsurf (38%) for speed and privacy, and Copilot (36%) for reliability.

[6] Sourcegraph Engineering. (2024). “The anatomy of an AI coding assistant”. Technical blog explaining how modern AI coding assistants work under the hood, detailing the vector embedding techniques (using OpenAI’s text-embedding-ada-002 or custom models), context fetching mechanisms, and how different features (autocomplete, chat, test generation) use specialized retrieval methods optimized for latency (76ms) or accuracy depending on the use case.

[7] Windsurf. (2025). “Getting Started with Windsurf”. Official documentation on getting started with the Windsurf VSCode fork.

[8] UI Bakery. (2025). “Windsurf AI Rules”. Best practices for working with Windsurf AI and optimizing its performance.

[9] Crawshaw, D. (2025). “Programming with Agents”. An in-depth look at the paradigm shift of programming with AI agents.

[10] Creator Economy. (2025). “12 Rules to Vibe Code Without Frustration”. A practical guide to effective AI-assisted programming techniques and best practices.

Startup advice

2024-08-12T00:00:00+00:00

From my time at Techstars SDSU.

It’s been quite the journey building General Folders so far. I learned things I wish I had known before starting the company. I also learned valuable lessons from past jobs that proved helpful in company building. Below is a list of these lessons. I hope that you find them useful on your journey!

⟢ ⟡ ⟣

Build a team. Bring along co-founders, but for the right reasons. Two can outperform one. Building the first team is your most important task.

Be genuinely passionate. The secret to hiring great people is to be knowledgeable and passionate about the product and business. Passion is contagious, and expertise is hard to fake. The same holds for sales: don’t underestimate the impact of stoke on selling a product.

Define the MVP. Determine the minimum amount of product or visuals required to get 1) customer buy-in and 2) relevant feedback.

Your ability to come up with a successful MVP is tied to being able to describe a very particular use case with a well-defined customer profile. Having a well-defined use case, more than anything, paves the way for successful distribution.

As it turns out, this is an exercise that a company inevitably repeats again and again. You can easily bankrupt a company as you spend to develop an upcoming version.

Sell early and often. A startup’s success hinges on its sales and the trajectory of those sales. Life is easier if the product can be sold early and often. A startup will struggle to survive with grand plans and lengthy sales cycles unless it secures substantial external funding.

Talk to customers. Nothing benefits a company more than customer feedback. Never lose sight of it. Feedback saves products and companies from veering off the tracks into irrelevance.

Embrace change. Don’t obsess over a particular decision or plan. If product reception is anything but stellar, update your strategy.

Follow the crowd. Tried-and-true practices are popular for a reason. It pays to follow the crowd if you know when and where to apply a technique.

Follow trends. Don’t ignore market trends. Following trends makes you more likely to get noticed, talked about, and funded. It’s an easier ride if you want to take it.

Hire an accountant. While it pays to follow trends, plan for the dry season. New trends always give way to newer trends — it’s funny how that works.

Stand out. Likewise, while following trends can be rewarding, sometimes it pays to focus on what you uniquely believe to be true. You can’t expect a significant return without taking on a risk nobody else wants to take.

Hustle. Aside from being a function of big risk, a big return is also an outcome of hard work. There are no shortcuts. Pick up the phone. Knock on doors. Get the product in front of customers. Put in the work.

Make the call. Running a business involves endless decisions with incomplete information. Indecision is a decision. Make the call. Use every decision as an opportunity to learn more about your business.

Move fast. Faster is better, but only when the goal is to serve customers and tend to their needs. Rushing for PR reasons comes at the expense of quality, ultimately tarnishing the brand.

Moving fast is not just about writing more code, sending more emails, and doing more busy work. Building requires clear thinking and planning. It requires coordinating with customers at every step. Skipping steps creates tech debt, ineffective plans, and underwhelmed customers. Rarely do products fail because there isn’t enough code and artifacts; rather, it is often due to inadequate thought. Don’t skip steps. And don’t settle for busy work.

Keep learning. In every scenario, assume you don’t have all the answers. Learn and evolve to become the leader the company needs at each stage. Can you hire yourself every day?

Ask for help. Ask for things and get help. The startup community is more generous, welcoming, and helpful than expected. Knowing when to ask, who to ask, and for what is a skill worth perfecting.

Go after capital. Raise capital to accelerate growth. Attract investors who are already sold on the vision and are ready to champion the company. If it’s a hard sell, it’s not a good fit.

Believe in yourself. Be kind, especially to yourself. Self-confidence is the only thing you have going for you for a long time. Things won’t ever be perfect, and that’s okay.

Keep going. People (mainly yourself) will tell you that what you’re doing is not enough and that if it were to work, it would have worked by now. They will tell you to give up. Don’t. Opportunities will come around.

Keep focused. Don’t compare your startup to other startups. Don’t read into startup news. Startup news doesn’t talk about the deal details. Focus on your early customers. Focus on your collaborators. Focus on well-defined local problems for your first few customers. Don’t attempt to solve global problems for imaginary customers.

Seize the day. You might not have achieved all the milestones, but you’ll never be this young ever again. 💁🏻‍♀️

⟢ ⟡ ⟣

What are some of your experiences from working at startups? I’d love to hear from you!

This article is cross-posted from my original publication on Medium.

The state of data exchange

2023-04-03T00:00:00+00:00

The abstract
The talk

Reactions to the talk

I just got back from Data Council. Thanks to Pete Soderling for an excellent conference where people gather to discuss the future of data infra, AI, and analytics. It was an energizing couple of days with many valuable takeaways as I work on General Folders.

A recording of the talk is available on YouTube.

Below you can find the full transcript. I would love to hear about your experiences with data exchange. Please reach out!

The abstract

Data exchange is integral to every business partnership. Yet data exchange practices are highly manual, prone to data leaks, difficult to validate, inherently impossible to monitor, and costly to audit. In this talk, we present an overview of the variety of methods enterprises use to share and transfer data. We talk about some of the challenges companies continue to face along the vectors of security, simplicity, and speed. We conclude by enumerating the properties of a good solution.

The talk

Business partners exchange data for a variety of reasons

It might be surprising to many, as it was for me, but they do this quite often.

One of the scenarios is SaaS vendors sending data to their customers. SaaS vendors offer some sort of service to their customer’s customers. The customers’ data then ends up in the vendor’s warehouse, and so the vendors need to make that data available to their customers.

Another scenario is when a company is doing market research and needs to incorporate market data into their analysis. They need to talk to data vendors to purchase a particular data set. The procurement process usually involves a data exploration and assessment aspect and the transaction could be a one-time transfer or require recurring updates.

Yet another scenario is when a healthcare, transportation, or energy company is responsible for sending data to the local government on a regular basis.

For AI SaaS evaluation, companies usually want to assess whether a new tool would work for their uses cases. Before buying, they usually transfer data to the vendor’s warehouse and ask that the vendor demonstrates the effectiveness of the new tool on real customer data.

In M&A, the acquirer needs to run their own diligence on the company they want to acquire. They usually ask for certain data sets and if the deal goes through, a full transfer of data assets.

Sometimes businesses need to collaborate on the intersection of their respective customers. For example, a makeup brand (i.e., the supplier) needs to share data with all retailers that carry the brand to find the marketing channels that work best in each cohort.

Yet in other cases, businesses don’t even know the overlap of their customers and that’s what they need to find out. We’ll get into more details about these applications later.

Okay, enough about use cases — we can go on forever.

Companies build solutions for data exchange today

Let’s go through some of these solutions and how they work. I’ve had a lot of conversations with companies about this topic and this list is a kind of summary of those conversations.

The goal is not to judge these methods, or even to assess whether a team made the right call to go with a certain approach — a lot of that depends on infrastructure limitations and deadlines. The goal is to enumerate some of the inevitably great many methods companies use to exchange data with their business partners.

Send or receive data as an Excel file or CSV by Gmail, Slack, Dropbox, or over SFTP

One of the most common ways to exchange data is by setting up an SFTP server, uploading a CSV file, then sharing a link. Some use Amazon Transfer Family and managed workflows to move data in and out of S3 over SFTP. Others use Airflow, Dagster, or cron jobs to set up and manage pipelines.

The data provider manages only half the pipeline. The rest of the pipeline is up to the data consumer.

What happens in this scenario is that the data provider ensures that the data is securely moved from S3 to the SFTP server. It is then up to the data consumer to decide how to handle the downstream data. They need to build their own pipelines.

Companies use DB replication tools to implement one-time or recurring syncs to and from customer data stores. You can use Fivetran or Airbyte, or even Debezium to build your own log-based CDC replication, should a log be available.

You might say, at this point, that sharing database credentials with business partners is not a secure solution. And this is true. But it happens quite often as businesses need to set up reliable end-to-end cross-company pipelines— particularly when one partner is short on staff.

It should be noted that AWS lets you create IAM roles to securely provide a business partner read or write access to your S3 buckets. This helps avoid sharing credentials in most cases.

Make data available by exposing an API

As a SaaS vendor, when you’re relatively sure you know what data a lot of your customers want access to, you can make that data available by exposing an API.

There are a lot of benefits to this: 1) consumers can pick and choose the data they need and access it when they need it, 2) there are standards for APIs that everyone can implement, like REST or GraphQL, 3) APIs are DB-independent. 4) APIs are open and not tied to a vendor, 5) the data contract is implicit in an API 6) APIs are easily testable, by both sides of the transaction. These are all properties of good data transfer technology.

Pull data by implementing an API

As a SaaS vendor partner, you can implement an API to pull your data from vendors. It turns out that’s a whole lot of work. APIs and schemas evolve a lot. The good news is that you can also use tools like Fivetran, Airbyte, Meltano, or Rudderstack that handle this for you.

Send data via an API

Moving data out of the warehouse and into operational systems like MailChimp and Braze is useful. These systems provide some sort of value on top of raw data. This is made possible by implementing the API they expose to send data. Given the work involved in maintaining that code, it’s simpler to use “reverse ETL” or “data activation” tools like Census, Hightouch, and Rudderstack.

Send data by implementing an SDK

You can integrate with a vendor’s existing SDK to send events data. For example, Braze’s backend service calculate metrics based on the SDK data it receives. Implementing SDKs is not easy and the integrations take months, even up to an entire year. Adding new data points also means new code to be tested, deployed, and shipped. But the data from implementing SDKs is reliable.

When transferring data is not necessary, sharing data is way more efficient. However, sharing works in very specific scenarios.

You can share data for read purposes only across accounts (which can span regions but of course needs to be on the same cloud), on both Snowflake and Redshift. The great thing about it is that it’s instant access. There’s no data copies or data movement. It’s a view of existing data and so any changes are captured instantaneously by the view. Because data never leaves your servers, you also get straightforward access to usage metrics. You can restrict or revoke access at any time.

With AWS Redshift, you can create data shares and share it with internal and external customers.

Companies like Stripe use Snowflake and Redshift data sharing capabilities to send all of their customers’ up-to-date data, avoiding an API integration. Of course, the API is still available for those that need it. Salesforce CDP also uses this “zero-copy” or “zero-ETL” strategy to data sharing to make their data available to Snowflake customers.

Make data available on a marketplace

With AWS Data Exchange, data vendors can provide easy and secure access to their data, with the ability to reach AWS customers. Consumers can get read access via Redshift sharing capabilities or write pipelines to export the data to S3 and use it from there.

This is Marketplace at Bruges by Samuel Prout.

With Snowflake Marketplace, data vendors can make their data available to Snowflake customers. With Snowflake data sharing, consumers get read access to the data instantly without the need for ELT integrations.

Collaborate on overlapping data with data clean rooms

Sometimes, all that business partners want is to collaborate on overlapping data without the need to fully exchange copies of data. This makes contracts easier and the whole procurement process faster.

From here.

Clean rooms allow business partners to collaborate on and analyze data in a secure environment, without having to share or reveal user-level data. For example, clean rooms can aid two business partners discover their shared customers. Further, clean rooms sometimes facilitate collaborations even before any sort of partnership contracts are signed.

With AWS Clean Rooms you can invite any AWS customer to collaborate, select datasets, and configure restrictions. You can analyze data with up to 4 parties in a single collaboration. You can set minimum aggregation thresholds while allowing collaborators to run their queries.

Similarly, Snowflake, InfoSum, and LiveRamp all offer some flavor of clean rooms. Some companies also implement their own. For example, Disney, Unilever, Hershey’s have all built out their own clean rooms to be able to collaborate with marketers and retailers.

But challenges still remain

Data exchange via Excel or CSV loses valuable type information

CSV suppresses type information that needs to be inferred later on when data is being uploaded back to the database or warehouse.

Data validation is manual in most cases

This is an issue on both sides of the transaction. This is not an issue for API-based integration. And it’s one of the main areas of strength for APIs.

Everywhere else, there is always the risk that your business partner might mistakenly send you data they should not be sending. And this is usually PII (personal identifiable information) or PHI (protected health information) where you don’t have the necessary contracts set into place, but also data for other parts of the business you may not need.

When you talk to data practitioners, you’ll see that they’ve been on the receiving side of a lot of PII data that customers just send by accident.

Not all data providers expose an API

Building and managing APIs is not easy. Keeping the API up-to-date and backwards compatible is also time consuming and requires a major commitment from the API provider. We can’t expect a partner API to be available for every data set we need from a business partner.

Not all vendor APIs are implemented by major integration companies

Connectors for all possible APIs under the sun just doesn’t exist. As a data consumer, implementing APIs and keeping the integration code up-to-date is very time consuming. Hence the need for integration companies in the first place.

Some data consumers are not staffed adequately

Data consumers are sometimes not staffed appropriately to manage the data they receive. This is especially an issue if data is sent via SFTP.

SFTP data exchange only handles one half of the transaction. Many data consumers are not savvy enough to manage the pipeline on their side. In many cases, you’ll find that data is downloaded onto a laptop before being uploaded to the cloud.

At a previous company, we needed to send data on a monthly cadence instead of an hourly one, because our partner only had a single staff member to manually download the data. In the meantime, our partner wanted us to provide real time dashboards to show how they were doing before they were able to run their own analytics on the raw data they were going to receive at the end of the month.

Building SDKs and APIs, but also integrating with them to send and receive data are all time consuming and require expertise.

Some data consumers ask for an obscure format

This adds the additional burden of obscure data transformations when building a transfer pipeline with a new business partner.

Pricing an exchange is complex. Who should pay? And what amount?

When setting up transfer pipelines, which side of the transaction should be paying for the pipelines, the storage, and the potential egress costs? Accounting for long-running and diverse transactions between two parties can quickly become complex.

Sharing credentials makes it even more difficult to do cost accounting.

Speed is a challenge

How fresh do we need the data to be? How quickly do we need the data to be made available in each batch? Should speed guarantees be included in partner contracts? It’s a challenge for teams to estimate what is possible right off the bat.

Security is a challenge. So is compliance

There is a lot to unroll here. Sharing credentials is not secure. Not knowing the location to which a file from SFTP is downloaded before it makes its way to the warehouse is worrisome. APIs expose backend data and are constantly a source for data breaches. We also see that without a thorough testing strategy, many partners tend to send or receive PII data that was intended to be sent as part of the contract.

Monitoring and maintaining so many different pipelines is a challenge

Partnership contracts are updated all the time. It’s a huge hassle to have to evolve pipelines alongside the evolving contracts.

Further, schemas change, APIs change, servers fail, and so maintaining the ever-growing list of highly diverse pipelines quickly becomes a burden on the business. Businesses end up in a place where they have a different type of pipeline for each business partner, and sometimes multiple types of pipelines for the same partner.

Auditing is a challenge

In a world where a company must maintain all the various disparate pipelines, auditing becomes increasingly complex and time consuming.

Decision fatigue is real

When you start speaking to a new business partner, or an old business partner about a new use case, you need to go through the list of all possible approaches, depending on the amount of time you have, number of people they have, your infrastructure, their infrastructure, the volume of data being transferred, the direction of the transfer, the cadence, the source, the destination, the required security and speed guarantees, and so on.

There’s a lot to consider!

What are the properties of a solution?

Now that we talked about all of the challenges, what are the properties of a good solution?

Here’s my wishlist.

The solution would manage the end-to-end cross-company pipeline, such that it is not relying on either partner to be particularly experienced in data transfer.
The solution would be cloud- and technology agnostic. Business partners shouldn’t need to consider each other’s tech stack before signing a contract.
The solution would make it easy to monitor and test all pipelines and to manage them all in one place. Both partners need to be alerted if a critical pipeline is failing.
A good solution would allow an easy way to incorporate the evolving contracts. The nature of partnerships change all the time.
The solution would maintain a ledger of all transactions that happen with a specific partner.
Lastly, yet most important, is that a good solution would make it easy for companies to stay secure and compliant. That’s easy to do when we have the right tools at our disposal.

Reactions to the talk

Below are reactions to the talk.

Thanks TJ Murphy for the shoutout.

Thank you for the support Abhi Sivasailam!

Thank you for the words of encouragement Matthew Mullins!

Thanks for the shoutout Ananth Packkildurai! Big fan of the DB comparison and of Data Engineering Weekly!

This post was originally published on Medium and is cross-posted here.

Models for integrating data science teams within organizations

2019-07-31T00:00:00+00:00

Introduction
The center-of-excellence model
Accounting model
- Drawbacks
- Benefits
The consultant model
- Benefits
- Drawbacks
The embedded model
- Benefits
- Drawbacks
The democratic model
- Benefits
- Drawbacks
The product data science model
- Benefits
- Drawbacks
Conclusion
References
Acknowledgements
Citations and coverage

Introduction

At our inaugural DS Crit meeting at Twitter HQ.

Beginning in the first decade of the 21st century, internet companies were able to gain visibility into the business in ways never possible in the age of spreadsheets and relational database management systems. No longer did they need to wait for end-of-quarter financial results in order to gauge business performance; and no more did they need to rely on extrapolations from samples to get a comprehensive view of what was working for all customers. In addition to improved visibility into the state of the business, the new data storage and aggregation capabilities enabled companies to build data products like search engines, language processors, and recommender systems.

What became important was to determine how this work could be achieved efficiently and effectively. Designing and building a data science organization is a complex problem, particularly when determining the nature of data science interactions with stakeholders.

A DS team isn’t just the people, it is the process and the interaction of the team with the rest of the company.

— DJ Patil [2]

In this post, I compare some of the popular models of integrating data science teams within companies. In determining the best model, I take into account the following factors:

Coordination efficiency. Every team creates new sources of knowledge. Incorporating that knowledge into the business in a timely and repeatable fashion requires robust organization design. Bad designs lead to failures and inefficiencies in knowledge sharing and coordination; this directly affects the speed and cost at which work is done.

The goal of work is some output, a strategy, product, marketing plan, budget, account plan, sale, feature, etc. Communication is a way of incorporating stakeholders into a plan before it is too far along to change or the cost is too high (or coworkers too angry!)

— Steven Sinofsky on Twitter

Employee happiness. No discussion of organizational structure is complete without considering employee happiness, motivation, and growth factors. This is not just about reducing the cost of recruiting in response to employee churn, but also about providing employees with the circumstances to do creative and effective work during their tenure. Designing structures without considering employee happiness is a costly failure.

Product success. Data scientists opportunity size new ideas, design experiments and metrics, and design and tune models. They promote the correct use of data within the company. New products shipped without these considerations usually contain deficiencies in instrumentation and implementation, and are potentially misaligned with company strategy. The customer voice is not accurately represented when experiments are incorrectly assessed and metrics incorrectly crafted. The decision making process is delayed without high quality data and metrics. Machine learning projects either fail or lack in quality without data science involvement.

I make a number of assumptions. The company is a single strategic business unit (SBU). The SBU is partitioned in two ways. First, it is partitioned into independent functions, with respect to specialization and responsibilities. Each function (e.g. design, marketing, or sales) is a group managing the needs of the business within the context of their specialization and responsibilities. Second, the SBU is partitioned with respect to outputs and services, into independent products. The products are independent in that they have independent launch timelines. Using these definitions, a product team is a subset of the SBU—with cross-functional membership—responsible for delivering a product or service.

A data scientist (DS) is skilled in data engineering, data management, data analysis, and machine learning; and data science is their work. The data science function is a group of data scientists and their managers.

We are now ready to review and compare the myriad ways that the data science function has been integrated within the SBU.

The center-of-excellence model

We start with the most centralized of all other models. In the center-of-excellence (CoE) model, also known as the research model, the expectation is that the data science team works independently to identify big bets and build prototypes. Under this model, the data science team is considered to be the company’s innovation arm.

Some misconceptions

There are some misconceptions that lead companies to choosing the CoE model for their data science team.

a. PhD graduates are hired to do research. Data science teams hire many PhD and Master’s graduates. The focus of most of these programs is research, and so there is a misconception that graduates of these programs are hired to do research. However, the true motivation for hiring PhD and Master’s graduates into DS roles is different. Data science is an interdisciplinary field with a wide variety of requirements. Data scientists are usually required to handle data engineering, statistical analysis, and machine learning in highly diverse domains (e.g. Finance, Marketing, Logistics, Healthcare, Social Media). The extra years of studies in engineering, mathematics, and statistics are meant to boost their abilities in producing quality analyses, in communicating results, in working with external stakeholders, and in designing of new methodology.

b. Innovation happens in a lab. In cases where companies rightly expect innovation from the data science team, there is a misconception that the innovation arm of the company needs to be freed and independent of the day-to-day requirements of the business. When teams do not consider the company’s existing business model and infrastructure, their output does not translate into functioning products. This is why despite some historical success stories, many companies refrain from such an investment even when it is affordable. The question then arises, “who is in charge of innovation?” That will need to be the topic of a future post.

Drawbacks

There are important drawbacks to having the data science team operate within the CoE model:

a. Lack of context about the challenges of the business. Without visibility into the day-to-day decision-making challenges, purely centralized data science teams find it difficult to identify the most important problems to tackle. They focus on pie in the sky ideas while the business suffers.

b. Difficulty in closing the loop. In cases where they are successful at identifying and solving an important problem, centralized research teams find it difficult to get the solution adopted by the product teams. The adoption of the proposed solution would likely disrupt a team’s existing roadmap—as the two teams are out of sync. Resolving this conflict usually requires actions by higher management, leading to unwelcome interruptions to existing teams and their roadmaps. If higher management does not step in, research teams become demotivated.

My view is everyone is on the same calendar/cadence. That’s a huge thing for me. If you don’t have that then split resources (all of them) by cadence. Teams on difference cadences can’t collaborate.

— Steven Sinofsky on Twitter

c. High cost associated with building new team to back initiative. Rather than disrupting existing roadmaps, an alternate path is to build a new product team to back a proposed solution. This team would have cross-functional membership to work on the proposals by the research team, making it a costly but valid endeavor. Valid, because ideas need to be backed by a complete team in order to be assessed correctly and quickly. It would be useless to measure the success of an idea if any part of the experience is lacking. A new feature requires design, engineering, data, marketing, comms, and sales involvement to realize its potential.

The cost of building a brand new product team further increases if the new product team does not form a partition along with existing product teams. Forming a partition with other product teams is important, otherwise roadmaps and responsibilities would be overlapping. This puts the new team’s longevity at jeopardy as they try to figure out their raison d’être—while confusing other teams.

d. Non-recurring and nondeterministic output. Under the research model, the product teams might be able to adopt and find value in a single output from the data science team, but wonder if there would be follow-through and continued feedback if they were to go ahead and make the proposed changes.

Benefits and success scenarios

It should be noted that the CoE model works for many types of teams. Centralization helps focus and agency. You should fully centralize that which you can clearly encapsulate from the rest of the organization. Full centralization works when coupling is low and joint meetings are few and far between.

As an example, consider tooling development teams. Once the company decides on a technology or programming language, tooling improvement efforts can happen more or less independently of product launch timelines.

Another example of a successful CoE team is Microsoft Research, a subsidiary of Microsoft. Formed in 1991, there is no expectation that the institute produce any result that would be applicable to core Microsoft products. It turns out that Microsoft is leading the patent race in AI as a result of its investment in a research institute.

Accounting model

In the accounting model, also known as the BI model, the central data science team produces reports and presentations on a recurring basis (usually monthly and quarterly). The data science team would inform the company of notable movements in top-level metrics. Once the team identifies an interesting or worrying trend, they would work with product teams to investigate the root cause. Thus, quite frequently, playing detective becomes a main activity of the data science team under the accounting model.

Drawbacks

There are three main drawbacks to this model:

a. Difficulty in attribution and closing the loop. As mentioned above, it is near impossible to reason based on global trends. This drawback becomes particularly pronounced when there are many product teams and hence many moving parts and levers.

b. Reorganization and the emergence of tiger teams. It is important to have analyses and metrics which are tied to levers (product teams) so that they are actionable quickly and with less cost and reorganization needs. Reorganization happens and new “tiger” teams emerge when the data science team is unable to identify the culprit and existing product teams are unable to own and prioritize a fix.

Tiger teams rarely form a partition with existing product teams and thus disrupt the flow of the organization. The emergence of tiger teams is a drawback of all fully centralized models.

c. Underutilizing technology. Having monthly and quarterly reports be the only function of the data science team is failing to fully gauge product quality before reaching certain calendar milestones. If launches are leading to less usage in a particular market, the drop happens a launch at a time, not a quarter or a week at a time. A product opens up to misuse a launch at a time. Data security is breached a launch at a time. Identifying the launch that led to decreased usage in Japan after many launches is an impossible task; so is determining the launch that created incorrect incentives for abusive behavior.

d. Low quality and stale data. Every launch creates new sources of data that need to be incorporated back into existing metrics, considered in future analyses, and incorporated in existing and future models. Accountant data scientists miss all important updates, and usually rely on stale data for analyses. It is difficult to be involved in instrumentation from the sidelines. This is a drawback of all fully centralized models.

Benefits

Reporting on quarterly trends of company metrics is valuable practice. The centralized aspect of the BI team allows for a holistic view of the SBU, inspiring decisions that lead to global optimizations that can balance and correct local decisions. This work is something that the data science team should be tackling as part of their charter, regardless of the model under practice.

The consultant model

In the consultant model, the central data science team is assigned tickets or emailed with questions. Data science managers then prioritize the tickets and questions and assign them to data scientists.

Benefits

In this model, the data science manager overrides any existing data science roadmaps to prioritize the questions and needs of stakeholders. Due to the symmetrical treatment of all members of the team, this model makes managing a data science team easy and cheap.

Drawbacks

There are many drawbacks with this model:

a. Communications overhead. Data scientists in a consulting position usually lack the context to resolve questions effectively in a timely manner. There is communications overhead involved in gaining familiarity with data sources and their creation process. Further, if a follow-up to an analysis is needed and the original consultant data scientist has other ongoing commitments, the work will get assigned to another data scientist. This requires yet another onboarding investment—and thus the cycle continues.

b. Unclear deadlines. It is difficult for stakeholders to know when work would get prioritized and assigned to a consultant data scientist. Additionally, the processes affecting the volume of incoming requests are not transparent to the data science team and their managers. Even after work gets assigned and prioritized, it is difficult for the data scientist to be able to estimate the amount of time needed to answer questions due to their unfamiliarity with the limitations and nuances of the ever-changing data sources.

c. Short-term ownership. Innovation happens when people plan for years, not days and weeks. Having data scientists act as short-term consultants makes it difficult to incentivize focus on complex or tedious work. This work is needed to ensure quality data, quality experimentation tools, quality data manipulation and visualization tools, and quality results.

d. Unclear ownership. A by-product of short-term ownership is unclear ownership. When projects are one-off and seemingly random, people are more likely to step on each other’s toes. This happens inadvertently but is a non-negligible source of inefficiency. It should be noted that this is a drawback of all fully centralized models.

e. Lack of motivation and unfulfilling work. Data scientists working under this model usually lack motivation as they are rarely involved in the product decision making process. They also usually find the work unfulfilling as they rarely see the results and impact of their work.

f. Low data quality and recurring emergencies. Without maintaining good data practices, products that rely on data as input fall prey to recurring bugs and emergencies. In this model, data scientists are pulled into a project in order to play detective and identify the source of the bug.

Apart from the unfamiliarity of the data scientist with the data creation process and the product’s evolution, there is also the possibility of missing data due to missing instrumentation. It is impossible to find a needle in a haystack when the needle is not instrumented. It is also painful to look for a needle in a haystack that is extended to the fourth dimension (of time).

Finding the culprit is nearly impossible in these situations. As discussed in the drawbacks of the accounting model, the organization usually responds by creating a tiger team, thus inducing further distractions and costs.

g. Unclear coverage of product areas. There are many allocation and prioritization challenges under this model. How does work get prioritized by the data science manager? Which product teams get the most attention—the successful products or the struggling ones? Which decisions are made with data in mind, which are made without, and who would be making these global decisions when the data science team lacks visibility?

h. No clear sizing and allocation strategy. As with any fully centralized model, it is always difficult to determine the number of data scientists needed. Does the size of the team grow with the size of the organization or with the number of requests? If the latter, how does one estimate the number of the requests and total scope? There is no simple strategy for determining the size of a fully centralized consultant team.

The embedded model

In this model, product teams hire their own data scientists. Each engineering manager is in charge of planning for data scientist headcount, hiring, allocation, and roadmap. The data scientist within each product team has the engineering team members as their peers.

Benefits

This model brings welcome independence to the teams and relieves the SBU of the management requirements of a fully centralized data science team. It solves problems with team sizing and communications by distributing responsibility. It also solves the ownership and motivation issues that exist in fully centralized models.

Drawbacks

While there are reductions in data science management costs, this model has important drawbacks:

a. Management complexity. Title and role diversity on the same team lead to management headaches. It is difficult for a single manager to maintain and assess multiple career ladders for different members of the team; managers rarely get it right even with a single ladder. Usually, the engineering manager is inadvertently biased towards assessment against the more common requirements—those of the standard engineering ladder. This incentivizes the data scientist to take on a role symmetrical to the other engineers on the team, undermining the original point of hiring a data scientist. Additionally, hiring data scientists, putting together the right interview panel, and on-boarding data scientists are all important challenges within this model.

b. Mentorship deficit and difficulty in maintaining uniform data standards and best practices. Data scientists benefit and learn from working closely with their peers, in particular during analysis reviews. An embedded model does not readily offer a path to a recurring and persistent relationship among data scientists. Further, independent data scientists on each team would design their own processes and standards. It should be noted that weak standards is a drawback of all fully decentralized models.

c. Underutilizing technology and data science de-prioritization. Some teams might put off hiring a data scientist due to pressing deadlines and costs. In the absence of good data, services are still deployed. This leads to important shortcomings in data quality and data products that becomes cumbersome and oftentimes impossible to fix later.

d. Local rather than global optimization. When there is no central ownership over metrics and key results, teams choose metrics and projects that lead to local optimization. Further, in this model, teams are incentivized to compete and ignore cannibalization effects. Local optimization is a drawback of all fully decentralized models.

The democratic model

In this model, it is believed that easy and straightforward access to data by product managers, designers, engineering managers, and engineers would lessen or remove the need for a data science role. Many identify the need for data scientists to be due to the lack of proper infrastructure for fast and easy dashboard creation.

Benefits

It is valuable to invest in data infrastructure and tooling that makes data access, processing, and visualization simpler everyday. This investment is particularly valuable to data scientists as it frees up time for proactive opportunity sizing, experiment design, metric design, model design, and general improvements in methodology.

Drawbacks

While ensuring everyone has direct and easy access to data is a noble goal, there are some drawbacks to this model:

a. Difficulty in mastering everything and maintaining data best practices. Usually, people are mostly specialized and interested in a particular set of tasks. Being skilled at a company’s engineering stack is already a big feat. It is fine to offload design work and sales work and data work. Data scientists enforce good data practices within the organization.

b. Dashboards are not the goal of data science, they are an intermediary step in the exploration of data.

The laws of shitty dashboards * Attack with Numbers (Wayback snapshot)
Disclosure: I have been responsible for building shitty dashboards. I personally made most of the errors below. I…

The product data science model

Between the extremes of the fully centralized model (the CoE model) and the fully decentralized model (the embedded model), there exists a spectrum of hybrid models that take characteristics from each of the aforementioned models. Taking advantage of the strengths of both models, while actively making up for their deficiencies is what makes hybrid models successful.

The product data science (PDS) model is inspired, only in part, by the matrix structure. Individuals are simultaneously members of the data science function and a product team. Data scientists—although each a member of a product team—report only to a central data science management team. Thus, unlike the matrix structure, there is unity of command under the PDS model.

Revisiting the assumptions we enumerated at the beginning, in the PDS model, the cross-functional product team would include data scientists.

Benefits

a. Clear ownership, actionable insights, and speed. One important benefit of the PDS model is clear ownership of projects by the data scientists, due to their membership in the various product teams. Membership in each product team gives data scientists a thorough understanding of that product, its limits, and its potential. This in turn allows a straightforward mapping of analysis to proposals for action. It is difficult to move fast if newly available insight does not map into reasonable and informed actions.

b. Quality data and quality data products. Data scientists close collaborations with a product team improves data quality. Every single launch changes the data, and so it is important to oversee its evolution with careful instrumentation.

Pete Skomoroch on Twitter

c. Standardized data science processes. Data science peers, working on different product teams, come together to establish best practices and onboarding flows within the data science team. They review one another’s code and analyses. They collaborate on complex projects. They benefit from a unified career ladder, with managers who can assess their impact and can plan for their growth.

d. Global optimization. The direct and recurring collaboration of data science peers from various product teams has other benefits. Due to their collective birds-eye-view of the business, they are able to connect the dots, identify inconsistencies, and optimize globally. This is similar to the way design teams should operate.

e. Sizing and allocation clarity. Another benefit of the PDS model is that it simplifies the task of determining the size of the data science team. Once you figure out how to partition the SBU into product teams, and you figure out the number of cross-functional stakeholders per product, the allocation of data scientists can be determined as a proportion. More available here.

Drawbacks

No model is perfect and each have their drawbacks.

Since there is no optimal or perfect organizational structure […] then the most important thing is to know the weaknesses of your structure and to compensate for them.

— Steven Sinofsky, Functional versus Unit Organizations

Below are some drawbacks of the PDS model.

a. Cost. One of the main arguments against the PDS model is the cost of hiring at least one data scientist for every product team; and the associated cost of a centralized data science management team. This assessment does not take into account the savings stemming from the increase in product and data quality, and the more effective use of data for the business. Having said that, organizations should do what they can afford. In the beginning everyone is responsible for engineering and data and design needs. As the SBU grows, one can have specialized functions handling each set of responsibilities.

b. Recurring conflicts due to lack of power parity. For success on cross-functional teams, all functional leads should have similar amounts of negotiating power. Without power parity, the benefits of cross-functional collaboration are lessened due to recurring conflicts, lack of context, late delivery, and thus suboptimal results. Power parity is ensured by parity in reporting structure and compensation. Many companies, to this day, lack data science representation at the executive level of the SBU.

Note that the importance of power parity is particularly pronounced when the company is behind on Data Science investments—whether it be in data infrastructure or in people. For data, the expectations are high and the stakeholders numerous. Short-term and long-term planning for the function needs to happen by someone who not only understands the requirements and challenges but is empowered to correctly and efficiently steer the data culture of an existing company.

c. Information overload and the data science manager. Gaining the right amount of knowledge about all products supported by the data science team is not straightforward. However, managers need to be informed and curious about the areas under their purview to be able to build a roadmap and effectively assess contributions, investments, timelines, and tradeoffs. They also need to be able to continually communicate the contract between the data science and stakeholder teams and be mindful of the team’s portfolio. This is a responsibility of the managers of every functional team—not just data science.

The drawbacks of the PDS model have relatively straightforward solutions, as described above.

Conclusion

Where an SBU is involved, I recommend the PDS model as the best in effectiveness and efficiency in leveraging data for the business.

The PDS model is compliant with Grove’s Law.

All large organizations with a common business purpose end up in a hybrid organizational form.

— Andy Grove

It is also aligned with Hayek’s views on the use of knowledge in society, where he motivates the need for a hybrid approach to organization and decision making. Neither end of the spectrum sufficiently meets the speed and context requirements of decision making in society.

We cannot expect that this problem will be solved by first communicating all this knowledge to a central board which, after integrating all knowledge, issues its orders. We must solve it by some form of decentralization. But this answers only part of our problem. We need decentralization because only thus can we insure that the knowledge of the particular circumstances of time and place will be promptly used. But the “man on the spot” cannot decide solely on the basis of his limited but intimate knowledge of the facts of his immediate surroundings. There still remains the problem of communicating to him such further information as he needs to fit his decisions into the whole pattern of changes of the larger economic system.

— F.A. Hayek

P.S. I had an easier time saying all of this in a Tweet,

Embedded for context, relevance, communication efficiency, and to be in sync. Centralized for hiring and promotion purposes, for peer review, for sharing and maintaining best practices [, for global optimization, and to align on strategy].

— @djpardis on Twitter

References

[1] Functional versus Unit Organizations by Steven Sinofsky

[2] Building Data Science Teams by DJ Patil

[3] Where should you put your data scientists by Daniel Tunkelang

[4] How to play well with others by Josh Wills

Acknowledgements

Thanks to Raki Wane, Peter Skomoroch, Sayan Sanyal, Josh Montague, Chris Albon, Josh Silverman, and Harish Krishnan for reviewing and providing valuable feedback.

Citations and coverage

University of Virgina Data Science (Wayback snapshot), 97 Ways (Matt Wright), Beyond the POC. How to Make Machine Learning Real in the Enterprise (Sam Charrington), Projects to Know (Amplify Partners, Sarah Catanzaro), The Data Science Roundup (Fishtown Analytics, Tristan Handy), Normcore Tech (Vicki Boykis), Femstreet (Sarah Nöckel), Linear Digressions, Analytical IQ (Adam Lorton), Hex Blog (Hex, Barry McCardel), Full Stack Deep Learning, The ML Times, nibble dispatch, Hiring Data Scientists and Machine Learning Engineers. A Practical Guide (Roy Keyes), Blog Cast (Sam Bail), dbt Blog (Erin Vaughan and Janessa Lantz), Building The Modern Data Team (Pedram Navid), Data Science Org Design for Startups (Nirant Kasliwal), On Search Leadership (Daniel Tunkelang), Building A Data Platform From Scratch At Collectors. Part 3 of 3 (Sam Bail), Modern Data Teams Hub (Amplify Partners, Emilie Schario), Data Science Management Resources (jm-contreras)

This post was originally published on Medium and is cross-posted here.

Hourly mentions of a word on Twitter

2015-05-23T00:00:00+00:00

Some time ago (OK, a month ago—time ✈️s), I saw this tweet:

Need a simple tool to track mentions of a keyword on Twitter by hour. Don’t need a bunch of bells and whistles. Thoughts?

— Kaegan Donnelly (@kaequan) • April 23, 2015

I thought, “Should be easy, lmgt.” However, results for the query “hourly mentions of a word on Twitter” didn’t offer clear solutions.

Days later I came across two relatively simple approaches to tackling the problem. The first is Tweepy. The other is Logstash.

Tweepy is an open source Python library for accessing the Twitter API, including the Twitter Streaming API.

Logstash is an open source tool for collecting, processing, and forwarding events. Logstash can read events from the Twitter Streaming API using its twitter plugin.

Having tried both, I recommend Logstash over Tweepy for two main reasons:

It deals with the Twitter API rate limits by default
It offers Elasticsearch and Kibana integration—simplifying the aggregation and visualization steps, respectively, that naturally follow the data (tweet) collection step

For both Tweepy and Logstash you need access to Twitter’s streaming API. Follow steps 2 and 3 here to create a Twitter app and obtain your Consumer Key, Consumer Key Secret, Access Token, and Access Token Secret.

The ELK solution

Download and install Elasticsearch, Logstash, and Kibana. If you are on a Mac, you can

brew install elasticsearch
brew install logstash

Make sure you have Elasticsearch and Kibana running. Before running Logstash, you need to prepare a configuration file. Below is a sample configuration file to collect tweets containing the word ireland (call it ireland.conf):

# a logstash config file has three sections:
# input{}, output{}, and (optional) filter{}; add plugins
# to specify how events should be handled in each section

input {
    twitter {
        # set key and token values from the previous step
        consumer_key => ""
        consumer_secret => ""
        oauth_token => ""
        oauth_token_secret => ""
        # assume we are interested in tracking all
        # mentions of the word "ireland"
        keywords => ["ireland"]
        # no need for all fields to get hourly counts
        full_tweet => false
    }
}

output {
	stdout {
		# include this to pretty-print the event's json to stdout
		codec => rubydebug
  	}
}

To start streaming tweets, run

logstash -f ireland.conf

At this point, tweets are written to stdout. In order to visualize tweet counts using Kibana, you need to save the tweets to Elasticsearch.

Add the elasticsearch plugin to the output section of the configuration:

output {
    elasticsearch {
        protocol => "http"
        host => "localhost"
        index => "irelandtweets"
    }

	stdout {
		# include this to pretty-print the event's json to stdout
		codec => rubydebug
  	}
}

Run Logstash again and have a look at:

http://localhost:9200/irelandtweets/_search/?pretty

Below is a sample of the output format. You can see, for example, that 65235 documents (tweets) have been stored in the irelandtweets index:

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 65235,
    "max_score" : 1.0,
    "hits" : [ {
      "_index" : "irelandtweets",
      "_type" : "logs",
      "_id" : "AU2B1MGZPj_44djTabLA",
      "_score" : 1.0,
      "_source":{"@timestamp":"2015-05-23T17:31:51.000Z","message":"Y'all have no idea how happy I am for Ireland 💗 Can my country say yes to equality too 😭","user":"LesbiForLauren","client":"<a href=\"http://twitter.com/download/iphone\" rel=\"nofollow\">Twitter for iPhone</a>","retweeted":false,"source":"http://twitter.com/LesbiForLauren/status/602165054042034176","@version":"1"}
    }, {
      "_index" : "irelandtweets",
      "_type" : "logs",
      "_id" : "AU2B1MGZPj_44djTabLF",
      "_score" : 1.0,
      "_source":{"@timestamp":"2015-05-23T17:31:51.000Z","message":"RT @muyskerm: @Jack_Septic_Eye Well done Ireland. The U.S. could take a lesson.","user":"SOUTHERNjamespb","client":"<a href=\"http://www.twitter.com\" rel=\"nofollow\">Twitter for BlackBerry</a>","retweeted":false,"source":"http://twitter.com/SOUTHERNjamespb/status/602165054889283584","@version":"1"}
    }, {
               ...

To start using Kibana, visit

http://localhost:5601/

On the Discover tab, there is a configuration form:

Check off the box: Index contains time-based events
Fill the Index name or pattern field with irelandtweets
Fill the Time-field name field with @timestamp

On the Visualize tab, choose visualization type Line chart.

Choose option From a saved search to use the same query you specified on the Discover tab
For metric aggregation (Y-Axis): Choose Count
For bucket aggregation (X-Axis): i. Aggregation: Date Histogram ii. Field: @timestamp iii. Interval: Minute
Click on the Refresh Interval tab at the top. Choose 5 seconds and see your line chart come alive 📈

Done. Thank you for starting the conversation Kaegan!

More resources

For details about Logstash plugins see this guide.

Anna Roes has written an excellent overview of Kibana in this tutorial.

Pardis Noorzad

The evolution of software engineering

Table of contents

The internet, cloud, and mobile eras put AI in context

Verification and maintenance costs determine whether AI displaces SaaS

Each past abstraction eliminated the need to acquire entire areas of knowledge

English is not a programming language

Open source creation and maintenance both benefit from AI

Languages, frameworks, and tools are consolidating, and AI may accelerate the trend

Can AI improve existing abstraction layers?

Revisiting Moneyball

Table of contents

Building an auth flow with AI

Solving the authentication challenge

Replit vs Windsurf: technical comparison

Replit features

Authentication and session management

Database integration

Deployment infrastructure

Replit limitations

Windsurf comparison

Data room platform implementation

Tech stack

Architecture decisions

Implementation results

Development timeline

Feedback request

Best practices in AI coding

The problem

The solution

The tools

Observations and best practices

Observations

Best practices

Advice from Windsurf itself

Elsewhere on the web

Next steps

References

Startup advice

The state of data exchange

Table of contents

Models for integrating data science teams within organizations

Table of contents