The Art of Unix Programming
by Eric S. Raymond
Eric Steven Raymond: The Art of Unix Programming, Addison-Wesley, 2003. ISBN 0-13-142901-9.
The book describes the design of the Unix operating system and the spirit that has grown up around it during decades of its successful use.
The book is dedicated to the creators of Unix and C:
To Ken Thompson and Dennis Ritchie, because you inspired me.
Unix Philosophy
There are no fix immutable ”Axioms“, but a set of rules and values that evolve with time as the context changes and show a pretty consistent core.
A sentence sums up the Unix philosophy: ”Keep It Simple, Stupid!“
Unix programmers have learned a view of the world in which simplicity is beauty is elegance is good, and in which complexity is ugliness is grotesquery is evil. [p. 282]
Unix tradition balances aesthetic values with practical ones.
One strain of Unix thinking emphasizes small sharp tools, starting designs from zero, and interfaces that are simple and consistent. This point of view has been most famously championed by Doug McIlroy. Another strain emphasizes doing simple implementations that work, and that ship quickly, even if the methods are brute-force and some edge cases have to be punted. Ken Thompson's code and his maxims about programming have often seemed to lean in this direction. [p. 285]
Critical Review
Chapter 20 points out some limitations of the Unix design, such as insufficient GUI support and some unclean design aspects. Some of these were addressed and elegantly resolved in Plan 9.
The Culture of Unix faces some problems, too. The author considers that the Unix people have fallen in elitism. Unix was once born inside the Time-Sharing movement that wanted to bring computer power to the people and rebeled against the priesthood of mainframes. Now Unix runs the Internet and the World Wide Web but it keeps itself apart from the mainstream of GUI interfaces and multimedia.
Macintosh programmers are all about the user experience. [...]
By contrast, Unix people are all about infrastructure. [...]
Both design philosophies have some validity, but the two camps have a great deal of difficulty seeing each other's points. The typical Unix developer's reflex is to dismiss Macintosh software as gaudy fluff, eye-candy for the ignorant, and to continue building software that appeals to ohter Unix developers. If end-users don't like it, so much the worse for the end users; they will come around when they get a clue.
Table of Contents
Copyright
Dedication
Addison-Wesley Professional Computing Series
List of Figures
List of Tables
List of Examples
Preface
Who Should Read This Book
How to Use This Book
Related References
Conventions Used in This Book
Our Case Studies
Author’s Acknowledgements
Part I: Context
Chapter 1. Philosophy: Philosophy Matters
Section 1.1. Culture? What Culture?
Section 1.2. The Durability of Unix
Section 1.3. The Case against Learning Unix Culture
Section 1.4. What Unix Gets Wrong
Section 1.5. What Unix Gets Right
Section 1.5.1. Open-Source Software
Section 1.5.2. Cross-Platform Portability and Open Standards
Section 1.5.3. The Internet and the World Wide Web
Section 1.5.4. The Open-Source Community
Section 1.5.5. Flexibility All the Way Down
Section 1.5.6. Unix Is Fun to Hack
Section 1.5.7. The Lessons of Unix Can Be Applied Elsewhere
Section 1.6. Basics of the Unix Philosophy
Section 1.6.1. Rule of Modularity: Write simple parts connected by clean interfaces.
Section 1.6.2. Rule of Clarity: Clarity is better than cleverness.
Section 1.6.3. Rule of Composition: Design programs to be connected with other programs.
Section 1.6.4. Rule of Separation: Separate policy from mechanism; separate interfaces from engines.
Section 1.6.5. Rule of Simplicity: Design for simplicity; add complexity only where you must.
Section 1.6.6. Rule of Parsimony: Write a big program only when it is clear by demonstration that nothing else will do.
Section 1.6.7. Rule of Transparency: Design for visibility to make inspection and debugging easier.
Section 1.6.8. Rule of Robustness: Robustness is the child of transparency and simplicity.
Section 1.6.9. Rule of Representation: Fold knowledge into data, so program logic can be stupid and robust.
Section 1.6.10. Rule of Least Surprise: In interface design, always do the least surprising thing.
Section 1.6.11. Rule of Silence: When a program has nothing surprising to say, it should say nothing.
Section 1.6.12. Rule of Repair: Repair what you can—but when you must fail, fail noisily and as soon as possible.
Section 1.6.13. Rule of Economy: Programmer time is expensive; conserve it in preference to machine time.
Section 1.6.14. Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.
Section 1.6.15. Rule of Optimization: Prototype before polishing. Get it working before you optimize it.
Section 1.6.16. Rule of Diversity: Distrust all claims for “one true way”.
Section 1.6.17. Rule of Extensibility: Design for the future, because it will be here sooner than you think.
Section 1.7. The Unix Philosophy in One Lesson
Section 1.8. Applying the Unix Philosophy
Section 1.9. Attitude Matters Too
Chapter 2. History: A Tale of Two Cultures
Section 2.1. Origins and History of Unix, 1969–1995
Section 2.1.1. Genesis: 1969–1971
Section 2.1.2. Exodus: 1971–1980
Section 2.1.3. TCP/IP and the Unix Wars: 1980–1990
Section 2.1.4. Blows against the Empire: 1991–1995
Section 2.2. Origins and History of the Hackers, 1961–1995
Section 2.2.1. At Play in the Groves of Academe: 1961–1980
Section 2.2.2. Internet Fusion and the Free Software Movement: 1981–1991
Section 2.2.3. Linux and the Pragmatist Reaction: 1991–1998
Section 2.3. The Open-Source Movement: 1998 and Onward
Section 2.4. The Lessons of Unix History
Chapter 3. Contrasts: Comparing the Unix Philosophy with Others
Section 3.1. The Elements of Operating-System Style
Section 3.1.1. What Is the Operating System’s Unifying Idea?
Section 3.1.2. Multitasking Capability
Section 3.1.3. Cooperating Processes
Section 3.1.4. Internal Boundaries
Section 3.1.5. File Attributes and Record Structures
Section 3.1.6. Binary File Formats
Section 3.1.7. Preferred User Interface Style
Section 3.1.8. Intended Audience
Section 3.1.9. Entry Barriers to Development
Section 3.2. Operating-System Comparisons
Section 3.2.1. VMS
Section 3.2.2. MacOS
Section 3.2.3. OS/2
Section 3.2.4. Windows NT
Section 3.2.5. BeOS
Section 3.2.6. MVS
Section 3.2.7. VM/CMS
Section 3.2.8. Linux
Section 3.3. What Goes Around, Comes Around
Part II: Design
Chapter 4. Modularity: Keeping It Clean, Keeping It Simple
Section 4.1. Encapsulation and Optimal Module Size
Section 4.2. Compactness and Orthogonality
Section 4.2.1. Compactness
Section 4.2.2. Orthogonality
Section 4.2.3. The SPOT Rule
Section 4.2.4. Compactness and the Strong Single Center
Section 4.2.5. The Value of Detachment
Section 4.3. Software Is a Many-Layered Thing
Section 4.3.1. Top-Down versus Bottom-Up
Section 4.3.2. Glue Layers
Section 4.3.3. Case Study: C Considered as Thin Glue
Section 4.4. Libraries
Section 4.4.1. Case Study: GIMP Plugins
Section 4.5. Unix and Object-Oriented Languages
Section 4.6. Coding for Modularity
Chapter 5. Textuality: Good Protocols Make Good Practice
Section 5.1. The Importance of Being Textual
Section 5.1.1. Case Study: Unix Password File Format
Section 5.1.2. Case Study: .newsrc Format
Section 5.1.3. Case Study: The PNG Graphics File Format
Section 5.2. Data File Metaformats
Section 5.2.1. DSV Style
Section 5.2.2. RFC 822 Format
Section 5.2.3. Cookie-Jar Format
Section 5.2.4. Record-Jar Format
Section 5.2.5. XML
Section 5.2.6. Windows INI Format
Section 5.2.7. Unix Textual File Format Conventions
Section 5.2.8. The Pros and Cons of File Compression
Section 5.3. Application Protocol Design
Section 5.3.1. Case Study: SMTP, a Simple Socket Protocol
Section 5.3.2. Case Study: POP3, the Post Office Protocol
Section 5.3.3. Case Study: IMAP, the Internet Message Access Protocol
Section 5.4. Application Protocol Metaformats
Section 5.4.1. The Classical Internet Application Metaprotocol
Section 5.4.2. HTTP as a Universal Application Protocol
Section 5.4.3. BEEP: Blocks Extensible Exchange Protocol
Section 5.4.4. XML-RPC, SOAP, and Jabber
Chapter 6. Transparency: Let There Be Light
Section 6.1. Studying Cases
Section 6.1.1. Case Study: audacity
Section 6.1.2. Case Study: fetchmail’s -v option
Section 6.1.3. Case Study: GCC
Section 6.1.4. Case Study: kmail
Section 6.1.5. Case Study: SNG
Section 6.1.6. Case Study: The Terminfo Database
Section 6.1.7. Case Study: Freeciv Data Files
Section 6.2. Designing for Transparency and Discoverability
Section 6.2.1. The Zen of Transparency
Section 6.2.2. Coding for Transparency and Discoverability
Section 6.2.3. Transparency and Avoiding Overprotectiveness
Section 6.2.4. Transparency and Editable Representations
Section 6.2.5. Transparency, Fault Diagnosis, and Fault Recovery
Section 6.3. Designing for Maintainability
Chapter 7. Multiprogramming: Separating Processes to Separate Function
Section 7.1. Separating Complexity Control from Performance Tuning
Section 7.2. Taxonomy of Unix IPC Methods
Section 7.2.1. Handing off Tasks to Specialist Programs
Section 7.2.2. Pipes, Redirection, and Filters
Section 7.2.3. Wrappers
Section 7.2.4. Security Wrappers and Bernstein Chaining
Section 7.2.5. Slave Processes
Section 7.2.6. Peer-to-Peer Inter-Process Communication
Section 7.3. Problems and Methods to Avoid
Section 7.3.1. Obsolescent Unix IPC Methods
Section 7.3.2. Remote Procedure Calls
Section 7.3.3. Threads—Threat or Menace?
Section 7.4. Process Partitioning at the Design Level
Chapter 8. Minilanguages: Finding a Notation That Sings
Section 8.1. Understanding the Taxonomy of Languages
Section 8.2. Applying Minilanguages
Section 8.2.1. Case Study: sng
Section 8.2.2. Case Study: Regular Expressions
Section 8.2.3. Case Study: Glade
Section 8.2.4. Case Study: m4
Section 8.2.5. Case Study: XSLT
Section 8.2.6. Case Study: The Documenter’s Workbench Tools
Section 8.2.7. Case Study: fetchmail Run-Control Syntax
Section 8.2.8. Case Study: awk
Section 8.2.9. Case Study: PostScript
Section 8.2.10. Case Study: bc and dc
Section 8.2.11. Case Study: Emacs Lisp
Section 8.2.12. Case Study: JavaScript
Section 8.3. Designing Minilanguages
Section 8.3.1. Choosing the Right Complexity Level
Section 8.3.2. Extending and Embedding Languages
Section 8.3.3. Writing a Custom Grammar
Section 8.3.4. Macros—Beware!
Section 8.3.5. Language or Application Protocol?
Chapter 9. Generation: Pushing the Specification Level Upwards
Section 9.1. Data-Driven Programming
Section 9.1.1. Case Study: ascii
Section 9.1.2. Case Study: Statistical Spam Filtering
Section 9.1.3. Case Study: Metaclass Hacking in fetchmailconf
Section 9.2. Ad-hoc Code Generation
Section 9.2.1. Case Study: Generating Code for the ascii Displays
Section 9.2.2. Case Study: Generating HTML Code for a Tabular List
Chapter 10. Configuration: Starting on the Right Foot
Section 10.1. What Should Be Configurable?
Section 10.2. Where Configurations Live
Section 10.3. Run-Control Files
Section 10.3.1. Case Study: The .netrc File
Section 10.3.2. Portability to Other Operating Systems
Section 10.4. Environment Variables
Section 10.4.1. System Environment Variables
Section 10.4.2. User Environment Variables
Section 10.4.3. When to Use Environment Variables
Section 10.4.4. Portability to Other Operating Systems
Section 10.5. Command-Line Options
Section 10.5.1. The -a to -z of Command-Line Options
Section 10.5.2. Portability to Other Operating Systems
Section 10.6. How to Choose among the Methods
Section 10.6.1. Case Study: fetchmail
Section 10.6.2. Case Study: The XFree86 Server
Section 10.7. On Breaking These Rules
Chapter 11. Interfaces: User-Interface Design Patterns in the Unix Environment
Section 11.1. Applying the Rule of Least Surprise
Section 11.2. History of Interface Design on Unix
Section 11.3. Evaluating Interface Designs
Section 11.4. Tradeoffs between CLI and Visual Interfaces
Section 11.4.1. Case Study: Two Ways to Write a Calculator Program
Section 11.5. Transparency, Expressiveness, and Configurability
Section 11.6. Unix Interface Design Patterns
Section 11.6.1. The Filter Pattern
Section 11.6.2. The Cantrip Pattern
Section 11.6.3. The Source Pattern
Section 11.6.4. The Sink Pattern
Section 11.6.5. The Compiler Pattern
Section 11.6.6. The ed pattern
Section 11.6.7. The Roguelike Pattern
Section 11.6.8. The ’Separated Engine and Interface’ Pattern
Section 11.6.9. The CLI Server Pattern
Section 11.6.10. Language-Based Interface Patterns
Section 11.7. Applying Unix Interface-Design Patterns
Section 11.7.1. The Polyvalent-Program Pattern
Section 11.8. The Web Browser as a Universal Front End
Section 11.9. Silence Is Golden
Chapter 12. Optimization
Section 12.1. Don’t Just Do Something, Stand There!
Section 12.2. Measure before Optimizing
Section 12.3. Nonlocality Considered Harmful
Section 12.4. Throughput vs. Latency
Section 12.4.1. Batching Operations
Section 12.4.2. Overlapping Operations
Section 12.4.3. Caching Operation Results
Chapter 13. Complexity: As Simple As Possible, but No Simpler
Section 13.1. Speaking of Complexity
Section 13.1.1. The Three Sources of Complexity
Section 13.1.2. Tradeoffs between Interface and Implementation Complexity
Section 13.1.3. Essential, Optional, and Accidental Complexity
Section 13.1.4. Mapping Complexity
Section 13.1.5. When Simplicity Is Not Enough
Section 13.2. A Tale of Five Editors
Section 13.2.1. ed
Section 13.2.2. vi
Section 13.2.3. Sam
Section 13.2.4. Emacs
Section 13.2.5. Wily
Section 13.3. The Right Size for an Editor
Section 13.3.1. Identifying the Complexity Problems
Section 13.3.2. Compromise Doesn’t Work
Section 13.3.3. Is Emacs an Argument against the Unix Tradition?
Section 13.4. The Right Size of Software
Part III: Implementation
Chapter 14. Languages: To C or Not To C?
Section 14.1. Unix’s Cornucopia of Languages
Section 14.2. Why Not C?
Section 14.3. Interpreted Languages and Mixed Strategies
Section 14.4. Language Evaluations
Section 14.4.1. C
Section 14.4.2. C++
Section 14.4.3. Shell
Section 14.4.4. Perl
Section 14.4.5. Tcl
Section 14.4.6. Python
Section 14.4.7. Java
Section 14.4.8. Emacs Lisp
Section 14.5. Trends for the Future
Section 14.6. Choosing an X Toolkit
Chapter 15. Tools: The Tactics of Development
Section 15.1. A Developer-Friendly Operating System
Section 15.2. Choosing an Editor
Section 15.2.1. Useful Things to Know about vi
Section 15.2.2. Useful Things to Know about Emacs
Section 15.2.3. The Antireligious Choice: Using Both
Section 15.3. Special-Purpose Code Generators
Section 15.3.1. yacc and lex
Section 15.3.2. Case Study: Glade
Section 15.4. make: Automating Your Recipes
Section 15.4.1. Basic Theory of make
Section 15.4.2. make in Non-C/C++ Development
Section 15.4.3. Utility Productions
Section 15.4.4. Generating Makefiles
Section 15.5. Version-Control Systems
Section 15.5.1. Why Version Control?
Section 15.5.2. Version Control by Hand
Section 15.5.3. Automated Version Control
Section 15.5.4. Unix Tools for Version Control
Section 15.6. Runtime Debugging
Section 15.7. Profiling
Section 15.8. Combining Tools with Emacs
Section 15.8.1. Emacs and make
Section 15.8.2. Emacs and Runtime Debugging
Section 15.8.3. Emacs and Version Control
Section 15.8.4. Emacs and Profiling
Section 15.8.5. Like an IDE, Only Better
Chapter 16. Reuse: On Not Reinventing the Wheel
Section 16.1. The Tale of J. Random Newbie
Section 16.2. Transparency as the Key to Reuse
Section 16.3. From Reuse to Open Source
Section 16.4. The Best Things in Life Are Open
Section 16.5. Where to Look?
Section 16.6. Issues in Using Open-Source Software
Section 16.7. Licensing Issues
Section 16.7.1. What Qualifies as Open Source
Section 16.7.2. Standard Open-Source Licenses
Section 16.7.3. When You Need a Lawyer
Part IV: Community
Chapter 17. Portability: Software Portability and Keeping Up Standards
Section 17.1. Evolution of C
Section 17.1.1. Early History of C
Section 17.1.2. C Standards
Section 17.2. Unix Standards
Section 17.2.1. Standards and the Unix Wars
Section 17.2.2. The Ghost at the Victory Banquet
Section 17.2.3. Unix Standards in the Open-Source World
Section 17.3. IETF and the RFC Standards Process
Section 17.4. Specifications as DNA, Code as RNA
Section 17.5. Programming for Portability
Section 17.5.1. Portability and Choice of Language
Section 17.5.2. Avoiding System Dependencies
Section 17.5.3. Tools for Portability
Section 17.6. Internationalization
Section 17.7. Portability, Open Standards, and Open Source
Chapter 18. Documentation: Explaining Your Code to a Web-Centric World
Section 18.1. Documentation Concepts
Section 18.2. The Unix Style
Section 18.2.1. The Large-Document Bias
Section 18.2.2. Cultural Style
Section 18.3. The Zoo of Unix Documentation Formats
Section 18.3.1. troff and the Documenter’s Workbench Tools
Section 18.3.2. TEX
Section 18.3.3. Texinfo
Section 18.3.4. POD
Section 18.3.5. HTML
Section 18.3.6. DocBook
Section 18.4. The Present Chaos and a Possible Way Out
Section 18.5. DocBook
Section 18.5.1. Document Type Definitions
Section 18.5.2. Other DTDs
Section 18.5.3. The DocBook Toolchain
Section 18.5.4. Migration Tools
Section 18.5.5. Editing Tools
Section 18.5.6. Related Standards and Practices
Section 18.5.7. SGML
Section 18.5.8. XML-DocBook References
Section 18.6. Best Practices for Writing Unix Documentation
Chapter 19. Open Source: Programming in the New Unix Community
Section 19.1. Unix and Open Source
Section 19.2. Best Practices for Working with Open-Source Developers
Section 19.2.1. Good Patching Practice
Section 19.2.2. Good Project- and Archive-Naming Practice
Section 19.2.3. Good Development Practice
Section 19.2.4. Good Distribution-Making Practice
Section 19.2.5. Good Communication Practice
Section 19.3. The Logic of Licenses: How to Pick One
Section 19.4. Why You Should Use a Standard License
Section 19.5. Varieties of Open-Source Licensing
Section 19.5.1. MIT or X Consortium License
Section 19.5.2. BSD Classic License
Section 19.5.3. Artistic License
Section 19.5.4. General Public License
Section 19.5.5. Mozilla Public License
Chapter 20. Futures: Dangers and Opportunities
Section 20.1. Essence and Accident in Unix Tradition
Section 20.2. Plan 9: The Way the Future Was
Section 20.3. Problems in the Design of Unix
Section 20.3.1. A Unix File Is Just a Big Bag of Bytes
Section 20.3.2. Unix Support for GUIs Is Weak
Section 20.3.3. File Deletion Is Forever
Section 20.3.4. Unix Assumes a Static Filesystem
Section 20.3.5. The Design of Job Control Was Badly Botched
Section 20.3.6. The Unix API Doesn’t Use Exceptions
Section 20.3.7. ioctl(2) and fcntl(2) Are an Embarrassment
Section 20.3.8. The Unix Security Model May Be Too Primitive
Section 20.3.9. Unix Has Too Many Different Kinds of Names
Section 20.3.10. File Systems Might Be Considered Harmful
Section 20.3.11. Towards a Global Internet Address Space
Section 20.4. Problems in the Environment of Unix
Section 20.5. Problems in the Culture of Unix
Section 20.6. Reasons to Believe
Appendix
Appendix A. Glossary of Abbreviations
Appendix B. References
Appendix C. Contributors
Appendix D. Rootless Root: The Unix Koans of Master Foo
Editor’s Introduction
Master Foo and the Ten Thousand Lines
Master Foo and the Script Kiddie
Master Foo Discourses on the Two Paths
Master Foo and the Methodologist
Master Foo Discourses on the Graphical User Interface
Master Foo and the Unix Zealot
Master Foo Discourses on the Unix-Nature
Master Foo and the End User
Index
