Main Learning Python: powerful object-oriented programming
Learning Python: powerful object-oriented programmingMark Lutz
Get a comprehensive, in-depth introduction to the core Python language with this hands-on book. Based on author Mark Lutz’s popular training course, this updated fifth edition will help you quickly write efficient, high-quality code with Python. It’s an ideal way to begin, whether you’re new to programming or a professional developer versed in other languages.
Complete with quizzes, exercises, and helpful illustrations, this easy-to-follow, self-paced tutorial gets you started with both Python 2.7 and 3.3—the latest releases in the 3.X and 2.X lines—plus all other releases in common use today. You’ll also learn some advanced language features that recently have become more common in Python code.
• Explore Python’s major built-in object types such as numbers, lists, and dictionaries
• Create and process objects with Python statements, and learn Python’s general syntax model
• Use functions to avoid code redundancy and package code for reuse
• Organize statements, functions, and other tools into larger components with modules
• Dive into classes: Python’s object-oriented programming tool for structuring code
• Write large programs with Python’s exception-handling model and development tools
• Learn advanced Python tools, including decorators, descriptors, metaclasses, and Unicode processing
Complete with quizzes, exercises, and helpful illustrations, this easy-to-follow, self-paced tutorial gets you started with both Python 2.7 and 3.3—the latest releases in the 3.X and 2.X lines—plus all other releases in common use today. You’ll also learn some advanced language features that recently have become more common in Python code.
• Explore Python’s major built-in object types such as numbers, lists, and dictionaries
• Create and process objects with Python statements, and learn Python’s general syntax model
• Use functions to avoid code redundancy and package code for reuse
• Organize statements, functions, and other tools into larger components with modules
• Dive into classes: Python’s object-oriented programming tool for structuring code
• Write large programs with Python’s exception-handling model and development tools
• Learn advanced Python tools, including decorators, descriptors, metaclasses, and Unicode processing
You may be interested in
Most frequently terms
FIFTH EDITION Learning Python Mark Lutz Beijing • Cambridge • Farnham • Köln • Sebastopol • Tokyo Learning Python, Fifth Edition by Mark Lutz Copyright © 2013 Mark Lutz. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or email@example.com. Editor: Rachel Roumeliotis Production Editor: Christopher Hearse Copyeditor: Rachel Monaghan Proofreader: Julie Van Keuren June 2013: Indexer: Lucie Haskins Cover Designer: Randy Comer Interior Designer: David Futato Illustrator: Rebecca Demarest Fifth Edition. Revision History for the Fifth Edition: 2013-06-07 First release See http://oreilly.com/catalog/errata.csp?isbn=9781449355739 for release details. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. Learning Python, 5th Edition, the image of a wood rat, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. ISBN: 978-1-449-35573-9 [QG] 1370970520 To Vera. You are my life. Table of Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxxiii Part I. Getting Started 1. A Python Q&A Session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Why Do People Use Python? Software Quality Developer Productivity Is Python a “Scripting Language”? OK, but What’s the Downside? Who Uses Python Today? What Can I Do with Python? Systems Programming GUIs Internet Scripting Component Integration Database Programming Rapid Prototyping Numeric and Scientific Programming And More: Gaming, Images, Data Mining, Robots, Excel... How Is Python Developed and Supported? Open Source Tradeoffs What Are Python’s Technical Strengths? It’s Object-Oriented and Functional It’s Free It’s Portable It’s Powerful It’s Mixable It’s Relatively Easy to Use It’s Relatively Easy to Learn It’s Named After Monty Python 3 4 5 5 7 9 10 11 11 11 12 12 13 13 14 15 15 16 16 17 17 18 19 19 20 20 v How Does Python Stack Up to Language X? Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 21 22 23 23 2. How Python Runs Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Introducing the Python Interpreter Program Execution The Programmer’s View Python’s View Execution Model Variations Python Implementation Alternatives Execution Optimization Tools Frozen Binaries Future Possibilities? Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 27 28 28 30 33 33 37 39 40 40 41 41 3. How You Run Programs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 The Interactive Prompt Starting an Interactive Session The System Path New Windows Options in 3.3: PATH, Launcher Where to Run: Code Directories What Not to Type: Prompts and Comments Running Code Interactively Why the Interactive Prompt? Usage Notes: The Interactive Prompt System Command Lines and Files A First Script Running Files with Command Lines Command-Line Usage Variations Usage Notes: Command Lines and Files Unix-Style Executable Scripts: #! Unix Script Basics The Unix env Lookup Trick The Python 3.3 Windows Launcher: #! Comes to Windows Clicking File Icons Icon-Click Basics Clicking Icons on Windows The input Trick on Windows Other Icon-Click Limitations vi | Table of Contents 43 44 45 46 47 48 49 50 52 54 55 56 57 58 59 59 60 60 62 62 63 63 66 Module Imports and Reloads Import and Reload Basics The Grander Module Story: Attributes Usage Notes: import and reload Using exec to Run Module Files The IDLE User Interface IDLE Startup Details IDLE Basic Usage IDLE Usability Features Advanced IDLE Tools Usage Notes: IDLE Other IDEs Other Launch Options Embedding Calls Frozen Binary Executables Text Editor Launch Options Still Other Launch Options Future Possibilities? Which Option Should I Use? Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Test Your Knowledge: Part I Exercises 66 66 68 71 72 73 74 75 76 77 78 79 81 81 82 82 82 83 83 85 85 86 87 Part II. Types and Operations 4. Introducing Python Object Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 The Python Conceptual Hierarchy Why Use Built-in Types? Python’s Core Data Types Numbers Strings Sequence Operations Immutability Type-Specific Methods Getting Help Other Ways to Code Strings Unicode Strings Pattern Matching Lists Sequence Operations Type-Specific Operations 93 94 95 97 99 99 101 102 104 105 106 108 109 109 109 Table of Contents | vii Bounds Checking Nesting Comprehensions Dictionaries Mapping Operations Nesting Revisited Missing Keys: if Tests Sorting Keys: for Loops Iteration and Optimization Tuples Why Tuples? Files Binary Bytes Files Unicode Text Files Other File-Like Tools Other Core Types How to Break Your Code’s Flexibility User-Defined Classes And Everything Else Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 110 110 111 113 114 115 116 118 120 121 122 122 123 124 126 126 128 129 130 130 131 131 5. Numeric Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Numeric Type Basics Numeric Literals Built-in Numeric Tools Python Expression Operators Numbers in Action Variables and Basic Expressions Numeric Display Formats Comparisons: Normal and Chained Division: Classic, Floor, and True Integer Precision Complex Numbers Hex, Octal, Binary: Literals and Conversions Bitwise Operations Other Built-in Numeric Tools Other Numeric Types Decimal Type Fraction Type Sets Booleans viii | Table of Contents 133 134 136 136 141 141 143 144 146 150 151 151 153 155 157 157 160 163 171 Numeric Extensions Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 172 172 173 173 6. The Dynamic Typing Interlude . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 The Case of the Missing Declaration Statements Variables, Objects, and References Types Live with Objects, Not Variables Objects Are Garbage-Collected Shared References Shared References and In-Place Changes Shared References and Equality Dynamic Typing Is Everywhere Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 175 176 177 178 180 181 183 185 186 186 186 7. String Fundamentals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 This Chapter’s Scope Unicode: The Short Story String Basics String Literals Single- and Double-Quoted Strings Are the Same Escape Sequences Represent Special Characters Raw Strings Suppress Escapes Triple Quotes Code Multiline Block Strings Strings in Action Basic Operations Indexing and Slicing String Conversion Tools Changing Strings I String Methods Method Call Syntax Methods of Strings String Method Examples: Changing Strings II String Method Examples: Parsing Text Other Common String Methods in Action The Original string Module’s Functions (Gone in 3.X) String Formatting Expressions Formatting Expression Basics Advanced Formatting Expression Syntax Advanced Formatting Expression Examples 189 189 190 192 193 193 196 198 200 200 201 205 208 209 209 210 211 213 214 215 216 217 218 220 Table of Contents | ix Dictionary-Based Formatting Expressions String Formatting Method Calls Formatting Method Basics Adding Keys, Attributes, and Offsets Advanced Formatting Method Syntax Advanced Formatting Method Examples Comparison to the % Formatting Expression Why the Format Method? General Type Categories Types Share Operation Sets by Categories Mutable Types Can Be Changed in Place Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 221 222 222 223 224 225 227 230 235 235 236 237 237 237 8. Lists and Dictionaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 Lists Lists in Action Basic List Operations List Iteration and Comprehensions Indexing, Slicing, and Matrixes Changing Lists in Place Dictionaries Dictionaries in Action Basic Dictionary Operations Changing Dictionaries in Place More Dictionary Methods Example: Movie Database Dictionary Usage Notes Other Ways to Make Dictionaries Dictionary Changes in Python 3.X and 2.7 Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 239 242 242 242 243 244 250 252 253 254 254 256 258 262 264 271 272 272 9. Tuples, Files, and Everything Else . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 Tuples Tuples in Action Why Lists and Tuples? Records Revisited: Named Tuples Files Opening Files Using Files x | Table of Contents 276 277 279 280 282 283 284 Files in Action Text and Binary Files: The Short Story Storing Python Objects in Files: Conversions Storing Native Python Objects: pickle Storing Python Objects in JSON Format Storing Packed Binary Data: struct File Context Managers Other File Tools Core Types Review and Summary Object Flexibility References Versus Copies Comparisons, Equality, and Truth The Meaning of True and False in Python Python’s Type Hierarchies Type Objects Other Types in Python Built-in Type Gotchas Assignment Creates References, Not Copies Repetition Adds One Level Deep Beware of Cyclic Data Structures Immutable Types Can’t Be Changed in Place Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Test Your Knowledge: Part II Exercises 285 287 288 290 291 293 294 294 295 297 297 300 304 306 306 308 308 308 309 310 311 311 311 312 313 Part III. Statements and Syntax 10. Introducing Python Statements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 The Python Conceptual Hierarchy Revisited Python’s Statements A Tale of Two ifs What Python Adds What Python Removes Why Indentation Syntax? A Few Special Cases A Quick Example: Interactive Loops A Simple Interactive Loop Doing Math on User Inputs Handling Errors by Testing Inputs Handling Errors with try Statements Nesting Code Three Levels Deep 319 320 322 322 323 324 327 329 329 331 332 333 335 Table of Contents | xi Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 336 336 336 11. Assignments, Expressions, and Prints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 Assignment Statements Assignment Statement Forms Sequence Assignments Extended Sequence Unpacking in Python 3.X Multiple-Target Assignments Augmented Assignments Variable Name Rules Expression Statements Expression Statements and In-Place Changes Print Operations The Python 3.X print Function The Python 2.X print Statement Print Stream Redirection Version-Neutral Printing Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 339 340 341 344 348 350 352 356 357 358 359 361 363 366 369 370 370 12. if Tests and Syntax Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 if Statements General Format Basic Examples Multiway Branching Python Syntax Revisited Block Delimiters: Indentation Rules Statement Delimiters: Lines and Continuations A Few Special Cases Truth Values and Boolean Tests The if/else Ternary Expression Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 371 371 372 372 375 376 378 379 380 382 385 385 386 13. while and for Loops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387 while Loops General Format Examples break, continue, pass, and the Loop else xii | Table of Contents 387 388 388 389 General Loop Format pass continue break Loop else for Loops General Format Examples Loop Coding Techniques Counter Loops: range Sequence Scans: while and range Versus for Sequence Shufflers: range and len Nonexhaustive Traversals: range Versus Slices Changing Lists: range Versus Comprehensions Parallel Traversals: zip and map Generating Both Offsets and Items: enumerate Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 389 390 391 391 392 395 395 395 402 402 403 404 405 406 407 410 413 414 414 14. Iterations and Comprehensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415 Iterations: A First Look The Iteration Protocol: File Iterators Manual Iteration: iter and next Other Built-in Type Iterables List Comprehensions: A First Detailed Look List Comprehension Basics Using List Comprehensions on Files Extended List Comprehension Syntax Other Iteration Contexts New Iterables in Python 3.X Impacts on 2.X Code: Pros and Cons The range Iterable The map, zip, and filter Iterables Multiple Versus Single Pass Iterators Dictionary View Iterables Other Iteration Topics Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 416 416 419 422 424 425 426 427 429 434 434 435 436 438 439 440 441 441 441 15. The Documentation Interlude . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443 Python Documentation Sources 443 Table of Contents | xiii # Comments The dir Function Docstrings: __doc__ PyDoc: The help Function PyDoc: HTML Reports Beyond docstrings: Sphinx The Standard Manual Set Web Resources Published Books Common Coding Gotchas Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Test Your Knowledge: Part III Exercises 444 444 446 449 452 461 461 462 463 463 465 466 466 467 Part IV. Functions and Generators 16. Function Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473 Why Use Functions? Coding Functions def Statements def Executes at Runtime A First Example: Definitions and Calls Definition Calls Polymorphism in Python A Second Example: Intersecting Sequences Definition Calls Polymorphism Revisited Local Variables Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 474 475 476 477 478 478 478 479 480 481 481 482 483 483 483 484 17. Scopes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485 Python Scope Basics Scope Details Name Resolution: The LEGB Rule Scope Example The Built-in Scope The global Statement xiv | Table of Contents 485 486 488 490 491 494 Program Design: Minimize Global Variables Program Design: Minimize Cross-File Changes Other Ways to Access Globals Scopes and Nested Functions Nested Scope Details Nested Scope Examples Factory Functions: Closures Retaining Enclosing Scope State with Defaults The nonlocal Statement in 3.X nonlocal Basics nonlocal in Action Why nonlocal? State Retention Options State with nonlocal: 3.X only State with Globals: A Single Copy Only State with Classes: Explicit Attributes (Preview) State with Function Attributes: 3.X and 2.X Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 495 497 498 499 500 500 501 504 508 508 509 512 512 513 513 515 519 519 520 18. Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 523 Argument-Passing Basics Arguments and Shared References Avoiding Mutable Argument Changes Simulating Output Parameters and Multiple Results Special Argument-Matching Modes Argument Matching Basics Argument Matching Syntax The Gritty Details Keyword and Default Examples Arbitrary Arguments Examples Python 3.X Keyword-Only Arguments The min Wakeup Call! Full Credit Bonus Points The Punch Line... Generalized Set Functions Emulating the Python 3.X print Function Using Keyword-Only Arguments Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 523 524 526 527 528 529 530 531 532 534 539 542 542 544 544 545 547 548 550 551 552 Table of Contents | xv 19. Advanced Function Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553 Function Design Concepts Recursive Functions Summation with Recursion Coding Alternatives Loop Statements Versus Recursion Handling Arbitrary Structures Function Objects: Attributes and Annotations Indirect Function Calls: “First Class” Objects Function Introspection Function Attributes Function Annotations in 3.X Anonymous Functions: lambda lambda Basics Why Use lambda? How (Not) to Obfuscate Your Python Code Scopes: lambdas Can Be Nested Too Functional Programming Tools Mapping Functions over Iterables: map Selecting Items in Iterables: filter Combining Items in Iterables: reduce Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 553 555 555 556 557 558 562 562 563 564 565 567 568 569 571 572 574 574 576 576 578 578 578 20. Comprehensions and Generations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 581 List Comprehensions and Functional Tools List Comprehensions Versus map Adding Tests and Nested Loops: filter Example: List Comprehensions and Matrixes Don’t Abuse List Comprehensions: KISS Generator Functions and Expressions Generator Functions: yield Versus return Generator Expressions: Iterables Meet Comprehensions Generator Functions Versus Generator Expressions Generators Are Single-Iteration Objects Generation in Built-in Types, Tools, and Classes Example: Generating Scrambled Sequences Don’t Abuse Generators: EIBTI Example: Emulating zip and map with Iteration Tools Comprehension Syntax Summary Scopes and Comprehension Variables Comprehending Set and Dictionary Comprehensions xvi | Table of Contents 581 582 583 586 588 591 592 597 602 604 606 609 614 617 622 623 624 Extended Comprehension Syntax for Sets and Dictionaries Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 625 626 626 626 21. The Benchmarking Interlude . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 629 Timing Iteration Alternatives Timing Module: Homegrown Timing Script Timing Results Timing Module Alternatives Other Suggestions Timing Iterations and Pythons with timeit Basic timeit Usage Benchmark Module and Script: timeit Benchmark Script Results More Fun with Benchmarks Other Benchmarking Topics: pystones Function Gotchas Local Names Are Detected Statically Defaults and Mutable Objects Functions Without returns Miscellaneous Function Gotchas Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Test Your Knowledge: Part IV Exercises 629 630 634 635 638 642 642 643 647 649 651 656 656 657 658 660 661 661 662 662 663 Part V. Modules and Packages 22. Modules: The Big Picture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 669 Why Use Modules? Python Program Architecture How to Structure a Program Imports and Attributes Standard Library Modules How Imports Work 1. Find It 2. Compile It (Maybe) 3. Run It Byte Code Files: __pycache__ in Python 3.2+ Byte Code File Models in Action 669 670 671 671 673 674 674 675 675 676 677 Table of Contents | xvii The Module Search Path Configuring the Search Path Search Path Variations The sys.path List Module File Selection Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 678 681 681 681 682 685 685 685 23. Module Coding Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 687 Module Creation Module Filenames Other Kinds of Modules Module Usage The import Statement The from Statement The from * Statement Imports Happen Only Once import and from Are Assignments import and from Equivalence Potential Pitfalls of the from Statement Module Namespaces Files Generate Namespaces Namespace Dictionaries: __dict__ Attribute Name Qualification Imports Versus Scopes Namespace Nesting Reloading Modules reload Basics reload Example Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 687 687 688 688 689 689 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 704 24. Module Packages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 707 Package Import Basics Packages and Search Path Settings Package __init__.py Files Package Import Example from Versus import with Packages Why Use Package Imports? A Tale of Three Systems Package Relative Imports xviii | Table of Contents 708 708 709 711 713 713 714 717 Changes in Python 3.X Relative Import Basics Why Relative Imports? The Scope of Relative Imports Module Lookup Rules Summary Relative Imports in Action Pitfalls of Package-Relative Imports: Mixed Use Python 3.3 Namespace Packages Namespace Package Semantics Impacts on Regular Packages: Optional __init__.py Namespace Packages in Action Namespace Package Nesting Files Still Have Precedence over Directories Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 718 718 720 722 723 723 729 734 735 736 737 738 740 742 742 742 25. Advanced Module Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 745 Module Design Concepts Data Hiding in Modules Minimizing from * Damage: _X and __all__ Enabling Future Language Features: __future__ Mixed Usage Modes: __name__ and __main__ Unit Tests with __name__ Example: Dual Mode Code Currency Symbols: Unicode in Action Docstrings: Module Documentation at Work Changing the Module Search Path The as Extension for import and from Example: Modules Are Objects Importing Modules by Name String Running Code Strings Direct Calls: Two Options Example: Transitive Module Reloads A Recursive Reloader Alternative Codings Module Gotchas Module Name Clashes: Package and Package-Relative Imports Statement Order Matters in Top-Level Code from Copies Names but Doesn’t Link from * Can Obscure the Meaning of Variables reload May Not Impact from Imports reload, from, and Interactive Testing 745 747 747 748 749 750 751 754 756 756 758 759 761 762 762 763 764 767 770 771 771 772 773 773 774 Table of Contents | xix Recursive from Imports May Not Work Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Test Your Knowledge: Part V Exercises 775 776 777 777 778 Part VI. Classes and OOP 26. OOP: The Big Picture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 783 Why Use Classes? OOP from 30,000 Feet Attribute Inheritance Search Classes and Instances Method Calls Coding Class Trees Operator Overloading OOP Is About Code Reuse Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 784 785 785 788 788 789 791 792 795 795 795 27. Class Coding Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 797 Classes Generate Multiple Instance Objects Class Objects Provide Default Behavior Instance Objects Are Concrete Items A First Example Classes Are Customized by Inheritance A Second Example Classes Are Attributes in Modules Classes Can Intercept Python Operators A Third Example Why Use Operator Overloading? The World’s Simplest Python Class Records Revisited: Classes Versus Dictionaries Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 797 798 798 799 801 802 804 805 806 808 809 812 814 815 815 28. A More Realistic Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 817 Step 1: Making Instances Coding Constructors Testing As You Go xx | Table of Contents 818 818 819 Using Code Two Ways Step 2: Adding Behavior Methods Coding Methods Step 3: Operator Overloading Providing Print Displays Step 4: Customizing Behavior by Subclassing Coding Subclasses Augmenting Methods: The Bad Way Augmenting Methods: The Good Way Polymorphism in Action Inherit, Customize, and Extend OOP: The Big Idea Step 5: Customizing Constructors, Too OOP Is Simpler Than You May Think Other Ways to Combine Classes Step 6: Using Introspection Tools Special Class Attributes A Generic Display Tool Instance Versus Class Attributes Name Considerations in Tool Classes Our Classes’ Final Form Step 7 (Final): Storing Objects in a Database Pickles and Shelves Storing Objects on a Shelve Database Exploring Shelves Interactively Updating Objects on a Shelve Future Directions Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 820 822 824 826 826 828 828 829 829 832 833 833 834 836 836 840 840 842 843 844 845 847 847 848 849 851 853 855 855 856 29. Class Coding Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 859 The class Statement General Form Example Methods Method Example Calling Superclass Constructors Other Method Call Possibilities Inheritance Attribute Tree Construction Specializing Inherited Methods Class Interface Techniques 859 860 860 862 863 864 864 865 865 866 867 Table of Contents | xxi Abstract Superclasses Namespaces: The Conclusion Simple Names: Global Unless Assigned Attribute Names: Object Namespaces The “Zen” of Namespaces: Assignments Classify Names Nested Classes: The LEGB Scopes Rule Revisited Namespace Dictionaries: Review Namespace Links: A Tree Climber Documentation Strings Revisited Classes Versus Modules Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 869 872 872 872 873 875 878 880 882 884 884 884 885 30. Operator Overloading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 887 The Basics Constructors and Expressions: __init__ and __sub__ Common Operator Overloading Methods Indexing and Slicing: __getitem__ and __setitem__ Intercepting Slices Slicing and Indexing in Python 2.X But 3.X’s __index__ Is Not Indexing! Index Iteration: __getitem__ Iterable Objects: __iter__ and __next__ User-Defined Iterables Multiple Iterators on One Object Coding Alternative: __iter__ plus yield Membership: __contains__, __iter__, and __getitem__ Attribute Access: __getattr__ and __setattr__ Attribute Reference Attribute Assignment and Deletion Other Attribute Management Tools Emulating Privacy for Instance Attributes: Part 1 String Representation: __repr__ and __str__ Why Two Display Methods? Display Usage Notes Right-Side and In-Place Uses: __radd__ and __iadd__ Right-Side Addition In-Place Addition Call Expressions: __call__ Function Interfaces and Callback-Based Code Comparisons: __lt__, __gt__, and Others The __cmp__ Method in Python 2.X xxii | Table of Contents 887 888 888 890 891 893 894 894 895 896 899 902 906 909 909 910 912 912 913 914 916 917 917 920 921 923 925 926 Boolean Tests: __bool__ and __len__ Boolean Methods in Python 2.X Object Destruction: __del__ Destructor Usage Notes Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 927 928 929 930 931 931 931 31. Designing with Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 933 Python and OOP Polymorphism Means Interfaces, Not Call Signatures OOP and Inheritance: “Is-a” Relationships OOP and Composition: “Has-a” Relationships Stream Processors Revisited OOP and Delegation: “Wrapper” Proxy Objects Pseudoprivate Class Attributes Name Mangling Overview Why Use Pseudoprivate Attributes? Methods Are Objects: Bound or Unbound Unbound Methods Are Functions in 3.X Bound Methods and Other Callable Objects Classes Are Objects: Generic Object Factories Why Factories? Multiple Inheritance: “Mix-in” Classes Coding Mix-in Display Classes Other Design-Related Topics Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 933 934 935 937 938 942 944 945 945 948 950 951 954 955 956 957 977 977 978 978 32. Advanced Class Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 979 Extending Built-in Types Extending Types by Embedding Extending Types by Subclassing The “New Style” Class Model Just How New Is New-Style? New-Style Class Changes Attribute Fetch for Built-ins Skips Instances Type Model Changes All Classes Derive from “object” Diamond Inheritance Change More on the MRO: Method Resolution Order Example: Mapping Attributes to Inheritance Sources 980 980 981 983 984 985 987 992 995 997 1001 1004 Table of Contents | xxiii New-Style Class Extensions Slots: Attribute Declarations Properties: Attribute Accessors __getattribute__ and Descriptors: Attribute Tools Other Class Changes and Extensions Static and Class Methods Why the Special Methods? Static Methods in 2.X and 3.X Static Method Alternatives Using Static and Class Methods Counting Instances with Static Methods Counting Instances with Class Methods Decorators and Metaclasses: Part 1 Function Decorator Basics A First Look at User-Defined Function Decorators A First Look at Class Decorators and Metaclasses For More Details The super Built-in Function: For Better or Worse? The Great super Debate Traditional Superclass Call Form: Portable, General Basic super Usage and Its Tradeoffs The super Upsides: Tree Changes and Dispatch Runtime Class Changes and super Cooperative Multiple Inheritance Method Dispatch The super Summary Class Gotchas Changing Class Attributes Can Have Side Effects Changing Mutable Class Attributes Can Have Side Effects, Too Multiple Inheritance: Order Matters Scopes in Methods and Classes Miscellaneous Class Gotchas KISS Revisited: “Overwrapping-itis” Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Test Your Knowledge: Part VI Exercises 1010 1010 1020 1023 1023 1024 1024 1025 1027 1028 1030 1031 1034 1035 1037 1038 1040 1041 1041 1042 1043 1049 1049 1050 1062 1064 1064 1065 1066 1068 1069 1070 1070 1071 1071 1072 Part VII. Exceptions and Tools 33. Exception Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1081 Why Use Exceptions? Exception Roles xxiv | Table of Contents 1081 1082 Exceptions: The Short Story Default Exception Handler Catching Exceptions Raising Exceptions User-Defined Exceptions Termination Actions Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 1083 1083 1084 1085 1086 1087 1089 1090 1090 34. Exception Coding Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1093 The try/except/else Statement How try Statements Work try Statement Clauses The try else Clause Example: Default Behavior Example: Catching Built-in Exceptions The try/finally Statement Example: Coding Termination Actions with try/finally Unified try/except/finally Unified try Statement Syntax Combining finally and except by Nesting Unified try Example The raise Statement Raising Exceptions Scopes and try except Variables Propagating Exceptions with raise Python 3.X Exception Chaining: raise from The assert Statement Example: Trapping Constraints (but Not Errors!) with/as Context Managers Basic Usage The Context Management Protocol Multiple Context Managers in 3.1, 2.7, and Later Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 1093 1094 1095 1098 1098 1100 1100 1101 1102 1104 1104 1105 1106 1107 1108 1110 1110 1112 1113 1114 1114 1116 1118 1119 1120 1120 35. Exception Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1123 Exceptions: Back to the Future String Exceptions Are Right Out! Class-Based Exceptions Coding Exceptions Classes 1124 1124 1125 1126 Table of Contents | xxv Why Exception Hierarchies? Built-in Exception Classes Built-in Exception Categories Default Printing and State Custom Print Displays Custom Data and Behavior Providing Exception Details Providing Exception Methods Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 1128 1131 1132 1133 1135 1136 1136 1137 1139 1139 1139 36. Designing with Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1141 Nesting Exception Handlers Example: Control-Flow Nesting Example: Syntactic Nesting Exception Idioms Breaking Out of Multiple Nested Loops: “go to” Exceptions Aren’t Always Errors Functions Can Signal Conditions with raise Closing Files and Server Connections Debugging with Outer try Statements Running In-Process Tests More on sys.exc_info Displaying Errors and Tracebacks Exception Design Tips and Gotchas What Should Be Wrapped Catching Too Much: Avoid Empty except and Exception Catching Too Little: Use Class-Based Categories Core Language Summary The Python Toolset Development Tools for Larger Projects Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers Test Your Knowledge: Part VII Exercises 1141 1143 1143 1145 1145 1146 1147 1148 1149 1149 1150 1151 1152 1152 1153 1155 1155 1156 1157 1160 1161 1161 1161 Part VIII. Advanced Topics 37. Unicode and Byte Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1165 String Changes in 3.X String Basics xxvi | Table of Contents 1166 1167 Character Encoding Schemes How Python Stores Strings in Memory Python’s String Types Text and Binary Files Coding Basic Strings Python 3.X String Literals Python 2.X String Literals String Type Conversions Coding Unicode Strings Coding ASCII Text Coding Non-ASCII Text Encoding and Decoding Non-ASCII text Other Encoding Schemes Byte String Literals: Encoded Text Converting Encodings Coding Unicode Strings in Python 2.X Source File Character Set Encoding Declarations Using 3.X bytes Objects Method Calls Sequence Operations Other Ways to Make bytes Objects Mixing String Types Using 3.X/2.6+ bytearray Objects bytearrays in Action Python 3.X String Types Summary Using Text and Binary Files Text File Basics Text and Binary Modes in 2.X and 3.X Type and Content Mismatches in 3.X Using Unicode Files Reading and Writing Unicode in 3.X Handling the BOM in 3.X Unicode Files in 2.X Unicode Filenames and Streams Other String Tool Changes in 3.X The re Pattern-Matching Module The struct Binary Data Module The pickle Object Serialization Module XML Parsing Tools Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 1167 1170 1171 1173 1174 1175 1176 1177 1178 1178 1179 1180 1181 1183 1184 1185 1187 1189 1189 1190 1191 1192 1192 1193 1195 1195 1196 1197 1198 1199 1199 1201 1204 1205 1206 1206 1207 1209 1211 1215 1215 1216 Table of Contents | xxvii 38. Managed Attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1219 Why Manage Attributes? Inserting Code to Run on Attribute Access Properties The Basics A First Example Computed Attributes Coding Properties with Decorators Descriptors The Basics A First Example Computed Attributes Using State Information in Descriptors How Properties and Descriptors Relate __getattr__ and __getattribute__ The Basics A First Example Computed Attributes __getattr__ and __getattribute__ Compared Management Techniques Compared Intercepting Built-in Operation Attributes Example: Attribute Validations Using Properties to Validate Using Descriptors to Validate Using __getattr__ to Validate Using __getattribute__ to Validate Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 1219 1220 1221 1222 1222 1224 1224 1226 1227 1229 1231 1232 1236 1237 1238 1241 1243 1245 1246 1249 1256 1256 1259 1263 1265 1266 1266 1267 39. Decorators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1269 What’s a Decorator? Managing Calls and Instances Managing Functions and Classes Using and Defining Decorators Why Decorators? The Basics Function Decorators Class Decorators Decorator Nesting Decorator Arguments Decorators Manage Functions and Classes, Too Coding Function Decorators xxviii | Table of Contents 1269 1270 1270 1271 1271 1273 1273 1277 1279 1281 1282 1283 Tracing Calls Decorator State Retention Options Class Blunders I: Decorating Methods Timing Calls Adding Decorator Arguments Coding Class Decorators Singleton Classes Tracing Object Interfaces Class Blunders II: Retaining Multiple Instances Decorators Versus Manager Functions Why Decorators? (Revisited) Managing Functions and Classes Directly Example: “Private” and “Public” Attributes Implementing Private Attributes Implementation Details I Generalizing for Public Declarations, Too Implementation Details II Open Issues Python Isn’t About Control Example: Validating Function Arguments The Goal A Basic Range-Testing Decorator for Positional Arguments Generalizing for Keywords and Defaults, Too Implementation Details Open Issues Decorator Arguments Versus Function Annotations Other Applications: Type Testing (If You Insist!) Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 1283 1285 1289 1295 1298 1301 1301 1303 1308 1309 1310 1312 1314 1314 1317 1318 1320 1321 1329 1330 1330 1331 1333 1336 1338 1340 1342 1343 1344 1345 40. Metaclasses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1355 To Metaclass or Not to Metaclass Increasing Levels of “Magic” A Language of Hooks The Downside of “Helper” Functions Metaclasses Versus Class Decorators: Round 1 The Metaclass Model Classes Are Instances of type Metaclasses Are Subclasses of Type Class Statement Protocol Declaring Metaclasses Declaration in 3.X 1356 1357 1358 1359 1361 1364 1364 1366 1367 1368 1369 Table of Contents | xxix Declaration in 2.X Metaclass Dispatch in Both 3.X and 2.X Coding Metaclasses A Basic Metaclass Customizing Construction and Initialization Other Metaclass Coding Techniques Inheritance and Instance Metaclass Versus Superclass Inheritance: The Full Story Metaclass Methods Metaclass Methods Versus Class Methods Operator Overloading in Metaclass Methods Example: Adding Methods to Classes Manual Augmentation Metaclass-Based Augmentation Metaclasses Versus Class Decorators: Round 2 Example: Applying Decorators to Methods Tracing with Decoration Manually Tracing with Metaclasses and Decorators Applying Any Decorator to Methods Metaclasses Versus Class Decorators: Round 3 (and Last) Chapter Summary Test Your Knowledge: Quiz Test Your Knowledge: Answers 1369 1370 1370 1371 1372 1373 1378 1381 1382 1388 1389 1390 1391 1391 1393 1394 1400 1400 1401 1403 1404 1407 1407 1408 41. All Good Things . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1409 The Python Paradox On “Optional” Language Features Against Disquieting Improvements Complexity Versus Power Simplicity Versus Elitism Closing Thoughts Where to Go From Here Encore: Print Your Own Completion Certificate! 1409 1410 1411 1412 1412 1413 1414 1414 Part IX. Appendixes A. Installation and Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1421 Installing the Python Interpreter Is Python Already Present? Where to Get Python Installation Steps xxx | Table of Contents 1421 1421 1422 1423 Configuring Python Python Environment Variables How to Set Configuration Options Python Command-Line Arguments Python 3.3 Windows Launcher Command Lines For More Help 1427 1427 1429 1432 1435 1436 B. The Python 3.3 Windows Launcher . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1437 The Unix Legacy The Windows Legacy Introducing the New Windows Launcher A Windows Launcher Tutorial Step 1: Using Version Directives in Files Step 2: Using Command-Line Version Switches Step 3: Using and Changing Defaults Pitfalls of the New Windows Launcher Pitfall 1: Unrecognized Unix !# Lines Fail Pitfall 2: The Launcher Defaults to 2.X Pitfall 3: The New PATH Extension Option Conclusions: A Net Win for Windows 1437 1438 1439 1441 1441 1444 1445 1447 1447 1448 1449 1450 C. Python Changes and This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1451 Major 2.X/3.X Differences 3.X Differences 3.X-Only Extensions General Remarks: 3.X Changes Changes in Libraries and Tools Migrating to 3.X Fifth Edition Python Changes: 2.7, 3.2, 3.3 Changes in Python 2.7 Changes in Python 3.3 Changes in Python 3.2 Fourth Edition Python Changes: 2.6, 3.0, 3.1 Changes in Python 3.1 Changes in Python 3.0 and 2.6 Specific Language Removals in 3.0 Third Edition Python Changes: 2.3, 2.4, 2.5 Earlier and Later Python Changes 1451 1452 1453 1454 1454 1455 1456 1456 1457 1458 1458 1458 1459 1460 1462 1463 D. Solutions to End-of-Part Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1465 Part I, Getting Started Part II, Types and Operations Part III, Statements and Syntax 1465 1467 1473 Table of Contents | xxxi Part IV, Functions and Generators Part V, Modules and Packages Part VI, Classes and OOP Part VII, Exceptions and Tools 1475 1485 1489 1497 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1507 xxxii | Table of Contents Preface If you’re standing in a bookstore looking for the short story on this book, try this: • Python is a powerful multiparadigm computer programming language, optimized for programmer productivity, code readability, and software quality. • This book provides a comprehensive and in-depth introduction to the Python language itself. Its goal is to help you master Python fundamentals before moving on to apply them in your work. Like all its prior editions, this book is designed to serve as a single, all-inclusive learning resource for all Python newcomers, whether they will be using Python 2.X, Python 3.X, or both. • This edition has been brought up to date with Python releases 3.3 and 2.7, and has been expanded substantially to reflect current practice in the Python world. This preface describes this book’s goals, scope, and structure in more detail. It’s optional reading, but is designed to provide some orientation before you get started with the book at large. This Book’s “Ecosystem” Python is a popular open source programming language used for both standalone programs and scripting applications in a wide variety of domains. It is free, portable, powerful, and is both relatively easy and remarkably fun to use. Programmers from every corner of the software industry have found Python’s focus on developer productivity and software quality to be a strategic advantage in projects both large and small. Whether you are new to programming or are a professional developer, this book is designed to bring you up to speed on the Python language in ways that more limited approaches cannot. After reading this book, you should know enough about Python to apply it in whatever application domains you choose to explore. By design, this book is a tutorial that emphasizes the core Python language itself, rather than specific applications of it. As such, this book is intended to serve as the first in a two-volume set: xxxiii • Learning Python, this book, teaches Python itself, focusing on language fundamentals that span domains. • Programming Python, among others, moves on to show what you can do with Python after you’ve learned it. This division of labor is deliberate. While application goals can vary per reader, the need for useful language fundamentals coverage does not. Applications-focused books such as Programming Python pick up where this book leaves off, using realistically scaled examples to explore Python’s role in common domains such as the Web, GUIs, systems, databases, and text. In addition, the book Python Pocket Reference provides reference materials not included here, and it is designed to supplement this book. Because of this book’s focus on foundations, though, it is able to present Python language fundamentals with more depth than many programmers see when first learning the language. Its bottom-up approach and self-contained didactic examples are designed to teach readers the entire language one step at a time. The core language skills you’ll gain in the process will apply to every Python software system you’ll encounter—be it today’s popular tools such as Django, NumPy, and App Engine, or others that may be a part of both Python’s future and your programming career. Because it’s based upon a three-day Python training class with quizzes and exercises throughout, this book also serves as a self-paced introduction to the language. Although its format lacks the live interaction of a class, it compensates in the extra depth and flexibility that only a book can provide. Though there are many ways to use this book, linear readers will find it roughly equivalent to a semester-long Python class. About This Fifth Edition The prior fourth edition of this book published in 2009 covered Python versions 2.6 and 3.0.1 It addressed the many and sometimes incompatible changes introduced in the Python 3.X line in general. It also introduced a new OOP tutorial, and new chapters on advanced topics such as Unicode text, decorators, and metaclasses, derived from both the live classes I teach and evolution in Python “best practice.” This fifth edition completed in 2013 is a revision of the prior, updated to cover both Python 3.3 and 2.7, the current latest releases in the 3.X and 2.X lines. It incorporates 1. And 2007’s short-lived third edition covered Python 2.5, and its simpler—and shorter—single-line Python world. See http://www.rmi.net/~lutz for more on this book’s history. Over the years, this book has grown in size and complexity in direct proportion to Python’s own growth. Per Appendix C, Python 3.0 alone introduced 27 additions and 57 changes in the language that found their way into this book, and Python 3.3 continues this trend. Today’s Python programmer faces two incompatible lines, three major paradigms, a plethora of advanced tools, and a blizzard of feature redundancy—most of which do not divide neatly between the 2.X and 3.X lines. That’s not as daunting as it may sound (many tools are variations on a theme), but all are fair game in an inclusive, comprehensive Python text. xxxiv | Preface all language changes introduced in each line since the prior edition was published, and has been polished throughout to update and sharpen its presentation. Specifically: • Python 2.X coverage here has been updated to include features such as dictionary and set comprehensions that were formerly for 3.X only, but have been back-ported for use in 2.7. • Python 3.X coverage has been augmented for new yield and raise syntax; the __pycache__ bytecode model; 3.3 namespace packages; PyDoc’s all-browser mode; Unicode literal and storage changes; and the new Windows launcher shipped with 3.3. • Assorted new or expanded coverage for JSON, timeit, PyPy, os.popen, generators, recursion, weak references, __mro__, __iter__, super, __slots__, metaclasses, descriptors, random, Sphinx, and more has been added, along with a general increase in 2.X compatibility in both examples and narrative. This edition also adds a new conclusion as Chapter 41 (on Python’s evolution), two new appendixes (on recent Python changes and the new Windows launcher), and one new chapter (on benchmarking: an expanded version of the former code timing example). See Appendix C for a concise summary of Python changes between the prior edition and this one, as well as links to their coverage in the book. This appendix also summarizes initial differences between 2.X and 3.X in general that were first addressed in the prior edition, though some, such as new-style classes, span versions and simply become mandated in 3.X (more on what the X’s mean in a moment). Per the last bullet in the preceding list, this edition has also experienced some growth because it gives fuller coverage to more advanced language features—which many of us have tried very hard to ignore as optional for the last decade, but which have now grown more common in Python code. As we’ll see, these tools make Python more powerful, but also raise the bar for newcomers, and may shift Python’s scope and definition. Because you might encounter any of these, this book covers them head-on, instead of pretending they do not exist. Despite the updates, this edition retains most of the structure and content of the prior edition, and is still designed to be a comprehensive learning resource for both the 2.X and 3.X Python lines. While it is primarily focused on users of Python 3.3 and 2.7— the latest in the 3.X line and the likely last in the 2.X line—its historical perspective also makes it relevant to older Pythons that still see regular use today. Though it’s impossible to predict the future, this book stresses fundamentals that have been valid for nearly two decades, and will likely apply to future Pythons too. As usual, I’ll be posting Python updates that impact this book at the book’s website described ahead. The “What’s New” documents in Python’s manuals set can also serve to fill in the gaps as Python surely evolves after this book is published. Preface | xxxv The Python 2.X and 3.X Lines Because it bears heavily on this book’s content, I need to say a few more words about the Python 2.X/3.X story up front. When the fourth edition of this book was written in 2009, Python had just become available in two flavors: • Version 3.0 was the first in the line of an emerging and incompatible mutation of the language known generically as 3.X. • Version 2.6 retained backward compatibility with the vast body of existing Python code, and was the latest in the line known collectively as 2.X. While 3.X was largely the same language, it ran almost no code written for prior releases. It: • Imposed a Unicode model with broad consequences for strings, files, and libraries • Elevated iterators and generators to a more pervasive role, as part of fuller functional paradigm • Mandated new-style classes, which merge with types, but grow more powerful and complex • Changed many fundamental tools and libraries, and replaced or removed others entirely The mutation of print from statement to function alone, aesthetically sound as it may be, broke nearly every Python program ever written. And strategic potential aside, 3.X’s mandatory Unicode and class models and ubiquitous generators made for a different programming experience. Although many viewed Python 3.X as both an improvement and the future of Python, Python 2.X was still very widely used and was to be supported in parallel with Python 3.X for years to come. The majority of Python code in use was 2.X, and migration to 3.X seemed to be shaping up to be a slow process. The 2.X/3.X Story Today As this fifth edition is being written in 2013, Python has moved on to versions 3.3 and 2.7, but this 2.X/3.X story is still largely unchanged. In fact, Python is now a dual-version world, with many users running both 2.X and 3.X according to their software goals and dependencies. And for many newcomers, the choice between 2.X and 3.X remains one of existing software versus the language’s cutting edge. Although many major Python packages have been ported to 3.X, many others are still 2.X-only today. To some observers, Python 3.X is now seen as a sandbox for exploring new ideas, while 2.X is viewed as the tried-and-true Python, which doesn’t have all of 3.X’s features but is still more pervasive. Others still see Python 3.X as the future, a view that seems supported by current core developer plans: Python 2.7 will continue to be supported but is to be the last 2.X, while 3.3 is the latest in the 3.X line’s continuing evolution. xxxvi | Preface On the other hand, initiatives such as PyPy—today a still 2.X-only implementation of Python that offers stunning performance improvements—represent a 2.X future, if not an outright faction. All opinions aside, almost five years after its release, 3.X has yet to supersede 2.X, or even match its user base. As one metric, 2.X is still downloaded more often than 3.X for Windows at python.org today, despite the fact that this measure would be naturally skewed to new users and the most recent release. Such statistics are prone to change, of course, but after five years are indicative of 3.X uptake nonetheless. The existing 2.X software base still trumps 3.X’s language extensions for many. Moreover, being last in the 2.X line makes 2.7 a sort of de facto standard, immune to the constant pace of change in the 3.X line—a positive to those who seek a stable base, and a negative to those who seek growth and ongoing relevance. Personally, I think today’s Python world is large enough to accommodate both 3.X and 2.X; they seem to satisfy different goals and appeal to different camps, and there is precedence for this in other language families (C and C++, for example, have a longstanding coexistence, though they may differ more than Python 2.X and 3.X). Moreover, because they are so similar, the skills gained by learning either Python line transfer almost entirely to the other, especially if you’re aided by dual-version resources like this book. In fact, as long as you understand how they diverge, it’s often possible to write code that runs on both. At the same time, this split presents a substantial dilemma for both programmers and book authors, which shows no signs of abating. While it would be easier for a book to pretend that Python 2.X never existed and cover 3.X only, this would not address the needs of the large Python user base that exists today. A vast amount of existing code was written for Python 2.X, and it won’t be going away anytime soon. And while some newcomers to the language can and should focus on Python 3.X, anyone who must use code written in the past needs to keep one foot in the Python 2.X world today. Since it may still be years before many third-party libraries and extensions are ported to Python 3.X, this fork might not be entirely temporary. Coverage for Both 3.X and 2.X To address this dichotomy and to meet the needs of all potential readers, this book has been updated to cover both Python 3.3 and Python 2.7, and should apply to later releases in both the 3.X and 2.X lines. It’s intended for programmers using Python 2.X, programmers using Python 3.X, and programmers stuck somewhere between the two. That is, you can use this book to learn either Python line. Although 3.X is often emphasized, 2.X differences and tools are also noted along the way for programmers using older code. While the two versions are largely similar, they diverge in some important ways, and I’ll point these out as they crop up. Preface | xxxvii For instance, I’ll use 3.X print calls in most examples, but will also describe the 2.X print statement so you can make sense of earlier code, and will often use portable printing techniques that run on both lines. I’ll also freely introduce new features, such as the nonlocal statement in 3.X and the string format method available as of 2.6 and 3.0, and will point out when such extensions are not present in older Pythons. By proxy, this edition addresses other Python version 2.X and 3.X releases as well, though some older version 2.X code may not be able to run all the examples here. Although class decorators are available as of both Python 2.6 and 3.0, for example, you cannot use them in an older Python 2.X that did not yet have this feature. Again, see the change tables in Appendix C for summaries of recent 2.X and 3.X changes. Which Python Should I Use? Version choice may be mandated by your organization, but if you’re new to Python and learning on your own, you may be wondering which version to install. The answer here depends on your goals. Here are a few suggestions on the choice. When to choose 3.X: new features, evolution If you are learning Python for the first time and don’t need to use any existing 2.X code, I encourage you to begin with Python 3.X. It cleans up some longstanding warts in the language and trims some dated cruft, while retaining all the original core ideas and adding some nice new tools. For example, 3.X’s seamless Unicode model and broader use of generators and functional techniques are seen by many users as assets. Many popular Python libraries and tools are already available for Python 3.X, or will be by the time you read these words, especially given the continual improvements in the 3.X line. All new language evolution occurs in 3.X only, which adds features and keeps Python relevant, but also makes language definition a constantly moving target—a tradeoff inherent on the leading edge. When to choose 2.X: existing code, stability If you’ll be using a system based on Python 2.X, the 3.X line may not be an option for you today. However, you’ll find that this book addresses your concerns, too, and will help if you migrate to 3.X in the future. You’ll also find that you’re in large company. Every group I taught in 2012 was using 2.X only, and I still regularly see useful Python software in 2.X-only form. Moreover, unlike 3.X, 2.X is no longer being changed—which is either an asset or liability, depending on whom you ask. There’s nothing wrong with using and writing 2.X code, but you may wish to keep tabs on 3.X and its ongoing evolution as you do. Python’s future remains to be written, and is largely up to its users, including you. When to choose both: version-neutral code Probably the best news here is that Python’s fundamentals are the same in both its lines—2.X and 3.X differ in ways that many users will find minor, and this book is designed to help you learn both. In fact, as long as you understand their differences, it’s often straightforward to write version-neutral code that runs on both xxxviii | Preface Pythons, as we regularly will in this book. See Appendix C for pointers on 2.X/3.X migration and tips on writing code for both Python lines and audiences. Regardless of which version or versions you choose to focus on first, your skills will transfer directly to wherever your Python work leads you. About the Xs: Throughout this book, “3.X” and “2.X” are used to refer collectively to all releases in these two lines. For instance, 3.X includes 3.0 through 3.3, and future 3.X releases; 2.X means all from 2.0 through 2.7 (and presumably no others). More specific releases are mentioned when a topic applies to it only (e.g., 2.7’s set literals and 3.3’s launcher and namespace packages). This notation may occasionally be too broad —some features labeled 2.X here may not be present in early 2.X releases rarely used today—but it accommodates a 2.X line that has already spanned 13 years. The 3.X label is more easily and accurately applied to this younger five-year-old line. This Book’s Prerequisites and Effort It’s impossible to give absolute prerequisites for this book, because its utility and value can depend as much on reader motivation as on reader background. Both true beginners and crusty programming veterans have used this book successfully in the past. If you are motivated to learn Python, and willing to invest the time and focus it requires, this text will probably work for you. Just how much time is required to learn Python? Although this will vary per learner, this book tends to work best when read. Some readers may use this book as an ondemand reference resource, but most people seeking Python mastery should expect to spend at least weeks and probably months going through the material here, depending on how closely they follow along with its examples. As mentioned, it’s roughly equivalent to a full-semester course on the Python language itself. That’s the estimate for learning just Python itself and the software skills required to use it well. Though this book may suffice for basic scripting goals, readers hoping to pursue software development at large as a career should expect to devote additional time after this book to large-scale project experience, and possibly to follow-up texts such as Programming Python.2 2. The standard disclaimer: I wrote this and another book mentioned earlier, which work together as a set: Learning Python for language fundamentals, Programming Python for applications basics, and Python Pocket Reference as a companion to the other two. All three derive from 1995’s original and broad Programming Python. I encourage you to explore the many Python books available today (I stopped counting at 200 at Amazon.com just now because there was no end in sight, and this didn’t include related subjects like Django). My own publisher has recently produced Python-focused books on instrumentation, data mining, App Engine, numeric analysis, natural language processing, MongoDB, AWS, and more—specific domains you may wish to explore once you’ve mastered Python language fundamentals here. The Python story today is far too rich for any one book to address alone. Preface | xxxix That may not be welcome news to people looking for instant proficiency, but programming is not a trivial skill (despite what you may have heard!). Today’s Python, and software in general, are both challenging and rewarding enough to merit the effort implied by comprehensive books such as this. Here are a few pointers on using this book for readers on both sides of the experience spectrum: To experienced programmers You have an initial advantage and can move quickly through some earlier chapters; but you shouldn’t skip the core ideas, and may need to work at letting go of some baggage. In general terms, exposure to any programming or scripting before this book might be helpful because of the analogies it may provide. On the other hand, I’ve also found that prior programming experience can be a handicap due to expectations rooted in other languages (it’s far too easy to spot the Java or C++ programmers in classes by the first Python code they write!). Using Python well requires adopting its mindset. By focusing on key core concepts, this book is designed to help you learn to code Python in Python. To true beginners You can learn Python here too, as well as programming itself; but you may need to work a bit harder, and may wish to supplement this text with gentler introductions. If you don’t consider yourself a programmer already, you will probably find this book useful too, but you’ll want to be sure to proceed slowly and work through the examples and exercises along the way. Also keep in mind that this book will spend more time teaching Python itself than programming basics. If you find yourself lost here, I encourage you to explore an introduction to programming in general before tackling this book. Python’s website has links to many helpful resources for beginners. Formally, this book is designed to serve as a first Python text for newcomers of all kinds. It may not be an ideal resource for someone who has never touched a computer before (for instance, we’re not going to spend any time exploring what a computer is), but I haven’t made many assumptions about your programming background or education. On the other hand, I won’t insult readers by assuming they are “dummies,” either, whatever that means—it’s easy to do useful things in Python, and this book will show you how. The text occasionally contrasts Python with languages such as C, C++, Java, and others, but you can safely ignore these comparisons if you haven’t used such languages in the past. This Book’s Structure To help orient you, this section provides a quick rundown of the content and goals of the major parts of this book. If you’re anxious to get to it, you should feel free to skip xl | Preface this section (or browse the table of contents instead). To some readers, though, a book this large probably merits a brief roadmap up front. By design, each part covers a major functional area of the language, and each part is composed of chapters focusing on a specific topic or aspect of the part’s area. In addition, each chapter ends with quizzes and their answers, and each part ends with larger exercises, whose solutions show up in Appendix D. Practice matters: I strongly recommend that readers work through the quizzes and exercises in this book, and work along with its examples in general if you can. In programming, there’s no substitute for practicing what you’ve read. Whether you do it with this book or a project of your own, actual coding is crucial if you want the ideas presented here to stick. Overall, this book’s presentation is bottom-up because Python is too. The examples and topics grow more challenging as we move along. For instance, Python’s classes are largely just packages of functions that process built-in types. Once you’ve mastered built-in types and functions, classes become a relatively minor intellectual leap. Because each part builds on those preceding it this way, most readers will find a linear reading makes the most sense. Here’s a preview of the book’s main parts you’ll find along the way: Part I We begin with a general overview of Python that answers commonly asked initial questions—why people use the language, what it’s useful for, and so on. The first chapter introduces the major ideas underlying the technology to give you some background context. The rest of this part moves on to explore the ways that both Python and programmers run programs. The main goal here is to give you just enough information to be able to follow along with later examples and exercises. Part II Next, we begin our tour of the Python language, studying Python’s major built-in object types and what you can do with them in depth: numbers, lists, dictionaries, and so on. You can get a lot done with these tools alone, and they are at the heart of every Python script. This is the most substantial part of the book because we lay groundwork here for later chapters. We’ll also explore dynamic typing and its references—keys to using Python well—in this part. Part III The next part moves on to introduce Python’s statements—the code you type to create and process objects in Python. It also presents Python’s general syntax model. Although this part focuses on syntax, it also introduces some related tools (such as the PyDoc system), takes a first look at iteration concepts, and explores coding alternatives. Preface | xli Part IV This part begins our look at Python’s higher-level program structure tools. Functions turn out to be a simple way to package code for reuse and avoid code redundancy. In this part, we will explore Python’s scoping rules, argument-passing techniques, the sometimes-notorious lambda, and more. We’ll also revisit iterators from a functional programming perspective, introduce user-defined generators, and learn how to time Python code to measure performance here. Part V Python modules let you organize statements and functions into larger components, and this part illustrates how to create, use, and reload modules. We’ll also look at some more advanced topics here, such as module packages, module reloading, package-relative imports, 3.3’s new namespace packages, and the __name__ variable. Part VI Here, we explore Python’s object-oriented programming tool, the class—an optional but powerful way to structure code for customization and reuse, which almost naturally minimizes redundancy. As you’ll see, classes mostly reuse ideas we will have covered by this point in the book, and OOP in Python is mostly about looking up names in linked objects with a special first argument in functions. As you’ll also see, OOP is optional in Python, but most find Python’s OOP to be much simpler than others, and it can shave development time substantially, especially for long-term strategic project development. Part VII We conclude the language fundamentals coverage in this text with a look at Python’s exception handling model and statements, plus a brief overview of development tools that will become more useful when you start writing larger programs (debugging and testing tools, for instance). Although exceptions are a fairly lightweight tool, this part appears after the discussion of classes because user-defined exceptions should now all be classes. We also cover some more advanced topics, such as context managers, here. Part VIII In the final part, we explore some advanced topics: Unicode and byte strings, managed attribute tools like properties and descriptors, function and class decorators, and metaclasses. These chapters are all optional reading, because not all programmers need to understand the subjects they address. On the other hand, readers who must process internationalized text or binary data, or are responsible for developing APIs for other programmers to use, should find something of interest in this part. The examples here are also larger than most of those in this book, and can serve as self-study material. Part IX The book wraps up with a set of four appendixes that give platform-specific tips for installing and using Python on various computers; present the new Windows xlii | Preface launcher that ships with Python 3.3; summarize changes in Python addressed by recent editions and give links to their coverage here; and provide solutions to the end-of-part exercises. Solutions to end-of-chapter quizzes appear in the chapters themselves. See the table of contents for a finer-grained look at this book’s components. What This Book Is Not Given its relatively large audience over the years, some have inevitably expected this book to serve a role outside its scope. So now that I’ve told you what this book is, I also want to be clear on what it isn’t: • This book is a tutorial, not a reference. • This book covers the language itself, not applications, standard libraries, or thirdparty tools. • This book is a comprehensive look at a substantial topic, not a watered-down overview. Because these points are key to this book’s content, I want to say a few more words about them up front. It’s Not a Reference or a Guide to Specific Applications This book is a language tutorial, not a reference, and not an applications book. This is by design: today’s Python—with its built-in types, generators, closures, comprehensions, Unicode, decorators, and blend of procedural, object-oriented, and functional programming paradigms—makes the core language a substantial topic all by itself, and a prerequisite to all your future Python work, in whatever domains you pursue. When you are ready for other resources, though, here are a few suggestions and reminders: Reference resources As implied by the preceding structural description, you can use the index and table of contents to hunt for details, but there are no reference appendixes in this book. If you are looking for Python reference resources (and most readers probably will be very soon in their Python careers), I suggest the previously mentioned book that I also wrote as a companion to this one—Python Pocket Reference—as well as other reference books you’ll find with a quick search, and the standard Python reference manuals maintained at http://www.python.org. The latter of these are free, always up to date, and available both on the Web and on your computer after a Windows install. Applications and libraries As also discussed earlier, this book is not a guide to specific applications such as the Web, GUIs, or systems programming. By proxy, this includes the libraries and Preface | xliii tools used in applications work; although some standard libraries and tools are introduced here—including timeit, shelve, pickle, struct, json, pdb, os, urllib, re, xml, random, PyDoc and IDLE—they are not officially in this book’s primary scope. If you’re looking for more coverage on such topics and are already proficient with Python, I recommend the follow-up book Programming Python, among others. That book assumes this one as its prerequisite, though, so be sure you have a firm grasp of the core language first. Especially in an engineering domain like software, one must walk before one runs. It’s Not the Short Story for People in a Hurry As you can tell from its size, this book also doesn’t skimp on the details: it presents the full Python language, not a brief look at a simplified subset. Along the way it also covers software principles that are essential to writing good Python code. As mentioned, this is a multiple-week or -month book, designed to impart the skill level you’d acquire from a full-term class on Python. This is also deliberate. Many of this book’s readers don’t need to acquire full-scale software development skills, of course, and some can absorb Python in a piecemeal fashion. At the same time, because any part of the language may be used in code you will encounter, no part is truly optional for most programmers. Moreover, even casual scripters and hobbyists need to know basic principles of software development in order to code well, and even to use precoded tools properly. This book aims to address both of these needs—language and principles—in enough depth to be useful. In the end, though, you’ll find that Python’s more advanced tools, such as its object-oriented and functional programming support, are relatively easy to learn once you’ve mastered their prerequisites—and you will, if you work through this book one chapter at a time. It’s as Linear as Python Allows Speaking of reading order, this edition also tries hard to minimize forward references, but Python 3.X’s changes make this impossible in some cases (in fact, 3.X sometimes seems to assume you already know Python while you’re learning it!). As a handful of representative examples: • Printing, sorts, the string format method, and some dict calls rely on function keyword arguments. • Dictionary key lists and tests, and the list calls used around many tools, imply iteration concepts. • Using exec to run code now assumes knowledge of file objects and interfaces. • Coding new exceptions requires classes and OOP fundamentals. xliv | Preface • And so on—even basic inheritance broaches advanced topics such as metaclasses and descriptors. Python is still best learned as a progression from simple to advanced, and a linear reading here still makes the most sense. Still, some topics may require nonlinear jumps and random lookups. To minimize these, this book will point out forward dependencies when they occur, and will ease their impacts as much as possible. But if your time is tight: Though depth is crucial to mastering Python, some readers may have limited time. If you are interested in starting out with a quick Python tour, I suggest Chapter 1, Chapter 4, Chapter 10, and Chapter 28 (and perhaps 26)—a short survey that will hopefully pique your interest in the more complete story told in the rest of the book, and which most readers will need in today’s Python software world. In general, this book is intentionally layered this way to make its material easier to absorb—with introductions followed by details, so you can start with overviews, and dig deeper over time. You don’t need to read this book all at once, but its gradual approach is designed to help you tackle its material eventually. This Book’s Programs In general, this book has always strived to be agnostic about both Python versions and platforms. It’s designed to be useful to all Python users. Nevertheless, because Python changes over time and platforms tend to differ in pragmatic ways, I need to describe the specific systems you’ll see in action in most examples here. Python Versions This fifth edition of this book, and all the program examples in it, are based on Python versions 3.3 and 2.7. In addition, many of its examples run under prior 3.X and 2.X releases, and notes about the history of language changes in earlier versions are mixed in along the way for users of older Pythons. Because this text focuses on the core language, however, you can be fairly sure that most of what it has to say won’t change very much in future releases of Python, as noted earlier. Most of this book applies to earlier Python versions, too, except when it does not; naturally, if you try using extensions added after a release you’re using, all bets are off. As a rule of thumb, the latest Python is the best Python if you are able to upgrade. Because this book focuses on the core language, most of it also applies to both Jython and IronPython, the Java- and .NET-based Python language implementations, as well as other Python implementations such as Stackless and PyPy (described in Chapter 2). Such alternatives differ mostly in usage details, not language. Preface | xlv Platforms The examples in this book were run on a Windows 7 and 8 ultrabook,3 though Python’s portability makes this mostly a moot point, especially in this fundamentals-focused book. You’ll notice a few Windows-isms—including command-line prompts, a handful of screenshots, install pointers, and an appendix on the new Windows launcher in 3.3—but this reflects the fact that most Python newcomers will probably get started on this platform, and these can be safely ignored by users of other operating systems. I also give a few launching details for other platforms like Linux, such as “#!” line use, but as we’ll see in Chapter 3 and Appendix B, the 3.3 Windows launcher makes even this a more portable technique. Fetching This Book’s Code Source code for the book’s examples, as well as exercise solutions, can be fetched as a zip file from the book’s website at the following address: http://oreil.ly/LearningPython-5E This site includes both all the code in this book as well as package usage instructions, so I’ll defer to it for more details. Of course, the examples work best in the context of their appearance in this book, and you’ll need some background knowledge on running Python programs in general to make use of them. We’ll study startup details in Chapter 3, so please stay tuned for information on this front. Using This Book’s Code The code in my Python books is designed to teach, and I’m glad when it assists readers in that capacity. O’Reilly itself has an official policy regarding reusing the book’s examples in general, which I’ve pasted into the rest of this section for reference: This book is here to help you get your job done. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. 3. Mostly under Windows 7, but it’s irrelevant to this book. At this writing, Python installs on Windows 8 and runs in its desktop mode, which is essentially the same as Windows 7 without a Start button as I write this (you may need to create shortcuts for former Start button menu items). Support for WinRT/ Metro “apps” is still pending. See Appendix A for more details. Frankly, the future of Windows 8 is unclear as I type these words, so this book will be as version-neutral as possible. xlvi | Preface We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Learning Python, Fifth Edition, by Mark Lutz. Copyright 2013 Mark Lutz, 978-1-4493-5573-9.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at firstname.lastname@example.org. Font Conventions This book’s mechanics will make more sense once you start reading it, of course, but as a reference, this book uses the following typographical conventions: Italic Used for email addresses, URLs, filenames, pathnames, and emphasizing new terms when they are first introduced Constant width Used for program code, the contents of files and the output from commands, and to designate modules, methods, statements, and system commands Constant width bold Used in code sections to show commands or text that would be typed by the user, and, occasionally, to highlight portions of code Constant width italic Used for replaceables and some comments in code sections Indicates a tip, suggestion, or general note relating to the nearby text. Indicates a warning or caution relating to the nearby text. You’ll also find occasional sidebars (delimited by boxes) and footnotes (at page end) throughout, which are often optional reading, but provide additional context on the topics being presented. The sidebars in “Why You Will Care: Slices” on page 204, for example, often give example use cases for the subjects being explored. Book Updates and Resources Improvements happen (and so do mis^H^H^H typos). Updates, supplements, and corrections (a.k.a. errata) for this book will be maintained on the Web, and may be suggested at either the publisher’s website or by email. Here are the main coordinates: Preface | xlvii Publisher’s site: http://oreil.ly/LearningPython-5E This site will maintain this edition’s official list of book errata, and chronicle specific patches applied to the text in reprints. It’s also the official site for the book’s examples as described earlier. Author’s site: http://www.rmi.net/~lutz/about-lp5e.html This site will be used to post more general updates related to this text or Python itself—a hedge against future changes, which should be considered a sort of virtual appendix to this book. My publisher also has an email address for comments and technical questions about this book: email@example.com For more information about my publisher’s books, conferences, Resource Centers, and the O’Reilly Network, see its general website: http://www.oreilly.com For more on my books, see my own book support site: http://rmi.net/~lutz Also be sure to search the Web if any of the preceding links become invalid over time; if I could become more clairvoyant, I would, but the Web changes faster than published books. Acknowledgments As I write this fifth edition of this book in 2013, it’s difficult to not be somewhat retrospective. I have now been using and promoting Python for 21 years, writing books about it for 18, and teaching live classes on it for 16. Despite the passage of time, I’m still regularly amazed at how successful Python has been—in ways that most of us could not possibly have imagined in the early 1990s. So at the risk of sounding like a hopelessly self-absorbed author, I hope you’ll pardon a few closing words of history and gratitude here. The Backstory My own Python history predates both Python 1.0 and the Web (and goes back to a time when an install meant fetching email messages, concatenating, decoding, and hoping it all somehow worked). When I first discovered Python as a frustrated C++ software developer in 1992, I had no idea what an impact it would have on the next two decades of my life. Two years after writing the first edition of Programming Python in 1995 for Python 1.3, I began traveling around the country and world teaching Python to beginners and experts. Since finishing the first edition of Learning Python in xlviii | Preface 1999, I’ve been an independent Python trainer and writer, thanks in part to Python’s phenomenal growth in popularity. Here’s the damage so far. I’ve now written 13 Python books (5 of this, and 4 of two others), which have together sold some 400,000 units by my data. I’ve also been teaching Python for over a decade and a half; have taught some 260 Python training sessions in the U.S., Europe, Canada, and Mexico; and have met roughly 4,000 students along the way. Besides propelling me toward frequent flyer utopia, these classes helped me refine this text and my other Python books. Teaching honed the books, and vice versa, with the net result that my books closely parallel what happens in my classes, and can serve as a viable alternative to them. As for Python itself, in recent years it has grown to become one of the top 5 to 10 most widely used programming languages in the world (depending on which source you cite and when you cite it). Because we’ll be exploring Python’s status in the first chapter of this book, I’ll defer the rest of this story until then. Python Thanks Because teaching teaches teachers to teach, this book owes much to my live classes. I’d like to thank all the students who have participated in my courses during the last 16 years. Along with changes in Python itself, your feedback played a major role in shaping this text; there’s nothing quite as instructive as watching 4,000 people repeat the same beginner mistakes live and in person! This book’s recent editions owe their trainingbased changes primarily to recent classes, though every class held since 1997 has in some way helped refine this book. I’d like to thank clients who hosted classes in Dublin, Mexico City, Barcelona, London, Edmonton, and Puerto Rico; such experiences have been one of my career’s most lasting rewards. Because writing teaches writers to write, this book also owes much to its audience. I want to thank the countless readers who took time to offer suggestions over the last 18 years, both online and in person. Your feedback has also been vital to this book’s evolution and a substantial factor in its success, a benefit that seems inherent in the open source world. Reader comments have run the gamut from “You should be banned from writing books” to “God bless you for writing this book”; if consensus is possible in such matters it probably lies somewhere between these two, though to borrow a line from Tolkien: the book is still too short. I’d also like to express my gratitude to everyone who played a part in this book’s production. To all those who have helped make this book a solid product over the years —including its editors, formatters, marketers, technical reviewers, and more. And to O’Reilly for giving me a chance to work on 13 book projects; it’s been net fun (and only feels a little like the movie Groundhog Day). Additional thanks is due to the entire Python community; like most open source systems, Python is the product of many unsung efforts. It’s been my privilege to watch Preface | xlix Python grow from a new kid on the scripting languages block to a widely used tool, deployed in some fashion by almost every organization writing software. Technical disagreements aside, that’s been an exciting endeavor to be a part of. I also want to thank my original editor at O’Reilly, the late Frank Willison. This book was largely Frank’s idea. He had a profound impact on both my career and the success of Python when it was new, a legacy that I remember each time I’m tempted to misuse the word “only.” Personal Thanks Finally, a few more personal notes of thanks. To the late Carl Sagan, for inspiring an 18-year-old kid from Wisconsin. To my Mother, for courage. To my siblings, for the truths to be found in museum peanuts. To the book The Shallows, for a much-needed wakeup call. To my son Michael and daughters Samantha and Roxanne, for who you are. I’m not quite sure when you grew up, but I’m proud of how you did, and look forward to seeing where life takes you next. And to my wife Vera, for patience, proofing, Diet Cokes, and pretzels. I’m glad I finally found you. I don’t know what the next 50 years hold, but I do know that I hope to spend all of them holding you. —Mark Lutz, Amongst the Larch, Spring 2013 l | Preface PART I Getting Started CHAPTER 1 A Python Q&A Session If you’ve bought this book, you may already know what Python is and why it’s an important tool to learn. If you don’t, you probably won’t be sold on Python until you’ve learned the language by reading the rest of this book and have done a project or two. But before we jump into details, this first chapter of this book will briefly introduce some of the main reasons behind Python’s popularity. To begin sculpting a definition of Python, this chapter takes the form of a question-and-answer session, which poses some of the most common questions asked by beginners. Why Do People Use Python? Because there are many programming languages available today, this is the usual first question of newcomers. Given that there are roughly 1 million Python users out there at the moment, there really is no way to answer this question with complete accuracy; the choice of development tools is sometimes based on unique constraints or personal preference. But after teaching Python to roughly 260 groups and over 4,000 students during the last 16 years, I have seen some common themes emerge. The primary factors cited by Python users seem to be these: Software quality For many, Python’s focus on readability, coherence, and software quality in general sets it apart from other tools in the scripting world. Python code is designed to be readable, and hence reusable and maintainable—much more so than traditional scripting languages. The uniformity of Python code makes it easy to understand, even if you did not write it. In addition, Python has deep support for more advanced software reuse mechanisms, such as object-oriented (OO) and function programming. Developer productivity Python boosts developer productivity many times beyond compiled or statically typed languages such as C, C++, and Java. Python code is typically one-third to 3 one-fifth the size of equivalent C++ or Java code. That means there is less to type, less to debug, and less to maintain after the fact. Python programs also run immediately, without the lengthy compile and link steps required by some other tools, further boosting programmer speed. Program portability Most Python programs run unchanged on all major computer platforms. Porting Python code between Linux and Windows, for example, is usually just a matter of copying a script’s code between machines. Moreover, Python offers multiple options for coding portable graphical user interfaces, database access programs, webbased systems, and more. Even operating system interfaces, including program launches and directory processing, are as portable in Python as they can possibly be. Support libraries Python comes with a large collection of prebuilt and portable functionality, known as the standard library. This library supports an array of application-level programming tasks, from text pattern matching to network scripting. In addition, Python can be extended with both homegrown libraries and a vast collection of third-party application support software. Python’s third-party domain offers tools for website construction, numeric programming, serial port access, game development, and much more (see ahead for a sampling). The NumPy extension, for instance, has been described as a free and more powerful equivalent to the Matlab numeric programming system. Component integration Python scripts can easily communicate with other parts of an application, using a variety of integration mechanisms. Such integrations allow Python to be used as a product customization and extension tool. Today, Python code can invoke C and C++ libraries, can be called from C and C++ programs, can integrate with Java and .NET components, can communicate over frameworks such as COM and Silverlight, can interface with devices over serial ports, and can interact over networks with interfaces like SOAP, XML-RPC, and CORBA. It is not a standalone tool. Enjoyment Because of Python’s ease of use and built-in toolset, it can make the act of programming more pleasure than chore. Although this may be an intangible benefit, its effect on productivity is an important asset. Of these factors, the first two (quality and productivity) are probably the most compelling benefits to most Python users, and merit a fuller description. Software Quality By design, Python implements a deliberately simple and readable syntax and a highly coherent programming model. As a slogan at a past Python conference attests, the net result is that Python seems to “fit your brain”—that is, features of the language interact 4 | Chapter 1: A Python Q&A Session in consistent and limited ways and follow naturally from a small set of core concepts. This makes the language easier to learn, understand, and remember. In practice, Python programmers do not need to constantly refer to manuals when reading or writing code; it’s a consistently designed system that many find yields surprisingly uniform code. By philosophy, Python adopts a somewhat minimalist approach. This means that although there are usually multiple ways to accomplish a coding task, there is usually just one obvious way, a few less obvious alternatives, and a small set of coherent interactions everywhere in the language. Moreover, Python doesn’t make arbitrary decisions for you; when interactions are ambiguous, explicit intervention is preferred over “magic.” In the Python way of thinking, explicit is better than implicit, and simple is better than complex.1 Beyond such design themes, Python includes tools such as modules and OOP that naturally promote code reusability. And because Python is focused on quality, so too, naturally, are Python programmers. Developer Productivity During the great Internet boom of the mid-to-late 1990s, it was difficult to find enough programmers to implement software projects; developers were asked to implement systems as fast as the Internet evolved. In later eras of layoffs and economic recession, the picture shifted. Programming staffs were often asked to accomplish the same tasks with even fewer people. In both of these scenarios, Python has shined as a tool that allows programmers to get more done with less effort. It is deliberately optimized for speed of development—its simple syntax, dynamic typing, lack of compile steps, and built-in toolset allow programmers to develop programs in a fraction of the time needed when using some other tools. The net effect is that Python typically boosts developer productivity many times beyond the levels supported by traditional languages. That’s good news in both boom and bust times, and everywhere the software industry goes in between. Is Python a “Scripting Language”? Python is a general-purpose programming language that is often applied in scripting roles. It is commonly defined as an object-oriented scripting language—a definition that blends support for OOP with an overall orientation toward scripting roles. If pressed for a one-liner, I’d say that Python is probably better known as a general-purpose pro1. For a more complete look at the Python philosophy, type the command import this at any Python interactive prompt (you’ll see how in Chapter 3). This invokes an “Easter egg” hidden in Python—a collection of design principles underlying Python that permeate both the language and its user community. Among them, the acronym EIBTI is now fashionable jargon for the “explicit is better than implicit” rule. These principles are not religion, but are close enough to qualify as a Python motto and creed, which we’ll be quoting from often in this book. Is Python a “Scripting Language”? | 5 gramming language that blends procedural, functional, and object-oriented paradigms— a statement that captures the richness and scope of today’s Python. Still, the term “scripting” seems to have stuck to Python like glue, perhaps as a contrast with larger programming effort required by some other tools. For example, people often use the word “script” instead of “program” to describe a Python code file. In keeping with this tradition, this book uses the terms “script” and “program” interchangeably, with a slight preference for “script” to describe a simpler top-level file and “program” to refer to a more sophisticated multifile application. Because the term “scripting language” has so many different meanings to different observers, though, some would prefer that it not be applied to Python at all. In fact, people tend to make three very different associations, some of which are more useful than others, when they hear Python labeled as such: Shell tools Sometimes when people hear Python described as a scripting language, they think it means that Python is a tool for coding operating-system-oriented scripts. Such programs are often launched from console command lines and perform tasks such as processing text files and launching other programs. Python programs can and do serve such roles, but this is just one of dozens of common Python application domains. It is not just a better shell-script language. Control language To others, scripting refers to a “glue” layer used to control and direct (i.e., script) other application components. Python programs are indeed often deployed in the context of larger applications. For instance, to test hardware devices, Python programs may call out to components that give low-level access to a device. Similarly, programs may run bits of Python code at strategic points to support end-user product customization without the need to ship and recompile the entire system’s source code. Python’s simplicity makes it a naturally flexible control tool. Technically, though, this is also just a common Python role; many (perhaps most) Python programmers code standalone scripts without ever using or knowing about any integrated components. It is not just a control language. Ease of use Probably the best way to think of the term “scripting language” is that it refers to a simple language used for quickly coding tasks. This is especially true when the term is applied to Python, which allows much faster program development than compiled languages like C++. Its rapid development cycle fosters an exploratory, incremental mode of programming that has to be experienced to be appreciated. Don’t be fooled, though—Python is not just for simple tasks. Rather, it makes tasks simple by its ease of use and flexibility. Python has a simple feature set, but it allows programs to scale up in sophistication as needed. Because of that, it is commonly used for quick tactical tasks and longer-term strategic development. 6 | Chapter 1: A Python Q&A Session So, is Python a scripting language or not? It depends on whom you ask. In general, the term “scripting” is probably best used to describe the rapid and flexible mode of development that Python supports, rather than a particular application domain. OK, but What’s the Downside? After using it for 21 years, writing about it for 18, and teaching it for 16, I’ve found that the only significant universal downside to Python is that, as currently implemented, its execution speed may not always be as fast as that of fully compiled and lower-level languages such as C and C++. Though relatively rare today, for some tasks you may still occasionally need to get “closer to the iron” by using lower-level languages such as these that are more directly mapped to the underlying hardware architecture. We’ll talk about implementation concepts in detail later in this book. In short, the standard implementations of Python today compile (i.e., translate) source code statements to an intermediate format known as byte code and then interpret the byte code. Byte code provides portability, as it is a platform-independent format. However, because Python is not normally compiled all the way down to binary machine code (e.g., instructions for an Intel chip), some programs will run more slowly in Python than in a fully compiled language like C. The PyPy system discussed in the next chapter can achieve a 10X to 100X speedup on some code by compiling further as your program runs, but it’s a separate, alternative implementation. Whether you will ever care about the execution speed difference depends on what kinds of programs you write. Python has been optimized numerous times, and Python code runs fast enough by itself in most application domains. Furthermore, whenever you do something “real” in a Python script, like processing a file or constructing a graphical user interface (GUI), your program will actually run at C speed, since such tasks are immediately dispatched to compiled C code inside the Python interpreter. More fundamentally, Python’s speed-of-development gain is often far more important than any speed-of-execution loss, especially given modern computer speeds. Even at today’s CPU speeds, though, there still are some domains that do require optimal execution speeds. Numeric programming and animation, for example, often need at least their core number-crunching components to run at C speed (or better). If you work in such a domain, you can still use Python—simply split off the parts of the application that require optimal speed into compiled extensions, and link those into your system for use in Python scripts. We won’t talk about extensions much in this text, but this is really just an instance of the Python-as-control-language role we discussed earlier. A prime example of this dual language strategy is the NumPy numeric programming extension for Python; by combining compiled and optimized numeric extension libraries with the Python language, NumPy turns Python into a numeric programming tool that is simultaneously efficient and easy to use. When needed, such extensions provide a powerful optimization tool. OK, but What’s the Downside? | 7 Other Python Tradeoffs: The Intangible Bits I mentioned that execution speed is the only major dow