Python for Data Science for Dummies

Introduction

About This Book

Foolish Assumptions

Icons Used in This Book

Beyond the Book

Where to Go from Here

Part I: Getting Started with Python for Data Science

Chapter 1: Discovering the Match between Data Science and Python

• Defining the Sexiest Job of the 21^st Century
• Considering the emergence of data science
• Outlining the core competencies of a data scientist
• Linking data science and big data
• Understanding the role of programming
• Creating the Data Science Pipeline
• Preparing the data
• Performing exploratory data analysis
• Learning from data
• Visualizing
• Obtaining insights and data products
• Understanding Python's Role in Data Science
• Considering the shifting profile of data scientists
• Working with a multipurpose, simple and efficient language
• Learning to Use Python Fast
• Loading data
• Training a model
• Viewing a result

Chapter 2: Introducing Python's Capabilities and Wonders

• Why Python?
• Grasping Python's core philosophy
• Discovering present and future development goals
• Working with Python
• Getting a taste of the language
• Understanding the need for indentation
• Working at the command line or in the IDE
• Performing Rapid Prototyping and Experimentation
• Considering Speed of Execution
• Visualizing Power
• Using the Python Ecosystem for Data Science
• Accessing scientific tools using SciPy
• Performing fundamental scientific computing using NumPy
• Performing data analysis using pandas
• Implementing machine learning using Scikit learn
• Plotting the data using matplotlib
• Parsing HTML documents using Beautiful Soup

Chapter 3: Setting Up Python for Data Science

• Considering the Off the Shelf Cross Platform Scientific Distributions
• Getting Continuum Analytics Anaconda
• Getting Enthought Canopy Express
• Getting pythonxy
• Getting WinPython
• Installing Anaconda on Windows
• Installing Anaconda on Linux
• Installing Anaconda on Mac OS X
• Downloading the Datasets and Example Code
• Using IPython Notebook
• Defining the code repository
• Understanding the datasets used in this book

Chapter 4: Reviewing Basic Python

• Working with Numbers and Logic
• Performing variable assignments
• Doing arithmetic
• Comparing data using Boolean expressions
• Creating and Using Strings
• Interacting with Dates
• Creating and Using Functions
• Creating reusable functions
• Calling functions in a variety of ways
• Using Conditional and Loop Statements
• Making decisions using the if statement
• Choosing between multiple options using nested decisions
• Performing repetitive tasks using for
• Using the while statement
• Storing Data Using Sets, Lists and Tuples
• Performing operations on sets
• Working with lists
• Creating and using Tuples
• Defining Useful Iterators
• Indexing Data Using Dictionaries

Part II: Getting Your Hands Dirty with Data

Chapter 5: Working with Real Data

• Uploading, Streaming and Sampling Data
• Uploading small amounts of data into memory
• Streaming large amounts of data into memory
• Sampling data
• Accessing Data in Structured Flat File Form
• Reading from a text file
• Reading CSV delimited format
• Reading Excel and other Microsoft Office files
• Sending Data in Unstructured File Form
• Managing Data from Relational Databases
• Interacting with Data from NoSQL Databases
• Accessing Data from the Web

Chapter 6: Conditioning Your Data

• Juggling between NumPy and pandas
• Knowing when to use NumPy
• Knowing when to use pandas
• Validating Your Data
• Figuring out what's in your data
• Removing duplicates
• Creating a data map and data plan
• Manipulating Categorical Variables
• Creating categorical variables
• Renaming levels
• Combining levels
• Dealing with Dates in Your Data
• Formatting date and time values
• Using the right time transformation
• Dealing with Missing Data
• Finding the missing data
• Encoding missingness
• Imputing missing data
• Slicing and Dicing and Selecting Data
• Slicing rows
• Slicing columns
• Dicing
• Concatenating and Transforming
• Adding new cases and variables
• Removing data
• Sorting and shuffling
• Aggregating Data at Any Level

Chapter 7: Shaping Data

• Working with HTML Pages
• Parsing XML and HTML
• Using XPath for data extraction
• Working with Raw Text
• Dealing with Unicode
• Stemming and removing stop words
• Introducing regular expressions
• Using the Bag of Words Model and Beyond
• Understanding the bag of words model
• Working with ngrams
• Implementing TFIDF transformations
• Working with Graph Data
• Understanding the adjacency matrix
• Using NetworkX basics

Chapter 8: Putting What You Know in Action

• Contextualizing Problems and Data
• Evaluating a data science problem
• Researching solutions
• Formulating a hypothesis
• Preparing your data
• Considering the Art of Feature Creation
• Defining feature creation
• Combining variables
• Understanding binning and discretization
• Using indicator variables
• Transforming distributions
• Performing Operations on Arrays
• Using vectorization
• Performing simple arithmetic on vectors and matrices
• Performing matrix vector multiplication
• Performing matrix multiplication

Part III: Visualizing the Invisible

Chapter 9: Getting a Crash Course in MatPlotLib

• Starting with a Graph

• Defining the plot
• Drawing multiple lines and plots
• Saving your work
• Setting the Axis, Ticks, Grids
• Getting the axes
• Formatting the axes
• Adding grids
• Defining the Line Appearance
• Working with line styles
• Using colors
• Adding markers
• Using Labels, Annotations and Legends
• Adding labels
• Annotating the chart
• Creating a legend

Chapter 10: Visualizing the Data

• Choosing the Right Graph
• Showing parts of a whole with pie charts
• Creating comparisons with bar charts
• Showing distributions using histograms
• Depicting groups using box plots
• Seeing data patterns using scatterplots
• Creating Advanced Scatterplots
• Depicting groups
• Showing correlations
• Plotting Time Series
• Representing time on axes
• Plotting trends over time
• Plotting Geographical Data
• Visualizing Graphs
• Developing undirected graphs
• Developing directed graphs

Chapter 11: Understanding the Tools

• Using the IPython Console
• Interacting with screen text
• Changing the window appearance
• Getting Python help
• Getting IPython help
• Using magic functions
• Discovering objects
• Using IPython Notebook
• Working with styles
• Restarting the kernel
• Restoring a checkpoint
• Performing Multimedia and Graphic Integration
• Embedding plots and other images
• Loading examples from online sites
• Obtaining online graphics and multimedia

Part IV: Wrangling Data

Chapter 12: Stretching Python's Capabilities

• Playing with Scikit learn
• Understanding classes in Scikit learn
• Defining applications for data science
• Performing the Hashing Trick
• Using hash functions
• Demonstrating the hashing trick
• Working with deterministic selection
• Considering Timing and Performance
• Benchmarking with time it
• Working with the memory profiler
• Running in Parallel
• Performing multicore parallelism
• Demonstrating multiprocessing

Chapter 13: Exploring Data Analysis

• The EDA Approach
• Defining Descriptive Statistics for Numeric Data
• Measuring central tendency
• Measuring variance and range
• Working with percentiles
• Defining measures of normality
• Counting for Categorical Data
• Understanding frequencies
• Creating contingency tables
• Creating Applied Visualization for EDA
• Inspecting boxplots
• Performing to tests after boxplots
• Observing parallel coordinates
• Graphing distributions
• Plotting scatterplots
• Understanding Correlation
• Using covariance and correlation
• Using nonparametric correlation
• Considering chi square for tables
• Modifying Data Distributions
• Using the normal distribution
• Creating a Z score standardization
• Transforming other notable distributions

Chapter 14: Reducing Dimensionality

• Understanding SVD
• Looking for dimensionality reduction
• Using SVD to measure the invisible
• Performing Factor and Principal Component Analysis
• Considering the psychometric model
• Looking for hidden factors
• Using components, not factors
• Achieving dimensionality reduction
• Understanding Some Applications
• Recognizing faces with PCA
• Extracting Topics with NMF
• Recommending movies

Chapter 15: Clustering

• Clustering with K means
• Understanding centroid based algorithms
• Creating an example with image data  
• Looking for optimal solutions
• Clustering big data
• Performing Hierarchical Clustering
• Moving Beyond the Round-Shaped Clusters: DBScan

Chapter 16: Detecting Outliers in Data

• Considering Detection of Outliers
• Finding more things that can go wrong
• Understanding anomalies and novel data
• Examining a Simple Univariate Method
• Leveraging on the Gaussian distribution
• Making assumptions and checking out
• Developing a Multivariate Approach
• Using principal component analysis
• Using cluster analysis
• Automating outliers detection with SVM

Part V: Learning from Data

Chapter 17: Exploring Four Simple and Effective Algorithms

• Guessing the Number: Linear Regression
• Defining the family of linear models
• Using more variables
• Understanding limitations and problems
• Moving to Logistic Regression
• Applying logistic regression
• Considering when classes are more
• Making Things as Simple as Naïve Bayes
• Finding out that Naïve Bayes isn't so naïve
• Predicting text classifications
• Learning Lazily with Nearest Neighbors
• Predicting after observing neighbors
• Choosing your k parameter wisely

Chapter 18: Performing Cross Validation, Selection and Optimization

• Pondering the Problem of Fitting a Model
• Understanding bias and variance
• Defining a strategy for picking models
• Dividing between training and test sets
• Cross Validating
• Using cross validation on k folds
• Sampling stratifications for complex data
• Selecting Variables Like a Pro
• Selecting by univariate measures
• Using a greedy search
• Pumping Up Your Hyper parameters
• Implementing a grid search  
• Trying a randomized search

Chapter 19: Increasing Complexity with Linear and Nonlinear Tricks

• Using Nonlinear Transformations
• Doing variable transformations
• Creating interactions between variables
• Regularizing Linear Models
• Relying on Ridge regression (L2)
• Using the Lasso (L1)
• Leveraging regularization
• Combining L1 & L2: Elasticnet
• Fighting with Big Data Chunk by Chunk
• Determining when there is too much data
• Implementing Stochastic Gradient Descent
• Understanding Support Vector Machines
• Relying on a computational method
• Fixing many new parameters
• Classifying with SVC
• Going nonlinear is easy
• Performing regression with SVR
• Creating a stochastic solution with SVM

Chapter 20: Understanding the Power of the Many

• Starting with a Plain Decision Tree
• Understanding a decision tree
• Creating classification and regression trees
• Making Machine Learning Accessible
• Working with a Random Forest classifier
• Working with a Random Forest regressor
• Optimizing a Random Forest
• Boosting Predictions
• Knowing that many weak predictors win
• Creating a gradient boosting classifier
• Creating a gradient boosting regressor
• Using GBM hyper parameters

Part VI: The Part of Tens

Chapter 21: Ten Essential Data Science Resource Collections

• Gaining Insights with Data Science Weekly
• Obtaining a Resource List at U Climb Higher
• Getting a Good Start with KDnuggets
• Accessing the Huge List of Resources on Data Science Central
• Obtaining the Facts of Open Source Data Science from Masters
• Locating Free Learning Resources with Quora
• Receiving Help with Advanced Topics at Conductrics
• Learning New Tricks from the Aspirational Data Scientist
• Finding Data Intelligence and Analytics Resources at Analytic Bridge
• Zeroing In on Developer Resources with Jonathan Bower

Chapter 22: Ten Data Challenges You Should Take

• Meeting the Data Science London + Scikit learn Challenge
• Predicting Survival on the Titanic
• Finding a Kaggle Competition that Suits Your Needs
• Honing Your Overfit Strategies
• Trudging Through the MovieLens Dataset  
• Getting Rid of Spam Emails
• Working with Handwritten Information
• Working with Pictures
• Analyzing Amazon.com Reviews
• Interacting with a Huge Graph

Index