The Pandas Library for Python

Introduction to Pandas

So, what is Pandas – practically speaking? In short, it’s the major data analysis library for Python. For scientists, students, and professional developers alike, Pandas represents a central reason for any learning or interaction with Python, as opposed to a statistics-specific language like R, or a proprietary academic package like SPSS or Matlab. (Fun fact – Pandas is named after the term Panel Data, and was originally created for the analysis of financial data tables). I like to think that the final “s” stands for Series or Statistics.

Although there are plenty of ways to explore numerical data with Python out-of-the box, these will universally involve some fairly low-performance results, with a ton of boilerplate. It may sound hard to believe, but Pandas is often recommended as the next stop for Excel users who are ready to take their data analysis to the next level. Nearly any problem that can be solved with a spreadsheet program can be solved in Pandas – without all the graphical cruft.

More importantly, because problems can be solved in Pandas via Python, solutions are already automated, or could be run as a service in the cloud. Further, Pandas makes heavy use of Numpy, relying on its low level calls to produce linear math results orders of magnitude more quickly than they would be handled by Python alone. These are just a few of the reasons Pandas is recommended as one of the first libraries to learn for all Pythonistas, and remains absolutely critical to Data Scientists.

About the Data

In this post, we’re going to be using a fascinating data set to demonstrate a useful slice of the Pandas library. This data set is particularly interesting as it’s part of a real world example, and we can all imagine people lined up at an airport (a place where things do occasionally go wrong). When looking at the data, I imagine people people sitting in those uncomfortable airport seats having just found out that their luggage is missing – not just temporarily, but it’s nowhere to be found in the system! Or, better yet, imagine that a hardworking TSA employee accidentally broke a precious family heirloom.

So it’s time to fill out another form, of course. Now, getting data from forms is an interesting process as far as data gathering is concerned, as we have a set of data that happens at specific times. This actually means we can interpret the entries as a Time Series. Also, because people are submitting the information, we can learn things about a group of people, too.

Back to our example: let’s say we work for the TSA and we’ve been tasked with getting some insights about when these accidents are most likely to happen, and make some recommendations for improving the service.

Pandas, luckily, is a one-stop shop for exploring and analyzing this data set. Feel free to download the excel file into your project folder to get started, or run the curl command below. Yes, pandas can read .xls or .xlsx files with a single call to pd.read_excel()! In fact, it’s often helpful for beginners experienced with .csv or excel files to think about how they would solve a problem in excel, and then experience how much easier it can be in Pandas.

So, without further ado, open your terminal, a text editor, or your favorite IDE, and take a look for yourself with the guidance below.

Example data:

Take for example, some claims made against the TSA during a screening process of persons or a passenger’s property due to an injury, loss, or damage. The claims data information includes claim number, incident date, claim type, claim amount, status, and disposition.

Directory: TSA Claims Data
Our Data Download: claims-2014.xls

Setup

To start off, let’s create a clean directory. You can put this wherever you’d like, or create a project folder in an IDE. Use your install method of choice to get Pandas: Pip is probably the easiest.

Install pandas along with xldr for loading Excel formatted files, matplotlib for plotting graphs, and Numpy for high-level mathematical functions.

Optional: download the example data with curl:

Launch Python:

Import packages:

Loading Data

Loading data with Pandas is easy. Pandas can accurately read data from almost any common format including JSON, CSV, and SQL. Data is loaded into Pandas’ “flagship” data structure, the DataFrame.

That’s a term you’ll want to remember. You’ll be hearing a lot about DataFrames. If that term seems confusing – think about a table in a database, or a sheet in Excel. The main point is that there is more than one column: each row or entry has multiple fields which are consistent from one row to the next.

You can load the example data straight from the web:

Less cooly, data can be loaded from a file:

Basic Operations

Print information about a DataFrame including the index dtype and column dtypes, non-null values, and memory usage. DataFrame.info() is one of the more useful and versatile methods attached to DataFrames (there are nearly 150!).

View the first n rows:

List all the columns in the DataFrame:

Return a single column (important – also referred to as a Series):

Hopefully, you’re starting to get an idea of what claims-2014.xls’s data is all about.

>>> df.info()
 'pandas.core.frame.dataframe'="">
RangeIndex: 8855 entries, 0 to 8854
Data columns (total 11 columns):
Date Received    8855 non-null datetime64[ns]
Incident Date    8855 non-null datetime64[ns]
Airport Code     8855 non-null object
Airport Name     8855 non-null object
Airline Name     8855 non-null object
Claim Type       8855 non-null object
Claim Site       8855 non-null object
Item Category    8855 non-null object
Close Amount     8855 non-null object
Disposition      8855 non-null object
dtypes: datetime64[ns](2), object(8)
memory usage: 761.1+ KB

>>> df.dtypes
Date Received    datetime64[ns]
Incident Date    datetime64[ns]
Airport Code             object
Airport Name             object
Airline Name             object
Claim Type               object
Claim Site               object
Item Category            object
Close Amount             object
Disposition              object
dtype: object

>>> grp = df.groupby(by=df['Incident Date'].dt.hour)
>>> grp['Item Category'].describe()
              count unique                   top freq
Incident Date
0              3421    146  Baggage/Cases/Purses  489
1                 6      5                 Other    2
2                11      9                     -    2
3                 5      5     Jewelry & Watches    1
4                49     18  Baggage/Cases/Purses    6
5               257     39                     -   33
6               357     54                     -   43
7               343     43              Clothing   41
8               299     47                     -   35
9               305     41                     -   31
10              349     45                 Other   43
11              343     41                     -   45
12              363     51                 Other   41
13              359     55                     -   45
14              386     60  Baggage/Cases/Purses   49
15              376     51                 Other   41
16              351     43  Personal Electronics   35
17              307     52                 Other   34
18              289     43  Baggage/Cases/Purses   37
19              241     46  Baggage/Cases/Purses   26
20              163     31  Baggage/Cases/Purses   23
21              104     32  Baggage/Cases/Purses   20
22              106     33  Baggage/Cases/Purses   19
23               65     25  Baggage/Cases/Purses   14

>>> df['Close Amount'].head()
0     0
1     0
2    50
3     0
4     0
Name: Close Amount, dtype: object

>>> df['Close Amount'].tail()
8850      0
8851    800
8852      0
8853    256
8854      -
Name: Close Amount, dtype: object

>>> def dash_to_zero(x):
>>>    if '-' in str(x):
>>>        return float() # 0.0
>>>    else:
>>>        return x  # just return the input value as-is

>>> df['Close Amount'] = df['Close Amount'].apply(dash_to_zero)
>>> df['Close Amount'].dtype
dtype('float64')

>>> df['Close Amount'].apply(lambda x: 0. if '-' in str(x) else x)

>>> df.sum()
Close Amount    538739.51
dtype: float64


>>> df.min()
Date Received              2014-01-01 00:00:00
Incident Date              2011-08-24 08:30:00
Airport Code                                 -
Airport Name      Albert J Ellis, Jacksonville
Airline Name                                 -
Claim Type                                   -
Claim Site                                   -
Item Category                                -
Close Amount                                 0
Disposition                                  -


>>> df.max()
Date Received                       2014-12-31 00:00:00
Incident Date                       2014-12-31 00:00:00
Airport Code                                        ZZZ
Airport Name                 Yuma International Airport
Airline Name                                 XL Airways
Claim Type                              Property Damage
Claim Site                                        Other
Item Category    Travel Accessories; Travel Accessories
Close Amount                                    25483.4
Disposition                                      Settle
dtype: object