MetaSnake - misc

Python for Finance: Pandas Resample, Groupby, and Rolling

2023-05-31T00:00:00-06:00

Republication note: I originally wrote this article for Ponder, where it was published on May 31, 2023. Ponder later removed the post following its acquisition, so I have preserved it here. The Ponder-specific code reflects the environment available when the article was written; the pandas concepts remain broadly applicable. View the archived original.

Exploring FDIC bank failures with pandas time-series tools.

Welcome back! Today we will explore how you can leverage pandas to understand trends in time-series data quickly, with a particular focus on time-series methods like .resample, grouping by dates, and performing .rolling operations to smooth out the results. Along the way, I might grumble a bit (I’d like to think that the grey strands in my beard are from wisdom and not fighting with a computer), but I intend to give you some superpowers.

If you work with time series data, you will want to understand .resample, grouping, and .rolling, as they will make your life much easier.

The code below preserves that original environment and demonstrates the pandas concepts as they appeared in the original post.

Let’s start by picking a dataset that might be relevant considering recent news… bank failures.

Data Overview: FDIC BankFind Suite

In recent years, the banking sector in the United States has witnessed many financial institution failures, leading to substantial economic losses and systemic risks. The Federal Deposit Insurance Corporation (FDIC) bank failure data provides valuable insights into the underlying causes of these failures and helps develop effective risk management and regulatory compliance strategies. By exploring this data with the aid of the popular data analysis library, pandas, one can identify patterns and trends, prepare data for modeling, and visualize findings effectively.

BankFind Suite is a comprehensive database provided by the FDIC, which allows users to search for current and former FDIC-insured banking institutions based on name, FDIC certificate number, website, or location. This tool offers detailed information on an institution’s financial history and trends, enabling users to track the performance of individual institutions, groups of institutions, or the industry as a whole.

Let’s use the Bank Failure data to explore. I will download the data from 1934 to 2023 containing over 4000 entries.

import json
import pandas as pd

>>> df = pd.read_csv('bankfail.csv')
>>> df
    CERT    CHCLASS1    CITYST          COST    FAILDATE    FIN ID  NAME                    QBFASSET    QBFDEP      RESTYPE RESTYPE1    SAVR
0   15426.0 NM      ALMENA, KS      16806.0 10/23/2020  10538   4104    ALMENA STATE BANK           65733.0     64941.0     FAILURE PA      DIF
1   16748.0 NM      FORT WALTON BEACH, FL   7247.0  10/16/2020  10537   4103    FIRST CITY BANK OF FLORIDA      136566.0    133936.0    FAILURE PA      DIF
2   14361.0 NM      BARBOURSVILLE, WV   45913.0 4/3/2020    10536   4102    THE FIRST STATE BANK            151808.0    143102.0    FAILURE PA      DIF
3   18265.0 NM      ERICSON, NE     25293.0 2/14/2020   10535   4101    ERICSON STATE BANK          100879.0    95159.0     FAILURE PA      DIF
4   21111.0 N       NEWARK, NJ      1946.0  11/1/2019   10534   4100    CITY NATIONAL BANK OF NEW JERSEY    120574.0    111234.0    FAILURE PA      DIF
... ... ...     ...         ... ...     ... ... ...                 ...     ...     ... ...     ...
4099    NaN NM      LEWISPORT, KY       NaN 8/6/1934    0   137 BANK OF LEWISPORT           81.0        68.0        FAILURE PO      BIF
4100    NaN N       LIMA, MT        NaN 7/18/1934   0   135 FIRST NATIONAL BANK OF LIMA     91.0        42.0        FAILURE PO      BIF
4101    NaN NM      FLORENCE, IN        NaN 7/18/1934   0   136 FLORENCE DEPOSIT BANK           105.0       69.0        FAILURE PO      BIF
4102    NaN NM      EAST PEORIA, IL     NaN 5/28/1934   0   133 FON DU LAC STATE BANK           374.0       238.0       FAILURE PO      BIF
4103    NaN NM      PITTSBURGH, PA      NaN 4/19/1934   0   134 BANK OF AMERICA TRUST CO.       1435.0      1064.0      FAILURE PO      BIF
4104 rows × 13 columns

It looks like my CSV export loaded ok. However, the existing column names are a little annoying. I’m going to clean them up.

Here’s some code to rename the columns, create a city and state column, and convert the date column to a datetime type.

>>> colnames = {'CERT': 'cert',
...   'CHCLASS1': 'charter_class',
...    'CITYST': 'location',
...    'COST': 'estimated_loss',
...    'FAILDATE': 'effective_date',
...    'FIN': 'fin',
...    'ID': 'id',
...    'NAME': 'institution',
...    'QBFASSET': 'total_assets',
...    'QBFDEP': 'total_deposits',
...    'RESTYPE': 'resolution',
...    'RESTYPE1': 'transaction_type',
...    'SAVR': 'insurance_fund'}

>>> def tweak_bank(df):
...   return (df
...           .assign(FAILDATE=pd.to_datetime(df.FAILDATE, format='%m/%d/%Y'),
...                   city=df.CITYST.str.split(',').str[0],
...                   state=df.CITYST.str.split(',').str[1],
...                  )
...           .drop(columns=['CITYST'])
...           .rename(columns=colnames)
...          )

>>> bank = tweak_bank(df)
>>> bank
    cert    charter_class   estimated_loss  effective_date  fin id  institution             total_assets    total_deposits  resolution  transaction_type    insurance_fund  city            state
0   15426.0 NM      16806.0     2020-10-23  10538   4104    ALMENA STATE BANK           65733.0     64941.0     FAILURE     PA          DIF     ALMENA          KS
1   16748.0 NM      7247.0      2020-10-16  10537   4103    FIRST CITY BANK OF FLORIDA      136566.0    133936.0    FAILURE     PA          DIF     FORT WALTON BEACH   FL
2   14361.0 NM      45913.0     2020-04-03  10536   4102    THE FIRST STATE BANK            151808.0    143102.0    FAILURE     PA          DIF     BARBOURSVILLE       WV
3   18265.0 NM      25293.0     2020-02-14  10535   4101    ERICSON STATE BANK          100879.0    95159.0     FAILURE     PA          DIF     ERICSON         NE
4   21111.0 N       1946.0      2019-11-01  10534   4100    CITY NATIONAL BANK OF NEW JERSEY    120574.0    111234.0    FAILURE     PA          DIF     NEWARK          NJ
... ... ...     ...     ...     ... ... ...                 ...     ...     ...     ...         ...     ...         ...
4099    NaN NM      NaN     1934-08-06  0   137 BANK OF LEWISPORT           81.0        68.0        FAILURE     PO          BIF     LEWISPORT       KY
4100    NaN N       NaN     1934-07-18  0   135 FIRST NATIONAL BANK OF LIMA     91.0        42.0        FAILURE     PO          BIF     LIMA            MT
4101    NaN NM      NaN     1934-07-18  0   136 FLORENCE DEPOSIT BANK           105.0       69.0        FAILURE     PO          BIF     FLORENCE        IN
4102    NaN NM      NaN     1934-05-28  0   133 FON DU LAC STATE BANK           374.0       238.0       FAILURE     PO          BIF     EAST PEORIA     IL
4103    NaN NM      NaN     1934-04-19  0   134 BANK OF AMERICA TRUST CO.       1435.0      1064.0      FAILURE     PO          BIF     PITTSBURGH      PA
4104 rows × 14 columns

Not bad. This is going to be great data for learning .resample, .groupby, and .rolling.

Data Dictionary

Here is the description of the columns for those (like me) who aren’t subject matter experts. (All descriptions taken from here.)

cert – Certificate number assigned by FDIC to identify the institution
charter_class – The FDIC assigns classification codes indicating an institution’s charter type (commercial bank, savings bank, or savings association), its chartering agent (state or federal government), its Federal Reserve membership status (member or nonmember), and its primary federal regulator (state-chartered institutions are subject to both federal and state supervision). These codes are:
N National chartered commercial bank supervised by the Office of the Comptroller of the
SM State charter Fed member commercial bank supervised by the Federal Reserve
NM State charter Fed nonmember commercial bank supervised by the FDIC
SA State or federal charter savings association supervised by the Office of Thrift Supervision or Office of the Comptroller of the Currency
SB State charter savings bank supervised by the FDIC
estimated_loss – the difference between the amount disbursed from the Deposit Insurance Fund (DIF) to cover obligations to insured depositors and the amount estimated to be ultimately recovered from the liquidation of the receivership estate. Estimated losses reflect unpaid principal amounts deemed unrecoverable and do not reflect interest that may be due on the DIF’s administrative or subrogated claims should its principal be repaid in full.
effective_date – date of failure.
fin – Financial institute number. Another unique identifier.
id – Institution directory number
name – Legal name of the institution.
total_assets – The Total assets owned by the institution, including cash, loans, securities, bank premises, and other assets as of the last Call Report or Thrift Financial Report.
total_deposits – Total including demand deposits, money market deposits, other savings deposits, time deposits and deposits in foreign offices as of the last Call Report.
resolution – Failure stature. Failure or assistance by merging with other institution.
transaction_type – Resolution type. Cat 1 – Institution’s charter survives. Cat 2 – Charter terminated. Cat 3 – Payout:
A/A – Cat1. (Assistance Transactions). Assistance was provided to the acquirer, who purchased the entire institution.
REP – Cat1. Re-privatization, management takeover with or without assistance at takeover, followed by a sale with or without additional assistance.
P&A – Cat2. Purchase and Assumption, where some or all of the deposits, certain other liabilities, and a portion of the assets (sometimes all of the assets) were sold to an acquirer. It was not determined if all the deposits (PA) or only the insured deposits (PI) were assumed.
PA – Cat2. Purchase and Assumption, where the insured and uninsured deposits, certain other liabilities, and a portion of the assets were sold to an acquirer.
PI – Cat2. Purchase and Assumption of the insured deposits only, where the traditional P&A was modified so that the acquiring institution assumed only the insured deposits.
IDT – Cat2. Insured deposit Transfer, where the acquiring institution served as a paying agent for the insurer, established accounts on their books for depositors and often acquired some assets as well.
ABT – Cat2. Asset-backed transfer, an FSLIC transaction that is very similar to an IDT.
MGR – Cat2.An institution where FSLIC took over management and generally provided financial assistance. FSLIC closed down before the institution was sold. transactions
PO – Cat3 – Payout, where the insurer paid the depositors directly and placed the assets in a liquidating receivership.
insurance_fund – Insurance fund.

Counts of failures by year

Let’s use pandas to find the count of failures by year.

When I hear the words “by year,” I immediately think we need to group this data by year. This is like one of those word math problems that everyone hated in junior high until they understood how to convert the words to math. The same thing happens here. When your boss tells you they want “failures by year,” you should think, “I’m going to use the year column in the pandas .groupby method.”

In this case, we don't have a year column. But due to the magic of datetime columns, we can access the year through the .dt accessor.

>>> bank.effective_date
0      2020-10-23
1      2020-10-16
2      2020-04-03
3      2020-02-14
4      2019-11-01
          ...    
4099   1934-08-06
4100   1934-07-18
4101   1934-07-18
4102   1934-05-28
4103   1934-04-19
Name: effective_date, Length: 4104, dtype: datetime64[ns]

>>> (bank
...  .groupby(bank.effective_date.dt.year)
...  .size()
... )
effective_date
1934     9
1935    25
1936    69
1937    75
1938    74
        ..
2015     8
2016     5
2017     8
2019     4
2020     4
Length: 84, dtype: int64

This is ok. I would prefer a visualization to this table of data. Let’s try a bar chart.

(bank
 .groupby(bank.effective_date.dt.year)
 .size()
 .plot.bar()
)

This is a start, but the x-axis is not happy. Just be aware that if you intend on making bar plots with pandas, it converts the index to categories. In this case, the dates are converted to strings. This makes me sad, but not every library is perfect.

https://github.com/pandas-dev/pandas/issues/17001

Let’s try and make a line plot instead.

(bank
 .groupby(bank.effective_date.dt.year)
 .size()
 .plot()
)

This looks good. Line plots in pandas (unlike bar plots) respect dates in the index.

Now let’s jump into grouping functionality that is time-series specific.

Using Offset Aliases with Pandas Resample

Hidden away in the pandas documentation is a mention of offset aliases. (Sadly, these are not accessible from the docstrings inside of Jupyter, which is one lament I have with pandas otherwise excellent documentation).

You can find them here, or documented in my book, Effective Pandas.

In short, Y, Q, M, W, and D mean year, quarter, month, week, and day respectively. If you memorize these single-letter shortcuts, the .resample method, or the pd.Grouper class, you can quickly aggregate date information by different time intervals.

Let's try and view failures by month using the .resample method. Pandas resampling is powerful because it lets you convert time-series data with one time interval into time-series data with different time intervals. You can upsample so you have more rows over shorter intervals, or downsample so you have fewer rows over longer intervals. The key to using this method is sticking a date column into the index and then calling .resample instead of .groupby:

>>> (bank
...  .set_index('effective_date')
...  .resample('M')
...  .size()
... )
effective_date
1934-04-30    1
1934-05-31    1
1934-07-31    2
1934-08-31    1
1934-09-30    1
             ..
2019-10-31    2
2019-11-30    1
2020-02-29    1
2020-04-30    1
2020-10-31    2
Length: 540, dtype: int64

Notice the index in the resulting Series. Each entry ends on the last day of the month. This is because we passed the M (month) offset alias into the .resample method.

I generally plot this by chaining on a call to .plot. This will plot the date on the x-axis and draw the values on the y-axis.

(bank
 .set_index('effective_date')
 .resample('M')
 .size()
 .plot()
)

Smoothing By Rougher Aggregations Using Pandas Resample

Let’s try something a little fancier. We will make a function that plots the monthly failures (which we already saw was bumpy) and then plot the yearly aggregation on top.

I'll use the .pipe method to refactor the plotting logic into a single function.

>>> import matplotlib.pyplot as plt
>>> fig, ax = plt.subplots(figsize=(10,4))
>>> def plot_monthly_and_yearly(df, ax):
...     (df
...       .resample('M')
...       .size()
...       .plot(ax=ax, label='Monthly')
...     )
...     (df
...       .resample('Y')
...       .size()
...       .plot(ax=ax, label='Annual')
...     )
...     ax.legend()
...     return df

>>> (bank
...  .set_index('effective_date')
...  .pipe(plot_monthly_and_yearly, ax=ax)
... )
        cert    charter_class   estimated_loss  fin id  institution             total_assets    total_deposits  resolution  transaction_type    insurance_fund  city            state
effective_date                                                  
2020-10-23  15426.0 NM      16806.0     10538   4104    ALMENA STATE BANK           65733.0     64941       FAILURE     PA          DIF     ALMENA          KS
2020-10-16  16748.0 NM      7247.0      10537   4103    FIRST CITY BANK OF FLORIDA      136566.0    133936      FAILURE     PA          DIF     FORT WALTON BEACH   FL
2020-04-03  14361.0 NM      45913.0     10536   4102    THE FIRST STATE BANK            151808.0    143102      FAILURE     PA          DIF     BARBOURSVILLE       WV
2020-02-14  18265.0 NM      25293.0     10535   4101    ERICSON STATE BANK          100879.0    95159       FAILURE     PA          DIF     ERICSON         NE
2019-11-01  21111.0 N       1946.0      10534   4100    CITY NATIONAL BANK OF NEW JERSEY    120574.0    111234      FAILURE     PA          DIF     NEWARK          NJ
...     ... ...     ...     ... ... ...                 ...     ...     ...     ...         ...     ...         ...
1934-08-06  NaN NM      NaN     0   137 BANK OF LEWISPORT           81.0        68      FAILURE     PO          BIF     LEWISPORT       KY
1934-07-18  NaN N       NaN     0   135 FIRST NATIONAL BANK OF LIMA     91.0        42      FAILURE     PO          BIF     LIMA            MT
1934-07-18  NaN NM      NaN     0   136 FLORENCE DEPOSIT BANK           105.0       69      FAILURE     PO          BIF     FLORENCE        IN
1934-05-28  NaN NM      NaN     0   133 FON DU LAC STATE BANK           374.0       238     FAILURE     PO          BIF     EAST PEORIA     IL
1934-04-19  NaN NM      NaN     0   134 BANK OF AMERICA TRUST CO.       1435.0      1064        FAILURE     PO          BIF     PITTSBURGH      PA
4104 rows x 13 columns

Resolution Type Over Time

I want to talk about one more feature of pandas: the pd.Grouper class. I use this class to group by different frequencies of a date column without having to stick the date in the index and call .resample. You might ask, why?

The .resample method only allows us to have a single grouping. If you have played around with .groupby, then you know that you can provide multiple columns or series to .groupby. In this example, let's assume this word problem:

What are the counts of the different resolution types by month?

This is a little tricky. We have “by month,” so we want to group by month, but we also really have “size by resolution type by month.”

A first stab might look like this:

>>> (bank
...  .groupby([bank.effective_date.dt.month, 'resolution'])
...  .size()
... )
effective_date  resolution
1               ASSISTANCE     22
                FAILURE       274
2               ASSISTANCE     26
                FAILURE       302
3               ASSISTANCE     32
                FAILURE       443
4               ASSISTANCE     87
                FAILURE       328
5               ASSISTANCE     30
                FAILURE       283
6               ASSISTANCE     28
                FAILURE       298
7               ASSISTANCE     48
                FAILURE       359
8               ASSISTANCE     62
                FAILURE       280
9               ASSISTANCE     57
                FAILURE       219
10              ASSISTANCE     47
                FAILURE       284
11              ASSISTANCE     30
                FAILURE       212
12              ASSISTANCE    110
                FAILURE       243
dtype: int64

This sort of works if we want to know how many banks failed in December. But we want monthly data over time. Because we also want to group by resolution, we can't use the .resample method. This is where pd.Grouper comes in.

We provide the name of a datetime column and a frequency (this is that offset alias that we talked about above). The code looks like this:

>>> (bank
...  .groupby([pd.Grouper(key='effective_date', freq='M'),
...            'resolution'])
...  .size()
... )
effective_date  resolution
1934-04-30      FAILURE       1
1934-05-31      FAILURE       1
1934-07-31      FAILURE       2
1934-08-31      FAILURE       1
1934-09-30      FAILURE       1
                             ..
2019-10-31      FAILURE       2
2019-11-30      FAILURE       1
2020-02-29      FAILURE       1
2020-04-30      FAILURE       1
2020-10-31      FAILURE       2
Length: 635, dtype: int64

By now, you know that I want to plot this. However, tacking on .plot is unsatisfying as we have a hierarchical index and this sticks both date and resolution as a tuple on the x-axis:

(bank
 .groupby([pd.Grouper(key='effective_date', freq='M'),
           'resolution'])
 .size()
 .plot()
)

What I want to do instead is unstack the innermost index, the resolution index. The .unstack method will pull out the inner index by default and stick it up into the columns. Now we have monthly dates in the index and a column for each resolution type.

>>> (bank
...  .groupby([pd.Grouper(key='effective_date', freq='M'),
...            'resolution'])
...  .size()
...  .unstack(level=1)
... )
        __reduced___ASSISTANCE  __reduced___FAILURE
effective_date      
1934-04-30  NaN         1
1934-05-31  NaN         1
1934-07-31  NaN         2
1934-08-31  NaN         1
1934-09-30  NaN         1
...     ...         ...
2019-10-31  NaN         2
2019-11-30  NaN         1
2020-02-29  NaN         1
2020-04-30  NaN         1
2020-10-31  NaN         2
540 rows x 2 columns

If we plot this, we will get a line for each resolution type over time. It looks like the FAILURE resolution is much more common since 1989.

def fix_cols(name):
    return name.replace('__reduced___', '')

(bank
 .groupby([pd.Grouper(key='effective_date', freq='M'),
           'resolution'])
 .size()
 .unstack(level=1)
 .fillna(0)
 .rename(columns=fix_cols)
 .plot()
)

Let’s do a 12-month rolling average to smooth this out a bit.

(bank
 .groupby([pd.Grouper(key='effective_date', freq='M'),
           'resolution'])
 .size()
 .unstack(level=1)
 .rename(columns=fix_cols)
 .fillna(0)
 .rolling(12)
 .mean()
 .plot()
)

Below I plot the yearly resolution size. Again, you can see that the basic shape is the same, but the yearly rolling average tells a better story.

(bank
 .groupby([pd.Grouper(key='effective_date', freq='Y'),
           'resolution'])
 .size()
 .unstack(level=1)
 .rename(columns=fix_cols)
 .fillna(0)
 .plot()
)

And finally, let’s look at the 12-month rolling average by resolution type.

(bank
 .groupby([pd.Grouper(key='effective_date', freq='M'),
           'transaction_type'])
 .size()
 .unstack(level=1)
 .rename(columns=fix_cols)
 .fillna(0)
 .rolling(12)
 .mean()
 .plot()
)

Now You Have Pandas Resample, Groupby, and Rolling Superpowers

I hope that this blog post exposed you to the powerful time-series functionality in pandas, particularly .resample, .groupby, and .rolling.

Folks often ask me why I use pandas to manipulate my data. Could I do this in SQL? Yes… probably. But I’ve written enough SQL in my life to know that it wouldn’t be fun for me.

I Wrote A Book

2023-03-27T00:00:00-06:00

TLDR - I made over $25,000 in the two-day launch of my new self-published book, Effective XGBoost. I discuss how that happened and if writing a book is worth the time and effort.

My new Effective XGBoost book walks through end-to-end classification from model creation to tuning to interpretation, to deployment.

Why Write a Book?

Every potential author has to answer this for themself. I have written and worked as a technical editor for many books (Illustrated Guide to Python 3, Intermediate Python, Effective Pandas, Machine Learning Pocket Reference, Learning the Pandas Library, Pandas 1.x Cookbook, Tiny Python Notebook (3.5-3.10), Effective Pycharm, Hands-on Machine Learning with Scikit-Learn, Keras & Tensorflow, Hands-on Unsupervised Learning Using Python, and maybe a few others), so I'm aware of what is involved in bringing a book from an idea to something you can hold in your hand. I also know and have interviewed over a dozen technical authors and know many other Python and data science authors. The general advice that you would make more per hour consulting is true.

Some of the books I have written.

Many folks write because they have the bug. Others write for credibility. Others want a book with an animal on the cover. Since transitioning from development into mostly corporate training and consulting, authoring has been great marketing for me.

With self-publishing being relatively easy these days, the sole gatekeeper to writing a book is yourself. Be aware, book creation is a marathon. Don't jump in thinking it is a sprint.

You Made More in Two Days Then I Make in...

Effective XGBoost has been my best launch to date. After two days I had over $25,000 in sales. Perhaps this feels like bragging or an overnight success. But I've been writing books since 2011 and finally feel like I'm starting to crack the nut (or at least listen to common advice like setting up a mailing list).

At this point, I have a few thousand on my mailing list, 140k followers on Twitter, and around 20k connections on LinkedIn. I've fully succumbed to the reality that I must do the marketing myself, and I've worked pretty hard (particularly on the Twitter side) to grow my audience.

Also, note that I've been working on this book since the summer of 2022. Cranking out a quality book is hard work and takes a lot of time. Creating video courses is much easier and less time-consuming for me. Due to the nature of my work, I was often able to spend entire days working on the book. That was not the case when I had fulltime employment and wrote my first books. Many days I woke up early to crank out content. My back-of-the-napkin calculation is that I spent 10 hours a week over 25 weeks writing this book. That comes out to about $100/hour.

If you are now itching to write a book, note that my consulting fees are slightly higher than $100 per hour. Plus, I've authored multiple books, and this book performed much better than previous launches. For comparison, Effective Pandas, only did around $8,000 on its release.

Audience

Since the dawn of self-publishing, authors have touted the importance of a mailing list. I didn't have an email list when I created my first books. Bad choice, Matt!

I used the Gumroad platform initially, and while the features were great for sales, that was about the end of the features. Eventually, I moved on to Podia, which includes mailing list functionality. Honestly, my mailing list-fu is lacking, but I'm continually looking to improve. I might create a Substack, but as a one-man shop, I like to keep things simple and limit platforms. The draw of Podia is that it is all in one.

Launch sales.

I have a few thousand subscribers on my mailing list right now, and they accounted for 35% of my sales.

My next largest source of sales was from Twitter. (If you are wondering how I tracked this, I used Podia coupons with platform-specific codes.) I doubled down on Twitter during Covid and now have 140K followers. They accounted for around 32% of my sales. With all of the hijinks Elon has been pulling recently, having all my "eggs" in the Twitter basket is scary.

LinkedIn came in around 11%, but I only have 20K followers there. I'm actively working on growing that audience. Just another thing to put in the to-do list...

Around 15% of my sales aren't attributable or are from folks who didn't use a coupon. Many of my followers pay full price to support creators like me. I'm grateful to those "super fans". Some of these folks might not have figured out how to enter the coupon code at checkout. Podia can be tricky like that sometimes.

Around 5% of the sales during this time were upsells. So these weren't sales of Effective XGBoost but in-cart discounts to my Effective Pandas material. Podia can be tricky like that sometimes.

You'll note that I haven't mentioned the elephant in the book sales business ... Amazon.

Physical Books

My initial release was only for a digital book with an option for a course and consulting. I highly recommend creating an accompanying course for your book. Around 70% of my sales included the course. Here's the math. 100 books at $50 = $5,000. Contrast that with having a course, offering 70 courses at $100 and 30 books = $7,000 + $1,500 = $8,500.

Oh yeah, Amazon. Amazon accounts for a large number of sales. However, they also take a large cut. And they don't give me access to my readers.

The compromise I have settled on is letting Amazon (and other long-tail providers) handle the physical book. Many prefer to read from a physical book, and Amazon does a great job pushing your book for you, especially if you can get that fly-wheel moving. However, I feed my family with my book sales and would prefer not to have a middleman for my audience's digital sales.

I created both a paperback and hardback book (Amazon now offers these) and announced to my mailing list that these books were available without disclosing them to the rest of the world. I also deeply discounted it (to the point where I'm making $1/sale) as a bonus for my mailing list and a way to say "thanks" to my audience.

Amazon ranking after launch to mailing list.

This (hopefully) kicks in the Amazon machine, as my mailing list pushed the book to the top of the lists and a rank of around 3,000 overall on Amazon.

As of this post, I've bumped the price back up to normal. Amazon can take the cut of sales they bring in, but I will push my digital version and let readers know about the Amazon option if they prefer physical books.

Based on the performance of Effective Pandas (which has done well on Amazon), the Amazon machine can get close to Podia in terms of ongoing revenue.

Note that I'm doing a full-color book on Amazon. These are expensive to print. I could probably double my Amazon income if I keep the same price but sold only a grayscale interior. However, my book has dozens of charts and accompanying code, and the color print on this batch of Effective XGBoost has been excellent. (I'm thinking that Amazon upgraded the POD printers, or I've been really lucky.)

Traditional Publishing?

What about traditional publishing? Are my readers getting an inferior product? My content is not prefect. I make tpyos, like the rest of the world. But editors are also not perfect and also miss typos as well. 😉

I feel like my process is decent, and having been through the process of two other publishers, I feel confident about the final outcome. I can "execute" my books via doctests or notebooks, so I know the code works. Grammarly helps with most typos these days.

I recruit dozens of volunteers from my mailing list review the drafts of the book. This turns out a quality product based on feedback and experience.

What about the other stuff? Layout, cover design, marketing? Yep, that is work. Consider using a publisher if you can't figure out how to do it professionally.

What about distribution? A publisher does provide more distribution than just Amazon. However, Amazon is the elephant here. And I've found that after any initial promotion, the onus is solely on me to promote my book. I use Ingram Spark to reach the long tail, but the squeeze is a lot of effort for the juice when I could be doing other things.

AI and the Death of Books

When I was about 90% done with the book, ChatGPT was released. As a technologist, I jumped on this boat and was really happy using it to augment where possible. As a one-man shop, Chat (as I like to call it) is a game-changer.

I won't go deep into how I used Chat here (perhaps in a different post). I will say, Chat's hallucinations are something to be aware of. Did I use Chat content in the book? Yes, however, I edited everything that Chat created.

Will I use Chat in future content creation? Yes.

Do I think Chat means the end of books? No. I don't feel like Chat can completely automate book creation now. I also think there is some creative direction that Chat can't create on its own.

Does that mean that books won't be written by Chat? No. They already are. This isn't particularly disturbing to me because I've seen folks creating garbage books since Amazon opened the floodgates of self-publishing. You can even pick out these scam books on the charts. Shady folks sell shady stuff to make a buck and then use bots to help game the system. Alas, this is the world we live in. I feel like my audience trusts me to provide a quality product, and the reviews confirm that.

Will Writing A Book Help You?

Thanks for getting this far.

Should you write a book? I get this question a lot.

As a corporate trainer, getting a foot in the door is much easier if you have "written the book on the subject". I know a lot of trainers, and many of them have written books... I wonder why that is? 😉

For more software or data-type folks, should you write a book? Again, if you want to be able to show expertise, this is one (albeit a long one) road to accomplish that task. Having a portfolio project and being able to discuss it in detail would probably accomplish the same for most folks but take much less time.

If you have a large platform or means to promote the book, it can be profitable. Note that this is a long tail game, and the news that you read about is typically the winners with outlier results.

Many just have a "book inside of them" that needs to get out. Just realize that getting it out is a marathon and like most things tech, will probably take two to three times as long as you think it will.

Let me know if you have any questions. I'm at @__mharrison__.

(Chat did not write this post.)

Pandas Series Introduction

2021-06-03T00:00:00-06:00

A Series is used to model one-dimensional data, similar to a list in Python. The Series object also has a few more bits of data, including an index and a name. A common idea through pandas is the notion of an axis. Because a series is one dimensiona-, it has a single axis—the index.

Below is a table of counts of songs artists composed. We will use this to explore how the series type functions:

Artist	Data
0	145
1	142
2	38
3	13

If you wanted to represent this data in pure Python, you could use a data structure similar to the one that follows. It is a dictionary that has a list of the data points stored under the 'data' key. In addition to an entry in the dictionary for the actual data, there is an explicit entry for the corresponding index values for the data (in the 'index' key), as well as an entry for the name of the data (in the 'name' key):

>>> ser = {
...   'index':[0, 1, 2, 3],
...   'data':[145, 142, 38, 13],
...   'name':'songs'
...   }

The get function defined below can pull items out of this data structure based on the index:

>>> def get(ser, idx):
...     value_idx = ser['index'].index(idx)
...     return ser['data'][value_idx]

>>> get(ser, 1)
142

Note

The code samples in this book are shown as if they were typed directly into an interpreter. Lines starting with >>> and ... are interpreter markers for the input prompt and continuation prompt respectively. Lines that are not prefixed by one of those sequences are the output from the interpreter after running the code.

The Python interpreter will print the return value of the last invocation (even if the print statement is missing) automatically. If you desire to use the code samples found in this book, leave the interpreter prompts out.

The index abstraction

This double abstraction of the index seems unnecessary at first glance—a list already has integer indexes. But there is a trick up pandas' sleeves. By allowing non-integer values, the data structure supports other index types such as strings, dates, as well as arbitrarily ordered indices or even duplicate index values.

Below is an example that has string values for the index:

>>> songs = {
...   'index':['Paul', 'John', 'George', 'Ringo'],
...   'data':[145, 142, 38, 13],
...   'name':'counts'
...   }

>>> get(songs, 'John')
142

The index is a core feature of pandas’ data structures given the library’s past in analysis of financial data or time series data. Many of the operations performed on a Series operate directly on the index or by index lookup.

The pandas `Series`

With that background in mind, let’s look at how to create a Series in pandas. It is easy to create a Series object from a list:

>>> import pandas as pd
>>> songs2 = pd.Series([145, 142, 38, 13],
...      name='counts')

>>> songs2
0    145
1    142
2     38
3     13
Name: counts, dtype: int64

When the interpreter prints our series, pandas makes a best effort to format it for the current terminal size. The leftmost column is the index column which contains entries for the index. The generic name for an index is an axis, and the values of the index—0, 1, 2, 3—are called axis labels. The two-dimensional structure in pandas—a DataFrame—has two axes, one for the rows and another for the columns.

The parts of a Series.

The rightmost column in the output contains the values of the series—145, 142, 38, and 13. In this case, they are integers (the console representation says dtype: int64, dtype meaning data type, and int64 meaning 64-bit integer), but in general values of a Series can hold strings, floats, booleans, or arbitrary Python objects. To get the best speed (and to leverage vectorized operations), the values should be of the same type, though this is not required.

It is easy to inspect the index of a series (or data frame), as it is an attribute of the object:

>>> songs2.index
RangeIndex(start=0, stop=4, step=1)

The default values for an index are monotonically increasing integers. songs2 has an integer-based index.

Note

The index can be string based as well, in which case pandas indicates that the datatype for the index is object (not string):

>>> songs3 = pd.Series([145, 142, 38, 13],
...      name='counts',
...      index=['Paul', 'John', 'George', 'Ringo'])

Note that the dtype that we see when we print a Series is the type of the values, not of the index:

>>> songs3
Paul      145
John      142
George     38
Ringo      13
Name: counts, dtype: int64

When we inspect the index attribute, we see that the dtype is object:

>>> songs3.index   # doctest: +NORMALIZE_WHITESPACE
Index(['Paul', 'John', 'George', 'Ringo'],
dtype='object')

The actual data for a series does not have to be numeric or homogeneous. We can insert Python objects into a series:

>>> class Foo:
...     pass

>>> ringo = pd.Series(
...      ['Richard', 'Starkey', 13, Foo()],
...      name='ringo')

>>> ringo  # doctest: +SKIP +NORMALIZE_WHITESPACE
0                             Richard
1                             Starkey
2                                  13
3    <__main__.Foo instance at 0x...>
Name: ringo, dtype: object

In the above case, the dtype-datatype-of the Series is object (meaning a Python object). This can be good or bad.

The object data type is used for strings. But, it is also used for values that have heterogeneous types. If you have just numeric data in a series, you wouldn't want it stored as a Python object, but rather as an int64 or float64, which allow you to do vectorized numeric operations.

If you have time data and it says it has the object type, you probably have strings for the dates. Using strings instead of date types is bad as you don't get the date operations that you would get if the type were datetime64[ns]. Strings, on the other hand, are stored in pandas as object. Don't worry; we will see how to convert types later in the book.

The `NaN` value

A value that may be familiar to NumPy users, but not Python users in general, is NaN. When pandas determines that a series holds numeric values, but it cannot find a number to represent an entry, it will use NaN. This value stands for Not A Number and is usually ignored in arithmetic operations. (Similar to NULL in SQL).

Here is a series that has NaN in it:

>>> import numpy as np
>>> nan_ser = pd.Series([2, np.nan],
...    index=['Ono', 'Clapton'])
>>> nan_ser
Ono        2.0
Clapton    NaN
dtype: float64

Note

One thing to note is that the type of this series is float64, not int64! The type is a float because float64 supports NaN, which int64 does not. When pandas sees numeric data (2) as well as the None, it coerced the 2 to a float value.

Below is an example of how pandas ignores NaN. The .count method, which counts the number of values in a series, disregards NaN. In this case, it indicates that the count of items in the Series is one, one for the value of 2 at index location Ono, ignoring the NaN value at index location Clapton:

>>> nan_ser.count()
1

Note

If you load data from a CSV file, an empty value for an otherwise numeric column will become NaN. Later, methods such as .fillna and .dropna will explain how to deal with NaN.

None, NaN, nan, and null are synonyms in this book when referring to empty or missing data found in a pandas series or data frame.

Optional Integer Support for `NaN`

As of pandas 0.24, there is optional support for the integer type to hold NaN values. The documentation calls this type the nullable integer type. When you create a series, you can pass in dtype='Int64' (note the capitalization):

>>> nan_ser2 = pd.Series([2, None],
...    index=['Ono', 'Clapton'],
...    dtype='Int64')
>>> nan_ser2
Ono           2
Clapton    <NA>
dtype: Int64

Operations on these series still ignore NaN:

>>> nan_ser2.count()
1

Note

You can use the .astype method to convert columns to the nullable integer type. Just use the string 'Int64' as the type:

>>> nan_ser.astype('Int64')
Ono           2
Clapton    <NA>
dtype: Int64

Similar to NumPy

The Series object behaves similarly to a NumPy array. As shown below, both types respond to index operations:

>>> import numpy as np
>>> numpy_ser = np.array([145, 142, 38, 13])
>>> songs3[1]
142
>>> numpy_ser[1]
142

They both have methods in common:

>>> songs3.mean()
84.5
>>> numpy_ser.mean()
84.5

They also both have a notion of a boolean array. A boolean array is a series with the same index as the series you are working with that has boolean values. It can be used as a mask to filter out items. Normal Python lists do not support such fancy index operations:

>>> mask = songs3 > songs3.median()  # boolean array

>>> mask
Paul       True
John       True
George    False
Ringo     False
Name: counts, dtype: bool

Once we have a mask, we can use that as a filter. We just need to pass the mask into an index operation. If the mask has a True value for a given index, the value is kept. Otherwise, the value is dropped. The mask above represents the locations that have a value higher than the median value of the series.

>>> songs3[mask]
Paul    145
John    142
Name: counts, dtype: int64

Filtering a series with a boolean array.

NumPy also has filtering by boolean arrays, but lacks the .median method on an array. Instead, NumPy provides a median function in the NumPy namespace:

>>> numpy_ser[numpy_ser > np.median(numpy_ser)]
array([145, 142])

Note

Both NumPy and pandas have adopted the convention of using import statements in combination with an as statement to rename their imports to two letter acronyms:

>>> import pandas as pd
>>> import numpy as np

Renaming imports provides a slight typing benefit (four fewer characters) while still allowing the user to be explicit with their namespaces.

Be careful, as you may see the following cast about in code samples, blogs, or documentation:

>>> from pandas import *

Though you see star imports frequently used in examples online, I would advise not to use star imports. I never use them in examples or my work. They have the potential to clobber items in your namespace and make tracing the source of a definition more difficult (especially if you have multiple star imports). As the Zen of Python states, “Explicit is better than implicit” [1].

[1]	Type `import this` into an interpreter to see the Zen of Python. Or search for "PEP 20".

Categorical Data

When you load data, you can indicate that the data is categorical. If we know that our data is limited to a few values; we might want to use categorical data. Categorical values have a few benefits:

Use less memory than strings
Can have an ordering
Can perform operations on categories
Enforce membership on values

Categories are not limited to strings; we can also use numbers or datetime values.

To create a category, we pass dtype="category" into the Series constructor. Alternatively, we can call the .as_type("category") method on a series:

>>> s = pd.Series(['m', 'l', 'xs', 's', 'xl'], dtype='category')
>>> s
0     m
1     l
2    xs
3     s
4    xl
dtype: category
Categories (5, object): ['l', 'm', 's', 'xl', 'xs']

If this series represents size, there is a natural ordering as a small is less than a medium. By default, categories don't have an ordering. We can verify this by inspecting the .cat attribute that has various properties:

>>> s.cat.ordered
False

To convert a non-categorical series to an ordered category, we can create a type with the CategoricalDtype constructor and the appropriate parameters. Then we pass this type into the .astype method:

>>> s2 = pd.Series(['m', 'l', 'xs', 's', 'xl'])
>>> size_type = pd.api.types.CategoricalDtype(
...     categories=['s','m','l'], ordered=True)
>>> s3 = s2.astype(size_type)
...
>>> s3
0      m
1      l
2    NaN
3      s
4    NaN
dtype: category
Categories (3, object): ['s' < 'm' < 'l']

In this case, we limited the categories to just 's', 'm', and 'l', but the data had values that were not in those categories. These extra values were replaced with NaN.

If we have ordered categories, we can do comparisons on them:

>>> s3 > 's'
0     True
1     True
2    False
3    False
4    False
dtype: bool

The prior example created a new Series from existing data that was not categorical. We can also add ordering information to categorical data. We just need to make sure that we specify all of the members of the category or pandas will throw a ValueError:

>>> s.cat.reorder_categories(['xs','s','m','l', 'xl'],
...                          ordered=True)
0     m
1     l
2    xs
3     s
4    xl
dtype: category
Categories (5, object): ['xs' < 's' < 'm' < 'l' < 'xl']

Note

String and datetime series have a str and dt attribute that allow us to perform common operations specific to that type. If we convert these types to categorical types, we can still use the str or dt attributes on them:

>>> s3.str.upper()
0      M
1      L
2    NaN
3      S
4    NaN
dtype: object

Summary

The Series object is a one-dimensional data structure. It can hold numerical data, time data, strings, or arbitrary Python objects. If you are dealing with numeric data, using pandas rather than a Python list will give you additional benefits. Pandas is faster, consumes less memory, and comes with built-in methods that are very useful to manipulate the data. Also, the index abstraction allows for accessing values by position or label. A Series can also have empty values and has some similarities to NumPy arrays. It is the basic workhorse of pandas; mastering it will pay dividends.

Pandas Data Structures

2021-06-02T00:00:00-06:00

One of the keys to understanding pandas is to understand the data model. At the core of pandas are two data structures:

Different dimensions of pandas data structures
Data Structure	Dimensionality	Spreadsheet Analog
`Series`	1D	Column
`DataFrame`	2D	Single Sheet

The most widely used data structures are the Series and the DataFrame that deal with array data and tabular data, respectively. An analogy with the spreadsheet world illustrates the basic differences between these types. A DataFrame is similar to a sheet with rows and columns, while a Series is similar to a single column of data.

Figure showing the relation between the main data structures in pandas. Namely, that a data frame can have multiple series.

Diving into these core data structures a little more is useful because a bit of understanding goes a long way towards better use of the library. We will spend a good portion of time discussing the Series and DataFrame. Both the Series and DataFrame share features. For example, they both have an index, which we will need to examine to understand how pandas works.

Also, because the DataFrame can be thought of as a collection of columns that are really Series objects, it is imperative that we have a comprehensive study of the Series first. Additionally (and perhaps odd to some), we will see this when we iterate over rows, and the rows are represented as Series.

Note

Some have compared the data structures to Python lists or dictionaries, and I think this is a stretch that doesn't provide much benefit. Mapping the list and dictionary methods on top of pandas' data structures just leads to confusion.

Summary

The pandas library includes three main data structures and associated functions for manipulating them. This book will focus on the Series and DataFrame. First, we will look at the Series as the DataFrame can be thought of as a collection of Series.

The Future of Corporate Training

2021-04-15T00:00:00-06:00

TLDR - I think in a post-Covid19 society, virtual training will be the default, but live training will return to office campuses and quarterly onsites.

Training Flipped

The last year threw corporate training for a loop. One week I was wrapping up a wonderful Machine Learning course for my biggest client and the next week they had dropped the dozen classes they had planned for the year.

It is never fun to lose your biggest client, but rather than look back, I doubled down and invested in upping my virtual training skills. I read a lot of material, bought equipment, and the feedback has been wonderful:

I have had many programming teachers who were super knowledgeable but didn't know much about pedagogy or vice versa and it was refreshing to be taught by someone who had both sets of skills.

Over all I think this was an excellent course and worthwhile. I like your teaching style and appreciate your willingness to answer questions, even when you had already explained it once before.

I really enjoyed it and learned a lot, thanks!

Default to Work from Home?

The jury is mixed on what the future of work will look like. Tech titans like Netflix and Google are insisted on a return to the office, while Twitter is leading the way pushing for permanent work from home. Between these extremes are the many companies that can operate remotely, but may do so only part time.

I asked my network on Twitter and LinkedIn and got very mixed results. The only take away is that a small percentage of my contacts will be going to the Netflix/Google model. The question is the degree from which they will work from home.

Where does Training Go?

My feeling is that much of training will remain virtual. However, I'm getting feeders from my network that in-person training is also desired. I suspect much of this may migrate to onsites.

I feel that even with my investment into virtual training, in-person is just a bit better. You get to see people (many are Zoom-shy or just Zoom-exhausted), you can help them easier, and they can help each other easier.

If you are looking for virtual or in-person (I'm vaccinated) training for Python or Data Science, reach out. Happy to help!

Best Way to Learn Python in 2021

2021-01-12T00:00:00-07:00

TLDR: There is a great demand for Python skills. There are also lots of ways to learn.

How does the saying go? The best time to learn Python was 20 years ago. The second best time is now.

That's not quite true, Python was a niche language 20 years ago. Today Python is ubiquitous and one of the top languages used to develop applications, analyze data, create predictive models, manage servers, and more.

When I learned Python in 2000, it was frowned upon as a less expressive Perl. It fit my brain, and I never looked back even though I have used Java, C#, Clojure, JavaScript, and other tools since.

A common question I get these days is: what is the best way to learn Python? I'm going to answer that with a staple of data science, a decision tree. The answer really depends, but after teaching Python to thousands over many years, I have a few opinions on the topic.

TLDR: Follow this for Python mastery

(Full disclosure: I make my living selling snake oil. I literally tell and enable people to do amazing things with Python that cure most ails.)

Are You a Corporation?

First things first. If you are a corporation, treat your employees well, and consider corporate training. There is something to be said about getting a team together and having them work through training together. (I said work because training can be intense). Your team will be on the same page at the end of the day. Training will fill in their knowledge gaps. They can address any issues or questions they have immediately with a knowledgeable instructor. Good trainers can facilitate discussions that bring out customers issues and solutions to those quickly.

Corporate Training is the quickest way to get your team on the same page.

Here is a quote from a manager at a large video company:

“Nice breadth, very practical, easy to follow along. Everybody loved it. We already used some of the skills to debug two different production issues. Awesome!”

Seriously, invest in your employees. Here is a common complaint. I talk to many companies that spend money on on-demand courses (full-disclosure: I have created content for many of these courseware providers), and while the content may be good, most employees are not motivated nor incentivized to go through it. The employers are not getting the value out of these courses. This is a typical criticism (in fact, most if not all of my clients have on-demand courses in addition to live training because they want their employees to have certain skills and most employees do not do on-demand.)

One more thing to consider with live corporate training. Not all corporate training is the same. Who would you rather have training you? Someone who is a professional slide reader? Or someone who has worked with the technology for 20+ years and also has had extensive experience tutoring, teaching, and writing?

Going with a quality trainer who has in-the-trenches experience and teaching skills can make a big difference. I generally only train on material that I have used as a consultant or as an employee.

If you are not interested in MetaSnake's offerings, reach out, I know a bunch of other quality trainers that also have more experience than just reciting Powerpoint presentations.

The Best for Individuals

How bad do you want to learn Python? I've come across countless people who claim they want to learn Python, but are not willing to invest anything to get results. The best results come from having a coach to identify weak spots and provide advice and practice to improve those faults.

Coach you probably don't recognize.

You have probably never heard of the coach above. His name is Brandon Payne. Still at a loss? His client is likely the best shooter in basketball history. Steph Curry has won NBA championships, MVPs, and made an amazing 100 three pointers in a row. Why would he need a coach? In Steph's own words:

I appreciate [Brandon's] perspective on things — everything from watching game film and cutting it up to advice and direction on reaching my full potential... And we created a vision for where I wanted to be in terms of ballhandling, footwork, balance and, of course, accountability.”

A coach is a partner who can help with reaching full potential and perhaps more importantly accountability.

Those who are at the top of their game use coaches: CEOs, athletes, entrepreneurs, and programmers. Why? Because they invest in their future. A coach can help identify and cure weaknesses and further strengthen existing skills. I could go on, but the point is that people who are serious about something invest in it, and coaching is probably the best way to improve.

A "free" option for coaching might be a more senior colleague who is willing to let you tag along and pair program. If this includes mentoring along the way, this can be invaluable. Some companies encourage this sort of behavior while others view it as "off task" and a waste of precious developer time.

Workshops & Live Courses

Similar to corporate training, workshops and live courses are the equivalents for an individual. Sometimes smaller companies or startups invest in one-off trainings for individuals when they don't have a whole team to train.

These courses lack many of the bonding and insight moments that come with team training, but if you want to grok something quickly, these are the way to go. Again, an experienced trainer can pinpoint weaknesses or gaps and develop them much quicker than going solo.

These may appear to cost more, but consider they can be like drinking from a firehose. You get comments like this:

I felt like I learned 3 months worth of material in 3 days!

After three days, most students are off and running with their new skills. Thrilled to be using thier new knowledge.

I was once in a training, and after demoing some code, a participant did a face palm. I asked what was the matter and they replied that the one line of code I had just showed them had taken them almost a month to implement. 🤦‍

Projects

Invariably, some are not sold on learning fast and thoroughly. (They have not allocated budget to invest in the future.)

At this point, there is a new requirement for learning Python (or any new skill), sustained motivation. Sitting through a couple of days of training is really easy compared to this, but I digress.

A great way to pick up Python is to choose a project and implement it in Python. In fact, this is how I learned Python. Was my code idiomatic Python when I was done? Nope, I was pretty clueless, but I enjoyed the language features that I did use. Did I have an expert helping me? No, it was the blind leading the blind. Was it quick? Somewhat, but again I wasn't using the tools how I should be. However, it did give me a good chance to play around with the language and compare it to other tools that I was used to.

If you have a hobby project that you are itching to do, or your boss is breathing down your neck about your next task, consider using this as a forcing function to get familiar with Python. There can be some inherent motivation here to learn.

Videos and On-demand Training

For those with sufficient self-motivation, on-demand video courses can be excellent. Again, the quality of the material can vary. You might want to consider if you are acquiring the courses from sources that focus on the topics at hand or if they are trying to be everything to everyone.

Over the years, many have asked me for training and I ask them how big their team is. When the response was "just me", I felt bad, but in the past year I have been developing a library of training for such individuals in addition to my corporate training. (Indeed, this can be complementary to corporate training as review material.)

I think my material is wonderful (but also realize that I'm heavily biased), and am willing to share my thoughts about other platforms and their pros and cons.

Again, the lack of budget might be a concern for some. I offer discounts to individuals not from the world's 30 wealthiest countries (or those laid off due to Covid19 and students). Still, for some, their willingness to invest in learning consists of spending no money but lots of time. (I hope you can see that I think this is the wrong tradeoff, especially for professionals).

In that case, most courses are paid courses. However, there are thousands if not millions of hours of free content on Youtube. These vary in quality of content, audio, and video. However, you can strike gold with these.

Unless you come across some University course (which might lean more theoretical and lacking application or at least experience in the battlefield), you will be picking up nuggets here and there without a coherent learning path. Again, not an effective use of time, in my opinion, but if the only currency you are willing to spend is time, this is an option.

Books

The final category of my learning options is the written word. You should ask yourself (or your company) "what are you willing to invest?". Many companies are willing to reimburse book purchases.

Some of the books that I have written

Why are books a good investment? Because the author was forced to fill out a proposal form that required that they think about taking the end-user, aka the reader, aka you, down a learning path. During the writing process, there are editors and reviewers whose purpose is to ensure that the learning happens.

Having written a few books (and seeing the reader feedback) as well as reading and reviewing many technical books, I'm a huge proponent of books. However, if you just read books, science tells us that you will not retain most of what you read. To retain information, you will need to apply it (or work on getting it into long term storage). This is probably my biggest beef with many books, they do not include exercises, or they are hard to apply (before they leave short-term memory).

Again, there are books of differing quality. I tend to read a lot of Python books, and am more than willing to proffer my opinion.

We are now back at the "lack of budget" folks. One option is the library. However most libraries do not have up-to-date material. Offering to be a technical reviewer might be a way to get your hands on newer material, but generally these are jobs that are requested from you rather than the other way around.

The cheap option, that sadly has suffered in the past few years with the rise of Twitter and blogging platforms like Medium, is the blog. This is the Youtube of written content. There is a lot out there. Some is gold, some is outdated. But, these posts generally lack a learning path and are not great for taking you along on a learning journey.

Get Started Learning

There you have it. My (obviously biased) thoughts around the most effective ways to learn Python after having been involved in almost every aspect of this for the past twenty years. Feel free to reach out on Twitter or LinkedIn to discuss.

Why Write a Book?

2020-11-13T00:00:00-07:00

The New Business Card

I never considered myself an author. But after answering many one-off calls asking for advice on book writing and realizing that I have published over a dozen books on Amazon, I guess I should admit it. I am an author!

Many of the books that I have written over the years.

Authoring has never been easier. Let's chat about some of the reasons for writing a book and see if they might resonate with you.

Note: I'm specifically talking about writing a non-fiction technical book. I've yet to tackle fiction (other than publishing a book of short stories from a book writing class I taught at the local elementary school.)

Lend me your ears and I'll sing you a song

Is there a story inside of you? I just said that I was not referring to fiction, so what does this even mean?

I started my writing journey after preparing and delivering a half-day Python tutorial. I was very excited about the tutorial's content and felt that it was "the tutorial that I wanted when I was learning Python." The tutorial feedback was extremely positive, and the tutorial had been accepted to be offered at another conference.

Rather than just rewriting my slides in preparation for the upcoming conference, I figured why not take this awesome content and turn it into a book? It should not be too hard, plus I had three months before the conference, so this would allow me to double-down and learn the material inside out.

Long story short, it took a bit more than three months to write the book, but this effort became the first version of what is now Illustrated Guide to Python 3. And this kickstarted my path as an author.

This seems to be a common occurrence. A (lead) engineer gets asked what resources they would recommend for ... and they come up empty. There are no good resources (or bikesheds that are their favorite color). So they set out to create their own. I have interviewed many authors who had this experience. They set out to create the resource they desired.

Recognition

After I published my book, I didn't want to make too much noise about it; because technical people realize that the value is in the product not the marketing or sales (how wrong I was).

Then I had the coolest experience, I was walking down the hallway at work, and a colleague approached me and said, "Hey, aren't you the author of Treading On Python?". I was dumbfounded. Without any mention on my part, the Amazon sales engine sold a book to someone who worked a few cubicles away from me. That felt awesome. But even better was that they had high praise for the book. Recognition is something that everyone craves.

Many potential authors want this recognition. Perhaps it can be better described as the author wants a "book with an animal on the cover" (along with their name).

The coveted animal cover.

I'm guilty of that. I have a framed copy of my book with a northern crested newt on the cover. (I wish it were a short-horned lizard, but I digress). This work has since been translated into seven languages and distributed around the world.

That experience of serendipitous recognition has been repeated over the years. I don't have a paparazzi following me, but...

Hustle

Recognition opens doors.

Many authors (or would-be authors) are interested in "the hustle." This could be looking for side gigs, augmenting income, looking for a better job, or starting consulting.

Book writing is an excellent business card.

One of my author friends, Mark Beckner, put it this way in his book The Coder's Path to Wealth and Independence:

If you publish intelligently, your business will be transformed. If you write with the intention that your publication will create unexpected opportunities, it will enable you to move to a new level in your business.

I have found this to be true. And I know and have spoken with many other authors who had this experience.

It is well known that the process of writing helps you to understand a subject better. You have literally written the book on the subject. With self-publishing (or even purchasing bulk copies from your publisher), you can use your book as a cost-effective business card booster. (In the current Covid-19 climate, this is a little harder. I have a bookshelf of books I meant to deliver to (potential) clients this year at conferences.)

This opens the doors to opportunities that would not come your way even if you were an expert in said technology.

Part of the hustle is money. While I know many authors who have sold six figures worth of books, these tend to be outliers. However, many authors I know have indicated that though they earned minimum wage from the writing, they have indirectly profited much more after the book from side gigs, job opportunities, and more.

Taking the Plunge

Having said this, writing a book is a marathon. Many have sprinted out of the gate only to fail. (I've been approached by a few publishers to "become a second author" on an in-progress book, which is code for "the author burned out").

Have you considered writing a book? I've created a course that walks from ideation to holding a book in your hand. It discusses the tradeoffs of self-publishing and going with a publisher, how to outline, what tools to use, how to get distribution, ebook and physical book tips, audiobook creation, and more. I've come full-circle. This is the course I wanted 10 years ago when I set out to write my first book.

Learn to use the assign method in Pandas (or how to deal with SettingWithCopyWarning issues)

2020-04-03T00:00:00-06:00

TLDR: Favor the .assign method, and you will not deal with SettingWithCopyWarning issues.

Master the .assign method and not index assignment!

Bad Code

Today I'm going to show you the solution to the dreaded SettingWithCopyWarning error that you eventually run into, but no one seems to explain it or give you a solution that you can understand.

I just saw another article discussing it today and thought I would share the best solution (that no one else talks about).

Here's the data referred to in the above article:

df = pd.DataFrame({
  'x':[1,5,4,3,4,5],
  'y':[.1,.5,.4,.3,.4,.5],
  'w':[11,15,14,13,14,15]})

The quest with this toy data is to change all values in the y column to 50 when the x column is greater than 3. Most try to do this by using indexing operations to filter out the appropriate rows and then doing an index assignment.

If you do an index operation directly on df, the update does not work. And you see the dreaded SettingWithCopyWarning warning:

df[df['x']>3]['y'] = 50

The dreaded warning!

The issue with this code is that df[df['x']>3] does not return the original dataframe. So the subsequent index assignment, ['y'] = 50, updates the new dataframe and not df. In effect, you create a new dataframe, assign to it, and do not capture it to a variable, so it is immediately available to be garbage collected.

The warning includes a tip to use .loc for indexing instead. Here is an example that works:

df.loc[df['x']>3,'y'] = 50

I recommend that you do not use this solution. I have a better solution below.

Three Rules for Better Pandas

However, I'll add three more rules of thumb that I have found to make pandas code easier to understand and less prone to errors:

Do not mutate objects
Favor chaining operations
Do not use assignment (the = operator) in pandas

The above code mutates df. However, most pandas operations do not mutate. For feeble-minded people like me, mixing mutation and immutability leads to confusing code and errors. My preferred solution to the above is this code:

df.assign(y=df.y.where(~(df.x > 3), 50))

In fact, whenever I create a new column (or update one) in pandas, I try to do it with the .assign method. This code also requires understanding the .where, method which takes some getting used to but obeys my three rules above.

A Chained Example

Here is an example from Chapter 5 of the The Pandas 1.x Cookbook. This code loads vehicle data from the fueleconomy.gov website. It applies a chain of operations to the dataframe when creating the data variable.

First, it filters out the makes that are Ford, Tesla, BMW, or Toyota with this line [fueleco.make.isin(makes)]. (It could have used .loc here as well). Next, it assigns a column, SClass, that is created from an inlined function (a lambda function) with this line .assign(SClass=.... If you use a function with the .assign method, you have access to the intermediate dataframe, referred to as df_ in the code. The lambda function dispatches to the generalize function.

The generalize function is a function I use all over the place when refining data. It takes a column and a list of tuples with match names and new values in it. It also accepts a default value. It loop over the list returns a new series where the match names are replaced with the new names. If there is no match, the default value is used. This function can be useful if you have a bunch of categorical data and want to limit it to a smaller subset:

url = 'https://github.com/mattharrison/datasets/raw/master/data/vehicles.csv.zip'
fueleco = pd.read_csv(url)

def generalize(ser, match_name, default):
    seen = None
    for match, name in match_name:
        mask = ser.str.contains(match)
        if seen is None:
            seen = mask
        else:
            seen |= mask
        ser = ser.where(~mask, name)
    ser = ser.where(seen, default)
    return ser

makes = ['Ford', 'Tesla', 'BMW', 'Toyota']
data = (fueleco
   [fueleco.make.isin(makes)]
   .assign(SClass=lambda df_: generalize(df_.VClass,
    [('Seaters', 'Car'), ('Car', 'Car'), ('Utility', 'SUV'),
     ('Truck', 'Truck'), ('Van', 'Van'), ('van', 'Van'),
     ('Wagon', 'Wagon')], 'other'))
)

data[['make', 'SClass', 'VClass']]

Master the .assign method and gain superpowers!

Conclusion

Try not to mutate data when using pandas. Even the pandas core developers say that you should not use mutation. If you use method chaining (note that index assignment does not work with chaining!), you will not mutate data. You will not have to deal with the SettingWithCopyWarning. Your code will be easier to read because it will read like processing steps.

If you want to see examples of pandas code that obey these rules see my latest book, Effective Pandas or follow me on twitter where I rant on such topics.

Setting up a Python Data Science Environment

2020-03-06T00:00:00-07:00

You can set up your environment using either method with three commands

Here are instructions for getting a Python stack installed on Windows, Mac, and Linux. These examples will use Python 3 (which I recommend as Python 2 is no longer being developed).

You have two main options for installing a Python environment. You can use the Python executable that you can get from the Python website, or you can use the Anaconda distribution. I will show you both. If you are new to Python, only doing "data science" work, or using Windows, I recommend using Anaconda.

Using Anaconda

TLDR: Use these to create an environment using Anaconda

Anaconda is both a product and a company. The company supports Python data tooling and makes the distribution available as a free product, but also as a supported enterprise paid for product. This page will demo the free option, but I'm assuming that the paid product is similar after install. There are versions for both Python 3.x and Python 2.7. Make sure you use the Python 3.x version.

The Anaconda distribution includes "Python" it just has some extras. It uses its own package system to make it simple to create working Python environments on most platforms. The steps you need to take if you want to use it are:

Install Anaconda (or Miniconda)
Create an environment
Install packages into the environment

Install Anaconda

You have two options for installing Anaconda (the product). The first is to install the whole kitchen sink, which is referred to as the "Anaconda Distribution" a 600MB+ download that includes many common libraries for machine learning, data analysis, and visualization. You can download it here:

The other option is to use Miniconda. This is a smaller download that does not include various libraries but rather lets you download and install them on demand. You can download it here:

(Many of my clients work inside a firewall and should use the Anaconda distribution that is available locally inside of their firewall.)

After you have installed Anaconda or Miniconda, you should have an executable called conda available. Test out that you have it. On UNIX platforms (Mac and Linux) open a terminal (if you are on Mac, you can hit Command-space and then type "Terminal" and hit enter). On Windows, go to the start menu and search for "Conda Prompt". Type into the terminal or command prompt:

conda --version

And it should print out the version, something like conda 4.7.5. If it does not do that, then you did not install Anaconda successfully.

Create an Environment

After you have installed the conda utility, you will need to make an "environment." An environment is a sandboxed installation of Python libraries and utilities. The purpose of an environment is to allow per project library dependencies.

For example, I consult with different clients. If I have multiple clients over the years, they might use different versions of libraries. Say, for example, client A is using Pandas 0.19, client B Pandas 0.25, and client C Pandas 1.0.1. Pandas in generally is pretty good about backward compatibility, but it does have breaking changes. Environments allow me to install specific packages for each client project. When I need to work on client A's work, I activate the environment for that project and have access to Pandas 0.19. When I need to shift to client C, I activate the corresponding environment and have access to Pandas 1.0.1.

To create an environment, you need to give it a name. For a machine learning class, you might create an environment named mlclass (you can specify another name if you like). You can also specify a Python version for the environment. To create an environment with Python 3.7 from a terminal or Anaconda Prompt (on Windows) type:

conda create --name mlclass python=3.7

You can create as many environments as you like. I create them on a per-project basis. That way, I can jump into a project and make sure that I can work on it without dependency conflicts.

After you create the environment, you should activate it. This updates the PATH environment variable so that when you invoke python or utilities, you use the executables found in the environment. Type:

conda activate mlclass

After running this, your prompt should indicate that you are in the environment, and it will change to look like this:

(mlclass)

Note that Anaconda, by default, stores its environments under your home directory. On my Mac, it would put it in ~/anaconda3/envs/mlclass.

Install Packages with Conda

At this point, you have installed conda, created an environment, and activated the environment. You are now ready to install some packages. Here is a command to install some of the common data analysis packages:

conda install notebook pandas scikit-learn seaborn xlrd districtdatalabs::yellowbrick py-xgboost notebook conda-forge::shap bokeh graphviz

The packages with colons in their names are specified as channel::package name. Anaconda ships with certain packages, but libraries can create their own "channel" to ship different versions of their library (or even package their library for Anaconda) on a different cadence than the Anaconda release cycle.

At this point, you should be ready to go with a basic installation. To launch Jupyter notebook type:

jupyter notebook

To learn more about the conda tool, check out the User Guide.

Installation with Python

TLDR: Use these commands on Linux or Mac if you do not want to use Anaconda. On Windows replace the second line with env\Scripts\activate

Note: If you are using Windows, it is recommended that you use Anaconda rather than this mechanism. Some of the scientific packages require installing and configuring a cross-compilation development environment, which can be annoying. If you are using a Mac and want to follow these instructions, it is recommended to install Python from Homebrew.

Install Python

If you are on Linux, use your package manager to install Python 3.x. On Mac, use Homebrew to install Python 3.x. If these instructions do not make sense to you, scroll up and use the instructions for Anaconda.

After Python is installed, open a terminal and check the version. Type:

python3 --version

It should say something like python 3.7.3. Make sure that you installed Python 3 and not Python 2.

Create a Virtual Environment

Python 3 ships with the ability to create environments that are very similar to Anaconda environments. However, they are not the same. Python calls its environments "Virtual Environments", so I will follow suit here. A virtual environment will allow you to create a sandbox for Python on a per-project basis.

To create a virtual environment named env in the current directory type:

python3 -m venv env

To activate this environment, type:

source env/bin/activate

Note that source is a UNIX Bash command (so this will not work on Windows. If you really want to use Windows, even though I suggested against it with this mechanism, you would type env\Scripts\activate.bat).

At this point, your prompt should change to something like this:

(env)

Install Packages with Pip

With Python installed and your virtual environment activated, you are ready to install packages. We use the pip tool to install packages from the Python Package Index. If your virtual environment is activated, then pip will install libraries into the virtual environment.

You should probably update pip with a new version. Type:

pip install -U pip

To install a base data science system type:

pip install notebook seaborn pandas scikit-learn xlrd yellowbrick xgboost shap bokeh graphviz

To launch Jupyter Notebook type the following:

jupyter notebook

There is a User Guide for pip as well.

Other Options

Can you use Docker? Probably. The above are the simplest ways of getting your environment set up. There may be an existing Docker container you can leverage. But Docker is usually orthogonal to what I'm teaching. Also, the Docker images will probably be using one of the two techniques I described above.

What about hosted services? Google COLAB is a great service (disclaimer: I'm a subscriber to their PRO account), but requires access to Google COLAB, which many of my clients do not have.

Summary

Setting up a Python development environment is three lines of code once you have Python 3 or Anaconda installed. If you have questions about this, feel free to tweet them. I try to keep this page up to date as I point my clients to it.

What I Use 2019 Edition

2019-11-20T00:00:00-07:00

It seems to be popular to share what gear you use. I'm pretty picky about some things, so I thought I would take a stab at it. Note that I'm just going to address physical hardware in this post, I'll save software for another post.

2015 Macbook Pro

I used Linux exclusively for over 15 years on ThinkPads, but due to the nature of my training and consulting got a Macbook in 2015. It is a decent piece of hardware. The screen went out on me a few years back, and now it seems the keyboard and trackpad are flaky sometimes. A few reboots seem to do the trick. Unless I'm training, I use an external keyboard and mouse, which oddly enough work even when the built-in devices do not. Feb 2020 Update - I have replaced the cable for the trackpad which seems to have solved the problem. Annoying, but easy if you have the correct tools and spend five minutes watching Youtube.

At the time it came out, the screen DPI was a wonder. Now it is a little more commonplace. I do most of my work on this machine. Again I will discuss software later. As of 2019, the 16inch version of this machine has reverted to using the same keyboard as I have. The butterfly keyboards of 2017-2019 were notoriously flaky, and I stayed away from those machines.

The battery life is still decent on this. I take it with me when traveling, and it performs well. I like the aluminum body, which is unparalleled among laptop manufacturers. I wish I could upgrade the SSD and RAM like I could with my ThinkPads. Se la vie.

My next laptop will probably be an Apple due to video editing software.

2017 Surface Book Pro

At some point, I realized that the vast majority of the world (and developers in said world) use Windows. I always missed my UNIX environment when forced to use Windows at work (in the NT days). Windows 10 is useable, and this device also travels with me (as both a backup and if I need to demo something on Windows).

The device comes in handy as a tablet as well. I've used the stylus and found it to be nice when editing books. I was a technical reviewer for Hands-On Unsupervised Learning Using Python and I liked being able to write on top of PDF's with the stylus.

Another area where the Mac excels is the trackpad. It is far superior to the MS version. However, I much prefer the ThinkPad's TrackPoint to either. I know ... gasp, the terror!

Standing Desk

I started standing at work around 2011. My job at the time was pre-standing desk movement. So I needed to use an adapter on top of my sitting desk. I used an Ergo Desktop Kangaroo Pro This worked ok. It was maybe not quite as stable but was able to transform from sitting to a standing position without falling over.

I mostly work from my home office these days and use a Tresanti Desk. I purchased mine at Costco, and I have few regrets. I'm 6'4", and the top hieght works for me. It has a glass top which my kids have used to write notes to me with whiteboard markers.

There are 3 USB outlets at the side, and no this is not a MIDI device, they are only used for charging. On that note, I have an IKEA Signum "Cable management" drilled into the bottom of the backside of the table. In the Signum is an Anker Dual Display Universal Docking Station. As you can tell from the photo, I'm not particularly concerned with eliminating wires. But it would be nice to have USB hub functionality in the desk.

Because it has a glass top mounting stands to it can prove challenging. My monitor stand works with it, but only because I could disassemble the clamp for it.

My other issue with this desk is the size. When I'm running a remote training and using the Mac and Windows, it can get a little crowded. I also use physical notes, so almost every bit of space is used.

I'm a huge fan of taking breaks during the day. Not only do they help your body; they help you with productivity. I recommend getting a standing desk. Do you have to stand eight hours a day? No. I used to stand most of the day. These days I sit most of the day because that feels better for my body. But I will still mix it up. I think the key is to be aware of your body and posture and take care of it.

Another thing I recommend if you are going the standing route is to pick up a padded mat to stand on.

ErgoDox Keyboard

I picked up an ErgoDox keyboard when Drop (the company formerly known as MassDrop) offered them initially. I was using a Unicomp (with a TrackPoint!), but my fingers were telling me that I needed to do something else. Come to think of it, that time probably correlated to when I was writing the first edition of what is now Illustrated Guide to Python 3.

I knew I wanted a keyboard that was ortholinear, mechanical, and split. Bonus points in that the firmware is opensource and you can program it how you want. This keyboard is an acquired taste, but I liked mine so much that when Drop offered them again, I picked up a second (partially out of fear that I would never be able to purchase one again, and partially because I wanted another for home use). Since that time, it appears that the DIY mechanical keyboard group has flourished. You can now purchase completely assembled models, and there are dozens of riffs on split keyboards.

I do not use QWERTY on this keyboard, but rather the Norman layout. Again, this is another acquired taste, but it has worked out for me. Also, I have keys that have a dual purpose. For example, I use emacs a lot. So my left thumb has Control under it. But if I tap it and release, it fires the Escape key.

I typically do not take this keyboard with me when traveling. I took it to PyCon once when it was novel, but it is not really portable (I have a travel case for it) in that it is bulky, and TSA gives you weird looks when you travel with it.

Again, you need to take care of yourself. Your fingers, wrists, and arms can suffer damage from too much use or non-ergonomic use. Take note and do what feels best.

Clearly Superior Technologies Trackball

Award for best-named company of 1995 goes to CST.

When I got my ErgoDox, I was considering modding it to add a TrackPoint. In the meantime, my previous blog had a couple of entries on the ErgoDox, and they were popular enough such that I sent Drop enough purchases of ErgoDoxen that they sent me a "Thank You" gift. The CST trackball mouse was in that gift box. It seemed to be the answer I was looking for. And it fits nicely in between the split keyboard.

I have also added additional "buttons" that plug into the mouse. I have them situated so that I can tap them with my palm. The left palm is a command modifier, and the right palm is the spacemacs leader key.

Apparently, the ball is the size of a billiard ball. I'm not a big pool player, but that seems like a fancy mod.

I also have two of these trackballs (one for "work" and one for home). As of 2019, CST seems to have retired and sold the business. Whether these high-quality devices will remain in production by the new owner remains to be seen. They are no longer available on Amazon. eBay lists one at over $400!

Again, listen to your body. If it tells you something when you are mousing about, maybe you should look into a change. A trackball certainly uses different body movements.

External Monitor

I bought an Auria 27 inch external monitor about ten years ago. At the time, it was a high DPI monitor that got good reviews for the price point. I will probably replace it with a 34 inch 4k monitor soon, but it has served me well.

Vivo Laptop Monitor Mount

I purchased the Vivo Laptop Monitor Mount in an attempt to get a little more real estate on top of my desk. This is a sturdy product that I enjoy, not in spark joy type of enjoy, but it does the job. It also makes standing more ergonomic as it enables me to raise my screen higher.

Audio-Technica BPHS1 Headset and Mic

I do up to a dozen virtual trainings per month, and this mic has served me well. The Audio-Technica BPHS1 has a long cable which allows me to walk around when teaching and not have to worry about standing in front of a mic. I also have a Blue Yeti, but this mic is more of a set it and forget it type. I feel like I'm always fussing with the levels on the Blue Yeti.

This is an XLR mic, so you will need a mic preamp like the Behringer Xenyx 302USB to use it with a USB interface.

I have a long power outlet attached to the Signum, and I hang the mic off of that when it is not in use.

MetaSnake - misc

Python for Finance: Pandas Resample, Groupby, and Rolling

Data Overview: FDIC BankFind Suite

Data Dictionary

Counts of failures by year

Using Offset Aliases with Pandas Resample

Smoothing By Rougher Aggregations Using Pandas Resample

Resolution Type Over Time

Now You Have Pandas Resample, Groupby, and Rolling Superpowers

I Wrote A Book

Why Write a Book?

You Made More in Two Days Then I Make in...

Audience

Physical Books

Traditional Publishing?

AI and the Death of Books

Will Writing A Book Help You?

Pandas Series Introduction

The index abstraction

The pandas Series

The NaN value

Optional Integer Support for NaN

Similar to NumPy

Categorical Data

Summary

Pandas Data Structures

Summary

The Future of Corporate Training

Training Flipped

Default to Work from Home?

Where does Training Go?

Best Way to Learn Python in 2021

Are You a Corporation?

The Best for Individuals

Workshops & Live Courses

Projects

Videos and On-demand Training

Books

Get Started Learning

Why Write a Book?

The New Business Card

Lend me your ears and I'll sing you a song

Recognition

Hustle

Taking the Plunge

Learn to use the assign method in Pandas (or how to deal with SettingWithCopyWarning issues)

Bad Code

Three Rules for Better Pandas

A Chained Example

Conclusion

Setting up a Python Data Science Environment

Using Anaconda

Install Anaconda

Create an Environment

Install Packages with Conda

Installation with Python

Install Python

Create a Virtual Environment

Install Packages with Pip

Other Options

Summary

What I Use 2019 Edition

2015 Macbook Pro

2017 Surface Book Pro

Standing Desk

ErgoDox Keyboard

Clearly Superior Technologies Trackball

External Monitor

Vivo Laptop Monitor Mount

Audio-Technica BPHS1 Headset and Mic

The pandas `Series`

The `NaN` value

Optional Integer Support for `NaN`