Pandas. Views versus copy

In pandas, it can be tricky setting the values of a dataframe. One can hit the ‘SettingWithCopy’ warning message.

The crux of the problem is this:

  • “The reason for having the SettingWithCopy warning is this. Sometimes when you slice an array you will simply get a view back, which means you can set it no problem. However, even a single dtyped array can generate a copy if it is sliced in a particular way. A multi-dtyped DataFrame (meaning it has say float and object data), will almost always yield a copy. Whether a view is created is dependent on the memory layout of the array”

–> Note, Pandas throws this warning because of the ambiguity. It doesnt know if the operation was successful or not etc. In some cases it actually fails, but in other cases it works fine.  So Pandas just warns user that its operating on a copy.

  • Sometimes a SettingWithCopy warning will arise at times when there’s no obvious chained indexing going on. Theseare the bugs that SettingWithCopy is designed to catch! Pandas is probably trying to warn you that you’ve done this:
    def do_something(df):
       foo = df[['bar', 'baz']]  
       # Is foo a view? A copy? Nobody knows!
       # ... many lines here ...
       foo['quux'] = value
       # We don't know whether this will modify df or not!
       return foo
    

    Yikes!

  • In particular, the warning can be a misnomer if there is no chained indexing going on (as described above)

__main__:3: SettingWithCopyWarning:
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

  • In case I know I  want to work with a copy,  I can create a copy explicitly (using .copy(). see below). This suppresses the warning.

 

Code:

 

Reference:

 

 

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s