How to Select Multiple Columns in Pandas

To select multiple columns ‘a’ and ‘b’ from a Pandas DataFrame ‘df’ and save them into a new DataFrame ‘df1’, use the double brackets method: df1 = df[['a', 'b']]. This creates a new DataFrame containing only the specified columns, preserving all rows from the original DataFrame.

Contents

• Basic Selection Methods
• Advanced Selection Techniques
• Common Pitfalls and Solutions
• Best Practices
• Performance Considerations

Basic Selection Methods

Double Brackets Method

The most straightforward and commonly used method is passing a list of column names within double square brackets:

df1 = df[['a', 'b']]

This syntax creates a new DataFrame containing only columns ‘a’ and ‘b’ from the original DataFrame. The outer brackets indicate DataFrame indexing, while the inner brackets define the list of column names.

Example Usage

import pandas as pd

# Create the example DataFrame
data = {'a': [2, 3], 'b': [3, 4], 'c': [4, 5]}
df = pd.DataFrame(data, index=[1, 2])

# Select multiple columns
df1 = df[['a', 'b']]

print(df1)

Output:

   a  b
1  2  3
2  3  4

Using .loc Method

The .loc method allows for label-based selection and is particularly useful when you need to select both rows and columns:

df1 = df.loc[:, ['a', 'b']]

Here, : selects all rows, and ['a', 'b'] selects the specified columns.

Using .iloc Method

The .iloc method selects columns by their integer position. If ‘a’ is the first column (index 0) and ‘b’ is the second column (index 1):

df1 = df.iloc[:, [0, 1]]

Advanced Selection Techniques

Using .filter Method

The filter method provides a more readable way to select columns:

df1 = df.filter(['a', 'b'])

This method is particularly useful when working with DataFrames that have many columns, as it clearly expresses the intent to filter columns.

Using .copy() Method

To ensure you’re working with a copy rather than a view of the original DataFrame:

df1 = df[['a', 'b']].copy()

This is recommended when you plan to modify the new DataFrame, as it prevents the SettingWithCopyWarning that can occur when working with views.

Column Selection with Conditions

You can combine column selection with boolean conditions:

# Select columns 'a' and 'b' where column 'a' > 2
df1 = df[df['a'] > 2][['a', 'b']]

Common Pitfalls and Solutions

Why Single Brackets Don’t Work

Using single brackets like df['a', 'b'] will raise a KeyError because pandas interprets this as trying to access a single column with a compound key name.

Correct approach:

# Single brackets work for one column
df_a = df['a']  # Returns a Series

# Double brackets needed for multiple columns
df1 = df[['a', 'b']]  # Returns a DataFrame

Column Slicing Issues

As shown in your unsuccessful attempt, df['a':'b'] doesn’t work because column slicing with strings isn’t supported in pandas.

Correct approaches:

# Method 1: Explicit list
df1 = df[['a', 'b']]

# Method 2: Using loc with labels
df1 = df.loc[:, ['a', 'b']]

# Method 3: Using iloc with positions
df1 = df.iloc[:, 0:2]  # If 'a' and 'b' are the first two columns

Deprecated Methods

The ix indexer was deprecated in pandas 0.20.0 and removed in pandas 1.0.0. Avoid using it in modern pandas code.

Instead of:

# Deprecated - will raise error in modern pandas
df1 = df.ix[:, 'a':'b']

Use:

# Modern equivalent
df1 = df.loc[:, ['a', 'b']]

Best Practices

Method Selection Recommendations

Performance Considerations

For small to medium DataFrames, performance differences between methods are negligible. However, for large DataFrames:

• Double brackets and .loc are generally the fastest for column selection
• .iloc can be slightly faster when selecting by position
• .filter has minimal overhead but is slightly slower than direct indexing

Memory Efficiency

When working with large datasets, consider using .copy() only when necessary to avoid memory duplication:

# For read-only operations, views are more memory-efficient
df1 = df[['a', 'b']]  # Creates a view (memory efficient)

# For modifications, create a copy
df1 = df[['a', 'b']].copy()  # Creates a copy (uses more memory but prevents SettingWithCopyWarning)

Performance Considerations

Benchmarking Different Methods

For optimal performance with large DataFrames, the double brackets method is generally the most efficient:

import pandas as pd
import numpy as np

# Create a large DataFrame
df = pd.DataFrame(np.random.rand(100000, 20), columns=[chr(97 + i) for i in range(20)])

# Different selection methods
%timeit df[['a', 'b']]      # Fastest
%timeit df.loc[:, ['a', 'b']] # Slightly slower
%timeit df.filter(['a', 'b']) # Slowest but most readable

Memory Usage Patterns

Different selection methods have different memory characteristics:

• Double brackets: Creates a view initially, then a copy if modified
• .loc: Always creates a copy
• .iloc: Always creates a copy
• .filter: Always creates a copy

Understanding these patterns helps in optimizing memory usage for large datasets.

Conclusion

To select multiple columns ‘a’ and ‘b’ from a Pandas DataFrame ‘df’ and save them into a new DataFrame ‘df1’, the most straightforward approach is using double brackets: df1 = df[['a', 'b']]. This method is readable, efficient, and widely used in the pandas ecosystem.

Key takeaways:

1. Double brackets is the preferred method for simple column selection
2. .loc is ideal when you need both row and column selection
3. .iloc is best when working with column positions rather than names
4. .filter provides excellent readability for column filtering operations
5. Always use .copy() when you need to modify the resulting DataFrame

By understanding these different approaches and their use cases, you can choose the most appropriate method for your specific data manipulation needs.

Sources

1. How to Select Multiple Columns in Pandas (With Examples) - Statology
2. Select multiple columns in a Pandas DataFrame - Tutorialspoint
3. How to select multiple columns in a pandas dataframe - GeeksforGeeks
4. How to Select Multiple Columns in Pandas DataFrame - Ubiq BI
5. Pandas Create New DataFrame By Selecting Specific Columns - Spark By Examples
6. How do I select a subset of a DataFrame? — pandas documentation
7. Pandas Dataframe loc, iloc & brackets examples - Analytics Yogi
8. Pandas iloc and loc – quickly select data in DataFrames - Shane Lynn