Complete Guide to Designing with Pandas in Data Science

1. Creating and Manipulating DataFrames

The DataFrame is the fundamental data structure in pandas . It is a two-dimensional table composed of rows and columns, similar to a spreadsheet or a database table. To create a DataFrame, you can start from a dictionary of lists, a NumPy array, or even read it directly from a CSV file.

 import pandas as pd 
# Creating a DataFrame from a dictionarydata = {'Name': ['Ana', 'Luis', 'Carlos'], 'Age': [23, 35, 45]}df = pd.DataFrame(data)print(df)

In this example, we have created a simple DataFrame with two columns: "Name" and "Age". This is just the beginning of what you can do with pandas.

Once you've created a DataFrame, pandas allows you to access and modify the data in a variety of ways. Indexing and data selection are crucial processes in any data analysis, and pandas offers multiple methods to do so.

• Selecting by Label : Using the loc method, you can select data by its row or column label.
• Selection by position : With iloc , you can access data using its numerical position.

# Selecting a columnprint(df['Name'])
# Selecting a rowprint(df.loc[1])
# Selecting by position print(df.iloc[0])

Pandas makes it easy to perform complex operations on your data, such as aggregating, filtering, grouping, and applying specific functions to entire columns. These operations are essential to effectively transform and analyze data.

• Add new columns: You can add new columns to your DataFrame based on calculations from other columns.

df['Double_Age'] = df['Age'] * 2print(df)

• Filtering data: Filtering your data to see only rows that meet certain conditions is one of the most common operations.

df_filter = df[df['Age'] > 30]print(df_filter)

• Grouping and summarizing data: The groupby function allows you to group data by one or more columns and then apply aggregation functions.

df_grouped = df.groupby('Name').mean()print(df_grouped)

1. Handling Missing Data

In any dataset, it is common to encounter missing values. Pandas offers powerful methods to deal with them, such as fillna to fill missing values ​​or dropna to remove rows or columns with missing values.

df_con_faltantes = df.copy()df_con_faltantes.loc[1, 'Age'] = Nonedf_rellenado = df_con_faltantes.fillna(0)print(df_rellenado)

In practice, it is often necessary to combine multiple datasets. Pandas provides several functions for joining DataFrames, such as merge and concat , which allow performing SQL-like joins.

data2 = {'Name': ['Ana', 'Carlos'], 'Salary': [5000, 6000]}df2 = pd.DataFrame(data2)
# Merging the DataFramesdf_unido = pd.merge(df, df2, on='Name', how='inner')print(df_unido)

Pivot tables and data reordering are useful techniques for transforming your data and making it easier to analyze. Pandas allows you to pivot data with pivot_table and reorder it with melt .

df_pivot = df.pivot_table(values='Age', index='Name', columns='Double_Age')print(df_pivot)
df_reshaped = pd.melt(df, id_vars=['Name'], value_vars=['Age', 'Double_Age'])print(df_reshaped)

As you work with larger datasets, the efficiency of your operations becomes critical. Pandas provides tools to optimize performance, such as vectorizing operations and using apply for custom operations.

# Vectorization example df['Age_triple'] = df['Age'] * 3
# Using apply for a custom function df['Category'] = df['Age'].apply(lambda x: 'Adult' if x >= 18 else 'Minor') print(df)

Designing and manipulating data with pandas are essential skills for any data science professional. From creating DataFrames to optimizing complex operations, pandas offers a wide range of tools that can help you get the most out of your data. With the techniques we’ve explored in this article, you’ll be well equipped to tackle any data challenge that comes your way.

If you want to deepen your knowledge and skills in pandas and Data Science, we invite you to explore our courses on G-Talent , where you will learn from experts in the field to master this powerful tool and become a highly skilled Data Science professional. Enroll today and take the next step in your career!