In this tutorial, we examine how to create lagged and lead variables: essential tools for time series and panel data analysis. Whether you’re modeling financial trends or running regressions, such transformations are often essential to compute new variables, such as Return on Assets or Revenue Growth. We’ll also explore how the Pychemist library simplifies this task.

We use pandas and the pychemist library to demonstrate this. For this tutorial we rely on a dataset of financial information for 20 large US companies, scraped from EDGAR. Some missing values in the dataset have been imputed for demonstration purposes.

While pandas lets you create lagged values using .shift(), it doesn’t always behave as expected for grouped data or irregular time series. Pychemist provides a simpler and more reliable alternative through its .chemaccessor.

Step 1: Load the Data and Libraries

We start by importing the necessary libraries and loading the dataset. pandas and numpy are used for working with panel datasets. Additionally, we import pychemist to download the financial dataset and to enable the custom pandas extensions (accessors) used in this example.

Note: If you haven’t installed pychemist yet, you can do so with the following command:

pip install pychemist

Import the required libraries and load the dataset into a DataFrame (df):

import pandas as pd
import numpy as np
import pychemist

df = pychemist.load('financials')

Let’s inspect the first few rows:

df.head()

5 rows × 6 columns

Step 2: Create a Lag Manually

First, we sort the dataset by company (ticker) and time (year). Second, we create a lagged variable total_assets_lag using Pandas’ built-in .shift() method.

df = df.sort_values(['ticker', 'year'])
df['total_assets_lag'] = df['total_assets'].shift()

Next, we inspect the DataFrame:

df.head(10)

10 rows × 7 columns

Problem: We see that indeed a lagged version of total assets is added to the DataFrame. However, this method does not take into account whether the previous row belongs to the same company. It simply shifts row-wise, even across companies. We see for example that total_assets_lag for Adobe in 2020 equals Apple’s Total Assets for 2024. Obviously, this is not what we want.

Step 3: Fix Grouping Issues Manually

To address this, we can conditionally shift values only if the previous row belongs to the same ticker:

df = df.sort_values(['ticker', 'year'])
df['total_assets_lag'] = np.where(
    df['ticker'] == df['ticker'].shift(),
    df['total_assets'].shift(),
    np.nan
)

Inspection of the first 10 rows of the DataFrame indicates that this indeed resolved the issue.

df.head(10)

10 rows × 7 columns

However, there are still cases where the calculation did not happen as expected. For example, take a look at the observations for Nvidia and Tesla:

df.query('ticker=="NVDA" or ticker=="TSLA"')

9 rows × 7 columns

Even when years are missing, the current logic carries forward the last available value, which may be from two or more years ago. As a result of data for NVIDIA for 2022 being missing, the lagged value total_assets_lag for 2023 is now incorrectly set equal to the total assets for 2021. Similarly, the lagged total assets for Tesla for 2022 are incorrectly set equal to the value for 2020.

Step 4: Handle Missing Years

Let’s refine the condition further. Our condition should verify that the previous row belongs to the same ticker and that the year increment equals one:

df = df.sort_values(['ticker', 'year'])
df['total_assets_lag'] = np.where(
    (df['ticker'] == df['ticker'].shift()) & 
    (df['year'] - df['year'].shift() == 1),
    df['total_assets'].shift(),
    np.nan
)

Let’s inspect the DataFrame again:

df.query('ticker=="NVDA" or ticker=="TSLA"')

9 rows × 7 columns

This approach works, but the code becomes verbose and difficult to scale. If we want a 2-year lag, the condition becomes even more complex:

df = df.sort_values(['ticker', 'year'])
df['total_assets_lag2'] = np.where(
    (df['ticker'] == df['ticker'].shift(2)) & 
    (df['year'] - df['year'].shift(2) == 2),
    df['total_assets'].shift(2),
    np.nan
)

Step 5: Let Pychemist Do the Work

To simplify and generalize this process, we can use Pychemist’s built-in lag function.

Note: this requires installing and importing the Pychemist library as done at the start of this tutorial.

We can now generate a lagged variable as follows:

df = df.chem.lag('total_assets', 'ticker', 'year')

This automatically handles company grouping and time consistency. The chem.lag method also handles important edge cases automatically. For instance, it only assigns lag values when both the grouping variable (e.g., ticker) matches and the year variable increments by exactly the lag interval. This means that if a year is missing in the dataset, the function will not incorrectly carry over data from a non-consecutive year, and will instead return NaN as expected. If the lagged column already exists, it will not be overwritten, and a warning will be issued to prevent accidental data loss.

Step 6: Lag Multiple Columns

To created multiple lagged variables at once, we can set a list of variables for which lagged variables should be computed. We set replace=True because the DataFrame already contains the lagged variable for total_assets. Without this argument the function would raise a warning that this variable already exists in the DataFrame. Alternatively, we could drop this column manually.

df = df.chem.lag(['total_assets', 'revenue'], 'ticker', 'year', replace=True)

We can see that the results are as expected:

df.query('ticker=="NVDA" or ticker=="TSLA"')

9 rows × 8 columns

To generate 2-year lags (or more), simply pass a ‘shift’ parameter:

df = df.chem.lag(['total_assets', 'revenue'], 'ticker', 'year', 2)

This results in the following DataFrame:

df.head()

5 rows × 10 columns

Step 7: Create Lead Variables

Creating lead variables (i.e., future values) is just as easy. Simply use .chem.lead instead:

df = df.chem.lead(['total_assets', 'revenue'], 'ticker', 'year')

The DataFrame would look as follows:

df.head()

5 rows × 12 columns

Step 8: Compute Derived Metrics

Now that we have lags, we can compute variables such as Return on Assets (ROA) or Revenue Growth:

df=df.eval("""
    roa=net_income / ((total_assets+total_assets_lag)/2)
    growth = (revenue-revenue_lag) / revenue_lag
    """)

Inspect the results:

df.query('ticker=="NVDA" or ticker=="TSLA"')

9 rows × 14 columns

Conclusion

In this tutorial, we learned how to create lagged and lead variables manually using pandas, and why that approach often falls short, especially in grouped, irregular panel data.

Then, we saw how the Pychemist library solves these problems elegantly:

• Accurate grouping
• Handles gaps in time
• Clean, concise API
• Easy multi-period shifts
• Works seamlessly with pandas

Whether you’re building financial models or prepping data for regression analysis, Pychemist streamlines your workflow and eliminates common mistakes.

In this tutorial, we introduce a more readable and expressive alternative using a custom pandas accessor provided by the pychemist library. By leveraging df.query under the hood, the .chem.mutate accessor allows you to perform chained, conditional assignments in a cleaner and more maintainable way.

To demonstrate how to modify variables, we’ll use the IBM HR Analytics Employee Attrition & Performance dataset, available on Kaggle:
https://www.kaggle.com/datasets/pavansubhasht/ibm-hr-analytics-attrition-dataset

We start by importing the required libraries and loading the dataset into a DataFrame:

import pandas as pd
import numpy as np

df = pd.read_csv('WA_Fn-UseC_-HR-Employee-Attrition.csv')

This creates a new DataFrame df.

We can examine the first five rows of the dataset to get a sense of the variables, their names, and the types of values they contain:

df.head()

5 rows × 35 columns

Conditional Mutation with Base Pandas

Imagine we want to decide which employees are eligible for a bonus. As an example, we could use Numpy’s where function (np.where) to flag employees who haven’t been promoted in 3 or more years:

df['promotion'] = np.where(df['YearsSinceLastPromotion'] >= 3, 1, 0)

Alternatively, the same logic using df.loc:

df['promotion'] = 0
df.loc[df['YearsSinceLastPromotion'] >= 3, 'promotion'] = 1

Both approaches generate a new column promotion with a value of 1 for employees who haven’t been promoted in 3 or more years, and 0 otherwise.

These methods work well for simple logic. However, suppose we now want to promote employees who meet all the following conditions:

• Haven’t been promoted in at least 3 years
• Hold the position of Manager
• Have a performance rating of 4 or higher

Using np.where, this looks like:

df['promotion'] = np.where(
    (df['YearsSinceLastPromotion'] >= 3) & 
    (df['JobRole'] == "Manager") & 
    (df['PerformanceRating'] >= 4),
    1, 0
    )

Or using df.loc:

df['promotion'] = 0
df.loc[
    (df['YearsSinceLastPromotion'] >= 3) & 
    (df['JobRole'] == "Manager") & 
    (df['PerformanceRating'] >= 4),
    'promotion'
    ] = 1

While both work, they’re increasingly verbose and harder to read and maintain, especially as the complexity of the conditions grows.

Using pychemist for Cleaner Mutation

Instead of repeating df[...] and writing complex boolean logic, we can use a custom accessor for pandas built on top of df.query and df.loc. This method enables readable, query-style conditional logic. We’ll use the pychemist package for this.

Install the package using pip:

pip install pychemist

Then, import the library to register the accessor and perform the conditional mutation:

import pychemist

df = df.chem.mutate('YearsSinceLastPromotion >= 3 & JobRole == "Manager" & PerformanceRating >= 4', 'Promotion', 1, 0)

How chem.mutate works

The method requires:

• Query string: A valid expression compatible with df.query
• Column name: The name of the column to create or modify
• Value if True: The value to assign to rows that match the query
• Value if False (optional): The value to assign to rows that don’t match

This syntax is significantly more readable and easier to chain.

Advanced Example: Chain Multiple Mutations

You can also chain multiple mutate calls together:

df = (df
    .chem.mutate('YearsSinceLastPromotion >= 3 & JobRole == "Manager" & PerformanceRating >= 4', 'Promotion', 1, 0)
    .chem.mutate('WorkLifeBalance<2 & OverTime=="Yes"', 'HighPressure', 'Yes', 'No')
    )

This generates two new columns:

• Promotion: 1 for managers with a high performance rating (4 or higher) that haven’t received a promotion for a least 3 years, else 0
• HighPressure: 'Yes' for employees who score low on work-life balance and work overtime, else 'No'

This chaining allows your transformations to remain tidy and declarative.

Summary

This tutorial demonstrated how to use pychemist’s custom mutate accessor to simplify data manipulation in pandas. It allows:

• More concise and readable conditional logic
• Easy chaining of multiple transformations
• Reduced risk of typos and parentheses mismatches

By abstracting away the boilerplate of df.loc and np.where, pychemist helps make your data preparation code more expressive and maintainable.