The .chem.lead() method is a custom pandas accessor that creates lead versions of one or more variables in a DataFrame. It works by shifting the specified columns backward in time (i.e., to future periods) and merging the result back onto the original DataFrame.

This is especially useful for panel or time-series data where each row represents an observation at a time point for a particular unit (e.g., experiment, company, individual).

For a concrete example of usage, refer to https://pychemist.com/creating-lag-and-lead-variables/.

Accessor Registration

This method is registered with pandas under the accessor name .chem:

Access the method like this:

df.chem.lead(...)

Method Signature

df.chem.lead(variables, identifier, time, shift=1, *, replace=False)

Parameters

Returns

• A modified copy of the original pd.DataFrame, with lead versions of the specified variables added as new columns.

Behavior

1. Validates parameter types and column existence.
2. Creates a shifted version of the selected variables by subtracting the shift from the time column.
3. Merges this lead DataFrame back into the original, using identifier and time as keys.
4. New columns are suffixed with:
   • _lead for shift=1
   • _leadN for shift=N (e.g., _lead3 for shift=3)
5. If replace=False and a target lead column already exists, a ValueError is raised.

Notes

• The original DataFrame remains unchanged.
• Supports multiple variables and vectorized group-wise operations.
• Useful for forecasting models or previewing future values.
• For backward-looking operations, see .chem.lag().

Example

df = pd.DataFrame({
    "id": [1, 1, 1, 2, 2, 2],
    "time": [1, 2, 3, 1, 2, 3],
    "mass": [10, 15, 20, 5, 10, 15]
})

df2 = df.chem.lead(variables="mass", identifier="id", time="time", shift=1)

This will produce a new column called mass_lead with the mass value from the next time step (grouped by id).

Or more concisely:

df2 = df.chem.lead("mass", "id", "time")

Common Use Cases

• Creating future-looking predictors in modeling.
• Forecast validation (comparing current and future state).
• Detecting upcoming transitions or changes in sequence.

Error Handling

• TypeError if variables is not a string or list of strings.
• TypeError if replace is not a boolean.
• TypeError if shift is not a positive integer.
• ValueError if a specified column does not exist.
• ValueError if a lead column already exists and replace=False.

Internals

The method:

1. Copies the relevant subset of the DataFrame.
2. Shifts the time column backward (df[time] - shift) to align future values.
3. Applies suffixes such as _lead or _leadN.
4. Merges the lead values back into the original DataFrame using pd.merge(...).

See Also

• .chem.lag() – for creating lagged (past) variables
• .chem.mutate() – for conditional column assignment
• pd.DataFrame.shift() – basic shifting

The .chem.lag() method is a custom pandas accessor that creates lagged versions of one or more variables in a DataFrame. It operates by shifting the specified columns by a given number of time periods and merging the result back onto the original DataFrame.

This is especially useful for time-series panel data where each row belongs to a unique unit (e.g., company, experiment, patient) over time.

For a concrete example of usage, refer to https://pychemist.com/creating-lag-and-lead-variables/.

Accessor Registration

This method is registered with pandas under the accessor name .chem:

Access the method like this:

df.chem.lag(...)

Method Signature

df.chem.lag(variables, identifier, time, shift=1, *, replace=False)

Parameters

Returns

• A modified copy of the original pd.DataFrame, with lagged versions of the specified variables added as new columns.

Behavior

1. Verifies input types and column existence.
2. Constructs a lagged version of the selected variables by shifting the time column forward by the given shift amount.
3. Merges this lagged DataFrame back onto the original, based on the identifier and time.
4. New columns are suffixed with:
   • _lag for shift=1
   • _lagN for shift=N (e.g., _lag3 for shift=3)
5. If replace=False and any of the output columns already exist, the method raises a ValueError.

Notes

• The original DataFrame remains unchanged.
• Supports multiple variables and vectorized operations.
• Designed for panel or longitudinal data.
• For negative shift (lead variables) refert to .chem.lead()

Example

df = pd.DataFrame({
    "id": [1, 1, 1, 2, 2, 2],
    "time": [1, 2, 3, 1, 2, 3],
    "mass": [10, 15, 20, 5, 10, 15]
})

df_lagged = df.chem.lag(variables="mass", identifier="id", time="time", shift=1)

This will produce a new column called mass_lag with the mass value from the previous time step (by id).

Or more concisely:

df_lagged = df.chem.lag("mass", "id", "time")

Common Use Cases

• Creating lagged predictors in time-series regression.
• Modeling delayed effects in experiments.
• Comparing values across sequential periods.

Error Handling

• TypeError if variables is not a string or list of strings.
• TypeError if replace is not a boolean.
• TypeError if shift is not a positive integer.
• ValueError if any specified variable is missing from the DataFrame.
• ValueError if a lagged column already exists and replace=False.

Internals

The method:

1. Creates a shifted copy of the target columns using df[time] + shift.
2. Applies suffixes like _lag, _lag2, etc.
3. Merges the lagged columns back onto the original DataFrame using pd.merge(...).

See Also

• .chem.mutate() – for conditional column assignment
• pd.DataFrame.shift() – basic shifting

The .chem.mutate() method is a custom pandas accessor that enables conditional assignment of values to a column in a DataFrame using a query string. It allows for clean, chainable DataFrame transformations and always returns a modified copy of the DataFrame.

This is particularly useful when working with chemistry-related tabular data and transformations, but it can be applied more broadly.

For a concrete example of usage, refer to https://pychemist.com/mutate/.

Accessor Registration

This method is registered using the pandas API extensions system. This means you can access the method via:

df.chem.mutate(...)

Method Signature

df.chem.mutate(query_str, column, value, other=None)

Parameters

Returns

• A modified copy of the original pd.DataFrame.

This method does not modify the DataFrame in-place.

Behavior

1. The method evaluates the query_str on the DataFrame.
2. For rows matching the query, the specified column is set to value.
3. For rows not matching the query, the column is set to other only if other is provided.
4. The column is created if it does not already exist.

Notes

• This method is part of a custom accessor named .chem.
• The original DataFrame is left unchanged.
• You can use it in method chains, e.g.: df2 = df.chem.mutate(query_str=”mass > 10″, column=”label”, value=”heavy”, other=”light”)
• value and other can be scalars or array-like, but must match the number of rows being assigned.

Or simply:

df2 = df.chem.mutate("mass > 10", "label", "heavy", "light")

Example

See https://pychemist.com/mutate/ for a practical example using .chem.mutate().

Common Use Cases

• Labeling chemical species by some threshold: df.chem.mutate(“concentration > 1.0”, “status”, “high”, “low”)
• Creating a new boolean flag: df.chem.mutate(“pH < 7”, “is_acidic”, True, other=False)

Error Handling

• Any syntax errors in query_str will raise a pandas exception at runtime.
• If value or other are array-like but do not match the shape of the selected rows, a ValueError will be raised.

Internals

Internally, the method:

1. Copies the DataFrame (df = self._obj.copy()),
2. Applies the query string using df.query(query_str),
3. Locates matching and non-matching row indices,
4. Assigns value and optionally other to the specified column,
5. Returns the modified DataFrame.

In this tutorial, we examine how to create lagged and lead variables: essential tools for time series and panel data analysis. Whether you’re modeling financial trends or running regressions, such transformations are often essential to compute new variables, such as Return on Assets or Revenue Growth. We’ll also explore how the Pychemist library simplifies this task.

We use pandas and the pychemist library to demonstrate this. For this tutorial we rely on a dataset of financial information for 20 large US companies, scraped from EDGAR. Some missing values in the dataset have been imputed for demonstration purposes.

While pandas lets you create lagged values using .shift(), it doesn’t always behave as expected for grouped data or irregular time series. Pychemist provides a simpler and more reliable alternative through its .chemaccessor.

Step 1: Load the Data and Libraries

We start by importing the necessary libraries and loading the dataset. pandas and numpy are used for working with panel datasets. Additionally, we import pychemist to download the financial dataset and to enable the custom pandas extensions (accessors) used in this example.

Note: If you haven’t installed pychemist yet, you can do so with the following command:

pip install pychemist

Import the required libraries and load the dataset into a DataFrame (df):

import pandas as pd
import numpy as np
import pychemist

df = pychemist.load('financials')

Let’s inspect the first few rows:

df.head()

5 rows × 6 columns

Step 2: Create a Lag Manually

First, we sort the dataset by company (ticker) and time (year). Second, we create a lagged variable total_assets_lag using Pandas’ built-in .shift() method.

df = df.sort_values(['ticker', 'year'])
df['total_assets_lag'] = df['total_assets'].shift()

Next, we inspect the DataFrame:

df.head(10)

10 rows × 7 columns

Problem: We see that indeed a lagged version of total assets is added to the DataFrame. However, this method does not take into account whether the previous row belongs to the same company. It simply shifts row-wise, even across companies. We see for example that total_assets_lag for Adobe in 2020 equals Apple’s Total Assets for 2024. Obviously, this is not what we want.

Step 3: Fix Grouping Issues Manually

To address this, we can conditionally shift values only if the previous row belongs to the same ticker:

df = df.sort_values(['ticker', 'year'])
df['total_assets_lag'] = np.where(
    df['ticker'] == df['ticker'].shift(),
    df['total_assets'].shift(),
    np.nan
)

Inspection of the first 10 rows of the DataFrame indicates that this indeed resolved the issue.

df.head(10)

10 rows × 7 columns

However, there are still cases where the calculation did not happen as expected. For example, take a look at the observations for Nvidia and Tesla:

df.query('ticker=="NVDA" or ticker=="TSLA"')

9 rows × 7 columns

Even when years are missing, the current logic carries forward the last available value, which may be from two or more years ago. As a result of data for NVIDIA for 2022 being missing, the lagged value total_assets_lag for 2023 is now incorrectly set equal to the total assets for 2021. Similarly, the lagged total assets for Tesla for 2022 are incorrectly set equal to the value for 2020.

Step 4: Handle Missing Years

Let’s refine the condition further. Our condition should verify that the previous row belongs to the same ticker and that the year increment equals one:

df = df.sort_values(['ticker', 'year'])
df['total_assets_lag'] = np.where(
    (df['ticker'] == df['ticker'].shift()) & 
    (df['year'] - df['year'].shift() == 1),
    df['total_assets'].shift(),
    np.nan
)

Let’s inspect the DataFrame again:

df.query('ticker=="NVDA" or ticker=="TSLA"')

9 rows × 7 columns

This approach works, but the code becomes verbose and difficult to scale. If we want a 2-year lag, the condition becomes even more complex:

df = df.sort_values(['ticker', 'year'])
df['total_assets_lag2'] = np.where(
    (df['ticker'] == df['ticker'].shift(2)) & 
    (df['year'] - df['year'].shift(2) == 2),
    df['total_assets'].shift(2),
    np.nan
)

Step 5: Let Pychemist Do the Work

To simplify and generalize this process, we can use Pychemist’s built-in lag function.

Note: this requires installing and importing the Pychemist library as done at the start of this tutorial.

We can now generate a lagged variable as follows:

df = df.chem.lag('total_assets', 'ticker', 'year')

This automatically handles company grouping and time consistency. The chem.lag method also handles important edge cases automatically. For instance, it only assigns lag values when both the grouping variable (e.g., ticker) matches and the year variable increments by exactly the lag interval. This means that if a year is missing in the dataset, the function will not incorrectly carry over data from a non-consecutive year, and will instead return NaN as expected. If the lagged column already exists, it will not be overwritten, and a warning will be issued to prevent accidental data loss.

Step 6: Lag Multiple Columns

To created multiple lagged variables at once, we can set a list of variables for which lagged variables should be computed. We set replace=True because the DataFrame already contains the lagged variable for total_assets. Without this argument the function would raise a warning that this variable already exists in the DataFrame. Alternatively, we could drop this column manually.

df = df.chem.lag(['total_assets', 'revenue'], 'ticker', 'year', replace=True)

We can see that the results are as expected:

df.query('ticker=="NVDA" or ticker=="TSLA"')

9 rows × 8 columns

To generate 2-year lags (or more), simply pass a ‘shift’ parameter:

df = df.chem.lag(['total_assets', 'revenue'], 'ticker', 'year', 2)

This results in the following DataFrame:

df.head()

5 rows × 10 columns

Step 7: Create Lead Variables

Creating lead variables (i.e., future values) is just as easy. Simply use .chem.lead instead:

df = df.chem.lead(['total_assets', 'revenue'], 'ticker', 'year')

The DataFrame would look as follows:

df.head()

5 rows × 12 columns

Step 8: Compute Derived Metrics

Now that we have lags, we can compute variables such as Return on Assets (ROA) or Revenue Growth:

df=df.eval("""
    roa=net_income / ((total_assets+total_assets_lag)/2)
    growth = (revenue-revenue_lag) / revenue_lag
    """)

Inspect the results:

df.query('ticker=="NVDA" or ticker=="TSLA"')

9 rows × 14 columns

Conclusion

In this tutorial, we learned how to create lagged and lead variables manually using pandas, and why that approach often falls short, especially in grouped, irregular panel data.

Then, we saw how the Pychemist library solves these problems elegantly:

• Accurate grouping
• Handles gaps in time
• Clean, concise API
• Easy multi-period shifts
• Works seamlessly with pandas

Whether you’re building financial models or prepping data for regression analysis, Pychemist streamlines your workflow and eliminates common mistakes.

In this tutorial, we introduce a more readable and expressive alternative using a custom pandas accessor provided by the pychemist library. By leveraging df.query under the hood, the .chem.mutate accessor allows you to perform chained, conditional assignments in a cleaner and more maintainable way.

To demonstrate how to modify variables, we’ll use the IBM HR Analytics Employee Attrition & Performance dataset, available on Kaggle:
https://www.kaggle.com/datasets/pavansubhasht/ibm-hr-analytics-attrition-dataset

We start by importing the required libraries and loading the dataset into a DataFrame:

import pandas as pd
import numpy as np

df = pd.read_csv('WA_Fn-UseC_-HR-Employee-Attrition.csv')

This creates a new DataFrame df.

We can examine the first five rows of the dataset to get a sense of the variables, their names, and the types of values they contain:

df.head()

5 rows × 35 columns

Conditional Mutation with Base Pandas

Imagine we want to decide which employees are eligible for a bonus. As an example, we could use Numpy’s where function (np.where) to flag employees who haven’t been promoted in 3 or more years:

df['promotion'] = np.where(df['YearsSinceLastPromotion'] >= 3, 1, 0)

Alternatively, the same logic using df.loc:

df['promotion'] = 0
df.loc[df['YearsSinceLastPromotion'] >= 3, 'promotion'] = 1

Both approaches generate a new column promotion with a value of 1 for employees who haven’t been promoted in 3 or more years, and 0 otherwise.

These methods work well for simple logic. However, suppose we now want to promote employees who meet all the following conditions:

• Haven’t been promoted in at least 3 years
• Hold the position of Manager
• Have a performance rating of 4 or higher

Using np.where, this looks like:

df['promotion'] = np.where(
    (df['YearsSinceLastPromotion'] >= 3) & 
    (df['JobRole'] == "Manager") & 
    (df['PerformanceRating'] >= 4),
    1, 0
    )

Or using df.loc:

df['promotion'] = 0
df.loc[
    (df['YearsSinceLastPromotion'] >= 3) & 
    (df['JobRole'] == "Manager") & 
    (df['PerformanceRating'] >= 4),
    'promotion'
    ] = 1

While both work, they’re increasingly verbose and harder to read and maintain, especially as the complexity of the conditions grows.

Using pychemist for Cleaner Mutation

Instead of repeating df[...] and writing complex boolean logic, we can use a custom accessor for pandas built on top of df.query and df.loc. This method enables readable, query-style conditional logic. We’ll use the pychemist package for this.

Install the package using pip:

pip install pychemist

Then, import the library to register the accessor and perform the conditional mutation:

import pychemist

df = df.chem.mutate('YearsSinceLastPromotion >= 3 & JobRole == "Manager" & PerformanceRating >= 4', 'Promotion', 1, 0)

How chem.mutate works

The method requires:

• Query string: A valid expression compatible with df.query
• Column name: The name of the column to create or modify
• Value if True: The value to assign to rows that match the query
• Value if False (optional): The value to assign to rows that don’t match

This syntax is significantly more readable and easier to chain.

Advanced Example: Chain Multiple Mutations

You can also chain multiple mutate calls together:

df = (df
    .chem.mutate('YearsSinceLastPromotion >= 3 & JobRole == "Manager" & PerformanceRating >= 4', 'Promotion', 1, 0)
    .chem.mutate('WorkLifeBalance<2 & OverTime=="Yes"', 'HighPressure', 'Yes', 'No')
    )

This generates two new columns:

• Promotion: 1 for managers with a high performance rating (4 or higher) that haven’t received a promotion for a least 3 years, else 0
• HighPressure: 'Yes' for employees who score low on work-life balance and work overtime, else 'No'

This chaining allows your transformations to remain tidy and declarative.

Summary

This tutorial demonstrated how to use pychemist’s custom mutate accessor to simplify data manipulation in pandas. It allows:

• More concise and readable conditional logic
• Easy chaining of multiple transformations
• Reduced risk of typos and parentheses mismatches

By abstracting away the boilerplate of df.loc and np.where, pychemist helps make your data preparation code more expressive and maintainable.