Airflow Xcom Exclusive -

from airflow.providers.common.sql.operators.sql import SQLExecuteQueryOperator purge_old_xcoms = SQLExecuteQueryOperator( task_id='purge_old_xcoms', conn_id='airflow_db', # Points to your backend metadata db sql=""" DELETE FROM xcom WHERE timestamp < NOW() - INTERVAL '30 days'; """ ) Use code with caution. 5. XCom Troubleshooting Guide

By default, Airflow uses JSON serialization for XComs. While secure and standard, it fails when tasks attempt to return non-serializable objects like complex custom Python classes, NumPy arrays, or Pandas DataFrames. airflow xcom exclusive

To achieve the benefits of XCom exclusive—efficient, lightweight, and maintainable data sharing—follow these guidelines: from airflow

There are two primary ways to handle XComs: the traditional approach and the TaskFlow API approach introduced in Airflow 2.0. A. The Traditional Approach ( xcom_push / xcom_pull ) airflow xcom exclusive

To ensure your Airflow pipelines remain performant, reliable, and clean, observe these strict XCom architectural patterns:

from datetime import datetime from airflow.decorators import dag, task @dag(start_date=datetime(2026, 1, 1), schedule=None) def taskflow_xcom_dag(): @task def train_model(): # Automatically pushed to key 'return_value' return 0.94 @task def evaluate_model(accuracy: float): # Automatically pulled from upstream task print(f"Model accuracy is accuracy") accuracy_data = train_model() evaluate_model(accuracy_data) taskflow_xcom_dag() Use code with caution.