Troubleshooting Airflow DAG and Task callbacks in MC

If you are unable to see Airflow DAG or task information in the Monte Carlo catalog, or you receive the following error:

"Invalid context received in MCD dag callback: no dag_run.end_date set, it looks like the dag is still running. Please check your callbacks are configured properly."

These problems are usually due to a misconfiguration in Airflow. It’s common to see DAG callbacks configured where Task callbacks should be, or sometimes only one type (task or DAG) is configured. 

 

Here are some common misconfiguration examples:

1. Incorrect Usage of default_args: defining DAG callbacks in "default_args." This is incorrect because "default_args" is used to configure Task callbacks, not DAG callbacks.

2. Tasks Stuck in Status Running in MC: MC receives tasks with status "Running". The callback might be configured to trigger at the wrong event, such as when the task starts running instead of when it completes (either successfully or with failure). Also, the callback configuration might be overwritten somewhere in your code, causing it to send the "Running" status instead of the final status.

To resolve these issues, double-check the placement and configuration of your task callbacks to ensure they are set correctly to trigger only on success or failure. Make sure that both task and DAG callbacks have been added according to our documentation, as both are required for monitoring to work correctly. If the issue persists or you need further assistance, please reach out to our team!