Upgrade to Prefect 3.0

Learn how to upgrade from Prefect 2.x to Prefect 3.0.

Prefect 3.0 introduces a number of enhancements to the OSS product: a new events & automations backend for event-driven workflows and observability, improved runtime performance, autonomous task execution and a streamlined caching layer based on transactional semantics.

The majority of these enhancements maintain compatibility with most Prefect 2.0 workflows, but there are a few caveats that you may need to adjust for.

To learn more about the enhanced performance and new features, see What’s new in Prefect 3.0.

For the majority of users, upgrading to Prefect 3.0 will be a seamless process that requires few or no code changes. This guide highlights key changes that you may need to consider when upgrading.

Prefect 2.0 refers to the 2.x lineage of the open source prefect package, and Prefect 3.0 refers exclusively to the 3.x lineage of the prefect package. Neither version is strictly tied to any aspect of Prefect’s commercial product, Prefect Cloud.

Quickstart

To upgrade to Prefect 3.0, run:

pip install -U prefect

If you self-host a Prefect server, run this command to update your database:

prefect server database upgrade

If you use a Prefect integration or extra, remember to upgrade it as well. For example:

pip install -U 'prefect[aws]'

Upgrade notes

Pydantic V2

This change affects you if: you use custom Pydantic models with Prefect features.

Prefect 3.0 is built with Pydantic 2.0 for improved performance. All Prefect objects will automatically upgrade, but if you use custom Pydantic models for flow parameters or custom blocks, you’ll need to ensure they are compatible with Pydantic 2.0. You can continue to use Pydantic 1.0 models in your own code if they do not interact directly with Prefect.

Refer to Pydantic’s migration guide for detailed information on necessary changes.

Module location and name changes

Some less-commonly used modules have been renamed, reorganized, or removed for clarity. The old import paths will continue to be supported for 6 months, but emit deprecation warnings. You can look at the deprecation code to see a full list of affected paths.

Async tasks in synchronous flows

In Prefect 2.0, it was possible to call native async tasks from synchronous flows, a pattern that is not normally supported in Python. Prefect 3.0.0 removes this behavior to reduce complexity and potential issues and edge cases. If you relied on asynchronous tasks in synchronous flows, you must either make your flow asynchronous or use a task runner that supports asynchronous execution.

Flow final states

In Prefect 2.0, the final state of a flow run was influenced by the states of its task runs; if any task run failed, the flow run was marked as failed.

In Prefect 3.0, the final state of a flow run is entirely determined by:

1. The return value of the flow function (same as in Prefect 2.0):

   • Literal values are considered successful.
   • Any explicit State that is returned will be considered the final state of the flow run. If an iterable of State objects is returned, all must be Completed for the flow run to be considered Completed. If any are Failed, the flow run will be marked as Failed.

2. Whether the flow function allows an exception to raise:

   • Exceptions that are allowed to propagate will result in a Failed state.
   • Exceptions suppressed with raise_on_failure=False will not affect the flow run state.

This change means that task failures within a flow do not automatically cause the flow run to fail unless they affect the flow’s return value or raise an uncaught exception.

When migrating from Prefect 2.0 to Prefect 3, be aware that flows may now complete successfully even if they contain failed tasks, unless you explicitly handle task failures.

To ensure your flow fails when critical tasks fail, consider these approaches:

1. Allow task exceptions to propagate by not using raise_on_failure=False.
2. Use return_state=True and explicitly check task states to conditionally raise the underlying exception or return a failed state.
3. Use try/except blocks to handle task failures and return appropriate states.

Examples

Choose the strategy that best fits your specific use case and error handling requirements.

---

Futures interface

PrefectFutures now have a standard synchronous interface, with an asynchronous one planned soon.

Automatic task caching

Prefect 3.0 introduces a powerful idempotency engine. By default, tasks in a flow run are automatically cached if they are called more than once with the same inputs. If you rely on tasks with side effects, this may result in surprising behavior. To disable caching, pass cache_policy=None to your task.

Workers

In Prefect 2.0, agents were deprecated in favor of next-generation workers. Workers are now standard in Prefect 3. For detailed information on upgrading from agents to workers, please refer to our upgrade guide.

Resolving common gotchas

AttributeError: 'coroutine' object has no attribute <some attribute>

When within an asynchronous task or flow context, if you do not await an asynchronous function or method, this error will be raised when you try to use the object.

To fix it, await the asynchronous function or method or use _sync=True.

For example, Block’s load method is asynchronous in an async context:

Similarly, if you never use an un-awaited coroutine, you may see a warning like this:

RuntimeWarning: coroutine 'some_async_callable' was never awaited
...
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

This is the same problem as above, and you may await the coroutine to fix it or use _sync=True.

TypeError: object <some Type> can't be used in 'await' expression

This error occurs when using the await keyword before an object that is not a coroutine.

To fix it, remove the await.

For example, my_task.submit(...) is always synchronous in Prefect 3.x:

See the Futures interface section for more information on this particular gotcha.

TypeError: Flow.deploy() got an unexpected keyword argument 'schedule'

In Prefect 3.0, the schedule argument has been removed in favor of the schedules argument.

This applies to both the Flow.serve and Flow.deploy methods.