The scalable approach is aggressive decoupling. The API endpoint should do three things only: validate the request (thank you, Pydantic), enqueue a job into a message broker like RabbitMQ or Redis, ...