python - Integrating Dagster with Django

Question

Hi I am trying to integrate Dagster into ongoing Django project. I am kind of struggling with providing Django context (models, apps, ...) to Dagster. As of now I am just checking wether dagit is present in sys.argv[0] in init.py of apps that are using Dagster.

<!-- language: python -->
## project/app/__init__.py
import sys

import django

if 'dagit-cli' in sys.argv[0]:
    print('dagit')
    django.setup()

Can anyone help me with setup?

Answer 1

I would also use the Django custom management command as Timothé said to execute a function or access ORM in the Django's context.

However, if your use case is required to access the Django ORM directly, you'll need to do a few steps:

1. Add the Django project to sys.path, so that Python can find packages.
2. Set up the Django environment.

import os
import sys

import django
from dagster import execute_pipeline, pipeline, solid


# Add the project to sys.path, so that Python can find packages
PROJECT_ROOT = os.path.join(os.path.dirname(__file__), 'demo')
sys.path.append(PROJECT_ROOT)

# Set up the Django environment
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'demo.settings')
django.setup()


from customers.models import Customer


@solid
def hello_django_orm(context):
    number_of_customers = Customer.objects.count()
    message = f'Found {number_of_customers} customers!'
    context.log.info(message)
    return number_of_customers


@pipeline
def hello_django_orm_pipeline():
    hello_django_orm()


if __name__ == '__main__':
    result = execute_pipeline(hello_django_orm_pipeline)
    assert result.success

See the result below.

Kindly see the full example here.

Answer 2

As far as I know, there is no high-level integration, but I found some workarounds to launch a pipeline from a django view:

• synchronously run a pipeline by calling package function execute_pipeline(<your_pipeline>) from python code, for example inside a View (even if it is a non-sense in terms of synchronous/asynchronous paradigm)

• create a custom management command and call it from inside python code with the django.core.management.call_command() hook