Dagster

What It Is

Dagster is the orchestration engine that runs Awali's data pipelines. Awali bundles a managed Dagster installation and exposes it through three surfaces: a status dashboard, a Workshop IDE for writing pipeline code, and the native Dagster web UI for power-user run management.

You'll find it under the Dagster menu link in the admin navigation.

What You Can Do

See pipeline status at a glance

The Dagster dashboard (/dagster) shows:

• Whether the Dagster dev server is running and healthy.
• How many assets are defined in your project.
• Recent run history with status, duration, and timestamps.
• Environment details — Python version, Dagster version, GraphQL connectivity.
• Quick start/stop controls for the dev server, sensors, and schedules.

Author pipelines in the Workshop

The Dagster Workshop (/dagster/workshop) is a Monaco-based IDE for writing Python pipeline code:

• Create, edit, rename, and delete asset definition files.
• Auto-save (two-second debounce) so nothing is ever lost.
• Format on demand with Black.
• Execute Python directly in an integrated terminal — useful for quick scratch work or testing a function in isolation.
• Run Dagster CLI commands: dagster asset list, dagster job list, and others.
• Files you delete go to a recoverable trash.

Materialize and monitor assets

• List all defined assets and their dependencies.
• Materialize any individual asset on demand.
• Inspect run logs, duration, and resource usage for completed runs.
• View the asset lineage graph — see exactly which upstream sources feed each asset and which downstream consumers depend on it.

Schedule and trigger runs

• List available schedules and sensors defined in your project.
• Enable or disable individual schedules and sensors from the dashboard.
• Trigger ad-hoc runs from the embedded native UI.

Drop into the full Dagster UI

For workflows that need the full native experience, the embedded UI (/dagster/ui) gives you the complete Dagster web interface — including the asset graph, run launcher, schedule and sensor configuration, partitions, and backfills.

What Pipelines Typically Do in Awali

Awali ships with example assets that demonstrate the patterns most projects follow:

1. Ingest raw data from CSV drops, REST APIs, or external databases.
2. Transform with Python and Pandas — clean, deduplicate, normalize, filter.
3. Materialize into data_db where the rest of Awali (dbt models, the Dashboard, Screen Builder screens) can consume it.
4. Prepare structured records as semantic documents for the AI knowledge base.
5. Aggregate across multiple sources into unified analytics tables.
6. Measure data quality — row counts, freshness, completeness — and surface the metrics back to the dashboard.

Common Workflows

Add a new ingestion asset

1. Open the Dagster Workshop and create a new .py file under your asset definitions directory.
2. Define an @asset function that extracts and returns your data.
3. Save (auto-save fires after a couple of seconds) and format with Black.
4. From the dashboard, click Materialize on the new asset to verify it runs.
5. Wire it up with a schedule or sensor in the same file when you're ready to automate it.

Investigate a failed run

1. From the Dagster dashboard, find the failed run in recent history.
2. Click in to see the run's log output and the step that failed.
3. If you need the full failure context, open the embedded UI to see step-by-step inputs and outputs.
4. Fix the asset code in the Workshop, save, and re-materialize.

Glossary

Asset

A data artifact Dagster produces — typically a table, file, or in-memory dataset. Each asset is defined by a Python function annotated with @asset.

Materialize

Running an asset to (re)produce its data.

Lineage

The dependency graph between assets — which assets feed which.

Job

A collection of asset materializations or ops that run together.

Schedule

A cron-style trigger that runs a job at fixed times.

Sensor

An event-driven trigger that runs a job in response to some condition (a new file, an upstream update, an external signal).

Run

One execution of a job, with its own log, status, and duration.

Dev server

The locally-managed Dagster process Awali runs to serve the UI and execute jobs.

Access & Security

• Admin role required, plus an active MFA session.
• Terminal commands are validated against an allowlist; shell metacharacters (&&, ||, |) and destructive operations (rm, mv, sudo) are blocked.
• The Workshop and terminal share infrastructure with DBT Workshop so the security model is identical.
• All file operations are scoped to the Dagster project directory.

← Back to overview