dagster-expert

分类: 开发与编程 | 上传者: dagster-io | 下载: 0 | 版本: v1.0（最新）

使用 Dagster 和 dg CLI 的专家指南。在执行任何需要 Dagster 特定知识或引用资产、具体化、组件、数据工具或数据管道的任务之前，请务必使用。常见任务可能包括创建新项目、添加新定义、了解当前项目结构、回答有关代码库的一般问题（查找资产、计划、传感器、组件或作业定义）、调试问题或提供有关特定 Dagster 概念的深入信息。 --- ## 核心 Dagster 概念仅简要定义（有关详细示例，请参阅参考文件）： - **资产**：由管道生成的持久对象（表、文件、模型） - **组件**：生成与特定域相关的定义（资产、计划、传感器、作业等）的可重用构建块。 ## 集成工作流程与任何外部工具或服务集成时，请阅读[集成库索引](./references/integrations/INDEX.md)。其中包含有关存在哪些集成库的信息，以及有关如何为没有已发布库的工具创建新的自定义集成的参考。 ## dg CLI `dg` CLI 是与 Dagster 以编程方式交互的推荐方式（添加定义、启动运行、探索项目结构等）。它作为“dagster-dg-cli”包的一部分安装。如果给定任务存在相关 CLI 命令，请始终尝试使用它。仅当完成用户目标绝对必要时才探索现有项目结构。在许多情况下，现有的 CLI 工具对项目结构有足够的了解，这意味着列出和读取现有文件是浪费且不必要的。几乎所有返回信息的“dg”命令都有一个“--json”标志，可用于获取机器可读格式的信息。除非您直接向用户显示信息，否则这应该优于默认表输出。＃＃紫外线

更新日志: Source: GitHub https://github.com/dagster-io/skills

目录结构

当前层级: tree/master/skills/dagster-expert/skills/dagster-expert/

📁 references/
- 📁 assets/
  - 📄 advanced-patterns.md 3.9 KB
  - 📄 definition-metadata.md 2.1 KB
  - 📄 dependencies.md 2.0 KB
  - 📄 INDEX.md 2.8 KB
- 📁 automation/
  - 📁 declarative-automation/
    
    📄 advanced.md 7.4 KB
    
    📄 core-concepts.md 6.8 KB
    
    📄 customization.md 4.5 KB
    
    📄 INDEX.md 2.6 KB
    
    📄 operands.md 5.8 KB
    
    📄 operators.md 4.6 KB
  - 📁 sensors/
    
    📄 asset-sensors.md 3.6 KB
    
    📄 basic-sensors.md 3.5 KB
    
    📄 run-status-sensors.md 3.5 KB
  - 📄 choosing-automation.md 2.0 KB
  - 📄 schedules.md 3.8 KB
- 📁 cli/
  - 📁 api/
    
    📁 agent/
    
    📄 get.md 133 B
    
    📄 INDEX.md 417 B
    
    📄 list.md 111 B
    
    📁 alert-policy/
    
    📄 INDEX.md 479 B
    
    📄 list.md 172 B
    
    📄 sync.md 348 B
    
    📁 artifact/
    
    📄 download.md 310 B
    
    📄 INDEX.md 466 B
    
    📄 upload.md 275 B
    
    📁 asset/
    
    📄 get-evaluations.md 392 B
    
    📄 get-events.md 409 B
    
    📄 get-health.md 293 B
    
    📄 get-partition-status.md 289 B
    
    📄 get.md 336 B
    
    📄 INDEX.md 808 B
    
    📄 list.md 243 B
    
    📁 code-location/
    
    📄 add.md 413 B
    
    📄 delete.md 193 B
    
    📄 get.md 183 B
    
    📄 INDEX.md 675 B
    
    📄 list.md 173 B
    
    📁 deployment/
    
    📄 delete.md 163 B
    
    📄 get.md 131 B
    
    📄 INDEX.md 765 B
    
    📄 list.md 126 B
    
    📄 settings-get.md 188 B
    
    📄 settings-set.md 277 B
    
    📁 organization/
    
    📄 INDEX.md 764 B
    
    📄 saml-remove.md 209 B
    
    📄 saml-upload.md 259 B
    
    📄 settings-get.md 196 B
    
    📄 settings-set.md 287 B
    
    📁 run/
    
    📄 get-events.md 609 B
    
    📄 get-logs.md 430 B
    
    📄 get.md 112 B
    
    📄 INDEX.md 684 B
    
    📄 list.md 297 B
    
    📁 schedule/
    
    📄 get-ticks.md 535 B
    
    📄 get.md 134 B
    
    📄 INDEX.md 619 B
    
    📄 list.md 188 B
    
    📁 secret/
    
    📄 get.md 233 B
    
    📄 INDEX.md 444 B
    
    📄 list.md 197 B
    
    📁 sensor/
    
    📄 get-ticks.md 525 B
    
    📄 get.md 126 B
    
    📄 INDEX.md 585 B
    
    📄 list.md 180 B
    
    📄 asset-check.md 728 B
    
    📄 general.md 806 B
    
    📄 INDEX.md 2.1 KB
    
    📄 job.md 364 B
  - 📁 list/
    
    📄 component-tree.md 131 B
    
    📄 envs.md 383 B
    
    📄 INDEX.md 711 B
    
    📄 projects.md 120 B
  - 📁 plus/
    
    📁 config/
    
    📄 set.md 434 B
    
    📄 view.md 234 B
    
    📁 create/
    
    📄 ci-api-token.md 283 B
    
    📁 deploy/
    
    📄 configure.md 1.2 KB
    
    📄 deploy.md 427 B
    
    📁 integrations/
    
    📄 dbt.md 1004 B
    
    📁 pull/
    
    📄 env.md 278 B
    
    📄 INDEX.md 1.3 KB
    
    📄 login.md 244 B
  - 📁 scaffold/
    
    📄 component.md 907 B
    
    📄 defs.md 1.8 KB
  - 📁 utils/
    
    📄 INDEX.md 711 B
    
    📄 inspect-component.md 1.2 KB
    
    📄 integrations.md 205 B
    
    📄 refresh-defs-state.md 284 B
  - 📄 check.md 454 B
  - 📄 create-dagster.md 878 B
  - 📄 dev.md 219 B
  - 📄 launch.md 780 B
  - 📄 list-components.md 210 B
  - 📄 list-defs.md 632 B
- 📁 components/
  - 📁 state-backed/
    
    📄 creating.md 6.7 KB
    
    📄 using.md 9.9 KB
  - 📄 creating-components.md 3.9 KB
  - 📄 designing-component-integrations.md 9.9 KB
  - 📄 resolved-framework.md 7.7 KB
  - 📄 subclassing-components.md 3.4 KB
  - 📄 template-variables.md 4.1 KB
- 📁 integrations/
  - 📁 dagster-airbyte/
    
    📄 INDEX.md 173 B
  - 📁 dagster-airlift/
    
    📄 INDEX.md 184 B
  - 📁 dagster-aws/
    
    📄 INDEX.md 160 B
  - 📁 dagster-azure/
    
    📄 INDEX.md 171 B
  - 📁 dagster-celery/
    
    📄 INDEX.md 164 B
  - 📁 dagster-celery-docker/
    
    📄 INDEX.md 193 B
  - 📁 dagster-celery-k8s/
    
    📄 INDEX.md 194 B
  - 📁 dagster-census/
    
    📄 INDEX.md 195 B
  - 📁 dagster-dask/
    
    📄 INDEX.md 164 B
  - 📁 dagster-databricks/
    
    📄 INDEX.md 179 B
  - 📁 dagster-datadog/
    
    📄 INDEX.md 170 B
  - 📁 dagster-datahub/
    
    📄 INDEX.md 181 B
  - 📁 dagster-dbt/
    
    📄 asset-checks.md 1.5 KB
    
    📄 component-based-integration.md 6.1 KB
    
    📄 dbt-cloud.md 2.1 KB
    
    📄 dependencies.md 1.2 KB
    
    📄 INDEX.md 1.5 KB
    
    📄 pythonic-integration.md 8.1 KB
    
    📄 scaffolding.md 1.9 KB
  - 📁 dagster-deltalake/
    
    📄 INDEX.md 208 B
  - 📁 dagster-deltalake-pandas/
    
    📄 INDEX.md 222 B
  - 📁 dagster-deltalake-polars/
    
    📄 INDEX.md 222 B
  - 📁 dagster-dlt/
    
    📄 INDEX.md 159 B
  - 📁 dagster-docker/
    
    📄 INDEX.md 161 B
  - 📁 dagster-duckdb/
    
    📄 INDEX.md 167 B
  - 📁 dagster-duckdb-pandas/
    
    📄 INDEX.md 203 B
  - 📁 dagster-duckdb-polars/
    
    📄 INDEX.md 203 B
  - 📁 dagster-duckdb-pyspark/
    
    📄 INDEX.md 203 B
  - 📁 dagster-embedded-elt/
    
    📄 INDEX.md 192 B
  - 📁 dagster-fivetran/
    
    📄 INDEX.md 177 B
  - 📁 dagster-gcp/
    
    📄 INDEX.md 167 B
  - 📁 dagster-gcp-pandas/
    
    📄 INDEX.md 216 B
  - 📁 dagster-gcp-pyspark/
    
    📄 INDEX.md 217 B
  - 📁 dagster-github/
    
    📄 INDEX.md 171 B
  - 📁 dagster-great-expectations/
    
    📄 INDEX.md 213 B
  - 📁 dagster-hightouch/
    
    📄 INDEX.md 181 B
  - 📁 dagster-iceberg/
    
    📄 INDEX.md 207 B
  - 📁 dagster-jupyter/
    
    📄 INDEX.md 163 B
  - 📁 dagster-k8s/
    
    📄 INDEX.md 165 B
  - 📁 dagster-looker/
    
    📄 INDEX.md 152 B
  - 📁 dagster-mlflow/
    
    📄 INDEX.md 181 B
  - 📁 dagster-msteams/
    
    📄 INDEX.md 182 B
  - 📁 dagster-mysql/
    
    📄 INDEX.md 157 B
  - 📁 dagster-omni/
    
    📄 INDEX.md 146 B
  - 📁 dagster-openai/
    
    📄 INDEX.md 156 B
  - 📁 dagster-pagerduty/
    
    📄 INDEX.md 176 B
  - 📁 dagster-pandas/
    
    📄 INDEX.md 171 B
  - 📁 dagster-pandera/
    
    📄 INDEX.md 169 B
  - 📁 dagster-papertrail/
    
    📄 INDEX.md 168 B
  - 📁 dagster-polars/
    
    📄 INDEX.md 203 B
  - 📁 dagster-polytomic/
    
    📄 INDEX.md 155 B
  - 📁 dagster-postgres/
    
    📄 INDEX.md 171 B
  - 📁 dagster-powerbi/
    
    📄 INDEX.md 154 B
  - 📁 dagster-prometheus/
    
    📄 INDEX.md 172 B
  - 📁 dagster-pyspark/
    
    📄 INDEX.md 169 B
  - 📁 dagster-sigma/
    
    📄 INDEX.md 157 B
  - 📁 dagster-slack/
    
    📄 INDEX.md 165 B
  - 📁 dagster-sling/
    
    📄 INDEX.md 153 B
  - 📁 dagster-snowflake/
    
    📄 INDEX.md 174 B
  - 📁 dagster-snowflake-pandas/
    
    📄 INDEX.md 221 B
  - 📁 dagster-snowflake-polars/
    
    📄 INDEX.md 221 B
  - 📁 dagster-snowflake-pyspark/
    
    📄 INDEX.md 221 B
  - 📁 dagster-spark/
    
    📄 INDEX.md 168 B
  - 📁 dagster-ssh/
    
    📄 INDEX.md 149 B
  - 📁 dagster-tableau/
    
    📄 INDEX.md 156 B
  - 📁 dagster-twilio/
    
    📄 INDEX.md 159 B
  - 📁 dagster-wandb/
    
    📄 INDEX.md 207 B
  - 📄 INDEX.md 9.7 KB
- 📁 migration/
  - 📄 INDEX.md 644 B
  - 📄 sensors-to-declarative-automation.md 373 B
- 📄 asset-selection.md 5.7 KB
- 📄 env-vars.md 3.4 KB
📄 SKILL.md 6.8 KB

SKILL.md

---
name: dagster-expert
description:
  Expert guidance for working with Dagster and the dg CLI. ALWAYS use before doing any task that requires
  knowledge specific to Dagster, or that references assets, materialization, components, data tools or data pipelines.
  Common tasks may include creating a new project, adding new definitions, understanding the current project structure, answering general questions about the codebase (finding asset, schedule, sensor, component or job definitions), debugging issues, or providing deep information about a specific Dagster concept.
---

## Core Dagster Concepts

Brief definitions only (see reference files for detailed examples):

- **Asset**: Persistent object (table, file, model) produced by your pipeline
- **Component**: Reusable building block that generates definitions (assets, schedules, sensors, jobs, etc.) relevant to a particular domain.

## Integration Workflow

When integrating with ANY external tool or service, read the [Integration libraries index](./references/integrations/INDEX.md). This contains information about which integration libraries exist, and references on how to create new custom integrations for tools that do not have a published library.

## dg CLI

The `dg` CLI is the recommended way to programmatically interact with Dagster (adding definitions, launching runs, exploring project structure, etc.). It is installed as part of the `dagster-dg-cli` package. If a relevant CLI command for a given task exists, always attempt to use it.

ONLY explore the existing project structure if it is strictly necessary to accomplish the user's goal. In many cases, existing CLI tools will have sufficient understanding of the project structure, meaning listing and reading existing files is wasteful and unnecessary.

Almost all `dg` commands that return information have a `--json` flag that can be used to get the information in a machine-readable format. This should be preferred over the default table output unless you are directly showing the information to the user.

## UV Compatibility

Projects typically use `uv` for dependency management, and it is recommended to use it for `dg` commands if possible:

```bash
uv run dg list defs
uv run dg launch --assets my_asset
```

## CRITICAL: Always Read Reference Files Before Answering

NEVER answer from memory or guess at CLI commands, APIs, or syntax. ALWAYS read the relevant reference file(s) from the Reference Index below before responding.

For every question, identify which reference file(s) are relevant using the index descriptions, read them, then answer based on what you read.

## Reference Index

<!-- BEGIN GENERATED INDEX -->

- [Asset Selection Syntax](./references/asset-selection.md) — filtering assets by tag, group, kind, upstream, or downstream; AssetSelection in Python, UI search bar, or CLI
- [Environment Variables](./references/env-vars.md) — configuring environment variables across different environments
- [Asset Patterns](./references/assets/INDEX.md) — defining assets, dependencies, metadata, partitions, or multi-asset definitions
- [Choosing an Automation Approach](./references/automation/choosing-automation.md) — deciding between schedules, sensors, and declarative automation
- [Schedules](./references/automation/schedules.md) — time-based automation with cron expressions
- [Declarative Automation](./references/automation/declarative-automation/INDEX.md) — asset-centric condition-based automation using AutomationCondition
- [Asset Sensors](./references/automation/sensors/asset-sensors.md) — triggering on asset materialization events
- [Basic Sensors](./references/automation/sensors/basic-sensors.md) — event-driven automation with file watching or custom polling
- [Run Status Sensors](./references/automation/sensors/run-status-sensors.md) — reacting to run success, failure, or other status changes
- [dg check](./references/cli/check.md) — validating project configuration or definitions
- [create-dagster](./references/cli/create-dagster.md) — creating a new Dagster project from scratch
- [dg dev](./references/cli/dev.md) — starting a local Dagster development instance
- [dg launch](./references/cli/launch.md) — materializing assets or executing jobs locally
- [dg list components](./references/cli/list-components.md) — seeing available component types for scaffolding
- [dg list defs](./references/cli/list-defs.md) — listing or filtering registered definitions
- [Dagster Plus API](./references/cli/api/INDEX.md) — dg api, programmatically querying or managing Dagster Plus resources (assets, runs, deployments, code locations, schedules, sensors, secrets, etc.)
- [dg list](./references/cli/list/INDEX.md) — exploring project structure (component tree, environment variables, workspace projects)
- [Dagster Plus CLI](./references/cli/plus/INDEX.md) — dg plus, Dagster Plus authentication, configuration, and deployment; logging in, setting config, creating API tokens, deploying code, pulling env vars, managing dbt manifests
- [dg scaffold component](./references/cli/scaffold/component.md) — creating a custom reusable component type
- [dg scaffold defs](./references/cli/scaffold/defs.md) — adding new definitions (assets, schedules, sensors, components) to a project
- [dg utilities](./references/cli/utils/INDEX.md) — dg utils, inspecting component types, viewing integrations, refreshing state-backed component cache
- [Creating Components](./references/components/creating-components.md) — building a new custom component from scratch
- [Designing Component Integrations](./references/components/designing-component-integrations.md) — designing a component that wraps an external service or tool; custom integrations
- [Resolved Framework](./references/components/resolved-framework.md) — defining custom YAML schema types using Resolver, Model, or Resolvable
- [Subclassing Components](./references/components/subclassing-components.md) — extending an existing component via subclassing; customize dagster integration component
- [Template Variables](./references/components/template-variables.md) — using Jinja2 template variables in component YAML (env, dg, context, or custom scopes)
- [Creating State-Backed Components](./references/components/state-backed/creating.md) — building a component that fetches and caches external state
- [Using State-Backed Components](./references/components/state-backed/using.md) — managing state-backed components in production, CI/CD, or refreshing state
- [Integration libraries index for 40+ tools and technologies (dbt, Fivetran, Snowflake, AWS, etc.).](./references/integrations/INDEX.md) — integration, external tool, dagster-\*; dbt, fivetran, airbyte, snowflake, bigquery, sling, aws, gcp
- [Migration Guides](./references/migration/INDEX.md) — migrating sensors to declarative automation, upgrading to modern Dagster patterns
<!-- END GENERATED INDEX -->

登录后下载/点赞/收藏 ❤ 108 | ★ 0

dagster-expert

目录结构

SKILL.md

举报内容

提示