airflow-dbt | Apache Airflow integration for dbt | BPM library

by gocardless Python Version: 0.4.0 License: MIT

X-Ray Key Features Code Snippets Community Discussions(2)Vulnerabilities Install Support

kandi X-RAY | airflow-dbt Summary

airflow-dbt is a Python library typically used in Automation, BPM applications. airflow-dbt has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has low support. You can install using 'pip install airflow-dbt' or download it from GitHub, PyPI.

Apache Airflow integration for dbt

Support

Quality

Security

License

Reuse

Support

airflow-dbt has a low active ecosystem.

It has 296 star(s) with 56 fork(s). There are 61 watchers for this library.

It had no major release in the last 12 months.

There are 23 open issues and 13 have been closed. On average issues are closed in 85 days. There are 7 open pull requests and 0 closed requests.

It has a neutral sentiment in the developer community.

The latest version of airflow-dbt is 0.4.0

Quality

airflow-dbt has 0 bugs and 4 code smells.

Security

airflow-dbt has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

airflow-dbt code analysis shows 0 unresolved vulnerabilities.

There are 1 security hotspots that need review.

License

airflow-dbt is licensed under the MIT License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

airflow-dbt releases are not available. You will need to build from source code and install.

Deployable package is available in PyPI.

Build file is available. You can build the component from source.

Installation instructions, examples and code snippets are available.

airflow-dbt saves you 132 person hours of effort in developing the same functionality from scratch.

It has 378 lines of code, 33 functions and 10 files.

It has low code complexity. Code complexity directly impacts maintainability of the code.

Top functions reviewed by kandi - BETA

kandi has reviewed airflow-dbt and discovered the below as its top functions. This is intended to give you an instant insight into airflow-dbt implemented functionality, and help decide if they suit your requirements.

Runs the script
Run the dbt command
Create the DbtHook instance
Dump the variables in JSON format
Builds the distribution
Print a status message

Get all kandi verified functions for this library.

airflow-dbt Key Features

No Key Features are available at this moment for airflow-dbt.

airflow-dbt Examples and Code Snippets

No Code Snippets are available at this moment for airflow-dbt.

Community Discussions

Trending Discussions on airflow-dbt

Generate dbt documentation from Google Cloud Composer

How to run DBT in airflow without copying our repo

QUESTION

Generate dbt documentation from Google Cloud Composer

Asked 2022-Jan-03 at 00:50

I have a dbt project running on Cloud Composer and all my models and snapshots are running sucessfully.

I'm having trouble generating the documentation once all the processing is finished.

The integration between dbt and cloud composer is done via airflow-dbt and I have setup a task for the DbtDocsGenerateOperator. The DAG actually runs fine, and I can see in the log that the catalog.json file is being written to the target folder in the correspondent cloud bucket, but the file is not there.

Doing some investigation on the GCP logging, I've notice that there's a process called gcs-syncd that is apparently removing the file.

Wondering if anyone has had success in this integration before and was able to generate the dbt docs from cloud composer?

...

ANSWER

Answered 2022-Jan-03 at 00:50

The problem here is that you're writing your catalog file to a location on a worker node that is mounted to the dags folder in gcs, which airflow and cloud composer manages. Per the documentation,

When you modify DAGs or plugins in the Cloud Storage bucket, Cloud Composer synchronizes the data across all the nodes in the cluster.

Cloud Composer synchronizes the dags/ and plugins/ folders uni-directionally by copying locally. Unidirectional synching means that local changes in these folders are overwritten.

The data/ and logs/ folders synchronize bi-directionally by using Cloud >Storage FUSE.

If you change the location of this file to /home/airflow/gcs/data/target/catalog.json, you should be fine as that syncs bi-directionally

Source https://stackoverflow.com/questions/69967480

QUESTION

How to run DBT in airflow without copying our repo

Asked 2020-Dec-23 at 11:51

We use DBT with GCP and BigQuery for transformations in BigQuery, and the simplest approach to scheduling our daily run dbt seems to be a BashOperator in Airflow. Currently we have two separate directories / github projects, one for DBT and another for Airflow. To schedule DBT to run with Airflow, it seems like our entire DBT project would need to be nested inside of our Airflow project, that way we can point to it for our dbt run bash command?

Is it possible to trigger our dbt run and dbt test without moving our DBT directory inside of our Airflow directory? With the airflow-dbt package, for the dir in the default_args, maybe it is possible to point to the gibhub link for the DBT project here?

...

ANSWER

Answered 2020-Dec-23 at 11:51

My advice would be to leave your dbt and airflow codebases separated. There is indeed a better way:

dockerise your dbt project in a simple python-based image where you COPY the codebase
push that to DockerHub or ECR or any other docker repository that you are using
use the DockerOperator in your airflow DAG to run that docker image with your dbt code

I'm assuming that you use the airflow LocalExecutor here and that you want to execute your dbt run workload on the server where airflow is running. If that's not the case and that you have access to a Kubernetes cluster, I would suggest instead to use the KubernetesPodOperator.

Source https://stackoverflow.com/questions/64890144

Community Discussions, Code Snippets contain sources that include Stack Exchange Network