Installation
Use the package manager pip to install data-pipelines-cli:
pip install data-pipelines-cli[<flags>]
Depending on the systems that you want to integrate with you need to provide different flags in square brackets. You can provide comma separate list of flags, for example:
pip install data-pipelines-cli[gcs,git,bigquery]
Depending on the data storage you have you can use:
bigquery
snowflake
redshift
postgres
If you need git integration for loading packages published by other projects or publish them by yourself you will need:
git
If you want to deploy created artifacts (docker images and DataHub metadata) add the following flags:
docker
datahub
These are not usually used by a person user.
If you need Business Intelligence integration you can use following options:
looker