Installation

Use the package manager pip to install data-pipelines-cli:

pip install data-pipelines-cli[<flags>]

Depending on the systems that you want to integrate with you need to provide different flags in square brackets. You can provide comma separate list of flags, for example:

pip install data-pipelines-cli[gcs,git,bigquery]

Depending on the data storage you have you can use:

  • bigquery

  • snowflake

  • redshift

  • postgres

If you need git integration for loading packages published by other projects or publish them by yourself you will need:

  • git

If you want to deploy created artifacts (docker images and DataHub metadata) add the following flags:

  • docker

  • datahub

These are not usually used by a person user.

If you need Business Intelligence integration you can use following options:

  • looker