data_pipelines_cli.cli_commands package
Submodules
data_pipelines_cli.cli_commands.clean module
data_pipelines_cli.cli_commands.compile module
- compile_project(env: str, docker_build: bool = False) None [source]
Create local working directories and build artifacts.
- Parameters
env (str) – Name of the environment
docker_build (bool) – Whether to build a Docker image
- Raises
- replace_image_settings(docker_args: data_pipelines_cli.data_structures.DockerArgs) None [source]
data_pipelines_cli.cli_commands.create module
- create(project_path: str, template_path: Optional[str]) None [source]
Create a new project using a template.
- Parameters
project_path (str) – Path to a directory to create
template_path (Optional[str]) – Path or URI to the repository of the project template
- Raises
DataPipelinesError – no template found in .dp.yml config file
data_pipelines_cli.cli_commands.deploy module
- class DeployCommand(env: str, docker_push: bool, dags_path: Optional[str], provider_kwargs_dict: Optional[Dict[str, Any]], datahub_ingest: bool)[source]
Bases:
object
A class used to push and deploy the project to the remote machine.
- blob_address_path: str
URI of the cloud storage to send build artifacts to
- datahub_ingest: bool
Whether to ingest DataHub metadata
- deploy() None [source]
Push and deploy the project to the remote machine.
- Raises
DependencyNotInstalledError – DataHub or Docker not installed
DataPipelinesError – Error while pushing Docker image
- docker_args: Optional[data_pipelines_cli.data_structures.DockerArgs]
Arguments required by the Docker to make a push to the repository. If set to None,
deploy()
will not make a push
- provider_kwargs_dict: Dict[str, Any]
Dictionary of arguments required by a specific cloud storage provider, e.g. path to a token, username, password, etc.
data_pipelines_cli.cli_commands.init module
- init(config_path: Optional[str]) None [source]
Configure the tool for the first time.
- Parameters
config_path (Optional[str]) – URI of the repository with a template of the config file
- Raises
DataPipelinesError – user do not want to overwrite existing config file
data_pipelines_cli.cli_commands.prepare_env module
- prepare_env(env: str) None [source]
Prepare local environment for use with dbt-related applications.
Prepare local environment for use with applications expecting a “traditional” dbt structure, such as plugins to VS Code. If in doubt, use
dp run
anddp test
instead.- Parameters
env (str) – Name of the environment