data_pipelines_cli.cli_commands package
Submodules
data_pipelines_cli.cli_commands.clean module
data_pipelines_cli.cli_commands.compile module
- compile_project(env: str, docker_repository_uri: Optional[str] = None, datahub_gms_uri: Optional[str] = None, docker_build: bool = False) None [source]
Create local working directories and build artifacts
- Parameters
env (str) – Name of the environment
docker_repository_uri (Optional[str]) – URI of the Docker repository
datahub_gms_uri (Optional[str]) – URI of the DataHub ingestion endpoint
docker_build (bool) – Whether to build a Docker image
- Raises
data_pipelines_cli.cli_commands.create module
- create(project_path: str, template_path: Optional[str]) None [source]
Create a new project using a template
- Parameters
project_path (str) – Path to a directory to create
template_path (Optional[str]) – Path or URI to the repository of the project template
- Raises
DataPipelinesError – no template found in .dp.yml config file
data_pipelines_cli.cli_commands.deploy module
- class DeployCommand(docker_push: Optional[str], blob_address: str, provider_kwargs_dict: Optional[Dict[str, Any]], datahub_ingest: bool)[source]
Bases:
object
A class used to push and deploy the project to the remote machine
- blob_address_path: str
URI of the cloud storage to send build artifacts to
- datahub_ingest: bool
Whether to ingest DataHub metadata
- deploy() None [source]
Push and deploy the project to the remote machine
- Raises
DependencyNotInstalledError – DataHub or Docker not installed
DataPipelinesError – Error while pushing Docker image
- docker_args: Optional[data_pipelines_cli.data_structures.DockerArgs]
Arguments required by the Docker to make a push to the repository. If set to None,
deploy()
will not make a push
- provider_kwargs_dict: Dict[str, Any]
Dictionary of arguments required by a specific cloud storage provider, e.g. path to a token, username, password, etc.
data_pipelines_cli.cli_commands.init module
- init(config_path: Optional[str]) None [source]
Configure the tool for the first time
- Parameters
config_path (Optional[str]) – URI of the repository with a template of the config file
- Raises
DataPipelinesError – user do not want to overwrite existing config file