Changelog
Unreleased
0.14.0 - 2022-02-02
0.13.0 - 2022-02-01
0.12.0 - 2022-01-31
dp publish
will push generated sources to external git repo
0.11.0 - 2022-01-18
Added
dp update
commanddp publish
command for creation of dbt package out of the project.
Changed
Docker response in
deploy
andcompile
gets printed as processed strings instead of plain dictionaries.dp compile
parses content ofdatahub.yml
and replaces Jinja variables in the form ofvar
orenv_var
.dags_path
is read from an envedairflow.yml
file.
0.10.0 - 2022-01-12
Changed
Run
dbt deps
at the end ofdp prepare-env
.
Fixed
dp run
anddp test
are no longer pointing toprofiles.yml
instead of the directory containing it.
0.9.0 - 2022-01-03
Added
--env
flag todp deploy
.
Changed
Docker repository URI gets read out of
build/config/{env}/k8s.yml
.
Removed
--docker-repository-uri
and--datahub-gms-uri
fromdp compile
anddp deploy
commands.dp compile
no longer replaces<INGEST_ENDPOINT>
indatahub.yml
, or<DOCKER_REPOSITORY_URL>
ink8s.yml
0.8.0 - 2021-12-31
Changed
dp init
anddp create
automatically adds.git
suffix to given template paths, if necessary.When reading dbt variables, global-scoped variables take precedence over project-scoped ones (it was another way around before).
Address argument for
dp deploy
is no longer mandatory. It should be either placed inairflow.yml
file as value ofdags_path
key, or provided with--dags-path
flag.
0.7.0 - 2021-12-29
Added
Add documentation in the style of Read the Docs.
Exception classes in
errors.py
, deriving fromDataPipelinesError
base exception class.Unit tests to massively improve code coverage.
--version
flag to dp command.Add
dp prepare-env
command that prepares local environment for standalone dbt (right now, it only generates and savesprofiles.yml
in$HOME/.dbt
).
Changed
dp compile
:--env
option has a default value:base
,--datahub
is changed to--datahub-gms-uri
,--repository
is changed to--docker-repository-uri
.
dp deploy
’s--docker-push
is not a flag anymore and requires a Docker repository URI parameter;--repository
got removed then.dp run
anddp test
rundp compile
before actual dbt command.Functions raise exceptions instead of exiting using
sys.exit(1)
;cli.cli()
entrypoint is expecting exception and exits only there.dp deploy
raises an exception if there is no Docker image to push orbuild/config/dag
directory does not exist.Rename
gcp
togcs
in requirements (now one should runpip install data-pipelines-cli[gcs]
).
0.6.0 - 2021-12-16
Modified
dp saves generated
profiles.yml
in eitherbuild/local
orbuild/env_execution
directories. dbt gets executed withenv_execution
as the target.
0.5.1 - 2021-12-14
Fixed
_dbt_compile
is no longer removing replaced<IMAGE_TAG>
.
0.5.0 - 2021-12-14
Added
echo_warning
function prints warning messages in yellow/orange color.
Modified
Docker image gets built at the end of
compile
command.dbt-related commands do not fail if no
$HOME/.dp.yml
exists (e.g.,dp run
).
Removed
Dropped
dbt-airflow-manifest-parser
dependency.
0.4.0 - 2021-12-13
Added
dp run
anddp test
commands.dp clean
command for removingbuild
andtarget
directories.File synchronization tests for Google Cloud Storage using
gcp-storage-emulator
.Read vars from config files (
$HOME/.dp.yml
,config/$ENV/dbt.yml
) and pass todbt
.
Modified
profiles.yml
gets generated and saved inbuild
directory indp compile
, instead of relying on a local one in the main project directory.dp dbt <command>
generatesprofiles.yml
inbuild
directory by default.dp init
is expectingconfig_path
argument to download config template with the help of thecopier
and save it in$HOME/.dp.yml
.dp template list
is renamed asdp template-list
.dp create
allows for providing extra argument calledtemplate-path
, being either name of one of templates defined in.dp.yml
config file or direct link to Git repository.
Removed
Support for manually created
profiles.yml
in main project directory.dp template new
command.username
field from$HOME/.dp.yml
file.
0.3.0 - 2021-12-06
Run
dbt deps
alongside rest ofdbt
commands indp compile
0.2.0 - 2021-12-03
Add support for GCP and S3 syncing in
dp deploy
0.1.2 - 2021-12-02
Fix: do not use styled
click.secho
for Docker push response, as it may not be astr
0.1.1 - 2021-12-01
Fix Docker SDK for Python’s bug related to tagging, which prevented Docker from pushing images.
0.1.0 - 2021-12-01
Added
Draft of
dp init
,dp create
,dp template new
,dp template list
anddp dbt
Draft of
dp compile
anddp deploy