Algorithms and implementations to participate in Kaggle YouTube-8M Video Understanding Competition
This PR fixes an issue when canceling BigQuery jobs. The issue arises when the service account is in a Google Cloud project A but the job runs in another project B. The default project and location won't work anymore.
Fix Vertex AI Custom Job training issue (#25367)
Remove extraneous word in installation guide (#25371)
Removes a single stray word from the installation instructions
updated documentation for databricks operator (#24599)
Remove unnecessary asset compilaton for prod images (#25374)
When prod image is built, we install airflow from packages and asset compilation happens as part of the package preparation.
There is no need whatsoever to repeat it here for PROD images (it is still needed for CI images though).
add downstream events to task instances (#25375)
Restore pushing CI image as latest to GHCR.io (#25380)
We stopped pushing latest CI/PROD images to ghcr.io when we started to run multiplatform builds and switched to --cache-from option of buildx. Hoever there are still some workflows that might require the latest image (for example Codespaces) and generally speaking someone could just pull the image if they are curious.
This PR adds back pushing the image (only "linux/amd" during image cache preparation.
fix - resolve bash by absolute path (#25331)
Co-authored-by: Matt Rixman MatrixManAtYrService@users.noreply.github.com
Convert ECS Fargate Sample DAG to System Test (#25316)
fix: change disable_verify_ssl behaviour (#25023)
The problem is that verify_ssl is overwritten by the configuration from the kube_config or load_incluster_config file.
Translate system tests migration (AIP-47) (#25340)
Use newer kubernetes authentication method in internal vault client (#25351)
This gets rid of the hvac deprecation warning about
being removed in version 1.0 that is currently printed when using this
Memorystore assets & system tests migration (AIP-47) (#25361)
Check expand_kwargs() input type before unmapping (#25355)
Change stdout and stderr access mode to append in commands (#25253)
Add missing import in best-practices code example (#25391)
This PR adds a missing import to the "unit test for a custom operator" code example. While this code example won't run on its own any way since
MyCustomOperator isn't defined (and probably shouldn't be for simplicity), I noticed when applying this code to my own custom operator that an import for
DAG was missing. This PR adds that back in, so the only missing import is for the user-added custom operator.
YandexCloud provider: Support new Yandex SDK features for DataProc (#25158)
Fix datasets list page (#25382)
add link to dag dependencies from datasets
fix dataset selection
Adjust limits when constructing cross-provider dependencies (#25364)
When we are constructing cross-provider dependencies, we should adjust the limits to account for some apache-airflow packages that are not yet released.
The adjustment is to add ".*" after the version number when
dependency is lower-bound with
>=. It's not explicitly mentioned in
PEP 440 - it is only mentioned there that it works for
equality, but since
>= is also part of
equal it also
Example is a common-sql package that might be limited to
in google provider as part of the change, but it might not yet be released
as it is being released together with the package it is needed by.
We need to adjust the version whenever in our providers we
refer to other providers with >= install clause because
pip only allows to install direct pre-release (and
development) dependencies but it does not modify requirements of
those package to also include pre-releases as transitive dependency.
Also, we need to remove the limits in
devel dependencies because
the packages are not yet released at the moment we use those
dependencies, so the limits in
devel should be removed, allowing
the developers to install Airflow without those devel packages.
Also a bug was found that would prevent to generate the dependencies in case provider.yaml file only changed (bad specification of .pre-commit include)
Move all "old" SQL operators to common.sql providers (#25350)
Previously, in #24836 we moved Hooks and added some new operators to the common.sql package. Now we are salso moving the operators and sensors to common.sql.
Add missing option when pushing latest images to cache :( (#25399)
The #25380 re-introduced pushing image as latest on main success and (as unfortunately happens) since this is only testable after merge, a small bug crippled in making main build fail.
This flag should fix the problem.