Cloud Composer1|Cloud Composer2
Apache Airflow includesa web user interface (UI) that you can use to manage workflows (DAGs), manage the Airflow environment,and perform administrative actions. For example, you can use the web interfaceto review the progress of a DAG, set up a new data connection, or review logsfrom previous DAG runs.
Airflow web server
Each Cloud Composer environment has a web server thatruns the Airflow web interface. The web server is a part ofCloud Composer environment architecture.
Before you begin
You must have a role that can view Cloud Composer environments.For more information, see Access control.
During the environment creation, Cloud Composer configures theURL for the web server that runs the Airflow web interface. The URL isnon-customizable.
- Cloud Composer2 supports the Airflow UI Access Control(Airflow Role-Based Access Control) feature for the Airflow web interface.
Accessing the Airflow web interface
The Airflow web server runsas a workload in your environment's cluster. The web server isdeployed to the composer.googleusercontent.com domain and provides access tothe Airflow web interface.
Cloud Composer2 provides access to the interface based on user identitiesand IAM policy bindings defined for users. Compared toCloud Composer1, Cloud Composer2 uses a different mechanism that does notrely on Identity-Aware Proxy.
Accessing the web interface from the Google Cloud console
To access the Airflow web interface from the Google Cloud console:
In the Google Cloud console, go to the Environments page.
Go to Environments
In the Airflow webserver column, follow the Airflow link for your environment.
Limiting access to the Airflow web server
Composer environments let you limit access to the Airflow web server.
You can block all access, or allow access from specific IPv4 or IPv6 external IP ranges.
Currently you cannot configure the allowed IP ranges using private IPaddresses.
Retrieving the web interface URL via the gcloud command-line tool
You can access the Airflow web interface from any web browser. To get the URLfor the web interface, enter the following gcloud command:
gcloud composer environments describe ENVIRONMENT_NAME \ --location LOCATION
Replace the following:
ENVIRONMENT_NAME: the name of your environment.LOCATION: the region where the environment is located.
The gcloud command shows the properties of a Cloud Composerenvironment, including the URLs for the web interface. The URLs arelisted as airflowUri and airflowByoidUri:
config: airflowUri: https://example-dot-us-central1.composer.googleusercontent.com airflowByoidUri: https://example-dot-us-central1.composer.byoid.googleusercontent.com
Restarting the web server
When debugging or troubleshooting Cloud Composer environments, some issuesmay be resolved by restarting the Airflow web server. You can restart the webserver using the restartWebServer APIor the restart-web-server gcloud command:
gcloud beta composer environments restart-web-server ENVIRONMENT_NAME \ --location=LOCATION
FAQs
Airflow provides various developer tools and integration capabilities such as REST APIs, Python SDK, and Jupyter Notebook integration. Google Cloud Composer provides an intuitive web-based interface, and users can also use command-line tools such as gcloud and Cloud SDK.
What is the difference between Apache Airflow and Cloud Composer? ›
Airflow provides various developer tools and integration capabilities such as REST APIs, Python SDK, and Jupyter Notebook integration. Google Cloud Composer provides an intuitive web-based interface, and users can also use command-line tools such as gcloud and Cloud SDK.
How do I connect Airflow to Google cloud? ›
Google Cloud Platform Connection
- Use Application Default Credentials, such as via the metadata server when running on Google Compute Engine.
- Use a service account key file (JSON format) on disk - Keyfile Path .
- Use a service account key file (JSON format) from connection configuration - Keyfile JSON .
Logging into Apache Airflow
- Open the Environments page on the Amazon MWAA console.
- Choose an environment.
- Choose Open Airflow UI.
Cloud Composer helps you create managed Airflow environments quickly and use Airflow-native tools, such as the powerful Airflow web interface and command-line tools, so you can focus on your workflows and not your infrastructure.
What are the three main features of Apache Airflow? ›
Principles of Airflow
- Dynamic: DAGs are written in Python, allowing for dynamic pipeline creation.
- Extensible: Easily create your own operators, executors, and libraries.
- Elegant: Airflow DAGs are lean and explicit. ...
- Scalable: Airflow is modular and uses a message queue to manage workers that execute tasks.
Learn More. Apache Airflow has become the most popular platform for data engineering, pipeline management, and workflow automation since its genesis as an internal data management tool at Airbnb a decade ago.
How do I connect to Airflow UI? ›
Creating a Connection with the UI
Open the Admin->Connections section of the UI. Click the Create link to create a new connection. Fill in the Connection Id field with the desired connection ID.
What is the difference between Cloud Composer 1 and 2? ›
Cloud Composer 1 environments are zonal. Cloud Composer 2 environments have a zonal Airflow Metadata DB and a regional Airflow scheduling & execution layer. Airflow schedulers, workers and web servers run in the Airflow execution layer.
How do I open Airflow UI locally? ›
Access the Airflow UI: Visit localhost:8080 in your browser and log in with the admin account details shown in the terminal. Enable the example_bash_operator DAG in the home page.
Accessing the web interface
The webserver is available at: http://localhost:8080 . The default account has the login airflow and the password airflow .
How to check if Airflow webserver is running? ›
Webserver Health Check Endpoint
To check the health status of your Airflow instance, you can simply access the endpoint /health . It will return a JSON object in which a high-level glance is provided.
When to use Cloud Composer? ›
Cloud Composer is best for batch workloads that can handle a few seconds of latency between task executions. You can use Cloud Composer to orchestrate services in your data pipelines, such as triggering a job in BigQuery or starting a Dataflow pipeline.
What is Airflow in Google cloud? ›
Airflow is a platform created by the community to programmatically author, schedule and monitor workflows.
Is Apache Airflow cloud based? ›
Airflow™ provides many plug-and-play operators that are ready to execute your tasks on Google Cloud Platform, Amazon Web Services, Microsoft Azure and many other third-party services. This makes Airflow easy to apply to current infrastructure and extend to next-gen technologies.
What is the difference between Kafka and Airflow? ›
Airflow is ideal for batch processing and workflow orchestration, with capabilities for scheduling and monitoring complex data pipelines. Kafka is designed for continuous data ingestion and processing, making it more suitable for real-time scenarios.
What is the difference between Cloud Composer and dataflow? ›
Cloud Composer is a cross platform orchestration tool that supports AWS, Azure and GCP (and more) with management, scheduling and processing abilities. Cloud Dataflow handles tasks.
What is the purpose of Apache Airflow? ›
Apache Airflow is an open-source tool to programmatically author, schedule, and monitor workflows. It is one of the most robust platforms used by Data Engineers for orchestrating workflows or pipelines. You can easily visualize your data pipelines' dependencies, progress, logs, code, trigger tasks, and success status.
What is the difference between Apache NiFi and Apache Airflow? ›
NiFi is ideal for real-time data ingestion, processing, and distribution scenarios. It excels in scenarios where data lineage and immediate data processing are required. Airflow is tailored for batch processing and is more suited for scheduled execution of complex workflows.
