You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have set up a multi node Airflow configuration. We have the below setup.
4 nodes
node 1: webserver, rabbitmq, flower
node 2: meta db
node 3: worker 1
node 4: worker 2
Although the setup is working fine and DAGs are getting distributed among the worker nodes, however the logs generated in the worker nodes are not reflecting in the webserver UI in the master node.
Below is the error:
***** Log file does not exist: /home/dvafadm/airflow/logs/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log
*** Fetching from: http://:8793/log/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log
*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd) !!!!!
****** See more at https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key
****** Failed to fetch log file from worker. Client error '403 FORBIDDEN' for url 'http://:8793/log/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log'
For more information check: https://httpstatuses.com/403**
We have checked all the folder permission, secret_key, timezone but still the problem persist.
Can you see if there are any files in the directory '/home/dvafadm/airflow/logs/' and if so, can you find the ones corresponding the the above DAG and list the file/directory structure? I'm curious if this is a name mismatch or if the logs just don't exist.
@jledoux-sonergia - The 2.4.0 is out today and It should have the IPV6 problem fixed. @devashish-parmar - if the request is forbidden, you should have looked at the webeserver logs and it should tell you the exact reason for the problem you have
We have checked all the folder permission, secret_key, timezone but still the problem persist.
Timezone is not an issue. You should check if TIME is synchronized on all your machines.
Converting that into a discussion as either some of the problems mentioned here (IPV6) have been fixed in 2.4.0 or they need more looking at the deployment side and checking the webserver logs.
Apache Airflow version
2.3.4
What happened
We have set up a multi node Airflow configuration. We have the below setup.
Although the setup is working fine and DAGs are getting distributed among the worker nodes, however the logs generated in the worker nodes are not reflecting in the webserver UI in the master node.
Below is the error:
***** Log file does not exist: /home/dvafadm/airflow/logs/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log
*** Fetching from: http://:8793/log/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log
*** !!!! Please make sure that all your Airflow components (e.g. schedulers, webservers and workers) have the same 'secret_key' configured in 'webserver' section and time is synchronized on all your machines (for example with ntpd) !!!!!
****** See more at https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key
****** Failed to fetch log file from worker. Client error '403 FORBIDDEN' for url 'http://:8793/log/dag_id=second_dag/run_id=manual__2022-09-07T10:16:57.050200+00:00/task_id=second_dag/attempt=1.log'
For more information check: https://httpstatuses.com/403**
We have checked all the folder permission, secret_key, timezone but still the problem persist.
can anyone help here?
What you think should happen instead
No response
How to reproduce
No response
Operating System
Linux Red Hat
Versions of Apache Airflow Providers
No response
Deployment
Virtualenv installation
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: