气流：芹菜工人连接太多

Question

我们正在运行气流1.10.1与芹菜。面对多个打开的连接。在DAG启动时，UI会挂起几分钟。

亮点：

• 所有节点都是裸金属的：CPU:40，MHz 2494.015，RAM 378G，10Gb NIC-

MySQL

+-------------------------+---------     + 
| Variable_name           | Value         |
+-------------------------+---------      +
| Thread pool_idle_threads | 0            |
| Thread pool_threads      | 0            |
| Threads_cached          | 775           |
| Threads_connected       | 5323          |
| Threads_created         | 4846609       |
| Threads_running         | 5             |
+-------------------------+---------      +

MySQL连接：

31  - worker1
215 - worker2
349 - worker53
335 - worker54
347 - worker55
336 - worker56
336 - worker57
354 - worker58
339 - worker59
328 - worker60
333 - worker61
337 - worker62
2   - scheduler

Worker.cfg

[core]
sql_alchemy_pool_size = 5
sql_alchemy_pool_recycle = 900
sql_alchemy_reconnect_timeout = 300
parallelism = 1200
dag_concurrency = 800
non_pooled_task_slot_count = 1200
max_active_runs_per_dag = 10
dagbag_import_timeout = 30
[celery]
worker_concurrency = 100

Scheduler.cfg：

   [core]
    sql_alchemy_pool_size = 30
    sql_alchemy_pool_recycle = 300
    sql_alchemy_reconnect_timeout = 300
    parallelism = 1200
    dag_concurrency = 800
    non_pooled_task_slot_count = 1200
    max_active_runs_per_dag = 10
    [scheduler]
    job_heartbeat_sec = 5
    scheduler_heartbeat_sec = 5
    run_duration = 1800
    min_file_process_interval = 10
    min_file_parsing_loop_time = 1
    dag_dir_list_interval = 300
    print_stats_interval = 30
    scheduler_zombie_task_threshold = 300
    max_tis_per_query = 1024
    max_threads = 29

另外，我正在运行1000个简单任务，比如sleep或ls

Answer 1

我们可以把连接从700-800切换到1-10

你可以做两件事：

1. setsql_alchemy_pool_enabled=False
2. 设置一个不同的result_backend从DB，在我们的情况下，我们使用redis作为result_backend和MySQL作为主DB