当前位置: 首页 > 面试题库 >

Django Celery Elastic Beanstalk主管无此处理错误

傅振濂
2023-03-14
问题内容

我的celery_config.txt在.ebextensions中的文件脚本中

#!/usr/bin/env bash

# Get django environment variables
celeryenv=`cat /opt/python/current/env | tr '\n' ',' | sed 's/export //g' | sed 's/$PATH/%(ENV_PATH)s/g' | sed 's/$PYTHONPATH//g' | sed 's/$LD_LIBRARY_PATH//g'`
celeryenv=${celeryenv%?}

# Create celery configuraiton script
celeryconf="[program:celeryd-worker]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A wellfie --loglevel=INFO

directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-worker.log
stderr_logfile=/var/log/celery-worker.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

environment=$celeryenv

[program:celeryd-beat]
; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery beat -A wellfie --loglevel=INFO --workdir=/tmp -S django

directory=/opt/python/current/app
user=nobody
numprocs=1
stdout_logfile=/var/log/celery-beat.log
stderr_logfile=/var/log/celery-beat.log
autostart=true
autorestart=true
startsecs=10

; Need to wait for currently executing tasks to finish at shutdown.
; Increase this if you have very long running tasks.
stopwaitsecs = 600

; When resorting to send SIGKILL to the program to terminate it
; send SIGKILL to its whole process group instead,
; taking care of its children as well.
killasgroup=true

; if rabbitmq is supervised, set its priority higher
; so it starts first
priority=998

environment=$celeryenv"

# Create the celery supervisord conf script
echo "$celeryconf" | tee /opt/python/etc/celery.conf

# Add configuration script to supervisord conf (if not there already)
if ! grep -Fxq "[include]" /opt/python/etc/supervisord.conf
  then
  echo "[include]" | tee -a /opt/python/etc/supervisord.conf
  echo "files: celery.conf" | tee -a /opt/python/etc/supervisord.conf
fi

# Reread the supervisord config
supervisorctl -c /opt/python/etc/supervisord.conf reread

# Update supervisord in cache without restarting all services
supervisorctl -c /opt/python/etc/supervisord.conf update

# Start/Restart celeryd through supervisord
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd

我的celery-worker.log

[2017-04-18 18:48:06,042: CRITICAL/MainProcess] Unrecoverable error: ImportError('No module named xxxx.toi3pr.0001.usw2.cache.amazonaws.com',)
Traceback (most recent call last):
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bootsteps.py", line 115, in start
    self.on_start()
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/apps/worker.py", line 143, in on_start
    self.emit_banner()
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/apps/worker.py", line 158, in emit_banner
    ' \n', self.startup_info(artlines=not use_image))),
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/apps/worker.py", line 221, in startup_info
    results=self.app.backend.as_uri(),
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/app/base.py", line 1182, in backend
    return self._get_backend()
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/app/base.py", line 900, in _get_backend
    self.loader)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/app/backends.py", line 65, in by_url
    return by_name(backend, loader), url
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/app/backends.py", line 45, in by_name
    cls = symbol_by_name(backend, aliases)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named xxxx.toi3pr.0001.usw2.cache.amazonaws.com

我的celery-beat.log

ImportError: No module named wellfie
Traceback (most recent call last):
  File "/opt/python/run/venv/bin/celery", line 9, in <module>
    load_entry_point('celery==4.0.2', 'console_scripts', 'celery')()
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/__main__.py", line 14, in main
    _main()
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/celery.py", line 326, in main
    cmd.execute_from_commandline(argv)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/base.py", line 279, in execute_from_commandline
    argv = self.setup_app_from_commandline(argv)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/base.py", line 481, in setup_app_from_commandline
    self.app = self.find_app(app)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/base.py", line 503, in find_app
    return find_app(app, symbol_by_name=self.symbol_by_name)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/app/utils.py", line 355, in find_app
    sym = symbol_by_name(app, imp=imp)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/bin/base.py", line 506, in symbol_by_name
    return imports.symbol_by_name(name, imp=imp)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/opt/python/run/venv/local/lib/python2.7/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
    return imp(module, package=package)
  File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named wellfie

我的eb-activity.log

[2017-04-18T19:50:23.354Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployEnactHook/01flip.py] : Starting activity...
[2017-04-18T19:50:28.095Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployEnactHook/01flip.py] : Completed activity. Result:
  httpd: stopped
  httpd: started
  httpd                            RUNNING   pid 11508, uptime 0:00:03
[2017-04-18T19:50:28.095Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployEnactHook/02create_pids_for_monitoring.sh] : Starting activity...
[2017-04-18T19:50:28.195Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployEnactHook/02create_pids_for_monitoring.sh] : Completed activity. Result:
  + chmod 0755 /var/run/httpd
  + /opt/elasticbeanstalk/bin/healthd-track-pidfile --proxy httpd
  + /opt/elasticbeanstalk/bin/healthd-track-pidfile --name application --location /opt/python/run/supervisord.pid
[2017-04-18T19:50:28.195Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployEnactHook] : Completed activity. Result:
  Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/enact.
[2017-04-18T19:50:28.195Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployPostHook] : Starting activity...
[2017-04-18T19:50:28.195Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Starting activity...
[2017-04-18T19:50:28.487Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployPostHook/run_supervised_celeryd.sh] : Completed activity. Result:
  [program:celeryd-worker]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery worker -A wellfie --loglevel=INFO

  directory=/opt/python/current/app
  user=nobody
  numprocs=1
  stdout_logfile=/var/log/celery-worker.log
  stderr_logfile=/var/log/celery-worker.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",RDS_PORT="5432",PYTHONPATH="/opt/python/current/app/wellfie:",REDIS_URL="bbbbb.bbbbb.0001.usw2.cache.amazonaws.com:6379",RDS_DB_NAME="bbbdb",DJANGO_SETTINGS_MODULE="wellfie.settings",RDS_USERNAME="aaaa",RDS_PASSWORD="bbbb",RDS_HOSTNAME="bbbbb.bbbbb.us-west-2.rds.amazonaws.com"

  [program:celeryd-beat]
  ; Set full path to celery program if using virtualenv
  command=/opt/python/run/venv/bin/celery beat -A wellfie --loglevel=INFO --workdir=/tmp -S django

  directory=/opt/python/current/app
  user=nobody
  numprocs=1
  stdout_logfile=/var/log/celery-beat.log
  stderr_logfile=/var/log/celery-beat.log
  autostart=true
  autorestart=true
  startsecs=10

  ; Need to wait for currently executing tasks to finish at shutdown.
  ; Increase this if you have very long running tasks.
  stopwaitsecs = 600

  ; When resorting to send SIGKILL to the program to terminate it
  ; send SIGKILL to its whole process group instead,
  ; taking care of its children as well.
  killasgroup=true

  ; if rabbitmq is supervised, set its priority higher
  ; so it starts first
  priority=998

  environment=PYTHONPATH="/opt/python/current/app/:",PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",RDS_PORT="5432",PYTHONPATH="/opt/python/current/app/wellfie:",REDIS_URL="bbbbb.bbbbb.0001.usw2.cache.amazonaws.com:6379",RDS_DB_NAME="bbbdb",DJANGO_SETTINGS_MODULE="wellfie.settings",RDS_USERNAME="aaaa",RDS_PASSWORD="bbbb",RDS_HOSTNAME="bbbbb.bbbbb.us-west-2.rds.amazonaws.com"
  No config updates to processes
  celeryd: ERROR (no such process)
  celeryd: ERROR (no such process)
[2017-04-18T19:50:28.487Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1/AppDeployPostHook] : Completed activity. Result:
  Successfully execute hooks in directory /opt/elasticbeanstalk/hooks/appdeploy/post.
[2017-04-18T19:50:28.487Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AppDeployStage1] : Completed activity. Result:
  Application version switch - Command CMD-AppDeploy stage 1 completed
[2017-04-18T19:50:28.487Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter] : Starting activity...
[2017-04-18T19:50:28.487Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigLogRotation] : Starting activity...
[2017-04-18T19:50:28.488Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigLogRotation/10-config.sh] : Starting activity...
[2017-04-18T19:50:28.563Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigLogRotation/10-config.sh] : Completed activity. Result:
  Disabled forced hourly log rotation.
[2017-04-18T19:50:28.564Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigLogRotation] : Completed activity. Result:
  Successfully execute hooks in directory /opt/elasticbeanstalk/addons/logpublish/hooks/config.
[2017-04-18T19:50:28.564Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigCWLAgent] : Starting activity...
[2017-04-18T19:50:28.564Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigCWLAgent/10-config.sh] : Starting activity...
[2017-04-18T19:50:28.677Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigCWLAgent/10-config.sh] : Completed activity. Result:
  Log streaming option setting is not specified, ignore cloudwatch logs setup.

  Disabled log streaming.
[2017-04-18T19:50:28.678Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter/ConfigCWLAgent] : Completed activity. Result:
  Successfully execute hooks in directory /opt/elasticbeanstalk/addons/logstreaming/hooks/config.
[2017-04-18T19:50:28.678Z] INFO  [11290] - [Application update app-721b-170418_214950@45/AddonsAfter] : Completed activity.
[2017-04-18T19:50:28.678Z] INFO  [11290] - [Application update app-721b-170418_214950@45] : Completed activity. Result:
  Application update - Command CMD-AppDeploy succeeded
[2017-04-18T19:50:40.404Z] INFO  [11617] - [CMD-TailLogs] : Starting activity...
[2017-04-18T19:50:40.404Z] INFO  [11617] - [CMD-TailLogs/AddonsBefore] : Starting activity...
[2017-04-18T19:50:40.404Z] INFO  [11617] - [CMD-TailLogs/AddonsBefore] : Completed activity.
[2017-04-18T19:50:40.404Z] INFO  [11617] - [CMD-TailLogs/TailLogs] : Starting activity...
[2017-04-18T19:50:40.404Z] INFO  [11617] - [CMD-TailLogs/TailLogs/TailLogs] : Starting activity...

我不确定是什么引起了错误,但是这条特殊的线引起了我的注意celeryd: ERROR (no such process)。在本地,我的设置像一个咒语一样工作,我完成了所有任务,在运行工人/节拍时没有零问题,但是在生产中,主管似乎很难访问celery工人,或者celery工人无法正常启动。

我已经调试了很多,但是在找出问题所在时遇到了问题。我已经删除了变量和网址,它们似乎都还不错。如果我手动将ssh插入实例,则可以顺利运行celery
beat和celery worker,但是主管遇到了问题。

这条路径也正确吗?

; Set full path to celery program if using virtualenv
command=/opt/python/run/venv/bin/celery worker -A wellfie --loglevel=INFO

我是AWS新手,因此,如果此解决方案显而易见,我深表歉意。

编辑:

01_packages.config

packages:
  yum:
    git: []
    postgresql93-devel: []
    libcurl-devel: []

02_python.config

container_commands:
  04_celery_tasks:
    command: "cat .ebextensions/files/celery_configuration.txt > /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh && chmod 744 /opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
    leader_only: true
  05_celery_tasks_run:
    command: "/opt/elasticbeanstalk/hooks/appdeploy/post/run_supervised_celeryd.sh"
    leader_only: true

django.config

option_settings:
  aws:elasticbeanstalk:application:environment:
     DJANGO_SETTINGS_MODULE: "wellfie.settings"
     PYTHONPATH: "/opt/python/current/app/wellfie:$PYTHONPATH"

  aws:elasticbeanstalk:container:python:
    WSGIPath: wellfie/wsgi.py

问题答案:

是的,路径没问题:

command=/opt/python/run/venv/bin/celery worker -A wellfie --loglevel=INFO

我注意到您已将PYTHONPATH变量设置了两次,就像您将用eb环境属性覆盖一样?一旦将其设置在应用程序上,然后再设置在应用程序wellfie软件包上。

您的日志条目:

environment=
PYTHONPATH="/opt/python/current/app/:",
PATH="/opt/python/run/venv/bin/:%ENV_PATH)s",
RDS_PORT="5432",
PYTHONPATH="/opt/python/current/app/wellfie:",
REDIS_URL="bbbbb.bbbbb.0001.usw2.cache.amazonaws.com:6379",
RDS_DB_NAME="bbbdb",
DJANGO_SETTINGS_MODULE="wellfie.settings",
RDS_USERNAME="aaaa",
RDS_PASSWORD="bbbb",
RDS_HOSTNAME="bbbbb.bbbbb.us-west-2.rds.amazonaws.com"

我的:

environment=
PYTHONPATH="/opt/python/current/app/:",
PATH="/opt/python/run/venv/bin/:%(ENV_PATH)s",
WEBSERVER="APACHE",
CNAME="xxxxx-prod-env",
PYCURL_SSL_LIBRARY="nss",
APP_ENV="prod"

因此,我将从调查开始。要记住的另一件事是,如果您使用某些AWS服务(例如Redis),则需要使用该服务的安全组设置对其进行开放式连接,否则将无法连接。

注意事项:

  1. 如果您在实例上播放,请记住有关virtualenv的信息。因此,您可以运行以下命令:

cd / opt / python / current / app &&来源/ opt / python / run / venv / bin /
activate && python manage.py

  1. “ ImportError:没有名为’wellfie’的模块”可能会发生一次,并由于系统正在启动并且进程可能尚未就绪而被登录到celery-beat.log,celery-worker.log中。

  2. ImportError(’没有名为xxxx.toi3pr.0001.usw2.cache.amazonaws.com的模块’)看起来像python初始化问题。

修改文件 root_folder / django_app / init.py

from __future__ import absolute_import, unicode_literals

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from django_app.celery import app as celery_app

__all__ = ['celery_app']

解决方法是更改​​:

supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd

进入

supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-worker
supervisorctl -c /opt/python/etc/supervisord.conf restart celeryd-beat


 类似资料:
  • 生产者属性 消费者财产 消费者服务 生产性服务业 创建主题的生产者配置 生产商工作正常,但消费者给出的错误如下 组织中的值和/或键反序列化器中的“ErrorHandlingDeserializer”。springframework。Kafka。侦听器。DefaultErrorHandler。handleOtherException(DefaultErrorHandler.java:149)~[sp

  • 我正在尝试在MVVM中实现一个ViewModel,提供可观察的东西作为“输入流”,提供观察者作为“输出流”供视图绑定。

  • 我worte在python多重处理程序。我使用在子进程中共享列表。首先,我在主进程中添加了一些任务。然后,启动一些子进程来完成共享列表中的任务,子进程也将任务添加到共享列表中。但我得到了一个例外如下: 我发现了一些关于如何在python多处理中使用共享列表的内容,如下所示。但也有一些例外。我不知道例外的含义。公共列表和经理列表有什么区别? 代码如下:

  • 是否可以配置Spring批处理管理员来启动主作业和从作业。我们有一个进程作为主节点和3-4个从节点。 Spring batch admin在单独的JVM进程中运行,但所有Spring批处理作业都使用相同的批处理数据库模式。

  • 我需要检查是否一个url是有效的和空值是可接受的以及。我是这个图书馆的新手,被这个错误难倒了。

  • 问题内容: 我正在使用Python在Selenium中编写自动化测试。一种元素可能存在或可能不存在。我试图用下面的代码来处理它,当存在元素时它可以工作。但是脚本在元素不存在时失败,如果元素不存在,我想继续下一个语句。 错误- 问题答案: 您可以查看元素是否存在,然后单击(如果存在)。无需例外。注意中的复数“ s” 。