python - CeleryBeat Process consumes all OS memory -

we using

django-celery==3.1.10 celery==3.1.20 python 2.7.13 

we have written customdatabasescheduler schedule task, schedules task on time. running celerybeat process init script, celerybeat consumes full memory of system i.e. 24gb in day.

i tried run pmap on celerybeat process, shows [anon] has took memory.

can please debug , fix this.

first of if on django 1.8 or above please use celery 4.0 , above. in case not require django-celery. in case follow tutorial. http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

coming problem may 1 of following reasons:

1. your workers overloaded. try using concurrency mentioned here
2. check if django settings debug set true. can cause memory leakages , celery advises against when run it.
3. check memory leak fixes in history. me seemed next version after yours, 3.1.21 precise has memory leak fixes. try upgrading latest 3.x version(only if cannot use 4.x reason.)
4. if else fails, try monitoring tools debug scenario happening. monitoring tools mentioned here.