I’m trying to get to the bottom of a problem that’s causing our Zope instance to lock up with increasing regularity. I’ve installed Products.signalstackLogger to try to get to the bottom of it and it appears it’s happening when one of the content rules we have defined in Plone is trying to send an email.
For example:
Thread 46447504 (GET /VirtualHostBase/https/xxx.xxx.xxx:443/plone/VirtualHostRoot/news/portal_factory/News Item/news_item.2012-09-18.0763501726/edit):
File "/home/zope/home/parts/zope2/lib/python/ZServer/PubCore/ZServerPublisher.py", line 31, in __init__
response=b)
File "/home/zope/home/parts/zope2/lib/python/ZPublisher/Publish.py", line 401, in publish_module
environ, debug, request, response)
File "/home/zope/home/parts/zope2/lib/python/ZPublisher/Publish.py", line 202, in publish_module_standard
response = publish(request, module_name, after_list, debug=debug)
File "/home/zope/home/parts/zope2/lib/python/ZPublisher/Publish.py", line 119, in publish
request, bind=1)
File "/home/zope/home/parts/zope2/lib/python/ZPublisher/mapply.py", line 88, in mapply
if debug is not None: return debug(object,args,context)
File "/home/zope/home/parts/zope2/lib/python/ZPublisher/Publish.py", line 42, in call_object
result=apply(object,args) # Type s<cr> to step into published object.
File "/home/zope/home/eggs/Plone-3.3.6-py2.4.egg/Products/CMFPlone/FactoryTool.py", line 375, in __call__
temp_obj = tempFolder.__getitem__(id)
File "/home/zope/home/eggs/Plone-3.3.6-py2.4.egg/Products/CMFPlone/FactoryTool.py", line 155, in __getitem__
self.invokeFactory(id=id, type_name=type_name)
File "/home/zope/home/eggs/Plone-3.3.6-py2.4.egg/Products/CMFPlone/PloneFolder.py", line 348, in invokeFactory
new_id = pt.constructContent(*args, **kw)
File "/home/zope/home/eggs/Products.CMFCore-2.1.3-py2.4.egg/Products/CMFCore/TypesTool.py", line 716, in constructContent
ob = info.constructInstance(container, id, *args, **kw)
File "/home/zope/home/eggs/Products.CMFCore-2.1.3-py2.4.egg/Products/CMFCore/TypesTool.py", line 276, in constructInstance
ob = self._constructInstance(container, id, *args, **kw)
File "/home/zope/home/eggs/Products.CMFCore-2.1.3-py2.4.egg/Products/CMFCore/TypesTool.py", line 450, in _constructInstance
newid = m(id, *args, **kw)
File "<string>", line 7, in addATNewsItem
File "/home/zope/home/parts/zope2/lib/python/OFS/ObjectManager.py", line 349, in _setObject
notifyContainerModified(self)
File "/home/zope/home/parts/zope2/lib/python/zope/app/container/contained.py", line 365, in notifyContainerModified
notify(ContainerModifiedEvent(object, *descriptions))
File "/home/zope/home/parts/zope2/lib/python/zope/event/__init__.py", line 23, in notify
subscriber(event)
File "/home/zope/home/parts/zope2/lib/python/zope/component/event.py", line 26, in dispatch
for ignored in zope.component.subscribers(event, None):
File "/home/zope/home/parts/zope2/lib/python/zope/component/_api.py", line 130, in subscribers
return sitemanager.subscribers(objects, interface)
File "/home/zope/home/parts/zope2/lib/python/zope/component/registry.py", line 290, in subscribers
return self.adapters.subscribers(objects, provided)
File "/home/zope/home/parts/zope2/lib/python/zope/interface/adapter.py", line 535, in subscribers
subscription(*objects)
File "/home/zope/home/eggs/plone.app.contentrules-1.1.7-py2.4.egg/plone/app/contentrules/handlers.py", line 174, in modified
execute(aq_parent(aq_inner(event.object)), event)
File "/home/zope/home/eggs/plone.app.contentrules-1.1.7-py2.4.egg/plone/app/contentrules/handlers.py", line 97, in execute
executor(event, bubbled=True, rule_filter=rule_filter)
File "/home/zope/home/eggs/plone.contentrules-1.1.0-py2.4.egg/plone/contentrules/engine/executor.py", line 27, in __call__
executable()
File "/home/zope/home/eggs/plone.contentrules-1.1.0-py2.4.egg/plone/contentrules/rule/rule.py", line 48, in __call__
if not executable():
File "/home/zope/home/eggs/plone.app.contentrules-1.1.7-py2.4.egg/plone/app/contentrules/actions/mail.py", line 103, in __call__
From=source)
File "/home/zope/home/eggs/Products.SecureMailHost-1.1.2-py2.4.egg/Products/SecureMailHost/SecureMailHost.py", line 246, in secureSend
return self._send(mfrom, all_recipients, msg, debug=debug)
File "/home/zope/home/eggs/Products.SecureMailHost-1.1.2-py2.4.egg/Products/SecureMailHost/SecureMailHost.py", line 276, in _send
mail.send()
File "/home/zope/home/eggs/Products.SecureMailHost-1.1.2-py2.4.egg/Products/SecureMailHost/mail.py", line 102, in send
smtpserver = smtplib.SMTP(self.host, self.port)
File "/usr/local/lib/python2.4/smtplib.py", line 244, in __init__
(code, msg) = self.connect(host, port)
File "/usr/local/lib/python2.4/smtplib.py", line 311, in connect
(code, msg) = self.getreply()
File "/usr/local/lib/python2.4/smtplib.py", line 352, in getreply
line = self.file.readline()
File "/usr/local/lib/python2.4/socket.py", line 332, in readline
data = self._sock.recv(self._rbufsize)
These notifications generally work without issue (many emails are sent per day) and have been configured well over 12 months with the problem only arising in the last 2 months. The only major change that proceeded the problem was switching from Apache load balancer to Haproxy
We’re using:
- Linux
- Local qmail server (localhost:25)
- Python 2.4.6
- Plone 3.3.6
This seems to be the result of our qmail server hitting its external connection limit. We have this configured to 30 and one IP (presumably a spambot) was taking up 20 of these.
Our Zope instances send email via qmail on the same virtual server but that is also our externally facing mail server. Once the connection limit is reached it seems that connection attempts from smtplib will wait with no timeout.
For now I’ve turned off the offending content rules. Longer term we may patch ucspi-tcp to limit connections by IP and switch to using a separate mail server for local and external emails.