Trac opens too many files

[matze]

The Issue-Tracker is down from time too time because the Python processes exceed the maximum number of open files (ulimit). Although I've already tried to just increase that limit, they continued to fail. Interestingly the log-files list errors for /dev/urandom exclusively, so this is most likely a software bug but not a scaling issue:

2015-08-22 06:42:48,281 Trac[chrome] ERROR: Error with navigation contributor QueryModule: OSError: [Errno 24] Too many open files: '/dev/urandom'
2015-08-22 06:42:48,293 Trac[chrome] ERROR: Error with navigation contributor TicketModule: OSError: [Errno 24] Too many open files: '/dev/urandom'
2015-08-22 06:42:48,306 Trac[chrome] ERROR: Error with navigation contributor AdminModule: OSError: [Errno 24] Too many open files: '/dev/urandom'
2015-08-22 06:42:48,307 Trac[chrome] ERROR: Error with navigation contributor SearchModule: OSError: [Errno 24] Too many open files: '/dev/urandom'
2015-08-22 06:42:48,323 Trac[chrome] ERROR: Error with navigation contributor QueryModule: OSError: [Errno 24] Too many open files: '/dev/urandom'
2015-08-22 06:42:48,375 Trac[chrome] ERROR: Error with navigation contributor TicketModule: OSError: [Errno 24] Too many open files: '/dev/urandom'

Meanwhile errors like the following are logged:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/trac/web/main.py", line 497, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/lib/python2.7/dist-packages/trac/web/main.py", line 170, in dispatch
    if handler.match_request(req):
  File "/usr/local/lib/python2.7/dist-packages/trac/attachment.py", line 468, in match_request
    req.args['realm'] = realm
  File "/usr/local/lib/python2.7/dist-packages/trac/web/api.py", line 316, in __getattr__
    value = self.callbacks[name](self)
  File "/usr/local/lib/python2.7/dist-packages/trac/web/api.py", line 299, in <lambda>
    'args': lambda req: arg_list_to_args(req.arg_list),
  File "/usr/local/lib/python2.7/dist-packages/trac/web/api.py", line 316, in __getattr__
    value = self.callbacks[name](self)
  File "/usr/local/lib/python2.7/dist-packages/trac/web/api.py", line 648, in _parse_arg_list
    fs = _FieldStorage(fp, environ=self.environ, keep_blank_values=True)
  File "/usr/lib/python2.7/cgi.py", line 508, in __init__
    self.read_multi(environ, keep_blank_values, strict_parsing)
  File "/usr/local/lib/python2.7/dist-packages/trac/web/api.py", line 173, in read_multi
    cgi.FieldStorage.read_multi(self, *args, **kwargs)
  File "/usr/lib/python2.7/cgi.py", line 637, in read_multi
    environ, keep_blank_values, strict_parsing)
  File "/usr/lib/python2.7/cgi.py", line 510, in __init__
    self.read_single()
  File "/usr/lib/python2.7/cgi.py", line 647, in read_single
    self.read_lines()
  File "/usr/lib/python2.7/cgi.py", line 669, in read_lines
    self.read_lines_to_outerboundary()
  File "/usr/lib/python2.7/cgi.py", line 697, in read_lines_to_outerboundary
    line = self.fp.readline(1<<16)
  File "/usr/local/lib/python2.7/dist-packages/trac/web/_fcgi.py", line 205, in readline
    self._waitForData()
  File "/usr/local/lib/python2.7/dist-packages/trac/web/_fcgi.py", line 159, in _waitForData
    self._conn.process_input()
  File "/usr/local/lib/python2.7/dist-packages/trac/web/_fcgi.py", line 685, in process_input
    rec.read(self._sock)
  File "/usr/local/lib/python2.7/dist-packages/trac/web/_fcgi.py", line 504, in read
    raise EOFError

Haven't looked into whether those are the reason for the issue or a symptom yet.