OSError: [Errno 13] Permission denied: '/srv/trac-hacks.org/trac/files'

[Ryan J Ollos]

There have been a few mailing list posts, including ​this recent one, about the Trac attachment migration not working. At least two of them were upgrading from 0.11 to 1.0. The migration doesn't seemed to have worked on t-h.o.

1. When I try to add an attachment to a ticket, I get the traceback below. Not surprising since there is no files directory in the environment.
2. There was no error during the trac-admin $ENV upgrade.
3. The Image macro shows attachments, but it is not possible to navigate directly to an attachment (try attachment:wiki:AccountManagerPlugin:components-admin.png​, which does exist).

We'll need some way to fix up the database now I assume. It would be nice to figure out why this isn't working in some cases and submit a patch to Trac.

drwxr-xr-x 12 root     root     4096 Jun  6 02:46 .
drwxr-xr-x  7 root     root     4096 Sep  6  2012 ..
drwxr-xr-x 16 www-data www-data 4096 Sep  5  2012 .egg-cache
drwxr-xr-x  2 www-data www-data 4096 Mar 22 17:03 .poll
drwxr-xr-x  2 www-data www-data 4096 Sep  4  2012 .reposearch
-rw-r--r--  1 root     root       98 Jan 23  2009 README
-rw-r--r--  1 root     root       27 Jan 23  2009 VERSION
drwxr-xr-x  4 www-data www-data 4096 Feb  4  2009 attachments
drwxr-xr-x  3 www-data www-data 4096 Jun  6 09:37 conf
drwxr-xr-x  2 root     root     4096 Jun  6 05:00 db
drwxr-xr-x  2 www-data www-data 4096 Jun  6 10:06 log
drwxr-xr-x  2 root     root     4096 Jun  6 02:46 plugins
drwxr-xr-x  2 root     root     4096 Mar  2  2009 templates
drwxr-xr-x  2 root     root     4096 Sep  6  2012 wiki-macros

File "/srv/trac-hacks.org/pve/lib/python2.6/site-packages/Trac-1.0.1-py2.6.egg/trac/web/main.py", line 497, in _dispatch_request
  dispatcher.dispatch(req)
File "/srv/trac-hacks.org/pve/lib/python2.6/site-packages/Trac-1.0.1-py2.6.egg/trac/web/main.py", line 214, in dispatch
  resp = chosen_handler.process_request(req)
File "/srv/trac-hacks.org/pve/lib/python2.6/site-packages/Trac-1.0.1-py2.6.egg/trac/attachment.py", line 512, in process_request
  data = self._do_save(req, attachment)
File "/srv/trac-hacks.org/pve/lib/python2.6/site-packages/Trac-1.0.1-py2.6.egg/trac/attachment.py", line 769, in _do_save
  attachment.insert(filename, upload.file, size)
File "/srv/trac-hacks.org/pve/lib/python2.6/site-packages/Trac-1.0.1-py2.6.egg/trac/attachment.py", line 319, in insert
  os.makedirs(dir)
File "/usr/lib/python2.6/os.py", line 150, in makedirs
  makedirs(head, mode)
File "/usr/lib/python2.6/os.py", line 150, in makedirs
  makedirs(head, mode)
File "/usr/lib/python2.6/os.py", line 150, in makedirs
  makedirs(head, mode)
File "/usr/lib/python2.6/os.py", line 150, in makedirs
  makedirs(head, mode)
File "/usr/lib/python2.6/os.py", line 157, in makedirs
  mkdir(name, mode)

[Jun Omae]

Cannot reproduce....

The migration is defined in ​trac/upgrades/db28.py. First, the function checks the existence of $ENV/attachments. If it exists, run do_upgrade().

But, if the os.path.exists(old_dir) returns False by some reasons, e.g. selinux? nfs cache?, it does nothing. The os.path.exists() returns False if os.stat() raises any os.error, even if not ENOENT.

trac/upgrades/db28.py

@@ -11 +11 @@
 # individuals. For the exact contribution history, see the revision
 # history and logs, available at http://trac.edgewall.org/.
 
+import errno
 import os.path
 
 from trac.attachment import Attachment
@@ -23 +24 @@
     the filenames in the process."""
     path = env.path
     old_dir = os.path.join(path, 'attachments')
-    if not os.path.exists(old_dir):
-        return
-    old_stat = os.stat(old_dir)
+    try:
+        old_stat = os.stat(old_dir)
+    except OSError, e:
+        if e.errno == errno.ENOENT:
+            return
+        raise
     new_dir = os.path.join(path, 'files', 'attachments')
-    if not os.path.exists(new_dir):
+    if not os.path.isdir(new_dir):
         os.makedirs(new_dir)
 
     cursor.execute("""

run-db28.py script

# -*- coding: utf-8 -*-
#
# Execute `do_upgrade` in trac/upgrades/db28.py
#
# Usage: python run-db28.py /path/to/tracenv
#

from __future__ import with_statement

from trac.env import Environment
from trac.upgrades import db28

def main(args):
    for arg in args:
        env = Environment(arg)
        with env.db_transaction as db:
            cursor = db.cursor()
            db28.do_upgrade(env, 28, cursor)

if __name__ == '__main__':
    import sys
    main(sys.argv[1:])

[Ryan J Ollos]

Hi Jun, Thank you for the patch. I tried running your run-db28.py locally on separate instances of the trac-hacks environment that I pulled off the server. I tried on one instance with the db28.py patched, and one without. In both cases the attachment upgrade was successful. Finally, I patched the db28.py on the trac-hacks server and ran the upgrade and it was successful as well.

Here is the output, which looks pretty standard to me.

rjollos@hacks:/srv/trac-1.0.1$ sudo /srv/trac-hacks.org/pve/bin/python run-db28.py /srv/trac-hacks.org/trac
The upgrade of attachments was successful, but the old attachments directory:

  /srv/trac-hacks.org/trac/attachments

couldn't be removed, possibly due to the presence of files that weren't
referenced in the database. The error was:

  OSError: [Errno 39] Directory not empty: '/srv/trac-hacks.org/trac/attachments/wiki/MyProjectPlugin/Project Files'

This error can be ignored, but for keeping your environment clean you should
backup any remaining files in that directory and remove it manually.

To summarize, as others on the mailing list have reported, running db28.py after a failed upgrade seems to work well, however I can't say if the patch had an effect. I will see if I can pull down a copy of the Trac 0.10.6 environment, install that locally and reproduce the failed upgrade.

On the trac-hacks server we are running,

$ /etc/apache2/sites-common$ cat /etc/debian_version
6.0.7
$ uname -a
Linux hacks 2.6.22-3-vserver-686 #1 SMP Tue Dec 4 08:57:30 UTC 2007 i686 GNU/Linux

It looks like we are not running SELinux.

[Jun Omae]

Thanks for the details.

...., however I can't say if the patch had an effect.

Indeed, it seems that the patch has no effects.

I will see if I can pull down a copy of the Trac 0.10.6 environment, install that locally and reproduce the failed upgrade.

Ok. I hope to reproduce this issue.

BTW, is there trac.log of the Trac environment on trac-hacks.org which was upgraded on Jun 6? Could you please post the trac.log?

[Ryan J Ollos]

Replying to jun66j5:

I will see if I can pull down a copy of the Trac 0.10.6 environment, install that locally and reproduce the failed upgrade.

Ok. I hope to reproduce this issue.

I can try to make a copy of the Trac 0.10.6 environment available to you if that would help.

BTW, is there trac.log of the Trac environment on trac-hacks.org which was upgraded on Jun 6? Could you please post the trac.log?

From what I see in /srv/trac-hacks.org/trac/log, there is a log rotate in place that retains 5 log files and we seemed to have just missed the opportunity to grab the one from June 6th. I'm not sure if logs are backed-up somewhere else for longer-term, but I can check into that.

[Ryan J Ollos]

​trac:#11370 has been opened for this issue. Since the issue is resolved on t-h.o, I'll close the ticket for now.