UnicodeDecode error when saving hours as csv (fix included)

[Dolf Andringa]

When downloading a csv file from the /hours url, sometimes I get a

UnicodeEncodeError: 'ascii' codec can't encode character ... in position 21: ordinal not in range(128)

This error occurs when a user entered unicode characters in tickets/hour logs. When downloading the csv file in queryhours2csv, this error was thrown.

The included diff fixes the error by properly encoding all strings as utf-8.

[Ryan J Ollos]

Duplicate of #8520 I think, but thanks for the patch. I will be able to test and apply it later this week.

#8520 closed as a duplicate.

[Ryan J Ollos]

I'm having trouble reproducing the issue. Could you provide a screen capture from just before the export, as well as the traceback you get when exporting?

[Ryan J Ollos]

Here is the traceback that results:

How to Reproduce

While doing a GET operation on /hours, Trac issued an internal error.

(please provide additional details here)

Request parameters:

{'format': u'csv'}

User agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.22 (KHTML, like Gecko) Ubuntu Chromium/25.0.1364.160 Chrome/25.0.1364.160 Safari/537.22

System Information

Trac	1.0.1
Babel	0.9.6
Genshi	0.6 (without speedups)
pysqlite	2.6.0
Python	2.7.3 (default, Aug 1 2012, 05:16:07) [GCC 4.6.3]
setuptools	0.6
SQLite	3.7.9
jQuery	1.7.2

Enabled Plugins

ComponentDependencyPlugin	0.1
PrivateReports	0.4dev
TicketSidebarProvider	0.0-r11698
TracHoursPlugin	0.6.0dev

Python Traceback

Traceback (most recent call last):
  File "/home/user/Workspace/th11047/trac-1.0.1/trac/web/main.py", line 497, in _dispatch_request
    dispatcher.dispatch(req)
  File "/home/user/Workspace/th11047/trac-1.0.1/trac/web/main.py", line 214, in dispatch
    resp = chosen_handler.process_request(req)
  File "/home/user/Workspace/trachacks2.git/trachoursplugin/trunk/trachours/hours.py", line 240, in process_request
    return self.process_timeline(req)
  File "/home/user/Workspace/trachacks2.git/trachoursplugin/trunk/trachours/hours.py", line 522, in process_timeline
    return self.display_html(req, query)
  File "/home/user/Workspace/trachacks2.git/trachoursplugin/trunk/trachours/hours.py", line 826, in display_html
    self.queryhours2csv(req, data)
  File "/home/user/Workspace/trachacks2.git/trachoursplugin/trunk/trachours/hours.py", line 1032, in queryhours2csv
    for header in data['headers']])
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-9: ordinal not in range(128)

[Ryan J Ollos]

#10848 closed as a duplicate.

[Ryan J Ollos]

(In [13210]) Fixes #10233: Encode data as utf-8 when exporting as CSV.

[Ryan J Ollos]

Replying to dolfandringa:

This error occurs when a user entered unicode characters in tickets/hour logs. When downloading the csv file in queryhours2csv, this error was thrown.

I think it can only happen if the ticket user, summary or similar field contains a unicode character. I can't see that the work comments ever get exported as CSV, regardless of how the query is setup. Please let me know if I'm overlooking something though, or if you can reproduce the issue still after 0.6.0dev-r13210.

[seqira@…]

The problem still exists if there are headers in UTF (e.g. if trac is localized).

I'm not good at python (it's my first python code), but following changes fixed problem in current trunk.

trachours/hours.py

@@ -968 +968 @@
         for groupname, results in data['groups']:
             if groupname:
                 writer.writerow(unicode(groupname))
-            writer.writerow([unicode(header['label'])
-                             for header in data['headers']])
+
+
+            #writer.writerow([unicode(header['label'])
+            #                 for header in data['headers']])
+            rowhead = []
+            for header in data['headers']:
+                rowhead.append(unicode(header['label']).encode('utf-8'))
+            writer.writerow(rowhead)
             for result in results:
                 row = []
                 for header in data['headers']:

[Ryan J Ollos]

In 14310:

0.6.0dev: Encode headers when exporting as CSV to fix traceback. Fixes #10233.