[TestManager 1.4.3] Parse error thrown everytime after trying to create a new catalogue

[Naresh Verma]

Description:

---

Trac TestManager functionality is completely broken after trying to create a new catalogue.

Trac TestManager plugin had been working well in past. There had been multiple catalogues, test plans earlier. The problem started while creating one sub-catalogue.

Traceback:

Most recent call last:
File "/usr/lib/python2.6/dist-packages/trac/web/main.py", line 450, in _dispatch_request
  dispatcher.dispatch(req)
File "/usr/lib/python2.6/dist-packages/trac/web/main.py", line 227, in dispatch
  data, content_type)
File "/usr/lib/python2.6/dist-packages/trac/web/chrome.py", line 745, in render_template
  stream |= self._filter_stream(req, method, filename, stream, data)
File "/usr/lib/pymodules/python2.6/genshi/core.py", line 132, in __or__
  return Stream(_ensure(function(self)), serializer=self.serializer)
File "/usr/lib/python2.6/dist-packages/trac/web/chrome.py", line 848, in inner
  data)
File "/opt/trac/coast/plugins/TestManager-1.4.3-py2.6.egg/testmanager/wiki.py", line 111, in filter_streamFile "/opt/trac/coast/plugins/TestManager-1.4.3-py2.6.egg/testmanager/wiki.py", line 184, in _catalog_wiki_viewFile "/usr/lib/pymodules/python2.6/genshi/input.py", line 425, in HTML
  return Stream(list(HTMLParser(StringIO(text), encoding=encoding)))
File "/usr/lib/pymodules/python2.6/genshi/core.py", line 288, in _ensure
  for event in stream:
File "/usr/lib/pymodules/python2.6/genshi/input.py", line 432, in _coalesce
  for kind, data, pos in chain(stream, [(None, None, None)]):
File "/usr/lib/pymodules/python2.6/genshi/input.py", line 339, in _generate
  raise ParseError(msg, self.filename, e.lineno, e.offset)

Observations:

---

1. Rest of the trac functionality is working fine apart from the TestManager plugin
2. The test manager stats is also working.
3. there is no mention about this problem on mailing list.
4. TestManager plugin is unusable completely after this problem.

Setup:

System Information

Trac:	0.11.7
Python:	2.6.6 (r266:84292, Dec 27 2010, 00:18:12) [GCC 4.4.5]
setuptools:	0.6
SQLite:	3.7.4
pysqlite:	2.6.0
Genshi:	0.6
mod_python:	3.3.1
Pygments:	1.4
Subversion:	1.6.12 (r955767)
jQuery:	1.5

'''Plaugins''':
components	autocompleteusers.*	enabled
ldapplugin.*	enabled
sqlexecutor.*	enabled
testmanager.api.testmanagersystem	enabled
testmanager.macros.testcasebreadcrumbmacro	enabled
testmanager.macros.testcasechangestatusmacro	enabled
testmanager.macros.testcasestatushistorymacro	enabled
testmanager.macros.testcasestatusmacro	enabled
testmanager.macros.testcasetreemacro	enabled
testmanager.macros.testplanlistmacro	enabled
testmanager.macros.testplantreemacro	enabled
testmanager.model.testmanagermodelprovider	enabled
testmanager.stats.teststatsplugin	enabled
testmanager.web_ui.testmanager	enabled
testmanager.web_ui.testmanagertemplateprovider	enabled
testmanager.wiki.wikitestmanagerinterface	enabled
testmanager.workflow.testmanagerworkflowinterface	enabled
ticket_clone.*	enabled
tickettemplate.*	enabled
tickettemplate.ttadmin.*	enabled
tracgenericclass.*	enabled
tracgenericworkflow.*	enabled
tracpygments.*	enabled
tracwysiwyg.*	enabled
webadmin.*	enabled
webadmin.basics.*	enabled
webadmin.logging.*	enabled
webadmin.perm.*	enabled
webadmin.plugin.*	enabled
webadmin.ticket.*	enabled
webadmin.web_ui.*	enabled

[anonymous]

Hi, I'm looking into your problem, but since I cannot reproduce, I'd like to ask you a couple of questions.

I assume you upgraded TestManager from a previous release, which one? Or it was a clean installation?

Please, could you turn debug on (from Admin panel, select Logging and then File), rerun your scenario and attach to this ticket the part of the log you think appropriate.

Ciao! Roberto

[Roberto Longobardi]

...forgot to sign in... (why do they keep allowing anonymous comments???)

[Naresh Verma]

[1] The installation was fresh. Trac system had been running for some time (I think for few years). [2] TestManager plugin was added a month back.

• Test Manager had been working well & we had used it extensively.
• Problem started when one of our tester copied text from our existing excel sheets to the Test Manager.

  What appears to us is this text probably contains some char that's been treated as EoF or other control char. The test case content which was pasted in the catalogue is attached in the bug in excel sheet named "BM- Test Plan.xlsx"

Debug logs are attached in the bug report "trac.log"

[Naresh Verma]

Trac Debug Logs

[Naresh Verma]

Test Case Used to paste in TestManager causing this problem

[Roberto Longobardi]

Hi, yes you're probably right... some weird character sequence in the text.

I'd like to take a look at your attachment, but since it contains blanks in the name it appears to be broken. Please, could you attach it again with a name with no blanks and special chars? (This track hacks site should really upgrade its Trac, it's still at 0.10...)

In the meantime, I'm thinking about how to workaround your problem anyway. You may try to delete the offending test case, if that's ok for you. The problem is how to identify it.

I'll try to set up something able to spot test cases with offending text in them, and let you know.

Ciao, Roberto

[Naresh Verma]

Test Plan without spaces in filename

[Naresh Verma]

• We too are struggling to identify the test case in the system. Not sure how to isolate it and delete it.
• The excel sheet is added without spaces in filename.

[Roberto Longobardi]

I've added a backdoor to this version of the plugin, to help you identify the test case or catalog containing the offending text.

1) Copy the attached plugin egg file into your environment's plugins directory (and backup the original one).

I didn't update the plugin version, so you won't be able to install using the admin console. I did this to facilitate you later putting the original one back into place. Let me know if this is a problem.

2) Enable DEBUG logging 3) Go to the Test Manager's home page (root catalog) by using the link in Trac's toolbar. This should cause your problem. 4) Edit the URL in the browser's address bar and add a "?verify" at the end.

Check the log file for this type of content:

13:48:09 Trac[wiki] DEBUG: >>> _verify_text_integrity
13:48:09 Trac[wiki] DEBUG:      >> Examining Test Catalog with path 'TC_TT0' and ID 0 ...
13:48:09 Trac[wiki] DEBUG:      >> Examining Test Catalog with path 'TC_TT1' and ID 1 ...
13:48:09 Trac[wiki] DEBUG:      >> Examining Test Case with path 'TC_TT1_TC4' and ID 4 ...
13:48:09 Trac[wiki] DEBUG:      >> Examining Test Catalog with path 'TC_TT1_TT2' and ID 2 ...
13:48:09 Trac[wiki] DEBUG:      >> Examining Test Case with path 'TC_TT1_TT2_TC0' and ID 0 ...
13:48:09 Trac[wiki] DEBUG:      >> Examining Test Case with path 'TC_TT1_TT2_TC1' and ID 1 ...
13:48:09 Trac[wiki] DEBUG:      >> Examining Test Case with path 'TC_TT1_TT2_TC2' and ID 2 ...
13:48:09 Trac[wiki] DEBUG:      >> Examining Test Case with path 'TC_TT1_TT2_TC3' and ID 3 ...
13:48:09 Trac[wiki] DEBUG:      >> Examining Test Catalog with path 'TC_TT3' and ID 3 ...
13:48:09 Trac[wiki] DEBUG:      >> Examining Test Catalog with path 'TC_TT3_TT4' and ID 4 ...
13:48:09 Trac[wiki] DEBUG:      >> Examining Test Case with path 'TC_TT3_TT4_TC5' and ID 5 ...
13:48:09 Trac[wiki] DEBUG:      >> Examining Test Case with path 'TC_TT3_TT4_TC6' and ID 6 ...
13:48:09 Trac[wiki] DEBUG: <<< _verify_text_integrity

and look for a line with:

13:48:09 Trac[wiki] DEBUG: !!! Exception trying to convert to HTML !!!

Take note of the line just preceding the error, for the object type (either catalog or test case) and ID.

5) Edit the simple Python file leveraging the Test Manager's XML-RPC api to delete the offending object.

You should replace your server's URL and the object type and ID as remarked in the python file.

6) Launch the python file simply with:

python delete_test_case.py

7) Kindly let me know if this works.

Hope this helps :-)

Ciao, roberto

[Roberto Longobardi]

Replacement test manager (only) plugin with backdoor for test

[Roberto Longobardi]

Python using the Test Manager XML RPC api to delete a test case

[Roberto Longobardi]

For your reference, this is the documentation for the XML-RPC remote API:

​http://trac-hacks.org/wiki/TestManagerForTracPluginRPCApi

Ciao, Roberto

[Naresh Verma]

1. Got the following error after copying the TestManager-1.4.3-py2.6.egg file (attached in the defect report):
   
2. This Zip import error does not show up with the original TestManager plugin file.
   
3. Observed this error when I clicked "Admin > Plugins"
   
4. As expected after this error the DEBUG logs does not show any logging of "_verify_text_integrity"
   

---

Trac detected an internal error:
ZipImportError: bad local file header in /opt/trac/coast/plugins/TestManager-1.4.3-py2.6.egg

---

Python Traceback

Most recent call last:
File "/usr/lib/python2.6/dist-packages/trac/web/main.py", line 450, in _dispatch_request
File "/usr/lib/python2.6/dist-packages/trac/web/main.py", line 206, in dispatch
File "/usr/lib/python2.6/dist-packages/trac/admin/web_ui.py", line 114, in process_request
File "/usr/lib/python2.6/dist-packages/trac/admin/web_ui.py", line 418, in render_admin_panel
File "/usr/lib/python2.6/dist-packages/trac/admin/web_ui.py", line 522, in _render_view
File "/usr/lib/python2.6/dist-packages/trac/util/init.py", line 410, in get_pkginfo
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 1166, in get_metadata
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 1231, in _get

System Information:

User Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-us) AppleWebKit/533.19.4 (KHTML, like Gecko)
Version/5.0.3 Safari/533.19.4

Trac: 0.11.7
Python: 2.6.6 (r266:84292, Dec 27 2010, 00:18:12) [GCC 4.4.5]
setuptools: 0.6
SQLite: 3.7.4
pysqlite: 2.6.0
Genshi: 0.6
mod_python: 3.3.1
Pygments: 1.4
Subversion: 1.6.12 (r955767)
jQuery: 1.5

[Roberto Longobardi]

Hi, this is strange, since the egg file is correct.

I'm afraid you may have downloaded the egg using the wrong procedure...

Did you go right-click on the attachment link above and "Save as"? Because this way you would have downloaded an HTML page with an egg file extension...

Please, make sure you navigate to the attachment download page first, and then download the file.

Let me know if this is the problem. Ciao. Roberto

[Naresh Verma]

I just retried using Firefox & Safari again and got the same error. Used Right click & "Save Link as..." in FIrefox & "Download Linked File As..." in Safari. I think these steps should be working if the file is ok.

[Roberto Longobardi]

Hi, nope, the problem is that the link above just brings you to a download page, not to the file itself :-)

Please, just click on the link above and then, from the download page, download the file, or just use the following direct link: TestManager-1.4.3-py2.6.egg

Ciao, roberto

[Naresh Verma]

File Download

[Naresh Verma]

Please refer the attached screenshot that shows when I downloaded the egg file.

Bellow is the debug log file output saying import error: ZipImportError: bad local file header in /opt/trac/coast/plugins/TestManager-1.4.3-py2.6.egg

[Roberto Longobardi]

This seems a known problem, actually, if you didn't restart Trac after replacing the egg file.

​t:#7014

Is this your case?

[Naresh Verma]

Yep, restarting trac removed the header issue. Thanks a lot.

Clicking the TestManager page with root catalogue gave me following trace which looks like "Object Not found": It appears the problem is before the Test Manager constructs the HTML.

2011-03-07 17:22:48,853 Trac[main] DEBUG: Dispatching <Request "GET u'/wiki/TC'">
2011-03-07 17:22:48,865 Trac[chrome] DEBUG: Prepare chrome data for request
2011-03-07 17:22:48,874 Trac[api] DEBUG: Updating wiki page index
2011-03-07 17:22:48,893 Trac[session] DEBUG: Retrieving session for ID u'naresh.verma'
2011-03-07 17:22:49,076 Trac[model] DEBUG: Title: None
2011-03-07 17:22:49,076 Trac[model] DEBUG: Description: None
2011-03-07 17:22:49,077 Trac[model] DEBUG: >>> get_fields
2011-03-07 17:22:49,077 Trac[model] DEBUG: <generator object options at 0xb9be5aa4>
2011-03-07 17:22:49,077 Trac[model] DEBUG: <generator object options at 0xb9be5acc>
2011-03-07 17:22:49,078 Trac[model] DEBUG: <generator object options at 0xb9be5af4>
2011-03-07 17:22:49,078 Trac[model] DEBUG: <generator object options at 0xb9be5b1c>
2011-03-07 17:22:49,079 Trac[model] DEBUG: <generator object options at 0xb9be5b44>
2011-03-07 17:22:49,079 Trac[model] DEBUG: Fields for realm testcase:
2011-03-07 17:22:49,079 Trac[model] DEBUG:    id : text
2011-03-07 17:22:49,080 Trac[model] DEBUG:    page_name : text
2011-03-07 17:22:49,080 Trac[model] DEBUG: Fields for realm testcatalog:
2011-03-07 17:22:49,080 Trac[model] DEBUG:    id : text
2011-03-07 17:22:49,081 Trac[model] DEBUG:    page_name : text
2011-03-07 17:22:49,081 Trac[model] DEBUG: Fields for realm resourceworkflowstate:
2011-03-07 17:22:49,081 Trac[model] DEBUG:    id : text
2011-03-07 17:22:49,081 Trac[model] DEBUG:    res_realm : text
2011-03-07 17:22:49,082 Trac[model] DEBUG:    state : text
2011-03-07 17:22:49,082 Trac[model] DEBUG: Fields for realm testplan:
2011-03-07 17:22:49,082 Trac[model] DEBUG:    id : text
2011-03-07 17:22:49,082 Trac[model] DEBUG:    catid : text
2011-03-07 17:22:49,083 Trac[model] DEBUG:    page_name : text
2011-03-07 17:22:49,083 Trac[model] DEBUG:    name : text
2011-03-07 17:22:49,083 Trac[model] DEBUG:    author : text
2011-03-07 17:22:49,083 Trac[model] DEBUG:    time : time
2011-03-07 17:22:49,084 Trac[model] DEBUG: Fields for realm testcaseinplan:
2011-03-07 17:22:49,084 Trac[model] DEBUG:    id : text
2011-03-07 17:22:49,084 Trac[model] DEBUG:    planid : text
2011-03-07 17:22:49,084 Trac[model] DEBUG:    page_name : text
2011-03-07 17:22:49,085 Trac[model] DEBUG:    status : text
2011-03-07 17:22:49,085 Trac[model] DEBUG: <<< get_fields
2011-03-07 17:22:49,085 Trac[model] DEBUG: >>> _fetch_object
2011-03-07 17:22:49,086 Trac[model] DEBUG: Searching for testcatalog: WHERE 1=1 AND id=%%s
2011-03-07 17:22:49,086 Trac[model] DEBUG: id = TC
2011-03-07 17:22:49,087 Trac[model] DEBUG: Object NOT found.
2011-03-07 17:22:49,087 Trac[model] DEBUG: Exists: False
2011-03-07 17:22:49,088 Trac[model] DEBUG: {'page_name': u'TC', 'id': u'TC'}
2011-03-07 17:22:49,402 Trac[main] ERROR: Internal Server Error: 
Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/trac/web/main.py", line 450, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/lib/python2.6/dist-packages/trac/web/main.py", line 227, in dispatch
    data, content_type)
  File "/usr/lib/python2.6/dist-packages/trac/web/chrome.py", line 745, in render_template
    stream |= self._filter_stream(req, method, filename, stream, data)
  File "/usr/lib/pymodules/python2.6/genshi/core.py", line 132, in __or__
    return Stream(_ensure(function(self)), serializer=self.serializer)
  File "/usr/lib/python2.6/dist-packages/trac/web/chrome.py", line 848, in inner
    data)
  File "/opt/trac/coast/plugins/TestManager-1.4.3-py2.6.egg/testmanager/wiki.py", line 112, in filter_stream
    return self._catalog_wiki_view(req, formatter, page_name, stream)
  File "/opt/trac/coast/plugins/TestManager-1.4.3-py2.6.egg/testmanager/wiki.py", line 190, in _catalog_wiki_view
    HTML(tree_macro.expand_macro(formatter, None, 'mode='+mode+',fulldetails='+fulldetails+',catalog_path='+page_name)),
  File "/usr/lib/pymodules/python2.6/genshi/input.py", line 425, in HTML
    return Stream(list(HTMLParser(StringIO(text), encoding=encoding)))
  File "/usr/lib/pymodules/python2.6/genshi/core.py", line 288, in _ensure
    for event in stream:
  File "/usr/lib/pymodules/python2.6/genshi/input.py", line 432, in _coalesce
    for kind, data, pos in chain(stream, [(None, None, None)]):
  File "/usr/lib/pymodules/python2.6/genshi/input.py", line 339, in _generate
    raise ParseError(msg, self.filename, e.lineno, e.offset)
ParseError: EOF in middle of construct: line 1, column 57906

[Roberto Longobardi]

OK, you're almost there.

Now, please add the "?verify" parameter to the test manager root URL as explained above (after turning debug on), cross your fingers and watch the log file for the offending test case.

Let me know. Ciao. Roberto

[Naresh Verma]

Thanks, I got it working as described. We had disabled the anonymous login and now I am getting the forbidden error. Is there any other way to get the python script working ?

[anonymous]

I've never used it with authentication, but from the XmlRpc plugin page

​http://trac-hacks.org/wiki/XmlRpcPlugin

it looks like you should be able to use the /login/rpc URL suffix and specify your password on the URL, as follows:

server = xmlrpclib.ServerProxy("​http://user:password@yourserver:port/yourproject/login/rpc")

[Naresh Verma]

Ok, figured out that my env was missing the XMLRPC plugin, so installed TracXMLRPC 1.1.2-r9913 and tried. As expected got multiple cases reported for HTML conversion like:

2011-03-07 22:11:23,832 Trac[wiki] DEBUG:      >> Examining Test Case with path 'TC_TT11_TT16_TT51_TC162' and ID 162 ...
2011-03-07 22:11:23,834 Trac[wiki] DEBUG: !!! Exception trying to convert to HTML !!!
2011-03-07 22:11:23,835 Trac[wiki] DEBUG:      >> Examining Test Case with path 'TC_TT11_TT16_TT51_TC163' and ID 163 ...
2011-03-07 22:11:23,836 Trac[wiki] DEBUG:      >> Examining Test Case with path 'TC_TT11_TT16_TT51_TC164' and ID 164 ...
2011-03-07 22:11:23,838 Trac[wiki] DEBUG:      >> Examining Test Case with path 'TC_TT11_TT16_TT51_TC165' and ID 165 ...
2011-03-07 22:11:23,840 Trac[wiki] DEBUG: !!! Exception trying to convert to HTML !!!

Then ran the python script but always got "false" when run with these IDs with both testcase & testcatalog:

# python delete_test_case.py 
>> Deleting test case
False
>> Deleted

Is there anything I am still missing? The anonymous has the XMLRPC privileges.

[Naresh Verma]

Got it working finally, the XMLRPC needed the authentication. Thanks a lot for helping out.

[anonymous]

I'm glad everything works now :-)

And please you find some problem or need some new feature don't hesitate to speak out and open a ticket here :-)

Ciao, Roberto