Confusion with timestamp - lack of documentation

[kubes@…]

I would like to start using Trac tickets, but first I need to synchronize thousands of ticket with trac via XmlRpcPlugin.

I cannot find any documentation with properties and their types to provide to ticket_create and ticket_update. Documentation ​http://www.jspwiki.org/Wiki.jsp?page=WikiRPCInterface2 linked from ​http://trac-hacks.org/wiki/XmlRpcPlugin doesn't work.

Can you please provide more complex example how to use timestamps when calling ticket_update? I cannot get it work. I would expect that I should provide timestamp of type int32 in _ts field? But when I call _trac.ticket_get(id) I obtain very long number ["_ts"] = "1380897120346000" which seem not be be int32 unix timestamp.

I probably use wrong type beause of "Server returned a fault exception: [1] 'unsupported type for timedelta microseconds component: unicode' while executing 'ticket.update()'". My code C# code is:

  public XmlRpcStruct GetProperties()
        {           
            XmlRpcStruct dict = new XmlRpcStruct();            
            dict.Add("type", Type.ToString());
            dict.Add("time", Time.ToString("yyyyMMddTHHmmsszzz"));
            dict.Add("changetime", Changetime.ToString("yyyyMMddTHHmmsszzz"));            
            dict.Add("_ts", DateTimeToUnixTimeStamp(DateTime.Now));
            dict.Add("component", Component ?? string.Empty);
            dict.Add("severity", Severity.ToString());
            dict.Add("priority", Priority.ToString());
            dict.Add("owner", Owner ?? string.Empty);
            dict.Add("reporter", Reporter ?? string.Empty);
            dict.Add("cc", Cc ?? string.Empty);
            dict.Add("version", Version ?? string.Empty);
            dict.Add("milestone", Milestone ?? string.Empty);
            dict.Add("status", Status.ToString());
            dict.Add("resolution", Resolution.ToString());
            dict.Add("summary", Summary ?? string.Empty);
            dict.Add("description", Description?? string.Empty);
            dict.Add("keywords", Keywords ?? string.Empty);                   
            return dict;
        }

        public static int DateTimeToUnixTimeStamp(DateTime dateTime)
        {         
            System.DateTime epoch = new DateTime(1970, 1, 1, 0, 0, 0, 0);
            return (int)((dateTime - epoch.ToLocalTime()).TotalSeconds);
        }

[osimons]

When updating a ticket, all you have to do is pass whatever _ts value you get from the server. The timestamp is a marker for the server to make sure that updates are performed based on latest values, and not on stale values. So, in order to do ticket.update you first have to do ticket.get.

What this value is exactly has changed with Trac. In earlier Trac versions (pre-0.12) timestamped based on seconds since epoch. However, with 0.12 this changed so that Trac now timestamps based on microseconds. So dividing by 1.000.000 will get you the seconds since epoch. The plugin works with both versions.

However, all this is implementation details you should not be concerned with. _ts is really just a token used to ensure valid updates.

As for available fields, that also depends on your server. If you have defined custom fields they will also appear as available fields for creation and updating. All fields are returned by the ticket.getTicketFields call. However, as you noticed, the plugin does not know anything about types, expected inputs, validation rules or similar. You may even have validation/manipulator plugins installed the plugin cannot know about. The plugin takes the data as if entered through web, and makes Trac perform validation of input - applying changes if all is correct, or returning all the warnings if problems are found. With fields as if entered through web, all ticket attributes should be passed as plain strings.

If you have specific suggestions for more correct documentation, then by all means add patch or suggestions to this ticket.

[anonymous]

I understand you point of view, that XmlRpcPlugin is just redirecting the call to trac. BUT trac has user interface and does not need end user documentation of methods and properties. If there is any documentation, how to call these internals, it would be OK, but I cannot found such documentation.

For me ,the end user, trac is blackbox and it is very hard for me to hunt for these properties and correct values and it is very easy form me to destroy Trac with invalid call through the plugin.

I have found waht was couse the issue and destroying trac database. I have comment out the followinf lines and get the _ts like object and send it again like a object (not int32):

 public XmlRpcStruct GetProperties()
        {           
            XmlRpcStruct dict = new XmlRpcStruct();            
            dict.Add("type", Type.ToString());
            //dict.Add("time", Time.ToString("yyyyMMddTHHmmsszzz"));*/
            //dict.Add("changetime", Changetime.ToString("yyyyMMddTHHmmsszzz"));    
            //dict.Add("_ts", DateTimeToUnixTimeStamp(DateTime.Now) + "000000");
            // http://trac-hacks.org/ticket/11340#comment:1    
            dict.Add("component", Component ?? string.Empty);
            dict.Add("severity", Severity.ToString());
            dict.Add("priority", Priority.ToString());
            dict.Add("owner", Owner ?? string.Empty);
            dict.Add("reporter", Reporter ?? string.Empty);
            dict.Add("cc", Cc ?? string.Empty);
            dict.Add("version", Version ?? string.Empty);
            dict.Add("milestone", Milestone ?? string.Empty);
            dict.Add("status", Status.ToString());
            dict.Add("resolution", Resolution.ToString());
            dict.Add("summary", Summary ?? string.Empty);
            dict.Add("description", Description?? string.Empty);
            dict.Add("keywords", Keywords ?? string.Empty);                   
            return dict;
        }

        private void KeepTryingUpdatingTicket(int id, Ticket t)
        {
            for (int i = 0; true; i++)
            {
                try
                {
                    string action;
                    switch (t.Resolution)
                    {
                        case Resolution.@fixed:
                            action = "resolve";
                            break;
                        case Resolution.invalid:
                        case Resolution.wontfix:
                        case Resolution.duplicate:
                        case Resolution.reject:
                        case Resolution.worksforme:
                        default:
                            action = string.Empty;
                            break;
                    }

                    object[] oldTicket = _trac.ticket_get(id);
                    //I don't concern with collision, it is first import, last is winner
                    XmlRpcStruct oldStruct = (XmlRpcStruct)oldTicket[3];
                    XmlRpcStruct newStruct = t.GetProperties();
                    //keep timestamp to confirm that it is not collision
                    object ts = oldStruct["_ts"];
                    newStruct["_ts"] = ts;

                    _trac.ticket_update(id, action, newStruct);
                    Console.WriteLine(t.Id + " updated");
                    return;
                }
                catch (WebException ex)//connection problem
                {
                    Console.WriteLine("problem: " + ex.Message);
                    Thread.Sleep(100 * i);
                    if (i == 10)//10 attempts
                        throw ex;
                }
            }
        }

Anyway I would like to put my own changetime and time. so what is the correct changetime format and time format?

[osimons]

All right. I see you got the _ts sorted by just returning whatever you got served when getting the ticket.

I said all ticket fields in Trac are in essence text fields. By that i mean XML-RPC String type. However, that is not correct for time and changetime. These fields are XML-RPC DateTime objects. The same is true for the when parameter supported by some of the methods, but in that case the call signature mentions it explicitly so details are not hidden in a general attributes construct. Anyway for DateTime your library no doubt has a way of creating those objects from native types.

PS! The reason ticket fields are 'hidden' in attributes instead of just being just regular keyword arguments to the method, is that the number of fields may vary both between Trac versions and between installations. A project can have any number of custom fields, called anything the admin wishes. I can't support that in the method signature - although explicit signatures is otherwise what I prefer when possible.

The plugin contains various tests, and these tests are all functional tests meaning that a test run actually starts a server and calls methods and checks result against a running server - calling just as you would do (only using Python of course). If you have a look at source:/xmlrpcplugin/trunk/tracrpc/tests/ticket.py​ you may get some pointers.

[kubes@…]

Finally I manage to write synchronization between our old tracking system and Trac using your plugin. The only thing I didn't manage to do is to put time and changetime property.

If I use XmlRpc.NET library and I use time as DateTime object, I got the same error as #10711 "'datetime.datetime' object has no attribute 'strip while executing 'ticket.create"

I understand the there is some method to trim the value and the datetime does not have trim method. It must be some issue connected with usage Trac + XmlRpc.NET. I have notice that in unittest xmlrpcplugin/trunk/tracrpc/tests/ticket.py​ on line 391 where do you use

dt = to_xmlrpc_datetime(to_datetime(None, utc))

so, it must be possible to send datetime to ticket create, which does not work for me. Is there some difference?

I use fiddler Http debugger to capture the XmlRpc request and response:

<?xml version="1.0"?>
<methodCall>
  <methodName>ticket.create</methodName>
  <params>
    <param>
      <value>
        <string>smazat</string>
      </value>
    </param>
    <param>
      <value>
        <string>smazat</string>
      </value>
    </param>
    <param>
      <value>
        <struct>
          <member>
            <name>type</name>
            <value>
              <string>defect</string>
            </value>
          </member>
          <member>
            <name>time</name>
            <value>
              <dateTime.iso8601>20131011T10:08:41</dateTime.iso8601>
            </value>
          </member>
          <member>
            <name>component</name>
            <value>
              <string />
            </value>
          </member>
          <member>
            <name>severity</name>
            <value>
              <string>highest</string>
            </value>
          </member>
          <member>
            <name>priority</name>
            <value>
              <string>blocker</string>
            </value>
          </member>
          <member>
            <name>owner</name>
            <value>
              <string />
            </value>
          </member>
          <member>
            <name>reporter</name>
            <value>
              <string>smazat</string>
            </value>
          </member>
          <member>
            <name>cc</name>
            <value>
              <string />
            </value>
          </member>
          <member>
            <name>version</name>
            <value>
              <string />
            </value>
          </member>
          <member>
            <name>milestone</name>
            <value>
              <string />
            </value>
          </member>
          <member>
            <name>status</name>
            <value>
              <string>new</string>
            </value>
          </member>
          <member>
            <name>resolution</name>
            <value>
              <string>invalid</string>
            </value>
          </member>
          <member>
            <name>summary</name>
            <value>
              <string>smazat</string>
            </value>
          </member>
          <member>
            <name>description</name>
            <value>
              <string>smazat</string>
            </value>
          </member>
          <member>
            <name>keywords</name>
            <value>
              <string />
            </value>
          </member>
        </struct>
      </value>
    </param>
  </params>
</methodCall>

<?xml version='1.0'?>
<methodResponse>
<fault>
<value><struct>
<member>
<name>faultCode</name>
<value><int>1</int></value>
</member>
<member>
<name>faultString</name>
<value><string>''datetime.datetime' object has no attribute 'strip'' while executing 'ticket.create()'</string></value>
</member>
</struct></value>
</fault>
</methodResponse>

Can you plese tell me what is wrong? What is different from the unittest?

[anonymous]

Replying to kubes@…:

Finally I manage to write synchronization between our old tracking system and Trac using your plugin. The only thing I didn't manage to do is to put time and changetime property.

If I use XmlRpc.NET library and I use time as DateTime object, I got the same error as ​http://trac.edgewall.org/ticket/10711 "'datetime.datetime' object has no attribute 'strip while executing 'ticket.create"

I understand the there is some method to trim the value and the datetime does not have trim method. It must be some issue connected with usage Trac + XmlRpc.NET. I have notice that in unittest xmlrpcplugin/trunk/tracrpc/tests/ticket.py​ on line 391 where do you use

dt = to_xmlrpc_datetime(to_datetime(None, utc))

so, it must be possible to send datetime to ticket create, which does not work for me. Is there some difference?

I use fiddler Http debugger to capture the XmlRpc request and response:

<?xml version="1.0"?>
<methodCall>
  <methodName>ticket.create</methodName>
  <params>
    <param>
      <value>
        <string>smazat</string>
      </value>
    </param>
    <param>
      <value>
        <string>smazat</string>
      </value>
    </param>
    <param>
      <value>
        <struct>
          <member>
            <name>type</name>
            <value>
              <string>defect</string>
            </value>
          </member>
          <member>
            <name>time</name>
            <value>
              <dateTime.iso8601>20131011T10:08:41</dateTime.iso8601>
            </value>
          </member>
          <member>
            <name>component</name>
            <value>
              <string />
            </value>
          </member>
          <member>
            <name>severity</name>
            <value>
              <string>highest</string>
            </value>
          </member>
          <member>
            <name>priority</name>
            <value>
              <string>blocker</string>
            </value>
          </member>
          <member>
            <name>owner</name>
            <value>
              <string />
            </value>
          </member>
          <member>
            <name>reporter</name>
            <value>
              <string>smazat</string>
            </value>
          </member>
          <member>
            <name>cc</name>
            <value>
              <string />
            </value>
          </member>
          <member>
            <name>version</name>
            <value>
              <string />
            </value>
          </member>
          <member>
            <name>milestone</name>
            <value>
              <string />
            </value>
          </member>
          <member>
            <name>status</name>
            <value>
              <string>new</string>
            </value>
          </member>
          <member>
            <name>resolution</name>
            <value>
              <string>invalid</string>
            </value>
          </member>
          <member>
            <name>summary</name>
            <value>
              <string>smazat</string>
            </value>
          </member>
          <member>
            <name>description</name>
            <value>
              <string>smazat</string>
            </value>
          </member>
          <member>
            <name>keywords</name>
            <value>
              <string />
            </value>
          </member>
        </struct>
      </value>
    </param>
  </params>
</methodCall>

<?xml version='1.0'?>
<methodResponse>
<fault>
<value><struct>
<member>
<name>faultCode</name>
<value><int>1</int></value>
</member>
<member>
<name>faultString</name>
<value><string>''datetime.datetime' object has no attribute 'strip'' while executing 'ticket.create()'</string></value>
</member>
</struct></value>
</fault>
</methodResponse>

Can you plese tell me what is wrong? What is different from the unittest?

[osimons]

Could you please also append the full error from trac.log? The traceback would be very useful for trying to understand your problem.

[kubes@…]

Here is a part of the trac.log:

2013-10-11 10:57:55,098 Trac[main] DEBUG: Dispatching <RequestWithSession "POST '/login/xmlrpc'">
2013-10-11 10:57:55,098 Trac[web_ui] DEBUG: LoginModule._remote_user: Authentication attempted for 'None'
2013-10-11 10:57:55,098 Trac[web_ui] DEBUG: LoginModule.authenticate: Set 'REMOTE_USER' = 'None'
2013-10-11 10:57:55,112 Trac[filter] DEBUG: HTTPAuthFilter: Authentication okay for builder
2013-10-11 10:57:55,112 Trac[session] DEBUG: Retrieving session for ID 'builder'
2013-10-11 10:57:55,128 Trac[main] DEBUG: Negotiated locale: None -> en_US
2013-10-11 10:57:55,628 Trac[svn_fs] DEBUG: Subversion bindings imported
2013-10-11 10:57:55,644 Trac[api] INFO: Synchronized '(default)' repository in 0.55 seconds
2013-10-11 10:57:55,644 Trac[web_ui] DEBUG: RPC incoming request of content type 'text/xml' dispatched to <tracrpc.xml_rpc.XmlRpcProtocol object at 0x01B37350>
2013-10-11 10:57:55,644 Trac[web_ui] DEBUG: RPC(XML-RPC) call by 'builder'
2013-10-11 10:57:55,644 Trac[xml_rpc] DEBUG: RPC(xml) call by 'builder', method 'ticket.query' with args: ('status!=asdfasdf&max=0',)
2013-10-11 10:57:55,644 Trac[api] DEBUG: action controllers for ticket workflow: ['ConfigurableTicketWorkflow']
2013-10-11 10:57:55,660 Trac[web_ui] DEBUG: RPC(XML-RPC) call by 'builder' ticket.query
2013-10-11 10:57:55,676 Trac[query] DEBUG: Count results in Query: 1
2013-10-11 10:57:55,676 Trac[xml_rpc] DEBUG: RPC(xml) 'ticket.query' result: [1]
2013-10-11 11:01:10,332 Trac[main] DEBUG: Dispatching <RequestWithSession "POST '/login/xmlrpc'">
2013-10-11 11:01:10,332 Trac[web_ui] DEBUG: LoginModule._remote_user: Authentication attempted for 'None'
2013-10-11 11:01:10,332 Trac[web_ui] DEBUG: LoginModule.authenticate: Set 'REMOTE_USER' = 'None'
2013-10-11 11:01:10,332 Trac[filter] DEBUG: HTTPAuthFilter: Authentication okay for builder
2013-10-11 11:01:10,332 Trac[session] DEBUG: Retrieving session for ID 'builder'
2013-10-11 11:01:10,348 Trac[main] DEBUG: Negotiated locale: None -> en_US
2013-10-11 11:01:10,598 Trac[api] INFO: Synchronized '(default)' repository in 0.27 seconds
2013-10-11 11:01:10,598 Trac[web_ui] DEBUG: RPC incoming request of content type 'text/xml' dispatched to <tracrpc.xml_rpc.XmlRpcProtocol object at 0x01B37350>
2013-10-11 11:01:10,598 Trac[web_ui] DEBUG: RPC(XML-RPC) call by 'builder'
2013-10-11 11:01:10,598 Trac[xml_rpc] DEBUG: RPC(xml) call by 'builder', method 'ticket.create' with args: ('smazat', 'smazat', {'status': 'new', 'severity': 'highest', 'reporter': 'smazat', 'cc': '', 'resolution': 'invalid', 'milestone': '', 'component': '', 'summary': 'smazat', 'priority': 'blocker', 'keywords': '', 'version': '', 'time': <DateTime '20131011T11:01:12' at 1b32c60>, 'owner': '', 'type': 'defect', 'description': 'smazat'})
2013-10-11 11:01:10,614 Trac[web_ui] DEBUG: RPC(XML-RPC) call by 'builder' ticket.create
2013-10-11 11:01:10,614 Trac[web_ui] ERROR: RPC(XML-RPC) Error
Traceback (most recent call last):
  File "build\bdist.win32\egg\tracrpc\web_ui.py", line 158, in _rpc_process
    result = (XMLRPCSystem(self.env).get_method(method_name)(req, args))[0]
  File "build\bdist.win32\egg\tracrpc\api.py", line 197, in __call__
    result = self.callable(req, *args)
  File "build\bdist.win32\egg\tracrpc\ticket.py", line 162, in create
    t[k] = v
  File "c:\docume~1\kubest~1.e-b\locals~1\temp\easy_install-7rt2br\Trac-1.0.1-py2.7-win32.egg.tmp\trac\ticket\model.py", line 160, in __setitem__
    value = value.strip()
ServiceException: 'datetime.datetime' object has no attribute 'strip'
2013-10-11 11:01:10,614 Trac[xml_rpc] ERROR: 'datetime.datetime' object has no attribute 'strip'
2013-10-11 11:01:10,614 Trac[xml_rpc] ERROR: Traceback (most recent call last):
  File "build\bdist.win32\egg\tracrpc\web_ui.py", line 158, in _rpc_process
    result = (XMLRPCSystem(self.env).get_method(method_name)(req, args))[0]
  File "build\bdist.win32\egg\tracrpc\api.py", line 197, in __call__
    result = self.callable(req, *args)
  File "build\bdist.win32\egg\tracrpc\ticket.py", line 162, in create
    t[k] = v
  File "c:\docume~1\kubest~1.e-b\locals~1\temp\easy_install-7rt2br\Trac-1.0.1-py2.7-win32.egg.tmp\trac\ticket\model.py", line 160, in __setitem__
    value = value.strip()
ServiceException: 'datetime.datetime' object has no attribute 'strip'

[osimons]

Ah. Re-reading the docs and code again, I see that you are indeed doing time wrong. This attribute has no meaning in Trac before the ticket actually exists, and therefore the call signature has a when argument to use for this special value. Also note that you need to be TRAC_ADMIN to be allowed to set a custom creation time.

Here is the actual current code for reference:

def create(self, req, summary, description, attributes={}, notify=False, when=None):
    """ Create a new ticket, returning the ticket ID.
    Overriding 'when' requires admin permission. """
    t = model.Ticket(self.env)
    t['summary'] = summary
    t['description'] = description
    t['reporter'] = req.authname
    for k, v in attributes.iteritems():
        t[k] = v
    t['status'] = 'new'
    t['resolution'] = ''
    # custom create timestamp?
    if when and not 'TICKET_ADMIN' in req.perm:
        self.log.warn("RPC ticket.create: %r not allowed to create with "
                "non-current timestamp (%r)", req.authname, when)
        when = None
    t.insert(when=when)
    if notify:
        try:
            tn = TicketNotifyEmail(self.env)
            tn.notify(t, newticket=True)
        except Exception, e:
            self.log.exception("Failure sending notification on creation "
                               "of ticket #%s: %s" % (t.id, e))
    return t.id

[kubes@…]

Briliant!!! That solve my issue. Thank you very much.

I had old signatures (from Trac 0.12) and there was string parameter, not DateTime. Then I regenerated method signatures and used the "when" parameter.

I suppose it would be more clear if the parameter names are consistant. The name of the time parameter is "time" in the ticket.get method and "when" in the ticket.create/update method.

Secondly what was confusing, that the time parameter can be obtain via dictionary, but cannot be set via dictionary, but via method parameter (which has different name).