Miscellaneous refactoring and minor clean ups

[Chris Nelson]

A ticket to link various small fixes to.

[Chris Nelson]

(In [11066]) Ignore .pyc files. Refs #9648.

[Chris Nelson]

(In [11067]) Don't know when this unnecessary blank line was introduced. Refs #9648.

[Chris Nelson]

(In [11068]) Fix resource name display. Refs #9648.

If no user names were configured, the resource showed up blank, not as a raw ID.

[Chris Nelson]

(In [11070]) White space, comments, minor refactoring. Refs #9648.

[Chris Nelson]

(In [11075]) Comments and white space. Refs #9648.

[Chris Nelson]

(In [11083]) Rework calendar offset adjustment to not go outside working hours. Refs #9648, #9533, #9300.

[Chris Nelson]

(In [11412]) Use [], not None, for children if parent not configured. Refs #9648.

We use the result of TracPM.children() in two ways:

• if children(ticket)
• for cid in children(ticket)

The former has the same result for None and [] but the latter needs a protecting if isCfg('parent') if children is None. Using [] saves that if.

[Chris Nelson]

(In [11413]) Allow configuration of start even when finish isn't. Refs #9648.

This isn't a common case but I'm trying to keep it flexible.

[Chris Nelson]

(In [11415]) Make TracPM DB access database agnostic. Refs #9648.

[Chris Nelson]

(In [11417]) Make TracPM.preQuery() safe for other plugins. Refs #9648.

When another plugin calls preQuery(), it may not have set default values for goal and root so rather than testing for the value, test to see if the value is present.

[Chris Nelson]

(In [11418]) It should be OK to have dependencies but not children. Refs #9648.

Showing tasks for goal= gets all predecessors and all predecessors' children. If no parent field is configured, don't try to process it.

[Chris Nelson]

(In [11421]) Move interface definitions to pmapi.py. Refs #9648.

[Chris Nelson]

(In [11425]) Computing the roots in a set of tasks is generally useful. Refs #9648.

Note that if no parent field is configured, all tickets are roots.

[Chris Nelson]

(In [11426]) A tree-aware task sorter. Refs #9833, #9042, #9290, #9300, #9691, #9784.

A task's priority depends on its parent's priority.

Includes a base sorter class to hold common sorter methods.

SimpleSorter and ProjectSorter derive from the base class.

(Copy effective priority list, just in case.)

Also remove unneeded debugging output. Refs #9648.

[Chris Nelson]

(In [11429]) Up version to 0.10. Refs #9648.

[falkb]

Your commits on 03/23/12 don't have made visible trouble here. After update it still seems to work well. Thanks.

[Chris Nelson]

Replying to falkb:

Your commits on 03/23/12 don't have made visible trouble here. After update it still seems to work well. Thanks.

That shouldn't be remarkable, of course, but I have had some missteps in the past and there was some major surgery here. I'm glad it's still working for you.

[Chris Nelson]

(In [11701]) Address pychecker complaints. Refs #9648.

Not all of them but at least all the local variable shadowing a built in and most of the issues with undeclared class variables.

Also renamed "cal" to "calendar" for clarity.

[Chris Nelson]

(In [11702]) Respond to code review feedback. Refs #9648.

[Chris Nelson]

(In [11703]) Don't resource level milestones. Refs #9648.

[Chris Nelson]

(In [12314]) Use TracPM.roots() to find root of set of tickets. Refs #9648.

[Chris Nelson]

(In [12315]) Allow parent-child relation. Refs #9648.

Mostly reuses logic from pred-succ relation so there isn't really any work for relation handling beyond adding the relation to the list of configurable relations; the rest is data-driven.

However, the relation handling builds lists and we previously handled parent as a singleton so some parent field handling needed tweaking:

• TracPM.parent() now returns None if there is no parent field or

relation configured or if there is no parent. Internal code was adjusted. The only user outside this module, tracjsgantt.py, works with this change.

• When creating a pseudo-ticket for milestones, set the parent list to

empty, not the previous place-holder ID 0.

• When normalizing and parsing a configured parent field, make a

one-element list of the parent ID

• When testing for children, check if the child ID is in the parent

list rather than the parent field is the child ID

Also fix a bug where parent/child field handling assumed the value of fieldName left over from a previous loop.

Note this was tested with a parent field, with a parent-child relation, and with no parent configuration.

Some plugins that support parent-child relationships may allow multiple parents (Subtickets does). Doing PM-ish things on projects broken down into phases broken down into deliverables broken down into tasks, etc. all assumes a tree structure which excludes multiple parents. If someone creates multiple parents, this will blow up. I'm OK with that.

[Chris Nelson]

(In [12316]) Refactor preQuery() to make _expand() usable elsewhere. Refs #9648.

Refactor _followLink() (nee _expand())) for flexibility.

• Don't include origin nodes in returned list
• Add a depth argument to control how many times to traverse link

[Chris Nelson]

(In [12317]) Add a function for finding related tickets. Refs #9648.

[Chris Nelson]

(In [12318]) Rename internal fields to avoid conflicts. Refs #9648.

Custom field names must start with a letter and have only letters, digits, and underscores. By adding a leading underscore to "calc_start" and "calc_finish", we avoid a potential conflict if the user used those as custom field names for their own purposes or even for fields.start and fields.finish values.

[Chris Nelson]

(In [12319]) Refactor date parsing. Refs #9648.

[Chris Nelson]

(In [12320]) More flexible setting of query fields. Refs #9648.

• TracPM.queryFields() lists all the fields that TracPM needs.
• Caller (in this case the Gantt chart) has to make sure to include those fields only once in query.

[Chris Nelson]

(In [12321]) Log some details in when scheduling goes a little wrong. Refs #9648.

[Chris Nelson]

(In [12322]) Make sure final tickets don't get scheduled twice. Refs #9648.

[Chris Nelson]

(In [12323]) Rename milestone_type to goal_ticket_type. Refs #9648.

Trying to eliminate confusion about what a "milestone" is.

[Chris Nelson]

(In [12324]) Assign open/closed status to milestone pseudo-tickets. Refs #9648.

[Chris Nelson]

(In [12330]) Refactor ticket filtering. Refs #9648.

This encapsulates a fairly long algorithm into a function for clarity.

Also, fix a bug where we sort too many times. Removing tickets in the filter shouldn't reorder them.

Also, always call _filter_tickets(); it applies omitMilestones so it has to be called even if no other display filter is specified.

[anonymous]

please, give a sign when your series of changes has finished, and we can dare update again

[Chris Nelson]

Replying to anonymous:

please, give a sign when your series of changes has finished, and we can dare update again

My goal is that the on-line version is at least stable at the end of each set of updates. The parent-child relation should be usable now.

I hope to push a major feature based on this refactoring this week.

[falkb]

Today I've tried an update to your latest SVN revision. I noticed it's all quite mixed up now, means it's worse than before.

I have my tickets grouped with parent tickets and display my milestone this way: [[TracJSGanttChart(milestone=foo,colorBy=status,userMap=0,format=week,schedule=alap,doResourceLeveling=0)]]

Before your latest commit avalanche the milestone looked like:

==========
 ----
     ---
        --
      =========
          ---
             --
           ==========
                  ---
                  ---

but now the parent tickets are not ordered by time anymore and it looks like

           ==========
                  ---
                  ---
==========
 ----
     ---
        --
      =========
          ---
             --

which results in a strange snarl of dependency arrows

[Chris Nelson]

Odd. I don't see strange sorting here. Can you try adding order=wbs to your chart and see if that helps?

[falkb]

nope, doesn't help. Also the milestone is displayed between the tickets.

[falkb]

How can I switch on some debug tracing for you?

[Chris Nelson]

Replying to falkb:

How can I switch on some debug tracing for you?

I haven't added very effective run-time logging. (I tend to put it in for development and take it out before release.)

There seem to be just 10 tickets. Can you post here an overview of their relationships:

Ticket	Duration	Due Date	Blocked By	Blocking	Parent
A
B				C	A
C			B		A

Even better if you can attach it as a CSV so I can just import it into my environment.

[falkb]

Good idea, please look at the attached csv file. I don't have duration but startdate and enddate. The according milestone is due 10/Dec/2012.

[falkb]

belongs to 40

[falkb]

Have a look at the attachment, it's a bit more than 10 tickets.

[Chris Nelson]

Replying to falkb:

Today I've tried an update to your latest SVN revision. I noticed it's all quite mixed up now, means it's worse than before.

I have my tickets grouped with parent tickets and display my milestone this way: [[TracJSGanttChart(milestone=foo,colorBy=status,userMap=0,format=week,schedule=alap,doResourceLeveling=0)]]

Before your latest commit avalanche the milestone looked like:

==========
 ----
     ---
        --
      =========
          ---
             --
           ==========
                  ---
                  ---

but now the parent tickets are not ordered by time anymore and it looks like

           ==========
                  ---
                  ---
==========
 ----
     ---
        --
      =========
          ---
             --

which results in a strange snarl of dependency arrows

I don't see any strange behavior. I'll attach what I see. The summary for the tickets imported is "th <your ticket number>-<relative ticket number>"

[Chris Nelson]

My Gantt of tickets from imported query.csv using current code at TH.

[anonymous]

Could you please give me an uncolapsed screenshot. Well, I have Trac-1.0, maybe some other versions of involved plugins, and some little hacks in your code. At least my hacks shouldn't matter, I've checked it already.

[Chris Nelson]

My Gantt with more tickets open

[Chris Nelson]

Replying to anonymous:

Could you please give me an uncolapsed screenshot.

Attached. I couldn't open 299 (your 2692) without going off the page and then I couldn't capture it.

Well, I have Trac-1.0, maybe some other versions of involved plugins, and some little hacks in your code. At least my hacks shouldn't matter, I've checked it already.

Yes, it might be some version dependency. Hmmm.

Can you post (or e-mail me) the embedded JavaScript in the page source? (You can clip it and obfuscate the ticket summaries, if you wish.)

[falkb]

OK with version before your refactoring

[falkb]

NOK with version after your refactoring

[falkb]

I analyzed a bit more (see result pictures below):

1. I used 0.12.2 again
2. I don't use my local hacks anymore, now clean from original svn sources

The related plugins I use are

• TracMasterTickets-3.0.2-py2.5.egg
• TracSubTicketsPlugin-0.1.0-py2.5.egg

Based on 1. and 2., this is what I see with revision r12213 before the refactoring session: and with latest revision r12330 now:

[falkb]

embedded javascript from page source, requested in 44

[falkb]

Replying to ChrisNelson:

Can you post (or e-mail me) the embedded JavaScript in the page source? (You can clip it and obfuscate the ticket summaries, if you wish.)

Here we go: attachment:falkb-open-0.12.2-r12330_javascript.txt​ (obfuscated)

[Chris Nelson]

Replying to falkb:

I analyzed a bit more (see result pictures below):

1. I used 0.12.2 again
2. I don't use my local hacks anymore, now clean from original svn sources

... Based on 1. and 2., this is what I see with revision r12213 before the refactoring session: ... and with latest revision r12330 now: ...

It appears to me that the tasks within each group are scheduled and ordered the same but that the top-level tickets are in a different order.

That order isn't numeric (ticket ID) or chronological (by start or finish). I'll look at the JS and see if I see a pattern.

[Chris Nelson]

Replying to falkb:

... The related plugins I use are

• TracMasterTickets-3.0.2-py2.5.egg

I'm using 2.2. However, I don't interact with the plugin, I just use its data.

• TracSubTicketsPlugin-0.1.0-py2.5.egg

I'm using the same.

...

[Chris Nelson]

Replying to ChrisNelson:

Replying to falkb:

... The related plugins I use are

• TracMasterTickets-3.0.2-py2.5.egg

I'm using 2.2. However, I don't interact with the plugin, I just use its data.

• TracSubTicketsPlugin-0.1.0-py2.5.egg

I'm using the same.

...

Are you using a field or a relation to configure the interface to those plugins? I have:

relation.parent-child = subtickets,parent,child
relation.pred-succ = mastertickets,source,dest

[falkb]

Replying to ChrisNelson:

Are you using a field or a relation to configure the interface to those plugins? I have:

relation.parent-child = subtickets,parent,child
relation.pred-succ = mastertickets,source,dest

Not sure what you mean. My trac.ini just contains for those fields:

[TracPM]
date_format = %Y-%m-%d
days_per_estimate = 0.125
fields.estimate = estimatedhours
fields.finish = userfinish
fields.parent = parents
fields.pred = blockedby
fields.start = userstart
fields.succ = blocking
fields.worked = totalhours
milestone_type = milestone

[Chris Nelson]

I don't see exactly what you see but I can make it worse with the before code (2677 ends up in the middle instead of at the end) if I use fields. Please try relations as noted above and see if it is better for you. (That is, replace the fields.parents, fields.pred, and fields.succ lines by the relation.* lines I quoted.

[falkb]

Do you mean from:

[TracPM]
fields.parent = parents
fields.pred = blockedby
fields.succ = blocking

to

relation.parent-child = subtickets,parent,child
relation.pred-succ = mastertickets,source,dest

or

relation.parent-child = subtickets,parents,?????
relation.pred-succ = mastertickets,blockedby,blocking

?

[Chris Nelson]

Replying to falkb:

Do you mean from:

[TracPM]
fields.parent = parents
fields.pred = blockedby
fields.succ = blocking

to

relation.parent-child = subtickets,parent,child
relation.pred-succ = mastertickets,source,dest

or

relation.parent-child = subtickets,parents,?????
relation.pred-succ = mastertickets,blockedby,blocking

?

The former. The setting is <table>,<from>,<to>. So, In the subtickets table (private to the Subtickets plugin), the parent field is the beginning of the arc and the child field is the end.

[falkb]

Using

[TracPM]
date_format = %Y-%m-%d
days_per_estimate = 0.125
fields.estimate = estimatedhours
fields.finish = userfinish
#fields.parent = parents
#fields.pred = blockedby
fields.start = userstart
#fields.succ = blocking
fields.worked = totalhours
relation.parent-child = subtickets,parent,child
relation.pred-succ = mastertickets,source,dest
milestone_type = milestone

Uhm.. well, now it's even worse. I don't have parent tickets anymore. It's simply an unstructured list of tickets. The milestone is in the middle, ticket order is different from what I've seen the tests before but still strangly mixed up.

[Chris Nelson]

Replying to falkb:

... Uhm.. well, now it's even worse. I don't have parent tickets anymore. It's simply an unstructured list of tickets. The milestone is in the middle, ticket order is different from what I've seen the tests before but still strangly mixed up.

relation wasn't supported in r12213. Are you using r12330?

[falkb]

yes, all my trials are with a running r12330

[Chris Nelson]

Replying to falkb:

yes, all my trials are with a running r12330

:-( I'll see if I can figure something out from the differences I see here.

[falkb]

Are you sure you committed everything? It's all very strange... I'm also stranded right now...

[Chris Nelson]

Replying to falkb:

Are you sure you committed everything? It's all very strange... I'm also stranded right now...

Yes. I checked out what I pushed to TH, rebuilt, and reinstalled and it's fine for me when I use relation. It's a little odd when I use fields. I still don't see what you see but that may be because I didn't set the task priorities so the scheduler may make slightly different choices. Sorry. I'll try to dig into this today.

[falkb]

The priorities are all default, except 3218 and 3456 are default+1, and 3449 is default+2

[Chris Nelson]

Replying to falkb:

I analyzed a bit more (see result pictures below):

1. I used 0.12.2 again
2. I don't use my local hacks anymore, now clean from original svn sources

The related plugins I use are

• TracMasterTickets-3.0.2-py2.5.egg
• TracSubTicketsPlugin-0.1.0-py2.5.egg

Based on 1. and 2., this is what I see with revision r12213 before the refactoring session: ... and with latest revision r12330 now: ...

While I doubt it is a significant difference, I note that in query.csv has 2682 as a child of 2681 but that is not shown on these charts.

[anonymous]

I'll check that at work tomorrow, I think it's because I retrieved the data from and did the tests with the copy of the live system and there were activities from one day to another.

[falkb]

Please, follow #10686 to read the further discussion between falkb and ChrisNelson because of the chart sorting problem.

[Chris Nelson]

(In [12426]) Assign WBS in correct order. Refs #10868, #9648.

The order of WBS assignments was broken by r12314.

TracPM.roots() returns an unordered list of ticket IDs so traversing that list meant we were no longer computing the WBS in the correct order. Now we use that list to determine what is a root ticket but traverse the ticket list in the scheduled (sorted) order.

Thanks to falkb for his help and patience while working on this.

[falkb]

Today I've found another issue which is broken since your latest activities within the last 4 weeks: I have a Trac custom report [[TracJSGanttChart(format=week,order=milestone|type|wbs,res=0,colorBy=priority,owner=$USER&status!=closed)]] which gave a survey over all open tickets to the user. Now it's not working anymore. No chart appears at all. I suspect it's because of no milestone is given.

[Chris Nelson]

(In [12746]) Make graph augmentation a TracPM method. Refs #9648.

Need to do this before trying to reschedule in ticket change listener.

[Chris Nelson]

(In [12747]) Add private table creation. Refs #9648.

I'm not sure that the name of the table, schedule, is sufficiently obscure that it won't step on users or other plugins.

We're going to store the schedule in a private table. We can use IEnvironmentSetupParticipant methods to check for the table and create as needed the first time the plugin is activated.

The general principles are at ​http://trac.edgewall.org/wiki/TracDev/PluginDevelopment/ExtensionPoints/trac.env.IEnvironmentSetupParticipant

The Trac schema is at ​http://trac.edgewall.org/browser/trunk/trac/db_default.py for comparison. I modeled schedule on ticket_change (int64 for datetimes) and ticket_custom (index by ticket to get values).

Some of the detail code came out of api.py in ​git://git.sixnetio.com/git/misc_tools/trac_plugin_source/trac-subtickets-plugin.git.

[Chris Nelson]

(In [12748]) Add a ticket change listener to detect related tickets. Refs #9648.

THIS CONTAINS SOME HARD-CODED FIELD NAMES AND WORKS ONLY WITH MASTERTICKETS AND SUBTICKETS. IT DOES NOT YET WORK WITH CHILDTICKETS OR OTHER RELATIONSHIP PLUGINS.

This doesn't do any rescheduling. It just notes how many tickets need to be rescheduled and how long it took to find them.

[Chris Nelson]

(In [12749]) Recompute schedule and save changes (if any) in a private table. Refs #9648.

• All datetimes used in schedule are now in local time zone.
  • Had to be "offset-aware" (to use to_utimestamp() utility function)
• TracPM.start(), .finish() now return calculated date, then stored date, then None depending on what is available.
• postQuery() now reads schedule from private table rather than having it queried from custom fields
• When the rescheduler is done calculating a schedule it uses INSERT or UPDATE to write only changed tickets back to the private schedule table.
• Changes to the schedule are stored in the schedle_history table.

This handles inserting initial schedule into DB and updating only changed tickets when rescheduling.

With the changes to TracPM.start(), .finish(), a caller (e.g., the Gantt chart) can:

• Query tickets
• Optionally schedule them
• Iterate over them getting the start and/or finish dates

If the scheduling step is skipped, the dates from the database are used.

There's a potentially inefficiency in how potentially inactive tickets are found. We might get this set by first getting all the affected tickets then following the predecessor links from the goal but there isn't a function for that yet and preQuery() is fairly efficient. I don't want to optimize prematurely.

For this to work, I have to splice the in-memory ticket graph based on ticket changes.

This has some gross code that is specific to plugins we use and makes TracPM less portable but I want to prove this works before trying to make it generic.

[Chris Nelson]

(In [12750]) Get actual start and finish for active and closed tickets. Refs #9648, #10717.

Use these dates in scheduling.

Note that closed tickets don't require resource leveling, display their actual start and finish times.

This defaults to disabled (0) and can be controlled by useActuals option in the TracPM section of trac.ini (1=use actual dates, 0=schedule). The useActuals parameter to the Gantt chart macro overrides the TracPM value.

[Chris Nelson]

(In [12753]) Add support for scheduled=1 to TracPM.query(). Refs #9648.

Allow retrieval of only tickets that have a precomputed schedule in the database.

[Chris Nelson]

(In [12754]) Scheduler more flexible about missing/defaulted options. Refs #9648.

The Gantt chart passes a bunch of options that come from the macro invocation or the chart configuration in Trac.ini. A lighter weight client -- like the ticket change listener -- may not have all those options and just wants default behaviors. Some missing values weren't handled as defaults before.

[Chris Nelson]

(In [12755]) Allow Gantt to request no scheduling (just show db values). Refs #9648.

This allows TracJSGanttChart(scheduled=1,schedule=none) to show tasks as they are in the schedule database.

[Chris Nelson]

(In [12757]) Improve "omitMilestones" handling. Refs #9648.

[Chris Nelson]

(In [12758]) Simplify Trac.pm.query() interface. Refs #9648.

[Chris Nelson]

(In [12759]) Move sort function to match master branch. Refs #9648.

[Chris Nelson]

(In [12760]) More consistent use of t.get(field) vs. t[field]. Refs #9648.

[Chris Nelson]

(In [12762]) White space. Refs #9648.

[Chris Nelson]

(In [12763]) Refine background ticket rescheduler. Refs #9648.

Move the exclusion of milestone pseudo-tickets closer to where it makes a difference (when we save the data).

This should be a no-op most of the time but without this the ticket set sometimes has dangling references which makes the plugin fragile when adding new function invocations in certain places.

[Chris Nelson]

(In [12764]) Bug in start/finish option experiment. Refs #9845, #9648.

[Chris Nelson]

(In [12969]) Background rescheduler sometimes fails to handle dependencies. Refs #9648.

[falkb]

Well, after a long time I've updated this Gantt chart plugin again. But now I came across those errors. Maybe you could have a look at this:

[2013-08-28 15:19:08,296 p2464:t4356] main.py:549 ERROR: Internal Server Error: 
Traceback (most recent call last):
  File "build\bdist.win32\egg\trac\web\main.py", line 497, in _dispatch_request
    dispatcher.dispatch(req)
  File "build\bdist.win32\egg\trac\web\main.py", line 214, in dispatch
    resp = chosen_handler.process_request(req)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 179, in process_request
    return self._process_ticket_request(req)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 614, in _process_ticket_request
    self._do_save(req, ticket, action)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 1328, in _do_save
    replyto=req.args.get('replyto'))
  File "build\bdist.win32\egg\trac\ticket\model.py", line 365, in save_changes
    listener.ticket_changed(self, comment, author, old_values)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2969, in ticket_changed
    self.rescheduleTickets(ticket, old_values)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2772, in rescheduleTickets
    tickets = self.queryTickets(ids)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2568, in queryTickets
    return self.pm.query(options, set())
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 1212, in query
    self.postQuery(options, tickets)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 1157, in postQuery
    self.getTicketDates(tickets)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 950, in getTicketDates
    ids)
  File "build\bdist.win32\egg\trac\db\util.py", line 65, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 78, in execute
    result = PyFormatCursor.execute(self, *args)
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 56, in execute
    args or [])
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 48, in _rollback_on_error
    return function(self, *args, **kwargs)
OperationalError: too many SQL variables
[2013-08-28 15:21:06,483 p2464:t4172] main.py:549 ERROR: Internal Server Error: 
Traceback (most recent call last):
  File "build\bdist.win32\egg\trac\web\main.py", line 497, in _dispatch_request
    dispatcher.dispatch(req)
  File "build\bdist.win32\egg\trac\web\main.py", line 214, in dispatch
    resp = chosen_handler.process_request(req)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 179, in process_request
    return self._process_ticket_request(req)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 614, in _process_ticket_request
    self._do_save(req, ticket, action)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 1328, in _do_save
    replyto=req.args.get('replyto'))
  File "build\bdist.win32\egg\trac\ticket\model.py", line 365, in save_changes
    listener.ticket_changed(self, comment, author, old_values)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2969, in ticket_changed
    self.rescheduleTickets(ticket, old_values)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2929, in rescheduleTickets
    values)
  File "build\bdist.win32\egg\trac\db\util.py", line 65, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 78, in execute
    result = PyFormatCursor.execute(self, *args)
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 56, in execute
    args or [])
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 48, in _rollback_on_error
    return function(self, *args, **kwargs)
OperationalError: near ",": syntax error
[2013-08-28 15:22:28,828 p2060:t3604] main.py:549 ERROR: Internal Server Error: 
Traceback (most recent call last):
  File "build\bdist.win32\egg\trac\web\main.py", line 497, in _dispatch_request
    dispatcher.dispatch(req)
  File "build\bdist.win32\egg\trac\web\main.py", line 214, in dispatch
    resp = chosen_handler.process_request(req)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 179, in process_request
    return self._process_ticket_request(req)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 614, in _process_ticket_request
    self._do_save(req, ticket, action)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 1328, in _do_save
    replyto=req.args.get('replyto'))
  File "build\bdist.win32\egg\trac\ticket\model.py", line 365, in save_changes
    listener.ticket_changed(self, comment, author, old_values)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2969, in ticket_changed
    self.rescheduleTickets(ticket, old_values)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2757, in rescheduleTickets
    ids = self._findAffected(ticket, old_values)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2459, in _findAffected
    toExplore = more(border)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2418, in more
    newOwners = ownersOf(n)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2338, in ownersOf
    list(ids))
  File "build\bdist.win32\egg\trac\db\util.py", line 65, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 78, in execute
    result = PyFormatCursor.execute(self, *args)
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 56, in execute
    args or [])
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 48, in _rollback_on_error
    return function(self, *args, **kwargs)
OperationalError: too many SQL variables

[Chris Nelson]

Are using the background ticket rescheduler? If not, try disabling it.

I can't tell what version you're running. Is it the most recent (r13368)? I don't see that code on those lines in that version.

I wonder if what you're seeing is another instance related to #11287.

[falkb]

Replying to ChrisNelson:

Are using the background ticket rescheduler? If not, try disabling it.

yes, I activated all of the plugin components as I noticed you've added new ones. Which ones are necessary?

I can't tell what version you're running. Is it the most recent (r13368)? I don't see that code on those lines in that version.

the latest SVN revision. I just had some small local patches but they should not matter in this case.

[Chris Nelson]

Replying to falkb:

Replying to ChrisNelson:

Are using the background ticket rescheduler? If not, try disabling it.

yes, I activated all of the plugin components as I noticed you've added new ones. Which ones are necessary?

The rescheduler is definitely optional, does nothing to make the Gantt work. I wish I could make it disabled by default but people who have tracjsgantt.* enabled get it when they update. (What is does is, whenever a schedule-related ticket change occurs -- ownership, estimate, etc. -- it reschedules all the related tickets and saves the schedule in a private table. We use that to drive our Task List reports but it's insufficiently documented for the rest of the world.)

I can't tell what version you're running. Is it the most recent (r13368)? I don't see that code on those lines in that version.

the latest SVN revision. I just had some small local patches but they should not matter in this case.

Oh, OK.

Can you send me a snippet that lets me know what

File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2338, in ownersOf
    list(ids))

refers to?

[Chris Nelson]

Yes, that's definitely related to #11287.

[falkb]

Replying to ChrisNelson:

The rescheduler is definitely optional, does nothing to make the Gantt work.

OK, now I've activated only

• TracJSGanttChart
• TracJSGanttSupport
• ResourceScheduler
• SimpleCalendar
• TracPM

The other components are off. I hope this is a valid combination.

Oh, OK.

My patch is that I also support versions beside milestones as schedule points in the chart (your internal milestones)

Can you send me a snippet that lets me know what

File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2338, in ownersOf
    list(ids))

refers to?

    def _findAffected(self, ticket, old_values):
        # Helper to find owners of tickets
        # @param ids set of tickets ID strings
        # @return set of owner strings
        def ownersOf(ids):
            db = self.env.get_db_cnx()
            inClause = "IN (%s)" % ','.join(('%s',) * len(ids))
            cursor = db.cursor()
            cursor.execute(("SELECT DISTINCT owner FROM ticket "
                            "WHERE id " + 
                           inClause),
L2338                      list(ids))
            owners = [row[0] for row in cursor]
            return set(owners)

[Chris Nelson]

See ​http://trac-hacks.org/ticket/11287#comment:4 (forgot a # in the commit message).

[Chris Nelson]

In 13401:

Trivial white space change. Refs #9648.

Need to test my ability to push to T-H.

[Chris Nelson]

In 13811:

Clean up .gitignore Refs #9648

[Chris Nelson]

In 13812:

Don't consider group start, finish when resource leveling. Refs #9648.

Consider one group and separate task. All tasks are one unit of work.

• Group A is owned by Mickey
• Tasks B, C, and D are children of A. All are owned by Minnie
• Task E is owned by Mickey and has no dependencies.

An ASAP scheduler would try to schedule, which requires scheduling B,
then C, then D. When resource leveling, they will happen one after the
other. The way the existing algorithm bubbles up finish dates, Group A
gets a finish of 3 and that is assigned to the group owner, Mickey.

Now the scheduler tries to schedule E. Checking the resource limits, it
thinks Mickey isn't available until 3 and scheduled E from 3 to 4,
leaving Mickey idle for three units at the start of the schedule.

Since there is no scheduled work in a group, the owner's availability is
not affected by the work done in the children. This change skips
consideration and update resource limits if a task has children.

[Chris Nelson]

In 13818:

Refactor body of ALAP function into a helper. Refs #9648.

This is the first step in making the logic common between ALAP and ASAP.

[Chris Nelson]

In 13819:

Make ancestor limit function a parameter. Refs #9648.

[Chris Nelson]

In 13820:

Make dependent limit function a parameter. Refs #9648.

[Chris Nelson]

In 13821:

Change 'finish' to 'from' Refs #9648.

The ALAP scheduler works *from* the finish toward the start. The ASAP
scheduler works from the start toward the finish. "from" is always the
"leading" edge of a task determined by other constraints.

[Chris Nelson]

In 13822:

Change 'start' to 'to'. Refs #9648.

The ALAP scheduler works from finish *to* start. The ASAP scheduler
works from start *to* finish. "to" is always the trailing edge of the
task, computed from "from" and the task duration.

[Chris Nelson]

In 13823:

Generic task date parser. Refs #9648.

Sometimes we parse start, sometimes finish.

[Chris Nelson]

In 13824:

Make limit comparison a function parameter. Refs #9648.

For ALAP we update resource limits going from finish to start. The
direction of comparision between a possible date and the resource limit
has a different sense than for ASAP; one uses < and > and the other uses

and <. We can't pass operators to the generic schedule function so we
pass a comparison function.

[Chris Nelson]

In 13825:

Make logs, comments generic. Refs #9648.

Take "start" and "finish" out of text.

Also clarify resource leveling criteria.

[Chris Nelson]

In 13826:

Make end of day wrapping a function parameter. Refs #9648.

An ALAP schedule wraps from the beginning of one day to the end of the
previous. An ASAP schedule wraps from the end of one to the beginning
of the next. This function encapsulates that logic.

[Chris Nelson]

In 13827:

Add Doxygen to _schedule_task(). Refs #9648.

[Chris Nelson]

In 13828:

Refactor _schedule_task_asap() using parameterized scheduler. Refs #9648.

All the heaving lifting was already done for ALAP.

[Chris Nelson]

In 13830:

Clarify an option description. Refs #9648.

[Chris Nelson]

In 13832:

Add a lot of logging (it's off by default). Refs #9648.

This is controlled by self.logEnabled which comes from logScheduling
in the [TracPM] section of trac.ini. It defaults to disabled if not
set.

[Chris Nelson]

In 13837:

Use numerical precedence levels for start/finish sources. Refs #9648.

Before a date was either explicit or not. Values in the database
(actual start/finish dates, previously scheduled values, and explicit
user-supplied values) were all explicit so a test for "better" between
an old schedule table value and a new actual value were not resolved.
Now each source of a date has a different precedence so we have enough
information to resolve a conflict.

The way dates were calculated, we should not have seen conflicts in
new values, only in the explicit ones. This change allows an actual
start to be higher precedence than a previously-scheduled finish.
However, there is still the potential conflict between items of the
same precedence.

• Two actual values should never be wrong (we can't finish before we start)
• Two scheduled values should never be wrong if we never save bad values
• If two user-supplied values are wrong, it's data entry, not

algorithm. We could validate this away but we don't currently even
allow explicit start dates.

[Chris Nelson]

In 13838:

Factor out schedule and schedule_change update. Refs #9648.

This makes the code which finds affected tickets clearer, closer.

[Chris Nelson]

In 13839:

Rewrite algorithm to identify tickets to reschedule. Refs #9648.

This is a shorter, cleaner algorithm:

• Find what's active now
• Find what was active
• wasActive - isActive should be idled
• isActive should be rescheduled
  • Save only if schedule changes

It also works. ;-) With the old algorithm if you has two goals scheduled
and interrupted the earlier one, the tasks for the second goal weren't
moved up. The code to find active and idled tickets was weak and
fragile and not worth fixing since it was also slow. In general, this
new approach handles a lot less data, defers getting ticket details as
long as possible, and gets them for many fewer tickets. In my
playground, I used to query 150 tickets (basically, everything in the
database) and now I query 4-5 tickets (the active set). I expect that
ratio to scale to production. Since querying for ticket details was the
longest part of rescheduling, this is much, much faster than the old
approach.

[Chris Nelson]

In 13840:

Factor our graph repair. Refs #9648.

[Chris Nelson]

In 13841:

Simplify interface to prune and repair routines. Refs #9648.

[Chris Nelson]

In 13842:

Don't schedule closed tickets in background rescheduler. Refs #9648.

Also add some logging.

[Chris Nelson]

In 14199:

Use Trac 1.0 DB API to follow links. Refs #9648.

[Chris Nelson]

In 14200:

Use Trac 1.0 DB API for finding scheduled tickets. Refs #9648.

[Chris Nelson]

In 14201:

Use Trac 1.0 DB API to find milestones. Refs #9648.

[Chris Nelson]

In 14202:

Use Trac 1.0 DB API to find ticket dates. Refs #9648.

[Chris Nelson]

In 14203:

Use Trac 1.0 DB API to follow relations. Refs #9648.

[Chris Nelson]

In 14204:

Use Trc 1.0 DB API to build enum map. Refs #9648.

[Chris Nelson]

In 14205:

Use Trac 1.0 DB API to find owners. Refs #9648.

Conflicts:

tracjsgantt/tracpm.py

[Chris Nelson]

In 14206:

Use Trac 1.0 DB API to find tickets by owner. Refs #9648.

[Chris Nelson]

In 14207:

Use Trac 1.0 DB API to find active tickets. Refs #9648.

[Chris Nelson]

In 14208:

Fix a typo. Refs #9648.