VCS and template object clean up to aid iteration speed and memory footprint

[durack1]

There have been a number of issues that describe code slowdowns and considerable memory growth when generate pngs using loops. Some of the problems are highlighted in the issues:
#241
#237
#236
#235
PCMDI/amipbcs#10
CDAT/cdat#1424
CDAT/cdat#1397

It appears the primary problem, is that vcs and EzTemplate objects are being created and never deleted, and over an iteration of ~8000 this can lead to a huge memory footprint (~1 -> 100Gb) and a very marked slowdown (~3 to 2000s) in the time taken to generate a single 3 panel png output file.

There are a number of ways to resolve this, and I believe the vcs.removeobject function should be augmented to provide a user the ability to purge all vcs (and EzTemplate or all used graphic) objects in the current session, so a command like:

vcs.removeobject('all',force=True)

This example function would purge ALL objects, even if there are current handles to them (the force=True option).

It would be useful to think about other use cases, and then incorporate these into the single function, rather than having multiple functions that do variants of the same thing.. Oh yes, and documentation of this functionality will be key

@lee1043 @doutriaux1 @danlipsa pinging you all here

[doutriaux1]

@durack1 this is misleading,

• all objects created by vcs are now being cleaned up. User generated object are NOT and SHOULD NOT be deleted by anyone but the user. Poor memory management by the user cannot and should not be blamed on the software. There is currently only one known case of memory growth which is when using vtk built against mesa. According to @danlipsa calling canvas.close() after a canvas.clear() fixes the issue, also switching to OpenGL2 (expected for the 3.0 release) solves the issue
• your proposed solution will NOT work as vcs.removeobject, removes ONE object. Agreed that we need to add code to make sure the object is not on use by any canvas before deleting it.
  remove("all") is not a viable solution as it s confusing, are your trying to remove a graphic method named "all" what about "all_boxfill", etc... The proper way to handle what you suggested is to implement a new function in Adding reset function for @durack1 WITH inline documentation #241 we named vcs.reset() to handle it. Probably vcs.clear() would be more appropriate, we can discuss the best possible name at length.
• Finally once we fixed removeobject to NOT delete objects in use, we should not implement a force=True option, but have removeobject return a clear error object indicating all dispalys and objects preventing the deletion of the requested object. So that the user can act accordingly.

[durack1]

@doutriaux1 feel free to edit the top #242 (comment) above, I was just trying to get down succinctly what the problem was, and what I thought was a reasonable solution - my aim was to reduce the tweaking of object deletion repetition (see below) and also the other kludges to make the software more user friendly..

FYI, my code includes the following:

vcs.removeobject(iso1)
vcs.removeobject(iso2)
vcs.removeobject(title)
vcs.removeobject(t1)
vcs.removeobject(t2)
vcs.removeobject(t3)
vcs.removeobject(tmpl)
...
gc.collect() ; # Attempt to force a memory flush
x.backend.renWin = None ; # @danlipsa fix UV-CDAT/vcs#237
x.close()

I am issuing an x.close at each loop iteration, but the objects associated with vcs canvas x (x = vcs.init()) were not being purged so the kludges above are currently required with the 2.10-19-gba336f2 nightly dated 22nd August

I would also note that a user cannot be expected to know to issue a x.backend.renWin = None command, and so there still is some cleaning up left to do

[durack1]

@downiec @jasonb5 @gabdulla this is the bug that I was talking about this morning

[durack1]

@downiec sounds good, I am also onsite tomorrow and this would certainly benefit from a screen showing. The script that really diagnoses the issue really well (plot timing starts at 3s and blows out to ~900+s after ~1000 loop steps) so it should be relatively easy to ascertain what the issue is and squash this long-standing bug once and for all

[durack1]

@downiec that demo file for testing is at https://github.com/PCMDI/amipbcs/pull/22/files

[durack1]

@forsyth2 @muryanto1 this was the VCS issue that I was having problems with

@downiec ping

[danlipsa]

@durack1 Did you tried your test with vcs nightly?