Tcl Source Code

Hi guys,

I'm trying to discover some memory leaking that I've in my application and I came up to the next code (see attached file).

The script creates a random number of global variables and dumps them to a file. Has you can see, after dumping the globals to the file the memory grows and this increment is never completely recovered.

Start - Physical: 14,404 / Virtual: 8,788
Globals Created - Physical: 82,588 / Virtual: 77,188 > Difference (Physical: 68,184 / Virtual: 68,400)
DB Saved - Physical: 101,696 / Virtual: 100,436 > Difference (Physical: 19,108 / Virtual: 23,248)
Modify - Physical: 92,500 / Virtual: 88,612 > Difference (Physical: -9,196 / Virtual: -11,824)
DB Saved - Physical: 100,264 / Virtual: 101,284 > Difference (Physical: 7,764 / Virtual: 12,672)
Modify - Physical: 91,060 / Virtual: 86,572 > Difference (Physical: -9,204 / Virtual: -14,712)
DB Saved - Physical: 100,420 / Virtual: 101,180 > Difference (Physical: 9,360 / Virtual: 14,608)
Modify - Physical: 93,220 / Virtual: 89,788 > Difference (Physical: -7,200 / Virtual: -11,392)
DB Saved - Physical: 99,944 / Virtual: 101,248 > Difference (Physical: 6,724 / Virtual: 11,460)

To execute the script in Windows you'll need the pslist.exe from PSTools, in Linux is using the ps command. I think that the ps command works different in OSX, but I haven't tested.

Am I missing some obvious behavior in Tcl? or is really some memory leaking?

Thanks a lot.

Logged In: YES 
user_id=75003
Originator: NO

Well, I do not see leaking in the version with threads. In most cycles the difference is 0, in a few cycles the difference is positive (= growing), in between are always cycles where the difference is negative (= memory usage goes down), and overall the memory stays the same. Over several hundred cycles (600) it started at 49500 virtual after initial allocations and never went above 50000 virtual. The physical side tracked this just at lower level. In other words I did not see any continuous increase.

> As a related, but a bit different question, do you known why is the 
> script using so much VM, compared with physical?

I cannot say. This is data from the operating system and not under the influence of the Tcl interpreter at all. The interpreter doesn't even know that some its memory (is / may be) virtual. At its level it simply has memory. Anything else is handled by the operating system.

File Added - 271946: memleak.tcl

Logged In: YES 
user_id=340696
Originator: YES

Hi Andreas,

In your first reply you said 

"in your actual server,do you have it growing with every dbsave ? if yes I
would assume that your are somewhere holding on to the old lists and thus
their memory is not released, i.e. I assume that their ref count never goes
to zero."

I think I found the "problem" using Threads. In our real server we use Threads in an intensive way. We almost don't use thread shared variables because we known by experience that are really slow for big variables, so everytime that we need to work with variables in threads, to read or modify them, we use the 'set' command.

Attached we can find a new modification of the script were we can see that the memory keeps increasing. In the example I've tried doing the 'set' directly and with a proc, with no difference. In our system it's worse since we have multiple threads.

Is something wrong in the way that I'm using the Threads? If so, please, how should I do it?

Thanks.
File Added: memleak.tcl

Logged In: YES 
user_id=75003
Originator: NO

My understanding of your code is this:

create_globals - creates 0-1499 variables, each contains a list between 0-2499 elements, each element a string of 0-14 characters.
modify_globals - you copy all variables around without changing them. using the loop you consume one additional list object and array of pointers to elements instead of just sharing the list. this blows the memory up some, but is stable as the old list object and pointer array should be returned to the allocator in each cycle.

dbsave - here you create the string-rep of your lists and save them to a file. memory usage grows again, allocating the space needed for the string-reps. the memory used by the string reps is released back to the allocator in the next round when the old list rep is destroyed by modify_globals. the channel system may use some memory for the buffering before writing to the OS. these are never released, but are re-used, so memory usage will not grow unbounded there either.

the ramp up of memory usage happens because you start with pure lists (i.e. without string rep), then you add memory for the second list, then the memory for the string rep of that list, and then the cycle begins where a second string rep is made and memory for old lists and string reps released, causing the pulsation between the two levels. in 8.4 you would see no pulsation as the memory is released to Tcl, but not the system allocator. there the max memory usage is simply a flat line.

in your actual server,do you have it growing with every dbsave ? if yes I would assume that your are somewhere holding on to the old lists and thus their memory is not released, i.e. I assume that their ref count never goes to zero.

this would have nothing to do with the file system, or sockets, or channels in general.

the test script behaves throughly normal.

File Added - 270307: memleak.tcl

Logged In: YES 
user_id=340696
Originator: YES

Hi Andreas,

Just added a "package require platform" and an update in the loop so I can see the progress in wish.

Regarding the 3 cycles, as you can see seems that only "leaks" in the first dbsave, after that seems stable. Anyway I'm still working to find a simple test for the next problem that I've, that seems related with this one somehow.

The other problem, in the real application, is that when a user connects to the server, the next dbsave consumes a bit more than 1 MB extra and 2 MB extra in VM. The connection process uses a socket and a thread. But I'm still trying to simplify the whole system to something small enough that can be tested and reproduced, and discard any bug in my code.

Anyway I would appreciate if the behavior that we see in this script is normal or not.

Thanks.
File Added: memleak.tcl

File Added - 270306: memleak.tcl.dat

Logged In: YES 
user_id=75003
Originator: NO

The attached file contains the output of the modified test script, for 14 modify/save iterations.
What I see is the memory usage ramping up in the first steps, and then cycling between two levels, both of which appear stable over the long-term. This is also seen in the memory usage differences which cycle between +/-N after the initial ramp-up.

I cannot confirm memory leaking. This was an 8.5.1+ build on linux, from around Feb 2. Using an 8.5.2b1 build using the CVS head as of last night shows the same pattern, i.e. ramp-up and cycle, but no long-term growth.

File Added: memleak.tcl.dat

Logged In: YES 
user_id=75003
Originator: NO

Attaching a revision of your test script. Two changes.
One, ensured aligned output to make the format tabular
Second, replaced the three modify/save cycles with an endless loop.

The first, because I can read table data much easier than something where the numbers zig-zag across the screen.
The second, because 3 iterations is often not enough to see how the memory behaves.


File Added: memleak.tcl

File Added - 270305: memleak.tcl

File Added - 270257: memleak.tcl