Memory leaks in linq on mono

Description

When I do many queries using LINQ, process allocates much memory and do not releases it, even if I invoke GC.Collect() method.

There is an archive attached that contains example project which reproduces this bug.
After 200-300 cycles of reading some data, process uses about 100MB and this value will continue to grow. And forced garbage collecting does not help.

When using Mapper.Fetch<>() with string cql query, memory usage is kept at a constant level. (for me about 45MB)

Environment

Mono v4.6.2 on Ubuntu

Activity

Show:
Jorge Bay Gondra
December 23, 2016, 10:15 AM

I'm not able to reproduce, both under Mono 4.4.2 and .NET 4.5.2 on Win 7 x64.

Under mono, the allocated memory (private working sets) grows until it reaches a limit, in my case ~70Mb.
The total number of bytes allocated according to the GC after collecting stays steady at ~9mb, ie:

Under .NET, the behaviour is more or less the same. The allocated memory limit is a little lower ~45mb and the total number of bytes allocated (after collecting) according to the GC stays steady at ~0.5mb.

Using dotMemory profiling session under .NET, we can see that the graph stays steady and most of the allocated memory is related to unmanaged code (probably due to IOCP calls):

The difference in the allocations (after GC) is likely due to what are the boundaries for managed / unmanaged code in one runtime and the other. .NET uses relies more on unmanaged code than mono.

Jorge Bay Gondra
December 23, 2016, 10:17 AM

BTW, thanks for providing a clean project to test with: nice!

Сергей Спиридонов
December 23, 2016, 10:55 AM

This issue occurs on Ubuntu. We reproduce it on Ubuntu 14.04 Server and Ubuntu 16.04 Desktop. In both cases the latest version of Mono was used (v4.6.2).

After 1st cycle of reading the level of allocated memory was at ~75MB
After 2nd => ~90MB
After 3rd => ~100MB

And GC always shows ~9MB after collecting

Now I will try to reproduce the bug on mono 4.4.2. I will write later about the results

Сергей Спиридонов
December 23, 2016, 12:21 PM
Edited

On Mono 4.4.2 on Ubuntu 16.04 Desktop I have the same situation. After executing the 3rd item 10 times (and memory collecting) the allocated memory grows to 140MB, but GC shows ~9MB

The memory will leak faster if one more column will be added into partition and also one or two clusterings will be added (to make linq query bigger). In this case I reach 200MB after executing 3rd item only 5 times (GC, of course, shows the same ~9MB)

Jorge Bay Gondra
December 24, 2016, 4:18 AM
Edited

The amount of memory allocated by the runtime (private working set) is not something to worry, as it's up to the runtime to decide and allocate that memory space preemptively from the OS (from time to time). You can expect different behaviours by: runtime, OS and hardware specs.

There are several memory heaps that as a sum, accounts for the working set. In the case of the .NET profiling session we saw before, unmanaged memory accounted in the base heap accounted for ~85% of the total.

To identify a memory leak, we should look that managed memory (visible by the GC) grows in time, which it doesn't look to be the case. When this happens, we usually see 2nd generation heap size to increase in time, without being affected by GC collections.

As a note, if you want to have a more deterministic view of the total size of the GC heap after a forced GC, you should use:

Which is somehow equivalent to:

GetTotalMemory(true) will call GC.WaitForPendingFinalizers(); GC.Collect(); until the value stabilizes.

I will close this ticket, if you see other indications of a memory leak, we can reopen it.
Thanks again for providing a concise code sample to test.

Not a Problem

Assignee

Unassigned

Reporter

Сергей Спиридонов

Labels

None

Reproduced in

3.2.0

PM Priority

None

Fix versions

None

External issue ID

None

Doc Impact

None

Reviewer

None

Pull Request

None

Epic Link

None

Sprint

None

Pull Requests

None

Size

None

Components

Affects versions

Priority

Major