memory problem

kimmokotanen
kimmokotanen Posts: 3 Observer

Subject: Kernel pool memory leak — rtp1.sys/rtp2.sys retain process object
references (~250,000 zombie processes, ~2.8 GB kernel pool in 48 h)

PRODUCT AND ENVIRONMENT

  • Product: F-Secure 26.6 (updated 2026-07-11)
  • Drivers involved: rtp1.sys and rtp2.sys — "Avira real-time protection
    filter driver", file version 1.1.2604.8565, file date 2026-06-16, both
    running as kernel services (rtp1, rtp2)
  • OS: Windows 11 Pro for Workstations, build 26200, x64
  • Hardware: 32 GB RAM
  • Machine profile: software development workstation that spawns large numbers
    of short-lived processes (Node.js/npm/tsc builds, scheduled batch jobs)

SYMPTOM
Available RAM declines steadily over days of uptime (measured: 13.7 GB free
-> 5.9 GB free in ~18 hours, continuing even while the machine is idle
overnight). The machine becomes progressively slower and only a reboot
recovers the memory. No user-mode process accounts for the loss; the growth
is in kernel pool.

MEASUREMENTS (after ~48 h uptime)
Kernel pools (Win32_PerfFormattedData_PerfOS_Memory):

  • Pool Nonpaged: 2.20 GB (normally ~0.3-0.8 GB on this machine after boot)
  • Pool Paged: 2.38 GB

Pool-tag breakdown (NtQuerySystemInformation / SystemPoolTagInformation,
equivalent to poolmon). "Outstanding" = allocations minus frees:

Tag Pool Used Outstanding allocs

Proc NonPaged 852.2 MB 249,317
MiP2 NonPaged 365.2 MB 249,318
AVBh NonPaged 72.4 MB 249,624 <- tag string found in rtp1.sys/rtp2.sys
Even NonPaged 64.7 MB 529,975
PsIn NonPaged 39.3 MB 249,316
SeTl NonPaged 34.7 MB 284,031
EtwS NonPaged 15.2 MB 249,291
PLE0 Paged 735.0 MB <- tag string found in rtp1.sys/rtp2.sys
Toke Paged 501.6 MB
SeAt Paged 105.8 MB

Driver attribution: findstr /m /l AVBh C:\Windows\System32\drivers\*.sys
and findstr /m /l PLE0 ... both match ONLY rtp1.sys and rtp2.sys.

ANALYSIS
The outstanding-allocation count ~249,000 repeats across Proc (process
objects), MiP2, PsIn, EtwS, AVBh and closely matches Toke (tokens) sizing.
This indicates ~249,000 terminated processes whose kernel objects (EPROCESS,
token, memory-management structures, security/telemetry blocks) are never
released — i.e. the real-time protection filter driver appears to take a
reference on every created process and never dereference it after the
process exits. On this development machine roughly 250,000 short-lived
processes are created over two days, so the pools grow by roughly 1.4 GB/day
until the system is starved and must be rebooted.

The count of retained objects grows in bursts that correlate with build
activity (hundreds of node/tsc processes per build), not with any specific
application.

ADDITIONAL NOTES

  • No Avira product is installed on this machine; the rtp drivers were
    delivered as part of F-Secure (Avira engine).
  • The symptom predates F-Secure 26.6 (the machine has required reboots due
    to memory exhaustion for weeks), so 26.6 did not introduce it, and did not
    fix it either.
  • Windows Task Manager shows nothing abnormal per-process, which is probably
    why this leak has few public reports; discussion
    https://community.f-secure.com/en/discussion/129269/endpoint-protection-service
    contains user complaints about Endpoint Protection Service resource usage
    that may share this root cause.

REQUEST

  1. Please confirm whether this is a known issue in the Avira RTP filter
    driver 1.1.2604.8565 and whether a fix is scheduled.

Thank you.

Answers

  • TVC15
    TVC15 Posts: 282 Rising Star
    edited July 13

    Hello,

    I'm running Total on 2 laptops, and have not noticed this issue, v26.6 and v26.6 beta 2, on Windows 11. So I'm not sure how widespread this may be and if there is a fix in the works?

    IMO, I would contact off forum support and see what they have to say, and if they wanted you to submit log files. Otherwise, you could continue to wait for a response here from a mod.

    https://www.f-secure.com/en/support

  • Chameni
    Chameni Posts: 308 Moderator

    Hi @kimmokotanen,

    Thank you for this, it's one of the most thorough reports we've received, and the pool-tag analysis and driver attribution are extremely helpful. This gives our team a strong starting point.

    We need to escalate your report to our respective directly with all the detail you've provided (outstanding-allocation counts across Proc/MiP2/PsIn/AVBh, the correlation with short-lived build processes, and the driver findstr attribution).

    To help them reproduce and investigate as quickly as possible, could you please provide:

    1. An FSDIAG diagnostic bundle — run the F-Secure Support Tool (fsdiag) while the machine is in the leaked state (after significant uptime), so the logs capture the condition rather than a fresh boot.
    2. A poolmon snapshot (or your NtQuerySystemInformation output) taken at two points — shortly after boot and after the leak has grown — so the delta is captured.
    3. Confirmation of whether disabling real-time protection temporarily halts the outstanding-allocation growth (this would help confirm the RTP driver as the source).

    I will send you a DM, you can reply me with all the details and ill make sure your report reaches the right team, and we'll update you as soon as we have their assessment. Thank you again for the exceptional level of detail, it's genuinely appreciated.

    Best regards,
    Chameni / F-Secure

  • hjellmarr
    hjellmarr Posts: 1 New Member

    Same issue on another machine, first on 26.6 and still on 26.8. What I can add: how much is retained per process, a trend on an otherwise idle machine, and why ordinary laptops may never show it.

    Here is the AI-generated report:

    System

    HP ZBook 15 G5, 32 GB RAM, Windows 11 Pro build 26200 x64.
    Windows Defender is off (AMRunningMode: Not running), so F-Secure is the only
    active AV. VBS and HVCI are on and enforced, Credential Guard is running,
    VirtualMachinePlatform is installed but the full Hyper-V role is not.

    State after 46 hours on 26.6

    physical in use : 31.27 GB of 31.59 GB      available     : 0.33 GB
    paged pool      :  7.35 GB                  standby cache : 0.32 GB
    nonpaged pool   :  4.03 GB
    
    Tag      Paged MB   Nonpaged MB   Outstanding
    PLE0       3952.8           0.0       445,417
    Proc          0.0        1524.6       446,621
    Toke        947.8           0.0       457,058
    MiP2          0.0         653.4       446,622
    DAV3        203.9           0.0       445,417
    DAV0        139.6           0.0       444,637
    AVBh          0.0         129.1       445,996
    DAV1         68.7           0.0       445,417
    

    With 0.32 GB of standby cache there is effectively no file cache left, which
    is what made the machine unusable rather than merely full.

    These eight tags hold 7.44 GB between them, about 17,900 bytes per retained
    process object. Their outstanding counts are within 3% of each other, so one
    allocation of each is retained together.

    Attribution

    Searching for the tag strings across all driver binaries maps each
    non-Microsoft tag to exactly two files:

    findstr /M /L /C:PLE0 C:\Windows\System32\drivers\*.sys
      C:\Windows\System32\drivers\rtp1.sys
      C:\Windows\System32\drivers\rtp2.sys
    

    The same holds for DAV0, DAV1, DAV3 and AVBh. Proc, Toke and MiP2 are
    Windows' own tags. The kernel frees an EPROCESS only when nothing references
    it any longer, so 446,621 outstanding Proc allocations mean process objects
    are kept alive long after their processes exited, and that count tracks tags
    that exist only in rtp1.sys and rtp2.sys.

    Open handles in user mode cannot explain it. About two hours before this
    sample, the handles open in all processes together, of every type, numbered
    209,408, less than half the outstanding Proc count above.

    Retention per process on 26.8

    The product updated from 26.6 to 26.8 between my measurements. rtp1.sys and
    rtp2.sys are byte-identical across that update (1.1.2607.9217, 2026-08-18,
    480,848 bytes). Everything below is on 26.8.

    Idle, load and idle intervals, where the load creates 200 short-lived
    processes with cmd.exe /c exit. Changes in outstanding allocations,
    measured 5 seconds after the load:

                     run 1, load 4.9 s              run 2, load 6.5 s
                 idle   load   per process     idle   load   per process
    Proc            5    204          1.02       49    213          1.07
    Toke            5    205          1.03      108    207          1.04
    PLE0            8    384          1.92       13    433          2.17
    

    "idle" is the idle interval of the same length right after the load, and
    "per process" is the load change divided by 200. The background was quiet in
    run 1, and subtracting its idle interval gives 1.00, 1.00 and 1.88. It was
    noisy in run 2, where idle changes around the load ranged from -139 to +108,
    so run 2 is best read as gross figures.

    Short-term and long-term behaviour

    Part of this is released again within minutes. Tracking 300 processes for
    7 minutes:

                  PLE0   Proc
    baseline      2278   1943
    +5 s          +498   +294
    +30 s         +278   +282
    +120 s        +217   +226
    +300 s        +380   +364
    +420 s        +498   +495
    

    Within two minutes PLE0 fell back by more than half and Proc by about a
    quarter. After that the counts rose again with no further load from the
    test, so other process creation on the machine dominated, and this test
    cannot say how much of the 300 is eventually released.

    The long-run counts answer that instead. Proc and PLE0 differed by 0.3% at
    46 hours, and by 0.4% at 4.8 hours on 26.8 (below). Over hours, roughly one
    PLE0 allocation remains per retained process object, and the counts keep
    growing until a reboot.

    A three-hour trend on an idle machine, 26.8

    Sampled every ten minutes with nobody using the machine:

    time     uptime h    PLE0      Proc     paged pool    available
    21:03        1.93    3,931     3,414       1,100 MB     15.73 GB
    22:03        2.93   49,987    49,503       1,849 MB     13.09 GB
    23:03        3.94   86,101    85,638       2,480 MB     10.20 GB
    23:53        4.77  113,095   112,667       2,814 MB      7.74 GB
    

    About 38,500 Proc allocations and 600 MB of paged pool per hour, and
    available memory halved in under three hours. Since the update the machine has run out
    of memory more than once, each time within a day of a reboot.

    Why ordinary laptops may never show it

    At about 18 KB per process this needs a process-dense workload. On a machine
    that starts only a few hundred processes a day it adds up to a few megabytes,
    which would fit the reply above that it does not show on two ordinary
    laptops.

    Developer tooling is enough, though. The trend above came from an idle
    machine with such tooling running: a directory watcher written as a shell
    loop runs grep for every file in a folder every few seconds, and under Git
    for Windows each external command costs several processes. Shortly after the
    trend, sampling the process list for 90 seconds found 537 new processes.
    Over the same interval the Proc count rose by 758, since a poll four times a
    second misses most processes that live only tens of milliseconds.

    Reproduction

    for ($i = 0; $i -lt 200; $i++) { & cmd.exe /c exit 0 }
    

    Read the PLE0 and Proc tags with poolmon (poolmon /iPLE0, poolmon /iProc)
    before and after, compared with an idle interval of the same length.

    Not included

    No FSDIAG bundle, and no test with real-time protection disabled. Defender is
    off on this machine, so that test needs the machine off the network first.