Windows Update - Server 2016 - Problem DeepGuard and Real-Time-Scanning

Hello,

 

In the course of the Windows Server 2016 - changeover I increasingly notice that the monthly cumulative updates fail, mostly with this error code 0x800705b4. The error code seems to be a TIMEOUT from Windows Update according to internet research.
There are many workarounds on the internet for troubleshooting.

After some testing I noticed that if I switch off the real-time scanning and the DeepGuard, the update process is completed much faster and successfully.
With activated DeepGuard and real-time scanning the installation usually fails and takes 2-3 hours.
With DeepGuard deactivated and real-time scanning - installation successful and takes 30min.
I have tested this variant with servers that get their Windows updates via the internal WSUS and install the updates normally via the menu function.

However, I have also tried other methods, where I also managed a successful installation of the updates with activated F-Secure.
Starting the Windows tool "sconfig" via an administrative cmd and installing Windows updates via it also led to success, but takes over 5h.
Furthermore, I downloaded the monthly update as a .msu file directly from Microsoft and then installed it manually, which also led to success. This took more than 4 hours and is of course not very effective.

In summary, you can say that F-Secure slows down the Microsoft update process on servers enormously or the update fails because of the timeout.

F-Secure Server Security Premium 12.11 is installed on the servers.

Can you trace or reproduce the above processes and do you have a solution?

 

Thank you very much for your help.

 

Stefan Schmidt

 

«1

Answers

  • We have the same issue with Windows Server 2016 and F-Secure Server Security 12.12 build 104.

     

    With F-secure enabled, windows update install via (internal) WSUS takes up to 8 hours (including several retries, errors and reboots) .

     

    When F-secure is completely disabled via services or uninstalled, windows update is completed within 15-30minutes without errors.

  • VadVad Posts: 1,043

    Hello,

     

    Please, check that you have both public hotfixes installed on your server products:

    https://www.f-secure.com/en/web/business_global/downloads/server-security

    Installing them helps to resolve the issue. If not, please, contact support.

     

    Best regards,

    Vad

  • MartdlMartdl Posts: 12

    Which specific hotfix do i need? We have much problems regarding patching windows via WSUS (takes up to 4 hours per server).

  • VadVad Posts: 1,043

    Hello Martdl,

     

    Please, start from checking the link. This hotfixes are for our SS product, not for Windows OS. Could be deployed form Policy Manager console, or locally.

     

    Best regards,

    Vad

  • The hotfixes have nothing to do with the above mentioned problem.

  • VadVad Posts: 1,043

    Hello ITMSuhl,

     

    This hotfixes help in several cases. If you have them installed and still can reproduce the problem, please, contact support.

     

    Best regards,

    Vad

  • Hallo,

    I have already contacted support and am waiting for feedback. Unfortunately it all takes a long time.

  • I installed theese two hotfixes on a clean test server and the FSPM server.

    --

    F-Secure Server Security Standard 12.x FSGKHS Hotfix
    December 20, 2018

    F-Secure Anti-Virus 9.52 Hotfix #9 952.09

    --

    F-Secure Server Security Standard 12.12 FSMA Hotfix
    October 29, 2018

    'F-Secure FSMA 10.10 Hotfix #3 1010.03'

    --

     

    Result:

    Windows update install took about 1 hour (however the Windows update GUI displays an error after 30minutes) 

    The restart after install took about 1hr, stuck on the blue screen "Installing updates, do not reboot..."

     

    Compared to result with no F-secure

    Install 10-15minutes

    Reboot 5-10min.

  • MJ-perCompMJ-perComp Posts: 1,098

    What is the status of ORSP connectivity before you start to deploy the Update?

    Check that using ORSPDIAG. (Do a "DIR orspdiag /s" to obtain the correct path on your system).

     

  • Orspdiag from my 'clean' server:

     

     

    Spoiler

    C:\Program Files (x86)\F-Secure\ORSP Client>orspdiag.exe
    ORSP DIAGNOSTIC DUMP

    ORSP: 1.2.17.257
    FS: F-Secure Server Security 12.12 build 104 (SVE)
    OS: Win64 10.0.14393 sp 0.0
    System: 6143 MB RAM, 2 CPUs

    Statistics start: 2019-01-16T17:25:45Z
    Statistics end:   2019-01-17T12:28:37Z

    General statistics:
    Number of HTTP queries:         13
    Number of HTTP submits:         1
    Number of HTTP timeouts:        0
    Number of HTTP errors:          0

    Number of 0 queries:            1
    Number of 0 responses:          1

    Statistics for type 2:
    Number of all placed queries:   439
    Number of application timeouts: 23
    Number of queries, that
            hit cache:              389
            hit server:             50
    Number of server hits that got
            response data:          50
            empty response:         0
    Server query roundtrip times (ms):
            min:                    0
            max:                    15111
            avg:                    491
            med:                    25
            stdev:                  2377
    Oldest cache entry (seconds):   168186
    Number of revoked entries:      0

    Statistics for type 1:
    Number of all placed queries:   4
    Number of application timeouts: 0
    Number of queries, that
            hit cache:              3
            hit server:             1
    Number of server hits that got
            response data:          1
            empty response:         0
    Server query roundtrip times (ms):
            min:                    52
            max:                    52
            avg:                    52
            med:                    52
            stdev:                  0
    Oldest cache entry (seconds):   2431263
    Number of revoked entries:      0

    Number of submits of type 0:    1 (1514 bytes)

    Tx: 9598 bytes, Rx: 10381 bytes

    Histogram of server query roundtrip times (ms):
    [0: 11] [20: 24] [40: 14] [80: 0] [160: 0] [320: 0] [640: 0] [1280: 0] [2560: 0] [5120: 1] [10240: 1]

    Histogram of NRS safe:
    [missing: 86] [empty: 0] [error: 0] [-100: 0] [-99: 0] [-79: 0] [-19: 0] [80: 46] [100: 0]

    Histogram of NRS lookups:
    [3: 96] [4: 29] [5: 7]

    Histogram of NHIPS ratings from cache:
    all:           [0: 69] [150: 3]
    last 14 days:  [0: 30] [150: 3]
    last 24 hours: [0: 4]

    UUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    Server: orsp-c3-ec1.aws
    Status: 200
    Connectivity state: Ok
    CRL state: Ok
    Proxies: -


    Current proxy: -

    Cache: 99/10000 entries (NHIPS: 72, NRS: 27), 23663 bytes

    C:\Program Files (x86)\F-Secure\ORSP Client>

     

     

    Compared to ORSPdiag from a server that has not installed windows/f-secure updates

     

    Spoiler
    Spoiler

    C:\Program Files (x86)\F-Secure\ORSP Client>orspdiag.exe
    ORSP DIAGNOSTIC DUMP

    ORSP: 1.2.17.257
    FS: F-Secure Server Security 12.12 build 104 (SVE)
    OS: Win64 10.0.14393 sp 0.0
    System: 4095 MB RAM, 2 CPUs

    Statistics start: 2019-01-17T01:07:33Z
    Statistics end:   2019-01-17T12:43:26Z

    General statistics:
    Number of HTTP queries:         12
    Number of HTTP submits:         1
    Number of HTTP timeouts:        0
    Number of HTTP errors:          0

    Number of 0 queries:            1
    Number of 0 responses:          1

    Statistics for type 2:
    Number of all placed queries:   285
    Number of application timeouts: 0
    Number of queries, that
            hit cache:              273
            hit server:             12
    Number of server hits that got
            response data:          12
            empty response:         0
    Server query roundtrip times (ms):
            min:                    0
            max:                    55
            avg:                    25
            med:                    21
            stdev:                  21
    Oldest cache entry (seconds):   57802
    Number of revoked entries:      0

    Statistics for type 1:
    Number of all placed queries:   5
    Number of application timeouts: 0
    Number of queries, that
            hit cache:              0
            hit server:             5
    Number of server hits that got
            response data:          5
            empty response:         0
    Server query roundtrip times (ms):
            min:                    24
            max:                    53
            avg:                    32
            med:                    26
            stdev:                  11
    Oldest cache entry (seconds):   3338665
    Number of revoked entries:      0

    Number of submits of type 0:    1 (1391 bytes)

    Tx: 8745 bytes, Rx: 7665 bytes

    Histogram of server query roundtrip times (ms):
    [0: 6] [20: 6] [40: 5] [80: 0] [160: 0] [320: 0] [640: 0] [1280: 0] [2560: 0] [5120: 0] [10240: 0]

    Histogram of NRS safe:
    [missing: 62] [empty: 0] [error: 0] [-100: 0] [-99: 0] [-79: 0] [-19: 0] [80: 26] [100: 0]

    Histogram of NRS lookups:
    [3: 71] [4: 13] [5: 4]

    Histogram of NHIPS ratings from cache:
    all:           [0: 1189] [150: 2]
    last 14 days:  [0: 5]
    last 24 hours: [0: 5]

    UUID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
    Server: orsp-c2-ec1.aws
    Status: 200
    Connectivity state: Ok
    CRL state: Ok
    Proxies: -
    Current proxy: -

    Cache: 1214/10000 entries (NHIPS: 1191, NRS: 23), 257270 bytes

    C:\Program Files (x86)\F-Secure\ORSP Client>

     

  • Answer from F-Secure-Support from Finland

     

    I've have a direct solution of the problems, you can be can be be be prepared for subsequent solutions, in the Echtzeit-Scan folgende Pfade ausschließen
    C:\Windows\WinSxSxS\
    C:\System Volume Information\
    C:\Windows\SoftwareDistribution\

     

    In Q1/Q2 this year an ultralight version for servers will be released, which skips all possible IO's --> I don't think it's so good now either

    Excluding the above mentioned paths at least leads to a successful installation of Windows Updates at Server 2016.

     

    Greetz

     

  • MartdlMartdl Posts: 12

    Any news on this?

    There is no improvement noticable on our Server 2016 systems. With F-secure enabled the update process is taking forever, stop all f-secure services and the server updates within an hour including reboots. A Server with F-secure enabled is taking over two hours updating.

    The new Capricorn engine makes no difference in the update process.

  • VadVad Posts: 1,043

    Hello Martdl,

     

    The only solution we have currently is the set of exclusions mentioned in the comment posted before yours.

     

    Best regards,

    Vad

  • Also have this problem on Windows 10.

     

    We use CS 12.

     

    Does the same problem occur with newer versions of CS?

  • MartdlMartdl Posts: 12

    That doesn't make any difference. Fix this please, we are currently disabling our SS products on servers during WSUS rounds. 

  • VadVad Posts: 1,043

    Hello Martdl, DavidCES,

     

    We have a hotfix now, which helps to resolve the issue for one of our customers. Please, contact support.

     

    Best regards,

    Vad


  • @ITMSuhl wrote:

    Answer from F-Secure-Support from Finland

     

    I've have a direct solution of the problems, you can be can be be be prepared for subsequent solutions, in the Echtzeit-Scan folgende Pfade ausschließen
    C:\Windows\WinSxSxS\
    C:\System Volume Information\
    C:\Windows\SoftwareDistribution\

     

    In Q1/Q2 this year an ultralight version for servers will be released, which skips all possible IO's --> I don't think it's so good now either

    Excluding the above mentioned paths at least leads to a successful installation of Windows Updates at Server 2016.

     

    Greetz

     


    This solution seems to work for us aswell.

     

    This is how i configured it:

     

    Fsecure policy manager settings

     

    Policy manager version 13.12.841

  • MartdlMartdl Posts: 12

    Is the * after the directorys needed? I've used this format: https://community.f-secure.com/t5/Business/Excluding-objects-from-Real-Time/ta-p/66013
    And it doesn't make a difference.

  • @Martdl Im not sure if the * is needed, but it seems to work for us.

    Did you set 'Excluded Objects Enabled = Enabled' setting?

  • MartdlMartdl Posts: 12

    @Henrik4 Yes, we have more exclusions set. According to F-secure documentation the * shouldn't be needed. I've made the change and am testing now to see if there's a improvement.....

  • There arent any hotfixes available for Client Security. I cant even find the downloads for v 12 as I believe its end of life.
  • Please differentiate Client and Server version

    Client = V14
    Server = V12

     

    https://www.f-secure.com/en/web/business_global/downloads/server-security

     

    We didn't put the asterisk behind it and successfully distributed the January updates on Server 2016

  • MartdlMartdl Posts: 12

    @ITMSuhl The problem isn't the distribution but the updates are taking far longer than usual. Normally a server is up and running again within an hour, now there are systems that are busy for > 4 hours. especially the reboots are taking much longer.

     

    Running PMS 13.11.84108

    SS clients: 12.12 with FSAV952-09 and FSMA1010-HF03 hotfixes

    ESS clients 12.12 with FSMA1010-HF03 hotfix

  • MartdlMartdl Posts: 12

    @Martdl wrote:

    @Henrik4 Yes, we have more exclusions set. According to F-secure documentation the * shouldn't be needed. I've made the change and am testing now to see if there's a improvement.....


    Well, that didn't work.

    2 NEW Windows server 2016 systems deployed with F-secure Server Security 12.12 with hotfixes.

    Disabled all F-secure services on one system. Downloaded all updates on both systems, same set, and started the update process. The Server with F-secure disabled is ready to go and rebooted within the hour (43 minutes). The second system is still busy "Getting Windows Ready, Don't turn off your computer" after  107 minutes.

    I'm done, going home now, but this is not working....

  • Did the excluded paths also reach the agent?Unbenannt.png

  • VadVad Posts: 1,043

    Hello everybody,

     

    The new hotfix was created yesterday. Not yet available on the web site. Please, contact support, and they will provide it to you.

    The hotfix is applicable to 12.x Server Security/Email and Server Security and Client Security. The hotfix id for the reference is FSAV952-10.

    Best regards,

    Vad

  • MartdlMartdl Posts: 12

    @ITMSuhl wrote:

    Did the excluded paths also reach the agent?Unbenannt.png


    Yes, checked that before i started the updates.
    I'm letting this rest now, monday is our production patching, so i'm not going to change any more and just disable f-secure on all servers.
    After that i will open a case with F-secure.

     

    I just heard from our Desktop team that they also experience problems with Windows 10 clients.

  • I'm glad my colleague found this thread.

    This issue has been costing us many (expensive) hours and frustration during downtime.

    Please provide a proper solution with high priority (product update)!

     

    Thanks.

  • VadVad Posts: 1,043

    Hello Solipsis-VdP,

     

    The solution (hotfix) is already available. Did you try it? If not, just contact support, and we will provide it to you.

     

    Best regards,

    Vad

  • MartdlMartdl Posts: 12

    Tried the hotfix on two new deployed machines. One with hotfix, one without.

    Server with hotfix is ready installing updates after 32 minutes.

    Server without hotfix is still busy after > 99 minutes.

    So i think i can safely conclude the hotfix makes a difference. I'll be deploying it asap.

     

    One last question, are the exclusions mentioned above still necessary?

This discussion has been closed.