SENSE does not always respond to DNS requests

Ozone

It look like SENSE does not always respond to DNS requests. I believe this was what was causing issues with my Bosch dishwasher (https://community.f-secure.com/t5/F-Secure-SENSE/Bosch-dishwasher-with-Home/m-p/95771) and now seeing it with the Garming Fenix 5 watch as well.

 

I have debugged this by dumping all traffic on both sides (the LAN and the Internet) of the SENSE. My watch sends DNS qureies for time-c.nist.gov and gold.garmin.com to SENSE. SENSE sends these queries onward to my Internet router, the Internet router replies with the addreses to SENSE, but the reply is never sent back to my watch by SENSE.

 

However, the same lookups work from other devices. The root issue may be related to ARP requests sent by SENSE, as I have seen ARP requets from SENSE to the dishwasher and the watch that have gone unanswered. I have not yet investigated thoroughly,  but with a quick look at the ARP requets sent by  SENSE seem to differ somewhat from requests sent by other devices. Given time I will investigate more thoroughly.

LKo

I can confirm that this is the case. SENSE does not always forward DNS replies. Happened three times today.

Ozone

I believe the root cause is the ARP requests that SENSE sends. This seems to be a problem for embedded devices, and less of a problem for more high end equipment like PCs, tables and phones. The problem has been with a Bosch dishwasher, Garmin Fenix 5 watch, and maybe once with a Withings scale. I've seen that there are no ARP replies with the dishwasher and the watch, but not confirmed it with the scale. The problems are not persistent, although right now I'm not able to get my watch to work no matter what. I have tried rebooting both SENSE and the watch. The watch has worked through the SENSE earlier, but also has had occational connection problems.

 

Here is as example of an ARP request sent by SENSE that the Fenix 5 does not respond to:

Ethernet II, Src: F-Secure_00:5e:71 (f0:27:45:00:5e:71), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
        Address: Broadcast (ff:ff:ff:ff:ff:ff)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
    Source: F-Secure_00:5e:71 (f0:27:45:00:5e:71)
        Address: F-Secure_00:5e:71 (f0:27:45:00:5e:71)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: ARP (0x0806)
    Padding: 0000000000006cd400006cd4000000050001
Address Resolution Protocol (request)
    Hardware type: Ethernet (1)
    Protocol type: IPv4 (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: request (1)
    Sender MAC address: F-Secure_00:5e:71 (f0:27:45:00:5e:71)
    Sender IP address: 192.168.71.1
    Target MAC address: Broadcast (ff:ff:ff:ff:ff:ff)
    Target IP address: 192.168.71.110

Here's an example of an ARP request sent by a Linux box that the Fenix 5 does respond to:

Ethernet II, Src: AsustekC_78:ef:41 (2c:56:dc:78:ef:41), Dst: Broadcast (ff:ff:ff:ff:ff:ff)
    Destination: Broadcast (ff:ff:ff:ff:ff:ff)
        Address: Broadcast (ff:ff:ff:ff:ff:ff)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
    Source: AsustekC_78:ef:41 (2c:56:dc:78:ef:41)
        Address: AsustekC_78:ef:41 (2c:56:dc:78:ef:41)
        .... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: ARP (0x0806)
Address Resolution Protocol (request)
    Hardware type: Ethernet (1)
    Protocol type: IPv4 (0x0800)
    Hardware size: 6
    Protocol size: 4
    Opcode: request (1)
    Sender MAC address: AsustekC_78:ef:41 (2c:56:dc:78:ef:41)
    Sender IP address: 192.168.71.3
    Target MAC address: Broadcast (ff:ff:ff:ff:ff:ff)
    Target IP address: 192.168.71.110

The only difference I see is the padding in the ethernet frame sent by SENSE.

Ozone

Now I once saw an ARP reply from the watch, even though there was the padding in the SENSE's ARP request, so that is probably not the problem. The watch still does not get DNS replies from SENSE, though. The connection once worked this morning, but unfortunately I did not dump the traffic then.

LKo

At least for me, I don't think it is related to ARP.

It is happening with all my devices: laptops, phones, tablets etc.

 

If I change the DNS server setting from using DHCP to static, let's say Google's 8.8.8.8, everything starts to work immediately.

Ozone

For reference, here is the sequence of events, from staring to sync with the Garmin watch when it powers up the WiFi, to when it fails and then shuts down WiFi:

09:44:44.534582 14:8f:21:d8:0a:b6 (oui Unknown) > Broadcast Null Unnumbered, xid, Flags [Response], length 42: 01 00
09:44:44.600589 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from 14:8f:21:d8:0a:b6 (oui Unknown), length 261
09:44:44.616790 IP 192.168.71.1.bootps > 192.168.71.110.bootpc: BOOTP/DHCP, Reply, length 300
09:44:44.628486 ARP, Request who-has 192.168.71.1 tell 192.168.71.110, length 28
09:44:44.629365 ARP, Reply 192.168.71.1 is-at f0:27:45:00:5e:71 (oui Unknown), length 46
09:44:44.631492 ARP, Request who-has 192.168.71.1 tell 192.168.71.110, length 28
09:44:44.632042 ARP, Reply 192.168.71.1 is-at f0:27:45:00:5e:71 (oui Unknown), length 46
09:44:44.633193 IP 192.168.71.110.50731 > 192.168.71.1.domain: 16275+ A? time-c.nist.gov. (33)
09:44:44.958584 ARP, Request who-has 192.168.71.110 (Broadcast) tell 192.168.71.1, length 46
09:44:45.055437 ARP, Reply 192.168.71.110 is-at 14:8f:21:d8:0a:b6 (oui Unknown), length 28
09:44:45.060443 IP 192.168.71.110.50731 > 192.168.71.1.domain: 45441+ A? gold.garmin.com. (33)
09:44:45.560852 IP 192.168.71.110.50731 > 192.168.71.1.domain: 16275+ A? time-c.nist.gov. (33)
09:44:46.061041 IP 192.168.71.110.50731 > 192.168.71.1.domain: 45441+ A? gold.garmin.com. (33)
09:44:46.560446 IP 192.168.71.110.50731 > 192.168.71.1.domain: 45441+ A? gold.garmin.com. (33)
09:44:47.063427 IP 192.168.71.110.50731 > 192.168.71.1.domain: 16275+ A? time-c.nist.gov. (33)
09:44:48.061436 IP 192.168.71.110.50731 > 192.168.71.1.domain: 45441+ A? gold.garmin.com. (33)
09:44:49.061430 IP 192.168.71.110.50731 > 192.168.71.1.domain: 16275+ A? time-c.nist.gov. (33)
09:45:06.669908 ARP, Request who-has 192.168.71.110 (Broadcast) tell 192.168.71.1, length 46
09:45:29.163847 ARP, Request who-has 192.168.71.110 (Broadcast) tell 192.168.71.1, length 46
09:45:49.797675 ARP, Request who-has 192.168.71.110 (Broadcast) tell 192.168.71.1, length 46
09:46:12.221250 ARP, Request who-has 192.168.71.110 (Broadcast) tell 192.168.71.1, length 46
09:46:34.651646 ARP, Request who-has 192.168.71.110 (Broadcast) tell 192.168.71.1, length 46
09:46:53.551692 ARP, Request who-has 192.168.71.110 (Broadcast) tell 192.168.71.1, length 46
09:47:16.053499 ARP, Request who-has 192.168.71.110 (Broadcast) tell 192.168.71.1, length 46
09:47:20.666520 ARP, Request who-has 192.168.71.110 tell 192.168.71.1, length 46
09:47:21.666509 ARP, Request who-has 192.168.71.110 tell 192.168.71.1, length 46
09:47:22.666650 ARP, Request who-has 192.168.71.110 tell 192.168.71.1, length 46

ARPs work, but there are no DNS replies. It could still be that the ARPs, since the SENSE continues to request the MAC address of the watch. At this point the watch has however given up and shut down the WiFi.

Ozone

When doing a DNS lookup with my iPad, I get a DNS reply:

10:10:59.038488 IP 192.168.71.109.62834 > 192.168.71.1.domain: 423+ A? gold.garmin.com. (33)
10:10:59.101232 IP 192.168.71.1.domain > 192.168.71.109.62834: 423 3/9/9 CNAME gold.garmin.com.edgekey.net., CNAME e8867.b.akama                                   iedge.net., A 104.75.82.229 (441)

So it's not only a DNS problem. Got to figure out the difference between the iPad and the Garming watch. One is that the Garmin is only briefly connected to the WLAN.

 

In both cases, the iPad and the Garmin watch, they are connected via another WLAN AP that has a wired connection to the SENSE. I'm dumping data on both the wire between the WLAN AP and SENSE, and the WLAN of the WLAN AP. I.e., none of the devices (either working or not working) are using the SENSE's WLAN.

[Deleted User]

Hello Ozone and LKo,

 

I have forwarded this post to the SENSE team's notice. They are investigating more on this.

Julle42

Sounds similar to the problem I have with Sense.

 

First I thought that Sense was really dropping WAN connection (my phone kept telling me 'no Internet connectivity'), but today I figured out that it was a DNS problem.

 

In my case, DNS stops working after 10 - 20 minutes. I can ping and access Internet IP addresses.

 

DNS functionality is restored by:

 

- booting Sense

- disconnecting and reconnecting WAN cable

- changing Sense DNS address manually (from default to 8.8.8.8)

- changing Sense IP address

 

My Sense is connected to Cisco EPC3825 (bridged mode) with CAT 5 cable.

 

 

Unknown

I also sometimes experience that some services or websites don't load while all others work. Example email sync didn't work but surfing web and WhatsApp worked just fine.

I'll now test with google dns server to see if I experience that behavior again, thanks for the info.

Julle42

Unfortunately, in my case, whatever I do, DNS stops working after 10 to 20 minutes.

 

So for example changing DNS to 8.8.8.8 is a very temporary solution. It seems that any network change (disconnecting cable, changing IP etc.) restores DNS temporarily.

 

Sense _was_ working nicely for a couple of weeks. In desperation I first connected Sense to my existing Wifi network and the also connected WAN port to my cable modem. In my opinion, that could at best mess things up, but surprisingly Sense worked perfectly. Could have been some other reason for that...

 

Yesterday I changed back to 'normal' configuration.

 

Clients I have used: Samsung S8+, Huawei P8 Lite, Huawei Mediapad M3 Lite, Microsoft Surface Pro 4 (cable connection and Wifi)

Unknown

Ok, well Google dns uses dnssec so it's safer/better than my internet provider's dns since they don't have that ON at least on by default.

LKo

Any update?

 

I find myself bypassing the whole SENSE every day, as it is unusable.

Ozone

Luckily I only have the issue with the Garmin watch, and I can sync it via Bluetooth, but it is still frustrating as that is not as convient as the WLAN syncing.

 

I'm still pretty convinced that it is related to ARP, since if you look at my previous dump, the Garmin responds to ARP requests but SENSE continues to  request the MAC address:

 

09:44:44.628486 ARP, Request who-has 192.168.71.1 tell 192.168.71.110, length 28
09:44:44.629365 ARP, Reply 192.168.71.1 is-at f0:27:45:00:5e:71 (oui Unknown), length 46
09:44:44.631492 ARP, Request who-has 192.168.71.1 tell 192.168.71.110, length 28
09:44:44.632042 ARP, Reply 192.168.71.1 is-at f0:27:45:00:5e:71 (oui Unknown), length 46
09:44:44.633193 IP 192.168.71.110.50731 > 192.168.71.1.domain: 16275+ A? time-c.nist.gov. (33)
09:44:44.958584 ARP, Request who-has 192.168.71.110 (Broadcast) tell 192.168.71.1, length 46
09:44:45.055437 ARP, Reply 192.168.71.110 is-at 14:8f:21:d8:0a:b6 (oui Unknown), length 28

The SENSE wouldn't do that it if gets the ARP reply and it updates its ARP table.

 

I believe I could figure out the difference between the watch and other devices with a few hours of work, but then I would hope to get compensated if I succeed. ;-)

LKo

Sure, I get it.

 

But the thing is, as you said, this could be figured out in a matter of hours if put some effort to it.

 

But I literally have no interest nor energy to put my time to this. I'm working as a systems specialist/administrator, so I have to investigate these things at work anyway.

 

I do not want to do that when I get home, and I paid for the device and service and I expect to get support for this.

So what I expect from F-Secure is to get some support personnel to tell me which commands to run and then I will mindlessly copy-paste them and send the output to them, so they can do the same work as I do every day at the office, which I get paid for

 

If I would still have time for this, as I used to when studying, I would be still running dd-wrt. But I don't, and that's why I invested into this product. Sad to see that the support level seems to in par with other F-Secure products. Please, prove me wrong.

[Deleted User]

Hi,

 

Thanks for the extensive investigation and apologies, this has somehow fallen between the cracks. I'll raise the priority of this. I cannot promise a solution will make the next firmware release which is coming 'soon', but once we are able to reproduce the problem we can attempt to fix it for the next one.

 

Best Regards:

Simo / SENSE QA Lead

Julle42

Thanks, Simo

 

My situation is as described earlier:

 

- connect directly to WAN with cable: DNS does not work after about 10 minutes

 

- connect as a Wifi extender to existing home Wifi: DNS stops working after about 10 minutes

 

- connect first as a Wifi extender and then also connect WAN cable: DNS works as it should.

 

Cannot explain why the last setup works, but I'm done with experimenting. Sense has been online with Wifi/WAN config for about a week.

[Deleted User]

Thanks for the additional information,

 

We made an improvement / fix to the DNS discovery logic and handling of upstream DNS servers. While we have not yet reproduced the exact issue, hopefully it helps your scenario. This should still make it for the next firmware release which goes into release testing very soon.

 

We'll continue looking into the reported ARP problem.

 

Best Regards:

Simo / SENSE QA Lead

LKo

Thank you Simo for taking an interest in this.

 

Definitely looking forward to get the next update!

User2028

Could this possibly be the cause of the Arlos disconnecting all the time that F-Secure has been ignoring?

 

https://community.f-secure.com/t5/F-Secure-SENSE/Connections-form-Arlo-not/td-p/98002

Unknown

Which version of the firmware is thr newest and has the fixes?

[Deleted User]

Current production firmware is 2017-07-05_01 - p1.2.11.744, which is so far the latest one available.

 

I will announce the build version of the next release with the changes into DNS handling once it goes through the release cycle.  We are still wrapping up QA, unless we find anything critical we are aiming to release this within a week. There's a significant feature included that took time to put in place so the process for this one has been a bit slow.

 

Thanks for bearing with us.

[Deleted User]

A new update is now live, your devices should've updated themselves around ~4AM unless you've changed  the schedule.

 

The version visible in the app hardware settings should be 2017-09-13_01 - P-1.3.9.21

 

Please let us know if the fixes / changes we have made helped at all.

Ozone

Thanks for the update!

 

I've only tried twice, but both times my Garmin watch has been able to sync after the firmware upgrade. It was occationally (but very seldom) able to sync previously as well, so I can't be fully sure it is fixed, but it looks good so far.

Ozone

Just got a connection failure with the Garmin watch... :-(

[Deleted User]

Sorry to hear that, please let us know if the situation is equally bad as before or if it has improved at all.

We weren't able to exactly reproduce the specific issue, but we made some general improvements in these areas.

 

We'll need to keep working on this if our improvements had no effect.

Julle42

I am happy to report that new software version (Security software 2017-09-13_01 - p-1.3.9.21) seems to have fixed my problem.

 

I did a factory reset two days ago and connected WAN port directly my cable modem.

 

Sense has been on line for two days, without any problems, including loss of DNS. Previously DNS functionality was lost after 10 to 20 minutes.