Dante- Can Ping cards but Dante Controller isn't seeing them

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
We're building a dante network between two venues and a broadcast room. We are having trouble seeing the board furthest from the broadcast. It seems like it works intermittently. I can ping all the devices from anywhere on the network. I subscribed some channels from the far board to the broadcast and they still worked, despite dante controller not being able to see it.

The network topology is very flat. Everything is on the same subnet and there are only 3 switch hops between buildings. The run goes over fiber between the buildings. What would cause the controller to fail from discovering the devices? A lot of the time when it does list all of the devices, information about the device will be unpopulated.

My thought is leaning towards something in the network filtering certain ports. That would explain why the audio is being passed (if it gets the chance to successfully subscribe) but other info about the devices is blocked? Anyone had this problem?
 

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
Here's the other fun part. I'm running DVS on a computer in the broadcast room. That DVS computer can be seen by controller at both ends of the campus. So why would DVS work consistently but I'm losing the cards in the mixers from point to point? M32's in both venues and an X32 producer in the broadcast room.
 

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
I can also see the latency stats for the board (called "Sanctuary") since there are still connections subscribed, I just cannot consistently change them through the controller. This is weird. I should also point out that when I had the computer audio channel subscribed (when I could change it) it worked flawlessly from an audio standpoint.
 

MNicolai

Well-Known Member
Fight Leukemia
Joined
Mar 30, 2008
Location
Sarasota, FL
Usually when I've had issues with segments of Dante networks falling offline it has something to do with a network switch configuration -- usually multicast or QoS related but can also be other configuration-related issues.

Other thing to look out for in the event you have a redundant network is make sure that the primary and secondary networks aren't getting crisscrossed or bridged somewhere along the way. Only takes one device in switched/bridged mode to take down both the primary and secondary of a "redundant" network.

In this case if you're crossing between multiple manufacturers' switches it's also possible that there's an issue with that handoff. It should not be taken for granted that all switch manufacturers will treat traffic the same way and that can create issues at the transitions where each side of the network may intermittently see the other side but not be able to subscribe to reliably.
 

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
Yeah I have no idea what's between the buildings. Traceroute shows no routers, but three switch hops and latency that's well within Dante specs. That said I'm thinking there are some consumer grade switches involved here. I don't understand why it would still allow the subscribed stream through and not "see" the device enough to modify subscriptions consistently.

This is all primary network for now. Our infrastructure doesn't allow for the creation of a redundant network at the moment.
 

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
So I checked out the layout and it looks like mostly Netgear switches, and it hops from our building to adjacent building on fiber, goes through a switch and then pops to the next building on fiber, and then to the device. I pulled this slide from the Dante level 3 training (which I had time to actually complete cuz corona)

We had some IP address conflicts the day we brought the network online and our IT guy was having trouble with the DHCP. I'm wondering if he might've locked down ports or something in his process to fix that. I'm waiting for an email back from him now.
 

Attachments

MRW Lights

Well-Known Member
Joined
Jan 4, 2017
Location
NYC
Remember from the training that multicast subscriptions can cause issues in switches using snmp snooping. DHCP should be okay, but you're likely looking at an uplink trunk conflict in your IP addressing scheme between switches/buildings. Without using DDM crossing subnets is impossible. You may be using the same subnet locally, but when you leave your network your uplink may be blocking your ports from communicating across your backbone.

Your setup should work fine. We have a very similar setup, 4 buildings, several venues and master control. We keep our Dante network isolated to our own switches that don't cross with anything else. Can you move your network to your own dedicated switches?
 

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
So far as I have been told, the whole campus is on the same subnet. When I asked if we could get our gear vlan-ed together I was met with a little resistance. After going back and forth a bit he didn't really understand the type of loading we are planning on putting on the network (also running NDI for video).

I'm leaning towards something is dumping the multicast traffic since when I was able to subscribe audio it worked beautifully, but only in unicast. I can't see past the main building to the other venue's board in controller, but all the data stuff about the devices is handled through the multicast ports. The IGMP snooping might be part of that. I can ping all the Dante hosts from anywhere on the campus, but anything related to multicast is wonky. I'm about to head over again, had to put my carp hat on this morning to build a camera platform.

Felt good to smell cut 3/4" ply again...
 
Last edited:
  • Like
Reactions: macsound

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
It's working better today. I still can't see the far side of the world, but earlier I was able to see it and subscribed to a channel just playing music. That's been running consistently, but I had clock sync issues with the far one. I set it to preferred master and it seems to have settled down, but now I can't see it and I keep getting notifications that a device named "001DC11A4AD6" is elevating to grand master, which I can only assume is the lost sheep Dante M32.

I'm getting audio from it, it's responding to pings from all the 224.0 addresses (but it wasn't for a while). I feel like there is something still on the network that is messing with the multicast traffic.
 

macsound

Well-Known Member
Joined
Jun 15, 2018
Location
San Francisco, CA
Are you allowed to use wirecast on your network? Sometimes school and corporate people discourage it.
Put a laptop at the switch that relays the data between buildings, check the time on the laptop to see it's accurate, then unplug and replug the dante on the M32 while remote desktoped into your controller that's in the other building. Make note of the time as well as if you can see the device, if it disappears and reappears or anything else.
Then go back to your wirecast machine and take a look at the logs based on the time you did you plugging, see if there's any egregious data like a reset.
 

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
Yeah, I think we can try that. So a bit of an update. We are using NDI for the video feed (1 out of a media computer and 2 cameras) from the remote venue to the broadcast, but we were unable to get the Dante stable enough for broadcast by last Sunday. What we did as a workaround was feed an out to one of the cameras with balanced inputs and piggybacked the audio along the NDI to get it to work. Wasn't ideal, but we also didn't have to cancel. Meanwhile, the NDI stuff (all Magewell gear) worked flawlessly and barely taxed network traffic.

Here's what I found out, because I realized I haven't updated this thread recently. The ability of Dante controller to see and interact with devices was intermittent, but it was consistent in that we could always see the close venue (we'll call it A) from the broadcast room and the far (B) was hit or miss. But when using Dante controller in the B venue booth we could see the B dante device, but not any of the A or broadcast room gak. The absolute wildcard was my laptop running Dante Virtual Soundcard. That would show up wherever, but anytime I subscribed audio it would either give me a triangle of death or it would initiate a cycle of clock warnings and failures.

So, we can ping all devices from anywhere, but what we can't do is ping mDNS and get replies from all of the devices from everywhere. And mDNS is how Dante handles discovery. So while the unicast audio would occasionally work, the random times I was able to subscribe anything from B, the clock would be unstable.

Fast forward to my conversation with our tech today. To be fair, he inherited this little network and it hasn't been more than 20ish computers between three buildings, a server and some printers until we showed up and burned everything down with COVID-induced technology. He eventually wants to assign static IP's to all of our devices and figure out some infrastructure plotting to clean up things. But he wasn't quite sure of any policy in place that would explain the dropped mDNS requests. So I asked him if I could look at the switches configs and he was cool with it. My boss already poked around a bit and discovered ICMP snooping enabled on one of the switches, so I'm hoping this is an easy fix now of just figuring out what policy is in place that's screwing it up. It now seems like the classic IGMP dumping the multicast into the wrong ports or just blocking it from the fiber uplink or something.

Also, Wireshark is awesome. That's saved my bacon a couple of times in the past.
 

macsound

Well-Known Member
Joined
Jun 15, 2018
Location
San Francisco, CA
Not sure if there's really only the 3 switches you mention, but something super handy about Unifi gear is you can see the topology live when using the controller. Seems based on your setup, you're not too strapped for cash, take a look at the unifi demo in a chrome browser. https://demo.ui.com
It's a little tough to see because they don't have any fake devices, but essentially, you can find all the hosts and the route they're taking, visually.
 

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
Unfortunately we have the Unifi switch in the broadcast room and everything else is a mix of Netgear or similar-grade hardware. The switches in the IDF's and MDF are all managed, but yeah. I did poke around on the Unifi controller and it does show all the clients and whatnot and has a really cool interface. I will definitely recommend that when we do upgrade we add more of them.
 

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
A couple of updates-

Accessing all the switches I was able to see I found a few running IGMP Snooping, I first tried enabling all of them and no effect. Then disabling all and we were able to see the other building! However, the clock was still unstable. Watching the far building device on controller it would swing high and low and lose sync periodically, maybe every couple of minutes. But we were able to get device info and subscribe audio.

At the IDF between the far building and the MDF there are a couple of media converters that are only 100Mbit. I'm not exactly sure how they have the fiber run (because there's actually two pairs running between each building), but would that be a potential bottleneck that's messing with the clock? Going to try some more work on Sunday, but was wondering if that was a red herring or not.
 

MNicolai

Well-Known Member
Fight Leukemia
Joined
Mar 30, 2008
Location
Sarasota, FL
100Mbit media converters are likely to create an issue. Off the top of my head, I think Dante only supports 48 channels over a 100Mbit connection, and if those links serve other purposes for the campus network then that will eat into that even more.
 
  • Like
Reactions: Ben Stiegler

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
Yeah we're only dumping at most 32, but try to keep that under. No sense subscribing to a channel that's unplugged. The network is a shared one though and in addition to the Dante, it's also running IP security cameras, access points, and the NDI video feeds. If a bottleneck would affect clock that might be another issue. The stream itself when working was fine. The clock just kept dropping out.
 

MNicolai

Well-Known Member
Fight Leukemia
Joined
Mar 30, 2008
Location
Sarasota, FL
IP cameras, WiFi, and video feeds will certainly bog down that 100Mbit link pretty quickly. I wouldn't be surprised if those are causing issues. It may not be the entire issue and you may need to resolve some switch configuration settings, but I would start there because that should be pretty inexpensive to upgrade.

You might be able to log into one of the switches and see how much bandwidth is moving across the link -- some will tell you, and others won't, but if the link the is saturated then your traffic is getting crunched by the QoS.

If you want to troubleshoot around it, I would look closer at all of the switch settings along the signal chain -- verify multicast settings, IGMP snooping on, QoS enabled with DSCP priority levels assigned, IEEE energy efficiency off, etc. This Yamaha page has examples of how that would be configured for Cisco SG300 switches. Some of the terminology would be different for someone else's switches but the concepts are generally the same. Here's Biamp's version of that for Netgear. As a shared network, you want to be careful with QoS though -- don't assume your network coordinator will be happy with throwing everyone to priority level 0 like the Yamaha guide would suggest. The best case scenario is to not saturate the bandwidth of the link in which case QoS doesn't come into play.
 
  • Like
Reactions: Ben Stiegler