Dante- Can Ping cards but Dante Controller isn't seeing them

That Netgear guide is great. Everything past the Unifi switch in the broadcast room is Netgear, a mix of 24 port and 8/10 port switches. All of them have IGMP support and QoS, but not all of them have very many settings that go that deep. I think only the 24 port guys let you even modify the queues for the DSCP tags, and even then I didn't see a way to add tags for it to recognize.

There's 6 strand fiber between the buildings and 2 pair are being used. The one fiber link is on a netgear switch with fiber connectors and it is a gigabit connection. The other one isn't (media converters, etc), but I'm not sure how those interact, it looks like there might be two 24 port switches in the MDF and the two fiber runs are both being used by each? I'm guessing both are being used to make a bigger pipe overall between the buildings, but again it's all on the same subnet.

It's been about 20 years since I've been in a networking infrastructure class, so I'm a bit rusty on what flies these days. Since there's an open pair of fiber on these runs I'm wondering if we couldn't just isolate all of our AV traffic to that pair. Can you patch fiber together at the rack? (I vaguely recall learning about how to splice it and taking a class that explained how bad it is to get fiber optic into your bloodstream, again like 20 years ago.) Or would you need some kinda SFP switch to connect those? The runs are not terribly long. This is our backup plan assuming configuration fails.
 
Yes, fiber strands can be patched together. It's just a matter of matching connectors and strand types. There is single mode and multimode, and you cannot mix the two. Single mode usually has yellow patch cords and multi is orange. Often, the patch panel has one kind of connector, and the switch has another, but patch cords come in varieties.
 
Pretty sure it's all multimode. Had some luck today, but determined that running IGMP Snooping on the switch in the broadcast room will in fact block all multicast data from the uplink port. I had wireshark going watching the PTP packets going and the second it was enabled everything stopped. It was hit or miss on the Netgear switches, which are a hodge-podge of make and model, but none of them have that many settings to modify. In fact, it turns out that Ubiquiti switch can't even do QoS and the VLAN capability is pretty limited as well.

Was able to see everything and make subscription changes with devices, but still experiencing PTP clock dropout about every 2-5 minutes from the long run with only a couple channels subscribed. I think it's odd that IGMP shut down the traffic, but I don't know how it actually works on that Ubiquiti switch.

The plan right now is to take over that last unused pair of fiber, patch together in the middle, stick media converters on either side and a hardware router in the MDF to connect, but segregate it from the rest of the network. I can only assume that the clock drop out is from some other conflicting traffic on the network, but I can't seem to figure out what it is.
 
See this article on IGMP. In short, you want it enabled otherwise your multicast traffic gets sent to many other segments of your network. The change you're seeing where it "blocks" the traffic probably isn't the case. What's happening is that the endpoint devices aren't making a successful subscription or the multicast pool isn't being advertised across the network for those devices to see there is something to subscribe to. Disabling IGMP to get traffic flowing is really just a band aid around a larger multicast issue and without it enabled, your Dante traffic will bog down other areas of the network because it opens the rest of the network up to those packets whether the other devices on the network are asking for them or not.


Moving to an isolated AV network is probably the way to go. Underlying problem could be a switch config issue but if the network is all different types of switches then the switch configs may not behave as intended even if they are set correctly.
 
I watched mDNS and PTP traffic packets completely stop interacting with the switch with it enabled. I thought it maybe was a discovery period where it had to setup the internal tables of which ports had hosts that would reply to multicast, but it was a hard switch. Dante controller immediately lost sight of all devices, including the board sitting next to the switch. I've read that Shure article, but from what I've seen with Audinate's guides IGMP only really becomes a dealbreaker when you're dealing with a much bigger network and crossing subnets. Now, I have no clue what the traffic from the security cameras is. I assume unicast RTSP, but I don't know. If they are like the ones I run at my house, they stream constantly and use the server to "trigger" for recording and consume more bandwidth on the network, but I think it's a proprietary setup so I'm thinking the cameras only initiate the stream when they detect? No clue though. IGMP is definitely ideal for the overall health of the network though.

Apparently there are some issues with how Unifi switches handle this and other multicast traffic- https://community.ui.com/questions/...n/f297dd0e-c424-4867-9f40-f4234a79e39c?page=1

There was an interesting comment in that thread where the Unifi has trouble going from 100 to 1G, our one venue has a 10/100 switch that we've been looking at switching out to a gigabit. There's just so many variables here, but how that Unifi is handling traffic through it's uplink to the rest of the network seems to be part of it.
 
I don't know how much further I would troubleshoot. It sounds like you need to upgrade the network to all Gigabit either isolated from the campus or get the campus to upgrade their switches. Should keep everything in the same model lineup, make sure all the switches are on the same firmware rev, and all switches running the same config file, etc. Everything should be Layer 2 Managed at a minimum.

If you are going to continue troubleshooting, I would try breaking the network into smaller pieces, stabilize one end of the network disconnected from the larger network and keep moving across the network to see at what point the Dante traffic breaks (may be hard to do since this probably requires messing with the campus LAN). Or, swap out the Ubiquiti with a Netgear and see if it starts to play any nicer. Gigabit switches aren't that expensive and your IT dept probably has a spare switch sitting on a shelf or in a rack that you can at least test with.

The perfect scenario is having AV on an isolated network. The next best scenario is having AV on its own infrastructure that is tied into the campus network -- but the campus network isn't inserted in the critical signal path for streaming media traffic. The next best scenario is piggybacking off a campus network that's on a uniform set of switches and configs. I generally would not recommend being on the network you've described where it's a hodgepodge of different makes/models/configs/bandwidths.

If you are purchasing new switches, the Cisco SG350 series is the industry standard for Dante. Certainly many other makes/models of switches will work, but the SG350 series poses the least amount of risk and with the SFP slots you can directly tie into that fiber run.
 
Have you considered that the switch doing your IGMP queries may not support the number of devices snooping? My colleague Wayne Howell (Artistic License) wrote three good articles in Lighting & Sound International magazine on this subject and some test tools available for download here.

For those watching this thread but need a bit more info on IGMP, both sACN or Dante make good use of it. I did two videos explaining the basics. The first is a primer where the second is a real world example.

If you're interested in why an Entertainment Class Switch is sometimes a good choice when building networks, vs choosing Enterprise Class switches, watch my video called Why VIA?. I discuss, among other things, why and what Wireshark does and how it helps to debug some of these issues.
 
That's a possibility. All I know is that with IGMP on with the Unifi switch, zero multicast traffic was allowed through the uplink port. I could not find a way to mitigate that through settings. I believe the rest of the switches on the network still have it enabled at this point. Only two of them have quieriers.

The plan now I think is to put media converters on either end of the fiber run. Go from broadcast room switch--->media converter--->fiber--->patch at the midpoint IDF--->fiber--->Media Converter--->booth switch. And then get an edge router in the broadcast room to connect to the main network. That way we cut out some switch hops and distance ourselves from the other traffic on the network.
 
That's a possibility. All I know is that with IGMP on with the Unifi switch, zero multicast traffic was allowed through the uplink port. I could not find a way to mitigate that through settings. I believe the rest of the switches on the network still have it enabled at this point. Only two of them have quieriers.

The plan now I think is to put media converters on either end of the fiber run. Go from broadcast room switch--->media converter--->fiber--->patch at the midpoint IDF--->fiber--->Media Converter--->booth switch. And then get an edge router in the broadcast room to connect to the main network. That way we cut out some switch hops and distance ourselves from the other traffic on the network.
Double check but I think you only want one IGMP quierier on.

Your plan of getting semi-isolated make a lot of sense and will probably clear up your problems and is a better solution.

Philip
 
The quierier apparently uses an election process not that dissimilar to the dante clock elections. I set one as the primary and the other as a secondary, but ideally the switches will decide which one is the better. Again, these are regular Netgear switches without many bells and whistles so it's anyones guess how well they handle it.
 
Try turning off all the non Dante equipment as an experiment. Kill the cameras, etc. if the Dante clock is happier, you have your answer re 100Mb bottleneck. QoS adjustments might also save you ... losing a few frames from a parking lot cam might be acceptable til you can get faster media converters. Also, let's not assume that the fiber was terminated cleanly. Many moons ago I did a training with Sprint North Supply (anyone remember them? Early distro giant)band it was not easy to do it right. Now imagine some electrician who watched a video the night before doing his first fiber terms on your campus...

you might also try swapping fiber pairs if there are spares.

keep us posted!
 
The quierier apparently uses an election process not that dissimilar to the dante clock elections. I set one as the primary and the other as a secondary, but ideally the switches will decide which one is the better. Again, these are regular Netgear switches without many bells and whistles so it's anyones guess how well they handle it.
Have you checked that the switch firmware is up to date and consistent across all the Netgears?
 
It doesn't seem to me that bad fiber termination would reduce the bandwidth. It's either above the minimum light level and working, or it's not working at all.
 
The firmware is a great point. I am almost certain they are not all up to date. And I'm afraid to attempt shutting down the network to do isolation troubleshooting. There's not much in the way of documentation on the infrastructure. I'm also afraid to update the switch firmware for similar reasons lol.

I'm hoping we can just use the two unused strands on the trunk lines and isolate it. We're going to try that this weekend. If not? Then we might have to go that route
 
A couple updates. We replaced one fast ethernet switch with a gigabit switch. This is the one that networked an Aviom/Dante box with an M32 in the venue we were able to link up to. It vastly improved life, the universe and everything between the broadcast room and the near venue. It also seemed to help out on some level at keeping the clock happier between the far building and broadcast, but we are still experiencing the occasional clock dropouts.

Fiber links between buildings is all OM1, and we got some media converters to try and utilize the unused pair, but with no success. I don't have any test equipment so the best I could do was polish the ends and try to get a link with the media converters on either segment, neither of which was successful.

It's a 6 strand fiber run, 4 of 6 are used. One link is gigabit, but the other is 10/100. After seeing the improvements from swapping the local switch with a gigabit I'm willing to bet that the traffic might be getting dumped over the slower link, but it's hard to tell with how the switches are configured on both ends.

Looking into a couple of options today. Trying to see if the distances between IDF's is under 300' and if we could just get away with running copper. If not, it's either upgrading the 10/100 media converters to something better, running new fiber, or crying in the corner. We're also going to attempt to repair that open pair on the existing fiber.

Thanks for all the help so far, gang. I just wanted to update to hopefully lead to some closure on this project.
 
Did you make the required crossover for the media converters? Tx to Rx. I've seen fiber installers do some funny things when they terminate cables.
 
Yeah, I tried both ways just to be sure. Here's the fun part. So MDF to IDF 1 has two fiber links (four strands out of six total being used). The one link is 10/100, the other was going out an SFP on a gigabit switch. In IDF that first link terminated into another 10/100 transceiver and then into a switch. The second link went into another switch that looks optimized for the access points we use. There are additional links between IDF 1 to IDF 2 (which is where the venue we're trying to hit is). The first link was 10/100 and terminated in IDF 2 into an ancient 10/100 switch with only one active host connected to it. Turns out that it was the AC controller, so we'll leave that be. The second fiber link between IDF 1 and 2 was a gigabit connection and terminated into the switch we were using to connect to the venue Dante card. So theoretically, we had a good fast link between IDF 1 and 2, but on IDF 1 to the MDF we were only going over 10/100.

I took our gigabit media converters and replaced the 10/100 ones on that link b/w MDF and IDF 1 so now the pipe should be gigabit all the way through. We were able to get link and talk on the devices happily for about 10 minutes and then as we turned on devices we started to have clock issues again. I took a step back and went conduit hunting, eventually we want to isolate the whole network which looks like it will mean new fiber. I came back to the broadcast room and found that the clock sync was dropping periodically.

I went back to the switch configs and noticed that we still had all of them running without IGMP since previously it seemed more stable. I went ahead and turned that all back on (except for the Unifi switch in the broadcast room- there's some issue with how IGMP snooping works on that hardware, it kills all multicast traffic). And so far....so good.

I'm hoping that we were still flooding the switches with clock sync packets and maybe that was causing it to lose clock? Now that we're gigabit all the way down hopefully that helps out.

We are still seeming to have trouble with an Aviom D400 with the dante card. Turning that thing on seemed to aggravate the clock situation, but we'll get to that in a bit. Right now I'm just letting it sync between the venue at IDF 2 and our broadcast room and it's pretty happy, at least it has been since I started typing. Just checked, it's still happy. I feel like a Bob Ross network analyst. Happy little link lights.

Fingers crossed.
 
Well the clock has been stable for about 30 minutes but I can no longer see the Dante card out past IDF 2. Still getting audio though.
 
Well! Have I got some news! On a separate note we managed to get some PTZ cameras in after them being backordered for months. I was helping my boss get those up and running when it just struck me once again that it was ridiculous that we could stream 3 HD video feeds down this pipe with minimal latency, send PTZ control info back up the pipe with minimal latency, throw VOIP Mumble chat everywhere on this network...

...and this dadgum Dante was still just not cooperating.

So I bit the bullet and went into the switch firmware. Initially I was hesitant to do this in a live production network (we share this with the normal staff of the church) but I started updating them one at a time, going from the furthest to the closest. I got as far as the IDF and all of a sudden everything started clicking. It turns out that one of the switches at the end of a fiber run had a bug. Here's the snippet from the release notes-

"When the device receives hundreds of Pulse Audio multicasts (with destination IP of 224.0.0.56) per second, GUI access and Remote Diagnostic are blocked."

Now I can't find anything that says that Dante specifically uses that multicast port, however Dante does dump A LOT of multicast traffic and this issue explains the issues we would experience where it would initially discover the board, eventually lose the device info and then ultimately lose all ability to subscribe audio, in spite of still happily transmitting the audio streams already subscribed in between fits of clock failure. I'm guessing whenever we turned the board on, it would begin to overwhelm the bugged switch with multicast packets until it finally shut down and blocked all the 224.0.0.0 addresses coming and going, but it didn't affect any of the unicast or broadcast messages, so all the rest of the normal traffic just kept on trucking along.

I need a beer.
 

Users who are viewing this thread

Back