DANTE Network with X32, 2 M32's and a D400 Aviom Switch- Audio Popping

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
So I posted this in the Music Tribe forums as well, but I'm not sure how much hope I have in getting a solution over there. Here's the deal:

We have two venues linked to a single broadcast room via an isolated network that supports both DANTE and NDI traffic. Despite DANTE Controller not showing any clock errors, we get popping artifacts in the audio stream on the output side of the DANTE network. The diagram shows the layout of our network:

SUMC Diagram.png

Originally we started using DANTE with a single M32 networked to the MacBook Pro running DVS. That setup by itself would cause pops to be heard, but again no clock issues showed up in Dante. All that was on the network at that point was the M32 Dante card, the D400 and the MBP running DVS. I need to do some more homework/troubleshooting, but I can't rule out the Aviom D400 as a problem child.

Anytime we try to set either M32 to sync to external, we start having loud pops in the PA as the clock sync with the DL32's gets messed up. Enabling sync to external in Dante Controller sorta works for a bit, but will ultimately cause clock failure as well. THe most stable config seems to be M32's sync internally, X32 producer set to sync to external and letting Dante Controller sort out the clock master. I should also point out that we are using shielded cable from M32 to DL32. We even took the DL32 out of the link in venue #2 to see if it made a difference and it did not.

I can't seem to find anyone else with similar issues. Any google-fu with M32/X32 and popping only leads to the STP issue which is not our problem. Also, the NDI equipment was added, but the popping problems existed prior to the addition of that hardware to our network so I don't think it's an issue with NDI multicast causing problems.

And I know about the Ubiquiti switches and their questionable support of multicast traffic. The initial setup had an old 10/100 switch and had the same symptoms.
 
  • Like
Reactions: RonHebbard

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
As far as I can tell via Dante Controller yes. The only two devices that aren't are the MBP DVS, because there was a bug with the latest firmware on that a few months back (might've been patched by now) and I guess the firmware on the DL32's themselves have been patched recently, though not directly related to the Dante network.
 

FMEng

Well-Known Member
Fight Leukemia
Joined
Mar 31, 2008
Location
Tacoma, WA
Keep in mind that I haven't done any Dante with my X32s in a few years, but I will be venturing into it again in the next few weeks. (I will be putting NDI and Dante together with Tricaster. Wish me luck.)

The latest Dante firmware is 4.08. Check it with Dante Firmware Update Manager. It would be a good idea to do the update, with the file directly from Behringer/Midas, to be sure. Use the latest version of Controller, too. The consoles must all be set to sync from X-Dante, and then one of them has to be chosen as sync master in Controller. I would set the consoles as preferred masters, and turn off others. Never make any sync changes with any speakers running.

I've heard about enough issues with Macs and Dante that I would suspect an issue there. Eliminate the MBP from the system and see if things improve, especially if it isn't running the current version of DVS. I know nothing about Macs, but in the PC world, I would experiment with not using any other sound devices while running DVS, making DVS the default sound device, and even going so far as disabling internal sound hardware. Conflicts there could ripple across the network, and multiple sound devices do conflict.
 
Last edited:

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
The mac in or out unfortunately doesn't make much different in the audio out to the X32. The other problem we have is that setting sync to the card on each M32 will result in loud pops through the PA as the DL32's start losing sync. I think that's our problem honestly, but I cannot find any documentation about it.
 
  • Like
Reactions: RonHebbard

MNicolai

Well-Known Member
Premium Member
Fight Leukemia
Joined
Mar 30, 2008
Location
Sarasota, FL
Originally we started using DANTE with a single M32 networked to the MacBook Pro running DVS.
I would start troubleshooting here. Break the network down into smaller pieces and work through one problem at a time and take notes as you connect up other parts of the system where the problems start.

If I recall correctly, the network switches you're using are not Layer 2 managed switches. Before you dig too far, I would swap out for Cisco SG350's. On a Layer 2 unmanaged switch, multicast becomes broadcast and QoS can fail to work (which is important for clocking, also if you have other forms of streaming media like the NDI's on the network).

Checking Dante firmware as described above is a good idea, though I doubt your issue is with DVS. Most of the Macbook issues I've heard about relate to the recent OS migration to exclusively 64-bit and it's been bumpy for Audinate and others to work out the kinks in their drivers.

Doesn't sound like a lot of equipment all things considered. You could also move everything into one room, eliminate the Ubiquiti and the fiber/media converter bits. Then get the network stable without the interconnections where you can troubleshoot everything at once in the same room before you start diagnosing issues with the interconnections between the systems.
 

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
Yeah, the first switch we had was an old LinkSys 10/100 "dumb" switch. These new ones are Netgears that are at least somewhat managed Layer 2 switches and I've pulled their model numbers and google fu says they should do alright with Dante. They can handle VLAN's but we haven't messed with that yet because I'm not sure it'd play nice between the two switch manufacturers. I haven't done a deep dive on setting it up through the Unifi.

We do have the option to add another fiber link to the mix between broadcast and venue #2, an option we were thinking of exploring if we added more cameras from that side. However with just 3 NDI sources on that pipe, it's not really filling it up.

I agree though, back to basics with maybe just an M32 and the X32 together and see if I can isolate it. I really feel it's related to the clock sync on the boards and how it handles clocking on that AES50 line. That's the only thing that's the same between our first foray into this world and where we are now. One thing I have not tried is removing the DL32 and setting clock to external on the M32. Like I said, it can't handle clocking on the dante card with a DL32 for some reason.

Edit- Oh! And that Aviom switch. I don't trust that thing. It sometimes has little panic moments when we bring another Dante device online.
 
Last edited:
  • Like
Reactions: RonHebbard

MNicolai

Well-Known Member
Premium Member
Fight Leukemia
Joined
Mar 30, 2008
Location
Sarasota, FL
Have you logged into the Netgears and configured the QoS priorities, disable IEEE, usual Dante preferences, etc?
 
  • Like
Reactions: RonHebbard

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
Yeah, I don't think these ones even support IEEE. The QoS is pretty basic unfortunately and I don't think the Unifi even supports QoS, which is odd. That whole switch is odd. Turning on IGMP snooping on that switch disables 100% of the Dante multicast traffic and kills it dead. It's known issue for them, at least on their forums.
 
  • Like
Reactions: RonHebbard

FMEng

Well-Known Member
Fight Leukemia
Joined
Mar 31, 2008
Location
Tacoma, WA
The whole system has to be clocked together, otherwise there will always be issues. The DL32 has to get its clock from the console. The console has to get its clock from Dante. Everything seems to point to a network issue causing the consoles not to get reliable sync.

Awhile back, I used a media converter, and found it to be a bottleneck, so I'm suspicious of that. I'm also leery of running ethernet between buildings (assuming Ubiquiti and Venue 1 are not co-located). Buildings are at different ground potentials, and the ethernet transformers may not provide enough isolation and noise immunity under those conditions. Non-audio data is more robust because packet re-sends cover up a multitude of sins. AoIP doesn't tolerate missing or corrupted packets at all.

The bottom line is you need serious switches with SFP modules for fiber, and runs between buildings to be fiber. I like Netgear switches for more typical uses, but for AoIP, I only use proven models, such as SG-350 series. The cost isn't that bad. In fact, I have two sitting next to some NDI cameras, on my dining table. When I started my project, I told myself I wouldn't do it without top quality network switches. I'm volunteering to do the installation, so the least the church can do is pay for components that won't waste large amounts of my time and make my hair turn gray.

To prove the point, buy one SG-350. Configure it properly. Put it and all of the audio devices together in the same room. Fire up the tone generator in one of the consoles and ship the audio around via Dante. I will bet it all works perfectly.
 
Last edited:

CrazyTechie

Well-Known Member
Joined
Jun 14, 2009
Location
Salt Lake City, UT
They can handle VLAN's but we haven't messed with that yet because I'm not sure it'd play nice between the two switch manufacturers.
This should be fine as long as both switches support VLANs, which it sounds like they do, and you use the same VLAN numbers on all the switches. VLAN tagging is built into the ethernet frame structure so it's universal. You'll just have to get the hang of tagged vs untagged ports.

So, if you setup VLAN 10 on both your Ubiquiti and Netgear switches, traffic will flow between them along your trunk ports.

The Cisco SG switches should serve you pretty well. The web interface is pretty easy to work in (waaaay better than Ubiquiti) especially if CLI isn't your preferred switch management method.
 

tmcgow1

Member
Joined
Aug 18, 2020
Location
Las Vegas
So I posted this in the Music Tribe forums as well, but I'm not sure how much hope I have in getting a solution over there. Here's the deal:

We have two venues linked to a single broadcast room via an isolated network that supports both DANTE and NDI traffic. Despite DANTE Controller not showing any clock errors, we get popping artifacts in the audio stream on the output side of the DANTE network. The diagram shows the layout of our network:

View attachment 21303
Originally we started using DANTE with a single M32 networked to the MacBook Pro running DVS. That setup by itself would cause pops to be heard, but again no clock issues showed up in Dante. All that was on the network at that point was the M32 Dante card, the D400 and the MBP running DVS. I need to do some more homework/troubleshooting, but I can't rule out the Aviom D400 as a problem child.

Anytime we try to set either M32 to sync to external, we start having loud pops in the PA as the clock sync with the DL32's gets messed up. Enabling sync to external in Dante Controller sorta works for a bit, but will ultimately cause clock failure as well. THe most stable config seems to be M32's sync internally, X32 producer set to sync to external and letting Dante Controller sort out the clock master. I should also point out that we are using shielded cable from M32 to DL32. We even took the DL32 out of the link in venue #2 to see if it made a difference and it did not.

I can't seem to find anyone else with similar issues. Any google-fu with M32/X32 and popping only leads to the STP issue which is not our problem. Also, the NDI equipment was added, but the popping problems existed prior to the addition of that hardware to our network so I don't think it's an issue with NDI multicast causing problems.

And I know about the Ubiquiti switches and their questionable support of multicast traffic. The initial setup had an old 10/100 switch and had the same symptoms.
This is not a simple problem to deal with. I would suggest looking at your source power as you stated the setup would pop by itself. Nothing in the data diagram is irregular or incompatible.
You are describing a classic power differential pop that is from items being out of phase and potential differential voltage between neutral and ground. Is everything under common switchgear? Same distro panel? Same phase? - this can be the first problem source of differential voltage on neutral legs. Common earth ground? - all earth ground potentials are not the same when dealing with clocks. ( that's another story) You should verify or self-install 8' copper grounding round as close to main distro as possible. Inspect it yourself and make sure there is not a 10-12 awg copper where there should be 4-8awg for sufficient ground. You can do the math for resistance and distance. In average soil there is 25 Ohms resistance already built into calculations. Plug strips and devices must all have earth grounding connectors. Racks should be connected to the common ground. Any lifted grounds should be primary suspects for induction to the common copper connection either audio and data resulting in a micro-ampere discharge when peak potential is achieved resulting in a POP. Ask why any ground is lifted? Toss any cheap plug strips and use IEEE certified power distribution hardware. Every discharge or pop upsets and/or restarts clocks, drains diodes, flashes memory, and causes havoc on a micro scale. Think of it as a mini bolt of lighting inside your power distro.
Then there is also rogue sources such as bell/alarm circuits, ethernet, telco/pots, cable tv, cable modem, surveillance that can carry a mish-mosh of voltage, grounding or lack-of that all puts demand on the grounding systems and draining of leaky voltage. External sources commonly cause noise either hums, pops, echo and timing errors between clocks. Have a common demarc ground for all external powered wiring such as a single lash up board. Our theater uses a 3P 200A voltage regulator before my fixed house distro with copper bonding to earth. All distro panels for stage, audio, lighting, data, all have common earth ground to eliminate any potential between devices. Any clock issues, pops, or hums can be tracked to interconnect cables or devices by process of elimination by type of system, lighting, data, video, audio. If you are subject to flooding or very high humidity, create a decision tree to eliminate moisture penetration inside devices or connect points.
If you do use lifted grounds or non-shielded anything, then consider voltage regulators and./or hum-buckers which should drain off the offset power charge. Once power is eliminated as a potential source for pops, look at your oldest gear that may have a leaky diode or faulty push-button that is on it's way out.

Clocks can be finicky on their own. The clock itself should not cause a pop sound. For Dante there is internal IEEE 1588 PTP or AES3 or word clock in a console. Only 1 clock can be selected as master clock, all else are slave clocks. Clock Synchronization (audinate.com)
 

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
That's a potential with the the building we are dealing with. Venue #1 and Broadcast both exist in the older side of the campus where Venue #2 was built about 15 years after. The only thing is we had this problem when we were just running a closed network with one M32-DL32, one laptop, and the Aviom box. All of which were in the same room, fed from the same panel. My troubleshooting gut is pointing towards something in that group causing the faults, but it is never a bad time to check out equipment grounding and make sure it's correct!
 
  • Love
Reactions: RonHebbard

MNicolai

Well-Known Member
Premium Member
Fight Leukemia
Joined
Mar 30, 2008
Location
Sarasota, FL
That's a potential with the the building we are dealing with. Venue #1 and Broadcast both exist in the older side of the campus where Venue #2 was built about 15 years after. The only thing is we had this problem when we were just running a closed network with one M32-DL32, one laptop, and the Aviom box. All of which were in the same room, fed from the same panel. My troubleshooting gut is pointing towards something in that group causing the faults, but it is never a bad time to check out equipment grounding and make sure it's correct!
It's not possible that it's a grounding/bonding issue between the buildings. Your fiber link between the buildings doesn't care about earth potential. For that matter, neither does unshielded CATx cable. The only links you could have an issue are on AES50 runs with shielded CATx cable, and the shield is required to be terminated at both ends by Behringer/Midas for the specific reason of solving their AES50 popping manufacturing defect.

Honestly, I would break it all down into just the one system and work out the issues on an island before even thinking about the interconnections to other rooms. You are seeing a Dante issue but are you absolutely sure it isn't an AES50 problem that's rippling through the entire system? Have you double checked that your STP CATx cables are terminated at both ends and verified with a continuity tester?
 

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
I made the Cat5e cables myself and they are grounded at both ends. The old wisdom of pulling the ground on one end is something I've always found slightly questionable. I'm hoping to get a chance tomorrow afternoon to do some isolation troubleshooting. We can simplify it to one room, I'm just wondering why I can't run the clock off the card without causing the DL32 to freak out.
 

FMEng

Well-Known Member
Fight Leukemia
Joined
Mar 31, 2008
Location
Tacoma, WA
I like Mike's idea of breaking the system down to it's parts, and making sure the basics are working before looking further.

Be sure to use Neutrik Ethercon shells on both ends of the AES50 cables. The shell is where the critical connection is made. Just using the ground connection on the RJ-45 causes trouble.

Finding out why the system won't run from the X-Dante card clock is key, because it absolutely must. It would be very useful to run just the console and the DL32, with the X-Dante card as the clock reference, but without any Dante network connection. That would tell us a lot. If that fails, then try the console without the DL32, clocked to the X-Dante card.

You live in humidity and salt central, and the accessory card slot connection does not inspire confidence. Maybe it's time for a spritz of DeoxIT D5 and and Gold G5.

In the analog audio world, dropping the shield at one end is what makes large systems work that would otherwise be a giant mess of hums and buzzes. I've built several radio facilities that way. Behringer's implementation of AES50 is a special case because they goofed on the circuit board design and caused what is called "the pin 1 problem." Basically, they made a high impedance ground that induces noise into adjacent circuitry. The Ethercon shell shunts enough noise to the chassis, preventing data corruption and sync losses.
 
Last edited:

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
The shell is where the critical connection is made. Just using the ground connection on the RJ-45 causes trouble.
Hmmm.

So that was a light bulb. We did use the shells, but the weird thing was when the cable strain was screwed all the way down it would not work correctly. The DL32 would show up, but sync was red. Here's my solution-

1609681978798.png


It seems pretty happy with this. So the XLR shield is not bonded to the RJ45 shield? Not sure why it doesn't like the back end screwed down. These are the "pass-through" RJ-45's, so maybe something is making contact when that is jammed in there?

I was able to switch the clock source over to the card on both venues (Venue 2 actually already has the DL32 out of the loop but still had the clock internal).

So far so good?
 

TimMc

Well-Known Member
Premium Member
Joined
Feb 15, 2017
Hmmm.

So that was a light bulb. We did use the shells, but the weird thing was when the cable strain was screwed all the way down it would not work correctly. The DL32 would show up, but sync was red. Here's my solution-

View attachment 21317

It seems pretty happy with this. So the XLR shield is not bonded to the RJ45 shield? Not sure why it doesn't like the back end screwed down. These are the "pass-through" RJ-45's, so maybe something is making contact when that is jammed in there?

I was able to switch the clock source over to the card on both venues (Venue 2 actually already has the DL32 out of the loop but still had the clock internal).

So far so good?
I would reterminate the RJ45 and cut back about 4" on the cable...

Here's my guess: the compression on the cable by the strain relief chuck is likely creating an impedence bump. You may need to modify the chuck (or find the larger, white plastic one) because STP has a greater diameter than UTP.

The other possibility is that when you screw down the tail piece that some mis-alignment of the RJ45 is occurring, whether that's in the contact mating or the shield ground/connector shell interface, I can't say.
 

StradivariusBone

Custom Title
Fight Leukemia
Joined
Aug 23, 2013
Location
Space Coast, FL
The RJ45 is very misaligned in the ethercon. That partially led to us deciding to go without them for the time being since they wouldn't really seat properly into the socket with the strain-relief screwed down. I've heard of these pass-through RJ-45's having issues like this, so I terminated them with a bit of the copper inside (not sticking out the other end), but I'm wondering if the angle of the RJ45 is causing some pins to not seat completely? That might explain the console recognizing the DL is connected, but not able to fully sync. That was the other part of abandoning the XLR barrel, it was happier without them.

In any event, it seems like this is solved. I was able to pull clock from the Dante cards and we had no audio distortions (outside of a funky RF mic issue- unrelated and mildly expected). Thanks to all!
 

Users who are viewing this thread