Complex MADI failover

ShowNet

Member
Joined
Dec 5, 2017
Location
New York
I have a fairly unique situation that we first tried to solve with network-based audio (Dante, namely), that hasn't worked and I was pointed in the direction of MADI, which I'm not particularly familiar with (I'm not an audio person by trade).

The TLDR is that I need redundant digital audio multicores that can intentionally be broken and restored within a matter of seconds, while audio continues to play seamlessly.

I have a large set piece that has 100 channels of audio - 50 to it, 50 from it. This piece moves around the space in ways that requires the audio to be switched from one multicore to another multicore, live. This is currently done by connecting the (very large, analog) B multi before disconnecting the A multi. This results in an audible pop (sometimes very loudly). And of course the cables are large.

The goal is to move to a digital audio system with redundant connections. So connection point A will have cables A1 and A2 (primary and backup). Connection point B will have cables B1 and B2. The sequence would go:
- Disconnect A1 (seamless failover to A2)
- Connect B1
- Wait for B1 digital handshake to complete, once audio is going through B1...
- Disconnect A2 (seamless failover to B1)
- Connect B2
- Move scenery

This whole sequence has to happen in a matter of seconds, under 10 seconds, ideally as quickly as the cables can be physically swapped, and, I repeat, audio is playing the entire time.

I wasn't there for the Dante testing, but as I understand it the failure point was usually on the wait time for the B1 connection to be established, and/or some variation of Dante basically choking if too much changed too quickly.

I was pointed in the direction of MADI, which I have no real-world experience with. Everything I've seen so far suggests it may well work for this scenario. From what I can tell, it will seamlessly failover from A1 to A2. The one thing I'm not sure of is that connection establishment time when we connect cable B1.

Any thoughts on if this would work?
 

RonHebbard

Well-Known Member
Premium Member
Joined
Jun 12, 2004
Location
Waterdown, ON, CA
I have a fairly unique situation that we first tried to solve with network-based audio (Dante, namely), that hasn't worked and I was pointed in the direction of MADI, which I'm not particularly familiar with (I'm not an audio person by trade).

The TLDR is that I need redundant digital audio multicores that can intentionally be broken and restored within a matter of seconds, while audio continues to play seamlessly.

I have a large set piece that has 100 channels of audio - 50 to it, 50 from it. This piece moves around the space in ways that requires the audio to be switched from one multicore to another multicore, live. This is currently done by connecting the (very large, analog) B multi before disconnecting the A multi. This results in an audible pop (sometimes very loudly). And of course the cables are large.

The goal is to move to a digital audio system with redundant connections. So connection point A will have cables A1 and A2 (primary and backup). Connection point B will have cables B1 and B2. The sequence would go:
- Disconnect A1 (seamless failover to A2)
- Connect B1
- Wait for B1 digital handshake to complete, once audio is going through B1...
- Disconnect A2 (seamless failover to B1)
- Connect B2
- Move scenery

This whole sequence has to happen in a matter of seconds, under 10 seconds, ideally as quickly as the cables can be physically swapped, and, I repeat, audio is playing the entire time.

I wasn't there for the Dante testing, but as I understand it the failure point was usually on the wait time for the B1 connection to be established, and/or some variation of Dante basically choking if too much changed too quickly.

I was pointed in the direction of MADI, which I have no real-world experience with. Everything I've seen so far suggests it may well work for this scenario. From what I can tell, it will seamlessly failover from A1 to A2. The one thing I'm not sure of is that connection establishment time when we connect cable B1.

Any thoughts on if this would work?
Calling @MNicolai and @TimMc and @TimmyP1955
Toodleoo!
Ron Hebbard
 
  • Like
Reactions: ShowNet

MNicolai

Well-Known Member
Fight Leukemia
Joined
Mar 30, 2008
Location
Sarasota, FL
I'm not surprised it wasn't flawless with Dante. Dante is intended to fail primary over to secondary. It gets a little wiggity whack when you try and fail primary to secondary back to primary right away. It's like if you have two treadmills side by side and try jumping between them. The first jump is easy but if the second one starts too soon you can may not have your balance well enough to bounce back gracefully. There's two things happening -- the stage boxes have to reestablish their clock sync with the console and the IP address and network paths have to renegotiate. There are things that you can do to make Dante negotiate faster but it's the kind of thing you need to test and putz with to work out the kinks.

I've heard some people claim they've seen faster negotiation with static IP's for all their endpoints instead of auto-negotiation or DHCP. Personally, I would try sticking a router on there and setting it up as DHCP server before resorting to static IP addresses everywhere. Dante is one of the few ethernet-based protocols that prefers auto-negotiation over statics.

The other thing is that if you use decent network switches and set them up correctly, Dante supposedly plays nicely with a "Ring" topology. In theory, you should be able to plug the new branch in before unplugging the old branch, and hopefully stay on the primary the entire time. I'm reasonably confident this is a solvable problem with Dante but it's one of the things I'd have to sit down and tinker with and stress test. You would actually need to play around with the configurations on your network switches more so than any settings on your actual console system.

I don't have enough experience with MADI to speak intelligently about it but Outline's Newton DSP is supposed to play nicely between MADI and Dante and may be worth investigating. The problem is that if you're using a console and stagebox system that has controllable head amp over Dante and then try to bridge that over to MADI for this transition and back over to Dante, your console will probably be worthless for doing any head amp control of the stage box inputs. That functionality would likely be broken though you would still transmit audio.

You could try reaching out to Digico and see what they say. I vaguely recall that officially they don't support patching MADI links around and want to be point-to-point without extraneous connections in the middle of the signal chain. Not because it doesn't work but because they don't want to be caught making any guarantees for the hundreds of different ways that could be abused.
 

DrewE

Well-Known Member
Joined
Mar 18, 2019
Location
Vermont
Since Dante is just networking over IP (as I understand things), could you not attack this problem at the router and network level rather than the audio transport level? Set up routers with redundant links, and let them handle the failover, and have the audio equipment be none the wiser? If done properly, I think the whole thing would just look like a single network the whole while.

Mind you, I'm enquiring as one with at best a little theoretical knowledge, and not as one with any practical experience in pulling this sort of thing off!
 

dmx

Active Member
Joined
Apr 10, 2010
Location
Virginia
I think DrewE and MNicolai might be headed in the right direction with regards to the Ring-type network setup with Dante. In theory, all of your connections on the set piece could be connected to a switch (not router) located within this set piece. Your house-side audio configuration would remain the same, with the router (and switches as necessary) located on the other side of the connection break. Again, theory, but you are just creating multiple links across the same network. (That all made sense in my head, but not sure if it comes out well) . I would connect both the primary and secondary links to the same network on both ends, and use your multiple CAT interconnects to serve as your redundancy.
 

rphilip

Active Member
Joined
Mar 5, 2008
Location
SW Michigan
To directions to consider:

Direct out https://www.directout.eu/en/products/
Makes various MADI routers. Perhaps one of their products could do the fail over in the MADI side

I also agree with the others who have suggested investigating using network protocols to manage the change over seperatly on each network. This would mean you wound need 4 network cables. I think it would be some on of the Spanning Tree protocols but I don’t know networking to that level. They can automatically handle network fail over but I don’t know if they can do it fast enough. Perhaps these some way to tell it to move between the two primarily link when both are connected.
 

TimMc

Well-Known Member
Premium Member
Joined
Feb 15, 2017
Regarding TCP/IP packet-switched network data:

Unless you are sending data packets of digital audio *between* different networks, you don't need a router. That box behind your desk, connected to the DSL or cable modem, that has some antennae on it? That's several devices in 1 box: router, firewall, DHCP server, MAC address filter, 4 port switch (usually), and Wireless Access Point. For a closed audio network - a network not connected to any other devices or networks - you don't need routers, firewalls, MAC filters, or WAPs. In a network where devices are not routinely changed out, a DHCP server isn't needed. If you're running one big happy Dante network, you really don't want a WAP. Wireless sounds like a good idea but Audinate will tell you "no". Believe them. If you need a WAP to run console remote control, put it on a VLAN and keep the Dante traffic off of it.

@rphilip mentions using Spanning Tree Protocol and there was some discussion of this over at the Live Audio Board forums. I don't recall the details but my foggy memory tells me the STP wasn't a panacea and created other potential issues... or I could have this all wrong, too. For some reason IT guys end up running sound and we have a handful of them on the LAB who were very helpful. I'll see if I can find the discussion and post a link.

@ShowNet - I think a good part of the Dante issues were mentioned by Mike, mostly word clock/sync and device identification/control connection. In addition to the main network to transport the audio packets, Dante uses a couple of subnets for digital housekeeping - clock sync and device telemetry/control. Mike alluded to a couple of tricks to make device discovery happen faster but nothing can happen until each "new" device has stable clocking. I'm not MADI fluent so the Soundcraft and DigiCo (MADI-connected) users here will be along shortly, I'm sure. :cool:

EDIT: Found the ProSoundWeb LAB subforum discussion
https://forums.prosoundweb.com/index.php/topic,165849.msg1529004.html#msg1529004
 
Last edited:

FMEng

Well-Known Member
Fight Leukemia
Joined
Mar 31, 2008
Location
Tacoma, WA
Any time you disconnect a digital connection, it's going to take a few frames for the receiving device to decide it needs to switch to the other source. It'll never be seamless. This needs to be solved in the audio domain with a digital audio router. The switch from source A to source B needs to be forced before source A is removed.

Borrowing from the broadcast world, I would do this with an audio routing network, either with TDM or AoIP. Lawo, Wheatstone, Axia and SAS are the biggies in switching and carrying large channel counts of audio in broadcast plants. This will take a large budget.
 

tdrga

Well-Known Member
Joined
Feb 13, 2011
Location
Central TX
Since Dante is just networking over IP (as I understand things), could you not attack this problem at the router and network level rather than the audio transport level? Set up routers with redundant links, and let them handle the failover, and have the audio equipment be none the wiser? If done properly, I think the whole thing would just look like a single network the whole while.

Mind you, I'm enquiring as one with at best a little theoretical knowledge, and not as one with any practical experience in pulling this sort of thing off!
I borrowed this link from a post on the Lighting forum - it shows a ring network failover and restore in action.



Not sure if it would heal fast enough for audio streams, but it seems to be fast enough to not cause DMX problems.

-Todd
 

ShowNet

Member
Joined
Dec 5, 2017
Location
New York
Thanks all, some great responses! Let's see if I can hit all the major points...

It gets a little wiggity whack when you try and fail primary to secondary back to primary right away.
@MNicolai Firstly, I love your wiggity whack terminology :) Also a great analogy, and one that feels spot-on.

Again, theory, but you are just creating multiple links across the same network. (That all made sense in my head, but not sure if it comes out well) . I would connect both the primary and secondary links to the same network on both ends, and use your multiple CAT interconnects to serve as your redundancy.
@dmx Your theory is in the right direction, and physically (topology-wise) is sound. If I go network based, I would have a switch (or two, we're big on redundancies here) located on the set piece, with multiple cables going back to network switches located "on land" as it were. I don't know if you're aware, but any time you have multiple cables connecting two switches, you need one or a combination of the following in order to avoid indefinite loops: 1) A link aggregation protocol (the multiple cables logically appear as one single cable), 2) a Spanning Tree Protocol variant, which (simplified) manages the enabling and disabling of ports such that traffic is only received on one cable and not multiple cables, 3) less commonly, a ring protocol, which you suggested. I'll come back this point.

@rphilip @TimMc You both mentioned Spanning Tree, which is the obvious first thing to look at. I forgot to mention my background is a decade in IT with a heavy emphasis on networking ;) We have a robust, enterprise-level show control network which does rely on Spanning Tree, and your feelings are right, STP is too slow and, I think, would cause the network switches to get too "wiggity whack." There are ways of optimizing STP, but I really haven't bothered because I just don't see it working.

Now, a ring protocol however... This is an important area I need to gain some experience in, but as shown in the video @tdrga posted it seems like it may well work. I need to do further research and bench testing to see how well they work in my scenario. But, having said that...

I'm a networking guy, I love networking! But ever since I started digging into this, I've been liking MADI more and more. Some of my concerns with a network-based solution is that I'm fighting to overcome two recovery points - the ethernet recovery, both layer 1 and layer 2, and then the layer 3 AoIP protocol recovery. That's three layers that have to be established before audio can proceed. So, at the moment, I'm not focusing on a network-based solution, but ring topology is something I need to explore.

As I understand it, MADI is essentially a one-way serial bit stream, which means there's very little handshaking involved when establishing a connection. Establish electrical connectivity, receiver locks onto incoming bit stream, BOOM, done! (I think). Doing MADI also removes the failure points and complexities of intermediate network switches, because MADI is point-to-point (unless there's a MADI router, but that's still a reduction in devices in the chain).

@FMEng Thanks for pointing out Wheatstone, Axia and SAS. I was aware of Lawo - it was standing in the Lawo booth at NAB last month that I first realized MADI might be a realistic solution. However I wasn't aware of the other vendors, so that's a great lead! Yes, they're pricey, but if it solves this problem the powers that be will go for it. One question, what do you mean by TDM? Time Division Multiplexing?

Great stuff so far, thanks!
 

ShowNet

Member
Joined
Dec 5, 2017
Location
New York
A couple other avenues I'm exploring:
- Network-wise, I'm mentally throwing around the idea of spanned layer 2 (dispersed network switches appearing, on the ethernet level, to be on the same switch/broadcast domain). This is more a datacenter technology and I've never worked with it, but it might help keep Dante happy if it alleviates route changes.

- Audio here is based on a DiGiCo Optocore ring. The idea of intentionally breaking that ring, the backbone of all audio for the show, it not likely to go well. But a DiGiCo rep mentioned a device that can create a "spoke," or breakout box of sorts, that keeps the primary ring fully intact. I'm looking for the business card of that rep, but if anyone else knows what device he's talking about... And @FMEng you mentioned that breaking any digital connection will never be seamless. The DiGiCo guys did warn me that with optocore there would be a "high ping" or.. "something" that happens. It's apparently very quick, but long enough to make you look up and go "huh? did something just happen?" I'm hoping MADI might prove your statement wrong, but if not, I can live with that kind of moment.
 

Ben Stiegler

Well-Known Member
Joined
Aug 3, 2017
Location
Sf Bay Area
This is a left field thought, but ... imagine failing over to a “Milli Vanilli” track for X seconds (or a verse, etc) while all the plumbing and logical layers which ride it are swapped.

What’s on the wagon ... just instrumentalists? Or vocalists who are carefully watched?Why are there 50 audio paths back to it? Is this like Radio City Music Hall scale?

Think also about hardened tactical fiber vs. copper for Dante, and whether either from the grid, the upstage wall or a wing,or thru a floor slot the fiber and it’s backup sister could be routed and cable managed without having to break and reconnect anything. It would be a much lighter and smaller jacket...


One more idea. Do you need phantom power for any of those channels?

If not, and if you stay in the analog realm, think about something like a high quality transformer-based mic splitter box. I’m guessing you get less or zero click/pop that way. And you can always have a phantom power injector on the wagon, powered with a small UPS to help ride across AC power swapping. Disable its beep alarm unless it’s in the right key...
 
Last edited:

Ancient Engineer

Well-Known Member
Joined
Sep 21, 2017
Location
Sandusky, Ohio
QSC and the Q-sys system might meet your needs. With that many paths you'd need to add a few extra channels worth of IO.

You'd need a core and a IOFrame or two on a switch located on the set piece.
 
  • Like
Reactions: RonHebbard

Ben Stiegler

Well-Known Member
Joined
Aug 3, 2017
Location
Sf Bay Area
QSC and the Q-sys system might meet your needs. With that many paths you'd need to add a few extra channels worth of IO.

You'd need a core and a IOFrame or two on a switch located on the set piece.
Would Q-Sys reconnect fast enough or does it handle dual network better?
 
  • Like
Reactions: RonHebbard

Ancient Engineer

Well-Known Member
Joined
Sep 21, 2017
Location
Sandusky, Ohio
No latency on switching from Network A to B.

About 15ms for a reconnect and sync.

So as long as you had at least one connection made you'd likely be fine.

Nathan Makaryrk at QSC could direct you the right engineer who could tell you exactly.

If you DM me I will give you his email.
 
  • Like
Reactions: RonHebbard

TimMc

Well-Known Member
Premium Member
Joined
Feb 15, 2017
The more I look at this the more I wonder why this isn't a pre-recorded, playback number.

In 2000 we had Riverdunce, The Show! in our PAC. Everything except the lead Irish and American tap routines, the Flamenco number and the vocal solo performances were playback, with the entire live side serving as backup for the Otari RADAR system. It was pathetically amusing to see the band mobbing the poor monitor mix engineer and going on about how they were disappointing the audience because their IEM and speaker mixes weren't perfect. I didn't have the heart to tell them that nothing they played on stage was coming out of the PA except for bleed into the soloists mics.

Hey, it's theatre. What do you mean it's supposed to be live? The investors, producer, director and designers want repeatable...