EOS firing OSC to Qlab - working, but inconsistently?

flamingo

Member
Hello, I've got a problem that I'm having a hard time troubleshooting. Our holiday is show up and running, but nearly every performance there is a random sound cue that does not fire when the linked light cue is taken.

Our setup: EOS show file cue fired > lighting network switch > cat 5 ethernet > sound mac computer > Qlab file cue is triggered

I'm new to the space and haven't had much time to wrap my head around the network set up. There's not a ton of detailed documentation unfortunately. We have a network switch for lighting, and a Cat 5 cable that runs down from the light booth upstairs to the sound mixing position FOH. It's a physical cable that was run thru a snakehole, and the other end plugs into sound's Qlab mac. Lighting is connected to the lighting network switch, and has a RPU as primary and EOS Ti as backup.

Every show, one or two sound cues are not firing correctly. It is not the same cue every time (although there have been a couple repeats as of this weekend).

Dropped cues have been in the middle of clusters that otherwise fired correctly, but have ALSO been cues that are fired after many minutes of no OSC triggers. For this reason I don't think that our switch is turning off ports or anything like that. The ethernet cable connection has been checked with a tester and found ok.

I have not power cycled the switch, and frankly I'm hesitant to, because I don't know exactly what all is plugged into it.

We are not able to replicate the issue, making the troubleshooting process incredibly frustrating. The next step is basically to abandon OSC and try MSC instead.
 
Sounds like a packet drop issue.
How long is the CAT5 cable run?

From the switch I would connect another laptop and then open cmd and type ping “Mac IP” /t and run it for the length of the show.

If it comes back fairly clean of packet drops then I would say run a ping test and tech the show while watching it to see if something else is effecting your bandwidth.

Also make sure you are running static IPs.
 
@Amiers - Static IPs all around. I'll see if we can get a laptop up there for this evening's show and check that out. I'm not used to digging into network stuff so I forgot that was possible!

@almorton - I agree :) And I understand this to be the case.
 
Does the sound cue always trigger, but is delayed? Or does it not trigger?

Are you using TCP or UDP OSC? If TCP, which port? If UDP, which ports? You can find this information in your Eos console.

Are you triggering by sending to a specific address (10.101.2.100), or by sending to a broadcast address (10.255.255.255)?

What kind of network switch?
 
@ScottT - That's a great question!
I'm struggling to confirm that we're using TCP. That's what I WANT to use. I have TCP selected in the shell on EOS, exactly how it says to set up on the website. But when we look at the settings in QLab, I can't figure out a way to confirm that we're using TCP. We're sending to the IP address of the Qlab computer.

Honestly I don't know what kind of switch, but it's Cisco... not very helpful I know. I haven't got that far along in my self-guided orientation LOL! But, pre-covid/my start date, OSC and midi have been used successfully between LX, PJ and sound on other shows using the same infrastructure. That's what I've been told.

I do not believe there is a delay; simply not firing.
 
I had the same issue working in a small theatre company several years ago. Running an ETC Ion triggering QLab 3 on a trash can MacPro via OSC. It worked 95% of the time, but occasionally (and randomly) QLab wouldn’t fire. At first we were connecting Cat 6 directly from console to computer, then we added an unmanaged switch. We swapped cables, changed ports, but nada. Never did nail down the issue and like you went to MIDI which has reliable worked for them for years now. It’s a shame too, with OSC needing no additional hardware or special cables.

OP, are you running QLab 4?
 
Following up to get more information here. I have been using OSC to take cues from EOS for years (10+). Usually with OSCRouter running on the Mac with QLab. Last several years, I use QLab to call OSC to EOS and this has always worked very reliably. However, a recent show made me go back in time to use OSC from the console to trigger QLab. I started up OSCRouter, configured as I always had, and it works. But about 1-4 cues per show would not fire Qlab. Sometimes same cues, different day. Sometimes different cues. I had cue 42 not fire OSC 3 days in a row. Then started working for 2 days, then tonight cue 42 did not fire OSC again.
I have tried everything I can think of. And continue to have issues. Here is the setup.

ION XE -> Cat6 -> Cisco Unmanaged 24 port switch -> Cat 6 -> QLab Mac. There is a Unifi Ubiquiti USG that I use all the time as a router connected to the switch as well. This setup has always been reliable in multiple venues and many shows. So what is different. ION XE running latest 3.1.1 and QLab running latest 4.6.11.

Both devices have static IPs assigned, and are always pingable. The network appears reliable. I have changed cables, tried a different switch, Unifi Ubiquiti, different Mac Laptop. Removed OSCRouter, and using OSC Cue Send String now. Still had same issue. Interesting thing, When using OSCRouter, I could see the logs and I noticed that I was not receiving certain cues from the ION XE. Log files would show Cue 40, Cue 41, Cue 43. and Skip cue 42, but I saw the Stage Manager press go and the lights would change for 42.

I can find no evidence of dropped packets. Using Unifi software I can monitor the devices and everything looks very normal with no Events or Alerts. Bandwidth is minimal. 1 Gbps switch, using less than 5 Mbps. I have to think this has something to do with the new ION XE or latest EOS software. I am yet to find evidence that the ION XE is sending every OSC string.

Again, I have used this setup for years on older 2.x ION and EOS software. Never had a miss that I couldn't track down to a bad cue I wrote in Qlab or missed cue number. Now I can't trust it, and that is a problem. I also used similar setup this past summer with EOS Nomad 3.1 on a Mac and it never missed a cue, ever. Very reliable. But I sent OSC to EOS directly from QLab, not the reverse, as I am doing now.

I am going to setup the EOS Diagnostic Tab tomorrow and see if I can get confirmation on OSC strings being sent straight from the console.
Any one else have any thoughts on this?? This is a real stumper and seeing others have this same problem leads me to believe there is a more legitimate bug someplace.
 
I've seen similar with a classic !on running 2.9.x - occasional missed cues from EOS to QLab. I haven't (yet) got around to running something like wireshark to packet sniff.
 
I did some troubleshooting on this. Only because it bugs me. I can't reproduce on command, but across ~150 cues, it will drop from 0-4 cues in 2 hours. It always looks similar to this.
In this case, cue 85 did not fire on QLab, but was clearly called on ION. You can see the ~3 min gap in time between cues. 84.5 completes to 100%. Then we miss the fire for 85, but it picks up the 85 @ 14% and show all remaining messages. Just skips the fire and start cues. All of the other messages are there. In this case, is skipped about 10 OSC messages from FIRE to pending/cue/text "86...."

This is what QLAB is receiving

Apr 22 20:55:12 STK-Work-MacBook-Pro QLab[45947]: <F53OSCSocket UDP 192.168.10.230:53001> sent OSC message: /eos/out/active/cue 0.69
Apr 22 20:55:12 STK-Work-MacBook-Pro QLab[45947]: <F53OSCSocket UDP 192.168.10.230:53001> sent OSC message: /eos/out/active/cue/text "84.5 0.9 69%"
Apr 22 20:55:13 STK-Work-MacBook-Pro QLab[45947]: <F53OSCSocket UDP 192.168.10.230:53001> sent OSC message: /eos/out/active/cue 1
Apr 22 20:55:13 STK-Work-MacBook-Pro QLab[45947]: <F53OSCSocket UDP 192.168.10.230:53001> sent OSC message: /eos/out/active/cue/text "84.5 3.0 100%"
Apr 22 20:58:17 STK-Work-MacBook-Pro QLab[45947]: <F53OSCSocket UDP 192.168.10.230:53001> sent OSC message: /eos/out/pending/cue/text "86 Moore Home Int 6.0"
Apr 22 20:58:18 STK-Work-MacBook-Pro QLab[45947]: <F53OSCSocket UDP 192.168.10.230:53001> sent OSC message: /eos/out/active/cue 0.14
Apr 22 20:58:18 STK-Work-MacBook-Pro QLab[45947]: <F53OSCSocket UDP 192.168.10.230:53001> sent OSC message: /eos/out/active/cue/text "85 6.0 14%"


This is what ION XE is sending

IMG_0203.jpeg
 
There has been some discussion of this issue on the QLab mailing list (https://groups.google.com/g/qlab/c/lOhimS59wug) and one solution is to use /eos/filter/add to restrict the messages being sent to QLab:

/eos/filter/add /cue/*

Note the space between the "add" and the "/cue/*"

Sending this from QLab to the EoS/Ion/Element will result in only receiving /cue/X/start OSC messages on every cue instead of the 10+ other messages that are sent by default.

Reducing the number of OSC messages appears to improve the reliability.
 
There has been some discussion of this issue on the QLab mailing list (https://groups.google.com/g/qlab/c/lOhimS59wug) and one solution is to use /eos/filter/add to restrict the messages being sent to QLab:

/eos/filter/add /cue/*

Note the space between the "add" and the "/cue/*"

Sending this from QLab to the EoS/Ion/Element will result in only receiving /cue/X/start OSC messages on every cue instead of the 10+ other messages that are sent by default.

Reducing the number of OSC messages appears to improve the reliability.
Wait. EOS sends *11 OSC messages* on a GO?

Also... oh, wait. I see. You send that filter command *in the OSC port*, and then EOS executes it. Got it.

@Zebetz: You already know this?
 
we occasionally trigger QLab for video from our Ion and have seen this issue as well. My understanding is that OSC uses UDO as its transmission protocol. UDP is designed to be fast, not reliable. UDP messages are occasionally dropped.

what I do when trying to talk to QLab from my ION is to simply send every message twice. IE I send the “/cue/15/start” mrssage in cue 34, then I write a cue 34.1 with the same command and put a follow in time of .01. Since we have been doing this we have not seen a problem ( although we do keep the QLab machine near the board op just in case)
 
OSC doesn’t care about the transport layer. It works with TCP, UDP, serial USB, and could probably tolerate semaphore if somebody was sillly enough to implement it. QLab expects UDP. It sure would be nice if Figure53 would provide support for TCP. FWIW, I use explicit OSC string cues instead of relying on implicit output from Eos, and haven’t seen a missed cue yet.

We have seen the Macs bog down if their wireless NIC is enabled so disabling it is part of our preshow checklist.
 
OSC doesn’t care about the transport layer. It works with TCP, UDP, serial USB, and could probably tolerate semaphore if somebody was sillly enough to implement it. QLab expects UDP. It sure would be nice if Figure53 would provide support for TCP. FWIW, I use explicit OSC string cues instead of relying on implicit output from Eos, and haven’t seen a missed cue yet.

We have seen the Macs bog down if their wireless NIC is enabled so disabling it is part of our preshow checklist.


Putting on my stupid hat. Can you clarify what you mean by “explicit OSC string cues”?
 
Eos has 2 modes of sending OSC strings: implicit and explicit. Explicit cues can be sent in a variety of ways including cues, macros, and magic sheets.

There's also this recently introduced syntax. I'm not sure how it's supposed to work, or whether the feature is fully baked.
Documentatiion on the feature is pretty scant. Allegedly, it applies to the cue list instead of a specific cue. It may be intended to provide a means of translating implicit OSC messages sent to other devices, or parsing incoming OSC messages received from other sources. There was a bug report that incorrect use of the feature could cause an infinite cue loop, but I haven't kept up on whether it's resolved.
 

Users who are viewing this thread

Back