Control/Dimming ETC Sensor & Unison control issues

EQUIPMENT (for reference)
175 circuits with ETC sensor modules
3 ETC CEM+ head units
2 large racks (unfamiliar with model)
1 smaller rack with phase loss detection (unfamiliar with model)
Unison control system (unfamiliar with model name, but if provided with a picture I can confirm/deny whether it's the right one)
Main Board - ION XE-20
All dimmer racks are connected using a 24-port cisco/linksys switch - with the dimmer packs being the only thing connected to it
DMX runs on 2 lines using DMX A and B on the CEM+ head units, one being the main board and the other being for unison control



Now here's gonna be a long one. I'm having an intermittent issue where someone will turn off the lights using a unison touch panel, which will either work as intended, not do anything, or in a rare situation, cause every dimmer rail to start rapidly flashing. Apparently, this has been an ongoing issue, where it would happen once or twice a month (I'm not sure if it only happens using the unison system or both using the board and unison) but it has ramped up to being more frequent, happening almost every day. For fairly obvious reasons, and with a large production happening in the coming weeks, this issue is simply detrimental for both fire safety reasons and audience enjoyment. Heres everything I know.

  • The fan in rack 1 (The one where most of the stage lights are routed) has stopped working, resulting in internal temps of around 110 degrees, however the fan in the adjacent rack 2 (which largely is turned off because of how the wiring is done) continues to work
  • I noticed that when I turned off the main board before turning off unison it caused everything to start strobing, but when I tried to recreate it, everything worked smoothly.
  • I restarted the unison head unit, which made no noticeable change with the error still occurring afterward, but we left it unplugged overnight and the next day (day of posting - maybe not a long-term fix) everything worked just fine.
  • The DMX runs are reversed on rack 1, with the board being DMX A and Unison being DMX B, which is opposite of racks 2 and 3
  • The facility was built in 2006/07 and everything has a thin layer of dust, with the exception that the filters on the racks were recently cleaned.

Any information is helpful, especially if there's any nice settings I may not know about, being new to ETC architectural controls and their dimmer racks.

As always, working on a budget here and replacing the fan with an OEM model is surprisingly expensive and there is going to be an LED refresh coming soon so that's why the seemingly obvious fix is being questioned first.

Thanks!!
Stephen
 
Rack fan is something to get rectified but I doubt it's the cause of your issues. This is an easy fix all things considered but given the arc flash and electrocution concerns, I would strongly recommend you leave this up to an electrician or ETC field tech (who may be a tech from your local ETC dealer) to remedy.

Do you have old Unison (Legacy) or Paradigm? There's a major difference between the two. Circa 2006/07, I would expect old Unison with floppy discs.

Is your Unison processor on a UPS? If so, has the battery been changed anytime recently? (Weird things can happen on a UPS with an old battery where the processor may start to get ugly power delivered to it).

What is the DMX distribution? Opto-splitters, network gateways, or direct from console to dimmer racks? If they're network gateways, that largely points away from this being a DMX issue and more toward it being a controller. If it's splitters or direct to console, there's a better chance this could be unhappy DMX errors that are propagating through your system.

It may be that your Unison processor is starting to die. You will very likely need to reach out to ETC Tech Support using the number on their website. Unless it's a more obvious UPS issue, I would plan for the likelihood you will need to send the Unison control module into ETC. The old-style Unison can have a variety of quirks that are not indicative of a permanent hardware failure where you may need to reload the config or "hit the jukebox" to clear out -- I wouldn't claim to be an expert on this voodoo as it's not something I personally deal with on a regular basis these days, but ETC tech support will often have you do a number of troubleshooting steps before you pull the controller and send it back to them.

If you do have old Unison, it is a good time to start budgeting for upgrades. Those controllers are quiet old at this point, you cannot get replacement parts to repair them unless you/ETC get lucky. I wouldn't bank on a legacy Unison processor being able to survive much longer than it already has.

ETC Tech Support has seen every possible issue an old Unison processor can possibly encounter. I wouldn't hesitate to call them directly. Your issues will either be 1) wholly unrelated to your processor or 2) definitely your processor and only they can do something about it. Either way they will be able to help you. If you have a dead rack fan, it's probably time to have a field tech do a once-over on the entire system anyway to clean dust of the racks and generally assess the current state of things.

It is not outside of the realm of possibility you may have a CEM+ problem instead, but that seems more likely to be an issue for a specific rack of dimmers whereas you seem to be seeing this issue across all racks. If your issues are across all of your dimmer racks, this seems less likely to be the culprit. May also be a good time to start budgeting an upgrade to CEM3 which includes upgrading the backplanes of your dimmer racks. This may or may not have any influence on your current problem but someday eventually you will likely have to do this. If you're talking LED refresh in the near future, also a good idea to budget R20AF modules into that upgrade so you're ready when that day comes.

Also -- which version of Eos software is running on the Ion XE, what is the speed of your DMX output set to, and is run into your system by traditional DMX cable over network control from a data cable?
 
Hi Stephen,
Yes, a call to ETC Tech Support is a good place to start.
Heat build up in the racks will cause a number of issues. The first thing is going to be getting the fan fixed.
If you are in Dana Point, I can recommend some authorized service providers in Southern California. Message me if you need some recommendations.
Take care,
John
 
Rack fan is something to get rectified but I doubt it's the cause of your issues. This is an easy fix all things considered but given the arc flash and electrocution concerns, I would strongly recommend you leave this up to an electrician or ETC field tech (who may be a tech from your local ETC dealer) to remedy.

Do you have old Unison (Legacy) or Paradigm? There's a major difference between the two. Circa 2006/07, I would expect old Unison with floppy discs.

Is your Unison processor on a UPS? If so, has the battery been changed anytime recently? (Weird things can happen on a UPS with an old battery where the processor may start to get ugly power delivered to it).

What is the DMX distribution? Opto-splitters, network gateways, or direct from console to dimmer racks? If they're network gateways, that largely points away from this being a DMX issue and more toward it being a controller. If it's splitters or direct to console, there's a better chance this could be unhappy DMX errors that are propagating through your system.

It may be that your Unison processor is starting to die. You will very likely need to reach out to ETC Tech Support using the number on their website. Unless it's a more obvious UPS issue, I would plan for the likelihood you will need to send the Unison control module into ETC. The old-style Unison can have a variety of quirks that are not indicative of a permanent hardware failure where you may need to reload the config or "hit the jukebox" to clear out -- I wouldn't claim to be an expert on this voodoo as it's not something I personally deal with on a regular basis these days, but ETC tech support will often have you do a number of troubleshooting steps before you pull the controller and send it back to them.

If you do have old Unison, it is a good time to start budgeting for upgrades. Those controllers are quiet old at this point, you cannot get replacement parts to repair them unless you/ETC get lucky. I wouldn't bank on a legacy Unison processor being able to survive much longer than it already has.

ETC Tech Support has seen every possible issue an old Unison processor can possibly encounter. I wouldn't hesitate to call them directly. Your issues will either be 1) wholly unrelated to your processor or 2) definitely your processor and only they can do something about it. Either way they will be able to help you. If you have a dead rack fan, it's probably time to have a field tech do a once-over on the entire system anyway to clean dust of the racks and generally assess the current state of things.

It is not outside of the realm of possibility you may have a CEM+ problem instead, but that seems more likely to be an issue for a specific rack of dimmers whereas you seem to be seeing this issue across all racks. If your issues are across all of your dimmer racks, this seems less likely to be the culprit. May also be a good time to start budgeting an upgrade to CEM3 which includes upgrading the backplanes of your dimmer racks. This may or may not have any influence on your current problem but someday eventually you will likely have to do this. If you're talking LED refresh in the near future, also a good idea to budget R20AF modules into that upgrade so you're ready when that day comes.

Also -- which version of Eos software is running on the Ion XE, what is the speed of your DMX output set to, and is run into your system by traditional DMX cable over network control from a data cable?
Yes, we are using legacy unison with passthrough for the dimmers

I'm going to reach out to ETC today.
 
You have a number of issues going on, not neccessarily related, fan on a dimmer rack as one, Unison processor for 2. A call to tech support is in order to maybe have them talk you through a reset of the Unison processor (There's a hidden reset button). You will be needing a service call however to get the rack fan fixed. Unison ER-4 proccessors were old in 2002, I'm surprised they installed legacy Unison in 2006 (any chance its actually a Paradigm ?, but possibly a system specified much earlier, so 20 years later means possibly a replacement is in order.
 
So for reference, after a call with ETC, we concluded that the CME processer (which was manufactured in the last production year, fun fact) has a deteriorating DMX processing chip. This would be a relatively easy fix, just being able to send off our current unit as they send in a loaner when they're working on it. This, of course, is complicated as we plan to upgrade to all led in the coming year, just needing to get through 2 shows. So our current plan is to ask for a lender unit when we get a quote back for the new system from our local ETC dealer (which should be happening in a matter of days). All in all, a great temporary fix has seemed to be unplugging the unison box for that one night, and just really being frugal about turning off the lights you don't need in that moment, because ultimately the way we design our shows leaves very few moments where the entire stage is lit at full intensity.

Thank you to everyone who took the time to respond!
 
SOLUTION

What ended up being the fix was a loose chip in the Unison processing unit, which I believe was responsible for writing garbled data to the DMX universe controlling the dimmers. We noticed this after restarting the unit and when it refused to boot (tl;dr), finally resorted to opening it up and seeing if anything was obviously wrong. We noticed that one of the socketed processors was up and out of the socket, so we pushed it back in so far we've had no issues. Yes, a system upgrade is coming and I am skeptical of it continuing to work through the upcoming show (which opens next Friday), but if you're right now in an electrical closet with doors being opened 30 minutes late, check to make sure everything is seated properly in your legacy unison processer! you never know what might just work!
 
Thats so odd. A rack mounted unit develops a loose part. I wonder what would cause that ?
 
Depending on the type of socket used, thermal cycling has a way of causing DIP chips to migrate slightly out of their sockets over time. 8 pin chips more than 16 pin and up. Its always one of the steps in any bench repair I do - Push all the socketed chips back in.
 

Users who are viewing this thread

Back