thelatinist
Member
Seems as though we might have determined the cause of our issue this afternoon when our transformer blew up. CL&P is on their way out to replace it sometime tonight. With any luck this will solve our problem...
Control/Dimming Flickering Lights
- Thread starterthelatinist
- Start date
Blowing up your transformer is a really unique way to diagnose your problem and not part of the standard troubleshooting process. 
Sent from my Galaxy Nexus using Tapatalk 2
Sent from my Galaxy Nexus using Tapatalk 2
It's a shame you didn't get that BMI/Power Quality Analyzer installed in time to prevent a catastrophic failure. Although it's possible/probable the local poco would not have done anything about it until there was an actual power interruption (and the associated loss of revenue). (No offense intended toward CL&P.)
Glad to hear (for once) it wasn't an issue with the dimmer rack. (No offense intended toward Strand Lighting.)
This has been a great thread and I do hope all your flickering problems disappear with the installation of the new mains transformer. I'm sure (99.44%) they will.
Last edited:
phenry
Active Member
No offense taken, I will assume that was a general comment that would have been used regardless of dimmer brand.
And no, blowing up the transformer is not a recommended troubleshooting procedure.
thelatinist
Member
So CL&P came and installed a new transformer on the mains. So we spent some time moving instruments back to the left leg, turned everything back on, and tested them. Alas, the flickering is unchanged! I was so sure when the transformer blew that that had been the cause of our problems all along. Now we're left...where? This is getting so frustrating.
Since several people have asked questions about what we have tried, I will summarize here:
1. Moved lighting console to second DMX wall jack.
2. Updated console software and firmware.
3. Unplugged all DMX-controlled moving heads.
4. Repatched DMX universe to DMX 2 output on console.
5. Opened and tested with known good show.
6. Repatched dimmers in console settings.
7. Tried alternate DMX source (Artnet-to-DMX box from PC with several software programs).
8. Turned off DMX opto-switch and disconnected DMX. Replicated problem under local control from C-21 front panel.
8. Removed and cleaned all dimmers.
9. Swapped dimmers one-for-one from left leg to right leg.
10. Removed and cleaned filters from rack.
11. Removed and cleaned C21 processor.
12. Turned off power and cleaned dust from the feed area beneath the rack.
13. Hired in an electrician, who spent nearly eight hours:
a. Checked all connections in power feed.
b. Checked bonding of ground and neutral.
c. Checked for voltage and amperage across ground and neutral
d. Checked for voltage and amperage between ground and ground from other panel.
e. Checked for and found amperage but no voltage on ground. Spent several hours chasing this amperage. Ultimately this amperage seemed to be induced current in the ground wire, which ran through the same conduit with 400-amp feed(s).
f. Checked tightness of connections in rack.
g. Checked voltage under load and found less than 1 volt decrease (121.0 to 120.2) under heavy load and while flickering.
h. Consulted with folks at Strand and at Century Lighting Service, and conducted several tests at their direction.
14. Strand technical support concluded that the problem was the C-21 processor. They shipped us a new processor, which we installed. No change.
15. Spoke extensively with Joel from Century Lighting service and performed several tests and changed several settings at his direction.
16. Hired in Joel from Century Lighting Service, who spent 6.5 hours doing the following:
a. Checked all connections in power feed.
b. Tested ground, neutral, etc. (all the stuff previously tested by our electrician).
c. Tried test dimmer in several placed throughout rack, determined that only the left leg was affected.
d. Checked many contacts in processor and CIC under power.
e. Removed, disassembled, thoroughly cleaned, tested, reassembled, and reinstalled: 1) C-21 processor 2) CIC 3) Phase Reference Card.
f. Replaced entire control housing (containing processor, CIC, Phase Reference Card, power supplies) with known working electronics.
g. Unplugged control out ribbon cables from board and connected LCD test board, observed flickering on the LCD panel.
h. Removed, swapped and reinstalled dimmers in an unsuccessful attempt to isolate the problem to certain locations/dimmer modules.
i. Disconnected all control in connections, plugged DMXster directly into CIC, duplicated problem. Noticed consistent flicker at approximately 14% while fading up.
j. Removed several dimmers and tested current across ground and neutral. Noticed a small spike (~2V) when flicker occurs.
g. Concluded that there is nothing wrong with our dimmer rack, suspected transformer or power line issue, recommended a recording line monitor.
16. We moved all instruments from the left leg for our show Friday and Saturday. Successfully ran show using only right leg.
17. Tuesday afternoon the mains transformer failed and was replaced.
After all this, our issue remains the same as it was when we started troubleshooting in early January.
Since several people have asked questions about what we have tried, I will summarize here:
1. Moved lighting console to second DMX wall jack.
2. Updated console software and firmware.
3. Unplugged all DMX-controlled moving heads.
4. Repatched DMX universe to DMX 2 output on console.
5. Opened and tested with known good show.
6. Repatched dimmers in console settings.
7. Tried alternate DMX source (Artnet-to-DMX box from PC with several software programs).
8. Turned off DMX opto-switch and disconnected DMX. Replicated problem under local control from C-21 front panel.
8. Removed and cleaned all dimmers.
9. Swapped dimmers one-for-one from left leg to right leg.
10. Removed and cleaned filters from rack.
11. Removed and cleaned C21 processor.
12. Turned off power and cleaned dust from the feed area beneath the rack.
13. Hired in an electrician, who spent nearly eight hours:
a. Checked all connections in power feed.
b. Checked bonding of ground and neutral.
c. Checked for voltage and amperage across ground and neutral
d. Checked for voltage and amperage between ground and ground from other panel.
e. Checked for and found amperage but no voltage on ground. Spent several hours chasing this amperage. Ultimately this amperage seemed to be induced current in the ground wire, which ran through the same conduit with 400-amp feed(s).
f. Checked tightness of connections in rack.
g. Checked voltage under load and found less than 1 volt decrease (121.0 to 120.2) under heavy load and while flickering.
h. Consulted with folks at Strand and at Century Lighting Service, and conducted several tests at their direction.
14. Strand technical support concluded that the problem was the C-21 processor. They shipped us a new processor, which we installed. No change.
15. Spoke extensively with Joel from Century Lighting service and performed several tests and changed several settings at his direction.
16. Hired in Joel from Century Lighting Service, who spent 6.5 hours doing the following:
a. Checked all connections in power feed.
b. Tested ground, neutral, etc. (all the stuff previously tested by our electrician).
c. Tried test dimmer in several placed throughout rack, determined that only the left leg was affected.
d. Checked many contacts in processor and CIC under power.
e. Removed, disassembled, thoroughly cleaned, tested, reassembled, and reinstalled: 1) C-21 processor 2) CIC 3) Phase Reference Card.
f. Replaced entire control housing (containing processor, CIC, Phase Reference Card, power supplies) with known working electronics.
g. Unplugged control out ribbon cables from board and connected LCD test board, observed flickering on the LCD panel.
h. Removed, swapped and reinstalled dimmers in an unsuccessful attempt to isolate the problem to certain locations/dimmer modules.
i. Disconnected all control in connections, plugged DMXster directly into CIC, duplicated problem. Noticed consistent flicker at approximately 14% while fading up.
j. Removed several dimmers and tested current across ground and neutral. Noticed a small spike (~2V) when flicker occurs.
g. Concluded that there is nothing wrong with our dimmer rack, suspected transformer or power line issue, recommended a recording line monitor.
16. We moved all instruments from the left leg for our show Friday and Saturday. Successfully ran show using only right leg.
17. Tuesday afternoon the mains transformer failed and was replaced.
After all this, our issue remains the same as it was when we started troubleshooting in early January.
So much for the 0.56% that Murphy says will bite you every time.
Hey epimetheus, could the problem be upstream of the local mains transformer? Did CL&P offer any explanation of how/why the old one failed? Was it spectacular?
I guess thelatinist still needs to put a BMI / Power Quality Analyzer on his service for a week or so to see what, if anything, it shows.
Although I have no idea how to work one, I'm a strong believer in them.
Note the author.
Hey epimetheus, could the problem be upstream of the local mains transformer? Did CL&P offer any explanation of how/why the old one failed? Was it spectacular?
I guess thelatinist still needs to put a BMI / Power Quality Analyzer on his service for a week or so to see what, if anything, it shows.
Although I have no idea how to work one, I'm a strong believer in them.
More at View attachment Hidden_Harmonic_Filter.pdf (PDF warning!)
Note the author.
thelatinist
Member
So much for the 0.56% that Murphy says will bite you every time.
Yeah. I feel like I spoke too soon last night.
I am wondering this myself. Anyone have any ideas?
Unfortunately, I wasn't there when CL&P replaced the transformer in the wee hours of the morning so I got no feedback from them. I was there when it failed, though. We had turned off the 400A breaker that feeds our rack while we cleaned the filters, etc. When we turned it back on, the breaker blew and all the power in the building went out. There was no fire, and by the time we checked the transformer housing there was no obvious sign of failure.
Yeah, that's the plan right now. Does anyone have a recommendation for a contractor [near Watertown, CT] to do a power quality analysis? We're hoping to avoid relying on CL&P for this.
More at(PDF warning!)
Note the author.
Interesting article. Joel mentioned that he'd run into that exact problem at an installation -- the customer had had a harmonic filter installed on the mains at some contractor's recommendation and was wondering why their lights were suddenly flickering.
Last edited by a moderator:
The harmonic filter was a good thought, but if there was one, it would not discriminate between one hot leg and the other. AFAIK, they are applied only to 3 phase systems and this is single phase. At this point, I am stumped. The power quality analyzer is certainly the next thing to do.
It would also be worth discussing the situation with an electrical engineer from the utility company. (This person is not the line crew foreman. The engineer would be someone in their office who does design work.) Perhaps the engineer would agree to investigate with your electrician. There's a good chance that whatever caused the transformer to fail is what is causing the odd dimmer behavior. The utility company has good incentive to look further into this because they stand to lose an expensive asset if the new transformer fails. That risk still exists. The engineer should be very capable of solving the mystery and would have access to good diagnostic tools.
It would also be worth discussing the situation with an electrical engineer from the utility company. (This person is not the line crew foreman. The engineer would be someone in their office who does design work.) Perhaps the engineer would agree to investigate with your electrician. There's a good chance that whatever caused the transformer to fail is what is causing the odd dimmer behavior. The utility company has good incentive to look further into this because they stand to lose an expensive asset if the new transformer fails. That risk still exists. The engineer should be very capable of solving the mystery and would have access to good diagnostic tools.
Last edited:
thelatinist
Member
Well, I was misinformed. The transformer was not replaced, as the plant manager had told me; it was merely tripped, probably due to overload, and was reset. So it remains possible that the issue is being caused by the transformer.
I called CL&P today to get the process started. As I feared, the CSR would not escalate the issue without the problem first being investigated by a lineman. So they sent out a lineman this morning (coincidentally the same one who originally installed the transformer ten years ago), who checked the voltage and, as expected, was unable to find any problem. His favorite phrase seemed to be "You've got 120 volts." Because he had no idea about...well, anything...he called in the meter tech (also coincidentally the same one who installed the meters and wiring from the transformer to the inside panel ten years ago). He checked voltages at the meter and inside, both before and after the stepdown transformers that feed the meter. He, too, then declared that we had 120 volts.
At that point they really wanted to declare it an interior problem and send me back to Strand and our electrician, but I pressed and he said that he could install a recording voltage meter. This meter, unfortunately, will record voltages only at 1-minute intervals, which is going to do nothing to identify our problem -- which occurs when we have 120 volts, anyway! When I expressed my doubts, he admitted that it was unlikely to find anything. He said that after that "if you're still not satisfied" they could escalate it to the power quality assurance group (or something to that effect), which could, in his words, "bring out the big-boy meters." Which is, of course, what I'd been requesting all along.
I gave them the written report from Century Lighting with its recommendation for a PQA. He said he will include it with his report and perhaps his superiors will agree to skip the intermediate step of a simple voltage meter, but he did not seem optimistic. I've resigned myself to the fact that procedures and red tape mean it will probably take at least a week -- if we're lucky -- to get someone out with the tools and skills to diagnose our issue.
It was a frustrating day all-around. Both the technicians were stuck on the fact that non-dimmed lights in the building were not flickering, and seemed not to understand that dimmers are sensitive to power issues that undimmed incandescents are not. And if I hear one more person tell me that we have 120 volts, I might scream.
I called CL&P today to get the process started. As I feared, the CSR would not escalate the issue without the problem first being investigated by a lineman. So they sent out a lineman this morning (coincidentally the same one who originally installed the transformer ten years ago), who checked the voltage and, as expected, was unable to find any problem. His favorite phrase seemed to be "You've got 120 volts." Because he had no idea about...well, anything...he called in the meter tech (also coincidentally the same one who installed the meters and wiring from the transformer to the inside panel ten years ago). He checked voltages at the meter and inside, both before and after the stepdown transformers that feed the meter. He, too, then declared that we had 120 volts.
At that point they really wanted to declare it an interior problem and send me back to Strand and our electrician, but I pressed and he said that he could install a recording voltage meter. This meter, unfortunately, will record voltages only at 1-minute intervals, which is going to do nothing to identify our problem -- which occurs when we have 120 volts, anyway! When I expressed my doubts, he admitted that it was unlikely to find anything. He said that after that "if you're still not satisfied" they could escalate it to the power quality assurance group (or something to that effect), which could, in his words, "bring out the big-boy meters." Which is, of course, what I'd been requesting all along.
I gave them the written report from Century Lighting with its recommendation for a PQA. He said he will include it with his report and perhaps his superiors will agree to skip the intermediate step of a simple voltage meter, but he did not seem optimistic. I've resigned myself to the fact that procedures and red tape mean it will probably take at least a week -- if we're lucky -- to get someone out with the tools and skills to diagnose our issue.
It was a frustrating day all-around. Both the technicians were stuck on the fact that non-dimmed lights in the building were not flickering, and seemed not to understand that dimmers are sensitive to power issues that undimmed incandescents are not. And if I hear one more person tell me that we have 120 volts, I might scream.
thelatinist
Member
Yesterday I spoke to the account executive from Northeast Utilities (the parent company of CL&P) in charge of our account. He agreed to send out a team from their Power Quality Analysis Group to investigate. They will install a PQA and measure over a three-day period. Their analysis should include current and voltage as well as harmonics and transients. I'm waiting to hear back with a date and time for the install. Wish us luck!
thelatinist
Member
Oh, yes, I fully intend to.
So I'm coming to this party just a little bit late, and I'm so impressed with the thoroughness with which thelatinist has undertaken his troubleshooting that I almost hesitate to ask these questions. Are there any other electrical systems in your facility that draw power from the same tap as your dimmer rack? If you have three-phase power in the building (I remember you said your dimmer rack has a single-phase supply, but does the building have 3-phase supply?), what is running on the other two phases? I'm no expert by any means, but have seen plenty of instances where other loads introduced enough garbage into the power supply to give sound and lighting systems fits. I'm thinking about things like gym lights (the ballasts do uh-uh-ugly things to voltage waveforms) and motors (is the woodshop anywhere nearby? what about HVAC, sump pumps, etc.?).
Also, thelatinist mentioned current on ground conductors and voltage between neutral and ground. Most people think of ground loops and the problems they cause in connection with audio systems (the dreaded 60-cycle hum), but grounding issues can affect dimmers, too. The largest facility I worked at had two 200-amp, one 400-amp, and one 600-amp three-phase disconnects in the stage electrical cage which were tied to a grounding grid buried about 20 feet below the surface to ensure a clean, single-point ground for all systems running off those discos. If thelatinist's facility has a flaky ground connection or multiple ground connections, there could be lots of junk introduced into the power supply.
Also, thelatinist mentioned current on ground conductors and voltage between neutral and ground. Most people think of ground loops and the problems they cause in connection with audio systems (the dreaded 60-cycle hum), but grounding issues can affect dimmers, too. The largest facility I worked at had two 200-amp, one 400-amp, and one 600-amp three-phase disconnects in the stage electrical cage which were tied to a grounding grid buried about 20 feet below the surface to ensure a clean, single-point ground for all systems running off those discos. If thelatinist's facility has a flaky ground connection or multiple ground connections, there could be lots of junk introduced into the power supply.
thelatinist
Member
Not at all. Please, ask away.
We do not have three-phase power in the building as it is not available in our rural area. We have two three-hundred amp legs with two load conductors for each leg, both legs in the same phase, entering the building from our transformer. These legs pass through a safety switch and into a cabinet where they are fed into two 400-amp 80:1 step-down transformers which feed the meter outside the building. The ground and neutral are bonded in this cabinet, which is sealed by CL&P. There are three conductors leading from each of these step-down transformers; two conductors from each leg feed into a two-leg 400-amp circuit breaker which feeds our dimmer rack; the remaining conductor from each leg feeds a total of five sub-panels that power the rest of the building.
And, yes, there is lots of junk on those other panels: gym lights, heating and ventilation blowers, fluorescent lights, sump pump, sewer pumps, etc. We have considered the possibility that one of these systems is causing the problem, but we have not noticed any difference in the flickering whether gym lights, heat, etc. are on or based on time of day, occupancy, or whether any other systems are turned on or off; and nothing has changed in the building since the lights were working in the fall. But this has been a nagging doubt in my mind. I have wanted to turn off everything else in the building and see if I can replicate the problem, but that would mean disabling the alarm, etc. I might go in tomorrow and give it a try.
We did investigate this on the advice of the folks from Strand. Our electrician checked and all the grounds are bonded to the neutral at the main panel (as described above). He also checked across grounds on different circuits throughout the building, including outlets fed by the sub-panels I mentioned above and our dimmer rack; as far as we could tell, everything appears to be in the same ground plane.
Kb
Once again, I admire your dogged determination to get to the bottom of your problem! One challenge the standard US electrical distribution system presents to any users that require a rock-solid ground connection is that the bonded neutral and ground means that your closest actual protective earth connection might be some distance away. I've even worked on outdoor gigs where we had to pound an eight-foot rod into the ground to get a clean enough ground. Unfortunately, adding a direct protective earth connection to an existing building can be expensive.
That brings up another question - what kind of neighbors do you have? Any businesses or industries that might be running equipment that might be mucking up your power? I'm glad that utility I'd going to monitor your supply and am interested to see what they find.
thelatinist
Member
Yeah, at this point we've all got our fingers crossed hoping the PQA reveals something. The request has been made by our account exec; now we just have to wait to hear back from the PQA Group...which apparently works on its own timeline. I'll keep you guys posted.
By the way, I just want to say again that the folks from Strand and Century Lighting Service have been really helpful throughout this process. They've really gone out of their way to help us out while minimizing our expenses. An even CL&P has been easier to work with than I had feared.
By the way, I just want to say again that the folks from Strand and Century Lighting Service have been really helpful throughout this process. They've really gone out of their way to help us out while minimizing our expenses. An even CL&P has been easier to work with than I had feared.
thelatinist
Member
RileyChilds
Member
We had this issue before,
What lights do you have?
