If more than doubling the calculated frame rate doesn't solve the sync issue (Nyquist frequency), I would try doing all the timing calculations inside the hardware and use the software only for basic commands like scene changes.
A quick fix that doesn't involve starting all over would be to connect an
Arduino or Teensy via USB, then use a smoothing algorithm before sending
DMX. Or if the program won't recognize this hardware, then just receive
DMX, smooth, then
send new
DMX.
Neither of these methods will solve the 8-bit steps issue in the dimming hardware but can be solved with hardware that smooths or 16-bit mode
DMX, as discussed previously. You can try putting capacitors in parallel with the
LED's, but that could potentially damage the hardware if the caps are too large or if the ESR is very low.
Edit: Actually, with current-limiting hardware, a large cap with low ESR and long calculated lifetime might work really well, but I don't want to promise anything and then you let the
smoke out while trying.