Depends if we take into consideration Legends Canon, it might be more of a fair fight. Or if we have Sith involved, as some Sith are genuinely terrifying. A key example would be Darth Nihilus, He was known to consume entire worlds by simply existing as he consumed the life-force of all he desired to satisfy his hunger. And if Viltrumites are capable of being harmed by the force, their are plenty of force abilities that can kill them. Like Force Crush, which crushes a person’s body, and Force Expand, which literally makes the victim explode from the inside via the expansion of their bones.

If we go for a more modern approach. Legends Palpatine was a fucking Menace, as he knew the technique of Force Storm. It would create storms capable of destroying planets and even ripping holes in the Space-Time Continuam. It could also transport and displace those caught in it to far out reaches of the galaxy.

So honestly, Legends Star Wars can definitely hold its own against the Viltrumite Empire.