Is it only with your power windows that it dims or do other devices cause your headlights to dim also?
If it was the alternator your battery should have gone flat a few times. A power window motor shouldn't draw enough power to make the headlights dim anyway.
Check the grouding of the power window motors and computer (probably inside the door).
Check your battery terminals are nice and clean.
If you want to check your alternator is working, get a multimeter and check the voltage with the engine running, should be around 13.8 volts ideally. Best to do this with your headlights or even high beam and a few other things running like interior fan.