Narrow it down after that. ie. if it's just one of the windows which causes the dimming effect, or all of them (they may all utilise a single grounding point).
Really, so long as your battery is charged, you should be able to run headlights and use your power windows at the same time without dimming anyhow (with the ignition off). Unlikely it's the alternator but doesn't hurt to check if you have a multimeter handy!