The spigot bearing will only make noise when the clutch is disengaged. otherwise the gearbox input shaft and crank where the spigot is is practically a solid piece (ie when the clutch is out), then the spigot bearing is not rotating and wont make any noise.
It could be the thrust bearing, but I would expect it to make noise in every gear (??)
Edit: Always change the stupid spigot bearing when you remove the clutch... such a pain in the butt to remove the stupid gearbox again when you find it makes an awful noise