Potter should certainly not refuse the chance to go to Chelsea. If he is successful it will catapult him into the top bracket of European managers and if he flops early (which would be unlikely considering he’s a very good manager) then he’ll earn a stack and still has his Brighton job reputation intact.
Any manager with any ambition would jump at the chance.