Rank Group is to pay a £5m penalty after regulators identified a series of compliance issues within three of its casino operators, including duty of care to customers.