Customer Address Design Probelm

I'm designing a database for E-commercial. So the Customer have many address also 1 address also has many customers owned so I have another CustomerAddress table (many-to-many relation). Here detail : 1. Customer: - CustomerID (PK) - Name - DateofBirth 2. AddressNote : - AddressNoteID (PK) - AddressID (FK) (has a relationship with the city table so I'm not mentioning it here) 3. CustomerAddress : - CustomerID (PK) - AddressNoteID (PK) - PhoneNumber (The phone number here indicates who will be called upon delivery at that address) My problem is that I want 1 client will have multiple addresses but will have 1 single default address. So I thinking to add one more column **IsDefault (bool)** in CustomerAddress table. Later, I feel wrong because that's still allowed customer have two different addresses and both set Default address. I believe in application level validation can handle this but I want database constraints. Is there anyway to design this constraints. I'm just beginner so thank for any instructions.