Adding FK to a newly attached partition slow

We have a partitioned table from which we attach and detach partitions for archiving purposes. When attempting to add a foreign key on the partitioned table, the process is taking a long time (up to 20 minutes) which is causing many problems. We have discussed using a NOT VALID option, but unfortunately this is not an option for partition tables, so just wanted to see if there were any options to add a foreign key to a partition