You can restrict the linked records options by adding a "conditional/dynamic filter" to your field. When a filter is selected on the form, it will limit the linked records available in the list. The dynamic filters for the form can only use the linked records field and lookup fields. To use this functionality, please first enable Dynamic filtering in the Filter section of your linked record field's settings.
To elaborate on this functionality, you can watch this video.
You can hide this filter from the field and use prefill to fill it up. You just need to set the Filter Field Visibility to the Hide filter fields (prefill using URL) option.
For each filter field you have the option to determine if filling the filter field should be required to add or remove records to/from the linked record field itself using these options, which can both be accessed by clicking on the filter field in the Filter Fields setting:
Allow adding linked records if this conditional filter is empty
Allow removing linked records if this conditional filter is empty