By default, records cannot be unlinked when the record finder is hidden. You can toggle this behavior on or off here:
Record Finder Options > Select Hide record finder button
Allow users to unlink existing records
In the linked records field setting, there's another option that you can use to achieve the same result. Turn on Read-only.