If you want to switch from one selected namespace to another, you cannot remove the old one with backspace, you have to add the new namespace (which even doesnt show up pinned at the top), but click out, click in again and deselect the old namespace.
So the current user experience is really shitty tbh, sorry.
Could you provide something like selectize? Easy to remove single namespaces:
