I've had that problem in the past on laptops using those really sort dongles. All I can say is try try again. I had to take a pair of pliers to one as I couldn't get a good grip but make sure you pull straight out or you may damage the port.
Moving this out of FAQs and Guides as it's neither.
It probably not a a bluetooth. Typically if it's specific for the mouse and not a universal Bluetooth adapter it called a USB receiver that won't have to be paired like bluetooth. The advantage of a Bluetooth mouse is it doesn't need a USB port and it free it up if your PC has bluetooth build in.