If you're about to try using a tealight oil warmer for the first time, be sure you clean up the warmer between each batch of oil. Old oil can begin to smell odd and collect dust if you leave any of it sitting in the warmer, and if you let the oil dry in the warmer, it becomes difficult to remove. Cleaning an oil warmer varies from brand to brand, but in general, you don't need any fancy materials.
Always let the burner and any leftover oil cool first. You don't want to leave the oil sitting for so long that it starts to evaporate without heat, of course. But don't grab the burner 5 minutes after you've blown out the candle.
Also, when you wash the burner, don't use boiling or very hot water. Not only will you burn your hands, but you'll increase the chances of the burner cracking. While the burner is built to handle heat, putting it through a warm-up with the burning oil, a cool-down right after, and then another rapid rise in temperature to near water's boiling point can stress the material too much.
Avoid using very abrasive materials so you don't leave scratches on the burner. Oil can seep into the scratches and leave residue that becomes almost impossible to get out. If you're really having trouble removing some of the oil, a mildly abrasive sponge or a little baking soda paste might work, but leave the steel wool in its box under your sink.
Some manufacturers claim their warmer oil is meant to evaporate and not leave residue. For these burners, place a clean piece of toweling in the oil bowl to soak up any wet residue you see. Remove the toweling and, with a clean piece of toweling, gently wipe the oil bowl out to ensure the oil is completely gone.
If you still smell the oil after wiping those out, or your burner oil is not supposed to evaporate away completely, try washing the container. Use warm water and dish soap to clean the oil burner a couple of times. Let that dry and see if you can feel oily residue or smell the old oil (remember to move the oil-soaked toweling out of the room so that you don't mistake the scent from that with leftover scent in the bowl).
If you can smell or feel the oil, wash the burner again. Do that a few more times -- it really can take a bit of time to get everything out. If you're still having trouble after that, then look at using something like a mild baking soda paste. Test it first, though -- ensure it won't leave a white residue.