This isn't new, and you should name your plugins better.
The update system uses both the directory name of the plugin and the name given in the plugin to look for matches. Also, it uses the "Plugin URI:" field in the plugin header, so make sure that you have set that to a unique value on a domain you control.
For naming, if you have a "pay" plugin that is not in the directory, then you should name it using your brand name, and not in a generic way. "Contact Form Pro" is extremely generic. But "BrandName - Contact Form Pro" is not. Brand things. Then you will have less conflicts.
Additionally, if you have a non-free plugin, then you should add code to it to prevent it from checking for updates in any case. There's plenty of examples on the web as to how to make a plugin prevent itself from updating to one from the directory, or examples of how to have it check somewhere else (like your site) for updates. Use those examples. Plugins can do anything they like, including control how they update.