If I understand you correctly, you are leaving CMS and back office Users as-is, and you want to allow Member login on the public facing side of your site with a custom provider?
Did you set the default provider to your new membership provider?
<membership defaultProvider="mynewMembershipProvider">
...
</membership>
The default provider in the membership section should always point to the provider being used for public-facing Members - the backoffice interally asks for the UsersMembershipProvider
every time it needs to authenticate, so changing the default will not affect it at all.
It is also possible depending on your set up (I haven't tried this) that if you did not include a <clear/>
directive then the original Umbraco member provider is causing problems.
Here is a full working <membership>
section using a custom BCrypt hashing provider from an Umbraco 7 install of mine:
<membership defaultProvider="BCryptMembershipProvider" userIsOnlineTimeWindow="15">
<providers>
<clear />
<add
name="BCryptMembershipProvider"
type="cFront.Web.Security.BCryptMembershipProvider"
connectionStringName="umbracoDbDSN"
requiresUniqueEmail="0"
/>
<add name="UsersMembershipProvider" type="umbraco.providers.UsersMembershipProvider" enablePasswordRetrieval="false" enablePasswordReset="false" requiresQuestionAndAnswer="false" passwordFormat="Hashed" />
</providers>
</membership>
<roleManager enabled="true" defaultProvider="BCryptRoleProvider">
<providers>
<clear />
<add name="BCryptRoleProvider" type="cFront.Web.Security.BCryptRoleProvider"
connectionStringName="umbracoDbDSN"
availableRoles="SuperUser,Administrator,Manager,User"
/>
</providers>
</roleManager>