=
and LIKE
comparisons in WHERE clauses apply a collation (not just a character set) to determine this kind of equality. This statement returns zero for the first two collations and one for the second two.
SELECT 'æ' = 'ae' COLLATE utf8mb4_unicode_ci, -- 0
'æ' = 'ae' COLLATE utf8mb4_general_ci, -- 0
'æ' = 'ae' COLLATE utf8mb4_unicode_520_ci, -- 1
'æ' = 'ae' COLLATE utf8mb4_german2_ci -- 1
It seems likely your default collation is one of the last two or some other collation that handles that equality test the way you don't want it.
You can see your connection's collation setting with this statement. I suspect it is utf8mb4_unicode_520_ci
.
SELECT @@collation_connection;
Be sure to define the collation for your columns with one you do want, and set your connection collation to the same thing. utf8mb4_unicode_ci
is suitable. Try this.
SET collation_connection = 'utf8mb4_unicode_ci';
SELECT 'æ' = 'ae' -- 0;
It's hard to give more specific advice without understanding your linguistic requirements better.
More info here: Difference between utf8mb4_unicode_ci and utf8mb4_unicode_520_ci collations in MariaDB/MySQL?