Sort your data for each country individually

Code
<?php

$array = array('a', 'g', 'A', 'ä', 'Ä', 'k', 'z');

sort($array);

echo 'sort = ' . implode(',', $array) . PHP_EOL;

$collator = new Collator('de_DE');
$collator->setAttribute(Collator::CASE_FIRST, Collator::LOWER_FIRST);
$collator->sort($array);

echo 'Collator::sort = ' . implode(',', $array) . PHP_EOL;
Result
sort = A,a,g,k,z,Ä,ä
Collator::sort = a,A,ä,Ä,g,k,z
Used Versions
PHP 8.3, Laminas MVC 3.2, Symfony 5.2, Laravel 8.28, PHPUnit 9.5, Doctrine ORM 2.8