Perform the following in the Azure Console:

1. Go to Azure Active Directory
2. Go to Users and group
3. Go to User settings
4. Set Restrict access to Azure AD administration portal to Yes

References:

1. https://docs.microsoft.com/en-us/azure/active-directory/active-directory-assign-admin-roles-azure-portal