Perform the following in the Azure Console:

1. Go to Azure Active Directory
2. Go to Users and group
3. Go to User settings
4. Set Guests can invite to No

References:

1. https://docs.microsoft.com/en-us/azure/active-directory/active-directory-b2b-delegate-invitations