Is there a way to configure the email notifications using NTLM?

Yes!

In a manual installation, you’ll need to install ruby-nltm, so ideally add this to your Gemfile.local and run bundle install.

In your mail configuration, you can then use authentication: :ntlm as a valid method and it should just work.

The packaged installation currently exposes a fixed set of authentication methods, and installing third party gems is not that trivial. We did however just recently add a method to add custom gems and plugins to your packaged installation: https://github.com/opf/openproject/blob/dev/doc/operation_guides/packager/installation-guide.md#adding-custom-plugins-to-the-installation

You would not be able to use the wizard to configure that authentication mode, though.

Best,
Oliver