Setting up an email server on a RaspberryPI (Postfix+Dovecot+MariaDB+Roundcube)

There’s a few things in this journey that you should be aware of when running your own mailserver before we begin.

  1. Invest in a static public IP
  2. Don’t open mail relay
  3. Leverage proper DNS records to help mitigate your email from being marked spam
  4. Leverage something to scrub your email
  5. Don’t open mail relay
  6. Don’t open mail relay
  7. Verify your domain or IP address hasn’t been placed on a blacklist from a previous owner

In my career in doing IT, handling email is one of the most tidiest tasks to setup/maintain due to so many moving pieces; many of which may be out of your control. Dealing with spam, blacklisting, having emails non-deliverable for several reasons, handling dns records, certificates, etc…. it’s sometimes worth paying a few extra bucks to have someone else host your email and have peace of mind the message will be delivered. That being said, if you have the extra time on your hands and like the challenge of solving problems, here’s a quick way to get started.

Preamble

This guide took me several hours to compile through trial and error. If you have any thoughts, notice any errors/typos, or have ideas on how to further secure/optimize, please leave feedback below to further improve this guide. Thank you and good luck on the deployment of your mail server!

Assumptions

  • You have previously followed my guide on building a LEMP stack
  • You are running Ubuntu or Debian as per the above guide (you can still follow this guide, you may have to slightly change which commands you use for your distribution — configuration should remain the same though)

DNS

Let’s first start at getting your DNS records configured properly. This guide will talk about configuring MX, SPF, and PTR records. We won’t be covering Domain Keys in this article, maybe in a separate article if someone donates to my paypal on the right side of the website 😉

MX Record

Via your nameservers, add a new mx record for your domain name. Here’s a list of tutorials for some of the major domain registrars:

SPF Record

Contrary to many websites that say you need to create a “SPF” record type, the SPF record type was never ratified by RFC standards. In this case, the proper way to create a SPF record is via a TXT record with the SPF value (as per RFC 7208).

You can leverage my SPF generator to create a new TXT record in the root of your domain.

PTR Record

To help decrease the odds of your emails being labeled as spam, I’d recommend creating a PTR record that will resolve your IP address to a DNS name (we call this a reverse lookup). For example, if my mail server’s domain name was mail.mydomain.com and it resolved to 123.123.123.123, I would create a PTR record for 123.123.123.123 that points to mail.mydomain.com.

In many cases, you will need to either work with your ISP (Internet Service Provider) or domain registrar if you own your own IP block to make changes to the record for your IP address block.

When you are ready, you can leverage the nslookup command on Windows to validate the name from the IP address.

nslookup 123.123.123.123

Or on linux you can leverage the host command to verify the reverse lookup as well:

host 123.123.123.123

Get the OS ready

Download the latest packages and actually perform any updates.

sudo sh -c 'apt update && apt upgrade'

Prepare MariaDB for virtual users/aliases

One of the primary reasons we need to configure a database is it is what will contain the information about all of our users and their corresponding email addresses (aliases). To do so, we need to create 3 new tables inside of a new database.

Login to the database

sudo mariadb -u root -p

Create the database, database user, and tables

Create a new database for our users (in this case, I’m calling the database mailserver). Note: This command must be run in the context of mariadb, this is not a bash command.

create database mailserver;

Create a new user called mailuser, grant them access to the entire database, require the user to only create connections from 127.0.0.1 (localhost), and specify a password for the user.

GRANT SELECT ON mailserver.* TO 'mailuser'@'127.0.0.1' IDENTIFIED BY 'mysupersecretpassword';

Execute the following command to apply the changes

FLUSH PRIVILEGES;

Create a table for each of the domain names we will leverage for our email addresses.

CREATE TABLE `mailserver`.`virtual_domains` (
  `id` int(11) NOT NULL auto_increment,
  `name` varchar(50) NOT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Create a table that will hold each of the users that will need mailboxes.

CREATE TABLE `mailserver`.`virtual_users` (
  `id` int(11) NOT NULL auto_increment,
  `domain_id` int(11) NOT NULL,
  `password` varchar(106) NOT NULL,
  `email` varchar(100) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `email` (`email`),
  FOREIGN KEY (domain_id) REFERENCES virtual_domains(id) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Create a table that will hold aliases (additional email addresses) for a particular user.

CREATE TABLE `mailserver`.`virtual_aliases` (
  `id` int(11) NOT NULL auto_increment,
  `domain_id` int(11) NOT NULL,
  `source` varchar(100) NOT NULL,
  `destination` varchar(100) NOT NULL,
  PRIMARY KEY (`id`),
  FOREIGN KEY (domain_id) REFERENCES virtual_domains(id) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Insert a new user into the database

First, we need to add our first domain name into the domains table

INSERT INTO `mailserver`.`virtual_domains`
  (`name`)
VALUES
  ('mydomain.com');

Second, we need to create the user. Replace mysupersecretpassword with your password.

INSERT INTO `mailserver`.`virtual_users`
  (`domain_id`, `password` , `email`)
VALUES
  ('1', ENCRYPT('mysupersecretpassword', CONCAT('$6$', SUBSTRING(SHA(RAND()), -16))), '[email protected]');

Third, we can optionally specify an alias (secondary email address) for the user.

INSERT INTO `mailserver`.`virtual_aliases`
  (`domain_id`, `source`, `destination`)
VALUES
  ('1', '[email protected]', '[email protected]');

Type exit once you are done to leave the context of MariaDB.

Install Packages for Postfix and Dovecot

Postfix is what we call a Mail Transport Agent (MTA) and is responsible for actually sending/receive the messages from the internet. Later, we will talk about Dovecot which will be our MDA (Mail Delivery Agent) (what actually interacts with the mailbox).

The following command will install postfix, dovecot, and pull the packages to interact with MySQL. Although these are labeled MySQL, they should interact fine with MariaDB.

sudo apt-get install postfix postfix-mysql dovecot-core dovecot-imapd dovecot-lmtpd dovecot-mysql

During the installation of Postfix, you will be prompted to configure the connection type to the mail server. In this case, select Internet for the mail configuration.

On the second installation prompt, it will ask for the domain name used in receiving email. In this prompt, specify one of the domain names you will be using for your users. For example, if your email addresses are going to be [email protected] you would specify mydomain.com for this prompt. Don’t worry if you have multiple email addresses, we will cover that later on.

Configure Postfix to leverage MariaDB

First, let’s create a backup of the Postfix configuration, so we have a baseline to refer back to.

sudo cp /etc/postfix/main.cf /etc/postfix/main.cf.bak

Copy the following configuration and replace the domain name example.com with yours. Credit to linode for sharing their configuration as it not only defines integration into a database, but also hardens the Postfix deployment.

# See /usr/share/postfix/main.cf.dist for a commented, more complete version

# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
#myorigin = /etc/mailname

smtpd_banner = $myhostname ESMTP $mail_name
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = no

# TLS parameters
smtpd_tls_cert_file=/etc/letsencrypt/live/mydomain.com/fullchain.pem
smtpd_tls_key_file=/etc/letsencrypt/live/mydomain.com/privkey.pem
smtpd_use_tls=yes
smtpd_tls_auth_only = yes
smtp_tls_security_level = may
smtpd_tls_security_level = may
smtpd_sasl_security_options = noanonymous, noplaintext
smtpd_sasl_tls_security_options = noanonymous

# Authentication
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

# Restrictions
smtpd_helo_restrictions =
        permit_mynetworks,
        permit_sasl_authenticated,
        reject_invalid_helo_hostname,
        reject_non_fqdn_helo_hostname
smtpd_recipient_restrictions =
        permit_mynetworks,
        permit_sasl_authenticated,
        reject_non_fqdn_recipient,
        reject_unknown_recipient_domain,
        reject_unlisted_recipient,
        reject_unauth_destination
smtpd_sender_restrictions =
        permit_mynetworks,
        permit_sasl_authenticated,
        reject_non_fqdn_sender,
        reject_unknown_sender_domain
smtpd_relay_restrictions =
        permit_mynetworks,
        permit_sasl_authenticated,
        defer_unauth_destination

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

myhostname = example.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydomain = mydomain.com
myorigin = $mydomain
mydestination = localhost
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
inet_protocols = all

# Handing off local delivery to Dovecot's LMTP, and telling it where to store mail
virtual_transport = lmtp:unix:private/dovecot-lmtp

# Virtual domains, users, and aliases
virtual_mailbox_domains = mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf
virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-mailbox-users.cf
virtual_alias_maps = mysql:/etc/postfix/mysql-virtual-mailbox-aliases.cf,
        mysql:/etc/postfix/mysql-virtual-mailbox-users.cf

# Even more Restrictions and MTA params
disable_vrfy_command = yes
strict_rfc821_envelopes = yes
#smtpd_etrn_restrictions = reject
#smtpd_reject_unlisted_sender = yes
#smtpd_reject_unlisted_recipient = yes
smtpd_delay_reject = yes
smtpd_helo_required = yes
smtp_always_send_ehlo = yes
#smtpd_hard_error_limit = 1
smtpd_timeout = 30s
smtp_helo_timeout = 15s
smtp_rcpt_timeout = 15s
smtpd_recipient_limit = 40
minimal_backoff_time = 180s
maximal_backoff_time = 3h

# Reply Rejection Codes
invalid_hostname_reject_code = 550
non_fqdn_reject_code = 550
unknown_address_reject_code = 550
unknown_client_reject_code = 550
unknown_hostname_reject_code = 550
unverified_recipient_reject_code = 550
unverified_sender_reject_code = 550

Next, we need to create the mappings of domain names, users, and aliases. In the same directory as the main.cf (/etc/postfix) we need to first create a file that will tell postfix how to lookup what domain names exist. You can open the documents with your favorite text editor; I use vi since it’s universally installed.

sudo vi /etc/postfix/mysql-virtual-mailbox-domains.cf

Press i to get vi into insert mode and paste the following, replacing the password with the mailuser we specified earlier in this tutorial.

user = mailuser
password = mysupersecretpassword
hosts = 127.0.0.1
dbname = mailserver
query = SELECT 1 FROM virtual_domains WHERE name='%s'

Press : and then type wq and press enter to write the changes to the file and quit in vi.

Next, we will create another file that is used to lookup each user’s mailbox.

sudo vi /etc/postfix/mysql-virtual-mailbox-users.cf

Press i to get vi into insert mode and paste the following, replacing the password with the mailuser we specified earlier in this tutorial.

user = mailuser
password = mysupersecretpassword
hosts = 127.0.0.1
dbname = mailserver
query = SELECT email FROM virtual_users WHERE email='%s'

Press : and then type wq and press enter to write the changes to the file and quit in vi.

Last, we will create another file that is used to map an alias to a user’s mailbox.

sudo vi /etc/postfix/mysql-virtual-mailbox-aliases.cf

Press i to get vi into insert mode and paste the following, replacing the password with the mailuser we specified earlier in this tutorial.

user = mailuser
password = mysupersecretpassword
hosts = 127.0.0.1
dbname = mailserver
query = SELECT destination FROM virtual_aliases WHERE source='%s'

Press : and then type wq and press enter to write the changes to the file and quit in vi.

Restart the Postfix service for the changes to take effect

sudo service postfix restart

Next, to enable port 587 and 465 to connect securely with email clients, we need to modify /etc/postfix/master.cf. First, let’s create a backup of the master.cf file.

sudo cp /etc/postfix/master.cf /etc/postfix/master.cf.bak

Next, we need to modify the master.cf file. Modify the document (mostly uncomment many of the lines) to look similar to the code below.

sudo vi /etc/postfix/mysql-virtual-mailbox-aliases.cf
submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_sasl_type=dovecot
  -o smtpd_sasl_path=private/auth
  -o smtpd_tls_auth_only=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_recipient_restrictions=
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING
smtps     inet  n       -       y       -       -       smtpd
  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_sasl_type=dovecot
  -o smtpd_sasl_path=private/auth
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_recipient_restrictions=
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

Press : and then type wq and press enter to write the changes to the file and quit in vi.

Restart the Postfix service for the changes to take effect

sudo service postfix restart

Configure Dovecot

Now that we have our MTA configured, we now need to configure our MDA. You can think of Postfix as a shipping center and Dovecot as the courier, who interfaces directly with your mailbox. Roundcube will be our MUA (mail user agent) that interfaces with Dovecot to display your mail. The goal for this section is to ensure Dovecot requires SSL.

First, we’ll create backups of each of the Dovecot configuration files

sudo cp /etc/dovecot/dovecot.conf /etc/dovecot/dovecot.conf.orig
sudo cp /etc/dovecot/conf.d/10-mail.conf /etc/dovecot/conf.d/10-mail.conf.orig
sudo cp /etc/dovecot/conf.d/10-auth.conf /etc/dovecot/conf.d/10-auth.conf.orig
sudo cp /etc/dovecot/dovecot-sql.conf.ext /etc/dovecot/dovecot-sql.conf.ext.orig
sudo cp /etc/dovecot/conf.d/10-master.conf /etc/dovecot/conf.d/10-master.conf.orig
sudo cp /etc/dovecot/conf.d/10-ssl.conf /etc/dovecot/conf.d/10-ssl.conf.orig

Execute the following command to enable support for imap and lmtp (pop3 can be added, but ensure you install the dovecot-pop3d package).

sudo sed -i '/^\!include_try \/usr\/share\/dovecot\/protocols.d\/\*.protocol/a protocols=imap lmtp' /etc/dovecot/dovecot.conf

Next, we need to edit /etc/dovecot/conf.d/10-mail.conf to define where mailboxes are stored. Execute the following commands:

sudo sed -i 's/mail_location = mbox.*/mail_location = maildir:\/var\/mail\/vhosts\/%d\/%n\//g' /etc/dovecot/conf.d/10-mail.conf
sudo sed -i 's/^#mail_privileged_group = mail/mail_privileged_group = mail/g' /etc/dovecot/conf.d/10-mail.conf

Next, we need to make directories for each of your domain names. Execute the following command for each of your domain names.

sudo mkdir -p /var/mail/vhosts/example.com

Now we need to create a user and group called vmail, assigned with an id of 5000, and set the directory with the owner of vmail

sudo groupadd -g 5000 vmail
sudo useradd -g vmail -u 5000 vmail -d /var/mail
sudo chown -R vmail:vmail /var/mail

Next we need to edit the user authentication file (/etc/dovecot/conf.d/10-auth.conf) to tell Dovecat to leverage MariaDB for our users. Execute the following commands:

sudo sed -i 's/^#disable_plaintext_auth = yes/disable_plaintext_auth = yes/g' /etc/dovecot/conf.d/10-auth.conf
sudo sed -i 's/^#auth_mechanisms = plain login/auth_mechanisms = plain login/g' /etc/dovecot/conf.d/10-auth.conf
sudo sed -i 's/^!include auth-system.conf.ext/#!include auth-system.conf.ext/g' /etc/dovecot/conf.d/10-auth.conf
sudo sed -i 's/^#!include auth-sql.conf.ext/!include auth-sql.conf.ext/g' /etc/dovecot/conf.d/10-auth.conf

Once we have the authentication file configured, we need to update the sql driver (/etc/dovecot/conf.d/auth-sql.conf.ext) to point to our mailboxes. You will need to uncomment the passdb section and uncomment the userdb driver that is static.

sudo vi /etc/dovecot/conf.d/auth-sql.conf.ext

Press i to get vi into insert mode and paste the following configuration

# Authentication for SQL users. Included from 10-auth.conf.
#
# <doc/wiki/AuthDatabase.SQL.txt>

passdb {
  driver = sql

  # Path for SQL configuration file, see example-config/dovecot-sql.conf.ext
  args = /etc/dovecot/dovecot-sql.conf.ext
}

# "prefetch" user database means that the passdb already provided the
# needed information and there's no need to do a separate userdb lookup.
# <doc/wiki/UserDatabase.Prefetch.txt>
#userdb {
#  driver = prefetch
#}

#userdb {
#  driver = sql
#  args = /etc/dovecot/dovecot-sql.conf.ext
#}

# If you don't have any user-specific settings, you can avoid the user_query
# by using userdb static instead of userdb sql, for example:
# <doc/wiki/UserDatabase.Static.txt>
userdb {
  driver = static
  args = uid=vmail gid=vmail home=/var/mail/vhosts/%d/%n
}

Press : and then type wq and press enter to write the changes to the file and quit in vi.

The final Dovecot file we need to modify will set our database settings (/etc/dovecot/dovecot-sql.conf.ext). Execute the following commands to uncomment the correct settings. Note: be sure to replace the password with the database password we configured earlier.

sudo sed -i 's/^#driver = /driver = mysql/g' /etc/dovecot/dovecot-sql.conf.ext
sudo sed -i 's/^#connect =/connect = host=127.0.0.1 dbname=mailserver user=mailuser password=mysupersecretpassword/g' /etc/dovecot/dovecot-sql.conf.ext
sudo sed -i 's/^#default_pass_scheme = MD5/default_pass_scheme = SHA512-CRYPT/g' /etc/dovecot/dovecot-sql.conf.ext
sudo sed -i '/^#password_query = \\/i password_query = SELECT email as user, password FROM virtual_users WHERE email=\x27%u\x27;' /etc/dovecot/dovecot-sql.conf.ext

After making the changes to the dovecot-sql.conf.ext file, next we need to change the owner and the group of the dovecot folder to the vmail user:

sudo chown -R vmail:dovecot /etc/dovecot
sudo chmod -R o-rwx /etc/dovecot 

Next, we need to disable the unencrypted versions of IMAP and SMTP.

sudo vi /etc/dovecot/conf.d/10-master.conf

We need to edit the /etc/dovecot/conf.d/10-master.conf file and set ports to 0 to disable non-encrypted imap/pop3. Find service imap-login { and make it look like the following.

service imap-login {
  inet_listener imap {
    port = 0
  }
  inet_listener imaps {
    port = 993
    ssl = yes
  }

  # Number of connections to handle before starting a new process. Typically
  # the only useful values are 0 (unlimited) or 1. 1 is more secure, but 0
  # is faster. <doc/wiki/LoginProcess.txt>
  #service_count = 1

  # Number of processes to always keep waiting for more connections.
  #process_min_avail = 0

  # If you set service_count=0, you probably need to grow this.
  #vsz_limit = $default_vsz_limit
}

service pop3-login {
  inet_listener pop3 {
    port = 0
  }
  inet_listener pop3s {
    port = 995
    ssl = yes
  }
}

In the same file, find service lmtp { and replace the whole block down to the third } with the following:

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    mode = 0600
    user = postfix
    group = postfix
  }

  # Create inet listener only if you can't use the above UNIX socket
  #inet_listener lmtp {
    # Avoid making LMTP visible for the entire internet
    #address =
    #port =
  #}
}

In the same file, find service auth { and replace the whole block down to the third } with the following:

service auth {
  # auth_socket_path points to this userdb socket by default. It's typically
  # used by dovecot-lda, doveadm, possibly imap process, etc. Users that have
  # full permissions to this socket are able to get a list of all usernames and
  # get the results of everyone's userdb lookups.
  #
  # The default 0666 mode allows anyone to connect to the socket, but the
  # userdb lookups will succeed only if the userdb returns an "uid" field that
  # matches the caller process's UID. Also if caller's uid or gid matches the
  # socket's uid or gid the lookup succeeds. Anything else causes a failure.
  #
  # To give the caller full permissions to lookup all users, set the mode to
  # something else than 0666 and Dovecot lets the kernel enforce the
  # permissions (e.g. 0777 allows everyone full permissions).
  unix_listener auth-userdb {
    mode = 0600
    user = vmail
    #group =
  }

  # Postfix smtp-auth
  unix_listener /var/spool/postfix/private/auth {
    mode = 0600
    user = postfix
    group = postfix
  }

  # Auth process is run as this user.
  user = dovecot
}

In the same file, find service auth-worker { and replace the whole block down to the } with the following:

service auth-worker {
  # Auth worker process is run as root by default, so that it can access
  # /etc/shadow. If this isn't necessary, the user should be changed to
  # $default_internal_user.
  user = vmail
}

Press : and then type wq and press enter to write the changes to the file and quit in vi.

Last, we need to tell dovecot where our SSL certificate is for encryption. We will modify the /etc/dovecot/conf.d/10-ssl.conf file. Make sure to update the directory with the correct path for your certificates.

Execute the following commands, replacing

sudo sed -i 's/^ssl = yes/ssl = required/g' f
sudo sed -i 's/^ssl_cert = .*/ssl_cert = <\/etc\/letsencrypt\/live\/mydomain.com\/fullchain.pem/g' /etc/dovecot/conf.d/10-ssl.conf
sudo sed -i 's/^ssl_key = .*/ssl_key = <\/etc\/letsencrypt\/live\/mydomain.com\/privkey.pem/g' /etc/dovecot/conf.d/10-ssl.conf

Last, restart devocot to enable all of our changes.

sudo service dovecot restart

Configure Roundcube

Install dependencies for Roundcube

Roundcube requires several PHP PEAR modules. To install the bare minimum featureset, execute the following command:

sudo apt-get install php7.3-mbstring php-pear php-net-idna2 php-net-smtp  php-mail-mime

Create a database for Roundcube

First, we need to create a new database and user for Roundcube. We can do this by logging into MariaDB and executing the create and grant commands.

sudo mariadb -u myusername -p
CREATE DATABASE roundcubemail CHARACTER SET utf8 COLLATE utf8_general_ci;
GRANT ALL PRIVILEGES ON roundcubemail.* TO [email protected] IDENTIFIED BY 'myreallyreallysecretpassword';
FLUSH PRIVILEGES;
exit

Request SSL Certificates for Roundcube

We will want to ensure all traffic to and from the client is encrypted in transit when trying to access Roundcube. To do this, I leverage Let’s Encrypt, which will allow you to request a free SSL certificate. If you have your own SSL certificate, go ahead and copy it to a location on the server so we can reference it later.

sudo apt-get install certbot
sudo certbot certonly --authenticator standalone webmail.mydomain.com --pre-hook "service nginx stop" --post-hook "service nginx start"

Create a directory for Roundcube

We will need to create a directory that will hold Roundcube’s files to serve to the web. Let’s create a new directory to serve these files and limit permissions to www-data.

sudo mkdir /var/www/webmail.mydomain.com
sudo chown -R www-data:www-data /var/www/webmail.mydomain.com

Copy Roundcube Files to the web directory

We will need to grab the latest copy of Roundcube’s code to run the website. Note: please ensure you substitute the correct version for Roundcube when executing the commands below as the version listed in the guide will likely be out of date as time goes on:

cd /tmp
wget https://github.com/roundcube/roundcubemail/releases/download/1.4.1/roundcubemail-1.4.1.tar.gz
tar -xf roundcubemail-1.4.1.tar.gz
mv roundcubemail-1.4.1 /var/www/webmail.mydomain.com

Populate the SQL Database

You will need to execute the following SQL command to populate your Roundcube database with the tables needed to run Roundcube. To do so, execute the following commands.

sudo mariadb roundcubemail < /var/www/webmail.mydomain.com/SQL/mysql.initial.sql

Install Roundcube dependencies

Roundcube doesn’t ship with several javascript dependencies. To ensure the Roundcube pages load properly, you will need to execute the following command to pull down the javascript dependencies.

sudo php /var/www/webmail.mydomain.com/bin/install-jsdeps.sh

Configure NGINX

Let’s configure NGINX to point to our web directory for the website. When doing so, it is very important you protect your installation by preventing access to some sensitive files from the web.

First, create a virtual-host file within the nginx sites-available folder:

sudo vi /etc/nginx/sites-available/webmail.mydomain.com

Press i to get vi into insert mode and paste the following. Note: Please replace the values with the path to your SSL Certificate we generated earlier.

##
# You should look at the following URL's in order to grasp a solid understanding
# of Nginx configuration files in order to fully unleash the power of Nginx.
# https://www.nginx.com/resources/wiki/start/
# https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/
# https://wiki.debian.org/Nginx/DirectoryStructure
#
# In most cases, administrators will remove this file from sites-enabled/ and
# leave it as reference inside of sites-available where it will continue to be
# updated by the nginx packaging team.
#
# This file will automatically load configuration files provided by other
# applications, such as Drupal or WordPress. These applications will be made
# available underneath a path with that package name, such as /drupal8.
#
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##

# Default server configuration
#

server {
        listen 443 ssl;
        listen [::]:443 ssl;
        server_name webmail.mydomain.com;

ssl_certificate /etc/letsencrypt/live/webmail.mydomain.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/webmail.mydomain.com/privkey.pem;
  ssl_session_timeout 1d;
  ssl_session_cache shared:SSL:50m;
  ssl_session_tickets off;
  ssl_protocols TLSv1.1 TLSv1.2;
  ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK';
  ssl_prefer_server_ciphers on;
  ssl_stapling on;
  ssl_stapling_verify on;
  ssl_trusted_certificate /etc/letsencrypt/live/webmail.mydomain.com/chain.pem;


        # SSL configuration
        #
        # listen 443 ssl default_server;
        # listen [::]:443 ssl default_server;
        #
        # Note: You should disable gzip for SSL traffic.
        # See: https://bugs.debian.org/773332
        #
        # Read up on ssl_ciphers to ensure a secure configuration.
        # See: https://bugs.debian.org/765782
        #
        # Self signed certs generated by the ssl-cert package
        # Don't use them in a production server!
        #
        # include snippets/snakeoil.conf;

        root /var/www/webmail.mydomain.com;

        # Add index.php to the list if you are using PHP
        index index.php index.html index.htm;

       # Revoke access to sensitive files and directories
       location ~ ^/(README|INSTALL|LICENSE|CHANGELOG|UPGRADING)$ {
                deny all;
       }
       location ~ ^/(config|temp|bin|SQL|logs)/ {
                deny all;
       }

        # pass PHP scripts to FastCGI server
        #
        location ~ \.php$ {
                include snippets/fastcgi-php.conf;
        #
        #       # With php-fpm (or other unix sockets):
                fastcgi_pass unix:/run/php/php7.3-fpm.sock;
        #       # With php-cgi (or other tcp sockets):
        #       fastcgi_pass 127.0.0.1:9000;
        }

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        location ~ /\. {
                deny all;
                access_log off;
                log_not_found off;
       }
}

Press : and then type wq and press enter to write the changes to the file and quit in vi.

Last, we need to create a link of the virtual host file to /etc/nginx/sites-enabled. You will need to execute the following commands to create the link as well as restart nginx to apply the changes.

sudo ln -s /etc/nginx/sites-available/webmail.mydomain.com /etc/nginx/sites-enabled/webmail.mydomain.com
sudo service nginx restart

Run the Roundcube installer

At this point, if you navigate to https://webmail.mydomain.com/installer, you should see the Roundcube Webmail Installer page. You should see a series of items show OK, NOT AVAILABLE, or NOT OK. You will need to remediate any items that show NOT OK for Roundcube to successfully run.

In this installer, I primarily focused on Step 1 (Checking the environment) and Step 2 (Checking the database). Once both show OK (don’t worry about if email is successful or fails (likely it is failing still), move the installer directory to your home drive to secure the environment (IT IS VERY DANGEROUS TO LEAVE THIS PAGE!!! DON’T SKIP THIS STEP).

sudo mv /var/www/webmail.mydomain.com/installer ~

Update Roundcube configuration

I couldn’t get Roundcube to actually work during the installation with this setup until I manually specified a few items via the Roundcube configuration file. Within the /var/www/webmail.mydomain.com/config/config.inc.php file, ensure you have the following code snippets to allow Roundcube to properly authenticate to your mailserver.

sudo vi /var/www/webmail.mydomain.com/config/config.inc.php

Ensure you have the following code snippets (typically there is a section under // IMAP that has the config we can start with). To do so, press i to get vi into insert mode and paste the following.

$config['default_port'] = 993;
$config['default_host'] = 'imaps://localhost';
$config['mail_domain'] = '%d';
$config['imap_conn_options'] = array(
 'ssl'         => array(
     'verify_peer'       => true,
     'verify_peer_name' => false,
  ),
);

// SMTP
$config['smtp_server'] = 'ssl://localhost';
$config['smtp_port'] = 465;
$config['smtp_auth_type'] = 'LOGIN';
// Required if you're running PHP 5.6 or later
$config['smtp_conn_options'] = array(
    'ssl' => array(
        'verify_peer'      => true,
        'verify_peer_name' => false,
    ),
);

Press : and then type wq and press enter to write the changes to the file and quit in vi.

Verify

At this point, you should be able to login to https://webmail.mydomain.com and send/receive email!

As with all technology, ensure you keep up-to-date with all the latest security patches to keep your environment stable and secure.

If you made it to this point, were able to successfully send/receive mail via Roundcube, pat yourself on the back and grab a fine beverage!

Troubleshooting

Here are some useful commands to help troubleshoot your deployment.

sudo postqueue -p can be used to check if any pending emails are queued.

sudo postmap -q mydomain.com mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf can be used to validate what domain names are accepted. You should receive the value of 1 if it exists.

sudo postmap -q [email protected] mysql:/etc/postfix/mysql-virtual-mailbox-users.cf will validate if a user account exists with the specified email address. You should receive the value of the email address of the user if it exists.

sudo postmap -q [email protected] mysql:/etc/postfix/mysql-virtual-mailbox-aliases.cf can be used to validate the alias of an email address. You should receive the email address of the user account if it does map back to another user.

tail -f /var/log/mail.log can be useful watching how emails are handled by postfix/dovecot to troubleshoot how messages are being handled

Roundcube installation instructions (documentation): https://github.com/roundcube/roundcubemail/wiki/Installation

[Tutorial] Using Azure Hybrid Connection Manager to reach resources on-premises without VPN Connections

One of the hidden gems of Azure is HCM (Hybrid Connection Manager), which addresses the issue of Azure’s App Services (Web App, API App, Functions) having the ability to connect to resources hosted in other Azure environments, clouds, or on-premises. In many cases, VPN or ExpressRoute connectivity may be overkill or not a possibility in establishing connectivity to the requested service. The great thing is Hybrid Connections is all the traffic will be egress TCP 443 traffic to Azure via TLS 1.2, which can easily attest to the needs of many secured environments and not require ports to be opened inbound into the environment.

There are two ways to leverage Hybrid Connections for App Services in Azure:

  1. Via WCF Hybrid Relays
  2. Via Hybrid Connections

For the purposes of this article, we are going to cover how to connect to a web service “on-premises” via the HCM Agent. While we are using a Web App as an example, keep in mind that this concept can be applied to all App Services such as Web Apps, API Apps, Logic Apps, and Azure Functions. In addition, this article will make a call to a web service on-premises, however keep in mind that HCM is able to connect to any TCP service such as MSSQL, MySQL, Oracle, Web Services, custom TCP service, mainframes, etc.

Tutorial

To begin, we will first deploy a Web App from the Azure Portal to give us access to the Hybrid Connection Manager blade. Note: You can leverage any App Service to create the hybrid connection manager instance, but you must be on a paid tier (Free tier will not work).

  1. Login to the Azure Portal (portal.azure.com)
  2. Select All services -> App Services -> click + Add
  3. Fill out the required information, ensuring you are on a plan greater than Free. Select Review + create and Create

Once deployed, navigate to your Web App, select Networking, and click on Configure your hybrid connection endpoints

On the Hybrid connections screen, click on Download connection manager.

Note: This is the agent you will need to install in the environment that contains the service you are trying to access. The agent itself can be deployed on any machine as long as the machine can access the service you are trying to reach.

Installation of the agent is very straightforward. Complete the steps below.

    1. Select HybridConnectionManager.msi
    2. Read the EULA, select I accept the terms in the License Agreement, and click Install
    3. Click Finish

Once installed, navigate back to the Azure Portal (portal.azure.com), click All services -> App Services -> Select your webapp, click Networking, select Configure your hybrid connection endpoints, and click Add hybrid connection.

Click Create new hybrid connection and enter the following:

  • Hybrid connection Name
    • MyService
  • Endpoint Host
    • IPAddress or DNSNameOfTheService
  • Endpoint Port
    • PortNumberofYourService
  • Servicebus namepsace
    • Create new
  • Location
    • Pick the location of the Azure region you want to go to
  • Name
    • Enter a unique name for the service bus resource that will be created. This is a globally unique name accross all of Azure and must only consist of lowercase letters, numbers, and hyphens.

Click OK once you have filled out the information above. Once Azure has created the connection, navigate back to the machine you installed the agent on. On the machine, click Start, HybridConnectionManager, and select Hybrid Connection Manager UI.

Once the agent has launched, select Add a new Hybrid Connection.

This will prompt you to enter your Azure credentials. Enter your credentials in the prompt.

Note: if the machine is locked down and cannot leverage javascript, you can close out of the sign-in window and select Enter Manually on the previous step. Back in the Azure Portal, you can select your connection and copy the “Gateway Connection String” to connect this agent to Azure.

Once you have authenticated click the Subscription dropdown to select your Azure Subscription, select the connection you created via the portal, and click Save.

Once Saved, you should see the connection we created via the Azure Portal with the Azure Status of “Connected”. If you don’t see “Connected”, double check you don’t have a proxy blocking outbound TCP 443 requests to the Service Bus instance we created earlier (azurehcmdemo.servicebus.windows.net).

Note: To help with resiliency, you can deploy multiple agents on different machines to ensure resiliency/availability/scalability. When you select the same connection endpoint, HCM will automatically begin to load balance traffic between the agents.

Once you see the agent connected on-premises, you can validate from the Azure Portal we see the agent is connected as well. Via All services -> App Services -> your app service -> Networking -> Configure your hybrid connection endpoints, you should see “Connected” via the Status column on your Hybrid connections blade.

At this point, within your application, you should be able to reference the contents of the on-premises machine via the same connection string you may have used before. Below I’ve added an example showing an on-premises IIS server that displays the text “Moo” when you browse to the web page. Via my Web App in Azure, I created a quick PHP script that will request the on-premises server, in which HCM on the App Service will place the request on a Service Bus queue, the HCM agent on-premises will pull down the request, forward the request to the Web App on-premises, place the response back on the queue, and the web app will display the result “Moo”.

Hope this helps! If you have any questions or comments feel free to reach out below.

Helpful Links/Sources

Azure Friday Video showing an example of this: https://www.youtube.com/watch?v=y_zAJZC_8Yk

Azure documentation on Hybrid Connections: https://docs.microsoft.com/en-us/azure/app-service/app-service-hybrid-connections

How to enable logging/debug HCM: https://blogs.msdn.microsoft.com/waws/2017/06/26/troubleshooting-hybrid-connections-with-logging/

Deploying FortiGate Virtual Appliances (FortiGate-VM) on Azure

Here is a recap of some of the reflections I have with deploying Fortinet’s FortiGate appliance on Azure. This is more of a reflection of the steps I took rather than a guide, but you can use the information below as you see fit.  At a high level, you will need to deploy the device on Azure and then configure the internal “guts” of the device to allow it to route traffic properly on your Virtual Network (VNet) in Azure. While Fortinet does have some documentation on deploying their appliance, I found it very confusing, so I hope this helps walk through deployment. At the time of writing this, v6.2 was the latest version; however I recommend using at least version 6.0 or greater as it provides support for auto-scaling, which is what we will be looking at for this guide.

First, just want to provide a quick overview of the different options you can take and a rough overview of each architecture:

Deploy the Appliance in Azure

As part of this tutorial, we will look at FortiGate’s Autoscaling deployment as this will allow us to dynamically scale up or down depending on load. In addition, this deployment will provide us high availability, so in the event we lose a VM, network traffic will automatically failover to another appliance.

Architecture

A high level overview of what resources are deployed

Deployment

  1. Login to the Azure Portal
    1. https://portal.azure.com
  2. Create two new Resource Groups
    1. Navigate to All services -> Resource Groups
    2. Click Add 
    3. Create two new resource groups with the following names (they can be different if you wish, but you will need at least 2)
      1. Fortigate-Handler-RG
      2. Fortigate-VMSS-RG
  3. Create a Service Principal
    1. Navigate to All services -> Azure Active Directory
    2. Select App registrations
    3. Click New Registration
      1. Name: Fortigate-NVA
      2. Supported account types: Accounts in this organizational directory only
      3. Redirect URI: leave blank
    4. Click Register
    5. Write down the Application (client) ID, Directory (tenant) ID, and Object ID.
    6. Click on Certificates & secrets
    7. Click on the New client secret button and set the description to Fortigate-NVA, set the password expiry to your preference and click Add
    8. Write down the value of your client secret
      1. Note: once you navigate away from the blade you won’t be able to retrieve it again
  4. Delegate the Service Principal
    1. Navigate to All services -> Subscriptions -> select your subscription -> and select Access control (IAM)
    2. Click Add, Add role assignment, and use the following configuration
      1. Role: Owner
      2. Assign access to: Azure AD user, group, or service principal
      3. Select: Search for Fortigate-NVA and select it
    3. Click Save

      1. Note: I didn’t have a chance to test, but I think these permissions could likely be delegated down at the resource group level vs subscription.  If someone could confirm, please leave a comment below.
  5. Deploy the Fortigate Handler (CosmosDB and Function App)
    1. Once you click the button above to deploy the template, use the following configuration
      1. Function App Name
        1. This is the name of the Azure Function resource that gets created.  This must be globally unique across all customers within Azure.
      2. Cosmos DB Name
        1. Name of the Cosmos DB that will be created. This field must be between 3 and 31 characters and can contain only lowercase letters, numbers and -.  This value should be globally unique across all customers within Azure.
      3. Storage Account Type: Standard_LRS
      4. Tenant ID
        1. Use the Directory (tenant) ID from the Service Principal we created earlier.
      5. Subscription ID
        1. Enter the subscription ID to the Azure Subscription you wish to deploy to.  You can find your subscription ID by navigating to All services -> Subscriptions and selecting your subscription.
      6. Rest App ID
        1. Use the Application (client) ID from the Service Principal we created earlier.
      7. Rest App Secret: iW8gS………………………pMX
        1. Use the value you wrote down when generating the Client Secret when creating the Service Principal.
      8. Heart Beat Loss Count: 3
        1. Number of consecutively lost heartbeats. When the heartbeat loss count has been reached, the VM is deemed unhealthy and failover activities commence.
      9. Scaling Group Resource Group Name: Fortigate-VMSS-RG
        1. This is the value of the secret Resource Group you created at the beginning of this guide.  This Resource Group will contain the VM Scale Set and it’s corresponding resources.
      10. Script Timeout: 230
        1. This is the timeout for the Function App script to run.  By default this is 230 seconds.
      11. Election Wait Time: 90
        1. This is the maximum time (in seconds) to wait for a master election for the FortiGate’s to complete.
      12. PSK Secret: mysupersecretpassphrase
        1. This is a random string of characters used by the FortiGates in the scale set to synchronize configuration items.
      13. Package Res URL: https://github.com/fortinet/fortigate-autoscale/releases/download/1.0.3/fortigate-autoscale-azure-funcapp.zip
        1. Grab the latest version of the package for the Azure Function App from GitHub.  You can find the latest compiled versions here: https://github.com/fortinet/fortigate-autoscale/releases
  6. Deploy the VM Scale Set
    1. Once you click the button above to deploy the template, use the following configuration
      1. Instance Type: Standard_F2
      2. FOS Version: 6.2.1
      3. VNet New Or Existing: new
        1. Select whether you wish to use an existing or new Virtual Network
      4. VNet Name: AzureHubVNet
        1. The name of the VNet to be used or created.
      5. Subnet Address Prefix: 10.0.0.0/16
        1. The address space of the VNet to be used or created.
      6. Subnet1Name: Untrust
        1. The name of the subnet that will be public facing to the internet.
      7. Subnet1Prefix: 10.0.1.0/24
        1. The address space of the subnet to be created for the public facing zone.
      8. Subnet2Name: Trust
        1. The name of the subnet that will contain the private NICs of the FortiGate’s.
      9. Subnet2Prefix: 10.0.2.0/24
        1. The address space of the subnet to be created for the private facing zone.
      10. Subnet2Load Balancer IP: 10.0.2.10
        1. The IP address of the load balancer in the private zone.
      11. Subnet3Name: Private
        1. The name of the subnet that will contain the private machines that are behind the FortiGate appliance.
      12. Subnet3Prefix: 10.0.3.0/24
        1. The address space of the subnet that will contain the private machines that are behind the FortiGate.  Note: this is more of a place holder in FortiGate’s template, you can create additional subnets later on/use a different subnet for your private resources.
      13. Public IP New or Existing: new
        1. The Public IP address to be associated as the VIP of the Azure Load Balancer for incoming traffic.
      14. Scaling Group Name Prefix: fgtasg
        1. The prefix each VMSS Name is given when deploying the FortiGate autoscale template. The value of this parameter should be the same as for deploy_funcapp.json. The prefix cannot contain special characters \/””[]:|<>+=;,?*@& or begin with ‘_’ or end with ‘.’ or ‘-‘.
      15. Initial Capacity: 2
        1. How many FortiGate’s should be deployed.  Default value is 1, however I recommend at least 2 for high availability.
      16. Min Capacity: 2
        1. The smallest amount of FortiGate’s that should be running.  Default value is 1, however I recommend at least 2 for high availability.
      17. Max Capacity: 3
        1. The max amount of FortiGate’s that should be deployed.
      18. Scale Out Threshold: 80
        1. Percentage of CPU utilization at which scale-out should occur.
      19. Scale In Threshold: 20
        1. Percentage of CPU utilization at which scale-in should occur.
      20. Admin Username: azureadmin
        1. FortiGate administrator username on all VMs.
      21. Admin Password: azurepassword
        1. FortiGate administrator password on all VMs. This field must be between 11 and 26 characters and must include at least one uppercase letter, one lowercase letter, one digit, and one special character such as (! @ # $ %).
      22. Endpoint URL: https://yourfunctionappurl.azurewebsites.net
        1. This can be found by navigating to All services -> Function App -> YourFunctionApp -> URL on the overview blade.

At this point, your FortiGate deployment should be completed. When a FortiGate appliance comes up, it will reach out to the Azure Function to pull down its base configuration. Any changes to the primary FortiGate will be synchronized to any additional FortiGates deployed as well.

For those using a hub/spoke network, you will want to associate a UDR to each of your subnets to force traffic back to the internal load balancer’s VIP. You can do this by creating a new Route Table, add a Route, set the next hop type to Virtual Appliance, and set the IP address to the IP address you specified for the “Subnet2Load Balancer IP”.

You can connect to the primary FortiGate for management via web console on Port 8443 (https://IP.AD.DR.ESS:8443) or via SSH on Port 22.

References

https://docs.fortinet.com/vm/azure/fortigate/6.2/azure-cookbook/6.2.0/128029/about-fortigate-vm-for-azure

Deploying Cisco Virtual Appliances (NGFWv) on Azure

Here is a recap of some of the reflections I have with deploying Cisco NGFWv (Next Generation Firewall Virtual) on Azure. This is more of a reflection of the steps I took rather than a guide, but you can use the information below as you see fit.  At a high level, you will need to deploy the device on Azure and then configure the internal “guts” of the Cisco device to allow it to route traffic properly on your Virtual Network (VNet) in Azure. While Cisco does have decent documentation on deploying a single appliance, the primary purpose of this document is to look at HA/Scale out deployments.

First, just want to provide a quick overview of some of Cisco’s offerings today for Azure:

  • Cisco CSR
  • Cisco Meraki
    • In Cisco’s words:
      • Virtual MX is a virtual instance of a Meraki security & SD-WAN appliance, dedicated specifically to providing the simple configuration benefits of site-to-site Auto VPN for customers running or migrating IT services to an Amazon Web Services or Microsoft Azure Virtual Private Cloud (VPC).
    • Source: https://meraki.cisco.com/products/appliances/vmx100
  • Cisco ASAv
    • In Cisco’s words:
      • The ASAv is a virtualized network security solution that provides policy enforcement and threat inspection across heterogeneous, multisite environments.
      • ASA firewall and VPN capabilities help safeguard traffic and multitenant architectures. Available in most hypervisor environments, the Cisco ASAv can be deployed exactly where it is needed to protect users and workloads on-premises or in the cloud.
    • Source: https://github.com/cisco/asav
  • Cisco Firepower NGFW (Threat Defense Virtual)
    • In Cisco’s words:
      • The Cisco Firepower® NGFW (next-generation firewall) is the industry’s first fully integrated, threat-focused next-gen firewall with unified management. It uniquely provides advanced threat protection before, during, and after attacks.
      • The Firepower Threat Defense Virtual (FTDv) is the virtualized component of the Cisco NGFW solution. Organizations employing SDN can rapidly provision and orchestrate flexible network protection with Firepower NGFWv. As well, organizations using NFV can further lower costs utilizing Firepower FTDv.
    • Source: https://github.com/cisco/firepower-ngfw

Deploy the Appliance in Azure

In deploying the Cisco appliances, you’ll notice you can deploy from the Azure Marketplace: https://azuremarketplace.microsoft.com/en-us/marketplace/apps/cisco.cisco-firepower-threat-defense-appliance?tab=Overview).  Personally, I’m not a big fan of deploying the appliance this way as I don’t have as much control over naming conventions, don’t have the ability to deploy more than one appliance for scale, cannot specify my availability set, etc. While Cisco does offer an ARM template, it doesn’t allow flexibility for more than two devices, nor configures anything from a load balancer perspective. In this case, I’ve written a custom ARM template that leverages managed disks, availability sets, consistent naming nomenclature, proper VM sizing, and most importantly, let you define how many virtual instances you’d like to deploy for scaling.

Note: this article doesn’t cover deployment of Cisco’s Firepower Management Center, which is what is used to centrally manage each of the scale-out instances in a “single pane of glass”.

With the above said, this article will cover what Cisco calls their “scalable design” model. Here is an example of what this visually looks like (taken from one of their slide decks listed in the notes section at the bottom of this article):

Scalable design model as per Cisco’s Reference Architecture

Below is a link to the ARM template I use.

Cisco-NGFWv-HA.json

Deployment of this template can be done by navigating to the Azure Portal (portal.azure.com), select Create a resource, type Template Deployment in the Azure Marketplace, click Create,  select Build your own template in the editor, and paste the code into the editor.

Alternatively, you can click this button here:

Here are some notes on what the parameters mean in the template:

VMsize: Per Cisco, the recommend VM sizes should be D3 or D3v2.  Interestingly, they don’t call out the use of Premium storage anywhere, which I would highly recommend using if this was a single instance machine (to get at least some sort of SLA by Azure).

CiscoSku: Here is where you can select to use bring-your-own-license or pay-as-you-go.  Plans are should be outlined in the following link, but oddly enough the BYOL image is only available via PowerShell and their plans don’t show it:  https://azuremarketplace.microsoft.com/en-us/marketplace/apps/cisco.cisco-firepower-threat-defense-appliance?tab=PlansAndPrice

CiscoVersion: The version of the Cisco appliance to deploy.

CiscoCount: This defines how many virtual instances you want deployed and placed behind load balancers.

VNetName: The name of your virtual network you have created.

VNetRG: The name of the resource group your virtual network is in.  This may be the same as the Resource Group you are placing the Cisco devices in, but this is a needed configurable option to prevent errors referencing a VNet in a different resource group.

envPrefix: All of the resources that get created (load balancer, virtual machines, public IPs, NICs, etc.) will use this naming nomenclature.

manPrivateIPPrefix, diagPrivateIPPrefix, trustPrivateIPPrefix, untrustPrivateIPPrefix: Corresponding subnet address range.  These should be the first 3 octets of the range followed by a period.  For example, 10.5.6. would be a valid value.

manPrivateIPFirst, diagPrivateIPFirst, trustPrivateIPFirst, untrustPrivateIPFirst: The first usable IP address on the subnet specified.  For example, if my subnet is 10.4.255.0/24, I would need to specify 4 as my first usable address.

Username: this is the name of the privileged account that should be used to ssh and login to the PanOS web portal.

NewStorageAccountName: this is the name of the storage account that will store boot diagnostics for the Cisco appliances. This will give you the ability to see what the serial console shows. This value should be alphanumeric and 3-24 characters.

Password: Password to the privileged account used to ssh and login to the device.

Configure the Appliance

Complete these steps for both devices.

  1. SSH to the device via it’s public or private IP address of the management interface
    1. Please note, SSH may not come up for another 10+ minutes after deployment has finished, even though the VMs show running. There are several tasks within the Cisco appliance that run post-provisioning which take awhile to complete before the ability to SSH works.
  2. Login using the following credentials
    1. Note: Even though we specified credentials within our template, cisco has a default set of admin credentials “baked” into the image and they should be specified during first login (which prompts you to immediately change). Please login using the default admin credentials.
    2. Username: admin
    3. Password: Admin123
      1. The password is case sensitive, you should use a capital A on Admin123.
  3. Change your password once prompted
  4. Enter y to configure IPv4
  5. Enter n to not configure IPv6
    1. As of 6/1/2019, Azure only has preview support for IPv6, so this article won’t cover any IPv6 specific items
  6. Enter dhcp to configure IPv4 with DHCP
    1. All addresses in Azure should be DHCP, static addresses are set within Azure, which essentially give the appliance a DHCP reservation
    2. Important Note: Once you configure this option, you’ll get an awkward “If your networking information has changed, you will need to reconnect” message and things will appears to be stuck. Be patient, it appears a script runs in the background, you’ll see it eventually prompt for the next question.
  7. Leave your SSH connection open for the next step

Configure NGFWv to use FirePower Management Center

Once you have gone through the initial configuration on both devices, you will need to register the sensor to a Firepower Management Center instance. To do this, you will need to run the configure manager command on both appliances. Please note I’ve listed the command below with the parameters it will accept, you will need to use the applicable values for your environment.

configure manager add {hostname | IPv4_address | IPv6_address | DONTRESOLVE} reg_key [nat_id]

Per Cisco’s documentation:

  • The registration key is a user-defined one-time use key that must not exceed 37 characters. Valid characters include alphanumeric characters (A–Z, a–z, 0–9) and the hyphen (-). You will need to remember this registration key when you add the device to the Firepower Management Center.
  • If the Firepower Management Center is not directly addressable, use DONTRESOLVE.
  • The NAT ID is an optional user-defined alphanumeric string that follows the same conventions as the registration key described above. It is required if the hostname is set to DONTRESOLVE. You will need to
    remember this NAT ID when you add the device to the Firepower Management Center

Add the appliances into FirePower Management Center

Repeat the following steps for each of the appliances you deployed

  1. Login to FirePower Management Center
  2. Select the Devices tab, click Device Management, and then click the Add button
  3. Enter the following
    1. Host: ManagementIP
    2. Device Name: FriendlyDeviceNameOrHostName
    3. Registration Key: KeyYouUsedWhenRunningConfigureManagerCommandAbove
    4. Access Control Panel
      1. Specify a Name, select Network Discovery
    5. Smart Licensing
      1. Check the following you are licensed for
        1. Malware
        2. Threat
        3. URL Filtering
    6. If you used NAT, configure NAT and specify the NAT ID
    7. Click Register

Initialize the interfaces on your appliances

Repeat the following steps for each of the appliances you deployed

  1. Select the Devices tab, click Device Management, and select the edit button (Pencil Icon) for your appliance
  2. Click the edit button (Pencil Icon) for GigabitEthernet 0/0
    1. Name: Untrust
    2. Check the Enabled checkbox
    3. Security Zone: Create a new zone called Untrusted
    4. Click the IPv4 tab
      1. IP Type: Use Static
      2. IP Address: IPAddressOfYourAppliance/SubnetSize
    1. Click OK
  3. Click the edit button (Pencil Icon) for GigabitEthernet 0/1
    1. Name: Trust
    2. Check the Enabled checkbox
    3. Security Zone: Create a new zone called Trusted
    4. Click the IPv4 tab
      1. P Type: Use Static
      2. IP Address: IPAddressOfYourAppliance/SubnetSize
    5. Click OK
  4. Click the Save button

Once you have completed the steps above, click Deploy, select each of your appliances, and click Deploy to push the configuration to the device

Configure static routes on your device

In this section, we will create several routes to handle the flow of traffic to and to/from your trusted subnets, traffic destined towards the internal, traffic destined towards the management interface (we’ll need this to help handle the health probes from the azure load balancer later on), and a specific route to define the Azure Health Probes themselves.

Repeat the following steps for each of the appliances you deployed.

  1. Select the Devices tab, click Device Management, and select the edit button (Pencil Icon) for your appliance
  2. Select the Routing tab and click Static Route
  3. Click the Add Route button
    1. Type: IPv4
    2. Interface: Trust
    3. Create new network objects
      1. Add network objects that represent each of the subnets you have in Azure that the device will need to return traffic to
        1. For example, you’d repeat these steps for each private subnet
          1. Name: DBServers
          2. Network: 10.3.5.0/24
          3. Click Save
      2. Add network object for the appliance’s management interface
        1. Name: YourAppliance-mgmt
        2. Network: IPAddressOfManagementInterface
          1. Use the private IP of your management interface
        3. Click Save
      3. Add network object for Azure Health Probes
        1. Name: Azure-LB-Probe
        2. Network: 168.63.129.16
        3. Click Save
    4. Add the defined network objects above to Selected Network box
    5. Gateway: Use the IP address of the default gateway of your subnet the Trust interface is deployed on
      1. Note: To find this, navigate to the Azure Portal (portal.azure.com) and select All Services -> Virtual Networks -> Your Virtual Network -> Subnets and use the first IP address of your subnet the trusted interface is on.  For example, if the address range of my subnet is 10.5.15.0/24, I would use 10.5.15.1 as my IP address.  If my subnet was 10.5.15.128/25, I would use 129 10.5.15.129 as my IP address
    6. Metric: 3
    7. Click OK
  4. Click the Add Route button
    1. Type: IPv4
    2. Interface: Untrust
    3. Add the any-ipv4 object to Selected Network box
      1. This will allow us to force all internet bound traffic through our Untrust interface
    4. Add the Azure-LB-Probe object to the Selected Network box
      1. This will allow health probes from the external azure load balancer probes to flow properly
    5. Add the YourAppliance-mgmt object to the Selected Network box
    6. Gateway: Use the IP address of the default gateway of your subnet the Untrust interface is deployed on
      1. Note: To find this, navigate to the Azure Portal (portal.azure.com) and select All Services -> Virtual Networks -> Your Virtual Network -> Subnets and use the first IP address of your subnet the untrusted interface is on.  For example, is the address range of my subnet is 10.5.15.0/24, I would use 10.5.15.1 as my IP address.  If my subnet was 10.5.15.128/25, I would use 129 10.5.15.129 as my IP address
    7. Metric: 2
    8. Click OK
  5. Click the Save button

Once you have completed the steps above, click Deploy, select each of your appliances, and click Deploy to push the configuration to the device

Configure NAT Policies

First create a NAT rule that will SNAT any traffic from our trusted zone to the Untrust interface. This is needed so Azure understands to return traffic through the external interface of your device for inspection.

  1. Select the Devices tab, click NAT, and select the Threat Defense NAT Policy link (or New Policy button)
  2. Select your first appliance, click the Add to Policy button, and click Save
  3. Click the Add Rule button
    1. NAT Rule: Auto NAT Rule
    2. Type: Dynamic
    3. Interface Objects Tab
      1. Select the Trusted Interface Object and click the Add to Source button
      2. Select the Untrusted Interface object and click the Add to Destination button
    4. Translation Tab
      1. Click the green button to add a new network object under Original Packet
        1. Name: any-ipv4
        2. Network: 0.0.0.0/0
        3. Click Save
      2. Original Source: any-ipv4
      3. Translated Source: Destination Interface IP
    5. Click OK

Next, we need to create a new NAT statement to handle traffic for our load balancer probes. We will need to configure two statements since we will receive health probes from the same IP address (168.63.129.16) to both NICs. On the same appliance, continue the following steps.

  1. Click the Add Rule button
    1. NAT Rule: Manual NAT Rule
    2. Type: Static
    3. Interface Objects Tab
      1. Select the Trusted Interface Object and click the Add to Source button
      2. Select the Untrusted Interface object and click the Add to Destination button
    4. Translation Tab
      1. Original Source: Azure-LB-Probe
      2. Original Destination: Source Interface IP
      3. Original Destination Port: SSH
      4. Translated Source: Destination Interface IP
      5. Translated Destination: YourAppliance-mgmt
      6. Translated Destination Port: SSH
    5. Click OK
  2. Click the Add Rule button
    1. NAT Rule: Manual NAT Rule
    2. Type: Static
    3. Interface Objects Tab
      1. Select the Untrusted Interface Object and click the Add to Source button
      2. Select the Trusted Interface object and click the Add to Destination button
    1. Translation Tab
      1. Original Source: Azure-LB-Probe
      2. Original Destination: Source Interface IP
      3. Original Destination Port: SSH
      4. Translated Source: Destination Interface IP
      5. Translated Destination: YourAppliance-mgmt
      6. Translated Destination Port: SSH
  3. Click OK

Optional Step: If you are using the appliances to front applications to the internet, you will also need to configure a NAT rule for ingress traffic. This is an optional step, but will show you how to configure traffic to let’s say a web server (which the ALB is configured to listen for per the template). If you do complete this step, make sure you add an access policy (Policies -> Access Control -> Select your policy -> click Add Rule).

  1. Click the Add Rule button
    1. NAT Rule: Manual NAT Rule
    2. Type: Static
    3. Interface Objects Tab
      1. Select the Untrusted Interface Object and click the Add to Source button
      2. Select the Trusted Interface object and click the Add to Destination button
    4. Translation Tab
      1. Original Source: any-ipv4
      2. Original Destination: Source Interface IP
      3. Original Destination Port: HTTP
      4. Translated Source: Destination Interface IP
      5. Translated Destination: webserver
        1. Click the green add button to create a new network object to define the private IP address of your web server.
      6. Translated Destination Port: HTTP
    1. Click OK

Click Save once you have finished adding the rules.

At this point, you will need to repeat the same steps above. The reason why we cannot apply the policy to both devices is when you configure the rule for the Azure Health Probes, you’ll need to specify the correct Translated Destination (I.e. Appliance1 should use the network object that resolves to appliance 1; Appliance2 should use the network object that resolves to appliance 2)

Once you have completed the steps above, click Deploy, select each of your appliances, and click Deploy to push the configuration to the device

Finalize the environment

Now that the environment is configured, there are two steps you will want to check back on.

  1. Add Route Tables to each subnet to force traffic to the Cisco appliances
    1. You will need to leverage route tables with custom routes to force traffic to the Cisco appliance. I’d highly recommend giving this a read to familiarize yourself with how Route Tables work in Azure: https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-udr-overview
  2. Ensure there is a Network Security Group (NSG) on the Untrust subnet
    1. As per Azure Load Balancer’s documentation, you will need an NSG associated to the NICs or subnet to allow traffic in from the internet. https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-standard-overview#securebydefault
  3. Remove the public IP from your management interface
    1. Considering at this point you’ve configured the device and have private connectivity via VPN or ExpressRoute, I’d remove the public IP from your management interface to prevent the public internet from accessing this interface
  4. Adjust NSG rules
    1. Similar to above, I’d scope down who/what network segments can pass traffic to the device. Go back and modify the NSG on the management interfaces to only allow traffic from specific source addresses.

References

Azure APIs and ARM Templating

One thing that has always been a mystery when leveraging Azure Resource Manager (ARM) templates is the “apiVersion” that needs to be specified for each resource that should be created in your template.

At a high level, every item processed by ARM will reach out to its corresponding service provider. Each of these providers have their own API for handling requests: https://docs.microsoft.com/en-us/rest/api/resources/

The tricky thing is many of the template examples that are provided out on GitHub in the quickstart gallery don’t align or may not be using the latest API. In this case, the question is… how do I find the latest API used by each resource provider?

Within PowerShell there’s a single command that can return back the API versions supported by each Resource Provider.

(Get-AzResourceProvider -ProviderNamespace "Microsoft.Storage").ResourceTypes

You’ll notice the follow result, which provides us exactly what we were looking for.

PS C:\Users\jstrom> (Get-AzResourceProvider -ProviderNamespace "Microsoft.Storage").ResourceTypes


ResourceTypeName : storageAccounts
Locations        : {East US, East US 2, West US, West Europe...}
ApiVersions      : {2019-04-01, 2018-11-01, 2018-07-01, 2018-03-01-preview...}

ResourceTypeName : operations
Locations        : {}
ApiVersions      : {2019-04-01, 2018-11-01, 2018-07-01, 2018-03-01-preview...}

ResourceTypeName : locations/asyncoperations
Locations        : {East US, East US 2, West US, West Europe...}
ApiVersions      : {2019-04-01, 2018-11-01, 2018-07-01, 2018-03-01-preview...}

ResourceTypeName : storageAccounts/listAccountSas
Locations        : {East US, East US 2, West US, West Europe...}
ApiVersions      : {2019-04-01, 2018-11-01, 2018-07-01, 2018-03-01-preview...}

ResourceTypeName : storageAccounts/listServiceSas
Locations        : {East US, East US 2, West US, West Europe...}
ApiVersions      : {2019-04-01, 2018-11-01, 2018-07-01, 2018-03-01-preview...}

ResourceTypeName : storageAccounts/blobServices
Locations        : {East US, East US 2, West US, West Europe...}
ApiVersions      : {2019-04-01, 2018-11-01, 2018-07-01, 2018-03-01-preview...}

ResourceTypeName : storageAccounts/tableServices
Locations        : {East US, East US 2, West US, West Europe...}
ApiVersions      : {2019-04-01, 2018-11-01, 2018-07-01, 2018-03-01-preview...}

ResourceTypeName : storageAccounts/queueServices
Locations        : {East US, East US 2, West US, West Europe...}
ApiVersions      : {2019-04-01, 2018-11-01, 2018-07-01, 2018-03-01-preview...}

ResourceTypeName : storageAccounts/fileServices
Locations        : {East US, East US 2, West US, West Europe...}
ApiVersions      : {2019-04-01, 2018-11-01, 2018-07-01, 2018-03-01-preview...}

ResourceTypeName : locations
Locations        : {}
ApiVersions      : {2019-04-01, 2018-11-01, 2018-07-01, 2018-03-01-preview...}

ResourceTypeName : locations/usages
Locations        : {East US, East US 2, West US, West Europe...}
ApiVersions      : {2019-04-01, 2018-11-01, 2018-07-01, 2018-03-01-preview...}

ResourceTypeName : locations/deleteVirtualNetworkOrSubnets
Locations        : {East US, East US 2, West US, West Europe...}
ApiVersions      : {2019-04-01, 2018-11-01, 2018-07-01, 2018-03-01-preview...}

ResourceTypeName : usages
Locations        : {}
ApiVersions      : {2019-04-01, 2018-11-01, 2018-07-01, 2018-03-01-preview...}

ResourceTypeName : checkNameAvailability
Locations        : {}
ApiVersions      : {2019-04-01, 2018-11-01, 2018-07-01, 2018-03-01-preview...}

ResourceTypeName : storageAccounts/services
Locations        : {East US, West US, East US 2 (Stage), West Europe...}
ApiVersions      : {2014-04-01}

ResourceTypeName : storageAccounts/services/metricDefinitions
Locations        : {East US, West US, East US 2 (Stage), West Europe...}

Happy ARM templating!

DPM 2016 – Reporting Services Server cannot connect to DPM Database

After installing System Center Data Protection Manager from scratch or after performing an upgrade from DPM 2012 R2, when you attempt to schedule a report to be mailed, you receive the following popup error or notice that when you generate a report you just receive a white page in your browser that continues to load indefinitely:


Reporting Services Server cannot connect to the DPM database.
To repair the configuration, follow steps for repairing DPM from DPM Setup Help.
ID: 3001
More Information
Just an endless loading page presented in your browser.

Unfortunately, following the repair steps suggested in the More Information link does not resolve the problem.

Resolution for SQL Server 2016 and later

  1. On the DPM server, open Computer Management, expand Local Users and Groups, click Groups, and create a new local group with the following information (replace items in red with the actual server name/hostname of your machine):
    1. Group name: DPMDBReaders$DPMSERVERNAME
    2. Description: This group is internally used by Microsoft System Center 2016 Data Protection Manager.
  2. On the DPM server, open Computer Management, expand Local Users and Groups, click Users, and create a new local user with the following information (replace items in red with the actual server name/hostname of your machine):
    1. User name: DPMR$DPMSERVERNAME
    2. Full name: DPMR$DPMSERVERNAME
    3. Description: This account is used for SQL reporting to generate reports for DPM 2016.
    4. Enter a strong password
    5. Check Password never expires
  3. Select your recently created DPMR$DPMSERVERNAME account and click on the Membership of tab
    1. Add the DPMDBReaders$DPMSERVERNAME group we created in step 1
    2. Click OK
  4. Start Microsoft SQL Server Management Studio and connect to the SQL instance used by DPM.
    1. Expand Security, right-click on the Logins, select New login
      1. Click on the Search… button and add the local group DPMDBReaders$DPMSERVERNAME
      2. Set the Default database to YourDPMDatabase

      3. Click on the User Mapping section, check the checkbox for YourDPMDatabase, and check the checkbox for the db_datareader role.
      4. Click OK
  5. In Microsoft SQL Server Management Studio, expand Databases, expand YourDPMDatabase, expand Security, expand Users, and right click Properties on the DPMDBReaders$DPMSERVERNAME group you granted access to in step 4.
    1. Click on Securables
    2. Click the Search… button
    3. Select Specific objects… and click OK
    4. Click the Object Types… button
    5. Check Stored precedures and click OK
    6. Click the Browse… button
    7. Check [dbo].[prc_MOM_Heartbeat_Get] and [dbo].[prc_MOM_ProductionServerGet] and click OK
    8. Click OK on the Select objects dialog box
    9. Place a checkbox in the Grant column for the Execute row.
      1. Make sure you do this step for both [dbo].[prc_MOM_Heartbeat_Get] and [dbo].[prc_MOM_ProductionServerGet], checking the box once will only update one of the storage procedures.
    10. Click OK
  6. Exit Microsoft SQL Server Management Studio.
  7. Open Reporting Services Configuration Manager and connect to the SqlServer and Instance hosting the DPM reports (as you go through this, replace the items in Red with your applicable values).
    1. Click on the Web Portal URL menu item and click on the listed URL for DPM.
    2. Click on the DPMReports_GUID folder to open the DPM reports page.
    3. Click on the DPMReporterDataSource Data Source to open its properties.
    4. Under Credentials, use the following configuration:
      1. Select the Using the following credentials radio button
      2. Type of credentials: Windows user name and password
      3. User name: DPMR$DPMSERVERNAME
      4. Password: EnterYourPasswordToTheAccount
      5. Ensure Log in using these credentials, but then try to impersonate the user viewing the report is unchecked
      6. Click the Test connection button
        1. Ensure it says Connected successfully

      7. Click Apply
    5. Close out of your web browser
  8. Back in the Reporting Services Configuration Manager window, complete the following steps
    1. Select the Service Account menu item
    2. Ensure Use built-in account is set to Network Service
      1. Check the Use built-in account radio button
      2. Set the account to Network Service
      3. Click Apply
      4. You will be prompted to save an encryption key. Save the key to location of your choosing, and type a password to be used to encrypt the file. Click OK
      5. You will then be prompted to specify an account with administrator privileges. Click OK to use your current account.
  9. Reboot the DPM Server

At this point, you should now be able to schedule e-mail reports without experiencing the original error and your reports should load properly!

DPM 2016 – Anonymous / Open Relay for SMTP Notifications

DPM 2016 is primarily geared towards using mail servers that require authentication (rightfully so, that’s a best security practice). However, many IT organizations have local mail relay servers with anonymous authentication that are used for several IT services in the organization. Unfortunately, DPM 2016 gets a bit wonky using unauthenticated mail servers and will likely give you a generic error that says:

Error ID: 2013
Details: The user name or password is incorrect

And if you ignore the error and head over to the notifications tab to configure a notification, you will be presented with another generic error:

An authentication error occured when trying to connect to the SMTP serve. (ID: 518)

You typed an incorrect user name, password, or SMTP server name. Type the correct user name or password to enable e-mail delivery of reports and alert notifications.

And if you are trying to configure scheduled emails you may receive an error about reporting services:

DPM Setup is unable to update the report server configuration to configure e-mail settings. (ID: 3040).

One thing I may do before getting too far ahead though is validate you can send an email from the DPM server. This can easily be done via PowerShell by executing the following command:

Send-MailMessage -SMTPServer localhost -To [email protected] -From [email protected] -Subject "Test Email from DPM Server" -Body "Howdy!  This is a test from the DPM Sever.  If you see this, mail relay is working!"

When executing the PowerShell command, it won’t return anything, but you should hopefully see a message in your mailbox. If you do, you’ve at least ruled out network/mail issues.

Once you’ve ruled out connectivity/the mail server, we will complete the following steps below to configure DPM.

  1. Configure E-mail for SQL Server Reporting Services
  2. Create a Local User Account
  3. Remove any artifacts left in the registry
  4. Update the SMTP settings in DPM.

Configuration

  1. Configure SQL Server Reporting Services
    1. Open Reporting Services Configuration Manager
    2. Sign into your DPM instance
    3. Select E-mail Settings and leverage the following configuration
      1. Sender Address: [email protected]
      2. SMTP Server: emailserver.yourdomain.com
      3. Authentication: No authentication

    4. Click Apply
  2. Create a local user account
    1. Open Computer Management, expand Local Users and Groups, select Users, and Create a new local user on the machine
      1. Create the user (I used anonemail as the account name, but anything can be specified)
      2. Remove all group membership
        1. This account doesn’t need to be a part of any group, including the Users group
        2. This account should not be a part of administrators (I’ve seen other blog posts mention you must use administrator, that is 100% not necessary and can be considered a security risk)
      3. Ensure the account is enabled
        1. A disabled account will not work
  3. Cleanup the registry
    1. Open registry editor (regedit.msc)
    2. Navigate to HKEY_LOCAL_MACHINE\Software\Microsoft\Microsoft Data Protection Manager\Notification
    3. Delete the following keys (if they exist):
      1. SmtpUserName
      2. SmtpPassword
  4. Reboot the DPM Server
    1. Technically, you could restart two services:
      SQL Server Reporting Services instance for DPM and the DPM service, but a reboot never hurts 😉
  5. Configure DPM to use SMTP relay
    1. Close out of the DPM and reopen
    2. Select Reporting, waiting for the screen to finish loading, and then select Action, Options
    3. Select the SMTP Server tab and enter
      1. SMTP sever name: relayserver.mydomain.com
      2. SMTP server port: 25
      3. “From” Address: [email protected]
      4. Username: .\localuserwecreatedearlier
        1. Ensure you have .\ to designate the user is local
      5. Password: LocalUserAccountPassword

    4. Click the Send Test E-Mail button and specify an email address to send a test email to validate all is well
    5. Success!
    6. Click OK on the Options window to save your settings

At this point, you should be able to relay emails through your open relay as well as schedule emails for reports without error.

DPM 2016 Installation – Error ID: 4387

When installing DPM 2016, you may get a really generic error during the “Prerequisites check” during installation. Looking online, there’s a ton of individuals that have this issue, but no one correlates the log files to specifically what is needed to solve each problem (yep, “each” problem, Error 4387 is a generic catch all for several issues during the prerequisits check).

Before I get into the article, the too long didn’t read (TLDR) version is make sure you are using both SQL Server 2016 (no service pack) and SSMS 16.5 or earlier to successfully install DPM 2016.

To get a bit more technical and find out what’s going on, open up the DPM Installation logs after you receive the error. The installation log files can be found by browsing to %ProgramFiles%\Microsoft System Center 2016\DPM\DPMLogs.  Documentation on where log files are stored by DPM can be found here: https://docs.microsoft.com/en-us/system-center/dpm/set-up-dpm-logging?view=sc-dpm-2016

Here’s a copy of my DpmSetup.log file, in which when looking through it, there isn’t a clear cut answer, just this generic line at the bottom ([3/1/2019 6:22:54 AM] *** Error : CurrentDomain_UnhandledException).

[3/1/2019 6:21:16 AM] Information : Microsoft System Center 2016 Data Protection Manager setup started.
[3/1/2019 6:21:16 AM] Data : Mode of setup = User interface
[3/1/2019 6:21:16 AM] Data : OSVersion = Microsoft Windows NT 10.0.14393.0
[3/1/2019 6:21:16 AM] Information : Check if the media is removable
[3/1/2019 6:21:16 AM] Data : Folder Path = C:\Program Files\Microsoft System Center 2016\DPM
[3/1/2019 6:21:16 AM] Data : Drive Name = C:\
[3/1/2019 6:21:16 AM] Data : Drive Type = 3
[3/1/2019 6:21:16 AM] Information : Check attributes of the directory
[3/1/2019 6:21:16 AM] Data : Folder Path = C:\Program Files\Microsoft System Center 2016\DPM
[3/1/2019 6:21:16 AM] Data : File Attributes = Directory
[3/1/2019 6:21:16 AM] Information : Check if the media is removable
[3/1/2019 6:21:16 AM] Data : Folder Path = C:\Program Files\Microsoft Data Protection Manager
[3/1/2019 6:21:16 AM] Data : Drive Name = C:\
[3/1/2019 6:21:16 AM] Data : Drive Type = 3
[3/1/2019 6:21:16 AM] Information : Check attributes of the directory
[3/1/2019 6:21:16 AM] Data : Folder Path = C:\Program Files\Microsoft Data Protection Manager
[3/1/2019 6:21:16 AM] * Exception : Ignoring the following exception intentionally => System.IO.FileNotFoundException: Could not find file 'C:\Program Files\Microsoft Data Protection Manager'.
File name: 'C:\Program Files\Microsoft Data Protection Manager'
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.GetAttributes(String path)
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Wizard.InstallLocationValidation.CheckForDirectoryAttributes(String path)
[3/1/2019 6:21:16 AM] Information : Check if the media is removable
[3/1/2019 6:21:16 AM] Data : Folder Path = C:\Program Files\Microsoft System Center 2016\DPM\DPM\DPMDB
[3/1/2019 6:21:16 AM] Data : Drive Name = C:\
[3/1/2019 6:21:16 AM] Data : Drive Type = 3
[3/1/2019 6:21:16 AM] Information : Check attributes of the directory
[3/1/2019 6:21:16 AM] Data : Folder Path = C:\Program Files\Microsoft System Center 2016\DPM\DPM\DPMDB
[3/1/2019 6:21:16 AM] * Exception : Ignoring the following exception intentionally => System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Program Files\Microsoft System Center 2016\DPM\DPM\DPMDB'.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.GetAttributes(String path)
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Wizard.InstallLocationValidation.CheckForDirectoryAttributes(String path)
[3/1/2019 6:21:17 AM] Information : The setup wizard is initialized.
[3/1/2019 6:21:17 AM] Information : Starting the setup wizard.
[3/1/2019 6:21:17 AM] Information : <<< Dialog >>> Welcome Page : Entering
[3/1/2019 6:22:33 AM] Information : <<< Dialog >>> Welcome Page : Leaving
[3/1/2019 6:22:33 AM] Information : <<< Dialog >>> Inspect Page : Entering
[3/1/2019 6:22:41 AM] Information : Query WMI provider for path of configuration file for SQL Server 2008 Reporting Services.
[3/1/2019 6:22:41 AM] Information : Querying WMI Namespace: \\DPM-SERVER\root\Microsoft\SqlServer\ReportServer\RS_DPM\V13\admin for query: SELECT * FROM MSReportServer_ConfigurationSetting WHERE InstanceName='DPM'
[3/1/2019 6:22:42 AM] Data : Path of configuration file for SQL Server 2008 Reporting Services = C:\Program Files\Microsoft SQL Server\MSRS13.DPM\Reporting Services\ReportServer\RSReportServer.config
[3/1/2019 6:22:42 AM] * Exception :  => System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.SqlServer.Smo, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91' or one of its dependencies. The system cannot find the file specified.
File name: 'Microsoft.SqlServer.Smo, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91'
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Helpers.MiscHelper.IsSqlClustered(String sqlMachineName, String sqlInstanceName)
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Helpers.MiscHelper.IsMachineClustered(String sqlMachineName, String sqlInstanceName)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

[3/1/2019 6:22:42 AM] * Exception :  => System.Management.ManagementException: Invalid namespace 
   at System.Management.ManagementException.ThrowWithExtendedInfo(ManagementStatus errorCode)
   at System.Management.ManagementScope.InitializeGuts(Object o)
   at System.Management.ManagementScope.Initialize()
   at System.Management.ManagementObjectSearcher.Initialize()
   at System.Management.ManagementObjectSearcher.Get()
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Helpers.WmiHelper.IsMachineClustered(String machineName, String instanceName)
[3/1/2019 6:22:42 AM] Information : OS >= win 8 , enable Dedupe role
[3/1/2019 6:22:53 AM] Information : output : True
.. 
 error : 
[3/1/2019 6:22:53 AM] Data : Path of inspection output xml = C:\Program Files\Microsoft System Center 2016\DPM\DPMLogs\InspectReport.xml
[3/1/2019 6:22:53 AM] Information : Instantiating inspect component.
[3/1/2019 6:22:53 AM] Data : Path of output xml = C:\Program Files\Microsoft System Center 2016\DPM\DPMLogs\InspectReport.xml
[3/1/2019 6:22:53 AM] Information : Deserializing the check XML from path : C:\Users\labuser.CONTOSO\AppData\Local\Temp\DPM8AFA.tmp\DPM2012\Setup\checks.xml
[3/1/2019 6:22:53 AM] Information : Loading the check XML from path : C:\Users\labuser.CONTOSO\AppData\Local\Temp\DPM8AFA.tmp\DPM2012\Setup\checks.xml
[3/1/2019 6:22:54 AM] Information : Deserialising the scenario XML from path : C:\Users\labuser.CONTOSO\AppData\Local\Temp\DPM8AFA.tmp\DPM2012\Setup\scenarios.xml
[3/1/2019 6:22:54 AM] Information : Loading the check XML from path : C:\Users\labuser.CONTOSO\AppData\Local\Temp\DPM8AFA.tmp\DPM2012\Setup\scenarios.xml
[3/1/2019 6:22:54 AM] Information : Getting scenarios for the product: DPM
[3/1/2019 6:22:54 AM] Information : Getting scenarios for DPM
[3/1/2019 6:22:54 AM] Information : Getting scenario for Mode:Install, DbLocation:Remote, SKU:Retail and CCMode:NotApplicable
[3/1/2019 6:22:54 AM] *** Error : Initialize the SQLSetUpHelper Object
[3/1/2019 6:22:54 AM] Information : [SQLSetupHelper.GetWMIReportingNamespace]. Reporting Namespace found. Reporting Namespace : V13
[3/1/2019 6:22:54 AM] Information : [SQLSetupHelper.GetWMISqlServerNamespace]. SQL Namespace found. SQL Namespace : \\DPM-SERVER\root\Microsoft\SqlServer\ComputerManagement13
[3/1/2019 6:22:54 AM] Information : Query WMI provider for SQL Server 2008.
[3/1/2019 6:22:54 AM] Information : Querying WMI Namespace: \\DPM-SERVER\root\Microsoft\SqlServer\ComputerManagement13 for query: Select * from SqlServiceAdvancedProperty where ServiceName='MSSQL$DPM' and PropertyName='Version'
[3/1/2019 6:22:54 AM] Information : SQL Server 2008 R2 SP2 instance DPM is present on this system.
[3/1/2019 6:22:54 AM] Information : Query WMI provider for SQL Server 2008.
[3/1/2019 6:22:54 AM] Information : Querying WMI Namespace: \\DPM-SERVER\root\Microsoft\SqlServer\ComputerManagement13 for query: Select * from SqlServiceAdvancedProperty where ServiceName='MSSQL$DPM' and PropertyName='Version'
[3/1/2019 6:22:54 AM] Information : [SQLSetupHelper.GetSQLDepedency]. Reporting Namespace and SQL namespace for installed SQL server which will be used as DPM DB. Reporting Namespace : \\DPM-SERVER\root\Microsoft\SqlServer\ReportServer\RS_DPM\V13\admin SQL Namespace : \\DPM-SERVER\root\Microsoft\SqlServer\ComputerManagement13
[3/1/2019 6:22:54 AM] Information : Check if SQL Server 2012 Service Pack 1 Tools is installed.
[3/1/2019 6:22:54 AM] Information : [SQLSetupHelper.GetSqlSetupRegKeyPath]. Registry Key path that contains SQL tools location: Software\Microsoft\Microsoft SQL Server\140\Tools\Setup\
[3/1/2019 6:22:54 AM] Information : Inspect.CheckSqlServerTools : MsiQueryProductState returned : INSTALLSTATE_DEFAULT
[3/1/2019 6:22:54 AM] *** Error : CurrentDomain_UnhandledException

Digging some more, I found that DPM seems to also place logs within the %temp% folder. Within this folder, I found that a tmpXXX.xml file was being created each time I ran through the installer and triggered an error. Upon opening the file, I see the following:

<?xml version="1.0" encoding="utf-16"?>
<WatsonInfo xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ExceptionList>
    <ExceptionEntry>
      <Exception_x0020_Type_x000D__x000A_>System.ArgumentNullException</Exception_x0020_Type_x000D__x000A_>
      <StackTrace_x000D__x000A_>   at System.Version.Parse(String input)
   at System.Version..ctor(String version)
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Inspect.InspectPrerequisites.CheckSqlServerTools(InspectContext context)
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Inspect.Inspect.InitializeContext(String sqlMachineName, String sqlInstanceName, String reportingMachineName, String reportingInstanceName, ConnectionOptions wmiSqlConnectionOptions, ConnectionOptions wmiReportingConnectionOptions, Boolean isRemoteDb, Boolean isSqlClustered, List`1 sqlClusterNodes, Boolean isRemoteReporting, String oldSqlMachineName, String oldSqlInstanceName, ProductNameEnum productName, InspectModeEnum inspectMode, Boolean remoteTriggerJob)
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Inspect.Inspect..ctor(String reportFilePath, String sqlMachineName, String sqlInstanceName, String reportingMachineName, String reportingInstanceName, ConnectionOptions wmiSqlConnectionOptions, ConnectionOptions wmiReportingConnectionOptions, Boolean isRemoteDb, Boolean isSqlClustered, List`1 sqlClusterNodes, Boolean isRemoteReporting, String oldSqlMachineName, String oldSqlInstanceName, InspectModeEnum inspectMode, InspectSkuEnum inspectSku, ProductNameEnum productName, InspectCCModeEnum ccMode, Boolean remoteTriggerJob)
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Wizard.BackEnd.InstantiateInspect(String inspectFile, String sqlMachineName, String sqlInstanceName, String reportingMachineName, String reportingInstanceName, ConnectionOptions wmiSqlConnectionOptions, ConnectionOptions wmiReportingConnectionOptions)
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Wizard.InspectPage.RunInspect()
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Wizard.InspectPage.InspectThreadEntry()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()</StackTrace_x000D__x000A_>
      <message>Value cannot be null.
Parameter name: input</message>
      <targetSite>System.Version Parse(System.String)</targetSite>
    </ExceptionEntry>
  </ExceptionList>
  <UICulture>en-US</UICulture>
  <Culture>en-US</Culture>
  <CLRVersion>4.0.30319.42000</CLRVersion>
  <OSVersion>Microsoft Windows NT 10.0.14393.0</OSVersion>
  <BucketingParametersValue>
    <Other>BC81BBF7</Other>
    <ExceptionPoint>System.Version.Parse</ExceptionPoint>
    <ExceptionName>System.ArgumentNullException</ExceptionName>
    <ModuleVersion>5.0.158.0</ModuleVersion>
    <ModuleName>SetupDpm.exe</ModuleName>
    <ApplicationVersion>5.0.158.0</ApplicationVersion>
    <ApplicationName>SetupDpm</ApplicationName>
  </BucketingParametersValue>
  <Info>Microsoft Data Protection Manager Exception Record</Info>
</WatsonInfo>

Looking through the above stack trace, I see hints that this is to SQL Server and in this case I’m receiving a null value for what looks like a version. So after reading other posts online, everyone said to downgrade to SQL Server 2016 RTM.

SQL Server 2016 version numbers: https://support.microsoft.com/en-us/help/3177312/sql-server-2016-build-versions

After downgrading to SQL Server 2016 RTM, I noticed I still received Error ID: 4387. This time I don’t see any files within the %temp% directory, but I did find in the DPMSetup.log file (within the DPMLogs directory) the following log:

[3/8/2019 5:13:09 AM] Information : Microsoft System Center 2016 Data Protection Manager setup started.
[3/8/2019 5:13:09 AM] Data : Mode of setup = User interface
[3/8/2019 5:13:09 AM] Data : OSVersion = Microsoft Windows NT 10.0.14393.0
[3/8/2019 5:13:09 AM] Information : Check if the media is removable
[3/8/2019 5:13:09 AM] Data : Folder Path = C:\Program Files\Microsoft System Center 2016\DPM
[3/8/2019 5:13:09 AM] Data : Drive Name = C:\
[3/8/2019 5:13:09 AM] Data : Drive Type = 3
[3/8/2019 5:13:09 AM] Information : Check attributes of the directory
[3/8/2019 5:13:09 AM] Data : Folder Path = C:\Program Files\Microsoft System Center 2016\DPM
[3/8/2019 5:13:09 AM] Data : File Attributes = Directory
[3/8/2019 5:13:09 AM] Information : Check if the media is removable
[3/8/2019 5:13:09 AM] Data : Folder Path = C:\Program Files\Microsoft Data Protection Manager
[3/8/2019 5:13:09 AM] Data : Drive Name = C:\
[3/8/2019 5:13:09 AM] Data : Drive Type = 3
[3/8/2019 5:13:09 AM] Information : Check attributes of the directory
[3/8/2019 5:13:09 AM] Data : Folder Path = C:\Program Files\Microsoft Data Protection Manager
[3/8/2019 5:13:09 AM] * Exception : Ignoring the following exception intentionally => System.IO.FileNotFoundException: Could not find file 'C:\Program Files\Microsoft Data Protection Manager'.
File name: 'C:\Program Files\Microsoft Data Protection Manager'
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.GetAttributes(String path)
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Wizard.InstallLocationValidation.CheckForDirectoryAttributes(String path)
[3/8/2019 5:13:09 AM] Information : Check if the media is removable
[3/8/2019 5:13:09 AM] Data : Folder Path = C:\Program Files\Microsoft System Center 2016\DPM\DPM\DPMDB
[3/8/2019 5:13:09 AM] Data : Drive Name = C:\
[3/8/2019 5:13:09 AM] Data : Drive Type = 3
[3/8/2019 5:13:09 AM] Information : Check attributes of the directory
[3/8/2019 5:13:09 AM] Data : Folder Path = C:\Program Files\Microsoft System Center 2016\DPM\DPM\DPMDB
[3/8/2019 5:13:09 AM] * Exception : Ignoring the following exception intentionally => System.IO.DirectoryNotFoundException: Could not find a part of the path 'C:\Program Files\Microsoft System Center 2016\DPM\DPM\DPMDB'.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.GetAttributes(String path)
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Wizard.InstallLocationValidation.CheckForDirectoryAttributes(String path)
[3/8/2019 5:13:10 AM] Information : The setup wizard is initialized.
[3/8/2019 5:13:10 AM] Information : Starting the setup wizard.
[3/8/2019 5:13:10 AM] Information : <<< Dialog >>> Welcome Page : Entering
[3/8/2019 5:13:55 AM] Information : <<< Dialog >>> Welcome Page : Leaving
[3/8/2019 5:13:55 AM] Information : <<< Dialog >>> Inspect Page : Entering
[3/8/2019 5:14:06 AM] Information : Query WMI provider for path of configuration file for SQL Server 2008 Reporting Services.
[3/8/2019 5:14:06 AM] Information : Querying WMI Namespace: \\DPM-SERVER\root\Microsoft\SqlServer\ReportServer\RS_DPM\V13\admin for query: SELECT * FROM MSReportServer_ConfigurationSetting WHERE InstanceName='DPM'
[3/8/2019 5:14:06 AM] Data : Path of configuration file for SQL Server 2008 Reporting Services = C:\Program Files\Microsoft SQL Server\MSRS13.DPM\Reporting Services\ReportServer\RSReportServer.config
[3/8/2019 5:14:06 AM] * Exception :  => System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.SqlServer.Smo, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91' or one of its dependencies. The system cannot find the file specified.
File name: 'Microsoft.SqlServer.Smo, Version=10.0.0.0, Culture=neutral, PublicKeyToken=89845dcd8080cc91'
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Helpers.MiscHelper.IsSqlClustered(String sqlMachineName, String sqlInstanceName)
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Helpers.MiscHelper.IsMachineClustered(String sqlMachineName, String sqlInstanceName)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

[3/8/2019 5:14:06 AM] * Exception :  => System.Management.ManagementException: Invalid namespace 
   at System.Management.ManagementException.ThrowWithExtendedInfo(ManagementStatus errorCode)
   at System.Management.ManagementScope.InitializeGuts(Object o)
   at System.Management.ManagementScope.Initialize()
   at System.Management.ManagementObjectSearcher.Initialize()
   at System.Management.ManagementObjectSearcher.Get()
   at Microsoft.Internal.EnterpriseStorage.Dls.Setup.Helpers.WmiHelper.IsMachineClustered(String machineName, String instanceName)
[3/8/2019 5:14:06 AM] Information : OS >= win 8 , enable Dedupe role
[3/8/2019 5:14:07 AM] Information : output : True
.. 
 error : 
[3/8/2019 5:14:08 AM] Data : Path of inspection output xml = C:\Program Files\Microsoft System Center 2016\DPM\DPMLogs\InspectReport.xml
[3/8/2019 5:14:08 AM] Information : Instantiating inspect component.
[3/8/2019 5:14:08 AM] Data : Path of output xml = C:\Program Files\Microsoft System Center 2016\DPM\DPMLogs\InspectReport.xml
[3/8/2019 5:14:08 AM] Information : Deserializing the check XML from path : C:\Users\labuser.CONTOSO\AppData\Local\Temp\DPM6EC.tmp\DPM2012\Setup\checks.xml
[3/8/2019 5:14:08 AM] Information : Loading the check XML from path : C:\Users\labuser.CONTOSO\AppData\Local\Temp\DPM6EC.tmp\DPM2012\Setup\checks.xml
[3/8/2019 5:14:08 AM] Information : Deserialising the scenario XML from path : C:\Users\labuser.CONTOSO\AppData\Local\Temp\DPM6EC.tmp\DPM2012\Setup\scenarios.xml
[3/8/2019 5:14:08 AM] Information : Loading the check XML from path : C:\Users\labuser.CONTOSO\AppData\Local\Temp\DPM6EC.tmp\DPM2012\Setup\scenarios.xml
[3/8/2019 5:14:08 AM] Information : Getting scenarios for the product: DPM
[3/8/2019 5:14:08 AM] Information : Getting scenarios for DPM
[3/8/2019 5:14:08 AM] Information : Getting scenario for Mode:Install, DbLocation:Remote, SKU:Retail and CCMode:NotApplicable
[3/8/2019 5:14:08 AM] *** Error : Initialize the SQLSetUpHelper Object
[3/8/2019 5:14:08 AM] Information : [SQLSetupHelper.GetWMIReportingNamespace]. Reporting Namespace found. Reporting Namespace : V13
[3/8/2019 5:14:08 AM] Information : [SQLSetupHelper.GetWMISqlServerNamespace]. SQL Namespace found. SQL Namespace : \\DPM-SERVER\root\Microsoft\SqlServer\ComputerManagement13
[3/8/2019 5:14:08 AM] Information : Query WMI provider for SQL Server 2008.
[3/8/2019 5:14:08 AM] Information : Querying WMI Namespace: \\DPM-SERVER\root\Microsoft\SqlServer\ComputerManagement13 for query: Select * from SqlServiceAdvancedProperty where ServiceName='MSSQL$DPM' and PropertyName='Version'
[3/8/2019 5:14:08 AM] Information : SQL Server 2008 R2 SP2 instance DPM is present on this system.
[3/8/2019 5:14:08 AM] Information : Query WMI provider for SQL Server 2008.
[3/8/2019 5:14:08 AM] Information : Querying WMI Namespace: \\DPM-SERVER\root\Microsoft\SqlServer\ComputerManagement13 for query: Select * from SqlServiceAdvancedProperty where ServiceName='MSSQL$DPM' and PropertyName='Version'
[3/8/2019 5:14:08 AM] Information : [SQLSetupHelper.GetSQLDepedency]. Reporting Namespace and SQL namespace for installed SQL server which will be used as DPM DB. Reporting Namespace : \\DPM-SERVER\root\Microsoft\SqlServer\ReportServer\RS_DPM\V13\admin SQL Namespace : \\DPM-SERVER\root\Microsoft\SqlServer\ComputerManagement13
[3/8/2019 5:14:08 AM] Information : Check if SQL Server 2012 Service Pack 1 Tools is installed.
[3/8/2019 5:14:08 AM] Information : [SQLSetupHelper.GetSqlSetupRegKeyPath]. Registry Key path that contains SQL tools location: Software\Microsoft\Microsoft SQL Server\140\Tools\Setup\
[3/8/2019 5:14:08 AM] Information : Inspect.CheckSqlServerTools : MsiQueryProductState returned : INSTALLSTATE_DEFAULT
[3/8/2019 5:14:08 AM] *** Error : CurrentDomain_UnhandledException

Looking at the above log, the last line hints we are looking for SQL Server Tools (in this case, what looks like some crazy old hints to depencies on SQL Server 2012). Unfortunately, installation of SQL Server 2016 will provide you the recommendation to grab SQL Server Management Studio 17.X, however DPM 2016 will only install with SQL Server Management Studio 16.5.X. You will need to uninstall the 17.X version of SSMS and install the 16.5.X build from the link below:
https://docs.microsoft.com/en-us/sql/ssms/sql-server-management-studio-changelog-ssms?view=sql-server-2017#download-ssms-1653

Alas! Upon installation and run through the DPM installation, no more Error 4387! Once DPM is installed, you can safely upgrade your SQL Server instance to 2017 if needed.

Hope this helps someone else! DPM can be picky and unforgiving in nature, but if you abide by exactly what their documentation calls out to a T and not venture anything outside of those parameters, you should be golden 🙂

https://docs.microsoft.com/en-us/system-center/dpm/install-dpm?view=sc-dpm-2016#setup-prerequisites

Closing notes, if the above items didn’t solve your problem. Please post your logs and let’s troubleshoot to document all solutions needed for all error logs. Thank you!

Deploying Palo Alto VM-Series on Azure

Here is a recap of some of the reflections I have with deploying Palo Alto’s VM-Series Virtual Appliance on Azure. This is more of a reflection of the steps I took rather than a guide, but you can use the information below as you see fit.  At a high level, you will need to deploy the device on Azure and then configure the internal “guts” of the Palo Alto to allow it to route traffic properly on your Virtual Network (VNet) in Azure.  The steps outlined should work for both the 8.0 and 8.1 versions of the Palo Alto VM-Series appliance.

Please note, this tutorial also assumes you are looking to deploy a scale-out architecture.  This can help ensure a single instance doesn’t get overwhelmed with the amount of bandwidth you are trying to push through it.  If you are looking for a single instance, you can still follow along.

Deploy the Appliance in Azure

In deploying the Virtual Palo Altos, the documentation recommends to create them via the Azure Marketplace (which can be found here: https://azuremarketplace.microsoft.com/en-us/marketplace/apps/paloaltonetworks.vmseries-ngfw?tab=Overview).  Personally, I’m not a big fan of deploying the appliance this way as I don’t have as much control over naming conventions, don’t have the ability to deploy more than one appliance for scale, cannot specify my availability set, cannot leverage managed disks, etc.  In addition, I noticed a really strange error that if you specify a password greater than 31 characters, the Palo Alto devices flat out won’t deploy on Azure.  In this case, I’ve written a custom ARM template that leverages managed disks, availability sets, consistent naming nomenclature, proper VM sizing, and most importantly, let you define how many virtual instances you’d like to deploy for scaling.

Note: this article doesn’t cover the concept of using Panorama, but that would centrally manage each of the scale-out instances in a “single pane of glass”.  Below, we will cover setting up a node manually to get it working.  It is possible to create a base-line configuration file that joins Panorama post-deployment to bootstrap the nodes upon deployment of the ARM template.  The bootstrap file is not something I’ve incorporated into this template, but the template could easily be modified to do so.

With the above said, this article will cover what Palo Alto considers their Shared design model. Here is an example of what this visually looks like (taken from Palo Alto’s Reference Architecture document listed in the notes section at the bottom of this article):

Shared design model as per Palo Alto’s Reference Architecture

Below is a link to the ARM template I use.

PaloAlto-HA.json

Deployment of this template can be done by navigating to the Azure Portal (portal.azure.com), select Create a resource, type Template Deployment in the Azure Marketplace, click Create,  select Build your own template in the editor, and paste the code into the editor.

Alternatively, you can click this button here:

Here are some notes on what the parameters mean in the template:

VMsize: Per Palo Alto, the recommend VM sizes should be DS3, DS4, or DS5.  Documentation on this can be found here.

PASku: Here is where you can select to use bring-your-own-license or pay-as-you-go.  Plans are outlined here: https://azuremarketplace.microsoft.com/en-us/marketplace/apps/paloaltonetworks.vmseries-ngfw?tab=PlansAndPrice

PAVersion: The version of PanOS to deploy.

PACount: This defines how many virtual instances you want deployed and placed behind load balancers.

VNetName: The name of your virtual network you have created.

VNetRG: The name of the resource group your virtual network is in.  This may be the same as the Resource Group you are placing the Palos in, but this is a needed configurable option to prevent errors referencing a VNet in a different resource group.

envPrefix: All of the resources that get created (load balancer, virtual machines, public IPs, NICs, etc.) will use this naming nomenclature.

manPrivateIPPrefix, trustPrivateIPPrefix, untrustPrivateIPPrefix: Corresponding subnet address range.  These should be the first 3 octets of the range followed by a period.  For example, 10.5.6. would be a valid value.

manPrivateIPFirst, trustPrivateIPFirst, untrustPrivateIPFirst: The first usable IP address on the subnet specified.  For example, if my subnet is 10.4.255.0/24, I would need to specify 4 as my first usable address.

Username: this is the name of the privileged account that should be used to ssh and login to the PanOS web portal.

Password: Password to the privileged account used to ssh and login to the PanOS web portal.  Must be 31 characters or less due to Pan OS limitation.


Configure the Appliance

Once the virtual appliance has been deployed, we need to configure the Palo Alto device itself to enable connectivity on our Trust/Untrust interfaces.

Activate the licenses on the VM-Series firewall.

Follow these steps if using the BYOL version

  1. Create a Support Account.
  2. Register the VM-Series Firewall
    (with auth code)
    .
  3. On the firewall web interface, select Device tab -> Licenses
    and select Activate feature using authentication code.
  4. Enter the capacity auth-code that you registered on the support
    portal. The firewall will connect to the update server (updates.paloaltonetworks.com), and download
    the license and reboot automatically.  If this doesn’t work, please continue below to configuring the interfaces of the device.
  5. Log back in to the web interface after reboot and confirm the following on the Dashboard:
    1. A valid serial number displays in Serial#.
      If the term Unknown displays, it means the device is not licensed. To view
      traffic logs on the firewall, you must install a valid capacity license.
    2. The VM Mode displays as Microsoft Azure.

Follow these steps if using the PAYG (Pay as you go) version

  1. Create a Support Account.
  2. Register the Usage-Based Model of
    the VM-Series Firewall in AWS and Azure (no auth code)
    .

Configure the Untrust/Trust interfaces

Configure the Untrust interface

  1. Select Network-> Interfaces ->Ethernet-> select the link for ethernet1/1 and configure as follows:
    1. Interface Type: Layer3 (default).
    2. On the Config tab, assign the interface to the Untrust-VR router.
    3. On the Config tab, expand the Security Zone drop-down and select New Zone. Define a new zone called Untrust, and then click OK.
  2. On the IPv4 tab, select DHCP Client if you plan to assign only one IP address on the interface. If you plan to assign more than one IP address select Static and manually enter the primary and secondary IP addresses assigned to the interface on the Azure portal.  The private IP address of the interface can be found by navigating to Virtual Machines -> YOURPALOMACHINE -> Networking and using the Private IP address specified on each tab.
    1. Note: Do not use the Public IP address to the Virtual Machine.  Azure automatically DNATs traffic to your private address so you will need to use the Private IP Address for your UnTrust interface.
  3. Clear the Automatically create default route to default gateway provided by server check box.
    1. Note: Disabling this option ensures that traffic handled by this interface does not flow directly to the default gateway in the VNet.
  4. Click OK

Note: For the untrust interface, within your Azure environment ensure you have a NSG associated to the untrust subnet or individual firewall interfaces as the template doesn’t deploy this for you (I could add this in, but if you already had an NSG I don’t want to overwrite it). As per Azure Load Balancer’s documentation, you will need an NSG associated to the NICs or subnet to allow traffic in from the internet.

Configure the Trust Interface

  1. Select Network-> Interfaces ->Ethernet-> select the link for ethernet1/2 and configure as follows:
    1. Interface Type: Layer3 (default).
    2. On the Config tab, assign the interface to the Trust-VR router.
    3. On the Config tab, expand the Security Zone drop-down and select New Zone. Define a new zone called Trust, and then click OK.
  2. On the IPv4 tab, select DHCP Client if you plan to assign only one IP address on the interface. If you plan to assign more than one IP address select Static and manually enter the primary and secondary IP addresses assigned to the interface on the Azure portal. The private IP address of the interface can be found by navigating to Virtual Machines -> YOURPALOMACHINE -> Networking and using the Private IP address specified on each tab.
    1. Clear the Automatically create default route to default gateway provided by server check box.
      1. Note: Disabling this option ensures that traffic handled by this interface does not flow directly to the default gateway in the VNet.
  3. Click OK

Click Commit in the top right.  Verify that the link state for the interfaces is up (the interfaces should turn green in the Palo Alto user interface).

Define Static Routes

The Palo Alto will need to understand how to route traffic to the internet and how to route traffic to your subnets.  As you will see in this section, we will need two separate virtual routers to help handle the processing of health probes submitted from each of the Azure Load Balancers.

Create a new Virtual Router and  Static Route to the internet

  1. Select Network -> Virtual Router
  2. Click Add at the bottom
  3. Set the Name to Untrust-VR
  4. Select Static Routes -> IPv4 -> Add
  5. Create a Static Route to egress internet traffic
    1. Name: Internet
    2. Destination: 0.0.0.0/0
    3. Interface: ethernet 1/1
    4. Next Hop: IP Address
    5. IP Address: Use the IP address of the default gateway of your subnet the Untrust interface is deployed on
      1. Note: To find this, navigate to the Azure Portal (portal.azure.com) and select All Services -> Virtual Networks -> Your Virtual Network -> Subnets and use the first IP address of your subnet the untrust interface is on.  For example, is the address range of my subnet is 10.5.15.0/24, I would use 10.5.15.1 as my IP address.  If my subnet was 10.5.15.128/25, I would use 129 10.5.15.129 as my IP address
  6. Create a Static Route to move traffic from the internet to your trusted VR
    1. Name: Internal Routes
    2. Destination: your vnet address space
    3. Interface: None
    4. Next Hop: Next VR
      1. Trust-VR
  7. Click OK

Create a new Virtual Router and Static Route to your Azure Subnets

  1. Select Network -> Virtual Router
  2. Click Add at the bottom
  3. Set the Name to Trust-VR
  4. Select Static Routes -> IPv4 -> Add
  5. Create a Static Route to send traffic to Azure from your Trusted interface
    1. Name: AzureVNet
    2. Destination: your vnet address space
    3. Interface: ethernet 1/2
    4. Next Hop: IP Address
    5. IP Address: Use the IP address of the default gateway of your subnet the Trust interface is deployed on
      1. Note: To find this, navigate to the Azure Portal (portal.azure.com) and select All Services -> Virtual Networks -> Your Virtual Network -> Subnets and use the first IP address of your subnet the trust interface is on.  For example, if the address range of my subnet is 10.5.15.0/24, I would use 10.5.15.1 as my IP address.  If my subnet was 10.5.15.128/25, I would use 129 10.5.15.129 as my IP address
  6. Create a Static Route to move internet traffic received on Trust to your Untrust Virtual Router
    1. Name: Internet
    2. Destination: 0.0.0.0/0
    3. Interface: None
    4. Next Hop: Next VR
      1. Untrust-VR
  7. Click OK

Click Commit in the top right.

Configure Health Probes for Azure Load Balancers

If deploying the Scale-Out scenario, you will need to approve TCP probes from 168.63.129.16, which is the IP address of the Azure Load Balancer.  Azure health probes come from a specific IP address (168.63.129.16).  In this case, we need a static route to allow the response back to the load balancer.   For the purpose of this article, we will configure SSH on the Trust interface strictly for the Azure Load Balancer to contact to validate the Palo Alto instances are healthy.

Configure Palo Alto SSH Service for the interfaces

First we need to create an Interface Management Profile

  1. Select Network -> Network Profiles -> Interface Mgmt
  2. Click Add in the button left
  3. Use the following configuration
    1. Name: SSH-MP
    2. Administrative Management Services: SSH
    3. Permitted IP Addresses: 168.63.129.16/32
  4. Click OK

Next, we need to assign the profile to the Trust interface

  1. Select Network -> Interfaces ->select the link for ethernet1/2
  2. Select the Advanced tab
  3. Set the Management Profile to SSH-MP
  4. Click OK

Next, we need to assign the profile to the Untrust interface

  1. Select Network -> Interfaces ->select the link for ethernet1/1
  2. Select the Advanced tab
  3. Set the Management Profile to SSH-MP
  4. Click OK

Create a Static Route for the Azure Load Balancer Health Probes on the Untrust Interface

Next we need to tell the health probes to flow out of the Untrust interface due to our 0.0.0.0/0 rule.

  1. Select Network -> Virtual Router -> Untrust-VR
  2. Select Static Routes -> IPv4 -> Add
  3. Use the following configuration
    1. Name: AzureLBHealthProbe
    2. Destination: 168.63.129.16/32
    3. Interface: ethernet 1/1
    4. Next Hop: IP Address
    5. IP Address: Use the IP address of the default gateway of your subnet the Trust interface is deployed on
      1. Note: To find this, navigate to the Azure Portal (portal.azure.com) and select All Services -> Virtual Networks -> Your Virtual Network -> Subnets and use the first IP address of your subnet the trust interface is on.  For example, if the address range of my subnet is 10.5.15.0/24, I would use 10.5.15.1 as my IP address.  If my subnet was 10.5.15.128/25, I would use 129 10.5.15.129 as my IP address
  4. Click OK

Create a Static Route for the Azure Load Balancer Health Probes on the Trust Interface

Next we need to tell the health probes to flow out of the Trust interface due to our 0.0.0.0/0 rule.

  1. Select Network -> Virtual Router -> Trust-VR
  2. Select Static Routes -> IPv4 -> Add
  3. Use the following configuration
    1. Name: AzureLBHealthProbe
    2. Destination: 168.63.129.16/32
    3. Interface: ethernet 1/2
    4. Next Hop: IP Address
    5. IP Address: Use the IP address of the default gateway of your subnet the Trust interface is deployed on
      1. Note: To find this, navigate to the Azure Portal (portal.azure.com) and select All Services -> Virtual Networks -> Your Virtual Network -> Subnets and use the first IP address of your subnet the trust interface is on.  For example, if the address range of my subnet is 10.5.15.0/24, I would use 10.5.15.1 as my IP address.  If my subnet was 10.5.15.128/25, I would use 129 10.5.15.129 as my IP address
  4. Click OK

Click Commit in the top right.

Create a NAT rule for internal traffic destined to the internet

You will need to NAT all egress traffic destined to the internet via the address of the Untrust interface, so return traffic from the Internet comes back through the Untrust interface of the device.

  1. Navigate to Policies -> NAT
  2. Click Add
  3. On the General tab use the following configuration
    1. Name: UntrustToInternet
    2. Description: Rule to NAT all trusted traffic destined to the Internet to the Untrust interface
  4. On the Original Packet tab use the following configuration
    1. Source Zone: Click Add and select Trust
    2. Destination Zone: Untrust
    3. Destination Interface: ethernet 1/1
    4. Service: Check Any
    5. Source Address: Click Add, use the Internal Address space of your Trust zones
    6. Destination address: Check Any
  5. On the Translated Packet tab use the following configuration
    1. Translation Type: Dynamic IP and Port
    2. Address Type: Interface Address
    3. Interface: ethernet 1/1
    4.  IP Address: None
    5. Destination Address Translation Translation Type: None
  6. Click OK

Click Commit in the top right.

Update your Palo Alto appliance

By default, Palo Alto deploys 8.0.0 for the 8.0.X series and 8.1.0 for the 8.1.X series.  In this case, Palo Alto will strongly recommend you upgrade the appliance to the latest version of that series before helping you with support cases.

To do this, go to Device -> Dynamic Updates -> click Check Now in the bottom left and download the latest build from the list of available updates.

Please note: the update process will require a reboot of the device and can take 20 minutes or so.

Summary

At this point you should have a working scaled out Palo Alto deployment.  If all went well, I would recommend removing the public IP to the management interface or at least scoping it down to the single public IP address you are coming from.  You can find your public IP address by navigating here: https://jackstromberg.com/whats-my-ip-address/

References

Official documentation from Palo Alto on deploying the VM-Series on Azure (took me forever to find this and doesn’t cover setting up the static routes or updating the appliance): https://docs.paloaltonetworks.com/vm-series/8-1/vm-series-deployment/set-up-the-vm-series-firewall-on-azure/deploy-the-vm-series-firewall-on-azure-solution-template.html

Official documentation from Palo Alto on Azure VM Sizing: https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?id=kA10g000000ClD7CAK

Documentation on architecture for the VM-Series on Azure (click the little download button towards the top of the page to grab a copy of the PDF):  https://www.paloaltonetworks.com/resources/guides/azure-architecture-guide

Neat video created by Palo Alto outlining the architecture of a scale-out VM-Series deployment: https://www.paloaltonetworks.com/resources/videos/vm-series-in-azure

Using Terraform with Azure VM Extensions

TLDR: There are two sections of this article; feel free to scroll down to the titles for the applicable section.

Using VM Extensions with Terraform to Domain Join Virtual Machines

VM Extensions are a fanastic way to yield post deployment configurations via template as code in Azure.  One of Azure’s most common VM Extensions is the JoinADDomainExtension, which will join your Azure VM to an Active Directory machine after the machine has successfully been provisioned.

Typically, this can be configured via the following block of ARM Template code (a fully working example building the virtual and running the extension can be found here).

{
    "apiVersion": "2015-06-15",
    "type": "Microsoft.Compute/virtualMachines/extensions",
    "name": "MYADJOINEDVM/joindomain",
    "location": "EastUS",
    "properties": {
        "publisher": "Microsoft.Compute",
        "type": "JsonADDomainExtension",
        "typeHandlerVersion": "1.3",
        "autoUpgradeMinorVersion": true,
        "settings": {
            "Name": "JACKSTROMBERG.COM",
            "OUPath": "OU=Users,OU=CustomOU,DC=jackstromberg,DC=com",
            "User": "JACKSTROMBERG.COM\\jack",
            "Restart": "true",
            "Options": "3"
        },
        "protectedSettings": {
            "Password": "SecretPassword!"
        }
    }
}

When looking at Terraform, the syntax is a bit different and there isn’t much documentation on how to handle the settings and most importantly, the password/secret used when joining the machine to the domain.  In this case, here is working translation of the ARM template to Terraform.

resource "azurerm_virtual_machine_extension" "MYADJOINEDVMADDE" {
  name                 = "MYADJOINEDVMADDE"
  location             = "EastUS"
  resource_group_name  = "MyRG"
  virtual_machine_name = "MYADJOINEDVM"
  publisher            = "Microsoft.Compute"
  type                 = "JsonADDomainExtension"
  type_handler_version = "1.3"

  # What the settings mean: https://docs.microsoft.com/en-us/windows/desktop/api/lmjoin/nf-lmjoin-netjoindomain

  settings = <<SETTINGS
    {
        "Name": "JACKSTROMBERG.COM",
        "OUPath": "OU=Users,OU=CustomOU,DC=jackstromberg,DC=com",
        "User": "JACKSTROMBERG.COM\\jack",
        "Restart": "true",
        "Options": "3"
    }
SETTINGS
  protected_settings = <<PROTECTED_SETTINGS
    {
      "Password": "SecretPassword!"
    }
  PROTECTED_SETTINGS
  depends_on = ["azurerm_virtual_machine.MYADJOINEDVM"]
}

The key pieces here are the SETTINGS and PROTECTED_SETTINGS blocks that allow you to pass the traditional JSON attributes as you would in the ARM template.  Luckily, terraform does a somewhat decent job documentation this on their public docs here, so if you have any additional questions on any of the attributes you can find them all here: https://www.terraform.io/docs/providers/azurerm/r/virtual_machine_extension.html

The last block of code I have specified at the very end is a depends_on statement.  This simpy ensures that this resource is not created until the Virtual Machine itself has successfully been provisioned and can be very beneficial if you have other scripts that may need to run prior to domain join.

Using VM Extensions with Terraform to customize a machine post deployment

Continueing along the lines of customizing a virtual machine post deployment, Azure has a handy dany extension called CustomSriptExtension.  What this extension does is allow you to arbitrarily download and execute files (typically PowerShell) after a virtual machine has been deployed.  Unlike the domain join example above, Azure has extensive documentation on this extension and provides support for both Windows and Linux (click the links for Windows or Linux to see the Azure docs on this).

Following similar suite as the above Domain Join example, within the ARM world, we can leverage the following template to execute code post deployment:

{
    "apiVersion": "2018-06-01",
    "type": "Microsoft.Compute/virtualMachines/extensions",
    "name": "config-app",
    "location": "EastUS",
    "properties": {
        "publisher": "Microsoft.Compute",
        "type": "CustomScriptExtension",
        "typeHandlerVersion": "1.9",
        "autoUpgradeMinorVersion": true,
        "settings": {
            "fileUris": [
                "script location"
            ]
        },
        "protectedSettings": {
            "commandToExecute": "myExecutionCommand",
            "storageAccountName": "mystorageaccountname",
            "storageAccountKey": "myStorageAccountKey"
        }
    }
}

When we look at the translation over to Terraform, for the most part the structure is the exact same.  Similiar to our Acitve Directory Domain Join script above, the tricky piece is knowing to use the PROTECTED_SETTINGS to encapsulate our block of code that in this case authenticates to the Azure Storage Account to pull down our post-deployment script.  Now per the Azure documentation, those variables are optional; if the scripts you have don’t contain sensitive information, you are more than welcome to simply specify the fileUri and specify the commandToExecute via the regular SETTINGS block.

resource "azurerm_virtual_machine_extension" "MYADJOINEDVMCSE" {
  name                 = "MYADJOINEDVMCSE"
  location             = "EastUS"
  resource_group_name  = "MyRG"
  virtual_machine_name = "MYADJOINEDVM"
  publisher            = "Microsoft.Compute"
  type                 = "CustomScriptExtension"
  type_handler_version = "1.9"

  # CustomVMExtension Documetnation: https://docs.microsoft.com/en-us/azure/virtual-machines/extensions/custom-script-windows

  settings = <<SETTINGS
    {
        "fileUris": ["https://mystorageaccountname.blob.core.windows.net/postdeploystuff/post-deploy.ps1"]
    }
SETTINGS
  protected_settings = <<PROTECTED_SETTINGS
    {
      "commandToExecute": "powershell -ExecutionPolicy Unrestricted -File post-deploy.ps1",
      "storageAccountName": "mystorageaccountname",
      "storageAccountKey": "myStorageAccountKey"
    }
  PROTECTED_SETTINGS
  depends_on = ["azurerm_virtual_machine_extension.MYADJOINEDVMADDE"]
}

At this point you should be able to leverage both extensions to join a machine to the domain and then customize virtually any aspect of the machine thereafter.

The only thing I’ll leave you with is typically it is recommended to not leave clear-text passwords scattered through your templates.  In either case, I highly recommend looking at leveraging Azure Key Vault or an alternative solution that can ensure proper security in handling those secrets.

Notes

Aside from Terraform, one question I’ve received is what happens if the extension runs against a machine that is already domain joined?
A: The VM extension will still install against the Azure Virtual Machine, but will immediately return back the following response: “Join completed for Domain ‘yourdomain.com'”

Specifically, the following is returned back to Azure: [{“version”:”1″,”timestampUTC”:”2019-03-27T16:30:57.9274393Z”,”status”:{“name”:”ADDomainExtension”,”operation”:”Join Domain/Workgroup”,”status”:”success”,”code”:0,”formattedMessage”:{“lang”:”en-US”,”message”:”Join completed for Domain ‘yourdomain.com'”},”substatus”:null}}]