Install Mailtrain v1.24 on Ubuntu 20.04 Server Without Docker

This tutorial will show you how to install Mailtrain on Ubuntu 20.04 without Docker. Mailtrain is an open-source self-hosted newsletter application, an alternative to commercial email service providers like Mailchimp. You can use Mailtrain to send newsletters to your email subscribers via your own email server or by using any SMTP relay service (Mailjet, SendGrid, AmazonSES, Mailgun, etc).

Mailtrain is released under the terms of GPL v3.0 license, built on Node.js and MySQL/MariaDB. The latest version is v1.24.1, released on September 28, 2018. Features of Mailtrain are as follows:

• It allows you to easily manage large mailing lists (like 1 million subscribers).
• You can add subscribers manually, through the API, or import from a CSV file.
• It supports custom fields (text fields, numbers, drop-downs or check boxes), merge tags and custom forms.
• List segmentation.
• RSS campaign: auto-generate newsletter from RSS feed and send it to subscribers.
• Subscribers can upload their GPG public keys and Mailtrain will encrypt the newsletter for them.
• Allows you to check individual click statistics for every link.
• Advanced email template editors and HTML code editor.
• Automation: send specific emails when user activates your predefined trigger.
• You can create open email list (allow public subscription) and closed email list (subscribers are added to the list by admin).

Requirements

To run Mailtrain, you need a server with at least 1GB RAM. You can click this special link to get $50 free credit on DigitalOcean. (For new users only). If you are already a DigitalOcean user, then you can click this special link to get $50 free credit on Vultr (for new users only). Once you have an account at DigitalOcean or Vultr, install Ubuntu 20.04 on your server and follow the instructions below.

You also need a domain name. I registered my domain name at NameCheap because the price is low and they give whois privacy protection free for life.

Install Mailtrain on Ubuntu 20.04 without Docker

In a previous tutorial, I explained how to install Mailtrain using Docker. Docker is great for people who want to get an application up and running quickly, but it’s also very resource hungry. On a VPS with 1GB RAM that has Mailtrain installed using Docker, the MySQL container often shuts down because RAM is running out. So I’m going to show you a RAM-friendly way to install Mailtrain on Ubuntu 20.04, which can save about 400MB RAM.

Note: Mailtrain provides an install script, but it may be outdated and you will likely encounter errors when executing the install script. This tutorial explains each step so that you can understand how it works and get it installed.

Step 1: Install MariaDB Database Server

Your subscribers data will be stored in a database. Mailtrain supports MySQL and MariaDB. MariaDB is a drop-in replacement for MySQL. It is developed by former members of MySQL team who are concerned that Oracle might turn MySQL into a closed-source product. So let’s install the MariaDB database server.

Enter the following command to install it on Ubuntu 20.04.

sudo apt install mariadb-server mariadb-client

After it’s installed, MariaDB server should be automatically stared. Use systemctl to check its status.

systemctl status mariadb

Sample output:

● mariadb.service - MariaDB 10.3.22 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: e>
     Active: active (running) since Mon 2020-04-20 15:31:14 HKT; 52s ago
       Docs: man:mysqld(8)
             https://mariadb.com/kb/en/library/systemd/
   Main PID: 1826628 (mysqld)
     Status: "Taking your SQL requests now..."
      Tasks: 30 (limit: 9451)
     Memory: 73.0M
     CGroup: /system.slice/mariadb.service
             └─1826628 /usr/sbin/mysqld

If it’s not running, start it with this command:

sudo systemctl start mariadb

To enable MariaDB to automatically start at system boot time, run

sudo systemctl enable mariadb

Now run the post installation security script.

sudo mysql_secure_installation

When it asks you to enter MariaDB root password, press Enter key as the root password isn’t set yet. Then enter y to set the root password for MariaDB server.

Next, you can press Enter to answer all remaining questions, which will remove anonymous user, disable remote root login and remove test database. This step is a basic requirement for MariaDB database security. (Note that the letter Y is capitalized, which means it’s the default answer.)

Check MariaDB server version information.

mariadb --version

Output:

mariadb  Ver 15.1 Distrib 10.3.22-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2

Step 2: Create a Database and User for Mailtrain

Now we need to log in to MariaDB console and create a database and user for Mailtrain. By default, the MaraiDB package on Ubuntu uses unix_socket to authenticate user login, which basically means you can use username and password of the OS to log into MariaDB console. So you can run the following command to login without providing MariaDB root password.

sudo mariadb -u root

Create a database for Mailtrain using the following command. I named it mailtrain, but you can use whatever name you like. (Don’t leave out the semicolon.)

create database mailtrain;

Then enter the command below to create a database user for Mailtrain and grant all privileges of the mailtrain database to the user. Replace mtuser and your-password with your preferred username and password.

grant all privileges on mailtrain.* to mtuser@localhost identified by 'mtuser_password';

Next, create a user with read only access to mailtrain database. I named this user mt_readonly.

grant select on mailtrain.* TO mt_readonly@localhost identified by 'mt_readonly_password';

Flush the privileges table for the changes to take effect and then get out of MariaDB console.

flush privileges;

exit;

Step 3: Install Node.js

Mailtrain is built on Node.js, which is a JavaScript run-time environment that translates human-readable JavaScript code into machine code. So we need to install Node.js on Ubuntu 20.04 in order to run Mailtrain. Mailtrain requires Node.js 7 . The latest version of Node.js is v14. However, I don’t recommend using the latest version, because it’s not compatible with Mailtrain v1.24. For best compatibility, I recommend installing Node.js 8 from the Snap Store.

Install the snap daemon.

sudo apt install snapd

There are multiple versions of Node.js in the Snap Store, which can be seen with:

snap info node

Output:

channels:
  latest/stable:    –                                                     
  latest/candidate: –                                                     
  latest/beta:      –                                                     
  latest/edge:      15.0.0-nightly2020042524a4e615 2020-04-25 (2668) 30MB classic
  14/stable:        14.0.0                         2020-04-23 (2647) 30MB classic
  14/candidate:     ↑                                                     
  14/beta:          ↑                                                     
  14/edge:          ↑                                                     
  13/stable:        13.13.0                        2020-04-14 (2635) 29MB classic
  13/candidate:     ↑                                                     
  13/beta:          ↑                                                     
  13/edge:          ↑                                                     
  12/stable:        12.16.2                        2020-04-23 (2644) 21MB classic
  12/candidate:     ↑                                                     
  12/beta:          ↑                                                     
  12/edge:          ↑                                                     
  11/stable:        11.15.0                        2019-06-26 (2336) 19MB classic
  11/candidate:     ↑                                                     
  11/beta:          ↑                                                     
  11/edge:          ↑                                                     
  10/stable:        10.20.1                        2020-04-23 (2638) 20MB classic
  10/candidate:     ↑                                                     
  10/beta:          ↑                                                     
  10/edge:          ↑                                                     
  9/stable:         9.11.2                         2018-12-14 (1407) 17MB classic
  9/candidate:      ↑                                                     
  9/beta:           ↑                                                     
  9/edge:           ↑                                                     
  8/stable:         8.16.0                         2019-06-24 (2310) 16MB classic
  8/candidate:      ↑                                                     
  8/beta:           ↑                                                     
  8/edge:           ↑                                                     
  6/stable:         6.17.1                         2019-06-24 (2311) 13MB classic
  6/candidate:      ↑                                                     
  6/beta:           ↑                                                     
  6/edge:           ↑                                                                    

We can install Node.js from the 8/stable channel.

sudo snap install node --classic --channel=8/stable

To check your Node.js and npm version, run

node -v

npm -v

Output:

Note: If this is the first time you install a Snap package on the server, you need to log out and log back in, in order to use the snap version of Ruby.

To compile and install native addons from npm you also need to install build tools:

sudo apt install -y build-essential

Step 4: Run Mailtrain

Go to /var/www/ and fetch Mailtrain files from Github.

cd /var/www/

sudo git clone https://github.com/Mailtrain-org/mailtrain.git

Import the initial SQL data to the mailtrain database by using the following command. You need to enter the password of mtuser.

mariadb -u mtuser -p mailtrain < /var/www/mailtrain/setup/sql/mailtrain.sql

Create the production.toml configuration file.

sudo nano /var/www/mailtrain/config/production.toml

In this file, you can add configurations that override the default configurations in /var/www/mailtrain/config/default.toml file. We only need to add the following configurations.

user="mailtrain"
group="mailtrain"
[log]
level="error"
[www]
secret="Replace this with some random characters"
[mysql]
user="mtuser"
password="mtuser_password"
[queue]
processes=5

Save and close the file. Then create the mailtrain user and group for the mailtrain daemon to run as. Note that we don’t need to create password for this user, so we create a system user instead of a normal user.

sudo adduser --system --group mailtrain

Create a configuration file for the report worker.

sudo nano /var/www/mailtrain/workers/reports/config/production.toml

Add the following configurations. This report worker will have read-only access to mailtrain database.

[log]
level="error"
[mysql]
user="mt_readonly"
password="mt_readonly_password"

Save and close the file. Then change the permission of /var/www/mailtrain/.

sudo chown mailtrain:mailtrain /var/www/mailtrain/ -R

sudo chmod o-rwx /var/www/mailtrain/config/

Install Python2.7 and create a symbolic link.

sudo apt install python2.7

sudo ln -s /usr/bin/python2.7 /usr/bin/python

And install required node packages.

cd /var/www/mailtrain

sudo -u mailtrain npm config set scripts-prepend-node-path true

sudo -u mailtrain npm install --no-progress --production --unsafe-perm=true

Copy the Mailtrain systemd service unit file to /etc/systemd/system/ directory.

sudo cp /var/www/mailtrain/setup/mailtrain.service /etc/systemd/system/

Open this file.

sudo nano /etc/systemd/system/mailtrain.service

Change /opt/mailtrain to /var/www/mailtrain. And change /usr/bin/node to /snap/bin/node because we are using the Snap version of Node.js.

You may also want to comment out the following line. Because, with the following line enabled, if MySQL/MariaDB restarts, then the mailtrain.service will also restart, and if the MySQL/MariaDB server is being upgraded, then mailtrain.service will stop. I think there’s no need to require the mysql.service.

Requires=mysql.service

Save and close this file. Then start mailtrain.service.

sudo systemctl start mailtrain.service

Check its status. Make sure it is running.

systemctl status mailtrain.service

Output:

● mailtrain.service - Mailtrain server
     Loaded: loaded (/etc/systemd/system/mailtrain.service; disabled; vendor preset: enabled)
     Active: active (running) since Sun 2020-04-26 10:26:37 HKT; 11s ago
   Main PID: 3058360 (mailtrain)
      Tasks: 20 (limit: 9451)
     Memory: 228.6M
     CGroup: /system.slice/mailtrain.service
             ├─3058360 mailtrain
             └─3058396 /snap/node/2310/bin/node /var/www/mailtrain/services/sender.js

Enable auto-start at system boot time.

sudo systemctl enable mailtrain.service

Now that Mailtrain is running, you can access Mailtrain web interface via port 3000.

your-server-ip:3000

Step 5: Set up Reverse Proxy and Enabling HTTPS

Before using the Mailtrain web interface, let’s put it behind Nginx and then enable HTTPS. Install Nginx web server on Ubuntu 20.04 with:

sudo apt install nginx

Now you can copy the example mailtrain-nginx.conf file to /etc/nginx/conf.d/ directory.

sudo cp /var/www/mailtrain/setup/mailtrain-nginx.conf /etc/nginx/conf.d/

Open this file.

sudo nano /etc/nginx/conf.d/mailtrain-nginx.conf

Find the following line.

server_name mailtrain.org www.mailtrain.org;

Change the value of server_name parameter to your own domain name like newsletter.your-domain.com. Don’t forget to create DNS A record for this sub-domain.

server_name newsletter.linuxbabe.com;

Save and close the file. Then test Nginx configurations.

sudo nginx -t

If the test if successful, reload Nginx for the changes to take effect.

sudo systemctl reload nginx

You should now be able to access Mailtrain web interface via your sub-domain: newsletter.your-domain.com.

Now edit /var/www/mailtrain/config/production.toml file.

sudo nano /var/www/mailtrain/config/production.toml

In the [www] section, add the following two lines to indicate that mailtrain is behind the Nginx proxy and make it listen on local host address only.

host="127.0.0.1"
proxy=true

Save and close the file. Then restart Mailtrain for the change to take effect.

sudo systemctl restart mailtrain

Step 6: Enable HTTPS With Let’s Encrypt

Install Let’s Encrypt client (certbot) on your Ubuntu 20.04 server.

sudo apt install certbot python3-certbot-nginx

Then you can use the Nginx plugin to automatically obtain and install a TLS certificate by executing the following command.

sudo certbot --nginx --agree-tos --redirect --hsts --staple-ocsp --email [email protected] -d newsletter.your-domain.com

Now you can access Mailtrain web interface via your domain name and a secure HTTPS connection.

htop command tells me that my server is using about 500MB RAM. If I run Mailtrain with Docker, it’s about 900MB.

Login with username admin and password test. Then change your account email address and password. Go to settings page to change default configurations. You should change the service address from http://localhost:3000/ to your sub-domain (https://newsletter.your-domain.com).

Note: The admin account is an easy target for hackers. For best security, it’s recommended that you change the username from admin to something else. The Mailtrain web interface doesn’t provide such option, but you can change the username in MariaDB database using SQL command.

In the Mailer Settings, you can use SMTP if you have your own email server or use Amazon SES. Actually, you can also use other SMTP relay services in the SMTP tab. In the screenshot below, I use my own email server. Emails will be submitted on port 587 with STARTTLS encryption.

If Mailtrain is installed on your email server, then you should use the following SMTP settings. The hostname should be 127.0.0.1 and port should 25. There’s no need to enable encryption or authentication when talking to localhost.

Save your settings. Then you can create a list to test the functionalities of Mailtrain.

How to Enable Redis Cache

You can install Redis server for session cache. This will allow Mailtrain to run 5 processes instead of the default single process, which will speed up email delivery if you have lots of email subscribers (Note that this will increase the memory usage like about 250MB).

sudo apt install redis

After it’s installed, Redis should be automatically started. You can check its status with:

systemctl status redis

Sample output:

● redis-server.service - Advanced key-value store
     Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2020-04-26 11:02:03 HKT; 4min 23s ago
       Docs: http://redis.io/documentation,
             man:redis-server(1)
   Main PID: 3059455 (redis-server)
      Tasks: 4 (limit: 9451)
     Memory: 2.3M
     CGroup: /system.slice/redis-server.service
             └─3059455 /usr/bin/redis-server 127.0.0.1:6379

If it’s not running, you can start it with:

sudo systemctl start redis

Enable auto-start at boot time:

sudo systemctl enable redis

Next, edit Mailtrain configuration file.

sudo nano /var/www/mailtrain/config/production.toml

Add the following two lines at the end of the file to enable Redis.

[redis]
enabled=true

Save and close the file. Then restart Mailtrain.

sudo systemctl restart mailtrain

How to Handle Bounce Messages in Mailtrain

Sooner or later, your email list will contain addresses that you can’t send emails to. For example, when a subscriber who uses a company email address leaves the company, that email address will be deleted. So your email server will receive bounce message saying that the email can’t be delivered.

If you use an SMTP relay service to send email, they will handle bounce messages for you. If you use your own email server to send email, then you need to handle bounce messages in Mailtrain. Mailtrain offers two ways to handle bounced messages.

• via VERP
• via Postfix log

I personally use the VERP method, because it’s widely used in the email community and also because the second method causes high CPU usage on my server.

VERP Bounce Handling

With VERP (variable envelope return path), your email list manager uses unique envelope addresses for each subscriber. To enable VERP, edit the production.toml file.

sudo nano /var/www/mailtrain/config/production.toml

If your Mailtrain host has no SMTP server running, then add the following text.

[verp]
enabled=true
port=25
disablesenderheader=true

If your Mailtrain server has a SMTP server like Postfix running, then add the following text. The bounce handling server will listen on 127.0.0.1:2525.

[verp] 
enabled=true 
port=2525 
host="127.0.0.1"
disablesenderheader=true

Save and close the file. Then restart Mailtrain for the changes to take effect.

sudo systemctl restart mailtrain

In Mailtrain web interface, go to Settings -> VERP Bounce Handlding. Check Use VERP to catch bounces. Save your settings.

Next, you need to create a MX record for the server hostname (bounces.your-domain.com), then add A record for this hostname, pointing to your Mailtrain host, so bounce messages can be sent to your Mailtrain host. Each subscriber in your list will have an unique envelope address like [email protected].

Note that if you deployed DMARC record for your domain name, then the SPF alignment must be set to relaxed mode. If it’s set to strict mode, then your newsletter could fail DMARC check.

If the Mailtrain bounce handling server is listening on port 2525 of 127.0.0.1 and Postfix SMTP server is listening on port 25 of the public IP address, then you need to set up transport map so that Postfix can relay the bounce message to Mailtrain. Create the transport map file.

sudo nano /etc/postfix/transport

Add the following line to this file. This tells Postfix to relay emails with address like [email protected] to the Mailtrain bounce handling server.

bounces.your-domain.com  smtp:[127.0.0.1]:2525

Save and close the file. Then generate the index file.

sudo postmap /etc/postfix/transport

Edit Postfix main configuration file.

sudo nano /etc/postfix/main.cf

Add the following line to the file.

transport_maps = hash:/etc/postfix/transport

Note: If you used iRedMail to set up your mail server, then the transport_maps parameter has some other values. You should append the value at the end like below.

transport_maps =
     proxy:mysql:/etc/postfix/mysql/transport_maps_user.cf
     proxy:mysql:/etc/postfix/mysql/transport_maps_maillist.cf
     proxy:mysql:/etc/postfix/mysql/transport_maps_domain.cf
     hash:/etc/postfix/transport

Save and close the file. Then restart Postfix for the change to take effect.

sudo systemctl restart postfix

Handling Bounced Messages via Postfix Log

This assumes that Mailtrain is installed on your email server.

Mailtrain is able to detect bounced messages from Postfix mail log by finding lines that contains status=bounced. First, you need to enable the Postfix log reading server. Edit the production.toml file.

sudo nano /var/www/mailtrain/config/production.toml

Add the following text.

[postfixbounce]
enabled=true

Save and close the file. Then restart Mailtrain for the changes to take effect.

sudo systemctl restart mailtrain

Now the Postfix log reading server is listening on 127.0.0.1:5699. To let it read Postfix log, run the following command.

tail -F /var/log/mail.log | nc localhost 5699 -

To make it run in the background, you can create a systemd service unit.

sudo nano /etc/systemd/system/bouncetail.service

Add the following text.

[Unit]
Description=Postfix bounce notifier
After=mailtrain.service

[Service]
ExecStart=/bin/sh -c '/usr/bin/tail -F /var/log/mail.log | nc localhost 5699 -'
Type=simple
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Save and close the file. Then start this service and enable auto-start at boot time.

sudo systemctl start bouncetail

sudo systemctl enable bouncetail

Be sure to restart bouncetail service after you restart mailtrain service. I found the bouncetail service causes high CPU usage on my server, so I don’t use this method.

Update

After using Mailtrain on my own server for some time, I found that they are other reasons that cause email bounce. The following are soft bounces.

• The recipient’s mailbox is full.
• Your IP address is on a blacklist.
• Recipient email server is down or offline.

There’s also hard bounce, i.e. the recipient’s email address doesn’t exist. By default, Mailtrain unsubscribes an email address if one bounce is detected, which is fine for hard bounce, but it’s a good idea to allow several soft bounces before you unsubscribe the bad email address. You can ask the receiving email server to remove your IP address from blacklist and then resubscribe the related email address.

How to Insert a Sign up Form on Your Website

First, you need to enable cross-origin resource sharing. Edit the production.toml file.

sudo nano /var/www/mailtrain/config/production.toml

Add the following line in the file to whitelist your website.

[cors]
# Allow subscription widgets to be embedded
origins=['https://www.example.com']

Save and close the file. Then restart Mailtrain for the change to take effect.

sudo systemctl restart mailtrain

Next, add the following code on your website to display the sign up form. Replace the red colored text with the Mailtrain domain name and your own list ID.

<div data-mailtrain-subscription-widget data-url="https://newsletter.example.com/subscription/8ljDT9KMGr/widget">
     <a href="https://newsletter.example.com/subscription/8ljDT9KMGr">Subscribe to our list