Malcolm is a simple, easy-to-use, and powerful network traffic analysis tool. It is capable of capturing artifacts (PCAP files) and Zeek logs. Malcolm provides an interconnected framework that makes it greater than the sum of its parts. Malcolm’s easy, flexible deployment and robust combination of tools fill a void in the network security space and make advanced network traffic analysis accessible to many in both the public and private sectors.
In this post, we will show you how to install Malcolm network traffic analysis tool on Ubuntu 22.04.
Prerequisites
- A server running Ubuntu 22.04.
- Minimum 16 GB RAM and 4 CPU core.
- A root password is configured on the server.
Create a Malcolm System User
First, you will need to create a dedicated user account to run Malcolm. You can create it with the following command:
useradd -m -d /opt/malcolm -s /bin/bash -G sudo malcolm
Next, set the user password with the following command:
passwd malcolm
Next, check the UID of the Malcolm user with the following command:
id malcolm
You should see the following output:
uid=1000(malcolm) gid=1000(malcolm) groups=1000(malcolm),27(sudo)
Install Malcolm Ubuntu 22.04
First, switch the user to Malcolm and download the latest version of Malcolm using the following command:
su - malcolm
git clone https://github.com/idaholab/Malcolm
Once the download is completed, change the directory to the downloaded directory and start installing Malcolm using the following command:
cd Malcolm
sudo ./scripts/install.py
During the installation, you will be asked several questions, as shown below:
Installing required packages: ['apache2-utils', 'make', 'openssl', 'python3-dialog'] "docker info" failed, attempt to install Docker? (Y/n): Y Attempt to install Docker using official repositories? (Y/n): Y Installing required packages: ['apt-transport-https', 'ca-certificates', 'curl', 'gnupg-agent', 'software-properties-common'] Installing docker packages: ['docker-ce', 'docker-ce-cli', 'containerd.io'] Installation of docker packages apparently succeeded Add a non-root user to the "docker" group?: Y Enter user account: malcolm Add another non-root user to the "docker" group?: n "docker-compose version" failed, attempt to install docker-compose? (Y/n): Y Install docker-compose directly from docker github? (Y/n): Y Download and installation of docker-compose apparently succeeded fs.file-max increases allowed maximum for file handles fs.file-max= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y fs.inotify.max_user_watches increases allowed maximum for monitored files fs.inotify.max_user_watches= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y fs.inotify.max_queued_events increases queue size for monitored files fs.inotify.max_queued_events= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y fs.inotify.max_user_instances increases allowed maximum monitor file watchers fs.inotify.max_user_instances= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y vm.max_map_count increases allowed maximum for memory segments vm.max_map_count= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y net.core.somaxconn increases allowed maximum for socket connections net.core.somaxconn= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y vm.dirty_background_ratio defines the percentage of system memory fillable with "dirty" pages before flushing vm.dirty_background_ratio= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y vm.dirty_ratio defines the maximum percentage of dirty system memory before committing everything vm.dirty_ratio= appears to be missing from /etc/sysctl.conf, append it? (Y/n): Y /etc/security/limits.d/limits.conf increases the allowed maximums for file handles and memlocked segments /etc/security/limits.d/limits.conf does not exist, create it? (Y/n): Y
Once the Malcolm is installed, you can proceed to the next step.
Configure Malcolm
After installing Malcolm, you will need to configure it using the following command:
sudo ./scripts/install.py --configure
During the Malcolm configuration, you will be asked for several questions. Answer all questions as shown below:
- Malcolm processes will run as UID 1000 and GID 1000. Is this OK? (Y/n): Y
- Setting 10g for OpenSearch and 3g for Logstash. Is this OK? yes
- Setting 3 workers for Logstash pipelines. Is this OK? (Y/n): yes
- Restart Malcolm upon system or Docker daemon restart? Yes Then, Choose the default option, unless-stopped.
- Choose whether to setup Malcolm with HTTPS: Yes
- Choose whether Malcolm will run behind any proxy: No
- Choose networking: Just enter to accept the defaults. Choose LDAP: No
- Store OpenSearch index snapshosts locally in /opt/malcolm/Malcom/opensearch-backup? Yes
- Choose to Compress OpenSearch index snapshots: Yes
- Whether you want to delete the oldest indices when the database exceeds a certain size: No
- Reverse DNS lookup locally for source and destination IP addresses in logs: No
- Hardware vendor OUI lookups for MAC addresses: Yes
- Perform string randomness scoring on some fields: yes
- Expose OpenSearch port to external hosts: no
- Expose Logstash port to external hosts: no
- Forward Logstash logs to external OpenSearch instance: no
- Expose Filebeat TCP port to external hosts: no
- Enable file extraction with Zeek: yes
- Choose interesting as the extraction behavior. Choose file preservation method: quarantine
- Scan extracted files/PE files with ClamAV: yes
- Scan extracted files/PE files with Yara: yes
- Scan extracted files/PE files with Capa: yes
- Lookup extracted file hashes with VirusTotal: no
- Download updated scanner signatures periodically: yes
- Should Malcolm capture network traffic to PCAP files: yes
- Specify capture interface(s) (comma-separated) on which Malcolm will use to network traffic: eth0
- Capture packets using netsniff-ng? (Y/n): yes
- Capture packets using tcpdump? (y/N): no
- Capture filter (tcpdump-like filter expression; leave blank to capture all traffic) (): You can disable traffic related to Elasticsearch (port 9200), Logstash (5044), Arkime(8005): not port 9200 and not port 5044 and not port 8005
- Disable capture interface hardware offloading and adjust ring buffer sizes? (y/N): n
Once the Malcolm has been configured, restart your system to apply the configuration.
sudo reboot
Create Malcolm Admin Account
Next, you will need to create an administrative account to access Malcolm web interface.
First, switch the user to Malcolm and navigate to the Malcolm directory using the following command:
su - malcolm
cd ~/Malcolm
Next, run the following command to create an admin account:
./scripts/auth_setup
Answer all questions as shown below to create an admin account:
- Store administrator username/password for local Malcolm access? yes
- Set administrator password and username.
- (Re)generate self-signed SSL certs for web traffic HTTPS: yes
- (Re)generate self-signed certificates for a remote log forwarder: yes
- Store username/password for forwarding Logstash events to a secondary, external OpenSearch instance: no. Store username/password for email alert sender account: no
Pull Malcolm Docker Images
Next, you will need to download all required Docker images from the Docker Hub registry. You can download all with the following command:
docker-compose pull
Once all the images are downloaded, verify all the images using the following command:
docker images
You should get the following output:
REPOSITORY TAG IMAGE ID CREATED SIZE malcolmnetsec/filebeat-oss 6.2.0 5e9fa4c8ea2d 11 days ago 648MB malcolmnetsec/arkime 6.2.0 4f4e6025c82d 11 days ago 793MB malcolmnetsec/zeek 6.2.0 5b117ad2b4bb 11 days ago 1.4GB malcolmnetsec/dashboards 6.2.0 9dcaff859eec 11 days ago 1.13GB malcolmnetsec/logstash-oss 6.2.0 cf4d75dcf4af 11 days ago 1.64GB malcolmnetsec/file-monitor 6.2.0 26227c3c7dc9 11 days ago 589MB malcolmnetsec/nginx-proxy 6.2.0 3b8b5413b52e 11 days ago 121MB malcolmnetsec/file-upload 6.2.0 2c704be24433 11 days ago 263MB malcolmnetsec/api 6.2.0 90f964b5d728 11 days ago 148MB malcolmnetsec/suricata 6.2.0 cbdb091d2df4 11 days ago 274MB malcolmnetsec/htadmin 6.2.0 2b55630700d1 11 days ago 242MB malcolmnetsec/opensearch 6.2.0 9ccb4665bd6c 11 days ago 1.25GB malcolmnetsec/pcap-monitor 6.2.0 d957d803bdbb 11 days ago 214MB malcolmnetsec/freq 6.2.0 3959a9daa952 11 days ago 131MB malcolmnetsec/dashboards-helper 6.2.0 7da9699a72b3 11 days ago 168MB malcolmnetsec/pcap-capture 6.2.0 df63e8daa369 11 days ago 121MB malcolmnetsec/name-map-ui 6.2.0 6bd6c7c58d36 11 days ago 120MB
Start Malcolm Service
At this point, all required components for Malcolm are ready. You can now start the Malcolm service using the following command:
./scripts/start
Wait for some time to start all services. Once all the services are started, verify all the running services using the following command:
docker ps -a
You should see all running containers in the following output:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 840ea2b0e9ad malcolmnetsec/nginx-proxy:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes (unhealthy) 0.0.0.0:443->443/tcp, 127.0.0.1:5601->5601/tcp, 0.0.0.0:488->488/tcp, 127.0.0.1:9200->9200/tcp malcolm-nginx-proxy-1 dd5c8c63816c malcolmnetsec/suricata:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes (unhealthy) malcolm-suricata-1 3112e1bd8f73 malcolmnetsec/filebeat-oss:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes (unhealthy) 127.0.0.1:5045->5045/tcp malcolm-filebeat-1 c93cfe93ad7e malcolmnetsec/file-upload:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes (unhealthy) 80/tcp, 127.0.0.1:8022->22/tcp malcolm-upload-1 18ee20b46f3c malcolmnetsec/dashboards:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes (unhealthy) 5601/tcp malcolm-dashboards-1 2c34206c06e4 malcolmnetsec/zeek:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes (unhealthy) malcolm-zeek-1 41103ef99ce1 malcolmnetsec/logstash-oss:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes (unhealthy) 9001/tcp, 127.0.0.1:5044->5044/tcp, 9600/tcp malcolm-logstash-1 0408f42a76c3 malcolmnetsec/dashboards-helper:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes (unhealthy) 28991/tcp malcolm-dashboards-helper-1 3e78024620de malcolmnetsec/arkime:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes (unhealthy) 8000/tcp, 8005/tcp, 8081/tcp malcolm-arkime-1 58cd869beced malcolmnetsec/pcap-monitor:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes (unhealthy) 30441/tcp malcolm-pcap-monitor-1 1040fa8bd6df malcolmnetsec/file-monitor:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes (unhealthy) 3310/tcp, 8440/tcp malcolm-file-monitor-1 25c83f14413d malcolmnetsec/zeek:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes malcolm-zeek-live-1 b321a96c0362 malcolmnetsec/api:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes (unhealthy) 5000/tcp malcolm-api-1 0f1f4ac023f9 malcolmnetsec/name-map-ui:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes (unhealthy) 8080/tcp malcolm-name-map-ui-1 ba4d553cf6b5 malcolmnetsec/suricata:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes malcolm-suricata-live-1 e4637d0ec04d malcolmnetsec/opensearch:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 13 minutes (health: starting) 9200/tcp, 9300/tcp, 9600/tcp, 9650/tcp malcolm-opensearch-1 ac002e31d9be malcolmnetsec/htadmin:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes (unhealthy) 80/tcp malcolm-htadmin-1 7223d5244a7b malcolmnetsec/pcap-capture:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes malcolm-pcap-capture-1 971931b21788 malcolmnetsec/freq:6.2.0 "https://www.howtoforge.com/usr/local/bin/dock…" 17 minutes ago Up 17 minutes (unhealthy) 10004/tcp malcolm-freq-1
You can also verify all listening ports using the following command:
ss -atlnp | grep -i docker
You should see the following output:
LISTEN 0 65535 127.0.0.1:5601 0.0.0.0:* users:(("docker-proxy",pid=7480,fd=4))
LISTEN 0 65535 0.0.0.0:488 0.0.0.0:* users:(("docker-proxy",pid=7519,fd=4))
LISTEN 0 65535 127.0.0.1:9200 0.0.0.0:* users:(("docker-proxy",pid=7443,fd=4))
LISTEN 0 65535 127.0.0.1:5044 0.0.0.0:* users:(("docker-proxy",pid=6247,fd=4))
LISTEN 0 65535 127.0.0.1:5045 0.0.0.0:* users:(("docker-proxy",pid=7063,fd=4))
LISTEN 0 65535 127.0.0.1:8022 0.0.0.0:* users:(("docker-proxy",pid=6826,fd=4))
LISTEN 0 65535 0.0.0.0:443 0.0.0.0:* users:(("docker-proxy",pid=7567,fd=4))
Access Malcolm
You can now access the Malcolm OpenSearch dashboard using the URL https://your-server-ip/dashboards/. You will be asked to provide your admin username and password:
Provide your admin username, password and click on the Sign in button. You should see the OpenSearch dashboard on the following screen:
To access the Malcolm Capture File and Log Archive Upload screen, type the URL https://your-server-ip/upload/.
To access the Host and subnet name mapping editor, type the URL https://your-server-ip/name-map-ui/.
To access the Account Management screen, use the URL https://your-server-ip:488/
Congratulations! you have successfully installed and configured Malcolm network traffic analysis tool on Ubuntu 22.04.
