Development and Custom Transforms

Production Transform Server using Gunicorn

Modified on: Wed, 10 May, 2023 at 12:42 PM

Overview

The instructions in this section serve as an example only. Your specific environment and requirements might require another solution. This article provides instructions for setting up a Transform host server with Python’s Gunicorn.

Gunicorn Server

The following section covers how to use Gunicorn as a Transform host server. This setup guide was tested on Ubuntu 18.04 LTS using Gunicorn and Python 3.

Note: This is only an example and requires additional design and configuration to serve as a secure and reliable production Transform Server. Generally Gunicorn runs behind a reverse proxy such as NGINX. Configuring NNX is not explained in this guide.

Note: Gunicorn is only supported on Python 3.

Install system and Python dependencies:

Refresh APT repositories:

sudo apt update

Install Python3:

 sudo apt install python3 -y

Install Python3 pip:

Configure Python3 and pip as the default for your system:

echo "alias python=python3" >> ~/.bash_aliases
echo "alias pip=pip3" >> ~/.bash_aliases
source ~/.bash_aliases

Install Maltego Python Transform Library:

sudo -H pip3 install maltego-trx

Install Gunicorn:

sudo -H pip3 install gunicorn

Prepare the Transform project. In this basic setup guide, it is assumed that the Transform project was created in the following folder:

$ sudo mkdir -p /var/www/    

$ cd /var/www/

$ sudo maltego-trx start TRX

This will create the Transform project folder "/var/www/TRX" with the recommend subfolder structure.

The Gunicorn web server will run as the “www-data” user. Set the “www-data” user as the owner of Transform project folder. Use the following command to change folder ownership:

$ sudo chown -R www-data:www-data /var/www/TRX/

Configure Gunicorn

It is possible to run Gunicorn once-off from the command line to test basic functionality. Change to the Transform project folder and execute the following command:

$ cd /var/www/TRX/

$ gunicorn --bind=0.0.0.0:8081 --threads=25 --workers=2 project:app

The Gunicorn process will start and bind to all host IP addresses on port 8081.

Use control + z to terminate Gunicorn process.

Note: The default (host) port can be configured here to use any other open port on which you prefer to expose the Transform.

To run Gunicorn as a persistent service, create the service manually as follows:

$ sudo nano /etc/systemd/system/TRX.service

The basic content of the service configuration file should be:

[Unit]
Description=Gunicorn instance to serve as transform server
After=network.target

[Service]
User=www-data
Group=www-data
WorkingDirectory=/var/www/TRX/
ExecStart= /usr/bin/python3 /usr/local/bin/gunicorn --bind=0.0.0.0:8081 --threads=25 --workers=2 project:app

[Install]
WantedBy=multi-user.target

Note that the "ExecStart" requires the absolute paths for the python and Gunicorn installations. Use the following command to determine the absolute path to python and Gunicorn:

$ which python3

$ which gunicorn

Note: In this case ‘python3’ must be specified.

Note: If you tested from the command line, please ensure that all running instances are stopped by using the following command:

$ ps aux | grep gunicorn | awk '{print $2;}' | xargs kill -9 2

Start and persist the Gunicorn service as follows:

$ sudo systemctl start TRX

$ sudo systemctl enable TRX

Confirm the status of the service as follows:

$ sudo systemctl status TRX

Test your configuration

Test the configuration by browsing to the following URL:

http://<your server name or IP>:8081/

* Update <your server name or IP> with appropriate values.

A valid and correct configuration will return the following response:

You have reached a Maltego Transform Server.

Retrieve the Transform links by running the following command from the Transform project folder:

$ cd /var/www/TRX/

$ python project.py list

The output of the “list” command will contain Transform server URLs:

= Transform Server URLs =
/run/dnstoip/: DNSToIP
/run/greetperson/: GreetPerson

Test the Transform server URLs by concatenating them to the Transform server URL:

http://<your server name or IP>:8081/run/dnstoip/