Production Transform Server

Modified on: Fri, 12 Jun, 2020 at 3:30 PM

Overview

The instructions in this section serves as an example only. Your specific environment and requirements might require another solution. This article provides instructions for setting up a transform host server with an Apache-based web server or with Python’s Gunicorn.


Apache Server

This basic setup guide was tested on Ubuntu 18.04 LTS using Apache2 and Python 3.


Note: This is only an example and requires additional design and configuration to serve as a secure and reliable production Transform server.


Install system and Python dependencies:

Refresh APT repositories:


$ sudo apt update


Install Apache2:


$ sudo apt install apache2 -y


Install WSGI Apache module:

For Python3


$ sudo apt install libapache2-mod-wsgi-py3 -y


Install Python3:


$ sudo apt install python3 -y


Install Python3 pip:


$ sudo apt install python3-pip -y


Configure Python3 and pip3 as the default for your system:


$ echo "alias python=python3" >> ~/.bash_aliases
$ echo "alias pip=pip3" >> ~/.bash_aliases
$ source ~/.bash_aliases


Confirm if the Apache2 server is running:


$ sudo systemctl status apache2


Or


$ sudo service apache2 status


Note: If you enabled a firewall on your server, you will be required to open the applicable ports before using the webserver.


Install Maltego Python Transform Library (Maltego-TRX):


$ sudo -H pip3 install maltego-trx


Prepare the Transform project

In this basic setup guide, it is assumed that the Transform project was created in the following folder:


$ cd /var/www/

$ sudo maltego-trx start TRX


This will create the Transform project folder "/var/www/TRX" with the recommend subfolder structure.


The Apache2 web server will run as the “www-data” user. Set the “www-data” user as the owner of the Transform project folder. Use the following command to change folder ownership:


$ sudo chown -R www-data:www-data /var/www/TRX/


Follow the instructions provided in the Transform writing guide to create and code Transforms. In this basic setup guide, it is assumed that an existing Transform project is ready to be hosted.


Configure Apache2

Various configuration files are required for Apache2 to serve the Transform project and act as a Transform Server.

Apache2 will run a WSGI server from the project file, e. g. "/var/www/TRX/project.py", listing on port "8080".

Create an apache configuration file, e.g. "TRX.conf”, and copy it to the Apache2 “sites-available” folder:


$ sudo vi TRX.conf

$ sudo cp ./TRX.conf /etc/apache2/sites-available/TRX.conf


The basic content of the configuration file should be:


<VirtualHost *:8080>
WSGIDaemonProcess TRX user=www-data group=www-data threads=25 python-path=/var/www/TRX/
WSGIScriptAlias / /var/www/TRX/project.py
      <Directory /var/www/TRX>
        WSGIProcessGroup TRX
          WSGIApplicationGroup %{GLOBAL}
          Order deny,allow
          Allow from all
      </Directory>
</VirtualHost>


Please take note that the “python-path” parameter must point to the Transform project folder, e.g. “/var/www/TRX/”.


Activate the Transform configuration file, e.g. “TRX.conf”:


$ sudo a2ensite TRX


Edit the Apache2 ports file, e.g. “/etc/apache2/ports.conf”, and add “Listen 8080” below the last “Listen” line:


$ sudo nano /etc/apache2/ports.conf
...
NameVirtualHost *:80
Listen 80
Listen 8080
...


Apply these configuration changes by restarting or reloading Apache2

To restart Apache2, use the following command:


$ sudo /etc/init.d/apache2 restart


Or to reload Apache2, use the following command:


$ sudo service apache2 reload


Adjust the ‘TRX’ site configuration defined in /etc/apache2/sites-available to meet your requirements. The presented configuration will route all traffic on port 8080 to the WSGI script TRX.wsgi located in /var/www/TRX.


Test your configuration

Test the configuration by browsing to the following URL:


http://<your server name or IP>:8080/

* Update <your server name or IP> with appropriate values.


A valid and correct configuration will return the following response:


You have reached a Maltego Transform Server.


Retrieve the Transform links by running the following command from the Transform project folder:


$ cd /var/www/TRX/

$ python project.py list


The output of the “list” command will contain Transform server URLs:


= Transform Server URLs =
/run/dnstoip/: DNSToIP
/run/greetperson/: GreetPerson


Test the Transform server URLs by concatenating them to the Transform server URL:


http://<your server name or IP>:8080/run/dnstoip/

* Update <your server name or IP> with appropriate values.


A valid and correct configuration will return the following response:


Transform found with name 'dnstoip', you will need to send a POST request to run it.


Gunicorn Server

The following section covers how to use Gunicorn as a transform host rather than Apache2 as explained above. This basic setup guide was tested on Ubuntu 18.04 LTS using Gunicorn and Python 3.


Note: This is only an example and requires additional design and configuration to serve as a secure and reliable production Transform Server. Generally Gunicorn runs behind a reverse proxy such as NGINX. Configuring NIGNX is not explained in this guide.


Gunicorn is only supported on Python 3.


Install system and Python dependencies:

Refresh APT repositories:


$ sudo apt update


Install Python3:


$ sudo apt install python3 -y


Install Python3 pip:

Configure Python3 and pip as the default for your system:


$ echo "alias python=python3" >> ~/.bash_aliases
$ echo "alias pip=pip3" >> ~/.bash_aliases
$ source ~/.bash_aliases


Install Maltego Python Transform Library:


$ sudo -H pip3 install maltego-trx


Install Gunicorn:


$ sudo -H pip3 install gunicorn


Prepare the Transform project. In this basic setup guide, it is assumed that the Transform project was created in the following folder:


$ sudo mkdir -p /var/www/    

$ cd /var/www/

$ sudo maltego-trx start TRX


This will create the Transform project folder "/var/www/TRX" with the recommend subfolder structure.


The Gunicorn web server will run as the “www-data” user. Set the “www-data” user as the owner of Transform project folder. Use the following command to change folder ownership:


$ sudo chown -R www-data:www-data /var/www/TRX/


Configure Gunicorn     

It is possible to run Gunicorn once-off from the command line to test basic functionality. Change to the Transform project folder and execute the following command:


$ cd /var/www/TRX/

$ gunicorn --bind=0.0.0.0:8081 --threads=25 --workers=2 project:app


The Gunicorn process will start and bind to all host IP addresses on port 8081.

Use control + z to terminate Gunicorn process.


To run Gunicorn as a persistent service, create the service manually as follows:


$ sudo nano /etc/systemd/system/TRX.service


The basic content of the service configuration file should be:


[Unit]
Description=Gunicorn instance to serve as transform server
After=network.target

[Service]
User=www-data
Group=www-data
WorkingDirectory=/var/www/TRX/
ExecStart= /usr/bin/python3 /usr/local/bin/gunicorn --bind=0.0.0.0:8081 --threads=25 --workers=2 project:app

[Install]
WantedBy=multi-user.target


Note that the "ExecStart" requires the absolute paths for the python and Gunicorn installations. Use the following command to determine the absolute path to python and Gunicorn:


$ which python3

$ which gunicorn


Note: In this case ‘python3’ must be specified.


Note: If you tested from the command line, please ensure that all running instances are stopped by using the following command:


$ ps aux | grep gunicorn | awk '{print $2;}' | xargs kill -9 2


Start and persist the Gunicorn service as follows:


$ sudo systemctl start TRX

$ sudo systemctl enable TRX


Confirm the status of the service as follows:


$ sudo systemctl status TRX


Test your configuration

Test the configuration by browsing to the following URL:


http://<your server name or IP>:8081/

* Update <your server name or IP> with appropriate values.


A valid and correct configuration will return the following response:


You have reached a Maltego Transform Server.


Retrieve the Transform links by running the following command from the Transform project folder:


$ cd /var/www/TRX/

$ python project.py list


The output of the “list” command will contain Transform server URLs:


= Transform Server URLs =
/run/dnstoip/: DNSToIP
/run/greetperson/: GreetPerson


Test the Transform server URLs by concatenating them to the Transform server URL:


http://<your server name or IP>:8081/run/dnstoip/

* Update <your server name or IP> with appropriate values.


A valid and correct configuration will return the following response:


Transform found with name 'dnstoip', you will need to send a POST request to run it.


Did you find it helpful? Yes No

Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.