This article will walk you through setting up a Python Transform using Gunicorn and our Maltego-TRX library hosted on Github.
The Maltego-TRX library is the official Maltego Python library used for developing local and TDS Transforms. Maltego-TRX 1.3.2 supports both Python2.7 & Python3
In order to use Gunicorn we will need to run it using Python3. We will be installing our dependencies using the pip3 command, to ensure that the Python dependencies are installed for Python 3.
Provisioning the host
We will be using an Ubuntu 18.04 host machine for our Transform server.
To setup the server we need install the following dependencies.
Install System Dependencies
apt-get update apt-get install python3-pip -y
Install Python Dependencies
pip3 install maltego-trx gunicorn
Start a new project
We will need to create a project folder to hold our Transform code. In this example, we will use the project folder "/var/www/TRX/".
We can create a new project folder, with the recommend project structure using the following commands:
mkdir /var/www/ cd /var/www/ maltego-trx start TRX
This will create the project folder "/var/www/TRX" with the recommend directory structure.
The user that runs the production server will be www-data. We will need to make www-data the owner of our project folder. We can do this using the following command:
chown -R www-data:www-data /var/www/TRX
Navigate to project directory using this command and follow one of the methods given below to start the server:
cd /var/www/TRX
Start the Server
Development Server
If we are developing and debugging new Transforms then it is better to use a lightweight Transform server that restarts every time the code is changed.
Remember to stop the Gunicorn first to free up port 8080:
ps aux | grep gunicorn | awk '{print $2;}' | xargs kill -9 2To run the development server,navigate to "/var/www/TRX" and run the following command:
python3 project.py runserver
The server should start on default port 8080 and print an output similar to the output given below having details such as available Transforms and their names:
=== Maltego Transform Server: v1.3.3 === = Transform Server URLs = /run/dnstoip/: DNSToIP /run/greetperson/: GreetPerson = Local Transform Names = dnstoip: DNSToIP greetperson: GreetPerson
The default project setup contains the DNSToIP and GreetPerson Transforms which we can see are listed above.
To test that the server is running correctly, we can navigate to http://<server_host>:<port> in our browser, and we should receive the following response:
You have reached a Maltego Transform Server.
Production Server
Once you are finished developing and testing your Transforms, it is better to run a more robust setup using Gunicorn.
First, make sure the development server is no longer running. Use 'Ctrl + C' to stop.
Then start the Gunicorn daemon by running the following command, from your project directory ( /var/www/TRX ):
gunicorn --bind=0.0.0.0:8080 --threads=25 --workers=2 project:app
Again, we can check that the server is running by opening http://<server_host>:<port> in our browser.
Running Gunicorn using the above command, has two issues:
- The server process is running as the root user
- If the host machine is restarted, gunicorn will not start on boot
We can solve the above two issues by creating a systemctl service that will manage the gunicorn process for us.
Creating a systemctl service for Gunicorn
First we will need to create a file ending in .service within the /etc/systemd/system directory. We can do this using the following command:
nano /etc/systemd/system/TRX.service
In the file we can then enter the following configuration:
[Unit] Description=Gunicorn instance to serve my transform server After=network.target [Service] User=www-data Group=www-data WorkingDirectory=/var/www/TRX/ ExecStart= /usr/bin/python3 /usr/local/bin/gunicorn --bind=0.0.0.0:8080 --threads=25 --workers=2 project:app [Install] WantedBy=multi-user.target
Use 'Ctrl + X' then press 'Y' and hit enter to save the file.
The "ExecStart" will need to use absolute paths to point to your python and gunicorn installations. To find the absolute paths on your machine, you can use the following commands:
which python3 which gunicorn
We can then start and enable our service using the following commands:
systemctl start TRX systemctl enable TRX
If we want to check the the service is running, we can use the following command:
systemctl status TRX
To list all available transforms you can use this command:
cd /var/www/TRX/ python3 project.py list
Then you should see a similar output:
= Transform Server URLs = /run/dnstoip/: DNSToIP /run/greetperson/: GreetPerson = Local Transform Names = dnstoip: DNSToIP greetperson: GreetPerson