Data Collection in Detail

Modified on: Thu, 17 Oct, 2024 at 6:00 AM

1. Collect the data of a target profile

In your usual browser, search for a profile whose data (friends, timeline, etc.) you would like collect with Maltego Evidence. Once you have found a profile, there are two ways to collect it in Maltego Evidence.

  1. Using Maltego Evidence without a browser - only with the URL of the target profile.
  2. The use of the Maltego Evidence Browser Extension


For variant 2., the Maltego Evidence Browser Extension must be installed for your browser beforehand. You can find this extension under the following links:


Chrome

https://chrome.google.com/webstore/detail/snh-titan-browser-extensi/bhajampliglbihiecgcppjajijeechbl

Firefox

Coming soon

Edge

Coming soon


1.1 Using Maltego Evidence without a browser

After creating a new project, you will land on the dashboard page of the project. If you want to collect more data in an existing project, click on the Maltego Evidence logo (top left) to go to your projects and select the desired project.


Alternatively, you can click on the project name and go directly to the corresponding project using the drop-down menu.


image-20230316101355-8.jpeg


Click on "Collect" in the left column and you will be taken to the survey page. Here you work your way through the selection fields from left to right.


1.1.1 Enter target profile

First, enter your target profile in the input field. It does not matter which social network it is. You can simply copy the address line from your browser and paste it into the input field.


image-20230316101355-9.jpeg


After entering the target profile, the program checks your input and automatically selects the corresponding network:


image-20230316101355-10.jpeg


1.1.2 Select crawling profile

In order for Maltego Evidence to be able to correctly save the data in full, it must log into a so-called crawling profile. These are mostly profiles created specifically for this purpose on the corresponding social networks. Please note that these profiles can occasionally also be blocked. Therefore, never use your private profile for the collection of data to prevent the loss and blocking of your main profile.


To assign a crawling profile to a collection job, you can either select an account that has already been entered with one click or you can enter a new account.


Important!
 Multiple accounts can also be selected for a collection! Especially for long timeline collections as well as friend collections this is absolutely recommended. Again, please do not forget that each of the selected profiles must have access to the corresponding content.


1.1.2.1 Enter a new crawling profile

Click the "Add Crawling Profile" or "Create Crawling Profile" button.


A sidebar will open. Please fill in the following data there:


Network

First, select which network the crawling profile is valid for. If you choose Telegram, the input mask will change to require a phone number instead of a username and password.

Name

You define this name yourself to be able to identify the profile later in the software. For example, "Max Mustermann".

Username / Vanity Name / E-Mail

This is the username or email address, or occasionally the phone number, that you would use to log into the network if you were also logging into the browser. For most networks, it is the value you would enter in the first field of the login screen.

Password

This is the password you use to log in. In special cases (e.g. Telegram or TikTok) the password can be left blank because other login methods are used.

Check login status

Here you can select whether Maltego Evidence should directly check and update the login status of the profile in the background. This is optional and offers you the possibility to find out in advance if the profile you have selected has possibly already been locked and if the specified data is valid.


Note
If you have further questions about creating crawling profiles, please consult the "Profile Manager" page in the User Guide.

Click Save. You can now select the profile in the crawling profiles overview.


If you have executed a login status check, this happens in the background and can take a few seconds to minutes, depending on the Internet connection and network.


It may happen in individual cases that Maltego Evidence needs your help to authenticate the login profile. E.g. for TikTok scanning a QR code, or for other networks entering a confirmation code sent by SMS or email. This would be communicated to you by means of a non-closable popup.


Note on Telegram
To store a Telegram profile, you only need your phone number. On your first login attempt (either the one you requested, or as part of the first executed survey), you will be asked for a confirmation code. The message sent to your cell phone will contain a warning that you should not give the code to any third party. 


In general, this is correct. However, in our specific case, login without this code is not possible. The access authorization generated with this code is only valid for your local PC. Maltego has no access to your profile and no possibility to access your data.


image-20230316101355-11.jpeg


1.1.3 Select survey options

Under the heading "What to crawl?" you must now select what exactly is to be saved.


image-20230316101355-12.jpeg


Here you can customize in detail which data should be saved for the profile. This is particularly relevant to ensure data protection. Please only collect data that is relevant to your case or investigation. In particular, when backing up friends or friends of friends, make sure that the collection of this data is relevant. You can add or remove individual options by checking or unchecking the boxes.


Depending on the network, different sub-items can be activated by clicking on the box above. If the mouse pointer is moved over an information sign, there is more information about this item.


You can repeat the above steps as many times as you like to collect multiple target profiles in one go. Click on "Add URL" to define additional targets. For each of these targets, you can customize your collection options and used collection profiles individually. You can also add targets from different networks in one query.


1.1.4 Survey options in detail

The survey options are presented in detail below:

Save Profile

Backs up the profile information. If you deselect this option, all other selected collection options will also be deselected. The reason for this is that each of the subsequent collection options necessarily backs up the profile as well.

Take Full Page Screenshot

By activating this option, a survey job is sent in which scrolls over the target profile. This will take several screenshots and stitch them together at the end to get a total screenshot of the timeline. You will find this in the detail bar of the target profile after completion and you can export this as PNG or PDF.

Save Friends

This save option creates a collection job where the friends of the target profile are saved. Depending on Save Friends / Save Followers, either friends and followers, or only one of the two will be backed up. Please make sure that a survey of friends is necessary for your purposes before selecting this option to ensure privacy.

Save Friends

If this option is selected, all visible friends of the target profile will be saved.

Save Followers

If you select this option, the followers of the target profile will be saved.

Save Timeline

Backing up a timeline has the effect of backing up all content published by the user on the timeline. This includes all users associated with it (e.g. authors of shared content and, depending on the options, also reacting and commenting users

Date Range

You can limit the collection of data to a certain period of time. Especially when backing up very large and active profiles, pages or groups, this is necessary to avoid blocking the collection profiles. We recommend to choose one day before the desired first posting and one day after the desired last posting. Depending on time and time zones in the network, it can lead to unsaved postings in rare cases, since time information is transmitted differently depending on the network.

Override Security Limit

Within some networks we have added a maximum default collection limit to avoid blocking of collection profiles. This value varies depending on the network. If you want to secure very distant postings by means of a time limit or simply secure a lot of postings in general, then you can override this security limit at your own risk. However, we do not recommend this. You may want to consider using single-posting collections instead.

Save Comments

Comments and their authors are only collected if you activate this option. Otherwise, only the number of comments is collected.

Save Reactions

This option is essential to collect reactions and related users. This option is useful when the friends list is not public, but you may want to identify connected / potentially friended users of the target profile.

Save Stories

In some networks, users can publish so-called "stories". These are usually only visible for 24 hours and then disappear automatically. With the "Save Stories" option, this content is also saved. However, we cannot save past Stories unless they are permanently visible as highlights (on Instagram, for example). 

Download Videos

The postings to videos are saved even without enabling this option. However, in SNH Titan you will then only see the preview of these videos. If you would like to be able to download the actual video content and also view it later, you must activate this option. However, it is important to mention that depending on the video length and quality, this will significantly increase the duration of the collection(s).

Save Media

In some cases, not all media are saved with the timeline collection. For example, on Facebook, not every media item necessarily has to have been shared beforehand. When you collect media, you also collect the media items that have not been shared, including the album structure of the target profiles.

Save Reactions

Depending on the network, media elements can also have reactions. In order for them to be saved, it is necessary to enable this option.

Save Comments

Depending on the network, media items may also have comments. In order for these to be saved, it is necessary to enable this option.

Save Friends of Friends

By activating this option, you will collect the friends of the target profile, as well as all visible friends of these friends. Please note that in most cases this collection will take several hours and it is not uncommon to collect more than 50,000 or 100,000 profiles. Use this collection only if it is absolutely necessary for your investigation and, if necessary, clarify the necessity of this collection with your supervisor or data protection officer beforehand.

Distribute to multiple runners

This option will appear in the following releases and allows you to distribute the workload of the many surveys among all runners. If you have multiple runners, this can speed up the collection by a factor of 3. However, it is important to mention that you will then not be able to continue running a survey in parallel until the collection job is completed or canceled.

Save Single Posting

You can also specify a single posting as the destination URL. Depending on the network, the query for the URL of a single posting is different. If you have specified a posting URL, you must deselect all other options and select this option instead. The SNH does not automatically recognize that it is a posting URL!

Save Comments

Saves the comments of the posting, including their authors.

Save Reactions

Saves the reactions of the posting, including the reacting users.

Download Videos

If the post contains one or more videos, they will be downloaded only if this option is enabled.


1.1.5 Sending the survey request

Click on the "Send tasks to runner" button to start the collection. The page will now refresh and after a few seconds you will see the currently running collection tasks in the top right menu bar.


By clicking on the tasks you can view the current status.


Icons-Sicherung.png


1.2 Using the Browser Extension

If you have installed the Maltego Evidence Browser Extension, you should already see a sidebar on the right side of the identified target profile.


When collapsed, you will see a small preview of the identified profile. Click on this preview box to display the collection options. In the resulting expanded sidebar, you will see the possible collection options. For an initial "quick start", we recommend keeping the default values for now, unless your profile contains extreme amounts of data. For example, backing up a newspaper's Facebook page might be an inconvenient way to start, due to the enormous number of posts that need to be backed up.


Click on "Send to Maltego Evidence" to send the collection job to Maltego Evidence. Important: Allow the browser to access the Maltego Evidence!


The Maltego Evidence desktop application will then open and offer you a few more collection options. You will be asked to select a project. Here, the project you just created should be selected by default. 


You will also be asked to select a collection profile. More about this in step 1.1.2.


Details and screenshots of this process can be seen in the Browser Extension submenu. 




2. Browser Extension

What is the browser extension?


We have developed a browser extension so that you can easily launch surveys from your usual browser. It gives you the possibility to switch smoothly between your browser and Maltego Evidence. The browser extension detects use of postings, profiles, groups and pages and offers you the option to specifically save them. This, unlike alternative software products, allows for maximum specificity and at the same time minimum data collection, as it really only collects the data from the social networks for which you have specifically requested a backup. Below we explain how you can install and use the extension.


How do I install the browser extension?

You can download the extension via the following links:

Chrome

https://chrome.google.com/webstore/detail/snh-titan-browser-extensi/bhajampliglbihiecgcppjajijeechbl

Firefox

Coming soon

Edge

Coming soon


We explain below how to install the browser extension using Chrome as an example.


Click the Add button in Chrome.

install-ce-1.png


Please confirm your selection by clicking the "Add extension" button.


__fileCreatedFromDataURI__.png


Now that you have added the extension, please visit one of the supported social networks:

  • Facebook
  • Instagram
  • Twitter
  • Youtube
  • VKontakte
  • Odnoklassniki


On the right side of the respective websites you will now see a window asking you to select whether it is a Desktop installation or an Enterprise installation.


Please select "Maltego Evidence Desktop" and click on "Activate".


extension-activation2.png

Installation in the other browsers works in the same way.


How can I tell that the browser extension has been installed?

When you visit one of the supported websites, you should see a Maltego Evidence logo in the bottom left corner. This shows you that the extension is active and searching for profiles, postings, groups, pages, etc.


Screenshot 2023-03-16 114327.png


How do I back up content with the browser extension?

Profiles, pages and groups


When you visit a profile, page or group, you will see a small preview of the profile you just visited on the right side.


Extension-Sidebar.png


Click on this preview to bring up the survey bar. 


Extension-Sidebar-Big.png


The options that are also offered in Maltego Evidence are displayed. You can configure the survey task here accordingly.

With a click on "Send task to Maltego Evidence" Maltego Evidence will either start or the already open window will be brought to the front and focused.


A survey mask opens in which you have to select the crawling profile and the project. These two selection fields are framed in red in the following screenshot. All other configurations have been pre-filled based on your selection in the browser. 


Click on the blue button at the bottom right to submit the survey jobs to let the runners collect the data completely.


task-from-ext.png


Backing up individual posts

There are two ways to save a single posting.


About the sidebar

In the first option you visit the page of a posting directly. On this page, as with profiles, you will see a sidebar. Here you can select which data you want to save. Clicking on the blue button will take you back to Maltego Evidence.


Single-Posting-Seitenleiste.png


About the Direct button

On some social networks, the extension also recognizes the postings directly in the feed and offers you a button either next to or within the posting to save this posting. You can see if Maltego Evidence has recognized a posting by the green frame and the save button.


In the screenshot you can see two examples:

detected-posting.pngSingle-Posting-Twitter.png


Clicking the button will take you directly back to Maltego Evidence and you can save the individual posting.


Since social networks regularly change in their design or structure, it may be that not all postings are perfectly recognized. In this case, please simply use method 1, which is usually more stable.


If you have no or hardly any postings recognized correctly, please report this to us via Discord, preferably with a screenshot.




3. Overview of Survey Tasks

How do I get to the overview of the survey tasks?

Project surveys

Sometimes it can be helpful to call up the survey tasks overview to see which tasks have already been processed, which tasks have not yet been started and which are currently being executed. You can view details about the survey there, as well as call up technical debug logs if errors occur.


There are several ways to view the current tasks. Usually the list of project tasks is the clearest way to view all tasks. To do this, click on the "Project Tasks" button in the upper right corner of a project, next to the tags.


project-tasks-button.png


There you will see in table form all pending (not yet started) surveys (A), all running (B) and completed or failed surveys (C).


For the ongoing surveys, you will see regular screenshots of the survey and the browser. If available, you also see a status display with a progress bar and the last sent status message of the survey modules. The view refreshes automatically.


Note
 Sometimes the view may take a few seconds to refresh. So, if the next survey job does not go directly to a running state after a previous one has been completed, we ask you to wait a few seconds. The next job will then start directly.


all-tasks-beschriftet.png


Show individual survey in detail

To view details about an individual survey, you can click on a survey and then view the details, as described in the screenshot below:


go-to-task-page (1).png


The survey page is divided into three sections.


At the top left you see information about the survey description, i.e. what is being surveyed.


At the bottom left you see the debug log. If the window is not visible, you need to expand the bottom area upwards by dragging the small bar above the "Debug Output" heading.


By default, the debug log is only loaded when you click Update Debug Log. If there was an error during the collection, then we ask you to send us this debug log. To do this, first click on "Update Debug Log" and then on the "Save" icon next to the "Debug Output" heading. Please send us this file.


On the right side you see status information about the survey job. If the job has not been started yet, you will be informed there that it is waiting until a runner becomes free. If the job is currently running, then you will see regular screenshots, as well as a progress bar and the status messages. If the survey is completed, you will see an information whether the survey was completed successfully, was cancelled by you, or an error occurred. In most cases, an error indicates that your crawling profile has been locked. Therefore, please check in the profile manager if your profile is displayed as faulty.


If you have canceled the survey, you can still have the data collected up to that point imported by clicking on the corresponding button for this on the right-hand side. However, not all data has always been collected up to the point of cancellation, so it is possible that the data is not complete. Please take this into account in your investigation.


single-task (2).png

3

Did you find it helpful? Yes No

Send feedback
Sorry we couldn't be helpful. Help us improve this article with your feedback.