Phrase To Website [using Search Engine]
Modified on: Fri, 17 Jul, 2020 at 9:40 AM
Transform Meta Info
|Display Name||To Website [using Search Engine]|
|Short Description||This Transform will search for the given phrase and show the sites where the phrase occurs.|
|Data Source||Bing API|
General to all search engine Transforms
There are a few Transforms that use search engines - all of them are very similar. The basic recipe for these Transforms is as follows:
- Expand the question. The question is the input from the GUI - be that a persons name, a domain or an phone number. When looking at a persons name, for instance, the name "Kosie Kramer", will be expanded to searches like Kosie Kramer, K Kramer, Kramer Kosie etc. In the case of a telephone number the search will be expanded to include most telephone notations used.
- Assign confidence levels. Because a search for "Kosie Kramer" is more likely to return better results - rather than a search for 'KramerK', the confidence level for the first search would be higher. The confidence levels are also used to assign preference to certain file types when doing searches on documents (these are configurable within the Transform itself). In the same way an XLS file containing the word is likely more interesting than a PDF file.
- Perform each search. The searches are performed and the snippets are obtained. It is important to note that only snippets are parsed. For parsing the entire page you need to dump to URL and process the URLs separately. Various search engines have various snippet lengths.
- Parse for output Entities. Depending on what output is required the snippets are parsed for Entities - in some cases the web site name is all that's required.
- Calculate weight. The weight is calculated from various factors - the confidence of the search, the frequency of the result, the importance of the originating website,, and in some cases, a correlation to the input.
- Normalise. The weights are now normalised using a fairly interesting algorithm that involves the mean and standard deviation of the spread of weights. It is important to understand that a search result with an equal spread of weights are often of no use.
General notes when using search engine Transforms
Maltego will sometimes give you results that seem incorrect. You need to keep in mind that the application will get desperate when it does not have results to return. So when you are searching for a person called "Vaxynutus Grabounill" and that person simply left no marks on the Internet, Maltego will eventually go after a search term "VG" despite a low confidence. However, you will still get some results. These results could seem completely off the mark, but should have very low weights. Always consider the weights.
Problems with parsing results
Some Entities are difficult to parse, such as telephone numbers. There is always a trade-off between missing numbers and parsing non-telephone numbers as phone numbers. With the current Transform we hope to reach the optimal balance.