This usually means that the Java Virtual Machine (JVM) could not allocate enough memory for the application. This almost always only occurs on older Java versions, we recommend updating to to the latest release of Java 8 or 11. You can download it here.
If you have allocated too much or too little memory to Maltego, you can easily fix this by running the config application. You can find the Maltego config application by searching for "Config" in your app list.