Generaly there are two reasons for OutOfmemoryError in NetBeans IDE. Either the application really requires more memory and you need to customize it as described in How do I set the JVM's heap size to be smaller? or there is a memory leak problem that needs to be fixed.
Recommendations for filing bug report are summarized in How to Write a Good Bug Report page and there are few more steps that can help in this specific case. Writing a detailed description of how the problem can be reproduced is the best way to make it easy to analyze the problem.
Custom modifications of netbeans.conf file should be mentioned especialy if they are related to memory management. Information about additional modules installed into IDE can be important (attaching ide.log suffices).
Adding PERFORMANCE keyword to a bug will attract attention to the bug.
Often it is hard to find what steps have to be performed to reproduce a leak. Some of the leaks are not massive and they are slowly degrading performance with more and more live objects on Java heap and finally resulting in an OOME. In these cases it might be useful to generate a dump of Java heap that can be used for postmortem analysis. NetBeans developers should be able to process these dumps to find what data structure is causing the problem and often this is the most important information required to fix the problem.
JVM memory dumpers
Alan Bateman's Heap dumps are back with a vengeance! blog entry gives an overview of how to generate a dump of Java heap for analysis. It is much easier to generate these dumps with Mustang (Java SE 6) including possibility to make a dump automatically when OutOfMemoryError occurs by adding -XX:+HeapDumpOnOutOfMemoryError flag into netbeans.conf file. If you want to produce the dump with Tiger (Java SE 5), please use the latest available version of JDK that incorporates recent bugfixes.
Trouble-Shooting and Diagnostic Guide for Java SE 5 and Trouble-Shooting and Diagnostic Guide for Java SE 6 contain useful information about memory leaks and tools that can be used to generate a dump of memory content. Slides from BOF presentation about OutOfMemoryError from JavaONE2005
Yet another possibility to generate a memory dump is INSANE tool. It works well with Java 1.4.2 and newer and provides two ways how to generate the dump. There is a button in memory toolbar and also it is possible to provoke the dumping by starting another instance of the IDE and passing --dumpfile option to it. Unfortunately there has to be some space on Java heap to produce the expected results.
Various types of memory leaks
Java heap space. This is the most common type of this error.
Perm Gen space. Often a problem with classloaders that are not released from memory. It can be caused by Ant tasks. Frequent module enabling and disabling (installing and uninstalling) can also cause this.
Source: NetBeans User FAQ