You need to increase the memory allocated to the compiler. In order to do this:

  1. Right click on the project node in the Projects window and select Properties
  2. In the project properties dialog, click on Build -> Compiling
  3. In the Additional Compiler Options text box, enter the Xms and Xms switches as needed (for example -J-Xms128m -J-Xmx512m)
  4. Click OK in the dialog
  5. Make sure external Javac compiler will be executed by entering the following line to of the project (Important Files/Per-user Project Properties):

That should resolve the issue.

Source: NetBeans User FAQ