randoop | Automatic test generation for Java | Unit Testing library

Each generated test may call any or all of the classes under test. For example, if your classlist file contains Class1.java, Class2.java, and Class3.java, then each generated test might call all three classes. Randoop does not create a separate set of tests for each class.

If there is only one class under test (which is definitely against the recommended practice!), then you could rename the files and the classes within them after they are generated. But that is not your situation, as you described above.

Randoop's input is .class or .jar files. Randoop generates Java JUnit test cases. If you are willing to have Java tests for your Kotlin code, then it should be possible to use Randoop.

If you want Randoop to generate Kotlin code, you would need to change the way that Randoop outputs its internal data structures into test file.

The only automated Kotlin test generator that I know of is EvoMaster, which generates system-level tests (not unit tests) for RESTful APIs that provide a schema in OpenAPI/Swagger format. It detects 500 status codes and mismatches from the API schemas.

If the outputLocation is null, you have to use the default output location javaProject.getOutputLocation() instead of entry.getPath().

See Javadoc of IClasspathEntry.getOutputLocation():

Returns:

the full path [...], or null if using default output folder

If in Project > Properties: Java Build Path tab Source the check box Allow output folders for source folders is not checked, IClasspathEntry::getOutputLocation() will always return null.

1. Why does not changing --output-limit change the number of generated test cases?

Randoop generates tests, then outputs a subset of them. For example, Randoop does not output subsumed tests, which appear as a subsequence of some longer test.

This is mentioned obliquely in the documentation for --output-limit.

two of which checking the method getClass does not return null value (even though that is not part of my specification)

getClass() is a method in Math (the class under test), so Randoop calls getClass(). At test generation time, the return value was not null, so Randoop made an assertion to that effect.

There is nothing special about getClass(); Randoop would create similar regression tests for other methods.

1. It seems that Randoop does not consult the specification inside spec.json

There was a bug in Randoop's handling of postcondition specifications for static methods. The bug has been fixed.

To report a bug, it is better to use Randoop's issue tracker, as noted in the Randoop manual. The options for getting help also include mailing lists. Unlike Stack Overflow, the issue tracker and mailing list permit discussions and tracking current status. Thanks!

Solved.

The jHipster genetator don't have the index.js on the main folder, so, the correct code to invoke it is:

First of all, you should override findClass instead of loadClass. The restriction that a class with a specific name can only be defined once in a loader does always apply and you may always encounter situations where the same class is requested multiple times (in any nontrivial scenario).

The loadClass implementation inherited from ClassLoader will already take care of this and return the existing definition, if there is one. Since it also queries the parent loader first, you should take care to specify the right parent loader, if the loader is supposed to define classes with names that are also in use by other class loaders. Using the default implementation of loadClass, your findClass method can stay that simple:

The error message gives you the answer:

Ignoring interface net.sf.javaml.core.Dataset specified via --classlist or --testclass. No classes to test

You are supposed to provide a class, not an interface, to the --testclass command-line argument.

By passing --testclass=net.sf.javaml.core.Dataset to Randoop, you indicated that you only want Randoop to create objects of type net.sf.javaml.core.Dataset. However, since that is an interface, it cannot be instantiated, and Randoop cannot create any objects, nor any tests.

[The paper] said that Randoop was earlier developed and evaluated on libraries.

This is largely true not just of Randoop, but of other test generation tools such as ARTOO, Check 'n' Crash, EvoSuite, GRT, QuickCheck, etc.

The paper is "Scaling up automated test generation: Automatically generating maintainable regression unit tests for programs" (ASE 2011). Its problem statement is that test generation tools have often been applied to libraries, which are easier to handle than programs. Its contribution is showing how to extend a test generation tool (Randoop) to programs.

An example of an earlier paper that applied Randoop to libraries is "Feedback-directed random test generation" (ICSE 2007). It reports finding dozens of important, previously-unknown errors.

I read it in a research paper published on IEEE which said that libraries dont change often and hence dont need much of regression testing.

The paper does not say libraries "don't need much regression testing". It actually says, "A library is less likely to be in need of a regression test suite. Libraries rarely change, and a library is likely to already have some tests." The main point is that the Randoop tool generates tests, and such a tool is more needed for components that don't have tests. As a general rule, libraries usually already have a human-written test suite. The library is also exercised by every program that uses it. By contrast, many programs exist that don't have a test suite, or whose test suite omits large parts of the program's behavior. Test generation is more needed for such components.

This is point #5 near the end of a list of 6 reasons to motivate extending Randoop to programs. The comment makes sense in that context but not when taken out of context or misquoted. The list starts with,

Randoop was originally targeted toward detecting existing bugs in data structure libraries libraries such as the JDK's java.util. Instead, we want to extend Randoop to generate maintainable regression tests for complex industrial software systems.

Data structure libraries tend to be easier for tools to handle in several ways. ...

Returning to one of your questions, every software component -- whether a program or a library -- needs a regression test suite to be run when it changes. Running the tests gives you confidence that your changes have not broken its functionality. If you never change a component, then you don't need a regression test suite for it.

Some libraries never change (because of policy, or because there is no need to change them), and others are being constantly updated.