Instrumentation and CF Analysis issues - what we know so far

[emaiannone]

Instrumentation

Instrumentation of classes and methods seems to work as follow:

1. EvoSuite target class is checked and its CFGs are registered (by a specific thread)
2. Inheritance tree and other things are initialized
3. For each class called by target class through a method call:
   3.1 The class is checked and its CFGs are registered. This is recursively repeated for each superclass and interface until an excluded class is encountered (e.g., Object)
   3.2 Exceptions specified by throws clause are checked and registered. This is recursively repeated for each superclass
   3.3. Class-level annotations are checked (not registered because they have no implemented method)
4. Each class NOT called by target class but references in instance variables is checked but not registered.

Note:

• Classes referenced by local variables are not checked at all.

Something strange happens when

When a polymorphic method is called, i.e., having an instance of B, subclass of A, and calling an overriden method through a reference of A, activating polymorphism.

This does not occur when polymorphism is avoided, i.e., having an instance of B, subclass of A, and calling the overriden method through a reference of B, avoiding polymorphism.

We thought that if the target class calls a method with an argument of a non-excluded class the problem would occured, but it is not.

What happens

In step 1 described above, additional classes are checked but not registered. In step 3 they are totally ignored because they have already been checked.

Steps to reproduce

VulnerableToySystemTest launches various tests. One of it involves MyCUTAnalyzePolymorphic target class and it exhibit the issue.

Why it happens?

I don't know!

Workaround?

This is an example of proposed solution in InstrumentingClassLoader at line 163:

Class<?> result = classes.get(name);
LoggingUtils.getEvoLogger().warn("Evaluating: " + name);
if (DependencyAnalysis.getInheritanceTree() == null) {
    if (name.equals(Properties.TARGET_CLASS)) {
        LoggingUtils.getEvoLogger().warn("This is target class" + name);
        Class<?> instrumentedClass = instrumentClass(name);
        return instrumentedClass;
    } else {
        return null;
    }
} else {
    if (result != null) {
        return result;
    } else {
        logger.info("Seeing class for first time: " + name);
        Class<?> instrumentedClass = instrumentClass(name);
        return instrumentedClass;
    }
}

It works? The above problem seems avoided, but much more other classes are being instumented.

Another approach is in TestSuiteGenerator class, in initializeTargetClass() method. The following code:

DependencyAnalysis.analyzeClass(Properties.TARGET_CLASS, Arrays.asList(cp.split(File.pathSeparator)));
LoggingUtils.getEvoLogger().info("* " + ClientProcess.getPrettyPrintIdentifier() + "Finished analyzing classpath");

should be moved before this:

DefaultTestCase test = buildLoadTargetClassTestCase(Properties.TARGET_CLASS);
ExecutionResult execResult = TestCaseExecutor.getInstance().execute(test, Integer.MAX_VALUE)

Control Flow Analysis

In any case, another issue is present: EvoSuite builds the call graph from CUT according to source code only, so it seems not to resolve runtime bindings (e.g., when polymorpism occurs). All CFGs are properly registered, but only the ones available in compile time are involved in the call graph, thus detailed with control dependencies (needed for the generation).

Workaround?

I don't know :(