Skip to content

Fix local class scanning#111

Open
wheelerlaw wants to merge 2 commits into
ExpediaGroup:masterfrom
cvp-ops:fix-local-class-scanning
Open

Fix local class scanning#111
wheelerlaw wants to merge 2 commits into
ExpediaGroup:masterfrom
cvp-ops:fix-local-class-scanning

Conversation

@wheelerlaw
Copy link
Copy Markdown
Contributor

@wheelerlaw wheelerlaw commented Oct 29, 2021

Note: The branch for this PR is rebased on top of the branch for the #110 because of the importance of fixing the master build process.

Summary

There are a few changes in this PR:

  1. Jenkins allows shared libraries to be written as a script, in addition to fully qualified class. The superclass of a script is groovy.lang.Script, whereas the superclass of a regular class is whatever the class extends (or java.lang.Object if it doesn't extend a class).

    The LocalProjectPipelineExtensionDetector class scans for classes in the project to instrument in the getClassesOfTypeInPackage method by using the Reflections package. It scans for classes that extend Object. Reflections, however, does not supporting scanning for classes that extend a class but are not direct subtypes of the same class. So the getClassesOfTypeInPackage method misses all Jenkins scripts that are a part of the shared library.

    An easy solution to this would be to also scan for classes that extend groovy.lang.Script. However, it is also perfectly valid for Jenkins shared libraries classes to extend other Jenkins shared library classes. The solution here is to use a class scanner that supports searching for classes that are descendants of a class.

  2. According to this comment, a ScanResult should be assigned within a try-with-resources, so I have made that change in the Local detector and the WholeClasspath detector.

  3. I found another Java 11+ compatibility issue (not catching NoClassDefFoundError exception) and fixed it.

Checklist

Testing

(Remove this checklist and replace it with "N/A - no code changes" if this PR does not modify source code)

  • I have manually verified that my code changes do the right thing.
  • I have run the tests and verified that my changes do not introduce any regressions.
  • I have written unit tests to verify that my code changes do the right thing and to protect my code against regressions

Documentation

(Remove this checklist and replace it with "N/A - no code changes" if this PR does not modify source code)

  • I have updated the "Unreleased" section of CHANGELOG.md with a brief description of my changes.
  • I have updated code comments - both GroovyDoc/JavaDoc-style comments and inline comments - where appropriate.
  • I have read CONTRIBUTING.md and have followed its guidance.

@kriegaex
Copy link
Copy Markdown

kriegaex commented Nov 25, 2021

I am not sure if this related, but after adding some dependencies to one of my projects, an old test using jenkins-spock is no longer working, complaining about missing OSGi and JNA classes while using the org.reflections tool. I have no idea why this is happening.

16:40:06.560 [main] INFO  org.reflections.Reflections - Reflections took 996 ms to scan 10 urls, producing 1502 keys and 8401 values 
16:40:06.568 [main] WARN  org.reflections.Reflections - could not get type for name org.osgi.framework.BundleActivator from any class loader
org.reflections.ReflectionsException: could not get type for name org.osgi.framework.BundleActivator
	at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:312)
	at org.reflections.Reflections.expandSuperTypes(Reflections.java:382)
	at org.reflections.Reflections.<init>(Reflections.java:140)
	at com.homeaway.devtools.jenkins.testing.LocalProjectPipelineExtensionDetector.getClassesOfTypeInPackage(LocalProjectPipelineExtensionDetector.java:61)
	at com.homeaway.devtools.jenkins.testing.LocalProjectPipelineExtensionDetector$getClassesOfTypeInPackage.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:135)
	at com.homeaway.devtools.jenkins.testing.JenkinsPipelineSpecification.setupSpec(JenkinsPipelineSpecification.groovy:1122)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at org.spockframework.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:200)
	at org.spockframework.runtime.model.MethodInfo.invoke(MethodInfo.java:113)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:484)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:467)
	at org.spockframework.runtime.BaseSpecRunner.doRunSetupSpec(BaseSpecRunner.java:177)
	at org.spockframework.runtime.BaseSpecRunner$3.invoke(BaseSpecRunner.java:161)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:484)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:467)
	at org.spockframework.runtime.BaseSpecRunner.runSetupSpec(BaseSpecRunner.java:154)
	at org.spockframework.runtime.BaseSpecRunner.doRunSetupSpec(BaseSpecRunner.java:174)
	at org.spockframework.runtime.BaseSpecRunner$3.invoke(BaseSpecRunner.java:161)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:484)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:467)
	at org.spockframework.runtime.BaseSpecRunner.runSetupSpec(BaseSpecRunner.java:154)
	at org.spockframework.runtime.BaseSpecRunner.runSetupSpec(BaseSpecRunner.java:149)
	at org.spockframework.runtime.BaseSpecRunner.doRunSpec(BaseSpecRunner.java:94)
	at org.spockframework.runtime.BaseSpecRunner$1.invoke(BaseSpecRunner.java:81)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:484)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:467)
	at org.spockframework.runtime.BaseSpecRunner.runSpec(BaseSpecRunner.java:73)
	at org.spockframework.runtime.BaseSpecRunner.run(BaseSpecRunner.java:64)
	at org.spockframework.runtime.Sputnik.run(Sputnik.java:63)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)
Caused by: java.lang.ClassNotFoundException: org.osgi.framework.BundleActivator
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:310)
	... 39 common frames omitted
16:40:06.581 [main] WARN  org.reflections.Reflections - could not get type for name org.osgi.framework.SynchronousBundleListener from any class loader
org.reflections.ReflectionsException: could not get type for name org.osgi.framework.SynchronousBundleListener
	at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:312)
	at org.reflections.Reflections.expandSuperTypes(Reflections.java:382)
	at org.reflections.Reflections.<init>(Reflections.java:140)
	at com.homeaway.devtools.jenkins.testing.LocalProjectPipelineExtensionDetector.getClassesOfTypeInPackage(LocalProjectPipelineExtensionDetector.java:61)
	at com.homeaway.devtools.jenkins.testing.LocalProjectPipelineExtensionDetector$getClassesOfTypeInPackage.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:115)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:135)
	at com.homeaway.devtools.jenkins.testing.JenkinsPipelineSpecification.setupSpec(JenkinsPipelineSpecification.groovy:1122)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at org.spockframework.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:200)
	at org.spockframework.runtime.model.MethodInfo.invoke(MethodInfo.java:113)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:484)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:467)
	at org.spockframework.runtime.BaseSpecRunner.doRunSetupSpec(BaseSpecRunner.java:177)
	at org.spockframework.runtime.BaseSpecRunner$3.invoke(BaseSpecRunner.java:161)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:484)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:467)
	at org.spockframework.runtime.BaseSpecRunner.runSetupSpec(BaseSpecRunner.java:154)
	at org.spockframework.runtime.BaseSpecRunner.doRunSetupSpec(BaseSpecRunner.java:174)
	at org.spockframework.runtime.BaseSpecRunner$3.invoke(BaseSpecRunner.java:161)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:484)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:467)
	at org.spockframework.runtime.BaseSpecRunner.runSetupSpec(BaseSpecRunner.java:154)
	at org.spockframework.runtime.BaseSpecRunner.runSetupSpec(BaseSpecRunner.java:149)
	at org.spockframework.runtime.BaseSpecRunner.doRunSpec(BaseSpecRunner.java:94)
	at org.spockframework.runtime.BaseSpecRunner$1.invoke(BaseSpecRunner.java:81)
	at org.spockframework.runtime.BaseSpecRunner.invokeRaw(BaseSpecRunner.java:484)
	at org.spockframework.runtime.BaseSpecRunner.invoke(BaseSpecRunner.java:467)
	at org.spockframework.runtime.BaseSpecRunner.runSpec(BaseSpecRunner.java:73)
	at org.spockframework.runtime.BaseSpecRunner.run(BaseSpecRunner.java:64)
	at org.spockframework.runtime.Sputnik.run(Sputnik.java:63)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)
Caused by: java.lang.ClassNotFoundException: org.osgi.framework.SynchronousBundleListener
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:310)
	... 39 common frames omitted
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...

Test ignored.

Test ignored.

java.lang.NoClassDefFoundError: com/sun/jna/Library

	at java.base/java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:1267)
	at java.base/java.lang.System$2.findBootstrapClassOrNull(System.java:2196)
	at java.base/jdk.internal.loader.ClassLoaders$BootClassLoader.loadClassOrNull(ClassLoaders.java:118)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:665)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:665)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:604)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at java.base/java.lang.Class.forName(Class.java:377)
	at com.homeaway.devtools.jenkins.testing.LocalProjectPipelineExtensionDetector.getClassesOfTypeInPackage(LocalProjectPipelineExtensionDetector.java:88)
	at com.homeaway.devtools.jenkins.testing.JenkinsPipelineSpecification.setupSpec(JenkinsPipelineSpecification.groovy:1122)

Would this PR fix a problem like this?

@kriegaex
Copy link
Copy Markdown

kriegaex commented Nov 25, 2021

Would this PR fix a problem like this?

Answering my own question: Yes, this PR fixes my problem. I just cloned the fork and built this PR's branch. I think it would make a lot of sense for the maintainer to actually review and, if it proves to be valid, merge this PR.

@awittha awittha self-assigned this Nov 29, 2021
@awittha awittha self-requested a review November 29, 2021 19:32
@awittha awittha added the bug Something isn't working label Nov 29, 2021
Copy link
Copy Markdown
Contributor

@awittha awittha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, but I have some questions I need cleared up before I can either approve or ask for changes.

Comment thread pom.xml
Comment on lines -274 to -278
<dependency>
<groupId>org.reflections</groupId>
<artifactId>reflections</artifactId>
<version>0.9.12</version>
</dependency>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The motivation behind using Reflections AND GlassGraph, was that Reflections would only see files in the local project; nothing on the extended classpath from Maven dependencies would be included.

How does one configure the current ClassGraph to only see files that are physically present on disk in the current project?

Or, to ask another way:

With this code change, if there's a dependency of the project that includes a file like the ScriptToTest.groovy that's added here, will these changes detect that script, or not?

They should not.

Is that done with the .disableJarScanning() here: https://github.com/ExpediaGroup/jenkins-spock/pull/111/files#diff-7cf77f094673fdcead882d1c0b654a186428288727ba01a59de640f2a9f80d7fR57 ?

}
Set<Class<?>> annotated_classes;
try (
ScanResult scanResult = new ClassGraph()
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this also .disableJarScanning()?


class LocalProjectPipelineExtensionDetectorSpec extends JenkinsPipelineSpecification {

def "LocalProjectPipelineExtensionDetectorSpec should detect classes that aren't direct subtypes of java-lang-Object" () {
Copy link
Copy Markdown
Contributor

@awittha awittha Nov 29, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
def "LocalProjectPipelineExtensionDetectorSpec should detect classes that aren't direct subtypes of java-lang-Object" () {
def "LocalProjectPipelineExtensionDetector should detect classes that aren't direct subtypes of java-lang-Object" () {

It's really the LocalProjectPipelineExtensionDetector that you're testing, right?

If you were testing the LoalProjectPipelineExtensionDetectorSpec, you'd be checking the DEFAULT_TEST_CLASSES ( https://github.com/ExpediaGroup/jenkins-spock/blob/master/src/main/groovy/com/homeaway/devtools/jenkins/testing/JenkinsPipelineSpecification.groovy#L1137 ) variable's contents, right?

I think this test can actually just extend Specification, since it's testing a standalone, non-Jenkins-specific class (LocalProjectPipelineExtensionDetector), right?

And if that's the case... since the class-under-test is in Java, can the test itself be written in ("matching") JUnit Java, too?

(These are actual questions, not change requests phrased as questions, by the way.)

@kriegaex
Copy link
Copy Markdown

kriegaex commented Feb 17, 2022

Why is nobody fixing this problem after such a long time? Locally, I am working with a snapshot version containing this fix, but my CI builds, e.g. on GitHub, cannot use that snapshot, because it is not deployed in any publicly available repository. So I have a choice to either skip my Jenkins Spock tests or let them fail with the error I posted above. It would be nice to get a maintenance release after almost 4 months.

With regard to code review feedback: Why not just be pragmatic, merge locally and fix what you think ought to be fixed, like a normal maintainer would, taking repsonsibility for what is merged into his own repository? You do not need to wait for the PR creator to commit your suggestions.

kriegaex added a commit to kriegaex/GebSpockSamples that referenced this pull request Feb 17, 2022
Use snapshot locally, but release on GitHub in order to avoid that Maven
cannot find the snapshot during the build. At the same time,
PublishBuildLogIT will be skipped if "jenkins-spock-2.1.5.jar" is found
on the classpath. I.e., it will be skipped on GitHub, because with that
buggy dependency it will fail. We can remove that restriction after
ExpediaGroup/jenkins-spock#111 was merged or the
corresponding bug fixed otherwise and a new release published.
kriegaex added a commit to kriegaex/GebSpockSamples that referenced this pull request Feb 18, 2022
Use snapshot locally, but release on GitHub in order to avoid that Maven
cannot find the snapshot during the build. At the same time,
PublishBuildLogIT will be skipped if "jenkins-spock-2.1.5.jar" is found
on the classpath. I.e., it will be skipped on GitHub, because with that
buggy dependency it will fail. We can remove that restriction after
ExpediaGroup/jenkins-spock#111 was merged or the
corresponding bug fixed otherwise and a new release published.
priyankarawat added a commit to konciergeMD/jenkins-spock that referenced this pull request Jun 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Development

Successfully merging this pull request may close these issues.

3 participants