Update the Ruby XML Test logic

[philsmart]

To support the changes in #130 and #18, we will need to update the logic in the Ruby XML script file in the following ways:

• By default, return the four new validators defined in Copy the 4 ukfed policy files into the testbed #130. Defined here:

  ukf-testbed/driver/lib/testbed/xml_test.rb

  Line 67 in 69a4dc0

  def validators_option(endpoint)

  • Each test will run through each of the four new policy files unless explicitly excluded.
• Change the endpoint parameter everywhere into a Ruby hash, allowing additional parameters without altering the method signature. For example:

  ukf-testbed/driver/lib/testbed/xml_test.rb

  Line 44 in 69a4dc0

  def option(optname, endpoint)

• Recast the options method so it can match on both endpoint and validator combinations for dealing with overrides. This enables:
  • Tests explicitly linked to specific policy validators by configuration (validators:).
  • Tests that run with any configured policy validator, but can be excluded from certain validators by configuration (overrides:)

[philsmart]

I added vals = option('validators', endpoint) || ['default', 'edugain', 'publication', 'registered_entity'] and all the validators are run as expected.

Noting default could become the development or something validator

[philsmart]

This will be part of #132

[philsmart]

As it has been a little while, I want to clear up the aims for this. We want to support the following.

1. By default, tests run through all the validators and all endpoints configured by the vals change made above. For example, this bad test will be checked against all validators (edugain etc) and all endpoints (prod and next):

expected:
  - status: error
    component_id: check_idpdisc
    message: incorrect Binding value on DiscoveryResponse

2. Tests linked to a specific set of validators. For example, this bad test will only be checked against the edugain and publication validators:

expected:
- status: error
  component_id: check_idpdisc
  message: incorrect Binding value on DiscoveryResponse
validators:
 - edugain
 - publication

3. Tests that are linked to all validators and endpoints, but can be specifically linked to validators attached to a specific endpoint, or exclude an endpoint completly. For example, the following bad test expects the general error message given, but excludes the 'next' endpoint entirely, and limits the 'prod' and 'other' endpoints to only using the default and edugain validators, expecting both to contain the expected error (the expected error being optional).

expected:
- status: error
  component_id: check_idpdisc
  message: incorrect Binding value on DiscoveryResponse
override:
  - endpoint: "next"
    skip: true
  - endpoint: ["prod","other"]
    validator: ["default", "edugain"]
    expected:
      - status: error
        message: a warning specific to these validators

4. And any combination thereof. For example, the follwing bad test defines a different expected error message on the edugain validator on prod, to the default validator also on prod.

expected:
- status: error
  component_id: check_idpdisc
  message: incorrect Binding value on DiscoveryResponse
override:
  - endpoint: "next"
    skip: true
  - endpoint: prod
    validator: default
    expected:
      - status: error
        message: a warning specific to the default validator on prod
 - endpoint: prod
    validator: edugain
    expected:
      - status: error
        message: a warning specific to the edugain validator on prod

[philsmart]

Another use case would be to have an override to skip only on validator? For example, run all endpoints and validators other than the default validator (on all endpoints).

expected:
- status: error
  component_id: check_idpdisc
  message: incorrect Binding value on DiscoveryResponse
override:
  - validator: default
    skip: true

[philsmart]

My Ruby skills are poor, so this could take a bit of time. I should also learn how to unit test it, otherwise it'll be even more difficult.

[alexstuart]

My Ruby skills are poor, so this could take a bit of time. I should also learn how to unit test it, otherwise it'll be even more difficult.

Sure, although Ian developed the framework to start with. You could ask him if he's able and willing to do the job.

[iay]

I can take a look at it at any point, of course. I have been assuming we want Phil to take the lead on it and come to me once he has something that works; we can make it a bit more Ruby-ish at that point if needed (not that I'm an authority either).

[philsmart]

Thanks, @iay. I have worked on something, so at some point, I will commit it for you to take a look. It may or may not be working—currently, it is not.

[philsmart]

Back to this...again. I think the exsiting ruby supports the use cases in #131 (comment). Which is confusing. But for example, this works as expected:

override:
  - endpoint: ["next"]
    validators: ["registered_entity"]
    expected:
      - status: error
        component_id: check_owner
        message: 'unknown owner name: An Example Organization'
      - status: error
        component_id: check_uk_mdui_dn_en_match
        message: 'mismatched xml:lang=''en'' DisplayNames: ''An Example Entity'' in mdui
          vs. ''An Example Organization'' in ODN'
  - endpoint: ["prod"]
    validators: ["default"]

Which would leave the only change the change to the default set of validators:

vals = option('validators', endpoint) || ['default', 'edugain', 'publication', 'registered_entity']

I am sure I am missing something. That said, the bigger issue is that other than the 'default' validator which contains our sanitised list of checks, all the other validators representing the policies fail most of the tests. This not unexpected as they contain additional checks we have not run before, and for metadata that is in a different state than our test metadata example (for example eduGAIN opt-in and opt-out). I'll dig into that a bit now.

[philsmart]

Maybe the use case was more than one endpoint validator override (which is perhaps a better fit with #131 (comment)). For example:

override:
  - endpoint: ["next"]
    validators: ["registered_entity"]
    expected:
      - status: error
        component_id: check_owner
        message: 'unknown owner name: An Example Organization'
      - status: error
        component_id: check_uk_mdui_dn_en_match
        message: 'mismatched xml:lang=''en'' DisplayNames: ''An Example Entity'' in mdui
          vs. ''An Example Organization'' in ODN'
  - endpoint: ["next"]
    validators: ["default"]
  - endpoint: ["prod"]
    validators: ["default"]

Next should run through the registered_entity and default validators with different expected outcomes. I think it only runs through the first one, right now.

[philsmart]

I'm still not very good at Ruby, so I relied heavily on the usual online resources, some Ruby docs, and CoPilot (I know) to better understand the syntax. I have made changes to the Ruby xml_test file to satisfy all of the above use cases (I think). This needs review by @iay.

I am going to try to explain the differences below.

---

(1)

I removed def option(optname, endpoint), and replaced it with def overrides_for(endpoint) (link). This method looks for ALL the overrides defined for a given endpoint. For example, given the following yaml:

override:
  - endpoint: ["next"]
      validators: ["registered_entity"]
      expected:
       ..
  - endpoint: ["next"]
       validators: ["default", "edugain"]
  - endpoint: ["prod"]
       validators: ["default"]

If the endpoint was 'next', it returns an array of the two 'next' endpoint overrides, both of which include a set of validators; the first also includes an expected status (not shown). For example:

[{"endpoint"=>["next"], "validators"=>["registered_entity"], "expected"=>[{...}]}, 
{"endpoint"=>["next"], "validators"=>["default", "edugain"]}]

---

(2)

Removed def validators_option(endpoint) and added def validator_executions(endpoint) link. This method creates a Hash of 'executions' from the output of overrides_for in (1). Each execution contains a single validator and all of its associated override configuration. For example, if we select 'next' from the first example, we now have a Hash of (Yes, the validators are repeated inside each validator element):

[{:validator=>"registered_entity", :override=>{"endpoint"=>["next"], "validators"=>["registered_entity"], "expected"=>[{" ...}]}}, 
{:validator=>"default", :override=>{"endpoint"=>["next"], "validators"=>["default", "edugain"]}}, 
{:validator=>"edugain", :override=>{"endpoint"=>["next"], "validators"=>["default", "edugain"]}}]

So the Hash is roughly:

{
  validator: "name",
  override:  { ...override hash... } 
}

If it can not find any validators in the overrides, it either takes the validators from the top level validators property, or the default set defined.

---

(3)

Changed def expected_option(endpoint) to def expected_option(override) to pull out the expected statuses (link). The difference being, we will already know the overrides for a given endpoint by this point (see (6)).

---

(4)

Changed def skip_option(endpoint) to def skip_option(override) to pull out the skip override (link). Again, the difference being, we will already know the overrides for a given endpoint by this point (see (6)).

---

(5)

Changed def execute_one(endpoint, &handler) to run each of the validators returned from validator_executions (link). Noting that the execution input to execute_validator is the hash returned from validator_executions in (2). In hindsight, perhaps this method should be execute_all rather than execute_one.

---

(6)

Changed def execute_validator(endpoint, validator, metadata, &handler) to def execute_validator(endpoint, execution, metadata, &handler) (link). This now pulls out the validator to run and the override config from the execution Hash passed from execute_one. It runs through the validator in the usual way, and takes statuses out of the override config.