Testing

Kilda consists of next testing layers:

• unit tests
• Functional tests
• ATDD-staging

Each module has a comprehensive README file, describing the purpose, how to run, etc.
Currently we actively develop 'unit tests' and 'functional tests' modules. ATDD-staging contains high-level cucumber scenarios which we run as smoke set and rarely update.

---

Coverage

It's really difficult to assess the coverage rate. We compiled a list of high-level test-cases, marked which ones are covered by automation and compiled a percentage 'rate' of how much is covered. But one should understand that provided numbers is a rough estimate.

Defects

Defects are logged to the backlog and assigned to the 'Open Kilda Board' project. Each defect should also have label 'Bug' and one of the 'priority' labels. As usual, defect should have steps to reproduce, expected and actual result.
If possible, it is preferable to provide a branch with functional-test that reproduces the issue.

Test Cases

This section contains description of all test cases that are currently covered by automation.

Switch-related

• DefaultRules
• DefaultRulesValidation
• FlowRules
• LagPort
• Meters
• MultiFloodlights
• PortAntiflap
• PortHistory
• PortHistoryIsolated
• PortProperties
• SwitchActivation
• SwitchDelete
• SwitchesFlowsV2
• Switches
• SwitchFailures
• SwitchMaintenance
• SwitchPortConfig
• SwitchProperties
• SwitchSync
• SwitchValidationSingleSwFlow
• SwitchValidation
• SwitchValidationV2

ISL-related

• Bfd
• IslMinPortSpeed
• IslReplug
• LinkMaintenance
• LinkProperties
• Link
• RoundTripIsl
• UnstableIsl

Flow-related

Regular Flow	Y-Flow	HA-Flow
AutoReroute	SubFlow	HaFlowCreate
AutoRerouteIsolated	YFlowCreate	HaFlowDiversity
Bandwidth	YFlowDiversity	HaFlowHistory
ConnectedDevices	YFlowPathSwap	HaFlowIntentionalReroute
DefaultFlow	YFlowPing	HaFlowPaths
FlowAffinity	YFlowProtected	HaFlowPathSwap
FlowCrud	YFlowReroute	HaFlowPing
FlowCrudV1	YFlowUpdate	HaFlowProtected
FlowDiversity	YFlowValidation	HaFlowReroute
FlowHistory		HaFlowSync
FlowLoop		HaFlowUpdate
FlowMonitoring		HaFlowValidation
FlowPing
FlowSync
FlowValidationNegative
IntentionalReroute
MaxLatency
MirrorEndpoints
MultiReroute
PartialUpdate
PinnedFlow
ProtectedPath
ProtectedPathV1
QinQFlow
SwapEndpoint
ThrottlingReroute
VxlanFlow

Server42-related

• Server42IslRtt
• Server42FlowRtt
• Server42YFlowRtt
• Server42HaFlowRtt

Network

• PathCheck
• PathComputation
• Paths

Statistics

• FlowStat
• YFlowStat
• HaFlowStat
• MflStat
• SimulateStats
• Tsdb

Toggles

• FeatureToggles
• FeatureTogglesV2

GRPC

• GrpcCommon
• License
• LogicalPort
• Log

Logging

• CheckLogging

Configuration

• Configuration

Resilience

• Chaos
• Contention
• FloodlightKafkaConnection
• Retries
• RetriesIsolated
• StormHeavyLoad
• StormLcm

---

Switch-related

---

DefaultRules

• Default rules are installed on switches #sw.hwSwString()

  Expect: Default rules are installed on the switch

• Default rules are installed when a new switch is connected

  Given: A switch with no rules installed and not connected to the controller

  When: Connect the switch to the controller

  Then: Default rules are installed on the switch

• Able to install default rule on #sw.hwSwString() [install-action=#data.installRulesAction]

  Given: A switch without any rules

  When: Install rules on the switch

  Then: The corresponding rules are really installed

• Able to install default rule on switch: #sw.hwSwString() [install-action=#data.installRulesAction]

  Given: A switch without rules

  When: Install rules on the switch

  Then: The corresponding rules are really installed

• Able to install default rules on #sw.hwSwString() [install-action=INSTALL_DEFAULTS]

  Given: A switch without any rules

  When: Install rules on the switch

  Then: The corresponding rules are really installed

• Able to delete default rule from #sw.hwSwString()[delete-action=#data.deleteRulesAction]

  When: Delete rules from the switch

  Then: The corresponding rules are really deleted

  And: Switch and rules validation shows that corresponding default rule is missing

• Able to delete default rule from #sw.hwSwString() [delete-action=#data.deleteRulesAction]

  When: Delete rule from the switch

  Then: The corresponding rule is really deleted

  And: Switch and rules validation shows that corresponding default rule is missing

• Able to delete/install the server42 Flow RTT turning rule on a switch

  Given: Select a switch which support server42 turning rule

  And: Server42 is enabled in feature toggle

  When: Delete the server42 turning rule from the switch

  Then: The delete rule response contains the server42 turning cookie only

  And: The corresponding rule is really deleted

  And: Switch and rules validation shows that corresponding rule is missing

  When: Install the server42 turning rule

  Then: The install rule response contains the server42 turning cookie only

  And: The corresponding rule is really installed

• Able to delete/install the server42 ISL RTT turning rule on a switch

  Given: Select a switch which support server42 turning rule

  And: Server42 is enabled in feature toggle

  And: server42IslRtt is enabled on the switch

  When: Delete the server42 ISL RTT turning rule from the switch

  Then: The delete rule response contains the server42 ISL RTT turning cookie only

  And: The corresponding rule is really deleted

  And: Switch and rules validation shows that corresponding rule is missing

  When: Install the server42 ISL RTT turning rule

  Then: The install rule response contains the server42 ISL RTT turning cookie only

  And: The corresponding rule is really installed

DefaultRulesValidation

This test suite checks the switch validate and rule validate features regarding default rules. System should be able to detect missing, misconfigured, proper and excess default rules. The real-life usecase is that we should properly detect and distinguish 'duplicate' default rules with the same cookie but different match/priority/etc.

• Switch and rule validation can properly detect default rules to 'proper' section (#sw.hwSwString() #propsDescr)

  Given: Clean switch without customer flows and with the given switchProps

  Expect: Switch validation shows all expected default rules in 'proper' section

  And: Rule validation shows all expected default rules in 'proper' section

FlowRules

Verify how Kilda behaves with switch rules (either flow rules or default rules) under different circumstances: e.g. persisting rules on newly connected switch, installing default rules on new switch etc.

• Pre-installed flow rules are not deleted from a new switch connected to the controller

  Given: A switch with proper flow rules installed (including default) and not connected to the controller

  When: Connect the switch to the controller

  Then: Previously installed rules are not deleted from the switch

• Able to delete rules from a switch (delete-action=#data.deleteRulesAction)

  Given: A switch with some flow rules installed

  When: Delete rules from the switch

  Then: The corresponding rules are really deleted

• Able to delete switch rules by cookie/priority #data.identifier

  Given: A switch with some flow rules installed

  When: Delete switch rules by #data.identifier

  Then: The requested rules are really deleted

• Attempt to delete switch rules by supplying non-existing #data.description leaves all rules intact

  Given: A switch with some flow rules installed

  When: Delete switch rules by non-existing #data.description

  Then: All rules are kept intact

• Able to delete switch rules by #data.description

  Given: A switch with some flow rules installed

  When: Delete switch rules by #data.description

  Then: The requested rules are really deleted

• Attempt to delete switch rules by supplying non-existing #data.description keeps all rules intact

  Given: A switch with some flow rules installed

  When: Delete switch rules by non-existing #data.description

  Then: All rules are kept intact

• Able to validate and sync missing rules for #description on terminating/transit switches

  Given: Two active not neighboring switches with a long available path

  And: Create a transit-switch flow going through these switches

  And: Remove flow rules so that they become 'missing'

  When: Synchronize rules on switches

  Then: The corresponding rules are installed on switches

  And: No missing rules were found after rules validation

• Unable to #action rules on a non-existent switch

  When: Try to #action rules on a non-existent switch

  Then: An error is received (404 code)

• Able to synchronize rules for a flow with protected path

  Given: Two active not neighboring switches

  And: Create a flow with protected path

  And: Delete flow rules(for main and protected paths) on involved switches for creating missing rules

  When: Synchronize rules on switches

  Then: No missing rules were found after rules synchronization

  And: Synced rules are exactly the same as before delete (ignoring irrelevant fields)

• Traffic counters in ingress rule are reset on flow rerouting(multiTable mode)

  Given: Two active neighboring switches and two possible flow paths at least

  And: Create a flow going through these switches

  When: Start traffic examination

  Then: Traffic counters in shared/ingress/egress rule on source and destination switches represent packets movement

  When: Break the flow ISL (bring switch port down) to cause flow rerouting

  Then: The flow was rerouted after reroute timeout

  And: Traffic counters in ingress rule on source and destination switches are reset

• Able to synchronize rules for a flow with VXLAN encapsulation

  Given: Two active not neighboring Noviflow switches

  And: Create a flow with vxlan encapsulation

  And: Delete flow rules so that they become 'missing'

  When: Synchronize rules on switches

  Then: The corresponding rules are installed on switches

  And: Rules are synced correctly

  And: No missing rules were found after rules validation

LagPort

Verify that flow can be created on a LAG port.

• Able to CRUD LAG port with lacp_reply=#lacpReply on #sw.hwSwString()

  Given: A switch

  When: Create a LAG

  Then: Response reports successful creation of the LAG port

  And: LAG port is really created

  And: LAG port is really created on the switch(check GRPC)

  And: Switch is valid

  When: Update the LAG port

  Then: Response reports successful updation of the LAG port

  And: LAG port is really updated

  And: LAG port is really updated on the switch(check GRPC)

  And: Switch is valid

  When: Delete the LAG port

  Then: Response reports successful deletion of the LAG port

  And: LAG port is really deleted from db

  And: LAG port is really deleted from switch

• Able to create a flow on a LAG port

  Given: A switchPair with a LAG port on the src switch

  When: Create a flow

  Then: Flow is valid and pingable

  And: System allows traffic on the flow

• Able to create a singleSwitchFlow on a LAG port

  Given: A switch with two traffgens and one LAG port

  And: A flow on the LAG port

  When: Create a flow

  Then: Flow is valid

  And: System allows traffic on the flow

• LAG port is not deleted after switch reconnecting

  Given: A switch with a LAG port

  When: Disconnect the switch

  And: Connect the switch back

  Then: The LAG port is still exist

  And: Switch is valid

• Unable to delete a LAG port in case flow on it

  Given: A flow on a LAG port

  When: When delete LAG port

  Then: Human readable error is returned

• Unable to create LAG on a port with flow on it

  Given: Active switch with flow on it

  When: Create a LAG port with flow's port

  Then: Human readable error is returned

• Unable to create a flow on port which is inside LAG group

  Given: An active switch with LAG port on it

  When: Create flow on ports which are in inside LAG group

  Then: Human readable error is returned

• Unable to create a LAG port with port which is used as mirrorPort

  Given: A flow with mirrorPoint

  When: Create a LAG port with port which is used as mirrorPort

  Then: Human readable error is returned

• Unable to create a LAG port in case port is #data.description

  When: Create a LAG port on a occupied port

  Then: Human readable error is returned

• Unable to create two LAG ports with the same physical port inside at the same time

  Given: A switch with a LAG port

  When: Try to create the same LAG port with the same physical ports inside

  Then: Human readable error is returned

• Unable to proceed incorrect delete LAG port request (#data.description)

  When: Send invalid delete LAG port request

  Then: Human readable error is returned

• System is able to detect and sync missed LAG port

  Given: A switch with a LAG port

  When: Delete LAG port via grpc

  Then: System detects that LAG port is missed

  When: Synchronize the switch

  Then: LAG port is reinstalled

• System is able to detect misconfigured LAG port

  Given: A switch with a LAG port

  When: Modify LAG port via grpc(delete, create with incorrect ports)

  Then: System detects misconfigured LAG port

• Able to create/update LAG port with duplicated port numbers on the switch

  Given: Switch and two ports

  When: Create LAG port with duplicated port numbers

  Then: Response shows that LAG port created successfully

  And: Request on user side shows that LAG port created

  And: Created port exists in a list of all LAG ports from switch side (GRPC)

  When: Update the LAG port with duplicated port numbers

  Then: Response shows that LAG port updated successfully

  And: Check on user side that LAG port updated successfully

  And: Check that LAG port updated successfully on switch side (via GRPC)

• Able to create and delete single LAG port with lacp_reply=#data.portLacpReply

  Given: A switch

  When: Create a LAG port

  Then: Response reports successful creation of the LAG port

  And: Correct rules and meters are on the switch

  When: Delete the LAG port

  Then: Response reports successful delete of the LAG port

  And: No LACP rules and meters on the switch

• Able to create and delete LAG port with #data.description

  Given: A switch with LAG port

  When: Create a LAG port

  Then: Response reports successful creation of the LAG port

  And: Correct rules and meters are on the switch

  When: Delete created LAG port

  Then: Response reports successful delete of the LAG port

  And: No LACP rules and meters of second LAG port on the switch

• Able to update #data.description for single LAG port

  Given: A switch

  And: A LAG port

  When: Update the LAG port

  Then: Response reports successful update of the LAG port

  And: Correct rules and meters are on the switch

• Able to update #data.description near to existing LAG port with lacp_reply=#data.existingPortLacpReply

  Given: A switch

  And: LAG port 1

  And: LAG port 2

  When: Update the LAG port

  Then: Response reports successful update of the LAG port

  And: Correct rules and meters are on the switch

• Unable decrease bandwidth on LAG port lower than connected flows bandwidth sum

  Given: Flows on a LAG port with switch ports

  When: Decrease LAG port bandwidth by deleting one port to make it lower than connected flows bandwidth sum

  Then: Human readable error is returned

  Then: No bandwidth changed for LAG port and all connected ports are in place

• Able to delete LAG port if it is already removed from switch

  Given: A switch with a LAG port

  When: Delete LAG port via grpc

  Then: Able to delete LAG port from switch with no exception

Meters

The test suite checks if traffic meters, including default, are set and deleted in a correct way. Note that many tests are bind to meter implementations of certain hardware manufacturers.

• Able to delete a meter from a #switchType switch

  Given: Select a #switchType switch and retrieve default meters

  When: A flow is created and its meter is deleted

  Then: Delete operation should be successful

  When: Delete the flow

  Then: No excessive meters are installed on the switch

• Unable to delete a meter with invalid ID=#meterId on a #switchType switch

  Given: Select a #switchType switch

  When: Try to delete meter with invalid ID

  Then: Got BadRequest because meter ID is invalid

• Default meters should express bandwidth in kbps re-calculated from pktps on Centec #sw.hwSwString()

  Expect: Only the default meters should be present on the switch

• Default meters should express bandwidth in pktps on Noviflow #sw.hwSwString()

  Expect: Only the default meters should be present on the switch

• Default meters should express bandwidth in kbps on Noviflow Wb5164 #sw.hwSwString()

  Expect: Only the default meters should be present on the switch

• Meters are created/deleted when creating/deleting a single-switch flow with ignore_bandwidth=#ignoreBandwidth on a #switchType switch

  Given: A #switchType switch with OpenFlow 1.3 support

  When: Get default meters from the switch

  And: Create a single-switch flow

  Then: New meters should appear after flow setup

  And: All new meters should have KBPS, BURST and STATS flags installed

  And: All new meters rate should be equal to flow's rate

  And: Switch validation shows no discrepancies in meters

  And: Flow validation shows no discrepancies in meters

  When: Delete the flow

  Then: New meters should disappear from the switch

• Meters are not created when creating a single-switch flow with maximum_bandwidth=0 on a #switchType switch

  Given: A #switchType switch with OpenFlow 1.3 support

  When: Get default meters from the switch

  And: Create a single-switch flow with maximum_bandwidth=0

  Then: Ony default meters should be present on the switch and new meters should not appear after flow setup

• Source/destination switches have meters only in flow ingress rule and intermediate switches don't have meters in flow rules at all (#srcSwitch - #dstSwitch flow)

  When: Create a flow between given switches

  Then: The source and destination switches have only one meter in the flow's ingress rule

  And: The source and destination switches have no meters in the flow's egress rule

  And: Intermediate switches don't have meters in flow rules at all

• Meter burst size is correctly set on #switchType switches for #flowRate flow rate

  Given: A single-switch flow with #flowRate kbps bandwidth is created on OpenFlow 1.3 compatible switch

  When: Update flow bandwidth to #flowRate kbps

  Then: New meters should be installed on the switch

  And: New meters rate should be equal to flow bandwidth

  And: New meters burst size matches the expected value for given switch model

  And: Switch validation shows no discrepancies in meters

  And: Flow validation shows no discrepancies in meters

• Flow burst should be correctly set on Centec switches in case of #flowRate kbps flow bandwidth

  Given: A single-switch flow with #flowRate kbps bandwidth is created on OpenFlow 1.3 compatible Centec switch

  When: Update flow bandwidth to #flowRate kbps

  Then: Meters with updated rate should be installed on the switch

  And: New meters burst size should respect the min/max border value for Centec

  And: Switch validation shows no discrepancies in meters

  And: Flow validation shows no discrepancies in meters

• Meter burst size is correctly set on Noviflow Wb5164 switches for #flowRate flow rate

  Given: A single-switch flow with #flowRate kbps bandwidth is created on OpenFlow 1.3 compatible switch

  When: Update flow bandwidth to #flowRate kbps

  Then: New meters should be installed on the switch

  And: New meters rate should be equal to flow bandwidth

  And: New meters burst size matches the expected value for given switch model

  And: Switch validation shows no discrepancies in meters

  And: Flow validation shows no discrepancies in meters

• System allows to reset meter values to defaults without reinstalling rules for #data.description flow

  Given: Switches combination (#data.description)

  And: A flow with custom meter rate and burst, that differ from defaults

  When: Ask system to reset meters for the flow

  Then: Response contains correct info about new meter values

  And: Default meters are unchanged

  And: Switch rules are unchanged

• Try to reset meters for unmetered flow

  Given: A flow with the 'bandwidth: 0' and 'ignoreBandwidth: true' fields

  When: Resetting meter burst and rate to default

  Then: Human readable error is returned

MultiFloodlights

Every switch can be connected to multiple floodlights. Floodlight can be either 'read-only'(stats floodlights) or 'read-write'(management floodlights). System chooses one 'master' fl between all RW fls. All switch floodlights can be checked via 'GET /api/v2/switches/{switchId}/connections'

• Switch remains online only if at least one of multiple RW floodlights is available

  Given: Switch simultaneously connected to 2 management floodlights

  And: Background observer monitoring the state of switch and its ISLs

  When: Switch loses connection to one of the regions

  Then: Switch can still return its rules

  When: Broken connection gets fixed, but the second floodlight loses connection to kafka

  Then: Switch can still return its rules

  And: All this time switch monitor saw switch as Active

  When: Switch loses connection to the last active region

  Then: Switch is marked as inactive

  When: Try getting switch rules

  Then: Human readable error is returned

  When: Broken region restores connection to kafka

  Then: Switch becomes activated and can respond

  When: Switch restores connection to the other region

  Then: Switch can still return its rules and remains active

  And: ISL monitor reports that all switch-related ISLs have been Up all the time

  Cleanup:

• System supports case when switch uses different networks to connect to FLs, i.e. has diff ips

  Given: A switch with at least 2 regions available

  And: The switch's ip in first region is different from other

  And: The switch is currently disconnected

  When: Test switch gets connected to both regions

  Then: Get switch connections API returns different ips for regions

  Cleanup:

PortAntiflap

Anti-flap feature is aimed to address the problem of blinking ports. Sometimes it happens that certain port on a switch begins to blink for a very short period of time. If it is a short-time behavior, then we don't want to init a port down event and reroute the flow. Otherwise we wait for 'antiflapWarmup' time and assign that blinking port a DOWN status. Then, when it comes back up it should remain in a stable 'up' state for at least 'antiflapCooldown' seconds to actually become UP for the rest of the system. Initially, port is considered 'flapping' if it changes status quicker than once in 'antiflap.min' seconds (port will change status from UP to DOWN only after 'antiflap.min' in case of a single-time change of status)

• Flapping port is brought down only after antiflap warmup and stable port is brought up only after cooldown timeout

  Given: Switch, port and ISL related to that port

  When: ISL port begins to blink

  Then: Right before warmup timeout runs out the related ISL remains up

  And: After warmup timeout the related ISL goes down

  And: ISL remains down even after cooldown timeout

  When: Port stops flapping, ending in UP state

  Then: Right before cooldown timeout runs out the ISL remains down

  And: After cooldown timeout the ISL goes up

• Port goes down in 'antiflap.min' seconds if no flapping occurs

  Given: Switch, port and ISL related to that port

  When: Port goes down

  Then: Related ISL goes down in about 'antiflapMin' seconds

• System properly registers events order when port flaps incredibly fast (end with Up)

  When: Port blinks rapidly for longer than 'antiflapWarmup' seconds, ending in UP state

  Then: Related ISL is FAILED

  And: After the port cools down the ISL is discovered again

• System properly registers events order when port flaps incredibly fast (end with Down)

  When: Port blinks rapidly for longer than 'antiflapWarmup' seconds, ending in DOWN state

  Then: Related ISL is FAILED

  And: ISL remains failed even after the port cools down

  When: restore broken ISL

  Then: ISL is restored

• A round-trip latency non-direct ISL goes UP according to antiflap

  Given: An active round-trip a-switch link

  When: Port down event happens

  And: Port up event happens

  Then: The ISL is failed till 'antiflap' is deactivated

PortHistoryIsolated

Verify that port history is created for the port up/down actions.

• Port history is able to show ANTI_FLAP statistic

  Given: floodlightRoutePeriodicSync is disabled

  And: A port in a stable state

  When: Execute port DOWN on the port

  Then: Port history is created for that port

  When: Blink port to generate antiflap statistic

  Then: Antiflap statistic is available in port history

PortHistory

Verify that port history is created for the port up/down actions.

• Port history are created for the port down/up actions when link is #islDescription

  Given: A link

  When: Execute port DOWN on the src switch

  Then: Port history is created on the src switch

  When: Execute port UP on the src switch

  Then: Port history is updated on the src switch

  And: Port history on the dst switch is not empty when link is direct

  And: Port history on the src switch is also available using default timeline

• Port history should not be returned in case timeline is incorrect (timeBefore > timeAfter)

  Given: A direct link with port history

  When: Get port history on the src switch for incorrect timeline

  Then: Port history is NOT returned

• Port history should not be returned in case port/switch have never existed

  When: Try to get port history for incorrect port and switch

  Then: Port history is empty

• Port history is available when switch is DEACTIVATED

  Given: A direct link

  When: Execute port DOWN/UP on the src switch

  And: Deactivate the src switch

  Then: Port history on the src switch is still available

• System shows antiflap statistic in the ANTI_FLAP_DEACTIVATED event when antiflap is deactivated before collecting ANTI_FLAP_PERIODIC_STATS

  Given: A direct link

  When: Execute port DOWN on the src switch for activating antiflap

  And: Generate antiflap statistic

  Then: Antiflap statistic is available in port history inside the ANTI_FLAP_DEACTIVATED event

PortProperties

Some switch ports should not be used in network discovery process. By default all ports on all switches are available for discovery. Admin has ability to enable/disable discovery on a specific port on a switch using Northbound REST API.

This spec assumes that port discovery property is enabled for all available ports.

• Able to manipulate port properties

  Given: A port with port properties

  When: Update port discovery property

  Then: Correct response is returned

  And: Port discovery property is really updated

• Informative error is returned when trying to get/update port properties with non-existing switch

  When: Try to get port properties info for non-existing switch

  Then: Human readable error is returned

  When: Try to update port discovery property for non-existing switch

  Then: Human readable error is returned

• Informative error is returned when trying to update port properties with non-existing port number

  When: Try to get port properties info for non-existing port

  Then: No error, default port propreties information is returned

  When: Try to update port discovery property for non-existing port

  Then: Human readable error is returned

• System doesn't discover link when port discovery property is disabled

  Given: A deleted link

  When: Disable port discovery property on the src and dst switches

  And: Bring port up on the src switch

  Then: Link is not detected

  When: Deactivate/activate src switch

  Then: Link is still not detected

  When: Enable port discovery property on the src switch

  Then: Link is detected and status of one-way ISL is FAILED

  When: Enable port discovery property on the dst switch

  Then: Link status is changed to DISCOVERED

• Link is stopped from being discovered after disabling port discovery property

  Given: An active link

  When: Disable port discovery property on the dst switch

  Then: One-way ISL status is changed to FAILED

  When: Enable port discovery property on the dst switch

  Then: Link state is changed to DISCOVERED

SwitchActivation

• Missing flow rules/meters are installed on a new switch before connecting to the controller

  Given: A switch with missing flow rules/meters and not connected to the controller

  When: Connect the switch to the controller

  Then: Missing flow rules/meters were synced during switch activation

• Excess transitVlanRules/meters are synced from a new switch before connecting to the controller

  Given: A switch with excess rules/meters and not connected to the controller

  When: Connect the switch to the controller

  Then: Excess meters/rules were synced during switch activation

• Excess vxlanRules/meters are synced from a new switch before connecting to the controller

  Given: A switch with excess rules/meters and not connected to the controller

  When: Connect the switch to the controller

  Then: Excess meters/rules were synced during switch activation

• New connected switch is properly discovered with related ISLs in a reasonable time

  Given: Disconnect one of the switches and remove it from DB. Pretend this switch never existed

  When: New switch connects

  Then: Switch is activated

  And: Related ISLs are discovered

SwitchDelete

• Unable to delete a nonexistent switch

  When: Try to delete a nonexistent switch

  Then: Get 404 NotFound error

• Unable to delete an active switch

  Given: An active switch

  When: Try to delete the switch

  Then: Get 400 BadRequest error because the switch must be deactivated first

• Unable to delete an inactive switch with active ISLs

  Given: An inactive switch with ISLs

  When: Try to delete the switch

  Then: Get 400 BadRequest error because the switch has ISLs

• Unable to delete an inactive switch with inactive ISLs (ISL ports are down)

  Given: An inactive switch with ISLs

  When: Try to delete the switch

  Then: Get 400 BadRequest error because the switch has ISLs

• Unable to delete an inactive switch with a #flowType flow assigned

  Given: A flow going through a switch

  When: Deactivate the switch

  And: Try to delete the switch

  Then: Got 400 BadRequest error because the switch has the flow assigned

• Able to delete an inactive switch without any ISLs

  Given: An inactive switch without any ISLs

  When: Try to delete the switch

  Then: The switch is actually deleted

• Able to delete an inactive switch with connected devices

  Given: An inactive switch without any ISLs but with connected devices

  When: Try to delete the switch

  Then: The switch is actually deleted

• Able to delete an active switch with active ISLs if using force delete

  Given: An active switch with active ISLs

  When: Try to force delete the switch

  Then: The switch is actually deleted

  Cleanup: Restore the switch, ISLs and reset costs

SwitchesFlowsV2

Verifies feature to retrieve list of flows passing the switch grouped by port number. Details: #5015

• System allows to get flows on particular ports on switch(several flows on the same port)

  Given: Y-Flow subflow which starts on switch

  And: List of the ports that subflow uses on switch, received from flow path

  When: Get all flows on the switch ports used by subflow under test

  Then: Each port in response has information about subflow

  And: Used port on shared endpoint shows both sub-flows

• System allows to get flows on particular ports on switch

  Given: Y-Flow subflow which ends on switch that is not in the path of another sub-flow or regular flow

  And: List of the ports that subflow uses on switch, received from flow path

  When: Get all flows on the switch ports used by subflow under test

  Then: Each port in response has information about the subflow

• System allows to get a flow that #switchRole switch

  Given: Flow that #switchRole switch

  When: Get all flows going through the switch

  Then: The created flows (including both y-flow subflows) are in the response list from the switch

• System allows to get a flow which protected path that goes through switch

  Given: Flow which protected path goes through switch

  When: Get all flows going through the switch

  Then: The flow's protected path is in the response list from the switch

• Mirror sink endpoint port is not listed in list of the ports used

  Given: Switch with flow on it and a free port

  When: Create mirror point on switch with sink pointing to free port

  Then: Mirror sink endpoint port is not listed in the ports list

• Empty list is returned if none of requested ports is busy with any flow

  Given: Switch with flow on it and ports this flow uses

  When: Request flows on several unused ports

  Then: Response is empty, but without errors

• One-switch Y-Flow subflows are listed in flows list

  Given: One switch Y-Flow

  When: Request flows on switch

  Then: Ports used by subflows on the switch are in response

Switches

• System is able to return a list of all switches

  Expect: System can return list of all switches

• System is able to return a certain switch info by its id

  When: Request info about certain switch from Northbound

  Then: Switch information is returned

• Informative error is returned when requesting switch info with non-existing id

  When: Request info about non-existing switch from Northbound

  Then: Not Found error is returned

• Systems allows to get a flow that goes through a switch

  Given: Two active not neighboring switches with two diverse paths at least

  And: A protected flow

  And: A single switch flow

  When: Get all flows going through the involved switches

  Then: The created flows are in the response list from the src switch

  And: Only the protectedFlow is in the response list from the involved switch(except the src switch)

  When: Get all flows going through the src switch based on the port of the main path

  Then: Only the protected flow is in the response list

  When: Get all flows going through the src switch based on the port of the protected path

  Then: Only the protected flow is in the response list

  When: Get all flows going through the src switch based on the dstPort of the single switch flow

  Then: Only the single switch flow is in the response list

  When: Get all flows going through the dst switch based on the dstPort of the protected flow

  Then: Only the protected flow is in the response list

  When: Create default flow on the same switches

  And: Get all flows going through the src switch

  Then: The created flows are in the response list

  When: Bring down all ports on src switch to make flow DOWN

  And: Get all flows going through the src switch

  Then: The created flows are in the response list

• Informative error is returned when requesting all flows going through non-existing switch

  When: Get all flows going through non-existing switch

  Then: Human readable error is returned

• Systems allows to get all flows that goes through a DEACTIVATED switch

  Given: Two active not neighboring switches

  And: A simple flow

  And: A single switch flow

  When: Deactivate the src switch

  And: Get all flows going through the deactivated src switch

  Then: The created flows are in the response list from the deactivated src switch

• System returns human readable error when requesting switch ports from non-existing switch

  When: Request all ports info from non-existing switch

  Then: Not Found error is returned

• System returns human readable error when requesting switch rules from non-existing switch

  When: Request all rules from non-existing switch

  Then: Not Found error is returned

• System returns human readable error when installing switch rules on non-existing switch

  When: Install switch rules on non-existing switch

  Then: Not Found error is returned

• System returns human readable error when deleting switch rules on non-existing switch

  When: Delete switch rules on non-existing switch

  Then: Not Found error is returned

• System returns human readable error when setting under maintenance non-existing switch

  When: set under maintenance non-existing switch

  Then: Not Found error is returned

• System returns human readable error when requesting all meters from non-existing switch

  When: Request all meters from non-existing switch

  Then: Not Found error is returned

• System returns human readable error when deleting meter on non-existing switch

  When: Delete meter on non-existing switch

  Then: Not Found error is returned

• System returns human readable error when configuring port on non-existing switch

  When: Configure port on non-existing switch

  Then: Not Found error is returned

• System returns human readable error when #data.descr non-existing switch

  When: Make action from description on non-existing switch

  Then: Not Found error is returned

• Able to partially update switch a 'location.#data.field' field

  Given: A switch

  When: Request a switch partial update for a #data.field field

  Then: Update response reflects the changes

  And: Changes actually took place

• Able to partially update switch a 'pop' field

  Given: A switch

  When: Request a switch partial update for a 'pop' field

  Then: Update response reflects the changes

  And: Changes actually took place

SwitchFailures

This spec verifies different situations when Kilda switches suddenly disconnect from the controller. Note: For now it is only runnable on virtual env due to no ability to disconnect hardware switches

• ISL is still able to properly fail even if switches have reconnected

  Given: A flow

  When: Two neighbouring switches of the flow go down simultaneously

  And: ISL between those switches looses connection

  And: Switches go back up

  Then: ISL still remains up right before discovery timeout should end

  And: ISL fails after discovery timeout

  And: The flow goes down OR changes path to avoid failed ISL after reroute timeout

• System is able to finish the reroute if switch blinks in the middle of it

  Given: A flow

  When: Current path breaks and reroute starts

  And: Switch reconnects in the middle of reroute

  Then: Flow reroute is successful

  And: Blinking switch has no rule anomalies

  And: Flow validation is OK

• System can handle situation when switch reconnects while flow is being created

  When: Start creating a flow between switches and lose connection to src before rules are set

  Then: Flow eventually goes DOWN

  And: Flow has no path associated

  And: Dst switch validation shows no missing rules

  When: Try to validate flow

  Then: Error is returned, explaining that this is impossible for DOWN flows

  When: Switch returns back UP

  Then: Flow is still down, because ISLs had not enough time to fail, so no ISLs are discovered and no reroute happen

  When: Reroute the flow

  Then: Flow is rerouted and in UP state

  And: Has a path now

  And: Can be validated

  And: Flow can be removed

SwitchMaintenance

• Maintenance mode can be set/unset for a particular switch

  Given: An active switch

  When: Set maintenance mode for the switch

  Then: Maintenance flag for the switch is really set

  And: Maintenance flag for all ISLs going through the switch is set as well

  And: Cost for ISLs is not changed

  When: Unset maintenance mode from the switch

  Then: Maintenance flag for the switch is really unset

  And: Maintenance flag for all ISLs going through the switch is unset as well

  And: Cost for ISLs is not changed

• Flows can be evacuated (rerouted) from a particular switch(several Isls are in use) when setting maintenance mode for it

  Given: Two active not neighboring switches and a switch to be maintained

  And: Create a Flow going through these switches

  And: Create an additional Flow that has the same intermediate switch, but other ISLs are involved into a path

  When: Set maintenance mode without flows evacuation flag for some intermediate switch involved in flow paths

  Then: Flows are not evacuated (rerouted) and have the same paths

  When: Set maintenance mode again with flows evacuation flag for the same switch

  Then: Flows are evacuated (rerouted)

  And: Switch under maintenance is not involved in new flow paths

• Both Y-Flow and Flow can be evacuated (rerouted) from a particular switch when setting maintenance mode for it

  Given: Switch triplet has been selected

  And: Create a Y-Flow going through selected switches and has intermediate switch in a path

  And: Two active not neighboring switches and preferable path with intermediate switch

  And: Create a Flow going through these switches

  When: Set maintenance mode without flows evacuation flag for some intermediate switch involved in flow paths

  Then: Both Flow and Y-Flow are not evacuated (rerouted) and have the same paths

  When: Set maintenance mode again with flows evacuation flag for the same switch

  Then: Both Flow and Y-Flow are evacuated (rerouted)

  And: Switch under maintenance is not involved in new flow paths

• Link discovered by a switch under maintenance is marked as maintained

  Given: An active switch with an active link

  And: Bring port down on the switch to fail the link

  And: Delete the link

  When: Set maintenance mode for the switch

  And: Bring port up to discover the deleted link

  Then: The link is discovered and marked as maintained

SwitchPortConfig

Verify that Kilda allows to properly control port state on switches (bring ports up or down).

• Able to bring ISL-busy port down/up on an #isl.srcSwitch.ofVersion switch #isl.srcSwitch.hwSwString

  When: Bring port down on the switch

  Then: Port failure is logged in TSDB

  When: Bring port up on the switch

  Then: Port UP event is logged in TSDB

• Able to bring ISL-free port down/up on #sw.hwSwString()

  When: Bring port down on the switch

  Then: Port is really DOWN

  When: Bring port up on the switch

  Then: Port is really UP

SwitchSync

• Able to synchronize switch without any rule and meter discrepancies (removeExcess=#removeExcess)

  Given: An active switch

  When: Synchronize the switch that doesn't have any rule and meter discrepancies

  Then: Operation is successful

• Able to synchronize switch (install missing rules and meters)

  Given: Two active not neighboring switches

  And: Create an intermediate-switch flow

  And: Drop all rules an meters from related switches (both default and non-default)

  When: Try to synchronize all switches

  Then: System detects missing rules and meters (both default and flow-related), then installs them

  And: Switch validation doesn't complain about any missing rules and meters

• Able to synchronize #switchKind switch (delete excess rules and meters)

  Given: Flow with intermediate switches

  And: Force install excess rules and meters on switch

  When: Try to synchronize the switch

  Then: System detects excess rules and meters

  And: Switch validation doesn't complain about excess rules and meters

• Able to synchronize switch with 'vxlan' rule(install missing rules and meters)

  Given: Two active not neighboring switches which support VXLAN encapsulation

  And: Create a flow with vxlan encapsulation

  And: Reproduce situation when switches have missing rules and meters

  When: Try to synchronize all switches

  Then: System detects missing rules and meters, then installs them

  And: Switch validation doesn't complain about missing rules and meters

  And: Rules are synced correctly

• Able to synchronize misconfigured default meter

  Given: An active switch with valid default rules and one misconfigured default meter

  When: Synchronize switch

  Then: The misconfigured meter is fixed and moved to the 'proper' section

• Able to synchronize misconfigured flow meter

  Given: An active switch with flow on it

  When: Update flow's meter

  Then: Flow is not valid

  And: Validate switch endpoint shows the updated meter as misconfigured

  When: Synchronize switch

  Then: The misconfigured meter was fixed and moved to the 'proper' section

  And: Flow is valid

SwitchValidationSingleSwFlow

This test suite checks the switch validate feature on a single flow switch. Description of fields:

• missing - those meters/rules, which are NOT present on a switch, but are present in db
• misconfigured - those meters which have different value (on a switch and in db) for the same parameter
• excess - those meters/rules, which are present on a switch, but are NOT present in db
• proper - meters/rules values are the same on a switch and in db

• Switch validation is able to store correct information on a #switchType switch in the 'proper' section

  Given: Select a #switchType switch and retrieve default meters

  When: Create a flow

  Then: Two meters are automatically created.

  And: The correct info is stored in the 'proper' section

  And: Created rules are stored in the 'proper' section

  And: The rest fields in the 'rule' section are empty

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections

• Switch validation is able to detect meter info into the 'misconfigured' section on a #switchType switch

  Given: Select a #switchType switch and retrieve default meters

  When: Create a flow

  And: Change bandwidth for the created flow directly in DB

  Then: Meters info is moved into the 'misconfigured' section

  And: Reason is specified why meters are misconfigured

  And: The rest fields are empty

  And: Created rules are still stored in the 'proper' section

  And: Flow validation shows discrepancies

  When: Reset meters for the flow

  Then: Misconfigured meters are reinstalled according to the new bandwidth and moved into the 'proper' section

  And: Flow validation shows no discrepancies

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections

• Switch validation is able to detect meter info into the 'missing' section on a #switchType switch

  Given: Select a #switchType switch and retrieve default meters

  When: Create a flow

  And: Remove created meter

  Then: Meters info/rules are moved into the 'missing' section

  And: The rest fields are empty

  When: Try to synchronize the switch

  Then: System detects missing rules and meters, then installs them

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections

• Switch validation is able to detect meter info into the 'excess' section on a #switchType switch

  Given: Select a #switchType switch and retrieve default meters

  When: Create a flow

  Then: Rules and meters are created

  When: Update meterId for created flow directly via db

  Then: Origin meters are moved into the 'excess' section

  And: Updated meters are stored in the 'missing' section

  And: Rules still exist in the 'proper' section

  When: Delete the flow

  And: Delete excess meters

  Then: Check that the switch validate request returns empty sections

• Switch validation is able to detect rule info into the 'missing' section on a #switchType switch

  Given: Select a #switchType switch and retrieve default meters

  When: Create a flow

  And: Delete created rules

  Then: Rule info is moved into the 'missing' section

  And: The rest fields in the 'rule' section are empty

  When: Try to synchronize the switch

  Then: System detects missing rules, then installs them

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections

• Switch validation is able to detect rule/meter info into the 'excess' section on a #switchType switch

  Given: Select a #switchType switch and no meters/rules exist on a switch

  When: Create excess rules/meter directly via kafka

  Then: System detects created rules/meter as excess rules

  When: Try to synchronize the switch

  Then: System detects excess rules and meters, then deletes them

  And: Switch validation doesn't complain about excess rules and meters

  Cleanup:

• Able to validate and sync a #switchType switch having missing rules of single-port single-switch flow

  Given: A single-port single-switch flow

  When: Remove flow rules from the switch, so that they become missing

  Then: Switch validation shows missing rules

  When: Synchronize switch

  Then: Switch validation shows no discrepancies

  When: Delete the flow

  Then: Switch validation returns empty sections

SwitchValidation

This test suite checks the switch validate feature followed by switch synchronization for different type of rules and different rules states. Rules states: excess, missing, proper. Rule types: ingress, egress, transit, special (protected path rules, connected devices rules etc.) Note that 'default' rules are not covered by this spec. Description of fields:

• missing - those meters/rules, which are NOT present on a switch, but are present in db
• misconfigured - those meters which have different value (on a switch and in db) for the same parameter
• excess - those meters/rules, which are present on a switch, but are NOT present in db
• proper - meters/rules values are the same on a switch and in db

• Able to validate and sync a terminating switch with proper rules and meters

  Given: A flow

  Expect: Validate switch for src and dst contains expected meters data in 'proper' section

  And: The rest fields in the 'meter' section are empty

  And: Created rules are stored in the 'proper' section

  And: The rest fields in the 'rule' section are empty

  And: Able to perform switch sync which does nothing

  When: Delete the flow

  Then: Switch validate request returns only default rules information

• Able to validate and sync a transit switch with proper rules and no meters

  Given: Two active not neighboring switches

  When: Create an intermediate-switch flow

  Then: The intermediate switch does not contain any information about meter

  And: Rules are stored in the 'proper' section on the transit switch

  And: Able to perform switch sync which does nothing

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections

• Able to validate switch with 'misconfigured' meters

  When: Create a flow

  And: Change bandwidth for the created flow directly in DB so that system thinks the installed meter is misconfigured

  And: Validate src and dst switches

  Then: Meters info is moved into the 'misconfigured' section

  And: Reason is specified why meter is misconfigured

  And: The rest fields of 'meter' section are empty

  And: Created rules are still stored in the 'proper' section

  And: Flow validation shows discrepancies

  When: Restore correct bandwidth via DB

  Then: Misconfigured meters are moved into the 'proper' section

  And: Flow validation shows no discrepancies

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections

• Able to validate and sync a switch with missing ingress rule + meter

  When: Create a flow

  And: Remove created meter on the srcSwitch

  Then: Meters info/rules are moved into the 'missing' section on the srcSwitch

  And: Meters info/rules are NOT moved into the 'missing' section on the dstSwitch

  When: Synchronize switch with missing rule and meter

  Then: Repeated validation shows no missing entities

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections

• Able to validate and sync a switch with missing ingress rule (unmetered)

  When: Create a flow

  And: Remove ingress rule on the srcSwitch

  Then: Ingress rule is moved into the 'missing' section on the srcSwitch

  When: Synchronize switch with missing unmetered rule

  Then: Repeated validation shows no missing entities

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections

• Able to validate and sync a switch with missing transit rule

  Given: Two active not neighboring switches

  And: Create an intermediate-switch flow

  When: Delete created rules on the transit

  Then: Rule info is moved into the 'missing' section

  When: Synchronize the switch

  Then: Repeated validation shows no discrepancies

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections on all involved switches

• Able to validate and sync a switch with missing egress rule

  Given: Two active not neighboring switches

  And: Create an intermediate-switch flow

  When: Delete created rules on the srcSwitch

  Then: Rule info is moved into the 'missing' section on the srcSwitch

  And: Rule info is NOT moved into the 'missing' section on the dstSwitch and transit switches

  When: Synchronize the switch

  Then: Repeated validation shows no discrepancies

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections on all involved switches

• Able to validate and sync an excess ingress/egress/transit rule + meter

  Given: Two active not neighboring switches

  And: Create an intermediate-switch flow

  When: Create excess rules on switches

  Then: Switch validation shows excess rules and store them in the 'excess' section

  And: Excess meter is shown on the srcSwitch only

  When: Try to synchronize every involved switch

  Then: System deletes excess rules and meters

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections on all involved switches

  Cleanup:

• Able to validate and sync a switch with missing 'vxlan' ingress/transit/egress rule + meter

  Given: Two active not neighboring VXLAN supported switches

  And: Create a flow with vxlan encapsulation

  And: Remove required rules and meters from switches

  Expect: Switch validation shows missing rules and meters on every related switch

  When: Try to synchronize all switches

  Then: System installs missing rules and meters

  And: Switch validation doesn't complain about missing rules and meters

  And: Rules are synced correctly

• Able to validate and sync a missing 'protected path' egress rule

  Given: A flow with protected path

  Expect: Upon validation all rules are stored in the 'proper' section

  When: Delete rule of protected path on the srcSwitch (egress)

  Then: Deleted rule is moved to the 'missing' section on the srcSwitch

  And: Rest switches are not affected by deleting the rule on the srcSwitch

  When: Synchronize switch with a missing protected path egress rule

  Then: Switch validation no longer shows missing rules

• Able to validate and sync a missing 'connected device' #data.descr rule

  Given: A flow with enabled connected devices

  Expect: Switch validation puts connected device lldp rule into 'proper' section

  When: Remove the connected device rule

  Then: Switch validation puts connected device rule into 'missing' section

  When: Synchronize the switch

  Then: Switch validation no longer shows any discrepancies in rules nor meters

  When: Delete the flow

  Then: Switch validation is empty

SwitchValidationV2

This test suite checks the switch validate feature followed by switch synchronization for different type of rules and different rules states. Rules states: excess, missing, proper. Rule types: ingress, egress, transit, special (protected path rules, connected devices rules etc.) Note that 'default' rules are not covered by this spec. Description of fields:

• missing - those meters/rules, which are NOT present on a switch, but are present in db
• misconfigured - those meters which have different value (on a switch and in db) for the same parameter
• excess - those meters/rules, which are present on a switch, but are NOT present in db
• proper - meters/rules values are the same on a switch and in db

• Able to validate and sync a terminating switch with proper rules and meters

  Given: A flow

  Expect: Validate switch for src and dst contains expected meters data in 'proper' section

  And: The rest fields in the 'meter' section are empty

  And: Created rules are stored in the 'proper' section

  And: The rest fields in the 'rule' section are empty

  And: Able to perform switch sync which does nothing

  When: Delete the flow

  Then: Switch validate request returns only default rules information

• Able to validate and sync a transit switch with proper rules and no meters

  Given: Two active not neighboring switches

  When: Create an intermediate-switch flow

  Then: The intermediate switch does not contain any information about meter

  And: Rules are stored in the 'proper' section on the transit switch

  And: Able to perform switch sync which does nothing

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections

• Able to validate switch with 'misconfigured' meters

  When: Create a flow

  And: Change bandwidth for the created flow directly in DB so that system thinks the installed meter is misconfigured

  And: Validate src and dst switches

  Then: Meters info is moved into the 'misconfigured' section

  And: Reason is specified why meter is misconfigured

  And: The rest fields of 'meter' section are empty

  And: Created rules are still stored in the 'proper' section

  And: Flow validation shows discrepancies

  When: Restore correct bandwidth via DB

  Then: Misconfigured meters are moved into the 'proper' section

  And: Flow validation shows no discrepancies

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections

• Able to validate and sync a switch with missing ingress rule + meter

  When: Create a flow

  And: Remove created meter on the srcSwitch

  Then: Meters info/rules are moved into the 'missing' section on the srcSwitch

  And: Meters info/rules are NOT moved into the 'missing' section on the dstSwitch

  When: Synchronize switch with missing rule and meter

  Then: Repeated validation shows no missing entities

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections

• Able to validate and sync a switch with missing ingress rule (unmetered)

  When: Create a flow

  And: Remove ingress rule on the srcSwitch

  Then: Ingress rule is moved into the 'missing' section on the srcSwitch

  When: Synchronize switch with missing unmetered rule

  Then: Repeated validation shows no missing entities

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections

• Able to validate and sync a switch with missing transit rule

  Given: Two active not neighboring switches

  And: Create an intermediate-switch flow

  When: Delete created rules on the transit

  Then: Rule info is moved into the 'missing' section

  When: Synchronize the switch

  Then: Repeated validation shows no discrepancies

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections on all involved switches

• Able to validate and sync a switch with missing egress rule

  Given: Two active not neighboring switches

  And: Create an intermediate-switch flow

  When: Delete created rules on the srcSwitch

  Then: Rule info is moved into the 'missing' section on the srcSwitch

  And: Rule info is NOT moved into the 'missing' section on the dstSwitch and transit switches

  When: Synchronize the switch

  Then: Repeated validation shows no discrepancies

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections on all involved switches

• Able to validate and sync an excess ingress/egress/transit rule + meter

  Given: Two active not neighboring switches

  And: Create an intermediate-switch flow

  When: Create excess rules on switches

  Then: Switch validation shows excess rules and store them in the 'excess' section

  And: Excess meter is shown on the srcSwitch only

  When: Try to synchronize every involved switch

  Then: System deletes excess rules and meters

  When: Delete the flow

  Then: Check that the switch validate request returns empty sections on all involved switches

  Cleanup:

• Able to validate and sync a switch with missing 'vxlan' ingress/transit/egress rule + meter

  Given: Two active not neighboring VXLAN supported switches

  And: Create a flow with vxlan encapsulation

  And: Remove required rules and meters from switches

  Expect: Switch validation shows missing rules and meters on every related switch

  When: Try to synchronize all switches

  Then: System installs missing rules and meters

  And: Switch validation doesn't complain about missing rules and meters

  And: Rules are synced correctly

• Able to validate and sync a missing 'protected path' egress rule

  Given: A flow with protected path

  Expect: Upon validation all rules are stored in the 'proper' section

  When: Delete rule of protected path on the srcSwitch (egress)

  Then: Deleted rule is moved to the 'missing' section on the srcSwitch

  And: Rest switches are not affected by deleting the rule on the srcSwitch

  When: Synchronize switch with a missing protected path egress rule

  Then: Switch validation no longer shows missing rules

• Able to validate and sync a missing 'connected device' #data.descr rule

  Given: A flow with enabled connected devices

  Expect: Switch validation puts connected device lldp rule into 'proper' section

  When: Remove the connected device rule

  Then: Switch validation puts connected device rule into 'missing' section

  When: Synchronize the switch

  Then: Switch validation no longer shows any discrepancies in rules nor meters

  When: Delete the flow

  Then: Switch validation is empty

• Able to filter results using request query, and asExpected field verification

  Given: Create a flow

  When: Perform result filtering

  And: Result contains only included sections

  Then: Delete the flow

  And: Check that the switch validate request returns empty sections

• Able to validate switch using #apiVersion API when GRPC is down

  Given: Random switch without LAG feature enabled

  And: GRPC container is down

  And: Switch has a new feature artificially added directly to DB

  When: Validate switch

  Then: Validation is successful

  Cleanup:

---

ISL-related

---

Bfd

BFD stands for Bidirectional Forwarding Detection. For now tested only on Noviflow switches. Main purpose is to detect ISL failure on switch level, which is times faster than a regular controller-involved discovery mechanism

• Able to create a valid BFD session between two Noviflow switches

  Given: An a-switch ISL between two Noviflow switches with BFD and RTL

  When: Create a BFD session on the ISL without props

  Then: Response reflects the requested bfd session with default prop values

  And: Link reflects that bfd is up

  And: Get link bfd API shows bfd props for link src/dst

  When: Interrupt ISL connection by breaking rule on a-switch

  Then: ISL immediately gets failed because bfd has higher priority than RTL

  And: Cost of ISL is unchanged and round trip latency status is ACTIVE

  When: Restore connection

  Then: ISL is rediscovered and bfd status is 'up'

  When: Remove existing BFD session

  Then: Bfd field is removed from isl

  And: Get BFD API shows '0' values in props

  When: Interrupt ISL connection by breaking rule on a-switch

  Then: ISL fails ONLY after discovery timeout

• Reacting on BFD events can be turned on/off by a feature toggle

  Given: An a-switch ISL between two Noviflow switches with BFD enabled

  When: Set BFD toggle to 'off' state

  And: Interrupt ISL connection by breaking rule on a-switch

  Then: ISL does not get FAILED immediately

  And: ISL fails after discovery timeout

  When: Set BFD toggle back to 'on' state and restore the ISL

  And: Again interrupt ISL connection by breaking rule on a-switch

  Then: ISL immediately gets failed

• Deleting a failed BFD link also removes the BFD session from it

  Given: An inactive a-switch link with BFD session

  When: Delete the link

  And: Discover the removed link again

  Then: Discovered link shows no bfd session

  And: Acts like there is no BFD session (fails only after discovery timeout)

• System is able to rediscover failed link after deleting BFD session

  Given: An interrupted a-switch ISL with BFD session

  When: Remove existing BFD session

  And: Restore connection

  Then: ISL is rediscovered

• Able to create/update BFD session with custom properties

  Given: An ISL between two Noviflow switches

  When: Create bfd session with custom properties

  Then: 'get ISL' and 'get BFD' api reflect the changes

  When: Update bfd session with custom properties

  Then: 'get ISL' and 'get BFD' api reflect the changes

• Unable to create bfd with #data.descr

  Given: An ISL between two Noviflow switches

  When: Try enabling bfd with forbidden properties

  Then: Error is returned

• Able to CRUD BFD sessions using v1 API

  Given: An ISL between two Noviflow switches

  When: Create a BFD session using v1 API

  Then: Response reports successful installation of the session

  And: Link reflects that bfd is up

  And: Get link bfd API shows default bfd props for link src/dst

  When: Disable bfd using v1 API

  Then: Response reports successful de-installation of the session

  And: Link reflects that bfd is removed

  And: Get link bfd API shows 'zero' bfd props for link src/dst

IslMinPortSpeed

Minimal port speed value is chosen for ISL capacity. Sometimes an ISL have different port speed on its edges. In that case, we need to set ISL capacity and all bandwidth parameters according to minimal speed value. Eg. 10G on one side, and 1G on another side, the ISL should have a 1G capacity.

• System sets min port speed for isl capacity

  Given: Two ports with different port speed

  When: Replug one end of the connected link to the destination switch(isl.srcSwitchId -> newDst.srcSwitchId)

  Then: Max bandwidth of new ISL is equal to the minimal port speed

IslReplug

Verify scenarios around replugging ISLs between different switches/ports.

• Round-trip ISL status changes to MOVED when replugging it into another switch

  Given: A connected a-switch link, round-trip-enabled

  And: A non-connected a-switch link with round-trip support

  When: Replug one end of the connected link to the not connected one

  Then: Replugged ISL status changes to MOVED

  And: Round trip status is not ACTIVE for the 'moved' ISL in both directions

  And: New ISL becomes DISCOVERED

  When: Replug the link back where it was

  Then: Original ISL becomes DISCOVERED again

  And: Replugged ISL status changes to MOVED

  When: Remove the MOVED ISL

  Then: Moved ISL is removed

  And: The src and dst switches of the isl pass switch validation

• ISL status changes to MOVED when replugging ISL into another switch

  Given: A connected a-switch link

  And: A non-connected a-switch link

  When: Replug one end of the connected link to the not connected one

  Then: Replugged ISL status changes to MOVED

  And: New ISL becomes DISCOVERED

  When: Replug the link back where it was

  Then: Original ISL becomes DISCOVERED again

  And: Replugged ISL status changes to MOVED

  When: Remove the MOVED ISL

  Then: Moved ISL is removed

  And: The src and dst switches of the isl pass switch validation

• New potential self-loop ISL (the same port on the same switch) is not getting discovered when replugging

  Given: A connected a-switch link

  When: Replug one end of the link into 'itself'

  Then: Replugged ISL status changes to FAILED

  And: The potential self-loop ISL is not present in the list of ISLs

  When: Replug the link back where it was

  Then: Original ISL becomes DISCOVERED again

  Cleanup:

• New potential self-loop ISL (different ports on the same switch) is not getting discovered when replugging

  Given: Two a-switch links on a single switch

  When: Plug an ISL between two ports on the same switch

  Then: The potential self-loop ISL is not present in the list of ISLs (wait for discovery interval)

  And: Self-loop rule packet counter is incremented and logged in tsdb

  Cleanup:

• User is able to replug ISL with enabled BFD, receive new ISL, enable bfd on it and replug back

  Given: An ISL with BFD and ability to replug

  When: Replug a bfd-enabled link to some other free port

  Then: Old ISL becomes Moved, new ISL is discovered

  And: Bfd on Moved ISL reports 'down' status

  When: Turn on BFD for new ISL

  Then: BFD is turned on according to getLink API

  When: Replug a new link back where it was before

  Then: Initial ISL becomes Discovered again, replugged ISL becomes Moved

  And: Bfd on Moved ISL reports 'down' status

  And: Bfd on Discovered ISL reports 'up' status

LinkMaintenance

• Maintenance mode can be set/unset for a particular link

  Given: An active link

  When: Set maintenance mode for the link

  Then: Maintenance flag for forward and reverse ISLs is really set

  And: Cost for ISLs is not changed

  When: Unset maintenance mode from the link

  Then: Maintenance flag for forward and reverse ISLs is really unset

  And: Cost for ISLs is changed to the default value

• Flows can be evacuated (rerouted) from a particular link when setting maintenance mode for it

  Given: Two active not neighboring switches with two possible paths at least

  And: Create a couple of flows going through these switches

  When: Set maintenance mode without flows evacuation flag for the first link involved in flow paths

  Then: Flows are not evacuated (rerouted) and have the same paths

  When: Set maintenance mode again with flows evacuation flag for the same link

  Then: Flows are evacuated (rerouted)

  And: Link under maintenance is not involved in new flow paths

• Y-Flows can be evacuated (rerouted) from a particular link when setting maintenance mode for it

  Given: Switch triplet with two possible paths at least for non-neighbouring switches

  And: Create Y-Flows going through selected switch triplet

  When: Set maintenance mode without flows evacuation flag for the first link involved in flow paths

  Then: Y-Flows are not evacuated (rerouted) and have the same paths

  When: Set maintenance mode again with flows evacuation flag for the same link

  Then: Y-Flows are evacuated (rerouted) and link under maintenance is not involved in new flow paths

  And: Link under maintenance is not involved in new Y-Flow paths

• Both Y-Flow and Flow can be evacuated (rerouted) from a particular link when setting maintenance mode for it

  Given: Switch triplet with active switches

  And: Create Y-Flows going through selected switch triplet

  And: Switch pair has been selected based on Y-Flow used Isl

  And: Create Flow going through selected switch pair

  When: Set maintenance mode without flows evacuation flag for the first link involved in flow paths

  Then: Both Y-Flow and Flow are not evacuated (rerouted) and have the same paths

  When: Set maintenance mode again with flows evacuation flag for the same link

  Then: Both Y-Flow and Flow are evacuated (rerouted)

  And: Link under maintenance is not involved in new flow paths

• Flows are rerouted to a path with link under maintenance when there are no other paths available

  Given: Two active not neighboring switches with two possible paths at least

  And: Create a couple of flows going through these switches

  And: Make only one alternative path available for both flows

  And: Set maintenance mode for the first link involved in alternative path

  When: Force flows to reroute by bringing port down on the source switch

  Then: Flows are rerouted to alternative path with link under maintenance

LinkProperties

• Link props are created with empty values when sending an invalid link props key

  When: Send link property request with invalid character

  Then: Response states that operation succeeded

  And: Link props are created but with empty values

• Unable to create link property with invalid switchId format

  When: Try creating link property with invalid switchId format

  Then: Response with error is received

• Unable to create link property with non-numeric value for #key

  When: Try creating link property with non-numeric values

  Then: Human readable error is returned

• Searching for link props with #data.descr

  Given: Preparing system for the following search

  When: Get link properties with search query

  Then: Returned props list match expected

  Cleanup:

• Updating cost, max bandwidth and description via link props actually updates cost, max bandwidth and description on ISLs

  Given: An active ISL

  When: Create link props of ISL to update cost and max bandwidth on the forward and reverse directions

  Then: Cost on forward and reverse ISLs is really updated

  And: Max bandwidth and description on forward and reverse ISLs is really updated as well

  When: Update link props on the forward direction of ISL to update cost one more time

  Then: Forward and reverse directions of the link props are really updated

  And: Cost on forward and reverse ISLs is really updated

  When: Delete link props

  Then: Cost on ISLs is changed to the default value

  And: Max bandwidth and description on forward and reverse ISLs is changed to the initial value as well

• Newly discovered link gets cost and max bandwidth from link props

  Given: An active ISL

  And: Bring port down on the source switch

  And: Delete the link

  And: Set cost and max bandwidth on the deleted link via link props

  When: Bring port up on the source switch to discover the deleted link

  Then: The discovered link gets cost from link props

  And: The discovered link gets max bandwidth from link props as well

Link

RoundTripIsl

• Isl with round-trip properly changes status after port events(#descr)

  Given: Round-trip ISL with a-switch

  When: Port down event happens

  And: Port up event happens, but traffic goes only in one direction

  Then: ISL is not getting discovered

  When: Traffic starts to flow in both directions

  Then: ISL gets discovered

• A round trip latency ISL doesn't go down when one switch lose connection to FL

  Given: A switch with/without round trip latency ISLs

  When: Simulate connection lose between the switch and FL, the switch becomes DEACTIVATED and remains operable

  And: Wait discoveryTimeout

  Then: All non round trip latency ISLs are FAILED

  And: All round trip latency ISLs are still DISCOVERED (the system uses round trip latency status for ISL alive confirmation)

• A round trip latency ISL goes down when both switches lose connection to FL

  Given: A round trip latency ISL

  When: Switches lose connection to FL, switches become DEACTIVATED but keep processing packets

  Then: The round trip latency ISL is FAILED (because round_trip_status is not available in DB for current ISL on both switches)

• A round trip latency ISL goes down when the src switch lose connection to FL and round trip latency rule is removed on the dst switch

  Given: A round trip latency ISL

  And: Round trip status is ACTIVE for the given ISL in both directions

  When: Simulate connection lose between the src switch and FL, switches become DEACTIVATED and remain operable

  Then: Round trip status for forward direction is not available and ACTIVE in reverse direction

  When: Delete ROUND_TRIP_LATENCY_RULE_COOKIE on the dst switch

  Then: The round trip latency ISL is FAILED

  And: Round trip status is not available for the given ISL in both directions

  When: Restore connection between the src switch and FL

  Then: Round trip isl is DISCOVERED

  And: Round trip status is available for the given ISL in forward direction only

  When: Install ROUND_TRIP_LATENCY_RULE_COOKIE on the dst switch

  Then: Round trip status is available for the given ISL in both directions

• A round trip latency ISL goes down when portDiscovery property is disabled on the src/dst ports

  Given: A round trip latency ISL

  When: Disable portDiscovery on the srcPort

  Then: Isl is still DISCOVERED

  When: Disable portDiscovery property on the dstPort

  Then: Status of the link is changed to FAILED

  When: Enable portDiscovery on the src/dst ports

  Then: ISL is rediscovered

• Able to delete failed ISL without force if it was discovered with disabled portDiscovery on a switch

  Given: A deleted round trip latency ISL

  When: Disable portDiscovery on the srcPort

  And: Revive the ISL back (bring switch port up)

  Then: The src/dst switches are valid

  When: Disable portDiscovery on the dstPort

  Then: The ISL is failed

  When: Delete the ISL without the 'force' option

  Then: The ISL is deleted

UnstableIsl

• ISL is NOT considered 'unstable' due to failing connection between switches (not port down)

  Given: ISL going through a-switch with link props created

  When: Remove a-switch rules to break link between switches

  Then: Status of forward and reverse ISLs becomes 'FAILED'

  And: Isl is not being 'unstable'

  When: Add a-switch rules to restore connection

  Then: Isl is not being 'unstable'

• ISL is not considered unstable after deactivating/activating switch

  Given: A switch that does not support round trip isl latency

  And: Cost of related ISLs

  When: Deactivate the switch

  Then: Switch ISL is not 'unstable'

  When: Activate the switch

  Then: Switch ISL is not 'unstable'

• ISL is marked as 'unstable' after port down and system takes it into account during flow creation

  Given: Two active neighboring switches with two parallel links

  And: Two possible paths for further manipulation with them

  And: All alternative paths are unavailable (bring ports down on the srcSwitch)

  And: First path is unstable (due to bringing port down/up)

  And: Cost of stable path is more preferable than the cost of unstable path (before penalties)

  When: Create a flow

  Then: Flow is created on the stable path(secondPath)

  When: Mark first path as stable(update the 'time_unstable' field in db)

  And: Reroute the flow

  Then: Flow is rerouted

---

Flow-related

---

AutoReroute

Verify different cases when Kilda is supposed to automatically reroute certain flow(s).

• Flow is rerouted when one of the #description flow ISLs fails

  Given: A flow with one alternative path at least

  When: Fail a flow ISL (bring switch port down)

  Then: The flow was rerouted after reroute delay

• Strict bandwidth true: Flow status is set to DOWN after reroute if no alternative path with enough bandwidth

  Given: A flow with one alternative path at least

  And: Alt path ISLs have not enough bandwidth to host the flow

  When: Fail a flow ISL (bring switch port down)

  Then: Flow history shows 3 retry attempts, eventually bringing flow to Down

  And: Flow path is unchanged

  When: Try to manually reroute the Down flow, while there is still not enough bandwidth

  Then: Error is returned, stating a 'not enough bandwidth' reason

  And: Flow history shows more reroute attempts after manual command

  And: Flow remains Down and on the same path

  When: Broken ISL on the original path is back online

  Then: The flow state has been changed to UP without rerouting(remains initial path)

• Single switch flow changes status on switch up/down events

  Given: Single switch flow

  When: The switch is disconnected

  Then: Flow becomes 'Down'

  When: Other isl fails

  Then: Flow remains 'DOWN'

  When: Other isl is back online

  Then: Flow remains 'DOWN'

  When: The switch is connected back

  Then: Flow becomes 'Up'

  And: Flow is valid

• Flow goes to 'Down' status when one of the flow ISLs fails and there is no alt path to reroute

  Given: A flow without alternative paths

  When: One of the flow ISLs goes down

  Then: The flow becomes 'Down'

  When: ISL goes back up

  Then: The flow becomes 'Up'

• Flow in 'Down' status is rerouted when discovering a new ISL

  Given: A flow between neighboring switches with one alternative path at least

  When: Bring all ports down on the source switch that are involved in the current and alternative paths

  Then: The flow goes to 'Down' status

  When: Bring all ports up on the source switch that are involved in the alternative paths

  Then: The flow goes to 'Up' status

  And: The flow was rerouted

• Flow in 'Up' status is not rerouted when discovering a new ISL and more preferable path is available

  Given: A flow with one alternative path at least

  And: Make the current flow path less preferable than others

  When: One of the links not used by flow goes down

  And: Failed link goes up

  Then: The flow is not rerouted and doesn't use more preferable path

• Flow in 'Up' status is not rerouted when connecting a new switch and more preferable path is available

  Given: A flow with one alternative path at least

  And: Make the current flow path less preferable than others

  When: Disconnect one of the switches not used by flow

  Then: The switch is really disconnected from the controller

  When: Connect the switch back to the controller

  Then: The switch is really connected to the controller

  And: The flow is not rerouted and doesn't use more preferable path

• Flow is not rerouted when one of the flow ports goes down

  Given: An intermediate-switch flow with one alternative path at least

  And: Make the current flow path less preferable than others

  When: Bring the flow port down on the source switch

  Then: The flow is not rerouted

  When: Bring the flow port down on the destination switch

  Then: The flow is not rerouted

• Flow in 'UP' status is not rerouted after switchUp event

  Given: Two active neighboring switches which support round trip latency

  And: A flow on the given switch pair

  When: Deactivate the src switch

  Then: Flow is UP

  When: Activate the src switch

  Then: System doesn't try to reroute the flow on the switchUp event because flow is already in UP state

• Flow is not rerouted when switchUp event appear for a switch which is not related to the flow

  Given: Given a flow in DOWN status on neighboring switches

  When: Generate switchUp event on switch which is not related to the flow

  Then: Flow is not triggered for reroute due to switchUp event because switch is not related to the flow

• System properly handles multiple flow reroutes if ISL on new path breaks while first reroute is in progress

  Given: Switch pair that have at least 3 paths and 2 paths that have at least 1 common isl

  And: A flow over these switches that uses one of the desired paths that have common ISL

  And: A potential 'backup' path that shares common isl has the preferred cost (will be preferred during reroute)

  When: An ISL which is unique for current path breaks, leading to a flow reroute

  And: Right when reroute starts: an ISL which is common for current path and potential backup path breaks too, triggering one more reroute of the current path

  Then: System reroutes the flow twice and flow ends up in UP state

  And: New flow path avoids both main and backup paths as well as broken ISLs

  And: Flow is pingable

AutoRerouteIsolated

Verify different cases when Kilda is supposed to automatically reroute certain flow(s).

• Flow in 'Down' status is rerouted after switchUp event

  Given: First switch pair with two parallel links and two available paths

  And: Second switch pair where the srс switch from the first switch pair is a transit switch

  And: First flow without transit switches

  And: Second flow with transit switch

  When: Disconnect the src switch of the first flow from the controller

  And: Mark the switch as ACTIVE in db

  And: Init auto reroute (bring ports down on the dstSwitch)

  Then: System tries to reroute a flow with transit switch

  And: Flows are 'Down'

  When: Connect the switch back to the controller

  Then: System tries to reroute the flow on switchUp event

• Strict bandwidth false: Flow is rerouted even if there is no available bandwidth on alternative path, sets status to Degraded

  Given: A flow with one alternative path at least

  And: Alt path ISLs have not enough bandwidth to host the flow

  When: Fail a flow ISL (bring switch port down)

  Then: Flow history shows two reroute attempts, second one succeeds with ignore bw

  And: The flow has changed path and has DEGRADED status

  When: Try to manually reroute the degraded flow, while there is still not enough bandwidth

  Then: Error is returned, stating a readable reason

  And: Flow remains DEGRADED and on the same path

  When: Trigger auto reroute by blinking not involved(in flow path) isl

  Then: System tries to reroute the DEGRADED flow

  And: Flow remains DEGRADED and on the same path

  When: Broken ISL on the original path is back online

  Then: Flow is rerouted to the original path to UP state

Bandwidth

Verify that ISL's bandwidth behaves consistently and does not allow any oversubscribtions etc.

• Available bandwidth on ISLs changes respectively when creating/updating/deleting a flow

  Given: Two active not neighboring switches

  When: Create a flow with a valid bandwidth

  Then: Available bandwidth on ISLs is changed in accordance with flow maximum bandwidth

  When: Update the flow with a valid bandwidth

  Then: The flow is successfully updated and has 'Up' status

  And: Available bandwidth on ISLs is changed in accordance with new flow maximum bandwidth

  When: Delete the flow

  Then: Available bandwidth on ISLs is changed to the initial value before flow creation

• Longer path is chosen in case of not enough available bandwidth on a shorter path

  Given: Two active switches with two possible flow paths at least

  When: Create a flow to reduce available bandwidth on links of the expected preferable path

  Then: The flow is really built through the expected preferable path

  When: Create another flow. One path is shorter but available bandwidth is not enough, another path is longer

  Then: The flow is built through longer path where available bandwidth is enough

• Unable to exceed bandwidth limit on ISL when creating a flow

  Given: Two active switches

  When: Create a flow with a bandwidth that exceeds available bandwidth on ISL

  Then: The flow is not created because flow path should not be found

• Unable to exceed bandwidth limit on ISL when updating a flow

  Given: Two active switches

  When: Create a flow with a valid bandwidth

  And: Update the flow with a bandwidth that exceeds available bandwidth on ISL

  Then: The flow is not updated because flow path should not be found

• Able to exceed bandwidth limit on ISL when creating/updating a flow with ignore_bandwidth=true

  Given: Two active switches

  When: Create a flow with a bandwidth that exceeds available bandwidth on ISL (ignore_bandwidth=true)

  Then: Available bandwidth on ISLs is not changed in accordance with flow maximum bandwidth

  When: Update the flow with a bandwidth that exceeds available bandwidth on ISL (ignore_bandwidth = true)

  Then: The flow is successfully updated and has 'Up' status

  And: Available bandwidth on ISLs is not changed in accordance with new flow maximum bandwidth

• Able to update bandwidth to maximum link speed without using alternate links

  Given: Two active neighboring switches

  When: Create a flow with a valid small bandwidth

  Then: Only one link is involved in flow path

  When: Update flow bandwidth to maximum link speed

  Then: The flow is successfully updated and has 'Up' status

  And: The same path is used by updated flow

• System doesn't allow to exceed bandwidth limit on ISL while updating a flow with ignore_bandwidth=false

  Given: Two active switches

  When: Create a flow with a bandwidth that exceeds available bandwidth on ISL (ignore_bandwidth=true)

  Then: Available bandwidth on ISLs is not changed in accordance with flow maximum bandwidth

  When: Update the flow (ignore_bandwidth = false)

  Then: Human readable error is returned

  And: The flow is not updated and has 'Up' status

  And: Available bandwidth on ISLs is not changed

• Unable to exceed bandwidth limit on ISL when creating a flow [v1 api]

  Given: Two active switches

  When: Create a flow with a bandwidth that exceeds available bandwidth on ISL

  Then: The flow is not created because flow path should not be found

ConnectedDevices

Verify ability to detect connected devices per flow endpoint (src/dst). Verify allocated Connected Devices resources and installed rules.

• Able to create a #flowDescr flow with lldp and arp enabled on #devicesDescr, encapsulation #data.encapsulation, swPair: #hw

  Given: A flow with enabled or disabled connected devices

  When: Create a flow with connected devices

  Then: Flow has been created successfully and validated

  And: Source and destination switches pass validation

  And: ARP/LLDP rules/meters have been installed

  When: Devices send lldp and arp packets on each flow endpoint

  Then: Getting connecting devices shows corresponding devices on each endpoint if enabled

  When: Delete the flow

  Then: Delete action removed all rules and meters

• Able to update flow from srcDevices=#oldSrcEnabled, dstDevices=#oldDstEnabled to srcDevices=#newSrcEnabled, dstDevices=#newDstEnabled

  Given: Flow with specific configuration of connected devices

  When: Update the flow with connected devices

  Then: Flow has been created successfully and validated

  And: ARP/LLDP rules have been installed

  And: Source and destination switches pass validation (includes meters check)

  When: Devices send lldp and arp packets on each flow endpoint

  Then: Getting connecting devices shows corresponding devices on each endpoint according to updated status

• Able to detect devices on a single-switch different-port flow

  Given: A flow between different ports on the same switch

  When: Device connects to src endpoint and send lldp and arp packets

  Then: LLDP and ARP connected devices are recognized and saved

  When: Remove the flow

  And: Try to get connected devices for removed flow

  Then: Error is returned

• Able to swap flow paths with connected devices (srcDevices=#srcEnabled, dstDevices=#dstEnabled)

  Given: Protected flow with connected devices

  And: Created protected flow with enabled or disabled connected devices

  When: Swap flow paths

  Then: Flow has been swapped successfully and validated

  And: ARP/LLDP rules have been installed

  And: Source and destination switches pass validation (includes meters check)

  When: Devices send lldp and arp packets on each flow endpoint

  Then: Getting connecting devices shows corresponding devices on each endpoint

• Able to handle 'timeLastSeen' field when receive repeating LLDP packets from the same device

  Given: Flow with connected devices has been created successfully

  When: Connected device sends lldp packet

  Then: Device is registered for the flow, with timeLastSeen and timeFirstSeen values

  When: Same packet is sent again

  Then: timeLastSeen is updated, timeFirstSeen remains the same

• Able to handle 'timeLastSeen' field when receive repeating ARP packets from the same device

  Given: Flow with connected devices has been created successfully

  And: A connected device

  When: Device sends ARP packet

  Then: Device is registered for the flow, with timeLastSeen and timeFirstSeen values

  When: Same packet is sent again

  Then: timeLastSeen is updated, timeFirstSeen remains the same

• Able to detect different devices on the same port (LLDP)

  Given: Flow with connected devices has been created successfully

  When: Two completely different lldp packets are sent

  Then: 2 devices are registered for the flow

  When: Same device (same mac address) sends lldp packet with updated port number

  Then: Device is recognized as new one and total of 3 devices are registered for the flow

  When: Request devices list with 'since' param equal to last registered device

  Then: Only 1 device is returned (the latest registered)

• Able to detect different devices on the same port (ARP)

  Given: Flow with connected devices has been created successfully

  When: Two completely different ARP packets are sent

  Then: 2 arp devices are registered for the flow

  When: Same device (same IP address) sends ARP packet with updated mac address

  Then: Device is recognized as new one and total of 3 arp devices are registered for the flow

  When: Request devices list with 'since' param equal to last registered device

  Then: Only 1 device is returned (the latest registered)

• System properly detects devices if feature is 'off' on switch level and 'on' on flow level

  Given: A switch with devices feature turned off

  When: Devices send arp and lldp packets into a free switch port

  Then: No devices are detected for the switch

  When: Flow is created on a target switch with devices feature 'on'

  And: Flow is valid and pingable

  And: Device sends an lldp+arp packet into a flow port on that switch (with a correct flow vlan)

  Then: LLDP and ARP devices are registered as flow devices

  And: Devices are registered per-switch

• System properly detects devices if feature is 'on' on switch level and 'off' on flow level

  Given: A switch with devices feature turned on

  And: Flow is created on a target switch with devices feature 'off'

  When: Devices send lldp and arp packets into a flow port

  Then: ARP and LLDP devices are registered per-switch

  Then: Devices are registered as flow devices

• Able to detect devices on free switch port (no flow or isl)

  Given: A switch with devices feature turned on

  When: Devices send lldp and arp packets into a free port

  Then: Corresponding devices are detected on a switch port

• Able to distinguish devices between default and non-default single-switch flows (#descr)

  Given: A switch with devices feature turned on

  And: A single-sw flow with devices feature 'on'

  And: A single-sw default flow with devices feature 'on'

  When: Devices send lldp and arp packets into a flow port with flow vlan

  Then: Corresponding devices are detected on a switch port with reference to corresponding non-default flow

  And: Devices are also visible as flow devices

  And: Devices are NOT visible as a default flow devices

  When: Other devices send lldp and arp packets into a flow port with vlan different from flow vlan

  Then: Corresponding devices are detected on a switch port with reference to corresponding default flow

  And: Devices are not visible as a flow devices

  And: Devices are visible as a default flow devices

  When: Other devices send lldp and arp packets into a flow port with double vlans different from flow vlan

  Then: Corresponding devices are detected on a switch port with reference to corresponding default flow

  And: Devices are not visible as a flow devices

  And: Devices are visible as a default flow devices

• System properly detects device vlan in case of #descr

  Given: Flow with connected devices

  And: A flow with enbaled connected devices, #descr

  When: Devices send lldp and arp packets on each flow endpoint

  Then: Devices are registered for the flow on src and dst

  And: Devices are registered on src switch

  And: Device are registered on dst switch

• System detects devices for a qinq(iVlan=#vlanId oVlan=#innerVlanId) flow with lldp and arp enabled on the src switch

  Given: Two switches connected to traffgen

  And: A QinQ flow with enabled connected devices on src has been created

  Expect: Flow is valid

  And: ARP/LLDP rules have been installed

  And: Source and destination switches pass validation (includes meters check)

  When: Devices send lldp and arp packets on each flow endpoint

  Then: Getting connecting devices shows corresponding devices on src endpoint

  And: Devices are registered on the src switch only

  And: Delete the flow

  And: Delete action removed all rules and meters

• System doesn't detect devices only if vlan match with outerVlan of qinq flow

  Given: Two switches connected to traffgen

  And: A QinQ flow with enabled connected devices

  When: Devices send lldp and arp packets on src flow endpoint and match outerVlan only

  Then: Getting connecting devices doesn't show corresponding devices on src endpoint

  And: Devices are not registered on the src/dst switches

• Able to detect devices on a qinq single-switch different-port flow

  Given: A flow between different ports on the same switch

  When: Device connects to src endpoint and send lldp and arp packets

  Then: LLDP and ARP connected devices are recognized and saved

  And: Devices are registered on the switch

• Able to detect devices when two qinq single-switch different-port flows exist with the same outerVlanId

  Given: Two flows between different ports on the same switch with the same outerVlanId

  When: Device connects to src endpoint and send lldp and arp packets for flow1 only

  Then: LLDP and ARP connected devices are recognized for flow1

  And: LLDP and ARP connected devices are not recognized for flow2

  And: Devices are registered on the switch

• Switch is not containing extra rules after connected devices removal

  When: A switch with devices feature turned on

  Then: The appropriate switch-related ARP/LLDP rules+meters are installed on switch

  And: Flow is created on a target switch with devices feature 'off'

  When: Turn LLDP and ARP detection off on switch

  And: Flow has been updated successfully

  Then: Check excess rules are not registered on device

  And: The appropriate switch-related ARP/LLDP rules+meters are removed from switch

• System starts detect connected after device properties turned on

  Given: A switch with devices feature turned off

  And: Flow is created on a target switch with devices feature 'off'

  When: update device properties and send lldp and arp packets on flow

  Then: Getting connecting devices show corresponding devices on src endpoint

  And: No connected devices for flow

• System stops receiving statistics if config is changed to 'off'

  Given: A switch with devices feature turned on

  And: Flow is created on a target switch with devices feature 'off'

  When: update device properties and send lldp and arp packets on flow

  Then: Getting connecting devices doesn't show corresponding devices on src endpoint

DefaultFlow

System allows to create default port(vlan=0) and simple flow(vlan=) on the same port. Default flow has lower priority than simple flow. Also system allows to pass tagged traffic via default flow.

• Systems allows to pass traffic via default/vlan and qinq flow when they are on the same port

  Given: At least 3 traffGen switches

  When: Create a vlan flow

  And: Create a default flow with the same srcSwitch and different dstSwitch

  And: Create a QinQ flow with the same src and dst switch

  Then: System allows traffic on the vlan flow

  And: System allows traffic on the default flow

  And: System allows traffic on the QinQ flow

• System allows tagged traffic via default flow(0<->0)

  Given: At least 2 traffGen switches

  When: Create a default flow

  Then: System allows tagged traffic on the default flow

• Unable to send traffic from simple flow into default flow and vice versa

  Given: At least 2 traffGen switches

  And: A default flow

  And: A simple flow

  When: Try to send traffic from simple flow into default flow and vice versa

  Then: System doesn't allow to send traffic in these directions

• Unable to create two default flow on the same port

  When: Create first default flow

  And: Try to create second default flow on the same port

  Then: Human readable error is returned

FlowAffinity

https://github.com/telstra/open-kilda/tree/develop/docs/design/solutions/pce-affinity-flows/

• Can create more than 2 affinity flows

  When: Create flow1

  And: Create flow2 with affinity to flow1

  And: Create flow3 with affinity to flow2

  Then: All flows have affinity with flow1

  And: Flows' histories contain 'affinityGroupId' information

  When: Delete flows

  Then: Flow1 history contains 'affinityGroupId' information in 'delete' operation

  And: Flow2 and flow3 histories does not have 'affinityGroupId' in 'delete' because it's gone after deletion of 'main' flow1

• Affinity flows are created close even if cost is not optimal, same dst

  Given: Two switch pairs with same dst and diverse paths available

  And: Existing flow over one of the switch pairs

  And: Potential affinity flow, which optimal path is diverse from the main flow, but it has a not optimal closer path

  When: Build affinity flow

  Then: Most optimal, but 'uncommon' to the main flow path is NOT picked, but path with least uncommon ISLs is chosen

  And: Path remains the same when manual reroute is called

• Affinity flows can have no overlapping switches at all

  Given: Two switch pairs with not overlapping short paths available

  And: First flow

  When: Create affinity flow

  Then: Affinity flow path has no overlapping ISLs with the first flow

• Affinity flow on the same endpoints #willOrNot take the same path if main path cost #exceedsOrNot affinity penalty

  Given: A neighboring switch pair with parallel ISLs

  And: First flow

  And: Isl which is taken by the main flow weighs more/less than the neighboring ISL, taking into account affinity penalty

  When: Create affinity flow on the same switch pair

  Then: It takes/doesn't take the path of the main flow

• Cannot create affinity flow if target flow has affinity with diverse flow

  Given: Existing flows with diversity

  When: Create affinity flow on the same switch pair

  Then: Error is returned

  When: Create affinity flow on the same switch pair

  Then: Error is returned

• Cannot create affinity flow if target flow has another diverse group

  Given: Existing flows with diversity

  When: Create an affinity flow that targets the diversity flow

  Then: Error is returned

• Able to create an affinity flow with a 1-switch flow

  Given: A one-switch flow

  When: Create an affinity flow targeting the one-switch flow

  Then: Both flows have affinity with flow1

  And: Affinity flow history contain 'affinityGroupId' information

• Error is returned if affinity_with references a non existing flow

  When: Create an affinity flow that targets non-existing flow

  Then: Error is returned

FlowCrud

"Verify CRUD operations and health of basic vlan flows on different types of switches. More specific cases like partialUpdate/protected/diverse etc. are covered in separate specifications

• Valid #data.description has been created (#verifyTraffic) and no rule discrepancies [#data.swPairDesc]

  Given: A flow

  Expect: No rule discrepancies on every switch of the flow

  And: No discrepancies when doing flow validation

  And: The flow allows traffic (only applicable flows are checked)

  When: Remove the flow

  Then: The flow is not present in NB

  And: ISL bandwidth is restored

  And: No rule discrepancies on every switch of the flow

• Able to create multiple flows with #data.description

  Given: Two potential flows that should not conflict

  When: Create the first flow

  And: Try creating a second flow with #data.description

  Then: Both flows are successfully created

• Able to create single switch single port flow with different vlan [#data.description]

  Given: A flow

  Expect: No rule discrepancies on the switch

  And: No discrepancies when doing flow validation

  When: Remove the flow

  Then: The flow is not present in NB

  And: No rule discrepancies on the switch after delete

• Able to validate flow with zero bandwidth

  Given: A flow with zero bandwidth

  When: Create a flow with zero bandwidth

  Then: Validation of flow with zero bandwidth must be succeed

• Unable to create single-switch flow with the same ports and vlans on both sides

  Given: Potential single-switch flow with the same ports and vlans on both sides

  When: Try creating such flow

  Then: Error is returned, stating a readable reason

• Unable to create flow with conflict data (#data.conflict)

  Given: A potential flow

  And: Another potential flow with #data.conflict

  When: Create the first flow

  And: Try creating the second flow which conflicts

  Then: Error is returned, stating a readable reason of conflict

  And: Main flow has been created successfully and in the UP state

  And: Conflict flow does not exist in the system

• A flow cannot be created with asymmetric forward and reverse paths

  Given: Two active neighboring switches with two possible flow paths at least and different number of hops

  And: Make all shorter(one isl) forward paths not preferable. Shorter reverse paths are still preferable

  When: Create a flow

  Then: The flow is built through one of the long paths

  And: The flow has symmetric forward and reverse paths even though there is a more preferable reverse path

• Error is returned if there is no available path to #data.isolatedSwitchType switch

  Given: A switch that has no connection to other switches

  When: Try building a flow using the isolated switch

  Then: Error is returned, stating that there is no path found for such flow

  And: Flow does not exist in the system

• Removing flow while it is still in progress of being set up should not cause rule discrepancies

  Given: A potential flow

  When: Init creation of a new flow

  And: Immediately remove the flow

  Then: System returns error as being unable to remove in progress flow

  And: Flow is not removed

  And: Flow eventually gets into UP state

  And: All related switches have no discrepancies in rules

• Unable to create a flow with #problem

  Given: A flow with #problem

  When: Try to create a flow

  Then: Flow is not created

• Unable to update to a flow with #problem

  Given: A flow

  When: Try to update the flow

  Then: Flow is not updated

• Unable to partially update to a flow with statistics vlan set to 0 or above 4095

  Given: A flow

  When: Try to partially update the flow

  Then: Flow is not updated

• Unable to create a flow on an isl port in case port is occupied on a #data.switchType switch

  Given: An isl

  When: Try to create a flow using isl port

  Then: Flow is not created

• Unable to create a flow on an isl port when ISL status is FAILED

  Given: An inactive isl with failed state

  When: Try to create a flow using ISL src port

  Then: Flow is not created

• Unable to create a flow on an isl port when ISL status is MOVED

  Given: An inactive isl with moved state

  When: Try to create a flow using ISL src port

  Then: Flow is not created

• System doesn't allow to create a one-switch flow on a DEACTIVATED switch

  Given: Disconnected switch

  When: Try to create a one-switch flow on a deactivated switch

  Then: Human readable error is returned

• System allows to CRUD protected flow

  Given: Two active not neighboring switches with two diverse paths at least

  When: Create flow with protected path

  Then: Flow is created with protected path

  And: Rules for main and protected paths are created

  And: Validation of flow must be successful

  When: Update flow: disable protected path(allocateProtectedPath=false)

  Then: Protected path is disabled

  And: Rules for protected path are deleted

• System allows to set/update description/priority/max-latency for a flow

  Given: Two active neighboring switches

  And: Value for each field

  When: Create a flow with predefined values

  Then: Flow is created with needed values

  When: Update predefined values

  Then: Flow is updated correctly

• System doesn't ignore encapsulationType when flow is created with ignoreBandwidth = true

  Given: Two active switches

  When: Create a flow with not supported encapsulation type on the switches

  Then: Human readable error is returned

• Flow status accurately represents the actual state of the flow and flow rules

  When: Create a flow on a long path

  Then: Flow status is changed to UP only when all rules are actually installed

  When: Delete flow

  Then: Flow is actually removed from flows dump only after all rules are removed

• Able to update a flow endpoint

  Given: Three active switches

  And: A vlan flow

  When: Update the flow: port number and vlan id on the src endpoint

  Then: Flow is really updated

  And: Flow history shows actual info into stateBefore and stateAfter sections

  And: Flow rules are recreated

  And: Flow is valid and pingable

  And: The src switch passes switch validation

  When: Update the flow: switch id on the dst endpoint

  Then: Flow is really updated

  And: Flow history shows actual info into stateBefore and stateAfter sections

  And: Flow rules are removed from the old dst switch

  And: Flow rules are installed on the new dst switch

  And: Flow is valid and pingable

  And: The new and old dst switches pass switch validation

• System reroutes flow to more preferable path while updating

  Given: Two active not neighboring switches with two possible paths at least

  And: A flow

  When: Make the current path less preferable than alternatives

  And: Update the flow

  Then: Flow is rerouted

  And: Flow is updated

  And: All involved switches pass switch validation

• System doesn't rebuild path for a flow to more preferable path while updating portNumber/vlanId

  Given: Two active switches connected to traffgens with two possible paths at least

  And: A flow

  When: Make the current path less preferable than alternatives

  And: Update the flow: vlanId on the src endpoint

  Then: Flow is really updated

  And: Flow path is not rebuild

  When: Update the flow: vlanId on the dst endpoint

  Then: Flow is really updated

  And: Flow path is not rebuild

  When: Update the flow: port number and vlanId on the src/dst endpoints

  Then: Flow is really updated

  And: Flow path is not rebuild

  And: Flow is valid

  When: Traffic starts to flow in both direction

  Then: Traffic flows in both direction

  And: All involved switches pass switch validation

• System allows to update single switch flow to multi switch flow

  Given: A single switch flow with enabled lldp/arp on the dst side

  When: Update the dst endpoint to make this flow as multi switch flow

  Then: Flow is really updated

  And: Flow is valid and pingable

  And: Involved switches pass switch validation

• Unable to create a flow with both strict_bandwidth and ignore_bandwidth flags

  When: Try to create a flow with strict_bandwidth:true and ignore_bandwidth:true

  Then: Bad Request response is returned

• Unable to update flow with incorrect id in request body

  Given: A flow

  When: Try to update flow with incorrect flow id in request body

  Then: Bad Request response is returned

• Unable to update flow with incorrect id in request path

  Given: A flow

  When: Try to update flow with incorrect flow id in request path

  Then: Bad Request response is returned

• Able to #method update with empty VLAN stats and non-zero VLANs (#5063)

  Given: A flow with non empty vlans stats and with src and dst vlans set to '0'

  When: Try to #method update flow with empty VLAN stats and non-zero VLANs

  Then: Flow is really updated

  When: Delete the flow

  Then: No excess rules left on the switches (#5141)

• Unable to update to a flow with maxLatencyTier2 higher as maxLatency)

  Given: A flow

  When: Try to update the flow

  Then: Bad Request response is returned

FlowCrudV1

Verify CRUD operations and health of most typical types of flows on different types of switches.

• Valid #data.description has been created (#verifyTraffic) and no rule discrepancies [#data.swPairDesc]

  Given: A flow

  Expect: No rule discrepancies on every switch of the flow

  And: No discrepancies when doing flow validation

  And: The flow allows traffic (only applicable flows are checked)

  When: Remove the flow

  Then: The flow is not present in NB

  And: ISL bandwidth is restored

  And: No rule discrepancies on every switch of the flow

• Able to create multiple flows with #data.description

  Given: Two potential flows that should not conflict

  When: Create the first flow

  And: Try creating a second flow with #data.description

  Then: Both flows are successfully created

• Able to create single switch single port flow with different vlan [#data.description]

  Given: A flow

  Expect: No rule discrepancies on the switch

  And: No discrepancies when doing flow validation

  When: Remove the flow

  Then: The flow is not present in NB

  And: No rule discrepancies on the switch after delete

• Able to validate flow with zero bandwidth

  Given: A flow with zero bandwidth

  When: Create a flow with zero bandwidth

  Then: Validation of flow with zero bandwidth must be succeed

• Unable to create single-switch flow with the same ports and vlans on both sides

  Given: Potential single-switch flow with the same ports and vlans on both sides

  When: Try creating such flow

  Then: Error is returned, stating a readable reason

• Unable to create flow with conflict data (#data.conflict)

  Given: A potential flow

  And: Another potential flow with #data.conflict

  When: Create the first flow

  And: Try creating the second flow which conflicts

  Then: Error is returned, stating a readable reason of conflict

• Unable to update flow when there are conflicting vlans((#data.conflict))

  Given: Two potential flows

  When: Create two flows

  And: Try updating the second flow which should conflict with the first one

  Then: Error is returned, stating a readable reason of conflict

• A flow cannot be created with asymmetric forward and reverse paths

  Given: Two active neighboring switches with two possible flow paths at least and different number of hops

  And: Make all shorter forward paths not preferable. Shorter reverse paths are still preferable

  When: Create a flow

  Then: The flow is built through one of the long paths

  And: The flow has symmetric forward and reverse paths even though there is a more preferable reverse path

• Error is returned if there is no available path to #data.isolatedSwitchType switch

  Given: A switch that has no connection to other switches

  When: Try building a flow using the isolated switch

  Then: Error is returned, stating that there is no path found for such flow

• Removing flow while it is still in progress of being created should not cause rule discrepancies

  Given: A potential flow

  When: Init creation of a new flow

  And: Immediately remove the flow

  Then: System returns error as being unable to remove in progress flow

  And: Flow is not removed

  And: Flow eventually gets into UP state

  And: All related switches have no discrepancies in rules

• Unable to create a flow on an isl port in case port is occupied on a #data.switchType switch

  Given: An isl

  When: Try to create a flow using isl port

  Then: Flow is not created

• Unable to update a flow in case new port is an isl port on a #data.switchType switch

  Given: An isl

  And: A flow has been created

  When: Try to edit port to isl port

  Then:

• Unable to create a flow on an isl port when ISL status is FAILED

  Given: An inactive isl with failed state

  When: Try to create a flow using ISL src port

  Then: Flow is not created

• Unable to create a flow on an isl port when ISL status is MOVED

  Given: An inactive isl with moved state

  When: Try to create a flow using ISL src port

  Then: Flow is not created

• Able to CRUD #flowDescription pinned flow

  When: Create a flow(pinned=true)

  Then: Pinned flow is created

  When: Update the flow (pinned=false)

  Then: The pinned option is disabled

• System doesn't allow to create a one-switch flow on a DEACTIVATED switch

  Given: A deactivated switch

  When: Create a flow

  Then: Human readable error is returned

• System recreates excess meter when flow is created with the same meterId

  Given: A Noviflow switch

  And: Create excess meters on the given switch

  When: Create several flows

  Then: Needed amount of flow was created

  And: Needed amount of meter was recreated

• System doesn't create flow when reverse path has different bandwidth than forward path on the second link

  Given: Two active not neighboring switches

  And: Select path for further manipulation with it

  And: Make all alternative paths unavailable (bring links down on the src/intermediate switches)

  And: Update reverse path to have not enough bandwidth to handle the flow

  When: Create a flow

  Then: Flow is not created

FlowDiversity

This test suite verifies the ability to create diverse flows in the system. Diverse flows are flows that should not overlap at all or the overlapping should be minimal. Such flows form a so-called diversity group. Ideally, the diversity group should not have flows with overlapping paths. But it depends on the cost of paths. The paths of flows from the same diversity group may overlap if the cost of each non-overlapping path is more than the cost of the overlapping path. The cost of paths for diverse flows is calculated in real time and consists of the following parameters:

1. The cost of ISL involved in the flow path (taken from DB);
2. (diversity.switch.cost) * (the number of diverse flows going through this switch);
3. (diversity.isl.cost) * (the number of diverse flows going through this ISL).

Refer to https://github.com/telstra/open-kilda/issues/1231 for more details.

• Able to create diverse flows

  Given: Two active neighboring switches with three not overlapping paths at least

  When: Create three flows with diversity enabled

  Then: Flow create response contains information about diverse flow

  And: All flows have diverse flow IDs in response

  And: All flows have different paths

  And: Flows' histories contain 'diverseGroupId' information

  When: Delete flows

  Then: Flows' histories contain 'diverseGroupId' information in 'delete' operation

• Able to update flows to become diverse

  Given: Two active neighboring switches with three not overlapping paths at least

  And: Create three flows

  When: Update the second flow to become diverse

  And: Second flow's history contains 'groupId' information

  Then: Update response contains information about diverse flow

  And: The flow became diverse and changed the path

  And: All flows except last one have the 'diverse_with' field

  When: Update the third flow to become diverse

  Then: The flow became diverse and all flows have different paths

• Able to update flows to become not diverse

  Given: Two active neighboring switches with three not overlapping paths at least

  And: Create three flows with diversity enabled

  And: Flow1 path is the most preferable

  When: Update the second flow to become not diverse

  Then: The flow became not diverse and rerouted to the more preferable path (path of the first flow)

  And: The 'diverse_with' field is removed

  And: The flow's history reflects the change of 'groupId' field

  When: Update the third flow to become not diverse

  Then: The flow became not diverse and rerouted to the more preferable path (path of the first flow)

  And: The 'diverse_with' field is removed

• Diverse flows are built through the same path if there are no alternative paths available

  Given: Two active neighboring switches with two not overlapping paths at least

  And: Create a flow going through these switches

  And: Make all alternative paths unavailable (bring ports down on the source switch)

  When: Create the second flow with diversity enabled

  Then: The second flow is built through the same path as the first flow

• Links and switches get extra cost that is considered while calculating diverse flow paths

  Given: Two active neighboring switches with three not overlapping paths at least

  And: Create a flow going through these switches

  And: Make each alternative path less preferable than the first flow path

  When: Create the second flow with diversity enabled

  Then: The flow is built through the most preferable path (path of the first flow)

  When: Create the third flow with diversity enabled

  Then: The flow is built through one of alternative paths because they are preferable already

• Able to get flow paths with correct overlapping segments stats (casual flows)

  Given: Two active neighboring switches with three not overlapping paths at least

  And: Create three flows with diversity enabled

  When: Get flow path for all flows

  Then: Flow path response for all flows has correct overlapping segments stats

• Able to get flow paths with correct overlapping segments stats (casual + single-switch flows)

  Given: Two active not neighboring switches

  And: Create a casual flow going through these switches

  And: Create a single-switch with diversity enabled on the source switch of the first flow

  And: Create a single-switch with diversity enabled on the destination switch of the first flow

  When: Get flow path for all flows

  Then: Flow path response for all flows has correct overlapping segments stats

  Cleanup:

• Able to update flow to become diverse and single-switch

  Given: Both switch pairs are with the same source switch

  And: Create two flows starting from the same switch

  When: Update the second flow to become diverse and single-switch

  Then: Update response contains information about diverse flow

• Able to create diverse flows [v1 api]

  Given: Two active neighboring switches with three not overlapping paths at least

  When: Create three flows with diversity enabled

  Then: All flows have diverse flow IDs in response

  And: All flows have different paths

FlowHistory

Verify that history records are created for the create/update actions. History record is created in case the create/update action is completed successfully.

• History records are created for the create/update actions using custom timeline

  When: Create a flow

  Then: History record is created

  And: Flow history contains all flow properties in the dump section

  When: Update the created flow

  Then: History record is created after updating the flow

  When: Delete the updated flow

  Then: History is still available for the deleted flow

  And: Flow history statuses returns all flow statuses for the whole life cycle

• History records are created for the create/update actions using default timeline

  When: Create a flow

  Then: History record is created

  When: Update the created flow

  Then: History record is created after updating the flow

  When: Delete the updated flow

  Then: History is still available for the deleted flow

• History records are created for the partial update actions #partialUpdateType

  Given: Flow has been created

  When: Update the created flow

  Then: History record is created after partial flow update

  And: History records are with correct flow details in the dump section

• History max_count cannot be <1

  When: Try to get history with max_count 0

  Then: Error due to invalid max_count returned

• Check history: #data.descr

  Expect: #data.descr

• Check history: timeBefore > timeAfter returns error

  When: Request history with timeBefore > timeAfter returns error

  Then: Error is returned

• Root cause is registered in flow history while rerouting

  Given: An active flow

  When: Deactivate the src switch and wait for links to change their state to FAILED

  Then: Flow goes DOWN

  And: The root cause('Switch is not active') is registered in flow history

• History records are created for the create/update actions using custom timeline [v1 api]

  When: Create a flow

  Then: History record is created

  When: Update the created flow

  Then: History record is created after updating the flow

  When: Delete the updated flow

  Then: History is still available for the deleted flow

FlowLoop

Flow loop feature designed for flow path testing. Loop provides additional flow rules on one of the terminating switch so any flow traffic is returned to switch-port where it was received. Such flow has 'looped=true' flag and supports all flow operations. When the loop removed system should restore the original flow rules. Enabling flowLoop in flow history is registered as the 'update' operation.

• Able to create flowLoop for a #data.flowDescription flow

  Given: An active and valid #data.flowDescription flow

  When: Create flowLoop on the src switch

  Then: Create flowLoop response contains flowId and src switchId

  And: Flow is UP and valid

  And: FlowLoop is really created on the src switch

  And: FlowLoop rules are created on the src switch

  And: FlowLoop rules are not created on the dst switch

  And: The src switch is valid

  When: Send traffic via flow in the forward direction

  Then: Flow doesn't allow traffic, because it is grubbed by flowLoop rules

  And: Counter only on the forward flowLoop rule is increased

  And: Counter on the simple(ingress/egress) flow rules is not increased on the src switch

  And: Counter on the simple(ingress/egress) flow rules on the dst switch is the same as reverseLoopPacketCount

  When: Send traffic via flow in the reverse direction

  Then: Flow doesn't allow traffic, because it is grubbed by flowLoop rules

  And: Counter only on the reverse flowLoop rule is increased

  And: Counter on the simple(ingress/egress) flow rules is not increased on the src switch

  And: Counter on the simple(ingress/egress) flow rules is increased on the dst switch

  When: Delete flowLoop

  Then: The delete flowLoop response contains the flowId

  And: FlowLoop is really deleted

  And: Flow is UP and valid

  And: FlowLoop rules are deleted from the src switch

  And: The src switch is valid

  And: Flow allows traffic

• Able to delete a flow with created flowLoop on it

  Given: A active multi switch flow

  When: Create flowLoop on the dst switch

  Then: FlowLoop is created on the dst switch

  And: FlowLoop rules are created

  And: Flow history contains info about flowLoop

  When: Delete the flow

  Then: FlowLoop rules are deleted from the dst switch

  And: Both switches are valid

• System is able to reroute a flow when flowLoop is created on it

  Given: A multi switch flow with one alternative path at least

  And: FlowLoop is created on the dst switch

  When: Fail a flow ISL (bring switch port down)

  Then: The flow was rerouted

  And: Flow is UP and valid

  And: FlowLoop is still present on the dst switch

  And: FlowLoop rules ar still present on the dst switch

  And: The src switch is valid

  And: Flow doesn't allow traffic, because it is grubbed by flowLoop rules

• System is able to detect and sync missing flowLoop rules

  Given: An active flow with created flowLoop on the src switch

  When: Delete flowLoop rules

  Then: System detects missing flowLoop rules

  When: Sync the src switch

  Then: Sync response contains flowLoop rules into the installed section

  Then: FlowLoop rules are synced

• Systems allows to get all flowLoops that goes through a switch

  Given: Two active switches

  And: Three multi switch flows

  When: Get all flowLoops from the src switch

  Then: There is no flowLoop because it is not created yet

  When: Create flowLoop for flow1 and flow2 on the src switch

  And: Create flowLoop for the flow3 on the dst switch

  And: Get all flowLoops from the src switch

  Then: The created flowLoops for flow1/flow2 are returned in the response list from the src switch

  When: Get all flowLoops from the dst switch

  Then: Only flow3 is in the response list from the dst switch

• System is able to autoSwapPath for a protected flow when flowLoop is created on it

  Given: Two active switches with three diverse paths at least

  And: A protected unmetered flow with flowLoop on the src switch

  When: Break ISL on the main path (bring port down) to init auto swap

  Then: Flow is switched to protected path

  And: FlowLoop is still present on the src switch

  And: FlowLoop rules are still exist on the src switch

  And: The src switch is valid

  When: Send traffic via flow

  Then: Flow doesn't allow traffic, because it is grubbed by flowLoop rules

  And: Counter on the flowLoop rules are increased

• Able to create flowLoop for a singleSwitch flow

  Given: An active singleSwitch flow

  When: Create flowLoop on the sw switch

  Then: Create flowLoop response contains flowId and src switchId

  And: Flow is UP and valid

  And: FlowLoop is really created on the switch

  And: FlowLoop rules are created

  And: The switch is valid

  When: Delete flowLoop

  Then: FlowLoop is deleted

  And: FlowLoop rules are deleted from the switch

  And: Flow is UP and valid

  And: The switch is valid

• Able to create flowLoop for a singleSwitchSinglePort flow

  Given: An active singleSwitchSinglePort flow

  When: Create flowLoop on the sw switch

  Then: Create flowLoop response contains flowId and src switchId

  And: Flow is UP and valid

  And: FlowLoop is really created on the switch

  And: FlowLoop rules are created

  And: The switch is valid

  When: Delete the flow with created flowLoop

  Then: FlowLoop rules are deleted from the switch

  And: The switch is valid

• Attempt to create the exact same flowLoop twice just reinstalls the rules

  Given: An active multi switch flow with created flowLoop on the src switch

  When: Try to create flowLoop on the src switch again

  Then: FlowLoop is still present for the src switch

  And: The src/dst switches are valid

  And: No extra rules are created on the src/dst switches

  When: Delete the flow with created flowLoop

  Then: FlowLoop rules are deleted

• Unable to create flowLoop when a switch is deactivated

  Given: An active flow

  And: Deactivated the src switch

  When: Try to create flowLoop on th src switch(deactivated)

  Then: Human readable error is returned

  And: FlowLoop is not created

  When: Try to create flowLoop on th dst switch(activated)

  Then: Human readable error is returned

  Then: FlowLoop is not created

  And: FlowLoop rules are not created on the dst switch

• Unable to create flowLoop on the src switch when it is already created on the dst switch

  Given: An active flow with created flowLoop on the src switch

  When: Try to create flowLoop on the dst switch

  Then: FlowLoop is not created on the dst switch

• Unable to create flowLoop on a transit switch

  Given: An active multi switch flow with transit switch

  When: Try to create flowLoop on the transit switch

  Then: Human readable error is returned

  Then: FlowLoop is not created

  And: FlowLoop rules are not created on the transit switch

• Unable to create flowLoop for a non existent flow

  When: Try to create flowLoop on the transit switch

  Then: Human readable error is returned

  And: FlowLoop rules are not created on the switch

  And: The switch is valid

• Unable to create flowLoop on a non existent switch

  Given: An active multi switch flow

  When: Try to create flowLoop on a non existent switch

  Then: Human readable error is returned

  And: FlowLoop rules are not created for the flow

FlowMonitoring

• Able to detect and reroute a flow with MAX_LATENCY strategy when main path does not satisfy latency SLA

  Given: 2 non-overlapping paths with 200 and 250 latency

  And: flowLatencyMonitoringReactions is enabled in featureToggle

  And: Disable s42 in featureToggle for generating flow-monitoring stats

  And: A flow with max_latency 210

  When: Main path does not satisfy SLA(update isl latency via db)

  Then: System detects that flowPathLatency doesn't satisfy max_latency and reroute the flow

  And: Flow history contains information that flow was rerouted due to SLA check

• System doesn't try to reroute a MAX_LATENCY flow when a flow path doesn't satisfy latency SLA and flowLatencyMonitoringReactions is disabled in featureToggle

  Given: 2 non-overlapping paths with 200 and 250 latency

  And: flowLatencyMonitoringReactions is disabled in featureToggle

  And: Disable s42 in featureToggle for generating flow-monitoring stats

  And: A flow with max_latency 210

  When: Main path does not satisfy SLA(update isl latency via db)

  Then: Flow is not rerouted because flowLatencyMonitoringReactions is disabled in featureToggle

  And: Flow path is not changed

FlowPing

This spec tests all the functionality related to flow pings. Flow ping feature sends a 'ping' packet at the one end of the flow, expecting that this packet will be delivered at the other end. 'Pings' the flow in both directions(forward and reverse).

• Able to ping a flow with vlan between #switchPairDescription

  Given: A flow with random vlan

  When: Ping the flow

  Then: Ping is successful

  And: No errors

  And: Unicast rule packet count is increased and logged to otsdb

• Able to ping a flow with vxlan

  Given: A flow with random vxlan

  When: Ping the flow

  Then: Ping is successful

  And: No errors

  And: Unicast rule packet count is increased and logged to otsdb

• Able to ping a flow with no vlan between #switchPairDescription

  Given: A flow with no vlan

  When: Ping the flow

  Then: Ping is successful

  And: No errors

• Flow ping can detect a broken path(#description) for a vlan flow

  Given: A flow with at least 1 a-switch link

  When: Break the flow by removing rules from a-switch

  And: Ping the flow

  Then: Ping response properly shows that certain direction is unpingable

• Unable to ping a single-switch flow

  Given: A single-switch flow

  When: Ping the flow

  Then: Error received

• Verify error if try to ping with wrong flowId

  When: Send ping request with non-existing flowId

  Then: Receive error response

• Flow ping can detect a broken path for a vxlan flow on an intermediate switch

  Given: A vxlan flow with intermediate switch(es)

  When: Break the flow by removing flow rules from the intermediate switch

  Then: Ping shows that path is broken

• Able to turn on periodic pings on a flow

  When: Create a flow with periodic pings turned on

  Then: Packet counter on catch ping rules grows due to pings happening

• Unable to create a single-switch flow with periodic pings

  When: Try to create a single-switch flow with periodic pings

  Then: Error is returned in response

FlowSync

• Able to synchronize a flow (install missing flow rules, reinstall existing) without rerouting

  Given: An intermediate-switch flow with deleted rules on src switch

  When: Synchronize the flow

  Then: The flow is not rerouted

  And: Missing flow rules are installed (existing ones are reinstalled) on all switches

  And: Flow is valid

FlowValidationNegative

The specification covers the following scenarios: -- Deleting flow rule from a switch and check if switch and flow validation fails. -- Failed switch validation should not cause validation errors for flows with all rules in place. Test case permutations (full-factored): - forward and reverse flows - ingress, transit and egress switches - Single switch, two switch and three+ switch flow spans.

• Flow and switch validation should fail in case of missing rules with #flowConfig configuration [#flowType]

  Given: Two flows with #flowConfig configuration

  And: Both flows have the same switches in path

  When: #flowType flow rule from first flow on #switchNo switch gets deleted

  Then: Intact flow should be validated successfully

  And: Damaged #flowType flow validation should fail, while other direction should be validated successfully

  And: Flow rule discrepancy should contain dpID of the affected switch and cookie of the damaged flow

  And: Affected switch should have one missing rule with the same cookie as the damaged flow

  And: There should be no excess rules on the affected switch

  And: Validation of non-affected switches (if any) should succeed

• Unable to #data.description a non-existent flow

  When: Trying to #action a non-existent flow

  Then: An error is received (404 code)

• Able to detect discrepancies for a flow with protected path

  When: Create a flow with protected path

  Then: Flow with protected path is created

  And: Validation of flow with protected path must be successful

  When: Delete rule of protected path on the srcSwitch

  Then: Flow validate detects discrepancies

  When: Delete all rules except default on the all involved switches

  Then: Flow validate detects discrepancies for all deleted rules

IntentionalReroute

Verify that on-demand reroute operations are performed accurately.

• Not able to reroute to a path with not enough bandwidth available

  Given: A flow with alternate paths available

  When: Make the current path less preferable than alternatives

  And: Make all alternative paths to have not enough bandwidth to handle the flow

  And: Init a reroute to a more preferable path

  Then: The flow is NOT rerouted because of not enough bandwidth on alternative paths

• Able to reroute to a better path if it has enough bandwidth

  Given: A flow with alternate paths available

  When: Make one of the alternative paths to be the most preferable among all others

  And: Make the future path to have exact bandwidth to handle the flow

  And: Init a reroute of the flow

  Then: The flow is successfully rerouted and goes through the preferable path

  And: 'Thin' ISL has 0 available bandwidth left

• Intentional flow reroute is not causing any packet loss

  Given: An unmetered flow going through a long not preferable path(reroute potential)

  When: Start traffic examination

  And: While traffic flow is active, request a flow reroute

  Then: Flow is rerouted

  And: Traffic examination result shows acceptable packet loss percentage

• Able to reroute to a path with not enough bandwidth available in case ignoreBandwidth=true

  Given: A flow with alternate paths available

  When: Make the current path less preferable than alternatives

  And: Make all alternative paths to have not enough bandwidth to handle the flow

  And: Init a reroute to a more preferable path

  Then: The flow is rerouted because ignoreBandwidth=true

  And: Available bandwidth was not changed while rerouting due to ignoreBandwidth=true

• Intentional flow reroute with VXLAN encapsulation is not causing any packet loss

  Given: A vxlan flow

  When: Start traffic examination

  And: While traffic flow is active, request a flow reroute

  Then: Flow is rerouted

  And: Traffic examination result shows acceptable packet loss percentage

• Not able to reroute to a path with not enough bandwidth available [v1 api]

  Given: A flow with alternate paths available

  When: Make the current path less preferable than alternatives

  And: Make all alternative paths to have not enough bandwidth to handle the flow

  And: Init a reroute to a more preferable path

  Then: The flow is NOT rerouted because of not enough bandwidth on alternative paths

• Able to reroute to a better path if it has enough bandwidth [v1 api]

  Given: A flow with alternate paths available

  When: Make one of the alternative paths to be the most preferable among all others

  And: Make the future path to have exact bandwidth to handle the flow

  And: Init a reroute of the flow

  Then: The flow is successfully rerouted and goes through the preferable path

  And: 'Thin' ISL has 0 available bandwidth left

MaxLatency

A flow with LATENCY strategy: - system tries to find the path with best latency, mL/mlT2 is not used during finding a path - system doesn't allow to create a flow in case flowPathLatency > max_latency_tier2 - system moves flow to DOWN state after attempt to reroute in case flowPathLatency > max_latency_tier2 - max_latency and max_latency_tier2 fields are used for setting the status of the flow (for example: two paths with 11 and 15 latency and two flows flow_1: mL=10, mlT2=12; flow_2: mL=14, mlT2=16 Both flows are build via the path with 11 latency, but the status is different: flow_1 - degraded, flow_2-up) - flowPathLatency <= max_latency - flow is in UP state after creating - flowPathLatency > max_latency - flow is in DEGRADED state after creating - max_latency_tier2 < max_latency - it is wrong, but the system allows it. In this case the system will consider that max_latency_tier2 = max_latency and inform us via kibana by warning message: log.warn("Bad flow params found: maxLatencyTier2 ({}) should be greater than maxLatency ({}). " + "Put maxLatencyTier2 = maxLatency during path calculation.", flow.getMaxLatencyTier2(), flow.getMaxLatency());

Special cases(covered by unit tests('MaxLatencyPathComputationStrategyBaseTest.java')):
- flow with MAX_LATENCY strategy and 'max-latency' set to 0 should pick path with least latency.
- flow with MAX_LATENCY strategy and 'max-latency' being unset(null) should pick path with least latency.

• Able to create protected flow with max_latency strategy if both paths satisfy SLA

  Given: 2 non-overlapping paths with 10 and 15 latency

  When: Create a flow with protected path, max_latency 16 and max_latency_tier_2 18

  Then: Flow is created, main path is the 15 latency path, protected is 10 latency

• Unable to create protected flow with max_latency strategy if #condition

  Given: 2 non-overlapping paths with 10 and 9 latency

  When: Create a flow with protected path and max_latency #testMaxLatency

  Then: Flow is not created, error returned describing that no paths found

• Able to create DEGRADED protected flow with max_latency strategy if maxLatency < protectedPathLatency < maxLatencyTier2

  Given: 2 non-overlapping paths with 10 and 15 latency

  When: Create a flow with protected path, maxLatency 11 and maxLatencyTier2 16

  Then: Flow is created, main path is the 10 latency path, protected is 15 latency

  And: Flow goes to DEGRADED state

• Able to create DEGRADED flow with max_latency strategy if maxLatencyTier2 > pathLatency > maxLatency

  Given: 2 non-overlapping paths with 11 and 15 latency

  When: Create a flow with max_latency 11 and max_latency_tier2 16

  Then: Flow is created, flow path is the 15 latency path

• Able to update DEGRADED flow with max_latency strategy if maxLatencyTier2 > pathLatency > maxLatency

  Given: 2 non-overlapping paths with 10 and 15 latency

  When: Create a flow with max_latency 11 and max_latency_tier2 16

  And: Update the flow(maxLatency: 10)

  Then: Flow is updated and goes to the DEGRADED state

• Able to reroute a MAX_LATENCY flow if maxLatencyTier2 > pathLatency > maxLatency

  Given: 2 non-overlapping paths with 10 and 15 latency

  When: Create a flow with max_latency 11 and max_latency_tier2 16

  And: Init auto reroute (bring port down on the src switch)

  Then: Flow is rerouted and goes to the DEGRADED state

• Able to create DEGRADED flow with LATENCY strategy if max_latency_tier_2 > flowPath > max_latency

  Given: 2 non-overlapping paths with 11 and 15 latency

  When: Create a flow, maxLatency 10 and maxLatencyTier2 12

  Then: Flow is created in DEGRADED state because flowPath doesn't satisfy max_latency value but satisfies max_latency_tier2

• Able to create a flow with LATENCY strategy when max_latency = pathLatency

  Given: 2 non-overlapping paths with 9 and 15 latency

  When: Create a flow, maxLatency 9 and maxLatencyTier2 12

  Then: Flow is created in UP

• Unable to create a flow with LATENCY strategy when pathLatency > max_latency_tier2

  Given: 2 non-overlapping paths with 12 and 13 latency

  When: Create a flow, maxLatency 10 and maxLatencyTier2 11

  Then: Flow is not created, human readable error is returned

• A flow with LATENCY strategy is DOWN after attempt to reroute in case pathLatency > max_latency_tier2

  Given: 2 non-overlapping paths with 11 and 15 latency

  And: A flow with maxLatency 11 and maxLatencyTier2 14 on the path with 11 latency

  When: Break the flow path to init autoReroute

  Then: Flow is not rerouted and moved to the DOWN state

  MirrorEndpoints

• Able to CRUD a mirror endpoint on the src switch, mirror to the same switch diff port #description

  Given: A flow

  When: Create a mirror point on src switch, pointing to a different port, random vlan

  Then: Mirror status changes to Active

  And: Flow history reports a successful mirror creation

  And: Mirror endpoint is visible in 'get flows', 'get single flow' and 'get mirror endpoint' APIs

  And: Mirror flow rule has an OF group action and higher prio than flow rule

  And: Flow rule of opposite direction does not have an OF group action and there is no mirror flow rule

  And: Related switches and flow pass validation

  When: Traffic briefly runs through the flow

  Then: OF group reports same amount of packets sent both to mirror and to main paths

  And: Original flow rule counter is not increased

  And: System collects stat for mirror cookie in tsdb

  And: Traffic is also received at the mirror point (check only if second tg available)

  When: Delete the mirror point

  Then: 'Mirror point delete' operation is present in flow history

  And: Mirror point is no longer present in flow and mirror APIs

  And: Mirror flow rule is removed and flow rule is intact

  And: OF group is removed

  And: Src switch and flow pass validation

  When: Delete the flow

  Then: Src switch pass validation

• Can create mirror point on protected flow and survive path swap, #mirrorDirection

  Given: A flow with protected path

  When: Create a mirror point

  Then: Mirror point is created and Active

  And: Flow and switch pass validation

  When: Swap flow paths

  Then: Flow and switch both pass validation

  And: Flow passes main traffic

  And: Flow passes mirrored traffic

• Can create mirror point on a VXLAN flow #description

  Given: A VXLAN flow

  When: Create a mirror point

  Then: Mirror point is created and Active

  And: Related switches and flow pass validation

  And: Flow passes traffic on main path as well as to the mirror (if possible to check)

• Flow with mirror point can survive flow sync, #encap, #mirrorDirection

  Given: A flow with given encapsulation type and mirror point

  When: Call flow sync for the flow

  Then: Related switches and flow pass validation

• Can create mirror point on unmetered pinned flow, #mirrorDirection

  Given: An unmetered pinned flow

  When: Create a mirror point on src

  Then: Mirror point is created and Active

  And: Flow and src switch both pass validation

• Can create a mirror point on the same port as flow, different vlan #description

  Given: A flow

  When: Add a mirror point on the same port with flow, different vlan

  Then: Mirror point is successfully created

  When: Delete the flow without deleting its mirror point

  Then: Mirror point is also deleted from db

  And: Related switch pass validation

• Can create multiple mirror points for the same flow and switch

  Given: A flow

  When: Add a Forward mirror point on the same port with flow, different vlan

  Then: Mirror point is created

  When: Add one more Forward mirror point on the same port with flow, different vlan

  Then: Mirror point is created

  And: Mirrorring group for forward path has 3 buckets (main flow + 2 mirrors)

  When: Add a Reverse mirror point on the different port with flow

  Then: Mirror point is created

  And: Mirrorring group for reverse path has 2 buckets (main flow + 1 mirror)

• System also updates mirror rule after flow partial update

  Given: A flow with mirror point

  When: Update flow port and vlan on the same endpoint where mirror is

  Then: Flow and affected switch are valid

  And: Mirror rule has updated port/vlan values

• Mirror point can be created for a default flow (0 vlan), #mirrorDirection

  Given: A default flow

  When: Create a mirror point

  Then: Mirror point is created and Active

  And: Related switches and flow pass validation

  And: Flow passes traffic on main path as well as to the mirror

• Flow mirror point works properly with a qinq flow, #mirrorDirection

  Given: A qinq flow

  When: Create a mirror point

  Then: Mirror point is created, flow and switches are valid

  And: Traffic examination reports packets on mirror point

• Unable to create a mirror endpoint with #data.testDescr on the transit switch

  Given: A flow with transit switch

  When: Try to add a mirror endpoint on the transit switch

  Then: Error is returned, cannot create mirror point on given sw

• Test possible error scenarios during mirror point creation: [#testData.testName, #testData.mirrorPointDirection]

  Given: A flow

  When: Try adding a mirror point with conflict

  Then: Error is returned, cannot create mirror point with given params

• Unable to create a mirror point with existing flow conflict, #mirrorDirection

  Given: A flow

  When: Try adding a mirror point that conflicts with other existing flow

  Then: Error is returned, cannot create mirror point with given params

• Unable to create a flow that conflicts with mirror point, #mirrorDirection

  Given: A flow with mirror point

  When: Try adding a flow that conflicts with existing mirror point

  Then: Error is returned, cannot create flow that conflicts with mirror point

• Unable to create mirror point with connected devices enabled, #mirrorDirection

  Given: A flow with connected devices enabled

  When: Try to create a mirror for the flow (on the same endpoint)

  Then: Error is returned, cannot create create mirror for a flow with devices

• Unable to update flow and enable connected devices if mirror is present, #mirrorDirection

  Given: A flow with a mirror point

  When: Try to partial update the flow and enable connected devices

  Then: Error is returned, cannot enable devices on a flow with mirror

• Cannot enable connected devices on switch if mirror is present

  Given: A flow with a mirror endpoint

  When: Try to enable connected devices for switch where mirror is created

  Then: Error is returned, cannot enable devices on a switch with mirror

• Cannot create mirror on a switch with enabled connected devices

  Given: A switch with enabled connected devices

  When: Try to create a mirror on the switch with devices

  Then: Error is returned, cannot create flow that conflicts with mirror point

• Unable to create a mirror point with existing mirror point conflict, #mirrorDirection

  Given: A flow with mirror point

  When: Try adding one more mirror point that conflicts with existing mirror point

  Then: Error is returned, cannot create flow that conflicts with mirror point

MultiReroute

• Simultaneous reroute of multiple flows should not oversubscribe any ISLs

  Given: Many flows on the same path, with alt paths available

  When: Make another path more preferable

  And: Make preferable path's ISL to have bandwidth to host only half of the rerouting flows

  And: Init simultaneous reroute of all flows by bringing current path's ISL down

  Then: Half of the flows are hosted on the preferable path

  And: Rest of the flows are hosted on another alternative paths

  And: None ISLs are oversubscribed

PartialUpdate

Covers PATCH /api/v2/flows/:flowId and PATCH /api/v1/flows/:flowId This API allows to partially update a flow, i.e. update a flow without specifying a full flow payload. Depending on changed fields flow will be either updated+rerouted or just have its values changed in database.

• Able to partially update flow '#data.field' without reinstalling its rules

  Given: A flow

  When: Request a flow partial update for a #data.field field

  Then: Update response reflects the changes

  And: Changes actually took place

  And: Flow rules have not been reinstalled

• Able to partially update flow #data.field without reinstalling its rules(v1)

  Given: A flow

  When: Request a flow partial update for a #data.field field

  Then: Update response reflects the changes

  And: Changes actually took place

  And: Flow rules have not been reinstalled

• Able to partially update flow #data.field which causes a reroute

  Given: A flow

  When: Request a flow partial update for a #data.field field

  Then: Update response reflects the changes

  And: Changes actually took place

  And: Flow rules have been reinstalled

• Able to turn on diversity feature using partial update

  Given: Two active neighboring switches with two not overlapping paths at least

  When: Create 2 not diverse flows going through these switches

  Then: Both flows use the same path

  When: Update second flow to become diverse with the first flow (partial update)

  Then: Flows use diverse paths

• Able to do partial update on a single-switch flow

  Given: A single-switch flow

  When: Request a flow partial update for a 'priority' field

  Then: Update response reflects the changes

  And: Changes actually took place

  And: Flow rules have not been reinstalled

• Able to update a flow port and vlan using partial update

  Given: Three active switches

  And: A vlan flow

  When: Update the flow: port number and vlan id on the src endpoint

  Then: Flow is really updated

  And: Flow is valid and pingable

  And: The src switch passes switch validation

• Able to update a flow endpoint using partial update

  Given: Three active switches

  And: A vlan flow

  When: Update the flow: switch id on the dst endpoint

  Then: Flow is really updated

  And: Flow rules are installed on the new dst switch

  And: Flow is valid and pingable

  And: The new and old dst switches pass switch validation

• Able to update flow encapsulationType using partial update

  Given: A flow with a 'transit_vlan' encapsulation

  When: Request a flow partial update for an encapsulationType field(vxlan)

  Then: Update response reflects the changes

  And: Changes actually took place

  And: Flow rules have been reinstalled

• Able to update a flow port and vlan for a single-switch flow using partial update

  Given: An active single-switch flow (different ports)

  When: Update the flow: port number and vlanId on the src endpoint

  Then: Flow is really updated

  And: Flow is valid

  And: The ingress/egress rules are really updated

  And: The switch passes switch validation

• Able to update a flow port and vlan for a single-switch single-port flow using partial update

  Given: An active single-switch single-port flow

  When: Update the flow: new port number on src+dst and new vlanId on the src endpoint

  Then: Flow is really updated

  And: Flow is valid

  And: The ingress/egress rules are really updated

  And: The switch passes switch validation

• Partial update with empty body does not actually update flow in any way(v1)

  Given: A flow

  When: Request a flow partial update without specifying any fields

  Then: Flow is left intact

  And: Flow rules have not been reinstalled

• Partial update with empty body does not actually update flow in any way

  Given: A flow

  When: Request a flow partial update without specifying any fields

  Then: Flow is left intact

  And: Flow rules have not been reinstalled

• Unable to partial update a flow in case new port is an isl port on a #data.switchType switch

  Given: Neighbouring switch pair

  And: A flow

  When: Try to edit port to isl port

  Then: Error is returned

• Unable to partial update flow when there are conflicting vlans (#data.conflict)

  Given: Two potential flows

  When: Create two flows

  And: Try updating the second flow which should conflict with the first one (partial update)

  Then: Error is returned, stating a readable reason of conflict

• Unable to update a flow to have both strict_bandwidth and ignore_bandwidth flags at the same time

  Given: An existing flow without flag conflicts

  When: Partial update the flow to have strict_bw-ignore_bw conflict

  Then: Bad Request response is returned

• Able to update vlanId via partialUpdate in case vlanId==0 and innerVlanId!=0

  Given: A default flow

  When: Update innerVlanId only via partialUpdate

  Then: Partial update response reflects the changes

  And: Changes actually took place

• Unable to partial update flow with maxLatency incorrect value(#description)

  Given: Two potential flows

  When: Partial update the flow

  Then: Bad Request response is returned

PinnedFlow

A new flag of flow that indicates that flow shouldn't be rerouted in case of auto-reroute.

• In case of isl down such flow should be marked as DOWN.
• On Isl up event such flow shouldn't be re-routed as well. Instead kilda should verify that it's path is online and mark flow as UP.

• Able to CRUD pinned flow

  When: Create a flow

  Then: Pinned flow is created

  When: Update the flow (pinned=false)

  Then: The pinned option is disabled

• Able to CRUD unmetered one-switch pinned flow

  When: Create a flow

  Then: Pinned flow is created

  When: Update the flow (pinned=false)

  Then: The pinned option is disabled

• System doesn't reroute(automatically) pinned flow when flow path is partially broken

  Given: A pinned flow going through a long not preferable path

  When: Make alt path more preferable than current path

  And: Init reroute by bringing current path's ISL down one by one

  Then: Flow is not rerouted and marked as DOWN when the first ISL is broken

  And: Rules and meters are not changed

  When: The broken ISLs are restored one by one

  And: Restore the last ISL

  Then: Flow is marked as UP when the last ISL is restored

• System is not rerouting pinned flow when 'reroute link flows' is called

  Given: A pinned flow with alt path available

  When: Make another path more preferable

  And: Init reroute of all flows that go through pinned flow's isl

  Then: Flow is not rerouted (but still present in reroute response)

• System returns error if trying to intentionally reroute a pinned flow

  Given: A pinned flow with alt path available

  When: Make another path more preferable

  And: Init manual reroute

  Then: Error is returned

• System doesn't allow to create pinned and protected flow at the same time

  When: Try to create pinned and protected flow

  Then: Human readable error is returned

• System doesn't allow to enable the protected path flag on a pinned flow

  Given: A pinned flow

  When: Update flow: enable the allocateProtectedPath flag(allocateProtectedPath=true)

  Then: Human readable error is returned

• System doesn't allow to create pinned and protected flow at the same time [v1 api]

  When: Try to create pinned and protected flow

  Then: Human readable error is returned

• System doesn't allow to enable the protected path flag on a pinned flow [v1 api]

  Given: A pinned flow

  When: Update flow: enable the allocateProtectedPath flag(allocateProtectedPath=true)

  Then: Human readable error is returned

ProtectedPath

Protected path - it is pre-calculated, reserved, and deployed (except ingress rule), so we can switch traffic fast.

• flow object is extended with a boolean parameter 'allocate_protected_path' with values false(default)
• /flows/{flow-id}/path returns 'main_path' + 'protected_path'.

System can start to use protected path in two case:

• main path is down;
• we send the 'swap' request for a flow with protected path('/v1/flows/{flow_id}/swap')

A flow has the status degraded in case when the main path is up and the protected path is down.

Main and protected paths can't use the same link.

• Able to create a flow with protected path when maximumBandwidth=#bandwidth, vlan=#vlanId

  Given: Two active not neighboring switches with two diverse paths at least

  When: Create flow with protected path

  Then: Flow is created with protected path

  And: Rules for main and protected paths are created

  And: Validation of flow must be successful

• Able to enable/disable protected path on a flow

  Given: Two active not neighboring switches with two diverse paths at least

  When: Create flow without protected path

  Then: Flow is created without protected path

  And: Source switch passes validation

  And: Cookies are created by flow

  When: Update flow: enable protected path(allocateProtectedPath=true)

  Then: Flow has been updated successfully and protected path is enabled

  And: Rules for main and protected paths are created

  When: Update flow: disable protected path(allocateProtectedPath=false)

  Then: Flow has been updated successfully and protected path is disabled

  And: Rules for protected path are deleted

• Able to swap main and protected paths manually

  When: Flow with protected path has been created successfully

  Then: Protected path is enabled

  And: Rules for main and protected paths are created

  And: Number of flow-related cookies has been collected for both source and destination switch

  And: No rule discrepancies on every switch of the flow on the main path

  And: No rule discrepancies on every switch of the flow on the protected path)

  And: The flow allows traffic(on the main path)

  When: Swap flow paths

  Then: Flow paths are swapped

  And: No rule discrepancies when doing flow validation

  And: Rules are updated

  And: New meter is created on the src and dst switches

  And: Old meter is deleted on the src and dst switches

  And: Initial transit switches store the correct info about rules and meters

  And: No rule discrepancies on every switch of the flow on the main path

  And: No rule discrepancies on every switch of the flow on the protected path)

  And: The flow allows traffic(on the protected path)

• System is able to switch #flowDescription flows to protected paths

  Given: Two active not neighboring switches with three diverse paths at least

  When: Create 5 flows with protected paths

  Then: Flows are created with protected path

  And: Current paths are not equal to protected paths

  And: Bandwidth is reserved for protected paths on involved ISLs

  When: Break ISL on the main path (bring port down) to init auto swap

  Then: Flows are switched to protected paths

  When: Restore port status

  Then: Path of the flow is not changed

• Flow swaps to protected path when main path gets broken, becomes DEGRADED if protected path is unable to reroute(no bw)

  Given: Two switches with 2 diverse paths at least

  When: Create flow with protected path

  And: Other paths have not enough bandwidth to host the flow in case of reroute

  And: Main flow path breaks

  Then: Main path swaps to protected, flow becomes degraded, main path UP, protected DOWN

  When: ISL gets back up

  Then: Main path remains the same, flow becomes UP, main path UP, protected UP

• Flow swaps to protected path when main path gets broken, becomes DEGRADED if protected path is unable to reroute(no path)

  Given: Two switches with 2 diverse paths at least

  When: Create flow with protected path

  And: Other paths are not available (ISLs are down)

  And: Main flow path breaks

  Then: Main path swaps to protected, flow becomes degraded, main path UP, protected DOWN

  When: ISL on broken path gets back up

  Then: Main path remains the same (no swap), flow becomes UP, main path remains UP, protected path becomes UP

• System reroutes #flowDescription flow to more preferable path and ignores protected path when reroute is intentional

  Given: Two active neighboring switches with four diverse paths at least

  And: A flow with protected path

  When: Make the current and protected path less preferable than alternatives

  And: Init intentional reroute

  Then: Flow is rerouted

  And: Path is not changed to protected path

• System is able to switch #flowDescription flow to protected path and ignores more preferable path when reroute is automatical

  Given: Two active not neighboring switches with three diverse paths at least

  And: A flow with protected path

  When: Make the current and protected path less preferable than alternatives

  And: Break ISL on the main path (bring port down) to init auto swap

  Then: Flow is switched to protected path

  When: Restore port status

  Then: Path of the flow is not changed

• A flow with protected path does not get rerouted if already on the best path

  Given: Two active neighboring switches and a flow

  When: Init intentional reroute

  Then: Flow is not rerouted

• Able to update a flow to enable protected path when there is not enough bandwidth

  Given: Two active neighboring switches

  And: Create flow without protected path

  And: Update all ISLs which can be used by protected path

  When: Update flow: enable protected path

  Then: Flow state is changed to DEGRADED

• Able to create a flow with protected path when there is not enough bandwidth and ignoreBandwidth=true

  Given: Two active neighboring switches

  And: Update all ISLs which can be used by protected path

  When: Create flow with protected path

  Then: Flow is created with protected path

  And: One transit vlan is created for main and protected paths

• System is able to recalculate protected path when protected path is broken

  Given: Two active not neighboring switches with three diverse paths at least

  When: Create a flow with protected path

  Then: Flow is created with protected path

  When: Break ISL on the protected path (bring port down) to init the recalculate procedure

  Then: Protected path is recalculated

  And: Current path is not changed

  And: Bandwidth is reserved for new protected path on involved ISLs

  And: Reservation is deleted on the broken ISL

  When: Restore port status

  Then: Path is not recalculated again

• Able to update #flowDescription flow to enable protected path if all alternative paths are unavailable

  Given: Two active neighboring switches with two not overlapping paths at least

  And: A flow without protected path

  And: All alternative paths are unavailable (bring ports down on the source switch)

  When: Update flow: enable protected path(allocateProtectedPath=true)

  Then: Flow state is changed to DEGRADED as protected path is DOWN

• #flowDescription flow is DEGRADED when protected and alternative paths are not available

  Given: Two active neighboring switches with two not overlapping paths at least

  And: A flow with protected path

  When: All alternative paths are unavailable

  Then: Flow status is DEGRADED

  When: Update flow: disable protected path(allocateProtectedPath=false)

  Then: Flow status is UP

• System properly reroutes both paths if protected path breaks during auto-swap

  Given: Switch pair with at least 4 diverse paths

  And: A protected flow

  When: Main paths breaks

  And: Protected path breaks when swap is in progress

  Then: Both paths are successfully evacuated from broken isls and the flow is UP

• System reuses current protected path when can't find new non overlapping protected path while intentional rerouting

  Given: Two active neighboring switches with three diverse paths

  And: A flow with protected path

  When: Make the current and protected path less preferable than alternatives

  And: Init intentional reroute

  Then: Flow should be rerouted

  And: Flow main path should be rerouted to a new path and ignore protected path

  And: Flow protected path shouldn't be rerouted due to lack of non overlapping path

  And: Flow and both its paths are UP

• Protected path is created in different POP even if this path is not preferable

  Given: Not neighboring switch pair with three diverse paths at least

  And: Src, dst and transit switches belongs to different POPs(src:1, dst:4, tr1/tr2:2, tr3:3)

  And: Path which contains tr3 is non preferable

  And: All alternative paths unavailable (bring ports down on the source switch)

  When: Create a protected flow

  Then: Main path is built through the preferable path(tr1 or tr2)

  And: Protected path is built through the non preferable path(tr3)

• Unable to perform the 'swap' request for a flow without protected path

  Given: Two active neighboring switches

  And: A flow without protected path

  When: Try to swap paths for flow that doesn't have protected path

  Then: Human readable error is returned

• Unable to swap paths for a non-existent flow

  When: Try to swap path on a non-existent flow

  Then: Human readable error is returned

• Unable to create a flow with protected path when there is not enough bandwidth

  Given: Two active neighboring switches

  And: Update all ISLs which can be used by protected path

  When: Create flow with protected path

  Then: Human readable error is returned

• Unable to swap paths for an inactive flow

  Given: Two active neighboring switches with two not overlapping paths at least

  And: A flow with protected path

  And: All alternative paths are unavailable (bring ports down on the source switch)

  When: Break ISL on a protected path (bring port down) for changing the flow state to DEGRADED

  Then: Flow state is changed to DEGRADED

  When: Break ISL on the main path (bring port down) for changing the flow state to DOWN

  Then: Flow state is changed to DOWN

  When: Try to swap paths when main/protected paths are not available

  Then: Human readable error is returned

  When: Restore ISL for the main path only

  Then: Flow state is still DEGRADED

  When: Try to swap paths when the main path is available and the protected path is not available

  Then: Human readable error is returned

  When: Restore ISL for the protected path

  Then: Flow state is changed to UP

• Unable to create a single switch flow with protected path

  Given: A switch

  When: Create single switch flow

  Then: Human readable error is returned

• Unable to update a single switch flow to enable protected path

  Given: A switch

  And: A flow without protected path

  When: Update flow: enable protected path

  Then: Human readable error is returned

• Unable to create #flowDescription flow with protected path if all alternative paths are unavailable

  Given: Two active neighboring switches without alt paths

  When: Try to create a new flow with protected path

  Then: Human readable error is returned

• System doesn't reroute main flow path when protected path is broken and new alt path is available(altPath is more preferable than mainPath)

  Given: Two active neighboring switches with three diverse paths at least

  And: A flow with protected path

  And: All alternative paths are unavailable (bring ports down on the source switch)

  And: ISL on a protected path is broken(bring port down) for changing the flow state to DEGRADED

  When: Make the current paths(main and protected) less preferable than alternative path

  And: Make alternative path available(bring port up on the source switch)

  Then: Reroute has been executed successfully and flow state is changed to UP

  And: Protected path is recalculated only

• System doesn't allow to enable the pinned flag on a protected flow

  Given: A protected flow

  When: Update flow: enable the pinned flag(pinned=true)

  Then: Human readable error is returned

ProtectedPathV1

Protected path - it is pre-calculated, reserved, and deployed (except ingress rule), so we can switch traffic fast.

• flow object is extended with a boolean parameter 'allocate_protected_path' with values false(default)
• /flows/{flow-id}/path returns 'main_path' + 'protected_path'.

System can start to use protected path in two case:

• main path is down;
• we send the 'swap' request for a flow with protected path('/v1/flows/{flow_id}/swap')

A flow has the status degraded in case when the main path is up and the protected path is down.

Main and protected paths can't use the same link.

• Able to create a flow with protected path when maximumBandwidth=#bandwidth, vlan=#vlanId

  Given: Two active not neighboring switches with two diverse paths at least

  When: Create flow with protected path

  Then: Flow is created with protected path

  And: Rules for main and protected paths are created

  And: Validation of flow must be successful

• Able to enable/disable protected path on a flow

  Given: Two active not neighboring switches with two diverse paths at least

  When: Create flow without protected path

  Then: Flow is created without protected path

  And: Source switch passes validation

  And: Cookies are created by flow

  When: Update flow: enable protected path(allocateProtectedPath=true)

  Then: Protected path is enabled

  And: Rules for main and protected paths are created

  When: Update flow: disable protected path(allocateProtectedPath=false)

  Then: Protected path is disabled

  And: Source switch passes validation

  And: Rules for protected path are deleted

• Unable to create a single switch flow with protected path

  Given: A switch

  When: Create single switch flow

  Then: Human readable error is returned

• Unable to update a single switch flow to enable protected path

  Given: A switch

  And: A flow without protected path

  When: Update flow: enable protected path

  Then: Human readable error is returned

• Unable to create a flow with protected path when there is not enough bandwidth

  Given: Two active neighboring switches

  And: Update all ISLs which can be used by protected path

  When: Create flow with protected path

  Then: Human readable error is returned

• Unable to create #flowDescription flow with protected path if all alternative paths are unavailable

  Given: Two active neighboring switches without alt paths

  When: Try to create a new flow with protected path

  Then: Human readable error is returned

• Able to swap main and protected paths manually

  Given: A simple flow

  And: Number of flow-related cookies has been collected for both source and destination switch

  When: Update flow: enable protected path(allocateProtectedPath=true)

  Then: Protected path is enabled

  And: Rules for main and protected paths are created

  And: Source and destination switches pass validation

  And: No rule discrepancies on every switch of the flow on the main path

  And: No rule discrepancies on every switch of the flow on the protected path)

  And: The flow allows traffic(on the main path)

  When: Swap flow paths

  Then: Flow paths are swapped

  And: New meter is created on the src and dst switches

  And: No rule discrepancies when doing flow validation

  And: Rules are updated

  And: Old meter is deleted on the src and dst switches

  And: Transit switches store the correct info about rules and meters

  And: No rule discrepancies on every switch of the flow on the main path

  And: No rule discrepancies on every switch of the flow on the protected path)

  And: The flow allows traffic(on the protected path)

• Unable to perform the 'swap' request for a flow without protected path

  Given: Two active neighboring switches

  And: A flow without protected path

  When: Try to swap paths for flow that doesn't have protected path

  Then: Human readable error is returned

• Unable to swap paths for a non-existent flow

  When: Try to swap path on a non-existent flow

  Then: Human readable error is returned

• Unable to swap paths for an inactive flow

  Given: Two active neighboring switches with two not overlapping paths at least

  And: A flow with protected path

  And: All alternative paths are unavailable (bring ports down on the source switch)

  When: Break ISL on a protected path (bring port down) for changing the flow state to DEGRADED

  Then: Flow state is changed to DEGRADED

  When: Break ISL on the main path (bring port down) for changing the flow state to DOWN

  Then: Flow state is changed to DOWN

  When: Try to swap paths when main/protected paths are not available

  Then: Human readable error is returned

  When: Restore ISL for the main path only

  Then: Flow state is still DEGRADED

  When: Try to swap paths when the main path is available and the protected path is not available

  Then: Human readable error is returned

  When: Restore ISL for the protected path

  Then: Flow state is changed to UP

• System doesn't allow to enable the pinned flag on a protected flow

  Given: A protected flow

  When: Update flow: enable the pinned flag(pinned=true)

  Then: Human readable error is returned

QinQFlow

• System allows to manipulate with QinQ flow #description

  When: Create a QinQ flow

  Then: Response contains correct info about vlanIds

  And: Flow is really created with requested vlanIds

  And: Flow is valid and pingable

  And: The flow allows traffic (if applicable)

  And: Involved switches pass switch validation

  When: Create a vlan flow on the same port as QinQ flow

  Then: Both existing flows are valid

  And: Involved switches pass switch validation

  And: Both flows are pingable

  Then: Both flows allow traffic

  When: Update the QinQ flow(outer/inner vlans)

  Then: Update response contains correct info about innerVlanIds

  And: Flow is really updated

  And: Flow history shows actual info into stateBefore and stateAfter sections

  Then: Both existing flows are still valid and pingable

  When: Delete the flows

  Then: Flows rules are deleted

  And: Shared rule of flow is deleted

• System allows to create a single switch QinQ flow #description

  Given: Switch default cookies before flow creation have been collected

  When: Create a single switch QinQ flow

  Then: Response contains correct info about vlanIds

  And: Flow is really created with requested vlanIds

  And: Flow is valid

  And: Unable to ping a one-switch qinq flow

  And: Involved switches pass switch validation

  And: Traffic examination is successful (if possible)

  When: Delete the flow

  Then: Flow rules are deleted

• System doesn't allow to create a QinQ flow with incorrect innerVlanIds (src:#srcInnerVlanId, dst:#dstInnerVlanId)

  Given: A switch pair with enabled multi table mode

  When: Try to create a QinQ flow with incorrect innerVlanId

  Then: Human readable error is returned

• Flow with innerVlan and vlanId=0 is transformed into a regular vlan flow without innerVlan

  When: Create a flow with vlanId=0 and innerVlanId!=0

  Then: Flow is created but with vlanId!=0 and innerVlanId==0

  And: Flow is valid

• System allow to create/update/delete a protected QinQ flow via APIv1

  Given: Two switches with enabled multi table mode

  When: Create a QinQ flow

  Then: Flow is really created with requested innerVlanId

  When: Update the flow(innerVlan/vlanId) via partialUpdate

  Then: Partial update response reflects the changes

  And: Flow is really updated with requested innerVlanId/vlanId

  And: Flow is valid and pingable

  When: Delete the flow via APIv1

  Then: Flows rules are deleted

  And: Shared rule of flow is deleted

• System allows to create QinQ flow and vlan flow with the same vlan on the same port

  Given: Two switches with enabled multi table mode

  When: Create a QinQ flow

  And: Create a flow without QinQ

  Then: Both flows allow traffic

• System detects conflict QinQ flows(oVlan: #conflictVlan, iVlan: #conflictInnerVlanId)

  Given: Two switches with enabled multi table mode

  When: Create a first flow

  And: Try to create a flow which conflicts(vlan) with first flow

  Then: Human readable error is returned

• System allows to create more than one QinQ flow on the same port and with the same vlan

  Given: Two switches connected to traffgen and enabled multiTable mode

  When: Create a first QinQ flow

  And: Create a second QinQ flow

  Then: Both flow are valid and pingable

  And: Flows allow traffic

  When: Delete the second flow

  Then: The first flow is still valid and pingable

  And: The first flow still allows traffic

• System allows to create a single-switch-port QinQ flow(srcVlanId: #srcVlanId, srcInnerVlanId: #srcInnerVlanId, dstVlanId: #dstVlanId, dstInnerVlanId: #dstInnerVlanId)

  Given: A switch with enabled multiTable mode

  When: Create a single switch QinQ flow

  Then: Response contains correct info about vlanIds

  And: Flow is really created with requested vlanIds

  And: Flow is valid

  And: Involved switches pass switch validation

  When: Delete the flow

  Then: Flow rules are deleted

• System allows to manipulate with QinQ vxlan flow #description

  When: Create QinQ vxlan flow

  Then: Response contains correct info about vlanIds

  And: Flow is really created with requested vlanIds

  And: Flow is valid and pingable

  And: The flow allows traffic (if applicable)

  And: Involved switches pass switch validation

  When: Create a vlan flow on the same port as QinQ flow

  Then: Both existing flows are valid

  And: Involved switches pass switch validation

  And: Both flows are pingable

  Then: Both flows allow traffic

  When: Update the QinQ flow(outer/inner vlans)

  Then: Update response contains correct info about innerVlanIds

  And: Flow is really updated

  Then: Both existing flows are still valid and pingable

  When: Delete the flows

  Then: Flows rules are deleted

  And: Shared rule of flow is deleted

• System is able to synchronize switch(flow rules)

  Given: Two switches connected to traffgen and enabled multiTable mode

  And: A QinQ flow on the given switches

  When: Delete all flow rules(ingress/egress/shared) on the src switch

  Then: System detects missing rules on the src switch

  And: Flow is valid

  And: The flow allows traffic

• System doesn't rebuild flow path to more preferable path while updating innerVlanId

  Given: Two active switches connected to traffgens with two possible paths at least

  And: A flow

  When: Make the current path less preferable than alternatives

  And: Update the flow: port number and vlanId on the src/dst endpoints

  Then: Flow is really updated

  And: Flow is not rerouted

  And: System allows traffic on the flow

  And: Flow is valid

  And: All involved switches pass switch validation

SwapEndpoint

• Able to swap endpoints(#data.description)

  Given: Some flows in the system according to preconditions

  When: Try to swap endpoints with #data.description

  Then: Endpoints are successfully swapped

  And: Flows validation doesn't show any rule discrepancies

  And: Switch validation doesn't show any missing/excess rules and meters

• Able to swap #data.endpointsPart (src1 <-> dst2, dst1 <-> src2) for two flows with the same source and different destination switches

  Given: Two flows with the same source and different destination switches

  When: Try to swap #endpointsPart for flows

  Then: #endpointsPart.capitalize() are successfully swapped

  And: Flows validation doesn't show any discrepancies

  And: Switch validation doesn't show any missing/excess rules and meters

• Able to swap endpoints (#data.description) for two flows with the same source and different destination switches

  Given: Two flows with the same source and different destination switches

  When: Try to swap endpoints for flows

  Then: Endpoints are successfully swapped

  And: Flows validation doesn't show any discrepancies

  And: Switch validation doesn't show any missing/excess rules and meters

• Able to swap #endpointsPart (#description) for two flows with different source and the same destination switches

  Given: Two flows with different source and the same destination switches

  When: Try to swap #endpointsPart for flows

  Then: #endpointsPart.capitalize() are successfully swapped

  And: Flows validation doesn't show any discrepancies

  And: Switch validation doesn't show any missing/excess rules and meters

• Able to swap #endpointsPart (#description) for two flows with different source and destination switches

  Given: Two flows with different source and destination switches

  When: Try to swap #endpointsPart for flows

  Then: #endpointsPart.capitalize() are successfully swapped

  And: Flows validation doesn't show any discrepancies

  And: Switch validation doesn't show any missing/excess rules and meters

• Able to swap endpoints (#data.description) for two flows with different source and destination switches

  Given: Two flows with different source and destination switches

  When: Try to swap endpoints for flows

  Then: Endpoints are successfully swapped

  And: Flows validation doesn't show any discrepancies

  And: Switch validation doesn't show any missing/excess rules and meters

• Unable to swap endpoints for existing flow and non-existing flow

  Given: An active flow

  When: Try to swap endpoints for existing flow and non-existing flow

  Then: An error is received (404 code)

• Unable to swap #data.endpointsPart for two flows: #data.description

  Given: Three active flows

  When: Try to swap #endpointsPart for two flows

  Then: An error is received (409 code)

• Unable to swap endpoints for two flows (#data.description)

  Given: Two active flows

  When: Try to swap endpoints for two flows

  Then: An error is received (400 code)

• Unable to swap ports for two flows (port is occupied by ISL on src switch)

  Given: Two active flows

  When: Try to swap ports for two flows

  Then: An error is received (400 code)

• Able to swap endpoints for two flows when all bandwidth on ISL is consumed

  Given: Create two flows with different source and the same destination switches

  And: Update the first flow so that it consumes all bandwidth on the link

  And: Break all alternative paths for the first flow

  And: Update max bandwidth for the second flow's link so that it is equal to max bandwidth of the first flow

  And: Break all alternative paths for the second flow

  When: Try to swap endpoints for two flows

  Then: Endpoints are successfully swapped

  And: Flows validation doesn't show any discrepancies

  And: Switch validation doesn't show any missing/excess rules and meters

• Unable to swap endpoints for two flows when not enough bandwidth on ISL

  Given: Create two flows with different source and the same destination switches

  And: Update the first flow so that it consumes all bandwidth on the link

  And: Break all alternative paths for the first flow

  And: Update max bandwidth for the second flow's link so that it is not enough bandwidth for the first flow

  And: Break all alternative paths for the second flow

  When: Try to swap endpoints for two flows

  Then: An error is received (500 code)

• Able to swap endpoints for two flows when not enough bandwidth on ISL and ignore_bandwidth=true

  Given: Create two flows with different source and the same destination switches

  And: Update the first flow so that it consumes all bandwidth on the link

  And: Break all alternative paths for the first flow

  And: Update max bandwidth for the second flow's link so that it is not enough bandwidth for the first flow

  And: Break all alternative paths for the second flow

  When: Try to swap endpoints for two flows

  Then: Endpoints are successfully swapped

  And: Flows validation doesn't show any discrepancies

  And: Switch validation doesn't show any missing/excess rules and meters

• Unable to swap endpoints for two flows when one of them is inactive

  Given: Create two flows with different source and the same destination switches

  And: Break all paths for the first flow

  When: Try to swap endpoints for two flows

  Then: An error is received (500 code)

  When: Get actual data of flow1 and flow2

  Then: Actual flow1, flow2 sources are different

  And: All involved switches are valid

• Able to swap endpoints (#data.description) for two protected flows

  Given: Two protected flows with different source and destination switches

  When: Try to swap endpoints for flows

  Then: Endpoints are successfully swapped

  And: Flows validation doesn't show any discrepancies

  And: Switch validation doesn't show any missing/excess rules and meters

• A protected flow with swapped endpoint allows traffic on main and protected paths

  Given: Two protected flows with different source and destination switches

  When: Try to swap endpoints for flows

  Then: Endpoints are successfully swapped

  And: Flows validation doesn't show any discrepancies

  And: Switch validation doesn't show any missing/excess rules and meters

  And: The first flow allows traffic on the main path

  And: The first flow allows traffic on the protected path

• Able to swap endpoints (#data.description) for two vxlan flows with the same source and destination switches

  Given: Two flows with the same source and destination switches

  When: Try to swap endpoints for flows

  Then: Endpoints are successfully swapped

  And: Flows validation doesn't show any discrepancies

  And: Switch validation doesn't show any missing/excess rules and meters

• Able to swap endpoints (#data.description) for two qinq flows with the same source and destination switches

  Given: Two flows with the same source and destination switches

  When: Try to swap endpoints for flows

  Then: Endpoints are successfully swapped

  And: Flows validation doesn't show any discrepancies

  And: Switch validation doesn't show any missing/excess rules and meters

• System reverts both flows if fails during rule installation when swapping endpoints

  Given: Two flows with different src switches and same dst

  When: Try to swap flow src endoints, but flow1 src switch does not respond

  Then: Receive error response

  And: First flow is reverted to Down

  And: Second flow is reverted to UP

  When: Delete both flows

  Then: Related switches have no rule anomalies

• Able to swap endpoints for a flow with flowLoop

  Given: Create two flows with the same src and different dst switches

  And: FlowLoop is created for the second flow on the dst switch

  When: Try to swap dst endpoints for two flows

  Then: Endpoints are successfully swapped

  And: Flows validation doesn't show any discrepancies

  And: FlowLoop is still created for the second flow but on the new dst switch

  And: FlowLoop rules are created on the new dst switch

  And: FlowLoop rules are deleted on the old dst switch

  And: Switch validation doesn't show any missing/excess rules and meters

  When: Send traffic via flow2

  Then: Flow doesn't allow traffic, because it is grubbed by flowLoop rules

  And: Counter on the flowLoop rules are increased

ThrottlingReroute

This test verifies that we do not perform a reroute as soon as we receive a reroute request (we talk only about automatic reroutes here; manual reroutes are still performed instantly). Instead, system waits for 'reroute.delay' seconds and if no new reroute requests are issued, it performs ONE reroute for each requested flowId. If ANY new reroute is issued during 'reroute.delay' the timer is refreshed. System should stop refreshing the timer if 'reroute.hardtimeout' is reached and perform all the queued reroutes (unique for each flowId).

• Reroute is not performed while new reroutes are being issued

  Given: Multiple flows that can be rerouted independently (use short unique paths)

  When: All flows break one by one

  Then: The oldest broken flow is still not rerouted before rerouteDelay run out

  And: The oldest broken flow is rerouted when the rerouteDelay runs out

  And: The rest of the flows are rerouted too

• Reroute is performed after hard timeout even though new reroutes are still being issued

  Given: Multiple flows that can be rerouted independently (use short unique paths)

  When: All flows begin to continuously reroute in a loop

  Then: Right until hard timeout should run out no flow reroutes happen

  And: Flows should start to reroute after hard timeout, even though reroutes are still being triggered

• Flow can be safely deleted while it is in the reroute window waiting for reroute

  Given: A flow

  When: Init a flow reroute by breaking current path

  And: Immediately remove the flow before reroute delay runs out and flow is actually rerouted

  Then: The flow is not present in NB

  And: Related switches have no excess rules, though need to wait until server42 rules are deleted

VxlanFlow

This spec checks basic functionality(simple flow(rules, ping, traffic, validate), pinned flow, flow with protected path, default flow) for a flow with VXLAN encapsulation.

NOTE: A flow with the 'VXLAN' encapsulation is supported on a Noviflow switches. So, flow can be created on a Noviflow(src/dst/transit) switches only.

• System allows to create/update encapsulation type for a flow #description

  When: Create a flow with #encapsulationCreate.toString() encapsulation type

  Then: Flow is created with the #encapsulationCreate.toString() encapsulation type

  And: Correct rules are installed

  And: Flow is valid

  And: Flow is pingable

  And: The flow allows traffic

  And: Flow is pingable

  When: Try to update the encapsulation type to #encapsulationUpdate.toString()

  Then: The encapsulation type is changed to #encapsulationUpdate.toString()

  And: Flow is valid

  And: Flow is pingable (though sometimes we have to wait)

  And: Rules are recreated

  And: New rules are installed correctly

  And: The flow allows traffic

• Able to CRUD a pinned flow with 'VXLAN' encapsulation

  When: Create a flow

  Then: Flow is created

  When: Update the flow (pinned=false)

  Then: The pinned option is disabled

• Able to CRUD a vxlan flow with protected path

  Given: Two active VXLAN supported switches with two available path at least

  When: Create a flow with protected path

  Then: Flow is created with protected path

  And: Rules for main and protected paths are created

  And: Validation of flow must be successful

  When: Update flow: disable protected path(allocateProtectedPath=false)

  Then: Protected path is disabled

  And: Rules for protected path are deleted

  And: And rules for main path are recreacted

  And: Validation of flow must be successful

• System allows tagged traffic via default flow(0<->0) with 'VXLAN' encapsulation

  Given: Two active VXLAN supported switches connected to traffgen

  When: Create a default flow

  Then: System allows tagged traffic on the default flow

• Unable to create a VXLAN flow when src and dst switches do not support it

  Given: Src and dst switches do not support VXLAN

  When: Try to create a VXLAN flow

  Then: Human readable error is returned

  When: Create a VLAN flow

  And: Try updated its encap type to VXLAN

  Then: Human readable error is returned

• System selects longer path if shorter path does not support required encapsulation type

  Given: Shortest path transit switch does not support VXLAN and alt paths with VXLAN are available

  When: Create a VXLAN flow

  Then: Flow is built through vxlan-enabled path, even though it is not the shortest

• Unable to create a vxlan flow when dst switch does not support it

  Given: VXLAN supported and not supported switches

  When: Try to create a flow

  Then: Human readable error is returned

• System allows to create/update encapsulation type for a one-switch flow(#encapsulationCreate.toString() -> #encapsulationUpdate.toString())

  When: Try to create a one-switch flow

  Then: Flow is created with the #encapsulationCreate.toString() encapsulation type

  And: Correct rules are installed

  And: Flow is valid

  And: Unable to ping a one-switch vxlan flow

  When: Try to update the encapsulation type to #encapsulationUpdate.toString()

  Then: The encapsulation type is changed to #encapsulationUpdate.toString()

  And: Flow is valid

  And: Rules are recreated

  And: New rules are installed correctly

---

HaFlowCreate

Verify create operations on HA-Flows.

• Valid HA-Flow can be created#trafficDisclaimer, covered cases: #coveredCases

  When: Create an HA-Flow of certain configuration

  Then: Traffic passes through HA-Flow

  And: HA-flow pass validation

  And: HA-Flow is pingable

  And: HA-Flow has been successfully deleted

  And: And involved switches pass validation

• User cannot create an HA-Flow with existent ha_flow_id

  Given: Existing HA-Flow

  When: Try to create the same HA-Flow

  Then: Error is received

• User cannot create an HA-Flow with equal A-B endpoints and different inner vlans at these endpoints

  Given: A switch triplet with equal A-B endpoint switches

  When: Try to create I-shaped HA-Flow request with equal A-B endpoint switches and different innerVlans

  Then: Error is received

• User cannot create a one switch HA-Flow

  Given: A switch

  When: Try to create one switch HA-Flow

  Then: Error is received

HaFlowDiversity

Verify the ability to create diverse HA-Flows in the system.

• Able to create diverse HA-Flows

  Given: Switches with three not overlapping paths at least

  When: Create three Ha-Flows with diversity enabled

  Then: HA-Flow create response contains info about diverse haFlow

  And: All Ha-Flows have diverse HA-Flow IDs in response

  And: All Ha-Flows have different paths

  And: HA-Flow passes flow validation

• Able to create HA-Flow diverse with regular flow that is already in diverse group with another HA-Flow

  Given: Switches with two not overlapping paths at least

  When: Create an HA-Flow without diversity

  And: Create a regular multiSwitch Flow diverse with previously created HA-Flow

  And: Create an additional HA-Flow diverse with simple flow that has another HA-Flow in diverse group

  Then: Create response contains correct info about diverse flows

  When: Get Flow and Ha-Flows details

  Then: All get Flow responses have correct diverse flow IDs

  And: HA-Flow passes flow validation

• Able to create HA-Flow diverse with Y-Flow that is in diverse group with another HA-Flow

  Given: Switches with three not overlapping paths at least

  When: Create an HA-Flow without diversity

  And: Create a Y-Flow diverse with previously created HA-Flow

  And: Create an additional HA-Flow diverse with Y-Flow that has another HA-Flow in diverse group

  Then: The last HA-Flow create response contains info about diverse haFlow

  When: Get Y-flow and Ha-Flows details

  Then: All get flow responses have correct diverse flow IDs

  And: HA-Flow passes flow validation

HaFlowHistory

Verify that history records are created for the basic actions applied to HA-Flow.

• History records with links details are created during link create operations and can be retrieved with timeline

  Given: HA-Flow has been created

  When: Request for history records

  Then: Correct event appears in HA-Flow history and can be retrieved without specifying timeline

  And: Flow history contains all flow properties in the 'state_after' dump section

• History records with links details are created during link #updateType operations and can be retrieved without timeline

  Given: HA-Flow has been created

  When: #type action has been executed

  Then: Correct event appears in HA-Flow history and can be retrieved without specifying timeline

  And: Flow history contains all flow properties in the 'state_before' dump section

  And: Flow history contains all flow properties in the 'state_after' dump section

• History records without links details are created during link deletion and can be retrieved with timeline

  Given: HA-Flow has been created

  When: Delete HA-Flow

  Then: Correct event appears in HA-Flow history and can be retrieved with specifying timeline

  And: All basic fields are correct

  And: Only 'state_before' dump section is present for deletion event

• History records can be retrieved with timeline in milliseconds format

  Given: HA-Flow

  When: Get timestamp after create event

  Then: The appropriate history record has been returned

• History records are created during link unsuccessful rerouting with root cause details and can be retrieved with or without timeline

  Given: HA-Flow has been created

  When: Deactivate the shared switch

  And: Related ISLs are FAILED

  And: HA-Flow goes DOWN

  Then: Correct event appears in HA-Flow history and can be retrieved without specifying timeline

  And: All basic fields are correct and rerouting failure details are available

  And: Correct event appears in HA-Flow history and can be retrieved with specifying timeline

  And: Both retrieved history records are identical

• Empty history returned in case filters return no results

  Given: HA-Flow

  When: Get timestamp after create event

  Then: Check HA-Flow history has no entries

• Only requested amount of history records are returned

  Given: HA-Flow has been created

  And: HA-Flow has been updated

  And: All history records have been retrieved

  When: Request specific amount of history record

  Then: The appropriate number of record has been returned

  When: Requested amount of history record is bigger than actual existing records

  Then: Only flow-specific records are available

• History max_count cannot be <1

  When: Try to get history with max_count 0

  Then: Error due to invalid max_count returned

HaFlowIntentionalReroute

Verify that on-demand HA-Flow reroute operations are performed accurately.

• Not able to reroute to a path with not enough bandwidth available

  Given: An HA-Flow with alternate paths available

  When: Make the current path less preferable than alternatives

  And: Make all alternative paths to have not enough bandwidth to handle the HA-Flow

  And: Init a reroute to a more preferable path

  Then: The HA-Flow is NOT rerouted because of not enough bandwidth on alternative paths

  And: And involved switches pass validation

  And: HA-Flow pass validation

• Able to reroute to a better path if it has enough bandwidth

  Given: An HA-Flow with alternate paths available

  When: Make one of the alternative paths to be the most preferable among all others

  And: Make the future path to have exact bandwidth to handle the HA-Flow

  And: Init a reroute of the HA-Flow

  Then: The HA-Flow is successfully rerouted and goes through the preferable path

  And: And involved switches pass validation

  And: HA-Flow pass validation

  And: 'Thin' ISL has 0 available bandwidth left

• Able to reroute to a path with not enough bandwidth available in case ignoreBandwidth=true

  Given: an HA-Flow with alternate paths available

  When: Make the current path less preferable than alternatives

  And: Make all alternative paths to have not enough bandwidth to handle the HA-Flow

  And: Init a reroute to a more preferable path

  Then: The HA-Flow is rerouted because ignoreBandwidth=true

  And: And involved switches pass validation

  And: HA-Flow pass validation

  And: Available bandwidth was not changed while rerouting due to ignoreBandwidth=true

HaFlowPaths

Verify paths response for ha-flows.

• Meaningful error is returned when requested paths for non-existing HA flow

  When: Request paths for non-existing HA flow

  Then: Meaningful error is returned

• HA flow main path is not overlapped with protected path

  Given: An HA-flow with protected path

  When: Request path for HA flow with protected path

  Then: HA flow main and protected paths do not have common ISLs

HaFlowPathSwap

Verify path swap operations on HA-flows.

• Able to swap main and protected paths manually

  Given: An HA-Flow with protected paths

  And: Current paths are not equal to protected paths

  When: Swap HA-Flow paths

  Then: The sub-flows are switched to protected paths

  And: HA-Flow and related sub-flows are valid

  And: All involved switches pass switch validation

  And: Traffic passes through HA-Flow

  Then: Stats are collected

• Unable to perform the 'swap' request for an HA-Flow without protected path

  Given: An HA-Flow without protected path

  When: Try to swap paths for HA-Flow that doesn't have a protected path

  Then: Human readable error is returned

• Unable to swap paths for a non-existent HA-Flow

  When: Try to swap path on a non-existent HA-Flow

  Then: Human readable error is returned

HaFlowPing

This spec tests 'periodic ping' functionality.

• Able to turn off periodic pings on an HA-Flow

  Given: An HA-Flow with periodic pings turned on

  And: Neither of the sub-flows end on Y-Point (ping is disabled for such kind of HA-Flow)

  When: Turn off periodic pings

  Then: Periodic pings are really disabled

  And: There is no metrics for HA-subflows

• Unable to ping one of the HA-subflows via periodic pings if related ISL is broken

  Given: Pinned HA-flow with periodic pings turned on which won't be rerouted after ISL fails

  And: Neither of the sub-flows end on Y-Point (ping is disabled for such kind of HA-Flow)

  When: Fail one of the HA-subflows ISL (bring switch port down)

  Then: Periodic pings are still enabled

  And: Metrics for the HA-subflow with broken ISL have 'error' status in tsdb

  And: Metrics for HA-subflow with active ISL have 'success' status in tsdb

• Able to turn on periodic pings on an Ha-flow

  Given: Create an Ha-flow without periodic pings turned on

  And: Neither of the sub-flows end on Y-Point (ping is disabled for such kind of HA-Flow)

  When: Turn on periodic ping on an HA-Flow

  Then: Periodic pings are really enabled

  And: Packet counter on catch ping rules grows due to pings happening

  And: Metrics for HA-subflows have 'success' in tsdb

• Unable to ping HA-Flow when one of subflows is one-switch one

  Given: HA-Flow which has one-switch subflow

  When: Ping HA-Flow

  Then: HA-Flow ping is not successful and the appropriate error message has been returned

• Able to ping HA-Flow when neither of the sub-flows end on Y-Point

  Given: HA-Flow has been created

  And: Neither of the sub-flows end on Y-Point (ping is disabled for such kind of HA-Flow)

  When: Ping HA-Flow

  Then: HA-Flow ping is successful

  When: Break one sub-flow by removing flow rules from the intermediate switch

  Then: HA-Flow ping is not successful, and ping for one sub-flow shows that path is broken

  When: Break another sub-flow by removing flow rules from the intermediate switch(after fixing previous discrepancy)

  Then: HA-Flow ping is not successful, and ping for another sub-flow shows that path is broken

  When: All required rules have been installed(sync)

  Then: HA-Flow ping is successful for both sub-flows

HaFlowProtected

Verify operations with protected paths on Ha-Flows.

• Able to enable protected path on an HA-Flow

  Given: A simple HA-Flow

  When: Update flow: enable protected path(allocateProtectedPath=true)

  Then: Update response contains enabled protected path

  And: Protected path is really enabled on the HA-Flow

  And: Protected path is really created

  And: HA-Flow pass validation

  And: All involved switches passes switch validation

• Able to disable protected path on an HA-Flow via partial update

  Given: An HA-Flow with protected path

  When: Patch flow: disable protected path(allocateProtectedPath=false)

  Then: Patch response contains disabled protected path

  And: Protected path is really disabled on the HA-Flow

  And: Protected path is really removed

  And: HA-Flow pass validation

  And: All involved switches passes switch validation

• User can update #data.descr of an HA-Flow with protected path

  Given: An HA-Flow with protected path

  When: Update the ha-Flow

  Then: Requested updates are reflected in the response and in 'get' API

  And: And involved switches pass validation

  And: HA-Flow pass validation

HaFlowReroute

Verify reroute operations on HA-flows.

• Valid HA-flow can be rerouted

  Given: An HA-flow

  When: Fail an HA-flow ISL (bring switch port down)

  Then: The HA-flow was rerouted after reroute delay

  And: History has relevant entries about HA-flow reroute

  And: HA-flow passes validation

  And: All involved switches pass switch validation

  And: Traffic passes through HA-Flow

  Then: Stats are collected

• HA-flow in 'Down' status is rerouted when discovering a new ISL

  Given: An HA-flow

  When: Bring all ports down on the shared switch that are involved in the current and alternative paths

  Then: The HA-flow goes to 'Down' status

  When: Bring all ports up on the shared switch that are involved in the alternative paths

  Then: The HA-flow goes to 'Up' state and the HA-flow was rerouted

  And: The first (shared) subFlow's ISl has been changed due to the ha-Flow reroute

  And: HA-flow passes validation

  And: All involved switches pass switch validation

• HA-flow goes to 'Down' status when ISl of the HA-flow fails and there is no alt path to reroute

  Given: An HA-flow without alternative paths

  And: All ISL ports on the shared switch that are involved in the alternative HA-flow paths are down

  When: Bring port down of ISL which is involved in the current HA-flow paths

  Then: The HA-flow goes to 'Down' status

  And: All involved switches pass switch validation

HaFlowSync

• Able to synchronize an HA-Flow (install missing rules, reinstall existing). protectedPath=#data.protectedPath

  Given: An HA-Flow with deleted rules on shared switch

  When: Synchronize the HA-Flow

  Then: The Ha-flow is synced

  And: The HA-Flow is not rerouted

  And: The HA-Flow is valid

  And: Missing HA-Flow rules are installed (existing ones are reinstalled) on all switches

• Able to synchronize an HA-Flow if HA-Flow switch is inactive protectedPath=#data.protectedPath

  Given: An HA-Flow with down shared endpoint

  When: Synchronize the HA-Flow

  Then: The HA-Flow is not synced

  And: Rules on down switch are not synced

  And: The HA-Flow is not rerouted

  And: Missing HA-Flow rules are installed (existing ones are reinstalled) on UP involved switches

HaFlowUpdate

Verify update and partial update operations on HA-Flows.

• User can update #data.descr of an HA-Flow

  Given: Existing HA-Flow

  When: Update the HA-Flow

  Then: Requested updates are reflected in the response and in 'get' API

  And: And involved switches pass validation

  And: HA-Flow pass validation

• User can update HA-Flow where one of subflows has both ends on shared switch

  Given: Existing HA-Flow where one of subflows has both ends on shared switch

  When: Update the HA-Flow

  Then: Requested updates are reflected in the response and in 'get' API

  And: And involved switches pass validation

  And: HA-Flow pass validation

• User can partially update #data.descr of an HA-Flow

  Given: Existing HA-Flow

  When: Partial update the HA-Flow

  Then: Requested updates are reflected in the response and in 'get' API

  And: And involved switches pass validation

  And: HA-Flow pass validation

• User cannot update an HA-Flow #data.descr

  Given: Existing HA-Flow

  When: Try to update the HA-Flow with invalid payload

  Then: Error is received

  And: And involved switches pass validation

  And: HA-Flow pass validation

• User cannot partial update an HA-Flow with #data.descr

  Given: Existing HA-Flow

  When: Try to partial update the HA-Flow with invalid payload

  Then: Error is received

  And: And involved switches pass validation

  And: HA-Flow pass validation

HaFlowValidation

Verify that missing HA-Flow rule is detected by switch/flow validations

• HA-Flow passes validation after creation

  Given: HA-Flow on non-neighbouring switches

  When: Validate HA-Flow

  Then: HA-Flow is validated successfully

• HA-Flow validation should fail in case of missing rule on #switchRole switch

  Given: HA-Flow on non-neighbouring switches

  When: Delete HA-Flow rule on switch

  Then: HA-Flow validation returns deleted rule in 'Discrepancies' section

• HA-Flow validation should fail in case of missing meter on #switchRole switch

  Given: HA-Flow on non-neighbouring switches

  When: Delete HA-Flow meter

  Then: HA-Flow validation returns rules related to deleted meter in 'Discrepancies' section

---

SubFlow

Verify different actions for a sub-flow.

• Unable to #data.action a sub-flow

  Given: Existing Y-Flow

  When: Invoke a certain action for a sub-flow

  Then: Human readable error is returned

  And: All involved switches pass switch validation

  And: Y-Flow and each sublows are in UP state

  And: Y-Flow passes flow validation

  And: SubFlow passes flow validation

  And: Flow history doesn't contain info about illegal action

  And: Sub flow is pingable

YFlowCreate

Verify create operations on y-flows.

• Valid Y-Flow can be created#trafficDisclaimer, covered cases: #coveredCases

  When: Create a Y-Flow of certain configuration

  Then: Y-Flow has been created successfully

  And: 2 sub-flows are created, visible via regular 'dump flows' API

  And: History has relevant entries about y-Flow creation

  And: User is able to view Y-Flow paths

  And: Y-Flow passes flow validation

  And: Both sub-flows pass flow validation

  And: YFlow is pingable

  And: All involved switches pass switch validation

  And: Bandwidth is properly consumed on shared and non-shared ISLs(not applicable for single switch Y-Flow)

  And: YFlow is pingable #2

  When: Traffic starts to flow on both sub-flows with maximum bandwidth (if applicable)

  Then: Traffic flows on both sub-flows, but does not exceed the Y-Flow bandwidth restriction (~halves for each sub-flow)

  When: Delete the y-flow

  Then: Y-Flow and related sub-flows are removed

  And: History of each subFlows has relevant entries about flow deletion

  And: All involved switches pass switch validation

• System forbids to create a Y-Flow with conflict: #data.descr

  When: Try creating a Y-Flow with one endpoint being in conflict with the other one

  Then: Error is received, describing the problem

  And: 'Get' Y-Flows doesn't return the flow

  And: 'Get' flows doesn't return the sub-flows

• System forbids to create a Y-Flow with conflict: subflow1 vlans are [0,X] and subflow2 vlans are [X,0] on shared endpoint

  When: Try creating a Y-Flow with one endpoint being in conflict with the other one

  Then: Error is received, describing the problem

  And: 'Get' y-flows doesn't return the flow

  And: 'Get' flows doesn't return the sub-flows

• System forbids to create a Y-Flow with conflict: shared endpoint port is inside a LAG group

  Given: A LAG port

  When: Try creating a Y-Flow with shared endpoint port being inside LAG

  Then: Error is received, describing the problem

  And: 'Get' y-flows doesn't return the flow

  And: 'Get' flows doesn't return the sub-flows

• System allows to create Y-Flow with bandwidth equal to link bandwidth between shared endpoint and y-point (#4965)

  Given: three switches and potential Y-Flow point

  When: Y-Flow plan for them with bandwidth equal to ISL bandwidth

  Then: Y-Flow is created and UP

YFlowDiversity

Verify the ability to create diverse y-flows in the system.

• Able to create diverse Y-Flows

  Given: Switches with three not overlapping paths at least

  When: Create three Y-Flows with diversity enabled

  Then: Y-Flow create response contains info about diverse yFlow

  And: All Y-Flows have diverse yFlow IDs in response

  And: All Y-Flows have different paths

  And: Y-Flows histories contains 'diverse' information

  And: Y-Flow passes flow validation

  When: Delete Y-Flows

  Then: Y-Flows' histories contain 'diverseGroupId' information in 'delete' operation

• Able to update Y-Flow to became diverse with simple multiSwitch flow

  Given: Switches with two not overlapping paths at least

  And: Y-Flow created

  And: Simple multiSwitch flow on the same path as first sub-flow

  When: Update Y-Flow to become diverse with simple multiSwitch flow

  Then: Update response contains information about diverse flow

  And: Y-Flow has been successfully updated with appropriate diverse info

  And: Simple multi switch flow has the 'diverse_with' field

  And: First sub-flow became diverse and changed path

  And: First sub flow history contains 'groupId' information

  And: Y-Flow passes flow validation

  And: Simple flow is valid

  When: Partially update Y-Flow to become not diverse with simple multiSwitch flow

  Then: Update response contains information about diverse flow

  And: Y-Flow is really updated

  And: Simple multi switch flow doesn't have the 'diverse_with' field

• Able to create Y-Flow with one switch sub flow and diverse with simple multiSwitch flow

  Given: Two switches with two not overlapping paths at least

  And: Simple multiSwitch flow

  When: Create a Y-Flow with one switch sub flow and diversity with simple flow

  Then: Create response contains information about diverse flow

  And: Y-Flow is diverse with Flow

  And: Flow is diverse with Y-Flow

  And: Y-Flow passes flow validation

  And: Simple Flow is valid

• Able to get Y-Flow paths with correct overlapping segments stats

  Given: Switches with three not overlapping paths at least

  When: Create three Y-Flows with diversity enabled

  And: Get flow path for all y-flows

  Then: All Y-Flow paths have diverse group information

• Able to get Y-Flow paths with correct overlapping segments stats with one-switch Y-Flow

  Given: Three switches

  When: Create y-flow

  And: Create one-switch y-flows on shared and ep1 switches in the same diversity group

  And: Get flow paths for all y-flows

  Then: All YFlows have correct diverse group information

• Able to get Y-Flow paths with with diversity part when flows become diverse after partial update

  Given: Switches with three not overlapping paths at least

  And: Create two Y-Flows

  When: Partially update Y-Flow to become diverse with another one

  And: Y-Flow has been updated successfully and become part of diverse group with another Y-Flow

  And: Request paths for both Y-Flows

  Then: Path request contain all the subflows in diverse group

YFlowPathSwap

Verify path swap operations on y-flows.

• Able to swap main and protected paths manually

  Given: A Y-Flow with protected paths

  And: Current paths are not equal to protected paths

  When: Swap Y-Flow paths

  Then: The sub-flows are switched to protected paths

  And: YFlow and related sub-flows are valid

  And: All involved switches passes switch validation

  When: Traffic starts to flow on both sub-flows with maximum bandwidth (if applicable)

  Then: Traffic flows on both sub-flows, but does not exceed the Y-Flow bandwidth restriction (~halves for each sub-flow)

  And: Y-Flow and subflows stats are available (flow.raw.bytes)

• System is able to switch a y-flow to protected paths

  Given: A y-flow with protected paths

  And: Current paths are not equal to protected paths

  And: Other ISLs have not enough bandwidth to host the flows in case of reroute

  When: Break ISL on the main path (bring port down) to init auto swap

  Then: The sub-flows are switched to protected paths

  And: YFlow and related sub-flows are valid

  And: All involved switches passes switch validation

  When: Restore port status

  Then: Paths of the y-flow is not changed

  And: All involved switches passes switch validation

• Unable to perform the 'swap' request for a flow without protected path

  Given: A y-flow without protected path

  When: Try to swap paths for y-flow that doesn't have a protected path

  Then: Human readable error is returned

• Unable to swap paths for a non-existent y-flow

  When: Try to swap path on a non-existent y-flow

  Then: Human readable error is returned

• Unable to swap paths for an inactive y-flow

  Given: A y-flow with protected paths

  And: Current paths are not equal to protected paths

  And: Other ISLs have not enough bandwidth to host the flows in case of reroute

  When: Break ISL on the protected path (bring port down) to make it INACTIVE

  Then: The sub-flows are switched to protected paths

  When: Try to swap paths when main/protected paths are not available

  Then: Human readable error is returned

  When: Restore port status

  Then: Paths of the y-flow is not changed

YFlowPing

This spec tests 'periodic ping' functionality.

• Able to turn on periodic pings on a y-flow

  When: Create a y-flow with periodic pings turned on

  Then: Periodic pings is really enabled

  And: Packet counter on catch ping rules grows due to pings happening

• Able to ping y-flow when one of subflows is one-switch one (#5019)

  Given: y-flow which has one-switch subflow

  When: ping y-flow

  Then: y-flow ping is not successful, but one of subflows ping is successful

  When: Break the flow by removing flow rules from the intermediate switch

  Then: y-flow ping is not successful, and ping for another sub-flow shows that path is broken

  When: All required rules have been installed(sync)

  Then: y-flow ping is not successful, but one of subflows ping is successful

• Able to ping y-flow and detect when path is broken

  Given: y-flow has been created

  And: ping for y-flow detects no discrepancy

  When: Break one sub-flow by removing flow rules from the intermediate switch

  Then: y-flow ping is not successful, and ping for one sub-flow shows that path is broken

  When: Break another sub-flow by removing flow rules from the intermediate switch(after fixing previous discrepancy)

  Then: y-flow ping is not successful, and ping for another sub-flow shows that path is broken

  When: All required rules have been installed(sync)

  Then: y-flow ping is successful for both sub-flows

YFlowProtected

Verify reroute operations on y-flows.

• Able to enable/disable protected path on a flow

  Given: A simple y-flow

  When: Update flow: enable protected path(allocateProtectedPath=true)

  Then: Protected path is really enabled on the YFlow

  And: Protected path is really enabled on the sub flows

  And: Protected path is really created

  And: YFlow and related sub-flows are valid

  And: All involved switches pass switch validation

  When: Disable protected path via partial update

  Then: Protected path is really disabled for YFlow/sub-flows

  And: Protected path is deleted

  And: YFlow and related sub-flows are valid

  And: All involved switches passes switch validation

YFlowReroute

Verify reroute operations on y-flows.

• Valid y-flow can be rerouted

  Given: A qinq y-flow

  When: Fail a flow ISL (bring switch port down)

  Then: The flow was rerouted after reroute delay

  And: History has relevant entries about y-flow reroute

  And: Y-flow passes flow validation

  And: Both sub-flows pass flow validation

  And: All involved switches pass switch validation

  When: Traffic starts to flow on both sub-flows with maximum bandwidth (if applicable)

  Then: Traffic flows on both sub-flows, but does not exceed the y-flow bandwidth restriction (~halves for each sub-flow)

  And: Subflows stats are available (flow.raw.bytes)

• Y-Flow reroute has not been executed when both sub-flows are on the best path

  Given: Y-Flow has been created successfully

  When: Y-Flow reroute has been called

  Then: The appropriate error has been returned

  And: Y-Flow path has not been changed

• Y-Flow reroute has been executed when more preferable path is available for both sub-flows (shared path cost was changed)

  Given: The appropriate switches have been collected

  And: The ISLs cost between switches has been changed to make preferable path

  And: Y-Flow with shared path has been created successfully

  And: Shared ISLs cost has been changed to provide on-demand Y-Flow reroute

  When: Y-Flow reroute has been called

  Then: Y-Flow reroute has been executed successfully

  And: Both sub-flows paths have been changed

• Y-Flow reroute has been executed when more preferable path is available for one of the sub-flows

  Given: The appropriate switches have been collected

  And: The ISLs cost between switches has been changed to make preferable path

  And: Y-Flow with shared path has been created successfully

  And: The required ISLs cost has been updated to make manual reroute available

  When: Y-Flow reroute has been called

  Then: Y-Flow has been rerouted successfully

  And: The appropriate flow has been rerouted

• Y-Flow reroute has not been executed when one sub-flow is on the best path and there is no alternative path for another sub-flow due to the down ISLs

  Given: Y-Flow has been created successfully

  And: Sub-flows not intersected ISLs have been collected

  And: Switch off all ISLs on the terminal switch

  When: Y-Flow reroute has been called

  Then: The appropriate error has been returned

  And: Y-Flow path has not been changed

YFlowUpdate

Verify update and partial update operations on y-flows.

• User can update #data.descr of a Y-Flow

  Given: Existing Y-Flow

  When: Update the y-flow

  Then: Requested updates are reflected in the response and in 'get' API

  And: All related switches have no discrepancies

• User can update y-flow where one of subflows has both ends on shared switch

  Given: Existing y-flow where one of subflows has both ends on shared switch

  When: Update the y-flow

  Then: Requested updates are reflected in the response and in 'get' API

  And: All related switches have no discrepancies

• User can partially update fields of one-switch y-flow

  Given: Existing one-switch y-flow

  When: Partial update the y-flow

  Then: Requested updates are reflected in the response and in 'get' API

  And: All related switches have no discrepancies

• User can partially update #data.descr of a y-flow

  Given: Existing y-flow

  When: Partial update the y-flow

  Then: Requested updates are reflected in the response and in 'get' API

  And: All related switches have no discrepancies

• User cannot update a y-flow with #data.descr

  Given: Existing y-flow

  When: Try to update the y-flow with invalid payload

  Then: Error is received

• User cannot partial update a y-flow with #data.descr

  Given: Existing y-flow

  When: Try to partial update the y-flow with invalid payload

  Then: Error is received

YFlowValidation

Verify that missing yFlow rule is detected by switch/flow validations. And make sure that the yFlow rule can be installed by syncSw/syncYFlow endpoints.

• Y-Flow validation should fail in case of missing Y-Flow shared rule (#data.description)

  Given: Existing Y-Flow

  When: Delete shared Y-Flow rule

  Then: Y-Flow validate detects discrepancies

  And: Simple flow validation detects discrepancies

  And: Switch validation detects missing Y-Flow rule

  When: Synchronize the shared switch

  Then: Y-Flow/subFlow passes flow validation

  And: Switch passes validation

• Y-Flow/subFlow validation should fail in case of missing subFlow rule (#data.description)

  Given: Existing Y-Flow

  When: Delete reverse rule of subFlow_1

  Then: Y-Flow is not valid

  And: Discrepancies has been detected for broken sub-flow REVERSE direction

  And: No discrepancies has been detected for broken sub-flow FORWARD direction

  And: No discrepancies has been detected for another sub-flow

  And: Simple flow validation detects discrepancies for the subFlow_1 REVERSE direction only

  And: Switch validation detects missing reverse rule only for the subFlow_1

  When: Sync Y-Flow

  Then: Y-Flow/subFlow passes flow validation

  And: Switches pass validation

• Unable to #data.action a non-existent Y-Flow

  When: Invoke a certain action for a non-existent Y-Flow

  Then: Human readable error is returned

---

Server42-related

---

Server42IslRtt

• ISL RTT stats are ONLY available if both server42_isl_rtt feature toggle is ON and server42_isl_rtt is enabled for switches

  Given: An active ISL with both switches having server42

  When: Enable server42IslRtt features toggle

  And: server42IslRtt is enabled for both switches(props)

  Then: ISL RTT stats are available for both forward and reverse direction

• ISL RTT stats are NOT available if featureToggle: #featureToggle and switchToggle: #switchToggle

  Given: An active ISL with both switches having server42

  When: server42IslRtt feature toggle is set #featureToggle

  And: server42IslRtt is set #switchToggle on src and dst switches

  Then: ISL RTT forward stats are NOT available for both forward and reverse directions

• ISL RTT stats are available if both endpoints are connected to the same server42 (same pop)

  Given: An active ISL with both switches connected to the same server42 instance

  When: server42IslRtt feature toggle is set to true

  And: server42IslRtt is enabled on src and dst switches

  Then: ISL RTT for both forward and reverse stats are available

  And: ISL latency value is updated in db in both direction from s42

• ISL RTT stats are not available for a moved link and available for a new link

  Given: An active a-switch ISL with both switches having server42

  And: A non-connected a-switch link with server42

  And: Replug one end of the connected link to the not connected one

  When: server42IslRtt feature toggle is set to true

  And: server42IslRtt is enabled on src and dst switches

  Then: ISL RTT rules are not deleted on the src switch for the moved link

  And: ISL RTT rules are not installed for the new link because it is the same as moved(portNumber)

  And: ISL RTT rules are installed on the new dst switch for the new link

  And: ISL RTT rules are not deleted on the origin dst switch for the moved link

  And: Involved switches pass the switch validation

  And: Expect ISL RTT for new ISL in forward/reverse directions

  And: Expect no ISL RTT for MOVED ISL in forward/reverse directions

  When: Replug the link back where it was

  And: Remove the MOVED ISL

  Then: Server42 ISL RTT rules are deleted on the dst switch of the moved link

  And: All involved switches pass switch validation

• No ISL RTT stats in both directions in case link is UP in forward direction only

  Given: An active a-switch ISL with both switches having server42 and with broken reverse direction

  When: server42IslRtt feature toggle is set to true

  And: server42IslRtt is enabled on src and dst switches

  Then: No ISL RTT stats in both directions because reverse direction is broken

  When: Restore link in reverse direction

  Then: ISL RTT stats for ISL in forward/reverse directions are available

• SERVER_42_ISL_RTT rules are updated according to changes in swProps

  When: server42IslRtt feature toggle is set to false

  And: server42IslRtt is disabled on the switch

  Then: No IslRtt rules on the switch

  When: server42IslRtt feature toggle is set to true

  Then: No IslRtt rules on the switch

  When: server42IslRtt feature toggle is set to false

  And: server42IslRtt is enabled on the switch

  Then: No IslRtt rules on the switch

  When: server42IslRtt feature toggle is set to true and enabled on the switch(previous step)

  Then: IslRtt rules are installed on the switch

  When: Update server42Port on the switch

  Then: SERVER_42_ISL_RTT_OUTPUT_COOKIE and SERVER_42_ISL_RTT_INPUT rules updated according to the changes

  And: SERVER_42_ISL_RTT_TURNING_COOKIE is not changed

  When: server42IslRtt feature toggle is set to false

  And: server42IslRtt is set to AUTO on the switch

  Then: No IslRtt rules on the switch

  When: server42IslRtt feature toggle is set true

  Then: IslRtt rules are installed

  When: Update server42Port on the switch(revert to origin)

  Then: SERVER_42_ISL_RTT_OUTPUT_COOKIE and SERVER_42_ISL_RTT_INPUT rules updated according to the changes

  And: SERVER_42_ISL_RTT_TURNING_COOKIE is not changed

• ISL Rtt stats are available in case link and switch are under maintenance

  Given: An active ISL under maintenance with both switches having server42, dst switch is under maintenance

  When: server42IslRtt feature toggle is turned on

  And: Enable server42IslRtt on the src and dst switches

  Then: Expect ISL RTT for ISL in forward/reverse directions

• ISL Rtt stats are available in case link is RTL and a switch is disconnected

  Given: An active RTL ISL with both switches having server42

  When: server42IslRtt feature toggle is turned on

  And: server42IslRtt is enabled on src and dst switches

  And: Deactivate the src switch

  Then: ISL RTT stats are available in both directions because RTL link is UP

  When: Connect back the src switch

  Then: ISL Rtt rules still exist on the src switch

  And: Switch is valid

  And: ISL RTT stats in both directions are available

• System is able to detect and sync missing ISL Rtt rules

  Given: An active ISL with both switches having server42

  When: Delete ISL Rtt rules on the src switch

  Then: Rules are really deleted

  And: Switch validation shows deleted rules as missing

  And: No ISL Rtt stats in forward/reverse directions

  When: Sync the src switch

  Then: Sync response contains ISL Rtt rules into the installed section

  And: ISL Rtt rules are really installed

  And: ISL Rtt stats are available in both directions

Server42FlowRtt

Verify that statistic is collected from server42 Rtt

• Create a #flowDescription flow with server42 Rtt feature and check datapoints in tsdb

  Given: Two active switches, src has server42 connected

  When: Set server42FlowRtt toggle to true

  And: server42FlowRtt is enabled on src and dst switches

  And: Create a flow

  Then: Check if stats for forward are available

• Flow rtt stats are available in forward and reverse directions for new flows

  Given: Two active switches with switch having server42

  And: server42FlowRtt feature toggle is set to true

  And: server42FlowRtt is enabled on src and dst switches

  When: Create a flow for forward metric

  And: Create a reversed flow for backward metric

  Then: Server42 input/ingress rules are installed

  And: Involved switches pass switch validation

  And: Check if stats for forward and reverse flows are available

• Flow rtt stats are available if both endpoints are connected to the same server42, same pop

  Given: Two active switches connected to the same server42 instance

  And: server42FlowRtt feature enabled globally and on src/dst switch

  When: Create a flow

  Then: Involved switches pass switch validation

  And: Stats for both directions are available

  When: Disable flow rtt on dst switch

  Then: Stats are available in forward direction

  And: Stats are not available in reverse direction

• Able to synchronize a flow (install missing server42 rules)

  Given: A switch pair connected to server42

  And: A flow on the given switch pair

  When: Delete ingress server42 rule related to the flow on the src switch

  Then: System detects missing rule on the src switch

  And: Flow is valid and UP

  And: server42 stats for forward direction are not increased

  And: server42 stats for reverse direction are increased

  When: Synchronize the flow

  Then: Missing ingress server42 rule is reinstalled on the src switch

  Then: server42 stats for forward direction are available again

• Able to swapEndpoint for a flow with enabled server42 on it

  Given: Two switch pairs with different src switches and the same dst switch

  And: server42 is enabled on the src sw of the first switch pair

  And: Two flows on the given switch pairs

  When: Try to swap src endpoints for two flows

  Then: Endpoints are successfully swapped

  And: Flows validation doesn't show any discrepancies

  And: All switches are valid

  And: server42 stats are available for the flow2 in the forward direction

  And: server42 stats are not available any more for the flow1 in the forward direction

• Rtt statistic is available for a flow in case switch is not connected to server42

  Given: Two active switches, only src has server42 connected

  When: Set server42FlowRtt toggle to true

  And: server42FlowRtt is enabled on src switch

  And: Create a flow

  Then: Stats from server42 only for forward direction are available

  And: Stats from flow monitoring feature for reverse direction only are available

  When: Disable server42FlowRtt on the src switch

  Then: Stats from flow monitoring feature for forward direction are available

• Flow rtt stats are still available after updating a #data.flowDescription flow

  Given: Two active switches, connected to the server42

  And: server42FlowRtt toggle is set to true

  And: server42FlowRtt is enabled on src and dst switches

  And: A flow

  When: Update the flow(vlan/innerVlan) via partialUpdate on the src/dst endpoint

  Then: Check if stats for forward/reverse directions are available

  And: Flow is valid

  And: The src switch is valid

• Flow rtt stats are available after updating switch properties related to server42

  Given: Two active switches, src has server42 connected with incorrect config in swProps

  When: Update the server42 in switch properties on ths src switch(incorrect port)

  Then: server42 rules on the switch are updated

  And: The src switch is valid

  When: Create a flow on the given switch pair

  Then: Flow rtt stats are not available due to incorrect s42 port on the src switch

  When: Set correct config for the server42 on the src switch

  Then: server42 related rules are updated according to the new config

  And: The src switch is valid

  And: Flow rtt stats are available

• Rtt statistic is available for a flow on a LAG port

  Given: Two active switches, both have server42 connected

  And: server42FlowRtt toggle is set to true

  And: server42FlowRtt is enabled on src/dst switches

  When: Create a LAG port on the src switch

  And: Create a flow

  Then: Stats from server42 for forward/reverse directions are available

• Stats are available only if both global and switch toggles are 'on' on both endpoints

  Given: Two active switches with having server42

  And: server42FlowRtt toggle is turned off

  And: server42FlowRtt is turned off on src and dst

  And: Flow for forward metric is created

  And: Reversed flow for backward metric is created

  Expect: Involved switches pass switch validation

  When: Wait for several seconds

  Then: Expect no flow rtt stats for forward flow

  And: Expect no flow rtt stats for reversed flow

  When: Enable global rtt toggle

  And: Wait for several seconds

  Then: Expect no flow rtt stats for forward flow

  And: Expect no flow rtt stats for reversed flow

  When: Enable switch rtt toggle on src and dst

  Then: Stats for forward and reverse flow are available

  When: Disable switch rtt toggle on dst (still enabled on src)

  Then: Stats for forward and reverse flow are available

  When: Disable global toggle

  And: Wait for several seconds

  Then: Expect no flow rtt stats for forward flow

  And: Expect no flow rtt stats for reversed flow

Server42YFlowRtt

• Create an Y-Flow (#description) with server42 Rtt feature and check datapoints in tsdb

  Given: Three active switches with server42 connected

  When: Set server42FlowRtt toggle to true

  And: server42FlowRtt is enabled on all switches

  And: Create a Y-Flow

  Then: Check if stats for FORWARD and REVERSE directions are available for the first sub-flow

  And: Check if stats for FORWARD and REVERSE directions are available for the second sub-flow

  When: Delete Y-Flow

  Then: All Server42 flow-related rules are deleted

• Y-Flow rtt stats are available if both endpoints are connected to the same server42(same pop)

  Given: Three active switches with server42 connected

  And: server42FlowRtt feature enabled globally and on src/dst switch

  When: Create a Y-Flow

  Then: Involved switches pass switch validation

  And: Stats for both directions are available for the first sub-flow

  And: Stats for both directions are available for the second sub-flow

  When: Disable flow rtt on shared switch

  Then: Stats are available in REVERSE direction for both sub-flows

  And: Stats are absent in FORWARD direction for both sub-flows

• Rtt statistic is available for a Y-Flow in case switch is not connected to server42

  Given: Three active switches with server42 connected

  And: Switches triplet with ONLY shared switch that supports server42 feature

  And: server42FlowRtt feature enabled globally and switch ON for appropriate switches(swT)

  When: Create a Y-Flow

  Then: Stats from server42 only for FORWARD direction are available for the first sub-flow

  And: Stats from server42 only for FORWARD direction are available for the second sub-flow

  And: Flow monitoring stats for REVERSE direction are available for the first sub-flow

  And: Flow monitoring stats for REVERSE direction are available for the second sub-flow

  When: Disable server42FlowRtt on the src switch

  Then: Flow monitoring stats for FORWARD direction are available for both sub-flows

• Able to swapEndpoint for a Y-Flow with enabled server42 on it

  Given: Three active switches with server42 connected

  And: Switches triplet doesn't contain WB164 switch

  And: server42FlowRtt feature enabled globally and switch ON for appropriate switches(swT)

  And: Create a Y-Flow

  And: Stats are available for both FORWARD and REVERSE directions are available for the first sub-flow

  And: Stats are available for both FORWARD and REVERSE directions are available for the second sub-flow

  When: Try to swap src endpoints for two flows

  Then: Endpoints are successfully swapped

  And: Y-Flow validation passes

  And: All switches are valid

  And: Stats are available for both FORWARD and REVERSE directions are available for the first sub-flow

  And: Stats are available for both FORWARD and REVERSE directions are available for the second sub-flow

  When: Delete Y-Flow

  Then: All Server42 flow-related rules are deleted

• Able to synchronize a Y-Flow (install missing server42 rules)

  Given: Three active switches with server42 connected

  And: server42FlowRtt feature enabled globally and switch ON for appropriate switches(swT)

  And: Create a Y-Flow

  And: Stats for both directions are available for the first sub-flow

  And: Stats for both directions are available for the second sub-flow

  When: Delete ingress server42 rule related to the flow on the shared switches

  Then: System detects missing rule on the shared switch

  And: Y-Flow is valid and UP

  Then: Stats are available in REVERSE direction for both sub-flows

  And: Stats are absent in FORWARD direction for both sub-flows

  When: Synchronize the Y-Flow

  Then: Missing ingress server42 rule is reinstalled on the shared switch

  Then: Server42 stats for FORWARD direction are available again

• Y-Flow rtt stats are still available after updating Y-Flow: #description

  Given: Three active switches with server42 connected

  And: Switches triplet doesn't contain WB164 switch

  And: server42FlowRtt feature enabled globally and switch ON for appropriate switches(swT)

  And: Create a Y-Flow

  When: Update the Y-Flow: #description

  Then: Y-Flow is 'Up' after updating

  And: Stats for both directions are available for the first sub-flow

  And: Stats for both directions are available for the second sub-flow

  And: Y-Flow is valid

  And: Each switch from triplet is valid

• Y-Flow rtt stats are available only if both global and switch toggles are 'ON' on both endpoints

  Given: Three active switches with server42 connected

  And: server42FlowRtt toggle is turned off

  And: server42FlowRtt is turned off on all switches

  When: Create a Y-Flow

  Then: Involved switches pass switch validation

  And: Expect no Y-Flow rtt stats for FORWARD and REVERSE directions

  When: Enable global rtt toggle

  Then: Expect no flow rtt stats for FORWARD and REVERSE direction for the first sub-flow

  And: Expect no flow rtt stats for FORWARD and REVERSE direction for the second sub-flow

  When: Enable switch rtt toggle on src and dst

  Then: Stats for FORWARD and REVERSE directions are available for the first sub-flow

  And: Stats for FORWARD and REVERSE directions are available for the second sub-flow

  When: Disable switch rtt toggle on ep1 and ep2 ends

  Then: Stats for FORWARD direction are available for both sub-flows

  When: Disable global toggle

  And: Wait for several seconds

  Then: Expect no flow rtt stats for FORWARD and REVERSE direction for the first sub-flow

  And: Expect no flow rtt stats for FORWARD and REVERSE direction for the second sub-flow

Server42HaFlowRtt

• Create an Ha-Flow (#description) with server42 Rtt feature and check datapoints in tsdb

  Given: Three active switches with server42 connected

  When: Set server42FlowRtt toggle to true

  And: server42FlowRtt is enabled on all switches

  And: Create Ha-Flow

  Then: Check if stats for FORWARD and REVERSE directions are available for the first sub-Flow

  And: Check if stats for FORWARD and REVERSE directions are available for the second sub-Flow

• Able to synchronize an Ha-Flow(shared path: #isHaFlowWithSharedPath) with the following installation of missing server42 rules

  Given: Three active switches with server42 connected

  And: Set server42FlowRtt toggle to true

  And: server42FlowRtt is enabled on all switches

  And: Create Ha-Flow

  And: Verify server42 rtt stats are available for both sub-Flows in forward and reverse direction

  When: Delete ingress server42 rule(s) related to the flow on the shared switch

  Then: System detects missing rule on the shared switch

  And: Ha-Flow is valid and UP

  And: The server42 stats for both sub-Flows in forward direction are not increased

  And: The server42 stats for both sub-Flows in reverse direction are increased

  When: Synchronize the Ha-Flow

  Then: Missing ingress server42 rules are reinstalled on the shared switch

  Then: The server42 stats for both FORWARD and REVERSE directions are available for the first sub-flow

  And: The server42 stats for both FORWARD and REVERSE directions are available for the second sub-flow

---

Network

---

PathCheck

• No path validation errors for valid path without limitations

  Given: Path for non-neighbouring switches

  When: Check the path without limitations

  Then: Path check result doesn't have validation messages

• Path check errors returned for each segment and each type of problem

  Given: Path of at least three switches

  And: Source switch supports only Transit VLAN encapsulation

  When: Check the path where one switch doesn't support flow encapsulation type and all links don't have enough BW

  Then: Path check result has multiple lack of bandwidth errors and at least one encapsulation support one

• Latency check errors are returned for the whole existing flow

  Given: Path of at least three switches

  And: Flow with cost computation strategy on that path

  When: Check the path (equal to the flow) if the computation strategy would be LATENCY and max_latency would be too low

  Then: Path check result returns latency validation errors (1 per tier1 and tier 2, per forward and revers paths)

• Path intersection check errors are returned for each segment of existing flow

  Given: Flow has been created successfully

  And: Path with intersected segment(s) for verification has been collected

  And: Involved ISls have been collected

  When: Check if the potential path has intersections with existing one

  Then: Path check reports expected amount of intersecting segments

• Path intersection check errors are returned for each segment of each flow in diverse group

  Given: List of required neighbouring switches has been collected

  And: Two flows in one diverse group have been created

  And: Paths for both flows have been collected

  When: Check potential path that has NO intersection with both flows from diverse group

  Then: Path check reports has No validation error about intersection

  When: Check potential path that has intersection ONLY with one flow from diverse group

  Then: Path check reports has ONLY one intersecting segment

  When: Check potential path that has intersection with both flows from diverse group

  Then: Path check reports has intersecting segments with both flows from diverse group

PathComputation

• Default path computation strategy is used when flow does not specify it

  Given: Default path computation strategy is COST

  And: Switch pair with two paths at least

  And: Update paths so that one path has minimal total latency and the other has minimal total cost

  When: Create flow without selecting path strategy

  Then: Flow is created with 'Cost' strategy (current default)

  And: Flow is actually built on the path with the least cost

  When: Update default strategy to LATENCY

  Then: Existing flow remains with COST strategy and on the same path

  And: Manual reroute of the flow responds that flow is already on the best path

  When: Create a new flow without specifying path computation strategy

  Then: New flow is created with 'Latency' strategy (current default)

  And: New flow actually uses path with the least latency (ignoring cost)

• Flow path computation strategy can be updated from LATENCY to COST

  Given: Switch pair with two paths at least

  And: Update paths so that one path has minimal total latency and the other has minimal total cost

  When: Create flow using Latency strategy

  Then: Flow is built on the least-latency path

  When: Update flow path strategy to 'Cost'

  Then: Flow path has changed to the least-cost path

• Target flow path computation strategy is not applied immediately in case flow was updated partially

  Given: Switch pair with two paths at least

  And: A flow with cost strategy

  When: Update path computation strategy(cost -> latency) via partialUpdate

  Then: Path computation strategy is not changed

  And: Flow is valid

  When: Sync the flow

  Then: Path computation strategy is updated and targetPathComputationStrategy is deleted

• Target path computation strategy is applied after updating/rerouting a flow

  Given: Switch pair with two paths at least

  And: A flow with cost strategy

  When: Update path computation strategy(cost -> latency) via partialUpdate

  And: Reroute the flow

  Then: Path computation strategy is updated and targetPathComputationStrategy is deleted

  When: Update path computation strategy(latency -> cost) via partialUpdate

  And: Update the flow

  Then: Path computation strategy is updated and targetPathComputationStrategy is deleted

Paths

• Get paths between not neighboring switches

  Given: Two active not neighboring switches

  And: Create a flow to reduce available bandwidth on some path between these two switches

  When: Get paths between switches

  Then: Paths will be sorted by bandwidth (descending order) and then by latency (ascending order)

  Then: Maximum count of paths can be changed during PCE calculations

• Able to get paths between switches for the LATENCY strategy

  Given: Two active not neighboring switches

  And: Create a flow to reduce available bandwidth on some path between these two switches

  When: Get paths between switches using the LATENCY strategy

  Then: Paths will be sorted by latency (ascending order) and then by bandwidth (descending order)

• Unable to get paths for non-existing switch

  When: Try to get paths between #problemDescription

  Then: Get error because request is invalid

• Unable to get paths for one switch

  When: Try to get paths between #problemDescription

  Then: Get error because request is invalid

• Unable to get paths with max_latency strategy without max latency parameter

  Given: Two active not neighboring switches

  When: Try to get paths between switches with max_latency strategy but without max_latency parameter

  Then: Human readable error is returned

• Unable to get a path for a 'vxlan' flowEncapsulationType when switches do not support it

  Given: Two active not supported 'vxlan' flowEncapsulationType switches

  When: Try to get a path for a 'vxlan' flowEncapsulationType between the given switches

  Then: Human readable error is returned

• Protected path is #isIncludedString included into path list if #isIncludedString requested

  Given: Two switches with potential protected path

  When: Request available protected paths between two switches

  Then: All paths #isIncludedString have protected ones and protected path differs from main path

• Protected path is null if it doesn't match criteria

  Given: Two non-neighbouring switches with the one path shorter than others

  When: Request available protected paths matching exactly one path

  Then:

---

Statistics

---

FlowStat

Verify that statistic is collected for different type of flow

• System is able to collect stats after intentional swapping flow path to protected

  Given: Two active neighboring switches with two diverse paths at least

  And: Flow with protected path

  When: Generate traffic on the given flow

  Then: Stats collects stat for main path cookies

  And: System collects stats for egress cookie of protected path with zero value

  When: Swap main and protected path

  And: Wait till stats from old main path are collected

  And: Generate traffic on the flow

  Then: System collects stats for previous egress cookie of protected path with non zero value

• System collects stats when a protected flow was intentionally rerouted

  Given: Two active not neighboring switches with three diverse paths at least

  And: A flow with protected path

  When: Generate traffic on the given flow

  Then: Stats is not empty for main path cookies

  And: Stats is empty for protected path egress cookie

  When: Make the current and protected path less preferable than alternatives

  And: Init intentional reroute

  And: Generate traffic on the flow

  Then: Stats is not empty for new main path cookies

  And: Stats is empty for a new protected path egress cookie

• System collects stats when a protected flow was automatically rerouted

  Given: Two active not neighboring switches with three not overlapping paths at least

  And: A flow with protected path

  When: Generate traffic on the given flow

  Then: System collects stats for main path cookies

  And: System collects stats for egress cookie of protected path with zero value

  When: Break ISL on the main path (bring port down) to init auto swap

  And: Generate traffic on the flow

  Then: System collects stats for previous egress cookie of protected path with non zero value

  And: System doesn't collect stats for previous main path cookies due to main path is broken

• System collects stat when protected flow is DEGRADED

  Given: Two active not neighboring switches with two not overlapping paths at least

  And: A flow with protected path

  And: All alternative paths are unavailable (bring ports down on the source switch)

  When: Break ISL on a protected path (bring port down) for changing the flow state to DEGRADED

  And: Generate traffic on the given flow

  Then: System collects stats for main path cookies

• System collects stats when flow is pinned and unmetered

  Given: Two active not neighboring switches

  And: An unmetered flow

  When: Generate traffic on the given flow

  Then: System collects stats for egress/ingress cookies

• System is able to collect stats after partial updating(port) on a flow endpoint

  Given: Two active neighboring switches connected to the traffgens

  And: A flow with updated port on src endpoint via partial update

  When: Generate traffic on the flow

  Then: System collects stats for ingress/egress cookies

• System is able to collect stats after partial updating(vlan) on a flow endpoint

  Given: Two active neighboring switches connected to the traffgens

  And: A flow with updated vlan on src endpoint via partial update

  When: Generate traffic on the flow

  Then: System collects stats for ingress/egress cookies

• System is able to collect stats after partial updating(inner vlan) on a flow endpoint

  Given: Two active neighboring switches connected to the traffgens

  And: A flow with updated inner vlan on src endpoint via partial update

  When: Generate traffic on the flow

  Then: System collects stats for ingress/egress cookies

YFlowStat

Verify that statistic is collected for different type of Y-flow

• System is able to collect #stat meter stats and they grow monotonically

  When: Stats were collected

  Then: #stat stats is available

• System is able to collect subflow #stat-#direction stats and they grow monotonically

  When: Stats were collected

  Then: #stat stats is available

HaFlowStat

Verify that statistic is collected for different type of HA-Flow

• System is able to collect #stat meter stats

  Expect: #stat stats is available

• System is able to collect #stat stats and they grow monotonically

  Expect: #stat stats is available

• System is able to collect latency stats for #description in #direction direction

  Expect: The appropriate statistics data is available

MflStat

Now we have two FL instances: Management and Statistics.

• FL Stats: collect statistics only from the switches.
• FL Management: do the other work and can collect statistics as well when a switch doesn't connect to FL Stats.

• System is able to collect stats from the statistic and management controllers

  Given: A flow

  When: Generate traffic on the given flow

  Then: Stat in TSDB is created

  When: Leave src switch only with management controller and disconnect from stats

  And: Generate traffic on the given flow

  Then: Stat on the src switch should be collected because management controller is set

  When: Leave src switch only with stats controller and disconnect from management

  And: Generate traffic on the given flow

  Then: Stat on the src switch should be collected because statistic controller is set

  When: Disconnect the src switch from both management and statistic controllers

  And: Generate traffic on the given flow

  Then: Stat on the src switch should not be collected because it is disconnected from controllers

  When: Restore default controllers on the src switches

  Then: Old statistic should be collected

• System is able to collect stats from the statistic and management controllers (v2)

  Given: A flow

  When: Generate traffic on the given flow

  Then: Stat in TSDB is created

  When: Src switch is only left with management controller (no stats controller)

  And: Generate traffic on the given flow

  Then: Stat on the src switch should be collected because management controller is still set

  When: Set only statistic controller on the src switch and disconnect from management

  And: Generate traffic on the given flow

  Then: Stat on the src switch should be collected because statistic controller is set

  When: Disconnect the src switch from both management and statistic controllers

  And: Generate traffic on the given flow

  Then: Stat on the src switch should not be collected because it is disconnected from controllers

  When: Restore default controllers on the src switch

  Then: Old statistic should be collected

• System is able to collect stats if at least 1 stats or management controller is available

  Given: A flow, src switch is connected to 2 RW and 2 RO floodlights

  When: Src switch is only left with 1 management controller (no stats controllers)

  And: Generate traffic on the given flow

  Then: Stat on the src switch should be collected (first RW switch available)

  When: Src switch is only left with the other management controller (no stats controllers)

  And: Generate traffic on the given flow

  Then: Stat on the src switch should be collected (second RW switch available)

  When: Set only 1 statistic controller on the src switch and disconnect from management

  And: Generate traffic on the given flow

  Then: Stat on the src switch should be collected (first RO switch available)

  When: Set only other statistic controller on the src switch and disconnect from management

  And: Generate traffic on the given flow

  Then: Stat on the src switch should be collected (second RO switch available)

SimulateStats

In this spec we'll try to simulate certain stats entries by pushing them directly to Kafka and then checking that they are correctly processed and saved to tsdb.

• Flow stats #metric with big values are properly being saved to stats db (noviflow boundaries)

  Expect: Corresponding entries appear in tsdb

Tsdb

Verify that basic stats logging happens.

• Basic stats are being logged for TX/RX metric:#metric

  Expect: At least 1 result in the past 15 minutes

• Basic stats are being logged for verification broadcast rule metric:#metric

  Expect: At least 1 result in the past 15 minutes

• Basic stats are being logged for metric:#metric (10min interval)

  Expect: At least 1 result in the past 15 minutes

• GRPC stats are being logged for metric:#metric, sw: #sw.hwSwString()

  Expect: At least 1 result in the past 15 minutes

---

Toggles

---

FeatureToggles

Feature Toggles is a special lever that allows to turn on/off certain Kilda features. For example, we can disable creation of new flows via Northbound API. This spec verifies that Feature Toggle restrictions are applied correctly.

• System forbids creating new flows when 'create_flow' toggle is set to false

  Given: Existing flow

  When: Set create_flow toggle to false

  And: Try to create a new flow

  Then: Error response is returned, explaining that feature toggle doesn't allow such operation

  And: Update of previously existing flow is still possible

  And: Delete of previously existing flow is still possible

• System forbids updating flows when 'update_flow' toggle is set to false

  Given: Existing flow

  When: Set update_flow toggle to false

  And: Try to update the flow

  Then: Error response is returned, explaining that feature toggle doesn't allow such operation

  And: Creating new flow is still possible

• System forbids deleting flows when 'delete_flow' toggle is set to false

  Given: Existing flow

  When: Set delete_flow toggle to false

  And: Try to delete the flow

  Then: Error response is returned, explaining that feature toggle doesn't allow such operation

  And: Creating new flow is still possible

  And: Updating of flow is still possible

  When: Set delete_flow toggle back to true

  Then: Able to delete flows

FeatureTogglesV2

Feature Toggles is a special lever that allows to turn on/off certain Kilda features. For example, we can disable creation of new flows via Northbound API. This spec verifies that Feature Toggle restrictions are applied correctly.

• System forbids creating new flows when 'create_flow' toggle is set to false

  Given: Existing flow

  When: Set create_flow toggle to false

  And: Try to create a new flow

  Then: Error response is returned, explaining that feature toggle doesn't allow such operation

  And: Update of previously existing flow is still possible

  And: Delete of previously existing flow is still possible

• System forbids updating flows when 'update_flow' toggle is set to false

  Given: Existing flow

  When: Set update_flow toggle to false

  And: Try to update the flow

  Then: Error response is returned, explaining that feature toggle doesn't allow such operation

  And: Creating new flow is still possible

• System forbids deleting flows when 'delete_flow' toggle is set to false

  Given: Existing flow

  When: Set delete_flow toggle to false

  And: Try to delete the flow

  Then: Error response is returned, explaining that feature toggle doesn't allow such operation

  And: Creating new flow is still possible

  And: Updating of flow is still possible

  When: Set delete_flow toggle back to true

  Then: Able to delete flows

• Flow encapsulation type is changed while auto rerouting according to 'flows_reroute_using_default_encap_type' feature toggle

  Given: A switch pair which supports 'transit_vlan' and 'vxlan' encapsulation types

  And: The 'flows_reroute_using_default_encap_type' feature is enabled

  And: A flow with default encapsulation

  When: Update default flow encapsulation type in kilda configuration

  And: Init a flow reroute by breaking current path

  Then: Flow is rerouted

  And: Encapsulation type is changed according to kilda configuration

  When: Update default flow encapsulation type in kilda configuration

  And: Disable the 'flows_reroute_using_default_encap_type' feature toggle

  And: Restore previous path

  And: Init a flow reroute by breaking a new current path

  Then: Flow is rerouted

  And: Encapsulation type is not changed according to kilda configuration

• Flow encapsulation type is not changed while syncing/auto rerouting/updating according to 'flows_reroute_using_default_encap_type' if switch doesn't support new type of encapsulation

  Given: A switch pair which supports 'transit_vlan' and 'vxlan' encapsulation types

  And: The 'vxlan' encapsulation type is disabled in swProps on the src switch

  And: The 'flows_reroute_using_default_encap_type' feature is enabled

  And: A flow with transit_vlan encapsulation

  When: Set vxlan as default flow encapsulation type in kilda configuration if it is not set

  And: Init a flow reroute by breaking current path

  Then: Flow is not rerouted

  And: Encapsulation type is NOT changed according to kilda configuration

  And: Flow is in DOWN state

  When: Update the flow

  Then: Encapsulation type is NOT changed according to kilda configuration

  And: Flow is rerouted and in UP state

  When: Synchronize the flow

  Then: Encapsulation type is NOT changed according to kilda configuration

  And: Flow is still in UP state

  Cleanup:

• System doesn't reroute flow when flows_reroute_on_isl_discovery: false

  Given: A flow with alternative paths

  And: Set flowsRerouteOnIslDiscovery=false

  When: Break the flow path(bring port down on the src switch)

  Then: The flow becomes 'Down'

  When: Restore all possible flow paths

  Then: The flow is still in 'Down' status, because flows_reroute_on_isl_discovery: false

  And: Flow is not rerouted

---

GRPC

---

GrpcCommon

• Able to get switch status on the #sw.hardware-#sw.software switch

  When: Get switch status

  Then: Response is not null and needed fields are returned

• Able to get switch packet in out stats on the #sw.hardware-#sw.software (#sw.description) switch

  When: Get switch packet in out stats

  Then: Response is not null and needed fields are returned

• Not able to get switch status from a non-existent switch address

  When: Get switch status

  Then: Human readable error is returned

License

NoviWare software license file is used to activate the basic and licensed features. If you want to test full functionality then you have to perform the following manual tests: - set license by 'file name'. File name is a file with license on switch. - set license by 'license data'

• Not able to set incorrect license on #sw.hardware-#sw.software

  When: Try to set incorrect license key

  Then: An error is received (400 code)

LogicalPort

This test suite checks the CRUD actions on a logical port. Logical ports are defined by associating a single physical port to them to define Bidirectional Forwarding Detection(BFD) ports or by associating a list of physical ports to them to create Link Aggregation Groups (LAG) or a list of BFD ports to them to create a LAG for fast-failover for BFD sessions.

NOTE: The GRPC implementation supports the LAG type only and it is set by default.

• Able to create/read/delete logicalport on the #sw.hardware-#sw.software switch

  When: Create logical port

  Then: Able to get the created logical port

  And: The created port is exist in a list of all logical port

  When: Try to delete the created logical port

  Then: Logical port is deleted

  When: Try to get the deleted logical port

  Then: Human readable error is returned

• Not able to create logical port with incorrect port number(lPort/sPort): #data.logicalPortNumber/#data.portNumber on the #sw.hardware-#sw.software switch

  When: Try to create logical port: #logicalPortNumber/#portNumber

  Then: Human readable error is returned.

• Not able to delete non-existent logical port number on the #sw.hardware-#sw.software switch

  When: Try to delete incorrect logicalPortNumber

  Then: Human readable error is returned.

Log

This test suite checks that we are able to enable/disable:

• log messages;
• OF log messages. And checks that we are able to do the CRUD actions with the remote log server configuration.

• Able to enable 'log messages' on the #sw.hardware-#sw.software switch

  When: Try to turn on 'log messages'

  Then: The 'log messages' is turned on

  When: Try to turn off 'log messages'

  Then: The 'log messages' is turned off

• Able to enable 'OF log messages' on #sw.hardware-#sw.software

  When: Try to turn on 'OF log messages'

  Then: The 'OF log messages' is turned on

  When: Try to turn off 'OF log messages'

  Then: The 'OF log messages' is turned off

• Able to manipulate(CRUD) with a remote log server on #sw.hardware-#sw.software

  When: Remove current remote log server configuration

  Then: Current remote log server configuration is deleted

  When: Try to set custom remote log server

  Then: New custom remote log server configuration is set

• Not able to set incorrect remote log server configuration(ip/port): #data.remoteIp/#data.remotePort on #sw.hardware-#sw.software

  When: Try to set incorrect configuration

  Then: Human readable error is returned

---

Logging

---

CheckLogging

This specification ensures that all logging facilities are up and running after Kilda deployment

• Check Floodlight logging

  When: Retrieve Floodlight logs for last 5 minutes

  Then: There should be discovery messages

• Check Northbound logging

  When: A non-existent switch is requested

  Then: An error is received (404 code)

  And: Northbound should log these actions within 30 seconds

• Check Storm logging

  When: A non-existent flow is requested

  Then: An error is received (404 code)

  And: Storm should log these actions within 30 seconds

---

Configuration

---

Configuration

Kilda configuration is a special lever that allows to change default flow encapsulation type while creating. This spec assumes that 'transit_vlan' is always default type

• System takes into account default flow encapsulation type while creating a flow

  When: Create a flow without encapsulation type

  Then: Flow is created with current default encapsulation type(transit_vlan)

  When: Update default flow encapsulation type

  Then: Correct response is returned

  And: Kilda configuration is really updated

  When: Create a flow without encapsulation type

  Then: Flow is created with new default encapsulation type(vxlan)

• System doesn't allow to update kilda configuration with wrong flow encapsulation type

  When: Try to set wrong flow encapsulation type

  Then: Human readable error is returned

---

Resilience

---

Chaos

Test system behavior under different factors and events that randomly appear across the topology

• Nothing breaks when multiple flows get rerouted due to randomly failing ISLs

  Given: Create multiple random flows

  When: Random ISLs 'blink' for some time

  Then: All flows remain up and valid

  And: All switches are valid

Contention

This spec is aimed to test different race conditions and system behavior in a concurrent environment (using v2 APIs)

• Parallel flow creation requests with the same name creates only 1 flow

  When: Create the same flow in parallel multiple times

  Then: One flow is created

  And: Other requests have received a decline

• Parallel flow crud requests properly allocate/deallocate bandwidth resources

  When: Create multiple flows on the same ISLs concurrently

  Then: Available bandwidth on related isls is reduced based on bandwidth of created flows

  When: Simultaneously remove all the flows

  Then: Available bandwidth on all related isls is reverted back to normal

• Reroute can be simultaneously performed with sync rules requests, removeExcess=#removeExcess

  Given: A flow with reroute potential

  When: Flow reroute is simultaneously requested together with sync rules requests for all related switches

  Then: Flow is Up and path has changed

  And: Related switches have no rule discrepancies

  And: Flow is healthy

FloodlightKafkaConnection

• System properly handles ISL statuses during connection problems between Floodlights and Kafka

  Given: All switches that have multiple management floodlights now remain with only 1

  And: Pick a region to break, find which isls are between regions

  When: Region 1 controller loses connection to Kafka

  Then: Non-rtl links between failed region and alive regions fail due to discovery timeout

  And: Right before controller alive timeout: switches are still active

  And: links inside regions are discovered

  And: rtl links between regions are discovered

  And: After controller alive timeout switches in broken region become inactive but links are still discovered

  When: System remains in this state for discovery timeout for ISLs

  Then: All links except for non-rtl transit ones are still discovered

  When: Controller restores connection to Kafka

  Then: All links are discovered and switches become active

  And: System is able to successfully create a valid flow between regions

  Cleanup:

• System can detect switch port changes if they happen while Floodlight was disconnected after it reconnects

  When: Controllers lose connection to kafka

  And: Switch port for certain ISL goes down

  And: Controllers restore connection to kafka

  Then: System detects that certain port has been brought down and fails the related link

  Cleanup:

• System can detect switch state changes if they happen while Floodlight was disconnected after it reconnects

  When: Controllers lose connection to kafka

  And: Switch loses connection to mgmt controllers

  And: Controllers restore connection to kafka

  Then: System detects that disconnected switch is no longer active

  When: Reconnect the switch back

  Then: Switch is Active again

  Cleanup:

Retries

• System retries the reroute (global retry) if it fails to install rules on one of the current target path's switches

  Given: Switch pair with at least 3 available paths, one path should have a transit switch that we will break and at least 1 path must remain safe

  And: A flow using given switch pair

  And: Switch on the preferred failover path will suddenly be unavailable for rules installation when the reroute starts

  When: Main path of the flow breaks initiating a reroute

  Then: System fails to install rules on desired path and tries to retry reroute and find new path (global retry)

  When: Switch is marked as offline

  Then: System finds another working path and successfully reroutes the flow (one of the retries succeeds)

  And: All related switches have no rule anomalies

• System tries to retry rule installation during #data.description if previous one is failed

  Given: Two active neighboring switches with two diverse paths at least

  And: All alternative paths unavailable (bring ports down)

  And: A protected flow

  When: Disconnect dst switch on protected path

  And: Mark the transit switch as ACTIVE in db

  And: Init flow #data.description

  Then: System retried to #data.description

  Then: Flow is DOWN

  And: Flow is not rerouted

  And: All involved switches pass switch validation(except dst switch)

  When: Connect dst switch back to the controller

  Then: Flow is UP

  And: Flow is valid and pingable

  Cleanup:

• Flow is successfully deleted from the system even if some rule delete commands fail (no rollback for delete)

  Given: A flow

  When: Send delete request for the flow

  And: One of the related switches does not respond

  Then: Flow history shows failed delete rule retry attempts but flow deletion is successful at the end

• System retries the intentional rerouteV1 if it fails to install rules on a switch

  Given: Two active neighboring switches with two diverse paths at least(main and backup paths)

  And: All alternative paths unavailable (bring ports down)

  And: A flow on the main path

  When: Make backupPath more preferable than mainPath

  And: Disconnect the dst switch

  And: Mark the dst switch as ACTIVE in db

  And: Init intentional flow reroute(APIv1)

  Then: System retries to install/delete rules on the dst switch

  Then: Flow is not rerouted

  And: All involved switches pass switch validation(except dst switch)

  When: Connect dst switch back to the controller

  Then: Flow is UP

  And: Flow is valid and pingable

  Cleanup:

RetriesIsolated

• System does not retry after global timeout for reroute operation

  Given: A flow with ability to reroute

  When: Break current path to trigger a reroute

  And: Connection to src switch is slow in order to simulate a global timeout on reroute operation

  Then: After global timeout expect flow reroute to fail and flow to become DOWN

  And: Flow remains down and no new history events appear for the next 5 seconds (no retry happens)

  And: The last reroute event was caused by delay on src for the alternative path

  And: Src/dst switches are valid

  Cleanup: ----

StormHeavyLoad

• Storm does not fail under heavy load of topo.disco topic

  When: Produce massive amount of messages into topo.disco topic

  Then: Still able to create and delete flows while Storm is swallowing the messages

  And: Topology is unchanged at the end

  Cleanup:

StormLcm

Storm Lifecycle Management: verifies system behavior after restart of WFM. This is required to simulate prod deployments, which are done on the live environment. Before restart the system will have some data (created flows etc.), verify their consistency after restart.

• System survives Storm topologies restart

  Given: Non-empty system with some flows created

  And: All created flows are valid

  And: Database dump

  When: Storm topologies are restarted

  Then: Database nodes and relations are unchanged

  And: Topology is recovered after storm topology restarting

  And: Flows remain valid in terms of installed rules and meters

  And: Flow can be updated

• System's able to fail an ISL if switches on both ends go offline during restart of network topology

  Given: Actual network topology

  When: Kill network topology

  And: Disconnect switches on both ends of ISL

  And: Deploy network topology back

  Then: Switches are recognized as being deactivated

  And: ISL between the switches gets failed after discovery timeout

  Cleanup: