diff --git a/README.md b/README.md index f83f19b..2b8cedf 100644 --- a/README.md +++ b/README.md @@ -1,76 +1,243 @@ # VMAX Cinder Driver -Copyright (c) 2014 EMC Corporation. +Copyright (c) 2016 EMC Corporation. All Rights Reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 -* The driver in the master branch supports Kilo and Juno. -* For the driver that supports Icehouse and Havana, go to the havana_icehouse branch. -# VMAX Driver (FC and iSCSI) +# EMC VMAX FC and iSCSI drivers ## Overview +The EMC VMAX drivers, *EMCVMAXISCSIDriver* and *EMCVMAXFCDriver*, support +the use of EMC VMAX storage arrays with Block Storage. They both provide +equivalent functions and differ only in support for their respective host +attachment methods. -This package consists of two drivers: -* EMCVMAXFCDriver, based on the Cinder FibreChannelDriver -* EMCVMAXISCSIDriver, based on the Cinder ISCSIDriver +The drivers perform volume operations by communicating with the back-end VMAX +storage. It uses a CIM client in Python called *PyWBEM* to perform CIM +operations over HTTP. -These drivers support the use of EMC VMAX storage arrays under OpenStack Cinder block management. They both provide equivalent functions and differ only in support for their respective host attachment methods. +The EMC CIM Object Manager (ECOM) is packaged with the EMC SMI-S provider. It +is a CIM server that enables CIM clients to perform CIM operations over HTTP by +using SMI-S in the back end for VMAX storage operations. -The drivers perform volume operations through use of the EMC SMI-S Provider, which is packaged with Solutions Enabler. The SMI-S Provider implements the SNIA Storage Management Initiative (SMI), an ANSI standard for storage management. +The EMC SMI-S Provider supports the SNIA Storage Management Initiative (SMI), +an ANSI standard for storage management. It supports the VMAX storage system. -EMC Cinder drivers also require PyWBEM, a client library written in Python that communicates with the SMI-S provider over HTTP. +## System requirements +The Cinder driver supports both VMAX-2 and VMAX-3 series. + +* For VMAX-2 series, SMI-S version V4.6.2.29 (Solutions Enabler 7.6.2.67) + or Solutions Enabler 8.1.2 is required. + +* For VMAX-3 [Hybrid], Solutions Enabler 8.1.2 is required. + +* For VMAX-3 [AFA], Solutions Enabler 8.3 is required. This is SSL only. + Refer to section below *SSL support*. + +When installing Solutions Enabler, make sure you explicitly add the SMI-S +component. + +You can download SMI-S from the EMC's support web site (login is required). +See the EMC SMI-S Provider release notes for installation instructions. + +Ensure that there is only one SMI-S (ECOM) server active on the same VMAX +array. + +Support Matrix: + + Microcode | Solutions Enabler/SMIS | VMAX2 | VMAX3 Hybrid | VMAX All Flash | + ------------- | :--------------------: | :-----------------: | :--------------: | :----------------: | + 5977.250.189 | SE7.6.2.64 | Yes | No | No | + 5977.813.785 | SE8.1.2 | Yes | Yes | No | + 5977.944.890 | SE8.3.0.1 | No | Yes | Yes | + + +## Required VMAX software suites for OpenStack +There are five Software Suites available for the VMAX All Flash and Hybrid: + +* Base Suite +* Advanced Suite +* Local Replication Suite +* Remote Replication Suite +* Total Productivity Pack + +Openstack requires the Advanced Suite and the Local Replication Suite +or the Total Productivity Pack (it includes the Advanced Suite and the +Local Replication Suite) for the VMAX All Flash and Hybrid. + +There are four bundled Software Suites for the VMAX2: + +* Advanced Software Suite +* Base Software Suite +* Enginuity Suite +* Symmetrix Management Suite + +OpenStack requires the Advanced Software Bundle for the VMAX2. + +or + +The VMAX2 Optional Software are: + +* EMC Storage Analytics (ESA) +* FAST VP +* Ionix ControlCenter and ProSphere Package +* Open Replicator for Symmetrix +* PowerPath +* RecoverPoint EX +* SRDF for VMAX 10K +* Storage Configuration Advisor +* TimeFinder for VMAX10K + +OpenStack requires TimeFinder for VMAX10K for the VMAX2. + +Each are licensed separately. For further details on how to get the +relevant license(s), reference eLicensing Support below. + +## eLicensing support +To activate your entitlements and obtain your VMAX license files, visit the +Service Center on [EMC support website](https://support.emc.com), as directed on your License +Authorization Code (LAC) letter emailed to you. + +* For help with missing or incorrect entitlements after activation + (that is, expected functionality remains unavailable because it is not + licensed), contact your EMC account representative or authorized reseller. + +* For help with any errors applying license files through Solutions Enabler, + contact the EMC Customer Support Center. + +* If you are missing a LAC letter or require further instructions on + activating your licenses through the Online Support site, contact EMC's + worldwide Licensing team at *licensing@emc.com* or call: + + North America, Latin America, APJK, Australia, New Zealand: SVC4EMC + (800-782-4362) and follow the voice prompts. + + EMEA: +353 (0) 21 4879862 and follow the voice prompts. ## OpenStack Release Support -This driver package supports the Juno and Kilo releases. Compared to previously released versions, enhancements include: -* Support for consistency groups. -* Support for live migration. -* Use lookup service in FC auto zoning. +In this release additional packages and the following features are supported. +Compared to previously released versions [OpenStack & EMC Git], +enhancements include: +* Support for VMAX All Flash. +* Oversubscription +* Consistency Group +* iSCSI multipath ## Supported Operations -The following operations are supported on VMAX arrays: -* Create volume -* Delete volume -* Extend volume -* Attach volume -* Detach volume -* Create snapshot -* Delete snapshot -* Create volume from snapshot -* Create cloned volume -* Copy image to volume -* Copy volume to image -* Create consistency group -* Delete consistency group -* Create Cgsnapshot (snapshot of a consistency group) -* Delete Cgsnapshot - -## Required Software Packages - -### Install SMI-S Provider with Solutions Enabler -* Required version: EMC SMI-S Provider 4.6.2.28 or higher (Solutions Enabler 7.6.2.28 or higher) -* SMI-S Provider is available from available from [EMC’s support website](https://support.emc.com) -* The SMI-S Provider with Solutions Enabler can be installed as a vApp, or on a Windows or Linux host +* Create, list, delete, attach, and detach volumes +* Create, list, and delete volume snapshots +* Copy an image to a volume +* Copy a volume to an image +* Clone a volume +* Extend a volume +* Retype a volume (Host assisted volume migration only) +* Create a volume from a snapshot +* Create and delete consistency group +* Create and delete consistency group snapshot +* Modify consistency group (add/remove volumes) +* Create consistency group from source (source can only be a CG snapshot) + +VMAX drivers also support the following features: + +* Dynamic masking view creation +* Dynamic determination of the target iSCSI IP address +* iSCSI multipath support + +VMAX2: + +* FAST automated storage tiering policy +* Striped volume creation + +VMAX All Flash and Hybrid: + +* Service Level support +* SnapVX support +* All Flash support + + +## Setup VMAX drivers ### Install PyWBEM -* Required version: PyWBEM 0.7 -* Available from [Sourceforge](http://sourceforge.net/projects/pywbem), or using the following commands: - * Install for Ubuntu: - - # apt-get install python-pywbem - * Install on openSUSE: - - # zypper install python-pywbem - - * Install on Fedora: +Ubuntu14.04(LTS),Ubuntu16.04(LTS),Red Hat Enterprise Linux, CentOS and Fedora: + + PyWBEM version | Python2 pip | Python2 Native | Python3 pip | Python3 Native | + ------------- | :-------------: | :-----------------: | :--------------: | :----------------: | + 0.9.0 | No | N/A | Yes | N/A | + 0.8.4 | No | N/A | Yes | N/A | + 0.7.0 | No | Yes | No | Yes | + +Note: +On Python2, use the updated distro version, for example: + + # apt-get install python-pywbem + +Note: + + On Python3, use the official pywbem version (V0.9.0 or v0.8.4). + + +Install the *python-pywbem* package for your distributution. +* Install for Ubuntu: + + # apt-get install python-pywbem + +* Install on openSUSE: + + # zypper install python-pywbem - # yum install pywbem +* Install on Red Hat Enterprise Linuxm Centos, and Fedora: + + # yum install pywbem + +Install iSCSI Utilities (iSCSI driver only). + +Download and configure the Cinder node as an iSCSI initiator +Install the *open-iscsi* package. + +* Install for Ubuntu: + + # apt-get install open-iscsi + +* Install on openSUSE: + + # zypper install ope-iscsi + +* Install on Red Hat Enterprise Linuxm Centos, and Fedora: + + # yum install scsi-target-utils.x86_64 + +Enable the iSCSI driver to start automatically. + +Download Solutions Enabler with SMI-S from [EMC support website](https://support.emc.com) +and install it. Add your VMAX arrays to SMI-S. + +You can install SMI-S on a non-OpenStack host. Supported platforms include +different flavors of Windows, Red Hat, and SUSE Linux. SMI-S can be +installed on a physical server or a VM hosted by an ESX server. Note that +the supported hypervisor for a VM running SMI-S is ESX only. See the EMC +SMI-S Provider release notes for more information on supported platforms and +installation instructions. + +Note: + + You must discover storage arrays on the SMI-S server before you can use + the VMAX drivers. Follow instructions in the SMI-S release notes. + +SMI-S is usually installed at */opt/emc/ECIM/ECOM/bin* on Linux and +*C:\Program Files\EMC\ECIM\ECOM\bin* on Windows. After you install and +configure SMI-S, go to that directory and type *TestSmiProvider.exe* +for windows and *./TestSmiProvider* for linux + +Use *addsys* in *TestSmiProvider* to add an array. Use *dv* and +examine the output after the array is added. Make sure that the arrays are +recognized by the SMI-S server before using the EMC VMAX drivers. ### Verify the EMC VMAX Cinder driver files EMC VMAX Drivers provided in the installer package consists of seven python files: @@ -96,7 +263,6 @@ The EMC VMAX drivers are written to support multiple types of storage, as config enabled_backends=CONF_GROUP_ISCSI, CONF_GROUP_FC [CONF_GROUP_ISCSI] - iscsi_ip_address = 10.10.0.50 volume_driver=cinder.volume.drivers.emc.emc_vmax_iscsi.EMCVMAXISCSIDriver cinder_emc_config_file=/etc/cinder/cinder_emc_config_CONF_GROUP_ISCSI.xml volume_backend_name=ISCSI_backend @@ -106,13 +272,11 @@ The EMC VMAX drivers are written to support multiple types of storage, as config cinder_emc_config_file=/etc/cinder/cinder_emc_config_CONF_GROUP_FC.xml volume_backend_name=FC_backend - -NOTE: iscsi_ip_address is required in an ISCSI configuration. This is the IP Address of the VMAX iscsi target. In this example, two backend configuration groups are enabled: CONF_GROUP_ISCSI and CONF_GROUP_FC. Each configuration group has a section describing unique parameters for connections, drivers, the volume_backend_name, and the name of the EMC-specific configuration file containing additional settings. Note that the file name is in the format /etc/cinder/cinder_emc_config_.xml. See the section below for a description of the file contents. Once the cinder.conf and EMC-specific configuration files have been created, cinder commands need to be issued in order to create and associate OpenStack volume types with the declared volume_backend_names: - + # cinder type-create VMAX_ISCSI # cinder type-key VMAX_ISCSI set volume_backend_name=ISCSI_backend # cinder type-create VMAX_FC @@ -128,6 +292,8 @@ Each enabled backend is configured via parameters contained in an EMC-specific c Here is an example and description of the contents: +VMAX2: + 10.108.246.202 @@ -142,67 +308,527 @@ Here is an example and description of the contents: FC_GOLD1 GOLD1 - -EcomServerIp, EcomServerPort, EcomUserName and EcomPassword identify the ECOM (EMC SMI-S Provider) server to be used, and provide logon credentials. - -PortGroups supplies the names of VMAX port groups that have been pre-configured to expose volumes managed by this Backend. Each supplied port group should have sufficient number and distribution of ports (across directors and switches) as to ensure adequate bandwidth and failure protection for the volume connections. PortGroups can contain one or more port groups of either iSCSI or FC ports. When a dynamic masking view is created by the VMAX driver, the port group is chosen randomly from the list above, to evenly distribute load across the set of groups provided. - -NOTE: Make sure that the PortGroups set contains either all FC or all iSCSI port groups (for a given backend), as appropriate for the configured driver (iSCSI or FC). -The Array tag holds the unique VMAX array serial number. +VMAX3 [All Flash and Hybrid]: -The Pool tag holds the unique pool name within a given array. + + + 1.1.1.1 + 00 + user1 + password1 + + OS-PORTGROUP1-PG + OS-PORTGROUP2-PG + + 111111111111 + SRP_1 + Gold + OLTP + + + +**EcomServerIp** + IP address of the ECOM server which is packaged with SMI-S. + +**EcomServerPort** + Port number of the ECOM server which is packaged with SMI-S. + +**EcomUserName** and **EcomPassword** + Cedentials for the ECOM server. + +**PortGroups** + Supplies the names of VMAX port groups that have been pre-configured to + expose volumes managed by this backend. Each supplied port group should + have sufficient number and distribution of ports (across directors and + switches) as to ensure adequate bandwidth and failure protection for the + volume connections. PortGroups can contain one or more port groups of + either iSCSI or FC ports. When a dynamic masking view is created by the + VMAX driver, the port group is chosen randomly from the PortGroup list, to + evenly distribute load across the set of groups provided. Make sure that + the PortGroups set contains either all FC or all iSCSI port groups (for a + given back end), as appropriate for the configured driver (iSCSI or FC). + +**Array** + Unique VMAX array serial number. + +**Pool** + Unique pool name within a given array. For back ends not using FAST + automated tiering, the pool is a single pool that has been created by the + administrator. For back ends exposing FAST policy automated tiering, the + pool is the bind pool to be used with the FAST policy. + +**FastPolicy** + VMAX2 only. Name of the FAST Policy to be used. By including this tag, + volumes managed by this back end are treated as under FAST control. + Omitting the ``FastPolicy`` tag means FAST is not enabled on the provided + storage pool. + +**SLO** + VMAX All Flash and Hybrid only. The Service Level Objective (SLO) that + manages the underlying storage to provide expected performance. Omitting + the ``SLO`` tag means that non FAST storage groups will be created instead + (storage groups not associated with any service level). + +**Workload** + VMAX All Flash and Hybrid only. When a workload type is added, the latency + range is reduced due to the added information. Omitting the ``Workload`` + tag means the latency range will be the widest for its SLO type. -NOTE: For this version of the driver, we do not support over subscription of pools. Creating a pool with max_subs_percent greater than 100 is not recommended. -For backends not using FAST automated tiering, the pool is a single pool that has been created by the admin. -For backends exposing FAST policy automated tiering, the pool name is the bind pool to be used with the FAST policy. +### FC Zoning with VMAX +Zone Manager is required when there is a fabric between the host and array. +This is necessary for larger configurations where pre-zoning would be too +complex and open-zoning would raise security concerns. -The FastPolicy tag conveys the name of the FAST Policy to be used. By including this tag, volumes managed by this backend are treated as under FAST control. Omitting the FastPolicy tag means FAST is not enabled on the provided storage pool. - -## Configuring Connectivity +### iSCSI with VMAX +* Make sure the *iscsi-initiator-utils* package is installed on all Compute + nodes. -### FC Zoning with VMAX +Note: -With the Icehouse release of OpenStack, a Zone Manager has been added to automate Fibre Channel zone management. Havana does not support this functionality. It is recommended to upgrade to the Juno release if you require FC zoning. + You can only ping the VMAX iSCSI target ports when there is a valid masking + view. An attach operation creates this masking view. -### iSCSI with VMAX -* Make sure the “iscsi-initiator-utils” package is installed on the host (use apt-get, zypper or yum, depending on Linux flavor) -* Verify host is able to ping VMAX iSCSI target ports ## VMAX Masking View and Group Naming Info ### Masking View Names -Masking views for the VMAX FC and iSCSI drivers are now dynamically created by the VMAX Cinder driver using the following naming conventions: +Masking views are dynamically created by the VMAX FC and iSCSI drivers using +the following naming conventions. **[protocol]** is either **I** for volumes +attached over iSCSI or **F** for volumes attached over Fiber Channel. + +VMAX2: + + OS-[shortHostName]-[poolName]-[protocol]-MV + +VMAX2 (where FAST policy is used): - OS--I-MV (for Masking Views using iSCSI) - OS--F-MV (for Masking Views using FC) + OS-[shortHostName]-[fastPolicy]-[protocol]-MV + +VMAX All Flash and Hybrid: + + OS-[shortHostName]-[SRP]-[SLO]-[workload]-[protocol]-MV ### Initiator Group Names -For each host that is attached to VMAX volumes using the Cinder drivers, an initiator group is created or re-used (per attachment type). All initiators of appropriate type known for that host are included in the group. At each new attach volume operation, the VMAX driver retrieves the initiators (either WWNNs or IQNs) from OpenStack and adds or updates the contents of the Initiator Group as required. Names are of the format: +For each host that is attached to VMAX volumes using the drivers, an initiator +group is created or re-used (per attachment type). All initiators of the +appropriate type known for that host are included in the group. At each new +attach volume operation, the VMAX driver retrieves the initiators (either WWNNs +or IQNs) from OpenStack and adds or updates the contents of the Initiator Group +as required. Names are of the following format. **[protocol]** is either **I** +for volumes attached over iSCSI or **F** for volumes attached over Fiber +Channel. + + OS-[shortHostName]-[protocol]-IG - OS--I-IG (for iSCSI initiators) - OS--F-IG (for Fibre Channel initiators) +Note: -Note: Hosts attaching to VMAX storage managed by the OpenStack environment cannot also be attached to storage on the same VMAX not being managed by OpenStack. This is due to limitations on VMAX Initiator Group membership. + Hosts attaching to OpenStack managed VMAX storage cannot also attach to + storage on the same VMAX that are not managed by OpenStack. -### FA Port Groups -VMAX array FA ports to be used in a new masking view are chosen from the list provided in the EMC configuration file. (See EMC-specific Configuration Files above) +### FA Port Groups +VMAX array FA ports to be used in a new masking view are chosen from the list +provided in the EMC configuration file. ### Storage Group Names -As volumes are attached to a host, they are either added to an existing storage group (if it exists) or a new storage group is created and the volume is then added. Storage groups contain volumes created from a pool (either single-pool or FAST-controlled), attached to a single host, over a single connection type (iSCSI or FC). Names are formed: +As volumes are attached to a host, they are either added to an existing storage +group (if it exists) or a new storage group is created and the volume is then +added. Storage groups contain volumes created from a pool (either single-pool +or FAST-controlled), attached to a single host, over a single connection type +(iSCSI or FC). **[protocol]** is either **I** for volumes attached over iSCSI +or **F** for volumes attached over Fiber Channel. + +VMAX2: + + OS-[shortHostName]-[poolName]-[protocol]-SG - OS--I-SG (attached over iSCSI) - OS--F-SG (attached over Fibre Channel) +VMAX2 (where FAST policy is used): -## Concatenated/Striped volumes -In order to support later expansion of created volumes, the VMAX Cinder drivers create concatenated volumes as the default layout. If later expansion is not required, users can opt to create striped volumes in order to optimize I/O performance. + OS-[shortHostName]-[fastPolicy]-[protocol]-SG + +VMAX All Flash and Hybrid: + + OS-[shortHostName]-[SRP]-[SLO]-[Workload]-[protocol]-SG + +## VMAX2 concatenated/Striped volumes +In order to support later expansion of created volumes, the VMAX Block Storage +drivers create concatenated volumes as the default layout. If later expansion +is not required, users can opt to create striped volumes in order to optimize +I/O performance. + +Below is an example of how to create striped volumes. First, create a volume +type. Then define the extra spec for the volume type +*storagetype:stripecount* representing the number of meta members in the +striped volume. The example below means that each volume created under the +*GoldStriped* volume type will be striped and made up of 4 meta members. -Below is an example of how to create striped volumes. First, create a volume type. Then define the extra spec for the volume type -- storagetype:stripecount represents the number of strips you want to make up your volume. The example below means that all volumes created under the GoldStriped volume type will be striped and made up of 4 stripes - # cinder type-create GoldStriped # cinder type-key GoldStriped set volume_backend_name=GOLD_BACKEND # cinder type-key GoldStriped set storagetype:stripecount=4 +## SSL support +Note: + + The ECOM component in Solutions Enabler enforces SSL in 8.3. + By default, this port is 5989. + +1. Get the CA certificate of the ECOM server. This pulls the CA cert file and + saves it as .pem file. *my_ecom_host* is the ip address or hostname of the + ECOM server. *ca_cert.pem* is the sample name of the .pem: + + # openssl s_client -showcerts -connect my_ecom_host:5989 /dev/null|openssl x509 -outform PEM >ca_cert.pem + +2. Copy the pem file to the system certificate directory: + + # cp cp ca_cert.pem /usr/share/ca-certificates/ca_cert.crt + +3. Update CA certificate database with the following commands: + Check that the new *ca_cert.crt* is going to be activiated by selecting + *ask* on the dialog. If it is not enabled for activation, down/up + key to select and space key to enable/disable. + + # dpkg-reconfigure ca-certificates + # sudo update-ca-certificates + +4. Update */etc/cinder/cinder.conf* to reflect SSL functionality by + adding the following to the back end block. *my_location* is the location + of the .pem file generated in step 1.: + + driver_ssl_cert_verify = False + driver_use_ssl = True + + If step 2 and 3 are skipped you must add the location of you .pem file. + + driver_ssl_cert_verify = False + driver_use_ssl = True + driver_ssl_cert_path = /my_location/ca_cert.pem + +5. Update EcomServerIp to ECOM host name and EcomServerPort to secure port + (5989 by default) in */etc/cinder/cinder_emc_config_.xml*. + +## iSCSI multipathing support + +* Install open-iscsi on all nodes on your system +* Do not install EMC PowerPath as they cannot co-exist with native multipath + software +* Multipath tools must be installed on all nova compute nodes + +On Ubuntu: + + # apt-get install open-iscsi #ensure iSCSI is installed + # apt-get install multipath-tools #multipath modules + # apt-get install sysfsutils sg3-utils #file system utilities + # apt-get install scsitools #SCSI tools + +On openSUSE and SUSE Linux Enterprise Server: + + # zipper install open-iscsi #ensure iSCSI is installed + # zipper install multipath-tools #multipath modules + # zipper install sysfsutils sg3-utils #file system utilities + # zipper install scsitools #SCSI tools + +On Red Hat Enterprise Linux and CentOS: + + # yum install iscsi-initiator-utils #ensure iSCSI is installed + # yum install device-mapper-multipath #multipath modules + # yum install sysfsutils sg3-utils #file system utilities + # yum install scsitools #SCSI tools + +### Multipath configuration file + +The multipath configuration file may be edited for better management and +performance. Log in as a privileged user and make the following changes to +*/etc/multipath.conf* on the Compute (nova) node(s). + + + devices { + # Device attributed for EMC VMAX + device { + vendor "EMC" + product "SYMMETRIX" + path_grouping_policy multibus + getuid_callout "/lib/udev/scsi_id --page=pre-spc3-83 --whitelisted --device=/dev/%n" + path_selector "round-robin 0" + path_checker tur + features "0" + hardware_handler "0" + prio const + rr_weight uniform + no_path_retry 6 + rr_min_io 1000 + rr_min_io_rq 1 + } + } + +You may need to reboot the host after installing the MPIO tools or restart +iSCSI and multipath services. + +On Ubuntu: + + # service open-iscsi restart + # service multipath-tools restart + +On On openSUSE, SUSE Linux Enterprise Server, Red Hat Enterprise Linux, and +CentOS: + + # systemctl restart open-iscsi + # systemctl restart multipath-tools + + + $ lsblk + NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT + sda 8:0 0 1G 0 disk + ..360000970000196701868533030303235 (dm-6) 252:6 0 1G 0 mpath + sdb 8:16 0 1G 0 disk + ..360000970000196701868533030303235 (dm-6) 252:6 0 1G 0 mpath + vda 253:0 0 1T 0 disk + +### OpenStack configurations + +On Compute (nova) node, add the following flag in the *[libvirt]* section of +*/etc/nova/nova.conf*: + + iscsi_use_multipath = True + +On cinder controller node, set the multipath flag to true in +*/etc/cinder.conf*: + + use_multipath_for_image_xfer = True + +Restart *nova-compute* and *cinder-volume* services after the change. + +### Verify you have multiple initiators available on the compute node for I/O + +* Create a 3GB VMAX volume. +* Create an instance from image out of native LVM storage or from VMAX + storage, for example, from a bootable volume +* Attach the 3GB volume to the new instance: + + # multipath -ll + mpath102 (360000970000196700531533030383039) dm-3 EMC,SYMMETRIX + size=3G features='1 queue_if_no_path' hwhandler='0' wp=rw + '-+- policy='round-robin 0' prio=1 status=active + 33:0:0:1 sdb 8:16 active ready running + '- 34:0:0:1 sdc 8:32 active ready running + +* Use the *lsblk* command to see the multipath device: + + # lsblk + NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT + sdb 8:0 0 3G 0 disk + ..360000970000196700531533030383039 (dm-6) 252:6 0 3G 0 mpath + sdc 8:16 0 3G 0 disk + ..360000970000196700531533030383039 (dm-6) 252:6 0 3G 0 mpath + vda + +## Consistency group support + +Consistency Groups operations are performed through the CLI using v2 of +the cinder API. + +*/etc/cinder/policy.json* may need to be updated to enable new API calls +for Consistency groups. + +Note: + Even though the terminology is 'Consistency Group' in OpenStack, a Storage + Group is created on the VMAX, and should not be confused with a VMAX + Consistency Group which is an SRDF construct. The Storage Group is not + associated with any FAST policy. + +### Operations + +* Create a Consistency Group: + + cinder --os-volume-api-version 2 consisgroup-create [--name ] + [--description ] [--availability-zone ] + + + # cinder --os-volume-api-version 2 consisgroup-create --name bronzeCG2 volume_type_1 + +* List Consistency Groups: + + cinder consisgroup-list [--all-tenants [<0|1>]] + + # cinder consisgroup-list + +* Show a Consistency Group: + + cinder consisgroup-show + + # cinder consisgroup-show 38a604b7-06eb-4202-8651-dbf2610a0827 + +* Update a consistency Group: + + cinder consisgroup-update [--name ] [--description ] + [--add-volumes ] [--remove-volumes ] + + + Change name: + + # cinder consisgroup-update --name updated_name 38a604b7-06eb-4202-8651-dbf2610a0827 + + Add volume(s) to a Consistency Group: + + # cinder consisgroup-update --add-volumes af1ae89b-564b-4c7f-92d9-c54a2243a5fe 38a604b7-06eb-4202-8651-dbf2610a0827 + + Delete volume(s) from a Consistency Group: + + # cinder consisgroup-update --remove-volumes af1ae89b-564b-4c7f-92d9-c54a2243a5fe 38a604b7-06eb-4202-8651-dbf2610a0827 + +* Create a snapshot of a Consistency Group: + + cinder cgsnapshot-create [--name ] [--description ] + + + # cinder cgsnapshot-create 618d962d-2917-4cca-a3ee-9699373e6625 + +* Delete a snapshot of a Consistency Group: + + cinder cgsnapshot-delete [ ...] + + # cinder cgsnapshot-delete 618d962d-2917-4cca-a3ee-9699373e6625 + +* Delete a Consistency Group: + + cinder consisgroup-delete [--force] [ ...] + + # cinder consisgroup-delete --force 618d962d-2917-4cca-a3ee-9699373e6625 + +* You can also create a volume in a consistency group in one step: + + cinder create [--consisgroup-id ] [--name ] + [--description ] [--volume-type ] + [--availability-zone ] + + # cinder create --volume-type volume_type_1 --name cgBronzeVol --consisgroup-id 1de80c27-3b2f-47a6-91a7-e867cbe36462 1 + +## Workload Planner (WLP) + +VMAX Hybrid allows you to manage application storage by using Service Level +Objectives (SLO) using policy based automation rather than the tiering in the +VMAX2. The VMAX Hybrid comes with up to 6 SLO policies defined. Each has a +set of workload characteristics that determine the drive types and mixes +which will be used for the SLO. All storage in the VMAX Array is virtually +provisioned, and all of the pools are created in containers called Storage +Resource Pools (SRP). Typically there is only one SRP, however there can be +more. Therefore, it is the same pool we will provision to but we can provide +different SLO/Workload combinations. + +The SLO capacity is retrieved by interfacing with Unisphere Workload Planner +(WLP). If you do not set up this relationship then the capacity retrieved is +that of the entire SRP. This can cause issues as it can never be an accurate +representation of what storage is available for any given SLO and Workload +combination. + +### Enabling WLP on Unisphere + +1. To enable WLP on Unisphere, click on the + *array-->Performance-->Settings*. +2. Set both the *Real Time* and the *Root Cause Analysis*. +3. Click *Register*. + +Note: + This should be set up ahead of time (allowing for several hours of data + collection), so that the Unisphere for VMAX Performance Analyzer can + collect rated metrics for each of the supported element types. + +### Using TestSmiProvider to add statistics access point + +After enabling WLP you must then enable SMI-S to gain access to the WLP data: + +1. Connect to the SMI-S Provider using TestSmiProvider. +2. Navigate to the *Active* menu. +3. Type *reg* and enter the noted responses to the questions: + + (EMCProvider:5989) ? reg + Current list of statistics Access Points: ? + Note: The current list will be empty if there are no existing Access Points. + Add Statistics Access Point {y|n} [n]: y + HostID [l2se0060.lss.emc.com]: ? + Note: Enter the Unisphere for VMAX location using a fully qualified Host ID. + Port [8443]: ? + Note: The Port default is the Unisphere for VMAX default secure port. If the secure port + is different for your Unisphere for VMAX setup, adjust this value accordingly. + User [smc]: ? + Note: Enter the Unisphere for VMAX username. + Password [smc]: ? + Note: Enter the Unisphere for VMAX password. + +4. Type *reg* again to view the current list: + + (EMCProvider:5988) ? reg + Current list of statistics Access Points: + HostIDs: + l2se0060.lss.emc.com + PortNumbers: + 8443 + Users: + smc + Add Statistics Access Point {y|n} [n]: n + +## Oversubscription support + +Oversubscription support requires the */etc/cinder/cinder.conf* to be +updated with two additional tags *max_over_subscription_ratio* and +*reserved_percentage*. In the sample below, the value of 2.0 for +*max_over_subscription_ratio* means that the pools in oversubscribed by a +factor of 2, or 200% oversubscribed. The *reserved_percentage* is the high +water mark where by the physical remaining space cannot be exceeded. +For example, if there is only 4% of physical space left and the reserve +percentage is 5, the free space will equate to zero. This is a safety +mechanism to prevent a scenario where a provisioning request fails due to +insufficient raw space. + +The parameter *max_over_subscription_ratio* and *reserved_percentage* are +optional. + +To set these parameter go to the configuration group of the volume type in +**/etc/cinder/cinder.conf**. + [VMAX_ISCSI_SILVER] + cinder_emc_config_file = /etc/cinder/cinder_emc_config_VMAX_ISCSI_SILVER.xml + volume_driver = cinder.volume.drivers.emc.emc_vmax_iscsi.EMCVMAXISCSIDriver + volume_backend_name = VMAX_ISCSI_SILVER + max_over_subscription_ratio = 2.0 + reserved_percentage = 10 + +For the second iteration of over subscription, take into account the +EMCMaxSubscriptionPercent property on the pool. This value is the highest +that a pool can be oversubscribed. + +### Scenario 1 + +*EMCMaxSubscriptionPercent* is 200 and the user defined +*max_over_subscription_ratio* is 2.5, the latter is ignored. +Oversubscription is 200%. + +### Scenario 2 + +*EMCMaxSubscriptionPercent* is 200 and the user defined +*max_over_subscription_ratio* is 1.5, 1.5 equates to 150% and is less than +the value set on the pool. Oversubscription is 150%. + +### Scenario 3 + +*EMCMaxSubscriptionPercent* is 0. This means there is no upper limit on the +pool. The user defined *max_over_subscription_ratio* is 1.5. +Oversubscription is 150%. + +### Scenario 4 + +*EMCMaxSubscriptionPercent* is 0. *max_over_subscription_ratio* is not +set by the user. We recommend to default to upper limit, this is 150%. + +note: + If FAST is set and multiple pools are associated with a FAST policy, + then the same rules apply. The difference is, the TotalManagedSpace and + EMCSubscribedCapacity for each pool associated with the FAST policy are + aggregated. + +### Scenario 5 + +*EMCMaxSubscriptionPercent* is 200 on one pool. It is 300 on another pool. +The user defined *max_over_subscription_ratio* is 2.5. Oversubscription is +200% on the first pool and 250% on the other. diff --git a/emc_vmax_common.py b/emc_vmax_common.py index e28c00b..c230849 100644 --- a/emc_vmax_common.py +++ b/emc_vmax_common.py @@ -1,4 +1,4 @@ -# Copyright (c) 2012 - 2014 EMC Corporation. +# Copyright (c) 2012 - 2015 EMC Corporation. # All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); you may @@ -13,20 +13,24 @@ # License for the specific language governing permissions and limitations # under the License. +import ast import os.path -from oslo.config import cfg +from oslo_config import cfg +from oslo_log import log as logging +from oslo_utils import units import six from cinder import exception from cinder.i18n import _, _LE, _LI, _LW -from cinder.openstack.common import log as logging +from cinder.objects import fields from cinder.volume.drivers.emc import emc_vmax_fast from cinder.volume.drivers.emc import emc_vmax_https from cinder.volume.drivers.emc import emc_vmax_masking from cinder.volume.drivers.emc import emc_vmax_provision from cinder.volume.drivers.emc import emc_vmax_provision_v3 from cinder.volume.drivers.emc import emc_vmax_utils +from cinder.volume import utils as volume_utils LOG = logging.getLogger(__name__) @@ -34,14 +38,7 @@ CONF = cfg.CONF try: - import inspect - import pywbem - argspec = inspect.getargspec(pywbem.WBEMConnection.__init__) - if any("ca_certs" in s for s in argspec.args): - updatedPywbem = True - else: - updatedPywbem = False pywbemAvailable = True except ImportError: pywbemAvailable = False @@ -49,6 +46,9 @@ CINDER_EMC_CONFIG_FILE = '/etc/cinder/cinder_emc_config.xml' CINDER_EMC_CONFIG_FILE_PREFIX = '/etc/cinder/cinder_emc_config_' CINDER_EMC_CONFIG_FILE_POSTFIX = '.xml' +BACKENDNAME = 'volume_backend_name' +PREFIXBACKENDNAME = 'capabilities:volume_backend_name' +PORTGROUPNAME = 'portgroupname' EMC_ROOT = 'root/emc' POOL = 'storagetype:pool' ARRAY = 'storagetype:array' @@ -59,10 +59,19 @@ MEMBERCOUNT = 'storagetype:membercount' STRIPED = 'striped' CONCATENATED = 'concatenated' +SMI_VERSION_8 = 800 # V3 SLO = 'storagetype:slo' WORKLOAD = 'storagetype:workload' +INTERVAL = 'storagetype:interval' +RETRIES = 'storagetype:retries' ISV3 = 'isV3' +TRUNCATE_5 = 5 +TRUNCATE_8 = 8 +SNAPVX = 7 +DISSOLVE_SNAPVX = 9 +CREATE_NEW_TARGET = 2 +SNAPVX_REPLICATION_TYPE = 6 emc_opts = [ cfg.StrOpt('cinder_emc_config_file', @@ -80,6 +89,7 @@ class EMCVMAXCommon(object): It supports VNX and VMAX arrays. """ + VERSION = "2.0.0" stats = {'driver_version': '1.0', 'free_capacity_gb': 0, @@ -89,12 +99,19 @@ class EMCVMAXCommon(object): 'vendor_name': 'EMC', 'volume_backend_name': None} - def __init__(self, prtcl, configuration=None): + pool_info = {'backend_name': None, + 'config_file': None, + 'arrays_info': {}, + 'max_over_subscription_ratio': None, + 'reserved_percentage': None + } + + def __init__(self, prtcl, version, configuration=None): if not pywbemAvailable: LOG.info(_LI( - 'Module PyWBEM not installed. ' - 'Install PyWBEM using the python-pywbem package.')) + "Module PyWBEM not installed. " + "Install PyWBEM using the python-pywbem package.")) self.protocol = prtcl self.configuration = configuration @@ -108,6 +125,33 @@ def __init__(self, prtcl, configuration=None): self.fast = emc_vmax_fast.EMCVMAXFast(prtcl) self.provision = emc_vmax_provision.EMCVMAXProvision(prtcl) self.provisionv3 = emc_vmax_provision_v3.EMCVMAXProvisionV3(prtcl) + self.version = version + self._gather_info() + + def _gather_info(self): + """Gather the relevant information for update_volume_stats.""" + if hasattr(self.configuration, 'cinder_emc_config_file'): + self.pool_info['config_file'] = ( + self.configuration.cinder_emc_config_file) + else: + self.pool_info['config_file'] = ( + self.configuration.safe_get('cinder_emc_config_file')) + + self.pool_info['backend_name'] = ( + self.configuration.safe_get('volume_backend_name')) + self.pool_info['max_over_subscription_ratio'] = ( + self.configuration.safe_get('max_over_subscription_ratio')) + self.pool_info['reserved_percentage'] = ( + self.configuration.safe_get('reserved_percentage')) + LOG.debug( + "Updating volume stats on file %(emcConfigFileName)s on " + "backend %(backendName)s.", + {'emcConfigFileName': self.pool_info['config_file'], + 'backendName': self.pool_info['backend_name']}) + + self.pool_info['arrays_info'] = ( + self.utils.parse_file_to_get_array_map( + self.pool_info['config_file'])) def create_volume(self, volume): """Creates a EMC(VMAX) volume from a pre-existing storage pool. @@ -118,13 +162,12 @@ def create_volume(self, volume): For a striped compositeType: The user must supply an extra spec to determine how many metas - will make up the striped volume.If the meta size is greater than - 240GB an error is returned to the user. Otherwise the + will make up the striped volume. If the meta size is greater + than 240GB an error is returned to the user. Otherwise the EMCNumberOfMembers is what the user specifies. :param volume: volume Object - :returns: volumeInstance, the volume instance - :raises: VolumeBackendAPIException + :returns: dict -- volumeDict - the volume dictionary """ volumeSize = int(self.utils.convert_gb_to_bits(volume['size'])) volumeName = volume['id'] @@ -133,14 +176,14 @@ def create_volume(self, volume): if extraSpecs[ISV3]: rc, volumeDict, storageSystemName = ( - self._create_v3_volume(volume, extraSpecs, - volumeName, volumeSize)) + self._create_v3_volume(volume, volumeName, volumeSize, + extraSpecs)) else: rc, volumeDict, storageSystemName = ( - self._create_composite_volume(volume, extraSpecs, - volumeName, volumeSize)) + self._create_composite_volume(volume, volumeName, volumeSize, + extraSpecs)) - # if volume is created as part of a consistency group + # If volume is created as part of a consistency group. if 'consistencygroup_id' in volume and volume['consistencygroup_id']: cgName = self.utils.truncate_string( volume['consistencygroup_id'], 8) @@ -156,14 +199,17 @@ def create_volume(self, volume): cgInstanceName, volumeInstance.path, cgName, - volumeName) + volumeName, + extraSpecs) LOG.info(_LI("Leaving create_volume: %(volumeName)s " "Return code: %(rc)lu " - "volume dict: %(name)s"), + "volume dict: %(name)s."), {'volumeName': volumeName, 'rc': rc, 'name': volumeDict}) + # Adding version information + volumeDict['version'] = self.version return volumeDict @@ -172,18 +218,20 @@ def create_volume_from_snapshot(self, volume, snapshot): For VMAX, replace snapshot with clone. - :param volume - volume Object - :param snapshot - snapshot object - :returns: cloneVolumeDict - the cloned volume dictionary + :param volume: volume Object + :param snapshot: snapshot object + :returns: dict -- the cloned volume dictionary + :raises: VolumeBackendAPIException """ - LOG.debug("Entering create_volume_from_snapshot") - self._initial_setup(volume) + LOG.debug("Entering create_volume_from_snapshot.") + snapshot['host'] = volume['host'] + extraSpecs = self._initial_setup(snapshot) self.conn = self._get_ecom_connection() snapshotInstance = self._find_lun(snapshot) storageSystem = snapshotInstance['SystemName'] syncName = self.utils.find_sync_sv_by_target( - self.conn, storageSystem, snapshotInstance, True) + self.conn, storageSystem, snapshotInstance, extraSpecs, True) if syncName is not None: repservice = self.utils.find_replication_service(self.conn, storageSystem) @@ -195,21 +243,24 @@ def create_volume_from_snapshot(self, volume, snapshot): data=exception_message) self.provision.delete_clone_relationship( - self.conn, repservice, syncName) + self.conn, repservice, syncName, extraSpecs) - return self._create_cloned_volume(volume, snapshot) + snapshot['host'] = volume['host'] + return self._create_cloned_volume(volume, snapshot, extraSpecs, False) def create_cloned_volume(self, cloneVolume, sourceVolume): """Creates a clone of the specified volume. - :param CloneVolume - clone volume Object - :param sourceVolume - volume object - :returns: cloneVolumeDict - the cloned volume dictionary + :param cloneVolume: clone volume Object + :param sourceVolume: volume object + :returns: cloneVolumeDict -- the cloned volume dictionary """ - return self._create_cloned_volume(cloneVolume, sourceVolume) + extraSpecs = self._initial_setup(sourceVolume) + return self._create_cloned_volume(cloneVolume, sourceVolume, + extraSpecs, False) def delete_volume(self, volume): - """Deletes a EMC(VMAX) volume + """Deletes a EMC(VMAX) volume. :param volume: volume Object """ @@ -218,20 +269,21 @@ def delete_volume(self, volume): rc, volumeName = self._delete_volume(volume) LOG.info(_LI("Leaving delete_volume: %(volumename)s Return code: " - "%(rc)lu"), + "%(rc)lu."), {'volumename': volumeName, 'rc': rc}) def create_snapshot(self, snapshot, volume): """Creates a snapshot. - For VMAX, replace snapshot with clone + For VMAX, replace snapshot with clone. :param snapshot: snapshot object :param volume: volume Object to create snapshot from - :returns: cloneVolumeDict,the cloned volume dictionary + :returns: dict -- the cloned volume dictionary """ - return self._create_cloned_volume(snapshot, volume, True) + extraSpecs = self._initial_setup(volume) + return self._create_cloned_volume(snapshot, volume, extraSpecs, True) def delete_snapshot(self, snapshot, volume): """Deletes a snapshot. @@ -239,37 +291,34 @@ def delete_snapshot(self, snapshot, volume): :param snapshot: snapshot object :param volume: volume Object to create snapshot from """ - LOG.info(_LI("Delete Snapshot: %(snapshotName)s "), + LOG.info(_LI("Delete Snapshot: %(snapshotName)s."), {'snapshotName': snapshot['name']}) + snapshot['host'] = volume['host'] self._delete_snapshot(snapshot) def _remove_members(self, controllerConfigService, - volumeInstance, extraSpecs, connector): + volumeInstance, connector, extraSpecs): """This method unmaps a volume from a host. Removes volume from the Device Masking Group that belongs to a Masking View. - Check if fast policy is in the extra specs, if it isn't we do - not need to do any thing for FAST + Check if fast policy is in the extra specs. If it isn't we do + not need to do any thing for FAST. Assume that isTieringPolicySupported is False unless the FAST - policy is in the extra specs and tiering is enabled on the array + policy is in the extra specs and tiering is enabled on the array. :param controllerConfigService: instance name of - ControllerConfigurationService - :param volume: volume Object - :param extraSpecs: the volume extra specs + ControllerConfigurationService + :param volumeInstance: volume Object :param connector: the connector object + :param extraSpecs: extra specifications + :returns: storageGroupInstanceName """ volumeName = volumeInstance['ElementName'] - LOG.debug("Detaching volume %s", volumeName) - try: - fastPolicyName = extraSpecs[FASTPOLICY] - except KeyError: - fastPolicyName = None - isV3 = extraSpecs[ISV3] + LOG.debug("Detaching volume %s.", volumeName) return self.masking.remove_and_reset_members( self.conn, controllerConfigService, volumeInstance, - fastPolicyName, volumeName, isV3, connector) + volumeName, extraSpecs, connector) def _unmap_lun(self, volume, connector): """Unmaps a volume from the host. @@ -280,14 +329,13 @@ def _unmap_lun(self, volume, connector): """ extraSpecs = self._initial_setup(volume) volumename = volume['name'] - LOG.info(_LI("Unmap volume: %(volume)s"), + LOG.info(_LI("Unmap volume: %(volume)s."), {'volume': volumename}) - device_info = self.find_device_number(volume, connector) - device_number = device_info['hostlunid'] - if device_number is None: + device_info = self.find_device_number(volume, connector['host']) + if 'hostlunid' not in device_info: LOG.info(_LI("Volume %s is not mapped. No volume to unmap."), - (volumename)) + volumename) return vol_instance = self._find_lun(volume) @@ -298,46 +346,64 @@ def _unmap_lun(self, volume, connector): if configservice is None: exception_message = (_("Cannot find Controller Configuration " "Service for storage system " - "%(storage_system)s") + "%(storage_system)s.") % {'storage_system': storage_system}) raise exception.VolumeBackendAPIException(data=exception_message) - self._remove_members(configservice, vol_instance, - extraSpecs, connector) + self._remove_members(configservice, vol_instance, connector, + extraSpecs) + livemigrationrecord = self.utils.get_live_migration_record(volume, + False) + if livemigrationrecord: + live_maskingviewdict = livemigrationrecord[0] + live_connector = livemigrationrecord[1] + live_extraSpecs = livemigrationrecord[2] + self._attach_volume( + volume, live_connector, live_extraSpecs, + live_maskingviewdict, True) + self.utils.delete_live_migration_record(volume) def initialize_connection(self, volume, connector): """Initializes the connection and returns device and connection info. The volume may be already mapped, if this is so the deviceInfo tuple is returned. If the volume is not already mapped then we need to - gather information to either 1. Create an new masking view or 2.Add - the volume to to an existing storage group within an already existing + gather information to either 1. Create an new masking view or 2. Add + the volume to an existing storage group within an already existing maskingview. The naming convention is the following: - initiatorGroupName = OS---IG - e.g OS-myShortHost-I-IG - storageGroupName = OS----SG - e.g OS-myShortHost-SATA_BRONZ1-I-SG - portGroupName = OS--PG The portGroupName will come from - the EMC configuration xml file. - These are precreated. If the portGroup does not exist - then a error will be returned to the user - maskingView = OS----MV - e.g OS-myShortHost-SATA_BRONZ1-I-MV + + .. code-block:: none + + initiatorGroupName = OS---IG + e.g OS-myShortHost-I-IG + storageGroupName = OS----SG + e.g OS-myShortHost-SATA_BRONZ1-I-SG + portGroupName = OS--PG The portGroupName will come from + the EMC configuration xml file. + These are precreated. If the portGroup does not + exist then an error will be returned to the user + maskingView = OS----MV + e.g OS-myShortHost-SATA_BRONZ1-I-MV :param volume: volume Object :param connector: the connector Object - :returns: deviceInfoDict, device information tuple + :returns: dict -- deviceInfoDict - device information dict :raises: VolumeBackendAPIException """ + portGroupName = None extraSpecs = self._initial_setup(volume) + is_multipath = connector.get('multipath', False) volumeName = volume['name'] - LOG.info(_LI("Initialize connection: %(volume)s"), + LOG.info(_LI("Initialize connection: %(volume)s."), {'volume': volumeName}) self.conn = self._get_ecom_connection() - deviceInfoDict = self._wrap_find_device_number(volume, connector) + deviceInfoDict = self._wrap_find_device_number( + volume, connector['host']) + maskingViewDict = self._populate_masking_dict( + volume, connector, extraSpecs) if ('hostlunid' in deviceInfoDict and deviceInfoDict['hostlunid'] is not None): @@ -348,19 +414,33 @@ def initialize_connection(self, volume, connector): deviceNumber = deviceInfoDict['hostlunid'] LOG.info(_LI("Volume %(volume)s is already mapped. " - "The device number is %(deviceNumber)s. "), + "The device number is %(deviceNumber)s."), {'volume': volumeName, 'deviceNumber': deviceNumber}) + # Special case, we still need to get the iscsi ip address. + portGroupName = ( + self._get_correct_port_group( + deviceInfoDict, maskingViewDict['storageSystemName'])) + else: - deviceInfoDict = self._attach_volume( - volume, connector, extraSpecs, True) + deviceInfoDict, portGroupName = self._attach_volume( + volume, connector, extraSpecs, maskingViewDict, True) else: - deviceInfoDict = self._attach_volume(volume, connector, extraSpecs) + deviceInfoDict, portGroupName = ( + self._attach_volume( + volume, connector, extraSpecs, maskingViewDict)) + + if self.protocol.lower() == 'iscsi': + deviceInfoDict['ip_and_iqn'] = ( + self._find_ip_protocol_endpoints( + self.conn, deviceInfoDict['storagesystem'], + portGroupName)) + deviceInfoDict['is_multipath'] = is_multipath return deviceInfoDict - def _attach_volume( - self, volume, connector, extraSpecs, isLiveMigration=None): + def _attach_volume(self, volume, connector, extraSpecs, + maskingViewDict, isLiveMigration=False): """Attach a volume to a host. If live migration is being undertaken then the volume @@ -368,39 +448,44 @@ def _attach_volume( :params volume: the volume object :params connector: the connector object - :params extraSpecs: the volume extra specs + :param extraSpecs: extra specifications + :param maskingViewDict: masking view information :param isLiveMigration: boolean, can be None - :returns: deviceInfoDict + :returns: dict -- deviceInfoDict + String -- port group name + :raises: VolumeBackendAPIException """ volumeName = volume['name'] maskingViewDict = self._populate_masking_dict( volume, connector, extraSpecs) if isLiveMigration: maskingViewDict['isLiveMigration'] = True + self.utils.insert_live_migration_record(volume, maskingViewDict, + connector, extraSpecs) else: maskingViewDict['isLiveMigration'] = False - rollbackDict = self.masking.get_or_create_masking_view_and_map_lun( - self.conn, maskingViewDict) + rollbackDict = self.masking.setup_masking_view( + self.conn, maskingViewDict, extraSpecs) # Find host lun id again after the volume is exported to the host. - deviceInfoDict = self.find_device_number(volume, connector) + deviceInfoDict = self.find_device_number(volume, connector['host']) if 'hostlunid' not in deviceInfoDict: # Did not successfully attach to host, # so a rollback for FAST is required. - LOG.error(_LE("Error Attaching volume %(vol)s. "), + LOG.error(_LE("Error Attaching volume %(vol)s."), {'vol': volumeName}) if ((rollbackDict['fastPolicyName'] is not None) or (rollbackDict['isV3'] is not None)): - (self.masking - ._check_if_rollback_action_for_masking_required( - self.conn, rollbackDict)) - exception_message = ("Error Attaching volume %(vol)s." + (self.masking._check_if_rollback_action_for_masking_required( + self.conn, rollbackDict)) + self.utils.delete_live_migration_record(volume) + exception_message = (_("Error Attaching volume %(vol)s.") % {'vol': volumeName}) raise exception.VolumeBackendAPIException( data=exception_message) - return deviceInfoDict + return deviceInfoDict, rollbackDict['pgGroupName'] def _is_same_host(self, connector, deviceInfoDict): """Check if the host is the same. @@ -410,8 +495,8 @@ def _is_same_host(self, connector, deviceInfoDict): live migration. :params connector: the connector object - :params deviceInfoDict: the device information - :returns: boolean True/False + :params deviceInfoDict: the device information dictionary + :returns: boolean -- True if the host is the same, False otherwise. """ if 'host' in connector: currentHost = connector['host'] @@ -421,28 +506,67 @@ def _is_same_host(self, connector, deviceInfoDict): return True return False - def _wrap_find_device_number(self, volume, connector): - """Aid for unit testing + def _get_correct_port_group(self, deviceInfoDict, storageSystemName): + """Get the portgroup name from the existing masking view. - :params volume: the volume Object - :params connector: the connector Object - :returns: deviceInfoDict + :params deviceInfoDict: the device info dictionary + :params storageSystemName: storage system name + :returns: String port group name + """ + if ('controller' in deviceInfoDict and + deviceInfoDict['controller'] is not None): + maskingViewInstanceName = deviceInfoDict['controller'] + try: + maskingViewInstance = ( + self.conn.GetInstance(maskingViewInstanceName)) + except Exception: + exception_message = (_("Unable to get the name of " + "the masking view.")) + raise exception.VolumeBackendAPIException( + data=exception_message) + + # Get the portgroup from masking view + portGroupInstanceName = ( + self.masking._get_port_group_from_masking_view( + self.conn, + maskingViewInstance['ElementName'], + storageSystemName)) + try: + portGroupInstance = ( + self.conn.GetInstance(portGroupInstanceName)) + portGroupName = ( + portGroupInstance['ElementName']) + except Exception: + exception_message = (_("Unable to get the name of " + "the portgroup.")) + raise exception.VolumeBackendAPIException( + data=exception_message) + else: + exception_message = (_("Cannot get the portgroup from " + "the masking view.")) + raise exception.VolumeBackendAPIException( + data=exception_message) + return portGroupName + + def check_ig_instance_name(self, initiatorGroupInstanceName): + """Check if an initiator group instance is on the array. + + :param initiatorGroupInstanceName: initiator group instance name + :returns: initiator group name, or None if deleted """ - return self.find_device_number(volume, connector) + return self.utils.check_ig_instance_name( + self.conn, initiatorGroupInstanceName) def terminate_connection(self, volume, connector): """Disallow connection from connector. :params volume: the volume Object - :params connectorL the connector Object + :params connector: the connector Object """ - self._initial_setup(volume) - volumename = volume['name'] - LOG.info(_LI("Terminate connection: %(volume)s"), + LOG.info(_LI("Terminate connection: %(volume)s."), {'volume': volumename}) - self.conn = self._get_ecom_connection() self._unmap_lun(volume, connector) def extend_volume(self, volume, newSize): @@ -451,15 +575,16 @@ def extend_volume(self, volume, newSize): Prequisites: 1. The volume must be composite e.g StorageVolume.EMCIsComposite=True 2. The volume can only be concatenated - e.g StorageExtent.IsConcatenated=True + e.g StorageExtent.IsConcatenated=True :params volume: the volume Object :params newSize: the new size to increase the volume to + :returns: dict -- modifiedVolumeDict - the extended volume Object :raises: VolumeBackendAPIException """ originalVolumeSize = volume['size'] volumeName = volume['name'] - self._initial_setup(volume) + extraSpecs = self._initial_setup(volume) self.conn = self._get_ecom_connection() volumeInstance = self._find_lun(volume) if volumeInstance is None: @@ -483,9 +608,14 @@ def extend_volume(self, volume, newSize): additionalVolumeSize = self.utils.convert_gb_to_bits( additionalVolumeSize) - # This is V2 - rc, modifiedVolumeDict = self._extend_composite_volume( - volumeInstance, volumeName, newSize, additionalVolumeSize) + if extraSpecs[ISV3]: + rc, modifiedVolumeDict = self._extend_v3_volume( + volumeInstance, volumeName, newSize, extraSpecs) + else: + # This is V2. + rc, modifiedVolumeDict = self._extend_composite_volume( + volumeInstance, volumeName, newSize, additionalVolumeSize, + extraSpecs) # Check the occupied space of the new extended volume. extendedVolumeInstance = self.utils.find_volume_instance( @@ -494,28 +624,28 @@ def extend_volume(self, volume, newSize): self.conn, extendedVolumeInstance) LOG.debug( "The actual volume size of the extended volume: %(volumeName)s " - "is %(volumeSize)s", + "is %(volumeSize)s.", {'volumeName': volumeName, 'volumeSize': extendedVolumeSize}) # If the requested size and the actual size don't - # tally throw an exception + # tally throw an exception. newSizeBits = self.utils.convert_gb_to_bits(newSize) diffVolumeSize = self.utils.compare_size( newSizeBits, extendedVolumeSize) if diffVolumeSize != 0: exceptionMessage = (_( "The requested size : %(requestedSize)s is not the same as " - "resulting size: %(resultSize)s") + "resulting size: %(resultSize)s.") % {'requestedSize': newSizeBits, 'resultSize': extendedVolumeSize}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) LOG.debug( - "Leaving extend_volume: %(volumeName)s " - "Return code: %(rc)lu " - "volume dict: %(name)s", + "Leaving extend_volume: %(volumeName)s. " + "Return code: %(rc)lu, " + "volume dict: %(name)s.", {'volumeName': volumeName, 'rc': rc, 'name': modifiedVolumeDict}) @@ -524,95 +654,109 @@ def extend_volume(self, volume, newSize): def update_volume_stats(self): """Retrieve stats info. - """ - if hasattr(self.configuration, 'cinder_emc_config_file'): - emcConfigFileName = self.configuration.cinder_emc_config_file - else: - emcConfigFileName = self.configuration.safe_get( - 'cinder_emc_config_file') - - backendName = self.configuration.safe_get('volume_backend_name') - LOG.debug( - "Updating volume stats on file %(emcConfigFileName)s on " - "backend %(backendName)s ", - {'emcConfigFileName': emcConfigFileName, - 'backendName': backendName}) - - if self.conn is None: - self._set_ecom_credentials(emcConfigFileName) - - arrayName = self.utils.parse_array_name_from_file(emcConfigFileName) - if arrayName is None: - LOG.error(_LE( - "Array Serial Number %(arrayName)s must be in the file " - "%(emcConfigFileName)s "), - {'arrayName': arrayName, - 'emcConfigFileName': emcConfigFileName}) - - poolName = self.utils.parse_pool_name_from_file(emcConfigFileName) - if poolName is None: - LOG.error(_LE( - "PoolName %(poolName)s must be in the file " - "%(emcConfigFileName)s "), - {'poolName': poolName, - 'emcConfigFileName': emcConfigFileName}) - - isV3 = self.utils.isArrayV3(self.conn, arrayName) - if isV3: - location_info, total_capacity_gb, free_capacity_gb = ( - self._update_srp_stats(emcConfigFileName, arrayName, poolName)) - else: - # This is V2 - location_info, total_capacity_gb, free_capacity_gb = ( - self._update_pool_stats( - emcConfigFileName, backendName, arrayName, poolName)) + """ + pools = [] + backendName = self.pool_info['backend_name'] + max_oversubscription_ratio = ( + self.pool_info['max_over_subscription_ratio']) + reservedPercentage = self.pool_info['reserved_percentage'] + array_max_over_subscription = None + array_reserve_percent = None + for arrayInfo in self.pool_info['arrays_info']: + self._set_ecom_credentials(arrayInfo) + # Check what type of array it is + isV3 = self.utils.isArrayV3(self.conn, arrayInfo['SerialNumber']) + if isV3: + (location_info, total_capacity_gb, free_capacity_gb, + provisioned_capacity_gb, + array_reserve_percent) = self._update_srp_stats(arrayInfo) + poolName = ("%(slo)s+%(poolName)s+%(array)s" + % {'slo': arrayInfo['SLO'], + 'poolName': arrayInfo['PoolName'], + 'array': arrayInfo['SerialNumber']}) + else: + # This is V2 + (location_info, total_capacity_gb, free_capacity_gb, + provisioned_capacity_gb, array_max_over_subscription) = ( + self._update_pool_stats(backendName, arrayInfo)) + poolName = ("%(poolName)s+%(array)s" + % {'poolName': arrayInfo['PoolName'], + 'array': arrayInfo['SerialNumber']}) + + pool = {'pool_name': poolName, + 'total_capacity_gb': total_capacity_gb, + 'free_capacity_gb': free_capacity_gb, + 'provisioned_capacity_gb': provisioned_capacity_gb, + 'QoS_support': False, + 'location_info': location_info, + 'consistencygroup_support': True, + 'thin_provisioning_support': True, + 'thick_provisioning_support': False, + 'max_over_subscription_ratio': max_oversubscription_ratio + } + if array_max_over_subscription: + pool['max_over_subscription_ratio'] = ( + self.utils.override_ratio( + max_oversubscription_ratio, + array_max_over_subscription)) + + if array_reserve_percent and ( + array_reserve_percent > reservedPercentage): + pool['reserved_percentage'] = array_reserve_percent + else: + pool['reserved_percentage'] = reservedPercentage + pools.append(pool) - data = {'total_capacity_gb': total_capacity_gb, - 'free_capacity_gb': free_capacity_gb, - 'reserved_percentage': 0, - 'QoS_support': False, - 'volume_backend_name': backendName or self.__class__.__name__, - 'vendor_name': "EMC", - 'driver_version': '2.1', + data = {'vendor_name': "EMC", + 'driver_version': self.version, 'storage_protocol': 'unknown', - 'location_info': location_info, - 'consistencygroup_support': True} - - self.stats = data + 'volume_backend_name': self.pool_info['backend_name'] or + self.__class__.__name__, + # Use zero capacities here so we always use a pool. + 'total_capacity_gb': 0, + 'free_capacity_gb': 0, + 'provisioned_capacity_gb': 0, + 'reserved_percentage': 0, + 'pools': pools} - return self.stats + return data - def _update_srp_stats(self, emcConfigFileName, arrayName, poolName): - """Update SRP stats + def _update_srp_stats(self, arrayInfo): + """Update SRP stats. - :param emcConfigFileName: the EMC configuration file - :param arrayName: the array - :param poolName: the pool - :returns: location_info, total_capacity_gb, free_capacity_gb + :param arrayInfo: array information + :returns: location_info + :returns: totalManagedSpaceGbs + :returns: remainingManagedSpaceGbs + :returns: provisionedManagedSpaceGbs + :returns: array_reserve_percent """ - totalManagedSpaceGbs, remainingManagedSpaceGbs = ( - self.utils.get_srp_pool_stats(self.conn, arrayName, poolName)) + (totalManagedSpaceGbs, remainingManagedSpaceGbs, + provisionedManagedSpaceGbs, array_reserve_percent) = ( + self.provisionv3.get_srp_pool_stats(self.conn, arrayInfo)) LOG.info(_LI( "Capacity stats for SRP pool %(poolName)s on array " - "%(arrayName)s (total_capacity_gb=%(total_capacity_gb)lu, " - "free_capacity_gb=%(free_capacity_gb)lu"), - {'poolName': poolName, - 'arrayName': arrayName, + "%(arrayName)s total_capacity_gb=%(total_capacity_gb)lu, " + "free_capacity_gb=%(free_capacity_gb)lu, " + "provisioned_capacity_gb=%(provisioned_capacity_gb)lu"), + {'poolName': arrayInfo['PoolName'], + 'arrayName': arrayInfo['SerialNumber'], 'total_capacity_gb': totalManagedSpaceGbs, - 'free_capacity_gb': remainingManagedSpaceGbs}) - slo = self.utils.parse_slo_from_file(emcConfigFileName) - workload = self.utils.parse_workload_from_file(emcConfigFileName) + 'free_capacity_gb': remainingManagedSpaceGbs, + 'provisioned_capacity_gb': provisionedManagedSpaceGbs}) location_info = ("%(arrayName)s#%(poolName)s#%(slo)s#%(workload)s" - % {'arrayName': arrayName, - 'poolName': poolName, - 'slo': slo, - 'workload': workload}) + % {'arrayName': arrayInfo['SerialNumber'], + 'poolName': arrayInfo['PoolName'], + 'slo': arrayInfo['SLO'], + 'workload': arrayInfo['Workload']}) - return location_info, totalManagedSpaceGbs, remainingManagedSpaceGbs + return (location_info, totalManagedSpaceGbs, + remainingManagedSpaceGbs, provisionedManagedSpaceGbs, + array_reserve_percent) def retype(self, ctxt, volume, new_type, diff, host): """Migrate volume to another host using retype. @@ -620,15 +764,15 @@ def retype(self, ctxt, volume, new_type, diff, host): :param ctxt: context :param volume: the volume object including the volume_type_id :param new_type: the new volume type. + :param diff: Unused parameter. :param host: The host dict holding the relevant target(destination) - information - :returns: boolean True/False - :returns: list + information + :returns: boolean -- True if retype succeeded, False if error """ volumeName = volume['name'] volumeStatus = volume['status'] - LOG.info(_LI("Migrating using retype Volume: %(volume)s"), + LOG.info(_LI("Migrating using retype Volume: %(volume)s."), {'volume': volumeName}) extraSpecs = self._initial_setup(volume) @@ -644,44 +788,47 @@ def retype(self, ctxt, volume, new_type, diff, host): if extraSpecs[ISV3]: return self._slo_workload_migration(volumeInstance, volume, host, volumeName, volumeStatus, - extraSpecs, new_type) + new_type, extraSpecs) else: return self._pool_migration(volumeInstance, volume, host, volumeName, volumeStatus, - extraSpecs[FASTPOLICY], new_type) + extraSpecs[FASTPOLICY], + new_type, extraSpecs) def migrate_volume(self, ctxt, volume, host, new_type=None): - """Migrate volume to another host + """Migrate volume to another host. :param ctxt: context :param volume: the volume object including the volume_type_id :param host: the host dict holding the relevant target(destination) - information + information :param new_type: None - :returns: boolean True/False - :returns: list - """ - LOG.warn(_LW("The VMAX plugin only supports Retype. " - "If a pool based migration is necessary " - "this will happen on a Retype " - "From the command line: " - "cinder --os-volume-api-version 2 retype " - " --migration-policy on-demand")) + :returns: boolean -- Always returns True + :returns: dict -- Empty dict {} + """ + LOG.warning(_LW("The VMAX plugin only supports Retype. " + "If a pool based migration is necessary " + "this will happen on a Retype " + "From the command line: " + "cinder --os-volume-api-version 2 retype " + " --migration-policy on-demand")) return True, {} def _migrate_volume( self, volume, volumeInstance, targetPoolName, - targetFastPolicyName, sourceFastPolicyName, new_type=None): - """Migrate volume to another host + targetFastPolicyName, sourceFastPolicyName, extraSpecs, + new_type=None): + """Migrate volume to another host. :param volume: the volume object including the volume_type_id :param volumeInstance: the volume instance :param targetPoolName: the target poolName :param targetFastPolicyName: the target FAST policy name, can be None :param sourceFastPolicyName: the source FAST policy name, can be None + :param extraSpecs: extra specifications :param new_type: None - :returns: boolean True/False - :returns: empty list + :returns: boolean -- True/False + :returns: list -- empty list """ volumeName = volume['name'] storageSystemName = volumeInstance['SystemName'] @@ -690,15 +837,16 @@ def _migrate_volume( self.conn, volumeInstance.path) moved, rc = self._migrate_volume_from( - volume, volumeInstance, targetPoolName, sourceFastPolicyName) + volume, volumeInstance, targetPoolName, sourceFastPolicyName, + extraSpecs) if moved is False and sourceFastPolicyName is not None: # Return the volume to the default source fast policy storage - # group because the migrate was unsuccessful - LOG.warn(_LW( + # group because the migrate was unsuccessful. + LOG.warning(_LW( "Failed to migrate: %(volumeName)s from " "default source storage group " - "for FAST policy: %(sourceFastPolicyName)s " + "for FAST policy: %(sourceFastPolicyName)s. " "Attempting cleanup... "), {'volumeName': volumeName, 'sourceFastPolicyName': sourceFastPolicyName}) @@ -706,12 +854,13 @@ def _migrate_volume( self.conn, volumeInstance.path): self._migrate_cleanup(self.conn, volumeInstance, storageSystemName, sourceFastPolicyName, - volumeName) + volumeName, extraSpecs) else: - # migrate was successful but still issues + # Migrate was successful but still issues. self._migrate_rollback( self.conn, volumeInstance, storageSystemName, - sourceFastPolicyName, volumeName, sourcePoolInstanceName) + sourceFastPolicyName, volumeName, sourcePoolInstanceName, + extraSpecs) return moved @@ -721,15 +870,16 @@ def _migrate_volume( if moved is True and targetFastPolicyName is not None: if not self._migrate_volume_fast_target( volumeInstance, storageSystemName, - targetFastPolicyName, volumeName): - LOG.warn(_LW( + targetFastPolicyName, volumeName, extraSpecs): + LOG.warning(_LW( "Attempting a rollback of: %(volumeName)s to " - "original pool %(sourcePoolInstanceName)s "), + "original pool %(sourcePoolInstanceName)s."), {'volumeName': volumeName, 'sourcePoolInstanceName': sourcePoolInstanceName}) self._migrate_rollback( self.conn, volumeInstance, storageSystemName, - sourceFastPolicyName, volumeName, sourcePoolInstanceName) + sourceFastPolicyName, volumeName, sourcePoolInstanceName, + extraSpecs) if rc == 0: moved = True @@ -738,24 +888,23 @@ def _migrate_volume( def _migrate_rollback(self, conn, volumeInstance, storageSystemName, sourceFastPolicyName, - volumeName, sourcePoolInstanceName): - """Full rollback + volumeName, sourcePoolInstanceName, extraSpecs): + """Full rollback. Failed on final step on adding migrated volume to new target - default storage group for the target FAST policy + default storage group for the target FAST policy. :param conn: connection info to ECOM :param volumeInstance: the volume instance :param storageSystemName: the storage system name :param sourceFastPolicyName: the source FAST policy name :param volumeName: the volume Name - - :returns: boolean True/False - :returns: int, the return code from migrate operation + :param sourcePoolInstanceName: the instance name of the source pool + :param extraSpecs: extra specifications """ - LOG.warn(_LW("_migrate_rollback on : %(volumeName)s from "), - {'volumeName': volumeName}) + LOG.warning(_LW("_migrate_rollback on : %(volumeName)s."), + {'volumeName': volumeName}) storageRelocationService = self.utils.find_storage_relocation_service( conn, storageSystemName) @@ -763,86 +912,85 @@ def _migrate_rollback(self, conn, volumeInstance, try: self.provision.migrate_volume_to_storage_pool( conn, storageRelocationService, volumeInstance.path, - sourcePoolInstanceName) + sourcePoolInstanceName, extraSpecs) except Exception: LOG.error(_LE( "Failed to return volume %(volumeName)s to " "original storage pool. Please contact your system " - "administrator to return it to the correct location "), + "administrator to return it to the correct location."), {'volumeName': volumeName}) if sourceFastPolicyName is not None: self.add_to_default_SG( conn, volumeInstance, storageSystemName, sourceFastPolicyName, - volumeName) + volumeName, extraSpecs) def _migrate_cleanup(self, conn, volumeInstance, storageSystemName, sourceFastPolicyName, - volumeName): - """If the migrate fails, put volume back to source FAST SG + volumeName, extraSpecs): + """If the migrate fails, put volume back to source FAST SG. :param conn: connection info to ECOM :param volumeInstance: the volume instance :param storageSystemName: the storage system name :param sourceFastPolicyName: the source FAST policy name :param volumeName: the volume Name - - :returns: boolean True/False - :returns: int, the return code from migrate operation + :param extraSpecs: extra specifications + :returns: boolean -- True/False """ - LOG.warn(_LW("_migrate_cleanup on : %(volumeName)s from "), - {'volumeName': volumeName}) - + LOG.warning(_LW("_migrate_cleanup on : %(volumeName)s."), + {'volumeName': volumeName}) + return_to_default = True controllerConfigurationService = ( self.utils.find_controller_configuration_service( conn, storageSystemName)) - # check to see what SG it is in - assocStorageGroupInstanceName = ( - self.utils.get_storage_group_from_volume(conn, - volumeInstance.path)) - # This is the SG it should be in + # Check to see what SG it is in. + assocStorageGroupInstanceNames = ( + self.utils.get_storage_groups_from_volume(conn, + volumeInstance.path)) + # This is the SG it should be in. defaultStorageGroupInstanceName = ( self.fast.get_policy_default_storage_group( conn, controllerConfigurationService, sourceFastPolicyName)) - # It is not in any storage group. Must add it to default source - if assocStorageGroupInstanceName is None: - self.add_to_default_SG(conn, volumeInstance, - storageSystemName, sourceFastPolicyName, - volumeName) - - # It is in the incorrect storage group - if (assocStorageGroupInstanceName is not None and - (assocStorageGroupInstanceName != - defaultStorageGroupInstanceName)): - self.provision.remove_device_from_storage_group( - conn, controllerConfigurationService, - assocStorageGroupInstanceName, volumeInstance.path, volumeName) - + for assocStorageGroupInstanceName in assocStorageGroupInstanceNames: + # It is in the incorrect storage group. + if (assocStorageGroupInstanceName != + defaultStorageGroupInstanceName): + self.provision.remove_device_from_storage_group( + conn, controllerConfigurationService, + assocStorageGroupInstanceName, + volumeInstance.path, volumeName, extraSpecs) + else: + # The volume is already in the default. + return_to_default = False + if return_to_default: self.add_to_default_SG( conn, volumeInstance, storageSystemName, sourceFastPolicyName, - volumeName) + volumeName, extraSpecs) + return return_to_default def _migrate_volume_fast_target( self, volumeInstance, storageSystemName, - targetFastPolicyName, volumeName): + targetFastPolicyName, volumeName, extraSpecs): """If the target host is FAST enabled. If the target host is FAST enabled then we need to add it to the - default storage group for that policy + default storage group for that policy. :param volumeInstance: the volume instance :param storageSystemName: the storage system name :param targetFastPolicyName: the target fast policy name :param volumeName: the volume name - :returns: boolean True/False + :param extraSpecs: extra specifications + :returns: boolean -- True/False """ falseRet = False LOG.info(_LI( "Adding volume: %(volumeName)s to default storage group " - "for FAST policy: %(fastPolicyName)s "), + "for FAST policy: %(fastPolicyName)s."), {'volumeName': volumeName, 'fastPolicyName': targetFastPolicyName}) @@ -853,11 +1001,11 @@ def _migrate_volume_fast_target( defaultStorageGroupInstanceName = ( self.fast.get_or_create_default_storage_group( self.conn, controllerConfigurationService, - targetFastPolicyName, volumeInstance)) + targetFastPolicyName, volumeInstance, extraSpecs)) if defaultStorageGroupInstanceName is None: LOG.error(_LE( "Unable to create or get default storage group for FAST policy" - ": %(fastPolicyName)s. "), + ": %(fastPolicyName)s."), {'fastPolicyName': targetFastPolicyName}) return falseRet @@ -865,42 +1013,44 @@ def _migrate_volume_fast_target( defaultStorageGroupInstanceName = ( self.fast.add_volume_to_default_storage_group_for_fast_policy( self.conn, controllerConfigurationService, volumeInstance, - volumeName, targetFastPolicyName)) + volumeName, targetFastPolicyName, extraSpecs)) if defaultStorageGroupInstanceName is None: LOG.error(_LE( "Failed to verify that volume was added to storage group for " - "FAST policy: %(fastPolicyName)s. "), + "FAST policy: %(fastPolicyName)s."), {'fastPolicyName': targetFastPolicyName}) return falseRet return True def _migrate_volume_from(self, volume, volumeInstance, - targetPoolName, sourceFastPolicyName): - """Check FAST policies and migrate from source pool + targetPoolName, sourceFastPolicyName, + extraSpecs): + """Check FAST policies and migrate from source pool. :param volume: the volume object including the volume_type_id :param volumeInstance: the volume instance :param targetPoolName: the target poolName :param sourceFastPolicyName: the source FAST policy name, can be None - :returns: boolean True/False - :returns: int, the return code from migrate operation + :param extraSpecs: extra specifications + :returns: boolean -- True/False + :returns: int -- the return code from migrate operation """ falseRet = (False, -1) volumeName = volume['name'] storageSystemName = volumeInstance['SystemName'] - LOG.debug("sourceFastPolicyName is : %(sourceFastPolicyName)s. ", + LOG.debug("sourceFastPolicyName is : %(sourceFastPolicyName)s.", {'sourceFastPolicyName': sourceFastPolicyName}) - # If the source volume is is FAST enabled it must first be removed - # from the default storage group for that policy + # If the source volume is FAST enabled it must first be removed + # from the default storage group for that policy. if sourceFastPolicyName is not None: self.remove_from_default_SG( self.conn, volumeInstance, storageSystemName, - sourceFastPolicyName, volumeName) + sourceFastPolicyName, volumeName, extraSpecs) - # migrate from one pool to another + # Migrate from one pool to another. storageRelocationService = self.utils.find_storage_relocation_service( self.conn, storageSystemName) @@ -908,27 +1058,26 @@ def _migrate_volume_from(self, volume, volumeInstance, self.conn, targetPoolName, storageSystemName) if targetPoolInstanceName is None: LOG.error(_LE( - "Error finding targe pool instance name for pool: " - "%(targetPoolName)s. "), + "Error finding target pool instance name for pool: " + "%(targetPoolName)s."), {'targetPoolName': targetPoolName}) return falseRet try: rc = self.provision.migrate_volume_to_storage_pool( self.conn, storageRelocationService, volumeInstance.path, - targetPoolInstanceName) - except Exception as e: - # rollback by deleting the volume if adding the volume to the - # default storage group were to fail - LOG.error(_LE("Exception: %s"), e) - LOG.error(_LE( + targetPoolInstanceName, extraSpecs) + except Exception: + # Rollback by deleting the volume if adding the volume to the + # default storage group were to fail. + LOG.exception(_LE( "Error migrating volume: %(volumename)s. " - "to target pool %(targetPoolName)s. "), + "to target pool %(targetPoolName)s."), {'volumename': volumeName, 'targetPoolName': targetPoolName}) return falseRet - # check that the volume is now migrated to the correct storage pool, - # if it is terminate the migrate session + # Check that the volume is now migrated to the correct storage pool, + # if it is terminate the migrate session. foundPoolInstanceName = self.utils.get_assoc_pool_from_volume( self.conn, volumeInstance.path) @@ -943,10 +1092,10 @@ def _migrate_volume_from(self, volume, volumeInstance, return falseRet else: - LOG.debug("Terminating migration session on : %(volumeName)s. ", + LOG.debug("Terminating migration session on: %(volumeName)s.", {'volumeName': volumeName}) self.provision._terminate_migrate_session( - self.conn, volumeInstance.path) + self.conn, volumeInstance.path, extraSpecs) if rc == 0: moved = True @@ -955,17 +1104,16 @@ def _migrate_volume_from(self, volume, volumeInstance, def remove_from_default_SG( self, conn, volumeInstance, storageSystemName, - sourceFastPolicyName, volumeName): - """For FAST, remove volume from default storage group + sourceFastPolicyName, volumeName, extraSpecs): + """For FAST, remove volume from default storage group. :param conn: connection info to ECOM :param volumeInstance: the volume instance :param storageSystemName: the storage system name :param sourceFastPolicyName: the source FAST policy name :param volumeName: the volume Name - - :returns: boolean True/False - :returns: int, the return code from migrate operation + :param extraSpecs: extra specifications + :raises: VolumeBackendAPIException """ controllerConfigurationService = ( self.utils.find_controller_configuration_service( @@ -974,22 +1122,22 @@ def remove_from_default_SG( defaultStorageGroupInstanceName = ( self.masking.remove_device_from_default_storage_group( conn, controllerConfigurationService, - volumeInstance.path, volumeName, sourceFastPolicyName)) - except Exception as ex: - LOG.error(_LE("Exception: %s"), ex) + volumeInstance.path, volumeName, sourceFastPolicyName, + extraSpecs)) + except Exception: exceptionMessage = (_( "Failed to remove: %(volumename)s. " "from the default storage group for " - "FAST policy %(fastPolicyName)s. ") + "FAST policy %(fastPolicyName)s.") % {'volumename': volumeName, 'fastPolicyName': sourceFastPolicyName}) - LOG.error(exceptionMessage) + LOG.exception(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) if defaultStorageGroupInstanceName is None: - LOG.warn(_LW( - "The volume: %(volumename)s. " + LOG.warning(_LW( + "The volume: %(volumename)s " "was not first part of the default storage " "group for FAST policy %(fastPolicyName)s."), {'volumename': volumeName, @@ -997,17 +1145,15 @@ def remove_from_default_SG( def add_to_default_SG( self, conn, volumeInstance, storageSystemName, - targetFastPolicyName, volumeName): - """For FAST, add volume to default storage group + targetFastPolicyName, volumeName, extraSpecs): + """For FAST, add volume to default storage group. :param conn: connection info to ECOM :param volumeInstance: the volume instance :param storageSystemName: the storage system name :param targetFastPolicyName: the target FAST policy name :param volumeName: the volume Name - - :returns: boolean True/False - :returns: int, the return code from migrate operation + :param extraSpecs: extra specifications """ controllerConfigurationService = ( self.utils.find_controller_configuration_service( @@ -1016,35 +1162,35 @@ def add_to_default_SG( self.fast .add_volume_to_default_storage_group_for_fast_policy( conn, controllerConfigurationService, volumeInstance, - volumeName, targetFastPolicyName)) + volumeName, targetFastPolicyName, extraSpecs)) if assocDefaultStorageGroupName is None: LOG.error(_LE( "Failed to add %(volumeName)s " "to default storage group for fast policy " - "%(fastPolicyName)s "), + "%(fastPolicyName)s."), {'volumeName': volumeName, 'fastPolicyName': targetFastPolicyName}) def _is_valid_for_storage_assisted_migration_v3( self, volumeInstanceName, host, sourceArraySerialNumber, - sourcePoolName, volumeName, volumeStatus): + sourcePoolName, volumeName, volumeStatus, sgName): """Check if volume is suitable for storage assisted (pool) migration. :param volumeInstanceName: the volume instance id :param host: the host object :param sourceArraySerialNumber: the array serial number of - the original volume - :param sourcePoolName: the pool name - the original volume + the original volume + :param sourcePoolName: the pool name of the original volume :param volumeName: the name of the volume to be migrated - :param volumeStatus: the status of the volume e.g - :returns: boolean, True/False - :returns: string, targetSlo - :returns: string, targetWorkload + :param volumeStatus: the status of the volume + :param sgName: storage group name + :returns: boolean -- True/False + :returns: string -- targetSlo + :returns: string -- targetWorkload """ falseRet = (False, None, None) if 'location_info' not in host['capabilities']: - LOG.error(_LE('Error getting array, pool, SLO and workload')) + LOG.error(_LE('Error getting array, pool, SLO and workload.')) return falseRet info = host['capabilities']['location_info'] @@ -1056,14 +1202,14 @@ def _is_valid_for_storage_assisted_migration_v3( targetPoolName = infoDetail[1] targetSlo = infoDetail[2] targetWorkload = infoDetail[3] - except Exception: - LOG.error(_LE("Error parsing array, pool, SLO and workload")) + except KeyError: + LOG.error(_LE("Error parsing array, pool, SLO and workload.")) if targetArraySerialNumber not in sourceArraySerialNumber: LOG.error(_LE( "The source array : %(sourceArraySerialNumber)s does not " - "match the target array: %(targetArraySerialNumber)s" - "skipping storage-assisted migration"), + "match the target array: %(targetArraySerialNumber)s " + "skipping storage-assisted migration."), {'sourceArraySerialNumber': sourceArraySerialNumber, 'targetArraySerialNumber': targetArraySerialNumber}) return falseRet @@ -1071,21 +1217,21 @@ def _is_valid_for_storage_assisted_migration_v3( if targetPoolName not in sourcePoolName: LOG.error(_LE( "Only SLO/workload migration within the same SRP Pool " - "Is supported in this version " + "is supported in this version " "The source pool : %(sourcePoolName)s does not " - "match the target array: %(targetPoolName)s" - "skipping storage-assisted migration"), + "match the target array: %(targetPoolName)s. " + "Skipping storage-assisted migration."), {'sourcePoolName': sourcePoolName, 'targetPoolName': targetPoolName}) return falseRet foundStorageGroupInstanceName = ( self.utils.get_storage_group_from_volume( - self.conn, volumeInstanceName)) + self.conn, volumeInstanceName, sgName)) if foundStorageGroupInstanceName is None: - LOG.warn(_LW( - "Volume : %(volumeName)s is not currently " - "belonging to any storage group "), + LOG.warning(_LW( + "Volume: %(volumeName)s is not currently " + "belonging to any storage group."), {'volumeName': volumeName}) else: @@ -1093,12 +1239,14 @@ def _is_valid_for_storage_assisted_migration_v3( foundStorageGroupInstanceName) emcFastSetting = self.utils._get_fast_settings_from_storage_group( storageGroupInstance) - targetCombination = targetSlo + '+' + targetWorkload + targetCombination = ("%(targetSlo)s+%(targetWorkload)s" + % {'targetSlo': targetSlo, + 'targetWorkload': targetWorkload}) if targetCombination in emcFastSetting: LOG.error(_LE( - "No action required. Volume : %(volumeName)s is " - "already part of slo/workload combination : " - "%(targetCombination)s"), + "No action required. Volume: %(volumeName)s is " + "already part of slo/workload combination: " + "%(targetCombination)s."), {'volumeName': volumeName, 'targetCombination': targetCombination}) return falseRet @@ -1113,16 +1261,16 @@ def _is_valid_for_storage_assisted_migration( :param volumeInstanceName: the volume instance id :param host: the host object :param sourceArraySerialNumber: the array serial number of - the original volume + the original volume :param volumeName: the name of the volume to be migrated :param volumeStatus: the status of the volume e.g - :returns: boolean, True/False - :returns: string, targetPool - :returns: string, targetFastPolicy + :returns: boolean -- True/False + :returns: string -- targetPool + :returns: string -- targetFastPolicy """ falseRet = (False, None, None) if 'location_info' not in host['capabilities']: - LOG.error(_LE('Error getting target pool name and array')) + LOG.error(_LE("Error getting target pool name and array.")) return falseRet info = host['capabilities']['location_info'] @@ -1133,39 +1281,39 @@ def _is_valid_for_storage_assisted_migration( targetArraySerialNumber = infoDetail[0] targetPoolName = infoDetail[1] targetFastPolicy = infoDetail[2] - except Exception: + except KeyError: LOG.error(_LE( - "Error parsing target pool name, array, and fast policy")) + "Error parsing target pool name, array, and fast policy.")) if targetArraySerialNumber not in sourceArraySerialNumber: LOG.error(_LE( "The source array : %(sourceArraySerialNumber)s does not " - "match the target array: %(targetArraySerialNumber)s" - "skipping storage-assisted migration"), + "match the target array: %(targetArraySerialNumber)s, " + "skipping storage-assisted migration."), {'sourceArraySerialNumber': sourceArraySerialNumber, 'targetArraySerialNumber': targetArraySerialNumber}) return falseRet - # get the pool from the source array and check that is is different - # to the pool in the target array + # Get the pool from the source array and check that is different + # to the pool in the target array. assocPoolInstanceName = self.utils.get_assoc_pool_from_volume( self.conn, volumeInstanceName) assocPoolInstance = self.conn.GetInstance( assocPoolInstanceName) if assocPoolInstance['ElementName'] == targetPoolName: LOG.error(_LE( - "No action required. Volume : %(volumeName)s is " - "already part of pool : %(pool)s"), + "No action required. Volume: %(volumeName)s is " + "already part of pool: %(pool)s."), {'volumeName': volumeName, 'pool': targetPoolName}) return falseRet - LOG.info(_LI("Volume status is: %s"), volumeStatus) + LOG.info(_LI("Volume status is: %s."), volumeStatus) if (host['capabilities']['storage_protocol'] != self.protocol and (volumeStatus != 'available' and volumeStatus != 'retyping')): LOG.error(_LE( "Only available volumes can be migrated between " - "different protocols")) + "different protocols.")) return falseRet return (True, targetPoolName, targetFastPolicy) @@ -1178,50 +1326,46 @@ def _set_config_file_and_get_extra_specs(self, volume, volumeTypeId=None): Based on the name of the config group, register the config file :param volume: the volume object including the volume_type_id - :returns: tuple the extra specs tuple - :returns: string configuration file + :param volumeTypeId: Optional override of volume['volume_type_id'] + :returns: dict -- the extra specs dict + :returns: string -- configuration file """ extraSpecs = self.utils.get_volumetype_extraspecs(volume, volumeTypeId) configGroup = None - # If there are no extra specs then the default case is assumed + # If there are no extra specs then the default case is assumed. if extraSpecs: configGroup = self.configuration.config_group - configurationFile = self._register_config_file_from_config_group( configGroup) return extraSpecs, configurationFile def _get_ecom_connection(self): - """Get the ecom connection + """Get the ecom connection. - :returns: conn,the ecom connection + :returns: pywbem.WBEMConnection -- conn, the ecom connection + :raises: VolumeBackendAPIException """ + ecomx509 = None if self.ecomUseSSL: + if (self.configuration.safe_get('driver_client_cert_key') and + self.configuration.safe_get('driver_client_cert')): + ecomx509 = {"key_file": + self.configuration.safe_get( + 'driver_client_cert_key'), + "cert_file": + self.configuration.safe_get( + 'driver_client_cert')} pywbem.cim_http.wbem_request = emc_vmax_https.wbem_request - if updatedPywbem: - conn = pywbem.WBEMConnection( - self.url, - (self.user, self.passwd), - default_namespace='root/emc', - x509={"key_file": - self.configuration.safe_get( - 'driver_client_cert_key'), - "cert_file": - self.configuration.safe_get('driver_client_cert')}, - ca_certs=self.ecomCACert, - no_verification=self.ecomNoVerification) - else: - conn = pywbem.WBEMConnection( - self.url, - (self.user, self.passwd), - default_namespace='root/emc', - x509={"key_file": - self.configuration.safe_get( - 'driver_client_cert_key'), - "cert_file": - self.configuration.safe_get('driver_client_cert')}) + conn = pywbem.WBEMConnection( + self.url, + (self.user, self.passwd), + default_namespace='root/emc', + x509=ecomx509, + ca_certs=self.configuration.safe_get('driver_ssl_cert_path'), + no_verification=not self.configuration.safe_get( + 'driver_ssl_cert_verify')) else: conn = pywbem.WBEMConnection( @@ -1229,9 +1373,8 @@ def _get_ecom_connection(self): (self.user, self.passwd), default_namespace='root/emc') - conn.debug = True if conn is None: - exception_message = (_("Cannot connect to ECOM server")) + exception_message = (_("Cannot connect to ECOM server.")) raise exception.VolumeBackendAPIException(data=exception_message) return conn @@ -1242,7 +1385,9 @@ def _find_pool_in_array(self, arrayStr, poolNameInStr, isV3): :param arrayStr: the array Serial number (String) :param poolNameInStr: the name of the poolname (String) :param isv3: True/False - :returns: foundPoolInstanceName, the CIM Instance Name of the Pool + :returns: foundPoolInstanceName - the CIM Instance Name of the Pool + :returns: string -- systemNameStr + :raises: VolumeBackendAPIException """ foundPoolInstanceName = None systemNameStr = None @@ -1280,7 +1425,6 @@ def _find_pool_in_array(self, arrayStr, poolNameInStr, isV3): def _find_lun(self, volume): """Given the volume get the instance from it. - :param conn: connection the the ecom server :param volume: volume object :returns: foundVolumeinstance """ @@ -1292,14 +1436,26 @@ def _find_lun(self, volume): self.conn = self._get_ecom_connection() if isinstance(loc, six.string_types): - name = eval(loc) + name = ast.literal_eval(loc) + keys = name['keybindings'] + systemName = keys['SystemName'] + + prefix1 = 'SYMMETRIX+' + prefix2 = 'SYMMETRIX-+-' + smiversion = self.utils.get_smi_version(self.conn) + if smiversion > SMI_VERSION_8 and prefix1 in systemName: + keys['SystemName'] = systemName.replace(prefix1, prefix2) + name['keybindings'] = keys instancename = self.utils.get_instance_name( name['classname'], name['keybindings']) - - # Handle the case where volume cannot be found. - foundVolumeinstance = self.utils.get_existing_instance( - self.conn, instancename) + # Allow for an external app to delete the volume. + LOG.debug("Volume instance name: %(in)s", + {'in': instancename}) + try: + foundVolumeinstance = self.conn.GetInstance(instancename) + except Exception: + foundVolumeinstance = None if foundVolumeinstance is None: LOG.debug("Volume %(volumename)s not found on the array.", @@ -1312,14 +1468,16 @@ def _find_lun(self, volume): return foundVolumeinstance - def _find_storage_sync_sv_sv(self, snapshot, volume, + def _find_storage_sync_sv_sv(self, snapshot, volume, extraSpecs, waitforsync=True): - """Find the storage synchronized name + """Find the storage synchronized name. :param snapshot: snapshot object :param volume: volume object - :returns: foundsyncname (String) - :returns: storage_system (String) + :param extraSpecs: extra specifications + :param waitforsync: boolean -- Wait for Solutions Enabler sync. + :returns: string -- foundsyncname + :returns: string -- storage_system """ snapshotname = snapshot['name'] volumename = volume['name'] @@ -1337,17 +1495,18 @@ def _find_storage_sync_sv_sv(self, snapshot, volume, if foundsyncname is None: LOG.debug( "Source: %(volumename)s Target: %(snapshotname)s. " - "Storage Synchronized not found. ", + "Storage Synchronized not found.", {'volumename': volumename, 'snapshotname': snapshotname}) else: - LOG.debug("Storage system: %(storage_system)s " + LOG.debug("Storage system: %(storage_system)s. " "Storage Synchronized instance: %(sync)s.", {'storage_system': storage_system, 'sync': foundsyncname}) - # Wait for SE_StorageSynchronized_SV_SV to be fully synced + # Wait for SE_StorageSynchronized_SV_SV to be fully synced. if waitforsync: - self.utils.wait_for_sync(self.conn, foundsyncname) + self.utils.wait_for_sync(self.conn, foundsyncname, + extraSpecs) return foundsyncname, storage_system @@ -1364,7 +1523,7 @@ def _find_initiator_names(self, connector): name = 'world wide port names' if foundinitiatornames is None or len(foundinitiatornames) == 0: - msg = (_("Error finding %s."), name) + msg = (_("Error finding %s.") % name) LOG.error(msg) raise exception.VolumeBackendAPIException(data=msg) @@ -1373,17 +1532,22 @@ def _find_initiator_names(self, connector): 'initiator': foundinitiatornames}) return foundinitiatornames - def find_device_number(self, volume, connector): + def _wrap_find_device_number(self, volume, host): + return self.find_device_number(volume, host) + + def find_device_number(self, volume, host): """Given the volume dict find a device number. - Find a device number that a host can see - for a volume + Find a device number that a host can see + for a volume. :param volume: the volume dict - :param connector: the connector dict - :returns: data, the data dict - + :param host: host from connector + :returns: dict -- the data dict """ + maskedvols = [] + data = {} + foundController = None foundNumDeviceNumber = None foundMaskingViewName = None volumeName = volume['name'] @@ -1401,9 +1565,9 @@ def find_device_number(self, volume, connector): if index > -1: unitinstance = self.conn.GetInstance(unitname, LocalOnly=False) - numDeviceNumber = int(unitinstance['DeviceNumber'], - 16) + numDeviceNumber = int(unitinstance['DeviceNumber'], 16) foundNumDeviceNumber = numDeviceNumber + foundController = controller controllerInstance = self.conn.GetInstance(controller, LocalOnly=False) propertiesList = controllerInstance.properties.items() @@ -1411,23 +1575,35 @@ def find_device_number(self, volume, connector): if properties[0] == 'ElementName': cimProperties = properties[1] foundMaskingViewName = cimProperties.value - break - break + devicedict = {'hostlunid': foundNumDeviceNumber, + 'storagesystem': storageSystemName, + 'maskingview': foundMaskingViewName, + 'controller': foundController} + maskedvols.append(devicedict) - if foundNumDeviceNumber is None: + if not maskedvols: LOG.debug( "Device number not found for volume " "%(volumeName)s %(volumeInstance)s.", {'volumeName': volumeName, 'volumeInstance': volumeInstance.path}) - - data = {'hostlunid': foundNumDeviceNumber, - 'storagesystem': storageSystemName, - 'maskingview': foundMaskingViewName} + else: + host = self.utils.get_host_short_name(host) + hoststr = ("-%(host)s-" + % {'host': host}) + for maskedvol in maskedvols: + if hoststr.lower() in maskedvol['maskingview'].lower(): + data = maskedvol + if not data: + if len(maskedvols) > 0: + data = maskedvols[0] + LOG.warning(_LW( + "Volume is masked but not to host %(host)s as is " + "expected. Assuming live migration."), + {'host': hoststr}) LOG.debug("Device info: %(data)s.", {'data': data}) - return data def get_target_wwns(self, storageSystem, connector): @@ -1435,7 +1611,8 @@ def get_target_wwns(self, storageSystem, connector): :param storageSystem: the storage system name :param connector: the connector dict - :returns: targetWwns, the target WWN list + :returns: list -- targetWwns, the target WWN list + :raises: VolumeBackendAPIException """ targetWwns = [] @@ -1446,44 +1623,42 @@ def get_target_wwns(self, storageSystem, connector): connector, storageHardwareService) LOG.debug( - "EMCGetTargetEndpoints: Service: %(service)s " + "EMCGetTargetEndpoints: Service: %(service)s, " "Storage HardwareIDs: %(hardwareIds)s.", {'service': storageHardwareService, 'hardwareIds': hardwareIdInstances}) for hardwareIdInstance in hardwareIdInstances: - LOG.debug("HardwareID instance is : %(hardwareIdInstance)s ", + LOG.debug("HardwareID instance is: %(hardwareIdInstance)s.", {'hardwareIdInstance': hardwareIdInstance}) try: - rc, targetEndpoints = ( # @UnusedVariable - self.provision.get_target_endpoints( - self.conn, storageHardwareService, hardwareIdInstance)) - except Exception as ex: - LOG.error(_LE("Exception: %s"), ex) + targetEndpoints = ( + self.utils.get_target_endpoints( + self.conn, hardwareIdInstance)) + except Exception: errorMessage = (_( "Unable to get target endpoints for hardwareId " - "%(hardwareIdInstance)s") + "%(hardwareIdInstance)s.") % {'hardwareIdInstance': hardwareIdInstance}) - LOG.error(errorMessage) + LOG.exception(errorMessage) raise exception.VolumeBackendAPIException(data=errorMessage) if targetEndpoints: - endpoints = targetEndpoints['TargetEndpoints'] - LOG.debug("There are %(len)lu endpoints ", - {'len': len(endpoints)}) - for targetendpoint in endpoints: + LOG.debug("There are %(len)lu endpoints.", + {'len': len(targetEndpoints)}) + for targetendpoint in targetEndpoints: wwn = targetendpoint['Name'] - # Add target wwn to the list if it is not already there + # Add target wwn to the list if it is not already there. if not any(d == wwn for d in targetWwns): targetWwns.append(wwn) else: LOG.error(_LE( - "Target end points do not exist for hardware Id : " - "%(hardwareIdInstance)s "), + "Target end points do not exist for hardware Id: " + "%(hardwareIdInstance)s."), {'hardwareIdInstance': hardwareIdInstance}) - LOG.debug("Target WWNs: : %(targetWwns)s ", + LOG.debug("Target WWNs: %(targetWwns)s.", {'targetWwns': targetWwns}) return targetWwns @@ -1494,8 +1669,8 @@ def _find_storage_hardwareids( :param connector: the connector dict :param hardwareIdManagementService: the storage Hardware - management service - :returns: foundInstances, the list of storage hardware ID instances + management service + :returns: list -- the list of storage hardware ID instances """ foundHardwareIdList = [] wwpns = self._find_initiator_names(connector) @@ -1512,7 +1687,7 @@ def _find_storage_hardwareids( instance = self.utils.get_existing_instance( self.conn, hardwareIdInstance.path) if instance is None: - # hardwareId doesn't exist any more. Skip it. + # HardwareId doesn't exist any more. Skip it. break foundHardwareIdList.append(hardwareIdInstance.path) break @@ -1528,52 +1703,60 @@ def _register_config_file_from_config_group(self, configGroupName): """Given the config group name register the file. :param configGroupName: the config group name - :returns: string configurationFile + :returns: string -- configurationFile - name of the configuration file """ if configGroupName is None: - self._set_ecom_credentials(CINDER_EMC_CONFIG_FILE) return CINDER_EMC_CONFIG_FILE if hasattr(self.configuration, 'cinder_emc_config_file'): configurationFile = self.configuration.cinder_emc_config_file else: configurationFile = ( - CINDER_EMC_CONFIG_FILE_PREFIX + configGroupName + - CINDER_EMC_CONFIG_FILE_POSTFIX) + ("%(prefix)s%(configGroupName)s%(postfix)s" + % {'prefix': CINDER_EMC_CONFIG_FILE_PREFIX, + 'configGroupName': configGroupName, + 'postfix': CINDER_EMC_CONFIG_FILE_POSTFIX})) # The file saved in self.configuration may not be the correct one, - # double check + # double check. if configGroupName not in configurationFile: configurationFile = ( - CINDER_EMC_CONFIG_FILE_PREFIX + configGroupName + - CINDER_EMC_CONFIG_FILE_POSTFIX) + ("%(prefix)s%(configGroupName)s%(postfix)s" + % {'prefix': CINDER_EMC_CONFIG_FILE_PREFIX, + 'configGroupName': configGroupName, + 'postfix': CINDER_EMC_CONFIG_FILE_POSTFIX})) - self._set_ecom_credentials(configurationFile) - return configurationFile - - def _set_ecom_credentials(self, configurationFile): - """Given the configuration file set the ecom credentials. - - :param configurationFile: name of the file (String) - :raises: VolumeBackendAPIException - """ if os.path.isfile(configurationFile): - LOG.debug("Configuration file : %(configurationFile)s exists", + LOG.debug("Configuration file : %(configurationFile)s exists.", {'configurationFile': configurationFile}) else: exceptionMessage = (_( - "Configuration file %(configurationFile)s does not exist ") + "Configuration file %(configurationFile)s does not exist.") % {'configurationFile': configurationFile}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) - ip, port = self.utils.get_ecom_server(configurationFile) - self.user, self.passwd = self.utils.get_ecom_cred(configurationFile) - self.ecomUseSSL, self.ecomCACert, self.ecomNoVerification = ( - self.utils.get_ecom_cred_SSL(configurationFile)) + return configurationFile + + def _set_ecom_credentials(self, arrayInfo): + """Given the array record set the ecom credentials. + + :param arrayInfo: record + :raises: VolumeBackendAPIException + """ + ip = arrayInfo['EcomServerIp'] + port = arrayInfo['EcomServerPort'] + self.user = arrayInfo['EcomUserName'] + self.passwd = arrayInfo['EcomPassword'] + self.ecomUseSSL = self.configuration.safe_get('driver_use_ssl') + ip_port = ("%(ip)s:%(port)s" + % {'ip': ip, + 'port': port}) if self.ecomUseSSL: - self.url = 'https://' + ip + ':' + port + self.url = ("https://%(ip_port)s" + % {'ip_port': ip_port}) else: - self.url = 'http://' + ip + ':' + port + self.url = ("http://%(ip_port)s" + % {'ip_port': ip_port}) self.conn = self._get_ecom_connection() def _initial_setup(self, volume, volumeTypeId=None): @@ -1588,41 +1771,43 @@ def _initial_setup(self, volume, volumeTypeId=None): the composite volume should be concatenated or striped. :param volume: the volume Object - :returns: tuple extra spec tuple - :returns: string the configuration file + :param volumeTypeId: Optional override of volume['volume_type_id'] + :returns: dict -- extra spec dict + :raises: VolumeBackendAPIException """ try: extraSpecs, configurationFile = ( self._set_config_file_and_get_extra_specs( volume, volumeTypeId)) - arrayName = self.utils.parse_array_name_from_file( + pool = self._validate_pool(volume) + LOG.debug("Pool returned is %(pool)s.", + {'pool': pool}) + arrayInfo = self.utils.parse_file_to_get_array_map( configurationFile) - if arrayName is None: + poolRecord = self.utils.extract_record(arrayInfo, pool) + + if not poolRecord: exceptionMessage = (_( - "The array cannot be null. The pool must be configured " - "either as a cinder extra spec for multi-backend or in " - "the EMC configuration file for the default case ")) - LOG.error(exceptionMessage) + "Unable to get corresponding record for pool.")) raise exception.VolumeBackendAPIException( data=exceptionMessage) - isV3 = self.utils.isArrayV3(self.conn, arrayName) + self._set_ecom_credentials(poolRecord) + isV3 = self.utils.isArrayV3( + self.conn, poolRecord['SerialNumber']) if isV3: - extraSpecs = self._set_v3_extra_specs( - extraSpecs, configurationFile, arrayName) + extraSpecs = self._set_v3_extra_specs(extraSpecs, poolRecord) else: # V2 extra specs - extraSpecs = self._set_v2_extra_specs( - extraSpecs, configurationFile, arrayName) + extraSpecs = self._set_v2_extra_specs(extraSpecs, poolRecord) except Exception: + import sys exceptionMessage = (_( - "Unable to get configuration information necessary to create " - "a volume. Please check that there is a configuration file " - "for each config group, if multi-backend is enabled. " - "The should be in the following format " - "/etc/cinder/cinder_emc_config_.xml")) + "Unable to get configuration information necessary to " + "create a volume: %(errorMessage)s.") + % {'errorMessage': sys.exc_info()[1]}) raise exception.VolumeBackendAPIException(data=exceptionMessage) return extraSpecs @@ -1630,9 +1815,10 @@ def _initial_setup(self, volume, volumeTypeId=None): def _get_pool_and_storage_system(self, extraSpecs): """Given the extra specs get the pool and storage system name. - :params extraSpecs: the extra spec tuple + :param extraSpecs: extra specifications :returns: poolInstanceName The pool instance name - :returns: String the storage system name + :returns: string -- the storage system name + :raises: VolumeBackendAPIException """ try: @@ -1641,13 +1827,13 @@ def _get_pool_and_storage_system(self, extraSpecs): array, extraSpecs[POOL], extraSpecs[ISV3]) except Exception: exceptionMessage = (_( - "You must supply an array in your EMC configuration file ")) + "You must supply an array in your EMC configuration file.")) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) if poolInstanceName is None or storageSystemStr is None: exceptionMessage = (_( - "Cannot get necessary pool or storage system information ")) + "Cannot get necessary pool or storage system information.")) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) @@ -1658,12 +1844,12 @@ def _populate_masking_dict(self, volume, connector, extraSpecs): :param volume: the volume object :param connector: the connector object - :param extraSpecs: the extra spec tuple - :returns: tuple maskingViewDict a tuple with masking view information + :param extraSpecs: extra specifications + :returns: dict -- a dictionary with masking view information """ maskingViewDict = {} hostName = connector['host'] - poolName = extraSpecs[POOL] + uniqueName = self.utils.generate_unique_trunc_pool(extraSpecs[POOL]) isV3 = extraSpecs[ISV3] maskingViewDict['isV3'] = isV3 protocol = self.utils.get_short_protocol_type(self.protocol) @@ -1673,24 +1859,31 @@ def _populate_masking_dict(self, volume, connector, extraSpecs): workload = extraSpecs[WORKLOAD] maskingViewDict['slo'] = slo maskingViewDict['workload'] = workload - maskingViewDict['pool'] = poolName - maskingViewDict['sgGroupName'] = ( - 'OS-' + shortHostName + '-' + poolName + '-' + slo + '-' + - workload + '-SG') - maskingViewDict['maskingViewName'] = ( - 'OS-' + shortHostName + '-' + poolName + '-' + slo + '-' + - workload + '-MV') + maskingViewDict['pool'] = uniqueName + prefix = ( + ("OS-%(shortHostName)s-%(poolName)s-%(slo)s-" + "%(workload)s-%(protocol)s" + % {'shortHostName': shortHostName, + 'poolName': uniqueName, + 'slo': slo, + 'workload': workload, + 'protocol': protocol})) else: - maskingViewDict['sgGroupName'] = ( - 'OS-' + shortHostName + '-' + poolName + '-' + - protocol + '-SG') - maskingViewDict['maskingViewName'] = ( - 'OS-' + shortHostName + '-' + poolName + '-' + - protocol + '-MV') - maskingViewDict['fastPolicy'] = ( - self.utils.parse_fast_policy_name_from_file( - self.configuration.cinder_emc_config_file)) - + maskingViewDict['fastPolicy'] = extraSpecs[FASTPOLICY] + if maskingViewDict['fastPolicy']: + uniqueName = self.utils.generate_unique_trunc_fastpolicy( + maskingViewDict['fastPolicy']) + '-FP' + prefix = ( + ("OS-%(shortHostName)s-%(poolName)s-%(protocol)s" + % {'shortHostName': shortHostName, + 'poolName': uniqueName, + 'protocol': protocol})) + + maskingViewDict['sgGroupName'] = ("%(prefix)s-SG" + % {'prefix': prefix}) + + maskingViewDict['maskingViewName'] = ("%(prefix)s-MV" + % {'prefix': prefix}) volumeName = volume['name'] volumeInstance = self._find_lun(volume) storageSystemName = volumeInstance['SystemName'] @@ -1698,13 +1891,13 @@ def _populate_masking_dict(self, volume, connector, extraSpecs): maskingViewDict['controllerConfigService'] = ( self.utils.find_controller_configuration_service( self.conn, storageSystemName)) - # The portGroup is gotten from emc xml config file - maskingViewDict['pgGroupName'] = ( - self.utils.parse_file_to_get_port_group_name( - self.configuration.cinder_emc_config_file)) + # The portGroup is gotten from emc xml config file. + maskingViewDict['pgGroupName'] = extraSpecs[PORTGROUPNAME] maskingViewDict['igGroupName'] = ( - 'OS-' + shortHostName + '-' + protocol + '-IG') + ("OS-%(shortHostName)s-%(protocol)s-IG" + % {'shortHostName': shortHostName, + 'protocol': protocol})) maskingViewDict['connector'] = connector maskingViewDict['volumeInstance'] = volumeInstance maskingViewDict['volumeName'] = volumeName @@ -1714,18 +1907,20 @@ def _populate_masking_dict(self, volume, connector, extraSpecs): def _add_volume_to_default_storage_group_on_create( self, volumeDict, volumeName, storageConfigService, - storageSystemName, fastPolicyName): + storageSystemName, fastPolicyName, extraSpecs): """Add the volume to the default storage group for that policy. On a create when fast policy is enable add the volume to the default - storage group for that policy. If it fails do the necessary rollback + storage group for that policy. If it fails do the necessary rollback. :param volumeDict: the volume dictionary :param volumeName: the volume name (String) :param storageConfigService: the storage configuration service :param storageSystemName: the storage system name (String) :param fastPolicyName: the fast policy name (String) - :returns: tuple maskingViewDict with masking view information + :param extraSpecs: extra specifications + :returns: dict -- maskingViewDict with masking view information + :raises: VolumeBackendAPIException """ try: volumeInstance = self.utils.find_volume_instance( @@ -1733,139 +1928,148 @@ def _add_volume_to_default_storage_group_on_create( controllerConfigurationService = ( self.utils.find_controller_configuration_service( self.conn, storageSystemName)) + defaultSgName = self.fast.format_default_sg_string(fastPolicyName) self.fast.add_volume_to_default_storage_group_for_fast_policy( self.conn, controllerConfigurationService, volumeInstance, - volumeName, fastPolicyName) + volumeName, fastPolicyName, extraSpecs) foundStorageGroupInstanceName = ( self.utils.get_storage_group_from_volume( - self.conn, volumeInstance.path)) + self.conn, volumeInstance.path, defaultSgName)) if foundStorageGroupInstanceName is None: exceptionMessage = (_( - "Error adding Volume: %(volumeName)s. " - "with instance path: %(volumeInstancePath)s. ") + "Error adding Volume: %(volumeName)s " + "with instance path: %(volumeInstancePath)s.") % {'volumeName': volumeName, 'volumeInstancePath': volumeInstance.path}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException( data=exceptionMessage) - except Exception as e: - # rollback by deleting the volume if adding the volume to the - # default storage group were to fail - LOG.error(_LE("Exception: %s"), e) + except Exception: + # Rollback by deleting the volume if adding the volume to the + # default storage group were to fail. errorMessage = (_( - "Rolling back %(volumeName)s by deleting it. ") + "Rolling back %(volumeName)s by deleting it.") % {'volumeName': volumeName}) - LOG.error(errorMessage) + LOG.exception(errorMessage) self.provision.delete_volume_from_pool( self.conn, storageConfigService, volumeInstance.path, - volumeName) + volumeName, extraSpecs) raise exception.VolumeBackendAPIException(data=errorMessage) def _create_and_get_unbound_volume( self, conn, storageConfigService, compositeVolumeInstanceName, - additionalSize): + additionalSize, extraSpecs): """Create an unbound volume. Create an unbound volume so it is in the correct state to add to a - composite volume + composite volume. :param conn: the connection information to the ecom server :param storageConfigService: the storage config service instance name :param compositeVolumeInstanceName: the composite volume instance name :param additionalSize: the size you want to increase the volume by + :param extraSpecs: extra specifications :returns: volume instance modifiedCompositeVolumeInstance """ assocPoolInstanceName = self.utils.get_assoc_pool_from_volume( conn, compositeVolumeInstanceName) appendVolumeInstance = self._create_and_get_volume_instance( conn, storageConfigService, assocPoolInstanceName, 'appendVolume', - additionalSize) + additionalSize, extraSpecs) isVolumeBound = self.utils.is_volume_bound_to_pool( conn, appendVolumeInstance) if 'True' in isVolumeBound: appendVolumeInstance = ( self._unbind_and_get_volume_from_storage_pool( - conn, storageConfigService, assocPoolInstanceName, - appendVolumeInstance.path, 'appendVolume')) + conn, storageConfigService, + appendVolumeInstance.path, 'appendVolume', extraSpecs)) return appendVolumeInstance def _create_and_get_volume_instance( self, conn, storageConfigService, poolInstanceName, - volumeName, volumeSize): + volumeName, volumeSize, extraSpecs): """Create and get a new volume. - :params conn: the connection information to the ecom server - :params storageConfigService: the storage config service instance name - :params poolInstanceName: the pool instance name - :params volumeName: the volume name - :params volumeSize: the size to create the volume - :returns: volumeInstance the volume instance + :param conn: the connection information to the ecom server + :param storageConfigService: the storage config service instance name + :param poolInstanceName: the pool instance name + :param volumeName: the volume name + :param volumeSize: the size to create the volume + :param extraSpecs: extra specifications + :returns: volumeInstance -- the volume instance """ - volumeDict, rc = ( # @UnusedVariable + volumeDict, _rc = ( self.provision.create_volume_from_pool( self.conn, storageConfigService, volumeName, poolInstanceName, - volumeSize)) + volumeSize, extraSpecs)) volumeInstance = self.utils.find_volume_instance( self.conn, volumeDict, volumeName) return volumeInstance def _unbind_and_get_volume_from_storage_pool( - self, conn, storageConfigService, poolInstanceName, - volumeInstanceName, volumeName): + self, conn, storageConfigService, + volumeInstanceName, volumeName, extraSpecs): """Unbind a volume from a pool and return the unbound volume. :param conn: the connection information to the ecom server :param storageConfigService: the storage config service instance name - :param poolInstanceName: the pool instance name :param volumeInstanceName: the volume instance name :param volumeName: string the volumeName - :returns: unboundVolumeInstance the unbound volume instance + :param extraSpecs: extra specifications + :returns: unboundVolumeInstance -- the unbound volume instance """ - - rc, job = ( # @UnusedVariable + _rc, _job = ( self.provision.unbind_volume_from_storage_pool( - conn, storageConfigService, poolInstanceName, - volumeInstanceName, - volumeName)) - volumeDict = self.provision.get_volume_dict_from_job(conn, job['Job']) - volumeInstance = self.utils.find_volume_instance( - self.conn, volumeDict, volumeName) + conn, storageConfigService, volumeInstanceName, + volumeName, extraSpecs)) + # Check that the volume in unbound + volumeInstance = conn.GetInstance(volumeInstanceName) + isVolumeBound = self.utils.is_volume_bound_to_pool( + conn, volumeInstance) + if 'False' not in isVolumeBound: + exceptionMessage = (_( + "Failed to unbind volume %(volume)s") + % {'volume': volumeInstanceName}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException(data=exceptionMessage) + return volumeInstance def _modify_and_get_composite_volume_instance( self, conn, elementCompositionServiceInstanceName, volumeInstance, - appendVolumeInstanceName, volumeName, compositeType): + appendVolumeInstanceName, volumeName, compositeType, extraSpecs): """Given an existing composite volume add a new composite volume to it. :param conn: the connection information to the ecom server :param elementCompositionServiceInstanceName: the storage element - composition service - instance name - :param volumeInstanceName: the volume instance name + composition service instance name + :param volumeInstance: the volume instance :param appendVolumeInstanceName: the appended volume instance name :param volumeName: the volume name :param compositeType: concatenated - :returns: int rc the return code - :returns: modifiedVolumeDict the modified volume Dict + :param extraSpecs: extra specifications + :returns: int -- the return code + :returns: dict -- modifiedVolumeDict - the modified volume dict """ isComposite = self.utils.check_if_volume_is_composite( self.conn, volumeInstance) if 'True' in isComposite: rc, job = self.provision.modify_composite_volume( conn, elementCompositionServiceInstanceName, - volumeInstance.path, appendVolumeInstanceName) + volumeInstance.path, appendVolumeInstanceName, extraSpecs) elif 'False' in isComposite: rc, job = self.provision.create_new_composite_volume( conn, elementCompositionServiceInstanceName, - volumeInstance.path, appendVolumeInstanceName, compositeType) + volumeInstance.path, appendVolumeInstanceName, compositeType, + extraSpecs) else: LOG.error(_LE( "Unable to determine whether %(volumeName)s is " - "composite or not "), + "composite or not."), {'volumeName': volumeName}) raise @@ -1876,7 +2080,7 @@ def _modify_and_get_composite_volume_instance( def _get_or_create_default_storage_group( self, conn, storageSystemName, volumeDict, volumeName, - fastPolicyName): + fastPolicyName, extraSpecs): """Get or create a default storage group for a fast policy. :param conn: the connection information to the ecom server @@ -1884,6 +2088,7 @@ def _get_or_create_default_storage_group( :param volumeDict: the volume dictionary :param volumeName: the volume name :param fastPolicyName: the fast policy name + :param extraSpecs: extra specifications :returns: defaultStorageGroupInstanceName """ controllerConfigService = ( @@ -1895,25 +2100,26 @@ def _get_or_create_default_storage_group( defaultStorageGroupInstanceName = ( self.fast.get_or_create_default_storage_group( self.conn, controllerConfigService, fastPolicyName, - volumeInstance)) + volumeInstance, extraSpecs)) return defaultStorageGroupInstanceName def _create_cloned_volume( - self, cloneVolume, sourceVolume, isSnapshot=False): + self, cloneVolume, sourceVolume, extraSpecs, isSnapshot=False): """Create a clone volume from the source volume. :param cloneVolume: clone volume :param sourceVolume: source of the clone volume - :returns: cloneDict the cloned volume dictionary + :param extraSpecs: extra specs + :param isSnapshot: boolean -- Defaults to False + :returns: dict -- cloneDict the cloned volume dictionary + :raises: VolumeBackendAPIException """ - extraSpecs = self._initial_setup(cloneVolume) - sourceName = sourceVolume['name'] cloneName = cloneVolume['name'] LOG.info(_LI( "Create a replica from Volume: Clone Volume: %(cloneName)s " - " Source Volume: %(sourceName)s"), + "Source Volume: %(sourceName)s."), {'cloneName': cloneName, 'sourceName': sourceName}) @@ -1921,16 +2127,28 @@ def _create_cloned_volume( sourceInstance = self._find_lun(sourceVolume) storageSystem = sourceInstance['SystemName'] + repServCapabilityInstanceName = ( + self.utils.find_replication_service_capabilities(self.conn, + storageSystem)) + is_clone_license = self.utils.is_clone_licensed( + self.conn, repServCapabilityInstanceName) + + if is_clone_license is False: + exceptionMessage = (_( + "Clone feature is not licensed on %(storageSystem)s.") + % {'storageSystem': storageSystem}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException(data=exceptionMessage) repServiceInstanceName = self.utils.find_replication_service( self.conn, storageSystem) - LOG.debug("Create volume replica: Volume: %(cloneName)s " - "Source Volume: %(sourceName)s " - "Method: CreateElementReplica " + LOG.debug("Create volume replica: Volume: %(cloneName)s " + "Source Volume: %(sourceName)s " + "Method: CreateElementReplica " "ReplicationService: %(service)s ElementName: " "%(elementname)s SyncType: 8 SourceElement: " - "%(sourceelement)s", + "%(sourceelement)s.", {'cloneName': cloneName, 'sourceName': sourceName, 'service': repServiceInstanceName, @@ -1942,7 +2160,8 @@ def _create_cloned_volume( cloneVolume, sourceVolume, sourceInstance, - isSnapshot) + isSnapshot, + extraSpecs) else: rc, cloneDict = self._create_clone_v2(repServiceInstanceName, cloneVolume, @@ -1951,33 +2170,38 @@ def _create_cloned_volume( isSnapshot, extraSpecs) LOG.debug("Leaving _create_cloned_volume: Volume: " - "%(cloneName)s Source Volume: %(sourceName)s " + "%(cloneName)s Source Volume: %(sourceName)s " "Return code: %(rc)lu.", {'cloneName': cloneName, 'sourceName': sourceName, 'rc': rc}) + # Adding version information + cloneDict['version'] = self.version return cloneDict def _add_clone_to_default_storage_group( - self, fastPolicyName, storageSystemName, cloneDict, cloneName): + self, fastPolicyName, storageSystemName, cloneDict, cloneName, + extraSpecs): """Helper function to add clone to the default storage group. :param fastPolicyName: the fast policy name :param storageSystemName: the storage system name :param cloneDict: clone dictionary :param cloneName: clone name + :param extraSpecs: extra specifications + :raises: VolumeBackendAPIException """ - # check if the clone/snapshot volume already part of the default sg + # Check if the clone/snapshot volume already part of the default sg. cloneInstance = self.utils.find_volume_instance( self.conn, cloneDict, cloneName) if self.fast.is_volume_in_default_SG(self.conn, cloneInstance.path): return - # if FAST enabled place clone volume or volume from snapshot to - # default storage group + # If FAST enabled place clone volume or volume from snapshot to + # default storage group. LOG.debug("Adding volume: %(cloneName)s to default storage group " - "for FAST policy: %(fastPolicyName)s ", + "for FAST policy: %(fastPolicyName)s.", {'cloneName': cloneName, 'fastPolicyName': fastPolicyName}) @@ -1988,11 +2212,11 @@ def _add_clone_to_default_storage_group( defaultStorageGroupInstanceName = ( self._get_or_create_default_storage_group( self.conn, storageSystemName, cloneDict, cloneName, - fastPolicyName)) + fastPolicyName, extraSpecs)) if defaultStorageGroupInstanceName is None: exceptionMessage = (_( "Unable to create or get default storage group for FAST " - "policy: %(fastPolicyName)s. ") + "policy: %(fastPolicyName)s.") % {'fastPolicyName': fastPolicyName}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException( @@ -2000,13 +2224,13 @@ def _add_clone_to_default_storage_group( self._add_volume_to_default_storage_group_on_create( cloneDict, cloneName, storageConfigService, storageSystemName, - fastPolicyName) + fastPolicyName, extraSpecs) def _delete_volume(self, volume): """Helper function to delete the specified volume. :param volume: volume object to be deleted - :returns: cloneDict the cloned volume dictionary + :returns: tuple -- rc (int return code), volumeName (string vol name) """ volumeName = volume['name'] @@ -2030,45 +2254,43 @@ def _delete_volume(self, volume): deviceId = volumeInstance['DeviceID'] if extraSpecs[ISV3]: - storageGroupName = self.utils.get_v3_storage_group_name( - extraSpecs[POOL], extraSpecs[SLO], extraSpecs[WORKLOAD]) rc = self._delete_from_pool_v3( storageConfigService, volumeInstance, volumeName, - deviceId, storageGroupName) + deviceId, extraSpecs) else: rc = self._delete_from_pool(storageConfigService, volumeInstance, volumeName, deviceId, - extraSpecs[FASTPOLICY]) + extraSpecs[FASTPOLICY], + extraSpecs) return (rc, volumeName) - def _pre_check_for_deletion(self, controllerConfigurationService, - volumeInstanceName, volumeName): - """Check is volume is part of a storage group prior to delete + def _remove_device_from_storage_group( + self, controllerConfigurationService, volumeInstanceName, + volumeName, extraSpecs): + """Check if volume is part of a storage group prior to delete. - Log a warning if volume is part of storage group + Log a warning if volume is part of storage group. :param controllerConfigurationService: controller configuration service :param volumeInstanceName: volume instance name :param volumeName: volume name (string) - :return storageGroupInstanceNames + :param extraSpecs: extra specifications """ storageGroupInstanceNames = ( self.masking.get_associated_masking_groups_from_device( self.conn, volumeInstanceName)) - if storageGroupInstanceNames is not None and \ - len(storageGroupInstanceNames) >= 1: - LOG.warn(_LW( - "Pre check for deletion " - "Volume: %(volumeName)s is part of a storage group " - "Attempting removal from %(storageGroupInstanceNames)s"), + if storageGroupInstanceNames: + LOG.warning(_LW( + "Pre check for deletion. " + "Volume: %(volumeName)s is part of a storage group. " + "Attempting removal from %(storageGroupInstanceNames)s."), {'volumeName': volumeName, 'storageGroupInstanceNames': storageGroupInstanceNames}) for storageGroupInstanceName in storageGroupInstanceNames: - self.provision.remove_device_from_storage_group( + self.masking.remove_device_from_storage_group( self.conn, controllerConfigurationService, - storageGroupInstanceName, - volumeInstanceName, volumeName) - return storageGroupInstanceNames + storageGroupInstanceName, volumeInstanceName, + volumeName, extraSpecs) def _find_lunmasking_scsi_protocol_controller(self, storageSystemName, connector): @@ -2111,8 +2333,11 @@ def _find_lunmasking_scsi_protocol_controller(self, storageSystemName, self.conn, controllerInstanceName) if instance is None: # Skip this controller as it doesn't exist - # any more + # any more. pass + else: + foundControllerInstanceName = ( + controllerInstanceName) break if foundControllerInstanceName is not None: @@ -2139,11 +2364,12 @@ def get_num_volumes_mapped(self, volume, connector): """Returns how many volumes are in the same zone as the connector. Find out how many volumes are mapped to a host - associated to the LunMaskingSCSIProtocolController + associated to the LunMaskingSCSIProtocolController. :param volume: volume object to be deleted :param connector: volume object to be deleted - :returns: int numVolumesMapped + :returns: int -- numVolumesMapped + :raises: VolumeBackendAPIException """ volumename = volume['name'] @@ -2167,7 +2393,7 @@ def get_num_volumes_mapped(self, volume, connector): 'connector': connector, 'ctrl': ctrl}) - # return 0 if masking view does not exist + # Return 0 if masking view does not exist. if ctrl is None: return 0 @@ -2189,12 +2415,12 @@ def _delete_snapshot(self, snapshot): """Helper function to delete the specified snapshot. :param snapshot: snapshot object to be deleted - :returns: None + :raises: VolumeBackendAPIException """ LOG.debug("Entering delete_snapshot.") snapshotname = snapshot['name'] - LOG.info(_LI("Delete Snapshot: %(snapshot)s "), + LOG.info(_LI("Delete Snapshot: %(snapshot)s."), {'snapshot': snapshotname}) extraSpecs = self._initial_setup(snapshot) @@ -2202,78 +2428,82 @@ def _delete_snapshot(self, snapshot): if not extraSpecs[ISV3]: snapshotInstance = self._find_lun(snapshot) + if snapshotInstance is None: + LOG.error(_LE( + "Snapshot %(snapshotname)s not found on the array. " + "No volume to delete."), + {'snapshotname': snapshotname}) + return (-1, snapshotname) storageSystem = snapshotInstance['SystemName'] - # wait for it to fully sync in case there is an ongoing - # create volume from snapshot request + # Wait for it to fully sync in case there is an ongoing + # create volume from snapshot request. syncName = self.utils.find_sync_sv_by_target( - self.conn, storageSystem, snapshotInstance, True) + self.conn, storageSystem, snapshotInstance, extraSpecs, + True) if syncName is None: LOG.info(_LI( - "Snapshot: %(snapshot)s: not found on the array. "), + "Snapshot: %(snapshot)s: not found on the array."), {'snapshot': snapshotname}) else: repservice = self.utils.find_replication_service(self.conn, storageSystem) if repservice is None: - exception_message = (_LE( + exception_message = _( "Cannot find Replication Service to" - " delete snapshot %s."), - snapshotname) + " delete snapshot %s.") % snapshotname raise exception.VolumeBackendAPIException( data=exception_message) - # break the replication relationship + # Break the replication relationship LOG.debug("Deleting snap relationship: Target: %(snapshot)s " - " Method: ModifyReplicaSynchronization " + "Method: ModifyReplicaSynchronization " "Replication Service: %(service)s Operation: 8 " "Synchronization: %(syncName)s.", {'snapshot': snapshotname, - 'service': str(repservice), - 'syncName': str(syncName)}) + 'service': repservice, + 'syncName': syncName}) self.provision.delete_clone_relationship( - self.conn, repservice, syncName, True) + self.conn, repservice, syncName, extraSpecs, True) - # delete the target device + # Delete the target device. self._delete_volume(snapshot) def create_consistencygroup(self, context, group): - """Creates a consistencygroup + """Creates a consistency group. - :param context: + :param context: the context :param group: the group object to be created - :returns: + :returns: dict -- modelUpdate = {'status': 'available'} + :raises: VolumeBackendAPIException """ - LOG.info(_LI("Create Consistency Group: %(group)s"), + LOG.info(_LI("Create Consistency Group: %(group)s."), {'group': group['id']}) - modelUpdate = {'status': 'available'} - - # Temp only - asking Xing why it is postfixed by comma + modelUpdate = {'status': fields.ConsistencyGroupStatus.AVAILABLE} volumeTypeId = group['volume_type_id'].replace(",", "") cgName = self.utils.truncate_string(group['id'], 8) extraSpecs = self._initial_setup(None, volumeTypeId) - poolInstanceName, storageSystem = ( # @UnusedVariable + _poolInstanceName, storageSystem = ( self._get_pool_and_storage_system(extraSpecs)) self.conn = self._get_ecom_connection() - # find storage system + # Find storage system. try: replicationService = self.utils.find_replication_service( self.conn, storageSystem) self.provision.create_consistency_group( - self.conn, replicationService, cgName) - except Exception as ex: - LOG.error(_LE("Exception: %(ex)s"), {'ex': ex}) + self.conn, replicationService, cgName, extraSpecs) + except Exception: exceptionMessage = (_("Failed to create consistency group:" - " %(cgName)s. ") + " %(cgName)s.") % {'cgName': cgName}) - LOG.error(exceptionMessage) + LOG.exception(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) return modelUpdate @@ -2281,24 +2511,25 @@ def create_consistencygroup(self, context, group): def delete_consistencygroup(self, context, group, volumes): """Deletes a consistency group. - :param context: + :param context: the context :param group: the group object to be deleted - :returns: + :param volumes: the list of volumes in the consisgroup to be deleted + :returns: dict -- modelUpdate + :returns: list -- list of volume objects + :raises: VolumeBackendAPIException """ - LOG.info(_LI("Delete Consistency Group: %(group)s"), + LOG.info(_LI("Delete Consistency Group: %(group)s."), {'group': group['id']}) cgName = self.utils.truncate_string(group['id'], 8) modelUpdate = {} modelUpdate['status'] = group['status'] - - # Temp only - asking Xing why it is postfixed by comma volumeTypeId = group['volume_type_id'].replace(",", "") extraSpecs = self._initial_setup(None, volumeTypeId) - poolInstanceName, storageSystem = ( # @UnusedVariable + _poolInstanceName, storageSystem = ( self._get_pool_and_storage_system(extraSpecs)) try: @@ -2311,7 +2542,7 @@ def delete_consistencygroup(self, context, group, volumes): cgInstanceName = self._find_consistency_group( replicationService, cgName) if cgInstanceName is None: - exception_message = (_("Cannot find CG group %s."), + exception_message = (_("Cannot find CG group %s.") % cgName) raise exception.VolumeBackendAPIException( data=exception_message) @@ -2321,69 +2552,91 @@ def delete_consistencygroup(self, context, group, volumes): self.provision.delete_consistency_group(self.conn, replicationService, - cgInstanceName, cgName) + cgInstanceName, cgName, + extraSpecs) - # do a bulk delete, a lot faster than single deletes + # Do a bulk delete, a lot faster than single deletes. if memberInstanceNames: - try: - controllerConfigurationService = ( - self.utils.find_controller_configuration_service( - self.conn, storageSystem)) - for memberInstanceName in memberInstanceNames: - self._pre_check_for_deletion( - controllerConfigurationService, memberInstanceName, - 'Member Volume') - if extraSpecs[ISV3]: - self.provisionv3.delete_volume_from_pool( - self.conn, storageConfigservice, - memberInstanceNames, None) - else: - self.provision.delete_volume_from_pool( - self.conn, storageConfigservice, - memberInstanceNames, None) - for volumeRef in volumes: - volumeRef['status'] = 'deleted' - except Exception: - for volumeRef in volumes: - volumeRef['status'] = 'error_deleting' - modelUpdate['status'] = 'error_deleting' - except Exception as ex: - LOG.error(_LE("Exception: %s"), ex) + volumes, modelUpdate = self._do_bulk_delete( + storageSystem, memberInstanceNames, storageConfigservice, + volumes, modelUpdate, extraSpecs[ISV3], extraSpecs) + + except Exception: exceptionMessage = (_( - "Failed to delete consistency group: %(cgName)s. ") + "Failed to delete consistency group: %(cgName)s.") % {'cgName': cgName}) - LOG.error(exceptionMessage) + LOG.exception(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) return modelUpdate, volumes - def create_cgsnapshot(self, context, cgsnapshot, db): + def _do_bulk_delete(self, storageSystem, memberInstanceNames, + storageConfigservice, volumes, modelUpdate, isV3, + extraSpecs): + """Do a bulk delete. + + :param storageSystem: storage system name + :param memberInstanceNames: volume Instance names + :param storageConfigservice: storage config service + :param volumes: volume objects + :param modelUpdate: dict + :param isV3: boolean + :param extraSpecs: extra specifications + :returns: list -- list of volume objects + :returns: dict -- modelUpdate + """ + try: + controllerConfigurationService = ( + self.utils.find_controller_configuration_service( + self.conn, storageSystem)) + for memberInstanceName in memberInstanceNames: + self._remove_device_from_storage_group( + controllerConfigurationService, memberInstanceName, + 'Member Volume', extraSpecs) + if isV3: + self.provisionv3.delete_volume_from_pool( + self.conn, storageConfigservice, + memberInstanceNames, None, extraSpecs) + else: + self.provision.delete_volume_from_pool( + self.conn, storageConfigservice, + memberInstanceNames, None, extraSpecs) + for volumeRef in volumes: + volumeRef['status'] = 'deleted' + except Exception: + for volumeRef in volumes: + volumeRef['status'] = 'error_deleting' + modelUpdate['status'] = 'error_deleting' + return volumes, modelUpdate + + def create_cgsnapshot(self, context, cgsnapshot, snapshots): """Creates a cgsnapshot. - :param context: + :param context: the context :param cgsnapshot: the consistency group snapshot to be created - :param db: cinder database - :returns: modelUpdate, list of snapshots + :param snapshots: snapshots + :returns: dict -- modelUpdate + :returns: list -- list of snapshots + :raises: VolumeBackendAPIException """ - consistencyGroup = db.consistencygroup_get( - context, cgsnapshot['consistencygroup_id']) + consistencyGroup = cgsnapshot.get('consistencygroup') + + snapshots_model_update = [] LOG.info(_LI( "Create snapshot for Consistency Group %(cgId)s " - "cgsnapshotID: %(cgsnapshot)s "), + "cgsnapshotID: %(cgsnapshot)s."), {'cgsnapshot': cgsnapshot['id'], 'cgId': cgsnapshot['consistencygroup_id']}) cgName = self.utils.truncate_string( cgsnapshot['consistencygroup_id'], 8) - modelUpdate = {'status': 'available'} - volumeTypeId = consistencyGroup['volume_type_id'].replace(",", "") extraSpecs = self._initial_setup(None, volumeTypeId) self.conn = self._get_ecom_connection() - poolInstanceName, storageSystem = ( # @UnusedVariable + _poolInstanceName, storageSystem = ( self._get_pool_and_storage_system(extraSpecs)) try: @@ -2393,20 +2646,20 @@ def create_cgsnapshot(self, context, cgsnapshot, db): cgInstanceName = ( self._find_consistency_group(replicationService, cgName)) if cgInstanceName is None: - exception_message = (_("Cannot find CG group %s."), cgName) + exception_message = (_("Cannot find CG group %s.") % cgName) raise exception.VolumeBackendAPIException( data=exception_message) memberInstanceNames = self._get_members_of_replication_group( cgInstanceName) - # create the target consistency group + # Create the target consistency group. targetCgName = self.utils.truncate_string(cgsnapshot['id'], 8) self.provision.create_consistency_group( - self.conn, replicationService, targetCgName) + self.conn, replicationService, targetCgName, extraSpecs) targetCgInstanceName = self._find_consistency_group( replicationService, targetCgName) - LOG.info(_LI("Create target consistency group %(targetCg)s "), + LOG.info(_LI("Create target consistency group %(targetCg)s."), {'targetCg': targetCgInstanceName}) for memberInstanceName in memberInstanceNames: @@ -2421,20 +2674,20 @@ def create_cgsnapshot(self, context, cgsnapshot, db): volumeSizeInbits))} if extraSpecs[ISV3]: - rc, volumeDict, storageSystemName = ( # @UnusedVariable + _rc, volumeDict, _storageSystemName = ( self._create_v3_volume( - volume, extraSpecs, - targetVolumeName, volumeSizeInbits)) + volume, targetVolumeName, volumeSizeInbits, + extraSpecs)) else: - rc, volumeDict, storageSystemName = ( # @UnusedVariable + _rc, volumeDict, _storageSystemName = ( self._create_composite_volume( - volume, extraSpecs, - targetVolumeName, volumeSizeInbits)) + volume, targetVolumeName, volumeSizeInbits, + extraSpecs)) targetVolumeInstance = self.utils.find_volume_instance( self.conn, volumeDict, targetVolumeName) LOG.debug("Create target volume for member volume " - "source volume: %(memberVol)s" - "target volume %(targetVol)s", + "Source volume: %(memberVol)s " + "Target volume %(targetVol)s.", {'memberVol': memberInstanceName, 'targetVol': targetVolumeInstance.path}) self.provision.add_volume_to_cg(self.conn, @@ -2442,109 +2695,107 @@ def create_cgsnapshot(self, context, cgsnapshot, db): targetCgInstanceName, targetVolumeInstance.path, targetCgName, - targetVolumeName) + targetVolumeName, + extraSpecs) - # less than 5 characters relationship name + # Less than 5 characters relationship name. relationName = self.utils.truncate_string(cgsnapshot['id'], 5) if extraSpecs[ISV3]: - rc, job = ( # @UnusedVariable - self.provisionv3.create_group_replica( - self.conn, replicationService, cgInstanceName, - targetCgInstanceName, relationName)) + self.provisionv3.create_group_replica( + self.conn, replicationService, cgInstanceName, + targetCgInstanceName, relationName, extraSpecs) else: - rc, job = ( # @UnusedVariable - self.provision.create_group_replica( - self.conn, replicationService, cgInstanceName, - targetCgInstanceName, relationName)) - # break the replica group relationship + self.provision.create_group_replica( + self.conn, replicationService, cgInstanceName, + targetCgInstanceName, relationName, extraSpecs) + # Break the replica group relationship. rgSyncInstanceName = self.utils.find_group_sync_rg_by_target( - self.conn, storageSystem, targetCgInstanceName, True) + self.conn, storageSystem, targetCgInstanceName, extraSpecs, + True) if rgSyncInstanceName is not None: repservice = self.utils.find_replication_service( self.conn, storageSystem) if repservice is None: exception_message = (_( - "Cannot find Replication service on system %s"), + "Cannot find Replication service on system %s.") % storageSystem) raise exception.VolumeBackendAPIException( data=exception_message) if extraSpecs[ISV3]: - # operation 7: dissolve for snapVx + # Operation 7: dissolve for snapVx. operation = self.utils.get_num(9, '16') self.provisionv3.break_replication_relationship( - self.conn, repservice, rgSyncInstanceName, operation) + self.conn, repservice, rgSyncInstanceName, operation, + extraSpecs) else: self.provision.delete_clone_relationship(self.conn, repservice, - rgSyncInstanceName) + rgSyncInstanceName, + extraSpecs) - except Exception as ex: - modelUpdate['status'] = 'error' - self.utils.populate_cgsnapshot_status( - context, db, cgsnapshot['id'], modelUpdate['status']) - LOG.error(_LE("Exception: %(ex)s"), {'ex': ex}) + except Exception: exceptionMessage = (_("Failed to create snapshot for cg:" - " %(cgName)s. ") + " %(cgName)s.") % {'cgName': cgName}) - LOG.error(exceptionMessage) + LOG.exception(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) - snapshots = self.utils.populate_cgsnapshot_status( - context, db, cgsnapshot['id'], modelUpdate['status']) - return modelUpdate, snapshots + for snapshot in snapshots: + snapshots_model_update.append( + {'id': snapshot['id'], 'status': 'available'}) + modelUpdate = {'status': fields.ConsistencyGroupStatus.AVAILABLE} - def delete_cgsnapshot(self, context, cgsnapshot, db): - """delete a cgsnapshot. + return modelUpdate, snapshots_model_update - :param context: + def delete_cgsnapshot(self, context, cgsnapshot, snapshots): + """Delete a cgsnapshot. + + :param context: the context :param cgsnapshot: the consistency group snapshot to be created - :param db: cinder database - :returns: modelUpdate, list of snapshots + :param snapshots: snapshots + :returns: dict -- modelUpdate + :returns: list -- list of snapshots + :raises: VolumeBackendAPIException """ - consistencyGroup = db.consistencygroup_get( - context, cgsnapshot['consistencygroup_id']) - snapshots = db.snapshot_get_all_for_cgsnapshot( - context, cgsnapshot['id']) - + consistencyGroup = cgsnapshot.get('consistencygroup') + model_update = {} + snapshots_model_update = [] LOG.info(_LI( "Delete snapshot for source CG %(cgId)s " - "cgsnapshotID: %(cgsnapshot)s"), + "cgsnapshotID: %(cgsnapshot)s."), {'cgsnapshot': cgsnapshot['id'], 'cgId': cgsnapshot['consistencygroup_id']}) - modelUpdate = {'status': 'deleted'} + model_update['status'] = cgsnapshot['status'] volumeTypeId = consistencyGroup['volume_type_id'].replace(",", "") extraSpecs = self._initial_setup(None, volumeTypeId) self.conn = self._get_ecom_connection() - poolInstanceName, storageSystem = ( # @UnusedVariable + _poolInstanceName, storageSystem = ( self._get_pool_and_storage_system(extraSpecs)) try: targetCgName = self.utils.truncate_string(cgsnapshot['id'], 8) - modelUpdate, snapshots = self._delete_cg_and_members( - storageSystem, extraSpecs, targetCgName, modelUpdate, - snapshots) - except Exception as ex: - modelUpdate['status'] = 'error_deleting' - self.utils.populate_cgsnapshot_status( - context, db, cgsnapshot['id'], modelUpdate['status']) - LOG.error(_LE("Exception: %(ex)s"), {'ex': ex}) + model_update, snapshots = self._delete_cg_and_members( + storageSystem, targetCgName, model_update, + snapshots, extraSpecs) + for snapshot in snapshots: + snapshots_model_update.append( + {'id': snapshot['id'], 'status': 'deleted'}) + except Exception: exceptionMessage = (_("Failed to delete snapshot for cg: " - "%(cgId)s. ") + "%(cgId)s.") % {'cgId': cgsnapshot['consistencygroup_id']}) - LOG.error(exceptionMessage) + LOG.exception(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) - snapshots = self.utils.populate_cgsnapshot_status( - context, db, cgsnapshot['id'], modelUpdate['status']) - return modelUpdate, snapshots + return model_update, snapshots_model_update def _find_consistency_group(self, replicationService, cgName): """Finds a CG given its name. :param replicationService: the replication service :param cgName: the consistency group name - :returns: + :returns: foundCgInstanceName """ foundCgInstanceName = None cgInstanceNames = ( @@ -2563,7 +2814,7 @@ def _get_members_of_replication_group(self, cgInstanceName): """Get the members of consistency group. :param cgInstanceName: the CG instance name - :returns: + :returns: list -- memberInstanceNames """ memberInstanceNames = self.conn.AssociatorNames( cgInstanceName, @@ -2572,37 +2823,47 @@ def _get_members_of_replication_group(self, cgInstanceName): return memberInstanceNames def _create_composite_volume( - self, volume, extraSpecs, volumeName, volumeSize): - """create a composite volume (V2) + self, volume, volumeName, volumeSize, extraSpecs, + memberCount=None): + """Create a composite volume (V2). :param volume: the volume object - :param extraSpecs: - :param volumeName: - :param volumeSize: - :returns: - """ - memberCount, errorDesc = self.utils.determine_member_count( - volume['size'], extraSpecs[MEMBERCOUNT], extraSpecs[COMPOSITETYPE]) - if errorDesc is not None: - exceptionMessage = (_("The striped meta count of %(memberCount)s " - "is too small for volume: %(volumeName)s. " - "with size %(volumeSize)s ") - % {'memberCount': memberCount, - 'volumeName': volumeName, - 'volumeSize': volume['size']}) - LOG.error(exceptionMessage) - raise exception.VolumeBackendAPIException(data=exceptionMessage) + :param volumeName: the name of the volume + :param volumeSize: the size of the volume + :param extraSpecs: extra specifications + :param memberCount: the number of meta members in a composite volume + :returns: int -- return code + :returns: dict -- volumeDict + :returns: string -- storageSystemName + :raises: VolumeBackendAPIException + """ + if not memberCount: + memberCount, errorDesc = self.utils.determine_member_count( + volume['size'], extraSpecs[MEMBERCOUNT], + extraSpecs[COMPOSITETYPE]) + if errorDesc is not None: + exceptionMessage = (_("The striped meta count of " + "%(memberCount)s is too small for " + "volume: %(volumeName)s, " + "with size %(volumeSize)s.") + % {'memberCount': memberCount, + 'volumeName': volumeName, + 'volumeSize': volume['size']}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) poolInstanceName, storageSystemName = ( self._get_pool_and_storage_system(extraSpecs)) - LOG.debug("Create Volume: %(volume)s Pool: %(pool)s " + LOG.debug("Create Volume: %(volume)s Pool: %(pool)s " "Storage System: %(storageSystem)s " - "Size: %(size)lu ", + "Size: %(size)lu MemberCount: %(memberCount)s.", {'volume': volumeName, 'pool': poolInstanceName, 'storageSystem': storageSystemName, - 'size': volumeSize}) + 'size': volumeSize, + 'memberCount': memberCount}) elementCompositionService = ( self.utils.find_element_composition_service(self.conn, @@ -2612,7 +2873,7 @@ def _create_composite_volume( self.conn, storageSystemName) # If FAST is intended to be used we must first check that the pool - # is associated with the correct storage tier + # is associated with the correct storage tier. if extraSpecs[FASTPOLICY] is not None: foundPoolInstanceName = self.fast.get_pool_associated_to_policy( self.conn, extraSpecs[FASTPOLICY], extraSpecs[ARRAY], @@ -2622,7 +2883,8 @@ def _create_composite_volume( "is not associated to storage tier for " "fast policy %(fastPolicy)s.") % {'poolName': extraSpecs[POOL], - 'fastPolicy': extraSpecs[FASTPOLICY]}) + 'fastPolicy': + extraSpecs[FASTPOLICY]}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException( data=exceptionMessage) @@ -2632,26 +2894,26 @@ def _create_composite_volume( volumeDict, rc = self.provision.create_composite_volume( self.conn, elementCompositionService, volumeSize, volumeName, - poolInstanceName, compositeType, memberCount) + poolInstanceName, compositeType, memberCount, extraSpecs) # Now that we have already checked that the pool is associated with # the correct storage tier and the volume was successfully created # add the volume to the default storage group created for - # volumes in pools associated with this fast policy + # volumes in pools associated with this fast policy. if extraSpecs[FASTPOLICY]: LOG.info(_LI( "Adding volume: %(volumeName)s to default storage group" - " for FAST policy: %(fastPolicyName)s "), + " for FAST policy: %(fastPolicyName)s."), {'volumeName': volumeName, 'fastPolicyName': extraSpecs[FASTPOLICY]}) defaultStorageGroupInstanceName = ( self._get_or_create_default_storage_group( self.conn, storageSystemName, volumeDict, - volumeName, extraSpecs[FASTPOLICY])) + volumeName, extraSpecs[FASTPOLICY], extraSpecs)) if not defaultStorageGroupInstanceName: exceptionMessage = (_( "Unable to create or get default storage group for " - "FAST policy: %(fastPolicyName)s. ") + "FAST policy: %(fastPolicyName)s.") % {'fastPolicyName': extraSpecs[FASTPOLICY]}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException( @@ -2659,18 +2921,21 @@ def _create_composite_volume( self._add_volume_to_default_storage_group_on_create( volumeDict, volumeName, storageConfigService, - storageSystemName, extraSpecs[FASTPOLICY]) + storageSystemName, extraSpecs[FASTPOLICY], extraSpecs) return rc, volumeDict, storageSystemName def _create_v3_volume( - self, volume, extraSpecs, volumeName, volumeSize): - """create a volume (V3) + self, volume, volumeName, volumeSize, extraSpecs): + """Create a volume (V3). :param volume: the volume object - :param extraSpecs: - :param volumeName: - :param volumeSize: - :returns: + :param volumeName: the volume name + :param volumeSize: the volume size + :param extraSpecs: extra specifications + :returns: int -- return code + :returns: dict -- volumeDict + :returns: string -- storageSystemName + :raises: VolumeBackendAPIException """ isValidSLO, isValidWorkload = self.utils.verify_slo_workload( extraSpecs[SLO], extraSpecs[WORKLOAD]) @@ -2678,7 +2943,7 @@ def _create_v3_volume( if not isValidSLO or not isValidWorkload: exceptionMessage = (_( "Either SLO: %(slo)s or workload %(workload)s is invalid. " - "Examine previous error statement for valid values") + "Examine previous error statement for valid values.") % {'slo': extraSpecs[SLO], 'workload': extraSpecs[WORKLOAD]}) LOG.error(exceptionMessage) @@ -2687,9 +2952,22 @@ def _create_v3_volume( poolInstanceName, storageSystemName = ( self._get_pool_and_storage_system(extraSpecs)) - LOG.debug("Create Volume: %(volume)s Pool: %(pool)s " + # Check to see if SLO and Workload are configured on the array. + storagePoolCapability = self.provisionv3.get_storage_pool_capability( + self.conn, poolInstanceName) + if storagePoolCapability: + self.provisionv3.get_storage_pool_setting( + self.conn, storagePoolCapability, extraSpecs[SLO], + extraSpecs[WORKLOAD]) + else: + exceptionMessage = (_( + "Cannot determine storage pool settings.")) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException(data=exceptionMessage) + + LOG.debug("Create Volume: %(volume)s Pool: %(pool)s " "Storage System: %(storageSystem)s " - "Size: %(size)lu ", + "Size: %(size)lu.", {'volume': volumeName, 'pool': poolInstanceName, 'storageSystem': storageSystemName, @@ -2698,72 +2976,51 @@ def _create_v3_volume( storageConfigService = self.utils.find_storage_configuration_service( self.conn, storageSystemName) - # check the SLO range - maximumVolumeSize, minimumVolumeSize = ( - self.provisionv3.get_volume_range( - self.conn, storageConfigService, poolInstanceName, - extraSpecs[SLO], extraSpecs[WORKLOAD])) - if not self.utils.is_in_range( - volumeSize, maximumVolumeSize, minimumVolumeSize): - LOG.warn(_LW( - "Volume: %(volume)s with size: %(volumeSize)s bits " - "is not in the Performance Capacity range : " - "%(minimumVolumeSize)s-%(maximumVolumeSize)s bits. " - "for SLO:%(slo)s and workload:%(workload)s. " - "Unpredictable results may occur. "), - {'volume': volumeName, - 'volumeSize': volumeSize, - 'minimumVolumeSize': minimumVolumeSize, - 'maximumVolumeSize': maximumVolumeSize, - 'slo': extraSpecs[SLO], - 'workload': extraSpecs[WORKLOAD] - }) - # A volume created without specifying a storage group during # creation time is allocated from the default SRP pool and - # assigned the optimized SLO + # assigned the optimized SLO. sgInstanceName = self._get_or_create_storage_group_v3( - extraSpecs[POOL], extraSpecs[SLO], extraSpecs[WORKLOAD], - storageSystemName) + extraSpecs[POOL], extraSpecs[SLO], + extraSpecs[WORKLOAD], storageSystemName, extraSpecs) volumeDict, rc = self.provisionv3.create_volume_from_sg( self.conn, storageConfigService, volumeName, - sgInstanceName, volumeSize) + sgInstanceName, volumeSize, extraSpecs) return rc, volumeDict, storageSystemName def _get_or_create_storage_group_v3( - self, poolName, slo, workload, storageSystemName): - """Get or create storage group_v3 (V3) + self, poolName, slo, workload, storageSystemName, extraSpecs): + """Get or create storage group_v3 (V3). :param poolName: the SRP pool nsmr :param slo: the SLO :param workload: the workload :param storageSystemName: storage system name - :returns: + :param extraSpecs: extra specifications + :returns: sgInstanceName """ - storageGroupName = self.utils.get_v3_storage_group_name( - poolName, slo, workload) - controllerConfigService = ( - self.utils.find_controller_configuration_service( - self.conn, storageSystemName)) - sgInstanceName = self.utils.find_storage_masking_group( - self.conn, controllerConfigService, storageGroupName) + storageGroupName, controllerConfigService, sgInstanceName = ( + self.utils.get_v3_default_sg_instance_name( + self.conn, poolName, slo, workload, storageSystemName)) if sgInstanceName is None: sgInstanceName = self.provisionv3.create_storage_group_v3( self.conn, controllerConfigService, storageGroupName, - poolName, slo, workload) + poolName, slo, workload, extraSpecs) return sgInstanceName def _extend_composite_volume(self, volumeInstance, volumeName, - newSize, additionalVolumeSize): - """extend a composite volume (V2) + newSize, additionalVolumeSize, extraSpecs): + """Extend a composite volume (V2). :param volumeInstance: the volume instance :param volumeName: the name of the volume :param newSize: in GBs - :param additionalVolumeSize: - :returns: + :param additionalVolumeSize: additional volume size + :param extraSpecs: extra specifications + :returns: int -- return code + :returns: dict -- modifiedVolumeDict + :raises: VolumeBackendAPIException """ # Is the volume extendable. isConcatenated = self.utils.check_if_volume_is_extendable( @@ -2779,7 +3036,7 @@ def _extend_composite_volume(self, volumeInstance, volumeName, else: compositeType = self.utils.get_composite_type(CONCATENATED) - LOG.debug("Extend Volume: %(volume)s New size: %(newSize)s GBs", + LOG.debug("Extend Volume: %(volume)s New size: %(newSize)s GBs.", {'volume': volumeName, 'newSize': newSize}) @@ -2787,7 +3044,7 @@ def _extend_composite_volume(self, volumeInstance, volumeName, storageSystemName = volumeInstance['SystemName'] LOG.debug( "Device ID: %(deviceid)s: Storage System: " - "%(storagesystem)s", + "%(storagesystem)s.", {'deviceid': deviceId, 'storagesystem': storageSystemName}) @@ -2798,26 +3055,27 @@ def _extend_composite_volume(self, volumeInstance, volumeName, self.utils.find_element_composition_service( self.conn, storageSystemName)) - # create a volume to the size of the - # newSize - oldSize = additionalVolumeSize + # Create a volume to the size of the + # newSize - oldSize = additionalVolumeSize. unboundVolumeInstance = self._create_and_get_unbound_volume( self.conn, storageConfigService, volumeInstance.path, - additionalVolumeSize) + additionalVolumeSize, extraSpecs) if unboundVolumeInstance is None: exceptionMessage = (_( - "Error Creating unbound volume on an Extend operation")) + "Error Creating unbound volume on an Extend operation.")) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) - # add the new unbound volume to the original composite volume + # Add the new unbound volume to the original composite volume. rc, modifiedVolumeDict = ( self._modify_and_get_composite_volume_instance( self.conn, elementCompositionService, volumeInstance, - unboundVolumeInstance.path, volumeName, compositeType)) + unboundVolumeInstance.path, volumeName, compositeType, + extraSpecs)) if modifiedVolumeDict is None: exceptionMessage = (_( "On an Extend Operation, error adding volume to composite " - "volume: %(volumename)s. ") + "volume: %(volumename)s.") % {'volumename': volumeName}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) @@ -2825,53 +3083,55 @@ def _extend_composite_volume(self, volumeInstance, volumeName, return rc, modifiedVolumeDict def _slo_workload_migration(self, volumeInstance, volume, host, - volumeName, volumeStatus, - extraSpecs, newType): - """migrate from SLO/Workload combination to another (V3) + volumeName, volumeStatus, newType, + extraSpecs): + """Migrate from SLO/Workload combination to another (V3). :param volumeInstance: the volume instance :param volume: the volume object :param host: the host object :param volumeName: the name of the volume :param volumeStatus: the volume status - :param extraSpecs: the extra specs dict - :param newType: - :returns: boolean + :param newType: the type to migrate to + :param extraSpecs: extra specifications + :returns: boolean -- True if migration succeeded, False if error. """ + storageGroupName = self.utils.get_v3_storage_group_name( + extraSpecs[POOL], extraSpecs[SLO], extraSpecs[WORKLOAD]) volumeInstanceName = volumeInstance.path isValid, targetSlo, targetWorkload = ( self._is_valid_for_storage_assisted_migration_v3( volumeInstanceName, host, extraSpecs[ARRAY], - extraSpecs[POOL], volumeName, volumeStatus)) + extraSpecs[POOL], volumeName, volumeStatus, + storageGroupName)) storageSystemName = volumeInstance['SystemName'] - if not isValid: LOG.error(_LE( "Volume %(name)s is not suitable for storage " - "assisted migration using retype"), + "assisted migration using retype."), {'name': volumeName}) return False if volume['host'] != host['host']: LOG.debug( "Retype Volume %(name)s from source host %(sourceHost)s " - "to target host %(targetHost)s", + "to target host %(targetHost)s.", {'name': volumeName, 'sourceHost': volume['host'], 'targetHost': host['host']}) return self._migrate_volume_v3( volume, volumeInstance, extraSpecs[POOL], targetSlo, - targetWorkload, storageSystemName, newType) + targetWorkload, storageSystemName, newType, extraSpecs) return False def _migrate_volume_v3( self, volume, volumeInstance, poolName, targetSlo, - targetWorkload, storageSystemName, newType): - """migrate from one slo/workload combination to another (V3) + targetWorkload, storageSystemName, newType, extraSpecs): + """Migrate from one slo/workload combination to another (V3). - This requires moving the volume from it's current SG to a - new or existing SG that has the target attributes + This requires moving the volume from its current SG to a + new or existing SG that has the target attributes. :param volume: the volume object :param volumeInstance: the volume instance @@ -2879,8 +3139,9 @@ def _migrate_volume_v3( :param targetSlo: the target SLO :param targetWorkload: the target workload :param storageSystemName: the storage system name - :param newType: - :returns: boolean + :param newType: the type to migrate to + :param extraSpecs: extra specifications + :returns: boolean -- True if migration succeeded, False if error. """ volumeName = volume['name'] @@ -2888,13 +3149,16 @@ def _migrate_volume_v3( self.utils.find_controller_configuration_service( self.conn, storageSystemName)) + defaultSgName = self.utils.get_v3_storage_group_name( + extraSpecs[POOL], extraSpecs[SLO], extraSpecs[WORKLOAD]) + foundStorageGroupInstanceName = ( self.utils.get_storage_group_from_volume( - self.conn, volumeInstance.path)) + self.conn, volumeInstance.path, defaultSgName)) if foundStorageGroupInstanceName is None: - LOG.warn(_LW( + LOG.warning(_LW( "Volume : %(volumeName)s is not currently " - "belonging to any storage group "), + "belonging to any storage group."), {'volumeName': volumeName}) else: self.provision.remove_device_from_storage_group( @@ -2902,15 +3166,15 @@ def _migrate_volume_v3( controllerConfigService, foundStorageGroupInstanceName, volumeInstance.path, - volumeName) - # check that it has been removed + volumeName, extraSpecs) + # Check that it has been removed. sgFromVolRemovedInstanceName = ( self.utils.wrap_get_storage_group_from_volume( - self.conn, volumeInstance.path)) + self.conn, volumeInstance.path, defaultSgName)) if sgFromVolRemovedInstanceName is not None: LOG.error(_LE( "Volume : %(volumeName)s has not been " - "removed from source storage group %(storageGroup)s"), + "removed from source storage group %(storageGroup)s."), {'volumeName': volumeName, 'storageGroup': sgFromVolRemovedInstanceName}) return False @@ -2919,24 +3183,25 @@ def _migrate_volume_v3( poolName, targetSlo, targetWorkload) targetSgInstanceName = self._get_or_create_storage_group_v3( - poolName, targetSlo, targetWorkload, storageSystemName) + poolName, targetSlo, targetWorkload, storageSystemName, + extraSpecs) if targetSgInstanceName is None: LOG.error(_LE( - "Failed to get or create storage group %(storageGroupName)s "), + "Failed to get or create storage group %(storageGroupName)s."), {'storageGroupName': storageGroupName}) return False self.masking.add_volume_to_storage_group( self.conn, controllerConfigService, targetSgInstanceName, - volumeInstance, volumeName, storageGroupName) - # check that it has been added + volumeInstance, volumeName, storageGroupName, extraSpecs) + # Check that it has been added. sgFromVolAddedInstanceName = ( self.utils.get_storage_group_from_volume( - self.conn, volumeInstance.path)) + self.conn, volumeInstance.path, storageGroupName)) if sgFromVolAddedInstanceName is None: LOG.error(_LE( "Volume : %(volumeName)s has not been " - "added to target storage group %(storageGroup)s"), + "added to target storage group %(storageGroup)s."), {'volumeName': volumeName, 'storageGroup': targetSgInstanceName}) return False @@ -2945,8 +3210,8 @@ def _migrate_volume_v3( def _pool_migration(self, volumeInstance, volume, host, volumeName, volumeStatus, - fastPolicyName, newType): - """migrate from one pool to another (V2) + fastPolicyName, newType, extraSpecs): + """Migrate from one pool to another (V2). :param volumeInstance: the volume instance :param volume: the volume object @@ -2954,8 +3219,9 @@ def _pool_migration(self, volumeInstance, volume, host, :param volumeName: the name of the volume :param volumeStatus: the volume status :param fastPolicyName: the FAST policy Name - :param newType: - :returns: boolean + :param newType: the type to migrate to + :param extraSpecs: extra specifications + :returns: boolean -- True if migration succeeded, False if error. """ storageSystemName = volumeInstance['SystemName'] isValid, targetPoolName, targetFastPolicyName = ( @@ -2966,100 +3232,95 @@ def _pool_migration(self, volumeInstance, volume, host, if not isValid: LOG.error(_LE( "Volume %(name)s is not suitable for storage " - "assisted migration using retype"), + "assisted migration using retype."), {'name': volumeName}) return False if volume['host'] != host['host']: LOG.debug( "Retype Volume %(name)s from source host %(sourceHost)s " - "to target host %(targetHost)s", + "to target host %(targetHost)s.", {'name': volumeName, 'sourceHost': volume['host'], 'targetHost': host['host']}) return self._migrate_volume( volume, volumeInstance, targetPoolName, targetFastPolicyName, - fastPolicyName, newType) + fastPolicyName, extraSpecs, newType) return False def _update_pool_stats( - self, emcConfigFileName, backendName, arrayName, poolName): - """update pool statistics (V2) + self, backendName, arrayInfo): + """Update pool statistics (V2). - :param emcConfigFileName: the EMC configuration file :param backendName: the backend name - :param arrayName: the array Name - :returns: location_info, total_capacity_gb, free_capacity_gb + :param arrayInfo: the arrayInfo + :returns: location_info, total_capacity_gb, free_capacity_gb, + provisioned_capacity_gb """ - # This value can be None - fastPolicyName = self.utils.parse_fast_policy_name_from_file( - emcConfigFileName) - if fastPolicyName is not None: + + if arrayInfo['FastPolicy']: LOG.debug( - "Fast policy %(fastPolicyName)s is enabled on %(arrayName)s. ", - {'fastPolicyName': fastPolicyName, - 'arrayName': arrayName}) + "Fast policy %(fastPolicyName)s is enabled on %(arrayName)s.", + {'fastPolicyName': arrayInfo['FastPolicy'], + 'arrayName': arrayInfo['SerialNumber']}) else: LOG.debug( "No Fast policy for Array:%(arrayName)s " - "backend:%(backendName)s", - {'arrayName': arrayName, + "backend:%(backendName)s.", + {'arrayName': arrayInfo['SerialNumber'], 'backendName': backendName}) storageSystemInstanceName = self.utils.find_storageSystem( - self.conn, arrayName) + self.conn, arrayInfo['SerialNumber']) isTieringPolicySupported = ( self.fast.is_tiering_policy_enabled_on_storage_system( self.conn, storageSystemInstanceName)) - if (fastPolicyName is not None and + if (arrayInfo['FastPolicy'] is not None and isTieringPolicySupported is True): # FAST enabled - total_capacity_gb, free_capacity_gb = ( + (total_capacity_gb, free_capacity_gb, provisioned_capacity_gb, + array_max_over_subscription) = ( self.fast.get_capacities_associated_to_policy( - self.conn, arrayName, fastPolicyName)) - LOG.info( - "FAST: capacity stats for policy %(fastPolicyName)s on " - "array %(arrayName)s (total_capacity_gb=%(total_capacity_gb)lu" - ", free_capacity_gb=%(free_capacity_gb)lu" - % {'fastPolicyName': fastPolicyName, - 'arrayName': arrayName, - 'total_capacity_gb': total_capacity_gb, - 'free_capacity_gb': free_capacity_gb}) + self.conn, arrayInfo['SerialNumber'], + arrayInfo['FastPolicy'])) + LOG.info(_LI( + "FAST: capacity stats for policy %(fastPolicyName)s on array " + "%(arrayName)s. total_capacity_gb=%(total_capacity_gb)lu, " + "free_capacity_gb=%(free_capacity_gb)lu."), + {'fastPolicyName': arrayInfo['FastPolicy'], + 'arrayName': arrayInfo['SerialNumber'], + 'total_capacity_gb': total_capacity_gb, + 'free_capacity_gb': free_capacity_gb}) else: # NON-FAST - total_capacity_gb, free_capacity_gb = ( - self.utils.get_pool_capacities(self.conn, poolName, arrayName)) - LOG.info( + (total_capacity_gb, free_capacity_gb, provisioned_capacity_gb, + array_max_over_subscription) = ( + self.utils.get_pool_capacities(self.conn, + arrayInfo['PoolName'], + arrayInfo['SerialNumber'])) + LOG.info(_LI( "NON-FAST: capacity stats for pool %(poolName)s on array " - "%(arrayName)s (total_capacity_gb=%(total_capacity_gb)lu, " - "free_capacity_gb=%(free_capacity_gb)lu" - % {'poolName': poolName, - 'arrayName': arrayName, - 'total_capacity_gb': total_capacity_gb, - 'free_capacity_gb': free_capacity_gb}) - - if poolName is None: - LOG.debug("Unable to get the poolName for location_info") - if arrayName is None: - LOG.debug("Unable to get the arrayName for location_info") - if fastPolicyName is None: - LOG.debug("FAST is not enabled for this configuration: " - "%(emcConfigFileName)s", - {'emcConfigFileName': emcConfigFileName}) + "%(arrayName)s total_capacity_gb=%(total_capacity_gb)lu, " + "free_capacity_gb=%(free_capacity_gb)lu."), + {'poolName': arrayInfo['PoolName'], + 'arrayName': arrayInfo['SerialNumber'], + 'total_capacity_gb': total_capacity_gb, + 'free_capacity_gb': free_capacity_gb}) location_info = ("%(arrayName)s#%(poolName)s#%(policyName)s" - % {'arrayName': arrayName, - 'poolName': poolName, - 'policyName': fastPolicyName}) + % {'arrayName': arrayInfo['SerialNumber'], + 'poolName': arrayInfo['PoolName'], + 'policyName': arrayInfo['FastPolicy']}) - return location_info, total_capacity_gb, free_capacity_gb + return (location_info, total_capacity_gb, free_capacity_gb, + provisioned_capacity_gb, array_max_over_subscription) - def _set_v2_extra_specs(self, extraSpecs, configurationFile, arrayName): - """set the VMAX V2 extra specs + def _set_v2_extra_specs(self, extraSpecs, poolRecord): + """Set the VMAX V2 extra specs. - :param extraSpecs: the extraSpecs (input) - :param configurationFile: the EMC configuration file - :param arrayName: the array serial number - :returns: extraSpecs (out) + :param extraSpecs: extra specifications + :param poolRecord: pool record + :returns: dict -- the extraSpecs + :raises: VolumeBackendAPIException """ try: stripedMetaCount = extraSpecs[STRIPECOUNT] @@ -3068,43 +3329,28 @@ def _set_v2_extra_specs(self, extraSpecs, configurationFile, arrayName): LOG.debug( "There are: %(stripedMetaCount)s striped metas in " - "the extra specs", + "the extra specs.", {'stripedMetaCount': stripedMetaCount}) - except Exception: + except KeyError: memberCount = '1' extraSpecs[MEMBERCOUNT] = memberCount extraSpecs[COMPOSITETYPE] = CONCATENATED - LOG.debug("StripedMetaCount is not in the extra specs") - pass - - poolName = self.utils.parse_pool_name_from_file(configurationFile) - if poolName is None: - exceptionMessage = (_( - "The pool cannot be null. The pool must be configured " - "either in the extra specs or in the EMC configuration " - "file corresponding to the Volume Type. ")) - LOG.error(exceptionMessage) - raise exception.VolumeBackendAPIException( - data=exceptionMessage) - - # Get the FAST policy from the file this value can be None if the - # user doesnt want to associate with any FAST policy - fastPolicyName = self.utils.parse_fast_policy_name_from_file( - configurationFile) - if fastPolicyName is not None: - LOG.debug("The fast policy name is: %(fastPolicyName)s. ", - {'fastPolicyName': fastPolicyName}) - - extraSpecs[POOL] = poolName - extraSpecs[ARRAY] = arrayName - extraSpecs[FASTPOLICY] = fastPolicyName + LOG.debug("StripedMetaCount is not in the extra specs.") + + # Get the FAST policy from the file. This value can be None if the + # user doesn't want to associate with any FAST policy. + if poolRecord['FastPolicy']: + LOG.debug("The fast policy name is: %(fastPolicyName)s.", + {'fastPolicyName': poolRecord['FastPolicy']}) + extraSpecs[FASTPOLICY] = poolRecord['FastPolicy'] extraSpecs[ISV3] = False + extraSpecs = self._set_common_extraSpecs(extraSpecs, poolRecord) LOG.debug("Pool is: %(pool)s " "Array is: %(array)s " "FastPolicy is: %(fastPolicy)s " "CompositeType is: %(compositeType)s " - "MemberCount is: %(memberCount)s ", + "MemberCount is: %(memberCount)s.", {'pool': extraSpecs[POOL], 'array': extraSpecs[ARRAY], 'fastPolicy': extraSpecs[FASTPOLICY], @@ -3112,46 +3358,69 @@ def _set_v2_extra_specs(self, extraSpecs, configurationFile, arrayName): 'memberCount': extraSpecs[MEMBERCOUNT]}) return extraSpecs - def _set_v3_extra_specs(self, extraSpecs, configurationFile, arrayName): - """set the VMAX V3 extra specs. + def _set_v3_extra_specs(self, extraSpecs, poolRecord): + """Set the VMAX V3 extra specs. If SLO or workload are not specified then the default values are NONE and the Optimized SLO will be assigned to the - volume - - :param extraSpecs: the extraSpecs (input) - :param configurationFile: the EMC configuration file - :param arrayName: the array serial number - :returns: extraSpecs (out) - """ - extraSpecs[SLO] = self.utils.parse_slo_from_file(configurationFile) - extraSpecs[WORKLOAD] = self.utils.parse_workload_from_file( - configurationFile) - extraSpecs[POOL] = self.utils.parse_pool_name_from_file( - configurationFile) - extraSpecs[ARRAY] = arrayName - extraSpecs[ISV3] = True + volume. + :param extraSpecs: extra specifications + :param poolRecord: pool record + :returns: dict -- the extra specifications dictionary + """ + extraSpecs[SLO] = poolRecord['SLO'] + extraSpecs[WORKLOAD] = poolRecord['Workload'] + extraSpecs[ISV3] = True + extraSpecs = self._set_common_extraSpecs(extraSpecs, poolRecord) LOG.debug("Pool is: %(pool)s " "Array is: %(array)s " "SLO is: %(slo)s " - "Workload is: %(workload)s ", + "Workload is: %(workload)s.", {'pool': extraSpecs[POOL], 'array': extraSpecs[ARRAY], 'slo': extraSpecs[SLO], 'workload': extraSpecs[WORKLOAD]}) return extraSpecs + def _set_common_extraSpecs(self, extraSpecs, poolRecord): + """Set common extra specs. + + The extraSpecs are common to v2 and v3 + + :param extraSpecs: extra specifications + :param poolRecord: pool record + :returns: dict -- the extra specifications dictionary + """ + extraSpecs[POOL] = poolRecord['PoolName'] + extraSpecs[ARRAY] = poolRecord['SerialNumber'] + extraSpecs[PORTGROUPNAME] = poolRecord['PortGroup'] + if 'Interval' in poolRecord and poolRecord['Interval']: + extraSpecs[INTERVAL] = poolRecord['Interval'] + LOG.debug("The user defined interval is : %(intervalInSecs)s.", + {'intervalInSecs': poolRecord['Interval']}) + else: + LOG.debug("Interval not overridden, default of 10 assumed.") + if 'Retries' in poolRecord and poolRecord['Retries']: + extraSpecs[RETRIES] = poolRecord['Retries'] + LOG.debug("The user defined retries is : %(retries)s.", + {'retries': poolRecord['Retries']}) + else: + LOG.debug("Retries not overridden, default of 60 assumed.") + return extraSpecs + def _delete_from_pool(self, storageConfigService, volumeInstance, - volumeName, deviceId, fastPolicyName): - """delete from pool (v2) + volumeName, deviceId, fastPolicyName, extraSpecs): + """Delete from pool (v2). :param storageConfigService: the storage config service :param volumeInstance: the volume instance :param volumeName: the volume Name :param deviceId: the device ID of the volume :param fastPolicyName: the FAST policy name(if it exists) - :returns: rc + :param extraSpecs: extra specifications + :returns: int -- return code + :raises: VolumeBackendAPIException """ storageSystemName = volumeInstance['SystemName'] controllerConfigurationService = ( @@ -3161,26 +3430,29 @@ def _delete_from_pool(self, storageConfigService, volumeInstance, defaultStorageGroupInstanceName = ( self.masking.remove_device_from_default_storage_group( self.conn, controllerConfigurationService, - volumeInstance.path, volumeName, fastPolicyName)) + volumeInstance.path, volumeName, fastPolicyName, + extraSpecs)) if defaultStorageGroupInstanceName is None: - LOG.warn(_LW( + LOG.warning(_LW( "The volume: %(volumename)s. was not first part of the " "default storage group for FAST policy %(fastPolicyName)s" "."), {'volumename': volumeName, 'fastPolicyName': fastPolicyName}) - # check if it is part of another storage group - self._pre_check_for_deletion(controllerConfigurationService, - volumeInstance.path, volumeName) + # Check if it is part of another storage group. + self._remove_device_from_storage_group( + controllerConfigurationService, + volumeInstance.path, volumeName, extraSpecs) else: - # check if volume is part of a storage group - self._pre_check_for_deletion(controllerConfigurationService, - volumeInstance.path, volumeName) - - LOG.debug("Delete Volume: %(name)s Method: EMCReturnToStoragePool " - "ConfigServic: %(service)s TheElement: %(vol_instance)s " - "DeviceId: %(deviceId)s ", + # Check if volume is part of a storage group. + self._remove_device_from_storage_group( + controllerConfigurationService, + volumeInstance.path, volumeName, extraSpecs) + + LOG.debug("Delete Volume: %(name)s Method: EMCReturnToStoragePool " + "ConfigService: %(service)s TheElement: %(vol_instance)s " + "DeviceId: %(deviceId)s.", {'service': storageConfigService, 'name': volumeName, 'vol_instance': volumeInstance.path, @@ -3188,11 +3460,11 @@ def _delete_from_pool(self, storageConfigService, volumeInstance, try: rc = self.provision.delete_volume_from_pool( self.conn, storageConfigService, volumeInstance.path, - volumeName) + volumeName, extraSpecs) - except Exception as e: - # if we cannot successfully delete the volume then we want to - # return the volume to the default storage group + except Exception: + # If we cannot successfully delete the volume then we want to + # return the volume to the default storage group. if (fastPolicyName is not None and defaultStorageGroupInstanceName is not None and storageSystemName is not None): @@ -3200,50 +3472,50 @@ def _delete_from_pool(self, storageConfigService, volumeInstance, self.fast .add_volume_to_default_storage_group_for_fast_policy( self.conn, controllerConfigurationService, - volumeInstance, volumeName, fastPolicyName)) + volumeInstance, volumeName, fastPolicyName, + extraSpecs)) if assocDefaultStorageGroupName is None: LOG.error(_LE( "Failed to Roll back to re-add volume %(volumeName)s " "to default storage group for fast policy " - "%(fastPolicyName)s: Please contact your sysadmin to " + "%(fastPolicyName)s. Please contact your sysadmin to " "get the volume returned to the default " - "storage group"), + "storage group."), {'volumeName': volumeName, 'fastPolicyName': fastPolicyName}) - LOG.error(_LE("Exception: %s"), e) - errorMessage = (_("Failed to delete volume %(volumeName)s"), + errorMessage = (_("Failed to delete volume %(volumeName)s.") % {'volumeName': volumeName}) - LOG.error(errorMessage) + LOG.exception(errorMessage) raise exception.VolumeBackendAPIException(data=errorMessage) - return rc def _delete_from_pool_v3(self, storageConfigService, volumeInstance, - volumeName, deviceId, storageGroupName): - """delete from pool (v3) + volumeName, deviceId, extraSpecs): + """Delete from pool (v3). :param storageConfigService: the storage config service :param volumeInstance: the volume instance :param volumeName: the volume Name :param deviceId: the device ID of the volume - :param storageGroupName: the name of the default SG - :returns: rc + :param extraSpecs: extra specifications + :returns: int -- return code + :raises: VolumeBackendAPIException """ storageSystemName = volumeInstance['SystemName'] controllerConfigurationService = ( self.utils.find_controller_configuration_service( self.conn, storageSystemName)) - # check if it is part of a storage group and delete it - # extra logic for case when volume is the last member - sgFromVolInstanceName = self.masking.remove_and_reset_members( + # Check if it is part of a storage group and delete it + # extra logic for case when volume is the last member. + self.masking.remove_and_reset_members( self.conn, controllerConfigurationService, volumeInstance, - None, volumeName, True, None, 'noReset') + volumeName, extraSpecs, None, False) LOG.debug("Delete Volume: %(name)s Method: EMCReturnToStoragePool " "ConfigServic: %(service)s TheElement: %(vol_instance)s " - "DeviceId: %(deviceId)s ", + "DeviceId: %(deviceId)s.", {'service': storageConfigService, 'name': volumeName, 'vol_instance': volumeInstance.path, @@ -3251,34 +3523,19 @@ def _delete_from_pool_v3(self, storageConfigService, volumeInstance, try: rc = self.provisionv3.delete_volume_from_pool( self.conn, storageConfigService, volumeInstance.path, - volumeName) - - except Exception as e: - # if we cannot successfully delete the volume then we want to - # return the volume to the default storage group - # which should be the SG it previously belonged to - storageGroupInstanceName = self.utils.find_storage_masking_group( - self.conn, controllerConfigurationService, storageGroupName) - - if sgFromVolInstanceName is not storageGroupInstanceName: - LOG.debug( - "Volume: %(volumeName)s was not previously part of " - " %(storageGroupInstanceName)s. " - "Returning to %(storageGroupName)s", - {'volumeName': volumeName, - 'storageGroupInstanceName': storageGroupInstanceName, - 'storageGroupName': storageGroupName}) + volumeName, extraSpecs) - if storageGroupInstanceName is not None: - self.masking.add_volume_to_storage_group( - self.conn, controllerConfigurationService, - storageGroupInstanceName, volumeInstance, volumeName, - storageGroupName) + except Exception: + # If we cannot successfully delete the volume, then we want to + # return the volume to the default storage group, + # which should be the SG it previously belonged to. + self.masking.return_volume_to_default_storage_group_v3( + self.conn, controllerConfigurationService, + volumeInstance, volumeName, extraSpecs) - LOG.error(_LE("Exception: %s"), e) - errorMessage = (_("Failed to delete volume %(volumeName)s"), + errorMessage = (_("Failed to delete volume %(volumeName)s.") % {'volumeName': volumeName}) - LOG.error(errorMessage) + LOG.exception(errorMessage) raise exception.VolumeBackendAPIException(data=errorMessage) return rc @@ -3286,59 +3543,60 @@ def _delete_from_pool_v3(self, storageConfigService, volumeInstance, def _create_clone_v2(self, repServiceInstanceName, cloneVolume, sourceVolume, sourceInstance, isSnapshot, extraSpecs): - """create a clone (v2) + """Create a clone (v2). :param repServiceInstanceName: the replication service :param cloneVolume: the clone volume object :param sourceVolume: the source volume object :param sourceInstance: the device ID of the volume :param isSnapshot: check to see if it is a snapshot - :param fastPolicyName: the FAST policy name(if it exists) - :returns: rc + :param extraSpecs: extra specifications + :returns: int -- return code + :raises: VolumeBackendAPIException """ - # check if the source volume contains any meta devices + # Check if the source volume contains any meta devices. metaHeadInstanceName = self.utils.get_volume_meta_head( self.conn, sourceInstance.path) - if metaHeadInstanceName is None: # simple volume + if metaHeadInstanceName is None: # Simple volume. return self._create_v2_replica_and_delete_clone_relationship( repServiceInstanceName, cloneVolume, sourceVolume, - sourceInstance, None, isSnapshot, extraSpecs) - else: # composite volume with meta device members - # check if the meta members' capacity + sourceInstance, None, extraSpecs, isSnapshot) + else: # Composite volume with meta device members. + # Check if the meta members capacity. metaMemberInstanceNames = ( - self.utils.get_meta_members_of_composite_volume( - self.conn, metaHeadInstanceName)) - volumeCapacities = self.utils.get_meta_members_capacity_in_bit( + self.utils.get_composite_elements( + self.conn, sourceInstance)) + volumeCapacities = self.utils.get_meta_members_capacity_in_byte( self.conn, metaMemberInstanceNames) - LOG.debug("Volume capacities: %(metasizes)s ", + LOG.debug("Volume capacities: %(metasizes)s.", {'metasizes': volumeCapacities}) if len(set(volumeCapacities)) == 1: - LOG.debug("Meta volume all of the same size") + LOG.debug("Meta volume all of the same size.") return self._create_v2_replica_and_delete_clone_relationship( repServiceInstanceName, cloneVolume, sourceVolume, - sourceInstance, None, isSnapshot, extraSpecs) + sourceInstance, None, extraSpecs, isSnapshot) LOG.debug("Meta volumes are of different sizes, " - "%d different sizes", len(set(volumeCapacities))) + "%d different sizes.", len(set(volumeCapacities))) baseTargetVolumeInstance = None for volumeSizeInbits in volumeCapacities: - if baseTargetVolumeInstance is None: # create base volume + if baseTargetVolumeInstance is None: # Create base volume. baseVolumeName = "TargetBaseVol" volume = {'size': int(self.utils.convert_bits_to_gbs( volumeSizeInbits))} - r, baseVolumeDict, storageSystemName = ( # @UnusedVariable + _rc, baseVolumeDict, storageSystemName = ( self._create_composite_volume( - volume, extraSpecs, - baseVolumeName, volumeSizeInbits)) + volume, baseVolumeName, volumeSizeInbits, + extraSpecs, 1)) baseTargetVolumeInstance = self.utils.find_volume_instance( self.conn, baseVolumeDict, baseVolumeName) LOG.debug("Base target volume %(targetVol)s created. " - "capacity in bits: %(capInBits)lu ", + "capacity in bits: %(capInBits)lu.", {'capInBits': volumeSizeInbits, 'targetVol': baseTargetVolumeInstance.path}) - else: # create append volume + else: # Create append volume targetVolumeName = "MetaVol" volume = {'size': int(self.utils.convert_bits_to_gbs( volumeSizeInbits))} @@ -3348,54 +3606,120 @@ def _create_clone_v2(self, repServiceInstanceName, cloneVolume, unboundVolumeInstance = ( self._create_and_get_unbound_volume( self.conn, storageConfigService, - baseTargetVolumeInstance.path, volumeSizeInbits)) + baseTargetVolumeInstance.path, volumeSizeInbits, + extraSpecs)) if unboundVolumeInstance is None: exceptionMessage = (_( "Error Creating unbound volume.")) LOG.error(exceptionMessage) + # Remove target volume + self._delete_target_volume_v2(storageConfigService, + baseTargetVolumeInstance, + extraSpecs) raise exception.VolumeBackendAPIException( data=exceptionMessage) - # append the new unbound volume to the - # base target composite volume + # Append the new unbound volume to the + # base target composite volume. baseTargetVolumeInstance = self.utils.find_volume_instance( self.conn, baseVolumeDict, baseVolumeName) - elementCompositionService = ( - self.utils.find_element_composition_service( - self.conn, storageSystemName)) - compositeType = self.utils.get_composite_type( - extraSpecs[COMPOSITETYPE]) - rc, modifiedVolumeDict = ( # @UnusedVariable - self._modify_and_get_composite_volume_instance( - self.conn, - elementCompositionService, - baseTargetVolumeInstance, - unboundVolumeInstance.path, - targetVolumeName, - compositeType)) - if modifiedVolumeDict is None: + try: + elementCompositionService = ( + self.utils.find_element_composition_service( + self.conn, storageSystemName)) + compositeType = self.utils.get_composite_type( + extraSpecs[COMPOSITETYPE]) + _rc, modifiedVolumeDict = ( + self._modify_and_get_composite_volume_instance( + self.conn, + elementCompositionService, + baseTargetVolumeInstance, + unboundVolumeInstance.path, + targetVolumeName, + compositeType, + extraSpecs)) + if modifiedVolumeDict is None: + exceptionMessage = (_( + "Error appending volume %(volumename)s to " + "target base volume.") + % {'volumename': targetVolumeName}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + except Exception: exceptionMessage = (_( - "Error appending volume %(volumename)s to " - "target base volume") - % {'volumename': targetVolumeName}) + "Exception appending meta volume to target volume " + "%(volumename)s.") + % {'volumename': baseVolumeName}) LOG.error(exceptionMessage) + # Remove append volume and target base volume + self._delete_target_volume_v2( + storageConfigService, unboundVolumeInstance, + extraSpecs) + self._delete_target_volume_v2( + storageConfigService, baseTargetVolumeInstance, + extraSpecs) + raise exception.VolumeBackendAPIException( data=exceptionMessage) - LOG.debug("Create V2 replica for meta members of different sizes") + LOG.debug("Create V2 replica for meta members of different sizes.") return self._create_v2_replica_and_delete_clone_relationship( repServiceInstanceName, cloneVolume, sourceVolume, - sourceInstance, baseTargetVolumeInstance, isSnapshot, - extraSpecs) + sourceInstance, baseTargetVolumeInstance, extraSpecs, + isSnapshot) def _create_v2_replica_and_delete_clone_relationship( self, repServiceInstanceName, cloneVolume, sourceVolume, - sourceInstance, targetInstance, isSnapshot, extraSpecs): + sourceInstance, targetInstance, extraSpecs, isSnapshot=False): + """Create a replica and delete the clone relationship. + + :param repServiceInstanceName: the replication service + :param cloneVolume: the clone volume object + :param sourceVolume: the source volume object + :param sourceInstance: the source volume instance + :param targetInstance: the target volume instance + :param extraSpecs: extra specifications + :param isSnapshot: check to see if it is a snapshot + :returns: int -- return code + :returns: dict -- cloneDict + """ sourceName = sourceVolume['name'] cloneName = cloneVolume['name'] - rc, job = self.provision.create_element_replica( - self.conn, repServiceInstanceName, cloneName, sourceName, - sourceInstance, targetInstance) + + try: + rc, job = self.provision.create_element_replica( + self.conn, repServiceInstanceName, cloneName, sourceName, + sourceInstance, targetInstance, extraSpecs) + except Exception: + exceptionMessage = (_( + "Exception during create element replica. " + "Clone name: %(cloneName)s " + "Source name: %(sourceName)s " + "Extra specs: %(extraSpecs)s ") + % {'cloneName': cloneName, + 'sourceName': sourceName, + 'extraSpecs': extraSpecs}) + LOG.error(exceptionMessage) + + if targetInstance is not None: + # Check if the copy session exists. + storageSystem = targetInstance['SystemName'] + syncInstanceName = self.utils.find_sync_sv_by_target( + self.conn, storageSystem, targetInstance, False) + if syncInstanceName is not None: + # Remove the Clone relationship. + rc, job = self.provision.delete_clone_relationship( + self.conn, repServiceInstanceName, syncInstanceName, + extraSpecs, True) + storageConfigService = ( + self.utils.find_storage_configuration_service( + self.conn, storageSystem)) + self._delete_target_volume_v2( + storageConfigService, targetInstance, extraSpecs) + + raise exception.VolumeBackendAPIException( + data=exceptionMessage) cloneDict = self.provision.get_volume_dict_from_job( self.conn, job['Job']) @@ -3404,25 +3728,29 @@ def _create_v2_replica_and_delete_clone_relationship( if fastPolicyName is not None: storageSystemName = sourceInstance['SystemName'] self._add_clone_to_default_storage_group( - fastPolicyName, storageSystemName, cloneDict, cloneName) - LOG.info("Snapshot creation %(cloneName)s completed. " - "Source Volume: %(sourceName)s " - % {'cloneName': cloneName, - 'sourceName': sourceName}) + fastPolicyName, storageSystemName, cloneDict, cloneName, + extraSpecs) + LOG.info(_LI("Snapshot creation %(cloneName)s completed. " + "Source Volume: %(sourceName)s."), + {'cloneName': cloneName, + 'sourceName': sourceName}) return rc, cloneDict cloneVolume['provider_location'] = six.text_type(cloneDict) syncInstanceName, storageSystemName = ( - self._find_storage_sync_sv_sv(cloneVolume, sourceVolume)) + self._find_storage_sync_sv_sv(cloneVolume, sourceVolume, + extraSpecs)) - # Remove the Clone relationship so it can be used as a regular lun - # 8 - Detach operation + # Remove the Clone relationship so it can be used as a regular lun. + # 8 - Detach operation. rc, job = self.provision.delete_clone_relationship( - self.conn, repServiceInstanceName, syncInstanceName) + self.conn, repServiceInstanceName, syncInstanceName, + extraSpecs) if fastPolicyName is not None: self._add_clone_to_default_storage_group( - fastPolicyName, storageSystemName, cloneDict, cloneName) + fastPolicyName, storageSystemName, cloneDict, cloneName, + extraSpecs) return rc, cloneDict @@ -3433,120 +3761,179 @@ def get_target_wwns_from_masking_view( :param storageSystem: the storage system name :param volume: volume to be attached :param connector: the connector dict - :returns: targetWwns, the target WWN list + :returns: list -- the target WWN list """ targetWwns = [] mvInstanceName = self.get_masking_view_by_volume(volume, connector) if mvInstanceName is not None: targetWwns = self.masking.get_target_wwns( self.conn, mvInstanceName) - LOG.info("Target wwns in masking view %(maskingView)s: " - "%(targetWwns)s" - % {'maskingView': mvInstanceName, - 'targetWwns': str(targetWwns)}) + LOG.info(_LI("Target wwns in masking view %(maskingView)s: " + "%(targetWwns)s."), + {'maskingView': mvInstanceName, + 'targetWwns': six.text_type(targetWwns)}) return targetWwns def get_port_group_from_masking_view(self, maskingViewInstanceName): + """Get the port groups in a masking view. + + :param maskingViewInstanceName: masking view instance name + :returns: portGroupInstanceName + """ return self.masking.get_port_group_from_masking_view( self.conn, maskingViewInstanceName) + def get_initiator_group_from_masking_view(self, maskingViewInstanceName): + """Get the initiator group in a masking view. + + :param maskingViewInstanceName: masking view instance name + :returns: initiatorGroupInstanceName + """ + return self.masking.get_initiator_group_from_masking_view( + self.conn, maskingViewInstanceName) + def get_masking_view_by_volume(self, volume, connector): - """Given volume, retrieve the masking view instance name + """Given volume, retrieve the masking view instance name. :param volume: the volume :param connector: the connector object - :returns maskingviewInstanceName + :returns: maskingviewInstanceName """ - LOG.debug("Finding Masking View for volume %(volume)s", + LOG.debug("Finding Masking View for volume %(volume)s.", {'volume': volume}) volumeInstance = self._find_lun(volume) return self.masking.get_masking_view_by_volume( self.conn, volumeInstance, connector) def get_masking_views_by_port_group(self, portGroupInstanceName): - """Given port group, retrieve the masking view instance name + """Given port group, retrieve the masking view instance name. - :param : the volume - :param mvInstanceName: masking view instance name - :returns: maksingViewInstanceNames + :param portGroupInstanceName: port group instance name + :returns: list -- maskingViewInstanceNames """ - LOG.debug("Finding Masking Views for port group %(pg)s", + LOG.debug("Finding Masking Views for port group %(pg)s.", {'pg': portGroupInstanceName}) return self.masking.get_masking_views_by_port_group( self.conn, portGroupInstanceName) + def get_masking_views_by_initiator_group( + self, initiatorGroupInstanceName): + """Given initiator group, retrieve the masking view instance name. + + :param initiatorGroupInstanceName: initiator group instance name + :returns: list -- maskingViewInstanceNames + """ + LOG.debug("Finding Masking Views for initiator group %(ig)s.", + {'ig': initiatorGroupInstanceName}) + return self.masking.get_masking_views_by_initiator_group( + self.conn, initiatorGroupInstanceName) + def _create_replica_v3( self, repServiceInstanceName, cloneVolume, - sourceVolume, sourceInstance, isSnapshot): - """VG3R specific function, create replica for source volume, + sourceVolume, sourceInstance, isSnapshot, extraSpecs): + """Create a replica. + + V3 specific function, create replica for source volume, including clone and snapshot. :param repServiceInstanceName: the replication service :param cloneVolume: the clone volume object + :param sourceVolume: the source volume object :param sourceInstance: the device ID of the volume - :param isSnapshot: check to see if it is a snapshot - :returns: rc + :param isSnapshot: boolean -- check to see if it is a snapshot + :param extraSpecs: extra specifications + :returns: int -- return code + :returns: dict -- cloneDict """ cloneName = cloneVolume['name'] - syncType = self.utils.get_num(8, '16') # default syncType 8: clone - - # create target volume - extraSpecs = self._initial_setup(cloneVolume) - - numOfBlocks = sourceInstance['NumberOfBlocks'] - blockSize = sourceInstance['BlockSize'] - volumeSizeInbits = numOfBlocks * blockSize - - volume = {'size': - int(self.utils.convert_bits_to_gbs(volumeSizeInbits))} - rc, volumeDict, storageSystemName = ( # @UnusedVariable - self._create_v3_volume( - volume, extraSpecs, cloneName, volumeSizeInbits)) - targetInstance = self.utils.find_volume_instance( - self.conn, volumeDict, cloneName) - LOG.debug("Create replica target volume " - "source volume: %(sourceVol)s" - "target volume %(targetVol)s", - {'sourceVol': sourceInstance.path, - 'targetVol': targetInstance.path}) + # SyncType 7: snap, VG3R default snapshot is snapVx. + syncType = self.utils.get_num(SNAPVX, '16') + # Operation 9: Dissolve for snapVx. + operation = self.utils.get_num(DISSOLVE_SNAPVX, '16') + rsdInstance = None + targetInstance = None + copyState = self.utils.get_num(4, '16') if isSnapshot: - # syncType 7: snap, VG3R default snapshot is snapVx - syncType = self.utils.get_num(7, '16') + rsdInstance = self.utils.set_target_element_supplier_in_rsd( + self.conn, repServiceInstanceName, SNAPVX_REPLICATION_TYPE, + CREATE_NEW_TARGET, extraSpecs) + else: + targetInstance = self._create_duplicate_volume( + sourceInstance, cloneName, extraSpecs) + + try: + _rc, job = ( + self.provisionv3.create_element_replica( + self.conn, repServiceInstanceName, cloneName, syncType, + sourceInstance, extraSpecs, copyState, targetInstance, + rsdInstance,)) + except Exception: + LOG.warning(_LW( + "Clone failed on V3. Cleaning up the target volume. " + "Clone name: %(cloneName)s "), + {'cloneName': cloneName}) + # Check if the copy session exists. + if targetInstance: + self._cleanup_target( + repServiceInstanceName, targetInstance, extraSpecs) + # Re-throw the exception. + raise - rc, job = ( # @UnusedVariable - self.provisionv3.create_element_replica( - self.conn, repServiceInstanceName, cloneName, syncType, - sourceInstance, targetInstance)) cloneDict = self.provisionv3.get_volume_dict_from_job( self.conn, job['Job']) + targetVolumeInstance = ( + self.provisionv3.get_volume_from_job(self.conn, job['Job'])) + LOG.info(_LI("The target instance device id is: %(deviceid)s."), + {'deviceid': targetVolumeInstance['DeviceID']}) cloneVolume['provider_location'] = six.text_type(cloneDict) - syncInstanceName, storageSystemName = ( # @UnusedVariable - self._find_storage_sync_sv_sv(cloneVolume, sourceVolume, True)) - - # detach/dissolve the clone/snap relationship - # 8 - Detach operation - # 9 - Dissolve operation - if isSnapshot: - # operation 7: dissolve for snapVx - operation = self.utils.get_num(9, '16') - else: - # operation 8: detach for clone - operation = self.utils.get_num(8, '16') + syncInstanceName, _storageSystem = ( + self._find_storage_sync_sv_sv(cloneVolume, sourceVolume, + extraSpecs, True)) rc, job = self.provisionv3.break_replication_relationship( self.conn, repServiceInstanceName, syncInstanceName, - operation) + operation, extraSpecs) return rc, cloneDict + def _cleanup_target( + self, repServiceInstanceName, targetInstance, extraSpecs): + """cleanup target after exception + + :param repServiceInstanceName: the replication service + :param targetInstance: the target instance + :param extraSpecs: extra specifications + """ + storageSystem = targetInstance['SystemName'] + syncInstanceName = self.utils.find_sync_sv_by_target( + self.conn, storageSystem, targetInstance, False) + if syncInstanceName is not None: + # Break the clone relationship. + self.provisionv3.break_replication_relationship( + self.conn, repServiceInstanceName, syncInstanceName, + DISSOLVE_SNAPVX, extraSpecs, True) + storageConfigService = ( + self.utils.find_storage_configuration_service( + self.conn, storageSystem)) + deviceId = targetInstance['DeviceID'] + volumeName = targetInstance['Name'] + self._delete_from_pool_v3( + storageConfigService, targetInstance, volumeName, + deviceId, extraSpecs) + def _delete_cg_and_members( - self, storageSystem, extraSpecs, cgName, modelUpdate, volumes): - """Helper function to delete a consistency group and its member volumes + self, storageSystem, cgName, modelUpdate, volumes, extraSpecs): + """Helper function to delete a consistencygroup and its member volumes. :param storageSystem: storage system - :param repServiceInstanceName: the replication service - :param cgInstanceName: consistency group instance name + :param cgName: consistency group name + :param modelUpdate: dict -- the model update dict + :param volumes: the list of member volumes + :param extraSpecs: extra specifications + :returns: dict -- modelUpdate + :returns: list -- the updated list of member volumes + :raises: VolumeBackendAPIException """ replicationService = self.utils.find_replication_service( self.conn, storageSystem) @@ -3556,15 +3943,18 @@ def _delete_cg_and_members( self.conn, storageSystem)) cgInstanceName = self._find_consistency_group( replicationService, cgName) + if cgInstanceName is None: - exception_message = (_("Cannot find CG group %s."), cgName) + exception_message = (_("Cannot find CG group %s.") % cgName) raise exception.VolumeBackendAPIException( data=exception_message) + memberInstanceNames = self._get_members_of_replication_group( cgInstanceName) self.provision.delete_consistency_group( - self.conn, replicationService, cgInstanceName, cgName) + self.conn, replicationService, cgInstanceName, cgName, + extraSpecs) if memberInstanceNames: try: @@ -3572,26 +3962,542 @@ def _delete_cg_and_members( self.utils.find_controller_configuration_service( self.conn, storageSystem)) for memberInstanceName in memberInstanceNames: - self._pre_check_for_deletion( + self._remove_device_from_storage_group( controllerConfigurationService, - memberInstanceName, 'Member Volume') + memberInstanceName, 'Member Volume', extraSpecs) LOG.debug("Deleting CG members. CG: %(cg)s " - "%(numVols)lu member volumes: %(memVols)s", + "%(numVols)lu member volumes: %(memVols)s.", {'cg': cgInstanceName, 'numVols': len(memberInstanceNames), 'memVols': memberInstanceNames}) if extraSpecs[ISV3]: self.provisionv3.delete_volume_from_pool( self.conn, storageConfigservice, - memberInstanceNames, None) + memberInstanceNames, None, extraSpecs) else: self.provision.delete_volume_from_pool( self.conn, storageConfigservice, - memberInstanceNames, None) + memberInstanceNames, None, extraSpecs) for volumeRef in volumes: - volumeRef['status'] = 'deleted' + volumeRef['status'] = 'deleted' except Exception: for volumeRef in volumes: volumeRef['status'] = 'error_deleting' modelUpdate['status'] = 'error_deleting' return modelUpdate, volumes + + def _delete_target_volume_v2( + self, storageConfigService, targetVolumeInstance, extraSpecs): + """Helper function to delete the clone target volume instance. + + :param storageConfigService: storage configuration service instance + :param targetVolumeInstance: clone target volume instance + :param extraSpecs: extra specifications + """ + deviceId = targetVolumeInstance['DeviceID'] + volumeName = targetVolumeInstance['Name'] + rc = self._delete_from_pool(storageConfigService, + targetVolumeInstance, + volumeName, deviceId, + extraSpecs[FASTPOLICY], + extraSpecs) + return rc + + def _validate_pool(self, volume): + """Get the pool from volume['host']. + + There may be backward compatibiliy concerns, so putting in a + check to see if a version has been added to provider_location. + If it has, we know we are at the current version, if not, we + assume it was created pre 'Pool Aware Scheduler' feature. + + :param volume: the volume Object + :returns: string -- pool + :raises: VolumeBackendAPIException + """ + pool = None + # Volume is None in CG ops. + if volume is None: + return pool + + # This check is for all operations except a create. + # On a create provider_location is None + try: + if volume['provider_location']: + version = self._get_version_from_provider_location( + volume['provider_location']) + if not version: + return pool + except KeyError: + return pool + try: + pool = volume_utils.extract_host(volume['host'], 'pool') + if pool: + LOG.debug("Pool from volume['host'] is %(pool)s.", + {'pool': pool}) + else: + exceptionMessage = (_( + "Pool from volume['host'] %(host)s not found.") + % {'host': volume['host']}) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + except Exception as ex: + exceptionMessage = (_( + "Pool from volume['host'] failed with: %(ex)s.") + % {'ex': ex}) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + return pool + + def _get_version_from_provider_location(self, loc): + """Get the version from the provider location. + + :param loc: the provider_location dict + :returns: version or None + """ + version = None + try: + if isinstance(loc, six.string_types): + name = ast.literal_eval(loc) + version = name['version'] + except KeyError: + pass + return version + + def manage_existing(self, volume, external_ref): + """Manages an existing VMAX Volume (import to Cinder). + + Renames the existing volume to match the expected name for the volume. + Also need to consider things like QoS, Emulation, account/tenant. + + :param volume: the volume object including the volume_type_id + :param external_ref: reference to the existing volume + :returns: dict -- model_update + :raises: VolumeBackendAPIException + """ + extraSpecs = self._initial_setup(volume) + self.conn = self._get_ecom_connection() + arrayName, deviceId = self.utils.get_array_and_device_id( + volume, external_ref) + + self.utils.check_volume_no_fast(extraSpecs) + + volumeInstanceName = ( + self.utils.find_volume_by_device_id_on_array( + arrayName, deviceId)) + + self.utils.check_volume_not_in_masking_view( + self.conn, volumeInstanceName, deviceId) + + cinderPoolInstanceName, storageSystemName = ( + self._get_pool_and_storage_system(extraSpecs)) + + self.utils.check_volume_not_replication_source( + self.conn, storageSystemName, deviceId) + + self.utils.check_is_volume_in_cinder_managed_pool( + self.conn, volumeInstanceName, cinderPoolInstanceName, + deviceId) + + volumeId = volume['name'] + volumeElementName = self.utils.get_volume_element_name(volumeId) + LOG.debug("Rename volume %(vol)s to %(volumeId)s.", + {'vol': volumeInstanceName, + 'volumeId': volumeElementName}) + + volumeInstance = self.utils.rename_volume(self.conn, + volumeInstanceName, + volumeElementName) + keys = {} + volpath = volumeInstance.path + keys['CreationClassName'] = volpath['CreationClassName'] + keys['SystemName'] = volpath['SystemName'] + keys['DeviceID'] = volpath['DeviceID'] + keys['SystemCreationClassName'] = volpath['SystemCreationClassName'] + + provider_location = {} + provider_location['classname'] = volpath['CreationClassName'] + provider_location['keybindings'] = keys + + model_update = {} + + volumeDisplayName = volume['display_name'] + model_update.update( + {'display_name': volumeDisplayName}) + model_update.update( + {'provider_location': six.text_type(provider_location)}) + return model_update + + def manage_existing_get_size(self, volume, external_ref): + """Return size of an existing VMAX volume to manage_existing. + + :param self: reference to class + :param volume: the volume object including the volume_type_id + :param external_ref: reference to the existing volume + :returns: size of the volume in GB + """ + LOG.debug("Volume in manage_existing_get_size: %(volume)s.", + {'volume': volume}) + arrayName, deviceId = self.utils.get_array_and_device_id(volume, + external_ref) + volumeInstanceName = ( + self.utils.find_volume_by_device_id_on_array(arrayName, deviceId)) + + try: + volumeInstance = self.conn.GetInstance(volumeInstanceName) + byteSize = self.utils.get_volume_size(self.conn, volumeInstance) + fByteSize = float(byteSize) + gbSize = int(fByteSize / units.Gi) + LOG.debug("Size of volume %(deviceID)s is %(volumeSize)s GB", + {'deviceID': deviceId, + 'volumeSize': gbSize}) + except Exception: + exceptionMessage = (_("Volume %(deviceID)s not found.") + % {'deviceID': deviceId}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException(data=exceptionMessage) + + return gbSize + + def unmanage(self, volume): + """Export VMAX volume from Cinder. + + Leave the volume intact on the backend array. + + :param volume: the volume object + :raises: VolumeBackendAPIException + """ + volumeName = volume['name'] + volumeId = volume['id'] + LOG.debug("Unmanage volume %(name)s, id=%(id)s", + {'name': volumeName, + 'id': volumeId}) + self._initial_setup(volume) + self.conn = self._get_ecom_connection() + volumeInstance = self._find_lun(volume) + if volumeInstance is None: + exceptionMessage = (_("Cannot find Volume: %(id)s. " + "unmanage operation. Exiting...") + % {'id': volumeId}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException(data=exceptionMessage) + + # Rename the volume to volumeId, thus remove the 'OS-' prefix. + self.utils.rename_volume(self.conn, volumeInstance, volumeId) + + def update_consistencygroup(self, group, add_volumes, + remove_volumes): + """Updates LUNs in consistency group. + + :param group: storage configuration service instance + :param add_volumes: the volumes uuids you want to add to the CG + :param remove_volumes: the volumes uuids you want to remove from + the CG + """ + LOG.info(_LI("Update Consistency Group: %(group)s. " + "This adds and/or removes volumes from a CG."), + {'group': group['id']}) + + modelUpdate = {'status': fields.ConsistencyGroupStatus.AVAILABLE} + volumeTypeId = group['volume_type_id'].replace(",", "") + + cg_name = self.utils.truncate_string(group['id'], 8) + + extraSpecs = self._initial_setup(None, volumeTypeId) + + _poolInstanceName, storageSystem = ( + self._get_pool_and_storage_system(extraSpecs)) + add_vols = [vol for vol in add_volumes] if add_volumes else [] + add_instance_names = self._get_volume_instance_names(add_vols) + remove_vols = [vol for vol in remove_volumes] if remove_volumes else [] + remove_instance_names = self._get_volume_instance_names(remove_vols) + self.conn = self._get_ecom_connection() + + try: + replicationService = self.utils.find_replication_service( + self.conn, storageSystem) + cgInstanceName = ( + self._find_consistency_group(replicationService, cg_name)) + if cgInstanceName is None: + raise exception.ConsistencyGroupNotFound( + consistencygroup_id=cg_name) + # Add volume(s) to a consistency group + if add_instance_names: + self.provision.add_volume_to_cg( + self.conn, replicationService, cgInstanceName, + add_instance_names, cg_name, None, + extraSpecs) + # Remove volume(s) from a consistency group + if remove_instance_names: + self.provision.remove_volume_from_cg( + self.conn, replicationService, cgInstanceName, + remove_instance_names, cg_name, None, + extraSpecs) + except exception.ConsistencyGroupNotFound: + raise + except Exception as ex: + LOG.error(_LE("Exception: %(ex)s"), {'ex': ex}) + exceptionMessage = (_("Failed to update consistency group:" + " %(cgName)s.") + % {'cgName': cg_name}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException(data=exceptionMessage) + + return modelUpdate, None, None + + def _get_volume_instance_names(self, volumes): + """Get volume instance names from volume. + + :param volumes: volume objects + :returns: volume instance names + """ + volumeInstanceNames = [] + for volume in volumes: + volumeInstance = self._find_lun(volume) + if volumeInstance is None: + LOG.error(_LE("Volume %(name)s not found on the array."), + {'name': volume['name']}) + else: + volumeInstanceNames.append(volumeInstance.path) + return volumeInstanceNames + + def create_consistencygroup_from_src(self, context, group, volumes, + cgsnapshot, snapshots, source_cg, + source_vols): + """Creates the consistency group from source. + + Currently the source can only be a cgsnapshot. + + :param context: the context + :param group: the consistency group object to be created + :param volumes: volumes in the consistency group + :param cgsnapshot: the source consistency group snapshot + :param snapshots: snapshots of the source volumes + :param source_cg: the source consistency group + :param source_vols: the source vols + :returns: model_update, volumes_model_update + model_update is a dictionary of cg status + volumes_model_update is a list of dictionaries of volume + update + """ + LOG.debug("Enter EMCVMAXCommon::create_consistencygroup_from_src. " + "Group to be created: %(cgId)s, " + "Source snapshot: %(cgSnapshot)s.", + {'cgId': group['id'], + 'cgSnapshot': cgsnapshot['consistencygroup_id']}) + + volumeTypeId = group['volume_type_id'].replace(",", "") + extraSpecs = self._initial_setup(None, volumeTypeId) + + self.create_consistencygroup(context, group) + targetCgName = self.utils.truncate_string(group['id'], TRUNCATE_8) + + if not snapshots: + exceptionMessage = (_("No source snapshots provided to create " + "consistency group %s.") % targetCgName) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + + modelUpdate = {'status': fields.ConsistencyGroupStatus.AVAILABLE} + + _poolInstanceName, storageSystem = ( + self._get_pool_and_storage_system(extraSpecs)) + try: + replicationService = self.utils.find_replication_service( + self.conn, storageSystem) + if replicationService is None: + exceptionMessage = (_( + "Cannot find replication service on system %s.") % + storageSystem) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + targetCgInstanceName = self._find_consistency_group( + replicationService, targetCgName) + LOG.debug("Create CG %(targetCg)s from snapshot.", + {'targetCg': targetCgInstanceName}) + + for volume, snapshot in zip(volumes, snapshots): + volumeSizeInbits = int(self.utils.convert_gb_to_bits( + snapshot['volume_size'])) + targetVolumeName = 'targetVol' + volume = {'size': int(self.utils.convert_bits_to_gbs( + volumeSizeInbits))} + if extraSpecs[ISV3]: + _rc, volumeDict, _storageSystemName = ( + self._create_v3_volume( + volume, targetVolumeName, volumeSizeInbits, + extraSpecs)) + else: + _rc, volumeDict, _storageSystemName = ( + self._create_composite_volume( + volume, targetVolumeName, volumeSizeInbits, + extraSpecs)) + targetVolumeInstance = self.utils.find_volume_instance( + self.conn, volumeDict, targetVolumeName) + LOG.debug("Create target volume for member snapshot. " + "Source snapshot: %(snapshot)s, " + "Target volume: %(targetVol)s.", + {'snapshot': snapshot['id'], + 'targetVol': targetVolumeInstance.path}) + + self.provision.add_volume_to_cg(self.conn, + replicationService, + targetCgInstanceName, + targetVolumeInstance.path, + targetCgName, + targetVolumeName, + extraSpecs) + + sourceCgName = self.utils.truncate_string(cgsnapshot['id'], + TRUNCATE_8) + sourceCgInstanceName = self._find_consistency_group( + replicationService, sourceCgName) + if sourceCgInstanceName is None: + exceptionMessage = (_("Cannot find source CG instance. " + "consistencygroup_id: %s.") % + cgsnapshot['consistencygroup_id']) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + relationName = self.utils.truncate_string(group['id'], TRUNCATE_5) + if extraSpecs[ISV3]: + self.provisionv3.create_group_replica( + self.conn, replicationService, sourceCgInstanceName, + targetCgInstanceName, relationName, extraSpecs) + else: + self.provision.create_group_replica( + self.conn, replicationService, sourceCgInstanceName, + targetCgInstanceName, relationName, extraSpecs) + # Break the replica group relationship. + rgSyncInstanceName = self.utils.find_group_sync_rg_by_target( + self.conn, storageSystem, targetCgInstanceName, extraSpecs, + True) + + if rgSyncInstanceName is not None: + if extraSpecs[ISV3]: + # Operation 9: dissolve for snapVx + operation = self.utils.get_num(9, '16') + self.provisionv3.break_replication_relationship( + self.conn, replicationService, rgSyncInstanceName, + operation, extraSpecs) + else: + self.provision.delete_clone_relationship( + self.conn, replicationService, + rgSyncInstanceName, extraSpecs) + except Exception: + cgSnapshotId = cgsnapshot['consistencygroup_id'] + exceptionMessage = (_("Failed to create CG %(cgName)s " + "from snapshot %(cgSnapshot)s.") + % {'cgName': targetCgName, + 'cgSnapshot': cgSnapshotId}) + LOG.exception(exceptionMessage) + raise exception.VolumeBackendAPIException(data=exceptionMessage) + volumes_model_update = self.utils.get_volume_model_updates( + context, volumes, group['id'], modelUpdate['status']) + + return modelUpdate, volumes_model_update + + def _find_ip_protocol_endpoints(self, conn, storageSystemName, + portgroupname): + """Find the IP protocol endpoint for ISCSI. + + :param storageSystemName: the system name + :param portgroupname: the portgroup name + :returns: foundIpAddresses + """ + LOG.debug("The portgroup name for iscsiadm is %(pg)s", + {'pg': portgroupname}) + foundipaddresses = [] + configservice = ( + self.utils.find_controller_configuration_service( + conn, storageSystemName)) + portgroupinstancename = ( + self.masking.find_port_group(conn, configservice, portgroupname)) + iscsiendpointinstancenames = ( + self.utils.get_iscsi_protocol_endpoints( + conn, portgroupinstancename)) + + for iscsiendpointinstancename in iscsiendpointinstancenames: + tcpendpointinstancenames = ( + self.utils.get_tcp_protocol_endpoints( + conn, iscsiendpointinstancename)) + for tcpendpointinstancename in tcpendpointinstancenames: + ipendpointinstancenames = ( + self.utils.get_ip_protocol_endpoints( + conn, tcpendpointinstancename)) + endpoint = {} + for ipendpointinstancename in ipendpointinstancenames: + endpoint = self.get_ip_and_iqn(conn, endpoint, + ipendpointinstancename) + if bool(endpoint): + foundipaddresses.append(endpoint) + return foundipaddresses + + def _extend_v3_volume(self, volumeInstance, volumeName, newSize, + extraSpecs): + """Extends a VMAX3 volume. + + :param volumeInstance: volume instance + :param volumeName: volume name + :param newSize: new size the volume will be increased to + :param extraSpecs: extra specifications + :returns: int -- return code + :returns: volumeDict + """ + new_size_in_bits = int(self.utils.convert_gb_to_bits(newSize)) + storageConfigService = self.utils.find_storage_configuration_service( + self.conn, volumeInstance['SystemName']) + volumeDict, rc = self.provisionv3.extend_volume_in_SG( + self.conn, storageConfigService, volumeInstance.path, + volumeName, new_size_in_bits, extraSpecs) + + return rc, volumeDict + + def _create_duplicate_volume( + self, sourceInstance, cloneName, extraSpecs): + """Create a volume in the same dimensions of the source volume. + + :param sourceInstance: the source volume instance + :param cloneName: the user supplied snap name + :param extraSpecs: additional info + :returns: targetInstance + """ + numOfBlocks = sourceInstance['NumberOfBlocks'] + blockSize = sourceInstance['BlockSize'] + volumeSizeInbits = numOfBlocks * blockSize + + volume = {'size': + int(self.utils.convert_bits_to_gbs(volumeSizeInbits))} + _rc, volumeDict, _storageSystemName = ( + self._create_v3_volume( + volume, cloneName, volumeSizeInbits, extraSpecs)) + targetInstance = self.utils.find_volume_instance( + self.conn, volumeDict, cloneName) + LOG.debug("Create replica target volume " + "Source Volume: %(sourceVol)s, " + "Target Volume: %(targetVol)s.", + {'sourceVol': sourceInstance.path, + 'targetVol': targetInstance.path}) + return targetInstance + + def get_ip_and_iqn(self, conn, endpoint, ipendpointinstancename): + """Get ip and iqn from the endpoint. + + :param conn: ecom connection + :param endpoint: end point + :param ipendpointinstancename: ip endpoint + :returns: endpoint + """ + if ('iSCSIProtocolEndpoint' in six.text_type( + ipendpointinstancename['CreationClassName'])): + iqn = self.utils.get_iqn(conn, ipendpointinstancename) + if iqn: + endpoint['iqn'] = iqn + elif ('IPProtocolEndpoint' in six.text_type( + ipendpointinstancename['CreationClassName'])): + ipaddress = ( + self.utils.get_iscsi_ip_address( + conn, ipendpointinstancename)) + if ipaddress: + endpoint['ip'] = ipaddress + + return endpoint diff --git a/emc_vmax_fast.py b/emc_vmax_fast.py index e9be9d3..3f88a53 100644 --- a/emc_vmax_fast.py +++ b/emc_vmax_fast.py @@ -1,4 +1,4 @@ -# Copyright (c) 2012 - 2014 EMC Corporation. +# Copyright (c) 2012 - 2015 EMC Corporation. # All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); you may @@ -12,9 +12,11 @@ # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. + +from oslo_log import log as logging + from cinder import exception from cinder.i18n import _, _LE, _LI, _LW -from cinder.openstack.common import log as logging from cinder.volume.drivers.emc import emc_vmax_provision from cinder.volume.drivers.emc import emc_vmax_utils @@ -38,8 +40,9 @@ def __init__(self, prtcl): def _check_if_fast_supported(self, conn, storageSystemInstanceName): """Check to see if fast is supported on the array. - :param conn: the connection to the ecom server + :param conn: the ecom connection :param storageSystemInstanceName: the storage system Instance name + :returns: boolean -- isTieringPolicySupported """ tierPolicyServiceInstanceName = self.utils.get_tier_policy_service( @@ -63,8 +66,8 @@ def is_tiering_policy_enabled(self, conn, tierPolicyServiceInstanceName): :param conn: the connection information to the ecom server :param tierPolicyServiceInstanceName: the tier policy service - instance name - :returns: foundIsSupportsTieringPolicies - True/False + instance name + :returns: boolean -- foundIsSupportsTieringPolicies """ foundIsSupportsTieringPolicies = None tierPolicyCapabilityInstanceNames = conn.AssociatorNames( @@ -76,7 +79,7 @@ def is_tiering_policy_enabled(self, conn, tierPolicyServiceInstanceName): tierPolicyCapabilityInstance = conn.GetInstance( tierPolicyCapabilityInstanceName, LocalOnly=False) propertiesList = (tierPolicyCapabilityInstance - .properties.items()) # ['SupportsTieringPolicies'] + .properties.items()) for properties in propertiesList: if properties[0] == 'SupportsTieringPolicies': cimProperties = properties[1] @@ -85,7 +88,7 @@ def is_tiering_policy_enabled(self, conn, tierPolicyServiceInstanceName): if foundIsSupportsTieringPolicies is None: LOG.error(_LE("Cannot determine if Tiering Policies " - "are supported")) + "are supported.")) return foundIsSupportsTieringPolicies @@ -103,7 +106,7 @@ def get_and_verify_default_storage_group( :param volumeInstanceName: the volume instance name :param volumeName: the volume name (String) :param fastPolicyName: the fast policy name (String) - :returns: foundDefaultStorageGroupInstanceName + :returns: foundDefaultStorageGroupInstanceName, defaultSgName """ foundDefaultStorageGroupInstanceName = None storageSystemInstanceName = self.utils.find_storage_system( @@ -111,21 +114,22 @@ def get_and_verify_default_storage_group( if not self._check_if_fast_supported(conn, storageSystemInstanceName): LOG.error(_LE( - "FAST is not supported on this array ")) + "FAST is not supported on this array.")) raise + defaultSgName = self.format_default_sg_string(fastPolicyName) assocStorageGroupInstanceName = ( - self.utils.get_storage_group_from_volume(conn, volumeInstanceName)) - defaultSgGroupName = (DEFAULT_SG_PREFIX + fastPolicyName + - DEFAULT_SG_POSTFIX) + self.utils.get_storage_group_from_volume(conn, volumeInstanceName, + defaultSgName)) + defaultStorageGroupInstanceName = ( self.utils.find_storage_masking_group(conn, controllerConfigService, - defaultSgGroupName)) + defaultSgName)) if defaultStorageGroupInstanceName is None: LOG.error(_LE( "Unable to find default storage group " - "for FAST policy : %(fastPolicyName)s "), + "for FAST policy : %(fastPolicyName)s."), {'fastPolicyName': fastPolicyName}) raise @@ -133,16 +137,27 @@ def get_and_verify_default_storage_group( foundDefaultStorageGroupInstanceName = ( assocStorageGroupInstanceName) else: - LOG.warn(_LW( + LOG.warning(_LW( "Volume: %(volumeName)s Does not belong " - "to storage storage group %(defaultSgGroupName)s. "), + "to storage group %(defaultSgName)s."), {'volumeName': volumeName, - 'defaultSgGroupName': defaultSgGroupName}) - return foundDefaultStorageGroupInstanceName + 'defaultSgName': defaultSgName}) + return foundDefaultStorageGroupInstanceName, defaultSgName + + def format_default_sg_string(self, fastPolicyName): + """Format the default storage group name + + :param fastPolicyName: the fast policy name + :returns: defaultSgName + """ + return ("%(prefix)s%(fastPolicyName)s%(postfix)s" + % {'prefix': DEFAULT_SG_PREFIX, + 'fastPolicyName': fastPolicyName, + 'postfix': DEFAULT_SG_POSTFIX}) def add_volume_to_default_storage_group_for_fast_policy( self, conn, controllerConfigService, volumeInstance, - volumeName, fastPolicyName): + volumeName, fastPolicyName, extraSpecs): """Add a volume to the default storage group for FAST policy. The storage group must pre-exist. Once added to the storage group, @@ -153,33 +168,33 @@ def add_volume_to_default_storage_group_for_fast_policy( :param volumeInstance: the volume instance :param volumeName: the volume name (String) :param fastPolicyName: the fast policy name (String) + :param extraSpecs: additional info :returns: assocStorageGroupInstanceName - the storage group - associated with the volume + associated with the volume """ failedRet = None - defaultSgGroupName = (DEFAULT_SG_PREFIX + fastPolicyName + - DEFAULT_SG_POSTFIX) + defaultSgName = self.format_default_sg_string(fastPolicyName) storageGroupInstanceName = self.utils.find_storage_masking_group( - conn, controllerConfigService, defaultSgGroupName) + conn, controllerConfigService, defaultSgName) if storageGroupInstanceName is None: LOG.error(_LE( - "Unable to create default storage group for" - " FAST policy : %(fastPolicyName)s "), - {'fastPolicyName': fastPolicyName}) + "Unable to get default storage group %(defaultSgName)s."), + {'defaultSgName': defaultSgName}) return failedRet self.provision.add_members_to_masking_group( conn, controllerConfigService, storageGroupInstanceName, - volumeInstance.path, volumeName) - # check to see if the volume is in the storage group + volumeInstance.path, volumeName, extraSpecs) + # Check to see if the volume is in the storage group. assocStorageGroupInstanceName = ( self.utils.get_storage_group_from_volume(conn, - volumeInstance.path)) + volumeInstance.path, + defaultSgName)) return assocStorageGroupInstanceName def _create_default_storage_group(self, conn, controllerConfigService, fastPolicyName, storageGroupName, - volumeInstance): + volumeInstance, extraSpecs): """Create a first volume for the storage group. This is necessary because you cannot remove a volume if it is the @@ -191,27 +206,28 @@ def _create_default_storage_group(self, conn, controllerConfigService, :param fastPolicyName: the fast policy name (String) :param storageGroupName: the storage group name (String) :param volumeInstance: the volume instance + :param extraSpecs: additional info :returns: defaultstorageGroupInstanceName - instance name of the - default storage group + default storage group """ failedRet = None firstVolumeInstance = self._create_volume_for_default_volume_group( - conn, controllerConfigService, volumeInstance.path) + conn, controllerConfigService, volumeInstance.path, extraSpecs) if firstVolumeInstance is None: LOG.error(_LE( - "Failed to create a first volume for storage" - " group : %(storageGroupName)s "), + "Failed to create a first volume for storage " + "group : %(storageGroupName)s."), {'storageGroupName': storageGroupName}) return failedRet defaultStorageGroupInstanceName = ( self.provision.create_and_get_storage_group( conn, controllerConfigService, storageGroupName, - firstVolumeInstance.path)) + firstVolumeInstance.path, extraSpecs)) if defaultStorageGroupInstanceName is None: LOG.error(_LE( "Failed to create default storage group for " - "FAST policy : %(fastPolicyName)s "), + "FAST policy : %(fastPolicyName)s."), {'fastPolicyName': fastPolicyName}) return failedRet @@ -220,35 +236,37 @@ def _create_default_storage_group(self, conn, controllerConfigService, tierPolicyServiceInstanceName = self.utils.get_tier_policy_service( conn, storageSystemInstanceName) - # get the fast policy instance name + # Get the fast policy instance name. tierPolicyRuleInstanceName = self._get_service_level_tier_policy( conn, tierPolicyServiceInstanceName, fastPolicyName) if tierPolicyRuleInstanceName is None: LOG.error(_LE( "Unable to get policy rule for fast policy: " - "%(fastPolicyName)s "), + "%(fastPolicyName)s."), {'fastPolicyName': fastPolicyName}) return failedRet - # now associate it with a FAST policy + # Now associate it with a FAST policy. self.add_storage_group_to_tier_policy_rule( conn, tierPolicyServiceInstanceName, defaultStorageGroupInstanceName, tierPolicyRuleInstanceName, - storageGroupName, fastPolicyName) + storageGroupName, fastPolicyName, extraSpecs) return defaultStorageGroupInstanceName def _create_volume_for_default_volume_group( - self, conn, controllerConfigService, volumeInstanceName): + self, conn, controllerConfigService, volumeInstanceName, + extraSpecs): """Creates a volume for the default storage group for a fast policy. Creates a small first volume for the default storage group for a fast policy. This is necessary because you cannot remove - the last volume from a storage group and this scenario is likely + the last volume from a storage group and this scenario is likely. :param conn: the connection information to the ecom server :param controllerConfigService: the controller configuration service :param volumeInstanceName: the volume instance name + :param extraSpecs: additional info :returns: firstVolumeInstanceName - instance name of the first volume in the storage group """ @@ -262,15 +280,15 @@ def _create_volume_for_default_volume_group( poolInstanceName = self.utils.get_assoc_pool_from_volume( conn, volumeInstanceName) if poolInstanceName is None: - LOG.error(_LE("Unable to get associated pool of volume")) + LOG.error(_LE("Unable to get associated pool of volume.")) return failedRet volumeName = 'vol1' volumeSize = '1' - volumeDict, rc = ( # @UnusedVariable + volumeDict, _rc = ( self.provision.create_volume_from_pool( conn, storageConfigurationInstanceName, volumeName, - poolInstanceName, volumeSize)) + poolInstanceName, volumeSize, extraSpecs)) firstVolumeInstanceName = self.utils.find_volume_instance( conn, volumeDict, volumeName) return firstVolumeInstanceName @@ -278,7 +296,7 @@ def _create_volume_for_default_volume_group( def add_storage_group_to_tier_policy_rule( self, conn, tierPolicyServiceInstanceName, storageGroupInstanceName, tierPolicyRuleInstanceName, - storageGroupName, fastPolicyName): + storageGroupName, fastPolicyName, extraSpecs): """Add the storage group to the tier policy rule. :param conn: the connection information to the ecom server @@ -287,8 +305,11 @@ def add_storage_group_to_tier_policy_rule( :param tierPolicyRuleInstanceName: tier policy instance name :param storageGroupName: the storage group name (String) :param fastPolicyName: the fast policy name (String) + :param extraSpecs: additional info + :returns: int -- return code + :raises: VolumeBackendAPIException """ - # 5 is ("Add InElements to Policy") + # 5 is ("Add InElements to Policy"). modificationType = '5' rc, job = conn.InvokeMethod( @@ -296,13 +317,14 @@ def add_storage_group_to_tier_policy_rule( PolicyRule=tierPolicyRuleInstanceName, Operation=self.utils.get_num(modificationType, '16'), InElements=[storageGroupInstanceName]) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( "Error associating storage group : %(storageGroupName)s. " "To fast Policy: %(fastPolicyName)s with error " - "description: %(errordesc)s") + "description: %(errordesc)s.") % {'storageGroupName': storageGroupName, 'fastPolicyName': fastPolicyName, 'errordesc': errordesc}) @@ -317,13 +339,13 @@ def _get_service_level_tier_policy( """Returns the existing tier policies for a storage system instance. Given the storage system instance name, get the existing tier - policies on that array + policies on that array. :param conn: the connection information to the ecom server :param tierPolicyServiceInstanceName: the policy service :param fastPolicyName: the fast policy name e.g BRONZE1 :returns: foundTierPolicyRuleInstanceName - the short name, - everything after the : + everything after the : """ foundTierPolicyRuleInstanceName = None @@ -343,9 +365,8 @@ def _get_existing_tier_policies(self, conn, tierPolicyServiceInstanceName): :param conn: the connection information to the ecom server :param tierPolicyServiceInstanceName: the tier policy service - instance Name - :returns: tierPolicyRuleInstanceNames - the tier policy rule - instance names + instance Name + :returns: list -- the tier policy rule instance names """ tierPolicyRuleInstanceNames = conn.AssociatorNames( tierPolicyServiceInstanceName, ResultClass='Symm_TierPolicyRule') @@ -358,8 +379,7 @@ def get_associated_tier_policy_from_storage_group( :param conn: the connection information to the ecom server :param storageGroupInstanceName: the storage group instance name - :returns: tierPolicyInstanceNames - the list of tier policy - instance names + :returns: list -- the list of tier policy instance names """ tierPolicyInstanceName = None @@ -380,8 +400,7 @@ def get_associated_tier_from_tier_policy( :param conn: the connection information to the ecom server :param tierPolicyRuleInstanceName: the tier policy rule instance name - :returns: storageTierInstanceNames - a list of storage tier - instance names + :returns: list -- a list of storage tier instance names """ storageTierInstanceNames = conn.AssociatorNames( tierPolicyRuleInstanceName, @@ -389,8 +408,8 @@ def get_associated_tier_from_tier_policy( if len(storageTierInstanceNames) == 0: storageTierInstanceNames = None - LOG.warn(_LW( - "Unable to get storage tiers from tier policy rule ")) + LOG.warning(_LW( + "Unable to get storage tiers from tier policy rule.")) return storageTierInstanceNames @@ -403,17 +422,16 @@ def get_policy_default_storage_group( :param conn: the connection information to the ecom server :param controllerConfigService: ControllerConfigurationService - instance name + instance name :param policyName: string value :returns: storageGroupInstanceName - instance name of the default - storage group + storage group """ foundStorageMaskingGroupInstanceName = None storageMaskingGroupInstances = conn.Associators( controllerConfigService, ResultClass='CIM_DeviceMaskingGroup') - for storageMaskingGroupInstance in \ - storageMaskingGroupInstances: + for storageMaskingGroupInstance in storageMaskingGroupInstances: if ('_default_' in storageMaskingGroupInstance['ElementName'] and policyName in storageMaskingGroupInstance['ElementName']): @@ -435,8 +453,7 @@ def _get_associated_storage_groups_from_tier_policy( :param conn: the connection information to the ecom server :param tierPolicyInstanceName: tier policy instance name - :returns: managedElementInstanceNames - the list of storage - instance names + :returns: list -- the list of storage instance names """ managedElementInstanceNames = conn.AssociatorNames( tierPolicyInstanceName, @@ -451,8 +468,7 @@ def get_associated_pools_from_tier( :param conn: the connection information to the ecom server :param storageTierInstanceName: the storage tier instance name - :returns: storagePoolInstanceNames - a list of storage tier - instance names + :returns: list -- a list of storage tier instance names """ storagePoolInstanceNames = conn.AssociatorNames( storageTierInstanceName, @@ -463,17 +479,18 @@ def get_associated_pools_from_tier( def add_storage_group_and_verify_tier_policy_assoc( self, conn, controllerConfigService, storageGroupInstanceName, - storageGroupName, fastPolicyName): + storageGroupName, fastPolicyName, extraSpecs): """Adds a storage group to a tier policy and verifies success. Add a storage group to a tier policy rule and verify that it was - successful by getting the association + successful by getting the association. :param conn: the connection to the ecom server :param controllerConfigService: the controller config service :param storageGroupInstanceName: the storage group instance name :param storageGroupName: the storage group name (String) :param fastPolicyName: the fast policy name (String) + :param extraSpecs: additional info :returns: assocTierPolicyInstanceName """ failedRet = None @@ -482,45 +499,44 @@ def add_storage_group_and_verify_tier_policy_assoc( conn, controllerConfigService) tierPolicyServiceInstanceName = self.utils.get_tier_policy_service( conn, storageSystemInstanceName) - # get the fast policy instance name + # Get the fast policy instance name. tierPolicyRuleInstanceName = self._get_service_level_tier_policy( conn, tierPolicyServiceInstanceName, fastPolicyName) if tierPolicyRuleInstanceName is None: LOG.error(_LE( - "Cannot find the fast policy %(fastPolicyName)s"), + "Cannot find the fast policy %(fastPolicyName)s."), {'fastPolicyName': fastPolicyName}) return failedRet else: LOG.debug( - "Adding storage group %(storageGroupInstanceName)s to" - " tier policy rule %(tierPolicyRuleInstanceName)s", + "Adding storage group %(storageGroupInstanceName)s to " + "tier policy rule %(tierPolicyRuleInstanceName)s.", {'storageGroupInstanceName': storageGroupInstanceName, 'tierPolicyRuleInstanceName': tierPolicyRuleInstanceName}) - # Associate the new storage group with the existing fast policy + # Associate the new storage group with the existing fast policy. try: self.add_storage_group_to_tier_policy_rule( conn, tierPolicyServiceInstanceName, storageGroupInstanceName, tierPolicyRuleInstanceName, - storageGroupName, fastPolicyName) - except Exception as ex: - LOG.error(_LE("Exception: %s"), ex) - LOG.error(_LE( + storageGroupName, fastPolicyName, extraSpecs) + except Exception: + LOG.exception(_LE( "Failed to add storage group %(storageGroupInstanceName)s " - " to tier policy rule %(tierPolicyRuleInstanceName)s"), + "to tier policy rule %(tierPolicyRuleInstanceName)s."), {'storageGroupInstanceName': storageGroupInstanceName, 'tierPolicyRuleInstanceName': tierPolicyRuleInstanceName}) return failedRet - # check that the storage group has been associated with with the - # tier policy rule + # Check that the storage group has been associated with with the + # tier policy rule. assocTierPolicyInstanceName = ( self.get_associated_tier_policy_from_storage_group( conn, storageGroupInstanceName)) LOG.debug( "AssocTierPolicyInstanceName is " - "%(assocTierPolicyInstanceName)s ", + "%(assocTierPolicyInstanceName)s.", {'assocTierPolicyInstanceName': assocTierPolicyInstanceName}) return assocTierPolicyInstanceName @@ -531,7 +547,7 @@ def get_associated_policy_from_storage_group( :param conn: the connection information to the ecom server :param storageGroupInstanceName: storage group instance name :returns: foundTierPolicyInstanceName - instance name of the - tier policy object + tier policy object """ foundTierPolicyInstanceName = None @@ -547,18 +563,20 @@ def get_associated_policy_from_storage_group( def delete_storage_group_from_tier_policy_rule( self, conn, tierPolicyServiceInstanceName, - storageGroupInstanceName, tierPolicyRuleInstanceName): + storageGroupInstanceName, tierPolicyRuleInstanceName, + extraSpecs): """Disassociate the storage group from its tier policy rule. :param conn: connection the ecom server :param tierPolicyServiceInstanceName: instance name of the tier policy - service + service :param storageGroupInstanceName: instance name of the storage group :param tierPolicyRuleInstanceName: instance name of the tier policy - associated with the storage group + associated with the storage group + :param extraSpecs: additional information """ modificationType = '6' - LOG.debug("Invoking ModifyStorageTierPolicyRule %s", + LOG.debug("Invoking ModifyStorageTierPolicyRule %s.", tierPolicyRuleInstanceName) try: rc, job = conn.InvokeMethod( @@ -566,17 +584,19 @@ def delete_storage_group_from_tier_policy_rule( PolicyRule=tierPolicyRuleInstanceName, Operation=self.utils.get_num(modificationType, '16'), InElements=[storageGroupInstanceName]) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: LOG.error(_LE("Error disassociating storage group from " - "policy: %s"), errordesc) + "policy: %s."), errordesc) else: - LOG.debug("Disassociated storage group from policy %s") + LOG.debug("Disassociated storage group from policy.") else: - LOG.debug("ModifyStorageTierPolicyRule completed") + LOG.debug("ModifyStorageTierPolicyRule completed.") except Exception as e: - LOG.info(_LI("Storage group not associated with the policy %s"), e) + LOG.info(_LI("Storage group not associated with the " + "policy. Exception is %s."), e) def get_pool_associated_to_policy( self, conn, fastPolicyName, arraySN, @@ -584,14 +604,14 @@ def get_pool_associated_to_policy( """Given a FAST policy check that the pool is linked to the policy. If it's associated return the pool instance, if not return None. - First check if FAST is enabled on the array + First check if FAST is enabled on the array. :param conn: the ecom connection :param fastPolicyName: the fast policy name (String) :param arraySN: the array serial number (String) :param storageConfigService: the storage Config Service :param poolInstanceName: the pool instance we want to check for - association with the fast storage tier + association with the fast storage tier :returns: foundPoolInstanceName """ storageSystemInstanceName = self.utils.find_storage_system( @@ -599,7 +619,7 @@ def get_pool_associated_to_policy( if not self._check_if_fast_supported(conn, storageSystemInstanceName): errorMessage = (_( - "FAST is not supported on this array ")) + "FAST is not supported on this array.")) LOG.error(errorMessage) exception.VolumeBackendAPIException(data=errorMessage) @@ -608,11 +628,11 @@ def get_pool_associated_to_policy( tierPolicyRuleInstanceName = self._get_service_level_tier_policy( conn, tierPolicyServiceInstanceName, fastPolicyName) - # Get the associated storage tiers from the tier policy rule + # Get the associated storage tiers from the tier policy rule. storageTierInstanceNames = self.get_associated_tier_from_tier_policy( conn, tierPolicyRuleInstanceName) - # For each gold storage tier get the associated pools + # For each gold storage tier get the associated pools. foundPoolInstanceName = None for storageTierInstanceName in storageTierInstanceNames: assocStoragePoolInstanceNames = ( @@ -634,8 +654,9 @@ def is_tiering_policy_enabled_on_storage_system( True if FAST policy enabled on the given storage system; False otherwise. + :param conn: the ecom connection :param storageSystemInstanceName: a storage system instance name - :returns: boolean + :returns: boolean -- isTieringPolicySupported """ try: tierPolicyServiceInstanceName = self.utils.get_tier_policy_service( @@ -643,7 +664,7 @@ def is_tiering_policy_enabled_on_storage_system( isTieringPolicySupported = self.is_tiering_policy_enabled( conn, tierPolicyServiceInstanceName) except Exception as e: - LOG.error(_LE("Exception: %s"), e) + LOG.error(_LE("Exception: %s."), e) return False return isTieringPolicySupported @@ -652,8 +673,10 @@ def get_tier_policy_by_name( self, conn, arrayName, policyName): """Given the name of the policy, get the TierPolicyRule instance name. - :param policyName: the name of policy rule, a string value - :returns: tierPolicyInstanceName - tier policy instance name + :param conn: the ecom connection + :param arrayName: the array + :param policyName: string -- the name of policy rule + :returns: tier policy instance name. None if not found """ tierPolicyInstanceNames = conn.EnumerateInstanceNames( 'Symm_TierPolicyRule') @@ -669,18 +692,24 @@ def get_capacities_associated_to_policy(self, conn, arrayName, policyName): Given the name of the policy, get the total capacity and un-used capacity in GB of all the storage pools associated with the policy. + :param conn: the ecom connection + :param arrayName: the array :param policyName: the name of policy rule, a string value - :returns: total_capacity_gb - total capacity in GB of all pools - associated with the policy - :returns: free_capacity_gb - (total capacity-EMCSubscribedCapacity) - in GB of all pools associated with - the policy + :returns: int -- total capacity in GB of all pools associated with + the policy + :returns: int -- real physical capacity in GB of all pools + available to be used + :returns: int -- (Provisioned capacity-EMCSubscribedCapacity) in GB + is the the capacity that has been provisioned + :returns: int -- the maximum oversubscription ration """ policyInstanceName = self.get_tier_policy_by_name( conn, arrayName, policyName) total_capacity_gb = 0 - allocated_capacity_gb = 0 + provisioned_capacity_gb = 0 + free_capacity_gb = 0 + array_max_over_subscription = None tierInstanceNames = self.get_associated_tier_from_tier_policy( conn, policyInstanceName) @@ -702,44 +731,53 @@ def get_capacities_associated_to_policy(self, conn, arrayName, policyName): break total_capacity_gb += self.utils.convert_bits_to_gbs( storagePoolInstance['TotalManagedSpace']) - allocated_capacity_gb += self.utils.convert_bits_to_gbs( + provisioned_capacity_gb += self.utils.convert_bits_to_gbs( storagePoolInstance['EMCSubscribedCapacity']) + free_capacity_gb += self.utils.convert_bits_to_gbs( + storagePoolInstance['RemainingManagedSpace']) + try: + array_max_over_subscription = ( + self.utils.get_ratio_from_max_sub_per( + storagePoolInstance['EMCMaxSubscriptionPercent'])) + except KeyError: + array_max_over_subscription = 65534 LOG.debug( - "policyName:%(policyName)s, pool: %(poolInstanceName)s, " - "allocated_capacity_gb = %(allocated_capacity_gb)lu", + "PolicyName:%(policyName)s, pool: %(poolInstanceName)s, " + "provisioned_capacity_gb = %(provisioned_capacity_gb)lu.", {'policyName': policyName, 'poolInstanceName': poolInstanceName, - 'allocated_capacity_gb': allocated_capacity_gb}) + 'provisioned_capacity_gb': provisioned_capacity_gb}) - free_capacity_gb = total_capacity_gb - allocated_capacity_gb - return (total_capacity_gb, free_capacity_gb) + return (total_capacity_gb, free_capacity_gb, + provisioned_capacity_gb, array_max_over_subscription) def get_or_create_default_storage_group( self, conn, controllerConfigService, fastPolicyName, - volumeInstance): + volumeInstance, extraSpecs): """Create or get a default storage group for FAST policy. :param conn: the ecom connection :param controllerConfigService: the controller configuration service :param fastPolicyName: the fast policy name (String) :param volumeInstance: the volume instance + :param extraSpecs: additional info :returns: defaultStorageGroupInstanceName - the default storage group instance name """ - defaultSgGroupName = (DEFAULT_SG_PREFIX + fastPolicyName + - DEFAULT_SG_POSTFIX) + defaultSgName = self.format_default_sg_string(fastPolicyName) defaultStorageGroupInstanceName = ( self.utils.find_storage_masking_group(conn, controllerConfigService, - defaultSgGroupName)) + defaultSgName)) if defaultStorageGroupInstanceName is None: - # create it and associate it with the FAST policy in question + # Create it and associate it with the FAST policy in question. defaultStorageGroupInstanceName = ( self._create_default_storage_group(conn, controllerConfigService, fastPolicyName, - defaultSgGroupName, - volumeInstance)) + defaultSgName, + volumeInstance, + extraSpecs)) return defaultStorageGroupInstanceName @@ -748,7 +786,7 @@ def _get_associated_tier_policy_from_pool(self, conn, poolInstanceName): :param conn: the connection information to the ecom server :param poolInstanceName: the pool instance name - :param fastPolicyName: the FAST Policy name (if it exists) + :returns: the FAST Policy name (if it exists) """ fastPolicyName = None @@ -771,9 +809,10 @@ def _get_associated_tier_policy_from_pool(self, conn, poolInstanceName): def is_volume_in_default_SG(self, conn, volumeInstanceName): """Check if the volume is already part of the default storage group. + :param conn: the ecom connection :param volumeInstanceName: the volume instance - :returns: True if the volume is already in default storage group - False otherwise + :returns: boolean -- True if the volume is already in default + storage group. False otherwise """ sgInstanceNames = conn.AssociatorNames( volumeInstanceName, diff --git a/emc_vmax_fc.py b/emc_vmax_fc.py index da6d5f3..c542f24 100644 --- a/emc_vmax_fc.py +++ b/emc_vmax_fc.py @@ -1,4 +1,4 @@ -# Copyright (c) 2014 EMC Corporation. +# Copyright (c) 2015 EMC Corporation. # All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); you may @@ -12,10 +12,14 @@ # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the # License for the specific language governing permissions and limitations # under the License. + +import ast + +from oslo_log import log as logging import six from cinder import context -from cinder.openstack.common import log as logging +from cinder.i18n import _LW from cinder.volume import driver from cinder.volume.drivers.emc import emc_vmax_common from cinder.zonemanager import utils as fczm_utils @@ -32,15 +36,49 @@ class EMCVMAXFCDriver(driver.FibreChannelDriver): performance enhancement. 2.0.0 - Add driver requirement functions 2.1.0 - Add consistency group functions + 2.1.1 - Fixed issue with mismatched config (bug #1442376) + 2.1.2 - Clean up failed clones (bug #1440154) + 2.1.3 - Fixed a problem with FAST support (bug #1435069) + 2.2.0 - Add manage/unmanage + 2.2.1 - Support for SE 8.0.3 + 2.2.2 - Update Consistency Group + 2.2.3 - Pool aware scheduler(multi-pool) support + 2.2.4 - Create CG from CG snapshot + 2.3.0 - Name change for MV and SG for FAST (bug #1515181) + - Fix for randomly choosing port group. (bug #1501919) + - get_short_host_name needs to be called in find_device_number + (bug #1520635) + - Proper error handling for invalid SLOs (bug #1512795) + - Extend Volume for VMAX3, SE8.1.0.3 + https://blueprints.launchpad.net/cinder/+spec/vmax3-extend-volume + - Incorrect SG selected on an attach (#1515176) + - Cleanup Zoning (bug #1501938) NOTE: FC only + - Last volume in SG fix + - _remove_last_vol_and_delete_sg is not being called + for VMAX3 (bug #1520549) + - necessary updates for CG changes (#1534616) + - Changing PercentSynced to CopyState (bug #1517103) + - Getting iscsi ip from port in existing masking view + - Replacement of EMCGetTargetEndpoints api (bug #1512791) + - VMAX3 snapvx improvements (bug #1522821) + 2.3.1 - VMAX2/VMAX3 iscsi multipath support (iscsi only) + 2.3.2 - VMAX oversubscription Support (blueprint vmax-oversubscription) + 2.3.3 - VMAX Driver - Live Migration for VMAX3 (bug #1587967) + 2.3.4 - additional locking (bug #1630535)(bug #1660374) + 2.3.5 - remove_and_reset_members fix + 2.3.6 - copy state fix (bug #1660378) + 2.3.7 - superfluous debug messages causing error(#1664630) + 2.3.8 - manage/unmanage volumes fix """ - VERSION = "2.1.0" + VERSION = "2.3.8" def __init__(self, *args, **kwargs): super(EMCVMAXFCDriver, self).__init__(*args, **kwargs) self.common = emc_vmax_common.EMCVMAXCommon( 'FC', + self.VERSION, configuration=self.configuration) self.zonemanager_lookup_service = fczm_utils.create_lookup_service() @@ -107,7 +145,7 @@ def ensure_export(self, context, volume): """Driver entry point to get the export info for an existing volume.""" pass - def create_export(self, context, volume): + def create_export(self, context, volume, connector): """Driver entry point to get the export info for a new volume.""" pass @@ -141,7 +179,7 @@ def initialize_connection(self, volume, connector): or - { + { 'driver_volume_type': 'fibre_channel' 'data': { 'target_discovered': True, @@ -177,53 +215,93 @@ def terminate_connection(self, volume, connector, **kwargs): if there isn't an initiator_target_map in the return of terminate_connection. - :returns: data - the target_wwns and initiator_target_map if the - zone is to be removed, otherwise empty + :param volume: the volume object + :param connector: the connector object + :returns: dict -- the target_wwns and initiator_target_map if the + zone is to be removed, otherwise empty """ - data = {} + data = {'driver_volume_type': 'fibre_channel', + 'data': {}} loc = volume['provider_location'] - name = eval(loc) + name = ast.literal_eval(loc) storage_system = name['keybindings']['SystemName'] - LOG.debug("Start FC detach process for volume: %(volume)s", + LOG.debug("Start FC detach process for volume: %(volume)s.", {'volume': volume['name']}) - target_wwns, init_targ_map = self._build_initiator_target_map( - storage_system, volume, connector) - mvInstanceName = self.common.get_masking_view_by_volume( volume, connector) if mvInstanceName is not None: portGroupInstanceName = ( self.common.get_port_group_from_masking_view( mvInstanceName)) + initiatorGroupInstanceName = ( + self.common.get_initiator_group_from_masking_view( + mvInstanceName)) LOG.debug("Found port group: %(portGroup)s " - "in masking view %(maskingView)s", + "in masking view %(maskingView)s.", {'portGroup': portGroupInstanceName, 'maskingView': mvInstanceName}) + # Map must be populated before the terminate_connection + target_wwns, init_targ_map = self._build_initiator_target_map( + storage_system, volume, connector) self.common.terminate_connection(volume, connector) - LOG.debug("Looking for masking views still associated with" - "Port Group %s", portGroupInstanceName) - mvInstances = self.common.get_masking_views_by_port_group( - portGroupInstanceName) - if len(mvInstances) > 0: - LOG.debug("Found %(numViews)lu MaskingViews.", - {'numViews': len(mvInstances)}) - data = {'driver_volume_type': 'fibre_channel', - 'data': {}} - else: # no views found - LOG.debug("No MaskingViews were found. Deleting zone.") + LOG.debug("Looking for masking views still associated with " + "Port Group %s.", portGroupInstanceName) + # check if the initiator group has been deleted + checkIgInstanceName = ( + self.common.check_ig_instance_name(initiatorGroupInstanceName)) + + # if it has not been deleted, check for remaining masking views + if checkIgInstanceName is not None: + mvInstances = self._get_common_masking_views( + portGroupInstanceName, initiatorGroupInstanceName) + + if len(mvInstances) > 0: + LOG.debug("Found %(numViews)lu MaskingViews.", + {'numViews': len(mvInstances)}) + data = {'driver_volume_type': 'fibre_channel', + 'data': {}} + else: # no masking views found + LOG.debug("No MaskingViews were found. Deleting zone.") + data = {'driver_volume_type': 'fibre_channel', + 'data': {'target_wwn': target_wwns, + 'initiator_target_map': init_targ_map}} + + LOG.debug("Return FC data for zone removal: %(data)s.", + {'data': data}) + + else: # The initiator group has been deleted + LOG.debug("Initiator Group has been deleted. Deleting zone.") data = {'driver_volume_type': 'fibre_channel', 'data': {'target_wwn': target_wwns, 'initiator_target_map': init_targ_map}} - LOG.debug("Return FC data for zone removal: %(data)s.", - {'data': data}) + LOG.debug("Return FC data for zone removal: %(data)s.", + {'data': data}) + else: + LOG.warning(_LW("Volume %(volume)s is not in any masking view."), + {'volume': volume['name']}) return data + def _get_common_masking_views( + self, portGroupInstanceName, initiatorGroupInstanceName): + """Check to see the existence of mv in list""" + mvInstances = [] + mvInstancesByPG = self.common.get_masking_views_by_port_group( + portGroupInstanceName) + + mvInstancesByIG = self.common.get_masking_views_by_initiator_group( + initiatorGroupInstanceName) + + for mvInstanceByPG in mvInstancesByPG: + if mvInstanceByPG in mvInstancesByIG: + mvInstances.append(mvInstanceByPG) + return mvInstances + def _build_initiator_target_map(self, storage_system, volume, connector): """Build the target_wwns and the initiator target map.""" target_wwns = [] @@ -241,7 +319,7 @@ def _build_initiator_target_map(self, storage_system, volume, connector): target_wwns.extend(map_d['target_port_wwn_list']) for initiator in map_d['initiator_port_wwn_list']: init_targ_map[initiator] = map_d['target_port_wwn_list'] - else: # no lookup service, pre-zoned case + else: # No lookup service, pre-zoned case. target_wwns = self.common.get_target_wwns(storage_system, connector) for initiator in initiator_wwns: @@ -256,7 +334,8 @@ def extend_volume(self, volume, new_size): def get_volume_stats(self, refresh=False): """Get volume stats. - If 'refresh' is True, run update the stats first. + :param refresh: boolean -- If True, run update the stats first. + :returns: dict -- the stats dict """ if refresh: self.update_volume_stats() @@ -274,27 +353,25 @@ def update_volume_stats(self): def migrate_volume(self, ctxt, volume, host): """Migrate a volume from one Volume Backend to another. - :param self: reference to class - :param ctxt: + :param ctxt: context :param volume: the volume object including the volume_type_id :param host: the host dict holding the relevant target(destination) - information - :returns: moved - :returns: list + information + :returns: boolean -- Always returns True + :returns: dict -- Empty dict {} """ return self.common.migrate_volume(ctxt, volume, host) def retype(self, ctxt, volume, new_type, diff, host): """Migrate volume to another host using retype. - :param self: reference to class - :param ctxt: + :param ctxt: context :param volume: the volume object including the volume_type_id :param new_type: the new volume type. + :param diff: Unused parameter. :param host: the host dict holding the relevant - target(destination) information - :returns: moved - "returns: list + target(destination) information + :returns: boolean -- True if retype succeeded, False if error """ return self.common.retype(ctxt, volume, new_type, diff, host) @@ -302,16 +379,65 @@ def create_consistencygroup(self, context, group): """Creates a consistencygroup.""" self.common.create_consistencygroup(context, group) - def delete_consistencygroup(self, context, group): + def delete_consistencygroup(self, context, group, volumes): """Deletes a consistency group.""" - volumes = self.db.volume_get_all_by_group(context, group['id']) return self.common.delete_consistencygroup( context, group, volumes) - def create_cgsnapshot(self, context, cgsnapshot): + def create_cgsnapshot(self, context, cgsnapshot, snapshots): """Creates a cgsnapshot.""" - return self.common.create_cgsnapshot(context, cgsnapshot, self.db) + return self.common.create_cgsnapshot(context, cgsnapshot, snapshots) - def delete_cgsnapshot(self, context, cgsnapshot): + def delete_cgsnapshot(self, context, cgsnapshot, snapshots): """Deletes a cgsnapshot.""" - return self.common.delete_cgsnapshot(context, cgsnapshot, self.db) + return self.common.delete_cgsnapshot(context, cgsnapshot, snapshots) + + def manage_existing(self, volume, external_ref): + """Manages an existing VMAX Volume (import to Cinder). + + Renames the Volume to match the expected name for the volume. + Also need to consider things like QoS, Emulation, account/tenant. + """ + return self.common.manage_existing(volume, external_ref) + + def manage_existing_get_size(self, volume, external_ref): + """Return size of an existing VMAX volume to manage_existing. + + :param self: reference to class + :param volume: the volume object including the volume_type_id + :param external_ref: reference to the existing volume + :returns: size of the volume in GB + """ + return self.common.manage_existing_get_size(volume, external_ref) + + def unmanage(self, volume): + """Export VMAX volume from Cinder. + + Leave the volume intact on the backend array. + """ + return self.common.unmanage(volume) + + def update_consistencygroup(self, context, group, + add_volumes, remove_volumes): + """Updates LUNs in consistency group.""" + return self.common.update_consistencygroup(group, add_volumes, + remove_volumes) + + def create_consistencygroup_from_src(self, context, group, volumes, + cgsnapshot=None, snapshots=None, + source_cg=None, source_vols=None): + """Creates the consistency group from source. + + Currently the source can only be a cgsnapshot. + + :param context: the context + :param group: the consistency group object to be created + :param volumes: volumes in the consistency group + :param cgsnapshot: the source consistency group snapshot + :param snapshots: snapshots of the source volumes + :param source_cg: the dictionary of a consistency group as source. + :param source_vols: a list of volume dictionaries in the source_cg. + """ + return self.common.create_consistencygroup_from_src( + context, group, volumes, cgsnapshot, snapshots, source_cg, + source_vols) diff --git a/emc_vmax_https.py b/emc_vmax_https.py index 6a9d677..aa9584f 100644 --- a/emc_vmax_https.py +++ b/emc_vmax_https.py @@ -1,4 +1,4 @@ -# Copyright (c) 2012 - 2014 EMC Corporation. +# Copyright (c) 2012 - 2015 EMC Corporation. # All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); you may @@ -14,26 +14,27 @@ # under the License. import base64 -import httplib import os import socket import ssl import string import struct -import urllib from eventlet import patcher -import OpenSSL +try: + import OpenSSL +except ImportError: + OpenSSL = None +from oslo_log import log as logging import six +from six.moves import http_client +from six.moves import urllib -from cinder.i18n import _LI -from cinder.openstack.common import log as logging +from cinder.i18n import _, _LI # Handle case where we are running in a monkey patched environment -if patcher.is_monkey_patched('socket'): - from eventlet.green.OpenSSL.SSL import GreenConnection as Connection -else: - raise ImportError +if OpenSSL and patcher.is_monkey_patched('socket'): + from eventlet.green.OpenSSL import SSL try: import pywbem @@ -53,7 +54,9 @@ def to_bytes(s): def get_default_ca_certs(): - """Try to find out system path with ca certificates. This path is cached and + """Gets the default CA certificates if found, otherwise None. + + Try to find out system path with ca certificates. This path is cached and returned. If no path is found out, None is returned. """ if not hasattr(get_default_ca_certs, '_path'): @@ -72,13 +75,13 @@ def get_default_ca_certs(): class OpenSSLConnectionDelegator(object): """An OpenSSL.SSL.Connection delegator. - Supplies an additional 'makefile' method which httplib requires + Supplies an additional 'makefile' method which http_client requires and is not present in OpenSSL.SSL.Connection. Note: Since it is not possible to inherit from OpenSSL.SSL.Connection a delegator must be used. """ def __init__(self, *args, **kwargs): - self.connection = Connection(*args, **kwargs) + self.connection = SSL.GreenConnection(*args, **kwargs) def __getattr__(self, name): return getattr(self.connection, name) @@ -87,7 +90,7 @@ def makefile(self, *args, **kwargs): return socket._fileobject(self.connection, *args, **kwargs) -class HTTPSConnection(httplib.HTTPSConnection): +class HTTPSConnection(http_client.HTTPSConnection): def __init__(self, host, port=None, key_file=None, cert_file=None, strict=None, ca_certs=None, no_verification=False): if not pywbemAvailable: @@ -99,20 +102,21 @@ def __init__(self, host, port=None, key_file=None, cert_file=None, else: excp_lst = () try: - httplib.HTTPSConnection.__init__(self, host, port, - key_file=key_file, - cert_file=cert_file) + http_client.HTTPSConnection.__init__(self, host, port, + key_file=key_file, + cert_file=cert_file) self.key_file = None if key_file is None else key_file self.cert_file = None if cert_file is None else cert_file self.insecure = no_verification - self.ca_certs = None if ca_certs is None else str(ca_certs) + self.ca_certs = ( + None if ca_certs is None else six.text_type(ca_certs)) self.set_context() # ssl exceptions are reported in various form in Python 3 # so to be compatible, we report the same kind as under # Python2 except excp_lst as e: - raise pywbem.cim_http.Error(str(e)) + raise pywbem.cim_http.Error(six.text_type(e)) @staticmethod def host_matches_cert(host, x509): @@ -124,49 +128,55 @@ def host_matches_cert(host, x509): or a Subject Alternative Name matches 'host'. """ def check_match(name): - # Directly match the name + # Directly match the name. if name == host: return True - # Support single wildcard matching + # Support single wildcard matching. if name.startswith('*.') and host.find('.') > 0: if name[2:] == host.split('.', 1)[1]: return True common_name = x509.get_subject().commonName - # First see if we can match the CN + # First see if we can match the CN. if check_match(common_name): return True - # Also try Subject Alternative Names for a match + # Also try Subject Alternative Names for a match. san_list = None for i in range(x509.get_extension_count()): ext = x509.get_extension(i) if ext.get_short_name() == b'subjectAltName': - san_list = str(ext) + san_list = six.text_type(ext) for san in ''.join(san_list.split()).split(','): if san.startswith('DNS:'): if check_match(san.split(':', 1)[1]): return True - # Server certificate does not match host - msg = ('Host "%s" does not match x509 certificate contents: ' - 'CommonName "%s"' % (host, common_name)) + # Server certificate does not match host. + msg = (_("Host %(host)s does not match x509 certificate contents: " + "CommonName %(commonName)s.") + % {'host': host, + 'commonName': common_name}) + if san_list is not None: - msg = msg + ', subjectAltName "%s"' % san_list + msg = (_("%(message)s, subjectAltName: %(sanList)s.") + % {'message': msg, + 'sanList': san_list}) raise pywbem.cim_http.AuthError(msg) def verify_callback(self, connection, x509, errnum, depth, preverify_ok): if x509.has_expired(): - msg = "SSL Certificate expired on '%s'" % x509.get_notAfter() + msg = msg = (_("SSL Certificate expired on %s.") + % x509.get_notAfter()) raise pywbem.cim_http.AuthError(msg) if depth == 0 and preverify_ok: # We verify that the host matches against the last - # certificate in the chain + # certificate in the chain. return self.host_matches_cert(self.host, x509) else: - # Pass through OpenSSL's default result + # Pass through OpenSSL's default result. return preverify_ok def set_context(self): @@ -184,32 +194,37 @@ def set_context(self): try: self.context.use_certificate_file(self.cert_file) except Exception as e: - msg = ('Unable to load cert from "%s" %s' - % (self.cert_file, e)) + msg = (_("Unable to load cert from %(cert)s %(e)s.") + % {'cert': self.cert_file, + 'e': e}) raise pywbem.cim_http.AuthError(msg) if self.key_file is None: - # We support having key and cert in same file + # We support having key and cert in same file. try: self.context.use_privatekey_file(self.cert_file) except Exception as e: - msg = ('No key file specified and unable to load key ' - 'from "%s" %s' % (self.cert_file, e)) + msg = (_("No key file specified and unable to load key " + "from %(cert)s %(e)s.") + % {'cert': self.cert_file, + 'e': e}) raise pywbem.cim_http.AuthError(msg) if self.key_file: try: self.context.use_privatekey_file(self.key_file) except Exception as e: - msg = ('Unable to load key from "%s" %s' - % (self.key_file, e)) + msg = (_("Unable to load key from %(cert)s %(e)s.") + % {'cert': self.cert_file, + 'e': e}) raise pywbem.cim_http.AuthError(msg) if self.ca_certs: try: self.context.load_verify_locations(to_bytes(self.ca_certs)) except Exception as e: - msg = ('Unable to load CA from "%s" %s' - % (self.ca_certs, e)) + msg = (_("Unable to load CA from %(cert)s %(e)s.") + % {'cert': self.cert_file, + 'e': e}) raise pywbem.cim_http.AuthError(msg) else: self.context.set_default_verify_paths() @@ -241,7 +256,7 @@ def wbem_request(url, data, creds, headers=None, debug=0, x509=None, """Send request over HTTP. Send XML data over HTTP to the specified url. Return the - response in XML. Uses Python's build-in httplib. x509 may be a + response in XML. Uses Python's build-in http_client. x509 may be a dictionary containing the location of the SSL certificate and key files. """ @@ -260,7 +275,7 @@ def wbem_request(url, data, creds, headers=None, debug=0, x509=None, localAuthHeader = None tryLimit = 5 - if isinstance(data, unicode): + if isinstance(data, six.text_type): data = data.encode('utf-8') data = '\n' + data @@ -269,7 +284,6 @@ def wbem_request(url, data, creds, headers=None, debug=0, x509=None, elif no_verification: ca_certs = None - # local = False if use_ssl: h = HTTPSConnection( host, @@ -296,10 +310,10 @@ def wbem_request(url, data, creds, headers=None, debug=0, x509=None, h.putheader('PegasusAuthorization', 'Local "%s"' % locallogin) for hdr in headers: - if isinstance(hdr, unicode): + if isinstance(hdr, six.text_type): hdr = hdr.encode('utf-8') s = map(lambda x: string.strip(x), string.split(hdr, ":", 1)) - h.putheader(urllib.quote(s[0]), urllib.quote(s[1])) + h.putheader(urllib.parse.quote(s[0]), urllib.parse.quote(s[1])) try: h.endheaders() @@ -315,13 +329,18 @@ def wbem_request(url, data, creds, headers=None, debug=0, x509=None, if response.status != 200: raise pywbem.cim_http.Error('HTTP error') - except httplib.BadStatusLine as arg: - raise pywbem.cim_http.Error("Bad Status line returned: '%s'" - % arg) - except socket.error as arg: - raise pywbem.cim_http.Error("Socket error: %s" % (arg,)) + except http_client.BadStatusLine as arg: + msg = (_("Bad Status line returned: %(arg)s.") + % {'arg': arg}) + raise pywbem.cim_http.Error(msg) except socket.sslerror as arg: - raise pywbem.cim_http.Error("SSL error: %s" % (arg,)) + msg = (_("SSL error: %(arg)s.") + % {'arg': arg}) + raise pywbem.cim_http.Error(msg) + except socket.error as arg: + msg = (_("Socket error: %(arg)s.") + % {'arg': arg}) + raise pywbem.cim_http.Error(msg) break diff --git a/emc_vmax_iscsi.py b/emc_vmax_iscsi.py index bb7a99c..e784e48 100644 --- a/emc_vmax_iscsi.py +++ b/emc_vmax_iscsi.py @@ -1,4 +1,4 @@ -# Copyright (c) 2012 - 2014 EMC Corporation. +# Copyright (c) 2012 - 2015 EMC Corporation. # All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); you may @@ -16,14 +16,12 @@ ISCSI Drivers for EMC VMAX arrays based on SMI-S. """ -import os - +from oslo_log import log as logging import six from cinder import context from cinder import exception from cinder.i18n import _, _LE, _LI -from cinder.openstack.common import log as logging from cinder.volume import driver from cinder.volume.drivers.emc import emc_vmax_common @@ -42,22 +40,56 @@ class EMCVMAXISCSIDriver(driver.ISCSIDriver): performance enhancement. 2.0.0 - Add driver requirement functions 2.1.0 - Add consistency group functions + 2.1.1 - Fixed issue with mismatched config (bug #1442376) + 2.1.2 - Clean up failed clones (bug #1440154) + 2.1.3 - Fixed a problem with FAST support (bug #1435069) + 2.2.0 - Add manage/unmanage + 2.2.1 - Support for SE 8.0.3 + 2.2.2 - Update Consistency Group + 2.2.3 - Pool aware scheduler(multi-pool) support + 2.2.4 - Create CG from CG snapshot + 2.3.0 - Name change for MV and SG for FAST (bug #1515181) + - Fix for randomly choosing port group. (bug #1501919) + - get_short_host_name needs to be called in find_device_number + (bug #1520635) + - Proper error handling for invalid SLOs (bug #1512795) + - Extend Volume for VMAX3, SE8.1.0.3 + https://blueprints.launchpad.net/cinder/+spec/vmax3-extend-volume + - Incorrect SG selected on an attach (#1515176) + - Cleanup Zoning (bug #1501938) NOTE: FC only + - Last volume in SG fix + - _remove_last_vol_and_delete_sg is not being called + for VMAX3 (bug #1520549) + - necessary updates for CG changes (#1534616) + - Changing PercentSynced to CopyState (bug #1517103) + - Getting iscsi ip from port in existing masking view + - Replacement of EMCGetTargetEndpoints api (bug #1512791) + - VMAX3 snapvx improvements (bug #1522821) + 2.3.1 - VMAX2/VMAX3 iscsi multipath support (iscsi only) + 2.3.2 - VMAX oversubscription Support (blueprint vmax-oversubscription) + 2.3.3 - VMAX Driver - Live Migration for VMAX3 (bug #1587967) + 2.3.4 - additional locking (bug #1630535)(bug #1660374) + 2.3.5 - remove_and_reset_members fix + 2.3.6 - copy state fix (bug #1660378) + 2.3.7 - superfluous debug messages causing error(#1664630) + 2.3.8 - manage/unmanage volumes fix """ - VERSION = "2.1.0" + VERSION = "2.3.8" def __init__(self, *args, **kwargs): super(EMCVMAXISCSIDriver, self).__init__(*args, **kwargs) - self.common =\ + self.common = ( emc_vmax_common.EMCVMAXCommon('iSCSI', - configuration=self.configuration) + self.VERSION, + configuration=self.configuration)) def check_for_setup_error(self): pass def create_volume(self, volume): - """Creates a EMC(VMAX/VNX) volume.""" + """Creates a VMAX volume.""" volpath = self.common.create_volume(volume) model_update = {} @@ -116,7 +148,7 @@ def ensure_export(self, context, volume): """Driver entry point to get the export info for an existing volume.""" pass - def create_export(self, context, volume): + def create_export(self, context, volume, connector): """Driver entry point to get the export info for a new volume.""" pass @@ -133,7 +165,10 @@ def initialize_connection(self, volume, connector): The iscsi driver returns a driver_volume_type of 'iscsi'. the format of the driver data is defined in smis_get_iscsi_properties. - Example return value:: + Example return value: + + .. code-block:: json + { 'driver_volume_type': 'iscsi' 'data': { @@ -143,40 +178,55 @@ def initialize_connection(self, volume, connector): 'volume_id': '12345678-1234-4321-1234-123456789012', } } + Example return value (multipath is enabled):: + { + 'driver_volume_type': 'iscsi' + 'data': { + 'target_discovered': True, + 'target_iqns': ['iqn.2010-10.org.openstack:volume-00001', + 'iqn.2010-10.org.openstack:volume-00002'], + 'target_portals': ['127.0.0.1:3260', '127.0.1.1:3260'], + 'target_luns': [1, 1], + } + } """ - self.common.initialize_connection( + device_info = self.common.initialize_connection( volume, connector) + try: + ip_and_iqn = device_info['ip_and_iqn'] + is_multipath = device_info['is_multipath'] + except KeyError as ex: + exception_message = (_("Cannot get iSCSI ipaddresses or " + "multipath flag. Exception is %(ex)s. ") + % {'ex': ex}) + raise exception.VolumeBackendAPIException(data=exception_message) iscsi_properties = self.smis_get_iscsi_properties( - volume, connector) + volume, connector, ip_and_iqn, is_multipath) - LOG.info(_LI("Leaving initialize_connection: %s"), (iscsi_properties)) + LOG.info(_LI("Leaving initialize_connection: %s"), iscsi_properties) return { 'driver_volume_type': 'iscsi', 'data': iscsi_properties } - def smis_do_iscsi_discovery(self, volume): - LOG.info(_LI("ISCSI provider_location not stored, using discovery")) - if not self._check_for_iscsi_ip_address(): - LOG.error(_LE( - "You must set your iscsi_ip_address in cinder.conf ")) + def _parse_target_list(self, targets): + """Parse target list into usable format. - (out, _err) = self._execute('iscsiadm', '-m', 'discovery', - '-t', 'sendtargets', '-p', - self.configuration.iscsi_ip_address, - run_as_root=True) - - LOG.info(_LI( - "smis_do_iscsi_discovery is: %(out)s"), - {'out': out}) - targets = [] - for target in out.splitlines(): - targets.append(target) - - return targets + :param targets: list of all targets + :return: outTargets + """ + outTargets = [] + for target in targets: + results = target.split(" ") + properties = {} + properties['target_portal'] = results[0].split(",")[0] + properties['target_iqn'] = results[1] + outTargets.append(properties) + return outTargets - def smis_get_iscsi_properties(self, volume, connector): + def smis_get_iscsi_properties(self, volume, connector, ip_and_iqn, + is_multipath): """Gets iscsi configuration. We ideally get saved information in the volume entity, but fall back @@ -192,48 +242,48 @@ def smis_get_iscsi_properties(self, volume, connector): present meaning no authentication, or auth_method == `CHAP` meaning use CHAP with the specified credentials. """ - properties = {} - location = self.smis_do_iscsi_discovery(volume) - if not location: - raise exception.InvalidVolume(_("Could not find iSCSI export " - " for volume %(volumeName)s") - % {'volumeName': volume['name']}) + device_info = self.common.find_device_number( + volume, connector['host']) - LOG.debug("ISCSI Discovery: Found %s", (location)) - properties['target_discovered'] = True + isError = False + if device_info: + try: + lun_id = device_info['hostlunid'] + except KeyError: + isError = True + else: + isError = True - device_info = self.common.find_device_number(volume, connector) - - if device_info is None or device_info['hostlunid'] is None: + if isError: + LOG.error(_LE("Unable to get the lun id")) exception_message = (_("Cannot find device number for volume " - "%(volumeName)s") + "%(volumeName)s.") % {'volumeName': volume['name']}) raise exception.VolumeBackendAPIException(data=exception_message) - device_number = device_info['hostlunid'] - - LOG.info(_LI( - "location is: %(location)s"), {'location': location}) - - for loc in location: - results = loc.split(" ") - properties['target_portal'] = results[0].split(",")[0] - properties['target_iqn'] = results[1] - - properties['target_lun'] = device_number - + properties = {} + if len(ip_and_iqn) > 1 and is_multipath: + properties['target_portals'] = ([t['ip'] + ":3260" for t in + ip_and_iqn]) + properties['target_iqns'] = ([t['iqn'].split(",")[0] for t in + ip_and_iqn]) + properties['target_luns'] = [lun_id] * len(ip_and_iqn) + properties['target_discovered'] = True + properties['target_iqn'] = ip_and_iqn[0]['iqn'].split(",")[0] + properties['target_portal'] = ip_and_iqn[0]['ip'] + ":3260" + properties['target_lun'] = lun_id properties['volume_id'] = volume['id'] LOG.info(_LI( - "ISCSI properties: %(properties)s"), {'properties': properties}) + "ISCSI properties: %(properties)s."), {'properties': properties}) LOG.info(_LI( - "ISCSI volume is: %(volume)s"), {'volume': volume}) + "ISCSI volume is: %(volume)s."), {'volume': volume}) if 'provider_auth' in volume: auth = volume['provider_auth'] LOG.info(_LI( - "AUTH properties: %(authProps)s"), {'authProps': auth}) + "AUTH properties: %(authProps)s."), {'authProps': auth}) if auth is not None: (auth_method, auth_username, auth_secret) = auth.split() @@ -242,7 +292,7 @@ def smis_get_iscsi_properties(self, volume, connector): properties['auth_username'] = auth_username properties['auth_password'] = auth_secret - LOG.info(_LI("AUTH properties: %s"), (properties)) + LOG.info(_LI("AUTH properties: %s."), properties) return properties @@ -274,27 +324,24 @@ def update_volume_stats(self): def migrate_volume(self, ctxt, volume, host): """Migrate a volume from one Volume Backend to another. - :param self: reference to class - :param ctxt: + + :param ctxt: context :param volume: the volume object including the volume_type_id - :param host: the host dict holding the relevant target(destination) - information - :returns: moved - :returns: list + :param host: the host dict holding the relevant target information + :returns: boolean -- Always returns True + :returns: dict -- Empty dict {} """ return self.common.migrate_volume(ctxt, volume, host) def retype(self, ctxt, volume, new_type, diff, host): """Migrate volume to another host using retype. - :param self: reference to class - :param ctxt: + :param ctxt: context :param volume: the volume object including the volume_type_id :param new_type: the new volume type. - :param host: the host dict holding the relevant target(destination) - information - :returns: moved - {} + :param diff: Unused parameter in common.retype + :param host: the host dict holding the relevant target information + :returns: boolean -- True if retype succeeded, False if error """ return self.common.retype(ctxt, volume, new_type, diff, host) @@ -302,27 +349,65 @@ def create_consistencygroup(self, context, group): """Creates a consistencygroup.""" self.common.create_consistencygroup(context, group) - def delete_consistencygroup(self, context, group): + def delete_consistencygroup(self, context, group, volumes): """Deletes a consistency group.""" - volumes = self.db.volume_get_all_by_group(context, group['id']) return self.common.delete_consistencygroup( context, group, volumes) - def create_cgsnapshot(self, context, cgsnapshot): + def create_cgsnapshot(self, context, cgsnapshot, snapshots): """Creates a cgsnapshot.""" - return self.common.create_cgsnapshot(context, cgsnapshot, self.db) + return self.common.create_cgsnapshot(context, cgsnapshot, snapshots) - def delete_cgsnapshot(self, context, cgsnapshot): + def delete_cgsnapshot(self, context, cgsnapshot, snapshots): """Deletes a cgsnapshot.""" - return self.common.delete_cgsnapshot(context, cgsnapshot, self.db) + return self.common.delete_cgsnapshot(context, cgsnapshot, snapshots) + + def manage_existing(self, volume, external_ref): + """Manages an existing VMAX Volume (import to Cinder). + + Renames the Volume to match the expected name for the volume. + Also need to consider things like QoS, Emulation, account/tenant. + """ + return self.common.manage_existing(volume, external_ref) + + def manage_existing_get_size(self, volume, external_ref): + """Return size of an existing VMAX volume to manage_existing. - def _check_for_iscsi_ip_address(self): - """Check to see if iscsi_ip_address is set in cinder.conf + :param self: reference to class + :param volume: the volume object including the volume_type_id + :param external_ref: reference to the existing volume + :returns: size of the volume in GB + """ + return self.common.manage_existing_get_size(volume, external_ref) - :returns: True/False + def unmanage(self, volume): + """Export VMAX volume from Cinder. + + Leave the volume intact on the backend array. + """ + return self.common.unmanage(volume) + + def update_consistencygroup(self, context, group, + add_volumes, remove_volumes): + """Updates LUNs in consistency group.""" + return self.common.update_consistencygroup(group, add_volumes, + remove_volumes) + + def create_consistencygroup_from_src(self, context, group, volumes, + cgsnapshot=None, snapshots=None, + source_cg=None, source_vols=None): + """Creates the consistency group from source. + + Currently the source can only be a cgsnapshot. + + :param context: the context + :param group: the consistency group object to be created + :param volumes: volumes in the consistency group + :param cgsnapshot: the source consistency group snapshot + :param snapshots: snapshots of the source volumes + :param source_cg: the dictionary of a consistency group as source. + :param source_vols: a list of volume dictionaries in the source_cg. """ - bExists = os.path.exists(CINDER_CONF) - if bExists: - if 'iscsi_ip_address' in open(CINDER_CONF).read(): - return True - return False + return self.common.create_consistencygroup_from_src( + context, group, volumes, cgsnapshot, snapshots, source_cg, + source_vols) diff --git a/emc_vmax_masking.py b/emc_vmax_masking.py index e7b4538..cf2ed8f 100644 --- a/emc_vmax_masking.py +++ b/emc_vmax_masking.py @@ -1,4 +1,4 @@ -# Copyright (c) 2012 - 2014 EMC Corporation. +# Copyright (c) 2012 - 2015 EMC Corporation. # All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); you may @@ -13,11 +13,12 @@ # License for the specific language governing permissions and limitations # under the License. +from oslo_concurrency import lockutils +from oslo_log import log as logging import six from cinder import exception from cinder.i18n import _, _LE, _LI, _LW -from cinder.openstack.common import log as logging from cinder.volume.drivers.emc import emc_vmax_fast from cinder.volume.drivers.emc import emc_vmax_provision from cinder.volume.drivers.emc import emc_vmax_provision_v3 @@ -33,6 +34,8 @@ FC = 'fc' EMC_ROOT = 'root/emc' +FASTPOLICY = 'storagetype:fastpolicy' +ISV3 = 'isV3' class EMCVMAXMasking(object): @@ -49,17 +52,30 @@ def __init__(self, prtcl): self.provision = emc_vmax_provision.EMCVMAXProvision(prtcl) self.provisionv3 = emc_vmax_provision_v3.EMCVMAXProvisionV3(prtcl) - def get_or_create_masking_view_and_map_lun(self, conn, maskingViewDict): + def setup_masking_view(self, conn, maskingViewDict, extraSpecs): + + @lockutils.synchronized(maskingViewDict['maskingViewName'], + "emc-mv-", True) + def do_get_or_create_masking_view_and_map_lun(): + return self.get_or_create_masking_view_and_map_lun(conn, + maskingViewDict, + extraSpecs) + return do_get_or_create_masking_view_and_map_lun() + + def get_or_create_masking_view_and_map_lun(self, conn, maskingViewDict, + extraSpecs): """Get or Create a masking view and add a volume to the storage group. Given a masking view tuple either get or create a masking view and add - the volume to the associated storage group + the volume to the associated storage group. If it is a live migration operation then we do not need to remove the volume from any storage group (default or otherwise). :param conn: the connection to ecom - :para maskingViewDict: the masking view tuple - :returns: dict rollbackDict + :param maskingViewDict: the masking view dict + :param extraSpecs: additional info + :returns: dict -- rollbackDict + :raises: VolumeBackendAPIException """ rollbackDict = {} @@ -69,34 +85,18 @@ def get_or_create_masking_view_and_map_lun(self, conn, maskingViewDict): volumeName = maskingViewDict['volumeName'] isV3 = maskingViewDict['isV3'] isLiveMigration = maskingViewDict['isLiveMigration'] + maskingViewDict['extraSpecs'] = extraSpecs defaultStorageGroupInstanceName = None fastPolicyName = None assocStorageGroupName = None + storageGroupInstanceName = None if isLiveMigration is False: if isV3: - assocStorageGroupInstanceName = ( - self.utils.get_storage_group_from_volume( - conn, volumeInstance.path)) - instance = conn.GetInstance( - assocStorageGroupInstanceName, LocalOnly=False) - assocStorageGroupName = instance['ElementName'] - defaultSgGroupName = self.utils.get_v3_storage_group_name( - maskingViewDict['pool'], - maskingViewDict['slo'], - maskingViewDict['workload']) - - if assocStorageGroupName != defaultSgGroupName: - LOG.warn(_LW( - "Volume: %(volumeName)s Does not belong " - "to storage storage group %(defaultSgGroupName)s. "), - {'volumeName': volumeName, - 'defaultSgGroupName': defaultSgGroupName}) - defaultStorageGroupInstanceName = assocStorageGroupInstanceName - - self._get_and_remove_from_storage_group_v3( - conn, controllerConfigService, volumeInstance.path, - volumeName, maskingViewDict, - defaultStorageGroupInstanceName) + defaultStorageGroupInstanceName = ( + self._get_v3_default_storagegroup_instancename( + conn, volumeInstance, maskingViewDict, + controllerConfigService, volumeName)) + else: fastPolicyName = maskingViewDict['fastPolicy'] # If FAST is enabled remove the volume from the default SG. @@ -105,22 +105,42 @@ def get_or_create_masking_view_and_map_lun(self, conn, maskingViewDict): self._get_and_remove_from_storage_group_v2( conn, controllerConfigService, volumeInstance.path, - volumeName, fastPolicyName)) - - # Validate new or existing masking view. - # Return the storage group so we can add the volume to it. - maskingViewInstanceName, storageGroupInstanceName, errorMessage = ( - self._validate_masking_view(conn, maskingViewDict, - defaultStorageGroupInstanceName)) + volumeName, fastPolicyName, + extraSpecs)) + else: + # Live Migration + self.remove_and_reset_members( + conn, controllerConfigService, volumeInstance, volumeName, + extraSpecs, maskingViewDict['connector'], False) - LOG.debug( - "The masking view in the attach operation is " - "%(maskingViewInstanceName)s. ", - {'maskingViewInstanceName': maskingViewInstanceName}) + # If anything has gone wrong with the masking view we rollback + try: + maskingViewInstanceName, storageGroupInstanceName, errorMessage = ( + self._validate_masking_view(conn, maskingViewDict, + defaultStorageGroupInstanceName, + extraSpecs)) + LOG.debug( + "The masking view in the attach operation is " + "%(maskingViewInstanceName)s. The storage group " + "in the masking view is %(storageGroupInstanceName)s.", + {'maskingViewInstanceName': maskingViewInstanceName, + 'storageGroupInstanceName': storageGroupInstanceName}) + except Exception as e: + LOG.exception(_LE( + "Masking View creation or retrieval was not successful " + "for masking view %(maskingViewName)s. " + "Attempting rollback."), + {'maskingViewName': maskingViewDict['maskingViewName']}) + errorMessage = e + + rollbackDict['pgGroupName'], errorMessage = ( + self._get_port_group_name_from_mv( + conn, maskingViewDict['maskingViewName'], + maskingViewDict['storageSystemName'])) if not errorMessage: - # Only after the masking view has been validated, add the volume - # to the storage group and recheck that it has been + # Only after the masking view has been validated, add the + # volume to the storage group and recheck that it has been # successfully added. errorMessage = self._check_adding_volume_to_storage_group( conn, maskingViewDict, storageGroupInstanceName) @@ -132,14 +152,16 @@ def get_or_create_masking_view_and_map_lun(self, conn, maskingViewDict): rollbackDict['volumeName'] = volumeName rollbackDict['fastPolicyName'] = fastPolicyName rollbackDict['isV3'] = isV3 + rollbackDict['extraSpecs'] = extraSpecs + rollbackDict['sgName'] = maskingViewDict['sgGroupName'] if errorMessage: # Rollback code if we cannot complete any of the steps above # successfully then we must roll back by adding the volume back to # the default storage group for that fast policy. if (fastPolicyName is not None): - # if the errorMessage was returned before the volume - # was removed from the default storage group no action + # If the errorMessage was returned before the volume + # was removed from the default storage group no action. self._check_if_rollback_action_for_masking_required( conn, rollbackDict) if isV3: @@ -151,8 +173,8 @@ def get_or_create_masking_view_and_map_lun(self, conn, maskingViewDict): exceptionMessage = (_( "Failed to get, create or add volume %(volumeName)s " - "to masking view %(maskingViewName)s " - "The error message received was %(errorMessage)s. ") + "to masking view %(maskingViewName)s. " + "The error message received was %(errorMessage)s.") % {'maskingViewName': maskingViewName, 'volumeName': volumeName, 'errorMessage': errorMessage}) @@ -161,15 +183,53 @@ def get_or_create_masking_view_and_map_lun(self, conn, maskingViewDict): return rollbackDict + def _get_v3_default_storagegroup_instancename(self, conn, volumeinstance, + maskingviewdict, + controllerConfigService, + volumeName): + defaultStorageGroupInstanceName = None + defaultSgGroupName = self.utils.get_v3_storage_group_name( + maskingviewdict['pool'], + maskingviewdict['slo'], + maskingviewdict['workload']) + assocStorageGroupInstanceNames = ( + self.utils.get_storage_groups_from_volume( + conn, volumeinstance.path)) + for assocStorageGroupInstanceName in ( + assocStorageGroupInstanceNames): + instance = conn.GetInstance( + assocStorageGroupInstanceName, LocalOnly=False) + assocStorageGroupName = instance['ElementName'] + + if assocStorageGroupName == defaultSgGroupName: + defaultStorageGroupInstanceName = ( + assocStorageGroupInstanceName) + break + if defaultStorageGroupInstanceName: + self._get_and_remove_from_storage_group_v3( + conn, controllerConfigService, volumeinstance.path, + volumeName, maskingviewdict, + defaultStorageGroupInstanceName) + else: + LOG.warning(_LW( + "Volume: %(volumeName)s does not belong " + "to storage group %(defaultSgGroupName)s."), + {'volumeName': volumeName, + 'defaultSgGroupName': defaultSgGroupName}) + return defaultStorageGroupInstanceName + def _validate_masking_view(self, conn, maskingViewDict, - defaultStorageGroupInstanceName): + defaultStorageGroupInstanceName, + extraSpecs): """Validate all the individual pieces of the masking view. - :param conn - the ecom connection - :param maskingViewDict - the masking view dictionary - :param defaultStorageGroupInstanceName - the default SG - :returns: maskingViewInstanceName, storageGroupInstanceName, - errorMessage + :param conn: the ecom connection + :param maskingViewDict: the masking view dictionary + :param defaultStorageGroupInstanceName: the default SG + :param extraSpecs: extra specifications + :returns: maskingViewInstanceName + :returns: storageGroupInstanceName, + :returns: string -- errorMessage """ storageSystemName = maskingViewDict['storageSystemName'] maskingViewName = maskingViewDict['maskingViewName'] @@ -179,24 +239,29 @@ def _validate_masking_view(self, conn, maskingViewDict, if maskingViewInstanceName is None: maskingViewInstanceName, storageGroupInstanceName, errorMessage = ( self._validate_new_masking_view( - conn, maskingViewDict, defaultStorageGroupInstanceName)) + conn, maskingViewDict, defaultStorageGroupInstanceName, + extraSpecs)) else: storageGroupInstanceName, errorMessage = ( self._validate_existing_masking_view( - conn, maskingViewDict, maskingViewInstanceName)) + conn, maskingViewDict, maskingViewInstanceName, + extraSpecs)) return maskingViewInstanceName, storageGroupInstanceName, errorMessage def _validate_new_masking_view(self, conn, maskingViewDict, - defaultStorageGroupInstanceName): + defaultStorageGroupInstanceName, + extraSpecs): """Validate the creation of a new masking view. - :param conn - the ecom connection - :param maskingViewDict - the masking view dictionary - :param defaultStorageGroupInstanceName - the default SG - :returns: maskingViewInstanceName, storageGroupInstanceName, - errorMessage + :param conn: the ecom connection + :param maskingViewDict: the masking view dictionary + :param defaultStorageGroupInstanceName: the default SG + :param extraSpecs: extra specifications + :returns: maskingViewInstanceName + :returns: storageGroupInstanceName, + :returns: string -- errorMessage """ controllerConfigService = maskingViewDict['controllerConfigService'] igGroupName = maskingViewDict['igGroupName'] @@ -220,14 +285,14 @@ def _validate_new_masking_view(self, conn, maskingViewDict, initiatorGroupInstanceName, errorMessage = ( self._check_initiator_group(conn, controllerConfigService, igGroupName, connector, - storageSystemName)) + storageSystemName, extraSpecs)) if errorMessage: return None, storageGroupInstanceName, errorMessage # Only after the components of the MV have been validated, # add the volume to the storage group and recheck that it # has been successfully added. This is necessary before - # creating a new masking view + # creating a new masking view. errorMessage = self._check_adding_volume_to_storage_group( conn, maskingViewDict, storageGroupInstanceName) if errorMessage: @@ -237,35 +302,38 @@ def _validate_new_masking_view(self, conn, maskingViewDict, self._check_masking_view( conn, controllerConfigService, maskingViewName, storageGroupInstanceName, - portGroupInstanceName, initiatorGroupInstanceName)) + portGroupInstanceName, initiatorGroupInstanceName, + extraSpecs)) return maskingViewInstanceName, storageGroupInstanceName, errorMessage def _validate_existing_masking_view(self, conn, maskingViewDict, - maskingViewInstanceName): + maskingViewInstanceName, extraSpecs): """Validate the components of an existing masking view. - :param conn - the ecom connection - :param maskingViewDict - the masking view dictionary - :param maskingViewInstanceName - the masking view instance name - :returns: storageGroupInstanceName, errorMessage + :param conn: the ecom connection + :param maskingViewDict: the masking view dictionary + :param maskingViewInstanceName: the masking view instance name + :param extraSpecs: extra specification + :returns: storageGroupInstanceName + :returns: string -- errorMessage """ storageGroupInstanceName = None controllerConfigService = maskingViewDict['controllerConfigService'] sgGroupName = maskingViewDict['sgGroupName'] - igGroupName = maskingViewDict['igGroupName'] - connector = maskingViewDict['connector'] - storageSystemName = maskingViewDict['storageSystemName'] - maskingViewName = maskingViewDict['maskingViewName'] + # igGroupName = maskingViewDict['igGroupName'] + # connector = maskingViewDict['connector'] + # storageSystemName = maskingViewDict['storageSystemName'] + # maskingViewName = maskingViewDict['maskingViewName'] # First verify that the initiator group matches the initiators. - errorMessage = self._check_existing_initiator_group( - conn, controllerConfigService, maskingViewName, - connector, storageSystemName, igGroupName) + # errorMessage = self._check_existing_initiator_group( + # conn, controllerConfigService, maskingViewName, + # connector, storageSystemName, igGroupName, extraSpecs) - if errorMessage: - return storageGroupInstanceName, errorMessage + # if errorMessage: + # return storageGroupInstanceName, errorMessage storageGroupInstanceName, errorMessage = ( self._check_existing_storage_group( @@ -278,199 +346,244 @@ def _check_storage_group(self, conn, maskingViewDict, storageGroupInstanceName): """Get the storage group and return it. - :param conn - the ecom connection - :param maskingViewDict - the masking view dictionary - :param defaultStorageGroupInstanceName - the default SG - :returns: storageGroupInstanceName, exceptionMessage + :param conn: the ecom connection + :param maskingViewDict: the masking view dictionary + :param storageGroupInstanceName: default storage group instance name + :returns: storageGroupInstanceName + :returns: string -- msg, the error message """ - exceptionMessage = None + msg = None storageGroupInstanceName = ( self._get_storage_group_instance_name( conn, maskingViewDict, storageGroupInstanceName)) if storageGroupInstanceName is None: - exceptionMessage = (_( + # This may be used in exception hence _ instead of _LE. + msg = (_( "Cannot get or create a storage group: %(sgGroupName)s" - " for volume %(volumeName)s ") - % {'sgGroupName': maskingViewDict['sgGroupName'], - 'volumeName': maskingViewDict['volumeName']}) - LOG.error(exceptionMessage) - return storageGroupInstanceName, exceptionMessage + " for volume %(volumeName)s ") % + {'sgGroupName': maskingViewDict['sgGroupName'], + 'volumeName': maskingViewDict['volumeName']}) + LOG.error(msg) + return storageGroupInstanceName, msg def _check_existing_storage_group( self, conn, controllerConfigService, sgGroupName, maskingViewInstanceName): """Check that we can get the existing storage group. - :param conn - the ecom connection - :param controllerConfigService - controller configuration service - :param sgGroupName - the storage group name - :param maskingViewInstanceName - the masking view instance name - - :returns: storageGroupInstanceName, exceptionMessage + :param conn: the ecom connection + :param controllerConfigService: controller configuration service + :param sgGroupName: the storage group name + :param maskingViewInstanceName: the masking view instance name + :returns: storageGroupInstanceName + :returns: string -- msg, the error message """ - exceptionMessage = None + msg = None sgFromMvInstanceName = ( self._get_storage_group_from_masking_view_instance( conn, maskingViewInstanceName)) if sgFromMvInstanceName is None: - exceptionMessage = (_( - "Cannot get storage group: %(sgGroupName)s" - " from masking view %(maskingViewInstanceName)s ") - % {'sgGroupName': sgGroupName, - 'maskingViewInstanceName': maskingViewInstanceName}) - LOG.error(exceptionMessage) - return sgFromMvInstanceName, exceptionMessage + # This may be used in exception hence _ instead of _LE. + msg = (_( + "Cannot get storage group: %(sgGroupName)s " + "from masking view %(maskingViewInstanceName)s. ") % + {'sgGroupName': sgGroupName, + 'maskingViewInstanceName': maskingViewInstanceName}) + LOG.error(msg) + return sgFromMvInstanceName, msg def _check_port_group(self, conn, controllerConfigService, pgGroupName): """Check that you can either get or create a port group. - :param conn - the ecom connection - :param controllerConfigService - controller configuration service - :param pgGroupName - the port group Name - :returns: portGroupInstanceName, exceptionMessage + :param conn: the ecom connection + :param controllerConfigService: controller configuration service + :param pgGroupName: the port group Name + :returns: portGroupInstanceName + :returns: string -- msg, the error message """ - exceptionMessage = None + msg = None portGroupInstanceName = self._get_port_group_instance_name( conn, controllerConfigService, pgGroupName) if portGroupInstanceName is None: - exceptionMessage = (_( - "Cannot get port group: %(pgGroupName)s. ") - % {'pgGroupName': pgGroupName}) - LOG.error(exceptionMessage) + # This may be used in exception hence _ instead of _LE. + msg = (_( + "Cannot get port group: %(pgGroupName)s. ") % + {'pgGroupName': pgGroupName}) + LOG.error(msg) - return portGroupInstanceName, exceptionMessage + return portGroupInstanceName, msg def _check_initiator_group( self, conn, controllerConfigService, igGroupName, - connector, storageSystemName): - """Check that initiator group can be either got or created. + connector, storageSystemName, extraSpecs): + """Check that initiator group can be either retrieved or created. - :param conn - the ecom connection - :param controllerConfigService - controller configuration service - :param igGroupName - the initiator group Name - :param connector - :param storageSystemName - the storage system name - :returns: initiatorGroupInstanceName, exceptionMessage + :param conn: the ecom connection + :param controllerConfigService: controller configuration service + :param igGroupName: the initiator group Name + :param connector: the connector object + :param storageSystemName: the storage system name + :param extraSpecs: extra specifications + :returns: initiatorGroupInstanceName + :returns: string -- the error message """ - exceptionMessage = None + msg = None initiatorGroupInstanceName = ( self._get_initiator_group_instance_name( conn, controllerConfigService, igGroupName, connector, - storageSystemName)) + storageSystemName, extraSpecs)) if initiatorGroupInstanceName is None: - exceptionMessage = (_( + # This may be used in exception hence _ instead of _LE. + msg = (_( "Cannot get or create initiator group: " - "%(igGroupName)s. ") - % {'igGroupName': igGroupName}) - LOG.error(exceptionMessage) + "%(igGroupName)s. ") % + {'igGroupName': igGroupName}) + LOG.error(msg) - return initiatorGroupInstanceName, exceptionMessage + return initiatorGroupInstanceName, msg def _check_existing_initiator_group( self, conn, controllerConfigService, maskingViewName, - connector, storageSystemName, igGroupName): + connector, storageSystemName, igGroupName, extraSpecs): """Check that existing initiator group in the masking view. Check if the initiators in the initiator group match those in the system. - :param controllerConfigService - controller configuration service - :param maskingViewName - the masking view name - :param connector - the connector object - :param storageSystemName - the storage system name - :param igGroupName - the initiator group name - :returns: maskingViewInstanceName, exceptionMessage + :param conn: the ecom connection + :param controllerConfigService: controller configuration service + :param maskingViewName: the masking view name + :param connector: the connector object + :param storageSystemName: the storage system name + :param igGroupName: the initiator group name + :param extraSpecs: extra specification + :returns: string -- msg, the error message """ - exceptionMessage = None + msg = None if not self._verify_initiator_group_from_masking_view( conn, controllerConfigService, maskingViewName, - connector, storageSystemName, igGroupName): - exceptionMessage = (_( + connector, storageSystemName, igGroupName, + extraSpecs): + # This may be used in exception hence _ instead of _LE. + msg = (_( "Unable to verify initiator group: %(igGroupName)s " - "in masking view %(maskingViewName)s. ") - % {'igGroupName': igGroupName, - 'maskingViewName': maskingViewName}) - LOG.error(exceptionMessage) - return exceptionMessage + "in masking view %(maskingViewName)s. ") % + {'igGroupName': igGroupName, + 'maskingViewName': maskingViewName}) + LOG.error(msg) + return msg def _check_masking_view( self, conn, controllerConfigService, maskingViewName, storageGroupInstanceName, - portGroupInstanceName, initiatorGroupInstanceName): + portGroupInstanceName, initiatorGroupInstanceName, extraSpecs): """Check that masking view can be either got or created. - :param controllerConfigService - controller configuration service - :param maskingViewName - the masking view name - :param storageGroupInstanceName - storage group instance name - :param portGroupInstanceName - port group instance name - :param initiatorGroupInstanceName - the initiator group instance name - :returns: maskingViewInstanceName, exceptionMessage + :param conn: the ecom connection + :param controllerConfigService: controller configuration service + :param maskingViewName: the masking view name + :param storageGroupInstanceName: storage group instance name + :param portGroupInstanceName: port group instance name + :param initiatorGroupInstanceName: the initiator group instance name + :param extraSpecs: extra specifications + :returns: maskingViewInstanceName + :returns: string -- msg, the error message """ - exceptionMessage = None + msg = None maskingViewInstanceName = ( self._get_masking_view_instance_name( conn, controllerConfigService, maskingViewName, storageGroupInstanceName, portGroupInstanceName, - initiatorGroupInstanceName)) + initiatorGroupInstanceName, extraSpecs)) if maskingViewInstanceName is None: - exceptionMessage = (_( - "Cannot create masking view: %(maskingViewName)s. ") - % {'maskingViewName': maskingViewName}) - LOG.error(exceptionMessage) + # This may be used in exception hence _ instead of _LE. + msg = (_( + "Cannot create masking view: %(maskingViewName)s. ") % + {'maskingViewName': maskingViewName}) + LOG.error(msg) - return maskingViewInstanceName, exceptionMessage + return maskingViewInstanceName, msg def _check_adding_volume_to_storage_group( self, conn, maskingViewDict, storageGroupInstanceName): """Add the volume to the storage group and double check it is there. - :param conn - the ecom connection - :param maskingViewDict - the masking view dictionary - :returns: exceptionMessage + :param conn: the ecom connection + :param maskingViewDict: the masking view dictionary + :param storageGroupInstanceName: storage group instance name + :returns: string -- the error message """ controllerConfigService = maskingViewDict['controllerConfigService'] sgGroupName = maskingViewDict['sgGroupName'] volumeInstance = maskingViewDict['volumeInstance'] - storageSystemName = maskingViewDict['storageSystemName'] volumeName = maskingViewDict['volumeName'] - exceptionMessage = None + msg = None if self._is_volume_in_storage_group( conn, storageGroupInstanceName, - volumeInstance): - LOG.warn(_LW( + volumeInstance, sgGroupName): + LOG.warning(_LW( "Volume: %(volumeName)s is already part " - "of storage group %(sgGroupName)s. "), + "of storage group %(sgGroupName)s."), {'volumeName': volumeName, 'sgGroupName': sgGroupName}) else: - self.add_volume_to_storage_group( - conn, controllerConfigService, - storageGroupInstanceName, volumeInstance, volumeName, - sgGroupName, storageSystemName) - if not self._is_volume_in_storage_group( - conn, storageGroupInstanceName, - volumeInstance): - exceptionMessage = (_( - "Volume: %(volumeName)s was not added " - "to storage group %(sgGroupName)s. ") - % {'volumeName': volumeName, - 'sgGroupName': sgGroupName}) - LOG.error(exceptionMessage) + msg = self._add_volume_to_sg_and_verify( + conn, controllerConfigService, storageGroupInstanceName, + volumeInstance, volumeName, sgGroupName, + maskingViewDict['extraSpecs']) + + return msg + + def _add_volume_to_sg_and_verify( + self, conn, controllerConfigService, storageGroupInstanceName, + volumeInstance, volumeName, sgGroupName, extraSpecs): + """Add the volume to the storage group and double check it is there. - return exceptionMessage + :param conn: the ecom connection + :param controllerConfigService: controller service + :param storageGroupInstanceName: storage group instance name + :param volumeInstance: the volume instance + :param volumeName: the volume name + :param sgGroupName: the storage group name + :param extraSpecs: the extra specifications + :returns: string -- the error message + """ + msg = None + self.add_volume_to_storage_group( + conn, controllerConfigService, storageGroupInstanceName, + volumeInstance, volumeName, sgGroupName, extraSpecs) + if not self._is_volume_in_storage_group( + conn, storageGroupInstanceName, volumeInstance, sgGroupName): + # This may be used in exception hence _ instead of _LE. + msg = (_( + "Volume: %(volumeName)s was not added " + "to storage group %(sgGroupName)s.") % + {'volumeName': volumeName, + 'sgGroupName': sgGroupName}) + LOG.error(msg) + else: + LOG.info(_LI("Successfully added %(volumeName)s to " + "%(sgGroupName)s."), + {'volumeName': volumeName, + 'sgGroupName': sgGroupName}) + return msg def _get_and_remove_from_storage_group_v2( self, conn, controllerConfigService, volumeInstanceName, - volumeName, fastPolicyName): + volumeName, fastPolicyName, extraSpecs): """Get the storage group and remove volume from it. - :param controllerConfigService - controller configuration service - :param volumeInstanceName - volume instance name - :param volumeName - volume name - :param fastPolicyName - fast name + :param conn: the ecom connection + :param controllerConfigService: controller configuration service + :param volumeInstanceName: volume instance name + :param volumeName: volume name + :param fastPolicyName: fast name + :param extraSpecs: additional info + :returns: defaultStorageGroupInstanceName + :raises: VolumeBackendAPIException """ defaultStorageGroupInstanceName = ( self.fast.get_and_verify_default_storage_group( @@ -479,7 +592,7 @@ def _get_and_remove_from_storage_group_v2( if defaultStorageGroupInstanceName is None: exceptionMessage = (_( "Cannot get the default storage group for FAST policy: " - "%(fastPolicyName)s. ") + "%(fastPolicyName)s.") % {'fastPolicyName': fastPolicyName}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException( @@ -488,62 +601,51 @@ def _get_and_remove_from_storage_group_v2( retStorageGroupInstanceName = ( self.remove_device_from_default_storage_group( conn, controllerConfigService, volumeInstanceName, - volumeName, fastPolicyName)) + volumeName, fastPolicyName, extraSpecs)) if retStorageGroupInstanceName is None: exceptionMessage = (_( - "Failed to remove volume %(volumeName)s from default SG: " - "%(volumeName)s. ") + "Failed to remove volume %(volumeName)s from default SG.") % {'volumeName': volumeName}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException( data=exceptionMessage) + return defaultStorageGroupInstanceName def _get_and_remove_from_storage_group_v3( self, conn, controllerConfigService, volumeInstanceName, volumeName, maskingViewDict, storageGroupInstanceName): """Get the storage group and remove volume from it. - :param controllerConfigService - controller configuration service - :param volumeInstanceName - volume instance name - :param volumeName - volume name - :param fastPolicyName - fast name + :param conn: the ecom connection + :param controllerConfigService: controller configuration service + :param volumeInstanceName: volume instance name + :param volumeName: volume name + :param maskingViewDict: the masking view dictionary + :param storageGroupInstanceName: storage group instance name + :raises: VolumeBackendAPIException """ + volInstance = conn.GetInstance(volumeInstanceName, LocalOnly=False) - assocVolumeInstanceNames = self.get_devices_from_storage_group( - conn, storageGroupInstanceName) - LOG.debug( - "There are %(length)lu associated with the default storage group " - "before removing volume %(volumeName)s.", - {'length': len(assocVolumeInstanceNames), - 'volumeName': volumeName}) - - self.provision.remove_device_from_storage_group( + self._remove_volume_from_sg( conn, controllerConfigService, storageGroupInstanceName, - volumeInstanceName, volumeName) + volInstance, maskingViewDict['extraSpecs']) - assocVolumeInstanceNames = self.get_devices_from_storage_group( - conn, storageGroupInstanceName) - LOG.debug( - "There are %(length)lu associated with the default storage group " - "after removing volume %(volumeName)s.", - {'length': len(assocVolumeInstanceNames), - 'volumeName': volumeName}) - - # required for unit tests + # Required for unit tests. emptyStorageGroupInstanceName = ( - self._wrap_get_storage_group_from_volume(conn, volumeInstanceName)) + self._wrap_get_storage_group_from_volume( + conn, volumeInstanceName, maskingViewDict['sgGroupName'])) if emptyStorageGroupInstanceName is not None: exceptionMessage = (_( "Failed to remove volume %(volumeName)s from default SG: " - "%(volumeName)s. ") + "%(volumeName)s.") % {'volumeName': volumeName}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException( data=exceptionMessage) def _is_volume_in_storage_group( - self, conn, storageGroupInstanceName, volumeInstance): + self, conn, storageGroupInstanceName, volumeInstance, sgName): """Check if the volume is already part of the storage group. Check if the volume is already part of the storage group, @@ -552,24 +654,25 @@ def _is_volume_in_storage_group( :param conn: the connection to ecom :param storageGroupInstanceName: the storage group instance name :param volumeInstance: the volume instance - :returns: boolean True/False + :param sgName: the storage group name + :returns: boolean """ foundStorageGroupInstanceName = ( self.utils.get_storage_group_from_volume( - conn, volumeInstance.path)) + conn, volumeInstance.path, sgName)) if foundStorageGroupInstanceName is not None: storageGroupInstance = conn.GetInstance( storageGroupInstanceName, LocalOnly=False) LOG.debug( "The existing storage group instance element name is: " - "%(existingElement)s. ", + "%(existingElement)s.", {'existingElement': storageGroupInstance['ElementName']}) foundStorageGroupInstance = conn.GetInstance( foundStorageGroupInstanceName, LocalOnly=False) LOG.debug( "The found storage group instance element name is: " - "%(foundElement)s. ", + "%(foundElement)s.", {'foundElement': foundStorageGroupInstance['ElementName']}) if (foundStorageGroupInstance['ElementName'] == ( storageGroupInstance['ElementName'])): @@ -583,7 +686,7 @@ def _find_masking_view(self, conn, maskingViewName, storageSystemName): :param conn: connection to the ecom server :param maskingViewName: the masking view name :param storageSystemName: the storage system name(String) - :returns: foundMaskingViewInstanceName masking view instance name + :returns: dict -- foundMaskingViewInstanceName """ foundMaskingViewInstanceName = None @@ -610,33 +713,27 @@ def _find_masking_view(self, conn, maskingViewName, storageSystemName): {'maskingViewName': maskingViewName}) else: LOG.info(_LI( - "Found existing masking view: %(maskingViewName)s "), + "Found existing masking view: %(maskingViewName)s."), {'maskingViewName': maskingViewName}) return foundMaskingViewInstanceName def _create_storage_group( - self, conn, maskingViewDict, - defaultStorageGroupInstanceName): + self, conn, maskingViewDict, defaultStorageGroupInstanceName): """Create a new storage group that doesn't already exist. If fastPolicyName is not none we attempt to remove it from the default storage group of that policy and associate to the new storage group that will be part of the masking view. Will not handle any exception in this method it will be handled - up the stack + up the stack. - :param conn: connection the ecom server - :param controllerConfigService: the controller configuration service - :param storageGroupName: the proposed group name (String) - :param volumeInstance: useful information on the volume - :param fastPolicyName: the fast policy name (String) can be None - :param volumeName: the volume name (String) - :param storageSystemName: the storage system name (String) + :param conn: connection to the ecom server + :param maskingViewDict: the masking view dictionary :param defaultStorageGroupInstanceName: the default storage group - instance name (Can be None) + instance name (Can be None) :returns: foundStorageGroupInstanceName the instance Name of the - storage group + storage group """ failedRet = None controllerConfigService = maskingViewDict['controllerConfigService'] @@ -650,48 +747,49 @@ def _create_storage_group( foundStorageGroupInstanceName = ( self.provisionv3.create_storage_group_v3( conn, controllerConfigService, storageGroupName, - pool, slo, workload)) + pool, slo, workload, maskingViewDict['extraSpecs'])) else: fastPolicyName = maskingViewDict['fastPolicy'] volumeInstance = maskingViewDict['volumeInstance'] foundStorageGroupInstanceName = ( self.provision.create_and_get_storage_group( conn, controllerConfigService, storageGroupName, - volumeInstance.path)) + volumeInstance.path, maskingViewDict['extraSpecs'])) if (fastPolicyName is not None and defaultStorageGroupInstanceName is not None): assocTierPolicyInstanceName = ( self.fast.add_storage_group_and_verify_tier_policy_assoc( conn, controllerConfigService, foundStorageGroupInstanceName, - storageGroupName, fastPolicyName)) + storageGroupName, fastPolicyName, + maskingViewDict['extraSpecs'])) if assocTierPolicyInstanceName is None: LOG.error(_LE( "Cannot add and verify tier policy association for " "storage group : %(storageGroupName)s to " - "FAST policy : %(fastPolicyName)s. "), + "FAST policy : %(fastPolicyName)s."), {'storageGroupName': storageGroupName, 'fastPolicyName': fastPolicyName}) return failedRet if foundStorageGroupInstanceName is None: LOG.error(_LE( - "Cannot get storage Group from job : %(storageGroupName)s. "), + "Cannot get storage Group from job : %(storageGroupName)s."), {'storageGroupName': storageGroupName}) return failedRet else: LOG.info(_LI( - "Created new storage group: %(storageGroupName)s "), + "Created new storage group: %(storageGroupName)s."), {'storageGroupName': storageGroupName}) return foundStorageGroupInstanceName - def _find_port_group(self, conn, controllerConfigService, portGroupName): + def find_port_group(self, conn, controllerConfigService, portGroupName): """Given the port Group name get the port group instance name. :param conn: connection to the ecom server :param controllerConfigService: the controller configuration service :param portGroupName: the name of the port group you are getting - :returns: foundPortGroup storage group instance name + :returns: foundPortGroupInstanceName """ foundPortGroupInstanceName = None portMaskingGroupInstances = conn.Associators( @@ -710,76 +808,84 @@ def _find_port_group(self, conn, controllerConfigService, portGroupName): if foundPortGroupInstanceName is None: LOG.error(_LE( - "Could not find port group : %(portGroupName)s. Check that the" - " EMC configuration file has the correct port group name. "), + "Could not find port group : %(portGroupName)s. Check that " + "the EMC configuration file has the correct port group name."), {'portGroupName': portGroupName}) return foundPortGroupInstanceName def _create_or_get_initiator_group( self, conn, controllerConfigService, igGroupName, - connector, storageSystemName): - """Attempt to create a initiatorGroup. - - If one already exists with the same Initiator/wwns then get it + connector, storageSystemName, extraSpecs): + """Attempt to create an initiatorGroup. + If one already exists with the same Initiator/wwns then get it. Check to see if an initiatorGroup already exists, that matches the - connector information + connector information. NOTE: An initiator/wwn can only belong to one initiatorGroup. If we were to attempt to create one with an initiator/wwn that - is already belong to another initiatorGroup, it would fail + is already belong to another initiatorGroup, it would fail. :param conn: connection to the ecom server :param controllerConfigService: the controller config Servicer :param igGroupName: the proposed name of the initiator group :param connector: the connector information to the host :param storageSystemName: the storage system name (String) + :param extraSpecs: extra specifications :returns: foundInitiatorGroupInstanceName """ - failedRet = None initiatorNames = self._find_initiator_names(conn, connector) - LOG.debug("The initiator name(s) are: %(initiatorNames)s ", + LOG.debug("The initiator name(s) are: %(initiatorNames)s.", {'initiatorNames': initiatorNames}) foundInitiatorGroupInstanceName = self._find_initiator_masking_group( conn, controllerConfigService, initiatorNames) # If you cannot find an initiatorGroup that matches the connector - # info create a new initiatorGroup + # info create a new initiatorGroup. if foundInitiatorGroupInstanceName is None: - # check that our connector information matches the - # hardwareId(s) on the symm + # Check that our connector information matches the + # hardwareId(s) on the vmax. storageHardwareIDInstanceNames = ( self._get_storage_hardware_id_instance_names( conn, initiatorNames, storageSystemName)) if not storageHardwareIDInstanceNames: - LOG.error(_LE( + LOG.info(_LI( "Initiator Name(s) %(initiatorNames)s are not on array " - "%(storageSystemName)s "), + "%(storageSystemName)s."), {'initiatorNames': initiatorNames, 'storageSystemName': storageSystemName}) - return failedRet + storageHardwareIDInstanceNames = ( + self._create_hardware_ids(conn, initiatorNames, + storageSystemName)) + if not storageHardwareIDInstanceNames: + msg = (_("Failed to create hardware id(s) on " + "%(storageSystemName)s.") + % {'storageSystemName': storageSystemName}) + LOG.error(msg) + raise exception.VolumeBackendAPIException(data=msg) foundInitiatorGroupInstanceName = self._create_initiator_Group( conn, controllerConfigService, igGroupName, - storageHardwareIDInstanceNames) + storageHardwareIDInstanceNames, extraSpecs) LOG.info(_LI( - "Created new initiator group name: %(igGroupName)s "), + "Created new initiator group name: %(igGroupName)s."), {'igGroupName': igGroupName}) else: LOG.info(_LI( - "Using existing initiator group name: %(igGroupName)s "), + "Using existing initiator group name: %(igGroupName)s."), {'igGroupName': igGroupName}) return foundInitiatorGroupInstanceName def _find_initiator_names(self, conn, connector): - """check the connector object for initiators(ISCSI) or wwpns(FC). + """Check the connector object for initiators(ISCSI) or wwpns(FC). :param conn: the connection to the ecom :param connector: the connector object - :returns list foundinitiatornames list of string initiator names + :returns: list -- list of found initiator names + :raises: VolumeBackendAPIException """ foundinitiatornames = [] name = 'initiator name' @@ -792,7 +898,7 @@ def _find_initiator_names(self, conn, connector): name = 'world wide port names' else: msg = (_("FC is the protocol but wwpns are " - "not supplied by Openstack")) + "not supplied by OpenStack.")) LOG.error(msg) raise exception.VolumeBackendAPIException(data=msg) @@ -814,11 +920,11 @@ def _find_initiator_masking_group( NOTE: An initiator/wwn can only belong to one initiatorGroup. If we were to attempt to create one with an initiator/wwn that is - already belong to another initiatorGroup, it would fail + already belong to another initiatorGroup, it would fail. :param conn: the connection to the ecom server :param controllerConfigService: the controller configuration service - :param initiatorName: the list of initiator names + :param initiatorNames: the list of initiator names :returns: foundInitiatorMaskingGroup """ foundInitiatorMaskingGroupInstanceName = None @@ -827,8 +933,8 @@ def _find_initiator_masking_group( conn.AssociatorNames(controllerConfigService, ResultClass='CIM_InitiatorMaskingGroup')) - for initiatorMaskingGroupInstanceName in \ - initiatorMaskingGroupInstanceNames: + for initiatorMaskingGroupInstanceName in ( + initiatorMaskingGroupInstanceNames): # Check that it hasn't been deleted. If it has, break out # of the for loop. instance = self.utils.get_existing_instance( @@ -845,8 +951,8 @@ def _find_initiator_masking_group( # we found the existing CIM_InitiatorMaskingGroup. hardwareid = storageHardwareIdInstance['StorageID'] for initiator in initiatorNames: - if six.text_type(hardwareid).lower() == \ - six.text_type(initiator).lower(): + if six.text_type(hardwareid).lower() == ( + six.text_type(initiator).lower()): foundInitiatorMaskingGroupInstanceName = ( initiatorMaskingGroupInstanceName) break @@ -863,9 +969,9 @@ def _get_storage_hardware_id_instance_names( """Given a list of initiator names find CIM_StorageHardwareID instance. :param conn: the connection to the ecom server - :param initiatorName: the list of initiator names + :param initiatorNames: the list of initiator names :param storageSystemName: the storage system name - :returns: foundHardwardIDsInstanceNames + :returns: list -- foundHardwardIDsInstanceNames """ foundHardwardIDsInstanceNames = [] @@ -880,17 +986,13 @@ def _get_storage_hardware_id_instance_names( for hardwareIdInstance in hardwareIdInstances: storageId = hardwareIdInstance['StorageID'] for initiatorName in initiatorNames: - LOG.debug("The storage Id is : %(storageId)s ", - {'storageId': storageId.lower()}) - LOG.debug("The initiatorName is : %(initiatorName)s ", - {'initiatorName': initiatorName.lower()}) if storageId.lower() == initiatorName.lower(): - # Check that the found hardwareId has been delete. + # Check that the found hardwareId has been deleted. # If it has, we don't want to add it to the list. instance = self.utils.get_existing_instance( conn, hardwareIdInstance.path) if instance is None: - # hardwareId doesn't exist. Skip it. + # HardwareId doesn't exist. Skip it. break foundHardwardIDsInstanceNames.append( @@ -898,17 +1000,17 @@ def _get_storage_hardware_id_instance_names( break LOG.debug( - "The found hardware IDs are : %(foundHardwardIDsInstanceNames)s ", + "The found hardware IDs are : %(foundHardwardIDsInstanceNames)s.", {'foundHardwardIDsInstanceNames': foundHardwardIDsInstanceNames}) return foundHardwardIDsInstanceNames def _get_initiator_group_from_job(self, conn, job): - """After creating an new initiator group find it and return it + """After creating an new initiator group find it and return it. :param conn: the connection to the ecom server :param job: the create initiator group job - :returns: dict initiatorDict + :returns: dict -- initiatorDict """ associators = conn.Associators( job['Job'], @@ -926,7 +1028,7 @@ def _get_initiator_group_from_job(self, conn, job): def _create_masking_view( self, conn, configService, maskingViewName, deviceMaskingGroup, - targetMaskingGroup, initiatorMaskingGroup): + targetMaskingGroup, initiatorMaskingGroup, extraSpecs): """After creating an new initiator group find it and return it. :param conn: the connection to the ecom server @@ -935,8 +1037,10 @@ def _create_masking_view( :param deviceMaskingGroup: device(storage) masking group (instanceName) :param targetMaskingGroup: target(port) masking group (instanceName) :param initiatorMaskingGroup: initiator masking group (instanceName) - :returns: int rc return code - :returns: dict job + :param extraSpecs: extra specifications + :returns: int -- return code + :returns: dict -- job + :raises: VolumeBackendAPIException """ rc, job = conn.InvokeMethod( 'CreateMaskingView', configService, ElementName=maskingViewName, @@ -944,12 +1048,13 @@ def _create_masking_view( DeviceMaskingGroup=deviceMaskingGroup, TargetMaskingGroup=targetMaskingGroup) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( "Error Create Masking View: %(groupName)s. " - "Return code: %(rc)lu. Error: %(error)s") + "Return code: %(rc)lu. Error: %(error)s.") % {'groupName': maskingViewName, 'rc': rc, 'error': errordesc}) @@ -957,16 +1062,17 @@ def _create_masking_view( raise exception.VolumeBackendAPIException( data=exceptionMessage) - LOG.info(_LI("Created new masking view : %(maskingViewName)s. "), - {'maskingViewName': maskingViewName}) + LOG.info(_LI( + "Created new masking view : %(maskingViewName)s."), + {'maskingViewName': maskingViewName}) return rc, job def find_new_masking_view(self, conn, jobDict): - """Find the newly created volume + """Find the newly created volume. :param conn: the connection to the ecom server - :param jobDict: the job tuple - :returns: instance maskingViewInstance + :param jobDict: the job dictionary + :returns: dict -- maskingViewInstance """ associators = conn.Associators( jobDict['Job'], @@ -1004,8 +1110,8 @@ def _get_storage_group_from_masking_view( {'view': maskingViewName, 'masking': foundStorageGroupInstanceName}) else: - LOG.warn(_LW("Unable to find Masking view: %(view)s. "), - {'view': maskingViewName}) + LOG.warning(_LW("Unable to find Masking view: %(view)s."), + {'view': maskingViewName}) return foundStorageGroupInstanceName @@ -1014,14 +1120,14 @@ def _get_storage_group_from_masking_view_instance( """Gets the Device Masking Group from masking view instance. :param conn: the connection to the ecom server - :param maskingViewInstance + :param maskingViewInstance: the masking view instance :returns: instance name foundStorageGroupInstanceName """ foundStorageGroupInstanceName = None groups = conn.AssociatorNames( maskingViewInstance, ResultClass='CIM_DeviceMaskingGroup') - if groups[0] > 0: + if len(groups) > 0: foundStorageGroupInstanceName = groups[0] return foundStorageGroupInstanceName @@ -1031,17 +1137,18 @@ def _get_storage_group_instance_name( defaultStorageGroupInstanceName): """Gets the storage group instance name. - If fastPolicy name is None - then NON FAST is assumed. If it is a valid fastPolicy name - then associate the new storage group with the fast policy. + If fastPolicy name is None then NON FAST is assumed. + If it is a valid fastPolicy name then associate the new storage + group with the fast policy. If we are using an existing storage group then we must check that - it is associated with the correct fast policy + it is associated with the correct fast policy. :param conn: the connection to the ecom server - :param maskingViewDict - the masking view dictionary + :param maskingViewDict: the masking view dictionary :param defaultStorageGroupInstanceName: default storage group instance - name (can be None for Non FAST) + name (can be None for Non FAST) :returns: instance name storageGroupInstanceName + :raises: VolumeBackendAPIException """ storageGroupInstanceName = self.utils.find_storage_masking_group( conn, maskingViewDict['controllerConfigService'], @@ -1054,7 +1161,7 @@ def _get_storage_group_instance_name( if storageGroupInstanceName is None: errorMessage = (_( "Cannot create or find an storage group with name " - "%(sgGroupName)s") + "%(sgGroupName)s.") % {'sgGroupName': maskingViewDict['sgGroupName']}) LOG.error(errorMessage) raise exception.VolumeBackendAPIException(data=errorMessage) @@ -1066,55 +1173,56 @@ def _get_port_group_instance_name( """Gets the port group instance name. The portGroup name has been defined in the EMC Config file if it - does not exist the operation should fail + does not exist the operation should fail. :param conn: the connection to the ecom server :param controllerConfigService: the controller configuration server :param pgGroupName: the port group name :returns: instance name foundPortGroupInstanceName """ - foundPortGroupInstanceName = self._find_port_group( + foundPortGroupInstanceName = self.find_port_group( conn, controllerConfigService, pgGroupName) if foundPortGroupInstanceName is None: LOG.error(_LE( "Cannot find a portGroup with name %(pgGroupName)s. " - "The port group for a masking view must be pre-defined"), + "The port group for a masking view must be pre-defined."), {'pgGroupName': pgGroupName}) return foundPortGroupInstanceName LOG.info(_LI( - "Port group instance name is %(foundPortGroupInstanceName)s"), + "Port group instance name is %(foundPortGroupInstanceName)s."), {'foundPortGroupInstanceName': foundPortGroupInstanceName}) return foundPortGroupInstanceName def _get_initiator_group_instance_name( self, conn, controllerConfigService, igGroupName, connector, - storageSystemName): + storageSystemName, extraSpecs): """Gets the initiator group instance name. :param conn: the connection to the ecom server :param controllerConfigService: the controller configuration server :param igGroupName: the port group name :param connector: the connector object - :param storageSystemName = the storage system name - :returns: instance name foundInitiatorGroupInstanceName + :param storageSystemName: the storage system name + :param extraSpecs: extra specifications + :returns: foundInitiatorGroupInstanceName """ foundInitiatorGroupInstanceName = (self._create_or_get_initiator_group( conn, controllerConfigService, igGroupName, connector, - storageSystemName)) + storageSystemName, extraSpecs)) if foundInitiatorGroupInstanceName is None: LOG.error(_LE( "Cannot create or find an initiator group with " - "name %(igGroupName)s"), + "name %(igGroupName)s."), {'igGroupName': igGroupName}) return foundInitiatorGroupInstanceName def _get_masking_view_instance_name( self, conn, controllerConfigService, maskingViewName, storageGroupInstanceName, portGroupInstanceName, - initiatorGroupInstanceName): - """Gets the masking view instance name + initiatorGroupInstanceName, extraSpecs): + """Gets the masking view instance name. :param conn: the connection to the ecom server :param controllerConfigService: the controller configuration server @@ -1122,18 +1230,19 @@ def _get_masking_view_instance_name( :param storageGroupInstanceName: the storage group instance name :param portGroupInstanceName: the port group instance name :param initiatorGroupInstanceName: the initiator group instance name + :param extraSpecs: extra specifications :returns: instance name foundMaskingViewInstanceName """ - rc, job = ( # @UnusedVariable + _rc, job = ( self._create_masking_view( conn, controllerConfigService, maskingViewName, storageGroupInstanceName, portGroupInstanceName, - initiatorGroupInstanceName)) + initiatorGroupInstanceName, extraSpecs)) foundMaskingViewInstanceName = self.find_new_masking_view(conn, job) if foundMaskingViewInstanceName is None: LOG.error(_LE( "Cannot find the new masking view just created with name " - "%(maskingViewName)s"), + "%(maskingViewName)s."), {'maskingViewName': maskingViewName}) return foundMaskingViewInstanceName @@ -1148,13 +1257,11 @@ def _check_if_rollback_action_for_masking_required( the exception occurred. :param conn: the connection to the ecom server - :param controllerConfigService: the controller config service - :param volumeInstanceName: the volume instance name - :param volumeName: the volume name (String) - :param fastPolicyName: the fast policy name (String) - :param defaultStorageGroupInstanceName: the default storage group - instance name + :param rollbackDict: the rollback dictionary + :returns: message + :raises: VolumeBackendAPIException """ + message = None try: if rollbackDict['isV3']: errorMessage = self._check_adding_volume_to_storage_group( @@ -1162,19 +1269,21 @@ def _check_if_rollback_action_for_masking_required( rollbackDict['defaultStorageGroupInstanceName']) if errorMessage: LOG.error(errorMessage) + message = (_("V3 rollback")) else: foundStorageGroupInstanceName = ( self.utils.get_storage_group_from_volume( - conn, rollbackDict['volumeInstance'].path)) - # volume is not associated with any storage group so add - # it back to the default - if len(foundStorageGroupInstanceName) == 0: - LOG.warn(_LW( + conn, rollbackDict['volumeInstance'].path, + rollbackDict['sgName'])) + # Volume is not associated with any storage group so add + # it back to the default. + if not foundStorageGroupInstanceName: + LOG.warning(_LW( "No storage group found. " "Performing rollback on Volume: %(volumeName)s " "To return it to the default storage group for FAST " - "policy %(fastPolicyName)s. "), + "policy %(fastPolicyName)s."), {'volumeName': rollbackDict['volumeName'], 'fastPolicyName': rollbackDict['fastPolicyName']}) assocDefaultStorageGroupName = ( @@ -1184,52 +1293,57 @@ def _check_if_rollback_action_for_masking_required( rollbackDict['controllerConfigService'], rollbackDict['volumeInstance'], rollbackDict['volumeName'], - rollbackDict['fastPolicyName'])) + rollbackDict['fastPolicyName'], + rollbackDict['extraSpecs'])) if assocDefaultStorageGroupName is None: LOG.error(_LE( "Failed to Roll back to re-add volume " "%(volumeName)s " "to default storage group for fast policy " "%(fastPolicyName)s: Please contact your sys " - "admin to get the volume re-added manually "), + "admin to get the volume re-added manually."), {'volumeName': rollbackDict['volumeName'], 'fastPolicyName': rollbackDict['fastPolicyName']}) - if len(foundStorageGroupInstanceName) > 0: + message = (_("V2 rollback, volume is not in any storage " + "group.")) + else: LOG.info(_LI( "The storage group found is " - "%(foundStorageGroupInstanceName)s: "), + "%(foundStorageGroupInstanceName)s."), {'foundStorageGroupInstanceName': foundStorageGroupInstanceName}) - # check the name see is it the default storage group or another - if (foundStorageGroupInstanceName != - rollbackDict['defaultStorageGroupInstanceName']): - # remove it from its current masking view and return it - # to its default masking view if fast is enabled - self.remove_and_reset_members( - conn, - rollbackDict['controllerConfigService'], - rollbackDict['volumeInstance'], - rollbackDict['fastPolicyName'], - rollbackDict['volumeName'], False) - except Exception as e: - LOG.error(_LE("Exception: %s"), e) + # Check the name, see is it the default storage group + # or another. + if (foundStorageGroupInstanceName != + rollbackDict['defaultStorageGroupInstanceName']): + # Remove it from its current masking view and return it + # to its default masking view if fast is enabled. + self.remove_and_reset_members( + conn, + rollbackDict['controllerConfigService'], + rollbackDict['volumeInstance'], + rollbackDict['volumeName'], + rollbackDict['extraSpecs']) + message = (_("V2 rollback - Volume in another storage " + "group besides default storage group.")) + except Exception: errorMessage = (_( "Rollback for Volume: %(volumeName)s has failed. " "Please contact your system administrator to manually return " "your volume to the default storage group for fast policy " - "%(fastPolicyName)s failed ") + "%(fastPolicyName)s failed.") % {'volumeName': rollbackDict['volumeName'], 'fastPolicyName': rollbackDict['fastPolicyName']}) - LOG.error(errorMessage) + LOG.exception(errorMessage) raise exception.VolumeBackendAPIException(data=errorMessage) + return message def _find_new_initiator_group(self, conn, maskingGroupDict): """After creating an new initiator group find it and return it. - :param conn: connection the ecom server + :param conn: connection to the ecom server :param maskingGroupDict: the maskingGroupDict dict - :param storageGroupName: storage group name (String) :returns: instance name foundInitiatorGroupInstanceName """ foundInitiatorGroupInstanceName = None @@ -1243,7 +1357,7 @@ def _get_initiator_group_from_masking_view( self, conn, maskingViewName, storageSystemName): """Given the masking view name get the initiator group from it. - :param conn: connection the the ecom server + :param conn: connection to the ecom server :param maskingViewName: the name of the masking view :param storageSystemName: the storage system name :returns: instance name foundInitiatorMaskingGroupInstanceName @@ -1263,14 +1377,14 @@ def _get_initiator_group_from_masking_view( {'view': maskingViewName, 'masking': foundInitiatorMaskingGroupInstanceName}) else: - LOG.warn(_LW("Unable to find Masking view: %(view)s ."), - {'view': maskingViewName}) + LOG.warning(_LW("Unable to find Masking view: %(view)s."), + {'view': maskingViewName}) return foundInitiatorMaskingGroupInstanceName def _verify_initiator_group_from_masking_view( self, conn, controllerConfigService, maskingViewName, connector, - storageSystemName, igGroupName): + storageSystemName, igGroupName, extraSpecs): """Check that the initiator group contains the correct initiators. If using an existing masking view check that the initiator group @@ -1280,12 +1394,14 @@ def _verify_initiator_group_from_masking_view( NOTE: EMC does not support ModifyMaskingView so we must first delete the masking view and recreate it. - :param conn: connection the ecom server + :param conn: connection to the ecom server :param controllerConfigService: the controller configuration service :param maskingViewName: maskingview name (String) :param connector: the connector dict :param storageSystemName: the storage System Name (string) :param igGroupName: the initiator group name (String) + :param extraSpecs: extra specifications + :returns: boolean """ initiatorNames = self._find_initiator_names(conn, connector) foundInitiatorGroupFromConnector = self._find_initiator_masking_group( @@ -1305,17 +1421,25 @@ def _verify_initiator_group_from_masking_view( self._get_storage_hardware_id_instance_names( conn, initiatorNames, storageSystemName)) if not storageHardwareIDInstanceNames: - LOG.error(_LE( + LOG.info(_LI( "Initiator Name(s) %(initiatorNames)s are not on " - "array %(storageSystemName)s "), + "array %(storageSystemName)s. "), {'initiatorNames': initiatorNames, 'storageSystemName': storageSystemName}) - return False + storageHardwareIDInstanceNames = ( + self._create_hardware_ids(conn, initiatorNames, + storageSystemName)) + if not storageHardwareIDInstanceNames: + LOG.error(_LE( + "Failed to create hardware id(s) on " + "%(storageSystemName)s."), + {'storageSystemName': storageSystemName}) + return False foundInitiatorGroupFromConnector = ( self._create_initiator_Group( conn, controllerConfigService, igGroupName, - storageHardwareIDInstanceNames)) + storageHardwareIDInstanceNames, extraSpecs)) storageGroupInstanceName = ( self._get_storage_group_from_masking_view( conn, maskingViewName, storageSystemName)) @@ -1326,51 +1450,55 @@ def _verify_initiator_group_from_masking_view( portGroupInstanceName is not None): self._delete_masking_view( conn, controllerConfigService, maskingViewName, - maskingViewInstanceName) + maskingViewInstanceName, extraSpecs) newMaskingViewInstanceName = ( self._get_masking_view_instance_name( conn, controllerConfigService, maskingViewName, storageGroupInstanceName, portGroupInstanceName, - foundInitiatorGroupFromConnector)) + foundInitiatorGroupFromConnector, extraSpecs)) if newMaskingViewInstanceName is not None: LOG.debug( "The old masking view has been replaced: " - "%(maskingViewName)s. ", + "%(maskingViewName)s.", {'maskingViewName': maskingViewName}) else: LOG.error(_LE( "One of the components of the original masking view " "%(maskingViewName)s cannot be retrieved so " "please contact your system administrator to check " - "that the correct initiator(s) are part of masking "), + "that the correct initiator(s) are part of masking."), {'maskingViewName': maskingViewName}) return False return True def _create_initiator_Group( self, conn, controllerConfigService, igGroupName, - hardwareIdinstanceNames): - """Create a new initiator group + hardwareIdinstanceNames, extraSpecs): + """Create a new initiator group. Given a list of hardwareId Instance name create a new - initiator group + initiator group. - :param conn: connection the ecom server + :param conn: connection to the ecom server :param controllerConfigService: the controller configuration service :param igGroupName: the initiator group name (String) :param hardwareIdinstanceNames: one or more hardware id instance names + :param extraSpecs: extra specifications + :returns: foundInitiatorGroupInstanceName + :raises: VolumeBackendAPIException """ rc, job = conn.InvokeMethod( 'CreateGroup', controllerConfigService, GroupName=igGroupName, Type=self.utils.get_num(INITIATORGROUPTYPE, '16'), Members=[hardwareIdinstanceNames[0]]) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( - "Error Create Group: %(groupName)s. " - "Return code: %(rc)lu. Error: %(error)s") + "Error Create Group: %(groupName)s. " + "Return code: %(rc)lu. Error: %(error)s.") % {'groupName': igGroupName, 'rc': rc, 'error': errordesc}) @@ -1388,12 +1516,14 @@ def _create_initiator_Group( MaskingGroup=foundInitiatorGroupInstanceName, Members=[hardwareIdinstanceNames[j]]) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = ( + self.utils.wait_for_job_complete(conn, job, + extraSpecs)) + if rc != 0: exceptionMessage = (_( "Error adding initiator to group : %(groupName)s. " - "Return code: %(rc)lu. Error: %(error)s") + "Return code: %(rc)lu. Error: %(error)s.") % {'groupName': igGroupName, 'rc': rc, 'error': errordesc}) @@ -1406,9 +1536,9 @@ def _create_initiator_Group( def _get_port_group_from_masking_view( self, conn, maskingViewName, storageSystemName): - """Given the masking view name get the port group from it + """Given the masking view name get the port group from it. - :param conn: connection the the ecom server + :param conn: connection to the ecom server :param maskingViewName: the name of the masking view :param storageSystemName: the storage system name :returns: instance name foundPortMaskingGroupInstanceName @@ -1417,40 +1547,43 @@ def _get_port_group_from_masking_view( foundPortMaskingGroupInstanceName = None foundView = self._find_masking_view( conn, maskingViewName, storageSystemName) + if foundView: + groups = conn.AssociatorNames( + foundView, + ResultClass='CIM_TargetMaskingGroup') + if len(groups) > 0: + foundPortMaskingGroupInstanceName = groups[0] - groups = conn.AssociatorNames( - foundView, - ResultClass='CIM_TargetMaskingGroup') - if len(groups) > 0: - foundPortMaskingGroupInstanceName = groups[0] - - LOG.debug( - "Masking view: %(view)s InitiatorMaskingGroup: %(masking)s.", - {'view': maskingViewName, - 'masking': foundPortMaskingGroupInstanceName}) + LOG.debug( + "Masking view: %(view)s InitiatorMaskingGroup: %(masking)s.", + {'view': maskingViewName, + 'masking': foundPortMaskingGroupInstanceName}) return foundPortMaskingGroupInstanceName def _delete_masking_view( self, conn, controllerConfigService, maskingViewName, - maskingViewInstanceName): - """Delete a masking view + maskingViewInstanceName, extraSpecs): + """Delete a masking view. - :param conn: connection the ecom server + :param conn: connection to the ecom server :param controllerConfigService: the controller configuration service :param maskingViewName: maskingview name (String) :param maskingViewInstanceName: the masking view instance name + :param extraSpecs: extra specifications + :raises: VolumeBackendAPIException """ rc, job = conn.InvokeMethod('DeleteMaskingView', controllerConfigService, ProtocolController=maskingViewInstanceName) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( "Error Modifying masking view : %(groupName)s. " - "Return code: %(rc)lu. Error: %(error)s") + "Return code: %(rc)lu. Error: %(error)s.") % {'groupName': maskingViewName, 'rc': rc, 'error': errordesc}) @@ -1460,12 +1593,12 @@ def _delete_masking_view( def get_masking_view_from_storage_group( self, conn, storageGroupInstanceName): - """Get the associated maskingview instance name + """Get the associated maskingview instance name. Given storage group instance name, get the associated masking - view instance name + view instance name. - :param conn: connection the ecom server + :param conn: connection to the ecom server :param storageGroupInstanceName: the storage group instance name :returns: instance name foundMaskingViewInstanceName """ @@ -1480,115 +1613,125 @@ def get_masking_view_from_storage_group( def add_volume_to_storage_group( self, conn, controllerConfigService, storageGroupInstanceName, - volumeInstance, volumeName, sgGroupName, storageSystemName=None): - """Add a volume to an existing storage group + volumeInstance, volumeName, sgGroupName, extraSpecs): + """Add a volume to an existing storage group. :param conn: connection to ecom server :param controllerConfigService: the controller configuration service - :param storageGroup: storage group instance + :param storageGroupInstanceName: storage group instance name :param volumeInstance: the volume instance :param volumeName: the name of the volume (String) :param sgGroupName: the name of the storage group (String) - :param storageSystemName: the storage system name (Optional Parameter), - if None plain operation assumed - :returns: int rc the return code of the job - :returns: dict the job dict + :param extraSpecs: additional info + :returns: int -- rc the return code of the job + :returns: dict -- the job dict """ self.provision.add_members_to_masking_group( conn, controllerConfigService, storageGroupInstanceName, - volumeInstance.path, volumeName) + volumeInstance.path, volumeName, extraSpecs) LOG.info(_LI( "Added volume: %(volumeName)s to existing storage group " - "%(sgGroupName)s. "), + "%(sgGroupName)s."), {'volumeName': volumeName, 'sgGroupName': sgGroupName}) def remove_device_from_default_storage_group( self, conn, controllerConfigService, volumeInstanceName, - volumeName, fastPolicyName): + volumeName, fastPolicyName, extraSpecs): """Remove the volume from the default storage group. Remove the volume from the default storage group for the FAST - policy and return the default storage group instance name + policy and return the default storage group instance name. :param conn: the connection to the ecom server :param controllerConfigService: the controller config service :param volumeInstanceName: the volume instance name :param volumeName: the volume name (String) :param fastPolicyName: the fast policy name (String) + :param extraSpecs: additional info :returns: instance name defaultStorageGroupInstanceName """ failedRet = None - defaultStorageGroupInstanceName = ( + defaultStorageGroupInstanceName, defaultSgName = ( self.fast.get_and_verify_default_storage_group( conn, controllerConfigService, volumeInstanceName, volumeName, fastPolicyName)) if defaultStorageGroupInstanceName is None: - LOG.warn(_LW( + LOG.warning(_LW( "Volume %(volumeName)s was not first part of the default " - "storage group for the FAST Policy"), + "storage group for the FAST Policy."), {'volumeName': volumeName}) return failedRet - assocVolumeInstanceNames = self.get_devices_from_storage_group( - conn, defaultStorageGroupInstanceName) + @lockutils.synchronized( + defaultStorageGroupInstanceName['ElementName'], + "emc-sg-", True) + def do_remove_vol_from_sg(): + assocVolumeInstanceNames = self.get_devices_from_storage_group( + conn, defaultStorageGroupInstanceName) - LOG.debug( - "There are %(length)lu associated with the default storage group " - "for fast before removing volume %(volumeName)s", - {'length': len(assocVolumeInstanceNames), - 'volumeName': volumeName}) + LOG.debug( + "There are %(length)lu associated with the default storage " + "group for fast before removing volume %(volumeName)s.", + {'length': len(assocVolumeInstanceNames), + 'volumeName': volumeName}) - self.provision.remove_device_from_storage_group( - conn, controllerConfigService, defaultStorageGroupInstanceName, - volumeInstanceName, volumeName) + self.provision.remove_device_from_storage_group( + conn, controllerConfigService, + defaultStorageGroupInstanceName, volumeInstanceName, + volumeName, extraSpecs) - assocVolumeInstanceNames = self.get_devices_from_storage_group( - conn, defaultStorageGroupInstanceName) - LOG.debug( - "There are %(length)lu associated with the default storage group " - "for fast after removing volume %(volumeName)s", - {'length': len(assocVolumeInstanceNames), - 'volumeName': volumeName}) + assocVolumeInstanceNames = self.get_devices_from_storage_group( + conn, defaultStorageGroupInstanceName) + LOG.debug( + "There are %(length)lu associated with the default storage " + "group for fast after removing volume %(volumeName)s.", + {'length': len(assocVolumeInstanceNames), + 'volumeName': volumeName}) - # required for unit tests + do_remove_vol_from_sg() + + # Required for unit tests. emptyStorageGroupInstanceName = ( - self._wrap_get_storage_group_from_volume(conn, volumeInstanceName)) + self._wrap_get_storage_group_from_volume(conn, volumeInstanceName, + defaultSgName)) if emptyStorageGroupInstanceName is not None: LOG.error(_LE( "Failed to remove %(volumeName)s from the default storage " - "group for the FAST Policy"), + "group for the FAST Policy."), {'volumeName': volumeName}) return failedRet return defaultStorageGroupInstanceName - def _wrap_get_storage_group_from_volume(self, conn, volumeInstanceName): - + def _wrap_get_storage_group_from_volume(self, conn, volumeInstanceName, + defaultSgName): """Wrapper for get_storage_group_from_volume. - Needed for override in tests + Needed for override in tests. :param conn: the connection to the ecom server :param volumeInstanceName: the volume instance name + :param defaultSgName: the default storage group name :returns: emptyStorageGroupInstanceName """ + return self.utils.get_storage_group_from_volume( - conn, volumeInstanceName) + conn, volumeInstanceName, defaultSgName) def get_devices_from_storage_group( self, conn, storageGroupInstanceName): - """Get the associated volume Instance names + """Get the associated volume Instance names. Given the storage group instance name get the associated volume - Instance names + Instance names. - :param conn: connection the the ecom server + :param conn: connection to the ecom server :param storageGroupInstanceName: the storage group instance name - :returns: list volumeInstanceNames list of volume instance names + :returns: list -- volumeInstanceNames list of volume instance names """ volumeInstanceNames = conn.AssociatorNames( storageGroupInstanceName, @@ -1598,6 +1741,15 @@ def get_devices_from_storage_group( def get_associated_masking_groups_from_device( self, conn, volumeInstanceName): + """Get the associated storage groups from the volume Instance name. + + Given the volume instance name get the associated storage group + instance names. + + :param conn: connection to the ecom server + :param volumeInstanceName: the volume instance name + :returns: list -- list of storage group instance names + """ maskingGroupInstanceNames = conn.AssociatorNames( volumeInstanceName, ResultClass='CIM_DeviceMaskingGroup', @@ -1605,131 +1757,329 @@ def get_associated_masking_groups_from_device( if len(maskingGroupInstanceNames) > 0: return maskingGroupInstanceNames else: - LOG.debug("Volume %(volumeName)s not in any storage group.", - {'volumeName': volumeInstanceName}) + LOG.info(_LI("Volume %(volumeName)s not in any storage group."), + {'volumeName': volumeInstanceName}) return None def remove_and_reset_members( self, conn, controllerConfigService, volumeInstance, - fastPolicyName, volumeName, isV3, connector=None, noReset=None): - """Part of unmap device or rollback. + volumeName, extraSpecs, connector=None, reset=True): + """This is called on a delete, unmap device or rollback. - Removes volume from the Device Masking Group that belongs to a - Masking View. Check if fast policy is in the extra specs, if it isn't - we do not need to do any thing for FAST. Assume that - isTieringPolicySupported is False unless the FAST policy is in - the extra specs and tiering is enabled on the array + If the connector is not None get the associated SG and remove volume + from the storage group, otherwise it is a VMAX3 deletion. - :param conn: connection the the ecom server + :param conn: connection to the ecom server :param controllerConfigService: the controller configuration service :param volumeInstance: the volume Instance - :param fastPolicyName: the fast policy name (if it exists) :param volumeName: the volume name - :param isV3: is array v2 or v3 + :param extraSpecs: additional info :param connector: optional - :param noReset: optional, if none, then reset - :returns: maskingGroupInstanceName + :param reset: reset, return to original SG (optional) + :returns: storageGroupInstanceName """ storageGroupInstanceName = None - if connector is not None: - storageGroupInstanceName = self._get_sg_associated_with_connector( - conn, controllerConfigService, volumeInstance.path, - volumeName, connector) - if storageGroupInstanceName is None: - return None - else: # connector is None in V3 volume deletion case - storageGroupInstanceNames = ( - self.get_associated_masking_groups_from_device( - conn, volumeInstance.path)) - if storageGroupInstanceNames: - storageGroupInstanceName = storageGroupInstanceNames[0] + if extraSpecs[ISV3]: + self._cleanup_deletion_v3( + conn, controllerConfigService, volumeInstance, extraSpecs) + else: + if connector is not None: + storageGroupInstanceName = ( + self._get_sg_associated_with_connector( + conn, controllerConfigService, volumeInstance.path, + volumeName, connector)) + if storageGroupInstanceName: + self._remove_volume_from_sg( + conn, controllerConfigService, + storageGroupInstanceName, volumeInstance, extraSpecs) else: - return None + LOG.warning(_LW("Cannot get storage from connector.")) + if reset: + self._return_back_to_default_sg( + conn, controllerConfigService, volumeInstance, volumeName, + extraSpecs) + + return storageGroupInstanceName + + def _cleanup_deletion_v3( + self, conn, controllerConfigService, volumeInstance, extraSpecs): + """Pre cleanup before VMAX3 deletion operation + + :param conn: the ecom connection + :param controllerConfigService: storage system instance name + :param volumeInstance: the volume instance + :param extraSpecs: the extra specifications + """ + storageGroupInstanceNames = ( + self.get_associated_masking_groups_from_device( + conn, volumeInstance.path)) + + if storageGroupInstanceNames: + sgNum = len(storageGroupInstanceNames) + if len(storageGroupInstanceNames) > 1: + LOG.warning(_LW("Volume %(volumeName)s is belong to " + "%(sgNum)s storage groups."), + {'volumeName': volumeInstance['ElementName'], + 'sgNum': sgNum}) + for storageGroupInstanceName in storageGroupInstanceNames: + self._remove_volume_from_sg( + conn, controllerConfigService, + storageGroupInstanceName, + volumeInstance, + extraSpecs) + + def _remove_volume_from_sg( + self, conn, controllerConfigService, storageGroupInstanceName, + volumeInstance, extraSpecs): + + """Remove volume from storage group + + :param conn: the ecom connection + :param controllerConfigService: storage system instance name + :param storageGroupInstanceName: the SG instance name + :param volumeInstance: the volume instance + :param extraSpecs: the extra specifications + """ instance = conn.GetInstance(storageGroupInstanceName, LocalOnly=False) storageGroupName = instance['ElementName'] + mvInstanceName = self.get_masking_view_from_storage_group( + conn, storageGroupInstanceName) + if mvInstanceName is None: + LOG.debug("Unable to get masking view %(maskingView)s " + "from storage group.", + {'maskingView': mvInstanceName}) + + @lockutils.synchronized(storageGroupName, "emc-sg", True) + def do_remove_volume_from_sg(): + volumeInstanceNames = self.get_devices_from_storage_group( + conn, storageGroupInstanceName) + numVolInStorageGroup = len(volumeInstanceNames) + LOG.debug( + "There are %(numVol)d volumes in the storage group " + "%(maskingGroup)s.", + {'numVol': numVolInStorageGroup, + 'maskingGroup': storageGroupInstanceName}) + + if numVolInStorageGroup == 1: + # Last volume in the storage group. + self._last_vol_in_SG( + conn, controllerConfigService, + storageGroupInstanceName, + storageGroupName, volumeInstance, + volumeInstance['ElementName'], extraSpecs) + else: + # Not the last volume so remove it from storage group + self._multiple_vols_in_SG( + conn, controllerConfigService, + storageGroupInstanceName, volumeInstance, + volumeInstance['ElementName'], + numVolInStorageGroup, extraSpecs) + + return do_remove_volume_from_sg() + else: + # need to lock masking view when we are locking the storage + # group to avoid possible deadlock situations from concurrent + # processes + maskingViewInstance = conn.GetInstance( + mvInstanceName, LocalOnly=False) + maskingViewName = maskingViewInstance['ElementName'] + + @lockutils.synchronized(maskingViewName, "emc-mv-", True) + @lockutils.synchronized(storageGroupName, "emc-sg", True) + def do_remove_volume_from_sg(): + volumeInstanceNames = self.get_devices_from_storage_group( + conn, storageGroupInstanceName) + numVolInStorageGroup = len(volumeInstanceNames) + LOG.debug( + "There are %(numVol)d volumes in the storage group " + "%(maskingGroup)s.", + {'numVol': numVolInStorageGroup, + 'maskingGroup': storageGroupInstanceName}) + + if numVolInStorageGroup == 1: + # Last volume in the storage group. + self._last_vol_in_SG( + conn, controllerConfigService, + storageGroupInstanceName, + storageGroupName, volumeInstance, + volumeInstance['ElementName'], extraSpecs) + else: + # Not the last volume so remove it from storage group + self._multiple_vols_in_SG( + conn, controllerConfigService, + storageGroupInstanceName, + volumeInstance, volumeInstance['ElementName'], + numVolInStorageGroup, extraSpecs) + return do_remove_volume_from_sg() + + def _last_vol_in_SG( + self, conn, controllerConfigService, storageGroupInstanceName, + storageGroupName, volumeInstance, volumeName, extraSpecs): + """Steps if the volume is the last in a storage group. + + 1. Check if the volume is in a masking view. + 2. If it is in a masking view, delete the masking view, remove the + initiators from the initiator group and delete the initiator + group if there are no other masking views associated with the + initiator group, remove the volume from the storage group, and + delete the storage group. + 3. If it is not in a masking view, remove the volume from the + storage group and delete the storage group. + + :param conn: the ecom connection + :param controllerConfigService: storage system instance name + :param storageGroupInstanceName: the SG instance name + :param storageGroupName: the Storage group name (String) + :param volumeInstance: the volume instance + :param volumeName: the volume name + :param extraSpecs: the extra specifications + """ + status = False + LOG.debug("Only one volume remains in storage group " + "%(sgname)s. Driver will attempt cleanup.", + {'sgname': storageGroupName}) + mvInstanceName = self.get_masking_view_from_storage_group( + conn, storageGroupInstanceName) + if mvInstanceName is None: + # Remove the volume from the storage group and delete the SG. + self._remove_last_vol_and_delete_sg( + conn, controllerConfigService, + storageGroupInstanceName, + storageGroupName, volumeInstance.path, + volumeName, extraSpecs) + status = True + else: + maskingViewInstance = conn.GetInstance( + mvInstanceName, LocalOnly=False) + maskingViewName = maskingViewInstance['ElementName'] + + def do_delete_mv_ig_and_sg(): + return self._delete_mv_ig_and_sg( + conn, controllerConfigService, mvInstanceName, + maskingViewName, storageGroupInstanceName, + storageGroupName, volumeInstance, volumeName, + extraSpecs) + do_delete_mv_ig_and_sg() + status = True + return status + + def _multiple_vols_in_SG( + self, conn, controllerConfigService, storageGroupInstanceName, + volumeInstance, volumeName, numVolsInSG, extraSpecs): + """If the volume is not the last in the storage group + + Remove the volume from the SG. + + :param conn: the ecom connection + :param controllerConfigService: storage system instance name + :param storageGroupInstanceName: the SG instance name + :param volumeInstance: the volume instance + :param volumeName: the volume name + :param numVolsInSG: the number of volumes in the SG + :param extraSpecs: the extra specifications + """ + + LOG.debug("Start: number of volumes in masking storage group: " + "%(numVol)d", {'numVol': numVolsInSG}) + self.provision.remove_device_from_storage_group( + conn, controllerConfigService, storageGroupInstanceName, + volumeInstance.path, volumeName, extraSpecs) + + LOG.debug( + "RemoveMembers for volume %(volumeName)s completed " + "successfully.", {'volumeName': volumeName}) + volumeInstanceNames = self.get_devices_from_storage_group( conn, storageGroupInstanceName) + LOG.debug( + "End: number of volumes in masking storage group: %(numVol)d.", + {'numVol': len(volumeInstanceNames)}) + + def _delete_mv_ig_and_sg( + self, conn, controllerConfigService, mvInstanceName, + maskingViewName, storageGroupInstanceName, storageGroupName, + volumeInstance, volumeName, extraSpecs): + """Delete the Masking view, the storage Group and the initiator group. + + :param conn: connection to the ecom server + :param controllerConfigService: the controller configuration service + :param mvInstanceName: masking view instance name + :param maskingViewName: masking view name + :param storageGroupInstanceName: storage group instance name + :param maskingViewName: masking view name + :param volumeInstance: the volume Instance + :param volumeName: the volume name + :param extraSpecs: extra specs + """ + isV3 = extraSpecs[ISV3] + fastPolicyName = extraSpecs.get(FASTPOLICY, None) + storageSystemInstanceName = self.utils.find_storage_system( conn, controllerConfigService) - - numVolInMaskingView = len(volumeInstanceNames) - LOG.debug( - "There are %(numVol)d volumes in the storage group " - "%(maskingGroup)s", - {'numVol': numVolInMaskingView, - 'maskingGroup': storageGroupInstanceName}) + initiatorGroupInstanceName = ( + self.get_initiator_group_from_masking_view(conn, mvInstanceName)) + self._last_volume_delete_masking_view( + conn, controllerConfigService, mvInstanceName, + maskingViewName, extraSpecs) + self._last_volume_delete_initiator_group( + conn, controllerConfigService, + initiatorGroupInstanceName, extraSpecs) if not isV3: isTieringPolicySupported, tierPolicyServiceInstanceName = ( self._get_tiering_info(conn, storageSystemInstanceName, fastPolicyName)) + self._get_and_remove_rule_association( + conn, fastPolicyName, + isTieringPolicySupported, + tierPolicyServiceInstanceName, + storageSystemInstanceName['Name'], + storageGroupInstanceName, extraSpecs) + + self._remove_last_vol_and_delete_sg( + conn, controllerConfigService, storageGroupInstanceName, + storageGroupName, volumeInstance.path, volumeName, + extraSpecs) - if numVolInMaskingView == 1: - # last volume in the storage group - self._last_volume_delete_masking_view( - conn, storageGroupInstanceName) - if not isV3: - self._get_and_remove_rule_association( - conn, fastPolicyName, - isTieringPolicySupported, - tierPolicyServiceInstanceName, - storageSystemInstanceName['name'], - storageGroupInstanceName) + LOG.debug( + "Volume %(volumeName)s successfully removed from SG and " + "Storage Group %(storageGroupName)s successfully deleted. ", + {'volumeName': volumeName, + 'storageGroupName': storageGroupName}) - self.provision.remove_device_from_storage_group( - conn, controllerConfigService, storageGroupInstanceName, - volumeInstance.path, volumeName) + def _return_back_to_default_sg( + self, conn, controllerConfigService, volumeInstance, volumeName, + extraSpecs): + """Return volume to default storage group - LOG.debug( - "Remove the last volume %(volumeName)s completed " - "successfully.", - {'volumeName': volumeName}) + Moving the volume to the default SG for VMAX3 and + FAST for VMAX2. - # Delete storage group - conn.DeleteInstance(storageGroupInstanceName) - if isV3: - if noReset is None: - self._return_volume_to_default_storage_group_v3( - conn, controllerConfigService, storageGroupName, - volumeInstance, volumeName, storageSystemInstanceName) - else: - if isTieringPolicySupported: - self._cleanup_tiering( - conn, controllerConfigService, fastPolicyName, - volumeInstance, volumeName) + :param conn: connection to the ecom server + :param controllerConfigService: the controller configuration service + :param volumeInstance: the volume Instance + :param volumeName: the volume name + :param extraSpecs: extra specs + """ + # Add it back to the default storage group. + if extraSpecs[ISV3]: + self.return_volume_to_default_storage_group_v3( + conn, controllerConfigService, + volumeInstance, volumeName, extraSpecs) else: - # not the last volume so remove it from storage group in - # the masking view - LOG.debug("start: number of volumes in masking storage group: " - "%(numVol)d", {'numVol': len(volumeInstanceNames)}) - self.provision.remove_device_from_storage_group( - conn, controllerConfigService, storageGroupInstanceName, - volumeInstance.path, volumeName) - - LOG.debug( - "RemoveMembers for volume %(volumeName)s completed " - "successfully.", {'volumeName': volumeName}) - - # add it back to the default storage group - if isV3: - self._return_volume_to_default_storage_group_v3( - conn, controllerConfigService, storageGroupName, - volumeInstance, volumeName, storageSystemInstanceName) - else: - # v2 if FAST POLICY enabled, move the volume to the default SG - if fastPolicyName is not None and isTieringPolicySupported: - self._cleanup_tiering( - conn, controllerConfigService, fastPolicyName, - volumeInstance, volumeName) - - volumeInstanceNames = self.get_devices_from_storage_group( - conn, storageGroupInstanceName) - LOG.debug( - "end: number of volumes in masking storage group: %(numVol)d", - {'numVol': len(volumeInstanceNames)}) - - return storageGroupInstanceName + # V2 if FAST POLICY enabled, move the volume to the default + # SG. + fastPolicyName = extraSpecs.get(FASTPOLICY, None) + storageSystemInstanceName = self.utils.find_storage_system( + conn, controllerConfigService) + isTieringPolicySupported, __ = ( + self._get_tiering_info(conn, storageSystemInstanceName, + fastPolicyName)) + if fastPolicyName is not None and isTieringPolicySupported: + self._cleanup_tiering( + conn, controllerConfigService, fastPolicyName, + volumeInstance, volumeName, extraSpecs) def _get_sg_associated_with_connector( self, conn, controllerConfigService, volumeInstanceName, @@ -1741,61 +2091,24 @@ def _get_sg_associated_with_connector( :param conn: the ecom connection :param controllerConfigService: storage system instance name - :param volumeInstanceName - :param volumeName - :param connector + :param volumeInstanceName: the volume instance name + :param volumeName: the volume name (String) + :param connector: the connector object :returns: storageGroupInstanceName(can be None) """ - storageGroupInstanceName = None - initiatorNames = self._find_initiator_names(conn, connector) - igInstanceNameFromConnector = self._find_initiator_masking_group( - conn, controllerConfigService, initiatorNames) - # a device can be shared by multi-SGs in a multi-host attach case - storageGroupInstanceNames = ( - self.get_associated_masking_groups_from_device( - conn, volumeInstanceName)) - LOG.debug("Found storage groups volume " - "%(volumeName)s is in: %(storageGroups)s", - {'volumeName': volumeName, - 'storageGroups': storageGroupInstanceNames}) - if storageGroupInstanceNames: # not empty - # get THE SG by IGs - for sgInstanceName in storageGroupInstanceNames: - # get maskingview from storage group - mvInstanceName = self.get_masking_view_from_storage_group( - conn, sgInstanceName) - LOG.debug("Found masking view associated with SG " - "%(storageGroup)s: %(maskingview)s", - {'maskingview': mvInstanceName, - 'storageGroup': sgInstanceName}) - # get initiator group from masking view - igInstanceName = ( - self.get_initiator_group_from_masking_view( - conn, mvInstanceName)) - LOG.debug("Initiator Group in masking view %(ig)s: " - "IG associated with connector%(igFromConnector)s", - {'ig': igInstanceName, - 'igFromConnector': igInstanceNameFromConnector}) - if igInstanceName == igInstanceNameFromConnector: - storageGroupInstanceName = sgInstanceName - LOG.debug("Found the storage group associated with " - "connector %(connector)s: %(storageGroup)s", - {'connector': initiatorNames, - 'storageGroup': storageGroupInstanceName}) - break - - return storageGroupInstanceName + return self._get_sg_or_mv_associated_with_initiator( + conn, controllerConfigService, volumeInstanceName, + volumeName, connector, True) def _get_tiering_info( self, conn, storageSystemInstanceName, fastPolicyName): - """get tiering specifics + """Get tiering specifics. :param conn: the ecom connection :param storageSystemInstanceName: storage system instance name - :param fastPolicyName - - :returns: isTieringPolicySupported, tierPolicyServiceInstanceName - + :param fastPolicyName: + :returns: boolean -- isTieringPolicySupported + :returns: tierPolicyServiceInstanceName """ isTieringPolicySupported = False tierPolicyServiceInstanceName = None @@ -1813,106 +2126,120 @@ def _get_tiering_info( return isTieringPolicySupported, tierPolicyServiceInstanceName def _last_volume_delete_masking_view( - self, conn, storageGroupInstanceName): - """delete the masking view + self, conn, controllerConfigService, mvInstanceName, + maskingViewName, extraSpecs): + """Delete the masking view. - delete the masking view if the volume is the last one in the - storage group + Delete the masking view if the volume is the last one in the + storage group. :param conn: the ecom connection - :param storageGroupInstanceName: storage group instance name + :param controllerConfigService: controller config service + :param mvInstanceName: masking view instance name + :param maskingViewName: masking view name + :param extraSpecs: extra specifications """ - # delete masking view - mvInstanceName = self.get_masking_view_from_storage_group( - conn, storageGroupInstanceName) - if mvInstanceName is not None: - LOG.debug( - "Last volume in the storage group, deleting masking view " - "%(mvInstanceName)s", - {'mvInstanceName': mvInstanceName}) - conn.DeleteInstance(mvInstanceName) + LOG.debug( + "Last volume in the storage group, deleting masking view " + "%(maskingViewName)s.", + {'maskingViewName': maskingViewName}) + self._delete_masking_view( + conn, controllerConfigService, maskingViewName, + mvInstanceName, extraSpecs) + + mvInstance = self.utils.get_existing_instance( + conn, mvInstanceName) + if mvInstance: + exceptionMessage = (_( + "Masking view %(maskingViewName)s " + "was not deleted successfully") % + {'maskingViewName': maskingViewName}) + + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + else: + LOG.info(_LI( + "Masking view %(maskingViewName)s successfully deleted."), + {'maskingViewName': maskingViewName}) def _get_and_remove_rule_association( self, conn, fastPolicyName, isTieringPolicySupported, tierPolicyServiceInstanceName, storageSystemName, - storageGroupInstanceName): - """remove the storage group from the policy rule + storageGroupInstanceName, extraSpecs): + """Remove the storage group from the policy rule. :param conn: the ecom connection + :param fastPolicyName: the fast policy name :param isTieringPolicySupported: boolean :param tierPolicyServiceInstanceName: the tier policy instance name :param storageSystemName: storage system name :param storageGroupInstanceName: the storage group instance name + :param extraSpecs: additional info """ - # disassociate storage group from FAST policy + # Disassociate storage group from FAST policy. if fastPolicyName is not None and isTieringPolicySupported is True: tierPolicyInstanceName = self.fast.get_tier_policy_by_name( conn, storageSystemName, fastPolicyName) - LOG.info(_LI( - "policy:%(policy)s, policy service:%(service)s, " - "masking group=%(maskingGroup)s"), + LOG.debug( + "Policy: %(policy)s, policy service:%(service)s, " + "masking group: %(maskingGroup)s.", {'policy': tierPolicyInstanceName, 'service': tierPolicyServiceInstanceName, 'maskingGroup': storageGroupInstanceName}) self.fast.delete_storage_group_from_tier_policy_rule( conn, tierPolicyServiceInstanceName, - storageGroupInstanceName, tierPolicyInstanceName) + storageGroupInstanceName, tierPolicyInstanceName, extraSpecs) - def _return_volume_to_default_storage_group_v3( - self, conn, controllerConfigService, storageGroupName, - volumeInstance, volumeName, storageSystemInstanceName): - """return volume to the default storage group in v3 + def return_volume_to_default_storage_group_v3( + self, conn, controllerConfigurationService, + volumeInstance, volumeName, extraSpecs): + """Return volume to the default storage group in v3. :param conn: the ecom connection :param controllerConfigService: controller config service - :param storageGroupInstanceName: storage group instance name :param volumeInstance: volumeInstance :param volumeName: the volume name - :param storageSystemInstanceName: the storage system instance name - """ - # First strip the shortHostname from the storage group name - defaultStorageGroupName, shorthostName = ( - self.utils.strip_short_host_name(storageGroupName)) - - # Check if host name exists which signifies detach operation - if shorthostName is not None: - # Populate maskingViewDict and storageGroupInstanceName - maskingViewDict = {} - maskingViewDict['sgGroupName'] = defaultStorageGroupName - maskingViewDict['volumeInstance'] = volumeInstance - maskingViewDict['volumeName'] = volumeName - maskingViewDict['controllerConfigService'] = \ - controllerConfigService - maskingViewDict['storageSystemName'] = \ - storageSystemInstanceName - sgInstanceName = self.utils.find_storage_masking_group( - conn, controllerConfigService, defaultStorageGroupName) - if sgInstanceName is not None: - errorMessage = ( - self._check_adding_volume_to_storage_group( - conn, maskingViewDict, - sgInstanceName)) - else: - errorMessage = (_( - "Storage group %(sgGroupName) " - "does not exist ") - % {'StorageGroup': defaultStorageGroupName}) + :param extraSpecs: additional info + :raises: VolumeBackendAPIException + """ + storageGroupName = self.utils.get_v3_storage_group_name( + extraSpecs[self.utils.POOL], extraSpecs[self.utils.SLO], + extraSpecs[self.utils.WORKLOAD]) + storageGroupInstanceName = self.utils.find_storage_masking_group( + conn, controllerConfigurationService, storageGroupName) + + if not storageGroupInstanceName: + storageGroupInstanceName = ( + self.provisionv3.create_storage_group_v3( + conn, controllerConfigurationService, storageGroupName, + extraSpecs[self.utils.POOL], extraSpecs[self.utils.SLO], + extraSpecs[self.utils.WORKLOAD], extraSpecs)) + if not storageGroupInstanceName: + errorMessage = (_("Failed to create storage group " + "%(storageGroupName)s.") % + {'storageGroupName': storageGroupName}) LOG.error(errorMessage) - raise exception.VolumeBackendAPIException( - data=errorMessage) + raise exception.VolumeBackendAPIException(data=errorMessage) + + self._add_volume_to_sg_and_verify( + conn, controllerConfigurationService, + storageGroupInstanceName, volumeInstance, volumeName, + storageGroupName, extraSpecs) def _cleanup_tiering( self, conn, controllerConfigService, fastPolicyName, - volumeInstance, volumeName): - """Cleanup tiering + volumeInstance, volumeName, extraSpecs): + """Clean up tiering. :param conn: the ecom connection :param controllerConfigService: the controller configuration service :param fastPolicyName: the fast policy name :param volumeInstance: volume instance :param volumeName: the volume name + :param extraSpecs: additional info """ defaultStorageGroupInstanceName = ( self.fast.get_policy_default_storage_group( @@ -1920,24 +2247,25 @@ def _cleanup_tiering( volumeInstanceNames = self.get_devices_from_storage_group( conn, defaultStorageGroupInstanceName) LOG.debug( - "start: number of volumes in default storage group: %(numVol)d", + "Start: number of volumes in default storage group: %(numVol)d.", {'numVol': len(volumeInstanceNames)}) defaultStorageGroupInstanceName = ( self.fast.add_volume_to_default_storage_group_for_fast_policy( conn, controllerConfigService, volumeInstance, volumeName, - fastPolicyName)) - # check default storage group number of volumes + fastPolicyName, extraSpecs)) + # Check default storage group number of volumes. volumeInstanceNames = self.get_devices_from_storage_group( conn, defaultStorageGroupInstanceName) LOG.debug( - "end: number of volumes in default storage group: %(numVol)d", + "End: number of volumes in default storage group: %(numVol)d.", {'numVol': len(volumeInstanceNames)}) def get_target_wwns(self, conn, mvInstanceName): - """Get the DA ports' wwns. + """Get the DA ports wwns. :param conn: the ecom connection :param mvInstanceName: masking view instance name + :returns: list -- the list of target wwns for the masking view """ targetWwns = [] targetPortInstanceNames = conn.AssociatorNames( @@ -1945,10 +2273,10 @@ def get_target_wwns(self, conn, mvInstanceName): ResultClass='Symm_FCSCSIProtocolEndpoint') numberOfPorts = len(targetPortInstanceNames) if numberOfPorts <= 0: - LOG.warn(_LW("No target ports found in " - "masking view %(maskingView)s"), - {'numPorts': len(targetPortInstanceNames), - 'maskingView': mvInstanceName}) + LOG.warning(_LW("No target ports found in " + "masking view %(maskingView)s."), + {'numPorts': len(targetPortInstanceNames), + 'maskingView': mvInstanceName}) for targetPortInstanceName in targetPortInstanceNames: targetWwns.append(targetPortInstanceName['Name']) return targetWwns @@ -1956,94 +2284,441 @@ def get_target_wwns(self, conn, mvInstanceName): def get_masking_view_by_volume(self, conn, volumeInstance, connector): """Given volume, retrieve the masking view instance name. + :param conn: the ecom connection :param volumeInstance: the volume instance :param connector: the connector object - :returns mvInstanceName: masking view instance name + :returns: masking view instance name """ - foundMVInstanceName = None - initiatorNames = self._find_initiator_names(conn, connector) + storageSystemName = volumeInstance['SystemName'] controllerConfigService = ( self.utils.find_controller_configuration_service( conn, storageSystemName)) - igInstanceNameFromConnector = self._find_initiator_masking_group( - conn, controllerConfigService, initiatorNames) - # a device can be shared by multi-SGs in a multi-host attach case - storageGroupInstanceNames = ( - self.get_associated_masking_groups_from_device( - conn, volumeInstance.path)) - LOG.debug("Found storage groups %(storageGroups)s", - {'storageGroups': storageGroupInstanceNames}) - if storageGroupInstanceNames: # not empty - # get THE SG by IGs - for sgInstanceName in storageGroupInstanceNames: - # get maskingview from storage group - mvInstanceName = self.get_masking_view_from_storage_group( - conn, sgInstanceName) - LOG.debug("Found masking view associated with SG " - "%(storageGroup)s: %(maskingview)s", - {'maskingview': mvInstanceName, - 'storageGroup': sgInstanceName}) - # get initiator group from masking view - igInstanceName = self.get_initiator_group_from_masking_view( - conn, mvInstanceName) - LOG.debug("Initiator Group in masking view %(ig)s: " - "IG associated with connector%(igFromConnector)s", - {'ig': igInstanceName, - 'igFromConnector': igInstanceNameFromConnector}) - if igInstanceName == igInstanceNameFromConnector: - foundMVInstanceName = mvInstanceName - LOG.debug("Found the masking view associated with " - "connector %(connector)s: %(maskingview)s", - {'connector': initiatorNames, - 'maskingview': foundMVInstanceName}) - break - - LOG.debug("Found Masking View %(mv)s: ", {'mv': foundMVInstanceName}) - return foundMVInstanceName + volumeName = volumeInstance['ElementName'] + mvInstanceName = ( + self._get_sg_or_mv_associated_with_initiator( + conn, controllerConfigService, volumeInstance.path, + volumeName, connector, False)) + return mvInstanceName def get_masking_views_by_port_group(self, conn, portGroupInstanceName): """Given port group, retrieve the masking view instance name. - :param : the volume - :param mvInstanceName: masking view instance name - :returns: maksingViewInstanceNames + :param conn: the ecom connection + :param portGroupInstanceName: the instance name of the port group + :returns: masking view instance names """ mvInstanceNames = conn.AssociatorNames( portGroupInstanceName, ResultClass='Symm_LunMaskingView') return mvInstanceNames + def get_masking_views_by_initiator_group( + self, conn, initiatorGroupInstanceName): + """Given initiator group, retrieve the masking view instance name. + + Retrieve the list of masking view instances associated with the + initiator group instance name. + + :param conn: the ecom connection + :param initiatorGroupInstanceName: the instance name of the + initiator group + :returns: list of masking view instance names + """ + mvInstanceNames = conn.AssociatorNames( + initiatorGroupInstanceName, ResultClass='Symm_LunMaskingView') + return mvInstanceNames + def get_port_group_from_masking_view(self, conn, maskingViewInstanceName): """Get the port group in a masking view. + :param conn: the ecom connection :param maskingViewInstanceName: masking view instance name :returns: portGroupInstanceName """ portGroupInstanceNames = conn.AssociatorNames( maskingViewInstanceName, ResultClass='SE_TargetMaskingGroup') if len(portGroupInstanceNames) > 0: - LOG.debug("Found port group %(pg)s in masking view %(mv)s", + LOG.debug("Found port group %(pg)s in masking view %(mv)s.", {'pg': portGroupInstanceNames[0], 'mv': maskingViewInstanceName}) return portGroupInstanceNames[0] else: - LOG.warn(_LW("No port group found in masking view %(mv)s"), - {'mv': maskingViewInstanceName}) + LOG.warning(_LW("No port group found in masking view %(mv)s."), + {'mv': maskingViewInstanceName}) def get_initiator_group_from_masking_view( self, conn, maskingViewInstanceName): """Get initiator group in a masking view. + :param conn: the ecom connection :param maskingViewInstanceName: masking view instance name - :returns: initiatorGroupInstanceName + :returns: initiatorGroupInstanceName or None if it is not found """ initiatorGroupInstanceNames = conn.AssociatorNames( maskingViewInstanceName, ResultClass='SE_InitiatorMaskingGroup') if len(initiatorGroupInstanceNames) > 0: - LOG.debug("Found initiator group %(pg)s in masking view %(mv)s", - {'pg': initiatorGroupInstanceNames[0], + LOG.debug("Found initiator group %(ig)s in masking view %(mv)s.", + {'ig': initiatorGroupInstanceNames[0], 'mv': maskingViewInstanceName}) return initiatorGroupInstanceNames[0] else: - LOG.warn(_LW("No port group found in masking view %(mv)s"), - {'mv': maskingViewInstanceName}) + LOG.warning(_LW("No Initiator group found in masking view " + "%(mv)s."), {'mv': maskingViewInstanceName}) + + def _get_sg_or_mv_associated_with_initiator( + self, conn, controllerConfigService, volumeInstanceName, + volumeName, connector, getSG=True): + """Get storage group or masking view associated with connector. + + If the connector gets passed then extra logic required to + get storage group. + + :param conn: the ecom connection + :param controllerConfigService: storage system instance name + :param volumeInstanceName: volume instance name + :param volumeName: volume element name + :param connector: the connector object + :param getSG: True if to get storage group; otherwise get masking + :returns: foundInstanceName(can be None) + """ + foundInstanceName = None + initiatorNames = self._find_initiator_names(conn, connector) + igInstanceNameFromConnector = self._find_initiator_masking_group( + conn, controllerConfigService, initiatorNames) + # Device can be shared by multi-SGs in a multi-host attach case. + storageGroupInstanceNames = ( + self.get_associated_masking_groups_from_device( + conn, volumeInstanceName)) + LOG.debug("Found storage groups volume " + "%(volumeName)s is in: %(storageGroups)s", + {'volumeName': volumeName, + 'storageGroups': storageGroupInstanceNames}) + if storageGroupInstanceNames: # not empty + # Get the SG by IGs. + for sgInstanceName in storageGroupInstanceNames: + # Get maskingview from storage group. + mvInstanceName = self.get_masking_view_from_storage_group( + conn, sgInstanceName) + if mvInstanceName: + LOG.debug("Found masking view associated with SG " + "%(storageGroup)s: %(maskingview)s", + {'maskingview': mvInstanceName, + 'storageGroup': sgInstanceName}) + # Get initiator group from masking view. + igInstanceName = ( + self.get_initiator_group_from_masking_view( + conn, mvInstanceName)) + LOG.debug("Initiator Group in masking view %(ig)s: " + "IG associated with connector " + "%(igFromConnector)s", + {'ig': igInstanceName, + 'igFromConnector': + igInstanceNameFromConnector}) + if igInstanceName == igInstanceNameFromConnector: + if getSG is True: + foundInstanceName = sgInstanceName + LOG.debug("Found the storage group associated " + "with initiator %(initiator)s: " + "%(storageGroup)s", + {'initiator': initiatorNames, + 'storageGroup': foundInstanceName}) + else: + foundInstanceName = mvInstanceName + LOG.debug("Found the masking view associated " + "with initiator %(initiator)s: " + "%(maskingview)s.", + {'initiator': initiatorNames, + 'maskingview': foundInstanceName}) + + break + return foundInstanceName + + def _remove_last_vol_and_delete_sg(self, conn, controllerConfigService, + storageGroupInstanceName, + storageGroupName, volumeInstanceName, + volumeName, extraSpecs): + """Remove the last volume and delete the storage group + + :param conn: the ecom connection + :param controllerConfigService: controller config service + :param storageGroupInstanceName: storage group instance name + :param storageGroupName: storage group name + :param volumeInstanceName: volume instance name + :param volumeName: volume name + :param extrSpecs: additional info + """ + self.provision.remove_device_from_storage_group( + conn, controllerConfigService, storageGroupInstanceName, + volumeInstanceName, volumeName, extraSpecs) + + LOG.debug( + "Remove the last volume %(volumeName)s completed " + "successfully.", + {'volumeName': volumeName}) + + # Delete storage group. + self._delete_storage_group(conn, controllerConfigService, + storageGroupInstanceName, + storageGroupName, extraSpecs) + storageGroupInstance = self.utils.get_existing_instance( + conn, storageGroupInstanceName) + if storageGroupInstance: + exceptionMessage = (_( + "Storage group %(storageGroupName)s " + "was not deleted successfully") % + {'storageGroupName': storageGroupName}) + + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + else: + LOG.info(_LI( + "Storage Group %(storageGroupName)s successfully deleted."), + {'storageGroupName': storageGroupName}) + + def _delete_storage_group(self, conn, controllerConfigService, + storageGroupInstanceName, storageGroupName, + extraSpecs): + """Delete empty storage group + + :param conn: the ecom connection + :param controllerConfigService: controller config service + :param storageGroupInstanceName: storage group instance name + :param storageGroupName: storage group name + :param extraSpecs: extra specifications + """ + rc, job = conn.InvokeMethod( + 'DeleteGroup', + controllerConfigService, + MaskingGroup=storageGroupInstanceName, + Force=True) + + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: + exceptionMessage = (_( + "Error Deleting Group: %(storageGroupName)s. " + "Return code: %(rc)lu. Error: %(error)s") + % {'storageGroupName': storageGroupName, + 'rc': rc, + 'error': errordesc}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + + def _delete_initiator_group(self, conn, controllerConfigService, + initiatorGroupInstanceName, initiatorGroupName, + extraSpecs): + """Delete an initiatorGroup. + + :param conn - connection to the ecom server + :param controllerConfigService - controller config service + :param initiatorGroupInstanceName - the initiator group instance name + :param initiatorGroupName - initiator group name + :param extraSpecs: extra specifications + """ + + rc, job = conn.InvokeMethod( + 'DeleteGroup', + controllerConfigService, + MaskingGroup=initiatorGroupInstanceName, + Force=True) + + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: + exceptionMessage = (_( + "Error Deleting Initiator Group: %(initiatorGroupName)s. " + "Return code: %(rc)lu. Error: %(error)s") + % {'initiatorGroupName': initiatorGroupName, + 'rc': rc, + 'error': errordesc}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + else: + LOG.debug("Initiator group %(initiatorGroupName)s " + "is successfully deleted.", + {'initiatorGroupName': initiatorGroupName}) + else: + LOG.debug("Initiator group %(initiatorGroupName)s " + "is successfully deleted.", + {'initiatorGroupName': initiatorGroupName}) + + def _delete_storage_hardware_id(self, + conn, + hardwareIdManagementService, + hardwareIdPath): + """Delete given initiator path + + Delete the initiator. Do not rise exception or failure if deletion + fails due to any reasons. + + :param conn - connection to the ecom server + :param hardwareIdManagementService - hardware id management service + :param hardwareIdPath - The path of the initiator object + """ + ret = conn.InvokeMethod('DeleteStorageHardwareID', + hardwareIdManagementService, + HardwareID = hardwareIdPath) + if ret == 0: + LOG.debug("Deletion of initiator path %(hardwareIdPath)s " + "is successful.", {'hardwareIdPath': hardwareIdPath}) + else: + LOG.warning(_LW("Deletion of initiator path %(hardwareIdPath)s " + "is failed."), {'hardwareIdPath': hardwareIdPath}) + + def _delete_initiators_from_initiator_group(self, conn, + controllerConfigService, + initiatorGroupInstanceName, + initiatorGroupName): + """Delete initiators + + Delete all initiators associated with the initiator group instance. + Cleanup whatever is possible. It will not return any failure or + rise exception if deletion fails due to any reasons. + + :param conn - connection to the ecom server + :param controllerConfigService - controller config service + :param initiatorGroupInstanceName - the initiator group instance name + """ + storageHardwareIdInstanceNames = ( + conn.AssociatorNames(initiatorGroupInstanceName, + ResultClass='SE_StorageHardwareID')) + if len(storageHardwareIdInstanceNames) == 0: + LOG.debug("No initiators found in Initiator group " + "%(initiatorGroupName)s.", + {'initiatorGroupName': initiatorGroupName}) + return + storageSystemName = controllerConfigService['SystemName'] + hardwareIdManagementService = ( + self.utils.find_storage_hardwareid_service(conn, + storageSystemName)) + for storageHardwareIdInstanceName in storageHardwareIdInstanceNames: + initiatorName = storageHardwareIdInstanceName['InstanceID'] + hardwareIdPath = storageHardwareIdInstanceName + LOG.debug("Initiator %(initiatorName)s " + "will be deleted from the Initiator group " + "%(initiatorGroupName)s. HardwareIdPath is " + "%(hardwareIdPath)s.", + {'initiatorName': initiatorName, + 'initiatorGroupName': initiatorGroupName, + 'hardwareIdPath': hardwareIdPath}) + self._delete_storage_hardware_id(conn, + hardwareIdManagementService, + hardwareIdPath) + + def _last_volume_delete_initiator_group( + self, conn, controllerConfigService, + initiatorGroupInstanceName, extraSpecs): + """Delete the initiator group + + Delete the Initiator group if there are no masking views associated + with the initiator group. + + :param conn: the ecom connection + :param controllerConfigService: controller config service + :param igInstanceNames: initiator group instance name + :param extraSpecs: extra specifications + """ + maskingViewInstanceNames = self.get_masking_views_by_initiator_group( + conn, initiatorGroupInstanceName) + initiatorGroupInstance = conn.GetInstance(initiatorGroupInstanceName) + initiatorGroupName = initiatorGroupInstance['ElementName'] + + if len(maskingViewInstanceNames) == 0: + LOG.debug( + "Last volume is associated with the initiator group, deleting " + "the associated initiator group %(initiatorGroupName)s.", + {'initiatorGroupName': initiatorGroupName}) + self._delete_initiators_from_initiator_group( + conn, controllerConfigService, initiatorGroupInstanceName, + initiatorGroupName) + self._delete_initiator_group(conn, controllerConfigService, + initiatorGroupInstanceName, + initiatorGroupName, extraSpecs) + else: + LOG.warning(_LW("Initiator group %(initiatorGroupName)s is " + "associated with masking views and can't be " + "deleted. Number of associated masking view is: " + "%(nmv)d."), + {'initiatorGroupName': initiatorGroupName, + 'nmv': len(maskingViewInstanceNames)}) + + def _create_hardware_ids( + self, conn, initiatorNames, storageSystemName): + """Create hardwareIds for initiator(s). + + :param conn: the connection to the ecom server + :param initiatorNames: the list of initiator names + :param storageSystemName: the storage system name + :returns: list -- foundHardwareIDsInstanceNames + """ + foundHardwareIDsInstanceNames = [] + + hardwareIdManagementService = ( + self.utils.find_storage_hardwareid_service( + conn, storageSystemName)) + for initiatorName in initiatorNames: + hardwareIdInstanceName = ( + self.utils.create_storage_hardwareId_instance_name( + conn, hardwareIdManagementService, initiatorName)) + LOG.debug( + "Created hardwareId Instance: %(hardwareIdInstanceName)s.", + {'hardwareIdInstanceName': hardwareIdInstanceName}) + foundHardwareIDsInstanceNames.append(hardwareIdInstanceName) + + return foundHardwareIDsInstanceNames + + def _get_port_group_name_from_mv(self, conn, maskingViewName, + storageSystemName): + """Get the port group name from the masking view. + + :param conn: the connection to the ecom server + :param maskingViewName: the masking view name + :param storageSystemName: the storage system name + :returns: String - port group name + String - error message + """ + errorMessage = None + portGroupName = None + portGroupInstanceName = ( + self._get_port_group_from_masking_view( + conn, maskingViewName, storageSystemName)) + if portGroupInstanceName is None: + LOG.error(_LE( + "Cannot get port group from masking view: " + "%(maskingViewName)s. "), + {'maskingViewName': maskingViewName}) + else: + try: + portGroupInstance = ( + conn.GetInstance(portGroupInstanceName)) + portGroupName = ( + portGroupInstance['ElementName']) + except Exception: + LOG.error(_LE( + "Cannot get port group name.")) + return portGroupName, errorMessage + + def remove_device_from_storage_group( + self, conn, controllerConfigService, storageGroupInstanceName, + volumeInstance, volumeName, extraSpecs): + """Remove a device from a storage group. + + :param conn: the connection to the ecom server + :param controllerConfigService: the controller config service + :param storageGroupInstanceName: the sg instance + :param volumeInstance: the volume instance + :param extraSpecs: the extra specifications + :return: do_remove_vol_from_sg() + """ + @lockutils.synchronized(storageGroupInstanceName['ElementName'], + "emc-sg-", True) + def do_remove_vol_from_sg(): + self.provision.remove_device_from_storage_group( + conn, controllerConfigService, storageGroupInstanceName, + volumeInstance, volumeName, extraSpecs) + return do_remove_vol_from_sg() diff --git a/emc_vmax_provision.py b/emc_vmax_provision.py index 200fd50..55b202f 100644 --- a/emc_vmax_provision.py +++ b/emc_vmax_provision.py @@ -1,4 +1,4 @@ -# Copyright (c) 2012 - 2014 EMC Corporation. +# Copyright (c) 2012 - 2015 EMC Corporation. # All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); you may @@ -14,11 +14,12 @@ # under the License. import time +from oslo_concurrency import lockutils +from oslo_log import log as logging import six from cinder import exception -from cinder.i18n import _, _LE -from cinder.openstack.common import log as logging +from cinder.i18n import _ from cinder.volume.drivers.emc import emc_vmax_utils @@ -30,6 +31,9 @@ EMC_ROOT = 'root/emc' THINPROVISIONINGCOMPOSITE = 32768 THINPROVISIONING = 5 +SYNC_CLONE_LOCAL = 10 +COPY_ON_WRITE = 6 +TF_CLONE = 8 class EMCVMAXProvision(object): @@ -43,15 +47,17 @@ def __init__(self, prtcl): self.utils = emc_vmax_utils.EMCVMAXUtils(prtcl) def delete_volume_from_pool( - self, conn, storageConfigservice, volumeInstanceName, volumeName): + self, conn, storageConfigservice, volumeInstanceName, volumeName, + extraSpecs): """Given the volume instance remove it from the pool. - :param conn: connection the the ecom server + :param conn: connection to the ecom server :param storageConfigservice: volume created from job :param volumeInstanceName: the volume instance name :param volumeName: the volume name (String) - :param - :param rc: return code + :param extraSpecs: additional info + :returns: int -- return code + :raises: VolumeBackendAPIException """ startTime = time.time() @@ -62,15 +68,16 @@ def delete_volume_from_pool( theElements = [volumeInstanceName] rc, job = conn.InvokeMethod( - 'EMCReturnToStoragePool', storageConfigservice, + 'ReturnElementsToStoragePool', storageConfigservice, TheElements=theElements) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( - "Error Delete Volume: %(volumeName)s. " - "Return code: %(rc)lu. Error: %(error)s") + "Error Delete Volume: %(volumeName)s. " + "Return code: %(rc)lu. Error: %(error)s.") % {'volumeName': volumeName, 'rc': rc, 'error': errordesc}) @@ -79,7 +86,7 @@ def delete_volume_from_pool( data=exceptionMessage) LOG.debug("InvokeMethod EMCReturnToStoragePool took: " - "%(delta)s H:MM:SS", + "%(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) @@ -87,16 +94,18 @@ def delete_volume_from_pool( def create_volume_from_pool( self, conn, storageConfigService, volumeName, - poolInstanceName, volumeSize): + poolInstanceName, volumeSize, extraSpecs): """Create the volume in the specified pool. :param conn: the connection information to the ecom server :param storageConfigService: the storage configuration service :param volumeName: the volume name (String) :param poolInstanceName: the pool instance name to create - the dummy volume in + the dummy volume in :param volumeSize: volume size (String) - :returns: volumeDict - the volume dict + :param extraSpecs: additional info + :returns: dict -- the volume dict + :raises: VolumeBackendAPIException """ startTime = time.time() @@ -108,16 +117,17 @@ def create_volume_from_pool( Size=self.utils.get_num(volumeSize, '64'), EMCBindElements=False) - LOG.debug("Create Volume: %(volumename)s Return code: %(rc)lu", + LOG.debug("Create Volume: %(volumename)s Return code: %(rc)lu.", {'volumename': volumeName, 'rc': rc}) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( - "Error Create Volume: %(volumeName)s. " - "Return code: %(rc)lu. Error: %(error)s") + "Error Create Volume: %(volumeName)s. " + "Return code: %(rc)lu. Error: %(error)s.") % {'volumeName': volumeName, 'rc': rc, 'error': errordesc}) @@ -126,24 +136,27 @@ def create_volume_from_pool( data=exceptionMessage) LOG.debug("InvokeMethod CreateOrModifyElementFromStoragePool " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) - # Find the newly created volume + # Find the newly created volume. volumeDict = self.get_volume_dict_from_job(conn, job['Job']) return volumeDict, rc def create_and_get_storage_group(self, conn, controllerConfigService, - storageGroupName, volumeInstanceName): + storageGroupName, volumeInstanceName, + extraSpecs): """Create a storage group and return it. :param conn: the connection information to the ecom server :param controllerConfigService: the controller configuration service :param storageGroupName: the storage group name (String :param volumeInstanceName: the volume instance name + :param extraSpecs: additional info :returns: foundStorageGroupInstanceName - instance name of the - default storage group + default storage group + :raises: VolumeBackendAPIException """ startTime = time.time() @@ -152,12 +165,13 @@ def create_and_get_storage_group(self, conn, controllerConfigService, Type=self.utils.get_num(STORAGEGROUPTYPE, '16'), Members=[volumeInstanceName]) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( - "Error Create Group: %(groupName)s. " - "Return code: %(rc)lu. Error: %(error)s") + "Error Create Group: %(groupName)s. " + "Return code: %(rc)lu. Error: %(error)s.") % {'groupName': storageGroupName, 'rc': rc, 'error': errordesc}) @@ -166,7 +180,7 @@ def create_and_get_storage_group(self, conn, controllerConfigService, data=exceptionMessage) LOG.debug("InvokeMethod CreateGroup " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) foundStorageGroupInstanceName = self._find_new_storage_group( @@ -175,14 +189,15 @@ def create_and_get_storage_group(self, conn, controllerConfigService, return foundStorageGroupInstanceName def create_storage_group_no_members( - self, conn, controllerConfigService, groupName): + self, conn, controllerConfigService, groupName, extraSpecs): """Create a new storage group that has no members. - :param conn: connection the ecom server + :param conn: connection to the ecom server :param controllerConfigService: the controller configuration service :param groupName: the proposed group name - :returns: foundStorageGroupInstanceName - the instance Name of - the storage group + :param extraSpecs: additional info + :returns: foundStorageGroupInstanceName + :raises: VolumeBackendAPIException """ startTime = time.time() @@ -191,12 +206,13 @@ def create_storage_group_no_members( Type=self.utils.get_num(STORAGEGROUPTYPE, '16'), DeleteWhenBecomesUnassociated=False) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( - "Error Create Group: %(groupName)s. " - "Return code: %(rc)lu. Error: %(error)s") + "Error Create Group: %(groupName)s. " + "Return code: %(rc)lu. Error: %(error)s.") % {'groupName': groupName, 'rc': rc, 'error': errordesc}) @@ -205,7 +221,7 @@ def create_storage_group_no_members( data=exceptionMessage) LOG.debug("InvokeMethod CreateGroup " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) @@ -216,9 +232,9 @@ def create_storage_group_no_members( def _find_new_storage_group( self, conn, maskingGroupDict, storageGroupName): - """After creating an new storage group find it and return it. + """After creating a new storage group find it and return it. - :param conn: connection the ecom server + :param conn: connection to the ecom server :param maskingGroupDict: the maskingGroupDict dict :param storageGroupName: storage group name (String) :returns: maskingGroupDict['MaskingGroup'] @@ -234,7 +250,7 @@ def get_volume_dict_from_job(self, conn, jobInstance): :param conn: the ecom connection :param jobInstance: the instance of a job - :returns: volumeDict - an instance of a volume + :returns: dict -- volumeDict - an instance of a volume """ associators = conn.Associators( jobInstance, @@ -252,35 +268,38 @@ def get_volume_dict_from_job(self, conn, jobInstance): return volumeDict def remove_device_from_storage_group( - self, conn, controllerConfigService, storageGroupInstanceName, - volumeInstanceName, volumeName): + self, conn, controllerConfigService, sgInstanceName, + volumeInstanceName, volumeName, extraSpecs): """Remove a volume from a storage group. :param conn: the connection to the ecom server :param controllerConfigService: the controller configuration service - :param storageGroupInstanceName: the instance name of the storage group + :param sgInstanceName: the instance name of the storage group :param volumeInstanceName: the instance name of the volume :param volumeName: the volume name (String) - :returns: rc - the return code of the job + :param extraSpecs: additional info + :returns: int -- the return code of the job + :raises: VolumeBackendAPIException """ startTime = time.time() rc, jobDict = conn.InvokeMethod('RemoveMembers', controllerConfigService, - MaskingGroup=storageGroupInstanceName, + MaskingGroup=sgInstanceName, Members=[volumeInstanceName]) - if rc != 0L: - rc, errorDesc = self.utils.wait_for_job_complete(conn, jobDict) - if rc != 0L: + if rc != 0: + rc, errorDesc = self.utils.wait_for_job_complete(conn, jobDict, + extraSpecs) + if rc != 0: exceptionMessage = (_( - "Error removing volume %(vol)s. %(error)s") + "Error removing volume %(vol)s. %(error)s.") % {'vol': volumeName, 'error': errorDesc}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException( data=exceptionMessage) LOG.debug("InvokeMethod RemoveMembers " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) @@ -288,69 +307,90 @@ def remove_device_from_storage_group( def add_members_to_masking_group( self, conn, controllerConfigService, storageGroupInstanceName, - volumeInstanceName, volumeName): + volumeInstanceName, volumeName, extraSpecs): """Add a member to a masking group group. + :param conn: the connection to the ecom server :param controllerConfigService: the controller configuration service :param storageGroupInstanceName: the instance name of the storage group :param volumeInstanceName: the instance name of the volume :param volumeName: the volume name (String) + :param extraSpecs: additional info + :raises: VolumeBackendAPIException """ - startTime = time.time() - - rc, job = conn.InvokeMethod( - 'AddMembers', controllerConfigService, - MaskingGroup=storageGroupInstanceName, - Members=[volumeInstanceName]) + try: + storageGroupInstance = conn.GetInstance(storageGroupInstanceName) + except Exception: + exceptionMessage = (_( + "Unable to get the name of the storage group.")) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: - exceptionMessage = (_( - "Error mapping volume %(vol)s. %(error)s") - % {'vol': volumeName, 'error': errordesc}) - LOG.error(exceptionMessage) - raise exception.VolumeBackendAPIException( - data=exceptionMessage) + @lockutils.synchronized(storageGroupInstance['ElementName'], + "emc-sg-", True) + def do_add_volume_to_sg(): + startTime = time.time() + rc, job = conn.InvokeMethod( + 'AddMembers', controllerConfigService, + MaskingGroup=storageGroupInstanceName, + Members=[volumeInstanceName]) + + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: + exceptionMessage = (_( + "Error adding volume %(vol)s to %(sg)s. %(error)s.") + % {'vol': volumeName, + 'sg': storageGroupInstance['ElementName'], + 'error': errordesc}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) - LOG.debug("InvokeMethod AddMembers " - "took: %(delta)s H:MM:SS", - {'delta': self.utils.get_time_delta(startTime, - time.time())}) + LOG.debug("InvokeMethod AddMembers " + "took: %(delta)s H:MM:SS.", + {'delta': self.utils.get_time_delta(startTime, + time.time())}) + return rc + return do_add_volume_to_sg() def unbind_volume_from_storage_pool( - self, conn, storageConfigService, poolInstanceName, - volumeInstanceName, volumeName): + self, conn, storageConfigService, + volumeInstanceName, volumeName, extraSpecs): """Unbind a volume from a pool and return the unbound volume. :param conn: the connection information to the ecom server :param storageConfigService: the storage configuration service - instance name - :param poolInstanceName: the pool instance name + instance name :param volumeInstanceName: the volume instance name :param volumeName: the volume name - :returns: unboundVolumeInstance - the unbound volume instance + :param extraSpecs: additional info + :returns: int -- return code + :returns: the job object + :raises: VolumeBackendAPIException """ startTime = time.time() rc, job = conn.InvokeMethod( 'EMCUnBindElement', storageConfigService, - InPool=poolInstanceName, TheElement=volumeInstanceName) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( - "Error unbinding volume %(vol)s from pool. %(error)s") + "Error unbinding volume %(vol)s from pool. %(error)s.") % {'vol': volumeName, 'error': errordesc}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException( data=exceptionMessage) LOG.debug("InvokeMethod EMCUnBindElement " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) @@ -358,7 +398,7 @@ def unbind_volume_from_storage_pool( def modify_composite_volume( self, conn, elementCompositionService, theVolumeInstanceName, - inVolumeInstanceName): + inVolumeInstanceName, extraSpecs): """Given a composite volume add a storage volume to it. @@ -366,9 +406,11 @@ def modify_composite_volume( :param elementCompositionService: the element composition service :param theVolumeInstanceName: the existing composite volume :param inVolumeInstanceName: the volume you wish to add to the - composite volume - :returns: rc - return code - :returns: job - job + composite volume + :param extraSpecs: additional info + :returns: int -- rc - return code + :returns: the job object + :raises: VolumeBackendAPIException """ startTime = time.time() @@ -378,39 +420,42 @@ def modify_composite_volume( TheElement=theVolumeInstanceName, InElements=[inVolumeInstanceName]) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( "Error adding volume to composite volume. " - "Error is: %(error)s") + "Error is: %(error)s.") % {'error': errordesc}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException( data=exceptionMessage) LOG.debug("InvokeMethod CreateOrModifyCompositeElement " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) return rc, job def create_composite_volume( self, conn, elementCompositionService, volumeSize, volumeName, - poolInstanceName, compositeType, numMembers): + poolInstanceName, compositeType, numMembers, extraSpecs): """Create a new volume using the auto meta feature. - :param conn: the connection the the ecom server + :param conn: connection to the ecom server :param elementCompositionService: the element composition service :param volumeSize: the size of the volume :param volumeName: user friendly name :param poolInstanceName: the pool to bind the composite volume to :param compositeType: the proposed composite type of the volume - e.g striped/concatenated + e.g striped/concatenated :param numMembers: the number of meta members to make up the composite. - If it is 1 then a non composite is created - :returns: rc - :returns: errordesc + If it is 1 then a non composite is created + :param extraSpecs: additional info + :returns: dict -- volumeDict + :returns: int -- return code + :raises: VolumeBackendAPIException """ startTime = time.time() @@ -423,7 +468,7 @@ def create_composite_volume( "newMembers: %(newMembers)lu " "poolInstanceName: %(poolInstanceName)s " "compositeType: %(compositeType)lu " - "numMembers: %(numMembers)s ", + "numMembers: %(numMembers)s.", {'elementCompositionService': elementCompositionService, 'provisioning': THINPROVISIONINGCOMPOSITE, 'volumeSize': volumeSize, @@ -442,12 +487,13 @@ def create_composite_volume( CompositeType=self.utils.get_num(compositeType, '16'), EMCNumberOfMembers=self.utils.get_num(numMembers, '32')) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( - "Error Create Volume: %(volumename)s. " - "Return code: %(rc)lu. Error: %(error)s") + "Error Create Volume: %(volumename)s. " + "Return code: %(rc)lu. Error: %(error)s.") % {'volumename': volumeName, 'rc': rc, 'error': errordesc}) @@ -456,31 +502,33 @@ def create_composite_volume( data=exceptionMessage) LOG.debug("InvokeMethod CreateOrModifyCompositeElement " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) - # Find the newly created volume + # Find the newly created volume. volumeDict = self.get_volume_dict_from_job(conn, job['Job']) return volumeDict, rc def create_new_composite_volume( self, conn, elementCompositionService, compositeHeadInstanceName, - compositeMemberInstanceName, compositeType): + compositeMemberInstanceName, compositeType, extraSpecs): """Creates a new composite volume. Given a bound composite head and an unbound composite member create a new composite volume. - :param conn: the connection the the ecom server + :param conn: connection to the ecom server :param elementCompositionService: the element composition service :param compositeHeadInstanceName: the composite head. This can be bound - :param compositeMemberInstanceName: the composite member. - This must be unbound + :param compositeMemberInstanceName: the composite member. This must be + unbound :param compositeType: the composite type e.g striped or concatenated - :returns: rc - return code - :returns: errordesc - descriptions of the error + :param extraSpecs: additional info + :returns: int -- return code + :returns: the job object + :raises: VolumeBackendAPIException """ startTime = time.time() @@ -491,12 +539,13 @@ def create_new_composite_volume( [compositeHeadInstanceName, compositeMemberInstanceName]), CompositeType=self.utils.get_num(compositeType, '16')) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( - "Error Creating new composite Volume Return code: %(rc)lu." - "Error: %(error)s") + "Error Creating new composite Volume Return code: " + "%(rc)lu. Error: %(error)s.") % {'rc': rc, 'error': errordesc}) LOG.error(exceptionMessage) @@ -504,7 +553,7 @@ def create_new_composite_volume( data=exceptionMessage) LOG.debug("InvokeMethod CreateOrModifyCompositeElement " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) @@ -512,15 +561,17 @@ def create_new_composite_volume( def _migrate_volume( self, conn, storageRelocationServiceInstanceName, - volumeInstanceName, targetPoolInstanceName): + volumeInstanceName, targetPoolInstanceName, extraSpecs): """Migrate a volume to another pool. :param conn: the connection to the ecom server :param storageRelocationServiceInstanceName: the storage relocation - service + service :param volumeInstanceName: the volume to be migrated :param targetPoolInstanceName: the target pool to migrate the volume to - :returns: rc - return code + :param extraSpecs: additional info + :returns: int -- return code + :raises: VolumeBackendAPIException """ startTime = time.time() @@ -530,19 +581,20 @@ def _migrate_volume( TheElements=[volumeInstanceName], TargetPool=targetPoolInstanceName) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( "Error Migrating volume from one pool to another. " - "Return code: %(rc)lu. Error: %(error)s") + "Return code: %(rc)lu. Error: %(error)s.") % {'rc': rc, 'error': errordesc}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException( data=exceptionMessage) LOG.debug("InvokeMethod RelocateStorageVolumesToStoragePool " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) @@ -550,16 +602,18 @@ def _migrate_volume( def migrate_volume_to_storage_pool( self, conn, storageRelocationServiceInstanceName, - volumeInstanceName, targetPoolInstanceName): + volumeInstanceName, targetPoolInstanceName, extraSpecs): """Given the storage system name, get the storage relocation service. :param conn: the connection to the ecom server :param storageRelocationServiceInstanceName: the storage relocation - service + service :param volumeInstanceName: the volume to be migrated :param targetPoolInstanceName: the target pool to migrate the - volume to. - :returns: rc + volume to. + :param extraSpecs: additional info + :returns: int -- rc, return code + :raises: VolumeBackendAPIException """ LOG.debug( "Volume instance name is %(volumeInstanceName)s. " @@ -570,59 +624,61 @@ def migrate_volume_to_storage_pool( try: rc = self._migrate_volume( conn, storageRelocationServiceInstanceName, - volumeInstanceName, targetPoolInstanceName) + volumeInstanceName, targetPoolInstanceName, extraSpecs) except Exception as ex: if 'source of a migration session' in six.text_type(ex): try: rc = self._terminate_migrate_session( - conn, volumeInstanceName) - except Exception as ex: - LOG.error(_LE('Exception: %s'), ex) + conn, volumeInstanceName, extraSpecs) + except Exception: exceptionMessage = (_( - "Failed to terminate migrate session")) - LOG.error(exceptionMessage) + "Failed to terminate migrate session.")) + LOG.exception(exceptionMessage) raise exception.VolumeBackendAPIException( data=exceptionMessage) try: rc = self._migrate_volume( conn, storageRelocationServiceInstanceName, - volumeInstanceName, targetPoolInstanceName) - except Exception as ex: - LOG.error(_LE('Exception: %s'), ex) + volumeInstanceName, targetPoolInstanceName, + extraSpecs) + except Exception: exceptionMessage = (_( - "Failed to migrate volume for the second time")) - LOG.error(exceptionMessage) + "Failed to migrate volume for the second time.")) + LOG.exception(exceptionMessage) raise exception.VolumeBackendAPIException( data=exceptionMessage) else: - LOG.error(_LE('Exception: %s'), ex) exceptionMessage = (_( - "Failed to migrate volume for the first time")) - LOG.error(exceptionMessage) + "Failed to migrate volume for the first time.")) + LOG.exception(exceptionMessage) raise exception.VolumeBackendAPIException( data=exceptionMessage) return rc - def _terminate_migrate_session(self, conn, volumeInstanceName): + def _terminate_migrate_session(self, conn, volumeInstanceName, + extraSpecs): """Given the volume instance terminate a migrate session. :param conn: the connection to the ecom server :param volumeInstanceName: the volume to be migrated - :returns: rc + :param extraSpecs: additional info + :returns: int -- return code + :raises: VolumeBackendAPIException """ startTime = time.time() rc, job = conn.InvokeMethod( 'RequestStateChange', volumeInstanceName, RequestedState=self.utils.get_num(32769, '16')) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( "Error Terminating migrate session. " - "Return code: %(rc)lu. Error: %(error)s") + "Return code: %(rc)lu. Error: %(error)s.") % {'rc': rc, 'error': errordesc}) LOG.error(exceptionMessage) @@ -630,7 +686,7 @@ def _terminate_migrate_session(self, conn, volumeInstanceName): data=exceptionMessage) LOG.debug("InvokeMethod RequestStateChange " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) @@ -638,65 +694,36 @@ def _terminate_migrate_session(self, conn, volumeInstanceName): def create_element_replica( self, conn, repServiceInstanceName, cloneName, - sourceName, sourceInstance, targetInstance, copyOnWrite=False): - """Make SMI-S call to create replica for source element - - :param conn - the connection to the ecom server - :param repServiceInstanceName - replication service - :param cloneName - replica name - :param sourceName - source volume name - :param sourceInstance - source volume instance + sourceName, sourceInstance, targetInstance, extraSpecs, + copyOnWrite=False): + """Make SMI-S call to create replica for source element. - :returns: rc - return code - :returns: job - job object of the replica creation operation + :param conn: the connection to the ecom server + :param repServiceInstanceName: replication service + :param cloneName: replica name + :param sourceName: source volume name + :param sourceInstance: source volume instance + :param targetInstance: the target instance + :param extraSpecs: additional info + :param copyOnWrite: optional + :returns: int -- return code + :returns: job object of the replica creation operation + :raises: VolumeBackendAPIException """ if copyOnWrite: startTime = time.time() - repServiceCapabilityInstanceNames = conn.AssociatorNames( - repServiceInstanceName, - ResultClass='CIM_ReplicationServiceCapabilities', - AssocClass='CIM_ElementCapabilities') - repServiceCapabilityInstanceName = \ - repServiceCapabilityInstanceNames[0] - - # ReplicationType 10 - Synchronous Clone Local - rc, rsd = conn.InvokeMethod( - 'GetDefaultReplicationSettingData', - repServiceCapabilityInstanceName, - ReplicationType=self.utils.get_num(10, '16')) - - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, rsd) - if rc != 0L: - exceptionMessage = (_( - "Error Create Cloned Volume: " - "Volume: %(cloneName)s Source Volume:" - "%(sourceName)s. Return code: %(rc)lu." - "Error: %(error)s") - % {'cloneName': cloneName, - 'sourceName': sourceName, - 'rc': rc, - 'error': errordesc}) - LOG.error(exceptionMessage) - raise exception.VolumeBackendAPIException( - data=exceptionMessage) - - LOG.debug("InvokeMethod GetDefaultReplicationSettingData " - "took: %(delta)s H:MM:SS", - {'delta': self.utils.get_time_delta(startTime, - time.time())}) - - # Set DesiredCopyMethodology to Copy-On-Write (6) - rsdInstance = rsd['DefaultInstance'] - rsdInstance['DesiredCopyMethodology'] = self.utils.get_num(6, '16') - - startTime = time.time() - - # SyncType 8 - Clone - # ReplicationSettingData.DesiredCopyMethodology 6 - Copy-On-Write + # ReplicationType 10 - Synchronous Clone Local. + # Set DesiredCopyMethodology to Copy-On-Write (6). + rsdInstance = self.utils.set_copy_methodology_in_rsd( + conn, repServiceInstanceName, SYNC_CLONE_LOCAL, + COPY_ON_WRITE, extraSpecs) + + # SyncType 8 - Clone. + # ReplicationSettingData.DesiredCopyMethodology Copy-On-Write (6). rc, job = conn.InvokeMethod( 'CreateElementReplica', repServiceInstanceName, - ElementName=cloneName, SyncType=self.utils.get_num(8, '16'), + ElementName=cloneName, + SyncType=self.utils.get_num(TF_CLONE, '16'), ReplicationSettingData=rsdInstance, SourceElement=sourceInstance.path) else: @@ -705,24 +732,25 @@ def create_element_replica( rc, job = conn.InvokeMethod( 'CreateElementReplica', repServiceInstanceName, ElementName=cloneName, - SyncType=self.utils.get_num(8, '16'), + SyncType=self.utils.get_num(TF_CLONE, '16'), SourceElement=sourceInstance.path) else: rc, job = conn.InvokeMethod( 'CreateElementReplica', repServiceInstanceName, ElementName=cloneName, - SyncType=self.utils.get_num(8, '16'), + SyncType=self.utils.get_num(TF_CLONE, '16'), SourceElement=sourceInstance.path, TargetElement=targetInstance.path) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( "Error Create Cloned Volume: " "Volume: %(cloneName)s Source Volume:" - "%(sourceName)s. Return code: %(rc)lu." - "Error: %(error)s") + "%(sourceName)s. Return code: %(rc)lu. " + "Error: %(error)s.") % {'cloneName': cloneName, 'sourceName': sourceName, 'rc': rc, @@ -732,30 +760,31 @@ def create_element_replica( data=exceptionMessage) LOG.debug("InvokeMethod CreateElementReplica " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) return rc, job def delete_clone_relationship( - self, conn, repServiceInstanceName, syncInstanceName, force=False): + self, conn, repServiceInstanceName, syncInstanceName, extraSpecs, + force=False): """Deletes the relationship between the clone and source volume. Makes an SMI-S call to break clone relationship between the clone - volume and the source + volume and the source. + 8/Detach - Delete the synchronization between two storage objects. + Treat the objects as independent after the synchronization is deleted. :param conn: the connection to the ecom server :param repServiceInstanceName: instance name of the replication service :param syncInstanceName: instance name of the - SE_StorageSynchronized_SV_SV object - :returns: rc - return code - :returns: job - job object of the replica creation operation + SE_StorageSynchronized_SV_SV object + :param extraSpecs: additional info + :param force: optional param + :returns: int -- return code + :returns: job object of the replica creation operation + :raises: VolumeBackendAPIException """ - - ''' - 8/Detach - Delete the synchronization between two storage objects. - Treat the objects as independent after the synchronization is deleted. - ''' startTime = time.time() rc, job = conn.InvokeMethod( @@ -764,18 +793,19 @@ def delete_clone_relationship( Synchronization=syncInstanceName, Force=force) - LOG.debug("Delete clone relationship: Sync Name: %(syncName)s " - "Return code: %(rc)lu", + LOG.debug("Delete clone relationship: Sync Name: %(syncName)s " + "Return code: %(rc)lu.", {'syncName': syncInstanceName, 'rc': rc}) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( "Error break clone relationship: " - "Sync Name: %(syncName)s " - "Return code: %(rc)lu. Error: %(error)s") + "Sync Name: %(syncName)s " + "Return code: %(rc)lu. Error: %(error)s.") % {'syncName': syncInstanceName, 'rc': rc, 'error': errordesc}) @@ -784,48 +814,23 @@ def delete_clone_relationship( data=exceptionMessage) LOG.debug("InvokeMethod ModifyReplicaSynchronization " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) return rc, job - def get_target_endpoints(self, conn, storageHardwareService, hardwareId): - """Given the hardwareId get the - - :param conn: the connection to the ecom server - :param storageHardwareService: the storage HardwareId Service - :param hardwareId: the hardware Id - :returns: rc - :returns: targetendpoints - """ - startTime = time.time() - - rc, targetEndpoints = conn.InvokeMethod( - 'EMCGetTargetEndpoints', storageHardwareService, - HardwareId=hardwareId) - - if rc != 0L: - exceptionMessage = (_("Error finding Target WWNs.")) - LOG.error(exceptionMessage) - raise exception.VolumeBackendAPIException(data=exceptionMessage) - - LOG.debug("InvokeMethod EMCGetTargetEndpoints " - "took: %(delta)s H:MM:SS", - {'delta': self.utils.get_time_delta(startTime, - time.time())}) - - return rc, targetEndpoints - def create_consistency_group( - self, conn, replicationService, consistencyGroupName): - """Create a new consistency group + self, conn, replicationService, consistencyGroupName, extraSpecs): + """Create a new consistency group. :param conn: the connection to the ecom server :param replicationService: the replication Service :param consistencyGroupName: the CG group name - :returns: rc - :returns: job + :param extraSpecs: additional info + :returns: int -- return code + :returns: job object + :raises: VolumeBackendAPIException """ startTime = time.time() @@ -834,13 +839,14 @@ def create_consistency_group( replicationService, GroupName=consistencyGroupName) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( "Failed to create consistency group: " "%(consistencyGroupName)s " - "Return code: %(rc)lu. Error: %(error)s") + "Return code: %(rc)lu. Error: %(error)s.") % {'consistencyGroupName': consistencyGroupName, 'rc': rc, 'error': errordesc}) @@ -849,7 +855,7 @@ def create_consistency_group( data=exceptionMessage) LOG.debug("InvokeMethod CreateGroup " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) @@ -857,16 +863,18 @@ def create_consistency_group( def delete_consistency_group( self, conn, replicationService, cgInstanceName, - consistencyGroupName): + consistencyGroupName, extraSpecs): - """Delete a consistency group + """Delete a consistency group. :param conn: the connection to the ecom server :param replicationService: the replication Service :param cgInstanceName: the CG instance name :param consistencyGroupName: the CG group name - :returns: rc - :returns: job + :param extraSpecs: additional info + :returns: int -- return code + :returns: job object + :raises: VolumeBackendAPIException """ startTime = time.time() @@ -876,13 +884,14 @@ def delete_consistency_group( ReplicationGroup=cgInstanceName, RemoveElements=True) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( "Failed to delete consistency group: " - "%(consistencyGroupName)s " - "Return code: %(rc)lu. Error: %(error)s") + "%(consistencyGroupName)s " + "Return code: %(rc)lu. Error: %(error)s.") % {'consistencyGroupName': consistencyGroupName, 'rc': rc, 'error': errordesc}) @@ -891,7 +900,7 @@ def delete_consistency_group( data=exceptionMessage) LOG.debug("InvokeMethod DeleteGroup " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) @@ -899,33 +908,42 @@ def delete_consistency_group( def add_volume_to_cg( self, conn, replicationService, cgInstanceName, - volumeInstanceName, cgName, volumeName): - """Add a volume to a consistency group + volumeInstanceName, cgName, volumeName, extraSpecs): + """Add a volume to a consistency group. :param conn: the connection to the ecom server :param replicationService: the replication Service - :param volumeInstanceName: the volume instance name :param cgInstanceName: the CG instance name + :param volumeInstanceName: the volume instance name :param cgName: the CG group name :param volumeName: the volume name - :returns: rc - :returns: job + :param extraSpecs: additional info + :returns: int -- return code + :returns: job object + :raises: VolumeBackendAPIException """ startTime = time.time() + if isinstance(volumeInstanceName, list): + theElements = volumeInstanceName + volumeName = 'Bulk Add' + else: + theElements = [volumeInstanceName] + rc, job = conn.InvokeMethod( 'AddMembers', replicationService, - Members=[volumeInstanceName], + Members=theElements, ReplicationGroup=cgInstanceName) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( - "Failed to add volume %(volumeName)s: " - "to consistency group %(cgName)s " - "Return code: %(rc)lu. Error: %(error)s") + "Failed to add volume %(volumeName)s " + "to consistency group %(cgName)s. " + "Return code: %(rc)lu. Error: %(error)s.") % {'volumeName': volumeName, 'cgName': cgName, 'rc': rc, @@ -935,41 +953,49 @@ def add_volume_to_cg( data=exceptionMessage) LOG.debug("InvokeMethod AddMembers " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) return rc, job def remove_volume_from_cg( self, conn, replicationService, cgInstanceName, - volumeInstanceName, cgName, volumeName): - """Remove a volume from a consistency group + volumeInstanceName, cgName, volumeName, extraSpecs): + """Remove a volume from a consistency group. :param conn: the connection to the ecom server :param replicationService: the replication Service - :param volumeInstanceName: the volume instance name :param cgInstanceName: the CG instance name + :param volumeInstanceName: the volume instance name :param cgName: the CG group name :param volumeName: the volume name - :returns: rc - :returns: job + :param extraSpecs: additional info + :returns: int -- return code + :returns: job object + :raises: VolumeBackendAPIException """ startTime = time.time() + if isinstance(volumeInstanceName, list): + theElements = volumeInstanceName + volumeName = 'Bulk Remove' + else: + theElements = [volumeInstanceName] + rc, job = conn.InvokeMethod( 'RemoveMembers', replicationService, - Members=[volumeInstanceName], - ReplicationGroup=cgInstanceName, - RemoveElements=True) + Members=theElements, + ReplicationGroup=cgInstanceName) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( - "Failed to remove volume %(volumeName)s: " - "to consistency group %(cgName)s " - "Return code: %(rc)lu. Error: %(error)s") + "Failed to remove volume %(volumeName)s " + "from consistency group %(cgName)s. " + "Return code: %(rc)lu. Error: %(error)s.") % {'volumeName': volumeName, 'cgName': cgName, 'rc': rc, @@ -979,50 +1005,54 @@ def remove_volume_from_cg( data=exceptionMessage) LOG.debug("InvokeMethod RemoveMembers " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) return rc, job def create_group_replica( self, conn, replicationService, - srcGroupInstanceName, tgtGroupInstanceName, relationName): - """Make SMI-S call to create replica for source group + srcGroupInstanceName, tgtGroupInstanceName, relationName, + extraSpecs): + """Make SMI-S call to create replica for source group. - :param conn - the connection to the ecom server - :param repServiceInstanceName - replication service - :param srcGroupInstanceName - source group instance name - :param tgtGroupInstanceName - target group instance name - :param cgName - target group name - - :returns: rc - return code - :returns: job - job object of the replica creation operation + :param conn: the connection to the ecom server + :param replicationService: replication service + :param srcGroupInstanceName: source group instance name + :param tgtGroupInstanceName: target group instance name + :param relationName: relation name + :param extraSpecs: additional info + :returns: int -- return code + :returns: job object of the replica creation operation + :raises: VolumeBackendAPIException """ LOG.debug( "Parameters for CreateGroupReplica: " - "replicationService: %(replicationService)s " + "replicationService: %(replicationService)s " "RelationName: %(relationName)s " "sourceGroup: %(srcGroup)s " - "targetGroup: %(tgtGroup)s ", + "targetGroup: %(tgtGroup)s.", {'replicationService': replicationService, 'relationName': relationName, 'srcGroup': srcGroupInstanceName, 'tgtGroup': tgtGroupInstanceName}) - # 8 for clone + # 8 for clone. rc, job = conn.InvokeMethod( 'CreateGroupReplica', replicationService, RelationshipName=relationName, SourceGroup=srcGroupInstanceName, TargetGroup=tgtGroupInstanceName, - SyncType=self.utils.get_num(8, '16')) + SyncType=self.utils.get_num(8, '16'), + WaitForCopyState=self.utils.get_num(4, '16')) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMsg = (_("Error CreateGroupReplica: " - "source: %(source)s target: %(target)s." - "Return code: %(rc)lu. Error: %(error)s") + "source: %(source)s target: %(target)s. " + "Return code: %(rc)lu. Error: %(error)s.") % {'source': srcGroupInstanceName, 'target': tgtGroupInstanceName, 'rc': rc, diff --git a/emc_vmax_provision_v3.py b/emc_vmax_provision_v3.py index a38a5d8..a85feb2 100644 --- a/emc_vmax_provision_v3.py +++ b/emc_vmax_provision_v3.py @@ -1,4 +1,4 @@ -# Copyright (c) 2012 - 2014 EMC Corporation. +# Copyright (c) 2012 - 2015 EMC Corporation. # All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); you may @@ -15,9 +15,12 @@ import time +from oslo_concurrency import lockutils +from oslo_log import log as logging +import six + from cinder import exception -from cinder.i18n import _, _LE -from cinder.openstack.common import log as logging +from cinder.i18n import _, _LE, _LW from cinder.volume.drivers.emc import emc_vmax_utils LOG = logging.getLogger(__name__) @@ -28,6 +31,10 @@ EMC_ROOT = 'root/emc' THINPROVISIONINGCOMPOSITE = 32768 THINPROVISIONING = 5 +INFO_SRC_V3 = 3 +ACTIVATESNAPVX = 4 +DEACTIVATESNAPVX = 19 +SNAPSYNCTYPE = 7 class EMCVMAXProvisionV3(object): @@ -41,15 +48,17 @@ def __init__(self, prtcl): self.utils = emc_vmax_utils.EMCVMAXUtils(prtcl) def delete_volume_from_pool( - self, conn, storageConfigservice, volumeInstanceName, volumeName): + self, conn, storageConfigservice, volumeInstanceName, volumeName, + extraSpecs): """Given the volume instance remove it from the pool. - :param conn: connection the the ecom server + :param conn: connection to the ecom server :param storageConfigservice: volume created from job :param volumeInstanceName: the volume instance name :param volumeName: the volume name (String) - :param - :param rc: return code + :param extraSpecs: additional info + :returns: int -- return code + :raises: VolumeBackendAPIException """ startTime = time.time() @@ -63,12 +72,13 @@ def delete_volume_from_pool( 'ReturnElementsToStoragePool', storageConfigservice, TheElements=theElements) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( - "Error Delete Volume: %(volumeName)s. " - "Return code: %(rc)lu. Error: %(error)s") + "Error Delete Volume: %(volumeName)s. " + "Return code: %(rc)lu. Error: %(error)s.") % {'volumeName': volumeName, 'rc': rc, 'error': errordesc}) @@ -77,7 +87,7 @@ def delete_volume_from_pool( data=exceptionMessage) LOG.debug("InvokeMethod ReturnElementsToStoragePool took: " - "%(delta)s H:MM:SS", + "%(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) @@ -85,8 +95,8 @@ def delete_volume_from_pool( def create_volume_from_sg( self, conn, storageConfigService, volumeName, - sgInstanceName, volumeSize): - """Create the volume and associate it with a storage group + sgInstanceName, volumeSize, extraSpecs): + """Create the volume and associate it with a storage group. We use EMCCollections parameter to supply a Device Masking Group to contain a newly created storage volume. @@ -95,53 +105,71 @@ def create_volume_from_sg( :param storageConfigService: the storage configuration service :param volumeName: the volume name (String) :param sgInstanceName: the storage group instance name - associated with an SLO + associated with an SLO :param volumeSize: volume size (String) - :returns: volumeDict - the volume dict + :param extraSpecs: additional info + :returns: dict -- volumeDict - the volume dict + :returns: int -- return code + :raises: VolumeBackendAPIException """ - startTime = time.time() - - rc, job = conn.InvokeMethod( - 'CreateOrModifyElementFromStoragePool', - storageConfigService, ElementName=volumeName, - EMCCollections=[sgInstanceName], - ElementType=self.utils.get_num(THINPROVISIONING, '16'), - Size=self.utils.get_num(volumeSize, '64')) - - LOG.debug("Create Volume: %(volumename)s Return code: %(rc)lu", - {'volumename': volumeName, - 'rc': rc}) - - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: - exceptionMessage = (_( - "Error Create Volume: %(volumeName)s. " - "Return code: %(rc)lu. Error: %(error)s") - % {'volumeName': volumeName, - 'rc': rc, - 'error': errordesc}) - LOG.error(exceptionMessage) - raise exception.VolumeBackendAPIException( - data=exceptionMessage) - - LOG.debug("InvokeMethod CreateOrModifyElementFromStoragePool " - "took: %(delta)s H:MM:SS", - {'delta': self.utils.get_time_delta(startTime, - time.time())}) + try: + storageGroupInstance = conn.GetInstance(sgInstanceName) + except Exception: + exceptionMessage = (_( + "Unable to get the name of the storage group")) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + + @lockutils.synchronized(storageGroupInstance['ElementName'], + "emc-sg-", True) + def do_create_volume_from_sg(): + startTime = time.time() - # Find the newly created volume - volumeDict = self.get_volume_dict_from_job(conn, job['Job']) - return volumeDict, rc + rc, job = conn.InvokeMethod( + 'CreateOrModifyElementFromStoragePool', + storageConfigService, ElementName=volumeName, + EMCCollections=[sgInstanceName], + ElementType=self.utils.get_num(THINPROVISIONING, '16'), + Size=self.utils.get_num(volumeSize, '64')) + + LOG.debug("Create Volume: %(volumename)s. Return code: %(rc)lu.", + {'volumename': volumeName, + 'rc': rc}) + + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: + exceptionMessage = (_( + "Error Create Volume: %(volumeName)s. " + "Return code: %(rc)lu. Error: %(error)s.") + % {'volumeName': volumeName, + 'rc': rc, + 'error': errordesc}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + + LOG.debug("InvokeMethod CreateOrModifyElementFromStoragePool " + "took: %(delta)s H:MM:SS.", + {'delta': self.utils.get_time_delta(startTime, + time.time())}) + + # Find the newly created volume. + volumeDict = self.get_volume_dict_from_job(conn, job['Job']) + return volumeDict, rc + + return do_create_volume_from_sg() def _find_new_storage_group( self, conn, maskingGroupDict, storageGroupName): """After creating an new storage group find it and return it. - :param conn: connection the ecom server + :param conn: connection to the ecom server :param maskingGroupDict: the maskingGroupDict dict :param storageGroupName: storage group name (String) - :returns: maskingGroupDict['MaskingGroup'] + :returns: maskingGroupDict['MaskingGroup'] or None """ foundStorageGroupInstanceName = None if 'MaskingGroup' in maskingGroupDict: @@ -154,12 +182,44 @@ def get_volume_dict_from_job(self, conn, jobInstance): :param conn: the ecom connection :param jobInstance: the instance of a job - :returns: volumeDict - an instance of a volume + :returns: dict -- volumeDict - an instance of a volume + """ + associators = conn.Associators( + jobInstance, + ResultClass='EMC_StorageVolume') + if len(associators) > 0: + return self.create_volume_dict(associators[0].path) + else: + exceptionMessage = (_( + "Unable to get storage volume from job.")) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException(data=exceptionMessage) + + def get_volume_from_job(self, conn, jobInstance): + """Given the jobInstance determine the volume Instance. + + :param conn: the ecom connection + :param jobInstance: the instance of a job + :returns: dict -- volumeDict - an instance of a volume """ associators = conn.Associators( jobInstance, ResultClass='EMC_StorageVolume') - volpath = associators[0].path + if len(associators) > 0: + return associators[0] + else: + exceptionMessage = (_( + "Unable to get storage volume from job.")) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException(data=exceptionMessage) + + def create_volume_dict(self, volumeInstanceName): + """Create volume dictionary + + :param volumeInstanceName: the instance of a job + :returns: dict -- volumeDict - an instance of a volume + """ + volpath = volumeInstanceName volumeDict = {} volumeDict['classname'] = volpath.classname keys = {} @@ -173,90 +233,157 @@ def get_volume_dict_from_job(self, conn, jobInstance): def create_element_replica( self, conn, repServiceInstanceName, - cloneName, syncType, sourceInstance, targetInstance=None): - """Make SMI-S call to create replica for source element - - :param conn - the connection to the ecom server - :param repServiceInstanceName - replication service - :param cloneName - clone volume name - :param syncType - 7: snapshot, 8: clone - :param sourceInstance - source volume instance - :param targetInstance - target volume instance - :returns: rc - return code + cloneName, syncType, sourceInstance, extraSpecs, + copyState=None, targetInstance=None, rsdInstance=None): + """Make SMI-S call to create replica for source element. + + :param conn: the connection to the ecom server + :param repServiceInstanceName: replication service + :param cloneName: clone volume name + :param syncType: 7=snapshot, 8=clone + :param sourceInstance: source volume instance + :param extraSpecs: additional info + :param copyState: wait for copy state + :param targetInstance: Target volume instance. Default None + :param rsdInstance: replication settingdata instance. Default None + :returns: int -- rc - return code :returns: job - job object of the replica creation operation + :raises: VolumeBackendAPIException """ startTime = time.time() + LOG.debug("Create replica: %(clone)s " + "syncType: %(syncType)s Source: %(source)s.", + {'clone': cloneName, + 'syncType': syncType, + 'source': sourceInstance.path}) + storageSystemName = sourceInstance['SystemName'] + __, __, sgInstanceName = ( + self.utils.get_v3_default_sg_instance_name( + conn, extraSpecs[self.utils.POOL], + extraSpecs[self.utils.SLO], + extraSpecs[self.utils.WORKLOAD], storageSystemName)) + try: + storageGroupInstance = conn.GetInstance(sgInstanceName) + except Exception: + exceptionMessage = (_( + "Unable to get the name of the storage group")) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + + @lockutils.synchronized(storageGroupInstance['ElementName'], + "emc-sg-", True) + def do_create_element_replica(): + if targetInstance is None and rsdInstance is None: + rc, job = conn.InvokeMethod( + 'CreateElementReplica', repServiceInstanceName, + ElementName=cloneName, + SyncType=self.utils.get_num(syncType, '16'), + SourceElement=sourceInstance.path, + Collections=[sgInstanceName]) + else: + rc, job = self._create_element_replica_extra_params( + conn, repServiceInstanceName, cloneName, syncType, + sourceInstance, targetInstance, rsdInstance, + sgInstanceName, copyState) + + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: + exceptionMessage = (_( + "Error Create Cloned Volume: %(cloneName)s " + "Return code: %(rc)lu. Error: %(error)s.") + % {'cloneName': cloneName, + 'rc': rc, + 'error': errordesc}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + + LOG.debug("InvokeMethod CreateElementReplica " + "took: %(delta)s H:MM:SS.", + {'delta': self.utils.get_time_delta(startTime, + time.time())}) + return rc, job + + return do_create_element_replica() + + def _create_element_replica_extra_params( + self, conn, repServiceInstanceName, cloneName, syncType, + sourceInstance, targetInstance, rsdInstance, sgInstanceName, + copyState): + """CreateElementReplica using extra parameters. - if targetInstance is None: - LOG.debug("Create targetless replica: %(clone)s " - "syncType: %(syncType)s Source: %(source)s", - {'clone': cloneName, - 'syncType': syncType, - 'source': sourceInstance.path}) + :param conn: the connection to the ecom server + :param repServiceInstanceName: replication service + :param cloneName: clone volume name + :param syncType: 7=snapshot, 8=clone + :param sourceInstance: source volume instance + :param targetInstance: Target volume instance. Default None + :param rsdInstance: replication settingdata instance. Default None + :param sgInstanceName: pool instance name + :param copyState: wait for copy state + :returns: int -- rc - return code + :returns: job - job object of the replica creation operation + """ + syncType = self.utils.get_num(syncType, '16') + if targetInstance and rsdInstance: rc, job = conn.InvokeMethod( 'CreateElementReplica', repServiceInstanceName, - ElementName=cloneName, SyncType=syncType, - SourceElement=sourceInstance.path) - else: - LOG.debug( - "Create replica: %(clone)s syncType: %(syncType)s " - "Source: %(source)s target: %(target)s", - {'clone': cloneName, - 'syncType': syncType, - 'source': sourceInstance.path, - 'target': targetInstance.path}) + ElementName=cloneName, + SyncType=syncType, + SourceElement=sourceInstance.path, + TargetElement=targetInstance.path, + ReplicationSettingData=rsdInstance) + elif targetInstance: rc, job = conn.InvokeMethod( 'CreateElementReplica', repServiceInstanceName, - ElementName=cloneName, SyncType=syncType, + ElementName=cloneName, + SyncType=syncType, SourceElement=sourceInstance.path, - TargetElement=targetInstance.path) - - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: - exceptionMessage = (_( - "Error Create Cloned Volume: %(cloneName)s " - "Return code: %(rc)lu. Error: %(error)s") - % {'cloneName': cloneName, - 'rc': rc, - 'error': errordesc}) - LOG.error(exceptionMessage) - raise exception.VolumeBackendAPIException( - data=exceptionMessage) + TargetElement=targetInstance.path, + WaitForCopyState=copyState) + elif rsdInstance: + rc, job = conn.InvokeMethod( + 'CreateElementReplica', repServiceInstanceName, + ElementName=cloneName, + SyncType=syncType, + SourceElement=sourceInstance.path, + ReplicationSettingData=rsdInstance, + Collections=[sgInstanceName], + WaitForCopyState=copyState) - LOG.debug("InvokeMethod CreateElementReplica " - "took: %(delta)s H:MM:SS", - {'delta': self.utils.get_time_delta(startTime, - time.time())}) return rc, job def break_replication_relationship( self, conn, repServiceInstanceName, syncInstanceName, - operation, force=False): + operation, extraSpecs, force=False): """Deletes the relationship between the clone/snap and source volume. Makes an SMI-S call to break clone relationship between the clone - volume and the source + volume and the source. :param conn: the connection to the ecom server :param repServiceInstanceName: instance name of the replication service :param syncInstanceName: instance name of the - SE_StorageSynchronized_SV_SV object - :param cloneName: replica name - :param sourceName: source volume name - :param sourceInstance: source volume instance + SE_StorageSynchronized_SV_SV object + :param operation: operation code + :param extraSpecs: additional info + :param force: force to break replication relationship if True :returns: rc - return code :returns: job - job object of the replica creation operation """ - LOG.debug("Break replication relationship: %(sv)s " - "operation: %(operation)s", + LOG.debug("Break replication relationship: %(sv)s " + "operation: %(operation)s.", {'sv': syncInstanceName, 'operation': operation}) return self._modify_replica_synchronization( - conn, repServiceInstanceName, syncInstanceName, operation, force) + conn, repServiceInstanceName, syncInstanceName, operation, + extraSpecs, force) def create_storage_group_v3(self, conn, controllerConfigService, - groupName, srp, slo, workload): + groupName, srp, slo, workload, extraSpecs): """Create the volume in the specified pool. :param conn: the connection information to the ecom server @@ -265,48 +392,50 @@ def create_storage_group_v3(self, conn, controllerConfigService, :param srp: the SRP (String) :param slo: the SLO (String) :param workload: the workload (String) - :returns: volumeDict - the volume dict - + :param extraSpecs: additional info + :returns: storageGroupInstanceName - storage group instance name """ startTime = time.time() - rc, job = conn.InvokeMethod( - 'CreateGroup', - controllerConfigService, - GroupName=groupName, - Type=self.utils.get_num(4, '16'), - EMCSRP=srp, - EMCSLO=slo, - EMCWorkload=workload) - - if rc != 0L: - rc, errordesc = rc, errordesc = self.utils.wait_for_job_complete( - conn, job) - if rc != 0L: - LOG.error(_LE( - 'Error Create Group: %(groupName)s. ' - 'Return code: %(rc)lu. Error: %(error)s'), - {'groupName': groupName, - 'rc': rc, - 'error': errordesc}) - raise - - LOG.debug("InvokeMethod CreateGroup " - "took: %(delta)s H:MM:SS", - {'delta': self.utils.get_time_delta(startTime, - time.time())}) - - foundStorageGroupInstanceName = self._find_new_storage_group( - conn, job, groupName) - - return foundStorageGroupInstanceName - - def _get_storage_pool_capability(self, conn, poolInstanceName): - """get the pool capability. + @lockutils.synchronized(groupName, "emc-sg-", True) + def do_create_storage_group_v3(): + rc, job = conn.InvokeMethod( + 'CreateGroup', + controllerConfigService, + GroupName=groupName, + Type=self.utils.get_num(4, '16'), + EMCSRP=srp, + EMCSLO=slo, + EMCWorkload=workload) + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete( + conn, job, extraSpecs) + if rc != 0: + LOG.error(_LE( + "Error Create Group: %(groupName)s. " + "Return code: %(rc)lu. Error: %(error)s."), + {'groupName': groupName, + 'rc': rc, + 'error': errordesc}) + raise + + LOG.debug("InvokeMethod CreateGroup " + "took: %(delta)s H:MM:SS.", + {'delta': self.utils.get_time_delta(startTime, + time.time())}) + + foundStorageGroupInstanceName = self._find_new_storage_group( + conn, job, groupName) + return foundStorageGroupInstanceName + + return do_create_storage_group_v3() + + def get_storage_pool_capability(self, conn, poolInstanceName): + """Get the pool capability. :param conn: the connection information to the ecom server :param poolInstanceName: the pool instance - :returns: storagePoolCapability - the storage pool capability instance + :returns: the storage pool capability instance. None if not found """ storagePoolCapability = None @@ -319,15 +448,15 @@ def _get_storage_pool_capability(self, conn, poolInstanceName): return storagePoolCapability - def _get_storage_pool_setting( + def get_storage_pool_setting( self, conn, storagePoolCapability, slo, workload): - """get the pool setting for pool capability. + """Get the pool setting for pool capability. :param conn: the connection information to the ecom server :param storagePoolCapability: the storage pool capability instance :param slo: the slo string e.g Bronze :param workload: the workload string e.g DSS_REP - :returns: foundStoragePoolSetting - the storage pool setting instance + :returns: the storage pool setting instance """ foundStoragePoolSetting = None @@ -337,25 +466,36 @@ def _get_storage_pool_setting( for storagePoolSetting in storagePoolSettings: settingInstanceID = storagePoolSetting['InstanceID'] - matchString = slo + ':' + workload + matchString = ("%(slo)s:%(workload)s" + % {'slo': slo, + 'workload': workload}) if matchString in settingInstanceID: foundStoragePoolSetting = storagePoolSetting break + if foundStoragePoolSetting is None: + exceptionMessage = (_( + "The array does not support the storage pool setting " + "for SLO %(slo)s and workload %(workload)s. Please " + "check the array for valid SLOs and workloads.") + % {'slo': slo, + 'workload': workload}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) return foundStoragePoolSetting - # New - # GetSupportedSizes, GetSupportedSizeRange updated to return - # SLO headroom information. def _get_supported_size_range_for_SLO( self, conn, storageConfigService, - srpPoolInstanceName, storagePoolSettingInstanceName): + srpPoolInstanceName, storagePoolSettingInstanceName, extraSpecs): """Gets available performance capacity per SLO. :param conn: the connection information to the ecom server + :param storageConfigService: the storage configuration service instance :param srpPoolInstanceName: the SRP storage pool instance - :param storagePoolSettingInstanceName: the SLO type - e.g Bronze - :returns: volumeDict - the volume dict + :param storagePoolSettingInstanceName: the SLO type, e.g Bronze + :param extraSpecs: additional info + :returns: dict -- supportedSizeDict - the supported size dict + :raises: VolumeBackendAPIException """ startTime = time.time() @@ -365,13 +505,13 @@ def _get_supported_size_range_for_SLO( ElementType=self.utils.get_num(3, '16'), Goal=storagePoolSettingInstanceName) - if rc != 0L: + if rc != 0: rc, errordesc = self.utils.wait_for_job_complete( - conn, supportedSizeDict) - if rc != 0L: + conn, supportedSizeDict, extraSpecs) + if rc != 0: exceptionMessage = (_( "Cannot get supported size range for %(sps)s " - "Return code: %(rc)lu. Error: %(error)s") + "Return code: %(rc)lu. Error: %(error)s.") % {'sps': storagePoolSettingInstanceName, 'rc': rc, 'error': errordesc}) @@ -380,120 +520,119 @@ def _get_supported_size_range_for_SLO( data=exceptionMessage) LOG.debug("InvokeMethod GetSupportedSizeRange " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) return supportedSizeDict def get_volume_range( - self, conn, storageConfigService, poolInstanceName, slo, workload): - """get upper and lower range for volume for slo/workload combination + self, conn, storageConfigService, poolInstanceName, slo, workload, + extraSpecs): + """Get upper and lower range for volume for slo/workload combination. :param conn: the connection information to the ecom server :param storageConfigService: the storage config service :param poolInstanceName: the pool instance - :param volumeSize: volume size (bits) :param slo: slo string e.g Bronze :param workload: workload string e.g DSS - :returns: storagePoolCapability - the storage pool capability instance + :param extraSpecs: additional info + :returns: supportedSizeDict """ - maximumVolumeSize = None - minimumVolumeSize = None - - storagePoolCapabilityInstanceName = self._get_storage_pool_capability( + supportedSizeDict = {} + storagePoolCapabilityInstanceName = self.get_storage_pool_capability( conn, poolInstanceName) if storagePoolCapabilityInstanceName: - storagePoolSettingInstanceName = self._get_storage_pool_setting( + storagePoolSettingInstanceName = self.get_storage_pool_setting( conn, storagePoolCapabilityInstanceName, slo, workload) - if storagePoolCapabilityInstanceName: - supportedSizeDict = self._get_supported_size_range_for_SLO( - conn, storageConfigService, poolInstanceName, - storagePoolSettingInstanceName) - - maximumVolumeSize = supportedSizeDict['MaximumVolumeSize'] - minimumVolumeSize = supportedSizeDict['MinimumVolumeSize'] - - return maximumVolumeSize, minimumVolumeSize + supportedSizeDict = self._get_supported_size_range_for_SLO( + conn, storageConfigService, poolInstanceName, + storagePoolSettingInstanceName, extraSpecs) + return supportedSizeDict def activate_snap_relationship( - self, conn, repServiceInstanceName, syncInstanceName): + self, conn, repServiceInstanceName, syncInstanceName, extraSpecs): """Activate snap relationship and start copy operation. :param conn: the connection to the ecom server :param repServiceInstanceName: instance name of the replication service :param syncInstanceName: instance name of the - SE_StorageSynchronized_SV_SV object - :returns: rc - return code - :returns: job - job object of the replica creation operation + SE_StorageSynchronized_SV_SV object + :param extraSpecs: additional info + :returns: int -- return code + :returns: job object of the replica creation operation """ - # operation 4: activate the snapVx - operation = self.utils.get_num(4, '16') + # Operation 4: activate the snapVx. + operation = ACTIVATESNAPVX - LOG.debug("Activate snap: %(sv)s operation: %(operation)s ", + LOG.debug("Activate snap: %(sv)s operation: %(operation)s.", {'sv': syncInstanceName, 'operation': operation}) return self._modify_replica_synchronization( - conn, repServiceInstanceName, syncInstanceName, operation) + conn, repServiceInstanceName, syncInstanceName, operation, + extraSpecs) def return_to_resource_pool(self, conn, repServiceInstanceName, - syncInstanceName): - """return the snap target resources back to the pool + syncInstanceName, extraSpecs): + """Return the snap target resources back to the pool. :param conn: the connection to the ecom server :param repServiceInstanceName: instance name of the replication service :param syncInstanceName: instance name of the + :param extraSpecs: additional info :returns: rc - return code - :returns: job - job object of the replica creation operation + :returns: job object of the replica creation operation """ - # operation 4: activate the snapVx - operation = self.utils.get_num(19, '16') + # Operation 4: activate the snapVx. + operation = DEACTIVATESNAPVX LOG.debug("Return snap resource back to pool: " - "%(sv)s operation: %(operation)s ", + "%(sv)s operation: %(operation)s.", {'sv': syncInstanceName, 'operation': operation}) return self._modify_replica_synchronization( - conn, repServiceInstanceName, syncInstanceName, operation) + conn, repServiceInstanceName, syncInstanceName, operation, + extraSpecs) def _modify_replica_synchronization( self, conn, repServiceInstanceName, syncInstanceName, - operation, force=False): - """Helper function to modify the relationship between the clone/snap - and source volume. + operation, extraSpecs, force=False): + """Modify the relationship between the clone/snap and source volume. - Makes an SMI-S call to break clone relationship between the clone - volume and the source + Helper function that makes an SMI-S call to break clone relationship + between the clone volume and the source. :param conn: the connection to the ecom server :param repServiceInstanceName: instance name of the replication service :param syncInstanceName: instance name of the - SE_StorageSynchronized_SV_SV object - :param cloneName: replica name - :param sourceName: source volume name - :param sourceInstance: source volume instance - :returns: rc - return code - :returns: job - job object of the replica creation operation + SE_StorageSynchronized_SV_SV object + :param operation: operation code + :param extraSpecs: additional info + :param force: force to modify replication synchronization if True + :returns: int -- return code + :returns: job object of the replica creation operation + :raises: VolumeBackendAPIException """ startTime = time.time() rc, job = conn.InvokeMethod( 'ModifyReplicaSynchronization', repServiceInstanceName, - Operation=operation, + Operation=self.utils.get_num(operation, '16'), Synchronization=syncInstanceName, Force=force) - LOG.debug("_modify_replica_synchronization: %(sv)s " - "operation: %(operation)s Return code: %(rc)lu", + LOG.debug("_modify_replica_synchronization: %(sv)s " + "operation: %(operation)s Return code: %(rc)lu.", {'sv': syncInstanceName, 'operation': operation, 'rc': rc}) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMessage = (_( - "Error modify replica synchronization: %(sv)s " - "operation: %(operation)s. " - "Return code: %(rc)lu. Error: %(error)s") + "Error modify replica synchronization: %(sv)s " + "operation: %(operation)s. " + "Return code: %(rc)lu. Error: %(error)s.") % {'sv': syncInstanceName, 'operation': operation, 'rc': rc, 'error': errordesc}) LOG.error(exceptionMessage) @@ -501,7 +640,7 @@ def _modify_replica_synchronization( data=exceptionMessage) LOG.debug("InvokeMethod ModifyReplicaSynchronization " - "took: %(delta)s H:MM:SS", + "took: %(delta)s H:MM:SS.", {'delta': self.utils.get_time_delta(startTime, time.time())}) @@ -509,44 +648,45 @@ def _modify_replica_synchronization( def create_group_replica( self, conn, replicationService, - srcGroupInstanceName, tgtGroupInstanceName, relationName): - """Make SMI-S call to create replica for source group + srcGroupInstanceName, tgtGroupInstanceName, relationName, + extraSpecs): + """Make SMI-S call to create replica for source group. - :param conn - the connection to the ecom server - :param repServiceInstanceName - replication service - :param srcGroupInstanceName - source group instance name - :param tgtGroupInstanceName - target group instance name - :param cgName - target group name - - :returns: rc - return code - :returns: job - job object of the replica creation operation + :param conn: the connection to the ecom server + :param replicationService: replication service + :param srcGroupInstanceName: source group instance name + :param tgtGroupInstanceName: target group instance name + :param relationName: replica relationship name + :param extraSpecs: additional info + :returns: int -- return code + :returns: job object of the replica creation operation + :raises: VolumeBackendAPIException """ LOG.debug( "Creating CreateGroupReplica V3: " "replicationService: %(replicationService)s " "RelationName: %(relationName)s " "sourceGroup: %(srcGroup)s " - "targetGroup: %(tgtGroup)s ", + "targetGroup: %(tgtGroup)s.", {'replicationService': replicationService, 'relationName': relationName, 'srcGroup': srcGroupInstanceName, 'tgtGroup': tgtGroupInstanceName}) - # 7 for snap - syncType = 7 rc, job = conn.InvokeMethod( 'CreateGroupReplica', replicationService, RelationshipName=relationName, SourceGroup=srcGroupInstanceName, TargetGroup=tgtGroupInstanceName, - SyncType=self.utils.get_num(syncType, '16')) + SyncType=self.utils.get_num(SNAPSYNCTYPE, '16')) - if rc != 0L: - rc, errordesc = self.utils.wait_for_job_complete(conn, job) - if rc != 0L: + if rc != 0: + rc, errordesc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: exceptionMsg = (_("Error CreateGroupReplica: " - "source: %(source)s target: %(target)s." - "Return code: %(rc)lu. Error: %(error)s") + "source: %(source)s target: %(target)s. " + "Return code: %(rc)lu. Error: %(error)s.") % {'source': srcGroupInstanceName, 'target': tgtGroupInstanceName, 'rc': rc, @@ -554,3 +694,164 @@ def create_group_replica( LOG.error(exceptionMsg) raise exception.VolumeBackendAPIException(data=exceptionMsg) return rc, job + + def get_srp_pool_stats(self, conn, arrayInfo): + """Get the totalManagedSpace, remainingManagedSpace. + + Capacity can be got in 2 ways depending on your configuration. + 1. The total capacity of the SRP, if you dont have WLP enabled. + 2. The SLO capacity, if you do + + :param conn: the connection to the ecom server + :param arrayInfo: the array dict + :returns: totalCapacityGb + :returns: remainingCapacityGb + :returns: subscribedCapacityGb + :returns: array_reserve_percent + """ + totalCapacityGb = -1 + remainingCapacityGb = -1 + subscribedCapacityGb = -1 + array_reserve_percent = -1 + storageSystemInstanceName = self.utils.find_storageSystem( + conn, arrayInfo['SerialNumber']) + + srpPoolInstanceNames = conn.AssociatorNames( + storageSystemInstanceName, + ResultClass='Symm_SRPStoragePool') + + for srpPoolInstanceName in srpPoolInstanceNames: + poolnameStr = self.utils.get_pool_name(conn, srpPoolInstanceName) + + if six.text_type(arrayInfo['PoolName']) == ( + six.text_type(poolnameStr)): + try: + # Check that pool hasn't suddently been deleted. + srpPoolInstance = conn.GetInstance(srpPoolInstanceName) + propertiesList = srpPoolInstance.properties.items() + for properties in propertiesList: + if properties[0] == 'TotalManagedSpace': + cimProperties = properties[1] + totalManagedSpace = cimProperties.value + totalCapacityGb = self.utils.convert_bits_to_gbs( + totalManagedSpace) + elif properties[0] == 'RemainingManagedSpace': + cimProperties = properties[1] + remainingManagedSpace = cimProperties.value + remainingCapacityGb = ( + self.utils.convert_bits_to_gbs( + remainingManagedSpace)) + elif properties[0] == 'EMCSubscribedCapacity': + cimProperties = properties[1] + subscribedManagedSpace = cimProperties.value + subscribedCapacityGb = ( + self.utils.convert_bits_to_gbs( + subscribedManagedSpace)) + elif properties[0] == 'EMCPercentReservedCapacity': + cimProperties = properties[1] + array_reserve_percent = int(cimProperties.value) + except Exception: + pass + remainingSLOCapacityGb = ( + self._get_remaining_slo_capacity_wlp( + conn, srpPoolInstanceName, arrayInfo, + storageSystemInstanceName['Name'])) + if remainingSLOCapacityGb != -1: + remainingCapacityGb = remainingSLOCapacityGb + else: + LOG.warning(_LW( + "Remaining capacity %(remainingCapacityGb)s " + "GBs is determined from SRP pool capacity " + "and not the SLO capacity. Performance may " + "not be what you expect."), + {'remainingCapacityGb': remainingCapacityGb}) + + return (totalCapacityGb, remainingCapacityGb, subscribedCapacityGb, + array_reserve_percent) + + def _get_remaining_slo_capacity_wlp(self, conn, srpPoolInstanceName, + arrayInfo, systemName): + """Get the remaining SLO capacity. + + This is derived from the WLP portion of Unisphere. Please + see the SMIProvider doc and the readme doc for details. + + :param conn: the connection to the ecom server + :param srpPoolInstanceName: SRP instance name + :param arrayInfo: the array dict + :param systemName: the system name + :returns: remainingCapacityGb + """ + remainingCapacityGb = -1 + storageConfigService = ( + self.utils.find_storage_configuration_service( + conn, systemName)) + + supportedSizeDict = ( + self.get_volume_range( + conn, storageConfigService, srpPoolInstanceName, + arrayInfo['SLO'], arrayInfo['Workload'], + None)) + try: + # Information source is V3. + if supportedSizeDict['EMCInformationSource'] == INFO_SRC_V3: + remainingCapacityGb = self.utils.convert_bits_to_gbs( + supportedSizeDict['EMCRemainingSLOCapacity']) + LOG.debug("Received remaining SLO Capacity " + "%(remainingCapacityGb)s GBs for SLO " + "%(SLO)s and workload %(workload)s.", + {'remainingCapacityGb': remainingCapacityGb, + 'SLO': arrayInfo['SLO'], + 'workload': arrayInfo['Workload']}) + except KeyError: + pass + return remainingCapacityGb + + def extend_volume_in_SG( + self, conn, storageConfigService, volumeInstanceName, + volumeName, volumeSize, extraSpecs): + """Extend a volume instance. + + :param conn: connection to the ecom server + :param storageConfigservice: the storage configuration service + :param volumeInstanceName: the volume instance name + :param volumeName: the volume name (String) + :param volumeSize: the volume size + :param extraSpecs: additional info + :returns: volumeDict + :returns: int -- return code + :raises: VolumeBackendAPIException + """ + startTime = time.time() + + rc, job = conn.InvokeMethod( + 'CreateOrModifyElementFromStoragePool', + storageConfigService, TheElement=volumeInstanceName, + Size=self.utils.get_num(volumeSize, '64')) + + LOG.debug("Extend Volume: %(volumename)s. Return code: %(rc)lu.", + {'volumename': volumeName, + 'rc': rc}) + + if rc != 0: + rc, error_desc = self.utils.wait_for_job_complete(conn, job, + extraSpecs) + if rc != 0: + exceptionMessage = (_( + "Error Extend Volume: %(volumeName)s. " + "Return code: %(rc)lu. Error: %(error)s.") + % {'volumeName': volumeName, + 'rc': rc, + 'error': error_desc}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + + LOG.debug("InvokeMethod CreateOrModifyElementFromStoragePool " + "took: %(delta)s H:MM:SS.", + {'delta': self.utils.get_time_delta(startTime, + time.time())}) + + # Find the newly created volume. + volumeDict = self.get_volume_dict_from_job(conn, job['Job']) + return volumeDict, rc diff --git a/emc_vmax_utils.py b/emc_vmax_utils.py index d604322..3349724 100644 --- a/emc_vmax_utils.py +++ b/emc_vmax_utils.py @@ -1,4 +1,4 @@ -# Copyright (c) 2012 - 2014 EMC Corporation. +# Copyright (c) 2012 - 2015 EMC Corporation. # All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); you may @@ -14,17 +14,20 @@ # under the License. import datetime +import hashlib +import os +import pickle import random import re -from xml.dom.minidom import parseString +from xml.dom import minidom +from oslo_log import log as logging +from oslo_service import loopingcall import six from cinder import context from cinder import exception -from cinder.i18n import _, _LE, _LI -from cinder.openstack.common import log as logging -from cinder.openstack.common import loopingcall +from cinder.i18n import _, _LE, _LI, _LW from cinder.volume import volume_types @@ -38,16 +41,24 @@ STORAGEGROUPTYPE = 4 POSTGROUPTYPE = 3 +CLONE_REPLICATION_TYPE = 10 +MAX_POOL_LENGTH = 16 +MAX_FASTPOLICY_LENGTH = 14 EMC_ROOT = 'root/emc' CONCATENATED = 'concatenated' CINDER_EMC_CONFIG_FILE_PREFIX = '/etc/cinder/cinder_emc_config_' CINDER_EMC_CONFIG_FILE_POSTFIX = '.xml' +LIVE_MIGRATION_FILE = '/etc/cinder/livemigrationarray' ISCSI = 'iscsi' FC = 'fc' JOB_RETRIES = 60 INTERVAL_10_SEC = 10 +INTERVAL = 'storagetype:interval' +RETRIES = 'storagetype:retries' CIM_ERR_NOT_FOUND = 6 +VOLUME_ELEMENT_NAME_PREFIX = 'OS-' +SYNCHRONIZED = 4 class EMCVMAXUtils(object): @@ -56,20 +67,24 @@ class EMCVMAXUtils(object): This Utility class is for EMC volume drivers based on SMI-S. It supports VMAX arrays. """ + SLO = 'storagetype:slo' + WORKLOAD = 'storagetype:workload' + POOL = 'storagetype:pool' def __init__(self, prtcl): if not pywbemAvailable: LOG.info(_LI( - 'Module PyWBEM not installed. ' - 'Install PyWBEM using the python-pywbem package.')) + "Module PyWBEM not installed. " + "Install PyWBEM using the python-pywbem package.")) self.protocol = prtcl def find_storage_configuration_service(self, conn, storageSystemName): - """Given the storage system name, get the storage configuration service + """Get storage configuration service with given storage system name. :param conn: connection to the ecom server :param storageSystemName: the storage system name - :returns: foundconfigService + :returns: foundConfigService + :raises: VolumeBackendAPIException """ foundConfigService = None configservices = conn.EnumerateInstanceNames( @@ -78,13 +93,13 @@ def find_storage_configuration_service(self, conn, storageSystemName): if storageSystemName == configservice['SystemName']: foundConfigService = configservice LOG.debug("Found Storage Configuration Service: " - "%(configservice)s", + "%(configservice)s.", {'configservice': configservice}) break if foundConfigService is None: exceptionMessage = (_("Storage Configuration Service not found " - "on %(storageSystemName)s") + "on %(storageSystemName)s.") % {'storageSystemName': storageSystemName}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) @@ -100,6 +115,7 @@ def find_controller_configuration_service(self, conn, storageSystemName): :param conn: connection to the ecom server :param storageSystemName: the storage system name :returns: foundconfigService + :raises: VolumeBackendAPIException """ foundConfigService = None configservices = conn.EnumerateInstanceNames( @@ -108,13 +124,13 @@ def find_controller_configuration_service(self, conn, storageSystemName): if storageSystemName == configservice['SystemName']: foundConfigService = configservice LOG.debug("Found Controller Configuration Service: " - "%(configservice)s", + "%(configservice)s.", {'configservice': configservice}) break if foundConfigService is None: exceptionMessage = (_("Controller Configuration Service not found " - "on %(storageSystemName)s") + "on %(storageSystemName)s.") % {'storageSystemName': storageSystemName}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) @@ -127,6 +143,7 @@ def find_element_composition_service(self, conn, storageSystemName): :param conn: the connection to the ecom server :param storageSystemName: the storage system name :returns: foundElementCompositionService + :raises: VolumeBackendAPIException """ foundElementCompositionService = None elementCompositionServices = conn.EnumerateInstanceNames( @@ -134,14 +151,15 @@ def find_element_composition_service(self, conn, storageSystemName): for elementCompositionService in elementCompositionServices: if storageSystemName == elementCompositionService['SystemName']: foundElementCompositionService = elementCompositionService - LOG.debug("Found Element Composition Service:" - "%(elementCompositionService)s" - % {'elementCompositionService': - elementCompositionService}) + LOG.debug( + "Found Element Composition Service: " + "%(elementCompositionService)s.", { + 'elementCompositionService': + elementCompositionService}) break if foundElementCompositionService is None: exceptionMessage = (_("Element Composition Service not found " - "on %(storageSystemName)s") + "on %(storageSystemName)s.") % {'storageSystemName': storageSystemName}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) @@ -154,6 +172,7 @@ def find_storage_relocation_service(self, conn, storageSystemName): :param conn: the connection to the ecom server :param storageSystemName: the storage system name :returns: foundStorageRelocationService + :raises: VolumeBackendAPIException """ foundStorageRelocationService = None storageRelocationServices = conn.EnumerateInstanceNames( @@ -163,13 +182,13 @@ def find_storage_relocation_service(self, conn, storageSystemName): foundStorageRelocationService = storageRelocationService LOG.debug( "Found Element Composition Service: " - "%(storageRelocationService)s", + "%(storageRelocationService)s.", {'storageRelocationService': storageRelocationService}) break if foundStorageRelocationService is None: exceptionMessage = (_("Storage Relocation Service not found " - "on %(storageSystemName)s") + "on %(storageSystemName)s.") % {'storageSystemName': storageSystemName}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) @@ -182,6 +201,7 @@ def find_storage_hardwareid_service(self, conn, storageSystemName): :param conn: the connection to the ecom server :param storageSystemName: the storage system name :returns: foundStorageRelocationService + :raises: VolumeBackendAPIException """ foundHardwareService = None storageHardwareservices = conn.EnumerateInstanceNames( @@ -190,13 +210,13 @@ def find_storage_hardwareid_service(self, conn, storageSystemName): if storageSystemName == storageHardwareservice['SystemName']: foundHardwareService = storageHardwareservice LOG.debug("Found Storage Hardware ID Management Service:" - "%(storageHardwareservice)s", + "%(storageHardwareservice)s.", {'storageHardwareservice': storageHardwareservice}) break if foundHardwareService is None: exceptionMessage = (_("Storage HardwareId mgmt Service not found " - "on %(storageSystemName)s") + "on %(storageSystemName)s.") % {'storageSystemName': storageSystemName}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) @@ -209,6 +229,7 @@ def find_replication_service(self, conn, storageSystemName): :param conn: the connection to the ecom server :param storageSystemName: the storage system name :returns: foundRepService + :raises: VolumeBackendAPIException """ foundRepService = None repservices = conn.EnumerateInstanceNames( @@ -222,7 +243,7 @@ def find_replication_service(self, conn, storageSystemName): break if foundRepService is None: exceptionMessage = (_("Replication Service not found " - "on %(storageSystemName)s") + "on %(storageSystemName)s.") % {'storageSystemName': storageSystemName}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) @@ -238,7 +259,8 @@ def get_tier_policy_service(self, conn, storageSystemInstanceName): :param conn: the connection information to the ecom server :param storageSystemInstanceName: the storageSystem instance Name :returns: foundTierPolicyService - the tier policy - service instance name + service instance name + :raises: VolumeBackendAPIException """ foundTierPolicyService = None groups = conn.AssociatorNames( @@ -251,49 +273,57 @@ def get_tier_policy_service(self, conn, storageSystemInstanceName): if foundTierPolicyService is None: exceptionMessage = (_( "Tier Policy Service not found " - "for %(storageSystemName)s") + "for %(storageSystemName)s.") % {'storageSystemName': storageSystemInstanceName}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) return foundTierPolicyService - def wait_for_job_complete(self, conn, job): + def wait_for_job_complete(self, conn, job, extraSpecs=None): """Given the job wait for it to complete. :param conn: connection to the ecom server :param job: the job dict - :returns: rc - the return code + :param extraSpecs: the extraSpecs dict. Defaults to None + :returns: int -- the return code :returns: errorDesc - the error description string """ jobInstanceName = job['Job'] - self._wait_for_job_complete(conn, job) + if extraSpecs and (INTERVAL in extraSpecs or RETRIES in extraSpecs): + self._wait_for_job_complete(conn, job, extraSpecs) + else: + self._wait_for_job_complete(conn, job) jobinstance = conn.GetInstance(jobInstanceName, LocalOnly=False) rc = jobinstance['ErrorCode'] errorDesc = jobinstance['ErrorDescription'] - LOG.debug('Return code is: %(rc)lu' - 'Error Description is: %(errorDesc)s', + LOG.debug("Return code is: %(rc)lu. " + "Error Description is: %(errorDesc)s.", {'rc': rc, 'errorDesc': errorDesc}) return rc, errorDesc - def _wait_for_job_complete(self, conn, job): + def _wait_for_job_complete(self, conn, job, extraSpecs=None): """Given the job wait for it to complete. :param conn: connection to the ecom server :param job: the job dict + :param extraSpecs: the extraSpecs dict. Defaults to None + :raises: loopingcall.LoopingCallDone + :raises: VolumeBackendAPIException """ def _wait_for_job_complete(): - """Called at an interval until the job is finished""" + # Called at an interval until the job is finished. + maxJobRetries = self._get_max_job_retries(extraSpecs) retries = kwargs['retries'] wait_for_job_called = kwargs['wait_for_job_called'] if self._is_job_finished(conn, job): raise loopingcall.LoopingCallDone() - if retries > JOB_RETRIES: + if retries > maxJobRetries: LOG.error(_LE("_wait_for_job_complete " "failed after %(retries)d " "tries."), @@ -305,23 +335,49 @@ def _wait_for_job_complete(): if not wait_for_job_called: if self._is_job_finished(conn, job): kwargs['wait_for_job_called'] = True - except Exception as e: - LOG.error(_LE("Exception: %s") % six.text_type(e)) + except Exception: exceptionMessage = (_("Issue encountered waiting for job.")) - LOG.error(exceptionMessage) + LOG.exception(exceptionMessage) raise exception.VolumeBackendAPIException(exceptionMessage) kwargs = {'retries': 0, 'wait_for_job_called': False} + + intervalInSecs = self._get_interval_in_secs(extraSpecs) + timer = loopingcall.FixedIntervalLoopingCall(_wait_for_job_complete) - timer.start(interval=INTERVAL_10_SEC).wait() + timer.start(interval=intervalInSecs).wait() + + def _get_max_job_retries(self, extraSpecs): + """Get max job retries either default or user defined + + :param extraSpecs: extraSpecs dict + :returns: JOB_RETRIES or user defined + """ + if extraSpecs and RETRIES in extraSpecs: + jobRetries = extraSpecs[RETRIES] + else: + jobRetries = JOB_RETRIES + return int(jobRetries) + + def _get_interval_in_secs(self, extraSpecs): + """Get interval in secs, either default or user defined + + :param extraSpecs: extraSpecs dict + :returns: INTERVAL_10_SEC or user defined + """ + if extraSpecs and INTERVAL in extraSpecs: + intervalInSecs = extraSpecs[INTERVAL] + else: + intervalInSecs = INTERVAL_10_SEC + return int(intervalInSecs) def _is_job_finished(self, conn, job): """Check if the job is finished. + :param conn: connection to the ecom server :param job: the job dict - - :returns: True if finished; False if not finished; + :returns: boolean -- True if finished; False if not finished; """ jobInstanceName = job['Job'] @@ -329,73 +385,75 @@ def _is_job_finished(self, conn, job): LocalOnly=False) jobstate = jobinstance['JobState'] # From ValueMap of JobState in CIM_ConcreteJob - # 2L=New, 3L=Starting, 4L=Running, 32767L=Queue Pending + # 2=New, 3=Starting, 4=Running, 32767=Queue Pending # ValueMap("2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13..32767, # 32768..65535"), # Values("New, Starting, Running, Suspended, Shutting Down, # Completed, Terminated, Killed, Exception, Service, # Query Pending, DMTF Reserved, Vendor Reserved")] - # NOTE(deva): string matching based on - # http://ipmitool.cvs.sourceforge.net/ - # viewvc/ipmitool/ipmitool/lib/ipmi_chassis.c - if jobstate in [2L, 3L, 4L, 32767L]: + if jobstate in [2, 3, 4, 32767]: return False else: return True - def wait_for_sync(self, conn, syncName): + def wait_for_sync(self, conn, syncName, extraSpecs=None): """Given the sync name wait for it to fully synchronize. :param conn: connection to the ecom server :param syncName: the syncName + :param extraSpecs: extra specifications + :raises: loopingcall.LoopingCallDone + :raises: VolumeBackendAPIException """ def _wait_for_sync(): - """Called at an interval until the synchronization is finished.""" + """Called at an interval until the synchronization is finished. + + :raises: loopingcall.LoopingCallDone + :raises: VolumeBackendAPIException + """ retries = kwargs['retries'] - wait_for_sync_called = kwargs['wait_for_sync_called'] - if self._is_sync_complete(conn, syncName): - raise loopingcall.LoopingCallDone() - if retries > JOB_RETRIES: - LOG.error(_LE("_wait_for_sync failed after %(retries)d " - "tries."), - {'retries': retries}) - raise loopingcall.LoopingCallDone() try: kwargs['retries'] = retries + 1 - if not wait_for_sync_called: + if not kwargs['wait_for_sync_called']: if self._is_sync_complete(conn, syncName): kwargs['wait_for_sync_called'] = True - except Exception as e: - LOG.error(_LE("Exception: %s") % six.text_type(e)) + except Exception: exceptionMessage = (_("Issue encountered waiting for " "synchronization.")) - LOG.error(exceptionMessage) + LOG.exception(exceptionMessage) raise exception.VolumeBackendAPIException(exceptionMessage) + if kwargs['retries'] > maxJobRetries: + LOG.error(_LE("_wait_for_sync failed after %(retries)d " + "tries."), + {'retries': retries}) + raise loopingcall.LoopingCallDone(retvalue=maxJobRetries) + if kwargs['wait_for_sync_called']: + raise loopingcall.LoopingCallDone() + + maxJobRetries = self._get_max_job_retries(extraSpecs) kwargs = {'retries': 0, 'wait_for_sync_called': False} + intervalInSecs = self._get_interval_in_secs(extraSpecs) timer = loopingcall.FixedIntervalLoopingCall(_wait_for_sync) - timer.start(interval=INTERVAL_10_SEC).wait() + rc = timer.start(interval=intervalInSecs).wait() + return rc def _is_sync_complete(self, conn, syncName): """Check if the job is finished. + :param conn: connection to the ecom server :param syncName: the sync name - :returns: True if fully synchronized; False if not; """ syncInstance = conn.GetInstance(syncName, LocalOnly=False) - percentSynced = syncInstance['PercentSynced'] + copyState = syncInstance['CopyState'] + LOG.debug("CopyState is %(copyState)lu.", + {'copyState': copyState}) - LOG.debug("percent synced is %(percentSynced)lu.", - {'percentSynced': percentSynced}) - - if percentSynced < 100: - return False - else: - return True + return copyState == SYNCHRONIZED def get_num(self, numStr, datatype): """Get the ecom int from the number. @@ -424,9 +482,9 @@ def find_storage_system(self, conn, configService): from it. :param conn: the connection to the ecom server - :param storageConfigService: the storage configuration service - :returns: rc - the return code of the job - :returns: jobDict - the job dict + :param configService: the storage configuration service + :returns: int -- rc - the return code of the job + :returns: dict -- jobDict - the job dict """ foundStorageSystemInstanceName = None groups = conn.AssociatorNames( @@ -436,21 +494,21 @@ def find_storage_system(self, conn, configService): if len(groups) > 0: foundStorageSystemInstanceName = groups[0] else: - LOG.error(_LE("Cannot get storage system")) + LOG.error(_LE("Cannot get storage system.")) raise return foundStorageSystemInstanceName - def get_storage_group_from_volume(self, conn, volumeInstanceName): + def get_storage_group_from_volume(self, conn, volumeInstanceName, sgName): """Returns the storage group for a particular volume. Given the volume instance name get the associated storage group if it - is belong to one + is belong to one. :param conn: connection to the ecom server :param volumeInstanceName: the volume instance name - :returns: foundStorageGroupInstanceName - the storage group - instance name + :param sgName: the storage group name + :returns: foundStorageGroupInstanceName """ foundStorageGroupInstanceName = None @@ -458,14 +516,50 @@ def get_storage_group_from_volume(self, conn, volumeInstanceName): volumeInstanceName, ResultClass='CIM_DeviceMaskingGroup') - if len(storageGroupInstanceNames) > 0: - foundStorageGroupInstanceName = storageGroupInstanceNames[0] + if len(storageGroupInstanceNames) > 1: + LOG.info(_LI( + "The volume belongs to more than one storage group. " + "Returning storage group %(sgName)s."), + {'sgName': sgName}) + for storageGroupInstanceName in storageGroupInstanceNames: + instance = self.get_existing_instance( + conn, storageGroupInstanceName) + if instance and sgName == instance['ElementName']: + foundStorageGroupInstanceName = storageGroupInstanceName + break return foundStorageGroupInstanceName - def wrap_get_storage_group_from_volume(self, conn, volumeInstanceName): + def get_storage_groups_from_volume(self, conn, volumeInstanceName): + """Returns all the storage group for a particular volume. + + Given the volume instance name get all the associated storage groups. + + :param conn: connection to the ecom server + :param volumeInstanceName: the volume instance name + :returns: foundStorageGroupInstanceName + """ + storageGroupInstanceNames = conn.AssociatorNames( + volumeInstanceName, + ResultClass='CIM_DeviceMaskingGroup') + + if storageGroupInstanceNames: + LOG.debug("There are %(len)d storage groups associated " + "with volume %(volumeInstanceName)s.", + {'len': len(storageGroupInstanceNames), + 'volumeInstanceName': volumeInstanceName}) + else: + LOG.debug("There are no storage groups associated " + "with volume %(volumeInstanceName)s.", + {'volumeInstanceName': volumeInstanceName}) + + return storageGroupInstanceNames + + def wrap_get_storage_group_from_volume(self, conn, volumeInstanceName, + sgName): """Unit test aid""" - return self.get_storage_group_from_volume(conn, volumeInstanceName) + return self.get_storage_group_from_volume(conn, volumeInstanceName, + sgName) def find_storage_masking_group(self, conn, controllerConfigService, storageGroupName): @@ -474,7 +568,7 @@ def find_storage_masking_group(self, conn, controllerConfigService, :param conn: connection to the ecom server :param controllerConfigService: the controllerConfigService :param storageGroupName: the name of the storage group you are getting - :param foundStorageGroup: storage group instance name + :returns: foundStorageMaskingGroupInstanceName """ foundStorageMaskingGroupInstanceName = None @@ -482,8 +576,7 @@ def find_storage_masking_group(self, conn, controllerConfigService, conn.Associators(controllerConfigService, ResultClass='CIM_DeviceMaskingGroup')) - for storageMaskingGroupInstance in \ - storageMaskingGroupInstances: + for storageMaskingGroupInstance in storageMaskingGroupInstances: if storageGroupName == storageMaskingGroupInstance['ElementName']: # Check that it has not been deleted recently. @@ -503,7 +596,7 @@ def find_storage_system_name_from_service(self, configService): """Given any service get the storage system name from it. :param configService: the configuration service - :returns: configService['SystemName'] - storage system name (String) + :returns: string -- configService['SystemName'] - storage system name """ return configService['SystemName'] @@ -513,7 +606,7 @@ def find_volume_instance(self, conn, volumeDict, volumeName): :param conn: connection to the ecom server :param volumeDict: the volume Dict :param volumeName: the user friendly name of the volume - :returns: foundVolumeInstance - the volume instance + :returns: foundVolumeInstance - the found volume instance """ volumeInstanceName = self.get_instance_name(volumeDict['classname'], volumeDict['keybindings']) @@ -538,28 +631,24 @@ def get_host_short_name(self, hostName): the full hostName is returned. :param hostName: the fully qualified host name () - :param shortHostName: the short hostName + :returns: string -- the short hostName """ shortHostName = None hostArray = hostName.split('.') - if len(hostArray) > 2: + if len(hostArray) > 1: shortHostName = hostArray[0] else: shortHostName = hostName - return shortHostName + return self.generate_unique_trunc_host(shortHostName) def get_instance_name(self, classname, bindings): """Get the instance from the classname and bindings. - NOTE: This exists in common too...will be moving it to other file - where both common and masking can access it - :param classname: class name for the volume instance :param bindings: volume created from job - :returns: foundVolumeInstance - the volume instance - + :returns: pywbem.CIMInstanceName -- instanceName """ instanceName = None try: @@ -572,216 +661,14 @@ def get_instance_name(self, classname, bindings): return instanceName - def get_ecom_server(self, filename): - """Given the file name get the ecomPort and ecomIP from it. - - :param filename: the path and file name of the emc configuration file - :returns: ecomIp - the ecom IP address - :returns: ecomPort - the ecom port - """ - ecomIp = self._parse_from_file(filename, 'EcomServerIp') - ecomPort = self._parse_from_file(filename, 'EcomServerPort') - if ecomIp is not None and ecomPort is not None: - LOG.debug("Ecom IP: %(ecomIp)s Port: %(ecomPort)s", - {'ecomIp': ecomIp, 'ecomPort': ecomPort}) - return ecomIp, ecomPort - else: - LOG.debug("Ecom server not found.") - return None - - def get_ecom_cred(self, filename): - """Given the filename get the ecomUser and ecomPasswd. - - :param filename: the path and filename of the emc configuration file - :returns: ecomUser - the ecom user - :returns: ecomPasswd - the ecom password - """ - ecomUser = self._parse_from_file(filename, 'EcomUserName') - ecomPasswd = self._parse_from_file(filename, 'EcomPassword') - if ecomUser is not None and ecomPasswd is not None: - return ecomUser, ecomPasswd - else: - LOG.debug("Ecom user not found.") - return None - - def get_ecom_cred_SSL(self, filename): - """Given the filename get the ecomUser and ecomPasswd. - - :param filename: the path and filename of the emc configuration file - :returns: ecomUser - the ecom user - :returns: ecomPasswd - the ecom password - """ - ecomUseSSL = self._parse_from_file(filename, 'EcomUseSSL') - ecomCACert = self._parse_from_file(filename, 'EcomCACert') - ecomNoVerification = self._parse_from_file( - filename, 'EcomNoVerification') - if ecomUseSSL is not None and ecomUseSSL == 'True': - ecomUseSSL = True - if ecomNoVerification is not None and ecomNoVerification == 'True': - ecomNoVerification = True - return ecomUseSSL, ecomCACert, ecomNoVerification - else: - ecomUseSSL = False - ecomNoVerification = False - return ecomUseSSL, ecomCACert, ecomNoVerification - - def parse_file_to_get_port_group_name(self, fileName): - """Parses a file and chooses a port group randomly. - - Given a file, parse it to get all the possible - portGroupElements and choose one randomly. - - :param fileName: the path and name of the file - :returns: portGroupName - the name of the port group chosen - """ - portGroupName = None - myFile = open(fileName, 'r') - data = myFile.read() - myFile.close() - dom = parseString(data) - portGroupElements = dom.getElementsByTagName('PortGroup') - - if portGroupElements is not None and len(portGroupElements) > 0: - portGroupNames = [] - for portGroupElement in portGroupElements: - if portGroupElement.hasChildNodes(): - portGroupName = portGroupElement.childNodes[0].nodeValue - portGroupName = portGroupName.replace('\n', '') - portGroupName = portGroupName.replace('\r', '') - portGroupName = portGroupName.replace('\t', '') - portGroupName = portGroupName.strip() - if portGroupName: - portGroupNames.append(portGroupName) - - LOG.debug("portGroupNames: %(portGroupNames)s", - {'portGroupNames': portGroupNames}) - numPortGroups = len(portGroupNames) - if numPortGroups > 0: - selectedPortGroupName = ( - portGroupNames[random.randint(0, numPortGroups - 1)]) - LOG.debug("Returning Selected Port Group: " - "'%(selectedPortGroupName)s'", - {'selectedPortGroupName': selectedPortGroupName}) - return selectedPortGroupName - - exception_message = (_("No Port Group elements found in config file.")) - LOG.error(exception_message) - raise exception.VolumeBackendAPIException(data=exception_message) - - def _parse_from_file(self, fileName, stringToParse): - """parse the string from XML. - - Remove newlines, tabs and trailing spaces - - :param fileName: the path and name of the file - :returns: retString - the returned string - """ - retString = None - myFile = open(fileName, 'r') - data = myFile.read() - myFile.close() - dom = parseString(data) - tag = dom.getElementsByTagName(stringToParse) - if tag is not None and len(tag) > 0: - strXml = tag[0].toxml() - strXml = strXml.replace('<%s>' % stringToParse, '') - strXml = strXml.replace('\n', '') - strXml = strXml.replace('\r', '') - strXml = strXml.replace('\t', '') - retString = strXml.replace('' % stringToParse, '') - retString = retString.strip() - return retString - - def parse_fast_policy_name_from_file(self, fileName): - """Parse the fast policy name from config file. - - If it is not there, then NON FAST is assumed. - - :param fileName: the path and name of the file - :returns: fastPolicyName - the fast policy name - """ - - fastPolicyName = self._parse_from_file(fileName, 'FastPolicy') - if fastPolicyName: - LOG.debug("File %(fileName)s: Fast Policy is %(fastPolicyName)s", - {'fileName': fileName, - 'fastPolicyName': fastPolicyName}) - return fastPolicyName - else: - LOG.info(_LI("Fast Policy not found.")) - return None - - def parse_array_name_from_file(self, fileName): - """Parse the array name from config file. - - If it is not there then there should only be one array configured to - the ecom. If there is more than one then erroneous results can occur. - - :param fileName: the path and name of the file - :returns: arrayName - the array name - """ - arrayName = self._parse_from_file(fileName, 'Array') - if arrayName: - return arrayName - else: - LOG.debug("Array not found from config file.") - return None - - def parse_pool_name_from_file(self, fileName): - """Parse the pool name from config file. - - If it is not there then we will attempt to get it from extra specs. - - :param fileName: the path and name of the file - :returns: poolName - the pool name - """ - poolName = self._parse_from_file(fileName, 'Pool') - if poolName: - return poolName - else: - LOG.debug("Pool not found from config file.") - return None - - def parse_slo_from_file(self, fileName): - """Parse the slo from config file. - - Please note that the string 'NONE' is returned if it not found. - - :param fileName: the path and name of the file - :returns: slo - the slo or 'NONE' - """ - slo = self._parse_from_file(fileName, 'SLO') - if slo: - return slo - else: - LOG.debug("SLO not in config file. " - "Defaulting to NONE") - return 'NONE' - - def parse_workload_from_file(self, fileName): - """Parse the workload from config file. - - Please note that the string 'NONE' is returned if it not found. - - :param fileName: the path and name of the file - :returns: workload - the workload or 'NONE' - """ - workload = self._parse_from_file(fileName, 'Workload') - if workload: - return workload - else: - LOG.debug("Workload not in config file. " - "Defaulting to NONE") - return 'NONE' - def parse_pool_instance_id(self, poolInstanceId): """Given the instance Id parse the pool name and system name from it. Example of pool InstanceId: Symmetrix+0001233455555+U+Pool 0 :param poolInstanceId: the path and name of the file - :returns: poolName - the pool name - :returns: systemName - the system name + :returns: string -- poolName - the pool name + :returns: string -- systemName - the system name """ poolName = None systemName = None @@ -791,12 +678,25 @@ def parse_pool_instance_id(self, poolInstanceId): idarray = poolInstanceId.split('+') if len(idarray) > 2: - systemName = idarray[0] + '+' + idarray[1] + systemName = self._format_system_name(idarray[0], idarray[1], '+') LOG.debug("Pool name: %(poolName)s System name: %(systemName)s.", {'poolName': poolName, 'systemName': systemName}) return poolName, systemName + def _format_system_name(self, part1, part2, sep): + """Join to make up system name + + :param part1: the prefix + :param sep: the separator + :param part2: the postfix + :returns: systemName + """ + return ("%(part1)s%(sep)s%(part2)s" + % {'part1': part1, + 'sep': sep, + 'part2': part2}) + def parse_pool_instance_id_v3(self, poolInstanceId): """Given the instance Id parse the pool name and system name from it. @@ -814,21 +714,22 @@ def parse_pool_instance_id_v3(self, poolInstanceId): idarray = poolInstanceId.split('-+-') if len(idarray) > 2: - systemName = idarray[0] + '-+-' + idarray[1] + systemName = ( + self._format_system_name(idarray[0], idarray[1], '-+-')) LOG.debug("Pool name: %(poolName)s System name: %(systemName)s.", {'poolName': poolName, 'systemName': systemName}) return poolName, systemName def convert_gb_to_bits(self, strGbSize): - """Convert GB(string) to bits(string). + """Convert GB(string) to bytes(string). :param strGB: string -- The size in GB - :returns: strBitsSize string -- The size in bits + :returns: string -- The size in bytes """ strBitsSize = six.text_type(int(strGbSize) * 1024 * 1024 * 1024) - LOG.debug("Converted %(strGbSize)s GBs to %(strBitsSize)s Bits", + LOG.debug("Converted %(strGbSize)s GBs to %(strBitsSize)s Bits.", {'strGbSize': strGbSize, 'strBitsSize': strBitsSize}) return strBitsSize @@ -838,7 +739,7 @@ def check_if_volume_is_composite(self, conn, volumeInstance): :param conn: the connection information to the ecom server :param volumeInstance: the volume Instance - :returns: 'True', 'False' or 'Undetermined' + :returns: string -- 'True', 'False' or 'Undetermined' """ propertiesList = volumeInstance.properties.items() for properties in propertiesList: @@ -882,7 +783,7 @@ def check_if_volume_is_extendable(self, conn, volumeInstance): :param conn: the connection information to the ecom server :param volumeInstance: the volume instance - :returns: 'True', 'False' or 'Undetermined' + :returns: string -- 'True', 'False' or 'Undetermined' """ isConcatenated = None @@ -919,7 +820,7 @@ def get_composite_type(self, compositeTypeStr): The default is '2' concatenated. :param compositeTypeStr: 'concatenated' or 'striped'. Cannot be None - :returns: compositeType = 2 or 3 + :returns: int -- compositeType = 2 for concatenated, or 3 for striped """ compositeType = 2 stripedStr = 'striped' @@ -927,18 +828,18 @@ def get_composite_type(self, compositeTypeStr): if compositeTypeStr.lower() == stripedStr.lower(): compositeType = 3 except KeyError: - # Default to concatenated if not defined + # Default to concatenated if not defined. pass return compositeType def is_volume_bound_to_pool(self, conn, volumeInstance): - '''Check if volume is bound to a pool. + """Check if volume is bound to a pool. :param conn: the connection information to the ecom server - :param storageServiceInstanceName: the storageSystem instance Name - :returns: foundIsSupportsTieringPolicies - true/false - ''' + :param volumeInstance: the volume instance + :returns: string -- 'True' 'False' or 'Undetermined' + """ propertiesList = volumeInstance.properties.items() for properties in propertiesList: if properties[0] == 'EMCIsBound': @@ -953,12 +854,12 @@ def is_volume_bound_to_pool(self, conn, volumeInstance): return 'Undetermined' def get_space_consumed(self, conn, volumeInstance): - '''Check the space consumed of a volume. + """Check the space consumed of a volume. :param conn: the connection information to the ecom server :param volumeInstance: the volume Instance :returns: spaceConsumed - ''' + """ foundSpaceConsumed = None unitnames = conn.References( volumeInstance, ResultClass='CIM_AllocatedFromStoragePool', @@ -977,14 +878,12 @@ def get_space_consumed(self, conn, volumeInstance): return foundSpaceConsumed def get_volume_size(self, conn, volumeInstance): - '''Get the volume size. - - ConsumableBlocks * BlockSize + """Get the volume size which is ConsumableBlocks * BlockSize. :param conn: the connection information to the ecom server :param volumeInstance: the volume Instance - :returns: volumeSizeOut - ''' + :returns: string -- volumeSizeOut + """ volumeSizeOut = 'Undetermined' numBlocks = 0 blockSize = 0 @@ -1014,12 +913,12 @@ def determine_member_count(self, sizeStr, memberCount, compositeType): :param sizeStr: the size in GBs of the proposed volume :param memberCount: the initial member count :param compositeType: the composite type - :returns: memberCount - string - :returns: errorDesc - the error description + :returns: string -- memberCount + :returns: string -- errorDesc - the error description """ errorDesc = None if compositeType in 'concatenated' and int(sizeStr) > 240: - newMemberCount = int(sizeStr) / 240 + newMemberCount = int(sizeStr) // 240 modular = int(sizeStr) % 240 if modular > 0: newMemberCount += 1 @@ -1039,7 +938,7 @@ def get_extra_specs_by_volume_type_name(self, volumeTypeName): """Gets the extra specs associated with a volume type. Given the string value of the volume type name, get the extra specs - object associated with the volume type + object associated with the volume type. :param volumeTypeName: string value of the volume type name :returns: extra_specs - extra specs object @@ -1054,16 +953,16 @@ def get_pool_capacities(self, conn, poolName, storageSystemName): """Get the total and remaining capacity in GB for a storage pool. Given the storage pool name, get the total capacity and remaining - capacity in GB + capacity in GB. :param conn: connection to the ecom server - :param storagePoolName: string value of the storage pool name - :returns: total_capacity_gb - total capacity of the storage pool in GB - :returns: free_capacity_gb - remaining capacity of the - storage pool in GB + :param poolName: string value of the storage pool name + :param storageSystemName: the storage system name + :returns: tuple -- (total_capacity_gb, free_capacity_gb, + provisioned_capacity_gb) """ LOG.debug( - "Retrieving capacity for pool %(poolName)s on array %(array)s", + "Retrieving capacity for pool %(poolName)s on array %(array)s.", {'poolName': poolName, 'array': storageSystemName}) @@ -1072,17 +971,24 @@ def get_pool_capacities(self, conn, poolName, storageSystemName): if poolInstanceName is None: LOG.error(_LE( "Unable to retrieve pool instance of %(poolName)s on " - "array %(array)s"), {'poolName': poolName, - 'array': storageSystemName}) + "array %(array)s."), + {'poolName': poolName, 'array': storageSystemName}) return (0, 0) storagePoolInstance = conn.GetInstance( poolInstanceName, LocalOnly=False) total_capacity_gb = self.convert_bits_to_gbs( storagePoolInstance['TotalManagedSpace']) - allocated_capacity_gb = self.convert_bits_to_gbs( + provisioned_capacity_gb = self.convert_bits_to_gbs( storagePoolInstance['EMCSubscribedCapacity']) - free_capacity_gb = total_capacity_gb - allocated_capacity_gb - return (total_capacity_gb, free_capacity_gb) + free_capacity_gb = self.convert_bits_to_gbs( + storagePoolInstance['RemainingManagedSpace']) + try: + array_max_over_subscription = self.get_ratio_from_max_sub_per( + storagePoolInstance['EMCMaxSubscriptionPercent']) + except KeyError: + array_max_over_subscription = 65534 + return (total_capacity_gb, free_capacity_gb, + provisioned_capacity_gb, array_max_over_subscription) def get_pool_by_name(self, conn, storagePoolName, storageSystemName): """Returns the instance name associated with a storage pool name. @@ -1090,19 +996,20 @@ def get_pool_by_name(self, conn, storagePoolName, storageSystemName): :param conn: connection to the ecom server :param storagePoolName: string value of the storage pool name :param storageSystemName: string value of array - :returns: poolInstanceName - instance name of storage pool + :returns: foundPoolInstanceName - instance name of storage pool """ foundPoolInstanceName = None LOG.debug( - "storagePoolName: %(poolName)s, storageSystemName: %(array)s", + "storagePoolName: %(poolName)s, storageSystemName: %(array)s.", {'poolName': storagePoolName, 'array': storageSystemName}) - poolInstanceNames = conn.EnumerateInstanceNames( - 'EMC_VirtualProvisioningPool') + storageSystemInstanceName = self.find_storageSystem(conn, + storageSystemName) + poolInstanceNames = conn.AssociatorNames( + storageSystemInstanceName, + ResultClass='EMC_VirtualProvisioningPool') for poolInstanceName in poolInstanceNames: - poolName, systemName = ( - self.parse_pool_instance_id(poolInstanceName['InstanceID'])) - if (poolName == storagePoolName and - storageSystemName in systemName): + poolName = self.get_pool_name(conn, poolInstanceName) + if (poolName == storagePoolName): # Check that the pool hasn't been recently deleted. instance = self.get_existing_instance(conn, poolInstanceName) if instance is None: @@ -1114,12 +1021,12 @@ def get_pool_by_name(self, conn, storagePoolName, storageSystemName): return foundPoolInstanceName def convert_bits_to_gbs(self, strBitSize): - """Convert Bits(string) to GB(string). + """Convert bytes(string) to GB(string). - :param strBitSize: string -- The size in bits - :returns: gbSize string -- The size in GB + :param strBitSize: string -- The size in bytes + :returns: int -- The size in GB """ - gbSize = int(strBitSize) / 1024 / 1024 / 1024 + gbSize = int(strBitSize) // 1024 // 1024 // 1024 return gbSize def compare_size(self, size1Str, size2Str): @@ -1127,7 +1034,7 @@ def compare_size(self, size1Str, size2Str): :param size1Str: the first bit size (String) :param size2Str: the second bit size (String) - :returns: size1GBs - size2GBs (int) + :returns: int -- size1GBs - size2GBs """ size1GBs = self.convert_bits_to_gbs(size1Str) size2GBs = self.convert_bits_to_gbs(size2Str) @@ -1138,7 +1045,8 @@ def get_volumetype_extraspecs(self, volume, volumeTypeId=None): """Compare the bit sizes to an approximate. :param volume: the volume dictionary - :returns: extraSpecs - the extra specs + :param volumeTypeId: Optional override for volume['volume_type_id'] + :returns: dict -- extraSpecs - the extra specs """ extraSpecs = {} @@ -1159,7 +1067,7 @@ def get_volume_type_name(self, volume): """Get the volume type name. :param volume: the volume dictionary - :returns: volumeTypeName - the volume type name + :returns: string -- volumeTypeName - the volume type name """ volumeTypeName = None @@ -1186,23 +1094,23 @@ def parse_volume_type_from_filename(self, emcConfigFile): return volumeTypeName def get_volumes_from_pool(self, conn, poolInstanceName): - '''Check the space consumed of a volume. + """Check the space consumed of a volume. :param conn: the connection information to the ecom server - :param volumeInstance: the volume Instance - :returns: spaceConsumed - ''' + :param poolInstanceName: the pool instance name + :returns: the volumes in the pool + """ return conn.AssociatorNames( poolInstanceName, AssocClass='CIM_AllocatedFromStoragePool', ResultClass='CIM_StorageVolume') def check_is_volume_bound_to_pool(self, conn, volumeInstance): - '''Check the space consumed of a volume. + """Check the space consumed of a volume. :param conn: the connection information to the ecom server :param volumeInstance: the volume Instance - :returns: spaceConsumed - ''' + :returns: string -- 'True', 'False' or 'Undetermined' + """ foundSpaceConsumed = None unitnames = conn.References( volumeInstance, ResultClass='CIM_AllocatedFromStoragePool', @@ -1225,11 +1133,11 @@ def check_is_volume_bound_to_pool(self, conn, volumeInstance): return 'Undetermined' def get_short_protocol_type(self, protocol): - '''Given the protocol type, return I for iscsi and F for fc + """Given the protocol type, return I for iscsi and F for fc. :param protocol: iscsi or fc - :returns: 'I' or 'F' - ''' + :returns: string -- 'I' for iscsi or 'F' for fc + """ if protocol.lower() == ISCSI.lower(): return 'I' elif protocol.lower() == FC.lower(): @@ -1244,7 +1152,7 @@ def get_hardware_id_instances_from_array( :param conn: connection to the ecom server :param: hardwareIdManagementService - hardware id management service :returns: hardwareIdInstances - the list of hardware - id instances + id instances """ hardwareIdInstances = ( conn.Associators(hardwareIdManagementService, @@ -1257,11 +1165,11 @@ def truncate_string(self, strToTruncate, maxNum): :param strToTruncate: the string to be truncated :param maxNum: the maximum number of characters - :returns: truncated string or original string + :returns: string -- truncated string or original string """ if len(strToTruncate) > maxNum: - newNum = len(strToTruncate) - maxNum / 2 - firstChars = strToTruncate[:maxNum / 2] + newNum = len(strToTruncate) - maxNum // 2 + firstChars = strToTruncate[:maxNum // 2] lastChars = strToTruncate[newNum:] strToTruncate = firstChars + lastChars @@ -1271,8 +1179,7 @@ def get_array(self, host): """Extract the array from the host capabilites. :param host: the host object - :param maxNum: the maximum number of characters - :returns: truncated string or original string + :returns: storageSystem - storage system represents the array """ try: @@ -1280,7 +1187,7 @@ def get_array(self, host): infoDetail = host.split('@') storageSystem = 'SYMMETRIX+' + infoDetail[0] except Exception: - LOG.error(_LE("Error parsing array from host capabilities. ")) + LOG.error(_LE("Error parsing array from host capabilities.")) return storageSystem @@ -1289,19 +1196,22 @@ def get_time_delta(self, startTime, endTime): :param startTime: the start time :param endTime: the end time - :returns: delta in string H:MM:SS + :returns: string -- delta in string H:MM:SS """ delta = endTime - startTime - return str(datetime.timedelta(seconds=int(delta))) + return six.text_type(datetime.timedelta(seconds=int(delta))) def find_sync_sv_by_target( - self, conn, storageSystem, target, waitforsync=True): + self, conn, storageSystem, target, extraSpecs, + waitforsync=True): """Find the storage synchronized name by target device ID. :param conn: connection to the ecom server :param storageSystem: the storage system name :param target: target volume object - :returns: foundSyncName (String) + :param extraSpecs: the extraSpecs dict + :param waitforsync: wait for the synchronization to complete if True + :returns: foundSyncInstanceName """ foundSyncInstanceName = None syncInstanceNames = conn.EnumerateInstanceNames( @@ -1311,36 +1221,38 @@ def find_sync_sv_by_target( if storageSystem != syncSvTarget['SystemName']: continue if syncSvTarget['DeviceID'] == target['DeviceID']: - # check that it hasn't recently been deleted + # Check that it hasn't recently been deleted. try: conn.GetInstance(syncInstanceName) foundSyncInstanceName = syncInstanceName LOG.debug("Found sync Name: " - "%(syncName)s", + "%(syncName)s.", {'syncName': foundSyncInstanceName}) except Exception: foundSyncInstanceName = None break if foundSyncInstanceName is None: - LOG.info(_LI( + LOG.warning(_LW( "Storage sync name not found for target %(target)s " - "on %(storageSystem)s"), + "on %(storageSystem)s."), {'target': target['DeviceID'], 'storageSystem': storageSystem}) else: - # Wait for SE_StorageSynchronized_SV_SV to be fully synced + # Wait for SE_StorageSynchronized_SV_SV to be fully synced. if waitforsync: - self.wait_for_sync(conn, foundSyncInstanceName) + self.wait_for_sync(conn, foundSyncInstanceName, extraSpecs) return foundSyncInstanceName def find_group_sync_rg_by_target( - self, conn, storageSystem, targetRgInstanceName, waitforsync=True): + self, conn, storageSystem, targetRgInstanceName, extraSpecs, + waitforsync=True): """Find the SE_GroupSynchronized_RG_RG instance name by target group. :param conn: connection to the ecom server :param storageSystem: the storage system name :param targetRgInstanceName: target group instance name + :param extraSpecs: the extraSpecs dict :param waitforsync: wait for synchronization to complete :returns: foundSyncInstanceName """ @@ -1350,58 +1262,36 @@ def find_group_sync_rg_by_target( for rgInstanceName in groupSyncRgInstanceNames: rgTarget = rgInstanceName['SyncedElement'] if targetRgInstanceName['InstanceID'] == rgTarget['InstanceID']: - # check that it has not recently been deleted + # Check that it has not recently been deleted. try: conn.GetInstance(rgInstanceName) foundSyncInstanceName = rgInstanceName LOG.debug("Found group sync name: " - "%(syncName)s", + "%(syncName)s.", {'syncName': foundSyncInstanceName}) except Exception: foundSyncInstanceName = None break if foundSyncInstanceName is None: - LOG.info(_LI( + LOG.warning(_LW( "Group sync name not found for target group %(target)s " - "on %(storageSystem)s"), + "on %(storageSystem)s."), {'target': targetRgInstanceName['InstanceID'], 'storageSystem': storageSystem}) else: - # Wait for SE_StorageSynchronized_SV_SV to be fully synced + # Wait for SE_StorageSynchronized_SV_SV to be fully synced. if waitforsync: - self.wait_for_sync(conn, foundSyncInstanceName) + self.wait_for_sync(conn, foundSyncInstanceName, extraSpecs) return foundSyncInstanceName - def populate_cgsnapshot_status( - self, context, db, cgsnapshot_id, status='available'): - """Update cgsnapshot status in the cinder database. - - :param context: - :param db: cinder database - :param cgsnapshot_id: cgsnapshot id - :param status: string value reflects the status of the member snapshot - :return snapshots: updated snapshots - """ - snapshots = db.snapshot_get_all_for_cgsnapshot(context, cgsnapshot_id) - LOG.info(_LI( - "Populating status for cgsnapshot: %(id)s "), - {'id': cgsnapshot_id}) - if snapshots: - for snapshot in snapshots: - snapshot['status'] = status - else: - LOG.info(_LI("No snapshot found for %(cgsnapshot)"), - {'cgsnapshot': cgsnapshot_id}) - return snapshots - def get_firmware_version(self, conn, arrayName): """Get the firmware version of array. :param conn: the connection to the ecom server :param arrayName: the array name - :returns: firmwareVersion (String) + :returns: string -- firmwareVersion """ firmwareVersion = None softwareIdentities = conn.EnumerateInstanceNames( @@ -1419,50 +1309,8 @@ def get_firmware_version(self, conn, arrayName): return firmwareVersion - def get_srp_pool_stats(self, conn, arrayName, poolName): - """Get the totalManagedSpace, remainingManagedSpace. - - :param conn: the connection to the ecom server - :param arrayName: the array name - :param poolName: the pool name - :returns: totalManagedSpace, remainingManagedSpace - """ - totalCapacityGb = -1 - remainingCapacityGb = -1 - storageSystemInstanceName = self.find_storageSystem(conn, arrayName) - - srpPoolInstanceNames = conn.AssociatorNames( - storageSystemInstanceName, - ResultClass='Symm_SRPStoragePool') - - for srpPoolInstanceName in srpPoolInstanceNames: - poolInstanceID = srpPoolInstanceName['InstanceID'] - poolnameStr, systemNameStr = ( # @UnusedVariable - self.parse_pool_instance_id_v3(poolInstanceID)) - - if six.text_type(poolName) == six.text_type(poolnameStr): - try: - # check that pool hasnt suddently been deleted - srpPoolInstance = conn.GetInstance(srpPoolInstanceName) - propertiesList = srpPoolInstance.properties.items() - for properties in propertiesList: - if properties[0] == 'TotalManagedSpace': - cimProperties = properties[1] - totalManagedSpace = cimProperties.value - totalCapacityGb = self.convert_bits_to_gbs( - totalManagedSpace) - elif properties[0] == 'RemainingManagedSpace': - cimProperties = properties[1] - remainingManagedSpace = cimProperties.value - remainingCapacityGb = self.convert_bits_to_gbs( - remainingManagedSpace) - except Exception: - pass - - return totalCapacityGb, remainingCapacityGb - def isArrayV3(self, conn, arrayName): - """Check is the array is V2 or V3. + """Check if the array is V2 or V3. :param conn: the connection to the ecom server :param arrayName: the array name @@ -1485,29 +1333,16 @@ def get_pool_and_system_name_v2( :param conn: the connection to the ecom server :param storageSystemInstanceName: the storage system instance name :param poolNameInStr: the pool name - :returns: foundPoolInstanceName, systemNameStr + :returns: foundPoolInstanceName + :returns: string -- systemNameStr """ - foundPoolInstanceName = None vpoolInstanceNames = conn.AssociatorNames( storageSystemInstanceName, ResultClass='EMC_VirtualProvisioningPool') - for vpoolInstanceName in vpoolInstanceNames: - poolInstanceId = vpoolInstanceName['InstanceID'] - # Example: SYMMETRIX+000195900551+TP+Sol_Innov - poolnameStr, systemNameStr = self.parse_pool_instance_id( - poolInstanceId) - if poolnameStr is not None and systemNameStr is not None: - if six.text_type(poolNameInStr) == six.text_type(poolnameStr): - # check that the pool hasnt recently been deleted - try: - conn.GetInstance(vpoolInstanceName) - foundPoolInstanceName = vpoolInstanceName - except Exception: - foundPoolInstanceName = None - break - - return foundPoolInstanceName, systemNameStr + return self._get_pool_instance_and_system_name( + conn, vpoolInstanceNames, storageSystemInstanceName, + poolNameInStr) def get_pool_and_system_name_v3( self, conn, storageSystemInstanceName, poolNameInStr): @@ -1516,34 +1351,67 @@ def get_pool_and_system_name_v3( :param conn: the connection to the ecom server :param storageSystemInstanceName: the storage system instance name :param poolNameInStr: the pool name - :returns: foundPoolInstanceName, systemNameStr + :returns: foundPoolInstanceName + :returns: string -- systemNameStr """ - foundPoolInstanceName = None srpPoolInstanceNames = conn.AssociatorNames( storageSystemInstanceName, ResultClass='Symm_SRPStoragePool') - for srpPoolInstanceName in srpPoolInstanceNames: - poolInstanceID = srpPoolInstanceName['InstanceID'] + return self._get_pool_instance_and_system_name( + conn, srpPoolInstanceNames, storageSystemInstanceName, + poolNameInStr) + + def _get_pool_instance_and_system_name( + self, conn, poolInstanceNames, storageSystemInstanceName, + poolname): + """Get the pool instance and the system name + + :param conn: the ecom connection + :param poolInstanceNames: list of pool instances + :param poolname: pool name (string) + :returns: foundPoolInstanceName, systemname + """ + foundPoolInstanceName = None + poolnameStr = None + systemNameStr = storageSystemInstanceName['Name'] + for poolInstanceName in poolInstanceNames: # Example: SYMMETRIX-+-000196700535-+-SR-+-SRP_1 - poolnameStr, systemNameStr = self.parse_pool_instance_id_v3( - poolInstanceID) - if poolnameStr is not None and systemNameStr is not None: - if six.text_type(poolNameInStr) == six.text_type(poolnameStr): + # Example: SYMMETRIX+000195900551+TP+Sol_Innov + poolnameStr = self.get_pool_name(conn, poolInstanceName) + if poolnameStr is not None: + if six.text_type(poolname) == six.text_type(poolnameStr): try: - conn.GetInstance(srpPoolInstanceName) - foundPoolInstanceName = srpPoolInstanceName + conn.GetInstance(poolInstanceName) + foundPoolInstanceName = poolInstanceName except Exception: foundPoolInstanceName = None break return foundPoolInstanceName, systemNameStr + def get_pool_name(self, conn, poolInstanceName): + """Get the pool name from the instance + + :param conn: the ecom connection + :param poolInstanceName: the pool instance + :returns: poolnameStr + """ + poolnameStr = None + try: + poolInstance = conn.GetInstance(poolInstanceName) + poolnameStr = poolInstance['ElementName'] + except Exception: + pass + return poolnameStr + def find_storageSystem(self, conn, arrayStr): - """Find an array instance name given the array name. + """Find an array instance name by the array name. + :param conn: the ecom connection :param arrayStr: the array Serial number (string) :returns: foundPoolInstanceName, the CIM Instance Name of the Pool + :raises: VolumeBackendAPIException """ foundStorageSystemInstanceName = None storageSystemInstanceNames = conn.EnumerateInstanceNames( @@ -1560,7 +1428,7 @@ def find_storageSystem(self, conn, arrayStr): LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException(data=exceptionMessage) - LOG.debug("Array Found: %(array)s..", + LOG.debug("Array Found: %(array)s.", {'array': arrayStr}) return foundStorageSystemInstanceName @@ -1571,7 +1439,7 @@ def is_in_range(self, volumeSize, maximumVolumeSize, minimumVolumeSize): :param volumeSize: volume size :param maximumVolumeSize: the max volume size :param minimumVolumeSize: the min volume size - :returns: true/false + :returns: boolean """ if (long(volumeSize) < long(maximumVolumeSize)) and ( @@ -1585,7 +1453,7 @@ def verify_slo_workload(self, slo, workload): :param slo: Service Level Object e.g bronze :param workload: workload e.g DSS - :returns: true/false + :returns: boolean """ isValidSLO = False isValidWorkload = False @@ -1626,20 +1494,14 @@ def get_v3_storage_group_name(self, poolName, slo, workload): :param workload: the workload string e.g DSS :returns: storageGroupName """ - return 'OS-' + poolName + '-' + slo + '-' + workload + '-SG' - - def strip_short_host_name(self, storageGroupName): - tempList = storageGroupName.split("-") - if len(tempList) == 6: - shorthostName = tempList.pop(1) - updatedStorageGroup = "-".join(tempList) - return updatedStorageGroup, shorthostName - else: - shorthostName = None - return storageGroupName, shorthostName + storageGroupName = ("OS-%(poolName)s-%(slo)s-%(workload)s-SG" + % {'poolName': poolName, + 'slo': slo, + 'workload': workload}) + return storageGroupName def _get_fast_settings_from_storage_group(self, storageGroupInstance): - """get the emc FAST setting from the storage group. + """Get the emc FAST setting from the storage group. :param storageGroupInstance: the storage group instance :returns: emcFastSetting @@ -1656,6 +1518,7 @@ def _get_fast_settings_from_storage_group(self, storageGroupInstance): def get_volume_meta_head(self, conn, volumeInstanceName): """Get the head of a meta volume. + :param conn: the ecom connection :param volumeInstanceName: the composite volume instance name :returns: the instance name of the meta volume head """ @@ -1668,7 +1531,7 @@ def get_volume_meta_head(self, conn, volumeInstanceName): metaHeadInstanceName = metaHeads[0] if metaHeadInstanceName is None: LOG.info(_LI( - "Volume %(volume)s does not have meta device members: "), + "Volume %(volume)s does not have meta device members."), {'volume': volumeInstanceName}) return metaHeadInstanceName @@ -1677,6 +1540,7 @@ def get_meta_members_of_composite_volume( self, conn, metaHeadInstanceName): """Get the member volumes of a composite volume. + :param conn: the ecom connection :param metaHeadInstanceName: head of the composite volume :returns: an array containing instance names of member volumes """ @@ -1684,26 +1548,34 @@ def get_meta_members_of_composite_volume( metaHeadInstanceName, AssocClass='CIM_BasedOn', ResultClass='EMC_PartialAllocOfConcreteExtent') - LOG.debug("metaMembers: %(members)s ", {'members': metaMembers}) + LOG.debug("metaMembers: %(members)s.", {'members': metaMembers}) return metaMembers - def get_meta_members_capacity_in_bit(self, conn, volumeInstanceNames): - """Get the capacity in bits of all meta device member volumes. + def get_meta_members_capacity_in_byte(self, conn, volumeInstanceNames): + """Get the capacity in byte of all meta device member volumes. + :param conn: the ecom connection :param volumeInstanceNames: array contains meta device member volumes :returns: array contains capacities of each member device in bits """ - capacitiesInBit = [] + capacitiesInByte = [] + headVolume = conn.GetInstance(volumeInstanceNames[0]) + totalSizeInByte = ( + headVolume['ConsumableBlocks'] * headVolume['BlockSize']) + volumeInstanceNames.pop(0) for volumeInstanceName in volumeInstanceNames: volumeInstance = conn.GetInstance(volumeInstanceName) numOfBlocks = volumeInstance['ConsumableBlocks'] blockSize = volumeInstance['BlockSize'] - volumeSizeInbits = numOfBlocks * blockSize - capacitiesInBit.append(volumeSizeInbits) - return capacitiesInBit + volumeSizeInByte = numOfBlocks * blockSize + capacitiesInByte.append(volumeSizeInByte) + totalSizeInByte = totalSizeInByte - volumeSizeInByte + + capacitiesInByte.insert(0, totalSizeInByte) + return capacitiesInByte def get_existing_instance(self, conn, instanceName): - """Check that the instance name still exists. + """Check that the instance name still exists and return the instance. :param conn: the connection to the ecom server :param instanceName: the instanceName to be checked @@ -1727,16 +1599,1320 @@ def process_exception_args(self, arg, instanceName): instance = None code, desc = arg[0], arg[1] if code == CIM_ERR_NOT_FOUND: - # Object doesn't exist any more + # Object doesn't exist any more. instance = None else: - # Something else that we cannot recover from has happened + # Something else that we cannot recover from has happened. LOG.error(_LE("Exception: %s"), six.text_type(desc)) exceptionMessage = (_( - "Cannot verify the existance of object:" + "Cannot verify the existence of object:" "%(instanceName)s.") % {'instanceName': instanceName}) LOG.error(exceptionMessage) raise exception.VolumeBackendAPIException( data=exceptionMessage) return instance + + def find_replication_service_capabilities(self, conn, storageSystemName): + """Find the replication service capabilities instance name. + + :param conn: the connection to the ecom server + :param storageSystemName: the storage system name + :returns: foundRepServCapability + """ + foundRepServCapability = None + repservices = conn.EnumerateInstanceNames( + 'CIM_ReplicationServiceCapabilities') + for repservCap in repservices: + if storageSystemName in repservCap['InstanceID']: + foundRepServCapability = repservCap + LOG.debug("Found Replication Service Capabilities: " + "%(repservCap)s", + {'repservCap': repservCap}) + break + if foundRepServCapability is None: + exceptionMessage = (_("Replication Service Capability not found " + "on %(storageSystemName)s.") + % {'storageSystemName': storageSystemName}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException(data=exceptionMessage) + + return foundRepServCapability + + def is_clone_licensed(self, conn, capabilityInstanceName): + """Check if the clone feature is licensed and enabled. + + :param conn: the connection to the ecom server + :param capabilityInstanceName: the replication service capabilities + instance name + :returns: True if licensed and enabled; False otherwise. + """ + capabilityInstance = conn.GetInstance(capabilityInstanceName) + propertiesList = capabilityInstance.properties.items() + for properties in propertiesList: + if properties[0] == 'SupportedReplicationTypes': + cimProperties = properties[1] + repTypes = cimProperties.value + LOG.debug("Found supported replication types: " + "%(repTypes)s", + {'repTypes': repTypes}) + if CLONE_REPLICATION_TYPE in repTypes: + # Clone is a supported replication type. + LOG.debug("Clone is licensed and enabled.") + return True + return False + + def create_storage_hardwareId_instance_name( + self, conn, hardwareIdManagementService, initiator): + """Create storage hardware ID instance name based on the WWPN/IQN. + + :param conn: connection to the ecom server + :param hardwareIdManagementService: the hardware ID management service + :param initiator: initiator(IQN or WWPN) to create the hardware ID + instance + :returns: hardwareIdList + """ + hardwareIdList = None + hardwareIdType = self._get_hardware_type(initiator) + rc, ret = conn.InvokeMethod( + 'CreateStorageHardwareID', + hardwareIdManagementService, + StorageID=initiator, + IDType=self.get_num(hardwareIdType, '16')) + + if 'HardwareID' in ret: + LOG.debug("Created hardware ID instance for initiator:" + "%(initiator)s rc=%(rc)d, ret=%(ret)s", + {'initiator': initiator, 'rc': rc, 'ret': ret}) + hardwareIdList = ret['HardwareID'] + else: + LOG.warning(_LW("CreateStorageHardwareID failed. initiator: " + "%(initiator)s, rc=%(rc)d, ret=%(ret)s."), + {'initiator': initiator, 'rc': rc, 'ret': ret}) + return hardwareIdList + + def _get_hardware_type( + self, initiator): + """Determine the hardware type based on the initiator. + + :param initiator: initiator(IQN or WWPN) + :returns: hardwareTypeId + """ + hardwareTypeId = 0 + try: + int(initiator, 16) + hardwareTypeId = 2 + except Exception: + if 'iqn' in initiator.lower(): + hardwareTypeId = 5 + if hardwareTypeId == 0: + LOG.warning(_LW("Cannot determine the hardware type.")) + return hardwareTypeId + + def _process_tag(self, element, tagName): + """Process the tag to get the value. + + :param element: the parent element + :param tagName: the tag name + :returns: nodeValue(can be None) + """ + nodeValue = None + try: + processedElement = element.getElementsByTagName(tagName)[0] + nodeValue = processedElement.childNodes[0].nodeValue + if nodeValue: + nodeValue = nodeValue.strip() + except IndexError: + pass + return nodeValue + + def _get_connection_info(self, ecomElement): + """Given the filename get the ecomUser and ecomPasswd. + + :param ecomElement: the ecom element + :returns: dict -- connargs - the connection info dictionary + :raises: VolumeBackendAPIException + """ + connargs = {} + connargs['EcomServerIp'] = ( + self._process_tag(ecomElement, 'EcomServerIp')) + connargs['EcomServerPort'] = ( + self._process_tag(ecomElement, 'EcomServerPort')) + connargs['EcomUserName'] = ( + self._process_tag(ecomElement, 'EcomUserName')) + connargs['EcomPassword'] = ( + self._process_tag(ecomElement, 'EcomPassword')) + + for k, __ in connargs.items(): + if connargs[k] is None: + exceptionMessage = (_( + "EcomServerIp, EcomServerPort, EcomUserName, " + "EcomPassword must have valid values.")) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + + # These can be None + connargs['EcomUseSSL'] = self._process_tag(ecomElement, 'EcomUseSSL') + connargs['EcomCACert'] = self._process_tag(ecomElement, 'EcomCACert') + connargs['EcomNoVerification'] = ( + self._process_tag(ecomElement, 'EcomNoVerification')) + + if connargs['EcomUseSSL'] and connargs['EcomUseSSL'] == 'True': + connargs['EcomUseSSL'] = True + if connargs['EcomNoVerification'] and ( + connargs['EcomNoVerification'] == 'True'): + connargs['EcomNoVerification'] = True + else: + connargs['EcomUseSSL'] = False + connargs['EcomNoVerification'] = False + + return connargs + + def _fill_record(self, connargs, serialNumber, poolName, + portGroup, element): + """Fill a single record. + + :param connargs: the connection info + :param serialNumber: the serial number of array + :param poolName: the poolname + :param portGroup: the portGroup + :param element: the parent element + :returns: dict -- kwargs + """ + kwargs = {} + kwargs['EcomServerIp'] = connargs['EcomServerIp'] + kwargs['EcomServerPort'] = connargs['EcomServerPort'] + kwargs['EcomUserName'] = connargs['EcomUserName'] + kwargs['EcomPassword'] = connargs['EcomPassword'] + kwargs['EcomUseSSL'] = connargs['EcomUseSSL'] + kwargs['EcomCACert'] = connargs['EcomCACert'] + kwargs['EcomNoVerification'] = connargs['EcomNoVerification'] + + slo = self._process_tag(element, 'SLO') + if slo is None: + slo = 'NONE' + kwargs['SLO'] = slo + workload = self._process_tag(element, 'Workload') + if workload is None: + workload = 'NONE' + kwargs['Workload'] = workload + fastPolicy = self._process_tag(element, 'FastPolicy') + kwargs['FastPolicy'] = fastPolicy + kwargs['SerialNumber'] = serialNumber + kwargs['PoolName'] = poolName + kwargs['PortGroup'] = portGroup + + return kwargs + + def _multi_pool_support(self, fileName): + """Multi pool support. + + + + + 10.108.246.202 + ... + + + 000198700439 + ... + + + FC_SLVR1 + ... + + + + + + + + + :param fileName: the configuration file + :returns: list + """ + myList = [] + connargs = {} + myFile = open(fileName, 'r') + data = myFile.read() + myFile.close() + dom = minidom.parseString(data) + interval = self._process_tag(dom, 'Interval') + retries = self._process_tag(dom, 'Retries') + try: + ecomElements = dom.getElementsByTagName('EcomServer') + if ecomElements and len(ecomElements) > 0: + for ecomElement in ecomElements: + connargs = self._get_connection_info(ecomElement) + arrayElements = ecomElement.getElementsByTagName('Array') + if arrayElements and len(arrayElements) > 0: + for arrayElement in arrayElements: + myList = self._get_pool_info(arrayElement, + fileName, connargs, + interval, retries, + myList) + else: + LOG.error(_LE( + "Please check your xml for format or syntax " + "errors. Please see documentation for more " + "details.")) + except IndexError: + pass + return myList + + def _single_pool_support(self, fileName): + """Single pool support. + + + 10.108.246.202 + 5988 + admin + #1Password + + OS-PORTGROUP1-PG + + 000198700439 + FC_SLVR1 + + + :param fileName: the configuration file + :returns: list + """ + myList = [] + kwargs = {} + connargs = {} + myFile = open(fileName, 'r') + data = myFile.read() + myFile.close() + dom = minidom.parseString(data) + try: + connargs = self._get_connection_info(dom) + interval = self._process_tag(dom, 'Interval') + retries = self._process_tag(dom, 'Retries') + portGroup = self._get_random_portgroup(dom) + + serialNumber = self._process_tag(dom, 'Array') + if serialNumber is None: + LOG.error(_LE( + "Array Serial Number must be in the file " + "%(fileName)s."), + {'fileName': fileName}) + poolName = self._process_tag(dom, 'Pool') + if poolName is None: + LOG.error(_LE( + "PoolName must be in the file " + "%(fileName)s."), + {'fileName': fileName}) + kwargs = self._fill_record( + connargs, serialNumber, poolName, portGroup, dom) + if interval: + kwargs['Interval'] = interval + if retries: + kwargs['Retries'] = retries + + myList.append(kwargs) + except IndexError: + pass + return myList + + def parse_file_to_get_array_map(self, fileName): + """Parses a file and gets array map. + + Given a file, parse it to get array and any pool(s) or + fast policy(s), SLOs, Workloads that might exist. + + :param fileName: the path and name of the file + :returns: list + """ + # Multi-pool support. + myList = self._multi_pool_support(fileName) + if len(myList) == 0: + myList = self._single_pool_support(fileName) + + return myList + + def extract_record(self, arrayInfo, pool): + """Given pool string determine the correct record. + + The poolName and the serialNumber will determine the + correct record to return in VMAX2. + The poolName, SLO and the serialNumber will determine the + correct record to return in VMAX3. + + :param arrayInfo: list of records + :param pool: e.g 'SATA_BRONZE1+000198700439' + 'SRP_1+Bronze+000198700555' + :returns: single record + """ + foundArrayInfoRec = {} + if pool: + for arrayInfoRec in arrayInfo: + if pool.count('+') == 2: + compString = ("%(slo)s+%(poolName)s+%(array)s" + % {'slo': arrayInfoRec['SLO'], + 'poolName': arrayInfoRec['PoolName'], + 'array': arrayInfoRec['SerialNumber']}) + else: + compString = ("%(poolName)s+%(array)s" + % {'poolName': arrayInfoRec['PoolName'], + 'array': arrayInfoRec['SerialNumber']}) + if compString == pool: + LOG.info(_LI( + "The pool_name from extraSpecs is %(pool)s."), + {'pool': pool}) + foundArrayInfoRec = arrayInfoRec + break + else: + foundArrayInfoRec = self._get_serial_number(arrayInfo) + + return foundArrayInfoRec + + def _get_random_portgroup(self, element): + """Get a portgroup from list of portgroup. + + Parse all available port groups under a particular + array and choose one. + + :param element: the parent element + :returns: the randomly chosen port group + :raises: VolumeBackendAPIException + """ + portGroupElements = element.getElementsByTagName('PortGroup') + if portGroupElements and len(portGroupElements) > 0: + portGroupNames = [] + for portGroupElement in portGroupElements: + if portGroupElement.childNodes: + portGroupName = portGroupElement.childNodes[0].nodeValue + if portGroupName: + portGroupNames.append(portGroupName.strip()) + portGroupNames = EMCVMAXUtils._filter_list(portGroupNames) + if len(portGroupNames) > 0: + return EMCVMAXUtils._get_random_pg_from_list(portGroupNames) + + exception_message = (_("No Port Group elements found in config file.")) + LOG.error(exception_message) + raise exception.VolumeBackendAPIException(data=exception_message) + + @staticmethod + def _get_random_pg_from_list(portgroupnames): + """From list of portgroup, choose one randomly + + :param portGroupNames: list of available portgroups + :returns: portGroupName - the random portgroup + """ + portgroupname = ( + portgroupnames[random.randint(0, len(portgroupnames) - 1)]) + + LOG.info(_LI("Returning random Port Group: " + "%(portGroupName)s."), + {'portGroupName': portgroupname}) + + return portgroupname + + @staticmethod + def _filter_list(portgroupnames): + """Clean up the port group list + + :param portgroupnames: list of available portgroups + :returns: portgroupnames - cleaned up list + """ + portgroupnames = filter(None, portgroupnames) + # Convert list to set to remove duplicate portgroups + portgroupnames = list(set(portgroupnames)) + return portgroupnames + + def _get_serial_number(self, arrayInfo): + """If we don't have a pool then we just get the serial number. + + If there is more then one serial number we must return an + error and a recommendation to edit the EMC conf file. + + :param arrayInfo: list of records + :returns: any record where serial number exists + :raises: VolumeBackendAPIException + """ + serialNumberList = [] + foundRecord = {} + + for arrayInfoRec in arrayInfo: + serialNumberList.append(arrayInfoRec['SerialNumber']) + foundRecord = arrayInfoRec + + if len(set(serialNumberList)) > 1: + # We have more than one serial number in the dict. + exception_message = (_("Multiple SerialNumbers found, when only " + "one was expected for this operation. " + "Please change your EMC config file.")) + raise exception.VolumeBackendAPIException(data=exception_message) + + return foundRecord + + def _get_pool_info(self, arrayElement, fileName, connargs, interval, + retries, myList): + """Get pool information from element. + + :param arrayElement: arrayElement + :param fileName: configuration file + :param connargs: connection arguments + :param interval: interval, can be None + :param retries: retries, can be None + :param myList: list (input) + :returns: list (output) + :raises: VolumeBackendAPIException + """ + kwargs = {} + portGroup = self._get_random_portgroup(arrayElement) + serialNumber = self._process_tag( + arrayElement, 'SerialNumber') + if serialNumber is None: + exceptionMessage = (_( + "SerialNumber must be in the file " + "%(fileName)s."), + {'fileName': fileName}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + + poolElements = arrayElement.getElementsByTagName('Pool') + if poolElements and len(poolElements) > 0: + for poolElement in poolElements: + poolName = self._process_tag(poolElement, 'PoolName') + if poolName is None: + exceptionMessage = (_( + "PoolName must be in the file " + "%(fileName)s."), + {'fileName': fileName}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + kwargs = self._fill_record(connargs, serialNumber, + poolName, portGroup, + poolElement) + if interval: + kwargs['Interval'] = interval + if retries: + kwargs['Retries'] = retries + myList.append(kwargs) + return myList + + def find_volume_by_device_id_on_array(self, storageSystem, deviceID): + """Find the volume by device ID on a specific array. + + :param conn: connection to the ecom server + :param storageSystem: the storage system name + :param deviceID: string value of the volume device ID + :returns: foundVolumeInstanceName + """ + systemName = 'SYMMETRIX-+-%s' % storageSystem + bindings = {'CreationClassName': 'Symm_StorageVolume', + 'SystemName': systemName, + 'DeviceID': deviceID, + 'SystemCreationClassName': 'Symm_StorageSystem'} + + instanceName = pywbem.CIMInstanceName( + classname='Symm_StorageVolume', + namespace=EMC_ROOT, + keybindings=bindings) + + LOG.debug("Retrieved volume from VMAX: %(instanceName)s", + {'instanceName': instanceName}) + + return instanceName + + def check_volume_no_fast(self, extraSpecs): + """Check if the volume's extraSpecs indicate FAST is enabled. + + :param extraSpecs: dict -- extra spec dict + :raises: VolumeBackendAPIException + """ + try: + if extraSpecs['storagetype:fastpolicy'] is not None: + LOG.warning(_LW( + "FAST is enabled. Policy: %(fastPolicyName)s."), + {'fastPolicyName': extraSpecs['storagetype:fastpolicy']}) + exceptionMessage = (_( + "Manage volume is not supported if FAST is enabled. " + "FAST policy: %(fastPolicyName)s." + ) % {'fastPolicyName': extraSpecs[ + 'storagetype:fastpolicy']}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + else: + return True + except KeyError: + return True + + def check_volume_not_in_masking_view(self, conn, volumeInstanceName, + deviceId): + """Check if volume is in Masking View. + + :param conn: connection to the ecom server + :param volumeInstanceName: the volume instance name + :param deviceId: :raises: VolumeBackendAPIException + :raises: VolumeBackendAPIException + :return: True if not in Masking View + """ + sgInstanceNames = ( + self.get_storage_groups_from_volume( + conn, volumeInstanceName)) + + mvInstanceName = None + for sgInstanceName in sgInstanceNames: + maskingViews = conn.AssociatorNames( + sgInstanceName, + ResultClass='Symm_LunMaskingView') + if len(maskingViews) > 0: + mvInstanceName = maskingViews[0] + if mvInstanceName: + exceptionMessage = (_( + "Unable to import volume %(deviceId)s to cinder. " + "Volume is in masking view %(mv)s.") + % {'deviceId': deviceId, 'mv': mvInstanceName}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + + if not mvInstanceName: + return True + + def check_volume_not_replication_source(self, conn, storageSystemName, + deviceId): + """Check volume not replication source. + + Check if the volume is the source of a replicated + volume. + + :param conn: connection to the ecom server + :param storageSystemName: the storage system name + :param deviceId: string value of the volume device ID + :raises: VolumeBackendAPIException + :returns: True if not replication source + """ + repSessionInstanceName = ( + self.get_associated_replication_from_source_volume( + conn, storageSystemName, deviceId)) + + if repSessionInstanceName: + exceptionMessage = (_( + "Unable to import volume %(deviceId)s to cinder. " + "It is the source volume of replication session %(sync)s.") + % {'deviceId': deviceId, 'sync': repSessionInstanceName}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + else: + return True + + def check_is_volume_in_cinder_managed_pool( + self, conn, volumeInstanceName, cinderPoolInstanceName, + deviceId): + """Check if volume is in a Cinder managed pool. + + :param conn: connection to the ecom server + :param volumeInstanceName: the volume instance name + :param cinderPoolInstanceName: the name of the storage pool + :param deviceId: string value of the volume device ID + :raises: VolumeBackendAPIException + :returns: True if volume in cinder managed pool + """ + volumePoolInstanceName = ( + self.get_assoc_v2_pool_from_volume(conn, + volumeInstanceName)) + if not volumePoolInstanceName: + volumePoolInstanceName = ( + self.get_assoc_v3_pool_from_volume(conn, + volumeInstanceName)) + + volumePoolName = volumePoolInstanceName['InstanceID'] + cinderPoolName = cinderPoolInstanceName['InstanceID'] + + LOG.debug("Storage pool of existing volume: %(volPool)s, " + "Storage pool currently managed by cinder: %(cinderPool)s.", + {'volPool': volumePoolName, + 'cinderPool': cinderPoolName}) + + if volumePoolName != cinderPoolName: + exceptionMessage = (_( + "Unable to import volume %(deviceId)s to cinder. The external " + "volume is not in the pool managed by current cinder host.") + % {'deviceId': deviceId}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + else: + return True + + def get_assoc_v2_pool_from_volume(self, conn, volumeInstanceName): + """Give the volume instance get the associated pool instance + + :param conn: connection to the ecom server + :param volumeInstanceName: the volume instance name + :returns: foundPoolInstanceName + """ + foundPoolInstanceName = None + + foundPoolInstanceNames = ( + conn.AssociatorNames(volumeInstanceName, + ResultClass='EMC_VirtualProvisioningPool')) + + if not foundPoolInstanceNames: + deviceID = volumeInstanceName['DeviceID'] + LOG.debug("Volume %(deviceId)s not in V3 SRP", + {'deviceId': deviceID}) + else: + LOG.debug("Retrieved pool: %(foundPoolInstanceNames)s", + {'foundPoolInstanceNames': foundPoolInstanceNames}) + + if len(foundPoolInstanceNames) > 0: + foundPoolInstanceName = foundPoolInstanceNames[0] + + return foundPoolInstanceName + + def get_assoc_v3_pool_from_volume(self, conn, volumeInstanceName): + """Give the volume instance get the associated pool instance + + :param conn: connection to the ecom server + :param volumeInstanceName: the volume instance name + :returns: foundPoolInstanceName + """ + foundPoolInstanceName = None + + foundPoolInstanceNames = ( + conn.AssociatorNames(volumeInstanceName, + ResultClass='Symm_SRPStoragePool')) + + if not foundPoolInstanceNames: + deviceID = volumeInstanceName['DeviceID'] + LOG.debug("Volume %(deviceId)s not in V3 SRP", + {'deviceId': deviceID}) + exceptionMessage = ("Unable to locate volume %(deviceId)s", + {'deviceId': deviceID}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + else: + LOG.debug("Retrieved pool: %(foundPoolInstanceNames)s", + {'foundPoolInstanceNames': foundPoolInstanceNames}) + + if len(foundPoolInstanceNames) > 0: + foundPoolInstanceName = foundPoolInstanceNames[0] + + return foundPoolInstanceName + + def get_volume_element_name(self, volumeId): + """Get volume element name follows naming convention, i.e. 'OS-UUID'. + + :param volumeId: volume id containing uuid + :returns: volume element name in format of OS-UUID + """ + elementName = volumeId + uuid_regex = (re.compile( + '[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}', + re.I)) + match = uuid_regex.search(volumeId) + if match: + volumeUUID = match.group() + elementName = ("%(prefix)s%(volumeUUID)s" + % {'prefix': VOLUME_ELEMENT_NAME_PREFIX, + 'volumeUUID': volumeUUID}) + LOG.debug( + "get_volume_element_name elementName: %(elementName)s.", + {'elementName': elementName}) + return elementName + + def rename_volume(self, conn, volume, newName): + """Change the volume ElementName to specified new name. + + :param conn: connection to the ecom server + :param volume: the volume instance name or volume instance + :param newName: new ElementName of the volume + :returns: volumeInstance after rename + """ + if type(volume) is pywbem.cim_obj.CIMInstance: + volumeInstance = volume + volumeInstance['ElementName'] = newName + else: + volumeInstance = conn.GetInstance(volume) + volumeInstance['ElementName'] = newName + + LOG.debug("Rename volume to new ElementName %(newName)s.", + {'newName': newName}) + + conn.ModifyInstance(volumeInstance, PropertyList=['ElementName']) + + return volumeInstance + + def get_array_and_device_id(self, volume, external_ref): + """Helper function for manage volume to get array name and device ID. + + :param volume: volume object from API + :param external_ref: the existing volume object to be manged + :returns: string value of the array name and device ID + """ + deviceId = external_ref.get(u'source-name', None) + arrayName = '' + for metadata in volume['volume_metadata']: + if metadata['key'].lower() == 'array': + arrayName = metadata['value'] + break + + if deviceId: + LOG.debug("Get device ID of existing volume - device ID: " + "%(deviceId)s, Array: %(arrayName)s.", + {'deviceId': deviceId, + 'arrayName': arrayName}) + else: + exception_message = (_("Source volume device ID is required.")) + raise exception.VolumeBackendAPIException( + data=exception_message) + return (arrayName, deviceId) + + def get_associated_replication_from_source_volume( + self, conn, storageSystem, sourceDeviceId): + """Get associated replication from source volume. + + Given the source volume device ID, find associated replication + storage synchronized instance names. + + :param conn: connection to the ecom server + :param storageSystem: the storage system name + :param source: target volume object + :returns: foundSyncName (String) + """ + foundSyncInstanceName = None + syncInstanceNames = conn.EnumerateInstanceNames( + 'SE_StorageSynchronized_SV_SV') + for syncInstanceName in syncInstanceNames: + sourceVolume = syncInstanceName['SystemElement'] + if storageSystem != sourceVolume['SystemName']: + continue + if sourceVolume['DeviceID'] == sourceDeviceId: + # Check that it hasn't recently been deleted. + try: + conn.GetInstance(syncInstanceName) + foundSyncInstanceName = syncInstanceName + LOG.debug("Found sync Name: " + "%(syncName)s.", + {'syncName': foundSyncInstanceName}) + except Exception: + foundSyncInstanceName = None + break + + if foundSyncInstanceName is None: + LOG.info(_LI( + "No replication synchronization session found associated " + "with source volume %(source)s on %(storageSystem)s."), + {'source': sourceDeviceId, 'storageSystem': storageSystem}) + + return foundSyncInstanceName + + def get_volume_model_updates( + self, context, volumes, cgId, status='available'): + """Update the volume model's status and return it. + + :param context: the context + :param volumes: volumes object api + :param cgId: cg id + :param status: string value reflects the status of the member volume + :returns: volume_model_updates - updated volumes + """ + volume_model_updates = [] + LOG.info(_LI( + "Updating status for CG: %(id)s."), + {'id': cgId}) + if volumes: + for volume in volumes: + volume_model_updates.append({'id': volume['id'], + 'status': status}) + else: + LOG.info(_LI("No volume found for CG: %(cg)s."), + {'cg': cgId}) + return volume_model_updates + + def get_smi_version(self, conn): + """Get the SMI_S version. + + :param conn: the connection to the ecom server + :returns: string -- version + """ + intVersion = 0 + swIndentityInstances = conn.EnumerateInstances( + 'SE_ManagementServerSoftwareIdentity') + if swIndentityInstances: + swIndentityInstance = swIndentityInstances[0] + majorVersion = swIndentityInstance['MajorVersion'] + minorVersion = swIndentityInstance['MinorVersion'] + revisionNumber = swIndentityInstance['RevisionNumber'] + + intVersion = int(six.text_type(majorVersion) + + six.text_type(minorVersion) + + six.text_type(revisionNumber)) + + LOG.debug("Major version: %(majV)lu, Minor version: %(minV)lu, " + "Revision number: %(revNum)lu, Version: %(intV)lu.", + {'majV': majorVersion, + 'minV': minorVersion, + 'revNum': revisionNumber, + 'intV': intVersion}) + return intVersion + + def get_composite_elements( + self, conn, volumeInstance): + """Get the meta members of a composite volume. + + :param conn: ECOM connection + :param volumeInstance: the volume instance + :returns memberVolumes: a list of meta members + """ + memberVolumes = None + storageSystemName = volumeInstance['SystemName'] + elementCompositionService = self.find_element_composition_service( + conn, storageSystemName) + rc, ret = conn.InvokeMethod( + 'GetCompositeElements', + elementCompositionService, + TheElement=volumeInstance.path) + + if 'OutElements' in ret: + LOG.debug("Get composite elements of volume " + "%(volume)s rc=%(rc)d, ret=%(ret)s", + {'volume': volumeInstance.path, 'rc': rc, 'ret': ret}) + memberVolumes = ret['OutElements'] + return memberVolumes + + def generate_unique_trunc_host(self, hostName): + """Create a unique short host name under 40 chars + + :param sgName: long storage group name + :returns: truncated storage group name + """ + if hostName and len(hostName) > 38: + hostName = hostName.lower() + m = hashlib.md5() + m.update(hostName.encode('utf-8')) + uuid = m.hexdigest() + return( + ("%(host)s%(uuid)s" + % {'host': hostName[-6:], + 'uuid': uuid})) + else: + return hostName + + def generate_unique_trunc_pool(self, poolName): + """Create a unique pool name under 16 chars + + :param poolName: long pool name + :returns: truncated pool name + """ + if poolName and len(poolName) > MAX_POOL_LENGTH: + return ( + ("%(first)s_%(last)s" + % {'first': poolName[:8], + 'last': poolName[-7:]})) + else: + return poolName + + def generate_unique_trunc_fastpolicy(self, fastPolicyName): + """Create a unique fast policy name under 14 chars + + :param fastPolicyName: long fast policy name + :returns: truncated fast policy name + """ + if fastPolicyName and len(fastPolicyName) > MAX_FASTPOLICY_LENGTH: + return ( + ("%(first)s_%(last)s" + % {'first': fastPolicyName[:7], + 'last': fastPolicyName[-6:]})) + else: + return fastPolicyName + + def get_iscsi_protocol_endpoints(self, conn, portgroupinstancename): + """Get the iscsi protocol endpoints of a port group. + + :param conn: the ecom connection + :param portgroupinstancename: the portgroup instance name + :returns: iscsiendpoints + """ + iscsiendpoints = conn.AssociatorNames( + portgroupinstancename, + AssocClass='CIM_MemberOfCollection') + return iscsiendpoints + + def get_tcp_protocol_endpoints(self, conn, iscsiendpointinstancename): + """Get the tcp protocol endpoints associated with an iscsi endpoint + + :param conn: the ecom connection + :param iscsiendpointinstancename: the iscsi endpoint instance name + :returns: tcpendpoints + """ + tcpendpoints = conn.AssociatorNames( + iscsiendpointinstancename, + AssocClass='CIM_BindsTo') + return tcpendpoints + + def get_ip_protocol_endpoints(self, conn, tcpendpointinstancename): + """Get the ip protocol endpoints associated with an tcp endpoint + + :param conn: the ecom connection + :param tcpendpointinstancename: the tcp endpoint instance name + :returns: ipendpoints + """ + ipendpoints = conn.AssociatorNames( + tcpendpointinstancename, + AssocClass='CIM_BindsTo') + return ipendpoints + + def get_iscsi_ip_address(self, conn, ipendpointinstancename): + """Get the IPv4Address from the ip endpoint instance name + + :param conn: the ecom connection + :param ipendpointinstancename: the ip endpoint instance name + :returns: foundIpAddress + """ + foundIpAddress = None + ipendpointinstance = conn.GetInstance(ipendpointinstancename) + propertiesList = ipendpointinstance.properties.items() + for properties in propertiesList: + if properties[0] == 'IPv4Address': + cimProperties = properties[1] + foundIpAddress = cimProperties.value + return foundIpAddress + + def get_target_endpoints(self, conn, hardwareId): + """Given the hardwareId get the target endpoints. + + :param conn: the connection to the ecom server + :param hardwareId: the hardware Id + :returns: targetEndpoints + :raises: VolumeBackendAPIException + """ + protocolControllerInstanceName = self.get_protocol_controller( + conn, hardwareId) + + targetEndpoints = conn.AssociatorNames( + protocolControllerInstanceName, + ResultClass='EMC_FCSCSIProtocolEndpoint') + + return targetEndpoints + + def get_protocol_controller(self, conn, hardwareinstancename): + """Get the front end protocol endpoints of a hardware instance + + :param conn: the ecom connection + :param hardwareinstancename: the hardware instance name + :returns: protocolControllerInstanceName + :raises: VolumeBackendAPIException + """ + protocolControllerInstanceName = None + protocol_controllers = conn.AssociatorNames( + hardwareinstancename, + ResultClass='EMC_FrontEndSCSIProtocolController') + if len(protocol_controllers) > 0: + protocolControllerInstanceName = protocol_controllers[0] + if protocolControllerInstanceName is None: + exceptionMessage = (_( + "Unable to get target endpoints for hardwareId " + "%(hardwareIdInstance)s.") + % {'hardwareIdInstance': hardwareinstancename}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException(data=exceptionMessage) + return protocolControllerInstanceName + + def get_replication_setting_data(self, conn, repServiceInstanceName, + replication_type, extraSpecs): + """Get the replication setting data + + :param conn: connection the ecom server + :param repServiceInstanceName: the storage group instance name + :param replication_type: the replication type + :param copy_methodology: the copy methodology + :returns: instance rsdInstance + """ + repServiceCapabilityInstanceNames = conn.AssociatorNames( + repServiceInstanceName, + ResultClass='CIM_ReplicationServiceCapabilities', + AssocClass='CIM_ElementCapabilities') + repServiceCapabilityInstanceName = ( + repServiceCapabilityInstanceNames[0]) + + rc, rsd = conn.InvokeMethod( + 'GetDefaultReplicationSettingData', + repServiceCapabilityInstanceName, + ReplicationType=self.get_num(replication_type, '16')) + + if rc != 0: + rc, errordesc = self.wait_for_job_complete(conn, rsd, + extraSpecs) + if rc != 0: + exceptionMessage = (_( + "Error getting ReplicationSettingData. " + "Return code: %(rc)lu. " + "Error: %(error)s.") + % {'rc': rc, + 'error': errordesc}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + return rsd + + def set_copy_methodology_in_rsd(self, conn, repServiceInstanceName, + replication_type, copy_methodology, + extraSpecs): + """Get the replication setting data + + :param conn: connection the ecom server + :param repServiceInstanceName: the storage group instance name + :param replication_type: the replication type + :param copy_methodology: the copy methodology + :returns: instance rsdInstance + """ + rsd = self.get_replication_setting_data( + conn, repServiceInstanceName, replication_type, extraSpecs) + rsdInstance = rsd['DefaultInstance'] + rsdInstance['DesiredCopyMethodology'] = ( + self.get_num(copy_methodology, '16')) + return rsdInstance + + def set_target_element_supplier_in_rsd( + self, conn, repServiceInstanceName, replication_type, + target_type, extraSpecs): + """Get the replication setting data + + :param conn: connection the ecom server + :param repServiceInstanceName: the storage group instance name + :param replication_type: the replication type + :param target_type: Use existing, Create new, Use and create + :returns: instance rsdInstance + """ + rsd = self.get_replication_setting_data( + conn, repServiceInstanceName, replication_type, extraSpecs) + rsdInstance = rsd['DefaultInstance'] + rsdInstance['TargetElementSupplier'] = ( + self.get_num(target_type, '16')) + + return rsdInstance + + def get_v3_default_sg_instance_name( + self, conn, poolName, slo, workload, storageSystemName): + """Get the V3 default instance name + + :param conn: the connection to the ecom server + :param poolName: the pool name + :param slo: the SLO + :param workload: the workload + :param storageSystemName: the storage system name + :returns: the storage group instance name + """ + storageGroupName = self.get_v3_storage_group_name( + poolName, slo, workload) + controllerConfigService = ( + self.find_controller_configuration_service( + conn, storageSystemName)) + sgInstanceName = self.find_storage_masking_group( + conn, controllerConfigService, storageGroupName) + return storageGroupName, controllerConfigService, sgInstanceName + + def check_ig_instance_name( + self, conn, initiatorGroupInstanceName): + """Check if a given Initiator Group Instance Name has been deleted. + + :param conn: the ecom connection + :param initiatorGroupInstanceName: the given IG instance name + :return: foundinitiatorGroupInstanceName or None if deleted + """ + foundinitiatorGroupInstanceName = self.get_existing_instance( + conn, initiatorGroupInstanceName) + if foundinitiatorGroupInstanceName is not None: + LOG.debug("Found initiator group name: " + "%(igName)s.", + {'igName': foundinitiatorGroupInstanceName}) + else: + LOG.debug("Could not find initiator group name: " + "%(igName)s.", + {'igName': foundinitiatorGroupInstanceName}) + return foundinitiatorGroupInstanceName + + def get_iqn(self, conn, ipendpointinstancename): + """Get the IPv4Address from the ip endpoint instance name. + + :param conn: the ecom connection + :param ipendpointinstancename: the ip endpoint instance name + :returns: foundIqn + """ + foundIqn = None + ipendpointinstance = conn.GetInstance(ipendpointinstancename) + propertiesList = ipendpointinstance.properties.items() + for properties in propertiesList: + if properties[0] == 'Name': + cimProperties = properties[1] + foundIqn = cimProperties.value + return foundIqn + + def get_total_cap_bytes(self, storagePoolInstance, + max_oversubscription_ratio): + """Get the total capactity in bytes + + This is determined based on a few factors. + 1. Is the MaxOverSubscriptionRatio set in the XML + 2. Is the EMCMaxSubscriptionPercent set on the pool + 3. Whether 1. is greater or less than 2. + 4. If neither 1. or 2. are set a default ratio is set + + :param storagePoolInstance: the pool instance + :param max_oversubscription_ratio: the max over subscription ratio, + can be None + :returns: total_cap_bytes + """ + max_sub_ratio_from_per = self.get_ratio_from_max_sub_per( + storagePoolInstance['EMCMaxSubscriptionPercent']) + if max_sub_ratio_from_per: + max_over_sub_ratio = ( + self.override_ratio(max_oversubscription_ratio, + max_sub_ratio_from_per)) + total_cap_bytes = storagePoolInstance['TotalManagedSpace'] + if not max_over_sub_ratio: + max_over_sub_ratio = max_oversubscription_ratio + + return self.get_oversubscription_capacity(total_cap_bytes, + max_over_sub_ratio) + + def get_ratio_from_max_sub_per(self, max_subscription_percent): + """Get ratio from max subscription percent if it exists. + + Check if the max subscription is set on the pool, if it is convert + it to a ratio. + + :param max_subscription_percent: max subscription percent + :returns: max_over_subscription_ratio + """ + if max_subscription_percent == '0': + return None + try: + max_subscription_percent_int = int(max_subscription_percent) + except ValueError: + LOG.error(_LE("Cannot convert max subscription percent to int.")) + return None + return float(max_subscription_percent_int) / 100 + + def override_ratio(self, max_over_sub_ratio, max_sub_ratio_from_per): + """Override ratio if necessary + + The over subscription ratio will be overriden if the max subscription + percent is less than the user supplied max oversubscription ratio. + + :param max_over_sub_ratio: user supplied over subscription ratio + :param max_sub_ratio_from_per: property on the pool + :returns: max_over_sub_ratio + """ + if max_over_sub_ratio: + try: + max_over_sub_ratio = max(float(max_over_sub_ratio), + float(max_sub_ratio_from_per)) + except ValueError: + max_over_sub_ratio = float(max_sub_ratio_from_per) + elif max_sub_ratio_from_per: + max_over_sub_ratio = float(max_sub_ratio_from_per) + + return max_over_sub_ratio + + def get_oversubscription_capacity(self, total_cap_bytes, + max_over_sub_ratio): + """Get the oversubscription capacity in bytes + + :param total_cap_bytes: capacity in byte + :param max_over_sub_ratio: max over subscription ratio (float) + :returns: over_sub_capacity_bytes + """ + try: + mos_ratio = float(max_over_sub_ratio) + except ValueError: + exceptionMessage = (_("Cannot convert %(maxOverSubRatio)s." + "Please check your value is in float " + "format. E.g 2.0") + % {'maxOverSubRatio': + max_over_sub_ratio}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException(data=exceptionMessage) + try: + total_cap_bytes_int = int(total_cap_bytes) + except ValueError: + exceptionMessage = (_("Cannot convert %(total_cap_bytes)s.") + % {'total_cap_bytes': + total_cap_bytes}) + LOG.error(exceptionMessage) + raise exception.VolumeBackendAPIException(data=exceptionMessage) + + return mos_ratio * total_cap_bytes_int + + def insert_live_migration_record(self, volume, maskingviewdict, + connector, extraSpecs): + """Insert a record of live migration destination into a temporary file + + :param volume: the volume dictionary + :param maskingviewdict: the storage group instance name + :param connector: the connector Object + :param extraSpecs: the extraSpecs dict + """ + live_migration_details = self.get_live_migration_record(volume, True) + if live_migration_details: + if volume['id'] not in live_migration_details: + live_migration_details[volume['id']] = [maskingviewdict, + connector, extraSpecs] + else: + live_migration_details = {volume['id']: [maskingviewdict, + connector, extraSpecs]} + try: + with open(LIVE_MIGRATION_FILE, "wb") as f: + pickle.dump(live_migration_details, f) + except Exception: + exceptionMessage = (_( + "Error in processing live migration file.")) + LOG.exception(exceptionMessage) + raise exception.VolumeBackendAPIException( + data=exceptionMessage) + + def delete_live_migration_record(self, volume): + """Delete record of live migration + + Delete record of live migration destination from file and if + after deletion of record, delete file if empty. + + :param volume: the volume dictionary + """ + live_migration_details = self.get_live_migration_record(volume, True) + if live_migration_details: + if volume['id'] in live_migration_details: + del live_migration_details[volume['id']] + with open(LIVE_MIGRATION_FILE, "wb") as f: + pickle.dump(live_migration_details, f) + else: + LOG.debug("%(Volume)s doesn't exist in live migration " + "record.", + {'Volume': volume['id']}) + if not live_migration_details: + os.remove(LIVE_MIGRATION_FILE) + + def get_live_migration_record(self, volume, returnallrecords): + """get record of live migration destination from a temporary file + + :param volume: the volume dictionary + :param returnallrecords: if true, return all records in file + :returns: returns a single record or all records depending on + returnallrecords flag + """ + returned_record = None + if os.path.isfile(LIVE_MIGRATION_FILE): + with open(LIVE_MIGRATION_FILE, "rb") as f: + live_migration_details = pickle.load(f) + if returnallrecords: + returned_record = live_migration_details + else: + if volume['id'] in live_migration_details: + returned_record = live_migration_details[volume['id']] + else: + LOG.debug("%(Volume)s doesn't exist in live migration " + "record.", + {'Volume': volume['id']}) + return returned_record diff --git a/test_emc_vmax.py b/test_emc_vmax.py index db18ce0..c00afbf 100644 --- a/test_emc_vmax.py +++ b/test_emc_vmax.py @@ -1,4 +1,4 @@ -# Copyright (c) 2012 - 2014 EMC Corporation, Inc. +# Copyright (c) 2012 - 2015 EMC Corporation, Inc. # All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); you may @@ -13,30 +13,34 @@ # License for the specific language governing permissions and limitations # under the License. +import ast import os import shutil +import sys import tempfile -import time -from xml.dom.minidom import Document +from xml.dom import minidom import mock +from oslo_service import loopingcall +from oslo_utils import units import six from cinder import exception -from cinder.openstack.common import log as logging -from cinder.openstack.common import loopingcall +from cinder.i18n import _ +from cinder.objects import fields from cinder import test -from cinder.volume.drivers.emc.emc_vmax_common import EMCVMAXCommon -from cinder.volume.drivers.emc.emc_vmax_fast import EMCVMAXFast -from cinder.volume.drivers.emc.emc_vmax_fc import EMCVMAXFCDriver -from cinder.volume.drivers.emc.emc_vmax_iscsi import EMCVMAXISCSIDriver -from cinder.volume.drivers.emc.emc_vmax_masking import EMCVMAXMasking -from cinder.volume.drivers.emc.emc_vmax_provision_v3 import EMCVMAXProvisionV3 -from cinder.volume.drivers.emc.emc_vmax_utils import EMCVMAXUtils -from cinder.volume import volume_types +from cinder.volume import configuration as conf +from cinder.volume.drivers.emc import emc_vmax_common +from cinder.volume.drivers.emc import emc_vmax_fast +from cinder.volume.drivers.emc import emc_vmax_fc +from cinder.volume.drivers.emc import emc_vmax_iscsi +from cinder.volume.drivers.emc import emc_vmax_masking +from cinder.volume.drivers.emc import emc_vmax_provision +from cinder.volume.drivers.emc import emc_vmax_provision_v3 +from cinder.volume.drivers.emc import emc_vmax_utils +from cinder.volume import volume_types -LOG = logging.getLogger(__name__) CINDER_EMC_CONFIG_DIR = '/etc/cinder/' @@ -76,6 +80,22 @@ class CIM_DeviceMaskingGroup(dict): pass +class EMC_LunMaskingSCSIProtocolController(dict): + pass + + +class CIM_TargetMaskingGroup(dict): + pass + + +class EMC_StorageHardwareID(dict): + pass + + +class CIM_IPProtocolEndpoint(dict): + pass + + class SE_ReplicationSettingData(dict): def __init__(self, *args, **kwargs): self['DefaultInstance'] = self.createInstance() @@ -84,7 +104,7 @@ def createInstance(self): self.DesiredCopyMethodology = 0 -class Fake_CIMProperty(): +class Fake_CIMProperty(object): def fake_getCIMProperty(self): cimproperty = Fake_CIMProperty() @@ -121,13 +141,31 @@ def fake_getRemainingManagedSpaceCIMProperty(self): cimproperty.value = '10000000000' return cimproperty - def fake_getElementNameCIMProperty(self): + def fake_getElementNameCIMProperty(self, name): + cimproperty = Fake_CIMProperty() + cimproperty.value = name + return cimproperty + + def fake_getSupportedReplicationTypes(self): + cimproperty = Fake_CIMProperty() + cimproperty.value = [2, 10] + return cimproperty + + def fake_getipv4address(self): cimproperty = Fake_CIMProperty() - cimproperty.value = 'OS-myhost-MV' + cimproperty.key = 'IPv4Address' + cimproperty.value = '10.10.10.10' return cimproperty + def fake_getiqn(self): + cimproperty = Fake_CIMProperty() + cimproperty.key = 'Name' + cimproperty.value = ( + 'iqn.1992-04.com.emc:600009700bca30c01b9c012000000003,t,0x0001') + return cimproperty -class Fake_CIM_TierPolicyServiceCapabilities(): + +class Fake_CIM_TierPolicyServiceCapabilities(object): def fake_getpolicyinstance(self): classinstance = Fake_CIM_TierPolicyServiceCapabilities() @@ -152,7 +190,7 @@ def fake_getinstancename(self, classname, bindings): return instancename -class FakeDB(): +class FakeDB(object): def volume_update(self, context, volume_id, model_update): pass @@ -183,7 +221,7 @@ def snapshot_get_all_for_cgsnapshot(self, context, cgsnapshot_id): return snapshots -class EMCVMAXCommonData(): +class EMCVMAXCommonData(object): wwpn1 = "123456789012345" wwpn2 = "123456789054321" connector = {'ip': '10.0.0.2', @@ -210,19 +248,23 @@ class EMCVMAXCommonData(): default_storage_group = ( u'//10.10.10.10/root/emc: SE_DeviceMaskingGroup.InstanceID=' '"SYMMETRIX+000198700440+OS_default_GOLD1_SG"') + default_sg_instance_name = { + 'CreationClassName': 'CIM_DeviceMaskingGroup', + 'ElementName': 'OS_default_GOLD1_SG', + 'SystemName': 'SYMMETRIX+000195900551'} storage_system = 'SYMMETRIX+000195900551' storage_system_v3 = 'SYMMETRIX-+-000197200056' - lunmaskctrl_id = \ - 'SYMMETRIX+000195900551+OS-fakehost-gold-MV' - lunmaskctrl_name = \ - 'OS-fakehost-gold-MV' - - initiatorgroup_id = \ - 'SYMMETRIX+000195900551+OS-fakehost-IG' - initiatorgroup_name = \ - 'OS-fakehost-IG' + port_group = 'OS-portgroup-PG' + lunmaskctrl_id = ( + 'SYMMETRIX+000195900551+OS-fakehost-gold-I-MV') + lunmaskctrl_name = ( + 'OS-fakehost-gold-I-MV') + + initiatorgroup_id = ( + 'SYMMETRIX+000195900551+OS-fakehost-IG') + initiatorgroup_name = 'OS-fakehost-IG' initiatorgroup_creationclass = 'SE_InitiatorMaskingGroup' - + iscsi_initiator = 'iqn.1993-08.org.debian' storageextent_creationclass = 'CIM_StorageExtent' initiator1 = 'iqn.1993-08.org.debian: 01: 1a2b3c4d5f6g' stconf_service_creationclass = 'Symm_StorageConfigurationService' @@ -241,13 +283,24 @@ class EMCVMAXCommonData(): storagepool_creationclass = 'Symm_VirtualProvisioningPool' srpstoragepool_creationclass = 'Symm_SRPStoragePool' storagegroup_creationclass = 'CIM_DeviceMaskingGroup' - hardwareid_creationclass = 'SE_StorageHardwareID' + hardwareid_creationclass = 'EMC_StorageHardwareID' replicationgroup_creationclass = 'CIM_ReplicationGroup' storagepoolid = 'SYMMETRIX+000195900551+U+gold' - storagegroupname = 'OS_default_GOLD1_SG' + storagegroupname = 'OS-fakehost-gold-I-SG' + defaultstoragegroupname = 'OS_default_GOLD1_SG' storagevolume_creationclass = 'EMC_StorageVolume' policyrule = 'gold' poolname = 'gold' + totalmanagedspace_bits = '1000000000000' + subscribedcapacity_bits = '500000000000' + remainingmanagedspace_bits = '500000000000' + maxsubscriptionpercent = 150 + totalmanagedspace_gbs = 931 + subscribedcapacity_gbs = 465 + remainingmanagedspace_gbs = 465 + fake_host = 'HostX@Backend#gold+1234567891011' + fake_host_v3 = 'HostX@Backend#Bronze+SRP_1+1234567891011' + fake_host_2_v3 = 'HostY@Backend#SRP_1+1234567891011' unit_creationclass = 'CIM_ProtocolControllerForUnit' storage_type = 'gold' @@ -260,11 +313,48 @@ class EMCVMAXCommonData(): 'SystemName': u'SYMMETRIX+000195900551', 'DeviceID': u'99999', 'SystemCreationClassName': u'Symm_StorageSystem'} + keybindings3 = {'CreationClassName': u'Symm_StorageVolume', + 'SystemName': u'SYMMETRIX+000195900551', + 'DeviceID': u'10', + 'SystemCreationClassName': u'Symm_StorageSystem'} provider_location = {'classname': 'Symm_StorageVolume', 'keybindings': keybindings} provider_location2 = {'classname': 'Symm_StorageVolume', 'keybindings': keybindings2} - + provider_location3 = {'classname': 'Symm_StorageVolume', + 'keybindings': keybindings3} + + provider_location_multi_pool = {'classname': 'Symm_StorageVolume', + 'keybindings': keybindings, + 'version': '2.2.0'} + + keybindings_manage = {'CreationClassName': 'Symm_StorageVolume', + 'SystemName': 'SYMMETRIX+000195900551', + 'DeviceID': '10', + 'SystemCreationClassName': 'Symm_StorageSystem'} + provider_location_manage = {'classname': 'Symm_StorageVolume', + 'keybindings': keybindings_manage} + + manage_vol = EMC_StorageVolume() + manage_vol['CreationClassName'] = 'Symm_StorageVolume' + manage_vol['ElementName'] = 'OS-Test_Manage_vol' + manage_vol['DeviceID'] = '10' + manage_vol['SystemName'] = 'SYMMETRIX+000195900551' + manage_vol['SystemCreationClassName'] = 'Symm_StorageSystem' + manage_vol.path = manage_vol + + block_size = 512 + majorVersion = 1 + minorVersion = 2 + revNumber = 3 + block_size = 512 + + metaHead_volume = {'DeviceID': 10, + 'ConsumableBlocks': 1000} + meta_volume1 = {'DeviceID': 11, + 'ConsumableBlocks': 200} + meta_volume2 = {'DeviceID': 12, + 'ConsumableBlocks': 300} properties = {'ConsumableBlocks': '12345', 'BlockSize': '512'} @@ -280,9 +370,9 @@ class EMCVMAXCommonData(): 'volume_type_id': 'abc', 'provider_location': six.text_type(provider_location), 'status': 'available', - 'host': 'fake-host', + 'host': fake_host, 'NumberOfBlocks': 100, - 'BlockSize': 512 + 'BlockSize': block_size } test_volume_v2 = {'name': 'vol1', @@ -297,9 +387,9 @@ class EMCVMAXCommonData(): 'volume_type_id': 'abc', 'provider_location': six.text_type(provider_location), 'status': 'available', - 'host': 'fake-host', + 'host': fake_host, 'NumberOfBlocks': 100, - 'BlockSize': 512 + 'BlockSize': block_size } test_volume_v3 = {'name': 'vol1', @@ -314,9 +404,9 @@ class EMCVMAXCommonData(): 'volume_type_id': 'abc', 'provider_location': six.text_type(provider_location), 'status': 'available', - 'host': 'fake-host', + 'host': fake_host_v3, 'NumberOfBlocks': 100, - 'BlockSize': 512 + 'BlockSize': block_size } test_volume_CG = {'name': 'volInCG', @@ -333,9 +423,26 @@ class EMCVMAXCommonData(): 'volume_type_id': 'abc', 'provider_location': six.text_type(provider_location), 'status': 'available', - 'host': 'fake-host' + 'host': fake_host } + test_volume_CG_v3 = {'name': 'volInCG', + 'consistencygroup_id': 'abc', + 'size': 1, + 'volume_name': 'volInCG', + 'id': 'volInCG', + 'device_id': '1', + 'provider_auth': None, + 'project_id': 'project', + 'display_name': 'volInCG', + 'display_description': + 'test volume in Consistency group', + 'volume_type_id': 'abc', + 'provider_location': + six.text_type(provider_location), + 'status': 'available', + 'host': fake_host_v3} + test_failed_volume = {'name': 'failed_vol', 'size': 1, 'volume_name': 'failed_vol', @@ -345,7 +452,8 @@ class EMCVMAXCommonData(): 'project_id': 'project', 'display_name': 'failed_vol', 'display_description': 'test failed volume', - 'volume_type_id': 'abc'} + 'volume_type_id': 'abc', + 'host': fake_host} failed_delete_vol = {'name': 'failed_delete_vol', 'size': '-1', @@ -357,59 +465,90 @@ class EMCVMAXCommonData(): 'display_name': 'failed delete vol', 'display_description': 'failed delete volume', 'volume_type_id': 'abc', - 'provider_location': six.text_type(provider_location2) - } + 'provider_location': + six.text_type(provider_location2), + 'host': fake_host} test_source_volume = {'size': 1, 'volume_type_id': 'sourceid', 'display_name': 'sourceVolume', 'name': 'sourceVolume', - 'id': 'sourceVolume', 'device_id': '1', 'volume_name': 'vmax-154326', 'provider_auth': None, - 'project_id': - 'project', 'id': '2', + 'project_id': 'project', + 'id': '2', + 'host': fake_host, 'provider_location': - six.text_type(provider_location), + six.text_type(provider_location), 'display_description': 'snapshot source volume'} + test_source_volume_v3 = {'size': 1, + 'volume_type_id': 'sourceid', + 'display_name': 'sourceVolume', + 'name': 'sourceVolume', + 'device_id': '1', + 'volume_name': 'vmax-154326', + 'provider_auth': None, + 'project_id': 'project', + 'id': '2', + 'host': fake_host_v3, + 'provider_location': + six.text_type(provider_location), + 'display_description': 'snapshot source volume'} + test_CG = {'name': 'myCG1', 'id': '12345abcde', 'volume_type_id': 'abc', - 'status': 'available', - 'host': 'fake-host' + 'status': fields.ConsistencyGroupStatus.AVAILABLE } test_snapshot = {'name': 'myCG1', 'id': '12345abcde', - 'status': 'available' + 'status': 'available', + 'host': fake_host } test_CG_snapshot = {'name': 'testSnap', 'id': '12345abcde', 'consistencygroup_id': '123456789', 'status': 'available', - 'snapshots': [] + 'snapshots': [], + 'consistencygroup': test_CG } location_info = {'location_info': '000195900551#silver#None', 'storage_protocol': 'ISCSI'} + location_info_v3 = {'location_info': '1234567891011#SRP_1#Bronze#DSS', + 'storage_protocol': 'FC'} test_host = {'capabilities': location_info, 'host': 'fake_host'} - - location_info_v3 = {'location_info': '0123456789#SRP_1#Bronze#DSS', - 'storage_protocol': 'FC'} test_host_v3 = {'capabilities': location_info_v3, - 'host': 'fake_v3_host'} + 'host': fake_host_2_v3} + initiatorNames = ["123456789012345", "123456789054321"] + storagegroups = [{'CreationClassName': storagegroup_creationclass, + 'ElementName': storagegroupname}, + {'CreationClassName': storagegroup_creationclass, + 'ElementName': 'OS-SRP_1-Bronze-DSS-SG'}] test_ctxt = {} new_type = {} diff = {} + extra_specs = {'storagetype:pool': u'SRP_1', + 'volume_backend_name': 'V3_BE', + 'storagetype:workload': u'DSS', + 'storagetype:slo': u'Bronze', + 'storagetype:array': u'1234567891011', + 'isV3': True, + 'portgroupname': u'OS-portgroup-PG'} + remainingSLOCapacity = '123456789' + SYNCHRONIZED = 4 + UNSYNCHRONIZED = 3 -class FakeLookupService(): + +class FakeLookupService(object): def get_device_mapping_from_network(self, initiator_wwns, target_wwns): return EMCVMAXCommonData.device_map -class FakeEcomConnection(): +class FakeEcomConnection(object): def __init__(self, *args, **kwargs): self.data = EMCVMAXCommonData() @@ -432,36 +571,37 @@ def InvokeMethod(self, MethodName, Service, ElementName=None, InPool=None, SourceGroup=None, TargetGroup=None, Goal=None, Type=None, EMCSRP=None, EMCSLO=None, EMCWorkload=None, EMCCollections=None, InitiatorMaskingGroup=None, - DeviceMaskingGroup=None, TargetMaskingGroup=None): + DeviceMaskingGroup=None, TargetMaskingGroup=None, + ProtocolController=None, StorageID=None, IDType=None, + WaitForCopyState=None, Collections=None): - rc = 0L + rc = 0 myjob = SE_ConcreteJob() myjob.classname = 'SE_ConcreteJob' myjob['InstanceID'] = '9999' myjob['status'] = 'success' myjob['type'] = ElementName - if Size == -1073741824 and \ - MethodName == 'CreateOrModifyCompositeElement': - rc = 0L + if Size == -1073741824 and ( + MethodName == 'CreateOrModifyCompositeElement'): + rc = 0 myjob = SE_ConcreteJob() myjob.classname = 'SE_ConcreteJob' myjob['InstanceID'] = '99999' myjob['status'] = 'success' myjob['type'] = 'failed_delete_vol' - if ElementName == 'failed_vol' and \ - MethodName == 'CreateOrModifyElementFromStoragePool': - rc = 10L + if ElementName == 'failed_vol' and ( + MethodName == 'CreateOrModifyElementFromStoragePool'): + rc = 10 myjob['status'] = 'failure' - elif TheElements and \ - TheElements[0]['DeviceID'] == '99999' and \ - MethodName == 'EMCReturnToStoragePool': - rc = 10L + elif TheElements and TheElements[0]['DeviceID'] == '99999' and ( + MethodName == 'ReturnElementsToStoragePool'): + rc = 10 myjob['status'] = 'failure' elif HardwareId: - rc = 0L + rc = 0 targetendpoints = {} endpoints = [] endpoint = {} @@ -474,12 +614,31 @@ def InvokeMethod(self, MethodName, Service, ElementName=None, InPool=None, endpoints.append(endpoint2) targetendpoints['TargetEndpoints'] = endpoints return rc, targetendpoints - elif ReplicationType and \ - MethodName == 'GetDefaultReplicationSettingData': - rc = 0L + elif ReplicationType and ( + MethodName == 'GetDefaultReplicationSettingData'): + rc = 0 rsd = SE_ReplicationSettingData() rsd['DefaultInstance'] = SE_ReplicationSettingData() return rc, rsd + if MethodName == 'CreateStorageHardwareID': + ret = {} + rc = 0 + ret['HardwareID'] = self.data.iscsi_initiator + return rc, ret + if MethodName == 'GetSupportedSizeRange': + ret = {} + rc = 0 + ret['EMCInformationSource'] = 3 + ret['EMCRemainingSLOCapacity'] = self.data.remainingSLOCapacity + return rc, ret + elif MethodName == 'GetCompositeElements': + ret = {} + rc = 0 + ret['OutElements'] = [self.data.metaHead_volume, + self.data.meta_volume1, + self.data.meta_volume2] + return rc, ret + job = {'Job': myjob} return rc, job @@ -501,6 +660,8 @@ def EnumerateInstanceNames(self, name): result = self._enum_storagevolumes() elif name == 'Symm_StorageVolume': result = self._enum_storagevolumes() + elif name == 'CIM_StorageVolume': + result = self._enum_storagevolumes() elif name == 'CIM_ProtocolControllerForUnit': result = self._enum_unitnames() elif name == 'EMC_LunMaskingSCSIProtocolController': @@ -513,6 +674,14 @@ def EnumerateInstanceNames(self, name): result = self._enum_storhdwids() elif name == 'EMC_StorageSystem': result = self._enum_storagesystems() + elif name == 'Symm_TierPolicyRule': + result = self._enum_policyrules() + elif name == 'CIM_ReplicationServiceCapabilities': + result = self._enum_repservcpbls() + elif name == 'SE_StorageSynchronized_SV_SV': + result = self._enum_storageSyncSvSv() + elif name == 'Symm_SRPStoragePool': + result = self._enum_srpstoragepool() else: result = self._default_enum() return result @@ -523,6 +692,8 @@ def EnumerateInstances(self, name): result = self._enum_pool_details() elif name == 'SE_StorageHardwareID': result = self._enum_storhdwids() + elif name == 'SE_ManagementServerSoftwareIdentity': + result = self._enum_sw_identity() else: result = self._default_enum() return result @@ -559,17 +730,26 @@ def GetInstance(self, objectpath, LocalOnly=False): result = self._getinstance_targetmaskinggroup(objectpath) elif name == 'CIM_DeviceMaskingGroup': result = self._getinstance_devicemaskinggroup(objectpath) + elif name == 'EMC_StorageHardwareID': + result = self._getinstance_storagehardwareid(objectpath) + elif name == 'Symm_VirtualProvisioningPool': + result = self._getinstance_pool(objectpath) + elif name == 'Symm_ReplicationServiceCapabilities': + result = self._getinstance_replicationServCapabilities(objectpath) else: result = self._default_getinstance(objectpath) return result + def ModifyInstance(self, objectpath, PropertyList=None): + pass + def DeleteInstance(self, objectpath): pass def Associators(self, objectpath, ResultClass='EMC_StorageHardwareID'): result = None - if ResultClass == 'EMC_StorageHardwareID': + if '_StorageHardwareID' in ResultClass: result = self._assoc_hdwid() elif ResultClass == 'EMC_iSCSIProtocolEndpoint': result = self._assoc_endpoint() @@ -577,12 +757,14 @@ def Associators(self, objectpath, ResultClass='EMC_StorageHardwareID'): result = self._assoc_storagevolume(objectpath) elif ResultClass == 'Symm_LunMaskingView': result = self._assoc_maskingview() - elif ResultClass == 'EMC_LunMaskingSCSIProtocolController': - result = self._assoc_maskingview() elif ResultClass == 'CIM_DeviceMaskingGroup': result = self._assoc_storagegroup() elif ResultClass == 'CIM_StorageExtent': result = self._assoc_storageextent() + elif ResultClass == 'EMC_LunMaskingSCSIProtocolController': + result = self._assoc_lunmaskctrls() + elif ResultClass == 'CIM_TargetMaskingGroup': + result = self._assoc_portgroup() else: result = self._default_assoc(objectpath) return result @@ -590,17 +772,40 @@ def Associators(self, objectpath, ResultClass='EMC_StorageHardwareID'): def AssociatorNames(self, objectpath, ResultClass='default', AssocClass='default'): result = None + if objectpath == 'point_to_storage_instance_names': + result = ['FirstStorageTierInstanceNames'] + + if ResultClass != 'default': + result = self.ResultClassHelper(ResultClass, objectpath) + + if result is None and AssocClass != 'default': + result = self.AssocClassHelper(AssocClass, objectpath) + if result is None: + result = self._default_assocnames(objectpath) + return result + + def AssocClassHelper(self, AssocClass, objectpath): + if AssocClass == 'CIM_HostedService': + result = self._assocnames_hostedservice() + elif AssocClass == 'CIM_AssociatedTierPolicy': + result = self._assocnames_assoctierpolicy() + elif AssocClass == 'CIM_OrderedMemberOfCollection': + result = self._enum_storagevolumes() + elif AssocClass == 'CIM_BindsTo': + result = self._assocnames_bindsto() + elif AssocClass == 'CIM_MemberOfCollection': + result = self._assocnames_memberofcollection() + else: + result = None + return result + def ResultClassHelper(self, ResultClass, objectpath): if ResultClass == 'EMC_LunMaskingSCSIProtocolController': result = self._assocnames_lunmaskctrl() - elif AssocClass == 'CIM_HostedService': - result = self._assocnames_hostedservice() elif ResultClass == 'CIM_TierPolicyServiceCapabilities': result = self._assocnames_policyCapabilities() elif ResultClass == 'Symm_TierPolicyRule': result = self._assocnames_policyrule() - elif AssocClass == 'CIM_AssociatedTierPolicy': - result = self._assocnames_assoctierpolicy() elif ResultClass == 'CIM_StoragePool': result = self._assocnames_storagepool() elif ResultClass == 'EMC_VirtualProvisioningPool': @@ -613,6 +818,8 @@ def AssociatorNames(self, objectpath, result = self._enum_storagevolumes() elif ResultClass == 'SE_InitiatorMaskingGroup': result = self._enum_initiatorMaskingGroup() + elif ResultClass == 'CIM_InitiatorMaskingGroup': + result = self._enum_initiatorMaskingGroup() elif ResultClass == 'CIM_StorageExtent': result = self._enum_storage_extent() elif ResultClass == 'SE_StorageHardwareID': @@ -621,10 +828,10 @@ def AssociatorNames(self, objectpath, result = self._enum_repservcpbls() elif ResultClass == 'CIM_ReplicationGroup': result = self._enum_repgroups() - elif AssocClass == 'CIM_OrderedMemberOfCollection': - result = self._enum_storagevolumes() elif ResultClass == 'Symm_FCSCSIProtocolEndpoint': result = self._enum_fcscsiendpoint() + elif ResultClass == 'EMC_FCSCSIProtocolEndpoint': + result = self._enum_fcscsiendpoint() elif ResultClass == 'Symm_SRPStoragePool': result = self._enum_srpstoragepool() elif ResultClass == 'Symm_StoragePoolCapabilities': @@ -632,13 +839,19 @@ def AssociatorNames(self, objectpath, elif ResultClass == 'CIM_storageSetting': result = self._enum_storagesettings() elif ResultClass == 'CIM_TargetMaskingGroup': - result = self._enum_targetMaskingGroup() + result = self._assocnames_portgroup() elif ResultClass == 'CIM_InitiatorMaskingGroup': result = self._enum_initMaskingGroup() elif ResultClass == 'Symm_LunMaskingView': result = self._enum_maskingView() + elif ResultClass == 'EMC_Meta': + result = self._enum_metavolume() + elif ResultClass == 'EMC_FrontEndSCSIProtocolController': + result = self._enum_maskingView() + elif ResultClass == 'CIM_TierPolicyRule': + result = self._assocnames_tierpolicy(objectpath) else: - result = self._default_assocnames(objectpath) + result = None return result def ReferenceNames(self, objectpath, @@ -672,8 +885,7 @@ def _ref_unitnames(self): return unitnames - def _ref_unitnames2(self): - unitnames = [] + def mv_entry(self, mvname): unitname = {} dependent = {} @@ -685,18 +897,32 @@ def _ref_unitnames2(self): antecedent = SYMM_LunMasking() antecedent['CreationClassName'] = self.data.lunmask_creationclass2 antecedent['SystemName'] = self.data.storage_system + antecedent['ElementName'] = mvname classcimproperty = Fake_CIMProperty() elementName = ( - classcimproperty.fake_getElementNameCIMProperty()) + classcimproperty.fake_getElementNameCIMProperty(mvname)) properties = {u'ElementName': elementName} antecedent.properties = properties unitname['Dependent'] = dependent unitname['Antecedent'] = antecedent unitname['CreationClassName'] = self.data.unit_creationclass + return unitname + + def _ref_unitnames2(self): + unitnames = [] + unitname = self.mv_entry('OS-myhost-MV') unitnames.append(unitname) + # Second masking + unitname2 = self.mv_entry('OS-fakehost-MV') + unitnames.append(unitname2) + + # third masking + amended = 'OS-rslong493156848e71b072a17c1c4625e45f75-MV' + unitname3 = self.mv_entry(amended) + unitnames.append(unitname3) return unitnames def _default_ref(self, objectpath): @@ -704,13 +930,20 @@ def _default_ref(self, objectpath): def _assoc_hdwid(self): assocs = [] - assoc = {} + assoc = EMC_StorageHardwareID() assoc['StorageID'] = self.data.connector['initiator'] + assoc['SystemName'] = self.data.storage_system + assoc['CreationClassName'] = 'EMC_StorageHardwareID' + assoc.path = assoc assocs.append(assoc) for wwpn in self.data.connector['wwpns']: - assoc2 = {} + assoc2 = EMC_StorageHardwareID() assoc2['StorageID'] = wwpn + assoc2['SystemName'] = self.data.storage_system + assoc2['CreationClassName'] = 'EMC_StorageHardwareID' + assoc2.path = assoc2 assocs.append(assoc2) + assocs.append(assoc) return assocs def _assoc_endpoint(self): @@ -723,14 +956,41 @@ def _assoc_endpoint(self): def _assoc_storagegroup(self): assocs = [] - assoc = CIM_DeviceMaskingGroup() - assoc['ElementName'] = 'OS_default_GOLD1_SG' + assoc1 = CIM_DeviceMaskingGroup() + assoc1['ElementName'] = self.data.storagegroupname + assoc1['SystemName'] = self.data.storage_system + assoc1['CreationClassName'] = 'CIM_DeviceMaskingGroup' + assoc1.path = assoc1 + assocs.append(assoc1) + assoc2 = CIM_DeviceMaskingGroup() + assoc2['ElementName'] = self.data.defaultstoragegroupname + assoc2['SystemName'] = self.data.storage_system + assoc2['CreationClassName'] = 'CIM_DeviceMaskingGroup' + assoc2.path = assoc2 + assocs.append(assoc2) + return assocs + + def _assoc_portgroup(self): + assocs = [] + assoc = CIM_TargetMaskingGroup() + assoc['ElementName'] = self.data.port_group assoc['SystemName'] = self.data.storage_system - assoc['CreationClassName'] = 'CIM_DeviceMaskingGroup' + assoc['CreationClassName'] = 'CIM_TargetMaskingGroup' assoc.path = assoc assocs.append(assoc) return assocs + def _assoc_lunmaskctrls(self): + ctrls = [] + ctrl = EMC_LunMaskingSCSIProtocolController() + ctrl['CreationClassName'] = self.data.lunmask_creationclass + ctrl['DeviceID'] = self.data.lunmaskctrl_id + ctrl['SystemName'] = self.data.storage_system + ctrl['ElementName'] = self.data.lunmaskctrl_name + ctrl.path = ctrl + ctrls.append(ctrl) + return ctrls + def _assoc_maskingview(self): assocs = [] assoc = SYMM_LunMasking() @@ -812,6 +1072,15 @@ def _assocnames_storagegroup(self): def _assocnames_storagevolume(self): return self._enum_storagevolume() + def _assocnames_portgroup(self): + return self._enum_portgroup() + + def _assocnames_memberofcollection(self): + return self._enum_hostedservice() + + def _assocnames_bindsto(self): + return self._enum_ipprotocolendpoint() + def _default_assocnames(self, objectpath): return objectpath @@ -845,6 +1114,7 @@ def _getinstance_initiatormaskinggroup(self, objectpath): self.data.initiatorgroup_creationclass) initiatorgroup['DeviceID'] = self.data.initiatorgroup_id initiatorgroup['SystemName'] = self.data.storage_system + initiatorgroup['ElementName'] = self.data.initiatorgroup_name initiatorgroup.path = initiatorgroup return initiatorgroup @@ -856,6 +1126,17 @@ def _getinstance_storagehardwareid(self, objectpath): hardwareid.path = hardwareid return hardwareid + def _getinstance_pool(self, objectpath): + pool = {} + pool['CreationClassName'] = 'Symm_VirtualProvisioningPool' + pool['ElementName'] = self.data.poolname + pool['SystemName'] = self.data.storage_system + pool['TotalManagedSpace'] = self.data.totalmanagedspace_bits + pool['EMCSubscribedCapacity'] = self.data.subscribedcapacity_bits + pool['RemainingManagedSpace'] = self.data.remainingmanagedspace_bits + pool['EMCMaxSubscriptionPercent'] = self.data.maxsubscriptionpercent + return pool + def _getinstance_replicationgroup(self, objectpath): replicationgroup = {} replicationgroup['CreationClassName'] = ( @@ -867,6 +1148,7 @@ def _getinstance_srpstoragepool(self, objectpath): srpstoragepool = SYMM_SrpStoragePool() srpstoragepool['CreationClassName'] = ( self.data.srpstoragepool_creationclass) + srpstoragepool['ElementName'] = 'SRP_1' classcimproperty = Fake_CIMProperty() totalManagedSpace = ( @@ -879,15 +1161,33 @@ def _getinstance_srpstoragepool(self, objectpath): return srpstoragepool def _getinstance_targetmaskinggroup(self, objectpath): - targetmaskinggroup = {} + targetmaskinggroup = CIM_TargetMaskingGroup() targetmaskinggroup['CreationClassName'] = 'CIM_TargetMaskingGroup' - targetmaskinggroup['ElementName'] = 'myPortGroup' + targetmaskinggroup['ElementName'] = self.data.port_group + targetmaskinggroup.path = targetmaskinggroup return targetmaskinggroup def _getinstance_devicemaskinggroup(self, objectpath): targetmaskinggroup = {} - targetmaskinggroup['CreationClassName'] = 'CIM_DeviceMaskingGroup' - targetmaskinggroup['ElementName'] = 'OS_default_GOLD1_SG' + if 'CreationClassName' in objectpath: + targetmaskinggroup['CreationClassName'] = ( + objectpath['CreationClassName']) + else: + targetmaskinggroup['CreationClassName'] = ( + 'CIM_DeviceMaskingGroup') + if 'ElementName' in objectpath: + targetmaskinggroup['ElementName'] = objectpath['ElementName'] + else: + targetmaskinggroup['ElementName'] = ( + self.data.storagegroupname) + if 'EMCMaximumIO' in objectpath: + targetmaskinggroup['EMCMaximumIO'] = objectpath['EMCMaximumIO'] + if 'EMCMaximumBandwidth' in objectpath: + targetmaskinggroup['EMCMaximumBandwidth'] = ( + objectpath['EMCMaximumBandwidth']) + if 'EMCMaxIODynamicDistributionType' in objectpath: + targetmaskinggroup['EMCMaxIODynamicDistributionType'] = ( + objectpath['EMCMaxIODynamicDistributionType']) return targetmaskinggroup def _getinstance_unit(self, objectpath): @@ -934,35 +1234,66 @@ def _getinstance_syncsvsv(self, objectpath): svInstance['SyncedElement'] = 'SyncedElement' svInstance['SystemElement'] = 'SystemElement' svInstance['PercentSynced'] = 100 + if 'PercentSynced' in objectpath and objectpath['PercentSynced'] < 100: + svInstance['PercentSynced'] = 50 + svInstance['CopyState'] = self.data.SYNCHRONIZED + if 'CopyState' in objectpath and ( + objectpath['CopyState'] != self.data.SYNCHRONIZED): + svInstance['CopyState'] = self.data.UNSYNCHRONIZED return svInstance + def _getinstance_replicationServCapabilities(self, objectpath): + repServCpblInstance = SYMM_SrpStoragePool() + classcimproperty = Fake_CIMProperty() + repTypesCimproperty = ( + classcimproperty.fake_getSupportedReplicationTypes()) + properties = {u'SupportedReplicationTypes': repTypesCimproperty} + repServCpblInstance.properties = properties + return repServCpblInstance + + def _getinstance_ipprotocolendpoint(self, objectpath): + return self._enum_ipprotocolendpoint()[0] + + def _getinstance_lunmaskingview(self, objectpath): + return self._enum_maskingView()[0] + def _default_getinstance(self, objectpath): return objectpath def _enum_stconfsvcs(self): conf_services = [] - conf_service = {} - conf_service['SystemName'] = self.data.storage_system - conf_service['CreationClassName'] = \ - self.data.stconf_service_creationclass - conf_services.append(conf_service) + conf_service1 = {} + conf_service1['SystemName'] = self.data.storage_system + conf_service1['CreationClassName'] = ( + self.data.stconf_service_creationclass) + conf_services.append(conf_service1) + conf_service2 = {} + conf_service2['SystemName'] = self.data.storage_system_v3 + conf_service2['CreationClassName'] = ( + self.data.stconf_service_creationclass) + conf_services.append(conf_service2) return conf_services def _enum_ctrlconfsvcs(self): conf_services = [] conf_service = {} conf_service['SystemName'] = self.data.storage_system - conf_service['CreationClassName'] = \ - self.data.ctrlconf_service_creationclass + conf_service['CreationClassName'] = ( + self.data.ctrlconf_service_creationclass) conf_services.append(conf_service) + conf_service1 = {} + conf_service1['SystemName'] = self.data.storage_system_v3 + conf_service1['CreationClassName'] = ( + self.data.ctrlconf_service_creationclass) + conf_services.append(conf_service1) return conf_services def _enum_elemcompsvcs(self): comp_services = [] comp_service = {} comp_service['SystemName'] = self.data.storage_system - comp_service['CreationClassName'] = \ - self.data.elementcomp_service_creationclass + comp_service['CreationClassName'] = ( + self.data.elementcomp_service_creationclass) comp_services.append(comp_service) return comp_services @@ -970,8 +1301,8 @@ def _enum_storrelocsvcs(self): reloc_services = [] reloc_service = {} reloc_service['SystemName'] = self.data.storage_system - reloc_service['CreationClassName'] = \ - self.data.storreloc_service_creationclass + reloc_service['CreationClassName'] = ( + self.data.storreloc_service_creationclass) reloc_services.append(reloc_service) return reloc_services @@ -979,16 +1310,21 @@ def _enum_replicsvcs(self): replic_services = [] replic_service = {} replic_service['SystemName'] = self.data.storage_system - replic_service['CreationClassName'] = \ - self.data.replication_service_creationclass + replic_service['CreationClassName'] = ( + self.data.replication_service_creationclass) replic_services.append(replic_service) + replic_service2 = {} + replic_service2['SystemName'] = self.data.storage_system_v3 + replic_service2['CreationClassName'] = ( + self.data.replication_service_creationclass) + replic_services.append(replic_service2) return replic_services def _enum_pools(self): pools = [] pool = {} - pool['InstanceID'] = self.data.storage_system + '+U+' + \ - self.data.storage_type + pool['InstanceID'] = ( + self.data.storage_system + '+U+' + self.data.storage_type) pool['CreationClassName'] = 'Symm_VirtualProvisioningPool' pool['ElementName'] = 'gold' pools.append(pool) @@ -997,8 +1333,8 @@ def _enum_pools(self): def _enum_pool_details(self): pools = [] pool = {} - pool['InstanceID'] = self.data.storage_system + '+U+' + \ - self.data.storage_type + pool['InstanceID'] = ( + self.data.storage_system + '+U+' + self.data.storage_type) pool['CreationClassName'] = 'Symm_VirtualProvisioningPool' pool['TotalManagedSpace'] = 12345678 pool['RemainingManagedSpace'] = 123456 @@ -1009,7 +1345,7 @@ def _enum_storagevolumes(self): vols = [] vol = EMC_StorageVolume() - vol['name'] = self.data.test_volume['name'] + vol['Name'] = self.data.test_volume['name'] vol['CreationClassName'] = 'Symm_StorageVolume' vol['ElementName'] = self.data.test_volume['id'] vol['DeviceID'] = self.data.test_volume['device_id'] @@ -1056,8 +1392,8 @@ def _enum_storagevolumes(self): # Added vol to vol.path failed_delete_vol['SystemCreationClassName'] = 'Symm_StorageSystem' failed_delete_vol.path = failed_delete_vol - failed_delete_vol.path.classname = \ - failed_delete_vol['CreationClassName'] + failed_delete_vol.path.classname = ( + failed_delete_vol['CreationClassName']) vols.append(failed_delete_vol) failed_vol = EMC_StorageVolume() @@ -1069,8 +1405,7 @@ def _enum_storagevolumes(self): # Added vol to vol.path failed_vol['SystemCreationClassName'] = 'Symm_StorageSystem' failed_vol.path = failed_vol - failed_vol.path.classname = \ - failed_vol['CreationClassName'] + failed_vol.path.classname = failed_vol['CreationClassName'] name_failed = {} name_failed['classname'] = 'Symm_StorageVolume' @@ -1084,6 +1419,31 @@ def _enum_storagevolumes(self): vols.append(failed_vol) + volumeHead = EMC_StorageVolume() + volumeHead.classname = 'Symm_StorageVolume' + blockSize = self.data.block_size + volumeHead['ConsumableBlocks'] = ( + self.data.metaHead_volume['ConsumableBlocks']) + volumeHead['BlockSize'] = blockSize + volumeHead['DeviceID'] = self.data.metaHead_volume['DeviceID'] + vols.append(volumeHead) + + metaMember1 = EMC_StorageVolume() + metaMember1.classname = 'Symm_StorageVolume' + metaMember1['ConsumableBlocks'] = ( + self.data.meta_volume1['ConsumableBlocks']) + metaMember1['BlockSize'] = blockSize + metaMember1['DeviceID'] = self.data.meta_volume1['DeviceID'] + vols.append(metaMember1) + + metaMember2 = EMC_StorageVolume() + metaMember2.classname = 'Symm_StorageVolume' + metaMember2['ConsumableBlocks'] = ( + self.data.meta_volume2['ConsumableBlocks']) + metaMember2['BlockSize'] = blockSize + metaMember2['DeviceID'] = self.data.meta_volume2['DeviceID'] + vols.append(metaMember2) + return vols def _enum_initiatorMaskingGroup(self): @@ -1128,6 +1488,14 @@ def _enum_hostedservice(self): hostedservice['CreationClassName'] = ( self.data.hostedservice_creationclass) hostedservice['SystemName'] = self.data.storage_system + hostedservice['Name'] = self.data.storage_system + hostedservice['TotalManagedSpace'] = self.data.totalmanagedspace_bits + hostedservice['EMCSubscribedCapacity'] = ( + self.data.subscribedcapacity_bits) + hostedservice['RemainingManagedSpace'] = ( + self.data.remainingmanagedspace_bits) + hostedservice['EMCMaxSubscriptionPercent'] = ( + self.data.maxsubscriptionpercent) hostedservices.append(hostedservice) return hostedservices @@ -1171,6 +1539,7 @@ def _enum_storagepool(self): storagepool['CreationClassName'] = self.data.storagepool_creationclass storagepool['InstanceID'] = self.data.storagepoolid storagepool['ElementName'] = 'gold' + storagepool['TotalManagedSpace'] = self.data.totalmanagedspace_bits storagepools.append(storagepool) return storagepools @@ -1197,7 +1566,7 @@ def _enum_storagesettings(self): storagesetting = {} storagesetting['CreationClassName'] = 'CIM_StoragePoolSetting' storagesetting['InstanceID'] = ('SYMMETRIX-+-000197200056-+-SBronze:' - 'NONE-+-F-+-0-+-SR-+-SRP_1') + 'DSS-+-F-+-0-+-SR-+-SRP_1') storagesettings.append(storagesetting) return storagesettings @@ -1205,7 +1574,7 @@ def _enum_targetMaskingGroup(self): targetMaskingGroups = [] targetMaskingGroup = {} targetMaskingGroup['CreationClassName'] = 'CIM_TargetMaskingGroup' - targetMaskingGroup['ElementName'] = 'myPortGroup' + targetMaskingGroup['ElementName'] = self.data.port_group targetMaskingGroups.append(targetMaskingGroup) return targetMaskingGroups @@ -1219,11 +1588,27 @@ def _enum_initMaskingGroup(self): def _enum_storagegroup(self): storagegroups = [] - storagegroup = {} - storagegroup['CreationClassName'] = ( + storagegroup1 = {} + storagegroup1['CreationClassName'] = ( + self.data.storagegroup_creationclass) + storagegroup1['ElementName'] = self.data.storagegroupname + storagegroups.append(storagegroup1) + storagegroup2 = {} + storagegroup2['CreationClassName'] = ( + self.data.storagegroup_creationclass) + storagegroup2['ElementName'] = self.data.defaultstoragegroupname + storagegroup2['SystemName'] = self.data.storage_system + storagegroups.append(storagegroup2) + storagegroup3 = {} + storagegroup3['CreationClassName'] = ( self.data.storagegroup_creationclass) - storagegroup['ElementName'] = self.data.storagegroupname - storagegroups.append(storagegroup) + storagegroup3['ElementName'] = 'OS-fakehost-SRP_1-Bronze-DSS-SG' + storagegroups.append(storagegroup3) + storagegroup4 = {} + storagegroup4['CreationClassName'] = ( + self.data.storagegroup_creationclass) + storagegroup4['ElementName'] = 'OS-SRP_1-Bronze-DSS-SG' + storagegroups.append(storagegroup4) return storagegroups def _enum_storagevolume(self): @@ -1246,6 +1631,7 @@ def _enum_storhdwids(self): hdwid = SE_StorageHardwareID() hdwid['CreationClassName'] = self.data.hardwareid_creationclass hdwid['StorageID'] = self.data.connector['wwpns'][0] + hdwid['InstanceID'] = "W-+-" + self.data.connector['wwpns'][0] hdwid.path = hdwid storhdwids.append(hdwid) @@ -1254,7 +1640,8 @@ def _enum_storhdwids(self): def _enum_storagesystems(self): storagesystems = [] storagesystem = {} - storagesystem['Name'] = 'SYMMETRIX+000195900551' + storagesystem['SystemName'] = self.data.storage_system + storagesystem['Name'] = self.data.storage_system storagesystems.append(storagesystem) return storagesystems @@ -1283,12 +1670,80 @@ def _enum_fcscsiendpoint(self): def _enum_maskingView(self): maskingViews = [] - maskingView = {} + maskingView = SYMM_LunMasking() maskingView['CreationClassName'] = 'Symm_LunMaskingView' - maskingView['ElementName'] = 'myMaskingView' + maskingView['ElementName'] = self.data.lunmaskctrl_name + + cimproperty = Fake_CIMProperty() + cimproperty.value = self.data.lunmaskctrl_name + properties = {u'ElementName': cimproperty} + maskingView.properties = properties + maskingViews.append(maskingView) return maskingViews + def _enum_portgroup(self): + portgroups = [] + portgroup = {} + portgroup['CreationClassName'] = ( + 'CIM_TargetMaskingGroup') + portgroup['ElementName'] = self.data.port_group + portgroups.append(portgroup) + return portgroups + + def _enum_metavolume(self): + return [] + + def _enum_storageSyncSvSv(self): + conn = FakeEcomConnection() + sourceVolume = {} + sourceVolume['CreationClassName'] = 'Symm_StorageVolume' + sourceVolume['DeviceID'] = self.data.test_volume['device_id'] + sourceInstanceName = conn.GetInstance(sourceVolume) + svInstances = [] + svInstance = {} + svInstance['SyncedElement'] = 'SyncedElement' + svInstance['SystemElement'] = sourceInstanceName + svInstance['CreationClassName'] = 'SE_StorageSynchronized_SV_SV' + svInstance['PercentSynced'] = 100 + svInstance['CopyState'] = self.data.UNSYNCHRONIZED + svInstances.append(svInstance) + return svInstances + + def _enum_sw_identity(self): + swIdentities = [] + swIdentity = {} + swIdentity['MajorVersion'] = self.data.majorVersion + swIdentity['MinorVersion'] = self.data.minorVersion + swIdentity['RevisionNumber'] = self.data.revNumber + swIdentities.append(swIdentity) + return swIdentities + + def _enum_ipprotocolendpoint(self): + ipprotocolendpoints = [] + ipprotocolendpoint = CIM_IPProtocolEndpoint() + ipprotocolendpoint['CreationClassName'] = 'CIM_IPProtocolEndpoint' + ipprotocolendpoint['SystemName'] = self.data.storage_system + classcimproperty = Fake_CIMProperty() + ipv4addresscimproperty = ( + classcimproperty.fake_getipv4address()) + properties = {u'IPv4Address': ipv4addresscimproperty} + ipprotocolendpoint.properties = properties + ipprotocolendpoint.path = ipprotocolendpoint + ipprotocolendpoints.append(ipprotocolendpoint) + iqnprotocolendpoint = CIM_IPProtocolEndpoint() + iqnprotocolendpoint['CreationClassName'] = ( + 'Symm_VirtualiSCSIProtocolEndpoint') + iqnprotocolendpoint['SystemName'] = self.data.storage_system + classcimproperty = Fake_CIMProperty() + iqncimproperty = ( + classcimproperty.fake_getiqn()) + properties = {u'Name': iqncimproperty} + iqnprotocolendpoint.properties = properties + iqnprotocolendpoint.path = iqnprotocolendpoint + ipprotocolendpoints.append(iqnprotocolendpoint) + return ipprotocolendpoints + def _default_enum(self): names = [] name = {} @@ -1305,36 +1760,93 @@ def setUp(self): self.tempdir = tempfile.mkdtemp() super(EMCVMAXISCSIDriverNoFastTestCase, self).setUp() self.config_file_path = None - self.config_file_1364232 = None self.create_fake_config_file_no_fast() self.addCleanup(self._cleanup) - - configuration = mock.Mock() - configuration.safe_get.return_value = 'ISCSINoFAST' - configuration.cinder_emc_config_file = self.config_file_path + configuration = conf.Configuration(None) + configuration.append_config_values = mock.Mock(return_value=0) configuration.config_group = 'ISCSINoFAST' - - self.stubs.Set(EMCVMAXISCSIDriver, 'smis_do_iscsi_discovery', - self.fake_do_iscsi_discovery) - self.stubs.Set(EMCVMAXCommon, '_get_ecom_connection', + configuration.cinder_emc_config_file = self.config_file_path + self.stubs.Set(configuration, 'safe_get', + self.fake_safe_get({'driver_use_ssl': + True, + 'volume_backend_name': + 'ISCSINoFAST'})) + self.stubs.Set(emc_vmax_common.EMCVMAXCommon, '_get_ecom_connection', self.fake_ecom_connection) instancename = FakeCIMInstanceName() - self.stubs.Set(EMCVMAXUtils, 'get_instance_name', + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'get_instance_name', instancename.fake_getinstancename) - self.stubs.Set(time, 'sleep', - self.fake_sleep) - - driver = EMCVMAXISCSIDriver(configuration=configuration) + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'isArrayV3', + self.fake_is_v3) + driver = emc_vmax_iscsi.EMCVMAXISCSIDriver(configuration=configuration) driver.db = FakeDB() self.driver = driver - self.driver.utils = EMCVMAXUtils(object) + self.driver.utils = emc_vmax_utils.EMCVMAXUtils(object) + + def fake_safe_get(self, values): + def _safe_get(key): + return values.get(key) + return _safe_get def create_fake_config_file_no_fast(self): - doc = Document() + doc = minidom.Document() + emc = doc.createElement("EMC") + doc.appendChild(emc) + doc = self.add_array_info(doc, emc) + filename = 'cinder_emc_config_ISCSINoFAST.xml' + self.config_file_path = self.tempdir + '/' + filename + + f = open(self.config_file_path, 'w') + doc.writexml(f) + f.close() + + def create_fake_config_file_no_fast_with_interval_retries(self): + + doc = minidom.Document() + emc = doc.createElement("EMC") + doc.appendChild(emc) + doc = self.add_array_info(doc, emc) + doc = self.add_interval_and_retries(doc, emc) + filename = 'cinder_emc_config_ISCSINoFAST_int_ret.xml' + config_file_path = self.tempdir + '/' + filename + + f = open(self.config_file_path, 'w') + doc.writexml(f) + f.close() + return config_file_path + + def create_fake_config_file_no_fast_with_interval(self): + + doc = minidom.Document() + emc = doc.createElement("EMC") + doc.appendChild(emc) + doc = self.add_array_info(doc, emc) + doc = self.add_interval_only(doc, emc) + filename = 'cinder_emc_config_ISCSINoFAST_int.xml' + config_file_path = self.tempdir + '/' + filename + + f = open(self.config_file_path, 'w') + doc.writexml(f) + f.close() + return config_file_path + + def create_fake_config_file_no_fast_with_retries(self): + + doc = minidom.Document() emc = doc.createElement("EMC") doc.appendChild(emc) + doc = self.add_array_info(doc, emc) + doc = self.add_retries_only(doc, emc) + filename = 'cinder_emc_config_ISCSINoFAST_ret.xml' + config_file_path = self.tempdir + '/' + filename + + f = open(self.config_file_path, 'w') + doc.writexml(f) + f.close() + return config_file_path + def add_array_info(self, doc, emc): array = doc.createElement("Array") arraytext = doc.createTextNode("1234567891011") emc.appendChild(array) @@ -1361,7 +1873,7 @@ def create_fake_config_file_no_fast(self): ecompassword.appendChild(ecompasswordtext) portgroup = doc.createElement("PortGroup") - portgrouptext = doc.createTextNode("myPortGroup") + portgrouptext = doc.createTextNode(self.data.port_group) portgroup.appendChild(portgrouptext) portgroups = doc.createElement("PortGroups") @@ -1374,7 +1886,7 @@ def create_fake_config_file_no_fast(self): pool.appendChild(pooltext) array = doc.createElement("Array") - arraytext = doc.createTextNode("0123456789") + arraytext = doc.createTextNode("1234567891011") emc.appendChild(array) array.appendChild(arraytext) @@ -1382,20 +1894,39 @@ def create_fake_config_file_no_fast(self): timeouttext = doc.createTextNode("0") emc.appendChild(timeout) timeout.appendChild(timeouttext) - - filename = 'cinder_emc_config_ISCSINoFAST.xml' - - self.config_file_path = self.tempdir + '/' + filename - - f = open(self.config_file_path, 'w') - doc.writexml(f) - f.close() + return doc + + def add_interval_and_retries(self, doc, emc): + interval = doc.createElement("Interval") + intervaltext = doc.createTextNode("5") + emc.appendChild(interval) + interval.appendChild(intervaltext) + + retries = doc.createElement("Retries") + retriestext = doc.createTextNode("40") + emc.appendChild(retries) + retries.appendChild(retriestext) + return doc + + def add_interval_only(self, doc, emc): + interval = doc.createElement("Interval") + intervaltext = doc.createTextNode("20") + emc.appendChild(interval) + interval.appendChild(intervaltext) + return doc + + def add_retries_only(self, doc, emc): + retries = doc.createElement("Retries") + retriestext = doc.createTextNode("70") + emc.appendChild(retries) + retries.appendChild(retriestext) + return doc # fix for https://bugs.launchpad.net/cinder/+bug/1364232 def create_fake_config_file_1364232(self): filename = 'cinder_emc_config_1364232.xml' - self.config_file_1364232 = self.tempdir + '/' + filename - text_file = open(self.config_file_1364232, "w") + config_file_1364232 = self.tempdir + '/' + filename + text_file = open(config_file_1364232, "w") text_file.write("\n\n" "10.10.10.10\n" "5988\n" @@ -1411,19 +1942,442 @@ def create_fake_config_file_1364232(self): "\nSILVER1\n" "") text_file.close() + return config_file_1364232 def fake_ecom_connection(self): conn = FakeEcomConnection() return conn - def fake_do_iscsi_discovery(self, volume): - output = [] - item = '10.10.0.50: 3260,1 iqn.1992-04.com.emc: 50000973f006dd80' - output.append(item) - return output + def fake_is_v3(self, conn, serialNumber): + return False + + def populate_masking_dict_setup(self): + extraSpecs = {'storagetype:pool': u'gold_pool', + 'volume_backend_name': 'GOLD_POOL_BE', + 'storagetype:array': u'1234567891011', + 'isV3': False, + 'portgroupname': u'OS-portgroup-PG', + 'storagetype:fastpolicy': u'GOLD'} + vol = {'SystemName': self.data.storage_system} + self.driver.common._find_lun = mock.Mock( + return_value=vol) + self.driver.common.utils.find_controller_configuration_service = ( + mock.Mock(return_value=None)) + return extraSpecs + + def test_populate_masking_dict_fast(self): + extraSpecs = self.populate_masking_dict_setup() + # If fast is enabled it will uniquely determine the SG and MV + # on the host along with the protocol(iSCSI) e.g. I + maskingViewDict = self.driver.common._populate_masking_dict( + self.data.test_volume, self.data.connector, extraSpecs) + self.assertEqual( + 'OS-fakehost-GOLD-FP-I-SG', maskingViewDict['sgGroupName']) + self.assertEqual( + 'OS-fakehost-GOLD-FP-I-MV', maskingViewDict['maskingViewName']) + + def test_populate_masking_dict_fast_more_than_14chars(self): + # If the length of the FAST policy name is greater than 14 chars + extraSpecs = self.populate_masking_dict_setup() + extraSpecs['storagetype:fastpolicy'] = 'GOLD_MORE_THAN_FOURTEEN_CHARS' + maskingViewDict = self.driver.common._populate_masking_dict( + self.data.test_volume, self.data.connector, extraSpecs) + self.assertEqual( + 'OS-fakehost-GOLD_MO__CHARS-FP-I-SG', + maskingViewDict['sgGroupName']) + self.assertEqual( + 'OS-fakehost-GOLD_MO__CHARS-FP-I-MV', + maskingViewDict['maskingViewName']) + + def test_populate_masking_dict_no_fast(self): + # If fast isn't enabled the pool will uniquely determine the SG and MV + # on the host along with the protocol(iSCSI) e.g. I + extraSpecs = self.populate_masking_dict_setup() + extraSpecs['storagetype:fastpolicy'] = None + maskingViewDict = self.driver.common._populate_masking_dict( + self.data.test_volume, self.data.connector, extraSpecs) + self.assertEqual( + 'OS-fakehost-gold_pool-I-SG', maskingViewDict['sgGroupName']) + self.assertEqual( + 'OS-fakehost-gold_pool-I-MV', maskingViewDict['maskingViewName']) + + def test_populate_masking_dict_fast_both_exceeding(self): + # If the length of the FAST policy name is greater than 14 chars and + # the length of the short host is more than 38 characters + extraSpecs = self.populate_masking_dict_setup() + connector = {'host': 'SHORT_HOST_MORE_THEN THIRTY_EIGHT_CHARACTERS'} + extraSpecs['storagetype:fastpolicy'] = ( + 'GOLD_MORE_THAN_FOURTEEN_CHARACTERS') + maskingViewDict = self.driver.common._populate_masking_dict( + self.data.test_volume, connector, extraSpecs) + self.assertLessEqual(len(maskingViewDict['sgGroupName']), 64) + self.assertLessEqual(len(maskingViewDict['maskingViewName']), 64) + + def test_populate_masking_dict_no_fast_both_exceeding(self): + # If the length of the FAST policy name is greater than 14 chars and + # the length of the short host is more than 38 characters + extraSpecs = self.populate_masking_dict_setup() + connector = {'host': 'SHORT_HOST_MORE_THEN THIRTY_EIGHT_CHARACTERS'} + extraSpecs['storagetype:pool'] = ( + 'GOLD_POOL_MORE_THAN_SIXTEEN_CHARACTERS') + extraSpecs['storagetype:fastpolicy'] = None + maskingViewDict = self.driver.common._populate_masking_dict( + self.data.test_volume, connector, extraSpecs) + self.assertLessEqual(len(maskingViewDict['sgGroupName']), 64) + self.assertLessEqual(len(maskingViewDict['maskingViewName']), 64) + + def test_populate_masking_dict_v3(self): + extraSpecs = {'storagetype:pool': u'SRP_1', + 'volume_backend_name': 'VMAX_ISCSI_BE', + 'storagetype:array': u'1234567891011', + 'isV3': True, + 'portgroupname': u'OS-portgroup-PG', + 'storagetype:slo': u'Diamond', + 'storagetype:workload': u'DSS'} + connector = {'host': 'fakehost'} + maskingViewDict = self.driver.common._populate_masking_dict( + self.data.test_volume, connector, extraSpecs) + self.assertEqual('OS-fakehost-SRP_1-Diamond-DSS-I-SG', + maskingViewDict['sgGroupName']) + self.assertEqual('OS-fakehost-SRP_1-Diamond-DSS-I-MV', + maskingViewDict['maskingViewName']) + + def test_filter_list(self): + portgroupnames = ['pg3', 'pg1', 'pg4', 'pg2'] + portgroupnames = ( + self.driver.common.utils._filter_list(portgroupnames)) + self.assertEqual(4, len(portgroupnames)) + self.assertEqual(['pg1', 'pg2', 'pg3', 'pg4'], sorted(portgroupnames)) + + portgroupnames = ['pg1'] + portgroupnames = ( + self.driver.common.utils._filter_list(portgroupnames)) + self.assertEqual(1, len(portgroupnames)) + self.assertEqual(['pg1'], portgroupnames) + + portgroupnames = ['only_pg', '', '', '', '', ''] + portgroupnames = ( + self.driver.common.utils._filter_list(portgroupnames)) + self.assertEqual(1, len(portgroupnames)) + self.assertEqual(['only_pg'], portgroupnames) + + def test_get_random_pg_from_list(self): + portGroupNames = ['pg1', 'pg2', 'pg3', 'pg4'] + portGroupName = ( + self.driver.common.utils._get_random_pg_from_list(portGroupNames)) + self.assertIn('pg', portGroupName) + + portGroupNames = ['pg1'] + portGroupName = ( + self.driver.common.utils._get_random_pg_from_list(portGroupNames)) + self.assertEqual('pg1', portGroupName) + + def test_get_random_portgroup(self): + # 4 portgroups + data = ("\n\n" + "" + "OS-PG1\n" + "OS-PG2\n" + "OS-PG3\n" + "OS-PG4\n" + "" + "") + dom = minidom.parseString(data) + portgroup = self.driver.common.utils._get_random_portgroup(dom) + self.assertIn('OS-PG', portgroup) + + # Duplicate portgroups + data = ("\n\n" + "" + "OS-PG1\n" + "OS-PG1\n" + "OS-PG1\n" + "OS-PG2\n" + "" + "") + dom = minidom.parseString(data) + portgroup = self.driver.common.utils._get_random_portgroup(dom) + self.assertIn('OS-PG', portgroup) + + def test_get_random_portgroup_exception(self): + # Missing PortGroup values + data = ("\n\n" + "" + "\n" + "\n" + "" + "") + dom = minidom.parseString(data) + self.assertRaises(exception.VolumeBackendAPIException, + self.driver.common.utils._get_random_portgroup, dom) + + # Missing portgroups + data = ("\n\n" + "" + "" + "") + dom = minidom.parseString(data) + self.assertRaises(exception.VolumeBackendAPIException, + self.driver.common.utils._get_random_portgroup, dom) - def fake_sleep(self, seconds): - return + def test_is_sync_complete(self): + conn = self.fake_ecom_connection() + syncname = SE_ConcreteJob() + syncname.classname = 'SE_StorageSynchronized_SV_SV' + syncname['CopyState'] = self.data.UNSYNCHRONIZED + issynched = self.driver.common.utils._is_sync_complete(conn, syncname) + self.assertFalse(issynched) + + def test_get_correct_port_group(self): + self.driver.common.conn = self.fake_ecom_connection() + maskingViewInstanceName = {'CreationClassName': 'Symm_LunMaskingView', + 'ElementName': 'OS-fakehost-gold-I-MV', + 'SystemName': 'SYMMETRIX+000195900551'} + deviceinfodict = {'controller': maskingViewInstanceName} + portgroupname = self.driver.common._get_correct_port_group( + deviceinfodict, self.data.storage_system) + self.assertEqual('OS-portgroup-PG', portgroupname) + + def test_generate_unique_trunc_pool(self): + pool_under_16_chars = 'pool_under_16' + pool1 = self.driver.utils.generate_unique_trunc_pool( + pool_under_16_chars) + self.assertEqual(pool_under_16_chars, pool1) + + pool_over_16_chars = ( + 'pool_over_16_pool_over_16') + # Should generate truncated string first 8 chars and + # last 7 chars + pool2 = self.driver.utils.generate_unique_trunc_pool( + pool_over_16_chars) + self.assertEqual('pool_ove_over_16', pool2) + + def test_generate_unique_trunc_host(self): + host_under_38_chars = 'host_under_38_chars' + host1 = self.driver.utils.generate_unique_trunc_host( + host_under_38_chars) + self.assertEqual(host_under_38_chars, host1) + + host_over_38_chars = ( + 'host_over_38_chars_host_over_38_chars_host_over_38_chars') + # Check that the same md5 value is retrieved from multiple calls + host2 = self.driver.utils.generate_unique_trunc_host( + host_over_38_chars) + host3 = self.driver.utils.generate_unique_trunc_host( + host_over_38_chars) + self.assertEqual(host2, host3) + + def test_find_ip_protocol_endpoints(self): + conn = self.fake_ecom_connection() + endpoint = self.driver.common._find_ip_protocol_endpoints( + conn, self.data.storage_system, self.data.port_group) + self.assertEqual('10.10.10.10', endpoint[0]['ip']) + + def test_find_device_number(self): + host = 'fakehost' + data = ( + self.driver.common.find_device_number(self.data.test_volume_v2, + host)) + self.assertEqual('OS-fakehost-MV', data['maskingview']) + + @mock.patch.object( + FakeEcomConnection, + 'ReferenceNames', + return_value=[]) + def test_find_device_number_false(self, mock_ref_name): + host = 'bogushost' + data = ( + self.driver.common.find_device_number(self.data.test_volume_v2, + host)) + self.assertFalse(data) + + def test_find_device_number_long_host(self): + # Long host name + host = 'myhost.mydomain.com' + data = ( + self.driver.common.find_device_number(self.data.test_volume_v2, + host)) + self.assertEqual('OS-myhost-MV', data['maskingview']) + + def test_find_device_number_short_name_over_38_chars(self): + # short name over 38 chars + host = 'myShortnameIsOverThirtyEightCharactersLong' + host = self.driver.common.utils.generate_unique_trunc_host(host) + amended = 'OS-' + host + '-MV' + v2_host_over_38 = self.data.test_volume_v2.copy() + # Pool aware scheduler enabled + v2_host_over_38['host'] = host + data = ( + self.driver.common.find_device_number(v2_host_over_38, + host)) + self.assertEqual(amended, data['maskingview']) + + def test_unbind_and_get_volume_from_storage_pool(self): + conn = self.fake_ecom_connection() + common = self.driver.common + common.utils.is_volume_bound_to_pool = mock.Mock( + return_value='False') + storageConfigService = ( + common.utils.find_storage_configuration_service( + conn, self.data.storage_system)) + volumeInstanceName = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + volumeName = "unbind-vol" + extraSpecs = {'volume_backend_name': 'GOLD_BE', + 'isV3': False} + volumeInstance = ( + common._unbind_and_get_volume_from_storage_pool( + conn, storageConfigService, + volumeInstanceName, volumeName, extraSpecs)) + self.assertEqual(self.data.storage_system, + volumeInstance['SystemName']) + self.assertEqual('1', volumeInstance['ElementName']) + + def test_create_hardware_ids(self): + conn = self.fake_ecom_connection() + connector = { + 'ip': '10.0.0.2', + 'initiator': self.data.iscsi_initiator, + 'host': 'fakehost'} + initiatorNames = ( + self.driver.common.masking._find_initiator_names(conn, connector)) + storageHardwareIDInstanceNames = ( + self.driver.common.masking._create_hardware_ids( + conn, initiatorNames, self.data.storage_system)) + self.assertEqual(self.data.iscsi_initiator, + storageHardwareIDInstanceNames[0]) + + def test_get_pool_instance_and_system_name(self): + conn = self.fake_ecom_connection() + # V2 - old '+' separator + storagesystem = {} + storagesystem['SystemName'] = self.data.storage_system + storagesystem['Name'] = self.data.storage_system + pools = conn.EnumerateInstanceNames("EMC_VirtualProvisioningPool") + poolname = 'gold' + poolinstancename, systemname = ( + self.driver.common.utils._get_pool_instance_and_system_name( + conn, pools, storagesystem, poolname)) + self.assertEqual(self.data.storage_system, systemname) + self.assertEqual(self.data.storagepoolid, + poolinstancename['InstanceID']) + # V3 - note: V2 can also have the '-+-' separator + storagesystem = {} + storagesystem['SystemName'] = self.data.storage_system_v3 + storagesystem['Name'] = self.data.storage_system_v3 + pools = conn.EnumerateInstanceNames('Symm_SRPStoragePool') + poolname = 'SRP_1' + poolinstancename, systemname = ( + self.driver.common.utils._get_pool_instance_and_system_name( + conn, pools, storagesystem, poolname)) + self.assertEqual(self.data.storage_system_v3, systemname) + self.assertEqual('SYMMETRIX-+-000197200056-+-SRP_1', + poolinstancename['InstanceID']) + # Invalid poolname + poolname = 'bogus' + poolinstancename, systemname = ( + self.driver.common.utils._get_pool_instance_and_system_name( + conn, pools, storagesystem, poolname)) + self.assertIsNone(poolinstancename) + self.assertEqual(self.data.storage_system_v3, systemname) + + def test_get_hardware_type(self): + iqn_initiator = 'iqn.1992-04.com.emc: 50000973f006dd80' + hardwaretypeid = ( + self.driver.utils._get_hardware_type(iqn_initiator)) + self.assertEqual(5, hardwaretypeid) + wwpn_initiator = '123456789012345' + hardwaretypeid = ( + self.driver.utils._get_hardware_type(wwpn_initiator)) + self.assertEqual(2, hardwaretypeid) + bogus_initiator = 'bogus' + hardwaretypeid = ( + self.driver.utils._get_hardware_type(bogus_initiator)) + self.assertEqual(0, hardwaretypeid) + + def test_check_if_rollback_action_for_masking_required(self): + conn = self.fake_ecom_connection() + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + extraSpecs = {'volume_backend_name': 'GOLD_BE', + 'isV3': False, + 'storagetype:fastpolicy': 'GOLD1'} + + vol = EMC_StorageVolume() + vol['name'] = self.data.test_volume['name'] + vol['CreationClassName'] = 'Symm_StorageVolume' + vol['ElementName'] = self.data.test_volume['id'] + vol['DeviceID'] = self.data.test_volume['device_id'] + vol['Id'] = self.data.test_volume['id'] + vol['SystemName'] = self.data.storage_system + vol['NumberOfBlocks'] = self.data.test_volume['NumberOfBlocks'] + vol['BlockSize'] = self.data.test_volume['BlockSize'] + + # Added vol to vol.path + vol['SystemCreationClassName'] = 'Symm_StorageSystem' + vol.path = vol + vol.path.classname = vol['CreationClassName'] + + rollbackDict = {} + rollbackDict['isV3'] = False + rollbackDict['defaultStorageGroupInstanceName'] = ( + self.data.default_storage_group) + rollbackDict['sgName'] = self.data.storagegroupname + rollbackDict['volumeName'] = 'vol1' + rollbackDict['fastPolicyName'] = 'GOLD1' + rollbackDict['volumeInstance'] = vol + rollbackDict['controllerConfigService'] = controllerConfigService + rollbackDict['extraSpecs'] = extraSpecs + # Path 1 - The volume is in another storage group that isn't the + # default storage group + expectedmessage = (_("V2 rollback - Volume in another storage " + "group besides default storage group.")) + message = ( + self.driver.common.masking. + _check_if_rollback_action_for_masking_required( + conn, rollbackDict)) + self.assertEqual(expectedmessage, message) + # Path 2 - The volume is not in any storage group + rollbackDict['sgName'] = 'sq_not_exist' + expectedmessage = (_("V2 rollback, volume is not in any storage " + "group.")) + message = ( + self.driver.common.masking. + _check_if_rollback_action_for_masking_required( + conn, rollbackDict)) + self.assertEqual(expectedmessage, message) + + def test_migrate_cleanup(self): + conn = self.fake_ecom_connection() + extraSpecs = {'volume_backend_name': 'GOLD_BE', + 'isV3': False, + 'storagetype:fastpolicy': 'GOLD1'} + + vol = EMC_StorageVolume() + vol['name'] = self.data.test_volume['name'] + vol['CreationClassName'] = 'Symm_StorageVolume' + vol['ElementName'] = self.data.test_volume['id'] + vol['DeviceID'] = self.data.test_volume['device_id'] + vol['Id'] = self.data.test_volume['id'] + vol['SystemName'] = self.data.storage_system + vol['NumberOfBlocks'] = self.data.test_volume['NumberOfBlocks'] + vol['BlockSize'] = self.data.test_volume['BlockSize'] + + # Added vol to vol.path + vol['SystemCreationClassName'] = 'Symm_StorageSystem' + vol.path = vol + vol.path.classname = vol['CreationClassName'] + # The volume is already belong to default storage group + return_to_default = self.driver.common._migrate_cleanup( + conn, vol, self.data.storage_system, 'GOLD1', + vol['name'], extraSpecs) + self.assertFalse(return_to_default) + # The volume does not belong to default storage group + return_to_default = self.driver.common._migrate_cleanup( + conn, vol, self.data.storage_system, 'BRONZE1', + vol['name'], extraSpecs) + self.assertTrue(return_to_default) def test_wait_for_job_complete(self): myjob = SE_ConcreteJob() @@ -1441,9 +2395,7 @@ def test_wait_for_job_complete(self): self.assertIsNone(rc) self.driver.utils._is_job_finished.assert_called_once_with( conn, myjob) - self.assertEqual( - True, - self.driver.utils._is_job_finished.return_value) + self.assertTrue(self.driver.utils._is_job_finished.return_value) self.driver.utils._is_job_finished.reset_mock() # Save the original state and restore it after this test @@ -1463,23 +2415,60 @@ def test_wait_for_sync(self): self.driver.utils._is_sync_complete = mock.Mock( return_value=True) rc = self.driver.utils.wait_for_sync(conn, mysync) - self.assertIsNone(rc) + self.assertIsNotNone(rc) self.driver.utils._is_sync_complete.assert_called_once_with( conn, mysync) - self.assertEqual( - True, - self.driver.utils._is_sync_complete.return_value) + self.assertTrue(self.driver.utils._is_sync_complete.return_value) self.driver.utils._is_sync_complete.reset_mock() # Save the original state and restore it after this test loopingcall_orig = loopingcall.FixedIntervalLoopingCall loopingcall.FixedIntervalLoopingCall = mock.Mock() rc = self.driver.utils.wait_for_sync(conn, mysync) - self.assertIsNone(rc) + self.assertIsNotNone(rc) + loopingcall.FixedIntervalLoopingCall.assert_called_once_with( + mock.ANY) + loopingcall.FixedIntervalLoopingCall.reset_mock() + loopingcall.FixedIntervalLoopingCall = loopingcall_orig + + def test_wait_for_sync_extra_specs(self): + mysync = 'fakesync' + conn = self.fake_ecom_connection() + file_name = ( + self.create_fake_config_file_no_fast_with_interval_retries()) + extraSpecs = {'volume_backend_name': 'ISCSINoFAST'} + pool = 'gold+1234567891011' + arrayInfo = self.driver.utils.parse_file_to_get_array_map( + self.config_file_path) + poolRec = self.driver.utils.extract_record(arrayInfo, pool) + extraSpecs = self.driver.common._set_v2_extra_specs(extraSpecs, + poolRec) + + self.driver.utils._is_sync_complete = mock.Mock( + return_value=True) + rc = self.driver.utils.wait_for_sync(conn, mysync, extraSpecs) + self.assertIsNotNone(rc) + self.driver.utils._is_sync_complete.assert_called_once_with( + conn, mysync) + self.assertTrue(self.driver.utils._is_sync_complete.return_value) + self.assertEqual(40, + self.driver.utils._get_max_job_retries(extraSpecs)) + self.assertEqual(5, + self.driver.utils._get_interval_in_secs(extraSpecs)) + self.driver.utils._is_sync_complete.reset_mock() + + # Save the original state and restore it after this test + loopingcall_orig = loopingcall.FixedIntervalLoopingCall + loopingcall.FixedIntervalLoopingCall = mock.Mock() + rc = self.driver.utils.wait_for_sync(conn, mysync) + self.assertIsNotNone(rc) loopingcall.FixedIntervalLoopingCall.assert_called_once_with( mock.ANY) loopingcall.FixedIntervalLoopingCall.reset_mock() loopingcall.FixedIntervalLoopingCall = loopingcall_orig + bExists = os.path.exists(file_name) + if bExists: + os.remove(file_name) # Bug 1395830: _find_lun throws exception when lun is not found. def test_find_lun(self): @@ -1509,19 +2498,19 @@ def test_find_lun(self): volume2 = EMC_StorageVolume() volume2['name'] = 'myVol' volume2['provider_location'] = six.text_type(provider_location2) - verify_orig = self.driver.common.utils.get_existing_instance - self.driver.common.utils.get_existing_instance = mock.Mock( + verify_orig = self.driver.common.conn.GetInstance + self.driver.common.conn.GetInstance = mock.Mock( return_value=None) findlun2 = self.driver.common._find_lun(volume2) # Not found. self.assertIsNone(findlun2) - instancename2 = self.driver.utils.get_instance_name( + self.driver.utils.get_instance_name( provider_location2['classname'], keybindings2) - self.driver.common.utils.get_existing_instance.assert_called_once_with( - self.driver.common.conn, instancename2) - self.driver.common.utils.get_existing_instance.reset_mock() - self.driver.common.utils.get_existing_instance = verify_orig + self.driver.common.conn.GetInstance.assert_called_once_with( + keybindings2) + self.driver.common.conn.GetInstance.reset_mock() + self.driver.common.conn.GetInstance = verify_orig keybindings3 = {'CreationClassName': u'Symm_StorageVolume', 'SystemName': u'SYMMETRIX+000195900551', @@ -1538,117 +2527,590 @@ def test_find_lun(self): self.driver.common.utils.process_exception_args, arg, instancename3) - def test_get_volume_stats_1364232(self): - self.create_fake_config_file_1364232() - self.assertEqual( - '000198700439', - self.driver.utils.parse_array_name_from_file( - self.config_file_1364232)) - self.assertEqual( - 'FC_SLVR1', - self.driver.utils.parse_pool_name_from_file( - self.config_file_1364232)) - self.assertEqual( - 'SILVER1', - self.driver.utils.parse_fast_policy_name_from_file( - self.config_file_1364232)) - self.assertTrue( - 'OS-PORTGROUP' in - self.driver.utils.parse_file_to_get_port_group_name( - self.config_file_1364232)) - bExists = os.path.exists(self.config_file_1364232) - if bExists: - os.remove(self.config_file_1364232) + # Bug 1403160 - make sure the masking view is cleanly deleted + def test_last_volume_delete_masking_view(self): + extraSpecs = {'volume_backend_name': 'ISCSINoFAST'} + conn = self.fake_ecom_connection() + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + + maskingViewInstanceName = ( + self.driver.common.masking._find_masking_view( + conn, self.data.lunmaskctrl_name, self.data.storage_system)) + + maskingViewName = conn.GetInstance( + maskingViewInstanceName)['ElementName'] + + # Deleting Masking View failed + self.assertRaises( + exception.VolumeBackendAPIException, + self.driver.common.masking._last_volume_delete_masking_view, + conn, controllerConfigService, maskingViewInstanceName, + maskingViewName, extraSpecs) + + # Deleting Masking view successful + self.driver.common.masking.utils.get_existing_instance = mock.Mock( + return_value=None) + self.driver.common.masking._last_volume_delete_masking_view( + conn, controllerConfigService, maskingViewInstanceName, + maskingViewName, extraSpecs) + + # Bug 1403160 - make sure the storage group is cleanly deleted + def test_remove_last_vol_and_delete_sg(self): + conn = self.fake_ecom_connection() + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + storageGroupName = self.data.storagegroupname + storageGroupInstanceName = ( + self.driver.utils.find_storage_masking_group( + conn, controllerConfigService, storageGroupName)) + + volumeInstanceName = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + volumeName = "1403160-Vol" + extraSpecs = {'volume_backend_name': 'GOLD_BE', + 'isV3': False} + + # Deleting Storage Group failed + self.assertRaises( + exception.VolumeBackendAPIException, + self.driver.common.masking._remove_last_vol_and_delete_sg, + conn, controllerConfigService, storageGroupInstanceName, + storageGroupName, volumeInstanceName, volumeName, extraSpecs) + + # Deleting Storage group successful + self.driver.common.masking.utils.get_existing_instance = mock.Mock( + return_value=None) + self.driver.common.masking._remove_last_vol_and_delete_sg( + conn, controllerConfigService, storageGroupInstanceName, + storageGroupName, volumeInstanceName, volumeName, extraSpecs) + + # Bug 1504192 - if the last volume is being unmapped and the masking view + # goes away, cleanup the initiators and associated initiator group. + def test_delete_initiators_from_initiator_group(self): + conn = self.fake_ecom_connection() + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + initiatorGroupName = self.data.initiatorgroup_name + initiatorGroupInstanceName = ( + self.driver.common.masking._get_initiator_group_from_masking_view( + conn, self.data.lunmaskctrl_name, self.data.storage_system)) + conn.InvokeMethod = mock.Mock(return_value=1) + # Deletion of initiators failed. + self.driver.common.masking._delete_initiators_from_initiator_group( + conn, controllerConfigService, initiatorGroupInstanceName, + initiatorGroupName) + conn.InvokeMethod = mock.Mock(return_value=0) + # Deletion of initiators successful. + self.driver.common.masking._delete_initiators_from_initiator_group( + conn, controllerConfigService, initiatorGroupInstanceName, + initiatorGroupName) + + # Bug 1504192 - if the last volume is being unmapped and the masking view + # goes away, cleanup the initiators and associated initiator group. + def test_last_volume_delete_initiator_group_exception(self): + extraSpecs = {'volume_backend_name': 'ISCSINoFAST'} + conn = self.fake_ecom_connection() + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + initiatorGroupInstanceName = ( + self.driver.common.masking._get_initiator_group_from_masking_view( + conn, self.data.lunmaskctrl_name, self.data.storage_system)) + job = { + 'Job': {'InstanceID': '9999', 'status': 'success', 'type': None}} + conn.InvokeMethod = mock.Mock(return_value=(4096, job)) + self.driver.common.masking.get_masking_views_by_initiator_group = ( + mock.Mock(return_value=[])) + self.driver.common.masking._delete_initiators_from_initiator_group = ( + mock.Mock(return_value=True)) + self.driver.common.masking.utils.wait_for_job_complete = ( + mock.Mock(return_value=(2, 'failure'))) + # Exception occurrs while deleting the initiator group. + self.assertRaises( + exception.VolumeBackendAPIException, + self.driver.common.masking._last_volume_delete_initiator_group, + conn, controllerConfigService, initiatorGroupInstanceName, + extraSpecs) + + # Bug 1504192 - if the last volume is being unmapped and the masking view + # goes away, cleanup the initiators and associated initiator group. + def test_last_volume_delete_initiator_group(self): + extraSpecs = {'volume_backend_name': 'ISCSINoFAST'} + conn = self.fake_ecom_connection() + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + initiatorGroupName = self.data.initiatorgroup_name + initiatorGroupInstanceName = ( + self.driver.common.masking._get_initiator_group_from_masking_view( + conn, self.data.lunmaskctrl_name, self.data.storage_system)) + self.assertEqual(initiatorGroupName, + conn.GetInstance( + initiatorGroupInstanceName)['ElementName']) + # masking view is associated with the initiator group and initiator + # group will not be deleted. + self.driver.common.masking._last_volume_delete_initiator_group( + conn, controllerConfigService, initiatorGroupInstanceName, + extraSpecs) + self.driver.common.masking.get_masking_views_by_initiator_group = ( + mock.Mock(return_value=[])) + self.driver.common.masking._delete_initiators_from_initiator_group = ( + mock.Mock(return_value=True)) + # No Masking view and initiators associated with the Initiator group + # and initiator group will be deleted. + self.driver.common.masking._last_volume_delete_initiator_group( + conn, controllerConfigService, initiatorGroupInstanceName, + extraSpecs) + job = { + 'Job': {'InstanceID': '9999', 'status': 'success', 'type': None}} + conn.InvokeMethod = mock.Mock(return_value=(4096, job)) + self.driver.common.masking.utils.wait_for_job_complete = ( + mock.Mock(return_value=(0, 'success'))) + # Deletion of initiator group is successful after waiting for job + # to complete. + self.driver.common.masking._last_volume_delete_initiator_group( + conn, controllerConfigService, initiatorGroupInstanceName, + extraSpecs) + + # Tests removal of last volume in a storage group V2 + def test_remove_and_reset_members(self): + extraSpecs = {'volume_backend_name': 'GOLD_BE', + 'isV3': False} + conn = self.fake_ecom_connection() + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + volumeInstanceName = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + volumeInstance = conn.GetInstance(volumeInstanceName) + volumeName = "Last-Vol" + self.driver.common.masking.get_devices_from_storage_group = mock.Mock( + return_value=['one_value']) + self.driver.common.masking.utils.get_existing_instance = mock.Mock( + return_value=None) + + self.driver.common.masking.remove_and_reset_members( + conn, controllerConfigService, volumeInstance, + volumeName, extraSpecs) + + # Bug 1393555 - masking view has been deleted by another process. + def test_find_maskingview(self): + conn = self.fake_ecom_connection() + foundMaskingViewInstanceName = ( + self.driver.common.masking._find_masking_view( + conn, self.data.lunmaskctrl_name, self.data.storage_system)) + # The masking view has been found. + self.assertEqual( + self.data.lunmaskctrl_name, + conn.GetInstance(foundMaskingViewInstanceName)['ElementName']) + + self.driver.common.masking.utils.get_existing_instance = mock.Mock( + return_value=None) + foundMaskingViewInstanceName2 = ( + self.driver.common.masking._find_masking_view( + conn, self.data.lunmaskctrl_name, self.data.storage_system)) + # The masking view has not been found. + self.assertIsNone(foundMaskingViewInstanceName2) + + # Bug 1393555 - port group has been deleted by another process. + def test_find_portgroup(self): + conn = self.fake_ecom_connection() + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + + foundPortGroupInstanceName = ( + self.driver.common.masking.find_port_group( + conn, controllerConfigService, self.data.port_group)) + # The port group has been found. + self.assertEqual( + self.data.port_group, + conn.GetInstance(foundPortGroupInstanceName)['ElementName']) + + self.driver.common.masking.utils.get_existing_instance = mock.Mock( + return_value=None) + foundPortGroupInstanceName2 = ( + self.driver.common.masking.find_port_group( + conn, controllerConfigService, self.data.port_group)) + # The port group has not been found as it has been deleted + # externally or by another thread. + self.assertIsNone(foundPortGroupInstanceName2) + + # Bug 1393555 - storage group has been deleted by another process. + def test_get_storage_group_from_masking_view(self): + conn = self.fake_ecom_connection() + foundStorageGroupInstanceName = ( + self.driver.common.masking._get_storage_group_from_masking_view( + conn, self.data.lunmaskctrl_name, self.data.storage_system)) + # The storage group has been found. + self.assertEqual( + self.data.storagegroupname, + conn.GetInstance(foundStorageGroupInstanceName)['ElementName']) + + self.driver.common.masking.utils.get_existing_instance = mock.Mock( + return_value=None) + foundStorageGroupInstanceName2 = ( + self.driver.common.masking._get_storage_group_from_masking_view( + conn, self.data.lunmaskctrl_name, self.data.storage_system)) + # The storage group has not been found as it has been deleted + # externally or by another thread. + self.assertIsNone(foundStorageGroupInstanceName2) + + # Bug 1393555 - initiator group has been deleted by another process. + def test_get_initiator_group_from_masking_view(self): + conn = self.fake_ecom_connection() + foundInitiatorGroupInstanceName = ( + self.driver.common.masking._get_initiator_group_from_masking_view( + conn, self.data.lunmaskctrl_name, self.data.storage_system)) + # The initiator group has been found. + self.assertEqual( + self.data.initiatorgroup_name, + conn.GetInstance(foundInitiatorGroupInstanceName)['ElementName']) + + self.driver.common.masking.utils.get_existing_instance = mock.Mock( + return_value=None) + foundInitiatorGroupInstanceName2 = ( + self.driver.common.masking._get_storage_group_from_masking_view( + conn, self.data.lunmaskctrl_name, self.data.storage_system)) + # The initiator group has not been found as it has been deleted + # externally or by another thread. + self.assertIsNone(foundInitiatorGroupInstanceName2) + + # Bug 1393555 - port group has been deleted by another process. + def test_get_port_group_from_masking_view(self): + conn = self.fake_ecom_connection() + foundPortGroupInstanceName = ( + self.driver.common.masking._get_port_group_from_masking_view( + conn, self.data.lunmaskctrl_name, self.data.storage_system)) + # The port group has been found. + self.assertEqual( + self.data.port_group, + conn.GetInstance(foundPortGroupInstanceName)['ElementName']) + + self.driver.common.masking.utils.get_existing_instance = mock.Mock( + return_value=None) + foundPortGroupInstanceName2 = ( + self.driver.common.masking._get_port_group_from_masking_view( + conn, self.data.lunmaskctrl_name, self.data.storage_system)) + # The port group has not been found as it has been deleted + # externally or by another thread. + self.assertIsNone(foundPortGroupInstanceName2) + + # Bug 1393555 - initiator group has been deleted by another process. + def test_find_initiator_group(self): + conn = self.fake_ecom_connection() + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + + foundInitiatorGroupInstanceName = ( + self.driver.common.masking._find_initiator_masking_group( + conn, controllerConfigService, self.data.initiatorNames)) + # The initiator group has been found. + self.assertEqual( + self.data.initiatorgroup_name, + conn.GetInstance(foundInitiatorGroupInstanceName)['ElementName']) + + self.driver.common.masking.utils.get_existing_instance = mock.Mock( + return_value=None) + foundInitiatorGroupInstanceName2 = ( + self.driver.common.masking._find_initiator_masking_group( + conn, controllerConfigService, self.data.initiatorNames)) + # The initiator group has not been found as it has been deleted + # externally or by another thread. + self.assertIsNone(foundInitiatorGroupInstanceName2) + + # Bug 1393555 - hardware id has been deleted by another process. + def test_get_storage_hardware_id_instance_names(self): + conn = self.fake_ecom_connection() + foundHardwareIdInstanceNames = ( + self.driver.common.masking._get_storage_hardware_id_instance_names( + conn, self.data.initiatorNames, self.data.storage_system)) + # The hardware id list has been found. + self.assertEqual( + '123456789012345', + conn.GetInstance( + foundHardwareIdInstanceNames[0])['StorageID']) + + self.driver.common.masking.utils.get_existing_instance = mock.Mock( + return_value=None) + foundHardwareIdInstanceNames2 = ( + self.driver.common.masking._get_storage_hardware_id_instance_names( + conn, self.data.initiatorNames, self.data.storage_system)) + # The hardware id list has not been found as it has been removed + # externally. + self.assertEqual(0, len(foundHardwareIdInstanceNames2)) + + # Bug 1393555 - controller has been deleted by another process. + def test_find_lunmasking_scsi_protocol_controller(self): + self.driver.common.conn = self.fake_ecom_connection() + foundControllerInstanceName = ( + self.driver.common._find_lunmasking_scsi_protocol_controller( + self.data.storage_system, self.data.connector)) + # The controller has been found. + self.assertEqual( + 'OS-fakehost-gold-I-MV', + self.driver.common.conn.GetInstance( + foundControllerInstanceName)['ElementName']) + + self.driver.common.utils.get_existing_instance = mock.Mock( + return_value=None) + foundControllerInstanceName2 = ( + self.driver.common._find_lunmasking_scsi_protocol_controller( + self.data.storage_system, self.data.connector)) + # The controller has not been found as it has been removed + # externally. + self.assertIsNone(foundControllerInstanceName2) + + # Bug 1393555 - storage group has been deleted by another process. + def test_get_policy_default_storage_group(self): + conn = self.fake_ecom_connection() + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + + foundStorageMaskingGroupInstanceName = ( + self.driver.common.fast.get_policy_default_storage_group( + conn, controllerConfigService, 'OS_default')) + # The storage group has been found. + self.assertEqual( + 'OS_default_GOLD1_SG', + conn.GetInstance( + foundStorageMaskingGroupInstanceName)['ElementName']) + + self.driver.common.fast.utils.get_existing_instance = mock.Mock( + return_value=None) + foundStorageMaskingGroupInstanceName2 = ( + self.driver.common.fast.get_policy_default_storage_group( + conn, controllerConfigService, 'OS_default')) + # The storage group has not been found as it has been removed + # externally. + self.assertIsNone(foundStorageMaskingGroupInstanceName2) + + # Bug 1393555 - policy has been deleted by another process. + def test_get_capacities_associated_to_policy(self): + conn = self.fake_ecom_connection() + (total_capacity_gb, free_capacity_gb, provisioned_capacity_gb, + array_max_over_subscription) = ( + self.driver.common.fast.get_capacities_associated_to_policy( + conn, self.data.storage_system, self.data.policyrule)) + # The capacities associated to the policy have been found. + self.assertEqual(self.data.totalmanagedspace_gbs, total_capacity_gb) + self.assertEqual(self.data.remainingmanagedspace_gbs, free_capacity_gb) + + self.driver.common.fast.utils.get_existing_instance = mock.Mock( + return_value=None) + (total_capacity_gb_2, free_capacity_gb_2, provisioned_capacity_gb_2, + array_max_over_subscription_2) = ( + self.driver.common.fast.get_capacities_associated_to_policy( + conn, self.data.storage_system, self.data.policyrule)) + # The capacities have not been found as the policy has been + # removed externally. + self.assertEqual(0, total_capacity_gb_2) + self.assertEqual(0, free_capacity_gb_2) + self.assertEqual(0, provisioned_capacity_gb_2) + + # Bug 1393555 - storage group has been deleted by another process. + def test_find_storage_masking_group(self): + conn = self.fake_ecom_connection() + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + + foundStorageMaskingGroupInstanceName = ( + self.driver.common.utils.find_storage_masking_group( + conn, controllerConfigService, self.data.storagegroupname)) + # The storage group has been found. + self.assertEqual( + self.data.storagegroupname, + conn.GetInstance( + foundStorageMaskingGroupInstanceName)['ElementName']) + + self.driver.common.utils.get_existing_instance = mock.Mock( + return_value=None) + foundStorageMaskingGroupInstanceName2 = ( + self.driver.common.utils.find_storage_masking_group( + conn, controllerConfigService, self.data.storagegroupname)) + # The storage group has not been found as it has been removed + # externally. + self.assertIsNone(foundStorageMaskingGroupInstanceName2) + + # Bug 1393555 - pool has been deleted by another process. + def test_get_pool_by_name(self): + conn = self.fake_ecom_connection() + + foundPoolInstanceName = self.driver.common.utils.get_pool_by_name( + conn, self.data.poolname, self.data.storage_system) + # The pool has been found. + self.assertEqual( + self.data.poolname, + conn.GetInstance(foundPoolInstanceName)['ElementName']) + + self.driver.common.utils.get_existing_instance = mock.Mock( + return_value=None) + foundPoolInstanceName2 = self.driver.common.utils.get_pool_by_name( + conn, self.data.poolname, self.data.storage_system) + # The pool has not been found as it has been removed externally. + self.assertIsNone(foundPoolInstanceName2) + + def test_get_volume_stats_1364232(self): + file_name = self.create_fake_config_file_1364232() + + arrayInfo = self.driver.utils.parse_file_to_get_array_map(file_name) + self.assertEqual( + '000198700439', arrayInfo[0]['SerialNumber']) + self.assertEqual( + 'FC_SLVR1', arrayInfo[0]['PoolName']) + self.assertEqual( + 'SILVER1', arrayInfo[0]['FastPolicy']) + self.assertIn('OS-PORTGROUP', arrayInfo[0]['PortGroup']) + bExists = os.path.exists(file_name) + if bExists: + os.remove(file_name) + + def test_intervals_and_retries_override( + self): + file_name = ( + self.create_fake_config_file_no_fast_with_interval_retries()) + extraSpecs = {'volume_backend_name': 'ISCSINoFAST'} + pool = 'gold+1234567891011' + arrayInfo = self.driver.utils.parse_file_to_get_array_map( + self.config_file_path) + poolRec = self.driver.utils.extract_record(arrayInfo, pool) + extraSpecs = self.driver.common._set_v2_extra_specs(extraSpecs, + poolRec) + self.assertEqual(40, + self.driver.utils._get_max_job_retries(extraSpecs)) + self.assertEqual(5, + self.driver.utils._get_interval_in_secs(extraSpecs)) + + bExists = os.path.exists(file_name) + if bExists: + os.remove(file_name) + + def test_intervals_and_retries_default(self): + extraSpecs = {'volume_backend_name': 'ISCSINoFAST'} + pool = 'gold+1234567891011' + arrayInfo = self.driver.utils.parse_file_to_get_array_map( + self.config_file_path) + poolRec = self.driver.utils.extract_record(arrayInfo, pool) + extraSpecs = self.driver.common._set_v2_extra_specs(extraSpecs, + poolRec) + self.assertEqual(60, + self.driver.utils._get_max_job_retries(extraSpecs)) + self.assertEqual(10, + self.driver.utils._get_interval_in_secs(extraSpecs)) + + def test_interval_only(self): + extraSpecs = {'volume_backend_name': 'ISCSINoFAST'} + file_name = self.create_fake_config_file_no_fast_with_interval() + pool = 'gold+1234567891011' + arrayInfo = self.driver.utils.parse_file_to_get_array_map( + self.config_file_path) + poolRec = self.driver.utils.extract_record(arrayInfo, pool) + extraSpecs = self.driver.common._set_v2_extra_specs(extraSpecs, + poolRec) + self.assertEqual(60, + self.driver.utils._get_max_job_retries(extraSpecs)) + self.assertEqual(20, + self.driver.utils._get_interval_in_secs(extraSpecs)) + + bExists = os.path.exists(file_name) + if bExists: + os.remove(file_name) + + def test_retries_only(self): + extraSpecs = {'volume_backend_name': 'ISCSINoFAST'} + file_name = self.create_fake_config_file_no_fast_with_retries() + pool = 'gold+1234567891011' + arrayInfo = self.driver.utils.parse_file_to_get_array_map( + self.config_file_path) + poolRec = self.driver.utils.extract_record(arrayInfo, pool) + extraSpecs = self.driver.common._set_v2_extra_specs(extraSpecs, + poolRec) + self.assertEqual(70, + self.driver.utils._get_max_job_retries(extraSpecs)) + self.assertEqual(10, + self.driver.utils._get_interval_in_secs(extraSpecs)) + + bExists = os.path.exists(file_name) + if bExists: + os.remove(file_name) @mock.patch.object( - EMCVMAXUtils, - 'find_storageSystem', - return_value=None) - @mock.patch.object( - EMCVMAXFast, - 'is_tiering_policy_enabled', + emc_vmax_utils.EMCVMAXUtils, + 'isArrayV3', return_value=False) @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'get_pool_capacities', - return_value=(1234, 1200)) - @mock.patch.object( - EMCVMAXUtils, - 'parse_array_name_from_file', - return_value="123456789") + return_value=(1234, 1200, 1200, 1)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', + emc_vmax_fast.EMCVMAXFast, + 'is_tiering_policy_enabled', return_value=False) - def test_get_volume_stats_no_fast(self, mock_storage_system, + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'find_storageSystem', + return_value=None) + def test_get_volume_stats_no_fast(self, + mock_storage_system, mock_is_fast_enabled, - mock_capacity, mock_array, + mock_capacity, mock_is_v3): self.driver.get_volume_stats(True) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) def test_create_volume_no_fast_success( - self, _mock_volume_type, mock_storage_system, mock_is_v3): + self, _mock_volume_type, mock_storage_system): self.driver.create_volume(self.data.test_volume_v2) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'storagetype: stripedmetacount': '4', 'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) def test_create_volume_no_fast_striped_success( - self, _mock_volume_type, mock_storage_system, mock_is_v3): + self, _mock_volume_type, mock_storage_system): self.driver.create_volume(self.data.test_volume_v2) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_create_volume_in_CG_no_fast_success( - self, _mock_volume_type, mock_storage_system, mock_is_v3): - self.driver.create_volume(self.data.test_volume_CG) - @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'ISCSINoFAST'}) + def test_create_volume_in_CG_no_fast_success( + self, _mock_volume_type, mock_storage_system): + self.driver.create_volume(self.data.test_volume_CG) + @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) def test_delete_volume_no_fast_success( - self, _mock_volume_type, mock_storage_system, mock_is_v3): + self, _mock_volume_type, mock_storage_system): self.driver.delete_volume(self.data.test_volume) def test_create_volume_no_fast_failed(self): @@ -1660,12 +3122,7 @@ def test_create_volume_no_fast_failed(self): volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_delete_volume_no_fast_notfound(self, _mock_volume_type, - mock_is_v3): + def test_delete_volume_no_fast_notfound(self, _mock_volume_type): notfound_delete_vol = {} notfound_delete_vol['name'] = 'notfound_delete_vol' notfound_delete_vol['id'] = '10' @@ -1675,208 +3132,197 @@ def test_delete_volume_no_fast_notfound(self, _mock_volume_type, notfound_delete_vol['SystemCreationClassName'] = 'Symm_StorageSystem' notfound_delete_vol['volume_type_id'] = 'abc' notfound_delete_vol['provider_location'] = None + notfound_delete_vol['host'] = self.data.fake_host name = {} name['classname'] = 'Symm_StorageVolume' keys = {} keys['CreationClassName'] = notfound_delete_vol['CreationClassName'] keys['SystemName'] = notfound_delete_vol['SystemName'] keys['DeviceID'] = notfound_delete_vol['DeviceID'] - keys['SystemCreationClassName'] = \ - notfound_delete_vol['SystemCreationClassName'] + keys['SystemCreationClassName'] = ( + notfound_delete_vol['SystemCreationClassName']) name['keybindings'] = keys self.driver.delete_volume(notfound_delete_vol) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) def test_delete_volume_failed( - self, _mock_volume_type, mock_storage_system, mock_is_v3): + self, _mock_volume_type, mock_storage_system): self.driver.create_volume(self.data.failed_delete_vol) self.assertRaises(exception.VolumeBackendAPIException, self.driver.delete_volume, self.data.failed_delete_vol) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXMasking, - '_wrap_get_storage_group_from_volume', - return_value=None) + emc_vmax_common.EMCVMAXCommon, + '_is_same_host', + return_value=True) @mock.patch.object( - EMCVMAXCommon, - '_wrap_find_device_number', + emc_vmax_common.EMCVMAXCommon, + 'find_device_number', return_value={'hostlunid': 1, 'storagesystem': EMCVMAXCommonData.storage_system}) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + emc_vmax_masking.EMCVMAXMasking, + '_wrap_get_storage_group_from_volume', + return_value=None) @mock.patch.object( - EMCVMAXCommon, - '_is_same_host', - return_value=True) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) def test_already_mapped_no_fast_success( - self, mock_is_same_host, mock_is_v3, mock_wrap_device, - mock_wrap_group, _mock_volume_type): + self, _mock_volume_type, mock_wrap_group, mock_wrap_device, + mock_is_same_host): + self.driver.common._get_correct_port_group = mock.Mock( + return_value=self.data.port_group) self.driver.initialize_connection(self.data.test_volume, self.data.connector) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXMasking, - '_wrap_get_storage_group_from_volume', + emc_vmax_masking.EMCVMAXMasking, + '_check_adding_volume_to_storage_group', return_value=None) @mock.patch.object( - EMCVMAXCommon, - '_wrap_find_device_number', - return_value={'storagesystem': EMCVMAXCommonData.storage_system}) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'find_storage_masking_group', return_value='value') @mock.patch.object( - EMCVMAXMasking, - '_check_adding_volume_to_storage_group', + emc_vmax_masking.EMCVMAXMasking, + '_wrap_get_storage_group_from_volume', return_value=None) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) def test_map_new_masking_view_no_fast_success( - self, mock_check, mock_storage_group, mock_is_v3, - mock_wrap_device, mock_wrap_group, mock_volume_type): + self, _mock_volume_type, mock_wrap_group, + mock_storage_group, mock_add_volume): + self.driver.common._wrap_find_device_number = mock.Mock( + return_value={}) self.driver.initialize_connection(self.data.test_volume, self.data.connector) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_is_same_host', + return_value=False) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + 'find_device_number', + return_value={'hostlunid': 1, + 'storagesystem': EMCVMAXCommonData.storage_system}) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXMasking, - '_wrap_get_storage_group_from_volume', - return_value=None) - @mock.patch.object( - EMCVMAXCommon, - '_wrap_find_device_number', - return_value={'storagesystem': EMCVMAXCommonData.storage_system}) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - @mock.patch.object( - EMCVMAXMasking, - '_find_masking_view', + def test_map_live_migration_no_fast_success(self, + _mock_volume_type, + mock_wrap_device, + mock_same_host): + emc_vmax_utils.LIVE_MIGRATION_FILE = (self.tempdir + + '/livemigrationarray') + extraSpecs = self.data.extra_specs + rollback_dict = self.driver.common._populate_masking_dict( + self.data.test_volume, self.data.connector, extraSpecs) + with mock.patch.object(self.driver.common.masking, + 'setup_masking_view', + return_value=rollback_dict): + self.driver.initialize_connection(self.data.test_volume, + self.data.connector) + + @mock.patch.object( + emc_vmax_masking.EMCVMAXMasking, + '_get_initiator_group_from_masking_view', return_value='value') @mock.patch.object( - EMCVMAXMasking, + emc_vmax_masking.EMCVMAXMasking, '_find_initiator_masking_group', return_value='value') @mock.patch.object( - EMCVMAXMasking, - '_get_initiator_group_from_masking_view', + emc_vmax_masking.EMCVMAXMasking, + '_find_masking_view', return_value='value') + @mock.patch.object( + emc_vmax_masking.EMCVMAXMasking, + '_wrap_get_storage_group_from_volume', + return_value=None) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) def test_map_existing_masking_view_no_fast_success( - self, mock_ig_from_mv, mock_initiator_group, mock_masking_view, - mock_is_v3, mock_wrap_device, mock_wrap_group, mock_volume_type): + self, _mock_volume_type, mock_wrap_group, mock_storage_group, + mock_initiator_group, mock_ig_from_mv): self.driver.initialize_connection(self.data.test_volume, self.data.connector) @mock.patch.object( - EMCVMAXMasking, - '_wrap_get_storage_group_from_volume', - return_value=None) - @mock.patch.object( - EMCVMAXCommon, - '_wrap_find_device_number', + emc_vmax_common.EMCVMAXCommon, + 'find_device_number', return_value={'storagesystem': EMCVMAXCommonData.storage_system}) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_map_no_fast_failed(self, mock_wrap_group, mock_wrap_device, - mock_is_v3): + emc_vmax_masking.EMCVMAXMasking, + '_wrap_get_storage_group_from_volume', + return_value=None) + def test_map_no_fast_failed(self, mock_wrap_group, mock_wrap_device): self.assertRaises(exception.VolumeBackendAPIException, self.driver.initialize_connection, self.data.test_volume, self.data.connector) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXUtils, - 'find_storage_masking_group', - return_value=EMCVMAXCommonData.storagegroupname) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + emc_vmax_masking.EMCVMAXMasking, + 'get_initiator_group_from_masking_view', + return_value='myInitGroup') @mock.patch.object( - EMCVMAXMasking, + emc_vmax_masking.EMCVMAXMasking, '_find_initiator_masking_group', return_value='myInitGroup') @mock.patch.object( - EMCVMAXMasking, - 'get_initiator_group_from_masking_view', - return_value='myInitGroup') + emc_vmax_utils.EMCVMAXUtils, + 'find_storage_masking_group', + return_value=EMCVMAXCommonData.storagegroupname) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) def test_detach_no_fast_success( - self, mock_volume_type, mock_storage_group, mock_is_v3, + self, mock_volume_type, mock_storage_group, mock_ig, mock_igc): - self.driver.terminate_connection( self.data.test_volume, self.data.connector) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'get_volume_size', return_value='2147483648') @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) def test_extend_volume_no_fast_success( - self, _mock_volume_type, mock_volume_size, - mock_is_v3): + self, _mock_volume_type, mock_volume_size): newSize = '2' self.driver.extend_volume(self.data.test_volume, newSize) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'check_if_volume_is_extendable', + return_value='False') @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'storagetype: stripedmetacount': '4', 'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXUtils, - 'check_if_volume_is_extendable', - return_value='False') - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) def test_extend_volume_striped_no_fast_failed( - self, _mock_volume_type, _mock_is_extendable, - mock_is_v3): + self, _mock_volume_type, _mock_is_extendable): newSize = '2' self.assertRaises(exception.VolumeBackendAPIException, self.driver.extend_volume, @@ -1884,38 +3330,34 @@ def test_extend_volume_striped_no_fast_failed( newSize) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSINoFAST'}) + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'get_meta_members_capacity_in_byte', + return_value=[1234567, 7654321]) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'get_volume_meta_head', + return_value=[EMCVMAXCommonData.test_volume]) @mock.patch.object( FakeDB, 'volume_get', return_value=EMCVMAXCommonData.test_source_volume) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - @mock.patch.object( - EMCVMAXUtils, - 'get_volume_meta_head', - return_value=[EMCVMAXCommonData.test_volume]) - @mock.patch.object( - EMCVMAXUtils, - 'get_meta_members_capacity_in_bit', - return_value=[1234567, 7654321]) - @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) def test_create_snapshot_different_sizes_meta_no_fast_success( - self, mock_volume_type, mock_volume, mock_is_v3, + self, mock_volume_type, mock_volume, mock_meta, mock_size, mock_pool): self.data.test_volume['volume_name'] = "vmax-1234567" common = self.driver.common volumeDict = {'classname': u'Symm_StorageVolume', 'keybindings': EMCVMAXCommonData.keybindings} common.provision.create_volume_from_pool = ( - mock.Mock(return_value=(volumeDict, 0L))) + mock.Mock(return_value=(volumeDict, 0))) common.provision.get_volume_dict_from_job = ( mock.Mock(return_value=volumeDict)) self.driver.create_snapshot(self.data.test_volume) @@ -1927,28 +3369,23 @@ def test_create_snapshot_no_fast_failed(self): self.data.test_volume) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXUtils, - 'find_sync_sv_by_target', - return_value=(None, None)) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + emc_vmax_utils.EMCVMAXUtils, + 'get_meta_members_capacity_in_byte', + return_value=[1234567]) @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'get_volume_meta_head', return_value=[EMCVMAXCommonData.test_volume]) @mock.patch.object( - EMCVMAXUtils, - 'get_meta_members_capacity_in_bit', - return_value=[1234567]) + emc_vmax_utils.EMCVMAXUtils, + 'find_sync_sv_by_target', + return_value=(None, None)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) def test_create_volume_from_same_size_meta_snapshot( - self, mock_volume_type, mock_sync_sv, mock_is_v3, - mock_meta, mock_size): + self, mock_volume_type, mock_sync_sv, mock_meta, mock_size): self.data.test_volume['volume_name'] = "vmax-1234567" self.driver.create_volume_from_snapshot( self.data.test_volume, self.data.test_volume) @@ -1961,182 +3398,472 @@ def test_create_volume_from_snapshot_no_fast_failed(self): self.data.test_volume) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - FakeDB, - 'volume_get', - return_value=EMCVMAXCommonData.test_source_volume) + emc_vmax_utils.EMCVMAXUtils, + 'get_volume_meta_head', + return_value=None) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_find_storage_sync_sv_sv', return_value=(None, None)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + FakeDB, + 'volume_get', + return_value=EMCVMAXCommonData.test_source_volume) @mock.patch.object( - EMCVMAXUtils, - 'get_volume_meta_head', - return_value=None) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) def test_create_clone_simple_volume_no_fast_success( self, mock_volume_type, mock_volume, mock_sync_sv, - mock_is_v3, mock_simple_volume): + mock_simple_volume): self.data.test_volume['volume_name'] = "vmax-1234567" self.driver.create_cloned_volume(self.data.test_volume, EMCVMAXCommonData.test_source_volume) - def test_create_clone_no_fast_failed(self): + # Bug https://bugs.launchpad.net/cinder/+bug/1440154 + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'get_volume_meta_head', + return_value=[EMCVMAXCommonData.test_volume]) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'get_meta_members_capacity_in_byte', + return_value=[1234567, 7654321]) + @mock.patch.object( + FakeDB, + 'volume_get', + return_value=EMCVMAXCommonData.test_source_volume) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) + @mock.patch.object( + emc_vmax_provision.EMCVMAXProvision, + 'create_element_replica') + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'find_sync_sv_by_target', + return_value=(None, None)) + def test_create_clone_assert_clean_up_target_volume( + self, mock_sync, mock_create_replica, mock_volume_type, + mock_volume, mock_capacities, mock_pool, mock_meta_volume): self.data.test_volume['volume_name'] = "vmax-1234567" + e = exception.VolumeBackendAPIException('CreateElementReplica Ex') + common = self.driver.common + common._delete_from_pool = mock.Mock(return_value=0) + conn = self.fake_ecom_connection() + storageConfigService = ( + common.utils.find_storage_configuration_service( + conn, self.data.storage_system)) + mock_create_replica.side_effect = e self.assertRaises(exception.VolumeBackendAPIException, self.driver.create_cloned_volume, self.data.test_volume, EMCVMAXCommonData.test_source_volume) + extraSpecs = common._initial_setup(self.data.test_volume) + fastPolicy = extraSpecs['storagetype:fastpolicy'] + targetInstance = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + common._delete_from_pool.assert_called_with(storageConfigService, + targetInstance, + targetInstance['Name'], + targetInstance['DeviceID'], + fastPolicy, + extraSpecs) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_migrate_volume_no_fast_success( - self, _mock_volume_type, mock_is_v3): + def test_migrate_volume_no_fast_success(self, _mock_volume_type): self.driver.migrate_volume(self.data.test_ctxt, self.data.test_volume, self.data.test_host) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'parse_pool_instance_id', return_value=('silver', 'SYMMETRIX+000195900551')) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) def test_retype_volume_no_fast_success( - self, _mock_volume_type, mock_values, mock_is_v3): + self, _mock_volume_type, mock_values): self.driver.retype( self.data.test_ctxt, self.data.test_volume, self.data.new_type, self.data.diff, self.data.test_host) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) def test_create_CG_no_fast_success( - self, _mock_volume_type, _mock_storage_system, mock_is_v3): + self, _mock_volume_type, _mock_storage_system): self.driver.create_consistencygroup( self.data.test_ctxt, self.data.test_CG) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSINoFAST'}) - @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) + emc_vmax_common.EMCVMAXCommon, + '_get_members_of_replication_group', + return_value=None) @mock.patch.object( FakeDB, 'volume_get_all_by_group', return_value=None) @mock.patch.object( - EMCVMAXCommon, - '_get_members_of_replication_group', - return_value=None) + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) def test_delete_CG_no_volumes_no_fast_success( self, _mock_volume_type, _mock_storage_system, - _mock_db_volumes, _mock_members, mock_is_v3): + _mock_db_volumes, _mock_members): self.driver.delete_consistencygroup( - self.data.test_ctxt, self.data.test_CG) + self.data.test_ctxt, self.data.test_CG, []) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'ISCSINoFAST'}) + def test_delete_CG_with_volumes_no_fast_success( + self, _mock_volume_type, _mock_storage_system): + self.driver.delete_consistencygroup( + self.data.test_ctxt, self.data.test_CG, []) + + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'find_group_sync_rg_by_target', + return_value="") + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_members_of_replication_group', + return_value=()) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_find_consistency_group', + return_value=(None, EMCVMAXCommonData.test_CG)) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_delete_CG_with_volumes_no_fast_success( - self, _mock_volume_type, _mock_storage_system, - mock_is_v3): - self.driver.delete_consistencygroup( - self.data.test_ctxt, self.data.test_CG) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) + def test_create_snapshot_for_CG_no_fast_success( + self, _mock_volume_type, _mock_storage, _mock_cg, _mock_members, + _mock_rg): + self.driver.create_cgsnapshot( + self.data.test_ctxt, self.data.test_CG_snapshot, []) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'ISCSINoFAST'}) + def test_delete_snapshot_for_CG_no_fast_success( + self, _mock_volume_type, _mock_storage): + self.driver.delete_cgsnapshot( + self.data.test_ctxt, self.data.test_CG_snapshot, []) + @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXCommon, - '_find_consistency_group', - return_value=(None, EMCVMAXCommonData.test_CG)) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) + def test_update_CG_add_volume_no_fast_success( + self, _mock_volume_type, _mock_storage_system): + add_volumes = [] + add_volumes.append(self.data.test_source_volume) + remove_volumes = None + self.driver.update_consistencygroup( + self.data.test_ctxt, self.data.test_CG, + add_volumes, remove_volumes) + # Multiple volumes + add_volumes.append(self.data.test_source_volume) + self.driver.update_consistencygroup( + self.data.test_ctxt, self.data.test_CG, + add_volumes, remove_volumes) + # Can't find CG + self.driver.common._find_consistency_group = mock.Mock( + return_value=None) + self.assertRaises(exception.ConsistencyGroupNotFound, + self.driver.update_consistencygroup, + self.data.test_ctxt, self.data.test_CG, + add_volumes, remove_volumes) + @mock.patch.object( - EMCVMAXCommon, - '_get_members_of_replication_group', - return_value=()) + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'find_group_sync_rg_by_target', - return_value="") + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) + def test_update_CG_remove_volume_no_fast_success( + self, _mock_volume_type, _mock_storage_system): + remove_volumes = [] + remove_volumes.append(self.data.test_source_volume) + add_volumes = None + self.driver.update_consistencygroup( + self.data.test_ctxt, self.data.test_CG, + add_volumes, remove_volumes) + # Multiple volumes + remove_volumes.append(self.data.test_source_volume) + self.driver.update_consistencygroup( + self.data.test_ctxt, self.data.test_CG, + add_volumes, remove_volumes) + + # Bug https://bugs.launchpad.net/cinder/+bug/1442376 + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_create_snapshot_for_CG_no_fast_success( - self, _mock_volume_type, _mock_storage, _mock_cg, _mock_members, - _mock_rg, mock_is_v3): - self.driver.create_cgsnapshot( - self.data.test_ctxt, self.data.test_CG_snapshot) - + emc_vmax_utils.EMCVMAXUtils, + 'get_meta_members_capacity_in_byte', + return_value=[1234567, 7654321]) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'get_volume_meta_head', + return_value=[EMCVMAXCommonData.test_volume]) + @mock.patch.object( + FakeDB, + 'volume_get', + return_value=EMCVMAXCommonData.test_source_volume) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'ISCSINoFAST'}) + def test_create_clone_with_different_meta_sizes( + self, mock_volume_type, mock_volume, + mock_meta, mock_size, mock_pool): + self.data.test_volume['volume_name'] = "vmax-1234567" + common = self.driver.common + volumeDict = {'classname': u'Symm_StorageVolume', + 'keybindings': EMCVMAXCommonData.keybindings} + volume = {'size': 0} + common.provision.create_volume_from_pool = ( + mock.Mock(return_value=(volumeDict, volume['size']))) + common.provision.get_volume_dict_from_job = ( + mock.Mock(return_value=volumeDict)) + + common._create_composite_volume = ( + mock.Mock(return_value=(0, + volumeDict, + EMCVMAXCommonData.storage_system))) + + self.driver.create_cloned_volume(self.data.test_volume, + EMCVMAXCommonData.test_source_volume) + extraSpecs = self.driver.common._initial_setup(self.data.test_volume) + common._create_composite_volume.assert_called_with( + volume, "TargetBaseVol", 1234567, extraSpecs, 1) + + def test_get_volume_element_name(self): + volumeId = 'ea95aa39-080b-4f11-9856-a03acf9112ad' + utils = self.driver.common.utils + volumeElementName = utils.get_volume_element_name(volumeId) + expectVolumeElementName = ( + emc_vmax_utils.VOLUME_ELEMENT_NAME_PREFIX + volumeId) + self.assertEqual(expectVolumeElementName, volumeElementName) + + def test_get_associated_replication_from_source_volume(self): + conn = self.fake_ecom_connection() + utils = self.driver.common.utils + repInstanceName = ( + utils.get_associated_replication_from_source_volume( + conn, self.data.storage_system, + self.data.test_volume['device_id'])) + expectInstanceName = ( + conn.EnumerateInstanceNames('SE_StorageSynchronized_SV_SV')[0]) + self.assertEqual(expectInstanceName, repInstanceName) + + def test_get_array_and_device_id_success(self): + deviceId = '0123' + arrayId = u'array1234' + external_ref = {u'source-name': deviceId} + volume = {'volume_metadata': [{'key': 'array', 'value': arrayId}] + } + utils = self.driver.common.utils + (arrId, devId) = utils.get_array_and_device_id(volume, external_ref) + self.assertEqual(arrayId, arrId) + self.assertEqual(deviceId, devId) + + def test_get_array_and_device_id_failed(self): + deviceId = '0123' + arrayId = u'array1234' + external_ref = {u'no-source-name': deviceId} + volume = {'volume_metadata': [{'key': 'array', 'value': arrayId}] + } + utils = self.driver.common.utils + self.assertRaises(exception.VolumeBackendAPIException, + utils.get_array_and_device_id, + volume, + external_ref) + + def test_rename_volume(self): + conn = self.fake_ecom_connection() + utils = self.driver.common.utils + newName = 'new_name' + volume = {} + volume['CreationClassName'] = 'Symm_StorageVolume' + volume['DeviceID'] = '1' + volume['ElementName'] = 'original_name' + pywbem = mock.Mock() + pywbem.cim_obj = mock.Mock() + pywbem.cim_obj.CIMInstance = mock.Mock() + emc_vmax_utils.pywbem = pywbem + volumeInstance = conn.GetInstance(volume) + originalName = volumeInstance['ElementName'] + volumeInstance = utils.rename_volume(conn, volumeInstance, newName) + self.assertEqual(newName, volumeInstance['ElementName']) + volumeInstance = utils.rename_volume( + conn, volumeInstance, originalName) + self.assertEqual(originalName, volumeInstance['ElementName']) + + def test_get_smi_version(self): + conn = self.fake_ecom_connection() + utils = self.driver.common.utils + version = utils.get_smi_version(conn) + expected = int(str(self.data.majorVersion) + + str(self.data.minorVersion) + + str(self.data.revNumber)) + self.assertEqual(version, expected) + + def test_get_pool_name(self): + conn = self.fake_ecom_connection() + utils = self.driver.common.utils + poolInstanceName = {} + poolInstanceName['InstanceID'] = "SATA_GOLD1" + poolInstanceName['CreationClassName'] = 'Symm_VirtualProvisioningPool' + poolName = utils.get_pool_name(conn, poolInstanceName) + self.assertEqual(poolName, self.data.poolname) + + def test_get_meta_members_capacity_in_byte(self): + conn = self.fake_ecom_connection() + utils = self.driver.common.utils + memberVolumeInstanceNames = [] + volumeHead = EMC_StorageVolume() + volumeHead.classname = 'Symm_StorageVolume' + blockSize = self.data.block_size + volumeHead['ConsumableBlocks'] = ( + self.data.metaHead_volume['ConsumableBlocks']) + volumeHead['BlockSize'] = blockSize + volumeHead['DeviceID'] = self.data.metaHead_volume['DeviceID'] + memberVolumeInstanceNames.append(volumeHead) + metaMember1 = EMC_StorageVolume() + metaMember1.classname = 'Symm_StorageVolume' + metaMember1['ConsumableBlocks'] = ( + self.data.meta_volume1['ConsumableBlocks']) + metaMember1['BlockSize'] = blockSize + metaMember1['DeviceID'] = self.data.meta_volume1['DeviceID'] + memberVolumeInstanceNames.append(metaMember1) + metaMember2 = EMC_StorageVolume() + metaMember2.classname = 'Symm_StorageVolume' + metaMember2['ConsumableBlocks'] = ( + self.data.meta_volume2['ConsumableBlocks']) + metaMember2['BlockSize'] = blockSize + metaMember2['DeviceID'] = self.data.meta_volume2['DeviceID'] + memberVolumeInstanceNames.append(metaMember2) + capacities = utils.get_meta_members_capacity_in_byte( + conn, memberVolumeInstanceNames) + headSize = ( + volumeHead['ConsumableBlocks'] - + metaMember1['ConsumableBlocks'] - + metaMember2['ConsumableBlocks']) + expected = [headSize * blockSize, + metaMember1['ConsumableBlocks'] * blockSize, + metaMember2['ConsumableBlocks'] * blockSize] + self.assertEqual(capacities, expected) + + def test_get_composite_elements(self): + conn = self.fake_ecom_connection() + utils = self.driver.common.utils + volumeInstanceName = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + volumeInstance = conn.GetInstance(volumeInstanceName) + memberVolumeInstanceNames = utils.get_composite_elements( + conn, volumeInstance) + expected = [self.data.metaHead_volume, + self.data.meta_volume1, + self.data.meta_volume2] + self.assertEqual(memberVolumeInstanceNames, expected) + + def test_get_volume_model_updates(self): + utils = self.driver.common.utils + status = 'status-string' + volumes = utils.get_volume_model_updates( + None, self.driver.db.volume_get_all_by_group("", 5), + self.data.test_CG['id'], + status) + self.assertEqual(status, volumes[0]['status']) + + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'find_group_sync_rg_by_target', + return_value="") + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_find_consistency_group', + return_value=(None, EMCVMAXCommonData.test_CG)) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_delete_snapshot_for_CG_no_fast_success( - self, _mock_volume_type, _mock_storage, mock_is_v3): - self.driver.delete_cgsnapshot( - self.data.test_ctxt, self.data.test_CG_snapshot) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSINoFAST'}) + def test_create_consistencygroup_from_src( + self, _mock_volume_type, _mock_storage, _mock_cg, _mock_rg): + volumes = [] + volumes.append(self.data.test_source_volume) + snapshots = [] + self.data.test_snapshot['volume_size'] = "10" + snapshots.append(self.data.test_snapshot) + model_update, volumes_model_update = ( + self.driver.create_consistencygroup_from_src( + self.data.test_ctxt, self.data.test_CG, volumes, + self.data.test_CG_snapshot, snapshots)) + self.assertEqual({'status': fields.ConsistencyGroupStatus.AVAILABLE}, + model_update) + self.assertEqual([{'status': 'available', 'id': '2'}], + volumes_model_update) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_update_pool_stats', + return_value={1, 2, 3, 4, 5}) + def test_ssl_support(self, pool_stats): + self.driver.common.update_volume_stats() + self.assertTrue(self.driver.common.ecomUseSSL) def _cleanup(self): - bExists = os.path.exists(self.config_file_path) - if bExists: - os.remove(self.config_file_path) + if self.config_file_path: + bExists = os.path.exists(self.config_file_path) + if bExists: + os.remove(self.config_file_path) shutil.rmtree(self.tempdir) @@ -2157,22 +3884,20 @@ def setUp(self): configuration.safe_get.return_value = 'ISCSIFAST' configuration.config_group = 'ISCSIFAST' - self.stubs.Set(EMCVMAXISCSIDriver, 'smis_do_iscsi_discovery', - self.fake_do_iscsi_discovery) - self.stubs.Set(EMCVMAXCommon, '_get_ecom_connection', + self.stubs.Set(emc_vmax_common.EMCVMAXCommon, '_get_ecom_connection', self.fake_ecom_connection) instancename = FakeCIMInstanceName() - self.stubs.Set(EMCVMAXUtils, 'get_instance_name', + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'get_instance_name', instancename.fake_getinstancename) - self.stubs.Set(time, 'sleep', - self.fake_sleep) - driver = EMCVMAXISCSIDriver(configuration=configuration) + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'isArrayV3', + self.fake_is_v3) + driver = emc_vmax_iscsi.EMCVMAXISCSIDriver(configuration=configuration) driver.db = FakeDB() self.driver = driver def create_fake_config_file_fast(self): - doc = Document() + doc = minidom.Document() emc = doc.createElement("EMC") doc.appendChild(emc) @@ -2212,7 +3937,7 @@ def create_fake_config_file_fast(self): timeout.appendChild(timeouttext) portgroup = doc.createElement("PortGroup") - portgrouptext = doc.createTextNode("myPortGroup") + portgrouptext = doc.createTextNode(self.data.port_group) portgroup.appendChild(portgrouptext) pool = doc.createElement("Pool") @@ -2220,11 +3945,6 @@ def create_fake_config_file_fast(self): emc.appendChild(pool) pool.appendChild(pooltext) - array = doc.createElement("Array") - arraytext = doc.createTextNode("0123456789") - emc.appendChild(array) - array.appendChild(arraytext) - portgroups = doc.createElement("PortGroups") portgroups.appendChild(portgroup) emc.appendChild(portgroups) @@ -2241,123 +3961,96 @@ def fake_ecom_connection(self): conn = FakeEcomConnection() return conn - def fake_do_iscsi_discovery(self, volume): - output = [] - item = '10.10.0.50: 3260,1 iqn.1992-04.com.emc: 50000973f006dd80' - output.append(item) - return output - - def fake_sleep(self, seconds): - return + def fake_is_v3(self, conn, serialNumber): + return False @mock.patch.object( - EMCVMAXUtils, - 'find_storageSystem', - return_value=None) + emc_vmax_fast.EMCVMAXFast, + 'get_capacities_associated_to_policy', + return_value=(1234, 1200, 1200, 1)) @mock.patch.object( - EMCVMAXFast, - 'is_tiering_policy_enabled', - return_value=True) + emc_vmax_utils.EMCVMAXUtils, + 'get_pool_capacities', + return_value=(1234, 1200, 1200, 1)) @mock.patch.object( - EMCVMAXFast, + emc_vmax_fast.EMCVMAXFast, 'get_tier_policy_by_name', return_value=None) @mock.patch.object( - EMCVMAXFast, - 'get_capacities_associated_to_policy', - return_value=(1234, 1200)) - @mock.patch.object( - EMCVMAXUtils, - 'parse_array_name_from_file', - return_value="123456789") + emc_vmax_fast.EMCVMAXFast, + 'is_tiering_policy_enabled', + return_value=True) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_get_volume_stats_fast(self, mock_storage_system, + emc_vmax_utils.EMCVMAXUtils, + 'find_storageSystem', + return_value=None) + def test_get_volume_stats_fast(self, + mock_storage_system, mock_is_fast_enabled, - mock_get_policy, mock_capacity, mock_array, - mock_is_v3): + mock_get_policy, + mock_pool_capacities, + mock_capacities_associated_to_policy): self.driver.get_volume_stats(True) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) + emc_vmax_fast.EMCVMAXFast, + 'get_pool_associated_to_policy', + return_value=1) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXFast, - 'get_pool_associated_to_policy', - return_value=1) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) def test_create_volume_fast_success( - self, _mock_volume_type, mock_storage_system, mock_pool_policy, - mock_is_v3): + self, _mock_volume_type, mock_storage_system, mock_pool_policy): self.driver.create_volume(self.data.test_volume_v2) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'storagetype: stripedmetacount': '4', - 'volume_backend_name': 'ISCSIFAST'}) + emc_vmax_fast.EMCVMAXFast, + 'get_pool_associated_to_policy', + return_value=1) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXFast, - 'get_pool_associated_to_policy', - return_value=1) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'storagetype: stripedmetacount': '4', + 'volume_backend_name': 'ISCSIFAST'}) def test_create_volume_fast_striped_success( - self, _mock_volume_type, mock_storage_system, mock_pool_policy, - mock_is_v3): + self, _mock_volume_type, mock_storage_system, mock_pool_policy): self.driver.create_volume(self.data.test_volume_v2) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) - @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) - @mock.patch.object( - EMCVMAXFast, + emc_vmax_fast.EMCVMAXFast, 'get_pool_associated_to_policy', return_value=1) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_create_volume_in_CG_fast_success( - self, _mock_volume_type, mock_storage_system, mock_pool_policy, - mock_is_v3): - self.driver.create_volume(self.data.test_volume_CG) - + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'ISCSIFAST'}) + def test_create_volume_in_CG_fast_success( + self, _mock_volume_type, mock_storage_system, mock_pool_policy): + self.driver.create_volume(self.data.test_volume_CG) + @mock.patch.object( - EMCVMAXMasking, + emc_vmax_masking.EMCVMAXMasking, '_wrap_get_storage_group_from_volume', return_value=None) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) def test_delete_volume_fast_success( - self, _mock_volume_type, mock_storage_group, mock_is_v3): + self, _mock_volume_type, mock_storage_group): self.driver.delete_volume(self.data.test_volume) def test_create_volume_fast_failed(self): @@ -2366,19 +4059,15 @@ def test_create_volume_fast_failed(self): self.data.test_failed_volume) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) - @mock.patch.object( - EMCVMAXMasking, + emc_vmax_masking.EMCVMAXMasking, '_wrap_get_storage_group_from_volume', return_value=None) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) def test_delete_volume_fast_notfound( - self, _mock_volume_type, mock_wrapper, mock_is_v3): + self, _mock_volume_type, mock_wrapper): notfound_delete_vol = {} notfound_delete_vol['name'] = 'notfound_delete_vol' notfound_delete_vol['id'] = '10' @@ -2386,151 +4075,132 @@ def test_delete_volume_fast_notfound( notfound_delete_vol['SystemName'] = self.data.storage_system notfound_delete_vol['DeviceID'] = notfound_delete_vol['id'] notfound_delete_vol['SystemCreationClassName'] = 'Symm_StorageSystem' + notfound_delete_vol['host'] = self.data.fake_host name = {} name['classname'] = 'Symm_StorageVolume' keys = {} keys['CreationClassName'] = notfound_delete_vol['CreationClassName'] keys['SystemName'] = notfound_delete_vol['SystemName'] keys['DeviceID'] = notfound_delete_vol['DeviceID'] - keys['SystemCreationClassName'] = \ - notfound_delete_vol['SystemCreationClassName'] + keys['SystemCreationClassName'] = ( + notfound_delete_vol['SystemCreationClassName']) name['keybindings'] = keys notfound_delete_vol['volume_type_id'] = 'abc' notfound_delete_vol['provider_location'] = None self.driver.delete_volume(notfound_delete_vol) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) - @mock.patch.object( - EMCVMAXMasking, - '_wrap_get_storage_group_from_volume', - return_value=None) + emc_vmax_fast.EMCVMAXFast, + 'get_pool_associated_to_policy', + return_value=1) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXFast, - 'get_pool_associated_to_policy', - return_value=1) + emc_vmax_masking.EMCVMAXMasking, + '_wrap_get_storage_group_from_volume', + return_value=None) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) def test_delete_volume_fast_failed( self, _mock_volume_type, _mock_storage_group, - mock_storage_system, mock_policy_pool, mock_is_v3): + mock_storage_system, mock_policy_pool): self.driver.create_volume(self.data.failed_delete_vol) self.assertRaises(exception.VolumeBackendAPIException, self.driver.delete_volume, self.data.failed_delete_vol) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) - @mock.patch.object( - EMCVMAXMasking, - '_wrap_get_storage_group_from_volume', - return_value=None) + emc_vmax_common.EMCVMAXCommon, + '_is_same_host', + return_value=True) @mock.patch.object( - EMCVMAXCommon, - '_wrap_find_device_number', + emc_vmax_common.EMCVMAXCommon, + 'find_device_number', return_value={'hostlunid': 1, 'storagesystem': EMCVMAXCommonData.storage_system}) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + emc_vmax_masking.EMCVMAXMasking, + '_wrap_get_storage_group_from_volume', + return_value=None) @mock.patch.object( - EMCVMAXCommon, - '_is_same_host', - return_value=True) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) def test_already_mapped_fast_success( self, _mock_volume_type, mock_wrap_group, mock_wrap_device, - mock_is_v3, mock_is_same_host): + mock_is_same_host): + self.driver.common._get_correct_port_group = mock.Mock( + return_value=self.data.port_group) self.driver.initialize_connection(self.data.test_volume, self.data.connector) @mock.patch.object( - EMCVMAXMasking, - '_wrap_get_storage_group_from_volume', - return_value=None) - @mock.patch.object( - EMCVMAXCommon, - '_wrap_find_device_number', + emc_vmax_common.EMCVMAXCommon, + 'find_device_number', return_value={'storagesystem': EMCVMAXCommonData.storage_system}) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_map_fast_failed(self, mock_wrap_group, mock_wrap_device, - mock_is_v3): + emc_vmax_masking.EMCVMAXMasking, + '_wrap_get_storage_group_from_volume', + return_value=None) + def test_map_fast_failed(self, mock_wrap_group, mock_wrap_device): self.assertRaises(exception.VolumeBackendAPIException, self.driver.initialize_connection, self.data.test_volume, self.data.connector) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) - @mock.patch.object( - EMCVMAXUtils, - 'find_storage_masking_group', - return_value=EMCVMAXCommonData.storagegroupname) + emc_vmax_provision.EMCVMAXProvision, + 'add_members_to_masking_group', + return_value=0) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + emc_vmax_masking.EMCVMAXMasking, + 'get_initiator_group_from_masking_view', + return_value='myInitGroup') @mock.patch.object( - EMCVMAXMasking, + emc_vmax_masking.EMCVMAXMasking, '_find_initiator_masking_group', return_value='myInitGroup') @mock.patch.object( - EMCVMAXMasking, - 'get_initiator_group_from_masking_view', - return_value='myInitGroup') - def test_detach_fast_success( - self, mock_volume_type, mock_storage_group, mock_is_v3, - mock_ig, mock_igc): - self.driver.terminate_connection( - self.data.test_volume, self.data.connector) - + emc_vmax_utils.EMCVMAXUtils, + 'find_storage_masking_group', + return_value=EMCVMAXCommonData.storagegroupname) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'ISCSIFAST'}) + def test_detach_fast_success( + self, mock_volume_type, mock_storage_group, + mock_ig, mock_igc, mock_add): + self.driver.terminate_connection( + self.data.test_volume, self.data.connector) + @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'get_volume_size', return_value='2147483648') @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) def test_extend_volume_fast_success( - self, _mock_volume_type, mock_volume_size, mock_is_v3): + self, _mock_volume_type, mock_volume_size): newSize = '2' self.driver.extend_volume(self.data.test_volume, newSize) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) - @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'check_if_volume_is_extendable', return_value='False') @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) def test_extend_volume_striped_fast_failed( - self, _mock_volume_type, _mock_is_extendable, - mock_is_v3): + self, _mock_volume_type, _mock_is_extendable): newSize = '2' self.assertRaises(exception.VolumeBackendAPIException, self.driver.extend_volume, @@ -2538,35 +4208,31 @@ def test_extend_volume_striped_fast_failed( newSize) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) - @mock.patch.object( - FakeDB, - 'volume_get', - return_value=EMCVMAXCommonData.test_source_volume) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - @mock.patch.object( - EMCVMAXFast, + emc_vmax_fast.EMCVMAXFast, 'get_pool_associated_to_policy', return_value=1) @mock.patch.object( - EMCVMAXUtils, + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'get_meta_members_capacity_in_byte', + return_value=[1234567, 7654321]) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, 'get_volume_meta_head', return_value=[EMCVMAXCommonData.test_volume]) @mock.patch.object( - EMCVMAXUtils, - 'get_meta_members_capacity_in_bit', - return_value=[1234567, 7654321]) + FakeDB, + 'volume_get', + return_value=EMCVMAXCommonData.test_source_volume) @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) def test_create_snapshot_different_sizes_meta_fast_success( - self, mock_volume_type, mock_volume, mock_is_v3, + self, mock_volume_type, mock_volume, mock_meta, mock_size, mock_pool, mock_policy): self.data.test_volume['volume_name'] = "vmax-1234567" common = self.driver.common @@ -2574,7 +4240,7 @@ def test_create_snapshot_different_sizes_meta_fast_success( volumeDict = {'classname': u'Symm_StorageVolume', 'keybindings': EMCVMAXCommonData.keybindings} common.provision.create_volume_from_pool = ( - mock.Mock(return_value=(volumeDict, 0L))) + mock.Mock(return_value=(volumeDict, 0))) common.provision.get_volume_dict_from_job = ( mock.Mock(return_value=volumeDict)) common.fast.is_volume_in_default_SG = ( @@ -2588,28 +4254,23 @@ def test_create_snapshot_fast_failed(self): self.data.test_volume) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) - @mock.patch.object( - EMCVMAXUtils, - 'find_sync_sv_by_target', - return_value=(None, None)) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + emc_vmax_utils.EMCVMAXUtils, + 'get_meta_members_capacity_in_byte', + return_value=[1234567]) @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'get_volume_meta_head', return_value=[EMCVMAXCommonData.test_volume]) @mock.patch.object( - EMCVMAXUtils, - 'get_meta_members_capacity_in_bit', - return_value=[1234567]) + emc_vmax_utils.EMCVMAXUtils, + 'find_sync_sv_by_target', + return_value=(None, None)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) def test_create_volume_from_same_size_meta_snapshot( - self, mock_volume_type, mock_sync_sv, mock_is_v3, - mock_meta, mock_size): + self, mock_volume_type, mock_sync_sv, mock_meta, mock_size): self.data.test_volume['volume_name'] = "vmax-1234567" common = self.driver.common common.fast.is_volume_in_default_SG = mock.Mock(return_value=True) @@ -2617,25 +4278,21 @@ def test_create_volume_from_same_size_meta_snapshot( self.data.test_volume, self.data.test_volume) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST', - 'FASTPOLICY': 'FC_GOLD1'}) + emc_vmax_utils.EMCVMAXUtils, + 'find_sync_sv_by_target', + return_value=(None, None)) @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'find_replication_service', return_value=None) @mock.patch.object( - EMCVMAXUtils, - 'find_sync_sv_by_target', - return_value=(None, None)) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST', + 'FASTPOLICY': 'FC_GOLD1'}) def test_create_volume_from_snapshot_fast_failed( self, mock_volume_type, - mock_rep_service, mock_sync_sv, mock_is_v3): + mock_rep_service, mock_sync_sv): self.data.test_volume['volume_name'] = "vmax-1234567" self.assertRaises(exception.VolumeBackendAPIException, self.driver.create_volume_from_snapshot, @@ -2643,68 +4300,35 @@ def test_create_volume_from_snapshot_fast_failed( EMCVMAXCommonData.test_source_volume) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) - @mock.patch.object( - FakeDB, - 'volume_get', - return_value=EMCVMAXCommonData.test_source_volume) - @mock.patch.object( - EMCVMAXCommon, - '_find_storage_sync_sv_sv', - return_value=(None, None)) + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + emc_vmax_utils.EMCVMAXUtils, + 'get_meta_members_capacity_in_byte', + return_value=[1234567, 7654321]) @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'get_volume_meta_head', - return_value=None) - def test_create_clone_simple_volume_fast_success( - self, mock_volume_type, mock_volume, mock_sync_sv, - mock_is_v3, mock_simple_volume): - self.data.test_volume['volume_name'] = "vmax-1234567" - self.driver.common.fast.is_volume_in_default_SG = ( - mock.Mock(return_value=True)) - self.driver.create_cloned_volume(self.data.test_volume, - EMCVMAXCommonData.test_source_volume) - + return_value=[EMCVMAXCommonData.test_volume]) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) + emc_vmax_fast.EMCVMAXFast, + 'get_pool_associated_to_policy', + return_value=1) @mock.patch.object( FakeDB, 'volume_get', return_value=EMCVMAXCommonData.test_source_volume) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - @mock.patch.object( - EMCVMAXFast, - 'get_pool_associated_to_policy', - return_value=1) - @mock.patch.object( - EMCVMAXUtils, - 'get_volume_meta_head', - return_value=[EMCVMAXCommonData.test_volume]) - @mock.patch.object( - EMCVMAXUtils, - 'get_meta_members_capacity_in_bit', - return_value=[1234567, 7654321]) - @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) def test_create_clone_fast_failed( - self, mock_volume_type, mock_vol, mock_is_v3, + self, mock_volume_type, mock_vol, mock_policy, mock_meta, mock_size, mock_pool): self.data.test_volume['volume_name'] = "vmax-1234567" self.driver.common._modify_and_get_composite_volume_instance = ( - mock.Mock(return_value=(1L, None))) + mock.Mock(return_value=(1, None))) self.assertRaises(exception.VolumeBackendAPIException, self.driver.create_cloned_volume, self.data.test_volume, @@ -2719,135 +4343,153 @@ def test_migrate_volume_fast_success(self, _mock_volume_type): self.data.test_host) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) + emc_vmax_masking.EMCVMAXMasking, + '_wrap_get_storage_group_from_volume', + return_value=None) @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'parse_pool_instance_id', return_value=('silver', 'SYMMETRIX+000195900551')) @mock.patch.object( - EMCVMAXMasking, - '_wrap_get_storage_group_from_volume', - return_value=None) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) def test_retype_volume_fast_success( - self, _mock_volume_type, mock_values, mock_wrap, mock_is_v3): + self, _mock_volume_type, mock_values, mock_wrap): self.driver.retype( self.data.test_ctxt, self.data.test_volume, self.data.new_type, self.data.diff, self.data.test_host) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) def test_create_CG_fast_success( - self, _mock_volume_type, _mock_storage_system, - mock_is_v3): + self, _mock_volume_type, _mock_storage_system): self.driver.create_consistencygroup( self.data.test_ctxt, self.data.test_CG) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) - @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) + emc_vmax_common.EMCVMAXCommon, + '_get_members_of_replication_group', + return_value=None) @mock.patch.object( FakeDB, 'volume_get_all_by_group', return_value=None) @mock.patch.object( - EMCVMAXCommon, - '_get_members_of_replication_group', - return_value=None) + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) def test_delete_CG_no_volumes_fast_success( self, _mock_volume_type, _mock_storage_system, - _mock_db_volumes, _mock_members, mock_is_v3): + _mock_db_volumes, _mock_members): self.driver.delete_consistencygroup( - self.data.test_ctxt, self.data.test_CG) + self.data.test_ctxt, self.data.test_CG, []) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) def test_delete_CG_with_volumes_fast_success( - self, _mock_volume_type, _mock_storage_system, - mock_is_v3): + self, _mock_volume_type, _mock_storage_system): self.driver.delete_consistencygroup( - self.data.test_ctxt, self.data.test_CG) + self.data.test_ctxt, self.data.test_CG, []) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'ISCSIFAST'}) + emc_vmax_utils.EMCVMAXUtils, + 'find_group_sync_rg_by_target', + return_value="") @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) + emc_vmax_common.EMCVMAXCommon, + '_get_members_of_replication_group', + return_value=()) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_find_consistency_group', return_value=(None, EMCVMAXCommonData.test_CG)) @mock.patch.object( - EMCVMAXCommon, - '_get_members_of_replication_group', - return_value=()) - @mock.patch.object( - EMCVMAXUtils, - 'find_group_sync_rg_by_target', - return_value="") + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) def test_create_snapshot_for_CG_no_fast_success( self, _mock_volume_type, _mock_storage, _mock_cg, _mock_members, - _mock_rg, mock_is_v3): + _mock_rg): self.driver.create_cgsnapshot( - self.data.test_ctxt, self.data.test_CG_snapshot) + self.data.test_ctxt, self.data.test_CG_snapshot, []) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'ISCSIFAST'}) + def test_delete_snapshot_for_CG_no_fast_success( + self, _mock_volume_type, _mock_storage): + self.driver.delete_cgsnapshot( + self.data.test_ctxt, self.data.test_CG_snapshot, []) + @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_delete_snapshot_for_CG_no_fast_success( - self, _mock_volume_type, _mock_storage, mock_is_v3): - self.driver.delete_cgsnapshot( - self.data.test_ctxt, self.data.test_CG_snapshot) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) + def test_update_CG_add_volume_fast_success( + self, _mock_volume_type, _mock_storage_system): + add_volumes = [] + add_volumes.append(self.data.test_source_volume) + remove_volumes = None + self.driver.update_consistencygroup( + self.data.test_ctxt, self.data.test_CG, + add_volumes, remove_volumes) + # Multiple volumes + add_volumes.append(self.data.test_source_volume) + self.driver.update_consistencygroup( + self.data.test_ctxt, self.data.test_CG, + add_volumes, remove_volumes) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'ISCSIFAST'}) + def test_update_CG_remove_volume_fast_success( + self, _mock_volume_type, _mock_storage_system): + remove_volumes = [] + remove_volumes.append(self.data.test_source_volume) + add_volumes = None + self.driver.update_consistencygroup( + self.data.test_ctxt, self.data.test_CG, + add_volumes, remove_volumes) + # Multiple volumes + remove_volumes.append(self.data.test_source_volume) + self.driver.update_consistencygroup( + self.data.test_ctxt, self.data.test_CG, + add_volumes, remove_volumes) def _cleanup(self): bExists = os.path.exists(self.config_file_path) @@ -2872,24 +4514,24 @@ def setUp(self): configuration.safe_get.return_value = 'FCNoFAST' configuration.config_group = 'FCNoFAST' - self.stubs.Set(EMCVMAXCommon, '_get_ecom_connection', + self.stubs.Set(emc_vmax_common.EMCVMAXCommon, '_get_ecom_connection', self.fake_ecom_connection) instancename = FakeCIMInstanceName() - self.stubs.Set(EMCVMAXUtils, 'get_instance_name', + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'get_instance_name', instancename.fake_getinstancename) - self.stubs.Set(time, 'sleep', - self.fake_sleep) + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'isArrayV3', + self.fake_is_v3) - driver = EMCVMAXFCDriver(configuration=configuration) + driver = emc_vmax_fc.EMCVMAXFCDriver(configuration=configuration) driver.db = FakeDB() driver.common.conn = FakeEcomConnection() driver.zonemanager_lookup_service = FakeLookupService() self.driver = driver - self.driver.utils = EMCVMAXUtils(object) + self.driver.utils = emc_vmax_utils.EMCVMAXUtils(object) def create_fake_config_file_no_fast(self): - doc = Document() + doc = minidom.Document() emc = doc.createElement("EMC") doc.appendChild(emc) @@ -2919,7 +4561,7 @@ def create_fake_config_file_no_fast(self): ecompassword.appendChild(ecompasswordtext) portgroup = doc.createElement("PortGroup") - portgrouptext = doc.createTextNode("myPortGroup") + portgrouptext = doc.createTextNode(self.data.port_group) portgroup.appendChild(portgrouptext) portgroups = doc.createElement("PortGroups") @@ -2931,11 +4573,6 @@ def create_fake_config_file_no_fast(self): emc.appendChild(pool) pool.appendChild(pooltext) - array = doc.createElement("Array") - arraytext = doc.createTextNode("0123456789") - emc.appendChild(array) - array.appendChild(arraytext) - timeout = doc.createElement("Timeout") timeouttext = doc.createTextNode("0") emc.appendChild(timeout) @@ -2953,100 +4590,74 @@ def fake_ecom_connection(self): conn = FakeEcomConnection() return conn - def fake_sleep(self, seconds): - return + def fake_is_v3(self, conn, serialNumber): + return False @mock.patch.object( - EMCVMAXUtils, - 'find_storageSystem', - return_value=None) + emc_vmax_utils.EMCVMAXUtils, + 'get_pool_capacities', + return_value=(1234, 1200, 1200, 1)) @mock.patch.object( - EMCVMAXFast, + emc_vmax_fast.EMCVMAXFast, 'is_tiering_policy_enabled', return_value=False) @mock.patch.object( - EMCVMAXUtils, - 'get_pool_capacities', - return_value=(1234, 1200)) - @mock.patch.object( - EMCVMAXUtils, - 'parse_array_name_from_file', - return_value="123456789") - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + emc_vmax_utils.EMCVMAXUtils, + 'find_storageSystem', + return_value=None) def test_get_volume_stats_no_fast(self, mock_storage_system, mock_is_fast_enabled, - mock_capacity, - mock_array, - mock_is_v3): + mock_capacity): self.driver.get_volume_stats(True) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCNoFAST'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCNoFAST'}) def test_create_volume_no_fast_success( - self, _mock_volume_type, mock_storage_system, mock_is_v3): + self, _mock_volume_type, mock_storage_system): self.driver.create_volume(self.data.test_volume_v2) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'storagetype: stripedmetacount': '4', 'volume_backend_name': 'FCNoFAST'}) - @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) def test_create_volume_no_fast_striped_success( - self, _mock_volume_type, mock_storage_system, mock_is_v3): + self, _mock_volume_type, mock_storage_system): self.driver.create_volume(self.data.test_volume_v2) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCNoFAST'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_create_volume_in_CG_no_fast_success( - self, _mock_volume_type, mock_storage_system, mock_is_v3): - self.driver.create_volume(self.data.test_volume_CG) - @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'FCNoFAST'}) + def test_create_volume_in_CG_no_fast_success( + self, _mock_volume_type, mock_storage_system): + self.driver.create_volume(self.data.test_volume_CG) + @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCNoFAST'}) def test_delete_volume_no_fast_success( - self, _mock_volume_type, mock_storage_system, mock_is_v3): + self, _mock_volume_type, mock_storage_system): self.driver.delete_volume(self.data.test_volume) def test_create_volume_no_fast_failed(self): @@ -3058,12 +4669,7 @@ def test_create_volume_no_fast_failed(self): volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'FCNoFAST'}) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_delete_volume_no_fast_notfound(self, _mock_volume_type, - mock_is_v3): + def test_delete_volume_no_fast_notfound(self, _mock_volume_type): notfound_delete_vol = {} notfound_delete_vol['name'] = 'notfound_delete_vol' notfound_delete_vol['id'] = '10' @@ -3071,62 +4677,56 @@ def test_delete_volume_no_fast_notfound(self, _mock_volume_type, notfound_delete_vol['SystemName'] = self.data.storage_system notfound_delete_vol['DeviceID'] = notfound_delete_vol['id'] notfound_delete_vol['SystemCreationClassName'] = 'Symm_StorageSystem' + notfound_delete_vol['host'] = self.data.fake_host name = {} name['classname'] = 'Symm_StorageVolume' keys = {} keys['CreationClassName'] = notfound_delete_vol['CreationClassName'] keys['SystemName'] = notfound_delete_vol['SystemName'] keys['DeviceID'] = notfound_delete_vol['DeviceID'] - keys['SystemCreationClassName'] = \ - notfound_delete_vol['SystemCreationClassName'] + keys['SystemCreationClassName'] = ( + notfound_delete_vol['SystemCreationClassName']) name['keybindings'] = keys notfound_delete_vol['volume_type_id'] = 'abc' notfound_delete_vol['provider_location'] = None self.driver.delete_volume(notfound_delete_vol) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCNoFAST'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCNoFAST'}) def test_delete_volume_failed( - self, _mock_volume_type, mock_storage_system, mock_is_v3): + self, _mock_volume_type, mock_storage_system): self.driver.create_volume(self.data.failed_delete_vol) self.assertRaises(exception.VolumeBackendAPIException, self.driver.delete_volume, self.data.failed_delete_vol) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCNoFAST', - 'FASTPOLICY': 'FC_GOLD1'}) + emc_vmax_common.EMCVMAXCommon, + '_is_same_host', + return_value=True) @mock.patch.object( - EMCVMAXMasking, + emc_vmax_masking.EMCVMAXMasking, 'get_masking_view_from_storage_group', return_value=EMCVMAXCommonData.lunmaskctrl_name) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - @mock.patch.object( - EMCVMAXCommon, - '_is_same_host', - return_value=True) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCNoFAST', + 'FASTPOLICY': 'FC_GOLD1'}) def test_map_lookup_service_no_fast_success( - self, _mock_volume_type, mock_maskingview, mock_is_v3, - mock_is_same_host): + self, _mock_volume_type, mock_maskingview, mock_is_same_host): self.data.test_volume['volume_name'] = "vmax-1234567" common = self.driver.common common.get_target_wwns_from_masking_view = mock.Mock( return_value=EMCVMAXCommonData.target_wwns) + common._get_correct_port_group = mock.Mock( + return_value=self.data.port_group) lookup_service = self.driver.zonemanager_lookup_service lookup_service.get_device_mapping_from_network = mock.Mock( return_value=EMCVMAXCommonData.device_map) @@ -3143,83 +4743,66 @@ def test_map_lookup_service_no_fast_success( for init, target in data['data']['initiator_target_map'].items(): self.assertEqual(init, target[0][::-1]) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + 'find_device_number', + return_value={'Name': "0001"}) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'FCNoFAST', 'FASTPOLICY': 'FC_GOLD1'}) - @mock.patch.object( - EMCVMAXCommon, - 'find_device_number', - return_value={'Name': "0001"}) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_map_no_fast_failed(self, mock_wrap_group, mock_maskingview, - mock_is_v3): + def test_map_no_fast_failed(self, _mock_volume_type, mock_wrap_device): self.assertRaises(exception.VolumeBackendAPIException, self.driver.initialize_connection, self.data.test_volume, self.data.connector) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCNoFAST'}) - @mock.patch.object( - EMCVMAXMasking, - 'get_masking_view_by_volume', - return_value=EMCVMAXCommonData.lunmaskctrl_name) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + emc_vmax_common.EMCVMAXCommon, + 'check_ig_instance_name', + return_value=None) @mock.patch.object( - EMCVMAXMasking, + emc_vmax_masking.EMCVMAXMasking, '_find_initiator_masking_group', return_value='myInitGroup') @mock.patch.object( - EMCVMAXMasking, - 'get_initiator_group_from_masking_view', - return_value='myInitGroup') - def test_detach_no_fast_last_volume_success( - self, mock_volume_type, mock_mv, mock_is_v3, mock_ig, mock_igc): - self.driver.terminate_connection(self.data.test_source_volume, - self.data.connector) - + emc_vmax_masking.EMCVMAXMasking, + 'get_masking_view_by_volume', + return_value=EMCVMAXCommonData.lunmaskctrl_name) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'FCNoFAST'}) + def test_detach_no_fast_last_volume_success( + self, mock_volume_type, mock_mv, mock_ig, mock_check_ig): + # last volume so initiatorGroup will be deleted by terminate connection + self.driver.terminate_connection(self.data.test_source_volume, + self.data.connector) + @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'get_volume_size', return_value='2147483648') @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCNoFAST'}) def test_extend_volume_no_fast_success(self, _mock_volume_type, - _mock_volume_size, mock_is_v3): + _mock_volume_size): newSize = '2' self.driver.extend_volume(self.data.test_volume, newSize) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCNoFAST'}) - @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'check_if_volume_is_extendable', return_value='False') @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCNoFAST'}) def test_extend_volume_striped_no_fast_failed( - self, _mock_volume_type, _mock_is_extendable, - mock_is_v3): + self, _mock_volume_type, _mock_is_extendable): newSize = '2' self.assertRaises(exception.VolumeBackendAPIException, self.driver.extend_volume, @@ -3230,172 +4813,174 @@ def test_extend_volume_striped_no_fast_failed( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'FCNoFAST'}) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_migrate_volume_no_fast_success(self, _mock_volume_type, - mock_is_v3): + def test_migrate_volume_no_fast_success(self, _mock_volume_type): self.driver.migrate_volume(self.data.test_ctxt, self.data.test_volume, self.data.test_host) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCNoFAST'}) - @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'parse_pool_instance_id', return_value=('silver', 'SYMMETRIX+000195900551')) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCNoFAST'}) def test_retype_volume_no_fast_success( - self, _mock_volume_type, mock_values, mock_is_v3): + self, _mock_volume_type, mock_values): self.driver.retype( self.data.test_ctxt, self.data.test_volume, self.data.new_type, self.data.diff, self.data.test_host) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCNoFAST'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCNoFAST'}) def test_create_CG_no_fast_success( - self, _mock_volume_type, _mock_storage_system, mock_is_v3): + self, _mock_volume_type, _mock_storage_system): self.driver.create_consistencygroup( self.data.test_ctxt, self.data.test_CG) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCNoFAST'}) - @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) + emc_vmax_common.EMCVMAXCommon, + '_get_members_of_replication_group', + return_value=None) @mock.patch.object( FakeDB, 'volume_get_all_by_group', return_value=None) @mock.patch.object( - EMCVMAXCommon, - '_get_members_of_replication_group', - return_value=None) + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCNoFAST'}) def test_delete_CG_no_volumes_no_fast_success( self, _mock_volume_type, _mock_storage_system, - _mock_db_volumes, _mock_members, mock_is_v3): + _mock_db_volumes, _mock_members): self.driver.delete_consistencygroup( - self.data.test_ctxt, self.data.test_CG) + self.data.test_ctxt, self.data.test_CG, []) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCNoFAST'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCNoFAST'}) def test_delete_CG_with_volumes_no_fast_success( - self, _mock_volume_type, _mock_storage_system, - mock_is_v3): + self, _mock_volume_type, _mock_storage_system): self.driver.delete_consistencygroup( - self.data.test_ctxt, self.data.test_CG) + self.data.test_ctxt, self.data.test_CG, []) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCNoFAST'}) + emc_vmax_utils.EMCVMAXUtils, + 'find_group_sync_rg_by_target', + return_value="") @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) + emc_vmax_common.EMCVMAXCommon, + '_get_members_of_replication_group', + return_value=()) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_find_consistency_group', return_value=(None, EMCVMAXCommonData.test_CG)) @mock.patch.object( - EMCVMAXCommon, - '_get_members_of_replication_group', - return_value=()) - @mock.patch.object( - EMCVMAXUtils, - 'find_group_sync_rg_by_target', - return_value="") + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCNoFAST'}) def test_create_snapshot_for_CG_no_fast_success( self, _mock_volume_type, _mock_storage, _mock_cg, _mock_members, - _mock_rg, mock_is_v3): + _mock_rg): self.driver.create_cgsnapshot( - self.data.test_ctxt, self.data.test_CG_snapshot) + self.data.test_ctxt, self.data.test_CG_snapshot, []) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCNoFAST'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCNoFAST'}) def test_delete_snapshot_for_CG_no_fast_success( - self, _mock_volume_type, _mock_storage, mock_is_v3): + self, _mock_volume_type, _mock_storage): self.driver.delete_cgsnapshot( - self.data.test_ctxt, self.data.test_CG_snapshot) - - def create_fake_config_file_parse_port_group(self): - filename = 'cinder_emc_config_file_parse_port_group.xml' - self.config_file_parse_port_group = self.tempdir + '/' + filename - text_file = open(self.config_file_parse_port_group, "w") - text_file.write( - "\n\n" - "10.108.246.202\n" - "5988\n" - "admin\t\n" - "#1Password\n" - "OS-PORTGROUP1-PG\r\n" - "\n" - "\n" - " \n" - "\n" - "\n000198700439" - " \n\nFC_SLVR1\n" - "\nSILVER1\n" - "") - text_file.close() + self.data.test_ctxt, self.data.test_CG_snapshot, []) - def test_get_port_group_parser(self): - self.create_fake_config_file_parse_port_group() - for _ in range(0, 10): - self.assertEqual( - u'OS-PORTGROUP1-PG', - self.driver.utils.parse_file_to_get_port_group_name( - self.config_file_parse_port_group)) - bExists = os.path.exists(self.config_file_parse_port_group) - if bExists: - os.remove(self.config_file_parse_port_group) + def test_unmanage_no_fast_success(self): + keybindings = {'CreationClassName': u'Symm_StorageVolume', + 'SystemName': u'SYMMETRIX+000195900000', + 'DeviceID': u'1', + 'SystemCreationClassName': u'Symm_StorageSystem'} + provider_location = {'classname': 'Symm_StorageVolume', + 'keybindings': keybindings} + + volume = {'name': 'vol1', + 'size': 1, + 'id': '1', + 'device_id': '1', + 'provider_auth': None, + 'project_id': 'project', + 'display_name': 'vol1', + 'display_description': 'test volume', + 'volume_type_id': 'abc', + 'provider_location': six.text_type(provider_location), + 'status': 'available', + 'host': self.data.fake_host, + 'NumberOfBlocks': 100, + 'BlockSize': self.data.block_size + } + common = self.driver.common + common._initial_setup = mock.Mock( + return_value={'volume_backend_name': 'FCNoFAST', + 'storagetype:fastpolicy': None}) + utils = self.driver.common.utils + utils.rename_volume = mock.Mock(return_value=None) + self.driver.unmanage(volume) + utils.rename_volume.assert_called_once_with( + common.conn, common._find_lun(volume), '1') + + def test_unmanage_no_fast_failed(self): + keybindings = {'CreationClassName': u'Symm_StorageVolume', + 'SystemName': u'SYMMETRIX+000195900000', + 'DeviceID': u'999', + 'SystemCreationClassName': u'Symm_StorageSystem'} + provider_location = {'classname': 'Symm_StorageVolume', + 'keybindings': keybindings} + + volume = {'name': 'NO_SUCH_VOLUME', + 'size': 1, + 'id': '999', + 'device_id': '999', + 'provider_auth': None, + 'project_id': 'project', + 'display_name': 'No such volume', + 'display_description': 'volume not on the array', + 'volume_type_id': 'abc', + 'provider_location': six.text_type(provider_location), + 'status': 'available', + 'host': self.data.fake_host, + 'NumberOfBlocks': 100, + 'BlockSize': self.data.block_size + } + common = self.driver.common + common._initial_setup = mock.Mock( + return_value={'volume_backend_name': 'FCNoFAST', + 'fastpolicy': None}) + self.assertRaises(exception.VolumeBackendAPIException, + self.driver.unmanage, + volume) def _cleanup(self): bExists = os.path.exists(self.config_file_path) @@ -3421,23 +5006,25 @@ def setUp(self): configuration.safe_get.return_value = 'FCFAST' configuration.config_group = 'FCFAST' - self.stubs.Set(EMCVMAXCommon, '_get_ecom_connection', + self.stubs.Set(emc_vmax_common.EMCVMAXCommon, '_get_ecom_connection', self.fake_ecom_connection) instancename = FakeCIMInstanceName() - self.stubs.Set(EMCVMAXUtils, 'get_instance_name', + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'get_instance_name', instancename.fake_getinstancename) - self.stubs.Set(time, 'sleep', - self.fake_sleep) + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'isArrayV3', + self.fake_is_v3) - driver = EMCVMAXFCDriver(configuration=configuration) + driver = emc_vmax_fc.EMCVMAXFCDriver(configuration=configuration) driver.db = FakeDB() driver.common.conn = FakeEcomConnection() driver.zonemanager_lookup_service = None self.driver = driver + self.driver.utils = emc_vmax_utils.EMCVMAXUtils(object) + self.driver.masking = emc_vmax_masking.EMCVMAXMasking('FC') def create_fake_config_file_fast(self): - doc = Document() + doc = minidom.Document() emc = doc.createElement("EMC") doc.appendChild(emc) @@ -3467,7 +5054,7 @@ def create_fake_config_file_fast(self): ecompassword.appendChild(ecompasswordtext) portgroup = doc.createElement("PortGroup") - portgrouptext = doc.createTextNode("myPortGroup") + portgrouptext = doc.createTextNode(self.data.port_group) portgroup.appendChild(portgrouptext) pool = doc.createElement("Pool") @@ -3476,7 +5063,7 @@ def create_fake_config_file_fast(self): pool.appendChild(pooltext) array = doc.createElement("Array") - arraytext = doc.createTextNode("0123456789") + arraytext = doc.createTextNode("1234567891011") emc.appendChild(array) array.appendChild(arraytext) @@ -3501,121 +5088,96 @@ def fake_ecom_connection(self): conn = FakeEcomConnection() return conn - def fake_sleep(self, seconds): - return + def fake_is_v3(self, conn, serialNumber): + return False @mock.patch.object( - EMCVMAXUtils, - 'find_storageSystem', - return_value=None) + emc_vmax_fast.EMCVMAXFast, + 'get_capacities_associated_to_policy', + return_value=(1234, 1200, 1200, 1)) @mock.patch.object( - EMCVMAXFast, - 'is_tiering_policy_enabled', - return_value=True) + emc_vmax_utils.EMCVMAXUtils, + 'get_pool_capacities', + return_value=(1234, 1200, 1200, 1)) @mock.patch.object( - EMCVMAXFast, + emc_vmax_fast.EMCVMAXFast, 'get_tier_policy_by_name', return_value=None) @mock.patch.object( - EMCVMAXFast, - 'get_capacities_associated_to_policy', - return_value=(1234, 1200)) - @mock.patch.object( - EMCVMAXUtils, - 'parse_array_name_from_file', - return_value="123456789") + emc_vmax_fast.EMCVMAXFast, + 'is_tiering_policy_enabled', + return_value=True) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + emc_vmax_utils.EMCVMAXUtils, + 'find_storageSystem', + return_value=None) def test_get_volume_stats_fast(self, mock_storage_system, mock_is_fast_enabled, mock_get_policy, - mock_capacity, - mock_array, - mock_is_v3): + mock_pool_capacities, + mock_capacities_associated_to_policy): self.driver.get_volume_stats(True) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST'}) + emc_vmax_fast.EMCVMAXFast, + 'get_pool_associated_to_policy', + return_value=1) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXFast, - 'get_pool_associated_to_policy', - return_value=1) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST'}) def test_create_volume_fast_success( - self, _mock_volume_type, mock_storage_system, mock_pool_policy, - mock_is_v3): + self, _mock_volume_type, mock_storage_system, mock_pool_policy): self.driver.create_volume(self.data.test_volume_v2) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'storagetype: stripedmetacount': '4', - 'volume_backend_name': 'FCFAST'}) + emc_vmax_fast.EMCVMAXFast, + 'get_pool_associated_to_policy', + return_value=1) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXFast, - 'get_pool_associated_to_policy', - return_value=1) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'storagetype: stripedmetacount': '4', + 'volume_backend_name': 'FCFAST'}) def test_create_volume_fast_striped_success( - self, _mock_volume_type, mock_storage_system, mock_pool_policy, - mock_is_v3): + self, _mock_volume_type, mock_storage_system, mock_pool_policy): self.driver.create_volume(self.data.test_volume_v2) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST'}) - @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) - @mock.patch.object( - EMCVMAXFast, + emc_vmax_fast.EMCVMAXFast, 'get_pool_associated_to_policy', return_value=1) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_create_volume_in_CG_fast_success( - self, _mock_volume_type, mock_storage_system, mock_pool_policy, - mock_is_v3): - self.driver.create_volume(self.data.test_volume_CG) - + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'FCFAST'}) + def test_create_volume_in_CG_fast_success( + self, _mock_volume_type, mock_storage_system, mock_pool_policy): + self.driver.create_volume(self.data.test_volume_CG) + @mock.patch.object( - EMCVMAXMasking, + emc_vmax_masking.EMCVMAXMasking, '_wrap_get_storage_group_from_volume', return_value=None) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST'}) def test_delete_volume_fast_success(self, _mock_volume_type, - mock_storage_group, - mock_is_v3): + mock_storage_group): self.driver.delete_volume(self.data.test_volume) def test_create_volume_fast_failed(self): @@ -3627,14 +5189,8 @@ def test_create_volume_fast_failed(self): volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'FCFAST'}) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_delete_volume_fast_notfound(self, _mock_volume_type, - mock_is_v3): - """We do not set the provider location. - """ + def test_delete_volume_fast_notfound(self, _mock_volume_type): + """"Test delete volume with volume not found.""" notfound_delete_vol = {} notfound_delete_vol['name'] = 'notfound_delete_vol' notfound_delete_vol['id'] = '10' @@ -3642,14 +5198,15 @@ def test_delete_volume_fast_notfound(self, _mock_volume_type, notfound_delete_vol['SystemName'] = self.data.storage_system notfound_delete_vol['DeviceID'] = notfound_delete_vol['id'] notfound_delete_vol['SystemCreationClassName'] = 'Symm_StorageSystem' + notfound_delete_vol['host'] = self.data.fake_host name = {} name['classname'] = 'Symm_StorageVolume' keys = {} keys['CreationClassName'] = notfound_delete_vol['CreationClassName'] keys['SystemName'] = notfound_delete_vol['SystemName'] keys['DeviceID'] = notfound_delete_vol['DeviceID'] - keys['SystemCreationClassName'] = \ - notfound_delete_vol['SystemCreationClassName'] + keys['SystemCreationClassName'] = ( + notfound_delete_vol['SystemCreationClassName']) name['keybindings'] = keys notfound_delete_vol['volume_type_id'] = 'abc' notfound_delete_vol['provider_location'] = None @@ -3657,56 +5214,49 @@ def test_delete_volume_fast_notfound(self, _mock_volume_type, self.driver.delete_volume(notfound_delete_vol) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST'}) - @mock.patch.object( - EMCVMAXMasking, - '_wrap_get_storage_group_from_volume', - return_value=None) + emc_vmax_fast.EMCVMAXFast, + 'get_pool_associated_to_policy', + return_value=1) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXFast, - 'get_pool_associated_to_policy', - return_value=1) + emc_vmax_masking.EMCVMAXMasking, + '_wrap_get_storage_group_from_volume', + return_value=None) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST'}) def test_delete_volume_fast_failed( self, _mock_volume_type, mock_wrapper, - mock_storage_system, mock_pool_policy, - mock_is_v3): + mock_storage_system, mock_pool_policy): self.driver.create_volume(self.data.failed_delete_vol) self.assertRaises(exception.VolumeBackendAPIException, self.driver.delete_volume, self.data.failed_delete_vol) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST', - 'FASTPOLICY': 'FC_GOLD1'}) + emc_vmax_common.EMCVMAXCommon, + '_is_same_host', + return_value=True) @mock.patch.object( - EMCVMAXMasking, + emc_vmax_masking.EMCVMAXMasking, 'get_masking_view_from_storage_group', return_value=EMCVMAXCommonData.lunmaskctrl_name) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - @mock.patch.object( - EMCVMAXCommon, - '_is_same_host', - return_value=True) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST', + 'FASTPOLICY': 'FC_GOLD1'}) def test_map_fast_success(self, _mock_volume_type, mock_maskingview, - mock_is_v3, mock_is_same_host): + mock_is_same_host): common = self.driver.common common.get_target_wwns = mock.Mock( return_value=EMCVMAXCommonData.target_wwns) + self.driver.common._get_correct_port_group = mock.Mock( + return_value=self.data.port_group) data = self.driver.initialize_connection( self.data.test_volume, self.data.connector) # Test the no lookup service, pre-zoned case. @@ -3715,50 +5265,48 @@ def test_map_fast_success(self, _mock_volume_type, mock_maskingview, for init, target in data['data']['initiator_target_map'].items(): self.assertIn(init[::-1], target) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + 'find_device_number', + return_value={'Name': "0001"}) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'FCFAST', 'FASTPOLICY': 'FC_GOLD1'}) - @mock.patch.object( - EMCVMAXCommon, - 'find_device_number', - return_value={'Name': "0001"}) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_map_fast_failed(self, mock_wrap_group, mock_maskingview, - mock_is_v3): + def test_map_fast_failed(self, _mock_volume_type, mock_wrap_device): self.assertRaises(exception.VolumeBackendAPIException, self.driver.initialize_connection, self.data.test_volume, self.data.connector) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST', - 'FASTPOLICY': 'FC_GOLD1'}) + emc_vmax_common.EMCVMAXCommon, + 'check_ig_instance_name', + return_value='myInitGroup') @mock.patch.object( - EMCVMAXMasking, - 'get_masking_view_by_volume', - return_value=EMCVMAXCommonData.lunmaskctrl_name) + emc_vmax_common.EMCVMAXCommon, + 'get_masking_views_by_port_group', + return_value=[]) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + emc_vmax_masking.EMCVMAXMasking, + 'get_initiator_group_from_masking_view', + return_value='myInitGroup') @mock.patch.object( - EMCVMAXMasking, + emc_vmax_masking.EMCVMAXMasking, '_find_initiator_masking_group', return_value='myInitGroup') @mock.patch.object( - EMCVMAXMasking, - 'get_initiator_group_from_masking_view', - return_value='myInitGroup') - def test_detach_fast_success( - self, mock_volume_type, mock_maskingview, mock_is_v3, mock_ig, - mock_igc): + emc_vmax_masking.EMCVMAXMasking, + 'get_masking_view_by_volume', + return_value=EMCVMAXCommonData.lunmaskctrl_name) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST', + 'FASTPOLICY': 'FC_GOLD1'}) + def test_detach_fast_success(self, mock_volume_type, mock_maskingview, + mock_ig, mock_igc, mock_mv, mock_check_ig): common = self.driver.common common.get_target_wwns = mock.Mock( return_value=EMCVMAXCommonData.target_wwns) @@ -3766,42 +5314,33 @@ def test_detach_fast_success( self.data.connector) common.get_target_wwns.assert_called_once_with( EMCVMAXCommonData.storage_system, EMCVMAXCommonData.connector) + numTargetWwns = len(EMCVMAXCommonData.target_wwns) + self.assertEqual(numTargetWwns, len(data['data'])) - self.assertEqual(0, len(data['data'])) - - @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST'}) @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'get_volume_size', return_value='2147483648') @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST'}) def test_extend_volume_fast_success(self, _mock_volume_type, - _mock_volume_size, - mock_is_v3): + _mock_volume_size): newSize = '2' self.driver.extend_volume(self.data.test_volume, newSize) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST'}) - @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'check_if_volume_is_extendable', return_value='False') @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_extend_volume_striped_fast_failed(self, _mock_volume_type, - _mock_is_extendable, - mock_is_v3): + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST'}) + def test_extend_volume_striped_fast_failed(self, + _mock_volume_type, + _mock_is_extendable): newSize = '2' self.assertRaises(exception.VolumeBackendAPIException, self.driver.extend_volume, @@ -3809,35 +5348,31 @@ def test_extend_volume_striped_fast_failed(self, _mock_volume_type, newSize) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST'}) - @mock.patch.object( - FakeDB, - 'volume_get', - return_value=EMCVMAXCommonData.test_source_volume) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - @mock.patch.object( - EMCVMAXFast, + emc_vmax_fast.EMCVMAXFast, 'get_pool_associated_to_policy', return_value=1) @mock.patch.object( - EMCVMAXUtils, + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'get_meta_members_capacity_in_byte', + return_value=[1234567, 7654321]) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, 'get_volume_meta_head', return_value=[EMCVMAXCommonData.test_volume]) @mock.patch.object( - EMCVMAXUtils, - 'get_meta_members_capacity_in_bit', - return_value=[1234567, 7654321]) + FakeDB, + 'volume_get', + return_value=EMCVMAXCommonData.test_source_volume) @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST'}) def test_create_snapshot_different_sizes_meta_fast_success( - self, mock_volume_type, mock_volume, mock_is_v3, + self, mock_volume_type, mock_volume, mock_meta, mock_size, mock_pool, mock_policy): self.data.test_volume['volume_name'] = "vmax-1234567" common = self.driver.common @@ -3845,42 +5380,41 @@ def test_create_snapshot_different_sizes_meta_fast_success( volumeDict = {'classname': u'Symm_StorageVolume', 'keybindings': EMCVMAXCommonData.keybindings} common.provision.create_volume_from_pool = ( - mock.Mock(return_value=(volumeDict, 0L))) + mock.Mock(return_value=(volumeDict, 0))) common.provision.get_volume_dict_from_job = ( mock.Mock(return_value=volumeDict)) common.fast.is_volume_in_default_SG = ( mock.Mock(return_value=True)) self.driver.create_snapshot(self.data.test_volume) - def test_create_snapshot_fast_failed(self): + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_validate_pool', + return_value=('Bogus_Pool')) + def test_create_snapshot_fast_failed(self, mock_pool): self.data.test_volume['volume_name'] = "vmax-1234567" self.assertRaises(exception.VolumeBackendAPIException, self.driver.create_snapshot, self.data.test_volume) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST'}) - @mock.patch.object( - EMCVMAXUtils, - 'find_sync_sv_by_target', - return_value=(None, None)) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + emc_vmax_utils.EMCVMAXUtils, + 'get_meta_members_capacity_in_byte', + return_value=[1234567]) @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'get_volume_meta_head', return_value=[EMCVMAXCommonData.test_volume]) @mock.patch.object( - EMCVMAXUtils, - 'get_meta_members_capacity_in_bit', - return_value=[1234567]) + emc_vmax_utils.EMCVMAXUtils, + 'find_sync_sv_by_target', + return_value=(None, None)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST'}) def test_create_volume_from_same_size_meta_snapshot( - self, mock_volume_type, mock_sync_sv, mock_is_v3, - mock_meta, mock_size): + self, mock_volume_type, mock_sync_sv, mock_meta, mock_size): self.data.test_volume['volume_name'] = "vmax-1234567" common = self.driver.common common.fast.is_volume_in_default_SG = mock.Mock(return_value=True) @@ -3888,94 +5422,76 @@ def test_create_volume_from_same_size_meta_snapshot( self.data.test_volume, self.data.test_volume) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST', - 'FASTPOLICY': 'FC_GOLD1'}) + emc_vmax_utils.EMCVMAXUtils, + 'find_sync_sv_by_target', + return_value=(None, None)) @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'find_replication_service', return_value=None) @mock.patch.object( - EMCVMAXUtils, - 'find_sync_sv_by_target', - return_value=(None, None)) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST', + 'FASTPOLICY': 'FC_GOLD1'}) def test_create_volume_from_snapshot_fast_failed( - self, mock_volume_type, - mock_rep_service, mock_sync_sv, mock_is_v3): + self, mock_volume_type, mock_rep_service, mock_sync_sv): self.data.test_volume['volume_name'] = "vmax-1234567" self.assertRaises(exception.VolumeBackendAPIException, self.driver.create_volume_from_snapshot, self.data.test_volume, EMCVMAXCommonData.test_source_volume) - @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST'}) - @mock.patch.object( - FakeDB, - 'volume_get', - return_value=EMCVMAXCommonData.test_source_volume) - @mock.patch.object( - EMCVMAXCommon, - '_find_storage_sync_sv_sv', - return_value=(None, None)) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - @mock.patch.object( - EMCVMAXUtils, - 'get_volume_meta_head', - return_value=None) - def test_create_clone_simple_volume_fast_success( - self, mock_volume_type, mock_volume, mock_sync_sv, - mock_is_v3, mock_simple_volume): + def test_create_clone_simple_volume_fast_success(self): + extraSpecs = {'storagetype:fastpolicy': 'FC_GOLD1', + 'volume_backend_name': 'FCFAST', + 'isV3': False} + self.driver.common._initial_setup = ( + mock.Mock(return_value=extraSpecs)) + self.driver.common.extraSpecs = extraSpecs + self.driver.utils.is_clone_licensed = ( + mock.Mock(return_value=True)) + FakeDB.volume_get = ( + mock.Mock(return_value=EMCVMAXCommonData.test_source_volume)) self.data.test_volume['volume_name'] = "vmax-1234567" self.driver.common.fast.is_volume_in_default_SG = ( mock.Mock(return_value=True)) + self.driver.utils.isArrayV3 = mock.Mock(return_value=False) + self.driver.common._find_storage_sync_sv_sv = ( + mock.Mock(return_value=(None, None))) self.driver.create_cloned_volume(self.data.test_volume, EMCVMAXCommonData.test_source_volume) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST'}) + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - FakeDB, - 'volume_get', - return_value=EMCVMAXCommonData.test_source_volume) + emc_vmax_utils.EMCVMAXUtils, + 'get_meta_members_capacity_in_byte', + return_value=[1234567, 7654321]) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + emc_vmax_utils.EMCVMAXUtils, + 'get_volume_meta_head', + return_value=[EMCVMAXCommonData.test_volume]) @mock.patch.object( - EMCVMAXFast, + emc_vmax_fast.EMCVMAXFast, 'get_pool_associated_to_policy', return_value=1) @mock.patch.object( - EMCVMAXUtils, - 'get_volume_meta_head', - return_value=[EMCVMAXCommonData.test_volume]) - @mock.patch.object( - EMCVMAXUtils, - 'get_meta_members_capacity_in_bit', - return_value=[1234567, 7654321]) + FakeDB, + 'volume_get', + return_value=EMCVMAXCommonData.test_source_volume) @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST'}) def test_create_clone_fast_failed( - self, mock_volume_type, mock_vol, mock_is_v3, mock_policy, + self, mock_volume_type, mock_vol, mock_policy, mock_meta, mock_size, mock_pool): self.data.test_volume['volume_name'] = "vmax-1234567" self.driver.common._modify_and_get_composite_volume_instance = ( - mock.Mock(return_value=(1L, None))) + mock.Mock(return_value=(1, None))) self.assertRaises(exception.VolumeBackendAPIException, self.driver.create_cloned_volume, self.data.test_volume, @@ -3985,143 +5501,147 @@ def test_create_clone_fast_failed( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'FCFAST'}) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) - def test_migrate_volume_fast_success(self, _mock_volume_type, mock_is_v3): + def test_migrate_volume_fast_success(self, _mock_volume_type): self.driver.migrate_volume(self.data.test_ctxt, self.data.test_volume, self.data.test_host) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST'}) + emc_vmax_masking.EMCVMAXMasking, + '_wrap_get_storage_group_from_volume', + return_value=None) @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'parse_pool_instance_id', return_value=('silver', 'SYMMETRIX+000195900551')) @mock.patch.object( - EMCVMAXMasking, - '_wrap_get_storage_group_from_volume', - return_value=None) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST'}) def test_retype_volume_fast_success( - self, _mock_volume_type, mock_values, mock_wrap, mock_is_v3): + self, _mock_volume_type, mock_values, mock_wrap): self.driver.retype( self.data.test_ctxt, self.data.test_volume, self.data.new_type, self.data.diff, self.data.test_host) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST'}) def test_create_CG_fast_success( - self, _mock_volume_type, _mock_storage_system, mock_is_v3): + self, _mock_volume_type, _mock_storage_system): self.driver.create_consistencygroup( self.data.test_ctxt, self.data.test_CG) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST'}) - @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) + emc_vmax_common.EMCVMAXCommon, + '_get_members_of_replication_group', + return_value=None) @mock.patch.object( FakeDB, 'volume_get_all_by_group', return_value=None) @mock.patch.object( - EMCVMAXCommon, - '_get_members_of_replication_group', - return_value=None) + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST'}) def test_delete_CG_no_volumes_fast_success( self, _mock_volume_type, _mock_storage_system, - _mock_db_volumes, _mock_members, mock_is_v3): + _mock_db_volumes, _mock_members): self.driver.delete_consistencygroup( - self.data.test_ctxt, self.data.test_CG) + self.data.test_ctxt, self.data.test_CG, []) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST'}) def test_delete_CG_with_volumes_fast_success( - self, _mock_volume_type, _mock_storage_system, - mock_is_v3): + self, _mock_volume_type, _mock_storage_system): self.driver.delete_consistencygroup( - self.data.test_ctxt, self.data.test_CG) + self.data.test_ctxt, self.data.test_CG, []) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST'}) + emc_vmax_utils.EMCVMAXUtils, + 'find_group_sync_rg_by_target', + return_value="") @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) + emc_vmax_common.EMCVMAXCommon, + '_get_members_of_replication_group', + return_value=()) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_find_consistency_group', return_value=(None, EMCVMAXCommonData.test_CG)) @mock.patch.object( - EMCVMAXCommon, - '_get_members_of_replication_group', - return_value=()) - @mock.patch.object( - EMCVMAXUtils, - 'find_group_sync_rg_by_target', - return_value="") + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST'}) def test_create_snapshot_for_CG_no_fast_success( self, _mock_volume_type, _mock_storage, _mock_cg, _mock_members, - _mock_rg, mock_is_v3): + _mock_rg): self.driver.create_cgsnapshot( - self.data.test_ctxt, self.data.test_CG_snapshot) + self.data.test_ctxt, self.data.test_CG_snapshot, []) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'FCFAST'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=False) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'FCFAST'}) def test_delete_snapshot_for_CG_no_fast_success( - self, _mock_volume_type, _mock_storage, mock_is_v3): + self, _mock_volume_type, _mock_storage): self.driver.delete_cgsnapshot( - self.data.test_ctxt, self.data.test_CG_snapshot) + self.data.test_ctxt, self.data.test_CG_snapshot, []) + + # Bug 1385450 + def test_create_clone_without_license(self): + mockRepServCap = {} + mockRepServCap['InstanceID'] = 'SYMMETRIX+1385450' + self.driver.utils.find_replication_service_capabilities = ( + mock.Mock(return_value=mockRepServCap)) + self.driver.utils.is_clone_licensed = ( + mock.Mock(return_value=False)) + self.assertRaises(exception.VolumeBackendAPIException, + self.driver.create_cloned_volume, + self.data.test_volume, + EMCVMAXCommonData.test_source_volume) + + def test_manage_existing_fast_failed(self): + volume = {} + metadata = {'key': 'array', + 'value': '12345'} + poolInstanceName = {} + storageSystem = {} + poolInstanceName['InstanceID'] = "SATA_GOLD1" + storageSystem['InstanceID'] = "SYMMETRIX+00019870000" + volume['volume_metadata'] = [metadata] + volume['name'] = "test-volume" + external_ref = {'source-name': '0123'} + common = self.driver.common + common._initial_setup = mock.Mock( + return_value={'volume_backend_name': 'FCFAST', + 'storagetype:fastpolicy': 'GOLD'}) + self.assertRaises(exception.VolumeBackendAPIException, + self.driver.manage_existing, + volume, + external_ref) def _cleanup(self): bExists = os.path.exists(self.config_file_path) @@ -4141,29 +5661,31 @@ def setUp(self): self.tempdir = tempfile.mkdtemp() super(EMCV3DriverTestCase, self).setUp() self.config_file_path = None - self.create_fake_config_file_fast() + self.create_fake_config_file_v3() self.addCleanup(self._cleanup) + self.set_configuration() + def set_configuration(self): configuration = mock.Mock() configuration.cinder_emc_config_file = self.config_file_path - configuration.safe_get.return_value = 'V3' + configuration.safe_get.return_value = 3 configuration.config_group = 'V3' - self.stubs.Set(EMCVMAXCommon, '_get_ecom_connection', + self.stubs.Set(emc_vmax_common.EMCVMAXCommon, '_get_ecom_connection', self.fake_ecom_connection) instancename = FakeCIMInstanceName() - self.stubs.Set(EMCVMAXUtils, 'get_instance_name', + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'get_instance_name', instancename.fake_getinstancename) - self.stubs.Set(time, 'sleep', - self.fake_sleep) + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'isArrayV3', + self.fake_is_v3) - driver = EMCVMAXFCDriver(configuration=configuration) + driver = emc_vmax_fc.EMCVMAXFCDriver(configuration=configuration) driver.db = FakeDB() self.driver = driver - def create_fake_config_file_fast(self): + def create_fake_config_file_v3(self): - doc = Document() + doc = minidom.Document() emc = doc.createElement("EMC") doc.appendChild(emc) @@ -4188,7 +5710,7 @@ def create_fake_config_file_fast(self): ecompassword.appendChild(ecompasswordtext) portgroup = doc.createElement("PortGroup") - portgrouptext = doc.createTextNode("myPortGroup") + portgrouptext = doc.createTextNode(self.data.port_group) portgroup.appendChild(portgrouptext) pool = doc.createElement("Pool") @@ -4197,7 +5719,7 @@ def create_fake_config_file_fast(self): pool.appendChild(pooltext) array = doc.createElement("Array") - arraytext = doc.createTextNode("0123456789") + arraytext = doc.createTextNode("1234567891011") emc.appendChild(array) array.appendChild(arraytext) @@ -4232,203 +5754,267 @@ def fake_ecom_connection(self): self.conn = FakeEcomConnection() return self.conn - def fake_sleep(self, seconds): - return + def fake_is_v3(self, conn, serialNumber): + return True + + def default_extraspec(self): + return {'storagetype:pool': 'SRP_1', + 'volume_backend_name': 'V3_BE', + 'storagetype:workload': 'DSS', + 'storagetype:slo': 'Bronze', + 'storagetype:array': '1234567891011', + 'isV3': True, + 'portgroupname': 'OS-portgroup-PG'} + + def default_vol(self): + vol = EMC_StorageVolume() + vol['name'] = self.data.test_volume['name'] + vol['CreationClassName'] = 'Symm_StorageVolume' + vol['ElementName'] = self.data.test_volume['id'] + vol['DeviceID'] = self.data.test_volume['device_id'] + vol['Id'] = self.data.test_volume['id'] + vol['SystemName'] = self.data.storage_system + vol['NumberOfBlocks'] = self.data.test_volume['NumberOfBlocks'] + vol['BlockSize'] = self.data.test_volume['BlockSize'] + # Added vol to vol.path + vol['SystemCreationClassName'] = 'Symm_StorageSystem' + vol.path = vol + vol.path.classname = vol['CreationClassName'] + return vol + + def default_storage_group(self): + storagegroup = {} + storagegroup['CreationClassName'] = ( + self.data.storagegroup_creationclass) + storagegroup['ElementName'] = 'no_masking_view' + return storagegroup @mock.patch.object( - EMCVMAXUtils, - 'find_storageSystem', - return_value=None) - @mock.patch.object( - EMCVMAXUtils, - 'parse_array_name_from_file', - return_value="123456789") + emc_vmax_masking.EMCVMAXMasking, + '_delete_mv_ig_and_sg') + def test_last_vol_in_SG_with_MV(self, mock_delete): + conn = self.fake_ecom_connection() + common = self.driver.common + controllerConfigService = ( + common.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + + extraSpecs = self.default_extraspec() + + storageGroupName = self.data.storagegroupname + storageGroupInstanceName = ( + common.utils.find_storage_masking_group( + conn, controllerConfigService, storageGroupName)) + vol = self.default_vol() + self.assertTrue(common.masking._last_vol_in_SG( + conn, controllerConfigService, storageGroupInstanceName, + storageGroupName, vol, vol['name'], extraSpecs)) + + def test_last_vol_in_SG_no_MV(self): + conn = self.fake_ecom_connection() + controllerConfigService = ( + self.driver.common.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + + extraSpecs = self.default_extraspec() + self.driver.common.masking.get_masking_view_from_storage_group = ( + mock.Mock(return_value=None)) + self.driver.common.masking.utils.get_existing_instance = ( + mock.Mock(return_value=None)) + storagegroup = self.default_storage_group() + + vol = self.default_vol() + self.assertTrue(self.driver.common.masking._last_vol_in_SG( + conn, controllerConfigService, storagegroup, + storagegroup['ElementName'], vol, vol['name'], extraSpecs)) + + def test_last_vol_in_SG_no_MV_fail(self): + self.driver.common.masking.utils.get_existing_instance = ( + mock.Mock(return_value='value')) + conn = self.fake_ecom_connection() + controllerConfigService = ( + self.driver.common.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + + extraSpecs = self.default_extraspec() + vol = self.default_vol() + storagegroup = self.default_storage_group() + storagegroup['ElementName'] = 'no_masking_view' + + self.assertRaises(exception.VolumeBackendAPIException, + self.driver.common.masking._last_vol_in_SG, + conn, controllerConfigService, + storagegroup, storagegroup['ElementName'], vol, + vol['name'], extraSpecs) + @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) + emc_vmax_utils.EMCVMAXUtils, + 'find_storageSystem', + return_value={'Name': EMCVMAXCommonData.storage_system_v3}) def test_get_volume_stats_v3( - self, mock_storage_system, mock_array, mock_is_v3): + self, mock_storage_system): self.driver.get_volume_stats(True) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'V3_BE'}) + emc_vmax_utils.EMCVMAXUtils, + 'get_v3_default_sg_instance_name', + return_value=(None, None, EMCVMAXCommonData.default_sg_instance_name)) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) - @mock.patch.object( - EMCVMAXProvisionV3, - '_get_supported_size_range_for_SLO', - return_value={'MaximumVolumeSize': '30000000000', - 'MinimumVolumeSize': '100000'}) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'V3_BE'}) def test_create_volume_v3_success( - self, _mock_volume_type, mock_storage_system, - mock_is_v3, mock_range): + self, _mock_volume_type, mock_storage_system, mock_sg): + self.data.test_volume_v3['host'] = self.data.fake_host_v3 + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) self.driver.create_volume(self.data.test_volume_v3) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'V3_BE'}) + emc_vmax_utils.EMCVMAXUtils, + 'get_v3_default_sg_instance_name', + return_value=(None, None, EMCVMAXCommonData.default_sg_instance_name)) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) - @mock.patch.object( - EMCVMAXProvisionV3, - '_get_supported_size_range_for_SLO', - return_value={'MaximumVolumeSize': '30000000000', - 'MinimumVolumeSize': '100000'}) - @mock.patch.object( - EMCVMAXUtils, - 'parse_slo_from_file', - return_value='NONE') - def test_create_volume_v3_no_slo_success( - self, _mock_volume_type, mock_storage_system, - mock_is_v3, mock_range, mock_slo): - self.driver.create_volume(self.data.test_volume_v3) - @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'V3_BE'}) - @mock.patch.object( - EMCVMAXCommon, + def test_create_volume_v3_no_slo_success( + self, _mock_volume_type, mock_storage_system, mock_sg): + v3_vol = self.data.test_volume_v3 + v3_vol['host'] = 'HostX@Backend#NONE+SRP_1+1234567891011' + instid = 'SYMMETRIX-+-000197200056-+-NONE:DSS-+-F-+-0-+-SR-+-SRP_1' + storagepoolsetting = ( + {'InstanceID': instid, + 'CreationClassName': 'CIM_StoragePoolSetting'}) + self.driver.common.provisionv3.get_storage_pool_setting = mock.Mock( + return_value=storagepoolsetting) + extraSpecs = {'storagetype:pool': 'SRP_1', + 'volume_backend_name': 'V3_BE', + 'storagetype:workload': 'DSS', + 'storagetype:slo': 'NONE', + 'storagetype:array': '1234567891011', + 'isV3': True, + 'portgroupname': 'OS-portgroup-PG'} + self.driver.common._initial_setup = mock.Mock( + return_value=extraSpecs) + + self.driver.create_volume(v3_vol) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) - @mock.patch.object( - EMCVMAXProvisionV3, - '_get_supported_size_range_for_SLO', - return_value={'MaximumVolumeSize': '30000000000', - 'MinimumVolumeSize': '100000'}) - @mock.patch.object( - EMCVMAXUtils, - 'parse_slo_from_file', - return_value='Bogus') + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'V3_BE'}) def test_create_volume_v3_invalid_slo_failed( - self, _mock_volume_type, mock_storage_system, - mock_is_v3, mock_range, mock_slo): + self, _mock_volume_type, mock_storage_system): + extraSpecs = {'storagetype:pool': 'SRP_1', + 'volume_backend_name': 'V3_BE', + 'storagetype:workload': 'DSS', + 'storagetype:slo': 'Bogus', + 'storagetype:array': '1234567891011', + 'isV3': True, + 'portgroupname': 'OS-portgroup-PG'} + self.driver.common._initial_setup = mock.Mock( + return_value=extraSpecs) + self.assertRaises(exception.VolumeBackendAPIException, self.driver.create_volume, self.data.test_volume) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'V3_BE'}) + emc_vmax_utils.EMCVMAXUtils, + 'get_v3_default_sg_instance_name', + return_value=(None, None, EMCVMAXCommonData.default_sg_instance_name)) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) - @mock.patch.object( - EMCVMAXProvisionV3, - '_get_supported_size_range_for_SLO', - return_value={'MaximumVolumeSize': '30000000000', - 'MinimumVolumeSize': '100000'}) - def test_create_volume_in_CG_v3_success( - self, _mock_volume_type, mock_storage_system, - mock_is_v3, mock_range): - self.driver.create_volume(self.data.test_volume_CG) - @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'V3_BE'}) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) - def test_delete_volume_v3_success( - self, _mock_volume_type, mock_is_v3): - self.driver.delete_volume(self.data.test_volume) + def test_create_volume_in_CG_v3_success( + self, _mock_volume_type, mock_storage_system, mock_sg): + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.driver.create_volume(self.data.test_volume_CG_v3) - @mock.patch.object( - FakeDB, - 'volume_get', - return_value=EMCVMAXCommonData.test_source_volume) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'V3_BE'}) + def test_delete_volume_v3_success(self, _mock_volume_type): + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.driver.delete_volume(self.data.test_volume_v3) + + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'get_v3_default_sg_instance_name', + return_value=(None, None, EMCVMAXCommonData.default_sg_instance_name)) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) - @mock.patch.object( - EMCVMAXProvisionV3, - '_get_supported_size_range_for_SLO', - return_value={'MaximumVolumeSize': '30000000000', - 'MinimumVolumeSize': '100000'}) - def test_create_snapshot_v3_success( - self, mock_volume_db, mock_type, moke_pool, mock_is_v3, mock_siz): - self.data.test_volume['volume_name'] = "vmax-1234567" - self.driver.create_snapshot(self.data.test_volume) - @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'V3_BE'}) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) @mock.patch.object( FakeDB, 'volume_get', - return_value=EMCVMAXCommonData.test_source_volume) - def test_delete_snapshot_v3_success( - self, mock_volume_type, mock_is_v3, mock_db): - self.data.test_volume['volume_name'] = "vmax-1234567" - self.driver.delete_snapshot(self.data.test_volume) + return_value=EMCVMAXCommonData.test_source_volume_v3) + def test_create_snapshot_v3_success( + self, mock_volume_db, mock_type, moke_pool, mock_sg): + self.data.test_volume_v3['volume_name'] = "vmax-1234567" + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.driver.create_snapshot(self.data.test_volume_v3) @mock.patch.object( FakeDB, 'volume_get', - return_value=EMCVMAXCommonData.test_source_volume) + return_value=EMCVMAXCommonData.test_source_volume_v3) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'V3_BE'}) + def test_delete_snapshot_v3_success(self, mock_volume_type, mock_db): + self.data.test_volume_v3['volume_name'] = "vmax-1234567" + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.driver.delete_snapshot(self.data.test_volume_v3) + @mock.patch.object( - EMCVMAXCommon, + emc_vmax_utils.EMCVMAXUtils, + 'get_v3_default_sg_instance_name', + return_value=(None, None, EMCVMAXCommonData.default_sg_instance_name)) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'V3_BE'}) @mock.patch.object( - EMCVMAXProvisionV3, - '_get_supported_size_range_for_SLO', - return_value={'MaximumVolumeSize': '30000000000', - 'MinimumVolumeSize': '100000'}) + FakeDB, + 'volume_get', + return_value=EMCVMAXCommonData.test_source_volume) def test_create_cloned_volume_v3_success( - self, mock_volume_db, mock_type, moke_pool, mock_is_v3, mock_size): - self.data.test_volume['volume_name'] = "vmax-1234567" - + self, mock_volume_db, mock_type, moke_pool, mock_sg): + self.data.test_volume_v3['volume_name'] = "vmax-1234567" cloneVol = {} cloneVol['name'] = 'vol1' cloneVol['id'] = '10' @@ -4439,217 +6025,236 @@ def test_create_cloned_volume_v3_success( cloneVol['volume_type_id'] = 'abc' cloneVol['provider_location'] = None cloneVol['NumberOfBlocks'] = 100 - cloneVol['BlockSize'] = 512 - name = {} - name['classname'] = 'Symm_StorageVolume' - keys = {} - keys['CreationClassName'] = cloneVol['CreationClassName'] - keys['SystemName'] = cloneVol['SystemName'] - keys['DeviceID'] = cloneVol['DeviceID'] - keys['NumberOfBlocks'] = cloneVol['NumberOfBlocks'] - keys['BlockSize'] = cloneVol['BlockSize'] - keys['SystemCreationClassName'] = \ - cloneVol['SystemCreationClassName'] - name['keybindings'] = keys - - self.driver.create_cloned_volume(cloneVol, self.data.test_volume) + cloneVol['BlockSize'] = self.data.block_size + cloneVol['host'] = self.data.fake_host_v3 + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.driver.create_cloned_volume(cloneVol, self.data.test_volume_v3) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'V3_BE'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) - def test_create_CG_v3_success( - self, _mock_volume_type, _mock_storage_system, mock_is_v3): - self.driver.create_consistencygroup( - self.data.test_ctxt, self.data.test_CG) - @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'V3_BE'}) + def test_create_CG_v3_success( + self, _mock_volume_type, _mock_storage_system): + self.driver.create_consistencygroup( + self.data.test_ctxt, self.data.test_volume_CG_v3) + @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) + emc_vmax_common.EMCVMAXCommon, + '_get_members_of_replication_group', + return_value=None) @mock.patch.object( FakeDB, 'volume_get_all_by_group', return_value=None) @mock.patch.object( - EMCVMAXCommon, - '_get_members_of_replication_group', - return_value=None) + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'V3_BE'}) def test_delete_CG_no_volumes_v3_success( self, _mock_volume_type, _mock_storage_system, - _mock_db_volumes, _mock_members, mock_is_v3): + _mock_db_volumes, _mock_members): self.driver.delete_consistencygroup( - self.data.test_ctxt, self.data.test_CG) + self.data.test_ctxt, self.data.test_CG, []) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'V3_BE'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'V3_BE'}) def test_delete_CG_with_volumes_v3_success( - self, _mock_volume_type, _mock_storage_system, - mock_is_v3): + self, _mock_volume_type, _mock_storage_system): self.driver.delete_consistencygroup( - self.data.test_ctxt, self.data.test_CG) + self.data.test_ctxt, self.data.test_CG, []) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'V3_BE'}) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) - def test_migrate_volume_v3_success(self, _mock_volume_type, mock_is_v3): + def test_migrate_volume_v3_success(self, _mock_volume_type): + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) self.driver.migrate_volume(self.data.test_ctxt, self.data.test_volume, self.data.test_host) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'V3_BE'}) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) + emc_vmax_provision.EMCVMAXProvision, + 'add_members_to_masking_group', + return_value=0) @mock.patch.object( - EMCVMAXUtils, - '_get_fast_settings_from_storage_group', - return_value='Gold+DSS_REP') + emc_vmax_provision_v3.EMCVMAXProvisionV3, + '_find_new_storage_group', + return_value='Any') @mock.patch.object( - EMCVMAXUtils, + emc_vmax_utils.EMCVMAXUtils, 'wrap_get_storage_group_from_volume', return_value=None) @mock.patch.object( - EMCVMAXProvisionV3, - '_find_new_storage_group', - return_value='Any') + emc_vmax_utils.EMCVMAXUtils, + '_get_fast_settings_from_storage_group', + return_value='Gold+DSS_REP') + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'V3_BE'}) def test_retype_volume_v3_success( - self, _mock_volume_type, mock_is_v3, mock_fast_settings, - mock_storage_group, mock_found_SG): + self, _mock_volume_type, mock_fast_settings, + mock_storage_group, mock_found_SG, mock_add): + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) self.assertTrue(self.driver.retype( - self.data.test_ctxt, self.data.test_volume, self.data.new_type, + self.data.test_ctxt, self.data.test_volume_v3, self.data.new_type, self.data.diff, self.data.test_host_v3)) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + '_get_fast_settings_from_storage_group', + return_value='Bronze+DSS') @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'V3_BE'}) - @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) - @mock.patch.object( - EMCVMAXUtils, - '_get_fast_settings_from_storage_group', - return_value='Bronze+DSS') def test_retype_volume_same_host_failure( - self, _mock_volume_type, mock_is_v3, mock_fast_settings): + self, _mock_volume_type, mock_fast_settings): + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) self.assertFalse(self.driver.retype( - self.data.test_ctxt, self.data.test_volume, self.data.new_type, + self.data.test_ctxt, self.data.test_volume_v3, self.data.new_type, self.data.diff, self.data.test_host_v3)) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'V3_BE'}) + emc_vmax_utils.EMCVMAXUtils, + 'find_group_sync_rg_by_target', + return_value=1) @mock.patch.object( - EMCVMAXCommon, - '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) + emc_vmax_common.EMCVMAXCommon, + '_get_members_of_replication_group', + return_value=()) @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_find_consistency_group', return_value=(None, EMCVMAXCommonData.test_CG)) @mock.patch.object( - EMCVMAXCommon, - '_get_members_of_replication_group', - return_value=()) - @mock.patch.object( - EMCVMAXUtils, - 'find_group_sync_rg_by_target', - return_value=1) + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'V3_BE'}) def test_create_cgsnapshot_v3_success( self, _mock_volume_type, _mock_storage, _mock_cg, _mock_members, - mock_rg, mock_is_v3): + mock_rg): provisionv3 = self.driver.common.provisionv3 - provisionv3.create_group_replica = mock.Mock(return_value=(0L, None)) + provisionv3.create_group_replica = mock.Mock(return_value=(0, None)) self.driver.create_cgsnapshot( - self.data.test_ctxt, self.data.test_CG_snapshot) + self.data.test_ctxt, self.data.test_CG_snapshot, []) repServ = self.conn.EnumerateInstanceNames("EMC_ReplicationService")[0] provisionv3.create_group_replica.assert_called_once_with( self.conn, repServ, (None, EMCVMAXCommonData.test_CG), - (None, EMCVMAXCommonData.test_CG), '12de') + (None, EMCVMAXCommonData.test_CG), '12de', + EMCVMAXCommonData.extra_specs) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'V3_BE'}) + def test_delete_cgsnapshot_v3_success( + self, _mock_volume_type, _mock_storage): + self.driver.delete_cgsnapshot( + self.data.test_ctxt, self.data.test_CG_snapshot, []) + @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, '_get_pool_and_storage_system', - return_value=(None, EMCVMAXCommonData.storage_system)) + return_value=(None, EMCVMAXCommonData.storage_system_v3)) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) - def test_delete_cgsnapshot_v3_success( - self, _mock_volume_type, _mock_storage, mock_is_v3): - self.driver.delete_cgsnapshot( - self.data.test_ctxt, self.data.test_CG_snapshot) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'V3_BE'}) + def test_update_CG_add_volume_v3_success( + self, _mock_volume_type, _mock_storage_system): + add_volumes = [] + add_volumes.append(self.data.test_source_volume) + remove_volumes = None + self.driver.update_consistencygroup( + self.data.test_ctxt, self.data.test_CG, + add_volumes, remove_volumes) + # Multiple volumes + add_volumes.append(self.data.test_source_volume) + self.driver.update_consistencygroup( + self.data.test_ctxt, self.data.test_CG, + add_volumes, remove_volumes) + # Can't find CG + self.driver.common._find_consistency_group = mock.Mock( + return_value=None) + self.assertRaises(exception.ConsistencyGroupNotFound, + self.driver.update_consistencygroup, + self.data.test_ctxt, self.data.test_CG, + add_volumes, remove_volumes) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system_v3)) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'V3_BE'}) + def test_update_CG_remove_volume_v3_success( + self, _mock_volume_type, _mock_storage_system): + remove_volumes = [] + remove_volumes.append(self.data.test_source_volume) + add_volumes = None + self.driver.update_consistencygroup( + self.data.test_ctxt, self.data.test_CG, + add_volumes, remove_volumes) + # Multiple volumes + remove_volumes.append(self.data.test_source_volume) + self.driver.update_consistencygroup( + self.data.test_ctxt, self.data.test_CG, + add_volumes, remove_volumes) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_is_same_host', + return_value=True) @mock.patch.object( - EMCVMAXMasking, + emc_vmax_masking.EMCVMAXMasking, 'get_masking_view_from_storage_group', return_value=EMCVMAXCommonData.lunmaskctrl_name) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) - @mock.patch.object( - EMCVMAXCommon, - '_is_same_host', - return_value=True) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'V3_BE'}) def test_map_v3_success( - self, _mock_volume_type, mock_maskingview, - mock_is_v3, mock_is_same_host): + self, _mock_volume_type, mock_maskingview, mock_is_same_host): common = self.driver.common common.get_target_wwns = mock.Mock( return_value=EMCVMAXCommonData.target_wwns) + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.driver.common._get_correct_port_group = mock.Mock( + return_value=self.data.port_group) data = self.driver.initialize_connection( - self.data.test_volume, self.data.connector) + self.data.test_volume_v3, self.data.connector) # Test the no lookup service, pre-zoned case. common.get_target_wwns.assert_called_once_with( EMCVMAXCommonData.storage_system, EMCVMAXCommonData.connector) @@ -4657,59 +6262,2206 @@ def test_map_v3_success( self.assertIn(init[::-1], target) @mock.patch.object( - volume_types, - 'get_volume_type_extra_specs', - return_value={'volume_backend_name': 'V3_BE'}) - @mock.patch.object( - EMCVMAXCommon, + emc_vmax_common.EMCVMAXCommon, 'find_device_number', return_value={'Name': "0001"}) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) - def test_map_v3_failed(self, mock_wrap_group, mock_maskingview, - mock_is_v3): + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'V3_BE'}) + def test_map_v3_failed(self, _mock_volume_type, mock_wrap_device): + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) self.assertRaises(exception.VolumeBackendAPIException, self.driver.initialize_connection, self.data.test_volume, self.data.connector) + @mock.patch.object( + emc_vmax_masking.EMCVMAXMasking, + 'remove_and_reset_members') + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'get_volume_element_name', + return_value='1') + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + 'check_ig_instance_name', + return_value='myInitGroup') + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + 'get_masking_views_by_port_group', + return_value=[]) + @mock.patch.object( + emc_vmax_masking.EMCVMAXMasking, + 'get_initiator_group_from_masking_view', + return_value='myInitGroup') + @mock.patch.object( + emc_vmax_masking.EMCVMAXMasking, + '_find_initiator_masking_group', + return_value='myInitGroup') + @mock.patch.object( + emc_vmax_masking.EMCVMAXMasking, + 'get_masking_view_from_storage_group', + return_value=EMCVMAXCommonData.lunmaskctrl_name) @mock.patch.object( volume_types, 'get_volume_type_extra_specs', return_value={'volume_backend_name': 'V3_BE'}) + def test_detach_v3_success(self, mock_volume_type, mock_maskingview, + mock_ig, mock_igc, mock_mv, mock_check_ig, + mock_element_name, mock_remove): + common = self.driver.common + with mock.patch.object(common, 'get_target_wwns', + return_value=EMCVMAXCommonData.target_wwns): + with mock.patch.object(common, '_initial_setup', + return_value=self.default_extraspec()): + data = self.driver.terminate_connection( + self.data.test_volume_v3, self.data.connector) + common.get_target_wwns.assert_called_once_with( + EMCVMAXCommonData.storage_system, + EMCVMAXCommonData.connector) + numTargetWwns = len(EMCVMAXCommonData.target_wwns) + self.assertEqual(numTargetWwns, len(data['data'])) + + # Bug https://bugs.launchpad.net/cinder/+bug/1440154 + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) @mock.patch.object( - EMCVMAXMasking, - 'get_masking_view_from_storage_group', - return_value=EMCVMAXCommonData.lunmaskctrl_name) + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'V3_BE'}) @mock.patch.object( - EMCVMAXUtils, - 'isArrayV3', - return_value=True) + FakeDB, + 'volume_get', + return_value=EMCVMAXCommonData.test_source_volume_v3) @mock.patch.object( - EMCVMAXMasking, - '_find_initiator_masking_group', - return_value='myInitGroup') + emc_vmax_provision_v3.EMCVMAXProvisionV3, + 'create_element_replica') @mock.patch.object( - EMCVMAXMasking, - 'get_initiator_group_from_masking_view', - return_value='myInitGroup') - def test_detach_v3_success( - self, mock_volume_type, mock_maskingview, mock_is_v3, - mock_ig, mock_igc): + emc_vmax_utils.EMCVMAXUtils, + 'find_sync_sv_by_target', + return_value=(None, None)) + def test_create_clone_v3_assert_clean_up_target_volume( + self, mock_sync, mock_create_replica, mock_volume_db, + mock_type, moke_pool): + self.data.test_volume['volume_name'] = "vmax-1234567" + e = exception.VolumeBackendAPIException('CreateElementReplica Ex') common = self.driver.common - common.get_target_wwns = mock.Mock( - return_value=EMCVMAXCommonData.target_wwns) - data = self.driver.terminate_connection(self.data.test_volume, - self.data.connector) - common.get_target_wwns.assert_called_once_with( - EMCVMAXCommonData.storage_system, EMCVMAXCommonData.connector) - - self.assertEqual(0, len(data['data'])) + volumeDict = {'classname': u'Symm_StorageVolume', + 'keybindings': EMCVMAXCommonData.keybindings} + common._create_v3_volume = ( + mock.Mock(return_value=(0, volumeDict, self.data.storage_system))) + conn = self.fake_ecom_connection() + storageConfigService = [] + storageConfigService = {} + storageConfigService['SystemName'] = EMCVMAXCommonData.storage_system + storageConfigService['CreationClassName'] = ( + self.data.stconf_service_creationclass) + common._delete_from_pool_v3 = mock.Mock(return_value=0) + mock_create_replica.side_effect = e + self.assertRaises(exception.VolumeBackendAPIException, + self.driver.create_cloned_volume, + self.data.test_volume_v3, + EMCVMAXCommonData.test_source_volume_v3) + extraSpecs = common._initial_setup(self.data.test_volume_v3) + targetInstance = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + deviceID = targetInstance['DeviceID'] + common._delete_from_pool_v3.assert_called_with(storageConfigService, + targetInstance, + targetInstance['Name'], + deviceID, + extraSpecs) + + def test_get_remaining_slo_capacity_wlp(self): + conn = self.fake_ecom_connection() + array_info = {'Workload': u'DSS', 'SLO': u'Bronze'} + storagesystem = self.data.storage_system_v3 + srpPoolInstanceName = {} + srpPoolInstanceName['InstanceID'] = ( + self.data.storage_system_v3 + '+U+' + 'SRP_1') + srpPoolInstanceName['CreationClassName'] = ( + 'Symm_VirtualProvisioningPool') + srpPoolInstanceName['ElementName'] = 'SRP_1' + + remainingCapacityGb = ( + self.driver.common.provisionv3._get_remaining_slo_capacity_wlp( + conn, srpPoolInstanceName, array_info, storagesystem)) + remainingSLOCapacityGb = self.driver.common.utils.convert_bits_to_gbs( + self.data.remainingSLOCapacity) + self.assertEqual(remainingSLOCapacityGb, remainingCapacityGb) + + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'get_volume_size', + return_value='2147483648') + def test_extend_volume(self, mock_volume_size): + newSize = '2' + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.driver.extend_volume(self.data.test_volume_v3, newSize) + + def test_extend_volume_smaller_size_exception(self): + test_local_volume = {'name': 'vol1', + 'size': 4, + 'volume_name': 'vol1', + 'id': 'vol1', + 'device_id': '1', + 'provider_auth': None, + 'project_id': 'project', + 'display_name': 'vol1', + 'display_description': 'test volume', + 'volume_type_id': 'abc', + 'provider_location': six.text_type( + self.data.provider_location), + 'status': 'available', + 'host': self.data.fake_host_v3, + 'NumberOfBlocks': 100, + 'BlockSize': self.data.block_size + } + newSize = '2' + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.assertRaises( + exception.VolumeBackendAPIException, + self.driver.extend_volume, + test_local_volume, newSize) + + def test_extend_volume_exception(self): + common = self.driver.common + newsize = '2' + common._initial_setup = mock.Mock(return_value=None) + common._find_lun = mock.Mock(return_value=None) + self.assertRaises( + exception.VolumeBackendAPIException, + common.extend_volume, + self.data.test_volume, newsize) + + def test_extend_volume_size_tally_exception(self): + common = self.driver.common + newsize = '2' + self.driver.common._initial_setup = mock.Mock( + return_value=self.data.extra_specs) + vol = {'SystemName': self.data.storage_system} + common._find_lun = mock.Mock(return_value=vol) + common._extend_v3_volume = mock.Mock(return_value=(0, vol)) + common.utils.find_volume_instance = mock.Mock( + return_value='2147483648') + common.utils.get_volume_size = mock.Mock(return_value='2147483646') + self.assertRaises( + exception.VolumeBackendAPIException, + common.extend_volume, + self.data.test_volume, newsize) def _cleanup(self): bExists = os.path.exists(self.config_file_path) if bExists: os.remove(self.config_file_path) shutil.rmtree(self.tempdir) + + +class EMCV2MultiPoolDriverTestCase(test.TestCase): + + def setUp(self): + self.data = EMCVMAXCommonData() + self.vol_v2 = self.data.test_volume_v2 + self.vol_v2['provider_location'] = ( + six.text_type(self.data.provider_location_multi_pool)) + self.tempdir = tempfile.mkdtemp() + super(EMCV2MultiPoolDriverTestCase, self).setUp() + self.config_file_path = None + self.create_fake_config_file_multi_pool() + self.addCleanup(self._cleanup) + + configuration = mock.Mock() + configuration.safe_get.return_value = 'MULTI_POOL' + configuration.cinder_emc_config_file = self.config_file_path + configuration.config_group = 'MULTI_POOL' + + self.stubs.Set(emc_vmax_common.EMCVMAXCommon, '_get_ecom_connection', + self.fake_ecom_connection) + instancename = FakeCIMInstanceName() + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'get_instance_name', + instancename.fake_getinstancename) + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'isArrayV3', + self.fake_is_v3) + + driver = emc_vmax_iscsi.EMCVMAXISCSIDriver(configuration=configuration) + driver.db = FakeDB() + self.driver = driver + self.driver.utils = emc_vmax_utils.EMCVMAXUtils(object) + + def create_fake_config_file_multi_pool(self): + doc = minidom.Document() + emc = doc.createElement("EMC") + doc.appendChild(emc) + + eComServers = doc.createElement("EcomServers") + emc.appendChild(eComServers) + + eComServer = doc.createElement("EcomServer") + eComServers.appendChild(eComServer) + + ecomserverip = doc.createElement("EcomServerIp") + eComServer.appendChild(ecomserverip) + ecomserveriptext = doc.createTextNode("1.1.1.1") + ecomserverip.appendChild(ecomserveriptext) + + ecomserverport = doc.createElement("EcomServerPort") + eComServer.appendChild(ecomserverport) + ecomserverporttext = doc.createTextNode("10") + ecomserverport.appendChild(ecomserverporttext) + + ecomusername = doc.createElement("EcomUserName") + eComServer.appendChild(ecomusername) + ecomusernametext = doc.createTextNode("user") + ecomusername.appendChild(ecomusernametext) + + ecompassword = doc.createElement("EcomPassword") + eComServer.appendChild(ecompassword) + ecompasswordtext = doc.createTextNode("pass") + ecompassword.appendChild(ecompasswordtext) + + arrays = doc.createElement("Arrays") + eComServer.appendChild(arrays) + + array = doc.createElement("Array") + arrays.appendChild(array) + + serialNo = doc.createElement("SerialNumber") + array.appendChild(serialNo) + serialNoText = doc.createTextNode("1234567891011") + serialNo.appendChild(serialNoText) + + portgroups = doc.createElement("PortGroups") + array.appendChild(portgroups) + + portgroup = doc.createElement("PortGroup") + portgroups.appendChild(portgroup) + portgrouptext = doc.createTextNode(self.data.port_group) + portgroup.appendChild(portgrouptext) + + pools = doc.createElement("Pools") + array.appendChild(pools) + + pool = doc.createElement("Pool") + pools.appendChild(pool) + poolName = doc.createElement("PoolName") + pool.appendChild(poolName) + poolNameText = doc.createTextNode("gold") + poolName.appendChild(poolNameText) + + pool2 = doc.createElement("Pool") + pools.appendChild(pool2) + pool2Name = doc.createElement("PoolName") + pool2.appendChild(pool2Name) + pool2NameText = doc.createTextNode("SATA_BRONZE1") + pool2Name.appendChild(pool2NameText) + pool2FastPolicy = doc.createElement("FastPolicy") + pool2.appendChild(pool2FastPolicy) + pool2FastPolicyText = doc.createTextNode("BRONZE1") + pool2FastPolicy.appendChild(pool2FastPolicyText) + + filename = 'cinder_emc_config_V2_MULTI_POOL.xml' + self.config_file_path = self.tempdir + '/' + filename + + f = open(self.config_file_path, 'w') + doc.writexml(f) + f.close() + + def fake_ecom_connection(self): + self.conn = FakeEcomConnection() + return self.conn + + def fake_is_v3(self, conn, serialNumber): + return False + + def default_extraspec(self): + return {'storagetype:pool': u'gold', + 'volume_backend_name': 'MULTI_POOL_BE', + 'storagetype:fastpolicy': None, + 'storagetype:compositetype': u'concatenated', + 'storagetype:membercount': 1, + 'storagetype:array': u'1234567891011', + 'isV3': False, + 'portgroupname': u'OS-portgroup-PG'} + + def test_validate_pool(self): + v2_valid_pool = self.data.test_volume_v2.copy() + # Pool aware scheduler enabled + v2_valid_pool['host'] = self.data.fake_host + pool = self.driver.common._validate_pool(v2_valid_pool) + self.assertEqual('gold+1234567891011', pool) + + # Cannot get the pool from the host + v2_valid_pool['host'] = 'HostX@Backend' + self.assertRaises(exception.VolumeBackendAPIException, + self.driver.common._validate_pool, + v2_valid_pool) + + # Legacy test. Provider Location does not have the version + v2_valid_pool['host'] = self.data.fake_host + v2_valid_pool['provider_location'] = self.data.provider_location + pool = self.driver.common._validate_pool(v2_valid_pool) + self.assertIsNone(pool) + + def test_array_info_multi_pool(self): + + arrayInfo = self.driver.utils.parse_file_to_get_array_map( + self.config_file_path) + self.assertEqual(2, len(arrayInfo)) + for arrayInfoRec in arrayInfo: + self.assertEqual( + '1234567891011', arrayInfoRec['SerialNumber']) + self.assertIn(self.data.port_group, arrayInfoRec['PortGroup']) + self.assertTrue( + self.data.poolname in arrayInfoRec['PoolName'] or + 'SATA_BRONZE1' in arrayInfoRec['PoolName']) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_POOL_BE'}) + def test_create_volume_multi_pool_success( + self, _mock_volume_type, mock_storage_system): + self.vol_v2['provider_location'] = None + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.driver.create_volume(self.vol_v2) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_POOL_BE'}) + def test_delete_volume_multi_pool_success( + self, _mock_volume_type, mock_storage_system): + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.driver.delete_volume(self.vol_v2) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_POOL_BE'}) + def test_create_volume_in_CG_multi_pool_success( + self, _mock_volume_type, mock_storage_system): + self.data.test_volume_CG['provider_location'] = None + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.driver.create_volume(self.data.test_volume_CG) + + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_POOL_BE'}) + def test_retype_volume_multi_pool_success( + self, _mock_volume_type): + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.driver.retype( + self.data.test_ctxt, self.vol_v2, self.data.new_type, + self.data.diff, self.data.test_host) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_POOL_BE'}) + # There is only one unique array in the conf file + def test_create_CG_multi_pool_success( + self, _mock_volume_type, _mock_storage_system): + self.driver.create_consistencygroup( + self.data.test_ctxt, self.data.test_CG) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_members_of_replication_group', + return_value=None) + @mock.patch.object( + FakeDB, + 'volume_get_all_by_group', + return_value=None) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_POOL_BE'}) + def test_delete_CG_no_volumes_multi_pool_success( + self, _mock_volume_type, _mock_storage_system, + _mock_db_volumes, _mock_members): + self.driver.delete_consistencygroup( + self.data.test_ctxt, self.data.test_CG, []) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_POOL_BE'}) + def test_delete_CG_with_volumes_multi_pool_success( + self, _mock_volume_type, _mock_storage_system): + self.driver.delete_consistencygroup( + self.data.test_ctxt, self.data.test_CG, []) + + def _cleanup(self): + bExists = os.path.exists(self.config_file_path) + if bExists: + os.remove(self.config_file_path) + shutil.rmtree(self.tempdir) + + +class EMCV3MultiSloDriverTestCase(test.TestCase): + + def setUp(self): + self.data = EMCVMAXCommonData() + self.vol_v3 = self.data.test_volume_v3 + self.vol_v3['provider_location'] = ( + six.text_type(self.data.provider_location_multi_pool)) + + self.tempdir = tempfile.mkdtemp() + super(EMCV3MultiSloDriverTestCase, self).setUp() + self.config_file_path = None + self.create_fake_config_file_multi_slo_v3() + self.addCleanup(self._cleanup) + self.set_configuration() + + def set_configuration(self): + configuration = mock.Mock() + configuration.safe_get.return_value = 'MULTI_SLO_V3' + configuration.cinder_emc_config_file = self.config_file_path + configuration.config_group = 'MULTI_SLO_V3' + + self.stubs.Set(emc_vmax_common.EMCVMAXCommon, '_get_ecom_connection', + self.fake_ecom_connection) + instancename = FakeCIMInstanceName() + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'get_instance_name', + instancename.fake_getinstancename) + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'isArrayV3', + self.fake_is_v3) + + driver = emc_vmax_fc.EMCVMAXFCDriver(configuration=configuration) + driver.db = FakeDB() + self.driver = driver + self.driver.utils = emc_vmax_utils.EMCVMAXUtils(object) + + def create_fake_config_file_multi_slo_v3(self): + doc = minidom.Document() + emc = doc.createElement("EMC") + doc.appendChild(emc) + + eComServers = doc.createElement("EcomServers") + emc.appendChild(eComServers) + + eComServer = doc.createElement("EcomServer") + eComServers.appendChild(eComServer) + + ecomserverip = doc.createElement("EcomServerIp") + eComServer.appendChild(ecomserverip) + ecomserveriptext = doc.createTextNode("1.1.1.1") + ecomserverip.appendChild(ecomserveriptext) + + ecomserverport = doc.createElement("EcomServerPort") + eComServer.appendChild(ecomserverport) + ecomserverporttext = doc.createTextNode("10") + ecomserverport.appendChild(ecomserverporttext) + + ecomusername = doc.createElement("EcomUserName") + eComServer.appendChild(ecomusername) + ecomusernametext = doc.createTextNode("user") + ecomusername.appendChild(ecomusernametext) + + ecompassword = doc.createElement("EcomPassword") + eComServer.appendChild(ecompassword) + ecompasswordtext = doc.createTextNode("pass") + ecompassword.appendChild(ecompasswordtext) + + arrays = doc.createElement("Arrays") + eComServer.appendChild(arrays) + + array = doc.createElement("Array") + arrays.appendChild(array) + + serialNo = doc.createElement("SerialNumber") + array.appendChild(serialNo) + serialNoText = doc.createTextNode("1234567891011") + serialNo.appendChild(serialNoText) + + portgroups = doc.createElement("PortGroups") + array.appendChild(portgroups) + + portgroup = doc.createElement("PortGroup") + portgroups.appendChild(portgroup) + portgrouptext = doc.createTextNode(self.data.port_group) + portgroup.appendChild(portgrouptext) + + vpools = doc.createElement("Pools") + array.appendChild(vpools) + vpool = doc.createElement("Pool") + vpools.appendChild(vpool) + poolName = doc.createElement("PoolName") + vpool.appendChild(poolName) + poolNameText = doc.createTextNode("SRP_1") + poolName.appendChild(poolNameText) + poolslo = doc.createElement("SLO") + vpool.appendChild(poolslo) + poolsloText = doc.createTextNode("Bronze") + poolslo.appendChild(poolsloText) + poolworkload = doc.createElement("Workload") + vpool.appendChild(poolworkload) + poolworkloadText = doc.createTextNode("DSS") + poolworkload.appendChild(poolworkloadText) + + vpool2 = doc.createElement("Pool") + vpools.appendChild(vpool2) + pool2Name = doc.createElement("PoolName") + vpool2.appendChild(pool2Name) + pool2NameText = doc.createTextNode("SRP_1") + pool2Name.appendChild(pool2NameText) + pool2slo = doc.createElement("SLO") + vpool2.appendChild(pool2slo) + pool2sloText = doc.createTextNode("Silver") + pool2slo.appendChild(pool2sloText) + pool2workload = doc.createElement("Workload") + vpool.appendChild(pool2workload) + pool2workloadText = doc.createTextNode("OLTP") + pool2workload.appendChild(pool2workloadText) + + filename = 'cinder_emc_config_MULTI_SLO_V3.xml' + self.config_file_path = self.tempdir + '/' + filename + + f = open(self.config_file_path, 'w') + doc.writexml(f) + f.close() + + def fake_ecom_connection(self): + self.conn = FakeEcomConnection() + return self.conn + + def fake_is_v3(self, conn, serialNumber): + return True + + def default_extraspec(self): + return {'storagetype:pool': u'SRP_1', + 'volume_backend_name': 'MULTI_SLO_BE', + 'storagetype:workload': u'DSS', + 'storagetype:slo': u'Bronze', + 'storagetype:array': u'1234567891011', + 'isV3': True, + 'portgroupname': u'OS-portgroup-PG'} + + def test_validate_pool(self): + v3_valid_pool = self.data.test_volume_v3.copy() + # Pool aware scheduler enabled + v3_valid_pool['host'] = self.data.fake_host_v3 + pool = self.driver.common._validate_pool(v3_valid_pool) + self.assertEqual('Bronze+SRP_1+1234567891011', pool) + + # Cannot get the pool from the host + v3_valid_pool['host'] = 'HostX@Backend' + self.assertRaises(exception.VolumeBackendAPIException, + self.driver.common._validate_pool, + v3_valid_pool) + # Legacy test. Provider Location does not have the version + v3_valid_pool['host'] = self.data.fake_host_v3 + v3_valid_pool['provider_location'] = self.data.provider_location + pool = self.driver.common._validate_pool(v3_valid_pool) + self.assertIsNone(pool) + + def test_array_info_multi_slo(self): + + arrayInfo = self.driver.utils.parse_file_to_get_array_map( + self.config_file_path) + self.assertEqual(2, len(arrayInfo)) + for arrayInfoRec in arrayInfo: + self.assertEqual( + '1234567891011', arrayInfoRec['SerialNumber']) + self.assertIn(self.data.port_group, arrayInfoRec['PortGroup']) + self.assertIn('SRP_1', arrayInfoRec['PoolName']) + self.assertTrue( + 'Bronze' in arrayInfoRec['SLO'] or + 'Silver' in arrayInfoRec['SLO']) + + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'get_v3_default_sg_instance_name', + return_value=(None, None, EMCVMAXCommonData.default_sg_instance_name)) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_SLO_BE'}) + def test_create_volume_multi_slo_success( + self, _mock_volume_type, mock_storage_system, mock_sg): + self.vol_v3['host'] = self.data.fake_host_v3 + self.vol_v3['provider_location'] = None + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.driver.create_volume(self.vol_v3) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_SLO_BE'}) + def test_delete_volume_multi_slo_success( + self, _mock_volume_type, mock_storage_system): + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.driver.delete_volume(self.vol_v3) + + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'get_v3_default_sg_instance_name', + return_value=(None, None, EMCVMAXCommonData.default_sg_instance_name)) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_SLO_BE'}) + def test_create_volume_in_CG_multi_slo_success( + self, _mock_volume_type, mock_storage_system, mock_sg): + self.data.test_volume_CG_v3['provider_location'] = None + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.driver.create_volume(self.data.test_volume_CG_v3) + + @mock.patch.object( + emc_vmax_provision.EMCVMAXProvision, + 'add_members_to_masking_group', + return_value=0) + @mock.patch.object( + emc_vmax_provision_v3.EMCVMAXProvisionV3, + '_find_new_storage_group', + return_value='Any') + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'wrap_get_storage_group_from_volume', + return_value=None) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + '_get_fast_settings_from_storage_group', + return_value='Gold+DSS_REP') + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_SLO_BE'}) + def test_retype_volume_multi_slo_success( + self, _mock_volume_type, mock_fast_settings, + mock_storage_group, mock_found_SG, mock_add): + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.assertTrue(self.driver.retype( + self.data.test_ctxt, self.data.test_volume_v3, self.data.new_type, + self.data.diff, self.data.test_host_v3)) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_SLO_BE'}) + # There is only one unique array in the conf file + def test_create_CG_multi_slo_success( + self, _mock_volume_type, _mock_storage_system): + self.driver.common._initial_setup = mock.Mock( + return_value=self.default_extraspec()) + self.driver.create_consistencygroup( + self.data.test_ctxt, self.data.test_CG) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_members_of_replication_group', + return_value=None) + @mock.patch.object( + FakeDB, + 'volume_get_all_by_group', + return_value=None) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_SLO_BE'}) + def test_delete_CG_no_volumes_multi_slo_success( + self, _mock_volume_type, _mock_storage_system, + _mock_db_volumes, _mock_members): + self.driver.delete_consistencygroup( + self.data.test_ctxt, self.data.test_CG, []) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_SLO_BE'}) + def test_delete_CG_with_volumes_multi_slo_success( + self, _mock_volume_type, _mock_storage_system): + self.driver.delete_consistencygroup( + self.data.test_ctxt, self.data.test_CG, []) + + def _cleanup(self): + bExists = os.path.exists(self.config_file_path) + if bExists: + os.remove(self.config_file_path) + shutil.rmtree(self.tempdir) + + +class EMCV2MultiPoolDriverMultipleEcomsTestCase(test.TestCase): + + def setUp(self): + + self.data = EMCVMAXCommonData() + self.vol_v2 = self.data.test_volume_v2 + self.vol_v2['provider_location'] = ( + six.text_type(self.data.provider_location_multi_pool)) + + self.tempdir = tempfile.mkdtemp() + super(EMCV2MultiPoolDriverMultipleEcomsTestCase, self).setUp() + self.config_file_path = None + self.create_fake_config_file_multi_ecom() + self.addCleanup(self._cleanup) + + configuration = mock.Mock() + configuration.cinder_emc_config_file = self.config_file_path + configuration.safe_get.return_value = 'MULTI_ECOM' + configuration.config_group = 'MULTI_ECOM' + + self.stubs.Set(emc_vmax_common.EMCVMAXCommon, '_get_ecom_connection', + self.fake_ecom_connection) + instancename = FakeCIMInstanceName() + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'get_instance_name', + instancename.fake_getinstancename) + self.stubs.Set(emc_vmax_utils.EMCVMAXUtils, 'isArrayV3', + self.fake_is_v3) + driver = emc_vmax_fc.EMCVMAXFCDriver(configuration=configuration) + driver.db = FakeDB() + driver.common.conn = FakeEcomConnection() + driver.zonemanager_lookup_service = FakeLookupService() + self.driver = driver + self.driver.utils = emc_vmax_utils.EMCVMAXUtils(object) + + def create_fake_config_file_multi_ecom(self): + doc = minidom.Document() + emc = doc.createElement("EMC") + doc.appendChild(emc) + + eComServers = doc.createElement("EcomServers") + emc.appendChild(eComServers) + + eComServer = doc.createElement("EcomServer") + eComServers.appendChild(eComServer) + + ecomserverip = doc.createElement("EcomServerIp") + eComServer.appendChild(ecomserverip) + ecomserveriptext = doc.createTextNode("1.1.1.1") + ecomserverip.appendChild(ecomserveriptext) + + ecomserverport = doc.createElement("EcomServerPort") + eComServer.appendChild(ecomserverport) + ecomserverporttext = doc.createTextNode("10") + ecomserverport.appendChild(ecomserverporttext) + + ecomusername = doc.createElement("EcomUserName") + eComServer.appendChild(ecomusername) + ecomusernametext = doc.createTextNode("user") + ecomusername.appendChild(ecomusernametext) + + ecompassword = doc.createElement("EcomPassword") + eComServer.appendChild(ecompassword) + ecompasswordtext = doc.createTextNode("pass") + ecompassword.appendChild(ecompasswordtext) + + arrays = doc.createElement("Arrays") + eComServer.appendChild(arrays) + + array = doc.createElement("Array") + arrays.appendChild(array) + + serialNo = doc.createElement("SerialNumber") + array.appendChild(serialNo) + serialNoText = doc.createTextNode("1110987654321") + serialNo.appendChild(serialNoText) + + portgroups = doc.createElement("PortGroups") + array.appendChild(portgroups) + + portgroup = doc.createElement("PortGroup") + portgroups.appendChild(portgroup) + portgrouptext = doc.createTextNode(self.data.port_group) + portgroup.appendChild(portgrouptext) + + pools = doc.createElement("Pools") + array.appendChild(pools) + + pool = doc.createElement("Pool") + pools.appendChild(pool) + poolName = doc.createElement("PoolName") + pool.appendChild(poolName) + poolNameText = doc.createTextNode("gold") + poolName.appendChild(poolNameText) + + pool2 = doc.createElement("Pool") + pools.appendChild(pool2) + pool2Name = doc.createElement("PoolName") + pool2.appendChild(pool2Name) + pool2NameText = doc.createTextNode("SATA_BRONZE1") + pool2Name.appendChild(pool2NameText) + pool2FastPolicy = doc.createElement("FastPolicy") + pool2.appendChild(pool2FastPolicy) + pool2FastPolicyText = doc.createTextNode("BRONZE1") + pool2FastPolicy.appendChild(pool2FastPolicyText) + + eComServer = doc.createElement("EcomServer") + eComServers.appendChild(eComServer) + + ecomserverip = doc.createElement("EcomServerIp") + eComServer.appendChild(ecomserverip) + ecomserveriptext = doc.createTextNode("1.1.1.1") + ecomserverip.appendChild(ecomserveriptext) + + ecomserverport = doc.createElement("EcomServerPort") + eComServer.appendChild(ecomserverport) + ecomserverporttext = doc.createTextNode("10") + ecomserverport.appendChild(ecomserverporttext) + + ecomusername = doc.createElement("EcomUserName") + eComServer.appendChild(ecomusername) + ecomusernametext = doc.createTextNode("user") + ecomusername.appendChild(ecomusernametext) + + ecompassword = doc.createElement("EcomPassword") + eComServer.appendChild(ecompassword) + ecompasswordtext = doc.createTextNode("pass") + ecompassword.appendChild(ecompasswordtext) + + arrays = doc.createElement("Arrays") + eComServer.appendChild(arrays) + + array = doc.createElement("Array") + arrays.appendChild(array) + + serialNo = doc.createElement("SerialNumber") + array.appendChild(serialNo) + serialNoText = doc.createTextNode("1234567891011") + serialNo.appendChild(serialNoText) + + portgroups = doc.createElement("PortGroups") + array.appendChild(portgroups) + + portgroup = doc.createElement("PortGroup") + portgroups.appendChild(portgroup) + portgrouptext = doc.createTextNode(self.data.port_group) + portgroup.appendChild(portgrouptext) + + pools = doc.createElement("Pools") + array.appendChild(pools) + + pool = doc.createElement("Pool") + pools.appendChild(pool) + poolName = doc.createElement("PoolName") + pool.appendChild(poolName) + poolNameText = doc.createTextNode("gold") + poolName.appendChild(poolNameText) + + pool2 = doc.createElement("Pool") + pools.appendChild(pool2) + pool2Name = doc.createElement("PoolName") + pool2.appendChild(pool2Name) + pool2NameText = doc.createTextNode("SATA_BRONZE1") + pool2Name.appendChild(pool2NameText) + pool2FastPolicy = doc.createElement("FastPolicy") + pool2.appendChild(pool2FastPolicy) + pool2FastPolicyText = doc.createTextNode("BRONZE1") + pool2FastPolicy.appendChild(pool2FastPolicyText) + + filename = 'cinder_emc_config_V2_MULTI_ECOM.xml' + self.config_file_path = self.tempdir + '/' + filename + + f = open(self.config_file_path, 'w') + doc.writexml(f) + f.close() + + def fake_ecom_connection(self): + self.conn = FakeEcomConnection() + return self.conn + + def fake_is_v3(self, conn, serialNumber): + return False + + def test_array_info_multi_ecom_no_fast(self): + pool = 'gold+1234567891011' + arrayInfo = self.driver.utils.parse_file_to_get_array_map( + self.config_file_path) + self.assertEqual(4, len(arrayInfo)) + poolRec = self.driver.utils.extract_record(arrayInfo, pool) + + self.assertEqual('1234567891011', poolRec['SerialNumber']) + self.assertEqual(self.data.port_group, poolRec['PortGroup']) + self.assertEqual(self.data.poolname, poolRec['PoolName']) + self.assertEqual('user', poolRec['EcomUserName']) + self.assertEqual('pass', poolRec['EcomPassword']) + self.assertIsNone(poolRec['FastPolicy']) + self.assertFalse(poolRec['EcomUseSSL']) + + def test_array_info_multi_ecom_fast(self): + pool = 'SATA_BRONZE1+1234567891011' + + arrayInfo = self.driver.utils.parse_file_to_get_array_map( + self.config_file_path) + self.assertEqual(4, len(arrayInfo)) + poolRec = self.driver.utils.extract_record(arrayInfo, pool) + + self.assertEqual('1234567891011', poolRec['SerialNumber']) + self.assertEqual(self.data.port_group, poolRec['PortGroup']) + self.assertEqual('SATA_BRONZE1', poolRec['PoolName']) + self.assertEqual('user', poolRec['EcomUserName']) + self.assertEqual('pass', poolRec['EcomPassword']) + self.assertEqual('BRONZE1', poolRec['FastPolicy']) + self.assertFalse(poolRec['EcomUseSSL']) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_ECOM_BE'}) + def test_create_volume_multi_ecom_success( + self, _mock_volume_type, mock_storage_system): + self.vol_v2['provider_location'] = None + self.driver.create_volume(self.vol_v2) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_ECOM_BE'}) + # If there are more than one unique arrays in conf file + def test_create_CG_multi_array_failure( + self, _mock_volume_type, _mock_storage_system): + self.assertRaises(exception.VolumeBackendAPIException, + self.driver.create_consistencygroup, + self.data.test_ctxt, + self.data.test_CG) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_members_of_replication_group', + return_value=None) + @mock.patch.object( + FakeDB, + 'volume_get_all_by_group', + return_value=None) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_ECOM_BE'}) + # There is more than one unique arrays in the conf file + def test_delete_CG_no_volumes_multi_array_failure( + self, _mock_volume_type, _mock_storage_system, + _mock_db_volumes, _mock_members): + self.assertRaises(exception.VolumeBackendAPIException, + self.driver.delete_consistencygroup, + self.data.test_ctxt, + self.data.test_CG, + []) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + @mock.patch.object( + volume_types, + 'get_volume_type_extra_specs', + return_value={'volume_backend_name': 'MULTI_ECOM_BE'}) + def test_create_volume_in_CG_multi_ecom_success( + self, _mock_volume_type, mock_storage_system): + self.data.test_volume_CG['provider_location'] = None + self.driver.create_volume(self.data.test_volume_CG) + + def _cleanup(self): + bExists = os.path.exists(self.config_file_path) + if bExists: + os.remove(self.config_file_path) + shutil.rmtree(self.tempdir) + + +class EMCVMAXProvisionV3Test(test.TestCase): + def setUp(self): + self.data = EMCVMAXCommonData() + + super(EMCVMAXProvisionV3Test, self).setUp() + + configuration = mock.Mock() + configuration.safe_get.return_value = 'ProvisionV3Tests' + configuration.config_group = 'ProvisionV3Tests' + emc_vmax_common.EMCVMAXCommon._gather_info = mock.Mock() + driver = emc_vmax_iscsi.EMCVMAXISCSIDriver(configuration=configuration) + driver.db = FakeDB() + self.driver = driver + + def test_get_storage_pool_setting(self): + provisionv3 = self.driver.common.provisionv3 + conn = FakeEcomConnection() + slo = 'Bronze' + workload = 'DSS' + poolInstanceName = {} + poolInstanceName['InstanceID'] = "SATA_GOLD1" + poolInstanceName['CreationClassName'] = ( + self.data.storagepool_creationclass) + + storagePoolCapability = provisionv3.get_storage_pool_capability( + conn, poolInstanceName) + storagepoolsetting = provisionv3.get_storage_pool_setting( + conn, storagePoolCapability, slo, workload) + self.assertIn('Bronze:DSS', storagepoolsetting['InstanceID']) + + def test_get_storage_pool_setting_exception(self): + provisionv3 = self.driver.common.provisionv3 + conn = FakeEcomConnection() + slo = 'Bronze' + workload = 'NONE' + poolInstanceName = {} + poolInstanceName['InstanceID'] = "SATA_GOLD1" + poolInstanceName['CreationClassName'] = ( + self.data.storagepool_creationclass) + + storagePoolCapability = provisionv3.get_storage_pool_capability( + conn, poolInstanceName) + self.assertRaises(exception.VolumeBackendAPIException, + provisionv3.get_storage_pool_setting, + conn, storagePoolCapability, slo, workload) + + def test_extend_volume_in_SG(self): + provisionv3 = self.driver.common.provisionv3 + conn = FakeEcomConnection() + storageConfigService = { + 'CreationClassName': 'Symm_ElementCompositionService', + 'SystemName': 'SYMMETRIX+000195900551'} + theVolumeInstanceName = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + inVolumeInstanceName = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + volumeSize = 3 + + extraSpecs = {'volume_backend_name': 'GOLD_BE', + 'isV3': True} + job = { + 'Job': {'InstanceID': '9999', 'status': 'success', 'type': None}} + conn.InvokeMethod = mock.Mock(return_value=(4096, job)) + provisionv3.utils.wait_for_job_complete = mock.Mock(return_value=( + 0, 'Success')) + volumeDict = {'classname': u'Symm_StorageVolume', + 'keybindings': EMCVMAXCommonData.keybindings} + provisionv3.get_volume_dict_from_job = ( + mock.Mock(return_value=volumeDict)) + result = provisionv3.extend_volume_in_SG(conn, storageConfigService, + theVolumeInstanceName, + inVolumeInstanceName, + volumeSize, extraSpecs) + self.assertEqual( + ({'classname': u'Symm_StorageVolume', + 'keybindings': { + 'CreationClassName': u'Symm_StorageVolume', + 'DeviceID': u'1', + 'SystemCreationClassName': u'Symm_StorageSystem', + 'SystemName': u'SYMMETRIX+000195900551'}}, 0), result) + + def test_extend_volume_in_SG_with_Exception(self): + provisionv3 = self.driver.common.provisionv3 + conn = FakeEcomConnection() + storageConfigService = { + 'CreationClassName': 'Symm_ElementCompositionService', + 'SystemName': 'SYMMETRIX+000195900551'} + theVolumeInstanceName = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + inVolumeInstanceName = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + volumeSize = 3 + + extraSpecs = {'volume_backend_name': 'GOLD_BE', + 'isV3': True} + job = { + 'Job': {'InstanceID': '9999', 'status': 'success', 'type': None}} + conn.InvokeMethod = mock.Mock(return_value=(4096, job)) + provisionv3.utils.wait_for_job_complete = mock.Mock(return_value=( + 2, 'Failure')) + self.assertRaises( + exception.VolumeBackendAPIException, + provisionv3.extend_volume_in_SG, conn, storageConfigService, + theVolumeInstanceName, inVolumeInstanceName, volumeSize, + extraSpecs) + + +class EMCVMAXMaskingTest(test.TestCase): + def setUp(self): + self.data = EMCVMAXCommonData() + + super(EMCVMAXMaskingTest, self).setUp() + + configuration = mock.Mock() + configuration.safe_get.return_value = 'MaskingTests' + configuration.config_group = 'MaskingTests' + emc_vmax_common.EMCVMAXCommon._get_ecom_connection = mock.Mock( + return_value=self.fake_ecom_connection()) + emc_vmax_common.EMCVMAXCommon._gather_info = mock.Mock( + return_value=self.fake_gather_info()) + instancename = FakeCIMInstanceName() + emc_vmax_utils.EMCVMAXUtils.get_instance_name = ( + instancename.fake_getinstancename) + driver = emc_vmax_iscsi.EMCVMAXISCSIDriver(configuration=configuration) + driver.db = FakeDB() + self.driver = driver + self.driver.utils = emc_vmax_utils.EMCVMAXUtils(object) + + def fake_ecom_connection(self): + conn = FakeEcomConnection() + return conn + + def fake_gather_info(self): + return + + def test_get_v3_default_storage_group_instance_name(self): + masking = self.driver.common.masking + conn = self.fake_ecom_connection() + extraSpecs = self.data.extra_specs + masking._get_and_remove_from_storage_group_v3 = mock.Mock() + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + maskingviewdict = self.driver.common._populate_masking_dict( + self.data.test_volume, self.data.connector, extraSpecs) + result = ( + masking._get_v3_default_storagegroup_instancename( + conn, maskingviewdict['volumeInstance'], + maskingviewdict, + controllerConfigService, maskingviewdict['volumeName'])) + self.assertEqual('OS-SRP_1-Bronze-DSS-SG', result['ElementName']) + + def test_get_v3_default_storage_group_instance_name_warning(self): + masking = self.driver.common.masking + conn = self.fake_ecom_connection() + extraSpecs = self.data.extra_specs + masking.utils.get_storage_groups_from_volume = mock.Mock( + return_value=[]) + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + maskingviewdict = self.driver.common._populate_masking_dict( + self.data.test_volume, self.data.connector, extraSpecs) + result = ( + masking._get_v3_default_storagegroup_instancename( + conn, maskingviewdict['volumeInstance'], + maskingviewdict, + controllerConfigService, maskingviewdict['volumeName'])) + self.assertIsNone(result) + + def test_return_volume_to_default_storage_group_v3(self): + masking = self.driver.common.masking + conn = self.fake_ecom_connection() + volumeInstanceName = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + volumeInstance = conn.GetInstance(volumeInstanceName) + volumeName = "V3-Vol" + extraSpecs = {'volume_backend_name': 'V3_BE', + 'isV3': True, + 'storagetype:pool': 'SRP_1', + 'storagetype:workload': 'DSS', + 'storagetype:slo': 'Bronze'} + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + masking.provisionv3.create_storage_group_v3 = mock.Mock( + return_value={'Value'}) + masking._is_volume_in_storage_group = mock.Mock( + return_value=True) + masking.return_volume_to_default_storage_group_v3 = mock.Mock() + masking._return_back_to_default_sg( + conn, controllerConfigService, volumeInstance, volumeName, + extraSpecs) + masking.return_volume_to_default_storage_group_v3.assert_called_with( + conn, controllerConfigService, + volumeInstance, volumeName, extraSpecs) + + def test_return_volume_to_default_storage_group_v3_exception(self): + masking = self.driver.common.masking + conn = self.fake_ecom_connection() + volumeInstanceName = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + volumeInstance = conn.GetInstance(volumeInstanceName) + volumeName = "V3-Vol" + extraSpecs = {'volume_backend_name': 'V3_BE', + 'isV3': True, + 'storagetype:pool': 'SRP_1', + 'storagetype:workload': 'DSS', + 'storagetype:slo': 'Bronze'} + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + + self.assertRaises( + exception.VolumeBackendAPIException, + masking.return_volume_to_default_storage_group_v3, + conn, controllerConfigService, + volumeInstance, volumeName, extraSpecs) + + def test_add_volume_to_sg_and_verify(self): + masking = self.driver.common.masking + conn = self.fake_ecom_connection() + volumeInstanceName = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + volumeInstance = conn.GetInstance(volumeInstanceName) + volumeName = "V3-Vol" + storageGroupInstanceName = self.data.storagegroups[0] + sgGroupName = self.data.storagegroupname + extraSpecs = {'volume_backend_name': 'V3_BE', + 'isV3': True, + 'storagetype:pool': 'SRP_1', + 'storagetype:workload': 'DSS', + 'storagetype:slo': 'Bronze'} + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + msg = masking._add_volume_to_sg_and_verify( + conn, controllerConfigService, storageGroupInstanceName, + volumeInstance, volumeName, sgGroupName, extraSpecs) + self.assertIsNone(msg) + + def test_cleanup_deletion_v3(self): + masking = self.driver.common.masking + conn = self.fake_ecom_connection() + volumeInstanceName = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + volumeInstance = conn.GetInstance(volumeInstanceName) + storageGroupInstanceName = self.data.storagegroups[1] + extraSpecs = {'volume_backend_name': 'V3_BE', + 'isV3': True, + 'storagetype:pool': 'SRP_1', + 'storagetype:workload': 'DSS', + 'storagetype:slo': 'Bronze'} + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + masking._remove_volume_from_sg = mock.Mock() + masking._cleanup_deletion_v3( + conn, controllerConfigService, volumeInstance, extraSpecs) + masking._remove_volume_from_sg.assert_called_with( + conn, controllerConfigService, storageGroupInstanceName, + volumeInstance, extraSpecs) + + def test_remove_volume_from_sg(self): + extraSpecs = self.data.extra_specs + conn = self.fake_ecom_connection() + common = self.driver.common + masking = common.masking + controllerConfigService = ( + common.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + storageGroupName = self.data.storagegroupname + storageGroupInstanceName = ( + self.driver.utils.find_storage_masking_group( + conn, controllerConfigService, storageGroupName)) + volumeInstanceNames = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")) + volumeInstanceName = volumeInstanceNames[0] + volumeInstance = conn.GetInstance(volumeInstanceName) + masking.get_devices_from_storage_group = ( + mock.Mock(return_value=volumeInstanceNames)) + masking._remove_volume_from_sg( + conn, controllerConfigService, storageGroupInstanceName, + volumeInstance, extraSpecs) + masking.get_devices_from_storage_group.assert_called_with( + conn, storageGroupInstanceName) + + +class EMCVMAXFCTest(test.TestCase): + def setUp(self): + self.data = EMCVMAXCommonData() + + super(EMCVMAXFCTest, self).setUp() + + configuration = mock.Mock() + configuration.safe_get.return_value = 'FCTests' + configuration.config_group = 'FCTests' + emc_vmax_common.EMCVMAXCommon._gather_info = mock.Mock() + driver = emc_vmax_fc.EMCVMAXFCDriver(configuration=configuration) + driver.db = FakeDB() + self.driver = driver + + def test_terminate_connection_ig_present(self): + common = self.driver.common + common.conn = FakeEcomConnection() + common._unmap_lun = mock.Mock() + common.get_masking_view_by_volume = mock.Mock( + return_value='testMV') + common.get_masking_views_by_port_group = mock.Mock( + return_value=[]) + common.get_target_wwns = mock.Mock( + return_value=EMCVMAXCommonData.target_wwns) + initiatorGroupInstanceName = ( + self.driver.common.masking._get_initiator_group_from_masking_view( + common.conn, self.data.lunmaskctrl_name, + self.data.storage_system)) + with mock.patch.object(self.driver.common, + 'check_ig_instance_name', + return_value=initiatorGroupInstanceName): + data = self.driver.terminate_connection(self.data.test_volume_v3, + self.data.connector) + common.get_target_wwns.assert_called_once_with( + EMCVMAXCommonData.storage_system, EMCVMAXCommonData.connector) + numTargetWwns = len(EMCVMAXCommonData.target_wwns) + self.assertEqual(numTargetWwns, len(data['data'])) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + 'check_ig_instance_name', + return_value=None) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + 'get_target_wwns', + return_value=EMCVMAXCommonData.target_wwns) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + 'get_masking_views_by_port_group', + return_value=[]) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + 'get_masking_view_by_volume', + return_value='testMV') + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_unmap_lun') + def test_terminate_connection_no_ig(self, mock_unmap, + mock_mv_vol, mock_mv_pg, + mock_wwns, mock_check_ig): + common = self.driver.common + common.conn = FakeEcomConnection() + data = self.driver.terminate_connection(self.data.test_volume_v3, + self.data.connector) + common.get_target_wwns.assert_called_once_with( + EMCVMAXCommonData.storage_system, EMCVMAXCommonData.connector) + numTargetWwns = len(EMCVMAXCommonData.target_wwns) + self.assertEqual(numTargetWwns, len(data['data'])) + + def test_get_common_masking_views_two_exist(self): + common = self.driver.common + common.conn = FakeEcomConnection() + maskingviews = [{'CreationClassName': 'Symm_LunMaskingView', + 'ElementName': 'MV1'}, + {'CreationClassName': 'Symm_LunMaskingView', + 'ElementName': 'MV2'}] + + portGroupInstanceName = ( + self.driver.common.masking._get_port_group_from_masking_view( + common.conn, self.data.lunmaskctrl_name, + self.data.storage_system)) + + initiatorGroupInstanceName = ( + self.driver.common.masking._get_initiator_group_from_masking_view( + common.conn, self.data.lunmaskctrl_name, + self.data.storage_system)) + common.get_masking_views_by_port_group = mock.Mock( + return_value=maskingviews) + common.get_masking_views_by_initiator_group = mock.Mock( + return_value=maskingviews) + + mvInstances = self.driver._get_common_masking_views( + portGroupInstanceName, initiatorGroupInstanceName) + self.assertEqual(2, len(mvInstances)) + + def test_get_common_masking_views_one_overlap(self): + common = self.driver.common + common.conn = FakeEcomConnection() + maskingviewsPG = [{'CreationClassName': 'Symm_LunMaskingView', + 'ElementName': 'MV1'}, + {'CreationClassName': 'Symm_LunMaskingView', + 'ElementName': 'MV2'}] + + maskingviewsIG = [{'CreationClassName': 'Symm_LunMaskingView', + 'ElementName': 'MV1'}] + + portGroupInstanceName = ( + self.driver.common.masking._get_port_group_from_masking_view( + common.conn, self.data.lunmaskctrl_name, + self.data.storage_system)) + + initiatorGroupInstanceName = ( + self.driver.common.masking._get_initiator_group_from_masking_view( + common.conn, self.data.lunmaskctrl_name, + self.data.storage_system)) + common.get_masking_views_by_port_group = mock.Mock( + return_value=maskingviewsPG) + common.get_masking_views_by_initiator_group = mock.Mock( + return_value=maskingviewsIG) + + mvInstances = self.driver._get_common_masking_views( + portGroupInstanceName, initiatorGroupInstanceName) + self.assertEqual(1, len(mvInstances)) + + def test_get_common_masking_views_no_overlap(self): + common = self.driver.common + common.conn = FakeEcomConnection() + maskingviewsPG = [{'CreationClassName': 'Symm_LunMaskingView', + 'ElementName': 'MV2'}] + + maskingviewsIG = [{'CreationClassName': 'Symm_LunMaskingView', + 'ElementName': 'MV1'}] + + portGroupInstanceName = ( + self.driver.common.masking._get_port_group_from_masking_view( + common.conn, self.data.lunmaskctrl_name, + self.data.storage_system)) + + initiatorGroupInstanceName = ( + self.driver.common.masking._get_initiator_group_from_masking_view( + common.conn, self.data.lunmaskctrl_name, + self.data.storage_system)) + common.get_masking_views_by_port_group = mock.Mock( + return_value=maskingviewsPG) + common.get_masking_views_by_initiator_group = mock.Mock( + return_value=maskingviewsIG) + + mvInstances = self.driver._get_common_masking_views( + portGroupInstanceName, initiatorGroupInstanceName) + self.assertEqual(0, len(mvInstances)) + + @mock.patch.object( + emc_vmax_provision.EMCVMAXProvision, + 'remove_device_from_storage_group') + def test_remove_device_from_storage_group(self, mock_remove): + conn = FakeEcomConnection() + common = self.driver.common + controllerConfigService = ( + common.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + volumeInstanceName = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + volumeName = 'vol1' + extraSpecs = {'volume_backend_name': 'V3_BE', + 'isV3': True, + 'storagetype:pool': 'SRP_1', + 'storagetype:workload': 'DSS', + 'storagetype:slo': 'Bronze'} + masking = common.masking + volumeInstance = conn.GetInstance(volumeInstanceName) + storageGroupName = self.data.storagegroupname + storageGroupInstanceName = ( + common.utils.find_storage_masking_group( + conn, controllerConfigService, storageGroupName)) + masking.remove_device_from_storage_group( + conn, controllerConfigService, storageGroupInstanceName, + volumeInstance, volumeName, extraSpecs) + masking.provision.remove_device_from_storage_group.assert_called_with( + conn, controllerConfigService, storageGroupInstanceName, + volumeInstanceName, volumeName, extraSpecs) + + +class EMCVMAXUtilsTest(test.TestCase): + def setUp(self): + self.data = EMCVMAXCommonData() + + super(EMCVMAXUtilsTest, self).setUp() + + configuration = mock.Mock() + configuration.safe_get.return_value = 'UtilsTests' + configuration.config_group = 'UtilsTests' + emc_vmax_common.EMCVMAXCommon._gather_info = mock.Mock() + driver = emc_vmax_iscsi.EMCVMAXISCSIDriver(configuration=configuration) + driver.db = FakeDB() + self.driver = driver + self.driver.utils = emc_vmax_utils.EMCVMAXUtils(object) + + def test_get_target_endpoints(self): + conn = FakeEcomConnection() + hardwareid = 123456789012345 + result = self.driver.utils.get_target_endpoints(conn, hardwareid) + self.assertEqual( + ([{'Name': '5000090000000000'}]), result) + + def test_get_protocol_controller(self): + conn = FakeEcomConnection() + hardwareid = 123456789012345 + result = self.driver.utils.get_protocol_controller(conn, hardwareid) + self.assertEqual( + ({'CreationClassName': 'Symm_LunMaskingView', + 'ElementName': 'OS-fakehost-gold-I-MV'}), result) + + def test_get_protocol_controller_exception(self): + conn = FakeEcomConnection() + conn.AssociatorNames = mock.Mock(return_value=[]) + hardwareid = 123456789012345 + self.assertRaises( + exception.VolumeBackendAPIException, + self.driver.utils.get_protocol_controller, + conn, hardwareid) + + def test_set_target_element_supplier_in_rsd(self): + conn = FakeEcomConnection() + extraSpecs = self.data.extra_specs + repServiceInstanceName = ( + self.driver.utils.find_replication_service( + conn, self.data.storage_system)) + rsdInstance = self.driver.utils.set_target_element_supplier_in_rsd( + conn, repServiceInstanceName, + emc_vmax_common.SNAPVX_REPLICATION_TYPE, + emc_vmax_common.CREATE_NEW_TARGET, extraSpecs) + self.assertIsNotNone(rsdInstance) + + def test_set_copy_methodology_in_rsd(self): + conn = FakeEcomConnection() + extraSpecs = self.data.extra_specs + repServiceInstanceName = ( + self.driver.utils.find_replication_service( + conn, self.data.storage_system)) + rsdInstance = self.driver.utils.set_copy_methodology_in_rsd( + conn, repServiceInstanceName, + emc_vmax_provision.SYNC_CLONE_LOCAL, + emc_vmax_provision.COPY_ON_WRITE, extraSpecs) + self.assertIsNotNone(rsdInstance) + + # bug #1605193 - Cleanup of Initiator Group fails + def test_check_ig_instance_name_present(self): + conn = FakeEcomConnection() + initiatorgroup = SE_InitiatorMaskingGroup() + initiatorgroup['CreationClassName'] = ( + self.data.initiatorgroup_creationclass) + initiatorgroup['DeviceID'] = self.data.initiatorgroup_id + initiatorgroup['SystemName'] = self.data.storage_system + initiatorgroup['ElementName'] = self.data.initiatorgroup_name + foundIg = self.driver.utils.check_ig_instance_name( + conn, initiatorgroup) + self.assertEqual(initiatorgroup, foundIg) + + # bug #1605193 - Cleanup of Initiator Group fails + def test_check_ig_instance_name_not_present(self): + conn = FakeEcomConnection() + initiatorgroup = None + with mock.patch.object(self.driver.utils, + 'get_existing_instance', + return_value=None): + foundIg = self.driver.utils.check_ig_instance_name( + conn, initiatorgroup) + self.assertIsNone(foundIg) + + def test_get_iqn(self): + conn = FakeEcomConnection() + iqn = "iqn.1992-04.com.emc:600009700bca30c01b9c012000000003,t,0x0001" + ipprotocolendpoints = conn._enum_ipprotocolendpoint() + foundIqn = self.driver.utils.get_iqn(conn, ipprotocolendpoints[1]) + self.assertEqual(iqn, foundIqn) + + def test_get_pool_capacities(self): + conn = FakeEcomConnection() + + (total_capacity_gb, free_capacity_gb, provisioned_capacity_gb, + array_max_over_subscription) = ( + self.driver.utils.get_pool_capacities( + conn, self.data.poolname, self.data.storage_system)) + self.assertEqual(931, total_capacity_gb) + self.assertEqual(465, free_capacity_gb) + self.assertEqual(465, provisioned_capacity_gb) + self.assertEqual(1.5, array_max_over_subscription) + + def test_get_pool_capacities_none_array_max_oversubscription(self): + conn = FakeEcomConnection() + null_emcmaxsubscriptionpercent = { + 'TotalManagedSpace': '1000000000000', + 'ElementName': 'gold', + 'RemainingManagedSpace': '500000000000', + 'SystemName': 'SYMMETRIX+000195900551', + 'CreationClassName': 'Symm_VirtualProvisioningPool', + 'EMCSubscribedCapacity': '500000000000'} + conn.GetInstance = mock.Mock( + return_value=null_emcmaxsubscriptionpercent) + (total_capacity_gb, free_capacity_gb, provisioned_capacity_gb, + array_max_over_subscription) = ( + self.driver.utils.get_pool_capacities( + conn, self.data.poolname, self.data.storage_system)) + self.assertEqual(65534, array_max_over_subscription) + + def test_get_ratio_from_max_sub_per(self): + max_subscription_percent_float = ( + self.driver.utils.get_ratio_from_max_sub_per(150)) + self.assertEqual(1.5, max_subscription_percent_float) + + def test_get_ratio_from_max_sub_per_none_value(self): + max_subscription_percent_float = ( + self.driver.utils.get_ratio_from_max_sub_per(str(0))) + self.assertIsNone(max_subscription_percent_float) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_find_lun', + return_value={'SystemName': EMCVMAXCommonData.storage_system}) + @mock.patch('builtins.open' if sys.version_info >= (3,) + else '__builtin__.open') + def test_insert_live_migration_record(self, mock_open, mock_lun): + conn = FakeEcomConnection() + self.driver.common.conn = conn + extraSpecs = self.data.extra_specs + connector = {'initiator': self.data.iscsi_initiator, + 'ip': '10.0.0.2', + 'platform': u'x86_64', + 'host': 'fakehost', + 'os_type': 'linux2', + 'multipath': False} + maskingviewdict = self.driver.common._populate_masking_dict( + self.data.test_volume, self.data.connector, extraSpecs) + emc_vmax_utils.LIVE_MIGRATION_FILE = ('/tempdir/livemigrationarray') + self.driver.utils.insert_live_migration_record( + self.data.test_volume, maskingviewdict, connector, extraSpecs) + mock_open.assert_called_once_with( + emc_vmax_utils.LIVE_MIGRATION_FILE, "wb") + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_find_lun', + return_value={'SystemName': EMCVMAXCommonData.storage_system}) + def test_delete_live_migration_record(self, mock_lun): + conn = FakeEcomConnection() + self.driver.common.conn = conn + extraSpecs = self.data.extra_specs + connector = {'initiator': self.data.iscsi_initiator, + 'ip': '10.0.0.2', + 'platform': u'x86_64', + 'host': 'fakehost', + 'os_type': 'linux2', + 'multipath': False} + maskingviewdict = self.driver.common._populate_masking_dict( + self.data.test_volume, self.data.connector, extraSpecs) + tempdir = tempfile.mkdtemp() + emc_vmax_utils.LIVE_MIGRATION_FILE = (tempdir + + '/livemigrationarray') + m = mock.mock_open() + with mock.patch('{}.open'.format(__name__), m, create=True): + with open(emc_vmax_utils.LIVE_MIGRATION_FILE, "wb") as f: + f.write('live migration details') + self.driver.utils.insert_live_migration_record( + self.data.test_volume, maskingviewdict, connector, extraSpecs) + self.driver.utils.delete_live_migration_record(self.data.test_volume) + m.assert_called_once_with(emc_vmax_utils.LIVE_MIGRATION_FILE, "wb") + shutil.rmtree(tempdir) + + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_find_lun', + return_value={'SystemName': EMCVMAXCommonData.storage_system}) + def test_get_live_migration_record(self, mock_lun): + conn = FakeEcomConnection() + self.driver.common.conn = conn + extraSpecs = self.data.extra_specs + connector = {'initiator': self.data.iscsi_initiator, + 'ip': '10.0.0.2', + 'platform': u'x86_64', + 'host': 'fakehost', + 'os_type': 'linux2', + 'multipath': False} + maskingviewdict = self.driver.common._populate_masking_dict( + self.data.test_volume, self.data.connector, extraSpecs) + tempdir = tempfile.mkdtemp() + emc_vmax_utils.LIVE_MIGRATION_FILE = (tempdir + + '/livemigrationarray') + self.driver.utils.insert_live_migration_record( + self.data.test_volume, maskingviewdict, connector, extraSpecs) + record = self.driver.utils.get_live_migration_record( + self.data.test_volume, False) + self.assertEqual(maskingviewdict, record[0]) + self.assertEqual(connector, record[1]) + os.remove(emc_vmax_utils.LIVE_MIGRATION_FILE) + shutil.rmtree(tempdir) + + def test_find_volume_by_device_id_on_array(self): + conn = FakeEcomConnection() + utils = self.driver.common.utils + + bindings = {'CreationClassName': 'Symm_StorageVolume', + 'SystemName': self.data.storage_system, + 'DeviceID': self.data.test_volume['device_id'], + 'SystemCreationClassName': 'Symm_StorageSystem'} + + inst = FakeCIMInstanceName() + fake_inst = inst.fake_getinstancename('Symm_StorageVolume', bindings) + utils.find_volume_by_device_id_on_array = mock.Mock( + return_value=fake_inst) + + volumeInstanceName = utils.find_volume_by_device_id_on_array( + self.data.storage_system, self.data.test_volume['device_id']) + + expectVolume = {} + expectVolume['CreationClassName'] = 'Symm_StorageVolume' + expectVolume['DeviceID'] = self.data.test_volume['device_id'] + expect = conn.GetInstance(expectVolume) + + provider_location = ast.literal_eval(expect['provider_location']) + bindings = provider_location['keybindings'] + + self.assertEqual(bindings, volumeInstanceName) + + def test_get_assoc_v2_pool_from_vol(self): + conn = FakeEcomConnection() + volumeInstanceName = {'CreationClassName': "Symm_StorageVolume", + 'DeviceID': "0123", + 'SystemName': "12345"} + + pool = conn.AssociatorNames( + volumeInstanceName, ResultClass='EMC_VirtualProvisioningPool') + poolName = self.driver.utils.get_assoc_v2_pool_from_volume( + conn, volumeInstanceName) + + self.assertEqual(pool[0]['ElementName'], poolName['ElementName']) + + def test_get_assoc_v2_pool_from_vol_fail(self): + conn = FakeEcomConnection() + volumeInstanceName = {'CreationClassName': "Symm_StorageVolume", + 'DeviceID': "0123", + 'SystemName': "12345"} + + conn.AssociatorNames = mock.Mock(return_value={}) + + poolName = self.driver.utils.get_assoc_v2_pool_from_volume( + conn, volumeInstanceName) + + self.assertIsNone(poolName) + + def test_get_assoc_v3_pool_from_vol(self): + conn = FakeEcomConnection() + volumeInstanceName = {'CreationClassName': "Symm_StorageVolume", + 'DeviceID': "0123", + 'SystemName': "12345"} + + pool = conn.AssociatorNames( + volumeInstanceName, ResultClass='Symm_SRPStoragePool') + poolName = self.driver.utils.get_assoc_v3_pool_from_volume( + conn, volumeInstanceName) + + self.assertEqual(pool[0]['ElementName'], poolName['ElementName']) + + def test_get_assoc_v3_pool_from_vol_fail(self): + conn = FakeEcomConnection() + volumeInstanceName = {'CreationClassName': "Symm_StorageVolume", + 'DeviceID': "0123", + 'SystemName': "12345"} + + conn.AssociatorNames = mock.Mock(return_value={}) + + self.assertRaises(exception.VolumeBackendAPIException, + self.driver.utils.get_assoc_v3_pool_from_volume, + conn, volumeInstanceName) + + def test_check_volume_no_fast_fail(self): + utils = self.driver.common.utils + initial_setup = {'volume_backend_name': 'FCFAST', + 'storagetype:fastpolicy': 'GOLD'} + + self.assertRaises(exception.VolumeBackendAPIException, + utils.check_volume_no_fast, + initial_setup) + + def test_check_volume_no_fast_pass(self): + utils = self.driver.common.utils + initial_setup = {'volume_backend_name': 'FCnoFAST', + 'storagetype:fastpolicy': None} + + self.assertTrue(utils.check_volume_no_fast( + initial_setup)) + + def test_check_volume_not_in_masking_view_pass(self): + conn = FakeEcomConnection() + utils = self.driver.common.utils + + bindings = {'CreationClassName': 'Symm_StorageVolume', + 'SystemName': self.data.storage_system, + 'DeviceID': self.data.test_volume['device_id'], + 'SystemCreationClassName': 'Symm_StorageSystem'} + inst = FakeCIMInstanceName() + fake_inst = inst.fake_getinstancename('Symm_StorageVolume', bindings) + + sgInstanceNames = conn.AssociatorNames(fake_inst, + ResultClass= + 'CIM_DeviceMaskingGroup') + + conn.AssociatorNames = mock.Mock(return_value={}) + + mock.patch.object(self.driver.utils, 'get_storage_groups_from_volume', + return_value=sgInstanceNames) + + self.assertTrue( + utils.check_volume_not_in_masking_view( + conn, fake_inst, self.data.test_volume['device_id'])) + + def test_check_volume_not_in_masking_view_fail(self): + conn = FakeEcomConnection() + utils = self.driver.common.utils + + bindings = {'CreationClassName': 'Symm_StorageVolume', + 'SystemName': self.data.storage_system, + 'DeviceID': self.data.test_volume['device_id'], + 'SystemCreationClassName': 'Symm_StorageSystem'} + inst = FakeCIMInstanceName() + fake_inst = inst.fake_getinstancename('Symm_StorageVolume', bindings) + + self.assertRaises(exception.VolumeBackendAPIException, + utils.check_volume_not_in_masking_view, + conn, fake_inst, self.data.test_volume['device_id']) + + def test_check_volume_not_replication_source_pass(self): + conn = FakeEcomConnection() + utils = self.driver.common.utils + + self.assertTrue( + utils.check_volume_not_replication_source( + conn, self.data.storage_system_v3, + self.data.test_volume['device_id'])) + + def test_check_volume_not_replication_source_fail(self): + conn = FakeEcomConnection() + utils = self.driver.common.utils + + replication_source = 'testReplicationSync' + + utils.get_associated_replication_from_source_volume = \ + mock.Mock(return_value=replication_source) + + self.assertRaises( + exception.VolumeBackendAPIException, + utils.check_volume_not_replication_source, conn, + self.data.storage_system_v3, self.data.test_volume['device_id']) + + def test_check_is_volume_in_cinder_managed_pool_fail(self): + conn = FakeEcomConnection() + utils = self.driver.common.utils + + volumeInstanceName = {'CreationClassName': "Symm_StorageVolume", + 'DeviceID': "0123", + 'SystemName': "12345"} + + poolInstanceName = {} + poolInstanceName['InstanceID'] = "SATA_GOLD1" + deviceId = '0123' + + self.assertRaises( + exception.VolumeBackendAPIException, + utils.check_is_volume_in_cinder_managed_pool, + conn, volumeInstanceName, poolInstanceName, deviceId) + + def test_check_is_volume_in_cinder_managed_pool_pass(self): + conn = FakeEcomConnection() + utils = self.driver.common.utils + + volumeInstanceName = {} + poolInstanceName = {} + poolInstanceName['InstanceID'] = "SATA_GOLD2" + deviceId = self.data.test_volume['device_id'] + + utils.get_assoc_v2_pool_from_volume = \ + mock.Mock(return_value=poolInstanceName) + + self.assertTrue( + utils.check_is_volume_in_cinder_managed_pool( + conn, volumeInstanceName, poolInstanceName, deviceId)) + + +class EMCVMAXCommonTest(test.TestCase): + def setUp(self): + self.data = EMCVMAXCommonData() + + super(EMCVMAXCommonTest, self).setUp() + + configuration = mock.Mock() + configuration.safe_get.return_value = 'CommonTests' + configuration.config_group = 'CommonTests' + emc_vmax_common.EMCVMAXCommon._gather_info = mock.Mock() + driver = emc_vmax_iscsi.EMCVMAXISCSIDriver(configuration=configuration) + driver.db = FakeDB() + self.driver = driver + self.driver.utils = emc_vmax_utils.EMCVMAXUtils(object) + + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'get_v3_default_sg_instance_name', + return_value=(None, None, EMCVMAXCommonData.default_sg_instance_name)) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=(None, EMCVMAXCommonData.storage_system)) + def test_create_duplicate_volume(self, mock_pool, mock_sg): + common = self.driver.common + common.conn = FakeEcomConnection() + volumeInstanceName = ( + common.conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + sourceInstance = common.conn.GetInstance(volumeInstanceName) + cloneName = "SS-V3-Vol" + extraSpecs = {'volume_backend_name': 'V3_BE', + 'isV3': True, + 'storagetype:pool': 'SRP_1', + 'storagetype:workload': 'DSS', + 'storagetype:slo': 'Bronze'} + targetInstance = common.conn.GetInstance(volumeInstanceName) + common.utils.find_volume_instance = mock.Mock( + return_value=targetInstance) + duplicateVolumeInstance = self.driver.common._create_duplicate_volume( + sourceInstance, cloneName, extraSpecs) + self.assertIsNotNone(duplicateVolumeInstance) + + def test_cleanup_target(self): + common = self.driver.common + common.conn = FakeEcomConnection() + volumeInstanceName = ( + common.conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + extraSpecs = {'volume_backend_name': 'V3_BE', + 'isV3': True, + 'storagetype:pool': 'SRP_1', + 'storagetype:workload': 'DSS', + 'storagetype:slo': 'Bronze'} + targetInstance = common.conn.GetInstance(volumeInstanceName) + repServiceInstanceName = ( + self.driver.utils.find_replication_service( + common.conn, self.data.storage_system)) + common.utils.find_sync_sv_by_target = mock.Mock( + return_value=(None, None)) + + self.driver.common._cleanup_target( + repServiceInstanceName, targetInstance, extraSpecs) + + def test_get_ip_and_iqn(self): + conn = FakeEcomConnection() + endpoint = {} + ipprotocolendpoints = conn._enum_ipprotocolendpoint() + ip_and_iqn = self.driver.common.get_ip_and_iqn(conn, endpoint, + ipprotocolendpoints[0]) + ip_and_iqn = self.driver.common.get_ip_and_iqn(conn, endpoint, + ipprotocolendpoints[1]) + self.assertEqual( + 'iqn.1992-04.com.emc:600009700bca30c01b9c012000000003,t,0x0001', + ip_and_iqn['iqn']) + self.assertEqual( + '10.10.10.10', ip_and_iqn['ip']) + + def test_manage_existing_get_size(self): + common = self.driver.common + common.conn = FakeEcomConnection() + + gb_size = 2 + exp_size = 2 + volume = {} + metadata = {'key': 'array', + 'value': '12345'} + volume['volume_metadata'] = [metadata] + external_ref = {'source-name': '0123'} + volumeInstanceName = {'CreationClassName': "Symm_StorageVolume", + 'DeviceID': "0123", + 'SystemName': "12345"} + + utils = self.driver.common.utils + utils.get_volume_size = mock.Mock( + return_value=int(gb_size * units.Gi)) + utils.find_volume_by_device_id_on_array = mock.Mock( + return_value=volumeInstanceName) + + size = self.driver.manage_existing_get_size(volume, external_ref) + self.assertEqual(exp_size, size) + + def test_manage_existing_get_size_fail(self): + common = self.driver.common + common.conn = FakeEcomConnection() + + gb_size = 2 + volume = {} + metadata = {'key': 'array', + 'value': '12345'} + volume['volume_metadata'] = [metadata] + external_ref = {'source-name': '0123'} + + utils = self.driver.common.utils + utils.get_volume_size = mock.Mock( + return_value=int(gb_size * units.Gi)) + + utils.find_volume_by_device_id_on_array = mock.Mock( + return_value=None) + + self.assertRaises(exception.VolumeBackendAPIException, + self.driver.common.manage_existing_get_size, + volume, external_ref) + + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'rename_volume', + return_value=EMCVMAXCommonData.manage_vol) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'check_is_volume_in_cinder_managed_pool', + return_value=True) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'check_volume_not_replication_source', + return_value=True) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_pool_and_storage_system', + return_value=('cinder_pool', 'vmax_storage_system')) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'check_volume_not_in_masking_view', + return_value=True) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'find_volume_by_device_id_on_array', + return_value=EMCVMAXCommonData.test_volume) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'check_volume_no_fast', + return_value=True) + @mock.patch.object( + emc_vmax_utils.EMCVMAXUtils, + 'get_array_and_device_id', + return_value=('12345', '1')) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_get_ecom_connection', + return_value=FakeEcomConnection()) + @mock.patch.object( + emc_vmax_common.EMCVMAXCommon, + '_initial_setup', + return_value=EMCVMAXCommonData.extra_specs) + def test_manage_existing( + self, mock_setup, mock_ecom, mock_ids, mock_vol_fast, + mock_vol_by_deviceId, mock_vol_in_mv, mock_pool_sg, + mock_vol_rep_src, mock_vol_in_mng_pool, mock_rename_vol): + common = self.driver.common + volume = self.data.test_volume + external_ref = {} + provider_location = six.text_type(self.data.provider_location_manage) + model_update = { + 'display_name': 'vol1', + 'provider_location': provider_location} + + new_model_update = common.manage_existing(volume, + external_ref) + + self.assertEqual(model_update, new_model_update) + + +class EMCVMAXISCSITest(test.TestCase): + def setUp(self): + self.data = EMCVMAXCommonData() + + super(EMCVMAXISCSITest, self).setUp() + + configuration = mock.Mock() + configuration.safe_get.return_value = 'iSCSITests' + configuration.config_group = 'iSCSITests' + emc_vmax_common.EMCVMAXCommon._gather_info = mock.Mock() + driver = emc_vmax_iscsi.EMCVMAXISCSIDriver(configuration=configuration) + driver.db = FakeDB() + self.driver = driver + + def test_smis_get_iscsi_properties(self): + device_info = {'hostlunid': 1} + self.driver.common.find_device_number = ( + mock.Mock(return_value=device_info)) + iqns_and_ips = ( + [{'iqn': 'iqn.1992-04.com.emc:50000973f006dd80,t,0x0001', + 'ip': '10.10.0.50'}, + {'iqn': 'iqn.1992-04.com.emc:50000973f006dd81,t,0x0001', + 'ip': '10.10.0.51'}]) + properties = self.driver.smis_get_iscsi_properties( + self.data.test_volume, self.data.connector, iqns_and_ips, True) + self.assertEqual([1, 1], properties['target_luns']) + self.assertEqual(['iqn.1992-04.com.emc:50000973f006dd80', + 'iqn.1992-04.com.emc:50000973f006dd81'], + properties['target_iqns']) + self.assertEqual(['10.10.0.50:3260', '10.10.0.51:3260'], + properties['target_portals']) + + +class EMCVMAXProvisionTest(test.TestCase): + def setUp(self): + self.data = EMCVMAXCommonData() + + super(EMCVMAXProvisionTest, self).setUp() + + configuration = mock.Mock() + configuration.safe_get.return_value = 'ProvisionTests' + configuration.config_group = 'ProvisionTests' + emc_vmax_common.EMCVMAXCommon._gather_info = mock.Mock() + driver = emc_vmax_iscsi.EMCVMAXISCSIDriver( + configuration=configuration) + driver.db = FakeDB() + self.driver = driver + self.driver.utils = emc_vmax_utils.EMCVMAXUtils(object) + + @mock.patch.object( + emc_vmax_provision.EMCVMAXProvision, + 'remove_device_from_storage_group') + def test_remove_device_from_storage_group(self, mock_remove): + conn = FakeEcomConnection() + controllerConfigService = ( + self.driver.utils.find_controller_configuration_service( + conn, self.data.storage_system)) + volumeInstanceName = ( + conn.EnumerateInstanceNames("EMC_StorageVolume")[0]) + volumeName = 'vol1' + extraSpecs = {'volume_backend_name': 'V3_BE', + 'isV3': True, + 'storagetype:pool': 'SRP_1', + 'storagetype:workload': 'DSS', + 'storagetype:slo': 'Bronze'} + masking = self.driver.common.masking + volumeInstance = conn.GetInstance(volumeInstanceName) + storageGroupName = self.data.storagegroupname + storageGroupInstanceName = ( + self.driver.common.utils.find_storage_masking_group( + conn, controllerConfigService, storageGroupName)) + numVolsInSG = 2 + masking._multiple_vols_in_SG( + conn, controllerConfigService, storageGroupInstanceName, + volumeInstance, volumeName, numVolsInSG, extraSpecs) + masking.provision.remove_device_from_storage_group.assert_called_with( + conn, controllerConfigService, storageGroupInstanceName, + volumeInstanceName, volumeName, extraSpecs)