How to Setup EMC PowerPath

From V.S.V., Inc.
Jump to: navigation, search

The basics of setting up EMC Powerpath on Linux on new installations

EMC PowerPath is used to manage the multipath logical unit number (LUN) presented to a server from the storage area

network. It dynamically load balances over multiple fibre controllers and provides automatic fail over in the event a

path or controller fails. This document will show you what packages and kits are needed and what you need to do to bring

the SAN presented LUN into the operating system

Listing of files and locations:

  • repo.example.com:/unixrepo/packages/EMC/ # Location of rpm files currently supported.

EMCpower.LINUX-5.3.1.00.00-111.rhel5.i386.rpm # This release supports: RHEL (4.2...4.8, 5.0...5.4)
EMCpower.LINUX-5.3.1.00.00-111.rhel5.ia64.rpm # This release supports: RHEL (4.2...4.8, 5.0...5.4)
EMCpower.LINUX-5.3.1.00.00-111.rhel5.ppc64.rpm # This release supports: RHEL (4.2...4.8, 5.0...5.4)
EMCpower.LINUX-5.3.1.00.00-111.rhel5.x86_64.rpm # This release supports: RHEL (4.2...4.8, 5.0...5.4)
EMCpower.LINUX-5.3.1.00.00-111.rhel.i386.rpm # This release supports: RHEL (4.2...4.8, 5.0...5.4)
EMCpower.LINUX-5.3.1.00.00-111.rhel.ia64.rpm # This release supports: RHEL (4.2...4.8, 5.0...5.4)
EMCpower.LINUX-5.3.1.00.00-111.rhel.x86_64.rpm # This release supports: RHEL (4.2...4.8, 5.0...5.4)
EMCPower.LINUX-5.5.0.00.00-275.RHEL5.i386.rpm # Linux Kernels: RHEL 5.0 to 5.4 (except ia64 xen) and RHEL 5.5;
EMCPower.LINUX-5.5.0.00.00-275.RHEL5.IA64.rpm # Linux Kernels: RHEL 5.0 to 5.4 (except ia64 xen) and RHEL 5.5;
EMCPower.LINUX-5.5.0.00.00-275.RHEL5.PPC64.rpm # Linux Kernels: RHEL 5.0 to 5.4 (except ia64 xen) and RHEL 5.5;
EMCPower.LINUX-5.5.0.00.00-275.RHEL5.x86_64.rpm # Linux Kernels: RHEL 5.0 to 5.4 (except ia64 xen) and RHEL 5.5;
naviagentcli-6.24.0.6.13-1.noarch.rpm # Currently used. Needed for EMC Clarion arrays.
naviagent-6.28.20.1.40-1.noarch.rpm # HAS NOT BEEN TESTED. NOT IN USE. Needed for EMC Clarion arrays.

  • repo.example.com:emc-controlcenter6.1-kit # EMC /ECC Master Agent needed for (Phone Home support)
                               	      # iso9660 file mounted loop and exported for NFS use.
  • /usr/local/bin/ql-scan-lun.sh # Qlogic scan bus script
    1. NOTE BEGIN ##

These older packages are no longer stored locally. If you need PowerPath for one of the these releases you will need to

go to the EMC Powerlink web site and download them.

PowerPath 4.3.x # supports the Linux 2.4 kernel. PowerPath 4.4 and PowerPath 4.5 # supports the Linux 2.6 kernel.

EMCPower.LINUX.4.3.6.Patch.b003.tar.gz # Support for RHEL 3 U9 EMCpower.LINUX.4.5.1.022.tar.gz # Support for RHEL 4.0 U3 EMCpower.LINUX.5.0.0.157.tar.gz # Support for RHEL 4 U2-U4 EMCpower.LINUX-5.1.2.00.00-021.tar.gz # Support for RHEL5 U2, RHEL5 U1, RHEL5, RHEL4.7

                                         NOTE:  V5.1.2 causes kernel panic on RHEL 5.3

EMCPower.LINUX.5.3.GA.b185.tar.gz # Support for RHEL5.3


    1. NOTE END ##

Pre-installation steps:


  • The PowerPath kits can be installed prior to the presentation of the storage LUNS.
   Just note that the powermt set_policy commands will not execute until the server and array communication is

established.

  • Arrange fibre connectivity with the storage team.
  • Identify the backend storage array . IE. Clarion or Symmetrix
  • Decide what version of EMC PowerPath to use. You will need to know the LINUX kernel version and architecture.
  • Obtain Logical Unit Number ID from the Storage Team. Used to identify the LUNS
  • Identify the file system to be used on the device. I.E. ext3, ocfs, ocfs2,

ASM

Installation steps:

      • See the simplified list of commands in the appendix if you have read the extend version already.
  • Feel free to install EMC Power packages prior to the presentation of luns.
  • Note: If any of the backend storage is presented from an EMC CLARIION array you need to install the EMC Naviagent. The

EMC Naviagent is a tool that initiates a bus scan from the CLARIION side and collects the server HBA information,

updating the CLARIION array. If backend storage is presented from an EMC Symmetrix array the installation of EMC

Naviagent is not needed.

$ mkdir /unixrepo

$ mount repo.example.com:/UNIX_TEAM/UNIX_TEAM /unixrepo

$ rpm -ivh /unixrepo/packages/EMC/naviagentcli-6.24.0.6.13-1.noarch.rpm

  • Install the appropriate PowerPath package from the above list.

$ sudo su -

$ rpm -ivh /unixrepo/packages/EMC/package-name-architecture.rpm

      • You must reboot the server after Power Path has been installed so the kernel modules will get loaded.
  • Install EMC Enterprise Control Center Agent, and make it start on boot. The ECC agent is a separate EMC monitoring

tool. It is used to notify EMC in the event that a SAN attached device has a problem. In turn EMC notifies the Storage

team of the problem, that is if we have not gotten to them first.

$ mkdir /mnt/ecc

$ mount repo.example.com:/emc-ctrlctr6.1-kit /mnt/ecc

$ /mnt/ecc/install_master.sh /mnt/ecc/ # install script with input of kit files location.

Please enter ECC root directory[/usr/ecc]: Do you want to create native installable images [Y|N]: N ECC Server Host Name 10.50.11.55 (eccserv101.example.com) ECC Server Port [5799] Master Agent\'s Listening Port [5798]

  1. Input Verification

ECC Server Host Name 10.50.11.55 (eccserv101.example.com) ECC Server Port 5799 Master Agent\'s Listening Port 5798

Are all inputs above acceptable [Y|N|Q]: Y . . . . Do you want EMC ControlCenter MasterAgent to start on reboot [Y|N]: Y

Done: error count = 0 EMC Control Center Master Agent installed successfully.

  • Ensure startup of EMC ECC.

$ /etc/init.d/eccmad start

  1. Verify it\'s running.

$ ps aux |grep mstragent # there should be 2 processes + your grep. root 10223 1 0 Mar27 S 00:00:00 /usr/ecc/exec/mstragent root 10316 10223 0 Mar27 S 00:00:36 /usr/ecc/exec/mstragent -s root 11410 10182 0 12:59 pts/1 00:00:00 grep mstragent

  • Register the EMC PowerPath License listed below to enable multipath availability. Without a license you have only one

path enabled. $ emcpreg -install # It will prompt you to enter the new key.

===== EMC PowerPath Registration =====

Do you have a new registration key or keys to enter?[n] y Enter the registration keys(s) for your product(s), one per line, pressing Enter after each key. After typing all keys, press Enter again.

Key (Enter if done): XXXX-XXXX-XXXX-XXXX-XXXX-XXXX # License provided by Storage Team. Use this key for all

installations. Key (Enter if done): <cr>

  • Check your License key to see if it installed.

$ emcpreg -list

Key XXXX-XXXX-XXXX-XXXX-XXXX-XXXX

 Product: PowerPath
 Capabilities: All
  • Set the PowerPath load balancing and fail over policy. Each storage array has a specific load balancing algorithm.
 PowerPath needs to be informed what storage type it is load balancing for (co=CLARIION, so=Symmetrix). 
 This command also enable the new license key functionality for load balancing and failover policy you will be setting

up next.

  • Policy setting should be one of the following, based on the array type.

o CLARIION: policy= CLAROpt o SYMMETRIX: policy= SymmOpt

$ powermt set policy=co dev=emcpowerh # Clariion Array devices. Use dev=all if all luns are Clariion $ powermt set policy=so dev=emcpowerh # Symmetrix Array devices. Use dev=all if all luns are Symmetric

  1. Example of device /dev/emcpowerh prior to setting the policy.

$ powermt display dev=emcpowerh

Pseudo name=emcpowerh CLARiiON ID=APM00000000001 [DBQARAC] Logical device ID=60060160705E1A002245B1552EF2DD11 [VOTE_2] state=alive; policy=BasicFailover; priority=0; queued-IOs=0 Owner: default=SP A, current=SP A Array failover mode: 1

==================================================================

Host --------------- - Stor - -- I/O Path - -- Stats ---
      1. HW Path I/O Paths Interf. Mode State Q-IOs Errors
==================================================================
  1 qla2xxx                   sde       SP A3     unlic   alive      0      0
  1 qla2xxx                   sdl       SP B3     unlic   alive      0      0
  2 qla2xxx                   sds       SP A2     active  alive      0      0
  2 qla2xxx                   sdz       SP B2     active  alive      0      0
  • Set the policy for a CLARIION presented LUN.

$ powermt set policy=co dev=emcpowerh # the dev= accepts comma separated device names.

  • Check /dev/emcpowerh again.

$ powermt display dev=emcpowerh

Pseudo name=emcpowerh CLARiiON ID=APM00000000001 [DBQARAC] Logical device ID=60060160705E1A002245B1552EF2DD11 [VOTE_2] state=alive; policy=CLAROpt; priority=0; queued-IOs=0 Owner: default=SP A, current=SP A Array failover mode: 1

==================================================================

Host --------------- - Stor - -- I/O Path - -- Stats ---
      1. HW Path I/O Paths Interf. Mode State Q-IOs Errors
==================================================================
  1 qla2xxx                   sde       SP A3     active   alive      0      0
  1 qla2xxx                   sdl       SP B3      active   alive      0      0
  2 qla2xxx                   sds       SP A2     active  alive      0      0
  2 qla2xxx                   sdz       SP B2     active  alive      0      0


  1. Package installations are complete.

$ umount /mnt/ecc

$ unmont /unixrepo

Configure the LUNS:

You can scan for presented luns using the Qlogic scan bus script or reboot the server after the storage team has

presented the LUNS.

  1. running the script in almost all cases will find the luns. Sometime it may take more than one execution. # For more

options use the -h switch.

$

  1. View the assigned emcpower devices to verify each logical device ID is what you expect it to be in size and policy

assignment.

$ powermt display dev=emcpowerh

Pseudo name=emcpowerh CLARiiON ID=APM00000000001 [DBQARAC] Logical device ID=60060160705E1A002245B1552EF2DD11 [VOTE_2] state=alive; policy=CLAROpt; priority=0; queued-IOs=0 Owner: default=SP A, current=SP A Array failover mode: 1

==================================================================

Host --------------- - Stor - -- I/O Path - -- Stats ---
      1. HW Path I/O Paths Interf. Mode State Q-IOs Errors
==================================================================
  1 qla2xxx                   sde       SP A3     active   alive      0      0
  1 qla2xxx                   sdl       SP B3      active   alive      0      0
  2 qla2xxx                   sds       SP A2     active  alive      0      0
  2 qla2xxx                   sdz       SP B2     active  alive      0      0

Partiton the the powerpath luns:

NOTE: For RHEL 5 use the GNU parted program for all disk partitions.

RHEL 5:

  1. create a new partition that uses the whole disk with a 64 block offset

$ parted /dev/emcpowerh mklabel msdos unit s mkpart primary \"128 -1\"

RHEL 4:

Note: GNU parted may also work with RHEL 4. It has not been tested.

$ fdisk -l /dev/emcpowerh

Disk /dev/emcpowerh: 966.3 GB, 966367641600 bytes 255 heads, 63 sectors/track, 117487 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes

        Device Boot    Start       End    Blocks   Id  System

Creating a partition on the device:

Setting up the partition on the new DMX and CLARIION luns requires an offset of 64 blocks. Linux systems using Intel

architecture are affected by the placement of metadata on the LUN, which causes the disk partition to be misaligned.

This is due to legacy BIOS code, which uses cylinder, head, and sector addressing instead of Logical Block Addressing

(LBA). When portioning the device with more than one partition, you only need to move the first partition to the required

offset.

  1. Create a new partition on all your EMC storage luns.

$ fdisk /dev/emcpowera n # create new partition p # make it primary 1 # identify the partition with a number starting with 1 <cr> # accept default <cr> # accept default only if using the entire device. If not, enter the end block for the size you need. x # enter expert mode. b # adjust the data partition starting block 1 # Enter the partition your moving 128 # move the beginning of partition data to block 128 w # write the new partition table.

Create the file system:

File system creations will be needed for most installations. The client will provide you with the file system they need.

Typically backup volumes are ext3. For Oracle it can be ext3, raw, ocfs and ocfs2 or ASM format.

Here are some examples that we have used in the past. You can safely use these examples, but for ocfs and ocfs2 you

should inquire about the block size and cluster size that the DBA wants to use. You will also have other kits and

packages to install.

Examples:

$ mke2fs -j -m% /dev/emcpowera1 -L label # ext3 file system $ mke2fs -j -m1 /dev/emcpowera1 -L /backup

$ mkfs.ocfs2 -L label-name /dev/emcpoweru1 $ mkfs.ocfs2 -L backups_new /dev/emcpoweru1 # formats the partition with default block and cluster sizes based on

device size calculations.

$ mkfs.ocfs -b block-size -u ownerID -g groupID-L label -m mount point /dev/emcpowerv1 $ mkfs.ocfs -b 128 -u oratrak -g oradba -L u02 -m /u02 /dev/emcpowerv1


  1. Guide for installing ASMlib http://www.oracle.com/technology/tech/linux/asmlib/install.html

$ /etc/init.d/oracleasm configure $ /etc/init.d/oracleasm createdisk label /dev/emcpowera1 # ASM format $ /etc/init.d/oracleasm createdisk DATA1 /dev/emcpowera1

          1. NOTES:

- raw devices: When setting up raw devices on (RHEL 4) for Oracle DB you make the device permissions persistent on reboots by creating

/etc/udev/permissions.d/40-udev.permissions.

Reference:http://www.oracle.com/technology/products/database/asm/pdf/device-mapper-udev-crs-asm%20rh4.pdf for more

details.

Example 1: /etc/udev/permissions.d/40-udev.permissions raw/raw20:root:dba:0660 raw/raw21:root:dba:0660 raw/raw30:orasoaprod:dba:0660 raw/raw31:orasoaprod:dba:0660 raw/raw32:orasoaprod:dba:0660

OR Example 2: /etc/udev/permissions.d/40-udev.permissions

raw/raw2[0-1]:root:dba:0660 raw/raw3[0-2]:orasoaprod:dba:0660

  1. Appendix:
  1. /usr/local/bin/ql-scan-lun.sh # Scan the bus LUNS
  2. rpm -ivh /unixrepo/packages/EMC/naviagentcli-6.24.0.6.13-1.noarch.rpm
  3. rpm -ivh /unixrepo/packages/EMC/EMCpower....rpm
  4. mkdir /mnt/ecc
  5. mount repo.example.com:/emc-ctrlctr6.1-kit /mnt/ecc
  6. /mnt/ecc/install_master.sh /mnt/ecc/

Please enter ECC root directory[/usr/ecc]: Do you want to create native installable images [Y|N]: N ECC Server Host Name 10.50.11.55 (eccserv101.example.com) ECC Server Port [5799] Master Agent\'s Listening Port [5798] Are all inputs above acceptable [Y|N|Q]: Y . . Do you want EMC ControlCenter MasterAgent to start on reboot [Y|N]: Y

  1. /etc/init.d/eccmad start
  2. ps aux |grep mstragent
  3. emcpreg -install

Key (Enter if done): XXXX-XXXX-XXXX-XXXX-XXXX-XXXX Key (Enter if done):

  1. emcpreg -list
  2. powermt set policy=co dev=emcpowera # Clariion Array devices. Use dev=all if all luns are Clariion
  3. powermt set policy=so dev=emcpowery # Symmetrix Array devices. Use dev=all if all luns are Symmetric
  4. powermt display dev=all # Make sure I/O Path Mode and State are active
  1. parted /dev/emcpowerh mklabel msdos unit s mkpart primary \"128 -1\" # for RHEL 5

or

  1. fdisk /dev/emcpowera # for RHEL 4

n # create new partition p # make it primary 1 # identify the partition with a number starting with 1 <cr> # accept default <cr> # accept default only if using the entire device. If not, enter the end block for the size you need. x # enter expert mode. b # adjust the data partition starting block 1 # enter the partition your moving 128 # move the beginning of partition data to block 128 w # write the new partition table.

  1. Make your file system.. See EXAMPLES above.

Getting the HBA WWN in RHEL 5.3 cat /sys/class/fc_host/host*/port_name

example cat /sys/class/fc_host/host*/port_name 0x2100001b3287a001 0x2100001b3287a501

is 21:00:00:1b:32:87:a0:01 21:00:00:1b:32:87:a5:01

Troubleshooting

If OEM tells you there is an error on a drive it shows it as if it was a local drive, i.e /dev/sde, rather than a power path drive, i.e /dev/emcpowere. In order to convert it to something useful for the SAN guys to investigate, run the following command:

$ sudo /usr/sbin/oracleasm querydisk -d /dev/sde1
Device "/dev/sde1" is marked an ASM disk with the label "ASMDISK01"

Main Page