IBM Systems Group
April 2005
AIX AK April 2005
Forschungszentrum Karlsruhe
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Topics
What is supported
The Linux SCSI subsystem
Linux Kernel version 2.6
Multipathing scenarios
Tape specifics
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
What Is Supported
What is supported
The Linux SCSI subsystem
Linux Kernel version 2.6
Multipathing scenarios
Tape specifics
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
IBM Storage Support for Linux
Enterprise Storage Server
DS4000 Storage Servers
Network Attached Storage
Linear Tape Open
Enterprise Tape
Storage Virtualization
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
pLinux Support Disk
ESS: SLES8, SLES9, RH-EL 3
SDD available
JS20, p5 and OP: SLES9, RH-EL 3
Remote boot supported
DS6000, DS8000: SLES8, SLES9, RH-EL 3
SDD available
JS20: SLES8, SLES9, RH-EL 3
p5 and OP: RH-EL 3 only
Remote boot supported
DS4000: SLES8, SLES9, RH-EL 3
Emulex Multipulse driver for multipathing
SLES 9 single path only
Remote boot with JS20, others require RPQ
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
pLinux Support Tape
LTO: SLES 8, SLES 9, RH-EL 3
Data Path (for 3584) and Media Changer failover supported
359x: SLES 8, SLES 9, RH-EL 3
Data path failover supported for 3592
Parallel SCSI attachment also supported
Advanced
IBMtape device driver
IBMtapeutil
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
The Linux SCSI Subsystem
What is supported
The Linux SCSI subsystem
Linux Kernel version 2.6
Multipathing scenarios
Tape specifics
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Linux Device Addressing
Everything is a file!
brw-rw---- 1 root
brw-rw---- 1 root
disk
8, 0 2003-03-14 14:07 /dev/sda
disk
8, 1 2003-03-14 14:07 /dev/sda1
brw-rw---- 1 root
disk
3, 0 2003-03-14 14:07 /dev/hda
crw-rw---- 1 root
crw-rw---- 1 root
crw-rw---- 1 root
crw-rw---- 1 root
disk
disk
disk
disk
9, 0 2003-03-14 14:07 /dev/st0
9, 96 2003-03-14 14:07 /dev/st0a
9, 32 2003-03-14 14:07 /dev/st0l
9, 64 2003-03-14 14:07 /dev/st0m
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Design
User Space
Kernel Space
SD
disks
block device
(sd_mod.o)
SR
ST
SG
IBMTape
char device
(IBMtape.o)
cdrom/dvd
block device
(sr_mod.o)
tapes
char device
(st.o)
pass-through
char device
(sg.o)
upper level
mid level
SCSI unifying layer
(scsi_mod.o, scsi*.[hc], hosts.[hc], constants.c)
SCSI / FC Host Bus
Pseudo drivers for
non SCSI buses
(e.g. ide-scsi.o)
lower level
Adapter drivers
(e.g. qla2300.o)
SCSI / FC
disks
SCSI / FC
disks
SCSI / FC
disks
Parallel SCSI / SAN
SCSI / FC
tape
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Linux Kernel Version 2.6
What is supported
The Linux SCSI subsystem
Linux Kernel version 2.6
Multipathing scenarios
Tape specifics
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Storage Changes in Linux Kernel 2.6
Increased number of SCSI devices
Persistent device names
Improved hotplugging
Native multipathing
LVM 2
Improved I/O performance
Larger devices and filesystems
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Wellknown Linux SCSI Limitations
Limited numbe
Fixed with Kernel
Up to 256 SC
2.6
Up to 256 SCSI gc
Up to 32 tape drives
Still there
Gaps in LUN sequen
Fixed with Kernel
Limited “on-the-fl
2.6
Conditionally
fixed with Kernel
2.6
Device re-orderi
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Other Problems and Pitfalls
Multiple LUN support of RH-EL
DS4000 Specific
QLogic failover driver configuration
Potential LUN thrashing
UTM (Access LUN)
ESS, DS6000, DS8000 Specific
SDD and LVM, ext3
Mounting PPRC targets
DS6000 Preferred Path
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Large filesystems support
Linux Kernel Limits
Max file size: 2 TB (2^41
bytes)
Max file system size: 8 ZB
(2^73 bytes)
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Multipathing Scenarios
What is supported
The Linux SCSI subsystem
Linux Kernel version 2.6
Multipathing scenarios
Tape specifics
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Multipathing Concepts
Linux host 1
HBA 0
Linux host 2
HBA 0
HBA 1
HBA 1
Linux host 1
Linux host 2
HBA 1
HBA 0
HBA 1 HBA 0
1
2
DS8000
Linux host 1
Linux host 2
HBA 1
Disk
HBA 0
HBA 1 HBA 0
LUN 0
1
2
DS6000
Disk
A
B
DS4000
LUN 0
Disk
LUN 0
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
LUN Transfer to Alternate Controller
DS4000 transfers LUNs to alternate controller
Volumes are owned by one controller
Volumes can be accessed through both controllers
Volume ownership is always transferred to the controller that is used
for volume access -> transfer time approx 1 s
Two multipathing solutions available
QLogic failover driver uses AVT
Difficult to configure
Potential LUN thrashing
RDAC uses inband communication
Self configuring
Suitable for data sharing scenarios
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
LUN Thrashing Scenario
Linux host 1
Linux host 2
HBA 0
HBA 0
HBA 1
HBA 1
A
B
DS4000
Disk
LUN 0
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
LUN Thrashing Scenario
Linux host 1
Linux host 2
HBA 0
HBA 0
HBA 1
HBA 1
A
B
DS4000
Disk
LUN 0
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
LUN Thrashing Scenario
Linux host 1
Linux host 2
HBA 0
HBA 0
HBA 1
HBA 1
A
B
DS4000
Disk
LUN 0
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Multipathing with RDAC
Must use QLogic non-failover driver
Always uses current path (as reported by DS4000)
RDAC installation
FC HBA driver must be installed and loaded
At least one LUN must be assigned and available
Must use Host Type LNXCLS - AVT turned off
Must update boot loader configuration
Must run mppUpdate after each configuration change
Updates RDAC configuration files
Rebuilds Initial RAMDisk
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
RDAC Shared Data Scenario
Linux host 1
HBA 0
Linux host 2
HBA 0
HBA 1
HBA 1
A
B
DS4000
Disk
LUN 0
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
RDAC Shared Data Scenario
Linux host 1
HBA 0
Linux host 2
HBA 0
HBA 1
HBA 1
A
B
DS4000
Disk
LUN 0
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
RDAC Shared Data Scenario
Linux host 1
HBA 0
Linux host 2
HBA 0
HBA 1
HBA 1
A
B
DS4000
Disk
LUN 0
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Preferred Path
DS6000 uses concept of preferred path
Volumes are owned by one controller
Volumes can be accessed through both controllers
Data is transferred to and from owning controller to requesting
controller internally -> performance penalty
SDD knows preferred path automatically
Access only through owning controller if possible
Dynamic load balancing across ports of preferred controller
Other multipathing solutions theoretically possible, but must
(still) be configured manually
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Preferred Path Shared Data Scenario
Linux host 1
HBA 0
Linux host 2
HBA 0
HBA 1
HBA 1
1
2
DS6000
Disk
LUN 0
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Preferred Path Shared Data Scenario
Linux host 1
HBA 0
Linux host 2
HBA 0
HBA 1
HBA 1
1
2
DS6000
Disk
LUN 0
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Preferred Path Shared Data Scenario
Linux host 1
HBA 0
Linux host 2
HBA 0
HBA 1
HBA 1
A
B
DS6000
Disk
LUN 0
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Preferred Path Shared Data Scenario
Linux host 1
HBA 0
Linux host 2
HBA 0
HBA 1
HBA 1
2
DS6000
Disk
LUN 0
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Host Ports Independent of Controller
ESS and DS8000 have independent host ports
Volumes are owned by one controller
All host ports can communicate with both controllers
Dynamic load balancing across all ports possible
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Independent Host Port Shared Data Scenario
Linux host 1
HBA 0
Linux host 2
HBA 0
HBA 1
HBA 1
1
2
DS8000
Disk
LUN 0
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Independent Host Port Shared Data Scenario
Linux host 1
HBA 0
Linux host 2
HBA 0
HBA 1
HBA 1
1
2
DS8000
Disk
LUN 0
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Independent Host Port Shared Data Scenario
Linux host 1
HBA 0
Linux host 2
HBA 0
HBA 1
HBA 1
2
DS8000
Disk
LUN 0
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Tape Specifics
What is supported
The Linux SCSI subsystem
Linux Kernel version 2.6
Multipathing scenarios
Tape specifics
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
IBMtape driver
For download as binary rpm package
Kernel module IBMtape.o
Required to utilize all LTO capabilies
Manages medium changer failover
Provides new devices and ioctl
Daemon IBMtaped
NDMC-7:/ # ls -l /dev/IBM*
crw-rw-rw- 1 root
crw-r--r-- 1 root
crw-rw-rw- 1 root
crw-rw-rw- 1 root
crw-rw-rw- 1 root
crw-rw-rw- 1 root
root
253, 128 Sep 25 11:18 /dev/IBMchanger0
253, 255 Dec 9 09:41 /dev/IBMtape
253, 0 Sep 25 11:18 /dev/IBMtape0
253, 64 Sep 25 11:18 /dev/IBMtape0n
253, 1 Sep 25 11:18 /dev/IBMtape1
253, 65 Sep 25 11:18 /dev/IBMtape1n
root
root
root
root
root
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
IBMtapeUtil
For download as source code
Exerciser tool
Software example
Build and install using make
-------------------------- General Commands: -----------------------------
Provides
1. Open a Device
2. Close a Device
3. Inquiry
4. Test Unit Ready
5. Reserve Device
6. Release Device
Q. Quit IBMtapeutil
7. Request Sense
8. Log Sense Page
9. Mode Sense Page
10. Switch Tape/Changer Device
11. Create Special Files
12. Query Driver Version
IBMtapeutil
IBMtapeconfig
------------------------- Medium Changer Commands: -----------------------
60. Element Information
61. Position To Element
62. Element Inventory
63. Exchange Medium
64. Move Medium
65. Load/Unload Medium
66. Initialize Element Status
67. Prevent/Allow Medium Removal
68. Initialize Element Status Range
69. Read Device Identifiers
-------------------------- Service Aid Commands: -------------------------
70. Dump Device
71. Force Dump
72. Load Ucode
73. Reset Drive
---------------------------------------------------------------------------
99. Back To Main Menu
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Use LTO Devices
Native
tools: mt, mtx, IBMtapeutil
applications: cpio, tar, taper, afio
3rd party applications
All major backup solutions available for Linux
Attention: some are only tested with parallel SCSI attachment
Check ISV Martrix for LTO
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Native Library Management
Linux tool for media changers: mtx
Linux host
HBA
Media changer is addressed through SCSI
generic device
SCSI / FC
tape
SCSI / FC
tape
/dev/sg0- internal SCSI disk, not relevant here
/dev/sg1- 1st SCSI tape drive
Robot
/dev/sg2- tape robot (media changer)
/dev/sg3– 2nd SCSI tape drive
mtx -f /dev/sg1 inquiry
mtx -f /dev/sg2 status
mtx -f /dev/sg2 load <slotnum> [ <drivenum> ]
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Medium Changer Failover
Automatitcally moves robot control to another drive in case of a
failure
Available for 2582, 3583, 3584
Enabled as an option for IBMtape driver
Check the /proc/scsi/IBMchanger file
Linux host
HBA
SCSI / FC
tape
SCSI / FC
tape
Robot
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Tape and Disk Connected to the Same HBA
Possible, but not recommended
Use separate switch zone, too
Linux host
One driver for all HBAs!
HBA 0
HBA 2
HBA 1
SCSI / FC
tape
SCSI / FC
tape
Disk
LUN 1
Disk
LUN 0
Linux host
HBA 0
HBA 1
Robot
SCSI / FC
tape
SCSI / FC
tape
Disk
LUN 1
Disk
LUN 0
Robot
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Questions & Discussion
More Questions?
What are your customers needs?
Contact: [email protected]
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
ESS / DS6000 / DS8000 Resources
Enterprise Storage Server interoperability matrix
Subsystem Device Driver (SDD)
Fibre channel host bus adapter firmware and driver level
Additional supported configurations
ESS host systems attachment guide
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
DS4000 Resources
DS4000 Storage interoperability matrix
Fibre channel host bus adapter firmware and driver level
Additional supported configurations
DS4000 Technical Support
DS4000 Downloads
AIX-AK FZK 2005
© 2005 IBM Corporation
IBM Systems Group
LTO Resources
LTO Compatibility Information
LTO ISV Matrix
LTO Downloads
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Redbooks
Implementing Linux with IBM Disk Storage
Linux with xSeries and FAStT: Essentials
Implementing IBM LTO in Linux and Windows
Linux Clustering with CSM and GPFS
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
White Papers
FAStT and Linux HowTo
FAStT and RH AS Cluster
ESS Attachment to United Linux 1 (IA-32)
Addendum to the Solution Assurance Process
© 2005 IBM Corporation
AIX-AK FZK 2005
IBM Systems Group
Legal Notices
Both Linux and Storage are rapidly changing environments.
This information is presented "as is" without any warranty of any kind. Customers are responsible for determining the
suitability to their respective environments.
Only a representative subset of the IBM offerings are presented here. Products not mentioned should not be interpreted as a
lack or withdrawal of support of those products.
Information concerning non-IBM products was obtained from a supplier of these products, published announcement material,
or other publicly available sources. Questions on the capability of non-IBM products should be addressed to the supplier of
those products.
Some information in this presentation addresses anticipated future capabilities. Such information is not intended as a
definitive statement of a commitment to specific levels of performance, function or delivery schedules with respect to any
future products. Such commitments are only made in official IBM product announcements. All statements regarding IBM
future direction and intent are subject to change or withdraw without notice, and represent goals and objectives only.
The information is presented here to communicate IBM's current investment and development activities as a good faith effort
to help with our customers' future planning. Contact your local IBM business contact for details on specific products,
programs or services.
The following are trademarks or registered trademarks of the International Business Machines Corporation:
AIX, AS/400, AS/400e, CICS, DB2, DB2 Universal Database, e-business (logo), Enterprise Storage Server, the eServer logo,
ESCON, FlashCopy, IBM, Intellistation, iSeries, Magstar, Modular Storage Server, MQSeries, Netfinity, NUMA-Q, OS/390,
OS/400, Parallel Sysplex, pSeries, RS/6000, S/390, SANergy, Seascape, Sequent, Sequent (logo), SP, SP2, SSA,
StorWatch, Thinkpad, Tivoli, Tivoli Storage Manager, Ultrastar, WebSphere, xSeries, zSeries.
Microsoft, Windows, Windows NT and the Windows logo are registered trademarks of Microsoft Corporation. Intel and
Pentium are registered trademarks of Intel Corporation. UNIX is a registered trademark licensed exclusively through the
OPEN group. LINUX is a registered trademark of Linus Torvalds. Java and all Java-based trademarks and logos are
trademarks of Sun Microsystems, Inc.
Red Hat, the Red Hat "Shadow Man" logo, RPM and all Red Hat-related logos are trademarks or registered trademarks of
Red Hat, Inc. Caldera Systems, the C-logo, SCO, and related logo, are trademarks or registered trademarks of Caldera
Systems, Inc. Turbolinux and "lightning bolt" logo are registered trademarks of Turbolinux, Inc. SuSE, and SuSE "lizard"
logo, are trademarks of SuSE, Inc.
Linux is a registered trademark of Linus Torvalds
Other company, product and service names may be trademarks or service marks of others.
© 2005 IBM Corporation
AIX-AK FZK 2005
|