VAX/VMS VOLUME SHADOWING What is VAX/VMS Volume Shadowing? o Cooperative effort between a HSC50 and VMS o Increased data availability - Survive failure of a single disk o Increased data accessibility - More read/write heads per datum Elements of a Shadow Set +----------+ | | | VIRTUAL | | UNIT | | | +----------+ +----------+ +----------+ ............ | | | | . . | PHYSICAL | | PHYSICAL | . . | UNIT | | UNIT | . . | | | | . . +----------+ +----------+ ............ Elements of a Shadow Set All physical unit in a shadow set MUST o Have the same geometry o Have the same physical properties - Write protection - Dual-path connections o Conform to the Digital Storage Architecture - Must have not bad blocks Cooperation Between HSC50 and VMS o Attempts to have each party perform functions best suited to its abilities o Defined by Mass Storage Control Protocol, part of Digital Storage Architecture o Most complex interactions occur during addition or removal of a member VMS Support for Volume Shadowing Transparent for FILES-11 ODS-2 volumes o When mounting a shadow set, guarantee consistancy o Automatic recovery from loss of a member o Notification of reduction of shadow set membership o Automatically rebuild shadow set after HSC50 reboot o Automatically fail shadow set to another HSC50, if possible FILES-11 Shadow Set Mount Maintaining Consistency o Volumes Identical o Volumes Different - Copy newest, most correct member to the other o Volumes Differ Only By Incomplete Write Requests - Copy either member to the other FILES-11 Shadow Set Mount Maintaining Consistency The following information is used to determine which type of copy operation is required, if any. o Volume Label o Correctly Dismounted o BACKUP Revision Number o Shadowing Generation Number FILES-11 Shadow Set Mount Maintaining Consistency Shadowing Generation Numbers Properties of a shadowing generation number o Monotonically increasing o Represents the time an event occured FILES-11 Shadow Set Mount Maintaining Consistency Shadowing Generation Numbers How to Make a Newer Generation Number if OLD_NUMBER < CURRENT_TIME then NEW_NUMBER <-- CURRENT_TIME else NEW_NUMBER <-- OLD_NUMBER + 1 FILES-11 Shadow Set Mount Maintaining Consistency Shadowing Generation Numbers When to update a shadowing generation number o Upon completion of processing for any change of shadow set membership - Addition of a volume - Removal of a volume due to hardware failure - Removal of a volume by user request o Immediately before dismounting the entire shadow set FILES-11 Shadow Set Mount Maintaining Consistency Shadowing Generation Numbers When are two shadow set volumes identical? Volume labels are Identical Volume correctly dismounted Yes BACKUP revision numbers are Equal Generation numbers are Equal In this case, no copy operation is required. FILES-11 Shadow Set Mount Maintaining Consistency Shadowing Generation Numbers When do two shadow set volumes differ by incomplete writes Volume labels are Identical Volume correctly dismounted No for both volumes BACKUP revision numbers are Equal Generation numbers are Equal In this case, a merge copy operation is required. FILES-11 Shadow Set Mount Maintaining Consistency Shadowing Generation Numbers When do two shadow set volumes differ due to an improper dismount of one Volume labels are Identical Volume correctly dismounted newer is Yes older is NO BACKUP revision numbers are Equal Generation numbers are Equal In this case, a normal copy operation is required from the newer to the older. FILES-11 Shadow Set Mount Maintaining Consistency Shadowing Generation Numbers When do two shadow set volumes differ due to shadow set membership change Volume labels are Identical Volume correctly dismounted not used BACKUP revision numbers are Equal Generation numbers are newer > older In this case, a normal copy operation is required from the newer to the older. FILES-11 Shadow Set Mount Maintaining Consistency Shadowing Generation Numbers When do two shadow set volumes differ due to BACKUP /IMAGE Volume labels are Identical Volume correctly dismounted not used BACKUP revision numbers are newer > older Generation numbers are not used In this case, a normal copy operation is required from the newer to the older. FILES-11 Shadow Set Mount The $MOUNT request completes as soon as the shadow copies are started o Shadow copies operations are time consuming o They can be performed in parallel with regular shadowed I/O requests o Special mount verification service controls shadow copies Removing a Failed Shadow Set Member o Shadow set member fails - HSC50 notes failure - Virtual unit becomes inaccessible o VMS "notices" failure or inaccessible virtual unit o Mount verification invoked Removing a Failed Shadow Set Member Mount Verification Processing o Query HSC50 for shadow set membership o Remove failed members o Add new members starting copies where appropriate Removing a Failed Shadow Set Member Mount Verification Processing o Make virtual unit accessible o Validate volume on virtual unit o Update generation number o Restart failed I/O operations and allow user access to virtual unit Shadow Set Reduced to Zero Members Attempt to rebuild shadow set using member(s) present before failure which produced the zero member shadow set until: 1. Shadow set rebuilds 2. Mount verification timeout interval expires Controller Failure Recovery o Wait for at least one member to be accessible somewhere, then wait five seconds more o Form shadow set from lowest working member, remove non-working members o Start merge copies to add remaining members, remove non-working members Controller Failure Recovery o Make virtual unit accessible o Validate volume on virtual unit o Update generation number o Restart failed I/O operations and allow user access to virtual unit Shadowed System Disks - Mechanics o One physical and one virtual unit number supplied to primary bootstrap - If virtual unit exists, primary bootstrap uses it - Otherwise, physical unit is used o Mount single volume shadowed system disk in SYSINIT o Add other system disk shadow set members with $MOUNT commands in SYSTARTUP Shadowed System Disks - Mechanics o Write dumps to: - Virtual unit, if it exists - Otherwise, physical unit o Update generation number Shadowed System Disks - Implications o Shadowed system disks for single nodes will often dissolve automatically - Boot-time system parameters and dumps will be written a single member of the shadow set - Copies to remaining members will be more frequent - Physical unit named to primary bootstrap is a single point of failure o Mounting the shadowed system disk on another cluster member solves these problems. Volume Shadowing and VAXclusters The shadow set membership must be the same on all VAXcluster nodes using the shadow set and on the HSC50. o Adding and removing shadow set members are automatically performed throughout the VAXcluster, regardless of /CLUSTER. o Mounting or dismounting the shadow set virtual unit is a per-node operation unless the /CLUSTER qualifier is used. $MOUNT Commands For a single FILES-11 volume $ MOUNT DJS23: - $_ /SHADOW = ( DJA9:, DJA5:, DJA10: ) - $_ MYVOLUME DISK$MYVOLUME $MOUNT Commands For a FILES-11 volume set $ MOUNT DUS23: - $_ /SHADOW = ( DUA9:, DUA5:, DUA10: ), - $_ DUS51: - $_ /SHADOW = ( DUA12:, DUA3:, DUA34: ) - $_ WORK1,WORK2 WRKD$ $DISMOUNT Commands o For a shadow set member $ DISMOUNT DUA34: o For a shadow set virtual unit $ DISMOUNT DJS23: