 |
|
This example deals with scenarios where files/folders in the Xsan need to be recovered under certain corruption scenarios. For example, a very low level hardware error maybe developing resulting in input/output errors while attempting to use the ls command to access files/folders. Such files/folders may not be seen through the Finder.
For another example, one or both the inodes sb_NTSecurityIdxInode and sb_NTSecurityDatInode referenced to in the Superblock might be damaged due to ACL corruption issues or low level disk issues. In the case of some major damage to sb_NTSecurityIdxInode, the file system might start, but the user may be unable to mount the volume. In the case of some major damage to sb_NTSecurityDatInode, the file system might appear to mount. Yet there might be input/output errors or entire files/folders may appear missing when listing through a terminal while Finder may not even show the file system or show incorrect information such as wrong icons, folders appearing as files etc.
Please note that, in some cases of damage to sb_NTSecurityDatInode, it may be possible to mount and access the contents of the volume this way: Edit the volume config file and use the value "No" for the following parameters: "Quotas," "WindowsSecurity," and "EnforceACLs," then restart and mount the volume (if the value for the "Quotas" parameter is "Yes," a value of "No" for the WindowsSecurity parameter has no effect, so we want to set the value "No" for both of them).
Also, according to the cvfs_config man page "Once WindowsSecurity has been enabled the file system will track Windows access lists for the duration of the file systems life, regardless of the WindowsSecurity value," so the sb_NTSecurityIdxInode and sb_NTSecurityDatInode will not be empty once ACLs have been activated and applied on, even if the ACLs are removed and deactivated at a later stage. Thus if the sb_NTSecurityIdxInode is damaged, the user may not be able to mount the volume even if the values for "Quotas," "WindowsSecurity," and "EnforceACLs" are set to "No" in the volume config file.
The following assumes that the administrator is operating from the metadata controller. The name of the Xsan volume is Testvol.
Step I.
The first step is to unmount the volume from the clients and then stop the volume. Then, executing the following command (as root or using sudo if not root) should generate and save a list of all files and folders Xsan can “see:”
# cvfsck –x Testvol > allfilesfolders
The file allfilesfolders generated above will have comma separated entries with the following information
Inode#, Mode, Size, Block Count, Affinity, Path, Extent Count, Extent Number, Storage pool, File Relative Block, Base, End, Depth, Breadth
Example entry:
3c4dcd,16877,2048,1,Data,Projects/sunshine,1,0,0,0,3858,3858,0,256
3c4de8,33188,0,0,none,Projects2/moonshine,0,0,0,0,0,0,0,0
Note: Despite what the man page for cvfsck says, the Inode# is reported in hex and not in decimal. The rest of the numerical values are in decimal.
Of particular interest is the second value, ie., mode. For example, if the mode value is 16877, then in octal it is 040775, so the entry corresponds to a directory with 755 permissions, if it is 33206, then in octal it is 100666, so file with permissions 666, if it is 33188, then in octal it is 100644, so file with permissions 644. If the mode is 41471, then it is 0120777 in octal, so the entry is a symbolic link.
If the only interest is in the “missing” Projects folder, something like below should give all the files and folders under the Projects hierarchy:
# grep ,[-_A-Za-z0-9][-_A-Za-z0-9]*,Projects, allfilesfolders > alllostProjects
# grep ,[-_A-Za-z0-9][-_A-Za-z0-9]*,Projects\/ allfilesfolders >> alllostProjects
Otherwise if you want to recover the whole Xsan use the allfilesfolders in the sequence below.
It may be a good idea to move the Path entries (the sixth field) to the last. This helps in sorting with specific numerical fields. Remember, Mac OS X Finder allows the ability of creating file and directory names with all kinds of special characters including commas, so something like below may be used to effect this:
# sed -e 's/^\([0-9a-f][0-9a-f]*\),\([0-9][0-9]*,[0-9][0-9]*,[0- 9][0-9]*,[-_A-Za-z0-9][-_A-Za-z0-9]*\),\(.*\),\([0-9][0-9]*,[0- 9][0-9]*,[0-9][0-9]*,[0-9][0-9]*,[0-9][0-9]*,[0-9][0-9]*,[0-9][0- 9]*,[0-9][0-9]*\)$/0x\1,\2,\4,\3/' > alllostProjects.$$ < alllostProjects
A file might have multiple extents and we need only one entry for this procedure. As mentioned before, the Finder allows all kinds of special characters as part of file or directory names (e.g, ', /, \,@,%,¬,®,€,£,~, [, }, and what not, in addition to Unicode characters such as Ç, Ä,ö,Ã,ß and so forth), so the most important thing is to escape single quotes since it is critical for a lot of commands to work. Something like below achieves all these objectives.
# sort -t , -k 1,6 -um < alllostProjects.$$ | sed -e "s/'/'\\\''/g" > alllostProjects. onlyonce.$$
Note: It is possible that files might cross multiple storage pools (eg., the customer has enabled the InodeStripeWidth parameter in his Xsan config file). In which case, the above sort sequence and pipe needs to be slightly modified based on the Pool number key to avoid wasting time due to restore of such files multiple times.
Step II.
An estimate of how much restore space is required can be made now before proceeding further.
estimatedsize=`awk -F, '$2 ~ /^3[0-9][0-9]*/ {s += $3}; END{print s}' < alllostProjects.onlyonce.$$ | xargs -J % echo % /1024 | bc`
Based on this size, a restore volume may be attached, say, /Volumes/restore, where all the files and folders will be restored.
At the outset, any Unix soft links or hard links can be separated. These can be manually dealt with later:
cut -f 1-13 -d"," alllostProjects.onlyonce.$$ | sort | uniq -d > alllostProjects _hlnopath.$$
exec 3<&0
exec 0< alllostProjects_hlnopath.$$
while read line
do
grep "$line" alllostProjects.onlyonce.$$ >> alllostProjects_hard_links.$$
sed -i tmp1 "/$line/d" alllostProjects.onlyonce.$$
done
exec 0<&3 3<&-
cut -f 1-13 -d"," alllostProjects_symbolic_links.$$ > alllostProjects_slnopath.$$
exec 3<&0
exec 0< alllostProjects_slnopath.$$
while read line
do
sed -i tmp2 "/$line/d" alllostProjects.onlyonce.$$
done
exec 0<&3 3<&-
Step III.
The inodes, modes, and paths may be extracted this way:
# sed -e 's/^\(0x[0-9a-f][0-9a-f]*\),\([0-9][0-9]*\),\([0-9][0- 9]*,[0-9][0-9]*,[-_A-Za-z0-9][-_A-Za-z0-9]*,[0-9][0-9]*,[0-9][0- 9]*,[0-9][0-9]*,[0-9][0-9]*,[0-9][0-9]*,[0-9][0-9]*,[0-9][0- 9]*,[0-9][0-9]*\),\(.*\)$/\1,\2,\4/' > alllostProjects_inode_mode_path.$$ < alllostProjects.onlyonce.$$
Step IV.
The directory entries maybe extracted using the mode value. However, it is possible that the damage is such that not all directories can be captured this way, so the last entry in the file paths can be chopped to sort and extract any missing directories; then all the directories can be created under /Volumes/restore:
# awk -F, '$2 ~ /^1[0-9][0-9]*/ {print}' < alllostProjects_inode_mode_path.$$ | sed -e "s/^\(0x[0-9a-f][0-9a-f]*\),\([0-9][0-9]*\),\(.*\)$/\3/g" > alllostProjects_dirlist.$$
# awk -F, '$2 ~ /^3[0-9][0-9]*/ {print}' < alllostProjects_inode_mode_path.$$ | sed -e "s/^\(0x[0-9a-f][0-9a- f]*\),\([0-9][0-9]*\),\(.*\)$/dirname '\3'/g" | /bin/sh >> alllostProjects_dirlist.$$
# sed -e "s/'/'\\\''/" < alllostProjects_dirlist.$$ | sed -e "s/\(.*\)/mkdir -p \/Volumes\/restore\/'\1'/" | sort -r | uniq | /bin/sh
Recall the ability of Finder to create file (and directory) names with all kinds of special characters. cvfsdb cannot handle these characters during the restore process, but the unix mv command can. The idea thus, is to associate each file with a unique alphanumeric identifier, files saved first to these identifiers, then moved back to their original names. A natural choice for this is md5 checksums: to each file path, a unique md5 checksum is associated:
# mkdir -p /Volumes/restore/md5_files
# awk -F, '$2 ~ /^3[0-9][0-9]*/ {print}' < alllostProjects_inode_mode_path.$$ | sed -e "s/^\(0x[0-9a-f][0-9a-f]*\),\([0-9][0-9]*\),\(.*\)$/\/bin\/echo \1,\`echo \/Volumes\/restore\/'\3' \| md5 \`,'\3'/g" | /bin/sh | sed -e "s/'/'\\\''/g" > alllostProjects.inodes_md5checksum_path.$$
The contents of the file nodes can now be copied to their respective md5 checksums using cvfsdb:
# awk -F, '{print "echo save "$1" \/Volumes\/restore\/md5_files\/"$2" \| cvfsdb Testvol"}' < alllostProjects.inodes_md5checksum_path.$$ | /bin/sh
Now, it is a simple matter of moving the md5 checksums to their corresponding file names:
# sed -e "s/^\(0x[0-9a-f][0-9a-f]*\),\([a-z0-9][a-z0- 9]*\),\(.*\)$/mv \/Volumes\/restore\/md5_files\/'\2' \/Volumes\/restore\/'\3'/" < alllostProjects.inodes_md5checksum_path.$$ | /bin/sh
Using the mode value (or extracting and parsing the idi_mode value using cvfsdb), correct file and directory permissions can actually be extracted and set. The mode value is used here:
# sed -e "s/^\(0x[0-9a-f][0-9a-f]*\),\([0-9][0-9]*\),\(.*\)$/chmod \`printf \"\%lo\" \2 \| cut -c 3-6 \| awk \'\{print \$1 + 0\}\'\` \/Volumes\/restore\/'\3'/g" < alllostProjects_inode_mode_path.$$ | /bin/sh
Using cvfsdb, appropriate ownership of files and directories can be extracted and the set in the restore area:
# sed -e "s/^\(0x[0-9a-f][0-9a-f]*\),\([0-9][0-9]*,[0-9][0-9]*,[0- 9][0-9]*,[-_A-Za-z0-9][-_A-Za-z0-9]*,[0-9][0-9]*,[0-9][0-9]*,[0- 9][0-9]*,[0-9][0-9]*,[0-9][0-9]*,[0-9][0-9]*,[0-9][0-9]*,[0-9][0- 9]*\),\(.*\)/echo show inode \1 \| cvfsdb Testvol \| grep -e idi_uid -e idi_gid \| awk -F\"= \" '\{printf \"\%d\\\n\",\$2\}' \| xargs -n 2 \| tr \" \" : \| xargs -J \% chown \% \/Volumes\/restore\/'\3'/" < alllostProjects.onlyonce.$$ | /bin/sh
Step V.
Now everything under Projects is restored under /Volumes/restore/Projects. All that remains is the hard links and symbolic links. Recall that the two files alllostProjects_symbolic_links.$$ and alllostProjects_hard_links.$$ were created earlier.
Use cvfsdb on each of the inodes in the alllostProjects_symbolic_links.$$ files and examine the “Extended Attributes” section to find the original file path and create the symbolic link appropriately in the restore area.
As far as the hard links go, all the links and the original file will have the same inode. Again use cvfsdb to show the inode information from the alllostProjects_hard_links.$$ file and examine the “Extended Attributes” area. The first entry will be the original file and the other entries will be the hard links, so you can restore the original file under /Volumes/restore using cvfsdb, then can create the hard links as appropriate.
A sample script along the above lines may be found here
Use it at your own risk.
|
|
Apple Solutions 
Back