UNIX find command to open permissions

Thanks to:
http://www.giannistsakiris.com/index.php/2008/12/20/unix-how-to-find-files-that-do-not-match-a-pattern-or-other-criteria/

and:
http://stackoverflow.com/questions/5119946/find-exec-with-multiple-commands

Using find we can query a storage volume and open permissions (or change permissions to other permissions) to ease permissions issues in a heterogeneous environment.

The "old" way to open all the permissions in the volume /media/volB is to run a chmod recursively on all the files in the directory:
chmod -R 777 /media/volB

This will touch each file and update its "Change" time.  That can cause issues if you are relying on that  for knowing when to archive files.

In steps using find to find everything that is not 777 permissions:
find /media/volB ! -perm 777
(The ! means "not")

adding the part that will change the permissions on the resulting files found:
find /media/volB ! -perm 777 -exec chmod 777 {} \;

done

From http://benward.me/blog/tumblr-22113784911

Accessing your iCloud Photostream on a Mac

iCloud's Photostream feature is neat. All the photographs and screen grabs I take with my iPhone or iPad are shared between the devices automatically. But getting to them on the Mac is more complicated, and if you're approaching it from a pure product perspective it appears that you have no choice but to use iPhoto or Aperture to get to the images on a Mac. Turns out this isn't true. Hunting for options, I found a Mac OSX Hints thread about finding the underlying iCloud sync location on disc, and automatically copying the files elsewhere so you can use them in Finder, or import into some other app. Read that thread if you like, but the most useful comment is the last one, by Joh:

Just use a saved search in Finder

A few nice ideas here, but the most simple way to get easily at your Photo Stream without iPhoto is to create a smart folder (saved search) in Finder with kind JPEG in the assets/sub folder. Easily accessible from the side bar, with preview icons and everything. You can even set sorting order and display type there and it will stick.

To extrapolate that into a handy step-by-step, here's how to get your Photostream in Finder:

1. Open Finder, hit ⌘⇧G, or hit Go: Go To Folder…
2. Enter ~/Library/Application Support/iLifeAssetManagement/assets/sub. Hit Enter.

You're now looking at the internal location where all the Photostream content gets synched to. It's a load of hashed directories, and within them are eventually the files themselves. If you do use iPhoto to manage Photostream, it copies new files into your library from this place.

1. Now that you're in the right place, hit ⌘F to open the search interface. You don't need to enter a search term.
2. Make sure that the search is focused on “sub”, and not “This Mac”, and then in the first Smart Search option, from the three drop-downs choose Kind is Image JPEG.
3. Now you're seeing all your Photostream photos.
4. Hit Save in the search interface, save the search as a Smart Folder in your Pictures directory, or wherever you like. You can now use this folder to access Photostream without importing into iPhoto first, and can use it to import into other apps such as Lightroom.

You can repeat the process, but choose a file type of PNG instead of JPEG, and create an equivalent Smart Folder for iOS screen grabs.

One of the other comments in the thread suggests you might still need to have the newest version of iPhoto to have the synching functionality in the first place, but since I have it already, I can't vouch for the functionality without.

Links

To share this entry, or reference it in commentary of your own, link to the following:

• Permalink: http://benward.me/blog/tumblr-22113784911
• Shortlink: http://bnwrd.me/1i48RW

Search Multiple Words / String Pattern Using grep Command

From http://www.cyberciti.biz/faq/searching-multiple-words-string-using-grep/

How do I search multiple strings or words using the grep command? For example I'd like to search word1, word2, word3 and so on within /path/to/file. How do I force grep to search multiple words?

The grep command supports regular expression pattern. To search multiple words, use following syntax:
grep 'word1\|word2\|word3' /path/to/file

In this example, search warning, error, and critical words in a text log file called /var/log/messages, enter:
$ grep 'warning\|error\|critical' /var/log/messages

To just match words, add -w swith:
$ grep -w 'warning\|error\|critical' /var/log/messages

Compare Directories using Diff in Linux

From: http://linuxcommando.blogspot.com/2008/05/compare-directories-using-diff-in-linux.html

To compare 2 files, we use the diff command. How do we compare 2 directories? Specifically, we want to know what files/subdirectories are common, what are only in 1 directory but not the other. 

Unix old-timers may remember the dircmp command. Alas, that command is not available in Linux. In Linux, we use the same diff command to compare directories as well as files.

$ diff  ~peter ~george
Only in /home/peter: announce.doc
diff /home/peter/.bashrc /home/george/.bashrc
76,83d72
<
< # Customization by Peter
< export LESS=-m
< export GREP_OPTIONS='--color=always'
< shopt -s histappend
< shopt -s cmdhist
< export PROMPT_COMMAND="history -a;$PROMPT_COMMAND"
< #echo keycode 58 = Escape |loadkeys -
Only in /home/george: .mcoprc
Only in /home/peter: .metacity
Only in /home/george: .newsticker-images
Only in /home/peter: .notifier.conf
Only in /home/george: targets.txt
Only in /home/peter: .xsession-errors

Without any option, diffing 2 directories will tell you which files only exist in 1 directory and not the other, and which are common files. Files that are common in both directories (e.g., .bashrc in the above listing) are diffed to see if and how the file contents differ.

 If you are NOT interested in file differences, just add the -q (or --brief) option.

diff -q ~peter ~george  |sort
Files /home/peter/.bashrc and /home/george/.bashrc differ
Only in /home/george: .mcoprc
Only in /home/george: .newsticker-images
Only in /home/george: targets.txt
Only in /home/peter: .metacity
Only in /home/peter: .notifier.conf
Only in /home/peter: .xsession-errors
Only in /home/peter: announce.doc

diff orders its output alphabetically by file/subdirectory name. I prefer to group them by whether they are common, and whether they only exist in the first or second directory. That is why I piped the output of diff through sort in the above command. Note that by default diff does not reach into the subdirectories to compare the files and subdirectories at that level. To change its behavior to recursively go down subdirectories, add -r.

  diff -qr ~peter ~george |sort

Creating a StorNext Filesystem (cvfs v4.3 or earlier)

You may have just provisioned some new DDN 9900 LUNs or otherwise added new storage to your StorNext based SAN.  You now need to create the filesystem.

Prerequisites:

• StorNext MDS/MDC (MetaData Server/Controller) in this example this will be built on RHEL
• A LUN presented to the MDS/MDC system(s) to be used for metadata storage.  This should be fast, capable of large IOPS, and as low latency as possible.
• LUNs presented to all of your client systems to be used for your data pool where the StorNext file system will be created and you data will be stored.

Overview:

• CVLABEL - Label the Disks / LUNs for use with StorNext
• Making the File System Config File
• CVMKFS - Make the File System
• Bringing the New File System Online
• Making the New File System Automatically Online and Mount Across Reboots

 

CVLABEL - Label the Disks / LUNs for use with StorNext

WARNING: The CVLABEL command is similar to formatting a disk!  Using the wrong device name could result in data loss!

First you need to view the LUNs that are available to StorNext.  This should be done via the StorNext command cvlabel:

NOTE: Using the command cvlabel -c will output information in the same format as is required to make a "cvlabel file" for "labeling" later in the process.

[user@system]# cvlabel -c
CvfsDisk_UNKNOWN /dev/sda    # host 2 lun 0 sectors 1170619103 sector_size 512 inquiry [DotHill DH3000          T230] serial 600C0FF000132F1A8BC8EB4E01000000
CvfsDisk_UNKNOWN /dev/sdb    # host 1 lun 0 sectors 487997407 sector_size 512 inquiry [Adaptec systemOS       V1.0] serial 166D4B28
CvfsDisk_UNKNOWN /dev/sdd    # host 2 lun 0 sectors 9376430047 sector_size 512 inquiry [DDN     S2A 9900        6.11] serial 60001FF01590A361000001000008BB85
CvfsDisk_UNKNOWN /dev/sdp    # host 2 lun 1 sectors 9376430047 sector_size 512 inquiry [DDN     S2A 9900        6.11] serial 60001FF0157EA47E010001000008BB85
CvfsDisk_UNKNOWN /dev/sdv    # host 2 lun 2 sectors 9376430047 sector_size 512 inquiry [DDN     S2A 9900        6.11] serial 60001FF01590A3C8020001000008BB85

Output this same info to a file cvlabel -c >> /path/to/file.txt to then manipulate it for "labeling".  Remove all LUNs that are not applicable by running the output through the grep command to focus on only the DotHill for metadata LUNs, DDN for data LUNs, or all LUNs from a particular host if you have multiple paths to your storage.  (ex: cvlabel -c | grep "host 2" | grep DDN >> /path to file or if you know how the system or OS disk will be presented you can show everything but that cvlabel -c | grep -v Adaptec >> /path to file)

 

The resulting info breaks down as follows:

CvfsDisk_UNKNOWN = StorNext Label

/dev/sda = Device

# host 2 lun 0 sectors 1170619103 sector_size 512 inquiry = commented (unused) info about the LUN to detarmain source

[DotHill DH3000 T230] = [ Vendor info ]

serial 600C0FF000132F1A8BC8EB4E01000000 = LUN / Disk serial number

 

Replace CvfsDisk_UNKNOWN with the desired label (ex: LUN14 to LUN16) in the text file.

META_LUN2 /dev/sda 1170619103 EFI    # host 2 lun 0 sectors 1170619103 sector_size 512 inquiry [DotHill DH3000          T230] serial 600C0FF000132F1A8BC8EB4E01000000
LUN14 /dev/sdd 9376430047 EFI    # host 2 lun 0 sectors 9376430047 sector_size 512 inquiry [DDN     S2A 9900        6.11] serial 60001FF01590A361000001000008BB85
LUN15 /dev/sdp 9376430047 EFI    # host 2 lun 1 sectors 9376430047 sector_size 512 inquiry [DDN     S2A 9900        6.11] serial 60001FF0157EA47E010001000008BB85
LUN16 /dev/sdv 9376430047 EFI    # host 2 lun 2 sectors 9376430047 sector_size 512 inquiry [DDN     S2A 9900        6.11] serial 60001FF01590A3C8020001000008BB85

 NOTE:  If you have redundant paths you only have to label each LUN once on any one of the paths, not once per path.  Also, be sure LUN labels are unique per LUN, meaning, LUN1 may be presented multiple times but in the storage device (DDN) there is only one LUN1.  If multiple unique LUNs are given the same names data corruption may occur!

 

Apply the label(s) using cvlabel /path/to/cvlabel/file.txt: (ex: cvlabel.txt)

NOTE: Be sure the LUN_label - to - device association in the file is correct or data loss could occur!  You will be prompted for each LUN prior to making changes, please read all the information completely before proceeding.

[user@system]# cvlabel cvlabel.txt

*WARNING* This program will over-write volume labels on the
          devices specified in the file "cvlabel.txt".

          After execution, the devices will only be usable by the
          StorNext. You will have to re-partition the
          devices to use them on a different file system.


Do you want to proceed? (Y / N) -> y

/dev/sdd   [DDN     S2A 9900        6.11] SNFS-EFI "LUN14"  Controller 'DAE54B28', Serial 'DAE54B28', Sector Size 512, Sectors 9376430047 (4.8TB), GUID 99a524b2-f943-11e2-b832-00259004d1b4 [Tue Jul 30 11:12:31 2013 00:25:90:04:d1:b4]

Disk "/dev/sdd" already has a valid label, and is being changed:

- The ctq_depth is changing from 0 to 16.
  This may change disk performance on IRIX systems.

Warning: The raw capacity (9376430047 sectors) of disk "/dev/sdd" might make it unusable by some older Solaris systems.

Warning: Because it will use an EFI label, disk "/dev/sdd" will not be usable by earlier StorNext releases.

Do you want to re-label it SNFS-EFI - Name: LUN14 Sectors: 9376430047 (Y / N) -> y
New Volume Label -Device: /dev/sda  SNFS Label: LUN14  Sectors: 9376430047.

 

Making the File System Config File

In some cases you are able to use an existing file system config file (or the following) and simply change a few parts to add additional file systems to an existing SAN.  For this example we will use the following existing file: volA.cfgx

<?xml version="1.0"?>
<configDoc xmlns="http://www.quantum.com/snfs" version="1.0">
  <config configVersion="0" name="volA" fsBlockSize="16384" journalSize="16777216">
    <globals>
      <abmFreeLimit>false</abmFreeLimit>
      <allocationStrategy>round</allocationStrategy>
      <haFsType>HaUnmanaged</haFsType>
      <bufferCacheSize>33554432</bufferCacheSize>
      <cvRootDir>/</cvRootDir>
      <storageManager>false</storageManager>
      <dataMigrationThreadPoolSize>128</dataMigrationThreadPoolSize>
      <debug>00000000</debug>
      <dirWarp>true</dirWarp>
      <extentCountThreshold>32768</extentCountThreshold>
      <enforceAcls>false</enforceAcls>
      <fileLocks>false</fileLocks>
      <fileLockResyncTimeOut>20</fileLockResyncTimeOut>
      <forcePerfectFit>false</forcePerfectFit>
      <fsCapacityThreshold>0</fsCapacityThreshold>
      <globalSuperUser>true</globalSuperUser>
      <inodeCacheSize>65536</inodeCacheSize>
      <inodeExpandMin>0</inodeExpandMin>
      <inodeExpandInc>0</inodeExpandInc>
      <inodeExpandMax>0</inodeExpandMax>
      <inodeDeleteMax>0</inodeDeleteMax>
      <inodeStripeWidth>0</inodeStripeWidth>
      <maxConnections>32</maxConnections>
      <maxLogSize>4194304</maxLogSize>
      <maxLogs>4</maxLogs>
      <remoteNotification>false</remoteNotification>
      <reservedSpace>true</reservedSpace>
      <fsmRealTime>false</fsmRealTime>
      <fsmMemLocked>false</fsmMemLocked>
      <opHangLimitSecs>180</opHangLimitSecs>
      <perfectFitSize>131072</perfectFitSize>
      <quotas>false</quotas>
      <restoreJournal>false</restoreJournal>
      <restoreJournalDir/>
      <restoreJournalMaxHours>0</restoreJournalMaxHours>
      <restoreJournalMaxMb>0</restoreJournalMaxMb>
      <stripeAlignSize>0</stripeAlignSize>
      <trimOnClose>0</trimOnClose>
      <threadPoolSize>64</threadPoolSize>
      <unixDirectoryCreationModeOnWindows>777</unixDirectoryCreationModeOnWindows>
      <unixIdFabricationOnWindows>false</unixIdFabricationOnWindows>
      <unixFileCreationModeOnWindows>777</unixFileCreationModeOnWindows>
      <unixNobodyUidOnWindows>501</unixNobodyUidOnWindows>
      <unixNobodyGidOnWindows>501</unixNobodyGidOnWindows>
      <windowsSecurity>false</windowsSecurity>
      <eventFiles>true</eventFiles>
      <eventFileDir/>
      <allocSessionReservationSize>1073741824</allocSessionReservationSize>
    </globals>
    <diskTypes>
      <diskType typeName="MetaDrive" sectors="1170619103" sectorSize="512"/>
      <diskType typeName="DataDrive" sectors="9376430047" sectorSize="512"/>
    </diskTypes>
    <stripeGroups>
      <stripeGroup index="0" name="MetaFiles" status="up" stripeBreadth="65536" read="true" write="true" metadata="true" journal="true" userdata="false" realTimeIOs="0" realTimeIOsReserve="0" realTimeMB="0" realTimeMBReserve="0" realTimeTokenTimeout="0" multipathMethod="rotate">
        <disk index="0" diskLabel="META_LUN0" diskType="MetaDrive" ordinal="0"/>
      </stripeGroup>
      <stripeGroup index="1" name="DataFiles_0" status="up" stripeBreadth="1048576" read="true" write="true" metadata="false" journal="false" userdata="true" realTimeIOs="0" realTimeIOsReserve="0" realTimeMB="0" realTimeMBReserve="0" realTimeTokenTimeout="0" multipathMethod="rotate">
        <affinities exclusive="false">
          <affinity>data0</affinity>
        </affinities>
        <disk index="0" diskLabel="LUN0" diskType="DataDrive" ordinal="1"/>
        <disk index="1" diskLabel="LUN1" diskType="DataDrive" ordinal="2"/>
        <disk index="2" diskLabel="LUN2" diskType="DataDrive" ordinal="3"/>
        <disk index="3" diskLabel="LUN3" diskType="DataDrive" ordinal="4"/>
        <disk index="4" diskLabel="LUN4" diskType="DataDrive" ordinal="5"/>
        <disk index="5" diskLabel="LUN5" diskType="DataDrive" ordinal="6"/>
        <disk index="6" diskLabel="LUN6" diskType="DataDrive" ordinal="7"/>
        <disk index="7" diskLabel="LUN7" diskType="DataDrive" ordinal="8"/>
        <disk index="8" diskLabel="LUN8" diskType="DataDrive" ordinal="9"/>
        <disk index="9" diskLabel="LUN9" diskType="DataDrive" ordinal="10"/>
      </stripeGroup>
    </stripeGroups>
  </config>
</configDoc>

 

The above file can be copied to make a new file system config file with some changes: volB.cfgx

<?xml version="1.0"?>
<configDoc xmlns="http://www.quantum.com/snfs" version="1.0">
  <config configVersion="0" name="volB" fsBlockSize="16384" journalSize="16777216">
    <globals>
      <abmFreeLimit>false</abmFreeLimit>
      <allocationStrategy>round</allocationStrategy>
      <haFsType>HaUnmanaged</haFsType>
      <bufferCacheSize>33554432</bufferCacheSize>
      <cvRootDir>/</cvRootDir>
      <storageManager>false</storageManager>
      <dataMigrationThreadPoolSize>128</dataMigrationThreadPoolSize>
      <debug>00000000</debug>
      <dirWarp>true</dirWarp>
      <extentCountThreshold>32768</extentCountThreshold>
      <enforceAcls>false</enforceAcls>
      <fileLocks>false</fileLocks>
      <fileLockResyncTimeOut>20</fileLockResyncTimeOut>
      <forcePerfectFit>false</forcePerfectFit>
      <fsCapacityThreshold>0</fsCapacityThreshold>
      <globalSuperUser>true</globalSuperUser>
      <inodeCacheSize>65536</inodeCacheSize>
      <inodeExpandMin>0</inodeExpandMin>
      <inodeExpandInc>0</inodeExpandInc>
      <inodeExpandMax>0</inodeExpandMax>
      <inodeDeleteMax>0</inodeDeleteMax>
      <inodeStripeWidth>0</inodeStripeWidth>
      <maxConnections>32</maxConnections>
      <maxLogSize>4194304</maxLogSize>
      <maxLogs>4</maxLogs>
      <remoteNotification>false</remoteNotification>
      <reservedSpace>true</reservedSpace>
      <fsmRealTime>false</fsmRealTime>
      <fsmMemLocked>false</fsmMemLocked>
      <opHangLimitSecs>180</opHangLimitSecs>
      <perfectFitSize>131072</perfectFitSize>
      <quotas>false</quotas>
      <restoreJournal>false</restoreJournal>
      <restoreJournalDir/>
      <restoreJournalMaxHours>0</restoreJournalMaxHours>
      <restoreJournalMaxMb>0</restoreJournalMaxMb>
      <stripeAlignSize>0</stripeAlignSize>
      <trimOnClose>0</trimOnClose>
      <threadPoolSize>64</threadPoolSize>
      <unixDirectoryCreationModeOnWindows>777</unixDirectoryCreationModeOnWindows>
      <unixIdFabricationOnWindows>false</unixIdFabricationOnWindows>
      <unixFileCreationModeOnWindows>777</unixFileCreationModeOnWindows>
      <unixNobodyUidOnWindows>501</unixNobodyUidOnWindows>
      <unixNobodyGidOnWindows>501</unixNobodyGidOnWindows>
      <windowsSecurity>false</windowsSecurity>
      <eventFiles>true</eventFiles>
      <eventFileDir/>
      <allocSessionReservationSize>1073741824</allocSessionReservationSize>
    </globals>
    <diskTypes>
      <diskType typeName="MetaDrive" sectors="xxxxxxxxxx" sectorSize="512"/>
      <diskType typeName="DataDrive" sectors="xxxxxxxxxx" sectorSize="512"/>
    </diskTypes>
    <stripeGroups>
      <stripeGroup index="0" name="MetaFiles" status="up" stripeBreadth="65536" read="true" write="true" metadata="true" journal="true" userdata="false" realTimeIOs="0" realTimeIOsReserve="0" realTimeMB="0" realTimeMBReserve="0" realTimeTokenTimeout="0" multipathMethod="rotate">
        <disk index="0" diskLabel="META_LUN2" diskType="MetaDrive" ordinal="0"/>
      </stripeGroup>
      <stripeGroup index="1" name="DataFiles_0" status="up" stripeBreadth="1048576" read="true" write="true" metadata="false" journal="false" userdata="true" realTimeIOs="0" realTimeIOsReserve="0" realTimeMB="0" realTimeMBReserve="0" realTimeTokenTimeout="0" multipathMethod="rotate">
        <affinities exclusive="false">
          <affinity>data0</affinity>
        </affinities>
        <disk index="0" diskLabel="LUN14" diskType="DataDrive" ordinal="1"/>
        <disk index="1" diskLabel="LUN15" diskType="DataDrive" ordinal="2"/>
        <disk index="2" diskLabel="LUN16" diskType="DataDrive" ordinal="3"/>
      </stripeGroup>
    </stripeGroups>
  </config>
</configDoc>

 NOTE: The following is the minimum amount of info that needs to be changed (needs to be unique across file systems or data loss can occur)!  The name of the file needs to match the file system name.  Additionally, based on the info gathered from running the CVLABEL command earlier, be sure to reflect the new metadata label and sectors and data label and sectors from the cvlabel output.

CVMKFS - Make the File System

The cvmkfs command is used to create / format the StorNext file system using the .cfgx file. Be sure to use the file system name and not the file name (ex: use cvmkfs volB not cvmkfs volB.cfgx)

[user@system]# cvmkfs volB
Checked Build disabled - default.
StorNext File System Initializer.

** WARNING ** This will destroy all contents and data of the

              --- [volB] ---

              file system. Re-initialization cannot be un-done!

Do you want to proceed? (Y/[N]) -> y

Re-initializing file system 'volB'.


*Warning*: Inode Expand Min is deprecated and will no longer be supported in future releases
*Warning*: Inode Expand Max is deprecated and will no longer be supported in future releases
*Warning*: Inode Expand Inc is deprecated and will no longer be supported in future releases
*Warning*: AllocationStrategy must be Round Robin ("round") when AllocSessionReservation is enabled.  Ignoring current setting.

Shared Meta Data File System.
Meta Data Root is on "MetaFiles".
Resetting Arbitration Block.
Writing Configuration Information Block.
Writing Root Directory Block.
Writing Disk Information Block.
Writing Stripe Group Index "MetaFiles".
Writing Stripe Group Index "DataFiles_0".
Resetting File System Journal.
Creating Free List.
Writing Initial 512 Inodes.
Inode block size is 1024 bytes.
Writing Initialization Control Block.
Setting "Force Stripe Alignment"
Resetting Super Block.
Writing Reserved Allocations.

File system 'volB' successfully created.

 

Bringing the New File System Online

Using the cvadmin interactive console you need to start then activate the file system:

[user@system]# cvadmin
StorNext Administrator

Enter command(s)
For command help, enter "help" or "?".

List FSS

File System Services (* indicates service is in control of FS):

No FSSs are active.
Select FSM "none"


snadmin> start volB
Starting FSS locally.
Start FSS "volB"

FSS 'volB' start initiated.
FSS 'volB' started.

snadmin> select
List FSS

File System Services (* indicates service is in control of FS):
 1> volB[0]              located on Venice02:36526 (pid 4814)

No FSSs are active.

snadmin> activate volB
Activate FSM "volB"

 Created           :    Wed Sep 11 17:37:36 2013
 Active Connections:    0
 Fs Block Size     :    4K
 Msg Buffer Size   :    4K
 Disk Devices      :    4
 Stripe Groups     :    2
 Fs Blocks         :    3071783232 (11.44 TB)
 Fs Blocks Free    :    3070700352 (11.44 TB) (99%)


snadmin (volB) > select
List FSS

File System Services (* indicates service is in control of FS):
 1>*volB[0]              located on Venice02:36526 (pid 4814)


snadmin (volB) > _

NOTE: the select command can be used to "refresh" the list of file systems.  Notice there is no * next to the file system prior to activation, the presence of an * indicates the file system is active.

 

Now the file system needs to me mounted for use:

[user@system]# mkdir /media/volB
[root@Venice02 ~]# mount -t cvfs volB /media/volB/
mount.cvfs: Filesystem volB mounted on /media/volB
[user@system]# mount
/dev/sdb1 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
/dev/cvfsctl1_volB on /media/volB type cvfs (rw,sparse=yes)

Your new file system should now me usable / mountable by any of your SAN clients.

Making the New File System Automatically Online and Mount Across Reboots

 The file system needs to be listed among the file systems being managed my this server (by the FSM service).  This means the file system name needs to be added to /usr/cvfs/config/fsmlist file within the MDS.

NOTE: If running multiple MDSes for failover / high availability you will need to do the following on both servers.

Edit the file /usr/cvfs/config/fsmlist and add the following:

volB . 0

NOTE: On the secondary MDS change the 0 to a 1:

volB . 1

You then need to add the mount to /etc/fstab:

LABEL=/                 /                       ext3    defaults        1 1
LABEL=backup            /mnt/backup             ext3    noauto          0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=swap              swap                    swap    defaults        0 0
volB                    /media/volB             cvfs    rw              0 0

 NOTE: It is a good idea to backup the file before making changes.

Your new filesystem should now be usable locally and persistent after reboot.

VIM substitution to prepare XML code samples for HTML blogs

I sometimes need to include XML code samples in my blogs.  When I do this I need a quick way to format it for HTML, more specifically, format it for whatever blogging software I'm using.  Since it is infrequent I just use VIM substitution (I know, I could do this in SED but for now VIM is good enough)

From http://vim.wikia.com/wiki/Search_and_replace:

:%s/foo/bar/g
Find each occurrence of 'foo' (in all lines), and replace it with 'bar'.

I search for < or > then replace them with < or > and done.

:%s/</\&lt\;/g to replace all instances of < with <
:%s/>/\&gt\;/g to replace all instances of > with >

Adding Disks and Creating LUNs in DDN9900 and DDN S7000 Chassis

Installing disks into a DDN enclosure

There are four different enclosure configurations for a DDN based SAN: 5, 10, or 20 enclosures

5 Enclosures (SS7000_1X Enclosure mapping)

BACK
I/O Modules PSU
25	26	27	28	29	30	DIMMs	25	26	27	28	29	30
19	20	21	22	23	24		19	20	21	22	23	24
13	14	15	16	17	18		13	14	15	16	17	18
7	8	9	10	11	12		7	8	9	10	11	12
1	2	3	4	5	6		1	2	3	4	5	6
A							B
FRONT

10 Enclosures (SS7000_2X Enclosure mapping)

BACK
I/O Modules PSU
49	50	51	52	53	54	DIMMs	55	56	57	58	59	60
37	38	39	40	41	42		43	44	45	46	47	48
25	26	27	28	29	30		31	32	33	34	35	36
13	14	15	16	17	18		19	20	21	22	23	24
1	2	3	4	5	6		7	8	9	10	11	12
A
FRONT

20 Enclosures (SS7000_2X Enclosure mapping)

BACK
I/O Modules PSU
49	50	51	52	53	54	DIMMs	55	56	57	58	59	60
37	38	39	40	41	42		43	44	45	46	47	48
25	26	27	28	29	30		31	32	33	34	35	36
13	14	15	16	17	18		19	20	21	22	23	24
1	2	3	4	5	6		7	8	9	10	11	12
A
FRONT

BACK
I/O Modules PSU
109	110	111	112	113	114	DIMMs	115	116	117	118	119	120
97	98	99	100	101	102		103	104	105	106	107	108
85	86	87	88	89	90		91	92	93	94	95	96
73	74	75	76	77	78		79	80	81	82	83	84
61	62	63	64	65	66		67	68	69	70	71	72
A
FRONT

 

Here is an example of a 5 enclosure install already populated with "14 tiers" of disks:

BACK
I/O Modules PSU
25	26	27	28	29	30	DIMMs	25	26	27	28	29	30
19	20	21	22	23	24		19	20	21	22	23	24
13	14	15	16	17	18		13	14	15	16	17	18
7	8	9	10	11	12		7	8	9	10	11	12
1	2	3	4	5	6		1	2	3	4	5	6
A							B
FRONT

BACK
I/O Modules PSU
25	26	27	28	29	30	DIMMs	25	26	27	28	29	30
19	20	21	22	23	24		19	20	21	22	23	24
13	14	15	16	17	18		13	14	15	16	17	18
7	8	9	10	11	12		7	8	9	10	11	12
1	2	3	4	5	6		1	2	3	4	5	6
C							D
FRONT

BACK
I/O Modules PSU
25	26	27	28	29	30	DIMMs	25	26	27	28	29	30
19	20	21	22	23	24		19	20	21	22	23	24
13	14	15	16	17	18		13	14	15	16	17	18
7	8	9	10	11	12		7	8	9	10	11	12
1	2	3	4	5	6		1	2	3	4	5	6
E							F
FRONT

BACK
I/O Modules PSU
25	26	27	28	29	30	DIMMs	25	26	27	28	29	30
19	20	21	22	23	24		19	20	21	22	23	24
13	14	15	16	17	18		13	14	15	16	17	18
7	8	9	10	11	12		7	8	9	10	11	12
1	2	3	4	5	6		1	2	3	4	5	6
G							H
FRONT

BACK
I/O Modules PSU
25	26	27	28	29	30	DIMMs	25	26	27	28	29	30
19	20	21	22	23	24		19	20	21	22	23	24
13	14	15	16	17	18		13	14	15	16	17	18
7	8	9	10	11	12		7	8	9	10	11	12
1	2	3	4	5	6		1	2	3	4	5	6
P							S
FRONT

 

Tier Information:
-----------------

                         Space
             Capacity  Available     Disk
Tier Owner     (MiB)     (MiB)      Status                LUN List
-----------------------------------------------------------------------------
   1 +  1     4578344          0  ABCDEFGHPS  0 
   2 +  2     4578344          0  ABCDEFGHPS  1 
   3 +  1     4578344          0  ABCDEFGHPS  2 
   4 +  2     4578344          0  ABCDEFGHPS  3 
   5 +  1     4578344          0  ABCDEFGHPS  4 
   6 +  2     4578344          0  ABCDEFGHPS  5 
   7 +  1     4578344          0  ABCDEFGHPS  6 
   8 +  2     4578344          0  ABCDEFGHPS  7 
   9 +  1     4578344          0  ABCDEFGHPS  8 
  10 +  2     4578344          0  ABCDEFGHPS  9 
  11 +  1     4578344          0  ABCDEFGHPS  10 
  12 +  2     4578344          0  ABCDEFGHPS  11 
  13 +  1     4578344          0  ABCDEFGHPS  12 
  14 +  2     4578344          0  ABCDEFGHPS  13 

Verifying TIER 1, 38.3%, Continuous.
Verifying TIER 3,  0.4%, Continuous.
Verifying TIER 8, 83.1%, Continuous.
Verifying TIER 10, 71.2%, Continuous.
All disks are healthy.

  Automatic disk restarting:   Disabled
  Automatic disk rebuilding:   Enabled
  Automatic SMART replacement: Disabled
  Maximum active rebuilds:     4
  Maximum active verifies:     2
  System rebuild extent:       64 MiB
  System rebuild delay:        10
  System delay IO limit:       0
  Journaling:                  Enabled                                     
                                     
              System Capacity 64096816 MiB, 0 MiB available.

 

If you were to add additional drives, say 30 drives for a total of 3 new tiers (10 drives per tier), you would add them to "15", "16", and "17" of each of the five enclosures.

BACK
I/O Modules PSU
25	26	27	28	29	30	DIMMs	25	26	27	28	29	30
19	20	21	22	23	24		19	20	21	22	23	24
13	14	15	16	17	18		13	14	15	16	17	18
7	8	9	10	11	12		7	8	9	10	11	12
1	2	3	4	5	6		1	2	3	4	5	6
A							B
FRONT

...

 Once the disks are installed you will create internal LUNs.  For ease of configuration use one LUN per Tier.  You will use the command lun add=<LUN#>:

NOTE: LUN numbers start at "0" but Tier numbers start at "1".  In our example, Tier 15 will be used to create LUN 14 which we label "LUN14"

S2A 9900[1]: lun add=14

Enter a label for LUN 14 (up to 16 characters): LUN14

Enter the capacity (in Mbytes) for LUN 14, LUN14

0 for all available capacity (default): 0

Enter the number of tiers (1..8)

Default will auto select, 'e' to escape: 1

Enter the tiers, each one on a new line, or 'e' to escape:
15

Enter the block size, (512, 1024, 2048, 4096)
Default is 512, 'e' to escape: 512

Attempting to add LUN 14
Operation successful: LUN 14 was added to the system.

The LUN must be formatted before it can be used.

Would you like to format the LUN now? (y/N): Y

INFO INIT_137 1-22 18:48:21 Starting Format of LUN 14

NOTE: The lines that say (default) in them will allow simply hitting the ENTER key to apply the default value instead of entering a value.

 

Repeat the above for each internal LUN / Tier that has been added.

NOTE: Each LUN / Tier will be "owned" by the controller that was used to create it.  To "load balance" the creation of LUNs you can alternate creation between both controllers.

 

 The following will need to be done on each controller individually

Next we need to present, or map, the internal LUN / Tier through the fiber ports as "external" LUNs.  This is done using the zoning edit=n command:

NOTE: zoning edit can be invoked without specifying a port number (=n) in which case you will be asked to specify a port number.

  S2A 9900[1]: zoning edit=1

  Enter the new LUN zoning for host port 1.

  Enter the unique LUN mapping, as follows:

  G.l   GROUP.LUN number
  P     Place-holder
  R     Before GROUP.LUN to indicate Read-Only
  N     Clear current assignment
 ENTER  No change
  E     Exit command
  ?     Display detailed help text

External LUN 0: is not mapped.  Enter new internal LUN:

NOTE: zoning edit will cycle through each external LUN number even after you have passed the number of available internal LUNs / Tiers, to escape the loop type e

CAUTION: Changes take effect immediately!  If you accidentally change the mapping for an existing LUN it could cause data loss/corruption and will most-likely at least interrupt (offline) the SAN.  Changes should be done carefully.  Mistakes are best avoided by focusing on each line individually, the line will indicate current LUN mappings.  When in doubt strike the ENTER key or use e to escape and start over.

 

Next, run tier verify=on to allow the controller to continuously check the tier for disk related problems.

S2A 9900[1]: tier verify=on

                             Tier Status     

                         Space
             Capacity  Available     Disk
Tier Owner     (MiB)     (MiB)      Status                LUN List
-----------------------------------------------------------------------------
   1 +  1     4578344          0  ABCDEFGHPS  0 
   2 +  2     4578344          0  ABCDEFGHPS  1 
   3 +  1     4578344          0  ABCDEFGHPS  2 
   4 +  2     4578344          0  ABCDEFGHPS  3 
   5 +  1     4578344          0  ABCDEFGHPS  4 
   6 +  2     4578344          0  ABCDEFGHPS  5 
   7 +  1     4578344          0  ABCDEFGHPS  6 
   8 +  2     4578344          0  ABCDEFGHPS  7 
   9 +  1     4578344          0  ABCDEFGHPS  8 
  10 +  2     4578344          0  ABCDEFGHPS  9 
  11 +  1     4578344          0  ABCDEFGHPS  10 
  12 +  2     4578344          0  ABCDEFGHPS  11 
  13 +  1     4578344          0  ABCDEFGHPS  12 
  14 +  2     4578344          0  ABCDEFGHPS  13
  15 +  1     4578344          0  ABCDEFGHPS  14
  16 +  2     4578344          0  ABCDEFGHPS  15
  17 +  1     4578344          0  ABCDEFGHPS  16

                         Tier Verification Status:

 TIER   Scheduled   Continuous  Progress    Passes  Last Verify Completion Time  

-------------------------------------------------------------------------------

   1                    Yes       0.8%          29     16:17:07 01/22/2011 
   2        No          Yes                     29     16:17:05 01/22/2011 
   3                    Yes                     28     19:25:51 01/12/2011 
   4        No          Yes                     28     11:47:25 01/16/2011 
   5                    Yes                     28     09:35:10 01/22/2011 
   6        No          Yes                     27     09:46:24 01/22/2011 
   7                    Yes                     24     15:09:06 11/28/2010 
   8        No          Yes                     24     15:20:22 12/01/2010 
   9                    Yes                     24     09:21:15 12/04/2010 
  10        No          Yes                     24     15:10:33 12/12/2010 
  11                    Yes                     24     23:28:35 12/13/2010 
  12        No          Yes                     24     12:19:44 12/21/2010 
  13                    Yes                     48     22:39:50 12/30/2010 
  14        No          Yes                      0     12:46:02 01/05/2011 
  15                    Yes                      0     21:10:56 01/05/2011 
  16        No          Yes                      0     23:02:30 01/05/2011
  17                    Yes                      0     23:02:30 01/05/2011 

  System verify extent:    32 Mbytes
  System verify delay:     40
  Maximum active verifies: 2


Please enter a TIER ('a' for all TIERs owned by this unit or 'q' to quit): a

ALL valid TIERs owned by this unit selected

Do you want the verify to run continuously? (y/N): y

 

Next, run cache writeback=on and av fastav=on

S2A 9900[1]: cache writeback=on

                          Current Cache settings

           Write   Maximum   MF  Prefetch
    LUN   Caching  Prefetch  Bit  Ceiling
 ------------------------------------------------------------------------------
      0   Enabled    x    1   On   65535
      1   Enabled    x    1   On   65535
      2   Enabled    x    1   On   65535
      3   Enabled    x    1   On   65535
      4   Enabled    x    1   On   65535
      5   Enabled    x    1   On   65535
      6   Enabled    x    1   On   65535
      7   Enabled    x    1   On   65535
      8   Enabled    x    1   On   65535
      9   Enabled    x    1   On   65535
     10   Enabled    x    1   On   65535
     11   Enabled    x    1   On   65535
     12   Enabled    x    1   On   65535
     13   Enabled    x    1   On   65535
     14   Enabled    x    1   On   65535
     15   Enabled    x    1   On   65535
     16   Enabled    x    1   On   65535

Writeback Limit: 75%

                     2560.0 Mbytes of Cache Installed
                      (2048 Segments of 1024 Kbytes)

S2A 9900[1]: av fastav=on

           Current LUN Audio/Visual settings

                                Read     Write   Maximum
  LUN  Label          FastAV Continuous Caching  Prefetch
-----------------------------------------------------------
    0 SAS_0              On      Off       On       x  1
    1 SAS_1              On      Off       On       x  1
    2 SAS_2              On      Off       On       x  1
    3 SAS_3              On      Off       On       x  1
    4 SAS_4              On      Off       On       x  1
    5 SAS_5              On      Off       On       x  1
    6 SAS_6              On      Off       On       x  1
    7 SAS_7              On      Off       On       x  1
    8 SAS_8              On      Off       On       x  1
    9 SAS_9              On      Off       On       x  1
   10 SAS_10             On      Off       On       x  1
   11 SAS_11             On      Off       On       x  1
   12 SAS_12             On      Off       On       x  1
   13 SAS_13             On      Off       On       x  1
   14 SAS_14             On      Off       On       x  1
   15 SAS_15             On      Off       On       x  1
   16 SAS_16             On      Off       On       x  1

  Disk Audio/Visual settings are:  Disabled  (Using disk defaults)
    Recovery Time Limit:           65535

  Ordered Tag Count:     0
  Unit Attention:        Enabled
  FASTAV Timeout:        200
  RC Timeout:            250
  Fail Check Condition:  Disabled

 

Multiple output files with dd utility: example

I just used this method to create a file MD5 "on the fly" meaning, one read operation when moving the file to a destination.

dd if=test.txt | tee >(md5 >> tmp/test.txt.md5) | dd of=tmp/test.txt or you can reverse the order of the destination and MD5
dd if=test.txt | tee >(dd of=tmp/test.txt) | md5 >> tmp/test.txt.md5

This worked for me.  The usage case would be when copying a file from "local" media to "remote" media (ex: local HDD to SAN volume)

Code Sample - BASH - FTP Server Management Script - IvansTool

I've created some scripts for managing an FTP server (for allowing non command line people a way to remotely manage the service via SSH)

While posting this script I see places that could be improved, simplified, and commented.  I may or may not come back and dress it up a bit... time permitting.

This script is called "IvansTool":

#!/bin/bash
#IvansTool
#
#------------------------------------------
#
#       Ivan Lawrence
#       written: Jan 26, 2010
#
#------------------------------------------
#
#   >>FUNCTIONS<<
function if0 {
if [ -z $1 ]
then
 echo "No empty variables!"
 echo "$2 was empty!"
 exit 0
#else
# echo "$2 = $1"
fi
}
function pause {
read -p "Press the [ENTER] or [RETURN] key to continue or CTR+C to cancel..."
}
#
# See what Productions have a mounted volume
function listProd {
GetProd=$(df -H | grep /data/ | awk -F" " '{print $5}' | awk -F"/" '{print $3}' | sort -d)
select ProdName in $GetProd
do
if [[ -n $ProdName ]]
then
 break
else
 echo "Select a number..."
fi
done
}
#
# See which users have been created
function listUser {
GetUser=$(grep /data/ /etc/passwd | awk -F":" '{print $1}' | sort -d)
select username in $GetUser
do
if [[ -n $username ]]
then
 break
else
 echo "Select a number..."
fi
done
}
#
# create a password from a limited "alphabet" excluding confusing chars
function passwordGen {
unset PASS
MATRIX="123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnpqrstuvwxyz!@#$%&*?-_=+)(}{][><"
LENGTH="8"
n=1
while [ "$n" -le "$LENGTH" ]
do
        RealRand=$(dd if=/dev/urandom count=1 2> /dev/null | cksum | cut -f1 -d" ")
        PASS="$PASS${MATRIX:$(($RealRand%${#MATRIX})):1}"
 #echo "$PASS"
        let n+=1
done
password=$PASS
}
#
# create a password from a limited "alphabet" excluding confusing chars and in a human friendly pattern
function custPassGen {
unset PASS
MATRIX="123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnpqrstuvwxyz!@#$%&*?-_=+)(}{][><"
Cap="ABCDEFGHJKLMNPQRSTUVWXYZ"
Low="abcdefghijkmnpqrstuvwxyz"
Punc="!@#$%&*?-_=+)(}{][><"
Num="123456789"
LENGTH="9"
n=1
while [ $n -le "$LENGTH" ]
do
 RealRand=$(dd if=/dev/urandom count=1 2> /dev/null | cksum | cut -f1 -d" ")
#       PASS="$PASS${MATRIX:$(($RealRand%${#MATRIX})):1}"
 if [[ (($n > 3)) && (($n < 7)) ]]
 then
  PASS="$PASS${Cap:$(($RealRand%${#Cap})):1}"
#  echo "$PASS"
 fi
 if [[ (($n < 3)) ]]
 then
  PASS="$PASS${Low:$(($RealRand%${#Low})):1}"
#  echo "$PASS"
 fi
 if [[ (($n == 3)) || (($n == 7)) ]]
 then
  PASS="$PASS${Punc:$(($RealRand%${#Punc})):1}"
#  echo "$PASS"
 fi
 if [[ (($n > 7)) ]]
 then
  PASS="$PASS${Num:$(($RealRand%${#Num})):1}"
#  echo "$PASS"
 fi
 let n+=1
# echo $n
done
password=$PASS
}
function emailRootAdminemail {
# experimental: I have not tested this loop for sending email to root per account!
emailDir=Scripts
address=root
if [ -f $emailDir/ftp_email_body ]
then
 sh $emailDir/ftp_email_body $1 $2 > $emailDir/ftpemail
 mail $address -s "FTP account created: $1" < $emailDir/ftpemail
 rm $emailDir/ftpemail
else
 echo "can't find $emailDir/ftp_email_body"
fi
}
function passGen {
echo "Either a totally random password:"
passwordGen
echo $password
echo "Or a formulaic password:"
echo "(lower x2, special x1, capital x3, special x1, integer x2)"
custPassGen
echo $password
}
function diskUsage {
echo "Display usage for"
echo "A - All"
echo "P - a Production"
echo "L - Logins in a Production"
read usageChoice
case $usageChoice in
 A | a ) echo "Dir  Size  Used  Avail  Use% "
 df -h | grep /data/ | awk -F" " '{print $5,$1,$2,$3,$4}' | sort -d
 ;;
 P | p ) listProd
 echo "Dir  Size  Used  Avail  Use% "
 df -h /data/$ProdName | grep /data/ | awk -F" " '{print $5,$1,$2,$3,$4}'
 ;;
 L | l ) listProd
 echo "*note: This could take a long time depending on the volume size"
 du -ch --max-depth=1 /data/$ProdName | awk -F"/" '{print $1,$4}' | grep -v lost+found
 ;;
 * ) echo "Enter your selection." ;;
esac
}
function rawSpace {
PEtotal=$(pvdisplay /dev/sdb1 | grep Total | awk '{print $3}')
PEfree=$(pvdisplay /dev/sdb1 | grep Free | awk -F" " '{print $3}')
PEsize=$(pvdisplay /dev/sdb1 | grep "PE Size" | awk -F" " '{print $4}')
GBfree=$(((($PEfree*32)-30627)/1024))
GBtotal=$(((($PEtotal*32)-30627)/1024))
echo "$GBfree GB free out of $GBtotal GB"
}
function usersFromCSV {
echo "If you used scp to get the file here I can move the file from your home dir to root's"
GetSSHuserhome=$(grep /home/ /etc/passwd | grep nologin -v | awk -F":" '{print $1}' | sort -d)
select SSHuserhome in $GetSSHuserhome root
do      
if [[ -n $SSHuserhome ]]
then
        break
else
        echo "Select a number..."
fi
done
if [[ $SSHuserhome == "root" ]]
then
 UserHomeCSV=$(find /$SSHuserhome -name *.csv)
else
 UserHomeCSV=$(find /home/$SSHuserhome -name *.csv)
 if [[ -f $UserHomeCSV ]]
 then
  read -p "Move $UserHomeCSV to /root for use? (y/n) " moveCSV
  if [[ $moveCSV == "y" || $moveCSV == "Y" ]]
  then
   mv $UserHomeCSV /root/
  else
   echo "Try again."
   exit
  fi
 else
  echo "No CSV file found in $SSHuserhome Try Again"
  exit
 fi
fi
echo "CSV must have no headers and be formatted: ProductionName, size (# in GB only), username, password, (additional info ignored)"
echo "Make sure there are no empty fields, empty lines, or unwanted spaces in lines!"
echo "*NOTE* You will enter VIM to edit the file, to remove lines navigate to the line to be deleted and strike the letter d twice."
echo "To exit VIM and continue strike ESC the type :q to quit, if changes have been made strike ESC then type :wq."
FindCSV=$(find /root -name *.csv)
read -p "I've found $FindCSV, would you like to use it? [y/n] " useResult
if [[ $useResult == "y" || $userResult == "Y" ]]
then
 vim $FindCSV
 CSVFILE=$FindCSV
else
 read -p "Path to CSV for user creation:" CSVFILE
fi
if0 $CSVFILE "CSVFILE"
read -p "Would you like a form email for each account sent to the admin account? [y/n] " sendRootEmail
echo "About to use File: $CSVFILE to create volumes and user accounts"
pause
if [ -f $CSVFILE ]
then
 exec < $CSVFILE
 while read line
 do
  TODAY=$(date +"%Y%m%d")
  TIME=$(date +"%H%M")
  ProductionName=$(echo $line | awk -F, '{print $1}')
  size=$(echo $line |awk -F, '{print $2}')
  username=$(echo $line | awk -F, '{print $3}')
  password=$(echo $line | awk -F, '{print $4}')
  if0 $ProductionName "ProductionName"
  if0 $size "size"
  if0 $username "username"
  if0 $password "password"
  echo "File entry = $ProductionName, $size G, $username, $password"
  if [ -d /data/$ProductionName ]
  then
   CHKVolGroup=$(lvdisplay | grep $ProductionName | awk -F" " '{print $3}')
   echo "Logical Volume already exists! ($CHKVolGroup)"
  else
   echo "Creating the volume for $ProductionName"
   lvcreate -L $size"g" -n $ProductionName VolGroupData 
   mkfs.ext3 /dev/mapper/VolGroupData-$ProductionName
   mkdir /data/$ProductionName
   mount /dev/VolGroupData/$ProductionName /data/$ProductionName
   cp /etc/fstab /etc/fstab.backups/fstab.$TODAY$TIME
   echo "/dev/VolGroupData/$ProductionName /data/$ProductionName ext3 defaults 0 0" >> /etc/fstab
   echo "Created volume: $ProductionName = $size G"
  fi
  id -un $username
  if [ $? == 0 ]
  then
   echo "User $username already exists!"
   echo "$password" | passwd $username --stdin
   echo "The password for $username has been updated to $password"
   # experimental: I have not tested this loop for sending email to root per account!
   if [[ $sendRootEmail == "y" || $sendRootEmail == "Y" ]]
   then
    emailRootAdminemail $username $password
   fi
  else
   useradd -m -d /data/$ProductionName/$username $username
   echo "$password" | passwd $username --stdin
   echo "User created: $username / $password"
   # experimental: I have not tested this loop for sending email to root per account!
   if [[ $sendRootEmail == "y" || $sendRootEmail == "Y" ]]
   then
    emailRootAdminemail $username $password
   fi
  fi
 #end while loop
 done
 if [[ $sendRootEmail == "y" || $sendRootEmail == "Y" ]]
 then
  echo "Accounts have been created and emails sent"
 else
  echo "Accounts have been added but no email sent"
  echo "NOTE: if you had answered Y to the erlier email question but are seeing this there was a problem"
 fi
 read -p "Delete $CSVFILE? (y/n) " CSVFileDEL
 echo "$CSVFILE and $CSVFileDEL"
 if [[ $CSVFileDEL == "y" || $CSVFileDEL == "Y" ]]
 then
  rm $CSVFILE
 else
  echo "*WARNING*: CSV file ($CSVFILE) still exists!" 
 fi
else
 echo "file not found"
 exit 0
fi
}
function userAdd {
echo "Add a Single User to an existing Production Partition"
echo "Select the Production:"
listProd
if0 $ProdName "ProdName"
read -p "Username: " username
if0 $username "username"
custPassGen
read -p "Use password $password [y] or create your own? [(c)reate]" genpass
if0 $genpass "genpass"
if [ $genpass == "c" ] || [ $genpass == "create" ]
then
 read -p "Password: " password
 if0 $password "password"
fi
read -p "Add $username to $ProdName using password $password ? [y/n]: " adduser
if0 $adduser "adduser"
if [ $adduser != "y" ]
then
 echo "No user added"
 exit 0
else
if [ -d /data/$ProdName ]
then
 id -un $username
 if [ $? == 0 ]
 then
  echo "User $username already exists!"
  exit 0
 else
  useradd -m -d /data/$ProdName/$username $username
  echo "$password" | passwd $username --stdin
  echo "User created: $username / $password"
  emailDir=Scripts
  address=dl-bur-serveradmin@starz.com
  if [ -f $emailDir/ftp_email_body ]
  then
   sh $emailDir/ftp_email_body $username $password > $emailDir/ftpemail
   mail $address -s "FTP account created: $username" < $emailDir/ftpemail
   rm $emailDir/ftpemail
  else
   echo "can't find $emailDir/ftp_email_body"
  fi
 fi
 else
  echo "Production does not exist!"
  exit 0
 fi
fi
}
function userProdAdd {
TODAY=$(date +"%Y%m%d")
TIME=$(date +"%H%M")
echo "Add a new Production Partition and user."
echo "ProductionName (all one word)"
echo "Capitalize the 1st letter of each word:"
echo "(ex: PostProd, IT, or Simpsons) "
read ProductionName
if0 $ProductionName "ProductionName"
read -p "Usable Gigs for $ProductionName: " size
if0 $size "size"
read -p "Username: " username
if0 $username "username"
#read -p "Password: " password
#if0 $password "password"
custPassGen
read -p "Use password $password [y] or create your own? [(c)reate]" genpass
if0 $genpass "genpass"
if [ $genpass == "c" ] || [ $genpass == "create" ]
then
 read -p "Password: " password
 if0 $password "password"
fi
#Check to see if the Production has a DIR
if [ -d /data/$ProductionName ]
then
 CHKVolGroup=$(lvdisplay | grep $ProductionName | awk -F" " '{print $3}')
 echo "Logical Volume / Production already exists! ($CHKVolGroup)"
 echo "Use a different Production name or choose 'Add a user to a Production'"
 exit 0
else
 echo "create the volume"
 #create the Logical Volume
 lvcreate -L $size"g" -n $ProductionName VolGroupData 
 mkfs.ext3 /dev/mapper/VolGroupData-$ProductionName
 mkdir /data/$ProductionName
 mount /dev/VolGroupData/$ProductionName /data/$ProductionName
 cp /etc/fstab /etc/fstab.backups/fstab.$TODAY$TIME
 echo "/dev/VolGroupData/$ProductionName /data/$ProductionName ext3 defaults 0 0" >> /etc/fstab
 echo "Created volume: $ProductionName = $size G"
fi
id -un $username
if [ $? == 0 ]
then
 echo "User $username already exists!"
 echo "Try Again!"
 exit 0
else
 useradd -m -d /data/$ProductionName/$username $username
 echo "$password" | passwd $username --stdin
 echo "User created: $username / $password"
 emailDir=Scripts
 address=dl-bur-serveradmin@starz.com
 if [ -f $emailDir/ftp_email_body ]
 then
  sh $emailDir/ftp_email_body $username $password > $emailDir/ftpemail
  mail $address -s "FTP account created: $username" < $emailDir/ftpemail
  rm $emailDir/ftpemail
 else
  echo "can't find $emailDir/ftp_email_body"
 fi
fi
}
function userDel {
echo "This will remove the FTP share and it's data"
listUser
usernameHome=$(grep $username /etc/passwd | cut -d ':' -f6)
id -un $username
if [ $? == 0 ]
then
 echo $username"'s home dir is " $usernameHome
 echo "If you want to also remove the Production partition then abort and choose that option."
 read -p "Continue removing only this user and it's data? [y/n]: " continue
 if [ $continue != "y" ]
 then
  echo "Nothing removed"
 else
  userdel -r $username
  echo "User $username has been deleted!"
 fi
else
 echo "User $username does not exist!"
 exit 0
fi
}
function userProdDel {
TODAY=$(date +"%Y%m%d")
TIME=$(date +"%H%M")
echo "Which production will you be destroying today?"
echo "Remember, this can not be undone!"
listProd
if0 $ProdName "ProdName"
ProdNameCK=$(grep "/data/$ProdName" /etc/fstab)
if [ -z "$ProdNameCK" ]
then
 echo "Production does not exists!"
 exit 0
else
 echo "The production has an entry in /etc/fstab of:"
 grep $ProdName /etc/fstab | cut -d ' ' -f1,2
 ProdNameFSTAB1=$(grep $ProdName /etc/fstab | cut -d ' ' -f1)
 ProdNameFSTAB2=$(grep $ProdName /etc/fstab | cut -d ' ' -f2)
 echo "The usernames associated with that Volume are:"
 ProdNamePASSWD=$(grep $ProdName /etc/passwd | cut -d ':' -f1)
 for u in $ProdNamePASSWD
 do
  echo "$u"
 done
 echo "Are you sure you want to continue?"
 echo "*** If there is no entry in /etc/fstab for $ProdName then do not continue!"
 echo "Check your spelling and try again (use the command df -H to list all volumes in use)"
 echo "Continuing will remove all the data in the FTP share(s) under $ProdName."
 read -p "Delete $ProdNameFSTAB2 **note: Once done you can not go back! [y/n] " answer1
 if [ $answer1 != "y" ]
 then
  exit
 else
  echo "Now deleteing $ProdNameFSTAB2 from the server"
  read -p "Remove users: $ProdNamePASSWD [y/n]: " userdel
  if [ $userdel != "y" ]
  then
   echo "user not deleted"
  else
   for u in $ProdNamePASSWD
   do
   userdel -r $u
   echo "User $u has been deleted!"
   done
  fi
  cp /etc/fstab /etc/fstab.backups/fstab.$TODAY$TIME
  grep $ProdNameFSTAB2 /etc/fstab >> /etc/fstab.removed
  echo $TODAY-$TIME >> /etc/fstab.removed
  grep -v $ProdNameFSTAB2 /etc/fstab > /etc/fstab.tmp
  cp /etc/fstab.tmp /etc/fstab
  umount $ProdNameFSTAB2
  lvremove $ProdNameFSTAB1
  read -p "Remove the DIR (rm -r $ProdNameFSTAB2) [y/n]: " dirrm
  if [ $dirrm != "y" ]
  then
   echo "dir not deleted"
  else
   rm -r $ProdNameFSTAB2
  fi
  echo "******** DONE ********"
 fi
fi
}
function volExtend {
echo "*Warning - Use caution, there is no checking so be sure your math is correct!*"
echo "Which production needs more space?"
listProd
LVPath=$(lvdisplay | grep $ProdName | awk '{print $3}')
CurrentSize=$(df -h | grep /data/$ProdName | awk '{print $1}' | sed s/.$//)
echo "You've selected $ProdName which is currently $CurrentSize G in size."
read -p "Please specify the new total (in Gigs)? " NewSize
echo "About to expand $ProdName (currently $CurrentSize G) to $NewSize G"
pause
#if [[ $CurrentSize -lt $NewSize ]]
#then
 lvextend -L $NewSize"g" $LVPath
 resize2fs $LVPath
#else
# echo "Error: Cannot srink the volume, the New size must be larger then the current size!"
#fi
}
function passReset {
custPassGen
read -p "Use password $password [y] or create your own? [(c)reate]" genpass
if [[ $genpass == c ]]
then
 read password
else
 echo "using $password"
fi
echo "List by production or user"
echo "P - Production"
echo "U - User"
read resetChoice
case $resetChoice in
 P | p ) listProd
 ProdNamePASSWD=$(grep $ProdName /etc/passwd | cut -d ':' -f1)
 echo "Reset all the users passwords?"
 for u in $ProdNamePASSWD
 do
  echo "Reset $u to $password"
 done
 pause
 for u in $ProdNamePASSWD
 do
  echo "$password" | passwd $u --stdin
  echo "User $u password = $password"
 done
 ;;
 U | u ) listUser
 read -p "reset $username password to $password [y/n]" ResetPass
 if [[ $ResetPass == y ]]
 then
  echo "$password" | passwd $username --stdin
  echo "user / pass is now $username / $password"
 else
  exit
 fi
 ;;
 * ) echo "Enter your selection." ;;
esac
}
function emailtest {
echo "enter the username/password you would like included in the FTP Account form email"
read -p "username:" emailtestuser
read -p "password:" emailtestpass
emailRootAdminemail $emailtestuser $emailtestpass
}
#Be sure it's running as root!
if [ "$(whoami)" != "root" ]
then
        echo "Error: You must be ROOT to add users!"
        exit 1
else
# selection=
# until [ "$selection" = "0" ]; do
  echo ""
  echo "#==> Make your choice! <==#"
  echo "1 - Display Production/disk usage"
  echo "2 - Display free raw disk space"
  echo "------------------------------"
  echo "3 - New Production and User"
  echo "4 - Add a User to a Production"
  echo "------------------------------"
  echo "5 - Remove a Production and User(s)"
  echo "6 - Remove a single User"
  echo "------------------------------"
  echo "7 - Add users or reset all passwords via CSV file"
  echo "8 - Remove users via CSV file"
  echo "------------------------------"
  echo "9 - Enlarge a production volume"
  echo "10 - Password Creator"
  echo "11 - Password Reset"
  echo "------------------------------"
  echo "12 - Generate FTP user/pass Admin email"
  echo ""
  echo "q - exit program"
  echo ""
  read -p "Enter selection: " selection
  case $selection in
   1 ) diskUsage ;;
   2 ) rawSpace ;;
   3 ) userProdAdd ;;
   4 ) userAdd ;;
   5 ) userProdDel ;;
   6 ) userDel ;;
   7 ) usersFromCSV ;;
   8 ) echo "under construction" ;;
   9 ) volExtend ;;
   10 ) passGen ;;
   11 ) passReset ;;
   12 ) emailtest ;;
   q ) exit ;;
   * ) echo "You're that stupid huh... Try Again!"
  esac
# done
fi

End