Saturday, November 10, 2012

netapp

<-- Please click if you found this site useful ;-)

NetApp

NetApp 101

https://netapp.myco.com/na_admin  # web gui URL.  Most feature avail there, including a console.

ssh netapp.myco.com   # ssh (or rsh, telnet in) for CLI access

get root mount of /vol/vol0/etc in a unix machint do to direct config on files.


NOW = NetApp Support Site
NetApp man pages ("mirror" by uwaterloo)
RAID-DP

IMHO Admin Notes

Notes about NetApp export, NFS and Windows CIFS ACL permission issues.

Best practices is for most (if not all) export points of NFS server is to  
implement root_squash.  root on
the nfs client is translated to user 'nobody' and would effectively have
the lowest access permission.  This is done to reduce accidents of user
wiping out the whole NFS server content from their desktops.

Sometime NetApp NFS exports are actually on top of filesystem using windows NT ACL,
their file permission may show up as 777, but when it comes to accessing
the file, it will require authentication from the Windows server (PDC/BDC 
or AD).  Any user login name that does not have a match in
windows user DB will have permission denied problems.  

Most unix client with automount can access nfs server thru /net.
However, admin should discourage the heavy reliance on /net.  It is good
for occassional use.
/home/SHARE_NAME or other mount points should be 
provided, such as /corp-eng and /corp-it.  This is because mount path will 
be more controllable, and also avoid older AIX bug of accessing /net when 
accessing NFS mounted volumes, access them as user instead of root, which 
get most priviledges squashed away.
If the FS is accessible by Windows and Unix, it is best to make share name
simple and keep them consistent.  Some admin like to create
matching 
\\net-app-svr1\share1   /net-app-svr1/share1
\\net-app-svr2\share2   /net-app-svr2/share2
I would recommend that in the unix side, that /net-app-svr1 be unified into a
single automount map called like /project .  This would mean 
all share names need to be uniq across all servers, but it help keep
transparency that allows for server migration w/o affecting user's work
behaviour.


Old Filer to New Filer Migration problems:

If copy files from Unix FS to Windows-style FS, there are likely going to 
be pitfalls. NDMP would copy the files, and permissions and date would be 
preserved, but ownership of the files may not be preserved.  XCOPY from 
DOS (or robocopy) may work a tad better in the sense that the files will 
go thru the normal windows access of checking access and ownership 
creation. Clear Case needed to run chown on the files that correspond to 
the view, and not having the ownership preserved becomes a big problem.  
Ultimately, User that run CC script for ownership change was made part of 
the NetApp Local Admin Group.  A more refined ACL would be safer.

Filer data migration:

NDMP is the quickest.  One can even turn off NFS and CIFS access to ensure 
no one is writting to the server anymore.  NDMP is a different protocol 
with its own access mechanism.


Mixed NFS and CIFS security mode:

Mix mode security (NT and Unix) is typically a real pain in the rear.
Migrating from NT a/o Unix to mix mode would mean filer has to fabricate
permissions, which may have unintenteded side effects.
Switch from mixed mode to either NT or Unix just drop the extra permission
info, thus some consultant say this is a safer step.

Clear Case and NetApp each point the other as recommending Mixed Mode
security.  It maybe nighmare if really used.  Unix mode worked flawlessly for
3+ years.

Different NetApp support/consultant says different things about mix mode, 
but my own experience match this description:
Mix-Mode means the filer either store Unix or NTFS acl on a file by file basis.
If a given file (or dir) ACL is set on unix, it will get to have only Unix ACL on it.
If last set on NTFS, then it will get Windows ACL.  
The dual mode options is not both stored, only one of the two is stored, and the rest
resolved in real time by the filer.  
This has a nasty side effect that flipping security style from mixed mode to say NTFS,
some files permissions are left alone and even windows admin can't change/erase the files, 
because they are not seen as root.
In short, avoid mix-mode like a plague!!


LVM


Layers:

Qtree, and/or subdirectories, export-able
  |
Volume (TradVol, FlexVol), export-able, snapshot configured at this level.
  |
agregate (OnTap 7.0 and up)
  |
 plex      (relevant mostly in mirror conf)
  |
raid group
  |
disk


Disks - Physical hardware device :)
   Spares are global, auto replace failed disk in any raid group.
   Sys will pick correct size spare.
   If no hot spare avail, filer run in degraded mode if disk fail, and
   def shutdown after 24 hours!  (options raid.timeout, in hours)

   sysconfig -d  # display all disk and some sort of id
   sysconfig -r  # contain info about usable and physical disk size
      # as well as which raid group the disk belongs to

 disk zero spare  # zero all spare disk so they can be added quickly to a volume.
 vol status -s  # check whether spare disks are zeroed 

   web gui: Filer, Status 
    = display number of spares avail on system

   web gui: Storage, Disk, Manage 
    = list of all disks, size, parity/data/spare/partner info, 
    which vol the disk is being used for.
    (raid group info is omited)

   Disk Naming:  
   .
   2a.17  SCSI adaptor 2, disk scsi id 17
   3b.97  SCSI adaptor 3, disk scsi id 97

   a = the main channel, typically for filer normal use
   b = secondary channel, typically hooked to partner's disk for takeover use only.



Raid group - a grouping of disks.  
  Should really have hot spare, or else degraded mode if disk fail, and shut
 down in 24 hours by def (so can't tolerate weekend failure).

        max raid group size:
              raid4    raid-dp   (def/max)
 FC            8/14     16/28 
 SATA, R200    7/7      14/16

 Some models are slightly diff than above.


 Raid-DP?  
 2 parity disk per raid group instead of 1 in raid4.
 If you are going to have a large volume/aggregate that spans 2 raid group (in
 a single plex), then may as well use raid-dp.
 Larger raid group size save storage by saving parity disk.
 at expense of slightly less data safety in case of multi-disks failure.
 

Plex
 - mirrored volume/aggregate have two plexes, one for each complete copy of the
     data.
        - raid4/raid_dp has only one plex, raid groups are "serialized".


aggregate - OnTap 7.0 addition, layer b/w volume and disk.  With this, NA
 recommend creating a huge aggregate that span all disks with 
 same RAID level, then carve out as many volume as desired.


Volume - traditional mgnt unit, called an "independent file system".
     aka Traditional Volume, starting in OnTap 7.0
     Made up of one ore more raid groups. 
     -  disk(s) can be added to volume, default add to existing raid group
 in the vol, but if it is maxed out, then it will create a new raid
 group.
     - vol size can be expanded , but no strink, concat or split.
     - vol can be exported to another filer (foreign vol).
     - small vol implies small raid group, therefore waste more space.
     - max size = 250 GB recommended max vol size in 6.0

     vol status -v [vol0] # display status of all [or specific] volume,
           # -v gives all details on volume options
     vol lang    vol0         # display [set] character set of a volume

     vol status -r  # display volume and raid status
     sysconfig  -r   # same as vol status -r

     vol create newvol  14 # create new vol w/ 14 disks
     vol create newvol2 -t raid4 -r 14 6@136G
  # vol size is 6 disks of 133 GB
  # use raid4 (alt, use raid_dp)
  # use raid group of 14 disks (def in cli), 
  # each raid group need a parity disk, so
  # larger raid group save space (at expense of ??)
  # 28 disks usable in raid_dp only?


     vol add newvol2 3  # add 3 more disks to a volume
     vol options vol1 nosnap on # turn off snapshot on a vol
     vol offline vol2
     vol online  vol2

FlexVol - OnTap 7.0 and up, resembles a TradVol, but build ontop of aggregate
    - grow and srink as needed


QTree  - "Quota Tree", store security style config, oplock, disk space usage and file limits.
      Multiple qtrees per volume.  QTrees are not req, NA can hae simple/plain 
      subdir at the the "root level" in a vol, but such dir cannot be converted to qtree.
      Any files/dirs not explicitly under any qtree will be placed in a
      default/system QTree 0.

    qtree create   /vol/vol1/qtree1  # create a qtree under vol1
    qtree security /vol/vol1/qtree1 unix # set unix security mode for the qtree
             # could also be ntfs or mixed
    qtree oplocks  /vol/vol1/qtree1 enable # enable oplock (windows access can perform catching) 

Config Approach
Aggregate:
Create largest aggregate, 1 per filer head is fine, unless need traditional vol.

Can create as many FlexVol as desired, since FlexVol can growth and srink as needed.
Max vol per aggregate = 100 ??

TradVol vs QTree?
- use fewer traditional volume when possible, since volume has parity disk overhead
- and space fragmentation problem.
- use QTree as size management unit.


FlexVol vs QTree?
- Use Volume for same "conceptual management unit"  
- Use diff vol to separate production data vs test data
- QTree should still be created under the volume instead of simple plain subdirectories
  at the "root" of the volume. 
  This way, quota can be turned on if just to monitor space usage.
- One FlexVol per Project is good.  Start Vol small and expand as needed.
  Strink as it dies off.
- Use QTree for different pieces of the same project.
- Depending on the backup approach, smaller volume may make backup easier.
  Should try to limit volume to 3 TB or less.


Quotas

mount root dir of the netapp volume in a unix or windows machine.
vi (/) etc/quotas   (in dos, use edit, not notepad!!)
then telnet to netapp server, issue command of quota resize vol1 .

quota on  vol1
quota off vol0
quota report
quota resize # update/re-read quotas (per-vol)
  # for user quota creation, may need to turn quota off,on for volume
  # for changes to be parsed correctly.

Netapp quota support hard limit, threshold, and soft limit.
However, only hard limit return error to FS.  The rest is largely useless, 
quota command on linux is not functional :(


Best Practices:

Other than user home directory, probably don't want to enforce quota limits.
However, still good to turn on quota so that space utilization can be monitored.


/etc/quotas
##                                           hard limit | thres |soft limit
##Quota Target       type                    disk  files| hold  |disk  file
##-------------      -----                   ----  -----  ----- ----- -----

*         tree@/vol/vol0   -       -       -       -       - # monitor usage on all qtree in vol0
*         tree@/vol/vol1   -       -       -       -       -
*         tree@/vol/vol2   -       -       -       -       -

/vol/vol2/qtree1     tree                200111000k  75K  - - - # enforce qtree quota, use kb is easier to compare on report
/vol/vol2/qtree2     tree                    -      - 1000M   - - # enable threshold notification for qtree (useless)


*                       user@/vol/vol2        - -       -       -       - # provide usage based on file ownership, w/in specified volume
tinh                    user                 50777000k -       5M      7M      - # user quota, on ALL fs ?!  may want to avoid
tinh                    user@/vol/vol2          10M     -       5M      7M      - # enforce user's quota w/in a specified volume
tinh         user@/vol/vol2/qtree1 100M    -       -    -       - # enforce user's quota w/in a specified qtree
            # exceptions for +/- space can be specified for given user/location


# 200111000k = 200 GB
#  50777000k =  50 GB
# they make output of quota report a bit easier to read

# * = default user/group/qtree 
# - = placeholder, no limit enforced, just enable stats collection

Snapshot

Snapshots are configured at the volume level. Thus, if different data need to have different snapshot characteristics, then they should be in different volume rather than just being in different QTree.
WAFL automatically reserve 20% for snapshot use.
snap list vol1
snap create vol1 snapname # manual snapshots creation.
snap sched   # print all snapshot schedules for all volumes
snap sched vol1 2 4   # scheduled snapshots for vol1: keep 2 weekly, 4 daily, 0 hourly snapshots
snap sched vol1 2 4 6  # same as above, but keep 6 hourly snapshots, 
snap sched vol1 2 4 6@9,16,20 # same as above, specifying which 3 hourly snapshot to keep + last 3 hours
snap reserve vol1     # display the percentage of space that is reserved for snapshot (def=20%)
snap reserve vol1 30  # set 30% of volume space for snapshot

vol options vol1 nosnap on # turn off snapshot, it is for whole volume!

gotchas, as per netapp:
"There is no way to tell how much space will be freed by deleting a particular snapshot or group of snapshots."

DeDup A/SIS

Advance Single Instance Storage (ie DeDuplication).
DeDuplication finds duplicate data and collapse them into a single unit. NetApp A/SIS works on the block-level (4KB), and operates in the background for individual FlexVol (not usable on Traditional Volume). Like snapshot that have inodes pointing to same block, SIS use the same tech to reduce storage need. "same" block are indexed by hash, and "sameness" is verified via a byte-by-byte comparison before re-org of the inode pointers to free space.

Performance impact:
  • File read just traverse thru a series of blocks in the i-node map. Random read is same. Sequential read may no longer be sequential, but large number of client request hardly makes read request really sequential anymore.
    Unlike EMC NS-series (as of Celerra v5.6), NetApp's dedup does not bundle together with compression, so there is no "re-hydration" time when accessing files (due to de-compression).
  • Write operations seems to take a real-time impact if SIS is turned on. Once SIS is on (and started), all write generate fingerprint on the fly and the info written to the change log. This calculation takes cpu power. Won't be impactful on system with less-than 50% load, but busy system can see degradation from 15% to 35% on FC disk.
    Page 6 of TR-3505:
    In real time, as additional data is written to the deduplicated volume, a fingerprint is created for each new block and written to a change log file. When deduplication is run subsequently, the change log is sorted and its sorted fingerprints are merged with those in the fingerprint file, and then the deduplication processing occurs.
    Note that there are really two change log files, so that as deduplication is running and merging the new blocks from one change log file into the fingerprint file, new data that is being written to the flexible volume is causing fingerprints for these new blocks to be written to the second change log file. The roles of the two files are then reversed the next time that deduplication is run.
    Page 15 of TR-3505:
    If the load on a system is low—that is, for systems in which the CPU utilization is around 50% or lower—there is a negligible difference in performance when writing data to a deduplicated volume, and there is no noticeable impact on other applications running on the system. On heavily used systems, however, where the system is nearly saturated with the amount of load on it, the impact on write performance can be expected to be around 15% for most NetApp systems. The performance impact is more noticeable on higher-end systems than on lower-end systems. On the FAS6080 system, this performance impact can be as much as 35%. The higher degradation is usually experienced in association with random writes. Note that these numbers are for FC drives; if ATA drives are used in a system, the performance impact would be greater.
  • Real dedup workload (finding duplicate block) can be scheduled to run at night or run on demand when sa knows filer is not busy.

    SIS won't operate on block marked by a snapshot, so saving maybe low when sis is turned on, till old snapshot expires. It is recommended to run sis before taking snapshot.
    
    sis on /vol/unixhome
    sis start -s /vol/unixhome # run scan for the first time (generate fingerprint)
    sis status   # show status and progress of scan if running
    df -s    # report on saving by dedup
    sis config    # see when sis is scheduled to run
    sis config -s auto /vol/home # use "auto" for when to rescan (when change amount is high)
        # recommend enable on all volume to reduce concurrent scan at mid-nite.
    
    sis off /vol/unixhome  # disable dedup.  stops fingerprint from being generated and written to change log
        # presumably with just this, write perf degradation should stops.
    sis undo /vol/unixhome  # recreate dedup block, delete fingerprint db when done.
        # use "priv set diag" to enter diag mode to run "undo".
    
    
    On a really busy FS but has slow cycles once in a while, perhaps dedup can result in no perf degradation yet save space:
    - sis on FlexVol
    - sis start -s FlexVol
    - sis off
    - (work)
    - sis start ...  (when system is idle)
    - sis off  (once scan is complete and busy working for user req again)
    
    Ref: TR-3050: NetApp Deduplication for FAS and V-Series Deployment and Implementation Guide

    NFS

    
    (/) etc/export
    is the file containing what is exported, and who can mount root fs as root.  Unix NFS related only.
    
    /vol/vol0        -access=sco-i:10.215.55.220,root=sco-i:10.215.55.220
    /vol/vol0/50gig  -access=alaska:siberia:root=alaska
    
    Unlike most Unices, NetApp allow export of ancestors and descendants.
    
    other options:
    -sec=sys # unix security, ie use uid/gid to define access
      # other options are kerberos-based.
    
    Besides just having export for nfs and ahare for cifs, 
    there is another setting about fs security permission style, nfs, ntfs, or mixed.  
    this control characteristic of chmod and files ACL.
    
    Once edit is done, telnet to netapp and issue cmd:
    exportfs -a  # re-add all exports as per new etc/export file
    exportfs -u  # unexport everything.  Careful!
    exportfs -u vol/vol1 # unexport vol1 (everything else remains intact)
    exportfs -r  # remove all exports that are no longer listed in etc/exports, maintain those that are still listed
       # -r is NOT the same as -au!
    
    The bug that Solaris and Linux NFS seems to exist on NetApp also.
    Hosts listed in exports sometime need to be given by IP address, or an
    explicit entry in the hosts file need to be setup.  Somehow, sometime
    the hostname does not get resolved thru DNS :(
    maybe it is a dns-cache poisoning problem...
    
    
    options nfs.per_client_stats.enable on
     # enable the collection of detained nfs stat per client 
    options nfs.v3.enable  on
    options nfs.tcp.enable on
     # enable NFS v3 and TCP for better performance.
    
    nfsstat  # display nfs sttistics, separte v2 and v3
    nfsstat -z  # zero the nfsstat counter
    nfsstat -h  # show detailed nfs statistics, several lines per client, since zero
    nfsstat -l  # show 1 line stat per client, since boot (non resetable stat)
    
    

    NIS domain

    changing NIS domain. no reboot should be necessary
    
    options nis.enable   off
    options nis.domain   new.nis.dom
    options nis.servers  10.0.91.44,10.0.91.82
    options nis.enable   on
    
    

    CIFS

    
    cifs disable  # turn off CIFS service
    cifs enable
    cifs setup  # configure domainname, wins.  only work when cifs is off.
    cifs testdc  # check registration w/ Windows Domain Controller
    
    
    
    cifs shares     # display info about all shares
    cifs shares -add sharename path -comment desc # create new share and give it some descriptive info
    cifs shares -change shrname -forcegroup grpname # specify that all cifs user will use a forced unix group on Unix-style FS.
          # this is for both read and write, so the mapping unix user need not be 
          # defined in this forcegroup in passwd or group map/file.
          # the groupname is a string, not gid number, this name need to be resolvable
          # from NIS, LDAP, or local group file.
    cifs shares -change shrname -umask 002     # define umask to be used.
    
    cifs access -delete wingrow  Everyone    
     # by default, share is accessible to "everyone" (who is connected to the domain)
     # above delete this default access
     # Note that this is equiv to exports, not file level ACL
    cifs access wingrow "authenticated users"  "Full Control" 
     # make share usable by authenticated users only
    cifs access it$ AD\Administrator "Full Control"    
     # make share "hidden" and only give access to admin  
     # (not sure if can use group "administrators")
    
    
    
    cifs sessions ...    # list current active cifs connections
    
    options cifs.wins_servers
     list what WINS server machine is using
    
    ifconfig  wins # enable  WINS on the given interface
    ifconfig  -wins # disable WINS on the given interface
    
    # WINS registration only happens when "cifs enable" is run.
    # re-registration means stopping and starting cifs service.
    # enabling or disabling wins on an interface will NOT cause re-registration
    
    etc/lslgroups.cfg  # list local group and membership SSID 
       # don't manually edit, use windows tool to update it!
    
    
    
    wcc  wafle cache control, oft use to check windows to unix mapping
     -u uid/uname uname may be a UNIX account name or a numeric UID
     -s sid/ntname  ntname may be an NT account name or a numeric SID
    
     SID has a long strings for domainname, then last 4-5 digits is the user.
     All computer in the same domain will use the domain SID.
    
     -x remove entries from WAFL cache
     -a add entrie
     -d display stats
    
    
    options wafl.default_nt_user username 
     # set what nt user will be mapped to unix by def (blank)
    options wafl.default_unix_user username
     # set what unix username will be used when mapped to NT (def = pcuser)
    
    
    user mapping b/w nt and unix, where user name are not the same.
    It is stored in the (/) etc/usermap.cfg file.
    
    NT acc   unix acc username
    Optionally, can have <= and => for single direction mapping instead of default both way.
    eg:
    
    tileg\Administrator      root
    tileg\fgutierrez         frankg
    tileg\vmaddipati         venkat
    tileg\thand              thand2
    tileg\thand              thand1
    tileg\kbhagavath         krishnan
    
    *\eric   => allen
    ad\administrator <= sunbox:root
    nt4dom\pcuser  <= tinh
    
    This mapping will be done so that users will gain full permission of the files under both env.
    a lot of time, they get nt account first, and thus end up with read only access to their 
    home dir in windows, as they are mapped as non owner.
    
    < !-- -- >
    usermap.cfg does get read by windows user writting to unix-style FS.
    Be careful when doing M-1 mapping.  While this may allow many unix user to use same NT account
    to gain access to NF-style FS as part of "everyone", the reverse access would be problematic.
    eg:
    hybridautoAD\tho sa
    hybridautoAD\tho tho
    While unix sa and tho maps to same user on windows, when Windows tho login, and try to write
    to UNIX-style FS, permission will assume that of unix user sa, will not be tho!!
    
    It maybe possible to use <== and ==> to indicate direction of mapping ??
    
    
    (??) another map does the reverse of windows mapping back to NFS when fs is NFS and access is from windows.
    (or was it the same file?).  It was pretty stupid in that it needed all users to be explicityly mapped.
    
    
    NetApp Web Interface control the share access (akin to exports)
    Windows Explorer file namager control each file ACL (akin to chmod on files).
    
    Can use Windows Manager to manage NetApp, general user can connect and browse.
    User list may not work too well.
    
    
    
    

    CIFS Commands

    
    cifs_setup   # configure CIFS, require CIFS service to be restarted 
        # - register computer to windows domain controller
        # - define WINS server
        
    options cifs.wins_server # display which WINS server machine is using
        # prior to OnTap 7.0.1, this is read only
    
    cifs domaininfo   # see DC info
    cifs testdc   # query DC to see if they are okay
    cifs prefdc print  # (display) which DC is used preferentially
        
    
    WINS info from NetApp, login req: http://now.netapp.com/Knowledgebase/solutionarea.asp?id=3.0.4321463.2683610
    # etc/cifsconfig_setup.cfg 
    # generated by cifs_setup, command is used to start up CIFS at boot
    # eg:
    cifs setup -w 192.168.20.2  -w 192.168.30.2 -security unix  -cp 437
    
    # usermap.cfg
    
    # one way mapping
    *\lys => lks
    NETAPP\administrator <= unixhost:root
    
    # two way mapping
    WINDOM\tinh tin
    
    ## these below are usually default, but sometime need to be explicitly set
    ## by some old NT DC config.
    WINDOM\* == * # map all user of a specific domain
    # *\*    == *   # map all user in all domains  
    

    Command

    Commands for NetApp CLI (logged in thru telnet/ssh/rsh)
    
    ? = help, cmd list
    help cmd 
    
    
    dns info  # display DNS domain, 
       # extracted from WINDOWS if not defined in resolve.conf
    options dns.domainname  # some /etc/rc script set domain here
    
    sysconfig -v 
    sysconfig -a # display netapp hw system info, include serial number and product model number
    sysconfig -c # check to ensure that there are no hardware misconfig problem, auto chk at boot
    
    sysstat 1  # show stats on the server, refresh every 1 sec.
    
    df -h
     similar to unix df, -h for "human readable"
     .snapshot should be subset of the actual volume 
    
    df -s report sis/dedup saving on a volume
    
    ndmpd status
     list active sessions
    
    ndmpd killall
     terminate all active ntmpd sessions.
     Needed sometime when backup software is hung.  kill ndmpd session to free it.
    
    useradmin useradd UID
     add new user (to telnet in for admin work)
    
    useradmin userlist
     list all users
    
    
    options   # list run time options.
    options KEY VALUE  # set specific options
    
    #eg, autosupport with email:
    options autosupport.mailhost  mailhost.myco.com,mailhost2.myco.com
     # comma list of up to 5 host (tried till one work?)
    options autosupport.support.transport smtp
    options autosupport.support.to autosupport@netapp.com
    options autosupport.to tin.ho@e-ville.com,bofh@e-ville.com
     # Change who receives notification emails.
    options autosupport.doit case_number_or_name
     # Generate an autosupport email to NetApp (to predefined users).
    
    # autosupport via web  (but then local admin don't get emaiL?)
    options autosupport.support.transport https
    options autosupport.support.proxy     na-useh-proxy:2010
    
    
    #find out about ntp config:
    cat registry| grep timed
    options.cf.timed.max_skew=
    options.service.cf.timed.enable=off
    options.service.timed.enable=on
    options.timed.log=off
    options.timed.max_skew=30m
    options.timed.min_skew=10
    options.timed.proto=ntp
    options.timed.sched=hourly
    options.timed.servers=time-server-name   # time server to use
    options.timed.window=0s
    state.timed.cycles_per_msec=2384372
    state.timed.extra_microseconds=-54
    state.timed.version=1
    
    rdfile   read data file (raw format)
      eg rdfile /etc/exports
      inside telnet channel, will read the root etc/exports file to std out.
      equiv to unix cat
    
    wrfile  write stdin to file  
      not edit, more like cat - > file kind of thing.
    
    
    FilerView
    FilerView is the Web GUI. If SSL certificate is broken, then it may load up a blank page.
    secureadmin status
    secureadmin disable ssl
    secureadmin setup -f ssl # follow prompt to setup new ssl cert
    
    SSH
    To allow root login to netapp w/o password, add root's id_dsa.pub to vol1/etc/sshd/root/.ssh/authorized_keys
    Beware of the security implications!

    Config Files

    
    all stored in etc folder.
    resolve.conf 
    nsswitch.conf
    
    # etc/exports
    
    /vol/unix02  -rw=192.168.1.0/24:172.27.1.5:www,root=www
    /vol/unix02/dir1 -rw=10.10.10.0/8
    
    # can export subdirs with separate permissions
    # issue exportfs -a to reread file
    
    

    Logs

    (/) etc/messages.* unix syslog style logs.  can configure to use remote syslog host.
    
    (/) etc/log/auditlog
     log all filer level command.  Not changes on done on the FS.
    
    
    
    The "root" of vol 0,1,etc in the netapp can be choose as the netapp root and store the /etc directory, 
    where all the config files are saved.  eg.
    /mnt/nar_200_vol0/etc
    /mnt/na4_vol1/etc
    
    other command that need to be issued is to be done via telnet/rsh/ssh to the netapp box.
    
    
    < ! - - - - >

    Howto

    Create new vol, qtree, and make access for CIFS
    vol create win01 ...
    
    qtree create   /vol/win01/wingrow
    qtree security /vol/win01/wingrow ntfs
    qtree oplocks  /vol/win01/wingrow enable
    cifs shares -add wingrow /vol/win01/wingrow -comment "Windows share growing"
    #-cifs access wingrow ad\tinh "Full Control"  # share level control is usually redundant
    cifs access -delete wingrow  Everyone
    cifs access wingrow "authenticated users"  "Full Control"
    
    # still need to go to the folder and set file/folder permission,
    # added corresponding department (MMC share, permission, type in am\Dept-S
    # the alt+k to complete list (ie, checK names).
    # also remove inherit from parent, so took out full access to everyone.
    
    

    Network Interface Config

    vif = virtual interface, eg use: create etherchannel
    
    link agregation (netapp typically calls it trunking, cisco EtherChannel).
    
    single mode  = HA fail over, only single link active at the same time.
    multi mode = Performance, multiple link active at the same time.  Req swich support
        Only good when multiple host access filer.  Switch do the
        traffic direction (per host).
    
    Many filer comes with 4 build in ethernet port, can do:
    
    2 pair of multi mode (e0a+e0b, e0c+e0d).
    then single mode on the above pair to get HA, filer will always have 2 link
    active at the same time.
    
    
    pktt start all -d /etc  # packets tracing, like tcpdump
    pktt stop all
    # trace all itnerfaces, put them in /etc dir, 
    # one file per interface.
    # files can be read by ethereal/wireshark
    
    
    

    Backup and Restore, Disaster Recovery


    NetApp supports dump/restore commands, a la Solaris format. Thus, the archive created can even be read by Solaris ufsrestore command.
    NetApp championed NDMP, and it is fast. But it backup whole volume as a unit, and restore has to be done as a whole unit. This may not be convinient.
    volcopy is fast, it will also copy all the snapshots associated with the volume.

    DFM

    Data Fabric Manager, now known as ...
    typically https://dfm:443/
    Ment to manage multiple filer in one place, but seems to just collect stats. Kinda slow at time. And to get volume config, still have to use FilerView, so not one-stop thing. ==> limited use.

    Links

    1. RAID_DP

    History

    
      - ca 2000 = Alpha Chip
    OnTap 5.3 - 
    OnTap 6.1  - 2003?  Intel based ? 
    OnTap 6.5 - 2004?  RAID_DP        Dual Paritiy introduced here.
    OnTap 7.0 - 2005?  Aggregate introduced here.
    OnTap 7.3.1 - 2008?  DeDuplication (a/sis) single instance storage available. 
    
    


    [Doc URL: http://www.grumpyxmas.com/netapp.html]
    [Doc URL: http://sn50.user.sonic.net/psg/netapp.html]
    [Doc URL: http://www.cs.fiu.edu/~tho01/psg/netapp.html]

    (cc) Tin Ho. See main page for copyright info.
    Last Updated: 2007-04-27, 2009-04-01


    "LYS on the outside, LKS in the inside"
    "AUHAUH on the outside, LAPPLAPP in the inside"
    psg101 sn50 tin6150
  • EMC Celerra

    Pocket Survival Guide - EMC Celerra
    <-- Please click if you found this site useful ;-)

    EMC Celerra

    EMC Celerra 101

    Celerra is the NAS offering from EMC.
    
    Control station is the management station where all admin commands are issued:
    
    https://celerra-cs0.myco.com/  # web gui URL.  Most feature avail there, including a console.
    
    ssh celerra-cs0.myco.com # ssh (or rsh, telnet in) for CLI access
    
    
    Layers:
      
    VDM (vdm2) / DM (server_2)  
      |
    Export/Share
      |
    Mount
      |
    File System
      |
    (AVM stripe, volume, etc)
      |
    storage pool (nas_pool)
      |
    disk
    
    Export can export subdirectory within a File System.
    All FS are native Unix FS.  CIFS features are added thru Samba (and other EMC add ons?).
    
    CIFS share recommended thru VDM, for easier migration, etc.  
    NFS share thru normal DM (server_X).  Physical DM can mount/export FS already shared by VDM, 
    but VDM can't access the "parent" export done by a DM.
    VDM mounts are accessible by underlaying DM via /root_vdm_N
    
    Quota can be on tree (directory), per user a/o group.
    
    
    
    Commands are to be issued thru the "control station" (ssh) 
    (or web gui (Celerra Manager) or Windows MMC SnapIn (Celerra Management).)
    
    Most commands are the form:
    server_...
    nas_...
    fs_...
    /nas/sbin/...
    
    typical options can be abreviated, albeit not listed in command usage:
    -l = -list
    -c = -create
    -n = -name
    -P = -Protocol
    
    
    nas_halt # orderly shutdown of the whole NS80 integrated.  
      # issue command from control station.
    
    

    IMHO Admin Notes

    Celerra sucks as it compares to the NetApp. If you have to manage one of these suckers, I am sorry for you (I am very sorry for myself too). I am so ready to convert my NS-80 integrated into a CX380 and chuck all the Data Mover that create the NAS head. There are lot of catchas. More often than not, it will bite you in your ass. Just be very careful, and know that when you need to most to change some option, count on it needing a reboot! The "I am sorry" quote came from a storage architect. One of my former boss used to be a big advocate of EMC Celerra but after having to plan multiple outage to fix things (which NetApp wouldn't have to), he became a Celerra hater. Comments apply to DART 5.5 and 5.6 (circa 2008, 2009)
    1. Windows files are stored as NFS, plus some hacking side addition for meta data.
      This mean from the getgo, need to decide how to store the userid and gid. UserMapper is a very different beast than the usermap.cfg used in NetApp.
    2. Quota is nightmare. Policy change is impossible. Turning it off require removing all files on the path.
    3. Web GUI is heavy Java, slow and clunky. And if you have the wrong java on your laptop, well, good luck!
    4. CLI is very unforgiven in specification of parameters and sequences.
    5. The nas_pool command shows how much space is available, but give no hints of virtual provisioning limit (NetApp may have the same problem though)
    Some good stuff, but only marginally:
    1. CheckPoint is more powerful than NetApp's Snapshot, but it requires a bit more setup. Arguably it does not hog up mainstream production file system space due to snapshot, and they can be deleted individually, so it is worth all the extra work it brings. :-)

    Sample Setup

    Below is a sample config for a brand new setup from scratch. The general flow is:
    1. Setup network connectivity, EtherChannel, etc
    2. Define Active/Standby server config
    3. Define basic network servers such as DNS, NIS, NTP
    4. Create Virtual CIFS server, join them to Windows Domain
    5. Create a storage pool for use with AVM
    6. Create file systems
    7. Mount file systems on DM/VDM, export/share them
    # Network configurations
    server_sysconfig server_2 -pci cge0 -o "speed=auto,duplex=auto"
    server_sysconfig server_2 -pci cge1 -o "speed=auto,duplex=auto"
    
    # Cisco EtherChannel (PortChannel)
    server_sysconfig server_2 -virtual -name TRK0 -create trk -option "device=cge0,cge1"
    server_sysconfig server_3 -virtual -name TRK0 -create trk -option "device=cge0,cge1"
    server_ifconfig  server_2 -c -D TRK0 -n TRK0 -p IP 10.10.91.107 255.255.255.0 10.10.91.255
    # ip, netmask, broadcast
    
    # Create default routes
    server_route server_2 -add default 10.10.91.1
    
    # Configure standby server
    server_standby server_2  -create mover=server_5 -policy auto
    
    # DNS, NIS, NTP setup
    server_dns  server_2 oak.net  10.10.91.47,162.86.50.204
    server_nis  server_2 oak.net  10.10.89.19,10.10.28.145
    server_date server_2 timesvc start ntp 10.10.91.10
    
     
    server_cifs ALL -add security=NT
    
    # Start CIFS services
    server_setup server_2 -P cifs -o start
    
    #Create Primary VDMs and VDM file system in one step.
    nas_server -name VDM2 -type vdm -create server_2 -setstate loaded
    
    #Define the CIFS environment on the VDM
    server_cifs VDM2 -add compname=winsvrname,domain=oak.net,interface=TRK0,wins=162.86.25.243:162.86.25.114
    server_cifs VDM2 -J compname=vdm2,domain=oak.net,admin=hotin,ou="ou=Computers:ou=EMC Celerra" -option reuse
    # ou is default location where object will be added to AD tree (read bottomm to top)
    # reuse option allows AD domain admin to pre-create computer account in AD, then join it from a reg user (pre-granted)
    # the ou definition is quite important, it need to be specified even when 
    # "reusing" an object, and the admin account used much be able to write to
    # that part of the AD tree defined by the ou.
    # EMC seems to need the OU to be defined in reverse order, 
    # from the bottom of the LDAP tree, separated by colon, working upward.
    # When in doubt, use the full domain account priviledges.
    
    # <!--
    
     
    
    
    
    
    # option to reset password if account password has changed but want to use same credential/object again...   resetserverpasswd
    
    
    other troubleshooting commands:
    ... server_kerberos -keytab ...
    server_cifssupport VDM2 -cred -name WinUsername -domain winDom   # test domain user credentials
    
    
    
    
    # Confirm d7 and d8 are the smaller LUNs on RG0
    nas_pool -create -name clar_r5_unused -description "RG0 LUNs" -volumes d7,d8
    
     
    # FS creation using AVM (Automatic Volume Management), which use pre-defined pools:
    # archive pool = ata drives
    # performance pool = fc drives
    
    nas_fs -name cifs1  -create size=80G pool=clar_archive
    server_mountpoint VDM2   -c  /cifs1  # mkdir 
    server_mount      VDM2 cifs1 /cifs1   # mount (fs given a name instead of traditional dev path) 
    server_export     VDM2 -name cifs1 /cifs1   # share, on VDM, automatically CIFS protocol
    ## Mount by VDM is accessible from a physical DM as /root_vdm_N (but N is not an obvious number)
    ## If FS export by NFS first, using DM /mountPoint as path, 
    ## then VDM won't be able to access that FS, and CIFS sharing would be limited to actual physical server
    
    nas_fs -name nfshome            -create size=20G pool=clar_r5_performance
    server_mountpoint server_4       -c  /nfshome
    server_mount      server_4   nfshome /nfshome
    server_export     server_4 -Protocol nfs -option root=10.10.91.44 /nfshome
    
    nas_fs -name MixedModeFS -create size=10G pool=clar_r5_performance
    server_mountpoint VDM4               -c  /MixedModeFS
    server_mount      VDM4       MixedModeFS /MixedModeFS
    server_export     VDM4 -name MixedModeFS /MixedModeFS
    server_export server_2 -Protocol nfs -option root=10.10.91.44 /root_vdm_6/MixedModeFS
    ##  Due to VDM sharing the FS, the mount path used by Physical DM (NFS) need to account for the /root_vdm_X prefix
    
    
    
    See additional notes in Config Approach below.

    Config Approach

  • Make decision whether to use USERMAPPER (okay in CIFS only world, but if there is any UNIX, most likely NO).
  • Decide on Quotas policy
  • Plan for Snapshots...
  • An IP address can be used by 1 NFS server and 1 CIFS server. server_ifconfig -D cge0 -n cge0-1 can be done for the DM; cge0-1 can still be the interface for CIFS in VDM. Alternatively, the DM can have other IP (eg cge0-2) if it is desired to match the IP/hostname of other CIFS/VDM.
  • Export FS thru VDM first, then NFS export use the /root_vdm_N/mountPoint path.

    Use VDM instead of DM (server_2) for CIFS server. A VDM is really just a file system. Thus, it can be copied/replicated. Because windows group and many other system data is not stored at the underlaying Unix FS, there was a need to easily backup/migrate CIFS server.
    For multi-protocol, it is best to have 1 VDM to provide CIFS access, and NFS will ride on the Physical DM.
    CAVA complication: The Antivirus scanning feature must be connected to a physical CIFS server, not to a VDM. This is because it is 1 CAVA for the whole DM, not multiple instance for multiple VDM that may exist on a DM. Global CIFS share is also required. May still want to just use physical DM with limited windows user/group config, as that may not readily migrate or backup.
    Overall, still think that there is a need of 2 IP per DM. Maybe VDM and NFS DM have same IP so that it can have same hostname. But the Global CIFS share will ride on a Physical DM with a separate IP that user don't need to know. Finally, perhaps scrap the idea of VDM, but then one may pay dearly in replication/backup...

    Celerra Howto

    Create a Server

    * Create a NFS server 
     - Really just ensuring a DM (eg server_2) is acting as primary, and
     - Create logical Network interface (server_ifconfig -c -n cge0-1 ...)
       (DM always exist, but if it is doing CIFS thru VDM only, then it has no IP and thus can't do NFS export).
    
    * Create Physical CIFS sesrver (server_setup server_2 -P cifs ...)  
        OR
      VDM to host CIFS server (nas_server -name VDM2 -type vdm -create server_2 -setstate loaded)
        + Start CIFS service (server_setup server_2 -P cifs -o start)
        + Join CIFS server to domain (server_cifs VDM2 -J ...)
    

    Create FS and Share

    1. Find space to host the FS (nas_pool for AVM, nas_disk for masoquistic MVM)
    2. Create the FS (nas_fs -n FSNAME -c ...)
    3. Mount FS in VDM, then DM (server_mountpoint -c, server_mount)
    4. Share it on windows via VDM (server_export -P cifs VDM2 -n FSNAME /FsMount)
    5. Export the share "via the vdm path" (server_export -o root=... /root_vdm_N/FsMount)
    Note that for server creation, DM for NFS is created first, then VDM for CIFS.
    But for FS sharing, it is first mounted/shared on VDM (CIFS), then DM (NFS).
    This is because VDM mount will dictate the path used by the DM as /root_vdm_N.
    It is kinda backward, almost like lower level DM need to go thru the higher level VDM, blame in on how the FS mount path ended up...

    File System, Mounts, Exports

    
    nas_fs -n FSNAME -create size=800G pool=clar_r5_performance # create fs
    nas_fs -d FSNAME      # delete fs
        
    nas_fs size FSNAME  # determine size
    nas_fs -list   # list all FS, including private root_* fs used by DM and VDM
    
    server_mount server_2  # show mounted FS for DM2
    server_mount VDM1  # show mounted FS for VDM1
    server_mount ALL  # show mounted FS on all servers
    
    server_mountpoint VDM1    -c  /FSName # create mountpoint (really mkdir on VDM1)
    server_mount      VDM1 FSNAME /FSName # mount the named FS at the defined mount point/path.
         # FSNAME is name of the file system, traditionally a disk/device in Unix
         # /FSName is the mount point, can be different than the name of the FS.
    
    server_mount server_2 -o accesspolicy=UNIX FSNAME /FSName
    # Other Access Policy (training book ch11-p15)
    # NT     (both unix and windows access check NTFS ACL)
    # UNIX   (both unix and windows access check NFS permission bits)
    # NATIVE (default, unix and nt perm kept independent, 
              careful with security implication!
       Ownership is only maintained once, Take Ownership in windows will 
       change file UID as viewed from Unix.)
    # SECURE (check ACL on both Unix and Win before granting access)
    # MIXED - Both NFS and CIFS client rights checked against ACL; Only a single set of security attributes maintained 
    # MIXED_COMPAT - MIXED with compatible features 
     
    NetApp Mixed Mode is like EMC Native.  Any sort of mixed mode is likely asking for problem.  
    Stict to either only NT or only Unix is the best bet.
    
    
    server_export ALL  # show all NFS export and CIFS share, vdm* and server_*
        # this is really like "looking at /etc/exports" and 
        # does not indicate actual live exports.
        # if FS is unmountable when DM booted up, server_export would 
        # still show the export even when it can't possibly exporting it
        # The entries are stored, so after FS is online, can just export w/ FS name, 
        # all other params will be looked up from "/etc/exports" 
    server_export server_4 -all # equivalent to "exportfs -all" on server_4.  
        # no way to do so for all DM at the same time.
    server_export VDM1 -name FSNAME /FSName
    server_export server_2 -Protocol nfs -option root=10.10.91.44 /root_vdm_6/FSName
    ##  Due to VDM sharing the FS, the mount path used by Physical DM (NFS) need to account for the /root_vdm_X prefix
    
    
    (1)  server_export server_4 -Protocol nfs -option root=host1:host2,rw=host1,host2 /myvol
    (2)  server_export server_4 -Protocol nfs -option rw=host3 /myvol
    (3)  server_export server_4 -Protocol nfs -option anon=0   /myvol
    
    # (1) export myvol as rw to host1 and host2, giving them root access.
    # subsequently add a new host to rw list.  
    # Celerra just append this whole "rw=host3" thing in there, so that the list end up having multiple rw= list.  
    # Hopefully Celerra add them all up together.
    # (2) Alternatively, unexport and reexport with the updated final list.
    # (3) The last export add mapping of anonymous user to map to 0 (root).  not recommended, but some crazy app need it some time.
    # there doesn't seems to be any root squash.  root= list is machine that is granted root access
    # all other are squashed?  
    
    
    WARNING
    The access= clause on Celerra is likely what one need to use in place of the traditional rw= list.
    ## root=host1:host2,
    ## rw=host1:host2:hostN,
    ## access=host1:host2:hostN
    
    ## Celerra require access to be assigned, which effectively limit which host can mount.
    ## the read/write list is not effective (I don't know what it is really good for)
    ## access= (open to all by default), and any host that can mount can write to the FS, 
    ## even those not listed in rw=...  
    ## (file system level NFS ACL still control who have write, but UID in NFS can easily be faked by client)
    ## In summary: for IP-based access limitation to Celerra, access= is needed.
    ## (can probably omit rw=)
    ## rw= is the correct settings as per man page on the control station.
    ## The PDF paints a different pictures though.  
    
    # NFS share is default if not specified
    # On VDM, export is only for CIFS protocol
    # NFS exports are stored in some file, 
    
    
    
    
    

    unshare/unmount

    
    server_export VDM1 -name ShareName\$  # share name with $ sign at end for hidden need to be escaped 
    server_export VDM1 -unexport -p -name ShareName # -p for permanent (-unexport = -u)
    server_umount VDM1 -p /FSName   # -p = permanent, if omitted, mount point remains
           # (marked with "unmounted" when listed by server_mount ALL)
          # FS can't be mounted elsewhere, server cannot be deleted, etc!
          # it really is rmdir on VDM1
    
    

    Advance FS cmd

    
    nas_fs -xtend FSNAME size=10G  ## ie ADD 10G to existing FS
      # extend/enlarge existing file system.
      # size is the NET NEW ADDITION tagged on to an existing FS,
      # and NOT the final size of the fs that is desired.
      # (more intuitive if use the +10G nomenclature, but it is EMC after all :-/
    
    nas_fs -modify FSNAME -auto_extend yes -vp yes -max_size 1T
      # modify FSNAM 
      # -auto_extend = enlarge automatically.  DEF=no
      # -vp yes  = use virtual provisioning
          If no, user see actual size of FS, but it can still grow on demand.
      # -max_size  = when FS will stop growing automatically, specify in G, T, etc.  
          Defualt to 16T, which is largest FS supported by DART 5.5
    
      # -hwm = high water mark in %, when FS will auto enlarge
         Default is 90
    
    
    nas_fs -n FSNAME -create size=100G pool=clarata_archive -auto_extend yes -max_size 1000G -vp yes
      # create a new File System
      # start with 100 GB, auto growth to 1 TB
      # use virtual provisioning, 
      # so nfs client df will report 1 TB when in fact FS could be smaller.
      # server_df will report actual size
      # nas_fs -info -size FSNAME will report current and max allowed size 
      #  (but need to dig thru the text)
    
    
    

    Server DM, VDM

    nas_server -list  # list physical server (Data Mover, DM)
    nas_server -list -all  # include Virtual Data Mover (VDM)
    server_sysconfig server_2 -pci
    
    nas_server -info server_2
    nas_server -v -l    # list vdm
    
    
    nas_server -v vdm1 -move server_3    # move vdm1 to DM3
      # disruptive, IP changed to the logica IP on destination server
      # logical interface (cge0-1) need to exist on desitnation server (with diff IP)
      # 
    
    
    server_setup server_3 -P cifs -o start  # create CIFS server on DM3, start it
          # req DM3 to be active, not standby (type 4)
    
    
    
    server_cifs  serve_2  -U compname=vdm2,domain=oak.net,admin=administrator # unjoin CIFS server from domain
    server_setup server_2 -P cifs -o delete  # delete the cifs server
    
    nas_server -d vdm1    # delete vdm (and all the CIFS server and user/group info contained in it)
    
    
    

    Storage Pool, Volume, Disk, Size

    AVM = Automatic Volume Management
    MVM = Manual Volume Management
    MVM is very tedious, and require lot of understanding of underlaying infrastructure and disk striping and concatenation. If not done properly, can create performance imbalance and degradation. Not really worth the headache. Use AVM, and all FS creation can be done via nas_fs pool=...

    
    
    nas_pool -size -all # find size of space of all hd managed by AVM
     potential_mb  = space that is avail on the raid group but not allocated to the pool yet??
     
    nas_pool -info -all # find which FS is defined on the storage pool
    
    
    server_df  # df, only reports in kb 
    server_df ALL  # list all *MOUNTED* FS and check points sizes
       # size is actual size of FS, NOT virtual provisioned size
       # (nfs client will see the virtual provisioned size)
    
    server_df  ALL | egrep -v ckpt\|root_vdm # get rid of duplicates due to VDM/server_x mount for CIFS+NFS access
    
    nas_fs -info size -all  # give size of fs, but long output rather than table format, hard to use.
    
    nas_fs -info -size -all | egrep name\|auto_ext\|size
       # somewhat usable space and virtual provisioning info
       # but too many "junk" fs like root_fs, ckpt, etc
    
    
    nas_volume -list # list disk volume, seldom used if using AVM.
    nas_disk -l
    
    
    /nas/sbin/rootnas_fs -info root_fs_vdm_vdm1 | grep _server  # find which DM host a VDM
    
    
    

    UserMapper

    Usermapper in EMC is substantially different than in the NetApp. RTFM!

    It is a program that generate UID for new windows user that it has never seen before. Files are stored in Unix style by the DM, thus SID need to have a translation DB. Usermapper provides this. A single Usermapper is used for the entire cabinet (server_2, _3, _4, VDM2, VDM3, etc) to provide consistency. If you are a Windows-ONLY shop, with only 1 Celerra, this maybe okay. But if there is any Unix, this is likely going to be a bad solution.
    If user get Unix UID, then the same user accessing files on windows or Unix is viewed as two different user, as UID from NIS will be different than UID created by usermapper!

    UID lookup sequence:
    1. SecMap Persistent Cache
    2. Global Data Mover SID Cache (seldom pose any problem)
    3. local passwd/group file
    4. NIS
    5. Active Directory Mapping Utility (schema extension to AD for EMC use)
    6. UserMapper database
    When a windows user hit the system (even for read access), Celerra need to find a UID for the user. Technically, it consults NIS and/or local passwd file first, failing that, it will dig in UserMapper. Failing that, it will generate a new UID as per UserMapper config.
    Howwever, to speed queries, a "cache" is used first all the time. The cache is called SecMap. However, it is really a binary database, and it is persisten across reboot. Thus, once a user has hit the Celerra, it will have an entry in the SecMap. There is no time out or reboot that will rid the user from SecMap. Any changes to NIS and/or UserMapper won't be effective until the SecMap entry is manually deleted.
    Overall, EMC admit this too, UserMapper should not be used in heterogeneous Windows/Unix environment. If UID cannot be guaranteed from NIS (or LDAP) then 3rd party tool from Centrify should be considered.
    
    server_usermapper server_2 -enable # enable usermapper service
    server_usermapper server_2 -disable
    # even with usermapper disabled, and passwd file in /.etc/passwd
    # somehow windows user file creation get some strange GID of 32770 (albeit UID is fine).
    # There is a /.etc/gid_map file, but it is not a text file, not sure what is in it.
    
    server_usermapper server_2 -Export -u passwd.txt # dump out usermapper db info for USER, storing it in .txt file
    server_usermapper server_2 -E      -g group.txt  # dump out usermapper db info for GROUP, storing it in file 
    
    # usermapper database should be back up periodically!
    server_usermapper server_2 -remove -all  # remove usermapper database
          # Careful, file owner will change in subsequent access!!
    
    There is no way to "edit" a single user, say to modify its UID.
    Only choice is to Export the database, edit that file, then re-Import it back.
    # as of Celerra version 5.5.32-4 (2008.06)
    
    
    When multiple Celerra exist, UserMapper should be synchronized (one become primary, rest secondary). server_usermapper ALL -enable primary=IP. Note that even when sync is setup, no entry will be populated on secondary until a user hit the Celerra with request. Ditto for the SecMap "cache" DB.
    
    p28 of configuring Celerra User Mapping PDF:
    
    Once you have NIS configured, the Data Mover automatically checks NIS for a user
    and group name. By default, it checks for a username in the form username.domain
    and a group name in the form groupname.domain. If you have added usernames
    and groupnames to NIS without a domain association, you can set the cifs resolver
    parameter so the Data Mover looks for the names without appending the domain.
    
    server_param server_2  -facility cifs -info resolver
    server_param server_2  -facility cifs -modify resolver -value 1
    repeat to all DM, but not applicable to VDM
    
    Setting the above will allow CIFS username lookup from NIS to match based on username, 
    without the .domain suffix.  Use it!  (Haven't seen a situation where this is bad)
    
    
    server_param server_2 -f cifs -m acl.useUnixGid -v 1
    
    Repeat for for all DM, but not for VDM.
    This setting affect only files created on windows.  UID is mapped by usermapper.
    GID of the file will by default map to whatever GID that Domain User maps to.
    Setting this setting, unix primary group of the user is looked up and used as 
    the GID of any files created from windows.
    Windows group permission settings retains whatever config is on windows 
    (eg inherit from parent folder).
    
    
    
    SecMap
    Unlike UserMapper, which is human readable database (and authority db) which exist one per NS80 cabinet (or sync b/w multiple cabinet), the SecMap database exist one per CIFS server (whether it is physcial DM or VDM).
    
    server_cifssupport VDM2 -secmap -list    # list SecMap entries 
    server_cifssupport ALL -secmap -list    # list SecMap entries on all svr, DM and VDM included.
    server_cifssupport VDM2 -secmap -delete -sid S-1-5-15-47af2515-307cfd67-28a68b82-4aa3e
    server_cifssupport ALL  -secmap -delete -sid S-1-5-15-47af2515-307cfd67-28a68b82-4aa3e
     # remove entry of a given SID (user) from the cache
     # delete would need to do for each CIFS server.  
     # Hopefully, this will trick EMC to query NIS for the UID instead of using one from UserMapper.
    
    server_cifssupport VDM2 -secmap -create -name USERNAME -domain AD-DOM
     # for 2nd usermapper, fetch the entry of the given user from primary usermapper db.
          
    
    
    
    
    
    

    General Command

    nas_version   # version of Celerra
        # older version only combatible with older JRE (eg 1.4.2 on 5.5.27 or older)
    
    server_log server_2  # read log file of server_2
    
    

    Config Files

    A number of files are stored in etc folder. retrieve/post using server_file server_2 -get/-put ...
    eg: server_file server_3 -get passwd ./server_3.passwd.txt would retrieve the passwd file local to that data mover.
    Each File System have a /.etc dir. It is best practice to create a subdirectory (QTree) below the root of the FS and then export this dir instead.


    On the control station, there are config files stored in:
  • /nas/server
  • /nas/site
    Server parameters (most of which require reboot to take effect), are stored in:
  • /nas/site/slot_param for the whole cabinet (all server_* and vdm)
  • /nas/server/slot_X/param (for each DM X)
    
    
    
    Celera Management
    Windows MMC Plug in thing...
    
    
    

    CheckPoint

    Snapshots are known as CheckPoint in EMC speak.
    Requires a SaveVol to keep the "copy on write" date. It is created automatically when first checkpoint is created, and by default grows automatically (at 90% high water mark). But it cannot be strunk. When the last checkpoint is deleted, the SaveVol is removed.
    GUI is the only sane way to edit it. Has abilities to create automated schedules for hourly, daily, weekly, monthly checkpoints.

    
    
    
    

    Backup and Restore, Disaster Recovery

    For NDMP backup, each Data Mover should be fiber connected to a tape drive (dedicated). Once zoning is in place, need to tell data mover to scan for the tapes.

    Quotas

    Change to use Filesize policy during initial setup as windows does not support block policy (which is Celerra default).
    Edit the /nas/site/slot_param on the control station (what happen to standby control station?) add the following entry:
    param quota policy=filesize
    
    Since this is a param change, retarded EMC requies a reboot:
    server_cpu server_2 -r now
    Repeat for additional DM that may exist on the same cabinet.


    ----
    Two "flavor" of quotas: Tree Quota, and User/Group quota. Both are per FS.
    Tree Quoata requires creating directory (like NetApp qtree, but at any level in the FS). There is no turning off tree quota, it can only be removed when all files in the tree is deleted.
    User/Group quota can be created per FS. Enableling require freezing of the FS for it to catalog/count the file size before it is available again! Disabling the quota has the same effect.
    User/Group quota default have 0 limit, which is monitoring only, but does not actually have hard quota or enforce anything.

    ----
    Each File System still need to have quota enabled... (?) Default behaviour is to deny when quota is exceeded. This "Deny Disk Space" can be changed (on the fly w/o reboot?)
  • GUI: File System Quotas, Settings.
  • CLI: nas_quotas -user -edit config -fs FSNAME ++ repeat for Tree Quota ?? But by default, quota limit is set to 0, which is to say it is only doing tracking, so may not need to change behaviour to allow.

    Celerra manager is easiest to use. GUI allows showing all QTree for all FS, but CLI don't have this capability. Sucks eh? :(

    EMC recommends turning on FileSystem quota whenever FS is created. But nas_quotas -on -tree ... -path / is denied, how to do this??!!
    # Create Tree Quota (NA QTree).  Should do this for each of the subdir in the FS that is directly exported.
    nas_quotas -on  -tree -fs CompChemHome -path /qtree # create qtree on a fs
    nas_quotas -off -tree -fs CompChemHome -path /qtree # destroy qtree on a fs (path has to be empty)
     # can remove qtree by removing dir on the FS from Unix host, seems to works fine.
    nas_quotas -report -tree -fs CompChemHome  # display qtree quota usage
    
    
    # per user quota, not too important other than Home dir... 
    # (and only if user home dir is not a qtree, useful in /home/grp/username FS tree)
    nas_quotas -on -user -fs CompChemHome   # track user usage on whole FS
           # def limit is 0 = tracking only
    nas_quotas -report -user -fs CompChemHome  # display users space usage on whole FS
    
    
    
    

    From Lab Exercise

    
    nas_quotas -user  -on -fs    FSNAME # enable user quota on FsNAMe.  Disruptive. (ch12, p22)   
    nas_quotas -group -on -mover server_2 # enable group quota on whole DM .  Disruptive.
    
    nas_quotas -both -off -mover server_2 # disable both group and user quota at the same time.
    
    ++ disruption...  ??? really?  just slow down?  or FS really unavailable?? ch 12, p22.
    
    nas_quotas -report -user -fs FSNAME 
    nas_quotas -report -user -mover server_2
    
    
    nas_quotas -edit -config -fs FsNAME  # Define default quota for a FS.
    
    
    nas_quotas -list -tree -fs FSNAME # list quota tree on the spefified FS.
     
    nas_quotas -edit -user -fs FSNAME user1 user2 ... # edit quota (vi interface)
    
    nas_quotas -user -edit -fs FSNAME -block 104 -inode 100 user1 # no vi!
    
    nas_quotas -u -e mover server_2 501 # user quota, edit, for uid 501, whole DM
    
    nas_quota -g -e -fs FSNAME 10  # group quota, edit, for gid 10, on a FS only.
    
    nas_quotas -user -clear -fs FSNAME # clear quota: reset to 0, turn quota off.
    
    

    Tree Quota

    
    nas_quotas -on -fs FSNAME -path /tree1  # create qtree on FS                (for user???) ++
    nas_quotas -on -fs FSNAME -path /subdir/tree2 # qtree can be a lower level dir
    
    nas_quotas -off -fs FSNAME -path /tree1  # disable user quota (why user?)
          # does it req dir to be empty??
    nas_quotas -e -fs FSNAME -path /tree1 user_id # -e,  -edit user quota
    nas_quotas -r -fs FSNAME -path /tree1  # -r = -report
    
    
    nas_quotas -t -on -fs FSNAME -path /tree3 # -t = tree quota, this eg turns it on on
          # if no -t defined, it is for the user??
    nas_quotas -t -list -fs FSNAME   # list tree quota
    
    
    To turn off Tree Quotas:
    - Path MUST BE EMPTY !!!!! ie, delete all the files, or move them out.  
        can one ask for a harder way of turning something off??!!
        Only alternative is to set quota value to 0 so it becomes tracking only, 
        but not fully off.
    
    
    Quota Policy change:
    - Quota check of block size (default) vs file size (windows only support this).
    - Exceed quota :: deny disk space or allow to continue.
    The policy need to be established from the getgo.  They can't really be changed as:
         - Param change require reboot
     - All quotas need to be turned OFF  (which requires path to be empty).
    
    Way to go EMC!  NetApp is much less draconian in such change.  
    Probably best to just not use quota at all on EMC!
    If everything is set to 0 and just use for tracking, maybe okay.  
    God forbid if you change your mind!
    
     
    
    

    CIFS Troubleshooting

    server_cifssupport VDM2 -cred -name WinUsername -domain winDom   # test domain user credentials
    
    server_cifs server_2  # if CIFS server is Unjoined from AD, it will state it next to the name in the listing
    server_cifs VDM2  # probbly should be VDM which is part of CIFS, not physical DM
    
    server_cifs VDM2 -Unjoin ... # to remove the object from AD tree
    
    server_cifs VDM2 -J compname=vdm2,domain=oak.net,admin=hotin,ou="ou=Computers:ou=EMC Celerra" -option reuse
    # note that by default the join will create a new "sub folder" called "EMC Celerra" in the tree, unless OU is overwritten
    
    
    
    server_cifs server_2 -Join compname=dm112-cge0,domain=nasdocs.emc.com,admin=administrator,ou="ou=Computers:ou=Engineering"
    
    
    ... server_kerberos -keytab ...
    
    

    Other Seldom Changed Config

    server_cpu server_2 -r now  # reboot DM2 (no fail over to standby will happen)
    
    server_devconfig
    server_devconfig server_2 -probe -scsi all # scan for new scsi hw, eg tape drive for NDMP
    server_devconfig ALL   -list -scsi -nondisks # display non disk items, eg tape drive
    
    
    /nas/sbin/server_tcpdump server_3 -start TRK0 -w /customer_dm3_fs/tcpdump.cap      # start tcpdump, 
     # file written on data mover, not control station!
     # /customer_dm3_fs is a file system exported by server_3
     # which can be accessed from control station via path of /nas/quota/slot_3/customer_dm3_fs
    /nas/sbin/server_tcpdump server_3 -stop  TRK0 
    /nas/sbin/server_tcpdump server_3 -display 
    # /nas/sbin/server_tcpdump maybe a sym link to /nas/bin/server_mgr
    
    
    /nas/quota/slot_2/ ... # has access to all mounted FS on server_2 
    # so ESRS folks have easy access to all the data!!
    
    /nas/tools/collect_support_materials  
    # "typically thing needed by support
    # file saved to /nas/var/emcsupport/...zip
    # ftp the zip file to emc.support.com/incoming/caseNumber
    # ftp from control station may need to use IP of the remote site.
       
    
    
    server_user ?? ... add    # add user into DM's /etc/passwd, eg use for NDMP
    

    Network interface config

    Physical network doesn't get an IP address (for Celera external perspective)
    All network config (IP, trunk, route, dns/nis/ntp server) applies to DM, not VDM.


    # define local network: ie assign IP 
    server_ifconfig server_2  -c      -D cge0  -n cge0-1     -p IP  10.10.53.152 255.255.255.224 10.10.53.158
    #      ifconfig of serv2  create  device  logical name  protocol      svr ip    netmask        broadcast
    
    server_ifconfig server_2 -a   # "ifconfig -a", has mac of trunk (which is what switch see)
    
    server_ifconfig server_2 cge0-2 down  ?? # ifconfig down for cge0-2 on server_2
    server_ifconfig server_2 -d cge0-2  # delete logical interfaces (ie IP associated with a NIC).
    ...
    
    server_ping  server_2 ip-to-ping  # run ping from server_2 
    
    server_route server_2 a default 10.10.20.1   # route add default 10.10.20.1  on DM2
    server_dns server_2    corp.hmarine.com ip-of-dns-svr  # define a DNS server to use.  It is per DM
    server_dns server_2 -d corp.hmarine.com    # delete DNS server settings
    server_nis server_2 hmarine.com ip-of-nis-svr   # define NIS server, again, per DM.
    server_date server_2 timesvc start ntp 10.10.91.10  # set to use NTP
    server_date server_2 0803132059   # set serverdate format is YY DD MM HH MM  sans space
          # good to use cron to set standby server clock once a day
          # as standby server can't get time from NTP.
     
    
    server_sysconfig server_2 -virtual  # list virtual devices configured on live DM.
    server_sysconfig server_4 -v -i TRK0   # display nic in TRK0
    server_sysconfig server_4 -pci cge0   # display tx and rx flowcontrol info
    server_sysconfig server_4 -pci cge4 -option "txflowctl=enable rxflowctl=enable"  # to enable rx on cge0
     # Flow Control is disabled by default.  But Cisco has enable and desirable by default, 
     # so it is best to enable them on the EMC.  Performance seems more reliable/repeatable in this config.
     # flow control can be changed on the fly and it will not cause downtime (amazing for EMC!)
     
    If performance is still unpredictable, there is a FASTRTO option, but that requires reboot!
    
    server_netstat server_4 -s -p tcp   # to check retrnsmits packets (sign of over-subscription)
    
    .server_config server_4 -v "bcm cge0 stat"  # to check ringbuffer and other paramaters 
          # also to see if eth link is up or down  (ie link LED on/off)
          # this get some info provided by ethtool
    
    .server_config server_4 -v "bcm cge0 showmac"  # show native and virtualized mac of the nic
    
    server_sysconfig server_2 -pci cge0 -option "lb=ip"
            # lb = load balance mechanism for the EtherChannel.  
            # ip based load balancing is the default
            # protocol defaults to lacp?  man page cisco side must support 802.3ad.  
            # but i thought cisco default to their own protocol.
            # skipping the "protocol=lacp" seems a safe bet
    
    
    

    Performance/Stats

    The .server_config is an undocumented command, and EMC does not recommended their use. Not sure why, I hope it doesn't crash the data mover :-P
    
    server_netstat server_x -i    # interface statistics
    server_sysconfig server_x -v    # List virtual devices
    server_sysconfig server_x -v -i vdevice_name  # Informational stats on the virtual device
    server_netstat server_x -s -a tcp   # retransmissions
    server_nfsstat server_x    # NFS SRTs
    server_nfsstat server_x -zero    # reset NFS stats
    
    
    
    # Rebooting the DMs will also reset all statistics.
    
    server_nfs server_2 -stats 
    server_nfs server_2 -secnfs -user -list
    
    
    .server_config server_x -v "printstats tcpstat"
    .server_config server_x -v "printstats tcpstat reset"
    .server_config server_x -v "printstats scsi full"
    .server_config server_x -v "printstats scsi reset"
    .server_config server_x -v "printstats filewrite"
    .server_config server_x -v "printstats filewrite reset"
    .server_config server_x -v "printstats fcp"
    .server_config server_x -v "printstats fcp reset"
    
    
    

    Standby Config

    Server failover:

    When server_2 fail over to server_3, then DM3 assume the role of server_2. VDM that was running on DM2 will move over to DM3 also. All IP address of all the DM and VDM are treansfered, including the MAC address.

    Note that when moving VDM from server_2 to server_3, outside of the fail over, the IP address are changed. This is because such a move is from one active DM to another.

    IP are kept only when failing over from Active to Standby.
    server_standby server_2 -c mover=server_3 -policy auto
    # assign server_3 as standby for server_2, using auto fail over policy
    
    Lab 6 page 89
    
    

    SAN backend

    
    If using the integrated model, there only way to peek into the CX backend is to use navicli command from the control station.
    
    navicli -h spa getcontrol -busy
     # see how busy the backend CX service processor A is
     # all navicli command works from the control station even when
     # it is integrated model that doesn't present navisphere to outside workd
     # spa is typically 128.221.252.200
     # spb is typically 128.221.252.201
     # they are coded in the /etc/hosts file under APM... or CK... (shelf name)
    
    
    cd /nas/sbin/setup_backend
    ./setup_clariion2 list config APM00074801759  # show lot of CX backend config, such as raid group config, lun, etc
    
    nas_storage -failback id=1 # if CX backend has trespassed disk, fail them back to original owning SP.
    
    
    

    Pro-actively replacing drive

    # Drive 1_0_7 will be replaced by a hot spare (run as root):
    # -h specify the backend CX controller, ip address in bottom of /etc/hosts of control station.
    # use of navicli instead of the secure one okay as it is a private network with no outside connections
    naviseccli -h xxx.xxx.xxx.xxx -user emc -password emc -scope 0 copytohotspare 1_0_7 -initiate
    # --or--
    /nas/sbin/navicli -h 128.221.252.200 -user nasadmin -scope 0 copytohotspare 1_0_7 -initiate 
    
    
    
    # find out status/progress of copy over (run as root)
    /nas/sbin/navicli -h 128.221.252.200 -user nasadmin -scope 0 getdisk 1_0_7 -state -rb
    

    User/security

    Sys admin can create accoutn for themselves into the /etc/passwd of the control station(s). Any user that have login via ssh to the control station can issue the bulk of the commands to control the Celerra. the nasadmin account is the same kind of generic user account. (ie, don't join the control station to NIS/LDAP for general user login!!)

    There is a root user, with password typically set to be same as nasadmin. root is needed on some special command in /nas/sbin, such as navicli to access the backend CX.

    All FS created on the Celerra can be accessed from the control station.

    Links

    1. EMC PowerLink
    2. EMC Lab access VDM2


    History

    
    DART 5.6 Released around 2009.0618.  Included Data Dedup, but must enable compression also
      which makes deflation a cpu and time expensive, not usable at all for high performance storage.
    DART 5.5 mainstream in 2007, 2008
    
    


    [Doc URL: http://www.grumpyxmas.com/emcCelerra.html ]
    [Doc URL: http://www.cs.fiu.edu/~tho01/psg/emcCelerra.html]

    (cc) Tin Ho. See main page for copyright info.
    Last Updated: 2008-03-22

    "LYS on the outside, LKS in the inside"
    "AUHAUH on the outside, LAPPLAPP in the inside"
    psg101 sn50 tin6150
  • isilon

    <-- Please click if you found this site useful ;-) If you want to add comments, please use Google's SideWiki.

    Isilon

    101

    
    isilon stores both windows sid and unix uid/gid with each file.
    When nfs client look at file created on windows, file may not have uid/gid in it.   
    isilon looks up the conversion from its mapping db.
    if it can't find one, it will generate a number, starting at 10000.
    
    
    

    maintenance commands

    isi_gather_info  # collect status of cluster and send to support (usually auto upload via ftp)
    

    HD Replacement

    isi devices     # list all devices of the node logged in
    isi devices -a status -d 14:bay28 # see statys of node 14, drive 28
    isi devices -a add    -d 14:28  # add the drive (after being replaced)
    isi devices -a format -d 14:28  # often need to format the drive for OneFS use first
         # it seems that after format it will automatically use drive (no ADD needed)
    
    # other actions are avail, eg smartfail a drive.
    
    isi_for_array -s 'isi devices | grep -v HEALTHY' # list all problematic dev across all nodes of the cluster.
    
    
    isi statistics drive --long  # 6.5 cmd to see utilization of a hd.
    

    user mapper stuff

    
    id username
    id windowsDomain\\windowsUser
        # Note that, username maybe case sensitive!!
    
    isi auth ads users  list --uid=50034
    isi auth ads users  list --sid=S-1-5-21-1202660629-813497703-682003330-518282
    isi auth ads groups list --gid=10002
    isi auth ads groups list --sid=S-1-5-21-1202660629-813497703-682003330-377106
    
    isi auth ads user list -n=ntdom\\username
    
    
    # find out Unix UID mapping to Windows SID mapping:
    # OneFS 6.5 has new commands vs 6.0
    isi auth mapping list  --source=UID:7868
    isi auth mapping rm    --source=UID:1000014
    isi auth mapping flush --source=UID:1000014   # this clear the cache
    isi auth mapping flush --all 
    isi auth local user list -n="ntdom\username" -v # list isilon local mapping
    
    isi auth mapping delete --source-sid=S-1-5-21-1202660629-813497703-682003330-518282 --target-uid=1000014 --2way
     # should delete the sid to uid mapping, both ways.
    isi auth mapping delete --target-sid=S-1-5-21-1202660629-813497703-682003330-518282 --source-uid=1000014
     # may repeat this if mapping not deleted.
    
    isi auth mapping dump | grep S-1-5-21-1202660629-813497703-682003330-518282
    
    isi auth ads group list --name
    
    isi auth local users delete --name=ntdom\\username --force
    
    rcf2307 is prefered auth mechanism... windows ad w/ services for unix.
    
    
    isi smb permission list --sharename=my_share
    
    
    
    
    
    
    # finding windows sid??  rm --2way ??
    
    
    
        # find out Unix UID mapping to Windows SID mapping:
        # OneFS 6.0: 
        isi auth ads users map list --uid=7868
        isi auth ads users map list --sid=S-1-5-21-1202660629-813497703-682003330-305726
    
        isi auth ads users map delete --uid=10020
        isi auth ads users map delete --uid=10021
        isi_for_array -s 'lw-ad-cache --delete-all'  # update the cache on all cluster node 
        # windows client need to unmap and remap drive for new UID to be looked up.
    
        # for OneFS 6.0.x only (not 6.5.x as it has new CIFS backend and also stopped using likewise)
        # this was lookup uid to gid map.
        
        sqlite3 /ifs/.ifsvar/likewise/db/idmap.db 'select sid,id from idmap_table where type=1;' # list user  sid to uid map
        sqlite3 /ifs/.ifsvar/likewise/db/idmap.db 'select sid,id from idmap_table where type=2;' # list group sid to gid map
    
        1:  The DB that you are looking at only has the fields that you are seeing listed.  
        With the current output it will give you the SID and UID of the users mapped.  
        With these commands you can find the username that is mapped to that information:
        #isi auth ads users list --uid={uid}
        or
        #isi auth ads users list --sid={sid}
    
        2:  The entries in the DB are made as the users authenticate to the cluster.  
        So when a client tries to access the share, the client sends over the SID, 
        we check the DB and if no entry is found, we check with NIS/LDAP, 
        if nothing is found there, we generate our own ID (10000 range) and add it to the DB.  
        Any subsequent access from that SID will be mapped to the UID in that DB.
    
        3:  You can run the following to get the groups and the same rules 
        apply for the GID and SID lookups:
        #sqlite3 /ifs/.ifsvar/likewise/db/idmap.db 'select sid,id from idmap_table where type=2;'
        #isi auth ads groups list --gid={gid}
        #isi auth ads groups list --sid={sid}
    
        4:  You can delete the entries in the database, 
        but the current permissions on files will remain the same.  
        So when the user re-accesses the cluster he will go through the 
        process outlined in question 1.
        
    
    
    
    

    Snapshot

    Snapshots take up space reported as usable space on the fs.
    cd .snapshot
    Admin can manually delete snapshot, or take snapshot of a specific directory tree instead of the whole OneFS.

    CIFS

    ACL

    ls -led   # show ACL for the current dir (or file if filename given)
    ls -l   # regular unix ls, but + after the permission bits indicate presence of CIFS ACL
    setfacl -b filename # remove all ACL for the file, turning it back to unix permission 
    chmod +a user DOMAIN\\username  allow generic_all /ifs/path/to/file.txt  # place NTFS ACL on file, granting user full access
    
    
    ls -lR | grep -e "+" -e "/" | grep -B 1 "+"    # recursively list files with NTFS ACL, short version
    ls -lR | grep -e "^.......... +" -e "/"  | grep -B 1 "^.......... +" # morse code version, works better if there are files w/ + in the name
    

    Time Sync

    
    isi_for_array -s 'isi auth ads dc' # check which Domain Controller each node is using
    isi_for_array -s 'isi auth ads dc --set-dc=MyDomainController # set DC across all nodes
    isi_for_array -s 'isi auth ads time'  # check clock on each node
    
    isi auth ads time --sync   # force cluster to sync time w/ DC (all nodes)
    
    isi auth ads status   # check join status to AD
    killall  lsassd    # reset daemon, auth off for ~30sec, should resolve offline AD problems
    
    

    "unix" config

    Syslog

    isi_log_server add SYSLOG_SVR_IP [FILTER]
    -or-
    vi /etc/mcp/templates/syslog.conf
    isi_for_array -sq 'killall -HUP syslogd'
    

    Disable user ssh login to isilon node

    For Isilon OneFS 6.0:
    vi /etc/mcp/templates/sshd_config
    
    add line
    AllowUsers root@* 
    
    Then copy this template to all the nodes:
    cp /etc/mcp/templates/sshd_config /ifs/ssh_config
    isi_for_array 'cp /ifs/ssh_config /etc/mcp/templates/sshd_config
    
    One may need to restart sshd, but in my experience sshd pick up this new template in less than a minute and users will be prevented from logging in via ssh.
    In OneFS 6.5, maybe the template will be replicated to all nodes? Or maybe that's only for syslogd, but not sshd, as they are concerned it may lock user out from all the nodes from ssh access...

    Links

    1. Isilon.com

    History

    OneFS 5.0
    OneFS 5.5
    OneFS 6.0 ca 2011.03 - Support mixed type of nodes - IQ 10000 and IQ 6000 in the same cluster. Single host entry in AD for whole Isilon cluster.
    OneFS 6.5 EA 2011.04 ? - SSD on the more high end node will catch meta data even for data in lower end node w/o SSD. CIFS is a completely new rewrite, and authentication with AD has changed. Test test test!!
    (2011) Acquired by EMC.
    
    
    


    [Doc URL: http://dl.dropbox.com/u/31360775/psg/isilon.html]

    (cc) Tin Ho. See main page for copyright info.
    Last Updated: 2012-06-21

    climate prediction banner boinc logo

    Valid CSS!

    Valid HTML 4.01 Strict


    "LYS on the outside, LKS in the inside"
    "AUHAUH on the outside, LAPPLAPP in the inside"
    psg101 sn50 tin6150

    welcome

    Sys Admin
    Pocket Survival Guide (PSG)



    Howdy there! This is my collection of notes that I have collected over the years as a Sys Admin. I am sharing them here hoping that they will be useful to you. These are concise notes to refresh your mind on how to do certain things, maybe you know what need to be done on one platform but don't know the commands on another. Or maybe just pointers for you to read up the relevan man pages or know what to google for :) They are work in progress, I know I need to polish up my notes so that other can make sense of them. As time permits...--which never really happens :-/

    Thanks for those who have send kind and encouraging words to me, it is nice to know that these notes are actually useful for someone other than myself!
    And special thanks to those who send me notes about mistakes, some pretty bad ones that could have caused someone to loose his/her job! So, be warned, check the man pages etc when doing something that could have negative impact (unexporting file system, shrinking volume, etc)!

    Okay, after a while of not having a home, I am hosting PSG on dropbox now. If you are bookmarking this, please use http://tiny.cc/tin6150/

    <-- Please click if you found this site useful ;-) IMHO

    Products Review

    Links

    References
    1. A Rosetta Stone of Unix OS Languages (as per the author). It is a pretty good translation between the many different Unix OS.
    2. A Rosetta Stone of Unix Shell Scripts
    3. Static table map of Unix OS commands from unixguide.net.
    4. AIX vs Solaris commands mapping
    5. Advanced Horizons has pretty good, concise info on AIX, Linux, Windows, Hardware, etc.
    6. Online Unix books (Mostly in Russian, some in English)
    7. Samba-CIFS
    8. Google for bookshelf
    9. .

    Lighter Readings
    1. Unixville Geeks galore :)
    2. Folklore on computer and companies (currently only apple).
    3. The New Hacker's Dictionary Hilarious read for the insiders. book ISBN 0262680920
    Tools
    1. HTML Validator
    2. Synergy Two computers, two monitors, ONE keyboard and mouse, with cut and paste!! synergy.conf
    3. crypt generation web form for generating encrypted password
    4. ASCII Art Generator, kinda like web-based figlet, pretty cool :) Another version hosted by a friend of mine.
    5. cacert.org Free Digital Certificate for secure web servers, etc.
    6. Sunfreeware main mirror site hosted by tds.net.
    7. .

    Books

    • Unix System Administration Handbook by Nemeth et al. This is the Bible for Unix Sys Admin. Excellent read and reference. Highly recommended.
    • Essential System Administration by Frisch. Another comprehensive Unix Sys Admin Book.
    • Tha Awk Programming Language by the original authors. Kinda old, but very concise yet easy to read, so still highly recommended. After reading Ch3, you can do most of what you would need to do with Awk. Unfortunately, its age does not cover new features implemented by gAWK :-(

    About me

    I have been "hacking" computers for 10+ years now. I started liking to fiddle with OS since college days in FIU, and it is still a hobby when I am not hiking and biking or studying for biotech :-)

    Copyright info about this work Creative Commons License

    This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike2.5 License.
    Pocket Sys Admin Survival Guide: for content that I wrote, (CC) 2005,2009 Tin Ho [ tin6150 (at) gmail.com ] some rights reserved. Some contents are "cached" here for easy reference. Sources include man pages, vendor documents, online references, discussion groups, etc. Copyright of those are obviously those of the vendor and original authors. I am merely caching them here for quick reference and avoid broken URL problems.
    DOC URL, Where is the "home" PSG ?
    http://tiny.cc/tin6150/ New home in 2011.06.
    http://unixville.com/~sn/psg/psg.html (coming soon)
    ftp://sn.is-a-geek.com/psg/psg.html My home "server". Up sporadically.
    http://www.cs.fiu.edu/~tho01/psg/psg.html (no longer updated as of 2007-06)
    http://www.fiu.edu/~tho01/psg/psg.html (no longer updated as of 2007-05)

    Google

    "LYS on the outside, LKS in the inside"
    "AUHAUH on the outside, LAPPLAPP in the inside"
    sn50 tin6150 psg101sn.is-a-geek.com