Solaris Live Upgrade Procedure for Patching

      Normally  CPU cluster patching for Solaris OS have taken an hour to complete, it increase the maintenance downtime window. In order to overcome that Solaris having the feature called  "LIVE UPGRADE" .
         The main advantages of Live upgrade is minimizing the downtime and providing the system admin to revert the original OS in case of any patching failure.

       In general, the Live Upgrade is briefly categorized as follows

• Create a new boot environment if you haven't already.
• Patch the new boot environment.
• Boot from the new boot environment.
• Check your results for the changes and see if they are acceptable 

Following are the steps to perform Solaris Live Upgrade for Patching

Step 1: Verify Live Upgrade packages are installed

           #pkgchk -v SUNWlucfg SUNWlur SUNWluu

Step 2: Create a current & new boot environment within the same root pool (rpool)
                #lucreate -c BE_APP01  -n new new_BE_APP01

Step 3: Verify step 2
           #zfs list 
          #lustatus

Step 4: Apply CPU onto the new boot environment 

           #cd 10_Recommend

           #./installpatchset -B new_BE_APP01 --s10patchset

Step 5: Activate the new boot environment
           #luactivate -s new_BE_APP01

Step 6: Reboot the server to boot to the new boot environment

           #shutdown -i 6 -g 0 -y

   Note: Do NOT use any other commands to reboot the server. you must use the above-mentioned command. Be patient, wait at least 5-10mins. if it still does not reboot, kindly type the command "reboot"

Step 7: Verify the status

           #lustatus 

Post Live Upgrade Procedures

• If there are no problems, you can delete the former boot environment to  save disk space by issuing the comman

          #ludelete BE_APP01  

• For furture patching, you only need to create a new boot environment within the same root pool (rpool), by issuing the command.

         #lucreate -n BE_APP01_yyyymmdd  

Solaris Hardening Procedure -- Part 1

       This articles elaborately describes about the hardening procedure of Solaris OS. Hardening process will take place in different segments of the OS, hence due to lengthy procedures have split into 3 parts. 
       
1. Services
    a) Disabling the Restricted Services
            i) Restricted services 
                  Stop the restricted services which will pose a risk to servers. The following are restricted services.
        telnet
        Uucp
        Netstat   
        Comsat
        Time
        Echo
        Discard
        ftp
        tftp
        Daytime
        Rquoted
        Rexecd
        Rpc.ttdbserverd
        finger
        talk
        chargen
        ident
        systat
        yppasswdd, ypserve, ypxfrd
        services (i.e. shell, login, klogin, exec, etc.) that listen to r-commands  (rlogin, rsh etc).
        ToolTalk (ttdbserverd)
        Calendar Manager (cmsd)
        statd (Unless required by NFS. See Use of NFS section for restrictions)
        sadmind (solstice admin daemon)
        rstatd   
        rusersd
        rwalld
        sprayd
        automount (Solaris)
           ii) SSH client and server
              Only Secure Shell protocol version 2 is allowed, SSH protocol v1 must be disabled. It is mentione in the file /etc/ssh/sshd_config
              #Protocol 1
              Protocol 2
          iii) Disable NIS services
             #svcadm disable svc:/network/nis/server:default   
             #svcadm disable svc:/network/nis/client:default 
          iv) Disable Sendmail
             #svcadm disable svc:/network/smtp:sendmail

2) Desktop environments
        i) X-Windows 
              X-Windows are not allowed in production, xhost must not be used.
              X-window traffic must be tunneled through SSH. To perform this comment out "X11Forwarding yes" in the file /etc/ssh/sshd_config 
       ii) Desktop Environment
              DE environments are not allowed. Disable dt login service
              #svcadm disable cde-login
       iii)#rm /usr/openwin/bin/xwd
           #rm /usr/openwin/bin/xwud

3) Password Security  
       i) Local Unix Password Baseline 
             Min no of alphabetic characters is 1
               /etc/default/passwd contains the setting MINALPHA=1
            Min no of special characters is 1
              /etc/default/passwd contains the setting"MINSPECIAL=1"
            Maximum number of repeatable characters is 1
              /etc/default/passwd contanis the MINREPEATS=1
        ii) Unix Password History 
             Set Prior password history to 10
                /etc/default/passwd contains the HISTORY=10
       iii) Unix Account unsuccessful login retries
              /etc/default/passwd contains "RETRIES=3"
              /etc/user_attr    contains "lock_after_retries=no" for root
             root::::auths=solaris.*,solaris.grant;profiles=Web Console Management,All;lock_after_retries=no;
       iv) Account Password life
             Password is valid for 30 days.
            #passwd -x 30 -n 7 -w 7 <username>
       v) Session Inactive  
            Enable inactive login session timeout to 15 mins (300 secs)  
            #cat /etc/default/login 
            ::: 
            TIMEOUT = 300
            :::
       vi) In addition, add the following lines in /etc/default/passwd 
            MAXWEEKS=4
            MINWEEKS=1
            PASSLENGTH=8
    
            MAXWEEKS - Maximum time period that a password is valid.
            MINWEEKS - Minimum time period before a password can be changed.
            PASSLENGTH - Minimum length of a password, in characters.           

4) Logging and Enabling User authentication auditing

            
         All Successful and failed logins are logged .
           Add "auth.info /var/log/authlog" to /etc/syslog.conf for capturing syslog events sent to LOG_AUTH. This contains information on successful and failed login and su (switch user) attempt 

             #touch /var/log/authlog
             #chown root:sys /var/log/authlog
             #chmod 600 /var/log/authlog

             #vi /etc/syslog.conf
              auth.info /var/log/authlog

          Logging only Failed Logins

                #cat /etc/default/login 

              SYSLOG=YES 

              SYSLOG_FAILED_LOGINS=3

             #touch /var/adm/loginlog
             #chmod 600 /var/adm/loginlog
             #chown root:sys /var/adm/loginlog

            

          Logging only Successful logins

              #touch /var/log/logins
              #chgrp sys logins
              #chmod 600 logins

              #cat /etc/syslog.conf

              local0.info /var/log/logins

              

         Added the following entry to /etc/profile and /etc/.login:   
         logger -p local0.info "User $LOGNAME has logged in"

         After editing the /etc/syslog.conf file restart the service

             #svcadm disable system-log

             #svcadm  enable system-log

         SU events logging

              #cat /etc/default/su
               SYSLOG=yes

               SULOG=/var/adm/sulog   

         Cron commands should be logged

               #cat /etc/default/cron
               CRONLOG=YES

5) Folder and File permissions 
        Set the permissions on the system important folders and files 
              #chmod 755 /etc /var /var/spool
              #chmod 700 /var/cron
              #chmod 750 /etc/security
              #chmod 600 /var/adm/messages /var/log/syslog /var/adm/loginlog
 
             
  

Sun SPARC Firmware (ILOM/BIOS) update

         In previous article, we have seen Firmware update for Sun x86 arch, we look it for Sun SPARC architecture. Follow the instructions to perform the firmware update.

Step 1
   Download the corresponding latest patches/updates image file from the Oracle downloads page
   http://www.oracle.com/technetwork/systems/patches/firmware/index.html
Step 2 
   The Sun_System_Firmware-7_4_0_b-SPARC_Enterprise_T5140+T5240.pkg file is the image that will need to be transferred to the Service Processor  for update purposes. To perform this transfer the firmware patch to the corresponding server.
Step 3
    Continue preparation of upgrading the SP by downloading the image from the host to the  SP.  Consider the image file in the path /tmp/images
  #cd /tmp/images
  #/usr/platform/sun4v/sbin/sysfwdownload Sun_System_Firmware-7_4_0_b-SPARC_Enterprise_T5140+T5240.pkg
.......... (9%).......... (18%).......... (27%).......... (37%).......... (46%).......... (55%).......... (64%).......... (74%).......... (83%).......... (92%)......... (100%)

    The above process will take 10-15 mins of time to download the patch to SP (located in /coredump/[image].pkg

Step 4

     Shutdown the host Solaris OS

  #shutdown -i0 -g0 -y

Step 5

Establish and connection to the Service Processor via ssh & login as Admin user with  ALOM mode

ssh root@xx.xxx.xx.x

...

Are you sure you want to continue connecting (yes/no)? yes

...

Password: password (nothing displayed)

Waiting for daemons to initialize...

Daemons ready

Sun(TM) Integrated Lights Out Manager

Version 2.0.0.0

Copyright 2007 Sun Microsystems, Inc. All rights reserved.

Use is subject to license terms.

Step 6

Verify that the host is powered off.If the host is not powered off, type the stop /SYS command.

      sc> stop /SYS

Step 7 

     Ensure the keyswitch is set to NORMAL. If it is set to LOCKED you will not be able to flash the firmware or send STOP-A to the system. If keyswitch is set to NORMAL start actual flashing process using flashupdate command

         sc> showkeyswitch

         Keyswitch is in the NORMAL position

   If the virtual key switch is in LOCKED position you can change that with the following command:

         sc> setkeyswitch -y normal

Step 8 

    Check the actual version

         sc>showhost 

Step 9

    Make the Flash update the downloaded Firmware Image

        sc>flashupdate -s 127.0.0.1

        SC Alert: System poweron is disabled.

        .........................................................................................................................................................

        Update complete. Reset device to use new software.

Step 10

      The Sun System Firmware has now been updated. For the system to use the new firmware you must reset the Service Processor . 

      sc> resetsc

      Are you sure you want to reset the SC [y/n]?  s

      Are you sure you want to reset the SC [y/n]?  y

      User Requested SC Shutdown

Step 11

      Now check the actual version

      sc>showsc

Step 12

       Now Poweron the server and boot the OS from SP

      sc>poweron

      sc>boot 

Note:

How to revert or downgrade the Sun Firmware in case of failure?

       You can always upgrade to new firmware, but downgrades are not guaranteed. Whether they work or not depends on whether your system's hardware configuration and the rest of the software stack (the Oracle Solaris OS, patches, and so on) have any dependencies on your newer firmware. If there are dependencies and you downgrade, you might find yourself unable to even boot. So in general, downgrades should be avoided unless you know for sure your compatibility matrix is verified. Also, it should be obvious that downgrading might cause you to lose features or functionality. For example, LDoms 1.0.1 requires System Firmware release 6.5.x. Downgrading to 6.4.6 would break LDoms functionality

Reference: http://www.oracle.com/technetwork/systems/patches/firmware/faq-jsp-139605.html

  

        

Sun x86 Firmware (ILOM/BIOS) Update Procedure

         This  articles describes SUN x86  Firmware Patches/Updates procedure.
This method of updating firmware patch is called Remote Flashing image method.
       In Sun Fire X4xxx series server with firmware that resides in the following locations:

     Server motherboard (system BIOS)
     Server’s Service Processor (ILOM)
     On-board LSI disk controller (HBA BIOS)

    Among these BIOS/ILOM are always updated together, We can update both ILOM and BIOS firmware even if one of their versions has not changed since the last software update.  

Step 1
       Download the corresponding latest patches/updates from the Oracle downloads page.
       http://www.oracle.com/technetwork/systems/patches/firmware/index.html

Step 2
      Ensure that you have the file: ILOM-3_0_6_15_d_r67788-Sun_Fire_X4150_X4250_X4450.pkg  on the PC that will access the ILOM web GUI. Also ensure the MD5 hash of the downloaded file is same as provided.

Step 3
    Start the Intergrated LOM SP web GUI by typing the IP address of the SP into a browser.

Step 4

 Log in to the SP, and backup the current configuration

 

     Using the option Backup/Restore in Maintenance tab, Select a transfer method  from the Transfer Method drop-down list as "Browser"
   With the Browser transfer method, the backup file is saved according to your web browser settings.

Step 5

    Click on the Maintenance Tab and choose Enter Upgrade Mode
  Step 6
    Click Browse and navigate to the location of the flash file

Step 7
   Click Upload

Step 8
    Once the firmware is uploaded, you will be asked to confirm the upgrade,as the current firmware and future firmware versions will be displayed. Please capture screenshot before proceeding.


   You will be presented with a screen that asks if you want to preserve or Not to preserve the current Settings, and if you want to delay BIOS upgrade until next server poweroff.
    Please Choose “Preserve” and do NOT delay BIOS upgrade
Step 9
    Click "Start Upgrade". The BIOS and SP firmware on the Sun Fire X4150/X4250/X4450 server will be updated. 




 
  
Step 10
    Reboot server for confirmation 
 Step 11    Log in to the SP and check the ILOM and BIOS version.

Check My Next article on Sun SPARC Firmware update