Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
25 replies Latest Post - ‏2012-05-08T16:28:11Z by SystemAdmin
SystemAdmin
SystemAdmin
6902 Posts
ACCEPTED ANSWER

Pinned topic HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

‏2012-04-28T12:33:39Z |
Dear All

i got error yesterday on my power P510 , and after rebooting the system hangs on AIX logo (when system starts) .
After investigation and check all my HD from SMS , i found that HD that holds the rootvg and AIX installation is FAILED .
(the server connected to a SAN storage contains Oracle Application ) , but no errors from SAN SCSI Disks .

I made a new installation on a new HD .. and when login , I did NOT find the :
oraclevg (Oracle data Volume group )and the 3 Filesystems was related to PV assigned to this VG .

I only find the rootvg (new) with only one HD that had the installation .

This is a Production Server ( i have backup from Oracle files but not AIX SYStem backup ) ..

Any help to restore the Previous VG and FS without losing data ..
Sorry , Iam not experienced enough in AIX .

Please ASSIST ASAP

Thanks
Abuzeid
Updated on 2012-05-08T16:28:11Z at 2012-05-08T16:28:11Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    6902 Posts
    ACCEPTED ANSWER

    Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

    ‏2012-04-28T13:39:31Z  in response to SystemAdmin
    Hi Friends ...

    Waiting your assistance ..
    Its a production server , and i need a scenario to restore previous VG ,PV and FS configuration .

    Thanks
    • orphy
      orphy
      480 Posts
      ACCEPTED ANSWER

      Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

      ‏2012-04-28T14:31:21Z  in response to SystemAdmin
      It's not a very good idea to wait for someone on this forum to help you recover a prod system!

      Don't you have AIX support for this prod system? If so, you should have called them with a sev 1 PMR by now (800-IBM-SERV) If not, assuming there is no change on the SAN side, you should have the Oracle hdisk back after the OS reinstall. If all you had on this box was rootvg and oraclevg, do a lspv to find out the name of the unassigned PV then do an importvg -y oraclevg hdiskX. You should then be able to mount your Oracle filesystems. You shouldn't have lost any Oracle data other than those on flight ones when the box crashed.

      You should plan on doing mksysb periodically.
      Orphy
      • SystemAdmin
        SystemAdmin
        6902 Posts
        ACCEPTED ANSWER

        Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

        ‏2012-04-28T15:00:57Z  in response to orphy
        Hi Orphy

        Thanks a lot for your immediate response .
        As i understood from your previous reply , that i can restore my data (FS) using importvg -y oraclevg hdiskX.
        That mean i should Know what exactly PV was assigned to this Oracle VG . (what to do if not , especially i have 9 HD in the SAN storage ???)

        Could you give me a detailed steps to do that .

        Thanks
        Abuzeid
        • orphy
          orphy
          480 Posts
          ACCEPTED ANSWER

          Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

          ‏2012-04-28T15:51:47Z  in response to SystemAdmin
          Are all nine PVs belonging to oraclevg? If so, just pick any unused one and do a importvg. Then do an "lsvg -l oraclevg" too see if those are the oracle FS in that VG. If so, you go on to mount the FS. If you don't see the Oracle FS, you have more than 2 VGs on this box. You then need to do a "varyoffvg oraclevg" and an "exportvg oraclevg". Repeat the same step using another PV. You will eventually find the PV belonging to oraclevg. Chances are that you have more than one PVs in the oraclevg but all you need is one PV for importvg since the VGDA on each hdisk knows all the PVs in the same VG. The importvg/exportvg won't harm your user data so no worry about trying it.
          Orphy
          • ColombianJoker
            ColombianJoker
            68 Posts
            ACCEPTED ANSWER

            Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

            ‏2012-04-28T16:25:34Z  in response to orphy
            Hello.

            Can you tell us:
            How many VGs you had before?
            Can you show us the output of:
            lspv
            lqueryvg -Atp $(lsdev -c disk -F name | tail -n1)
            • SystemAdmin
              SystemAdmin
              6902 Posts
              ACCEPTED ANSWER

              Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

              ‏2012-04-28T21:11:49Z  in response to ColombianJoker
              Hello ,

              i think i facing a problem regarding to the SSA SAN storage ,,
              i cannot see or detect the SAN physical volumes from command #lspv or from smit tool .

              How to ensure the SAN storage connected properly ? (from command line or smit) may this help to solve my issue .

              do you need me to upload any command result or any log file ?
              Thanks
              Abuzeid
              • orphy
                orphy
                480 Posts
                ACCEPTED ANSWER

                Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

                ‏2012-04-28T22:36:55Z  in response to SystemAdmin
                Do you know if you have AIX support for this box? If so, call them! If you are not sure, call them to check.

                Are these 7133 SSA? I haven't touched them for a long time so don't remember a whole lot about them. You can run cfgmgr to see if it complains about missing any device drivers since you reinstalled AIX. Check the hardware to make sure they are good (i.e. no fault lights, still power up, etc).

                Do you see any errors when you run errpt?
                Orphy
  • SystemAdmin
    SystemAdmin
    6902 Posts
    ACCEPTED ANSWER

    Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

    ‏2012-04-30T00:46:47Z  in response to SystemAdmin
    Hi abuzeid_maher,

    It seems you need to install Powerpath and ODM on your Server, as you said, you made fresh install the Server after it failed.
    Once you installed the Powerpath and ODM, you may now run commands to detect your SAN disks, anyway, you didn't mention if your SAN is Clariion or DMX.

    Regards,
    Zieggy
  • SystemAdmin
    SystemAdmin
    6902 Posts
    ACCEPTED ANSWER

    Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

    ‏2012-05-04T17:51:17Z  in response to SystemAdmin
    Hi , all
    Now i can detect the SSA SAN storage (replacing Fiber cables).
    when booting in maintenance mode , i can see the VGs .

    Also , i made a fsck as per note :http://www-01.ibm.com/support/docview.wss?rs=111&context=SWG10&dc=DA480&dc=DB520&q1=LED+518&uid=isg3T1000131&loc=en_US&cs=utf-8&lang=en

    fsck /dev/hd4
    fsck /dev/hd2
    fsck /dev/hd3
    fsck /dev/hd9var
    fsck /dev/hd1

    and i could mount (/ , tmp , usr , and var filesystems) .

    I found that the hostname for the server is changed to localhost !!!
    i change it from smit and sync;sync;sync;reboot

    after normal booting , the previous error disappear.
    but after code 0c32 the display has no code and system stuck and no response.(i tried many times).

    any advice .

    thanks
    • orphy
      orphy
      480 Posts
      ACCEPTED ANSWER

      Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

      ‏2012-05-04T19:30:02Z  in response to SystemAdmin
      You were able to boot up AIX without any issue before. So are you saying now that somehow rootvg got in trouble after you you replaced some cables? Do you have a console/monitor attached to this server? If so, answer the question on the console to select that as the console display. After you boot up the box, use "smitty hostname" to change the hostname.
      Orphy
  • SystemAdmin
    SystemAdmin
    6902 Posts
    ACCEPTED ANSWER

    Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

    ‏2012-05-04T17:56:17Z  in response to SystemAdmin
    hi ,
    each time i booted in service mode , i found the hostname back to localhost ,
    and didnot changed ,
    also, i couldn't up the en0 lan (may be in single user mode) .
    how to permanent change the hostname from maintenance mode ?

    thanks
  • SystemAdmin
    SystemAdmin
    6902 Posts
    ACCEPTED ANSWER

    Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

    ‏2012-05-04T19:40:11Z  in response to SystemAdmin
    hello ,

    the status now is :
    aix booting and code 0c32 appear on the code display .
    it supposed to get aix screen (services starting ) , but the screen go offline .
    and stuck in this ..
    as you said , its a console issue . could you tell me how to fix it please ..
    can i set the console from SMS menu ??? or from where ?

    thanks
  • SystemAdmin
    SystemAdmin
    6902 Posts
    ACCEPTED ANSWER

    Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

    ‏2012-05-05T14:24:47Z  in response to SystemAdmin
    Hello ALL

    I installed a new AIX on another HD .
    and i can see all SAN SSA disks ,but the both old Volume groups not shown for sure .
    i need to restore my volume groups (oravg and datavg) with my old configuration and Filesystems .
    The first VG (oravg) was on one ligical volume lv0 and contains (8 physical volumes HD)on the SSA SAN .
    i tried to create a VG with the same old name oravg but he asked me to assign a physical volume to it .
    Does this will affect my data on the HDs , and as i told you i have multiple volumes HD was assigned to this VG ,
    how to add them all ?
    could you please tell me how to restore this in details .
    I am sorry , but its a critical issue now ...
    Please Assist ..

    Regards

    Abuzeid
    • orphy
      orphy
      480 Posts
      ACCEPTED ANSWER

      Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

      ‏2012-05-05T14:36:52Z  in response to SystemAdmin
      Do not make a VG with those PVs or your data will be gone! Did you ever try my suggestions (posted Apr 28, 2012 11:51:47 AM above) using importvg/exportvg? If not, you should.
      Orphy
  • SystemAdmin
    SystemAdmin
    6902 Posts
    ACCEPTED ANSWER

    Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

    ‏2012-05-05T15:50:29Z  in response to SystemAdmin
    Hello ,

    ok ,i will use importvg .
    but i have mutiple physical volumes assigned to these volumegroups .
    could you provide me with commands >

    Thanks
    • orphy
      orphy
      480 Posts
      ACCEPTED ANSWER

      Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

      ‏2012-05-05T16:07:23Z  in response to SystemAdmin
      Don't worry about the number of disks you have in oraclevg. Run these commands.

      lspv # find the first unassigned PV, e.g. hdisk1
      importvg -y oravg hdisk1
      lsvg -l oraclevg

      If you find lv0, you can mount the oracle filesystem and move on to do lspv to find the other unused PVs to importvg for datavg. If you don't see lv0, this is probably datavg. Then do these

      varyoffvg oravg
      exportvg oravg
      importvg -y datavg hdisk1
      lsvg -l datavg
      mount ...

      Once you have datavg, use lspv to find the other 8 unused PVs (HDs) and use one of them to do importvg for oravg as mentioned above.
      Orphy
      • orphy
        orphy
        480 Posts
        ACCEPTED ANSWER

        Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

        ‏2012-05-05T16:09:50Z  in response to orphy
        I meant oravg if my phone changed it to oraclevg for me so just use oravg.
        Orphy
        • SystemAdmin
          SystemAdmin
          6902 Posts
          ACCEPTED ANSWER

          Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

          ‏2012-05-05T17:06:40Z  in response to orphy
          Hello Orphy ..

          i got this error when try to varyon the volume group:
          0516-082 varyonvg unable to access a special device ..

          Thanks
          • orphy
            orphy
            480 Posts
            ACCEPTED ANSWER

            Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

            ‏2012-05-05T17:37:04Z  in response to SystemAdmin
            What was the varyonvg for? Can you show me outputs of these commands?

            history
            lsdev -Ccdisk
            lspv
            Orphy
            • SystemAdmin
              SystemAdmin
              6902 Posts
              ACCEPTED ANSWER

              Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

              ‏2012-05-05T18:06:56Z  in response to orphy
              Hello Orphy ,

              Good News : the OLD system is up and running now ..
              (i use diagnostics from service mode and found a problem related to Monitor , after change it
              the AIX start in Gui mode ).

              BUT , the BAD news is : i could NOT find my oracle File systems /oradata and /oratech ..

              Here are the requested commands output :

              bash-3.2# lsdev -Ccdisk
              hdisk0 Available 1S-08-00-5,0 16 Bit LVD SCSI Disk Drive
              hdisk1 Available 1S-08-00-8,0 16 Bit LVD SCSI Disk Drive
              hdisk2 Available 1n-08-L SSA Logical Disk Drive
              hdisk3 Available 1n-08-L SSA Logical Disk Drive
              hdisk4 Available 1n-08-L SSA Logical Disk Drive
              hdisk5 Available 1n-08-L SSA Logical Disk Drive
              hdisk6 Available 1n-08-L SSA Logical Disk Drive
              hdisk7 Available 1n-08-L SSA Logical Disk Drive
              hdisk8 Defined 1S-08-00-4,0 16 Bit LVD SCSI Disk Drive
              hdisk9 Available 1S-08-00-4,0 Other SCSI Disk Drive
              hdisk10 Available 1S-08-00-3,0 Other SCSI Disk Drive
              hdisk11 Available 1n-08-L SSA Logical Disk Drive
              hdisk12 Available 1n-08-L SSA Logical Disk Drive
              hdisk13 Available 1n-08-L SSA Logical Disk Drive
              hdisk14 Available 1n-08-L SSA Logical Disk Drive



              bash-3.2# lspv
              hdisk0 005fbbaf82025c36 rootvg
              hdisk1 005fbbafa44aa990 rootvg
              hdisk2 005fbbafc376f887 oravg
              hdisk3 005fbbafd5591e3c oravg
              hdisk4 005fbbaf7692daf5 oravg
              hdisk5 005fbbaf76933c62 oravg
              hdisk6 005fbbaf3d8dd464 oravg
              hdisk7 005fbbaf0a9b49e4 rootvg
              hdisk9 005fbbafb58c961d datavg
              hdisk10 005fbbaff33a9a29 datavg
              hdisk11 005582dae5772092 None
              hdisk12 005582dae5a9cf7c None
              hdisk13 005582dae5c1a85d None
              hdisk14 005582dabbd16004 None



              bash-3.2# df
              Filesystem 512-blocks Free %Used Iused %Iused Mounted on
              /dev/hd4 655360 578632 12% 1747 2% /
              /dev/hd2 8388608 6879896 18% 20960 2% /usr
              /dev/hd9var 131072 31512 76% 2063 13% /var
              /dev/hd3 7077888 1493696 79% 705 1% /tmp
              /dev/hd1 131072 126808 4% 22 1% /home
              /proc - - - - - /proc
              /dev/hd10opt 393216 339792 14% 481 1% /opt
              /dev/lv01 110100480 24655256 78% 882498 25% /oraapp


              Check this also :

              bash-3.2# mount -a
              mount: 0506-324 Cannot mount /dev/hd1 on /home: The requested resource is busy.
              mount: 0506-324 Cannot mount /proc on /proc: The requested resource is busy.
              mount: 0506-324 Cannot mount /dev/hd10opt on /opt: The requested resource is busy.
              mount: 0506-324 Cannot mount /dev/lv02 on /oratech: A file or directory in the path name does not exist.
              mount: 0506-324 Cannot mount /dev/lv00 on /oradata: A file or directory in the path name does not exist.
              mount: 0506-324 Cannot mount /dev/lv01 on /oraapp: The requested resource is busy.
              mount: 0506-324 Cannot mount /dev/datalv on /data: A file or directory in the path name does not exist.
              bash-3.2#

              I could not find the /dev/lv02 ,/dev/lv00 ,/dev/datalv ..

              Waiting your Assistance , and thanks alot

              Regards
              Abuzeid
              • orphy
                orphy
                480 Posts
                ACCEPTED ANSWER

                Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

                ‏2012-05-05T18:43:48Z  in response to SystemAdmin
                Are you sure you don't have another VG containing /oradata and /oratech? Would they be in hdisk11-14? Those SSA disks are unassigned and you only have 5 PVs in oravg but you said oravg is supposed to have 8. Why not try importvg using hdisk11-14 to see what you have there?

                If you can't find them there, you should just recreate them and restore. I think you mentioned having backups.

                Orphy
          • SystemAdmin
            SystemAdmin
            6902 Posts
            ACCEPTED ANSWER

            Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

            ‏2012-05-05T18:12:02Z  in response to SystemAdmin
            Please find the attached file contains /etc/filesystems

            Thanks
  • SystemAdmin
    SystemAdmin
    6902 Posts
    ACCEPTED ANSWER

    Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

    ‏2012-05-05T19:30:20Z  in response to SystemAdmin
    Hello ALL ,

    thanks for you all , especially Orphy .

    I used redefinevg command to make varyonvg work , then synclvodm .
    After that i mount -a and then found all my data under filesystems ..
    you can contact me for further details ..

    Thanks again ..

    Abuzeid
    • orphy
      orphy
      480 Posts
      ACCEPTED ANSWER

      Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

      ‏2012-05-06T00:41:57Z  in response to SystemAdmin
      Good for you! After everything is settled down, you should put a backup/recovery plan in place. I suggest doing a mksysb at least once a month and maybe run a set of command to save your system info for future reference. If you don't want to figure out what to save, look into the snap command and simply save its output which usually have enough info on another box. Good luck with your new venture!
      Orphy
      • SystemAdmin
        SystemAdmin
        6902 Posts
        ACCEPTED ANSWER

        Re: HOW restore my Filesystems or volume group after reinstalling the AIX 5.3

        ‏2012-05-08T16:28:11Z  in response to orphy
        Thanks ALL for support .

        and i will schedule a system backup monthly , and databackup weekly ..

        Regards

        Abuzeid