Topic
20 replies Latest Post - ‏2010-11-16T14:05:07Z by SystemAdmin
PKrogh
PKrogh
11 Posts
ACCEPTED ANSWER

Pinned topic FBSS5007W Failed to complete procedure on sectors

‏2010-11-10T08:57:29Z |
Hi

Help me please with this error:FBSS5007W Failed to complete procedure on sectors

I have attached log files!
Updated on 2010-11-16T14:05:07Z at 2010-11-16T14:05:07Z by SystemAdmin
  • PKrogh
    PKrogh
    11 Posts
    ACCEPTED ANSWER

    Re: FBSS5007W Failed to complete procedure on sectors

    ‏2010-11-10T09:13:53Z  in response to PKrogh
    And log file from server.
    • PKrogh
      PKrogh
      11 Posts
      ACCEPTED ANSWER

      Re: FBSS5007W Failed to complete procedure on sectors

      ‏2010-11-10T09:35:58Z  in response to PKrogh
      Again!
      • MichaelSternberg
        MichaelSternberg
        24 Posts
        ACCEPTED ANSWER

        Re: FBSS5007W Failed to complete procedure on sectors

        ‏2010-11-10T14:00:40Z  in response to PKrogh
        According to FAST_BACK_CLIENT040.sf log file that was supplied, snapshot of volume M: was aborted at Nov 10 09:02:42:434. There are no log files of FB Server inside of "serverLog andFBSS5007W.zip" that was supplied - you accidentally put log files of FB Manager inside. Without FB Server logs I can not proceed in analysis of this issue.
        • PKrogh
          PKrogh
          11 Posts
          ACCEPTED ANSWER

          Re: FBSS5007W Failed to complete procedure on sectors

          ‏2010-11-11T06:44:33Z  in response to MichaelSternberg
          Ohh!

          I got a new set off log files. It failed again!

          Attachments

          • MichaelSternberg
            MichaelSternberg
            24 Posts
            ACCEPTED ANSWER

            Re: FBSS5007W Failed to complete procedure on sectors

            ‏2010-11-11T09:18:42Z  in response to PKrogh
            According to screenshot, that was supplied in previous message, the snapshot was aborted at 11/11/2010 3:15:36 AM.

            FB Client logs that were supplied starts at Nov 10 03:57 and ends at Nov 11 01:20
            FB Server logs that were supplied starts at Nov 11 00:29 and ends at Nov 11 01:07

            I need logs of FB Client and FB Server that will cover time that the problem has occurred.
            Those are text files, can be opened in any text editor or text viewer. Please verify that logs, that are supplied, cover the time that problem has occurred.
            • PKrogh
              PKrogh
              11 Posts
              ACCEPTED ANSWER

              Re: FBSS5007W Failed to complete procedure on sectors

              ‏2010-11-11T09:33:17Z  in response to MichaelSternberg
              Try again!

              Attachments

              • MichaelSternberg
                MichaelSternberg
                24 Posts
                ACCEPTED ANSWER

                Re: FBSS5007W Failed to complete procedure on sectors

                ‏2010-11-11T12:51:04Z  in response to PKrogh
                And again we don't have logs of FB Server from the time that snapshot was aborted.

                When analyzing supplied log file of FB Client following snapshots aborts can be spotted:
                1493 was aborted at Nov 11 02:07:25:787 due to network disconnect between FB Client and FB Server.
                1579 was aborted at Nov 11 07:55:33:211 from unknown reason.

                Supplied log files of FB Server start at Nov 11 02:43:47:280 and end at Nov 11 06:21:40:944.

                Please check above snapshot abort timestamps and FB Server traces time range and understand that they are not overlapping !!

                Maybe it will be easier just to zip and attach all available *.sf files from FB Client and FB Server logs folders ?
  • SystemAdmin
    SystemAdmin
    68 Posts
    ACCEPTED ANSWER

    Re: FBSS5007W Failed to complete procedure on sectors

    ‏2010-11-12T15:14:54Z  in response to PKrogh
    Hi,

    Sorry, I know there had already been a few iteration on this, and Michael, the guy who answered from IBM, will now be out of office for the next 2 weeks.

    I looked in the server logs from the last post, there wasn't even one abort with the error event: "Failed to complete procedure on sectors". However there are a few aborts with the error event: "FBSS5002W Pre consistency-point script timeout on okprd1613". From those snapshots, for only one (1489) I could find both client and server logs in the posts. It looks like some timeout issuebetween client and server but not exactly clear yet.

    Can you provide please complete client + server logs (all .sf and .txt files from both client and server log folders). It important to send also the .txt files which are the configuration files.

    Thanks,

    Uri Wolloch
    Software Arcitect - Tivoli Storage manager FastBack
  • SystemAdmin
    SystemAdmin
    68 Posts
    ACCEPTED ANSWER

    Re: FBSS5007W Failed to complete procedure on sectors

    ‏2010-11-14T08:14:08Z  in response to PKrogh
    Hi,

    In order to do a thorough survey of your environment, I'm still missing the server configuration files: history.txt, state.txt, inprogress.txt, conf.txt, orabr_conf.txt, XpressClients.txt, FXVersion.txt, vminfo.txt + Locations.ini (this is a hidden system file, so you may not see it at first). And also the customer logs (clog*.sf|)
    From what I see so far:

    • You have a dedupe repository.
    • you may start around 21 snapshots in parallel.
    • Not sure if you changed the default number of simultaneous fulls, but you should not change in a dedupe repository.

    Conclusion:
    This looks like a scalability issue. I would try to spread the policies to not run so many snapshots at the same time.
    If you want to improve performance, you may want to consider not using dedupe.

    I can maybe say more after I get all the files.

    Thanks,

    -Uri
    • PKrogh
      PKrogh
      11 Posts
      ACCEPTED ANSWER

      Re: FBSS5007W Failed to complete procedure on sectors

      ‏2010-11-14T10:14:51Z  in response to SystemAdmin
      Hi

      I have try running 1 snapshot for the OKPRD1613 E-drive, but same result!
      It's the same on Video930 M-Drive!

      New Log attached.

      Attachments

  • SystemAdmin
    SystemAdmin
    68 Posts
    ACCEPTED ANSWER

    Re: FBSS5007W Failed to complete procedure on sectors

    ‏2010-11-15T08:26:37Z  in response to PKrogh
    Hi,

    It looks like there is some issue in the dedupe repository. In order to debug I will need all the files in the sn folder (next to the server folder on the server side).

    As a general note, a system that is demanding in terms of I/O activity will probably work better with standard repository and not dedupe.

    Again, sorry for the multiple iterations...

    Thanks,

    -Uri
  • SystemAdmin
    SystemAdmin
    68 Posts
    ACCEPTED ANSWER

    Re: FBSS5007W Failed to complete procedure on sectors

    ‏2010-11-15T09:53:10Z  in response to PKrogh
    Hi,

    What we see here is basically a scalability issue with the FastBack deduplication service. This is relatively a demanding environment. I would suggest to change the repository to be a regular repository (non dedupe). Please bear in mind that even with a regular repository, FastBack enjoys excellent data reduction because of the nature of the block level incremental backup technology (as opposed to a traditional file level backup product).
    We are working to improve the scalability of the product as a whole and dedupe in particular in the coming FastBack releases. But for your environment to run stably in the current version, the best thing to do is to use a regular repository.

    Hope this helps,

    -Uri
  • SystemAdmin
    SystemAdmin
    68 Posts
    ACCEPTED ANSWER

    Re: FBSS5007W Failed to complete procedure on sectors

    ‏2010-11-15T09:56:46Z  in response to PKrogh
    One her thing that is possible to investigate ,although I wouldn't put too much hope into, is to see if the repository disk performs well enough. That can be easily done by running a utility like IOMeter, and check performance of random writes (try 64kb chunks as an example)

    -Uri
    • PKrogh
      PKrogh
      11 Posts
      ACCEPTED ANSWER

      Re: FBSS5007W Failed to complete procedure on sectors

      ‏2010-11-15T10:48:27Z  in response to SystemAdmin
      Is it possible to go from dedupe to not-dedupe with the existing repository without destroy it?
  • SystemAdmin
    SystemAdmin
    68 Posts
    ACCEPTED ANSWER

    Re: FBSS5007W Failed to complete procedure on sectors

    ‏2010-11-15T12:00:43Z  in response to PKrogh
    Hi,

    Unfortunately there is no migration process between the two types of repositories. I saw you are also running DR, so you can keep the DR repository until you have new backups with the new repository. That way you stay protected and you have the DR data in case you need to restore.

    Thanks,

    -Uri
    • PKrogh
      PKrogh
      11 Posts
      ACCEPTED ANSWER

      Re: FBSS5007W Failed to complete procedure on sectors

      ‏2010-11-15T13:16:51Z  in response to SystemAdmin
      What is the best solution for the least possible reconfiguration where we can retain most possible configuration? if we choose to turn it into non-dedupe.
  • SystemAdmin
    SystemAdmin
    68 Posts
    ACCEPTED ANSWER

    Re: FBSS5007W Failed to complete procedure on sectors

    ‏2010-11-16T14:05:07Z  in response to PKrogh
    Hi,

    What you can do is to log in to FastBack manager in technician mode (user name: technician, pasword same as admin 9default admin123). Please don't use this mode normally!

    Go to "General Configuration"->"Technician options" and press "Delete repository". (!) This will delete all the data from your repository, but it will leave the configuration in tact.

    After that logout, login back as admin. And then remove the repository and add a new one (non dedupe).

    This should make the system work much better, but bear in mind that polices that backup 20 volumes in parallel are still not recommend. I would spread it much thinner.

    Hope this will help,

    -Uri