Topic
20 replies Latest Post - ‏2014-10-10T15:57:19Z by Asim80
SystemAdmin
SystemAdmin
6772 Posts
ACCEPTED ANSWER

Pinned topic SOMA - automated backup of domains - downloading files to where?

‏2009-04-16T16:29:49Z |
I'm attempting again to use SOMA for domain backups. I will hopefully place the XML requests on a Unix machine to have them be cron job capable. Otherwise, I know no other way of automating domain backups. Does anyone know of any other way through SOMA to automate domain backups, say weekly? I've got this request below calling the 2004 version so I can have multiple actions in one request element, all with unique names.

If they do not have unique names, and I can't download them somewhere, say an NFS share or other Windows machine somewhere, what's the point of 'backup' ?



<?xml version="1.0" encoding="UTF-8"?>
<env:Envelope xmlns:env="http://schemas.xmlsoap.org/soap/envelope/">
<env:Body>
<dp:request xmlns:dp="http://www.datapower.com/schemas/management">
<dp:do-backup format="ZIP">
<dp:user-comment>DEV_ADI backup</dp:user-comment>
<dp:domain name="DEV_ADI"/>
</dp:do-backup>
<dp:do-backup format="ZIP">
<dp:user-comment>DEV_CGI backup</dp:user-comment>
<dp:domain name="DEV_CGI"/>
</dp:do-backup>
</dp:request>
</env:Body>
</env:Envelope>
Updated on 2013-02-08T16:02:31Z at 2013-02-08T16:02:31Z by HermannSW
  • zachahuy83
    zachahuy83
    45 Posts
    ACCEPTED ANSWER

    Re: SOMA - automated backup of domains - downloading files to where?

    ‏2009-04-16T21:57:18Z  in response to SystemAdmin
    Hey kpc,

    I have tried this before and attach is the write up doc of this app. It backup to a FTP server and use the schedule rule to automate this.
    • zachahuy83
      zachahuy83
      45 Posts
      ACCEPTED ANSWER

      Re: SOMA - automated backup of domains - downloading files to where?

      ‏2009-04-16T21:59:09Z  in response to zachahuy83
      One thing you want to be careful when backing up is you dont want to backup too many domain at once. There is a known problem when ftping a large file. So break it down to several different backup calls e.g. first backup call backup domain a, b, c then ftp off device, second backup backup d, e, f...etc.
      • SystemAdmin
        SystemAdmin
        6772 Posts
        ACCEPTED ANSWER

        Re: SOMA - automated backup of domains - downloading files to where?

        ‏2009-04-17T14:01:32Z  in response to zachahuy83
        Thanks zachahuy83
        I'm gonna use the doc and try it out. Very much appreciated. I wonder why the schema to soma, or amp, just doesn't have the operations needed to make this an easier process. Perhaps I'm missing something. Regardless I will use this process you have listed. If it works it works :)
        • zachahuy83
          zachahuy83
          45 Posts
          ACCEPTED ANSWER

          Re: SOMA - automated backup of domains - downloading files to where?

          ‏2009-04-17T19:00:17Z  in response to SystemAdmin
          Btw, can you also give me some feedback regarding how the article is written. It got accepted by developWork, but I just have not have time to get some peer review on it and convert it to XML and post it. Any constructive feedback would help. Thanks.
          • SystemAdmin
            SystemAdmin
            6772 Posts
            ACCEPTED ANSWER

            Re: SOMA - automated backup of domains - downloading files to where?

            ‏2009-04-17T19:21:56Z  in response to zachahuy83
            Definitely. I'll get back to you specifically as soon as I've put it in place. It's really well thought out, that's for sure, and the fact that it can work internally or via browser is pretty snazzy.
            • SystemAdmin
              SystemAdmin
              6772 Posts
              ACCEPTED ANSWER

              Re: SOMA - automated backup of domains - downloading files to where?

              ‏2009-04-22T20:02:43Z  in response to SystemAdmin
              Works well! It can span across all domains too! I have set it up so that a different XSLT is used to backup a different domain, each one of them in a different request rule, all in the same firewall. The browser calls add files to the location, while if left only to the scheduling processing policy, the files overwrite themselves, which I like.

              An internal, auto-backup service covering all domains, and having naming control with different backup zip files per domain. Exactly what I wanted. Good stuff... Thanks!

              -Kirby
              • SystemAdmin
                SystemAdmin
                6772 Posts
                ACCEPTED ANSWER

                Re: SOMA - automated backup of domains - downloading files to where?

                ‏2009-04-28T20:29:06Z  in response to SystemAdmin
                Hi,

                Thanks for sharing the doc. I'm testing xml FW from the browser. I'm got the below error

                url-open: Remote error on url 'ftp://xx.xx.xx.xx/czaqdp1Backup2009-04-28T13a.zip'

                I'm using the FileZilla Server for testing FTP server. I have configured and checked that the user and permission on FileZilla server are correct.

                What could be the reason?

                Can we use sftp instead of ftp? Does new version DataPower XI50.3.7.2.0 Supports SFTP ?

                Thanks,
                Raj
                • SystemAdmin
                  SystemAdmin
                  6772 Posts
                  ACCEPTED ANSWER

                  Re: SOMA - automated backup of domains - downloading files to where?

                  ‏2009-04-28T21:44:22Z  in response to SystemAdmin
                  Raj

                  I've attached the xsl file I use. Pay attention to the following line in the file:

                  <!-- Extract the export content string -->
                  <xsl:variable name="encBackUpFile" select="string($BackUpFile//mgmt:file)"/>

                  Make sure the //mgmt:file is exactly like that. I replaced it with //dp:file earlier on and it did not work remotely. I was getting the error of "could not open file" if that helps. Otherwise, I have had no hangups with the xsl file. Check your FTP filesystem for permissions of course. Once it works in one domain, you can add request rules with appropriate matches to span across domains.

                  -Kirby
                  • SystemAdmin
                    SystemAdmin
                    6772 Posts
                    ACCEPTED ANSWER

                    Re: SOMA - automated backup of domains - downloading files to where?

                    ‏2009-04-28T21:54:59Z  in response to SystemAdmin
                    Raj

                    I've actually attached the file this time.

                    -K
                    • SystemAdmin
                      SystemAdmin
                      6772 Posts
                      ACCEPTED ANSWER

                      Re: SOMA - automated backup of domains - downloading files to where?

                      ‏2009-04-29T22:20:17Z  in response to SystemAdmin
                      Hi KPC,

                      Thanks for the xslt file. I have replaced the hostnames and port for ftp and xml mgmt servers in the xslt file. I'm getting the below error. Any idea?

                      0x80e00040 xmlfirewall (BackupService): url-open: Remote error on url 'ftp://ftphostname/Downloads/SSGTestDomain_2009-04-29T18.zip'
                      0x80e00040 xmlfirewall (BackupService): url-open: Remote error on url 'https://xx.xx.xx.xx:5550/service/mgmt/2004'
                      • SystemAdmin
                        SystemAdmin
                        6772 Posts
                        ACCEPTED ANSWER

                        Re: SOMA - automated backup of domains - downloading files to where?

                        ‏2009-04-29T23:11:00Z  in response to SystemAdmin
                        Not sure Raj on those errors alone. Check your request inside the xslt (well-written by zachahuy83) and verify datapower can ping the ftp server and not just your desktop machine, would be one guess.
                        Otherwise, if the export works locally on the datapower, it's a matter of something between you and the ftp server from the datapower would be my guess.
                      • DaveHubbard
                        DaveHubbard
                        135 Posts
                        ACCEPTED ANSWER

                        Re: SOMA - automated backup of domains - downloading files to where?

                        ‏2009-04-30T10:28:57Z  in response to SystemAdmin
                        RajDN

                        I might suggest you check your default domain log - it may be that you getting a security error connecting back into the device.

                        You may need to check how your mgmt interface is protected and that the URI (2004) is enabled. Then possibly use a user-agent (in your client XMLManager) to supply required credentials on the call.

                        My further suspicion might be that the ftp would fail as a result of the mgmt call failing i.e no zip produced.

                        Just a thought.

                        Dave
                        • go_ram
                          go_ram
                          1 Post
                          ACCEPTED ANSWER

                          Re: SOMA - automated backup of domains - downloading files to where?

                          ‏2010-12-10T18:04:49Z  in response to DaveHubbard
                          Has anyone found a solution this error of the Ftp failing. Getting the same error. I checked the appliance and was able to ping the ftp server and connectivity is good. I see the datpower has logged in to this.
                          The export was also created in the device in the export directory. But the file is not getting ftped.
    • Asim80
      Asim80
      19 Posts
      ACCEPTED ANSWER

      Re: SOMA - automated backup of domains - downloading files to where?

      ‏2014-10-10T15:57:19Z  in response to zachahuy83

      This is good write up. If we use 'all-domains' in the do-export call, will it take too long in a situation where you have about 25 domains? I haven't tried it yet, wanted to get an advise first. Thanks 

  • SystemAdmin
    SystemAdmin
    6772 Posts
    ACCEPTED ANSWER

    Re: SOMA - automated backup of domains - downloading files to where?

    ‏2012-03-20T09:06:35Z  in response to SystemAdmin
    hi all,

    based on the ideas presented here and the document by zachahuy83, i had developed an automated backup solution. now that i've some time, i'd like to share the solution.

    please find the attached export of an XML Firewall service named BackupDomains, and referenced objects and files.

    hope it helps.

    --

    • the solution utilizes SOMA and AMP interfaces of DataPower appliance.
    • you can specify as many appliances to be backed up.
    • you can choose to backup an entire device or be selective on domains.
    • you can use ftp or sftp to offload the backup files.

    --

    xml firewall policy uses a configuration file (local://backup/config/backup-config.xml) where you define the appliance(s) and the destination for backup files. the configuration file itself is documented enough. basically,
    • write the xml management interface's ip address and port per appliance (port can be configured globally if common)
    • determine if an entire or domain-selective backup desired per appliance (and list the interested domains)
    • specify the type of server for backup storage: ftp or sftp
    • detail the connection and directory parameters of storage server: ip, port (optional), backup directory and user credentials (optional)

    --

    other points:
    • make sure that XML Management Interface enabled on each device with SOMA and AMP services
    • edit the SSL Proxy Profile Policy setting of the referenced User Agent (Backup) to set secure connection to appliances: just add SSL certificates of devices (used on XML Management Interface endpoint) into the referenced Validation Credentials (Backup)
    • edit Basic-Auth Policy of the referenced User Agent (Backup) accordingly to authenticate to the XML Management Interfaces of appliances: add credentials of users on each device used to get a backup
    • edit SFTP Client Policies setting of the referenced User Agent (Backup) and set the authentication method and the credentials for SFTP server to store the backup files: edit the referenced SSH Client Profile (Backup). no need for any SFTP C. Policy if FTP is preferred over SFTP.
    • alternatively, SFTP user credentials can be specified in the configuration file rather than configuring an SFTP C. Policy
    • if the storage method is FTP and no user is specified in the configuration file, make sure that the server supports anonymous login.
    • used XSL files can be found under the directory local://backup/xsl: backup.xsl and backup-utils.xsl
    • SystemAdmin
      SystemAdmin
      6772 Posts
      ACCEPTED ANSWER

      Re: SOMA - automated backup of domains - downloading files to where?

      ‏2013-02-07T15:16:25Z  in response to SystemAdmin
      Thanks Teqfoo. Does this preform a normal "backup" or secure backup? I am wondering if keys, certs, and files are backed up as they are with a Secure Backup? Or if it's just the domain configs, like a normal backup.
      • HermannSW
        HermannSW
        4155 Posts
        ACCEPTED ANSWER

        Re: SOMA - automated backup of domains - downloading files to where?

        ‏2013-02-07T15:22:06Z  in response to SystemAdmin
        Looking into the stylesheet local/backup/backup.xsl of attached config shows that "normal" backups are done.
        So all config, but no crypto material will be backed up.

        For backing up crypto material your only choice is secure-backup, and that requires a reinit if not enabled on the device already.

         
        Hermann<myXsltBlog/> <myXsltTweets/>
        • SystemAdmin
          SystemAdmin
          6772 Posts
          ACCEPTED ANSWER

          Re: SOMA - automated backup of domains - downloading files to where?

          ‏2013-02-08T14:34:08Z  in response to HermannSW
          Yes, my devices have DR/secure backup mode enabled. I am wanting to automate backing up the entire device daily or weekly. I am wondering what the best practice is to accomplish that. From my little research, the easiest way appears to be creating a script on a server, in our case a unix server and a shell script. And having it SSH to the box, run CLI commands, and then copy the file to an off storage location.

          Any thoughts?
          • HermannSW
            HermannSW
            4155 Posts
            ACCEPTED ANSWER

            Re: SOMA - automated backup of domains - downloading files to where?

            ‏2013-02-08T16:02:31Z  in response to SystemAdmin
            Why do you want to copy off the device yourself.
            You can have the destination point remotely.
            That has the advantage, that no space for the full backup is needed on the device.

            
            xi50(config)# help secure-backup secure-backup <cert> <destination> [include-iscsi] [include-raid] Create a secure backup of the appliance that includes the config, certificates and associated files into a set of secure (encrypted) files. ... destination is the URL of a directory where the backup will be stored. Supported protocols are local:, temporary: and ftp: ... For example: secure-backup myCryptoCert temporary:
            ///myBackupDir/  Encrypts the backup to temporary:
            ///myBackupDir/ using myCryptoCert, with the iSCSI and RAID data included.   xi50(config)#
            


            Btw, you can alternatively do a request against XML management interface instead of having your script "ssh" into the box.
            From "store:///xml-mgmt.xsd":
            
            ... <!--Secure Backup--> <xsd:complexType name=
            "ActionSecureBackup"> <xsd:sequence> <xsd:element name=
            "cert" type=
            "tns:dmReference" minOccurs=
            "1" maxOccurs=
            "1" /> <xsd:element name=
            "destination" type=
            "tns:dmURL" minOccurs=
            "1" maxOccurs=
            "1" /> <xsd:element name=
            "include-iscsi" type=
            "tns:dmToggle" minOccurs=
            "0" maxOccurs=
            "1" /> <xsd:element name=
            "include-raid" type=
            "tns:dmToggle" minOccurs=
            "0" maxOccurs=
            "1" /> </xsd:sequence> </xsd:complexType> ...
            


             
            Hermann<myXsltBlog/> <myXsltTweets/>
    • zed145
      zed145
      1 Post
      ACCEPTED ANSWER

      Re: SOMA - automated backup of domains - downloading files to where?

      ‏2014-03-21T20:29:39Z  in response to SystemAdmin

      Hi SystemAdmin

      Your zip file doesn't open. Can you please verify.