IC SunsetThe developerWorks Connections platform will be sunset on December 31, 2019. On January 1, 2020, this community and its apps will no longer be available. More details available on our FAQ.
Topic
  • 6 replies
  • Latest Post - ‏2019-09-25T13:53:04Z by PabloTo
PabloTo
PabloTo
35 Posts

Pinned topic RSync in an RPGLE program

‏2017-06-02T15:06:13Z |

Hi all, 

I love this computer!

Recently I need to syncronize  the stream file between two IBM I system and I try with one of the new tools of the OPS product: RSync.

I was happy of this solution so I want to share with you this utility, I made a demo version I hope this works well.

I hope it will be useful for someone.

I create two table, the first one which contain the list of the directory to syncronize:

Create or replace Table DemoIFS0f
  (Dataora Timestamp Not Null With Default Current Timestamp,
   Directory Varchar(1024) Not Null With Default,
   FlagAnn    char(1) Not null with default,
   DataOraUpd Timestamp for each row on update
                   as row change timestamp not null ,
  Primary key (Directory))
  Rcdfmt DemoIfs;

Label On Table DemoIfs0f Is
     'Directory to sincronie' ;

Label On Column DemoIfs0f
  (Dataora Is 'Input               Timestamp',
   Directory Is 'Directory ',
   FlagAnn Is 'Flag di             Annullamento',
   DataOraUpd is 'Update Timestamp');

 

the second table contain the list of exclusion for every directory:

Create or replace Table  DemoExcl1f
  (Dataora    Timestamp     Not Null With Default Current Timestamp,
   StmftoExcl Varchar(1024) Not Null With Default,
   FlagAnn    char(1)       Not null with default,
   DataOraUpd Timestamp for each row on update
                   as row change timestamp not null ,
  Primary key (StmftoExcl))
  Rcdfmt DemoIfs;

Label On Table  DemoExcl1f Is
     'File to exclude from Rsync' ;

Label On Column DemoDat / DemoExcl1f
  (Dataora    Is 'Input               Timestamp',
   StmftoExcl is 'Stmf to             Exclude  ',
   FlagAnn    Is 'Flag di             Annullamento',
   DataOraUpd is 'Update Timestamp');

 

Then I make two program for the syncronization:

**FREE
// 
//                                                  
//
// Read DEMOIFS0F ed make RSync on remote server
                                                             
Ctl-Opt DftActGrp(*No)
        ActGrp(*StgMdl)
        StgMdl(*TeraSpace)
        BndDir( 'QC2LE' : 'AL400MNUV2')
        DatFmt(*Iso) TimFmt(*Iso)
        Alwnull(*UsrCtl)
        Option(*SrcStmt:*NoDebugIo :*NoUnRef)
        Debug;

// Prototipi
Dcl-Pr DemoIfs01r ExtPgm('DEMOIFS01R');
End-Pr;

Dcl-Pi DemoIfs01r;
End-Pi;

Dcl-Pr DemoExcl1r ExtPgm('DEMOEXCL1R');
  Directory Varchar(1024) Const;
  Rc        Int(5);
End-Pr;

dcl-pr RSyncpase char(400);
       directory varchar(1024);
End-pr RSyncpase;

 // Set ERRNO                              
Dcl-Pr Qp2Shell ExtPgm('QP2SHELL');
       parm1 char(32765) Const Options(*VarSize :*trim);
       parm2 Char(32765) Const Options(*VarSize :*trim);
       parm3 Char(32765) Const Options(*VarSize :*trim);
End-Pr Qp2Shell;

Dcl-s parm1 VarChar(32765) inz('/QOpenSys/QIBM/ProdData/OPS/tools/bin/-bash');
Dcl-s parm2 VarChar(32765) inz('-c');
Dcl-s parm3 VarChar(32765);


dcl-s PaseRSync       char(32767) inz;
Dcl-s Directory       VarChar(1024);
Dcl-s ContatoreStampe Int(10);
Dcl-s Rc              Int(5);
Dcl-C  A          '''';

*inlr = *on;
parm1 = Parm1 + X'00';
parm2 = Parm2 + X'00';

Exec Sql Set Option DatFmt = *Iso, TimFmt = *Iso, CloSqlCsr = *EndMod,
             Commit = *none;

Exec Sql Declare DemoIfs cursor for
  Select Directory from DemoIfs0f where FlagAnn = '';

Exec Sql Open DemoIfs;

Dow SqlState <> '02000';

  Exec Sql Fetch next from DemoIfs into :Directory;

  If SqlState = '02000';
    Leave;
  EndIf;

  // PASE command line (Call qp2shell) this command as to be made
  //  with qsecofr to transmit at to remote system the ownership of the stream files
  //  replace remotesytem with your system name
  Parm3 = 'rsync -avRqhpe "ssh -oGlobalKnownHostsFile=+
            /home/ssh_global_known_hosts/known_hosts +
           -oUserKnownHostsFile=/home/ssh_global_known_hosts/known_hosts +
           -oIdentityFile=/home/ssh_global_known_hosts/id_rsa_d1"  ' +
           directory + ' qsecofr@remotesystem:/ --delete ';

  // generate the list of the stream file you want to exclude
  DemoExcl1r(Directory :Rc);

  Select;
    // Error in exclusion list generation abort
    When Rc = -1;
      Dsply 'Error in list exclusion generation';
      Return;

    // if there are not error I just add the exclusion list
    When Rc = 0;
      Parm3 += ' --exclude-from '+A+ '/utemp/RsyncExcludelist.txt' +A+ X'00';


    // with rc = -2 there are no exclusion for this directory
    When Rc = -2;
      Parm3 += X'00';

  EndSl;

  Qp2Shell(Parm1 :parm2 :parm3);

EndDo;

Exec Sql Close DemoIfs;

Exec Sql Select Count(*) Into :ContatoreStampe
          from Qsys2.Output_Queue_Entries
          where Job_name = qsys2.Job_name;

// If there some error in Rsync command the system generate a spool file 
If ContatoreStampe > 0;
  Dsply 'Error in Rsync look at spool file';
EndIf;

Return;

**FREE
// Read DEMOEXCL0F and write a stream file in /tmp/RsyncExcludelist.txt
//  which contain the list of stream file to exclude
           
Ctl-Opt DftActGrp(*No)
        ActGrp(*StgMdl)
        StgMdl(*TeraSpace)
        BndDir( 'QC2LE' )
        DatFmt(*Iso) TimFmt(*Iso)
        Alwnull(*UsrCtl)
        Option(*SrcStmt:*NoDebugIo :*NoUnRef)
        Debug;

Dcl-Pr DemoExcl1r ExtPgm('DEMOEXCL1R');
  Directory Varchar(1024) Const;
  Rc        Int(5);
End-Pr;

Dcl-Pi DemoExcl1r;
  Directory Varchar(1024) Const;
  Rc        Int(5);
End-Pi;

Dcl-s  ExcludeList VarChar(1024) ;

Dcl-s  StmfField  SqlType(Clob: 10000000) CcsId( 1208);

Dcl-s  SqlString  VarChar(1000);
Dcl-s  StmfToExcl SqlType(Clob_File) CCsId(1208);
Dcl-s  IdxExcl    Int(5);
Dcl-C  StmfName   '/tmp/RsyncExcludelist.txt';
Dcl-C  A          '''';
Dcl-s  Crlf       Char(1) Inz(X'0D');
Dcl-s  Exclusion  Ind;

//                                                  

*inlr = *on;

Exec Sql Set Option DatFmt = *Iso, TimFmt = *Iso, CloSqlCsr = *EndMod,
             Commit = *none;

StmfToExcl_Name = StmfName;
StmfToExcl_NL = %Len(StmfName);
StmfToExcl_FO = SQFOVR; // OverWrite
StmfField_len  = 0;

SqlString = 'Select StmfToExcl +
            from DemoExcl1f +
            where FlagAnn = ' +A+ ' ' +A+
            ' and StmfToExcl like ' +A+ Directory + '%' +A;

Exec Sql Prepare Stmfprep from :SqlString;

Exec Sql Declare StmfCur cursor for StmfPrep;

Exec Sql Open StmfCur;

Dow SqlState <> '02000';

  Exec Sql Fetch next from StmfCur into :ExcludeList;

  If SqlState = '02000';
    Leave;
  EndIf;

  Exclusion = *on;

  If Stmffield_len = 0;
    StmfField_Data = ExcludeList + CrLf;
  Else;
    StmfField_Data = %Subst(StmfField_Data :1 :StmfField_len) + ExcludeList + CrLf;
  EndIf;
  StmfField_len  = StmfField_len + %len(ExcludeList) + 1;

EndDo;

Exec Sql Close StmfCur;

If Exclusion = *Off;
  Rc = -2;
  Return;
EndIf;

Exec Sql Set :StmfToExcl = :StmfField;

If SqlState <> '00000';
  Rc = -1;
  Return;
EndIf;

Rc = 0;

Return;

 

I hope i'll like it.

Bye 

 

Updated on 2017-06-02T15:25:03Z at 2017-06-02T15:25:03Z by PabloTo
  • _vlad_
    _vlad_
    87 Posts

    Re: RSync in an RPGLE program

    ‏2017-06-10T21:53:29Z  

    Can you publish it on github or bitbucket?

    Among other benefits, it will give the users a clear message regarding the license under which the code is released

    Thanks

  • DidierDecath
    DidierDecath
    2 Posts

    Re: RSync in an RPGLE program

    ‏2019-05-22T12:45:23Z  

    Hello Pablo To,

    Can you confirm that you still use the command rsync inside RPGLE ? 

    If yes, have you any updates since you wrote this very interesting document ?

     

    Best regards,

     

    Didier   

     

    sample :

    rsync -dvIe ssh source_path/*                

    profile@remote server:/remote_path                
    building file list ... done                                                  
    xxxx                                                                    
    xxxx                                                                     
    xxxx                                                                    
                                                                                 
    sent 210 bytes  received 121 bytes  220.67 bytes/sec                         
    total size is 5,106  speedup is 15.43    

                                                               

  • PabloTo
    PabloTo
    35 Posts

    Re: RSync in an RPGLE program

    ‏2019-05-23T16:00:37Z  

    Hello Pablo To,

    Can you confirm that you still use the command rsync inside RPGLE ? 

    If yes, have you any updates since you wrote this very interesting document ?

     

    Best regards,

     

    Didier   

     

    sample :

    rsync -dvIe ssh source_path/*                

    profile@remote server:/remote_path                
    building file list ... done                                                  
    xxxx                                                                    
    xxxx                                                                     
    xxxx                                                                    
                                                                                 
    sent 210 bytes  received 121 bytes  220.67 bytes/sec                         
    total size is 5,106  speedup is 15.43    

                                                               

    Hi Didier,

    Yes, I continue use rsync inside RPGLE, I have a scheduled job than every day sync some directory from my production system to the demo system.

    I'd like to make a little rpgle utilily. I hope I can make it soon.

    Bye

  • PabloTo
    PabloTo
    35 Posts

    Re: RSync in an RPGLE program

    ‏2019-09-10T13:14:45Z  

    Hello Pablo To,

    Can you confirm that you still use the command rsync inside RPGLE ? 

    If yes, have you any updates since you wrote this very interesting document ?

     

    Best regards,

     

    Didier   

     

    sample :

    rsync -dvIe ssh source_path/*                

    profile@remote server:/remote_path                
    building file list ... done                                                  
    xxxx                                                                    
    xxxx                                                                     
    xxxx                                                                    
                                                                                 
    sent 210 bytes  received 121 bytes  220.67 bytes/sec                         
    total size is 5,106  speedup is 15.43    

                                                               

    Hi,

    I develop a beta version of Rsync in RPGLE, would you like to be the beta tester?

    I run it to replicate my ifs from twto IBM I, and it works. 

    Tell me if you want try.

    Bye

  • DidierDecath
    DidierDecath
    2 Posts

    Re: RSync in an RPGLE program

    ‏2019-09-25T13:21:24Z  
    • PabloTo
    • ‏2019-09-10T13:14:45Z

    Hi,

    I develop a beta version of Rsync in RPGLE, would you like to be the beta tester?

    I run it to replicate my ifs from twto IBM I, and it works. 

    Tell me if you want try.

    Bye

    Hi,

    I have received your utility but i can't restore the savf on v7r2.

    On my side, I am using rsync to replace nfs between iseries and aix servers for flow exchanges.

    Not far of the end Wink

     

    Bye

  • PabloTo
    PabloTo
    35 Posts

    Re: RSync in an RPGLE program

    ‏2019-09-25T13:53:04Z  

    Hi,

    I have received your utility but i can't restore the savf on v7r2.

    On my side, I am using rsync to replace nfs between iseries and aix servers for flow exchanges.

    Not far of the end Wink

     

    Bye

    Hi, I put the savf in V7R2. please try.

    Bye