Topic
3 replies Latest Post - ‏2012-06-29T11:20:22Z by wellseb
SystemAdmin
SystemAdmin
926 Posts
ACCEPTED ANSWER

Pinned topic COPY successful but file not available immediately

‏2012-01-26T22:16:45Z |
How long after a C:D process that includes a copy step should the file that was transferred actually show up on the file system??

We've been under the impression that the process itself wouldn't proceed beyond a successful COPY step till it is complete

The step in question retrieves a file from a WIntel server onto the local UNIX
....
MYGET COPY
FROM (
FILE="&remoteOutputFile"
SNODE
SYSOPTS="DATATYPE(BINARY)"
)
TO (
FILE="&localOutputFile"
PNODE
DISP=RPL
SYSOPTS=":datatype=binary:"
)
...

The size of the file dictates how long it takes to "show up", necessitating checks such as the following before proceeding:

...
let ZZ=600;
let SNOOZE=20;
let nn=$ZZ/$SNOOZE;
while ! -f $MYFILE -a 1 -eq $(( $nn > 0 ))
do
let nn--
sleep $SNOOZE
done
if -f $MYFYLE
then
echo yay
else
echo boo hoo
fi
...


Should this even be necessary? We've never required such checks on WIntel nor z/OS with even incredibly large files. As soon as any COPY step has ended, the files are always available on the target system. I'm guessing possibly some latency on the file system maybe?

The "uname -a" output of the Unix server is:

SunOS myserver 5.10 Generic_144488-17 sun4u sparc SUNW,Sun-Fire-15000


Rahul
Updated on 2012-06-29T11:20:22Z at 2012-06-29T11:20:22Z by wellseb
  • SystemAdmin
    SystemAdmin
    926 Posts
    ACCEPTED ANSWER

    Re: COPY successful but file not available immediately

    ‏2012-02-22T15:14:31Z  in response to SystemAdmin
    No, there is no need for such checks !
    If you want to do something using the transferred file,
    use IF-THEN_ELSE logic within your process

    MYGET COPY
    FROM (
    FILE="&remoteOutputFile"
    SNODE
    SYSOPTS="DATATYPE(BINARY)"
    )
    TO (
    FILE="&localOutputFile"
    PNODE
    DISP=RPL
    SYSOPTS=":datatype=binary:"
    )

    IFSTEP IF (MYGET EQ 0) THEN
    RTASKOK RUN TASK
    SYSOPTS="echo yay"

    ELSE

    RTASKNOK RUN TASK
    SYSOPTS="echo boo hoo"

    EIF

    Sasha
    • SystemAdmin
      SystemAdmin
      926 Posts
      ACCEPTED ANSWER

      Re: COPY successful but file not available immediately

      ‏2012-02-22T16:17:34Z  in response to SystemAdmin
      Thanks, Sasha.

      Actually, soon after I had posted my message I began to suspect what was going on. There are 2 transfers involved, from the same PNODE to different SNODEs invoked from within the same shell script.

      We incorrectly assumed the calls to "direct" using the using here-doc (<<) syntax would be synchronous and finish only after the C:D process is done.

      I'm including here the gist of what was happening, to benefit others in the future.
      #

      #!/usr/bin/ksh
      1. ...
      2. bunch of declarations
      3. ...
      /opt/cdunix/ndm/bin/direct <<EOF
      sub bla1 process snode=$SNODE1
      MYGET COPY
      FROM (
      FILE="&remoteOutputFile"
      SNODE
      SYSOPTS="DATATYPE(BINARY)"
      )
      TO (
      FILE="&localOutputFile"
      PNODE
      DISP=RPL
      SYSOPTS=":datatype=binary:"
      )
      pend;
      quit;
      EOF

      1. the above "direct" simply submits the process listing and returns
      2. that's why the next step doesn't work always for large files
      3. and that's why the snooze logic was inserted

      /opt/cdunix/ndm/bin/direct <<EOF
      sub sendit process snode=$SNODE2
      MYPUT copy
      from (file=$localOutputFile
      pnode
      sysopts=":datatype=binary:"
      )
      to (file=$DSNMF
      snode
      disp=new
      )
      pend;
      quit;
      EOF

      #

      There are several ways to address this, including
      (1) One long "sleep" between calls to "direct"
      (2) looping short "sleep"s till the file is detected
      (3) Submitting the 2nd CDP, moved to a completely separate script, invoked via RUN TASK upon successful completion of the MYCOPY task

      ~Rahul
      • wellseb
        wellseb
        1 Post
        ACCEPTED ANSWER

        Re: COPY successful but file not available immediately

        ‏2012-06-29T11:20:22Z  in response to SystemAdmin
        Hi Rahul,

        Although not explicitly noted in the Connect:Direct UNIX documentation (however it is mentioned in the documentation for other environments), we found that using the "maxdelay=0" parameter solved our problem:

        
        . $NDMBIN/ndmcli -x << EOJ submit maxdelay=0 $procname process snode=$
        {remote_ip
        } snodeid=($
        {pqid
        }) step1 copy from (snode file=$
        {remote_file
        } disp=(shr) )   to   (pnode file=$
        {local_file
        } disp=(rpl) sysopts=
        "$sysopts" )   pend ; # end submit and eoj out of ndmcli EOJ