Topic
  • 3 replies
  • Latest Post - ‏2012-06-29T11:20:22Z by wellseb
SystemAdmin
SystemAdmin
926 Posts

Pinned topic COPY successful but file not available immediately

‏2012-01-26T22:16:45Z |
How long after a C:D process that includes a copy step should the file that was transferred actually show up on the file system??

We've been under the impression that the process itself wouldn't proceed beyond a successful COPY step till it is complete

The step in question retrieves a file from a WIntel server onto the local UNIX
....
MYGET COPY
FROM (
FILE="&remoteOutputFile"
SNODE
SYSOPTS="DATATYPE(BINARY)"
)
TO (
FILE="&localOutputFile"
PNODE
DISP=RPL
SYSOPTS=":datatype=binary:"
)
...

The size of the file dictates how long it takes to "show up", necessitating checks such as the following before proceeding:

...
let ZZ=600;
let SNOOZE=20;
let nn=$ZZ/$SNOOZE;
while ! -f $MYFILE -a 1 -eq $(( $nn > 0 ))
do
let nn--
sleep $SNOOZE
done
if -f $MYFYLE
then
echo yay
else
echo boo hoo
fi
...


Should this even be necessary? We've never required such checks on WIntel nor z/OS with even incredibly large files. As soon as any COPY step has ended, the files are always available on the target system. I'm guessing possibly some latency on the file system maybe?

The "uname -a" output of the Unix server is:

SunOS myserver 5.10 Generic_144488-17 sun4u sparc SUNW,Sun-Fire-15000


Rahul
Updated on 2012-06-29T11:20:22Z at 2012-06-29T11:20:22Z by wellseb
  • SystemAdmin
    SystemAdmin
    926 Posts

    Re: COPY successful but file not available immediately

    ‏2012-02-22T15:14:31Z  
    No, there is no need for such checks !
    If you want to do something using the transferred file,
    use IF-THEN_ELSE logic within your process

    MYGET COPY
    FROM (
    FILE="&remoteOutputFile"
    SNODE
    SYSOPTS="DATATYPE(BINARY)"
    )
    TO (
    FILE="&localOutputFile"
    PNODE
    DISP=RPL
    SYSOPTS=":datatype=binary:"
    )

    IFSTEP IF (MYGET EQ 0) THEN
    RTASKOK RUN TASK
    SYSOPTS="echo yay"

    ELSE

    RTASKNOK RUN TASK
    SYSOPTS="echo boo hoo"

    EIF

    Sasha
  • SystemAdmin
    SystemAdmin
    926 Posts

    Re: COPY successful but file not available immediately

    ‏2012-02-22T16:17:34Z  
    No, there is no need for such checks !
    If you want to do something using the transferred file,
    use IF-THEN_ELSE logic within your process

    MYGET COPY
    FROM (
    FILE="&remoteOutputFile"
    SNODE
    SYSOPTS="DATATYPE(BINARY)"
    )
    TO (
    FILE="&localOutputFile"
    PNODE
    DISP=RPL
    SYSOPTS=":datatype=binary:"
    )

    IFSTEP IF (MYGET EQ 0) THEN
    RTASKOK RUN TASK
    SYSOPTS="echo yay"

    ELSE

    RTASKNOK RUN TASK
    SYSOPTS="echo boo hoo"

    EIF

    Sasha
    Thanks, Sasha.

    Actually, soon after I had posted my message I began to suspect what was going on. There are 2 transfers involved, from the same PNODE to different SNODEs invoked from within the same shell script.

    We incorrectly assumed the calls to "direct" using the using here-doc (<<) syntax would be synchronous and finish only after the C:D process is done.

    I'm including here the gist of what was happening, to benefit others in the future.
    #

    #!/usr/bin/ksh
    1. ...
    2. bunch of declarations
    3. ...
    /opt/cdunix/ndm/bin/direct <<EOF
    sub bla1 process snode=$SNODE1
    MYGET COPY
    FROM (
    FILE="&remoteOutputFile"
    SNODE
    SYSOPTS="DATATYPE(BINARY)"
    )
    TO (
    FILE="&localOutputFile"
    PNODE
    DISP=RPL
    SYSOPTS=":datatype=binary:"
    )
    pend;
    quit;
    EOF

    1. the above "direct" simply submits the process listing and returns
    2. that's why the next step doesn't work always for large files
    3. and that's why the snooze logic was inserted

    /opt/cdunix/ndm/bin/direct <<EOF
    sub sendit process snode=$SNODE2
    MYPUT copy
    from (file=$localOutputFile
    pnode
    sysopts=":datatype=binary:"
    )
    to (file=$DSNMF
    snode
    disp=new
    )
    pend;
    quit;
    EOF

    #

    There are several ways to address this, including
    (1) One long "sleep" between calls to "direct"
    (2) looping short "sleep"s till the file is detected
    (3) Submitting the 2nd CDP, moved to a completely separate script, invoked via RUN TASK upon successful completion of the MYCOPY task

    ~Rahul
  • wellseb
    wellseb
    1 Post

    Re: COPY successful but file not available immediately

    ‏2012-06-29T11:20:22Z  
    Thanks, Sasha.

    Actually, soon after I had posted my message I began to suspect what was going on. There are 2 transfers involved, from the same PNODE to different SNODEs invoked from within the same shell script.

    We incorrectly assumed the calls to "direct" using the using here-doc (<<) syntax would be synchronous and finish only after the C:D process is done.

    I'm including here the gist of what was happening, to benefit others in the future.
    #

    #!/usr/bin/ksh
    1. ...
    2. bunch of declarations
    3. ...
    /opt/cdunix/ndm/bin/direct <<EOF
    sub bla1 process snode=$SNODE1
    MYGET COPY
    FROM (
    FILE="&remoteOutputFile"
    SNODE
    SYSOPTS="DATATYPE(BINARY)"
    )
    TO (
    FILE="&localOutputFile"
    PNODE
    DISP=RPL
    SYSOPTS=":datatype=binary:"
    )
    pend;
    quit;
    EOF

    1. the above "direct" simply submits the process listing and returns
    2. that's why the next step doesn't work always for large files
    3. and that's why the snooze logic was inserted

    /opt/cdunix/ndm/bin/direct <<EOF
    sub sendit process snode=$SNODE2
    MYPUT copy
    from (file=$localOutputFile
    pnode
    sysopts=":datatype=binary:"
    )
    to (file=$DSNMF
    snode
    disp=new
    )
    pend;
    quit;
    EOF

    #

    There are several ways to address this, including
    (1) One long "sleep" between calls to "direct"
    (2) looping short "sleep"s till the file is detected
    (3) Submitting the 2nd CDP, moved to a completely separate script, invoked via RUN TASK upon successful completion of the MYCOPY task

    ~Rahul
    Hi Rahul,

    Although not explicitly noted in the Connect:Direct UNIX documentation (however it is mentioned in the documentation for other environments), we found that using the "maxdelay=0" parameter solved our problem:

    
    . $NDMBIN/ndmcli -x << EOJ submit maxdelay=0 $procname process snode=$
    {remote_ip
    } snodeid=($
    {pqid
    }) step1 copy from (snode file=$
    {remote_file
    } disp=(shr) )   to   (pnode file=$
    {local_file
    } disp=(rpl) sysopts=
    "$sysopts" )   pend ; # end submit and eoj out of ndmcli EOJ