Topic
  • 7 replies
  • Latest Post - ‏2012-11-28T15:07:40Z by Kostek
Kostek
Kostek
6 Posts

Pinned topic Problem with runnig in subshell

‏2012-11-24T16:21:25Z |
Hi All

I've got a script which sometimes runs ok and sometimes fails. I localized that the problem is with simple piece of code that uses subshell. Then I've created script that contains only those few lines and it fails(sometimes) as well.

This few lines remove trailing blanks from file in one step.

#!/bin/bash

cat file.txt | (rm file.txt ; touch file.txt ;
while read File_Line; do

echo ${File_Line} | sed 's/ *$//g' >> file.txt

done)

It sometimes fails with: cat: cannot open file.txt
The problem is that it not happens every time, few times it works fine and then is one time with error.

Do You know what is going on and what should be done to avoid this error?
AIX Version 6.1

cheers
kostek
Updated on 2012-11-28T15:07:40Z at 2012-11-28T15:07:40Z by Kostek
  • SystemAdmin
    SystemAdmin
    6902 Posts

    Re: Problem with running in subshell

    ‏2012-11-26T16:50:33Z  
    I'm not exactly sure why that's happening. My first thought without attempting to duplicate the problem is that it might be a race condition. With that said, I'd accomplish what you're trying to do using Perl.

    perl -wpi -e 's/\s+$//' file.txt
  • Kostek
    Kostek
    6 Posts

    Re: Problem with running in subshell

    ‏2012-11-26T23:32:23Z  
    I'm not exactly sure why that's happening. My first thought without attempting to duplicate the problem is that it might be a race condition. With that said, I'd accomplish what you're trying to do using Perl.

    perl -wpi -e 's/\s+$//' file.txt
    Thanks for post, I'll consider to use perl if no way out.

    Pleasy try to duplicate problem on You system, maybe then You will find what's up.

    I duplicated the same error on linux debian, so i think it's not the AIX problem but rather something with subshell. I know this is strange but in this case it seems like sometimes command 'rm' in subshell is executed before 'cat' and then cat doesn't see file... ?
  • Holgervk
    Holgervk
    56 Posts

    Re: Problem with runnig in subshell

    ‏2012-11-27T08:53:11Z  
    you are reading and writing to the same file
    this gives unpredictable results. always avoid this.

    you might do something like

    cp file.txt file2.txt
    cat file2.txt ...(rest of your code)
    rm file2.txt
  • The_Doctor
    The_Doctor
    98 Posts

    Re: Problem with runnig in subshell

    ‏2012-11-27T17:18:24Z  
    As previously stated, this is a classic race condition between 2 different processes, giving unpredictable results.

    The "cat" process is forked. The "rm" process is forked. These 2 forked processes run asynchronously.... not serially.... thus the unpredictable results.

    Change your code to remove the reliance on "rm" & "touch" and you'll be good to go. aka.... there are many ways to do accomplish what you want without "rm" & "touch".

    BTW.... your use of echo makes sed redundant. You can remove the pipe to sed & speed things up dramatically. Like I said.... many ways to do it, here's but one example:
    
    #!/bin/ksh > temp.txt                         # ensure our temp file is empty 
    
    while read File_Line 
    
    do   echo $
    {File_Line
    } >> temp.txt   # sed 
    's/ *$//g' ..... the string passed to sed is redundant # remove the pipe to sed and # avoid the fork of a 
    
    new sed process 
    
    for every line in the file # 1,000,000 lines = 1,000,000 forks = slow   done                  < file.txt   # or use the cat | format 
    
    if you don
    't like this   mv temp.txt file.txt               # destroy original file 
    
    if you wish
    
  • SystemAdmin
    SystemAdmin
    6902 Posts

    Re: Problem with runnig in subshell

    ‏2012-11-27T23:24:12Z  
    For some reason I can't reply to The_Doctor's post despite trying three different browsers. He's of course right, but at the risk of sounding pedantic, it's not the echo that strips the line, but the read. A quick change of IFS and you'll see that the read no longer strips off the trailing white space, e.g.

    $ cat print.bash
    #!/bin/bash

    IFS=':'
    read LINE < $1
    export LINE
    perl -le 'print $ENV{LINE}' | od -c
    $ od -c file.txt
    0000000 t h i s i s a t e s t
    0000020 \t \t \n
    0000024
    $ ./print.bash file.txt
    0000000 t h i s i s a t e s t
    0000020 \t \t \n
    0000024
    $ perl -wpi -e 's/IFS/#IFS/' print.bash
    $ ./print.bash file.txt
    0000000 t h i s i s a t e s t \n
    0000017
  • The_Doctor
    The_Doctor
    98 Posts

    Re: Problem with runnig in subshell

    ‏2012-11-27T23:48:02Z  
    For some reason I can't reply to The_Doctor's post despite trying three different browsers. He's of course right, but at the risk of sounding pedantic, it's not the echo that strips the line, but the read. A quick change of IFS and you'll see that the read no longer strips off the trailing white space, e.g.

    $ cat print.bash
    #!/bin/bash

    IFS=':'
    read LINE < $1
    export LINE
    perl -le 'print $ENV{LINE}' | od -c
    $ od -c file.txt
    0000000 t h i s i s a t e s t
    0000020 \t \t \n
    0000024
    $ ./print.bash file.txt
    0000000 t h i s i s a t e s t
    0000020 \t \t \n
    0000024
    $ perl -wpi -e 's/IFS/#IFS/' print.bash
    $ ./print.bash file.txt
    0000000 t h i s i s a t e s t \n
    0000017
    yeah, you're right

    I believe it's:

    • the read that removes the leading & trailing blanks from the complete line....
    • while the echo (without double quotes) takes any consecutive imbedded blanks and reduces then down to 1 blank

    For example:
    
    echo 
    "${File_Line}"           # retain any imbedded blanks with 
    
    double quotes, 
    
    while echo $
    {File_Line
    }            # the original echo takes 2 or more consecutive blanks & reduces them down to a single blank
    
  • Kostek
    Kostek
    6 Posts

    Re: Problem with runnig in subshell

    ‏2012-11-28T15:07:40Z  
    yeah, you're right

    I believe it's:

    • the read that removes the leading & trailing blanks from the complete line....
    • while the echo (without double quotes) takes any consecutive imbedded blanks and reduces then down to 1 blank

    For example:
    <pre class="jive-pre"> echo "${File_Line}" # retain any imbedded blanks with double quotes, while echo $ {File_Line } # the original echo takes 2 or more consecutive blanks & reduces them down to a single blank </pre>
    Thanks a lot guys

    I didn't know that system treats those commands like two independent processes.
    Thanks for performance tips as well.