Topic
14 replies Latest Post - ‏2012-09-28T23:28:46Z by outlaw
lzrycki
lzrycki
29 Posts
ACCEPTED ANSWER

Pinned topic COBOL for AIX - DDMEA Files

‏2012-08-10T12:41:14Z |
Hi people,

I start working upgrading COBOL programs from release 2 to 4. All my experience come from mainframe field.

Could someone explain me what is a DDMEA file, existing with COBOL 2 but not with COBOL 4.

What kind of filesystem must I declare to have the identical type of files in both COBOL version ? ( FILESYS(RSD) , FILESYS(RAW), etc )

Thanks in adveance

Leonardo
Buenos Aires
Argentina
Updated on 2012-09-28T23:28:46Z at 2012-09-28T23:28:46Z by outlaw
  • outlaw
    outlaw
    39 Posts
    ACCEPTED ANSWER

    Re: COBOL for AIX - DDMEA Files

    ‏2012-09-11T16:45:52Z  in response to lzrycki
    A DDMEA file contains metadata (file attributes), and is used by the SdU (Smart data Utilities) routines that provides the FILESYS(VSAM) support.

    Both v2 (no longer supported) and v4 support the VSAM file system, and it is the default FILESYS value.

    You don't say what level of v4 you are on, the latest is 4.1.1.6 (iirc), and it is highly recommended as it corrects many issues with several file systems (and improves I/O performance for some of them). Some of those items are very important to your issue.

    Note that forward compatibility(v2->v4) is promised, but backwards compatibility(v4->v2) is not - but we do our best to provide both, where possible.

    There are also very few cases, where to be standards compliant, behaviour has changed in a way that may cause issues in a migration such as this. Some recent examples:
    LINAGE support (all file systems)
    READ NEXT (STL)
    File Status 02 (duplicate record) (STL)

    You also don't mention what kind of files you want to be the same betwixt v2 and v4.
    Here's a brief description of what is available, and where:

    Sequential only file systems
    QSAM: FILESYS(QSAM), FILESYS(RAW) -- V4.1.1 and above (RAW deprecated)
    supports fixed, variable length, and spanned variable length files
    these files can be shipped back & forth with z/OS via FTP with site rdw (and bin)
    -- not editable/printable
    RSD: FILESYS(RSD) -- V2 and above
    v2 only supports fixed length, v4 also supports variable (and linage)
    Think of RSD as being Line Sequential, but record oriented (2 short reads return different records)
    can be edited/printed/etc.
    LSQ: FILESYS(LSQ), FILESYS(NAT) -- NAT is v2+, LSQ is 4.1.1+ (NAT deprecated)
    4.1.1.6 has limited linage support
    Is variable length, by definition, but can store fixed length
    Is not record oriented (2 short reads can pull from the same record)
    can be edited/printed/etc.

    Sequential, Relative, and Indexed files
    VSAM: FILESYS(VSAM), default -- V2 and above
    A bug in early 3.1/4.1 broke forwards/backwards compat with fixed length sequential files
    4.1.1.6 fixes the forward compatibility, but unfortunately supporting backwards is not possible
    not editable
    STL: FILESYS(STL) -- V2 and above
    I don't know of any forward/backward compat issues with any STL organization
    not editable
    DB2: FILESYS(DB2) -- V3.1 and above
    I don't know of any forward/backward compat issues with any DB2 organization
    not editable

    Typically, I would recommend using the default, it if suites your needs, and overriding for special cases, like needing to view (RSD), or having special needs (backup, transactional, etc) (DB2/SFS).

    However, for the releases under question, and back/forward compat, I would recommend STL over VSAM, for performance, correctness, and likelyhood of meeting your needs.

    If it is a file you intend to print or view, RSD and/or LSQ(NAT) are the only choices.

    Unfortunately, changing file systems can, sometimes, be as painful, if not more so, than changing releases as each file system tends to have its own little quirks :(

    In 4.1.1, we greatly improved overall file system semantic consistency across several file systems, along with much higher I/O throughput rates. We'll continue that work going forward.

    Richard A Nelson (Rick)
    COBOL Development IBM Silicon Valley Laboratory
    http://www.ibm.com/software/awdtools/cobol/
    • lzrycki
      lzrycki
      29 Posts
      ACCEPTED ANSWER

      Re: COBOL for AIX - DDMEA Files

      ‏2012-09-12T12:17:32Z  in response to outlaw
      Hi Richard,

      thanks for your answer. We use COBOL 4.1.1, installed 45 days ago.

      In our COBOL upgrade test we saw :

      1. All our scripts use DDMEA files
      2. Programs compiled with COBOL 2 creates DDMEA files
      3. Programs compiled with COBOL 4 did not creates DDMEA files; for compatibility we use with COBOL 4 compiled programs, COBRTOPT=FILESYS(RAW)

      It's possible to create DDMEA files with COBOL 4 and FILESYS(RAW) ?

      Thanks in advance

      Leonardo
      • outlaw
        outlaw
        39 Posts
        ACCEPTED ANSWER

        Re: COBOL for AIX - DDMEA Files

        ‏2012-09-12T15:14:17Z  in response to lzrycki
        Hello Leonardo

        I'll answer/comment on your update in a bottom-up way.

        No, it is not possible to create DDMEA files via FILESYS(QSAM), or its alias FILESYS(RAW). DDMEA files can only be created or used by the VSAM file system.

        Both v2 and v4 default to FILESYS(VSAM) if not otherwise overridden, and the version of the compiler should make no difference in behaviour here... Unless you have explicit file names in the 'ASSIGN' clause in the COBOL program. You can test part of this by compiling a program on V2, but running it on the V4 machine.

        The fact you're trying QSAM, and wanting DDMEA leads me to believe you are being hit by the bug I mentioned above: VSAM Fixed length sequential file format was changed somewhere in the 4.1 timeframe.

        The effect of this behaviour change is that newly created files now contain imbedded metadata, where they did not before. This change went unnoticed for a while.

        A later PTF (after 4.1.1) allowed COBOL programs to read either the older (non metadata) or newer (metadata) file formats. However, new files continue be created only in the newer (metadata) format.

        As a point of clarification - I believe you are using DDMEA to describe the file format:
        Fixed length sequential files, with no record delimiters ... Simply a byte stream of data.

        When I speak of DDMEA, it is in reference to the actual metadata file associated with every VSAM file:
        If you have a file called 'myfile', and do `ls myfile* .myfile.*` you will see:
        myfile << The data itself
        .myfile.DDMEA << The file metadata (record length, #records, etc.)

        So, assuming the above is mostly correct, I can see why you keep mentioning RAW - it has the same basic file format (byte stream, no delimiters or metadata), but lacks the actual .<filename>.DDMEA file.

        Which brings us to the first point 'All our scripts use DDMEA files'... In what way, are you reading just the file data, knowing the record length ? Do you anything with the .<filename>.DDMEA file ?

        The answer to how you are using the file will guide what the possibilities are going forward.

        Also, please double check the full COBOL version:
        lslpp -l cobol.rte

        You really need to be on the latest PTF, which is, iirc, 4.1.1.6, otherwise you may not have the fix for the above mentioned VSAM file forward compatibility bug!

        Richard A Nelson (Rick)
        COBOL Development IBM Silicon Valley Laboratory
        http://www.ibm.com/software/awdtools/cobol/
        • lzrycki
          lzrycki
          29 Posts
          ACCEPTED ANSWER

          Re: COBOL for AIX - DDMEA Files

          ‏2012-09-12T18:29:51Z  in response to outlaw
          Hi Rick,

          I use the command that you sent me and the COBOL Version is

          lslpp -l cobol.rte
          Fileset Level State Description

          Path: /usr/lib/objrepos
          cobol.rte 4.1.1.0 COMMITTED IBM COBOL for AIX Runtime
          We use FLESYS(RAW) because the files generated by programs compiled by COBOL 2 are exactly the same as the files generated by the same programs compiled by COBOL 4; the only difference: DDMEA files are not created by COBOL 4.

          Our actual system is a mainframe migration without AIX re-engineering. A lot of IEBGENER and IDCAMS have been substituted by a set of utilities that use DDMEA files. Because of this, I ask about FILESYS(RAW) and DDMEA files.

          All our Cobol programs works exactly the same in both environments, with only 2 differences:

          1. one program works with a file created by the execution of an Oracle Select SQL statement + SORT; the program compiled by Cobol 2 works fine with ORGANIZATION IS SEQUENTIAL, but the same program under COBOL 4 see the file as LINE SEQUENTIAL.

          2. one program works with a file crated by the execution of an Oracle Select SQL statement; the program compiled by Cobol 2 works fine with ORGANIZATION IS RELATIVE - ACCESS IS DYNAMIC ( OPEN OK, START OK, all the READ NEXT OK ), but the same program under COBOL 4 with the same ORG and ACCESS expose 6 bytes in the header of each record and gave as a file status error ( OPEN OK, START OK, the first READ NEXT file status '04'). This 6 bytes exists in the physical file. Our solution: add a filler PIC X(6).

          We don't know why we have this differences.
          • outlaw
            outlaw
            39 Posts
            ACCEPTED ANSWER

            Re: COBOL for AIX - DDMEA Files

            ‏2012-09-12T22:59:37Z  in response to lzrycki
            Caveat Emptor:
            I am a very technical person, tend towards being overly verbose, and will switch betwixt the 5000' view, and the 1" view at the drop of a hat, I will carp ad nauseam until we are on the same page (or agree to disagree), but none of this is meant to be disparaging, so please don't take it that way... these are just a few of the plethora of my 'character traits'. Please bear with me and keep asking/refining your issues until I understand how to help!

            You are significantly down-level on maintenance (by ~12 months), you should get that corrected ASAP.

            I was correct in my assumption on your use of QSAM (RAW), but while the file data format is is the same, there is not, nor will there ever will be a .<file>.DDMEA created by FS(QSAM)- that just isn't the use case for FS(QSAM).

            You are still using 'DDMEA files' for two different things, and that is hindering my ability to be of assistance.

            As I asked in the last post: Do your utilities process just the data file (<file>), or do they also attempt to process the (undocumented) .<file>.DDMEA file ?

            As to the specific points:

            1) There are only a few ways for a file to be marked as 'LINE SEQUENTIAL' vs 'SEQUENTIAL':
            ...A) ORGANIZATION IS LINE SEQUENTIAL << in the COBOL program
            ...B) ASSIGN ... TO 'NAT-<filename>' << in the COBOL program
            ...C) ASSIGN ... USING <variable>; where <variable> contains NAT-<filename> << in the COBOL program
            ...D) Use of an environment variable for the assign; where the variable contains NAT-<filename>

            LINE SEQUENTIAL an SEQUENTIAL are completely orthogonal beasts. The former is \n delimited, and the later's format depends upon the underlying file system (SdU, STL, SFS, DB2, etc.)

            If the V2 program worked with this file, it is obviously a fixed length sequential file - which as I've already mentioned is a known issue, and V4 can read/update those files, IFF, you install the requisite maintenance level... otherwise, you are stuck in the current situation.

            2) This one I'm having a hard time wrapping my head around, there just isn't enough detail.
            What is in the 6 byte prefix? I see no file system code (Sdu or STL) that would create a 6 byte prefix!
            The only way I know of get a 6 byte prefix is FS(SDU),FM(S) on V3.1, or FS(SDU) on V2 ... But I'll grant that this may've been one of the many issues with the I/O rewrite started in the 4.1.1.0 level.

            In any case, adding your own PIC X(6)in an attempt to kluge around this will only make matters worse for both of us in the near future.

            If you install the latest PTF, again, iirc, 4.1.1.6, you may find:
            A) The second issue just goes away (subject to your answer above on what is in those 6 bytes).
            ...My guess is the first 4 bytes are binary field, and the last two bytes are always 0x144a
            ...You can use the `od -x` or `xxd` command to display the file contents in hexadecimal.
            B) That V4 will be able to read/update a FS(VSAM) file created by V2, but not visa versa.

            Even were you to open a PMR on this, today, the first course of action would be to have you apply the latest maintenance, and see what, if anything, remains an issue :(

            Richard A Nelson (Rick)
            COBOL Development IBM Silicon Valley Laboratory
            http://www.ibm.com/software/awdtools/cobol/
            • lzrycki
              lzrycki
              29 Posts
              ACCEPTED ANSWER

              Re: COBOL for AIX - DDMEA Files

              ‏2012-09-13T10:25:40Z  in response to outlaw
              Hi Rick,

              thanks for your comments. I'll take into account.

              I ask the technical people why we install COBOL V4.1.1 45 days ago and not the 4.1.1.6 version and if we can install the 4.1.1.6 PTF; they answer me, because IBM recommend us exactly this version to be 100% compliant with TX Series 7.1. IBM said that COBOL 4.1.16 is not full compliant with TX Series 7.1.

              Regards

              Leonardo
              • outlaw
                outlaw
                39 Posts
                ACCEPTED ANSWER

                Re: COBOL for AIX - DDMEA Files

                ‏2012-09-13T18:49:10Z  in response to lzrycki
                Sigh... typical administrivia headaches, perpetrated upon the beleaguered by the clueless.

                There is a rhyme & reason for 'Version'.'Release'.'Modification level'.'I forget this part'...

                4.1.1.6 is simply a maintenance update for the 4.1.1 release. Such a point release is assumed to have the same behaviour/API/ABI/etc... as Version 4, Release 1 !

                At each product launch, a list of related products, and the Version & Release tested (supported) is put in the docs. Typically the only time these lists are updated is when there a new version or release of a related product.

                TX Series will not, nor should the be expected to, adjust their documentation for every PTF we put out, unless that PTF is something they need, or an important (again, for them) fix.

                Likewise, we test COBOL & TXSeries 6.1 & 7.1 ... and don't list any PTFs, as there are none that we know of (to date) that impact COBOL.

                Does that mean TXSeries doesn't support COBOL for AIX 4.1.1.6 ... absolutely not... they said 4.1.1 (actually, probably just 4.1), and 4.1.1.6 is simply a rolled-up collection of bug fixes and enhancements.

                Does that mean COBOL doesn't support whatever PTF/fixpack is current for TXSeries, of course not!

                My advise remains the same (upgrade to 4.1.1.6), and that is also the line you'll get from a PMR, since you are not at the current maintenance level.

                There have been so many changes in the year since 4.1.1 went out, that I can't really provide support, or offer more assistance than I already have on 4.1.1, sorry :(

                Richard A Nelson (Rick)
                COBOL Development IBM Silicon Valley Laboratory
                http://www.ibm.com/software/awdtools/cobol/
                • lzrycki
                  lzrycki
                  29 Posts
                  ACCEPTED ANSWER

                  Re: COBOL for AIX - DDMEA Files

                  ‏2012-09-13T19:18:26Z  in response to outlaw
                  Rick,

                  thanks a lot for your comments; I'll pass your recommendation to our technical people; I supose they will send it to IBM, because we are updating a set of products like AIX , COBOL 2 -> 4, TX Series 6->7 and Oracle 10g to 11g. IBM was in charge to make the release-level-etc recommendation.

                  Regards

                  Leonardo
                  • outlaw
                    outlaw
                    39 Posts
                    ACCEPTED ANSWER

                    Re: COBOL for AIX - DDMEA Files

                    ‏2012-09-13T19:59:49Z  in response to lzrycki
                    Rewording my prior note:

                    4.1.1.6 IS 4.1.1, in fact, it is the only supported 4.1.1 (if a customer reports a problem on 4.1.1.0, or 4.1.1.3, we will ask them to upgrade and reproduce on the supported level), COBOL does not issue fixes on prior PTF levels, as each PTF is a complete roll-up of all prior updates to 4.1.1.

                    The situation would be different if I was asking you to go to the (non-existant) 4.2.0

                    Richard A Nelson (Rick)
                    COBOL Development IBM Silicon Valley Laboratory
                    http://www.ibm.com/software/awdtools/cobol/
                    • lzrycki
                      lzrycki
                      29 Posts
                      ACCEPTED ANSWER

                      Re: COBOL for AIX - DDMEA Files

                      ‏2012-09-14T13:48:05Z  in response to outlaw
                      OK.

                      For IBM
                      4.1.1 = 4.1.1.0 = 4.1.1.1 = 4.1.1.6 = 4.1.1.x ∀x ∈ N

                      I sent your comments to our technical people to review the software release/level/etc grid that we are upgrading. It's possible that this grid was done time ago when 4.1.1.0 was the 'actual' release.

                      Leonardo

                      PD: sorry for my english, I'm a spanish speaker
                    • lzrycki
                      lzrycki
                      29 Posts
                      ACCEPTED ANSWER

                      Re: COBOL for AIX - DDMEA Files

                      ‏2012-09-19T02:50:51Z  in response to outlaw
                      Rick,

                      we'll install the COBOL for Aix PTF to level 4.1.1.6 .

                      Thanks

                      Leonardo
                      • outlaw
                        outlaw
                        39 Posts
                        ACCEPTED ANSWER

                        Re: COBOL for AIX - DDMEA Files

                        ‏2012-09-20T20:24:16Z  in response to lzrycki
                        Cool, and no worries about language usage... English is my native tongue, and I'm still not very good with it ;)

                        I'd like to hear how things go for you after the install!
                        Richard A Nelson (Rick)
                        COBOL Development IBM Silicon Valley Laboratory
                        http://www.ibm.com/software/awdtools/cobol/
                        • lzrycki
                          lzrycki
                          29 Posts
                          ACCEPTED ANSWER

                          Re: COBOL for AIX - DDMEA Files

                          ‏2012-09-27T19:18:16Z  in response to outlaw
                          Hi Rick,

                          about the issue that we have working with Cobol 4.1.1 and sequential files.

                          We run a program with COBRTOPT=ERRCOUNT(99) that opens a sequential file; open file status = '00' , read file status = '04' and the record is garbage; if we change the organization as LINE SEQUENTIAL, open file status = '00', read file status = '00' and the record is OK. ( first solution ) but if we maintain the original program with Organization is Sequential and run the program without COBRTOPT=ERRCOUNT(99) , the program runs perfect. ( second solution and better than 1st ).

                          I can't understand the relation between ERRCOUNT(99) and the reading failing process for sequential files. Do you know if this is a bug solved in 4.1.1.6 ?

                          Thanks in advance

                          Regards

                          Leonardo
                          ERRCOUNT
                          ERRCOUNT indicates how many warning messages can occur before the run unit terminates abnormally.
                          ERRCOUNT option syntax

                          .-20-----.
                          >>-ERRCOUNT(-+--------+-)--------------------------------------><
                          '

                          Default: ERRCOUNT(20)

                          number, if positive, is the number of warning messages that can occur while the run unit is running. If the number of warning messages exceeds number, the run unit terminates abnormally.

                          number, if 0, indicates that an unlimited number of warning messages can occur. number cannot be negative.

                          Any message due to a condition that has a severity higher than a warning results in termination of the run unit regardless of the value of the ERRCOUNT option.
                          • outlaw
                            outlaw
                            39 Posts
                            ACCEPTED ANSWER

                            Re: COBOL for AIX - DDMEA Files

                            ‏2012-09-28T23:28:46Z  in response to lzrycki
                            There isn't enough information for me to hazard a guess :(

                            ERRCOUNT should only play a role if you are getting Warning messages issued, but you don't say.

                            As for the crux of the matter (VSAM/SdU sequential file processing) - As I said above, very early:
                            VSAM: FILESYS(VSAM), default -- V2 and above
                            A bug in early 3.1/4.1 broke forwards/backwards compat with fixed length sequential files
                            4.1.1.6 fixes the forward compatibility, but unfortunately supporting backwards is not possible

                            So yes, the reading of a V2 created sequential file should work fine when you get to 4.1.1.6.

                            However, you can't take a v4 created sequential file and read it on the system with v2 installed.

                            If you need to transport a sequential file and process it on both v2 and v4, you need to create the file with v2 and ship both the base file, and the .DDMEA file to the v4 system, where you can alter it at will, and return both files to the v2 system.

                            Richard A Nelson (Rick)
                            COBOL Development IBM Silicon Valley Laboratory
                            http://www.ibm.com/software/awdtools/cobol/