CMS/TSO Pipelines enhancements since 1.1.11 (c) Copyright IBM Corp. 2000, 2010. DISCLAIMER: =========== This file describes enhancements to the "CMS Pipelines Run-time Library" since 1.1.11. The fact that an enhancement is described here must not be construed as a commitment by IBM to supply such enhancement to the z/VM product or any other product. The enhancements are subject to change without notice, including complete withdrawal. Sometimes things just don't pan out. Please read the section INCOMPATIBILITIES below. Refer also to the CMS Pipelines home page: http://vm.marist.edu/%7Epipeline The "CMS Pipelines Run-time Library", which currently is CMS Pipelines sublevel 20 as of January 9, 2009, is available for download from the CMS Pipelines home page. References to requirement numbers refer to the fifty requirements submitted by SHARE Inc. in March 1996 and the twenty-five submitted since then. See http://vm.marist.edu/%7Epipeline/sharereq.html This file is written bottom-up. The latest items are first. ----------------------------------------------------------------------- VM SERVICE: A problem with ABENDs in BSAM/QSAM has been identified in CMS. When a SYNAD exit is missing, the ABEND 1 is reported from the SVC that called the command that fails rather than the actual ABEND instruction. This means that I/O errors in commands issued through COMMAND or CMS often are perceived as problems in CMS Pipelines, when CMS PIPELINES is merely the messenger. CMS may be changed to correct this in a future release (future as of May 3, 2002). OTHER SERVICE: High level assembler Release 4 requires a PTF for APAR PQ72467 to avoid message 1321 (and you need sublevel 6 for the CMS/TSO Pipelines code to issue message 1321). PTFs: R4 R3 CMS UQ75787 UQ75785 MVS UQ75801 UQ75789 VSE UQ75837 UQ75835 OS/400 requires PTF# MF30781 for APAR MA27293 to operate correctly with TCPCLIENT. ----------------------------------------------------------------------- INCOMPATIBILITIES/SPECIFICATION CHANGE MD5 Is superseded by DIGEST MD5, which has additional function. MD5 will be removed from the runtime distribution in a future sublevel. Assigning the string contents of a counter that has more than 18 characters to the same or another counter gets an assert failure. SPECs The scope of the STRIP keyword in a specification item was changed in sublevel 8. STATE ISODATE output format no longer has leading blank in times before 10AM. That is, it now conforms to the standard. TIMEstamp as of sublevel 15, the two last digits representing hundredths of seconds may be non-zero. On the other hand, TIMEstamp can no longer run in the DOS environment. ----------------------------------------------------------------------- Using PL/j. If you are assembling PL/j programs with the High Level Assembler, be sure not to specify COMPAT(MACROCASE) as the assembler does not proccess the DEFCHARS macro correctly. You'll just have to be a good boy and put the unquoted operands in uppercase. ----------------------------------------------------------------------- ARCHITECTURE LEVEL SET As of 1.1.11/21, CMS/TSO Pipelines supports data spaces; for VM, in an XC virtual machine only. This support requires the ESA/390 architecture level set, notably the LAE instruction. A G3 level set may be assumed by some future sublevel. The the only supported releases of the underlying operating systems are in both cases z/Architecture, but the P390 is still close to my heart. TCPCKSUM introduced in 1.1.11/001F requires the G3 level set for the CKSM instruction, which is part of the P390 repertoire. z/Architecture is supported on CMS, but not required. That is, CMS Pipelines supports the version of CMS that runs in 64-bit mode. Note that other ESA and XC virtual machines run in 390 architecture mode even when CP runs in 64-bit mode. ----------------------------------------------------------------------- Sublevel 21: Add global and local option QUALIFY. The local option may also specify NOQUALIFY or QUALIFY . (period) to disable the global qualifier for the stage. A stage's qualifier is inherited when an encoded pipeline is issued that specifies callpipe, but not for addpipe (and not for any of the pipeline commands issued from REXX). This allows constructs such as IF and TOTARGET to reference the current qualifier, which is impossible otherwise. Add support for arrays, arrays of structures, and subscripting. A subscript is a positive integer enclosed in parentheses after the member identifier it applies to (e.g., struct.substruct(6).member or struct.member(3)). When the last identifier references an array, but without providing a subscript, the entire array is selected. In this case, the array must have a definite bound. (See STRUCT below.) The index origin is 1; thus, subscripts must be unsigned positive numbers. The identifier following the MEMBER keyword may be prefixed one or two periods. A single period means to apply the current qualifier, which must have been specified previously. Two periods means that a fully-qualified member name follows; thus, any active qualifier is ignored. When the member was defined as a field, word, or autofield in SUBSTR ... OF MEMBER, you would get the specified member to the end of the record; that is, the substring operation and the count/ending in the member definition were both ignored. Five structures are predifined in a global scope, which is searched last, so that it cannot obscure user-defined structures: * The CP data areas VMCMHDR and VMCPARM (use with the VMCF stages), * FPLASIT, which describes the first part of a data space created with ADRSPACE CREATE INITALISE. * FPLSTORBUF, the output record from INSTORE. * EVENTRECORD, the output record from RUNPIPE EVENTS. Structures in global scope can be listed by STRUCT LIST as long as they are not obscured; they are not listed by STRUCT LISTALL; and they cannot be deleted. ADRSPACE CREATE [ALET] [INITialise] DESTROY ISOLATE PERMIT [ USER | VCIT ] [WRITE] QUERY [ [USER] ] ADRSPACE exposes the CP macro by the same name. It is available on CMS only. The virtual machine must be in XC mode (this precludes zCMS). CREATE: The input must contain two or three blank-delimited words: The name of the address space; its size in pages (decimal); and the storage key (hex, typically e0). When ALET is specified, the function of ALSERV ADD WRITE is also performed. If INITIALISE is specified, ALET is implied; the first bytes of the new address space are initalised as follows: 0-7 The constant 'FPLASIT1'. 8-F The ASIT of the address space. 10-13 Number of pages allocated. 14-17 Zeros. May be used as a lock. 18-1F The name of the user that created the address space. 20-38 The name of the data space. 38-3b Next available byte (=x'50'). 3c-3F Last available byte (= page count * 4096 -1). 40-4F Zeros. When ALET and INITALISE are omitted, the output is the eight-byte ASIT of the newly created address space and a four-byte binary count of pages in the address space. The count is a multiple of 256. When ALET is specified or implied, the output is 16 bytes with the ALET appended to the basic record. DESTROY: The input must be eight, twelve, or sixteen-byte records containing the ASIT to destroy in the first eight positions. ISOLATE: Ditto, the ASIT to isolate. PERMIT: Specify the user ID to be authorised or the ASIT of its primary space (which is called a VCIT). Pass the ASIT(s) to authorise on the input. These ASITs may be suffixed four or eight bytes. Access is read only unless WRITE is specified. Note that an IPL of either virtual machine will revoke the permission. When neiter USER nor VCIT is specified, the input record must be 16 bytes and specify the ASIT followed by the user ID. QUERY: Specify the user you wish to query as the operand and pass the address space name(s) on the input. The name BASE returns the ASIT of the primary space. The output contains an eight-byte ASIT followed by a four-byte binary number containing the number of pages in the address space. When the secondary output stream is defined, input records for non-extant objects are, in general, written to this stream; otherwise they attract an error or warning message. For PERMIT, the secondary stream is used to indicate that the specified user or VCIT already has permission to the address space; a non-extant ASIT, user, or VCIT attracts an error message. ADRSPACE may be used to access the main storage from an adjunct virtual machine. As the adjunct is a separate machine, but shares its name with the base, getting permission is a bit tricky. First the adjunct must inform the main of its VCIT to allow the main to permit the adjunct. Then the main must communicate its ASIT to the adjunct, which can then perform ALSERV ADD. Note that PERMITs are dropped on IPL of either virtual machine, permitter as well as permittee. Any ALETs created while the permission lasted remain, however, but in a revoked state. ADRSPACE starts on commit level -2, veriries that the secondary input is unconnected, and commits to zero. ALSERV { ADD [WRITE] | REMOVE | TEST } ALSERV exposes the CP macro by the same name. It is available on CMS only. The virtual machine must be in XC mode (this precludes zCMS). In addition, it supplies a way to test the validity of an ALET. ADD: Pass the ASIT(s) to be added on the input. (Four or eight bytes may be appended.) The output is a four-byte ALET. REMOVE: Pass the ALET(s) for the address space to remove from you configuration. The now defunct ALET is passed to the output after it has been removed. TEST: Pass the ALET(s) you wish to verify on the input. Valid ALETs are passed to the primary output. If a secondary output is defined, invalid and revoked ones are passed to it. Otherwise a message is issued if the TEST ACCESS (MVS) or TEST PROTECT (CMS) instruction sets condition code 3. (The astute reader has noted that it looks as if TAR is broken in XC mode.) ADRSPACE starts on commit level -2, veriries that the secondary input is unconnected, and commits to zero. DISKID Issue the DISKID function to obtain characteristics of a reserved minidisk. DISKID [ ...] DISKID first queries the device numbers specified in the operand list, then reads its input, if any, and performs the query for each word on each input line. The output record contains the last sixteen bytes of the DISKID parameter list: DS XL2 Virtual device number DS H Blocksize DS F Offset DC D'0' Reserved INSTORE Add ALET . ALET and PGMLIST are mutually exclusive. The ALET is typically generated by ADRSPACE CREATE INITialise. The address space must have been created with the key that CMS/TSO Pipelines executes with, which is X'E0' on CMS and X'80' on MVS. The buffer is built in the specified address space rather than in free storage. The specified address space cannot be used by other stages while it is active with INSTORE. The address space is released for other use when INSTORE terminates. The file may be extracted by passing the record to OUTSTORE or it may be extracted in a different virtual machine that has obtained access to the data space, as long as the output record is not consumed in the creating virtual machine. The extracting virtual machine will typically use a different ALET from the creating virtual machine. INSTORE now starts at commit level -2. It commits to zero when the contents of the address space have been validated. Using the ALET operand may offer virtual storage constraint release, as well as inter-machine data transfer. MAPMDISK DEFINE [ FETCH | ZERO | RETAIN ] The input record describes a data space to be created: 1. The ASIT for the data space (hex). 2. The number of the first page to map (decimal, nonnegative). This number is multiplied by 4096 before it is stored in the parameter list. 3. The count of pages (decimal, positive). 4. The number of the first pool block (decimal, nonnegative). The input record is echoed to the output when the pool extent has been mapped. Thus, the output record may be passed to MAPMDISK REMOVE when the pool extent should no longer be mapped. MAPMDISK IDENTIFY Create a minidisk pool. MAPMDISK reads all input and then creates a single pool, which replaces any existing pool. Each input record describes one extent of the pool: 1. Device number. 2. Block offset (from DISKID). 3. Block count (from LISTFILE). The devices are assigned contiguous pool block numbers strating from 0. MAPMDISK REMOVE The input record has the same format as for CREATE, except that the first three words only are used. The input record is echoed to the output when the pool extent has been unmapped. MAPMDISK SAVE The input record specifies the ASIT to be saved and a number of s of pages to be saved. (That is, the block form of the macro is used.) The output record contains the eight-byte error buffer followed by the interrupt subcode (which should be x'01') and the completion status (x'00' for all saved) for a total of ten bytes. Starts at commit level -2 and establishes an external interrupt handler and then commits to level 0. NOT and DIGEST are just the messengers. The scanner processes NOT DIGEST SHA512 incorrectly, issuing message 113. This has been broken ever since it was written in 1992. OUTSTORE Add ALET . This can be specified only when OUTSTORE is first in a pipeline. The address space must not be fetch protected. The file is extracted from the specified address space. The file is extracted from the data space where it was built by INSTORE when OUTSTORE is not first in a pipeline and the input record indicates an ALET for the contents of the file. OUTSTORE now starts at commit level -2. It commits to zero when the contents of the address space have been validated. PICK Support manifest constants in numeric compares. RITA Broke with the fix to issue CP messages from immediate commands. It has reported 0 time since December 16 2008. SORT With UNIQUE and a secondary stream, the duplicate records were still written to the primary stream. SPEC Avoid program checks: 1) when referencing an unsigned binary structure member in a counter expression; 2) when upgrading integer to floating point, e.g., length(a)-length(b). It is now documented that on the run-out cycle, if an EOF item is present, it will be the first issued. It has always been that way, and I dare not change it, even though it might be unintuitive that, e.g., PAD items are not issued. Provide the correct sign when a PICTURE is used with ROUND and a negative number and the rounding results in a carry out of the most significant digit. Array subscripting. =================== NOTE: This describes changes to the handling of counter expressions ONLY. The handling of an is as described above. That is, a construct such as "SPEC member q.m(5) 1" is not described here, but to make it excrusiatingly clear, expressions are not supported in this construct. Nor are they supported in qualifiers. Support references to array members in a counter expression (in addition to the support). You may refer to a member with an expression in parentheses. Such expressions may reference array members that are structures too. Assuming that struct test2 is defined and has an array member a: literal abcdefghijklm|spec while (#0+=1)<5 do print test2.a(#0) nw done As in an , a member name may be prefixed one or two periods (e.g., .member and ..member). When prefixed a single period it means that a qualifier must have been declared. Using two periods ignores any established qualifier; that is, it indicates a fully qualified member name follows. Using parentheses for subscripting avoids codepage issues inherent in braces and brackets, but the slight downside is that built-in functions have priority over simple array member names (thus you cannot just refer to member "length"). However, you can always prefix a single period to an unqualified member name to state explicitly that this is a member that needs qualification (a fully qualified member name must have at least one period, which ensures no ambiguity). Another consequence of this is that a misspelled function name is reported as an error in specifying a member name. You may also refer to the counter array directly by #(). Specify the global operand COUNTERS to ensure that counters 0 through the number are allocated. Manifest constants ================== Manifest constants are supported as terms in a counter expression and as constant input fields. As a term in a counter expression, it simply a binary number that is treated like all other numbers: print x.c 1 As a constant field, the manifest constant is a four-byte signed binary field of type d, but it is right-aligned by default. You can assign it as all other typed members: member x.c c2d 1 Often you may want to assign this binary value, as here to set the function code in a VMCF parameter list: spec qualify vmcparm 1-* 1 member vmcprepl member vmcpfunc STORAGE [ALET ] READ [ALET ] [ ] [ALET ] When READ is specified, STORAGE is not sensitive to its position in the pipeline; it reads storage unconditionally. Using first the area specified by the operands, if any; then the areas specified on each input record. CAVEAT EMPTOR: The area specified by the operands is verified to be addressable, but the area specified on input records is not as this test is rather expensive. Thus, improper input record contents can cause program checks in STORAGE (if a nonzero ALET is specified) or subsequent stages (otherwise). Thus STORAGE READ is designed for high-performance use where the address is already verified to be within the specified addess space. Use of STORAGE READ is discouraged on MVS, as it is almost impossible to predict whether an address is valid or not. The specified ALET is used to read or write storage. This requires an XC mode virtual machine for CMS. The ALET operand is supported on both CMS and MVS, but on MVS you must discover the ALET yourself. As VM uses only the primary list, this flag is added on CMS. On MVS the ALET must be specified exactly. STRUCT ADD [ANYcase] [THREAD] ANYCASE: Case is ignored in comparing the structure name and the member names, but case may be respected in nested structures that were defined by another STRUCT ADD stage that did not specify ANYCASE. Add support for manifest constants and arrays. Also support manifest constant with the LENGTH option. An array specified as (*) is varying length; it can be accessed only with a subscript. The index origin is 1. Multidimensional arrays are not supported. ::= : + ::= | ::= = ::= [] | - ::= | ( "*" | ) ::= | ::= | ::= [WORDSep | FIELDsep ] [Word | Field | AUTOField] ::= STRUCT | Member | Length ::= | Member | Next SUBSTR Issue correct messages when an is not present or the range is a manifest constant. TIMESTAMP Fix bad output when the number specified is larger than 16. Also fixed the table of timestamp lengths in the article on sublevel 15 enhancements. UTF Convert between UTF-8, UTF-16, and UTF-32. UTF [FROM] [TO] [REPORT] ::= [MODIFIED] UTF-8 | UTF-16 | UTF-32 Modified UTF-8 is the encoding favoured by Java. It has the property that x'00' does not occur (and thus it may be used to delimit strings in the "normal" way. Specify REPORT to issue a message and stop when the input is not valid for the format specified. For UTF-8 this includes "overlong" encodings. VMC Interoperates with VMCLISTEN, but as it deblocks the response it receives into 80-byte records, it may not be particularly useful. Still, it saves you building the VMCF parameter list. VMCDATA Receive data; send reply or reject message. No arguments. The input record must be at least 40 bytes. It is typically derived from the output from VMCLISTEN. The valid function codes are Receive, reply, and reject, though a send/receive function is interpreted as reply. Any reply data must be appended to the interrupt header. EOF on the output is ignored. VMCLIENT [] Send VMCF requests to other virtual machines. If specified, the word is the user ID to receive the messages; this user ID is inserted in all parameter lists. If no word is specified, you must supply the traget virtual machine user ID in the parameter list. The input record must be at least 40 bytes. It consists of a VMCF parameter list with send data appended. Valid function codes are send, sendx, send/recive, and identify. The output is the final interrupt header appended any reply data (send/receive only). The message identifier is ignored for requests other than identify (it is used by CMS Pipelines to identify the particular client stage). VMCLISTEN [RECEIVE] Write VMCF requests into the pipeline. Must be first. There can be only one VMCLISTEN stage active in a virtual machine. Each VMCF interrupt header is written to the output stream including sendx data. Send data are appended to the message header when RECEIVE is specified. VMCF interrupts are disabled while the record is being processed by the subsequent stages; if such stages wish to pass a record to VMCLIENT, they should COPY the record to consume it. ----------------------------------------------------------------------- Sublevel 20: (January 9, 2009) SPEC Process the keyword STRING correctly. (That is, as it was in sublevel 1E.) ----------------------------------------------------------------------- Sublevel 1F: (January 7, 2009) The conversion routine for U2C did not strip leading or trailing blanks. As a result it would fault input fields that contain blanks. This surfaces in, e.g., SPEC print #0 u2c, as the default picture suppresses leading zeros. DIGEST Add NOT to reverse the meaning of the output streams. This allows a verifying application a simple way to propagate EOF backwards when the desired output is the secondary output. This allows a way to stop on the first mismatch. JUXTAPOSE Support any number of input streams. This is a convenience for FANINANY feeding to the secondary input stream. For input streams, substitute "a positive numbered" for "the secondary". PICK Support data type U for unsigned binary. SPEC Process a one-letter identifier with qualifier correctly. Improve performance of scanner. Support data type U for unsigned binary. Support the keyword STRING to indicate that a delimited string follows, e.g., SPEC STRING 1-1 1 will write "-". [Deleted in sublevel 20.] TCPCKSUM Compute ones-complement checksum of a message. TCPCKSUM [] Without an operand, TCPCKSUM computes the checksum of each input record and produces a 16-bit result checksum on the output. This result should be all zeros when the input message contains a valid TCP/IP checksum field. When specified, the operand designates the begin column of the 16-bit checksum field within the record. The checksum of the record is computed and stored into the specified position; the updated record is then written to the output. For correct interoperation with TCP/IP, the checksum field in the input record should contain binary zeros. TCPCKSUM interoperates with the IP, TCP, and UDP headers. Refer to RFCs 1071, 1141, 1624, 1936. ----------------------------------------------------------------------- Sublevel 1E: (December 16, 2008) Do not loop on a stall when there are stages in internal wait. And do report stalls that involve stages in internal wait. PIPMOD Add the immediate commands ACTIVE and STOP ACTIVE. Responses are CP messages as the code is running disabled. The commands apply to all threads. This discussion assumes that only the main thread is running a pipeline; that is, we ignore CMS multitasking. PIPMOD ACTIVE shows the currently running stage. PIPMOD STOP ACTIVE sets a flag for the dispatcher to terminate the stage. Whether the dispatcher sees this flag is another matter. A stopable stage (sometimes called summarily stopable) is a stage that has specified that it uses only resources that the dispatcher knows how to release. It is not documented which stages are stopable (and this attribute may indeed change for a stage over time), but most filters are, including SPEC. If the running stage is stopable and the I/O PSW does not indicate that control is in the pipeline dispatcher, it will be forced to stop by making it return to the dispatcher with return code -4091 when DMSITI returns to the interrupted program. A non-stopable stage that is in a loop will not return to the dispatcher and therefore not cause it to terminate the stage. HX is still the only option for such a thing. Also note that CMS will interpret the PIPMOD command as a normal command when entered at the Ready prompt. Neither ACTIVE nor STOP are valid PIPMOD subcommands. CHOP Support a negative number, indicating the length of the chopped-off piece (the record on the secondary output). Note that -0 is not an according to the pipeline syntax rules. CRC Add the keyword CRCFIRST, which is optional after EACH. When specified, the CRC is written before the input record is passed to the primary output. When EACH is specified without CRCFIRST, the records are written first to the primary output and then the secondary. The record in the secondary output is no longer delayed. SPEC Fix assert failure in conjunction with storing many and long strings in counters. The contents of fields that are defined by a field identifier is not converted correctly to binary in some contexts, notably in an evaluated output position. Do not run out of storage when a LENGTH * member is used as the output specification; take the length of the input instead. Return correct data in record() after a READ item. Add storage(string, int) function. The string is printable hexadecimal; the length is a plain number. spec eof print storage('230', 32) 1 storage() does not verify the validity of the address specified; in particular, it does not enforce that the address be within the virtual machine's storage, which allows you to dig stuff out of a shared segment. Definitely drive with due care and attention. storage('560', 4) Returns the size of the virtual machine for your validation pleasure. Add ability to retain the output buffer between records when write is suppressed due to break levels. This is enabled by specifying KEEP after PRINTONLY , e.g., spec printonly eof keep a: w1 . w2-* (a) The first word specifies where to print the rest of the input line. Without KEEP you would see only the last input line. Requirements: 21 The requirement is satisfied with the two immediate commands. ----------------------------------------------------------------------- Sublevel 1D: (October 28, 2008) PICK Storage leakage with complex parameter list when the pipeline fails its syntax check (be that PICK or some other stage). Also possibility of message about storage corruption. Respect data types (D, F, and P) in numerical compares and convert appropriately including packed scaling. SPEC Avoid 0C5 in reverse() when the argument is null. Restore WORD() to accept three arguments and also set the default word separator correctly. Do not force numeric conversion when a field identifier is used with ||=. Correct length of an output member. Correct 0C7 when null strings for arithmetic. Issue message for remainder by zero. Set correct exactness for P2C conversions; also enforce correctness of truncated numbers. Correct X2D to handle negative numbers (it was doing the same as X2U). Add support for the P data type in a member. The data are treated as signed packed decimal integer. A scale factor may be specified in parentheses immediately after the P. A positive scale indicates the number of decimal places in the number. Any fraction (after scaling) is truncated when used as an output field without a scale factor. STRUCT Fix ABEND with more than 128 members in a structure. (Blush.) Support "Length *" to indicate a variable length member. This turns off the position so that any following member must have an explicit position. This is typically used in the last member of a structure, but it can be put anywhere. Support a halfword scale factor in parentheses after the type character. Just as the type has no meaning to STRUCTURE, neither has the scale factor. They are both ignored in most contexts. Minor enhancements/fixes to the formatting. Add yet one more message to help in finding the location of a problem. ----------------------------------------------------------------------- Sublevel 1C: (September 16, 2008) (September 19, 2008) This sublevel was sent twice to Marist because the file pool was restored from a backup sometime after the morning of 2008-09-16. PICK Now has a contorted test case for parentheses. And when you get into that game, you'll need some help figuring out what goes wrong in your matching. Turn on message level 512 to list compares that fail. ----------------------------------------------------------------------- Sublevel 1B: (September 13, 2008) PICK Now has test cases for parentheses. It has also been modified to retest the FROM condition when a TO without AFTER has matched. The description below has been updated. ----------------------------------------------------------------------- Sublevel 1A: (September 7, 2008) Traps: A trap facility is implemented to allow the user to specify CP commands to issue when a particular message is issued. The facility is enabled for messages FPLDSK124E and FPLDSK129E. There is no user interface to enable other messages, though it would be possible to write a REXX program to enable other messages in the field, should this be necessary. When a trap-enabled message has been issued, the global variable corresponding to the module, message, and severity (e.g., DSK124E) is fetched from the group FPLTRAP and inpsected. If the contents of the variable is not blank, it is stripped of leading blanks and passed directly to CP with diagnose 8 and then logged with message 1453. The return code is ignored. The most useful command to issue by these means is expected to be VMDUMP, but you could also display storage or send a message to someone instead (or given sufficent privileges, shut down the system). The usual rules apply: The command is limited to 240 bytes; it should be uppercase unless for strings that you wish to be mixed case; and line end characters separate multiple commands. It is deliberate that the command is issued directly to CP; this makes the current register set easy to find (registers 0 through 12 are the ones of the program that issued the message); had a CMS command been issued instead, it would be non-trivial to find the registers (though clearly possible given sufficient stamina). Assuming you create a VMDUMP SPOOL file, you can use DUMPLOAD to read it in, then access MAINT 193 and inspect it with VMDUMPTL. The CMS/TSO Pipelines home page contains a sample VM Dump Toool macro, FPLDSK VMDT, to extract information from the dump file for the two enabled messages. PICK Fix incorrectly issued message 54s. Remove the restriction on the number of compares that can be joined with AND or OR (was 10). Add the operators IN and NOTIN. For IN, a null left range matches anything; and a null right range matches nothing except a null left range; each of the characters of the left range must be present somewhere in the right range to match. Note that IN is asymmetric, unlike the compares. The typical use of IN would be to test whether a one-character input range contains one of a set of characters. Add the ability to specify that one comparand has the same length as the other by allowing +. The number specifies the beginning column. The length used will be the samller of the length of the other comparand and the length of the rest of the record from the specified column. The obvious application is to avoid counting the length of a string constant, but other uses are possible. Parentheses can now be used to group comparisons. The only sensible use is to enclose OR items in conjunction with AND, but anything syntactically correct is accepted. For equal compare (= and ==), the right-hand input range may be repeated by coding a list of comma-separated strings or input ranges. Implied length is not allowed on the left-hand side in such a construct. The following two PICKs are identical: PICK 1 == /a/ or 1 == /b/ or 1 == /c/ PICK 1 == /a/, /b/, /c/ The motivation is laziness, but with a complex left-hand input range there may be some measurable performance increase with large files (or possibly less performance decrease caused by scanning for the new syntax). PICK is further turned into a do-it-yourself kit for partitioning the stream. The full syntax is now: PICK [ANYCASE] [PAD ] [ | {{FROM | TO} [AFTER] | WHILE] }] {TO [AFTER] | COUNT } ::= | {AND | OR} ::= | ( ) ::= | + | When neither FROM, TO, nor WHILE is specified, PICK selects lines that match the list of comparisons and discards those that do not (as it has done all the time). When FROM, TO, or WHILE is specified without a second list, PICK partitions the file at the first line that matches (or not): FROM rejects the part of the file up to the first matching record; the remainder of the file is selected. TO selects the part of the file up to the first matching record; the remainder of the file is rejected. WHILE selects records until the first that does not match; it rejects the balance of the file. When COUNT or TO is specified with FROM, PICK discards records up to the first record that is matched by the FROM list. It then selects either the number of records specified by COUNT or up to the next record that matches the TO list. When TO is specified without AFTER, the matching record is not written immediately; instead the FROM clause is retested against the record to see whether it starts a new range to be selected (this is important when the TO is a subset of the FROM clause). It then goes back to rejecting records until another one is matched by the FROM list. Thus, BETWEEN/INSIDE /abc/ /def/ can be implemented as respectively: PICK from 1.3 == /abc/ to after 1.3 == /def/ PICK from after 1.3 == /abc/ to 1.3 == /def/ Other variations of AFTER are available that cannot be implemented with any of the BETWEEN family of selection stages. For example, to skip to the first page of the HASM/HLASM object listing (which usually contains "- Loc", but can contain a nonblank character in the second column); and to stop at the relocation dictionary or the cross reference, whichever occur first, but resuming at the next object listing: PICK anycase from after 1 == /-/ and 2 in / CDR/ and 3.4 == / Loc/ to 1+ == /- Pos.Id/ or 1+ == /-Symbol/ This will work with multiple stacked listing files, as produced, for example, by a BATCH assembly. To select the first assembly only: PICK anycase from after 1 == /-/ and 2 in / CDR/ and 3.4 == / Loc/ PICK anycase to 1+ == /- Pos.Id/ or 1+ == /-Symbol/ (The wrinkle here is that when HLASM is run with LC(0), there are no page ejects, so checking for, e.g., the string "Relocation Dictionary" might be tripped by a sufficiently contortedly constructed MNOTE.) Notes: An must be terminated in a blank (unlike a ). PICK 1 == 2, 3 will give a rather misleading diagnostic. The following syntactically is valid, but unlikely to do what you want, because it will not select anything as the second comparand is six-byte literal string: PICK 1 == 2, 3, 12 Consider a stacked COPY file where members are separated by a *COPY record. To select all members beginnig with A: pick from 1+ == /*COPY / and substr 1 of w2 == /A/ to 1+ == /*COPY / POLISH Does not default the HEX option. And the message (42) is not helpful. SPEC Fix c2u conversion not to return '0' regardless of the input. c2u(8) conversion and the c2u() bif are OK. Fix functions to verify their arguments better. Add the ability for counters to hold string values; and the operators || and ||= for string catenation. Beware of the need to escape; you will likely be using four bars. Note that catenation with blank (the blank operator) is not implemented. As in REXX, || has precedence between & and +. Values are converted between numbers and strings as required, except that assignment and print directly from a field identifier forces conversion to a number to remain compatible with the past. (This becomes apparent when a field identifier is used directly in a PRINT specification item without a picture, where the default picture applies.) The string(x) cast can be used to assign the contents of an input range as a string regardless. Counters are now initialised to the NULL value, which converts to a zero or a null string; thus JOIN * can now be implemented much slower as: SPEC printonly eof 1-* a. set #0||=a eof print a 1 Add the built-in functions that work like the REXX ones (and enhance SUBSTR to allow padding). This is the complete list of REXX-compatible functions implemented: ABBREV DELSTR MIN TRANSLATE ABS DELWORD OVERLAY VERIFY BITAND FIND POS WORD BITOR INDEX REVERSE WORDINDEX BITXOR INSERT RIGHT WORDLENGTH CENTRE JUSTIFY SIGN WORDPOS COMPARE LASTPOS SPACE WORDS COPIES LEFT STRIP XRANGE C2X LENGTH SUBSTR X2C DATATYPE MAX SUBWORD (These are the reducible functions from PATTERN, if anyone is interested; the SPEC-specific functions remain unchanged, of course.) Not that loading a number into a counter is fast, but id did get 10% faster. STRUCT ADD now accepts a type character with filler items (those where a hyphen or period is specified instead of a member name). It also attempts to issue an explanatory mesage to inform you roughly where a problem is. STRUCT cannot issue messages that relate to the original records because it conditions the input stream so that a structure definition does not span record boundaries. TRACKXPAND was documented, but the actual entry point was TRACKEXPAND. The variant without the E is now a synonym. Requirements: 36 Is satisfied with the enhancements to SPEC. Whether this is "high performance" may be open to discussion. And clearly some functions, such as sourceline, have no meaning in a pipeline context. ----------------------------------------------------------------------- Sublevel 19: (August 5, 2008) LOOKUP Stalls when the quinternary input is connected, but neither the tertiary nor the quarternary are. (It insists on reading from the primary only.) SPEC Fix datatype() to be in agreement with REXX on blank and null strings. datatype("", "x") now returns 1 datatype("", "n") now returns 0 datatype(" ", "n") now returns 0 datatype("", "w") now returns 0 datatype(" ", "w") now returns 0 datatype("") now returns CHAR datatype(" ") now returns CHAR Note that datatype returns NUM when the first string can be assigned to a counter without error. It returns NUM for many strings that cannot be processed by the d2c conversion on a specification item. String function results that are longer than 18 bytes, but shorter than 22 had the final bytes trashed. Bites for example with print record(#7, 20). (Blush.) Add unsigned decimal conversions C2U and U2C plus the same extended conversions as supported by C2D and D2C. Add built-in functions C2U and X2U. TRACKWRITE Fix channel program check when reading the disk label and a non-zero begin of extent is specified. ----------------------------------------------------------------------- Sublevel 18: (May 16, 2008) >SFS hole | >sfs x y .z f coerce Attracted assert code 111 for a nonextant file. It now attracts a message to specify the record length. AFTFST Add support for STRing as in TIMESTAMP. CRC The complete description of the CRC stage, which is now supported, follows. (ADDLENGTH and CKSUM are new.) CRC [APPEND] [EACH] [ADDLENGTH] [CRC-32 | CRC-16 | CRC-16I | CCITT-16 | CKSUM | ] ::= {16-BIT|32-BIT} [PRELOAD ] [REFLIN] [REFLOUT] [COMPLEMENT | XOROUT ] [ADDLENgth] Compute the cyclic redundancy code of a message. When one output stream is defined, the default is to read the entire file, computing the CRC, and write a single record at end-of-file. The record contains two, four, or twelwe bytes of binary data. When the secondary output stream is defined, the input is passed to the primary output without being delayed; the CRC is written to the secondary at end-of-file. CRC starts on commit level -2, verifies that the secondary input is not connected, and then commits to level 0. The CRC operands are of two types, general flags, and operands to specify the algorithm. The general flags are: APPEND Pass the input to the primary output. Write the CRC as a separate final record. Secondary streams are not allowed. EACH Write the CRC for each record and reset to initial conditions. ADDLENGTH Logically append the number of bytes in the record or file to the data being checksummed. Only the significant bytes of this count are included. The length is processed with the loworder byte first. Algorithmic operands are specified with 16-BIT or 32-BIT, one of which is required, followed by the remaining operands. 16-BIT Use a 16-bit polynominal, which is specified in hexadecimal as the following word. Only the loworder 16 bits are used. 32-BIT Use a 32-bit polynominal, which is specified in hexadecimal as the following word. PRELOAD Specify the initial accumulator contents. The default is 0. REFLIN "Reflect input." Transpose the bitorder of the input bytes so that the least significant bit is leftmost. REFLOUT "Reflect output." Transpose the bitorder of each byte of the resulting CRC before applying XOROUT, but after applying ADDLENGTH. XOROUT Exclusive OR the CRC with the hexadecimal word that is specified after the keyword. COMPLEMENT Equivalent to XOROUT FFFFFFFF. ADDLENGTH As described above. This is an alternative way to protect against a burst of leading zeros, as all CRC algoritmns produce zero for input bytes of zero CRC when the accumulator is zero. That is, consider using ADDLENGTH when PRELOAD is zero. A number of "canned" CRC algorithms may be selected, but they cannot be modified by further options. These are summarised below. Name Width Poly Initial Xorout R/i R/o Adl crc-32 32-bit 04C11DB7 FFFFFFFF FFFFFFFF yes yes no crc-16 16-bit 00008005 00000000 00000000 yes yes no crc-16i 16-bit 00008005 00008000 00000000 no no no ccitt-16 16-bit 00001021 FFFFFFFF FFFFFFFF no no no cksum 32-bit 04C11DB7 00000000 FFFFFFFF no no yes AUTODIN2 is a synonym for CRC-32 that hints at the heritage of the polynominum. A twelve-byte CRC record is produced by CRC CKSUM (only--specifying ADDLENGTH does not add this field). This record contains the four-byte CRC followed by an eight-byte binary count of the number of bytes included in the CRC. CRC CKSUM interoperates with the POSIX cksum command, which produces the equivalent of crc 32-bit 04c11db7 complement addlength Note, however, that the UNIX command echo -e -n abc|cksum Produces a different CRC than does pipe literal abc|crc cksum because the input data are not the same (x'414243' vs x'818283'). Also note that the POSIX cksum produces two unsigned decimal numbers (the second is the byte count) whereas CRC produces a 2-byte or 4-byte binary number. References: Ross Williams: A PAINLESS GUIDE TO CRC ERROR DETECTION ALGORITHMS. CRC32: Ethernet specifications (Xerox, DEC, Intel), September 30 1980. CRC16: GA22-6844-4 (IBM 2701 Data Adapter Unit OEM). CKSUM: http://www.opengroup.org/onlinepubs/009695399/ utilities/cksum.html DEAL SECONDARY LATCH Like DEAL SECONDARY in the handling of the output steram selection based on records from the secondary input. With LATCH, DEAL waits for a record to arrive on its input streams. When a record arrives at the secondary input, the specified output stream is selected; when a record arrives at the primary input, it is copied to the currently selected output. LATCH implies RELEASE; the record on the secondary input is consumed immediately. The primary output stream is selected initially. IDRDEBLOCK Records written to the tertiary stream are now deblocked according the the IDR format; thus, the first byte is the subtype rather than X'80'. The high-order bit is removed from the subtype flag. The documentation below has been updated. PICK Add the ability to compare up to ten pairs connected with AND or OR. AND may be expressed as that word, "&", and "&&". Likewise OR may be expressed as "OR", "|", or "||". This allows for ands to be doubled and ORs not to be, but self-escaped. Compairson proceeds from left to right. When a pair compares and the connective is OR, the record is matched; when the connective is AND, the following pair is also compared, and so on. When a pair does not compare, pairs are skipped until an OR connector is met, at which point comparisons are resumed; the record is rejected if no OR follows the mismatching pair. POLISH The following applies to the HEX parser only (yes, there is none other). Add the ? suffix operator and the "dot" qualifier operator. The ? operator is monadic in the output record (as the evaluator needs not distinguish between prefix and suffix operators). ? binds tighter than prefix operators. The dot operator may be a binary operator between two or three identifiers or in front of an identifier as a monadic operator. In all cases, the output is a "qualify" record. The qualifiers are written in the reverse order they appear in the expression to cater for the first identifier(s) being optional. The dot opeartor binds closer than anything else, including ?. ? might be interpreted by your evaluator as the indirection operator. Note that the precedence is nowhere near TSO TEST's straight left-to-right evaluation. The expression below might compute the address of the field pointed to by the fullword 12 bytes after the address pointed to by fullword addressed by the contents of register 4 (depending on your evaluator, of course): (r4?+c)? If R4 contains 10 and storage location 10 contains 20, the result could be the contents of location 2C. No parenthesis would be required by TSO TEST to obtain the same result (in fact, TSO TEST does not support parentheses in address expressions). Prefix plus is no longer quietly ignored; you might want to interpret it as meaning a 31-bit address. Passing the string ". v - z+x.y?" to POLISH HEX produces this output and the obligatory null record: qualifier v identifier z binary - qualifier y x monadic ? binary + SPEC When a computed output column is used in a PRINT item, the value printed was in fact the output column. (My test case used the same value for both.) Computed output positions can now include the field length and a placement option: ( [, [, ]]) The first number must evalueate to positive; the second to not negative; and the string must contain the pacement keyword. Note that a literal placement must be enclosed in quotes. A length of zero means take the length from the input data, which was essentially the only option available before. These two output specifications are identical, except that the second one will take somewhat longer than the first one: ... 1.8 c ... ... (1, 8, "c") ... STATE(W) Add support for STRing as for TIMESTAMP. STRUCT Add support for members that are words, fields, and autofields, as in normal input ranges. Such members do not contribute to the length of a structure and they cannot be chained to the previous or following member using Lenght or Next. This is the syntax of a member: * ::= [] * 04500000 * | - * 04510000 * * 04520000 * ::= * 04530000 * | * 04540000 * * 04550000 * ::= [WORDSep | FIELDsep ] * 04560000 * [Word | Field | AUTOField] * 04570000 * * 04580000 * * 04590000 * ::= STRUCT * 04600000 * | Member * 04610000 * | Length * 04620000 * * 04630000 * ::= * 04640000 * | Member * 04650000 * | Next * 04660000 Asterisks are accepted in the range when WORD, FIELD, or AUTOFIELD is specified, but not otherwise. TIMESTAMP While the infrastructure was throughly tested, the interface from the actual filter was not. Happily, this resulted in an error message only. %t Tens and hundredth of a second. (Present in sublevel 15, but not documented.) (And the Single Unix standard requires it to substitute a tab, so this is not portable.) But this is new: %F Equivalent to %Y-%m-%d (the ISO 8601 date format). Requirements: 5 is satisfied by the documentation of CRC. ----------------------------------------------------------------------- Sublevel 17: (March 16, 2008) The syntax for a member definition is amended: * ::= { | - } * 04470000 * * 04480000 * ::= [] * 04490000 * | STRUCT * 04500000 * { | Member | Next } * 04510000 * * 04520000 * ::= { * 04530000 * | Member * 04540000 * | Length } * 04550000 That is, you can specify a filler (an unnamed member) by a hyphen. You can reset the location counter to an already defined member to implement variant records without having to know its position. The blank after the keyword Length is no longer optional. NOTA BENE: The interaction between SUBSTR and nested member names is not yet understood and we invite discussion of this. It would appear that if you have two member names in such a construct, they cannot both be qualified by the same qualifier, as that would imply a recursive inclusion of the qualifying structure. So while you can at the moment specify: ... QUALIFY mystr SUBSTR member left of member right ... It would appear to make no sense. If you decide to use such constructs, the only way to ensure forward compatibility is to avoid the qualifier and specify fully qualified names for all members. SPEC now rejects QUALIFY within a SUBSTR construct. POLISH Parse an expression and generate the reverse polish list of operations to perform. POLISH {HEXadecimal} Each line is parsed according to the specified parser and a list of actions to perform is written. A null line is written at the end of the expression. Output lines contain a word specifying the action type an evaluator would take and whatever type of action this is. These are: constant A numeric constant ("abc" is a numeric constant with the HEX parser). string A quoted string. Both single and double quotes are supported for the string begin character. identifier An identifier (that does not also parse as a hexadecimal constant for the HEX parser). For HEX parsers, the character set is the same as for the High-level Assembler. Case is preserverd in identifiers, but your evaluator may, of course, decide to fold the names. function Call a function. The third word contains the number of arguments to pop off the stack. monadic A monadic operator. The evaluator should modify the stack top. binary A dyadic operator. The evaluator should pop two values and push the result. For the HEX parser, these are the supported operators in order of increasing precedence. All operators are left associative. + - * / % // | & prefix - + Example: Input: max ( g1+10, deadbeaf&30) identifier g1 constant 10 binary + constant deadbeaf constant 30 binary & function max 2 You cannot modify the underlying grammar. In particular, the precedence of the operators cannot be changed. You may implement a subset of the grammar by rejecting, e.g., particular operators. SPEC NOTA BENE: The interpretation of the number after QUALIFY has changed. IF and NOWRITE items were incorrectly ignored during the runout cycle. SPECs now has a qualifier associated with each stream. The QUALIFY construct is enhanced to support this syntax: QUALIFY [ALL] [INPUT | OUTPUT | BOTH] [] Where the default is BOTH. The number specifies the first column of the structure; the default is 1. Note that output streams are selected with OUTSTREAM. The same qualifier is used for both reading stations (even though the BREAK comparison may be between any stream and the second reading). Add built-in functions C2F, X2D, and X2F. C2F takes two to sixteen bytes and converts them from internal hexadecimal float to a counter. X2D and X2F are similar to C2D and C2F, except that the input is a string of hexadecimal digits. An odd unber of digits is prefixed a nibble of zeros (thus it can never be negative). The number specified with QUALIFY is now relative to 1. The desription in sublevel 16 has been updated and enhanced in several places. SAMEASABove REXX is a sample that shows how to use various aspectes of the recent enhancements to SPECs. +============================================= Beginning of SAMEASAB REXX /* Process same as above lines from CP display/dump response */ /* John Hartmann 12 Mar 2008 13:59:29 */ /*%test: pipe < sameas above|sameasabove|cons */ Signal on novalue 'callpipe (end \ name SAMEASAB.REXX:5)', '\*:', '|spec', ' a: substr 2-* of w1 . ', /* Low address */ ' b: w2 . ', /* possibly "to" */ ' c: w3 . ', /* High address, maybe */ ' select second d: substr 2-* of w1 .', /* Previous addr */ ' if eof() then ', ' nowrite ', /* Ignore runout */ ' elseif b=="to" then ', /* An x to y lines */ ' set ', ' ( ', ' #0:=x2d(a); ', /* Current address */ ' #1:=#0-x2d(d); ', /* Stride */ ' #0-=#1; ',/* back up address for pre-increment */ ' #2:=x2d(c) ', /* Limit */ ' ) ', ' while (#0+=#1)<=#2 do ', /* Do all inclusive */ ' 1 1 ', /* R, possibly */ ' pad 0 print #0 strip d2x 2.8 r', /* Address */ ' pad blank 12-* 12 write',/* Data from the original line */ ' done ', ' nowrite ', /* No final null record */ ' else ', ' select 0 1-* 1 ', /* Plain old line */ ' endif ', '|*:' exit RC -=================================================== End of SAMEASAB REXX * SAMEASAB REXX C1 V 71 34 1 3/12/08 18:13:40 ----------------------------------------------------------------------- Sublevel 16: (March 9, 2008) This sublevel adds the ability to refer to symbolic names in and SPEC's Output Placement. Such references are to "members" of "structures". Structures are managed with the new STRUCTURE stage. The structure must be defined to the pipeline before a stage can refer to a member. This is accomplished with the STRUCT ADD stage. The input to the stage follows this syntax: ::= + ::= ":" + ::= ::= STRUCT { | Next} | [] { | {Length }} That is, the input contains the definition of one or more structures. Each structure is opened with a colon followed by an identifier, which is the name of the structure. A structure contains one or more members. A member is specified by its name and either a structure reference or a range. The keyword STRUCT indicates that the member is an imbedded structure is followed by the name of the structure and its position or the keyword Next. A member that represents a data item is defined by an optional type character followed by a range or the keyword Length with a number. Valid type characters are any letter except L (as L is the beginning of Length). The type is translated to uppercase. Types used by SPECs (which is the only program that currently can obtain the type information) are a subset of the ones that can be specified as conversion for SPECs. These are currently assigned: D The field contains binary fixed point, twos complement. F The field contains hexadecimal floating point. Note, however, that unlike the conversion routine mnemonics, these refer to the internal System/360 representation of the data. When SPEC loads data from such a field into a counter, it assumes the internal format of the field when the type is specified; otherwise it falls back on the current behaviour of expecting a number in scientific notation. To be visible to SPEC, the member must be the first in a substring (or not be after SUBSTR at all). If a range is specified, the components must be positive. Specifying NEXT or LENGTH assigns the position after the previous member. When an imbedded STRUCT is specified, it is resolved at the time STRUCT processes the structure definition, not when the member is referenced by an . Structure and member identifers are constructed as valid REXX simple variables. The first character must be a letter or one of the characters "#@$!?_". Subsequent characters, if any, may additionally contain a digit. Unlike REXX, identifiers are case sensitive; they can be any length. Note that member names used as terms in expressions are limited to alphanumerics and underscore as the other characters are operators or have special meaning; a single character is scanned as a field identifier. The scope of the structure definitions can be the thread; or the pipeline set in which they are defined and all nested pipeline sets. In particular, all structure definitions in a pipeline set are discarded when the PIPE command ends or the RUNPIPE stage consumes the record. Structures in the current pipeline set can be deleted with STRUCT DELETE. For member references in inputRanges, structures are resolved first in the current pipeline set; then in the containing pipeline set, and so on. Finally thread scope structures are inspected. To refer to a member of a structure, is modified to accept [Qualify []] Member ::= {"."}* The qualifier remains active for the remainder of the stage's arguments. A new qualifier can be specified; it replaces the qualifier. Specifying a single period with QUALIFY disables the qualifier. The qualifier can be a single identifier, in which case this refers to a structure; or it may be several identifiers joined by periods. If a number is specified after the qualifier, that number specifies the beginning column of the structure; the default is 1. The number less one is added to all member positions. Thus, to specify that structure s begins in column 5, you should specify QUALIFY s 5. Care has to be exercised to ensure that structures are defined before they are referenced, because defining a structure implies reading input records, which must happen on commit level 0. Thus, it is likely that any reference to a structure in the same pipeline specification would not resolve. This can be resolved in three ways. First, one can define the structures in thread scope, but this may make them more visible than desired. Second, one can have the PIPE command run a REXX program that issues CALLPIPE to add the structure definitions before it issues the "real" pipeline, also with CALLPIPE. Third, one can use STRUCT ADD|APPEND ... to issue the pipeline after STRUCT ADD has completed. The advantage of the first two approaches is that one can inspect the return code from the pipeline that loads the structures before issuing the "real" pipeline. -oOo- AUTOFIELD still had a problem with fields relative to the end when a field separator was present at the end of the record. CRC Add EACH to this undocumented stage. The CRC is computed for each record. Without a secondary stream the CRC replaces the contents of the record. APPEND EACH, while currently valid, makes no sense; results are undefined. PICK Has stopped at some unknown time to issue error messages when numeric compares received non-numeric data. This applies both the constants in the parameter list and to input ranges. QUERY Add SUBSTITUTIONS. This writes a single line containing up to four items of substitution data from the last message issued. The format of a substitution item is described in PIPMITEM in FPLOM MACLIB. When used in conjunction with QUERY MSGLIST, this can allow an application a way to capture failure data without going through the contortions of RUNPIPE. Nota Bene: Character items (highorder byte of length zero) are not reliable as their addresses are usually point into a copy of the pipeline specification string that is long gone, but return codes from underlying commands should be obtainable. SPECs Add several built-in functions, a facility to iterate, and the ability to reference members of records. Built-in functions: DELTA() Convenience for PRIMARY(x)-SECONDARY(x). One of the terms is zero on the run-in and on the run-out cycle. DELTA implies both SELECT 0 and SELECT SECOND. FIELD(, [, ]) Return the nth field of the string. The number must be nonzero. The field number is counted from the end of the string when the number is negative. The character argument must have length 1; it specifies the field separator; the default is X'05', horisontal tab. PRESENT() Return 1 when the field is present in the record and 0 otherwise. LENGTH() for a field is 0 both when the field is present, but null, and when the field is not present. PRIMARY() Return the numeric contents of the member from the first reading. The member must have a type that is D or F. PRIMARY implies SELECT 0. RECORD([ [, ]]) Return the current input record or a substring of it. If present, the first number specifies the starting position of the result within the record; it must be nonzero. When this number is negative, the position is relative to the end of the record. The second number specifies the maximum number of columns to include in the result. The default is to the end of the record. The second number must be nonnegative. The result is never padded. This facility implements the equivalent of the computed output placement. SECONDary() Return the numeric contents of the member from the second reading. The member must have a type that is D or F. SECIND implies SELECT SECOND. SUBSTR(, [, ]) Return a substring of the string argument. The first number specifies the starting position of the result within the record. The second number must be nonzero. When this number is negative, the position is relative to the end of the record. The second number specifies the maximum number of columns to include in the result. The default is to the end of the string. The second number must be nonnegative. The result is never padded. TYPE() Return the data type associated with the field (if it is a member of a strucure) or a single blank. WORD(, [, ]) Return the nth word of the string. The number must be nonzero. The word number is counted from the end of the string when the number is negative. The character argument must have length 1; it specifies the word separator; the default is blank. Note that it is not possible to assign a string result to a counter. You may specify a member name as a term of an expression. If a qualifier is in effect, it is observed. If there is no qualifier in effect, you must specify the fully qualified name starting with the name of the structure; such references assume a structure offset of zero. The member must have a type of C, D, F, or no type. It is converted as appropriate for the type. A member that has the type C is considered a string and can be used only where a string value is acceptable. Type D and F members are numeric; untyped members are considered as containing a character string representing a number. For the expression parser to recognise a member name, it must not contain an exclamation point or a question mark, as these are operators in the expression language. A number sign (#) is scanned as a counter number when it is followed by only digits; otherwise it is scanned as a member identifier. A single character is scanned as a field identifier if a field has been identified by that name; otherwise it is taken as a member identifier. Symbolic input ranges: When loading the contents of a field that is specified as a member into a counter and the member is defined with a type, types D and F are interpreted to mean fixed point twos complement binary, and hexadecimal floating point, respectively. Other types may be specified, but they will be ignored; either way, such members are converted from printable scientific notation, as before. This applies to a reference by a single-letter identifier, not to a direct reference to a member. When an output field is specified as a member with the type D or F and the input source is a counter (PRINT) without a picture, the floating decimal number is converted directly to the requested representation. Similarly, if the input source is a member that also has a supported type, the conversion is directly between the two formats. In either case, STRIP and explicit conversion also reverts processing to untyped data. Istead of specifying QUALIFY in concjunction with MEMBER, as described in general input ranges above, SPEC scans for the keyword QUALIFY on its own. QUALIFY [INPUT|OUTPUT|BOTH] [] The number specifies the column where the qualifier is based. The default is 1. That is, QUALIFY str.sub 7 means that the structure "str" is assumed to begin in column 7. "sub" is an imbedded structure with "str". Iteration: A series of specification items are iterated using the new WHILE construct: WHILE DO DONE While and if can be nested to a depth of 16 (unchanged). Do be careful to have the iteration terminate. There are no safeguards. Eample: Assuming that DIFFMATRIX produces a number of records containing a number of abutted fullword counters, this might format for comfortable viewing. diffmatrix |spec set #0:=0 while #0... Remove a structure from the current pipeline set or from thread scope (if THREAD is specified). Additional identifiers may be passed on the primary input stream. All deletions in the argument string or an input record are processed as a whole; they are either all removed or they all remain. In general a structure must not be referenced from other structures to be removed, except when those referencing structures are also removed in the same input record (or in the argument string). To remove the single structure THREAD, pass the literal on the primary input stream. STRUCTURE LIST ... Additional identifiers may be passed on the primary input stream. The structure definition is written to the primary output. The output can be saved and passed to STRUCT ADD except when the structure is referenced (in which case a reference count is shown in parentheses). In general, however, the output CHOPped at a left parenthesis can always be fed back to STRUCT ADD. STRUCTURE LISTALL [MEMBERS] List all active structure definitions and the relative set level where they are defined. When MEMBERS is specified, the members of the structure are also listed (as for STRUCT LIST); otherwise just the structure name is written. THREEWAY Message 222 was issued when the secondary stream is present, but the tertiary is missing. EOF on output was not handled correctly. It looked like 3WAY would terminate as soon as it detects EOF, but it also consumed the record. It now does what the book says; it terminates when all output streams have been sensed at EOF without consuming the input record where this is discovered. Requirements: 43 is satisfied by the structure stuff. ----------------------------------------------------------------------- Sublevel 15: (December 15, 2007) The concept of internal wait is introduced. Such a stage will either be posted by other stages in the pipeline set (WARP), or it does not wish to cause the pipeline to enter a real wait state (IMMCMD INTERNAL). Thus, a stage that uses internal wait cannot obscure a stall. It has for a while been random whether STALLACTION STATUS was observed or not. An assert B2 could result after a terminating a stage due to the STOPERROR action. When STOPERROR was triggered, stages might not have been resumed correctly. A message is issued when a pipeline specification is committed if the global option TRACE is specified for the pipeline specification. Message 28 has been enhanced with the commit level where the stage starts. The pipeline stall dump did not format the stage's commit level correctly. The design of handling committing a stage that waits for a pipeline specification without active stages was flawed. Fix the undocumented AUTOFIELD in conjuction with substrings. Once AUTOFIELD was specified, it would remain on for the rest of the field defenition parse. FINDINST Look for instructions in TXT data. FINDINST [NOSUPervisor] [NOFP] {DOS | 370 | XA | ESA| ZOP | YOP | ZS3| ZS4} A rather arcane program, but when you need it, you need it real bad. The delimited string contains an instruction stream, e.g., x0A0B. It may contain several instructions. It must contain an even number of bytes. NOSUPervisor means that privileged instructions are not considered valid. This should be specified for an MVS program that is not authorised and when a CMS program is known not to use privileged instructions. NOFP means that floating point instructions are not considered valid. In addition binary and/or decimal floating point may be considered invalid, depending on the operation code table selected. The last operand specifies the operation code table to use. See High Level Assembler User's Guide. The input record contains a four-byte prefix followed by the text of a control section. The prefix contains the the assigned address. An output record is produced for each occurrence of the delimited string that is at an even address. It contains: 1 4 Assigned address of the signature. This is the contents of the first four bytes of the input record plus the offset to the beginning of the signature. 5 4 Reserved; alignment. 9 8 First eight bytes of input record (normally beginning of section). 17 16 Sixteen bytes context before the string. 33 16 Sixteen bytes context from the string. 49 4 Binary count of valid instructions from the match. 53 4 Binary count of BAL, BAS, BASSM, etc., met. 57 1 Flag byte .... ...1 The instruction at the beginning of the delimited string is always executed. That is, it is a definitive instruction boundary. .... ...0 The instruction at the beginning of the delimited string may be a subset of a previous instructions. That is, this may be a false positive. .... ..1. A program check will occur if the instruction sequence were to be executed. Thus this is a negative match. .... ..0. The program will reach an unconditional branch or a BAL-type instruction were it to be executed at the point of match 58 1 Reason code. Binary integer that explains how the instruction sequence terminates. 1 An unconditional branch is reached. 2 The end of the CSECT is reached. 3 A privileged operation is reached (NOSUP). 4 A hex floating point operation is reached. 5 A binary floating point operation is reached. 6 A decimal floating point operation is reached. 7 Stores into page 0 (0C4) 8 Odd number for even/odd pair (0C6) 59 4 Popularity index for three instructions. 59 First instruction in signature. 60 Two bytes before signature. 61 Four bytes before signature. 62 Six bytes before signature. The popularity index is an indication of when the instruction was introduced. The binary values are: 0 Not a valid instruction. 1 Base 370 with extensions. 2 XA with extensions. 3 ESA with extensions. Notably, G3 level set. 4 Basic z/Architecture. 5 z900 dyslexic instructions. 6 z9 instructions. 7 z10 instructions. IDRDEBlock [ FIRST | LAST | BOTH ] Process IDR records in a load module (PDS format). The input record must be 4 to 256 bytes and the first column must contain X'80'. IDRDEBLOCK processes only type-4 records, which contain translator identification. The primary output stream contains the 15-byte or 30-byte translator identification, depending on the precence of two identification fields and the operand. Clearly, if one only is present, the question of the operand is moot. If BOTH is specified or defaulted, all 30 bytes are written; if FIRST is specified, the first 15 bytes are written; and if LAST is specified, the last 15 bytes. The list of CESD numbers is written to the secondary output after the record on the primary stream. The flag idicating end of list has been removed. Other identification records are passed to the tertiary output, if one is defined. The data includes the flag byte (offset 2) for the length in the second byte. The flag indicating end of IDRs is removed in the first byte of the record. IDRDEBLOCK starts at commit level -2; it verifies that the primary input stream is the only one connected. It terminates on EOF on the primary output or the secondary output. IMMCMD Add the INTERNAL operand after the immediate command name. When INTERNAL is specified, the immediate command will not of itself cause a wait state, but it will wait while other stages wait for external events (even other IMMCMD stages). Essentially, this boils down to "do not wait for me". IMMCMD ... INTERNAL cannot obscure a stall. LOOKUP The master is reset when a record arrives at the quinternary input stream. Before resetting the master, LOOKUP writes records to the quarternary output describing the contents of the master file, as it does on the tertiary output prior to termination. All streams remain connected. PIPESTOP cannot be first in a pipeline; this is not documented. SPACE The operands are parsed incorrectly in some cases when the number is 0. STRING was not rejected and ANYOF not processed correctly. TIMESTAMP Make year 2100 ready. Add formatting option. Supply tenth and hundredth of a second. TIMESTAMP [ { [] } | STAndard | ISO | SHOrtdate | FULldate | STRing ] The formatted timestamp can be the "raw" 16-byte sorted timestamp, a predifined format, or a custom format. When the operands are numeric, the first number specifies the position, relative to the end of the formatted timestamp, of the first character to include; it is quietly limited to 16. The second number specifies the count of characters to include. The default is the same as the first number. It is quietly restricted to this number. Standard, ISO, short, and full work as for STATE. The length of the formatted timestamps are: Standard 14 ISO 19 Full 19 Short 17 STRing specifies a custom format in the POSIX style for strftime(). The delimited string specifies formatting. Substitutions are indicated by %; other characters are copied to the formatted string unchanged. The conversion strings that are recognised are: %% Insert single % %Y Four-digit year including century %y Two-digit year of century %m Two-digit month (01-12) %n Two-digit month with initial zero changed to blank ( 1-12) %d Two-digit day of month %e Two-digit day of month with initial zero changed to blank ( 1-31) %j Julian day of year (TIMEstamp only) %H Hour, 24-hour clock (00-23) %k Hour, 24-hour clock first leading zero blank ( 0-23) %M Minute %S Second %T Short for %H:%M:%S %t Tens and hundredth of a second. TXTDEBlock Process control and text records in a load module in PDS format. A record is written to the primary output stream for each section CESD ID present. The record contains four bytes binary data. A record is then read from the secondary input. The section is processed if the first byte of this record contains c'1'; the section is ignored otherwise. The text data for the section is then written as one or more records on the secondary output. Such a record contains the CESD ID (four bytes), the assigned address (four bytes), and the text itself (variable). RLD data are ignored. Other record types are passed to the tertiary output, if one is defined, or discarded. WARP Pipeline warp. WARP The word specifies the wormhole's name. Eight characters, case respected. Each pipeline set (PIPE command, RUNPIPE) can have only one wormhole by a particular name. When WARP is first in the pipeline, it waits for records to fall out of the wormhole and then copies them to the output. Such a WARP stage is called a catcher WARP stage. There can be only one catcher for a particular wormhole for the entire pipeline set. When WARP is not first in the pipeline, it sends its input records through the wormhole. Such a WARP stage is called a pitching warp stage. There can be any number of pitchers in a pipeline set (and they can be added with CALLPIPE). It will terminate without consuming the record if the wormhole cannot write the record. Records are then copied to the output, but EOF on output is ignored. That is EOF is propagated from the wormhole listener's output to the senders' input. A pitching WARP stage waits for the wormhole to consume the record. WARP starts at commit level -20. When it is first in the pipeline it verifies that no other wormholes exist with the specified name and then commits to zero. When it is not first in the pipeline, it commits to -1 to give a catcher time to start; if there is still no catcher, the stage terminates with an error message; otherwise it commits to zero. The workings of WARP are similar to a unix named pipe. In fact, a pitcher may establish a catcher to receive a reply and send its name in the message it pitches. Though in general, such practice is not considered pipethink. WARP sets the internal wait flag, which means that it cannot cause the pipeline to go into a wait state; thus it cannot obscure a stall. HEALTH WARNINGS: Apprentice plumbers should probably look elsewhere. WARPLIST Write a record for each active wormhole. ----------------------------------------------------------------------- Sublevel 14: (June 29, 2007) Scanning an for words where the first number is negative and the second number is positive returned a word too many. E.g., pipe literal one two three|substr w-1;2|cons should get you a null record, but it got you the third word. Fix a potential storage leak in conjunction with CMS Multitasking that was introduced in sublevel 6. HLASMERR would overrun its output buffer when the assembler produces invalid format ADATA message records. (E.g., ICTL without operands.) PICK Fields were not compared numerically when the operand string has trailing blank(s). TRFREAD The SPOOL file was not closed in some cases where EOF propagates into TRFREAD. ----------------------------------------------------------------------- Sublevel 13: (January 15, 2007) The new edition of the Author's Edition and help files applies to this sublevel. In the process of updating it, more test cases were added and several minor problems and discrepancies were solved. Thus, the documentation in the book is authoritative. ----------------------------------------------------------------------- This file documents CMS Pipeline enhancements since 1.1.11/0013 For enhancements from 1.1.10(VM/ESA 2.3.0) see PIPELINE NEWS1111 For enhancements from 1.1.9 (VM/ESA 2.1.0) see PIPELINE NEWS1110 For enhancements from 1.1.8 (VM/ESA 1.2.2) see PIPELINE NEWS119 For enhancements from 1.1.7 (VM/ESA 1.2.1) see PIPELINE NEWS118 For enhancements from 1.1.6 (VM/ESA 1.1.1) see PIPELINE NEWS117