Pipeline enhancements since 1.1.9 (VM/ESA Version 2 Release 1 Mod 0) (c) Copyright IBM Corp. 1995-1997. DISCLAIMER: =========== This file describes enhancements that were made to bring CMS Pipelines up to level 1.1.10. These enhancements are included in CMS14. Refer to PIPELINE NEWS for a statement about support. Refer also to the CMS Pipelines home page: http://pucc.princeton.edu/%7Epipeline The "CMS Pipelines Run-time Library", is available for download from the CMS Pipelines home page. References to requirement numbers refer to the fifty requirements submitted by SHARE Inc. in March 1996. See http://pucc.princeton.edu/%7Epipeline/sharereq.html The enhancements are subject to change without notice. Including complete withdrawal. Sometimes things just don't pan out. This file is written bottom-up. The latest items are first. Pre-requisite service: ====================== CMS APAR VM49553 is required for PIPE HELP to work on CMS7 and CMS8. HLASM PTF PN69921 is required for "hlasm" and "hlasmerr" to work. CMS APAR VM61261 is recommended for users of CMS 13. CMS Pipelines has been changed to circumvent the failure, but other OS simulation applications may fail. ----------------------------------------------------------------------- Level 1.1.10 sublevel 20 (decimal 32) is the level integrated with CMS14. SQL Connect with both userid and database was rejected with syntax check. Code now follows the SQL syntax strictly. Thus, anyone who have discovered to elide IDENTIFIED BY will punished in future. ----------------------------------------------------------------------- The PIPDESC macro created a reference to an undefined symbol when used outside a PL/j procedure. The pipeline level was not stored correctly in the Pipeline Services Transfer Vector (it was all zeros). Minidisk device drivers for V format files tested a more or less random byte to suppress the fast interface. This could let to > reporting message 134 even when writing a V-format file. DATECONVERT More fixes. HELP Insist on being a first stage. JEREMY/PIPDUMP a few changes. SQL CONNECT TO padded the database name with nulls rather than blanks. This leads to SQLCODE -940 and similar, depending on the SQL release. -------- Shipped to VMTOOLS 15 Oct 1997 17:22:07 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel 001C (28 in decimal) >SFS (and other SFS write drivers). Process file mode number correctly. Retain existing mode number when replacing unless an explicit mode is specified. JEREMY and PIPDUMP followed a stale pointer when a stage has terminated after streams had been added with ADDSTREAM. DATECONVERT More fixes. STATE(W) Should show -1 (hyphen and mode number) when the directory is not accessed. The mode number was not stored. -------- Shipped to VMTOOLS 19 Sep 1997 14:55:47 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel 001B (27 in decimal) Restored some messages that were deleted in error. -------- Shipped to VMTOOLS 18 Sep 1997 10:11:57 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel 001A (26 in decimal) The values for configuration variables are now uppercased. The default for DiskTempFileType is now CMSUT1 in all styles. With the proper PIPELINE HELPLIB file, HELP MENU now supports an additional keyword, BUILTINS, COMMANDS, HOST, MESSAGES, OTHER, SYNTAX. BUILTINS is the default; all menus offer the other menus. -------- Shipped to VMTOOLS 6 Sep 1997 11:43:58 ------------------------ PIPELINE MODULE Modification 1.0110 sublevel 19 (25 in decimal) When writing a file, the SFS drivers would issue a generic message when a file is not writable because it is a directory control directory that is accessed read only (or somebody else has accessed r/w). DATECONVert is now updated with the latest from its author. IF Changed to support propagating EOF forward. Thus, it now works as expected, also with partitioning selection stages. But partitioning selection stages sever an output stream; so the dispatch order of the stages after IF and the selection stage is undefined after the selection stage severs its primary (or secondary) output. -------- Shipped to VMTOOLS 1 Sep 1997 11:30:40 ------------------------ PIPELINE MODULE Modification 1.0110 sublevel 18 (24 in decimal) PIPMOD DEFAULT is now removed. The SUBSTR keyword in an inputRange can now be written SUBSTRING as well. The default STALLACTION in the FPL style is JEREMY; this was changed in sublevel 16, but was not documented. LOOKUP The increment was not deleted from the detail record before it is passed to the primary output. Deleted master records are not written to the quaternary output. STRLITERAL Add keyword APPEND (before the string). The record is then inserted after the contents of the input rather than before. PREFACE is allowed as a default. AFTER is also recognised. The delimited string is now optional; a null record is used when it is omitted. -------- Shipped to VMTOOLS 18 Aug 1997 11:33:16 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel 17 (23 in decimal) The DISKINTERFACE configuration variable has been dropped. When using a message repository, the severity code is now displayed correctly. CONFIGURE Interface to the configuration variables added in sublevel 16. This replaces the FPLOPTQR service aid. When CONFIGURE is first in a pipeline, it writes all configuration variables that are currently set to the output. When CONFIGURE is not first in a pipeline, non-blank lines contain the name of a configuration variable and optionally a value or keyword to set. When a variable is changed, the change takes effect immediately and across all running pipelines. Caveat emptor. The output record is the name and (possibly new) value of the variable. If an unset variable is queried, the value is established at that time from the global variable repository or from its default. PIPMOD DEFAULT will be removed in the next sublevel. STATE(W) Now sets the real file mode of the file, when this mode is available. They are further modified when a directory path is specified as the third word. If the file resides in a directory, the fully qualified path to the actual directory is appended as described below. Consider this scenario: H R/W 165 DIR DKIBM2JH:JOHN.FPL.DELTA J/H R/O 637 DIR DKIBM2JH:JOHN.FPL.SRC K/H R/O 6549 151 CONCPG Further, file FPLACN ASSEMBLE resides on mode J; file PIPACN TEXT resides on mode K. "STATE FPLACN ASSEMBLE .FPL.DELTA" will show mode J and DKIBM2JH:JOHN.FPL.SRC as the directory path. "STATE PIPACN TEXT .FPL.DELTA" will show mode K and no directory path. -------- Shipped to VMTOOLS 23 Jul 1997 14:17:45 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel 16 (22 in decimal) Endicott/Copenhagen Convergence =============================== Over time the "Endicott pipeline" and the "Copenhagen pipeline" have diverged in some subtle ways. Apart from confusing customers, maintaining two versions is also wasting money for IBM. Thus, the two versions are being consolidated into one base from which they will be generated. Apart from the default style (which will be described shortly), the contents of the two modules will be identical. The Endicott module will continue to be named DMSPIPE and the Copenhagen module will continue to be called PIPELINE on VMTOOLS and on the CMS Pipelines home page. The bootstrap modules (PIPE) will remain unchanged. Simply changing the behaviour of either module will not cut it; there is compatibility to consider. However, the Endicott module is bowing towards Copenhagen; it will use the original message numbers. Thus, anything that parses the output of PIPE COMMAND PIPE xxx will break if the output includes error messages. Also, the output from RUNPIPE (omitting EVENTS) will change. Dramatically. Also, the return codes will be the Copenhagen ones rather than the ones associated with the DMS messages. Default Styles -------------- To determine how the pipeline should behave in other instances where there traditionally used to be a difference between the Copenhagen and the Endicott version, it now maintains defaults, which it derives from the variables set in the GLOBALV group FPL (by default) and in lieu of a GLOBALV, some built-in defaults. The command PIPMOD DEFAULT can be used to change the setting for a default at any time; it takes effect immediately. (And it works even on CMS5.) If a default has not been set by the PIPMOD command before its value is needed, the corresponding variable is fetched from the GLOBALV pool (for CMS5 and previous, all global variables are considered unset). Once fetched, the value is cached within the pipeline module; the repository is not queried again. The variable STYLE is a summary way of making one's views known without going through the nitty-gritty of setting all of the variables. A value of DMS for the STYLE variable will select the traditional Endicott style; PIP will select the traditional Copenhagen style; FPL will select the "preferred" style. In the FPL style, the APPLID in messages is FPL rather than PIP; other defaults are also different, as noted below. If the variable STYLE is not set, DMS is the default style for modules generated in Endicott; PIP is the default style for modules generated in Copenhagen. Defaults -------- REPOSITORY Controls how messages are issued. The REPOSITORY default specifies the application identifier for the message repository to use. The default is FPL in DMS style (and that is expected to be in the system shared segment by the time it ships); it is - in the PIP and FPL styles. - means not to use a message repository, but to use the author's English message. The good news is that the pipeline will cope gracefully with missing message repositories &c. If all else fails, it will have the author's message squirrelled away; and it will issue it. GROUP The GLOBALV group to consult. The default is FPL in all styles. The group can be changed only by the PIPMOD DEFAULTS command. DISKTEMPFILETYPE Controls how >mdsk replaces a file in a directory that is accessed as a mode letter. The problem here is that identical server machines that perform identical actions will also use identical file names and file types for the temporary files; and that causes havoc when the files are in the same SFS directory. The real solution to this problem is the >sfs driver (or simply using the directory rather than the mode letter), but a tactical solution was devised in response to an APAR. TOD (which is the default in the DMS style) creates a work file the name and type of which is determined by the TOD clock; this name will be unique across the entire system, but not necessarily across the collection. CMSUT1 (which is the default in the PIP and FPL styles) uses the filetype CMSUT1 and a file name related to the particular stage; this will be unique within the virtual machine. USERID uses the user ID as reported by diagnose 0 for the file type. The file name is related to the particular stage. Thus, for all virtual machines that use the USERID option, the temporary file will be unique, but also easily found. (This is an enhancement.) SQLPGMOWNER The SQL/DS user ID by whom the access module is prepared. The default is DMSPIPE in DMS and FPL styles; it is 5785RAC in PIP style. SQLPGMNAME The name of the access module. The default is DMSPQI in DMS and FPL styles; it is PIPSQI in PIP style. STALLACTION How much information to provide on a stall. The default is STATUSDUMP in DMS and PIP styles; it is JEREMY in the FPL style. QUIET Issue message 29 only. No dump. JEREMY Issue message 29 only. Use JEREMY to dump the status of stages. STATUS Issue message 29 and message 30 for each stage. No dump. STATUSDUMP Issue message 29 and message 30 for each stage. Produce PIPDUMP dump. STALLFILETYPE The file type to use for the dump on a stall. The default is LIST00* in DMS style; it is LISTING in PIP and FPL styles. The asterisk indicates that CMS Pipelines should try to write an individual file for the dump; only when all 100 files (LIST0000 through LIST0099) exist is the dump appended to the last file looked for. Specifying LIST* would allow for 10,000 individual dumps. To make a setting permanent, use GLOBALV; for example: GLOBALV SELECT FPL SETP STALLACTION JEREMY Other general Changes ===================== The pipeline now uses the FPL module prefix for its modules. Several message texts have been revised. The actual installed module names are now substituted in some messages (PIPMOD and PIPE, by default). This means that the actual command to issue will be correct even when the pipeline is installed in a non-standard way. The pipeline now supports recursions on different processes and threads. To this end, the SCBWKWRD of the PIPRUN nucleus extension now points to an area that is global to all threads. The pipeline has no concept of the "current thread"; pipeline sets run completely in parallel on their respective threads. How control passes between threads is none of the pipeline's business. It is the intent that this multithreading will also support and be supported by Sterling Software's VM: multitasking subsystem. When and whether this will happen is clearly out of IBM's control (or at least not entirely within IBM's control ;-) The PIPE command incorrectly processed the flags in the user save area; this means that for a while it has refused to run when invoked by SVC 202. The support to use suspend and resume using MT introduced in sublevel 7 was incorrect and would lead to a wild branch if it were ever used. The PIPDUMP EXEC is no longer used at a stall. See below. Before using the CVT pointer in low storage to find MVS services, which are also provided on CMS, the pointer is now sanity-checked. Users who turn off the PCP flag in offset x'74' will now receive a nuisance message, but CMS Pipelines will never behave as TSO Pipelines (unless you know where to zap the module). An is now scanned according to this syntax: ::= * {SUBSTR OF}* ::= {FIELDSEPARATOR | WORDSEPARATOR} ::= * [WORD | FIELD] and so on Previously, the leading was scanned by SPECs only, but not by stages such as LOCATE, CHANGE, &c. Individual stages: ================== >sfs Incorrect message when creation date is specified for an >>sfs existing file. Modification date processed incorrectly. CHOP Support ANYCASE as the first keyword. CURRDIR [] [SFS|POSIX] Experimental. Maintain a current directory similarly to POSIX. There are a few wrinkles. The input range specifies a subset of the record. The input range can contain an absolute directory, a relative directory, a hyphen, or nothing. A relative directory can refer to the parent directory; this is done with a single hyphen in SFS style and two dots in POSIX style. SFS means SFS-style directory structure using dots to separate levels. It uses the DMSVALDT function under the covers. POSIX style uses slashes to delimit the components. A hyphen switches back to the previous directory; the current directory becomes the previous, so multiple lines containing a single hyphen will toggle. A blank or null input range writes the current directory. DAM Verify that the primary output stream is not connected GATE before looking to propagate EOF between streams that are not connected when the stage starts. DATECONVERT Perform date conversion and validation. DATECONVert is a front end to the underlying CSL routine, DateTimeSubtract. This is an experimental filter. It requires VM/ESA 2.2.0 or a later release. In practice, you'll need a DMSDTS CSLLIB file that contains the latest DateTimeSubtract. You can obtain this from the Pipelines home page (see above for the URL). DATECONVert [] [ []] [WINDOW | [WINDOW] BASEYEAR ] The default is 1-* shortdate isodate window -50. The output record written to the primary output stream contains the converted date (if the input field contains a date that is valid for its format). If the input field does not contain a valid date in the particular format, the record is passed to the secondary output stream, if one is defined; otherwise DATECONVERT terminates with an error message. Valid input and output formats are: FULLDATE ISODATE SHORTDATE Rx ISO_SHORT CSL PIPE_SHORT EUR_SHORT EUR JULIAN_SHORT JULIAN TOD_ABSOLUTE SCIENTIFIC_ABSOLUTE POSIX TOD_RELATIVE SCIENTIFIC_RELATIVE MET REXX_DATE_E_LONG REXX_DATE_J_LONG REXX_DATE_N_SHORT FULLDATE, ISODATE, and SHORTDATE are as for other CMS commands, such as LISTFILE. Rx specifies a REXX date format, where x is the first letter of the first argument to the date() function (i.e., B, E, J, N, O, S, U, C, D, M, or W). The second list of formats are the ones recognised by the underlying CSL routine, DateTimeSubtract. The third list of formats are extensions of the Rexx Date functions REXX_DATE_E_LONG (dd/mm/yyyy) and REXX_DATE_J_LONG (yyyyddd) and REXX_DATE_N_SHORT has a 2 digit year (dd mmm yy). For interpreting two-digit years, DATECONVERT uses by default a sliding window that begins fifty years in the past. This is equivalent to WINDOW -50. You can specify any signed number. The BASEYEAR keyword specifies that the window does not depend on the current year; it begins at the year specified. Note that BASEYEAR 50 specifies a window that begins almost two thousand years ago. Be sure to specify a four-digit year. FITTING Restore the fitting name in the RPL when the stage terminates. The RPL can be reused with a subsequent pipeline specification if its state is changed to one of the acceptable initial states. FPLxxxxx These built-in programs are reserved for CMS/TSO Pipelines use. They are not in general documented. FROMTARGET When used with two output streams, EOF was propagated backwards from the secondary output. This leads to the primary output receiving records before the target. HLASM Updated to support release 2 of the assembler. HLASMERR JEREMY Is now part of the pipeline module. Whenever it reads a record, it writes the pipeline status to an output stream. When the secondary output is defined, the pipeline status is written to that stream and the input record is then passed to the output. Otherwise the dump is written to the primary output and the input record is then discarded. JEREMY does not delay the record. LOCATE Support ANYOF to indicate that the target is any of the NLOCATE characters in the string. Unlike CHOP et al, this ANYOF is not the default ;-) There is also a NOT which is like CHOP &c's NOT, but it really is for testing only. LOOKUP Add SETCOUNT, TRACKCOUNT, and INCREMENT. Make counter stick at INT_MAX (rather than wrap to negative). SETCOUNT specifies that the initial count for the master record is provided in the first ten columns of the master record. These ten columns are not stored in the reference, even when KEYONLY is omitted. The input range for the master should not take these ten characters into account. TRACKCOUNT specifies that the current counter is prefixed to the master record when it is written to the primary output. INCREMENT specifies that the amount to increment is specified in the first ten columns of the detail record. This field is deleted before the record is matched. The input range should not take this field into account. COUNTSOFAR is a convenience for COUNT TRACKCOUNT. COUNT affects the tertiary and quaternary outputs only. NOEOFBACK Pass input to output without propagating EOF backwards. (Needed to fix TO/FROMTARGET.) PIPDUMP Write a dump of the pipeline's control blocks to the output stream. Any arguments are used as heading text. Apart from the contents of the output, its behaviour is like JEREMY. RETAB Ran out of storage. SPLIT Support ANYCASE as the first keyword. STATE(W) Support SFS directory names. If the directory is accessed as a mode, the mode letter is returned and the request is processed as if the request was entered with that mode letter. If the directory is not accessed, the mode letter is set to a hyphen (-). When NOFORMAT is specified and a directory is specified that is not accessed, a skeleton FST is built and written. (The directory is not written.) When NOFORMAT is omitted and a directory is specified that is not accessed, the output record contains the full directory path after the timestamp. That is, the directory path is word -1. STRIP Support ANYCASE as the first keyword. STORAGE Test correctly for end of virtual machine storage; it incorrectly accepted the byte just beyond the end of the virtual machine. TOTARGET When used with two output streams, EOF was propagated backwards from the primary output stream. This leads to the secondary receiving records before the target. UPDATE Terminate with return code 0 when an output stream sees end-of-file. WELLIE (That is, all the xxOE and HFSxxxx stages.) Messages 709 and 710 were substituted incorrectly. Access to the shell's filedescriptors for stdin/out/err did not work on CMS when the files were connected to the terminal. In fact, CMS PIPE from the shell got CMS very confused. Add reason codes to some messages. REXX Interface Enhancements for Parsing ======================================= These enhancements are done to address SHARE requirement 22. With these facilities, as much of the requirement is addressed as is feasible within the existing structure for CMS Pipelines built-in programs. There is no simple way to parse an argument string the same way a built-in does. This part of the requirement is likely to be rejected. TESTKEYWORD is now the preferred spelling of PIPTSTKW, which was introduced in sublevel 6. (Since the list of keyword names is not published, this may be of limited practical utility, but it certainly is useful for me.) SCANRANGE {OPTional|REQuired} {|.} [] SCANRANGE can be used by the filter programmer to parse an argument string containing an inputRange specification in the same way that CMS/TSO Pipelines built-in programs scan their arguments when an inputRange can be specified. SCANRANGE scans the beginning of the string according to the inputRange syntax and sets two variables. The first variable is set to a token representing the inputRange specification; it can be used in subsequent GETRANGE pipeline commands to specify that input range. (The contents of this token are unspecified; other tokens can represent other inputRanges.) The second variable is set to the unscanned portion of the input string (that is, the part of the string that was left after an inputRange was scanned). The first word of SCANRANGE's argument string is a required keyword. The second word is the name of the variable to receive the token representing the inputRange specification. The third word is the name of the variable to receive the residual string; specify a period to discard the residual string. The remainder of the argument string is the string to be scanned. Leading blanks are ignored. When OPTional is specified, a missing or incorrect inputRange specification is treated as 1-*; otherwise a non-zero return code is set when an inputRange cannot be scanned. Note that the keyword is required. Note that and are names of variables, which are set, but is a string. SCANRANGE is typically used at the beginning of a REXX filter where it scans its arguments string. For example, to scan an argument string that may specify one input range but no other arguments: parse arg argstring 'scanrange optional range_definition rest' argstring if RC^=0 then exit RC /* Messages are already issued */ if rest^='' then call err 112, rest /* Too much */ To scan an argument string that must specify two input ranges and no other arguments: parse arg argstring 'scanrange required first rest' argstring if RC^=0 then exit RC 'scanrange required second rest' rest if RC^=0 then exit RC if rest^='' then call err 112, rest /* Too much */ Note: The token representing the input range remains valid only as long as the stage is running. Once the stage has terminated, the contents of the token are stale. Using it (e.g., in some other REXX filter) may cause random ABENDs. GETRANGE {VARiable|STEM} [] GETRANGE can be used by the filter programmer to extract the part of a record to be processed in the same way CMS/TSO Pipelines built-in programs select a substring of the input record. The location of the substring to be extracted is defined by a token representing an inputRange specification; such tokens are created by the SCANRANGE pipeline command. The first word of the argument string contains the name of a variable whose value has been set by SCANRANGE. The second word is a required keyword. The third word contains the name of a variable or a stem to be set to the contents of the substring. The remainder of the argument string (after exactly one blank) is the string from which the substring will be taken. The value of the variable whose name is must have been produced by the SCANRANGE pipeline command. The content of this variable is unspecified; it must not be changed by the REXX filter. When VAR is specified in the second word, represents the name of a single variable, which is set to a substring of the input string, as determined by the contents of the token. When STEM is specified, is the name of a stemmed array, as defined for the STEM device driver. When the substring is not present in the string, as opposed to being of zero length, the compound variables are set as follows: 0 1 1 The input string. When the substring is present in the string, the compound variables are set as follows: 0 3 1 The part of up to the beginning of the input range. 2 The contents of the specified input range. 3 The balance of the string. Note that and are names of variables, which are set, but is a string. Also note that the string begins after exactly one blank. Example: parse arg inputRange 'scanrange required field rest' inputRange if RC^=0 then exit RC signal on error do forever 'peekto line' 'getrange field var range' line 'output' reverse(range) 'readto' end error: ... SCANSTRING {|.} [] Scan the beginning of the string according to the syntax for a delimitedString. Leading blanks are ignored. Store the string into the variable that is named by the first word; store the residual string into the variable that is named by the second word. (Or discard it if the second word is a period.) Example: parse arg argstring 'scanstring word rest' argstring Argstring could contain "/abc/" or "xf1f2f3". Note that and are names of variables, which are set, but is a string. Control Blocks ============== IBM does not encourage looking into the CMS Pipelines data areas. But if you do: Tools that inspect the control block structure must be revised as a result of the multitasking support. The PIPPHDR is replaced by PIPGLOB and PIPTHREAD data areas; the PIPHDR is extended, but unchanged in its previous extent; all other control blocks are unchanged. Utilities that inspect the control block structure may issue this pipeline command to test whether or not the control block structure has been revised. RESOLVE FPLBLOKS If this built-in is available (the return code is non-zero), you can invoke it to get control block addresses. It writes the addresses (in printable hex) of these control blocks, on one line in this order: 1. PIPGLOB 2. PIPTHREAD 3. PIPHDR (for the pipeline set). 4. PIPVECT (for the pipeline specification that contains FPLBLOKS). 5. PIPEBLOK (for FPLBLOKS itself). If you issue a callpipe to get the control blocks--for example, callpipe fplbloks | split | stem cpx. --the last two addresses will be stale by the time you see them. -------- Shipped to VMTOOLS 20 Jan 1997 20:51:31 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel 11 (17 in decimal) Fixed READTO when EOFREPORT is not CURRENT so that it does not do a PEEKTO first, as this sets a return code that prevents you from consuming the last record when EOF propagates backwards. This does highlight the fact that PEEKTO has changed when EOFREPORT is used. With EOFREPORT CURRENT, any number of PEEKTOs give the same result; with EOFREPORT ANY (or ALL), subsequent PEEKTOs may set return code 4 or 8. It could be argued that the dispatcher should take care of this situation, but that would mean adding to the path length for the general case. When their secondary output was connected, the TCP/IP stages did not terminate if they were unable to create or bind the socket. Instead they continued, which resulted in an avalanche of error messages. JOINCONT The preferred keyword to specify an input range is now RANGE, though COLumns will remain a synonym. SFS Writing a file in F-format or appending to an existing file causes an error on the DMSCLOSE on CMS13, which actually enables some code that has quietly been waiting for Y2K support. And that code had a slight problem. 407 ABEND in floating point multiply if one number has less than 16 significant digits and the other has more and needs to have its precision reduced. -------- Shipped to VMTOOLS 30 Nov 1996 16:51:01 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel 11 (17 in decimal) The extensions to the type-9 event record are back in. -------- Shipped to VMTOOLS 13 Nov 1996 21:31:55 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel F (15 in decimal) The extensions to the type-9 event record have been regressed. They cause breakage. GATE Was not careful enough about its output streams when propagating EOF. -------- Shipped to VMTOOLS 12 Nov 1996 14:50:13 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel D (13 in decimal) The SUBSTRing facility failed when a SUBSTR-ed range could be inherited, as in LOOKUP. The type-09 event records (end-of-stage) are extended with two fullwords: The size of the work area allocated and the highwater mark of other allocated storage. Both numbers reflect doublewords rather than bytes. UNIQUE PAIRWISE abends when other than a column range is specified. Further, it terminates when the primary is at EOF, which breaks not unique. -------- Shipped to VMTOOLS 26 Oct 1996 16:14:51 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel D (13 in decimal) When the scanner terminates and the PIPE command was issued by Xedit, message 1145 is now issued to alert the user to the possible truncation. When an input range is a field, a null field is now processed differently than a missing field. This has corrected the behaviour of CHANGE, (N)LOCATE, and SPEC. An input range can now be SUBSTRing'ed. The syntax of an input range is: ::= {SUBSTR OF}* ::= the old definition. To select the second character of the third word: spec substr 2 of word 3 1 Substrings can be nested to any depth. (Share requirement 18.) CHANGE Incorrectly appended the change string when the first string is null and the field or word was not present in the record. HOSTBYxxx wrote random data when RXSOCKET was not already NUCXLOADed and it therefore initialised itself. INSERT [BEFORE|AFTER] [] Insert the delimited string in each input record. The input range specifies the place to insert the record. The keyword specifies whether the insert is to be before or after the input range. The default is to insert before. The default range is the entire record. When the input range is not present in the record, the range used is the first or the last column of the record, depending on whether the first part of the inputRange is relative to the beginning or the end of the record. Note that the full inputRange specification is available, including words and fields. (Share requirement 2.) 407 Divide lost precision massively with long dividends. So much that the result became zero. The handling of extreme cases with pictures was badly broken. -------- Shipped to VMTOOLS 7 Sep 1996 17:17:02 ------------------------ PIPELINE MODULE Modification 1.0110 sublevel B (11 in decimal) DAM Converse (almost) of GATE. Any number of streams. Primary output must not be connected. When a record arrives on the primary input, the higher-numbered streams are shorted in the stream order. The record on the primary input is not consumed. When EOF arrives on the primary input, DAM terminates without producing output. EOF on higher-numbered streams is propagated as appropriate. Note that DAM does not buffer anything. FANIN Terminate when the secondary output stream is connected. -------- Shipped to VMTOOLS 28 Aug 1996 22:22:11 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel A (10 in decimal) PREDSELECT Discard any records on the secondary and tertiary input that arrive before the first record on the primary input. (Rather than write a null buffer.) STATE Accept RC 36 as well as 28 from CMS. The return code will be aggregated in the normal fashion. -------- Shipped to VMTOOLS 27 Aug 1996 19:06:42 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel 9 (9 in decimal) FANOUTWO Did not notice if its output was severed when it starts. Since EOFBACK uses FANOUTWO, it suffered the same problem. HOSTBYxxx Write all errnos to the secondary output stream, if one is defined. The record contains the host or address to be resolved, the numeric errno, and the symbolic errno. PREDSELECT Consumed an input record it shouldn't. This bug was introduced in sublevel 7. TCPDATA Tried to write an error record to the secondary output even when it wasn't defined. -------- Shipped to VMTOOLS 25 Aug 1996 12:57:36 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel 8 (8 in decimal) DEAL, GATHER, FANOUTWO, and possibly other FANxxx filters did not select streams properly in all instances. DEAL Add suboption STRIP to KEY and STREAMID. The key/ID field is then deleted from the output record. This may save a bunch of stages to delete the field on each output stream. The key must be at the beginning or at the end of the record; this is enforced at run time. HOSTBYxx Split input lines into words and process each word separately. Both discard blank and null lines. LOOKUP Propagate EOF from the primary input to the primary and secondary output. Propagate EOF on all of the three first output streams to the primary input. Propagate EOF on the tertiary input to the output stream 4. Propagate EOF on input stream 3 to output streams 3 and 5. (Share requirement 13.) TCPxxx and UDP would abend when given a domain name for an IP address and the name server returned more than 16 bytes of data. (Eg., ns.princeton.edu returns two addresses.) -------- Shipped to VMTOOLS 23 Aug 1996 20:14:09 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel 7 (7 in decimal) The PL/I and C interface now suppresses gratuitous messages that are issued by AD/Cycle when it maps the old format plist to a newer one. The internal wait/post routine does thread suspend/resume when multitasking is active. Thus, the pipeline will allow other MT applications to run concurrently. They will still not be able to do pipes. (SHARE requirement 39.) Wherever the TCPIP stages and their supporting cast would accept a dotted-decimal number, they now take anything else as a host name and resolve it via RXSOCKET Version 2 or later. If you haven't got RXSOCKET, you'll get a message to that effect. Errors in name resolution are reported in a distinctly RXSOCKET-y way. RXSOCKET Version 1 is used for name resolution if it is already initialised and the name is resolvable. (This completes requirement 45.) 1. New and revised Pipeline Commands: EOFREPORT A new pipeline command, EOFREPORT, is introduced to control the return code on SELECT ANYINPUT, PEEKTO, and OUTPUT. With the default, CURRENT, all is as it always was. EOFREPORT ALL will report the severance of the last connected output stream with return code 8 during PEEKTO and SELECT ANYINPUT. They will also report RC=8 on subsequent invocations. EOFREPORT ANY will report return code 8 as done by ALL, and in addition set RC=4 if a stream is severed during PEEKTO and SELECT ANYINPUT. It will report RC=4 on OUTPUT if a stream was severed before the record was peeked by the consumer stage. Note that events that occur between the time the stage is made runable and until it is dispatched are not reported. EOFREPORT ANY/ALL also performs the STREAMSTATE SUMMARY function. The return code is set accordingly. (Share requirement 13.) STREAMSTATE Add SUMMARY (only operand). The return code is zero when there is both a connected input stream and a connected output stream. It is 8 otherwise. (Share requirement 13.) 2. Stages: ADTFST Add SORTED keyword to produce the timestamp as a single word FMTFST which is suitable for numerical comparison in eg, pick. STATE(W) The timestamp could be 19960725160437. Do remember numeric digits if you ever use them in a REXX expression; with the default of 9 digits, you'll only be comparing to the nearest hour. FANOUTWO Propagates EOF on the primary stream even faster than it used to do. (But this is visible only in pipelines that wait for events.) (Share requirement 13.) GATE A new dispatcher function allows GATE to be more responsive to streams being severed. Fix a bug that made STRICT test for end-of-file on the primary rather than for a record being present, as it should have done. (Share requirement 13, part 2.) HELP Turn TEXT off after the Xedit session, if PIPE HELP turned it on before the session. That is, undo the damage. HOSTBYADDR Reverse name resolution. Input lines contain dotted-decimal IP addresses. The fully-qualified host name is written to the primary output. If the secondary output is defined and the IP address is not resolved, the input record is written to the secondary output. If the secondary output stream is not connected, the only indication of an address not being resolved is the lack of an output record. That is, no return code is set. Name resolution requires RXSOCKET version 2 or later. It is controlled by the contents of the file TCPIP DATA. (Share requirement 45.) HOSTBYNAME Resolve a host name to an IP address. Input lines contain host names in the local domain (if they have no periods) or fully qualified host names if they contain dots. The IP address (dotted-decimal) is written to the primary output. If the secondary output is defined and the host name is not resolved, the input record is written to the secondary output. If the secondary output stream is not connected, the only indication of a name not being resolved is the lack of an output record. That is, no return code is set. Name resolution requires RXSOCKET version 2 or later. It is controlled by the contents of the file TCPIP DATA. However, if RXSOCKET Version 1 is initialised and if the host name can be resolved, CMS Pipelines bows towards 555 west 57th street and formats the four bytes of binary text returned into an IP address. (Share requirement 45.) IF An adaptation of a REXX filter written by Steve Hayes. Process records conditionally. When IF has two streams defined, the records that are selected by the selection stage are processed by the stages between the two occurrences of the label on the IF stage. The output flows from the secondary output | if: if locate /oscar/ | | if: | When IF has three streams defined, the records that are selected by the selection stage are processed by the stages between the first two occurrences of the label on the IF stage. Records that are rejected are processed between the last two occurrences of the label on the IF stage. The output flows from the tertiary output | if: if find Davidsen | | if: | | if: | Note that no end-characters are used. IF does the faninany-ing under its covers. JUXTAPOSE Add COUNT option and support for secondary output stream. When COUNT is specified, each record on the primary output is prefixed with a ten-character count of the number of records on the secondary input that has referenced this record from the primary input. When a record on the primary input has not been matched by any record, this record is written to the secondary output (but it is not prefixed a count). This output record is delayed one record relative to the primary input stream. JUXTAPOSE terminates when both output streams are severed. LOCATE and NLOCATE always assumed they had a secondary stream. Bug was introduced with mixed-case support. In some cases, (N)LOCATE would go one longer than the range in its compare. For example, locate 1.10 /==========/ would also match the string 2.10 if it contained all equals. LOOKUP Add COUNTSOFAR, AUTOADD BEFORE, KEYONLY. COUNTSOFAR activates the COUNT option and also writes the current count (inclusive of the current detail) preceding the master record when it is written to the primary. (COUNT should have worked this way always. Sorry.) AUTOADD BEFORE adds an unmatched detail to the reference before it looks for it. Thus, no output is ever generated on the secondary stream. KEYONLY stores only the key field in the master file and thus writes just the key for the master records. The three operands are designed to be used together to build several stemmed arrays concurrently without delaying the record: |l:lookup countsofar autoadd before keyonly w1 master detail |spec /=/ 1 11-* n /./ n 1.10 strip n /=/ next read w2-* n |i:fanin .... \l: \l: |spec /=/ 1 11-* n /.0=/ n 1.10 strip n |i: This subroutine is baked in as STEMBUILD. PREDSELECT Now works reliably as far as the input streams going to end-of-file is concerned. EOF is now propagated immediately from the secondary input to the primary output and from the tertiary input to the secondary output. EOF on the outputs is not propagated before both output streams are at EOF. EOF on the primary input causes PREDSELECT to terminate. This will cause breakage for those who were too lazy to connect the secondary output from the selection stage to the tertiary input of PREDSELECT, and still rely on records on the secondary output. The secondary output will be severed immediately if the tertiary input is not defined. (Share requirement 13, part 2.) RETAB Converse of UNTAB. Same arguments. REXX Add STREAMSTATE ALL . The states of all defined streams are written into the variable as a string of blank- delimited words. Each word has the format :. The first return code is for the input side; the second is for the output side. See also the new pipeline commands. SORT COUNT and UNIQUE: Write duplicate records to the secondary output stream without delay, if the secondary output stream is defined. Only EOF on the primary output will terminate SORT; EOF on the secondary output is ignored. SORT now starts on commit level -2, as do all selection stages. SPEC 407: Allow leading v and punctuation in a picture. Fix formatting of pictures that have no digit selects before the v or decimal point. SQLSELECT Format packed data with an even number of digits correctly. STATE(W) Set return code when a file is not present and it terminates because of an output not being connected. Add ASIS. When ASIS is specified, STATE(W) does not attempt to uppercase the file name. STEMBUILD Build multiple stemmed arrays for use by VARSET. The first word contains the stem name; the rest are the data. The output records contain =.= and after EOF, ==n. The data records are not delayed. (See LOOKUP.) TCPLISTEN Test the state of the output stream before doing the accept. This guards against the output becoming not connected while a server is being started up. It also guards against the output not being connected in the first instance. URLDEBLOCK Deblock the url-x format. The processing assumes an ASCII encoding of the input data. Each input record is one url to be expanded. An output record is written for each part of the input record separated by ampersand or semicolon. Plus signs are changed to blanks. %xx is replaced by the single character x'xx'. XLATE Add codepage 37-2. This is the 3270 codepage used by C/370. It differs from 1047 by having hat and not reversed (5f and b0). When uploading C from a workstation, this is the one you want. YEAR2000 Support. The sfs device drivers use the FULLDATE option internally on systems that support it. Since the dates are already specified with the century, no external changes are required and no change to existing pipelines is needed. -------- Shipped to VMTOOLS 1 May 1996 09:41:45 ------------------------ PIPELINE MODULE Modification 1.0110 sublevel 6 (6 in decimal) New pipeline commands: (Also available to REXX filters.) PIPTSTKW Test if the string can be decoded as the keyword code. The keyword code is one or two characters. Case is ignored in both operands. The return code is 0 if the string contains the keyword; it is 1 if the string is not the keyword; other values indicate trouble in the arguments (first one too long; missing or too many arguments). To test if the variable 'word' contains the keyword NUMBER: 'piptstkw rn' word if rc=0 then /* we have that keyword */ else if rc=1 then /* we ain't got it */ Only keywords that are generated into the main pipeline module can be tested. DMSPRM contains this table. (Part of requirement 22.) YEAR2000 Support. The Keyword SHOrtdate ISOdate FULldate are added to AFTFST, FMTFST, STATE(W). The default is SHORTDATE. DEBLOCK Add RDW, the do-it-yourself deblocking kit. DEBLOCK RDW [STRIP] The range specifies the location of the record descriptor, which contains the length of the record less the second number. When the record descriptor begins in the first column, you may specify STRIP to delete the record descriptor from the output record (this is not possible if the record descriptor is imbedded in the record). For example, DEBLOCK RFC959 is processed internally the same as DEBLOCK RDW 2.2 3. DEBLOCK SF could be DEBLOCK RDW 1.2 0 STRIP. Fix DEBLOCK SF4 not to write negative-length record when the record descriptor is bad. To deblock a LIST3820 or PSEG3820 that has lost its record structure: DEBLOCK RDW 2.2 1. (Part of requirement 37.) PIPCMD Support the keyword STOP as for FANOUT. The default is not to test the state of the output streams. Unlike FANOUT, PIPCMD consumes the input record before it tests the state of the streams, because it cannot detect whether EOF has been seen by the subroutine pipeline. If it is important that the input record not be consumed, the subroutine should return a negative return code. This causes (has always caused) PIPCMD to terminate without consuming an input record. (Pre-requisite for requirement 16.) SORT The fix for 0c5 abends when fields are beyond 4K of the beginning of the record compared random data on descending. SPECs Bug if there is no reference to an input record before the first select item *and* that item is issued conditionally. 407: The multiply operator did not observe one of the rules of the /390 game. 0C6 as a result with more than 15 digits precision. -------- Shipped to VMTOOLS 22 Mar 1996 09:36:56 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel 3 (3 in decimal) MDSKxxxx Inspects x'10' in offset x'3D' of the FST in FVS as exposed by the FSSTATE macro. If this bit is on, the full block interface will not be used. With a companion change to HSX (3.4.0), the two packages are now compatible. SFSxxxx Issue meaningful message on reason 90530 from DMSVALDT. This reason code was added to CMS 11. Normally it indicates that you forgot to put a dot in a directory name. Or that something became a directory which shouldn't have been: < / profile exec SPEC 407: Negative exponents were treated as positive when converting a number to the internal representation. -------- Shipped to VMTOOLS 12 Mar 1996 00:03:54 ----------------------- PIPELINE MODULE Modification 1.0110 sublevel 2 (2 in decimal) DEAL Add sub-keyword RELEASE after SECONDARY. When RELEASE is specified, the record on the secondary input is released before the record is read on the primary input. This allows for the secondary input to be a derivative of the record, assuming it is produced first. DEBLOCK Add RFC959, which deblocks according to RFC 959. Each logical record contains a flag byte followed by the two-byte length of the remainder of the record. The flag byte is not inspected. Note that since the length field is imbedded in the record, it is not removed in the output record. IP2SOCKA Generate the sockaddr_in structure, which is used in the UDP stage's input data. No arguments. The input record contains three blank-delimited words: 1. The literal "AF_INET" (in any case). 2. A port number. 3. A dotted-decimal IP address. (See TCPLISTEN, eg.) For each non-null input record, the output record contains sixteen bytes (in network byte order): 0 2 '0002'x AF_INET 2 2 Port number (unsigned short) 4 4 IP address (unsigned long) 8 8 Binary zeros. (See also TCPDOTS.) (Part of requirement 45.) SOCKA2IP Format the contents of the sockaddr_in structure. No arguments are allowed. The operation is the inverse of ip2socka above. The input record is sixteen bytes in the format defined for ip2socka above. The output record contains three blank-delimited words, as described for the input to ip2socka. The output record contains one blank between words. (Part of requirement 45.) SPECs Do not consume input records from streams that are not referenced by a SELECT clause. This bug was introduced in February 1995. It made it into ESA V2. TCPCLIENT When the GETSOCKNAME operand is specified, the first output TCPDATA record is now the raw sixteen-byte sockaddr_in structure. TCPLISTEN Use SOCKA2IP to convert. UDP (Part of requirement 45.) -------- Shipped to VMTOOLS 27 Jan 1996 17:57:05 ----------------------- PIPELINE MODULE Modification 1.0109 sublevel 60 ( 3C in hex) Regressed module due to failure on ancient CMS. -------- Shipped to VMTOOLS 25 Jan 1996 16:07:13 ----------------------- PIPELINE MODULE Modification 1.0109 sublevel 58 ( 3A in hex) It is now enforced correctly and reliably that the pipeline is not subjected to recursion from a different process/thread. And this is done without incurring the MT overhead when MT is not being used for other purposes. BFSxxxx Refuse to run in subset and with DOS ON, since this would lead to OpenEdition (or multitasking) abending. BFSSTATE Format unknown userid/groupid as unsigned decimal rather than hex. This is more unixy. DEBLOCK Fixed DEBLOCK LINEND option to scan ; it scanned for something else. JOINCONT Add ANYCASE support. Note that ANYof takes precedence. Add COLUMN to specify any position. COLUMN implies KEEP. COLUMN refers to the first record. Keywords can be specified in any order. (Part of requirement 25.) SORT Bugfix: descending and anycase only sorts the first column of a field. Also the case with descending and very long fields. SPEC Fix it not to abend on PRINT "Z" 1. TCPCLIENT Support the new keyword GROUP in conjunction with DEBLOCK. TCPDATA GROUP takes a delimited string, which can specify any stage. You can use a built-in (eg, GROUP /nfind -/; or GROUP ?JOINCONT TRAILING /-/?) or a REXX filter; you cannot specify a cascade of filters. When the stage decides to read from the socket, it will now insist on getting complete records (as defined by the DEBLOCK option). It will also insist on the stage specified with GROUP writing at least one output record before it will allow the socket to be idle. (Part of requirement 45.) ------------------ Changes in 1995: ----------------------------------- Message 795 issued incorrectly in some instances. Rather annoying. A word range now has no leading or trailing word separators, even when the end of the range is beyond the input record. Thus, w1-* and w1-2000000000 represents the same string. The TCP/IP device drivers were generating an incorrect IUCV expansion which only stored the low-order halfword of the length to be sent. It turns out that the IUCV macro has an undocumented format of the BUFLEN parameter to support fullword lengths. EOFBACK Run a device driver with EOF being propagated backwards. The record is written to the device driver only after it has been consumed by the following stage. LOCATE Fix recent enhancements, which were regressed. SPEC Tolerate xC sign in C2I conversion. Some programs do produce packed data for dates. TCPDOTS Converts a dotted-decimal IP address to a fullword integer. If a second argument is specified, it is a port number. A socket address (AF_INET) structure is built containing the IP address and the port number; the output record is then 16 bytes. The input may be specified as the argument or as input records, or both. (Part of requirement 45.) UDP Fix breakage introduced when the TCPxxxxx stages were added. -------- Shipped to VMTOOLS 22 Nov 1995 21:42:30 ----------------------- PIPELINE MODULE Modification 1.0109 sublevel 56 ( 38 in hex) Regress the MT support. Regress changes to LOCATE. -------- Shipped to VMTOOLS 6 Nov 1995 14:48:31 ------------------------ PIPSYSF MODULE Edition .4005 The initialisation code no longer uses a MT call to determine the thread and process ID when MT has not already been activated. This should improve on end-of-command processing since MT won't be activated. (This is surely a design flaw in MT.) An enhancement in 1.1.8 that was never documented and possibly should not be is that an inputRange can also be an AUTOField (also AField). This specifies that the first character of the record is used as the field separator. This character (and a matching one at the end of the record if there is one) is ignored in the field counting. Thus, for example to translate the variable name in the input to VARSET to uppercase: |xlate af 1|varset direct BETWEEN And its brethren (OUTSIDE INSIDE NOTINSIDE) fixed to issue message 113 when there are no further operands. The message currently issued (15) is misleading. CHANGE Added a spurious character to the end of the record when FS is used and the field is not present in the record. Actually, it was SPECS that was broken, but CHANGE is the messenger. CHOP Do not go in front of the input record (before). That is, do not write negative-length records. HELP Issue message 172 correctly when help is not stored. HOSTID Write the default IP address associated with the host. You can specify USERID for the name of the TCPIP server. (Part of requirement 45.) HOSTNAME Write the host name. Note that this does not include the domain name. This is probably the same as the RSCS host which you can find out via the CMS IDENTIFY command. You can specify USERID for the name of the TCPIP server. (Part of requirement 45.) LOCATE Support ANYCASE keyword at beginning of operands. May be NLOCATE slow for long strings. But then even the standard algorithm is slow for long strings. (Part of requirement 25.) PARCEL Enforce that records must be complete. Consume input record as soon as its data are processed. SPEC Fix storage overlay when an odd number of counters is used and additional counters are allocated in free storage. STRxxxxx Support ANYCASE as a keyword before the delimited string. In those cases where it wasn't already, if any. Not as fast, particularly STR(N)FIND (and STRASM(N)FIND). (Part of requirement 25.) TCPxxx Set ERRNO=0 when socket is closed. Set ERRNO=5001 when terminated by PIPMOD stop or similar. The GETSOCKNAME option causes the stage to write as its first output record the actual port number assigned by TCP/IP (or the one requested). To listen on a dynamically assigned port (which you'd somehow communicate to your clients): TCPLISTEN 0 GETSOCKNAME | t: take 1 | ... The output record contains the port number followed by the dotted-decimal host ID (or a single zero if the ANY host was used to bind the socket). These socket options can now be enabled: KEEPALIVE OOBINLINE REUSEADDR The LINGER option is now implemented as a socket option as well as being the delay after end-of-file on the input before the socket is closed. (TCPCLIENT and TCPDATA.) (Part of requirement 45.) TCPCLIENT Add these options: GREETING The server sends a line of greeting when the client connects. This line will be written to the output before reading input records. (Part of requirement 45.) TCPCLIENT Add these options: TCPLISTEN UDP LOCALIPaddress < ANY | HOSTID | > The default will remain ANY (which binds to all or any of the IP addresses). ANY is most appropriate to TCPLISTEN. For TCPCLIENT, you may wish to specify HOSTID to use the default IP address; or specify a particular network interface. (Part of requirement 45.) TCPCLIENT Add these options: TCPDATA DEBLOCK sf | sf4 | The data received are deblocked in the format specified, but no additional information is added to the data transmitted (cf. the SF and SF4 operands alone). For (refer to DEBLOCK), TERMINATE is always in force; EOF is not accepted. (Part of requirement 45.) TCPDATA Now Starts at commit level -10, as documented. UDP Add the socket options BROADCAST and REUSEADDR. Support GETSOCKNAME like TCPxxxx. (Part of requirement 45.) ----------------------------------------------------------------------- This file documents CMS Pipeline enhancements since 1.1.9 (VM/ESA 2.1.0). The VM/ESA level shipped is sublevel x'33'. For enhancements from 1.1.8 (VM/ESA 1.2.2) see PIPELINE NEWS119 on IBMVM For enhancements from 1.1.7 (VM/ESA 1.2.1) see PIPELINE NEWS118 on IBMVM For enhancements from 1.1.6 (VM/ESA 1.1.1) see PIPELINE NEWS117 on IBMVM (c) Copyright IBM Corp. 1995-1997.