Parallel Library User Notes (For PLIB version V2.0) Craig W. Yankes Digital Equipment Corporation Mid-Range Systems Advanced Development 305 Foster Street (LTN2-2/H17) Littleton, Massachusetts 01460 This document describes software being placed into the public domain by Digital Equipment Corporation on an "as is" basis. No commitments for support are either expressed or implied for the software described herein or any potential future versions thereof. Digital Equipment Corporation assumes no liability for any loss caused by use of the software described herein or any potential future versions thereof. The information in this document is subject to change without notice and should not be construed as a commitment by Digital Equipment Corporation. Digital Equipment Corporation assumes no responsibility for any errors that may appear in this document. No responsibility is assumed for the use or reliability of software on equipment that is not supplied by Digital Equipment Corporation or its affiliated companies. Comments or suggestions about the software described herein may be forwarded to the name and address listed above. Parallel Library V2 User Notes Page 2 CONTENTS 1 INTRODUCTION . . . . . . . . . . . . . . . . . . . . 3 2 SUMMARY . . . . . . . . . . . . . . . . . . . . . . 4 2.1 Subprocess Management . . . . . . . . . . . . . . 4 2.2 Memory Sharing . . . . . . . . . . . . . . . . . . 4 2.3 Process Synchronization . . . . . . . . . . . . . 4 2.4 Initialization Routines . . . . . . . . . . . . . 4 3 SUBPROCESS MANAGEMENT ROUTINES . . . . . . . . . . . 5 3.1 Plib$create_subprocesses . . . . . . . . . . . . . 5 3.2 Plib$delete_subprocesses . . . . . . . . . . . . . 6 3.3 Plib$current_image . . . . . . . . . . . . . . . . 7 4 MEMORY SHARING . . . . . . . . . . . . . . . . . . . 8 4.1 Plib$share_memory . . . . . . . . . . . . . . . . 8 4.2 Plib$share_code . . . . . . . . . . . . . . . . 10 4.3 Plib$current_pc . . . . . . . . . . . . . . . . 12 4.4 Plib$main_pid . . . . . . . . . . . . . . . . . 13 5 PROCESS SYNCHRONIZATION . . . . . . . . . . . . . 14 5.1 Plib$signal_subprocesses . . . . . . . . . . . . 14 5.2 Plib$signal_main . . . . . . . . . . . . . . . . 15 5.3 Plib$synch . . . . . . . . . . . . . . . . . . . 16 5.4 Plib$signal_and_wait . . . . . . . . . . . . . . 17 5.5 Plib$set_bit_interlocked . . . . . . . . . . . . 19 5.6 Plib$clear_bit_interlocked . . . . . . . . . . . 21 5.7 Plib$add_word_interlocked . . . . . . . . . . . 22 6 INITIALIZATION ROUTINES . . . . . . . . . . . . . 23 6.1 Plib$init_signals . . . . . . . . . . . . . . . 23 6.2 Plib$init_pagefile . . . . . . . . . . . . . . . 25 7 APPENDIX 1 - SAMPLE PROGRAM . . . . . . . . . . . 26 7.1 General Overview . . . . . . . . . . . . . . . . 26 Parallel Library V2 User Notes Page 3 1 INTRODUCTION The Parallel Library is a set of routines that provide many of the functions required to implement parallel programming. In general, they are jacket routines around the appropriate VMS system service calls and reduce the parameter lists to those predominantly required by parallel programming. The intended audience for these routines is the applications programmer. Use of the Parallel Library routines does not require an understanding of how the system services operate. This document will not attempt to be a "how to" manual for parallel programming, but rather will explain the functions of these routines. A heavily documented program will be included as a sample program to demonstrate the potential usage of these routines. These routines are for use with VMS Version 4.x. Parallel Library V2 User Notes Page 4 2 SUMMARY These routines can be categorized into 4 groups of functions. These areas are: 2.1 Subprocess Management Parallelism is usually implemented under a main process / multiple subprocesses model. This group of routines assists with the creation and deletion of the subprocesses. 2.2 Memory Sharing Establishing a shared memory region accessible by the main process and the subprocesses is usually a requirement for parallelism. This group of routines provides the functionality of sharing memory and executable code. 2.3 Process Synchronization There are two forms of process synchronization: barrier synchronization to control the overall flow of the program and task assignment for the subprocess to allocate the next piece of work. The routines in this group assist in providing these operations. 2.4 Initialization Routines This group of routines provides various initialization functions that are required for subsequent operations. Specifically, these routine provide for the initialization of the paging file and the signaling mechanisms. Parallel Library V2 User Notes Page 5 3 SUBPROCESS MANAGEMENT ROUTINES 3.1 Plib$create_subprocesses This routine will request that the system create the number of subprocesses that has been specified. Format: istat = plib$create_subprocesses (number_of_subprocesses, exe_name, sys_input, sys_output, return_pids) Arguments: number_of_subprocesses - Count of the number of subprocesses to be started. Longword passed by reference. exe_name - Name of the .exe file that this subprocess will execute. Character string (maximum of 128 characters) passed by descriptor. sys_input - String to assign to the sys$input logical. Character string passed by descriptor. sys_output - String to assign to the sys$output and sys$error logicals. Character string passed by descriptor. return_pids - An array of NUMBER_OF_SUBPROCESSES longwords that will return the PIDs of the created subprocesses. This array space must be allocated by the calling process. Return value: The return value is whatever value is returned from the $CREPRC system service call. Restrictions: Comments: To avoid problems if the executable image disk name has changed or is being run from a different directory, the PLIB$CURRENT_IMAGE routine should be used to obtain the image name. Parallel Library V2 User Notes Page 6 3.2 Plib$delete_subprocesses This routine uses the Delete Process system service to delete some or all of the subprocesses. Format: istat = plib$delete_subprocesses (inum, ipids) Arguments: INUM - The number of subprocesses to delete. Longword passed by reference. IPIDS - Longword array of subprocess PIDs to delete. (Note, these pids can be obtained when the subprocesses are created by the PLIB$CREATE_SUBPROCESSES routine.) Only the first INUM number of subprocesses will be deleted from this list. Return value: 1 = all were deleted successfully 0 = At least one subprocess was not deleted. Restrictions: Comments: Parallel Library V2 User Notes Page 7 3.3 Plib$current_image The PLIB$CURRENT_IMAGE routine will return the name of the image currently being executed. Format: istat = plib$current_image (image_name) Arguments: image_name - Character string of 128 characters in which the name of the current executable image will be returned. Return value: The return value is the status value from the $GETJPI system service. Restrictions: Comments: This routine should be used to determine the name of the image to be passed to the PLIB$CREATE_SUBPROCESSES routine and minimizes the problems of images being executed from other directories or being renamed. Parallel Library V2 User Notes Page 8 4 MEMORY SHARING 4.1 Plib$share_memory The PLIB$SHARE_MEMORY routine attempts to map to an existing global section and will create the global section if it did not previously exist. Format: istat = plib$share_memory (low_address, high_address, section_name) Arguments: LOW_ADDRESS - The starting virtual address for the range of memory to be shared. Longword passed by reference. HIGH_ADDRESS - The ending virtual address for the range of memory to be shared. Longword passed by reference. SECTION_NAME - Name of the global section to either map to or create. Return value: If the process-private page file does not have enough free space to contain the shared memory range being requested, the process will abort and state the size of the paging file that is required. The return value is that from the $CRMPSC (Create and Map Global Section) system service. Restrictions: The process-private paging file must be initialized (by calling PLIB$INIT_PAGEFILE) prior to calling this routine. The PLIB$SHARE_CODE routine, if used in the application, must be called prior to this routine. Comments: When the global section is created, it is defined as being demand-zero and thus the entire global section is zeroed. No attempt is made to preserve existing data in this shared range at the time of the global section creation. Note, this operation of zeroing the data only occurs when the global section is created. If the main process creates it and deposits values into it, those values are accessible to Parallel Library V2 User Notes Page 9 the subprocesses after they have mapped to the global section. Two things to be careful of: 1) Since global sections are always on full page boundaries, the memory sharing is actually done from the first byte of the page containing the LOW_ADDRESS address to the last byte of the page containing the HIGH_ADDRESS address. This has the side effect that more memory is shared than what was requested in the parameters. The sample program's comments will show some methods to avoid the potential problems this can create. 2) Global sections are accessible by either any process in the creator's UIC group or by any process in the system. This routine creates the global section to be accessible throughout the UIC group. Be careful, therefore, that two or more processes in the same UIC group don't accidentally try to access a global section that has the same name! The PLIB$MAIN_PID routine can be used to derive a process-specific unique name that can be appended to the Global Section name to avoid this problem. Parallel Library V2 User Notes Page 10 4.2 Plib$share_code The PLIB$SHARE_CODE routine will use Global Sections to share executable code. This code sharing does not require any privileges to implement. Format: istat = plib$share_code (low_address, high_address, section_name) Arguments: LOW_ADDRESS - The starting virtual address for the range of memory to be shared. Longword passed by reference. HIGH_ADDRESS - The ending virtual address for the range of memory to be shared. Longword passed by reference. SECTION_NAME - Name of the global section to either map to or create. Return value: If the process-private page file does not have enough free space to contain the shared memory range being requested, the process will abort and state the size of the paging file that is required. The return value is that from the $CRMPSC (Create and Map Global Section) system service. Restrictions: The process-private paging file must be initialized (by calling PLIB$INIT_PAGEFILE) prior to calling this routine. This routine may only be called once in a program. Any executable code to be shared should be in contiguous virtual addresses and be set up as shareable through this one call. Comments: Two things to be careful of: 1) Since global sections are always on full page boundaries, the memory sharing is actually done from the first byte of the page containing the LOW_ADDRESS address to the last byte of the page containing the HIGH_ADDRESS address. This has the side effect that more memory is shared than what was Parallel Library V2 User Notes Page 11 requested in the parameters. The sample program's comments in will show some methods to avoid the potential problems this can create. 2) Global sections are accessible by either any process in the creator's UIC group or by any process in the system. This routine creates the global section to be accessible throughout the UIC group. Be careful, therefore, that two or more processes in the same UIC group don't accidentally try to access a global section that has the same name! The PLIB$MAIN_PID routine can be used to derive a process-specific unique name that can be appended to the Global Section name to avoid this problem. In some languages, FORTRAN as an example, it is not straightforward to determine where the executable code starts and ends. Please refer to the sample program to demonstrate the usage of PLIB$CURRENT_PC and the "book marker" routine to determine the range to be shared. Parallel Library V2 User Notes Page 12 4.3 Plib$current_pc This routine is used in conjunction with the PLIB$SHARE_CODE and assists in determining the starting virtual address for the range to be shared. Format: iaddr = plib$current_pc() Arguments: - none - Return value: The returned value in IADDR is the virtual address of where this routine was called from. Restrictions: Comments: Refer to the sample program for an explanation of how this routine assists with the sharing of executable code. Parallel Library V2 User Notes Page 13 4.4 Plib$main_pid The plib$main_pid routine returns the process identification (PID) of the parent process if called from a subprocess. If called by a main process, it returns its own PID. Format: my_pid = plib$main_pid (parent_pid) Arguments: PARENT_PID - Character string variable of eight characters in length in which the ascii representation of the parent process's PID will be returned. Return value: 0 - This routine has been called by a main process. non-0 - This routine was called by a subprocess and the value returned is the integer longword value of the subprocess's own PID. Restrictions: Comments: Appending the main process's PID to global section and private page file names is an easy method of avoiding process-to-process interactions. The value returned in PARENT_PID is the PID of the parent process. In an environment of a main process with multiple subprocesses, this equates to returning the PID of the main process. Be careful if subprocesses are creating other subprocesses since the second level subprocesses will not receive the PID of the main process but rather of its parent! Parallel Library V2 User Notes Page 14 5 PROCESS SYNCHRONIZATION 5.1 Plib$signal_subprocesses The PLIB$SIGNAL_SUBPROCESSES routine is used to awaken the subprocesses and to tell them which parallel section to execute. Format: istat = plib$signal_subprocesses (section_number, wait_option) Arguments: SECTION_NUMBER - A longword whose value will be passed to each subprocess when they are awakened. Passed by reference. WAIT_OPTION - This argument specifies whether the main process should wait for the subprocesses to report completion before continuing execution. The options are: 0 - Don't wait for the subprocesses to report completion. Using this option will request that this routine returns to the calling routine immediately after awakening the subprocesses. non-0 - Wait for the subprocesses to report completion. With this option, this routine will not return to the calling routine until the subprocesses have signaled their completion. Return value: This routine will always return a 1. Restrictions: The PLIB$INIT_SIGNALS routine must be called prior to using this routine. Comments: The section_number argument is normally used to instruct the subprocesses as to which section of code to execute. Since the Parallel Library routines don't do anything with this value except to pass it to the subprocesses, however, this longword can be used for any purpose. Parallel Library V2 User Notes Page 15 5.2 Plib$signal_main The PLIB$SIGNAL_MAIN routine is used by the subprocesses to report their completion to the main process. This routine will not return until the next time the main process has awakened the subprocesses by the PLIB$SIGNAL_SUBPROCESSES routine. Format: istat = plib$signal_main (next_part) Arguments: NEXT_PART - This argument is how the subprocesses receive the SECTION_NUMBER value that the main process specified in PLIB$SIGNAL_SUBPROCESSES. Return value: This routine will return a 1 in all cases. Restrictions: The PLIB$INIT_SIGNALS routine must be called prior to using this routine. Comments: Every subprocess is to call this routine when it has completed its own functions. The main process will be signaled only when every subprocess has called this routine. Parallel Library V2 User Notes Page 16 5.3 Plib$synch This routine is used by the main process to determine when the subprocesses have completed their work if the subprocesses were awakened with the NO_WAIT option in PLIB$SIGNAL_SUBPROCESSES. Format: istat = plib$synch () Arguments: - none - Return value: This routine returns a 1 in all cases. Restrictions: The PLIB$INIT_SIGNALS routine must be called prior to using this routine. Comments: This routine is used to pause the main process until the subprocesses have signaled completion if the main process has been executing at the same time. This can be caused by two situations: 1) The main process awoke the subprocesses using the NO_WAIT option to PLIB$SIGNAL_SUBPROCESSES, or, 2) The subprocesses were just created. In either case, the main process will eventually reach a point where it must know that the subprocesses have completed their previous task (or are ready in the case of creating the subprocesses) before the main process can continue. The PLIB$SYNCH routine provides this functionality and will pause the main process until the subprocesses have all reported completion by calling PLIB$SIGNAL_MAIN. Parallel Library V2 User Notes Page 17 5.4 Plib$signal_and_wait This routine can be used to signal the subprocesses and wait for them to report their completion. NOTE - This routine is being kept in the V2 of the Parallel Library only for backwards compatibility with V1. With the ease of using the V2 routines, it is fully expected that PLIB$SIGNAL_SUBPROCESSES, PLIB$SIGNAL_MAIN and PLIB$SYNCH would be used instead of PLIB$SIGNAL_AND_WAIT. Format: istat = plib$signal_and_wait (sub_ef, main_ef) Arguments: SUB_EF - Longword passed by reference whose value is an event flag number to be set to awaken the subprocesses. MAIN_EF - Longword passed by reference whose value is the event flag that the subprocesses will set to reawaken the main process. Return value: This routine always returns a 1. Restrictions: Comments: This routine has the effect of setting the event flag specified in SUB_EF, wait for the event flag specified in MAIN_EF to be set and then clear both event flags before returning. Be careful when using this routine to avoid the "sticky flag" problem. For example, if the subprocesses will always be awakened by event flag 64 and will awaken the main process with 65, the following will happen: Subprocesses wait for event flag 64. The main process sets 64 and waits for 65. The subprocesses awake, perform their action, set event flag 65 to report completion and wait for 64. The subprocesses immediately continue since event flag 64 is still set. Parallel Library V2 User Notes Page 18 The "sticky flag" problem is that there are no guaranteed ways of clearing event flag 64 that doesn't have timing windows. To get around this, the program can use two separate event flags to awaken the subprocesses - the first time they wait for 64 and the next time they wait for 66, for example, with the "next flag" continually toggling back and forth. This is not hard to code once it is conceptually understood, but is prone to errors. The hassles involved in using this routine was the basis for creating the PLIB$SIGNAL_MAIN and PLIB$SIGNAL_SUBPROCESSES routines that perform this event flag toggling automatically. Parallel Library V2 User Notes Page 19 5.5 Plib$set_bit_interlocked This routine uses the VAX BBSSI instruction to set the least significant bit of the first argument interlocked. This routine will continually attempt to set the bit until it has been successful. Format: istat = plib$set_bit_interlocked (flag, [loop_count, entry_count]) Arguments: FLAG - Longword passed by reference whose least significant bit is to be set using the VAX BBSSI interlocked bit setting instruction. LOOP_COUNT - A longword passed by reference whose value is incremented every time the bit setting is attempted. ENTRY_COUNT - A longword passed by reference whose value is incremented once upon calling this routine. Return value: This routine always returns 1 since it does not return until the bit setting operation has been successful. Restrictions: Comments: This routine is used to implement critical sections and permits only one one process to enter the critical section at a time. A coding example is: istat = plib$set_bit_interlocked (flag, loop_count, entry_count) my_variable = shared_variable shared_variable = shared_variable + 10 istat = plib$clear_bit_interlocked (flag) If two or more processes try to enter this critical section at the same time, only one will be permitted to enter since only one process could (by definition of the VAX interlocked instructions) have set the bit in the shared FLAG variable and return from this routine. The other process will remain Parallel Library V2 User Notes Page 20 in the PLIB$SET_BIT_INTERLOCKED routine until the first has cleared the bit via the PLIB$CLEAR_BIT_INTERLOCKED routine. The loop_count and entry_count arguments are useful in determining how often two or more processes are blocking each other for access to the critical sections. The numbers individually have little meaning, but comparing the ratio between the two can yield valid performance and bottleneck information. ***** Since these numbers are only valid when compared together, they must either both be in the argument list or neither. ***** Examples of the two possible calling formats are: istat = plib$set_bit_interlocked (flag) istat = plib$set_bit_interlocked (flag, loop_count, entry_count) In the first example, note the absence of an extra comma to denote defaulted arguments. Parallel Library V2 User Notes Page 21 5.6 Plib$clear_bit_interlocked This routine uses the VAX BBCCI instruction to clear the least significant bit of the argument. Format: istat = plib$clear_bit_interlocked (flag) Arguments: FLAG - Longword passed by reference whose least significant bit is to be cleared using the VAX BBCCI interlocked instruction. Return value: 1 - The bit was set upon calling this routine and has been cleared. 0 - The bit was not set upon calling this routine. Restrictions: Comments: As in the coding example at the end of the PLIB$SET_BIT_INTERLOCKED description, this routine is used to clear the flag bit to permit other processes to enter into the critical section. And yes, setting FLAG = 0 will accomplish the same function. However, the error checking that is done by this routine is generally worth the added overhead of the subroutine call. For example, accidentally saying FLOG = 0 would be a fatal algorithmic error in that the FLAG variable will not be cleared but there would obviously not be any compiler errors. Using this routine and accidentally typing FLOG instead of FLAG will generate the error return status since FLOG will probably be zero upon entry. Parallel Library V2 User Notes Page 22 5.7 Plib$add_word_interlocked This routine uses the VAX Add Aligned Word Interlocked instruction (ADAWI) to perform an interlocked addition. Format: istat = plib$add_word_interlocked (add_value, sum_value) Arguments: ADD_VALUE - The word value (passed by reference) to be added to the SUM_VALUE. SUM_VALUE - The word value (passed by reference) to which the ADD_VALUE argument will be added to. Return value: 0 - The final answer in SUM_VALUE was zero. 1 - The final answer in SUM_VALUE was non-zero. Restrictions: Comments: Be careful when using this routine that the program does not use the answer that is in SUM_VALUE. The problem is that the addition itself is performed interlocked, but that the memory fetching to use this value is not part of the interlocked sequence and thus another process could have already modified it by another call to PLIB$ADD_WORD_INTERLOCKED. (The return value for this routine is based upon the condition codes at the end of the ADAWI instruction and thus can be trusted.) This routine does have a useful purpose, however, and that is to implement count-down semaphores where the only interest is when the final value is zero. Testing the return value instead of SUM_VALUE can be used to check for this condition. Parallel Library V2 User Notes Page 23 6 INITIALIZATION ROUTINES 6.1 Plib$init_signals Format: istat = plib$init_signals (ef_start, num, proc_num) Arguments: EF_START - This argument specifies the first of three consecutive event flag numbers that the various Parallel Library routines can use to perform the subprocess synchronization. Longword passed by reference. NUM - The number of subprocesses that are going to be used in this application. Longword passed by reference. PROC_NUM - This argument is returned from the routine and specifies if the calling process was the main process or a subprocess. Possible values: 0 - This routine was called from a main process. >0 - This routine was called by a subprocess. Each subprocess will receive a unique number in the range of 1 to NUM based upon the order of calling the routine. Return value: The possible return values are: 1 - Success. 0 - An illegal range of event flags was specified. Anything else - Return value from a system service. Restrictions: Since the event flags must be shared, the entire event flag cluster that contains the specified event flags is set up as shared between the main process and the subprocesses. To minimize the errors caused by accidentally sharing event flags, specifying an event flag range that crosses a cluster boundary is illegal. This routine (along with the rest of the Parallel Library) require exclusive access to a PSECT (FORTRAN common block) named PLIB_COM. This common block will be set up as a shared global section whose name is the eight character PID of the main process appended to "PLIBSH". Parallel Library V2 User Notes Page 24 This routine must be called after PLIB$INIT_PAGEFILE has been called and after the optional call to PLIB$SHARE_CODE. (There are no restrictions on who to call first, this routine or calls to PLIB$SHARE_MEMORY.) Comments: To clarify the required calling sequence, this is the order in which the functions must be called: 1) PLIB$INIT_PAGEFILE must be called first. 2) PLIB$SHARE_CODE is optionally called next and can only be called once. 3) The call to PLIB$INIT_SIGNALS and the various optional calls to PLIB$SHARE_MEMORY can occur in any order. Parallel Library V2 User Notes Page 25 6.2 Plib$init_pagefile The PLIB$INIT_PAGEFILE routine initializes the access to the process private paging file and will, optionally, create the page file. Format: istat = plib$init_pagefile (page_file_name, create_option) Arguments: PAGE_FILE_NAME - The name of the page file to use. Character variable of up to 128 characters passed by reference. CREATE_OPTION - Defines if this routine is to create a new page file or to use an existing file. Values are: 0 - Use the existing file specified in PAGE_FILE_NAME. >0 - Create a new page file called PAGE_FILE_NAME with a size of CREATE_OPTION number of blocks. Return value: 1 - Success 0 - Error encountered initializing the page file. Restrictions: Comments: This routine, along with the other initialization routines, can be safely called from subprocesses with the appropriate action being taken. For example, if this routine is called by the main process and multiple subprocesses and requests that a new page file be created, only one page file will be created. To minimize problems of multiple processes being run from the same directory and trying to use the same filename, the suggestion is to append the main process's PID to the filename just as with the global section names. Parallel Library V2 User Notes Page 26 7 APPENDIX 1 - SAMPLE PROGRAM Parallel Library Userguide Sample Program Summary 7.1 General Overview The sample program included as part of the Parallel Library originated in a real paralleled program, but has had all of the "real work" removed to leave behind only the constructs required to implement and coordinate the parallelism. Error checking of the Parallel Library routine return statuses has also been eliminated to minimize the amount of non-coordination code. There are two parallel sections in this program, both of which do a few lines of code once a task has been allocated. (This program is quite heavily commented to explain various parallelism concepts. Without comments it is roughly one printed page. With comments, it is 12 pages long!) The program is called SAMPLE.FOR and is included as part of the Parallel Library package. *** Note *** The functions in these parallel sections are very simple for demonstration purposes and probably will never be replicated in a real program. The parallel sections of code that the subprocesses will execute are: 1) The first section contains 100 tasks. The only work involved for the task is writing out the task number and which subprocess executed it. 2) The second section copies a 100 element array into another 100 element array in ten tasks of ten locations each. As mentioned, these are not representative of real world programs, but the desire was to minimize the lines of code that weren't directly involved with coordinating the parallelism. Parallel Library V2 User Notes Page 27 The program flow is as follows (assuming chronological order in the vertical axis): Main process Subprocesses ------------ ------------ 1) Is executed. 2) Creates the shared memory global section and the shared event flags. 3) Creates the subprocesses. Are created. 4) Waits for the subprocesses Maps to the existing shared to report in as ready. memory global section and the shared event flags. 5) Reports back when all subprocesses are ready. Waits for main to signal work. 6) Main awakens. Sets up for and signals section 1 to be executed. 7) Main waits for subs to report Subprocesses awaken and completion of section 1. execute section 1. 8) Subprocesses perform section 1 and report when all are finished. Wait for main to signal the next work. 9) Main awakens and executes some dummy single-stream code to show where it could be done between sections 1 and 2. 9) Main repeats steps 6 Subprocesses repeat steps through 8 for section 2. 6 through 8 for section 2. 10) Main deletes the subprocesses and exits. By examining this program, the standard flow of a parallel program should be observable. The reader is directed to look at the sample program with the flow outline given above to observe how the Parallel Library routines can be used in a parallel application. (Any comment line starting with two asterisks (c**) has been added to this Parallel Library V2 User Notes Page 28 demonstration program specifically to help explain the background concepts.)