In Unix and other Unix-like operating systems, a pipeline is a set of process (computing)es chained by their standard streams, so that the output of each process ( stdout ) feeds directly as input ( stdin ) of the next one. filter (Unix)s are often used in this way. The concept was named by analogy to a physical pipeline.

This feature of Unix was borrowed by other operating systems, such as Taos operating system and MS-DOS, and eventually became the pipes and filters design pattern (computer science) of software engineering. Unix pipelines should not be confused with other pipeline (computer)s found in modern computer systems, although the general concept is quite similar.

= Pipelines in Command line interfaces =

Most Unix shells have a special syntax construct for the creation of pipelines. Typically, one simply writes the filter commands in sequence, separated by the ASCII vertical bar character | (which, for this reason, is often called pipe (computing) by Unix users). The shell starts the processes and arranges for the necessary connections between their standard streams (including some amount of buffer storage). Multi-line entry is possible with the break slash, after each pipe.

==Error stream==

By default, the standard error streams ( stderr ) of the processes are not passed on through the pipe; instead, they are merged and directed to the computer console. However, many shells have additional syntax for changing this behaviour. In the csh shell, for instance, using |& instead of | signifies that the standard error stream too should be merged with the standard output and fed to the next process.

= Pipelines in Graphical user interfaces =

Graphical environments such as RISC OS and ROX Desktop also make use of pipelines. Rather than providing a save dialog box containing a file manager to let the user specify where a program (computing) should write data, RISC OS and ROX provide a save dialog box containing an icon (computing) (and a field to specify the name). The destination is specified by drag-and-drop the icon. The user can drop the icon anywhere an already-saved file could be dropped, including onto icons of other programs. If the icon is dropped onto a program s icon, it s loaded and the contents that would otherwise have been saved are passed in on the new program s standard input stream.

For instance, a user browsing the world-wide web might come across a .gz compressed image which they want to edit and re-upload. Using GUI pipelines, they could drag the link to their de-archiving program, drag the icon representing the extracted contents to their image editor, edit it, open the save as dialog, and drag its icon to their uploading software.

Conceptually, this method could be used with a conventional minifiler savebox, but this would require the users programs to have an obvious and easily-accessible location in the filesystem that can be navigated to. In practice, this is often not the case, so GUI pipelines are rare.

=Example=

Below is an example of a pipeline that implements a kind of spell checker for the World Wide Web resource indicated by a Uniform Resource Locator [http://en.wikipedia.org/wiki/Pipeline].

CURL http://en.wikipedia.org/wiki/Pipeline | Sed s/[^a-zA-Z ]//g | tr A-Z a-z | Grep [a-z] | sort -u | comm (Unix) -23 - /usr/dict/words

Here is an explanation of the pipeline:

*First the curl program obtains the HTML contents of a web page. *The contents of this page are piped through sed, which removes all characters which are not spaces or letters. *tr then changes all of the uppercase letters into their corresponding lowercase counterparts, and converts the spaces in the lines of text to newlines. *Each word is now on a separate line. *grep is used to remove lines of whitespace. *sort sorts the list of words into alphabetical order, and removes duplicates. *Finally, comm finds which of the words in the list are not in the given dictionary file (in this case, /usr/dict/words).

=Creating pipelines by program=

Pipelines can be created under program control. The pipe() system call asks the operating system to construct a new pipe object. This results in two new, opened file descriptors in the process: the read-only end of the pipe, and the write-only end. The pipe ends then appear as normal, anonymous Computer file except that they have no ability to seek.

To avoid deadlock and exploit parallelism, the process with one or more new pipes will then, generally, call fork (computing) to create new processes. Each process will close the end(s) of the pipe that it will not be using, before producing and consuming data. Alternatively, the first process may simply create a new pthreads and ensure only one thread has access to one pipe end.

Named pipes may also be created using mkfifo() or mknod() and then presented as the input or output file to programs as they are invoked.

=Implementation=

In most Unix-like systems, all processes of a pipeline are started at the same time, with their streams appropriately connected, and managed by the , by the operating system so that the receiving program need not worry about dropping data on account of it being too busy to receive it. Buffers also collect data from their senders as soon as it is made available, so that a sender need not finish its job, or exit, before a receiver can start its work on the product.

Other implementations of pipes have provided pipe-like functionality without multitasking. Under MS-DOS, for example, only one process could be running at the same time, but it could start another process, which would then need to complete before the initial process could recover control. So when pipes were used, the COMMAND.COM shell would first create a temporary buffer file, making sure this file is its standard output. Then it would start the sending process, which would inherit the buffer file as its standard output and would write its entire output to it. Once the sending process had terminated, the shell would close the buffer file and open it again, in read mode, as its standard input. The shell would then run the second, receiving process, which would inherit the buffer file as standard input. This provided similar functionality to Unix shell pipes, but required processes to complete their work before handing their ouput off to the receiver. This was impractical for long-running processes, therefore MS-DOS programs often offered their own output pagination if they output lots of text data on their standard output, rather than relying on the user to pipe them through the standard more.exe utility. It should be noted that many (if not most) non-trivial MS-DOS programs, did not use MS-DOS file handles for either input or output, and instead read the keyboard directly at BIOS level or performed BIOS calls for output (or even wrote directly to video memory). Such programs could not be redirected through pipes in either direction.

Tools like Netcat can connect pipes to TCP/IP sockets, following the Unix philosophy of everything is a file .

CMS Pipelines is a port of the pipeline idea to VM/CMS and MVS systems. It supports much more complex pipeline structures than Unix shells, with steps taking multiple input streams and producing multiple output shells. (Such functionality is supported by the Unix kernel, but few programs use it.) Due to the different nature of IBM mainframe operating systems, it implements many steps inside CMS Pipelines which in Unix are separate external programs, but can also call separate external programs for their functionality. Also, due to the record-oriented nature of files on IBM mainframes, the pipelines also operate in a record-oriented rather than stream-oriented manner.

= History=

The pipeline concept and the vertical-bar notation was invented by Douglas McIlroy, one of the authors of the early Unix shell, after he noticed that much of the time they were processing the output of one program as the input to another. The idea was eventually ported to other operating systems, such as DOS, OS/2, Windows NT, BeOS, and Mac OS X, often with the same notation.

= See also =