Pipeline

A pipeline has meaning in electrical and mechanical systems, as well as in software. In general, the term represents the concept of splitting a job into subprocesses in which the output of one subprocess feeds into the next (much like water flows from one pipe segment to the next).

A mechanical example of a pipeline is a washer/dryer system for clothing. Instead of having one unit that both washes and dries, we have two units that together form a pipeline (the output of the washer enters the drier). If washing takes 1 hour and drying takes 1 hour, the pipeline allows us to finish a full load of laundry every hour, compared to every 2 hours if you had a single (non-pipelined) unit that washed and then dried. It still requires two hours for an item of clothing to complete its wash/dry cycle of course.

Electrically, pipelines are used in microprocessors to allow complex logic sequences to execute at faster speeds. Pipelines are related to the engineering concepts of throughput and latency. In contrast to the washer/drier example above, an instruction pipelining process can increase the instruction throughput without a significant increase in circuit complexity, indeed perhaps the simplest analogy would be a washer/drier mechanism connected to a single drive shaft using a single driving motor? Seven or more pipelining stages are common in microprocessor designs.

In computer software, a pipeline is a [command line]? feature prevalent in UNIX and other unix-like operating systems. The authors of the early UNIX command shells noticed that much of the time they were processing the output of one program as the input to another. They established a means of chaining the running programs together as co-processes? so that the output of the first program becomes the input to the second. A pipeline may be extended to any number of commands with the output of one serving as the input to the next.

Most UNIX commands obey a few conventions so that they may be used in a pipeline. These include: line structured records, reading data from the standard input, and writing to the standard output.

Below is an example of a pipeline that implements a kind of spell checker for this page.

 fetch -o - http://www.wikipedia.com/wiki.cgi?Pipeline |
 sed 's/[^a-zA-Z]/  /g' |
 tr 'A-Z' 'a-z' |
 sort -u |
 comm -23 - /usr/share/dict/web2