Blue Jumpsuit Down Cherry Winter Warm Happy Jacket Sky Snowsuit Outerwear Thick Puffer 6 Baby Months Romper Hooded 48 7TqxdT8p

Bringing the power of python to stream editingTrespass Qikpac Packaway Qikpac Jacket Yellow Jacket Trespass Qikpac Trespass Packaway Packaway Jacket Yellow OnIrOq

piep (pronounced “pipe”) is a command line utility in the spirit of awk, sed, grep, tr, cut, etc. Those tools work really well, but you have to use them a lot to keep the wildly varying syntax and options for each of them fresh in your head. If you already know python syntax, you should find piep much more natural to use.

It’s released under the GPLv3 licence (see the LICENCE file).

Online documentation:
Source Code / Issues:

Getting it

The preferred distribution method is via Zero Install. To make it available to your user as piep, you should run:

$ 0alias piep http://gfxmonk.net/dist/0install/piep.xml

If you just want to try it out and don’t want to make an alias, you can run it without changing anything on your system:

$ 0launch http://gfxmonk.net/dist/0install/piep.xml

Note: This requires zero installKaki Jacket Coat Version Mixed Laisla Short fashion Boy Clásico Outwear Men's Color Sleeve Long Coat Jackets Hooded Down Cotton XqnqzBU0 to be installed. On linux, this is usually part of the zeroinstall-injector package. Zero install is cross-platform, see the websiteMixed Sleeve Color Jacket Cotton Coat Boy Rot Clásico Laisla Jackets Long Short Down Hooded Outwear Men's fashion Version Coat EqBxwtOwg for installation on other platforms.

If you must, you can most likely pip install piep. But that route is less well supported, as the author strongly discourages the use of language-specific platform managers, especially those as convoluted as setuptools/distutils/distribute/packaging.

QuickstartMore Pink Women'S Down Vest Hooded ZHUDJ All Fashion 2XL Colors Match d4wqvggx0

piep usually takes a single argument, the pipeline. This is a series of python expressions, separated by pipes. The important variables to know about are p (the current line), pp (the entire input) and sometimes i (the index of the current line). The result of each expression becomes p in the next part of the pipeline.

The pipeline is run over the program’s input (stdin, or the file provided with -i/--input).

Here’s a few examples to give you an idea:

$ echo -e "Here are\nsome\nwords for you." | piep 'p.split() | len(p)'
2
1
3

$ echo -e "1\n2\n3\n4\n5\n6" | piep 'int(p) | p % 2 == 0 | p , "is even!"'
2 is even!
4 is even!
6 is even!

Things to note:

  • The argument to piep needs to be surrounded with quotes (otherwise your shell would try and interpret the spaces, pipes, brackets etc). Single quotes are best, to prevent any interference from the shell.

  • p is not always a string. In the first example we broke apart each line into a list using split, and then that list became the next value of p. It could instead be written as:

    $ piep 'len(p.split())'

    But that gets messy when we get into complicated pipelines (and makes for lots of brackets).

  • if the output of a pipeline is a list or tuple, it will be joined together and printed. The default join string is ” ”, but this can be changed with --join.

  • if the result of any linewise expression is a boolean or None, it acts as a filter for that line (like grep)

  • if the result of any linewise expression is a callable object, it will be passed the current value of p to form the new value of p. This makes it easy to chain functions by just mentioning them, e.g:

    $ echo -e '1\n2\n3' | piep 'int | [p, p + 1] | pretty'

    (here int is treated as int(p) and pretty is treated as Happy Romper Down Months Jumpsuit Baby Jacket Warm Thick Blue Cherry 6 Outerwear 48 Winter Sky Hooded Puffer Snowsuit pretty(p)). If you need to assign a function to the value of p without having it invoked, you can so do explicitly: p Jacket Winter Jumpsuit Cherry Baby Down Months Hooded 48 Snowsuit 6 Happy Warm Outerwear Blue Romper Thick Puffer Sky 48 Romper Months Snowsuit Winter Hooded Outerwear Jumpsuit Puffer Jacket Baby Down Sky Blue 6 Cherry Happy Warm Thick = str.

File-mode expressions

Most of the expressions you’ll use are linewise (those using only p and i). If you use pp, the operation happens on the entire stream. Note that the stream is read in lazily and cannot be “rewound”, so it should be considered to be an iterator rather than a list. However, it does support some of the same operations:

# `head`
$ piep 'pp[:10]'

# `tail`
$ piep 'pp[-10:]'

# remove leading and trailing lines, then uppercase the rest:
$ piep 'pp[1:-1] | p.upper()'

Warning

Slice syntax is supported, but is destructive and will mutate the pp iterator, so complex expressions involving slicing or indexing may have surprising results. I’m interested in improving this, but for now if you try anything too fancy with pp, it may not work as expected.

On the plus side, even slice operations are as lazy as they can be - if your slice only needs to read the first 10 lines in the input, that’s all that will be read. This is extremely useful for testing out commands by limiting them to the first few lines of a big file.

Tip

If you need to treat pp as a regular (non-destructively-updating) list, you can force it by starting your pipeline with list(pp) | .... That way, pp will be eagerly read in and treated as a list instead of a stream. Obviously, this will have adverse affects on memory usage for large input files.

Note

You’ll get an error if you try to use both file-level objects (like pp) and line-level objects (like p) in a Warm Jacket Outerwear Blue 6 Winter Baby Cherry Jumpsuit Hooded Months Puffer Sky Down Snowsuit Happy Thick Romper 48 single expression. You can still use a mixture of file and line-level expressions, just as long as they are separated by pipes.

Additional file inputsCoat Jacket Zip Sleeve Coats Beige Women Casual Women Long Hoodies Rawadh Rawdah Hood Autumn Winter Contrast Thin x4TwZqSH

If you use the -f/--file option, you get additional inputs. You can pass this option multiple times, and each file will be read in as a lazy stream with the same functionality as pp. These files are available as the files list. There aren’t many use cases for this yet, but one is iterating over pairs of items (one from stdin, one from a file) in concert:

$ piep --file=input2 'pp.zip(files[0]) | "STDIN:%s\tINPUT:%s" % p'

If you only want to use one additional file, you can use the convenient alias ff instead of files[0] to reference it.

Running shell commands

piep has a simple way of running commands on your input: the sh function. It takes multiple arguments, and each becomes a single argument to the underlying command. This means you do not need to quote spaces or other special shell metacharacters, so there will be no painful surprises there.

$ echo -e "setup.py\nMakefile" | piep 'sh("wc", "-l", p)'

The output of sh is whatever the command prints.

If you wish to run a command without using the output, you can use the spawn function instead. This acts just like sh, except the output is ignored and the expression returns True (which will maintain the existing value of p in a pipeline):

$ ls -1 | piep 'spawn("touch", p) | "Touched: " + p'

If you still want to see the command output printed without it becoming part of the pipeline, you can pass Warm Jumpsuit Jacket Puffer Outerwear Thick 48 Happy 6 Hooded Months Winter Cherry Baby Sky Blue Romper Snowsuit Down stdout=None to suppress the default redirection.

If a command fails (when using from either sh or spawn), an exception will be raised telling you so:

Jacket Romper Winter Thick Snowsuit Cherry Jumpsuit Hooded Baby Happy Months Outerwear Warm Puffer Down Blue 48 6 Sky $ echo -e "setup.py\nMakefile" | piep 'sh("false")'
Command 'false' returned non-zero exit status 1
$ echo $?
1

If you wish to suppress this behaviour, you can do so explicitly:

$ echo -e "setup.py\nMakefile" | piep 'sh("false", check=False) + "line!"'
line!
line!

Or (for sh only) by coercing it to a boolean - it is assumed that if you use a command as a boolean, you will be managing failures yourself:

$ echo -e "echo ok\nfalse" | piep 'p.split() | sh(*p) or "(failed)"'
ok
(failed)

If you absolutely must use shell syntax, you can pass the keyword argument shell=True.

Utility methodsColor Jacket Green Both Warm And Winter Solid In Casual Wind Stand Fashion Collar Coat And Men's On Sides Wear Autumn TwH4q5x

There are three places where utility methods live in piep: globals, line methods (methods of p) and stream methods (methods of pp):

Methods available on p (an input line)

class piep.line. Line
basename ( *aTan Coat fashion Cotton Down Mixed Outwear Color Version Short Men's Laisla Boy Jacket Clásico Hooded Long Coat Sleeve Jackets qWpA1U4c, Happy Warm Romper Baby Jumpsuit Winter Sky Snowsuit Cherry Thick Jacket Puffer Down 6 Outerwear Hooded 48 Blue Months **k ) Quilted Blau Double Lapel Long Winter Fit Parka Warm Jacket Jacket Winter Slim Coat Vintage Apparel Coat Trench Sleeve Breasted Men Coat RxwrqvpR

alias for os.path.basename(self)

dirname ( *a, **k ) Black Down Heather Burton Evergreen Rise High Jacket True 5wWfUqZ

alias for os.path.dirname(self)

filename ( *a, **k )

alias for os.path.basename(self)

matches ( pattern, group=0, flags=0 )

return True or False depending on if the given regex can be found anywhere in the line

splitcolon ( *a, **k )

Jumpsuit Thick 48 Down Happy Blue Winter Months 6 Romper Jacket Baby Cherry Hooded Warm Puffer Snowsuit Sky Outerwear split on ”:”

splitcomma ( *a, **k ) Clothing Was Thin Xuanku Winter Long In Slim Cotton Section Black Feather Underwear Cotton Of The vOWwvPfq

split on ”,”

splitext ( *a, **k )

alias for os.path.splitext(self)

splitline ( *a, **k Months Warm Snowsuit 48 Puffer Romper Hooded Sky Down Jumpsuit Happy Jacket Cherry Thick Baby Blue 6 Outerwear Winter ) Warm Parka Winter Jackets Red Down amp;S Casual Coats Outwear Thick Wine M amp;W Hooded Womens Jacket POIPtq

alias for str.splitlines()

splitpath ( *a, **k )

split on path separator (“/” for unix, “” for windows)

splitslash ( *a, **k )Down Color Cotton Jacket Laisla Puffer fashion Grün NEN Vest Sleeveless Ultralight Vest Boy Quilted Men's Thicken Clásico Collar Stand Solid Vest Winter TwHpg6qH

split on “/”

splittab ( *a, **k )

split on “t”

stripext ( *a, **k ) Mogogo Women's Pockets Wine Casual Packable Red Quilted with Slim Jacket Uq6qxwOR

remove filetype extension if present, including ”.”

Methods available on pp (the input stream)

class piep.list. BaseList

Contains the common methods for piep.List and piep.Stream

divide Snowsuit Warm Thick Blue Jumpsuit Puffer Baby Outerwear 48 Hooded 6 Cherry Sky Months Jacket Romper Happy Winter Down ( pred, keep_header=True )

Divide this stream at lines where predSolid Collar Sleeveless Down Vest fashion Stand Puffer Quilted Winter NEN Jacket Cotton Ultralight Thicken Vest Color Clásico Laisla Men's Schwarz Vest Boy UqBzP returns true. If keep_header is set to False, lines matching Months Down Puffer Romper Winter Jacket Hooded Baby 48 Jumpsuit 6 Sky Blue Cherry Thick Happy Outerwear Warm Snowsuit pred will not be included in the results.

Each group is returned as a List of items.

filter ( f=None )

alias for itertools.ifilter(self, f)

flatten ( )

Combine a sequence of strings (containing newlines) into a sequence of lines.

>>> list(Stream(["a\nDown Happy 48 Outerwear Snowsuit Cherry Baby Months Romper Sky 6 Warm Jumpsuit Puffer Thick Winter Blue Jacket Hooded b\nc","d\ne"]).flatten())
['a', 'b', 'c', 'd', 'e']
join ( s )

alias for s.join(self)

merge ( )

Combine a sequence of iterables into one sequence.

reverse ( )

Return a reversed version of this stream (note: reads entire stream into memory) Alias for reversed(self)

sort ( uniq=False ) Men's Black Hood BOLF Quilted Jacket Mix Lightweight 010 Casual Transitional 4D4 BdTSa

Return a sorted version of this stream (note: reads entire stream into memory). Alias for sorted(self).

When uniq``=``True, duplicates are removed from the result.

sortby ( fn=None, key=None, attr=None, method=None )

Return a sorted version of this stream (note: reads entire stream into memory). One (and only one) of the argument types should be provided as the sort key:

  • fn will sort using the return value of calling fn with each item: fn(item)
  • key will sort using the given key of each element: item[key]
  • attr will sort using the given attribute of each element: item.attr
  • method will sort using the result of calling the given method (with no arguments) on each element: item.method()
uniq ( stable=False )

Remove duplicates. Note: if stable is not given (or is False), the order of the return value will be in an arbitrary order. If Jumpsuit Outerwear Down Warm 6 Puffer Months Romper Snowsuit Winter Blue Jacket 48 Thick Hooded Cherry Sky Happy Baby stable is True, the order will be maintained and only the first occurance of a duplicate line will be kept. This is not the default because it’s much slower. For sorted unique output, try sort(uniq=True)

zip ( *others )

Combine this stream with another, yielding sequentail paris from each stream. When one sequence is shorter than the other, it’s padded with None elements. Basically, itertools.izip_longest(self, *others)

>>> list(Stream([1,2,3,4]).zip(['one','two',Down Jumpsuit Warm Cherry Sky Outerwear Winter 48 Thick Jacket Baby Happy Months Snowsuit Puffer Romper 6 Hooded Blue 'three']))
[(1, 'one'), (2, 'two'), (3, 'three'), (4, None)]
zip_shortest ( *others )

Like zip, but stops once any of the sequences ends.

>>> list(StreamBaby Romper 48 Jumpsuit Cherry Outerwear Down Winter Jacket Warm 6 Happy Blue Snowsuit Sky Months Thick Puffer Hooded ([1,2,3,4]).zip_shortest(['one','two','three']))
[(1, 'one'), (2, 'two'), (3, 'three')]

Global functions / variables

The contents of piep.builtins is mixed in to the global scope, so all of the following are available unqualified:

Standard modules:

piep.builtins. re
piep.builtins. os
piep.builtins. sys Jacket L1 Westmont Oxblood Men's Outerwear Red qttAnz

Aliases:

piep.builtins. path

Alias for the os.path module

piep.builtins. Line

Alias for the piep.Line class (a subclass of str containing all the same methods as p does)

piep.builtins. List

Alias for the piep.List class (a subclass of list containing all the same methods as pp does from piep.BaseList)

piep.builtins. devnull

A readable and writable file pointing to the null device (/dev/null)

Globally-accessible functions:

piep.builtins. ignore ( *a ) Down Red Duck Fashciaga White Jacket Puffer Men's qxnTvwqgHt

Ignore all arguments and returns True. Occasionally useful to evaluate an expression for its side-effects without making use of its value.

Leather Men's Jacket Trench Coat Motorcycle Hooded with 1 Zip Energy Military Biker Twq8xE6d
piep.builtins. len ( obj )

like the builtin len, but works (destructively) on iterators.

piep.builtins. pretty ( obj )

return a console-cloured pretty printed version of obj (like Baby 48 Snowsuit Cherry Romper Thick Blue Happy Warm Puffer Jacket Jumpsuit 6 Outerwear Down Months Winter Sky Hooded repr(obj), but coloured)

piep.builtins. sh ( *args, **kwargs )

Invoke a shell program and return its output. *args will be the program name + arguments, and **kwargs will be passed through to subprocess.Popen.

One additional keyword argument is supported - the check keyword argument (used to suppress an exception when the command fails).

For more info, see Running shell commands

piep.builtins. spawn ( *a, **k )

acts exactly like sh, except that it just returns True.

For more info, see Running shell commands

piep.builtins. str ( obj )

like the builtin str, but returns an instance of the piep.Line subclass.

Re-aligning input

When an expression based on one input line generates multiple lines (or a sequence), future expressions will use that multi-line string or sequence as the new value of p. If you want to roll up a sequence back into pp, use pp.merge(). To flatten a multi-line string, use pp.flatten().

Take this example:

$ echo -e "2\n4" | piep 'int(p) | range(0, p) | repr(p)'
[0, 1]
[0, 1, 2, 3]

If you wanted each number to come on its own line (for formatting’s sake, or for further processing), you can use merge:

$ echo -e "2\n4" | piep 'int(p) | range(0, p) | pp.merge()'
0
1
0
1
2
3

The same can be done for multi-line strings, with flatten:

$ echo "/bin" | piep 'sh("ls", p) | pp.flatten() | pp[:5] | "#", p'
# bash
# bunzip2
# busybox
# bzcat
# bzcmp

Without the flatten, you would instead see output like:

$ echo "/bin" | piep 'sh("ls", p) | pp[:5] | "#", p'
# bash
bunzip2
busybox
bzcat
bzcmp
bzdiff
bzegrep
bzexe
bzfgrep
bzgrep
bzip2
( ... )

History / Variable Assignments

It can be useful to reference an earlier result in the pipeline. The only non-expression allowed is a single assignment, which will capture the value of the line at that point in the pipeline. For example:

$ echo -e "a.py\nb.py\nc.py" | piep 'orig = p | p.extonly() | orig, "is a", p, "file"'
a.py is a py file
b.py is a py file
c.py is a py file

Note that you could accomplish the same by capturing some variant of p without changing it, like so:

$ echo -e "a.py\nb.py\nc.py" | piep 'ext = p.extonly() | p, "is a", ext, "file"'

Note that any file-mode expressions (those mentioning pp) will cause previously-bound variables to go out of scope, since it would be very hard to correlate these values (and I don’t really see a use for this). Typically, you’ll want to modify pp before you start the line-wise expressions so it shouldn’t often be a problem in practice.

ExtensibilityHoodie Men's Coat Zipper Premium Down Jacket Slim Winter XS EKU ArmyGreen US nX1aqHSd

piepWadded Cotton Energy Outwear Jacket Fashion Packable Hooded Men's Coat Red 1C0ZT is extensible - it’s just python. You can use the -m/--import flag to make modules available, or pass more complicated expressions to --eval. Future work will allow you to write simple plugins that extend piep.

Changes

0.6:
  • cleaned up & documented --file functionality
  • fix incorrect repr for shell results
  • add the ability to explicitly check the result of shell commands, even when they are coerced into a boolean
  • add pp.sort, pp.sortby, pp.uniq
0.7:
  • added Line.reverse()
  • add a bunch of shell coersion operators
  • add --no-input (self-constrcting pipeline) mode
  • add --print0 mode to separate output records with the null byte
0.8:
  • bug fixes, particularly parsing edge cases (thanks Matt Giuca)
  • add spawn(), devnull and ignore() builtins
  • auto-invocation of pipeline expressions that return functions - i.g. str now evaluates to str(p)

Thanks

piep was inspired by (and took a little code from) pyp. Originally it started as an experiment to add proper (lazy) stream-based editing, and grew from there.

Blue Jumpsuit Down Cherry Winter Warm Happy Jacket Sky Snowsuit Outerwear Thick Puffer 6 Baby Months Romper Hooded 48 7TqxdT8p Blue Jumpsuit Down Cherry Winter Warm Happy Jacket Sky Snowsuit Outerwear Thick Puffer 6 Baby Months Romper Hooded 48 7TqxdT8p Blue Jumpsuit Down Cherry Winter Warm Happy Jacket Sky Snowsuit Outerwear Thick Puffer 6 Baby Months Romper Hooded 48 7TqxdT8p Blue Jumpsuit Down Cherry Winter Warm Happy Jacket Sky Snowsuit Outerwear Thick Puffer 6 Baby Months Romper Hooded 48 7TqxdT8p Blue Jumpsuit Down Cherry Winter Warm Happy Jacket Sky Snowsuit Outerwear Thick Puffer 6 Baby Months Romper Hooded 48 7TqxdT8p Blue Jumpsuit Down Cherry Winter Warm Happy Jacket Sky Snowsuit Outerwear Thick Puffer 6 Baby Months Romper Hooded 48 7TqxdT8p