py3line 0.3.0

Installation

py3line is on PyPI, so simply run:

pip install py3line

or

sudo curl -L "https://61-63976011-gh.circle-artifacts.com/0/py3line-$(uname -s)-$(uname -m)" -o /usr/local/bin/py3line
sudo chmod +x /usr/local/bin/py3line

to have it installed in your environment.

For installing from source, clone the repo and run:

python setup.py install

Tutorial

Lets start with examples, we want to evaluate a number of words in each line:

$ echo -e "Here are\nsome\nwords for you." | ./py3line.py "x = len(line.split(' ')); print(x, line)"
2 Here are
1 some
3 words for you.

Py3line process input stream by python code line by line.

• echo -e “Here arensomenwords for you.” – create an input stream consists of three lines

• | – pipeline input stream to py3line

• “x = len(line.split()); print(x, line)” – define 2 actions: “x = len(line.split(’ ‘))” evaluate number of words in each line, then “print(x, line)” print the result. Each action apply to the input stream step by step.

The example above can be represented as the following python code:

import sys

def process(stream):
    for line in stream:
        x = len(line.split(' '))  # action 1
        print(x, line)            # action 2
        yield line

stream = (line.rstrip("\r\n") for line in sys.stdin if line)
stream = process(stream)
for line in stream: pass

You can also get the executed python code by --pycode argument.

$ ./py3line.py "x = len(line.split(' ')); print(x, line)" --pycode  #skipbashtest
...

$ echo -e "Here are\nsome\nwords for you." | ./py3line.py "line = len(line.split()); print(sum(stream))"
6

import sys

def process(stram):
    for line in stream:
        line = len(line.split())  # action 1
        yield line

def transform(stream):
    print(sum(stream))            # action 2
    return stream

stream = (line.rstrip("\r\n") for line in sys.stdin if line)
stream = transform(process(stream))
for line in stream: pass

$ ./py3line.py "line = len(line.split()); print(sum(stream))" --pycode  #skipbashtest
...

$ echo -e "Here are\nsome\nwords for you." | ./py3line.py "line = len(line.split()); print(sum(stream)); print(max(stream))"  #skipbashtest
6
2019-05-05 14:55:09,353 | ERROR   | Traceback (most recent call last):
  File "<string>", line 15, in <module>
    stream = transform2(process1(stream))
  File "<string>", line 10, in transform2
    print(max(stream))
ValueError: max() arg is an empty sequence

$ echo -e "Here are\nsome\nwords for you." | ./py3line.py "line = len(line.split()); stream = list(stream); print(sum(stream), max(stream))"
6 3

import sys

def process(stram):
    for line in stream:
        line = len(line.split())     # action 1
        yield line

def transform(stream):
    stream = list(stream)            # action 2
    print(sum(stream), max(stream))  # action 3
    return stream

stream = (line.rstrip("\r\n") for line in sys.stdin if line)
stream = transform(process(stream))
for line in stream: pass

$ echo -e "Here are\nsome\nwords for you." | ./py3line.py "s = 0; m = 0; num_of_words = len(line.split()); s += num_of_words; m = max(m, num_of_words); print(s, m)"
2 2
3 2
6 3

import sys

def process(stram):
    s = 0                                 # action 1
    m = 0                                 # action 2
    for line in stream:
        num_of_words = len(line.split())  # action 3
        s += num_of_words                 # action 4
        m = max(m, num_of_words)          # action 5
        print(s, m)                       # action 6
        yield line

stream = (line.rstrip("\r\n") for line in sys.stdin if line)
stream = process(stream)
for line in stream: pass

$ echo -e "Here are\nsome\nwords for you." | ./py3line.py "s = 0; m = 0; num_of_words = len(line.split()); s += num_of_words; m = max(m, num_of_words); for line in stream: pass; print(s, m)"
6 3

import sys

def process(stram):
    global s, m
    s = 0                                 # action 1
    m = 0                                 # action 2
    for line in stream:
        num_of_words = len(line.split())  # action 3
        s += num_of_words                 # action 4
        m = max(m, num_of_words)          # action 5
        yield line

def transform(stream):
    global s, m
    for line in stream: pass              # action 6
    print(s, m)                           # action 7
    return stream

stream = (line.rstrip("\r\n") for line in sys.stdin if line)
stream = transform(process(stream))
for line in stream: pass

$ echo -e "Here are\nsome\nwords for you." | ./py3line.py "for line in stream: print(1); for line in stream: print(1)"
1
1
1

import sys

def transform(stream):
    for line in stream: pass              # action 1 (3 iterations)
    for line in stream: pass              # action 2 (0 iterations)
    return stream

stream = (line.rstrip("\r\n") for line in sys.stdin if line)
stream = transform(stream)
for line in stream: pass                  # (0 iterations)

Details

Let us define some terminology. py3line “action1; action2; action3

We have actions: action1, action2 and action3. Each action have type. It may be element processing or stream transformation.

We can understand the type of action based on the variables used in it. We have two variables: line and stream. They are markers that define the type of action.

Lets look at some types from examples abow:

x = line.split()                 -- element processing
print(x, line)                   -- element processing
print(sum(stream))               -- stream transformation
stream = list(stream)            -- stream transformation
print(sum(stream), max(stream))  -- stream transformation
s = 0                            -- unidentified
m = 0                            -- unidentified
num_of_words = len(line.split()) -- element processing
s += num_of_words                -- unidentified
m = max(m, num_of_words)         -- unidentified
print(s, m)                      -- unidentified
for line in stream: pass         -- stream transformation

[rule1] If an action has an undefined type, it inherits its type from the previous action. [rule2] If there is no previous action, then the action is considered a stream transformation.

Examples:

s = 0                            -- stream transformation (because of [rule2])
num_of_words = len(line.split()) -- element processing (because of `line` marker)
s += num_of_words                -- element processing (because of [rule1])
print(s)                         -- element processing (because of [rule1])

And if we want to do print at the and, we should have some stream marker in actions before.

s = 0                            -- stream transformation (because of [rule2])
num_of_words = len(line.split()) -- element processing (because of `line` marker)
s += num_of_words                -- element processing (because of [rule1])
stream                           -- stream transformation (because of `stream` marker)
print(s)                         -- stream transformation (because of [rule1])

Unfortunately, it is not so clearly to people who are not familiar with the the implementation. Therefore, it is better to use a more explicit to readers actions like for line in stream: pass.

s = 0                            -- stream transformation (because of [rule2])
num_of_words = len(line.split()) -- element processing (because of `line` marker)
s += num_of_words                -- element processing (because of [rule1])
for line in stream: pass         -- stream transformation (because of `stream` marker)
print(s)                         -- stream transformation (because of [rule1])

Some examples

# Print every line (null transform)
$ cat ./testsuit/test.txt | ./py3line.py "print(line)"
This is my cat,
 whose name is Betty.
This is my dog,
 whose name is Frank.
This is my fish,
 whose name is George.
This is my goat,
 whose name is Adam.

# Number every line
$ cat ./testsuit/test.txt | ./py3line.py "stream = enumerate(stream); print(line)"
(0, 'This is my cat,')
(1, ' whose name is Betty.')
(2, 'This is my dog,')
(3, ' whose name is Frank.')
(4, 'This is my fish,')
(5, ' whose name is George.')
(6, 'This is my goat,')
(7, ' whose name is Adam.')

# Number every line
$ cat ./testsuit/test.txt | ./py3line.py "stream = enumerate(stream); print(line[0], line[1])"
0 This is my cat,
1  whose name is Betty.
2 This is my dog,
3  whose name is Frank.
4 This is my fish,
5  whose name is George.
6 This is my goat,
7  whose name is Adam.

Or just cat ./testsuit/test.txt | ./py3line.py "stream = enumerate(stream); print(*line)"

# Print every first and last word
$ cat ./testsuit/test.txt | ./py3line.py "s = line.split(); print(s[0], s[-1])"
This cat,
whose Betty.
This dog,
whose Frank.
This fish,
whose George.
This goat,
whose Adam.

# Split into words and print as list (strip al non word char like comma, dot, etc)
$ cat ./testsuit/test.txt | ./py3line.py "print(re.findall(r'\w+', line))"
['This', 'is', 'my', 'cat']
['whose', 'name', 'is', 'Betty']
['This', 'is', 'my', 'dog']
['whose', 'name', 'is', 'Frank']
['This', 'is', 'my', 'fish']
['whose', 'name', 'is', 'George']
['This', 'is', 'my', 'goat']
['whose', 'name', 'is', 'Adam']

# Split into words (strip al non word char like comma, dot, etc)
$ cat ./testsuit/test.txt | ./py3line.py "print(*re.findall(r'\w+', line))"
This is my cat
whose name is Betty
This is my dog
whose name is Frank
This is my fish
whose name is George
This is my goat
whose name is Adam

# Find all three letter words
$ cat ./testsuit/test.txt | ./py3line.py "print(re.findall(r'\b\w\w\w\b', line))"
['cat']
[]
['dog']
[]
[]
[]
[]
[]

# Find all three letter words + skip empty lists
cat ./testsuit/test.txt | ./py3line.py "line = re.findall(r'\b\w\w\w\b', line); if not line: continue; print(line)"
['cat']
['dog']

# Regex matching with groups
$ cat ./testsuit/test.txt | ./py3line.py "line = re.findall(r' is ([A-Z]\w*)', line); if not line: continue; print(*line)"
Betty
Frank
George
Adam

# cat ./testsuit/test.txt | ./py3line.py "line = re.search(r' is ([A-Z]\w*)', line); if not line: continue; line.group(1)"
$ cat ./testsuit/test.txt | ./py3line.py "rgx = re.compile(r' is ([A-Z]\w*)'); line = rgx.search(line); if not line: continue; print(line.group(1))"
Betty
Frank
George
Adam

# head -n 2
# cat ./testsuit/test.txt | ./py3line.py "stream = enumerate(stream); if line[0] >= 2: break; print(line[1])"
$ cat ./testsuit/test.txt | ./py3line.py "stream = list(stream)[:2]; print(line)"
This is my cat,
 whose name is Betty.

# Print just the URLs in the access log
$ cat ./testsuit/nginx.log | ./py3line.py "print(shlex.split(line)[13])"
HEAD / HTTP/1.0
HEAD / HTTP/1.0
HEAD / HTTP/1.0
HEAD / HTTP/1.0
HEAD / HTTP/1.0
GET /admin/moktoring/session/add/ HTTP/1.1
GET /admin/jsi18n/ HTTP/1.1
GET /static/admin/img/icon-calendar.svg HTTP/1.1
GET /static/admin/img/icon-clock.svg HTTP/1.1
HEAD / HTTP/1.0
HEAD / HTTP/1.0
HEAD / HTTP/1.0
HEAD / HTTP/1.0
HEAD / HTTP/1.0
GET /logout/?reason=startApplication HTTP/1.1
GET / HTTP/1.1
GET /login/?next=/ HTTP/1.1
POST /admin/customauth/user/?q=%D0%9F%D0%B0%D1%81%D0%B5%D1%87%D0%BD%D0%B8%D0%BA HTTP/1.1

# Print most common accessed urls and filter accessed more then 5 times
$ cat ./testsuit/nginx.log | ./py3line.py "line = shlex.split(line)[13]; stream = collections.Counter(stream).most_common(); if line[1] < 5: continue; print(line)"
('HEAD / HTTP/1.0', 10)

# create directory tree
echo -e "y1\nx2\nz3" | ./py3line.py "pathlib.Path('/DATA/' + line +'/db-backup/').mkdir(parents=True, exist_ok=True)"

group by 3 lines ... (https://askubuntu.com/questions/1052622/separate-log-text-according-to-paragraph)

$ ./py3line.py --help
usage: py3line.py [-h] [-v] [-q] [--version] [--pycode]
                  [expression [expression ...]]

Py3line is a UNIX command-line tool for a simple text stream processing by the
Python one-liner scripts. Like grep, sed and awk.

positional arguments:
  expression     python comma separated expressions

optional arguments:
  -h, --help     show this help message and exit
  -v, --verbose
  -q, --quiet
  --version      print the version string
  --pycode       show generated python code