codetransformer 0.5.0

visit_{OP}

Just like the NodeTransformer, we write visit_* methods that define how we act on an instruction.

For example (taken from my lazy library):

def visit_UNARY_NOT(self, instr):
    """
    Replace the `not` operator to act on the values that the thunks
    represent.
    This makes `not` lazy.
    """
    yield self.LOAD_CONST(_lazy_not).steal(instr)
    # TOS  = _lazy_not
    # TOS1 = arg

    yield ROT_TWO()
    # TOS  = arg
    # TOS1 = _lazy_not

    yield CALL_FUNCTION(1)
    # TOS = _lazy_not(arg)

This visitor is applied to a unary not instruction (not a) and replaces it with code that is like: _lazy_not(a)

These methods will act on any opcode.

These methods are passed an Instruction object as the argument.

visit_{OTHER}

Code objects also have some data other than their bytecode. We can act on these things as well.

The following methods act in the form of visit_* -> co_*, for example, visit_name acts on the co_name field.

1. visit_name

2. visit_names

3. visit_varnames

4. visit_freevars

5. visit_cellvars

6. visit_defaults

7. visit_consts

A note about visit_const: One should be sure to call super().visit_const(const) inside of their definiton to recursivly apply your transformer to nested code objects.

const_index

One of the best uses of a bytecode transform is to make something available at runtime without putting a name in the namespace. We can do this by putting a new entry in the co_consts.

The const_index function accepts the value you want to put into the consts and returns the index as an int. This will create a new entry if needed.

The LOAD_CONST method of a CodeTransformer is a shortcut that returns a LOAD_CONST instruction object with the argument as the index of the object passed.

steal

steal is a method of the Instruction object that steals the jump target of another instruction. For example, if an instruction a is jumping to instruction b and instruction c steals b, then a will jump to b. This is useful when you are replacing an instruction with a transformer but want to preserve jumps.

Applying a Transformer to a Function

An instance of CodeTransformer is callable, accepting a function and returning a new function with the bytecode modified based on the rules of the transformer. This allows a CodeTransformer to be used as a decorator, for example:

>>> @mytransformer()
... def f(*args):
...     ...
...     return None

asconstants

This decorator will inline objects into a piece of code so that the names do not need to be looked up at runtime.

Example:

>>> from codetransformer.transformers import asconstants
>>> @asconstants(a=1)
>>> def f():
...     return a
...
>>> f()
1
>>> a = 5
>>> f()
1

This will work in a fresh session where a is not defined because the name a will be inlined with the constant value: 1. If a is defined, it will still be overridden with the new value.

This decorator can also take a variable amount of of builtin names:

>>> tuple = None
>>> @asconstants('tuple', 'list')
... def f(a):
...     if a:
...         return tuple
...     return list
...
>>> f(True) is tuple
False

These strings are take as the original builtin values, even if they have been overridden. These will still be faster than doing a global lookup to find the object. If no arguments are passed, it means: assume all the builtin names are constants.

optimize

The CPython peephole optimizer is only run once over the bytecode; however, sometimes some optimizations do not present themselves until a second pass has been made. One example of this is De Morgan’s Laws. Using the following code as an example:

>>> from dis import dis
>>> def f(a, b):
...     if not a and not b: return None
...
>>> dis(f)
2           0 LOAD_FAST                0 (a)
            3 UNARY_NOT
            4 POP_JUMP_IF_FALSE       18
            7 LOAD_FAST                1 (b)
           10 UNARY_NOT
           11 POP_JUMP_IF_FALSE       18
           14 LOAD_CONST               0 (None)
           17 RETURN_VALUE
      >>   18 LOAD_CONST               0 (None)
           21 RETURN_VALUE
>>> from codetransformer.transformers import optimize
>>> @optimize()
... def g(a, b):
...     if not a and not b: return None
...
>>> dis(g)
3           0 LOAD_FAST                0 (a)
            3 POP_JUMP_IF_TRUE        16
            6 LOAD_FAST                1 (b)
            9 POP_JUMP_IF_TRUE        16
           12 LOAD_CONST               0 (None)
           15 RETURN_VALUE
      >>   16 LOAD_CONST               0 (None)
           19 RETURN_VALUE

This shows that we can get a pretty decent win for no effort at all. The optimize transformer takes a keyword argument: passes, that denotes the number of passes of the peephole optimizer to run. Just like this optimization is ironed out on the second pass, there may exist some that require 2 or 3 passes to work.

Overloaded Literals

The codetransfomer.transformers.literals module includes transformers designed to allow for overloading the meaning of certain literal values. This allows us to front load some work to compile time and make some operations for readable. One example is ordereddict_literals. This transformer instance changes all dictionary literals into collection.OrderedDict instances. For example:

>>> from codetransfomer.transformers.literals import ordereddict_literals
>>> @ordereddict_literals
... def f():
...     return {'a': 1, 'b': 2, 'c': 3}
...
>>> f()
OrderedDict([('a', 1), ('b', 2), ('c', 3)])

Another example is the decimal transformer. This transformer turns float literals into Decimal literals. For example:

>>> from codetransfomer.transformers.literals import decimal_literals
>>> @decimal_literals
... def f():
...     return 1.5
...
>>> f()
Decimal('1.5')