codetransformer 0.7.2

Overloading Literals

While this can be done as an AST transformation, we will often need to execute the constructor for the literal multiple times. Also, we need to be sure that any additional names required to run our code are provided when we run. With codetransformer, we can pre compute our new literals and emit code that is as fast as loading our unmodified literals without requiring any additional names be available implicitly.

In the following block we demonstrate overloading dictionary syntax to result in collections.OrderedDict objects. OrderedDict is like a dict; however, the order of the keys is preserved.

>>> from codetransformer.transformers.literals import ordereddict_literals
>>> @ordereddict_literals
... def f():
...     return {'a': 1, 'b': 2, 'c': 3}
>>> f()
OrderedDict([('a', 1), ('b', 2), ('c', 3)])

This also supports dictionary comprehensions:

>>> @ordereddict_literals
... def f():
...     return {k: v for k, v in zip('abc', (1, 2, 3))}
>>> f()
OrderedDict([('a', 1), ('b', 2), ('c', 3)])

The next block overrides float literals with decimal.Decimal objects. These objects support arbitrary precision arithmetic.

>>> from codetransformer.transformers.literals import decimal_literals
>>> @decimal_literals
... def f():
...     return 1.5
>>> f()
Decimal('1.5')

Pattern Matched Exceptions

Pattern matched exceptions are a good example of a CodeTransformer that would be very complicated to implement at the AST level. This transformation extends the try/except syntax to accept instances of BaseException as well subclasses of BaseException. When excepting an instance, the args of the exception will be compared for equality to determine which exception handler should be invoked. For example:

>>> @pattern_matched_exceptions()
... def foo():
...     try:
...         raise ValueError('bar')
...     except ValueError('buzz'):
...         return 'buzz'
...     except ValueError('bar'):
...         return 'bar'
>>> foo()
'bar'

This function raises an instance of ValueError and attempts to catch it. The first check looks for instances of ValueError that were constructed with an argument of 'buzz'. Because our custom exception is raised with 'bar', these are not equal and we do not enter this handler. The next handler looks for ValueError('bar') which does match the exception we raised. We then enter this block and normal python rules take over.

We may also pass their own exception matching function:

>>> def match_greater(match_expr, exc_type, exc_value, exc_traceback):
...     return math_expr > exc_value.args[0]

>>> @pattern_matched_exceptions(match_greater)
... def foo():
...     try:
...         raise ValueError(5)
...     except 4:
...         return 4
...     except 5:
...         return 5
...     except 6:
...         return 6
>>> foo()
6

This matches on when the match expression is greater in value than the first argument of any exception type that is raised. This particular behavior would be very hard to mimic through AST level transformations.

Instructions

The Instruction object represents an atomic operation that can be performed by the CPython virtual machine. These are things like LOAD_NAME which loads a name onto the stack, or ROT_TWO which rotates the top two stack elements.

Some instructions accept an argument, for example LOAD_NAME, which modifies the behavior of the instruction. This is much like a function call where some functions accept arguments. Because the bytecode is always packed as raw bytes, the argument must be some integer (CPython stores all arguments two in bytes). This means that things that need a more rich argument system (like LOAD_NAME which needs the actual name to look up) must carry around the actual arguments in some table and use the integer as an offset into this array. One of the key abstractions of the Instruction object is that the argument is always some python object that represents the actual argument. Any lookup table management is handled for the user. This is helpful because some arguments share this table so we don’t want to add extra entries or forget to add them at all.

Another annoyance is that the instructions that handle control flow use their argument to say what bytecode offset to jump to. Some jumps use the absolute index, others use a relative index. This also makes it hard if you want to add or remove instructions because all of the offsets must be recomputed. In codetransformer, the jump instructions all accept another Instruction as the argument so that the assembler can manage this for the user. We also provide an easy way for new instructions to “steal” jumps that targeted another instruction so that can manage altering the bytecode around jump targets.

Code

Code objects are a nice abstraction over python’s types.CodeType. Quoting the CodeType constructor docstring:

code(argcount, kwonlyargcount, nlocals, stacksize, flags, codestring,
      constants, names, varnames, filename, name, firstlineno,
      lnotab[, freevars[, cellvars]])

Create a code object.  Not for the faint of heart.

The codetransformer abstraction is designed to make it easy to dynamically construct and inspect these objects. This allows us to easy set things like the argument names, and manipulate the line number mappings.

The Code object provides methods for converting to and from Python’s code representation:

1. from_pycode

2. to_pycode.

This allows us to take an existing function, parse the meaning from it, modify it, and then assemble this back into a new python code object.