pygolang 0.0.5

GPython

Command gpython provides Python interpreter that supports lightweight threads via tight integration with gevent. The standard library of GPython is API compatible with Python standard library, but inplace of OS threads lightweight coroutines are provided, and IO is internally organized via libuv/libev-based IO scheduler. Consequently programs can spawn lots of coroutines cheaply, and modules like time, socket, ssl, subprocess etc - all could be used from all coroutines simultaneously, and in the same blocking way as if every coroutine was a full OS thread. This gives ability to scale programs without changing concurrency model and existing code.

Additionally GPython sets UTF-8 to be default encoding always, and puts go, chan, select etc into builtin namespace.

Goroutines and channels

go spawns a coroutine, or thread if gevent was not activated. It is possible to exchange data in between either threads or coroutines via channels. chan creates a new channel with Go semantic - either synchronous or buffered. Use chan.recv, chan.send and chan.close for communication. nilchan stands for nil channel. select can be used to multiplex on several channels. For example:

ch1 = chan()    # synchronous channel
ch2 = chan(3)   # channel with buffer of size 3

def _():
    ch1.send('a')
    ch2.send('b')
go(_)

ch1.recv()      # will give 'a'
ch2.recv_()     # will give ('b', True)

ch2 = nilchan   # rebind ch2 to nil channel
_, _rx = select(
    ch1.recv,           # 0
    ch1.recv_,          # 1
    (ch1.send, obj),    # 2
    ch2.recv,           # 3
    default,            # 4
)
if _ == 0:
    # _rx is what was received from ch1
    ...
if _ == 1:
    # _rx is (rx, ok) of what was received from ch1
    ...
if _ == 2:
    # we know obj was sent to ch1
    ...
if _ == 3:
    # this case will be never selected because
    # send/recv on nil channel block forever.
    ...
if _ == 4:
    # default case
    ...

By default chan creates new channel that can carry arbitrary Python objects. However type of channel elements can be specified via chan(dtype=X) - for example chan(dtype=’C.int’) creates new channel whose elements are C integers. chan.nil(X) creates typed nil channel. Cython/nogil API explains how channels with non-Python dtypes, besides in-Python usage, can be additionally used for interaction in between Python and nogil worlds.

Methods

func decorator allows to define methods separate from class.

For example:

@func(MyClass)
def my_method(self, ...):
    ...

will define MyClass.my_method().

func can be also used on just functions, for example:

@func
def my_function(...):
    ...

Defer / recover / panic

defer allows to schedule a cleanup to be executed when current function returns. It is similar to try/finally but does not force the cleanup part to be far away in the end. For example:

wc = wcfs.join(zurl)    │     wc = wcfs.join(zurl)
defer(wc.close)         │     try:
                        │        ...
...                     │        ...
...                     │        ...
...                     │     finally:
                        │        wc.close()

If deferred cleanup fails, previously unhandled exception, if any, won’t be lost - it will be chained with (PEP 3134) and included into traceback dump even on Python2.

For completeness there is recover and panic that allow to program with Go-style error handling, for example:

def _():
   r = recover()
   if r is not None:
      print("recovered. error was: %s" % (r,))
defer(_)

...

panic("aaa")

But recover and panic are probably of less utility since they can be practically natively modelled with try/except.

If defer is used, the function that uses it must be wrapped with @func decorator.

Import

gimport provides way to import python modules by full path in a Go workspace.

For example

lonet = gimport('lab.nexedi.com/kirr/go123/xnet/lonet')

will import either

• lab.nexedi.com/kirr/go123/xnet/lonet.py, or

• lab.nexedi.com/kirr/go123/xnet/lonet/__init__.py

located in src/ under $GOPATH.

Cython/nogil API

Cython package golang provides nogil API with goroutines, channels and other features that mirror corresponding Python package. Cython API is not only faster compared to Python version, but also, due to nogil property, allows to build concurrent systems without limitations imposed by Python’s GIL. All that while still programming in Python-like language. Brief description of Cython/nogil API follows:

go spawns new task - a coroutine, or thread, depending on activated runtime. chan[T] represents a channel with Go semantic and elements of type T. Use makechan[T] to create new channel, and chan[T].recv, chan[T].send, chan[T].close for communication. nil stands for nil channel. select can be used to multiplex on several channels. For example:

cdef nogil:
   struct Point:
      int x
      int y

   void worker(chan[int] chi, chan[Point] chp):
      chi.send(1)

      cdef Point p
      p.x = 3
      p.y = 4
      chp.send(p)

   void myfunc():
      cdef chan[int]   chi = makechan[int]()       # synchronous channel of integers
      cdef chan[Point] chp = makechan[Point](3)    # channel with buffer of size 3 and Point elements

      go(worker, chi, chp)

      i = chi.recv()    # will give 1
      p = chp.recv()    # will give Point(3,4)

      chp = nil         # rebind chp to nil channel
      cdef cbool ok
      cdef int j = 33
      _ = select([
          chi.recvs(&i)          # 0
          chi.recvs(&i, &ok),    # 1
          chi.sends(&j),         # 2
          chp.recvs(&p),         # 3
          default,               # 4
      ])
      if _ == 0:
          # i is what was received from chi
          ...
      if _ == 1:
          # (i, ok) is what was received from chi
          ...
      if _ == 2:
          # we know j was sent to chi
          ...
      if _ == 3:
          # this case will be never selected because
          # send/recv on nil channel block forever.
          ...
      if _ == 4:
          # default case
          ...

Python channels are represented by pychan cdef class. Python channels that carry non-Python elements (pychan.dtype != DTYPE_PYOBJECT) can be converted to Cython/nogil chan[T] via pychan.chan_*(). Similarly Cython/nogil chan[T] can be wrapped into pychan via pychan.from_chan_*(). This provides interaction mechanism in between nogil and Python worlds. For example:

def myfunc(pychan pych):
   if pych.dtype != DTYPE_INT:
      raise TypeError("expected chan[int]")

   cdef chan[int] ch = pych.chan_int()  # pychan -> chan[int]
   with nogil:
      # use ch in nogil code. Both Python and nogil parts can
      # send/receive on the channel simultaneously.
      ...

def mytick(): # -> pychan
   cdef chan[int] ch
   with nogil:
      # create a channel that is connected to some nogil task of the program
      ch = ...

   # wrap the channel into pychan. Both Python and nogil parts can
   # send/receive on the channel simultaneously.
   cdef pychan pych = pychan.from_chan_int(ch)  # pychan <- chan[int]
   return pych

panic stops normal execution of current goroutine by throwing a C-level exception. On Python/C boundaries C-level exceptions have to be converted to Python-level exceptions with topyexc. For example:

cdef void _do_something() nogil:
   ...
   panic("bug")   # hit a bug

# do_something is called by Python code - it is thus on Python/C boundary
cdef void do_something() nogil except +topyexc:
   _do_something()

def pydo_something():
   with nogil:
      do_something()

See libgolang.h and golang.pxd for details of the API. See also testprog/golang_pyx_user/ for demo project that uses Pygolang in Cython/nogil mode.

Additional packages and utilities

The following additional packages and utilities are also provided to close gaps between Python/Cython and Go environments:

print('hello %s' % qq(name))

def bench_add(b):
    x, y = 1, 2
    for i in xrange(b.N):
        x + y

$ py.bench --count=3 x.py
...
pymod: bench_add.py
Benchmarkadd    50000000        0.020 µs/op
Benchmarkadd    50000000        0.020 µs/op
Benchmarkadd    50000000        0.020 µs/op

Pygolang change history