F#, python, piping generators/iterators/IOs

A few weeks ago, I was watching this excellent introduction to F#, a programming language greatly inspired by other ML like languages.

http://channel9.msdn.com/pdc2008/TL11/

The pipeline operator |> is demonstrated in the video. It is argued that using this operator allows programmers to organize their code easily in a natural way.

This is meaningful for me because I've been avoiding a functional approach when using python because I would fear it would affect the readability of the code.

Look at at the following example, it starts with a few ordinary python examples, and then I added my "python pipe" response to F#.

# My Maths
def fA(x):
 print "In fA"
 return x+1

def fB(x):
 return x+2

def fC(x):
 print "In fC"
 return x+3

myInput = [1,2,3]

ugly = ( fA(i) for i in ( i+2 for i in ( fC(i) for i in myInput)))
print "About to process."
ugly = list(ugly) # execution starts here
print "ugly: %s" % ugly

# Lamda makes generator expressions reusable with different inputs
genA = lambda x : ( fA(i) for i in x)
genB = lambda x : ( i+2 for i in x)
genC = lambda x : ( fC(i) for i in x)

lessUgly = genA(genB(genC(myInput)))
lessUgly = list(lessUgly)
print "lessUgly: %s" % lessUgly

#or
genCi = genC(myInput)
genBi =genB(genCi)
genAi = genA(genBi)
# At least that way, generators appear in the "right" order

result = list(genAi)
print "result: %s" % result

#In newer versions of python map returns an iterator
# in older version take a look at imap in itertools
# (I'm currently python 2.6)

mapResult = map(fA, map(fB, map(fC, myInput)))
print "mapResult: %s" % mapResult

#or 

mapC = map(fC, myInput)
mapB = map(fB, mapC)
mapA = map(fA, mapB)

mapResult = list(mapA)
print "mapResult: %s" % mapResult


# Now with a pipe!

class Pipe:
 "Handles flow."
 
 def __init__ (self, source):
  self.lastIter = iter(source)
 
 def __iter__(self):
  return self.lastIter
 
 def __or__(self, right):
  try:
   it = iter(right)
  except:
   return NotImplemented
  self.lastIter = it
  return self


p = Pipe(myInput)
# Notice the generators appear in the order they are executed
p | ( fC(i) for i in p) | ( i+2 for i in p) | ( fA(i) for i in p)
result = list(p)
print "Piperesult: %s" % result

#or

p = Pipe(myInput)

p | ( fC(i) for i in p)
p | ( i+2 for i in p)
p | ( fA(i) for i in p)
result = list(p)
print "Piperesult: %s" % result

# or with map
p = Pipe(myInput)

p | map(fC, p)
p | map(fB, p)
p | map(fA, p)
result = list(p)
print "Piperesult: %s" % result

If you did watch the whole F# video, you might recall that at the end, threading is added just by adding a few keywords and symbols. I'm sure they are also many easy way to add threading to this approach in python. Maybe with a threaded map? tmap(numberOfThreads=4, inputBuffer=5, outputBuffer=5, myFunction, myInput). Or if the Pipe class would handle functions directly instead of iterators, it could decide on a threading strategy by itself?

Anyways that's it for today.