Thoughts on Cython

Cython, is an epic project lets just be up front about it. I do make and have made some use of it professionally in previous projects.

There is one thing that keeps coming up and it really bugs me: “Cython to speed up your Python programs”. This in my opinion is a mis-representation of what cython is, but i am however being “hyper-critical”.

Cython has many uses i have illustrated them probably more than most different cases aimed at the hacker. Extending pure C/C++ projects directly without writing any C/C++ and doing it in Python is where it can seriously shine and be insane how powerful it can become, if your setup your project accordingly. My frustration comes from some small criticism on how i don’t really talk about speeding up _your_ python code.

There are several reasons why i seriously avoiding talking about this in the book and my recent cython article for linux format. But to be clear you need to understand what cython is and what it does.

Cython provides a way of natively working with C/C++ types and code. And the side-effect is your can compile cython code which just so happens to look a lot like python to be really efficient and strip out the use of the Python Runtime since you are directly using C types. This is fine but let’s be clear. Cython and Python are two very different Programming language, it just so happens that Cython can compile (most) Python code but Python cannot compile Cython code. This is essentially the problem.

So when your brand cython as a way to speed up python programs your essentially saying that you need to re-write your code in cython. And outside of the toy examples of single cython files it will not work. Remember cython compiles single files into single python modules. Meaning you cannot therefore point cython at a package or set of modules to compile this all down magically without some serious downsides on project structure and common-sense.

Cython just so happens to fit extremely well into scientific computing which is entirely different topic to normal software engineering and not only that scientific code is an entirely different beast of engineering. And in the scientific world having some python scripts with single files that do heavy computation using cython here to use C Types and compile it as a shared module to be imported into Python code works extremely well. Even on lists or maps scientific people will use the libPython api to directly access data. I feel there was 2 sides to the coin those who are scientific which will want this section and another section of people who want to know what can i do with cython in my code (games, servers, applications), really anything outside of statistical computation programs. Not only that but having a solid understanding of the uses i demonstrate will make this side seem trivial outside of the more fine-grained ways of making it just that little bit faster using more cython specifics and libpython specifics which would fit better as a paper than a technical book since it would up to much more change.

Emacs Reformatting a huge code base

Hey so one thing that bugged me and probably others if you join a company on a huge legacy code base and the code formatting or style is all over the place. Example tabs vs spaces, gnu vs linux vs something else.And you open it in vim or in gedit or emacs and it all looks different in each editor.

So i decided to learn a little lisp:

(defun format-me ()
   (c-set-style "linux") ;; preferred c style
   (indent-region (point-min) (point-max) nil) ;; format it
   (untabify (point-min) (point-max)) ;; untabify
   (save-buffer) ;;save

So this emacs lisp function will do the formatting i want on a specified file only trouble i had was, i was unable to figure out how to in emacs lisp recursively walk the directory tree to find all the *.c *.cc *.cpp *.h *.hh to run this on, so i ended up using bash so example to run this code from bash on a file i simple run:

$ emacs -batch <file-to-format> -l ~/format-lisp-code.el -f format-me

This runs and does it in 1 go so all i had to do now was find the files i wanted to reformat:

$ find -type f -regex ".*/.*\.\(c\|cpp\|cc\|h\|hh\)"

This returned me all i cared about in our code base then all i had to do was run:

for i in `find -type f -regex ".*/.*\.\(c\|cpp\|cc\|h\|hh\)"`; do emacs -batch $i -l ~/format-lisp-code.el -f format-me; done

And voila magic emacs you are so amazing!

Maven fail!

TLDR: Rant post about maven, java and jenkins plugin development

So recently i done some work on the Coverity Plugin for Jenkins.
Note: I think coverity is a complete waste of money by the way for C/C++ development the latest GCC/LLVM shows you as much as you need.

I’ve got some good experience with Maven from hadoop development. I understand the problem its trying to solve huge multi-module projects have so many dependencies. The java ecosystem is just HUGE, i mean to build this plugin from a clean maven it pulls down hundreds of jar’s as dependencies. Its kind of insane.

I want as you read this blog post to remember that, all this plugin actually does is runs your jenkins build as you would expepect but runs some extra coverity tools which you need to manually have installed on your slave automatically.

But working with jenkins plugins with maven is sooo horrible. I mean because the pom.xml had issues and warnings i couldn’t figure out how to use it with intellij, the thing would never build but just give really cryptic errors. Then i just go back to emacs, commandline and run mvn compile to add in a MacroExpansionToken then to test it i have a local tomcat instance, so i have the jenkins.war already running there with a helloworld build to test the code. I run mvn clean compile install -DskipTests to create the jenkins .hpi then manually go to jenkins manage plugins and upload it. Then restart jenkins run the build and then find a bug go back to the begining.

Its so slow and painful and plugin development is sooooo awful with jenkins. I mean its the problem with java overall, making everything i mean everything so generic, yeah sure you can really easily reuse _any_ piece of code or embed some cool service into an application. But because everything has such generic interfaces programming with them you usually have really obscure abstract interfaces to work with and you need to wrap your head around before you can actually solve the problem.

I mean maven isn’t any kind of useful build tool it doesn’t even build software well, its a lump of horrible xml where you don’t even declare what you want it just automatically treats anything in src/main/{java/groovy/test/…} as a .jar unless you want a jar. This xml is really is awful to work with no documentation and a plugin for everything.

I mean if you don’t have an IDE like eclipse of intellij java development is almost just entirely impossible these days. There is just so much crap and boilerplate to fill out you won’t remember all of it to do it correctly.

It instantly forces tonnes of convention before it actually solves anything. Ant and ivy was so nice to work with before. I dont understand how maven made anything easier i mean i can sort of see it with hadoop it works pretty ok there. But even then does it?

With C/C++ or even Python or Node everything is so simple and if you need more complex systems there is alternatives for that. Everything about those languages they are very focused on solving an issue. Golang is about the only language that exists that almost reminds me of maven like a revel project or even just a simple library the dependency management is nice but it forces a fair amount of convention, but i think its bearable because its been like this always not like maven just poping along after a while and everything is quite fragmented.

I mean Java or any JVM or any .NET software is the only languages where you can spend literally weeks declaring boilerplate data-flow and object models for weeks. Before you actually write code to do what you need.

i386 Assembly Programming Tutorial pt1

Assembly programming is a black hole for new programmers these days, its almost impossible to find really easy to follow tutorial on the subject but some people in work were talking about it and i felt i could shed some light on it.

Consider a really simple program in C

int main (void)
  return 2;

You can compile this:

$ gcc -g -O2 -Wall test.c -o test

And whoop we compiled a very useless program but when we run it we get:

$ ./test; echo $?

So we done something! But how does all of that work i mean how was this able to return a value to the shell. We could write this in assembly (i386/x86) and i can show you what happens here to return this value its call the c-decl ABI (

.global	main
	mov $2, %eax

To compile this you simply:

$ as fib2.s -o fib2.o
$ gcc -o test fib2.o
$ ./test; echo $?

So what does this mean, well i am sure it looks some what clear whats going on, we declared a global symbol main and its code simply puts the value 2 into the register eax and returns. The ret opcode is a fairly useful one, it is responsible for restoring the instruction pointer for the program back to the caller. You can do all of this manually by messing with the instruction pointer yourself but thats for another day see (

In the next post i will write how we can call into libc functions such as printf for a hello world from assembly in i386. This will introduce the stack.

Career vs Life

Hey all reading this, its a fairly personal post and probably controversial for me to make.

About a year ago i was working for a great wee company WANdisco i had such a great set of people to work with, we were all so close and work was such a joy to go to. Exciting and vibrant when i was working there for bit over a year and i started to yearn for a more ‘low-level’ challenge such as a low-latency C/C++ application. I felt i had so much to learn in this area and its where i wanted my career to go towards.

So i joined where i am now (not going to write to avoid google picking it up) and i began to realise the pain of legacy systems programming and in my opinion i found it a challenge to some extent but it was never the code base i had issues with i mean with my experience in open source and GCC which is older and bigger than most legacy code-bases you will ever find. The issues i had were more high-level, we were never given freedom to be pro-active about maintaining code (fixing things before they become an issue). This lead to much time being very idle and frustrated at being quite literately un-challenged. And this was seen to be ‘busy’ with everyone embellishing their job and work to sound better to hopefully eventually gain some kind of promotion or some kind of management role.

And though this job i found such frustration i became very depressed to the point i though almost that Software Engineering isn’t a career path that i would want to pursue any longer….

But now its all fixed and for the first time in probably 2 years i feel settled, i mean i went to lunch and everything seemed brighter and better than i have felt in so long and i have been smiling all day. I never realised how depressed i had become, its so dangerous when i bottled things up so much that it would personify as physical illness such as bad colds and coughs on and off. I would bottle up so much because my Significant Other i really didn’t want for her to have to deal with my problems i love her so much and i just want to make her happy but my career was bringing us both down.

But you know what i finally have a new job where i will be starting in about a month (will reveal closer to the time). And you know what the people were so nice to me and challenged me and made me work in the interview i felt at peace again. Like i knew what i was doing again being proactive challenging myself and had the desire to push myself. And more importantly understood why i participate in open-source, at this job very very frequently you will be forced to deal with this question ‘sure, why would you do that? Were you paid? No? What…?’.

Its probably important to note, their reasons were i guess to some extent legitimate such as project approvals from such a large corporate entity and product stability. But regardless of this i would still argue that failure to fix and update is kind of like buying a house that really needs the plumbing and electrics re-done, sure you can live in it and use it but its not that pleasant and has a very high likely hood of costing you a lot in the long run.

Though this year i have talked to many many people on-line and off-line; strangers who become friends on a bus from different walks of life and different careers. And the universal notion we all have had is that, career will only take you so far because in the end it is we who have to live our lives and we can’t let circumstances get in the way of enjoying life.

My depression at this job was so great i believe if it went on for another year or 2 it probably was getting close to costing me my relationship with my Significant Other and maybe even my life. I probably shouldn’t go into the nitty details of what it was like here but if your in a similar position i would love to hear from you via email or if your local to Northern Ireland i would like to share a coffee with you.

Take care of your mental health i mean seriously the stigma it has in some countries still scares me such as Northern Ireland. Your likely to hear people equate Mental Health with the ‘luney bin’.