Thoughts on Cython

Cython, is an epic project lets just be up front about it. I do make and have made some use of it professionally in previous projects.

There is one thing that keeps coming up and it really bugs me: “Cython to speed up your Python programs”. This in my opinion is a mis-representation of what cython is, but i am however being “hyper-critical”.

Cython has many uses i have illustrated them probably more than most different cases aimed at the hacker. Extending pure C/C++ projects directly without writing any C/C++ and doing it in Python is where it can seriously shine and be insane how powerful it can become, if your setup your project accordingly. My frustration comes from some small criticism on how i don’t really talk about speeding up _your_ python code.

There are several reasons why i seriously avoiding talking about this in the book and my recent cython article for linux format. But to be clear you need to understand what cython is and what it does.

Cython provides a way of natively working with C/C++ types and code. And the side-effect is your can compile cython code which just so happens to look a lot like python to be really efficient and strip out the use of the Python Runtime since you are directly using C types. This is fine but let’s be clear. Cython and Python are two very different Programming language, it just so happens that Cython can compile (most) Python code but Python cannot compile Cython code. This is essentially the problem.

So when your brand cython as a way to speed up python programs your essentially saying that you need to re-write your code in cython. And outside of the toy examples of single cython files it will not work. Remember cython compiles single files into single python modules. Meaning you cannot therefore point cython at a package or set of modules to compile this all down magically without some serious downsides on project structure and common-sense.

Cython just so happens to fit extremely well into scientific computing which is entirely different topic to normal software engineering and not only that scientific code is an entirely different beast of engineering. And in the scientific world having some python scripts with single files that do heavy computation using cython here to use C Types and compile it as a shared module to be imported into Python code works extremely well. Even on lists or maps scientific people will use the libPython api to directly access data. I feel there was 2 sides to the coin those who are scientific which will want this section and another section of people who want to know what can i do with cython in my code (games, servers, applications), really anything outside of statistical computation programs. Not only that but having a solid understanding of the uses i demonstrate will make this side seem trivial outside of the more fine-grained ways of making it just that little bit faster using more cython specifics and libpython specifics which would fit better as a paper than a technical book since it would up to much more change.

Emacs Reformatting a huge code base

Hey so one thing that bugged me and probably others if you join a company on a huge legacy code base and the code formatting or style is all over the place. Example tabs vs spaces, gnu vs linux vs something else.And you open it in vim or in gedit or emacs and it all looks different in each editor.

So i decided to learn a little lisp:

(defun format-me ()
   (c-set-style "linux") ;; preferred c style
   (indent-region (point-min) (point-max) nil) ;; format it
   (untabify (point-min) (point-max)) ;; untabify
   (save-buffer) ;;save
)

So this emacs lisp function will do the formatting i want on a specified file only trouble i had was, i was unable to figure out how to in emacs lisp recursively walk the directory tree to find all the *.c *.cc *.cpp *.h *.hh to run this on, so i ended up using bash so example to run this code from bash on a file i simple run:

$ emacs -batch <file-to-format> -l ~/format-lisp-code.el -f format-me

This runs and does it in 1 go so all i had to do now was find the files i wanted to reformat:

$ find -type f -regex ".*/.*\.\(c\|cpp\|cc\|h\|hh\)"

This returned me all i cared about in our code base then all i had to do was run:

for i in `find -type f -regex ".*/.*\.\(c\|cpp\|cc\|h\|hh\)"`; do emacs -batch $i -l ~/format-lisp-code.el -f format-me; done

And voila magic emacs you are so amazing!

Maven fail!

TLDR: Rant post about maven, java and jenkins plugin development

So recently i done some work on the Coverity Plugin for Jenkins.
Note: I think coverity is a complete waste of money by the way for C/C++ development the latest GCC/LLVM shows you as much as you need.

I’ve got some good experience with Maven from hadoop development. I understand the problem its trying to solve huge multi-module projects have so many dependencies. The java ecosystem is just HUGE, i mean to build this plugin from a clean maven it pulls down hundreds of jar’s as dependencies. Its kind of insane.

I want as you read this blog post to remember that, all this plugin actually does is runs your jenkins build as you would expepect but runs some extra coverity tools which you need to manually have installed on your slave automatically.

But working with jenkins plugins with maven is sooo horrible. I mean because the pom.xml had issues and warnings i couldn’t figure out how to use it with intellij, the thing would never build but just give really cryptic errors. Then i just go back to emacs, commandline and run mvn compile to add in a MacroExpansionToken then to test it i have a local tomcat instance, so i have the jenkins.war already running there with a helloworld build to test the code. I run mvn clean compile install -DskipTests to create the jenkins .hpi then manually go to jenkins manage plugins and upload it. Then restart jenkins run the build and then find a bug go back to the begining.

Its so slow and painful and plugin development is sooooo awful with jenkins. I mean its the problem with java overall, making everything i mean everything so generic, yeah sure you can really easily reuse _any_ piece of code or embed some cool service into an application. But because everything has such generic interfaces programming with them you usually have really obscure abstract interfaces to work with and you need to wrap your head around before you can actually solve the problem.

I mean maven isn’t any kind of useful build tool it doesn’t even build software well, its a lump of horrible xml where you don’t even declare what you want it just automatically treats anything in src/main/{java/groovy/test/…} as a .jar unless you want a jar. This xml is really is awful to work with no documentation and a plugin for everything.

I mean if you don’t have an IDE like eclipse of intellij java development is almost just entirely impossible these days. There is just so much crap and boilerplate to fill out you won’t remember all of it to do it correctly.

It instantly forces tonnes of convention before it actually solves anything. Ant and ivy was so nice to work with before. I dont understand how maven made anything easier i mean i can sort of see it with hadoop it works pretty ok there. But even then does it?

With C/C++ or even Python or Node everything is so simple and if you need more complex systems there is alternatives for that. Everything about those languages they are very focused on solving an issue. Golang is about the only language that exists that almost reminds me of maven like a revel project or even just a simple library the dependency management is nice but it forces a fair amount of convention, but i think its bearable because its been like this always not like maven just poping along after a while and everything is quite fragmented.

I mean Java or any JVM or any .NET software is the only languages where you can spend literally weeks declaring boilerplate data-flow and object models for weeks. Before you actually write code to do what you need.

i386 Assembly Programming Tutorial pt1

Assembly programming is a black hole for new programmers these days, its almost impossible to find really easy to follow tutorial on the subject but some people in work were talking about it and i felt i could shed some light on it.

Consider a really simple program in C

int main (void)
{
  return 2;
}

You can compile this:

$ gcc -g -O2 -Wall test.c -o test

And whoop we compiled a very useless program but when we run it we get:

$ ./test; echo $?
2

So we done something! But how does all of that work i mean how was this able to return a value to the shell. We could write this in assembly (i386/x86) and i can show you what happens here to return this value its call the c-decl ABI (http://en.wikipedia.org/wiki/X86_calling_conventions)

.code32
.global	main
main:
	mov $2, %eax
	ret

To compile this you simply:

$ as fib2.s -o fib2.o
$ gcc -o test fib2.o
$ ./test; echo $?
2

So what does this mean, well i am sure it looks some what clear whats going on, we declared a global symbol main and its code simply puts the value 2 into the register eax and returns. The ret opcode is a fairly useful one, it is responsible for restoring the instruction pointer for the program back to the caller. You can do all of this manually by messing with the instruction pointer yourself but thats for another day see (http://pdos.csail.mit.edu/6.893/2009/readings/i386/RET.htm)

In the next post i will write how we can call into libc functions such as printf for a hello world from assembly in i386. This will introduce the stack.

Career vs Life

Hey all reading this, its a fairly personal post and probably controversial for me to make.

About a year ago i was working for a great wee company WANdisco i had such a great set of people to work with, we were all so close and work was such a joy to go to. Exciting and vibrant when i was working there for bit over a year and i started to yearn for a more ‘low-level’ challenge such as a low-latency C/C++ application. I felt i had so much to learn in this area and its where i wanted my career to go towards.

So i joined where i am now (not going to write to avoid google picking it up) and i began to realise the pain of legacy systems programming and in my opinion i found it a challenge to some extent but it was never the code base i had issues with i mean with my experience in open source and GCC which is older and bigger than most legacy code-bases you will ever find. The issues i had were more high-level, we were never given freedom to be pro-active about maintaining code (fixing things before they become an issue). This lead to much time being very idle and frustrated at being quite literately un-challenged. And this was seen to be ‘busy’ with everyone embellishing their job and work to sound better to hopefully eventually gain some kind of promotion or some kind of management role.

And though this job i found such frustration i became very depressed to the point i though almost that Software Engineering isn’t a career path that i would want to pursue any longer….

But now its all fixed and for the first time in probably 2 years i feel settled, i mean i went to lunch and everything seemed brighter and better than i have felt in so long and i have been smiling all day. I never realised how depressed i had become, its so dangerous when i bottled things up so much that it would personify as physical illness such as bad colds and coughs on and off. I would bottle up so much because my Significant Other i really didn’t want for her to have to deal with my problems i love her so much and i just want to make her happy but my career was bringing us both down.

But you know what i finally have a new job where i will be starting in about a month (will reveal closer to the time). And you know what the people were so nice to me and challenged me and made me work in the interview i felt at peace again. Like i knew what i was doing again being proactive challenging myself and had the desire to push myself. And more importantly understood why i participate in open-source, at this job very very frequently you will be forced to deal with this question ‘sure, why would you do that? Were you paid? No? What…?’.

Its probably important to note, their reasons were i guess to some extent legitimate such as project approvals from such a large corporate entity and product stability. But regardless of this i would still argue that failure to fix and update is kind of like buying a house that really needs the plumbing and electrics re-done, sure you can live in it and use it but its not that pleasant and has a very high likely hood of costing you a lot in the long run.

Though this year i have talked to many many people on-line and off-line; strangers who become friends on a bus from different walks of life and different careers. And the universal notion we all have had is that, career will only take you so far because in the end it is we who have to live our lives and we can’t let circumstances get in the way of enjoying life.

My depression at this job was so great i believe if it went on for another year or 2 it probably was getting close to costing me my relationship with my Significant Other and maybe even my life. I probably shouldn’t go into the nitty details of what it was like here but if your in a similar position i would love to hear from you via email or if your local to Northern Ireland i would like to share a coffee with you.

Take care of your mental health i mean seriously the stigma it has in some countries still scares me such as Northern Ireland. Your likely to hear people equate Mental Health with the ‘luney bin’.

http://www.nidirect.gov.uk/mental-health

Global Warming – Investment Banking the issue

So when i was travelling to work this morning i couldn’t not read this article http://www.bbc.co.uk/news/science-environment-26810559, i have several things to say about this article in particular.

I personally find when articles like this come about its kind of overwhelming in how under-prepared our society is for the future but so it may seem.  I find it so depressing and disgusting that we simply believe we are all so un-prepared for the future when you consider how much man kind has already endured. And now all of a sudden people are worried were all going have to live off insects to survive.

The article basically hits on 2 main stress points:

  • Food shortages
  • Energy supply shortages

I am going to show you several things that might open your eyes and also discuss my experience from growing up in Northern Ireland and how the complacency of man has created these problems out of nothing.

TL;DR: i believe we are going to be ok, tackling food waste and regulating investment banking will stop making the already rich even more so and the normal farmer to be more fairly paid giving rise to more money into research and development into farming and also education in food waste and nutrition and food heritage. I also believe the energy problem is already worked out but there is no financial gains for the already wealthy until its needed. We will be ok and we will be prosperous as a race.


Food Shortages and Crop Yields

When you consider this most news people and scientist at this high high level really don’t even know the simplest of details, for instance i live on a farm and i see the effects of complacency stronger than you would dare to believe.

For instance the fields behind where i live are owned by our landlord, its a vast vast amount of land with which he even has more on the west of northern Ireland. All for growing crops wheat is a big issue at the moment, my girlfriend’s father was al most put out of business because of this.

Northern Ireland has been dealing with huge rainfall for the last year or so. This means dairy and meat farmers are forced to bring cattle in doors what does this mean. Usually in good weather you can leave your cattle out and feed on the grass on the fields and you rotate them around so they always have grass to feed on. So this is great the farmer has no need to buy artificial feeds made from wheat etc.

With the large down pours the demand for wheat has increased but the supply has been lower than previous years not drastically lower but lower more and more smaller farmers have been forced to buy from abroad etc. The demand has increased because in the rain farmers are forced to bring the animals inside into shelters and feed them a mixture of wheat feeds, straw and cut grass. But why cant you just leave them out in the rain the whole time. This is because if you leave cattle out in the fields you fields will be utterly RUINED and you will NOT be able to grow grass usable to feed your animals for probably best part of a year. So your will be forced to buy food and you cannot grow at least some grass.

So this is 1 aspect the effect it has on dairy farmers so ok this sounds bad with the more rain they cant feed animals of their own back with such low pay for milk which by the way is insane milk is cheaper than water which makes no sense at all.

So the people growing the wheat remember i said i live on a farm, with all the rain the fields are entirely water logged and haven’t produced anything in the last 4 months and are trying to be fixed. And i even had a chat with green grocers in my town and how most of the veg we all eat here is from Spain and Holland. Now i have mostly conjectures no hard truth to say there are many effects to consider to understand here.

  • Adequate drainage was never implemented for these fields ever
  • Poor poor maintenance of the fields
  • Poor maintenance of equipment
  • Not enough workers and low pay

So you can see there are quite a few reasons here for this situation the fields are in. This has so many effects and problems to be considered to fully appreciate the problem lets think first low pay which is the easiest for most skilled workers in the western society to understand.

Supply and Demand of Raw Produce

So when i see our fields behind out house when we first moved in there were several wheat fields they were brimming with produce and busy happy workers. But when it even vaguely came closer to winter everything shuts down. And there were equipment problems for them and not enough money available to warrant the amount of effort required to make a living. Never mind the completely un-prepared fields with no drainage etc which was complacent on the farmer. So when i had that conversation with the green grocers all they had to say on the matter is “you’ll never beat the dutch in growing”. So why is this statement true why cant we in nothern Ireland grow as well as the dutch is it to do with our climates probably something to do with it and the fact they do grow marijuana there which has had an awful lot of research and development into that probably can have an effect on normal growing by a far greater understanding. But why isn’t there more research into bio domes and more available funding for bio domes and micro climates etc for vegetable growing and farming a much more vibrant industry for a newer younger generation to be involved in.

I wouldn’t say the work is any more difficult, for me as a software engineer sometimes helping on my partners families farm is the first peace of mind i feel in a long time. The stress and anxiety of a corporate world is overwhelming sometimes. But this can then lead into the next issue BANKS and large Super Markets.


Banks and Investment Banking

I am not going to pretend i don’t go to Tesco or Sainsburies’, but do you ever look onto BBC news and see oh Tesco’s operating profits have dropped and Tesco credit cards and Tesco banks and Marks and Spencer bank accounts and ISA’s etc.

Well that the hell is this, since when where i buy my potatoes and sausages all of a sudden turn into a bank where i can get a loan and get my wages paid into. Well this is where it gets complicated and intentionally so which you might find. When you see these articles on these super markets dropping in profits or growing in profits; its never that more customers decided “i don’t link Tesco” all of a sudden and go elsewhere, generally the same customers are going to these shops.

But what’s happened that they are loosing or gaining money, when you spent money in these super markets they are using the profits to start putting it into investments abroad housing mortgage’s, stock exchanges, stocks on commodities or shares in companies. And gambling their profits to gain money.

For example the Co-Op bank has been on bbc news a lot recently but co-op is simply a small shop round the corner. Since when did it become a bank (doesn’t matter). What’s happening these smalls shops started out doing fairly well but over time they ended up using profits over time investing into anything and everything and if they put money into apple shares in 2002 then 2004 sure fuck me they are probably loaded. But this is why investment banking is so insane its just a gamble with money. The 2008 financial crisis similar but more complicated issue we as tax payers pay for their gambling debt and i say gambling debt because it is.

But wait how does this related to the issue of wheat supply though-out the world. Well when the wheat prices effect the dairy farmers which is a good example creameries are under so much pressure of super markets such as Tesco to drive price DOWN DOWN DOWN. So the effect hits on the dairy farmer so he is taking the hit here and believe you me this is a big hit small milk farming of around 100 cows is almost dead because of this. Even though this actually produces a fair amount of raw produces you can trade.

And what’s worse is this is almost un talked about and unheard of. To most people they might think 100 cows is a lot and it is. For example the Artificial insemination cost of breeding cattle, and different breeds different yields and effects of milk. Then health cost of cattle these days is going up and up veterinary bills and so on.

I mean me and my partner were with her father and we had to bring tepid water to a cow who was bleeding from the inside out after a bad calving where the calf at least lived, but the cow died even though we had the vet out and the they took blood from another cow to help and sown her up and everything. Me and my partner went back to work the next day and that night we were told the cow died. It was so sad i remember that day so well i remember where i was sitting and what we were doing.

I mean its one cow and the farmer bared the cost, but 1 cow gives so much to society. Even farmers these days by law have to pay the government to dispose of the animal when it dies and the farmer bears this cost also.

Beef and milk are the raw produce from a single cow. We can make yoghurt, cheese, milk, cream, butter milk, steak… so much. The costs in doing this are becoming so great for smaller farmers that small local producers are almost died out because of the big competition of big farms. So ok there are huge farms here and in England but are you going to bet you whole food supply on 1 guy to do so?

And the other issue is supermarkets are hiding the cost of the issues, when they drive down prices to creameries, creameries are forced to talk it because of such high competition of super market pressure they have to accept the best they can find.

The farmer is the most important man in the world and no one notices. I only choose the dairy and wheat farmer as examples because i have first hand experience with this. So when you think of massive monopolies on food such as Microsoft had on software for years this isn’t good for society and each person around us. Competition although difficult but choice is just so important because of our buy/sell and trading style economics it keeps vendors in check to be fair to the end customers because the buyer or consumer has a choice.

You can argue these super markets aren’t doing anything wrong by branching into investment banking but its created this gambling society throwing money out of our country if it fails and giving vast wealth to the lazy and already wealthy when it goes well.

People creating the commodities are completely forgotten about.

The other side of the coin is this type of investment banking into stock exchanges funding companies and tech companies has brought about so much research and development and vastly changed how we think and communicate with technology but its time investment banking gets tougher and tougher regulation. I don’t believe super markets should be allowed to do any kind of investment banking because your playing with local rural economies which are already weak because of a lower population and investment and consolidating wealth and prosperity into single cities such as London in the United Kingdoms case. I mean the same job here in Northern Ireland will offer me around £30k a year but in London they will offer in excess of £60-70k a year and the end result in often cases will be worse in London than the lower paid guy from here.

The consolidation of money is the weakness to our economy and high liquidity keeps everything in check and to have high liquidity you need more competition of smaller vendors and avoiding huge monopolies on anything. For example if someone person owned all the oil in the world and all the food in the world. What would the world look like? And if there are lots of people who own a little bit so we can trade fairly what would the world look like?

Then what about tacking food shortages, i really don’t believe in any way were all going to have to start eating insects even though something like 50% of the world does eat insects as part of a healthy diet. Whats happening is that no one has any life skills any-more in my experience, its only the last 2 years the cut of meat Brisket is becoming known in the United Kingdom the amount of food waste there must have been for such a long time. And un-fashionable vegetables such as a simple Turnip which can grow easily here any root vegetable even, there are even crops that like harsher weather the means for us to survive magically already exist but are forgotten about.

For food security the United Kingdom must respect its old food heritage of Nose to Tail eating and old traditional Game meats and offals its only recently i really started to take the notion of eating liver seriously and calves liver is very good even ox-heart is not organ’y at all its meant to be like a steak and when cut and prepared properly meant to be beautiful but i cant just go to down the street and get this. Fish and shell fish ,we live on an island and there is 1 part time fish mongers in Belfast Wednesday to Friday and on Friday and Saturday in the market.

So tackle the issue of Food waste then tell me were all going to have to eat insects and i might believe you, there is simply just so much we are over looking in food waste and exporting and not eating i refuse to believe that statement. I am not saying the issue is a simple one but education on food and food preparation should be mandatory in school. Not only this but the notion of food waste must be discussed and nutrition. For me i work full time i leave my house at 8am, and i am not home until 6pm going to my local green grocer and butchers is almost impossible.

Except if i come home early and work from home more. And i think the working life needs an adjustment the corporate world rewards compliance and mundane process over end results which simply makes no sense in a changing world the attitudes to the world need to change to have much more freedom as a human being.

its even getting to the stage in the United Kingdom that having a family a traditional nuclear family i think they call it is almost impossible now the pressures of money, housing, living costs are pushing people to live alone the cost of having a family too great. For me i think its going to be possible to find a job where i work from home full time so at least my partner might be able to resume some what of a career so she can live her life the way she wants and i will support her every way. Just the way she has supported me so strongly with my frustrations of working where i am.


When you then think about the energy supplies, you immediately think of fossil fuels, coal oil and gas and how we’ve been told for the last 30 years that its going to run out no one knows what were going to do and its all going to go to hell.

I really hate this statement you see it made all the time on the news and in school and its just scaring people. All you need to do is watch this video and you will feel security that the world isn’t going to be over in a long time.

I believe the human race will adapt like it always has and it will work and we will survive and we will be prosperous and happy. When articles like this come about people forget where we all came from and how we’ve dealt with stuff in the past.

Thanks for reading and: BE EXCELLENT TO EACH OTHER!