RedBrain @ $HOME : ~ $ » tutorial

Crules and Intermediate Representation

redbrain — Thu, 03 Dec 2009 02:47:00 +0000

If you follow my blog you may have noticed i have been quiet and thats because i’ve been working on http://crules.org Its the new homepage for my project Crules, its got some good content on it but nothing really on the language yet. Though today i spend a very long time on the HACKING article And i explain a lot on how Intermediate Representation’s work and 3-address code.

So lets just take an extract since i am lazy: So in compilers or interpreters how do we represent:

(( x + y ) -- ( ( x + y ) * ( x -- y ) ) ) + ( ( x + y ) * ( x -- y ) )

Lets see it in a diagram first then in IR code:

Expression

So what is this? Well early on when i was getting into compiler construction when doing semantic analysis you will need to find a way of representing these semantics for your language, and you will find generally that there are 3 things you need to know: DESTINATION OPERAND A and OPERAND B. This is what’s called 3-address code. This is what sets compilers apart and can make or break them! Its what gives implementations flexibility and the ability to ‘control the flow‘ of execution! Any good compiler book will give you more detailed discussion on this topic its my main interest in compilers since this is what gives an implementation its logic. So lets give it a syntax we can write down to illustrate how mine works: it just so happens a LISP ‘ish syntax can show this very well!

( ( IDENTIFIER ( TYPE ) ) => ( OPERAND A ) => ( OPERAND B ) );

Since the parser is precedence aware it will construct the tree as shown in a symbol language which looks like this if it was output ( outputting the IR like this is a future TODO ); It may look a little like GIMPLE but this IR I have as I will demonstrate more examples can encompass much much more.. ( this is a 3-address code by the way )

( ( NIL ( OP_ADD ) )
=> ( ( ( NIL ( OP_SUBTRACT ) )
=> ( ( ( NIL ( OP_ADD ) )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘x’ )
=> ( NIL )
); )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘y’ )
=> ( NIL )
); )
) ; )
=> ( ( ( NIL ( OP_MULTIPLY ) )
=> ( ( ( NIL ( OP_ADD ) )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘x’ )
=> ( NIL )
); )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘y’ )
=> ( NIL )
); )
); )
=> ( ( ( NIL ( OP_SUBTRACT ) )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘x’ )
=> ( NIL )
); )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘y’ )
=> ( NIL )
); )
); )
); )
); )
=> ( ( ( NIL ( OP_MULTIPLY ) )
=> ( ( ( NIL ( OP_ADD ) )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘x’ )
=> ( NIL )
); )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘y’ )
=> ( NIL )
); )
); )
=> ( ( ( NIL ( OP_SUBTRACT ) )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘x’ )
=> ( NIL )
); )
=> ( ( ( NIL ( SYMBOL_ACCESS ) )
=> ( ‘y’ )
=> ( NIL )
); )
); )
); )
);

busy!

Though one big note its is ok for me to leave the representation like this since in my interpreter i just evaluate this at run time but how does this affect things like code-generators like proper Compilers? Well its quite simple in a sane compiler like GCC ( although most compilers will follow this idiom i am sure) introduce another layer for IR called RTL ( Register Transfer Language ). The wikipedia article lacks and most compiler books overlook this section very much since, most books are written by the academics and generally stay away from the back-end of compilers since that is the actual hard part since a code-gen is very specific to the instruction set your targeting and register allocation algorithms are very difficult and is a huge research project. Its said in computer science the most difficult things to build software wise is a good JIT or any Code-Generator for that matter. Anyways back to the problem lets take a smaller expression for sleep’s sake the expression:

unsigned int retval = ( x * 2 ) / ( 7 + y );

Will generate something like:

t1 = x * 2;
t2 = 7 + y;
t3 = t1 / t2;
retval = t3;

Anyways you’ll notice in my the IR keywords such as SYMBOL_ACCESS or SYMBOL_ITEM or OP_ADD etc, these are what’s called OP_Codes and are generally represented by some symbolic hex values since its cheap to check against for the implementation to figure out what to do with symbol ‘x’; So ok thats a quick and dirty intro to Intermediate Representation’s!

I think this lolcat will sum up your feelings at the end of reading this blog post:

And i started getting out my old grunge cd’s which include my Silverchair collection and this is on of my favourite bands of all time and this is a pretty good song!

Update – Interpreter CRULES & Jamedo

redbrain — Mon, 27 Jul 2009 23:51:42 +0000

So hey, its been a little longer than i intended to update my blog! But i’ve been busy, so i have got rid of my personal work being on repositories all over the show. So I have made:

http://code.redbrain.co.uk

There will be public git access soon, but you can download the snapshot tarballs of the branches. And I have a wiki too there isn’t anything on there yet but it will give me room to stop putting long tutorial article’s as pages on my blog to proper wiki article’s:

http://wiki.redbrain.co.uk

So Compilers, Interpreters and programming languages are my interests… *yawn! So let me show you my interpreter pipeline for my interpreted language ‘CRULES’:

Crules

So I have to admit that’s the most impressive art I ever done (I know I’m lame ). Its my pipeline for my programming language, but I guess a lot of programming languages would follow this kind of model/architecture. So it got me thinking this is nearly the basics for any semi-serious interpreter or ‘a’ language so lets talk though it.

I am still not going to talk about the details of my language because its not ready yet. Other than I am doing fun things like, so far the only data-structure I let the developer have is lists, so if you do something like:

% foo := { 2,3,4,5,6,67,7}

#is a list and strings are lists!

% str := “bla”

#is the same as

% str := { ‘b’, ‘l’, ‘a’ }

It works a lot around the shell so you can also do:

% foo := `ls`

#it produces a delimited list of strings with the delimited as carriage return or new line etc.

Though note I am going to implement an api for arrays. But not yet, because lists are going to cause problems unless I implemented a table/array of accessors to the elements in the list so you can do fast access to the elements because it doesn’t scale no matter what anyone says when say you are doing image processing on a 1024×1024 list… accessing the elements isn’t going to scale.

Anyway so lets get back to looking into the pipeline of it: So first a user creates a program lets look at the helloworld.crl:

rule main()

{

echo “Hello World!”

foo := 5 + 2 – 7

}

all := main;

So first my interpreter will start lexing this using LEX, into tokens like: rule, main ‘(‘ ‘)’…. and then the parser starts applying the grammar implemented using YACC. So I start producing symbols to build a symbol table the data-structure looks like this its basicly a more high-level 3 address code:

struct symtab {
const char *identifier;
uint8_t symType;
uint8_t op_a_t;
uint8_t op_b_t;
union {
_SYM_LIST_ *list;
float flt;
int32_t integer;
char ch;
//sym link to function call
struct symtab *syms;
} op_a;
union {
_SYM_LIST_ *list;
float flt;
int32_t integer;
//params to a func_call
struct symstack *syms;
struct symtab *sy;
} op_b;
struct symtab *next;
} __attribute__((aligned));

So first it creates the rule definition! For now just think as rules as entry points to a program so you can add more identifiers to that ‘all’ list not just main. It has a deeper meaning but I don’t want to talk or show this until I have the interpreter and language definitions more concrete!

So the the rule ‘main’ creates a new symbol like this:

//the symbol takes the identifer of the rule identifier

symbol->identifer = ‘main’

//its a rule symbol

symbol->symType= _SYM _RULE _

symbol union_a is the link to the first symbol in that block of statements and union_b is NULL because I don’t need it until I have rule dependencies it needs to list.

symbol->next= NULL, because it is just a definition

Then each expression like ‘ echo “Hello World” ‘, creates a symbol and its next = the next expression like the ‘ foo := 5 + 2 – 7 ‘. This creates an interesting one, because this can’t be reduced to a 3-address code until you split it up, for now I just assume the developer wants it to run as 5+2=7 then 7-7=0; But I am going to add precedence later. So you can be more expressive to have nested expressions in brackets to be executed first etc..

But yeah this expression is split up into 3 symbols for now, its an over all variable Assignment but it depends an expression of 5+2 as a temporary symbol then this temporary symbol – 7 then foo = this new temporary symbol and I can push these dead symbols to the garbage collector.

When a symbol is allocated, it is allocated memory of a preallocated memory heap of default 32MB, it can be extended or shortened if needs be, but it allows me to control and see how much memory is being occupied at anyone time in the runtime of the interpreter. Any symbols to be kept such as rule definitions, function definitions or variables are kept on one massive stack. So ok, that’s a problem if you need to search for them, the answer I found was having 3 other tables that link to the symbols in the stack. When any link is broke it becomes garbage, so when I have a new variable re-assignment I can break that link and remove the entry in the variable table it still exists in the symbol stack but the Garbage collector is always called on EXIT and the exit of any block statements like loops functions etc, so when you pop your way out of the stack you can have local variables instead of everything global and you have conflicts which will cause a lot of problems. Also the garbage collector is implemented as a pthread so you have have to pause for several seconds waiting for it to free the garbage. I haven’t done this yet because its going to be very awkward to keep the concurrency, the pthread mutex api should handle a lot for me but its still going to be complicated. I have thought I might like to try implementing a lot of different internal using OPEN/MP to make it even speedier, but that won’t be for a while.

And then I have been playing with LLVM to create a JIT execution system, you can see my posts on ‘Managing the JIT’ on comp.compilers, I have posted there quite a few times before, its a great list of basically all programming language, compiler and interpreter experts in the world!

So I have a variable table that links to the variables in the symbol stack and the same for the rules and functions. This speed up searching for specific identifiers.

So that’s the basics of parsing into symbols I guess, it pains me to say but many people thing a programming language is mainly a parser is so wrong the parser just lets you build your symbols the execution and management of these symbols is much, much more complicated. That’s not to say implementing a simple programming language is difficult, the problem I see a lot is people are scared of doing it because the vocabulary is huge! I mean if you start working at an interpreter from first principle’s I’ll guarantee if you have half a brain you’ll come to a similar architecture as this.

I found it difficult at the start when I knew anything in a programming language can always be brought to 3 things and I got really confused when people talked about symbol tables, 3 address code, IR (intermediate representation) and parse trees or syntax tree’s. I think a lot of developers have their own prefered meanings to them, but it is mainly a barrier of academics trying to make the subject more complicated than needs be, although when you go at this from first principle’s you’ll will soon learn .

So thats the main focus of my personal work but I am now also soon to be an approved GNU/gcc and GNU/automake developer! Can’t wait, got my FSF paper work signed and sent back it may arrive in Boston soon I guess but yeah I am excited. I am updating the automake documentation to show the best way to handle multiple LEX and YACC in one program. Its something only one of two people do but its a problem none the less and to make it portable is the actual problem. But I have the solutions to it now .

Got your attention

Thing is I would really love it if I had an AIX/FreeBSD/Solaris machine to do more testing on because now that I am doing work on GCC it would be so helpful to have more machines to do testing. But i’m ok for now, so far in GCC I am doing regression testing to find a regression I can tackle, its extremely difficult some are defunct on the bugzilla some are ancient and not perused. And the rest is very active but for MIPS or SPARC, and I don’t have a scratchbox setup to even test that yet. Would just be great to get my hands on even an old ibook for a power-pc cpu or an old IBM p-server. If anyone knows of any available I would be really grateful!

Ok this has become a very long post so I’ll shut up! . See you guys from Beflast LUG on Wednesday! One last thing i want to link you to 5 of my favourite bands and they are all off Jamendo:

Tutorials and Pages on the go!

redbrain — Fri, 06 Mar 2009 10:57:34 +0000

So i have been spending the last few days and nights trying to make an awesome tutorial on writing a c/c++ application with autoconf and automake + libtool, creating a library out of it with relevant headers and swig python bindings and then all the way to packaging to a PPA on launchpad which will be debs no rpms here no idea how to make those!

So as you can see there is alot there! So good wordpress is saving my stuff. When i am writing my tutorials now and articles i am starting to write them in OpenOffice first to make it much more professional of me. Posts are still quite spontaneous on me

But i think for my pages etc.. it all has to take its time and be done right. On a plus things are going good now no more ranting from me anymores

So yeah i am going to post updates of my OS work soon i am going to take it of gitorious probably because its too awkward to keep update because i spend so much time on launchpad now because of drizzle trying to merge my changes with the trunk there are a few problems but its because my tree was pretty old. And there was a good few autoconf updates since then for the project like more depandancies etc. So yeah i am working on it somtimes but drizzle is a little more important to me because it looks better on me i guess. But stay tunned and i will keep you up todate and more tutorials etc!

not-real

So next moving on here is something i am looking forward to: http://www.imdb.com/title/tt0438488/

There is a new trailer it looks sweet! Anyways follow me on identi.ca i keep it pretty uptodate

Autoconf for dummies :)

redbrain — Tue, 20 Jan 2009 17:24:11 +0000

I have been making a projects up wth autoconf recently and i intend to clean up all my projects here if its java to be with ant build file! And then on with c projects with autoconf etc..

Anyways yeah read it because it makes it so easy this tutorial!

Whoops forgot to link: http://www.galassi.org/mark//mydocs/autoconf_tutorial_toc.html

ciao for now!

String Matching algorithms?! This is my one :)

redbrain — Fri, 16 Jan 2009 10:32:44 +0000

Hoping wordpress can make this code look half decent! Its in C:
//it is like the Boyer Moore algorithm but i dont think its fully optimized

/**
 * @eturn returns the postition of the comparison
 * returns -1 if it cannot be found else i
 * @param str is the string you want to search through!
 * @param comp is the string you want to search for!
 **/
static size_t str_getPlace( char* str, char* comp ) {

  size_t i, counter, complen=strlen(comp), skip_counter=0;
  i=complen-1;

  for (;;) {
    //printf("no: %i - orig: %c - %c\n",i,str[i],comp[complen-1]);
    if( comp[complen-1] == str[i] ) {
      //reset counter = 1 as we found one char already
      counter=1;
      if(complen > 1) {
	int j;
	for(j=complen-2; j>-1; --j){
	  --i;
	  if( comp[j] == str[i] )
	    counter++;
	  else{
	    //skipping length + the counter of already checked
	    i+=complen+counter;
	    break;
	  }
	}
      }
      //printf("Counters: comp length: %i -counter %i\n",strlen(comp),counter);
      if ( counter == strlen(comp) )
	return i;
    } else {
      //if i is bigger than length of string or the remaining
      //length is less than the length of the search
      if (i >= strlen(str) || (strlen(str) -i) <= complen ){
	skip_counter++;
	//if skipped more than the length of the search as there is no match!
	if( skip_counter >= complen )
	  break;
	else
	  i=complen+skip_counter-1;
      } else
	i+=complen;
    }
  }
  return -1;
}