RedBrain @ $HOME : ~ $ » BLUG

Update – Interpreter CRULES & Jamedo

redbrain — Mon, 27 Jul 2009 23:51:42 +0000

So hey, its been a little longer than i intended to update my blog! But i’ve been busy, so i have got rid of my personal work being on repositories all over the show. So I have made:

http://code.redbrain.co.uk

There will be public git access soon, but you can download the snapshot tarballs of the branches. And I have a wiki too there isn’t anything on there yet but it will give me room to stop putting long tutorial article’s as pages on my blog to proper wiki article’s:

http://wiki.redbrain.co.uk

So Compilers, Interpreters and programming languages are my interests… *yawn! So let me show you my interpreter pipeline for my interpreted language ‘CRULES’:

Crules

So I have to admit that’s the most impressive art I ever done (I know I’m lame ). Its my pipeline for my programming language, but I guess a lot of programming languages would follow this kind of model/architecture. So it got me thinking this is nearly the basics for any semi-serious interpreter or ‘a’ language so lets talk though it.

I am still not going to talk about the details of my language because its not ready yet. Other than I am doing fun things like, so far the only data-structure I let the developer have is lists, so if you do something like:

% foo := { 2,3,4,5,6,67,7}

#is a list and strings are lists!

% str := “bla”

#is the same as

% str := { ‘b’, ‘l’, ‘a’ }

It works a lot around the shell so you can also do:

% foo := `ls`

#it produces a delimited list of strings with the delimited as carriage return or new line etc.

Though note I am going to implement an api for arrays. But not yet, because lists are going to cause problems unless I implemented a table/array of accessors to the elements in the list so you can do fast access to the elements because it doesn’t scale no matter what anyone says when say you are doing image processing on a 1024×1024 list… accessing the elements isn’t going to scale.

Anyway so lets get back to looking into the pipeline of it: So first a user creates a program lets look at the helloworld.crl:

rule main()

{

echo “Hello World!”

foo := 5 + 2 – 7

}

all := main;

So first my interpreter will start lexing this using LEX, into tokens like: rule, main ‘(‘ ‘)’…. and then the parser starts applying the grammar implemented using YACC. So I start producing symbols to build a symbol table the data-structure looks like this its basicly a more high-level 3 address code:

struct symtab {
const char *identifier;
uint8_t symType;
uint8_t op_a_t;
uint8_t op_b_t;
union {
_SYM_LIST_ *list;
float flt;
int32_t integer;
char ch;
//sym link to function call
struct symtab *syms;
} op_a;
union {
_SYM_LIST_ *list;
float flt;
int32_t integer;
//params to a func_call
struct symstack *syms;
struct symtab *sy;
} op_b;
struct symtab *next;
} __attribute__((aligned));

So first it creates the rule definition! For now just think as rules as entry points to a program so you can add more identifiers to that ‘all’ list not just main. It has a deeper meaning but I don’t want to talk or show this until I have the interpreter and language definitions more concrete!

So the the rule ‘main’ creates a new symbol like this:

//the symbol takes the identifer of the rule identifier

symbol->identifer = ‘main’

//its a rule symbol

symbol->symType= _SYM _RULE _

symbol union_a is the link to the first symbol in that block of statements and union_b is NULL because I don’t need it until I have rule dependencies it needs to list.

symbol->next= NULL, because it is just a definition

Then each expression like ‘ echo “Hello World” ‘, creates a symbol and its next = the next expression like the ‘ foo := 5 + 2 – 7 ‘. This creates an interesting one, because this can’t be reduced to a 3-address code until you split it up, for now I just assume the developer wants it to run as 5+2=7 then 7-7=0; But I am going to add precedence later. So you can be more expressive to have nested expressions in brackets to be executed first etc..

But yeah this expression is split up into 3 symbols for now, its an over all variable Assignment but it depends an expression of 5+2 as a temporary symbol then this temporary symbol – 7 then foo = this new temporary symbol and I can push these dead symbols to the garbage collector.

When a symbol is allocated, it is allocated memory of a preallocated memory heap of default 32MB, it can be extended or shortened if needs be, but it allows me to control and see how much memory is being occupied at anyone time in the runtime of the interpreter. Any symbols to be kept such as rule definitions, function definitions or variables are kept on one massive stack. So ok, that’s a problem if you need to search for them, the answer I found was having 3 other tables that link to the symbols in the stack. When any link is broke it becomes garbage, so when I have a new variable re-assignment I can break that link and remove the entry in the variable table it still exists in the symbol stack but the Garbage collector is always called on EXIT and the exit of any block statements like loops functions etc, so when you pop your way out of the stack you can have local variables instead of everything global and you have conflicts which will cause a lot of problems. Also the garbage collector is implemented as a pthread so you have have to pause for several seconds waiting for it to free the garbage. I haven’t done this yet because its going to be very awkward to keep the concurrency, the pthread mutex api should handle a lot for me but its still going to be complicated. I have thought I might like to try implementing a lot of different internal using OPEN/MP to make it even speedier, but that won’t be for a while.

And then I have been playing with LLVM to create a JIT execution system, you can see my posts on ‘Managing the JIT’ on comp.compilers, I have posted there quite a few times before, its a great list of basically all programming language, compiler and interpreter experts in the world!

So I have a variable table that links to the variables in the symbol stack and the same for the rules and functions. This speed up searching for specific identifiers.

So that’s the basics of parsing into symbols I guess, it pains me to say but many people thing a programming language is mainly a parser is so wrong the parser just lets you build your symbols the execution and management of these symbols is much, much more complicated. That’s not to say implementing a simple programming language is difficult, the problem I see a lot is people are scared of doing it because the vocabulary is huge! I mean if you start working at an interpreter from first principle’s I’ll guarantee if you have half a brain you’ll come to a similar architecture as this.

I found it difficult at the start when I knew anything in a programming language can always be brought to 3 things and I got really confused when people talked about symbol tables, 3 address code, IR (intermediate representation) and parse trees or syntax tree’s. I think a lot of developers have their own prefered meanings to them, but it is mainly a barrier of academics trying to make the subject more complicated than needs be, although when you go at this from first principle’s you’ll will soon learn .

So thats the main focus of my personal work but I am now also soon to be an approved GNU/gcc and GNU/automake developer! Can’t wait, got my FSF paper work signed and sent back it may arrive in Boston soon I guess but yeah I am excited. I am updating the automake documentation to show the best way to handle multiple LEX and YACC in one program. Its something only one of two people do but its a problem none the less and to make it portable is the actual problem. But I have the solutions to it now .

Got your attention

Thing is I would really love it if I had an AIX/FreeBSD/Solaris machine to do more testing on because now that I am doing work on GCC it would be so helpful to have more machines to do testing. But i’m ok for now, so far in GCC I am doing regression testing to find a regression I can tackle, its extremely difficult some are defunct on the bugzilla some are ancient and not perused. And the rest is very active but for MIPS or SPARC, and I don’t have a scratchbox setup to even test that yet. Would just be great to get my hands on even an old ibook for a power-pc cpu or an old IBM p-server. If anyone knows of any available I would be really grateful!

Ok this has become a very long post so I’ll shut up! . See you guys from Beflast LUG on Wednesday! One last thing i want to link you to 5 of my favourite bands and they are all off Jamendo:

Identi.ca -> Curl -> Update

redbrain — Mon, 25 May 2009 23:34:14 +0000

So yeah, i told you guys in my last post i would be doing a tutorial on compiler work, i am still working at this, tbh there is so much writing i am thinking i might make like a sub-domain for it like a latex-html type thing, i have been writing a long .odt in openOffice, but i think if i want to put it on this blog it will be far too big. But anyways well see i updated some of my stuff here on the blog feeling much happier i think i have found a theme i can settle on!

Oh and i added the link for my hello-world gcc-frontend

$ git clone git://github.com/redbrain/gcc-sbsh.git

$ ./configure

$ make (..sleep for 2 hours…)

//what your interested in is gcc/sbsh/ * all the files in there is all the stuff you have to do to make a hello world front-end you need all those definitions etc.. i think i have broke my Make-lang.in but i’ll fix it asap. If your on a mac i posted the fix for mpfr on the gcc-help mailing list.

To compile on a mac:

$ wget -c ftp://ftp.gmplib.org/pub/gmp-4.3.1/gmp-4.3.1.tar.gz

$ tar xvf gmp-4.3.1.tar.gz

$ ./configure

$ make

$ sudo make install

$ sudo port install mpfr

$ cd gcc-sbsh

$ ./configure –prefix= –with-mpfr-lib=/opt/local/lib –with-mpfr-include=/opt/local/include

Now it should configure properly! http://www.nabble.com/Configuration-problem:-not-recognizing-MPFR-td23556792.html. You may find my on gcc-help pretty often now and automake bison and autoconf. I use these apps alot now and the mailing lists are really intereasting. Linux-kernel list sucks its far far too fast to follow unless you a full time-developer drizzle i dont have enough time for either KVM i think they are all over the place

And here is the git-repos of my work for the hello-world interpreter its still a work in progress but i think its ok for one evenings work

$ git clone git://github.com/redbrain/simple-programming-language.git

$ apt-get install build-essential autoconf automake libtool

$ sh autorun.sh

$ ./configure

$ make

$ ./src/simple –help

$ ./src/simple tests/helloworld.sim

Oh and i have done some massive updates to the Belfast LUG wiki in general: http://www.belfastlinux.org/wiki/Main_Page

http://www.belfastlinux.org/wiki/2009-04-06_mobile-monday

http://www.belfastlinux.org/wiki/Meetings

I will aim to keep the Meetings page updated with my email after the meeting with what happened at the Meet like in: http://www.belfastlinux.org/wiki/2009-03-25_pubbage. I will maby start a page in the future about going to LugRadio live 2009 but thats a good few months off yet..

So yeah This last week and a few days i have spent my evenings doing some writing on compilers etc… for the tutorial and playing with haskell.. I am trying to learn the curl api to make an identi.ca client written in haskell and it cant be hard because the code to do it in C is well tiny look i knocked this up in a few mins:

#include "stdio.h"
#include "curl/curl.h"
#define IDENTICA "http://identi.ca/api/statuses/update.xml"
int main( int argc, char *argv[] )
{
CURL *curl;
CURLcode res;
curl = curl_easy_init();
if ( curl ) {
char *data="status=hello world identi.ca libcurl – c";
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, data);
//username:password… male sure to keep the ‘:’
curl_easy_setopt(curl, CURLOPT_USERPWD, "username:password");
curl_easy_setopt(curl, CURLOPT_URL, IDENTICA);
res = curl_easy_perform(curl);
printf("result -> %i\n", res);
/* always cleanup */
curl_easy_cleanup(curl);
}
return 0;
}

You can compile with:

$ gcc -o identica main.c `curl-config –libs –cflags`

So as you can see posting to identi.ca or twitter is a simple task. But haskell… well it isnt simple I do love haskell in some ways its very much the way you think of things if your doing a very long definition in Pure Math or a proof in Applied Math. Yeah i do a Math degree… i hate it a i love it just i prefer doing programming mostly.

Anyways the problem i have with the language is its becoming EXTREMLY messy in my opinion, instead of people concentrating on making haskell useful to actually make apps and wrappers to useful libraries they seem to keep adding features to the language. Like the way things work in it take a long time to get your head around but to manipulate data is why i LOVE it. So its like look at this tutorial it just explains why i love haskell http://www.haskell.org/~pairwise/intro/section1.html#part4

lolcat

Thats just an awesome idea lazy evaluation, i am getting to grips with it but i think it would be better if i get a book on it rather than reading a sticking bits of what i see around the interenet together.Its just its all very well being able to make a list and do silly evaluations on them etc.. But you need to have some good networking apis or some decent file-read-write apis there are some but its just very centric around read or getch type functions. Then GHC dam its a huge and the main implemenation of haskell, but its sooo bloated. I was toying with my own idea to make a gcc-haskell front-end but i don’t know if i like the language enough to do it, i would probably prefer to make my own version of it. But then it would take a long time to persude people to move over. etc…

Anyways its nearly 1AM and i have work in the morning Remember to post comments guys!