So hey, its been a little longer than i intended to update my blog! But i’ve been busy, so i have got rid of my personal work being on repositories all over the show. So I have made:

There will be public git access soon, but you can download the [...]]]>

So hey, its been a little longer than i intended to update my blog! But i’ve been busy, so i have got rid of my personal work being on repositories all over the show. So I have made:

• http://code.redbrain.co.uk

There will be public git access soon, but you can download the snapshot tarballs of the branches. And I have a wiki too there isn’t anything on there yet but it will give me room to stop putting long tutorial article’s as pages on my blog to proper wiki article’s:

• http://wiki.redbrain.co.uk

So Compilers, Interpreters and programming languages are my interests… *yawn! So let me show you my interpreter pipeline for my interpreted language ‘CRULES’:

Crules

So I have to admit that’s the most impressive art I ever done (I know I’m lame ). Its my pipeline for my programming language, but I guess a lot of programming languages would follow this kind of model/architecture. So it got me thinking this is nearly the basics for any semi-serious interpreter or ‘a’ language so lets talk though it.

I am still not going to talk about the details of my language because its not ready yet. Other than I am doing fun things like, so far the only data-structure I let the developer have is lists, so if you do something like:

% foo := { 2,3,4,5,6,67,7}

#is a list and strings are lists!

% str := “bla”

#is the same as

% str := { ‘b’, ‘l’, ‘a’ }

It works a lot around the shell so you can also do:

% foo := `ls`

#it produces a delimited list of strings with the delimited as carriage return or new line etc.

Though note I am going to implement an api for arrays. But not yet, because lists are going to cause problems unless I implemented a table/array of accessors to the elements in the list so you can do fast access to the elements because it doesn’t scale no matter what anyone says when say you are doing image processing on a 1024×1024 list… accessing the elements isn’t going to scale.

Anyway so lets get back to looking into the pipeline of it: So first a user creates a program lets look at the helloworld.crl:

rule main()

{

echo “Hello World!”

foo := 5 + 2 – 7

}

all := main;

So first my interpreter will start lexing this using LEX, into tokens like: rule, main ‘(‘ ‘)’…. and then the parser starts applying the grammar implemented using YACC. So I start producing symbols to build a symbol table the data-structure looks like this its basicly a more high-level 3 address code:

So first it creates the rule definition! For now just think as rules as entry points to a program so you can add more identifiers to that ‘all’ list not just main. It has a deeper meaning but I don’t want to talk or show this until I have the interpreter and language definitions more concrete!

So the the rule ‘main’ creates a new symbol like this:

//the symbol takes the identifer of the rule identifier

symbol->identifer = ‘main’

//its a rule symbol

symbol->symType= _SYM _RULE _

symbol union_a is the link to the first symbol in that block of statements and union_b is NULL because I don’t need it until I have rule dependencies it needs to list.

symbol->next= NULL, because it is just a definition

Then each expression like ‘ echo “Hello World” ‘, creates a symbol and its next = the next expression like the ‘ foo := 5 + 2 – 7 ‘. This creates an interesting one, because this can’t be reduced to a 3-address code until you split it up, for now I just assume the developer wants it to run as 5+2=7 then 7-7=0; But I am going to add precedence later. So you can be more expressive to have nested expressions in brackets to be executed first etc..

But yeah this expression is split up into 3 symbols for now, its an over all variable Assignment but it depends an expression of 5+2 as a temporary symbol then this temporary symbol – 7 then foo = this new temporary symbol and I can push these dead symbols to the garbage collector.

When a symbol is allocated, it is allocated memory of a preallocated memory heap of default 32MB, it can be extended or shortened if needs be, but it allows me to control and see how much memory is being occupied at anyone time in the runtime of the interpreter. Any symbols to be kept such as rule definitions, function definitions or variables are kept on one massive stack. So ok, that’s a problem if you need to search for them, the answer I found was having 3 other tables that link to the symbols in the stack. When any link is broke it becomes garbage, so when I have a new variable re-assignment I can break that link and remove the entry in the variable table it still exists in the symbol stack but the Garbage collector is always called on EXIT and the exit of any block statements like loops functions etc, so when you pop your way out of the stack you can have local variables instead of everything global and you have conflicts which will cause a lot of problems. Also the garbage collector is implemented as a pthread so you have have to pause for several seconds waiting for it to free the garbage. I haven’t done this yet because its going to be very awkward to keep the concurrency, the pthread mutex api should handle a lot for me but its still going to be complicated. I have thought I might like to try implementing a lot of different internal using OPEN/MP to make it even speedier, but that won’t be for a while.

And then I have been playing with LLVM to create a JIT execution system, you can see my posts on ‘Managing the JIT’ on comp.compilers, I have posted there quite a few times before, its a great list of basically all programming language, compiler and interpreter experts in the world!

So I have a variable table that links to the variables in the symbol stack and the same for the rules and functions. This speed up searching for specific identifiers.

So that’s the basics of parsing into symbols I guess, it pains me to say but many people thing a programming language is mainly a parser is so wrong the parser just lets you build your symbols the execution and management of these symbols is much, much more complicated. That’s not to say implementing a simple programming language is difficult, the problem I see a lot is people are scared of doing it because the vocabulary is huge! I mean if you start working at an interpreter from first principle’s I’ll guarantee if you have half a brain you’ll come to a similar architecture as this.

I found it difficult at the start when I knew anything in a programming language can always be brought to 3 things and I got really confused when people talked about symbol tables, 3 address code, IR (intermediate representation) and parse trees or syntax tree’s. I think a lot of developers have their own prefered meanings to them, but it is mainly a barrier of academics trying to make the subject more complicated than needs be, although when you go at this from first principle’s you’ll will soon learn .

So thats the main focus of my personal work but I am now also soon to be an approved GNU/gcc and GNU/automake developer! Can’t wait, got my FSF paper work signed and sent back it may arrive in Boston soon I guess but yeah I am excited. I am updating the automake documentation to show the best way to handle multiple LEX and YACC in one program. Its something only one of two people do but its a problem none the less and to make it portable is the actual problem. But I have the solutions to it now .

Got your attention

Thing is I would really love it if I had an AIX/FreeBSD/Solaris machine to do more testing on because now that I am doing work on GCC it would be so helpful to have more machines to do testing. But i’m ok for now, so far in GCC I am doing regression testing to find a regression I can tackle, its extremely difficult some are defunct on the bugzilla some are ancient and not perused. And the rest is very active but for MIPS or SPARC, and I don’t have a scratchbox setup to even test that yet. Would just be great to get my hands on even an old ibook for a power-pc cpu or an old IBM p-server. If anyone knows of any available I would be really grateful!

Ok this has become a very long post so I’ll shut up! . See you guys from Beflast LUG on Wednesday! One last thing i want to link you to 5 of my favourite bands and they are all off Jamendo:

• http://www.jamendo.com/en/album/26656
• http://www.jamendo.com/en/album/34523
• http://www.jamendo.com/en/album/42122
• http://www.jamendo.com/en/album/34714
• http://www.jamendo.com/en/album/3661

I decided for a VPS off: http://www.vpsville.ca/index

I got the CITY plan Its very powerful because its running in [...]]]> So its been a while since i have updated, while i was offline i guess, i have been moving to a new server! I have had a few teething problems but i think its all going fine now!

I decided for a VPS off: http://www.vpsville.ca/index

I got the CITY plan Its very powerful because its running in some kind of container VPS, i get to see the full ‘$ cat /proc/cpuinfo’. And it 8 core’s of the new Intel core i7! But i only have 256mb ram which was the teething problem so i turned off innodb in mysql and limited php memory usage to help compensate. It was nice Debian Server has all the mysql.conf and php.conf files all documented very well so you generally have an idea what your changing.

This is a very useful link for mysql tunning: http://howtoforge.com/tuning-mysql-performance-with-mysqltuner

——– General Statistics ————————————————–
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.0.75-0ubuntu10.2
[OK] Operating on 32-bit architecture with less than 2GB RAM

——– Storage Engine Statistics ——————————————-
[--] Status: +Archive -BDB -Federated -InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 940K (Tables: 32)
[--] Data in MEMORY tables: 0B (Tables: 1)
[!!] Total fragmented tables: 2

——– Performance Metrics ————————————————-
[--] Up for: 2h 45m 48s (6K q [0.612 qps], 214 conn, TX: 11M, RX: 1M)
[--] Reads / Writes: 74% / 26%
[--] Total buffers: 58.0M global + 2.6M per thread (100 max threads)
[!!] Maximum possible memory usage: 320.5M (125% of installed RAM)
[OK] Slow queries: 0% (0/6K)
[OK] Highest usage of available connections: 4% (4/100)
[OK] Key buffer size / total MyISAM indexes: 16.0M/331.0K
[OK] Key buffer hit rate: 99.7% (29K cached / 87 reads)
[OK] Query cache efficiency: 58.5% (2K cached / 4K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 291 sorts)
[!!] Temporary tables created on disk: 30% (103 on disk / 339 total)
[OK] Thread cache hit rate: 98% (4 created / 214 connections)
[OK] Table cache hit rate: 89% (50 open / 56 opened)
[OK] Open file limit used: 9% (98/1K)
[OK] Table locks acquired immediately: 100% (3K immediate / 3K locks)

——– Recommendations —————————————————–
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours – recommendations may be inaccurate
Reduce your overall MySQL memory footprint for system stability
Enable the slow query log to troubleshoot bad queries
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Variables to adjust:
*** MySQL’s maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
tmp_table_size (> 32M)
max_heap_table_size (> 16M)

That’s the kind of output i got in my first go so i start going though each of these recommendations: This link shows you each of the different variables you can change: http://dev.mysql.com/doc/refman/5.1/en/innodb-configuration.html, I think you can change a lot of Apache  memory variables but i haven’t looked into this yet!

But i also decided to make a swap file, this is very easy in Linux if your system is all running directly off a rootfs you can:

$ dd if=/dev/zero of=/myswapfile bs=1024 count=1048576

#this should make you a 1G file which we can use for a Swap file!

$ mkswap /myswapfile

$ swapon /myswapfile

#this should work but in my VPS i keep getting ‘swapon: swapfile: Operation not permitted’

#you can also add this to your fstab like so:

$ echo “/myswapfile       none    swap    sw      0       0″ >> /etc/fstab

So yeah that’s quite useful to know! I have to say i have upgraded my wordpress to 2.8 and very unhappy to be honest, there are a lot of little bits and pieces annoying me 2.7.1 was a very good release! But this 2.8 is very memory hungry, but from what i read is that they have introduced a new API for widgets, plugins and themes. So i guess there will be problems for a while!

Anyways what i am hosting on my VPS is 4 sites:

http://redbrain.co.uk

http://brendanmullan.com

http://team-qcg.com

http://animeequation.com

They are the websites so i run: mySQL,   Exim mail server(because noodles recommended it to me), Apache (might test light httpd), PHP plus various php plugins, git (using those python-tools): http://scie.nti.st/2007/11/14/hosting-git-repositories-the-easy-and-secure-way

I have to say anyone who works with me will know i hate SVN and CVS. Mainly because they are just retarded and your locked in too much and if the server goes down, then what the hell are you meant to do. My main reason for sticking with Linux was the fact now your no longer locked into software. A lot of people i know in the windows world use windows but use ‘cracked’ copies of it, as-well as photoshop, games, office etc… all downloaded etc. Using free software alternatives means your no longer reliant on these kinds of software and you know whats available and you learn alot more about your computer and why!

So anyways i thought i would post a screenshot of my current desktop i think it looks pretty nice tbh! I was using Xmonad for ages there, its a great window manager written in haskell its great because i run a server on my work desktop! And then i still need a way to run firefox, email and emacs. So this was very useful, and its very lightweight and usable. Other tiling window managers are light but very complicated!

my laptop

So i am also giving Haskell another go, i got the book ‘Real World Haskell’, its actually pretty good there is a helluva lot of learn, but its going to take a while before i’ll be seriously able to say i can write a mail client in haskell rofl! I would seriously like to make a GHS right now a gcc implementation of Haskell, because i have started the gcc-front-end documentation with a friend.

http://gcc.gnu.org/wiki/FrontEnd/skeleton

Problem is were kind of stuck right now, i posted most of it all so far and put up the code, etc but there have been quite a few changes in the latest gcc-4.5-snapshots so trying to keep up with them. But the real problem is i posted the code etc, but posting source code on a wiki page… its not that nice! So this is another point where the gcc-developers on oftc.net have said should really get my FSF-copyright assignment document stuff done, because i have my automake work and now gcc, and they would like to see official documentation in the GCC-manual in the next releases i think this would really help GCC.

If you working in industry like me, you’ll find the only real thing keeping compilers/interpreters and programming languages alive in large business driven companies is the idea of parallelism, this is quite old right now but its only really getting together now! I also have a book on OPEN/MP and know some of the basics of it in C/C++. Its very impressive the implementation in GCC and the Glibc. There is only a little theory to learn about OPENMP because actualy coding with it, is very simple! But you wont understand the code until you learn a little about shared memory systems and threading on this! Because OPENMP at the end of the day is starting threads for faster execution behind the scene. But it does all the work for accessing and manipulating memory without breaking your code for you! So in that sense its awesome

So yeah i also joined http://wakoopa.com/redbrain Its quite an interesting idea! And also pointing out that: http://outlawpla.net/ This is a Linux outlaws Planet, it should be cools i am hoping to get my blog on it as Dan and Fab pointed out in the show yesterday!

Anyways spent most of my time reading tbh recently so i havent been coding much, re-writing my OS from-scratch again i am starting to understand the i386 kind of now, and i want to structure my kernel correctly and make sure its as similar to just coding with libc as much as possible so i dont implement a stupidly named API of mad functions that only i understand! But i have done bits and pieces of cleaning my simple-programming language. Need to finish that tbh, there isn’t much more work for me to do to demonstrate the basics for it all. But been doing reading and writing takes alot of time when i work around my Job which takes up most of my time and i haven’t had a chance to get a break to do serious work at work, been documenating and transferring knowledge of work systems on Unix to new guys which is taking some time as they dont have a background in Linux or UNIX so its kind of frustrating to say the least.

Anyways i better head on looking forward to the BLUG meet on Wednesday! See you there guys!

no-space