RedBrain @ $HOME : ~ $

Swap my memory!

So its been a while since i have updated, while i was offline i guess, i have been moving to a new server! I have had a few teething problems but i think its all going fine now!

I decided for a VPS off: http://www.vpsville.ca/index

I got the CITY plan Its very powerful because its running in some kind of container VPS, i get to see the full ‘$ cat /proc/cpuinfo’. And it 8 core’s of the new Intel core i7! But i only have 256mb ram which was the teething problem so i turned off innodb in mysql and limited php memory usage to help compensate. It was nice Debian Server has all the mysql.conf and php.conf files all documenated very well so you generaly have an idea what your changing.

This is a very useful link for mysql tunning: http://howtoforge.com/tuning-mysql-performance-with-mysqltuner

——– General Statistics ————————————————–
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.0.75-0ubuntu10.2
[OK] Operating on 32-bit architecture with less than 2GB RAM

——– Storage Engine Statistics ——————————————-
[--] Status: +Archive -BDB -Federated -InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 940K (Tables: 32)
[--] Data in MEMORY tables: 0B (Tables: 1)
[!!] Total fragmented tables: 2

——– Performance Metrics ————————————————-
[--] Up for: 2h 45m 48s (6K q [0.612 qps], 214 conn, TX: 11M, RX: 1M)
[--] Reads / Writes: 74% / 26%
[--] Total buffers: 58.0M global + 2.6M per thread (100 max threads)
[!!] Maximum possible memory usage: 320.5M (125% of installed RAM)
[OK] Slow queries: 0% (0/6K)
[OK] Highest usage of available connections: 4% (4/100)
[OK] Key buffer size / total MyISAM indexes: 16.0M/331.0K
[OK] Key buffer hit rate: 99.7% (29K cached / 87 reads)
[OK] Query cache efficiency: 58.5% (2K cached / 4K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 291 sorts)
[!!] Temporary tables created on disk: 30% (103 on disk / 339 total)
[OK] Thread cache hit rate: 98% (4 created / 214 connections)
[OK] Table cache hit rate: 89% (50 open / 56 opened)
[OK] Open file limit used: 9% (98/1K)
[OK] Table locks acquired immediately: 100% (3K immediate / 3K locks)

——– Recommendations —————————————————–
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
MySQL started within last 24 hours – recommendations may be inaccurate
Reduce your overall MySQL memory footprint for system stability
Enable the slow query log to troubleshoot bad queries
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Variables to adjust:
*** MySQL’s maximum memory usage is dangerously high ***
*** Add RAM before increasing MySQL buffer variables ***
tmp_table_size (> 32M)
max_heap_table_size (> 16M)

That’s the kind of output i got in my first go so i start going though each of these recommendations: This link shows you each of the different variables you can change: http://dev.mysql.com/doc/refman/5.1/en/innodb-configuration.html, I think you can change a lot of Apache  memory variables but i haven’t looked into this yet!

But i also decided to make a swap file, this is very easy in Linux if your system is all running directly off a rootfs you can:

$ dd if=/dev/zero of=/myswapfile bs=1024 count=1048576

#this should make you a 1G file which we can use for a Swap file!

$ mkswap /myswapfile

$ swapon /myswapfile

#this should work but in my VPS i keep getting ’swapon: swapfile: Operation not permitted’

#you can also add this to your fstab like so:

$ echo “/myswapfile       none    swap    sw      0       0″ >> /etc/fstab

So yeah that’s quite useful to know! I have to say i have upgraded my wordpress to 2.8 and very unhappy to be honest, there are a lot of little bits and pieces annoying me 2.7.1 was a very good release! But this 2.8 is very memory hungry, but from what i read is that they have introduced a new API for widgets, plugins and themes. So i guess there will be problems for a while!

Anyways what i am hosting on my VPS is 4 sites:

http://redbrain.co.uk

http://brendanmullan.com

http://team-qcg.com

http://animeequation.com

They are the websites so i run: mySQL,   Exim mail server(because noodles recommended it to me), Apache (might test light httpd), PHP plus various php plugins, git (using those python-tools): http://scie.nti.st/2007/11/14/hosting-git-repositories-the-easy-and-secure-way

I have to say anyone who works with me will know i hate SVN and CVS. Mainly because they are just retarded and your locked in too much and if the server goes down, then what the hell are you meant to do. My main reason for sticking with Linux was the fact now your no longer locked into software. A lot of people i know in the windows world use windows but use ‘cracked’ copies of it, as-well as photoshop, games, office etc… all downloaded etc. Using free software alternatives means your no longer reliant on these kinds of software and you know whats available and you learn alot more about your computer and why!

So anyways i thought i would post a screenshot of my current desktop i think it looks pretty nice tbh! I was using Xmonad for ages there, its a great window manager written in haskell its great because i run a server on my work desktop! And then i still need a way to run firefox, email and emacs. So this was very useful, and its very lightweight and usable. Other tiling window managers are light but very complicated!

my laptop

So i am also giving Haskell another go, i got the book ‘Real World Haskell’, its actually pretty good there is a helluva lot of learn, but its going to take a while before i’ll be seriously able to say i can write a mail client in haskell rofl! I would seriously like to make a GHS right now a gcc implementation of Haskell, because i have started the gcc-front-end documentation with a friend.

http://gcc.gnu.org/wiki/FrontEnd/skeleton

Problem is were kind of stuck right now, i posted most of it all so far and put up the code, etc but there have been quite a few changes in the latest gcc-4.5-snapshots so trying to keep up with them. But the real problem is i posted the code etc, but posting source code on a wiki page… its not that nice! So this is another point where the gcc-developers on oftc.net have said should really get my FSF-copyright assignment document stuff done, because i have my automake work and now gcc, and they would like to see official documentation in the GCC-manual in the next releases i think this would really help GCC.

If you working in industry like me, you’ll find the only real thing keeping compilers/interpreters and programming languages alive in large business driven companies is the idea of parallelism, this is quite old right now but its only really getting together now! I also have a book on OPEN/MP and know some of the basics of it in C/C++. Its very impressive the implementation in GCC and the Glibc. There is only a little theory to learn about OPENMP because actualy coding with it, is very simple! But you wont understand the code until you learn a little about shared memory systems and threading on this! Because OPENMP at the end of the day is starting threads for faster execution behind the scene. But it does all the work for accessing and manipulating memory without breaking your code for you! So in that sense its awesome

So yeah i also joined http://wakoopa.com/redbrain Its quite an interesting idea! And also pointing out that: http://outlawpla.net/ This is a Linux outlaws Planet, it should be cools i am hoping to get my blog on it as Dan and Fab pointed out in the show yesterday!

Anyways spent most of my time reading tbh recently so i havent been coding much, re-writing my OS from-scratch again i am starting to understand the i386 kind of now, and i want to structure my kernel correctly and make sure its as similar to just coding with libc as much as possible so i dont implement a stupidly named API of mad functions that only i understand! But i have done bits and pieces of cleaning my simple-programming language. Need to finish that tbh, there isn’t much more work for me to do to demonstrate the basics for it all. But been doing reading and writing takes alot of time when i work around my Job which takes up most of my time and i haven’t had a chance to get a break to do serious work at work, been documenating and transferring knowledge of work systems on Unix to new guys which is taking some time as they dont have a background in Linux or UNIX so its kind of frustrating to say the least.

Anyways i better head on looking forward to the BLUG meet on Wednesday! See you there guys!

no-space

Identi.ca -> Curl -> Update

So yeah, i told you guys in my last post i would be doing a tutorial on compiler work, i am still working at this, tbh there is so much writing i am thinking i might make like a sub-domain for it like a latex-html type thing, i have been writing a long .odt in openOffice, but i think if i want to put it on this blog it will be far too big. But anyways well see i updated some of my stuff here on the blog feeling much happier i think i have found a theme i can settle on!

Oh and i added the link for my hello-world gcc-frontend

$ git clone git://github.com/redbrain/gcc-sbsh.git

$ ./configure

$ make (..sleep for 2 hours…)

//what your interested in is gcc/sbsh/ * all the files in there is all the stuff you have to do to make a hello world front-end you need all those definitions etc.. i think i have broke my Make-lang.in but i’ll fix it asap. If your on a mac i posted the fix for mpfr on the gcc-help mailing list.

To compile on a mac:

$ wget -c ftp://ftp.gmplib.org/pub/gmp-4.3.1/gmp-4.3.1.tar.gz

$ tar xvf gmp-4.3.1.tar.gz

$ ./configure

$ make

$ sudo make install

$ sudo port install mpfr

$ cd gcc-sbsh

$ ./configure –prefix=<maby put a prefix for install here instead of sys-wide install> –with-mpfr-lib=/opt/local/lib –with-mpfr-include=/opt/local/include

Now it should configure properly!   http://www.nabble.com/Configuration-problem:-not-recognizing-MPFR-td23556792.html. You may find my on gcc-help pretty often now and automake bison and autoconf. I use these apps alot now and the mailing lists are really intereasting. Linux-kernel list sucks its far far too fast to follow unless you a full time-developer drizzle i dont have enough time for either KVM i think they are all over the place

And here is the git-repos of my work for the hello-world interpreter its still a work in progress but i think its ok for one evenings work

$ git clone git://github.com/redbrain/simple-programming-language.git

$ apt-get install build-essential autoconf automake libtool

$ sh autorun.sh

$ ./configure

$ make

$ ./src/simple –help

$ ./src/simple tests/helloworld.sim

Oh and i have done some massive updates to the Belfast LUG wiki in general: http://www.belfastlinux.org/wiki/Main_Page

http://www.belfastlinux.org/wiki/2009-04-06_mobile-monday

http://www.belfastlinux.org/wiki/Meetings

I will aim to keep the Meetings page updated with my email after the meeting with what happened at the Meet like in: http://www.belfastlinux.org/wiki/2009-03-25_pubbage. I will maby start a page in the future about going to LugRadio live 2009 but thats a good few months off yet..

So yeah This last week and a few days i have spent my evenings doing some writing on compilers etc… for the tutorial and playing with haskell.. I am trying to learn the curl api to make an identi.ca client written in haskell and it cant be hard because the code to do it in C is well tiny look i knocked this up in a few mins:

#include "stdio.h"
#include "curl/curl.h"
 
#define IDENTICA     "http://identi.ca/api/statuses/update.xml"
 
int main( int argc, char *argv[] )
{
  CURL *curl;
  CURLcode res;
 
  curl = curl_easy_init();
 
  if ( curl ) {
    char *data="status=hello world identi.ca libcurl - c";
    curl_easy_setopt(curl, CURLOPT_POSTFIELDS, data);
 
    //username:password... male sure to keep the ':'
    curl_easy_setopt(curl, CURLOPT_USERPWD, "username:password");
    curl_easy_setopt(curl, CURLOPT_URL, IDENTICA);
    res = curl_easy_perform(curl);
 
    printf("result -> %i\n", res);
    /* always cleanup */
    curl_easy_cleanup(curl);
  }
  return 0;
}

You can compile with:

$ gcc -o identica main.c `curl-config –libs –cflags`

So as you can see posting to identi.ca or twitter is a simple task. But haskell… well it isnt simple I do love haskell in some ways its very much the way you think of things if your doing a very long definition in Pure Math or a proof in Applied Math. Yeah i do a Math degree… i hate it a i love it just i prefer doing programming mostly.

Anyways the problem i have with the language is its becoming EXTREMLY messy in my opinion, instead of people concentrating on making haskell useful to actually make apps and wrappers to useful libraries they seem to keep adding features to the language. Like the way things work in it take a long time to get your head around but to manipulate data is why i LOVE it. So its like look at this tutorial it just explains why i love haskell http://www.haskell.org/~pairwise/intro/section1.html#part4

lolcat

Thats just an awesome idea lazy evaluation, i am getting to grips with it but i think it would be better if i get a book on it rather than reading a sticking bits of what i see around the interenet together.Its just its all very well being able to make a list and do silly evaluations on them etc.. But you need to have some good networking apis or some decent file-read-write apis there are some but its just very centric around read or getch type functions. Then GHC dam its a huge and the main implemenation of haskell, but its sooo bloated. I was toying with my own idea to make a gcc-haskell front-end but i don’t know if i like the language enough to do it, i would probably prefer to make my own version of it. But then it would take a long time to persude people to move over. etc…

Anyways its nearly 1AM and i have work in the morning Remember to post comments guys!

Programming languages implementation!

So sorry for the long time without a post but I have been so busy its not even funny, I have had to take away my time from Drizzle on launchpad now to concentrate on my own stuff. So I also am taking some vaction off from work, so I have some time off which is just brilliant at the moment, I have been playing Zelda on the wii, and I updated lots of material on the Belfast Lug Wiki. And worked on what I am about to explain.

So yeah just want put in this link for anyone who has had problems with Multiple Lexers or Parsers with Flex and Bison in one project i posted my solution to the automake mailing lists last month because i was working alot with that trying to figure it out:  http://lists.gnu.org/archive/html/automake/2009-04/msg00095.html

So in my spare time I have been working on a programming language that’s cool (I think any-ways), its taking some ideas from the GNU/make and extending them to be used for easy rules programming  to be used as a standalone language or embeddable into applications for any kind of application to make decisions much easier. I am working with this on launchpad:

https://launchpad.net/silix

There is my Kernel on there but i havent had any time to work on it ages, there is my crules language which is the rules language i was talking about and scc which is the simple C compiler it isnt much implemented yet though. Not much time recently.

But i’ll introduce my ideas for my programming language. Hmm i am not sure how i am going to illustrate this without showing code.. maby i’ll wait untill next time its like make but you can override rules depandant on what variables are in scope. Thats the wordy answer

But before I show how it looks soon in the future when I get it to a decent working stage I want to put a tutorial on how you can go about making any kind of programming language right down to how you would go about making a full ELF binary etc!

Because this is something NO-ONE has really done in ages, no-one really writes full compilers or assemblers and linkers anymore, thats’ with the whole GCC project Ian Taylor from google has shown some work recently on where he is writing a completely new linker to replace gnu/LD which is called gold. This is very important not only because if that isn’t properly maintained the knowledge goes too on binary formats and how to generate them from object code made from assemblers. Which is something in my opinion very few people know how to do now adays.

GOLD := http://airs.com/ian/gold-slides.pdf

What the tutorial I am working on is how to make a programming language compiler or interpreter. So I made up a VERY simple language just to illustrate the ideas and how to implement these ideas. I do it all in C with Flex and Bison.

I am sick of people talking about writing compilers/interpreters in Java, saying its a valid language for such a thing I beg to differ for too many reasons.

Mainly because of this say you implement a Language X this is the stack of software that X depends:

X → x-interpreter → Java-runtime → C-programs → Kernel → Hardware

So why use something that limits you, I always find with java you concentrate on how java says you should implement your ideas, how about NO you should concentrate on your ideas and your shouldn’t be limited by your language, so I just like C .

So I will try and get this tutorial up ASAP its taking a while to write up lots to write finishing off the code just need to implement support for methods and I think its complete enough for this kind of hello world tutorial on how to use Bison and Flex to help you make your interpreter, and I will show how you can go about changing how it works to be more like a compiler to generate an output.

Compiler-life-guard