I am asking for help from you or from your friend.. anyone, all you need is:

all minezz

• 5 minutes
• text editor
• basic knowledge in python

I am trying to build up a test suite for this compiler and i would love to have it all built from scratch for this project rather than taking something from CPython where i think licenses will clash and i want to avoid anything like that. So what does this all mean?

I need people to write up their favourite python snippets, but there are requirements: what you can do is use anything EXCEPT any imports. Imports are something which will be handled soon, and isn’t that important  yet for the core language implementation.

So I have set-up this mailing list and i would love it if as many of you would join: http://crules.org/cgi-bin/mailman/listinfo/gccpy

So join the mailing list or simply post a message if you prefer and we will try to remember to cc you with in. One thing i would ask is only submit a small test case if you are ok with this being pretty much un-licensed because GNU projects require copy approval for you to submit code, but we can negate this by going though me just submit this to me under a http://sam.zoy.org/wtfpl/ (Whatever you want to do license) . This just saves any pain of licensing, you will of course still be attributed! Leave your name and an email within the test-case file, since really that’s all we care about as hackers anyway we don’t care about licensing so long as we have some attribution. To give you a flavour of how simple these test cases can be lets look at one i made earlier:

Things for testing the expression syntax handling or calling functions, parameter passing like keywords parameters and positional parameters, class’s anything! Even if it is as small as one expression and a print for the result it is more than enough for a test case! The reason I want as many people involved is so i can see what features/trends matter most important to python users and then I can focus on making them extra awesome. And if it only takes you 5 minutes to write a small piece of python code and send an email while you get your name within the Gcc sources the more the better:).

When you create a test case simply send an email to the mailing list with the subject “Test Case <name>”. You can submit as many as you like the more the better  and the funnier the better…. * looks at Jezra ! @yamatt i am reusing that python while loop you sent me ages ago to freak out my processor! Thanks so much every one, watch this space in the next week for links on how to use and see your test case running in Gccpy, so you can compile your python code to an executable!

So remember the mailing list is over here. http://crules.org/cgi-bin/mailman/listinfo/gccpy

So why not make a game engine using my language with crules as a scripting language so I can see what the API _SHOULD_ look like for the client. So i choose doom since i am terrible with graphics, i can use the doom.wad and concentrate on programming; So in the past i have written two separate 2D game engines one was a Java 2D Monkey Island clone i done for university and it was terrible since Java is a terrible platform in my opinion, then my 2nd was a port  of it over to C++/SDL and the code was terrible so don’t bother asking me about it.

Anyway’s so the long and short i am writing doom engine from scratch, my original idea was to work with an existing game engine like Crystal Space or Doomsday Engine though, Crystal Space i was put off since it already uses Python and i don’t really feel like trawling though lots of code and forking it essentially would be really more work than its worth. Doomsday Engine is the cream of the crop of Doom Ports at the moment, all others are practically the same in essence using the same ancient code and hacks from throughout the 90′s. And really the technology has changed somewhat since then. Though Doomsday Engine is pretty amazing and works well, it seems to need a massive cleanup and i don’t mean to be nasty on the developers or anything they have done a fantastic job!  And they have really made the code much cleaner than any other port about! Plus the lead developer has been really nice to me . Anyway’s Games have always been a great passion to me and really doom can’t be that hard to re-implement from scratch and really it isn’t i already have a basic game engine working using OpenGL and SDL{ _ttf, _mixer }. With audio, and fps counter and input bla bla. But i am just concentrating on differentiating the wad sprite formats etc while revising for exams and working on crules and many other bits and pieces. So in the end  i’ll just stick in an extract of the log of my game engine when it starts parsing the doom.wad and explain a little on how to parse the wad!

debug: main.c:28 -> Trying to start crldoom wad: data/doom.wad!
debug: doom.c:147 -> parsing wad file <data/doom.wad>!
debug: doom.c:80 -> is a wad file :: <IWAD>!
log: doom.c:155 -> wad file data/doom.wad has length 12408292!
debug: doom.c:27 -> debug seeking…
debug: doom.c:30 -> reading….
debug: doom.c:34 -> parsing…
log: doom.c:49 -> header:: ident -> 1145132873, n_lumps -> 2306, infotableofs -> 12371396!
log: doom.c:62 -> directory:: filepos -> 12, size -> 10752, name -> 1497451600, ident -> PLAYPAL!
log: doom.c:62 -> directory:: filepos -> 10764, size -> 8704, name -> 1330401091, ident -> COLORMAP!
log: doom.c:62 -> directory:: filepos -> 19468, size -> 4000, name -> 1329876549, ident -> ENDOOM!
log: doom.c:62 -> directory:: filepos -> 23468, size -> 6854, name -> 1330464068, ident -> DEMO1!
log: doom.c:62 -> directory:: filepos -> 30324, size -> 9402, name -> 1330464068, ident -> DEMO2!
log: doom.c:62 -> directory:: filepos -> 39728, size -> 15466, name -> 1330464068, ident -> DEMO3!
log: doom.c:62 -> directory:: filepos -> 55196, size -> 3286, name -> 1330464068, ident -> DEMO4!
log: doom.c:62 -> directory:: filepos -> 58484, size -> 0, name -> 827142469, ident -> E1M1!
log: doom.c:62 -> directory:: filepos -> 58484, size -> 1430, name -> 1313425492, ident -> THINGS!
log: doom.c:62 -> directory:: filepos -> 59916, size -> 6804, name -> 1162758476, ident -> LINEDEFS!
log: doom.c:62 -> directory:: filepos -> 66720, size -> 19980, name -> 1162103123, ident -> SIDEDEFS!
log: doom.c:62 -> directory:: filepos -> 86700, size -> 1880, name -> 1414677846, ident -> VERTEXES!
log: doom.c:62 -> directory:: filepos -> 88580, size -> 8964, name -> 1397179731, ident -> SEGS!
log: doom.c:62 -> directory:: filepos -> 97544, size -> 956, name -> 1128616787, ident -> SSECTORS!
log: doom.c:62 -> directory:: filepos -> 98500, size -> 6664, name -> 1162104654, ident -> NODES!
log: doom.c:62 -> directory:: filepos -> 105164, size -> 2288, name -> 1413694803, ident -> SECTORS!
log: doom.c:62 -> directory:: filepos -> 107452, size -> 968, name -> 1162495314, ident -> REJECT!
log: doom.c:62 -> directory:: filepos -> 108420, size -> 6948, name -> 1129270338, ident -> BLOCKMAP!

So this is just some of the output of the code i had written it parses out whats called the ‘directories’ in the file and then i also have functions to parse out whats called the ‘lumps’ from these directories the lumps are the actual data. I’ll explain more detail when i talk about how to parse this. And how to convert the music lumps to proper midi so you can play it in SDL_mixer or even in Totem if in in gnome.

So what do we need, if you have yourself a Doom.wad file lying about i am sure if you go through your old boxes you’ll find yourself a copy of doom, or if you cant find it you can find it on torrent sites but you didn’t read that here . Thing is I have bought the game like 3/4 times I am sure hehe. Lets get started.

In the Doom wad there is whats called the ‘HEADER’ which is of length 12 bytes which contains 3 * 4-byte integers which contain the data we care about, so lets do that lets make some code (i just wrote this code in like 10 min to illustrate the idea its ok does the job!):

So lets compile and test this code <remember to change the path to your doom wad>:

gcc doom_wad_test.c

./a.out

So right what does any of this code mean i just chucked it up there, i wrote it up very quickly from scratch. So as i stated earlier there is a wad header which is a 12 bytes in length which i refer to as ‘WAD_HEADER_LENGTH’ which contains 3 * 4-byte integers. So from the code we have fopen’d the wad_file and i use fread to read in the 12 bytes in a 12 byte ‘unsigned char[]‘ buffer. So to read out the integers in a neutral endian way i found a nice parse_int function in some of the wad documentation i found and in old doom source code which was GPL so it was like…. yoink… but its pretty easy to understand if you know your representations well enough but i wont go into detail.

The first integer is the IWAD or PWAD string, IWAD is what you will find mostly its the deployment version of the wad, the PWAD is a patch wad and very few of them are still around and i think it has similar structure. Anyways, you parse out the first integer like this

Remember to add null string ‘\0′, so we don’t get buffer overflow! This string is always a length of 4 so no need to mess about! And then the next integer is the number of lumps in the wad the lumps are the actual binary data like music and graphics. Third int is the ‘directory offset’ which contains pointers to where the lumps are their idents and the size of the lumps which have lengths of 16 ‘WAD_DIRECTORY_LENGTH’ until the end of the wad file! so note i find the length of the file and do:

So how do we parse out these directories? We move to the directory offset and then read the 16 bytes into a buffer, which is 4 integers this time. But still contains only 3 things means the ident strings can be longer or can be with some padding between each directory lump.

Now we know the file offset of the lump the size of the lump and the ident of the lump and we make sure to add the null terminator again to avoid buffer overflow. And then we can parse out the lumps simply just fseek and fread Done its so simple and seems like there would be more to it, yet when you think of it is very much common sense there really isn’t that many ways to pack data into a file like this! If your interested i’ll be talking about it soon… on how to convert the D_E1M* lumps which are music lumps to compatible midi tracks. And finally on how BSP works for the levels and maybe how to draw them in C/SDL/OpenGL… if you ask i prefer C over C++, yeah when it comes to building this game engine there have been times i though woo having an object would be really nice here but then i realise well no because that’s what the scripting engine is going to do… Mainly due to the fact i dont like C++ syntax it feels messy to me and C is just nice and simple

Finally I just want to wrap up with saying John Carmack has to be one of my biggest Hero’s in Computer Science along with Migel De Icaza, Linus Torvalds, Various GCC hackers… like Ian Lance Taylor But John Carmack in particular since he made Doom! But also he attended the University of Missouri–Kansas City for two semesters before withdrawing to work as a freelance programmer. Which i think that is brilliant! Since University does not teach students to be serious programmers or technology well very few do anyway and i think its great he had the balls to do that and so did Migel De Icaza and they have both done some of the most amazing things in computer science!

Anyways if your interested in Crules or CrlDoom see: http://crules.org code for crldoom is over @ http://code.redbrain.co.uk/cgit.cgi/crldoom

This is a project i love Classic Doom 3 it was Doom re-implemented ontop of the Doom3 Engine what more do you want!!!

So lets just take an extract since i am lazy: So in compilers or interpreters how do we represent:

Lets see it in a diagram first then in IR code:

Expression

So what is this? Well early on when i was getting into compiler construction when doing semantic analysis you will need to find a way of representing these semantics for your language, and you will find generally that there are 3 things you need to know: DESTINATION OPERAND A and OPERAND B. This is what’s called 3-address code. This is what sets compilers apart and can make or break them! Its what gives implementations flexibility and the ability to ‘control the flow‘ of execution! Any good compiler book will give you more detailed discussion on this topic its my main interest in compilers since this is what gives an implementation its logic. So lets give it a syntax we can write down to illustrate how mine works: it just so happens a LISP ‘ish syntax can show this very well!

Since the parser is precedence aware it will construct the tree as shown in a symbol language which looks like this if it was output ( outputting the IR like this is a future TODO ); It may look a little like GIMPLE but this IR I have as I will demonstrate more examples can encompass much much more.. ( this is a 3-address code by the way )

busy!

Though one big note its is ok for me to leave the representation like this since in my interpreter i just evaluate this at run time but how does this affect things like code-generators like proper Compilers? Well its quite simple in a sane compiler like GCC ( although most compilers will follow this idiom i am sure) introduce another layer for IR called RTL ( Register Transfer Language ). The wikipedia article lacks and most compiler books overlook this section very much since, most books are written by the academics and generally stay away from the back-end of compilers since that is the actual hard part since a code-gen is very specific to the instruction set your targeting and register allocation algorithms are very difficult and is a huge research project.  Its said in computer science the most difficult things to build software wise is a good JIT or any Code-Generator for that matter.  Anyways back to the problem lets take a smaller expression for sleep’s sake the expression:

Will generate something like:

Anyways you’ll notice in my the IR keywords such as SYMBOL_ACCESS or SYMBOL_ITEM or OP_ADD etc, these are what’s called OP_Codes and are generally represented by some symbolic hex values since its cheap to check against for the implementation to figure out what to do with symbol ‘x’; So ok thats a quick and dirty intro to Intermediate Representation’s!

I think this lolcat will sum up your feelings at the end of reading this blog post:

And i started getting out my old grunge cd’s which include my Silverchair collection and this is on of my favourite bands of all time and this is a pretty good song!

First off go and download the 750mb tarball of the Intel C compilers over at: http://software.intel.com/en-us/intel-compilers/

% tar zxvf l_cproc_p_11.1.059_ia32.tgz

% cd l_cproc_p_11.1.059_ia32

% ./install.sh

# follow all the instructions it complained i didn’t have java installed but you don’t need it. Once your finished it should all be installed to /opt/intel/Compiler

% PATH=$PATH:/opt/intel/Compiler/11.0/069/bin/ia32

% LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/Compiler/11.1/059/lib/ia32/:/usr/lib/

You should be able to run ‘icc’ but you will probably get the error: ” /opt/intel/Compiler/11.1/059/bin/ia32/mcpcom: error while loading shared libraries: libstdc++.so.5: cannot open shared object file: No such file or directory

compilation aborted for test.c (code 127)”

So you need to install an older libstdc++ on debian it is:

% aptitude install libstdc++5

That should be it lets give it a test:

% icc test.c

Should work now! If your project uses a Configure.ac lets make a quick change:

Change your AC_PROG_CC to:
AC_PROG_CC([icc gcc cc cl])

This is the order in which the configure should search for each compiler. Since you need GCC installed to use Intel C++ you need to put ICC at the top of the list! Hope this helps someone!

So i am making a Python compiler(front-end), well starting too its going quite slow because i am in the middle of finishing up some automake work, and crules definitions. And GCC is quite complex in its development but its quite amazing and the most amazing project i have ever seen. I mean its development consists of many little files in different languages that aren’t even front-ends in GCC. For example a .md machine description is written in LISP that describes the architecture of an OS or Processor this makes it easier to port GCC to new systems, GIMPLE the IR was a language called SIMPLE but its GNU/Simple = GIMPLE, this gives the IR a better formalization, lang.opt files for command line options so you dont have to do any command line parsing in your front-end thats up to the gcc driver program( I’ll explain this some other time), Make-lang.in GNU/Make on how your code should be compiled, config-lang.in its in bash but does all the configuration on how the front-end is build with respect to Stage-x as in the boot-strap if it requires libraries to be built like JAVA needs libjava or if its an optional language.

The force!

Thats a lot of stuff people probably don’t care about! But its interesting what other project uses a cool a setup as that! I mean LISP, there isn’t even a lisp compiler in there but there its a bit of a one in the backend. There are some exciting things coming from GCC at the moment, LTO or graphite springs to mind, they aren’t ready yet, but LTO mean Link time optimization, so when you compile your program with gcc in whatever language, you compile each file to object code individually but you use gcc to link each object code and library together instead of using LD explicitly because it works nicer, so on Link time optimization when your linking all the code together another pass of the optimizations goes over a huge IR of all the object code to re-structure of remove dead code etc all the usual optimizations. And with OPEN/MP becoming pretty popular graphite you can pass flags like -fparallelize-all to extract loops even dependence loops to push in open/mp pragmas to make fully parallelized code without having to lift a finger. I’m working quite slowly to write 3 papers in total, a long thesis on Crules you can check it in my git-repos over at http://code.redbrain.co.uk and i am working on a thesis on the python-front-end as-well as a little paper on the gcc architecture. There is a tagged branch for a front-end-skeleton here on my git-repos: http://code.redbrain.co.uk/cgit.cgi/gcc-python/

So there is a lot of work there. And i chose to do a python front-end due to the fact everyone can say python is an awesome language no-one can deny that, it has been very well designed; language design is something a lot of people can over-look or take too far and python was a good mix in my opinion. What i mean is if you spend too long on language design your language will become very messy and hard for users to become comfortable with, something that has happened Haskell in my opinion although the way non-determinism is very clever in how Haskell has implemented it. On the other hand if you don’t spend enough time designing your language its going to be very messy and not much re-use of syntax. One thing to remember in design there is always a reason for the smallest of things be it the reason of using an identifier to using a ‘.’ for access or ‘- >;’ for pointer access, keywords have to make sense etc.

I was invited to join the graphite project i posted a quick patch to fix a part of their build-system, but i am not sure what i can really contribute because i would need to work full-time on it because its been going for some time now. Some of the work is pretty exciting. I am going to see later if i can maby merge the actual python parser from python into my GCC front-end because this would make a very nice setup to mirror the language features of python to an extend but there will be differences an GNU’isms because the mere fact that this will be compiled handling things like IO will rely on linking against libraries and the python pre-processor functions like import and from etc will be fiddly to work but i should be  able to take alot of work from the gnuJava front-end as there are similarities in how certain things work.

Something to Distract you!

Anyways from my goggling i haven’t been able to find an implementation of python that is compiled properly i mean natively compiled, some people mix the up the fact that python does create byte-code like java think its compiled code and it really isn’t. I remember at university a lecturer told us that compiling JAVA code you were creating Machine code to run natively on a computer. And he got the pipeline so wrong in creating a binary. They always made it seem that a compiler was a piece of magic that made a binary and java is NOT a compiled language unless you use GCJ. I personally find linkers are a piece of magic these days a lot of the know-how in how linkers are build is quite complicated in some ways. Although what you are essentially doing is taking the object code output from an assembler and creating an ELF format file with the object code in the correct segments and data in the top segment etc. And on windows its a COFF format i think, but even at that what are you doing and what does object code look-like is it literally the binary format of instructions in order from the corresponding assembly code?!

I would love to understand more on how that all works, but i mean an object code file is a not a plain text binary file so yeah i even posted to comp.compilers a thread on manging the jit you can look for it in the archives but its something that is still quite hidden, and only a few people who implemented it understand, i see how it all partly works because of the libjit project. like the defintions of an a.out and ELF were quite understandable i guess. But with a jit if your able to for example jit a function like:

If you are able to make target assembly code for this, and you assemble it and link it properly you have a binary how do you actually execute it in a useful way for an interpreter?! I mean do you keep adding new functions into this image and use dlopen functions and treat the symbols like a dynamic library and execute them that way through dlopen? I mean this is what the LLVM project hides from the users very well but anyways enough of that.

Anyways thats what I’ve been doing for a summer what have you been up to?! You can follow some of my stuff on several mailing lists like bison,autoconf, automake, gcc-help, gcc,comp.compilers. I helped a guy a while back on his bison parser, bison is a very useful tool, but for doing more free-form grammars is a pain I’ve found and i like to think i know YACC pretty well now from using it for about a year.

By the way I’ll be posting a wiki links on crules soon for example on how the language works etc and why but i would really like it if people read my thesis when its finished because i designed this language from a very specific set of reasons in the beginning for a specific application, but its become a fully multi-paradigm language in its design now, and this is what my thesis on it describes.

Anyways i’m going back to get some work done and i have BLUG tomorrow! Oh hey i found a new favourite band bought their album as soon as i heard this, its a polish metal band they are probably the a new Nightwish i guess! They are very good!

I think thats enough for now i have some photos to upload but i’ll do that next time

]]> http://redbrain.co.uk/index.php/about-me/python-gcc-and-crules/feed/ 1