Normally I sort of skip the hacking scene and just wait for the possibility of running code on the devices I own, after which I only hop in if it is feasible and not too complicated. The PSP was just one device (a software exploit). The iPhone should have been easy as well, but the hackers there don't release full working compilers on the latest version (that runs on PC's anyways). Looks like for the iPhone you just have to invest in a MAC, but then you also have to pay $99 per year just to have the right to have your own code running legally on the iPhone. Something is not right about that... Even on windows, you just have to pay for a compiler and you can make programs for life of the PC... Not $99 PER year even if you make nothing back from your programs..
Since this is about PS3 security, this time instead of documenting the internals of the PS3, I will talk about the efforts of some people who have made strides and places where people can get more information. I treat these forums as sort of a blog (I somehow don't like blogs as they are not structured like threads or wikis are), so there may or may not be edits or updates on this.
The newest security layer on consoles is the hypervisor. This piece of software (along with hardware) is sort of the gatekeeper. It has its hands on the decryption process (the most important layer) and other mudane low level stuff. Some of the stuff it does is... manipulating memory (keeping programs from corrupting each other), handle interrupts (IRQ) for devices that want immediate attention, handle all aspects of the SPE (7 of them on the PS3), some lpm (Logical Performance Monitor?) stuff, device communication, pci, network, GPU, and storage. Because it talks to the SPE, it also has its hands on the one SPE in isolation mode that does decryption for many of the security on the PS3. The hypervisor does all of the above using 256 "functions" or system calls available to any program with the right privilege level. It is like the kernel in the traditional sense, but kernels can run on top of it now. It seems like it only takes up 2MB of memory to store all those 256 hypervisor code in main memory (xdr).
Many people have access to the gameOS code already (on test consoles and tool consoles, not retail), and a while back there was even a leak publicly of an old version of the sdk ( http://rghost.ru/896794 ) and there were even some hacks into the retail PS3 hypervisor portion of xdr memory ( http://geohotps3.blogspot.com ). The retail PS3 came with the ability to run code on the linux portion of the machine, but the hypervisor was running intact, and as you can see the linux kernel needed to talk to it to do anything useful (as listed above in hypervisor calls). The main difference between the PS3 running in linux mode and GameOS mode is that in the GameOS mode, the isolated SPE is constantly running, decrypting and encrypting stuff from disc, harddrives and other security checking, which means a lot of PS3 games won't be running from the linux side (or GameOS side for that matter) if that SPE is not doing work. Which begs the question... what is preventing PS3 programs (not using the SPE) from full access of the machine? Well, for one, the concentration should be to access all the other pieces of the PS3 WITHOUT using the hypervisor. In other words, disable the hypervisor (free up the 2 MB) and remove it from memory, then write the kernel directly accessing all the PS3 device (one less hypervisor layer to go through). In other words, permanently disable the security layer (hardware abstraction layer). Code running on the PS3 would now be a little faster, and restrictions on the GPU would only be lack of appropriate drivers. Linux or some other operating system would run faster as well. It may be even possible that game backups may work if debug versions of games are patched, but this would be drifting into piracy and that is not he main purpose of this post.
