Next gen mobile operating systems

[edepot]

While doing a review on the Sony PSP a long while back, I had some ideas of how a custom firmware will improve the interface on mobile gadgets. Some of them were mentioned on http://www.edepot.com/reviews_sony_psp.html and since then some people have gone ahead and actually try to do a replacement for the XMB operating system that Sony provided. I thought that maybe I should throw out some ideas that may work well with small footprint display devices like the PSP.

Since Xerox PARC (and Apple) pioneered the windowing model, a lot of the operating systems these days seem to use this similar model, rehashed over and over again. Although the model works, the main problem with small displays like the PSP is that once you fill up a window on that small display, you don't have enough screen footprint for manipulating other windows. Until the PSP accepts a goggle or transparent glasses for virtual reality, you have a limitation on display footprint. If you were to use the TV out, that might help a bit, but in the case of the PSP, you are still constrained. Imagine running Windows XP on only 480x272... you can't do much with windowing stuff and still be productive. Even with TVout 720x480, it is still a little small. That is why they recommended that you run 800x600 on the lcd display for many OS'es. So the future is to forget the standard windowing interface (you can keep the scrolling stuff WITHIN a window, but multiple windows on small display may be a problem). New concepts should be implemented to deal with the problem of small mobile displays (when mobile). One area worth doing is using 3D stuff to represent applications, or go fully 3D with traversing a 3D environment with each application represented in the 3D environment. Since many mobile gadgets now have camera support (some able to do facial recognition), it should be easy to code or modify existing apps to recognize the outline of your hand or fingers, which you can flick your fingers and get the OS to respond (many creative ideas can be sprung up here). You can even invent an avatar that represents you in the environment and depending on your fingers you guide it to move to areas either free floating in 3D space, or like a person walking on the ground going to different applications.

Even running applications can utilize next gen 3D manipulation. As an example, let me describe a typical typing application... When you run it (moving your visible or invisible avatar to the application in 3D for example), you get a display for typing. Moving your finger to the left will show a big semitransparent A at the cursor position. Moving your finger right will show the Z, and in between, the different letters of the alphabet will scroll at the cursor position. Move your finger down and back up, and you just typed the letter you selected, allowing the next letter to be entered. You can change the way the letters are selected, but you get the idea. If the camera is not that sensitive, you modify it a bit, like moving your finger in a circle will scroll through the letters like on the old ipods with a wheel. If the camera can detect multiple fingers even better; You can move your fingers in typing position left and right in front of the camera and the screen will show the current "scrolled" five transparent characters which you can move the appropriate finger down to "type" it in.

To switch between applications, you can shake the gadget to drop the current application that is stuck to the display. Since the PSP doesn't have motion detection you can use the left trigger and right trigger. They use similar concepts in the psp browser to switch between browser tabs. When the application is switched, show the application in 3D being taken away from the display (it gets smaller or something when it drops from the display), and the next application growing bigger and filling the display. You can probably map it to a cube that you turn, or a book that you turn pages (each page is like an application). There are lots of ways in 3D to show this transition between apps.

Because mobile devices usually have internet, the os should be net aware so that when it is connected to the internet, you allow it check into a database of homebrew software, and automatically present a list that the user can choose to download by clicking and confirming (like windowsupdate.com but let different authors put their stuff on the site, categorized of course into different types of apps and games). Because peer-to-peer is getting popular, making a torrent client and using it like a "windowsupdate.com" to a torrent site would serve as an easy way to obtain software via the internet. If going the whole torrent site route, the PSP torrent should only show listings that the OS knows is compatible with itself (either through a search keyword or some sort of torrent file keyword filter).

In addition to the internet, most devices have built in wi-fi. Making the OS PAN aware (personal area network) via the wifi is also beneficial to mobile devices. Simlar to the NDS lite, you can instantly connect and chat with fellow people who are close to you. The goal here is to allow easy trading of files (pics, music, whatever) and chatting and even gaming (if homebrew authors get access to an easy communication sdk to makes compatible apps for the users to send to each other based on psp id). Even a PS3 Home copycat (full 3d avatars) with just adhoc for connected PSP within a wifi area would not be out of the question here.

Now we get to the problem of multitasking. Some mobile devices do not have CPUs that can multitask well. Sometimes it is up to each individual author to code an app that seemingly can multitask, the problem however is being able to multitask apps by different authors. Because current PSP pspsdk homebrew does not allow you to do multitasking of apps by different authors, the OS should allocate a simple method to switch between code that the users have created. The best way is to let the CPU stop a thread, save its context, and load and switch another thread to run. If this ability is not possible, then you have a problem with different apps from different authors wanting to coexist, without waiting for one app to exit first.

Therefore, the next operating system for the next generation mobile devices should take into account the small screen size, and try to negate the negative aspects by concentrating on new 3D operating systems.

[edepot]

One of the trends these days in mobile devices is to use the touch screen. This technology is very useful for dynamic keys that can change in size and location on the touch sensitive display. When mobile devices reach a standard of having this built into every display, they will find that there is another dimension to add for input, and that is the third dimension. Touch screens are 2D (width and height) input devices. If you also include the Z direction (up), you can do 3D. And thus, the next trend after 2D is to go with 3D input, and that can be achieved by the utilization of a camera that has associated software or hardware that can track objects in 3D. Since this is an evolutionary extension of the touchscreen, a good way to utilize "Camera"-sense is to have code that tracks finger movements in 3D space that is in front of a camera attached to the mobile device, and is facing the user. The easiest utility for this 3D input is to track your finger as it moves in front of the camera. By dividing the area into squares, you can track very quickly any movement that falls within the squares by checking the center pixel of each square. Varying the size of the squares allows adjusting the sensitivity of the tracking software. This basic quick-and-dirty method allows tracking the finger in 2D space. For the third dimension, the size of the fingertip can be also be considered. The closer the fingertip is to the camera, the bigger the round circle, and the farther away, the smaller the circle. Quick routines for tracking fingers can be accomplished by first checking the center pixel for the color of the finger, then if it "hits" check adjacent pixels outward within a circular pattern, bigger and bigger until you hit a non-finger color circle. The number of pixels to check within a circular outline can vary depending on the speed of the CPU, size of the squares, and the efficiency of the code. So, in a sense, touch-screen can actually evolve into 3D input devices. Of course this does not preclude the possibility of putting different color marks on your fingers and let the software track multiple fingers at once.

However, this type of camera-sense (as opposed to touch-sense) is better if it is augmented with virtual reality glasses, preferrably light transparent ones with adjustable opaque settings. While this technology is still being resolved, the next best thing is being able to see 3D within a 2D surface (like how most games are these days).