BomjPacket: A WAP-based Internet browser

Imagine a cellphone that only has a GPRS/WAP feature and understands WML but not XHTML. These kind of phones are typically in the $100 price range. People use them to make calls but rarely do they think of using them to browse Internet. There are mobile browsers that allow one to browse Internet using a cellphone, for example Opera Mini. But they are resource demanding, whereas the idea of this project is to use minimal resources while giving the ability to browse the Internet. In short, BomjPacket is like OperaMini, but it is WML-based and it is open-source!

Bomjpacket-2.0 Siberian Cutter advances mobile browsing to a completely new level. It supports JavaScript and CSS thus allowing to execute interactive Internet applications on a cellphone. Siberian Cutter is Nikita Mikhalkov's movie in which a crazy Siberian engineer spends his life building his dream gadget, just like me.
Of course, it is impossible for just one developer to implement a comprehensive browser. Therefore, I leveraged effort of a number of developers using their freely available projects as parts of my browser:

Download Siberian Cutter

Give it a try!
Enter the following URL into your WAP browser:
http://bp.alexeysmirnov.name.

MetaPHP: PHP-based context-free grammar (CFG) parser generator
J4P5: PHP JavaScript grammar and RT environment
CSS Parser
PHP DOM Implementation
libcurl, included in PHP distribution.

Google

Google News

Amazon

We all use Google. Remember that pull-down menu on top of the window? Siberian Cutter supports this type of menu.

INSTALL

INTERNALS

README


Suppose you own a website and you would like to host this HTML->WML
converter for your own purposes or for public use. Then copy all PHP
files to your website and configure the main script.

You will need to define the following variables:

*  $basedir="./results/";
   This is the local directory to where the WML pages are written

*  $baseurl="http://mysite.com/";
   This is the URL of your website

*  $scripturl = "http://mysite.com/browser.php";
   This is where you installed this script

There are a number of other variables that you might want to change:

*  $trace=0 - turns debug output on/off

*  $need_dump=0; - prints out the results of HTML->XHTML conversion and stops

*  $output_type=1 - there are two types of WML output: hierarchical
   and using inlining. The latter one is the default.

*  $no_title=1 - allows to save one line in on the mobile screen

The README file will provide more insight into the meaning of these
variables.


This file gives an idea of the inner workings of this converter. See
INSTALL for the administrative questions.

HTML to WML conversion. Uses expat XML parser.

This is performed in two steps:

HTML -> XHTML -> WML

The differences between HTML and XHTML are:

* matched and properly nested tags, e.g. <b><i>Test</i></b>
* all attributes are in quotes: <a href="url">Interesting page</a>
* no empty attributes: <table border> is unacceptable
* case of the opening and closing tag mathces, i. e. <b></B> is invalid

The main difference between XHTML and WML is that the latter supports
only a subset of the former's tags. 

We will describe the two steps in greater detail below:

First, however, we need to do a few preprocessing actions, namely get rid of Javascript and CSS as they are
different languages. Then we get rid of HTML comments. After that, we need
to take care of encoded characters b/c the parser does not understand
them. For example ™ gets converted to TM.


 HTML to XHTML
--------------------------------------------------------------------

Function need_quotes() adds quotes to attributes wherever they need them 
For example, <p align=center/> gets converted to <p align="center"/>

Then we need to fix unclosed quotes in tags and remove tags from
strings. Function fix_quotes_tags1() takes care of that. For example, 

<a href="url1>Page1</a><a href="url2">Page2</a>

gets converted to

<a href=url1>"Page1"</a><a href="url2">Page2</a>

using the following rules:

"a<..." -> "a"<.....
"a<.../>b" -> "a"<.../>"b"
".../>a" -> ..../>"a"


Imagine a string with quotes inside: 

title="this is "OK" button"

We will need to get rid of internal quotes. The first quote is extended until
the next tag. Function fix_quotes_text() takes care of that.

Function tags_toupper($data) convert tag names to uppercase.

Function fix_tags($data) does a number of things:

  * replace these tags with line breaks:
    </TR> -> <BR/>
    </P> -> <BR/>
    <BR> -> <BR/>
    </BR> -> <BR/>

  * eliminate all the tags except WML ones. We only need the following
    tags:

    'NOP', 'HTML', 'HEAD', 'TITLE', 'BODY', 'H1', 'H2', 'H3',
    'H4', 'H5', 'H6', 'A',  'BR', 'B', 'I', 'EM', 'LI', 'ADDRESS',
    'DIV', 'CODE', 'BLOCKQUOTE', 'TT', 'PRE', 'STRONG', 'SMALL',
    'SUP', 'SUB'

  * add opening and closing essential HTML tags if they are missing:
    <HTML>, <BODY>, <HEAD> as we need them in each HTML document


We have already taken care of the attributes adding quotes around
them, but function fix_attrs() does more:

  * Eliminate the following attributes as they might contain
    javascript:  "onclick",  "onchange",  "onfocus",  "onmouseover",
    "onmouseout",  "onmousedown",  "onmouseup",  "onkeyup",
    "onkeydown",  "onkeypress",  "onsubmit"

  * Remove empty attributes

  * For the A, IMG, and DIV tags keep only certain attributes. We need
    to filter out certain attributes because the parser gets confused when the same attribute repeats which might happen.

  * Certain tags do not allow other tags nested into them, for example
    <a href="">NO TAGS HERE</a>. Function filter_content($data, $tag)
    takes care of that.

Finally, add closing tags if they are missing and remove redundant
ones, e. g.

<div>

</div>
</div>

gets converted to <div> </div>

Function pair_tags($data) does that.

 XHTML to WML
--------------------------------------------------------------------

Now we have a nice XHTML file which is what the cellphone wants. I
guess the reason why this is necessary is because cellphones do not
have processing power to fix HTML file if it is not properly
formatted. Therefore, this is what the server-side component has to
do. But XHTML file on its own is not enough either. Typically, it is a
very long web page, but think of a mobile's screen - it is very
short. Therefore, we need to break the XHTML file into a number of
smaller files, and we also convert them into WML language - a subset
of HTML that cellphones understand. A few of them understand XHTML
also but WML is simpler.

The breakdown algorithm takes into account the layout of the original
HTML page. A typical page includes a numebr of DIV elements which are
nested into each other. For example, DIV id="wrapper" might have DIV
id="menu" and DIV id="content" inside. Therefore, we will place two
links on the first WML page:

MENU
CONTENT

When a user clicks either of them (s)he goes to the appropriate WML
page of that section. Therefore, there are 3 WML pages in total.

Unfortunately, the names or IDs of the sections are not always
self-explanatory. Therefore, guessing what hides behind a given link
is often difficult. Instead of including links to the sections we give their preview,
that is, include a few lines from each section and insert a link at
the end that allows the user to expand that section:

HOME - NEWS - ARTICLES-...
This is a very interest...

Only when the user clicks on the "..." link does the cellphone go to
the appropriate section.

Bomjpacket is HTML to WML conversion. Uses expat XML parser and CURL library to
fetch pages.

List of features:

* Converts an arbitrary web page to a WML representation using DIV
  tags as the basis for breaking down a complex HTML file to a number
  of simplified WML files.

* Images less than 100x100 are loaded.

* GET/POST queries.

* Cookies.

* SSL has not been tested but likely possible.


Demo site:

http://bp.alexeysmirnov.name

Browsers	Mobile Software	Cool Gadgets	Academic Research
Wapsilon - WAP emulator. http://wapsilon.com/ Mobile Emulator. View Bomjpacket in this emulator. WinWAP emulator. http://www.winwap.com Web Browser for S60 Nokia MiniMap Browser. http://www.nokia.com/browser OperaMini. http://operamini.com TeaShark - a Java-based WAP browser. http://teashark.com/. Gmail on the move. http://sourceforge.net/projects/gmail-mobile/ WAPReader. http://sourceforge.net/projects/wapreader/ Math4Mobile. http://www.math4mobile.com/ http://gateway.oceanlake.com Phonifier optimizes web pages for mobile phones. http://phonifier.com/ Siemens A60 - our main test phone. Its OpenWave browser is very fast to open long pages. And it costs less than $100. Siemens M65 - a similar cellphone with a larger screen. SonyErocsson K500i. Sony Ericsson Z558i - an inexpensive cellphone with touchscreen Fly SX210 - a cellphone with a touchscreen and web-camera. Research Papers: O. Buyukkokten, H. Garcia-Molina, and A. Paepcke. Focused Web Searching with PDAs E. Kaasinen et al. Two Approaches to Bringing Internet Services to WAP Devices O. Buyukkokten, H. Garcia-Molina, and A. Paepcke. Accordion Summarization for End-Game Browsing on PDAs and Cellular Phones Tapan Parikh works on mobile tools for developing economies. He was selected as one of the MIT TR 2007 Young Innovators. Google is also working on the content adaptation for cell phones. Try google.com from your cell phone.

^*The name of the project means Homeless people's food in Russian.

BOMJ-Packet*: Full-Fledged Internet on a Primitive Phone

BOMJ-Packet^*: Full-Fledged Internet on a Primitive Phone