HTML 5 had taken steps in specifying how web content (e.g. news story, blog entry) should be represented in a page. The specification has attempted to structure a web page by separating different parts of a web page such as headers, footers, navigation, content etc.  The elements of HTML 5 that will help with content extraction are <section> and <article>.

The <section> element is described in the HTML 5 specification as follows,

“The section element represents a generic document or application section. A section, in this context, is a thematic grouping of content, typically with a header, possibly with a footer.

Examples of sections would be chapters, the various tabbed pages in a tabbed dialog box, or the numbered sections of a thesis. A Web site’s home page could be split into sections for an introduction, news items, contact information.”

Having an HTML element that groups content is very welcome.  The <section> element can be used to contain content such as a news article.  HTML 5 has gone one step further to make this possible by introducing the <article> element which the specification described as follows,

“The article element represents a section of a page that consists of a composition that forms an independent part of a document, page, or site. This could be a forum post, a magazine or newspaper article, a Web log entry, a user-submitted comment, or any other independent item of content.

An article element is “independent” in that its contents could stand alone, for example in syndication. However, the element is still associated with its ancestors; for instance, contact information that applies to a parent body element still covers the article as well.”

 A structured implementation of the <section> and <article> elements by content publishers will go a long way in making content extraction simpler thereby providing for a small step in making web content semantic analysis easier.

The HTML 5 specification has been out there for some time, its time for rendering engines to start implementing some of the new semantic oriented elements in the specification (some rendering engines have already started implementing parts of the specification). 2009 sounds like a good year for rendering engines, content publishers and content generation software to come together and help chart the course for web semantic analysis-based applications.

NOTE: HTML 5 contains other descriptive elements that help with the expression of semantics of textual data. I will get to those in future posts.

As I have said in the past, the goal of the semantic web is still a long ways to be realized. However, rather than wait for every website owner to build semantic web conforming website (or retrofit their past content to be semantic web compliant), we should seek to derive web semantics at the application level using a whole new set of applications, web semantic analysis-based applications. Yahoo’s BOSS can be one of the missing components that pushes the ball forward towards this goal.

Surprisingly, one of the challenges of deriving web semantics is as simple as programmatically identifying and extracting the content from a web page (I have talked about this in a previous post: A case for standardizing blog templates).  Before semantic analysis can be performed on a web page, the proper content must be extracted fom the web page first. As humans, when we look at a web page, we can readily distinguish the “main content” of a web page from  navigation bar, header, links or ads.  This is not so easy for computer programs to accomplish.  At Filtrbox, we have developed algorithms to accomplish this with a very high success rate only because we have devoted time and resources into the algorithms because they are core to our business.  Other application developers wishing to leverage web content semantics may not have the time and resources to build such algorithms because that is not core to their business. This is where Yahoo BOSS comes into the picture. We know that Yahoo has built its massive Web index by indexing the “main content” extracted from web pages.  Yahoo has  invested time and resources to solve the content extraction problem. In addition, they have built a massive infrastructure to index and store web content.  Therefore, instead of re-inventing the wheel, developers of applications that leverage web semantics can take advantage of Yahoo’s content extraction through the Yahoo BOSS API. However, Yahoo needs to open up a little more for this to be possible.

Here is where Yahoo needs to open up: Although Yahoo currently performs content extraction and content indexing, unfortunately the Yahoo BOSS API is not geared towards applications that analyze web data semantics.  The Yahoo BOSS API in its current form is geared towards web searches.  It is keyword query-based and returns at least TITLE, URL and ABSTRACT/EXCERPT.  Unfortunately, to move towards web semantic analysis-based applications, the ABSTRACT/EXCERPT alone is not enough.  Instead, the Yahoo BOSS API should return the WHOLE “main content” (not links,ads and navigation etc) of a web page.  Returning the whole content enables applications to perform semantic analysis on the data from millions of web pages that is stored in Yahoo’s web index, thereby adding value to the data and moving the ball forward towards unlocking the hidden value in web data using web semantic analysis-based applications.

One of the questions that I am often asked is how Filtrbox is different from traditional RSS readers and aggregators.  The following are the major differences:

Closed Search Domain vs. Open Search Domain

When using traditional RSS aggregators, the user supplies the list of RSS feeds. This means that the domain of information gathered by a traditional RSS reader/aggregator is limited to the RSS feeds that are known to the user.  I call this a closed search domain. However, in an environment such the one we have today where thousands of new content sources are being created on a daily basis and anyone can potentially become a publisher, it is unrealistic to put the burden on the user to keep up with the thousands of new content sources that are sprouting up each day.  Filtrbox takes this burdensome responsibility away from the user and discovers the new content sources for the user because Filtrbox’s search domain covers all the new content sources. I call this an open search domain. The user can also add RSS feeds to the search domain, thereby guaranteeing that their RSS feeds of interest are searched. This approach leads to the user discovering new content sources.

Publisher centric vs. Content centric

Traditional RSS readers/aggregators present to the user all the content that is published by a specific publisher regardless of whether the user is interested in the content or not. Thus, the traditional RSS readers/aggregators implement a publisher centric information consumption model. On the other hand, Filtrbox implements a content centric information consumption model.  Rather than deliver to the user all the content published by a specific publisher, whether its relevant or not, Filtrbox allows the user to filter for the content that they are interested in from ANY publisher by providing contextual keywords. The content centric model implemented by Filtrbox greatly reduces information overload because each piece of content is examined and filtered for contextual relevance before it is delivered to the user.

No filtering vs. Contextual relevance filtering

As indicated above, traditional RSS aggregators do not filter the content.  All content published by a publisher in the user’s closed search domain is delivered to the user regardless of whether it is relevant or not.  Filtrbox applies algorithms that filter content from an open search domain of publishers for contextual relevance.  Filtrbox uses multiple factors to determine the contextual relevance of content and assigns a score called FiltrRank.  The most important feature of the algorithm is that the contextual relevance algorithm learns from a Filtrbox user’s implicit interests and applies the implicit interest to future contextual relevance filtering. This means that the content delivered to the user is content that that specific user is interested in and not content other people are interested in.  Contextual relevance filtering plays a large part in the reduction of information overload.

Beyond RSS

To help answer this question, here is the Web 2.0 school of thought that I subscribe to:  Let’s start off with an enterprise database analogy. The basic assumption is that blogs are nothing but a data store.  While information in a blog makes for an interesting read, it is about as interesting as reading data in a text column in a relational database.  While the data in a single text column may have a lot of meaning, its meaning and usefulnes is enhanced when the data is combined with other columns in the same table in database, or with other tables in the same database, or even with data in other databases. The wealth of data is hidden in its interconnections with other data. In order to harvest the wealth of data in databases, applications are built on top of the databases that reference and make relational semantic inferences between the data in the database(s).  Today, blogs are the database(s). What is lacking are the applications that harvest the wealth of information stored in the blogs.  These are the applications that the next wave of Web 2.0 companies (including myself) are working on. 

The pace of these next generation applications is being hindered by the lack of a consistent structure (standard) in blog data. What Isikold is bringing attention to is that unlike relational databases, which adhere to relational database management system standard (characterized by a simple TABLE/COLUMN/ROW+SQL structure that has been consistent over the years), blogs have no such standard. The structure of blogs is currently left up to the blogging platforms such a WordPress, Typepad etc. Blogging standards today are akin to having Oracle, SQL Server, MySQL each using a different standard for storing and retrieving information. Not only a different a standard for each of the databases, but a different standard for each version of each database.  Exacerbating the problem further, each of the different databases being customizable by anyone and anyone can change the standard to a standard of their liking. If these databases were is such a state, it would be very difficult to write any applications that leverage data from these databases. ODBC and JDBC standards would be very unreliable, if not useless.  Such is the state of the blogosphere today when one looks at it from a data interface perspective.  

As many of you know, I am currently devoted to work on the layer of applications that leverages the data in blogs and beyond in order make such data more useful to users.  The lack of standardization (as described above) makes it difficult to identify the content in blogs.  Content identification is important because an application needs to be able to identify the difference between actual blog post text and some other text on the blog so that analyses and inferences can be established appropriately.  I have been monitoring the different types of templates in an attempt to predict template patterns for the different blogging platforms (mainly WordPress, TypePad, Blogger, MovableType).  I came to the conclusion that pattern prediction is only successful to a certain point due to the following

1) the original templates from the blogging platform vendor consists of multiple major and minor versions that do not have a predictable consistency in the template content tagging and

2) there are modified/hand coded templates floating out there which are totally unreliable.

As a result of these observations, I have resorted to writing my own content identification algorithms that include a combination of template pattern predictor algorithms and NLP based semantic blog post text identification algorithms.  While this has served me well up to now, a blog template standard will be very beneficial not only to myself but many people who have not figured out how get past the problem.  

Isikold is suggesting that a standard be adopted with the goal of giving blog templates a consistent structure.  This means the adoption of a template standard that identifies the different types of data on the different parts of bogs post. Isikold is suggesting that on a blog post, the template should make it easy to identify the blog post text, the side bar, the name of the author, the data that blog post was published, the tags for the blog post content and the blog posts comments.  I believe an adoption of this simple template will go a long way in helping to bring the next wave of Web 2.0 applications to market faster.  I support a blog template standard.

The applications that I am working on are reliant on some calculations using RSS dates.  I have noticed that the RSS date specification is probably the most taken for granted part of the RSS spec.  It is taken for granted because many consumers of RSS program around the date inconsistencies so there is not much of an outcry.  However, when you see a date like 01/02/07, you have to stop and say something. 

To those developers generating RSS feeds, please take a look at the RSS date format specifications as per the RSS specification.  I will summarize it here: 

The RSS date must conform to the RFC-822 (refer to the BNF for “date-time”  in section 5) date time format.  Examples of this format are: 

Wed, 04 Feb 2008 08:00:00 EST

Wed, 04 Feb 2008 13:00:00 GMT

Wed, 04 Feb 2008 15:00:00 +0200 

Do not just execute a stringifying method on your date object before writing it to the RSS feed.  Set the date format to the above mentioned format first before writing it to the RSS feed. 

To validate whether your date is correct, you can use http://feedvalidator.org