Think Different

Thursday

Face Book Open Source Technologies

Introduction

Facebook has been developed from the ground up using open source software. Developers building with Platform scale their own applications using many of the same infrastructure technologies that power Facebook.

Platform

Our Platform engineering team has released and maintains open source SDKs for Android, C#, iPhone, JavaScript, PHP, and Python.

Developer tools

codemod assists with large-scale codebase refactors that can be partially automated but still require human oversight and occasional intervention.

Facebook Animation is a JavaScript library for creating customizable animations using DOM and CSS manipulation.

flvtool++ is a tool for hinting and manipulating the metadata of FLV files. It was originally created for Facebook Video.

Online Schema Change for MySQL lets you alter large database tables without taking your cluster offline.

PHPEmbed makes embedding PHP truly simple for all of our developers (and indeed the world) we developed this PHPEmbed library which is just a more accessible and simplified API built on top of the PHP SAPI.

phpsh provides an interactive shell for PHP that features readline history, tab completion, and quick access to documentation. It is ironically written mostly in Python.

Three20 is an Objective-C library for iPhone developers which provides many UI elements and data helpers behind our iPhone application.

XHP is a PHP extension which augments the syntax of the language such that XML document fragments become valid expressions.

XHProf is a function-level hierarchical profiler for PHP with a simple HTML-based navigational interface.

Infrastructure

Apache Cassandra is a distributed storage system for managing structured data that is designed to scale to a very large size across many commodity servers, with no single point of failure.

Apache Hive is data warehouse infrastructure built on top of Hadoop that provides tools to enable easy data summarization, adhoc querying and analysis of large datasets.

FlashCache is a general purpose writeback block cache for Linux. It was developed as a loadable Linux kernel module, using the Device Mapper and sits below the filesystem.

HipHop for PHP transforms PHP source code into highly optimized C++. HipHop offers large performance gains and was developed over the past two years.

Scribe is a scalable service for aggregating log data streamed in real time from a large number of servers.

Thrift provides a framework for scalable cross-language services development in C++, Java, Python, PHP, and Ruby.

Tornado is a relatively simple, non-blocking web server framework written in Python. It is designed to handle thousands of simultaneous connections, making it ideal for real-time Web services.

Engineers contribute to

Apache Hadoop provides reliable, scalable, distributed computing infrastructure which we use for data analysis.

Cfengine is a rule-based configuration system that is used to automate the config and maintenance of servers. Facebook uses Cfengine to maintain host configs and to automate many janitorial operations on our production tiers.

memcached is a distributed memory object caching system. Memcached was not originally developed at Facebook, but we have become the largest user of the technology.

MySQL is the backbone of our database infrastructure. You can find our patches on Launchpad and learn more about how we use it on the MySQL@Facebook page.

PHP is an incredibly popular scripting language which makes up the majority of our code-base. Its simple syntax lets us move fast and iterate on products.

Varnish serves billions of requests every day to Facebook users around the world. Whenever you load photos and profile pictures of your friends, there's a very good chance that Varnish is involved.

Mirror

We host a public mirror for projects such as Apache, Centos, CPAN, Fedora, GNU, Mozilla, MySQL, and much more...

Tuesday

PHP for Android Project Launched

irontec have just launched an open source project to bring PHP to Android platform. PHP for Android project (PFA) aims to make PHP development in Android not only possible but also feasable providing tools and documentation. The project already have an APK which provides PHP support to Android Scripting Environment (ASE). To get started you can follow the screencast below :

APK and source code both available at http://phpforandroid.net. Minimum requirement to get PHP for Android running is Android 1.5 phone or emulator. There is even an unofficial ASE build with PHP 5.3 support included. Now Rasmus can get an Android phone and start scripting on mobile.

Getting Started with iPFaces PHP Mobile Application Framework

iPFaces is a flexible solution for easy development of form-oriented network mobile applications. With the iPFaces solution, mobile devices are able to render content received from a server using their native UI components. It uses thin presentation client (must be installed on device) to render application content. Using iPFaces it is possible to build an application where users can use their device's specific component behavior and additional device features, such as location service and additional graphic components of the device (lists, pickers etc.).

Architecture

The solution is based on the use of a thin presentation client installed on the device and an application/web server which generates the content for clients. The client and the server communicate with each other using the network.

The idea is similar to the web browser - web server model. The client sends HTTP(S) requests to the server and receives iPFaces specific HTTP(S) responses, where the content is an XML representation of the application's form which is be rendered on client-side together with the form's data.

How to start?

Development of a complex iPFaces application is really simple because the simulation mode can be used. This mode is capable of transfering XML content to a HTML page, which can be displayed in a web browser. It is a helpful tool for developers who can see their iPFaces application in the browser window and they do not need a real iPhone device for main development.

Developers can build and deploy an application to the application server and the browser will show them the GUI which is almost the same as a screen in an iPhone application. There is a difference in the GPS elements. A GPS field is working in a browser only as text field that can be filled by user and GPS coordinates will be submitted. The GPS field is hidden on mobile devices, because the location of the device is detected without interaction with the user.

Figure 1: Form representation on the iPhone device and in a web browser

Hello World Example

To use PHP iPFaces library, just include "ipfaces-php-lib-1.1.php" file, construct the component tree and call "render()" method on the component form.

view plain print ?

require "path/to/ipfaces/library/ipfaces-php-lib-1.1.php";
$ipf_form = new IPFForm();
$ipf_screen = $ipf_form->addScreen("screen", "Hello World Application");
$ipf_screen->addLabel("label", "Hello World!");
$ipf_form->render();

Figure 2: Hello World example

More complex example

The following example is not complete. It only illustrates how easy it is to define forms for iPhone using iPFaces. For complete examples please visit http://www.ipfaces.org

Example: Use of location service

To obtain a user location from a mobile device use the IPFGsm class. Upon submission of a form, the location data will be sent as a parameter with the selected name (gpsElement in this example).

require "../lib/ipfaces-php-lib-1.1.php";
require "citydatabase.php";
$form = new IPFForm();
$screen = $form->addScreen("How Far Is It?");
$screen->addLabel("You can find distance between your position and selected city.");
$parser = new CityDatabase();
if (isset($_COOKIE["city"])){
$value = $_COOKIE["city"];
}
$select = new IPFSelect("citySelect", $value, "Distance to" ,"list" );
$options = array();
for($row = 0; $row < count($parser->data); $row++){
$options[] = new IPFOption($row, $parser->data[$row]["City"]);
}
$select->Icon = "../img/distcalc.png";
$select->addOptions($options);
$screen->addItem($select);
$screen->addGps("gpsElement");
$screen->addButton("backButton", "1", "Examples", "../index.php", IPFButton::BUTTON_TYPE_LINK, IPFButton::BUTTON_POSITION_BACK);
$screen->addButton("submitButton", "0", "Calculate" , "distance.php", IPFButton::BUTTON_TYPE_SUBMIT, IPFButton::BUTTON_POSITION_FORWARD);
$form->render();

curtsy: Hatem

Monday

Scaling the BBC iPlayer to handle demand with PHP

One of the key goals we set ourselves when we developed the new iPlayer was that it would have to be fast to use. We understand that any delay in getting you to the video is frustrating as the site is just a jumping off point into TV and Radio content.

But how do we make things fast? Displaying a web page in the browser contains many steps, some we can control some we can't. Time spent for the request and response travelling over the network we can't control, but we can control how long the pages take to generate and how large they are. We also have a degree of control over how long those pages can take to render in your browser.

We had our work cut out for us on the new version of iPlayer.

Personalised websites require much more processing power and data storage

The current site uses one back-end service that we pull data from to build the pages. The new site uses many more, and we both post and pull data from them.

This means that every returning user gets a different homepage. There's already a small amount of difference between each homepage on our current site (your recently played) but the new site is driven much more by your favourites, recommendations and friends; they're key parts of the experience and they have to be fast.

We started developing in PHP

The BBC is standardising on PHP as its web tier development tool. Our current site is developed using Perl and Server Side Includes, and it's something that's well understood, but our new web tier framework (based on Zend) means that teams can share components and modules. In fact, the team responsible for the social networking functionality develop modules that anyone within the BBC can integrate into their site easily.

This does come at a cost though: the usage of a framework sometimes introduces delay in generating a page as it needs to get hold of resources to do so. In some cases this is necessary, especially if there's an element of personalisation, but in others our web tier is just repeating the same tasks.

All this against a growing demand

The site will have to support a massive amount of page views and users every day, on average 8 million a day for 1.3 million users. Previous versions of the site were able to grow into this demand; we'll have to hit the ground running from day one.

This graph shows our growth over the last year in terms of monthly page views.

So how do we do this?

One of the first things we can do is optimise the time it takes to generate the page.

Although changing architectures can be risky, we were confident that the one we moved to would enable us to meet all the challenges. At the heart of page generation is a PHP and customised Zend-based layer called PAL. This system then needs to integrate with our login system, BBC iD, our programme metadata system (Dynamite), our social networking systems, a Key Value data store and a few others. The homepage alone for a logged-in user with friends requires 15 calls across these services. Even if each of those calls take a few milliseconds, we can spend a second or two just collecting the information required, which would push us well out of our 2.5s target.

We proved our architecture before we built it

At the start of re-architecting iPlayer, we did what we could to eliminate guesswork. We developed a number of architectures based on our requirements, and then built prototypes of three of them; all built to serve the homepage, which we then tested against some basic volumetrics. This gave us plenty of data about how many requests we could serve a second and CPU loads, which we could then weigh up against other softer factors, like how our dev team could work with it.

We actually ended up going for the one which offered us a good balance between these factors, as this enabled us to be the most flexible in building pages, rather than constraining what we could with the site just to squeeze the extra speed out.

We cache a lot

Caching means storing a copy of the data in memory so subsequent requests for that data don't have to do the expensive things such as database queries.

It also allows us to get around any delays introduced by our framework starting up, as there's no such delay when delivering from cache.

Caching has its problems though. The data may have changed in the underlying system (programmes become available to play for example) but the change won't be reflected in our cache. This means we can only cache for seconds or minutes, but with the millions of page views we get, it can still make a crucial difference.

Data caching We cache the data returned from the services. We use Memcached for this. Sometimes we share data between pages.
HTML caching We also cache the resulting HTML for a short time. When you're hitting a page, it's highly likely you're just seeing the cached page. We use Varnish for this. Caching in this way is nothing new, but Varnish has a few tricks up its sleeve that we use which I'll explain later.

We broke the page into personalised and standard components

If you look at our homepage, many of those components are the same for everyone, but some are just for you. With traditional page caching in some reverse HTML caches, it's not possible to do this; so we break the page up. The main build of the page is cached; then when the page loads we use XHR and Ajax to load in the personalised components. Varnish gives us the ability to control the caching at a low-level like this. Every time we generate a page or a fragment, we can tell Varnish how long we want to cache it for. The main bulk of the homepage doesn't need caching for long to get some benefit, but your favourites we can cache for longer (although still only for a few minutes), and we know when you add a new favourite so we can clear out the cache and replace it with the new content. This means as you browse the site, the page loads quicker and your experience is smoother.

We use loads of servers

After we've optimised all we can using a single server, we then scale horizontally using multiple servers joined together in a pool. None of our web servers store any state about who you are and what you're doing, so your request can go to any server at any time.

We also serve pages out of two locations (or data centres). This gives us a higher degree of resilience to failures; we can lose an entire data centre and still be able serve the site.

We load tested the site before we launched

We're able to track how the site is used, so this gives us the ability to produce detailed volumetrics of how we think the new site is going to be used. Some of it is estimation, but it's always backed up with data. We can then produce detailed load tests, so we can simulate usage of the site. This enables us to find and resolve any problems we may experience under load, before we go live.

The end result

We're not 100% there yet (this is a beta after all) but from this sample 24 hours of monitoring data you can see that, apart from a couple of spikes, we're doing well at keeping to our target of 2.5 seconds. (We were also able to track down the spikes to some misbehaving components on the platform).

We're currently working hard behind the scenes at making sure we can continue to serve at this speed as usage increases, spreading the load across our infrastructure.

At the end of this though, we hope the result of our efforts is that you won't notice a thing: it'll just work.

curtsy: Simon Frost is Technical Architect for BBC iPlayer .

Tuesday

PHP's Place in the Enterprise

PHP in the enterprise

PHP claims to be the most widely used programming language on the web. A quick look at http://langpop.com/ supports this – it’s almost certainly the most common for smaller web projects. PHP was not originally designed as an enterprise-level language, but as it has evolved, it has become suitable for much larger projects than were originally envisaged when Rasmus Lerdorf produced PHP/FI in 1995 (source). PHP now supports SOAP, XML-RPC, JSON and any database platform you care to mention.

For something to be considered “Enterprise-level”, i.e. ready for use in the enterprise, it should meet the criteria of the Enterprise Challenges examined in our previous blog post.

PHP as an Enterprise-level language

With PHP 5.3, PHP is a full object oriented language with exception handling and useful features such as closures. Let’s take a look at PHP in light of enterprise challenges.

Scalability

PHP is very scalable owing to its shared nothing architecture – Facebook, Yahoo and Flickr, for example, are huge apps that tackle many Enterprise problems such as scalability, security and robustness using PHP.

Whilst PHP does not itself explicitly support concurrency (each PHP request is a single request and response), its host server can handle the instantiation of requests. PHP is very fast, and can horizontally scale extremely easily, often as simply as adding instances to a cluster of servers. This means the data throughput and concurrent users challenges can be overcome.

Robustness

Because each request runs in isolation, PHP is less likely to become deadlocked than a multithreaded language such as Java. There are tradeoffs, however – Java’s sophisticated mutual exclusivity functionality can protect vital areas of data, whereas two PHP processes may update them at the same time. This can push a certain amount of work onto the database, but as such, is easily solved (and, some may argue, may be better handled by the database system anyway).

Frameworks

Enterprise application development can be expedited with sophisticated frameworks to abstract away tasks, help organise code, and provide functionality rather than re-inventing the wheel. There are frameworks available for PHP, the most prominent being Zend Framework. Zend Framework is robust, thoroughly unit tested and can be a strong platform for enterprise applications. Jim Plush, a senior developer at Panasonic, blogs about his experiences with it. eZ Components is another enterprise framework for PHP focusing on providing re-usable components.

Security

Compared with other languages, PHP is neither particularly more or less secure; out of the box it may lack much of the explicit security support provided in .Net, for example, but one of PHP’s main strengths is the wealth of libraries available through PEAR and PECL, providing a range of simple and sophisticated security options.

Summary

PHP warrants its burgeoning place in Enterprise application development, but should be used where appropriate, wherever possible taking advantage of existing libraries and frameworks rather than rewriting. As web developers, as our medium moves to a more social environment, we can learn much from the established enterprise frameworks and applications that are already serving millions of users.

Curtsy: Posted by Gavin Davies on 18th Dec 2009

Monday

Dynamic content for BBC

Matt McDonnell wrote about the new BBC Topic Pages Beta. I'd now like explain how some of the many components that build those pages all work together.

The point of the Topic Pages is that they bring together content from all around bbc.co.uk. Obviously, many different systems produce all that content, and in general they don't tend to share content very well. Our challenge was to build a platform that could make sense of the different interfaces to those systems to make sharing that content easier.

The first thing to note is that the Topic Pages themselves are dynamic, unlike the vast majority of pages on bbc.co.uk. Essentially, this means that the HTML of the page isn't stored as a physical file on a hard-disk, but instead is built up dynamically when the page is requested.

This is done by the "Page Assembly Layer" or "PAL", a brand new component written in the PHP programming language. In the future, the intention is that most pages on bbc.co.uk will be produced dynamically using the PHP layer, and the Topic Pages system has blazed a trail, being the first released on this new platform.

The PAL itself does a fairly simple job, in principle. First, it receives a request for a Topic Page; it then looks up which modules (ie, the different blocks of content on the page, such as BBC News, Programmes and Weather) it needs to build that page; it grabs all those different modules, which originate on various different systems and finally it assembles them before returning the page to the user.

The really important part here is that the PAL is grabbing all the useful content dynamically, and not storing any content itself (apart from a bit of caching, to help smooth out any spikes in load). This means that the PAL is a really generic system that can be used for building other sorts of pages as well.

The PAL actually requests the various content modules from another component, which itself routes these requests on to the underlying systems. This Module Routing system is implemented in Apache Cocoon, an open source framework, released by the Apache Software Foundation. This way, the PAL can access content through a simple, uniform interface (which is based on REST principles, for those who are into such things), rather than having to deal directly with many complex interfaces to multiple systems.

This gives us two big benefits. First, if we want to change how a particular module is implemented, we can just reconfigure the Module Routing, and don't have to alter the PAL (which we don't want to tinker with too much, as it is busy serving pages); second, it makes it easier for any other system, or other page on bbc.co.uk, to reuse the modules. This model also makes it easier to add new content modules to the PAL, as most of the logic about how to access a new module can be handled in the Module Routing layer.

As Matt said, "Topics are automatically updated web pages, each one covering a different person, country or subject." We wanted to present content from across the various departments within the BBC. The Search Engines are the one place that all that content from bbc.co.uk (well, most of that content) is brought together (nearly 2 million documents from www.bbc.co.uk and the Newsand Sport sites). So it makes a lot of sense to create many of the modules from the Search indexes (the News module, and the Programmes module, for example, are both created like this). This is done using elaborate search queries, which are a bit fancier than the one- or two-keyword queries which the Search Engines normally receive - the query for the Fashion topic, for example, contains 364 terms! These queries are built using a variety of techniques by the Search Editorial Team.

We also use content from other sources than the Search indexes - for example, the Weather modules, and the information boxes about countries, which come from the BBC News website. In order to make it as easy as possible to share and reuse these content modules, we are attempting to create some standards for the formats in which the data is passed around (rather than just allowing every system to specify its own format).

Having looked around a bit, we decided to base our data structure on the Atom format, which was originally created to describe feeds from Blogs. Atom is a rich XML format, and one of its most attractive features is that you can add your own data elements. This means we can have a standard container for our data, but also include extra elements where appropriate (such as in the weather modules, to denote temperatures, for example).

This approach is similar to Google's GDATA, which is also based on ATOM. We have a prototype of this XML format - called BBC Module XML - in use in the Topics pages, and in the next few months we will be looking at refining and improving it, and hopefully making it useful for other systems across bbc.co.uk. We use XSLT to transform from the XML to the nice XHTML modules which are placed on the page.

The only parts that I haven't touched upon so far are the admin systems for creating and maintaining the pages and also the modules that go on them. At the moment, these admin systems aren't as joined up as they should be (and are implemented in different programming languages in some cases). This is a headache for the editorial team that maintain the Topic Pages. Our next priority is figuring out how we can bring those systems closer together, and generally improve the workflow for the editorial staff, so that they can easily add more Topic Pages.

Once we've made their lives a little easier, we will look at more feature enhancements. These include providing RSS feeds of the Topic Pages so that people can more easily stay up to date with their favourite topics. Additionally, we want to improve our systems for sharing metadata, so that it will be easier to automatically link to relevant Topic Pages from other pages on bbc.co.uk. And we will also add more types of content modules, to increase the range of content on the Topic Pages.

As you hopefully can see, the Topic Page project was pretty complex, and involved creating many new systems. Wherever possible, we have developed those systems to be generic and extensible, to provide not just Topic Pages, but also a platform for sharing and reusing content, and building other products in the future. This has all been possible thanks to some fantastic, far-sighted and occasionally frenetic work from the BBC Search Team and colleagues in FM&T Journalism and FM&T Internet - thanks to all of them.

N.B. More information on /topics can be found here.

Stephen Betts is Search Technical Team Leader, BBC Future Media & Technology

courtsy: bbc

Friday

BBC World Cup 2010 dynamic semantic publishing

The World Cup 2010 website is a significant step change in the way that content is published. From first using the site, the most striking changes are the horizontal navigation and the larger, format high-quality video. As you navigate through the site it becomes apparent that this is a far deeper and richer use of content than can be achieved through traditional CMS-driven publishing solutions.

The site features 700-plus team, group and player pages, which are powered by a high-performance dynamic semantic publishing framework. This framework facilitates the publication of automated metadata-driven web pages that are light-touch, requiring minimal journalistic management, as they automatically aggregate and render links to relevant stories.

Dynamic aggregation examples include:

The underlying publishing framework does not author content directly; rather it publishes data about the content - metadata. The published metadata describes the world cup content at a fairly low-level of granularity, providing rich content relationships and semantic navigation. By querying this published metadata we are able to create dynamic page aggregations for teams, groups and players.

The foundation of these dynamic aggregations is a rich ontological domain model. The ontology describes entity existence, groups and relationships between the things/concepts that describe the World Cup. For example, "Frank Lampard" is part of the "England Squad" and the "England Squad" competes in "Group C" of the "FIFA World Cup 2010".

The ontology also describes journalist-authored assets (stories, blogs, profiles, images, video and statistics) and enables them to be associated to concepts within the domain model. Thus a story with an "England Squad" concept relationship provides the basis for a dynamic query aggregation for the England Squad page "All stories tagged with England Squad".

This diagram gives a high-level overview of the main architectural components of this domain-driven, dynamic rendering framework.

The journalists use a web tool, called 'Graffiti', for the selective association - or tagging - of concepts to content. For example, a journalist may associate the concept "Frank Lampard" with the story "Goal re-ignites technology row".

In addition to the manual selective tagging process, journalist-authored content is automatically analysed against the World Cup ontology. A natural language and ontological determiner process automatically extracts World Cup concepts embedded within a textual representation of a story. The concepts are moderated and, again, selectively applied before publication. Moderated, automated concept analysis improves the depth, breadth and quality of metadata publishing.

Journalist-published metadata is captured and made persistent for querying using the resource description framework (RDF) metadata representation and triple store technology. A RDF triplestore and SPARQL approach was chosen over and above traditional relational database technologies due to the requirements for interpretation of metadata with respect to an ontological domain model. The high level goal is that the domain ontology allows for intelligent mapping of journalist assets to concepts and queries. The chosen triplestore provides reasoning following the forward-chaining model and thus implied inferred statements are automatically derived from the explicitly applied journalist metadata concepts. For example, if a journalist selects and applies the single concept "Frank Lampard", then the framework infers and applies concepts such as "England Squad", "Group C" and "FIFA World Cup 2010" (as generated triples within the triple store).

This inference capability makes both the journalist tagging and the triple store powered SPARQL queries simpler and indeed quicker than a traditional SQL approach. Dynamic aggregations based on inferred statements increase the quality and breadth of content across the site. The RDF triple approach also facilitates agile modeling, whereas traditional relational schema modeling is less flexible and also increases query complexity.

Our triple store is deployed multi-data center in a resilient, clustered, performant and horizontally scalable fashion, allowing future expansion for additional ontologies and indeed linked open data (LOD) sets.

The triple store is abstracted via a JAVA/Spring/CXF JSR 311 compliant REST service. The REST API is accessible via HTTPs with an appropriate certificate. The API is designed as a generic façade onto the triplestore allowing RDF data to be re-purposed and re-used pan BBC. This service orchestrates SPARQL queries and ensures that results are dynamically cached with a low 'time-to-live' (TTL) (1 minute) expiry cross data center using memcached.

All RDF metadata transactions sent to the API for CRUD operations are validated against associated ontologies before any persistence operations are invoked. This validation process ensures that RDF conforms to underlying ontologies and ensures data consistency. The validation libraries used include Jena Eyeball. The API also performs content transformations between the various flavors of RDF such as N3 or XML RDF. Example RDF views on the data include:

Automated XML sports stats feeds from various sources are delivered and processed by the BBC. These feeds are now also transformed into an RDF representation. The transformation process maps feed supplier ids onto corresponding ontology concepts and thus aligns external provider data with the RDF ontology representation with the triple store. Sports stats for matches, teams and players are aggregated inline and served dynamically from the persistent triple store.

The following "Frank Lampard" player page includes dynamic sports stats data served via SPARQL queries from the persistent triple store:

The dynamic aggregation and publishing page-rendering layer is built using a Zend PHP and memcached stack. The PHP layer requests an RDF representation of a particular concept or concepts from the REST service layer based on the audience's URL request. If an "England Squad" page request is received by the PHP code several RDF queries will be invoked over HTTPs to the REST service layer below.

The render layer will then dynamically aggregate several asset types (stories, blogs, feeds, images, profiles and statistics) for a particular concept such as "England Squad". The resultant view and RDF is cached with a low TTL (1 minute) at the render layer for subsequent requests from the audience. The PHP layer dynamically renders views based on HTTP headers providing content negotiated HTML and/or RDF for each and every page.

To make use of the significant number of existing static news kit and architecture (apache servers, HTTP load balancers and gateway architecture) all HTTP responses are annotated with appropriate low (1 minute) cache expires headers. This HTTP caching increases the scalability of the platform and also allows content delivery network caching (CDN) if demand requires.

This dynamic semantic publishing architecture has been serving millions of page requests a day throughout the World Cup with continually changing OWL reasoned semantic RDF data. The platform currently serves an average of a million SPARQL queries a day with a peak RDF transaction rate of 100s of player statistics per minute. Cache expiry at all layers within the framework is 1 minute proving a dynamic, rapidly changing domain and statistic-driven user experience.

The development of this new high-performance dynamic semantic publishing stack is a great innovation for the BBC as we are the first to use this technology on such a high-profile site. It also puts us at the cutting edge of development for the next phase of the Internet, Web 3.0.

So what's next for the platform after the World Cup? There are many engaging expansion possibilities: such as extending the World Cup approach throughout the sport site; making BBC assets geographically 'aware' is another possibility; as is aligning news stories to BBC programs. This is all still to be decided, but one thing we are certain of is that this technological approach will play a key role in the creation, navigation and management of over 12,000 athletes and index pages for the London 2012 Olympics.

Jem Rayfield is Senior Technical Architect, BBC News and Knowledge. Read the previous post on the Internet blog that covers the BBC World Cup website, The World Cup and a call to action around Linked Data.

Metadata is data about data - it describes other data. In this instance, it provides information about the content of a digital asset. For example, a World Cup story may include metadata that describes which football players are mentioned within the text of a story. The metadata may also describe the associated team, group or organization associated to the story.

IBM LanguageWare Language and ontological linguistic platform.

RDF is based upon the idea of making statements about concepts/resources in the form of subject-predicate-object expressions. These expressions are known as triples in RDF terminology. The subject denotes the resource; and the predicate denotes traits or aspects of the resource and expresses a relationship between the subject and the object. For example, to represent the notion "Frank Lampard plays for England" in RDF is as a triple, the subject is "Frank Lampard"; the predicate is "plays for" and the object is "England Squad".

SPARQL (pronounced "sparkle") is an RDF query language its name is a recursive acronym (i.e. an acronym that refers to itself) that stands for SPARQL Protocol and RDF Query Language.

BigOWLIM A high performance, scalable, resilient triplestore with robust OWL reasoning support

LOD The term Linked Open Data is used to describe a method of exposing, sharing, and connecting data via dereferenceable URIs on the Web.

JAVA Object-orientated programming language developed by Sun Microsystems.

Spring Rich JAVA framework for managing POJOs providing facilities such as inversion of control (ioc) and aspect orientated programming

Apache CXF JAVA Web services framework for JAX-WS and JAR-RS

JSR 311 Java standard specification API for RESTful web services.

Memcached Distributed memory caching system (deployed multi datacenter)

Jena Eyeball JAVA RDF validation library for checking ontological issues with RDF

N3 Shorthand textual representation of RDF designed with human readability in mind.

XML RDF XML representation of an RDF graph.

XML (Extensible Markup Language) is a set of rules for encoding documents and data in machine-readable form

Zend Open source scripting virtual machine for PHP, facilitating common programming patterns such as model view controller.

PHP Hypertext Preprocessor general-purpose dynamic web scripting language, use to create dynamic web pages.

CDN A content delivery network or content distribution network (CDN) is a collection of computers usually hosted within Internet Service Provider hosting facilities. The CDN servers cache local copies of content to maximize bandwidth and reduce requests to origin servers.

OWL Web Ontology Language (OWL) is a family of knowledge representation languages for authoring ontologies.

courtsy: BBC Jem Rayfield