Saturday

The future of PHP

PHP is already popular, used in millions of domains (according to Netcraft), supported by most ISPs and used by household-name Web companies like Yahoo! The upcoming versions of PHP aim to add to this success by introducing new features that make PHP more usable in some cases and more secure in others. Are you ready for PHP V6? If you were upgrading tomorrow, would your scripts execute just fine or would you have work to do? This article focuses on the changes for PHP V6 — some of them back-ported to versions PHP V5.x — that could require some tweaks to your current scripts.

If you're not using PHP yet and have been thinking about it, take a look at its latest features. These features, from Unicode to core support for XML, make it even easier for you to write feature-filled PHP applications.

New PHP V6 features

PHP V6 is currently available as a developer snapshot, so you can download and try out many of the features and changes listed in this article. For features that have been implemented in the current snapshot, see Resources.

Improved Unicode support

Much improved for PHP V6 is support for Unicode strings in many of the core functions. This new feature has a big impact because it will allow PHP to support a broader set of characters for international support. So, if you're a developer or architect using a different language, such as the Java™ programming language, because it has better internationalization (i18n) support than PHP, it'll be time to take another look at PHP when the support improves.

Because you can download and use a developer's version of PHP V6 today, you will see some functions already supporting Unicode strings. For a list of functions that have been tested and verified to handle Unicode, see Resources.

What is Unicode?

Unicode is an industry-standard set of characters, character encoding, and encoding methodologies primarily aimed at enabling i18n and localization (i10n). The Unicode Transformation Format (UTF) specifies a way to encode characters for Unicode. For more information about Unicode and UTF, see Resources.

Namespaces

Namespaces are a way of avoiding name collisions between functions and classes without using prefixes in naming conventions that make the names of your methods and classes unreadable. So by using namespaces, you can have class names that someone else might use, but now you don't have to worry about running into any problems. Listing 1 provides an example of a namespace in PHP.

You won't have to update or change anything in your code because any PHP code you write that doesn't include namespaces will run just fine. Because the namespaces feature appears to be back-ported to V5.3 of PHP, when it becomes available, you can start to introduce namespaces into your own PHP applications.


Listing 1. Example of a namespace
                  

Web 2.0 features

Depending on how you use PHP and what your scripts look like now, the language and syntax differences in PHP V6 may or may not affect you as much as the next features, which are those that directly allow you to introduce Web 2.0 features into your PHP application.

SOAP

SOAP is one of the protocols that Web services "speak" and is supported in quite a few other languages, such as the Java programming language and Microsoft® .NET. Although there are other ways to consume and expose Web services, such as Representational State Transfer (REST), SOAP remains a common way of allowing different platforms to have interoperability. In addition to SOAP modules in the PHP Extension and Application Repository (PEAR) library, a SOAP extension to PHP was introduced in V5. This extension wasn't enabled by default, so you have to enable the extension or hope your ISP did. In addition, PEAR packages are available that allow you to build SOAP clients and servers, such as the SOAP package.

Unless you change the default, the SOAP extension will be enabled for you in V6. These extensions provide an easy way to implement SOAP clients and SOAP servers, allowing you to build PHP applications that consume and provide Web services.

If SOAP extensions are on by default, that means you won't have to configure them in PHP. If you develop PHP applications and publish them to an ISP, you may need to check with your ISP to verify that SOAP extensions will be enabled for you when they upgrade.

XML

As of PHP V5.1, XMLReader and XMLWriter have been part of the core of PHP, which makes it easier for you to work with XML in your PHP applications. Like the SOAP extensions, this can be good news if you use SOAP or XML because PHP V6 will be a better fit for you than V4 out of the box.

The XMLWriter and XMLReader are stream-based object-oriented classes that allow you to read and write XML without having to worry about the XML details.


Things removed

In addition to having new features, PHP V6 will not have some other functions and features that have been in previous versions. Most of these things, such as register_globals and safe_mode, are widely considered "broken" in current PHP, as they may expose security risks. In an effort to clean up PHP, the functions and features listed in the next section will be removed, or deprecated, from PHP. Opponents of this removal will most likely cite issues with existing scripts breaking after ISPs or enterprises upgrade to PHP V6, but proponents of this cleanup effort will be happy that the PHP team is sewing up some holes and providing a cleaner, safer implementation.

Features that will be removed from the PHP version include:

  • magic_quotes
  • register_globals
  • register_long_arrays
  • safe_mode

magic_quotes

Citing portability, performance, and inconvenience, the PHP documentation discourages the use of magic_quotes. It's so discouraged that it's being removed from PHP V6 altogether, so before upgrading to PHP V6, make sure that all your code avoids using magic_quotes. If you're using magic_quotes to escape strings for database calls, use your database implementation's parameterized queries, if they're supported. If not, use your database implementation's escape function, such as mysql_escape_string for MySQL or pg_escape_string for PostgreSQL. Listing 2 shows an example of magic_quotes use.


Listing 2. Using magic_quotes (discouraged)
                  

After preparing your PHP code for the new versions of PHP, your code should look like that in Listing 3.


Listing 3. Using parameterized queries (recommended)
                 prepare("INSERT INTO USERS (USERNAME) VALUES ?"); $statement->execute(array($_GET['username'])); ?> 

Now that support for magic_quotes will be completely removed, the get_magic_quotes_gpc() function will no longer be available. This may affect some of the older PHP scripts, so before updating, make sure you fix any locations in which this functions exists.

register_globals

The register_globals configuration key was already defaulted to off in PHP V4.2, which was controversial at the time. When register_globals is turned on, it was easy to use variables that could be injected with values from HTML forms. These variables don't really require initialization in your scripts, so it's easy to write scripts with gaping security holes. The register_globals documentation (see Resources) provides much more information about register_globals. See Listing 4 for an example of using register_globals.


Listing 4. Using register_globals (discouraged)
                  

If your PHP code uses global variables, you should update it. If you don't update your code to get prepared for newer versions of PHP, consider updating it for security reasons. When you're finished, your code should look like Listing 5.


Listing 5. Being specific instead (recommended)
                  

register_long_arrays

The register_long_arrays setting, when turned on, registers the $HTTP_*_VARS predefined variables. If you're using the longer variables, update now to use the shorter variables. This setting was introduced in PHP V5 — presumably for backward-compatibility — and the PHP folks recommend turning it off for performance reasons. Listing 6 shows an example of register_long-arrays use.


Listing 6. Using deprecated registered arrays (discouraged)
                  

If your PHP code looks like that shown in Listing 6, update it to look like that in Listing 7. Shut off the register_long_arrays setting if it's on and test your scripts again.


Listing 7. Using $_GET (recommended)
                  

safe_mode

The safe_mode configuration key, when turned on, ensures that the owner of a file being operated on matches the owner of the script that is executing. It was originally a way to attempt to handle security when operating in a shared server environment, like many ISPs would have. (For a link to a list of the functions affected by this safe_mode change, see Resources.) Your PHP code will be unaffected by this change, but it's good to be aware of it in case you're setting up PHP in the future or counting on safe_mode in your scripts.

PHP tags

Microsoft Active Server Pages (ASP)-style tags — the shorter version of the PHP tags — are no longer supported. To make sure this is not an issue for your scripts, verify that you aren't using the <% or %> tags in your PHP files. Replace them with and ?>.

FreeType 1 and GD 1

The PHP team is removing support for both FreeType 1 and GD 1, citing the age and lack of ongoing developments of both libraries as the reason. Newer versions of both of these libraries are available that provide better functionality. For more information about FreeType and GD, see Resources.

ereg

The ereg extension, which supports Portable Operating System Interface (POSIX) regular expressions, is being removed from core PHP support. If you are using any of the POSIX regex functions, this change will affect you unless you include the ereg functionality. If you're using POSIX regex today, consider taking the time to update your regex functions to use the Perl-Compatible Regular Expression (PCRE) functions because they give you more features and perform better. Table 1 provides a list of the POSIX regex functions that will not be available after ereg is removed. Their PCRE replacements are also shown.


Table 1. ereg() functions and their PCRE equivalents
ereg() functionSimilar preg() function
ereg(), eregi() preg_match()
ereg_replace(), ereg_replacei() preg_replace()

PHP V5.3

Some of the features mentioned here have also been ported to PHP V5.3, which is scheduled to be released during the first quarter of 2008. You may want to upgrade to V5.3 and start using these features now, so that when you move to V6 of PHP, it'll be less of a jump. The following list of features have been back-ported to V5.3:

  • Namespaces
  • XMLReader and XMLWriter in core by default

About the author

Nathan Good lives in the Twin Cities area of Minnesota. Professionally, he does software development, software architecture, and systems administration. When he's not writing software, he enjoys building PCs and servers, reading about and working with new technologies, and trying to get his friends to make the move to open source software. He's written and co-written many books and articles, including Professional Red Hat Enterprise Linux 3, Regular Expression Recipes: A Problem-Solution Approach, and Foundations of PEAR: Rapid PHP Development.

Wednesday

Facebook looks to speed up PHP

The internally-developed HipHop for PHP source code transformer is being offered via open source


Technologists at Facebook on Tuesday are releasing a source code transformer intended to boost the performance of PHP.

In development for the past two years, the technology, called HipHop for PHP, has been used to reduce the CPU usage on Facebook Web servers by an average of about 50 percent, said Haiping Zhao, senior server engineer at Facebook.

[ InfoWorld reported on PHP 6 highlights, including internationalization, that were detailed at a technical conference late last year. ]

"The project has had a tremendous impact on Facebook. We feel the Web at large can benefit from HipHop, so we are releasing it as open source this evening in hope that it brings a new focus toward scaling large, complex Web sites with PHP," Zhao said in a statement on the Facebook Developers page.

The technology is not complete and users need to be comfortable with HipHop before trying it out, said Zhao.

HipHop, he said, technically is not a compiler. It features a code transformer, a reimplementation of PHP's runtime system and a rewrite of some common PHP extensions to boost performance optimizations.

"HipHop programmatically transforms your PHP source code into highly optimized C++ and then uses g++ to compile it," Zhao said. "HipHop executes the source code in a semantically equivalent manner and sacrifices some rarely used features -- such as eval() -- in exchange for improved performance."

PHP, Zhao said, offers benefits in terms of programmer productivity, as do other scripting languages such as Perl, Python and Ruby. But scripting languages are known to be less efficient in terms of CPU and memory usage, he said.

"Because of this, it's been challenging to scale Facebook to over 400 billion PHP-based page views every month," Zhao said.

HipHop allows developers to keep the best aspects of PHP while taking advantage of performance of C++, said Zhao.

"In total, we have written over 300,000 lines of code and more than 5,000 unit tests. All of this will be released this evening on GitHub under the open source PHP license," Zhao said.

The CEO of PHP tools maker Zend Technologies questioned how much of HipHop was Facebook-specific.

"It will be interesting to learn what HipHop concepts apply to the broad PHP community and what are specific to Facebook," said Andi Gutmans, of Zend. "We have always adapted to changes within the PHP runtime, whether these changes were made by us or by the community and will be glad to continue doing so. We believe it is important to continue to fold in new ideas and innovations into the community-based runtime."

This story, "Facebook looks to speed up PHP," was originally published at InfoWorld.com. Follow the latest developments in software development at InfoWorld.com.

Paul Krill is an editor at large at InfoWorld, focusing on coverage of application development (desktop and mobile) and core Web technologies such as HTML5, Java, and Flash.

Adobe, Zend combine Flash and PHP development

Flash Builder 4.5 for PHP helps developers build apps for Web, desktop, and mobile systems, including Apple's iOS devices. But you can use Eclipse with plug-in available along with Flex sdk to compile in Eclipse itself.


Zend Technologies and Adobe Systems today announced Flash Builder 4.5 for PHP, which enables developers to use PHP and Flash development skills to build rich Internet applications for mobile, Web, and desktop platforms.

The product provides an IDE combining Adobe's Flash Builder 4.5, for ActionScript-based Flash client development, and Zend Studio 8, for server-side PHP capabilities. Accentuating the use for mobile deployments, the two vendors said applications can be delivered to Google Android, Research in Motion BlackBerry Tablet OS, and Apple iOS systems. For iOS, the IDE exports the applications into native code, rather than run in an the AIR (Adobe Integrated Runtime) software, which Apple prohibits on iOS. Android apps are also exported to a native package.

[ Adobe is backing Android tablets with Flash Player 10.2. | Keep up on key mobile developments and with InfoWorld's Mobile Edge blog and Mobilize newsletter. | Follow Paul Krill on Twitter. ]

Flash Builder 4.5 for PHP, said Zend CEO Andi Gutmans, "makes it very easy for PHP developers to also build mobile apps, and it makes very easy for mobile developers to build server-side apps. We're coming at this from both angles." Applications themselves can be built on Windows or Mac clients.

"Developers will now be able to use one tool, one framework, and one common code base and build apps that run on Android, BlackBerry, and iOS," said Dave Gruber, Adobe group product manager. Flash Builder 4.5 for PHP is due by May 11, with the premium edition, featuring network tracking and memory and performance profilers, priced at $799. A standard edition lacking these features will cost $399. Adobe and Zend are making the announcement concurrently with Adobe's rollout of its Adobe Creative Suite 5.5 product line.

This article, "Adobe, Zend combine Flash and PHP development," was originally published at InfoWorld.com. Follow the latest developments in business technology news and get a digest of the key stories each day in the InfoWorld Daily newsletter. For the latest developments in business technology news, follow InfoWorld.com on Twitter.

Paul Krill is an editor at large at InfoWorld, focusing on coverage of application development (desktop and mobile) and core Web technologies such as HTML5, Java, and Flash.

Cloud computing underwhelms PHP developers

Developers still see it as the future, but they think it is currently overhyped and vendor-driven


While technology vendors continue to pound home the message of cloud computing, PHP developers Tuesday viewed the concept as overhyped and were not in agreement on its benefits.

Developers at the ZendCon 2010 PHP conference in Santa, Clara, Calif., heard Zend Technologies CEO Andi Gutmans tout the company's cloud computing plans, which involve developing Zend PHP Cloud Platform. During his presentation, however, developers appeared mostly underwhelmed when Gutmans asked if cloud computing was game-changing or just hype. Afterward, developers gave cloud computing mixed reviews.

[ Oracle discussed its cloud plans on Monday . | Stay on top of the latest app dev news with the Developer World newsletter. ]

"I guess I have a feeling that 10, 15 years from now, maybe we'll all be using this stuff, but right now, it's entirely pushed by vendors," said Phillip Winn, back-end developer for games builder Tapulous.

"I don't see value in it," Winn said. "I don't have a strong opinion. I don't care. It doesn't affect me."

Winn recalled a former employer who thought cloud computing could be used to cut costs and reduce staff levels. "Economically, it ended up not making any sense for them," said Winn.

Cloud computing, said attendee Chuck Hudson, founder of Aduci, a consulting firm, has been the subject of some hype. "But there's definitely some opportunity there to leverage cloud computing," with developers able to rapidly develop systems and for enterprises to save on infrastructure and maintenance costs, Hudson said.

Rather than view cloud computing as a potential job-killer, Hudson sees it as a chance for IT persons to expand horizons. "I think it's more an opportunity for people in their current roles to learn the new technology and apply it. So I think it's just retooling your toolset."

Cloud computing, said Joseph Munowenyu, computer programmer at Valley City State University, in North Dakota, is "where everything is headed." At consulting firm Atos Origin, the company does not yet use cloud computing, said Atos developer Chris Campbell. "It's something we've been looking at." But he also saw "an element of hype" to the concept.

After his presentation, Gutmans acknowledged people could be "a bit tired of hearing about [cloud computing] because there's so much talk about it." But customers are nonetheless interested in leveraging its benefits, Gutmans said. Zend Cloud Platform will feature portable and native cloud services, application platform monitoring, cluster management, application deployment, configuration management, and IDE integration.

Within the same building complex as ZendCon, attendees at the Cloud Computing Conference & Expo conference Tuesday were more upbeat about cloud computing, as would be expected.

"We're definitely interested in cloud computing and right now, I'm on a research project where we're actually using the Amazon Web Services [cloud] environment to do all of our research work," said Jim Cannaliato, vice president of technology at SAIC.

Another attendee noted his company's growing use of cloud computing. "We've got some bits and pieces, so we're not fully cloud-enabled, but that's the direction we're heading," said Sadri Behbahany, senior director of IT at Wacom, which makes tablet input devices.

This article, "Cloud computing underwhelms PHP developers," was originally published at InfoWorld.com. Follow the latest developments in business technology news and get a digest of the key stories each day in the InfoWorld Daily newsletter.

Read more about cloud computing in InfoWorld's Cloud Computing Channel.

Paul Krill is an editor at large at InfoWorld, focusing on coverage of application development (desktop and mobile) and core Web technologies such as HTML5, Java, and Flash.

Google releases video chat source code

Google has released the code for WebRTC, a voice and video codec for the Web


Google has released the source code for a technology that it hopes developers will use to embed real-time video and voice chat functionality in their Web applications.

Google acquired the technology, called WebRTC (Web Real Time Communication), when it purchased VoIP (Voice over IP) software developer Global IP Solutions in 2010, for approximately $68.2 million. The company said it would open source the technology early last month.

[ Track the latest trends in open source with InfoWorld's Open Sources blog and Technology: Open Source newsletter. ]

WebRTC is a set of voice and video signal processing technologies, which can be accessed by developers through HTML tags and JavaScript APIs (application programming interfaces).

Today, Internet audio and video chat services from companies such as Skype are chiefly proprietary, accessible through plug-ins and client downloads. Last month, Microsoft agreed to purchase Skype for $8.5 billion.

Google wants third-party developers to use the voice and video engines to create chat applications that can be run directly from within a browser. Global IP Solutions has built WebRTC-based mobile clients for Android, Windows Mobile and the iPhone. Ericsson Labs built a videoconference prototype with the technology as well.

The move to open source WebRTC echoes a similar move Google made when it acquired video compression provider On2 Technologies in 2010. Google subsequently released On2's VP8 video codec as open source to provide a royalty-free alternative to the widely used H.264 standard.

Google is working with other browser developers, such as Mozilla and Opera, in hopes they will support the technology in their browsers. The company is also participating in W3C (World Wide Web Consortium) and IETF (Internet Engineering Task Force) projects for creating real-time communication Web standards. WebRTC is based on the W3C's Web Applications 1.0 API.

The source code is available under a royalty-free BSD (Berkeley Software Distribution)-style license.

Google did not immediately respond to requests for comment.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's email address is Joab_Jackson@idg.com

Building Semantic Web CRUD operations using PHP

When developing a Web application, it's standard practice to create a database structure on which server-side code is placed for the logic and UI layers. To connect to the database, the server-side code needs to do some basic creating, updating, deleting, and — most importantly — reading of records. As databases behind Web applications are typically relational databases, these CRUD operations are done using the well-known language, SQL. However, as Web development is increasingly occurring through object-oriented programming (OOP), the model is changing.

The Resource Description Framework (RDF) is a perfect way to describe objects while maintaining the meaning of that data. Simple Protocol and RDF Query Language (SPARQL — pronounced "sparkle") is the language typically used to query against that data, as it syntactically matches the structure of RDF itself. Both RDF and SPARQL are technologies within what has been dubbed the Semantic Web stack.

To fully embrace the Semantic Web idea, you can apply traditional Web-development techniques to RDF data using SPARQL. This article shows how to use a simplified Model-View-Controller (MVC) design pattern, the PHP server-side scripting language, and SPARQL for connecting to RDF — as opposed to using SQL on a relational database system.

SQL and SPARQL CRUD operations

Prerequisites

This article assumes a basic understanding of SQL, PHP, and Web application development. An understanding of Semantic Web is also beneficial. To run the create, update, and delete commands on Semantic Web-based data, you need a Semantic Web data store that supports the SPARQL/Update specification.

It's worth taking a look at the similarities and differences between CRUD operations when developed in SQL and SPARQL. Listing 1 shows the SQL code for a read operation.


Listing 1. SQL for the read operation
SELECT realname, dob, location FROM UserTable  WHERE realname = "John Smith"; 

Compare that SQL-based code with the SPARQL-based code shown in Listing 2. These are two read operations because they're the easiest to understand, implement, and explain. This is true for both SQL and SPARQL.


Listing 2. SPARQL for the read operation
     PREFIX foaf:   PREFIX rdf:  SELECT ?uri ?name ?dob ?location FROM  WHERE { ?uri rdf:type foaf:Person ; foaf:name "John Smith" ; foaf:birthday ?dob ; foaf:location ?location . } ; 

Your first thought when comparing the two listings is likely to be that the SPARQL version clearly has many more lines than the SQL version. That is true, but don't be tricked into thinking that the SQL is necessarily simpler and cleaner. SPARQL, depending on the engine that you run it against, can be completely distributed through something known as the linked data effect. In addition, it allows you to have dynamic schemas because of its interlinked object-oriented perspective, in contrast to SQL's strictly relational perspective. If you were to split relational database tables into as many islands of data, you would actually have many more lines of SQL in comparison to SPARQL — not to mention that the SQL would be full of those nasty JOIN descriptors.

The first two lines of the SPARQL are the PREFIX declarations. According to Semantic Web theory, everything — whether an object or a data graph source (also an object) — has a Uniform Resource Identifier (URI). The PREFIX lines are simply applying a temporary label to some URIs — in this case, the Friend of a Friend and RDF schemas. The benefit here is that you can use the PREFIX declarations later in the query instead of having to use the full URIs.

The next line of the SPARQL code describes the query request. It's essentially the same as the SQL statement, except for the additional request for the URI. Take note of the use of question marks (?) to indicate that the term is a variable.

The FROM statement describes where to grab data. It's the same in SQL and SPARQL, except that in SPARQL, the data source name is a URI, rather than a string denoting a physical location on your computer or network.

The WHERE statements are quite different from each other because with SPARQL, you must specify which schemas to use to fetch data. Once again, if you tried to do this using traditional methods, you would need a lot more than plain SQL: You'd need to use the PHP, the Java™ programming language, or some other server-side language to do checking between data sources. It is reasonably clear what the lines of SPARQL do, which includes ensuring that the data being retrieved is only of the type Person. SPARQL fetches a name and a location while doing some pattern matching to find the right John Smith.

Create

CRUD operations in SPARQL are typically a bit more shrouded in mystery than the read operation. However, they can be done. To begin, the create operation inserts a new record or object into the table or graph.


Listing 3. SQL for the create operation
     INSERT  INTO UserTable (realname, dob, location)  VALUES ("John Smith", "1985-01-01", "Bristol, UK");  

Now, compare the SQL-based code in Listing 3 with the SPARQL-based code in Listing 4 for the same create operation.


Listing 4. SPARQL for the create operation
     PREFIX foaf:   PREFIX rdf:  INSERT  INTO GRAPH   (?realname, ?dob, ?location)  {  rdf:Type  foaf:Person ;   foaf:name "John Smith" ;   foaf:birthday  <1985-01-01T00:00:00> ;  foaf:location "Bristol, UK"  } 

Once again, notice that the PREFIX lines work exactly as they do in the read operation with SPARQL. The INSERT INTO works similarly to SQL, but again, this is URI-based rather than string-, table-, and name-based, which allows the operation to be done across HTTP. You must also specify the schema again. Here, it's slightly easier to understand than in the read operation, as you can have practically any kind of property so long as it is compatible with the schema. This is a benefit and a beauty of the distributed dynamically extensible objects formalism RDF provides.

Delete

If you create, at some point, you are probably going to want to delete. For instance, users may want to delete their account on your site (obviously unfortunate that they want to leave, but they may have valid reasons). Listing 5 provides the SQL code for a typical delete operation.


Listing 5. SQL for the delete operation
     DELETE FROM UserTable  WHERE realname = "John Smith" 

Now, compare the SQL-based code in Listing 5 with the SPARQL-based code in Listing 6.


Listing 6. SPARQL for the delete operation
     DELETE  FROM GRAPH  { ?predicate ?object } 

The fundamental difference between the SQL and the SPARQL code is that the SQL deletes a row in a table, whereas the SPARQL deletes "all triples" relating to the "John Smith" object denoted by http://www.example.org/graph/johnsmith#me. This difference is the result the graph-based nature of the RDF model.

Update

Many Web applications allow users to update their information. The UPDATE operation is what makes that possible. Listings 7 and 8 demonstrate how to code this in SQL and SPARQL.


Listing 7. SQL for the update operation
      UPDATE UserTable SET location = "Weston-super-Mare, UK" WHERE realname = "Joanne Smith" 

Now compare the SQL-based code in Listing 7 with the SPARQL-based code in Listing 8 for the update operation.


Listing 8. SPARQL for the update operation
     PREFIX foaf:  MODIFY   DELETE {?uri foaf:location ?location}  INSERT {?uri foaf:location "Weston-super-Mare, UK"} WHERE { ?uri foaf:name "Joanne Smith" } 

UPDATE with SPARQL may seem incredibly silly, but it's completely valid when you understand that you aren't updating a relational table row — you're updating one very specific relationship within a graph. The easiest way to do this without attaining multiple locations is to delete and insert. The MODIFY keyword is used to establish the connection to the right graph.

Connecting to SQL and SPARQL database systems

To execute the above SQL and SPARQL statements, you must connect to the system somehow. Different systems, obviously, have different connection methods. One common method is to connect to a generic database using Open Database Connectivity (ODBC) drivers, which are often included in current versions of Mac OS X and Linux® systems, and are installable on other operating systems such as Windows®. ODBC essentially provides a simple generic API to connect to an SQL database of your choice. Interestingly, ODBC also works with some Semantic Web data stores such as OpenLink Virtuoso. However, most other Semantic Web data stores require some custom connection procedure or a custom-made generic connection system such as RDF2Go (at the time of writing, RDF2Go is only for Java technology) that work with a variety of systems such as Seseme and Jena. An alternative to consider if your data is going to be exposed over HTTP is a SPARQL connection method over HTTP, which makes your data "Linked Data"-ready and can be completely distributed. Because the range of connection methods for SQL and SPARQL vary, it isn't feasible to cover these in any detail here.

SQL and SPARQL through PHP

After selecting a connection method, the traditional next step is to establish common operations in PHP. For a customized system, this is usually done using SQL strings with PHP variables injected and passed to it through function parameters. The function will then connect to the database and execute this transaction. The proposal here is to do exactly the same for a SPARQL-RDF connection as would be done for an SQL-RDBMS connection.

So, take a gander at another code comparison — this time using the PHP language — starting with read, then going on to create, delete, and update, as with the code comparisons above. Use a hypothetical query execution function called query_execute, which takes in a string representation of the SQL/SPARQL statement.

Read

First up is the simple read operation. Variables can be injected into the query using string concatenation in PHP.


Listing 9. SQL for the read operation
     function readUserInfo($realname) {     $sqlstatement = "SELECT realname, dob, location FROM UserTable WHERE     realname = \"" + realname + "\";";     return query_execute($sqlstatement);     } 

Now, compare the SQL-based PHP code in Listing 9 with the SPARQL-based PHP code in Listing 10 for the humble read operation.


Listing 10. SPARQL for the read operation
      function readUserInfo($realname) {     $sqlstatement = "PREFIX foaf:      PREFIX rdf:  SELECT ?uri ?name ?dob ?location  FROM  WHERE  { ?uri rdf:type foaf:Person ;  foaf:name \"" + $realname + "\" ;  foaf:birthday ?dob ; foaf:location ?location . } ;";     return query_execute($sparqlstatement); } 

As you can see, the above function has been developed in such a way that the function name and parameters are identical, which means that you could start using the SPARQL version straightaway. Plus, if you're using an ODBC system, you won't need to worry about return types changing.

Create

As above, so below. The following PHP methods inject variables into a string to run a dynamic query. Listing 11 shows how to do that with a create operation.


Listing 11. SQL for the create operation
      function createUserInfo($realname, $dob,     $location) {     $sqlstatement = "INSERT INTO UserTable (realname, dob,     location) VALUES (\"" + $realname + "\", \"" + $dob + "\", \"" +     $location + "\");"; return query_execute($sqlstatement); } 

Now compare the SQL-based PHP code in Listing 11 with the SPARQL-based PHP code in Listing 12 for the create operation


Listing 12. SPARQL for the create operation
      function createUserInfo($uri, $realname,     $dob, $location) {     $sparqlstatement = "PREFIX foaf:      PREFIX rdf:  INSERT INTO GRAPH      (?realname, ?dob, ?location)      {      " + $uri + " rdf:Type     foaf:Person ;     foaf:name \"" + $realname + "\" ;     foaf:birthday         <" + $dob + ">> ;         foaf:location \"" + $location + "\}" return query_execute($sparqlstatement);     } 

For simplicity, the SPARQL version of this function includes a URI parameter, although it is simple enough to do some additional string concatenation to make the function declaration identical to the SQL.

Delete

Use injection through concatenation in PHP to perform the delete operation. Please note, as mentioned in the previous section, that different variables pass into the SQL and the SPARQL versions. Listing 13 shows the SQL code for the delete operation.


Listing 13. SQL for the delete operation
      function deleteUserInfo($realname) {     $sqlstatement = "DELETE FROM UserTable WHERE realname = \"" + John Smith + "\"";         return query_execute($sqlstatement);  } 

Now, compare the SQL-based PHP code in Listing 13 with the SPARQL-based PHP code in Listing 14 for the delete operation.


Listing 14. SPARQL for the delete operation
      function deleteUserInfo($uri) {     $sparqlstatement = "DELETE FROM GRAPH      {<" + $uri +     "> ?predicate ?object }";      return query_execute($sparqlstatement);  } 

Once again, for simplicity, the SPARQL version of the function takes in a URI, rather than the real name. You could use the DELETE SPARQL statement with a WHERE clause to find the data by name rather than URI, which would enable you to have an identical function header to the SQL version.

Update

Updating a record is fairly simple in both SQL and SPARQL through PHP. Just be aware of the different structures that relational databases and RDF provide. Listing 15 shows the SQL code for the update operation.


Listing 15. SQL for the update operation
      function updateUserInfo($realname, $location)     {         $sqlstatement = "UPDATE UserTable SET location = \"" + $location + "\" WHERE realname = \"" + $realname+ "\";";             return query_execute($sqlstatement);      } 

Now compare the SQL-based PHP code in Listing 15 with the SPARQL-based PHP code in Listing 16 for the update operation.


Listing 16. SPARQL for the update operation
      function updateUserInfo($realname,     $location) {      $sparqlstatement = " PREFIX foaf:  MODIFY  DELETE {?uri foaf:location ?location}  INSERT {?uri foaf:location \"" + $location + "\"} WHERE {      ?uri foaf:name \"" + $realname + "\"     }         ";         return query_execute($sparqlstatement);     } 

The function declarations in the SQL and SPARQL versions are identical. As a result, swapping from SQL to SPARQL is simple.

Although it's quite easy to move from a data source with an SQL endpoint to a data source with a SPARQL endpoint, there are two important areas you need to be aware of: levels of abstraction and the similarities and differences between SQL and SPARQL. You want to avoid that trapped feeling that comes from thinking that one language is practically identical to another. So it's best to understand a language's limitations while at the same time exploiting the language's features — in particular, the various syntax sugar and graphical representations provided by different languages.

Levels of abstraction

The example code in this article is strongly coupled to the database structure. In theory, there is only a certain level of loose coupling that SQL and RDBMS can cope with before table structure reorganization must occur. However, this is not the case with SPARQL and RDF. With SPARQL, your coupling with data can be very abstract because of the distributed but interlinked nature of the RDF. The reusability of the examples in this article could potentially be improved by increasing their abstractness. However, for our purposes, simple closely coupled functions are used to exemplify the similarities and differences between SPARQL and SQL.

As you can see from the example code, there are many similarities between SQL and SPARQL. The differences come in when understanding the Web-based, graphical, and object-oriented nature of RDF and how that filters into the SPARQL language. As a simplified rule of thumb, you can imagine the triple structure in RDF and SPARQL as basically representing, in order, the unique primary key of a row (the subject), the attribute/column name (the predicate or relationship) and the cell data that is based on the row and column (the object). In addition, SPARQL can take full advantage of HTTP communication, and, therefore, data can be (but doesn't have to be) distributed over intranets, extranets, and the wider Internet.

Why move from SQL to SPARQL?

There are many reasons why you would want to move from SQL to SPARQL. The details extend beyond the scope of this article, but you could be motivated by the following points:

  • You want a more distributed data solution.
  • You want to expose your data on the Web for people to use and link to.
  • You may find Node-Arc-Node relationships (triple) easier to understand than relational database models.
  • You may want to understand your data in a pure object-oriented fashion to work with an OOP paradigm (PHP V5 and later supports OOP).
  • You want to build generic agents that can connect to data sources on the Web.

Of course, there are also reasons why you may not want to move away from SQL, and they are probably perfectly valid reasons. SPARQL is an additional method of querying, not necessarily an immediate replacement of SQL. The same goes for relational data and Semantic Web-based data. These are not replacements. Instead, it's best to think in terms of merging newer and older techniques to produce a hybrid system that can handle and be handled by older legacy systems and the current and future systems.

curtsy: Daniel Lewis

Friday

Is This Web 3.0?

Not everyone agrees on exactly what Web 2.0 entails. As with all great buzzwords and concepts, people are already predicting what Web 3.0 will be. Will rich internet applications dominate it?

RIAs are still in their infancy, but when done right they're incredibly powerful tools. When Google launched Google Maps a few years ago, it opened people's eyes to the fact that web browsers can do much more than merely display pictures and text.

Currently, there are four mainstream mechanisms being used to develop RIAs.


AJAX/JavaScript: AJAX is a web development technique for using JavaScript with XML to create a rich internet application by dynamically and asynchronously exchanging data in the background without having to refresh the page. Google Maps and Gmail demonstrated what could be done with simple existing technologies like JavaScript and XMLHttpRequest. Google, Microsoft and Yahoo! all now promote their own AJAX toolkits to assist in building AJAX-rich media functionality.

Flash/Flex: The first horse in the RIA race was Flash. Adobe/Macromedia with its Flash/Flex infrastructure is still the leader in online video. Combining the programming capabilities of Flex makes an incredibly powerful toolset for creating internet applications. Flash has strong penetration and when used effectively can enhance your website.

Silverlight/.NET: Microsoft is barreling ahead with Silverlight, a browser plug-in to deliver interactive web applications that should be taken seriously. The company launched Silverlight earlier this year and is promoting it heavily to its large partner development network. Silverlight is delivered to a browser via XAML, which is a text-based markup language. This makes it easier for search engines to scan Silverlight vs. Flash.

OpenLaszlo: Finally, even though you don't see it much, there's an open-source platform for RIAs called OpenLaszlo. Initially developed as a proprietary system by Laszlo Systems, it was made open source in 2004. Not wanting to be left out of the RIA race, IBM--consistent with its embracing of Linux and other open source--has helped propel OpenLaszlo. The company worked with Laszlo Systems to use the open source Eclipse development platform with OpenLaszlo. Applications for OpenLaszlo can be run in Flash or in DHTML.

One current issue with Flash is that while search engines can index it, they don't index it as well as with text because Flash is a binary compiled file. That's why most websites aren't entirely created in Flash. Accessibility and keyboard navigation can be issues with these rich applications as well. If you don't have a mouse or can't use one, then you'll have problems with these technologies. Also, while the plug-ins have sizable browser penetration, they're problematic for some users.

What It Means for You
What does all of this mean for business owners in the Web 2.0 era? For the tech entrepreneur it means new opportunities. Many traditional client server applications are being pressured to move their applications to the web. Entrepreneurs can potentially displace client server apps with new innovative web applications. For other entrepreneurs, you have to evaluate your business and what specific benefits you can get from adding rich features to your website.

Ever Evolving
There will be other emerging technologies in the RIA area. The combination of these rich features will help trends like social networking continue to evolve. We'll likely see many websites with more drag-and-drop-type features in the next few years. The online/offline office also will continue to develop as predominant internet companies compete with Microsoft Office for the next generation office applications. Web applications will continue to become more robust and feature rich than ever before.

courtsy: Frank Bell

Thursday

NASA updated to MySql

The NASA Acquisition Internet Service (NAIS) (http://nais.nasa.gov/) is responsible for providing the general public with information regarding contract opportunities with the revered space organization. A network of servers interconnecting 12 of NASA's field installations, NAIS is the only means for obtaining acquisition information for contracts ranging between $25,000 and $500,000. Saving NASA and its partners roughly $4 million annually, the NAIS model has been so successful that it has been adopted by the U.S. FedBizOpps program (http://www.eps.gov/) as a means for providing access to contracting opportunities for the entire U.S. Government. Furthermore, NAIS supports several thousand users, and receives on average 300,000 hits each month.

Given NAIS' mission-critical purpose at NASA, quite a few heads turned when they announced the successful conversion of the NASA Acquisition Internet Service database backend from Oracle to MySQL. Restructuring of Oracle licensing agreements would have left NAIS facing a serious budgeting dilemma. As a result, the NAIS team began searching out a more cost-effective database solution. The obvious choice? Open Source. And within the Open Source arena, the NAIS team settled upon what they considered to be the most robust database product available: MySQL.

Asked what aspects of MySQL were most appealing, NAIS director Jim Bradford responded with three, paraphrased here:

  • Cost: Because the total cost of MySQL is limited to the cost of technical support, given that MySQL is available for free download and use in most cases.
  • Support: Due to the large developer community which can be found on the Internet. Although NASA has used direct support from the MySQL developers infrequently, he stated that "they were very helpful and responsive when needed". Furthermore, NAIS developer John Sudderth stated in an article discussing the switch that the cost for official support was about 1 percent of the technical support expenditure for Oracle (http://www.gcn.com/vol19_no33/enterprise/3275-1.html).
  • Compatibility: MySQL can easily interface with most SQL-compliant applications through ODBC.

Perhaps a fourth advantage to making the switch to MySQL could be attributed to performance. "We noticed an increase in [speed of] performance since the change and have not experienced any problems with the product.", says NAIS Computer Systems Analyst and project leader Dwight Clark stated in an article for Federal Computer Week (http://www.fcw.com/fcw/articles/2000/1204/pol-nasa-12-04-00.asp).

courtsy: W.J. Gilmore

MySQL and "LAMP" Save istockphoto.com $900K

Istockphoto.com is the biggest royalty-free stock photo community in the world, and its sister company, istockpro.com, is home to a host of illustrious professional photographers. Every week, approximately 5,000 photographers upload more than 2,500 photos (2.5 GB) to MySQL® -- the world's most popular open source database -- and approximately 1,250(1.25 GB) are accepted and posted to istock Web sites. More than 200,000 customers, including corporations, advertising and public relations agencies, and individuals access these photo databases and download 20-30 GB daily for a variety of uses. The MySQL database tracks every photo submitted and manages the permissions and billing, all for the fraction of the cost of traditional database systems.

"We operate a classic LAMP system," says Patrick Lor, executive vice president. "MySQL enables us to grow our business at a rate of 15 to 30 percent a month.

“A traditional database would have required at least a million dollars in funding for design, hardware, and all of the other costs associated with such solutions. We still wouldn't have gained the flexibility and stability that MySQL provides. With the LAMP approach, we've built a business based on sweat and ideas, and are growing it exponentially. And, it only cost about $100,000 to implement.”

According to Lor, istockphoto.com has a unique billing system that saves the company substantial costs associated with invoicing. In this model, users pay a $10 fee and then download photos against that sum. With each download, the user is issued a receipt. When the $10 runs out, users simply make another deposit. The MySQL database monitors the entire process.

Istockpro.com, the other istock site, operates somewhat differently. Professional photographers submit their photos, which are then posted to the Web site. Users then pay by credit card. MySQL tracks permissions and transactions, and provides some minimal accounting features to the photographers. Again, email receipts are issued, avoiding many of the costs typically associated with invoicing.

Prior to moving to an open source solution, istock Web sites relied on Cold Fusion. Lor states that, in addition to costs associated with this product, they were working with a version that they thought needed more work before it could help achieve their goals.

“We realized that MySQL and other open source products opened more options for us, ” says Lor. “The talent pool was more varied and the costs were certainly lower. More importantly, open source provides stability that is critical to our operations. Traffic is king in this business, and if we can't reliably support our photographers and our customers, we're out of business.”

Lor says that his company also jumped on what it saw as a trend toward using open source solutions for the enterprise, with unexpected results. The increasing legitimacy of open source solutions enables istock to approach much larger partners, such as Adobe Systems, Inc., than would otherwise be possible.

“After all, we're using the same database that Google and Yahoo! rely on,” Lor says. “That means that our choice of database is an asset, and not a liability when we go after partners and new business. It helps us grow.”

MySQL helps istock grow in other ways, as well. According to Lor, the 250,000-strong customer database records comprise active users who frequently ask for new features. MySQL helps accommodate these requests, enabling Lor and his team to make changes “on the fly,” in most cases within an hour or two.

Additionally, the istockphoto site is particularly active, with photographers - and their photographs - changing on a daily basis. MySQL tracks and manages these updates seamlessly.

When you're growing as rapidly as istock, scalability becomes mission-critical. Over the last year, the istockphoto configuration has grown from three to 10 Intel processors running Apache and Unix. This database comprises 250 GB of data, with 115 tables and 7.7 million records.

The istockpro database comprises 500 GB and runs on six Intel processors. It supports 41 tables and 2.7 million records.

“MySQL is such an integral part of our business, that it literally allows us to exist,” says Lor. “The fact that we can grow as quickly as we do is also attributable in large part to MySQL and the open source community.”

courtsy: mysql

Face Book Open Source Technologies

Introduction

Facebook has been developed from the ground up using open source software. Developers building with Platform scale their own applications using many of the same infrastructure technologies that power Facebook.


Platform

Our Platform engineering team has released and maintains open source SDKs for Android, C#, iPhone, JavaScript, PHP, and Python.


Developer tools

codemod assists with large-scale codebase refactors that can be partially automated but still require human oversight and occasional intervention.

Facebook Animation is a JavaScript library for creating customizable animations using DOM and CSS manipulation.

flvtool++ is a tool for hinting and manipulating the metadata of FLV files. It was originally created for Facebook Video.

Online Schema Change for MySQL lets you alter large database tables without taking your cluster offline.

PHPEmbed makes embedding PHP truly simple for all of our developers (and indeed the world) we developed this PHPEmbed library which is just a more accessible and simplified API built on top of the PHP SAPI.

phpsh provides an interactive shell for PHP that features readline history, tab completion, and quick access to documentation. It is ironically written mostly in Python.

Three20 is an Objective-C library for iPhone developers which provides many UI elements and data helpers behind our iPhone application.

XHP is a PHP extension which augments the syntax of the language such that XML document fragments become valid expressions.

XHProf is a function-level hierarchical profiler for PHP with a simple HTML-based navigational interface.


Infrastructure

Apache Cassandra is a distributed storage system for managing structured data that is designed to scale to a very large size across many commodity servers, with no single point of failure.

Apache Hive is data warehouse infrastructure built on top of Hadoop that provides tools to enable easy data summarization, adhoc querying and analysis of large datasets.

FlashCache is a general purpose writeback block cache for Linux. It was developed as a loadable Linux kernel module, using the Device Mapper and sits below the filesystem.

HipHop for PHP transforms PHP source code into highly optimized C++. HipHop offers large performance gains and was developed over the past two years.

Scribe is a scalable service for aggregating log data streamed in real time from a large number of servers.

Thrift provides a framework for scalable cross-language services development in C++, Java, Python, PHP, and Ruby.

Tornado is a relatively simple, non-blocking web server framework written in Python. It is designed to handle thousands of simultaneous connections, making it ideal for real-time Web services.


Engineers contribute to

Apache Hadoop provides reliable, scalable, distributed computing infrastructure which we use for data analysis.

Cfengine is a rule-based configuration system that is used to automate the config and maintenance of servers. Facebook uses Cfengine to maintain host configs and to automate many janitorial operations on our production tiers.

memcached is a distributed memory object caching system. Memcached was not originally developed at Facebook, but we have become the largest user of the technology.

MySQL is the backbone of our database infrastructure. You can find our patches on Launchpad and learn more about how we use it on the MySQL@Facebook page.

PHP is an incredibly popular scripting language which makes up the majority of our code-base. Its simple syntax lets us move fast and iterate on products.

Varnish serves billions of requests every day to Facebook users around the world. Whenever you load photos and profile pictures of your friends, there's a very good chance that Varnish is involved.


Mirror

We host a public mirror for projects such as Apache, Centos, CPAN, Fedora, GNU, Mozilla, MySQL, and much more...

Tuesday

PHP for Android Project Launched

irontec have just launched an open source project to bring PHP to Android platform. PHP for Android project (PFA) aims to make PHP development in Android not only possible but also feasable providing tools and documentation. The project already have an APK which provides PHP support to Android Scripting Environment (ASE). To get started you can follow the screencast below :

APK and source code both available at http://phpforandroid.net. Minimum requirement to get PHP for Android running is Android 1.5 phone or emulator. There is even an unofficial ASE build with PHP 5.3 support included. Now Rasmus can get an Android phone and start scripting on mobile.

Getting Started with iPFaces PHP Mobile Application Framework

iPFaces is a flexible solution for easy development of form-oriented network mobile applications. With the iPFaces solution, mobile devices are able to render content received from a server using their native UI components. It uses thin presentation client (must be installed on device) to render application content. Using iPFaces it is possible to build an application where users can use their device's specific component behavior and additional device features, such as location service and additional graphic components of the device (lists, pickers etc.).

Architecture

The solution is based on the use of a thin presentation client installed on the device and an application/web server which generates the content for clients. The client and the server communicate with each other using the network.

The idea is similar to the web browser - web server model. The client sends HTTP(S) requests to the server and receives iPFaces specific HTTP(S) responses, where the content is an XML representation of the application's form which is be rendered on client-side together with the form's data.

image01.png

How to start?

Development of a complex iPFaces application is really simple because the simulation mode can be used. This mode is capable of transfering XML content to a HTML page, which can be displayed in a web browser. It is a helpful tool for developers who can see their iPFaces application in the browser window and they do not need a real iPhone device for main development.

Developers can build and deploy an application to the application server and the browser will show them the GUI which is almost the same as a screen in an iPhone application. There is a difference in the GPS elements. A GPS field is working in a browser only as text field that can be filled by user and GPS coordinates will be submitted. The GPS field is hidden on mobile devices, because the location of the device is detected without interaction with the user.

image02.png
image03.png
Figure 1: Form representation on the iPhone device and in a web browser

Hello World Example

To use PHP iPFaces library, just include "ipfaces-php-lib-1.1.php" file, construct the component tree and call "render()" method on the component form.

  1. require "path/to/ipfaces/library/ipfaces-php-lib-1.1.php";
  2. $ipf_form = new IPFForm();
  3. $ipf_screen = $ipf_form->addScreen("screen", "Hello World Application");
  4. $ipf_screen->addLabel("label", "Hello World!");
  5. $ipf_form->render();
image04.png
image05.png
Figure 2: Hello World example

More complex example

The following example is not complete. It only illustrates how easy it is to define forms for iPhone using iPFaces. For complete examples please visit http://www.ipfaces.org

Example: Use of location service

To obtain a user location from a mobile device use the IPFGsm class. Upon submission of a form, the location data will be sent as a parameter with the selected name (gpsElement in this example).

  1. require "../lib/ipfaces-php-lib-1.1.php";
  2. require "citydatabase.php";
  3. $form = new IPFForm();
  4. $screen = $form->addScreen("How Far Is It?");
  5. $screen->addLabel("You can find distance between your position and selected city.");
  6. $parser = new CityDatabase();
  7. if (isset($_COOKIE["city"])){
  8. $value = $_COOKIE["city"];
  9. }
  10. $select = new IPFSelect("citySelect", $value, "Distance to" ,"list" );
  11. $options = array();
  12. for($row = 0; $row < count($parser->data); $row++){
  13. $options[] = new IPFOption($row, $parser->data[$row]["City"]);
  14. }
  15. $select->Icon = "../img/distcalc.png";
  16. $select->addOptions($options);
  17. $screen->addItem($select);
  18. $screen->addGps("gpsElement");
  19. $screen->addButton("backButton", "1", "Examples", "../index.php", IPFButton::BUTTON_TYPE_LINK, IPFButton::BUTTON_POSITION_BACK);
  20. $screen->addButton("submitButton", "0", "Calculate" , "distance.php", IPFButton::BUTTON_TYPE_SUBMIT, IPFButton::BUTTON_POSITION_FORWARD);
  21. $form->render();
image06.png

curtsy:

Monday

Scaling the BBC iPlayer to handle demand with PHP

One of the key goals we set ourselves when we developed the new iPlayer was that it would have to be fast to use. We understand that any delay in getting you to the video is frustrating as the site is just a jumping off point into TV and Radio content.

But how do we make things fast? Displaying a web page in the browser contains many steps, some we can control some we can't. Time spent for the request and response travelling over the network we can't control, but we can control how long the pages take to generate and how large they are. We also have a degree of control over how long those pages can take to render in your browser.

We had our work cut out for us on the new version of iPlayer.

Personalised websites require much more processing power and data storage

The current site uses one back-end service that we pull data from to build the pages. The new site uses many more, and we both post and pull data from them.

This means that every returning user gets a different homepage. There's already a small amount of difference between each homepage on our current site (your recently played) but the new site is driven much more by your favourites, recommendations and friends; they're key parts of the experience and they have to be fast.

We started developing in PHP

The BBC is standardising on PHP as its web tier development tool. Our current site is developed using Perl and Server Side Includes, and it's something that's well understood, but our new web tier framework (based on Zend) means that teams can share components and modules. In fact, the team responsible for the social networking functionality develop modules that anyone within the BBC can integrate into their site easily.

This does come at a cost though: the usage of a framework sometimes introduces delay in generating a page as it needs to get hold of resources to do so. In some cases this is necessary, especially if there's an element of personalisation, but in others our web tier is just repeating the same tasks.

All this against a growing demand

The site will have to support a massive amount of page views and users every day, on average 8 million a day for 1.3 million users. Previous versions of the site were able to grow into this demand; we'll have to hit the ground running from day one.

page-views_595.png

This graph shows our growth over the last year in terms of monthly page views.


So how do we do this?

One of the first things we can do is optimise the time it takes to generate the page.

Although changing architectures can be risky, we were confident that the one we moved to would enable us to meet all the challenges. At the heart of page generation is a PHP and customised Zend-based layer called PAL. This system then needs to integrate with our login system, BBC iD, our programme metadata system (Dynamite), our social networking systems, a Key Value data store and a few others. The homepage alone for a logged-in user with friends requires 15 calls across these services. Even if each of those calls take a few milliseconds, we can spend a second or two just collecting the information required, which would push us well out of our 2.5s target.

We proved our architecture before we built it

At the start of re-architecting iPlayer, we did what we could to eliminate guesswork. We developed a number of architectures based on our requirements, and then built prototypes of three of them; all built to serve the homepage, which we then tested against some basic volumetrics. This gave us plenty of data about how many requests we could serve a second and CPU loads, which we could then weigh up against other softer factors, like how our dev team could work with it.

We actually ended up going for the one which offered us a good balance between these factors, as this enabled us to be the most flexible in building pages, rather than constraining what we could with the site just to squeeze the extra speed out.

We cache a lot

Caching means storing a copy of the data in memory so subsequent requests for that data don't have to do the expensive things such as database queries.

It also allows us to get around any delays introduced by our framework starting up, as there's no such delay when delivering from cache.

Caching has its problems though. The data may have changed in the underlying system (programmes become available to play for example) but the change won't be reflected in our cache. This means we can only cache for seconds or minutes, but with the millions of page views we get, it can still make a crucial difference.

  • Data caching We cache the data returned from the services. We use Memcached for this. Sometimes we share data between pages.
  • HTML caching We also cache the resulting HTML for a short time. When you're hitting a page, it's highly likely you're just seeing the cached page. We use Varnish for this. Caching in this way is nothing new, but Varnish has a few tricks up its sleeve that we use which I'll explain later.

We broke the page into personalised and standard components

If you look at our homepage, many of those components are the same for everyone, but some are just for you. With traditional page caching in some reverse HTML caches, it's not possible to do this; so we break the page up. The main build of the page is cached; then when the page loads we use XHR and Ajax to load in the personalised components. Varnish gives us the ability to control the caching at a low-level like this. Every time we generate a page or a fragment, we can tell Varnish how long we want to cache it for. The main bulk of the homepage doesn't need caching for long to get some benefit, but your favourites we can cache for longer (although still only for a few minutes), and we know when you add a new favourite so we can clear out the cache and replace it with the new content. This means as you browse the site, the page loads quicker and your experience is smoother.

We use loads of servers

After we've optimised all we can using a single server, we then scale horizontally using multiple servers joined together in a pool. None of our web servers store any state about who you are and what you're doing, so your request can go to any server at any time.

We also serve pages out of two locations (or data centres). This gives us a higher degree of resilience to failures; we can lose an entire data centre and still be able serve the site.

We load tested the site before we launched

We're able to track how the site is used, so this gives us the ability to produce detailed volumetrics of how we think the new site is going to be used. Some of it is estimation, but it's always backed up with data. We can then produce detailed load tests, so we can simulate usage of the site. This enables us to find and resolve any problems we may experience under load, before we go live.

The end result

We're not 100% there yet (this is a beta after all) but from this sample 24 hours of monitoring data you can see that, apart from a couple of spikes, we're doing well at keeping to our target of 2.5 seconds. (We were also able to track down the spikes to some misbehaving components on the platform).

page-times_595.png

We're currently working hard behind the scenes at making sure we can continue to serve at this speed as usage increases, spreading the load across our infrastructure.

At the end of this though, we hope the result of our efforts is that you won't notice a thing: it'll just work.

curtsy: Simon Frost is Technical Architect for BBC iPlayer .

Tuesday

PHP's Place in the Enterprise

PHP in the enterprise

PHP claims to be the most widely used programming language on the web. A quick look at http://langpop.com/ supports this – it’s almost certainly the most common for smaller web projects. PHP was not originally designed as an enterprise-level language, but as it has evolved, it has become suitable for much larger projects than were originally envisaged when Rasmus Lerdorf produced PHP/FI in 1995 (source). PHP now supports SOAP, XML-RPC, JSON and any database platform you care to mention.

For something to be considered “Enterprise-level”, i.e. ready for use in the enterprise, it should meet the criteria of the Enterprise Challenges examined in our previous blog post.

PHP as an Enterprise-level language

With PHP 5.3, PHP is a full object oriented language with exception handling and useful features such as closures. Let’s take a look at PHP in light of enterprise challenges.

Scalability

PHP is very scalable owing to its shared nothing architecture – Facebook, Yahoo and Flickr, for example, are huge apps that tackle many Enterprise problems such as scalability, security and robustness using PHP.

Whilst PHP does not itself explicitly support concurrency (each PHP request is a single request and response), its host server can handle the instantiation of requests. PHP is very fast, and can horizontally scale extremely easily, often as simply as adding instances to a cluster of servers. This means the data throughput and concurrent users challenges can be overcome.

Robustness

Because each request runs in isolation, PHP is less likely to become deadlocked than a multithreaded language such as Java. There are tradeoffs, however – Java’s sophisticated mutual exclusivity functionality can protect vital areas of data, whereas two PHP processes may update them at the same time. This can push a certain amount of work onto the database, but as such, is easily solved (and, some may argue, may be better handled by the database system anyway).

Frameworks

Enterprise application development can be expedited with sophisticated frameworks to abstract away tasks, help organise code, and provide functionality rather than re-inventing the wheel. There are frameworks available for PHP, the most prominent being Zend Framework. Zend Framework is robust, thoroughly unit tested and can be a strong platform for enterprise applications. Jim Plush, a senior developer at Panasonic, blogs about his experiences with it. eZ Components is another enterprise framework for PHP focusing on providing re-usable components.

Security

Compared with other languages, PHP is neither particularly more or less secure; out of the box it may lack much of the explicit security support provided in .Net, for example, but one of PHP’s main strengths is the wealth of libraries available through PEAR and PECL, providing a range of simple and sophisticated security options.

Summary

PHP warrants its burgeoning place in Enterprise application development, but should be used where appropriate, wherever possible taking advantage of existing libraries and frameworks rather than rewriting. As web developers, as our medium moves to a more social environment, we can learn much from the established enterprise frameworks and applications that are already serving millions of users.

Curtsy: Posted by Gavin Davies on 18th Dec 2009

Monday

Dynamic content for BBC

Matt McDonnell wrote about the new BBC Topic Pages Beta. I'd now like explain how some of the many components that build those pages all work together.

The point of the Topic Pages is that they bring together content from all around bbc.co.uk. Obviously, many different systems produce all that content, and in general they don't tend to share content very well. Our challenge was to build a platform that could make sense of the different interfaces to those systems to make sharing that content easier.

The first thing to note is that the Topic Pages themselves are dynamic, unlike the vast majority of pages on bbc.co.uk. Essentially, this means that the HTML of the page isn't stored as a physical file on a hard-disk, but instead is built up dynamically when the page is requested.

This is done by the "Page Assembly Layer" or "PAL", a brand new component written in the PHP programming language. In the future, the intention is that most pages on bbc.co.uk will be produced dynamically using the PHP layer, and the Topic Pages system has blazed a trail, being the first released on this new platform.

The PAL itself does a fairly simple job, in principle. First, it receives a request for a Topic Page; it then looks up which modules (ie, the different blocks of content on the page, such as BBC News, Programmes and Weather) it needs to build that page; it grabs all those different modules, which originate on various different systems and finally it assembles them before returning the page to the user.

The really important part here is that the PAL is grabbing all the useful content dynamically, and not storing any content itself (apart from a bit of caching, to help smooth out any spikes in load). This means that the PAL is a really generic system that can be used for building other sorts of pages as well.

topics_diagram2.jpg

The PAL actually requests the various content modules from another component, which itself routes these requests on to the underlying systems. This Module Routing system is implemented in Apache Cocoon, an open source framework, released by the Apache Software Foundation. This way, the PAL can access content through a simple, uniform interface (which is based on REST principles, for those who are into such things), rather than having to deal directly with many complex interfaces to multiple systems.

This gives us two big benefits. First, if we want to change how a particular module is implemented, we can just reconfigure the Module Routing, and don't have to alter the PAL (which we don't want to tinker with too much, as it is busy serving pages); second, it makes it easier for any other system, or other page on bbc.co.uk, to reuse the modules. This model also makes it easier to add new content modules to the PAL, as most of the logic about how to access a new module can be handled in the Module Routing layer.

As Matt said, "Topics are automatically updated web pages, each one covering a different person, country or subject." We wanted to present content from across the various departments within the BBC. The Search Engines are the one place that all that content from bbc.co.uk (well, most of that content) is brought together (nearly 2 million documents from www.bbc.co.uk and the Newsand Sport sites). So it makes a lot of sense to create many of the modules from the Search indexes (the News module, and the Programmes module, for example, are both created like this). This is done using elaborate search queries, which are a bit fancier than the one- or two-keyword queries which the Search Engines normally receive - the query for the Fashion topic, for example, contains 364 terms! These queries are built using a variety of techniques by the Search Editorial Team.

We also use content from other sources than the Search indexes - for example, the Weather modules, and the information boxes about countries, which come from the BBC News website. In order to make it as easy as possible to share and reuse these content modules, we are attempting to create some standards for the formats in which the data is passed around (rather than just allowing every system to specify its own format).

Having looked around a bit, we decided to base our data structure on the Atom format, which was originally created to describe feeds from Blogs. Atom is a rich XML format, and one of its most attractive features is that you can add your own data elements. This means we can have a standard container for our data, but also include extra elements where appropriate (such as in the weather modules, to denote temperatures, for example).

This approach is similar to Google's GDATA, which is also based on ATOM. We have a prototype of this XML format - called BBC Module XML - in use in the Topics pages, and in the next few months we will be looking at refining and improving it, and hopefully making it useful for other systems across bbc.co.uk. We use XSLT to transform from the XML to the nice XHTML modules which are placed on the page.

The only parts that I haven't touched upon so far are the admin systems for creating and maintaining the pages and also the modules that go on them. At the moment, these admin systems aren't as joined up as they should be (and are implemented in different programming languages in some cases). This is a headache for the editorial team that maintain the Topic Pages. Our next priority is figuring out how we can bring those systems closer together, and generally improve the workflow for the editorial staff, so that they can easily add more Topic Pages.

Once we've made their lives a little easier, we will look at more feature enhancements. These include providing RSS feeds of the Topic Pages so that people can more easily stay up to date with their favourite topics. Additionally, we want to improve our systems for sharing metadata, so that it will be easier to automatically link to relevant Topic Pages from other pages on bbc.co.uk. And we will also add more types of content modules, to increase the range of content on the Topic Pages.

As you hopefully can see, the Topic Page project was pretty complex, and involved creating many new systems. Wherever possible, we have developed those systems to be generic and extensible, to provide not just Topic Pages, but also a platform for sharing and reusing content, and building other products in the future. This has all been possible thanks to some fantastic, far-sighted and occasionally frenetic work from the BBC Search Team and colleagues in FM&T Journalism and FM&T Internet - thanks to all of them.

N.B. More information on /topics can be found here.

Stephen Betts is Search Technical Team Leader, BBC Future Media & Technology

courtsy: bbc

ML self-service pipeline that abstracts Kubernetes complexity

  To successfully bridge the gap between machine learning engineering and cluster operations, you need to build a self-service pipeline that...