geschätzte Lesezeit: 5 min

November 16, 2008

There are different kinds of HTTP caches that are useful for different kinds of things. I want to talk about gateway caches – or, “reverse proxy caches” – and consider their effects on modern, dynamic web application design.

Draw an imaginary vertical line, situated between Alice and Cache, from the very top of the diagram to the very bottom. That line is your public, internet facing interface. In other words, everything from Cache back is “your site” as far as Alice is concerned.

Alice is actually Alice’s web browser, or perhaps some other kind of HTTP user-agent. There’s also Bob and Carol. Gateway caches are primarily interesting when you consider their effects across multiple clients.

Cache is an HTTP gateway cache, like Varnish, Squid in reverse proxy mode, Django’s cache framework, or my personal favorite: rack-cache. In theory, this could also be a CDN, like Akamai.

And that brings us to Backend, a dynamic web application built with only the most modern and sophisticated web framework. Interpreted language, convenient routing, an ORM, slick template language, and various other crap – all adding up to amazing developer productivity. In other words, it’s horribly slow and bloated… and awesome! There’s probably many of these processes, possibly running on multiple machines.

(One would typically have a separate web server -- like Nginx, Apache or lighttpd -- and maybe a load balancer sitting in here as well but that's largely irrelevant to this discussion and has been omitted from the diagrams.)

Expiration

Most people understand the expiration model well enough. You specify how long a response should be considered “fresh” by including either or both of the Cache-Control: max-age=N or Expires headers. Caches that understand expiration will not make the same request until the cached version reaches its expiration time and becomes “stale”.

A gateway cache dramatically increases the benefits of providing expiration information in dynamically generated responses. To illustrate, let’s suppose Alice requests a welcome page:

Since the cache has no previous knowledge of the welcome page, it forwards the request to the backend. The backend generates the response, including a Cache-Control header that indicates the response should be considered fresh for ten minutes. The cache then shoots the response back to Alice while storing a copy for itself.

Thirty seconds later, Bob comes along and requests the same welcome page:

The cache recognizes the request, pulls up the stored response, sees that it’s still fresh, and sends the cached response back to Bob, ignoring the backend entirely.

Note that we’ve experienced no significant bandwidth savings here – the entire response was delivered to both Alice and Bob. We see savings in CPU usage, database round trips, and the various other resources required to generate the response at the backend.

Validation

Expiration is ideal when you can get away with it. Unfortunately, there are many situations where it doesn’t make sense, and this is especially true for heavily dynamic web apps where changes in resource state can occur frequently and unpredictably. The validation model is designed to support these cases.

Again, we’ll suppose Alice makes the initial request for the welcome page:

The Last-Modified and ETag header values are called “cache validators” because they can be used by the cache on subsequent requests to validate the freshness of the stored response without requiring the backend to generate or transmit the response body. You don’t need both validators – either one will do, though both have pros and cons, the details of which are outside the scope of this document.

So Bob comes along at some point after Alice and requests the welcome page:

The cache sees that it has a copy of the welcome page but can’t be sure of its freshness so it needs to pass the request to the backend. But, before doing so, the cache adds the If-Modified-Since and If-None-Match headers to the request, setting them to the original response’s Last-Modified and ETag values, respectively. These headers make the request conditional. Once the backend receives the request, it generates the current cache validators, checks them against the values provided in the request, and immediately shoots back a 304 Not Modified response without generating the response body. The cache, having validated the freshness of its copy, is now free to respond to Bob.

This requires a round-trip with the backend, but if the backend generates cache validators up front and in an efficient manner, it can avoid generating the response body. This can be extremely significant. A backend that takes advantage of validation need not generate the same response twice.

Combining Expiration and Validation

The expiration and validation models form the basic foundation of HTTP caching. A response may include expiration information, validation information, both, or neither. So far we’ve seen what each looks like independently. It’s also worth looking at how things work when they’re combined.

Suppose, again, that Alice makes the initial request:

The backend specifies that the response should be considered fresh for sixty seconds and also includes the Last-Modified cache validator.

Bob comes along thirty seconds later. Since the response is still fresh, validation is not required; he’s served directly from cache:

But then Carol makes the same request, thirty seconds after Bob:

The cache relies on expiration if at all possible before falling back on validation. Note also that the 304 Not Modified response includes updated expiration information, so the cache knows that it has another sixty seconds before it needs to perform another validation request.

More

The basic mechanisms shown here form the conceptual foundation of caching in HTTP – not to mention the Cache architectural constraint as defined by REST. There’s more to it, of course: a cache’s behavior can be further constrained with additional Cache-Control directives, and the Vary header narrows a response’s cache suitability based on headers of subsequent requests. For a more thorough look at HTTP caching, I suggest Mark Nottingham’s excellent Caching Tutorial for Web Authors and Webmasters. Paul James’s HTTP Caching is also quite good and bit shorter. And, of course, the relevant sections of RFC 2616 are highly recommended.

(Oh, and the diagrams were made using websequencediagrams.com, a very simple, text-based sequence diagram generating web service thingy.)

geschätzte Lesezeit: 6 min

This blog post is about open vs private or closed APIs. For the web service specification, see OpenAPI Specification.

A few months ago I wrote a blog post about how important it is to make dev teams at FAIRTIQ independent and specifically how we leveraged Kafka to address a facet of it. In this blog post I’m looking at the same topic with a broader view: using open APIs to make companies independent while still having them benefit from one another. To illustrate it, I created an app that connects FAIRTIQ and Slack.

You’ve experienced it, the symbiosis

Over the past decade more and more companies have allowed developers to build applications on top of their platform and data. You are probably familiar with the “Sign in with Google” button that provides a 1-click registration to your favourite website using your Google account data, or the “Impress your friends” in your awesome video game’s leaderboard, which would make all your Facebook buddies jealous about your new high score.

A high score screen with a button to share on Facebook. Source: Pinterest

This kind of feature is possible thanks to open APIs. A company makes some data and actions publicly available for any developer to use in their own applications (in contrast to closed or private APIs). Generally, that benefits both parties. In the examples above, Google and Facebook get more accounts or activity on their services, whereas the developers can leverage the platforms to get more visibility or provide simplified user registration.

Open APIs allow platform providers to leverage the creativity of an ever growing developer community to come up with innovative applications that add value to their core business without investing directly in the development of these.

When any developer can use an open API to make cool things within our accounts, they can also do things which are less cool, all the way down to malicious. The data privacy question comes naturally to our mind. It turns out that the rules regarding who can do what and see what are set by the platform provider, i.e. the one providing the open API, holding the data and enabling the predefined actions. In most cases there is an authorisation flow, where the user grants the permission for instance to Acme inc to post on their Facebook feed on their behalf. Once the permission is granted, it remains valid until the user explicitly revoked it.

Open APIs and FAIRTIQ

Whereas FAIRTIQ provides open APIs for the public transport organisations (to connect their pricing systems for instance) it doesn’t provide a user-facing open API as of now. Let’s see how it could look like in a very simple form. An open API can be seen as a set of data to read, events to be notified about and actions to be performed.

FAIRTIQ’s hypothetical simple user-facing open API

One could argue that this is rather limited. However the full power comes when combined with other services or open APIs. Let’s say you want to create an app called “Travel with me buddy”. You could ask the user’s permission to access their FAIRTIQ check-in information and their Facebook friends. When two friends check in at the same time and place, you would send both a notification. As an event organiser, you could promote public transport to join the venue. You’d connect your event app to FAIRTIQ so that participants get credits to spend on-site based on how many kilometers they travelled with FAIRTIQ. Or as a home hacker, you’d make sure your vacuum cleaner robot starts worrying about your floors as soon as you leave your home and check-in.

The possibilities are endless. Some of them are home hacks and fun whereas others can be large scale revenue generating businesses. Given the potential, I want people at FAIRTIQ to start thinking “open API”. It’s a mindset. Something to have in the back of your head anytime regardless whether you are a developer, a sales person or a product designer. In order to make it more tangible and sparkle some ideas, I created the Commuting bot.

Check in with FAIRTIQ, followed by a status change on Slack

The Commuting bot is an app that automatically changes an employee's Slack status to “Commuting” whenever they are checked in with FAIRTIQ and restores it to whatever it was before when they check out. It leveragesSlack’s open API and FAIRTIQ’s inexistant open API (an internal simulation).

For this to work, the Commuting bot must obtain the user’s permission to be notified by FAIRTIQ whenever they check in and out, and from Slack to read and change their status. On the FAIRTIQ side, we do not support this authorisation model yet as we don’t provide an open API, so I hard coded it. On Slack, this is achieved by setting up a Slack App and setting the appropriate scopes.

Commuting bot slack app.

It’s worth noting that the Slack app contains no app logic. It’s only a way to request the appropriate permissions to the Slack user when they will install the Commuting bot. I created a basic web interface that allows FAIRTIQ employees to connect their Slack account to their FAIRTIQ account. After authenticating with FAIRTIQ and pressing “Connect” they are redirected to a page provided by Slack where they authorise the Commuting bot to edit their Slack profile on their behalf.

Commuting bot web interface

Slack’s authorisation page

From now on, the Commuting bot monitors the check in and checkout events on FAIRTIQ and adjusts the user’s Slack status in consequence. Behind the scenes, when the user grants permission to edit their Slack status, the Commuting bot obtains a token that allows it to act on Slack on the user’s behalf (as if it was the user themselves). This authorisation remains until the token is revoked. That can be done by clicking “Disconnect” in the Commuting bot or by uninstalling the Commuting bot app from Slack.

And it scales

Both companies FAIRTIQ and Slack most likely don’t care that much about Michel’s Commuting bot. That said, they also haven’t invested a single penny in it*. They only provided an open API that millions of Michels can use to innovate and/or serve their own business needs. Surely, providing an open API has a cost, however it’s mostly independent from the number of use cases that run on it. Make the open API once, let millions use it.

There are actually operational costs when running an open API, which are dependent on the usage, but they are usually marginal, provided that you have measures in place to prevent abuse (e.g. a developer sending billions of requests). For large scale usage, such APIs can also be monetised.

* Ok, I’m an employee of FAIRTIQ so that wasn’t completely free for us, but assume I wasn’t.

Conclusion

The benefits for both the open API provider and the consumer depend a lot on the use case being developed. Usually it’s a win-win situation: both parties mind their own business and on top they help each other without specifically investing in doing that. The open API provider leverages worldwide innovation for free, while at the same time unlocking new business opportunities for all the others. These new use cases, when successful, generally have a positive direct or indirect impact on the open API provider’s business.

Open doesn’t necessarily mean free. It means it’s not restricted to specific people or companies. An open API can be monetised if it makes sense to do so. Nevertheless, it must be done very carefully as it could easily defeat the purpose if it becomes a barrier to adoption. Freemium models (pay only if you make money with it) can be adequate.

Finally, open APIs are a great way to make systems interoperable. This becomes increasingly important, as systems grow in complexity and size. A big fat system operated by a single company will age much faster than a composition of multiple smaller, interoperable systems, each maintained by another specialised company.

Ah, and the title’s question? Well, now you know…

geschätzte Lesezeit: 4 min

This article demonstrates how to recursively update specific key in multidimensional array in PHP. 

Multidimensional arrays add alot to the complexity than a linear array. Similarly, updating specific keys in a multidimensional array using an iterative approach is complex and unpredictable. 

So, in this article, we take up the challenge and explore how to update specific key in multidimensional array in PHP. 

The article “How to update key value in array PHP” explores three options for changing keys in a linear array. 

• A loop approach.
• The direct indexing approach.
• Using PHP array_search function.

Out of these three, direct indexing stands out in terms of efficiency. That said, this article will use a mix of these to develop a generic function to handle multidimensional arrays.

Before diving into code examples, let’s first see the multidimensional array.

$students =
[
   "Sam" =>   ["Exam#1" => 87],
   "Brian" => ["Exam#1" => 80],
   "Chris" => ["Exam#1" => 92],
   "Sarah" => ["Exam#1"=> 90]
];

The array $students is a two-dimensional array. The objectives here are to

• Find the key in multidimensional array.
• Update specific key in multidimensional array.

Let’s see an example that uses a loop and direct indexing to update the key “Exam#1” to “MidTerm” in all the subarrays. 

Advertisements

<?php
$students =
[
   "Sam" =>   ["Exam#1" => 87],
   "Brian" => ["Exam#1" => 80],
   "Chris" => ["Exam#1" => 92],
   "Sarah" => ["Exam#1"=> 90]
];
function changeName($array, $oldKey, $newKey)
{
    foreach($array as $k => &$v)
    {
        if(array_key_exists($oldKey, $v)) //If key exists in the subarray
        {
            $value = $v[$oldKey];    //Passes the value to a variable using direct indexing.
            $v[$newKey] = $value; //Make a pair with the new key.
            unset($v[$oldKey]); //Unsets the old key and value pair.
        }
    }
    return $array;
}
$students = changeName($students, 'Exam#1', 'MidTerm');
?>

Here’s the breakdown of the function changeName.

Advertisements

• The loop iterates through the array and $v refers to the subarrays.
• PHP array_key_exists checks if the key is available a subarray.
• The block within array_key_exists updates the key and unsets the older one.
• Finally, the function returns the updated array.

 So, here’s the output of the function for the array above.

Array
(
    [Sam] => Array
        (
            [MidTerm] => 87
        )
    [Brian] => Array
        (
            [MidTerm] => 80
        )
    [Chris] => Array
        (
            [MidTerm] => 92
        )
    [Sarah] => Array
        (
            [MidTerm] => 90
        )
)

Looks cool! But let’s assume the multidimensional array changes to something as follows.

Advertisements

$students =
[
   [
       "Name" => "Sam",
       "Exams" => [
           "Exam#1" => 87
       ]
   ],
   [
    "Name" => "Brian",
    "Exams" => [
        "Exam#1" => 80
    ],
    [
        "Name" => "Chris",
        "Exams" => [
            "Exam#1" => 92
        ]
    ],
    [
        "Name" => "Sarah",
        "Exams" => [
            "Exam#1" => 90
        ]
    ],
],
];

It seems gibberish but worse than that is the function doesn’t support this structure. It will instantly throw an error. In simpler terms, it is not generic enough. So, what’s the way out then?

The way out is writing a generic function using recursion. So, up next we’ll see how to update specific key in multidimensional array using recursion.

Recursive algorithms adapt to arrays regardless of their dimensions and that’s why they are super flexible and robust. The following example uses a recursive function to update specific key in multidimensional array in PHP.

function changeName(&$array, $oldKey, $newKey)
{
    foreach($array as $k => &$v)
    {
        if(gettype($v) == 'array')
        {
            if(array_key_exists($oldKey, $v)) //If key exists in the subarray
            {
                $value = $v[$oldKey];    //Passes the value to a variable using direct indexing.
                $v[$newKey] = $value; //Make a pair with the new key.
                unset($v[$oldKey]); //Unsets the old key and value pair.
            }
            else
            {
                changeName($v, $oldKey, $newKey); //Go a level deep to find the key.
            }
        }
    }
}

The changeName function has been modified into a recursive function. The core logic of updating the key looks the same. The recursive calls happen if the function doesn’t find the key in a subarray. 

Advertisements

So, it will go as many levels down as possible just to find and update a specific key. Also, note that we are not returning from the function because it takes the array by reference thus making in place changes.

Here’s the output.

Array
(
    [0] => Array
        (
            [Name] => Sam
            [Exams] => Array
                (
                    [MidTerm] => 87
                )
        )
    [1] => Array
        (
            [Name] => Brian
            [Exams] => Array
                (
                    [MidTerm] => 80
                )
            [0] => Array
                (
                    [Name] => Chris
                    [Exams] => Array
                        (
                            [MidTerm] => 92
                        )
                )
            [1] => Array
                (
                    [Name] => Sarah
                    [Exams] => Array
                        (
                            [MidTerm] => 90
                        )
                )
        )
)

Voila! We’ve made a generic function to update specific key in a multidimensional array in PHP.

The article overviews the challenges in working with a multidimensional array and then explores an example that is non-generic but successfully updates a specific key. Then, the article moves to a generic recursive function that updates a specific key in a multidimensional array of any depth.

Hopefully, you’ve liked this article. Learn more about PHP at FuelingPHP. 

We have many fun articles related to PHP. You can explore these to learn more about PHP.