Category Archives: Development

A Simple Yet Powerful Cache Tagging Strategy for Redis

Posted on by
Reply

When using Redis for caching data related to specific entities, a basic requirement is being able to invalidate all the data related to any entity we update, so that any subsequent request related to that entity will refresh its cache. This is usually called “tagging”, as the basic idea is to tag the data you’re caching with the identifiers of the entities related to that data, so that when one of your entities changes, you (somehow) invalidate all the data tagged with that entity’s identifier.

Active Data Invalidation

The usual advice for implementing this on Redis is based on an active data invalidation strategy, where we actively invalidate (read: delete) all of the cache entries for any tag at the point where we’re updating its data.

Writing to the Cache

  1. Insert a record with the data you’re caching, usually as a string, with some identifier of your data as the key and your serialized data as the value.
    SET mydata1 "Serialized data for mydata1"
    SET mydata2 "Serialized data for mydata2"
    SET mydata3 "Serialized data for mydata3"
  2. Take the data identifier key you used in (1) and add it to a set or list corresponding to each associated tag record.
    SADD tag:mytag1 mydata1 mydata2
    SADD tag:mytag2 mydata3

Reading from the Cache

  1. Just get the cached data by key, using its identifier, and deserialize it.
    GET mydata1
  2. If there’s no data in the cache: retrieve it from persistent storage, generate it, etc. and then write it to cache.

Invalidating Tagged Data in the Cache

  1. Get the record corresponding to the tag you want to invalidate.
    SMEMBERS tag:mytag1
  2. For each data identifier key in that record, delete the record by key.
    DEL mydata1 mydata2
  3. Delete the record for the tag, effectively invalidating it.
    DEL tag:mytag1

Pros and Cons

Advantages

  1. Relatively simple to implement

Disadvantages

  1. Slow invalidation, as it first requires retrieving the list of keys associated with that tag, then deleting them as well as the tag record. This slows down any write involving updating a tag’s associated data.
  2. Greater risk of inconsistent data reads, as another client might read stale cache data while keys are being invalidated.
  3. If any of your tag records get evicted or deleted by mistake, there’s no easy way to invalidate those records.

Variants

There are slightly more elaborate approaches of this basic idea, for specific use cases, for example when you need to be able to easily access the tags corresponding to each cached data record you can use primary and secondary indices.

The Alternative: Passive Data Invalidation

This tagging strategy is based on keeping numeric “versions” of each tag. The invalidation becomes much easier and quicker as the tag’s version number simply has to be increased, at the cost of making the cache retrieval slighly more complex since you need to check if the version number of the data corresponds to the tag’s current version.

For this we can use Redis hashes, which are a very handy data structure that works as a collection of field-value pairs.

Writing to the Cache

All the tag versions will be saved in a hash record with an easily remembered key, for example tags, @tags, etc. We’ll use @tags in the rest of this example.

  1. Read your @tags record to get the current versions of your tags. If it hasn’t been created yet it will just be empty.
    HGETALL @tags
  2. Insert a record for the data you’re caching as another hash, with some identifier of your data as the key.  The serialized data you’re caching would go into a specific field in that hash, for example let’s call it @data. Also, for each tag associated with this data we add a field named after the tag, with the current version of that tag, as read in step (1), as the value. For tags not present in @tags we use 0 as the version.
    HMSET mydata1 @data "Serialized data for data1" mytag1 1 mytag2 0

Reading from the Cache

  1. Retrieve the current tag versions from @tags
    HGETALL @tags
  2. Get the cached data hash by key, using its identifier. If it’s not in cache: retrieve it from persistent storage, generate it, etc., and then write it to cache.
    HGETALL mydata1
  3. Compare the version of each tag in your cached data hash with its current version in @tags. If it is not exactly the same for any of the tags then ignore the cached data, and instead: retrieve it from persistent storage, generate it, etc., and then write it to cache.
  4. If all the tag versions in your cached data are current, then deserialize the @data field.

Invalidating Tagged Data in the Cache

  1. Increase by 1 the value of the field named after your tag, inside your @tags record. Helpfully, Redis creates this field automatically if it doesn’t exist, and will end up with value 1 after the increment.
    HINCRBY @tags mytag1 1

Pros and Cons

Advantages

  1. Invalidating tags is super quick and easy, as you just increment a value for each tag you updated.
  2. Less risk of inconsistent data reads, as tag invalidations are atomic.
  3. If you’re evicting by least recently or least frequently used, your @tags record should not be evicted as you will be accessing it all the time.

Disadvantages

  1. It’s a bit more complex to implement, especially the reading part.
  2. Reading data from cache requires also reading the @tags record. This can be mitigated by reading that record right after connecting to Redis and keeping it in memory for any subsequent cache retrievals during the same connection. Doing that also helps to keep the data consistent, as its less probable that you will mix stale and updated data in the same connection.

Conclusion

I have successfully used this approach for caching DB results while tagging them based on the tables the data was retrieved from. When writing to any table I’m also increasing its tag version, and all subsequent reads of the cached results based on statements such as SELECT or JOIN involving that table will automatically refresh that cached data. Systems that have a more robust entity tagging system can use the same idea for tagging individual entities inside the DB, like a particular user, blog post, etc.

If there’s interest, I might add an example of a PhpRedis implementation for setting, getting, and invalidating tagged data using this strategy. In the meantime, if you end up implementing this strategy, please let me know how you implemented and/or modified it for your particular use case, and what cool things you learned in the process!

MySQL Function for Calculating the Working Days between 2 Dates

Posted on by
Reply

Lately I needed a MySQL stored function to calculate the working days (or business days) between 2 dates, however all the solutions I found online were either not configurable in terms of which week days count as working days, or really hard to read/understand. So I rolled my own, and decided to post it here in case anyone else finds it useful.

Here’s the function declaration code, as well as a usage example, hosted in GitHub (or you can find it in https://gist.github.com/kazeno/8bad9453d1e4d2aed33e6af14d1aa7a1 if it’s not showing in your browser):

The function accepts 2 dates, as well as a string that specifies which week days should count as working days. The week days are input as the integers corresponding to their WEEKDAY function representation, i.e.:

0 = Monday
1 = Tuesday
2 = Wednesday
3 = Thursday
4 = Friday
5 = Saturday
6 = Sunday

Thus if the working days are Monday to Friday, the workdays argument would be ‘01234’, and if for a more abstact example the working days are Tuesday to Thursday plus Saturday, the workdays argument would then become ‘1235’.

The function itself determines the start and end dates from the first 2 arguments (so you can use it with the earlier and later dates in any position), counts the number of whole weeks (Monday to Sunday) between the 2 dates, and then loops through the remaining days not belonging to a whole week and counts them if they are contained in the 3rd argument.

Hope you found it useful!

Clear the Field of an non-Editable AngularUI Typeahead

Posted on by
Reply

When working with non-editable AngularUI Typeaheads (i.e. ones that have the typeahead-editable attribute set to false), a logical thing to do would be to erase the typeahead input field’s view value if the user doesn’t select a valid typeahead option. However, there’s no built-in option to do just that, so here’s a workaround.

The main idea here is to use the ng-blur attribute to set a function that will clear the typeahead field if no valid option was selected. To do that first we need to access by name the form controller object containing the typeahead. If the typeahead is not contained in a form element, what we can do is declare one of its parent elements as a form controller using ng-form, and give names to both of them:

<div ng-form name="myForm"> <input type="text" name="myField" ng-model="myField" ng-blur="clearUnselected()" typeahead="myField in myFields" typeahead-editable="false" /> </div>

Now we can access the the typeahead’s form properties from the AngularJS controller, and reset its $viewValue property if no valid option was selected by the user. We wrap this into an AngularJS $timeout because there’s a delay between clicking on an option and the corresponding model updating. Don’t forget to inject the $timeout service into your controller!

//inside your controller $scope.clearUnselected = function () { $timeout(function () { if (!$scope.myField) { //the model was not set by the typeahead $scope.myForm.myField.$setViewValue(''); $scope.myForm.myField.$render(); } }, 250); //a 250 ms delay should be safe enough }

How to fix blank screen in Preferences > Themes on the Prestashop Back Office

Posted on by
Reply

The Symptoms

You’ve linked your shop to your Prestashop Addons account, where you’ve previously or recently bought a theme. You try opening the Preferences > Themes menu, just to find that after trying to load the page for a while, you either get a blank page (even with dev mode on) or a cryptic 500 error.

The cause

When the Admin Themes Controller initializes, it checks if you’re logged into Prestashop Addons, and if so, whether there’s a theme in your account that hasn’t been installed in your store. If there is, it tries to download it automatically, and here’s where the problem lies: your server is timing out during this connection. If you go to your server error log, you’ll probably find an error such as mod_fcgid: read data timeout in xx seconds if your PHP installation is running on FastCGI.

The Solution

Unfortunately, if you are running on a shared host you almost certainly won’t have the required access necessary to change the server’s timeout parameters. In that case the only course of action is to disable the theme auto-download functionality in AdminThemesController.php, and this can be cleanly done in an override. This is all the code you need in your override: 

<?php
class AdminThemesController extends AdminThemesControllerCore
{
    public function init()
	{
        $this->logged_on_addons = false;
        parent::init();
	}
}

If you don’t know how to create overrides don’t worry, you can download the following zip with the override already in place. All you need to do is extract the folder within, upload it into your store’s root Prestashop folder, and delete the file cache/class_index.php on your store so the override is detected.

Access Private and Protected Properties of Objects in PHP

Posted on by
4

The theory goes that forcibly overriding Class behaviour from outside its scope is bad practice. However, in the real world we developers often times find ourselves having to adapt to, or work on top of, existing code outside of our control. That’s where hacks like these come in handy, though you should really use them with caution, and always because there’s no other choice available.

The Array Cast Override

This is the conceptually simplest method I’ve found, and the only one I know that works on PHP versions previous to 5.3. It involves casting an Object as an Array, and looking through the Array elements for the property you need. When doing this, the index of the Array element corresponding to each property is named according to special rules, depending on its visibility inside the Object. The next pseudo-code snippet shows these rules:

switch ($visibility) { case 'public': $array_index = $property_name; break; case 'protected': $array_index = "\0*\0" . $property_name; break; case 'private': $array_index = "\0" . $class_name . "\0" . $property_name; break; }

This means that the way you access the property depends on its visibility. As you can see PHP sandwiches either an asterisk or the Class name (depending on if the visibility is protected or private, respectively) between two NULL bytes, and prepends this to the property name to form the Array index. You can thus access the hidden properties of an object as follows:

class MyExampleClass //example of a class with hidden properties { public $myPublicVar = "I'm public!"; protected $myProtectedVar = "I'm protected!"; private $myPrivateVar = "I'm private!"; } $obj = new MyExampleClass(); $obj_array = (Array)$obj; //cast the object as an Array echo $obj_array['myPublicVar']; //echoes: I'm public! echo $obj_array["\0*\0" . 'myProtectedVar']; //echoes: I'm protected! echo $obj_array["\0" . 'MyExampleClass' . "\0" . 'myPrivateVar']; //echoes: I'm private!

The type of the property doesn’t matter, I’ve even accessed resource handlers this way.

The Reflection Override (PHP >= 5.3)

When talking about overriding a property’s visibility, this is the method most seasoned developers know. Using the Reflection Class, you reflect an existing object and change the accessibility of each property you need by using the ReflectionProperty::setAccessible method. Here is an example:

class MyExampleClass //example of a class with hidden properties { public $myPublicVar = "I'm public!"; protected $myProtectedVar = "I'm protected!"; private $myPrivateVar = "I'm private!"; } $obj = new MyExampleClass(); $refObj = new ReflectionObject($obj); $refProp1 = $refObj->getProperty('myProtectedVar'); $refProp1->setAccessible(TRUE); $refProp2 = $refObj->getProperty('myPrivateVar'); $refProp2->setAccessible(TRUE); echo $refProp1->getValue($obj); //echoes: I'm protected! echo $refProp2->getValue($obj); //echoes: I'm private!

Using Reflection is supposedly really slow, but the upside is that you can not only read the value of the property, but also change it using the ReflectionProperty::setValue method.

The Closure Bind Override (PHP >= 5.4)

This is the newest method I’ve come across, and it’s also the quickest and most painless one. It uses the bind static method of the Closure class to bind a Closure to our Object, which we’ll use to return the properties we want. The usage is really straightforward:

class MyExampleClass //example of a class with hidden properties { public $myPublicVar = "I'm public!"; protected $myProtectedVar = "I'm protected!"; private $myPrivateVar = "I'm private!"; } $obj = new MyExampleClass(); $propGetter = Closure::bind( function($prop){return $this->$prop;}, $obj, $obj ); echo $propGetter('myProtectedVar'); //echoes: I'm protected! echo $propGetter('myPrivateVar'); //echoes: I'm private!

It works almost like magic, the disadvantage being that most shared hosting providers are lazy and haven’t bothered to update their PHP versions since the Late Pleistocene. If you can use it, this is the method I would recommend the most.

Do you know any other methods to access hidden properties of objects? If so, I would love to read you comments below.

Acknowledgements and References

fmmarzoa at librexpresion dot org
Tobias Schlitt
PHP.Net

Install PHPUnit with VisualPHPUnit on a Shared Hosting Server

Posted on by
2

Update: The 4.x branch of PHPUnit has some changes, such as no longer using Autoload.php, that make this tutorial outdated. I’ll try to see if the new versions can be run on shared hosting as well, but in the meantime, the following instructions only apply for PHPUnit 3.x.

Here’s how to install and run PHPUnit with the VisualPHPUnit graphical interface, on any shared hosting server, without need for PEAR, composer, phar, or SSH command line; in fact all you need is your web browser, FTP access, and a code editor such as Notepad++.

PHPUnit: Dependency Madness

PHPUnit is the best case against the DRY philosophy I’ve ever came across, since it’s chock-full of random dependencies you won’t even be using for anything else. As such, be prepared to download and extract a helluva lot of packages.

First of all create a folder on your computer for housing all necessary files and directories, I’ve named mine Utilities for reference. Now go to https://github.com/sebastianbergmann/phpunit/branches and check the requirements of every numbered version until you find the newest one that will run on the php version you’ve got on your server. If you don’t know which version of php you’re using, run phpinfo() to find out. Download the zip for your version, and extract just the PHPUnit folder inside into Utilities.

Go to https://github.com/sebastianbergmann/php-file-iterator, download the zip and extract just the File folder into Utilities.

Go to https://github.com/sebastianbergmann/php-text-template, download the zip and extract just the Text folder into Utilities.

For the next 3 packages, download the zip files and extract just the PHP folder in each zip into Utilities (merge the folders): https://github.com/sebastianbergmann/php-code-coverage
https://github.com/sebastianbergmann/php-timer
https://github.com/sebastianbergmann/php-token-stream

Now create a folder inside Utilities called SebastianBergmann . Now create 3 folders inside SebastianBergmann, called: Diff , Exporter , and Version . For the next 3 packages, download the zip files and extract just the contents of the src folder (not the folder itself, only its contents) in each one into the corresponding folder inside SebastianBergmann: https://github.com/sebastianbergmann/diff
https://github.com/sebastianbergmann/exporter
https://github.com/sebastianbergmann/version

Finally, go to https://github.com/sebastianbergmann/phpunit-mock-objects, download the zip and extract just the PHPUnit folder into Utilities (merge it with the existing folder).

Great, now that you’ve got PHPUnit and all its dependencies, upload the “Utilities” folder to your server, preferably into a non web-accesible location. For reference, your folder tree should look like this:

folder tree

VisualPHPUnit: Let there be GUI

Now that PHPUnit is on your server, let’s make it usable with a web browser based GUI. Download the zip from https://github.com/NSinopoli/VisualPHPUnit, extract it, and rename the main folder if you want (I creatively renamed mine to VisualPHPUnit for reference).

Go into app/config/bootstrap.php and make the following changes:

  • Comment or delete the following line: 'pear_path' => '/usr/share/pear',
  • The default directory for tests is VisualPHPUnit/app/test , if you wish to change it or add other directories, modify the ‘test_directories’ array. By the way, $root points to the VisualPHPUnit folder.
  • Comment or delete the following lines: set_include_path( get_include_path() . PATH_SEPARATOR . $root . PATH_SEPARATOR . $config['pear_path'] );
  • Below the lines you just commented, add the following two, changing the relative paths so that they point to your Utilities folder from the web-accessible folder where you’ll upload the VisualPHPUnit package: set_include_path(get_include_path() . PATH_SEPARATOR . dirname(__FILE__) . '/../Utilities'); set_include_path(get_include_path() . PATH_SEPARATOR . dirname(__FILE__) . '/../Utilities' . '/PHPUnit');

Now upload your VisualPHPUnit folder to a web-accessible location on your server. Fire up your browser, open your VisualPHPUnit location, and see if you feel lucky.

Well do you, punk?

If your stars have aligned now you should have a working install of PHPUnit with VisualPHPUnit. If not, it’s troubleshooting time!

“Fatal error: require_once() [function.require]: Failed opening required ‘PHPUnit/Autoload.php'”

Your include paths in VisualPHPUnit/app/config/bootstrap.php are wrong, sort them out

“Fatal error: require_once() [function.require]: Failed opening required {any other file related to phpunit}“

You missed a PHPUnit dependency, you can find all the packages you need (and then some more) at https://github.com/sebastianbergmann?tab=repositories

“404: Not Found” error / You just get the Index of your /VisualPHPUnit folder

You’ve got the curse of the .htaccess, brought on by lazy open-source web development practices. There are 2 .htaccess files in VisualPHPUnit that are used to redirect requests, and we need to fix them.

First open the .htaccess file in the “VisualPHPUnit” folder.

  • Find the following line:
    RewriteRule ^$ app/public/ [L] add an [R] flag so that you can see the redirected address in your browser, so the line becomes:
    RewriteRule ^$ app/public/ [L,R]
  • Go to your web browser, reload your VisualPHPUnit tab, and check the address to which you’re being redirected. It should be {your VisualPHPUnit location}/app/public/ . Since it’s not, find the following line in your .htaccess file:
    RewriteEngine on and insert the following line after it: RewriteBase /
  • Reload again your browser tab, and check where you’re being redirected now. Part of the path will probably be missing in the middle, so you’ll need to edit the line you just added so that your browser ends up redirecting you correctly to {your VisualPHPUnit location}/app/public/ . In my case I had to change it to this:
    RewriteBase /VisualPHPUnit/
  • Once you get redirected to the correct address, you need to remove the [R] flag you set at the beginning, so that the line you modified to
    RewriteRule ^$ app/public/ [L,R] becomes once again
    RewriteRule ^$ app/public/ [L]
Your browser should load VisualPHPUnit now, but tests will not work since we need to fix the remaining redirection. Go into VisualPHPUnit/app/public and open the .htaccess file there.
  • Find the following line:
    RewriteRule ^(.*)$ index.php [QSA,L] and insert the [R] redirection flag as before, so the line becomes:
    RewriteRule ^(.*)$ index.php [QSA,L,R]
  • Go to your browser and open the link on the top of your VisualPHPUnit tab that says “Archives”. Check the address bar to see where you are being redirected. You should be redirected to {your VisualPHPUnit location}/app/public/index.php . Since you’re not, you know the drill, find the following line in your .htaccess file:
    RewriteEngine on and insert the following line after it: RewriteBase /
  • Reload your browser tab, check where you’re being redirected and what’s missing in the path, and modify the line you just added until the redirection takes you to the correct address. In my case, I had to change it to this:
    RewriteBase /VisualPHPUnit/app/public/
  • Once you get redirected to the correct address, you need to remove the [R] flag you set, so that the line you modified to
    RewriteRule ^(.*)$ index.php [QSA,L,R] becomes once again
    RewriteRule ^(.*)$ index.php [QSA,L]
Now you should be able to see the files in your test directory and run tests.

Tests don’t run and return empty results

You’re using a PHPUnit version that requires a newer PHP version to run. Get an older version of PHPUnit.

“Fatal error: Uncaught exception ‘PHPUnit_Framework_Error_Notice’ with message ‘Non-static method PHP_Timer::start() should not be called statically, assuming $this from incompatible context'” when you try running tests

You’ve got the curse of the shitty non-backwards-compatible dependency refactoring, go into your Utilities folder and open PHPUnit/Framework/TestResult.php and make the following changes:

  • Find the following line:
    PHP_Timer::start(); comment it out or delete it, and insert the following 2 lines after it: $timer = new PHP_Timer; $timer->start();
  • Find the following line:
    $time = PHP_Timer::stop(); comment it out or delete it, and insert the following line after it: $time = $timer->stop();
Your tests should run fine now.