As I mentioned in a previous blog post, Rails 2.1 natively supports incoming JSON requests. Unfortunately, it still struggles with JSON data containing non-ASCII characters.
According to the JSON spec, JSON fully supports UTF-8 encoded text, so with a few exceptions it generally should not be necessary to escape non-ASCII characters with \u Unicode escape sequences. However, many JSON libraries appear to escape all non-ASCII text in this fashion. This in itself should not be a problem, but
ActiveSupport::JSON currently does not properly parse JSON containing \u escapes, resulting in strings with literal \u escape sequences rather than the desired UTF-8 encoded characters. This is especially confusing since
ActiveSupport:JSON itself encodes all non-ASCII characters as \u escapes, so one might think that the reverse transformation yields the original data. But this behavior is likely explained by an odd implementation choice for its decoder: Rather than using the
json-pure) library, it converts the JSON data to YAML and then uses the YAML library to decode the data into Ruby objects.
Monkey-patching to the rescue! I decided to replace
ActiveSupport::JSON::decode with an implementation that uses the
json library. The easiest way is to stick the following code into a file named something like
activesupport_json_unicode_patch.rb inside the
config/initializers/ directory, where Rails will automatically pick it up.
You can verify the fix by adding a test case (I added a file named
activesupport_json_test.rb to the
This test should fail without the patch and pass after adding it.
In addition to fixing the JSON / Unicode problem, this patch should also provide a nice speed boost, as we’re replacing the somewhat roundabout YAML based JSON decode method with a native one (particularly if you’re using the native
json implementation rather than