[CMS] Map locale names to match what Crowdin expects during synchronisation

[kzar]

Background

In issue #3049 it was reported the wrong language (English) was being served for canonical URLs on adblockbrowser.org when the Accept-Language header contained a hyphen. This regression was caused when locale directories were renamed to match the names Crowdin expected.

Now that we have to undo that change we will again be unable to easily synchronise translations with Crowdin. We will therefore need a way to map our locale names to the Crowdin equivalents.

What to change

Update the Crowdin synchronisation script to map our locale names to the Crowdin equivalents. To turn a local locale name into the Crowdin version the following steps should be taken:

1.) If the name contains an underscore, we can just replace it with a hyphen.
2.) Otherwise check if the name is supported by Crowdin using the ​supported languages API. If already supported, we're done without modifications.
3.) Otherwise take an upper-case version of the locale name and append it after a hyphen. For example "xx" becomes "xx-XX". If that's supported we're now done.
4.) (If at step 1 or 3 we realise the locale is unsupported, the _unmodified_ locale name should be used when issuing a warning to the user.)

Example mappings:

es = es-ES
de = de
pt = pt-PT
zh_TW = zh-TW

[sebastian]

The background section is inaccurate. #3049 isn't a regression. This issue existed ever since for adblockbrowser.org. We used CrowdIn locales there from the beginning.

Moreover, I don't think this mapping need to be configurable but should rather be hard-coded to enforce consistency and single point of failure. As far as I can tell it can even be expressed with a simple function:

def convert_from_crowdin_locale(locale)
  parts = locale.split('-')
  if len(parts) == 1 or parts[0] == parts[1].lower():
    return parts[0]
  return '_'.join(parts[:2])

[kzar]

Replying to sebastian:

The background section is inaccurate. #3049 isn't a regression. This issue existed ever since for adblockbrowser.org. We used CrowdIn locales there from the beginning.

Well I thought so too but then I noticed the change was made in ​d19bfbe47ed8. (However that commit was pushed before the site went live I suppose.)

Moreover, I don't think this mapping need to be configurable but should rather be hard-coded to enforce consistency and single point of failure. As far as I can tell it can even be expressed with a simple function:

def convert_from_crowdin_locale(locale)
  parts = locale.split('-')
  if len(parts) == 1 or parts[0] == parts[1].lower():
    return parts[0]
  return '_'.join(parts[:2])

Perhaps we can do this without manually configuring the mappings, but in that case we will also need a function to convert from the local locale names to the Crowdin ones. How can that function know that "es" should be "es-ES" but "de" should remain as "de"? (I certainly agree that making this automatic is desirable.)

[sebastian]

My point was that this doesn't need to be configurable. But should rather be a hard-coded mapping, if not a function. Of course I'd prefer to use the algorithm above rather than hard-coding the mapping. But sure, it doesn't work the other way around. But if we know the list of CrowdIn locales at runtime, we can easily build a reverse mapping using the same function. Or we can try different variants at runtime, so if there is no es on CrowdIn, fallback to es-ES.

[kzar]

Whoops, of course. I was thinking about this last night and realised it was actually pretty simple to implement with the mapping working automatically like you suggested. I've updated the ticket.

[kzar]

​https://hg.adblockplus.org/cms/rev/96297eb4bc64