Automatically find, translate and save missing translation keys.
For a project that I am managing with Vue and Blade files, I wanted to support another language (Dutch) and, of course, without too much of a hassle.
I therefore wrote a simple Laravel command that:
finds all the translation keys in the source files (Blade & Vue)
compares them with the translation keys that were already saved to find the new ones
translates the new keys automatically to Dutch
saves the new keys with the translations
Writing the translation keys
Laravel supports two ways of adding translation keys:
The first way is to have a group followed by a key like this: @lang('messages.welcome')
Those files are saved as php files. In the above case, it would be: /resources/lang/en/messages.php and for the dutch version /resources/lang/nl/messages.php
Since Laravel version 5.6, you can use translation strings as keys. The above example would instead look like @lang('Welcome to my website!')
You notice that the group is gone and the actual text has become the key. This has the big advantage, that the output will always look ok, even if it's not added as a key to your language file. The other big difference is that it will be written to a json format in stead of php which is ideal for this type of project as I will need to load the translations inside our vue components.
For vue, I added the vue-i18n dependency. Translatable strings are written either as as $t('Welcome to my website!') in the template files or outside the templates with the following format: i18n.t('Welcome to my website!')
Find all the translation keys in the source files
private function findKeysInFiles(): array
{
$path = [resource_path('js/views'), resource_path('views')];
$functions = ['\$t', 'i18n.t', '@lang'];
$pattern =
"[^\w|>]". // Must not have an alphanum or _ or > before real method
"(".implode('|', $functions) .")". // Must start with one of the functions
"\(". // Match opening parentheses
"[\'\"]". // Match " or '
"(". // Start a new group to match:
"([^\1)]+)+". // this is the key/value
")". // Close group
"[\'\"]". // Closing quote
"[\),]"; // Close parentheses or new parameter
$finder = new Finder();
$finder->in($path)->exclude('storage')->name(['*.vue', '*.php'])->files();
$keys = [];
foreach ($finder as $file) {
if (preg_match_all("/$pattern/siU", $file->getContents(), $matches)) {
foreach ($matches[2] as $key) {
$keys[$key] = '';
}
}
}
return $keys;
}
The code above loops over all our vue and blade files and will capture anything between $t(''), i18n.t('') (for vue) or @lang('') (for blade). The regex used for this was copied from Laravel translation manager.
A big thank you to Barry van den Heuvel because regex is a bit like the barf of programming languages :) more meme wisdom
get the translation keys that were already saved
private function loadAllSavedTranslations(): array
{
$path = Storage::disk(self::STORAGE_DISK)->path('');
$finder = new Finder();
$finder->in($path)->name(['*.json'])->files();
$translations = [];
foreach ($finder as $file) {
$locale = $file->getFilenameWithoutExtension();
if (!in_array($locale, self::LOCALES)) {
continue;
}
$this->info('loading: ' . $locale);
$jsonString = $file->getContents();
$translations[$locale] = json_decode($jsonString, true);
}
return $translations;
}
The above code, will load the keys of the language files that we already have saved. In our case: en.json and nl.json.
We need to load those so we can later add our new keys to it.
Translates the new keys automatically to Dutch
private function translateAndSaveNewKeys(array $translationsKeys, array $alreadyTranslated)
{
foreach (self::LOCALES as $locale) {
$newKeysFound = array_diff_key($translationsKeys, $alreadyTranslated[$locale]);
if (count($newKeysFound) < 1) {
continue;
}
$newKeysWithValues = $this->translateKeys($locale, $newKeysFound);
$this->saveToFile($locale, $newKeysWithValues, $alreadyTranslated[$locale]);
}
}
With the results of the functions above, we can now get the new keys with arraydiffkey and translate them.
Of course, it is not recommended to use these translations in production! As a first draft however, it is very handy.
For this I used the Google Translate API PHP Package
After installing the package and adding GoogleTranslate as a dependency injection to the class, you can simply call:
$this->googleTranslate->setTarget($locale);
$translated = $this->googleTranslate->translate($key);
Save the new keys with the translations
private function saveToFile(string $locale, array $newKeysWithValues, array $alreadyTranslated)
{
$localeTranslations = array_merge($newKeysWithValues, $alreadyTranslated);
uksort($localeTranslations, 'strnatcasecmp');
Storage::disk(self::STORAGE_DISK)
->put($locale .'.json', json_encode($localeTranslations, JSON_UNESCAPED_UNICODE | JSON_PRETTY_PRINT));
}
array_merge will combine the keys that we have already saved with our new keys. Basically only adding the new keys so existing translations will not get overwritten.
uksort($localeTranslations, 'strnatcasecmp'); sorts the keys alphabetically but case insensitive.
The JSONPRETTYPRINT flag writes each key on a new line resulting in the following nl.json file:
{
"Account deleted": "Account verwijderd",
"all": "alle",
"allowed attempts:": "toegestane pogingen:",
"All rights reserved.": "Alle rechten voorbehouden.",
"Already have an account?": "Heb je al een account?",
"An invitation was sent to {email} to accept your challenge.": "Een uitnodiging is verzonden naar {email} om je uitdaging te accepteren.",
"Answer": "Antwoord",
...
All the code for the command:
namespace App\Console\Commands;
use Exception;
use Illuminate\Console\Command;
use Illuminate\Support\Facades\Log;
use Illuminate\Support\Facades\Storage;
use Stichoza\GoogleTranslate\GoogleTranslate;
use Symfony\Component\Finder\Finder;
class FindAndAddLanguageKeysCommand extends Command
{
const STORAGE_DISK = 'lang';
const LOCALES = ['nl', 'en'];
/**
* @var string
*/
protected $signature = 'language:find-and-add';
/**
* @var string
*/
protected $description = 'Automatically find, translate and save missing translation keys.';
private GoogleTranslate $googleTranslate;
public function __construct(GoogleTranslate $googleTranslate)
{
$this->googleTranslate = $googleTranslate;
parent::__construct();
}
public function handle()
{
$alreadyTranslated = $this->loadAllSavedTranslations();
$translationsKeys = $this->findKeysInFiles();
$this->translateAndSaveNewKeys($translationsKeys, $alreadyTranslated);
$this->info('success');
}
private function loadAllSavedTranslations(): array
{
$path = Storage::disk(self::STORAGE_DISK)->path('');
$finder = new Finder();
$finder->in($path)->name(['*.json'])->files();
$translations = [];
foreach ($finder as $file) {
$locale = $file->getFilenameWithoutExtension();
if (!in_array($locale, self::LOCALES)) {
continue;
}
$this->info('loading: ' . $locale);
$jsonString = $file->getContents();
$translations[$locale] = json_decode($jsonString, true);
}
return $translations;
}
private function findKeysInFiles(): array
{
$path = [resource_path('js/views'), resource_path('views')];
$functions = ['\$t', 'i18n.t', '@lang'];
$pattern =
"[^\w|>]". // Must not have an alphanum or _ or > before real method
"(".implode('|', $functions) .")". // Must start with one of the functions
"\(". // Match opening parenthese
"[\'\"]". // Match " or '
"(". // Start a new group to match:
"([^\1)]+)+". // this is the key/value
")". // Close group
"[\'\"]". // Closing quote
"[\),]"; // Close parentheses or new parameter
$finder = new Finder();
$finder->in($path)->exclude('storage')->name(['*.vue', '*.php'])->files();
$this->info('> ' . $finder->count() . ' vue & php files found');
$keys = [];
foreach ($finder as $file) {
if (preg_match_all("/$pattern/siU", $file->getContents(), $matches)) {
if (count($matches) < 2) {
continue;
}
$this->info('>> ' . count($matches[2]) . ' keys found for ' . $file->getFilename());
foreach ($matches[2] as $key) {
if (strlen($key) < 2) {
continue;
}
$keys[$key] = '';
}
}
}
uksort($keys, 'strnatcasecmp');
return $keys;
}
private function translateAndSaveNewKeys(array $translationsKeys, array $alreadyTranslated)
{
foreach (self::LOCALES as $locale) {
$newKeysFound = array_diff_key($translationsKeys, $alreadyTranslated[$locale]);
if (count($newKeysFound) < 1) {
continue;
}
$this->info(count($newKeysFound) . ' new keys found for "' . $locale . '"');
$newKeysWithValues = $this->translateKeys($locale, $newKeysFound);
$this->saveToFile($locale, $newKeysWithValues, $alreadyTranslated[$locale]);
}
}
private function translateKeys(string $locale, array $keys): array
{
foreach ($keys as $keyIndex => $keyValue) {
$keys[$keyIndex] = $this->translateKey($locale, $keyIndex);
}
return $keys;
}
private function translateKey(string $locale, string $key): string
{
if ($locale === 'en') {
return $key;
}
try {
$this->googleTranslate->setTarget($locale);
$translated = $this->googleTranslate->translate($key);
} catch (Exception $exception) {
Log::warning('Google translate issue with ' . $key . ': ' . $exception->getMessage());
$translated = $key;
}
return $translated;
}
private function saveToFile(string $locale, array $newKeysWithValues, array $alreadyTranslated)
{
$localeTranslations = array_merge($newKeysWithValues, $alreadyTranslated);
uksort($localeTranslations, 'strnatcasecmp');
Storage::disk(self::STORAGE_DISK)
->put($locale .'.json', json_encode($localeTranslations, JSON_UNESCAPED_UNICODE | JSON_PRETTY_PRINT));
}
}