Remove Apple unnecessary empty lines

Signed-off-by: Roman Tsisyk <roman@tsisyk.com>

.github/workflows/test.yml

@@ -0,0 +1,37 @@
+# This workflow uses actions that are not certified by GitHub.
+# They are provided by a third-party and are governed by
+# separate terms of service, privacy policy, and support
+# documentation.
+# This workflow will download a prebuilt Ruby version, install dependencies and run tests with Rake
+# For more information see: https://github.com/marketplace/actions/setup-ruby-jruby-and-truffleruby
+
+name: Test
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+
+permissions:
+  contents: read
+
+jobs:
+  test:
+
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        ruby-version: ['2.6', '2.7', '3.0', '3.1']
+
+    steps:
+    - uses: actions/checkout@v3
+    - name: Set up Ruby
+      uses: ruby/setup-ruby@v1
+      with:
+        ruby-version: ${{ matrix.ruby-version }}
+        bundler-cache: true # runs 'bundle install' and caches installed gems automatically
+    - name: Install dependencies
+      run: bundle install
+    - name: Run tests
+      run: bundle exec rake test

.gitignore

@@ -1,3 +1,5 @@
 *.gem
 .idea/
 *.lock
+.ruby-version
+.DS_Store

CHANGELOG.md

@@ -0,0 +1,59 @@
+# 1.1.2 (2022-11-15)
+
+- Bugfix: Fixed a runtime error caused by a missing rexml dependency in Ruby 3 (#312)
+
+# 1.1.1 (2021-01-28)
+
+- Bugfix: Properly parse multiline comments in Android XML files (#300)
+
+# 1.1 (2020-07-09)
+
+- Feature: Add --escape-all-tags option to force escaping of Android styling tags (#281)
+- Improvement: Twine now requires Ruby 2.4 or greater and rubyzip 2.0 or greater (#297)
+- Bugfix: Fix issues with the Django formatter (#289)
+
+# 1.0.6 (2019-05-28)
+
+- Improvement: Support more Android styling tags (#278)
+- Improvement: Update Android output path for default language (#276)
+
+# 1.0.5 (2019-02-24)
+
+- Bugfix: Incorrect language detection when reading localization files (#251)
+- Bugfix: Double quotes in Android files could be converted to single quotes (#254)
+- Bugfix: Properly escape quotes when writing gettext files (#268)
+
+# 1.0.4 (2018-05-30)
+
+- Feature: Add a --quiet option (#245)
+- Bugfix: Consume child HTML tags in Android formatter (#247)
+- Bugfix: Let consume-localization-archive return a non-zero status (#246)
+
+# 1.0.3 (2018-01-26)
+
+- Bugfix: Workaround a possible crash in safe_yaml (#237)
+- Bugfix: Fix an error caused by combining %@ with other placeholders (#235)
+
+# 1.0.2 (2018-01-20)
+
+- Improvement: Better support for placeholders in HTML styled Android strings (#212)
+
+# 1.0.1 (2017-10-17)
+
+- Bugfix: Always prefer the passed-in formatter (#221)
+
+# 1.0 (2017-10-16)
+
+- Feature: Fail twine commands if there's more than one formatter candidate (#201)
+- Feature: In the Apple formatter, use developer language for the base localization (#219)
+- Bugfix: Preserve basic HTML styling for Android strings (#212)
+- Bugfix: `twine --version` reports "unknown" (#213)
+- Bugfix: Support spaces in command line arguments (#206)
+- Bugfix: 'r' missing before region code in Android output folder (#203)
+- Bugfix: .po formatter language detection (#199)
+- Bugfix: Add 'q' placeholder for android strings (#194)
+- Bugfix: Boolean command line parameters are always true (#191)
+
+# 0.10.1 (2017-01-19)
+
+- Bugfix: Xcode integration (#184)

ISSUE_TEMPLATE.md

@@ -0,0 +1,6 @@
+If you're reporting a bug, please do the following:
+
+- Mention the Twine version you're using
+- Provide a minimal but complete input file causing the error (inline with triple ` is fine)
+- Quote the exact twine command that's being run
+- Provide the complete _expected_ output (again, inline is fine)

README.md

@@ -1,157 +1,168 @@
 # Twine
 
-Twine is a command line tool for managing your strings and their translations. These strings are all stored in a master text file and then Twine uses this file to import and export strings in a variety of file types, including iOS and Mac OS X `.strings` files, Android `.xml` files, gettext `.po` files, and [jquery-localize][jquerylocalize] `.json` files. This allows individuals and companies to easily share strings across multiple projects, as well as export strings in any format the user wants.
+Twine is a command line tool for managing your strings and their translations. These are all stored in a single text file and then Twine uses this file to import and export localization files in a variety of types, including iOS and Mac OS X `.strings` files, Android `.xml` files, gettext `.po` files, and [jquery-localize][jquerylocalize] `.json` files. This allows individuals and companies to easily share translations across multiple projects, as well as export localization files in any format the user wants.
 
 ## Install
 
-### As a Gem
-
 Twine is most easily installed as a Gem.
 
 	$ gem install twine
 
-### From Source
+## Twine File Format
 
-You can also run Twine directly from source. However, it requires [rubyzip][rubyzip] in order to create and read standard zip files.
+Twine stores everything in a single file, the Twine data file. The format of this file is a slight variant of the [Git][git] config file format, which itself is based on the old [Windows INI file][INI] format. The entire file is broken up into sections, which are created by placing the section name between two pairs of square brackets. Sections are optional, but they are the recommended way of grouping your definitions into smaller, more manageable chunks.
 
-	$ gem install rubyzip
-	$ git clone git://github.com/mobiata/twine.git
-	$ cd twine
-	$ ./twine --help
+Each grouping section contains N definitions. These definitions start with the key placed within a single pair of square brackets. It then contains a number of key-value pairs, including a comment, a comma-separated list of tags and all of the translations.
 
-Make sure you run the `twine` executable at the root of the project as it properly sets up your Ruby library path. The `bin/twine` executable does not.
+### Placeholders
 
-## String File Format
-
-Twine stores all of its strings in a single file. The format of this file is a slight variant of the [Git][git] config file format, which itself is based on the old [Windows INI file][INI] format. The entire file is broken up into sections, which are created by placing the section name between two pairs of square brackets. Sections are optional, but they are a recommended way of breaking your strings into smaller, more manageable chunks.
-
-Each grouping section contains N string definitions. These string definitions start with the string key placed within a single pair of square brackets. This string definition then contains a number of key-value pairs, including a comment, a comma-separated list of tags (which are used by Twine to select a subset of strings), and all of the translations.
+Twine supports [`printf` style placeholders][printf] with one peculiarity: `@` is used for strings instead of `s`. This is because Twine started out as a tool for iOS and OS X projects.
 
 ### Tags
 
-Tags are used by Twine as a way to only work with a subset of your strings at any given point in time. Each string can be assigned zero or more tags which are separated by commas. Tags are optional, though highly recommended. You can get a list of all strings currently missing tags by executing the `generate-report` command.
+Tags are used by Twine as a way to only work with a subset of your definitions at any given point in time. Each definition can be assigned zero or more tags which are separated by commas. Tags are optional, though highly recommended. You can get a list of all definitions currently missing tags by executing the [`validate-twine-file`](#validate-twine-file) command with the `--pedantic` option.
+
+When generating a localization file, you can specify which definitions should be included using the `--tags` option. Provide a comma separated list of tags to match all definitions that contain any of the tags (`--tags tag1,tag2` matches all definitions tagged with `tag1` _or_ `tag2`). Provide multiple `--tags` options to match definitions containing all specified tags (`--tags tag1 --tags tag2` matches all definitions tagged with `tag1` _and_ `tag2`). You can match definitions _not_ containing a tag by prefixing the tag with a tilde (`--tags ~tag1` matches all definitions _not_ tagged with `tag1`). All three options are combinable.
 
 ### Whitespace
 
 Whitepace in this file is mostly ignored. If you absolutely need to put spaces at the beginning or end of your translated string, you can wrap the entire string in a pair of `` ` `` characters. If your actual string needs to start *and* end with a grave accent, you can wrap it in another pair of `` ` `` characters. See the example, below.
 
+### References
+
+If you want a definition to inherit the values of another definition, you can use a reference. Any property not specified for a definition will be taken from the reference.
+
 ### Example
 
-	[[General]]
-		[yes]
-			en = Yes
-			es = Sí
-			fr = Oui
-			ja = はい
-		[no]
-			en = No
-			fr = Non
-			ja = いいえ
-	
-	[[Errors]]
-		[path_not_found_error]
-			en = The file '%@' could not be found.
-			tags = app1,app6
-			comment = An error describing when a path on the filesystem could not be found.
-		[network_unavailable_error]
-			en = The network is currently unavailable.
-			tags = app1
-			comment = An error describing when the device can not connect to the internet.
-	
-	[[Escaping Example]]
-		[list_item_separator]
-			en = `, `
-			tags = mytag
-			comment = A string that should be placed between multiple items in a list. For example: Red, Green, Blue
-		[grave_accent_quoted_string]
-			en = ``%@``
-			tags = myothertag
-			comment = This string will evaluate to `%@`.
+```ini
+[[General]]
+	[yes]
+		en = Yes
+		es = Sí
+		fr = Oui
+		ja = はい
+	[no]
+		en = No
+		fr = Non
+		ja = いいえ
+
+[[Errors]]
+	[path_not_found_error]
+		en = The file '%@' could not be found.
+		tags = app1,app6
+		comment = An error describing when a path on the filesystem could not be found.
+	[network_unavailable_error]
+		en = The network is currently unavailable.
+		tags = app1
+		comment = An error describing when the device can not connect to the internet.
+	[dismiss_error]
+		ref = yes
+		en = Dismiss
+
+[[Escaping Example]]
+	[list_item_separator]
+		en = `, `
+		tags = mytag
+		comment = A string that should be placed between multiple items in a list. For example: Red, Green, Blue
+	[grave_accent_quoted_string]
+		en = ``%@``
+		tags = myothertag
+		comment = This string will evaluate to `%@`.
+```
 
 ## Supported Output Formats
 
-Twine currently supports the following formats for outputting strings:
+Twine currently supports the following output formats:
 
 * [iOS and OS X String Resources][applestrings] (format: apple)
 * [Android String Resources][androidstrings] (format: android)
+    * HTML tags will be escaped by replacing `<` with `&lt`
+    * Tags inside `<![CDATA[` won't be escaped.
+    * Supports [basic styling][androidstyling] according to [Android documentation](https://developer.android.com/guide/topics/resources/string-resource.html#StylingWithHTML).  All of the documented tags are supported, in addition to `<a>` links.
+	    * These tags will *not* be escaped if the string doesn't contain placeholders. You can reference them directly in your layouts or by using [`getText()`](https://developer.android.com/reference/android/content/res/Resources.html#getText(int)) to read them programmatically.
+	    * These tags *will* be escaped if the string contains placeholders. You can use [`getString()`](https://developer.android.com/reference/android/content/res/Resources.html#getString(int,%20java.lang.Object...)) combined with [`fromHtml`](https://developer.android.com/reference/android/text/Html.html#fromHtml(java.lang.String)) as shown in the [documentation][androidstyling] to display them.
+	* See [\#212](https://github.com/scelis/twine/issues/212) for details.
 * [Gettext PO Files][gettextpo] (format: gettext)
 * [jquery-localize Language Files][jquerylocalize] (format: jquery)
 * [Django PO Files][djangopo] (format: django)
 * [Tizen String Resources][tizen] (format: tizen)
+* [Flash/Flex Properties][flash] (format: flash)
 
-If you would like to enable twine to create language files in another format, create an appropriate formatter in `lib/twine/formatters`.
+If you would like to enable Twine to create localization files in another format, read the wiki page on how to create an appropriate formatter.
 
 ## Usage
 
-	Usage: twine COMMAND STRINGS_FILE [INPUT_OR_OUTPUT_PATH] [--lang LANG1,LANG2...] [--tags TAG1,TAG2,TAG3...] [--format FORMAT]
-	
+	Usage: twine COMMAND TWINE_FILE [INPUT_OR_OUTPUT_PATH] [--lang LANG1,LANG2...] [--tags TAG1,TAG2,TAG3...] [--format FORMAT]
+
 ### Commands
 
-#### `generate-string-file`
+#### `generate-localization-file`
 
-This command creates an Apple or Android strings file from the master strings data file.
+This command creates a localization file from the Twine data file. If the output file would not contain any translations, Twine will exit with an error.
 
-	$ twine generate-string-file /path/to/strings.txt values-ja.xml --tags common,app1
-	$ twine generate-string-file /path/to/strings.txt Localizable.strings --lang ja --tags mytag
-	$ twine generate-string-file /path/to/strings.txt all-english.strings --lang en
+	$ twine generate-localization-file /path/to/twine.txt values-ja.xml --tags common,app1
+	$ twine generate-localization-file /path/to/twine.txt Localizable.strings --lang ja --tags mytag
+	$ twine generate-localization-file /path/to/twine.txt all-english.strings --lang en
 
-#### `generate-all-string-files`
+#### `generate-all-localization-files`
 
-This command is a convenient way to call `generate-string-file` multiple times. It uses standard Mac OS X, iOS, and Android conventions to figure out exactly which files to create given a parent directory. For example, if you point it to a parent directory containing `en.lproj`, `fr.lproj`, and `ja.lproj` subdirectories, Twine will create a `Localizable.strings` file of the appropriate language in each of them. This is often the command you will want to execute during the build phase of your project.
+This command is a convenient way to call [`generate-localization-file`](#generate-localization-file) multiple times. It uses standard conventions to figure out exactly which files to create given a parent directory. For example, if you point it to a parent directory containing `en.lproj`, `fr.lproj`, and `ja.lproj` subdirectories, Twine will create a `Localizable.strings` file of the appropriate language in each of them. However, files that would not contain any translations will not be created; instead warnings will be logged to `stderr`. This is often the command you will want to execute during the build phase of your project.
 
-	$ twine generate-all-string-files /path/to/strings.txt /path/to/project/locales/directory --tags common,app1
+	$ twine generate-all-localization-files /path/to/twine.txt /path/to/project/locales/directory --tags common,app1
 
-#### `consume-string-file`
+#### `consume-localization-file`
 
-This command slurps all of the strings from a `.strings` or `.xml` file and incorporates the translated text into the master strings data file. This is a simple way to incorporate any changes made to a single file by one of your translators. It will only identify strings that already exist in the master data file.
+This command slurps all of the translations from a localization file and incorporates the translated strings into the Twine data file. This is a simple way to incorporate any changes made to a single file by one of your translators. It will only identify definitions that already exist in the data file.
 
-	$ twine consume-string-file /path/to/strings.txt fr.strings
-	$ twine consume-string-file /path/to/strings.txt Localizable.strings --lang ja
-	$ twine consume-string-file /path/to/strings.txt es.xml
+	$ twine consume-localization-file /path/to/twine.txt fr.strings
+	$ twine consume-localization-file /path/to/twine.txt Localizable.strings --lang ja
+	$ twine consume-localization-file /path/to/twine.txt es.xml
 
-#### `consume-all-string-files`
+#### `consume-all-localization-files`
 
-This command reads in a folder containing many `.strings` or `.xml` files. These files should be in a standard folder hierarchy so that twine knows the language of each file. When combined with the `--developer-language`, `--consume-comments`, and `--consume-all` flags, this command is a great way to create your initial strings data file from an existing iOS or Android project. Just make sure that you create a blank strings.txt file, first!
+This command reads in a folder containing many localization files. These files should be in a standard folder hierarchy so that Twine knows the language of each file. When combined with the `--developer-language`, `--consume-comments`, and `--consume-all` flags, this command is a great way to create your initial Twine data file from an existing project. Just make sure that you create a blank Twine data file first!
 
-	$ twine consume-all-string-files strings.txt Resources/Locales --developer-language en --consume-all --consume-comments
+	$ twine consume-all-localization-files twine.txt Resources/Locales --developer-language en --consume-all --consume-comments
 
-#### `generate-loc-drop`
+#### `generate-localization-archive`
 
-This command is a convenient way to generate a zip file containing files created by the `generate-string-file` command. It is often used for creating a single zip containing a large number of strings in all languages which you can then hand off to your translation team.
+This command is a convenient way to generate a zip file containing files created by the [`generate-localization-file`](#generate-localization-file) command. If a file would not contain any translated strings, it is skipped and a warning is logged to `stderr`. This command can be used to create a single zip containing a large number of translations in all languages which you can then hand off to your translation team.
 
-	$ twine generate-loc-drop /path/to/strings.txt LocDrop1.zip
-	$ twine generate-loc-drop /path/to/strings.txt LocDrop2.zip --lang en,fr,ja,ko --tags common,app1
+	$ twine generate-localization-archive /path/to/twine.txt LocDrop1.zip
+	$ twine generate-localization-archive /path/to/twine.txt LocDrop2.zip --lang en,fr,ja,ko --tags common,app1
 
-#### `consume-loc-drop`
+#### `consume-localization-archive`
 
-This command is a convenient way of taking a zip file and executing the `consume-string-file` command on each file within the archive. It is most often used to incorporate all of the changes made by the translation team after they have completed work on a localization drop.
+This command is a convenient way of taking a zip file and executing the [`consume-localization-file`](#consume-localization-file) command on each file within the archive. It is most often used to incorporate all of the changes made by the translation team after they have completed work on a localization archive.
 
-	$ twine consume-loc-drop /path/to/strings.txt LocDrop2.zip
+	$ twine consume-localization-archive /path/to/twine.txt LocDrop2.zip
 
-#### `generate-report`
+#### `validate-twine-file`
 
-This command gives you useful information about your strings. It will tell you how many strings you have, how many have been translated into each language, and whether your master strings data file has any duplicate string keys.
+This command validates that the Twine data file can be parsed, contains no duplicate keys, and that no key contains invalid characters. It will exit with a non-zero status code if any of those criteria are not met.
 
-	$ twine generate-report /path/to/strings.txt
+	$ twine validate-twine-file /path/to/twine.txt
 
-## Creating Your First strings.txt File
+## Creating Your First Twine Data File
 
-The easiest way to create your first strings.txt file is to run the `consume-all-string-files` command. The one caveat is to first create a blank strings.txt file to use as your starting point. Then, just point the `consume-all-string-files` command at a directory in your project containing all of your iOS, OS X, or Android strings files.
+The easiest way to create your first Twine data file is to run the [`consume-all-localization-files`](#consume-all-localization-files) command. The one caveat is to first create a blank file to use as your starting point. Then, just point the `consume-all-localization-files` command at a directory in your project containing all of your localization files.
 
-	$ touch strings.txt
-	$ twine consume-all-string-files strings.txt Resources/Locales --developer-language en --consume-all --consume-comments
+	$ touch twine.txt
+	$ twine consume-all-localization-files twine.txt Resources/Locales --developer-language en --consume-all --consume-comments --format apple/android/gettext/jquery/django/tizen/flash
 
 ## Twine and Your Build Process
 
+### Xcode
+
 It is easy to incorporate Twine right into your iOS and OS X app build processes.
 
 1. In your project folder, create all of the `.lproj` directories that you need. It does not really matter where they are. We tend to put them in `Resources/Locales/`.
-2. Run the `generate-all-string-files` command to create all of the string files you need in these directories. For example,
+2. Run the [`generate-all-localization-files`](#generate-all-localization-files) command to create all of the `.strings` files you need in these directories. For example,
 
-		$ twine generate-all-string-files strings.txt Resources/Locales/ --tags tag1,tag2
+		$ twine generate-all-localization-files twine.txt Resources/Locales/ --tags tag1,tag2
 
-	Make sure you point Twine at your strings data file, the directory that contains all of your `.lproj` directories, and the tags that describe the strings you want to use for this project.
-3. Drag the `Resources/Locales/` directory to the Xcode project navigator so that Xcode knows to include all of these strings files in your build.
+	Make sure you point Twine at your data file, the directory that contains all of your `.lproj` directories, and the tags that describe the definitions you want to use for this project.
+3. Drag the `Resources/Locales/` directory to the Xcode project navigator so that Xcode knows to include all of these `.strings` files in your build.
 4. In Xcode, navigate to the "Build Phases" tab of your target.
 5. Click on the "Add Build Phase" button and select "Add Run Script".
 6. Drag the new "Run Script" build phase up so that it runs earlier in the build process. It doesn't really matter where, as long as it happens before the resources are copied to your bundle.
@@ -159,44 +170,82 @@ It is easy to incorporate Twine right into your iOS and OS X app build processes
 
 Now, whenever you build your application, Xcode will automatically invoke Twine to make sure that your `.strings` files are up-to-date.
 
+### Android Studio/Gradle
+
+#### Standard
+
+Add the following code to `app/build.gradle`:
+
+```
+task generateLocalizations {
+	String script = 'if hash twine 2>/dev/null; then twine generate-localization-file twine.txt ./src/main/res/values/generated_strings.xml; fi'
+	exec {
+		executable "sh"
+		args '-c', script
+	}
+}
+
+preBuild {
+	dependsOn generateLocalizations
+}
+```
+
+#### Using [jruby](http://jruby.org)
+
+With this approach, developers do not need to manually install ruby, gem, or twine.
+
+Add the following code to `app/build.gradle`:
+
+```
+buildscript {
+	repositories { jcenter() }
+
+	dependencies {
+		/* NOTE: Set your preferred version of jruby here. */
+		classpath "com.github.jruby-gradle:jruby-gradle-plugin:1.5.0"
+	}
+}
+
+apply plugin: 'com.github.jruby-gradle.base'
+
+dependencies {
+	/* NOTE: Set your preferred version of twine here. */
+	jrubyExec 'rubygems:twine:1.1'
+}
+
+task generateLocalizations (type: JRubyExec) {
+	dependsOn jrubyPrepare
+	jrubyArgs '-S'
+	script "twine"
+	scriptArgs 'generate-localization-file', 'twine.txt', './src/main/res/values/generated_strings.xml'
+}
+
+preBuild {
+	dependsOn generateLocalizations
+}
+```
+
 ## User Interface
 
-* [Twine TextMate 2 Bundle](https://github.com/mobiata/twine.tmbundle) — This [TextMate 2](https://github.com/textmate/textmate) bundle will make it easier for you to work with Twine strings files. In particular, it lets you use code folding to easily collapse and expand both strings and sections.
-* [twine_ui](https://github.com/Daij-Djan/twine_ui) — A user interface for Twine written by [Dominik Pich](https://github.com/Daij-Djan/). Consider using this if you would prefer to use Twine without dropping to a command line.
+* [Twine TextMate 2 Bundle](https://github.com/mobiata/twine.tmbundle) — This [TextMate 2](https://github.com/textmate/textmate) bundle will make it easier for you to work with Twine files. In particular, it lets you use code folding to easily collapse and expand both definitions and sections.
 
-## Plugin Support
+## Extending Twine
 
-Twine supports a basic plugin infrastructure, allowing third-party code to provide support for additional formatters. Twine will read a yaml config file specifying which plugins to load from three locations.
-
-0. `./twine.yml`    The current working directory
-0. `~/.twine`       The home directory
-0. `/etc/twine.yml` The etc directory
-
-Plugins are specified as values for the `gems` key. The following is an example config:
-
-```
-gems: appium_twine
-```
-
-Multiple gems can also be specfied in the yaml file.
-
-```
-gems: [appium_twine, some_other_plugin]
-```
-
-[appium_twine](https://github.com/appium/appium_twine) is a sample plugin used to provide a C# formatter.
+If there's a format Twine does not yet support and you're keen to change that, check out the [documentation](documentation/formatters.md).
 
 ## Contributors
 
 Many thanks to all of the contributors to the Twine project, including:
 
 * [Blake Watters](https://github.com/blakewatters)
+* [bootstraponline](https://github.com/bootstraponline)
 * [Ishitoya Kentaro](https://github.com/kent013)
 * [Joseph Earl](https://github.com/JosephEarl)
 * [Kevin Everets](https://github.com/keverets)
 * [Kevin Wood](https://github.com/kwood)
 * [Mohammad Hejazi](https://github.com/MohammadHejazi)
 * [Robert Guo](http://www.robertguo.me/)
+* [Sebastian Ludwig](https://github.com/sebastianludwig)
 * [Sergey Pisarchik](https://github.com/SergeyPisarchik)
 * [Shai Shamir](https://github.com/pichirichi)
 
@@ -204,9 +253,12 @@ Many thanks to all of the contributors to the Twine project, including:
 [rubyzip]: http://rubygems.org/gems/rubyzip
 [git]: http://git-scm.org/
 [INI]: http://en.wikipedia.org/wiki/INI_file
-[applestrings]: http://developer.apple.com/documentation/Cocoa/Conceptual/LoadingResources/Strings/Strings.html
+[applestrings]: https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/LoadingResources/Strings/Strings.html
 [androidstrings]: http://developer.android.com/guide/topics/resources/string-resource.html
+[androidstyling]: http://developer.android.com/guide/topics/resources/string-resource.html#FormattingAndStyling
 [gettextpo]: http://www.gnu.org/savannah-checkouts/gnu/gettext/manual/html_node/PO-Files.html
 [jquerylocalize]: https://github.com/coderifous/jquery-localize
 [djangopo]: https://docs.djangoproject.com/en/dev/topics/i18n/translation/
 [tizen]: https://developer.tizen.org/documentation/articles/localization
+[flash]: http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/mx/resources/IResourceManager.html#getString()
+[printf]: https://en.wikipedia.org/wiki/Printf_format_string

Rakefile

@@ -2,7 +2,7 @@ require 'rake'
 require 'rake/testtask'
 
 Rake::TestTask.new do |t|
-  t.test_files = %w(test/twine_test.rb)
+  t.libs = ['lib', 'test']
 end
 
 task :default => :test

documentation/formatters.md

@@ -0,0 +1,89 @@
+# How Formatters Work
+
+It's a formatters job to transform a Twine file into a localization file of a specific format. That task is solved using a hierarchical approach. This document describes the process in detail using the following Twine file as an example.
+
+```
+[[General]]
+	[yes]
+		en = Yes
+	[no]
+		en = No
+
+[[Messages]]
+	[success]
+		en = All good
+		comment = Everything worked
+```
+
+A Twine file consists of multiple _sections_, each _section_ contains _keys_ with translated _values_, _comments_ and so on. If we highlight the components of the example file, it looks like this:
+
+![Highlighted Twine file components](assets/formatter_1.png)
+
+A formatter takes each of these components (and a few more) and formats them, working it's way from the outside inwards. First the method `format_file` is called, which doesn't do much more than calling `format_sections` which in turn calls `format_section` for each section and so on. To get an overview which method calls which, look at the next picture, where each method is represented by a block that is called by its surrounding block - read it from top to bottom and from the outside in.
+
+![formatting method calling structure](assets/formatter_2.png)
+
+If we map the _input_ for each method to the example file, it looks like this
+
+![highlighted formatter method input](assets/formatter_3.png)
+
+As stated at the beginning, the output produced by a formatter depends on the formatter. The output of the Apple formatter would for example be
+
+```
+/********** General **********/
+
+"yes" = "Yes";
+
+"no" = "No";
+
+
+/********** Messages **********/
+
+/* Everything worked */
+"success" = "All good";
+```
+
+Or, highlighted by the method that produces each piece of the output
+
+![highlighted formatter method output](assets/formatter_4.png)
+
+The process described above is implemented by the structure giving formatter base class [`Abstract`](/lib/twine/formatters/abstract.rb). To generate a desired output format, a formatter overwrites just enough methods to gain as much control as it needs - it's basically the [Template Method Pattern](https://en.wikipedia.org/wiki/Template_method_pattern) applied again and again.
+
+# Write a Formatter
+
+Formatters inherit from [`Abstract`](/lib/twine/formatters/abstract.rb) and need to specify some information about the format they are supporting like the format name, the default file name, if they can handle a directory of localization files and so on - just take a look at the class and the [other formatters](/lib/twine/formatters) to get an idea.
+
+The `Abstract` formatter also specifies two utility methods to be used when read a localization file, `set_translation_for_key` and `set_comment_for_key`, however the actual parsing is too formatter specific and must be implemented in the `read` method of a formatter.
+
+Which methods to overwrite to produce the desired output depends pretty much on the format. Again, looking at the [bundled formatters](/lib/twine/formatters) will provide some insight.
+
+Finally, to make a formatter available, it needs to be added to the list of formatters
+
+```
+Twine::Formatters.formatters << MyWickedFormatter.new
+```
+
+# Plugins
+
+Once a formatter has been developed, it can be distributed as a gem and loaded as a plugin. Twine will read a yaml config file specifying which plugins to load from three locations.
+
+0. `./twine.yml`    The current working directory
+0. `~/.twine`       The home directory
+0. `/etc/twine.yml` The etc directory
+
+Plugins are specified as values for the `gems` key. The following is an example config:
+
+```
+gems: wicked_twine
+```
+
+Multiple gems can also be specified in the yaml file.
+
+```
+gems: [wicked_twine, some_other_plugin]
+```
+
+## Sample Plugins
+
+* [appium-twine](https://github.com/appium/appium_twine)
+* [twine-flutter](https://github.com/tiknil/twine-flutter)

lib/twine.rb

@@ -1,12 +1,42 @@
 module Twine
+  @@stdout = STDOUT
+  @@stderr = STDERR
+
+  def self.stdout
+    @@stdout
+  end
+
+  def self.stdout=(out)
+    @@stdout = out
+  end
+
+  def self.stderr
+    @@stderr
+  end
+
+  def self.stderr=(err)
+    @@stderr = err
+  end
+
   class Error < StandardError
   end
 
-  require 'twine/plugin'
-  require 'twine/cli'
-  require 'twine/encoding'
-  require 'twine/formatters'
-  require 'twine/runner'
-  require 'twine/stringsfile'
   require 'twine/version'
+  require 'twine/plugin'
+  require 'twine/twine_file'
+  require 'twine/encoding'
+  require 'twine/output_processor'
+  require 'twine/placeholders'
+  require 'twine/formatters'
+  require 'twine/formatters/abstract'
+  require 'twine/formatters/android'
+  require 'twine/formatters/apple'
+  require 'twine/formatters/apple_plural'
+  require 'twine/formatters/django'
+  require 'twine/formatters/flash'
+  require 'twine/formatters/gettext'
+  require 'twine/formatters/jquery'
+  require 'twine/formatters/tizen'
+  require 'twine/runner'
+  require 'twine/cli'
 end

lib/twine/cli.rb

@@ -1,186 +1,433 @@
 require 'optparse'
+require 'io/console'
 
 module Twine
-  class CLI
-    def initialize(args, options)
-      @options = options
-      @args = args
+  module CLI
+    ALL_FORMATS = Formatters.formatters.map(&:format_name).map(&:downcase)
+    OPTIONS = {
+      consume_all: {
+        switch: ['-a', '--[no-]consume-all'],
+        description: 'Normally Twine will ignore any translation keys that do not exist in your Twine file.',
+        boolean: true
+      },
+      consume_comments: {
+        switch: ['-c', '--[no-]consume-comments'],
+        description: <<-DESC,
+          Normally Twine will ignore all comments in the file. With this flag set, any 
+          comments encountered will be read and parsed into the Twine data file. This is especially useful 
+          when creating your first Twine data file from an existing project.
+        DESC
+        boolean: true
+      },
+      create_folders: {
+        switch: ['-r', '--[no-]create-folders'],
+        description: <<-DESC,
+          This flag may be used to create output folders for all languages, if they don't exist yet.
+          As a result all languages will be exported, not only the ones where an output folder already exists.
+        DESC
+        boolean: true
+      },
+      developer_language: {
+        switch: ['-d', '--developer-language LANG'],
+        description: <<-DESC,
+          When writing the Twine data file, set the specified language as the "developer language". In 
+          practice, this just means that this language will appear first in the Twine data file. When 
+          generating files this language will be used as default language and its translations will be 
+          used if a definition is not localized for the output language.
+        DESC
+      },
+      encoding: {
+        switch: ['-e', '--encoding ENCODING'],
+        description: <<-DESC,
+          Twine defaults to encoding all output files in UTF-8. This flag will tell Twine to use an alternate
+          encoding for these files. For example, you could use this to write Apple .strings files in UTF-16. 
+          When reading files, Twine does its best to determine the encoding automatically. However, if the 
+          files are UTF-16 without BOM, you need to specify if it's UTF-16LE or UTF16-BE.
+        DESC
+      },
+      escape_all_tags: {
+        switch: ['--[no-]escape-all-tags'],
+        description: <<-DESC,
+          Always escape all HTML tags. By default the Android formatter will ONLY escape styling tags, if a 
+          string also contains placeholders. This flag enforces that styling tags are escaped regardless of
+          placeholders.
+        DESC
+      },
+      file_name: {
+        switch: ['-n', '--file-name FILE_NAME'],
+        description: 'This flag may be used to overwrite the default file name of the format.'
+      },
+      format: {
+        switch: ['-f', '--format FORMAT', ALL_FORMATS],
+        description: <<-DESC,
+          The file format to read or write: (#{ALL_FORMATS.join(', ')}). Additional formatters can be placed in the formats/ directory.
+        DESC
+      },
+      :include => {
+        switch: ['-i', '--include SET', [:all, :translated, :untranslated]],
+        description: <<-DESC,
+          This flag will determine which definitions are included. It's possible values are:
+            all: All definitions both translated and untranslated for the specified language are included. 
+              This is the default value.
+            translated: Only definitions with translation for the specified language are included.
+            untranslated: Only definitions without translation for the specified language are included.
+        DESC
+        default: :all
+      },
+      languages: {
+        switch: ['-l', '--lang LANGUAGES', Array],
+        description: 'Comma separated list of language codes to use for the specified action.'
+      },
+      output_path: {
+        switch: ['-o', '--output-file OUTPUT_FILE'],
+        description: 'Write a new Twine file at this location instead of replacing the original file.'
+      },
+      pedantic: {
+        switch: ['-p', '--[no-]pedantic'],
+        description: 'When validating a Twine file, perform additional checks that go beyond pure validity (like presence of tags).'
+      },
+      quiet: {
+        switch: ['-q', '--[no-]quiet'],
+        description: 'Suppress all console output except error messages.'
+      },
+      tags: {
+        switch: ['-t', '--tags TAG1,TAG2,TAG3', Array],
+        description: <<-DESC,
+          Only definitions with ANY of the specified tags will be processed. Specify this option multiple
+          times to only include definitions with ALL of the specified tags. Prefix a tag with ~ to include
+          definitions NOT containing that tag. Omit this option to match all definitions in the Twine data file.
+        DESC
+        repeated: true
+      },
+      untagged: {
+        switch: ['-u', '--[no-]untagged'],
+        description: <<-DESC,
+          If you have specified tags using the --tags flag, then only those tags will be selected. If you also 
+          want to select all definitions that are untagged, then you can specify this option to do so.
+        DESC
+      },
+      validate: {
+        switch: ['--[no-]validate'],
+        description: 'Validate the Twine file before formatting it.'
+      }
+    }
+
+    COMMANDS = {
+      'generate-localization-file' => {
+        description: 'Generates a localization file in a certain LANGUAGE given a particular FORMAT. This script will attempt to guess both the language and the format given the filename and extension. For example, "ko.xml" will generate a Korean language file for Android.',
+        arguments: [:twine_file, :output_path],
+        optional_options: [
+          :developer_language,
+          :encoding,
+          :escape_all_tags,
+          :format,
+          :include,
+          :languages,
+          :quiet,
+          :tags,
+          :untagged,
+          :validate
+        ],
+        option_validation: Proc.new { |options|
+          if options[:languages] and options[:languages].length > 1
+            raise Twine::Error.new 'specify only a single language for the `generate-localization-file` command.'
+          end
+        },
+        example: 'twine generate-localization-file twine.txt ko.xml --tags FT'
+      },
+      'generate-all-localization-files' => {
+        description: 'Generates all the localization files necessary for a given project. The parent directory to all of the locale-specific directories in your project should be specified as the INPUT_OR_OUTPUT_PATH. This command will most often be executed by your build script so that each build always contains the most recent translations.',
+        arguments: [:twine_file, :output_path],
+        optional_options: [
+          :create_folders,
+          :developer_language,
+          :encoding,
+          :escape_all_tags,
+          :file_name,
+          :format,
+          :include,
+          :quiet,
+          :tags,
+          :untagged,
+          :validate
+        ],
+        example: 'twine generate-all-localization-files twine.txt Resources/Locales/ --tags FT,FB'
+      },
+      'generate-localization-archive' => {
+        description: 'Generates a zip archive of localization files in a given format. The purpose of this command is to create a very simple archive that can be handed off to a translation team. The translation team can unzip the archive, translate all of the strings in the archived files, zip everything back up, and then hand that final archive back to be consumed by the consume-localization-archive command.',
+        arguments: [:twine_file, :output_path],
+        required_options: [
+          :format
+        ],
+        optional_options: [
+          :developer_language,
+          :encoding,
+          :escape_all_tags,
+          :include,
+          :quiet,
+          :tags,
+          :untagged,
+          :validate
+        ],
+        example: 'twine generate-localization-archive twine.txt LocDrop5.zip --tags FT,FB --format android --lang de,en,en-GB,ja,ko'
+      },
+      'consume-localization-file' => {
+        description: 'Slurps all of the translations from a localization file into the specified TWINE_FILE. If you have some files returned to you by your translators you can use this command to incorporate all of their changes. This script will attempt to guess both the language and the format given the filename and extension. For example, "ja.strings" will assume that the file is a Japanese iOS strings file.',
+        arguments: [:twine_file, :input_path],
+        optional_options: [
+          :consume_all,
+          :consume_comments,
+          :developer_language,
+          :encoding,
+          :format,
+          :languages,
+          :output_path,
+          :quiet,
+          :tags
+        ],
+        option_validation: Proc.new { |options|
+          if options[:languages] and options[:languages].length > 1
+            raise Twine::Error.new 'specify only a single language for the `consume-localization-file` command.'
+          end
+        },
+        example: 'twine consume-localization-file twine.txt ja.strings'
+      },
+      'consume-all-localization-files' => {
+        description: 'Slurps all of the translations from a directory into the specified TWINE_FILE. If you have some files returned to you by your translators you can use this command to incorporate all of their changes. This script will attempt to guess both the language and the format given the filename and extension. For example, "ja.strings" will assume that the file is a Japanese iOS strings file.',
+        arguments: [:twine_file, :input_path],
+        optional_options: [
+          :consume_all,
+          :consume_comments,
+          :developer_language,
+          :encoding,
+          :format,
+          :output_path,
+          :quiet,
+          :tags
+        ],
+        example: 'twine consume-all-localization-files twine.txt Resources/Locales/ --developer-language en --tags DefaultTag1,DefaultTag2'
+      },
+      'consume-localization-archive' => {
+        description: 'Consumes an archive of translated files. This archive should be in the same format as the one created by the generate-localization-archive command.',
+        arguments: [:twine_file, :input_path],
+        optional_options: [
+          :consume_all,
+          :consume_comments,
+          :developer_language,
+          :encoding,
+          :format,
+          :output_path,
+          :quiet,
+          :tags
+        ],
+        example: 'twine consume-localization-archive twine.txt LocDrop5.zip'
+      },
+      'validate-twine-file' => {
+        description: 'Validates that the given Twine file is parseable, contains no duplicates, and that no key contains invalid characters. Exits with a non-zero exit code if those criteria are not met.',
+        arguments: [:twine_file],
+        optional_options: [
+          :developer_language,
+          :pedantic,
+          :quiet
+        ],
+        example: 'twine validate-twine-file twine.txt'
+      }
+    }
+    DEPRECATED_COMMAND_MAPPINGS = {
+      'generate-loc-drop' => 'generate-localization-archive',   # added on 17.01.2017 - version 0.10
+      'consume-loc-drop' => 'consume-localization-archive'      # added on 17.01.2017 - version 0.10
+    }
+
+    def self.parse(args)
+      command = args.select { |a| a[0] != '-' }[0]
+      args = args.reject { |a| a == command }
+
+      if args.any? { |a| a == '--version' }
+        Twine::stdout.puts "Twine version #{Twine::VERSION}"
+        return false
+      end
+
+      mapped_command = DEPRECATED_COMMAND_MAPPINGS[command]
+      if mapped_command
+        Twine::stdout.puts "WARNING: Twine commands names have changed. `#{command}` is now `#{mapped_command}`. The old command is deprecated and will soon stop working. For more information please check the documentation at https://github.com/mobiata/twine"
+        command = mapped_command
+      end
+
+      if command.nil?
+        print_help(args)
+        return false
+      elsif not COMMANDS.keys.include? command
+        raise Twine::Error.new "Invalid command: #{command}"
+      end
+
+      parse_command_options(command, args)
     end
 
-    def self.parse_args(args, options)
-      new(args, options).parse_args
+    private
+
+    def self.print_help(args)
+      verbose = false
+
+      help_parser = OptionParser.new
+      help_parser.banner = 'Usage: twine [command] [options]'
+
+      help_parser.define('-h', '--help', 'Show this message.')
+      help_parser.define('--verbose', 'More detailed help.') { verbose = true }
+
+      help_parser.parse!(args)
+
+      Twine::stdout.puts help_parser.help
+      Twine::stdout.puts ''
+      
+
+      Twine::stdout.puts 'Commands:'
+
+      COMMANDS.each do |name, properties|
+        if verbose
+          Twine::stdout.puts ''
+          Twine::stdout.puts ''
+          Twine::stdout.puts "# #{name}"
+          Twine::stdout.puts ''
+          Twine::stdout.puts properties[:description]
+        else 
+          Twine::stdout.puts "- #{name}"
+        end
+      end
+
+      Twine::stdout.puts ''
+      Twine::stdout.puts 'type `twine [command] --help` for further information about a command.'
     end
 
-    def parse_args
-      parser = OptionParser.new do |opts|
-        opts.banner = 'Usage: twine COMMAND STRINGS_FILE [INPUT_OR_OUTPUT_PATH] [--lang LANG1,LANG2...] [--tags TAG1,TAG2,TAG3...] [--format FORMAT]'
-        opts.separator ''
-        opts.separator 'The purpose of this script is to convert back and forth between multiple data formats, allowing us to treat our strings (and translations) as data stored in a text file. We can then use the data file to create drops for the localization team, consume similar drops returned by the localization team, generate reports on the strings, as well as create formatted string files to ship with your products. Twine currently supports iOS, OS X, Android, gettext, and jquery-localize string files.'
-        opts.separator ''
-        opts.separator 'Commands:'
-        opts.separator ''
-        opts.separator 'generate-string-file -- Generates a string file in a certain LANGUAGE given a particular FORMAT. This script will attempt to guess both the language and the format given the filename and extension. For example, "ko.xml" will generate a Korean language file for Android.'
-        opts.separator ''
-        opts.separator 'generate-all-string-files -- Generates all the string files necessary for a given project. The parent directory to all of the locale-specific directories in your project should be specified as the INPUT_OR_OUTPUT_PATH. This command will most often be executed by your build script so that each build always contains the most recent strings.'
-        opts.separator ''
-        opts.separator 'consume-string-file -- Slurps all of the strings from a translated strings file into the specified STRINGS_FILE. If you have some files returned to you by your translators you can use this command to incorporate all of their changes. This script will attempt to guess both the language and the format given the filename and extension. For example, "ja.strings" will assume that the file is a Japanese iOS strings file.'
-        opts.separator ''
-        opts.separator 'consume-all-string-files -- Slurps all of the strings from a directory into the specified STRINGS_FILE. If you have some files returned to you by your translators you can use this command to incorporate all of their changes. This script will attempt to guess both the language and the format given the filename and extension. For example, "ja.strings" will assume that the file is a Japanese iOS strings file.'
-        opts.separator ''
-        opts.separator 'generate-loc-drop -- Generates a zip archive of strings files in any format. The purpose of this command is to create a very simple archive that can be handed off to a translation team. The translation team can unzip the archive, translate all of the strings in the archived files, zip everything back up, and then hand that final archive back to be consumed by the consume-loc-drop command. This command assumes that --include-untranslated has been specified on the command line.'
-        opts.separator ''
-        opts.separator 'consume-loc-drop -- Consumes an archive of translated files. This archive should be in the same format as the one created by the generate-loc-drop command.'
-        opts.separator ''
-        opts.separator 'generate-report -- Generates a report containing data about your strings. For example, it will tell you if you have any duplicate strings or if any of your strings are missing tags. In addition, it will tell you how many strings you have and how many of those strings have been translated into each language.'
-        opts.separator ''
-        opts.separator 'General Options:'
-        opts.separator ''
-        opts.on('-l', '--lang LANGUAGES', Array, 'The language code(s) to use for the specified action.') do |langs|
-          @options[:languages] = langs
+    # source: https://www.safaribooksonline.com/library/view/ruby-cookbook/0596523696/ch01s15.html
+    def self.word_wrap(s, width)
+      s.gsub(/(.{1,#{width}})(\s+|\Z)/, "\\1\n").rstrip
+    end
+
+    def self.indent(string, first_line, following_lines)
+      lines = string.split("\n")
+      indentation = ' ' * following_lines
+      lines.map! { |line| indentation + line }
+      result = lines.join("\n").strip
+      ' ' * first_line + result
+    end
+
+    # ensure the description forms a neat block on the right
+    def self.prepare_description!(options, summary_width)
+      lines = options[:description].split "\n"
+
+      # remove leadinge HEREDOC spaces
+      space_match = lines[0].match(/^\s+/)
+      if space_match
+        leading_spaces = space_match[0].length
+        lines.map! { |l| l[leading_spaces..-1] }  
+      end
+
+      merged_lines = []
+      lines.each do |line|
+        # if the line is a continuation of the previous one
+        if not merged_lines.empty? and (line[0] != ' ' or line[0, 4] == '    ')
+          merged_lines[-1] += ' ' + line.strip
+        else
+          merged_lines << line.rstrip
         end
-        opts.on('-t', '--tags TAGS', Array, 'The tag(s) to use for the specified action. Only strings with that tag will be processed. Do not specify any tags to match all strings in the strings data file.') do |tags|
-          @options[:tags] = tags
+      end
+
+      if IO.console
+        console_width = IO.console.winsize[1]
+      else
+        console_width = 100
+      end
+      summary_width += 7  # account for description padding
+      max_description_width = console_width - summary_width
+      merged_lines.map! do |line|
+        if line[0] == ' '
+          line = word_wrap(line.strip, max_description_width - 2)
+          line = indent(line, 2, 4)
+        else
+          line = word_wrap(line, max_description_width)
         end
-        opts.on('-u', '--untagged', 'If you have specified tags using the --tags flag, then only those tags will be selected. If you also want to select all strings that are untagged, then you can specify this option to do so.') do |u|
-          @options[:untagged] = true
-        end
-        formats = []
-        Formatters.formatters.each do |formatter|
-          formats << formatter::FORMAT_NAME
-        end
-        opts.on('-f', '--format FORMAT', "The file format to read or write (#{formats.join(', ')}). Additional formatters can be placed in the formats/ directory.") do |format|
-          lformat = format.downcase
-          if !formats.include?(lformat)
-            STDERR.puts "Invalid format: #{format}"
+        line
+      end
+
+      options[:switch] << indent(merged_lines.join("\n"), 0, summary_width)
+    end
+
+    def self.parse_command_options(command_name, args)
+      command = COMMANDS[command_name]
+
+      result = {
+        command: command_name
+      }
+
+      parser = OptionParser.new
+      parser.banner = "Usage: twine #{command_name} #{command[:arguments].map { |c| "[#{c}]" }.join(' ')} [options]"
+
+      [:required_options, :optional_options].each do |option_type|
+        options = command[option_type]
+        if options and options.size > 0
+          parser.separator ''
+          parser.separator option_type.to_s.gsub('_', ' ').capitalize + ":"
+          
+          options.each do |option_name|
+            option = OPTIONS[option_name]
+
+            result[option_name] = option[:default] if option[:default]
+
+            prepare_description!(option, parser.summary_width)
+
+            parser.define(*option[:switch]) do |value|
+              if option[:repeated]
+                result[option_name] = (result[option_name] || []) << value
+              else
+                result[option_name] = value
+              end
+            end
           end
-          @options[:format] = lformat
         end
-        opts.on('-a', '--consume-all', 'Normally, when consuming a string file, Twine will ignore any string keys that do not exist in your master file.') do |a|
-          @options[:consume_all] = true
-        end
-        opts.on('-s', '--include-untranslated', 'This flag will cause any Android string files that are generated to include strings that have not yet been translated for the current language.') do |s|
-          @options[:include_untranslated] = true
-        end
-        opts.on('-o', '--output-file OUTPUT_FILE', 'Write the new strings database to this file instead of replacing the original file. This flag is only useful when running the consume-string-file or consume-loc-drop commands.') do |o|
-          @options[:output_path] = o
-        end
-        opts.on('-n', '--file-name FILE_NAME', 'When running the generate-all-string-files command, this flag may be used to overwrite the default file name of the format.') do |n|
-          @options[:file_name] = n
-        end
-        opts.on('-d', '--developer-language LANG', 'When writing the strings data file, set the specified language as the "developer language". In practice, this just means that this language will appear first in the strings data file.') do |d|
-          @options[:developer_language] = d
-        end
-        opts.on('-c', '--consume-comments', 'Normally, when consuming a string file, Twine will ignore all comments in the file. With this flag set, any comments encountered will be read and parsed into the strings data file. This is especially useful when creating your first strings data file from an existing project.') do |c|
-          @options[:consume_comments] = true
-        end
-        opts.on('-e', '--encoding ENCODING', 'Twine defaults to encoding all output files in UTF-8. This flag will tell Twine to use an alternate encoding for these files. For example, you could use this to write Apple .strings files in UTF-16. This flag currently only works with Apple .strings files and is currently only supported in Ruby 1.9.3 or greater.') do |e|
-          if !"".respond_to?(:encode)
-            raise Twine::Error.new "The --encoding flag is only supported on Ruby 1.9.3 or greater."
+      end
+
+      parser.define('-h', '--help', 'Show this message.') do
+        Twine::stdout.puts parser.help
+        return false
+      end
+
+      parser.separator ''
+      parser.separator 'Examples:'
+      parser.separator ''
+      parser.separator "> #{command[:example]}"
+
+      begin
+        parser.parse! args
+      rescue OptionParser::ParseError => e
+        raise Twine::Error.new e.message
+      end
+
+      arguments = args.reject { |a| a[0] == '-' }
+      number_of_missing_arguments = command[:arguments].size - arguments.size
+      if number_of_missing_arguments > 0
+        missing_arguments = command[:arguments][-number_of_missing_arguments, number_of_missing_arguments]
+        raise Twine::Error.new "#{number_of_missing_arguments} missing argument#{number_of_missing_arguments > 1 ? "s" : ""}: #{missing_arguments.join(', ')}. Check `twine #{command_name} -h`"
+      end
+
+      if args.length > command[:arguments].size
+        raise Twine::Error.new "Unknown argument: #{args[command[:arguments].size]}"
+      end
+
+      if command[:required_options]
+        command[:required_options].each do |option_name|
+          if result[option_name] == nil
+            raise Twine::Error.new "missing option: #{OPTIONS[option_name][:switch][0]}"
           end
-          @options[:output_encoding] = e
-        end
-        opts.on('-h', '--help', 'Show this message.') do |h|
-          puts opts.help
-          exit
-        end
-        opts.on('--version', 'Print the version number and exit.') do |x|
-          puts "Twine version #{Twine::VERSION}"
-          exit
-        end
-        opts.separator ''
-        opts.separator 'Examples:'
-        opts.separator ''
-        opts.separator '> twine generate-string-file strings.txt ko.xml --tags FT'
-        opts.separator '> twine generate-all-string-files strings.txt Resources/Locales/ --tags FT,FB'
-        opts.separator '> twine consume-string-file strings.txt ja.strings'
-        opts.separator '> twine consume-all-string-files strings.txt Resources/Locales/ --developer-language en --tags DefaultTag1,DefaultTag2'
-        opts.separator '> twine generate-loc-drop strings.txt LocDrop5.zip --tags FT,FB --format android --lang de,en,en-GB,ja,ko'
-        opts.separator '> twine consume-loc-drop strings.txt LocDrop5.zip'
-        opts.separator '> twine generate-report strings.txt'
-      end
-      parser.parse! @args
-
-      if @args.length == 0
-        puts parser.help
-        exit
-      end
-
-      @options[:command] = @args[0]
-
-      if !VALID_COMMANDS.include? @options[:command]
-        raise Twine::Error.new "Invalid command: #{@options[:command]}"
-      end
-
-      if @args.length == 1
-        raise Twine::Error.new 'You must specify your strings file.'
-      end
-
-      @options[:strings_file] = @args[1]
-
-      case @options[:command]
-      when 'generate-string-file'
-        if @args.length == 3
-          @options[:output_path] = @args[2]
-        elsif @args.length > 3
-          raise Twine::Error.new "Unknown argument: #{@args[3]}"
-        else
-          raise Twine::Error.new 'Not enough arguments.'
-        end
-        if @options[:languages] and @options[:languages].length > 1
-          raise Twine::Error.new 'Please only specify a single language for the generate-string-file command.'
-        end
-      when 'generate-all-string-files'
-        if ARGV.length == 3
-          @options[:output_path] = @args[2]
-        elsif @args.length > 3
-          raise Twine::Error.new "Unknown argument: #{@args[3]}"
-        else
-          raise Twine::Error.new 'Not enough arguments.'
-        end
-      when 'consume-string-file'
-        if @args.length == 3
-          @options[:input_path] = @args[2]
-        elsif @args.length > 3
-          raise Twine::Error.new "Unknown argument: #{@args[3]}"
-        else
-          raise Twine::Error.new 'Not enough arguments.'
-        end
-        if @options[:languages] and @options[:languages].length > 1
-          raise Twine::Error.new 'Please only specify a single language for the consume-string-file command.'
-        end
-      when 'consume-all-string-files'
-        if @args.length == 3
-          @options[:input_path] = @args[2]
-        elsif @args.length > 3
-          raise Twine::Error.new "Unknown argument: #{@args[3]}"
-        else
-          raise Twine::Error.new 'Not enough arguments.'
-        end
-      when 'generate-loc-drop'
-        @options[:include_untranslated] = true
-        if @args.length == 3
-          @options[:output_path] = @args[2]
-        elsif @args.length > 3
-          raise Twine::Error.new "Unknown argument: #{@args[3]}"
-        else
-          raise Twine::Error.new 'Not enough arguments.'
-        end
-        if !@options[:format]
-          raise Twine::Error.new 'You must specify a format.'
-        end
-      when 'consume-loc-drop'
-        if @args.length == 3
-          @options[:input_path] = @args[2]
-        elsif @args.length > 3
-          raise Twine::Error.new "Unknown argument: #{@args[3]}"
-        else
-          raise Twine::Error.new 'Not enough arguments.'
-        end
-      when 'generate-report'
-        if @args.length > 2
-          raise Twine::Error.new "Unknown argument: #{@args[2]}"
         end
       end
+
+      command[:option_validation].call(result) if command[:option_validation]
+
+      command[:arguments].each do |argument_name|
+        result[argument_name] = args.shift
+      end
+
+      result
     end
   end
 end

lib/twine/encoding.rb

@@ -1,20 +1,22 @@
 module Twine
   module Encoding
-    def self.encoding_for_path path
-      File.open(path, 'rb') do |f|
-        begin
-          a = f.readbyte
-          b = f.readbyte
-          if (a == 0xfe && b == 0xff)
-            return 'UTF-16BE'
-          elsif (a == 0xff && b == 0xfe)
-            return 'UTF-16LE'
-          end
-        rescue EOFError
-        end
-      end
 
-      'UTF-8'
+    def self.bom(path)
+      first_bytes = IO.binread(path, 2)
+      return nil unless first_bytes
+      first_bytes = first_bytes.codepoints.map.to_a
+      return 'UTF-16BE' if first_bytes == [0xFE, 0xFF]
+      return 'UTF-16LE' if first_bytes == [0xFF, 0xFE]
+    rescue EOFError
+      return nil
+    end
+
+    def self.has_bom?(path)
+      !bom(path).nil?
+    end
+
+    def self.encoding_for_path(path)
+      bom(path) || 'UTF-8'      
     end
   end
 end

lib/twine/formatters.rb

@@ -1,15 +1,6 @@
-require 'twine/formatters/abstract'
-require 'twine/formatters/android'
-require 'twine/formatters/apple'
-require 'twine/formatters/flash'
-require 'twine/formatters/gettext'
-require 'twine/formatters/jquery'
-require 'twine/formatters/django'
-require 'twine/formatters/tizen'
-
 module Twine
   module Formatters
-    @formatters = [Formatters::Apple, Formatters::Android, Formatters::Gettext, Formatters::JQuery, Formatters::Flash, Formatters::Django, Formatters::Tizen]
+    @formatters = []
 
     class << self
       attr_reader :formatters
@@ -22,9 +13,8 @@ module Twine
       # returns array of active formatters
       #
       def register_formatter formatter_class
-        raise "#{formatter_class} already registered" if @formatters.include? formatter_class
-        @formatters << formatter_class
+        @formatters << formatter_class.new
       end
     end
   end
-end
+end

lib/twine/formatters/abstract.rb

@@ -1,146 +1,188 @@
+require 'fileutils'
+
 module Twine
   module Formatters
     class Abstract
-      attr_accessor :strings
+      SUPPORTS_PLURAL = false
+      LANGUAGE_CODE_WITH_OPTIONAL_REGION_CODE = "[a-z]{2}(?:-[A-Za-z]{2})?"
+
+      attr_accessor :twine_file
       attr_accessor :options
 
-      def self.can_handle_directory?(path)
-        return false
+      def initialize
+        @twine_file = TwineFile.new
+        @options = {}
       end
 
-      def initialize(strings, options)
-        @strings = strings
-        @options = options
+      def format_name
+        raise NotImplementedError.new("You must implement format_name in your formatter class.")
       end
 
-      def iosify_substitutions(str)
-        # use "@" instead of "s" for substituting strings
-        str.gsub!(/%([0-9\$]*)s/, '%\1@')
-        return str
+      def extension
+        raise NotImplementedError.new("You must implement extension in your formatter class.")
       end
 
-      def androidify_substitutions(str)
-        # 1) use "s" instead of "@" for substituting strings
-        str.gsub!(/%([0-9\$]*)@/, '%\1s')
-
-        # 1a) escape strings that begin with a lone "@"
-        str.sub!(/^@ /, '\\@ ')
-
-        # 2) if there is more than one substitution in a string, make sure they are numbered
-        substituteCount = 0
-        startFound = false
-        str.each_char do |c|
-          if startFound
-            if c == "%"
-              # ignore as this is a literal %
-            elsif c.match(/\d/)
-              # leave the string alone if it already has numbered substitutions
-              return str
-            else
-              substituteCount += 1
-            end
-            startFound = false
-          elsif c == "%"
-            startFound = true
-          end
-        end
-
-        if substituteCount > 1
-          currentSub = 1
-          startFound = false
-          newstr = ""
-          str.each_char do |c|
-            if startFound
-              if !(c == "%")
-                newstr = newstr + "#{currentSub}$"
-                currentSub += 1
-              end
-              startFound = false
-            elsif c == "%"
-              startFound = true
-            end
-            newstr = newstr + c
-          end
-          return newstr
-        else
-          return str
-        end
-      end
-      
-      def set_translation_for_key(key, lang, value)
-        if @strings.strings_map.include?(key)
-          @strings.strings_map[key].translations[lang] = value
-        elsif @options[:consume_all]
-          STDERR.puts "Adding new string '#{key}' to strings data file."
-          arr = @strings.sections.select { |s| s.name == 'Uncategorized' }
-          current_section = arr ? arr[0] : nil
-          if !current_section
-            current_section = StringsSection.new('Uncategorized')
-            @strings.sections.insert(0, current_section)
-          end
-          current_row = StringsRow.new(key)
-          current_section.rows << current_row
-          
-          if @options[:tags] && @options[:tags].length > 0
-              current_row.tags = @options[:tags]            
-          end
-          
-          @strings.strings_map[key] = current_row
-          @strings.strings_map[key].translations[lang] = value
-        else
-          STDERR.puts "Warning: '#{key}' not found in strings data file."
-        end
-        if !@strings.language_codes.include?(lang)
-          @strings.add_language_code(lang)
-        end
-      end
-
-      def set_comment_for_key(key, comment)
-        if @strings.strings_map.include?(key)
-          @strings.strings_map[key].comment = comment
-        end
+      def can_handle_directory?(path)
+        Dir.entries(path).any? { |item| /^.+#{Regexp.escape(extension)}$/.match(item) }
       end
 
       def default_file_name
         raise NotImplementedError.new("You must implement default_file_name in your formatter class.")
       end
 
+      def set_translation_for_key(key, lang, value)
+        value = value.gsub("\n", "\\n")
+
+        if @twine_file.definitions_by_key.include?(key)
+          definition = @twine_file.definitions_by_key[key]
+          reference = @twine_file.definitions_by_key[definition.reference_key] if definition.reference_key
+
+          if !reference or value != reference.translations[lang]
+            definition.translations[lang] = value
+          end
+        elsif @options[:consume_all]
+          Twine::stdout.puts "Adding new definition '#{key}' to twine file."
+          current_section = @twine_file.sections.find { |s| s.name == 'Uncategorized' }
+          unless current_section
+            current_section = TwineSection.new('Uncategorized')
+            @twine_file.sections.insert(0, current_section)
+          end
+          current_definition = TwineDefinition.new(key)
+          current_section.definitions << current_definition
+          
+          if @options[:tags] && @options[:tags].length > 0
+            current_definition.tags = @options[:tags]            
+          end
+          
+          @twine_file.definitions_by_key[key] = current_definition
+          @twine_file.definitions_by_key[key].translations[lang] = value
+        else
+          Twine::stdout.puts "WARNING: '#{key}' not found in twine file."
+        end
+        if !@twine_file.language_codes.include?(lang)
+          @twine_file.add_language_code(lang)
+        end
+      end
+
+      def set_comment_for_key(key, comment)
+        return unless @options[:consume_comments]
+        
+        if @twine_file.definitions_by_key.include?(key)
+          definition = @twine_file.definitions_by_key[key]
+          
+          reference = @twine_file.definitions_by_key[definition.reference_key] if definition.reference_key
+
+          if !reference or comment != reference.raw_comment
+            definition.comment = comment
+          end
+        end
+      end
+
       def determine_language_given_path(path)
-        raise NotImplementedError.new("You must implement determine_language_given_path in your formatter class.")
+        only_language_and_region = /^#{LANGUAGE_CODE_WITH_OPTIONAL_REGION_CODE}$/i
+        basename = File.basename(path, File.extname(path))
+        return basename if basename =~ only_language_and_region
+        return basename if @twine_file.language_codes.include? basename
+        
+        path.split(File::SEPARATOR).reverse.find { |segment| segment =~ only_language_and_region }
       end
 
-      def read_file(path, lang)
-        raise NotImplementedError.new("You must implement read_file in your formatter class.")
+      def output_path_for_language(lang)
+        lang
       end
 
-      def write_file(path, lang)
-        raise NotImplementedError.new("You must implement write_file in your formatter class.")
+      def read(io, lang)
+        raise NotImplementedError.new("You must implement read in your formatter class.")
       end
 
-      def write_all_files(path)
-        if !File.directory?(path)
-          raise Twine::Error.new("Directory does not exist: #{path}")
+      def format_file(lang)
+        output_processor = Processors::OutputProcessor.new(@twine_file, @options)
+        processed_twine_file = output_processor.process(lang)
+
+        return nil if processed_twine_file.definitions_by_key.empty?
+
+        header = format_header(lang)
+        result = ""
+        result += header + "\n" if header
+        result += format_sections(processed_twine_file, lang)
+      end
+
+      def format_header(lang)
+      end
+
+      def format_sections(twine_file, lang)
+        sections = twine_file.sections.map { |section| format_section(section, lang) }
+        sections.compact.join("\n")
+      end
+
+      def format_section_header(section)
+      end
+
+      def should_include_definition(definition, lang)
+        return !definition.translation_for_lang(lang).nil?
+      end
+
+      def format_section(section, lang)
+        definitions = section.definitions.select { |definition| should_include_definition(definition, lang) }
+        return if definitions.empty?
+
+        result = ""
+
+        if section.name && section.name.length > 0
+          section_header = format_section_header(section)
+          result += "\n#{section_header}" if section_header
         end
 
-        file_name = @options[:file_name] || default_file_name
-        langs_written = []
-        Dir.foreach(path) do |item|
-          if item == "." or item == ".."
-            next
-          end
-          item = File.join(path, item)
-          if File.directory?(item)
-            lang = determine_language_given_path(item)
-            if lang
-              write_file(File.join(item, file_name), lang)
-              langs_written << lang
-            end
-          end
+        definitions.map! { |definition| format_definition(definition, lang) }
+        definitions.compact! # remove nil definitions
+        definitions.map! { |definition| "\n#{definition}" }  # prepend newline
+        result += definitions.join
+      end
+
+      def format_definition(definition, lang)
+        formatted_definition = [format_comment(definition, lang)]
+        if self.class::SUPPORTS_PLURAL && definition.is_plural?
+          formatted_definition << format_plural(definition, lang)
+        else
+          formatted_definition << format_key_value(definition, lang)
         end
-        if langs_written.empty?
-          raise Twine::Error.new("Failed to generate any files: No languages found at #{path}")
+        formatted_definition.compact.join
+      end
+
+      def format_comment(definition, lang)
+      end
+
+      def format_key_value(definition, lang)
+        value = definition.translation_for_lang(lang)
+        key_value_pattern % { key: format_key(definition.key.dup), value: format_value(value.dup) }
+      end
+
+      def format_plural(definition, lang)
+        plural_hash = definition.plural_translation_for_lang(lang)
+        if plural_hash
+          format_plural_keys(definition.key.dup, plural_hash)
         end
       end
+
+      def key_value_pattern
+        raise NotImplementedError.new("You must implement key_value_pattern in your formatter class.")
+      end
+
+      def format_plural_keys(key, plural_hash)
+        raise NotImplementedError.new("You must implement format_plural_keys in your formatter class.")
+      end
+
+      def format_key(key)
+        key
+      end
+
+      def format_value(value)
+        value
+      end
+
+      def escape_quotes(text)
+        text.gsub('"', '\\\\"')
+      end
     end
   end
 end

lib/twine/formatters/android.rb

@@ -5,151 +5,199 @@ require 'rexml/document'
 module Twine
   module Formatters
     class Android < Abstract
-      FORMAT_NAME = 'android'
-      EXTENSION = '.xml'
-      DEFAULT_FILE_NAME = 'strings.xml'
+      include Twine::Placeholders
+
+      SUPPORTS_PLURAL = true
       LANG_CODES = Hash[
         'zh' => 'zh-Hans',
-        'zh-rCN' => 'zh-Hans',
-        'zh-rHK' => 'zh-Hant',
-        'en-rGB' => 'en-UK',
+        'zh-CN' => 'zh-Hans',
+        'zh-HK' => 'zh-Hant',
+        # See https://developer.android.com/reference/java/util/Locale#legacy-language-codes
+        'iw' => 'he',
         'in' => 'id',
-        'nb' => 'no'
-        # TODO: spanish
-      ]
-      DEFAULT_LANG_CODES = Hash[
-        'zh-TW' => 'zh-Hant' # if we don't have a zh-TW translation, try zh-Hant before en
+        'ji' => 'yi'
       ]
 
-      def self.can_handle_directory?(path)
+      def format_name
+        'android'
+      end
+
+      def extension
+        '.xml'
+      end
+
+      def can_handle_directory?(path)
         Dir.entries(path).any? { |item| /^values.*$/.match(item) }
       end
 
       def default_file_name
-        return DEFAULT_FILE_NAME
+        'strings.xml'
       end
 
       def determine_language_given_path(path)
         path_arr = path.split(File::SEPARATOR)
         path_arr.each do |segment|
           if segment == 'values'
-            return @strings.language_codes[0]
+            return @twine_file.language_codes[0]
           else
-            match = /^values-(.*)$/.match(segment)
+            # The language is defined by a two-letter ISO 639-1 language code, optionally followed by a two letter ISO 3166-1-alpha-2 region code (preceded by lowercase "r").
+            # see http://developer.android.com/guide/topics/resources/providing-resources.html#AlternativeResources
+            match = /^values-([a-z]{2}(-r[a-z]{2})?)$/i.match(segment)
+
             if match
-              lang = match[1]
-              lang = LANG_CODES.fetch(lang, lang)
-              lang.sub!('-r', '-')
-              return lang
+              lang = match[1].sub('-r', '-')
+              return LANG_CODES.fetch(lang, lang)
             end
           end
         end
 
-        return
+        return super
       end
 
-      def read_file(path, lang)
-        resources_regex = /<resources(?:[^>]*)>(.*)<\/resources>/m
-        key_regex = /<string name="(\w+)">/
-        comment_regex = /<!-- (.*) -->/
-        value_regex = /<string name="\w+">(.*)<\/string>/
-        key = nil
-        value = nil
+      def output_path_for_language(lang)
+        if lang == @twine_file.language_codes[0]
+          "values"
+        else
+          "values-#{lang}".gsub(/-(\p{Lu})/, '-r\1')
+        end
+      end
+
+      def set_translation_for_key(key, lang, value)
+        value = CGI.unescapeHTML(value)
+        value.gsub!('\\\'', '\'')
+        value.gsub!('\\"', '"')
+        value = convert_placeholders_from_android_to_twine(value)
+        value.gsub!('\@', '@')
+        value.gsub!(/(\\u0020)*|(\\u0020)*\z/) { |spaces| ' ' * (spaces.length / 6) }
+        super(key, lang, value)
+      end
+
+      def read(io, lang)
+        document = REXML::Document.new io, :compress_whitespace => %w{ string }
+        document.context[:attribute_quote] = :quote
         comment = nil
+        document.root.children.each do |child|
+          if child.is_a? REXML::Comment
+            content = child.string.strip
+            content.gsub!(/[\s]+/, ' ')
+            comment = content if content.length > 0 and not content.start_with?("SECTION:")
+          elsif child.is_a? REXML::Element
+            next unless child.name == 'string'
 
-        File.open(path, 'r:UTF-8') do |f|
-          content_match = resources_regex.match(f.read)
-          if content_match
-            for line in content_match[1].split(/\r?\n/)
-              key_match = key_regex.match(line)
-              if key_match
-                key = key_match[1]
-                value_match = value_regex.match(line)
-                if value_match
-                  value = value_match[1]
-                  value = CGI.unescapeHTML(value)
-                  value.gsub!('\\\'', '\'')
-                  value.gsub!('\\"', '"')
-                  value = iosify_substitutions(value)
-                  value.gsub!(/(\\u0020)*|(\\u0020)*\z/) { |spaces| ' ' * (spaces.length / 6) }
-                else
-                  value = ""
-                end
-                set_translation_for_key(key, lang, value)
-                if comment and comment.length > 0 and !comment.start_with?("SECTION:")
-                  set_comment_for_key(key, comment)
-                end
-                comment = nil
-              end
+            key = child.attributes['name']
 
-              comment_match = comment_regex.match(line)
-              if comment_match
-                comment = comment_match[1]
-              end
-            end
-          end
+            content = child.children.map(&:to_s).join
+            set_translation_for_key(key, lang, content)
+            set_comment_for_key(key, comment) if comment
+
+            comment = nil
+          end 
         end
       end
 
-      def write_file(path, lang)
-        default_lang = nil
-        if DEFAULT_LANG_CODES.has_key?(lang)
-          default_lang = DEFAULT_LANG_CODES[lang]
-        end
-        File.open(path, 'w:UTF-8') do |f|
-          f.puts "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<!-- Android Strings File -->\n<!-- Generated by Twine #{Twine::VERSION} -->\n<!-- Language: #{lang} -->"
-          f.write '<resources>'
-          @strings.sections.each do |section|
-            printed_section = false
-            section.rows.each do |row|
-              if row.matches_tags?(@options[:tags], @options[:untagged])
-                if !printed_section
-                  f.puts ''
-                  if section.name && section.name.length > 0
-                    section_name = section.name.gsub('--', '—')
-                    f.puts "\t<!-- SECTION: #{section_name} -->"
-                  end
-                  printed_section = true
-                end
+      def format_header(lang)
+        "<?xml version=\"1.0\" encoding=\"utf-8\"?>"
+      end
 
-                key = row.key
+      def format_sections(twine_file, lang)
+        result = '<resources>'
+        
+        result += super + "\n"
 
-                value = row.translated_string_for_lang(lang, default_lang)
-                if !value && @options[:include_untranslated]
-                  value = row.translated_string_for_lang(@strings.language_codes[0])
-                end
+        result += "</resources>\n"
+      end
 
-                if value # if values is nil, there was no appropriate translation, so let Android handle the defaulting
-                  value = String.new(value) # use a copy to prevent modifying the original
+      def format_section_header(section)
+        "#{space(4)}<!-- SECTION: #{section.name} -->"
+      end
 
-                  # Android enforces the following rules on the values
-                  #  1) apostrophes and quotes must be escaped with a backslash
-                  value.gsub!('\'', '\\\\\'')
-                  value.gsub!('"', '\\\\"')
-                  #  2) HTML escape the string
-                  value = CGI.escapeHTML(value)
-                  #  3) fix substitutions (e.g. %s/%@)
-                  value = androidify_substitutions(value)
-                  #  4) replace beginning and end spaces with \0020. Otherwise Android strips them.
-                  value.gsub!(/\A *| *\z/) { |spaces| '\u0020' * spaces.length }
+      def format_comment(definition, lang)
+        "#{space(4)}<!-- #{definition.comment.gsub('--', '—')} -->\n" if definition.comment
+      end
 
-                  comment = row.comment
-                  if comment
-                    comment = comment.gsub('--', '—')
-                  end
+      def key_value_pattern
+        "#{space(4)}<string name=\"%{key}\">%{value}</string>"
+      end
 
-                  if comment && comment.length > 0
-                    f.puts "\t<!-- #{comment} -->\n"
-                  end
-                  f.puts "\t<string name=\"#{key}\">#{value}</string>"
-                end
-              end
-            end
-          end
+      def format_plural_keys(key, plural_hash)
+        result = "#{space(4)}<plurals name=\"#{key}\">\n"
+        result += plural_hash.map{|quantity,value| "#{space(8)}<item quantity=\"#{quantity}\">#{escape_value(value)}</item>"}.join("\n")
+        result += "\n#{space(4)}</plurals>"
+      end
 
-          f.puts '</resources>'
+      def space(level)
+        ' ' * level
+      end
+
+      def gsub_unless(text, pattern, replacement)
+        text.gsub(pattern) do |match|
+          match_start_position = Regexp.last_match.offset(0)[0]
+          yield(text[0, match_start_position]) ? match : replacement
         end
       end
+
+      # http://developer.android.com/guide/topics/resources/string-resource.html#FormattingAndStyling
+      def escape_value(value)
+        inside_cdata = /<\!\[CDATA\[((?!\]\]>).)*$/              # opening CDATA tag ('<![CDATA[') not followed by a closing tag (']]>')
+        inside_opening_tag = /<(a|font|span|p)\s?((?!>).)*$/     # tag start ('<a ', '<font ', '<span ' or '<p ') not followed by a '>'
+
+        # escape double and single quotes and & signs
+        value = gsub_unless(value, '"', '\\"') { |substring| substring =~ inside_cdata || substring =~ inside_opening_tag }
+        value = gsub_unless(value, "'", "\\'") { |substring| substring =~ inside_cdata }
+        value = gsub_unless(value, /&/, '&amp;') { |substring| substring =~ inside_cdata || substring =~ inside_opening_tag }
+
+        # if `value` contains a placeholder, escape all angle brackets
+        # if not, escape opening angle brackes unless it's a supported styling tag
+        # https://github.com/scelis/twine/issues/212
+        # https://stackoverflow.com/questions/3235131/#18199543
+        if number_of_twine_placeholders(value) > 0 or @options[:escape_all_tags]
+          # matches all `<` but <![CDATA
+          angle_bracket = /<(?!(\/?(\!\[CDATA)))/
+        else
+          # matches all '<' but <b>, <em>, <i>, <cite>, <dfn>, <big>, <small>, <font>, <tt>, <s>,
+          # <strike>, <del>, <u>, <super>, <sub>, <ul>, <li>, <br>, <div>, <span>, <p>, <a>
+          # and <![CDATA
+          angle_bracket = /<(?!(\/?(b|em|i|cite|dfn|big|small|font|tt|s|strike|del|u|super|sub|ul|li|br|div|span|p|a|\!\[CDATA)))/
+        end
+        value = gsub_unless(value, angle_bracket, '&lt;') { |substring| substring =~ inside_cdata }
+        value = gsub_unless(value, '\n', "\n\\n") { |substring| substring =~ inside_cdata }
+
+        # escape non resource identifier @ signs (http://developer.android.com/guide/topics/resources/accessing-resources.html#ResourcesFromXml)
+        resource_identifier_regex = /@(?!([a-z\.]+:)?[a-z+]+\/[a-zA-Z_]+)/   # @[<package_name>:]<resource_type>/<resource_name>
+        value.gsub(resource_identifier_regex, '\@')
+      end
+
+      # see http://developer.android.com/guide/topics/resources/string-resource.html#FormattingAndStyling
+      # however unescaped HTML markup like in "Welcome to <b>Android</b>!" is stripped when retrieved with getString() (http://stackoverflow.com/questions/9891996/)
+      def format_value(value)
+        value = value.dup
+
+        # convert placeholders (e.g. %@ -> %s)
+        value = convert_placeholders_from_twine_to_android(value)
+
+        # capture xliff tags and replace them with a placeholder
+        xliff_tags = []
+        value.gsub! /<xliff:g.+?<\/xliff:g>/ do
+          xliff_tags << $&
+          'TWINE_XLIFF_TAG_PLACEHOLDER'
+        end
+
+        # escape everything outside xliff tags
+        value = escape_value(value)
+
+        # put xliff tags back into place
+        xliff_tags.each do |xliff_tag|
+          # escape content of xliff tags
+          xliff_tag.gsub! /(<xliff:g.*?>)(.*)(<\/xliff:g>)/ do "#{$1}#{escape_value($2)}#{$3}" end
+          value.sub! 'TWINE_XLIFF_TAG_PLACEHOLDER', xliff_tag
+        end
+        
+        # replace beginning and end spaces with \u0020. Otherwise Android strips them.
+        value.gsub(/\A *| *\z/) { |spaces| '\u0020' * spaces.length }
+      end
+
     end
   end
 end
+
+Twine::Formatters.formatters << Twine::Formatters::Android.new

lib/twine/formatters/apple.rb

@@ -1,16 +1,22 @@
 module Twine
   module Formatters
     class Apple < Abstract
-      FORMAT_NAME = 'apple'
-      EXTENSION = '.strings'
-      DEFAULT_FILE_NAME = 'Localizable.strings'
+      include Twine::Placeholders
 
-      def self.can_handle_directory?(path)
+      def format_name
+        'apple'
+      end
+
+      def extension
+        '.strings'
+      end
+
+      def can_handle_directory?(path)
         Dir.entries(path).any? { |item| /^.+\.lproj$/.match(item) }
       end
 
       def default_file_name
-        return DEFAULT_FILE_NAME
+        'Localizable.strings'
       end
 
       def determine_language_given_path(path)
@@ -18,108 +24,79 @@ module Twine
         path_arr.each do |segment|
           match = /^(.+)\.lproj$/.match(segment)
           if match
-            return match[1]
+            if match[1] == "Base"
+              return @options[:developer_language]
+            else
+              return match[1]
+            end
           end
         end
 
-        return
+        return super
       end
 
-      def read_file(path, lang)
-        encoding = Twine::Encoding.encoding_for_path(path)
-        sep = nil
-        if !encoding.respond_to?(:encode)
-          # This code is not necessary in 1.9.3 and does not work as it did in 1.8.7.
-          if encoding.end_with? 'LE'
-            sep = "\x0a\x00"
-          elsif encoding.end_with? 'BE'
-            sep = "\x00\x0a"
+      def output_path_for_language(lang)
+        "#{lang}.lproj"
+      end
+
+      def read(io, lang)
+        last_comment = nil
+        while line = io.gets
+          # matches a `key = "value"` line, where key may be quoted or unquoted. The former may also contain escaped characters
+          match = /^\s*((?:"(?:[^"\\]|\\.)+")|(?:[^"\s=]+))\s*=\s*"((?:[^"\\]|\\.)*)"/.match(line)
+          if match
+            key = match[1]
+            key = key[1..-2] if key[0] == '"' and key[-1] == '"'
+            key.gsub!('\\"', '"')
+            value = match[2]
+            value.gsub!('\\"', '"')
+            set_translation_for_key(key, lang, value)
+            if last_comment
+              set_comment_for_key(key, last_comment)
+            end
+          end
+
+          match = /\/\* (.*) \*\//.match(line)
+          if match
+            last_comment = match[1]
           else
-            sep = "\n"
-          end
-        end
-
-        if encoding.index('UTF-16')
-          mode = "rb:#{encoding}"
-        else
-          mode = "r:#{encoding}"
-        end
-
-        File.open(path, mode) do |f|
-          last_comment = nil
-          while line = (sep) ? f.gets(sep) : f.gets
-            if encoding.index('UTF-16')
-              if line.respond_to? :encode!
-                line.encode!('UTF-8')
-              else
-                require 'iconv'
-                line = Iconv.iconv('UTF-8', encoding, line).join
-              end
-            end
-            match = /"((?:[^"\\]|\\.)+)"\s*=\s*"((?:[^"\\]|\\.)*)"/.match(line)
-            if match
-              key = match[1]
-              key.gsub!('\\"', '"')
-              value = match[2]
-              value.gsub!('\\"', '"')
-              value = iosify_substitutions(value)
-              set_translation_for_key(key, lang, value)
-              if last_comment
-                set_comment_for_key(key, last_comment)
-              end
-            end
-            if @options[:consume_comments]
-              match = /\/\* (.*) \*\//.match(line)
-              if match
-                last_comment = match[1]
-              else
-                last_comment = nil
-              end
-            end
+            last_comment = nil
           end
         end
       end
 
-      def write_file(path, lang)
-        default_lang = @strings.language_codes[0]
-        encoding = @options[:output_encoding] || 'UTF-8'
-        File.open(path, "w:#{encoding}") do |f|
-          f.puts "/**\n * Apple Strings File\n * Generated by Twine #{Twine::VERSION}\n * Language: #{lang}\n */"
-          @strings.sections.each do |section|
-            printed_section = false
-            section.rows.each do |row|
-              if row.matches_tags?(@options[:tags], @options[:untagged])
-                f.puts ''
-                if !printed_section
-                  if section.name && section.name.length > 0
-                    f.print "/********** #{section.name} **********/\n\n"
-                  end
-                  printed_section = true
-                end
+      def format_file(lang)
+        result = super
+        result += "\n"
+      end
 
-                key = row.key
-                key = key.gsub('"', '\\\\"')
+      def format_section_header(section)
+        "\n/********** #{section.name} **********/\n"
+      end
 
-                value = row.translated_string_for_lang(lang, default_lang)
-                if value
-                  value = value.gsub('"', '\\\\"')
+      def key_value_pattern
+        "\"%{key}\" = \"%{value}\";"
+      end
 
-                  comment = row.comment
-                  if comment
-                    comment = comment.gsub('*/', '* /')
-                  end
+      def format_comment(definition, lang)
+        "\n/* #{definition.comment.gsub('*/', '* /')} */\n" if definition.comment
+      end
 
-                  if comment && comment.length > 0
-                    f.print "/* #{comment} */\n"
-                  end
+      def format_key(key)
+        escape_quotes(key)
+      end
 
-                  f.print "\"#{key}\" = \"#{value}\";\n"
-                end
-              end
-            end
-          end
-        end
+      def format_value(value)
+        # Replace Android's %s with iOS %@
+        value = convert_placeholders_from_android_to_twine(value)
+        escape_quotes(value)
+      end
+
+      def should_include_definition(definition, lang)
+        return !definition.is_plural? && super
       end
     end
   end
 end
+
+Twine::Formatters.formatters << Twine::Formatters::Apple.new

lib/twine/formatters/apple_plural.rb

@@ -0,0 +1,72 @@
+module Twine
+  module Formatters
+    class ApplePlural < Apple
+      include Twine::Placeholders
+
+      SUPPORTS_PLURAL = true
+
+      def format_name
+        'apple-plural'
+      end
+
+      def extension
+        '.stringsdict'
+      end
+
+      def default_file_name
+        'Localizable.stringsdict'
+      end
+
+      def format_footer(lang)
+        footer = "</dict>\n</plist>"
+      end
+
+      def format_file(lang)
+        result = super
+        result += format_footer(lang)
+      end
+
+      def format_header(lang)
+        header =  "<\?xml version=\"1.0\" encoding=\"UTF-8\"\?>\n"
+        header += "<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">\n"
+        header += "<plist version=\"1.0\">\n<dict>"
+      end
+
+      def format_section_header(section)
+        "<!-- ********** #{section.name} **********/ -->\n"
+      end
+
+      def format_plural_keys(key, plural_hash)
+        result = "\t<key>#{key}</key>\n"
+        result += "\t<dict>\n"
+        result += "\t\t<key>NSStringLocalizedFormatKey</key>\n"
+        result += "\t\t<string>\%\#@value@</string>\n"
+        result += "\t\t<key>value</key>\n"
+        result += "\t\t<dict>\n"
+        result += "\t\t\t<key>NSStringFormatSpecTypeKey</key>\n"
+        result += "\t\t\t<string>NSStringPluralRuleType</string>\n"
+        result += "\t\t\t<key>NSStringFormatValueTypeKey</key>\n"
+        result += "\t\t\t<string>d</string>\n"
+        # Replace Android's %s with iOS %@
+        result += plural_hash.map{|quantity,value| "\t\t\t<key>#{quantity}</key>\n\t\t\t<string>#{convert_placeholders_from_android_to_twine(value)}</string>"}.join("\n")
+        result += "\n"
+        result += "\t\t</dict>\n"
+        result += "\t</dict>\n"
+      end
+
+      def format_comment(definition, lang)
+        "<!-- #{definition.comment.gsub('--', '—')} -->\n" if definition.comment
+      end
+
+      def read(io, lang)
+        raise NotImplementedError.new("Reading \".stringdict\" files not implemented yet")
+      end
+
+      def should_include_definition(definition, lang)
+        return definition.is_plural? && definition.plural_translation_for_lang(lang)
+      end
+    end
+  end
+end
+
+Twine::Formatters.formatters << Twine::Formatters::ApplePlural.new

lib/twine/formatters/django.rb

@@ -1,143 +1,94 @@
 module Twine
   module Formatters
+    # For a description of the .po file format, see https://www.gnu.org/software/gettext/manual/html_node/PO-Files.html
     class Django < Abstract
-      FORMAT_NAME = 'django'
-      EXTENSION = '.po'
-      DEFAULT_FILE_NAME = 'strings.po'
+      def format_name
+        'django'
+      end
 
-      def self.can_handle_directory?(path)
-      Dir.entries(path).any? { |item| /^.+\.po$/.match(item) }
+      def extension
+        '.po'
       end
 
       def default_file_name
-        return DEFAULT_FILE_NAME
+        'strings.po'
       end
 
-      def determine_language_given_path(path)
-          path_arr = path.split(File::SEPARATOR)
-          path_arr.each do |segment|
-              match = /(..)\.po$/.match(segment)
-              if match
-                  return match[1]
-              end
+      def read(io, lang)
+        comment_regex = /^\s*#\. *"?(.*)"?$/
+        key_regex = /^msgid *"(.*)"$/
+        value_regex = /^msgstr *"(.*)"$/m
+
+        while line = io.gets          
+          comment_match = comment_regex.match(line)
+          if comment_match
+            comment = comment_match[1]
           end
-          
-        return
-      end
 
-      def read_file(path, lang)
-        comment_regex = /#.? *"(.*)"$/
-        key_regex = /msgid *"(.*)"$/
-        value_regex = /msgstr *"(.*)"$/m
-
-        encoding = Twine::Encoding.encoding_for_path(path)
-        sep = nil
-        if !encoding.respond_to?(:encode)
-          # This code is not necessary in 1.9.3 and does not work as it did in 1.8.7.
-          if encoding.end_with? 'LE'
-            sep = "\x0a\x00"
-          elsif encoding.end_with? 'BE'
-            sep = "\x00\x0a"
-          else
-            sep = "\n"
+          key_match = key_regex.match(line)
+          if key_match
+            key = key_match[1].gsub('\\"', '"')
+          end
+          value_match = value_regex.match(line)
+          if value_match
+            value = value_match[1].gsub(/"\n"/, '').gsub('\\"', '"')
           end
-        end
 
-        if encoding.index('UTF-16')
-          mode = "rb:#{encoding}"
-        else
-          mode = "r:#{encoding}"
-        end
-
-        File.open(path, mode) do |f|
-          last_comment = nil
-          while line = (sep) ? f.gets(sep) : f.gets
-            if encoding.index('UTF-16')
-              if line.respond_to? :encode!
-                line.encode!('UTF-8')
-              else
-                require 'iconv'
-                line = Iconv.iconv('UTF-8', encoding, line).join
-              end
+          if key and key.length > 0 and value and value.length > 0
+            set_translation_for_key(key, lang, value)
+            if comment and comment.length > 0 and !comment.start_with?("--------- ")
+              set_comment_for_key(key, comment)
             end
-            if @options[:consume_comments]
-               comment_match = comment_regex.match(line)
-               if comment_match
-                   comment = comment_match[1]
-               end
-            else
-                comment = nil
-            end
-            key_match = key_regex.match(line)
-            if key_match
-                key = key_match[1].gsub('\\"', '"')
-            end
-            value_match = value_regex.match(line)
-            if value_match
-                value = value_match[1].gsub(/"\n"/, '').gsub('\\"', '"')
-            end
-
-
-            if key and key.length > 0 and value and value.length > 0
-                set_translation_for_key(key, lang, value)
-                if comment and comment.length > 0 and !comment.start_with?("--------- ")
-                    set_comment_for_key(key, comment)
-                end
-                comment = nil
-            end
-
+            key = nil
+            value = nil
+            comment = nil
           end
         end
       end
 
-      def write_file(path, lang)
-        default_lang = @strings.language_codes[0]
-        encoding = @options[:output_encoding] || 'UTF-8'
-        File.open(path, "w:#{encoding}") do |f|
-          f.puts "##\n # Django Strings File\n # Generated by Twine #{Twine::VERSION}\n # Language: #{lang}\n "
-          @strings.sections.each do |section|
-            printed_section = false
-            section.rows.each do |row|
-              if row.matches_tags?(@options[:tags], @options[:untagged])
-                f.puts ''
-                if !printed_section
-                  if section.name && section.name.length > 0
-                    f.print "#--------- #{section.name} ---------#\n\n"
-                  end
-                  printed_section = true
-                end
-                
-                basetrans = row.translated_string_for_lang(default_lang)
+      def format_file(lang)
+        @default_lang = @twine_file.language_codes[0]
+        result = super
+        @default_lang = nil
+        result
+      end
 
-                key = row.key
-                key = key.gsub('"', '\\\\"')
+      def format_header(lang)
+        # see https://www.gnu.org/software/trans-coord/manual/gnun/html_node/PO-Header.html for details
+        "# Django Strings File\n# Generated by Twine\n# Language: #{lang}\nmsgid \"\"\nmsgstr \"\"\n\"Content-Type: text/plain; charset=UTF-8\\n\""
+      end
 
-                value = row.translated_string_for_lang(lang, default_lang)
-                if value
-                  value = value.gsub('"', '\\\\"')
+      def format_section_header(section)
+        "# --------- #{section.name} --------- #\n"
+      end
 
-                  comment = row.comment
+      def format_definition(definition, lang)
+        [format_comment(definition, lang), format_base_translation(definition), format_key_value(definition, lang)].compact.join
+      end
 
-                  if comment
-                     comment = comment.gsub('"', '\\\\"')
-                  end
+      def format_base_translation(definition)
+        base_translation = definition.translations[@default_lang]
+        "# base translation: \"#{base_translation}\"\n" if base_translation
+      end
 
-                  if comment && comment.length > 0
-                    f.print "#. #{comment} \n"
-                  end
+      def key_value_pattern
+        "msgid \"%{key}\"\n" +
+        "msgstr \"%{value}\"\n"
+      end
 
-                  if basetrans && basetrans.length > 0
-                    f.print "# base translation: \"#{basetrans}\"\n"
-                  end
+      def format_comment(definition, lang)
+        "#. #{escape_quotes(definition.comment)}\n" if definition.comment
+      end
 
-                  f.print "msgid \"#{key}\"\n"
-                  f.print "msgstr \"#{value}\"\n"
-                end
-              end
-            end
-          end
-        end
+      def format_key(key)
+        escape_quotes(key)
+      end
+
+      def format_value(value)
+        escape_quotes(value)
       end
     end
   end
 end
+
+Twine::Formatters.formatters << Twine::Formatters::Django.new

lib/twine/formatters/flash.rb

@@ -1,110 +1,67 @@
 module Twine
   module Formatters
     class Flash < Abstract
-      FORMAT_NAME = 'flash'
-      EXTENSION = '.properties'
-      DEFAULT_FILE_NAME = 'resources.properties'
+      include Twine::Placeholders
 
-      def self.can_handle_directory?(path)
-        return false
+      def format_name
+        'flash'
+      end
+
+      def extension
+        '.properties'
       end
 
       def default_file_name
-        return DEFAULT_FILE_NAME
+        'resources.properties'
       end
 
-      def determine_language_given_path(path)
-        return
+      def set_translation_for_key(key, lang, value)
+        value = convert_placeholders_from_flash_to_twine(value)
+        super(key, lang, value)
       end
 
-      def read_file(path, lang)
-        encoding = Twine::Encoding.encoding_for_path(path)
-        sep = nil
-        if !encoding.respond_to?(:encode)
-          # This code is not necessary in 1.9.3 and does not work as it did in 1.8.7.
-          if encoding.end_with? 'LE'
-            sep = "\x0a\x00"
-          elsif encoding.end_with? 'BE'
-            sep = "\x00\x0a"
-          else
-            sep = "\n"
-          end
-        end
+      def read(io, lang)
+        last_comment = nil
+        while line = io.gets
+          match = /((?:[^"\\]|\\.)+)\s*=\s*((?:[^"\\]|\\.)*)/.match(line)
+          if match
+            key = match[1]
+            value = match[2].strip
 
-        if encoding.index('UTF-16')
-          mode = "rb:#{encoding}"
-        else
-          mode = "r:#{encoding}"
-        end
-
-        File.open(path, mode) do |f|
-          last_comment = nil
-          while line = (sep) ? f.gets(sep) : f.gets
-            if encoding.index('UTF-16')
-              if line.respond_to? :encode!
-                line.encode!('UTF-8')
-              else
-                require 'iconv'
-                line = Iconv.iconv('UTF-8', encoding, line).join
-              end
-            end
-            match = /((?:[^"\\]|\\.)+)\s*=\s*((?:[^"\\]|\\.)*)/.match(line)
-            if match
-              key = match[1]
-              value = match[2]
-              value.gsub!(/\{[0-9]\}/, '%@')
-              set_translation_for_key(key, lang, value)
-              if last_comment
-                set_comment_for_key(key, last_comment)
-              end
-            end
-            if @options[:consume_comments]
-              match = /#(.*)/.match(line)
-              if match
-                last_comment = match[1]
-              else
-                last_comment = nil
-              end
-            end
+            set_translation_for_key(key, lang, value)
+            set_comment_for_key(key, last_comment) if last_comment
           end
+          
+          match = /# *(.*)/.match(line)
+          last_comment = match ? match[1] : nil
         end
       end
 
-      def write_file(path, lang)
-        default_lang = @strings.language_codes[0]
-        encoding = @options[:output_encoding] || 'UTF-8'
-        File.open(path, "w:#{encoding}") do |f|
-          f.puts "## Flash Strings File\n## Generated by Twine #{Twine::VERSION}\n## Language: #{lang}\n"
-          @strings.sections.each do |section|
-            printed_section = false
-            section.rows.each do |row|
-              if row.matches_tags?(@options[:tags], @options[:untagged])
-                f.puts ''
-                if !printed_section
-                  if section.name && section.name.length > 0
-                    f.print "## #{section.name} ##\n\n"
-                  end
-                  printed_section = true
-                end
+      def format_sections(twine_file, lang)
+        super + "\n"
+      end
 
-                key = row.key
-                value = row.translated_string_for_lang(lang, default_lang)
-                if value
-                  placeHolderNumber = -1
-                  value = value.gsub(/%[d@]/) { placeHolderNumber += 1; '{%d}' % placeHolderNumber }
-                  
-                  comment = row.comment
-                  if comment && comment.length > 0
-                    f.print "# #{comment}\n"
-                  end
+      def format_header(lang)
+        "## Flash Strings File\n## Generated by Twine\n## Language: #{lang}"
+      end
 
-                  f.print "#{key}=#{value}"
-                end
-              end
-            end
-          end
-        end
+      def format_section_header(section)
+        "## #{section.name} ##\n"
+      end
+
+      def format_comment(definition, lang)
+        "# #{definition.comment}\n" if definition.comment
+      end
+
+      def key_value_pattern
+        "%{key}=%{value}"
+      end
+
+      def format_value(value)
+        convert_placeholders_from_twine_to_flash(value)
       end
     end
   end
 end
+
+Twine::Formatters.formatters << Twine::Formatters::Flash.new

lib/twine/formatters/gettext.rb

@@ -3,108 +3,91 @@
 module Twine
   module Formatters
     class Gettext < Abstract
-      FORMAT_NAME = 'gettext'
-      EXTENSION = '.po'
-      DEFAULT_FILE_NAME = 'strings.po'
+      def format_name
+        'gettext'
+      end
 
-      def self.can_handle_directory?(path)
-        Dir.entries(path).any? { |item| /^.+\.po$/.match(item) }
+      def extension
+        '.po'
       end
 
       def default_file_name
-        return DEFAULT_FILE_NAME
+        'strings.po'
       end
 
-      def determine_language_given_path(path)
-        path_arr = path.split(File::SEPARATOR)
-        path_arr.each do |segment|
-          match = /(..)\.po$/.match(segment)
-          if match
-            return match[1]
-          end
-        end
-
-        return
-      end
-
-      def read_file(path, lang)
+      def read(io, lang)
         comment_regex = /#.? *"(.*)"$/
         key_regex = /msgctxt *"(.*)"$/
         value_regex = /msgstr *"(.*)"$/m
-        File.open(path, 'r:UTF-8') do |f|
-          while item = f.gets("\n\n")
-            key = nil
-            value = nil
-            comment = nil
+        
+        while item = io.gets("\n\n")
+          key = nil
+          value = nil
+          comment = nil
 
-            comment_match = comment_regex.match(item)
-            if comment_match
-              comment = comment_match[1]
-            end
-            key_match = key_regex.match(item)
-            if key_match
-              key = key_match[1].gsub('\\"', '"')
-            end
-            value_match = value_regex.match(item)
-            if value_match
-              value = value_match[1].gsub(/"\n"/, '').gsub('\\"', '"')
-            end
-            if key and key.length > 0 and value and value.length > 0
-              set_translation_for_key(key, lang, value)
-              if comment and comment.length > 0 and !comment.start_with?("SECTION:")
-                set_comment_for_key(key, comment)
-              end
-              comment = nil
+          comment_match = comment_regex.match(item)
+          if comment_match
+            comment = comment_match[1]
+          end
+          key_match = key_regex.match(item)
+          if key_match
+            key = key_match[1].gsub('\\"', '"')
+          end
+          value_match = value_regex.match(item)
+          if value_match
+            value = value_match[1].gsub(/"\n"/, '').gsub('\\"', '"')
+          end
+          if key and key.length > 0 and value and value.length > 0
+            set_translation_for_key(key, lang, value)
+            if comment and comment.length > 0 and !comment.start_with?("SECTION:")
+              set_comment_for_key(key, comment)
             end
+            comment = nil
           end
         end
       end
 
-      def write_file(path, lang)
-        default_lang = @strings.language_codes[0]
-        encoding = @options[:output_encoding] || 'UTF-8'
-        File.open(path, "w:#{encoding}") do |f|
-          f.puts "msgid \"\"\nmsgstr \"\"\n\"Language: #{lang}\\n\"\n\"X-Generator: Twine #{Twine::VERSION}\\n\"\n\n"
-          @strings.sections.each do |section|
-            printed_section = false
-            section.rows.each do |row|
-              if row.matches_tags?(@options[:tags], @options[:untagged])
-                if !printed_section
-                  f.puts ''
-                    if section.name && section.name.length > 0
-                      section_name = section.name.gsub('--', '—')
-                      f.puts "# SECTION: #{section_name}"
-                    end
-                  printed_section = true
-                end
+      def format_file(lang)
+        @default_lang = twine_file.language_codes[0]
+        result = super
+        @default_lang = nil
+        result
+      end
 
-                basetrans = row.translated_string_for_lang(default_lang)
+      def format_header(lang)
+        "msgid \"\"\nmsgstr \"\"\n\"Language: #{lang}\"\n\"X-Generator: Twine #{Twine::VERSION}\"\n"
+      end
 
-                if basetrans
-                  key = row.key
-                  key = key.gsub('"', '\\\\"')
+      def format_section_header(section)
+        "# SECTION: #{section.name}"
+      end
 
-                  comment = row.comment
-                  if comment
-                    comment = comment.gsub('"', '\\\\"')
-                  end
+      def should_include_definition(definition, lang)
+        super and !definition.translation_for_lang(@default_lang).nil?
+      end
 
-                  if comment && comment.length > 0
-                    f.print "#. \"#{comment}\"\n"
-                  end
+      def format_comment(definition, lang)
+        "#. \"#{escape_quotes(definition.comment)}\"\n" if definition.comment
+      end
 
-                  f.print "msgctxt \"#{key}\"\nmsgid \"#{basetrans}\"\n"
-                  value = row.translated_string_for_lang(lang)
-                  if value
-                    value = value.gsub('"', '\\\\"')
-                  end
-                  f.print "msgstr \"#{value}\"\n\n"
-                end
-              end
-            end
-          end
-        end
+      def format_key_value(definition, lang)
+        value = definition.translation_for_lang(lang)
+        [format_key(definition.key.dup), format_base_translation(definition), format_value(value.dup)].compact.join
+      end
+
+      def format_key(key)
+        "msgctxt \"#{escape_quotes(key)}\"\n"
+      end
+
+      def format_base_translation(definition)
+        "msgid \"#{escape_quotes(definition.translations[@default_lang])}\"\n"
+      end
+
+      def format_value(value)
+        "msgstr \"#{escape_quotes(value)}\"\n"
       end
     end
   end
 end
+
+Twine::Formatters.formatters << Twine::Formatters::Gettext.new

lib/twine/formatters/jquery.rb

@@ -1,87 +1,74 @@
 module Twine
   module Formatters
     class JQuery < Abstract
-      FORMAT_NAME = 'jquery'
-      EXTENSION = '.json'
-      DEFAULT_FILE_NAME = 'localize.json'
+      def format_name
+        'jquery'
+      end
 
-      def self.can_handle_directory?(path)
-        Dir.entries(path).any? { |item| /^.+\.json$/.match(item) }
+      def extension
+        '.json'
       end
 
       def default_file_name
-        return DEFAULT_FILE_NAME
+        'localize.json'
       end
 
       def determine_language_given_path(path)
-        path_arr = path.split(File::SEPARATOR)
-        path_arr.each do |segment|
-          match = /^((.+)-)?([^-]+)\.json$/.match(segment)
-          if match
-            return match[3]
-          end
-        end
+        match = /^.+([a-z]{2}-[A-Z]{2})\.json$/.match File.basename(path)
+        return match[1] if match
 
-        return
+        return super
       end
 
-      def read_file(path, lang)
+      def read(io, lang)
         begin
           require "json"
         rescue LoadError
-          raise Twine::Error.new "You must run 'gem install json' in order to read or write jquery-localize files."
+          raise Twine::Error.new "You must run `gem install json` in order to read or write jquery-localize files."
         end
 
-        open(path) do |io|
-          json = JSON.load(io)
-          json.each do |key, value|
-            value.gsub!("\n","\\n")
-            set_translation_for_key(key, lang, value)
-          end
+        json = JSON.load(io)
+        json.each do |key, value|
+          set_translation_for_key(key, lang, value)
         end
       end
 
-      def write_file(path, lang)
-        begin
-          require "json"
-        rescue LoadError
-          raise Twine::Error.new "You must run 'gem install json' in order to read or write jquery-localize files."
-        end
+      def format_file(lang)
+        result = super
+        return result unless result
+        "{\n#{super}\n}\n"
+      end
 
-        printed_string = false
-        default_lang = @strings.language_codes[0]
-        encoding = @options[:output_encoding] || 'UTF-8'
-        File.open(path, "w:#{encoding}") do |f|
-          f.print "{"
+      def format_sections(twine_file, lang)
+        sections = twine_file.sections.map { |section| format_section(section, lang) }
+        sections.delete_if(&:empty?)
+        sections.join(",\n\n")
+      end
 
-          @strings.sections.each_with_index do |section, si|
-            printed_section = false
-            section.rows.each_with_index do |row, ri|
-              if row.matches_tags?(@options[:tags], @options[:untagged])
-                if printed_string
-                  f.print ",\n"
-                end
+      def format_section_header(section)
+      end
 
-                if !printed_section
-                  f.print "\n"
-                  printed_section = true
-                end
+      def format_section(section, lang)
+        definitions = section.definitions.dup
 
-                key = row.key
-                key = key.gsub('"', '\\\\"')
+        definitions.map! { |definition| format_definition(definition, lang) }
+        definitions.compact! # remove nil definitions
+        definitions.join(",\n")
+      end
 
-                value = row.translated_string_for_lang(lang, default_lang)
-                value = value.gsub('"', '\\\\"')
+      def key_value_pattern
+        "\"%{key}\":\"%{value}\""
+      end
 
-                f.print "\"#{key}\":\"#{value}\""
-                printed_string = true
-              end
-            end
-          end
-          f.puts "\n}"
+      def format_key(key)
+        escape_quotes(key)
+      end
 
-        end
+      def format_value(value)
+        escape_quotes(value)
       end
     end
   end
 end
+
+Twine::Formatters.formatters << Twine::Formatters::JQuery.new

lib/twine/formatters/tizen.rb

@@ -5,9 +5,8 @@ require 'rexml/document'
 module Twine
   module Formatters
     class Tizen < Abstract
-      FORMAT_NAME = 'tizen'
-      EXTENSION = '.xml'
-      DEFAULT_FILE_NAME = 'strings.xml'
+      include Twine::Placeholders
+
       LANG_CODES = Hash[
         'eng-GB' => 'en',
         'rus-RU' => 'ru',
@@ -20,39 +19,21 @@ module Twine
         'por-PT' => 'pt',
         'ukr-UA' => 'uk'
       ]
-      DEFAULT_LANG_CODES = Hash[
-      ]
 
-      def self.can_handle_directory?(path)
+      def format_name
+        'tizen'
+      end
+
+      def extension
+        '.xml'
+      end
+
+      def can_handle_directory?(path)
         Dir.entries(path).any? { |item| /^values.*$/.match(item) }
       end
 
       def default_file_name
-        return DEFAULT_FILE_NAME
-      end
-
-      def write_all_files(path)
-        if !File.directory?(path)
-          raise Twine::Error.new("Directory does not exist: #{path}")
-        end
-
-        langs_written = []
-        Dir.foreach(path) do |item|
-          if item == "." or item == ".."
-            next
-          end
-          item = File.join(path, item)
-          if !File.directory?(item)
-            lang = determine_language_given_path(item)
-            if lang
-              write_file(item, lang)
-              langs_written << lang
-            end
-          end
-        end
-        if langs_written.empty?
-          raise Twine::Error.new("Failed to genertate any files: No languages found at #{path}")
-        end
+        'strings.xml'
       end
 
       def determine_language_given_path(path)
@@ -68,7 +49,7 @@ module Twine
         return
       end
 
-      def read_file(path, lang)
+      def read(io, lang)
         resources_regex = /<resources(?:[^>]*)>(.*)<\/resources>/m
         key_regex = /<string name="(\w+)">/
         comment_regex = /<!-- (.*) -->/
@@ -77,99 +58,80 @@ module Twine
         value = nil
         comment = nil
 
-        File.open(path, 'r:UTF-8') do |f|
-          content_match = resources_regex.match(f.read)
-          if content_match
-            for line in content_match[1].split(/\r?\n/)
-              key_match = key_regex.match(line)
-              if key_match
-                key = key_match[1]
-                value_match = value_regex.match(line)
-                if value_match
-                  value = value_match[1]
-                  value = CGI.unescapeHTML(value)
-                  value.gsub!('\\\'', '\'')
-                  value.gsub!('\\"', '"')
-                  value = iosify_substitutions(value)
-                  value.gsub!(/(\\u0020)*|(\\u0020)*\z/) { |spaces| ' ' * (spaces.length / 6) }
-                else
-                  value = ""
-                end
-                set_translation_for_key(key, lang, value)
-                if comment and comment.length > 0 and !comment.start_with?("SECTION:")
-                  set_comment_for_key(key, comment)
-                end
-                comment = nil
+        content_match = resources_regex.match(io.read)
+        if content_match
+          for line in content_match[1].split(/\r?\n/)
+            key_match = key_regex.match(line)
+            if key_match
+              key = key_match[1]
+              value_match = value_regex.match(line)
+              if value_match
+                value = value_match[1]
+                value = CGI.unescapeHTML(value)
+                value.gsub!('\\\'', '\'')
+                value.gsub!('\\"', '"')
+                value = convert_placeholders_from_android_to_twine(value)
+                value.gsub!(/(\\u0020)*|(\\u0020)*\z/) { |spaces| ' ' * (spaces.length / 6) }
+              else
+                value = ""
               end
+              set_translation_for_key(key, lang, value)
+              if comment and comment.length > 0 and !comment.start_with?("SECTION:")
+                set_comment_for_key(key, comment)
+              end
+              comment = nil
+            end
 
-              comment_match = comment_regex.match(line)
-              if comment_match
-                comment = comment_match[1]
-              end
+            comment_match = comment_regex.match(line)
+            if comment_match
+              comment = comment_match[1]
             end
           end
         end
       end
 
-      def write_file(path, lang)
-        default_lang = nil
-        if DEFAULT_LANG_CODES.has_key?(lang)
-          default_lang = DEFAULT_LANG_CODES[lang]
-        end
-        File.open(path, 'w:UTF-8') do |f|
-          f.puts "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<!-- Tizen Strings File -->\n<!-- Generated by Twine #{Twine::VERSION} -->\n<!-- Language: #{lang} -->"
-          f.write '<string_table  Bversion="2.0.0.201311071819" Dversion="20120315">'
-          @strings.sections.each do |section|
-            printed_section = false
-            section.rows.each do |row|
-              if row.matches_tags?(@options[:tags], @options[:untagged])
-                if !printed_section
-                  f.puts ''
-                  if section.name && section.name.length > 0
-                    section_name = section.name.gsub('--', '—')
-                    f.puts "\t<!-- SECTION: #{section_name} -->"
-                  end
-                  printed_section = true
-                end
+      def format_header(lang)
+        "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<!-- Tizen Strings File -->\n<!-- Generated by Twine -->\n<!-- Language: #{lang} -->"
+      end
 
-                key = row.key
+      def format_sections(twine_file, lang)
+        result = '<string_table  Bversion="2.0.0.201311071819" Dversion="20120315">'
+        
+        result += super + "\n"
 
-                value = row.translated_string_for_lang(lang, default_lang)
-                if !value && @options[:include_untranslated]
-                  value = row.translated_string_for_lang(@strings.language_codes[0])
-                end
+        result += "</string_table>\n"
+      end
 
-                if value # if values is nil, there was no appropriate translation, so let Tizen handle the defaulting
-                  value = String.new(value) # use a copy to prevent modifying the original
+      def format_section_header(section)
+        "\t<!-- SECTION: #{section.name} -->"
+      end
 
-                  # Tizen enforces the following rules on the values
-                  #  1) apostrophes and quotes must be escaped with a backslash
-                  value.gsub!('\'', '\\\\\'')
-                  value.gsub!('"', '\\\\"')
-                  #  2) HTML escape the string
-                  value = CGI.escapeHTML(value)
-                  #  3) fix substitutions (e.g. %s/%@)
-                  value = androidify_substitutions(value)
-                  #  4) replace beginning and end spaces with \0020. Otherwise Tizen strips them.
-                  value.gsub!(/\A *| *\z/) { |spaces| '\u0020' * spaces.length }
+      def format_comment(definition, lang)
+        "\t<!-- #{definition.comment.gsub('--', '—')} -->\n" if definition.comment
+      end
 
-                  comment = row.comment
-                  if comment
-                    comment = comment.gsub('--', '—')
-                  end
+      def key_value_pattern
+        "\t<text id=\"IDS_%{key}\">%{value}</text>"
+      end
 
-                  if comment && comment.length > 0
-                    f.puts "\t<!-- #{comment} -->\n"
-                  end
-                  f.puts "\t<text id=\"IDS_#{key.upcase}\">#{value}</text>"
-                end
-              end
-            end
-          end
+      def format_key(key)
+        key.upcase
+      end
 
-          f.puts '</string_table>'
-        end
+      def format_value(value)
+        value = escape_quotes(value)
+        # Tizen enforces the following rules on the values
+        #  1) apostrophes and quotes must be escaped with a backslash
+        value.gsub!("'", "\\\\'")
+        #  2) HTML escape the string
+        value = CGI.escapeHTML(value)
+        #  3) fix substitutions (e.g. %s/%@)
+        value = convert_placeholders_from_twine_to_android(value)
+        #  4) replace beginning and end spaces with \0020. Otherwise Tizen strips them.
+        value.gsub(/\A *| *\z/) { |spaces| '\u0020' * spaces.length }
       end
     end
   end
 end
+
+Twine::Formatters.formatters << Twine::Formatters::Tizen.new

lib/twine/output_processor.rb

@@ -0,0 +1,71 @@
+module Twine
+  module Processors
+
+    class OutputProcessor
+      def initialize(twine_file, options)
+        @twine_file = twine_file
+        @options = options
+      end
+
+      def default_language
+        @options[:developer_language] || @twine_file.language_codes[0]
+      end
+
+      def fallback_languages(language)
+        fallback_mapping = {
+          'zh-CN' => 'zh-Hans', # if we don't have a zh-CN translation, try zh-Hans before en
+          'zh-TW' => 'zh-Hant' # if we don't have a zh-TW translation, try zh-Hant before en
+        }
+
+        # Regional dialect fallbacks to generic language (for example: 'es-MX' to 'es' instead of default 'en').
+        if language.match(/([a-zA-Z]{2})-[a-zA-Z]+/)
+          generic_language = language.gsub(/([a-zA-Z])-[a-zA-Z]+/, '\1')
+        end
+
+        [fallback_mapping[language], generic_language, default_language].flatten.compact
+      end
+
+      def process(language)
+        result = TwineFile.new
+
+        result.language_codes.concat @twine_file.language_codes
+        @twine_file.sections.each do |section|
+          new_section = TwineSection.new section.name
+
+          section.definitions.each do |definition|
+            next unless definition.matches_tags?(@options[:tags], @options[:untagged])
+
+            value = definition.translation_for_lang(language)
+
+            next if value && @options[:include] == :untranslated
+
+            if value.nil? && @options[:include] != :translated
+              value = definition.translation_for_lang(fallback_languages(language))
+            end
+
+            next unless value
+
+            new_definition = definition.dup
+            new_definition.translations[language] = value
+
+            if definition.is_plural?
+              # If definition is plural, but no translation found -> create
+              # Then check 'other' key
+              if !(new_definition.plural_translations[language] ||= {}).key? 'other'
+                new_definition.plural_translations[language]['other'] = value
+              end
+            end
+
+            new_section.definitions << new_definition
+            result.definitions_by_key[new_definition.key] = new_definition
+          end
+
+          result.sections << new_section
+        end
+
+        return result
+      end
+    end
+
+  end
+end

lib/twine/placeholders.rb

@@ -0,0 +1,82 @@
+module Twine
+  module Placeholders
+    extend self
+
+    # Note: the ` ` (single space) flag is NOT supported
+    PLACEHOLDER_FLAGS_WIDTH_PRECISION_LENGTH = '([-+0#])?(\d+|\*)?(\.(\d+|\*))?(hh?|ll?|L|z|j|t|q)?'
+    PLACEHOLDER_PARAMETER_FLAGS_WIDTH_PRECISION_LENGTH = '(\d+\$)?' + PLACEHOLDER_FLAGS_WIDTH_PRECISION_LENGTH
+    PLACEHOLDER_TYPES = '[diufFeEgGxXoscpaA]'
+    PLACEHOLDER_REGEX = /%#{PLACEHOLDER_PARAMETER_FLAGS_WIDTH_PRECISION_LENGTH}#{PLACEHOLDER_TYPES}/
+
+    def number_of_twine_placeholders(input)
+      input.scan(PLACEHOLDER_REGEX).size
+    end
+
+    def convert_twine_string_placeholder(input)
+      # %@ -> %s
+      input.gsub(/(%#{PLACEHOLDER_PARAMETER_FLAGS_WIDTH_PRECISION_LENGTH})@/, '\1s')
+    end
+
+    # http://developer.android.com/guide/topics/resources/string-resource.html#FormattingAndStyling
+    # http://stackoverflow.com/questions/4414389/android-xml-percent-symbol
+    # https://github.com/mobiata/twine/pull/106
+    def convert_placeholders_from_twine_to_android(input)
+      # %@ -> %s
+      value = convert_twine_string_placeholder(input)
+
+      number_of_placeholders = number_of_twine_placeholders(value)
+
+      return value if number_of_placeholders == 0
+
+      # got placeholders -> need to double single percent signs
+      # % -> %% (but %% -> %%, %d -> %d)
+      placeholder_syntax = PLACEHOLDER_PARAMETER_FLAGS_WIDTH_PRECISION_LENGTH + PLACEHOLDER_TYPES
+      single_percent_regex = /([^%])(%)(?!(%|#{placeholder_syntax}))/
+      value.gsub! single_percent_regex, '\1%%'
+
+      return value if number_of_placeholders < 2
+
+      # number placeholders
+      non_numbered_placeholder_regex = /%(#{PLACEHOLDER_FLAGS_WIDTH_PRECISION_LENGTH}#{PLACEHOLDER_TYPES})/
+
+      number_of_non_numbered_placeholders = value.scan(non_numbered_placeholder_regex).size
+
+      return value if number_of_non_numbered_placeholders == 0
+
+      raise Twine::Error.new("The value \"#{input}\" contains numbered and non-numbered placeholders") if number_of_placeholders != number_of_non_numbered_placeholders
+
+      # %d -> %$1d
+      index = 0
+      value.gsub!(non_numbered_placeholder_regex) { "%#{index += 1}$#{$1}" }
+
+      value
+    end
+
+    def convert_placeholders_from_android_to_twine(input)
+      placeholder_regex = /(%#{PLACEHOLDER_PARAMETER_FLAGS_WIDTH_PRECISION_LENGTH})s/
+
+      # %s -> %@
+      input.gsub(placeholder_regex, '\1@')
+    end
+
+    # http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/mx/resources/IResourceManager.html#getString()
+    # http://soenkerohde.com/2008/07/flex-localization/comment-page-1/
+    def convert_placeholders_from_twine_to_flash(input)
+      value = convert_twine_string_placeholder(input)
+
+      value.gsub(PLACEHOLDER_REGEX).each_with_index do |match, index|
+        "{#{index}}"
+      end
+    end
+
+    def convert_placeholders_from_flash_to_twine(input)
+      input.gsub /\{\d+\}/, '%@'
+    end
+
+    # Python supports placeholders in the form of `%(amount)03d`
+    # see https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting
+    def contains_python_specific_placeholder(input)
+      /%\([a-zA-Z0-9_-]+\)#{PLACEHOLDER_PARAMETER_FLAGS_WIDTH_PRECISION_LENGTH}#{PLACEHOLDER_TYPES}/.match(input) != nil
+    end
+  end
+end

lib/twine/plugin.rb

@@ -1,3 +1,4 @@
+require 'date'
 require 'safe_yaml/load'
 
 SafeYAML::OPTIONS[:suppress_warnings] = true
@@ -56,7 +57,7 @@ module Twine
     end
 
     def join_path *paths
-      File.expand_path File.join *paths
+      File.expand_path File.join(*paths)
     end
   end
 end

lib/twine/runner.rb

@@ -1,303 +1,357 @@
 require 'tmpdir'
+require 'fileutils'
 
 Twine::Plugin.new # Initialize plugins first in Runner.
 
 module Twine
-  VALID_COMMANDS = ['generate-string-file', 'generate-all-string-files', 'consume-string-file', 'consume-all-string-files', 'generate-loc-drop', 'consume-loc-drop', 'generate-report']
-
   class Runner
-    def initialize(args)
-      @options = {}
-      @args = args
+    class NullOutput
+      def puts(message)
+      end
+      def string
+        ""
+      end
     end
 
     def self.run(args)
-      new(args).run
+      options = CLI.parse(args)
+
+      return unless options
+      
+      twine_file = TwineFile.new
+      twine_file.read options[:twine_file]
+      runner = new(options, twine_file)
+
+      case options[:command]
+      when 'generate-localization-file'
+        runner.generate_localization_file
+      when 'generate-all-localization-files'
+        runner.generate_all_localization_files
+      when 'consume-localization-file'
+        runner.consume_localization_file
+      when 'consume-all-localization-files'
+        runner.consume_all_localization_files
+      when 'generate-localization-archive'
+        runner.generate_localization_archive
+      when 'consume-localization-archive'
+        runner.consume_localization_archive
+      when 'validate-twine-file'
+        runner.validate_twine_file
+      end
     end
 
-    def run
-      # Parse all CLI arguments.
-      CLI::parse_args(@args, @options)
-      read_strings_data
-      execute_command
+    def initialize(options = {}, twine_file = TwineFile.new)
+      @options = options
+      @twine_file = twine_file
+      if @options[:quite]
+        Twine::stdout = NullOutput.new
+      end
     end
 
-    def read_strings_data
-      @strings = StringsFile.new
-      @strings.read @options[:strings_file]
-    end
-
-    def write_strings_data(path)
+    def write_twine_data(path)
       if @options[:developer_language]
-        @strings.set_developer_language_code(@options[:developer_language])
+        @twine_file.set_developer_language_code(@options[:developer_language])
       end
-      @strings.write(path)
+      @twine_file.write(path)
     end
 
-    def execute_command
-      case @options[:command]
-      when 'generate-string-file'
-        generate_string_file
-      when 'generate-all-string-files'
-        generate_all_string_files
-      when 'consume-string-file'
-        consume_string_file
-      when 'consume-all-string-files'
-        consume_all_string_files
-      when 'generate-loc-drop'
-        generate_loc_drop
-      when 'consume-loc-drop'
-        consume_loc_drop
-      when 'generate-report'
-        generate_report
-      end
-    end
+    def generate_localization_file
+      validate_twine_file if @options[:validate]
 
-    def generate_string_file
       lang = nil
-      if @options[:languages]
-        lang = @options[:languages][0]
-      end
+      lang = @options[:languages][0] if @options[:languages]
 
-      read_write_string_file(@options[:output_path], false, lang)
+      formatter, lang = prepare_read_write(@options[:output_path], lang)
+      output = formatter.format_file(lang)
+
+      raise Twine::Error.new "Nothing to generate! The resulting file would not contain any translations." unless output
+
+      IO.write(@options[:output_path], output, encoding: output_encoding)
     end
 
-    def generate_all_string_files
+    def generate_all_localization_files
+      validate_twine_file if @options[:validate]
+
       if !File.directory?(@options[:output_path])
-        raise Twine::Error.new("Directory does not exist: #{@options[:output_path]}")
-      end
-
-      format = @options[:format]
-      if !format
-        format = determine_format_given_directory(@options[:output_path])
-      end
-      if !format
-        raise Twine::Error.new "Could not determine format given the contents of #{@options[:output_path]}"
-      end
-
-      formatter = formatter_for_format(format)
-
-      formatter.write_all_files(@options[:output_path])
-    end
-
-    def consume_string_file
-      lang = nil
-      if @options[:languages]
-        lang = @options[:languages][0]
-      end
-
-      read_write_string_file(@options[:input_path], true, lang)
-      output_path = @options[:output_path] || @options[:strings_file]
-      write_strings_data(output_path)
-    end
-
-    def consume_all_string_files
-      if !File.directory?(@options[:input_path])
-        raise Twine::Error.new("Directory does not exist: #{@options[:output_path]}")
-      end
-
-      Dir.glob(File.join(@options[:input_path], "**/*")) do |item|
-        if File.file?(item)
-          begin
-            read_write_string_file(item, true, nil)
-          rescue Twine::Error => e
-            STDERR.puts "#{e.message}"
-          end
+        if @options[:create_folders]
+          FileUtils.mkdir_p(@options[:output_path])
+        else
+          raise Twine::Error.new("Directory does not exist: #{@options[:output_path]}")
         end
       end
 
-      output_path = @options[:output_path] || @options[:strings_file]
-      write_strings_data(output_path)
-    end
-
-    def read_write_string_file(path, is_read, lang)
-      if is_read && !File.file?(path)
-        raise Twine::Error.new("File does not exist: #{path}")
-      end
-
-      format = @options[:format]
-      if !format
-        format = determine_format_given_path(path)
-      end
-      if !format
-        raise Twine::Error.new "Unable to determine format of #{path}"
-      end
-
-      formatter = formatter_for_format(format)
-
-      if !lang
-        lang = determine_language_given_path(path)
-      end
-      if !lang
-        lang = formatter.determine_language_given_path(path)
-      end
-      if !lang
-        raise Twine::Error.new "Unable to determine language for #{path}"
-      end
-
-      if !@strings.language_codes.include? lang
-        @strings.language_codes << lang
-      end
-
-      if is_read
-        formatter.read_file(path, lang)
+      if @options[:format]
+        formatter = formatter_for_format(@options[:format])
       else
-        formatter.write_file(path, lang)
+        formatter = find_formatter { |f| f.can_handle_directory?(@options[:output_path]) }
       end
+      
+      unless formatter
+        raise Twine::Error.new "Could not determine format given the contents of #{@options[:output_path]}. Try using `--format`."
+      end
+
+      file_name = @options[:file_name] || formatter.default_file_name
+      if @options[:create_folders]
+        @twine_file.language_codes.each do |lang|
+          output_path = File.join(@options[:output_path], formatter.output_path_for_language(lang))
+
+          FileUtils.mkdir_p(output_path)
+
+          file_path = File.join(output_path, file_name)
+
+          output = formatter.format_file(lang)
+          unless output
+            Twine::stdout.puts "Skipping file at path #{file_path} since it would not contain any translations."
+            next
+          end
+
+          IO.write(file_path, output, encoding: output_encoding)
+        end
+      else
+        language_found = false
+        Dir.foreach(@options[:output_path]) do |item|
+          next if item == "." or item == ".."
+
+          output_path = File.join(@options[:output_path], item)
+          next unless File.directory?(output_path)
+
+          lang = formatter.determine_language_given_path(output_path)
+          next unless lang
+
+          language_found = true
+
+          file_path = File.join(output_path, file_name)
+          output = formatter.format_file(lang)
+          unless output
+            Twine::stdout.puts "Skipping file at path #{file_path} since it would not contain any translations."
+            next
+          end
+
+          IO.write(file_path, output, encoding: output_encoding)
+        end
+
+        unless language_found
+          raise Twine::Error.new("Failed to generate any files: No languages found at #{@options[:output_path]}")
+        end
+      end
+
     end
 
-    def generate_loc_drop
-      begin
-        require 'zip/zip'
-      rescue LoadError
-        raise Twine::Error.new "You must run 'gem install rubyzip' in order to create or consume localization drops."
-      end
+    def generate_localization_archive
+      validate_twine_file if @options[:validate]
+      
+      require_rubyzip
 
       if File.file?(@options[:output_path])
         File.delete(@options[:output_path])
       end
 
-      Dir.mktmpdir do |dir|
-        Zip::ZipFile.open(@options[:output_path], Zip::ZipFile::CREATE) do |zipfile|
+      Dir.mktmpdir do |temp_dir|
+        Zip::File.open(@options[:output_path], Zip::File::CREATE) do |zipfile|
           zipfile.mkdir('Locales')
 
           formatter = formatter_for_format(@options[:format])
-          @strings.language_codes.each do |lang|
+          @twine_file.language_codes.each do |lang|
             if @options[:languages] == nil || @options[:languages].length == 0 || @options[:languages].include?(lang)
-              file_name = lang + formatter.class::EXTENSION
-              real_path = File.join(dir, file_name)
+              file_name = lang + formatter.extension
+              temp_path = File.join(temp_dir, file_name)
               zip_path = File.join('Locales', file_name)
-              formatter.write_file(real_path, lang)
-              zipfile.add(zip_path, real_path)
+
+              output = formatter.format_file(lang)
+              unless output
+                Twine::stdout.puts "Skipping file #{file_name} since it would not contain any translations."
+                next
+              end
+              
+              IO.write(temp_path, output, encoding: output_encoding)
+              zipfile.add(zip_path, temp_path)
             end
           end
         end
       end
     end
 
-    def consume_loc_drop
+    def consume_localization_file
+      lang = nil
+      if @options[:languages]
+        lang = @options[:languages][0]
+      end
+
+      read_localization_file(@options[:input_path], lang)
+      output_path = @options[:output_path] || @options[:twine_file]
+      write_twine_data(output_path)
+    end
+
+    def consume_all_localization_files
+      if !File.directory?(@options[:input_path])
+        raise Twine::Error.new("Directory does not exist: #{@options[:input_path]}")
+      end
+
+      Dir.glob(File.join(@options[:input_path], "**/*")) do |item|
+        if File.file?(item)
+          begin
+            read_localization_file(item)
+          rescue Twine::Error => e
+            Twine::stderr.puts "#{e.message}"
+          end
+        end
+      end
+
+      output_path = @options[:output_path] || @options[:twine_file]
+      write_twine_data(output_path)
+    end
+
+    def consume_localization_archive
+      require_rubyzip
+
       if !File.file?(@options[:input_path])
         raise Twine::Error.new("File does not exist: #{@options[:input_path]}")
       end
 
-      begin
-        require 'zip/zip'
-      rescue LoadError
-        raise Twine::Error.new "You must run 'gem install rubyzip' in order to create or consume localization drops."
-      end
-
-      Dir.mktmpdir do |dir|
-        Zip::ZipFile.open(@options[:input_path]) do |zipfile|
+      error_encountered = false
+      Dir.mktmpdir do |temp_dir|
+        Zip::File.open(@options[:input_path]) do |zipfile|
           zipfile.each do |entry|
-            if !entry.name.end_with?'/' and !File.basename(entry.name).start_with?'.'
-              real_path = File.join(dir, entry.name)
-              FileUtils.mkdir_p(File.dirname(real_path))
-              zipfile.extract(entry.name, real_path)
-              begin
-                read_write_string_file(real_path, true, nil)
-              rescue Twine::Error => e
-                STDERR.puts "#{e.message}"
-              end
+            next if entry.name.end_with? '/' or File.basename(entry.name).start_with? '.'
+
+            real_path = File.join(temp_dir, entry.name)
+            FileUtils.mkdir_p(File.dirname(real_path))
+            zipfile.extract(entry.name, real_path)
+            begin
+              read_localization_file(real_path)
+            rescue Twine::Error => e
+              Twine::stderr.puts "#{e.message}"
+              error_encountered = true
             end
           end
         end
       end
 
-      output_path = @options[:output_path] || @options[:strings_file]
-      write_strings_data(output_path)
+      output_path = @options[:output_path] || @options[:twine_file]
+      write_twine_data(output_path)
+
+      if error_encountered
+        raise Twine::Error.new("At least one file could not be consumed")
+      end
     end
 
-    def generate_report
-      total_strings = 0
-      strings_per_lang = {}
+    def validate_twine_file
+      total_definitions = 0
       all_keys = Set.new
       duplicate_keys = Set.new
       keys_without_tags = Set.new
-      @strings.language_codes.each do |code|
-        strings_per_lang[code] = 0
-      end
+      invalid_keys = Set.new
+      keys_with_python_only_placeholders = Set.new
+      valid_key_regex = /^[A-Za-z0-9_]+$/
 
-      @strings.sections.each do |section|
-        section.rows.each do |row|
-          total_strings += 1
+      @twine_file.sections.each do |section|
+        section.definitions.each do |definition|
+          total_definitions += 1
 
-          if all_keys.include? row.key
-            duplicate_keys.add(row.key)
-          else
-            all_keys.add(row.key)
-          end
+          duplicate_keys.add(definition.key) if all_keys.include? definition.key
+          all_keys.add(definition.key)
 
-          row.translations.each_key do |code|
-            strings_per_lang[code] += 1
-          end
+          keys_without_tags.add(definition.key) if definition.tags == nil or definition.tags.length == 0
 
-          if row.tags == nil || row.tags.length == 0
-            keys_without_tags.add(row.key)
-          end
+          invalid_keys << definition.key unless definition.key =~ valid_key_regex
+
+          keys_with_python_only_placeholders << definition.key if definition.translations.values.any? { |v| Placeholders.contains_python_specific_placeholder(v) }
         end
       end
 
-      # Print the report.
-      puts "Total number of strings = #{total_strings}"
-      @strings.language_codes.each do |code|
-        puts "#{code}: #{strings_per_lang[code]}"
+      errors = []
+      join_keys = lambda { |set| set.map { |k| "  " + k }.join("\n") }
+
+      unless duplicate_keys.empty?
+        errors << "Found duplicate key(s):\n#{join_keys.call(duplicate_keys)}"
       end
 
-      if duplicate_keys.length > 0
-        puts "\nDuplicate string keys:"
-        duplicate_keys.each do |key|
-          puts key
+      if @options[:pedantic]
+        if keys_without_tags.length == total_definitions
+          errors << "None of your definitions have tags."
+        elsif keys_without_tags.length > 0
+          errors << "Found definitions without tags:\n#{join_keys.call(keys_without_tags)}"
         end
       end
 
-      if keys_without_tags.length == total_strings
-        puts "\nNone of your strings have tags."
-      elsif keys_without_tags.length > 0
-        puts "\nStrings without tags:"
-        keys_without_tags.each do |key|
-          puts key
-        end
+      unless invalid_keys.empty?
+        errors << "Found key(s) with invalid characters:\n#{join_keys.call(invalid_keys)}"
       end
+
+      unless keys_with_python_only_placeholders.empty?
+        errors << "Found key(s) with placeholders that are only supported by Python:\n#{join_keys.call(keys_with_python_only_placeholders)}"
+      end
+
+      raise Twine::Error.new errors.join("\n\n") unless errors.empty?
+
+      Twine::stdout.puts "#{@options[:twine_file]} is valid."
     end
 
-    def determine_language_given_path(path)
-      code = File.basename(path, File.extname(path))
-      if !@strings.language_codes.include? code
-        code = nil
-      end
+    private
 
-      code
+    def output_encoding
+      @options[:encoding] || 'UTF-8'
     end
 
-    def determine_format_given_path(path)
-      ext = File.extname(path)
-      Formatters.formatters.each do |formatter|
-        if formatter::EXTENSION == ext
-          return formatter::FORMAT_NAME
-        end
+    def require_rubyzip
+      begin
+        require 'zip'
+      rescue LoadError
+        raise Twine::Error.new "You must run 'gem install rubyzip' in order to create or consume localization archives."
       end
-
-      return
-    end
-
-    def determine_format_given_directory(directory)
-      Formatters.formatters.each do |formatter|
-        if formatter.can_handle_directory?(directory)
-          return formatter::FORMAT_NAME
-        end
-      end
-
-      return
     end
 
     def formatter_for_format(format)
-      Formatters.formatters.each do |formatter|
-        if formatter::FORMAT_NAME == format
-          return formatter.new(@strings, @options)
-        end
+      find_formatter { |f| f.format_name == format }
+    end
+
+    def find_formatter(&block)
+      formatters = Formatters.formatters.select(&block)
+      if formatters.empty?
+        return nil
+      elsif formatters.size > 1
+        raise Twine::Error.new("Unable to determine format. Candidates are: #{formatters.map(&:format_name).join(', ')}. Please specify the format you want using `--format`")
+      end
+      formatter = formatters.first
+      formatter.twine_file = @twine_file
+      formatter.options = @options
+      formatter
+    end
+
+    def read_localization_file(path, lang = nil)
+      unless File.file?(path)
+        raise Twine::Error.new("File does not exist: #{path}")
       end
 
-      return
+      formatter, lang = prepare_read_write(path, lang)
+
+      external_encoding = @options[:encoding] || Twine::Encoding.encoding_for_path(path)
+
+      IO.open(IO.sysopen(path, 'rb'), 'rb', external_encoding: external_encoding, internal_encoding: 'UTF-8') do |io|
+        io.read(2) if Twine::Encoding.has_bom?(path)
+        formatter.read(io, lang)
+      end
+    end
+
+    def prepare_read_write(path, lang)
+      if @options[:format]
+        formatter = formatter_for_format(@options[:format])
+      else
+        formatter = find_formatter { |f| f.extension == File.extname(path) }
+      end
+      
+      unless formatter
+        raise Twine::Error.new "Unable to determine format of #{path}. Try using `--format`."
+      end      
+
+      lang = lang || formatter.determine_language_given_path(path)
+      unless lang
+        raise Twine::Error.new "Unable to determine language for #{path}. Try using `--lang`."
+      end
+
+      @twine_file.language_codes << lang unless @twine_file.language_codes.include? lang
+
+      return formatter, lang
     end
   end
 end

lib/twine/stringsfile.rb

@@ -1,201 +0,0 @@
-module Twine
-  class StringsSection
-    attr_reader :name
-    attr_reader :rows
-
-    def initialize(name)
-      @name = name
-      @rows = []
-    end
-  end
-
-  class StringsRow
-    attr_reader :key
-    attr_accessor :comment
-    attr_accessor :tags
-    attr_reader :translations
-
-    def initialize(key)
-      @key = key
-      @comment = nil
-      @tags = nil
-      @translations = {}
-    end
-
-    def matches_tags?(tags, include_untagged)
-      if tags == nil || tags.length == 0
-        # The user did not specify any tags. Everything passes.
-        return true
-      elsif @tags == nil || @tags.length == 0
-        # This row has no tags.
-        return (include_untagged) ? true : false
-      else
-        tags.each do |tag|
-          if @tags.include? tag
-            return true
-          end
-        end
-      end
-
-      return false
-    end
-
-    def translated_string_for_lang(lang, default_lang=nil)
-      if @translations[lang]
-        return @translations[lang]
-      elsif default_lang.respond_to?("each")
-        default_lang.each do |def_lang|
-          if @translations[def_lang]
-            return @translations[def_lang]
-          end
-        end
-        return nil
-      else
-        return @translations[default_lang]
-      end
-    end
-  end
-
-  class StringsFile
-    attr_reader :sections
-    attr_reader :strings_map
-    attr_reader :language_codes
-
-    def initialize
-      @sections = []
-      @strings_map = {}
-      @language_codes = []
-    end
-
-    def read(path)
-      if !File.file?(path)
-        raise Twine::Error.new("File does not exist: #{path}")
-      end
-
-      File.open(path, 'r:UTF-8') do |f|
-        line_num = 0
-        current_section = nil
-        current_row = nil
-        while line = f.gets
-          parsed = false
-          line.strip!
-          line_num += 1
-
-          if line.length == 0
-            next
-          end
-
-          if line.length > 4 && line[0, 2] == '[['
-            match = /^\[\[(.+)\]\]$/.match(line)
-            if match
-              current_section = StringsSection.new(match[1])
-              @sections << current_section
-              parsed = true
-            end
-          elsif line.length > 2 && line[0, 1] == '['
-            match = /^\[(.+)\]$/.match(line)
-            if match
-              current_row = StringsRow.new(match[1])
-              @strings_map[current_row.key] = current_row
-              if !current_section
-                current_section = StringsSection.new('')
-                @sections << current_section
-              end
-              current_section.rows << current_row
-              parsed = true
-            end
-          else
-            match = /^([^=]+)=(.*)$/.match(line)
-            if match
-              key = match[1].strip
-              value = match[2].strip
-              if value[0,1] == '`' && value[-1,1] == '`'
-                value = value[1..-2]
-              end
-
-              case key
-              when "comment"
-                current_row.comment = value
-              when 'tags'
-                current_row.tags = value.split(',')
-              else
-                if !@language_codes.include? key
-                  add_language_code(key)
-                end
-                current_row.translations[key] = value
-              end
-              parsed = true
-            end
-          end
-
-          if !parsed
-            raise Twine::Error.new("Unable to parse line #{line_num} of #{path}: #{line}")
-          end
-        end
-      end
-    end
-
-    def write(path)
-      dev_lang = @language_codes[0]
-
-      File.open(path, 'w:UTF-8') do |f|
-        @sections.each do |section|
-          if f.pos > 0
-            f.puts ''
-          end
-
-          f.puts "[[#{section.name}]]"
-
-          section.rows.each do |row|
-            f.puts "\t[#{row.key}]"
-            value = row.translations[dev_lang]
-            if !value
-              puts "Warning: #{row.key} does not exist in developer language '#{dev_lang}'"
-            else
-              if value[0,1] == ' ' || value[-1,1] == ' ' || (value[0,1] == '`' && value[-1,1] == '`')
-                value = '`' + value + '`'
-              end
-              f.puts "\t\t#{dev_lang} = #{value}"
-            end
-
-            if row.tags && row.tags.length > 0
-              tag_str = row.tags.join(',')
-              f.puts "\t\ttags = #{tag_str}"
-            end
-            if row.comment && row.comment.length > 0
-              f.puts "\t\tcomment = #{row.comment}"
-            end
-            @language_codes[1..-1].each do |lang|
-              value = row.translations[lang]
-              if value
-                if value[0,1] == ' ' || value[-1,1] == ' ' || (value[0,1] == '`' && value[-1,1] == '`')
-                  value = '`' + value + '`'
-                end
-                f.puts "\t\t#{lang} = #{value}"
-              end
-            end
-          end
-        end
-      end
-    end
-
-    def add_language_code(code)
-      if @language_codes.length == 0
-        @language_codes << code
-      elsif !@language_codes.include?(code)
-        dev_lang = @language_codes[0]
-        @language_codes << code
-        @language_codes.delete(dev_lang)
-        @language_codes.sort!
-        @language_codes.insert(0, dev_lang)
-      end
-    end
-
-    def set_developer_language_code(code)
-      if @language_codes.include?(code)
-        @language_codes.delete(code)
-      end
-      @language_codes.insert(0, code)
-    end
-  end
-end

lib/twine/twine_file.rb

@@ -0,0 +1,256 @@
+module Twine
+  class TwineDefinition
+    PLURAL_KEYS = %w(zero one two few many other)
+
+    attr_reader :key
+    attr_accessor :comment
+    attr_accessor :tags
+    attr_reader :translations
+    attr_reader :plural_translations
+    attr_reader :is_plural
+    attr_accessor :reference
+    attr_accessor :reference_key
+
+    def initialize(key)
+      @key = key
+      @comment = nil
+      @tags = nil
+      @translations = {}
+      @plural_translations = {}
+    end
+
+    def comment
+      raw_comment || (reference.comment if reference)
+    end
+
+    def raw_comment
+      @comment
+    end
+
+    # [['tag1', 'tag2'], ['~tag3']] == (tag1 OR tag2) AND (!tag3)
+    def matches_tags?(tags, include_untagged)
+      if tags == nil || tags.empty?  # The user did not specify any tags. Everything passes.
+        return true
+      elsif @tags == nil  # This definition has no tags -> check reference (if any)
+        return reference ? reference.matches_tags?(tags, include_untagged) : include_untagged
+      elsif @tags.empty?
+        return include_untagged
+      else
+        return tags.all? do |set|
+          regular_tags, negated_tags = set.partition { |tag| tag[0] != '~' }
+          negated_tags.map! { |tag| tag[1..-1] }
+          matches_regular_tags = (!regular_tags.empty? && !(regular_tags & @tags).empty?)
+          matches_negated_tags = (!negated_tags.empty? && (negated_tags & @tags).empty?)
+          matches_regular_tags or matches_negated_tags
+        end
+      end
+
+      return false
+    end
+
+    def translation_for_lang(lang)
+      translation = [lang].flatten.map { |l| @translations[l] }.compact.first
+
+      translation = reference.translation_for_lang(lang) if translation.nil? && reference
+
+      return translation
+    end
+
+    def plural_translation_for_lang(lang)
+      if @plural_translations.has_key? lang
+        @plural_translations[lang].dup.sort_by { |key,_| TwineDefinition::PLURAL_KEYS.index(key) }.to_h
+      end
+    end
+
+    def is_plural?
+      !@plural_translations.empty?
+    end
+  end
+
+  class TwineSection
+    attr_reader :name
+    attr_reader :definitions
+
+    def initialize(name)
+      @name = name
+      @definitions = []
+    end
+  end
+
+  class TwineFile
+    attr_reader :sections
+    attr_reader :definitions_by_key
+    attr_reader :language_codes
+
+    private
+
+    def match_key(text)
+      match = /^\[(.+)\]$/.match(text)
+      return match[1] if match
+    end
+
+    public
+
+    def initialize
+      @sections = []
+      @definitions_by_key = {}
+      @language_codes = []
+    end
+
+    def add_language_code(code)
+      if @language_codes.length == 0
+        @language_codes << code
+      elsif !@language_codes.include?(code)
+        dev_lang = @language_codes[0]
+        @language_codes << code
+        @language_codes.delete(dev_lang)
+        @language_codes.sort!
+        @language_codes.insert(0, dev_lang)
+      end
+    end
+
+    def set_developer_language_code(code)
+      @language_codes.delete(code)
+      @language_codes.insert(0, code)
+    end
+
+    def read(path)
+      if !File.file?(path)
+        raise Twine::Error.new("File does not exist: #{path}")
+      end
+
+      File.open(path, 'r:UTF-8') do |f|
+        line_num = 0
+        current_section = nil
+        current_definition = nil
+        while line = f.gets
+          parsed = false
+          line.strip!
+          line_num += 1
+
+          if line.length == 0
+            next
+          end
+
+          if line.length > 4 && line[0, 2] == '[['
+            match = /^\[\[(.+)\]\]$/.match(line)
+            if match
+              current_section = TwineSection.new(match[1])
+              @sections << current_section
+              parsed = true
+            end
+          elsif line.length > 2 && line[0, 1] == '['
+            key = match_key(line)
+            if key
+              current_definition = TwineDefinition.new(key)
+              @definitions_by_key[current_definition.key] = current_definition
+              if !current_section
+                current_section = TwineSection.new('')
+                @sections << current_section
+              end
+              current_section.definitions << current_definition
+              parsed = true
+            end
+          else
+            match = /^([^:=]+)(?::([^=]+))?=(.*)$/.match(line)
+            if match
+              key = match[1].strip
+              plural_key = match[2].to_s.strip
+              value = match[3].strip
+
+              value = value[1..-2] if value[0] == '`' && value[-1] == '`'
+
+              case key
+              when 'comment'
+                current_definition.comment = value
+              when 'tags'
+                current_definition.tags = value.split(',')
+              when 'ref'
+                current_definition.reference_key = value if value
+              else
+                if !@language_codes.include? key
+                  add_language_code(key)
+                end
+                # Providing backward compatibility
+                # for formatters without plural support
+                if plural_key.empty? || plural_key == 'other'
+                  current_definition.translations[key] = value
+                end
+                if !plural_key.empty?
+                  if !TwineDefinition::PLURAL_KEYS.include? plural_key
+                    warn("Unknown plural key #{plural_key}")
+                    next
+                  end
+                  (current_definition.plural_translations[key] ||= {})[plural_key] = value
+                end
+              end
+              parsed = true
+            end
+          end
+
+          if !parsed
+            raise Twine::Error.new("Unable to parse line #{line_num} of #{path}: #{line}")
+          end
+        end
+      end
+
+      # resolve_references
+      @definitions_by_key.each do |key, definition|
+        next unless definition.reference_key
+        definition.reference = @definitions_by_key[definition.reference_key]
+      end
+    end
+
+    def write(path)
+      dev_lang = @language_codes[0]
+
+      File.open(path, 'w:UTF-8') do |f|
+        @sections.each do |section|
+          if f.pos > 0
+            f.puts ''
+          end
+
+          f.puts "[[#{section.name}]]"
+
+          section.definitions.each do |definition|
+            f.puts "\t[#{definition.key}]"
+
+            value = write_value(definition, dev_lang, f)
+            if !value && !definition.reference_key
+              Twine::stdout.puts "WARNING: #{definition.key} does not exist in developer language '#{dev_lang}'"
+            end
+            
+            if definition.reference_key
+              f.puts "\t\tref = #{definition.reference_key}"
+            end
+            if definition.tags && definition.tags.length > 0
+              tag_str = definition.tags.join(',')
+              f.puts "\t\ttags = #{tag_str}"
+            end
+            if definition.raw_comment and definition.raw_comment.length > 0
+              f.puts "\t\tcomment = #{definition.raw_comment}"
+            end
+            @language_codes[1..-1].each do |lang|
+              write_value(definition, lang, f)
+            end
+          end
+        end
+      end
+    end
+
+    private
+
+    def write_value(definition, language, file)
+      value = definition.translations[language]
+      return nil unless value
+
+      if value[0] == ' ' || value[-1] == ' ' || (value[0] == '`' && value[-1] == '`')
+        value = '`' + value + '`'
+      end
+
+      file.puts "\t\t#{language} = #{value}"
+      return value
+    end
+
+  end
+end

lib/twine/version.rb

@@ -1,3 +1,3 @@
 module Twine
-  VERSION = '0.6.0'
+  VERSION = '1.1.2-om'
 end

test/command_test.rb

@@ -0,0 +1,14 @@
+require 'twine_test'
+
+class CommandTest < TwineTest
+  def prepare_mock_formatter(formatter_class, clear_other_formatters = true)
+    twine_file = Twine::TwineFile.new
+    twine_file.language_codes.concat KNOWN_LANGUAGES
+
+    formatter = formatter_class.new
+    formatter.twine_file = twine_file
+    Twine::Formatters.formatters.clear if clear_other_formatters
+    Twine::Formatters.formatters << formatter
+    formatter
+  end
+end

test/fixtures/en-1.json

@@ -1,5 +0,0 @@
-{
-"key1":"key1-english",
-"key3":"key3-english",
-"key5":"A new string"
-}

test/fixtures/en-1.po

@@ -1,16 +0,0 @@
-msgid ""
-msgstr ""
-"Language: en\n"
-"X-Generator: Twine\n"
-
-msgctxt "key1"
-msgid "key1-english"
-msgstr "key1-english"
-
-msgctxt "key3"
-msgid "key3-english"
-msgstr ""
-
-msgctxt "key5"
-msgid "A new string"
-msgstr "A new string"

test/fixtures/en-1.strings

@@ -1,10 +0,0 @@
-/**
- * iOS Strings File
- * Generated by Twine
- * Language: en
- */
-
-/* My Strings */
-"key1" = "key1-english";
-"key3" = "key3-english";
-"key5" = "A new string";

test/fixtures/en-2.po

@@ -1,23 +0,0 @@
-msgid ""
-msgstr ""
-"Language: en\n"
-"X-Generator: Twine\n"
-
-msgctxt "key1"
-msgid "key1-english"
-msgstr "key1-english"
-
-msgctxt "key3"
-msgid "key3-english"
-msgstr ""
-
-msgctxt "key4"
-msgid "key4"
-"multiline"
-msgstr "A multi"
-"line string\n"
-"can occur"
-
-msgctxt "key5"
-msgid "A new string"
-msgstr "A new string"

test/fixtures/en-3.xml

@@ -1,8 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<!-- Android Strings File -->
-<!-- Generated by Twine 0.5.0 -->
-<!-- Language: en -->
-<resources>
-	<!-- SECTION: My Strings -->
-	<string name="string_with_spaces">\u0020string with spaces\u0020\u0020</string>
-</resources>

test/fixtures/enc_utf8.dummy

@@ -0,0 +1,2 @@
+Üß`
+da

test/fixtures/formatter_android.xml

@@ -0,0 +1,12 @@
+<?xml version="1.0" encoding="utf-8"?>
+<resources>
+    <!-- SECTION: Section 1 -->
+    <!-- comment key1 -->
+    <string name="key1">value1-english</string>
+    <string name="key2">value2-english</string>
+
+    <!-- SECTION: Section 2 -->
+    <string name="key3">value3-english</string>
+    <!-- comment key4 -->
+    <string name="key4">value4-english</string>
+</resources>

test/fixtures/formatter_apple.strings

@@ -0,0 +1,14 @@
+/********** Section 1 **********/
+
+/* comment key1 */
+"key1" = "value1-english";
+
+"key2" = "value2-english";
+
+
+/********** Section 2 **********/
+
+"key3" = "value3-english";
+
+/* comment key4 */
+"key4" = "value4-english";

test/fixtures/formatter_django.po

@@ -0,0 +1,29 @@
+# Django Strings File
+# Generated by Twine
+# Language: en
+msgid ""
+msgstr ""
+"Content-Type: text/plain; charset=UTF-8\n"
+
+# --------- Section 1 --------- #
+
+#. comment key1
+# base translation: "value1-english"
+msgid "key1"
+msgstr "value1-english"
+
+# base translation: "value2-english"
+msgid "key2"
+msgstr "value2-english"
+
+
+# --------- Section 2 --------- #
+
+# base translation: "value3-english"
+msgid "key3"
+msgstr "value3-english"
+
+#. comment key4
+# base translation: "value4-english"
+msgid "key4"
+msgstr "value4-english"

test/fixtures/formatter_flash.properties

@@ -0,0 +1,15 @@
+## Flash Strings File
+## Generated by Twine
+## Language: en
+
+## Section 1 ##
+
+# comment key1
+key1=value1-english
+key2=value2-english
+
+## Section 2 ##
+
+key3=value3-english
+# comment key4
+key4=value4-english

test/fixtures/formatter_gettext.po

@@ -0,0 +1,26 @@
+msgid ""
+msgstr ""
+"Language: en"
+"X-Generator: Twine <%= Twine::VERSION %>"
+
+
+# SECTION: Section 1
+#. "comment key1"
+msgctxt "key1"
+msgid "value1-english"
+msgstr "value1-english"
+
+msgctxt "key2"
+msgid "value2-english"
+msgstr "value2-english"
+
+
+# SECTION: Section 2
+msgctxt "key3"
+msgid "value3-english"
+msgstr "value3-english"
+
+#. "comment key4"
+msgctxt "key4"
+msgid "value4-english"
+msgstr "value4-english"

test/fixtures/formatter_gettext_quotes.po

@@ -0,0 +1,10 @@
+msgid ""
+msgstr ""
+"Language: en"
+"X-Generator: Twine <%= Twine::VERSION %>"
+
+
+# SECTION: Section
+msgctxt "key"
+msgid "foo \"bar\" baz"
+msgstr "foo \"bar\" baz"

test/fixtures/formatter_jquery.json

@@ -0,0 +1,7 @@
+{
+"key1":"value1-english",
+"key2":"value2-english",
+
+"key3":"value3-english",
+"key4":"value4-english"
+}

test/fixtures/formatter_tizen.xml

@@ -0,0 +1,15 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Tizen Strings File -->
+<!-- Generated by Twine -->
+<!-- Language: en -->
+<string_table  Bversion="2.0.0.201311071819" Dversion="20120315">
+	<!-- SECTION: Section 1 -->
+	<!-- comment key1 -->
+	<text id="IDS_KEY1">value1-english</text>
+	<text id="IDS_KEY2">value2-english</text>
+
+	<!-- SECTION: Section 2 -->
+	<text id="IDS_KEY3">value3-english</text>
+	<!-- comment key4 -->
+	<text id="IDS_KEY4">value4-english</text>
+</string_table>

test/fixtures/fr-1.xml

@@ -1,10 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<!-- Android Strings File -->
-<!-- Generated by Twine -->
-<!-- Language: fr -->
-<resources>
-	<!-- This is a comment -->
-	<string name="key1">key1-french</string>
-	<string name="key2">key2-french</string>
-	<string name="key3">key3-french</string>
-</resources>

test/fixtures/gettext_multiline.po

@@ -0,0 +1,10 @@
+msgid ""
+msgstr ""
+"Language: en\n"
+"X-Generator: Twine\n"
+
+msgctxt "key1"
+msgid "key1"
+msgstr "multi"
+"line\n"
+"string"

test/fixtures/strings-1.txt

@@ -1,17 +0,0 @@
-[[My Strings]]
-	[key1]
-		en = key1-english
-		tags = tag1
-		comment = This is a comment
-		es = key1-spanish
-		fr = key1-french
-	[key2]
-		en = key2-english
-		tags = tag2
-		fr = key2-french
-	[key3]
-		en = key3-english
-		tags = tag1,tag2
-		es = key3-spanish
-	[key4]
-		en = key4-english

test/fixtures/strings-2.txt

@@ -1,5 +0,0 @@
-[[My Strings]]
-	[key with space ]
-		en = `string with space `
-		tags = tag1
-		comment = String ends with space

test/fixtures/strings-3.txt

@@ -1,5 +0,0 @@
-[[My Strings]]
-    [parameterized_string]
-        en = The %@ brown fox jumps over the %@ dog %d times.
-    [percentage_string]
-        en = This product is %d%% off.

test/fixtures/test-json-line-breaks/consumed.txt

@@ -1,5 +0,0 @@
-[[Line Break Strings]]
-	[line_breaking]
-		en = This\nstring\ncontains\nline\nbreaks
-		tags = tag1
-		fr = This\nstring\nalso\ncontains\nline\nbreaks

test/fixtures/test-json-line-breaks/generated.json

@@ -1,3 +0,0 @@
-{
-"line_breaking":"This\nstring\ncontains\nline\nbreaks"
-}

test/fixtures/test-json-line-breaks/line-breaks.json

@@ -1,3 +0,0 @@
-{
-"line_breaking":"This\nstring\nalso\ncontains\nline\nbreaks"
-}

test/fixtures/test-json-line-breaks/line-breaks.txt

@@ -1,4 +0,0 @@
-[[Line Break Strings]]
-    [line_breaking]
-        en = This\nstring\ncontains\nline\nbreaks
-        tags = tag1

test/fixtures/test-output-1.txt

@@ -1,12 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<!-- Android Strings File -->
-<!-- Generated by Twine <%= Twine::VERSION %> -->
-<!-- Language: fr -->
-<resources>
-	<!-- SECTION: My Strings -->
-	<!-- This is a comment -->
-	<string name="key1">key1-french</string>
-	<string name="key2">key2-french</string>
-	<string name="key3">key3-english</string>
-	<string name="key4">key4-english</string>
-</resources>

test/fixtures/test-output-10.txt

@@ -1,9 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<!-- Android Strings File -->
-<!-- Generated by Twine 0.5.0 -->
-<!-- Language: en -->
-<resources>
-	<!-- SECTION: My Strings -->
-	<!-- String ends with space -->
-	<string name="key with space ">string with space\u0020</string>
-</resources>

test/fixtures/test-output-11.txt

@@ -1,9 +0,0 @@
-[[Uncategorized]]
-	[string_with_spaces]
-		en = ` string with spaces  `
-
-[[My Strings]]
-	[key with space ]
-		en = `string with space `
-		tags = tag1
-		comment = String ends with space

test/fixtures/test-output-12.txt

@@ -1,12 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<!-- Tizen Strings File -->
-<!-- Generated by Twine <%= Twine::VERSION %> -->
-<!-- Language: fr -->
-<string_table  Bversion="2.0.0.201311071819" Dversion="20120315">
-	<!-- SECTION: My Strings -->
-	<!-- This is a comment -->
-	<text id="IDS_KEY1">key1-french</text>
-	<text id="IDS_KEY2">key2-french</text>
-	<text id="IDS_KEY3">key3-english</text>
-	<text id="IDS_KEY4">key4-english</text>
-</string_table>

test/fixtures/test-output-2.txt

@@ -1,12 +0,0 @@
-/**
- * Apple Strings File
- * Generated by Twine <%= Twine::VERSION %>
- * Language: en
- */
-
-/********** My Strings **********/
-
-/* This is a comment */
-"key1" = "key1-english";
-
-"key3" = "key3-english";

test/fixtures/test-output-3.txt

@@ -1,18 +0,0 @@
-[[My Strings]]
-	[key1]
-		en = key1-english
-		tags = tag1
-		comment = This is a comment
-		es = key1-spanish
-		fr = key1-french
-	[key2]
-		en = key2-english
-		tags = tag2
-		fr = key2-french
-	[key3]
-		en = key3-english
-		tags = tag1,tag2
-		es = key3-spanish
-		fr = key3-french
-	[key4]
-		en = key4-english

test/fixtures/test-output-4.txt

@@ -1,21 +0,0 @@
-[[Uncategorized]]
-	[key5]
-		en = A new string
-
-[[My Strings]]
-	[key1]
-		en = key1-english
-		tags = tag1
-		comment = This is a comment
-		es = key1-spanish
-		fr = key1-french
-	[key2]
-		en = key2-english
-		tags = tag2
-		fr = key2-french
-	[key3]
-		en = key3-english
-		tags = tag1,tag2
-		es = key3-spanish
-	[key4]
-		en = key4-english

test/fixtures/test-output-5.txt

@@ -1,4 +0,0 @@
-{
-"key1":"key1-english",
-"key3":"key3-english"
-}

test/fixtures/test-output-6.txt

@@ -1,10 +0,0 @@
-/**
- * Apple Strings File
- * Generated by Twine <%= Twine::VERSION %>
- * Language: en
- */
-
-/********** My Strings **********/
-
-/* String ends with space */
-"key with space " = "string with space ";

test/fixtures/test-output-7.txt

@@ -1,16 +0,0 @@
-msgid ""
-msgstr ""
-"Language: en\n"
-"X-Generator: Twine <%= Twine::VERSION %>\n"
-
-
-# SECTION: My Strings
-#. "This is a comment"
-msgctxt "key1"
-msgid "key1-english"
-msgstr "key1-english"
-
-msgctxt "key3"
-msgid "key3-english"
-msgstr "key3-english"
-

test/fixtures/test-output-8.txt

@@ -1,9 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<!-- Android Strings File -->
-<!-- Generated by Twine <%= Twine::VERSION %> -->
-<!-- Language: en -->
-<resources>
-	<!-- SECTION: My Strings -->
-	<string name="parameterized_string">The %1$s brown fox jumps over the %2$s dog %3$d times.</string>
-	<string name="percentage_string">This product is %d%% off.</string>
-</resources>

test/fixtures/test-output-9.txt

@@ -1,21 +0,0 @@
-[[Uncategorized]]
-	[key5]
-		en = A new string
-
-[[My Strings]]
-	[key1]
-		en = key1-english
-		tags = tag1
-		comment = This is a comment
-		es = key1-spanish
-		fr = key1-french
-	[key2]
-		en = key2-english
-		tags = tag2
-		fr = key2-french
-	[key3]
-		en = key3-english
-		tags = tag1,tag2
-		es = key3-spanish
-	[key4]
-		en = A multiline string\ncan occur

test/fixtures/twine_accent_values.txt

@@ -0,0 +1,13 @@
+[[Section]]
+	[value_with_leading_accent]
+		en = `value
+	[value_with_trailing_accent]
+		en = value`
+	[value_with_leading_space]
+		en = ` value`
+	[value_with_trailing_space]
+		en = `value `
+	[value_wrapped_by_spaces]
+		en = ` value `
+	[value_wrapped_by_accents]
+		en = ``value``

test/test_abstract_formatter.rb

@@ -0,0 +1,165 @@
+require 'twine_test'
+
+class TestAbstractFormatter < TwineTest
+  class SetTranslation < TwineTest
+    def setup
+      super
+
+      @twine_file = build_twine_file 'en', 'fr' do
+        add_section 'Section' do
+          add_definition key1: 'value1-english'
+          add_definition key2: { en: 'value2-english', fr: 'value2-french' }
+        end
+      end
+
+      @formatter = Twine::Formatters::Abstract.new
+      @formatter.twine_file = @twine_file
+    end
+
+    def test_set_translation_updates_existing_value
+      @formatter.set_translation_for_key 'key1', 'en', 'value1-english-updated'
+
+      assert_equal 'value1-english-updated', @twine_file.definitions_by_key['key1'].translations['en']
+    end
+
+    def test_set_translation_does_not_alter_other_language
+      @formatter.set_translation_for_key 'key2', 'en', 'value2-english-updated'
+
+      assert_equal 'value2-french', @twine_file.definitions_by_key['key2'].translations['fr']
+    end
+
+    def test_set_translation_escapes_newlines
+      @formatter.set_translation_for_key 'key1', 'en', "new\nline"
+
+      assert_equal 'new\nline', @twine_file.definitions_by_key['key1'].translations['en']
+    end
+
+    def test_set_translation_adds_translation_to_existing_key
+      @formatter.set_translation_for_key 'key1', 'fr', 'value1-french'
+
+      assert_equal 'value1-french', @twine_file.definitions_by_key['key1'].translations['fr']
+    end
+
+    def test_set_translation_does_not_add_new_key
+      @formatter.set_translation_for_key 'new-key', 'en', 'new-key-english'
+
+      assert_nil @twine_file.definitions_by_key['new-key']
+    end
+
+    def test_set_translation_consume_all_adds_new_key
+      formatter = Twine::Formatters::Abstract.new
+      formatter.twine_file = @twine_file
+      formatter.options = { consume_all: true }
+      formatter.set_translation_for_key 'new-key', 'en', 'new-key-english'
+
+      assert_equal 'new-key-english', @twine_file.definitions_by_key['new-key'].translations['en']
+    end
+
+    def test_set_translation_consume_all_adds_tags
+      random_tag = SecureRandom.uuid
+      formatter = Twine::Formatters::Abstract.new
+      formatter.twine_file = @twine_file
+      formatter.options = { consume_all: true, tags: [random_tag] }
+      formatter.set_translation_for_key 'new-key', 'en', 'new-key-english'
+
+      assert_equal [random_tag], @twine_file.definitions_by_key['new-key'].tags
+    end
+
+    def test_set_translation_adds_new_keys_to_category_uncategoriezed
+      formatter = Twine::Formatters::Abstract.new
+      formatter.twine_file = @twine_file
+      formatter.options = { consume_all: true }
+      formatter.set_translation_for_key 'new-key', 'en', 'new-key-english'
+
+      assert_equal 'Uncategorized', @twine_file.sections[0].name 
+      assert_equal 'new-key', @twine_file.sections[0].definitions[0].key
+    end
+  end
+
+  class ValueReference < TwineTest
+    def setup
+      super
+
+      @twine_file = build_twine_file 'en', 'fr' do
+        add_section 'Section' do
+          add_definition refkey: 'ref-value'
+          add_definition key: :refkey
+        end
+      end
+
+      @formatter = Twine::Formatters::Abstract.new
+      @formatter.twine_file = @twine_file
+    end
+
+    def test_set_translation_does_not_add_unchanged_translation
+      @formatter.set_translation_for_key 'key', 'en', 'ref-value'
+
+      assert_nil @twine_file.definitions_by_key['key'].translations['en']
+    end
+
+    def test_set_translation_adds_changed_translation
+      @formatter.set_translation_for_key 'key', 'en', 'changed value'
+
+      assert_equal 'changed value', @twine_file.definitions_by_key['key'].translations['en']
+    end
+  end
+
+  class SetComment < TwineTest
+    def setup
+      super
+
+      @twine_file = build_twine_file 'en' do
+        add_section 'Section' do
+          add_definition key: 'value'
+        end
+      end
+    end
+
+    def test_set_comment_for_key_does_not_update_comment
+      formatter = Twine::Formatters::Abstract.new
+      formatter.twine_file = @twine_file
+      formatter.set_comment_for_key('key', 'comment')
+
+      assert_nil formatter.twine_file.definitions_by_key['key'].comment
+    end
+
+    def test_set_comment_for_key_updates_comment_with_update_comments
+      formatter = Twine::Formatters::Abstract.new
+      formatter.twine_file = @twine_file
+      formatter.options = { consume_comments: true }
+      formatter.set_comment_for_key('key', 'comment')
+
+      assert_equal 'comment', formatter.twine_file.definitions_by_key['key'].comment
+    end
+  end
+
+  class CommentReference < TwineTest
+    def setup
+      super
+
+      @twine_file = build_twine_file 'en' do
+        add_section 'Section' do
+          add_definition refkey: 'ref-value', comment: 'reference comment'
+          add_definition key: 'value', ref: :refkey
+        end
+      end
+
+      @formatter = Twine::Formatters::Abstract.new
+      @formatter.twine_file = @twine_file
+      @formatter.options = { consume_comments: true }
+    end
+
+    def test_set_comment_does_not_add_unchanged_comment
+      @formatter.set_comment_for_key 'key', 'reference comment'
+      
+      assert_nil @twine_file.definitions_by_key['key'].raw_comment
+    end
+
+    def test_set_comment_adds_changed_comment
+      @formatter.set_comment_for_key 'key', 'changed comment'
+
+      assert_equal 'changed comment', @twine_file.definitions_by_key['key'].raw_comment
+    end
+  end
+
+end

test/test_cli.rb

@@ -0,0 +1,467 @@
+require 'twine_test'
+
+class CLITest < TwineTest
+  def setup
+    super()
+
+    @twine_file_path = File.join @output_dir, SecureRandom.uuid
+    @input_path = File.join @output_dir, SecureRandom.uuid
+    @input_dir = @output_dir
+  end
+
+  def parse(command)
+    @options = Twine::CLI::parse command.split
+  end
+
+  def parse_with(parameters)
+    raise "you need to implement `parse_with` in your test class"
+  end
+
+  def assert_help
+    parse_with '--help'
+    assert_equal @options, false
+    assert_match /Usage: twine.*Examples:/m, Twine::stdout.string
+  end
+
+  def assert_option_consume_all
+    parse_with '--consume-all'
+    assert @options[:consume_all]
+    parse_with '--no-consume-all'
+    refute @options[:consume_all]
+  end
+
+  def assert_option_consume_comments
+    parse_with '--consume-comments'
+    assert @options[:consume_comments]
+    parse_with '--no-consume-comments'
+    refute @options[:consume_comments]
+  end
+
+  def assert_option_developer_language
+    random_language = KNOWN_LANGUAGES.sample
+    parse_with "--developer-language #{random_language}"
+    assert_equal random_language, @options[:developer_language]
+  end
+
+  def assert_option_encoding
+    parse_with '--encoding UTF16'
+    assert_equal 'UTF16', @options[:encoding]
+  end
+
+  def assert_option_escape_all_tags
+    parse_with "--escape-all-tags"
+    assert @options[:escape_all_tags]
+    parse_with "--no-escape-all-tags"
+    refute @options[:escape_all_tags]
+  end
+
+  def assert_option_format
+    random_format = Twine::Formatters.formatters.sample.format_name.downcase
+    parse_with "--format #{random_format}"
+    assert_equal random_format, @options[:format]
+  end
+
+  def assert_option_include
+    random_set = [:all, :translated, :untranslated].sample
+    parse_with "--include #{random_set}"
+    assert_equal random_set, @options[:include]
+  end
+
+  def assert_option_single_language
+    random_language = KNOWN_LANGUAGES.sample
+    parse_with "--lang #{random_language}"
+    assert_equal [random_language], @options[:languages]
+  end
+
+  def assert_option_multiple_languages
+    random_languages = KNOWN_LANGUAGES.shuffle[0, 3]
+    parse_with "--lang #{random_languages.join(',')}"
+    assert_equal random_languages.sort, @options[:languages].sort
+  end
+
+  def assert_option_languages
+    assert_option_single_language
+    assert_option_multiple_languages
+  end
+
+  def assert_option_output_path
+    parse_with "--output-file #{@output_path}"
+    assert_equal @output_path, @options[:output_path]  
+  end
+
+  def assert_option_quiet
+    parse_with '--quiet'
+    assert @options[:quiet]
+    parse_with '--no-quiet'
+    refute @options[:quiet]
+  end
+
+  def assert_option_tags
+    # single tag
+    random_tag = "tag#{rand(100)}"
+    parse_with "--tags #{random_tag}"
+    assert_equal [[random_tag]], @options[:tags]
+
+    # multiple OR tags
+    random_tags = ["tag#{rand(100)}", "tag#{rand(100)}", "tag#{rand(100)}"]
+    parse_with "--tags #{random_tags.join(',')}"
+    sorted_tags = @options[:tags].map { |tags| tags.sort }
+    assert_equal [random_tags.sort], sorted_tags
+
+    # multiple AND tags
+    random_tag_1 = "tag#{rand(100)}"
+    random_tag_2 = "tag#{rand(100)}"
+    parse_with "--tags #{random_tag_1} --tags #{random_tag_2}"
+    assert_equal [[random_tag_1], [random_tag_2]], @options[:tags]
+    
+    # NOT tag
+    random_tag = "~tag#{rand(100)}"
+    parse_with "--tags #{random_tag}"
+    assert_equal [[random_tag]], @options[:tags]
+  end
+
+  def assert_option_untagged
+    parse_with '--untagged'
+    assert @options[:untagged]
+    parse_with '--no-untagged'
+    refute @options[:untagged]
+  end
+
+  def assert_option_validate
+    parse_with "--validate"
+    assert @options[:validate]
+    parse_with "--no-validate"
+    refute @options[:validate]
+  end
+end
+
+class TestCLI < CLITest
+  def test_version
+    parse "--version"
+
+    assert_equal @options, false
+    assert_equal "Twine version #{Twine::VERSION}\n", Twine::stdout.string
+  end
+
+  def test_help
+    parse ""
+    assert_match 'Usage: twine', Twine::stdout.string
+  end
+
+  def test_invalid_command
+    assert_raises Twine::Error do
+      parse "not a command"
+    end
+  end
+end
+
+class TestGenerateLocalizationFileCLI < CLITest
+  def parse_with(parameters)
+    parse "generate-localization-file #{@twine_file_path} #{@output_path} " + parameters
+  end
+
+  def test_command
+    parse_with ""
+
+    assert_equal 'generate-localization-file', @options[:command]
+    assert_equal @twine_file_path, @options[:twine_file]
+    assert_equal @output_path, @options[:output_path]
+  end
+
+  def test_missing_argument
+    assert_raises Twine::Error do
+      parse "generate-localization-file #{@twine_file_path}"
+    end
+  end
+
+  def test_extra_argument
+    assert_raises Twine::Error do
+      parse_with "extra"
+    end
+  end
+
+  def test_options
+    assert_help
+    assert_option_developer_language
+    assert_option_encoding
+    assert_option_escape_all_tags
+    assert_option_format
+    assert_option_include
+    assert_option_single_language
+    assert_raises(Twine::Error) { assert_option_multiple_languages }
+    assert_option_quiet
+    assert_option_tags
+    assert_option_untagged
+    assert_option_validate
+  end
+end
+
+class TestGenerateAllLocalizationFilesCLI < CLITest
+  def parse_with(parameters)
+    parse "generate-all-localization-files #{@twine_file_path} #{@output_dir} " + parameters
+  end
+
+  def test_command
+    parse_with ""
+
+    assert_equal 'generate-all-localization-files', @options[:command]
+    assert_equal @twine_file_path, @options[:twine_file]
+    assert_equal @output_dir, @options[:output_path]
+  end
+
+  def test_missing_argument
+    assert_raises Twine::Error do
+      parse "generate-all-localization-files twine_file"
+    end
+  end
+
+  def test_extra_arguemnt
+    assert_raises Twine::Error do
+      parse_with "extra"
+    end
+  end
+
+  def test_options
+    assert_help
+    assert_option_developer_language
+    assert_option_encoding
+    assert_option_escape_all_tags
+    assert_option_format
+    assert_option_include
+    assert_option_quiet
+    assert_option_tags
+    assert_option_untagged
+    assert_option_validate
+  end
+
+  def test_option_create_folders
+    parse_with '--create-folders'
+    assert @options[:create_folders]
+    parse_with '--no-create-folders'
+    refute @options[:create_folders]
+  end
+
+  def test_option_file_name
+    random_filename = "#{rand(10000)}"
+    parse_with "--file-name #{random_filename}"
+    assert_equal random_filename, @options[:file_name]
+  end
+end
+
+class TestGenerateLocalizationArchiveCLI < CLITest
+  def parse_with(parameters)
+    parse "generate-localization-archive #{@twine_file_path} #{@output_path} --format apple " + parameters
+  end
+
+  def test_command
+    parse_with ""
+
+    assert_equal 'generate-localization-archive', @options[:command]
+    assert_equal @twine_file_path, @options[:twine_file]
+    assert_equal @output_path, @options[:output_path]
+  end
+
+  def test_missing_argument
+    assert_raises Twine::Error do
+      parse "generate-localization-archive twine_file --format apple"
+    end
+  end
+
+  def test_extra_argument
+    assert_raises Twine::Error do
+      parse_with "extra"
+    end
+  end
+
+  def test_options
+    assert_help
+    assert_option_developer_language
+    assert_option_encoding
+    assert_option_escape_all_tags
+    assert_option_include
+    assert_option_quiet
+    assert_option_tags
+    assert_option_untagged
+    assert_option_validate
+  end
+
+  def test_option_format_required
+    assert_raises Twine::Error do
+      parse "generate-localization-archive twine_file output"
+    end
+  end
+
+  def test_supports_deprecated_command
+    parse "generate-loc-drop #{@twine_file_path} #{@output_path} --format apple"
+    assert_equal 'generate-localization-archive', @options[:command]
+  end
+
+  def test_deprecated_command_prints_warning
+    parse "generate-loc-drop #{@twine_file_path} #{@output_path} --format apple"
+    assert_match "WARNING: Twine commands names have changed.", Twine::stdout.string
+  end
+end
+
+class TestConsumeLocalizationFileCLI < CLITest
+  def parse_with(parameters)
+    parse "consume-localization-file #{@twine_file_path} #{@input_path} " + parameters
+  end
+
+  def test_command
+    parse_with ""
+
+    assert_equal 'consume-localization-file', @options[:command]
+    assert_equal @twine_file_path, @options[:twine_file]
+    assert_equal @input_path, @options[:input_path]
+  end
+
+  def test_missing_argument
+    assert_raises Twine::Error do
+      parse "consume-localization-file twine_file"
+    end
+  end
+
+  def test_extra_argument
+    assert_raises Twine::Error do
+      parse_with "extra"
+    end
+  end
+
+  def test_options
+    assert_help
+    assert_option_consume_all
+    assert_option_consume_comments
+    assert_option_developer_language
+    assert_option_encoding
+    assert_option_format
+    assert_option_single_language
+    assert_raises(Twine::Error) { assert_option_multiple_languages }
+    assert_option_output_path
+    assert_option_quiet
+    assert_option_tags
+  end
+end
+
+class TestConsumeAllLocalizationFilesCLI < CLITest
+  def parse_with(parameters)
+    parse "consume-all-localization-files #{@twine_file_path} #{@input_dir} " + parameters
+  end
+
+  def test_command
+    parse_with ""
+
+    assert_equal 'consume-all-localization-files', @options[:command]
+    assert_equal @twine_file_path, @options[:twine_file]
+    assert_equal @input_dir, @options[:input_path]
+  end
+
+  def test_missing_argument
+    assert_raises Twine::Error do
+      parse "consume-all-localization-files twine_file"
+    end
+  end
+
+  def test_extra_argument
+    assert_raises Twine::Error do
+      parse_with "extra"
+    end
+  end
+
+  def test_options
+    assert_help
+    assert_option_consume_all
+    assert_option_consume_comments
+    assert_option_developer_language
+    assert_option_encoding
+    assert_option_format
+    assert_option_output_path
+    assert_option_quiet
+    assert_option_tags
+  end
+end
+
+class TestConsumeLocalizationArchiveCLI < CLITest
+  def parse_with(parameters)
+    parse "consume-localization-archive #{@twine_file_path} #{@input_path} " + parameters
+  end
+
+  def test_command
+    parse_with  ""
+
+    assert_equal 'consume-localization-archive', @options[:command]
+    assert_equal @twine_file_path, @options[:twine_file]
+    assert_equal @input_path, @options[:input_path]
+  end
+
+  def test_missing_argument
+    assert_raises Twine::Error do
+      parse "consume-localization-archive twine_file"
+    end
+  end
+
+  def test_extra_argument
+    assert_raises Twine::Error do
+      parse_with "extra"
+    end
+  end
+
+  def test_options
+    assert_help
+    assert_option_consume_all
+    assert_option_consume_comments
+    assert_option_developer_language
+    assert_option_encoding
+    assert_option_format
+    assert_option_output_path
+    assert_option_quiet
+    assert_option_tags
+  end
+
+  def test_supports_deprecated_command
+    parse "consume-loc-drop #{@twine_file_path} #{@input_path}"
+    assert_equal 'consume-localization-archive', @options[:command]
+  end
+
+  def test_deprecated_command_prints_warning
+    parse "consume-loc-drop #{@twine_file_path} #{@input_path}"
+    assert_match "WARNING: Twine commands names have changed.", Twine::stdout.string
+  end
+end
+
+class TestValidateTwineFileCLI < CLITest
+  def parse_with(parameters)
+    parse "validate-twine-file #{@twine_file_path} " + parameters
+  end
+
+  def test_command
+    parse_with ""
+
+    assert_equal 'validate-twine-file', @options[:command]
+    assert_equal @twine_file_path, @options[:twine_file]
+  end
+
+  def test_missing_argument
+    assert_raises Twine::Error do
+      parse 'validate-twine-file'
+    end
+  end
+
+  def test_extra_argument
+    assert_raises Twine::Error do
+      parse_with 'extra'
+    end
+  end
+
+  def test_options
+    assert_help
+    assert_option_developer_language
+    assert_option_quiet
+  end
+
+  def test_option_pedantic
+    parse "validate-twine-file #{@twine_file_path} --pedantic"
+    assert @options[:pedantic]
+    parse "validate-twine-file #{@twine_file_path} --no-pedantic"
+    refute @options[:pedantic]
+  end
+end

test/test_consume_localization_archive.rb

@@ -0,0 +1,33 @@
+require 'command_test'
+
+class TestConsumeLocalizationArchive < CommandTest
+  def setup
+    super
+
+    @twine_file = build_twine_file 'en', 'es' do
+      add_section 'Section' do
+        add_definition key1: 'value1'
+      end
+    end
+  end
+
+  def new_runner(options = {})
+    options[:input_path] = fixture_path 'consume_localization_archive.zip'
+    options[:output_path] = @output_path
+
+    Twine::Runner.new(options, @twine_file)
+  end
+
+  def test_consumes_zip_file
+    new_runner(format: 'android').consume_localization_archive
+
+    assert @twine_file.definitions_by_key['key1'].translations['en'], 'value1-english'
+    assert @twine_file.definitions_by_key['key1'].translations['es'], 'value1-spanish'
+  end
+
+  def test_raises_error_if_format_ambiguous
+    assert_raises Twine::Error do
+      new_runner.consume_localization_archive
+    end
+  end
+end

test/test_consume_localization_file.rb

@@ -0,0 +1,119 @@
+require 'command_test'
+
+class TestConsumeLocalizationFile < CommandTest
+  def new_runner(language, file)
+    options = {}
+    options[:output_path] = File.join(@output_dir, file) if file
+    options[:input_path] = File.join(@output_dir, file) if file
+    FileUtils.touch options[:input_path]
+    options[:languages] = language if language
+
+    @twine_file = Twine::TwineFile.new
+    @twine_file.language_codes.concat KNOWN_LANGUAGES
+
+    Twine::Runner.new(options, @twine_file)
+  end
+
+  def prepare_mock_read_formatter(formatter_class)
+    formatter = prepare_mock_formatter(formatter_class)
+    formatter.expects(:read)
+  end
+
+  def test_deducts_android_format_from_output_path
+    prepare_mock_read_formatter Twine::Formatters::Android
+
+    new_runner('fr', 'fr.xml').consume_localization_file
+  end
+
+  def test_deducts_apple_format_from_output_path
+    prepare_mock_read_formatter Twine::Formatters::Apple
+
+    new_runner('fr', 'fr.strings').consume_localization_file
+  end
+
+  def test_deducts_jquery_format_from_output_path
+    prepare_mock_read_formatter Twine::Formatters::JQuery
+
+    new_runner('fr', 'fr.json').consume_localization_file
+  end
+
+  def test_deducts_gettext_format_from_output_path
+    prepare_mock_read_formatter Twine::Formatters::Gettext
+
+    new_runner('fr', 'fr.po').consume_localization_file
+  end
+
+  def test_deducts_language_from_input_path
+    random_language = KNOWN_LANGUAGES.sample
+    formatter = prepare_mock_formatter Twine::Formatters::Android
+    formatter.expects(:read).with(anything, random_language)
+
+    new_runner(nil, "#{random_language}.xml").consume_localization_file
+  end
+
+  class TestEncodings < CommandTest
+    class DummyFormatter < Twine::Formatters::Abstract
+      attr_reader :content
+
+      def extension
+        '.dummy'
+      end
+
+      def format_name
+        'dummy'
+      end
+
+      def read(io, lang)
+        @content = io.read
+      end
+    end
+
+    def new_runner(input_path, encoding = nil)
+      options = {}
+      options[:output_path] = @output_path
+      options[:input_path] = input_path
+      options[:encoding] = encoding if encoding
+      options[:languages] = 'en'
+
+      @twine_file = Twine::TwineFile.new
+      @twine_file.language_codes.concat KNOWN_LANGUAGES
+
+      Twine::Runner.new(options, @twine_file)
+    end
+
+    def setup
+      super
+      @expected_content = "Üß`\nda\n"
+    end
+
+    def test_reads_utf8
+      formatter = prepare_mock_formatter DummyFormatter
+      new_runner(fixture_path('enc_utf8.dummy')).consume_localization_file
+      assert_equal @expected_content, formatter.content
+    end
+
+    def test_reads_utf16le_bom
+      formatter = prepare_mock_formatter DummyFormatter
+      new_runner(fixture_path('enc_utf16le_bom.dummy')).consume_localization_file
+      assert_equal @expected_content, formatter.content
+    end
+
+    def test_reads_utf16be_bom
+      formatter = prepare_mock_formatter DummyFormatter
+      new_runner(fixture_path('enc_utf16be_bom.dummy')).consume_localization_file
+      assert_equal @expected_content, formatter.content
+    end
+
+    def test_reads_utf16le
+      formatter = prepare_mock_formatter DummyFormatter
+      new_runner(fixture_path('enc_utf16le.dummy'), 'UTF-16LE').consume_localization_file
+      assert_equal @expected_content, formatter.content
+    end
+
+    def test_reads_utf16be
+      formatter = prepare_mock_formatter DummyFormatter
+      new_runner(fixture_path('enc_utf16be.dummy'), 'UTF-16BE').consume_localization_file
+      assert_equal @expected_content, formatter.content
+    end
+  end
+end

test/test_formatters.rb

@@ -0,0 +1,624 @@
+require 'twine_test'
+
+class FormatterTest < TwineTest
+  def setup(formatter_class)
+    super()
+
+    @twine_file = build_twine_file 'en' do
+      add_section 'Section 1' do
+        add_definition key1: 'value1-english', comment: 'comment key1'
+        add_definition key2: 'value2-english'
+      end
+
+      add_section 'Section 2' do
+        add_definition key3: 'value3-english'
+        add_definition key4: 'value4-english', comment: 'comment key4'
+      end
+    end
+
+    @empty_twine_file = Twine::TwineFile.new
+    @formatter = formatter_class.new
+    @formatter.twine_file = @empty_twine_file
+    @formatter.options = { consume_all: true, consume_comments: true }
+  end
+
+  def assert_translations_read_correctly
+    1.upto(4) do |i|
+      assert_equal "value#{i}-english", @empty_twine_file.definitions_by_key["key#{i}"].translations['en']
+    end
+  end
+
+  def assert_file_contents_read_correctly
+    assert_translations_read_correctly
+
+    assert_equal "comment key1", @empty_twine_file.definitions_by_key["key1"].comment
+    assert_equal "comment key4", @empty_twine_file.definitions_by_key["key4"].comment
+  end
+end
+
+class TestAndroidFormatter < FormatterTest
+  def setup
+    super Twine::Formatters::Android
+
+    @escape_test_values = {
+      'this & that'               => 'this &amp; that',
+      'this < that'               => 'this &lt; that',
+      "it's complicated"          => "it\\'s complicated",
+      'a "good" way'              => 'a \"good\" way',
+
+      '<b>bold</b>'               => '<b>bold</b>',
+      '<em>bold</em>'             => '<em>bold</em>',
+
+      '<i>italic</i>'             => '<i>italic</i>',
+      '<cite>italic</cite>'       => '<cite>italic</cite>',
+      '<dfn>italic</dfn>'         => '<dfn>italic</dfn>',
+
+      '<big>larger</big>'         => '<big>larger</big>',
+      '<small>smaller</small>'    => '<small>smaller</small>',
+
+      '<font color="#45C1D0">F</font>' => '<font color="#45C1D0">F</font>',
+
+      '<tt>monospaced</tt>'       => '<tt>monospaced</tt>',
+
+      '<s>strike</s>'             => '<s>strike</s>',
+      '<strike>strike</strike>'   => '<strike>strike</strike>',
+      '<del>strike</del>'         => '<del>strike</del>',
+
+      '<u>underline</u>'          => '<u>underline</u>',
+
+      '<super>superscript</super>'=> '<super>superscript</super>',
+
+      '<sub>subscript</sub>'      => '<sub>subscript</sub>',
+
+      '<ul>bullet point</ul>'     => '<ul>bullet point</ul>',
+      '<li>bullet point</li>'     => '<li>bullet point</li>',
+
+      '<br>line break'            => '<br>line break',
+
+      '<div>division</div>'       => '<div>division</div>',
+
+      '<span style="color:#45C1D0">inline</span>' => '<span style="color:#45C1D0">inline</span>',
+
+      '<p>para</p>'               => '<p>para</p>',
+      '<p dir="ltr">para</p>'     => '<p dir="ltr">para</p>',
+
+      '<b>%@</b>'                 => '&lt;b>%s&lt;/b>',
+      '<em>%@</em>'               => '&lt;em>%s&lt;/em>',
+
+      '<i>%@</i>'                 => '&lt;i>%s&lt;/i>',
+      '<cite>%@</cite>'           => '&lt;cite>%s&lt;/cite>',
+      '<dfn>%@</dfn>'             => '&lt;dfn>%s&lt;/dfn>',
+
+      '<big>%@</big>'             => '&lt;big>%s&lt;/big>',
+      '<small>%@</small>'         => '&lt;small>%s&lt;/small>',
+
+      '<font color="#45C1D0>%@</font>' => '&lt;font color="#45C1D0>%s&lt;/font>',
+
+      '<tt>%@</tt>'               => '&lt;tt>%s&lt;/tt>',
+
+      '<s>%@</s>'                 => '&lt;s>%s&lt;/s>',
+      '<strike>%@</strike>'       => '&lt;strike>%s&lt;/strike>',
+      '<del>%@</del>'             => '&lt;del>%s&lt;/del>',
+
+      '<u>%@</u>'                 => '&lt;u>%s&lt;/u>',
+
+      '<super>%@</super>'         => '&lt;super>%s&lt;/super>',
+
+      '<sub>%@</sub>'             => '&lt;sub>%s&lt;/sub>',
+
+      '<ul>%@</ul>'               => '&lt;ul>%s&lt;/ul>',
+      '<li>%@</li>'               => '&lt;li>%s&lt;/li>',
+
+      '<br>%@'                    => '&lt;br>%s',
+
+      '<div>%@</div>'             => '&lt;div>%s&lt;/div>',
+
+      '<span style="color:#45C1D0">%@</span>' => '&lt;span style="color:#45C1D0">%s&lt;/span>',
+
+      '<p>%@</p>'                 => '&lt;p>%s&lt;/p>',
+      '<p dir="ltr">%@</p>'       => '&lt;p dir="ltr">%s&lt;/p>',
+
+      '<a href="target">link</a>'     => '<a href="target">link</a>',
+      '<a href="target">"link"</a>'   => '<a href="target">\"link\"</a>',
+      '<a href="target"></a>"out"'    => '<a href="target"></a>\"out\"',
+      '<a href="http://url.com?param=1&param2=3&param3=%20">link</a>'   =>   '<a href="http://url.com?param=1&param2=3&param3=%20">link</a>',
+
+      '<q>escaped</q><![CDATA[]]>'                => '&lt;q>escaped&lt;/q><![CDATA[]]>',
+      '<![CDATA[]]><q>escaped</q>'                => '<![CDATA[]]>&lt;q>escaped&lt;/q>',
+      '<![CDATA[<q>unescaped</q>]]>'              => '<![CDATA[<q>unescaped</q>]]>',
+      '<![CDATA[<q>unescaped with %@</q>]]>'      => '<![CDATA[<q>unescaped with %s</q>]]>',
+      '<![CDATA[]]><![CDATA[<q>unescaped</q>]]>'  => '<![CDATA[]]><![CDATA[<q>unescaped</q>]]>',
+
+      '<![CDATA[&]]>'  => '<![CDATA[&]]>',
+      '<![CDATA[\']]>' => '<![CDATA[\']]>',
+      '<![CDATA["]]>'  => '<![CDATA["]]>',
+
+      '<xliff:g></xliff:g>' => '<xliff:g></xliff:g>',
+      '<xliff:g>untouched</xliff:g>' => '<xliff:g>untouched</xliff:g>',
+      '<xliff:g id="42">untouched</xliff:g>' => '<xliff:g id="42">untouched</xliff:g>',
+      '<xliff:g id="1">first</xliff:g> inbetween <xliff:g id="2">second</xliff:g>' => '<xliff:g id="1">first</xliff:g> inbetween <xliff:g id="2">second</xliff:g>'
+    }
+    @escape_all_test_values = {
+      '<b>bold</b>'               => '&lt;b>bold&lt;/b>',
+      '<i>italic</i>'             => '&lt;i>italic&lt;/i>',
+      '<u>underline</u>'          => '&lt;u>underline&lt;/u>'      
+    }
+  end
+
+  def test_read_format
+    @formatter.read content_io('formatter_android.xml'), 'en'
+
+    assert_file_contents_read_correctly
+  end
+
+  def test_read_multiline_translation
+    content = <<-EOCONTENT
+      <?xml version="1.0" encoding="utf-8"?>
+      <resources>
+        <string name="foo">This is
+     a string</string>
+      </resources>
+    EOCONTENT
+
+    io = StringIO.new(content)
+
+    @formatter.read io, 'en'
+
+    assert_equal 'This is\n a string', @empty_twine_file.definitions_by_key["foo"].translations['en']
+  end
+
+  def test_read_multiline_comment
+    content = <<-EOCONTENT
+      <?xml version="1.0" encoding="utf-8"?>
+      <resources>
+        <!-- multiline
+        comment -->
+        <string name="foo">This is
+     a string</string>
+      </resources>
+    EOCONTENT
+
+    io = StringIO.new(content)
+
+    @formatter.read io, 'en'
+
+    assert_equal 'multiline comment', @empty_twine_file.definitions_by_key["foo"].comment
+  end
+
+  def test_read_html_tags
+    content = <<-EOCONTENT
+      <?xml version="1.0" encoding="utf-8"?>
+      <resources>
+        <string name="foo">Hello, <b>BOLD</b></string>
+      </resources>
+    EOCONTENT
+
+    io = StringIO.new(content)
+
+    @formatter.read io, 'en'
+
+    assert_equal 'Hello, <b>BOLD</b>', @empty_twine_file.definitions_by_key["foo"].translations['en']
+  end
+
+  def test_double_quotes_are_not_modified
+    content = <<-EOCONTENT
+      <?xml version="1.0" encoding="utf-8"?>
+      <resources>
+        <string name="foo">Hello, <a href="http://www.foo.com">BOLD</a></string>
+      </resources>
+    EOCONTENT
+
+    io = StringIO.new(content)
+
+    @formatter.read io, 'en'
+
+    assert_equal 'Hello, <a href="http://www.foo.com">BOLD</a>', @empty_twine_file.definitions_by_key["foo"].translations['en']
+  end
+
+  def test_set_translation_converts_leading_spaces
+    @formatter.set_translation_for_key 'key1', 'en', "\u0020value"
+    assert_equal ' value', @empty_twine_file.definitions_by_key['key1'].translations['en']
+  end
+
+  def test_set_translation_coverts_trailing_spaces
+    @formatter.set_translation_for_key 'key1', 'en', "value\u0020\u0020"
+    assert_equal 'value  ', @empty_twine_file.definitions_by_key['key1'].translations['en']
+  end
+
+  def test_set_translation_converts_string_placeholders
+    @formatter.set_translation_for_key 'key1', 'en', "value %s"
+    assert_equal 'value %@', @empty_twine_file.definitions_by_key['key1'].translations['en']
+  end
+
+  def test_set_translation_unescapes_at_signs
+    @formatter.set_translation_for_key 'key1', 'en', '\@value'
+    assert_equal '@value', @empty_twine_file.definitions_by_key['key1'].translations['en']
+  end
+
+  def test_set_translation_unescaping
+    @escape_test_values.each do |expected, input|
+      @formatter.set_translation_for_key 'key1', 'en', input
+      assert_equal expected, @empty_twine_file.definitions_by_key['key1'].translations['en']
+    end
+
+    @escape_all_test_values.each do |expected, input|
+      @formatter.set_translation_for_key 'key1', 'en', input
+      assert_equal expected, @empty_twine_file.definitions_by_key['key1'].translations['en']
+    end
+  end
+
+  def test_format_file
+    formatter = Twine::Formatters::Android.new
+    formatter.twine_file = @twine_file
+    assert_equal content('formatter_android.xml'), formatter.format_file('en')
+  end
+
+  def test_format_key_with_space
+    assert_equal 'key ', @formatter.format_key('key ')
+  end
+
+  def test_format_value_with_leading_space
+    assert_equal "\\u0020value", @formatter.format_value(' value')
+  end
+
+  def test_format_value_with_trailing_space
+    assert_equal "value\\u0020", @formatter.format_value('value ')
+  end
+
+  def test_format_value_string_placeholder
+    assert_equal "The file %s could not be found.", @formatter.format_value("The file %@ could not be found.")
+  end
+
+  def test_format_value_escaping
+    @escape_test_values.each do |input, expected|
+      assert_equal expected, @formatter.format_value(input)
+    end
+
+    @formatter.options.merge!({ escape_all_tags: true })
+    @escape_all_test_values.each do |input, expected|
+      assert_equal expected, @formatter.format_value(input)
+    end
+  end
+
+  def test_format_value_escapes_non_resource_identifier_at_signs
+    assert_equal '\@whatever  \@\@', @formatter.format_value('@whatever  @@')
+  end
+
+  def test_format_value_does_not_modify_resource_identifiers
+    identifier = '@android:string/cancel'
+    assert_equal identifier, @formatter.format_value(identifier)
+  end
+
+  def test_deducts_language_from_filename
+    language = KNOWN_LANGUAGES.sample
+    assert_equal language, @formatter.determine_language_given_path("#{language}.xml")
+  end
+
+  def test_recognize_every_twine_language_from_filename
+    twine_file = build_twine_file "not-a-lang-code" do
+      add_section "Section" do
+        add_definition key: "value"
+      end
+    end
+
+    @formatter.twine_file = twine_file
+    assert_equal "not-a-lang-code", @formatter.determine_language_given_path("not-a-lang-code.xml")
+  end
+
+  def test_deducts_language_from_resource_folder
+    language = KNOWN_LANGUAGES.sample
+    assert_equal language, @formatter.determine_language_given_path("res/values-#{language}")
+  end
+
+  def test_deducts_language_and_region_from_resource_folder
+    assert_equal 'de-AT', @formatter.determine_language_given_path("res/values-de-rAT")
+  end
+
+  def test_does_not_deduct_language_from_device_capability_resource_folder
+    assert_nil @formatter.determine_language_given_path('res/values-w820dp')
+  end
+
+  def test_output_path_is_prefixed
+    assert_equal 'values-en', @formatter.output_path_for_language('en')
+  end
+
+  def test_output_path_with_region
+    assert_equal 'values-en-rGB', @formatter.output_path_for_language('en-GB')
+  end
+
+  def test_output_path_respects_default_lang
+    @formatter.twine_file.language_codes.concat KNOWN_LANGUAGES
+    non_default_language = KNOWN_LANGUAGES[1..-1].sample
+    assert_equal 'values', @formatter.output_path_for_language(KNOWN_LANGUAGES[0])
+    assert_equal "values-#{non_default_language}", @formatter.output_path_for_language(non_default_language)
+  end
+end
+
+class TestAppleFormatter < FormatterTest
+  def setup
+    super Twine::Formatters::Apple
+  end
+
+  def test_read_format
+    @formatter.read content_io('formatter_apple.strings'), 'en'
+
+    assert_file_contents_read_correctly
+  end
+
+  def test_deducts_language_from_filename
+    language = KNOWN_LANGUAGES.sample
+    assert_equal language, @formatter.determine_language_given_path("#{language}.strings")
+  end
+
+  def test_recognize_every_twine_language_from_filename
+    twine_file = build_twine_file "not-a-lang-code" do
+      add_section "Section" do
+        add_definition key: "value"
+      end
+    end
+
+    @formatter.twine_file = twine_file
+    assert_equal "not-a-lang-code", @formatter.determine_language_given_path("not-a-lang-code.strings")
+  end
+
+  def test_deducts_language_from_resource_folder
+    language = %w(en de fr).sample
+    assert_equal language, @formatter.determine_language_given_path("#{language}.lproj/Localizable.strings")
+  end
+
+  def test_deducts_base_language_from_resource_folder
+    @formatter.options = { consume_all: true, consume_comments: true, developer_language: 'en' }
+    assert_equal 'en', @formatter.determine_language_given_path('Base.lproj/Localizations.strings')
+  end
+
+  def test_reads_quoted_keys
+    @formatter.read StringIO.new('"key" = "value"'), 'en'
+    assert_equal 'value', @empty_twine_file.definitions_by_key['key'].translations['en']
+  end
+
+  def test_reads_unquoted_keys
+    @formatter.read StringIO.new('key = "value"'), 'en'
+    assert_equal 'value', @empty_twine_file.definitions_by_key['key'].translations['en']
+  end
+
+  def test_ignores_leading_whitespace_before_quoted_keys
+    @formatter.read StringIO.new("\t  \"key\" = \"value\""), 'en'
+    assert_equal 'value', @empty_twine_file.definitions_by_key['key'].translations['en']
+  end
+
+  def test_ignores_leading_whitespace_before_unquoted_keys
+    @formatter.read StringIO.new("\t  key = \"value\""), 'en'
+    assert_equal 'value', @empty_twine_file.definitions_by_key['key'].translations['en']
+  end
+
+  def test_allows_quotes_in_quoted_keys
+    @formatter.read StringIO.new('"ke\"y" = "value"'), 'en'
+    assert_equal 'value', @empty_twine_file.definitions_by_key['ke"y'].translations['en']
+  end
+
+  def test_does_not_allow_quotes_in_quoted_keys
+    @formatter.read StringIO.new('ke"y = "value"'), 'en'
+    assert_nil @empty_twine_file.definitions_by_key['key']
+  end
+
+  def test_allows_equal_signs_in_quoted_keys
+    @formatter.read StringIO.new('"k=ey" = "value"'), 'en'
+    assert_equal 'value', @empty_twine_file.definitions_by_key['k=ey'].translations['en']
+  end
+
+  def test_does_not_allow_equal_signs_in_unquoted_keys
+    @formatter.read StringIO.new('k=ey = "value"'), 'en'
+    assert_nil @empty_twine_file.definitions_by_key['key']
+  end
+
+  def test_format_file
+    formatter = Twine::Formatters::Apple.new
+    formatter.twine_file = @twine_file
+    assert_equal content('formatter_apple.strings'), formatter.format_file('en')
+  end
+
+  def test_format_key_with_space
+    assert_equal 'key ', @formatter.format_key('key ')
+  end
+
+  def test_format_value_with_leading_space
+    assert_equal ' value', @formatter.format_value(' value')
+  end
+
+  def test_format_value_with_trailing_space
+    assert_equal 'value ', @formatter.format_value('value ')
+  end
+end
+
+class TestJQueryFormatter < FormatterTest
+
+  def setup
+    super Twine::Formatters::JQuery
+  end
+
+  def test_read_format
+    @formatter.read content_io('formatter_jquery.json'), 'en'
+
+    assert_translations_read_correctly
+  end
+
+  def test_format_file
+    formatter = Twine::Formatters::JQuery.new
+    formatter.twine_file = @twine_file
+    assert_equal content('formatter_jquery.json'), formatter.format_file('en')
+  end
+
+  def test_empty_sections_are_removed
+    @twine_file = build_twine_file 'en' do
+      add_section 'Section 1' do
+      end
+
+      add_section 'Section 2' do
+        add_definition key: 'value'
+      end
+    end
+    formatter = Twine::Formatters::JQuery.new
+    formatter.twine_file = @twine_file
+    refute_includes formatter.format_file('en'), ','
+  end
+
+  def test_format_value_with_newline
+    assert_equal "value\nwith\nline\nbreaks", @formatter.format_value("value\nwith\nline\nbreaks")
+  end
+
+  def test_deducts_language_from_filename
+    language = KNOWN_LANGUAGES.sample
+    assert_equal language, @formatter.determine_language_given_path("#{language}.json")
+  end
+
+  def test_deducts_language_from_extended_filename
+    language = KNOWN_LANGUAGES.sample
+    assert_equal language, @formatter.determine_language_given_path("something-#{language}.json")
+  end
+
+  def test_deducts_language_from_path
+    language = %w(en-GB de fr).sample
+    assert_equal language, @formatter.determine_language_given_path("/output/#{language}/#{@formatter.default_file_name}")
+  end
+end
+
+class TestGettextFormatter < FormatterTest
+  def setup
+    super Twine::Formatters::Gettext
+  end
+
+  def test_read_format
+    @formatter.read content_io('formatter_gettext.po'), 'en'
+
+    assert_file_contents_read_correctly
+  end
+
+  def test_read_with_multiple_line_value
+    @formatter.read content_io('gettext_multiline.po'), 'en'
+
+    assert_equal 'multiline\nstring', @empty_twine_file.definitions_by_key['key1'].translations['en']
+  end
+
+  def test_format_file
+    formatter = Twine::Formatters::Gettext.new
+    formatter.twine_file = @twine_file
+    assert_equal content('formatter_gettext.po'), formatter.format_file('en')
+  end
+
+  def test_deducts_language_and_region
+    language = "en-GB"
+    assert_equal language, @formatter.determine_language_given_path("#{language}.po")
+  end
+
+  def test_deducts_language_from_path
+    language = %w(en-GB de fr).sample
+    assert_equal language, @formatter.determine_language_given_path("/output/#{language}/#{@formatter.default_file_name}")
+  end
+
+  def test_quoted_strings
+    formatter = Twine::Formatters::Gettext.new
+    formatter.twine_file = build_twine_file "not-a-lang-code" do
+      add_section "Section" do
+        add_definition key: "foo \"bar\" baz"
+      end
+    end
+    assert_equal content('formatter_gettext_quotes.po'), formatter.format_file('en')
+  end
+end
+
+class TestTizenFormatter < FormatterTest
+
+  def setup
+    super Twine::Formatters::Tizen
+  end
+
+  def test_read_format
+    skip 'the current implementation of Tizen formatter does not support reading'
+    @formatter.read content_io('formatter_tizen.xml'), 'en'
+
+    assert_file_contents_read_correctly
+  end
+
+  def test_format_file
+    formatter = Twine::Formatters::Tizen.new
+    formatter.twine_file = @twine_file
+    assert_equal content('formatter_tizen.xml'), formatter.format_file('en')
+  end
+end
+
+class TestDjangoFormatter < FormatterTest
+  def setup
+    super Twine::Formatters::Django
+  end
+
+  def test_read_format
+    @formatter.read content_io('formatter_django.po'), 'en'
+
+    assert_file_contents_read_correctly
+  end
+
+  def test_format_file
+    formatter = Twine::Formatters::Django.new
+    formatter.twine_file = @twine_file
+    assert_equal content('formatter_django.po'), formatter.format_file('en')
+  end
+
+  def test_deducts_language_and_region
+    language = "en-GB"
+    assert_equal language, @formatter.determine_language_given_path("#{language}.po")
+  end
+
+  def test_deducts_language_from_path
+    language = %w(en-GB de fr).sample
+    assert_equal language, @formatter.determine_language_given_path("/output/#{language}/#{@formatter.default_file_name}")
+  end
+
+  def test_ignores_commented_out_strings
+    content = <<-EOCONTENT
+      #~ msgid "foo"
+      #~ msgstr "This should be ignored"
+    EOCONTENT
+
+    io = StringIO.new(content)
+
+    @formatter.read io, 'en'
+
+    assert_nil @empty_twine_file.definitions_by_key["foo"]
+  end
+end
+
+class TestFlashFormatter < FormatterTest
+  def setup
+    super Twine::Formatters::Flash
+  end
+
+  def test_read_format
+    @formatter.read content_io('formatter_flash.properties'), 'en'
+
+    assert_file_contents_read_correctly
+  end
+
+  def test_set_translation_converts_placeholders
+    @formatter.set_translation_for_key 'key1', 'en', "value {#{rand(10)}}"
+    assert_equal 'value %@', @empty_twine_file.definitions_by_key['key1'].translations['en']
+  end
+
+  def test_format_file
+    formatter = Twine::Formatters::Flash.new
+    formatter.twine_file = @twine_file
+    assert_equal content('formatter_flash.properties'), formatter.format_file('en')
+  end
+
+  def test_format_value_converts_placeholders
+    assert_equal "value {0}", @formatter.format_value('value %d')
+  end
+
+  def test_deducts_language_from_resource_folder
+    language = %w(en de fr).sample
+    assert_equal language, @formatter.determine_language_given_path("locale/#{language}/#{@formatter.default_file_name}")
+  end
+
+  def test_deducts_language_and_region_from_resource_folder
+    assert_equal 'de-AT', @formatter.determine_language_given_path("locale/de-AT/#{@formatter.default_file_name}")
+  end
+end

test/test_generate_all_localization_files.rb

@@ -0,0 +1,140 @@
+require 'command_test'
+
+class TestGenerateAllLocalizationFiles < CommandTest
+  def new_runner(create_folders, twine_file = nil, options = {})
+    default_options = {}
+    default_options[:output_path] = @output_dir
+    default_options[:format] = 'apple'
+    default_options[:create_folders] = create_folders
+
+    options = default_options.merge options
+
+    unless twine_file
+      twine_file = build_twine_file 'en', 'es' do
+        add_section 'Section' do
+          add_definition key: 'value'
+        end
+      end
+    end
+
+    Twine::Runner.new(options, twine_file)
+  end
+
+  class TestFormatterSelection < TestGenerateAllLocalizationFiles
+    def setup
+      super
+      Dir.mkdir File.join @output_dir, 'values-en'
+    end
+
+    def new_runner(options = {})
+      super(true, nil, options)
+    end
+
+    def test_returns_error_for_ambiguous_output_path
+      assert_raises Twine::Error do
+        new_runner(format: nil).generate_all_localization_files
+      end
+    end
+
+    def test_uses_specified_formatter_to_resolve_ambiguity
+      # implicit assert that this call doesn't raise an exception
+      new_runner(format: 'android').generate_all_localization_files
+    end
+  end
+
+  class TestDoNotCreateFolders < TestGenerateAllLocalizationFiles
+    def new_runner(twine_file = nil, options = {})
+      super(false, twine_file, options)
+    end
+
+    def test_fails_if_output_folder_does_not_exist
+      assert_raises Twine::Error do
+        new_runner.generate_all_localization_files
+      end
+    end
+
+    def test_does_not_create_language_folders
+      Dir.mkdir File.join @output_dir, 'en.lproj'
+      new_runner.generate_all_localization_files
+      refute File.exist?(File.join(@output_dir, 'es.lproj')), "language folder should not be created"
+    end
+
+    def test_prints_empty_file_warnings
+      Dir.mkdir File.join @output_dir, 'en.lproj'
+      empty_twine_file = build_twine_file('en') {}
+      new_runner(empty_twine_file).generate_all_localization_files
+      assert_match "Skipping file at path", Twine::stdout.string
+    end
+
+    def test_does_not_print_empty_file_warnings_if_quite
+      Dir.mkdir File.join @output_dir, 'en.lproj'
+      empty_twine_file = build_twine_file('en') {}
+      new_runner(empty_twine_file, quite: true).generate_all_localization_files
+      refute_match "Skipping file at path", Twine::stdout.string
+    end
+  end
+
+  class TestCreateFolders < TestGenerateAllLocalizationFiles
+    def new_runner(twine_file = nil, options = {})
+      super(true, twine_file, options)
+    end
+
+    def test_creates_output_folder
+      FileUtils.remove_entry_secure @output_dir
+      new_runner.generate_all_localization_files
+      assert File.exist? @output_dir
+    end
+
+    def test_creates_language_folders
+      new_runner.generate_all_localization_files
+      assert File.exist?(File.join(@output_dir, 'en.lproj')), "language folder 'en.lproj' should be created"
+      assert File.exist?(File.join(@output_dir, 'es.lproj')), "language folder 'es.lproj' should be created"
+    end
+
+    def test_prints_empty_file_warnings
+      empty_twine_file = build_twine_file('en') {}
+      new_runner(empty_twine_file).generate_all_localization_files
+
+      assert_match "Skipping file at path", Twine::stdout.string
+    end
+
+    def test_does_not_print_empty_file_warnings_if_quite
+      empty_twine_file = build_twine_file('en') {}
+      new_runner(empty_twine_file, quite: true).generate_all_localization_files
+
+      refute_match "Skipping file at path", Twine::stdout.string
+    end
+  end
+
+  class TestValidate < CommandTest
+    def new_runner(validate)
+      Dir.mkdir File.join @output_dir, 'values-en'
+
+      options = {}
+      options[:output_path] = @output_dir
+      options[:format] = 'android'
+      options[:validate] = validate
+
+      twine_file = build_twine_file 'en' do
+        add_section 'Section' do
+          add_definition key: 'value'
+          add_definition key: 'value'
+        end
+      end
+
+      Twine::Runner.new(options, twine_file)
+    end
+
+    def test_does_not_validate_twine_file
+      prepare_mock_formatter Twine::Formatters::Android
+
+      new_runner(false).generate_all_localization_files
+    end
+
+    def test_validates_twine_file_if_validate
+      assert_raises Twine::Error do
+        new_runner(true).generate_all_localization_files
+      end
+    end
+  end
+end

test/test_generate_localization_archive.rb

@@ -0,0 +1,85 @@
+require 'command_test'
+
+class TestGenerateLocalizationArchive < CommandTest
+  def new_runner(twine_file = nil, options = {})
+    options[:output_path] = @output_path
+    options[:format] = 'apple'
+
+    unless twine_file
+      twine_file = build_twine_file 'en', 'fr' do
+        add_section 'Section' do
+          add_definition key: 'value'
+        end
+      end
+    end
+
+    Twine::Runner.new(options, twine_file)
+  end
+
+  def test_generates_zip_file
+    new_runner.generate_localization_archive
+
+    assert File.exist?(@output_path), "zip file should exist"
+  end
+
+  def test_zip_file_structure
+    new_runner.generate_localization_archive
+
+    names = []
+    Zip::File.open(@output_path) do |zipfile|
+      zipfile.each do |entry|
+        names << entry.name
+      end
+    end
+    assert_equal ['Locales/', 'Locales/en.strings', 'Locales/fr.strings'], names
+  end
+
+  def test_uses_formatter
+    formatter = prepare_mock_formatter Twine::Formatters::Apple
+    formatter.expects(:format_file).twice
+
+    new_runner.generate_localization_archive
+  end
+
+  def test_prints_empty_file_warnings
+    empty_twine_file = build_twine_file('en') {}
+    new_runner(empty_twine_file).generate_localization_archive
+    assert_match "Skipping file", Twine::stdout.string
+  end
+
+  def test_does_not_print_empty_file_warnings_if_quite
+    empty_twine_file = build_twine_file('en') {}
+    new_runner(empty_twine_file, quite: true).generate_localization_archive
+    refute_match "Skipping file", Twine::stdout.string
+  end
+
+  class TestValidate < CommandTest
+    def new_runner(validate)
+      options = {}
+      options[:output_path] = @output_path
+      options[:format] = 'android'
+      options[:validate] = validate
+
+      twine_file = build_twine_file 'en' do
+        add_section 'Section' do
+          add_definition key: 'value'
+          add_definition key: 'value'
+        end
+      end
+
+      Twine::Runner.new(options, twine_file)
+    end
+
+    def test_does_not_validate_twine_file
+      prepare_mock_formatter Twine::Formatters::Android
+
+      new_runner(false).generate_localization_archive
+    end
+
+    def test_validates_twine_file_if_validate
+      assert_raises Twine::Error do
+        new_runner(true).generate_localization_archive
+      end
+    end
+  end
+end

test/test_generate_localization_file.rb

@@ -0,0 +1,119 @@
+require 'command_test'
+
+class TestGenerateLocalizationFile < CommandTest
+  def new_runner(language, file, options = {})
+    options[:output_path] = File.join(@output_dir, file) if file
+    options[:languages] = language if language
+
+    twine_file = Twine::TwineFile.new
+    twine_file.language_codes.concat KNOWN_LANGUAGES
+
+    Twine::Runner.new(options, twine_file)
+  end
+
+  def prepare_mock_format_file_formatter(formatter_class)
+    formatter = prepare_mock_formatter(formatter_class)
+    formatter.expects(:format_file).returns(true)
+  end
+
+  def test_deducts_android_format_from_output_path
+    prepare_mock_format_file_formatter Twine::Formatters::Android
+
+    new_runner('fr', 'fr.xml').generate_localization_file
+  end
+
+  def test_deducts_apple_format_from_output_path
+    prepare_mock_format_file_formatter Twine::Formatters::Apple
+
+    new_runner('fr', 'fr.strings').generate_localization_file
+  end
+
+  def test_deducts_jquery_format_from_output_path
+    prepare_mock_format_file_formatter Twine::Formatters::JQuery
+
+    new_runner('fr', 'fr.json').generate_localization_file
+  end
+
+  def test_deducts_gettext_format_from_output_path
+    prepare_mock_format_file_formatter Twine::Formatters::Gettext
+
+    new_runner('fr', 'fr.po').generate_localization_file
+  end
+
+  def test_deducts_django_format_from_output_path
+    prepare_mock_format_file_formatter Twine::Formatters::Django
+
+    new_runner('fr', 'fr.po').generate_localization_file
+  end
+
+  def test_returns_error_for_ambiguous_output_path
+    # both Gettext and Django use .po
+    gettext_formatter = prepare_mock_formatter(Twine::Formatters::Gettext)
+    gettext_formatter.stubs(:format_file).returns(true)
+    django_formatter = prepare_mock_formatter(Twine::Formatters::Django, false)
+    django_formatter.stubs(:format_file).returns(true)
+
+    assert_raises Twine::Error do
+      new_runner('fr', 'fr.po').generate_localization_file
+    end
+  end
+
+  def test_uses_specified_formatter_to_resolve_ambiguity
+    # both Android and Tizen use .xml
+    android_formatter = prepare_mock_formatter(Twine::Formatters::Android)
+    android_formatter.stubs(:format_file).returns(true)
+    tizen_formatter = prepare_mock_formatter(Twine::Formatters::Tizen, false)
+    tizen_formatter.stubs(:format_file).returns(true)
+
+    # implicit assert that this call doesn't raise an exception
+    new_runner('fr', 'fr.xml', format: 'android').generate_localization_file
+  end
+
+  def test_deducts_language_from_output_path
+    random_language = KNOWN_LANGUAGES.sample
+    formatter = prepare_mock_formatter Twine::Formatters::Android
+    formatter.expects(:format_file).with(random_language).returns(true)
+
+    new_runner(nil, "#{random_language}.xml").generate_localization_file
+  end
+
+  def test_returns_error_if_nothing_written
+    formatter = prepare_mock_formatter Twine::Formatters::Android
+    formatter.expects(:format_file).returns(false)
+
+    assert_raises Twine::Error do
+      new_runner('fr', 'fr.xml').generate_localization_file
+    end
+  end
+
+  class TestValidate < CommandTest
+    def new_runner(validate)
+      options = {}
+      options[:output_path] = @output_path
+      options[:languages] = ['en']
+      options[:format] = 'android'
+      options[:validate] = validate
+
+      twine_file = build_twine_file 'en' do
+        add_section 'Section' do
+          add_definition key: 'value'
+          add_definition key: 'value'
+        end
+      end
+
+      Twine::Runner.new(options, twine_file)
+    end
+
+    def test_does_not_validate_twine_file
+      prepare_mock_formatter Twine::Formatters::Android
+
+      new_runner(false).generate_localization_file
+    end
+
+    def test_validates_twine_file_if_validate
+      assert_raises Twine::Error do
+        new_runner(true).generate_localization_file
+      end
+    end
+  end
+end

test/test_output_processor.rb

@@ -0,0 +1,85 @@
+require 'twine_test'
+
+class TestOutputProcessor < TwineTest
+  def setup
+    super
+
+    @twine_file = build_twine_file 'en', 'fr' do
+      add_section 'Section' do
+        add_definition key1: 'value1', tags: ['tag1']
+        add_definition key2: 'value2', tags: ['tag1', 'tag2']
+        add_definition key3: 'value3', tags: ['tag2']
+        add_definition key4: { en: 'value4-en', fr: 'value4-fr' }
+      end
+    end
+  end
+
+  def test_includes_all_keys_by_default
+    processor = Twine::Processors::OutputProcessor.new(@twine_file, {})
+    result = processor.process('en')
+
+    assert_equal %w(key1 key2 key3 key4), result.definitions_by_key.keys.sort
+  end
+
+  def test_filter_by_tag
+    processor = Twine::Processors::OutputProcessor.new(@twine_file, { tags: [['tag1']] })
+    result = processor.process('en')
+
+    assert_equal %w(key1 key2), result.definitions_by_key.keys.sort
+  end
+
+  def test_filter_by_multiple_tags
+    processor = Twine::Processors::OutputProcessor.new(@twine_file, { tags: [['tag1', 'tag2']] })
+    result = processor.process('en')
+
+    assert_equal %w(key1 key2 key3), result.definitions_by_key.keys.sort
+  end
+
+  def test_filter_untagged
+    processor = Twine::Processors::OutputProcessor.new(@twine_file, { tags: [['tag1']], untagged: true })
+    result = processor.process('en')
+
+    assert_equal %w(key1 key2 key4), result.definitions_by_key.keys.sort
+  end
+
+  def test_include_translated
+    processor = Twine::Processors::OutputProcessor.new(@twine_file, { include: :translated })
+    result = processor.process('fr')
+
+    assert_equal %w(key4), result.definitions_by_key.keys.sort
+  end
+
+  def test_include_untranslated
+    processor = Twine::Processors::OutputProcessor.new(@twine_file, { include: :untranslated })
+    result = processor.process('fr')
+
+    assert_equal %w(key1 key2 key3), result.definitions_by_key.keys.sort
+  end
+
+  class TranslationFallback < TwineTest
+    def setup
+      super
+
+      @twine_file = build_twine_file 'en', 'fr', 'de' do
+        add_section 'Section' do
+          add_definition key1: { en: 'value1-en', fr: 'value1-fr' }
+        end
+      end
+    end
+
+    def test_fallback_to_default_language
+      processor = Twine::Processors::OutputProcessor.new(@twine_file, {})
+      result = processor.process('de')
+      
+      assert_equal 'value1-en', result.definitions_by_key['key1'].translations['de']
+    end
+
+    def test_fallback_to_developer_language
+      processor = Twine::Processors::OutputProcessor.new(@twine_file, {developer_language: 'fr'})
+      result = processor.process('de')
+      
+      assert_equal 'value1-fr', result.definitions_by_key['key1'].translations['de']
+    end
+  end
+
+end

test/test_placeholders.rb

@@ -0,0 +1,142 @@
+require 'twine_test'
+
+class PlaceholderTest < TwineTest
+  def assert_starts_with(prefix, value)
+    msg = message(nil) { "Expected #{mu_pp(value)} to start with #{mu_pp(prefix)}" }
+    assert value.start_with?(prefix), msg
+  end
+
+  def placeholder(type = nil)
+    # %[parameter][flags][width][.precision][length]type (see https://en.wikipedia.org/wiki/Printf_format_string#Format_placeholder_specification)
+    lucky = lambda { rand > 0.5 }
+    placeholder = '%'
+    placeholder += (rand * 20).to_i.to_s + '$' if lucky.call
+    placeholder += '-+0#'.chars.to_a.sample if lucky.call
+    placeholder += (0.upto(20).map(&:to_s) << "*").sample if lucky.call
+    placeholder += '.' + (0.upto(20).map(&:to_s) << "*").sample if lucky.call
+    placeholder += %w(h hh l ll L z j t).sample if lucky.call
+    placeholder += type || 'diufFeEgGxXocpaA'.chars.to_a.sample # this does not contain s or @ because strings are a special case
+  end
+
+  class ToAndroid < PlaceholderTest
+    def to_android(value)
+      Twine::Placeholders.convert_placeholders_from_twine_to_android(value)
+    end
+
+    def test_replaces_simple_string_placeholder
+      assert_equal "some '%s' value", to_android("some '%@' value")
+    end
+
+    def test_replaces_complicated_string_placeholder
+      placeholder = placeholder('@')
+      expected = placeholder.dup
+      expected[-1] = 's'
+      assert_equal "some #{expected} value", to_android("some #{placeholder} value")
+    end
+
+    def test_does_not_change_regular_at_signs
+      input = "some @ more @@ signs @"
+      assert_equal input, to_android(input)
+    end
+
+    def test_does_not_modify_single_percent_signs
+      assert_equal "some % value", to_android("some % value")
+    end
+
+    def test_does_not_modify_single_percent_signs_when_followed_by_space_and_format_letter
+      # Said differently: formartter parser should not recognize %a in "70% and"
+      assert_equal 'If 70% and 30% dog 80% end', to_android('If 70% and 30% dog 80% end')
+    end
+
+    def test_escapes_single_percent_signs_if_placeholder_present
+      assert_starts_with "some %% v", to_android("some % value #{placeholder}")
+    end
+
+    def test_does_not_modify_double_percent_signs
+      assert_equal "some %% value", to_android("some %% value")
+    end
+
+    def test_does_not_modify_double_percent_signs_if_placeholder_present
+      assert_starts_with "some %% v", to_android("some %% value #{placeholder}")
+    end
+
+    def test_does_not_modify_single_placeholder
+      input = "some #{placeholder} text"
+      assert_equal input, to_android(input)
+    end
+
+    def test_numbers_multiple_placeholders
+      assert_equal "first %1$d second %2$f", to_android("first %d second %f")
+    end
+
+    def test_does_not_modify_numbered_placeholders
+      input = "second %2$f first %1$d"
+      assert_equal input, to_android(input)
+    end
+
+    def test_raises_an_error_when_mixing_numbered_and_non_numbered_placeholders
+      assert_raises Twine::Error do
+        to_android("some %d second %2$f")
+      end
+    end
+
+    def test_complicated_float_placeholders
+      assert_equal "%1$.0f%2$s (apparent: %3$.0f)", to_android("%.0f%@ (apparent: %.0f)")
+    end
+  end
+
+  class FromAndroid < PlaceholderTest
+    def from_android(value)
+      Twine::Placeholders.convert_placeholders_from_android_to_twine(value)
+    end
+
+    def test_replaces_string_placeholder
+      assert_equal "some %@ value", from_android("some %s value")
+    end
+  end
+
+  class ToFlash < PlaceholderTest
+    def to_flash(value)
+      Twine::Placeholders.convert_placeholders_from_twine_to_flash(value)
+    end
+
+    def test_replaces_placeholder
+      assert_equal "some {0} text", to_flash("some #{placeholder} text")
+    end
+
+    def test_replaces_string_placeholder
+      assert_equal "some {0} text", to_flash("some #{placeholder('@')} text")
+    end
+
+    def test_numbers_placeholders
+      assert_equal "some {0} more {1} text {2}", to_flash("some #{placeholder('@')} more #{placeholder('@')} text #{placeholder('@')}")
+    end
+  end
+
+  class FromFlash < PlaceholderTest
+    def from_flash(value)
+      Twine::Placeholders.convert_placeholders_from_flash_to_twine(value)
+    end
+
+    def test_maps_all_placeholders_to_string
+      assert_equal "some %@ more %@ text %@", from_flash("some {0} more {1} text {2}")
+    end
+  end
+
+  class PythonPlaceholder < PlaceholderTest
+    def test_negative_for_regular_placeholders
+      assert_equal false, Twine::Placeholders.contains_python_specific_placeholder(placeholder)
+    end
+
+    def test_positive_for_named_placeholders
+      inputs = [
+        "%(language)s has",
+        "For %(number)03d quotes",
+        "bought on %(app_name)s"
+      ]
+      inputs.each do |input|
+        assert_equal true, Twine::Placeholders.contains_python_specific_placeholder(input)
+      end
+    end
+  end
+end

test/test_twine_definition.rb

@@ -0,0 +1,111 @@
+require 'twine_test'
+
+class TestTwineDefinition < TwineTest
+  class TestTags < TwineTest
+    def setup
+      super
+      @definition = Twine::TwineDefinition.new 'key'
+    end
+
+    def test_include_untagged
+      assert @definition.matches_tags?([[rand(100000).to_s]], true)
+    end
+
+    def test_matches_no_given_tags
+      assert @definition.matches_tags?([], false)
+    end
+
+    def test_matches_tag
+      @definition.tags = ['tag1']
+
+      assert @definition.matches_tags?([['tag1']], false)
+    end
+
+    def test_matches_any_tag
+      @definition.tags = ['tag1']
+
+      assert @definition.matches_tags?([['tag0', 'tag1', 'tag2']], false)
+    end
+
+    def test_matches_all_tags
+      @definition.tags = ['tag1', 'tag2']
+
+      assert @definition.matches_tags?([['tag1'], ['tag2']], false)
+    end
+
+    def test_does_not_match_all_tags
+      @definition.tags = ['tag1']
+
+      refute @definition.matches_tags?([['tag1'], ['tag2']], false)
+    end
+
+    def test_does_not_match_excluded_tag
+      @definition.tags = ['tag1']
+
+      refute @definition.matches_tags?([['~tag1']], false)
+    end
+
+    def test_matches_excluded_tag
+      @definition.tags = ['tag2']
+
+      assert @definition.matches_tags?([['~tag1']], false)
+    end
+
+    def test_complex_rules
+      @definition.tags = ['tag1', 'tag2', 'tag3']
+
+      assert @definition.matches_tags?([['tag1']], false)
+      assert @definition.matches_tags?([['tag1', 'tag4']], false)
+      assert @definition.matches_tags?([['tag1'], ['tag2'], ['tag3']], false)
+      refute @definition.matches_tags?([['tag1'], ['tag4']], false)
+
+      assert @definition.matches_tags?([['tag4', '~tag5']], false)
+    end
+  end
+
+  class TestReferences < TwineTest
+    def setup
+      super
+
+      @reference = Twine::TwineDefinition.new 'reference-key'
+      @reference.comment = 'reference comment'
+      @reference.tags = ['ref1']
+      @reference.translations['en'] = 'ref-value'
+
+      @definition = Twine::TwineDefinition.new 'key'
+      @definition.reference_key = @reference.key
+      @definition.reference = @reference
+    end
+
+    def test_reference_comment_used
+      assert_equal 'reference comment', @definition.comment
+    end
+
+    def test_reference_comment_override
+      @definition.comment = 'definition comment'
+
+      assert_equal 'definition comment', @definition.comment
+    end
+
+    def test_reference_tags_used
+      assert @definition.matches_tags?([['ref1']], false)
+    end
+
+    def test_reference_tags_override
+      @definition.tags = ['tag1']
+
+      refute @definition.matches_tags?([['ref1']], false)
+      assert @definition.matches_tags?([['tag1']], false)
+    end
+
+    def test_reference_translation_used
+      assert_equal 'ref-value', @definition.translation_for_lang('en')
+    end
+
+    def test_reference_translation_override
+      @definition.translations['en'] = 'value'
+
+      assert_equal 'value', @definition.translation_for_lang('en')
+    end
+  end
+end

test/test_twine_file.rb

@@ -0,0 +1,58 @@
+require 'twine_test'
+
+class TestTwineFile < TwineTest
+  class Reading < TwineTest
+    def setup
+      super
+
+      @twine_file = Twine::TwineFile.new
+      @twine_file.read fixture_path('twine_accent_values.txt')
+    end
+
+    def test_reading_keeps_leading_accent
+      assert_equal '`value', @twine_file.definitions_by_key['value_with_leading_accent'].translations['en']
+    end
+
+    def test_reading_keeps_trailing_accent
+      assert_equal 'value`', @twine_file.definitions_by_key['value_with_trailing_accent'].translations['en']
+    end
+
+    def test_reading_keeps_leading_space
+      assert_equal ' value', @twine_file.definitions_by_key['value_with_leading_space'].translations['en']
+    end
+
+    def test_reading_keeps_trailing_space
+      assert_equal 'value ', @twine_file.definitions_by_key['value_with_trailing_space'].translations['en']
+    end
+
+    def test_reading_keeps_wrapping_spaces
+      assert_equal ' value ', @twine_file.definitions_by_key['value_wrapped_by_spaces'].translations['en']
+    end
+
+    def test_reading_keeps_wrapping_accents
+      assert_equal '`value`', @twine_file.definitions_by_key['value_wrapped_by_accents'].translations['en']
+    end
+  end
+
+  class Writing < TwineTest
+
+    def test_accent_wrapping
+      @twine_file = build_twine_file 'en' do
+        add_section 'Section' do
+          add_definition value_with_leading_accent: '`value'
+          add_definition value_with_trailing_accent: 'value`'
+          add_definition value_with_leading_space: ' value'
+          add_definition value_with_trailing_space: 'value '
+          add_definition value_wrapped_by_spaces: ' value '
+          add_definition value_wrapped_by_accents: '`value`'
+        end
+      end
+
+      @twine_file.write @output_path
+
+      assert_equal content('twine_accent_values.txt'), File.read(@output_path)
+    end
+
+  end
+
+end

test/test_validate_twine_file.rb

@@ -0,0 +1,69 @@
+# encoding: utf-8
+
+require 'command_test'
+
+class TestValidateTwineFile < CommandTest
+  def setup
+    super
+    @options = { twine_file: 'input.txt' }
+
+    @twine_file = build_twine_file 'en' do
+      add_section 'Section 1' do
+        add_definition key1: 'value1', tags: ['tag1']
+        add_definition key2: 'value2', tags: ['tag1']
+      end
+
+      add_section 'Section 2' do
+        add_definition key3: 'value3', tags: ['tag1', 'tag2']
+        add_definition key4: 'value4', tags: ['tag2']
+      end
+    end
+  end
+
+  def random_definition
+    @twine_file.definitions_by_key[@twine_file.definitions_by_key.keys.sample]
+  end
+
+  def test_recognizes_valid_file
+    Twine::Runner.new(@options, @twine_file).validate_twine_file
+    assert_equal "input.txt is valid.\n", Twine::stdout.string
+  end
+
+  def test_reports_duplicate_keys
+    @twine_file.sections[0].definitions << random_definition
+
+    assert_raises Twine::Error do
+      Twine::Runner.new(@options, @twine_file).validate_twine_file
+    end
+  end
+
+  def test_reports_invalid_characters_in_keys
+    random_definition.key[0] = "!?;:,^`´'\"\\|/(){}[]~-+*=#$%".chars.to_a.sample
+
+    assert_raises Twine::Error do
+      Twine::Runner.new(@options, @twine_file).validate_twine_file
+    end
+  end
+
+  def test_does_not_reports_missing_tags_by_default
+    random_definition.tags.clear
+
+    Twine::Runner.new(@options, @twine_file).validate_twine_file
+  end
+
+  def test_reports_missing_tags
+    random_definition.tags.clear
+
+    assert_raises Twine::Error do
+      Twine::Runner.new(@options.merge(pedantic: true), @twine_file).validate_twine_file
+    end
+  end
+
+  def test_reports_python_specific_placeholders
+    random_definition.translations["en"] = "%(python_only)s"
+
+    assert_raises Twine::Error do
+      Twine::Runner.new(@options, @twine_file).validate_twine_file
+    end
+  end
+end

test/twine_file_dsl.rb

@@ -0,0 +1,46 @@
+module TwineFileDSL
+  def build_twine_file(*languages)
+    @currently_built_twine_file = Twine::TwineFile.new
+    @currently_built_twine_file.language_codes.concat languages
+    yield
+    result = @currently_built_twine_file
+    @currently_built_twine_file = nil
+    return result
+  end
+
+  def add_section(name)
+    return unless @currently_built_twine_file
+    @currently_built_twine_file_section = Twine::TwineSection.new name
+    @currently_built_twine_file.sections << @currently_built_twine_file_section
+    yield
+    @currently_built_twine_file_section = nil
+  end
+
+  def add_definition(parameters)
+    return unless @currently_built_twine_file
+    return unless @currently_built_twine_file_section
+
+    # this relies on Ruby preserving the order of hash elements
+    key, value = parameters.first
+    definition = Twine::TwineDefinition.new(key.to_s)
+    if value.is_a? Hash
+      value.each do |language, translation|
+        definition.translations[language.to_s] = translation
+      end
+    elsif !value.is_a? Symbol
+      language = @currently_built_twine_file.language_codes.first
+      definition.translations[language] = value
+    end
+
+    definition.comment = parameters[:comment] if parameters[:comment]
+    definition.tags = parameters[:tags] if parameters[:tags]
+    if parameters[:ref] || value.is_a?(Symbol)
+      reference_key = (parameters[:ref] || value).to_s
+      definition.reference_key = reference_key
+      definition.reference = @currently_built_twine_file.definitions_by_key[reference_key]
+    end
+
+    @currently_built_twine_file_section.definitions << definition
+    @currently_built_twine_file.definitions_by_key[definition.key] = definition
+  end
+end

test/twine_test.rb

@@ -1,138 +1,48 @@
 require 'erb'
-require 'rubygems'
-require 'test/unit'
+require 'minitest/autorun'
+require "mocha/minitest"
+require 'securerandom'
+require 'stringio'
 require 'twine'
+require 'twine_file_dsl'
 
-class TwineTest < Test::Unit::TestCase
-  def test_generate_string_file_1
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'fr.xml')
-      Twine::Runner.run(%W(generate-string-file test/fixtures/strings-1.txt #{output_path} --include-untranslated))
-      assert_equal(ERB.new(File.read('test/fixtures/test-output-1.txt')).result, File.read(output_path))
-    end
+class TwineTest < Minitest::Test
+  include TwineFileDSL
+
+  KNOWN_LANGUAGES = %w(en fr de es)
+  
+  def setup
+    super
+    Twine::stdout = StringIO.new
+    Twine::stderr = StringIO.new
+
+    @formatters = Twine::Formatters.formatters.dup
+
+    @output_dir = Dir.mktmpdir
+    @output_path = File.join @output_dir, SecureRandom.uuid
   end
 
-  def test_generate_string_file_2
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'en.strings')
-      Twine::Runner.run(%W(generate-string-file test/fixtures/strings-1.txt #{output_path} -t tag1))
-      assert_equal(ERB.new(File.read('test/fixtures/test-output-2.txt')).result, File.read(output_path))
-    end
+  def teardown
+    FileUtils.remove_entry_secure @output_dir if File.exist? @output_dir
+    Twine::Formatters.formatters.clear
+    Twine::Formatters.formatters.concat @formatters
+    super
   end
 
-  def test_generate_string_file_3
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'en.json')
-      Twine::Runner.run(%W(generate-string-file test/fixtures/strings-1.txt #{output_path} -t tag1))
-      assert_equal(ERB.new(File.read('test/fixtures/test-output-5.txt')).result, File.read(output_path))
-    end
+  def execute(command)
+    command += " -o #{@output_path}"
+    Twine::Runner.run(command.split(" "))
   end
 
-  def test_generate_string_file_4
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'en.strings')
-      Twine::Runner.run(%W(generate-string-file test/fixtures/strings-2.txt #{output_path} -t tag1))
-      assert_equal(ERB.new(File.read('test/fixtures/test-output-6.txt')).result, File.read(output_path))
-    end
+  def fixture_path(filename)
+    File.join File.dirname(__FILE__), 'fixtures', filename
   end
 
-  def test_generate_string_file_5
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'en.po')
-      Twine::Runner.run(%W(generate-string-file test/fixtures/strings-1.txt #{output_path} -t tag1))
-      assert_equal(ERB.new(File.read('test/fixtures/test-output-7.txt')).result, File.read(output_path))
-    end
+  def content(filename)
+    ERB.new(File.read fixture_path(filename)).result
   end
 
-  def test_generate_string_file_6
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'en.xml')
-      Twine::Runner.run(%W(generate-string-file test/fixtures/strings-3.txt #{output_path}))
-      assert_equal(ERB.new(File.read('test/fixtures/test-output-8.txt')).result, File.read(output_path))
-    end
-  end
-
-  def test_generate_string_file_7
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'en.xml')
-      Twine::Runner.run(%W(generate-string-file test/fixtures/strings-2.txt #{output_path} -t tag1))
-      assert_equal(ERB.new(File.read('test/fixtures/test-output-10.txt')).result, File.read(output_path))
-    end
-  end
-
-  def test_generate_string_file_8
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'fr.xml')
-      Twine::Runner.run(%W(generate-string-file --format tizen test/fixtures/strings-1.txt #{output_path} --include-untranslated))
-      assert_equal(ERB.new(File.read('test/fixtures/test-output-12.txt')).result, File.read(output_path))
-    end
-  end
-
-  def test_consume_string_file_1
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'strings.txt')
-      Twine::Runner.run(%W(consume-string-file test/fixtures/strings-1.txt test/fixtures/fr-1.xml -o #{output_path} -l fr))
-      assert_equal(File.read('test/fixtures/test-output-3.txt'), File.read(output_path))
-    end
-  end
-
-  def test_consume_string_file_2
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'strings.txt')
-      Twine::Runner.run(%W(consume-string-file test/fixtures/strings-1.txt test/fixtures/en-1.strings -o #{output_path} -l en -a))
-      assert_equal(File.read('test/fixtures/test-output-4.txt'), File.read(output_path))
-    end
-  end
-
-  def test_consume_string_file_3
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'strings.txt')
-      Twine::Runner.run(%W(consume-string-file test/fixtures/strings-1.txt test/fixtures/en-1.json -o #{output_path} -l en -a))
-      assert_equal(File.read('test/fixtures/test-output-4.txt'), File.read(output_path))
-    end
-  end
-
-  def test_consume_string_file_4
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'strings.txt')
-      Twine::Runner.run(%W(consume-string-file test/fixtures/strings-1.txt test/fixtures/en-1.po -o #{output_path} -l en -a))
-      assert_equal(File.read('test/fixtures/test-output-4.txt'), File.read(output_path))
-    end
-  end
-
-  def test_consume_string_file_5
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'strings.txt')
-      Twine::Runner.run(%W(consume-string-file test/fixtures/strings-1.txt test/fixtures/en-2.po -o #{output_path} -l en -a))
-      assert_equal(File.read('test/fixtures/test-output-9.txt'), File.read(output_path))
-    end
-  end
-
-  def test_consume_string_file_6
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'strings.txt')
-      Twine::Runner.run(%W(consume-string-file test/fixtures/strings-2.txt test/fixtures/en-3.xml -o #{output_path} -l en -a))
-      assert_equal(File.read('test/fixtures/test-output-11.txt'), File.read(output_path))
-    end
-  end
-
-  def test_generate_report_1
-    Twine::Runner.run(%w(generate-report test/fixtures/strings-1.txt))
-  end
-
-  def test_json_line_breaks_consume
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'strings.txt')
-      Twine::Runner.run(%W(consume-string-file test/fixtures/test-json-line-breaks/line-breaks.txt test/fixtures/test-json-line-breaks/line-breaks.json -l fr -o #{output_path}))
-      assert_equal(File.read('test/fixtures/test-json-line-breaks/consumed.txt'), File.read(output_path))
-    end
-  end
-
-  def test_json_line_breaks_generate
-    Dir.mktmpdir do |dir|
-      output_path = File.join(dir, 'en.json')
-      Twine::Runner.run(%W(generate-string-file test/fixtures/test-json-line-breaks/line-breaks.txt #{output_path}))
-      assert_equal(File.read('test/fixtures/test-json-line-breaks/generated.json'), File.read(output_path))
-    end
+  def content_io(filename)
+    StringIO.new ERB.new(File.read fixture_path(filename)).result
   end
 end

twine

@@ -1,3 +1,3 @@
 #!/bin/sh
 BASEDIR=$(dirname $0)
-ruby -rubygems -I $BASEDIR/lib $BASEDIR/bin/twine $@
+ruby -rrubygems -I $BASEDIR/lib $BASEDIR/bin/twine "$@"

twine.gemspec

@@ -5,22 +5,27 @@ Gem::Specification.new do |s|
   s.name         = "twine"
   s.version      = Twine::VERSION
   s.date         = Time.now.strftime('%Y-%m-%d')
-  s.summary      = "Manage strings and their translations for your iOS and Android projects."
-  s.homepage     = "https://github.com/mobiata/twine"
+  s.summary      = "Manage strings and their translations for your iOS, Android and other projects."
+  s.homepage     = "https://github.com/scelis/twine"
   s.email        = "twine@mobiata.com"
   s.authors      = [ "Sebastian Celis" ]
   s.has_rdoc     = false
+  s.license      = "BSD-3-Clause"
 
   s.files        = %w( Gemfile README.md LICENSE )
   s.files       += Dir.glob("lib/**/*")
   s.files       += Dir.glob("bin/**/*")
   s.files       += Dir.glob("test/**/*")
-  s.test_file    = 'test/twine_test.rb'
+  s.test_files   = Dir.glob("test/test_*")
 
-  s.required_ruby_version = ">= 1.8.7"
-  s.add_runtime_dependency('rubyzip', "~> 0.9.5")
-  s.add_runtime_dependency('safe_yaml', "~> 1.0.3")
-  s.add_development_dependency('rake', "~> 0.9.2")
+  s.required_ruby_version = ">= 2.6"
+  s.add_runtime_dependency('rexml', "~> 3.2")
+  s.add_runtime_dependency('rubyzip', "~> 2.0")
+  s.add_runtime_dependency('safe_yaml', "~> 1.0")
+  s.add_development_dependency('rake', "~> 13.0")
+  s.add_development_dependency('minitest', "~> 5.5")
+  s.add_development_dependency('minitest-ci', "~> 3.0")
+  s.add_development_dependency('mocha', "~> 1.1")
 
   s.executables  = %w( twine )
   s.description  = <<desc