use unix lineendings even on windows

docs/_summary.md

@@ -22,6 +22,7 @@
   * QMK API
     * [Overview](api_overview.md)
     * [API Documentation](api_docs.md)
+    * [KLE To info.json](kle2json_guide.md)
     * [Keyboard Support](reference_configurator_support.md)
     * [Adding Default Keymaps](configurator_default_keymaps.md)
 

docs/cli_commands.md

@@ -308,24 +308,32 @@ qmk generate-rgb-breathe-table [-q] [-o OUTPUT] [-m MAX] [-c CENTER]
 
 ## `qmk kle2json`
 
-This command allows you to convert from raw KLE data to QMK Configurator JSON. It accepts either an absolute file path, or a file name in the current directory. By default it will not overwrite `info.json` if it is already present. Use the `-f` or `--force` flag to overwrite.
+This command allows you to convert [Keyboard-Layout-Editor.com](http://keyboard-layout-editor.com) layouts into `info.json` layouts. It will also create a `keymap.json` file for your layout. This saves a lot of time when setting up a new keyboard.
+
+To use this command your KLE will need to follow a specific format. See [KLE To info.json](kle2json_guide.md) for more details.
 
 **Usage**:
 
 ```
-qmk kle2json [-f] <filename>
+qmk kle2json -kb <keyboard> [-km KEYMAP] [-l LAYOUT] <filename-or-kle-id>
 ```
 
 **Examples**:
 
-```
-$ qmk kle2json kle.txt 
-☒ File info.json already exists, use -f or --force to overwrite.
-```
+With only a KLE id:
 
 ```
-$ qmk kle2json -f kle.txt -f
-Ψ Wrote out to info.json
+$ qmk kle2json -kb clueboard/new60 70aaa4bed76d0b2f67fd165641239552
+Ψ Wrote file keyboards/clueboard/new60/info.json
+Ψ Wrote file keyboards/clueboard/new60/keymaps/default/keymap.json
+```
+
+With a full URL:
+
+```
+$ qmk kle2json -kb clueboard/new60 'http://www.keyboard-layout-editor.com/#/gists/70aaa4bed76d0b2f67fd165641239552'
+Ψ Wrote file keyboards/clueboard/new60/info.json
+Ψ Wrote file keyboards/clueboard/new60/keymaps/default/keymap.json
 ```
 
 ## `qmk pyformat`

docs/kle2json_guide.md

@@ -0,0 +1,62 @@
+# KLE To info.json
+
+This page describes how to create a [Keyboard Layout Editor](http://keyboard-layout-editor.com) layout that works with `qmk kle2json`. You will also be able to use the same KLE layout to create VIA support for your keyboard.
+
+## Overview
+
+QMK uses `info.json` to store metadata about the keyboard including the matrix pins, row and column configuration, and keyboard layout. You can encode some of this data into a KLE layout and then convert that to `info.json`, which saves you time and headaches creating files from scratch.
+
+Our reference layout is here: <http://www.keyboard-layout-editor.com/#/gists/70aaa4bed76d0b2f67fd165641239552>
+
+Keep that open in a separate window so that you can refer to it while reading this document.
+
+## Layout Structure
+
+The first thing you should notice is that the main layout for the keyboard is in the middle. This is the default layout. Around the sides you'll find layout options. These are groups of keys that can replace specific keys in the default layout. More on that later.
+
+## Anatomy of a Key
+
+Every key in KLE has 12 labels- top left, top center, top right, center left, center, center right, bottom left, bottom center, bottom right, front left, front center, front right. We do not use all 12 labels, but we do some of them.
+
+The labels we use are below:
+
+| | | |
+|-|-|-|
+| **Matrix** | (unused) | (unused) |
+| **Keycode** | (unused) | (unused) |
+| (unused) | (unused) | **Option Group** |
+| **Option Name** | (unused) | (unused) |
+
+### Matrix
+
+This is the location of the key in the keyboard matrix. It should be two whole numbers separated by a comma representing row and column. For example, the top left location of the keyboard is usually `0,0`, to the right of that key is `0,1`, below that key is `1,0`, and so on.
+
+### Keycode
+
+This is the default keycode for layer 0. You should supply this now even if you won't use the generated keymap because QMK will make greater use of it in the future.
+
+### Option Group
+
+This identifies what Layout Options (if any) this key is part of. If this value is empty the key is not part of any Layout Options.
+
+The value is two whole numbers separated by a comma. The first number is the Layout Option Group Number, which identifies the specific Layout Option this key is associated with. The second number is the specific Layout Option Choice this key is associated with.
+
+### Option Name
+
+This identifies the name for each Option Group Choice. This will be used in the future to automatically build layouts from each option. All keys with the same Option Group identifier should have the same Option Name.
+
+## Running qmk kle2json
+
+Once you have built your KLE layout you are ready to convert it to `info.json`. Start by clicking `Sign In with GitHub` on Keyboard Layout Editor. Once you have signed in you can click `Save` to save your KLE. Once it's saved grab the URL from the bar, it should end end in `/#/gists/<long_number>`.
+
+To use the reference layout as an example, I could update the keyboard `clueboard/60` from that KLE using this command:
+
+```shell
+qmk kle2json -kb clueboard/60 -l LAYOUT_60_ansi 'http://www.keyboard-layout-editor.com/#/gists/70aaa4bed76d0b2f67fd165641239552'
+```
+
+It will fetch the KLE, generate a new `LAYOUT_60_ansi`, and either add a new layout or change the existing layout to match.
+
+## Adding more layouts
+
+At the current time QMK only supports the base layout. Layout Options are allowed for VIA compatibility purposes and will be implemented in QMK at a later date. For now we recommend you create a KLE for each layout you want to support in QMK and call `qmk kle2json` once per layout to set up the info.json.

lib/python/qmk/cli/chibios/confmigrate.py

@@ -145,17 +145,17 @@ def chibios_confmigrate(cli):
         if "CHCONF_H" in input_defs["dict"] or "_CHCONF_H_" in input_defs["dict"]:
             migrate_chconf_h(to_override, outfile=sys.stdout)
             if cli.args.overwrite:
-                with open(cli.args.input, "w") as out_file:
+                with open(cli.args.input, "w", newline='\n') as out_file:
                     migrate_chconf_h(to_override, outfile=out_file)
 
         elif "HALCONF_H" in input_defs["dict"] or "_HALCONF_H_" in input_defs["dict"]:
             migrate_halconf_h(to_override, outfile=sys.stdout)
             if cli.args.overwrite:
-                with open(cli.args.input, "w") as out_file:
+                with open(cli.args.input, "w", newline='\n') as out_file:
                     migrate_halconf_h(to_override, outfile=out_file)
 
         elif "MCUCONF_H" in input_defs["dict"] or "_MCUCONF_H_" in input_defs["dict"]:
             migrate_mcuconf_h(to_override, outfile=sys.stdout)
             if cli.args.overwrite:
-                with open(cli.args.input, "w") as out_file:
+                with open(cli.args.input, "w", newline='\n') as out_file:
                     migrate_mcuconf_h(to_override, outfile=out_file)

lib/python/qmk/cli/kle2json.py

@@ -4,34 +4,121 @@ import json
 import os
 from pathlib import Path
 
+import requests
 from milc import cli
 from kle2xy import KLE2xy
 
+import qmk.path
 from qmk.converter import kle2qmk
+from qmk.info import info_json
 from qmk.info_json_encoder import InfoJSONEncoder
 
 
-@cli.argument('filename', help='The KLE raw txt to convert')
-@cli.argument('-f', '--force', action='store_true', help='Flag to overwrite current info.json')
-@cli.subcommand('Convert a KLE layout to a Configurator JSON', hidden=False if cli.config.user.developer else True)
+def fetch_json(url):
+    """Gets the JSON from a url.
+    """
+    response = fetch_url(url)
+
+    if response.status_code == 200:
+        return response.json()
+
+    print(f'ERROR: {url} returned {response.status_code}: {response.text}')
+    return {}
+
+
+def fetch_url(url):
+    """Fetch a URL.
+    """
+    response = requests.get(url, timeout=30)
+    response.encoding = 'utf-8-sig'
+
+    return response
+
+
+def fetch_gist(id):
+    """Retrieve a gist from gist.github.com
+    """
+    url = f'https://api.github.com/gists/{id}'
+    gist = fetch_json(url)
+
+    for data in gist['files'].values():
+        if data['filename'].endswith('kbd.json'):
+            if data.get('truncated'):
+                return fetch_url(data['raw_url']).text
+            else:
+                return data['content']
+
+    return None
+
+
+def fetch_kle(id):
+    """Fetch the kle data from a gist ID.
+    """
+    gist = fetch_gist(id)
+
+    return gist[1:-1]
+
+
+@cli.argument('kle', arg_only=True, help='A file or KLE id to convert')
+@cli.argument('-l', '--layout', arg_only=True, default='LAYOUT', help='The LAYOUT name this KLE represents')
+@cli.argument('-kb', '--keyboard', arg_only=True, required=True, help='The folder name for the keyboard')
+@cli.argument('-km', '--keymap', arg_only=True, default='default', help='The name of the keymap to write (Default: default)')
+@cli.subcommand('Use a KLE layout to build info.json and a keymap', hidden=False if cli.config.user.developer else True)
 def kle2json(cli):
     """Convert a KLE layout to QMK's layout format.
-    """  # If filename is a path
-    if cli.args.filename.startswith("/") or cli.args.filename.startswith("./"):
-        file_path = Path(cli.args.filename)
-    # Otherwise assume it is a file name
+    """
+    file_path = Path(os.environ['ORIG_CWD'], cli.args.kle)
+
+    # Find our KLE text
+    if cli.args.kle.startswith('http') and '#' in cli.args.kle:
+        kle_path = cli.args.kle.split('#', 1)[1]
+        if 'gists' not in kle_path:
+            cli.log.error('Invalid KLE url: {fg_cyan}%s', cli.args.kle)
+            return False
+        else:
+            raw_code = fetch_kle(kle_path.split('/')[-1])
+
+    elif file_path.exists():
+        raw_code = file_path.open().read()
+
     else:
-        file_path = Path(os.environ['ORIG_CWD'], cli.args.filename)
-    # Check for valid file_path for more graceful failure
-    if not file_path.exists():
-        cli.log.error('File {fg_cyan}%s{style_reset_all} was not found.', file_path)
-        return False
-    out_path = file_path.parent
-    raw_code = file_path.open().read()
-    # Check if info.json exists, allow overwrite with force
-    if Path(out_path, "info.json").exists() and not cli.args.force:
-        cli.log.error('File {fg_cyan}%s/info.json{style_reset_all} already exists, use -f or --force to overwrite.', out_path)
+        raw_code = fetch_kle(cli.args.kle)
+        if not raw_code:
+            cli.log.error('File {fg_cyan}%s{style_reset_all} was not found.', file_path)
+            return False
+
+    # Make sure the user supplied a keyboard
+    if not cli.args.keyboard:
+        cli.log.error('You must pass --keyboard or be in a keyboard directory!')
+        cli.print_usage()
         return False
+
+    # Check for an existing info.json
+    if qmk.path.is_keyboard(cli.args.keyboard):
+        kb_info_json = info_json(cli.args.keyboard)
+    else:
+        kb_info_json = {
+            "keyboard_name": cli.args.keyboard,
+            "maintainer": "",
+            "features": {
+                "console": True,
+                "extrakey": True,
+                "mousekey": True,
+                "nkro": True
+            },
+            "matrix_pins": {
+                "cols": [],
+                "rows": [],
+            },
+            "usb": {
+              "device_ver": "0x0001",
+              "pid": '0x0000',
+              "vid": '0x03A8',
+            },
+            "layouts": {},
+        }
+
+    # Build and merge in the new layout
     try:
         # Convert KLE raw to x/y coordinates (using kle2xy package from skullydazed)
         kle = KLE2xy(raw_code)
@@ -39,22 +126,44 @@ def kle2json(cli):
         cli.log.error('Could not parse KLE raw data: %s', raw_code)
         cli.log.exception(e)
         return False
-    keyboard = {
-        'keyboard_name': kle.name,
-        'url': '',
-        'maintainer': 'qmk',
-        'width': kle.columns,
-        'height': kle.rows,
-        'layouts': {
-            'LAYOUT': {
-                'layout': kle2qmk(kle)
-            }
-        },
-    }
+
+    if 'layouts' not in kb_info_json:
+        kb_info_json['layouts'] = {}
+
+    if cli.args.layout not in kb_info_json['layouts']:
+        kb_info_json['layouts'][cli.args.layout] = {}
+
+    kb_info_json['layouts'][cli.args.layout]['layout'] = kle2qmk(kle)
 
     # Write our info.json
-    keyboard = json.dumps(keyboard, indent=4, separators=(', ', ': '), sort_keys=False, cls=InfoJSONEncoder)
-    info_json_file = out_path / 'info.json'
+    keyboard_dir = qmk.path.keyboard(cli.args.keyboard)
+    keyboard_dir.mkdir(exist_ok=True, parents=True)
+    info_json_file = keyboard_dir / 'info.json'
 
-    info_json_file.write_text(keyboard)
-    cli.log.info('Wrote out {fg_cyan}%s/info.json', out_path)
+    json.dump(kb_info_json, info_json_file.open('w', newline='\n'), indent=4, separators=(', ', ': '), sort_keys=False, cls=InfoJSONEncoder)
+    cli.log.info('Wrote file {fg_cyan}%s', info_json_file)
+
+    # Generate and write a keymap
+    keymap_path = keyboard_dir / 'keymaps' / cli.args.keymap
+    keymap_file = keymap_path / 'keymap.json'
+
+    if keymap_path.exists():
+        cli.log.warning('{fg_cyan}%s{fg_reset} already exists, not generating a keymap.', keymap_path)
+    else:
+        keymap = [key.get('label', 'KC_NO') for key in kb_info_json['layouts'][cli.args.layout]['layout']]
+        keymap_json = {
+                'version': 1,
+                'documentation': "This file is a QMK Keymap. You can compile it with `qmk compile` or import it at <https://config.qmk.fm>. It can also be used directly with QMK's source code.",
+                'author': '',
+                'keyboard': kb_info_json['keyboard_name'],
+                'keymap': cli.args.keymap,
+                'layout': cli.args.layout,
+                'layers': [
+                    keymap,
+                    ['KC_TRNS' for key in keymap],
+                ],
+        }
+        keymap_path.mkdir(exist_ok=True, parents=True)
+
+        json.dump(keymap_json, keymap_file.open('w', newline='\n'), indent=4, separators=(', ', ': '), sort_keys=False)
+        cli.log.info('Wrote file %s', keymap_file)

lib/python/qmk/converter.py

@@ -1,33 +1,64 @@
 """Functions to convert to and from QMK formats
 """
-from collections import OrderedDict
+from milc import cli
 
 
 def kle2qmk(kle):
     """Convert a KLE layout to QMK's layout format.
     """
     layout = []
+    top_left_corner = None
 
+    # Iterate through the KLE classifying keys by layout
     for row in kle:
         for key in row:
             if key['decal']:
                 continue
 
-            qmk_key = OrderedDict(
-                label="",
-                x=key['column'],
-                y=key['row'],
-            )
+            if key['label_style'] in [0, 4]:
+                key_name = key['name'].split('\n')
+                if len(key_name) == 7:
+                    matrix, _, _, alt_layout, layout_name, _, keycode = key_name
+                elif len(key_name) == 5:
+                    matrix, _, _, alt_layout, layout_name = key_name
+                    cli.log.warning('Missing Keycode for key at matrix %s layout %s.', matrix, alt_layout)
+                else:
+                    cli.log.error('Unknown label format: %s', repr(key['name']))
+                    continue
+            else:
+                cli.log.error('Unknown label style: %s', key['label_style'])
+                continue
 
+            matrix = list(map(int, matrix.split(',', 1)))
+
+            qmk_key = {
+                'label': keycode,
+                'x': key['column'],
+                'y': key['row'],
+                'matrix': matrix,
+            }
+
+            if not top_left_corner and (not alt_layout or alt_layout.endswith(',0')):
+                top_left_corner = key['column'], key['row']
+
+            # Figure out what layout this key is part of
+            # FIXME(skullydazed): In the future this will populate `layout_options` in info.json
+            if alt_layout:
+                alt_group, layout_index = map(int, alt_layout.split(',', 1))
+                if layout_index != 0:
+                    continue
+
+            # Set the key size
             if key['width'] != 1:
                 qmk_key['w'] = key['width']
             if key['height'] != 1:
                 qmk_key['h'] = key['height']
-            if 'name' in key and key['name']:
-                qmk_key['label'] = key['name'].split('\n', 1)[0]
-            else:
-                del (qmk_key['label'])
 
             layout.append(qmk_key)
 
+    # Adjust the keys to account for the top-left corner
+    for key in layout:
+        key['x'] -= top_left_corner[0]
+        key['y'] -= top_left_corner[1]
+
     return layout

lib/python/qmk/info_json_encoder.py

@@ -17,6 +17,17 @@ class InfoJSONEncoder(json.JSONEncoder):
         if not self.indent:
             self.indent = 4
 
+    def default(self, obj):
+        """Fix certain objects that don't encode.
+        """
+        if isinstance(obj, Decimal):
+            if obj == int(obj):
+                return int(obj)
+
+            return float(obj)
+
+        return json.JSONEncoder.default(self, obj)
+
     def encode(self, obj):
         """Encode JSON objects for QMK.
         """

requirements.txt

@@ -5,3 +5,4 @@ colorama
 hjson
 milc
 pygments
+requests