1.1.2

A fix for a security vulnerability reported by Maximilian Hils.

.gitignore

@@ -1,4 +1,27 @@
 *~
-*.js
+lib/*.map
+lib/api-functions.js
+lib/auth.js
+lib/common.js
+lib/config.js
+lib/cwmp.js
+lib/db.js
+lib/fs.js
+lib/nbi.js
+lib/query.js
+lib/server.js
+lib/session.js
 node_modules
-debug/*
+debug
+config/auth.js
+config/config.json
+config/cwmp.key
+config/cwmp.crt
+config/cwmp.ca-bundle
+config/nbi.key
+config/nbi.crt
+config/nbi.ca-bundle
+config/fs.key
+config/fs.crt
+config/fs.ca-bundle
+config/ext

.npmignore

@@ -1,14 +1,21 @@
 *~
 .npmignore
 
-config/*
-!config/*-sample.*
-!config/custom_commands
+config/auth.js
+config/config.json
+config/cwmp.key
+config/cwmp.crt
+config/cwmp.ca-bundle
+config/nbi.key
+config/nbi.crt
+config/nbi.ca-bundle
+config/fs.key
+config/fs.crt
+config/fs.ca-bundle
+config/ext
 
 debug/*
 !debug/.gitkeep
 
 lib/*.coffee
-tools/*.coffee
-test/*.coffee
-
+lib/*.map

CHANGELOG.md

@@ -0,0 +1,91 @@
+# Change Log
+
+## 1.1.2 (2018-02-24)
+
+- A large number of bug fixes as well as stability and performance improvements.
+- Three security vulnerabilities disclosed by Maximilian Hils have been patched.
+- New config option UDP_CONNECTION_REQUEST_PORT to specify binding port for UDP
+connection requests.
+- New config option DATETIME_MILLISECONDS to strip milliseconds from dateTime
+values.
+- New config option BOOLEAN_LITERAL to use 1/0 or true/false for boolean values.
+- Parameter values that cannot be parsed according to the reported type now show
+a warning message.
+- Virtual parameter scripts now use the variable 'args' instead of the special
+'TIMESTAMPS' and 'VALUES' variables. The content of the args array is: {declare
+timestamps}, {declare values}, {current timestamps}, {current values}.
+- Virtual parameter value types are now inferred from the JavaScript type if the
+returned value attribute is not a value-type pair.
+- Show a fault when a virtual parameter script doesn't return the required
+attributes.
+- Redis is now optional (and disabled by default), reducing the complexity of
+scalable deployments.
+- Better detection of cyclical presets resulting in fewer faults for complex
+provisioning scripts.
+- Math.random() is now deterministic on per-device basis. A function has been
+added to allow specifying a seed value (e.g. Math.random.seed(Date.now())).
+- Overload spikes are now handled gracefully by refusing to accept new sessions
+temporarily when under abnormal load.
+- Added log messages for session timeouts, connection drops, and XML parsing
+errors.
+- Date.now() now takes an optional argument to specify "time steps" (in
+milliseconds). This can be used to ensure a group of parameters are all
+refreshed at the same time intervals.
+- Only the non-default configuration options are now logged at process start.
+- Faults caused by errors from extensions now show a cleaner stack trace.
+- Exit main process if there are too many worker crashes (e.g. when DB is down).
+- Updated dependencies and included a lockfile to ensure installations get the
+exact dependencies it was tested against.
+
+## 1.1.1 (2017-03-23)
+
+- Avoid crashing when connection request credentials are missing.
+- Show a warning instead of crashing when failing to parse parameter values
+according to the expected value type.
+- Add missing "Registered" event.
+- Fix bug where in certain cases many more instances than declared are created.
+- Fix parameter discovery bug when declared path timestamp is 1 or is not set.
+- Fix preset precondition failing when testing against datetime parameters and
+certain other parameters like \_deviceId.\_ProductClass.
+
+## 1.1.0 (2017-03-10)
+
+- Provisions enable implementing dynamic device configuration or complex device
+provisioning work flow using arbitrary scripts.
+- Virtual parameters are user-defined parameters whose values are evaluated from
+a custom script.
+- Extensions are sandboxed Node.js scripts that are accessible from provision
+and virtual parameter scripts to facilitate integration with external entities.
+- Support for UDP/STUN based connection requests for reaching devices behind
+NAT (TR-069 Annex G).
+- Presets can now be scheduled using a cron-like expression.
+- Presets can now be tied to specific device events (e.g. boot).
+- Presets precondition queries no longer support "$or" or other MongoDB logical
+operators.
+- Faults are no longer a part of tasks but are now first class objects.
+- Presets are now assigned to channels. A fault in one channel only blocks
+presets in that channel.
+- New API CRUD functions for provisions, virtual parameters, and faults.
+- New config options for XML output.
+- API responses now include "GenieACS-Version" header.
+- Graceful shutdown when receiving SIGINT and SIGTERM events.
+- Support SSL intermediate certificate chains.
+- Supported Node.js versions are 6.x and 7.x.
+- Supported MongoDB versions are 2.6 through 3.4.
+- Expect performance differences due to major under the hood changes. Some
+operations are faster and some are slower. Overall performance is improved.
+- GenieACS will no longer fetch the entire device data model upon first contact
+but will instead only fetch the parameters it needs to fulfill the presets.
+- Logs have been overhauled and split into two streams: process log (stderr) and
+access log (stdout). Also added config options to dump logs to files rather than
+standard streams.
+- Connection request authentication credentials are picked up from the device
+data model if available. config/auth.js is still supported as a fallback and now
+supports an optional callback argument.
+- Custom commands have been removed. Use virtual parameters and/or extensions.
+- Aliases and value normalizers (config/parameters.json) have been removed. Use
+virtual parameters.
+- The API /devices/<device_id>/preset has been removed.
+- Rarely used RequestDownload method no longer supported.
+- The TR-069 client simulator has moved to its own repo at
+https://github.com/zaidka/genieacs-sim

CONTRIBUTING.md

@@ -0,0 +1,39 @@
+# Have a question?
+
+Please ask in the [Users mailing list][mailing-lists] instead of opening a
+Github Issue. The Github issue tracker is strictly for bug reports.
+
+# Issues
+
+We use GitHub issues to track bugs. Please ensure your bug description is
+clear and has sufficient instructions to be able to reproduce the issue.
+
+If you face interoperability issues with your device, more often than not it's
+a device related issue. Please consult the [Users mailing list][mailing-lists]
+before submitting an Issue.
+
+# Pull Requests
+
+Pull requests are welcome. If you have a fix for a bug, go ahead and create a
+pull request. For other kinds of enhancements, please discuss in the [Users
+mailing list][mailing-lists] first.
+
+Please follow the guidelines [here][commit-guidelines] when committing your
+changes.
+
+## Contributor License Agreement
+
+By submitting a pull request to this repository, you acknowledge that, while
+maintaining copyright, you grant Zaid Abdulla a perpetual, worldwide, non-
+exclusive, no-charge, royalty-free, irrevocable license to reproduce, prepare
+derivative works of, publicly display, publicly perform, sublicense, and
+distribute your contributions and such derivative works under the AGPLv3
+license or any other license terms, including, but not limited to, proprietary
+or commercial license terms.
+
+You confirm that you own or have rights to distribute and sublicense the
+source code contained therein, and that your content does not infringe upon
+the intellectual property rights of a third party.
+
+[commit-guidelines]: http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html
+[mailing-lists]: http://lists.genieacs.com

README.md

@@ -1,65 +1,80 @@
 # GenieACS
 
-GenieACS is a *blazing fast* TR-069 auto configuration server (ACS) built with Node.js, Redis, and MongoDB. It's technology neutral and configurable to fit any service provider needs. This is the core back end component. Official front end GUI is available at https://github.com/zaidka/genieacs-gui.
+GenieACS is a fast and lightweight TR-069 auto configuration server (ACS). This
+is the core back end component. A GUI front end is available at
+https://github.com/genieacs/genieacs-gui.
 
 ## Features
 
-* **Massively concurrent**: Can handle hundreds of thousands of connected devices on a single server even with low inform interval.
-* **Preset-based configuration**: Define sets of configurations that devices will pick up and apply as needed based on given preconditions.
-* **Tagging**: Use tags to group devices for more manageable presets.
-* **Searching**: Query for devices on any parameter (supports all common operators including regular expressions).
-* **Parameter aliasing**: Define aliases to unify parameter paths from different types of devices. Aliases behave like normal parameters (i.e. can be queried or modified).
-* **Extensive API**: A simple yet rich HTTP-based API allows easy integration with other system.
+- **Battle-tested at scale**: Can scale to manage hundreds of thousands and
+potentially millions of devices
+- **Powerful provisioning system**: Scriptable configuration to handle complex
+device provisioning scenarios and other automated operations such as firmware
+upgrade
+- **Extensive API**: A rich HTTP-based API allows easy integration with other
+systems
 
-## Getting started
+## Quick Start
 
-Install [Node.js](http://nodejs.org/), [Redis](http://redis.io/), and [MongoDB](http://www.mongodb.org/). Refer to the corresponding documentation for installation guides. Then use NPM to install GenieACS by typing:
+Install [Node.js](http://nodejs.org/) and [MongoDB](http://www.mongodb.org/).
+Refer to their corresponding documentation for installation instructions. The
+supported versions are:
+
+- Node.js: 6.x and 8.x (8.x recommended)
+- MongoDB: 2.6 through 3.4
+
+Install build tools and libxml2 development files from your system's package
+manager.
+
+Then install GenieACS using NPM:
 
     npm install -g genieacs
 
-You may need to modify the configuration files under "config" directory (in /lib/node_modules/genieacs/config) depending on your setup.
-
 Alternatively, you can install from source by cloning the git repository:
 
-    cd /opt
-    git clone https://github.com/zaidka/genieacs.git --branch v1.0
+    git clone https://github.com/genieacs/genieacs.git
     cd genieacs
+    git checkout $(git tag -l v1.1.* --sort=-v:refname | head -n 1)
     npm install
-    npm run configure
     npm run compile
 
-Finally, run the following in GNU Screen session or something similar:
+Before proceeding, find and review the file "config.json" in "config" directory
+where GenieACS is downloaded.
+
+Finally, run the following (from bin directory if installing from source):
 
     genieacs-cwmp
 
-This is the service that the CPEs will communicate with. It listens to port 7547 by default (see config/config.json). Configure the ACS URL of your devices accordingly.
+This is the service that the CPEs will communicate with. It listens to port 7547
+by default (see config/config.json). Configure the ACS URL of your devices
+accordingly.
 
     genieacs-nbi
 
-This is the northbound interface module. It exposes a REST API on port 7557 by default. This is needed for the GUI front end to communicate with.
+This is the northbound interface module. It exposes a REST API on port 7557 by
+default. This must be running for the GUI front end to work.
 
     genieacs-fs
 
-This is the file server from which the CPEs will download firmware images and other types of files.
+This is the file server from which the CPEs will download firmware images and
+such.
 
-You can use stream redirection to output to log files:
+Note: For production deployment make sure to run those as background services.
 
-    genieacs-cwmp >> /var/log/genieacs-cwmp.log 2>> /var/log/genieacs-cwmp-err.log
-    genieacs-nbi >> /var/log/genieacs-nbi.log 2>> /var/log/genieacs-nbi-err.log
-    genieacs-fs >> /var/log/genieacs-fs.log 2>> /var/log/genieacs-fs-err.log
-
-For further details about installation and configuration, refer to the [wiki section](https://github.com/zaidka/genieacs/wiki).
-
-You may now proceed with installing [GenieACS GUI front end](https://github.com/zaidka/genieacs-gui).
+For further details about installation and configuration, refer to the
+[wiki section](https://github.com/genieacs/genieacs/wiki).
 
 ## Support
 
-The [Users mailing list](http://lists.genieacs.com) is a good place to get guidance and help from the community. Head on over and join the conversation! In addition, the [wiki](https://github.com/zaidka/genieacs/wiki) provides useful documentation and tips from GenieACS users.
+The [Users mailing list](http://lists.genieacs.com) is a good place to get
+guidance and help from the community. Head on over and join the conversation!
+In addition, the [wiki](https://github.com/genieacs/genieacs/wiki) provides useful
+documentation and tips from GenieACS users.
 
-You may submit bug reports or feature requests [here](https://github.com/zaidka/genieacs/issues). For device interoperability issues, please consult the mailing list first — it's likely that a workaround already exists.
+For commercial support options and professional services, please visit
+[genieacs.com](https://genieacs.com).
 
-For commercial support options and professional services, please visit [genieacs.com](https://genieacs.com).
+## License
 
-## Contributing
-
-Contributions are welcome. Fork this repo and open a pull request and wait for feedback. You can also contribute by enhancing the documentation in the [wiki section](https://github.com/zaidka/genieacs/wiki).
+Copyright 2013-2018 GenieACS Inc. GenieACS is released under the
+[AGPLv3 license terms](https://raw.githubusercontent.com/genieacs/genieacs/master/LICENSE).

bin/genieacs-cwmp

@@ -5,5 +5,5 @@ var fs = require('fs');
 var dir = path.resolve(path.dirname(fs.realpathSync(__filename)), '..');
 process.chdir(dir);
 
-cluster = require(path.resolve(dir, 'lib/cluster'));
+var cluster = require(path.resolve(dir, 'lib/cluster'));
 cluster.start('cwmp');

bin/genieacs-fs

@@ -5,5 +5,5 @@ var fs = require('fs');
 var dir = path.resolve(path.dirname(fs.realpathSync(__filename)), '..');
 process.chdir(dir);
 
-cluster = require(path.resolve(dir, 'lib/cluster'));
+var cluster = require(path.resolve(dir, 'lib/cluster'));
 cluster.start('fs');

bin/genieacs-nbi

@@ -5,5 +5,5 @@ var fs = require('fs');
 var dir = path.resolve(path.dirname(fs.realpathSync(__filename)), '..');
 process.chdir(dir);
 
-cluster = require(path.resolve(dir, 'lib/cluster'));
+var cluster = require(path.resolve(dir, 'lib/cluster'));
 cluster.start('nbi');

config/auth-sample.js

@@ -1,4 +1,7 @@
-exports.connectionRequest = function(deviceId) {
-  // return username/password pair for a given device
-  return ["", ""];
+"use strict";
+
+function connectionRequest(deviceId, url, username, password, callback) {
+  return callback(username || deviceId, password || "");
 }
+
+exports.connectionRequest = connectionRequest;

config/config-sample.json

@@ -1,7 +1,5 @@
 {
   "MONGODB_CONNECTION_URL" : "mongodb://127.0.0.1/genieacs",
-  "REDIS_PORT" : "6379",
-  "REDIS_HOST" : "127.0.0.1",
   "CWMP_INTERFACE" : "0.0.0.0",
   "CWMP_PORT" : 7547,
   "CWMP_SSL" : false,
@@ -9,7 +7,6 @@
   "NBI_PORT" : 7557,
   "FS_INTERFACE" : "0.0.0.0",
   "FS_PORT" : 7567,
-  "FS_IP" : "192.168.0.1",
-  "LOG_INFORMS" : true,
+  "FS_HOSTNAME" : "acs.example.com",
   "DEBUG" : false
 }

config/custom_commands-sample.json

@@ -1,4 +0,0 @@
-{
-  "seowon_frequencies" : "/^[0-9]+\\-SWU\\%2D9200\\-[0-9A-Z]+$/",
-  "greenpacket_frequencies" : "/^[0-9A-Z]+\\-DX350\\-[0-9A-Z]+$/"
-}

config/custom_commands/greenpacket_frequencies.js

@@ -1,210 +0,0 @@
-/*
- * Copyright 2013 Fanoos Telecom
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-var net = require('net');
-var PORT = 23;
-var USERNAME = 'mt7109';
-var PASSWORD = 'wimax';
-var TIMEOUT = 5000;
-
-var FREQUENCIES_REGEX = /([0-9,]+)/
-var BANDWIDTHS_REGEX = /[0-9]+\-([0-9,]+)/
-var DEFAULT_BW_REGEX = /([0-9]+)\-[0-9,]+/
-var telnetConnect = function(host, callback) {
-  var client = net.connect(PORT, host, function() {
-    // enter username
-    telnetExecute(USERNAME, client, "Password: ", function(err, response) {
-      if (err) return callback(err);
-      // enter password
-      telnetExecute(PASSWORD, client, "\n# ", function(err, response) {
-        return callback(err);
-      });
-    });
-  });
-  return client;
-}
-
-var telnetExecute = function(command, connection, prompt, callback) {
-  var response = '';
-  command += "\n";
-  var listener = function(chunk) {
-    response += chunk.toString();
-    if (response.slice(0 - prompt.length) == prompt) {
-      connection.removeListener('data', listener);
-      return callback(null, response.slice(command.length, 0 - prompt.length));
-    }
-  };
-
-  var end = function() {
-    connection.removeListener('end', end);
-    return callback(null, response.slice(command.length));
-  };
-
-  if (prompt)
-    connection.on('data', listener);
-  else
-    connection.on('end', end);
-
-  if (command)
-    connection.write(command);
-}
-
-var getDeviceIp = function(deviceId, callback) {
-  var db = require('../../lib/db');
-  var URL = require('url');
-
-  db.devicesCollection.findOne({_id : deviceId}, {'InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value' : 1}, function(err, device) {
-    var connectionRequestUrl = device.InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value;
-    var url = URL.parse(connectionRequestUrl);
-    return callback(url.hostname);
-  });
-};
-
-exports.init = function(deviceId, args, callback) {
-  return exports.get(deviceId, args, callback);
-}
-
-exports.get = function(deviceId, args, callback) {
-  var currentFrequencies = [];
-  var currentBandwidths = [];
-  var defaultBandwidth = '';
-  var final = [];
-
-  getDeviceIp(deviceId, function(ip) {
-    var client = telnetConnect(ip, function(err) {
-      // get frequencies
-      telnetExecute("sncfg get WMX_FREQ_LIST", client, "\n# ", function(err, response) {
-        if (err) return callback(err);
-        currentFrequencies = FREQUENCIES_REGEX.exec(response)[1].split(',')
-        // get bandwidths
-        telnetExecute("sncfg get WMX_FREQ_BANDWDITH_FL", client, "\n# ", function(err, response) {
-          if (err) return callback(err);
-          currentBandwidths = BANDWIDTHS_REGEX.exec(response)[1].split(',')
-          defaultBandwidth = DEFAULT_BW_REGEX.exec(response)[1]
-          // generate freq/bw pairs string
-          for (var i = 0; i < currentFrequencies.length; i++) {
-            final.push(currentFrequencies[i] + ' ' + currentBandwidths[i])
-          }
-          // log out
-          telnetExecute("exit", client, null, function(err, response) {
-            client.end();
-            return callback(err, defaultBandwidth + '-' + final.join(','));
-          });
-        });
-      });
-    });
-
-    client.on("error", function(err) {
-      client.destroy();
-      callback(err);
-    });
-
-    client.setTimeout(TIMEOUT, function() {
-      client.destroy();
-      callback("timeout");
-    });
-  });
-};
-
-var parsePairs = function(pairsString) {
-  var a = pairsString.split(',');
-  var f = [];
-  var b = [];
-  var l = [];
-  for (i in a) {
-    var sp = a[i].trim().split(' ');
-    f.push(parseInt(sp[0]));
-    b.push(parseFloat(sp[1]));
-  }
-  l.push(f.join(','));
-  l.push(b.join(','));
-  return l;
-}
-
-exports.set = function(deviceId, args, callback) {
-  executeSet(deviceId, args, ['set'], callback);
-};
-
-exports.dset = function(deviceId, args, callback) {
-  executeSet(deviceId, args, ['dset'], callback);
-};
-
-exports.setdset = function(deviceId, args, callback) {
-  executeSet(deviceId, args, ['dset', 'set'], callback);
-};
-
-var executeSet = function(deviceId, args, method, callback) {
-  var tmp = args.split('-');
-  var defaultBandwidth = tmp[0];
-  var pairs = parsePairs(tmp[1]);
-  var frequenciesString = pairs[0];
-  var bandwidthsString = pairs[1];
-
-  getDeviceIp(deviceId, function(ip) {
-    var client = telnetConnect(ip, function(err) {
-      var telnet_cmd = "sncfg " + method[0];
-      telnetExecute(telnet_cmd + " WMX_FREQ_BANDWDITH_FL '" + defaultBandwidth + "-" + bandwidthsString + "'", client, "\n# ", function(err, response) {
-        if (err) return callback(err);
-        telnetExecute(telnet_cmd + " WMX_FREQ_LIST '" + frequenciesString + "'", client, "\n# ", function(err, response) {
-          if (err) return callback(err);
-          if (method.length > 1) {
-            telnet_cmd = "sncfg " + method[1];
-            telnetExecute(telnet_cmd + " WMX_FREQ_BANDWDITH_FL '" + defaultBandwidth + "-" + bandwidthsString + "'", client, "\n# ", function(err, response) {
-              if (err) return callback(err);
-              telnetExecute(telnet_cmd + " WMX_FREQ_LIST '" + frequenciesString + "'", client, "\n# ", function(err, response) {
-                if (err) return callback(err);
-                telnetExecute("sncfg commit", client, "\n# ", function(err, response) {
-                  if (err) return callback(err);
-                  // log out
-                  telnetExecute("exit", client, null, function(err, response) {
-                    client.end();
-                    return callback(null, args);
-                  });
-                });
-              });
-            });
-          }
-          else {
-            telnetExecute("sncfg commit", client, "\n# ", function(err, response) {
-            if (err) return callback(err);
-              // log out
-              telnetExecute("exit", client, null, function(err, response) {
-                client.end();
-                return callback(null, args);
-              });
-            });
-          }
-        });
-      });
-    });
-
-    client.on("error", function(err) {
-      client.destroy();
-      callback(err);
-    });
-
-    client.setTimeout(TIMEOUT, function() {
-      client.destroy();
-      callback("timeout");
-    });
-  });
-};

config/custom_commands/seowon_frequencies.js

@@ -1,230 +0,0 @@
-/*
- * Copyright 2013 Fanoos Telecom
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-// Change frequency plans for Seowon devices.
-// Args format: 3532500 5000,3537500 5000,3542500 5000,3547500 5000,3552500 5000,3557500 5000,3505000 10000,3515000 10000,3525000 10000,3535000 10000,3545000 10000,3555000 10000
-var net = require('net');
-var PORT = 23;
-var USERNAME = 'admin';
-var PASSWORD = 'admin';
-var TIMEOUT = 5000;
-
-var FREQUENCIES_REGEX = /\s*([\d]+)\.\s*Frequency\s*:\s*([\d]+)\s\(KHz\),\s*Bandwidth\s*:\s*([\d\.]+)\s*\(MHz\)/gm
-
-var telnetConnect = function(host, callback) {
-  var client = net.connect(PORT, host, function() {
-    // enter username
-    telnetExecute(USERNAME, client, "Password: ", function(err, response) {
-      if (err) return callback(err);
-      // enter password
-      telnetExecute(PASSWORD, client, "MT7109> ", function(err, response) {
-        return callback(err);
-      });
-    });
-  });
-  return client;
-}
-
-var telnetExecute = function(command, connection, prompt, callback) {
-  var response = '';
-  command += "\n";
-  var listener = function(chunk) {
-    response += chunk.toString();
-    if (response.slice(0 - prompt.length) == prompt) {
-      connection.removeListener('data', listener);
-      return callback(null, response.slice(command.length, 0 - prompt.length));
-    }
-  };
-
-  var end = function() {
-    connection.removeListener('end', end);
-    return callback(null, response.slice(command.length));
-  };
-
-  if (prompt)
-    connection.on('data', listener);
-  else
-    connection.on('end', end);
-
-  if (command)
-    connection.write(command);
-}
-
-var parseFrequencies = function(frequenciesString) {
-  var a = frequenciesString.split(',');
-  var l = [];
-  for (i in a) {
-    var sp = a[i].trim().split(' ');
-    var frequency = parseInt(sp[0]);
-    var bandwidth = parseFloat(sp[1]);
-    l.push([frequency, bandwidth]);
-  }
-  return l;
-}
-
-var findFrequency = function(haystack, needle) {
-  for (i in haystack) {
-    if (haystack[i][0] == needle[0] && haystack[i][1] == needle[1])
-      return i;
-  }
-  return -1;
-}
-
-var getDeviceIp = function(deviceId, callback) {
-  var db = require('../../lib/db');
-  var URL = require('url');
-
-  db.devicesCollection.findOne({_id : deviceId}, {'InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value' : 1}, function(err, device) {
-    var connectionRequestUrl = device.InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value;
-    var url = URL.parse(connectionRequestUrl);
-    return callback(url.hostname);
-  });
-};
-
-exports.init = function(deviceId, args, callback) {
-  return exports.get(deviceId, args, callback);
-};
-
-exports.get = function(deviceId, args, callback) {
-  var currentFrequencies = [];
-
-  getDeviceIp(deviceId, function(ip) {
-    var client = telnetConnect(ip, function(err) {
-      // enter privileged mode
-      telnetExecute("enable", client, "MT7109# ", function(err, response) {
-        if (err) return callback(err);
-        // show frequencies
-        telnetExecute("show wmx freq", client, "MT7109# ", function(err, response) {
-          if (err) return callback(err);
-          while (f = FREQUENCIES_REGEX.exec(response)) {
-            var frequency = parseInt(f[2]);
-            var bandwidth = Math.floor(parseFloat(f[3]) * 1000);
-            var freqStr = String(frequency) + ' ' + String(bandwidth);
-            currentFrequencies.push(freqStr);
-          }
-          // log out
-          telnetExecute("logout", client, null, function(err, response) {
-            client.end();
-            return callback(err, currentFrequencies.join(','));
-          });
-        });
-      });
-    });
-
-    client.on("error", function(err) {
-      client.destroy();
-      callback(err);
-    });
-
-    client.setTimeout(TIMEOUT, function() {
-      client.destroy();
-      callback("timeout");
-    });
-  });
-};
-
-exports.set = function(deviceId, args, callback) {
-  var newFrequencies = parseFrequencies(args);
-  var addFrequencies = [];
-  var delFrequencies = [];
-
-  getDeviceIp(deviceId, function(ip) {
-    var client = telnetConnect(ip, function(err) {
-      // enter privileged mode
-      telnetExecute("enable", client, "MT7109# ", function(err, response) {
-        if (err) return callback(err);
-        // show frequencies
-        telnetExecute("show wmx freq", client, "MT7109# ", function(err, response) {
-          if (err) return callback(err);
-          while (f = FREQUENCIES_REGEX.exec(response)) {
-            var index = parseInt(f[1]);
-            var frequency = parseInt(f[2]);
-            var bandwidth = Math.floor(parseFloat(f[3]));
-
-            var i = findFrequency(newFrequencies, [frequency, bandwidth * 1000]);
-            if (i == -1)
-              delFrequencies.push(index);
-            else
-              newFrequencies.splice(i, 1);
-          }
-          for (f in newFrequencies)
-            addFrequencies.push(newFrequencies[f]);
-
-          if (addFrequencies.length == 0 && delFrequencies.length == 0) {
-            // log out
-            telnetExecute("logout", client, null, function(err, response) {
-              client.end();
-              return callback(err, args);
-            });
-            return;
-          }
-
-          telnetExecute("wimax", client, "MT7109 (WiMax)# ", function(err, response) {
-            if (err) return callback(err);
-            var add, del;
-            del = function() {
-              if (delFrequencies.length == 0) {
-                add();
-                return;
-              }
-              var i = delFrequencies.pop();
-              telnetExecute("wmx freq del " + i, client, "MT7109 (WiMax)# ", function(err, response) {
-                if (err) return callback(err);
-                del();
-              });
-            };
-            add = function() {
-              if (addFrequencies.length == 0) {
-                // commit
-                telnetExecute("commit", client, "MT7109 (WiMax)# ", function(err, response) {
-                  if (err) return callback(err);
-                  // log out
-                  telnetExecute("logout", client, null, function(err, response) {
-                    client.end();
-                    return callback(null, args);
-                  });
-                });
-                return;
-              }
-              var i = addFrequencies.pop();
-              telnetExecute("wmx freq add " + i[0] + ' ' + i[1]/1000, client, "MT7109 (WiMax)# ", function(err, response) {
-                if (err) return callback(err);
-                add();
-              });
-            };
-            del();
-          });
-        });
-      });
-    });
-
-    client.on("error", function(err) {
-      client.destroy();
-      callback(err);
-    });
-
-    client.setTimeout(TIMEOUT, function() {
-      client.destroy();
-      callback("timeout");
-    });
-  });
-};

config/ext-sample.js

@@ -0,0 +1,34 @@
+// This is an exmaple GenieACS extension to get the current latitude/longitude
+// of the International Space Station. Why, you ask? Because why not.
+// To install, copy this file to config/ext/iss.js.
+
+"use strict";
+
+const http = require("http");
+
+let cache = null;
+let cacheExpire = 0;
+
+function latlong(args, callback) {
+  if (Date.now() < cacheExpire)
+    return callback(null, cache);
+
+  http.get("http://api.open-notify.org/iss-now.json", (res) => {
+    if (res.statusCode !== 200)
+      return callback(new Error(`Request failed (status code: ${res.statusCode})`));
+
+    let rawData = "";
+    res.on("data", (chunk) => rawData += chunk);
+
+    res.on("end", () => {
+      let pos = JSON.parse(rawData)["iss_position"];
+      cache = [+pos["latitude"], +pos["longitude"]];
+      cacheExpire = Date.now() + 10000;
+      callback(null, cache);
+    });
+  }).on("error", (err) => {
+    callback(err);
+  });
+}
+
+exports.latlong = latlong;

config/parameters-sample.json

@@ -1,27 +0,0 @@
-{
-  "_lastInform" : {"alias" : "summary.lastInform", "type" : "date"},
-  "_lastBoot" : {"alias" : "summary.lastBoot", "type" : "date"},
-  "_lastBootstrap" : {"alias" : "summary.lastBootstrap", "type" : "date"},
-  "_deviceId._SerialNumber" : {"alias" : "summary.serialNumber"},
-  "_deviceId._ProductClass" : {"alias" : "summary.productClass"},
-  "_deviceId._OUI" : {"alias" : "summary.oui"},
-  "_deviceId._Manufacturer" : {"alias" : "summary.manufacturer"},
-  "InternetGatewayDevice.DeviceInfo.HardwareVersion" : {"alias" : "summary.hardwareVersion"},
-  "InternetGatewayDevice.DeviceInfo.SoftwareVersion" : {"alias" : "summary.softwareVersion"},
-  "Device.DeviceInfo.HardwareVersion" : {"alias" : "summary.hardwareVersion"},
-  "Device.DeviceInfo.SoftwareVersion" : {"alias" : "summary.softwareVersion"},
-
-  "InternetGatewayDevice.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANIPConnection./^\\d+$/.MACAddress" : {"alias" : "summary.mac", "type" : "mac"},
-  "InternetGatewayDevice.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANIPConnection./^\\d+$/.ExternalIPAddress" : {"alias" : "summary.ip"},
-  "InternetGatewayDevice.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANPPPConnection./^\\d+$/.MACAddress" : {"alias" : "summary.mac", "type" : "mac"},
-  "InternetGatewayDevice.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANPPPConnection./^\\d+$/.ExternalIPAddress" : {"alias" : "summary.ip"},
-  "Device.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANIPConnection./^\\d+$/.MACAddress" : {"alias" : "summary.mac", "type" : "mac"},
-  "Device.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANIPConnection./^\\d+$/.ExternalIPAddress" : {"alias" : "summary.ip"},
-  "Device.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANPPPConnection./^\\d+$/.MACAddress" : {"alias" : "summary.mac", "type" : "mac"},
-  "Device.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANPPPConnection./^\\d+$/.ExternalIPAddress" : {"alias" : "summary.ip"},
-
-  "InternetGatewayDevice.LANDevice./^\\d+$/.WLANConfiguration./^\\d+$/.SSID" : {"alias" : "summary.wlanSsid"},
-  "InternetGatewayDevice.LANDevice./^\\d+$/.WLANConfiguration./^\\d+$/.PreSharedKey.1.KeyPassphrase" : {"alias" : "summary.wlanPassphrase"},
-  "Device.LANDevice./^\\d+$/.WLANConfiguration./^\\d+$/.SSID" : {"alias" : "summary.wlanSsid"},
-  "Device.LANDevice./^\\d+$/.WLANConfiguration./^\\d+$/.PreSharedKey.1.KeyPassphrase" : {"alias" : "summary.wlanPassphrase"}
-}

lib/api-functions.coffee

@@ -1,5 +1,7 @@
 ###
-# Copyright 2013, 2014  Zaid Abdulla
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
 #
 # GenieACS is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as
@@ -13,31 +15,11 @@
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-#
-# This file incorporates work covered by the following copyright and
-# permission notice:
-#
-# Copyright 2013 Fanoos Telecom
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
 ###
 
+crypto = require 'crypto'
+dgram = require 'dgram'
+
 config = require './config'
 db = require './db'
 http = require 'http'
@@ -45,11 +27,52 @@ common = require './common'
 util = require 'util'
 URL = require 'url'
 auth = require './auth'
+cache = require './cache'
 
 
-connectionRequest = (deviceId, callback) ->
-  # ensure socket is reused in case of digest authentication
-  agent = new http.Agent({maxSockets : 1})
+udpConReq = (address, un, key, callback) ->
+  return false if not address
+  [host, port] = address.split(':', 2)
+  if port is undefined
+    port = 80
+  else
+    port = parseInt(port)
+    return false if isNaN(port)
+
+  ts = Math.trunc(Date.now() / 1000)
+  id = Math.trunc(Math.random() * 4294967295)
+  cn = crypto.randomBytes(8).toString('hex')
+  sig = crypto.createHmac('sha1', key).update("#{ts}#{id}#{un}#{cn}").digest('hex')
+  uri = "http://#{address}?ts=#{ts}&id=#{id}&un=#{un}&cn=#{cn}&sig=#{sig}"
+
+  message = Buffer.from("GET #{uri} HTTP/1.1\r\nHost: #{address}\r\n\r\n")
+
+  client = dgram.createSocket({type: 'udp4', reuseAddr: true})
+  UDP_CONNECTION_REQUEST_PORT = config.get('UDP_CONNECTION_REQUEST_PORT')
+  if UDP_CONNECTION_REQUEST_PORT
+    # When a device is NAT'ed, the UDP Connection Request must originate from the same address and port used by the STUN server, in order to traverse the firewall.
+    # This does require that the Genieacs NBI and STUN server are allowed to bind to the same address and port.
+    # The STUN server needs to open its UDP port with the SO_REUSEADDR option, allowing the NBI to also bind to the same port.
+    client.bind({port: UDP_CONNECTION_REQUEST_PORT, exclusive: true})
+
+  count = 3
+  client.send(message, 0, message.length, port, host, f = (err) ->
+    if err or -- count <= 0
+      client.close()
+      return callback(err)
+    client.send(message, 0, message.length, port, host, f)
+  )
+  return true
+
+
+httpConReq = (url, username, password, allowBasicAuth, timeout, callback) ->
+  options = URL.parse(url)
+
+  if options.protocol != 'http:'
+    return callback(new Error('Invalid connection request URL or protocol'))
+
+  # Ensure socket is reused in case of digest authentication
+  options.agent = new http.Agent({maxSockets : 1})
 
   statusToError = (statusCode) ->
     switch statusCode
@@ -62,102 +85,131 @@ connectionRequest = (deviceId, callback) ->
       else
         new Error("Unexpected response code from device: #{statusCode}")
 
-  conReq = (url, authString, callback) ->
-    options = URL.parse(url)
-    options.agent = agent
-
-    if authString
-      options.headers = {'Authorization' : authString}
-
   request = http.get(options, (res) ->
     if res.statusCode == 401 and res.headers['www-authenticate']?
       authHeader = auth.parseAuthHeader(res.headers['www-authenticate'])
-      callback(res.statusCode, authHeader)
-      # don't need body, go ahead and emit events to free up the socket for possible reuse
+      if authHeader.method is 'Basic'
+        if not allowBasicAuth
+          request.abort()
+          callback(new Error('Basic HTTP authentication not allowed'))
+          return
+        options.headers = {'Authorization' : auth.basic(username or '', password or '')}
+      else if authHeader.method is 'Digest'
+        options.headers = {
+          'Authorization' : auth.digest(username or '', password or '', options.path, 'GET', null, authHeader)
+        }
+
+      request = http.get(options, (res) ->
+        if res.statusCode == 0
+          # Workaround for some devices unexpectedly closing the connection
+          request = http.get(options, (res) ->
+            callback(statusToError(res.statusCode))
             res.resume()
-    )
-
-    request.on('error', (err) ->
-      # error event when request is aborted
+          ).on('error', (err) ->
             request.abort()
-      callback(0)
+            callback(statusToError(0))
+          ).on('socket', (socket) ->
+            socket.setTimeout(timeout)
+            socket.on('timeout', () -> request.abort())
           )
-
-    request.on('socket', (socket) ->
-      socket.setTimeout(2000)
-      socket.on('timeout', () ->
+        else
+          callback(statusToError(res.statusCode))
+        res.resume()
+      ).on('error', (err) ->
         request.abort()
+        callback(statusToError(0))
+      ).on('socket', (socket) ->
+        socket.setTimeout(timeout)
+        socket.on('timeout', () -> request.abort())
       )
+    else
+      callback(statusToError(res.statusCode))
+
+    # No listener for data so emit resume
+    res.resume()
+  ).on('error', (err) ->
+    request.abort()
+    callback(statusToError(0))
+  ).on('socket', (socket) ->
+    socket.setTimeout(timeout)
+    socket.on('timeout', () -> request.abort())
   )
 
+
+connectionRequest = (deviceId, callback) ->
+  CONNECTION_REQUEST_TIMEOUT = config.get('CONNECTION_REQUEST_TIMEOUT', deviceId)
+  CONNECTION_REQUEST_ALLOW_BASIC_AUTH = config.get('CONNECTION_REQUEST_ALLOW_BASIC_AUTH', deviceId)
   proj = {
     'Device.ManagementServer.ConnectionRequestURL._value' : 1,
-    'InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value' : 1
+    'Device.ManagementServer.UDPConnectionRequestAddress._value' : 1,
+    'Device.ManagementServer.ConnectionRequestUsername._value' : 1,
+    'Device.ManagementServer.ConnectionRequestPassword._value' : 1,
+    'InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value' : 1,
+    'InternetGatewayDevice.ManagementServer.UDPConnectionRequestAddress._value' : 1,
+    'InternetGatewayDevice.ManagementServer.ConnectionRequestUsername._value' : 1,
+    'InternetGatewayDevice.ManagementServer.ConnectionRequestPassword._value' : 1
   }
 
   db.devicesCollection.findOne({_id : deviceId}, proj, (err, device)->
-    if err
-      callback(err)
-      return
+    return callback(err) if err
 
     if device.Device? # TR-181 data model
       connectionRequestUrl = device.Device.ManagementServer.ConnectionRequestURL._value
+      udpConnectionRequestAddress = device.Device.ManagementServer?.UDPConnectionRequestAddress?._value
+      username = device.Device.ManagementServer.ConnectionRequestUsername?._value
+      password = device.Device.ManagementServer.ConnectionRequestPassword?._value
     else # TR-098 data model
       connectionRequestUrl = device.InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value
+      udpConnectionRequestAddress = device.InternetGatewayDevice.ManagementServer?.UDPConnectionRequestAddress?._value
+      username = device.InternetGatewayDevice.ManagementServer.ConnectionRequestUsername?._value
+      password = device.InternetGatewayDevice.ManagementServer.ConnectionRequestPassword?._value
 
-    # for testing
-    #connectionRequestUrl = connectionRequestUrl.replace(/^(http:\/\/)([0-9\.]+)(\:[0-9]+\/[a-zA-Z0-9]+\/?$)/, '$110.1.1.254$3')
-    conReq(connectionRequestUrl, null, (statusCode, authHeader) ->
-      if statusCode == 401
-        [username, password] = config.auth.connectionRequest(deviceId)
-        if authHeader.method is 'Basic'
-          authString = auth.basic(username, password)
-        else if authHeader.method is 'Digest'
-          uri = URL.parse(connectionRequestUrl)
-          authString = auth.digest(username, password, uri.path, 'GET', null, authHeader)
+    conReq = () ->
+      udpSent = udpConReq(udpConnectionRequestAddress, username, password, (err) -> throw err if err)
 
-        conReq(connectionRequestUrl, authString, (statusCode, authHeader) ->
-          if statusCode == 0
-            # Workaround for some devices unexpectedly closing the connection
-            return conReq(connectionRequestUrl, authString, (statusCode) -> callback(statusToError(statusCode)))
-          callback(statusToError(statusCode))
+      httpConReq(connectionRequestUrl, username, password, CONNECTION_REQUEST_ALLOW_BASIC_AUTH, CONNECTION_REQUEST_TIMEOUT, (err) ->
+        if udpSent
+          return callback()
+        callback(err)
       )
-      else
-        callback(statusToError(statusCode))
+
+    if config.auth?.connectionRequest?
+      # Callback is optional for backward compatibility
+      if config.auth.connectionRequest.length > 4
+        return config.auth.connectionRequest(deviceId, connectionRequestUrl, username, password, (u, p) ->
+          username = u
+          password = p
+          conReq()
         )
+      [username, password] = config.auth.connectionRequest(deviceId, connectionRequestUrl, username, password)
+    conReq()
   )
 
 
-watchTask = (taskId, timeout, callback) ->
+watchTask = (deviceId, taskId, timeout, callback) ->
   setTimeout(() ->
-    db.tasksCollection.findOne({_id : taskId}, {'_id' : 1, 'fault' : 1}, (err, task) ->
+    db.tasksCollection.findOne({_id : taskId}, {'_id' : 1}, (err, task) ->
       return callback(err) if err
 
-      if task
-        timeout -= 500
-        if task.fault?
-          callback(null, 'fault')
-        else if timeout <= 0
-          callback(null, 'timeout')
-        else
-          watchTask(taskId, timeout, callback)
-      else
-        callback(null, 'completed')
+      if not task
+        return callback(null, 'completed')
+
+      db.faultsCollection.findOne({_id : "#{deviceId}:task_#{taskId}"}, {'_id' : 1}, (err, fault) ->
+        return callback(err) if err
+
+        if fault
+          return callback(null, 'fault')
+
+        if (timeout -= 500) <= 0
+          return callback(null, 'timeout')
+
+        watchTask(deviceId, taskId, timeout, callback)
+      )
     )
   , 500)
 
 
-expandParam = (param, aliases) ->
-  params = [param]
-  for a,aa of aliases
-    if a == param or common.startsWith(a, "#{param}.")
-      for p in aa
-        params.push(p) if p[p.lastIndexOf('.') + 1] != '_'
-
-  return params
-
-
-sanitizeTask = (task, aliases, callback) ->
+sanitizeTask = (task, callback) ->
   task.timestamp = new Date(task.timestamp ? Date.now())
   if task.expiry?
     if common.typeOf(task.expiry) is common.DATE_TYPE or isNaN(task.expiry)
@@ -165,67 +217,21 @@ sanitizeTask = (task, aliases, callback) ->
     else
       task.expiry = new Date(task.timestamp.getTime() + +task.expiry * 1000)
 
-  switch task.name
-    when 'getParameterValues'
-      projection = {}
-      for p in task.parameterNames
-        for pp in expandParam(p, aliases)
-          projection[pp] = 1
-      db.devicesCollection.findOne({_id : task.device}, projection, (err, device) ->
-        parameterNames = []
-        for k of projection
-          if common.getParamValueFromPath(device, k)?
-            parameterNames.push(k)
-        task.parameterNames = parameterNames
-        callback(task)
-      )
-    when 'setParameterValues'
-      projection = {}
-      values = {}
-      for p in task.parameterValues
-        for pp in expandParam(p[0], aliases)
-          projection[pp] = 1
-          values[pp] = p[1]
-      db.devicesCollection.findOne({_id : task.device}, projection, (err, device) ->
-        parameterValues = []
-        for k of projection
-          param = common.getParamValueFromPath(device, k)
-          if param?
-            parameterValues.push([k, values[k], param._type])
-        task.parameterValues = parameterValues
-        callback(task)
-      )
-    else
-      # TODO implement setParameterValues
   callback(task)
 
 
-addAliases = (device, aliases) ->
-  for k,v of aliases
-    for p in v
-      pp = p.split('.')
-      obj = device
-      for i in pp
-        if not obj[i]?
-          obj = null
-          break
-        obj = obj[i]
-
-      device[k] = obj if obj?
-
-
-insertTasks = (tasks, aliases, callback) ->
+insertTasks = (tasks, callback) ->
   if tasks? and common.typeOf(tasks) isnt common.ARRAY_TYPE
     tasks = [tasks]
   else if not tasks? or tasks.length == 0
-    return callback(tasks)
+    return callback(null, tasks or [])
 
   counter = tasks.length
 
   for task in tasks
-    sanitizeTask(task, aliases, (t) ->
+    sanitizeTask(task, (t) ->
       if t.uniqueKey?
-        db.tasksCollection.remove({device : t.device, uniqueKey : t.uniqueKey}, (err, removed) ->
+        db.tasksCollection.remove({device : t.device, uniqueKey : t.uniqueKey}, (err) ->
         )
 
       --counter
@@ -238,19 +244,21 @@ insertTasks = (tasks, aliases, callback) ->
 
 
 deleteDevice = (deviceId, callback) ->
-  db.tasksCollection.remove({'device' : deviceId}, (err, removed) ->
+  db.tasksCollection.remove({'device' : deviceId}, (err) ->
     return callback(err) if err
-    db.devicesCollection.remove({'_id' : deviceId}, (err, removed) ->
+    db.devicesCollection.remove({'_id' : deviceId}, (err) ->
       return callback(err) if err
-      db.redisClient.del("#{deviceId}_presets_hash", "#{deviceId}_inform_hash", (err) ->
-        callback(err)
+      db.faultsCollection.remove({'_id' : {'$regex' : "^#{common.escapeRegExp(deviceId)}\\:"}}, (err) ->
+        return callback(err) if err
+        db.operationsCollection.remove({'_id' : {'$regex' : "^#{common.escapeRegExp(deviceId)}\\:"}}, (err) ->
+          return callback(err) if err
+          cache.del("#{deviceId}_tasks_faults_operations", callback)
+        )
       )
     )
   )
 
 
-exports.addAliases = addAliases
-exports.sanitizeTask = sanitizeTask
 exports.connectionRequest = connectionRequest
 exports.watchTask = watchTask
 exports.insertTasks = insertTasks

lib/auth.coffee

@@ -1,5 +1,7 @@
 ###
-# Copyright 2013, 2014  Zaid Abdulla
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
 #
 # GenieACS is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as
@@ -19,18 +21,30 @@ crypto = require 'crypto'
 
 
 exports.parseAuthHeader = (authHeader) ->
-  res = {}
-  i = authHeader.indexOf(' ')
-  res['method'] = authHeader.slice(0, i)
+  authHeader = authHeader.trim()
+  method = authHeader.split(' ', 1)[0]
+  res = {method: method}
+  parts = authHeader.slice(method.length + 1).split(',')
 
-  options = authHeader.slice(i + 1)
-  regex = /([a-z]+)="([a-zA-Z0-9\/\.@\-\+,= ]+)"/g
-  while (r = regex.exec(options)) != null
-    res[r[1]] = r[2]
+  while (part = parts.shift())?
+    name = part.split('=', 1)[0]
+    throw new Error('Unable to parse auth header') if name.length == part.length
+    value = part.slice(name.length + 1)
+    if not /^\s*"/.test(value)
+      value = value.trim()
+    else
+      while not /[^\\]"\s*$/.test(value)
+        p = parts.shift()
+        throw new Error('Unable to parse auth header') if not p?
+        value += ',' + p
+
+      try
+        value = JSON.parse(value)
+      catch
+        throw new Error('Unable to parse auth header')
+
+    res[name.trim()] = value
 
-  regex = /([a-z]+)=([a-zA-Z0-9\/\.@\-\+]+)/g
-  while (r = regex.exec(options)) != null
-    res[r[1]] = r[2]
   return res
 
 
@@ -72,9 +86,7 @@ exports.digest = (username, password, uri, httpMethod, body, authHeader) ->
   authString += ",nonce=\"#{authHeader.nonce}\""
   authString += ",uri=\"#{uri}\""
   authString += ",algorithm=#{authHeader.algorithm}" if authHeader.algorithm?
-  authString += ",qop=#{qop}"
-  authString += ",nc=#{nc}"
-  authString += ",cnonce=\"#{cnonce}\""
+  authString += ",qop=#{qop},nc=#{nc},cnonce=\"#{cnonce}\"" if qop?
   authString += ",response=\"#{response}\""
-  authString += ",opaque=\"#{authHeader.opaque}\""
+  authString += ",opaque=\"#{authHeader.opaque}\"" if authHeader.opaque?
   return authString

lib/cache.js

@@ -0,0 +1,257 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const config = require("./config");
+
+const MAX_CACHE_TTL = config.get("MAX_CACHE_TTL");
+
+let redisClient;
+let mongoDb;
+let mongoCollection;
+let mongoTimeOffset = 0;
+
+
+function connect(callback) {
+  const REDIS_HOST = config.get("REDIS_HOST");
+
+  if (REDIS_HOST) {
+    const redis = require("redis");
+    const REDIS_PORT = config.get("REDIS_PORT");
+    const REDIS_DB = config.get("REDIS_DB");
+    redisClient = redis.createClient(REDIS_PORT, REDIS_HOST);
+    redisClient.select(REDIS_DB, function(err) {
+      exports.get = redisGet;
+      exports.set = redisSet;
+      exports.del = redisDel;
+      exports.pop = redisPop;
+      exports.lock = redisLock;
+      callback(err);
+    });
+  }
+  else {
+    const mongodb = require("mongodb");
+    const MONGODB_CONNECTION_URL = config.get("MONGODB_CONNECTION_URL");
+
+    mongodb.MongoClient.connect(MONGODB_CONNECTION_URL, function(err, db) {
+      if (err)
+        return callback(err);
+
+      mongoDb = db;
+      mongoCollection = db.collection("cache");
+      mongoCollection.ensureIndex({"expire": 1}, {expireAfterSeconds: 0});
+
+      const now = Date.now();
+      mongoDb.command({hostInfo: 1}, function(err, res) {
+        if (err)
+          return callback(err);
+        mongoTimeOffset = res.system.currentTime.getTime() - now;
+        exports.get = mongoGet;
+        exports.set = mongoSet;
+        exports.del = mongoDel;
+        exports.pop = mongoPop;
+        exports.lock = mongoLock;
+        callback();
+      });
+    });
+  }
+}
+
+
+function disconnect() {
+  if (redisClient)
+    redisClient.quit();
+
+  if (mongoDb)
+    mongoDb.close();
+}
+
+
+function redisGet(key, callback) {
+  if (Array.isArray(key))
+    redisClient.mget(key, callback);
+  else
+    redisClient.get(key, callback);
+}
+
+
+function redisDel(key, callback) {
+  redisClient.del(key, callback);
+}
+
+
+function redisSet(key, value, ttl, callback) {
+  if (!callback && typeof v === "function") {
+    callback = ttl;
+    ttl = null;
+  }
+
+  if (!ttl)
+    ttl = MAX_CACHE_TTL;
+
+  redisClient.setex(key, ttl, value, callback);
+}
+
+
+function redisPop(key, callback) {
+  const POP_SCRIPT = "local v=redis.call(\"get\",KEYS[1]);redis.call(\"del\",KEYS[1]);return v;";
+  redisClient.eval(POP_SCRIPT, 1, key, callback);
+}
+
+
+function redisLock(lockName, ttl, callback) {
+  const UNLOCK_SCRIPT = "if redis.call(\"get\", KEYS[1]) == ARGV[1] then return redis.call(\"del\", KEYS[1]) else return 0 end";
+  const EXTEND_SCRIPT = "if redis.call(\"get\", KEYS[1]) == ARGV[1] then return redis.call(\"pexpire\", KEYS[1], ARGV[2]) else return 0 end";
+  const token = Math.random().toString(36).slice(2);
+
+  function unlockOrExtend(ttl) {
+    if (!ttl) {
+      redisClient.eval(UNLOCK_SCRIPT, 1, lockName, token, function(err, res) {
+        if (err || !res)
+          throw err || new Error("Lock expired");
+      });
+    }
+    else {
+      redisClient.eval(EXTEND_SCRIPT, 1, lockName, token, ttl, function(err, res) {
+        if (err || !res)
+          throw err || new Error("Lock expired");
+      });
+    }
+  }
+
+  redisClient.set(lockName, token, "NX", "EX", ttl, function(err, res) {
+    if (err || res)
+      return callback(err, unlockOrExtend);
+
+    setTimeout(function() {
+      redisLock(lockName, ttl, callback);
+    }, 200);
+  });
+}
+
+
+function mongoGet(key, callback) {
+  let expire = new Date(Date.now() - mongoTimeOffset);
+  if (Array.isArray(key))
+    mongoCollection.find({"_id": {"$in": key}}).toArray(function(err, res) {
+      if (err)
+        return callback(err);
+
+      let indices = {};
+      key.forEach(function(v, i) {
+        indices[v] = i;
+      });
+
+      let values = [];
+      res.forEach(function(r) {
+        if (r["expire"] > expire)
+          values[indices[r["_id"]]] = r["value"];
+      });
+      callback(null, values);
+    });
+  else
+    mongoCollection.findOne({"_id": {"$in": [key]}}, function(err, res) {
+      if (err || !res)
+        return callback(err);
+
+      if (res["expire"] > expire)
+        return callback(null, res["value"]);
+
+      callback();
+    });
+}
+
+
+function mongoDel(key, callback) {
+  if (Array.isArray(key))
+    mongoCollection.remove({"_id": {"$in": key}}, callback);
+  else
+    mongoCollection.remove({"_id": key}, callback);
+}
+
+
+function mongoSet(key, value, ttl, callback) {
+  if (!callback && typeof v === "function") {
+    callback = ttl;
+    ttl = null;
+  }
+
+  if (!ttl)
+    ttl = MAX_CACHE_TTL;
+
+  let expire = new Date((Date.now() - mongoTimeOffset) + (ttl * 1000));
+  mongoCollection.save({"_id": key, "value": value, "expire": expire}, callback);
+}
+
+
+function mongoPop(key, callback) {
+  mongoCollection.findAndModify({"_id": key}, null, null, {remove: true}, function(err, res) {
+    if (err || !res["value"])
+      return callback(err);
+
+    if (res["value"]["expire"] > new Date(Date.now() - mongoTimeOffset))
+      return callback(null, res["value"]["value"]);
+
+    callback();
+  });
+}
+
+
+function mongoLock(lockName, ttl, callback) {
+  const token = Math.random().toString(36).slice(2);
+
+  function unlockOrExtend(ttl) {
+    if (!ttl) {
+      mongoCollection.remove({"_id": lockName, "value": token}, function(err, res) {
+        if (err || res["result"]["n"] !== 1)
+          throw err || new Error("Lock expired");
+      });
+    }
+    else {
+      let expire = new Date((Date.now() - mongoTimeOffset) + (ttl * 1000));
+      mongoCollection.update(
+        {"_id": lockName, "value": token},
+        {"expire": expire},
+        function(err, res) {
+          if (err || res["result"]["n"] !== 1)
+            throw err | new Error("Lock expired");
+        }
+      );
+    }
+  }
+
+  let expireTest = new Date(Date.now() - mongoTimeOffset);
+  let expireSet = new Date((Date.now() - mongoTimeOffset) + (ttl * 1000));
+
+  mongoCollection.update(
+    {"_id": lockName, "expire": {"$lte": expireTest}},
+    {"$set": {"value" : token, "expire": expireSet}},
+    {upsert: true},
+    function(err, res) {
+      if (err && err.code === 11000)
+        return setTimeout(function() {mongoLock(lockName, ttl, callback)}, 200);
+
+      return callback(err, unlockOrExtend);
+    }
+  );
+}
+
+
+exports.connect = connect;
+exports.disconnect = disconnect;

lib/cluster.coffee

@@ -1,50 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-###
-
-config = require './config'
-cluster = require 'cluster'
-util = require 'util'
-
-# Keep current configuration to pass that to workers as environment variables
-ENVIRONMENT_VARIABLES = {}
-for k, v of config.allConfig
-  ENVIRONMENT_VARIABLES["GENIEACS_#{k}"] = v
-
-
-cluster.on('listening', (worker, address) ->
-  util.log("Worker #{worker.process.pid} listening to #{address.address}:#{address.port}")
-)
-
-cluster.on('exit', (worker, code, signal) ->
-  util.log("Worker #{worker.process.pid} died (#{worker.process.exitCode})")
-  cluster.fork(ENVIRONMENT_VARIABLES)
-)
-
-
-start = (service) ->
-  workerProcesses = config.get("#{service.toUpperCase()}_WORKER_PROCESSES")
-
-  cluster.setupMaster({
-      exec : 'lib/server',
-      args : [service]
-  })
-
-  for [0 ... workerProcesses]
-    cluster.fork(ENVIRONMENT_VARIABLES)
-
-
-exports.start = start

lib/cluster.js

@@ -0,0 +1,177 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const cluster = require("cluster");
+const logger = require("./logger");
+const config = require("./config");
+
+const VERSION = require("../package.json").version;
+
+let respawnTimestamp = 0;
+let tooManyCrashesTimestamp = 0;
+
+function getDependencyVersions() {
+  let dependencies = [`node@${process.versions.node}`];
+  try {
+    let npmls = JSON.parse(require("child_process").execSync("npm ls --prod --json --depth 0"));
+
+    for (let k in npmls.dependencies)
+      dependencies.push(`${k}@${npmls.dependencies[k].version}`);
+  } catch (err) {}
+
+  return dependencies.join(",");
+}
+
+
+function getConfig() {
+  let conf = [];
+  const keys = Object.keys(process.env);
+  for (let k of keys)
+    if (k.startsWith("GENIEACS_")) {
+      const name = k.slice(9);
+      const value = process.env[k];
+      const def = config.getDefault(name);
+      if (def == null || value !== `${def}`)
+        conf.push(`${name}=${value}`);
+    }
+
+  return conf.join(",");
+}
+
+
+function restartWorker(worker, code, signal) {
+  let msg = {
+    message: "Worker died",
+    pid: worker.process.pid
+  };
+
+  if (code != null)
+    msg.exitCode = code;
+
+  if (signal != null)
+    msg.signal = signal;
+
+  logger.error(msg);
+
+  const now = Date.now();
+  respawnTimestamp = Math.max(now, respawnTimestamp + 2000);
+  if (respawnTimestamp === now) {
+    tooManyCrashesTimestamp = now;
+    cluster.fork();
+    return
+  }
+
+  if (now - tooManyCrashesTimestamp > 60000) {
+    process.exitCode = 1;
+    cluster.removeListener("exit", restartWorker);
+    for (let pid in cluster.workers)
+      cluster.workers[pid].kill();
+
+    logger.error({
+      message: "Too many crashes, exiting",
+      pid: process.pid
+    });
+    return;
+  }
+
+  setTimeout(function() {
+    if (process.exitCode) return;
+    cluster.fork();
+  }, respawnTimestamp - now);
+}
+
+
+function start(service) {
+  logger.init(service, VERSION);
+  const config = require("./config");
+  const SERVICE_ADDRESS = config.get(`${service.toUpperCase()}_INTERFACE`)
+  const SERVICE_PORT = config.get(`${service.toUpperCase()}_PORT`)
+
+  logger.info({
+    message: `GenieACS (genieacs-${service}) starting`,
+    pid: process.pid,
+    version: VERSION,
+    dependencies: getDependencyVersions(),
+    config: getConfig()
+  });
+
+  cluster.on("listening", function(worker, address) {
+    if ((address.addressType === 4 || address.addressType === 6) &&
+        address.address === SERVICE_ADDRESS && address.port === SERVICE_PORT) {
+      logger.info({
+        message: "Worker listening",
+        pid: worker.process.pid,
+        address: address.address,
+        port: address.port
+      });
+    }
+  });
+
+  cluster.on("exit", restartWorker);
+
+  process.on("SIGINT", function() {
+    logger.info({
+      message: "Received signal SIGINT, exiting",
+      pid: process.pid
+    });
+
+    cluster.removeListener("exit", restartWorker);
+  });
+
+  process.on("SIGTERM", function() {
+    logger.info({
+      message: "Received signal SIGTERM, exiting",
+      pid: process.pid
+    });
+
+    cluster.removeListener("exit", restartWorker);
+    for (let pid in cluster.workers)
+      cluster.workers[pid].kill();
+  });
+
+  cluster.setupMaster({
+    exec: "lib/server",
+    args: [service]
+  });
+
+  let workerCount = config.get(`${service.toUpperCase()}_WORKER_PROCESSES`);
+
+  if (!workerCount)
+    workerCount = Math.max(2, require("os").cpus().length);
+
+  for (let i = 0; i < workerCount; ++ i)
+    cluster.fork();
+}
+
+
+process.on("uncaughtException", function(err) {
+  logger.error({
+    message: "Uncaught exception in master process, exiting",
+    exception: err,
+    pid: process.pid
+  });
+  process.exitCode = 1;
+  cluster.removeListener("exit", restartWorker);
+  for (let pid in cluster.workers)
+    cluster.workers[pid].kill();
+});
+
+
+exports.start = start;

lib/common.coffee

@@ -1,27 +1,23 @@
 ###
-# Copyright 2013 Fanoos Telecom
+# Copyright 2013-2017  Zaid Abdulla
 #
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
+# This file is part of GenieACS.
 #
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
+# GenieACS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
 #
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
+# GenieACS is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
 ###
 
 buffer = require 'buffer'
-querystring = require 'querystring'
 
 UNDEFINED_TYPE = '[object Undefined]'
 NULL_TYPE = '[object Null]'
@@ -38,21 +34,6 @@ typeOf = (obj) ->
   Object.prototype.toString.call(obj)
 
 
-exports.endsWith = (str, suffix) ->
-  str.indexOf(suffix, str.length - suffix.length) isnt -1
-
-
-exports.startsWith = (str, prefix) ->
-  str.substring(0, prefix.length) == prefix
-
-
-exports.arrayToHash = (arr) ->
-  hash = {}
-  for i in arr
-    hash[i[0]] = i[1]
-  return hash
-
-
 exports.generateDeviceId = (deviceIdStruct) ->
   # Percent escaping function. Escapes everything except alphanumerics and underscore
   esc = (str) ->
@@ -70,34 +51,6 @@ exports.generateDeviceId = (deviceIdStruct) ->
   return "#{esc(deviceIdStruct['OUI'])}-#{esc(deviceIdStruct['SerialNumber'])}"
 
 
-exports.parseDeviceId = (deviceId) ->
-  parts = deviceId.split('-')
-  ret = {oui : querystring.unescape(parts[0])}
-  if parts.length == 3
-    ret.productClass = querystring.unescape(parts[1])
-    ret.serialNumber = querystring.unescape(parts[2])
-  else
-    ret.serialNumber = querystring.unescape(parts[1])
-  return ret
-
-
-exports.extend = (obj, mixin) ->
-  obj[name] = method for name, method of mixin
-  obj
-
-
-exports.flattenObject = (object) ->
-  newObj = {}
-  f = (obj, prefix) ->
-    for k, v of obj
-      if typeof(v) is 'object'
-        f(v, "#{prefix}#{k}.")
-      else
-        newObj["#{prefix}#{k}"] = v
-  f(object, '')
-  return newObj
-
-
 exports.getParamValueFromPath = (obj, path) ->
   pp = path.split('.')
   ref = obj
@@ -109,17 +62,156 @@ exports.getParamValueFromPath = (obj, path) ->
     return undefined
 
 
-exports.matchType = (src, dst) ->
-  switch typeOf(src)
-    when STRING_TYPE
-      String(dst)
-    when NUMBER_TYPE
-      Number(dst)
-    when BOOLEAN_TYPE
-      v = String(dst).trim().toLowerCase()
-      v == 'true' or v == 'on' or v == 'yes' or v == '1'
+pathOverlap = (a, b, start) ->
+  if a.length == b.length
+    res = 3
+  else if a.length > b.length
+    res = 2
   else
-      dst
+    res = 1
+
+  for i in [(start ? 0)...Math.min(a.length, b.length)] by 1
+    if a[i]?
+      if not b[i]?
+        res &= 2
+      else if a[i] != b[i]
+        return 0
+    else if b[i]?
+      res &= 1
+
+    return 0 if not res
+
+  return res
+
+
+parseAlias = (pattern, start, res) ->
+  aliases = []
+  i = start
+  while i < pattern.length and pattern[i] != ']'
+    alias = []
+    i = j = parsePath(pattern, i, alias) + 1
+
+    while pattern[j] not in [']', ',']
+      if pattern[j] == '"' and i == j
+        ++ j
+        while pattern[j] != '"' or pattern[j - 1] == '\\'
+          if ++ j >= pattern.length
+            throw new Error('Invalid alias expression')
+      if ++ j >= pattern.length
+        throw new Error('Invalid alias expression')
+    value = pattern[i...j].trim()
+    i = j
+    if value[0] == '"'
+      try
+        value = JSON.parse(value)
+      catch err
+        throw new Error('Invalid alias expression')
+    alias.push(value)
+    aliases.push(alias)
+    ++ i if pattern[i] == ','
+
+  # Need to sort to ensure identical expressions have idential string representation
+  srt = (a, b) ->
+    for j in [0...Math.min(a.length, b.length)] by 2
+      for k in [0...Math.min(a[j].length, b[j].length)] by 1
+        if Array.isArray(a[j][k])
+          if Array.isArray(b[j][k])
+            return srt(a[j][k], b[j][k])
+          else if not b[j][k]?
+            return -1
+          else
+            return 1
+        else if not a[j][k]?
+          if not b[j][k]?
+            return 0
+          else
+            return 1
+        else if not b[j][k]? or Array.isArray(b[j][k])
+            return -1
+        else if a[j][k] > b[j][k]
+          return 1
+        else if a[j][k] < b[j][k]
+          return -1
+
+      if a[j].length > b[j].length
+        return -1
+      else if a[j].length < b[j].length
+        return 1
+
+      if a[j + 1] > b[j + 1]
+        return -1
+      else if a[j + 1] < b[j + 1]
+        return 1
+
+    if a.length > b.length
+      return -1
+    else if a.length < b.length
+      return 1
+
+    return 0
+
+  aliases.sort(srt)
+  res.push([].concat.apply([], aliases))
+  return i
+
+
+parsePath = (pattern, start, res) ->
+  path = []
+  path.wildcard = 0
+  path.alias = 0
+  i = start ? 0
+
+  # Colon separator is needed for parseAlias
+  if i < pattern.length and pattern[i] != ':'
+    while true
+      if pattern[i] == '['
+        path.alias |= 1 << path.length
+        i = parseAlias(pattern, i + 1, path) + 1
+      else
+        j = i
+        while i < pattern.length and pattern[i] != ':' and pattern[i] != '.'
+          ++ i
+        n = pattern.slice(j, i)
+        path.wildcard |= 1 << path.length if n == '*'
+        path.push(n)
+
+      if i >= pattern.length or pattern[i] == ':'
+        break
+      else if pattern[i] != '.'
+        throw new Error('Invalid alias expression')
+
+      ++ i
+
+  Object.freeze(path)
+
+  if not res?
+    return path
+
+  res.push(path)
+  return i
+
+
+addPathMeta = (path) ->
+  return path if path.alias? or path.wildcard?
+
+  path.alias = 0
+  path.wildcard = 0
+
+  for p, i in path
+    if typeOf(p) is ARRAY_TYPE
+      path.alias |= 1 << i
+      for j in [0...p.length] by 2
+        addPathMeta(p[j])
+    else if p == '*'
+      path.wildcard |= 1 << i
+
+  Object.freeze(path)
+  return path
+
+
+# Source: http://stackoverflow.com/a/6969486
+escapeRegExp = (str) ->
+  return str.replace(/[\-\[\]\/\{\}\(\)\*\+\?\.\\\^\$\|]/g, "\\$&")
 
 
 exports.UNDEFINED_TYPE = UNDEFINED_TYPE
@@ -131,3 +223,7 @@ exports.OBJECT_TYPE = OBJECT_TYPE
 exports.REGEXP_TYPE = REGEXP_TYPE
 
 exports.typeOf = typeOf
+exports.pathOverlap = pathOverlap
+exports.parsePath = parsePath
+exports.addPathMeta = addPathMeta
+exports.escapeRegExp = escapeRegExp

lib/config.coffee

@@ -1,5 +1,7 @@
 ###
-# Copyright 2013, 2014  Zaid Abdulla
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
 #
 # GenieACS is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as
@@ -19,48 +21,63 @@ path = require 'path'
 common = require './common'
 
 options = {
-  CONFIG_DIR : {type : 'string', default : 'config'},
+  CONFIG_DIR : {type : 'path', default : 'config'},
   MONGODB_CONNECTION_URL : {type : 'string', default : 'mongodb://127.0.0.1/genieacs'},
   REDIS_PORT : {type : 'int', default : 6379},
-  REDIS_HOST : {type : 'string', default : '127.0.0.1'},
+  REDIS_HOST : {type : 'string', default : ''},
   REDIS_DB : {type : 'int', default : 0},
 
-  CWMP_WORKER_PROCESSES : {type : 'int', default : 4},
+  CWMP_WORKER_PROCESSES : {type : 'int', default : 0},
   CWMP_PORT : {type : 'int', default : 7547},
   CWMP_INTERFACE : {type : 'string', default : '0.0.0.0'},
   CWMP_SSL : {type : 'bool', default : false},
+  CWMP_LOG_FILE : {type: 'path', default : ''},
+  CWMP_ACCESS_LOG_FILE : {type : 'path', default : ''},
 
-  NBI_WORKER_PROCESSES : {type : 'int', default : 2},
+  NBI_WORKER_PROCESSES : {type : 'int', default : 0},
   NBI_PORT : {type : 'int', default : 7557},
   NBI_INTERFACE : {type : 'string', default : '0.0.0.0'},
   NBI_SSL : {type : 'bool', default : false},
+  NBI_LOG_FILE : {type: 'path', default : ''},
+  NBI_ACCESS_LOG_FILE : {type : 'path', default : ''},
 
-  FS_WORKER_PROCESSES : {type : 'int', default : 2},
+  FS_WORKER_PROCESSES : {type : 'int', default : 0},
   FS_PORT : {type : 'int', default : 7567},
   FS_INTERFACE : {type : 'string', default : '0.0.0.0'},
   FS_SSL : {type : 'bool', default : false},
-  FS_IP : {type : 'string', default : '192.168.0.1'},
+  FS_HOSTNAME : {type : 'string', default : 'acs.example.com'},
+  FS_LOG_FILE : {type: 'path', default : ''},
+  FS_ACCESS_LOG_FILE : {type : 'path', default : ''},
 
-  PRESETS_CACHE_DURATION : {type : 'int', default : 86400},
-  LOG_INFORMS : {type : 'bool', default : true},
+  UDP_CONNECTION_REQUEST_PORT : {type : 'int', default : 0},
+
+  DOWNLOAD_TIMEOUT: {type : 'int', default : 3600},
+  EXT_TIMEOUT: {type: 'int', default: 3000},
+  MAX_CACHE_TTL : {type : 'int', default : 86400},
   DEBUG : {type : 'bool', default : false},
   RETRY_DELAY : {type : 'int', default : 300},
-  IGNORE_XML_NAMESPACES : {type : 'bool', default : false},
   SESSION_TIMEOUT : {type : 'int', default : 30},
-  GET_PARAMETER_NAMES_DEPTH_THRESHOLD : {type : 'int', default : 0},
-  TASK_PARAMETERS_BATCH_SIZE : {type : 'int', default : 32},
+  CONNECTION_REQUEST_TIMEOUT : {type : 'int', default: 2000},
+  GPN_NEXT_LEVEL : {type : 'int', default : 0},
+  GPV_BATCH_SIZE : {type : 'int', default : 32},
+  MAX_DEPTH : {type: 'int', default : 16},
   COOKIES_PATH : {type : 'string'},
+  LOG_FORMAT : {type : 'string', default : 'simple'},
+  ACCESS_LOG_FORMAT : {type : 'string', default : ''},
+  MAX_CONCURRENT_REQUESTS : {type : 'int', default: 20},
+  DATETIME_MILLISECONDS : {type: 'bool', default: true},
+  BOOLEAN_LITERAL : {type: 'bool', default: true},
+  CONNECTION_REQUEST_ALLOW_BASIC_AUTH: {type: 'bool', default: false}
 
-  # Libxml related configuration
-  XML_PARSE_RECOVER : {type : 'bool'},
-  XML_PARSE_NOENT : {type : 'bool'},
-  XML_PARSE_NOBLANKS : {type : 'bool'},
-  XML_PARSE_NSCLEAN : {type : 'bool'},
-  XML_PARSE_NOCDATA : {type : 'bool'},
-  XML_PARSE_IGNORE_ENC : {type : 'bool'},
+  # XML configuration
+  XML_RECOVER : {type : 'bool', default : false},
+  XML_IGNORE_ENC : {type : 'bool', default : false},
+  XML_FORMAT : {type : 'bool', default : false},
+  XML_NO_DECL : {type : 'bool', default : false},
+  XML_NO_EMPTY : {type : 'bool', default : false},
+  XML_IGNORE_NAMESPACE : {type : 'bool', default : false},
 
   # Should probably never be changed
-  PRESETS_TIME_PADDING : {type : 'int', default : 1},
   DEVICE_ONLINE_THRESHOLD : {type : 'int', default : 4000}
 }
 
@@ -70,6 +87,28 @@ allConfig = {}
 setConfig = (name, value, commandLineArgument) ->
   return true if allConfig[name]?
 
+  # For compatibility with v1.0
+  if name in ['PRESETS_CACHE_DURATION', 'presets-cache-duration']
+    setConfig('MAX_CACHE_TTL', value)
+
+  if name in ['GET_PARAMETER_NAMES_DEPTH_THRESHOLD', 'get-parameter-names-depth-threshold']
+    setConfig('GPN_NEXT_LEVEL', value)
+
+  if name in ['TASK_PARAMETERS_BATCH_SIZE', 'task-parameters-batch-size']
+    setConfig('GPV_BATCH_SIZE', value)
+
+  if name in ['IGNORE_XML_NAMESPACES', 'ignore-xml-namespaces']
+    setConfig('XML_IGNORE_NAMESPACE', value)
+
+  if name in ['XML_PARSE_IGNORE_ENC', 'xml-parse-ignore-enc']
+    setConfig('XML_IGNORE_ENC', value)
+
+  if name in ['XML_PARSE_RECOVER', 'xml-parse-recover']
+    setConfig('XML_RECOVER', value)
+
+  if name in ['FS_IP', 'fs-ip']
+    setConfig('FS_HOSTNAME', value)
+
   cast = (val, type) ->
     switch type
       when 'int'
@@ -78,6 +117,8 @@ setConfig = (name, value, commandLineArgument) ->
         String(val).trim().toLowerCase() in ['true', 'on', 'yes', '1']
       when 'string'
         String(val)
+      when 'path'
+        if val then path.resolve(val) else ''
       else
         null
 
@@ -91,30 +132,26 @@ setConfig = (name, value, commandLineArgument) ->
     if name == n
       _value = cast(value, optionDetails.type)
       n = optionName
-    else if common.startsWith(name, "#{n}-")
+    else if name.startsWith("#{n}-")
       _value = cast(value, optionDetails.type)
       n = "#{optionName}-#{name[optionName.length+1..]}"
 
     if _value?
       allConfig[n] = _value
       # Save as environmnet variable to pass on to any child processes
-      process.env[n] = _value
+      process.env["GENIEACS_#{n}"] = _value
       return true
 
   return false
 
 
 # Command line arguments
-exports.argv = []
 argv = process.argv[2..]
 while argv.length
   arg = argv.shift()
   if arg[0] == '-'
     v = argv.shift()
-    exports.argv[arg] = v
     setConfig(arg[2..], v, true)
-  else
-    exports.argv.push(arg)
 
 
 # Environment variable
@@ -124,18 +161,14 @@ for k, v of process.env
   setConfig(k, v)
 
 
-# Find config dir
-if exports.argv['--config-dir']?
-  allConfig.CONFIG_DIR = exports.argv['--config-dir']
-else if process.env['GENIEACS_CONFIG_DIR']?
-  allConfig.CONFIG_DIR = process.env['GENIEACS_CONFIG_DIR']
-else
-  allConfig.CONFIG_DIR = options.CONFIG_DIR.default
-
+# Use default config dir if none defined
+setConfig('CONFIG_DIR', options['CONFIG_DIR'].default)
 
 # Configuration file
 for k, v of require(path.resolve(allConfig.CONFIG_DIR, 'config'))
-  setConfig(k, v)
+  if not setConfig(k, v)
+    # Pass as environment variable to be accessable by extensions
+    process.env["GENIEACS_#{k}"] = v
 
 
 # Defaults
@@ -158,14 +191,18 @@ get = (option, deviceId) ->
   return allConfig[option]
 
 
-# load parameter configurations
-exports.PARAMETERS = require(path.resolve(allConfig.CONFIG_DIR, 'parameters'))
+getDefault = (optionName) ->
+  option = options[optionName]
+  return null if not option
+  val = option.default
+  if val and option.type == 'path'
+    val = path.resolve(val)
+  return val
+
 
 # load authentication scripts
+try
   exports.auth = require(path.resolve(allConfig.CONFIG_DIR, 'auth'))
 
-exports.CUSTOM_COMMANDS = require(path.resolve(allConfig.CONFIG_DIR, 'custom_commands'))
-
 exports.get = get
-
-exports.allConfig = allConfig
+exports.getDefault = getDefault

lib/custom-commands.coffee

@@ -1,61 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-###
-
-path = require 'path'
-config = require './config'
-common = require './common'
-
-
-FILES = {}
-getFile = (filename) ->
-  if filename not of FILES
-    FILES[filename] = require(path.resolve(config.get('CONFIG_DIR'), "custom_commands/#{filename}"))
-
-  return FILES[filename]
-
-
-exports.execute = (deviceId, command, callback) ->
-  args = command.split(/\s+/)
-  f = args.shift()
-  c = args.shift()
-
-  file = getFile(f)
-  file[c](deviceId, args.join(' '), callback)
-
-
-exports.getFileCommands = (filename) ->
-  commands = []
-  f = getFile(filename)
-  for k,v of f
-    commands.push(k)
-  return commands
-
-
-exports.getDeviceCustomCommands = (deviceId) ->
-  commands = {}
-  for k,v of config.CUSTOM_COMMANDS
-    if eval(v).test(deviceId)
-      commands[k] = exports.getFileCommands(k)
-  return commands
-
-
-exports.getDeviceCustomCommandNames = (deviceId) ->
-  commands = []
-  for k,v of config.CUSTOM_COMMANDS
-    if eval(v).test(deviceId)
-      commands.push(k)
-  return commands

lib/db.coffee

@@ -1,5 +1,7 @@
 ###
-# Copyright 2013, 2014  Zaid Abdulla
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
 #
 # GenieACS is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as
@@ -17,42 +19,22 @@
 
 config = require './config'
 mongodb = require 'mongodb'
-redis = require('redis')
-parameters = require './parameters'
+common = require './common'
 
-redisClient = null
 tasksCollection = null
 devicesCollection = null
 presetsCollection = null
 objectsCollection = null
-
-
-# Only flush redis DB if aliases or custom commands config has changed
-flushRedis = (callback) ->
-  redisClient.get('config_hash', (err, res) ->
-    return callback(err) if err
-
-    hash = require('crypto')
-      .createHash('md5')
-      .update(JSON.stringify(config.PARAMETERS))
-      .update(JSON.stringify(config.CUSTOM_COMMANDS))
-      .digest('hex')
-
-    if res == hash
-      return callback()
-
-    redisClient.flushdb((err) ->
-      return callback(err) if err
-      redisClient.set('config_hash', hash, (err, res) ->
-        return callback(err)
-      )
-    )
-  )
+provisionsCollection = null
+virtualParametersCollection = null
+faultsCollection = null
+filesCollection = null
+operationsCollection = null
 
 
 connect = (callback) ->
-  callbackCounter = 6
-  mongodb.MongoClient.connect(config.get('MONGODB_CONNECTION_URL'), {db:{w:1},server:{autoReconnect:true}}, (err, db) ->
+  callbackCounter = 9
+  mongodb.MongoClient.connect(config.get('MONGODB_CONNECTION_URL'), (err, db) ->
     return callback(err) if err
     exports.mongoDb = db
     db.collection('tasks', (err, collection) ->
@@ -93,138 +75,578 @@ connect = (callback) ->
         return callback(err)
     )
 
-    exports.redisClient = redisClient = redis.createClient(config.get('REDIS_PORT'), config.get('REDIS_HOST'))
-    redisClient.select(config.get('REDIS_DB'), (err) ->
-      if err
-        callbackCounter = 0
-        return callback(err)
-
-      flushRedis((err) ->
+    db.collection('provisions', (err, collection) ->
+      exports.provisionsCollection = provisionsCollection = collection
       if --callbackCounter == 0 or err
         callbackCounter = 0
         return callback(err)
     )
+
+    db.collection('virtualParameters', (err, collection) ->
+      exports.virtualParametersCollection = virtualParametersCollection = collection
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
     )
+
+    db.collection('faults', (err, collection) ->
+      exports.faultsCollection = faultsCollection = collection
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
+    )
+
+    db.collection('operations', (err, collection) ->
+      exports.operationsCollection = operationsCollection = collection
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
+    )
+
   )
 
 
 disconnect = () ->
-  exports.mongoDb.close()
-  redisClient.quit()
+  exports.mongoDb?.close()
 
 
-getTask = (taskId, callback) ->
-  tid = String(taskId)
-  redisClient.get(tid, (err, res) ->
-    return callback(err) if err
-    if res?
-      callback(err, JSON.parse(res))
+# Optimize projection by removing overlaps
+# This can modify the object
+optimizeProjection = (obj) ->
+  if obj['']
+    return {'': obj['']}
+
+  keys = Object.keys(obj).sort()
+  return obj if keys.length <= 1
+  i = 1
+  while i < keys.length
+    a = keys[i-1]
+    b = keys[i]
+    if b.startsWith(a)
+      if b.charAt(a.length) == '.' or b.charAt(a.length - 1) == '.'
+        delete obj[b]
+        keys.splice(i--, 1)
+    ++ i
+  return obj
+
+
+fetchDevice = (id, timestamp, patterns, callback) ->
+  MAX_DEPTH = config.get('MAX_DEPTH', id)
+
+  if not patterns?.length
+    patterns = [[[], (1 << MAX_DEPTH) - 1]]
+
+  projection = {}
+  projectionTree = {}
+  func = (path, pats, projTree) ->
+    children = {}
+    for pat in pats
+      fragment = pat[0][path.length] or '*'
+
+      if fragment == '*'
+        projection[path.join('.')] = 1 if pat[1] << path.length
+        return
+
+      projTree[fragment] ?= {}
+
+      if pat[1] & (1 << path.length)
+        projection[path.concat('_timestamp').join('.')] = 1
+        projection[path.concat('_object').join('.')] = 1
+        projection[path.concat('_instance').join('.')] = 1
+        projection[path.concat([fragment, '_timestamp']).join('.')] = 1
+        projection[path.concat([fragment, '_value']).join('.')] = 1
+        projection[path.concat([fragment, '_type']).join('.')] = 1
+        projection[path.concat([fragment, '_object']).join('.')] = 1
+        projection[path.concat([fragment, '_instance']).join('.')] = 1
+        projection[path.concat([fragment, '_writable']).join('.')] = 1
+        projection[path.concat([fragment, '_orig']).join('.')] = 1
+
+      if pat[1] >> (path.length + 1)
+        children[fragment] ?= []
+        children[fragment].push(pat)
+
+    for k, v of children
+      func(path.concat(k), v, projTree[k])
+
+  func([], patterns, projectionTree)
+
+  delete projectionTree['DeviceID']
+  delete projectionTree['Events']
+  delete projectionTree['Tags']
+
+  res = []
+  loaded = []
+
+  projection = optimizeProjection(projection)
+
+  for k, v of projection
+    if k == '' or k == 'Events' or k.startsWith('Events.')
+      if k == '' or k == 'Events' or k == 'Events._writable'
+        res.push([['Events'], timestamp,
+          {object: [timestamp, 1], writable: [timestamp, 0]}])
+        loaded.push([['Events'], 1]) if k == 'Events._writable'
+      if k == 'Events'
+        projection['_lastInform'] = 1
+        projection['_lastBoot'] = 1
+        projection['_lastBootstrap'] = 1
+        projection['_registered'] = 1
+        loaded.push([['Events'], ((1 << MAX_DEPTH) - 1)])
+      else if k == 'Events.Inform._writable' or k == 'Events.Inform'
+        projection['_lastInform'] = 1
+        loaded.push([['Events', 'Inform'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k == 'Events.1_BOOT._writable' or k == 'Events.1_BOOT'
+        projection['_lastBoot'] = 1
+        loaded.push([['Events', '1_BOOT'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k == 'Events.0_BOOTSTRAP._writable' or k == 'Events.0_BOOTSTRAP'
+        projection['_lastBootstrap'] = 1
+        loaded.push([['Events', '0_BOOTSTRAP'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k == 'Events.Registered._writable' or k == 'Events.Registered'
+        projection['_registered'] = 1
+        loaded.push([['Events', 'Registered'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k.endsWith('._writable') and k != 'Events._writable'
+        loaded.push([k.split('.').slice(0, 2), 1 ^ ((1 << MAX_DEPTH) - 1)])
+      delete projection[k] if k != ''
+
+    if k == '' or k == 'DeviceID' or k.startsWith('DeviceID.')
+      if k == '' or k == 'DeviceID' or k == 'DeviceID._writable'
+        res.push([['DeviceID'], timestamp,
+          {object: [timestamp, 1], writable: [timestamp, 0]}])
+        loaded.push([['DeviceID'], 1]) if k == 'DeviceID._writable'
+      if k == 'DeviceID'
+        projection['_id'] = 1
+        projection['_deviceId'] = 1
+        loaded.push([['DeviceID'], ((1 << MAX_DEPTH) - 1)])
+      else if k == 'DeviceID.ID._writable' or k == 'DeviceID.ID'
+        projection['_id'] = 1
+        loaded.push([['DeviceID', 'ID'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k == 'DeviceID.Manufacturer._writable' or k == 'DeviceID.DeManufacturer'
+        projection['_deviceId._Manufacturer'] = 1
+        loaded.push([['DeviceID', 'Manufacturer'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k == 'DeviceID.ProductClass._writable' or k == 'DeviceID.ProductClass'
+        projection['_deviceId._ProductClass'] = 1
+        loaded.push([['DeviceID', 'ProductClass'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k == 'DeviceID.OUI._writable' or k == 'DeviceID.OUI'
+        projection['_deviceId._OUI'] = 1
+        loaded.push([['DeviceID', 'ProductClass'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k == 'DeviceID.SerialNumber._writable' or k == 'DeviceID.SerialNumber'
+        projection['_deviceId._SerialNumber'] = 1
+        loaded.push([['DeviceID', 'SerialNumber'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k.endsWith('._writable') and k != 'DeviceID._writable'
+        loaded.push([k.split('.').slice(0, 2), 1 ^ ((1 << MAX_DEPTH) - 1)])
+      delete projection[k] if k != ''
+
+    if k == 'Tags' or k.startsWith('Tags.')
+      if not projection['_tags']
+        projection['_tags'] = 1
+        loaded.push([['Tags'], ((1 << MAX_DEPTH) - 1)])
+      delete projection[k]
+
+  if projection['']
+    proj = {}
+  else if Object.keys(projection).length == 0
+    proj = {'_id': 1}
   else
-      tasksCollection.findOne({_id : mongodb.ObjectID(tid)}, (err, task) ->
-        callback(err, task)
-      )
-  )
+    proj = projection
 
+  devicesCollection.findOne({'_id' : id}, proj, (err, device) ->
+    return callback(err) if err or not device?
 
-getCached = (name, valueCallback, valueExpiry, callback) ->
-  redisClient.get(name, (err, res) ->
-    return callback(err) if err
-    return callback(null, JSON.parse(res)) if res?
-
-    lockAcquired = () ->
-      valueCallback((err, value) ->
-        return callback(err, value) if err
-        redisClient.setex(name, valueExpiry, JSON.stringify(value), (err) ->
-          return callback(err, value) if err
-          redisClient.del("lock.#{name}", (err) ->
-            callback(err, value)
-          )
-        )
-      )
-
-    redisClient.setnx("lock.#{name}", Date.now() + 30000, (err, res) ->
-      return callback(err) if err
-      if res == 1
-        lockAcquired()
+    storeParams = (obj, path, timestamp, descendantsFetched, projTree) ->
+      if descendantsFetched
+        thisFetched = true
       else
-        setTimeout(() ->
-          redisClient.get("lock.#{name}", (err, timestamp) ->
-            return callback(err) if err
-            now = Date.now()
-            if not timestamp?
-              getCached(name, valueCallback, valueExpiry, callback)
-            else if timestamp > now
-              setTimeout(() ->
-                getCached(name, valueCallback, valueExpiry, callback)
-              , 1000)
+        if projection[path.join('.')]
+          descendantsFetched = true
+          if path.length and projection[path.slice(0, -1).concat("_timestamp").join('.')]
+            thisFetched = true
+            loaded.push([path, ((1 << (path.length - 1)) - 1) ^ ((1 << MAX_DEPTH) - 1)])
           else
-              redisClient.getset("lock.#{name}", now + 30000, (err, timestamp) ->
-                return callback(err) if err
-                if timestamp > now
-                  setTimeout(() ->
-                    getCached(name, valueCallback, valueExpiry, callback)
-                  , 1000)
+            loaded.push([path, ((1 << path.length) - 1) ^ ((1 << MAX_DEPTH) - 1)])
+        else if projection[path.concat('_writable').join('.')]
+          loaded.push([path, 1 << (path.length - 1)])
+          thisFetched = true
+
+      if obj['_timestamp']?
+        obj['_timestamp'] = +obj['_timestamp']
+
+      # For compatibility with v1.0 database
+      if obj['_instance'] and not obj['_object']?
+        obj['_object'] = true
+
+      if thisFetched
+        attrs = {}
+
+        t = obj['_timestamp'] or 1
+        t = timestamp if timestamp > t
+
+        if obj['_value']?
+          attrs.value = [obj['_timestamp'] or 1, [obj['_value'], obj['_type']]]
+          attrs.value[1][0] = +attrs.value[1][0] if obj['_type'] == 'xsd:dateTime'
+          obj['_object'] = false
+
+        if obj['_writable']?
+          attrs.writable = [timestamp or 1, if obj['_writable'] then 1 else 0]
+
+        if obj['_object']?
+          attrs.object = [t, if obj['_object'] then 1 else 0]
+
+        res.push([path, t, attrs])
+
+      for k, v of obj
+        if not k.startsWith('_')
+          kk = k
+          obj['_object'] = true
+          storeParams(v, path.concat(k), obj['_timestamp'], descendantsFetched, projTree?[k])
+          delete projTree[kk] if projTree
+
+      if not descendantsFetched
+        for k, v of projTree
+          p = path.concat(k)
+          loaded.push([p, ((1 << path.length) - 1) ^ ((1 << MAX_DEPTH) - 1)])
+          if (obj['_object'] or path.length == 0) and obj['_timestamp']
+            res.push([p, obj['_timestamp']])
+      else if (obj['_object'] or path.length == 0) and obj['_timestamp']
+          res.push([path.concat('*'), obj['_timestamp']])
+
+    for k, v of device
+      switch k
+        when '_lastInform'
+          res.push([['Events', 'Inform'], +v,
+            {object: [+v, 0], writable: [+v, 0], value: [+v, [+v, 'xsd:dateTime']]}])
+          delete device[k]
+        when '_lastBoot'
+          res.push([['Events', '1_BOOT'], +v,
+            {object: [+v, 0], writable: [+v, 0], value: [+v, [+v, 'xsd:dateTime']]}])
+          delete device[k]
+        when '_lastBootstrap'
+          res.push([['Events', '0_BOOTSTRAP'], +v,
+            {object: [+v, 0], writable: [+v, 0], value: [+v, [+v, 'xsd:dateTime']]}])
+          delete device[k]
+        when '_registered'
+          # Use current timestamp for registered event attribute timestamps
+          res.push([['Events', 'Registered'], timestamp,
+            {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [+v, 'xsd:dateTime']]}])
+          delete device[k]
+        when '_id'
+          if projection[''] or projection['_id']
+            res.push([['DeviceID', 'ID'], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v, 'xsd:string']]}])
+          delete device[k]
+        when '_tags'
+          if v.length
+            res.push([['Tags'], timestamp,
+              {object: [timestamp, 1], writable: [timestamp, 0]}])
+          for t in v
+            t = t.replace(/[^a-zA-Z0-9\-]+/g, '_')
+            res.push([['Tags', t], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 1], value: [timestamp, [true, 'xsd:boolean']]}])
+          delete device[k]
+        when '_deviceId'
+          if v['_Manufacturer']?
+            res.push([['DeviceID', 'Manufacturer'], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v['_Manufacturer'], 'xsd:string']]}])
+          if v['_OUI']?
+            res.push([['DeviceID', 'OUI'], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v['_OUI'], 'xsd:string']]}])
+          if v['_ProductClass']?
+            res.push([['DeviceID', 'ProductClass'], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v['_ProductClass'], 'xsd:string']]}])
+          if v['_SerialNumber']?
+            res.push([['DeviceID', 'SerialNumber'], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v['_SerialNumber'], 'xsd:string']]}])
+          delete device[k]
+
+    storeParams(device, [], 0, false, projectionTree)
+
+    return callback(null, res, loaded)
+  )
+
+
+saveDevice = (deviceId, deviceData, isNew, sessionTimestamp, callback) ->
+  update = {'$set' : {}, '$unset' : {}, '$addToSet' : {}, '$pull' : {}}
+
+  iter = deviceData.timestamps.diff()
+  while diff = iter.next().value
+    continue if diff[0].wildcard != (1 << (diff[0].length - 1))
+    continue if diff[0][0] in ['Events', 'DeviceID', 'Tags']
+
+    # Param timestamps may be greater than session timestamp to track revisions
+    if diff[2] > sessionTimestamp
+      diff[2] = sessionTimestamp
+
+    if not diff[2]? and diff[1]?
+      update['$unset'][diff[0].slice(0, -1).concat('_timestamp').join('.')] = 1
+    else if diff[2] != diff[1]
+      parent = deviceData.paths.get(diff[0].slice(0, -1))
+      if parent and (parent.length == 0 or deviceData.attributes.has(parent))
+        update['$set'][diff[0].slice(0, -1).concat('_timestamp').join('.')] = new Date(diff[2])
+
+  iter = deviceData.attributes.diff()
+  while diff = iter.next().value
+    continue if diff[1] == diff[2]
+    path = diff[0]
+    switch path[0]
+      when 'Events'
+        if diff[0].length == 2 and diff[2]?.value?[1][0] != diff[1]?.value?[1][0]
+          if not diff[2]
+            switch path[1]
+              when 'Inform'
+                update['$unset']['_lastInform'] = 1
+              when '1_BOOT'
+                update['$unset']['_lastBoot'] = 1
+              when '0_BOOTSTRAP'
+                update['$unset']['_lastBootstrap'] = 1
+              when 'Registered'
+                update['$unset']['_registered'] = 1
           else
-                  lockAcquired()
-              )
-          )
-        , 1000)
-    )
+            t = new Date(diff[2].value[1][0])
+            switch path[1]
+              when 'Inform'
+                update['$set']['_lastInform'] = t
+              when '1_BOOT'
+                update['$set']['_lastBoot'] = t
+              when '0_BOOTSTRAP'
+                update['$set']['_lastBootstrap'] = t
+              when 'Registered'
+                update['$set']['_registered'] = t
+      when 'DeviceID'
+        if diff[2].value?[1]?[0] != diff[1]?.value?[1]?[0]
+          v = diff[2].value[1][0]
+          switch path[1]
+            when 'ID'
+              update['$set']['_id'] = v
+            when 'Manufacturer'
+              update['$set']['_deviceId._Manufacturer'] = v
+            when 'OUI'
+              update['$set']['_deviceId._OUI'] = v
+            when 'ProductClass'
+              update['$set']['_deviceId._ProductClass'] = v
+            when 'SerialNumber'
+              update['$set']['_deviceId._SerialNumber'] = v
+      when 'Tags'
+        if diff[2]?.value?[1]?[0] != diff[1]?.value?[1]?[0]
+          v = diff[2]?.value?[1][0]
+          if v?
+            update['$addToSet']['_tags'] ?= {'$each' : []}
+            update['$addToSet']['_tags']['$each'].push(diff[0][1])
+          else
+            update['$pull']['_tags'] ?= {'$in' : []}
+            update['$pull']['_tags']['$in'].push(diff[0][1])
+      else
+        if not diff[2]
+          update['$unset'][diff[0].join('.')] = 1
+          continue
+
+        for attrName of diff[2]
+          # Param timestamps may be greater than session timestamp to track revisions
+          if diff[2][attrName][0] > sessionTimestamp
+            diff[2][attrName][0] = sessionTimestamp
+
+          if diff[2][attrName][1]?
+            switch attrName
+              when 'value'
+                if diff[2].value[1][0] != diff[1]?.value?[1]?[0]
+                  if diff[2].value[1][1] == 'xsd:dateTime' and Number.isInteger(diff[2].value[1][0])
+                    update['$set'][path.concat('_value').join('.')] = new Date(diff[2].value[1][0])
+                  else
+                    update['$set'][path.concat('_value').join('.')] = diff[2].value[1][0]
+                if diff[2].value[1][1] != diff[1]?.value?[1]?[1]
+                  update['$set'][path.concat('_type').join('.')] = diff[2].value[1][1]
+                if diff[2].value[0] != diff[1]?.value?[0]
+                  update['$set'][path.concat('_timestamp').join('.')] = new Date(diff[2].value[0])
+              when 'object'
+                if not diff[1]?.object or diff[2].object[1] != diff[1].object?[1]
+                  update['$set'][path.concat('_object').join('.')] = !!diff[2].object[1]
+              when 'writable'
+                if not diff[1]?.writable or diff[2].writable[1] != diff[1].writable?[1]
+                  update['$set'][path.concat('_writable').join('.')] = !!diff[2].writable[1]
+
+        if diff[1]
+          for attrName of diff[1]
+            if diff[1][attrName][1]? and not diff[2]?[attrName]?[1]?
+              update['$unset'][path.concat("_#{attrName}").join('.')] = 1
+              if attrName is 'value'
+                update['$unset'][path.concat('_type').join('.')] = 1
+                update['$unset'][path.concat('_timestamp').join('.')] = 1
+
+  update['$unset'] = optimizeProjection(update['$unset'])
+
+  # Remove overlap possibly caused by parameters changing from objects
+  # to regular parameters or vice versa. Reason being that _timestamp
+  # represents two different things depending on whether the parameter
+  # is an object or not.
+  for k of update['$unset']
+    delete update['$unset'][k] if update['$set'][k]?
+
+  # Remove empty keys
+  for k of update
+    if k == '$addToSet'
+      for kk of update[k]
+        delete update[k][kk] if update[k][kk]['$each'].length == 0
+    else if k == '$pull'
+      for kk of update[k]
+        delete update[k][kk] if update[k][kk]['$in'].length == 0
+    delete update[k] if Object.keys(update[k]).length == 0
+
+  return callback() if Object.keys(update).length == 0
+
+  if update['$addToSet'] and update['$pull']
+    # Mongo doesn't allow $addToSet and $pull at the same time
+    update2 = {'$pull': update['$pull']}
+    delete update['$pull']
+
+  devicesCollection.update({'_id' : deviceId}, update, {upsert: isNew}, (err, result) ->
+    if not err and result.result.n != 1
+      return callback(new Error("Device #{deviceId} not found in database"))
+
+    if update2
+      return devicesCollection.update({'_id': deviceId}, update2, callback)
+
+    return callback(err)
   )
 
 
-getAliases = (callback) ->
-  getCached('aliases', parameters.compileAliases, config.get('PRESETS_CACHE_DURATION'), (err, res) ->
-    throw err if err
-    callback(res)
-  )
-
-
-getPresetsObjectsAliases = (callback) ->
-  redisClient.mget('presets', 'objects', 'aliases', (err, res) ->
-    presets = JSON.parse(res[0])
-    objects = JSON.parse(res[1])
-    aliases = JSON.parse(res[2])
-
-    if presets and objects and aliases
-      return callback(presets, objects, aliases)
-
-    if not presets
-      getCached('presets', (callback) ->
-        presetsCollection.find().toArray((err, res) ->
-          callback(err, res)
-        )
-      , config.get('PRESETS_CACHE_DURATION'), (err, res) ->
-        throw err if err
-        presets = res
-        callback(presets, objects, aliases) if objects and aliases
-      )
-
-    if not objects
-      getCached('objects', (callback) ->
-        objectsCollection.find().toArray((err, res) ->
+getFaults = (deviceId, callback) ->
+  faultsCollection.find({'_id' : {'$regex' : "^#{common.escapeRegExp(deviceId)}\\:"}}).toArray((err, res) ->
     return callback(err) if err
-          objs = {}
-          objs[r._id] = r for r in res
-          callback(null, objs)
-        )
-      , config.get('PRESETS_CACHE_DURATION'), (err, res) ->
-        throw err if err
-        objects = res
-        callback(presets, objects, aliases) if presets and aliases
-      )
 
-    if not aliases
-      getCached('aliases', parameters.compileAliases, config.get('PRESETS_CACHE_DURATION'), (err, res) ->
-        throw err if err
-        aliases = res
-        callback(presets, objects, aliases) if presets and objects
-      )
+    faults = {}
+    for r in res
+      channel = r._id.slice(deviceId.length + 1)
+      delete r._id
+      delete r.channel
+      delete r.device
+      r.timestamp = +r.timestamp
+      r.provisions = JSON.parse(r.provisions)
+      faults[channel] = r
+
+    callback(err, faults)
   )
 
 
-exports.getTask = getTask
-exports.getPresetsObjectsAliases = getPresetsObjectsAliases
-exports.getAliases = getAliases
+saveFault = (deviceId, channel, fault, callback) ->
+  fault = Object.assign({}, fault)
+  fault._id = "#{deviceId}:#{channel}"
+  fault.device = deviceId
+  fault.channel = channel
+  fault.timestamp = new Date(fault.timestamp)
+  fault.provisions = JSON.stringify(fault.provisions)
+  faultsCollection.save(fault, callback)
+
+
+deleteFault = (deviceId, channel, callback) ->
+  faultsCollection.remove({_id: "#{deviceId}:#{channel}"}, callback)
+
+
+getDueTasks = (deviceId, timestamp, callback) ->
+  cur = tasksCollection.find({'device' : deviceId}).sort(['timestamp'])
+
+  tasks = []
+
+  cur.nextObject(f = (err, task) ->
+    return callback(err) if err
+
+    if not task?
+      return callback(null, tasks, null)
+
+    task.timestamp = +task.timestamp if task.timestamp?
+    task.expiry = +task.expiry if task.expiry?
+
+    if task.timestamp >= timestamp
+      return callback(null, tasks, +task.timestamp)
+
+    task._id = String(task._id)
+    tasks.push(task)
+
+    # For API compatibility
+    if task.name is 'download' and task.file?
+      if mongodb.ObjectID.isValid(task.file)
+        q = {_id: {'$in' : [task.file, new mongodb.ObjectID(task.file)]}}
+      else
+        q = {_id: task.file}
+
+      filesCollection.find(q).toArray((err, res) ->
+        return callback(err) if err
+        if res[0]?
+          task.fileType ?= res[0].metadata.fileType
+          task.fileName ?= res[0].filename or res[0]._id.toString()
+        cur.nextObject(f)
+      )
+    else
+      cur.nextObject(f)
+  )
+
+
+clearTasks = (deviceId, taskIds, callback) ->
+  tasksCollection.remove({'_id' : {'$in' : (new mongodb.ObjectID(id) for id in taskIds)}}, callback)
+
+
+getOperations = (deviceId, callback) ->
+  operationsCollection.find({'_id' : {'$regex' : "^#{common.escapeRegExp(deviceId)}\\:"}}).toArray((err, res) ->
+    return callback(err) if err
+
+    operations = {}
+    for r in res
+      commandKey = r._id.slice(deviceId.length + 1)
+      delete r._id
+      r.timestamp = +r.timestamp
+      r.args = JSON.parse(r.args) if r.args
+      r.provisions = JSON.parse(r.provisions)
+      r.retries = JSON.parse(r.retries)
+      operations[commandKey] = r
+
+    callback(err, operations)
+  )
+
+
+saveOperation = (deviceId, commandKey, operation, callback) ->
+  operation = Object.assign({}, operation)
+  operation._id = "#{deviceId}:#{commandKey}"
+  operation.timestamp = new Date(operation.timestamp)
+  operation.provisions = JSON.stringify(operation.provisions)
+  operation.retries = JSON.stringify(operation.retries)
+  operation.args = JSON.stringify(operation.args)
+  operationsCollection.save(operation, callback)
+
+
+deleteOperation = (deviceId, commandKey, callback) ->
+  operationsCollection.remove({_id: "#{deviceId}:#{commandKey}"}, callback)
+
+
+getPresets = (callback) ->
+  presetsCollection.find().toArray(callback)
+
+
+getObjects = (callback) ->
+  objectsCollection.find().toArray(callback)
+
+
+getProvisions = (callback) ->
+  provisionsCollection.find().toArray(callback)
+
+
+getVirtualParameters = (callback) ->
+  virtualParametersCollection.find().toArray(callback)
+
+
+getFiles = (callback) ->
+  filesCollection.find().toArray(callback)
+
+
 exports.connect = connect
 exports.disconnect = disconnect
+exports.fetchDevice = fetchDevice
+exports.saveDevice = saveDevice
+exports.getFaults = getFaults
+exports.saveFault = saveFault
+exports.deleteFault = deleteFault
+exports.clearTasks = clearTasks
+exports.saveOperation = saveOperation
+exports.deleteOperation = deleteOperation
+exports.getPresets = getPresets
+exports.getObjects = getObjects
+exports.getProvisions = getProvisions
+exports.getVirtualParameters = getVirtualParameters
+exports.getFiles = getFiles
+exports.getDueTasks = getDueTasks
+exports.getOperations = getOperations

lib/default-provisions.js

@@ -0,0 +1,117 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const config = require("./config");
+const common = require("./common");
+const device = require("./device");
+const scheduling = require("./scheduling");
+
+const MAX_DEPTH = config.get("MAX_DEPTH");
+
+
+function refresh(sessionContext, provision, declarations, startRevision, endRevision) {
+  let path = common.parsePath(provision[1]).slice();
+  let l = path.length;
+  path.length = MAX_DEPTH;
+  path.fill("*", l);
+  let every = 1000 * (provision[2] || 1);
+  let offset = scheduling.variance(sessionContext.deviceId, every);
+  let t = scheduling.interval(sessionContext.timestamp, every, offset);
+
+  for (let i = l; i < path.length; ++ i)
+    declarations.push([path.slice(0, i), t, {object: 1, writable: 1, value: t}]);
+
+  return true;
+}
+
+
+function value(sessionContext, provision, declarations, startRevision, endRevision) {
+  let path = common.parsePath(provision[1]);
+  declarations.push([path, 1, {value: 1}, null, {value: [provision[2]]}]);
+
+  return true;
+}
+
+
+function tag(sessionContext, provision, declarations, startRevision, endRevision) {
+  let path = ["Tags", provision[1]];
+  declarations.push([path, 1, {value: 1}, null, {value: [provision[2]]}]);
+
+  return true;
+}
+
+
+function reboot(sessionContext, provision, declarations, startRevision, endRevision) {
+  declarations.push([["Reboot"], 1, {value: 1}, null, {value: [sessionContext.timestamp]}]);
+
+  return true;
+}
+
+
+function reset(sessionContext, provision, declarations, startRevision, endRevision) {
+  declarations.push([["FactoryReset"], 1, {value: 1}, null, {value: [sessionContext.timestamp]}]);
+
+  return true;
+}
+
+
+function download(sessionContext, provision, declarations, startRevision, endRevision) {
+  let alias = [["FileType"], provision[1] || "", ["FileName"], provision[2] || "",
+    ["TargetFileName"], provision[3] || ""];
+
+  declarations.push([["Downloads", alias], 1, {}, 1]);
+  declarations.push([["Downloads", alias, "Download"], 1, {value: 1}, null,
+    {value: [sessionContext.timestamp]}]);
+
+  return true;
+}
+
+
+function instances(sessionContext, provision, declarations, startRevision, endRevision) {
+  let count = Number(provision[2]);
+
+  if (Number.isNaN(count))
+    return true;
+
+  let path = common.parsePath(provision[1]);
+
+  if (provision[2][0] === "+" || provision[2][0] === "-") {
+    declarations.push([path, 1]);
+
+    if (endRevision === startRevision)
+      return false;
+
+    let unpacked = device.unpack(sessionContext.deviceData, path, startRevision + 1);
+    count = Math.max(0, unpacked.length + count);
+  }
+
+  declarations.push([path, 1, null, count]);
+
+  return true;
+}
+
+
+exports.refresh = refresh;
+exports.value = value;
+exports.tag = tag;
+exports.reboot = reboot;
+exports.reset = reset;
+exports.download = download;
+exports.instances = instances;

lib/device.js

@@ -0,0 +1,393 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const common = require('./common');
+
+
+const CHANGE_FLAGS = {
+  object: 2,
+  writable: 4,
+  value: 8
+};
+
+
+function parseBool(v) {
+  v = "" + v;
+  if (v === "true" || v === "TRUE" || v === "True" || v === "1")
+    return true;
+  else if (v === "false" || v === "FALSE" || v === "False" || v === "0")
+    return false;
+  else
+    return null;
+}
+
+
+function sanitizeParameterValue(parameterValue) {
+  if (parameterValue[0] != null) {
+    switch (parameterValue[1]) {
+      case 'xsd:boolean':
+        if (typeof parameterValue[0] !== 'boolean') {
+          let v = parseBool(parameterValue[0]);
+          if (v == null)
+            v = '' + parameterValue[0];
+          parameterValue[0] = v;
+        }
+        break;
+      case 'xsd:string':
+        if (typeof parameterValue[0] !== 'string') {
+          parameterValue[0] = '' + parameterValue[0];
+        }
+        break;
+      case 'xsd:int':
+      case 'xsd:unsignedInt':
+        if (typeof parameterValue[0] !== 'number') {
+          let v = parseInt(parameterValue[0]);
+          if (isNaN(v))
+            v = '' + parameterValue[0];
+          parameterValue[0] = v;
+        }
+        break;
+      case 'xsd:dateTime':
+        if (typeof parameterValue[0] !== 'number') {
+          // Don't use parseInt because it reads date string as a number
+          let v = +parameterValue[0];
+          if (isNaN(v)) {
+            v = Date.parse(parameterValue[0]);
+            if (isNaN(v))
+              v = '' + parameterValue[0];
+          }
+          parameterValue[0] = v;
+        }
+        break;
+      default:
+        if (parameterValue[1] != null)
+          throw new Error(`Parameter value type "${parameterValue[1]}" not recognized.`);
+
+        parameterValue[0] = JSON.parse(JSON.stringify(parameterValue[0]));
+        break;
+    }
+  }
+  return parameterValue;
+}
+
+
+function getAliasDeclarations(path, timestamp) {
+  var decs = [];
+
+  function recursive(pattern, prefix) {
+    var pattern = prefix.concat(pattern);
+
+    for (let i = prefix.length; i < pattern.length; ++ i) {
+      if (Array.isArray(pattern[i])) {
+        let pat = pattern[i];
+        pattern[i] = '*';
+        for (let j = 0; j < pat.length; j += 2)
+          recursive(pat[j], pattern.slice(0, i + 1));
+      }
+    }
+
+    if (prefix.length)
+      decs.push([pattern, timestamp, {value: timestamp}]);
+    else
+      decs.push([pattern, timestamp]);
+  }
+
+  recursive(path, []);
+
+  return decs;
+}
+
+
+function unpack(deviceData, path, revision) {
+  let allMatches = [];
+  if (!path.alias) {
+    for (let p of deviceData.paths.find(path, false, true))
+      if (deviceData.attributes.has(p, revision))
+        allMatches.push(p);
+  }
+  else {
+    let wildcardPath = path.slice();
+    wildcardPath.wildcard = path.wildcard;
+    for (let i = 0; i < wildcardPath.length; ++ i)
+      if (Array.isArray(wildcardPath[i])) {
+        wildcardPath[i] = '*';
+        wildcardPath.wildcard |= 1 << i;
+      }
+
+    for (let p of deviceData.paths.find(wildcardPath, false, true))
+      if (deviceData.attributes.has(p, revision))
+        allMatches.push(p);
+
+    for (let i = path.length - 1; i >= 0; -- i) {
+      if (Array.isArray(path[i])) {
+        for (let j = 0; j < path[i].length; j += 2) {
+          let p = wildcardPath.slice(0, i + 1).concat(path[i][j]);
+          p.alias = (wildcardPath.alias & ((1 << (i + 1)) - 1)) + path[i][j].alias << (i + 1);
+          p.wildcard = (wildcardPath.wildcard & ((1 << (i + 1)) - 1)) + path[i][j].wildcard << (i + 1);
+          let unpacked = unpack(deviceData, p, revision);
+          let filtered = [];
+          for (let up of unpacked) {
+            let attributes = deviceData.attributes.get(up, revision);
+            if (attributes && attributes.value && attributes.value[1] &&
+                sanitizeParameterValue([path[i][j + 1], attributes.value[1][1]])[0] === attributes.value[1][0]) {
+              for (let m = 0; m < allMatches.length; ++ m) {
+                let k;
+                for (k = i; k >= 0; -- k)
+                  if (allMatches[m][k] != up[k])
+                    break
+
+                if (k < 0) {
+                  filtered.push(allMatches[m]);
+                  allMatches[m] = [];
+                }
+              }
+            }
+          }
+          allMatches = filtered;
+        }
+      }
+    }
+  }
+
+  allMatches.sort(function(a, b) {
+    for (let i = 0; i < a.length; ++ i) {
+      if (a === b)
+        continue;
+
+      // Use numeric sorting for numbers
+      let ia = parseInt(a), ib = parseInt(b);
+      if (ia.toString() === a && ib.toString() === b) {
+        a = ia;
+        b = ib;
+      }
+
+      if (a < b)
+        return -1;
+      else
+        return 1;
+    }
+    return 0;
+  });
+
+  return allMatches;
+}
+
+
+function clear(deviceData, path, timestamp, attributes, changeFlags = 0) {
+  let changeTrackers = {};
+
+  timestamp = timestamp || 0;
+
+  let descendantsTimestamp = timestamp;
+  if (attributes && attributes.object) {
+    if (attributes.object > descendantsTimestamp)
+      descendantsTimestamp = attributes.object;
+    if (!(attributes.object <= attributes.value))
+      attributes.value = attributes.object;
+  }
+
+  for (let p of deviceData.paths.find(path, true, true, descendantsTimestamp ? 99 : path.length)) {
+    let tracker = deviceData.trackers.get(p);
+    for (let k in tracker)
+      changeTrackers[k] |= tracker[k];
+
+    let currentTimestamp = deviceData.timestamps.get(p);
+    if (currentTimestamp === undefined)
+      continue;
+
+    if (timestamp > currentTimestamp ||
+      (descendantsTimestamp > currentTimestamp && p.length > path.length)) {
+      deviceData.timestamps.delete(p);
+      deviceData.attributes.delete(p);
+      changeFlags |= 1;
+    }
+    else if (attributes && p.length == path.length) {
+      let currentAttributes = deviceData.attributes.get(p);
+      if (currentAttributes) {
+        let newAttrs;
+        for (let attrName in attributes) {
+          if (attrName in currentAttributes &&
+              attributes[attrName] > currentAttributes[attrName][0]) {
+            changeFlags |= CHANGE_FLAGS[attrName];
+            if (!newAttrs) {
+              newAttrs = Object.assign({}, currentAttributes);
+              deviceData.attributes.set(p, newAttrs);
+            }
+            delete newAttrs[attrName];
+          }
+        }
+      }
+    }
+  }
+
+  // Note: For performance, we're merging all changes together rather than
+  // mark changes based the exact parameters affected.
+  for (let k in changeTrackers) {
+    if (changeTrackers[k] & changeFlags)
+      deviceData.changes.add(k);
+  }
+}
+
+
+function compareEquality(a, b) {
+  var t = typeof a;
+  if (a === null ||
+      a === undefined ||
+      t === 'number' ||
+      t === 'boolean' ||
+      t === 'string' ||
+      t === 'symbol')
+    return a === b;
+
+  return JSON.stringify(a) === JSON.stringify(b);
+}
+
+
+function set(deviceData, path, timestamp, attributes, toClear) {
+  path = deviceData.paths.add(path);
+
+  let currentTimestamp = deviceData.timestamps.get(path);
+
+  let currentAttributes;
+
+  if (path.wildcard)
+    attributes = undefined;
+  else if (currentTimestamp)
+    currentAttributes = deviceData.attributes.get(path);
+
+  let changeFlags = 0;
+
+  if (attributes) {
+    if (attributes.value && attributes.value[1]
+        && attributes.value[0] >= (attributes.object ? attributes.object[0] : 0))
+      attributes.object = [attributes.value[0], 0];
+
+    if (attributes.object && attributes.object[1]
+      && attributes.object[0] >= (attributes.value ? attributes.value[0] : 0))
+      attributes.value = [attributes.object[0]];
+
+    let newAttributes = Object.assign({}, currentAttributes, attributes);
+
+    if (currentAttributes) {
+      for (let attrName in attributes) {
+        timestamp = Math.max(timestamp, attributes[attrName][0]);
+        if (!(attrName in currentAttributes))
+          changeFlags |= CHANGE_FLAGS[attrName];
+        else if (attributes[attrName][0] <= currentAttributes[attrName][0])
+          newAttributes[attrName] = currentAttributes[attrName];
+        else if (!compareEquality(attributes[attrName][1], currentAttributes[attrName][1]))
+          changeFlags |= CHANGE_FLAGS[attrName];
+      }
+    }
+    else
+      changeFlags |= 1;
+
+    deviceData.attributes.set(path, newAttributes);
+
+    if (!(timestamp <= currentTimestamp)) {
+      deviceData.timestamps.set(path, timestamp);
+      if (path.length > 1)
+        toClear = set(deviceData, path.slice(0, path.length - 1), timestamp,
+          {object: [timestamp, 1]}, toClear);
+    }
+  }
+  else if (!(timestamp <= currentTimestamp)) {
+    deviceData.timestamps.set(path, timestamp);
+
+    if (currentAttributes) {
+      deviceData.attributes.delete(path);
+      changeFlags |= 1;
+    }
+    else if (path.wildcard) {
+      for (let p of deviceData.paths.find(path, false, true, path.length))
+        if (timestamp > deviceData.timestamps.get(p)) {
+          toClear = toClear || [];
+          toClear.push([p, timestamp]);
+        }
+    }
+  }
+
+  if (changeFlags) {
+    if (changeFlags & 1) {
+      toClear = toClear || [];
+      toClear.push([path, timestamp, null, changeFlags]);
+    }
+    else if (changeFlags & CHANGE_FLAGS.object) {
+      toClear = toClear || [];
+      toClear.push([path, 0, {object: attributes.object[0]}, changeFlags]);
+    }
+    else {
+      for (let p of deviceData.paths.find(path, true, false, path.length)) {
+        let tracker = deviceData.trackers.get(p);
+        for (let k in tracker) {
+          if (tracker[k] & changeFlags)
+            deviceData.changes.add(k);
+        }
+      }
+    }
+  }
+
+  return toClear;
+}
+
+
+function track(deviceData, path, marker, attributes) {
+  path = deviceData.paths.add(path);
+  let f = 1;
+
+  if (attributes)
+    for (let attrName of attributes)
+      f |= CHANGE_FLAGS[attrName];
+
+  let cur = deviceData.trackers.get(path);
+  if (!cur) {
+    cur = {};
+    deviceData.trackers.set(path, cur);
+  }
+
+  cur[marker] |= f;
+}
+
+
+function clearTrackers(deviceData, tracker) {
+  if (Array.isArray(tracker)) {
+    deviceData.trackers.forEach(function(v, k) {
+      for (let t of tracker)
+        delete v[t];
+    });
+    for (let t of tracker)
+      deviceData.changes.delete(t);
+  }
+  else {
+    deviceData.trackers.forEach(function(v, k) {
+      delete v[tracker];
+    });
+    deviceData.changes.delete(tracker);
+  }
+}
+
+
+exports.sanitizeParameterValue = sanitizeParameterValue;
+exports.getAliasDeclarations = getAliasDeclarations;
+exports.unpack = unpack;
+exports.set = set;
+exports.clear = clear;
+exports.track = track;
+exports.clearTrackers = clearTrackers;

lib/extension-wrapper.js

@@ -0,0 +1,90 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const jobs = new Set();
+const fileName = process.argv[2];
+let script;
+
+function errorToFault(err) {
+  if (!err)
+    return null;
+
+  if (!err.name)
+    return {code: "ext", message: `${err}`};
+
+  let fault = {
+    code: `ext.${err.name}`,
+    message: err.message,
+    detail: {
+      name: err.name,
+      message: err.message
+    }
+  };
+
+  if (err.stack) {
+    fault.detail.stack = err.stack;
+    // Trim the stack trace
+    let stackTrimIndex = fault.detail.stack.match(/\s+at\s[^\s]+\s\(.*extension-wrapper\.js:.+\)/);
+    if (stackTrimIndex)
+      fault.detail.stack = fault.detail.stack.slice(0, stackTrimIndex.index);
+  }
+
+  return fault;
+}
+
+process.on("uncaughtException", function(err) {
+  let fault = errorToFault(err);
+  jobs.forEach(function(jobId) {
+    process.send([jobId, fault]);
+  });
+  jobs.clear();
+  process.disconnect();
+});
+
+process.on("message", function(message) {
+  jobs.add(message[0]);
+
+  if (!script) {
+    let cwd = process.env["GENIEACS_CONFIG_DIR"] + "/ext";
+    process.chdir(cwd);
+    script = require(`${cwd}/${fileName}`);
+  }
+
+  const funcName = message[1][0];
+
+  if (!script[funcName]) {
+    let fault = {
+      code: "ext",
+      message: `No such function '${funcName}' in extension '${fileName}'`
+    };
+    process.send([message[0], fault]);
+    return;
+  }
+
+  script[funcName](message[1].slice(1), function(err, res) {
+    if (!jobs.delete(message[0]))
+      return;
+
+    process.send([message[0], errorToFault(err), res]);
+  });
+});
+
+// Ignore SIGINT
+process.on('SIGINT', function() {});

lib/extensions.js

@@ -0,0 +1,117 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const childProcess = require("child_process");
+const crypto = require ("crypto");
+
+const config = require("./config");
+
+const TIMEOUT = config.get('EXT_TIMEOUT');
+
+const processes = {};
+const jobs = {};
+
+function messageHandler(message) {
+  let func = jobs[message[0]];
+  if (func) {
+    delete jobs[message[0]];
+    func(null, message[1], message[2]);
+  }
+}
+
+
+function run(args, callback) {
+  let scriptName = args[0];
+
+  if (!processes[scriptName]) {
+    let p = childProcess.fork('lib/extension-wrapper', [scriptName]);
+    processes[scriptName] = p;
+
+    p.on('error', function() {
+      if (processes[scriptName] === p) {
+        kill(processes[scriptName]);
+        delete processes[scriptName];
+      }
+    });
+
+    p.on('disconnect', function() {
+      if (processes[scriptName] === p) {
+        kill(processes[scriptName]);
+        delete processes[scriptName];
+      }
+    });
+
+    p.on('message', messageHandler);
+  }
+
+  let id = crypto.randomBytes(8).toString('hex');
+  jobs[id] = callback;
+  setTimeout(function() {
+    if (id in jobs) {
+      delete jobs[id];
+      return callback(null, {code: "timeout", message: "Extension timed out"});
+    }
+  }, TIMEOUT);
+
+  return processes[scriptName].send([id, args.slice(1)]);
+};
+
+
+function kill(process, callback) {
+  if (process.signalCode !== null || process.exitCode !== null)
+    return;
+
+  let timeToKill = Date.now() + 5000;
+
+  process.kill();
+
+  let t = setInterval(function() {
+    if (process.signalCode !== null || process.exitCode !== null) {
+      clearInterval(t);
+      if (callback)
+        callback();
+    }
+    else if (Date.now() > timeToKill) {
+      process.kill('SIGKILL');
+      clearInterval(t);
+      if (callback)
+        callback();
+    }
+  }, 100);
+}
+
+
+function killAll(callback) {
+  let counter = 1;
+  for (let n in processes) {
+    ++counter;
+    kill(processes[n], function() {
+      if (--counter == 0 && callback)
+        callback();
+    });
+    delete processes[n];
+  }
+  if (--counter == 0 && callback)
+    callback();
+}
+
+
+exports.run = run;
+exports.killAll = killAll;

lib/gpn-heuristic.js

@@ -0,0 +1,82 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const WILDCARD_MULTIPLIER = 2;
+const UNDISCOVERED_DEPTH = 7;
+
+// Simple heuristic to estimate GPN count given a set of patterns to be
+// discovered. Used to decide whether to use nextLevel = false in GPN.
+// gpnPatterns is [pattern, flags]
+// pattern is a path (array)
+// flags is an int where its bits mark the segments in the pattern that
+// need refreshing. Leading 0s indicate that the pattern up to that
+// point has been discovered.
+function estimateGpnCount(gpnPatterns, depth = 0) {
+  const children = {};
+  const wildcardChildren = [];
+  let wildcardDiscovered = false;
+  let gpnCount = 0;
+
+  for (let pattern of gpnPatterns) {
+    const path = pattern[0];
+    const flags = pattern[1] >> depth;
+
+    let k = path[depth];
+
+    if (!k) {
+      if (flags & 1)
+        gpnCount = 1;
+      continue;
+    }
+
+    if (flags & 1) {
+      gpnCount = 1;
+
+      if (depth > UNDISCOVERED_DEPTH)
+        continue;
+    }
+    else if (k === "*")
+      wildcardDiscovered = true;
+
+    if (k === "*") {
+      wildcardChildren.push(pattern);
+    }
+    else {
+      children[k] = children[k] || [];
+      children[k].push(pattern);
+    }
+  }
+
+  let wildcardGpnCount = 0;
+  if (!wildcardDiscovered && wildcardChildren.length)
+      wildcardGpnCount += estimateGpnCount(wildcardChildren, depth + 1) * WILDCARD_MULTIPLIER;
+
+  for (let k of Object.keys(children)) {
+    let c = estimateGpnCount(children[k].concat(wildcardChildren), depth + 1);
+    wildcardGpnCount -= c;
+    gpnCount += c;
+  }
+
+  gpnCount += Math.max(0, wildcardGpnCount);
+
+  return gpnCount;
+}
+
+exports.estimateGpnCount = estimateGpnCount;

lib/instance-set.js

@@ -0,0 +1,138 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+
+class InstanceSet {
+
+  constructor() {
+    this.set = new Set();
+  }
+
+  add(instance) {
+    this.set.add(instance);
+  }
+
+  delete(instance) {
+    this.set.delete(instance);
+  }
+
+  superset(instance) {
+    let res = [];
+    for (let inst of this.set) {
+      let match = true;
+      for (let k in instance) {
+        if (inst[k] !== instance[k]) {
+          match = false;
+          break;
+        }
+      }
+
+      if (match)
+        res.push(inst);
+    }
+
+    res.sort(function(a, b) {
+      let keysA = Object.keys(a);
+      let keysB = Object.keys(b);
+
+      if (keysA.length != keysB.length)
+        return keysB.length - keysA.length;
+
+      keysA.sort();
+      keysB.sort();
+
+      for (let i = 0; i < keysA.length; ++ i) {
+        if (keysA[i] > keysB[i])
+          return 1;
+        else if (keysA[i] < keysB[i])
+          return -1;
+        else if (a[keysA[i]] > b[keysB[i]])
+          return 1;
+        else if (a[keysA[i]] < b[keysB[i]])
+          return -1;
+      }
+
+      return 0;
+    });
+
+    return res;
+  }
+
+  subset(instance) {
+    let res = [];
+
+    for (let inst of this.set) {
+      let match = true;
+      for (let k in inst) {
+        if (inst[k] !== instance[k]) {
+          match = false;
+          break;
+        }
+      }
+
+      if (match)
+        res.push(inst);
+    }
+
+    res.sort(function(a, b) {
+      let keysA = Object.keys(a);
+      let keysB = Object.keys(b);
+
+      if (keysA.length != keysB.length)
+        return keysA.length - keysB.length;
+
+      keysA.sort();
+      keysB.sort();
+
+      for (let i = 0; i < keysA.length; ++ i) {
+        if (keysA[i] > keysB[i])
+          return 1;
+        else if (keysA[i] < keysB[i])
+          return -1;
+        else if (a[keysA[i]] > b[keysB[i]])
+          return 1;
+        else if (a[keysA[i]] < b[keysB[i]])
+          return -1;
+      }
+
+      return 0;
+    });
+
+    return res;
+  }
+
+  forEach(callback) {
+    return this.set.forEach(callback);
+  }
+
+  values() {
+    return this.set.values();
+  }
+
+  clear() {
+    this.set.clear();
+  }
+
+  get size() {
+    return this.set.size;
+  }
+}
+
+module.exports = InstanceSet;

lib/local-cache.js

@@ -0,0 +1,341 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const vm = require("vm");
+const crypto = require("crypto");
+
+const db = require("./db");
+const cache = require("./cache");
+const query = require("./query");
+const logger = require("./logger");
+const scheduling = require("./scheduling");
+
+const REFRESH = 3000;
+
+let nextRefresh = 1;
+let hash = null;
+let presets, provisions, virtualParameters, files;
+
+
+function computeHash() {
+  // MD5 hash for presets, provisions, virtual parameters for detecting changes
+  let h = crypto.createHash("md5");
+  for (let p of presets) {
+    h.update(JSON.stringify(p.name));
+    h.update(JSON.stringify(p.channel));
+    h.update(JSON.stringify(p.schedule));
+    h.update(JSON.stringify(p.events));
+    h.update(JSON.stringify(p.precondition));
+    h.update(JSON.stringify(p.provisions));
+  }
+
+  let keys;
+
+  keys = Object.keys(provisions).sort();
+  h.update(JSON.stringify(keys));
+  for (let k of keys)
+    h.update(provisions[k].md5);
+
+  keys = Object.keys(virtualParameters).sort();
+  h.update(JSON.stringify(keys));
+  for (let k of keys)
+    h.update(virtualParameters[k].md5);
+
+  hash = h.digest("hex");
+}
+
+
+function refresh(callback) {
+  if (!nextRefresh)
+    return setTimeout(function() {refresh(callback);}, 20);
+
+  nextRefresh = 0;
+  const now = Date.now();
+
+  cache.get("presets_hash", function(err, res) {
+    if (err)
+      return callback(err);
+
+    if (hash && res === hash) {
+      nextRefresh = now + (REFRESH - (now % REFRESH));
+      return callback();
+    }
+
+    cache.lock("presets_hash_lock", 3, function(err, unlockOrExtend) {
+      if (err)
+        return callback(err);
+
+      let counter = 3;
+
+      counter += 2;
+      db.getPresets(function(err, res) {
+        if (err) {
+          if (counter & 1)
+            callback(err);
+          return counter = 0;
+        }
+
+        db.getObjects(function(err, objects) {
+          if (err) {
+            if (counter & 1)
+              callback(err);
+            return counter = 0;
+          }
+
+          // If no keys are defined, consider all parameters as keys to keep the
+          // same behavior from v1.0
+          for (let obj of objects) {
+            if (!obj["_keys"] || !obj["_keys"].length)
+              obj["_keys"] = Object.keys(obj).filter(k => !k.startsWith("_"));
+          }
+
+          res.sort((a, b) => {
+            if (a.weight === b.weight)
+              return a._id > b._id;
+            else
+              return a.weight - b.weight;
+          });
+
+          presets = [];
+          for (let preset of res) {
+            let schedule = null;
+            if (preset.schedule) {
+              let parts = preset.schedule.trim().split(/\s+/);
+              schedule = {
+                md5: crypto.createHash("md5").update(preset.schedule).digest("hex")
+              };
+
+              try {
+                schedule.duration = +(parts.shift()) * 1000;
+                schedule.schedule = scheduling.parseCron(parts.join(" "));
+              }
+              catch (err) {
+                logger.warn({
+                  message: "Invalid preset schedule",
+                  preset: preset._id,
+                  schedule: preset.schedule
+                });
+                schedule.schedule = false;
+              }
+            }
+
+            let events = preset.events || {};
+            let precondition = query.convertMongoQueryToFilters(JSON.parse(preset.precondition));
+            let _provisions = preset.provisions || [];
+
+            // Generate provisions from the old configuration format
+            for (let c of preset.configurations) {
+              switch (c.type) {
+                case "age":
+                  _provisions.push(["refresh", c.name, c.age]);
+                  break;
+
+                case "value":
+                  _provisions.push(["value", c.name, c.value]);
+                  break;
+
+                case "add_tag":
+                  _provisions.push(["tag", c.tag, true]);
+                  break;
+
+                case "delete_tag":
+                  _provisions.push(["tag", c.tag, false]);
+                  break;
+
+                case "provision":
+                  _provisions.push([c.name].concat(c.args || []));
+                  break;
+
+                case "add_object":
+                  for (let obj of objects) {
+                    if (obj["_id"] === c.object) {
+                      let alias = obj["_keys"].map((k) => `${k}:${JSON.stringify(obj[k])}`).join(",");
+                      let p = `${c.name}.[${alias}]`;
+                      _provisions.push(["instances", p, 1]);
+
+                      for (let k in obj)
+                        if (!k.startsWith("_") && !(obj["_keys"].indexOf(k) !== -1))
+                          _provisions.push(["value", `${p}.${k}`, obj[k]]);
+                    }
+                  }
+                  break;
+
+                case "delete_object":
+                  for (let obj of objects) {
+                    if (obj["_id"] === c.object) {
+                      let alias = obj["_keys"].map((k) => `${k}:${JSON.stringify(obj[k])}`).join(",");
+                      let p = `${c.name}.[${alias}]`;
+                      _provisions.push(["instances", p, 0]);
+                    }
+                  }
+                  break;
+
+                default:
+                  if (counter & 1)
+                    callback(new Error(`Unknown configuration type ${c.type}`));
+                  return counter = 0;
+                  break;
+              }
+            }
+
+            presets.push({
+              name: preset._id,
+              channel: preset.channel || "default",
+              schedule: schedule,
+              events: events,
+              precondition: precondition,
+              provisions: _provisions
+            });
+          }
+
+          if ((counter -= 2) === 1) {
+            computeHash();
+            cache.set("presets_hash", hash, 300, function(err) {
+              unlockOrExtend(0);
+              nextRefresh = now + (REFRESH - (now % REFRESH));
+              callback(err);
+            });
+          }
+        });
+      });
+
+      counter += 2;
+      db.getProvisions(function(err, res) {
+        if (err) {
+          if (counter & 1)
+            callback(err);
+          return counter = 0;
+        }
+
+        provisions = {};
+        for (let r of res) {
+          provisions[r._id] = {};
+          provisions[r._id].md5 = crypto.createHash("md5").update(r.script).digest("hex");
+          provisions[r._id].script = new vm.Script(`"use strict";(function(){\n${r.script}\n})();`, {filename: r._id, lineOffset: -1, timeout: 50});
+        }
+
+        if ((counter -= 2) === 1) {
+          computeHash();
+          cache.set("presets_hash", hash, 300, function(err) {
+            unlockOrExtend(0);
+            nextRefresh = now + (REFRESH - (now % REFRESH));
+            callback(err);
+          });
+        }
+      });
+
+      counter += 2;
+      db.getVirtualParameters(function(err, res) {
+        if (err) {
+          if (counter & 1)
+            callback(err);
+          return counter = 0;
+        }
+
+        virtualParameters = {};
+        for (let r of res) {
+          virtualParameters[r._id] = {}
+          virtualParameters[r._id].md5 = crypto.createHash("md5").update(r.script).digest("hex");
+          virtualParameters[r._id].script = new vm.Script(`"use strict";(function(){\n${r.script}\n})();`, {filename: r._id, lineOffset: -1, timeout: 50});
+        }
+
+        if ((counter -= 2) === 1) {
+          computeHash();
+          cache.set("presets_hash", hash, 300, function(err) {
+            unlockOrExtend(0);
+            nextRefresh = now + (REFRESH - (now % REFRESH));
+            callback(err);
+          });
+        }
+      });
+
+      counter += 2;
+      db.getFiles(function(err, res) {
+        if (err) {
+          if (counter & 1)
+            callback(err);
+          return counter = 0;
+        }
+
+        files = {};
+        for (let r of res) {
+          const id = r.filename || r._id.toString();
+          files[id] = {};
+          files[id].length = r.length;
+          files[id].md5 = r.md5;
+          files[id].contentType = r.contentType;
+        }
+
+        if ((counter -= 2) === 1) {
+          computeHash();
+          cache.set("presets_hash", hash, 300, function(err) {
+            unlockOrExtend(0);
+            nextRefresh = now + (REFRESH - (now % REFRESH));
+            callback(err);
+          });
+        }
+      });
+
+      if ((counter -= 2) === 1) {
+        computeHash();
+        cache.set("presets_hash", hash, 300, function(err) {
+          unlockOrExtend(0);
+          nextRefresh = now + (REFRESH - (now % REFRESH));
+          callback(err);
+        });
+      }
+    });
+  });
+}
+
+
+function getPresets(callback) {
+  if (Date.now() < nextRefresh)
+    return callback(null, hash, presets);
+
+  refresh(function(err) {
+    callback(err, hash, presets);
+  });
+}
+
+
+function getFiles(callback) {
+  if (Date.now() < nextRefresh)
+    return callback(null, hash, files);
+
+  refresh(function(err) {
+    callback(err, hash, files);
+  })
+}
+
+
+function getProvisionsAndVirtualParameters(callback) {
+  if (Date.now() < nextRefresh)
+    return callback(null, hash, provisions, virtualParameters);
+
+  refresh(function(err) {
+    callback(err, hash, provisions, virtualParameters);
+  });
+}
+
+
+exports.getPresets = getPresets;
+exports.getFiles = getFiles;
+exports.getProvisionsAndVirtualParameters = getProvisionsAndVirtualParameters;

lib/logger.js

@@ -0,0 +1,320 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const fs = require("fs");
+const os = require("os");
+
+const config = require("./config");
+
+const REOPEN_EVERY = 60000;
+
+const LOG_FORMAT = config.get("LOG_FORMAT");
+const ACCESS_LOG_FORMAT = config.get("ACCESS_LOG_FORMAT") || LOG_FORMAT;
+
+const defaultMeta = {};
+
+let LOG_SYSTEMD = false;
+let ACCESS_LOG_SYSTEMD = false;
+
+let LOG_FILE, ACCESS_LOG_FILE;
+
+let logStream = fs.createWriteStream(null, {fd: process.stderr.fd});
+let logStat = fs.fstatSync(logStream.fd);
+let accessLogStream = fs.createWriteStream(null, {fd: process.stdout.fd});
+let accessLogStat = fs.fstatSync(accessLogStream.fd);
+
+
+// Reopen if original files have been moved (e.g. logrotate)
+function reopen() {
+  let counter = 1;
+
+  if (LOG_FILE) {
+    ++ counter;
+    fs.stat(LOG_FILE, function(err, stat) {
+      if (err && !err.message.startsWith("ENOENT:"))
+        throw err;
+
+      if (!(stat && stat.dev === logStat.dev && stat.ino === logStat.ino)) {
+        logStream.end();
+        logStream = fs.createWriteStream(null,
+          {fd: fs.openSync(LOG_FILE, "a")});
+        logStat = fs.fstatSync(logStream.fd);
+      }
+
+      if (-- counter === 0)
+        setTimeout(reopen, REOPEN_EVERY - (Date.now() % REOPEN_EVERY)).unref();
+    });
+  }
+
+  if (ACCESS_LOG_FILE) {
+    ++ counter;
+    fs.stat(ACCESS_LOG_FILE, function(err, stat) {
+      if (err && !err.message.startsWith("ENOENT:"))
+        throw err;
+
+      if (!(stat && stat.dev === accessLogStat.dev && stat.ino === accessLogStat.ino)) {
+        accessLogStream.end();
+        accessLogStream = fs.createWriteStream(null,
+          {fd: fs.openSync(ACCESS_LOG_FILE, "a")});
+        accessLogStat = fs.fstatSync(accessLogStream.fd);
+      }
+
+      if (-- counter === 0)
+        setTimeout(reopen, REOPEN_EVERY - (Date.now() % REOPEN_EVERY)).unref();
+    });
+  }
+
+  if (-- counter === 0)
+    setTimeout(reopen, REOPEN_EVERY - (Date.now() % REOPEN_EVERY)).unref();
+}
+
+
+function init(service, version) {
+  defaultMeta.hostname = os.hostname();
+  defaultMeta.pid = process.pid;
+  defaultMeta.name = `genieacs-${service}`;
+  defaultMeta.version = version;
+
+  LOG_FILE = config.get(`${service.toUpperCase()}_LOG_FILE`);
+  ACCESS_LOG_FILE = config.get(`${service.toUpperCase()}_ACCESS_LOG_FILE`);
+
+  if (LOG_FILE) {
+    logStream = fs.createWriteStream(null,
+      {fd: fs.openSync(LOG_FILE, "a")});
+    logStat = fs.fstatSync(logStream.fd);
+  }
+
+  if (ACCESS_LOG_FILE) {
+    accessLogStream = fs.createWriteStream(null,
+      {fd: fs.openSync(ACCESS_LOG_FILE, "a")});
+    accessLogStat = fs.fstatSync(accessLogStream.fd);
+  }
+
+  // Determine if logs are going to journald
+  const JOURNAL_STREAM = process.env["JOURNAL_STREAM"];
+
+  if (JOURNAL_STREAM) {
+    let [dev, inode] = JOURNAL_STREAM.split(":");
+    dev = parseInt(dev);
+    inode = parseInt(inode);
+
+    LOG_SYSTEMD = (logStat.dev === dev && logStat.ino === inode);
+    ACCESS_LOG_SYSTEMD = (accessLogStat.dev === dev && accessLogStat.ino === inode);
+  }
+
+  if (LOG_FILE || ACCESS_LOG_FILE) {
+    // Can't use setInterval as we need all workers to cehck at the same time
+    setTimeout(reopen, REOPEN_EVERY - (Date.now() % REOPEN_EVERY)).unref();
+  }
+}
+
+
+function close() {
+  accessLogStream.end();
+  logStream.end();
+}
+
+
+function flatten(details) {
+  if (details.sessionContext) {
+    details.deviceId = details.sessionContext.deviceId;
+    details.remoteAddress = details.sessionContext.httpRequest.connection.remoteAddress;
+    delete details.sessionContext;
+  }
+
+  if (details.exception) {
+    details.exceptionName = details.exception.name;
+    details.exceptionMessage = details.exception.message;
+    details.exceptionStack = details.exception.stack;
+    delete details.exception;
+  }
+
+  if (details.task) {
+    details.taskId = details.task._id;
+    delete details.task;
+  }
+
+  if (details.rpc) {
+    if (details.rpc.acsRequest) {
+      details.acsRequestId = details.rpc.id;
+      details.acsRequestName = details.rpc.acsRequest.name;
+      if (details.rpc.acsRequest.commandKey)
+        details.acsRequestCommandKey = details.rpc.acsRequest.commandKey;
+    }
+    else if (details.rpc.cpeRequest) {
+      details.cpeRequestId = details.rpc.id;
+      if (details.rpc.cpeRequest.name === "Inform") {
+        details.informEvent = details.rpc.cpeRequest.event.join(",");
+        details.informRetryCount = details.rpc.cpeRequest.retryCount;
+      }
+      else {
+        details.cpeRequestName = details.rpc.cpeRequest.name;
+        if (details.rpc.cpeRequest.commandKey)
+          details.cpeRequestCommandKey = details.rpc.cpeRequest.commandKey;
+      }
+    }
+    else if (details.rpc.cpeFault) {
+      details.acsRequestId = details.rpc.id;
+      details.cpeFaultCode = details.rpc.cpeFault.detail.faultCode;
+      details.cpeFaultString = details.rpc.cpeFault.detail.faultString;
+    }
+    delete details.rpc;
+  }
+
+  if (details.fault) {
+    details.faultCode = details.fault.code;
+    details.faultMessage = details.fault.message;
+    delete details.fault;
+  }
+
+  return details;
+}
+
+
+function formatJson(details, systemd) {
+  if (systemd) {
+    let severity = "";
+    if (details.severity === "info")
+      severity = "<6>";
+    else if (details.severity === "warn")
+      severity = "<4>";
+    else if (details.severity === "error")
+      severity = "<3>";
+
+    return `${severity}${JSON.stringify(flatten(details))}${os.EOL}`;
+  }
+
+  return `${JSON.stringify(flatten(details))}${os.EOL}`;
+}
+
+
+function formatSimple(details, systemd) {
+  flatten(details);
+
+  let remote = "";
+  if (details.remoteAddress) {
+    if (details.deviceId)
+      remote = `${details.remoteAddress} ${details.deviceId}: `;
+    else
+      remote = `${details.remoteAddress}: `;
+  }
+
+  const keys = Object.keys(details);
+  const skip = {
+    deviceId: 1,
+    remoteAddress: 1,
+    severity: 1,
+    timestamp: 1,
+    message: 1
+  };
+
+  let meta = "";
+
+  let kv = [];
+  for (let k of keys)
+    if (!skip[k])
+      kv.push(`${k}=${JSON.stringify(details[k])}`);
+
+  if (kv.length)
+    meta = `; ${kv.join(" ")}`;
+
+  if (systemd) {
+    let severity = "";
+    if (details.severity === "info")
+      severity = "<6>";
+    else if (details.severity === "warn")
+      severity = "<4>";
+    else if (details.severity === "error")
+      severity = "<3>";
+
+    return `${severity}${remote}${details.message}${meta}${os.EOL}`;
+  }
+
+  return `${details.timestamp} [${details.severity.toUpperCase()}] ${remote}${details.message}${meta}${os.EOL}`;
+}
+
+
+function log(details) {
+  details.timestamp = new Date().toISOString();
+  if (LOG_FORMAT === "json") {
+    details = Object.assign({}, defaultMeta, details);
+    logStream.write(formatJson(details, LOG_SYSTEMD));
+  } else {
+    logStream.write(formatSimple(details, LOG_SYSTEMD));
+  }
+}
+
+
+function info(details) {
+  details.severity = "info";
+  log(details);
+}
+
+
+function warn(details) {
+  details.severity = "warn";
+  log(details);
+}
+
+
+function error(details) {
+  details.severity = "error";
+  log(details);
+}
+
+
+function accessLog(details) {
+  details.timestamp = new Date().toISOString();
+  if (ACCESS_LOG_FORMAT === "json") {
+    Object.assign(details, defaultMeta);
+    accessLogStream.write(formatJson(details, ACCESS_LOG_SYSTEMD));
+  } else {
+    accessLogStream.write(formatSimple(details, ACCESS_LOG_SYSTEMD));
+  }
+}
+
+
+function accessInfo(details) {
+  details.severity = "info";
+  accessLog(details);
+}
+
+
+function accessWarn(details) {
+  details.severity = "warn";
+  accessLog(details);
+}
+
+
+function accessError(details) {
+  details.severity = "error";
+  accessLog(details);
+}
+
+
+exports.init = init;
+exports.close = close;
+exports.flatten = flatten;
+exports.info = info;
+exports.warn = warn;
+exports.error = error;
+exports.accessInfo = accessInfo;
+exports.accessWarn = accessWarn;
+exports.accessError = accessError;

lib/mongo-query.coffee

@@ -1,141 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-#
-# This file incorporates work covered by the following copyright and
-# permission notice:
-#
-# Copyright 2013 Fanoos Telecom
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-###
-
-common = require './common'
-
-
-test = (obj, query) ->
-  for k,v of query
-    if k.charAt(0) == '$' # this is a logical operator
-      res = testExpressions(obj, v, k)
-    else
-      value = common.getParamValueFromPath(obj, k)
-
-      if common.typeOf(v) isnt common.OBJECT_TYPE
-        # TODO comparing array to regex, array to array, and object to object
-        if common.typeOf(value) is common.ARRAY_TYPE
-          res = value.indexOf(v) != -1
-        else
-          if common.typeOf(v) is common.REGEXP_TYPE
-            res = v.test(value)
-          else
-            res = v == value
-      else
-        for k2,v2 of v
-          switch k2
-            when '$ne'
-              if common.typeOf(value) is common.ARRAY_TYPE
-                res = value.indexOf(v2) == -1
-              else
-                res = value != v2
-            when '$lt'
-              res = value < v2
-            when '$lte'
-              res = value <= v2
-            when '$gt'
-              res = value > v2
-            when '$gte'
-              res = value >= v2
-            when '$regex'
-              res = v2.test(value)
-            when '$in'
-              throw new Error('Operator not supported')        
-            when '$nin'
-              throw new Error('Operator not supported')        
-            when '$all'
-              throw new Error('Operator not supported')        
-            when '$exists'
-              throw new Error('Operator not supported')        
-            else
-              throw new Error('Operator not supported')        
-
-    if not res
-      return false
-
-  return true
-
-
-testExpressions = (obj, expressions, lop) ->
-  for f in expressions
-    res = test(obj, f)
-
-    switch lop
-      when '$and'
-        return false if not res
-      when '$or'
-        return true if res
-      when '$nor'
-        return false if res
-
-  return switch lop
-    when '$and' then true
-    when '$or' then false
-    when '$nor' then true
-    else throw new Error('Unknown logical operator')
-
-
-# generate field projection from a query
-projection = (query, proj) ->
-  for k,v of query
-    if k.charAt(0) == '$' # this is a logical operator
-      for q in v
-        projection(q, proj)
-    else
-      proj[k] = 1
-
-# optimize projection by removing overlaps
-optimizeProjection = (obj) ->
-  keys = Object.keys(obj).sort()
-  return if keys.length <= 1
-  i = 1
-  while i < keys.length
-    a = keys[i-1]
-    b = keys[i]
-    if common.startsWith(b, a)
-      if b.charAt(a.length) == '.' or b.charAt(a.length - 1) == '.'
-        delete obj[b]
-        keys.splice(i--, 1)
-    ++ i
-  return
-
-
-exports.test = test
-exports.projection = projection
-exports.optimizeProjection = optimizeProjection

lib/nbi.coffee

@@ -1,5 +1,7 @@
 ###
-# Copyright 2013, 2014  Zaid Abdulla
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
 #
 # GenieACS is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as
@@ -38,15 +40,19 @@
 # THE SOFTWARE.
 ###
 
-config = require './config'
 url = require 'url'
-common = require './common'
-db = require './db'
 mongodb = require 'mongodb'
 querystring = require 'querystring'
+vm = require 'vm'
+
+config = require './config'
+common = require './common'
+db = require './db'
 query = require './query'
 apiFunctions = require './api-functions'
-presets = require './presets'
+cache = require './cache'
+
+VERSION = require('../package.json').version
 
 # regular expression objects
 DEVICE_TASKS_REGEX = /^\/devices\/([a-zA-Z0-9\-\_\%]+)\/tasks\/?$/
@@ -55,20 +61,29 @@ TAGS_REGEX = /^\/devices\/([a-zA-Z0-9\-\_\%]+)\/tags\/([a-zA-Z0-9\-\_\%]+)\/?$/
 PRESETS_REGEX = /^\/presets\/([a-zA-Z0-9\-\_\%]+)\/?$/
 OBJECTS_REGEX = /^\/objects\/([a-zA-Z0-9\-\_\%]+)\/?$/
 FILES_REGEX = /^\/files\/([a-zA-Z0-9\%\!\*\'\(\)\;\:\@\&\=\+\$\,\?\#\[\]\-\_\.\~]+)\/?$/
-PING_REGEX = /^\/ping\/([a-zA-Z0-9\-\_\.]+)\/?$/
-QUERY_REGEX = /^\/([a-zA-Z0-9_]+s)\/?$/
-DEVICE_PRESET_REGEX = /^\/devices\/([a-zA-Z0-9\-\_\%]+)\/preset\/?$/
+PING_REGEX = /^\/ping\/([a-zA-Z0-9\-\_\.\:]+)\/?$/
+QUERY_REGEX = /^\/([a-zA-Z0-9_]+)\/?$/
 DELETE_DEVICE_REGEX = /^\/devices\/([a-zA-Z0-9\-\_\%]+)\/?$/
+PROVISIONS_REGEX = /^\/provisions\/([a-zA-Z0-9\-\_\%]+)\/?$/
+VIRTUAL_PARAMETERS_REGEX = /^\/virtual_parameters\/([a-zA-Z0-9\-\_\%]+)\/?$/
+FAULTS_REGEX = /^\/faults\/([a-zA-Z0-9\-\_\%\:]+)\/?$/
 
 
-errorToString = (err) ->
-  "#{err.name}: #{err.message}"
+
+throwError = (err, httpResponse) ->
+  if httpResponse
+    httpResponse.writeHead(500, {'Connection' : 'close'})
+    httpResponse.end("#{err.name}: #{err.message}")
+
+  throw err
 
 
 listener = (request, response) ->
   chunks = []
   bytes = 0
 
+  response.setHeader('GenieACS-Version', VERSION)
+
   request.addListener 'data', (chunk) ->
     chunks.push(chunk)
     bytes += chunk.length
@@ -93,34 +108,22 @@ listener = (request, response) ->
         preset._id = presetName
 
         db.presetsCollection.save(preset, (err) ->
-          db.redisClient.del('presets', 'presets_hash', (err) ->
-            throw err if err
-          )
-          if err
-            response.writeHead(500)
-            response.end(errorToString(err))
-            return
+          return throwError(err, response) if err
+          cache.del('presets_hash', (err) ->
+            return throwError(err, response) if err
             response.writeHead(200)
             response.end()
           )
+        )
       else if request.method == 'DELETE'
-        db.presetsCollection.remove({'_id' : presetName}, (err, removedCount) ->
-          db.redisClient.del('presets', 'presets_hash', (err) ->
-            throw err if err
-          )
-          if err
-            response.writeHead(500)
-            response.end(errorToString(err))
-            return
-
-          if removedCount == 0
-            response.writeHead(404)
-            response.end()
-            return
-
+        db.presetsCollection.remove({'_id' : presetName}, (err) ->
+          return throwError(err, response) if err
+          cache.del('presets_hash', (err) ->
+            return throwError(err, response) if err
             response.writeHead(200)
             response.end()
           )
+        )
       else
         response.writeHead 405, {'Allow': 'PUT, DELETE'}
         response.end('405 Method Not Allowed')
@@ -131,34 +134,90 @@ listener = (request, response) ->
         object._id = objectName
 
         db.objectsCollection.save(object, (err) ->
-          db.redisClient.del('objects', 'presets_hash', (err) ->
-            throw err if err
-          )
-          if err
-            response.writeHead(500)
-            response.end(errorToString(err))
-            return
+          return throwError(err, response) if err
+          cache.del('presets_hash', (err) ->
+            return throwError(err, response) if err
             response.writeHead(200)
             response.end()
           )
+        )
       else if request.method == 'DELETE'
-        db.objectsCollection.remove({'_id' : objectName}, (err, removedCount) ->
-          db.redisClient.del('objects', 'presets_hash', (err) ->
-            throw err if err
-          )
-          if err
-            response.writeHead(500)
-            response.end(errorToString(err))
-            return
-
-          if removedCount == 0
-            response.writeHead(404)
-            response.end()
-            return
-
+        db.objectsCollection.remove({'_id' : objectName}, (err) ->
+          return throwError(err, response) if err
+          cache.del('presets_hash', (err) ->
+            return throwError(err, response) if err
             response.writeHead(200)
             response.end()
           )
+        )
+      else
+        response.writeHead 405, {'Allow': 'PUT, DELETE'}
+        response.end('405 Method Not Allowed')
+    else if PROVISIONS_REGEX.test(urlParts.pathname)
+      provisionName = querystring.unescape(PROVISIONS_REGEX.exec(urlParts.pathname)[1])
+      if request.method == 'PUT'
+        object = {
+          _id: provisionName
+          script: body.toString()
+        }
+        try
+          new vm.Script("\"use strict\";(function(){\n#{object.script}\n})();")
+        catch err
+          response.writeHead(400)
+          response.end("#{err.name}: #{err.message}")
+          return
+
+        db.provisionsCollection.save(object, (err) ->
+          return throwError(err, response) if err
+          cache.del('presets_hash', (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
+      else if request.method == 'DELETE'
+        db.provisionsCollection.remove({'_id' : provisionName}, (err) ->
+          return throwError(err, response) if err
+          cache.del('presets_hash', (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
+      else
+        response.writeHead 405, {'Allow': 'PUT, DELETE'}
+        response.end('405 Method Not Allowed')
+    else if VIRTUAL_PARAMETERS_REGEX.test(urlParts.pathname)
+      virtualParameterName = querystring.unescape(VIRTUAL_PARAMETERS_REGEX.exec(urlParts.pathname)[1])
+      if request.method == 'PUT'
+        object = {
+          _id: virtualParameterName
+          script: body.toString()
+        }
+        try
+          new vm.Script("\"use strict\";(function(){\n#{object.script}\n})();")
+        catch err
+          response.writeHead(400)
+          response.end("#{err.name}: #{err.message}")
+          return
+
+        db.virtualParametersCollection.save(object, (err) ->
+          return throwError(err, response) if err
+          cache.del('presets_hash', (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
+      else if request.method == 'DELETE'
+        db.virtualParametersCollection.remove({'_id' : virtualParameterName}, (err) ->
+          return throwError(err, response) if err
+          cache.del('presets_hash', (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
       else
         response.writeHead 405, {'Allow': 'PUT, DELETE'}
         response.end('405 Method Not Allowed')
@@ -167,59 +226,56 @@ listener = (request, response) ->
       deviceId = querystring.unescape(r[1])
       tag = querystring.unescape(r[2])
       if request.method == 'POST'
-        db.devicesCollection.update({'_id' : deviceId}, {'$addToSet' : {'_tags' : tag}}, {safe: true}, (err, updatedCount) ->
-          db.redisClient.del("#{deviceId}_presets_hash", (err) ->
-            throw err if err
-          )
-          if err
-            response.writeHead(500)
-            response.end(errorToString(err))
-            return
-
-          if updatedCount == 0
-            response.writeHead(404)
-            response.end()
-            return
-
+        db.devicesCollection.update({'_id' : deviceId}, {'$addToSet' : {'_tags' : tag}}, {safe: true}, (err) ->
+          return throwError(err, response) if err
           response.writeHead(200)
           response.end()
         )
       else if request.method == 'DELETE'
-        db.devicesCollection.update({'_id' : deviceId}, {'$pull' : {'_tags' : tag}}, {safe: true}, (err, updatedCount) ->
-          db.redisClient.del("#{deviceId}_presets_hash", (err) ->
-            throw err if err
-          )
-          if err
-            response.writeHead(500)
-            response.end(errorToString(err))
-            return
-
-          if updatedCount == 0
-            response.writeHead(404)
-            response.end()
-            return
-
+        db.devicesCollection.update({'_id' : deviceId}, {'$pull' : {'_tags' : tag}}, {safe: true}, (err) ->
+          return throwError(err, response) if err
           response.writeHead(200)
           response.end()
         )
       else
         response.writeHead 405, {'Allow': 'POST, DELETE'}
         response.end('405 Method Not Allowed')
+    else if FAULTS_REGEX.test(urlParts.pathname)
+      if request.method == 'DELETE'
+        faultId = querystring.unescape(FAULTS_REGEX.exec(urlParts.pathname)[1])
+        deviceId = faultId.split(':', 1)[0]
+        channel = faultId.slice(deviceId.length + 1)
+        db.faultsCollection.remove({_id : faultId}, (err) ->
+          return throwError(err, response) if err
+          if channel.startsWith('task_')
+            return db.tasksCollection.remove({_id : new mongodb.ObjectID(channel.slice(5))}, (err) ->
+              return throwError(err, response) if err
+              cache.del("#{deviceId}_tasks_faults_operations", (err) ->
+                return throwError(err, response) if err
+                response.writeHead(200)
+                response.end()
+              )
+            )
+
+          cache.del("#{deviceId}_tasks_faults_operations", (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
+      else
+        response.writeHead 405, {'Allow': 'DELETE'}
+        response.end('405 Method Not Allowed')
     else if DEVICE_TASKS_REGEX.test(urlParts.pathname)
       if request.method == 'POST'
         deviceId = querystring.unescape(DEVICE_TASKS_REGEX.exec(urlParts.pathname)[1])
         if body.length
           task = JSON.parse(body)
           task.device = deviceId
-          db.getAliases((aliases) ->
-            apiFunctions.insertTasks(task, aliases, (err) ->
-              db.redisClient.del("#{deviceId}_presets_hash", (err) ->
-                throw err if err
-              )
-              if err
-                response.writeHead(500)
-                response.end(errorToString(err))
-                return
+          apiFunctions.insertTasks(task, (err) ->
+            return throwError(err, response) if err
+            cache.del("#{deviceId}_tasks_faults_operations", (err) ->
+              return throwError(err, response) if err
 
               if urlParts.query.connection_request?
                 apiFunctions.connectionRequest(deviceId, (err) ->
@@ -227,21 +283,15 @@ listener = (request, response) ->
                     response.writeHead(202, err.message, {'Content-Type' : 'application/json'})
                     response.end(JSON.stringify(task))
                   else
-                    apiFunctions.watchTask(task._id, config.get('DEVICE_ONLINE_THRESHOLD', deviceId), (err, status) ->
-                      if err
-                        response.writeHead(500)
-                        response.end(errorToString(err))
-                        return
+                    apiFunctions.watchTask(deviceId, task._id, config.get('DEVICE_ONLINE_THRESHOLD', deviceId), (err, status) ->
+                      return throwError(err, response) if err
 
                       if status is 'timeout'
                         response.writeHead(202, 'Task queued but not processed', {'Content-Type' : 'application/json'})
                         response.end(JSON.stringify(task))
                       else if status is 'fault'
                         db.tasksCollection.findOne({_id : task._id}, (err, task) ->
-                          if err
-                            response.writeHead(500)
-                            response.end(errorToString(err))
-                            return
+                          return throwError(err, response) if err
 
                           response.writeHead(202, 'Task faulted', {'Content-Type' : 'application/json'})
                           response.end(JSON.stringify(task))
@@ -261,7 +311,7 @@ listener = (request, response) ->
           apiFunctions.connectionRequest(deviceId, (err) ->
             if err
               response.writeHead 504
-              response.end(errorToString(err))
+              response.end("#{err.name}: #{err.message}")
               return
             response.writeHead 200
             response.end()
@@ -272,54 +322,49 @@ listener = (request, response) ->
       else
         response.writeHead 405, {'Allow': 'POST'}
         response.end('405 Method Not Allowed')
-    else if DEVICE_PRESET_REGEX.test(urlParts.pathname)
-      deviceId = querystring.unescape(DEVICE_PRESET_REGEX.exec(urlParts.pathname)[1])
-      if request.method is 'GET'
-        db.getPresetsObjectsAliases((allPresets, allObjects, allAliases) ->
-          presets.getDevicePreset(deviceId, allPresets, allObjects, allAliases, (devicePreset) ->
-            response.writeHead(200, {'Content-Type' : 'application/json'})
-            response.end(JSON.stringify(devicePreset))
-          )
-        )
-      else
-        response.writeHead 405, {'Allow': 'GET'}
-        response.end('405 Method Not Allowed')
     else if TASKS_REGEX.test(urlParts.pathname)
       r = TASKS_REGEX.exec(urlParts.pathname)
-      taskId = mongodb.ObjectID(querystring.unescape(r[1]))
+      taskId = new mongodb.ObjectID(querystring.unescape(r[1]))
       action = r[2]
       if not action? or action is '/'
         if request.method == 'DELETE'
-          db.tasksCollection.remove({'_id' : taskId}, (err, removedCount) ->
-            if err
-              response.writeHead(500)
-              response.end(errorToString(err))
-              return
-
-            if removedCount == 0
+          db.tasksCollection.findOne({'_id' : taskId}, {'device' : 1}, (err, task) ->
+            return throwError(err, response) if err
+            if not task?
               response.writeHead(404)
-              response.end()
+              response.end("Task not found")
               return
 
+            deviceId = task.device
+            db.tasksCollection.remove({'_id' : taskId}, (err) ->
+              return throwError(err, response) if err
+              db.faultsCollection.remove({_id : "#{deviceId}:task_#{String(taskId)}"}, (err) ->
+                return throwError(err, response) if err
+                cache.del("#{deviceId}_tasks_faults_operations", (err) ->
+                  return throwError(err, response) if err
                   response.writeHead(200)
                   response.end()
                 )
+              )
+            )
+          )
         else
           response.writeHead 405, {'Allow': 'PUT DELETE'}
           response.end('405 Method Not Allowed')
       else if action is '/retry'
         if request.method == 'POST'
-          db.tasksCollection.update({_id : taskId}, {$unset : {fault : 1}, $set : {timestamp : new Date()}}, (err, updatedCount) ->
-            # TODO need to invalidate presets hash for the device
-            throw err if err
-            if updatedCount == 0
-              response.writeHead(404)
-              response.end()
-              return
-
+          db.tasksCollection.findOne({'_id' : taskId}, {'device' : 1}, (err, task) ->
+            return throwError(err, response) if err
+            deviceId = task.device
+            db.faultsCollection.remove({_id : "#{deviceId}:task_#{String(taskId)}"}, (err) ->
+              return throwError(err, response) if err
+              cache.del("#{deviceId}_tasks_faults_operations", (err) ->
+                return throwError(err, response) if err
                 response.writeHead(200)
                 response.end()
               )
+            )
+          )
         else
           response.writeHead 405, {'Allow': 'POST'}
           response.end('405 Method Not Allowed')
@@ -336,12 +381,12 @@ listener = (request, response) ->
           version : request.headers.version,
         }
 
-        gs = new mongodb.GridStore(db.mongoDb, filename, 'w', {metadata : metadata})
+        gs = new mongodb.GridStore(db.mongoDb, filename, filename, 'w', {metadata : metadata})
         gs.open((err, gs) ->
           gs.write(body, (err, res) ->
-            throw err if err
+            return throwError(err, response) if err
             gs.close((err) ->
-              throw err if err
+              return throwError(err, response) if err
               response.writeHead(201)
               response.end()
             )
@@ -360,7 +405,7 @@ listener = (request, response) ->
       require('child_process').exec("ping -w 1 -i 0.2 -c 3 #{host}", (err, stdout, stderr) ->
         if err
           response.writeHead(404, {'Cache-Control' : 'no-cache'})
-          response.end(errorToString(err))
+          response.end("#{err.name}: #{err.message}")
           return
         response.writeHead(200, {'Content-Type' : 'text/plain', 'Cache-Control' : 'no-cache'})
         response.end(stdout)
@@ -373,15 +418,21 @@ listener = (request, response) ->
 
       deviceId = querystring.unescape(DELETE_DEVICE_REGEX.exec(urlParts.pathname)[1])
       apiFunctions.deleteDevice(deviceId, (err) ->
-        if err
-          response.writeHead(500)
-          response.end(errorToString(err))
-          return
+        return throwError(err, response) if err
         response.writeHead(200)
         response.end()
       )
     else if QUERY_REGEX.test(urlParts.pathname)
       collectionName = QUERY_REGEX.exec(urlParts.pathname)[1]
+
+      # Convert to camel case
+      i = collectionName.indexOf('_')
+      while i >= 0
+        ++ i
+        up = if i < collectionName.length then collectionName[i].toUpperCase() else ''
+        collectionName = collectionName.slice(0, i - 1) + up + collectionName.slice(i + 1)
+        i = collectionName.indexOf('_', i)
+
       if request.method not in ['GET', 'HEAD']
         response.writeHead 405, {'Allow' : 'GET, HEAD'}
         response.end('405 Method Not Allowed')
@@ -392,41 +443,42 @@ listener = (request, response) ->
         response.end('404 Not Found')
         return
 
-      func = (aliases) ->
       if urlParts.query.query?
         try
           q = JSON.parse(urlParts.query.query)
         catch err
           response.writeHead(400)
-            response.end(errorToString(err))
+          response.end("#{err.name}: #{err.message}")
           return
       else
         q = {}
-        q = query.expand(q, aliases) if collectionName is 'devices'
-        q = query.substituteObjectId(q) if collectionName is 'tasks'
+
+      switch collectionName
+        when 'devices'
+          q = query.expand(q)
+        when 'tasks'
+          q = query.sanitizeQueryTypes(q, {
+            _id: ((v) -> return new mongodb.ObjectID(v))
+            timestamp: ((v) -> return new Date(v))
+            retries: Number
+          })
+        when 'faults'
+          q = query.sanitizeQueryTypes(q, {
+            timestamp: ((v) -> return new Date(v))
+            retries: Number
+          })
 
       if urlParts.query.projection?
         projection = {}
         for p in urlParts.query.projection.split(',')
           p = p.trim()
           projection[p] = 1
-            if collectionName is 'devices'
-              for k,v of aliases
-                if k == p or common.startsWith(k, "#{p}.")
-                  projection[a] = 1 for a in v
 
       cur = collection.find(q, projection, {batchSize : 50})
       if urlParts.query.sort?
         s = JSON.parse(urlParts.query.sort)
         sort = {}
         for k,v of s
-            if aliases[k]?
-              for a in aliases[k]
-                if a[a.lastIndexOf('.')+1] != '_' and collectionName is 'devices'
-                  sort["#{a}._value"] = v
-                else
-                  sort[a] = v
-            else
           if k[k.lastIndexOf('.') + 1] != '_' and collectionName is 'devices'
             sort["#{k}._value"] = v
           else
@@ -434,8 +486,8 @@ listener = (request, response) ->
         cur.sort(sort)
 
       cur.skip(parseInt(urlParts.query.skip)) if urlParts.query.skip?
-        cur.limit(parseInt(urlParts.query.limit)) if urlParts.query.limit?
-        cur.count((err, total) ->
+      cur.limit(limit = parseInt(urlParts.query.limit)) if urlParts.query.limit?
+      cur.count(false, (err, total) ->
         response.writeHead(200, {'Content-Type' : 'application/json', 'total' : total})
         if request.method is 'HEAD'
           response.end()
@@ -443,21 +495,19 @@ listener = (request, response) ->
         response.write("[\n")
         i = 0
         cur.each((err, item) ->
-            if item is null
-              response.end("\n]")
-            else
+          if err
+            throwError(err)
+            return false
+
+          if item?
             response.write(",\n") if i++
-              apiFunctions.addAliases(item, aliases) if collectionName is 'devices'
             response.write(JSON.stringify(item))
+
+          if not item? or (limit? and i >= limit)
+            response.end("\n]")
         )
       )
 
-      if collectionName is 'devices'
-        db.getAliases((aliases) ->
-          func(aliases)
-        )
-      else
-        func({})
     else
       response.writeHead 404
       response.end('404 Not Found')

lib/normalize.coffee

@@ -1,132 +0,0 @@
-###
-# Copyright 2013 Fanoos Telecom
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-###
-
-# Credit to Netanel Rubin from Check Point Software Technologies for discovering
-# vulnerability caused by use of eval to generate regex from user input.
-
-common = require './common'
-parameters = require './parameters'
-
-
-stringToRegexp = (input, flags) ->
-  if (input.indexOf('*') == -1)
-    return false
-
-  output = input.replace(/[\[\]\\\^\$\.\|\?\+\(\)]/, "\\$&")
-  if output[0] == '*'
-    output = output.replace(/^\*+/g, '')
-  else
-    output = '^' + output
-
-  if output[output.length - 1] == '*'
-    output = output.replace(/\*+$/g, '')
-  else
-    output = output + '$'
-
-  output = output.replace(/[\*]/, '.*')
-
-  return new RegExp(output, flags)
-
-
-normalizers = {}
-
-normalizers.default = (input, normType) ->
-  if normType is 'query' and common.typeOf(input) is common.STRING_TYPE
-    vals = []
-    str = normalizers.string(input, normType)
-    if common.typeOf(str) == common.ARRAY_TYPE
-      vals = vals.concat(str)
-    else
-      vals.push(str)
-
-    f = parseFloat(input)
-    if not isNaN(f)
-      vals.push(f)
-
-    d = new Date(input)
-    if input.length >= 8 and d.getFullYear() > 1983
-      vals.push(d)
-
-    r = stringToRegexp(input)
-    if r isnt false
-      vals.push({'$regex' : r})
-
-    return vals
-  return input
-
-
-FLOAT_REGEX = /^([\-\+]?[0-9\.]+)/
-normalizers.float = (input, normType) ->
-  res = FLOAT_REGEX.exec(input)
-  if res != null
-    parseFloat(res[1])
-  else
-    null
-
-
-normalizers.date = (input, normType) ->
-  return new Date(input)
-
-
-normalizers.string = (input, normType) ->
-  if normType is 'query'
-    if (m = /^\/(.*?)\/(g?i?m?y?)$/.exec(input))
-      return [{'$regex' : new RegExp(m[1], m[2])}, input]
-  return input
-
-
-colonizeMac = (input) ->
-  if (i = input.indexOf(':')) != -1
-    i = if (i % 2) == 0 then 0 else 1
-    while i < input.length
-      if input[i-1] != ':' and input[i-2] != ':'
-        if input[i] != ':' and i != 0
-          input = input.substring(0, i) + ':' + input.substring(i)
-          ++i
-        ++i
-      ++ i
-    return input
-  else
-    return input.replace(/[0-9A-F]{1}(?!$)/g, '$&:?')
-
-
-normalizers.mac = (input, normType) ->
-  if normType isnt 'query'
-    return input
-
-  input = input.trim().toUpperCase().replace(/[^0-9A-F\*\-\:]/g, '').replace(/[\-\:]/g, ':')
-
-  if input.indexOf('*') != -1
-    input = (colonizeMac(a) for a in input.split('*')).join('*')
-    return stringToRegexp(input, 'i')
-
-  input = colonizeMac(input)
-
-  if input.length == 17
-    return [input, input.toLowerCase()]
-
-  return {'$regex' : new RegExp(input, 'i')}
-
-
-exports.normalize = (path, value, normType) ->
-  type = parameters.getType(path) ? 'default'
-  normalizers[type](value, normType)

lib/parameters.coffee

@@ -1,212 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-###
-
-config = require './config'
-db = require './db'
-
-
-PATH_REGEX = /\w+|(\/.+?\/\w*)/g
-
-# Javascript represents numbers as 64-bit floating point numbers
-# Integers up to 2^53 can be encoded precisely
-MAX_PARAMETERS = 52
-
-
-# get the index of the least significant bit
-getLsb = (num) ->
-  i = -1
-  while num > 0
-    ++ i
-    break if num & 1
-    num >>= 1
-  return i
-
-
-createParameterAttributeFinder = (attributeName) ->
-  # compile attributes into data types optimized for quick lookup
-  attributes = []
-  stringSegments = {}
-  regexSegments = {}
-  paramRegex = {}
-
-  for k, v of config.PARAMETERS
-    continue if not v[attributeName]?
-    attributeIndex = attributes.push(v[attributeName]) - 1
-    throw new Error("There cannot be more than #{MAX_PARAMETERS} configured parameters") if attributeIndex >= MAX_PARAMETERS
-    segmentIndex = 0
-    parts = k.match(PATH_REGEX)
-    for i in [0 ... parts.length]
-      part = parts[i]
-      positionHash = i + parts.length * MAX_PARAMETERS
-
-      if part[0] == '/' # this is a regex part
-        j = part.lastIndexOf('/')
-        regExp = new RegExp(part[1...j], part[j+1..])
-        regexSegments[positionHash] ?= 0
-        regexSegments[positionHash] |= Math.pow(2, attributeIndex)
-        paramRegex[attributeIndex] ?= {}
-        paramRegex[attributeIndex][i] = regExp
-      else
-        stringSegments[positionHash] ?= {}
-        stringSegments[positionHash][part] ?= 0
-        stringSegments[positionHash][part] |= Math.pow(2, attributeIndex)
-
-  # Finder function
-  return (param) ->
-    parts = param.split('.')
-    positionHash = parts.length * MAX_PARAMETERS
-
-    stringIndices = 0 | stringSegments[positionHash]?[parts[0]]
-    allIndices = stringIndices | regexSegments[positionHash]
-
-    for i in [1...parts.length]
-      positionHash = i + parts.length * MAX_PARAMETERS
-      strIdx = stringSegments[positionHash]?[parts[i]]
-      rgxIdx = regexSegments[positionHash]
-      stringIndices &= strIdx
-      allIndices &= (strIdx | rgxIdx)
-
-    if stringIndices > 0
-      return attributes[getLsb(stringIndices)]
-
-    attributeIndex = 0
-    while allIndices > 0
-      if allIndices & 1
-        match = true
-        for i, r of paramRegex[attributeIndex]
-          if not r.test(parts[i])
-            match = false
-            break
-
-        if match
-          return attributes[attributeIndex]
-
-      ++ attributeIndex
-      allIndices >>= 1
-
-    return null
-
-
-splitIds = (batches, callback) ->
-  db.devicesCollection.count((err, count) ->
-    throw err if err
-    return callback(null) if count == 0
-    ids = [{}]
-    return callback(ids) if count <= 1000 or batches == 1
-    batchSize = Math.floor(count / batches)
-    for i in [1 ... batches]
-      ids[i] = {}
-      do (i) ->
-        cursor = db.devicesCollection.find({}, {_id:1}).sort({'_id':1}).skip(i * batchSize).limit(1)
-        cursor.nextObject((err, obj) ->
-          ids[i]['$gte'] = obj._id
-          ids[i-1]['$lt'] = obj._id
-          if i == batches - 1
-            callback(ids)
-        )
-  )
-
-
-compileAliases = (callback) ->
-  map = () ->
-    recurse = (partIndex, ref) ->
-      part = parts[partIndex]
-      res = []
-      if part.test?
-        for p of ref
-          if part.test(p)
-            if partIndex == parts.length - 1
-              res.push(p)
-            else
-              for r in recurse(partIndex + 1, ref[p])
-                res.push("#{p}.#{r}")
-      else
-        if ref[part]?
-          if partIndex == parts.length - 1
-            return [part]
-          else
-            for r in recurse(partIndex + 1, ref[part])
-              res.push("#{part}.#{r}")
-
-      return res
-
-    for key, parts of keys
-      for alias in recurse(0, this)
-        emit(alias, key)
-    return
-
-  reduce = (k, v) ->
-    return v[0]
-
-  aliases = {}
-  keys = {}
-  for k, v of config.PARAMETERS
-    alias = v.alias
-    continue if not alias?
-    isStatic = true
-    ar = []
-    for p in k.match(/\w+|(\/.+?\/\w*)/g)
-      if p[0] == '/' # this is a regex
-        isStatic = false
-        i = p.lastIndexOf('/')
-        r = new RegExp(p[1...i], p[i+1..])
-        ar.push(r)
-      else
-        ar.push(p)
-
-    if isStatic
-      aliases[alias] ?= []
-      aliases[alias].push(k)
-    else
-      keys[k] = ar
-
-  counter = 0
-  splitIds(4, (batches) ->
-    if not batches?
-      return callback(null, aliases)
-
-    for batch in batches
-      options = {
-        out : {inline : 1}
-        jsMode : true
-        sort : {$natural : 1}
-        scope : {keys : keys, flags : {}}
-        query : {_id : batch} if batches.length > 1
-      }
-
-      db.devicesCollection.mapReduce(map, reduce, options, (err, out) ->
-        if err
-          callback?(err)
-          return callback = null
-
-        for o in out
-          alias = config.PARAMETERS[o.value].alias
-          aliases[alias] ?= []
-          id = String(o._id)
-          if id not in aliases[alias]
-            aliases[alias].push(id)
-
-        if ++counter == batches.length
-          callback?(null, aliases)
-      )
-    return
-  )
-
-
-exports.getType = createParameterAttributeFinder('type')
-exports.getAlias = createParameterAttributeFinder('alias')
-exports.compileAliases = compileAliases

lib/path-set.js

@@ -0,0 +1,179 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+
+class PathSet {
+
+  constructor() {
+    this.lengthIndex = [];
+    this.fragmentIndex = [];
+  }
+
+
+  get depth() {
+    return this.lengthIndex.length;
+  }
+
+
+  add(path) {
+    let p = this.get(path);
+
+    if (p)
+      return p;
+
+    if (!this.lengthIndex[path.length])
+      this.lengthIndex[path.length] = new Set();
+
+    this.lengthIndex[path.length].add(path);
+
+    let wildcard = 0;
+    for (let i = 0; i < path.length; ++ i) {
+      let fragment = path[i];
+      if (fragment === "*")
+        wildcard = wildcard | (1 << i);
+
+      if (!this.fragmentIndex[i])
+        this.fragmentIndex[i] = {};
+
+      if (!this.fragmentIndex[i][fragment])
+        this.fragmentIndex[i][fragment] = new Set();
+
+      this.fragmentIndex[i][fragment].add(path);
+    }
+
+    if (path.wildcard == null) {
+      path.alias = 0;
+      path.wildcard = wildcard;
+      Object.freeze(path);
+    }
+    return path;
+  }
+
+
+  get(pattern) {
+    if (!this.lengthIndex[pattern.length])
+      return null;
+
+    if (this.lengthIndex[pattern.length].has(pattern))
+      return pattern;
+
+    let sets = [];
+
+    sets.push(this.lengthIndex[pattern.length]);
+
+    for (let i = 0; i < pattern.length; ++ i) {
+      if (!this.fragmentIndex[i])
+        return null;
+
+      let fragment = pattern[i];
+
+      if (!this.fragmentIndex[i][fragment])
+        return null;
+
+      sets.push(this.fragmentIndex[i][fragment]);
+    }
+
+    sets.sort(function(a, b) {
+      return a.size - b.size;
+    });
+
+    let smallestSet = sets.shift();
+    for (let path of smallestSet) {
+      for (let s of sets) {
+        if (!s.has(path)) {
+          path = null;
+          break
+        }
+      }
+      if (path)
+        return path;
+    }
+
+    return null;
+  }
+
+
+  find(pattern, superset = false, subset = false, depth = pattern.length) {
+    let res = [];
+    let groups = [[]];
+
+    for (let i = pattern.length; i <= depth; ++ i)
+      if (this.lengthIndex[i])
+        groups[0].push(this.lengthIndex[i]);
+
+    if (!groups[0].length)
+      return res;
+
+    for (let i = 0; i < pattern.length; ++ i) {
+      if (!this.fragmentIndex[i])
+        return res;
+
+      let fragment = pattern[i];
+
+      if (fragment === "*" && subset)
+        continue;
+
+      let g = [];
+
+      if (this.fragmentIndex[i][fragment])
+        g.push(this.fragmentIndex[i][fragment]);
+
+      if (fragment !== "*" && superset && this.fragmentIndex[i]["*"])
+        g.push(this.fragmentIndex[i]["*"]);
+
+      if (!g.length)
+        return res;
+
+      groups.push(g);
+    }
+
+    groups.sort(function(a, b) {
+      return a.reduce(function(prev, cur) {return prev + cur.size;}, 0) -
+        b.reduce(function(prev, cur) {return prev + cur.size;}, 0);
+    });
+
+    let smallestGroup = groups.shift();
+    for (let set of smallestGroup) {
+      for (let path of set) {
+        for (let group of groups) {
+          let found = false;
+          for (let s of group) {
+            if (s.has(path)) {
+              found = true;
+              break;
+            }
+          }
+          if (!found) {
+            path = null;
+            break;
+          }
+        }
+        if (path)
+          res.push(path);
+      }
+    }
+
+    return res;
+  }
+
+}
+
+
+module.exports = PathSet;

lib/presets.coffee

@@ -1,336 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-#
-# This file incorporates work covered by the following copyright and
-# permission notice:
-#
-# Copyright 2013 Fanoos Telecom
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-###
-
-config = require './config'
-common = require './common'
-db = require './db'
-mongoQuery = require './mongo-query'
-query = require './query'
-log = require('util').log
-customCommands = require './custom-commands'
-
-
-exports.calculatePresetsHash = (presets, objects) ->
-  crypto = require('crypto')
-  hash = crypto.createHash('md5').update(JSON.stringify(presets) + JSON.stringify(objects)).digest('hex')
-  return hash
-
-
-matchObject = (object, param) ->
-  flatObject = common.flattenObject(object)
-  if object._keys?.length
-    keys = object._keys
-  else
-    # Consider all parameters as keys if none are defined
-    keys = (k for k in Object.keys(flatObject) when k[0] != '_')
-
-  for k in keys
-    p = common.getParamValueFromPath(param, k)
-    return false if not p?
-    v = common.matchType(p._value, flatObject[k])
-
-    if p._value != common.matchType(p._value, flatObject[k])
-      return false
-
-  return true
-
-
-exports.getDevicePreset = (deviceId, presets, objects, aliases, callback) ->
-  deviceCustomCommands = customCommands.getDeviceCustomCommands(deviceId)
-  # only fetch relevant params
-  projection = {_id : 1}
-  for p in presets
-    if common.typeOf(p.precondition) is common.STRING_TYPE
-      p.precondition = query.expand(JSON.parse(p.precondition), aliases)
-    else
-      # Accept an object for backward compatiblity
-      p.precondition = query.expand(p.precondition ? {}, aliases)
-    mongoQuery.projection(p.precondition, projection)
-
-    for c in p.configurations
-      switch c.type
-        when 'value', 'age'
-          projection[c.name] = 1
-        when 'custom_command_value', 'custom_command_age'
-          commandfile = c.command.split(' ', 1)
-          if commandfile of deviceCustomCommands
-            projection["_customCommands.#{c.command.split(' ', 1)}"] = 1
-        when 'software_version'
-          projection['Device.DeviceInfo.SoftwareVersion'] = 1
-          projection['InternetGatewayDevice.DeviceInfo.SoftwareVersion'] = 1
-        when 'add_tag', 'delete_tag'
-          projection['_tags'] = 1
-        when 'add_object', 'delete_object'
-          projection[c.name] = 1
-        else
-          throw new Error('Unknown configuration type')
-
-  mongoQuery.optimizeProjection(projection)
-
-  db.devicesCollection.findOne({'_id' : deviceId}, projection, (err, device) ->
-    throw err if err
-    devicePresets = []
-    for p in presets
-      if mongoQuery.test(device, p.precondition)
-        devicePresets.push(p)
-
-    configurations = accumulateConfigurations(devicePresets, objects)
-
-    getParamDetails = (device, path) ->
-      p = common.getParamValueFromPath(device, path)
-      return undefined if not p?
-      details = {}
-      for k,v of p
-        details[k] = v if k[0] == '_'
-      return details
-
-    devicePreset = {}
-    for c in configurations
-      switch c.type
-        when 'value'
-          devicePreset.parameters ?= {}
-          devicePreset.parameters[c.name] ?= {}
-          devicePreset.parameters[c.name].preset ?= {}
-          devicePreset.parameters[c.name].preset.value = c.value
-          devicePreset.parameters[c.name].current = getParamDetails(device, c.name)
-
-        when 'age'
-          devicePreset.parameters ?= {}
-          devicePreset.parameters[c.name] ?= {}
-          devicePreset.parameters[c.name].preset ?= {}
-          devicePreset.parameters[c.name].preset.expiry = parseInt(c.age)
-          devicePreset.parameters[c.name].current = getParamDetails(device, c.name)
-
-        when 'add_tag'
-          devicePreset.tags ?= {}
-          devicePreset.tags[c.tag] = 2 | c.tag in (device._tags ? [])
-
-        when 'delete_tag'
-          devicePreset.tags ?= {}
-          devicePreset.tags[c.tag] = 0 | c.tag in (device._tags ? [])
-
-        when 'add_object'
-          devicePreset.objects ?= {}
-          devicePreset.objects[c.name] ?= {}
-          devicePreset.objects[c.name][c.object] ?= {}
-          devicePreset.objects[c.name][c.object].preset = objects[c.object]
-          continue if not param = common.getParamValueFromPath(device, c.name)
-          devicePreset.objects[c.name][c.object].current ?= {}
-          for k,p of param
-            continue if k[0] == '_'
-            if matchObject(objects[c.object], p)
-              devicePreset.objects[c.name][c.object].current[k] = p
-
-        when 'delete_object'
-          devicePreset.objects ?= {}
-          devicePreset.objects[c.name] ?= {}
-          devicePreset.objects[c.name][c.object] ?= {}
-          continue if not param = common.getParamValueFromPath(device, c.name)
-          devicePreset.objects[c.name][c.object].current ?= {}
-          for k,p of param
-            continue if k[0] == '_'
-            if matchObject(objects[c.object], p)
-              devicePreset.objects[c.name][c.object].current[k] = p
-
-        when 'custom_command_value'
-          [filename, commandName] = c.command.split(' ', 2)
-          devicePreset.customCommands ?= {}
-          devicePreset.customCommands[filename] ?= {}
-          devicePreset.customCommands[filename].preset ?= {}
-          devicePreset.customCommands[filename].preset.value = c.value
-          devicePreset.customCommands[filename].preset.valueCommand = c.command
-          continue if not deviceCustomCommands[filename]? or commandName not in deviceCustomCommands[filename]
-          if cmd = common.getParamValueFromPath(device, "_customCommands.#{filename}")
-            devicePreset.customCommands[filename].current = cmd
-          else
-            devicePreset.customCommands[filename].current ?= {}
-
-        when 'custom_command_age'
-          [filename, commandName] = c.command.split(' ', 2)
-          devicePreset.customCommands ?= {}
-          devicePreset.customCommands[filename] ?= {}
-          devicePreset.customCommands[filename].preset ?= {}
-          devicePreset.customCommands[filename].preset.expiry = parseInt(c.age)
-          devicePreset.customCommands[filename].preset.expiryCommand = c.command
-          continue if not deviceCustomCommands[filename]? or commandName not in deviceCustomCommands[filename]
-          if cmd = common.getParamValueFromPath(device, "_customCommands.#{filename}")
-            devicePreset.customCommands[filename].current = cmd
-          else
-            devicePreset.customCommands[filename].current ?= {}
-
-        when 'software_version'
-          devicePreset.softwareVersion ?= {}
-          devicePreset.softwareVersion.preset = c.software_version
-          if device?.Device? # TR-181 data model
-            devicePreset.softwareVersion.current = common.getParamValueFromPath(device, 'Device.DeviceInfo.SoftwareVersion')
-          else # TR-098 data model
-            devicePreset.softwareVersion.current = common.getParamValueFromPath(device, 'InternetGatewayDevice.DeviceInfo.SoftwareVersion')
-
-        else
-          throw new Error('Unknown configuration type')
-
-    callback(devicePreset)
-  )
-
-
-exports.processDevicePreset = (deviceId, devicePreset, callback) ->
-  PRESETS_TIME_PADDING = config.get('PRESETS_TIME_PADDING')
-  now = new Date()
-  expiry = config.get('PRESETS_CACHE_DURATION', deviceId)
-  getParameterValues = []
-  setParameterValues = []
-  addTags = []
-  deleteTags = []
-  taskList = []
-
-  # Parameters
-  for parameterPath, parameterDetails of devicePreset.parameters ? {}
-    continue if not parameterDetails.current?
-    expiry = Math.min(expiry, parameterDetails.preset.expiry) if parameterDetails.preset.expiry?
-    presetValue = parameterDetails.preset.value
-    currentValue = parameterDetails.current._value
-    if presetValue? and currentValue != (presetValue = common.matchType(currentValue, presetValue))
-      setParameterValues.push([parameterPath, presetValue, parameterDetails.current._type])
-    else if parameterDetails.preset.expiry?
-      diff = parameterDetails.preset.expiry - (now - (parameterDetails.current._timestamp ? 0)) / 1000
-      if diff <= PRESETS_TIME_PADDING
-        if parameterDetails.current._value?
-          getParameterValues.push(parameterPath)
-        else
-          taskList.push({device : deviceId, name : 'refreshObject', objectName : parameterPath})
-      else
-        expiry = Math.min(expiry, diff)
-
-  # Objects
-  for parameterPath, object of devicePreset.objects ? {}
-    for objectName, objectDetails of object
-      continue if not objectDetails.current?
-      if objectDetails.preset?
-        flatObject = common.flattenObject(objectDetails.preset)
-        if Object.keys(objectDetails.current).length > 0
-          for i, obj of objectDetails.current
-            for paramName, paramValue of flatObject
-              continue if paramName[0] == '_'
-              currentValue = common.getParamValueFromPath(obj, paramName)
-              if currentValue?
-                presetValue = common.matchType(currentValue._value, flatObject[paramName])
-                if currentValue._value != presetValue
-                  setParameterValues.push(["#{parameterPath}.#{i}.#{paramName}", presetValue, currentValue._type])
-        else
-          vals = []
-          for k,v of flatObject
-            vals.push([k, v]) if k[0] != '_'
-          taskList.push({device : deviceId, name : 'addObject', objectName : parameterPath, parameterValues : vals})
-      else if Object.keys(objectDetails.current).length > 0
-        for i, obj of objectDetails.current
-          taskList.push({device : deviceId, name : 'deleteObject', objectName : "#{parameterPath}.#{i}"})
-
-  # Tags
-  for tag, v of devicePreset.tags ? {}
-    if v == 1
-      deleteTags.push(tag)
-    else if v == 2
-      addTags.push(tag)
-
-  # Custom commands
-  for commandName, commandDetails of devicePreset.customCommands ? {}
-    continue if not commandDetails.current?
-    expiry = Math.min(expiry, commandDetails.preset.expiry) if commandDetails.preset.expiry?
-    currentValue = commandDetails.current._value
-    presetValue = common.matchType(currentValue, commandDetails.preset.value)
-    if commandDetails.preset.value? and currentValue != presetValue
-      taskList.push({device : deviceId, name : 'customCommand', command : commandDetails.preset.valueCommand})
-    else if commandDetails.preset.expiry?
-      diff = commandDetails.preset.expiry - (now - (commandDetails.current._timestamp ? 0)) / 1000
-      if diff <= PRESETS_TIME_PADDING
-        taskList.push({device : deviceId, name : 'customCommand', command : commandDetails.preset.expiryCommand})
-      else
-        expiry = Math.min(expiry, diff)
-
-  if setParameterValues.length
-    taskList.push {device : deviceId, name : 'setParameterValues', parameterValues: setParameterValues, timestamp : now}
-
-  if getParameterValues.length
-    taskList.push {device : deviceId, name : 'getParameterValues', parameterNames: getParameterValues, timestamp : now}
-
-  callback(taskList, addTags, deleteTags, expiry)
-
-
-getObjectHash = (object) ->
-  if object._keys?.length
-    keys = object._keys
-  else
-    keys = (k for k in Object.keys(object) when k[0] != '_')
-
-  keys.sort()
-
-  hash = []
-  for k in keys
-    hash.push(k)
-    hash.push(common.getParamValueFromPath(object, k))
-
-  return JSON.stringify(hash)
-
-
-accumulateConfigurations = (presets, objects) ->
-  maxWeights = {}
-  configurations = {}
-  for p in presets
-    for c in p.configurations
-      configurationHash = switch c.type
-        when 'add_tag', 'delete_tag'
-          "tag_#{c.tag}"
-        when 'add_object', 'delete_object'
-          objectHash = getObjectHash(objects[c.object])
-          "object_#{c.name}_#{objectHash}"
-        when 'software_version'
-          'software_version'
-        when 'custom_command_age', 'custom_command_value'
-          [filename, commandName] = c.command.split(' ', 2)
-          "#{c.type}_#{filename}_#{commandName}"
-        else
-          "#{c.type}_#{c.name}"
-
-      if not maxWeights[configurationHash]? or p.weight > maxWeights[configurationHash]
-        configurations[configurationHash] = c
-        maxWeights[configurationHash] = p.weight
-
-  configurationsList = (configurations[c] for c of configurations)
-  return configurationsList

lib/query.coffee

@@ -1,4 +1,24 @@
 ###
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
+#
+# GenieACS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# GenieACS is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This file incorporates work covered by the following copyright and
+# permission notice:
+#
 # Copyright 2013 Fanoos Telecom
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
@@ -21,18 +41,59 @@
 ###
 
 common = require './common'
-normalize = require './normalize'
 mongodb = require 'mongodb'
 
 
-expandValue = (param, value) ->
+stringToRegexp = (input, flags) ->
+  if (input.indexOf('*') == -1)
+    return false
+
+  output = input.replace(/[\[\]\\\^\$\.\|\?\+\(\)]/, "\\$&")
+  if output[0] == '*'
+    output = output.replace(/^\*+/g, '')
+  else
+    output = '^' + output
+
+  if output[output.length - 1] == '*'
+    output = output.replace(/\*+$/g, '')
+  else
+    output = output + '$'
+
+  output = output.replace(/[\*]/, '.*')
+
+  return new RegExp(output, flags)
+
+
+normalize = (input) ->
+  if common.typeOf(input) is common.STRING_TYPE
+    vals = [input]
+    if (m = /^\/(.*?)\/(g?i?m?y?)$/.exec(input))
+      vals.push({'$regex' : new RegExp(m[1], m[2])})
+
+    f = parseFloat(input)
+    if not isNaN(f)
+      vals.push(f)
+
+    d = new Date(input)
+    if input.length >= 8 and d.getFullYear() > 1983
+      vals.push(d)
+
+    r = stringToRegexp(input)
+    if r isnt false
+      vals.push({'$regex' : r})
+
+    return vals
+  return input
+
+
+expandValue = (value) ->
   if common.typeOf(value) is common.ARRAY_TYPE
     a = []
     for j in value
-      a = a.concat(expandValue(param, j))
+      a = a.concat(expandValue(j))
     return [a]
   else if common.typeOf(value) isnt common.OBJECT_TYPE
-    n = normalize.normalize(param, value, 'query')
+    n = normalize(value)
     if common.typeOf(n) isnt common.ARRAY_TYPE
       return [n]
     else
@@ -44,7 +105,7 @@ expandValue = (param, value) ->
   values = []
   for k,v of value
     keys.push(k)
-    values.push(expandValue(param, v))
+    values.push(expandValue(v))
     indices.push(0)
 
   i = 0
@@ -62,72 +123,249 @@ expandValue = (param, value) ->
   return objs
 
 
-permute = (param, val, aliases) ->
-  keys = []
-  if aliases[param]?
-    for p in aliases[param]
-      keys.push(p)
-  else
-    keys.push(param)
-
+permute = (param, val) ->
   conditions = []
-  for k in keys
-    values = expandValue(k, val)
-    if k[k.lastIndexOf('.') + 1] != '_'
-      k += '._value'
+
+  values = expandValue(val)
+  if param[param.lastIndexOf('.') + 1] != '_'
+    param += '._value'
 
   for v in values
     obj = {}
-      obj[k] = v
+    obj[param] = v
     conditions.push(obj)
 
   return conditions
 
 
-expand = (query, aliases) ->
+expand = (query) ->
   new_query = {}
   for k,v of query
     if k[0] == '$' # operator
       expressions = []
       for e in v
-        expressions.push(expand(e, aliases))
+        expressions.push(expand(e))
       new_query[k] = expressions
     else
-      conditions = permute(k, v, aliases)
+      conditions = permute(k, v)
       if conditions.length > 1
-        if new_query['$and']?
+        new_query['$and'] ?= []
+        if v?['$ne']?
+          for c in conditions
+            new_query['$and'].push(c)
+        else
           new_query['$and'].push({'$or' : conditions})
       else
-          new_query['$and'] = [{'$or' : conditions}]
-      else
-        common.extend(new_query, conditions[0])
+        Object.assign(new_query, conditions[0])
 
   return new_query
 
-# Replace _id string values with ObjectID type
-substituteObjectId = (query) ->
+
+# Generate parameter projection from a query
+# If second arg is given, it's edited and returned
+queryProjection = (query, proj) ->
+  proj ?= {}
+  for k,v of query
+    if k.charAt(0) == '$' # this is a logical operator
+      for q in v
+        queryProjection(q, proj)
+    else
+      proj[k] = 1
+  return proj
+
+
+testExpressions = (params, expressions, lop) ->
+  for f in expressions
+    res = test(params, f)
+
+    switch lop
+      when '$and'
+        return false if not res
+      when '$or'
+        return true if res
+      when '$nor'
+        return false if res
+
+  return switch lop
+    when '$and' then true
+    when '$or' then false
+    when '$nor' then true
+    else throw new Error('Unknown logical operator')
+
+
+test = (params, query) ->
+  for k,v of query
+    if k.charAt(0) == '$' # this is a logical operator
+      res = testExpressions(params, v, k)
+    else
+      value = params[k]
+
+      if common.typeOf(v) isnt common.OBJECT_TYPE
+        # TODO comparing array to regex, array to array, and object to object
+        if common.typeOf(value) is common.ARRAY_TYPE
+          res = value.indexOf(v) != -1
+        else
+          if common.typeOf(v) is common.REGEXP_TYPE
+            res = v.test(value)
+          else
+            res = v == value
+      else
+        for k2,v2 of v
+          switch k2
+            when '$ne'
+              if common.typeOf(value) is common.ARRAY_TYPE
+                res = value.indexOf(v2) == -1
+              else
+                res = value != v2
+            when '$lt'
+              res = value < v2
+            when '$lte'
+              res = value <= v2
+            when '$gt'
+              res = value > v2
+            when '$gte'
+              res = value >= v2
+            when '$regex'
+              res = v2.test(value)
+            when '$in'
+              throw new Error('Operator not supported')
+            when '$nin'
+              throw new Error('Operator not supported')
+            when '$all'
+              throw new Error('Operator not supported')
+            when '$exists'
+              throw new Error('Operator not supported')
+            else
+              throw new Error('Operator not supported')
+
+    if not res
+      return false
+
+  return true
+
+
+matchType = (src, dst) ->
+  if typeof src is 'string'
+    return String(dst)
+  else if typeof src is 'number'
+    if +dst == parseFloat(dst)
+      return +dst
+    else if not isNaN(Date.parse(dst))
+      return Date.parse(dst)
+  else if typeof src is 'boolean'
+    v = String(dst).trim()
+    if v is 'true' or v is 'TRUE' or v is 'True' or v is '1'
+      return true
+    else if v is 'false' or v is 'FALSE' or v is 'False' or v is '0'
+      return false
+
+  return dst
+
+
+testFilter = (obj, filter) ->
+  for k, v of filter
+    [param, op] = k.split(/([^a-zA-Z0-9\-\_\.].*)/, 2)
+    val = matchType(obj[param], v)
+    switch op
+      when '=', undefined
+        return false if obj[param] != val
+      when '>'
+        return false if not (obj[param] > val)
+      when '>='
+        return false if not (obj[param] >= val)
+      when '<'
+        return false if not (obj[param] < val)
+      when '<='
+        return false if not (obj[param] <= val)
+      when '!'
+        return false if obj[param] == val
+      else
+        throw new Error("Unrecognized operator #{op}")
+
+  return true
+
+
+convertMongoQueryToFilters = (query, filters) ->
+  filters ?= {}
+  for k, v of query
+    if k[0] == '$'
+      if k == '$and'
+        for vv in v
+          convertMongoQueryToFilters(vv, filters)
+      else
+        throw new Error("Operator #{k} not supported")
+    else if k == '_tags'
+      if common.typeOf(v) is common.OBJECT_TYPE
+        for kk, vv of v
+          vv = vv.replace(/[^a-zA-Z0-9\-]+/g, '_')
+          switch kk
+            when '$ne'
+              filters["Tags.#{vv}!"] = true
+            else
+              throw new Error("Operator #{kk} not supported")
+      else
+        v = v.replace(/[^a-zA-Z0-9\-]+/g, '_')
+        filters["Tags.#{v}"] = true
+    else
+      switch k
+        when '_id' then k = 'DeviceID.ID'
+        when '_deviceId._Manufacturer' then k = 'DeviceID.Manufacturer'
+        when '_deviceId._OUI' then k = 'DeviceID.OUI'
+        when '_deviceId._ProductClass' then k = 'DeviceID.ProductClass'
+        when '_deviceId._SerialNumber' then k = 'DeviceID.SerialNumber'
+        when '_lastInform' then k = 'Events.Inform'
+        when '_lastBootstrap' then k = 'Events.0_BOOTSTRAP'
+        when '_lastBoot' then k = 'Events.1_BOOT'
+        when '_registered' then k = 'Events.Registered'
+
+      if common.typeOf(v) is common.OBJECT_TYPE
+        for kk, vv of v
+          switch kk
+            when '$eq'
+              filters[k] = vv
+            when '$ne'
+              filters["#{k}!"] = vv
+            when '$lt'
+              filters["#{k}<"] = vv
+            when '$lte'
+              filters["#{k}<="] = vv
+            when '$gt'
+              filters["#{k}>"] = vv
+            when '$gte'
+              filters["#{k}>="] = vv
+            else
+              throw new Error("Oprator #{kk} not supported")
+      else
+        filters[k] = v
+
+  return filters
+
+
+sanitizeQueryTypes = (query, types) ->
   for k, v of query
     if k[0] == '$' # logical operator
-      for i in [0...v.length]
-        substituteObjectId(v[i])
-    else if k == '_id'
-      if common.typeOf(v) is common.STRING_TYPE
-        query[k] = mongodb.ObjectID(v) if v.length == 24
-      else if common.typeOf(v) is common.OBJECT_TYPE
+      sanitizeQueryTypes(vv, types) for vv in v
+    else if k of types
+      if common.typeOf(v) is common.OBJECT_TYPE
         for kk, vv of v
           switch kk # operator
             when '$in', '$nin'
-              for i in [0...vv.length]
-                vv[i] = mongodb.ObjectID(vv[i]) if vv[i].length == 24
+              vv[i] = types[k](vv[i]) for i in [0...vv.length]
             when '$eq', '$gt', '$gte', '$lt', '$lte', '$ne'
-              v[kk] = mongodb.ObjectID(vv) if vv.length == 24
+              v[kk] = types[k](vv)
             when '$exists', '$type' then
               # ignore
             else
               throw new Error('Operator not supported')
+      else
+        query[k] = types[k](query[k])
 
   return query
 
 
 exports.expand = expand
-exports.substituteObjectId = substituteObjectId
+exports.queryProjection = queryProjection
+exports.test = test
+exports.sanitizeQueryTypes = sanitizeQueryTypes
+exports.convertMongoQueryToFilters = convertMongoQueryToFilters
+exports.testFilter = testFilter

lib/sandbox.js

@@ -0,0 +1,364 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const vm = require("vm");
+
+const seedrandom = require("seedrandom");
+
+const common = require("./common");
+const device = require("./device");
+const extensions = require("./extensions");
+const logger = require("./logger");
+const scheduling = require("./scheduling");
+
+
+// Used for throwing to exit user script and commit
+const COMMIT = Symbol();
+
+// Used to execute extensions and restart
+const EXT = Symbol();
+
+const context = vm.createContext();
+
+let state;
+
+
+class SandboxDate {
+  constructor() {
+    if (arguments.length)
+      return new Date(...arguments);
+
+    return new Date(state.sessionContext.timestamp);
+  }
+
+  static now(intervalOrCron, variance) {
+    let t = state.sessionContext.timestamp;
+
+    if (typeof intervalOrCron === "number") {
+
+      if (variance == null)
+        variance = intervalOrCron;
+
+      let offset = 0;
+      if (variance)
+        offset = scheduling.variance(state.sessionContext.deviceId, variance);
+
+      t = scheduling.interval(t, intervalOrCron, offset);
+    }
+    else if (typeof intervalOrCron === "string") {
+      let offset = 0;
+      if (variance)
+        offset = scheduling.variance(state.sessionContext.deviceId, variance);
+      let cron = scheduling.parseCron(intervalOrCron);
+      t = scheduling.cron(t, cron, offset)[0];
+    }
+    else if (intervalOrCron) {
+      throw new Error("Invalid Date.now() argument");
+    }
+
+    return t;
+  }
+};
+
+function random() {
+  if (!state.rng)
+    state.rng = seedrandom(state.sessionContext.deviceId);
+
+  return state.rng();
+}
+
+random.seed = function(s) {
+  state.rng = seedrandom(s);
+}
+
+
+class ParameterWrapper {
+  constructor(path, attributes, unpacked, unpackedRevision) {
+    for (let attrName of attributes) {
+      Object.defineProperty(this, attrName, {get: function() {
+        if (state.uncommitted)
+          commit();
+
+        if (state.revision !== unpackedRevision) {
+          unpackedRevision = state.revision;
+          unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision);
+        }
+
+        if (!unpacked.length)
+          return undefined;
+
+        let attr = state.sessionContext.deviceData.attributes.get(unpacked[0], state.revision)[attrName];
+
+        if (!attr)
+          return undefined;
+
+        return attr[1];
+      }});
+    }
+
+    Object.defineProperty(this, "path", {get: function() {
+      if (state.uncommitted)
+        commit();
+
+      if (state.revision !== unpackedRevision) {
+        unpackedRevision = state.revision;
+        unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision);
+      }
+
+      if (!unpacked.length)
+        return undefined;
+
+      return unpacked[0].join(".");
+    }});
+
+    Object.defineProperty(this, "size", {get: function() {
+      if (state.uncommitted)
+        commit();
+
+      if (state.revision !== unpackedRevision) {
+        unpackedRevision = state.revision;
+        unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision);
+      }
+
+      if (!unpacked.length)
+        return undefined;
+
+      return unpacked.length;
+    }});
+
+    this[Symbol.iterator] = function*() {
+      if (state.uncommitted)
+        commit();
+
+      if (state.revision !== unpackedRevision) {
+        unpackedRevision = state.revision;
+        unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision);
+      }
+
+      for (let p of unpacked)
+        yield new ParameterWrapper(p, attributes, [p], state.revision);
+    }
+  }
+}
+
+
+function declare(path, timestamps, values) {
+  state.uncommitted = true;
+  if (!timestamps)
+    timestamps = {};
+
+  if (!values)
+    values = {};
+
+  let parsedPath = common.parsePath(path);
+
+  let declaration = [parsedPath, 1];
+  let attrs = {};
+
+  for (let attrName in values) {
+    if (!timestamps[attrName])
+      timestamps[attrName] = 1;
+
+    if (attrName === "path")
+      declaration[3] = values.path;
+    else {
+      attrs[attrName] = 1;
+      declaration[4] = declaration[4] || {};
+      if (attrName === "value" && !Array.isArray(values.value))
+        declaration[4].value = [values.value];
+      else
+        declaration[4][attrName] = values[attrName];
+    }
+  }
+
+  for (let attrName in timestamps) {
+    if (attrName === "path")
+      declaration[1] = timestamps.path;
+    else {
+      attrs[attrName] = 1;
+      declaration[2] = declaration[2] || {};
+      declaration[2][attrName] = timestamps[attrName];
+    }
+  }
+
+  state.declarations.push(declaration);
+
+  return new ParameterWrapper(parsedPath, Object.keys(attrs));
+}
+
+
+function clear(path, timestamp, attributes) {
+  state.uncommitted = true;
+
+  if (state.revision === state.maxRevision)
+    state.clear.push([common.parsePath(path), timestamp, attributes]);
+}
+
+
+function commit() {
+  ++ state.revision;
+  state.uncommitted = false;
+
+  if (state.revision === state.maxRevision + 1)
+    throw COMMIT;
+  else if (state.revision > state.maxRevision + 1)
+    throw new Error("Declare function should not be called from within a try/catch block");
+}
+
+
+function ext() {
+  ++ state.extCounter;
+  let extCall = Array.from(arguments).map(String);
+  let key = `${state.revision}: ${JSON.stringify(extCall)}`;
+
+  if (key in state.sessionContext.extensionsCache)
+    return state.sessionContext.extensionsCache[key];
+
+  state.extensions[key] = extCall;
+  throw EXT;
+}
+
+
+function log(msg, meta) {
+  if (state.revision === state.maxRevision && state.extCounter >= 0) {
+    const details = Object.assign({}, meta, {
+      sessionContext: state.sessionContext,
+      message: `Script: ${msg}`
+    });
+
+    delete details.hostname;
+    delete details.pid;
+    delete details.name;
+    delete details.version;
+    delete details.deviceId;
+    delete details.remoteAddress;
+
+    logger.accessInfo(details);
+  }
+}
+
+
+Object.defineProperty(context, "Date", {value: SandboxDate});
+Object.defineProperty(context, "declare", {value: declare});
+Object.defineProperty(context, "clear", {value: clear});
+Object.defineProperty(context, "commit", {value: commit});
+Object.defineProperty(context, "ext", {value: ext});
+Object.defineProperty(context, "log", {value: log});
+
+// Monkey-patch Math.random() to make it deterministic
+context.random = random;
+vm.runInContext("Math.random = random;", context);
+delete context.random;
+
+
+function errorToFault(err) {
+  if (!err)
+    return null;
+
+  if (!err.name)
+    return {code: "script", message: `${err}`};
+
+  let fault = {
+    code: `script.${err.name}`,
+    message: err.message,
+    detail: {
+      name: err.name,
+      message: err.message
+    }
+  };
+
+  if (err.stack) {
+    // Trim the stack trace at the self-executing anonymous wrapper function
+    let stackTrimIndex = err.stack.match(/\s+at\s[^\s]+\s+at\s[^\s]+\s\(vm\.js.+\)/).index;
+    fault.detail.stack = err.stack.slice(0, stackTrimIndex);
+  }
+
+  return fault;
+}
+
+
+function run(script, globals, sessionContext, startRevision, maxRevision, callback, extCounter = 0) {
+  state = {
+    sessionContext: sessionContext,
+    revision: startRevision,
+    maxRevision: maxRevision,
+    uncommitted: false,
+    declarations: [],
+    extensions: {},
+    clear: [],
+    rng: null,
+    extCounter: extCounter
+  };
+
+  for (let n of Object.keys(context))
+    delete context[n];
+
+  Object.assign(context, globals);
+
+  let ret, status;
+
+  try {
+    ret = script.runInContext(context, {displayErrors: false});
+    status = 0;
+  }
+  catch (err) {
+    if (err === COMMIT)
+      status = 1;
+    else if (err === EXT)
+      status = 2;
+    else
+      return callback(null, errorToFault(err));
+  }
+
+  let _state = state;
+  let args = Array.from(arguments);
+
+  // Need to maintain a counter of ext() calls to avoid calling certain
+  // functions (e.g. log()) multiple times as calling ext() results in
+  // re-execution of script without revision being incremented.
+  args[6] = 0 - (state.extCounter - extCounter);
+
+  let counter = 3;
+  for (let key of Object.keys(_state.extensions)) {
+    counter += 2;
+    extensions.run(_state.extensions[key], function(err, fault, res) {
+      if (err || fault) {
+        if (counter & 1)
+          callback(err, fault);
+        return counter = 0;
+      }
+      _state.sessionContext.extensionsCache[key] = res;
+      if ((counter -= 2) === 1)
+        if (status === 2)
+          return run.apply(null, args);
+        else
+          return callback(null, null, _state.clear, _state.declarations, status === 0, ret);
+    });
+  }
+
+  if ((counter -= 2) === 1)
+    if (status === 2)
+      return run.apply(null, args);
+    else
+      return callback(null, null, _state.clear, _state.declarations, status === 0, ret);
+}
+
+
+exports.run = run;

lib/scheduling.js

@@ -0,0 +1,78 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const crypto = require("crypto");
+
+const later = require("later");
+
+
+function md532(str) {
+  const digest = crypto
+    .createHash("md5")
+    .update(str)
+    .digest();
+  return (
+    digest.readUInt32LE(0) ^
+    digest.readUInt32LE(4) ^
+    digest.readUInt32LE(8) ^
+    digest.readUInt32LE(12)
+  );
+}
+
+
+function variance(deviceId, variance) {
+  return (md532(deviceId) >>> 0) % variance;
+}
+
+
+function interval(timestamp, interval, offset = 0) {
+  return Math.trunc((timestamp + offset) / interval) * interval - offset;
+}
+
+
+function parseCron(cronExp) {
+  let parts = cronExp.trim().split(/\s+/);
+  if (parts.length === 5)
+    parts.unshift("*");
+
+  return later.schedule(later.parse.cron(parts.join(" "), true));
+}
+
+
+function cron(timestamp, schedule, offset = 0) {
+  // TODO later.js doesn't throw erorr if expression is invalid!
+  const ret = [0, 0];
+
+  let prev = schedule.prev(1, new Date(timestamp + offset));
+  if (prev)
+    ret[0] = prev.setMilliseconds(0) - offset;
+
+  let next = schedule.next(1, new Date(timestamp + offset + 1000));
+  if (next)
+    ret[1] = next.setMilliseconds(0) - offset;
+
+  return ret;
+}
+
+
+exports.variance = variance;
+exports.interval = interval;
+exports.parseCron = parseCron;
+exports.cron = cron;

lib/server.coffee

@@ -1,5 +1,7 @@
 ###
-# Copyright 2013, 2014  Zaid Abdulla
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
 #
 # GenieACS is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as
@@ -15,52 +17,74 @@
 # along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
 ###
 
-domain = require 'domain'
-util = require 'util'
-cluster = require 'cluster'
-config = require './config'
-db = require './db'
-
-service = config.argv[0]
+service = process.argv[2]
 if not service?
   throw new Error('Missing argument cwmp, fs, or nbi')
 
+util = require 'util'
+cluster = require 'cluster'
+logger = require './logger'
+logger.init(service, require('../package.json').version)
+config = require './config'
+db = require './db'
+cache = require './cache'
+extensions = require './extensions'
+
 networkInterface = config.get("#{service.toUpperCase()}_INTERFACE")
 port = config.get("#{service.toUpperCase()}_PORT")
 useHttps = config.get("#{service.toUpperCase()}_SSL")
 
-serviceListener = require("./#{service}").listener
+listener = require("./#{service}").listener
+onConnection = require("./#{service}").onConnection
 
 server = null
 
-listener = (httpRequest, httpResponse) ->
-  d = domain.create()
+closeServer = (timeout, callback) ->
+  return callback() if not server
 
-  d.on('error', (err) ->
-    util.error("#{new Date().toISOString()} - #{err.stack}\n")
-    try
-      killTimer = setTimeout(() ->
-        process.exit(1)
-      , 30000)
+  setTimeout(() ->
+    return if not callback
 
-      killTimer.unref()
+    # Ignore HTTP requests from connection that may still be open
+    server.removeListener('request', listener)
+    server.setTimeout(1)
+
+    cb = callback
+    callback = null
+    setTimeout(cb, 1000)
+  , timeout).unref()
 
-      cluster.worker?.disconnect()
   server.close(() ->
+    return if not callback
+
+    cb = callback
+    callback = null
+    cb()
+  )
+
+
+exit = () ->
+  setTimeout(() ->
+    extensions.killAll(() ->
+      process.exit(1)
+    )
+  , 30000).unref()
+
+  closeServer(20000, () ->
     db.disconnect()
+    cache.disconnect()
+    extensions.killAll()
+    cluster.worker?.disconnect()
   )
 
-      httpResponse.writeHead(500, {'Connection' : 'close'})
-      httpResponse.end("#{err.name}: #{err.message}")
-    catch err2
-      util.error("#{new Date().toISOString()} - #{err2.stack}\n")
-  )
 
-  d.add(httpRequest)
-  d.add(httpResponse)
-
-  d.run(() ->
-    serviceListener(httpRequest, httpResponse)
+process.on('uncaughtException', (err) ->
+  logger.error({
+    message: 'Uncaught exception'
+    exception: err,
+    pid: process.pid
+  })
+  exit()
 )
 
 
@@ -69,15 +93,31 @@ if useHttps
   fs = require 'fs'
   httpsKey = path.resolve(config.get('CONFIG_DIR'), "#{service}.key")
   httpsCert = path.resolve(config.get('CONFIG_DIR'), "#{service}.crt")
+  httpsCa = path.resolve(config.get('CONFIG_DIR'), "#{service}.ca-bundle")
   options = {
     key: fs.readFileSync(httpsKey),
     cert: fs.readFileSync(httpsCert)
   }
+
+  try
+    # Use intermediate certificates if available
+    options.ca = fs.readFileSync(httpsCa).toString()
+      .match(/\-+BEGIN CERTIFICATE\-+[0-9a-zA-Z\+\-\/\=\s]+?\-+END CERTIFICATE\-+/g)
+
   server = require('https').createServer(options, listener)
+  server.on('secureConnection', onConnection) if onConnection?
 else
   server = require('http').createServer(listener)
+  server.on('connection', onConnection) if onConnection?
 
 db.connect((err) ->
+  throw err if err
+  cache.connect((err) ->
     throw err if err
     server.listen(port, networkInterface)
   )
+)
+
+
+process.on('SIGINT', exit)
+process.on('SIGTERM', exit)

lib/soap.coffee

@@ -1,433 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-#
-# This file incorporates work covered by the following copyright and
-# permission notice:
-#
-# Copyright 2013 Fanoos Telecom
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-###
-
-config = require './config'
-libxmljs = require 'libxmljs'
-
-SERVER_NAME = "GenieACS/#{require('../package.json').version}"
-
-NAMESPACES = {
-  '1.0' : {
-    'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-    'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-    'xsd' : 'http://www.w3.org/2001/XMLSchema',
-    'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-    'cwmp' : 'urn:dslforum-org:cwmp-1-0'
-  },
-  '1.1' : {
-    'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-    'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-    'xsd' : 'http://www.w3.org/2001/XMLSchema',
-    'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-    'cwmp' : 'urn:dslforum-org:cwmp-1-1'
-  },
-  '1.2' : {
-    'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-    'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-    'xsd' : 'http://www.w3.org/2001/XMLSchema',
-    'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-    'cwmp' : 'urn:dslforum-org:cwmp-1-2'
-  },
-  '1.3' : {
-    'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-    'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-    'xsd' : 'http://www.w3.org/2001/XMLSchema',
-    'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-    'cwmp' : 'urn:dslforum-org:cwmp-1-2'
-  },
-  '1.4' : {
-    'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-    'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-    'xsd' : 'http://www.w3.org/2001/XMLSchema',
-    'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-    'cwmp' : 'urn:dslforum-org:cwmp-1-3'
-  }
-}
-
-# Generate Libxmljs options
-LIBXMLJS_OPTIONS = {}
-for k, v of config.allConfig
-  if k.lastIndexOf('XML_PARSE_', 0) == 0
-    LIBXMLJS_OPTIONS[k] = v
-
-
-# Workaround for devices that don't use correct namespace declarations or prefixes
-if config.get('IGNORE_XML_NAMESPACES')
-  libxmljs.Element.prototype.__find = libxmljs.Element.prototype.find
-  libxmljs.Element.prototype.find = (xpath, namespaces) ->
-    # Modify xpath queries to work regardless of element's namespace
-    p = xpath.replace(/([^\/:]+:)?([^\/]+)(\/|$)/g, '$2$3') \
-      .replace(/([a-zA-Z0-9_-]+)([^\/]*)(\/|$)/g, "*[local-name()='$1']$2$3")
-    libxmljs.Element.prototype.__find.call(this, p, namespaces)
-
-
-event = (xml) ->
-  e.text().trim() for e in xml.find('EventStruct/EventCode')
-
-
-parameterInfoList = (xml) ->
-  infoList = []
-  for e in xml.find('ParameterInfoStruct')
-    name = e.get('Name').text().trim()
-    writable = Boolean(JSON.parse(e.get('Writable').text()))
-    infoList.push([name, writable])
-  return infoList
-
-
-parameterValueList = (xml) ->
-  valueList = []
-  for e in xml.find('ParameterValueStruct')
-    valueType = e.get('Value').attr('type').value().trim()
-    name = e.get('Name').text().trim()
-    value = e.get('Value').text().trim()
-    try
-      switch valueType
-        when 'xsd:boolean'
-          value = Boolean(JSON.parse(value))
-        when 'xsd:unsignedInt'
-          value = JSON.parse(value)
-        when 'xsd:int'
-          value = JSON.parse(value)
-    catch err
-      value = null
-    valueList.push([name, value, valueType])
-  return valueList
-
-
-cpeGetParameterNames = (xml, methodRequest) ->
-  el = xml.node('cwmp:GetParameterNames')
-  el.node('ParameterPath').text(methodRequest.parameterPath)
-  el.node('NextLevel').text(+methodRequest.nextLevel)
-
-
-cpeGetParameterNamesResponse = (xml) ->
-  {parameterList : parameterInfoList(xml.get('ParameterList'))}
-
-
-cpeGetParameterValues = (xml, methodRequest) ->
-  el = xml.node('cwmp:GetParameterValues').node('ParameterNames')
-  el.attr({'soap-enc:arrayType' : "xsd:string[#{methodRequest.parameterNames.length}]"})
-  for p in methodRequest.parameterNames
-    el.node('string').text(p)
-
-
-cpeGetParameterValuesResponse = (xml) ->
-  {parameterList : parameterValueList(xml.get('ParameterList'))}
-
-
-cpeSetParameterValues = (xml, methodRequest) ->
-  el = xml.node('cwmp:SetParameterValues')
-  paramList = el.node('ParameterList')
-  paramList.attr({'soap-enc:arrayType' : "cwmp:ParameterValueStruct[#{methodRequest.parameterList.length}]"})
-  for i in methodRequest.parameterList
-    pvs = paramList.node('ParameterValueStruct')
-    pvs.node('Name').text(i[0])
-    v = pvs.node('Value')
-    v.text(i[1])
-    v.attr({'xsi:type' : i[2]}) if i[2]?
-  el.node('ParameterKey').text(methodRequest.parameterKey ? '')
-
-
-cpeSetParameterValuesResponse = (xml) ->
-  {status : JSON.parse(xml.get('Status').text())}
-
-
-cpeAddObject = (xml, methodRequest) ->
-  el = xml.node('cwmp:AddObject')
-  el.node('ObjectName').text(methodRequest.objectName)
-  el.node('ParameterKey').text(methodRequest.parameterKey ? '')
-
-
-cpeAddObjectResponse = (xml) ->
-  {
-    instanceNumber : parseInt(xml.get('InstanceNumber').text()),
-    status : parseInt(xml.get('Status').text())
-  }
-
-
-cpeDeleteObject = (xml, methodRequest) ->
-  el = xml.node('cwmp:DeleteObject')
-  el.node('ObjectName').text(methodRequest.objectName)
-  el.node('ParameterKey').text(methodRequest.parameterKey ? '')
-
-
-cpeDeleteObjectResponse = (xml) ->
-  {status : parseInt(xml.get('Status').text())}
-
-
-cpeReboot = (xml, methodRequest) ->
-  el = xml.node('cwmp:Reboot')
-  el.node('CommandKey').text(methodRequest.commandKey ? '')
-
-
-cpeRebootResponse = (xml) ->
-  {}
-
-
-cpeFactoryReset = (xml, methodRequest) ->
-  el = xml.node('cwmp:FactoryReset')
-
-
-cpeFactoryResetResponse = (xml, methodRequest) ->
-  {}
-
-
-cpeDownload = (xml, methodRequest) ->
-  el = xml.node('cwmp:Download')
-  el.node('CommandKey').text(methodRequest.commandKey ? '')
-  el.node('FileType').text(methodRequest.fileType)
-  el.node('URL').text(methodRequest.url)
-  el.node('Username').text(methodRequest.username ? '')
-  el.node('Password').text(methodRequest.password ? '')
-  el.node('FileSize').text(methodRequest.fileSize ? '0')
-  el.node('TargetFileName').text(methodRequest.TargetFileName ? '')
-  el.node('DelaySeconds').text(methodRequest.delaySeconds ? '0')
-  el.node('SuccessURL').text(methodRequest.successUrl ? '')
-  el.node('FailureURL').text(methodRequest.failureUrl ? '')
-
-
-cpeDownloadResponse = (xml) ->
-  {status : JSON.parse(xml.get('Status').text())}
-
-
-acsInform = (xml) ->
-  {
-    parameterList : parameterValueList(xml.get('ParameterList')),
-    deviceId : traverseXml(xml.get('DeviceId')),
-    event : event(xml.get('Event')),
-    retryCount : parseInt(xml.get('RetryCount').text())
-  }
-
-
-acsInformResponse = (xml, methodResponse) ->
-  xml.node('cwmp:InformResponse').node('MaxEnvelopes').text(1)
-
-
-acsGetRPCMethods = (xml) ->
-  {}
-
-
-acsGetRPCMethodsResponse = (xml, methodResponse) ->
-  el = xml.node('cwmp:GetRPCMethodsResponse').node('MethodList')
-  el.attr({'soap-enc:arrayType' : "xsd:string[#{methodResponse.methodList.length}]"})
-  for m in methodResponse.methodList
-    el.node('string').text(m)
-
-
-acsTransferComplete = (xml) ->
-  {
-    commandKey : xml.get('CommandKey').text(),
-    faultStruct : traverseXml(xml.get('FaultStruct')),
-    startTime : xml.get('StartTime').text(), # TODO convert to datetime
-    completeTime : xml.get('CompleteTime').text() # TODO convert to datetime
-  }
-
-
-acsTransferCompleteResponse = (xml, methodResponse) ->
-  xml.node('cwmp:TransferCompleteResponse').text('')
-
-
-acsRequestDownload = (xml) ->
-  # TODO FileTypeArg
-  {
-    fileType : xml.get('FileType').text()
-  }
-
-
-acsRequestDownloadResponse = (xml, methodResponse) ->
-  xml.node('cwmp:RequestDownloadResponse').text('')
-
-
-traverseXml = (xml) ->
-  obj = {}
-  for n in xml.childNodes()
-    if n.type() == 'element'
-      obj[n.name()] = traverseXml(n)
-
-  if Object.keys(obj).length == 0
-    xml.text()
-  else
-    obj
-
-
-fault = (xml) ->
-  return traverseXml(xml)
-
-
-exports.request = (httpRequest, cwmpVersion) ->
-  cwmpRequest = {cwmpVersion : cwmpVersion}
-
-  data = httpRequest.getBody()
-
-  if data.length > 0
-    xml = libxmljs.parseXml(data, LIBXMLJS_OPTIONS)
-
-    if not cwmpRequest.cwmpVersion?
-      # cwmpVersion not passed, thus it's an inform request
-      methodElement = xml.get('/soap-env:Envelope/soap-env:Body/*', NAMESPACES['1.0'])
-      switch methodElement.namespace().href()
-        when 'urn:dslforum-org:cwmp-1-0'
-          cwmpRequest.cwmpVersion = '1.0'
-        when 'urn:dslforum-org:cwmp-1-1'
-          cwmpRequest.cwmpVersion = '1.1'
-        when 'urn:dslforum-org:cwmp-1-2'
-          cwmpRequest.sessionTimeout = try xml.get('/soap-env:Envelope/soap-env:Header/cwmp:sessionTimeout', NAMESPACES['1.2']).text() catch then null
-          if cwmpRequest.sessionTimeout?
-            cwmpRequest.cwmpVersion = '1.3'
-          else
-            cwmpRequest.cwmpVersion = '1.2'
-        when 'urn:dslforum-org:cwmp-1-3'
-          cwmpRequest.cwmpVersion = '1.4'
-        else
-          throw new Error('Unrecognized CWMP version')
-
-      cwmpRequest.sessionTimeout ?= try xml.get('/soap-env:Envelope/soap-env:Header/cwmp:sessionTimeout', NAMESPACES[cwmpRequest.cwmpVersion]).text() catch then null
-    else
-      methodElement = xml.get('/soap-env:Envelope/soap-env:Body/cwmp:*', NAMESPACES[cwmpRequest.cwmpVersion])
-
-    cwmpRequest.id = try xml.get('/soap-env:Envelope/soap-env:Header/cwmp:ID', NAMESPACES[cwmpRequest.cwmpVersion]).text() catch then null
-
-    if methodElement? and not (config.get('IGNORE_XML_NAMESPACES') and methodElement.name() is 'Fault')
-      switch methodElement.name()
-        when 'Inform'
-          cwmpRequest.methodRequest = acsInform(methodElement)
-          cwmpRequest.methodRequest.type = 'Inform'
-        when 'GetRPCMethods'
-          cwmpRequest.methodRequest = acsGetRPCMethods(methodElement)
-          cwmpRequest.methodRequest.type = 'GetRPCMethods'
-        when 'TransferComplete'
-          cwmpRequest.methodRequest = acsTransferComplete(methodElement)
-          cwmpRequest.methodRequest.type = 'TransferComplete'
-        when 'RequestDownload'
-          cwmpRequest.methodRequest = acsRequestDownload(methodElement)
-          cwmpRequest.methodRequest.type = 'RequestDownload'
-        when 'GetParameterNamesResponse'
-          cwmpRequest.methodResponse = cpeGetParameterNamesResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'GetParameterNamesResponse'
-        when 'GetParameterValuesResponse'
-          cwmpRequest.methodResponse = cpeGetParameterValuesResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'GetParameterValuesResponse'
-        when 'SetParameterValuesResponse'
-          cwmpRequest.methodResponse = cpeSetParameterValuesResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'SetParameterValuesResponse'
-        when 'AddObjectResponse'
-          cwmpRequest.methodResponse = cpeAddObjectResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'AddObjectResponse'
-        when 'DeleteObjectResponse'
-          cwmpRequest.methodResponse = cpeDeleteObjectResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'DeleteObjectResponse'
-        when 'RebootResponse'
-          cwmpRequest.methodResponse = cpeRebootResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'RebootResponse'
-        when 'FactoryResetResponse'
-          cwmpRequest.methodResponse = cpeFactoryResetResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'FactoryResetResponse'
-        when 'DownloadResponse'
-          cwmpRequest.methodResponse = cpeDownloadResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'DownloadResponse'
-        else
-          throw Error('8000 Method not supported ' + methodElement.name())
-    else
-      faultElement = xml.get('/soap-env:Envelope/soap-env:Body/soap-env:Fault', NAMESPACES[cwmpRequest.cwmpVersion])
-      cwmpRequest.fault = fault(faultElement)
-    
-  return cwmpRequest
-
-
-createSoapEnv = (cwmpVersion) ->
-  xml = libxmljs.Document()
-  env = xml.node('soap-env:Envelope')
-  for prefix, url of NAMESPACES[cwmpVersion]
-    env.defineNamespace prefix, url
-  return env
-
-
-exports.response = (cwmpResponse) ->
-  headers = {
-    'Server' : SERVER_NAME,
-    'SOAPServer' : SERVER_NAME,
-  }
-
-  if cwmpResponse?.methodResponse?
-    env = createSoapEnv(cwmpResponse.cwmpVersion)
-    header = env.node('soap-env:Header')
-    header.node('cwmp:ID').attr({'soap-env:mustUnderstand' : 1}).text(cwmpResponse.id)
-    body = env.node('soap-env:Body')
-    switch cwmpResponse.methodResponse.type
-      when 'InformResponse'
-        acsInformResponse(body, cwmpResponse.methodResponse)
-      when 'GetRPCMethodsResponse'
-        acsGetRPCMethodsResponse(body, cwmpResponse.methodResponse)
-      when 'TransferCompleteResponse'
-        acsTransferCompleteResponse(body, cwmpResponse.methodResponse)
-      when 'RequestDownloadResponse'
-        acsRequestDownloadResponse(body, cwmpResponse.methodResponse)
-      else
-        throw Error("Unknown method response type #{cwmpResponse.methodResponse.type}")
-  else if cwmpResponse?.methodRequest?
-    env = createSoapEnv(cwmpResponse.cwmpVersion)
-    header = env.node('soap-env:Header')
-    header.node('cwmp:ID').attr({'soap-env:mustUnderstand' : 1}).text(cwmpResponse.id)
-    body = env.node('soap-env:Body')
-    switch cwmpResponse.methodRequest.type
-      when 'GetParameterNames'
-        cpeGetParameterNames(body, cwmpResponse.methodRequest)
-      when 'GetParameterValues'
-        cpeGetParameterValues(body, cwmpResponse.methodRequest)
-      when 'SetParameterValues'
-        cpeSetParameterValues(body, cwmpResponse.methodRequest)
-      when 'AddObject'
-        cpeAddObject(body, cwmpResponse.methodRequest)
-      when 'DeleteObject'
-        cpeDeleteObject(body, cwmpResponse.methodRequest)
-      when 'Reboot'
-        cpeReboot(body, cwmpResponse.methodRequest)
-      when 'FactoryReset'
-        cpeFactoryReset(body, cwmpResponse.methodRequest)
-      when 'Download'
-        cpeDownload(body, cwmpResponse.methodRequest)
-      else
-        throw Error("Unknown method request #{cwmpResponse.methodRequest.type}")
-  
-  if env?
-    headers['Content-Type'] = 'text/xml; charset="utf-8"'
-    return {code: 200, headers: headers, data: new Buffer(env.doc().toString(false))}
-  else
-    return {code: 204, headers: headers, data: new Buffer(0)}

lib/soap.js

@@ -0,0 +1,612 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const libxmljs = require("libxmljs");
+
+const config = require("./config");
+
+const SERVER_NAME = `GenieACS/${require("../package.json").version}`;
+
+const NAMESPACES = {
+  "1.0": {
+    "soap-enc": "http://schemas.xmlsoap.org/soap/encoding/",
+    "soap-env": "http://schemas.xmlsoap.org/soap/envelope/",
+    "xsd": "http://www.w3.org/2001/XMLSchema",
+    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
+    "cwmp": "urn:dslforum-org:cwmp-1-0"
+  },
+  "1.1": {
+    "soap-enc": "http://schemas.xmlsoap.org/soap/encoding/",
+    "soap-env": "http://schemas.xmlsoap.org/soap/envelope/",
+    "xsd": "http://www.w3.org/2001/XMLSchema",
+    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
+    "cwmp": "urn:dslforum-org:cwmp-1-1"
+  },
+  "1.2": {
+    "soap-enc": "http://schemas.xmlsoap.org/soap/encoding/",
+    "soap-env": "http://schemas.xmlsoap.org/soap/envelope/",
+    "xsd": "http://www.w3.org/2001/XMLSchema",
+    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
+    "cwmp": "urn:dslforum-org:cwmp-1-2"
+  },
+  "1.3": {
+    "soap-enc": "http://schemas.xmlsoap.org/soap/encoding/",
+    "soap-env": "http://schemas.xmlsoap.org/soap/envelope/",
+    "xsd": "http://www.w3.org/2001/XMLSchema",
+    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
+    "cwmp": "urn:dslforum-org:cwmp-1-2"
+  },
+  "1.4": {
+    "soap-enc": "http://schemas.xmlsoap.org/soap/encoding/",
+    "soap-env": "http://schemas.xmlsoap.org/soap/envelope/",
+    "xsd": "http://www.w3.org/2001/XMLSchema",
+    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
+    "cwmp": "urn:dslforum-org:cwmp-1-3"
+  }
+};
+
+const LIBXMLJS_PARSE_OPTIONS = {
+  "nocdata": true,
+  "recover": config.get("XML_RECOVER"),
+  "ignore_enc": config.get("XML_IGNORE_ENC")
+};
+
+const LIBXMLJS_SAVE_OPTIONS = {
+  "format": config.get("XML_FORMAT"),
+  "declaration": !config.get("XML_NO_DECL"),
+  "selfCloseEmpty": !config.get("XML_NO_EMPTY")
+};
+
+const XML_IGNORE_NAMESPACE = config.get("XML_IGNORE_NAMESPACE");
+
+if (XML_IGNORE_NAMESPACE) {
+  libxmljs.Element.prototype.__find = libxmljs.Element.prototype.find;
+  libxmljs.Element.prototype.find = function(xpath, namespaces) {
+    const p = xpath.replace(/([^\/:]+:)?([^\/]+)(\/|$)/g, "$2$3").replace(/([a-zA-Z0-9_-]+)([^\/]*)(\/|$)/g, "*[local-name()=\"$1\"]$2$3");
+    return libxmljs.Element.prototype.__find.call(this, p, namespaces);
+  };
+}
+
+let warnings;
+
+
+function parseBool(v) {
+  v = "" + v;
+  if (v === "true" || v === "TRUE" || v === "True" || v === "1")
+    return true;
+  else if (v === "false" || v === "FALSE" || v === "False" || v === "0")
+    return false;
+  else
+    return null;
+}
+
+
+function event(xml) {
+  return xml.find("EventStruct/EventCode").map(e => e.text().trim());
+}
+
+
+function parameterInfoList(xml) {
+  return xml.find("ParameterInfoStruct").map(function(e) {
+    let param = e.get("Name").text();
+    let value = e.get("Writable").text();
+    let parsed = parseBool(value);
+
+    if (parsed === null) {
+      warnings.push({
+        message: "Invalid writable attribute",
+        parameter: param
+      });
+      parsed = false;
+    }
+
+    return [param, parsed];
+  });
+}
+
+
+function parameterValueList(xml) {
+  return xml.find("ParameterValueStruct").map(function(e) {
+    const valueElement = e.get("Value");
+    const valueType = valueElement.attr("type").value();
+    let param = e.get("Name").text();
+    let value = valueElement.text();
+    let parsed = value;
+    if (valueType === "xsd:boolean") {
+      parsed = parseBool(value);
+      if (parsed === null) {
+        warnings.push({
+          message: "Invalid value attribute",
+          parameter: param
+        });
+        parsed = value;
+      }
+    }
+    else if (valueType === "xsd:int" || valueType === "xsd:unsignedInt") {
+      parsed = parseInt(value);
+      if (isNaN(parsed)) {
+        warnings.push({
+          message: "Invalid value attribute",
+          parameter: param
+        });
+        parsed = value;
+      }
+    }
+    else if (valueType === "xsd:dateTime") {
+      parsed = Date.parse(value);
+      if (isNaN(parsed)) {
+        warnings.push({
+          message: "Invalid value attribute",
+          parameter: param
+        });
+        parsed = value;
+      }
+    }
+
+   return [param, parsed, valueType];
+  });
+}
+
+
+function GetParameterNames(xml, methodRequest) {
+  const el = xml.node("cwmp:GetParameterNames");
+  el.node("ParameterPath").text(methodRequest.parameterPath);
+  el.node("NextLevel").text(+methodRequest.nextLevel);
+}
+
+
+function GetParameterNamesResponse(xml) {
+  return {
+    name: "GetParameterNamesResponse",
+    parameterList: parameterInfoList(xml.get("ParameterList"))
+  };
+}
+
+
+function GetParameterValues(xml, methodRequest) {
+  const el = xml.node("cwmp:GetParameterValues").node("ParameterNames");
+  el.attr({
+    "soap-enc:arrayType": `xsd:string[${methodRequest.parameterNames.length}]`
+  });
+
+  for (const p of methodRequest.parameterNames)
+    el.node("string").text(p);
+}
+
+
+function GetParameterValuesResponse(xml) {
+  return {
+    name: "GetParameterValuesResponse",
+    parameterList: parameterValueList(xml.get("ParameterList"))
+  };
+}
+
+
+function SetParameterValues(xml, methodRequest) {
+  const el = xml.node("cwmp:SetParameterValues");
+  const paramList = el.node("ParameterList");
+  paramList.attr({
+    "soap-enc:arrayType": `cwmp:ParameterValueStruct[${methodRequest.parameterList.length}]`
+  });
+
+  for (const p of methodRequest.parameterList) {
+    const pvs = paramList.node("ParameterValueStruct");
+    pvs.node("Name").text(p[0]);
+    let val = p[1];
+    if (p[2] === 'xsd:dateTime' && typeof val === 'number') {
+      val = (new Date(val)).toISOString();
+      if (methodRequest.DATETIME_MILLISECONDS === false)
+        val = val.replace('.000', '');
+    }
+    if (p[2] === 'xsd:boolean' && typeof val === 'boolean') {
+      if (methodRequest.BOOLEAN_LITERAL === false)
+        val = +val;
+    }
+    pvs.node("Value").attr({"xsi:type": p[2]}).text(val);
+  }
+
+  el.node("ParameterKey").text(methodRequest.parameterKey || "");
+}
+
+
+function SetParameterValuesResponse(xml) {
+  return {
+    name: "SetParameterValuesResponse",
+    status: parseInt(xml.get("Status").text())
+  };
+}
+
+
+function AddObject(xml, methodRequest) {
+  const el = xml.node("cwmp:AddObject");
+  el.node("ObjectName").text(methodRequest.objectName);
+  el.node("ParameterKey").text(methodRequest.parameterKey || "");
+}
+
+
+function AddObjectResponse(xml) {
+  return {
+    name: "AddObjectResponse",
+    instanceNumber: parseInt(xml.get("InstanceNumber").text()),
+    status: parseInt(xml.get("Status").text())
+  };
+}
+
+
+function DeleteObject(xml, methodRequest) {
+  const el = xml.node("cwmp:DeleteObject");
+  el.node("ObjectName").text(methodRequest.objectName);
+  el.node("ParameterKey").text(methodRequest.parameterKey || "");
+}
+
+
+function DeleteObjectResponse(xml) {
+  return {
+    name: "DeleteObjectResponse",
+    status: parseInt(xml.get("Status").text())
+  };
+}
+
+
+function Reboot(xml, methodRequest) {
+  const el = xml.node("cwmp:Reboot");
+  el.node("CommandKey").text(methodRequest.commandKey || "");
+}
+
+
+function RebootResponse(xml) {
+  return {
+    name: "RebootResponse"
+  };
+}
+
+
+function FactoryReset(xml, methodRequest) {
+  xml.node("cwmp:FactoryReset");
+}
+
+
+function FactoryResetResponse(xml, methodRequest) {
+  return {
+    name: "FactoryResetResponse"
+  };
+}
+
+
+function Download(xml, methodRequest) {
+  const el = xml.node("cwmp:Download");
+  el.node("CommandKey").text(methodRequest.commandKey || "");
+  el.node("FileType").text(methodRequest.fileType);
+  el.node("URL").text(methodRequest.url);
+  el.node("Username").text(methodRequest.username || "");
+  el.node("Password").text(methodRequest.password || "");
+  el.node("FileSize").text(methodRequest.fileSize || "0");
+  el.node("TargetFileName").text(methodRequest.targetFileName || "");
+  el.node("DelaySeconds").text(methodRequest.delaySeconds || "0");
+  el.node("SuccessURL").text(methodRequest.successUrl || "");
+  el.node("FailureURL").text(methodRequest.failureUrl || "");
+}
+
+
+function DownloadResponse(xml) {
+  const res = {
+    name: "DownloadResponse",
+    status: parseInt(xml.get("Status").text())
+  };
+
+  if (res.status === 0) {
+    res.startTime = Date.parse(xml.get("StartTime").text());
+    res.completeTime = Date.parse(xml.get("CompleteTime").text());
+  }
+
+  return res;
+}
+
+
+function Inform(xml) {
+  return {
+    name: "Inform",
+    parameterList: parameterValueList(xml.get("ParameterList")),
+    deviceId: {
+      "Manufacturer": xml.get("DeviceId/Manufacturer").text(),
+      "OUI": xml.get("DeviceId/OUI").text(),
+      "ProductClass": xml.get("DeviceId/ProductClass").text(),
+      "SerialNumber": xml.get("DeviceId/SerialNumber").text()
+    },
+    event: event(xml.get("Event")),
+    retryCount: parseInt(xml.get("RetryCount").text())
+  };
+}
+
+
+function InformResponse(xml, methodResponse) {
+  xml.node("cwmp:InformResponse").node("MaxEnvelopes").text(1);
+}
+
+
+function GetRPCMethods(xml) {
+  return {name: "GetRPCMethods"};
+}
+
+
+function GetRPCMethodsResponse(xml, methodResponse) {
+  const el = xml.node("cwmp:GetRPCMethodsResponse").node("MethodList");
+  el.attr({
+    "soap-enc:arrayType": `xsd:string[${methodResponse.methodList.length}]`
+  });
+
+  for (const m of methodResponse.methodList)
+    el.node("string").text(m);
+}
+
+
+function TransferComplete(xml) {
+  return {
+    name: "TransferComplete",
+    commandKey: xml.get("CommandKey").text(),
+    faultStruct: faultStruct(xml.get("FaultStruct")),
+    startTime: Date.parse(xml.get("StartTime").text()),
+    completeTime: Date.parse(xml.get("CompleteTime").text())
+  };
+}
+
+
+function TransferCompleteResponse(xml, methodResponse) {
+  xml.node("cwmp:TransferCompleteResponse").text("");
+}
+
+
+function RequestDownload(xml) {
+  return {
+    name: "RequestDownload",
+    fileType: xml.get("FileType").text()
+  };
+}
+
+
+function RequestDownloadResponse(xml, methodResponse) {
+  xml.node("cwmp:RequestDownloadResponse").text("");
+}
+
+
+function faultStruct(xml) {
+  const f = {
+    faultCode: xml.get("FaultCode").text(),
+    faultString: xml.get("FaultString").text()
+  };
+
+  for (const e of xml.find("SetParameterValuesFault")) {
+    if (!f.setParameterValuesFault)
+      f.setParameterValuesFault = [];
+
+    f.setParameterValuesFault.push({
+      parameterName: e.get("ParameterName").text(),
+      faultCode: e.get("FaultCode").text(),
+      faultString: e.get("FaultString").text()
+    });
+  }
+
+  return f;
+}
+
+
+function fault(xml, cwmpVersion) {
+  return {
+    faultCode: xml.get("faultcode").text(),
+    faultString: xml.get("faultstring").text(),
+    detail: faultStruct(xml.get("detail/cwmp:Fault", NAMESPACES[cwmpVersion]))
+  };
+}
+
+
+function request(data, cwmpVersion, warn) {
+  warnings = warn;
+
+  const rpc = {
+    cwmpVersion: cwmpVersion
+  };
+
+  if (!data.length)
+    return rpc;
+
+  const xml = libxmljs.parseXml(data, LIBXMLJS_PARSE_OPTIONS);
+
+  let methodElement;
+
+  if (rpc.cwmpVersion) {
+    methodElement = xml.get("/soap-env:Envelope/soap-env:Body/cwmp:*", NAMESPACES[rpc.cwmpVersion]);
+  }
+  else {
+    // cwmpVersion not passed, thus it's an inform request
+    methodElement = xml.get("/soap-env:Envelope/soap-env:Body/*", NAMESPACES["1.0"]);
+    switch (methodElement.namespace().href()) {
+      case "urn:dslforum-org:cwmp-1-0":
+        rpc.cwmpVersion = "1.0";
+        break;
+      case "urn:dslforum-org:cwmp-1-1":
+        rpc.cwmpVersion = "1.1";
+        break;
+      case "urn:dslforum-org:cwmp-1-2":
+        const timeoutElement = xml.get("/soap-env:Envelope/soap-env:Header/cwmp:sessionTimeout", NAMESPACES["1.2"]);
+        if (timeoutElement) {
+          rpc.sessionTimeout = parseInt(timeoutElement.text());
+          rpc.cwmpVersion = "1.3";
+        }
+        else {
+          rpc.cwmpVersion = "1.2";
+        }
+        break;
+      case "urn:dslforum-org:cwmp-1-3":
+        rpc.cwmpVersion = "1.4";
+        break;
+      default:
+        throw new Error("Unrecognized CWMP version");
+    }
+
+    if (!rpc.sessionTimeout) {
+      const timeoutElement = xml.get("/soap-env:Envelope/soap-env:Header/cwmp:sessionTimeout", NAMESPACES[rpc.cwmpVersion]);
+      if (timeoutElement)
+        rpc.sessionTimeout = parseInt(timeoutElement.text());
+    }
+  }
+
+  let idElement = xml.get("/soap-env:Envelope/soap-env:Header/cwmp:ID", NAMESPACES[rpc.cwmpVersion]);
+  if (idElement)
+    rpc.id = idElement.text();
+
+  if (methodElement && !(XML_IGNORE_NAMESPACE && methodElement.name() === "Fault")) {
+    switch (methodElement.name()) {
+      case "Inform":
+        rpc.cpeRequest = Inform(methodElement);
+        break;
+      case "GetRPCMethods":
+        rpc.cpeRequest = GetRPCMethods(methodElement);
+        break;
+      case "TransferComplete":
+        rpc.cpeRequest = TransferComplete(methodElement);
+        break;
+      case "RequestDownload":
+        rpc.cpeRequest = RequestDownload(methodElement);
+        break;
+      case "GetParameterNamesResponse":
+        rpc.cpeResponse = GetParameterNamesResponse(methodElement);
+        break;
+      case "GetParameterValuesResponse":
+        rpc.cpeResponse = GetParameterValuesResponse(methodElement);
+        break;
+      case "SetParameterValuesResponse":
+        rpc.cpeResponse = SetParameterValuesResponse(methodElement);
+        break;
+      case "AddObjectResponse":
+        rpc.cpeResponse = AddObjectResponse(methodElement);
+        break;
+      case "DeleteObjectResponse":
+        rpc.cpeResponse = DeleteObjectResponse(methodElement);
+        break;
+      case "RebootResponse":
+        rpc.cpeResponse = RebootResponse(methodElement);
+        break;
+      case "FactoryResetResponse":
+        rpc.cpeResponse = FactoryResetResponse(methodElement);
+        break;
+      case "DownloadResponse":
+        rpc.cpeResponse = DownloadResponse(methodElement);
+        break;
+      default:
+        throw new Error(`8000 Method not supported ${methodElement.name()}`);
+    }
+  }
+  else {
+    const faultElement = xml.get("/soap-env:Envelope/soap-env:Body/soap-env:Fault", NAMESPACES[rpc.cwmpVersion]);
+    rpc.cpeFault = fault(faultElement, rpc.cwmpVersion);
+  }
+
+  return rpc;
+}
+
+
+function createSoapEnv(cwmpVersion) {
+  const xml = libxmljs.Document();
+  const env = xml.node("soap-env:Envelope");
+
+  for (const prefix of Object.keys(NAMESPACES[cwmpVersion]))
+    env.defineNamespace(prefix, NAMESPACES[cwmpVersion][prefix]);
+
+  return env;
+}
+
+
+function response(rpc) {
+  const headers = {
+    "Server": SERVER_NAME,
+    "SOAPServer": SERVER_NAME
+  };
+
+  if (!rpc)
+    return {code: 204, headers: headers, data: new Buffer(0)};
+
+  const env = createSoapEnv(rpc.cwmpVersion);
+  const header = env.node("soap-env:Header");
+  header.node("cwmp:ID").attr({
+    "soap-env:mustUnderstand": 1
+  }).text(rpc.id);
+
+  const body = env.node("soap-env:Body");
+
+  if (rpc.acsResponse) {
+    switch (rpc.acsResponse.name) {
+      case "InformResponse":
+        InformResponse(body, rpc.acsResponse);
+        break;
+      case "GetRPCMethodsResponse":
+        GetRPCMethodsResponse(body, rpc.acsResponse);
+        break;
+      case "TransferCompleteResponse":
+        TransferCompleteResponse(body, rpc.acsResponse);
+        break;
+      case "RequestDownloadResponse":
+        RequestDownloadResponse(body, rpc.acsResponse);
+        break;
+      default:
+        throw new Error(`Unknown method response type ${rpc.acsResponse.name}`);
+    }
+  }
+  else if (rpc.acsRequest) {
+    switch (rpc.acsRequest.name) {
+      case "GetParameterNames":
+        GetParameterNames(body, rpc.acsRequest);
+        break;
+      case "GetParameterValues":
+        GetParameterValues(body, rpc.acsRequest);
+        break;
+      case "SetParameterValues":
+        SetParameterValues(body, rpc.acsRequest);
+        break;
+      case "AddObject":
+        AddObject(body, rpc.acsRequest);
+        break;
+      case "DeleteObject":
+        DeleteObject(body, rpc.acsRequest);
+        break;
+      case "Reboot":
+        Reboot(body, rpc.acsRequest);
+        break;
+      case "FactoryReset":
+        FactoryReset(body, rpc.acsRequest);
+        break;
+      case "Download":
+        Download(body, rpc.acsRequest);
+        break;
+      default:
+        throw new Error(`Unknown method request ${rpc.acsRequest.name}`);
+    }
+  }
+
+  headers["Content-Type"] = "text/xml; charset=\"utf-8\"";
+  return {
+    code: 200,
+    headers: headers,
+    data: new Buffer(env.doc().toString(LIBXMLJS_SAVE_OPTIONS))
+  };
+}
+
+
+exports.request = request;
+exports.response = response;

lib/tasks.coffee

@@ -1,367 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-#
-# This file incorporates work covered by the following copyright and
-# permission notice:
-#
-# Copyright 2013 Fanoos Telecom
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-###
-
-config = require './config'
-common = require './common'
-db = require './db'
-mongodb = require 'mongodb'
-customCommands = require './custom-commands'
-url = require 'url'
-
-exports.STATUS_OK = STATUS_OK = 1
-exports.STATUS_FAULT = STATUS_FAULT = 2
-exports.STATUS_COMPLETED = STATUS_COMPLETED = 4
-exports.STATUS_SAVE = STATUS_SAVE = 128 # not mutually exclusive with the rest
-
-
-this.refreshObject = (task, methodResponse, callback) ->
-  task.session = {} if not task.session?
-
-  if not task.session.subtask?
-    path = task.objectName
-    path += '.' if path != '' and not common.endsWith(path, '.')
-    task.session.subtask = {device : task.device, name : 'getParameterNames', parameterPath : path}
-
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if task.session.subtask.name is 'getParameterNames'
-    this.getParameterNames(task.session.subtask, methodResponse, (err, status, methodRequest, deviceUpdates) =>
-      task.session.parameterNames ?= []
-      if deviceUpdates?.parameterNames?
-        for p in deviceUpdates.parameterNames
-          if not common.endsWith(p[0], '.')
-            task.session.parameterNames.push(p[0])
-
-      if status & STATUS_COMPLETED
-        task.session.subtask = {device : task.device, name : 'getParameterValues', parameterNames : task.session.parameterNames}
-        delete task.session.parameterNames
-
-        this.getParameterValues(task.session.subtask, {}, (err, status, methodRequest) ->
-          # ignore deviceUpdates returned by firt call to getParameterValues
-          callback(err, status, methodRequest, deviceUpdates)
-        )
-      else if status & STATUS_OK
-        callback(err, STATUS_OK, methodRequest, deviceUpdates)
-      else
-        throw new Error('Unexpected subtask status')
-    )
-  else if task.session.subtask.name is 'getParameterValues'
-    if methodResponse.faultcode?
-      task.fault = methodResponse
-      return callback(null, STATUS_FAULT)
-
-    this.getParameterValues(task.session.subtask, methodResponse, (err, status, methodRequest, deviceUpdates) ->
-      callback(err, status, methodRequest, deviceUpdates)
-    )
-  else
-    throw Error('Unexpected subtask name')
-
-
-this.getParameterNames = (task, methodResponse, callback) ->
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  GET_PARAMETER_NAMES_DEPTH_THRESHOLD = config.get('GET_PARAMETER_NAMES_DEPTH_THRESHOLD', task.device)
-
-  getParameterDepth = (param) ->
-    return 0 if !param
-    return (param[...-1] + '.').split('.').length - 1
-
-  if methodResponse.type is 'GetParameterNamesResponse'
-    path = task.session.queue.pop()
-
-    # If parameter depth higher than the threshold, nextLevel was set to false
-    if not task.nextLevel? and getParameterDepth(path) < GET_PARAMETER_NAMES_DEPTH_THRESHOLD
-      for p in methodResponse.parameterList
-        task.session.queue.push(p[0]) if p[0][-1..] == '.'
-
-    deviceUpdates = {parameterNames : methodResponse.parameterList}
-
-    # Make sure that for each parameter, all its parents are explicitly included
-    found = {}
-    found[path] = 0 if !!path
-    for p in deviceUpdates.parameterNames
-      param = p[0]
-      i = param.length
-      while (i = param.lastIndexOf('.', i-1)) > path.length
-        pp = param.slice(0, i + 1)
-        break if found[pp]?
-        found[pp] = 0
-      found[p[0]] = 1
-
-    for k, v of found
-      deviceUpdates.parameterNames.push([k]) if v == 0
-  else
-    task.session = {queue : [task.parameterPath]}
-
-  if task.session.queue.length > 0
-    path = task.session.queue[-1..][0]
-    methodRequest = {
-      type : 'GetParameterNames',
-      parameterPath : path,
-      nextLevel : task.nextLevel ? getParameterDepth(path) < GET_PARAMETER_NAMES_DEPTH_THRESHOLD
-    }
-    return callback(null, STATUS_OK, methodRequest, deviceUpdates)
-  else
-    return callback(null, STATUS_COMPLETED, null, deviceUpdates)
-
-
-this.getParameterValues = (task, methodResponse, callback) ->
-  task.session = {} if not task.session?
-
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if not task.session.currentIndex?
-    task.session.currentIndex = 0
-  else if methodResponse.parameterList?
-    task.session.currentIndex = task.session.nextIndex
-
-  task.session.nextIndex = Math.min(task.session.currentIndex + config.get('TASK_PARAMETERS_BATCH_SIZE', task.device), task.parameterNames.length)
-  names = task.parameterNames.slice(task.session.currentIndex, task.session.nextIndex)
-
-  if methodResponse.type is 'GetParameterValuesResponse'
-    deviceUpdates = {parameterValues : methodResponse.parameterList}
-
-  if names.length == 0
-    callback(null, STATUS_COMPLETED, null, deviceUpdates)
-  else
-    methodRequest = {
-      type : 'GetParameterValues',
-      parameterNames : names
-    }
-    callback(null, STATUS_OK, methodRequest, deviceUpdates)
-
-
-this.setParameterValues = (task, methodResponse, callback) ->
-  task.session = {} if not task.session?
-
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if not task.session.currentIndex?
-    task.session.currentIndex = 0
-  else if methodResponse.type is 'SetParameterValuesResponse'
-    prevValues = task.parameterValues.slice(task.session.currentIndex, task.session.nextIndex)
-    task.session.currentIndex = task.session.nextIndex
-
-  task.session.nextIndex = Math.min(task.session.currentIndex + config.get('TASK_PARAMETERS_BATCH_SIZE', task.device), task.parameterValues.length)
-  values = task.parameterValues.slice(task.session.currentIndex, task.session.nextIndex)
-
-  if prevValues?
-    deviceUpdates = {parameterValues : prevValues}
-
-  if values.length == 0
-    callback(null, STATUS_COMPLETED, null, deviceUpdates)
-  else
-    callback(null, STATUS_OK, {type : 'SetParameterValues', parameterList : values}, deviceUpdates)
-
-
-this.addObject = (task, methodResponse, callback) ->
-  task.session = {} if not task.session?
-
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  allDeviceUpdates = {}
-
-  if not task.session.instanceNumber?
-    if methodResponse.type is 'AddObjectResponse'
-      status_save = STATUS_SAVE
-      task.session.instanceNumber = methodResponse.instanceNumber
-      # TODO Don't specify nextLevel explicity. Also consider using refreshObject instead.
-      task.session.subtask = {device : task.device, name : 'getParameterNames', parameterPath : "#{task.objectName}.#{task.session.instanceNumber}.", nextLevel : false}
-      task.session.appliedParameterValues = []
-      task.session.parameterNames = []
-    else
-      callback(null, STATUS_OK, {type : 'AddObject', objectName : "#{task.objectName}."})
-      return
-
-  subtask = () =>
-    switch task.session.subtask.name
-      when 'getParameterNames'
-        this.getParameterNames(task.session.subtask, methodResponse, (err, status, methodRequest, deviceUpdates) =>
-          common.extend(allDeviceUpdates, deviceUpdates)
-          if deviceUpdates and deviceUpdates.parameterNames
-            for p in deviceUpdates.parameterNames
-              task.session.parameterNames.push(p[0]) if not common.endsWith(p[0], '.')
-
-          if status & STATUS_COMPLETED
-            task.session.subtask = {device : task.device, name : 'getParameterValues', parameterNames : task.session.parameterNames}
-            subtask()
-          else if status & STATUS_OK
-            # Use STATUS_SAVE to avoid adding duplicate object in case of error
-            callback(err, STATUS_OK | status_save, methodRequest, allDeviceUpdates)
-          else
-            throw Error('Unexpected subtask status')
-        )
-      when 'getParameterValues'
-        this.getParameterValues(task.session.subtask, methodResponse, (err, status, methodRequest, deviceUpdates) =>
-          common.extend(allDeviceUpdates, deviceUpdates)
-          # if values are given, compare with default values
-          if task.parameterValues?
-            if deviceUpdates and deviceUpdates.parameterValues?
-              for p1 in deviceUpdates.parameterValues
-                for p2 in task.parameterValues
-                  if p1[0] == "#{task.objectName}.#{task.session.instanceNumber}.#{p2[0]}"
-                    t = if p2[2] then p2[2] else p1[2]
-                    v = common.matchType(p1[1], p2[1])
-                    # TODO only include if writable
-                    task.session.appliedParameterValues.push([p1[0], v, t])
-
-          if methodResponse.faultcode?
-            task.fault = methodResponse
-            return callback(null, STATUS_FAULT)
-
-          if status & STATUS_COMPLETED and task.session.appliedParameterValues.length > 0
-            task.session.subtask = {device : task.device, name : 'setParameterValues', parameterValues : task.session.appliedParameterValues}
-            subtask()
-          else
-            callback(err, status, methodRequest, allDeviceUpdates)
-        )
-      when 'setParameterValues'
-        this.setParameterValues(task.session.subtask, methodResponse, (err, status, methodRequest, deviceUpdates) =>
-          common.extend(allDeviceUpdates, deviceUpdates)
-          callback(err, status, methodRequest, allDeviceUpdates)
-        )
-  subtask()
-
-
-this.deleteObject = (task, methodResponse, callback) ->
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if methodResponse.type is 'DeleteObjectResponse'
-    callback(null, STATUS_COMPLETED, null, {deletedObjects : [task.objectName]})
-  else
-    methodRequest = {
-      type : 'DeleteObject',
-      objectName : "#{task.objectName}."
-    }
-    callback(null, STATUS_OK, methodRequest)
-
-
-this.reboot = (task, methodResponse, callback) ->
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if methodResponse.type isnt 'RebootResponse'
-    callback(null, STATUS_OK, {type : 'Reboot'})
-  else
-    callback(null, STATUS_COMPLETED)
-
-
-this.factoryReset = (task, methodResponse, callback) ->
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if methodResponse.type isnt 'FactoryResetResponse'
-    callback(null, STATUS_OK, {type : 'FactoryReset'})
-  else
-    callback(null, STATUS_COMPLETED)
-
-
-this.download = (task, methodResponse, callback) ->
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if methodResponse.type isnt 'DownloadResponse'
-    db.filesCollection.findOne({_id : mongodb.ObjectID(String(task.file))}, (err, file) ->
-      if not file?
-        callback('File not found')
-        return
-      else if err?
-        callback(err)
-        return
-
-      l = {
-        protocol : if config.get('FS_SSL') then 'https' else 'http',
-        hostname : config.get('FS_IP'),
-        port : config.get('FS_PORT'),
-        pathname : encodeURIComponent(file.filename)
-      }
-
-      methodRequest = {
-        type : 'Download',
-        fileType : file.metadata.fileType,
-        fileSize : file.length,
-        url : url.format(l),
-        successUrl : task.successUrl,
-        failureUrl : task.failureUrl
-      }
-      callback(null, STATUS_OK, methodRequest)
-    )
-  else
-    callback(null, STATUS_COMPLETED)
-
-
-this.customCommand = (task, methodResponse, callback) ->
-  # TODO implement timeout
-  customCommands.execute(task.device, task.command, (err, value) ->
-    if err
-      task.fault = {name : err.name, message : err.message}
-      callback(null, STATUS_FAULT)
-    else
-      commandName = task.command.split(' ', 2)[0]
-      callback(null, STATUS_COMPLETED, null, {customCommands : [[commandName, value]]})
-  )
-
-
-exports.task = (task, methodResponse, callback) ->
-  this[task.name](task, methodResponse, callback)

lib/update-device.coffee

@@ -1,217 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-###
-
-common = require './common'
-db = require './db'
-normalize = require('./normalize').normalize
-mongoQuery = require './mongo-query'
-parameters = require './parameters'
-
-OBJECT_REGEX = /\.$/
-INSTANCE_REGEX = /\.[\d]+\.$/
-
-
-queueUpdates = (deviceId, deviceUpdates, callback) ->
-  return callback() if not deviceUpdates? or Object.keys(deviceUpdates).length == 0
-  # TODO consider setting expiry
-  db.redisClient.rpush("#{deviceId}_updates", JSON.stringify(deviceUpdates), (err, res) ->
-    return callback(err)
-  )
-
-
-clearUpdatesQueue = (deviceId, callback) ->
-  db.redisClient.del("#{deviceId}_updates", (err, res) ->
-    return callback(err)
-  )
-
-
-commitUpdates = (deviceId, deviceUpdates, commitQueue, callback) ->
-  now = new Date(Date.now())
-  update = {'$set' : {}, '$unset' : {}}
-
-  # Used to find parameters that no longer exist in device
-  presentPaths = null
-  projection = null
-
-  # Keep track of alias changes to clear cache if needed
-  newAliases = null
-
-  processBatch = (updatesBatch, cb) ->
-    if updatesBatch.informEvents?
-      update['$set']['_lastInform'] = now
-      update['$set']['_lastBoot'] = now if '1 BOOT' in updatesBatch.informEvents
-      update['$set']['_lastBootstrap'] = now if '0 BOOTSTRAP' in updatesBatch.informEvents
-
-    if updatesBatch.parameterValues?
-      for p in updatesBatch.parameterValues
-        origValue = p[1]
-        v = normalize(p[0], origValue)
-
-        path = if common.endsWith(p[0], '.') then p[0] else "#{p[0]}."
-        if v == origValue
-          update['$unset']["#{path}_orig"] = 1
-        else
-          update['$set']["#{path}_orig"] = origValue
-        update['$set']["#{path}_value"] = v
-        update['$set']["#{path}_timestamp"] = now
-        if p[2]?
-          update['$set']["#{path}_type"] = p[2]
-
-    if updatesBatch.deletedObjects?
-      for p in updatesBatch.deletedObjects
-        update['$unset'][p] = 1
-
-    if updatesBatch.parameterNames?
-      presentPaths ?= {}
-      projection ?= {}
-
-      for p in updatesBatch.parameterNames
-        if OBJECT_REGEX.test(p[0])
-          path = p[0].slice(0, -1)
-          if INSTANCE_REGEX.test(p[0])
-            update['$set']["#{path}._instance"] = true
-          else
-            update['$set']["#{path}._object"] = true
-        else
-          path = p[0]
-        update['$set']["#{path}._writable"] = p[1] if p[1]?
-        update['$set']["#{path}._timestamp"] = now
-
-        # Add parameter (sans any dot suffix) if has an alias
-        if (a = parameters.getAlias(path))?
-          newAliases ?= {}
-          newAliases[path] = a
-
-        projection[path] = 1
-
-        # Store this param path and all its parents for later look up
-        presentPaths[path] = true
-        i = p[0].indexOf('.')
-        while i != -1
-          presentPaths[p[0][0...i]] = true
-          i = p[0].indexOf('.', i + 1)
-
-    if updatesBatch.customCommands?
-      for p in updatesBatch.customCommands
-        commandName = p[0]
-        update['$set']["_customCommands.#{commandName}._value"] = p[1]
-        update['$set']["_customCommands.#{commandName}._timestamp"] = now
-
-  processBatches = (cb) ->
-    if not commitQueue
-      processBatch(deviceUpdates) if deviceUpdates?
-      return cb()
-
-    db.redisClient.lpop("#{deviceId}_updates", (err, res) ->
-      callback(err) if err
-
-      if not res?
-        processBatch(deviceUpdates) if deviceUpdates?
-        return cb()
-
-      processBatch(JSON.parse(res))
-      return processBatches(cb)
-    )
-
-  commit = () ->
-    for k of update
-      delete update[k] if Object.keys(update[k]).length == 0
-
-    if Object.keys(update).length
-      db.devicesCollection.update({'_id' : deviceId}, update, {}, (err, count) ->
-        return callback(err) if err
-
-        # Clear aliases cache if there's a new aliased parameter that has not been included in aliases cache
-        if newAliases?
-          newAliasFound = false
-          db.getAliases((aliases) ->
-            for p, a of newAliases
-              if not aliases[a]? or p not in aliases[a]
-                newAliasFound = true
-                break
-
-            return callback() if not newAliasFound
-            db.redisClient.del('aliases', (err, res) ->
-              return callback(err)
-            )
-          )
-        else
-          return callback()
-      )
-    else
-      return callback()
-
-  processBatches(() ->
-    if projection?
-      mongoQuery.optimizeProjection(projection)
-
-      # Fetch device from DB to discover and delete parameters that no longer exist
-      # While we're fetching the object, we might as well use the information
-      # to optimize our update query
-      db.devicesCollection.findOne({'_id' : deviceId}, projection, (err, device) ->
-        return callback(err) if err
-
-        recursive = (obj, prefix) ->
-          for k, v of obj
-            p = prefix + k
-
-            # Mark unset operations we want to keep
-            if update['$unset'][p]?
-              update['$unset'][p] = 2
-              continue
-
-            if k[0] == '_'
-              # Remove set operations for unmodified values for optimization
-              if update['$set'][p] == v
-                delete update['$set'][p]
-              continue
-
-            if presentPaths[p]
-              recursive(v, "#{p}.")
-            else
-              update['$unset'][p] = 2
-
-        # Scan the DB object and remove parameters that no longer exists in device
-        for path of projection
-          root = device
-          rootPath = ''
-          for p in path.split('.')
-            rootPath += "#{p}."
-            root = root[p]
-            break if not root?
-
-          if root
-            recursive(root, rootPath)
-          else
-            if not presentPaths[rootPath.slice(0, -1)]?
-              update['$unset'][rootPath.slice(0, -1)] = 2
-
-          # Remove unnecessary unset updates
-          for k, v of update['$unset']
-            if common.startsWith(k, path + '.') and v != 2
-              delete update['$unset'][k]
-
-        commit()
-      )
-    else
-      commit()
-  )
-
-
-exports.queueUpdates = queueUpdates
-exports.clearUpdatesQueue = clearUpdatesQueue
-exports.commitUpdates = commitUpdates

lib/versioned-map.js

@@ -0,0 +1,267 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const NONEXISTENT = new Object();
+
+
+function compareEquality(a, b) {
+  var t = typeof a;
+  if (t === 'number' || (a == null) || t === 'boolean' || t === 'string' || t === 'symbol')
+    return a === b;
+
+  return JSON.stringify(a) === JSON.stringify(b);
+}
+
+
+class VersionedMap {
+
+  constructor() {
+    this._sizeDiff = [0];
+    this._revision = 0;
+    this.map = new Map();
+    this.dirty = 0;
+  }
+
+
+  get size() {
+    return this.map.size + this._sizeDiff[this.revision];
+  }
+
+
+  get revision() {
+    return this._revision;
+  }
+
+
+  set revision(rev) {
+    for (let i = this._sizeDiff.length; i <= rev; ++ i)
+      this._sizeDiff[i] = this._sizeDiff[i - 1];
+
+    this._revision = rev;
+  }
+
+
+  get(key, rev) {
+    if (rev == null)
+      rev = this._revision
+
+    var v = this.map.get(key);
+    if (!v)
+      return;
+
+    for (let i = Math.min(v.length - 1, rev); i >= 0; -- i)
+      if (i in v)
+        if (v[i] === NONEXISTENT)
+          return undefined;
+        else
+          return v[i];
+  }
+
+
+  has(key, rev) {
+    if (rev == null)
+      rev = this._revision
+
+    var v = this.map.get(key);
+
+    if (v == null)
+      return false;
+
+    for (let i = Math.min(v.length - 1, rev); i >= 0; -- i)
+      if (i in v)
+        if (v[i] === NONEXISTENT)
+          return false;
+        else
+          return true;
+
+    return false;
+  }
+
+
+  set(key, value, rev) {
+    if (rev == null)
+      rev = this._revision;
+
+    var v = this.map.get(key);
+    if (!v) {
+      this.dirty |= 1 << rev;
+      for (let i = 0; i < rev; ++ i)
+        this._sizeDiff[i] -= 1;
+
+      v = [];
+      v[rev] = value;
+      this.map.set(key, v);
+      return this;
+    }
+
+    var old = v[Math.min(rev, v.length - 1)];
+
+    if (rev < v.length - 1) {
+      if (compareEquality(value, old))
+        return
+      else
+        throw new Error('Cannot modify old revisions');
+    }
+
+    this.dirty |= 1 << rev;
+    if (old === NONEXISTENT)
+      ++ this._sizeDiff[rev];
+
+    v[rev] = value;
+  }
+
+
+  delete(key, rev) {
+    if (rev == null)
+      rev = this._revision;
+
+    var v = this.map.get(key);
+    if (!v)
+      return false;
+
+    if (rev < v.length - 1)
+      throw new Error('Cannot modify old revisions');
+
+    var old = v[v.length - 1];
+    if (old === NONEXISTENT)
+      return false;
+
+    this.dirty |= 1 << rev;
+    -- this._sizeDiff[rev];
+    v[rev] = NONEXISTENT;
+
+    return true;
+  }
+
+
+  getRevisions(key) {
+    var v = this.map.get(key);
+    if (!v)
+      return null;
+
+    var res = {};
+
+    for (let i in v)
+      if (v[i] === NONEXISTENT)
+        res.delete |= 1 << i;
+      else
+        res[i] = v[i];
+
+    return res;
+  }
+
+
+  setRevisions(key, revisions) {
+    var del = 0;
+    var rev = [];
+    var minKey = 999;
+    for (let k in revisions) {
+      if (k === 'delete')
+        del = revisions[k];
+      else {
+        let r = parseInt(k);
+        minKey = Math.min(minKey, r);
+        for (let i = this._sizeDiff.length; i <= r; ++ i)
+          this._sizeDiff[i] = this._sizeDiff[i - 1];
+
+        this.dirty |= 1 << r;
+        rev[r] = revisions[k];
+      }
+    }
+
+    for (let i = 0; i < minKey; ++ i)
+      -- this._sizeDiff[i];
+
+    this.dirty |= del;
+
+    for (let i = 0; del > 0; del >>= 1, ++ i)
+      if (del & 1)
+        rev[i] = NONEXISTENT;
+
+    this.map.set(key, rev);
+  }
+
+
+  getDiff(key) {
+    var revisions = this.map.get(key);
+    if (!revisions)
+      return [];
+
+    var current = revisions[revisions.length - 1];
+    if (current === NONEXISTENT) {
+      if (0 in revisions)
+        return [key, revisions[0]];
+      else
+        return [key];
+    } else if (0 in revisions)
+      return [key, revisions[0], current];
+    else
+      return [key, , current];
+  }
+
+
+  *diff() {
+    for (let pair of this.map) {
+      let current = pair[1][pair[1].length - 1];
+      if (current === NONEXISTENT) {
+        if (0 in pair[1])
+          yield [pair[0], pair[1][0]];
+        else
+          yield [pair[0]];
+      } else if (0 in pair[1])
+        yield [pair[0], pair[1][0], current];
+      else
+        yield [pair[0], , current];
+    }
+  }
+
+
+  collapse(revision) {
+    if (this._sizeDiff.length <= revision)
+      return;
+
+    this._sizeDiff[revision] = this._sizeDiff[this._sizeDiff.length - 1];
+    this._sizeDiff.splice(revision + 1, this._sizeDiff.length);
+
+    let d = this.dirty >> revision;
+    this.dirty = this.dirty ^ (d << revision);
+    this.dirty |= (+!!d) << revision;
+
+    for (let pair of this.map) {
+      let k = pair[0];
+      let v = pair[1];
+      let last = v[v.length - 1];
+      if (last === NONEXISTENT) {
+        v.splice(revision, v.length);
+        if (v.some(function(val) { return true; }))
+          v[revision] = NONEXISTENT;
+        else
+          this.map.delete(k);
+      }
+      else {
+        v[revision] = last;
+        v.splice(revision + 1, v.length);
+      }
+    }
+  }
+
+}
+
+module.exports = VersionedMap;

npm-shrinkwrap.json

@@ -0,0 +1,926 @@
+{
+  "name": "genieacs",
+  "version": "1.1.2",
+  "lockfileVersion": 1,
+  "requires": true,
+  "dependencies": {
+    "abbrev": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/abbrev/-/abbrev-1.1.1.tgz",
+      "integrity": "sha512-nne9/IiQ/hzIhY6pdDnbBtz7DjPTKrY00P/zvPSm5pOFkl6xuGrGnXn/VtTNNfNtAfZ9/1RtehkszU9qcTii0Q=="
+    },
+    "ajv": {
+      "version": "4.11.8",
+      "resolved": "https://registry.npmjs.org/ajv/-/ajv-4.11.8.tgz",
+      "integrity": "sha1-gv+wKynmYq5TvcIK8VlHcGc5xTY=",
+      "requires": {
+        "co": "4.6.0",
+        "json-stable-stringify": "1.0.1"
+      }
+    },
+    "ansi-regex": {
+      "version": "2.1.1",
+      "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-2.1.1.tgz",
+      "integrity": "sha1-w7M6te42DYbg5ijwRorn7yfWVN8="
+    },
+    "aproba": {
+      "version": "1.2.0",
+      "resolved": "https://registry.npmjs.org/aproba/-/aproba-1.2.0.tgz",
+      "integrity": "sha512-Y9J6ZjXtoYh8RnXVCMOU/ttDmk1aBjunq9vO0ta5x85WDQiQfUF9sIPBITdbiiIVcBo03Hi3jMxigBtsddlXRw=="
+    },
+    "are-we-there-yet": {
+      "version": "1.1.4",
+      "resolved": "https://registry.npmjs.org/are-we-there-yet/-/are-we-there-yet-1.1.4.tgz",
+      "integrity": "sha1-u13KOCu5TwXhUZQ3PRb9O6HKEQ0=",
+      "requires": {
+        "delegates": "1.0.0",
+        "readable-stream": "2.3.4"
+      }
+    },
+    "asn1": {
+      "version": "0.2.3",
+      "resolved": "https://registry.npmjs.org/asn1/-/asn1-0.2.3.tgz",
+      "integrity": "sha1-2sh4dxPJlmhJ/IGAd36+nB3fO4Y="
+    },
+    "assert-plus": {
+      "version": "0.2.0",
+      "resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-0.2.0.tgz",
+      "integrity": "sha1-104bh+ev/A24qttwIfP+SBAasjQ="
+    },
+    "asynckit": {
+      "version": "0.4.0",
+      "resolved": "https://registry.npmjs.org/asynckit/-/asynckit-0.4.0.tgz",
+      "integrity": "sha1-x57Zf380y48robyXkLzDZkdLS3k="
+    },
+    "aws-sign2": {
+      "version": "0.6.0",
+      "resolved": "https://registry.npmjs.org/aws-sign2/-/aws-sign2-0.6.0.tgz",
+      "integrity": "sha1-FDQt0428yU0OW4fXY81jYSwOeU8="
+    },
+    "aws4": {
+      "version": "1.6.0",
+      "resolved": "https://registry.npmjs.org/aws4/-/aws4-1.6.0.tgz",
+      "integrity": "sha1-g+9cqGCysy5KDe7e6MdxudtXRx4="
+    },
+    "balanced-match": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.0.tgz",
+      "integrity": "sha1-ibTRmasr7kneFk6gK4nORi1xt2c="
+    },
+    "bcrypt-pbkdf": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/bcrypt-pbkdf/-/bcrypt-pbkdf-1.0.1.tgz",
+      "integrity": "sha1-Y7xdy2EzG5K8Bf1SiVPDNGKgb40=",
+      "optional": true,
+      "requires": {
+        "tweetnacl": "0.14.5"
+      }
+    },
+    "bindings": {
+      "version": "1.2.1",
+      "resolved": "https://registry.npmjs.org/bindings/-/bindings-1.2.1.tgz",
+      "integrity": "sha1-FK1hE4EtLTfXLme0ystLtyZQXxE="
+    },
+    "block-stream": {
+      "version": "0.0.9",
+      "resolved": "https://registry.npmjs.org/block-stream/-/block-stream-0.0.9.tgz",
+      "integrity": "sha1-E+v+d4oDIFz+A3UUgeu0szAMEmo=",
+      "requires": {
+        "inherits": "2.0.3"
+      }
+    },
+    "boom": {
+      "version": "2.10.1",
+      "resolved": "https://registry.npmjs.org/boom/-/boom-2.10.1.tgz",
+      "integrity": "sha1-OciRjO/1eZ+D+UkqhI9iWt0Mdm8=",
+      "requires": {
+        "hoek": "2.16.3"
+      }
+    },
+    "brace-expansion": {
+      "version": "1.1.11",
+      "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.11.tgz",
+      "integrity": "sha512-iCuPHDFgrHX7H2vEI/5xpz07zSHB00TpugqhmYtVmMO6518mCuRMoOYFldEBl0g187ufozdaHgWKcYFb61qGiA==",
+      "requires": {
+        "balanced-match": "1.0.0",
+        "concat-map": "0.0.1"
+      }
+    },
+    "bson": {
+      "version": "1.0.4",
+      "resolved": "https://registry.npmjs.org/bson/-/bson-1.0.4.tgz",
+      "integrity": "sha1-k8ENOeqltYQVy8QFLz5T5WKwtyw="
+    },
+    "buffer-shims": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/buffer-shims/-/buffer-shims-1.0.0.tgz",
+      "integrity": "sha1-mXjOMXOIxkmth5MCjDR37wRKi1E="
+    },
+    "caseless": {
+      "version": "0.12.0",
+      "resolved": "https://registry.npmjs.org/caseless/-/caseless-0.12.0.tgz",
+      "integrity": "sha1-G2gcIf+EAzyCZUMJBolCDRhxUdw="
+    },
+    "co": {
+      "version": "4.6.0",
+      "resolved": "https://registry.npmjs.org/co/-/co-4.6.0.tgz",
+      "integrity": "sha1-bqa989hTrlTMuOR7+gvz+QMfsYQ="
+    },
+    "code-point-at": {
+      "version": "1.1.0",
+      "resolved": "https://registry.npmjs.org/code-point-at/-/code-point-at-1.1.0.tgz",
+      "integrity": "sha1-DQcLTQQ6W+ozovGkDi7bPZpMz3c="
+    },
+    "combined-stream": {
+      "version": "1.0.6",
+      "resolved": "https://registry.npmjs.org/combined-stream/-/combined-stream-1.0.6.tgz",
+      "integrity": "sha1-cj599ugBrFYTETp+RFqbactjKBg=",
+      "requires": {
+        "delayed-stream": "1.0.0"
+      }
+    },
+    "concat-map": {
+      "version": "0.0.1",
+      "resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz",
+      "integrity": "sha1-2Klr13/Wjfd5OnMDajug1UBdR3s="
+    },
+    "console-control-strings": {
+      "version": "1.1.0",
+      "resolved": "https://registry.npmjs.org/console-control-strings/-/console-control-strings-1.1.0.tgz",
+      "integrity": "sha1-PXz0Rk22RG6mRL9LOVB/mFEAjo4="
+    },
+    "core-util-is": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.2.tgz",
+      "integrity": "sha1-tf1UIgqivFq1eqtxQMlAdUUDwac="
+    },
+    "cryptiles": {
+      "version": "2.0.5",
+      "resolved": "https://registry.npmjs.org/cryptiles/-/cryptiles-2.0.5.tgz",
+      "integrity": "sha1-O9/s3GCBR8HGcgL6KR59ylnqo7g=",
+      "requires": {
+        "boom": "2.10.1"
+      }
+    },
+    "dashdash": {
+      "version": "1.14.1",
+      "resolved": "https://registry.npmjs.org/dashdash/-/dashdash-1.14.1.tgz",
+      "integrity": "sha1-hTz6D3y+L+1d4gMmuN1YEDX24vA=",
+      "requires": {
+        "assert-plus": "1.0.0"
+      },
+      "dependencies": {
+        "assert-plus": {
+          "version": "1.0.0",
+          "resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-1.0.0.tgz",
+          "integrity": "sha1-8S4PPF13sLHN2RRpQuTpbB5N1SU="
+        }
+      }
+    },
+    "debug": {
+      "version": "2.6.9",
+      "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
+      "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
+      "requires": {
+        "ms": "2.0.0"
+      }
+    },
+    "deep-extend": {
+      "version": "0.4.2",
+      "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.4.2.tgz",
+      "integrity": "sha1-SLaZwn4zS/ifEIkr5DL25MfTSn8="
+    },
+    "delayed-stream": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz",
+      "integrity": "sha1-3zrhmayt+31ECqrgsp4icrJOxhk="
+    },
+    "delegates": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/delegates/-/delegates-1.0.0.tgz",
+      "integrity": "sha1-hMbhWbgZBP3KWaDvRM2HDTElD5o="
+    },
+    "detect-libc": {
+      "version": "1.0.3",
+      "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-1.0.3.tgz",
+      "integrity": "sha1-+hN8S9aY7fVc1c0CrFWfkaTEups="
+    },
+    "ecc-jsbn": {
+      "version": "0.1.1",
+      "resolved": "https://registry.npmjs.org/ecc-jsbn/-/ecc-jsbn-0.1.1.tgz",
+      "integrity": "sha1-D8c6ntXw1Tw4GTOYUj735UN3dQU=",
+      "optional": true,
+      "requires": {
+        "jsbn": "0.1.1"
+      }
+    },
+    "es6-promise": {
+      "version": "3.2.1",
+      "resolved": "https://registry.npmjs.org/es6-promise/-/es6-promise-3.2.1.tgz",
+      "integrity": "sha1-7FYjOGgDKQkgcXDDlEjiREndH8Q="
+    },
+    "extend": {
+      "version": "3.0.1",
+      "resolved": "https://registry.npmjs.org/extend/-/extend-3.0.1.tgz",
+      "integrity": "sha1-p1Xqe8Gt/MWjHOfnYtuq3F5jZEQ="
+    },
+    "extsprintf": {
+      "version": "1.3.0",
+      "resolved": "https://registry.npmjs.org/extsprintf/-/extsprintf-1.3.0.tgz",
+      "integrity": "sha1-lpGEQOMEGnpBT4xS48V06zw+HgU="
+    },
+    "forever-agent": {
+      "version": "0.6.1",
+      "resolved": "https://registry.npmjs.org/forever-agent/-/forever-agent-0.6.1.tgz",
+      "integrity": "sha1-+8cfDEGt6zf5bFd60e1C2P2sypE="
+    },
+    "form-data": {
+      "version": "2.1.4",
+      "resolved": "https://registry.npmjs.org/form-data/-/form-data-2.1.4.tgz",
+      "integrity": "sha1-M8GDrPGTJ27KqYFDpp6Uv+4XUNE=",
+      "requires": {
+        "asynckit": "0.4.0",
+        "combined-stream": "1.0.6",
+        "mime-types": "2.1.18"
+      }
+    },
+    "fs.realpath": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/fs.realpath/-/fs.realpath-1.0.0.tgz",
+      "integrity": "sha1-FQStJSMVjKpA20onh8sBQRmU6k8="
+    },
+    "fstream": {
+      "version": "1.0.11",
+      "resolved": "https://registry.npmjs.org/fstream/-/fstream-1.0.11.tgz",
+      "integrity": "sha1-XB+x8RdHcRTwYyoOtLcbPLD9MXE=",
+      "requires": {
+        "graceful-fs": "4.1.11",
+        "inherits": "2.0.3",
+        "mkdirp": "0.5.1",
+        "rimraf": "2.6.2"
+      }
+    },
+    "fstream-ignore": {
+      "version": "1.0.5",
+      "resolved": "https://registry.npmjs.org/fstream-ignore/-/fstream-ignore-1.0.5.tgz",
+      "integrity": "sha1-nDHa40dnAY/h0kmyTa2mfQktoQU=",
+      "requires": {
+        "fstream": "1.0.11",
+        "inherits": "2.0.3",
+        "minimatch": "3.0.4"
+      }
+    },
+    "gauge": {
+      "version": "2.7.4",
+      "resolved": "https://registry.npmjs.org/gauge/-/gauge-2.7.4.tgz",
+      "integrity": "sha1-LANAXHU4w51+s3sxcCLjJfsBi/c=",
+      "requires": {
+        "aproba": "1.2.0",
+        "console-control-strings": "1.1.0",
+        "has-unicode": "2.0.1",
+        "object-assign": "4.1.1",
+        "signal-exit": "3.0.2",
+        "string-width": "1.0.2",
+        "strip-ansi": "3.0.1",
+        "wide-align": "1.1.2"
+      }
+    },
+    "getpass": {
+      "version": "0.1.7",
+      "resolved": "https://registry.npmjs.org/getpass/-/getpass-0.1.7.tgz",
+      "integrity": "sha1-Xv+OPmhNVprkyysSgmBOi6YhSfo=",
+      "requires": {
+        "assert-plus": "1.0.0"
+      },
+      "dependencies": {
+        "assert-plus": {
+          "version": "1.0.0",
+          "resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-1.0.0.tgz",
+          "integrity": "sha1-8S4PPF13sLHN2RRpQuTpbB5N1SU="
+        }
+      }
+    },
+    "glob": {
+      "version": "7.1.2",
+      "resolved": "https://registry.npmjs.org/glob/-/glob-7.1.2.tgz",
+      "integrity": "sha512-MJTUg1kjuLeQCJ+ccE4Vpa6kKVXkPYJ2mOCQyUuKLcLQsdrMCpBPUi8qVE6+YuaJkozeA9NusTAw3hLr8Xe5EQ==",
+      "requires": {
+        "fs.realpath": "1.0.0",
+        "inflight": "1.0.6",
+        "inherits": "2.0.3",
+        "minimatch": "3.0.4",
+        "once": "1.4.0",
+        "path-is-absolute": "1.0.1"
+      }
+    },
+    "graceful-fs": {
+      "version": "4.1.11",
+      "resolved": "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.1.11.tgz",
+      "integrity": "sha1-Dovf5NHduIVNZOBOp8AOKgJuVlg="
+    },
+    "har-schema": {
+      "version": "1.0.5",
+      "resolved": "https://registry.npmjs.org/har-schema/-/har-schema-1.0.5.tgz",
+      "integrity": "sha1-0mMTX0MwfALGAq/I/pWXDAFRNp4="
+    },
+    "har-validator": {
+      "version": "4.2.1",
+      "resolved": "https://registry.npmjs.org/har-validator/-/har-validator-4.2.1.tgz",
+      "integrity": "sha1-M0gdDxu/9gDdID11gSpqX7oALio=",
+      "requires": {
+        "ajv": "4.11.8",
+        "har-schema": "1.0.5"
+      }
+    },
+    "has-unicode": {
+      "version": "2.0.1",
+      "resolved": "https://registry.npmjs.org/has-unicode/-/has-unicode-2.0.1.tgz",
+      "integrity": "sha1-4Ob+aijPUROIVeCG0Wkedx3iqLk="
+    },
+    "hawk": {
+      "version": "3.1.3",
+      "resolved": "https://registry.npmjs.org/hawk/-/hawk-3.1.3.tgz",
+      "integrity": "sha1-B4REvXwWQLD+VA0sm3PVlnjo4cQ=",
+      "requires": {
+        "boom": "2.10.1",
+        "cryptiles": "2.0.5",
+        "hoek": "2.16.3",
+        "sntp": "1.0.9"
+      }
+    },
+    "hoek": {
+      "version": "2.16.3",
+      "resolved": "https://registry.npmjs.org/hoek/-/hoek-2.16.3.tgz",
+      "integrity": "sha1-ILt0A9POo5jpHcRxCo/xuCdKJe0="
+    },
+    "http-signature": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/http-signature/-/http-signature-1.1.1.tgz",
+      "integrity": "sha1-33LiZwZs0Kxn+3at+OE0qPvPkb8=",
+      "requires": {
+        "assert-plus": "0.2.0",
+        "jsprim": "1.4.1",
+        "sshpk": "1.13.1"
+      }
+    },
+    "inflight": {
+      "version": "1.0.6",
+      "resolved": "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz",
+      "integrity": "sha1-Sb1jMdfQLQwJvJEKEHW6gWW1bfk=",
+      "requires": {
+        "once": "1.4.0",
+        "wrappy": "1.0.2"
+      }
+    },
+    "inherits": {
+      "version": "2.0.3",
+      "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.3.tgz",
+      "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4="
+    },
+    "ini": {
+      "version": "1.3.5",
+      "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.5.tgz",
+      "integrity": "sha512-RZY5huIKCMRWDUqZlEi72f/lmXKMvuszcMBduliQ3nnWbx9X/ZBQO7DijMEYS9EhHBb2qacRUMtC7svLwe0lcw=="
+    },
+    "is-fullwidth-code-point": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-1.0.0.tgz",
+      "integrity": "sha1-754xOG8DGn8NZDr4L95QxFfvAMs=",
+      "requires": {
+        "number-is-nan": "1.0.1"
+      }
+    },
+    "is-typedarray": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/is-typedarray/-/is-typedarray-1.0.0.tgz",
+      "integrity": "sha1-5HnICFjfDBsR3dppQPlgEfzaSpo="
+    },
+    "isarray": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/isarray/-/isarray-1.0.0.tgz",
+      "integrity": "sha1-u5NdSFgsuhaMBoNJV6VKPgcSTxE="
+    },
+    "isstream": {
+      "version": "0.1.2",
+      "resolved": "https://registry.npmjs.org/isstream/-/isstream-0.1.2.tgz",
+      "integrity": "sha1-R+Y/evVa+m+S4VAOaQ64uFKcCZo="
+    },
+    "jsbn": {
+      "version": "0.1.1",
+      "resolved": "https://registry.npmjs.org/jsbn/-/jsbn-0.1.1.tgz",
+      "integrity": "sha1-peZUwuWi3rXyAdls77yoDA7y9RM=",
+      "optional": true
+    },
+    "json-schema": {
+      "version": "0.2.3",
+      "resolved": "https://registry.npmjs.org/json-schema/-/json-schema-0.2.3.tgz",
+      "integrity": "sha1-tIDIkuWaLwWVTOcnvT8qTogvnhM="
+    },
+    "json-stable-stringify": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/json-stable-stringify/-/json-stable-stringify-1.0.1.tgz",
+      "integrity": "sha1-mnWdOcXy/1A/1TAGRu1EX4jE+a8=",
+      "requires": {
+        "jsonify": "0.0.0"
+      }
+    },
+    "json-stringify-safe": {
+      "version": "5.0.1",
+      "resolved": "https://registry.npmjs.org/json-stringify-safe/-/json-stringify-safe-5.0.1.tgz",
+      "integrity": "sha1-Epai1Y/UXxmg9s4B1lcB4sc1tus="
+    },
+    "jsonify": {
+      "version": "0.0.0",
+      "resolved": "https://registry.npmjs.org/jsonify/-/jsonify-0.0.0.tgz",
+      "integrity": "sha1-LHS27kHZPKUbe1qu6PUDYx0lKnM="
+    },
+    "jsprim": {
+      "version": "1.4.1",
+      "resolved": "https://registry.npmjs.org/jsprim/-/jsprim-1.4.1.tgz",
+      "integrity": "sha1-MT5mvB5cwG5Di8G3SZwuXFastqI=",
+      "requires": {
+        "assert-plus": "1.0.0",
+        "extsprintf": "1.3.0",
+        "json-schema": "0.2.3",
+        "verror": "1.10.0"
+      },
+      "dependencies": {
+        "assert-plus": {
+          "version": "1.0.0",
+          "resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-1.0.0.tgz",
+          "integrity": "sha1-8S4PPF13sLHN2RRpQuTpbB5N1SU="
+        }
+      }
+    },
+    "later": {
+      "version": "1.2.0",
+      "resolved": "https://registry.npmjs.org/later/-/later-1.2.0.tgz",
+      "integrity": "sha1-8s9sTdeVbdL1IK3wMpg26YdrrQ8="
+    },
+    "libxmljs": {
+      "version": "0.18.7",
+      "resolved": "https://registry.npmjs.org/libxmljs/-/libxmljs-0.18.7.tgz",
+      "integrity": "sha1-NnPrF8dMuv/e+fM8uD3dgqJQVbc=",
+      "requires": {
+        "bindings": "1.2.1",
+        "nan": "2.5.1",
+        "node-pre-gyp": "0.6.39"
+      }
+    },
+    "mime-db": {
+      "version": "1.33.0",
+      "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.33.0.tgz",
+      "integrity": "sha512-BHJ/EKruNIqJf/QahvxwQZXKygOQ256myeN/Ew+THcAa5q+PjyTTMMeNQC4DZw5AwfvelsUrA6B67NKMqXDbzQ=="
+    },
+    "mime-types": {
+      "version": "2.1.18",
+      "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.18.tgz",
+      "integrity": "sha512-lc/aahn+t4/SWV/qcmumYjymLsWfN3ELhpmVuUFjgsORruuZPVSwAQryq+HHGvO/SI2KVX26bx+En+zhM8g8hQ==",
+      "requires": {
+        "mime-db": "1.33.0"
+      }
+    },
+    "minimatch": {
+      "version": "3.0.4",
+      "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.0.4.tgz",
+      "integrity": "sha512-yJHVQEhyqPLUTgt9B83PXu6W3rx4MvvHvSUvToogpwoGDOUQ+yDrR0HRot+yOCdCO7u4hX3pWft6kWBBcqh0UA==",
+      "requires": {
+        "brace-expansion": "1.1.11"
+      }
+    },
+    "minimist": {
+      "version": "0.0.8",
+      "resolved": "https://registry.npmjs.org/minimist/-/minimist-0.0.8.tgz",
+      "integrity": "sha1-hX/Kv8M5fSYluCKCYuhqp6ARsF0="
+    },
+    "mkdirp": {
+      "version": "0.5.1",
+      "resolved": "https://registry.npmjs.org/mkdirp/-/mkdirp-0.5.1.tgz",
+      "integrity": "sha1-MAV0OOrGz3+MR2fzhkjWaX11yQM=",
+      "requires": {
+        "minimist": "0.0.8"
+      }
+    },
+    "mongodb": {
+      "version": "2.2.34",
+      "resolved": "https://registry.npmjs.org/mongodb/-/mongodb-2.2.34.tgz",
+      "integrity": "sha1-o09Zu+thdUrsQy3nLD/iFSakTBo=",
+      "requires": {
+        "es6-promise": "3.2.1",
+        "mongodb-core": "2.1.18",
+        "readable-stream": "2.2.7"
+      },
+      "dependencies": {
+        "process-nextick-args": {
+          "version": "1.0.7",
+          "resolved": "https://registry.npmjs.org/process-nextick-args/-/process-nextick-args-1.0.7.tgz",
+          "integrity": "sha1-FQ4gt1ZZCtP5EJPyWk8q2L/zC6M="
+        },
+        "readable-stream": {
+          "version": "2.2.7",
+          "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.2.7.tgz",
+          "integrity": "sha1-BwV6y+JGeyIELTb5jFrVBwVOlbE=",
+          "requires": {
+            "buffer-shims": "1.0.0",
+            "core-util-is": "1.0.2",
+            "inherits": "2.0.3",
+            "isarray": "1.0.0",
+            "process-nextick-args": "1.0.7",
+            "string_decoder": "1.0.3",
+            "util-deprecate": "1.0.2"
+          }
+        }
+      }
+    },
+    "mongodb-core": {
+      "version": "2.1.18",
+      "resolved": "https://registry.npmjs.org/mongodb-core/-/mongodb-core-2.1.18.tgz",
+      "integrity": "sha1-TEYTm986HwMt7ZHbSfOO7AFlkFA=",
+      "requires": {
+        "bson": "1.0.4",
+        "require_optional": "1.0.1"
+      }
+    },
+    "ms": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
+      "integrity": "sha1-VgiurfwAvmwpAd9fmGF4jeDVl8g="
+    },
+    "nan": {
+      "version": "2.5.1",
+      "resolved": "https://registry.npmjs.org/nan/-/nan-2.5.1.tgz",
+      "integrity": "sha1-1bAWkSUzJql6K77p5hxV2NYDUeI="
+    },
+    "node-pre-gyp": {
+      "version": "0.6.39",
+      "resolved": "https://registry.npmjs.org/node-pre-gyp/-/node-pre-gyp-0.6.39.tgz",
+      "integrity": "sha512-OsJV74qxnvz/AMGgcfZoDaeDXKD3oY3QVIbBmwszTFkRisTSXbMQyn4UWzUMOtA5SVhrBZOTp0wcoSBgfMfMmQ==",
+      "requires": {
+        "detect-libc": "1.0.3",
+        "hawk": "3.1.3",
+        "mkdirp": "0.5.1",
+        "nopt": "4.0.1",
+        "npmlog": "4.1.2",
+        "rc": "1.2.5",
+        "request": "2.81.0",
+        "rimraf": "2.6.2",
+        "semver": "5.5.0",
+        "tar": "2.2.1",
+        "tar-pack": "3.4.1"
+      }
+    },
+    "nopt": {
+      "version": "4.0.1",
+      "resolved": "https://registry.npmjs.org/nopt/-/nopt-4.0.1.tgz",
+      "integrity": "sha1-0NRoWv1UFRk8jHUFYC0NF81kR00=",
+      "requires": {
+        "abbrev": "1.1.1",
+        "osenv": "0.1.5"
+      }
+    },
+    "npmlog": {
+      "version": "4.1.2",
+      "resolved": "https://registry.npmjs.org/npmlog/-/npmlog-4.1.2.tgz",
+      "integrity": "sha512-2uUqazuKlTaSI/dC8AzicUck7+IrEaOnN/e0jd3Xtt1KcGpwx30v50mL7oPyr/h9bL3E4aZccVwpwP+5W9Vjkg==",
+      "requires": {
+        "are-we-there-yet": "1.1.4",
+        "console-control-strings": "1.1.0",
+        "gauge": "2.7.4",
+        "set-blocking": "2.0.0"
+      }
+    },
+    "number-is-nan": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/number-is-nan/-/number-is-nan-1.0.1.tgz",
+      "integrity": "sha1-CXtgK1NCKlIsGvuHkDGDNpQaAR0="
+    },
+    "oauth-sign": {
+      "version": "0.8.2",
+      "resolved": "https://registry.npmjs.org/oauth-sign/-/oauth-sign-0.8.2.tgz",
+      "integrity": "sha1-Rqarfwrq2N6unsBWV4C31O/rnUM="
+    },
+    "object-assign": {
+      "version": "4.1.1",
+      "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz",
+      "integrity": "sha1-IQmtx5ZYh8/AXLvUQsrIv7s2CGM="
+    },
+    "once": {
+      "version": "1.4.0",
+      "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
+      "integrity": "sha1-WDsap3WWHUsROsF9nFC6753Xa9E=",
+      "requires": {
+        "wrappy": "1.0.2"
+      }
+    },
+    "os-homedir": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/os-homedir/-/os-homedir-1.0.2.tgz",
+      "integrity": "sha1-/7xJiDNuDoM94MFox+8VISGqf7M="
+    },
+    "os-tmpdir": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/os-tmpdir/-/os-tmpdir-1.0.2.tgz",
+      "integrity": "sha1-u+Z0BseaqFxc/sdm/lc0VV36EnQ="
+    },
+    "osenv": {
+      "version": "0.1.5",
+      "resolved": "https://registry.npmjs.org/osenv/-/osenv-0.1.5.tgz",
+      "integrity": "sha512-0CWcCECdMVc2Rw3U5w9ZjqX6ga6ubk1xDVKxtBQPK7wis/0F2r9T6k4ydGYhecl7YUBxBVxhL5oisPsNxAPe2g==",
+      "requires": {
+        "os-homedir": "1.0.2",
+        "os-tmpdir": "1.0.2"
+      }
+    },
+    "path-is-absolute": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz",
+      "integrity": "sha1-F0uSaHNVNP+8es5r9TpanhtcX18="
+    },
+    "performance-now": {
+      "version": "0.2.0",
+      "resolved": "https://registry.npmjs.org/performance-now/-/performance-now-0.2.0.tgz",
+      "integrity": "sha1-M+8wxcd9TqIcWlOGnZG1bY8lVeU="
+    },
+    "process-nextick-args": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/process-nextick-args/-/process-nextick-args-2.0.0.tgz",
+      "integrity": "sha512-MtEC1TqN0EU5nephaJ4rAtThHtC86dNN9qCuEhtshvpVBkAW5ZO7BASN9REnF9eoXGcRub+pFuKEpOHE+HbEMw=="
+    },
+    "punycode": {
+      "version": "1.4.1",
+      "resolved": "https://registry.npmjs.org/punycode/-/punycode-1.4.1.tgz",
+      "integrity": "sha1-wNWmOycYgArY4esPpSachN1BhF4="
+    },
+    "qs": {
+      "version": "6.4.0",
+      "resolved": "https://registry.npmjs.org/qs/-/qs-6.4.0.tgz",
+      "integrity": "sha1-E+JtKK1rD/qpExLNO/cI7TUecjM="
+    },
+    "rc": {
+      "version": "1.2.5",
+      "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.5.tgz",
+      "integrity": "sha1-J1zWh/bjs2zHVrqibf7oCnkDAf0=",
+      "requires": {
+        "deep-extend": "0.4.2",
+        "ini": "1.3.5",
+        "minimist": "1.2.0",
+        "strip-json-comments": "2.0.1"
+      },
+      "dependencies": {
+        "minimist": {
+          "version": "1.2.0",
+          "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
+          "integrity": "sha1-o1AIsg9BOD7sH7kU9M1d95omQoQ="
+        }
+      }
+    },
+    "readable-stream": {
+      "version": "2.3.4",
+      "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.4.tgz",
+      "integrity": "sha512-vuYxeWYM+fde14+rajzqgeohAI7YoJcHE7kXDAc4Nk0EbuKnJfqtY9YtRkLo/tqkuF7MsBQRhPnPeyjYITp3ZQ==",
+      "requires": {
+        "core-util-is": "1.0.2",
+        "inherits": "2.0.3",
+        "isarray": "1.0.0",
+        "process-nextick-args": "2.0.0",
+        "safe-buffer": "5.1.1",
+        "string_decoder": "1.0.3",
+        "util-deprecate": "1.0.2"
+      }
+    },
+    "request": {
+      "version": "2.81.0",
+      "resolved": "https://registry.npmjs.org/request/-/request-2.81.0.tgz",
+      "integrity": "sha1-xpKJRqDgbF+Nb4qTM0af/aRimKA=",
+      "requires": {
+        "aws-sign2": "0.6.0",
+        "aws4": "1.6.0",
+        "caseless": "0.12.0",
+        "combined-stream": "1.0.6",
+        "extend": "3.0.1",
+        "forever-agent": "0.6.1",
+        "form-data": "2.1.4",
+        "har-validator": "4.2.1",
+        "hawk": "3.1.3",
+        "http-signature": "1.1.1",
+        "is-typedarray": "1.0.0",
+        "isstream": "0.1.2",
+        "json-stringify-safe": "5.0.1",
+        "mime-types": "2.1.18",
+        "oauth-sign": "0.8.2",
+        "performance-now": "0.2.0",
+        "qs": "6.4.0",
+        "safe-buffer": "5.1.1",
+        "stringstream": "0.0.5",
+        "tough-cookie": "2.3.3",
+        "tunnel-agent": "0.6.0",
+        "uuid": "3.2.1"
+      }
+    },
+    "require_optional": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/require_optional/-/require_optional-1.0.1.tgz",
+      "integrity": "sha512-qhM/y57enGWHAe3v/NcwML6a3/vfESLe/sGM2dII+gEO0BpKRUkWZow/tyloNqJyN6kXSl3RyyM8Ll5D/sJP8g==",
+      "requires": {
+        "resolve-from": "2.0.0",
+        "semver": "5.5.0"
+      }
+    },
+    "resolve-from": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/resolve-from/-/resolve-from-2.0.0.tgz",
+      "integrity": "sha1-lICrIOlP+h2egKgEx+oUdhGWa1c="
+    },
+    "rimraf": {
+      "version": "2.6.2",
+      "resolved": "https://registry.npmjs.org/rimraf/-/rimraf-2.6.2.tgz",
+      "integrity": "sha512-lreewLK/BlghmxtfH36YYVg1i8IAce4TI7oao75I1g245+6BctqTVQiBP3YUJ9C6DQOXJmkYR9X9fCLtCOJc5w==",
+      "requires": {
+        "glob": "7.1.2"
+      }
+    },
+    "safe-buffer": {
+      "version": "5.1.1",
+      "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.1.tgz",
+      "integrity": "sha512-kKvNJn6Mm93gAczWVJg7wH+wGYWNrDHdWvpUmHyEsgCtIwwo3bqPtV4tR5tuPaUhTOo/kvhVwd8XwwOllGYkbg=="
+    },
+    "seedrandom": {
+      "version": "2.4.3",
+      "resolved": "https://registry.npmjs.org/seedrandom/-/seedrandom-2.4.3.tgz",
+      "integrity": "sha1-JDhQTa0zkXMUv/GKxNeU8W1qrsw="
+    },
+    "semver": {
+      "version": "5.5.0",
+      "resolved": "https://registry.npmjs.org/semver/-/semver-5.5.0.tgz",
+      "integrity": "sha512-4SJ3dm0WAwWy/NVeioZh5AntkdJoWKxHxcmyP622fOkgHa4z3R0TdBJICINyaSDE6uNwVc8gZr+ZinwZAH4xIA=="
+    },
+    "set-blocking": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/set-blocking/-/set-blocking-2.0.0.tgz",
+      "integrity": "sha1-BF+XgtARrppoA93TgrJDkrPYkPc="
+    },
+    "signal-exit": {
+      "version": "3.0.2",
+      "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-3.0.2.tgz",
+      "integrity": "sha1-tf3AjxKH6hF4Yo5BXiUTK3NkbG0="
+    },
+    "sntp": {
+      "version": "1.0.9",
+      "resolved": "https://registry.npmjs.org/sntp/-/sntp-1.0.9.tgz",
+      "integrity": "sha1-ZUEYTMkK7qbG57NeJlkIJEPGYZg=",
+      "requires": {
+        "hoek": "2.16.3"
+      }
+    },
+    "sshpk": {
+      "version": "1.13.1",
+      "resolved": "https://registry.npmjs.org/sshpk/-/sshpk-1.13.1.tgz",
+      "integrity": "sha1-US322mKHFEMW3EwY/hzx2UBzm+M=",
+      "requires": {
+        "asn1": "0.2.3",
+        "assert-plus": "1.0.0",
+        "bcrypt-pbkdf": "1.0.1",
+        "dashdash": "1.14.1",
+        "ecc-jsbn": "0.1.1",
+        "getpass": "0.1.7",
+        "jsbn": "0.1.1",
+        "tweetnacl": "0.14.5"
+      },
+      "dependencies": {
+        "assert-plus": {
+          "version": "1.0.0",
+          "resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-1.0.0.tgz",
+          "integrity": "sha1-8S4PPF13sLHN2RRpQuTpbB5N1SU="
+        }
+      }
+    },
+    "string-width": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/string-width/-/string-width-1.0.2.tgz",
+      "integrity": "sha1-EYvfW4zcUaKn5w0hHgfisLmxB9M=",
+      "requires": {
+        "code-point-at": "1.1.0",
+        "is-fullwidth-code-point": "1.0.0",
+        "strip-ansi": "3.0.1"
+      }
+    },
+    "string_decoder": {
+      "version": "1.0.3",
+      "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.0.3.tgz",
+      "integrity": "sha512-4AH6Z5fzNNBcH+6XDMfA/BTt87skxqJlO0lAh3Dker5zThcAxG6mKz+iGu308UKoPPQ8Dcqx/4JhujzltRa+hQ==",
+      "requires": {
+        "safe-buffer": "5.1.1"
+      }
+    },
+    "stringstream": {
+      "version": "0.0.5",
+      "resolved": "https://registry.npmjs.org/stringstream/-/stringstream-0.0.5.tgz",
+      "integrity": "sha1-TkhM1N5aC7vuGORjB3EKioFiGHg="
+    },
+    "strip-ansi": {
+      "version": "3.0.1",
+      "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-3.0.1.tgz",
+      "integrity": "sha1-ajhfuIU9lS1f8F0Oiq+UJ43GPc8=",
+      "requires": {
+        "ansi-regex": "2.1.1"
+      }
+    },
+    "strip-json-comments": {
+      "version": "2.0.1",
+      "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz",
+      "integrity": "sha1-PFMZQukIwml8DsNEhYwobHygpgo="
+    },
+    "tar": {
+      "version": "2.2.1",
+      "resolved": "https://registry.npmjs.org/tar/-/tar-2.2.1.tgz",
+      "integrity": "sha1-jk0qJWwOIYXGsYrWlK7JaLg8sdE=",
+      "requires": {
+        "block-stream": "0.0.9",
+        "fstream": "1.0.11",
+        "inherits": "2.0.3"
+      }
+    },
+    "tar-pack": {
+      "version": "3.4.1",
+      "resolved": "https://registry.npmjs.org/tar-pack/-/tar-pack-3.4.1.tgz",
+      "integrity": "sha512-PPRybI9+jM5tjtCbN2cxmmRU7YmqT3Zv/UDy48tAh2XRkLa9bAORtSWLkVc13+GJF+cdTh1yEnHEk3cpTaL5Kg==",
+      "requires": {
+        "debug": "2.6.9",
+        "fstream": "1.0.11",
+        "fstream-ignore": "1.0.5",
+        "once": "1.4.0",
+        "readable-stream": "2.3.4",
+        "rimraf": "2.6.2",
+        "tar": "2.2.1",
+        "uid-number": "0.0.6"
+      }
+    },
+    "tough-cookie": {
+      "version": "2.3.3",
+      "resolved": "https://registry.npmjs.org/tough-cookie/-/tough-cookie-2.3.3.tgz",
+      "integrity": "sha1-C2GKVWW23qkL80JdBNVe3EdadWE=",
+      "requires": {
+        "punycode": "1.4.1"
+      }
+    },
+    "tunnel-agent": {
+      "version": "0.6.0",
+      "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz",
+      "integrity": "sha1-J6XeoGs2sEoKmWZ3SykIaPD8QP0=",
+      "requires": {
+        "safe-buffer": "5.1.1"
+      }
+    },
+    "tweetnacl": {
+      "version": "0.14.5",
+      "resolved": "https://registry.npmjs.org/tweetnacl/-/tweetnacl-0.14.5.tgz",
+      "integrity": "sha1-WuaBd/GS1EViadEIr6k/+HQ/T2Q=",
+      "optional": true
+    },
+    "uid-number": {
+      "version": "0.0.6",
+      "resolved": "https://registry.npmjs.org/uid-number/-/uid-number-0.0.6.tgz",
+      "integrity": "sha1-DqEOgDXo61uOREnwbaHHMGY7qoE="
+    },
+    "util-deprecate": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
+      "integrity": "sha1-RQ1Nyfpw3nMnYvvS1KKJgUGaDM8="
+    },
+    "uuid": {
+      "version": "3.2.1",
+      "resolved": "https://registry.npmjs.org/uuid/-/uuid-3.2.1.tgz",
+      "integrity": "sha512-jZnMwlb9Iku/O3smGWvZhauCf6cvvpKi4BKRiliS3cxnI+Gz9j5MEpTz2UFuXiKPJocb7gnsLHwiS05ige5BEA=="
+    },
+    "verror": {
+      "version": "1.10.0",
+      "resolved": "https://registry.npmjs.org/verror/-/verror-1.10.0.tgz",
+      "integrity": "sha1-OhBcoXBTr1XW4nDB+CiGguGNpAA=",
+      "requires": {
+        "assert-plus": "1.0.0",
+        "core-util-is": "1.0.2",
+        "extsprintf": "1.3.0"
+      },
+      "dependencies": {
+        "assert-plus": {
+          "version": "1.0.0",
+          "resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-1.0.0.tgz",
+          "integrity": "sha1-8S4PPF13sLHN2RRpQuTpbB5N1SU="
+        }
+      }
+    },
+    "wide-align": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/wide-align/-/wide-align-1.1.2.tgz",
+      "integrity": "sha512-ijDLlyQ7s6x1JgCLur53osjm/UXUYD9+0PbYKrBsYisYXzCxN+HC3mYDNy/dWdmf3AwqwU3CXwDCvsNgGK1S0w==",
+      "requires": {
+        "string-width": "1.0.2"
+      }
+    },
+    "wrappy": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
+      "integrity": "sha1-tSQ9jz7BqjXxNkYFvA0QNuMKtp8="
+    }
+  }
+}

package.json

@@ -1,40 +1,42 @@
 {
   "name": "genieacs",
-  "version": "1.0.6",
-  "description": "A blazing fast TR-069 auto configuration server (ACS) built with Node.js, Redis, and MongoDB",
+  "version": "1.1.2",
+  "description": "A TR-069 Auto Configuration Server (ACS)",
   "repository": {
     "type": "git",
-    "url": "https://github.com/zaidka/genieacs.git"
+    "url": "https://github.com/genieacs/genieacs.git"
   },
+  "homepage": "https://genieacs.com",
   "keywords": [
     "TR-069",
     "CWMP",
     "ACS"
   ],
-  "author": "Zaid Abdulla",
+  "author": "Zaid Abdulla <zaid@genieacs.com>",
   "license": "AGPL-3.0",
-  "readmeFilename": "README.md",
   "preferGlobal": true,
   "directories": {
-    "test": "test",
     "bin": "bin"
   },
   "dependencies": {
-    "libxmljs": "0.11.0 - 0.14.x",
-    "redis": "^0.12.1",
-    "hiredis": "0.1.17 - 0.4.x",
-    "mongodb": "^1.4.8"
+    "later": "^1.2.0",
+    "libxmljs": "^0.18.7",
+    "mongodb": "^2.2.34",
+    "seedrandom": "^2.4.3"
+  },
+  "optionalDependencies": {
+    "redis": "^2.6.5"
   },
   "devDependencies": {
-    "coffee-script": "^1.7.1"
+    "coffeescript": "^1.12.7"
   },
   "engines": {
-    "node": ">=0.10.0 <0.12"
+    "node": ">=6.9.2 <9.0"
   },
   "scripts": {
     "prepublish": "npm run compile",
-    "compile": "coffee -cb lib/*.coffee test/*.coffee tools/*.coffee",
-    "configure": "cp ./config/config-sample.json ./config/config.json; cp ./config/parameters-sample.json ./config/parameters.json; cp ./config/custom_commands-sample.json ./config/custom_commands.json; cp ./config/auth-sample.js ./config/auth.js",
+    "compile": "coffee -cb lib/*.coffee",
+    "configure": "cp -n ./config/config-sample.json ./config/config.json",
     "install": "npm run configure",
     "test": "node test"
   }

test/methods.coffee

@@ -1,155 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-###
-
-common = require '../lib/common'
-
-NAMESPACES = {
-  'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-  'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-  'xsd' : 'http://www.w3.org/2001/XMLSchema',
-  'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-  'cwmp' : 'urn:dslforum-org:cwmp-1-0',
-}
-
-
-exports.inform = (device, xmlOut, callback) ->
-  body = xmlOut.root().childNodes()[1]
-  inform = body.node('cwmp:Inform')
-
-  deviceId = inform.node('DeviceId')
-  deviceId.node('Manufacturer', device['InternetGatewayDevice.DeviceInfo.Manufacturer'][1])
-  deviceId.node('OUI', device['InternetGatewayDevice.DeviceInfo.ManufacturerOUI'][1])
-  deviceId.node('ProductClass', device['InternetGatewayDevice.DeviceInfo.ProductClass'][1])
-  deviceId.node('SerialNumber', device['InternetGatewayDevice.DeviceInfo.SerialNumber'][1])
-
-  eventStruct = inform.node('Event').attr({'soap-enc:arrayType' : 'cwmp:EventStruct[1]'}).node('EventStruct')
-  eventStruct.node('EventCode', '2 PERIODIC')
-  eventStruct.node('CommandKey')
-
-  inform.node('MaxEnvelopes', '1')
-  inform.node('CurrentTime', new Date().toISOString())
-  inform.node('RetryCount', '0')
-
-  parameterList = inform.node('ParameterList').attr({'soap-enc:arrayType' : 'cwmp:ParameterValueStruct[7]'})
-  for p in [
-      'InternetGatewayDevice.DeviceInfo.SpecVersion',
-      'InternetGatewayDevice.DeviceInfo.HardwareVersion',
-      'InternetGatewayDevice.DeviceInfo.SoftwareVersion',
-      'InternetGatewayDevice.DeviceInfo.ProvisioningCode',
-      'InternetGatewayDevice.ManagementServer.ParameterKey',
-      'InternetGatewayDevice.ManagementServer.ConnectionRequestURL',
-      'InternetGatewayDevice.WANDevice.1.WANConnectionDevice.1.WANIPConnection.1.ExternalIPAddress']
-    parameterValueStruct = parameterList.node('ParameterValueStruct')
-    parameterValueStruct.node('Name', p)
-    parameterValueStruct.node('Value', device[p][1]).attr({'xsi:type' : "#{device[p][2]}"})
-
-  callback(xmlOut)
-
-
-exports.GetParameterNames = (device, xmlIn, xmlOut, callback) ->
-  parameterNames = Object.keys(device).sort()
-  parameterPath = xmlIn.get('/soap-env:Envelope/soap-env:Body/cwmp:GetParameterNames/ParameterPath', NAMESPACES).text()
-  nextLevel = !!eval(xmlIn.get('/soap-env:Envelope/soap-env:Body/cwmp:GetParameterNames/NextLevel', NAMESPACES).text())
-
-  parameterList = []
-  if nextLevel
-    for p in parameterNames
-      if common.startsWith(p, parameterPath) and p.length != parameterPath.length
-        i = p.indexOf('.', parameterPath.length)
-        if i == -1 or i == p.length - 1
-          parameterList.push(p)
-  else
-    for p in parameterNames
-      if common.startsWith(p, parameterPath)
-        parameterList.push(p)
-
-  getParameterNamesResponseNode = xmlOut.root().childNodes()[1].node('cwmp:GetParameterNamesResponse')
-  parameterListNode = getParameterNamesResponseNode.node('ParameterList')
-  parameterListNode.attr({'soap-enc:arrayType' : "cwmp:ParameterInfoStruct[#{parameterList.length}]"})
-  for p in parameterList
-    parameterInfoStructNode = parameterListNode.node('ParameterInfoStruct')
-    parameterInfoStructNode.node('Name', p)
-    parameterInfoStructNode.node('Writable', String(device[p][0]))
-
-  callback(xmlOut)
-
-
-exports.GetParameterValues = (device, xmlIn, xmlOut, callback) ->
-  parameterNames = xmlIn.find('/soap-env:Envelope/soap-env:Body/cwmp:GetParameterValues/ParameterNames/*', NAMESPACES)
-
-  parameterList = xmlOut.root().childNodes()[1].node('cwmp:GetParameterValuesResponse').node('ParameterList')
-  parameterList.attr({'soap-enc:arrayType' : "cwmp:ParameterValueStruct[#{parameterNames.length}]"})
-
-  for p in parameterNames
-    name = p.text()
-    value = device[name][1]
-    type = device[name][2]
-    valueStruct = parameterList.node('ParameterValueStruct')
-    valueStruct.node('Name', name)
-    valueStruct.node('Value', device[name][1]).attr({'xsi:type' : "#{type}"})
-
-  callback(xmlOut)
-
-
-exports.SetParameterValues = (device, xmlIn, xmlOut, callback) ->
-  parameterValues = xmlIn.find('/soap-env:Envelope/soap-env:Body/cwmp:SetParameterValues/ParameterList/*', NAMESPACES)
-
-  for p in parameterValues
-    name = p.get('Name').text()
-    value = p.get('Value')
-    device[name][1] = value.text()
-    device[name][2] = value.attr('xsi:type')
-
-  responseNode = xmlOut.root().childNodes()[1].node('cwmp:SetParameterValuesResponse')
-  responseNode.node('Status', '0')
-
-  callback(xmlOut)
-
-
-exports.AddObject = (device, xmlIn, xmlOut, callback) ->
-  objectName = xmlIn.get('/soap-env:Envelope/soap-env:Body/cwmp:AddObject/ObjectName', NAMESPACES).text()
-
-  parameters = []
-  instances = {}
-  instanceNumber = 1
-  while device[objectName + instanceNumber + '.']?
-    instanceNumber += 1
-
-  for p in Object.keys(device).sort()
-    if common.startsWith(p, objectName) and p.length > objectName.length
-      n = objectName + instanceNumber + p.slice(p.indexOf('.', objectName.length))
-      if not device[n]?
-        device[n] = [device[p][0], '', device[p][2]]
-
-  responseNode = xmlOut.root().childNodes()[1].node('cwmp:AddObjectResponse')
-  responseNode.node('InstanceNumber', String(instanceNumber))
-  responseNode.node('Status', '0')
-
-  callback(xmlOut)
-
-
-exports.DeleteObject = (device, xmlIn, xmlOut, callback) ->
-  objectName = xmlIn.get('/soap-env:Envelope/soap-env:Body/cwmp:DeleteObject/ObjectName', NAMESPACES).text()
-
-  for p in Object.keys(device)
-    if common.startsWith(p, objectName)
-      delete device[p]
-
-  responseNode = xmlOut.root().childNodes()[1].node('cwmp:DeleteObjectResponse')
-  responseNode.node('Status', '0')
-
-  callback(xmlOut)

test/test.coffee

@@ -1,173 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-###
-
-http = require 'http'
-libxmljs = require 'libxmljs'
-deviceTemplate = require './huawei_bm632w'
-methods = require './methods'
-util = require 'util'
-
-NAMESPACES = {
-  'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-  'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-  'xsd' : 'http://www.w3.org/2001/XMLSchema',
-  'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-  'cwmp' : 'urn:dslforum-org:cwmp-1-0',
-}
-
-
-createSoapDocument = (id) ->
-  xml = libxmljs.Document()
-  env = xml.node('soap-env:Envelope')
-  for prefix, url of NAMESPACES
-    env.defineNamespace prefix, url
-
-  header = env.node('soap-env:Header')
-  header.node('cwmp:ID').attr({'soap-env:mustUnderstand' : 1}).text(id)
-  body = env.node('soap-env:Body')
-  return xml
-
-
-createDevice = (serial) ->
-  device = JSON.parse(JSON.stringify(deviceTemplate))
-  device['InternetGatewayDevice.DeviceInfo.SerialNumber'][1] = serial
-  return device
-
-
-sendRequest = (device,xml, callback) ->
-  headers = {}
-  if xml?
-    body = xml.toString()
-  else
-    body = ''
-
-  headers['Content-Length'] = body.length
-  headers['Content-Type'] = 'text/xml; charset="utf-8"'
-
-  if device.Cookies?
-    headers['Cookie'] = device['Cookies']
-
-  options = {
-    hostname: 'localhost',
-    port: 7547,
-    path: '/',
-    method: 'POST',
-    headers: headers
-  }
-
-  request = http.request(options, (response) ->
-    chunks = []
-    bytes = 0
-    response.on('data', (chunk) ->
-      chunks.push(chunk)
-      bytes += chunk.length
-    )
-    response.on('end', () ->
-      body = new Buffer(bytes)
-      offset = 0
-      chunks.forEach((chunk) ->
-        chunk.copy(body, offset, 0, chunk.length)
-        offset += chunk.length
-      )
-
-      if +response.headers['Content-Length'] > 0 or body.length > 0
-        xml = libxmljs.parseXml(body)
-      else
-        xml = null
-
-      if response.headers['set-cookie']?
-        # TODO set individual cookies
-        device['Cookies'] = [true, response.headers['set-cookie'], 'string']
-
-      callback(xml)
-    )
-  )
-  request.end(body)
-
-
-startSession = (device) ->
-  xmlOut = createSoapDocument(0)
-  methods.inform(device, xmlOut, (xml) ->
-    sendRequest(device, xml, (xml) ->
-      if not xml?
-        console.log('No inform response, trying again in 10 seconds.')
-        setTimeout(() ->
-          startSession(device)
-        , 10000)
-        return
-      sendRequest(device, null, (xml) ->
-        handleMethod(device, xml)
-      )
-    )
-  )
-
-
-createFaultResponse = (xmlOut, code, message) ->
-  body = xmlOut.root().childNodes()[1]
-
-  soapFault = body.node('soap-env:Fault')
-  soapFault.node('faultcode').text('Client')
-  soapFault.node('faultstring').text('CWMP fault')
-
-  fault = soapFault.node('detail').node('cwmp:Fault')
-  fault.node('FaultCode').text(code)
-  fault.node('FaultString').text(message)
-
-
-handleMethod = (device, xml) ->
-  if not xml?
-    setTimeout(() ->
-      startSession(device)
-    , 1000 * parseInt(device['InternetGatewayDevice.ManagementServer.PeriodicInformInterval'][1]))
-    return
-
-  requestId = xml.get('/soap-env:Envelope/soap-env:Header/cwmp:ID', NAMESPACES).text()
-
-  xmlOut = createSoapDocument(requestId)
-
-  element = xml.get('/soap-env:Envelope/soap-env:Body/cwmp:*', NAMESPACES)
-  method = methods[element.name()]
-
-  if method?
-    methods[element.name()](device, xml, xmlOut, (xml) ->
-      sendRequest(device, xml, (xml) ->
-        handleMethod(device, xml)
-      )
-    )
-  else
-    createFaultResponse(xmlOut, 9000, 'Method not supported')
-    sendRequest(device, xmlOut, (xml) ->
-      handleMethod(device, xml)
-    )
-
-
-cluster = require 'cluster'
-
-NUM_WORKERS = 1
-NUM_DEVICES = 1
-
-if cluster.isMaster
-  for i in [0 ... NUM_WORKERS]
-    cluster.fork({SERIAL_PREFIX : i})
-else
-  for i in [0...NUM_DEVICES]
-    serial = process.env.SERIAL_PREFIX + ("00000#{i}".slice(-6))
-    device = createDevice(serial)
-    do (device) ->
-      setTimeout(() ->
-        startSession(device)
-      , i * 100)

tools/dump-data-model

@@ -0,0 +1,91 @@
+#!/usr/bin/env node
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const fs = require("fs");
+const path = require("path");
+
+const outputFilename = path.resolve(`data_model_${process.argv[2]}.json`);
+process.chdir(path.resolve(path.dirname(fs.realpathSync(__filename)), ".."));
+
+const db = require("../lib/db");
+
+const dataModel = {};
+
+
+function recursive(object, path) {
+  if (!path)
+    path = "";
+
+  for (let key in object) {
+    if (key[0] === "_" || !object[key])
+      continue;
+
+    let isObject = object[key]["_object"];
+    let isValue = "_value" in object[key];
+
+    if ((isObject && isValue) || (!isObject && !isValue))
+      throw new Error("Unexpected object");
+
+    if (isValue) {
+      let value = object[key]["_value"];
+      if (typeof value === "boolean")
+        value = +value;
+
+      let values = [object[key]["_writable"] || false, value.toString()];
+      if (object[key]["_type"])
+        values.push(object[key]["_type"]);
+
+      dataModel[`${path}${key}`] = values;
+    }
+
+    if (isObject) {
+      let values = [object[key]["_writable"] || false];
+      dataModel[`${path}${key}.`] = values;
+      recursive(object[key], `${path}${key}.`);
+    }
+  }
+}
+
+
+db.connect(function(err) {
+  if (err)
+    throw err;
+
+  return db.devicesCollection.findOne({
+    _id: process.argv[2]
+  }, {}, function(err, device) {
+    if (err)
+      throw err;
+
+    if (!device)
+      throw new Error("Device not found");
+
+    recursive(device);
+
+    fs.writeFile(outputFilename, JSON.stringify(dataModel, null, 2), function() {
+      if (err)
+        throw err;
+
+      console.log(`${outputFilename} saved`);
+      process.exit(0);
+    });
+  });
+});

tools/genie_auth.pl

@@ -1,27 +0,0 @@
-#!/usr/bin/perl
-use Digest::MD5 qw(md5_base64);
-use URI::Escape;
-print "Enter the details *exactly* as sent in device TR-069 informs (case sensitive).\n";
-print 'Serial number: ';
-chomp($serial = <>);
-$serial = uri_escape($serial, '^A-Za-z0-9_');
-
-print 'OUI: ';
-chomp($oui = <>);
-$oui = uri_escape($oui, '^A-Za-z0-9_');
-
-print 'Product class (leave blank if not available): ';
-chomp($product_class = <>);
-$product_class = uri_escape($product_class, '^A-Za-z0-9_');
-
-if ($product_class eq '') {
-  $device_id = "$oui-$serial"
-}
-else {
-  $device_id = "$oui-$product_class-$serial"
-}
-
-print "ACS username: $device_id\n";
-print 'ACS password: ' . md5_base64("${device_id} Open Sesame") . "\n";
-print "Connection request username: $device_id\n";
-print 'Connection request password: ' . md5_base64("${device_id} Sesame, open") . "\n";

tools/update-deviceid.coffee

@@ -1,37 +0,0 @@
-# This script will generate _deviceId struct for all devices in DB.
-# To be used in migrating existing data to the recent update which introduced _deviceId object.
-config = require '../config'
-mongodb = require 'mongodb'
-
-dbserver = new mongodb.Server(config.MONGODB_SOCKET, 0, {auto_reconnect: true})
-db = new mongodb.Db(config.DATABASE_NAME, dbserver, {native_parser: true, safe: true})
-
-query = {}
-
-projection = {
-  _id : true,
-  'InternetGatewayDevice.DeviceInfo' : true
-}
-
-db.open((err, db) ->
-  db.collection('devices', (err, collection) ->
-    cur = collection.find(query)
-    cur.each((err, doc) ->
-      if not doc?
-        db.close()
-        return
-      serialNumber = doc['InternetGatewayDevice']['DeviceInfo']['SerialNumber']['_value']
-      oui = doc['InternetGatewayDevice']['DeviceInfo']['ManufacturerOUI']['_value']
-      manufacturer = doc['InternetGatewayDevice']['DeviceInfo']['Manufacturer']['_value']
-      productClass = doc['InternetGatewayDevice']['DeviceInfo']['ProductClass']['_value']
-      id = doc['_id']
-      collection.update({_id : id}, {$set : {
-        '_deviceId._Manufacturer' : manufacturer,
-        '_deviceId._OUI' : oui,
-        '_deviceId._ProductClass' : productClass,
-        '_deviceId._SerialNumber' : serialNumber
-      }}, () ->
-      )
-    )
-  )
-)