1.1.1

Because RequestDownload method is no longer supported.

.gitignore

@@ -1,4 +1,28 @@
 *~
-*.js
+lib/*.map
+lib/api-functions.js
+lib/auth.js
+lib/cache.js
+lib/common.js
+lib/config.js
+lib/cwmp.js
+lib/db.js
+lib/fs.js
+lib/nbi.js
+lib/query.js
+lib/server.js
+lib/session.js
 node_modules
-debug/*
+debug
+config/auth.js
+config/config.json
+config/cwmp.key
+config/cwmp.crt
+config/cwmp.ca-bundle
+config/nbi.key
+config/nbi.crt
+config/nbi.ca-bundle
+config/fs.key
+config/fs.crt
+config/fs.ca-bundle
+config/ext

.npmignore

@@ -1,14 +1,21 @@
 *~
 .npmignore
 
-config/*
+config/auth.js
-!config/*-sample.*
+config/config.json
-!config/custom_commands
+config/cwmp.key
+config/cwmp.crt
+config/cwmp.ca-bundle
+config/nbi.key
+config/nbi.crt
+config/nbi.ca-bundle
+config/fs.key
+config/fs.crt
+config/fs.ca-bundle
+config/ext
 
 debug/*
 !debug/.gitkeep
 
 lib/*.coffee
-tools/*.coffee
+lib/*.map
-test/*.coffee
-

CHANGELOG.md

@@ -0,0 +1,54 @@
+# Change Log
+
+## 1.1.1 (2017-03-23)
+
+- Avoid crashing when connection request credentials are missing.
+- Show a warning instead of crashing when failing to parse parameter values
+according to the expected value type.
+- Add missing "Registered" event.
+- Fix bug where in certain cases many more instances than declared are created.
+- Fix parameter discovery bug when declared path timestamp is 1 or is not set.
+- Fix preset precondition failing when testing against datetime parameters and
+certain other parameters like \_deviceId.\_ProductClass.
+
+## 1.1.0 (2017-03-10)
+
+- Provisions enable implementing dynamic device configuration or complex device
+provisioning work flow using arbitrary scripts.
+- Virtual parameters are user-defined parameters whose values are evaluated from
+a custom script.
+- Extensions are sandboxed Node.js scripts that are accessible from provision
+and virtual parameter scripts to facilitate integration with external entities.
+- Support for UDP/STUN based connection requests for reaching devices behind
+NAT (TR-069 Annex G).
+- Presets can now be scheduled using a cron-like expression.
+- Presets can now be tied to specific device events (e.g. boot).
+- Presets precondition queries no longer support "$or" or other MongoDB logical
+operators.
+- Faults are no longer a part of tasks but are now first class objects.
+- Presets are now assigned to channels. A fault in one channel only blocks
+presets in that channel.
+- New API CRUD functions for provisions, virtual parameters, and faults.
+- New config options for XML output.
+- API responses now include "GenieACS-Version" header.
+- Graceful shutdown when receiving SIGINT and SIGTERM events.
+- Support SSL intermediate certificate chains.
+- Supported Node.js versions are 6.x and 7.x.
+- Supported MongoDB versions are 2.6 through 3.4.
+- Expect performance differences due to major under the hood changes. Some
+operations are faster and some are slower. Overall performance is improved.
+- GenieACS will no longer fetch the entire device data model upon first contact
+but will instead only fetch the parameters it needs to fulfill the presets.
+- Logs have been overhauled and split into two streams: process log (stderr) and
+access log (stdout). Also added config options to dump logs to files rather than
+standard streams.
+- Connection request authentication credentials are picked up from the device
+data model if available. config/auth.js is still supported as a fallback and now
+supports an optional callback argument.
+- Custom commands have been removed. Use virtual parameters and/or extensions.
+- Aliases and value normalizers (config/parameters.json) have been removed. Use
+virtual parameters.
+- The API /devices/<device_id>/preset has been removed.
+- Rarely used RequestDownload method no longer supported.
+- The TR-069 client simulator has moved to its own repo at
+https://github.com/zaidka/genieacs-sim

CONTRIBUTING.md

@@ -0,0 +1,39 @@
+# Have a question?
+
+Please ask in the [Users mailing list][mailing-lists] instead of opening a
+Github Issue. The Github issue tracker is strictly for bug reports.
+
+# Issues
+
+We use GitHub issues to track bugs. Please ensure your bug description is
+clear and has sufficient instructions to be able to reproduce the issue.
+
+If you face interoperability issues with your device, more often than not it's
+a device related issue. Please consult the [Users mailing list][mailing-lists]
+before submitting an Issue.
+
+# Pull Requests
+
+Pull requests are welcome. If you have a fix for a bug, go ahead and create a
+pull request. For other kinds of enhancements, please discuss in the [Users
+mailing list][mailing-lists] first.
+
+Please follow the guidelines [here][commit-guidelines] when committing your
+changes.
+
+## Contributor License Agreement
+
+By submitting a pull request to this repository, you acknowledge that, while
+maintaining copyright, you grant Zaid Abdulla a perpetual, worldwide, non-
+exclusive, no-charge, royalty-free, irrevocable license to reproduce, prepare
+derivative works of, publicly display, publicly perform, sublicense, and
+distribute your contributions and such derivative works under the AGPLv3
+license or any other license terms, including, but not limited to, proprietary
+or commercial license terms.
+
+You confirm that you own or have rights to distribute and sublicense the
+source code contained therein, and that your content does not infringe upon
+the intellectual property rights of a third party.
+
+[commit-guidelines]: http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html
+[mailing-lists]: http://lists.genieacs.com

README.md

@@ -1,65 +1,75 @@
 # GenieACS
 
-GenieACS is a *blazing fast* TR-069 auto configuration server (ACS) built with Node.js, Redis, and MongoDB. It's technology neutral and configurable to fit any service provider needs. This is the core back end component. Official front end GUI is available at https://github.com/zaidka/genieacs-gui.
+GenieACS is a fast and lightweight TR-069 auto configuration server (ACS). This
+is the core back end component. A GUI front end is available at
+https://github.com/zaidka/genieacs-gui.
 
 ## Features
 
-* **Massively concurrent**: Can handle hundreds of thousands of connected devices on a single server even with low inform interval.
+- **Battle-tested at scale**: Can scale to manage hundreds of thousands and
-* **Preset-based configuration**: Define sets of configurations that devices will pick up and apply as needed based on given preconditions.
+potentially millions of devices
-* **Tagging**: Use tags to group devices for more manageable presets.
+- **Powerful provisioning system**: Scriptable configuration to handle complex
-* **Searching**: Query for devices on any parameter (supports all common operators including regular expressions).
+device provisioning scenarios and other automated operations such as firmware
-* **Parameter aliasing**: Define aliases to unify parameter paths from different types of devices. Aliases behave like normal parameters (i.e. can be queried or modified).
+upgrade
-* **Extensive API**: A simple yet rich HTTP-based API allows easy integration with other system.
+- **Extensive API**: A rich HTTP-based API allows easy integration with other
+systems
 
-## Getting started
+## Quick Start
 
-Install [Node.js](http://nodejs.org/), [Redis](http://redis.io/), and [MongoDB](http://www.mongodb.org/). Refer to the corresponding documentation for installation guides. Then use NPM to install GenieACS by typing:
+Install [Node.js](http://nodejs.org/), [Redis](http://redis.io/), and
+[MongoDB](http://www.mongodb.org/). Refer to the corresponding documentation for
+installation guides. Supported versions are:
+
+- Node.js: 6.x and 7.x (7.x recommended)
+- Redis: 2.4 through 3.2
+- MongoDB: 2.6 through 3.4
+
+Then install GenieACS using NPM:
 
     npm install -g genieacs
 
-You may need to modify the configuration files under "config" directory (in /lib/node_modules/genieacs/config) depending on your setup.
-
 Alternatively, you can install from source by cloning the git repository:
 
-    cd /opt
+    git clone https://github.com/zaidka/genieacs.git
-    git clone https://github.com/zaidka/genieacs.git --branch v1.0
     cd genieacs
+    git checkout $(git tag -l v1.1.* --sort=-v:refname | head -n 1)
     npm install
-    npm run configure
     npm run compile
 
-Finally, run the following in GNU Screen session or something similar:
+Finally, run the following (from bin directory if installing from source):
 
     genieacs-cwmp
 
-This is the service that the CPEs will communicate with. It listens to port 7547 by default (see config/config.json). Configure the ACS URL of your devices accordingly.
+This is the service that the CPEs will communicate with. It listens to port 7547
+by default (see config/config.json). Configure the ACS URL of your devices
+accordingly.
 
     genieacs-nbi
 
-This is the northbound interface module. It exposes a REST API on port 7557 by default. This is needed for the GUI front end to communicate with.
+This is the northbound interface module. It exposes a REST API on port 7557 by
+default. This must be running for the GUI front end to work.
 
     genieacs-fs
 
-This is the file server from which the CPEs will download firmware images and other types of files.
+This is the file server from which the CPEs will download firmware images and
+such.
 
-You can use stream redirection to output to log files:
+Note: For production deployment make sure to run those as background services.
 
-    genieacs-cwmp >> /var/log/genieacs-cwmp.log 2>> /var/log/genieacs-cwmp-err.log
+For further details about installation and configuration, refer to the
-    genieacs-nbi >> /var/log/genieacs-nbi.log 2>> /var/log/genieacs-nbi-err.log
+[wiki section](https://github.com/zaidka/genieacs/wiki).
-    genieacs-fs >> /var/log/genieacs-fs.log 2>> /var/log/genieacs-fs-err.log
-
-For further details about installation and configuration, refer to the [wiki section](https://github.com/zaidka/genieacs/wiki).
-
-You may now proceed with installing [GenieACS GUI front end](https://github.com/zaidka/genieacs-gui).
 
 ## Support
 
-The [Users mailing list](http://lists.genieacs.com) is a good place to get guidance and help from the community. Head on over and join the conversation! In addition, the [wiki](https://github.com/zaidka/genieacs/wiki) provides useful documentation and tips from GenieACS users.
+The [Users mailing list](http://lists.genieacs.com) is a good place to get
+guidance and help from the community. Head on over and join the conversation!
+In addition, the [wiki](https://github.com/zaidka/genieacs/wiki) provides useful
+documentation and tips from GenieACS users.
 
-You may submit bug reports or feature requests [here](https://github.com/zaidka/genieacs/issues). For device interoperability issues, please consult the mailing list first — it's likely that a workaround already exists.
+For commercial support options and professional services, please visit
+[genieacs.com](https://genieacs.com).
 
-For commercial support options and professional services, please visit [genieacs.com](https://genieacs.com).
+## License
 
-## Contributing
+Copyright 2013-2017 Zaid Abdulla. GenieACS is released under the
-
+[AGPLv3 license terms](https://raw.githubusercontent.com/zaidka/genieacs/master/LICENSE).
-Contributions are welcome. Fork this repo and open a pull request and wait for feedback. You can also contribute by enhancing the documentation in the [wiki section](https://github.com/zaidka/genieacs/wiki).

bin/genieacs-cwmp

@@ -5,5 +5,5 @@ var fs = require('fs');
 var dir = path.resolve(path.dirname(fs.realpathSync(__filename)), '..');
 process.chdir(dir);
 
-cluster = require(path.resolve(dir, 'lib/cluster'));
+var cluster = require(path.resolve(dir, 'lib/cluster'));
 cluster.start('cwmp');

bin/genieacs-fs

@@ -5,5 +5,5 @@ var fs = require('fs');
 var dir = path.resolve(path.dirname(fs.realpathSync(__filename)), '..');
 process.chdir(dir);
 
-cluster = require(path.resolve(dir, 'lib/cluster'));
+var cluster = require(path.resolve(dir, 'lib/cluster'));
 cluster.start('fs');

bin/genieacs-nbi

@@ -5,5 +5,5 @@ var fs = require('fs');
 var dir = path.resolve(path.dirname(fs.realpathSync(__filename)), '..');
 process.chdir(dir);
 
-cluster = require(path.resolve(dir, 'lib/cluster'));
+var cluster = require(path.resolve(dir, 'lib/cluster'));
 cluster.start('nbi');

config/auth-sample.js

@@ -1,4 +1,7 @@
-exports.connectionRequest = function(deviceId) {
+"use strict";
-  // return username/password pair for a given device
+
-  return ["", ""];
+function connectionRequest(deviceId, url, username, password, callback) {
+  return callback(username || deviceId, password || "");
 }
+
+exports.connectionRequest = connectionRequest;

config/custom_commands-sample.json

@@ -1,4 +0,0 @@
-{
-  "seowon_frequencies" : "/^[0-9]+\\-SWU\\%2D9200\\-[0-9A-Z]+$/",
-  "greenpacket_frequencies" : "/^[0-9A-Z]+\\-DX350\\-[0-9A-Z]+$/"
-}

config/custom_commands/greenpacket_frequencies.js

@@ -1,210 +0,0 @@
-/*
- * Copyright 2013 Fanoos Telecom
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-var net = require('net');
-var PORT = 23;
-var USERNAME = 'mt7109';
-var PASSWORD = 'wimax';
-var TIMEOUT = 5000;
-
-var FREQUENCIES_REGEX = /([0-9,]+)/
-var BANDWIDTHS_REGEX = /[0-9]+\-([0-9,]+)/
-var DEFAULT_BW_REGEX = /([0-9]+)\-[0-9,]+/
-var telnetConnect = function(host, callback) {
-  var client = net.connect(PORT, host, function() {
-    // enter username
-    telnetExecute(USERNAME, client, "Password: ", function(err, response) {
-      if (err) return callback(err);
-      // enter password
-      telnetExecute(PASSWORD, client, "\n# ", function(err, response) {
-        return callback(err);
-      });
-    });
-  });
-  return client;
-}
-
-var telnetExecute = function(command, connection, prompt, callback) {
-  var response = '';
-  command += "\n";
-  var listener = function(chunk) {
-    response += chunk.toString();
-    if (response.slice(0 - prompt.length) == prompt) {
-      connection.removeListener('data', listener);
-      return callback(null, response.slice(command.length, 0 - prompt.length));
-    }
-  };
-
-  var end = function() {
-    connection.removeListener('end', end);
-    return callback(null, response.slice(command.length));
-  };
-
-  if (prompt)
-    connection.on('data', listener);
-  else
-    connection.on('end', end);
-
-  if (command)
-    connection.write(command);
-}
-
-var getDeviceIp = function(deviceId, callback) {
-  var db = require('../../lib/db');
-  var URL = require('url');
-
-  db.devicesCollection.findOne({_id : deviceId}, {'InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value' : 1}, function(err, device) {
-    var connectionRequestUrl = device.InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value;
-    var url = URL.parse(connectionRequestUrl);
-    return callback(url.hostname);
-  });
-};
-
-exports.init = function(deviceId, args, callback) {
-  return exports.get(deviceId, args, callback);
-}
-
-exports.get = function(deviceId, args, callback) {
-  var currentFrequencies = [];
-  var currentBandwidths = [];
-  var defaultBandwidth = '';
-  var final = [];
-
-  getDeviceIp(deviceId, function(ip) {
-    var client = telnetConnect(ip, function(err) {
-      // get frequencies
-      telnetExecute("sncfg get WMX_FREQ_LIST", client, "\n# ", function(err, response) {
-        if (err) return callback(err);
-        currentFrequencies = FREQUENCIES_REGEX.exec(response)[1].split(',')
-        // get bandwidths
-        telnetExecute("sncfg get WMX_FREQ_BANDWDITH_FL", client, "\n# ", function(err, response) {
-          if (err) return callback(err);
-          currentBandwidths = BANDWIDTHS_REGEX.exec(response)[1].split(',')
-          defaultBandwidth = DEFAULT_BW_REGEX.exec(response)[1]
-          // generate freq/bw pairs string
-          for (var i = 0; i < currentFrequencies.length; i++) {
-            final.push(currentFrequencies[i] + ' ' + currentBandwidths[i])
-          }
-          // log out
-          telnetExecute("exit", client, null, function(err, response) {
-            client.end();
-            return callback(err, defaultBandwidth + '-' + final.join(','));
-          });
-        });
-      });
-    });
-
-    client.on("error", function(err) {
-      client.destroy();
-      callback(err);
-    });
-
-    client.setTimeout(TIMEOUT, function() {
-      client.destroy();
-      callback("timeout");
-    });
-  });
-};
-
-var parsePairs = function(pairsString) {
-  var a = pairsString.split(',');
-  var f = [];
-  var b = [];
-  var l = [];
-  for (i in a) {
-    var sp = a[i].trim().split(' ');
-    f.push(parseInt(sp[0]));
-    b.push(parseFloat(sp[1]));
-  }
-  l.push(f.join(','));
-  l.push(b.join(','));
-  return l;
-}
-
-exports.set = function(deviceId, args, callback) {
-  executeSet(deviceId, args, ['set'], callback);
-};
-
-exports.dset = function(deviceId, args, callback) {
-  executeSet(deviceId, args, ['dset'], callback);
-};
-
-exports.setdset = function(deviceId, args, callback) {
-  executeSet(deviceId, args, ['dset', 'set'], callback);
-};
-
-var executeSet = function(deviceId, args, method, callback) {
-  var tmp = args.split('-');
-  var defaultBandwidth = tmp[0];
-  var pairs = parsePairs(tmp[1]);
-  var frequenciesString = pairs[0];
-  var bandwidthsString = pairs[1];
-
-  getDeviceIp(deviceId, function(ip) {
-    var client = telnetConnect(ip, function(err) {
-      var telnet_cmd = "sncfg " + method[0];
-      telnetExecute(telnet_cmd + " WMX_FREQ_BANDWDITH_FL '" + defaultBandwidth + "-" + bandwidthsString + "'", client, "\n# ", function(err, response) {
-        if (err) return callback(err);
-        telnetExecute(telnet_cmd + " WMX_FREQ_LIST '" + frequenciesString + "'", client, "\n# ", function(err, response) {
-          if (err) return callback(err);
-          if (method.length > 1) {
-            telnet_cmd = "sncfg " + method[1];
-            telnetExecute(telnet_cmd + " WMX_FREQ_BANDWDITH_FL '" + defaultBandwidth + "-" + bandwidthsString + "'", client, "\n# ", function(err, response) {
-              if (err) return callback(err);
-              telnetExecute(telnet_cmd + " WMX_FREQ_LIST '" + frequenciesString + "'", client, "\n# ", function(err, response) {
-                if (err) return callback(err);
-                telnetExecute("sncfg commit", client, "\n# ", function(err, response) {
-                  if (err) return callback(err);
-                  // log out
-                  telnetExecute("exit", client, null, function(err, response) {
-                    client.end();
-                    return callback(null, args);
-                  });
-                });
-              });
-            });
-          }
-          else {
-            telnetExecute("sncfg commit", client, "\n# ", function(err, response) {
-            if (err) return callback(err);
-              // log out
-              telnetExecute("exit", client, null, function(err, response) {
-                client.end();
-                return callback(null, args);
-              });
-            });
-          }
-        });
-      });
-    });
-
-    client.on("error", function(err) {
-      client.destroy();
-      callback(err);
-    });
-
-    client.setTimeout(TIMEOUT, function() {
-      client.destroy();
-      callback("timeout");
-    });
-  });
-};

config/custom_commands/seowon_frequencies.js

@@ -1,230 +0,0 @@
-/*
- * Copyright 2013 Fanoos Telecom
- *
- * Permission is hereby granted, free of charge, to any person obtaining a copy
- * of this software and associated documentation files (the "Software"), to deal
- * in the Software without restriction, including without limitation the rights
- * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
- * copies of the Software, and to permit persons to whom the Software is
- * furnished to do so, subject to the following conditions:
- *
- * The above copyright notice and this permission notice shall be included in
- * all copies or substantial portions of the Software.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
- * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
- * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
- * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
- * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
- * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
- * THE SOFTWARE.
- */
-
-// Change frequency plans for Seowon devices.
-// Args format: 3532500 5000,3537500 5000,3542500 5000,3547500 5000,3552500 5000,3557500 5000,3505000 10000,3515000 10000,3525000 10000,3535000 10000,3545000 10000,3555000 10000
-var net = require('net');
-var PORT = 23;
-var USERNAME = 'admin';
-var PASSWORD = 'admin';
-var TIMEOUT = 5000;
-
-var FREQUENCIES_REGEX = /\s*([\d]+)\.\s*Frequency\s*:\s*([\d]+)\s\(KHz\),\s*Bandwidth\s*:\s*([\d\.]+)\s*\(MHz\)/gm
-
-var telnetConnect = function(host, callback) {
-  var client = net.connect(PORT, host, function() {
-    // enter username
-    telnetExecute(USERNAME, client, "Password: ", function(err, response) {
-      if (err) return callback(err);
-      // enter password
-      telnetExecute(PASSWORD, client, "MT7109> ", function(err, response) {
-        return callback(err);
-      });
-    });
-  });
-  return client;
-}
-
-var telnetExecute = function(command, connection, prompt, callback) {
-  var response = '';
-  command += "\n";
-  var listener = function(chunk) {
-    response += chunk.toString();
-    if (response.slice(0 - prompt.length) == prompt) {
-      connection.removeListener('data', listener);
-      return callback(null, response.slice(command.length, 0 - prompt.length));
-    }
-  };
-
-  var end = function() {
-    connection.removeListener('end', end);
-    return callback(null, response.slice(command.length));
-  };
-
-  if (prompt)
-    connection.on('data', listener);
-  else
-    connection.on('end', end);
-
-  if (command)
-    connection.write(command);
-}
-
-var parseFrequencies = function(frequenciesString) {
-  var a = frequenciesString.split(',');
-  var l = [];
-  for (i in a) {
-    var sp = a[i].trim().split(' ');
-    var frequency = parseInt(sp[0]);
-    var bandwidth = parseFloat(sp[1]);
-    l.push([frequency, bandwidth]);
-  }
-  return l;
-}
-
-var findFrequency = function(haystack, needle) {
-  for (i in haystack) {
-    if (haystack[i][0] == needle[0] && haystack[i][1] == needle[1])
-      return i;
-  }
-  return -1;
-}
-
-var getDeviceIp = function(deviceId, callback) {
-  var db = require('../../lib/db');
-  var URL = require('url');
-
-  db.devicesCollection.findOne({_id : deviceId}, {'InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value' : 1}, function(err, device) {
-    var connectionRequestUrl = device.InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value;
-    var url = URL.parse(connectionRequestUrl);
-    return callback(url.hostname);
-  });
-};
-
-exports.init = function(deviceId, args, callback) {
-  return exports.get(deviceId, args, callback);
-};
-
-exports.get = function(deviceId, args, callback) {
-  var currentFrequencies = [];
-
-  getDeviceIp(deviceId, function(ip) {
-    var client = telnetConnect(ip, function(err) {
-      // enter privileged mode
-      telnetExecute("enable", client, "MT7109# ", function(err, response) {
-        if (err) return callback(err);
-        // show frequencies
-        telnetExecute("show wmx freq", client, "MT7109# ", function(err, response) {
-          if (err) return callback(err);
-          while (f = FREQUENCIES_REGEX.exec(response)) {
-            var frequency = parseInt(f[2]);
-            var bandwidth = Math.floor(parseFloat(f[3]) * 1000);
-            var freqStr = String(frequency) + ' ' + String(bandwidth);
-            currentFrequencies.push(freqStr);
-          }
-          // log out
-          telnetExecute("logout", client, null, function(err, response) {
-            client.end();
-            return callback(err, currentFrequencies.join(','));
-          });
-        });
-      });
-    });
-
-    client.on("error", function(err) {
-      client.destroy();
-      callback(err);
-    });
-
-    client.setTimeout(TIMEOUT, function() {
-      client.destroy();
-      callback("timeout");
-    });
-  });
-};
-
-exports.set = function(deviceId, args, callback) {
-  var newFrequencies = parseFrequencies(args);
-  var addFrequencies = [];
-  var delFrequencies = [];
-
-  getDeviceIp(deviceId, function(ip) {
-    var client = telnetConnect(ip, function(err) {
-      // enter privileged mode
-      telnetExecute("enable", client, "MT7109# ", function(err, response) {
-        if (err) return callback(err);
-        // show frequencies
-        telnetExecute("show wmx freq", client, "MT7109# ", function(err, response) {
-          if (err) return callback(err);
-          while (f = FREQUENCIES_REGEX.exec(response)) {
-            var index = parseInt(f[1]);
-            var frequency = parseInt(f[2]);
-            var bandwidth = Math.floor(parseFloat(f[3]));
-
-            var i = findFrequency(newFrequencies, [frequency, bandwidth * 1000]);
-            if (i == -1)
-              delFrequencies.push(index);
-            else
-              newFrequencies.splice(i, 1);
-          }
-          for (f in newFrequencies)
-            addFrequencies.push(newFrequencies[f]);
-
-          if (addFrequencies.length == 0 && delFrequencies.length == 0) {
-            // log out
-            telnetExecute("logout", client, null, function(err, response) {
-              client.end();
-              return callback(err, args);
-            });
-            return;
-          }
-
-          telnetExecute("wimax", client, "MT7109 (WiMax)# ", function(err, response) {
-            if (err) return callback(err);
-            var add, del;
-            del = function() {
-              if (delFrequencies.length == 0) {
-                add();
-                return;
-              }
-              var i = delFrequencies.pop();
-              telnetExecute("wmx freq del " + i, client, "MT7109 (WiMax)# ", function(err, response) {
-                if (err) return callback(err);
-                del();
-              });
-            };
-            add = function() {
-              if (addFrequencies.length == 0) {
-                // commit
-                telnetExecute("commit", client, "MT7109 (WiMax)# ", function(err, response) {
-                  if (err) return callback(err);
-                  // log out
-                  telnetExecute("logout", client, null, function(err, response) {
-                    client.end();
-                    return callback(null, args);
-                  });
-                });
-                return;
-              }
-              var i = addFrequencies.pop();
-              telnetExecute("wmx freq add " + i[0] + ' ' + i[1]/1000, client, "MT7109 (WiMax)# ", function(err, response) {
-                if (err) return callback(err);
-                add();
-              });
-            };
-            del();
-          });
-        });
-      });
-    });
-
-    client.on("error", function(err) {
-      client.destroy();
-      callback(err);
-    });
-
-    client.setTimeout(TIMEOUT, function() {
-      client.destroy();
-      callback("timeout");
-    });
-  });
-};

config/ext-sample.js

@@ -0,0 +1,34 @@
+// This is an exmaple GenieACS extension to get the current latitude/longitude
+// of the International Space Station. Why, you ask? Because why not.
+// To install, copy this file to config/ext/iss.js.
+
+"use strict";
+
+const http = require("http");
+
+let cache = null;
+let cacheExpire = 0;
+
+function latlong(args, callback) {
+  if (Date.now() < cacheExpire)
+    return callback(null, cache);
+
+  http.get("http://api.open-notify.org/iss-now.json", (res) => {
+    if (res.statusCode !== 200)
+      return callback(new Error(`Request failed (status code: ${res.statusCode})`));
+
+    let rawData = "";
+    res.on("data", (chunk) => rawData += chunk);
+
+    res.on("end", () => {
+      let pos = JSON.parse(rawData)["iss_position"];
+      cache = [+pos["latitude"], +pos["longitude"]];
+      cacheExpire = Date.now() + 10000;
+      callback(null, cache);
+    });
+  }).on("error", (err) => {
+    callback(err);
+  });
+}
+
+exports.latlong = latlong;

config/parameters-sample.json

@@ -1,27 +0,0 @@
-{
-  "_lastInform" : {"alias" : "summary.lastInform", "type" : "date"},
-  "_lastBoot" : {"alias" : "summary.lastBoot", "type" : "date"},
-  "_lastBootstrap" : {"alias" : "summary.lastBootstrap", "type" : "date"},
-  "_deviceId._SerialNumber" : {"alias" : "summary.serialNumber"},
-  "_deviceId._ProductClass" : {"alias" : "summary.productClass"},
-  "_deviceId._OUI" : {"alias" : "summary.oui"},
-  "_deviceId._Manufacturer" : {"alias" : "summary.manufacturer"},
-  "InternetGatewayDevice.DeviceInfo.HardwareVersion" : {"alias" : "summary.hardwareVersion"},
-  "InternetGatewayDevice.DeviceInfo.SoftwareVersion" : {"alias" : "summary.softwareVersion"},
-  "Device.DeviceInfo.HardwareVersion" : {"alias" : "summary.hardwareVersion"},
-  "Device.DeviceInfo.SoftwareVersion" : {"alias" : "summary.softwareVersion"},
-
-  "InternetGatewayDevice.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANIPConnection./^\\d+$/.MACAddress" : {"alias" : "summary.mac", "type" : "mac"},
-  "InternetGatewayDevice.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANIPConnection./^\\d+$/.ExternalIPAddress" : {"alias" : "summary.ip"},
-  "InternetGatewayDevice.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANPPPConnection./^\\d+$/.MACAddress" : {"alias" : "summary.mac", "type" : "mac"},
-  "InternetGatewayDevice.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANPPPConnection./^\\d+$/.ExternalIPAddress" : {"alias" : "summary.ip"},
-  "Device.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANIPConnection./^\\d+$/.MACAddress" : {"alias" : "summary.mac", "type" : "mac"},
-  "Device.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANIPConnection./^\\d+$/.ExternalIPAddress" : {"alias" : "summary.ip"},
-  "Device.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANPPPConnection./^\\d+$/.MACAddress" : {"alias" : "summary.mac", "type" : "mac"},
-  "Device.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANPPPConnection./^\\d+$/.ExternalIPAddress" : {"alias" : "summary.ip"},
-
-  "InternetGatewayDevice.LANDevice./^\\d+$/.WLANConfiguration./^\\d+$/.SSID" : {"alias" : "summary.wlanSsid"},
-  "InternetGatewayDevice.LANDevice./^\\d+$/.WLANConfiguration./^\\d+$/.PreSharedKey.1.KeyPassphrase" : {"alias" : "summary.wlanPassphrase"},
-  "Device.LANDevice./^\\d+$/.WLANConfiguration./^\\d+$/.SSID" : {"alias" : "summary.wlanSsid"},
-  "Device.LANDevice./^\\d+$/.WLANConfiguration./^\\d+$/.PreSharedKey.1.KeyPassphrase" : {"alias" : "summary.wlanPassphrase"}
-}

lib/api-functions.coffee

@@ -1,5 +1,7 @@
 ###
-# Copyright 2013, 2014  Zaid Abdulla
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
 #
 # GenieACS is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as
@@ -13,31 +15,11 @@
 #
 # You should have received a copy of the GNU Affero General Public License
 # along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-#
-# This file incorporates work covered by the following copyright and
-# permission notice:
-#
-# Copyright 2013 Fanoos Telecom
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
 ###
 
+crypto = require 'crypto'
+dgram = require 'dgram'
+
 config = require './config'
 db = require './db'
 http = require 'http'
@@ -47,9 +29,39 @@ URL = require 'url'
 auth = require './auth'
 
 
-connectionRequest = (deviceId, callback) ->
+udpConReq = (address, un, key, callback) ->
-  # ensure socket is reused in case of digest authentication
+  return false if not address
-  agent = new http.Agent({maxSockets : 1})
+  [host, port] = address.split(':', 2)
+  if port is undefined
+    port = 80
+  else
+    port = parseInt(port)
+    return false if isNaN(port)
+
+  ts = Math.trunc(Date.now() / 1000)
+  id = Math.trunc(Math.random() * 4294967295)
+  cn = crypto.randomBytes(8).toString('hex')
+  sig = crypto.createHmac('sha1', key).update("#{ts}#{id}#{un}#{cn}").digest('hex')
+  uri = "http://#{address}?ts=#{ts}&id=#{id}&un=#{un}&cn=#{cn}&sig=#{sig}"
+
+  message = Buffer.from("GET #{uri} HTTP/1.1\r\nHost: #{address}\r\n\r\n")
+
+  client = dgram.createSocket('udp4')
+
+  count = 3
+  client.send(message, 0, message.length, port, host, f = (err) ->
+    if err or -- count <= 0
+      client.close()
+      return callback(err)
+    client.send(message, 0, message.length, port, host, f)
+  )
+  return true
+
+
+httpConReq = (url, username, password, timeout, callback) ->
+  options = URL.parse(url)
+  # Ensure socket is reused in case of digest authentication
+  options.agent = new http.Agent({maxSockets : 1})
 
   statusToError = (statusCode) ->
     switch statusCode
@@ -62,102 +74,126 @@ connectionRequest = (deviceId, callback) ->
       else
         new Error("Unexpected response code from device: #{statusCode}")
 
-  conReq = (url, authString, callback) ->
-    options = URL.parse(url)
-    options.agent = agent
-
-    if authString
-      options.headers = {'Authorization' : authString}
-
   request = http.get(options, (res) ->
     if res.statusCode == 401 and res.headers['www-authenticate']?
       authHeader = auth.parseAuthHeader(res.headers['www-authenticate'])
-      callback(res.statusCode, authHeader)
+      if authHeader.method is 'Basic'
-      # don't need body, go ahead and emit events to free up the socket for possible reuse
+        options.headers = {'Authorization' : auth.basic(username or '', password or '')}
+      else if authHeader.method is 'Digest'
+        options.headers = {
+          'Authorization' : auth.digest(username or '', password or '', options.path, 'GET', null, authHeader)
+        }
+
+      request = http.get(options, (res) ->
+        if res.statusCode == 0
+          # Workaround for some devices unexpectedly closing the connection
+          request = http.get(options, (res) ->
+            callback(statusToError(res.statusCode))
             res.resume()
-    )
+          ).on('error', (err) ->
-
-    request.on('error', (err) ->
-      # error event when request is aborted
             request.abort()
-      callback(0)
+            callback(statusToError(0))
+          ).on('socket', (socket) ->
+            socket.setTimeout(timeout)
+            socket.on('timeout', () -> request.abort())
           )
-
+        else
-    request.on('socket', (socket) ->
+          callback(statusToError(res.statusCode))
-      socket.setTimeout(2000)
+        res.resume()
-      socket.on('timeout', () ->
+      ).on('error', (err) ->
         request.abort()
+        callback(statusToError(0))
+      ).on('socket', (socket) ->
+        socket.setTimeout(timeout)
+        socket.on('timeout', () -> request.abort())
       )
+    else
+      callback(statusToError(res.statusCode))
+
+    # No listener for data so emit resume
+    res.resume()
+  ).on('error', (err) ->
+    request.abort()
+    callback(statusToError(0))
+  ).on('socket', (socket) ->
+    socket.setTimeout(timeout)
+    socket.on('timeout', () -> request.abort())
   )
 
+
+connectionRequest = (deviceId, callback) ->
+  CONNECTION_REQUEST_TIMEOUT = config.get('CONNECTION_REQUEST_TIMEOUT', deviceId)
   proj = {
     'Device.ManagementServer.ConnectionRequestURL._value' : 1,
-    'InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value' : 1
+    'Device.ManagementServer.UDPConnectionRequestAddress._value' : 1,
+    'Device.ManagementServer.ConnectionRequestUsername._value' : 1,
+    'Device.ManagementServer.ConnectionRequestPassword._value' : 1,
+    'InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value' : 1,
+    'InternetGatewayDevice.ManagementServer.UDPConnectionRequestAddress._value' : 1,
+    'InternetGatewayDevice.ManagementServer.ConnectionRequestUsername._value' : 1,
+    'InternetGatewayDevice.ManagementServer.ConnectionRequestPassword._value' : 1
   }
 
   db.devicesCollection.findOne({_id : deviceId}, proj, (err, device)->
-    if err
+    return callback(err) if err
-      callback(err)
-      return
 
     if device.Device? # TR-181 data model
       connectionRequestUrl = device.Device.ManagementServer.ConnectionRequestURL._value
+      udpConnectionRequestAddress = device.Device.ManagementServer?.UDPConnectionRequestAddress?._value
+      username = device.Device.ManagementServer.ConnectionRequestUsername?._value
+      password = device.Device.ManagementServer.ConnectionRequestPassword?._value
     else # TR-098 data model
       connectionRequestUrl = device.InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value
+      udpConnectionRequestAddress = device.InternetGatewayDevice.ManagementServer?.UDPConnectionRequestAddress?._value
+      username = device.InternetGatewayDevice.ManagementServer.ConnectionRequestUsername?._value
+      password = device.InternetGatewayDevice.ManagementServer.ConnectionRequestPassword?._value
 
-    # for testing
+    conReq = () ->
-    #connectionRequestUrl = connectionRequestUrl.replace(/^(http:\/\/)([0-9\.]+)(\:[0-9]+\/[a-zA-Z0-9]+\/?$)/, '$110.1.1.254$3')
+      udpSent = udpConReq(udpConnectionRequestAddress, username, password, (err) -> throw err if err)
-    conReq(connectionRequestUrl, null, (statusCode, authHeader) ->
-      if statusCode == 401
-        [username, password] = config.auth.connectionRequest(deviceId)
-        if authHeader.method is 'Basic'
-          authString = auth.basic(username, password)
-        else if authHeader.method is 'Digest'
-          uri = URL.parse(connectionRequestUrl)
-          authString = auth.digest(username, password, uri.path, 'GET', null, authHeader)
 
-        conReq(connectionRequestUrl, authString, (statusCode, authHeader) ->
+      httpConReq(connectionRequestUrl, username, password, CONNECTION_REQUEST_TIMEOUT, (err) ->
-          if statusCode == 0
+        if udpSent
-            # Workaround for some devices unexpectedly closing the connection
+          return callback()
-            return conReq(connectionRequestUrl, authString, (statusCode) -> callback(statusToError(statusCode)))
+        callback(err)
-          callback(statusToError(statusCode))
       )
-      else
+
-        callback(statusToError(statusCode))
+    if config.auth?.connectionRequest?
+      # Callback is optional for backward compatibility
+      if config.auth.connectionRequest.length > 4
+        return config.auth.connectionRequest(deviceId, connectionRequestUrl, username, password, (u, p) ->
+          username = u
+          password = p
+          conReq()
         )
+      [username, password] = config.auth.connectionRequest(deviceId, connectionRequestUrl, username, password)
+    conReq()
   )
 
 
-watchTask = (taskId, timeout, callback) ->
+watchTask = (deviceId, taskId, timeout, callback) ->
-  setTimeout( () ->
+  setTimeout(() ->
-    db.tasksCollection.findOne({_id : taskId}, {'_id' : 1, 'fault' : 1}, (err, task) ->
+    db.tasksCollection.findOne({_id : taskId}, {'_id' : 1}, (err, task) ->
       return callback(err) if err
 
-      if task
+      if not task
-        timeout -= 500
+        return callback(null, 'completed')
-        if task.fault?
+
-          callback(null, 'fault')
+      db.faultsCollection.findOne({_id : "#{deviceId}:task_#{taskId}"}, {'_id' : 1}, (err, fault) ->
-        else if timeout <= 0
+        return callback(err) if err
-          callback(null, 'timeout')
+
-        else
+        if fault
-          watchTask(taskId, timeout, callback)
+          return callback(null, 'fault')
-      else
+
-        callback(null, 'completed')
+        if (timeout -= 500) <= 0
+          return callback(null, 'timeout')
+
+        watchTask(deviceId, taskId, timeout, callback)
+      )
     )
   , 500)
 
 
-expandParam = (param, aliases) ->
+sanitizeTask = (task, callback) ->
-  params = [param]
-  for a,aa of aliases
-    if a == param or common.startsWith(a, "#{param}.")
-      for p in aa
-        params.push(p) if p[p.lastIndexOf('.') + 1] != '_'
-
-  return params
-
-
-sanitizeTask = (task, aliases, callback) ->
   task.timestamp = new Date(task.timestamp ? Date.now())
   if task.expiry?
     if common.typeOf(task.expiry) is common.DATE_TYPE or isNaN(task.expiry)
@@ -165,56 +201,10 @@ sanitizeTask = (task, aliases, callback) ->
     else
       task.expiry = new Date(task.timestamp.getTime() + +task.expiry * 1000)
 
-  switch task.name
-    when 'getParameterValues'
-      projection = {}
-      for p in task.parameterNames
-        for pp in expandParam(p, aliases)
-          projection[pp] = 1
-      db.devicesCollection.findOne({_id : task.device}, projection, (err, device) ->
-        parameterNames = []
-        for k of projection
-          if common.getParamValueFromPath(device, k)?
-            parameterNames.push(k)
-        task.parameterNames = parameterNames
-        callback(task)
-      )
-    when 'setParameterValues'
-      projection = {}
-      values = {}
-      for p in task.parameterValues
-        for pp in expandParam(p[0], aliases)
-          projection[pp] = 1
-          values[pp] = p[1]
-      db.devicesCollection.findOne({_id : task.device}, projection, (err, device) ->
-        parameterValues = []
-        for k of projection
-          param = common.getParamValueFromPath(device, k)
-          if param?
-            parameterValues.push([k, values[k], param._type])
-        task.parameterValues = parameterValues
-        callback(task)
-      )
-    else
-      # TODO implement setParameterValues
   callback(task)
 
 
-addAliases = (device, aliases) ->
+insertTasks = (tasks, callback) ->
-  for k,v of aliases
-    for p in v
-      pp = p.split('.')
-      obj = device
-      for i in pp
-        if not obj[i]?
-          obj = null
-          break
-        obj = obj[i]
-
-      device[k] = obj if obj?
-
-
-insertTasks = (tasks, aliases, callback) ->
   if tasks? and common.typeOf(tasks) isnt common.ARRAY_TYPE
     tasks = [tasks]
   else if not tasks? or tasks.length == 0
@@ -223,9 +213,9 @@ insertTasks = (tasks, aliases, callback) ->
   counter = tasks.length
 
   for task in tasks
-    sanitizeTask(task, aliases, (t) ->
+    sanitizeTask(task, (t) ->
       if t.uniqueKey?
-        db.tasksCollection.remove({device : t.device, uniqueKey : t.uniqueKey}, (err, removed) ->
+        db.tasksCollection.remove({device : t.device, uniqueKey : t.uniqueKey}, (err) ->
         )
 
       --counter
@@ -238,19 +228,26 @@ insertTasks = (tasks, aliases, callback) ->
 
 
 deleteDevice = (deviceId, callback) ->
-  db.tasksCollection.remove({'device' : deviceId}, (err, removed) ->
+  db.tasksCollection.remove({'device' : deviceId}, (err) ->
     return callback(err) if err
-    db.devicesCollection.remove({'_id' : deviceId}, (err, removed) ->
+    db.devicesCollection.remove({'_id' : deviceId}, (err) ->
       return callback(err) if err
-      db.redisClient.del("#{deviceId}_presets_hash", "#{deviceId}_inform_hash", (err) ->
+      db.faultsCollection.remove({'_id' : {'$regex' : "^#{common.escapeRegExp(deviceId)}\\:"}}, (err) ->
-        callback(err)
+        return callback(err) if err
+        db.operationsCollection.remove({'_id' : {'$regex' : "^#{common.escapeRegExp(deviceId)}\\:"}}, (err) ->
+          return callback(err) if err
+          db.redisClient.del("#{deviceId}_presets_hash",
+            "#{deviceId}_inform_hash",
+            "#{deviceId}_faults",
+            "#{deviceId}_tasks",
+            "#{deviceId}_operations",
+            callback)
+        )
       )
     )
   )
 
 
-exports.addAliases = addAliases
-exports.sanitizeTask = sanitizeTask
 exports.connectionRequest = connectionRequest
 exports.watchTask = watchTask
 exports.insertTasks = insertTasks

lib/auth.coffee

@@ -1,5 +1,7 @@
 ###
-# Copyright 2013, 2014  Zaid Abdulla
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
 #
 # GenieACS is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as
@@ -19,18 +21,10 @@ crypto = require 'crypto'
 
 
 exports.parseAuthHeader = (authHeader) ->
-  res = {}
+  res = {method: authHeader.slice(0, authHeader.indexOf(' '))}
-  i = authHeader.indexOf(' ')
+  re = /([a-z0-9_-]+)=(?:"([^"]+)"|([a-z0-9_-]+))/gi
-  res['method'] = authHeader.slice(0, i)
+  while match = re.exec(authHeader)
-
+    res[match[1]] = match[2] or match[3]
-  options = authHeader.slice(i + 1)
-  regex = /([a-z]+)="([a-zA-Z0-9\/\.@\-\+,= ]+)"/g
-  while (r = regex.exec(options)) != null
-    res[r[1]] = r[2]
-
-  regex = /([a-z]+)=([a-zA-Z0-9\/\.@\-\+]+)/g
-  while (r = regex.exec(options)) != null
-    res[r[1]] = r[2]
   return res
 
 
@@ -72,9 +66,7 @@ exports.digest = (username, password, uri, httpMethod, body, authHeader) ->
   authString += ",nonce=\"#{authHeader.nonce}\""
   authString += ",uri=\"#{uri}\""
   authString += ",algorithm=#{authHeader.algorithm}" if authHeader.algorithm?
-  authString += ",qop=#{qop}"
+  authString += ",qop=#{qop},nc=#{nc},cnonce=\"#{cnonce}\"" if qop?
-  authString += ",nc=#{nc}"
-  authString += ",cnonce=\"#{cnonce}\""
   authString += ",response=\"#{response}\""
-  authString += ",opaque=\"#{authHeader.opaque}\""
+  authString += ",opaque=\"#{authHeader.opaque}\"" if authHeader.opaque?
   return authString

lib/cache.coffee

@@ -0,0 +1,310 @@
+###
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
+#
+# GenieACS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# GenieACS is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+###
+
+vm = require 'vm'
+crypto = require 'crypto'
+
+later = require 'later'
+
+db = require './db'
+query = require './query'
+
+
+REFRESH = 3000
+
+nextRefresh = 0
+hash = null
+presets = null
+provisions = null
+virtualParameters = null
+files = null
+
+
+UNLOCK_SCRIPT = 'if redis.call("get", KEYS[1]) == ARGV[1] then return redis.call("del", KEYS[1]) else return 0 end'
+EXTEND_SCRIPT = 'if redis.call("get", KEYS[1]) == ARGV[1] then return redis.call("pexpire", KEYS[1], ARGV[2]) else return 0 end'
+
+lock = (lockName, ttl, callback) ->
+  token = Math.random().toString(36).slice(2)
+  unlockOrExtend = (ttl) ->
+    if not ttl
+      db.redisClient.eval(UNLOCK_SCRIPT, 1, lockName, token, (err, res) ->
+        throw err or new Error('Lock expired') if err or not res
+      )
+    else
+      db.redisClient.eval(EXTEND_SCRIPT, 1, lockName, token, ttl, (err, res) ->
+        throw err or new Error('Lock expired') if err or not res
+      )
+    return
+
+  db.redisClient.set(lockName, token, 'NX', 'PX', ttl, (err, res) ->
+    if err or res
+      return callback(err, unlockOrExtend)
+
+    setTimeout(() ->
+      return lock(lockName, ttl, callback)
+    , 200)
+  )
+
+
+computeHash = () ->
+  # MD5 hash for presets, provisions, virtual parameters for detecting changes
+  h = crypto.createHash('md5')
+  for p in presets
+    h.update(JSON.stringify(p.name))
+    h.update(JSON.stringify(p.channel))
+    h.update(JSON.stringify(p.schedule))
+    h.update(JSON.stringify(p.events))
+    h.update(JSON.stringify(p.precondition))
+    h.update(JSON.stringify(p.provisions))
+
+  keys = Object.keys(provisions).sort()
+  h.update(JSON.stringify(keys))
+  h.update(provisions[k].md5) for k in keys
+
+  keys = Object.keys(virtualParameters).sort()
+  h.update(JSON.stringify(keys))
+  h.update(virtualParameters[k].md5) for k in keys
+
+  hash = h.digest('hex')
+
+
+refresh = (callback) ->
+  now = Date.now()
+  db.redisClient.get('presets_hash', (err, res) ->
+    return callback(err) if err
+
+    if hash? and res == hash
+      nextRefresh = now + REFRESH
+      return callback()
+
+    lock('presets_hash_lock', 3000, (err, unlockOrExtend) ->
+      return callback(err) if err
+      counter = 3
+
+      counter += 2
+      db.presetsCollection.find().toArray((err, res) ->
+        if err
+          callback(err) if counter & 1
+          return counter = 0
+
+        db.objectsCollection.find().toArray((err, objects) ->
+          if err
+            callback(err) if (counter & 1)
+            return counter = 0
+
+          res.sort((a, b) ->
+            if a.weight == b.weight
+              return a._id > b._id
+            else
+              return a.weight - b.weight
+          )
+
+          presets = []
+
+          for preset in res
+            schedule = null
+            if preset.schedule
+              parts = preset.schedule.trim().split(/\s+/)
+              schedule = {
+                md5: crypto.createHash('md5').update(preset.schedule).digest('hex')
+              }
+
+              try
+                schedule.duration = +(parts.shift()) * 1000
+                parts.unshift('0') if parts.length == 5
+                # TODO later.js doesn't throw erorr if expression is invalid!
+                schedule.schedule = later.schedule(later.parse.cron(parts.join(' '), true))
+              catch err
+                # TODO show a warning
+                schedule.schedule = false
+
+            events = preset.events ? {}
+
+            precondition = query.convertMongoQueryToFilters(JSON.parse(preset.precondition))
+
+            _provisions = preset.provisions or []
+
+            # Generate provisions from the old configuration format
+            for c in preset.configurations
+              switch c.type
+                when 'age'
+                  _provisions.push(['refresh', c.name, c.age])
+                when 'value'
+                  _provisions.push(['value', c.name, c.value])
+                when 'add_tag'
+                  _provisions.push(['tag', c.tag, true])
+                when 'delete_tag'
+                  _provisions.push(['tag', c.tag, false])
+                when 'provision'
+                  _provisions.push([c.name].concat(c.args or []))
+                when 'add_object'
+                  for obj in objects
+                    if obj['_id'] == c.object
+                      alias = ("#{k}:#{JSON.stringify(obj[k])}" for k in obj['_keys']).join(',')
+                      p = "#{c.name}.[#{alias}]"
+                      _provisions.push(['instances', p, 1])
+                      for k of obj
+                        if k[0] != '_' and k not in obj['_keys']
+                          _provisions.push(['value', "#{p}.#{k}", obj[k]])
+                when 'delete_object'
+                  for obj in objects
+                    if obj['_id'] == c.object
+                      alias = ("#{k}:#{JSON.stringify(obj[k])}" for k in obj['_keys']).join(',')
+                      p = "#{c.name}.[#{alias}]"
+                      _provisions.push(['instances', p, 0])
+                else
+                  callback(new Error("Unknown configuration type #{c.type}")) if counter & 1
+                  return counter = 0
+
+            presets.push({name: preset._id, channel: preset.channel or 'default', schedule: schedule, events: events, precondition: precondition, provisions: _provisions})
+
+          if (counter -= 2) == 1
+            computeHash()
+            db.redisClient.setex("presets_hash", 300, hash, (err) ->
+              unlockOrExtend(0)
+              nextRefresh = now + REFRESH
+              return callback()
+            )
+        )
+      )
+
+      counter += 2
+      db.provisionsCollection.find().toArray((err, res) ->
+        if err
+          callback(err) if counter & 1
+          return counter = 0
+
+        provisions = {}
+        for r in res
+          provisions[r._id] = {}
+          provisions[r._id].md5 = crypto.createHash('md5').update(r.script).digest('hex')
+          provisions[r._id].script = new vm.Script("\"use strict\";(function(){\n#{r.script}\n})();", {filename: r._id, lineOffset: -1, timeout: 50})
+
+        if (counter -= 2) == 1
+          computeHash()
+          db.redisClient.setex("presets_hash", 300, hash, (err) ->
+            unlockOrExtend(0)
+            nextRefresh = now + REFRESH
+            return callback()
+          )
+      )
+
+      counter += 2
+      db.virtualParametersCollection.find().toArray((err, res) ->
+        if err
+          callback(err) if counter & 1
+          return counter = 0
+
+        virtualParameters = {}
+        for r in res
+          virtualParameters[r._id] = {}
+          virtualParameters[r._id].md5 = crypto.createHash('md5').update(r.script).digest('hex')
+          virtualParameters[r._id].script = new vm.Script("\"use strict\";(function(){\n#{r.script}\n})();", {filename: r._id, lineOffset: -1, timeout: 50})
+
+        if (counter -= 2) == 1
+          computeHash()
+          db.redisClient.setex("presets_hash", 300, hash, (err) ->
+            unlockOrExtend(0)
+            nextRefresh = now + REFRESH
+            return callback()
+          )
+      )
+
+      counter += 2
+      db.filesCollection.find().toArray((err, res) ->
+        if err
+          callback(err) if counter & 1
+          return counter = 0
+
+        files = {}
+        for r in res
+          id = r.filename or r._id.toString()
+          files[id] = {}
+          files[id].length = r.length
+          files[id].md5 = r.md5
+          files[id].contentType = r.contentType
+
+        if (counter -= 2) == 1
+          computeHash()
+          db.redisClient.setex("presets_hash", 300, hash, (err) ->
+            unlockOrExtend(0)
+            nextRefresh = now + REFRESH
+            return callback()
+          )
+      )
+
+      if (counter -= 2) == 1
+        computeHash()
+        db.redisClient.setex("presets_hash", 300, hash, (err) ->
+          unlockOrExtend(0)
+          nextRefresh = now + REFRESH
+          return callback()
+        )
+    )
+  )
+
+
+getPresets = (callback) ->
+  if Date.now() < nextRefresh
+    return callback(null, hash, presets)
+
+  refresh((err) ->
+    return callback(err, hash, presets)
+  )
+
+
+getProvisions = (callback) ->
+  if Date.now() < nextRefresh
+    return callback(null, hash, provisions)
+
+  refresh((err) ->
+    return callback(err, hash, provisions)
+  )
+
+
+getVirtualParameters = (callback) ->
+  if Date.now() < nextRefresh
+    return callback(null, hash, virtualParameters)
+
+  refresh((err) ->
+    return callback(err, hash, virtualParameters)
+  )
+
+
+getFiles = (callback) ->
+  if Date.now() < nextRefresh
+    return callback(null, hash, files)
+
+  refresh((err) ->
+    return callback(err, hash, files)
+  )
+
+
+getProvisionsAndVirtualParameters = (callback) ->
+  if Date.now() < nextRefresh
+    return callback(null, hash, provisions, virtualParameters)
+
+  refresh((err) ->
+    return callback(err, hash, provisions, virtualParameters)
+  )
+
+exports.getPresets = getPresets
+exports.getProvisions = getProvisions
+exports.getFiles = getFiles
+exports.getProvisionsAndVirtualParameters = getProvisionsAndVirtualParameters

lib/cluster.coffee

@@ -1,50 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-###
-
-config = require './config'
-cluster = require 'cluster'
-util = require 'util'
-
-# Keep current configuration to pass that to workers as environment variables
-ENVIRONMENT_VARIABLES = {}
-for k, v of config.allConfig
-  ENVIRONMENT_VARIABLES["GENIEACS_#{k}"] = v
-
-
-cluster.on('listening', (worker, address) ->
-  util.log("Worker #{worker.process.pid} listening to #{address.address}:#{address.port}")
-)
-
-cluster.on('exit', (worker, code, signal) ->
-  util.log("Worker #{worker.process.pid} died (#{worker.process.exitCode})")
-  cluster.fork(ENVIRONMENT_VARIABLES)
-)
-
-
-start = (service) ->
-  workerProcesses = config.get("#{service.toUpperCase()}_WORKER_PROCESSES")
-
-  cluster.setupMaster({
-      exec : 'lib/server',
-      args : [service]
-  })
-
-  for [0 ... workerProcesses]
-    cluster.fork(ENVIRONMENT_VARIABLES)
-
-
-exports.start = start

lib/cluster.js

@@ -0,0 +1,139 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const cluster = require("cluster");
+const logger = require("./logger");
+
+const VERSION = require("../package.json").version;
+
+
+function getDependencyVersions() {
+  let dependencies = [`node@${process.versions.node}`];
+  try {
+    let npmls = JSON.parse(require("child_process").execSync("npm ls --prod --json"));
+
+    for (let k in npmls.dependencies)
+      dependencies.push(`${k}@${npmls.dependencies[k].version}`);
+  } catch (err) {}
+
+  return dependencies.join(",");
+}
+
+
+function getConfig() {
+  let conf = [];
+  const keys = Object.keys(process.env);
+  for (let k of keys)
+    if (k.startsWith("GENIEACS_"))
+      conf.push(`${k.slice(9)}=${process.env[k]}`);
+
+  return conf.join(",");
+}
+
+
+function restartWorker(worker, code, signal) {
+  let msg = {
+    message: "Worker died",
+    pid: worker.process.pid
+  };
+
+  if (code != null)
+    msg.exitCode = code;
+
+  if (signal != null)
+    msg.signal = signal;
+
+  logger.error(msg);
+  cluster.fork();
+}
+
+
+function start(service) {
+  logger.init(service, VERSION);
+  const config = require("./config");
+
+  logger.info({
+    message: `GenieACS (genieacs-${service}) starting`,
+    pid: process.pid,
+    version: VERSION,
+    dependencies: getDependencyVersions(),
+    config: getConfig()
+  });
+
+  cluster.on("listening", function(worker, address) {
+    logger.info({
+      message: "Worker listening",
+      pid: worker.process.pid,
+      address: address.address,
+      port: address.port
+    });
+  });
+
+  cluster.on("exit", restartWorker);
+
+  process.on("SIGINT", function() {
+    logger.info({
+      message: "Received signal SIGINT, exiting",
+      pid: process.pid
+    });
+
+    cluster.removeListener("exit", restartWorker);
+  });
+
+  process.on("SIGTERM", function() {
+    logger.info({
+      message: "Received signal SIGTERM, exiting",
+      pid: process.pid
+    });
+
+    cluster.removeListener("exit", restartWorker);
+    for (let pid in cluster.workers)
+      cluster.workers[pid].kill();
+  });
+
+  cluster.setupMaster({
+    exec: "lib/server",
+    args: [service]
+  });
+
+  let workerCount = config.get(`${service.toUpperCase()}_WORKER_PROCESSES`);
+
+  if (!workerCount)
+    workerCount = Math.max(2, require("os").cpus().length);
+
+  for (let i = 0; i < workerCount; ++ i)
+    cluster.fork();
+}
+
+
+process.on("uncaughtException", function(err) {
+  logger.error({
+    message: "Uncaught exception in master process, exiting",
+    exception: err
+  });
+  cluster.removeListener("exit", restartWorker);
+  for (let pid in cluster.workers)
+    cluster.workers[pid].kill();
+
+  logger.close();
+});
+
+
+exports.start = start;

lib/common.coffee

@@ -1,23 +1,20 @@
 ###
-# Copyright 2013 Fanoos Telecom
+# Copyright 2013-2017  Zaid Abdulla
 #
-# Permission is hereby granted, free of charge, to any person obtaining a copy
+# This file is part of GenieACS.
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
 #
-# The above copyright notice and this permission notice shall be included in
+# GenieACS is free software: you can redistribute it and/or modify
-# all copies or substantial portions of the Software.
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
 #
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# GenieACS is distributed in the hope that it will be useful,
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# GNU Affero General Public License for more details.
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+#
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# You should have received a copy of the GNU Affero General Public License
-# THE SOFTWARE.
+# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
 ###
 
 buffer = require 'buffer'
@@ -38,21 +35,6 @@ typeOf = (obj) ->
   Object.prototype.toString.call(obj)
 
 
-exports.endsWith = (str, suffix) ->
-  str.indexOf(suffix, str.length - suffix.length) isnt -1
-
-
-exports.startsWith = (str, prefix) ->
-  str.substring(0, prefix.length) == prefix
-
-
-exports.arrayToHash = (arr) ->
-  hash = {}
-  for i in arr
-    hash[i[0]] = i[1]
-  return hash
-
-
 exports.generateDeviceId = (deviceIdStruct) ->
   # Percent escaping function. Escapes everything except alphanumerics and underscore
   esc = (str) ->
@@ -70,34 +52,6 @@ exports.generateDeviceId = (deviceIdStruct) ->
   return "#{esc(deviceIdStruct['OUI'])}-#{esc(deviceIdStruct['SerialNumber'])}"
 
 
-exports.parseDeviceId = (deviceId) ->
-  parts = deviceId.split('-')
-  ret = {oui : querystring.unescape(parts[0])}
-  if parts.length == 3
-    ret.productClass = querystring.unescape(parts[1])
-    ret.serialNumber = querystring.unescape(parts[2])
-  else
-    ret.serialNumber = querystring.unescape(parts[1])
-  return ret
-
-
-exports.extend = (obj, mixin) ->
-  obj[name] = method for name, method of mixin
-  obj
-
-
-exports.flattenObject = (object) ->
-  newObj = {}
-  f = (obj, prefix) ->
-    for k, v of obj
-      if typeof(v) is 'object'
-        f(v, "#{prefix}#{k}.")
-      else
-        newObj["#{prefix}#{k}"] = v
-  f(object, '')
-  return newObj
-
-
 exports.getParamValueFromPath = (obj, path) ->
   pp = path.split('.')
   ref = obj
@@ -109,17 +63,173 @@ exports.getParamValueFromPath = (obj, path) ->
     return undefined
 
 
-exports.matchType = (src, dst) ->
+pathOverlap = (a, b, start) ->
-  switch typeOf(src)
+  if a.length == b.length
-    when STRING_TYPE
+    res = 3
-      String(dst)
+  else if a.length > b.length
-    when NUMBER_TYPE
+    res = 2
-      Number(dst)
-    when BOOLEAN_TYPE
-      v = String(dst).trim().toLowerCase()
-      v == 'true' or v == 'on' or v == 'yes' or v == '1'
   else
-      dst
+    res = 1
+
+  for i in [(start ? 0)...Math.min(a.length, b.length)] by 1
+    if a[i]?
+      if not b[i]?
+        res &= 2
+      else if a[i] != b[i]
+        return 0
+    else if b[i]?
+      res &= 1
+
+    return 0 if not res
+
+  return res
+
+
+parseAlias = (pattern, start, res) ->
+  aliases = []
+  i = start
+  while i < pattern.length and pattern[i] != ']'
+    alias = []
+    i = j = parsePath(pattern, i, alias) + 1
+
+    while pattern[j] not in [']', ',']
+      if pattern[j] == '"' and i == j
+        ++ j
+        while pattern[j] != '"' or pattern[j - 1] == '\\'
+          if ++ j >= pattern.length
+            throw new Error('Invalid alias expression')
+      if ++ j >= pattern.length
+        throw new Error('Invalid alias expression')
+    value = pattern[i...j].trim()
+    i = j
+    if value[0] == '"'
+      try
+        value = JSON.parse(value)
+      catch err
+        throw new Error('Invalid alias expression')
+    alias.push(value)
+    aliases.push(alias)
+    ++ i if pattern[i] == ','
+
+  # Need to sort to ensure identical expressions have idential string representation
+  srt = (a, b) ->
+    for j in [0...Math.min(a.length, b.length)] by 2
+      for k in [0...Math.min(a[j].length, b[j].length)] by 1
+        if Array.isArray(a[j][k])
+          if Array.isArray(b[j][k])
+            return srt(a[j][k], b[j][k])
+          else if not b[j][k]?
+            return -1
+          else
+            return 1
+        else if not a[j][k]?
+          if not b[j][k]?
+            return 0
+          else
+            return 1
+        else if not b[j][k]? or Array.isArray(b[j][k])
+            return -1
+        else if a[j][k] > b[j][k]
+          return 1
+        else if a[j][k] < b[j][k]
+          return -1
+
+      if a[j].length > b[j].length
+        return -1
+      else if a[j].length < b[j].length
+        return 1
+
+      if a[j + 1] > b[j + 1]
+        return -1
+      else if a[j + 1] < b[j + 1]
+        return 1
+
+    if a.length > b.length
+      return -1
+    else if a.length < b.length
+      return 1
+
+    return 0
+
+  aliases.sort(srt)
+  res.push([].concat.apply([], aliases))
+  return i
+
+
+parsePath = (pattern, start, res) ->
+  path = []
+  path.wildcard = 0
+  path.alias = 0
+  i = start ? 0
+
+  # Colon separator is needed for parseAlias
+  if i < pattern.length and pattern[i] != ':'
+    while true
+      if pattern[i] == '['
+        path.alias |= 1 << path.length
+        i = parseAlias(pattern, i + 1, path) + 1
+      else
+        j = i
+        while i < pattern.length and pattern[i] != ':' and pattern[i] != '.'
+          ++ i
+        n = pattern.slice(j, i)
+        path.wildcard |= 1 << path.length if n == '*'
+        path.push(n)
+
+      if i >= pattern.length or pattern[i] == ':'
+        break
+      else if pattern[i] != '.'
+        throw new Error('Invalid alias expression')
+
+      ++ i
+
+  Object.freeze(path)
+
+  if not res?
+    return path
+
+  res.push(path)
+  return i
+
+
+addPathMeta = (path) ->
+  return path if path.alias? or path.wildcard?
+
+  path.alias = 0
+  path.wildcard = 0
+
+  for p, i in path
+    if typeOf(p) is ARRAY_TYPE
+      path.alias |= 1 << i
+      for j in [0...p.length] by 2
+        addPathMeta(p[j])
+    else if p == '*'
+      path.wildcard |= 1 << i
+
+  Object.freeze(path)
+  return path
+
+
+hammingWeight = (flags) ->
+  flags -= ((flags >> 1) & 0x55555555)
+  flags = (((flags >> 2) & 0x33333333) + (flags & 0x33333333))
+  flags = (((flags >> 4) + flags) & 0x0f0f0f0f)
+  flags += (flags >> 8)
+  flags += (flags >> 16)
+  return flags & 0x0000003f
+
+
+# Source: http://stackoverflow.com/a/6969486
+escapeRegExp = (str) ->
+  return str.replace(/[\-\[\]\/\{\}\(\)\*\+\?\.\\\^\$\|]/g, "\\$&")
+
+
+# djb2 hashing algorithm adapted from http://www.cse.yorku.ca/~oz/hash.html
+djb2 = (str) ->
+  hash = 5381
+  for i in [0...str.length]
+    hash = ((hash << 5) + hash) ^ str.charCodeAt(i)
+  return hash
 
 
 exports.UNDEFINED_TYPE = UNDEFINED_TYPE
@@ -131,3 +241,9 @@ exports.OBJECT_TYPE = OBJECT_TYPE
 exports.REGEXP_TYPE = REGEXP_TYPE
 
 exports.typeOf = typeOf
+exports.pathOverlap = pathOverlap
+exports.parsePath = parsePath
+exports.addPathMeta = addPathMeta
+exports.hammingWeight = hammingWeight
+exports.escapeRegExp = escapeRegExp
+exports.djb2 = djb2

lib/config.coffee

@@ -1,5 +1,7 @@
 ###
-# Copyright 2013, 2014  Zaid Abdulla
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
 #
 # GenieACS is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as
@@ -19,48 +21,57 @@ path = require 'path'
 common = require './common'
 
 options = {
-  CONFIG_DIR : {type : 'string', default : 'config'},
+  CONFIG_DIR : {type : 'path', default : 'config'},
   MONGODB_CONNECTION_URL : {type : 'string', default : 'mongodb://127.0.0.1/genieacs'},
   REDIS_PORT : {type : 'int', default : 6379},
   REDIS_HOST : {type : 'string', default : '127.0.0.1'},
   REDIS_DB : {type : 'int', default : 0},
 
-  CWMP_WORKER_PROCESSES : {type : 'int', default : 4},
+  CWMP_WORKER_PROCESSES : {type : 'int', default : 0},
   CWMP_PORT : {type : 'int', default : 7547},
   CWMP_INTERFACE : {type : 'string', default : '0.0.0.0'},
   CWMP_SSL : {type : 'bool', default : false},
+  CWMP_LOG_FILE : {type: 'path', default : ''},
+  CWMP_ACCESS_LOG_FILE : {type : 'path', default : ''},
 
-  NBI_WORKER_PROCESSES : {type : 'int', default : 2},
+  NBI_WORKER_PROCESSES : {type : 'int', default : 0},
   NBI_PORT : {type : 'int', default : 7557},
   NBI_INTERFACE : {type : 'string', default : '0.0.0.0'},
   NBI_SSL : {type : 'bool', default : false},
+  NBI_LOG_FILE : {type: 'path', default : ''},
+  NBI_ACCESS_LOG_FILE : {type : 'path', default : ''},
 
-  FS_WORKER_PROCESSES : {type : 'int', default : 2},
+  FS_WORKER_PROCESSES : {type : 'int', default : 0},
   FS_PORT : {type : 'int', default : 7567},
   FS_INTERFACE : {type : 'string', default : '0.0.0.0'},
   FS_SSL : {type : 'bool', default : false},
-  FS_IP : {type : 'string', default : '192.168.0.1'},
+  FS_HOSTNAME : {type : 'string', default : 'acs.example.com'},
+  FS_LOG_FILE : {type: 'path', default : ''},
+  FS_ACCESS_LOG_FILE : {type : 'path', default : ''},
 
-  PRESETS_CACHE_DURATION : {type : 'int', default : 86400},
+  DOWNLOAD_TIMEOUT: {type : 'int', default : 3600},
-  LOG_INFORMS : {type : 'bool', default : true},
+  EXT_TIMEOUT: {type: 'int', default: 3000},
+  MAX_CACHE_TTL : {type : 'int', default : 86400},
   DEBUG : {type : 'bool', default : false},
   RETRY_DELAY : {type : 'int', default : 300},
-  IGNORE_XML_NAMESPACES : {type : 'bool', default : false},
   SESSION_TIMEOUT : {type : 'int', default : 30},
-  GET_PARAMETER_NAMES_DEPTH_THRESHOLD : {type : 'int', default : 0},
+  CONNECTION_REQUEST_TIMEOUT : {type : 'int', default: 2000},
-  TASK_PARAMETERS_BATCH_SIZE : {type : 'int', default : 32},
+  GPN_NEXT_LEVEL : {type : 'int', default : 0},
+  GPV_BATCH_SIZE : {type : 'int', default : 32},
+  MAX_DEPTH : {type: 'int', default : 16},
   COOKIES_PATH : {type : 'string'},
+  LOG_FORMAT : {type : 'string', default : 'simple'},
+  ACCESS_LOG_FORMAT : {type : 'string', default : ''},
 
-  # Libxml related configuration
+  # XML configuration
-  XML_PARSE_RECOVER : {type : 'bool'},
+  XML_RECOVER : {type : 'bool', default : false},
-  XML_PARSE_NOENT : {type : 'bool'},
+  XML_IGNORE_ENC : {type : 'bool', default : false},
-  XML_PARSE_NOBLANKS : {type : 'bool'},
+  XML_FORMAT : {type : 'bool', default : false},
-  XML_PARSE_NSCLEAN : {type : 'bool'},
+  XML_NO_DECL : {type : 'bool', default : false},
-  XML_PARSE_NOCDATA : {type : 'bool'},
+  XML_NO_EMPTY : {type : 'bool', default : false},
-  XML_PARSE_IGNORE_ENC : {type : 'bool'},
+  XML_IGNORE_NAMESPACE : {type : 'bool', default : false},
 
   # Should probably never be changed
-  PRESETS_TIME_PADDING : {type : 'int', default : 1},
   DEVICE_ONLINE_THRESHOLD : {type : 'int', default : 4000}
 }
 
@@ -70,6 +81,28 @@ allConfig = {}
 setConfig = (name, value, commandLineArgument) ->
   return true if allConfig[name]?
 
+  # For compatibility with v1.0
+  if name in ['PRESETS_CACHE_DURATION', 'presets-cache-duration']
+    setConfig('MAX_CACHE_TTL', value)
+
+  if name in ['GET_PARAMETER_NAMES_DEPTH_THRESHOLD', 'get-parameter-names-depth-threshold']
+    setConfig('GPN_NEXT_LEVEL', value)
+
+  if name in ['TASK_PARAMETERS_BATCH_SIZE', 'task-parameters-batch-size']
+    setConfig('GPV_BATCH_SIZE', value)
+
+  if name in ['IGNORE_XML_NAMESPACES', 'ignore-xml-namespaces']
+    setConfig('XML_IGNORE_NAMESPACE', value)
+
+  if name in ['XML_PARSE_IGNORE_ENC', 'xml-parse-ignore-enc']
+    setConfig('XML_IGNORE_ENC', value)
+
+  if name in ['XML_PARSE_RECOVER', 'xml-parse-recover']
+    setConfig('XML_RECOVER', value)
+
+  if name in ['FS_IP', 'fs-ip']
+    setConfig('FS_HOSTNAME', value)
+
   cast = (val, type) ->
     switch type
       when 'int'
@@ -78,6 +111,8 @@ setConfig = (name, value, commandLineArgument) ->
         String(val).trim().toLowerCase() in ['true', 'on', 'yes', '1']
       when 'string'
         String(val)
+      when 'path'
+        if val then path.resolve(val) else ''
       else
         null
 
@@ -91,30 +126,26 @@ setConfig = (name, value, commandLineArgument) ->
     if name == n
       _value = cast(value, optionDetails.type)
       n = optionName
-    else if common.startsWith(name, "#{n}-")
+    else if name.startsWith("#{n}-")
       _value = cast(value, optionDetails.type)
       n = "#{optionName}-#{name[optionName.length+1..]}"
 
     if _value?
       allConfig[n] = _value
       # Save as environmnet variable to pass on to any child processes
-      process.env[n] = _value
+      process.env["GENIEACS_#{n}"] = _value
       return true
 
   return false
 
 
 # Command line arguments
-exports.argv = []
 argv = process.argv[2..]
 while argv.length
   arg = argv.shift()
   if arg[0] == '-'
     v = argv.shift()
-    exports.argv[arg] = v
     setConfig(arg[2..], v, true)
-  else
-    exports.argv.push(arg)
 
 
 # Environment variable
@@ -124,18 +155,14 @@ for k, v of process.env
   setConfig(k, v)
 
 
-# Find config dir
+# Use default config dir if none defined
-if exports.argv['--config-dir']?
+setConfig('CONFIG_DIR', options['CONFIG_DIR'].default)
-  allConfig.CONFIG_DIR = exports.argv['--config-dir']
-else if process.env['GENIEACS_CONFIG_DIR']?
-  allConfig.CONFIG_DIR = process.env['GENIEACS_CONFIG_DIR']
-else
-  allConfig.CONFIG_DIR = options.CONFIG_DIR.default
-
 
 # Configuration file
 for k, v of require(path.resolve(allConfig.CONFIG_DIR, 'config'))
-  setConfig(k, v)
+  if not setConfig(k, v)
+    # Pass as environment variable to be accessable by extensions
+    process.env["GENIEACS_#{k}"] = v
 
 
 # Defaults
@@ -158,14 +185,8 @@ get = (option, deviceId) ->
   return allConfig[option]
 
 
-# load parameter configurations
-exports.PARAMETERS = require(path.resolve(allConfig.CONFIG_DIR, 'parameters'))
-
 # load authentication scripts
-exports.auth = require(path.resolve(allConfig.CONFIG_DIR, 'auth'))
+try
-
+  exports.auth = require(path.resolve(allConfig.CONFIG_DIR, 'auth'))
-exports.CUSTOM_COMMANDS = require(path.resolve(allConfig.CONFIG_DIR, 'custom_commands'))
 
 exports.get = get
-
-exports.allConfig = allConfig

lib/custom-commands.coffee

@@ -1,61 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-###
-
-path = require 'path'
-config = require './config'
-common = require './common'
-
-
-FILES = {}
-getFile = (filename) ->
-  if filename not of FILES
-    FILES[filename] = require(path.resolve(config.get('CONFIG_DIR'), "custom_commands/#{filename}"))
-
-  return FILES[filename]
-
-
-exports.execute = (deviceId, command, callback) ->
-  args = command.split(/\s+/)
-  f = args.shift()
-  c = args.shift()
-
-  file = getFile(f)
-  file[c](deviceId, args.join(' '), callback)
-
-
-exports.getFileCommands = (filename) ->
-  commands = []
-  f = getFile(filename)
-  for k,v of f
-    commands.push(k)
-  return commands
-
-
-exports.getDeviceCustomCommands = (deviceId) ->
-  commands = {}
-  for k,v of config.CUSTOM_COMMANDS
-    if eval(v).test(deviceId)
-      commands[k] = exports.getFileCommands(k)
-  return commands
-
-
-exports.getDeviceCustomCommandNames = (deviceId) ->
-  commands = []
-  for k,v of config.CUSTOM_COMMANDS
-    if eval(v).test(deviceId)
-      commands.push(k)
-  return commands

lib/db.coffee

@@ -1,5 +1,7 @@
 ###
-# Copyright 2013, 2014  Zaid Abdulla
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
 #
 # GenieACS is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as
@@ -18,41 +20,23 @@
 config = require './config'
 mongodb = require 'mongodb'
 redis = require('redis')
-parameters = require './parameters'
+common = require './common'
 
 redisClient = null
 tasksCollection = null
 devicesCollection = null
 presetsCollection = null
 objectsCollection = null
-
+provisionsCollection = null
-
+virtualParametersCollection = null
-# Only flush redis DB if aliases or custom commands config has changed
+faultsCollection = null
-flushRedis = (callback) ->
+filesCollection = null
-  redisClient.get('config_hash', (err, res) ->
+operationsCollection = null
-    return callback(err) if err
-
-    hash = require('crypto')
-      .createHash('md5')
-      .update(JSON.stringify(config.PARAMETERS))
-      .update(JSON.stringify(config.CUSTOM_COMMANDS))
-      .digest('hex')
-
-    if res == hash
-      return callback()
-
-    redisClient.flushdb((err) ->
-      return callback(err) if err
-      redisClient.set('config_hash', hash, (err, res) ->
-        return callback(err)
-      )
-    )
-  )
 
 
 connect = (callback) ->
-  callbackCounter = 6
+  callbackCounter = 10
-  mongodb.MongoClient.connect(config.get('MONGODB_CONNECTION_URL'), {db:{w:1},server:{autoReconnect:true}}, (err, db) ->
+  mongodb.MongoClient.connect(config.get('MONGODB_CONNECTION_URL'), (err, db) ->
     return callback(err) if err
     exports.mongoDb = db
     db.collection('tasks', (err, collection) ->
@@ -93,138 +77,587 @@ connect = (callback) ->
         return callback(err)
     )
 
-    exports.redisClient = redisClient = redis.createClient(config.get('REDIS_PORT'), config.get('REDIS_HOST'))
+    db.collection('provisions', (err, collection) ->
-    redisClient.select(config.get('REDIS_DB'), (err) ->
+      exports.provisionsCollection = provisionsCollection = collection
-      if err
+      if --callbackCounter == 0 or err
         callbackCounter = 0
         return callback(err)
+    )
 
-      flushRedis((err) ->
+    db.collection('virtualParameters', (err, collection) ->
+      exports.virtualParametersCollection = virtualParametersCollection = collection
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
+    )
+
+    db.collection('faults', (err, collection) ->
+      exports.faultsCollection = faultsCollection = collection
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
+    )
+
+    db.collection('operations', (err, collection) ->
+      exports.operationsCollection = operationsCollection = collection
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
+    )
+
+    exports.redisClient = redisClient = redis.createClient(config.get('REDIS_PORT'), config.get('REDIS_HOST'))
+    redisClient.select(config.get('REDIS_DB'), (err) ->
       if --callbackCounter == 0 or err
         callbackCounter = 0
         return callback(err)
     )
   )
-  )
 
 
 disconnect = () ->
-  exports.mongoDb.close()
+  exports.mongoDb?.close()
-  redisClient.quit()
+  redisClient?.quit()
 
 
-getTask = (taskId, callback) ->
+# Optimize projection by removing overlaps
-  tid = String(taskId)
+# This modifies given object and returns it
-  redisClient.get(tid, (err, res) ->
+optimizeProjection = (obj) ->
-    return callback(err) if err
+  keys = Object.keys(obj).sort()
-    if res?
+  return if keys.length <= 1
-      callback(err, JSON.parse(res))
+  i = 1
+  while i < keys.length
+    a = keys[i-1]
+    b = keys[i]
+    if b.startsWith(a)
+      if b.charAt(a.length) == '.' or b.charAt(a.length - 1) == '.'
+        delete obj[b]
+        keys.splice(i--, 1)
+    ++ i
+  return obj
+
+
+fetchDevice = (id, timestamp, patterns, callback) ->
+  MAX_DEPTH = config.get('MAX_DEPTH', id)
+
+  projection = {_id: 1}
+  projectionTree = {}
+  func = (path, pats, projTree) ->
+    children = {}
+    for pat in pats
+      fragment = pat[0][path.length] or '*'
+      load = 0
+      if fragment == '*'
+        projection[path.join('.')] = 1
+        return
+
+      projTree[fragment] ?= {}
+
+      if pat[1] & (1 << path.length)
+        projection[path.concat('_timestamp').join('.')] = 1
+        projection[path.concat([fragment, '_timestamp']).join('.')] = 1
+        projection[path.concat([fragment, '_value']).join('.')] = 1
+        projection[path.concat([fragment, '_type']).join('.')] = 1
+        projection[path.concat([fragment, '_object']).join('.')] = 1
+        projection[path.concat([fragment, '_instance']).join('.')] = 1
+        projection[path.concat([fragment, '_writable']).join('.')] = 1
+        projection[path.concat([fragment, '_orig']).join('.')] = 1
+
+      if pat[1] >> (path.length + 1)
+        children[fragment] ?= []
+        children[fragment].push(pat)
+
+    for k, v of children
+      func(path.concat(k), v, projTree[k])
+
+  func([], patterns, projectionTree)
+
+  delete projectionTree['DeviceID']
+  delete projectionTree['Events']
+  delete projectionTree['Tags']
+
+  if projection['']?
+    proj = {}
   else
-      tasksCollection.findOne({_id : mongodb.ObjectID(tid)}, (err, task) ->
+    proj = JSON.parse(JSON.stringify(projection))
-        callback(err, task)
-      )
-  )
 
+  for k, v of proj
+    if k == '' or k == 'Events' or k.startsWith('Events.')
+      if k.length < 7 or k.startsWith('Events.Inform.') or k == 'Events.Inform'
+        proj['_lastInform'] = 1
+      if k.length < 7 or k.startsWith('Events.1_BOOT.') or k == 'Events.1_BOOT'
+        proj['_lastBoot'] = 1
+      if k.length < 7 or k.startsWith('Events.0_BOOTSTRAP.') or k == 'Events.0_BOOTSTRAP'
+        proj['_lastBootstrap'] = 1
+      if k.length < 7 or k.startsWith('Events.Registered.') or k == 'Events.Registered'
+        proj['_registered'] = 1
+      delete proj[k] if k != ''
 
-getCached = (name, valueCallback, valueExpiry, callback) ->
+    if k == '' or k == 'DeviceID' or k.startsWith('DeviceID.')
-  redisClient.get(name, (err, res) ->
+      if k.length < 9 or k.startsWith('DeviceID.ID.') or k == 'DeviceID.ID'
-    return callback(err) if err
+        proj['_id'] = 1
-    return callback(null, JSON.parse(res)) if res?
+      if k.length < 9 or k.startsWith('DeviceID.Manufacturer.') or k == 'DeviceID.DeManufacturer'
+        proj['_deviceId._Manufacturer'] = 1
+      if k.length < 9 or k.startsWith('DeviceID.ProductClass.') or k == 'DeviceID.ProductClass'
+        proj['_deviceId._ProductClass'] = 1
+      if k.length < 9 or k.startsWith('DeviceID.OUI.') or k == 'DeviceID.OUI'
+        proj['_deviceId._OUI'] = 1
+      if k.length < 9 or k.startsWith('DeviceID.SerialNumber.') or k == 'DeviceID.SerialNumber'
+        proj['_deviceId._SerialNumber'] = 1
+      if k.length < 9
+        proj['_deviceId'] = 1
+      delete proj[k] if k != ''
 
-    lockAcquired = () ->
+    if k == '' or k == 'Tags' or k.startsWith('Tags.')
-      valueCallback((err, value) ->
+      proj['_tags'] = 1
-        return callback(err, value) if err
+      delete proj[k] if k != ''
-        redisClient.setex(name, valueExpiry, JSON.stringify(value), (err) ->
-          return callback(err, value) if err
-          redisClient.del("lock.#{name}", (err) ->
-            callback(err, value)
-          )
-        )
-      )
 
-    redisClient.setnx("lock.#{name}", Date.now() + 30000, (err, res) ->
+  optimizeProjection(proj)
-      return callback(err) if err
+
-      if res == 1
+  res = []
-        lockAcquired()
+  loaded = []
+
+  devicesCollection.findOne({'_id' : id}, proj, (err, device) ->
+    return callback(err) if err or not device?
+
+    storeParams = (obj, path, timestamp, descendantsFetched, projTree) ->
+      if descendantsFetched
+        thisFetched = true
       else
-        setTimeout(() ->
+        if projection[path.join('.')]
-          redisClient.get("lock.#{name}", (err, timestamp) ->
+          descendantsFetched = true
-            return callback(err) if err
+          loaded.push([path, ((1 << path.length) - 1) ^ ((1 << MAX_DEPTH) - 1)])
-            now = Date.now()
+        if projection[path.concat('_writable').join('.')]
-            if not timestamp?
+          loaded.push([path, 1 << (path.length - 1)])
-              getCached(name, valueCallback, valueExpiry, callback)
+          thisFetched = true
-            else if timestamp > now
+
-              setTimeout(() ->
+      if obj['_timestamp']?
-                getCached(name, valueCallback, valueExpiry, callback)
+        obj['_timestamp'] = +obj['_timestamp']
-              , 1000)
+
+      if thisFetched
+        attrs = {}
+
+        t = if obj['_timestamp'] > timestamp then obj['_timestamp'] else timestamp
+
+        if obj['_value']?
+          attrs.value = [obj['_timestamp'] or 1, [obj['_value'], obj['_type']]]
+          attrs.value[1][0] = +attrs.value[1][0] if obj['_type'] == 'xsd:dateTime'
+          obj['_object'] = false
+
+        if obj['_writable']?
+          attrs.writable = [timestamp, if obj['_writable'] then 1 else 0]
+
+        if obj['_object']? or obj['_instance']?
+          attrs.object = [t, if (obj['_object'] or obj['_instance']) then 1 else 0]
+
+        res.push([path, t, attrs])
+
+      for k, v of obj
+        if k[0] != '_'
+          kk = k
+          obj['_object'] = true
+          storeParams(v, path.concat(k), obj['_timestamp'] or 1, descendantsFetched, projTree?[k])
+          delete projTree[kk] if projTree
+
+      if obj['_object'] or obj['_instance']
+        if descendantsFetched
+          res.push([path.concat('*'), obj['_timestamp']]) if obj['_timestamp']
         else
-              redisClient.getset("lock.#{name}", now + 30000, (err, timestamp) ->
+          for k, v of projTree
-                return callback(err) if err
+            p = path.concat(k)
-                if timestamp > now
+            res.push([p, obj['_timestamp']]) if obj['_timestamp']
-                  setTimeout(() ->
+            loaded.push([p, ((1 << path.length) - 1) ^ ((1 << MAX_DEPTH) - 1)])
-                    getCached(name, valueCallback, valueExpiry, callback)
+
-                  , 1000)
+    for k, v of device
+      switch k
+        when '_lastInform'
+          res.push([['Events'], timestamp,
+            {object: [timestamp, 1], writable: [timestamp, 0]}])
+          res.push([['Events', 'Inform'], +v,
+            {object: [+v, 0], writable: [+v, 0], value: [+v, [+v, 'xsd:dateTime']]}])
+          delete device[k]
+        when '_lastBoot'
+          res.push([['Events'], timestamp,
+            {object: [timestamp, 1], writable: [timestamp, 0]}])
+          res.push([['Events', '1_BOOT'], +v,
+            {object: [+v, 0], writable: [+v, 0], value: [+v, [+v, 'xsd:dateTime']]}])
+          delete device[k]
+        when '_lastBootstrap'
+          res.push([['Events'], timestamp,
+            {object: [timestamp, 1], writable: [timestamp, 0]}])
+          res.push([['Events', '0_BOOTSTRAP'], +v,
+            {object: [+v, 0], writable: [+v, 0], value: [+v, [+v, 'xsd:dateTime']]}])
+          delete device[k]
+        when '_registered'
+          res.push([['Events'], timestamp,
+            {object: [timestamp, 1], writable: [timestamp, 0]}])
+          # Use current timestamp for registered event attribute timestamps
+          res.push([['Events', 'Registered'], timestamp,
+            {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [+v, 'xsd:dateTime']]}])
+          delete device[k]
+        when '_id'
+          res.push([['DeviceID'], timestamp,
+            {object: [timestamp, 1], writable: [timestamp, 0]}])
+          res.push([['DeviceID', 'ID'], timestamp,
+            {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v, 'xsd:string']]}])
+          delete device[k]
+        when '_tags'
+          res.push([['Tags'], timestamp,
+            {object: [timestamp, 1], writable: [timestamp, 0]}])
+          for t in v
+            t = t.replace(/[^a-zA-Z0-9\-]+/g, '_')
+            res.push([['Tags', t], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 1], value: [timestamp, [true, 'xsd:boolean']]}])
+          delete device[k]
+        when '_deviceId'
+          res.push([['DeviceID'], timestamp,
+            {object: [timestamp, 1], writable: [timestamp, 0]}])
+          if v['_Manufacturer']?
+            res.push([['DeviceID', 'Manufacturer'], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v['_Manufacturer'], 'xsd:string']]}])
+          if v['_OUI']?
+            res.push([['DeviceID', 'OUI'], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v['_OUI'], 'xsd:string']]}])
+          if v['_ProductClass']?
+            res.push([['DeviceID', 'ProductClass'], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v['_ProductClass'], 'xsd:string']]}])
+          if v['_SerialNumber']?
+            res.push([['DeviceID', 'SerialNumber'], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v['_SerialNumber'], 'xsd:string']]}])
+          delete device[k]
+
+    storeParams(device, [], 0, false, projectionTree)
+
+    return callback(null, res, loaded)
+  )
+
+
+saveDevice = (deviceId, deviceData, isNew, callback) ->
+  update = {'$set' : {}, '$unset' : {}, '$addToSet' : {}, '$pull' : {}}
+
+  iter = deviceData.timestamps.diff()
+  while diff = iter.next().value
+    continue if diff[0].wildcard != (1 << (diff[0].length - 1))
+    continue if diff[0][0] in ['Events', 'DeviceID', 'Tags']
+
+    if not diff[2]? and diff[1]?
+      update['$unset'][diff[0].slice(0, -1).concat('_timestamp').join('.')] = 1
+    else if diff[2] != diff[1]
+      parent = deviceData.paths.get(diff[0].slice(0, -1))
+      if parent and (parent.length == 0 or deviceData.attributes.has(parent))
+        update['$set'][diff[0].slice(0, -1).concat('_timestamp').join('.')] = new Date(diff[2])
+
+  iter = deviceData.attributes.diff()
+  while diff = iter.next().value
+    continue if diff[1] == diff[2]
+    path = diff[0]
+    switch path[0]
+      when 'Events'
+        if diff[0].length == 2 and diff[2]?.value?[1][0] != diff[1]?.value?[1][0]
+          if not diff[2]
+            switch path[1]
+              when 'Inform'
+                update['$unset']['_lastInform'] = 1
+              when '1_BOOT'
+                update['$unset']['_lastBoot'] = 1
+              when '0_BOOTSTRAP'
+                update['$unset']['_lastBootstrap'] = 1
+              when 'Registered'
+                update['$unset']['_registered'] = 1
           else
-                  lockAcquired()
+            t = new Date(diff[2].value[1][0])
-              )
+            switch path[1]
-          )
+              when 'Inform'
-        , 1000)
+                update['$set']['_lastInform'] = t
-    )
+              when '1_BOOT'
+                update['$set']['_lastBoot'] = t
+              when '0_BOOTSTRAP'
+                update['$set']['_lastBootstrap'] = t
+              when 'Registered'
+                update['$set']['_registered'] = t
+      when 'DeviceID'
+        if diff[2].value?[1]?[0] != diff[1]?.value?[1]?[0]
+          v = diff[2].value[1][0]
+          switch path[1]
+            when 'ID'
+              update['$set']['_id'] = v
+            when 'Manufacturer'
+              update['$set']['_deviceId._Manufacturer'] = v
+            when 'OUI'
+              update['$set']['_deviceId._OUI'] = v
+            when 'ProductClass'
+              update['$set']['_deviceId._ProductClass'] = v
+            when 'SerialNumber'
+              update['$set']['_deviceId._SerialNumber'] = v
+      when 'Tags'
+        if diff[2]?.value?[1]?[0] != diff[1]?.value?[1]?[0]
+          v = diff[2]?.value?[1][0]
+          if v?
+            update['$addToSet']['_tags'] ?= {'$each' : []}
+            update['$addToSet']['_tags']['$each'].push(diff[0][1])
+          else
+            update['$pull']['_tags'] ?= {'$in' : []}
+            update['$pull']['_tags']['$in'].push(diff[0][1])
+      else
+        if not diff[2]
+          update['$unset'][diff[0].join('.')] = 1
+          continue
+
+        for attrName of diff[2]
+          if diff[2][attrName][1]? or diff[1]?[attrName]?[1]?
+            switch attrName
+              when 'value'
+                if not diff[1] or diff[2].value[1][0] != diff[1].value?[1][0]
+                  if diff[2].value[1][1] == 'xsd:dateTime'
+                    update['$set'][path.concat('_value').join('.')] = new Date(diff[2].value[1][0])
+                  else
+                    update['$set'][path.concat('_value').join('.')] = diff[2].value[1][0]
+                if not diff[1] or diff[2].value[1][1] != diff[1].value?[1][1]
+                  update['$set'][path.concat('_type').join('.')] = diff[2].value[1][1]
+                if not diff[1] or diff[2].value[0] != diff[1].value?[0]
+                  update['$set'][path.concat('_timestamp').join('.')] = new Date(diff[2].value[0])
+              when 'object'
+                if not diff[1]?.object or diff[2].object[1] != diff[1].object?[1]
+                  update['$set'][path.concat('_object').join('.')] = !!diff[2].object[1]
+              when 'writable'
+                if not diff[1]?.writable or diff[2].writable[1] != diff[1].writable?[1]
+                  update['$set'][path.concat('_writable').join('.')] = !!diff[2].writable[1]
+
+        if diff[1]
+          for attrName of diff[1]
+            if attrName not of diff[2]
+              update['$unset'][path.concat("_#{attrName}").join('.')] = 1
+              if attrName is 'value'
+                update['$unset'][path.concat('_type').join('.')] = 1
+                update['$unset'][path.concat('_timestamp').join('.')] = 1
+
+  # Remove empty keys
+  for k of update
+    if k == '$addToSet'
+      for kk of update[k]
+        delete update[k][kk] if update[k][kk]['$each'].length == 0
+    else if k == '$pull'
+      for kk of update[k]
+        delete update[k][kk] if update[k][kk]['$in'].length == 0
+    delete update[k] if Object.keys(update[k]).length == 0
+
+  return callback() if Object.keys(update).length == 0
+
+  if (update['$unset']?)
+    optimizeProjection(update['$unset'])
+
+  devicesCollection.update({'_id' : deviceId}, update, {upsert: isNew}, (err, result) ->
+    if not err and result.result.n != 1
+      return callback(new Error("Device #{deviceId} not found in database"))
+
+    return callback(err)
   )
 
 
-getAliases = (callback) ->
+getDueTasksAndFaultsAndOperations = (deviceId, timestamp, callback) ->
-  getCached('aliases', parameters.compileAliases, config.get('PRESETS_CACHE_DURATION'), (err, res) ->
+  redisClient.mget("#{deviceId}_tasks", "#{deviceId}_faults", "#{deviceId}_operations", (err, res) ->
-    throw err if err
-    callback(res)
-  )
-
-
-getPresetsObjectsAliases = (callback) ->
-  redisClient.mget('presets', 'objects', 'aliases', (err, res) ->
-    presets = JSON.parse(res[0])
-    objects = JSON.parse(res[1])
-    aliases = JSON.parse(res[2])
-
-    if presets and objects and aliases
-      return callback(presets, objects, aliases)
-
-    if not presets
-      getCached('presets', (callback) ->
-        presetsCollection.find().toArray((err, res) ->
-          callback(err, res)
-        )
-      , config.get('PRESETS_CACHE_DURATION'), (err, res) ->
-        throw err if err
-        presets = res
-        callback(presets, objects, aliases) if objects and aliases
-      )
-
-    if not objects
-      getCached('objects', (callback) ->
-        objectsCollection.find().toArray((err, res) ->
     return callback(err) if err
-          objs = {}
+
-          objs[r._id] = r for r in res
+    if res[0]
-          callback(null, objs)
+      tasks = JSON.parse(res[0])
-        )
+
-      , config.get('PRESETS_CACHE_DURATION'), (err, res) ->
+    if res[1]?
-        throw err if err
+      faults = JSON.parse(res[1])
-        objects = res
+
-        callback(presets, objects, aliases) if presets and aliases
+    if res[2]?
+      operations = JSON.parse(res[2])
+
+    if tasks? and faults? and operations?
+      return callback(null, tasks, faults, operations)
+
+    MAX_CACHE_TTL = config.get('MAX_CACHE_TTL', deviceId)
+
+    if not faults?
+      getFaults(deviceId, (err, flts) ->
+        if err
+          callback?(err)
+          return callback = null
+
+        faults = flts
+        if tasks? and operations?
+          return callback(null, tasks, faults, operations)
       )
 
-    if not aliases
+    if not tasks?
-      getCached('aliases', parameters.compileAliases, config.get('PRESETS_CACHE_DURATION'), (err, res) ->
+      getDueTasks(deviceId, timestamp, (err, dueTasks, nextTimestamp) ->
-        throw err if err
+        if err
-        aliases = res
+          callback?(err)
-        callback(presets, objects, aliases) if presets and objects
+          return callback = null
+
+        if nextTimestamp?
+          exp = Math.min(0, Math.trunc((nextTimestamp - Date.now()) / 1000))
+        else
+          exp = MAX_CACHE_TTL
+
+        redisClient.setex("#{deviceId}_tasks", exp, JSON.stringify(dueTasks), (err) ->
+          if err
+            callback?(err)
+            return callback = null
+
+          tasks = dueTasks
+          if faults? and operations?
+            return callback(null, tasks, faults, operations)
+        )
+      )
+
+    if not operations?
+      getOperations(deviceId, (err, _operations) ->
+        if err
+          callback?(err)
+          return callback = null
+
+        operations = _operations
+        if tasks? and faults?
+          return callback(null, tasks, faults, operations)
+      )
+
+  )
+
+
+getFaults = (deviceId, callback) ->
+  faultsCollection.find({'_id' : {'$regex' : "^#{common.escapeRegExp(deviceId)}\\:"}}).toArray((err, res) ->
+    return callback(err) if err
+
+    faults = {}
+    for r in res
+      channel = r._id.slice(deviceId.length + 1)
+      delete r._id
+      delete r.channel
+      delete r.device
+      r.timestamp = +r.timestamp
+      r.provisions = JSON.parse(r.provisions)
+      faults[channel] = r
+
+    MAX_CACHE_TTL = config.get('MAX_CACHE_TTL', deviceId)
+
+    redisClient.setex("#{deviceId}_faults", MAX_CACHE_TTL, JSON.stringify(faults), (err) ->
+      callback(err, faults)
     )
   )
 
 
-exports.getTask = getTask
+saveFault = (deviceId, channel, fault, callback) ->
-exports.getPresetsObjectsAliases = getPresetsObjectsAliases
+  fault._id = "#{deviceId}:#{channel}"
-exports.getAliases = getAliases
+  fault.device = deviceId
+  fault.channel = channel
+  fault.timestamp = new Date(fault.timestamp)
+  fault.provisions = JSON.stringify(fault.provisions)
+  faultsCollection.save(fault, (err) ->
+    return callback(err) if err
+    redisClient.del("#{deviceId}_faults", callback)
+  )
+
+
+deleteFault = (deviceId, channel, callback) ->
+  faultsCollection.remove({_id: "#{deviceId}:#{channel}"}, (err) ->
+    return callback(err) if err
+    redisClient.del("#{deviceId}_faults", callback)
+  )
+
+
+
+getDueTasks = (deviceId, timestamp, callback) ->
+  cur = tasksCollection.find({'device' : deviceId}).sort(['timestamp'])
+
+  tasks = []
+
+  cur.nextObject(f = (err, task) ->
+    return callback(err) if err
+
+    if not task?
+      return callback(null, tasks, null)
+
+    task.timestamp = +task.timestamp if task.timestamp?
+    task.expiry = +task.expiry if task.expiry?
+
+    if task.timestamp >= timestamp
+      return callback(null, tasks, +task.timestamp)
+
+    task._id = String(task._id)
+    tasks.push(task)
+
+    # For API compatibility
+    if task.name is 'download' and task.file?
+      if mongodb.ObjectID.isValid(task.file)
+        q = {_id: {'$in' : [task.file, new mongodb.ObjectID(task.file)]}}
+      else
+        q = {_id: task.file}
+
+      filesCollection.find(q).toArray((err, res) ->
+        return callback(err) if err
+        if res[0]?
+          task.fileType ?= res[0].metadata.fileType
+          task.fileName ?= res[0].filename or res[0]._id.toString()
+        cur.nextObject(f)
+      )
+    else
+      cur.nextObject(f)
+  )
+
+
+clearTasks = (deviceId, taskIds, callback) ->
+  if not taskIds?.length
+    return callback()
+
+  tasksCollection.remove({'_id' : {'$in' : (new mongodb.ObjectID(id) for id in taskIds)}}, (err) ->
+    return callback(err) if err
+    redisClient.del("#{deviceId}_tasks", callback)
+  )
+
+
+getOperations = (deviceId, callback) ->
+  operationsCollection.find({'_id' : {'$regex' : "^#{common.escapeRegExp(deviceId)}\\:"}}).toArray((err, res) ->
+    return callback(err) if err
+
+    operations = {}
+    for r in res
+      commandKey = r._id.slice(deviceId.length + 1)
+      delete r._id
+      r.timestamp = +r.timestamp
+      r.args = JSON.parse(r.args) if r.args
+      r.provisions = JSON.parse(r.provisions)
+      r.retries = JSON.parse(r.retries)
+      operations[commandKey] = r
+
+    MAX_CACHE_TTL = config.get('MAX_CACHE_TTL', deviceId)
+
+    redisClient.setex("#{deviceId}_operations", MAX_CACHE_TTL, JSON.stringify(operations), (err) ->
+      callback(err, operations)
+    )
+  )
+
+
+saveOperation = (deviceId, commandKey, operation, callback) ->
+  operation._id = "#{deviceId}:#{commandKey}"
+  operation.timestamp = new Date(operation.timestamp)
+  operation.provisions = JSON.stringify(operation.provisions)
+  operation.retries = JSON.stringify(operation.retries)
+  operation.args = JSON.stringify(operation.args)
+  operationsCollection.save(operation, (err) ->
+    return callback(err) if err
+    redisClient.del("#{deviceId}_operations", callback)
+  )
+
+
+deleteOperation = (deviceId, commandKey, callback) ->
+  operationsCollection.remove({_id: "#{deviceId}:#{commandKey}"}, (err) ->
+    return callback(err) if err
+    redisClient.del("#{deviceId}_operations", callback)
+  )
+
+
 exports.connect = connect
 exports.disconnect = disconnect
+exports.fetchDevice = fetchDevice
+exports.saveDevice = saveDevice
+exports.getDueTasksAndFaultsAndOperations = getDueTasksAndFaultsAndOperations
+exports.getFaults = getFaults
+exports.saveFault = saveFault
+exports.deleteFault = deleteFault
+exports.clearTasks = clearTasks
+exports.saveOperation = saveOperation
+exports.deleteOperation = deleteOperation

lib/default-provisions.js

@@ -0,0 +1,116 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const config = require("./config");
+const common = require("./common");
+const device = require("./device");
+
+const MAX_DEPTH = config.get("MAX_DEPTH");
+
+
+function refresh(sessionContext, provision, declarations, startRevision, endRevision) {
+  let path = common.parsePath(provision[1]).slice();
+  let l = path.length;
+  path.length = MAX_DEPTH;
+  path.fill("*", l);
+  let every = 1000 * (provision[2] || 1);
+  let offset = Math.abs(common.djb2(sessionContext.deviceId)) % 86400000;
+  let t = Math.trunc((sessionContext.timestamp + offset) / every) * every - offset;
+
+  for (let i = l; i < path.length; ++ i)
+    declarations.push([path.slice(0, i), t, {object: 1, writable: 1, value: t}]);
+
+  return true;
+}
+
+
+function value(sessionContext, provision, declarations, startRevision, endRevision) {
+  let path = common.parsePath(provision[1]);
+  declarations.push([path, 1, {value: 1}, null, {value: [provision[2]]}]);
+
+  return true;
+}
+
+
+function tag(sessionContext, provision, declarations, startRevision, endRevision) {
+  let path = ["Tags", provision[1]];
+  declarations.push([path, 1, {value: 1}, null, {value: [provision[2]]}]);
+
+  return true;
+}
+
+
+function reboot(sessionContext, provision, declarations, startRevision, endRevision) {
+  declarations.push([["Reboot"], 1, {value: 1}, null, {value: [sessionContext.timestamp]}]);
+
+  return true;
+}
+
+
+function reset(sessionContext, provision, declarations, startRevision, endRevision) {
+  declarations.push([["FactoryReset"], 1, {value: 1}, null, {value: [sessionContext.timestamp]}]);
+
+  return true;
+}
+
+
+function download(sessionContext, provision, declarations, startRevision, endRevision) {
+  let alias = ["FileType", provision[1] || "", "FileName", provision[2] || "",
+    "TargetFileName", provision[3] || ""];
+
+  declarations.push([["Downloads", alias], 1, {}, 1]);
+  declarations.push([["Downloads", alias, "Download"], 1, {value: 1}, null,
+    {value: [sessionContext.timestamp]}]);
+
+  return true;
+}
+
+
+function instances(sessionContext, provision, declarations, startRevision, endRevision) {
+  let count = Number(provision[2]);
+
+  if (Number.isNaN(count))
+    return true;
+
+  let path = common.parsePath(provision[1]);
+
+  if (provision[2][0] === "+" || provision[2][0] === "-") {
+    declarations.push([path, 1]);
+
+    if (endRevision === startRevision)
+      return false;
+
+    let unpacked = device.unpack(sessionContext.deviceData, path, startRevision + 1);
+    count = Math.max(0, unpacked.length + count);
+  }
+
+  declarations.push([path, 1, null, count]);
+
+  return true;
+}
+
+
+exports.refresh = refresh;
+exports.value = value;
+exports.tag = tag;
+exports.reboot = reboot;
+exports.reset = reset;
+exports.download = download;
+exports.instances = instances;

lib/device.js

@@ -0,0 +1,380 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const common = require('./common');
+
+
+const CHANGE_FLAGS = {
+  object: 4,
+  writable: 16,
+  value: 64
+};
+
+
+function sanitizeParameterValue(parameterValue) {
+  if (parameterValue[0] != null) {
+    switch (parameterValue[1]) {
+      case 'xsd:boolean':
+        if (typeof parameterValue[0] !== 'boolean') {
+          parameterValue[0] = !!JSON.parse(parameterValue[0]);
+        }
+        break;
+      case 'xsd:string':
+        if (typeof parameterValue[0] !== 'string') {
+          parameterValue[0] = '' + parameterValue[0];
+        }
+        break;
+      case 'xsd:int':
+      case 'xsd:unsignedInt':
+        if (typeof parameterValue[0] !== 'number') {
+          parameterValue[0] = +parameterValue[0];
+        }
+        break;
+      case 'xsd:dateTime':
+        if (typeof parameterValue[0] !== 'number') {
+          if (parameterValue[0].getTime() != null)
+            parameterValue[0] = parameterValue[0].getTime();
+          else if (isNaN(parameterValue[0]))
+            parameterValue[0] = Date.parse(parameterValue[0]);
+          else
+            parameterValue[0] = +parameterValue[0];
+        }
+        break;
+      default:
+        if (parameterValue[1] != null)
+          throw new Error(`Parameter value type "${parameterValue[1]}" not recognized.`);
+
+        parameterValue[0] = JSON.parse(JSON.stringify(parameterValue[0]));
+    }
+  }
+  return parameterValue;
+}
+
+
+function getAliasDeclarations(path, timestamp) {
+  var decs = [];
+
+  function recursive(pattern, prefix) {
+    var pattern = prefix.concat(pattern);
+
+    for (let i = prefix.length; i < pattern.length; ++ i) {
+      if (Array.isArray(pattern[i])) {
+        let pat = pattern[i];
+        pattern[i] = '*';
+        for (let j = 0; j < pat.length; j += 2)
+          recursive(pat[j], pattern.slice(0, i + 1));
+      }
+    }
+
+    if (prefix.length)
+      decs.push([pattern, timestamp, {value: timestamp}]);
+    else
+      decs.push([pattern, timestamp]);
+  }
+
+  recursive(path, []);
+
+  return decs;
+}
+
+
+function unpack(deviceData, path, revision) {
+  let allMatches = [];
+  if (!path.alias) {
+    for (let p of deviceData.paths.find(path, false, true))
+      if (deviceData.attributes.has(p, revision))
+        allMatches.push(p);
+  }
+  else {
+    let wildcardPath = path.slice();
+    wildcardPath.wildcard = path.wildcard;
+    for (let i = 0; i < wildcardPath.length; ++ i)
+      if (Array.isArray(wildcardPath[i])) {
+        wildcardPath[i] = '*';
+        wildcardPath.wildcard |= 1 << i;
+      }
+
+    for (let p of deviceData.paths.find(wildcardPath, false, true))
+      if (deviceData.attributes.has(p, revision))
+        allMatches.push(p);
+
+    for (let i = path.length - 1; i >= 0; -- i) {
+      if (Array.isArray(path[i])) {
+        for (let j = 0; j < path[i].length; j += 2) {
+          let p = wildcardPath.slice(0, i + 1).concat(path[i][j]);
+          p.alias = (wildcardPath.alias & ((1 << (i + 1)) - 1)) + path[i][j].alias << (i + 1);
+          p.wildcard = (wildcardPath.wildcard & ((1 << (i + 1)) - 1)) + path[i][j].wildcard << (i + 1);
+          let unpacked = unpack(deviceData, p, revision);
+          let filtered = [];
+          for (let up of unpacked) {
+            let attributes = deviceData.attributes.get(up, revision);
+            if (attributes && attributes.value && attributes.value[1] &&
+                sanitizeParameterValue([path[i][j + 1], attributes.value[1][1]])[0] === attributes.value[1][0]) {
+              for (let m = 0; m < allMatches.length; ++ m) {
+                let k;
+                for (k = i; k >= 0; -- k)
+                  if (allMatches[m][k] != up[k])
+                    break
+
+                if (k < 0) {
+                  filtered.push(allMatches[m]);
+                  allMatches[m] = [];
+                }
+              }
+            }
+          }
+          allMatches = filtered;
+        }
+      }
+    }
+  }
+
+  allMatches.sort(function(a, b) {
+    for (let i = 0; i < a.length; ++ i) {
+      if (a === b)
+        continue;
+
+      // Use numeric sorting for numbers
+      let ia = parseInt(a), ib = parseInt(b);
+      if (ia.toString() === a && ib.toString() === b) {
+        a = ia;
+        b = ib;
+      }
+
+      if (a < b)
+        return -1;
+      else
+        return 1;
+    }
+    return 0;
+  });
+
+  return allMatches;
+}
+
+
+function clear(deviceData, path, timestamp, attributes) {
+  for (let p of deviceData.paths.find(path, true, true, timestamp ? 99 : path.length)) {
+    for (let k in deviceData.trackers.get(p))
+      deviceData.changes.add(k);
+
+    let t = deviceData.timestamps.get(p);
+    if (!t)
+      continue;
+
+    if (!(timestamp <= t)) {
+      deviceData.timestamps.delete(p);
+      deviceData.attributes.delete(p);
+    } else if (t && attributes) {
+      let currentAttributes = deviceData.attributes.get(p);
+      if (currentAttributes) {
+        let newAttrs;
+        for (let attrName in attributes) {
+          if (attrName in currentAttributes &&
+              attributes[attrName][0] > currentAttributes[attrName][0]) {
+            if (!newAttrs)
+              newAttrs = JSON.parse(JSON.stringify(currentAttributes));
+            delete newAttrs[attrName];
+          }
+        }
+
+        if (newAttrs)
+          deviceData.attributes.set(p, newAttrs);
+      }
+    }
+  }
+}
+
+
+function compareEquality(a, b) {
+  var t = typeof a;
+  if (a === null ||
+      a === undefined ||
+      t === 'number' ||
+      t === 'boolean' ||
+      t === 'string' ||
+      t === 'symbol')
+    return a === b;
+
+  return JSON.stringify(a) === JSON.stringify(b);
+}
+
+
+function set(deviceData, path, timestamp, attributes, toClear) {
+  path = deviceData.paths.add(path);
+
+  let currentTimestamp = deviceData.timestamps.get(path);
+
+  let currentAttributes;
+
+  if (path.wildcard)
+    attributes = undefined;
+  else
+    currentAttributes = deviceData.attributes.get(path);
+
+  if (attributes) {
+    if (attributes.value && attributes.value[1]
+        && attributes.value[0] >= (attributes.object ? attributes.object[0] : 0))
+      attributes.object = [attributes.value[0], 0];
+
+    if (attributes.object && attributes.object[1]
+      && attributes.object[0] >= (attributes.value ? attributes.value[0] : 0))
+      attributes.value = [attributes.object[0]];
+
+    if (currentAttributes) {
+      let changeFlags = 0;
+
+      for (let attrName in attributes) {
+        timestamp = Math.max(timestamp, attributes[attrName][0]);
+        if (attrName in currentAttributes) {
+          if (attributes[attrName][0] > currentAttributes[attrName][0]) {
+            changeFlags |= CHANGE_FLAGS[attrName];
+            if (compareEquality(attributes[attrName][1], currentAttributes[attrName][1]))
+              changeFlags |= CHANGE_FLAGS[attrName] << 1;
+          }
+          else
+            delete attributes[attrName];
+        }
+        else {
+          changeFlags |= CHANGE_FLAGS[attrName];
+          changeFlags |= CHANGE_FLAGS[attrName] << 1;
+        }
+      }
+
+      if (Object.keys(attributes).length) {
+        if (attributes.object && currentAttributes.object &&
+            attributes.object[1] !== currentAttributes.object[1]) {
+          toClear = toClear || [];
+          toClear.push([path.concat('*'), attributes.object[0]]);
+        }
+        for (let attrName in currentAttributes)
+          if (!(attrName in attributes))
+            attributes[attrName] = currentAttributes[attrName];
+
+        deviceData.attributes.set(path, attributes);
+      }
+
+      if (timestamp > currentTimestamp) {
+        changeFlags |= 1;
+        deviceData.timestamps.set(path, timestamp);
+
+        if (path.length > 1)
+          toClear = set(deviceData, path.slice(0, path.length - 1), timestamp,
+            {object: [timestamp, 1]}, toClear);
+      }
+
+      if (changeFlags)
+        for (let sup of deviceData.paths.find(path, true, false)) {
+          let c = deviceData.trackers.get(sup);
+          for (let k in c)
+            if (c[k] & changeFlags) {
+              deviceData.changes.add(k);
+            }
+        }
+    }
+    else if (!(timestamp <= currentTimestamp)) {
+      deviceData.timestamps.set(path, timestamp);
+      deviceData.attributes.set(path, attributes);
+
+      if (path.length > 1)
+        toClear = set(deviceData, path.slice(0, path.length - 1), timestamp,
+          {object: [timestamp, 1]}, toClear);
+
+      toClear = toClear || [];
+      toClear.push([path, timestamp]);
+    }
+  }
+  else if (!(timestamp <= currentTimestamp)) {
+    deviceData.timestamps.set(path, timestamp);
+
+    if (currentAttributes) {
+      deviceData.attributes.delete(path);
+      toClear = toClear || [];
+      toClear.push([path, timestamp]);
+    }
+
+    if (path.wildcard)
+      for (let sub of deviceData.paths.find(path, false, true))
+        if (timestamp > deviceData.timestamps.get(sub)) {
+          deviceData.timestamps.delete(sub);
+          if (deviceData.attributes.delete(sub)) {
+            toClear = toClear || [];
+            toClear.push([sub, timestamp]);
+          }
+        }
+  }
+
+  return toClear;
+}
+
+
+function track(deviceData, path, marker, timestamp, attributeTimestamps, exist, attributeValues) {
+  path = deviceData.paths.add(path);
+  let f = 0;
+
+  if (timestamp)
+    f |= 1;
+
+  if (exist)
+    f |= 2;
+
+  if (attributeTimestamps)
+    for (let attrName of attributeTimestamps)
+      f |= CHANGE_FLAGS[attrName];
+
+  if (attributeValues)
+    for (let attrName of attributeValues)
+      f |= CHANGE_FLAGS[attrName] << 1;
+
+  let cur = deviceData.trackers.get(path);
+  if (!cur) {
+    cur = {};
+    deviceData.trackers.set(path, cur);
+  }
+
+  cur[marker] |= f;
+}
+
+
+function clearTrackers(deviceData, tracker) {
+  if (Array.isArray(tracker)) {
+    deviceData.trackers.forEach(function(v, k) {
+      for (let t of tracker)
+        delete v[t];
+    });
+    for (let t of tracker)
+      deviceData.changes.delete(t);
+  }
+  else {
+    deviceData.trackers.forEach(function(v, k) {
+      delete v[tracker];
+    });
+    deviceData.changes.delete(tracker);
+  }
+}
+
+
+exports.sanitizeParameterValue = sanitizeParameterValue;
+exports.getAliasDeclarations = getAliasDeclarations;
+exports.unpack = unpack;
+exports.set = set;
+exports.clear = clear;
+exports.track = track;
+exports.clearTrackers = clearTrackers;

lib/extension-wrapper.js

@@ -0,0 +1,64 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const jobs = new Set();
+const fileName = process.argv[2];
+let script;
+
+function errorToFault(err) {
+  if (!err)
+    return null;
+
+  return {
+    code: `ext.${err.name}`,
+    message: err.message,
+    detail: {
+      name: err.name,
+      message: err.message,
+      stack: err.stack
+    }
+  }
+}
+
+process.on("uncaughtException", function(err) {
+  let fault = errorToFault(err);
+  jobs.forEach(function(jobId) {
+    process.send([jobId, fault]);
+  });
+  jobs.clear();
+  process.disconnect();
+});
+
+process.on("message", function(message) {
+  jobs.add(message[0]);
+
+  if (!script) {
+    let cwd = process.env["GENIEACS_CONFIG_DIR"] + "/ext";
+    process.chdir(cwd);
+    script = require(`${cwd}/${fileName}`);
+  }
+
+  script[message[1][0]](message[1].slice(1), function(err, res) {
+    if (!jobs.delete(message[0]))
+      return;
+
+    process.send([message[0], errorToFault(err), res]);
+  });
+});

lib/extensions.js

@@ -0,0 +1,100 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const childProcess = require("child_process");
+const crypto = require ("crypto");
+
+const config = require("./config");
+
+const TIMEOUT = config.get('EXT_TIMEOUT');
+
+const processes = {};
+const jobs = {};
+
+function messageHandler(message) {
+  let func = jobs[message[0]];
+  if (func) {
+    delete jobs[message[0]];
+    func(null, message[1], message[2]);
+  }
+}
+
+
+function run(args, callback) {
+  let scriptName = args[0];
+
+  if (!processes[scriptName]) {
+    let p = childProcess.fork('lib/extension-wrapper', [scriptName]);
+
+    p.on('error', function() {
+      kill(processes[scriptName]);
+      delete processes[scriptName];
+    });
+
+    p.on('disconnect', function() {
+      kill(processes[scriptName]);
+      delete processes[scriptName];
+    });
+
+    p.on('message', messageHandler);
+    processes[scriptName] = p;
+  }
+
+  let id = crypto.randomBytes(8).toString('hex');
+  jobs[id] = callback;
+  setTimeout(function() {
+    if (id in jobs) {
+      delete jobs[id];
+      return callback(null, {code: "timeout", message: "Extension timed out"});
+    }
+  }, TIMEOUT);
+
+  return processes[scriptName].send([id, args.slice(1)]);
+};
+
+
+function kill(process) {
+  if (process.signalCode !== null || process.exitCode !== null)
+    return;
+
+  let timeToKill = Date.now() + 5000;
+
+  process.kill();
+
+  let t = setInterval(function() {
+    if (process.signalCode !== null || process.exitCode !== null) {
+      clearInterval(t);
+    }
+    else if (Date.now() > timeToKill) {
+      process.kill('SIGKILL');
+      clearInterval(t);
+    }
+  }, 100);
+}
+
+
+function killAll() {
+  for (let n in processes)
+    kill(processes[n]);
+}
+
+
+exports.run = run;
+exports.killAll = killAll;

lib/instance-set.js

@@ -0,0 +1,138 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+
+class InstanceSet {
+
+  constructor() {
+    this.set = new Set();
+  }
+
+  add(instance) {
+    this.set.add(instance);
+  }
+
+  delete(instance) {
+    this.set.delete(instance);
+  }
+
+  superset(instance) {
+    let res = [];
+    for (let inst of this.set) {
+      let match = true;
+      for (let k in instance) {
+        if (inst[k] !== instance[k]) {
+          match = false;
+          break;
+        }
+      }
+
+      if (match)
+        res.push(inst);
+    }
+
+    res.sort(function(a, b) {
+      let keysA = Object.keys(a);
+      let keysB = Object.keys(b);
+
+      if (keysA.length != keysB.length)
+        return keysB.length - keysA.length;
+
+      keysA.sort();
+      keysB.sort();
+
+      for (let i = 0; i < keysA.length; ++ i) {
+        if (keysA[i] > keysB[i])
+          return 1;
+        else if (keysA[i] < keysB[i])
+          return -1;
+        else if (a[keysA[i]] > b[keysB[i]])
+          return 1;
+        else if (a[keysA[i]] < b[keysB[i]])
+          return -1;
+      }
+
+      return 0;
+    });
+
+    return res;
+  }
+
+  subset(instance) {
+    let res = [];
+
+    for (let inst of this.set) {
+      let match = true;
+      for (let k in inst) {
+        if (inst[k] !== instance[k]) {
+          match = false;
+          break;
+        }
+      }
+
+      if (match)
+        res.push(inst);
+    }
+
+    res.sort(function(a, b) {
+      let keysA = Object.keys(a);
+      let keysB = Object.keys(b);
+
+      if (keysA.length != keysB.length)
+        return keysA.length - keysB.length;
+
+      keysA.sort();
+      keysB.sort();
+
+      for (let i = 0; i < keysA.length; ++ i) {
+        if (keysA[i] > keysB[i])
+          return 1;
+        else if (keysA[i] < keysB[i])
+          return -1;
+        else if (a[keysA[i]] > b[keysB[i]])
+          return 1;
+        else if (a[keysA[i]] < b[keysB[i]])
+          return -1;
+      }
+
+      return 0;
+    });
+
+    return res;
+  }
+
+  forEach(callback) {
+    return this.set.forEach(callback);
+  }
+
+  values() {
+    return this.set.values();
+  }
+
+  clear() {
+    this.set.clear();
+  }
+
+  get size() {
+    return this.set.size;
+  }
+}
+
+module.exports = InstanceSet;

lib/logger.js

@@ -0,0 +1,319 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const fs = require("fs");
+const os = require("os");
+
+const config = require("./config");
+
+const REOPEN_EVERY = 60000;
+
+const LOG_FORMAT = config.get("LOG_FORMAT");
+const ACCESS_LOG_FORMAT = config.get("ACCESS_LOG_FORMAT") || LOG_FORMAT;
+
+const defaultMeta = {};
+
+let LOG_SYSTEMD = false;
+let ACCESS_LOG_SYSTEMD = false;
+
+let LOG_FILE, ACCESS_LOG_FILE;
+
+let logStream = fs.createWriteStream(null, {fd: process.stderr.fd});
+let logStat = fs.fstatSync(logStream.fd);
+let accessLogStream = fs.createWriteStream(null, {fd: process.stdout.fd});
+let accessLogStat = fs.fstatSync(accessLogStream.fd);
+
+
+// Reopen if original files have been moved (e.g. logrotate)
+function reopen() {
+  let counter = 1;
+
+  if (LOG_FILE) {
+    ++ counter;
+    fs.stat(LOG_FILE, function(err, stat) {
+      if (err && !err.message.startsWith("ENOENT:"))
+        throw err;
+
+      if (!(stat && stat.dev === logStat.dev && stat.ino === logStat.ino)) {
+        logStream.end();
+        logStream = fs.createWriteStream(null,
+          {fd: fs.openSync(LOG_FILE, "a")});
+        logStat = fs.fstatSync(logStream.fd);
+      }
+
+      if (-- counter === 0)
+        setTimeout(reopen, REOPEN_EVERY - (Date.now() % REOPEN_EVERY)).unref();
+    });
+  }
+
+  if (ACCESS_LOG_FILE) {
+    ++ counter;
+    fs.stat(ACCESS_LOG_FILE, function(err, stat) {
+      if (err && !err.message.startsWith("ENOENT:"))
+        throw err;
+
+      if (!(stat && stat.dev === accessLogStat.dev && stat.ino === accessLogStat.ino)) {
+        accessLogStream.end();
+        accessLogStream = fs.createWriteStream(null,
+          {fd: fs.openSync(ACCESS_LOG_FILE, "a")});
+        accessLogStat = fs.fstatSync(accessLogStream.fd);
+      }
+
+      if (-- counter === 0)
+        setTimeout(reopen, REOPEN_EVERY - (Date.now() % REOPEN_EVERY)).unref();
+    });
+  }
+
+  if (-- counter === 0)
+    setTimeout(reopen, REOPEN_EVERY - (Date.now() % REOPEN_EVERY)).unref();
+}
+
+
+function init(service, version) {
+  defaultMeta.hostname = os.hostname();
+  defaultMeta.pid = process.pid;
+  defaultMeta.name = `genieacs-${service}`;
+  defaultMeta.version = version;
+
+  LOG_FILE = config.get(`${service.toUpperCase()}_LOG_FILE`);
+  ACCESS_LOG_FILE = config.get(`${service.toUpperCase()}_ACCESS_LOG_FILE`);
+
+  if (LOG_FILE) {
+    logStream = fs.createWriteStream(null,
+      {fd: fs.openSync(LOG_FILE, "a")});
+    logStat = fs.fstatSync(logStream.fd);
+  }
+
+  if (ACCESS_LOG_FILE) {
+    accessLogStream = fs.createWriteStream(null,
+      {fd: fs.openSync(ACCESS_LOG_FILE, "a")});
+    accessLogStat = fs.fstatSync(accessLogStream.fd);
+  }
+
+  // Determine if logs are going to journald
+  const JOURNAL_STREAM = process.env["JOURNAL_STREAM"];
+
+  if (JOURNAL_STREAM) {
+    let [dev, inode] = JOURNAL_STREAM.split(":");
+    dev = parseInt(dev);
+    inode = parseInt(inode);
+
+    LOG_SYSTEMD = (logStat.dev === dev && logStat.ino === inode);
+    ACCESS_LOG_SYSTEMD = (accessLogStat.dev === dev && accessLogStat.ino === inode);
+  }
+
+  if (LOG_FILE || ACCESS_LOG_FILE) {
+    // Can't use setInterval as we need all workers to cehck at the same time
+    setTimeout(reopen, REOPEN_EVERY - (Date.now() % REOPEN_EVERY)).unref();
+  }
+}
+
+
+function close() {
+  accessLogStream.end();
+  logStream.end();
+}
+
+
+function flatten(details) {
+  if (details.sessionContext) {
+    details.deviceId = details.sessionContext.deviceId;
+    details.remoteAddress = details.sessionContext.httpRequest.connection.remoteAddress;
+    delete details.sessionContext;
+  }
+
+  if (details.exception) {
+    details.exceptionName = details.exception.name;
+    details.exceptionMessage = details.exception.message;
+    details.exceptionStack = details.exception.stack;
+    delete details.exception;
+  }
+
+  if (details.task) {
+    details.taskId = details.task._id;
+    delete details.task;
+  }
+
+  if (details.rpc) {
+    if (details.rpc.acsRequest) {
+      details.acsRequestId = details.rpc.id;
+      details.acsRequestName = details.rpc.acsRequest.name;
+      if (details.rpc.acsRequest.commandKey)
+        details.acsRequestCommandKey = details.rpc.acsRequest.commandKey;
+    }
+    else if (details.rpc.cpeRequest) {
+      details.cpeRequestId = details.rpc.id;
+      if (details.rpc.cpeRequest.name === "Inform") {
+        details.informEvent = details.rpc.cpeRequest.event.join(",");
+        details.informRetryCount = details.rpc.cpeRequest.retryCount;
+      }
+      else {
+        details.cpeRequestName = details.rpc.cpeRequest.type;
+        if (details.rpc.cpeRequest.commandKey)
+          details.cpeRequestCommandKey = details.rpc.cpeRequest.commandKey;
+      }
+    }
+    else if (details.rpc.cpeFault) {
+      details.acsRequestId = details.rpc.id;
+      details.cpeFaultCode = details.rpc.cpeFault.faultCode;
+      details.cpeFaultString = details.rpc.cpeFault.faultString;
+    }
+    delete details.rpc;
+  }
+
+  if (details.fault) {
+    details.faultCode = details.fault.code;
+    details.faultMessage = details.fault.message;
+    delete details.fault;
+  }
+
+  return details;
+}
+
+
+function formatJson(details, systemd) {
+  if (systemd) {
+    let severity = "";
+    if (details.severity === "info")
+      severity = "<6>";
+    else if (details.severity === "warn")
+      severity = "<4>";
+    else if (details.severity === "error")
+      severity = "<3>";
+
+    return `${severity}${JSON.stringify(flatten(details))}${os.EOL}`;
+  }
+
+  return `${JSON.stringify(flatten(details))}${os.EOL}`;
+}
+
+
+function formatSimple(details, systemd) {
+  flatten(details);
+
+  let remote = "";
+  if (details.remoteAddress) {
+    if (details.deviceId)
+      remote = `${details.remoteAddress} ${details.deviceId}: `;
+    else
+      remote = `${details.remoteAddress}: `;
+  }
+
+  const keys = Object.keys(details);
+  const skip = {
+    deviceId: 1,
+    remoteAddress: 1,
+    severity: 1,
+    timestamp: 1,
+    message: 1
+  };
+
+  let meta = "";
+
+  let kv = [];
+  for (let k of keys)
+    if (!skip[k])
+      kv.push(`${k}=${JSON.stringify(details[k])}`);
+
+  if (kv.length)
+    meta = `; ${kv.join(" ")}`;
+
+  if (systemd) {
+    let severity = "";
+    if (details.severity === "info")
+      severity = "<6>";
+    else if (details.severity === "warn")
+      severity = "<4>";
+    else if (details.severity === "error")
+      severity = "<3>";
+
+    return `${severity}${remote}${details.message}${meta}${os.EOL}`;
+  }
+
+  return `${details.timestamp} [${details.severity.toUpperCase()}] ${remote}${details.message}${meta}${os.EOL}`;
+}
+
+
+function log(details) {
+  details.timestamp = new Date().toISOString();
+  if (LOG_FORMAT === "json") {
+    details = Object.assign({}, defaultMeta, details);
+    logStream.write(formatJson(details, LOG_SYSTEMD));
+  } else {
+    logStream.write(formatSimple(details, LOG_SYSTEMD));
+  }
+}
+
+
+function info(details) {
+  details.severity = "info";
+  log(details);
+}
+
+
+function warn(details) {
+  details.severity = "warn";
+  log(details);
+}
+
+
+function error(details) {
+  details.severity = "error";
+  log(details);
+}
+
+
+function accessLog(details) {
+  details.timestamp = new Date().toISOString();
+  if (ACCESS_LOG_FORMAT === "json") {
+    Object.assign(details, defaultMeta);
+    accessLogStream.write(formatJson(details, ACCESS_LOG_SYSTEMD));
+  } else {
+    accessLogStream.write(formatSimple(details, ACCESS_LOG_SYSTEMD));
+  }
+}
+
+
+function accessInfo(details) {
+  details.severity = "info";
+  accessLog(details);
+}
+
+
+function accessWarn(details) {
+  details.severity = "warn";
+  accessLog(details);
+}
+
+
+function accessError(details) {
+  details.severity = "error";
+  accessLog(details);
+}
+
+
+exports.init = init;
+exports.close = close;
+exports.info = info;
+exports.warn = warn;
+exports.error = error;
+exports.accessInfo = accessInfo;
+exports.accessWarn = accessWarn;
+exports.accessError = accessError;

lib/mongo-query.coffee

@@ -1,141 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-#
-# This file incorporates work covered by the following copyright and
-# permission notice:
-#
-# Copyright 2013 Fanoos Telecom
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-###
-
-common = require './common'
-
-
-test = (obj, query) ->
-  for k,v of query
-    if k.charAt(0) == '$' # this is a logical operator
-      res = testExpressions(obj, v, k)
-    else
-      value = common.getParamValueFromPath(obj, k)
-
-      if common.typeOf(v) isnt common.OBJECT_TYPE
-        # TODO comparing array to regex, array to array, and object to object
-        if common.typeOf(value) is common.ARRAY_TYPE
-          res = value.indexOf(v) != -1
-        else
-          if common.typeOf(v) is common.REGEXP_TYPE
-            res = v.test(value)
-          else
-            res = v == value
-      else
-        for k2,v2 of v
-          switch k2
-            when '$ne'
-              if common.typeOf(value) is common.ARRAY_TYPE
-                res = value.indexOf(v2) == -1
-              else
-                res = value != v2
-            when '$lt'
-              res = value < v2
-            when '$lte'
-              res = value <= v2
-            when '$gt'
-              res = value > v2
-            when '$gte'
-              res = value >= v2
-            when '$regex'
-              res = v2.test(value)
-            when '$in'
-              throw new Error('Operator not supported')        
-            when '$nin'
-              throw new Error('Operator not supported')        
-            when '$all'
-              throw new Error('Operator not supported')        
-            when '$exists'
-              throw new Error('Operator not supported')        
-            else
-              throw new Error('Operator not supported')        
-
-    if not res
-      return false
-
-  return true
-
-
-testExpressions = (obj, expressions, lop) ->
-  for f in expressions
-    res = test(obj, f)
-
-    switch lop
-      when '$and'
-        return false if not res
-      when '$or'
-        return true if res
-      when '$nor'
-        return false if res
-
-  return switch lop
-    when '$and' then true
-    when '$or' then false
-    when '$nor' then true
-    else throw new Error('Unknown logical operator')
-
-
-# generate field projection from a query
-projection = (query, proj) ->
-  for k,v of query
-    if k.charAt(0) == '$' # this is a logical operator
-      for q in v
-        projection(q, proj)
-    else
-      proj[k] = 1
-
-# optimize projection by removing overlaps
-optimizeProjection = (obj) ->
-  keys = Object.keys(obj).sort()
-  return if keys.length <= 1
-  i = 1
-  while i < keys.length
-    a = keys[i-1]
-    b = keys[i]
-    if common.startsWith(b, a)
-      if b.charAt(a.length) == '.' or b.charAt(a.length - 1) == '.'
-        delete obj[b]
-        keys.splice(i--, 1)
-    ++ i
-  return
-
-
-exports.test = test
-exports.projection = projection
-exports.optimizeProjection = optimizeProjection

lib/nbi.coffee

@@ -1,5 +1,7 @@
 ###
-# Copyright 2013, 2014  Zaid Abdulla
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
 #
 # GenieACS is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as
@@ -38,15 +40,18 @@
 # THE SOFTWARE.
 ###
 
-config = require './config'
 url = require 'url'
-common = require './common'
-db = require './db'
 mongodb = require 'mongodb'
 querystring = require 'querystring'
+vm = require 'vm'
+
+config = require './config'
+common = require './common'
+db = require './db'
 query = require './query'
 apiFunctions = require './api-functions'
-presets = require './presets'
+
+VERSION = require('../package.json').version
 
 # regular expression objects
 DEVICE_TASKS_REGEX = /^\/devices\/([a-zA-Z0-9\-\_\%]+)\/tasks\/?$/
@@ -57,18 +62,27 @@ OBJECTS_REGEX = /^\/objects\/([a-zA-Z0-9\-\_\%]+)\/?$/
 FILES_REGEX = /^\/files\/([a-zA-Z0-9\%\!\*\'\(\)\;\:\@\&\=\+\$\,\?\#\[\]\-\_\.\~]+)\/?$/
 PING_REGEX = /^\/ping\/([a-zA-Z0-9\-\_\.]+)\/?$/
 QUERY_REGEX = /^\/([a-zA-Z0-9_]+s)\/?$/
-DEVICE_PRESET_REGEX = /^\/devices\/([a-zA-Z0-9\-\_\%]+)\/preset\/?$/
 DELETE_DEVICE_REGEX = /^\/devices\/([a-zA-Z0-9\-\_\%]+)\/?$/
+PROVISIONS_REGEX = /^\/provisions\/([a-zA-Z0-9\-\_\%]+)\/?$/
+VIRTUAL_PARAMETERS_REGEX = /^\/virtual_parameters\/([a-zA-Z0-9\-\_\%]+)\/?$/
+FAULTS_REGEX = /^\/faults\/([a-zA-Z0-9\-\_\%\:]+)\/?$/
 
 
-errorToString = (err) ->
+
-  "#{err.name}: #{err.message}"
+throwError = (err, httpResponse) ->
+  if httpResponse
+    httpResponse.writeHead(500, {'Connection' : 'close'})
+    httpResponse.end("#{err.name}: #{err.message}")
+
+  throw err
 
 
 listener = (request, response) ->
   chunks = []
   bytes = 0
 
+  response.setHeader('GenieACS-Version', VERSION)
+
   request.addListener 'data', (chunk) ->
     chunks.push(chunk)
     bytes += chunk.length
@@ -93,34 +107,22 @@ listener = (request, response) ->
         preset._id = presetName
 
         db.presetsCollection.save(preset, (err) ->
-          db.redisClient.del('presets', 'presets_hash', (err) ->
+          return throwError(err, response) if err
-            throw err if err
+          db.redisClient.del('presets_hash', (err) ->
-          )
+            return throwError(err, response) if err
-          if err
-            response.writeHead(500)
-            response.end(errorToString(err))
-            return
             response.writeHead(200)
             response.end()
           )
+        )
       else if request.method == 'DELETE'
-        db.presetsCollection.remove({'_id' : presetName}, (err, removedCount) ->
+        db.presetsCollection.remove({'_id' : presetName}, (err) ->
-          db.redisClient.del('presets', 'presets_hash', (err) ->
+          return throwError(err, response) if err
-            throw err if err
+          db.redisClient.del('presets_hash', (err) ->
-          )
+            return throwError(err, response) if err
-          if err
-            response.writeHead(500)
-            response.end(errorToString(err))
-            return
-
-          if removedCount == 0
-            response.writeHead(404)
-            response.end()
-            return
-
             response.writeHead(200)
             response.end()
           )
+        )
       else
         response.writeHead 405, {'Allow': 'PUT, DELETE'}
         response.end('405 Method Not Allowed')
@@ -131,34 +133,90 @@ listener = (request, response) ->
         object._id = objectName
 
         db.objectsCollection.save(object, (err) ->
+          return throwError(err, response) if err
           db.redisClient.del('objects', 'presets_hash', (err) ->
-            throw err if err
+            return throwError(err, response) if err
-          )
-          if err
-            response.writeHead(500)
-            response.end(errorToString(err))
-            return
             response.writeHead(200)
             response.end()
           )
+        )
       else if request.method == 'DELETE'
-        db.objectsCollection.remove({'_id' : objectName}, (err, removedCount) ->
+        db.objectsCollection.remove({'_id' : objectName}, (err) ->
+          return throwError(err, response) if err
           db.redisClient.del('objects', 'presets_hash', (err) ->
-            throw err if err
+            return throwError(err, response) if err
-          )
-          if err
-            response.writeHead(500)
-            response.end(errorToString(err))
-            return
-
-          if removedCount == 0
-            response.writeHead(404)
-            response.end()
-            return
-
             response.writeHead(200)
             response.end()
           )
+        )
+      else
+        response.writeHead 405, {'Allow': 'PUT, DELETE'}
+        response.end('405 Method Not Allowed')
+    else if PROVISIONS_REGEX.test(urlParts.pathname)
+      provisionName = querystring.unescape(PROVISIONS_REGEX.exec(urlParts.pathname)[1])
+      if request.method == 'PUT'
+        object = {
+          _id: provisionName
+          script: body.toString()
+        }
+        try
+          new vm.Script("\"use strict\";(function(){\n#{object.script}\n})();")
+        catch err
+          response.writeHead(400)
+          response.end("#{err.name}: #{err.message}")
+          return
+
+        db.provisionsCollection.save(object, (err) ->
+          return throwError(err, response) if err
+          db.redisClient.del('presets_hash', (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
+      else if request.method == 'DELETE'
+        db.provisionsCollection.remove({'_id' : provisionName}, (err) ->
+          return throwError(err, response) if err
+          db.redisClient.del('presets_hash', (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
+      else
+        response.writeHead 405, {'Allow': 'PUT, DELETE'}
+        response.end('405 Method Not Allowed')
+    else if VIRTUAL_PARAMETERS_REGEX.test(urlParts.pathname)
+      virtualParameterName = querystring.unescape(VIRTUAL_PARAMETERS_REGEX.exec(urlParts.pathname)[1])
+      if request.method == 'PUT'
+        object = {
+          _id: virtualParameterName
+          script: body.toString()
+        }
+        try
+          new vm.Script("\"use strict\";(function(){\n#{object.script}\n})();")
+        catch err
+          response.writeHead(400)
+          response.end("#{err.name}: #{err.message}")
+          return
+
+        db.virtualParametersCollection.save(object, (err) ->
+          return throwError(err, response) if err
+          db.redisClient.del('presets_hash', (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
+      else if request.method == 'DELETE'
+        db.virtualParametersCollection.remove({'_id' : virtualParameterName}, (err) ->
+          return throwError(err, response) if err
+          db.redisClient.del('presets_hash', (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
       else
         response.writeHead 405, {'Allow': 'PUT, DELETE'}
         response.end('405 Method Not Allowed')
@@ -167,59 +225,62 @@ listener = (request, response) ->
       deviceId = querystring.unescape(r[1])
       tag = querystring.unescape(r[2])
       if request.method == 'POST'
-        db.devicesCollection.update({'_id' : deviceId}, {'$addToSet' : {'_tags' : tag}}, {safe: true}, (err, updatedCount) ->
+        db.devicesCollection.update({'_id' : deviceId}, {'$addToSet' : {'_tags' : tag}}, {safe: true}, (err) ->
+          return throwError(err, response) if err
           db.redisClient.del("#{deviceId}_presets_hash", (err) ->
-            throw err if err
+            return throwError(err, response) if err
-          )
-          if err
-            response.writeHead(500)
-            response.end(errorToString(err))
-            return
-
-          if updatedCount == 0
-            response.writeHead(404)
-            response.end()
-            return
-
             response.writeHead(200)
             response.end()
           )
+        )
       else if request.method == 'DELETE'
-        db.devicesCollection.update({'_id' : deviceId}, {'$pull' : {'_tags' : tag}}, {safe: true}, (err, updatedCount) ->
+        db.devicesCollection.update({'_id' : deviceId}, {'$pull' : {'_tags' : tag}}, {safe: true}, (err) ->
+          return throwError(err, response) if err
           db.redisClient.del("#{deviceId}_presets_hash", (err) ->
-            throw err if err
+            return throwError(err, response) if err
-          )
-          if err
-            response.writeHead(500)
-            response.end(errorToString(err))
-            return
-
-          if updatedCount == 0
-            response.writeHead(404)
-            response.end()
-            return
-
             response.writeHead(200)
             response.end()
           )
+        )
       else
         response.writeHead 405, {'Allow': 'POST, DELETE'}
         response.end('405 Method Not Allowed')
+    else if FAULTS_REGEX.test(urlParts.pathname)
+      if request.method == 'DELETE'
+        faultId = querystring.unescape(FAULTS_REGEX.exec(urlParts.pathname)[1])
+        deviceId = faultId.split(':', 1)[0]
+        channel = faultId.slice(deviceId.length + 1)
+        db.faultsCollection.remove({_id : faultId}, (err) ->
+          return throwError(err, response) if err
+          if channel.startsWith('task_')
+            return db.tasksCollection.remove({_id : new mongodb.ObjectID(channel.slice(5))}, (err) ->
+              return throwError(err, response) if err
+              db.redisClient.del("#{deviceId}_tasks", "#{deviceId}_faults", (err) ->
+                return throwError(err, response) if err
+                response.writeHead(200)
+                response.end()
+              )
+            )
+
+          db.redisClient.del("#{deviceId}_faults", (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
+      else
+        response.writeHead 405, {'Allow': 'DELETE'}
+        response.end('405 Method Not Allowed')
     else if DEVICE_TASKS_REGEX.test(urlParts.pathname)
       if request.method == 'POST'
         deviceId = querystring.unescape(DEVICE_TASKS_REGEX.exec(urlParts.pathname)[1])
         if body.length
           task = JSON.parse(body)
           task.device = deviceId
-          db.getAliases((aliases) ->
+          apiFunctions.insertTasks(task, (err) ->
-            apiFunctions.insertTasks(task, aliases, (err) ->
+            return throwError(err, response) if err
-              db.redisClient.del("#{deviceId}_presets_hash", (err) ->
+            db.redisClient.del("#{deviceId}_tasks", (err) ->
-                throw err if err
+              return throwError(err, response) if err
-              )
-              if err
-                response.writeHead(500)
-                response.end(errorToString(err))
-                return
 
               if urlParts.query.connection_request?
                 apiFunctions.connectionRequest(deviceId, (err) ->
@@ -227,21 +288,15 @@ listener = (request, response) ->
                     response.writeHead(202, err.message, {'Content-Type' : 'application/json'})
                     response.end(JSON.stringify(task))
                   else
-                    apiFunctions.watchTask(task._id, config.get('DEVICE_ONLINE_THRESHOLD', deviceId), (err, status) ->
+                    apiFunctions.watchTask(deviceId, task._id, config.get('DEVICE_ONLINE_THRESHOLD', deviceId), (err, status) ->
-                      if err
+                      return throwError(err, response) if err
-                        response.writeHead(500)
-                        response.end(errorToString(err))
-                        return
 
                       if status is 'timeout'
                         response.writeHead(202, 'Task queued but not processed', {'Content-Type' : 'application/json'})
                         response.end(JSON.stringify(task))
                       else if status is 'fault'
                         db.tasksCollection.findOne({_id : task._id}, (err, task) ->
-                          if err
+                          return throwError(err, response) if err
-                            response.writeHead(500)
-                            response.end(errorToString(err))
-                            return
 
                           response.writeHead(202, 'Task faulted', {'Content-Type' : 'application/json'})
                           response.end(JSON.stringify(task))
@@ -261,7 +316,7 @@ listener = (request, response) ->
           apiFunctions.connectionRequest(deviceId, (err) ->
             if err
               response.writeHead 504
-              response.end(errorToString(err))
+              response.end("#{err.name}: #{err.message}")
               return
             response.writeHead 200
             response.end()
@@ -272,54 +327,44 @@ listener = (request, response) ->
       else
         response.writeHead 405, {'Allow': 'POST'}
         response.end('405 Method Not Allowed')
-    else if DEVICE_PRESET_REGEX.test(urlParts.pathname)
-      deviceId = querystring.unescape(DEVICE_PRESET_REGEX.exec(urlParts.pathname)[1])
-      if request.method is 'GET'
-        db.getPresetsObjectsAliases((allPresets, allObjects, allAliases) ->
-          presets.getDevicePreset(deviceId, allPresets, allObjects, allAliases, (devicePreset) ->
-            response.writeHead(200, {'Content-Type' : 'application/json'})
-            response.end(JSON.stringify(devicePreset))
-          )
-        )
-      else
-        response.writeHead 405, {'Allow': 'GET'}
-        response.end('405 Method Not Allowed')
     else if TASKS_REGEX.test(urlParts.pathname)
       r = TASKS_REGEX.exec(urlParts.pathname)
-      taskId = mongodb.ObjectID(querystring.unescape(r[1]))
+      taskId = new mongodb.ObjectID(querystring.unescape(r[1]))
       action = r[2]
       if not action? or action is '/'
         if request.method == 'DELETE'
-          db.tasksCollection.remove({'_id' : taskId}, (err, removedCount) ->
+          db.tasksCollection.findOne({'_id' : taskId}, {'device' : 1}, (err, task) ->
-            if err
+            return throwError(err, response) if err
-              response.writeHead(500)
+            deviceId = task.device
-              response.end(errorToString(err))
+            db.tasksCollection.remove({'_id' : taskId}, (err) ->
-              return
+              return throwError(err, response) if err
-
+              db.faultsCollection.remove({_id : "#{deviceId}:task_#{String(taskId)}"}, (err) ->
-            if removedCount == 0
+                return throwError(err, response) if err
-              response.writeHead(404)
+                db.redisClient.del("#{deviceId}_tasks", "#{deviceId}_faults", (err) ->
-              response.end()
+                  return throwError(err, response) if err
-              return
-
                   response.writeHead(200)
                   response.end()
                 )
+              )
+            )
+          )
         else
           response.writeHead 405, {'Allow': 'PUT DELETE'}
           response.end('405 Method Not Allowed')
       else if action is '/retry'
         if request.method == 'POST'
-          db.tasksCollection.update({_id : taskId}, {$unset : {fault : 1}, $set : {timestamp : new Date()}}, (err, updatedCount) ->
+          db.tasksCollection.findOne({'_id' : taskId}, {'device' : 1}, (err, task) ->
-            # TODO need to invalidate presets hash for the device
+            return throwError(err, response) if err
-            throw err if err
+            deviceId = task.device
-            if updatedCount == 0
+            db.faultsCollection.remove({_id : "#{deviceId}:task_#{String(taskId)}"}, (err) ->
-              response.writeHead(404)
+              return throwError(err, response) if err
-              response.end()
+              db.redisClient.del("#{deviceId}_faults", (err) ->
-              return
+                return throwError(err, response) if err
-
                 response.writeHead(200)
                 response.end()
               )
+            )
+          )
         else
           response.writeHead 405, {'Allow': 'POST'}
           response.end('405 Method Not Allowed')
@@ -336,12 +381,12 @@ listener = (request, response) ->
           version : request.headers.version,
         }
 
-        gs = new mongodb.GridStore(db.mongoDb, filename, 'w', {metadata : metadata})
+        gs = new mongodb.GridStore(db.mongoDb, filename, filename, 'w', {metadata : metadata})
         gs.open((err, gs) ->
           gs.write(body, (err, res) ->
-            throw err if err
+            return throwError(err, response) if err
             gs.close((err) ->
-              throw err if err
+              return throwError(err, response) if err
               response.writeHead(201)
               response.end()
             )
@@ -360,7 +405,7 @@ listener = (request, response) ->
       require('child_process').exec("ping -w 1 -i 0.2 -c 3 #{host}", (err, stdout, stderr) ->
         if err
           response.writeHead(404, {'Cache-Control' : 'no-cache'})
-          response.end(errorToString(err))
+          response.end("#{err.name}: #{err.message}")
           return
         response.writeHead(200, {'Content-Type' : 'text/plain', 'Cache-Control' : 'no-cache'})
         response.end(stdout)
@@ -373,15 +418,21 @@ listener = (request, response) ->
 
       deviceId = querystring.unescape(DELETE_DEVICE_REGEX.exec(urlParts.pathname)[1])
       apiFunctions.deleteDevice(deviceId, (err) ->
-        if err
+        return throwError(err, response) if err
-          response.writeHead(500)
-          response.end(errorToString(err))
-          return
         response.writeHead(200)
         response.end()
       )
     else if QUERY_REGEX.test(urlParts.pathname)
       collectionName = QUERY_REGEX.exec(urlParts.pathname)[1]
+
+      # Convert to camel case
+      i = collectionName.indexOf('_')
+      while i >= 0
+        ++ i
+        up = if i < collectionName.length then collectionName[i].toUpperCase() else ''
+        collectionName = collectionName.slice(0, i - 1) + up + collectionName.slice(i + 1)
+        i = collectionName.indexOf('_', i)
+
       if request.method not in ['GET', 'HEAD']
         response.writeHead 405, {'Allow' : 'GET, HEAD'}
         response.end('405 Method Not Allowed')
@@ -392,41 +443,42 @@ listener = (request, response) ->
         response.end('404 Not Found')
         return
 
-      func = (aliases) ->
       if urlParts.query.query?
         try
           q = JSON.parse(urlParts.query.query)
         catch err
           response.writeHead(400)
-            response.end(errorToString(err))
+          response.end("#{err.name}: #{err.message}")
           return
       else
         q = {}
-        q = query.expand(q, aliases) if collectionName is 'devices'
+
-        q = query.substituteObjectId(q) if collectionName is 'tasks'
+      switch collectionName
+        when 'devices'
+          q = query.expand(q)
+        when 'tasks'
+          q = query.sanitizeQueryTypes(q, {
+            _id: ((v) -> return new mongodb.ObjectID(v))
+            timestamp: ((v) -> return new Date(v))
+            retries: Number
+          })
+        when 'faults'
+          q = query.sanitizeQueryTypes(q, {
+            timestamp: ((v) -> return new Date(v))
+            retries: Number
+          })
 
       if urlParts.query.projection?
         projection = {}
         for p in urlParts.query.projection.split(',')
           p = p.trim()
           projection[p] = 1
-            if collectionName is 'devices'
-              for k,v of aliases
-                if k == p or common.startsWith(k, "#{p}.")
-                  projection[a] = 1 for a in v
 
       cur = collection.find(q, projection, {batchSize : 50})
       if urlParts.query.sort?
         s = JSON.parse(urlParts.query.sort)
         sort = {}
         for k,v of s
-            if aliases[k]?
-              for a in aliases[k]
-                if a[a.lastIndexOf('.')+1] != '_' and collectionName is 'devices'
-                  sort["#{a}._value"] = v
-                else
-                  sort[a] = v
-            else
           if k[k.lastIndexOf('.') + 1] != '_' and collectionName is 'devices'
             sort["#{k}._value"] = v
           else
@@ -434,8 +486,8 @@ listener = (request, response) ->
         cur.sort(sort)
 
       cur.skip(parseInt(urlParts.query.skip)) if urlParts.query.skip?
-        cur.limit(parseInt(urlParts.query.limit)) if urlParts.query.limit?
+      cur.limit(limit = parseInt(urlParts.query.limit)) if urlParts.query.limit?
-        cur.count((err, total) ->
+      cur.count(false, (err, total) ->
         response.writeHead(200, {'Content-Type' : 'application/json', 'total' : total})
         if request.method is 'HEAD'
           response.end()
@@ -443,21 +495,19 @@ listener = (request, response) ->
         response.write("[\n")
         i = 0
         cur.each((err, item) ->
-            if item is null
+          if err
-              response.end("\n]")
+            throwError(err, response)
-            else
+            return false
+
+          if item?
             response.write(",\n") if i++
-              apiFunctions.addAliases(item, aliases) if collectionName is 'devices'
             response.write(JSON.stringify(item))
+
+          if not item? or (limit? and i >= limit)
+            response.end("\n]")
         )
       )
 
-      if collectionName is 'devices'
-        db.getAliases((aliases) ->
-          func(aliases)
-        )
-      else
-        func({})
     else
       response.writeHead 404
       response.end('404 Not Found')

lib/normalize.coffee

@@ -1,132 +0,0 @@
-###
-# Copyright 2013 Fanoos Telecom
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-###
-
-# Credit to Netanel Rubin from Check Point Software Technologies for discovering
-# vulnerability caused by use of eval to generate regex from user input.
-
-common = require './common'
-parameters = require './parameters'
-
-
-stringToRegexp = (input, flags) ->
-  if (input.indexOf('*') == -1)
-    return false
-
-  output = input.replace(/[\[\]\\\^\$\.\|\?\+\(\)]/, "\\$&")
-  if output[0] == '*'
-    output = output.replace(/^\*+/g, '')
-  else
-    output = '^' + output
-
-  if output[output.length - 1] == '*'
-    output = output.replace(/\*+$/g, '')
-  else
-    output = output + '$'
-
-  output = output.replace(/[\*]/, '.*')
-
-  return new RegExp(output, flags)
-
-
-normalizers = {}
-
-normalizers.default = (input, normType) ->
-  if normType is 'query' and common.typeOf(input) is common.STRING_TYPE
-    vals = []
-    str = normalizers.string(input, normType)
-    if common.typeOf(str) == common.ARRAY_TYPE
-      vals = vals.concat(str)
-    else
-      vals.push(str)
-
-    f = parseFloat(input)
-    if not isNaN(f)
-      vals.push(f)
-
-    d = new Date(input)
-    if input.length >= 8 and d.getFullYear() > 1983
-      vals.push(d)
-
-    r = stringToRegexp(input)
-    if r isnt false
-      vals.push({'$regex' : r})
-
-    return vals
-  return input
-
-
-FLOAT_REGEX = /^([\-\+]?[0-9\.]+)/
-normalizers.float = (input, normType) ->
-  res = FLOAT_REGEX.exec(input)
-  if res != null
-    parseFloat(res[1])
-  else
-    null
-
-
-normalizers.date = (input, normType) ->
-  return new Date(input)
-
-
-normalizers.string = (input, normType) ->
-  if normType is 'query'
-    if (m = /^\/(.*?)\/(g?i?m?y?)$/.exec(input))
-      return [{'$regex' : new RegExp(m[1], m[2])}, input]
-  return input
-
-
-colonizeMac = (input) ->
-  if (i = input.indexOf(':')) != -1
-    i = if (i % 2) == 0 then 0 else 1
-    while i < input.length
-      if input[i-1] != ':' and input[i-2] != ':'
-        if input[i] != ':' and i != 0
-          input = input.substring(0, i) + ':' + input.substring(i)
-          ++i
-        ++i
-      ++ i
-    return input
-  else
-    return input.replace(/[0-9A-F]{1}(?!$)/g, '$&:?')
-
-
-normalizers.mac = (input, normType) ->
-  if normType isnt 'query'
-    return input
-
-  input = input.trim().toUpperCase().replace(/[^0-9A-F\*\-\:]/g, '').replace(/[\-\:]/g, ':')
-
-  if input.indexOf('*') != -1
-    input = (colonizeMac(a) for a in input.split('*')).join('*')
-    return stringToRegexp(input, 'i')
-
-  input = colonizeMac(input)
-
-  if input.length == 17
-    return [input, input.toLowerCase()]
-
-  return {'$regex' : new RegExp(input, 'i')}
-
-
-exports.normalize = (path, value, normType) ->
-  type = parameters.getType(path) ? 'default'
-  normalizers[type](value, normType)

lib/parameters.coffee

@@ -1,212 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-###
-
-config = require './config'
-db = require './db'
-
-
-PATH_REGEX = /\w+|(\/.+?\/\w*)/g
-
-# Javascript represents numbers as 64-bit floating point numbers
-# Integers up to 2^53 can be encoded precisely
-MAX_PARAMETERS = 52
-
-
-# get the index of the least significant bit
-getLsb = (num) ->
-  i = -1
-  while num > 0
-    ++ i
-    break if num & 1
-    num >>= 1
-  return i
-
-
-createParameterAttributeFinder = (attributeName) ->
-  # compile attributes into data types optimized for quick lookup
-  attributes = []
-  stringSegments = {}
-  regexSegments = {}
-  paramRegex = {}
-
-  for k, v of config.PARAMETERS
-    continue if not v[attributeName]?
-    attributeIndex = attributes.push(v[attributeName]) - 1
-    throw new Error("There cannot be more than #{MAX_PARAMETERS} configured parameters") if attributeIndex >= MAX_PARAMETERS
-    segmentIndex = 0
-    parts = k.match(PATH_REGEX)
-    for i in [0 ... parts.length]
-      part = parts[i]
-      positionHash = i + parts.length * MAX_PARAMETERS
-
-      if part[0] == '/' # this is a regex part
-        j = part.lastIndexOf('/')
-        regExp = new RegExp(part[1...j], part[j+1..])
-        regexSegments[positionHash] ?= 0
-        regexSegments[positionHash] |= Math.pow(2, attributeIndex)
-        paramRegex[attributeIndex] ?= {}
-        paramRegex[attributeIndex][i] = regExp
-      else
-        stringSegments[positionHash] ?= {}
-        stringSegments[positionHash][part] ?= 0
-        stringSegments[positionHash][part] |= Math.pow(2, attributeIndex)
-
-  # Finder function
-  return (param) ->
-    parts = param.split('.')
-    positionHash = parts.length * MAX_PARAMETERS
-
-    stringIndices = 0 | stringSegments[positionHash]?[parts[0]]
-    allIndices = stringIndices | regexSegments[positionHash]
-
-    for i in [1...parts.length]
-      positionHash = i + parts.length * MAX_PARAMETERS
-      strIdx = stringSegments[positionHash]?[parts[i]]
-      rgxIdx = regexSegments[positionHash]
-      stringIndices &= strIdx
-      allIndices &= (strIdx | rgxIdx)
-
-    if stringIndices > 0
-      return attributes[getLsb(stringIndices)]
-
-    attributeIndex = 0
-    while allIndices > 0
-      if allIndices & 1
-        match = true
-        for i, r of paramRegex[attributeIndex]
-          if not r.test(parts[i])
-            match = false
-            break
-
-        if match
-          return attributes[attributeIndex]
-
-      ++ attributeIndex
-      allIndices >>= 1
-
-    return null
-
-
-splitIds = (batches, callback) ->
-  db.devicesCollection.count((err, count) ->
-    throw err if err
-    return callback(null) if count == 0
-    ids = [{}]
-    return callback(ids) if count <= 1000 or batches == 1
-    batchSize = Math.floor(count / batches)
-    for i in [1 ... batches]
-      ids[i] = {}
-      do (i) ->
-        cursor = db.devicesCollection.find({}, {_id:1}).sort({'_id':1}).skip(i * batchSize).limit(1)
-        cursor.nextObject((err, obj) ->
-          ids[i]['$gte'] = obj._id
-          ids[i-1]['$lt'] = obj._id
-          if i == batches - 1
-            callback(ids)
-        )
-  )
-
-
-compileAliases = (callback) ->
-  map = () ->
-    recurse = (partIndex, ref) ->
-      part = parts[partIndex]
-      res = []
-      if part.test?
-        for p of ref
-          if part.test(p)
-            if partIndex == parts.length - 1
-              res.push(p)
-            else
-              for r in recurse(partIndex + 1, ref[p])
-                res.push("#{p}.#{r}")
-      else
-        if ref[part]?
-          if partIndex == parts.length - 1
-            return [part]
-          else
-            for r in recurse(partIndex + 1, ref[part])
-              res.push("#{part}.#{r}")
-
-      return res
-
-    for key, parts of keys
-      for alias in recurse(0, this)
-        emit(alias, key)
-    return
-
-  reduce = (k, v) ->
-    return v[0]
-
-  aliases = {}
-  keys = {}
-  for k, v of config.PARAMETERS
-    alias = v.alias
-    continue if not alias?
-    isStatic = true
-    ar = []
-    for p in k.match(/\w+|(\/.+?\/\w*)/g)
-      if p[0] == '/' # this is a regex
-        isStatic = false
-        i = p.lastIndexOf('/')
-        r = new RegExp(p[1...i], p[i+1..])
-        ar.push(r)
-      else
-        ar.push(p)
-
-    if isStatic
-      aliases[alias] ?= []
-      aliases[alias].push(k)
-    else
-      keys[k] = ar
-
-  counter = 0
-  splitIds(4, (batches) ->
-    if not batches?
-      return callback(null, aliases)
-
-    for batch in batches
-      options = {
-        out : {inline : 1}
-        jsMode : true
-        sort : {$natural : 1}
-        scope : {keys : keys, flags : {}}
-        query : {_id : batch} if batches.length > 1
-      }
-
-      db.devicesCollection.mapReduce(map, reduce, options, (err, out) ->
-        if err
-          callback?(err)
-          return callback = null
-
-        for o in out
-          alias = config.PARAMETERS[o.value].alias
-          aliases[alias] ?= []
-          id = String(o._id)
-          if id not in aliases[alias]
-            aliases[alias].push(id)
-
-        if ++counter == batches.length
-          callback?(null, aliases)
-      )
-    return
-  )
-
-
-exports.getType = createParameterAttributeFinder('type')
-exports.getAlias = createParameterAttributeFinder('alias')
-exports.compileAliases = compileAliases

lib/path-set.js

@@ -0,0 +1,179 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+
+class PathSet {
+
+  constructor() {
+    this.lengthIndex = [];
+    this.fragmentIndex = [];
+  }
+
+
+  get depth() {
+    return this.lengthIndex.length;
+  }
+
+
+  add(path) {
+    let p = this.get(path);
+
+    if (p)
+      return p;
+
+    if (!this.lengthIndex[path.length])
+      this.lengthIndex[path.length] = new Set();
+
+    this.lengthIndex[path.length].add(path);
+
+    let wildcard = 0;
+    for (let i = 0; i < path.length; ++ i) {
+      let fragment = path[i];
+      if (fragment === "*")
+        wildcard = wildcard | (1 << i);
+
+      if (!this.fragmentIndex[i])
+        this.fragmentIndex[i] = {};
+
+      if (!this.fragmentIndex[i][fragment])
+        this.fragmentIndex[i][fragment] = new Set();
+
+      this.fragmentIndex[i][fragment].add(path);
+    }
+
+    if (path.wildcard == null) {
+      path.alias = 0;
+      path.wildcard = wildcard;
+      Object.freeze(path);
+    }
+    return path;
+  }
+
+
+  get(pattern) {
+    if (!this.lengthIndex[pattern.length])
+      return null;
+
+    if (this.lengthIndex[pattern.length].has(pattern))
+      return pattern;
+
+    let sets = [];
+
+    sets.push(this.lengthIndex[pattern.length]);
+
+    for (let i = 0; i < pattern.length; ++ i) {
+      if (!this.fragmentIndex[i])
+        return null;
+
+      let fragment = pattern[i];
+
+      if (!this.fragmentIndex[i][fragment])
+        return null;
+
+      sets.push(this.fragmentIndex[i][fragment]);
+    }
+
+    sets.sort(function(a, b) {
+      return a.size - b.size;
+    });
+
+    let smallestSet = sets.shift();
+    for (let path of smallestSet) {
+      for (let s of sets) {
+        if (!s.has(path)) {
+          path = null;
+          break
+        }
+      }
+      if (path)
+        return path;
+    }
+
+    return null;
+  }
+
+
+  find(pattern, superset = false, subset = false, depth = pattern.length) {
+    let res = [];
+    let groups = [[]];
+
+    for (let i = pattern.length; i <= depth; ++ i)
+      if (this.lengthIndex[i])
+        groups[0].push(this.lengthIndex[i]);
+
+    if (!groups[0].length)
+      return res;
+
+    for (let i = 0; i < pattern.length; ++ i) {
+      if (!this.fragmentIndex[i])
+        return res;
+
+      let fragment = pattern[i];
+
+      if (fragment === "*" && subset)
+        continue;
+
+      let g = [];
+
+      if (this.fragmentIndex[i][fragment])
+        g.push(this.fragmentIndex[i][fragment]);
+
+      if (fragment !== "*" && superset && this.fragmentIndex[i]["*"])
+        g.push(this.fragmentIndex[i]["*"]);
+
+      if (!g.length)
+        return res;
+
+      groups.push(g);
+    }
+
+    groups.sort(function(a, b) {
+      return a.reduce(function(prev, cur) {return prev + cur.size;}, 0) -
+        b.reduce(function(prev, cur) {return prev + cur.size;}, 0);
+    });
+
+    let smallestGroup = groups.shift();
+    for (let set of smallestGroup) {
+      for (let path of set) {
+        for (let group of groups) {
+          let found = false;
+          for (let s of group) {
+            if (s.has(path)) {
+              found = true;
+              break;
+            }
+          }
+          if (!found) {
+            path = null;
+            break;
+          }
+        }
+        if (path)
+          res.push(path);
+      }
+    }
+
+    return res;
+  }
+
+}
+
+
+module.exports = PathSet;

lib/presets.coffee

@@ -1,336 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-#
-# This file incorporates work covered by the following copyright and
-# permission notice:
-#
-# Copyright 2013 Fanoos Telecom
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-###
-
-config = require './config'
-common = require './common'
-db = require './db'
-mongoQuery = require './mongo-query'
-query = require './query'
-log = require('util').log
-customCommands = require './custom-commands'
-
-
-exports.calculatePresetsHash = (presets, objects) ->
-  crypto = require('crypto')
-  hash = crypto.createHash('md5').update(JSON.stringify(presets) + JSON.stringify(objects)).digest('hex')
-  return hash
-
-
-matchObject = (object, param) ->
-  flatObject = common.flattenObject(object)
-  if object._keys?.length
-    keys = object._keys
-  else
-    # Consider all parameters as keys if none are defined
-    keys = (k for k in Object.keys(flatObject) when k[0] != '_')
-
-  for k in keys
-    p = common.getParamValueFromPath(param, k)
-    return false if not p?
-    v = common.matchType(p._value, flatObject[k])
-
-    if p._value != common.matchType(p._value, flatObject[k])
-      return false
-
-  return true
-
-
-exports.getDevicePreset = (deviceId, presets, objects, aliases, callback) ->
-  deviceCustomCommands = customCommands.getDeviceCustomCommands(deviceId)
-  # only fetch relevant params
-  projection = {_id : 1}
-  for p in presets
-    if common.typeOf(p.precondition) is common.STRING_TYPE
-      p.precondition = query.expand(JSON.parse(p.precondition), aliases)
-    else
-      # Accept an object for backward compatiblity
-      p.precondition = query.expand(p.precondition ? {}, aliases)
-    mongoQuery.projection(p.precondition, projection)
-
-    for c in p.configurations
-      switch c.type
-        when 'value', 'age'
-          projection[c.name] = 1
-        when 'custom_command_value', 'custom_command_age'
-          commandfile = c.command.split(' ', 1)
-          if commandfile of deviceCustomCommands
-            projection["_customCommands.#{c.command.split(' ', 1)}"] = 1
-        when 'software_version'
-          projection['Device.DeviceInfo.SoftwareVersion'] = 1
-          projection['InternetGatewayDevice.DeviceInfo.SoftwareVersion'] = 1
-        when 'add_tag', 'delete_tag'
-          projection['_tags'] = 1
-        when 'add_object', 'delete_object'
-          projection[c.name] = 1
-        else
-          throw new Error('Unknown configuration type')
-
-  mongoQuery.optimizeProjection(projection)
-
-  db.devicesCollection.findOne({'_id' : deviceId}, projection, (err, device) ->
-    throw err if err
-    devicePresets = []
-    for p in presets
-      if mongoQuery.test(device, p.precondition)
-        devicePresets.push(p)
-
-    configurations = accumulateConfigurations(devicePresets, objects)
-
-    getParamDetails = (device, path) ->
-      p = common.getParamValueFromPath(device, path)
-      return undefined if not p?
-      details = {}
-      for k,v of p
-        details[k] = v if k[0] == '_'
-      return details
-
-    devicePreset = {}
-    for c in configurations
-      switch c.type
-        when 'value'
-          devicePreset.parameters ?= {}
-          devicePreset.parameters[c.name] ?= {}
-          devicePreset.parameters[c.name].preset ?= {}
-          devicePreset.parameters[c.name].preset.value = c.value
-          devicePreset.parameters[c.name].current = getParamDetails(device, c.name)
-
-        when 'age'
-          devicePreset.parameters ?= {}
-          devicePreset.parameters[c.name] ?= {}
-          devicePreset.parameters[c.name].preset ?= {}
-          devicePreset.parameters[c.name].preset.expiry = parseInt(c.age)
-          devicePreset.parameters[c.name].current = getParamDetails(device, c.name)
-
-        when 'add_tag'
-          devicePreset.tags ?= {}
-          devicePreset.tags[c.tag] = 2 | c.tag in (device._tags ? [])
-
-        when 'delete_tag'
-          devicePreset.tags ?= {}
-          devicePreset.tags[c.tag] = 0 | c.tag in (device._tags ? [])
-
-        when 'add_object'
-          devicePreset.objects ?= {}
-          devicePreset.objects[c.name] ?= {}
-          devicePreset.objects[c.name][c.object] ?= {}
-          devicePreset.objects[c.name][c.object].preset = objects[c.object]
-          continue if not param = common.getParamValueFromPath(device, c.name)
-          devicePreset.objects[c.name][c.object].current ?= {}
-          for k,p of param
-            continue if k[0] == '_'
-            if matchObject(objects[c.object], p)
-              devicePreset.objects[c.name][c.object].current[k] = p
-
-        when 'delete_object'
-          devicePreset.objects ?= {}
-          devicePreset.objects[c.name] ?= {}
-          devicePreset.objects[c.name][c.object] ?= {}
-          continue if not param = common.getParamValueFromPath(device, c.name)
-          devicePreset.objects[c.name][c.object].current ?= {}
-          for k,p of param
-            continue if k[0] == '_'
-            if matchObject(objects[c.object], p)
-              devicePreset.objects[c.name][c.object].current[k] = p
-
-        when 'custom_command_value'
-          [filename, commandName] = c.command.split(' ', 2)
-          devicePreset.customCommands ?= {}
-          devicePreset.customCommands[filename] ?= {}
-          devicePreset.customCommands[filename].preset ?= {}
-          devicePreset.customCommands[filename].preset.value = c.value
-          devicePreset.customCommands[filename].preset.valueCommand = c.command
-          continue if not deviceCustomCommands[filename]? or commandName not in deviceCustomCommands[filename]
-          if cmd = common.getParamValueFromPath(device, "_customCommands.#{filename}")
-            devicePreset.customCommands[filename].current = cmd
-          else
-            devicePreset.customCommands[filename].current ?= {}
-
-        when 'custom_command_age'
-          [filename, commandName] = c.command.split(' ', 2)
-          devicePreset.customCommands ?= {}
-          devicePreset.customCommands[filename] ?= {}
-          devicePreset.customCommands[filename].preset ?= {}
-          devicePreset.customCommands[filename].preset.expiry = parseInt(c.age)
-          devicePreset.customCommands[filename].preset.expiryCommand = c.command
-          continue if not deviceCustomCommands[filename]? or commandName not in deviceCustomCommands[filename]
-          if cmd = common.getParamValueFromPath(device, "_customCommands.#{filename}")
-            devicePreset.customCommands[filename].current = cmd
-          else
-            devicePreset.customCommands[filename].current ?= {}
-
-        when 'software_version'
-          devicePreset.softwareVersion ?= {}
-          devicePreset.softwareVersion.preset = c.software_version
-          if device?.Device? # TR-181 data model
-            devicePreset.softwareVersion.current = common.getParamValueFromPath(device, 'Device.DeviceInfo.SoftwareVersion')
-          else # TR-098 data model
-            devicePreset.softwareVersion.current = common.getParamValueFromPath(device, 'InternetGatewayDevice.DeviceInfo.SoftwareVersion')
-
-        else
-          throw new Error('Unknown configuration type')
-
-    callback(devicePreset)
-  )
-
-
-exports.processDevicePreset = (deviceId, devicePreset, callback) ->
-  PRESETS_TIME_PADDING = config.get('PRESETS_TIME_PADDING')
-  now = new Date()
-  expiry = config.get('PRESETS_CACHE_DURATION', deviceId)
-  getParameterValues = []
-  setParameterValues = []
-  addTags = []
-  deleteTags = []
-  taskList = []
-
-  # Parameters
-  for parameterPath, parameterDetails of devicePreset.parameters ? {}
-    continue if not parameterDetails.current?
-    expiry = Math.min(expiry, parameterDetails.preset.expiry) if parameterDetails.preset.expiry?
-    presetValue = parameterDetails.preset.value
-    currentValue = parameterDetails.current._value
-    if presetValue? and currentValue != (presetValue = common.matchType(currentValue, presetValue))
-      setParameterValues.push([parameterPath, presetValue, parameterDetails.current._type])
-    else if parameterDetails.preset.expiry?
-      diff = parameterDetails.preset.expiry - (now - (parameterDetails.current._timestamp ? 0)) / 1000
-      if diff <= PRESETS_TIME_PADDING
-        if parameterDetails.current._value?
-          getParameterValues.push(parameterPath)
-        else
-          taskList.push({device : deviceId, name : 'refreshObject', objectName : parameterPath})
-      else
-        expiry = Math.min(expiry, diff)
-
-  # Objects
-  for parameterPath, object of devicePreset.objects ? {}
-    for objectName, objectDetails of object
-      continue if not objectDetails.current?
-      if objectDetails.preset?
-        flatObject = common.flattenObject(objectDetails.preset)
-        if Object.keys(objectDetails.current).length > 0
-          for i, obj of objectDetails.current
-            for paramName, paramValue of flatObject
-              continue if paramName[0] == '_'
-              currentValue = common.getParamValueFromPath(obj, paramName)
-              if currentValue?
-                presetValue = common.matchType(currentValue._value, flatObject[paramName])
-                if currentValue._value != presetValue
-                  setParameterValues.push(["#{parameterPath}.#{i}.#{paramName}", presetValue, currentValue._type])
-        else
-          vals = []
-          for k,v of flatObject
-            vals.push([k, v]) if k[0] != '_'
-          taskList.push({device : deviceId, name : 'addObject', objectName : parameterPath, parameterValues : vals})
-      else if Object.keys(objectDetails.current).length > 0
-        for i, obj of objectDetails.current
-          taskList.push({device : deviceId, name : 'deleteObject', objectName : "#{parameterPath}.#{i}"})
-
-  # Tags
-  for tag, v of devicePreset.tags ? {}
-    if v == 1
-      deleteTags.push(tag)
-    else if v == 2
-      addTags.push(tag)
-
-  # Custom commands
-  for commandName, commandDetails of devicePreset.customCommands ? {}
-    continue if not commandDetails.current?
-    expiry = Math.min(expiry, commandDetails.preset.expiry) if commandDetails.preset.expiry?
-    currentValue = commandDetails.current._value
-    presetValue = common.matchType(currentValue, commandDetails.preset.value)
-    if commandDetails.preset.value? and currentValue != presetValue
-      taskList.push({device : deviceId, name : 'customCommand', command : commandDetails.preset.valueCommand})
-    else if commandDetails.preset.expiry?
-      diff = commandDetails.preset.expiry - (now - (commandDetails.current._timestamp ? 0)) / 1000
-      if diff <= PRESETS_TIME_PADDING
-        taskList.push({device : deviceId, name : 'customCommand', command : commandDetails.preset.expiryCommand})
-      else
-        expiry = Math.min(expiry, diff)
-
-  if setParameterValues.length
-    taskList.push {device : deviceId, name : 'setParameterValues', parameterValues: setParameterValues, timestamp : now}
-
-  if getParameterValues.length
-    taskList.push {device : deviceId, name : 'getParameterValues', parameterNames: getParameterValues, timestamp : now}
-
-  callback(taskList, addTags, deleteTags, expiry)
-
-
-getObjectHash = (object) ->
-  if object._keys?.length
-    keys = object._keys
-  else
-    keys = (k for k in Object.keys(object) when k[0] != '_')
-
-  keys.sort()
-
-  hash = []
-  for k in keys
-    hash.push(k)
-    hash.push(common.getParamValueFromPath(object, k))
-
-  return JSON.stringify(hash)
-
-
-accumulateConfigurations = (presets, objects) ->
-  maxWeights = {}
-  configurations = {}
-  for p in presets
-    for c in p.configurations
-      configurationHash = switch c.type
-        when 'add_tag', 'delete_tag'
-          "tag_#{c.tag}"
-        when 'add_object', 'delete_object'
-          objectHash = getObjectHash(objects[c.object])
-          "object_#{c.name}_#{objectHash}"
-        when 'software_version'
-          'software_version'
-        when 'custom_command_age', 'custom_command_value'
-          [filename, commandName] = c.command.split(' ', 2)
-          "#{c.type}_#{filename}_#{commandName}"
-        else
-          "#{c.type}_#{c.name}"
-
-      if not maxWeights[configurationHash]? or p.weight > maxWeights[configurationHash]
-        configurations[configurationHash] = c
-        maxWeights[configurationHash] = p.weight
-
-  configurationsList = (configurations[c] for c of configurations)
-  return configurationsList

lib/query.coffee

@@ -1,4 +1,24 @@
 ###
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
+#
+# GenieACS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# GenieACS is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This file incorporates work covered by the following copyright and
+# permission notice:
+#
 # Copyright 2013 Fanoos Telecom
 #
 # Permission is hereby granted, free of charge, to any person obtaining a copy
@@ -21,18 +41,59 @@
 ###
 
 common = require './common'
-normalize = require './normalize'
 mongodb = require 'mongodb'
 
 
-expandValue = (param, value) ->
+stringToRegexp = (input, flags) ->
+  if (input.indexOf('*') == -1)
+    return false
+
+  output = input.replace(/[\[\]\\\^\$\.\|\?\+\(\)]/, "\\$&")
+  if output[0] == '*'
+    output = output.replace(/^\*+/g, '')
+  else
+    output = '^' + output
+
+  if output[output.length - 1] == '*'
+    output = output.replace(/\*+$/g, '')
+  else
+    output = output + '$'
+
+  output = output.replace(/[\*]/, '.*')
+
+  return new RegExp(output, flags)
+
+
+normalize = (input) ->
+  if common.typeOf(input) is common.STRING_TYPE
+    vals = [input]
+    if (m = /^\/(.*?)\/(g?i?m?y?)$/.exec(input))
+      vals.push({'$regex' : new RegExp(m[1], m[2])})
+
+    f = parseFloat(input)
+    if not isNaN(f)
+      vals.push(f)
+
+    d = new Date(input)
+    if input.length >= 8 and d.getFullYear() > 1983
+      vals.push(d)
+
+    r = stringToRegexp(input)
+    if r isnt false
+      vals.push({'$regex' : r})
+
+    return vals
+  return input
+
+
+expandValue = (value) ->
   if common.typeOf(value) is common.ARRAY_TYPE
     a = []
     for j in value
-      a = a.concat(expandValue(param, j))
+      a = a.concat(expandValue(j))
     return [a]
   else if common.typeOf(value) isnt common.OBJECT_TYPE
-    n = normalize.normalize(param, value, 'query')
+    n = normalize(value)
     if common.typeOf(n) isnt common.ARRAY_TYPE
       return [n]
     else
@@ -44,7 +105,7 @@ expandValue = (param, value) ->
   values = []
   for k,v of value
     keys.push(k)
-    values.push(expandValue(param, v))
+    values.push(expandValue(v))
     indices.push(0)
 
   i = 0
@@ -62,72 +123,249 @@ expandValue = (param, value) ->
   return objs
 
 
-permute = (param, val, aliases) ->
+permute = (param, val) ->
-  keys = []
-  if aliases[param]?
-    for p in aliases[param]
-      keys.push(p)
-  else
-    keys.push(param)
-
   conditions = []
-  for k in keys
+
-    values = expandValue(k, val)
+  values = expandValue(val)
-    if k[k.lastIndexOf('.') + 1] != '_'
+  if param[param.lastIndexOf('.') + 1] != '_'
-      k += '._value'
+    param += '._value'
 
   for v in values
     obj = {}
-      obj[k] = v
+    obj[param] = v
     conditions.push(obj)
 
   return conditions
 
 
-expand = (query, aliases) ->
+expand = (query) ->
   new_query = {}
   for k,v of query
     if k[0] == '$' # operator
       expressions = []
       for e in v
-        expressions.push(expand(e, aliases))
+        expressions.push(expand(e))
       new_query[k] = expressions
     else
-      conditions = permute(k, v, aliases)
+      conditions = permute(k, v)
       if conditions.length > 1
-        if new_query['$and']?
+        new_query['$and'] ?= []
+        if v?['$ne']?
+          for c in conditions
+            new_query['$and'].push(c)
+        else
           new_query['$and'].push({'$or' : conditions})
       else
-          new_query['$and'] = [{'$or' : conditions}]
+        Object.assign(new_query, conditions[0])
-      else
-        common.extend(new_query, conditions[0])
 
   return new_query
 
-# Replace _id string values with ObjectID type
+
-substituteObjectId = (query) ->
+# Generate parameter projection from a query
+# If second arg is given, it's edited and returned
+queryProjection = (query, proj) ->
+  proj ?= {}
+  for k,v of query
+    if k.charAt(0) == '$' # this is a logical operator
+      for q in v
+        queryProjection(q, proj)
+    else
+      proj[k] = 1
+  return proj
+
+
+testExpressions = (params, expressions, lop) ->
+  for f in expressions
+    res = test(params, f)
+
+    switch lop
+      when '$and'
+        return false if not res
+      when '$or'
+        return true if res
+      when '$nor'
+        return false if res
+
+  return switch lop
+    when '$and' then true
+    when '$or' then false
+    when '$nor' then true
+    else throw new Error('Unknown logical operator')
+
+
+test = (params, query) ->
+  for k,v of query
+    if k.charAt(0) == '$' # this is a logical operator
+      res = testExpressions(params, v, k)
+    else
+      value = params[k]
+
+      if common.typeOf(v) isnt common.OBJECT_TYPE
+        # TODO comparing array to regex, array to array, and object to object
+        if common.typeOf(value) is common.ARRAY_TYPE
+          res = value.indexOf(v) != -1
+        else
+          if common.typeOf(v) is common.REGEXP_TYPE
+            res = v.test(value)
+          else
+            res = v == value
+      else
+        for k2,v2 of v
+          switch k2
+            when '$ne'
+              if common.typeOf(value) is common.ARRAY_TYPE
+                res = value.indexOf(v2) == -1
+              else
+                res = value != v2
+            when '$lt'
+              res = value < v2
+            when '$lte'
+              res = value <= v2
+            when '$gt'
+              res = value > v2
+            when '$gte'
+              res = value >= v2
+            when '$regex'
+              res = v2.test(value)
+            when '$in'
+              throw new Error('Operator not supported')
+            when '$nin'
+              throw new Error('Operator not supported')
+            when '$all'
+              throw new Error('Operator not supported')
+            when '$exists'
+              throw new Error('Operator not supported')
+            else
+              throw new Error('Operator not supported')
+
+    if not res
+      return false
+
+  return true
+
+
+matchType = (src, dst) ->
+  if typeof src is 'string'
+    return String(dst)
+  else if typeof src is 'number'
+    if +dst == parseFloat(dst)
+      return +dst
+    else if not isNaN(Date.parse(dst))
+      return Date.parse(dst)
+  else if typeof src is 'boolean'
+    v = String(dst).trim()
+    if v is 'true' or v is 'TRUE' or v is 'True' or v is '1'
+      return true
+    else if v is 'false' or v is 'FALSE' or v is 'False' or v is '0'
+      return false
+
+  return dst
+
+
+testFilter = (obj, filter) ->
+  for k, v of filter
+    [param, op] = k.split(/([^a-zA-Z0-9\-\_\.].*)/, 2)
+    val = matchType(obj[param], v)
+    switch op
+      when '=', undefined
+        return false if obj[param] != val
+      when '>'
+        return false if not (obj[param] > val)
+      when '>='
+        return false if not (obj[param] >= val)
+      when '<'
+        return false if not (obj[param] < val)
+      when '<='
+        return false if not (obj[param] <= val)
+      when '!'
+        return false if obj[param] == val
+      else
+        throw new Error("Unrecognized operator #{op}")
+
+  return true
+
+
+convertMongoQueryToFilters = (query, filters) ->
+  filters ?= {}
+  for k, v of query
+    if k[0] == '$'
+      if k == '$and'
+        for vv in v
+          convertMongoQueryToFilters(vv, filters)
+      else
+        throw new Error("Operator #{k} not supported")
+    else if k == '_tags'
+      if common.typeOf(v) is common.OBJECT_TYPE
+        for kk, vv of v
+          vv = vv.replace(/[^a-zA-Z0-9\-]+/g, '_')
+          switch kk
+            when '$ne'
+              filters["Tags.#{vv}!"] = true
+            else
+              throw new Error("Operator #{kk} not supported")
+      else
+        v = v.replace(/[^a-zA-Z0-9\-]+/g, '_')
+        filters["Tags.#{v}"] = true
+    else
+      switch k
+        when '_id' then k = 'DeviceID.ID'
+        when '_deviceId._Manufacturer' then k = 'DeviceID.Manufacturer'
+        when '_deviceId._OUI' then k = 'DeviceID.OUI'
+        when '_deviceId._ProductClass' then k = 'DeviceID.ProductClass'
+        when '_deviceId._SerialNumber' then k = 'DeviceID.SerialNumber'
+        when '_lastInform' then k = 'Events.Inform'
+        when '_lastBootstrap' then k = 'Events.0_BOOTSTRAP'
+        when '_lastBoot' then k = 'Events.1_BOOT'
+        when '_registered' then k = 'Events.Registered'
+
+      if common.typeOf(v) is common.OBJECT_TYPE
+        for kk, vv of v
+          switch kk
+            when '$eq'
+              filters[k] = vv
+            when '$ne'
+              filters["#{k}!"] = vv
+            when '$lt'
+              filters["#{k}<"] = vv
+            when '$lte'
+              filters["#{k}<="] = vv
+            when '$gt'
+              filters["#{k}>"] = vv
+            when '$gte'
+              filters["#{k}>="] = vv
+            else
+              throw new Error("Oprator #{kk} not supported")
+      else
+        filters[k] = v
+
+  return filters
+
+
+sanitizeQueryTypes = (query, types) ->
   for k, v of query
     if k[0] == '$' # logical operator
-      for i in [0...v.length]
+      sanitizeQueryTypes(vv, types) for vv in v
-        substituteObjectId(v[i])
+    else if k of types
-    else if k == '_id'
+      if common.typeOf(v) is common.OBJECT_TYPE
-      if common.typeOf(v) is common.STRING_TYPE
-        query[k] = mongodb.ObjectID(v) if v.length == 24
-      else if common.typeOf(v) is common.OBJECT_TYPE
         for kk, vv of v
           switch kk # operator
             when '$in', '$nin'
-              for i in [0...vv.length]
+              vv[i] = types[k](vv[i]) for i in [0...vv.length]
-                vv[i] = mongodb.ObjectID(vv[i]) if vv[i].length == 24
             when '$eq', '$gt', '$gte', '$lt', '$lte', '$ne'
-              v[kk] = mongodb.ObjectID(vv) if vv.length == 24
+              v[kk] = types[k](vv)
             when '$exists', '$type' then
               # ignore
             else
               throw new Error('Operator not supported')
+      else
+        query[k] = types[k](query[k])
 
   return query
 
 
 exports.expand = expand
-exports.substituteObjectId = substituteObjectId
+exports.queryProjection = queryProjection
+exports.test = test
+exports.sanitizeQueryTypes = sanitizeQueryTypes
+exports.convertMongoQueryToFilters = convertMongoQueryToFilters
+exports.testFilter = testFilter

lib/sandbox.js

@@ -0,0 +1,328 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const vm = require("vm");
+
+const seedrandom = require("seedrandom");
+
+const common = require("./common");
+const device = require("./device");
+const extensions = require("./extensions");
+const logger = require("./logger");
+
+
+// Used for throwing to exit user script and commit
+const COMMIT = Symbol();
+
+// Used to execute extensions and restart
+const EXT = Symbol();
+
+const context = vm.createContext();
+
+let state;
+
+
+class SandboxDate {
+  constructor(arg) {
+    if (arguments.length)
+      return new (Function.prototype.bind.apply(Date, arguments));
+
+    return new Date(state.sessionContext.timestamp);
+  }
+
+  static now() {
+    return state.sessionContext.timestamp;
+  }
+};
+
+function random() {
+  if (!state.rng)
+    state.rng = seedrandom(sessionContext.timestamp);
+
+  return state.rng();
+}
+
+
+class ParameterWrapper {
+  constructor(path, attributes, unpacked, unpackedRevision) {
+    for (let attrName of attributes) {
+      Object.defineProperty(this, attrName, {get: function() {
+        if (state.uncommitted)
+          commit();
+
+        if (state.revision !== unpackedRevision) {
+          unpackedRevision = state.revision;
+          unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision + 1);
+        }
+
+        if (!unpacked.length)
+          return undefined;
+
+        let attr = state.sessionContext.deviceData.attributes.get(unpacked[0], state.revision + 1)[attrName];
+
+        if (!attr)
+          return undefined;
+
+        return attr[1];
+      }});
+    }
+
+    Object.defineProperty(this, "path", {get: function() {
+      if (state.uncommitted)
+        commit();
+
+      if (state.revision !== unpackedRevision) {
+        unpackedRevision = state.revision;
+        unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision + 1);
+      }
+
+      if (!unpacked.length)
+        return undefined;
+
+      return unpacked[0].join(".");
+    }});
+
+    Object.defineProperty(this, "size", {get: function() {
+      if (state.uncommitted)
+        commit();
+
+      if (state.revision !== unpackedRevision) {
+        unpackedRevision = state.revision;
+        unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision + 1);
+      }
+
+      if (!unpacked.length)
+        return undefined;
+
+      return unpacked.length;
+    }});
+
+    this[Symbol.iterator] = function*() {
+      if (state.uncommitted)
+        commit();
+
+      if (state.revision !== unpackedRevision) {
+        unpackedRevision = state.revision;
+        unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision + 1);
+      }
+
+      for (let p of unpacked)
+        yield new ParameterWrapper(p, attributes, [p], state.revision);
+    }
+  }
+}
+
+
+function declare(path, timestamps, values) {
+  state.uncommitted = true;
+  if (!timestamps)
+    timestamps = {};
+
+  if (!values)
+    values = {};
+
+  let parsedPath = common.parsePath(path);
+
+  let declaration = [parsedPath, 1];
+  let attrs = {};
+
+  for (let attrName in values) {
+    if (!timestamps[attrName])
+      timestamps[attrName] = 1;
+
+    if (attrName === "path")
+      declaration[3] = values.path;
+    else {
+      attrs[attrName] = 1;
+      declaration[4] = declaration[4] || {};
+      if (attrName === "value" && !Array.isArray(values.value))
+        declaration[4].value = [values.value];
+      else
+        declaration[4][attrName] = values[attrName];
+    }
+  }
+
+  for (let attrName in timestamps) {
+    if (attrName === "path")
+      declaration[1] = timestamps.path;
+    else {
+      attrs[attrName] = 1;
+      declaration[2] = declaration[2] || {};
+      declaration[2][attrName] = timestamps[attrName];
+    }
+  }
+
+  state.declarations.push(declaration);
+
+  return new ParameterWrapper(parsedPath, Object.keys(attrs));
+}
+
+
+function clear(path, timestamp, attributes) {
+  state.uncommitted = true;
+
+  if (state.revision === state.maxRevision)
+    state.clear.push([common.parsePath(path), timestamp, attributes]);
+}
+
+
+function commit() {
+  ++ state.revision;
+  state.uncommitted = false;
+
+  if (state.revision === state.maxRevision + 1)
+    throw COMMIT;
+  else if (state.revision > state.maxRevision + 1)
+    throw new Error("Declare function should not be called from within a try/catch block");
+}
+
+
+function ext() {
+  ++ state.extCounter;
+  let extCall = Array.from(arguments).map(String);
+  let key = `${state.revision}: ${JSON.stringify(extCall)}`;
+
+  if (key in state.sessionContext.extensionsCache)
+    return state.sessionContext.extensionsCache[key];
+
+  state.extensions[key] = extCall;
+  throw EXT;
+}
+
+
+function log(msg, meta) {
+  if (state.revision === state.maxRevision && state.extCounter >= 0) {
+    const details = Object.assign({}, meta, {
+      sessionContext: state.sessionContext,
+      message: `Script: ${msg}`
+    });
+
+    delete details.hostname;
+    delete details.pid;
+    delete details.name;
+    delete details.version;
+    delete details.deviceId;
+    delete details.remoteAddress;
+
+    logger.accessInfo(details);
+  }
+}
+
+
+Object.defineProperty(context, "Date", {value: SandboxDate});
+Object.defineProperty(context, "declare", {value: declare});
+Object.defineProperty(context, "clear", {value: clear});
+Object.defineProperty(context, "commit", {value: commit});
+Object.defineProperty(context, "ext", {value: ext});
+Object.defineProperty(context, "log", {value: log});
+
+// Monkey-patch Math.random() to make it deterministic
+context.random = random;
+vm.runInContext("Math.random = random;", context);
+delete context.random;
+
+
+function errorToFault(err) {
+  if (!err)
+    return null;
+
+  // Trim the stack trace at the self-executing anonymous wrapper function
+  let stackTrimIndex = err.stack.match(/\s+at\s[^\s]+\s+at\s[^\s]+\s\(vm\.js.+\)/).index;
+
+  return {
+    code: `script.${err.name}`,
+    message: err.message,
+    detail: {
+      name: err.name,
+      message: err.message,
+      stack: err.stack.slice(0, stackTrimIndex)
+    }
+  }
+}
+
+
+function run(script, globals, sessionContext, startRevision, maxRevision, callback, extCounter = 0) {
+  state = {
+    sessionContext: sessionContext,
+    revision: startRevision,
+    maxRevision: maxRevision,
+    uncommitted: false,
+    declarations: [],
+    extensions: {},
+    clear: [],
+    rng: null,
+    extCounter: extCounter
+  };
+
+  for (let n of Object.keys(context))
+    delete context[n];
+
+  Object.assign(context, globals);
+
+  let ret, status;
+
+  try {
+    ret = script.runInContext(context, {displayErrors: false});
+    status = 0;
+  }
+  catch (err) {
+    if (err === COMMIT)
+      status = 1;
+    else if (err === EXT)
+      status = 2;
+    else
+      return callback(null, errorToFault(err));
+  }
+
+  let _state = state;
+  let args = Array.from(arguments);
+
+  // Need to maintain a counter of ext() calls to avoid calling certain
+  // functions (e.g. log()) multiple times as calling ext() results in
+  // re-execution of script without revision being incremented.
+  args[6] = 0 - (state.extCounter - extCounter);
+
+  let counter = 3;
+  for (let key of Object.keys(_state.extensions)) {
+    counter += 2;
+    extensions.run(_state.extensions[key], function(err, fault, res) {
+      if (err || fault) {
+        if (counter & 1)
+          callback(err, fault);
+        return counter = 0;
+      }
+      _state.sessionContext.extensionsCache[key] = res;
+      if ((counter -= 2) === 1)
+        if (status === 2)
+          return run.apply(null, args);
+        else
+          return callback(null, null, _state.clear, _state.declarations, status === 0, ret);
+    });
+  }
+
+  if ((counter -= 2) === 1)
+    if (status === 2)
+      return run.apply(null, args);
+    else
+      return callback(null, null, _state.clear, _state.declarations, status === 0, ret);
+}
+
+
+exports.run = run;

lib/server.coffee

@@ -1,5 +1,7 @@
 ###
-# Copyright 2013, 2014  Zaid Abdulla
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
 #
 # GenieACS is free software: you can redistribute it and/or modify
 # it under the terms of the GNU Affero General Public License as
@@ -15,53 +17,54 @@
 # along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
 ###
 
-domain = require 'domain'
+service = process.argv[2]
-util = require 'util'
-cluster = require 'cluster'
-config = require './config'
-db = require './db'
-
-service = config.argv[0]
 if not service?
   throw new Error('Missing argument cwmp, fs, or nbi')
 
+util = require 'util'
+cluster = require 'cluster'
+logger = require './logger'
+logger.init(service, require('../package.json').version)
+config = require './config'
+db = require './db'
+extensions = require './extensions'
+
 networkInterface = config.get("#{service.toUpperCase()}_INTERFACE")
 port = config.get("#{service.toUpperCase()}_PORT")
 useHttps = config.get("#{service.toUpperCase()}_SSL")
 
-serviceListener = require("./#{service}").listener
+listener = require("./#{service}").listener
+onConnection = require("./#{service}").onConnection
 
 server = null
 
-listener = (httpRequest, httpResponse) ->
+exit = () ->
-  d = domain.create()
+  setTimeout(() ->
-
-  d.on('error', (err) ->
-    util.error("#{new Date().toISOString()} - #{err.stack}\n")
-    try
-      killTimer = setTimeout(() ->
     process.exit(1)
-      , 30000)
+  , 30000).unref()
-
-      killTimer.unref()
 
   cluster.worker?.disconnect()
+
+  if not server
+    db.disconnect()
+    extensions.killAll()
+    logger.close()
+    return
+
   server.close(() ->
     db.disconnect()
+    extensions.killAll()
+    logger.close()
   )
 
-      httpResponse.writeHead(500, {'Connection' : 'close'})
-      httpResponse.end("#{err.name}: #{err.message}")
-    catch err2
-      util.error("#{new Date().toISOString()} - #{err2.stack}\n")
-  )
 
-  d.add(httpRequest)
+process.on('uncaughtException', (err) ->
-  d.add(httpResponse)
+  logger.error({
-
+    message: 'Uncaught exception'
-  d.run(() ->
+    exception: err
-    serviceListener(httpRequest, httpResponse)
+  })
-  )
+  exit()
+)
 
 
 if useHttps
@@ -69,15 +72,28 @@ if useHttps
   fs = require 'fs'
   httpsKey = path.resolve(config.get('CONFIG_DIR'), "#{service}.key")
   httpsCert = path.resolve(config.get('CONFIG_DIR'), "#{service}.crt")
+  httpsCa = path.resolve(config.get('CONFIG_DIR'), "#{service}.ca-bundle")
   options = {
     key: fs.readFileSync(httpsKey),
     cert: fs.readFileSync(httpsCert)
   }
+
+  try
+    # Use intermediate certificates if available
+    options.ca = fs.readFileSync(httpsCa).toString()
+      .match(/\-+BEGIN CERTIFICATE\-+[0-9a-zA-Z\+\-\/\=\s]+?\-+END CERTIFICATE\-+/g)
+
   server = require('https').createServer(options, listener)
+  server.on('secureConnection', onConnection) if onConnection?
 else
   server = require('http').createServer(listener)
+  server.on('connection', onConnection) if onConnection?
 
 db.connect((err) ->
   throw err if err
   server.listen(port, networkInterface)
 )
+
+
+process.on('SIGINT', exit)
+process.on('SIGTERM', exit)

lib/soap.coffee

@@ -1,433 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-#
-# This file incorporates work covered by the following copyright and
-# permission notice:
-#
-# Copyright 2013 Fanoos Telecom
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-###
-
-config = require './config'
-libxmljs = require 'libxmljs'
-
-SERVER_NAME = "GenieACS/#{require('../package.json').version}"
-
-NAMESPACES = {
-  '1.0' : {
-    'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-    'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-    'xsd' : 'http://www.w3.org/2001/XMLSchema',
-    'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-    'cwmp' : 'urn:dslforum-org:cwmp-1-0'
-  },
-  '1.1' : {
-    'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-    'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-    'xsd' : 'http://www.w3.org/2001/XMLSchema',
-    'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-    'cwmp' : 'urn:dslforum-org:cwmp-1-1'
-  },
-  '1.2' : {
-    'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-    'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-    'xsd' : 'http://www.w3.org/2001/XMLSchema',
-    'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-    'cwmp' : 'urn:dslforum-org:cwmp-1-2'
-  },
-  '1.3' : {
-    'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-    'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-    'xsd' : 'http://www.w3.org/2001/XMLSchema',
-    'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-    'cwmp' : 'urn:dslforum-org:cwmp-1-2'
-  },
-  '1.4' : {
-    'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-    'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-    'xsd' : 'http://www.w3.org/2001/XMLSchema',
-    'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-    'cwmp' : 'urn:dslforum-org:cwmp-1-3'
-  }
-}
-
-# Generate Libxmljs options
-LIBXMLJS_OPTIONS = {}
-for k, v of config.allConfig
-  if k.lastIndexOf('XML_PARSE_', 0) == 0
-    LIBXMLJS_OPTIONS[k] = v
-
-
-# Workaround for devices that don't use correct namespace declarations or prefixes
-if config.get('IGNORE_XML_NAMESPACES')
-  libxmljs.Element.prototype.__find = libxmljs.Element.prototype.find
-  libxmljs.Element.prototype.find = (xpath, namespaces) ->
-    # Modify xpath queries to work regardless of element's namespace
-    p = xpath.replace(/([^\/:]+:)?([^\/]+)(\/|$)/g, '$2$3') \
-      .replace(/([a-zA-Z0-9_-]+)([^\/]*)(\/|$)/g, "*[local-name()='$1']$2$3")
-    libxmljs.Element.prototype.__find.call(this, p, namespaces)
-
-
-event = (xml) ->
-  e.text().trim() for e in xml.find('EventStruct/EventCode')
-
-
-parameterInfoList = (xml) ->
-  infoList = []
-  for e in xml.find('ParameterInfoStruct')
-    name = e.get('Name').text().trim()
-    writable = Boolean(JSON.parse(e.get('Writable').text()))
-    infoList.push([name, writable])
-  return infoList
-
-
-parameterValueList = (xml) ->
-  valueList = []
-  for e in xml.find('ParameterValueStruct')
-    valueType = e.get('Value').attr('type').value().trim()
-    name = e.get('Name').text().trim()
-    value = e.get('Value').text().trim()
-    try
-      switch valueType
-        when 'xsd:boolean'
-          value = Boolean(JSON.parse(value))
-        when 'xsd:unsignedInt'
-          value = JSON.parse(value)
-        when 'xsd:int'
-          value = JSON.parse(value)
-    catch err
-      value = null
-    valueList.push([name, value, valueType])
-  return valueList
-
-
-cpeGetParameterNames = (xml, methodRequest) ->
-  el = xml.node('cwmp:GetParameterNames')
-  el.node('ParameterPath').text(methodRequest.parameterPath)
-  el.node('NextLevel').text(+methodRequest.nextLevel)
-
-
-cpeGetParameterNamesResponse = (xml) ->
-  {parameterList : parameterInfoList(xml.get('ParameterList'))}
-
-
-cpeGetParameterValues = (xml, methodRequest) ->
-  el = xml.node('cwmp:GetParameterValues').node('ParameterNames')
-  el.attr({'soap-enc:arrayType' : "xsd:string[#{methodRequest.parameterNames.length}]"})
-  for p in methodRequest.parameterNames
-    el.node('string').text(p)
-
-
-cpeGetParameterValuesResponse = (xml) ->
-  {parameterList : parameterValueList(xml.get('ParameterList'))}
-
-
-cpeSetParameterValues = (xml, methodRequest) ->
-  el = xml.node('cwmp:SetParameterValues')
-  paramList = el.node('ParameterList')
-  paramList.attr({'soap-enc:arrayType' : "cwmp:ParameterValueStruct[#{methodRequest.parameterList.length}]"})
-  for i in methodRequest.parameterList
-    pvs = paramList.node('ParameterValueStruct')
-    pvs.node('Name').text(i[0])
-    v = pvs.node('Value')
-    v.text(i[1])
-    v.attr({'xsi:type' : i[2]}) if i[2]?
-  el.node('ParameterKey').text(methodRequest.parameterKey ? '')
-
-
-cpeSetParameterValuesResponse = (xml) ->
-  {status : JSON.parse(xml.get('Status').text())}
-
-
-cpeAddObject = (xml, methodRequest) ->
-  el = xml.node('cwmp:AddObject')
-  el.node('ObjectName').text(methodRequest.objectName)
-  el.node('ParameterKey').text(methodRequest.parameterKey ? '')
-
-
-cpeAddObjectResponse = (xml) ->
-  {
-    instanceNumber : parseInt(xml.get('InstanceNumber').text()),
-    status : parseInt(xml.get('Status').text())
-  }
-
-
-cpeDeleteObject = (xml, methodRequest) ->
-  el = xml.node('cwmp:DeleteObject')
-  el.node('ObjectName').text(methodRequest.objectName)
-  el.node('ParameterKey').text(methodRequest.parameterKey ? '')
-
-
-cpeDeleteObjectResponse = (xml) ->
-  {status : parseInt(xml.get('Status').text())}
-
-
-cpeReboot = (xml, methodRequest) ->
-  el = xml.node('cwmp:Reboot')
-  el.node('CommandKey').text(methodRequest.commandKey ? '')
-
-
-cpeRebootResponse = (xml) ->
-  {}
-
-
-cpeFactoryReset = (xml, methodRequest) ->
-  el = xml.node('cwmp:FactoryReset')
-
-
-cpeFactoryResetResponse = (xml, methodRequest) ->
-  {}
-
-
-cpeDownload = (xml, methodRequest) ->
-  el = xml.node('cwmp:Download')
-  el.node('CommandKey').text(methodRequest.commandKey ? '')
-  el.node('FileType').text(methodRequest.fileType)
-  el.node('URL').text(methodRequest.url)
-  el.node('Username').text(methodRequest.username ? '')
-  el.node('Password').text(methodRequest.password ? '')
-  el.node('FileSize').text(methodRequest.fileSize ? '0')
-  el.node('TargetFileName').text(methodRequest.TargetFileName ? '')
-  el.node('DelaySeconds').text(methodRequest.delaySeconds ? '0')
-  el.node('SuccessURL').text(methodRequest.successUrl ? '')
-  el.node('FailureURL').text(methodRequest.failureUrl ? '')
-
-
-cpeDownloadResponse = (xml) ->
-  {status : JSON.parse(xml.get('Status').text())}
-
-
-acsInform = (xml) ->
-  {
-    parameterList : parameterValueList(xml.get('ParameterList')),
-    deviceId : traverseXml(xml.get('DeviceId')),
-    event : event(xml.get('Event')),
-    retryCount : parseInt(xml.get('RetryCount').text())
-  }
-
-
-acsInformResponse = (xml, methodResponse) ->
-  xml.node('cwmp:InformResponse').node('MaxEnvelopes').text(1)
-
-
-acsGetRPCMethods = (xml) ->
-  {}
-
-
-acsGetRPCMethodsResponse = (xml, methodResponse) ->
-  el = xml.node('cwmp:GetRPCMethodsResponse').node('MethodList')
-  el.attr({'soap-enc:arrayType' : "xsd:string[#{methodResponse.methodList.length}]"})
-  for m in methodResponse.methodList
-    el.node('string').text(m)
-
-
-acsTransferComplete = (xml) ->
-  {
-    commandKey : xml.get('CommandKey').text(),
-    faultStruct : traverseXml(xml.get('FaultStruct')),
-    startTime : xml.get('StartTime').text(), # TODO convert to datetime
-    completeTime : xml.get('CompleteTime').text() # TODO convert to datetime
-  }
-
-
-acsTransferCompleteResponse = (xml, methodResponse) ->
-  xml.node('cwmp:TransferCompleteResponse').text('')
-
-
-acsRequestDownload = (xml) ->
-  # TODO FileTypeArg
-  {
-    fileType : xml.get('FileType').text()
-  }
-
-
-acsRequestDownloadResponse = (xml, methodResponse) ->
-  xml.node('cwmp:RequestDownloadResponse').text('')
-
-
-traverseXml = (xml) ->
-  obj = {}
-  for n in xml.childNodes()
-    if n.type() == 'element'
-      obj[n.name()] = traverseXml(n)
-
-  if Object.keys(obj).length == 0
-    xml.text()
-  else
-    obj
-
-
-fault = (xml) ->
-  return traverseXml(xml)
-
-
-exports.request = (httpRequest, cwmpVersion) ->
-  cwmpRequest = {cwmpVersion : cwmpVersion}
-
-  data = httpRequest.getBody()
-
-  if data.length > 0
-    xml = libxmljs.parseXml(data, LIBXMLJS_OPTIONS)
-
-    if not cwmpRequest.cwmpVersion?
-      # cwmpVersion not passed, thus it's an inform request
-      methodElement = xml.get('/soap-env:Envelope/soap-env:Body/*', NAMESPACES['1.0'])
-      switch methodElement.namespace().href()
-        when 'urn:dslforum-org:cwmp-1-0'
-          cwmpRequest.cwmpVersion = '1.0'
-        when 'urn:dslforum-org:cwmp-1-1'
-          cwmpRequest.cwmpVersion = '1.1'
-        when 'urn:dslforum-org:cwmp-1-2'
-          cwmpRequest.sessionTimeout = try xml.get('/soap-env:Envelope/soap-env:Header/cwmp:sessionTimeout', NAMESPACES['1.2']).text() catch then null
-          if cwmpRequest.sessionTimeout?
-            cwmpRequest.cwmpVersion = '1.3'
-          else
-            cwmpRequest.cwmpVersion = '1.2'
-        when 'urn:dslforum-org:cwmp-1-3'
-          cwmpRequest.cwmpVersion = '1.4'
-        else
-          throw new Error('Unrecognized CWMP version')
-
-      cwmpRequest.sessionTimeout ?= try xml.get('/soap-env:Envelope/soap-env:Header/cwmp:sessionTimeout', NAMESPACES[cwmpRequest.cwmpVersion]).text() catch then null
-    else
-      methodElement = xml.get('/soap-env:Envelope/soap-env:Body/cwmp:*', NAMESPACES[cwmpRequest.cwmpVersion])
-
-    cwmpRequest.id = try xml.get('/soap-env:Envelope/soap-env:Header/cwmp:ID', NAMESPACES[cwmpRequest.cwmpVersion]).text() catch then null
-
-    if methodElement? and not (config.get('IGNORE_XML_NAMESPACES') and methodElement.name() is 'Fault')
-      switch methodElement.name()
-        when 'Inform'
-          cwmpRequest.methodRequest = acsInform(methodElement)
-          cwmpRequest.methodRequest.type = 'Inform'
-        when 'GetRPCMethods'
-          cwmpRequest.methodRequest = acsGetRPCMethods(methodElement)
-          cwmpRequest.methodRequest.type = 'GetRPCMethods'
-        when 'TransferComplete'
-          cwmpRequest.methodRequest = acsTransferComplete(methodElement)
-          cwmpRequest.methodRequest.type = 'TransferComplete'
-        when 'RequestDownload'
-          cwmpRequest.methodRequest = acsRequestDownload(methodElement)
-          cwmpRequest.methodRequest.type = 'RequestDownload'
-        when 'GetParameterNamesResponse'
-          cwmpRequest.methodResponse = cpeGetParameterNamesResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'GetParameterNamesResponse'
-        when 'GetParameterValuesResponse'
-          cwmpRequest.methodResponse = cpeGetParameterValuesResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'GetParameterValuesResponse'
-        when 'SetParameterValuesResponse'
-          cwmpRequest.methodResponse = cpeSetParameterValuesResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'SetParameterValuesResponse'
-        when 'AddObjectResponse'
-          cwmpRequest.methodResponse = cpeAddObjectResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'AddObjectResponse'
-        when 'DeleteObjectResponse'
-          cwmpRequest.methodResponse = cpeDeleteObjectResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'DeleteObjectResponse'
-        when 'RebootResponse'
-          cwmpRequest.methodResponse = cpeRebootResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'RebootResponse'
-        when 'FactoryResetResponse'
-          cwmpRequest.methodResponse = cpeFactoryResetResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'FactoryResetResponse'
-        when 'DownloadResponse'
-          cwmpRequest.methodResponse = cpeDownloadResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'DownloadResponse'
-        else
-          throw Error('8000 Method not supported ' + methodElement.name())
-    else
-      faultElement = xml.get('/soap-env:Envelope/soap-env:Body/soap-env:Fault', NAMESPACES[cwmpRequest.cwmpVersion])
-      cwmpRequest.fault = fault(faultElement)
-    
-  return cwmpRequest
-
-
-createSoapEnv = (cwmpVersion) ->
-  xml = libxmljs.Document()
-  env = xml.node('soap-env:Envelope')
-  for prefix, url of NAMESPACES[cwmpVersion]
-    env.defineNamespace prefix, url
-  return env
-
-
-exports.response = (cwmpResponse) ->
-  headers = {
-    'Server' : SERVER_NAME,
-    'SOAPServer' : SERVER_NAME,
-  }
-
-  if cwmpResponse?.methodResponse?
-    env = createSoapEnv(cwmpResponse.cwmpVersion)
-    header = env.node('soap-env:Header')
-    header.node('cwmp:ID').attr({'soap-env:mustUnderstand' : 1}).text(cwmpResponse.id)
-    body = env.node('soap-env:Body')
-    switch cwmpResponse.methodResponse.type
-      when 'InformResponse'
-        acsInformResponse(body, cwmpResponse.methodResponse)
-      when 'GetRPCMethodsResponse'
-        acsGetRPCMethodsResponse(body, cwmpResponse.methodResponse)
-      when 'TransferCompleteResponse'
-        acsTransferCompleteResponse(body, cwmpResponse.methodResponse)
-      when 'RequestDownloadResponse'
-        acsRequestDownloadResponse(body, cwmpResponse.methodResponse)
-      else
-        throw Error("Unknown method response type #{cwmpResponse.methodResponse.type}")
-  else if cwmpResponse?.methodRequest?
-    env = createSoapEnv(cwmpResponse.cwmpVersion)
-    header = env.node('soap-env:Header')
-    header.node('cwmp:ID').attr({'soap-env:mustUnderstand' : 1}).text(cwmpResponse.id)
-    body = env.node('soap-env:Body')
-    switch cwmpResponse.methodRequest.type
-      when 'GetParameterNames'
-        cpeGetParameterNames(body, cwmpResponse.methodRequest)
-      when 'GetParameterValues'
-        cpeGetParameterValues(body, cwmpResponse.methodRequest)
-      when 'SetParameterValues'
-        cpeSetParameterValues(body, cwmpResponse.methodRequest)
-      when 'AddObject'
-        cpeAddObject(body, cwmpResponse.methodRequest)
-      when 'DeleteObject'
-        cpeDeleteObject(body, cwmpResponse.methodRequest)
-      when 'Reboot'
-        cpeReboot(body, cwmpResponse.methodRequest)
-      when 'FactoryReset'
-        cpeFactoryReset(body, cwmpResponse.methodRequest)
-      when 'Download'
-        cpeDownload(body, cwmpResponse.methodRequest)
-      else
-        throw Error("Unknown method request #{cwmpResponse.methodRequest.type}")
-  
-  if env?
-    headers['Content-Type'] = 'text/xml; charset="utf-8"'
-    return {code: 200, headers: headers, data: new Buffer(env.doc().toString(false))}
-  else
-    return {code: 204, headers: headers, data: new Buffer(0)}

lib/soap.js

@@ -0,0 +1,603 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const libxmljs = require("libxmljs");
+
+const config = require("./config");
+
+const SERVER_NAME = `GenieACS/${require("../package.json").version}`;
+
+const NAMESPACES = {
+  "1.0": {
+    "soap-enc": "http://schemas.xmlsoap.org/soap/encoding/",
+    "soap-env": "http://schemas.xmlsoap.org/soap/envelope/",
+    "xsd": "http://www.w3.org/2001/XMLSchema",
+    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
+    "cwmp": "urn:dslforum-org:cwmp-1-0"
+  },
+  "1.1": {
+    "soap-enc": "http://schemas.xmlsoap.org/soap/encoding/",
+    "soap-env": "http://schemas.xmlsoap.org/soap/envelope/",
+    "xsd": "http://www.w3.org/2001/XMLSchema",
+    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
+    "cwmp": "urn:dslforum-org:cwmp-1-1"
+  },
+  "1.2": {
+    "soap-enc": "http://schemas.xmlsoap.org/soap/encoding/",
+    "soap-env": "http://schemas.xmlsoap.org/soap/envelope/",
+    "xsd": "http://www.w3.org/2001/XMLSchema",
+    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
+    "cwmp": "urn:dslforum-org:cwmp-1-2"
+  },
+  "1.3": {
+    "soap-enc": "http://schemas.xmlsoap.org/soap/encoding/",
+    "soap-env": "http://schemas.xmlsoap.org/soap/envelope/",
+    "xsd": "http://www.w3.org/2001/XMLSchema",
+    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
+    "cwmp": "urn:dslforum-org:cwmp-1-2"
+  },
+  "1.4": {
+    "soap-enc": "http://schemas.xmlsoap.org/soap/encoding/",
+    "soap-env": "http://schemas.xmlsoap.org/soap/envelope/",
+    "xsd": "http://www.w3.org/2001/XMLSchema",
+    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
+    "cwmp": "urn:dslforum-org:cwmp-1-3"
+  }
+};
+
+const LIBXMLJS_PARSE_OPTIONS = {
+  "nocdata": true,
+  "noent": true,
+  "recover": config.get("XML_RECOVER"),
+  "ignore_enc": config.get("XML_IGNORE_ENC")
+};
+
+const LIBXMLJS_SAVE_OPTIONS = {
+  "format": config.get("XML_FORMAT"),
+  "declaration": !config.get("XML_NO_DECL"),
+  "selfCloseEmpty": !config.get("XML_NO_EMPTY")
+};
+
+const XML_IGNORE_NAMESPACE = config.get("XML_IGNORE_NAMESPACE");
+
+if (XML_IGNORE_NAMESPACE) {
+  libxmljs.Element.prototype.__find = libxmljs.Element.prototype.find;
+  libxmljs.Element.prototype.find = function(xpath, namespaces) {
+    const p = xpath.replace(/([^\/:]+:)?([^\/]+)(\/|$)/g, "$2$3").replace(/([a-zA-Z0-9_-]+)([^\/]*)(\/|$)/g, "*[local-name()=\"$1\"]$2$3");
+    return libxmljs.Element.prototype.__find.call(this, p, namespaces);
+  };
+}
+
+let warnings;
+
+
+function parseBool(v) {
+  v = "" + v;
+  if (v === "true" || v === "TRUE" || v === "True" || v === "1")
+    return true;
+  else if (v === "false" || v === "FALSE" || v === "False" || v === "0")
+    return false;
+  else
+    return null;
+}
+
+
+function event(xml) {
+  return xml.find("EventStruct/EventCode").map(e => e.text().trim());
+}
+
+
+function parameterInfoList(xml) {
+  return xml.find("ParameterInfoStruct").map(function(e) {
+    let param = e.get("Name").text();
+    let value = e.get("Writable").text();
+    let parsed = parseBool(value);
+
+    if (parsed === null) {
+      warnings.push({
+        message: "Invalid writable attribute",
+        parameter: param
+      });
+      parsed = false;
+    }
+
+    return [param, parsed];
+  });
+}
+
+
+function parameterValueList(xml) {
+  return xml.find("ParameterValueStruct").map(function(e) {
+    const valueElement = e.get("Value");
+    const valueType = valueElement.attr("type").value();
+    let param = e.get("Name").text();
+    let value = valueElement.text();
+    let parsed = value;
+    if (valueType === "xsd:boolean") {
+      parsed = parseBool(value);
+      if (parsed === null) {
+        warnings.push({
+          message: "Invalid value attribute",
+          parameter: param
+        });
+        parsed = value;
+      }
+    }
+    else if (valueType === "xsd:int" || valueType === "xsd:unsignedInt") {
+      parsed = parseInt(value);
+      if (isNaN(parsed)) {
+        warnings.push({
+          message: "Invalid value attribute",
+          parameter: param
+        });
+        parsed = value;
+      }
+    }
+    else if (valueType === "xsd:dateTime") {
+      parsed = Date.parse(value);
+      if (isNaN(parsed)) {
+        warnings.push({
+          message: "Invalid value attribute",
+          parameter: param
+        });
+        parsed = value;
+      }
+    }
+
+   return [param, parsed, valueType];
+  });
+}
+
+
+function GetParameterNames(xml, methodRequest) {
+  const el = xml.node("cwmp:GetParameterNames");
+  el.node("ParameterPath").text(methodRequest.parameterPath);
+  el.node("NextLevel").text(+methodRequest.nextLevel);
+}
+
+
+function GetParameterNamesResponse(xml) {
+  return {
+    name: "GetParameterNamesResponse",
+    parameterList: parameterInfoList(xml.get("ParameterList"))
+  };
+}
+
+
+function GetParameterValues(xml, methodRequest) {
+  const el = xml.node("cwmp:GetParameterValues").node("ParameterNames");
+  el.attr({
+    "soap-enc:arrayType": `xsd:string[${methodRequest.parameterNames.length}]`
+  });
+
+  for (const p of methodRequest.parameterNames)
+    el.node("string").text(p);
+}
+
+
+function GetParameterValuesResponse(xml) {
+  return {
+    name: "GetParameterValuesResponse",
+    parameterList: parameterValueList(xml.get("ParameterList"))
+  };
+}
+
+
+function SetParameterValues(xml, methodRequest) {
+  const el = xml.node("cwmp:SetParameterValues");
+  const paramList = el.node("ParameterList");
+  paramList.attr({
+    "soap-enc:arrayType": `cwmp:ParameterValueStruct[${methodRequest.parameterList.length}]`
+  });
+
+  for (const p of methodRequest.parameterList) {
+    const pvs = paramList.node("ParameterValueStruct");
+    pvs.node("Name").text(p[0]);
+    pvs.node("Value").attr({"xsi:type": p[2]}).text(p[1]);
+  }
+
+  el.node("ParameterKey").text(methodRequest.parameterKey || "");
+}
+
+
+function SetParameterValuesResponse(xml) {
+  return {
+    name: "SetParameterValuesResponse",
+    status: parseInt(xml.get("Status").text())
+  };
+}
+
+
+function AddObject(xml, methodRequest) {
+  const el = xml.node("cwmp:AddObject");
+  el.node("ObjectName").text(methodRequest.objectName);
+  el.node("ParameterKey").text(methodRequest.parameterKey || "");
+}
+
+
+function AddObjectResponse(xml) {
+  return {
+    name: "AddObjectResponse",
+    instanceNumber: parseInt(xml.get("InstanceNumber").text()),
+    status: parseInt(xml.get("Status").text())
+  };
+}
+
+
+function DeleteObject(xml, methodRequest) {
+  const el = xml.node("cwmp:DeleteObject");
+  el.node("ObjectName").text(methodRequest.objectName);
+  el.node("ParameterKey").text(methodRequest.parameterKey || "");
+}
+
+
+function DeleteObjectResponse(xml) {
+  return {
+    name: "DeleteObjectResponse",
+    status: parseInt(xml.get("Status").text())
+  };
+}
+
+
+function Reboot(xml, methodRequest) {
+  const el = xml.node("cwmp:Reboot");
+  el.node("CommandKey").text(methodRequest.commandKey || "");
+}
+
+
+function RebootResponse(xml) {
+  return {
+    name: "RebootResponse"
+  };
+}
+
+
+function FactoryReset(xml, methodRequest) {
+  xml.node("cwmp:FactoryReset");
+}
+
+
+function FactoryResetResponse(xml, methodRequest) {
+  return {
+    name: "FactoryResetResponse"
+  };
+}
+
+
+function Download(xml, methodRequest) {
+  const el = xml.node("cwmp:Download");
+  el.node("CommandKey").text(methodRequest.commandKey || "");
+  el.node("FileType").text(methodRequest.fileType);
+  el.node("URL").text(methodRequest.url);
+  el.node("Username").text(methodRequest.username || "");
+  el.node("Password").text(methodRequest.password || "");
+  el.node("FileSize").text(methodRequest.fileSize || "0");
+  el.node("TargetFileName").text(methodRequest.targetFileName || "");
+  el.node("DelaySeconds").text(methodRequest.delaySeconds || "0");
+  el.node("SuccessURL").text(methodRequest.successUrl || "");
+  el.node("FailureURL").text(methodRequest.failureUrl || "");
+}
+
+
+function DownloadResponse(xml) {
+  const res = {
+    name: "DownloadResponse",
+    status: parseInt(xml.get("Status").text())
+  };
+
+  if (res.status === 0) {
+    res.startTime = Date.parse(xml.get("StartTime").text());
+    res.completeTime = Date.parse(xml.get("CompleteTime").text());
+  }
+
+  return res;
+}
+
+
+function Inform(xml) {
+  return {
+    name: "Inform",
+    parameterList: parameterValueList(xml.get("ParameterList")),
+    deviceId: {
+      "Manufacturer": xml.get("DeviceId/Manufacturer").text(),
+      "OUI": xml.get("DeviceId/OUI").text(),
+      "ProductClass": xml.get("DeviceId/ProductClass").text(),
+      "SerialNumber": xml.get("DeviceId/SerialNumber").text()
+    },
+    event: event(xml.get("Event")),
+    retryCount: parseInt(xml.get("RetryCount").text())
+  };
+}
+
+
+function InformResponse(xml, methodResponse) {
+  xml.node("cwmp:InformResponse").node("MaxEnvelopes").text(1);
+}
+
+
+function GetRPCMethods(xml) {
+  return {name: "GetRPCMethods"};
+}
+
+
+function GetRPCMethodsResponse(xml, methodResponse) {
+  const el = xml.node("cwmp:GetRPCMethodsResponse").node("MethodList");
+  el.attr({
+    "soap-enc:arrayType": `xsd:string[${methodResponse.methodList.length}]`
+  });
+
+  for (const m of methodResponse.methodList)
+    el.node("string").text(m);
+}
+
+
+function TransferComplete(xml) {
+  return {
+    name: "TransferComplete",
+    commandKey: xml.get("CommandKey").text(),
+    faultStruct: faultStruct(xml.get("FaultStruct")),
+    startTime: Date.parse(xml.get("StartTime").text()),
+    completeTime: Date.parse(xml.get("CompleteTime").text())
+  };
+}
+
+
+function TransferCompleteResponse(xml, methodResponse) {
+  xml.node("cwmp:TransferCompleteResponse").text("");
+}
+
+
+function RequestDownload(xml) {
+  return {
+    name: "RequestDownload",
+    fileType: xml.get("FileType").text()
+  };
+}
+
+
+function RequestDownloadResponse(xml, methodResponse) {
+  xml.node("cwmp:RequestDownloadResponse").text("");
+}
+
+
+function faultStruct(xml) {
+  const f = {
+    faultCode: xml.get("FaultCode").text(),
+    faultString: xml.get("FaultString").text()
+  };
+
+  for (const e of xml.find("SetParameterValuesFault")) {
+    if (!f.setParameterValuesFault)
+      f.setParameterValuesFault = [];
+
+    f.setParameterValuesFault.push({
+      parameterName: e.get("ParameterName").text(),
+      faultCode: e.get("FaultCode").text(),
+      faultString: e.get("FaultString").text()
+    });
+  }
+
+  return f;
+}
+
+
+function fault(xml, cwmpVersion) {
+  return {
+    faultCode: xml.get("faultcode").text(),
+    faultString: xml.get("faultstring").text(),
+    detail: faultStruct(xml.get("detail/cwmp:Fault", NAMESPACES[cwmpVersion]))
+  };
+}
+
+
+function request(httpRequest, cwmpVersion, warn) {
+  warnings = warn;
+
+  const rpc = {
+    cwmpVersion: cwmpVersion
+  };
+
+  const data = httpRequest.getBody();
+
+  if (!data.length)
+    return rpc;
+
+  const xml = libxmljs.parseXml(data, LIBXMLJS_PARSE_OPTIONS);
+
+  let methodElement;
+
+  if (rpc.cwmpVersion) {
+    methodElement = xml.get("/soap-env:Envelope/soap-env:Body/cwmp:*", NAMESPACES[rpc.cwmpVersion]);
+  }
+  else {
+    // cwmpVersion not passed, thus it's an inform request
+    methodElement = xml.get("/soap-env:Envelope/soap-env:Body/*", NAMESPACES["1.0"]);
+    switch (methodElement.namespace().href()) {
+      case "urn:dslforum-org:cwmp-1-0":
+        rpc.cwmpVersion = "1.0";
+        break;
+      case "urn:dslforum-org:cwmp-1-1":
+        rpc.cwmpVersion = "1.1";
+        break;
+      case "urn:dslforum-org:cwmp-1-2":
+        const timeoutElement = xml.get("/soap-env:Envelope/soap-env:Header/cwmp:sessionTimeout", NAMESPACES["1.2"]);
+        if (timeoutElement) {
+          rpc.sessionTimeout = parseInt(timeoutElement.text());
+          rpc.cwmpVersion = "1.3";
+        }
+        else {
+          rpc.cwmpVersion = "1.2";
+        }
+        break;
+      case "urn:dslforum-org:cwmp-1-3":
+        rpc.cwmpVersion = "1.4";
+        break;
+      default:
+        throw new Error("Unrecognized CWMP version");
+    }
+
+    if (!rpc.sessionTimeout) {
+      const timeoutElement = xml.get("/soap-env:Envelope/soap-env:Header/cwmp:sessionTimeout", NAMESPACES[rpc.cwmpVersion]);
+      if (timeoutElement)
+        rpc.sessionTimeout = parseInt(timeoutElement.text());
+    }
+  }
+
+  rpc.id = xml.get("/soap-env:Envelope/soap-env:Header/cwmp:ID", NAMESPACES[rpc.cwmpVersion]).text();
+
+  if (methodElement && !(XML_IGNORE_NAMESPACE && methodElement.name() === "Fault")) {
+    switch (methodElement.name()) {
+      case "Inform":
+        rpc.cpeRequest = Inform(methodElement);
+        break;
+      case "GetRPCMethods":
+        rpc.cpeRequest = GetRPCMethods(methodElement);
+        break;
+      case "TransferComplete":
+        rpc.cpeRequest = TransferComplete(methodElement);
+        break;
+      case "RequestDownload":
+        rpc.cpeRequest = RequestDownload(methodElement);
+        break;
+      case "GetParameterNamesResponse":
+        rpc.cpeResponse = GetParameterNamesResponse(methodElement);
+        break;
+      case "GetParameterValuesResponse":
+        rpc.cpeResponse = GetParameterValuesResponse(methodElement);
+        break;
+      case "SetParameterValuesResponse":
+        rpc.cpeResponse = SetParameterValuesResponse(methodElement);
+        break;
+      case "AddObjectResponse":
+        rpc.cpeResponse = AddObjectResponse(methodElement);
+        break;
+      case "DeleteObjectResponse":
+        rpc.cpeResponse = DeleteObjectResponse(methodElement);
+        break;
+      case "RebootResponse":
+        rpc.cpeResponse = RebootResponse(methodElement);
+        break;
+      case "FactoryResetResponse":
+        rpc.cpeResponse = FactoryResetResponse(methodElement);
+        break;
+      case "DownloadResponse":
+        rpc.cpeResponse = DownloadResponse(methodElement);
+        break;
+      default:
+        throw new Error(`8000 Method not supported ${methodElement.name()}`);
+    }
+  }
+  else {
+    const faultElement = xml.get("/soap-env:Envelope/soap-env:Body/soap-env:Fault", NAMESPACES[rpc.cwmpVersion]);
+    rpc.cpeFault = fault(faultElement, rpc.cwmpVersion);
+  }
+
+  return rpc;
+}
+
+
+function createSoapEnv(cwmpVersion) {
+  const xml = libxmljs.Document();
+  const env = xml.node("soap-env:Envelope");
+
+  for (const prefix of Object.keys(NAMESPACES[cwmpVersion]))
+    env.defineNamespace(prefix, NAMESPACES[cwmpVersion][prefix]);
+
+  return env;
+}
+
+
+function response(rpc) {
+  const headers = {
+    "Server": SERVER_NAME,
+    "SOAPServer": SERVER_NAME
+  };
+
+  if (!rpc)
+    return {code: 204, headers: headers, data: new Buffer(0)};
+
+  const env = createSoapEnv(rpc.cwmpVersion);
+  const header = env.node("soap-env:Header");
+  header.node("cwmp:ID").attr({
+    "soap-env:mustUnderstand": 1
+  }).text(rpc.id);
+
+  const body = env.node("soap-env:Body");
+
+  if (rpc.acsResponse) {
+    switch (rpc.acsResponse.name) {
+      case "InformResponse":
+        InformResponse(body, rpc.acsResponse);
+        break;
+      case "GetRPCMethodsResponse":
+        GetRPCMethodsResponse(body, rpc.acsResponse);
+        break;
+      case "TransferCompleteResponse":
+        TransferCompleteResponse(body, rpc.acsResponse);
+        break;
+      case "RequestDownloadResponse":
+        RequestDownloadResponse(body, rpc.acsResponse);
+        break;
+      default:
+        throw new Error(`Unknown method response type ${rpc.acsResponse.name}`);
+    }
+  }
+  else if (rpc.acsRequest) {
+    switch (rpc.acsRequest.name) {
+      case "GetParameterNames":
+        GetParameterNames(body, rpc.acsRequest);
+        break;
+      case "GetParameterValues":
+        GetParameterValues(body, rpc.acsRequest);
+        break;
+      case "SetParameterValues":
+        SetParameterValues(body, rpc.acsRequest);
+        break;
+      case "AddObject":
+        AddObject(body, rpc.acsRequest);
+        break;
+      case "DeleteObject":
+        DeleteObject(body, rpc.acsRequest);
+        break;
+      case "Reboot":
+        Reboot(body, rpc.acsRequest);
+        break;
+      case "FactoryReset":
+        FactoryReset(body, rpc.acsRequest);
+        break;
+      case "Download":
+        Download(body, rpc.acsRequest);
+        break;
+      default:
+        throw new Error(`Unknown method request ${rpc.acsRequest.name}`);
+    }
+  }
+
+  headers["Content-Type"] = "text/xml; charset=\"utf-8\"";
+  return {
+    code: 200,
+    headers: headers,
+    data: new Buffer(env.doc().toString(LIBXMLJS_SAVE_OPTIONS))
+  };
+}
+
+
+exports.request = request;
+exports.response = response;

lib/tasks.coffee

@@ -1,367 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-#
-# This file incorporates work covered by the following copyright and
-# permission notice:
-#
-# Copyright 2013 Fanoos Telecom
-#
-# Permission is hereby granted, free of charge, to any person obtaining a copy
-# of this software and associated documentation files (the "Software"), to deal
-# in the Software without restriction, including without limitation the rights
-# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-# copies of the Software, and to permit persons to whom the Software is
-# furnished to do so, subject to the following conditions:
-#
-# The above copyright notice and this permission notice shall be included in
-# all copies or substantial portions of the Software.
-#
-# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-# THE SOFTWARE.
-###
-
-config = require './config'
-common = require './common'
-db = require './db'
-mongodb = require 'mongodb'
-customCommands = require './custom-commands'
-url = require 'url'
-
-exports.STATUS_OK = STATUS_OK = 1
-exports.STATUS_FAULT = STATUS_FAULT = 2
-exports.STATUS_COMPLETED = STATUS_COMPLETED = 4
-exports.STATUS_SAVE = STATUS_SAVE = 128 # not mutually exclusive with the rest
-
-
-this.refreshObject = (task, methodResponse, callback) ->
-  task.session = {} if not task.session?
-
-  if not task.session.subtask?
-    path = task.objectName
-    path += '.' if path != '' and not common.endsWith(path, '.')
-    task.session.subtask = {device : task.device, name : 'getParameterNames', parameterPath : path}
-
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if task.session.subtask.name is 'getParameterNames'
-    this.getParameterNames(task.session.subtask, methodResponse, (err, status, methodRequest, deviceUpdates) =>
-      task.session.parameterNames ?= []
-      if deviceUpdates?.parameterNames?
-        for p in deviceUpdates.parameterNames
-          if not common.endsWith(p[0], '.')
-            task.session.parameterNames.push(p[0])
-
-      if status & STATUS_COMPLETED
-        task.session.subtask = {device : task.device, name : 'getParameterValues', parameterNames : task.session.parameterNames}
-        delete task.session.parameterNames
-
-        this.getParameterValues(task.session.subtask, {}, (err, status, methodRequest) ->
-          # ignore deviceUpdates returned by firt call to getParameterValues
-          callback(err, status, methodRequest, deviceUpdates)
-        )
-      else if status & STATUS_OK
-        callback(err, STATUS_OK, methodRequest, deviceUpdates)
-      else
-        throw new Error('Unexpected subtask status')
-    )
-  else if task.session.subtask.name is 'getParameterValues'
-    if methodResponse.faultcode?
-      task.fault = methodResponse
-      return callback(null, STATUS_FAULT)
-
-    this.getParameterValues(task.session.subtask, methodResponse, (err, status, methodRequest, deviceUpdates) ->
-      callback(err, status, methodRequest, deviceUpdates)
-    )
-  else
-    throw Error('Unexpected subtask name')
-
-
-this.getParameterNames = (task, methodResponse, callback) ->
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  GET_PARAMETER_NAMES_DEPTH_THRESHOLD = config.get('GET_PARAMETER_NAMES_DEPTH_THRESHOLD', task.device)
-
-  getParameterDepth = (param) ->
-    return 0 if !param
-    return (param[...-1] + '.').split('.').length - 1
-
-  if methodResponse.type is 'GetParameterNamesResponse'
-    path = task.session.queue.pop()
-
-    # If parameter depth higher than the threshold, nextLevel was set to false
-    if not task.nextLevel? and getParameterDepth(path) < GET_PARAMETER_NAMES_DEPTH_THRESHOLD
-      for p in methodResponse.parameterList
-        task.session.queue.push(p[0]) if p[0][-1..] == '.'
-
-    deviceUpdates = {parameterNames : methodResponse.parameterList}
-
-    # Make sure that for each parameter, all its parents are explicitly included
-    found = {}
-    found[path] = 0 if !!path
-    for p in deviceUpdates.parameterNames
-      param = p[0]
-      i = param.length
-      while (i = param.lastIndexOf('.', i-1)) > path.length
-        pp = param.slice(0, i + 1)
-        break if found[pp]?
-        found[pp] = 0
-      found[p[0]] = 1
-
-    for k, v of found
-      deviceUpdates.parameterNames.push([k]) if v == 0
-  else
-    task.session = {queue : [task.parameterPath]}
-
-  if task.session.queue.length > 0
-    path = task.session.queue[-1..][0]
-    methodRequest = {
-      type : 'GetParameterNames',
-      parameterPath : path,
-      nextLevel : task.nextLevel ? getParameterDepth(path) < GET_PARAMETER_NAMES_DEPTH_THRESHOLD
-    }
-    return callback(null, STATUS_OK, methodRequest, deviceUpdates)
-  else
-    return callback(null, STATUS_COMPLETED, null, deviceUpdates)
-
-
-this.getParameterValues = (task, methodResponse, callback) ->
-  task.session = {} if not task.session?
-
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if not task.session.currentIndex?
-    task.session.currentIndex = 0
-  else if methodResponse.parameterList?
-    task.session.currentIndex = task.session.nextIndex
-
-  task.session.nextIndex = Math.min(task.session.currentIndex + config.get('TASK_PARAMETERS_BATCH_SIZE', task.device), task.parameterNames.length)
-  names = task.parameterNames.slice(task.session.currentIndex, task.session.nextIndex)
-
-  if methodResponse.type is 'GetParameterValuesResponse'
-    deviceUpdates = {parameterValues : methodResponse.parameterList}
-
-  if names.length == 0
-    callback(null, STATUS_COMPLETED, null, deviceUpdates)
-  else
-    methodRequest = {
-      type : 'GetParameterValues',
-      parameterNames : names
-    }
-    callback(null, STATUS_OK, methodRequest, deviceUpdates)
-
-
-this.setParameterValues = (task, methodResponse, callback) ->
-  task.session = {} if not task.session?
-
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if not task.session.currentIndex?
-    task.session.currentIndex = 0
-  else if methodResponse.type is 'SetParameterValuesResponse'
-    prevValues = task.parameterValues.slice(task.session.currentIndex, task.session.nextIndex)
-    task.session.currentIndex = task.session.nextIndex
-
-  task.session.nextIndex = Math.min(task.session.currentIndex + config.get('TASK_PARAMETERS_BATCH_SIZE', task.device), task.parameterValues.length)
-  values = task.parameterValues.slice(task.session.currentIndex, task.session.nextIndex)
-
-  if prevValues?
-    deviceUpdates = {parameterValues : prevValues}
-
-  if values.length == 0
-    callback(null, STATUS_COMPLETED, null, deviceUpdates)
-  else
-    callback(null, STATUS_OK, {type : 'SetParameterValues', parameterList : values}, deviceUpdates)
-
-
-this.addObject = (task, methodResponse, callback) ->
-  task.session = {} if not task.session?
-
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  allDeviceUpdates = {}
-
-  if not task.session.instanceNumber?
-    if methodResponse.type is 'AddObjectResponse'
-      status_save = STATUS_SAVE
-      task.session.instanceNumber = methodResponse.instanceNumber
-      # TODO Don't specify nextLevel explicity. Also consider using refreshObject instead.
-      task.session.subtask = {device : task.device, name : 'getParameterNames', parameterPath : "#{task.objectName}.#{task.session.instanceNumber}.", nextLevel : false}
-      task.session.appliedParameterValues = []
-      task.session.parameterNames = []
-    else
-      callback(null, STATUS_OK, {type : 'AddObject', objectName : "#{task.objectName}."})
-      return
-
-  subtask = () =>
-    switch task.session.subtask.name
-      when 'getParameterNames'
-        this.getParameterNames(task.session.subtask, methodResponse, (err, status, methodRequest, deviceUpdates) =>
-          common.extend(allDeviceUpdates, deviceUpdates)
-          if deviceUpdates and deviceUpdates.parameterNames
-            for p in deviceUpdates.parameterNames
-              task.session.parameterNames.push(p[0]) if not common.endsWith(p[0], '.')
-
-          if status & STATUS_COMPLETED
-            task.session.subtask = {device : task.device, name : 'getParameterValues', parameterNames : task.session.parameterNames}
-            subtask()
-          else if status & STATUS_OK
-            # Use STATUS_SAVE to avoid adding duplicate object in case of error
-            callback(err, STATUS_OK | status_save, methodRequest, allDeviceUpdates)
-          else
-            throw Error('Unexpected subtask status')
-        )
-      when 'getParameterValues'
-        this.getParameterValues(task.session.subtask, methodResponse, (err, status, methodRequest, deviceUpdates) =>
-          common.extend(allDeviceUpdates, deviceUpdates)
-          # if values are given, compare with default values
-          if task.parameterValues?
-            if deviceUpdates and deviceUpdates.parameterValues?
-              for p1 in deviceUpdates.parameterValues
-                for p2 in task.parameterValues
-                  if p1[0] == "#{task.objectName}.#{task.session.instanceNumber}.#{p2[0]}"
-                    t = if p2[2] then p2[2] else p1[2]
-                    v = common.matchType(p1[1], p2[1])
-                    # TODO only include if writable
-                    task.session.appliedParameterValues.push([p1[0], v, t])
-
-          if methodResponse.faultcode?
-            task.fault = methodResponse
-            return callback(null, STATUS_FAULT)
-
-          if status & STATUS_COMPLETED and task.session.appliedParameterValues.length > 0
-            task.session.subtask = {device : task.device, name : 'setParameterValues', parameterValues : task.session.appliedParameterValues}
-            subtask()
-          else
-            callback(err, status, methodRequest, allDeviceUpdates)
-        )
-      when 'setParameterValues'
-        this.setParameterValues(task.session.subtask, methodResponse, (err, status, methodRequest, deviceUpdates) =>
-          common.extend(allDeviceUpdates, deviceUpdates)
-          callback(err, status, methodRequest, allDeviceUpdates)
-        )
-  subtask()
-
-
-this.deleteObject = (task, methodResponse, callback) ->
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if methodResponse.type is 'DeleteObjectResponse'
-    callback(null, STATUS_COMPLETED, null, {deletedObjects : [task.objectName]})
-  else
-    methodRequest = {
-      type : 'DeleteObject',
-      objectName : "#{task.objectName}."
-    }
-    callback(null, STATUS_OK, methodRequest)
-
-
-this.reboot = (task, methodResponse, callback) ->
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if methodResponse.type isnt 'RebootResponse'
-    callback(null, STATUS_OK, {type : 'Reboot'})
-  else
-    callback(null, STATUS_COMPLETED)
-
-
-this.factoryReset = (task, methodResponse, callback) ->
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if methodResponse.type isnt 'FactoryResetResponse'
-    callback(null, STATUS_OK, {type : 'FactoryReset'})
-  else
-    callback(null, STATUS_COMPLETED)
-
-
-this.download = (task, methodResponse, callback) ->
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if methodResponse.type isnt 'DownloadResponse'
-    db.filesCollection.findOne({_id : mongodb.ObjectID(String(task.file))}, (err, file) ->
-      if not file?
-        callback('File not found')
-        return
-      else if err?
-        callback(err)
-        return
-
-      l = {
-        protocol : if config.get('FS_SSL') then 'https' else 'http',
-        hostname : config.get('FS_IP'),
-        port : config.get('FS_PORT'),
-        pathname : encodeURIComponent(file.filename)
-      }
-
-      methodRequest = {
-        type : 'Download',
-        fileType : file.metadata.fileType,
-        fileSize : file.length,
-        url : url.format(l),
-        successUrl : task.successUrl,
-        failureUrl : task.failureUrl
-      }
-      callback(null, STATUS_OK, methodRequest)
-    )
-  else
-    callback(null, STATUS_COMPLETED)
-
-
-this.customCommand = (task, methodResponse, callback) ->
-  # TODO implement timeout
-  customCommands.execute(task.device, task.command, (err, value) ->
-    if err
-      task.fault = {name : err.name, message : err.message}
-      callback(null, STATUS_FAULT)
-    else
-      commandName = task.command.split(' ', 2)[0]
-      callback(null, STATUS_COMPLETED, null, {customCommands : [[commandName, value]]})
-  )
-
-
-exports.task = (task, methodResponse, callback) ->
-  this[task.name](task, methodResponse, callback)

lib/update-device.coffee

@@ -1,217 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-###
-
-common = require './common'
-db = require './db'
-normalize = require('./normalize').normalize
-mongoQuery = require './mongo-query'
-parameters = require './parameters'
-
-OBJECT_REGEX = /\.$/
-INSTANCE_REGEX = /\.[\d]+\.$/
-
-
-queueUpdates = (deviceId, deviceUpdates, callback) ->
-  return callback() if not deviceUpdates? or Object.keys(deviceUpdates).length == 0
-  # TODO consider setting expiry
-  db.redisClient.rpush("#{deviceId}_updates", JSON.stringify(deviceUpdates), (err, res) ->
-    return callback(err)
-  )
-
-
-clearUpdatesQueue = (deviceId, callback) ->
-  db.redisClient.del("#{deviceId}_updates", (err, res) ->
-    return callback(err)
-  )
-
-
-commitUpdates = (deviceId, deviceUpdates, commitQueue, callback) ->
-  now = new Date(Date.now())
-  update = {'$set' : {}, '$unset' : {}}
-
-  # Used to find parameters that no longer exist in device
-  presentPaths = null
-  projection = null
-
-  # Keep track of alias changes to clear cache if needed
-  newAliases = null
-
-  processBatch = (updatesBatch, cb) ->
-    if updatesBatch.informEvents?
-      update['$set']['_lastInform'] = now
-      update['$set']['_lastBoot'] = now if '1 BOOT' in updatesBatch.informEvents
-      update['$set']['_lastBootstrap'] = now if '0 BOOTSTRAP' in updatesBatch.informEvents
-
-    if updatesBatch.parameterValues?
-      for p in updatesBatch.parameterValues
-        origValue = p[1]
-        v = normalize(p[0], origValue)
-
-        path = if common.endsWith(p[0], '.') then p[0] else "#{p[0]}."
-        if v == origValue
-          update['$unset']["#{path}_orig"] = 1
-        else
-          update['$set']["#{path}_orig"] = origValue
-        update['$set']["#{path}_value"] = v
-        update['$set']["#{path}_timestamp"] = now
-        if p[2]?
-          update['$set']["#{path}_type"] = p[2]
-
-    if updatesBatch.deletedObjects?
-      for p in updatesBatch.deletedObjects
-        update['$unset'][p] = 1
-
-    if updatesBatch.parameterNames?
-      presentPaths ?= {}
-      projection ?= {}
-
-      for p in updatesBatch.parameterNames
-        if OBJECT_REGEX.test(p[0])
-          path = p[0].slice(0, -1)
-          if INSTANCE_REGEX.test(p[0])
-            update['$set']["#{path}._instance"] = true
-          else
-            update['$set']["#{path}._object"] = true
-        else
-          path = p[0]
-        update['$set']["#{path}._writable"] = p[1] if p[1]?
-        update['$set']["#{path}._timestamp"] = now
-
-        # Add parameter (sans any dot suffix) if has an alias
-        if (a = parameters.getAlias(path))?
-          newAliases ?= {}
-          newAliases[path] = a
-
-        projection[path] = 1
-
-        # Store this param path and all its parents for later look up
-        presentPaths[path] = true
-        i = p[0].indexOf('.')
-        while i != -1
-          presentPaths[p[0][0...i]] = true
-          i = p[0].indexOf('.', i + 1)
-
-    if updatesBatch.customCommands?
-      for p in updatesBatch.customCommands
-        commandName = p[0]
-        update['$set']["_customCommands.#{commandName}._value"] = p[1]
-        update['$set']["_customCommands.#{commandName}._timestamp"] = now
-
-  processBatches = (cb) ->
-    if not commitQueue
-      processBatch(deviceUpdates) if deviceUpdates?
-      return cb()
-
-    db.redisClient.lpop("#{deviceId}_updates", (err, res) ->
-      callback(err) if err
-
-      if not res?
-        processBatch(deviceUpdates) if deviceUpdates?
-        return cb()
-
-      processBatch(JSON.parse(res))
-      return processBatches(cb)
-    )
-
-  commit = () ->
-    for k of update
-      delete update[k] if Object.keys(update[k]).length == 0
-
-    if Object.keys(update).length
-      db.devicesCollection.update({'_id' : deviceId}, update, {}, (err, count) ->
-        return callback(err) if err
-
-        # Clear aliases cache if there's a new aliased parameter that has not been included in aliases cache
-        if newAliases?
-          newAliasFound = false
-          db.getAliases((aliases) ->
-            for p, a of newAliases
-              if not aliases[a]? or p not in aliases[a]
-                newAliasFound = true
-                break
-
-            return callback() if not newAliasFound
-            db.redisClient.del('aliases', (err, res) ->
-              return callback(err)
-            )
-          )
-        else
-          return callback()
-      )
-    else
-      return callback()
-
-  processBatches(() ->
-    if projection?
-      mongoQuery.optimizeProjection(projection)
-
-      # Fetch device from DB to discover and delete parameters that no longer exist
-      # While we're fetching the object, we might as well use the information
-      # to optimize our update query
-      db.devicesCollection.findOne({'_id' : deviceId}, projection, (err, device) ->
-        return callback(err) if err
-
-        recursive = (obj, prefix) ->
-          for k, v of obj
-            p = prefix + k
-
-            # Mark unset operations we want to keep
-            if update['$unset'][p]?
-              update['$unset'][p] = 2
-              continue
-
-            if k[0] == '_'
-              # Remove set operations for unmodified values for optimization
-              if update['$set'][p] == v
-                delete update['$set'][p]
-              continue
-
-            if presentPaths[p]
-              recursive(v, "#{p}.")
-            else
-              update['$unset'][p] = 2
-
-        # Scan the DB object and remove parameters that no longer exists in device
-        for path of projection
-          root = device
-          rootPath = ''
-          for p in path.split('.')
-            rootPath += "#{p}."
-            root = root[p]
-            break if not root?
-
-          if root
-            recursive(root, rootPath)
-          else
-            if not presentPaths[rootPath.slice(0, -1)]?
-              update['$unset'][rootPath.slice(0, -1)] = 2
-
-          # Remove unnecessary unset updates
-          for k, v of update['$unset']
-            if common.startsWith(k, path + '.') and v != 2
-              delete update['$unset'][k]
-
-        commit()
-      )
-    else
-      commit()
-  )
-
-
-exports.queueUpdates = queueUpdates
-exports.clearUpdatesQueue = clearUpdatesQueue
-exports.commitUpdates = commitUpdates

lib/versioned-map.js

@@ -0,0 +1,255 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const NONEXISTENT = new Object();
+
+
+function compareEquality(a, b) {
+  var t = typeof a;
+  if (t === 'number' || (a == null) || t === 'boolean' || t === 'string' || t === 'symbol')
+    return a === b;
+
+  return JSON.stringify(a) === JSON.stringify(b);
+}
+
+
+class VersionedMap {
+
+  constructor() {
+    this._sizeDiff = [0];
+    this._revision = 0;
+    this.map = new Map();
+  }
+
+
+  get size() {
+    return this.map.size + this._sizeDiff[this.revision];
+  }
+
+
+  get revision() {
+    return this._revision;
+  }
+
+
+  set revision(rev) {
+    for (let i = this._sizeDiff.length; i <= rev; ++ i)
+      this._sizeDiff[i] = this._sizeDiff[i - 1];
+
+    this._revision = rev;
+  }
+
+
+  get(key, rev) {
+    if (rev == null)
+      rev = this._revision
+
+    var v = this.map.get(key);
+    if (!v)
+      return;
+
+    for (let i = Math.min(v.length - 1, rev); i >= 0; -- i)
+      if (i in v)
+        if (v[i] === NONEXISTENT)
+          return undefined;
+        else
+          return v[i];
+  }
+
+
+  has(key, rev) {
+    if (rev == null)
+      rev = this._revision
+
+    var v = this.map.get(key);
+
+    if (v == null)
+      return false;
+
+    for (let i = Math.min(v.length - 1, rev); i >= 0; -- i)
+      if (i in v)
+        if (v[i] === NONEXISTENT)
+          return false;
+        else
+          return true;
+
+    return false;
+  }
+
+
+  set(key, value, rev) {
+    if (rev == null)
+      rev = this._revision;
+
+    var v = this.map.get(key);
+    if (!v) {
+      for (let i = 0; i < rev; ++ i)
+        this._sizeDiff[i] -= 1;
+
+      v = [];
+      v[rev] = value;
+      this.map.set(key, v);
+      return this;
+    }
+
+    var old = v[Math.min(rev, v.length - 1)];
+
+    if (rev < v.length - 1) {
+      if (compareEquality(value, old))
+        return
+      else
+        throw new Error('Cannot modify old revisions');
+    }
+
+    if (old === NONEXISTENT)
+      ++ this._sizeDiff[rev];
+
+    v[rev] = value;
+  }
+
+  delete(key, rev) {
+    if (rev == null)
+      rev = this._revision;
+
+    var v = this.map.get(key);
+    if (!v)
+      return false;
+
+    if (rev < v.length - 1)
+      throw new Error('Cannot modify old revisions');
+
+    var old = v[v.length - 1];
+    if (old === NONEXISTENT)
+      return false;
+
+    -- this._sizeDiff[rev];
+    v[rev] = NONEXISTENT;
+
+    return old;
+  }
+
+
+  getRevisions(key) {
+    var v = this.map.get(key);
+    if (!v)
+      return null;
+
+    var res = {};
+
+    for (let i in v)
+      if (v[i] === NONEXISTENT)
+        res.delete |= 1 << i;
+      else
+        res[i] = v[i];
+
+    return res;
+  }
+
+
+  setRevisions(key, revisions) {
+    var del = 0;
+    var rev = [];
+    var minKey = 999;
+    for (let k in revisions) {
+      if (k === 'delete')
+        del = revisions[k];
+      else {
+        let r = parseInt(k);
+        minKey = Math.min(minKey, r);
+        for (let i = this._sizeDiff.length; i <= r; ++ i)
+          this._sizeDiff[i] = this._sizeDiff[i - 1];
+
+        rev[r] = revisions[k];
+      }
+    }
+
+    for (let i = 0; i < minKey; ++ i)
+      -- this._sizeDiff[i];
+
+    for (let i = 0; del > 0; del >>= 1, ++ i)
+      if (del & 1)
+        rev[i] = NONEXISTENT;
+
+    this.map.set(key, rev);
+  }
+
+
+  getDiff(key) {
+    var revisions = this.map.get(key);
+    if (!revisions)
+      return [];
+
+    var current = revisions[revisions.length - 1];
+    if (current === NONEXISTENT) {
+      if (0 in revisions)
+        return [key, revisions[0]];
+      else
+        return [key];
+    } else if (0 in revisions)
+      return [key, revisions[0], current];
+    else
+      return [key, , current];
+  }
+
+
+  *diff() {
+    for (let pair of this.map) {
+      let current = pair[1][pair[1].length - 1];
+      if (current === NONEXISTENT) {
+        if (0 in pair[1])
+          yield [pair[0], pair[1][0]];
+        else
+          yield [pair[0]];
+      } else if (0 in pair[1])
+        yield [pair[0], pair[1][0], current];
+      else
+        yield [pair[0], , current];
+    }
+  }
+
+
+  collapse(revision) {
+    if (this._sizeDiff.length <= revision)
+      return;
+
+    this._sizeDiff[revision] = this._sizeDiff[this._sizeDiff.length - 1];
+    this._sizeDiff.splice(revision + 1, this._sizeDiff.length);
+
+    for (let pair of this.map) {
+      let k = pair[0];
+      let v = pair[1];
+      let last = v[v.length - 1];
+      if (last === NONEXISTENT) {
+        v.splice(revision, v.length);
+        if (v.some(function(val) { return true; }))
+          v[revision] = NONEXISTENT;
+        else
+          this.map.delete(k);
+      }
+      else {
+        v[revision] = last;
+        v.splice(revision + 1, v.length);
+      }
+    }
+  }
+
+}
+
+module.exports = VersionedMap;

package.json

@@ -1,40 +1,41 @@
 {
   "name": "genieacs",
-  "version": "1.0.6",
+  "version": "1.1.1",
-  "description": "A blazing fast TR-069 auto configuration server (ACS) built with Node.js, Redis, and MongoDB",
+  "description": "A TR-069 Auto Configuration Server (ACS)",
   "repository": {
     "type": "git",
     "url": "https://github.com/zaidka/genieacs.git"
   },
+  "homepage": "https://genieacs.com",
   "keywords": [
     "TR-069",
     "CWMP",
     "ACS"
   ],
-  "author": "Zaid Abdulla",
+  "author": "Zaid Abdulla <zaid@genieacs.com>",
   "license": "AGPL-3.0",
-  "readmeFilename": "README.md",
   "preferGlobal": true,
   "directories": {
-    "test": "test",
     "bin": "bin"
   },
   "dependencies": {
-    "libxmljs": "0.11.0 - 0.14.x",
+    "hiredis": "^0.5.0",
-    "redis": "^0.12.1",
+    "later": "^1.2.0",
-    "hiredis": "0.1.17 - 0.4.x",
+    "libxmljs": "^0.18.4",
-    "mongodb": "^1.4.8"
+    "mongodb": "^2.2.24",
+    "redis": "^2.6.5",
+    "seedrandom": "^2.4.2"
   },
   "devDependencies": {
-    "coffee-script": "^1.7.1"
+    "coffeescript": "^1.12.4"
   },
   "engines": {
-    "node": ">=0.10.0 <0.12"
+    "node": ">=6.9.2 <8.0"
   },
   "scripts": {
     "prepublish": "npm run compile",
-    "compile": "coffee -cb lib/*.coffee test/*.coffee tools/*.coffee",
+    "compile": "coffee -cb lib/*.coffee",
-    "configure": "cp ./config/config-sample.json ./config/config.json; cp ./config/parameters-sample.json ./config/parameters.json; cp ./config/custom_commands-sample.json ./config/custom_commands.json; cp ./config/auth-sample.js ./config/auth.js",
+    "configure": "cp -n ./config/config-sample.json ./config/config.json",
     "install": "npm run configure",
     "test": "node test"
   }

test/methods.coffee

@@ -1,155 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-###
-
-common = require '../lib/common'
-
-NAMESPACES = {
-  'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-  'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-  'xsd' : 'http://www.w3.org/2001/XMLSchema',
-  'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-  'cwmp' : 'urn:dslforum-org:cwmp-1-0',
-}
-
-
-exports.inform = (device, xmlOut, callback) ->
-  body = xmlOut.root().childNodes()[1]
-  inform = body.node('cwmp:Inform')
-
-  deviceId = inform.node('DeviceId')
-  deviceId.node('Manufacturer', device['InternetGatewayDevice.DeviceInfo.Manufacturer'][1])
-  deviceId.node('OUI', device['InternetGatewayDevice.DeviceInfo.ManufacturerOUI'][1])
-  deviceId.node('ProductClass', device['InternetGatewayDevice.DeviceInfo.ProductClass'][1])
-  deviceId.node('SerialNumber', device['InternetGatewayDevice.DeviceInfo.SerialNumber'][1])
-
-  eventStruct = inform.node('Event').attr({'soap-enc:arrayType' : 'cwmp:EventStruct[1]'}).node('EventStruct')
-  eventStruct.node('EventCode', '2 PERIODIC')
-  eventStruct.node('CommandKey')
-
-  inform.node('MaxEnvelopes', '1')
-  inform.node('CurrentTime', new Date().toISOString())
-  inform.node('RetryCount', '0')
-
-  parameterList = inform.node('ParameterList').attr({'soap-enc:arrayType' : 'cwmp:ParameterValueStruct[7]'})
-  for p in [
-      'InternetGatewayDevice.DeviceInfo.SpecVersion',
-      'InternetGatewayDevice.DeviceInfo.HardwareVersion',
-      'InternetGatewayDevice.DeviceInfo.SoftwareVersion',
-      'InternetGatewayDevice.DeviceInfo.ProvisioningCode',
-      'InternetGatewayDevice.ManagementServer.ParameterKey',
-      'InternetGatewayDevice.ManagementServer.ConnectionRequestURL',
-      'InternetGatewayDevice.WANDevice.1.WANConnectionDevice.1.WANIPConnection.1.ExternalIPAddress']
-    parameterValueStruct = parameterList.node('ParameterValueStruct')
-    parameterValueStruct.node('Name', p)
-    parameterValueStruct.node('Value', device[p][1]).attr({'xsi:type' : "#{device[p][2]}"})
-
-  callback(xmlOut)
-
-
-exports.GetParameterNames = (device, xmlIn, xmlOut, callback) ->
-  parameterNames = Object.keys(device).sort()
-  parameterPath = xmlIn.get('/soap-env:Envelope/soap-env:Body/cwmp:GetParameterNames/ParameterPath', NAMESPACES).text()
-  nextLevel = !!eval(xmlIn.get('/soap-env:Envelope/soap-env:Body/cwmp:GetParameterNames/NextLevel', NAMESPACES).text())
-
-  parameterList = []
-  if nextLevel
-    for p in parameterNames
-      if common.startsWith(p, parameterPath) and p.length != parameterPath.length
-        i = p.indexOf('.', parameterPath.length)
-        if i == -1 or i == p.length - 1
-          parameterList.push(p)
-  else
-    for p in parameterNames
-      if common.startsWith(p, parameterPath)
-        parameterList.push(p)
-
-  getParameterNamesResponseNode = xmlOut.root().childNodes()[1].node('cwmp:GetParameterNamesResponse')
-  parameterListNode = getParameterNamesResponseNode.node('ParameterList')
-  parameterListNode.attr({'soap-enc:arrayType' : "cwmp:ParameterInfoStruct[#{parameterList.length}]"})
-  for p in parameterList
-    parameterInfoStructNode = parameterListNode.node('ParameterInfoStruct')
-    parameterInfoStructNode.node('Name', p)
-    parameterInfoStructNode.node('Writable', String(device[p][0]))
-
-  callback(xmlOut)
-
-
-exports.GetParameterValues = (device, xmlIn, xmlOut, callback) ->
-  parameterNames = xmlIn.find('/soap-env:Envelope/soap-env:Body/cwmp:GetParameterValues/ParameterNames/*', NAMESPACES)
-
-  parameterList = xmlOut.root().childNodes()[1].node('cwmp:GetParameterValuesResponse').node('ParameterList')
-  parameterList.attr({'soap-enc:arrayType' : "cwmp:ParameterValueStruct[#{parameterNames.length}]"})
-
-  for p in parameterNames
-    name = p.text()
-    value = device[name][1]
-    type = device[name][2]
-    valueStruct = parameterList.node('ParameterValueStruct')
-    valueStruct.node('Name', name)
-    valueStruct.node('Value', device[name][1]).attr({'xsi:type' : "#{type}"})
-
-  callback(xmlOut)
-
-
-exports.SetParameterValues = (device, xmlIn, xmlOut, callback) ->
-  parameterValues = xmlIn.find('/soap-env:Envelope/soap-env:Body/cwmp:SetParameterValues/ParameterList/*', NAMESPACES)
-
-  for p in parameterValues
-    name = p.get('Name').text()
-    value = p.get('Value')
-    device[name][1] = value.text()
-    device[name][2] = value.attr('xsi:type')
-
-  responseNode = xmlOut.root().childNodes()[1].node('cwmp:SetParameterValuesResponse')
-  responseNode.node('Status', '0')
-
-  callback(xmlOut)
-
-
-exports.AddObject = (device, xmlIn, xmlOut, callback) ->
-  objectName = xmlIn.get('/soap-env:Envelope/soap-env:Body/cwmp:AddObject/ObjectName', NAMESPACES).text()
-
-  parameters = []
-  instances = {}
-  instanceNumber = 1
-  while device[objectName + instanceNumber + '.']?
-    instanceNumber += 1
-
-  for p in Object.keys(device).sort()
-    if common.startsWith(p, objectName) and p.length > objectName.length
-      n = objectName + instanceNumber + p.slice(p.indexOf('.', objectName.length))
-      if not device[n]?
-        device[n] = [device[p][0], '', device[p][2]]
-
-  responseNode = xmlOut.root().childNodes()[1].node('cwmp:AddObjectResponse')
-  responseNode.node('InstanceNumber', String(instanceNumber))
-  responseNode.node('Status', '0')
-
-  callback(xmlOut)
-
-
-exports.DeleteObject = (device, xmlIn, xmlOut, callback) ->
-  objectName = xmlIn.get('/soap-env:Envelope/soap-env:Body/cwmp:DeleteObject/ObjectName', NAMESPACES).text()
-
-  for p in Object.keys(device)
-    if common.startsWith(p, objectName)
-      delete device[p]
-
-  responseNode = xmlOut.root().childNodes()[1].node('cwmp:DeleteObjectResponse')
-  responseNode.node('Status', '0')
-
-  callback(xmlOut)

test/test.coffee

@@ -1,173 +0,0 @@
-###
-# Copyright 2013, 2014  Zaid Abdulla
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-###
-
-http = require 'http'
-libxmljs = require 'libxmljs'
-deviceTemplate = require './huawei_bm632w'
-methods = require './methods'
-util = require 'util'
-
-NAMESPACES = {
-  'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-  'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-  'xsd' : 'http://www.w3.org/2001/XMLSchema',
-  'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-  'cwmp' : 'urn:dslforum-org:cwmp-1-0',
-}
-
-
-createSoapDocument = (id) ->
-  xml = libxmljs.Document()
-  env = xml.node('soap-env:Envelope')
-  for prefix, url of NAMESPACES
-    env.defineNamespace prefix, url
-
-  header = env.node('soap-env:Header')
-  header.node('cwmp:ID').attr({'soap-env:mustUnderstand' : 1}).text(id)
-  body = env.node('soap-env:Body')
-  return xml
-
-
-createDevice = (serial) ->
-  device = JSON.parse(JSON.stringify(deviceTemplate))
-  device['InternetGatewayDevice.DeviceInfo.SerialNumber'][1] = serial
-  return device
-
-
-sendRequest = (device,xml, callback) ->
-  headers = {}
-  if xml?
-    body = xml.toString()
-  else
-    body = ''
-
-  headers['Content-Length'] = body.length
-  headers['Content-Type'] = 'text/xml; charset="utf-8"'
-
-  if device.Cookies?
-    headers['Cookie'] = device['Cookies']
-
-  options = {
-    hostname: 'localhost',
-    port: 7547,
-    path: '/',
-    method: 'POST',
-    headers: headers
-  }
-
-  request = http.request(options, (response) ->
-    chunks = []
-    bytes = 0
-    response.on('data', (chunk) ->
-      chunks.push(chunk)
-      bytes += chunk.length
-    )
-    response.on('end', () ->
-      body = new Buffer(bytes)
-      offset = 0
-      chunks.forEach((chunk) ->
-        chunk.copy(body, offset, 0, chunk.length)
-        offset += chunk.length
-      )
-
-      if +response.headers['Content-Length'] > 0 or body.length > 0
-        xml = libxmljs.parseXml(body)
-      else
-        xml = null
-
-      if response.headers['set-cookie']?
-        # TODO set individual cookies
-        device['Cookies'] = [true, response.headers['set-cookie'], 'string']
-
-      callback(xml)
-    )
-  )
-  request.end(body)
-
-
-startSession = (device) ->
-  xmlOut = createSoapDocument(0)
-  methods.inform(device, xmlOut, (xml) ->
-    sendRequest(device, xml, (xml) ->
-      if not xml?
-        console.log('No inform response, trying again in 10 seconds.')
-        setTimeout(() ->
-          startSession(device)
-        , 10000)
-        return
-      sendRequest(device, null, (xml) ->
-        handleMethod(device, xml)
-      )
-    )
-  )
-
-
-createFaultResponse = (xmlOut, code, message) ->
-  body = xmlOut.root().childNodes()[1]
-
-  soapFault = body.node('soap-env:Fault')
-  soapFault.node('faultcode').text('Client')
-  soapFault.node('faultstring').text('CWMP fault')
-
-  fault = soapFault.node('detail').node('cwmp:Fault')
-  fault.node('FaultCode').text(code)
-  fault.node('FaultString').text(message)
-
-
-handleMethod = (device, xml) ->
-  if not xml?
-    setTimeout(() ->
-      startSession(device)
-    , 1000 * parseInt(device['InternetGatewayDevice.ManagementServer.PeriodicInformInterval'][1]))
-    return
-
-  requestId = xml.get('/soap-env:Envelope/soap-env:Header/cwmp:ID', NAMESPACES).text()
-
-  xmlOut = createSoapDocument(requestId)
-
-  element = xml.get('/soap-env:Envelope/soap-env:Body/cwmp:*', NAMESPACES)
-  method = methods[element.name()]
-
-  if method?
-    methods[element.name()](device, xml, xmlOut, (xml) ->
-      sendRequest(device, xml, (xml) ->
-        handleMethod(device, xml)
-      )
-    )
-  else
-    createFaultResponse(xmlOut, 9000, 'Method not supported')
-    sendRequest(device, xmlOut, (xml) ->
-      handleMethod(device, xml)
-    )
-
-
-cluster = require 'cluster'
-
-NUM_WORKERS = 1
-NUM_DEVICES = 1
-
-if cluster.isMaster
-  for i in [0 ... NUM_WORKERS]
-    cluster.fork({SERIAL_PREFIX : i})
-else
-  for i in [0...NUM_DEVICES]
-    serial = process.env.SERIAL_PREFIX + ("00000#{i}".slice(-6))
-    device = createDevice(serial)
-    do (device) ->
-      setTimeout(() ->
-        startSession(device)
-      , i * 100)

tools/dump-data-model

@@ -0,0 +1,91 @@
+#!/usr/bin/env node
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const fs = require("fs");
+const path = require("path");
+
+const outputFilename = path.resolve(`data_model_${process.argv[2]}.json`);
+process.chdir(path.resolve(path.dirname(fs.realpathSync(__filename)), ".."));
+
+const db = require("../lib/db");
+
+const dataModel = {};
+
+
+function recursive(object, path) {
+  if (!path)
+    path = "";
+
+  for (let key in object) {
+    if (key[0] === "_" || !object[key])
+      continue;
+
+    let isObject = object[key]["_object"];
+    let isValue = "_value" in object[key];
+
+    if ((isObject && isValue) || (!isObject && !isValue))
+      throw new Error("Unexpected object");
+
+    if (isValue) {
+      let value = object[key]["_value"];
+      if (typeof value === "boolean")
+        value = +value;
+
+      let values = [object[key]["_writable"] || false, value.toString()];
+      if (object[key]["_type"])
+        values.push(object[key]["_type"]);
+
+      dataModel[`${path}${key}`] = values;
+    }
+
+    if (isObject) {
+      let values = [object[key]["_writable"] || false];
+      dataModel[`${path}${key}.`] = values;
+      recursive(object[key], `${path}${key}.`);
+    }
+  }
+}
+
+
+db.connect(function(err) {
+  if (err)
+    throw err;
+
+  return db.devicesCollection.findOne({
+    _id: process.argv[2]
+  }, {}, function(err, device) {
+    if (err)
+      throw err;
+
+    if (!device)
+      throw new Error("Device not found");
+
+    recursive(device);
+
+    fs.writeFile(outputFilename, JSON.stringify(dataModel, null, 2), function() {
+      if (err)
+        throw err;
+
+      console.log(`${outputFilename} saved`);
+      process.exit(0);
+    });
+  });
+});

tools/genie_auth.pl

@@ -1,27 +0,0 @@
-#!/usr/bin/perl
-use Digest::MD5 qw(md5_base64);
-use URI::Escape;
-print "Enter the details *exactly* as sent in device TR-069 informs (case sensitive).\n";
-print 'Serial number: ';
-chomp($serial = <>);
-$serial = uri_escape($serial, '^A-Za-z0-9_');
-
-print 'OUI: ';
-chomp($oui = <>);
-$oui = uri_escape($oui, '^A-Za-z0-9_');
-
-print 'Product class (leave blank if not available): ';
-chomp($product_class = <>);
-$product_class = uri_escape($product_class, '^A-Za-z0-9_');
-
-if ($product_class eq '') {
-  $device_id = "$oui-$serial"
-}
-else {
-  $device_id = "$oui-$product_class-$serial"
-}
-
-print "ACS username: $device_id\n";
-print 'ACS password: ' . md5_base64("${device_id} Open Sesame") . "\n";
-print "Connection request username: $device_id\n";
-print 'Connection request password: ' . md5_base64("${device_id} Sesame, open") . "\n";

tools/update-deviceid.coffee

@@ -1,37 +0,0 @@
-# This script will generate _deviceId struct for all devices in DB.
-# To be used in migrating existing data to the recent update which introduced _deviceId object.
-config = require '../config'
-mongodb = require 'mongodb'
-
-dbserver = new mongodb.Server(config.MONGODB_SOCKET, 0, {auto_reconnect: true})
-db = new mongodb.Db(config.DATABASE_NAME, dbserver, {native_parser: true, safe: true})
-
-query = {}
-
-projection = {
-  _id : true,
-  'InternetGatewayDevice.DeviceInfo' : true
-}
-
-db.open((err, db) ->
-  db.collection('devices', (err, collection) ->
-    cur = collection.find(query)
-    cur.each((err, doc) ->
-      if not doc?
-        db.close()
-        return
-      serialNumber = doc['InternetGatewayDevice']['DeviceInfo']['SerialNumber']['_value']
-      oui = doc['InternetGatewayDevice']['DeviceInfo']['ManufacturerOUI']['_value']
-      manufacturer = doc['InternetGatewayDevice']['DeviceInfo']['Manufacturer']['_value']
-      productClass = doc['InternetGatewayDevice']['DeviceInfo']['ProductClass']['_value']
-      id = doc['_id']
-      collection.update({_id : id}, {$set : {
-        '_deviceId._Manufacturer' : manufacturer,
-        '_deviceId._OUI' : oui,
-        '_deviceId._ProductClass' : productClass,
-        '_deviceId._SerialNumber' : serialNumber
-      }}, () ->
-      )
-    )
-  )
-)