1.1.1

Because RequestDownload method is no longer supported.

.gitignore

@@ -1,3 +1,28 @@
 *~
-*.js
+lib/*.map
+lib/api-functions.js
+lib/auth.js
+lib/cache.js
+lib/common.js
+lib/config.js
+lib/cwmp.js
+lib/db.js
+lib/fs.js
+lib/nbi.js
+lib/query.js
+lib/server.js
+lib/session.js
 node_modules
+debug
+config/auth.js
+config/config.json
+config/cwmp.key
+config/cwmp.crt
+config/cwmp.ca-bundle
+config/nbi.key
+config/nbi.crt
+config/nbi.ca-bundle
+config/fs.key
+config/fs.crt
+config/fs.ca-bundle
+config/ext

.npmignore

@@ -1,14 +1,21 @@
 *~
 .npmignore
 
-config/*
-!config/*-sample.*
-!config/custom_commands
+config/auth.js
+config/config.json
+config/cwmp.key
+config/cwmp.crt
+config/cwmp.ca-bundle
+config/nbi.key
+config/nbi.crt
+config/nbi.ca-bundle
+config/fs.key
+config/fs.crt
+config/fs.ca-bundle
+config/ext
 
 debug/*
 !debug/.gitkeep
 
 lib/*.coffee
-tools/*.coffee
-test/*.coffee
-
+lib/*.map

CHANGELOG.md

@@ -0,0 +1,54 @@
+# Change Log
+
+## 1.1.1 (2017-03-23)
+
+- Avoid crashing when connection request credentials are missing.
+- Show a warning instead of crashing when failing to parse parameter values
+according to the expected value type.
+- Add missing "Registered" event.
+- Fix bug where in certain cases many more instances than declared are created.
+- Fix parameter discovery bug when declared path timestamp is 1 or is not set.
+- Fix preset precondition failing when testing against datetime parameters and
+certain other parameters like \_deviceId.\_ProductClass.
+
+## 1.1.0 (2017-03-10)
+
+- Provisions enable implementing dynamic device configuration or complex device
+provisioning work flow using arbitrary scripts.
+- Virtual parameters are user-defined parameters whose values are evaluated from
+a custom script.
+- Extensions are sandboxed Node.js scripts that are accessible from provision
+and virtual parameter scripts to facilitate integration with external entities.
+- Support for UDP/STUN based connection requests for reaching devices behind
+NAT (TR-069 Annex G).
+- Presets can now be scheduled using a cron-like expression.
+- Presets can now be tied to specific device events (e.g. boot).
+- Presets precondition queries no longer support "$or" or other MongoDB logical
+operators.
+- Faults are no longer a part of tasks but are now first class objects.
+- Presets are now assigned to channels. A fault in one channel only blocks
+presets in that channel.
+- New API CRUD functions for provisions, virtual parameters, and faults.
+- New config options for XML output.
+- API responses now include "GenieACS-Version" header.
+- Graceful shutdown when receiving SIGINT and SIGTERM events.
+- Support SSL intermediate certificate chains.
+- Supported Node.js versions are 6.x and 7.x.
+- Supported MongoDB versions are 2.6 through 3.4.
+- Expect performance differences due to major under the hood changes. Some
+operations are faster and some are slower. Overall performance is improved.
+- GenieACS will no longer fetch the entire device data model upon first contact
+but will instead only fetch the parameters it needs to fulfill the presets.
+- Logs have been overhauled and split into two streams: process log (stderr) and
+access log (stdout). Also added config options to dump logs to files rather than
+standard streams.
+- Connection request authentication credentials are picked up from the device
+data model if available. config/auth.js is still supported as a fallback and now
+supports an optional callback argument.
+- Custom commands have been removed. Use virtual parameters and/or extensions.
+- Aliases and value normalizers (config/parameters.json) have been removed. Use
+virtual parameters.
+- The API /devices/<device_id>/preset has been removed.
+- Rarely used RequestDownload method no longer supported.
+- The TR-069 client simulator has moved to its own repo at
+https://github.com/zaidka/genieacs-sim

CONTRIBUTING.md

@@ -0,0 +1,39 @@
+# Have a question?
+
+Please ask in the [Users mailing list][mailing-lists] instead of opening a
+Github Issue. The Github issue tracker is strictly for bug reports.
+
+# Issues
+
+We use GitHub issues to track bugs. Please ensure your bug description is
+clear and has sufficient instructions to be able to reproduce the issue.
+
+If you face interoperability issues with your device, more often than not it's
+a device related issue. Please consult the [Users mailing list][mailing-lists]
+before submitting an Issue.
+
+# Pull Requests
+
+Pull requests are welcome. If you have a fix for a bug, go ahead and create a
+pull request. For other kinds of enhancements, please discuss in the [Users
+mailing list][mailing-lists] first.
+
+Please follow the guidelines [here][commit-guidelines] when committing your
+changes.
+
+## Contributor License Agreement
+
+By submitting a pull request to this repository, you acknowledge that, while
+maintaining copyright, you grant Zaid Abdulla a perpetual, worldwide, non-
+exclusive, no-charge, royalty-free, irrevocable license to reproduce, prepare
+derivative works of, publicly display, publicly perform, sublicense, and
+distribute your contributions and such derivative works under the AGPLv3
+license or any other license terms, including, but not limited to, proprietary
+or commercial license terms.
+
+You confirm that you own or have rights to distribute and sublicense the
+source code contained therein, and that your content does not infringe upon
+the intellectual property rights of a third party.
+
+[commit-guidelines]: http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html
+[mailing-lists]: http://lists.genieacs.com

README.md

@@ -1,66 +1,75 @@
 # GenieACS
 
-GenieACS is a *blazing fast* TR-069 auto configuration server (ACS) built with Node.js, Redis, and MongoDB. It's technology neutral and configurable to fit any service provider needs. This is the core back end component. Official front end GUI is available at https://github.com/zaidka/genieacs-gui.
+GenieACS is a fast and lightweight TR-069 auto configuration server (ACS). This
+is the core back end component. A GUI front end is available at
+https://github.com/zaidka/genieacs-gui.
 
 ## Features
 
-* **Massively concurrent**: Can handle hundreds of thousands of connected devices on a single server even with low inform interval.
-* **Preset-based configuration**: Define sets of configurations that devices will pick up and apply as needed based on given preconditions.
-* **Tagging**: Use tags to group devices for more manageable presets.
-* **Searching**: Query for devices on any parameter (supports all common operators including regular expressions).
-* **Parameter aliasing**: Define aliases to unify parameter paths from different types of devices. Aliases behave like normal parameters (i.e. can be queried or modified).
-* **Extensive API**: A simple yet rich HTTP-based API allows easy integration with other system.
+- **Battle-tested at scale**: Can scale to manage hundreds of thousands and
+potentially millions of devices
+- **Powerful provisioning system**: Scriptable configuration to handle complex
+device provisioning scenarios and other automated operations such as firmware
+upgrade
+- **Extensive API**: A rich HTTP-based API allows easy integration with other
+systems
 
-## Getting started
+## Quick Start
 
-Install [Node.js](http://nodejs.org/), [Redis](http://redis.io/), and [MongoDB](http://www.mongodb.org/). Refer to the corresponding documentation for installation guides. Then use NPM to install GenieACS by typing:
+Install [Node.js](http://nodejs.org/), [Redis](http://redis.io/), and
+[MongoDB](http://www.mongodb.org/). Refer to the corresponding documentation for
+installation guides. Supported versions are:
+
+- Node.js: 6.x and 7.x (7.x recommended)
+- Redis: 2.4 through 3.2
+- MongoDB: 2.6 through 3.4
+
+Then install GenieACS using NPM:
 
     npm install -g genieacs
 
-You may need to modify the configuration files under "config" directory (in /lib/node_modules/genieacs/config) depending on your setup.
+Alternatively, you can install from source by cloning the git repository:
 
-Alternatively, you can use git to get the latest development version (not recommended for production use):
-
-    cd /opt
     git clone https://github.com/zaidka/genieacs.git
     cd genieacs
+    git checkout $(git tag -l v1.1.* --sort=-v:refname | head -n 1)
     npm install
-    npm run configure
     npm run compile
 
-Finally, run the following in GNU Screen session or something similar:
+Finally, run the following (from bin directory if installing from source):
 
     genieacs-cwmp
 
-This is the service that the CPEs will communicate with. It listens to port 7547 by default (see config/config.json). Configure the ACS URL of your devices accordingly.
+This is the service that the CPEs will communicate with. It listens to port 7547
+by default (see config/config.json). Configure the ACS URL of your devices
+accordingly.
 
     genieacs-nbi
 
-This is the northbound interface module. It exposes a REST API on port 7557 by default. This is needed for the GUI front end to communicate with.
+This is the northbound interface module. It exposes a REST API on port 7557 by
+default. This must be running for the GUI front end to work.
 
     genieacs-fs
 
-This is the file server from which the CPEs will download firmware images and other types of files.
+This is the file server from which the CPEs will download firmware images and
+such.
 
-You can use stream redirection to output to log files:
+Note: For production deployment make sure to run those as background services.
 
-    genieacs-cwmp >> /var/log/genieacs-cwmp.log 2>> /var/log/genieacs-cwmp-err.log
-    genieacs-nbi >> /var/log/genieacs-nbi.log 2>> /var/log/genieacs-nbi-err.log
-    genieacs-fs >> /var/log/genieacs-fs.log 2>> /var/log/genieacs-fs-err.log
-
-For further details about installation and configuration, refer to the [wiki section](https://github.com/zaidka/genieacs/wiki).
-
-You may now proceed with installing [GenieACS GUI front end](https://github.com/zaidka/genieacs-gui).
+For further details about installation and configuration, refer to the
+[wiki section](https://github.com/zaidka/genieacs/wiki).
 
 ## Support
 
-Documentation is still work in progress. Feel free to contact me if you require any assistance in installation, configuration, or using the APIs.
+The [Users mailing list](http://lists.genieacs.com) is a good place to get
+guidance and help from the community. Head on over and join the conversation!
+In addition, the [wiki](https://github.com/zaidka/genieacs/wiki) provides useful
+documentation and tips from GenieACS users.
 
-You may submit bug reports or feature requests [here](https://github.com/zaidka/genieacs/issues).
+For commercial support options and professional services, please visit
+[genieacs.com](https://genieacs.com).
 
-For commercial support options, please contact me.
-
-## Contributing
-
-Contributions are welcome. Fork this repo and open a pull request and wait for feedback. You can also contribute by enhancing the documentation in the [wiki section](https://github.com/zaidka/genieacs/wiki).
+## License
 
+Copyright 2013-2017 Zaid Abdulla. GenieACS is released under the
+[AGPLv3 license terms](https://raw.githubusercontent.com/zaidka/genieacs/master/LICENSE).

bin/genieacs-cwmp

@@ -2,9 +2,8 @@
 var path = require('path');
 var fs = require('fs');
 
-var dir = path.join(path.dirname(fs.realpathSync(__filename)), '../lib/');
-process.title = 'genieacs-cwmp';
+var dir = path.resolve(path.dirname(fs.realpathSync(__filename)), '..');
 process.chdir(dir);
 
-require(path.join(dir, './cwmp'));
-
+var cluster = require(path.resolve(dir, 'lib/cluster'));
+cluster.start('cwmp');

bin/genieacs-fs

@@ -2,9 +2,8 @@
 var path = require('path');
 var fs = require('fs');
 
-var dir = path.join(path.dirname(fs.realpathSync(__filename)), '../lib/');
-process.title = 'genieacs-fs';
+var dir = path.resolve(path.dirname(fs.realpathSync(__filename)), '..');
 process.chdir(dir);
 
-require(path.join(dir, './fs'));
-
+var cluster = require(path.resolve(dir, 'lib/cluster'));
+cluster.start('fs');

bin/genieacs-nbi

@@ -2,9 +2,8 @@
 var path = require('path');
 var fs = require('fs');
 
-var dir = path.join(path.dirname(fs.realpathSync(__filename)), '../lib/');
-process.title = 'genieacs-nbi';
+var dir = path.resolve(path.dirname(fs.realpathSync(__filename)), '..');
 process.chdir(dir);
 
-require(path.join(dir, './nbi'));
-
+var cluster = require(path.resolve(dir, 'lib/cluster'));
+cluster.start('nbi');

config/auth-sample.js

@@ -1,4 +1,7 @@
-exports.connectionRequest = function(deviceId) {
-  // return username/password pair for a given device
-  return ["", ""];
+"use strict";
+
+function connectionRequest(deviceId, url, username, password, callback) {
+  return callback(username || deviceId, password || "");
 }
+
+exports.connectionRequest = connectionRequest;

config/config-sample.json

@@ -1,16 +1,15 @@
 {
-  "DATABASE_NAME" : "genieacs",
-  "MONGODB_SOCKET" : "/tmp/mongodb-27017.sock",
-  "REDIS_SOCKET" : "6379",
-  "ACS_INTERFACE" : "0.0.0.0",
-  "ACS_PORT" : 7547,
-  "ACS_HTTPS_INTERFACE" : "0.0.0.0",
-  "ACS_HTTPS_PORT" : 7548,
-  "API_INTERFACE" : "0.0.0.0",
-  "API_PORT" : 7557,
-  "FILES_INTERFACE" : "0.0.0.0",
-  "FILES_PORT" : 7567,
-  "FILES_IP" : "192.168.0.1",
+  "MONGODB_CONNECTION_URL" : "mongodb://127.0.0.1/genieacs",
+  "REDIS_PORT" : "6379",
+  "REDIS_HOST" : "127.0.0.1",
+  "CWMP_INTERFACE" : "0.0.0.0",
+  "CWMP_PORT" : 7547,
+  "CWMP_SSL" : false,
+  "NBI_INTERFACE" : "0.0.0.0",
+  "NBI_PORT" : 7557,
+  "FS_INTERFACE" : "0.0.0.0",
+  "FS_PORT" : 7567,
+  "FS_IP" : "192.168.0.1",
   "LOG_INFORMS" : true,
-  "DEBUG_DEVICES" : {}
+  "DEBUG" : false
 }

config/custom_commands-sample.json

@@ -1,4 +0,0 @@
-{
-  "seowon_frequencies" : "/^[0-9]+\\-SWU\\%2D9200\\-[0-9A-Z]+$/",
-  "greenpacket_frequencies" : "/^[0-9A-Z]+\\-DX350\\-[0-9A-Z]+$/"
-}

config/custom_commands/greenpacket_frequencies.js

@@ -1,188 +0,0 @@
-var net = require('net');
-var PORT = 23;
-var USERNAME = 'mt7109';
-var PASSWORD = 'wimax';
-var TIMEOUT = 5000;
-
-var FREQUENCIES_REGEX = /([0-9,]+)/
-var BANDWIDTHS_REGEX = /[0-9]+\-([0-9,]+)/
-var DEFAULT_BW_REGEX = /([0-9]+)\-[0-9,]+/
-var telnetConnect = function(host, callback) {
-  var client = net.connect(PORT, host, function() {
-    // enter username
-    telnetExecute(USERNAME, client, "Password: ", function(err, response) {
-      if (err) return callback(err);
-      // enter password
-      telnetExecute(PASSWORD, client, "\n# ", function(err, response) {
-        return callback(err);
-      });
-    });
-  });
-  return client;
-}
-
-var telnetExecute = function(command, connection, prompt, callback) {
-  var response = '';
-  command += "\n";
-  var listener = function(chunk) {
-    response += chunk.toString();
-    if (response.slice(0 - prompt.length) == prompt) {
-      connection.removeListener('data', listener);
-      return callback(null, response.slice(command.length, 0 - prompt.length));
-    }
-  };
-
-  var end = function() {
-    connection.removeListener('end', end);
-    return callback(null, response.slice(command.length));
-  };
-
-  if (prompt)
-    connection.on('data', listener);
-  else
-    connection.on('end', end);
-
-  if (command)
-    connection.write(command);
-}
-
-var getDeviceIp = function(deviceId, callback) {
-  var db = require('../../db');
-  var URL = require('url');
-
-  db.devicesCollection.findOne({_id : deviceId}, {'InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value' : 1}, function(err, device) {
-    var connectionRequestUrl = device.InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value;
-    var url = URL.parse(connectionRequestUrl);
-    return callback(url.hostname);
-  });
-};
-
-exports.init = function(deviceId, args, callback) {
-  return exports.get(deviceId, args, callback);
-}
-
-exports.get = function(deviceId, args, callback) {
-  var currentFrequencies = [];
-  var currentBandwidths = [];
-  var defaultBandwidth = '';
-  var final = [];
-
-  getDeviceIp(deviceId, function(ip) {
-    var client = telnetConnect(ip, function(err) {
-      // get frequencies
-      telnetExecute("sncfg get WMX_FREQ_LIST", client, "\n# ", function(err, response) {
-        if (err) return callback(err);
-        currentFrequencies = FREQUENCIES_REGEX.exec(response)[1].split(',')
-        // get bandwidths
-        telnetExecute("sncfg get WMX_FREQ_BANDWDITH_FL", client, "\n# ", function(err, response) {
-          if (err) return callback(err);
-          currentBandwidths = BANDWIDTHS_REGEX.exec(response)[1].split(',')
-          defaultBandwidth = DEFAULT_BW_REGEX.exec(response)[1]
-          // generate freq/bw pairs string
-          for (var i = 0; i < currentFrequencies.length; i++) {
-            final.push(currentFrequencies[i] + ' ' + currentBandwidths[i])
-          }
-          // log out
-          telnetExecute("exit", client, null, function(err, response) {
-            client.end();
-            return callback(err, defaultBandwidth + '-' + final.join(','));
-          });
-        });
-      });
-    });
-
-    client.on("error", function(err) {
-      client.destroy();
-      callback(err);
-    });
-
-    client.setTimeout(TIMEOUT, function() {
-      client.destroy();
-      callback("timeout");
-    });
-  });
-};
-
-var parsePairs = function(pairsString) {
-  var a = pairsString.split(',');
-  var f = [];
-  var b = [];
-  var l = [];
-  for (i in a) {
-    var sp = a[i].trim().split(' ');
-    f.push(parseInt(sp[0]));
-    b.push(parseFloat(sp[1]));
-  }
-  l.push(f.join(','));
-  l.push(b.join(','));
-  return l;
-}
-
-exports.set = function(deviceId, args, callback) {
-  executeSet(deviceId, args, ['set'], callback);
-};
-
-exports.dset = function(deviceId, args, callback) {
-  executeSet(deviceId, args, ['dset'], callback);
-};
-
-exports.setdset = function(deviceId, args, callback) {
-  executeSet(deviceId, args, ['dset', 'set'], callback);
-};
-
-var executeSet = function(deviceId, args, method, callback) {
-  var tmp = args.split('-');
-  var defaultBandwidth = tmp[0];
-  var pairs = parsePairs(tmp[1]);
-  var frequenciesString = pairs[0];
-  var bandwidthsString = pairs[1];
-
-  getDeviceIp(deviceId, function(ip) {
-    var client = telnetConnect(ip, function(err) {
-      var telnet_cmd = "sncfg " + method[0];
-      telnetExecute(telnet_cmd + " WMX_FREQ_BANDWDITH_FL '" + defaultBandwidth + "-" + bandwidthsString + "'", client, "\n# ", function(err, response) {
-        if (err) return callback(err);
-        telnetExecute(telnet_cmd + " WMX_FREQ_LIST '" + frequenciesString + "'", client, "\n# ", function(err, response) {
-          if (err) return callback(err);
-          if (method.length > 1) {
-            telnet_cmd = "sncfg " + method[1];
-            telnetExecute(telnet_cmd + " WMX_FREQ_BANDWDITH_FL '" + defaultBandwidth + "-" + bandwidthsString + "'", client, "\n# ", function(err, response) {
-              if (err) return callback(err);
-              telnetExecute(telnet_cmd + " WMX_FREQ_LIST '" + frequenciesString + "'", client, "\n# ", function(err, response) {
-                if (err) return callback(err);
-                telnetExecute("sncfg commit", client, "\n# ", function(err, response) {
-                  if (err) return callback(err);
-                  // log out
-                  telnetExecute("exit", client, null, function(err, response) {
-                    client.end();
-                    return callback(null, args);
-                  });
-                });
-              });
-            });
-          }
-          else {
-            telnetExecute("sncfg commit", client, "\n# ", function(err, response) {
-            if (err) return callback(err);
-              // log out
-              telnetExecute("exit", client, null, function(err, response) {
-                client.end();
-                return callback(null, args);
-              });
-            });
-          }
-        });
-      });
-    });
-
-    client.on("error", function(err) {
-      client.destroy();
-      callback(err);
-    });
-
-    client.setTimeout(TIMEOUT, function() {
-      client.destroy();
-      callback("timeout");
-    });
-  });
-};

config/custom_commands/seowon_frequencies.js

@@ -1,208 +0,0 @@
-// Change frequency plans for Seowon devices.
-// Args format: 3532500 5000,3537500 5000,3542500 5000,3547500 5000,3552500 5000,3557500 5000,3505000 10000,3515000 10000,3525000 10000,3535000 10000,3545000 10000,3555000 10000
-var net = require('net');
-var PORT = 23;
-var USERNAME = 'admin';
-var PASSWORD = 'admin';
-var TIMEOUT = 5000;
-
-var FREQUENCIES_REGEX = /\s*([\d]+)\.\s*Frequency\s*:\s*([\d]+)\s\(KHz\),\s*Bandwidth\s*:\s*([\d\.]+)\s*\(MHz\)/gm
-
-var telnetConnect = function(host, callback) {
-  var client = net.connect(PORT, host, function() {
-    // enter username
-    telnetExecute(USERNAME, client, "Password: ", function(err, response) {
-      if (err) return callback(err);
-      // enter password
-      telnetExecute(PASSWORD, client, "MT7109> ", function(err, response) {
-        return callback(err);
-      });
-    });
-  });
-  return client;
-}
-
-var telnetExecute = function(command, connection, prompt, callback) {
-  var response = '';
-  command += "\n";
-  var listener = function(chunk) {
-    response += chunk.toString();
-    if (response.slice(0 - prompt.length) == prompt) {
-      connection.removeListener('data', listener);
-      return callback(null, response.slice(command.length, 0 - prompt.length));
-    }
-  };
-
-  var end = function() {
-    connection.removeListener('end', end);
-    return callback(null, response.slice(command.length));
-  };
-
-  if (prompt)
-    connection.on('data', listener);
-  else
-    connection.on('end', end);
-
-  if (command)
-    connection.write(command);
-}
-
-var parseFrequencies = function(frequenciesString) {
-  var a = frequenciesString.split(',');
-  var l = [];
-  for (i in a) {
-    var sp = a[i].trim().split(' ');
-    var frequency = parseInt(sp[0]);
-    var bandwidth = parseFloat(sp[1]);
-    l.push([frequency, bandwidth]);
-  }
-  return l;
-}
-
-var findFrequency = function(haystack, needle) {
-  for (i in haystack) {
-    if (haystack[i][0] == needle[0] && haystack[i][1] == needle[1])
-      return i;
-  }
-  return -1;
-}
-
-var getDeviceIp = function(deviceId, callback) {
-  var db = require('../../db');
-  var URL = require('url');
-
-  db.devicesCollection.findOne({_id : deviceId}, {'InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value' : 1}, function(err, device) {
-    var connectionRequestUrl = device.InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value;
-    var url = URL.parse(connectionRequestUrl);
-    return callback(url.hostname);
-  });
-};
-
-exports.init = function(deviceId, args, callback) {
-  return exports.get(deviceId, args, callback);
-};
-
-exports.get = function(deviceId, args, callback) {
-  var currentFrequencies = [];
-
-  getDeviceIp(deviceId, function(ip) {
-    var client = telnetConnect(ip, function(err) {
-      // enter privileged mode
-      telnetExecute("enable", client, "MT7109# ", function(err, response) {
-        if (err) return callback(err);
-        // show frequencies
-        telnetExecute("show wmx freq", client, "MT7109# ", function(err, response) {
-          if (err) return callback(err);
-          while (f = FREQUENCIES_REGEX.exec(response)) {
-            var frequency = parseInt(f[2]);
-            var bandwidth = Math.floor(parseFloat(f[3]) * 1000);
-            var freqStr = String(frequency) + ' ' + String(bandwidth);
-            currentFrequencies.push(freqStr);
-          }
-          // log out
-          telnetExecute("logout", client, null, function(err, response) {
-            client.end();
-            return callback(err, currentFrequencies.join(','));
-          });
-        });
-      });
-    });
-
-    client.on("error", function(err) {
-      client.destroy();
-      callback(err);
-    });
-
-    client.setTimeout(TIMEOUT, function() {
-      client.destroy();
-      callback("timeout");
-    });
-  });
-};
-
-exports.set = function(deviceId, args, callback) {
-  var newFrequencies = parseFrequencies(args);
-  var addFrequencies = [];
-  var delFrequencies = [];
-
-  getDeviceIp(deviceId, function(ip) {
-    var client = telnetConnect(ip, function(err) {
-      // enter privileged mode
-      telnetExecute("enable", client, "MT7109# ", function(err, response) {
-        if (err) return callback(err);
-        // show frequencies
-        telnetExecute("show wmx freq", client, "MT7109# ", function(err, response) {
-          if (err) return callback(err);
-          while (f = FREQUENCIES_REGEX.exec(response)) {
-            var index = parseInt(f[1]);
-            var frequency = parseInt(f[2]);
-            var bandwidth = Math.floor(parseFloat(f[3]));
-
-            var i = findFrequency(newFrequencies, [frequency, bandwidth * 1000]);
-            if (i == -1)
-              delFrequencies.push(index);
-            else
-              newFrequencies.splice(i, 1);
-          }
-          for (f in newFrequencies)
-            addFrequencies.push(newFrequencies[f]);
-
-          if (addFrequencies.length == 0 && delFrequencies.length == 0) {
-            // log out
-            telnetExecute("logout", client, null, function(err, response) {
-              client.end();
-              return callback(err, args);
-            });
-            return;
-          }
-
-          telnetExecute("wimax", client, "MT7109 (WiMax)# ", function(err, response) {
-            if (err) return callback(err);
-            var add, del;
-            del = function() {
-              if (delFrequencies.length == 0) {
-                add();
-                return;
-              }
-              var i = delFrequencies.pop();
-              telnetExecute("wmx freq del " + i, client, "MT7109 (WiMax)# ", function(err, response) {
-                if (err) return callback(err);
-                del();
-              });
-            };
-            add = function() {
-              if (addFrequencies.length == 0) {
-                // commit
-                telnetExecute("commit", client, "MT7109 (WiMax)# ", function(err, response) {
-                  if (err) return callback(err);
-                  // log out
-                  telnetExecute("logout", client, null, function(err, response) {
-                    client.end();
-                    return callback(null, args);
-                  });
-                });
-                return;
-              }
-              var i = addFrequencies.pop();
-              telnetExecute("wmx freq add " + i[0] + ' ' + i[1]/1000, client, "MT7109 (WiMax)# ", function(err, response) {
-                if (err) return callback(err);
-                add();
-              });
-            };
-            del();
-          });
-        });
-      });
-    });
-
-    client.on("error", function(err) {
-      client.destroy();
-      callback(err);
-    });
-
-    client.setTimeout(TIMEOUT, function() {
-      client.destroy();
-      callback("timeout");
-    });
-  });
-};

config/ext-sample.js

@@ -0,0 +1,34 @@
+// This is an exmaple GenieACS extension to get the current latitude/longitude
+// of the International Space Station. Why, you ask? Because why not.
+// To install, copy this file to config/ext/iss.js.
+
+"use strict";
+
+const http = require("http");
+
+let cache = null;
+let cacheExpire = 0;
+
+function latlong(args, callback) {
+  if (Date.now() < cacheExpire)
+    return callback(null, cache);
+
+  http.get("http://api.open-notify.org/iss-now.json", (res) => {
+    if (res.statusCode !== 200)
+      return callback(new Error(`Request failed (status code: ${res.statusCode})`));
+
+    let rawData = "";
+    res.on("data", (chunk) => rawData += chunk);
+
+    res.on("end", () => {
+      let pos = JSON.parse(rawData)["iss_position"];
+      cache = [+pos["latitude"], +pos["longitude"]];
+      cacheExpire = Date.now() + 10000;
+      callback(null, cache);
+    });
+  }).on("error", (err) => {
+    callback(err);
+  });
+}
+
+exports.latlong = latlong;

config/httpscert-sample.crt

@@ -1,17 +0,0 @@
------BEGIN CERTIFICATE-----
-MIICqTCCAhICCQDSrolRw1qPBzANBgkqhkiG9w0BAQUFADCBmDELMAkGA1UEBhMC
-SVExFTATBgNVBAgMDFN1bGF5bWFuaXlhaDEVMBMGA1UEBwwMU3VsYXltYW5peWFo
-MQ8wDQYDVQQKDAZGYW5vb3MxEzARBgNVBAsMClRlY2hub2xvZ3kxDjAMBgNVBAMM
-BWdlbmllMSUwIwYJKoZIhvcNAQkBFhZ6YWlkLmFiZHVsbGFAZmFub29zLmlxMB4X
-DTEzMDIxNzEyNTYzN1oXDTE0MDIxNzEyNTYzN1owgZgxCzAJBgNVBAYTAklRMRUw
-EwYDVQQIDAxTdWxheW1hbml5YWgxFTATBgNVBAcMDFN1bGF5bWFuaXlhaDEPMA0G
-A1UECgwGRmFub29zMRMwEQYDVQQLDApUZWNobm9sb2d5MQ4wDAYDVQQDDAVnZW5p
-ZTElMCMGCSqGSIb3DQEJARYWemFpZC5hYmR1bGxhQGZhbm9vcy5pcTCBnzANBgkq
-hkiG9w0BAQEFAAOBjQAwgYkCgYEAxtHC3jlOoVsZQ9I8Wgd+8NkEaRfc01lRIAzj
-eWuUO/xNY54EUDHecRxwolHtoGkkZAp4kKR3uUVdAo9uXmNT2CeeZ5lfRoa8WmgG
-Yd6PExsdc4ZeKcw20XrKtQICqmU6rGlLERMFX97ejQh3eUmqwbzNaXiURxcKGdeK
-2C/NxoUCAwEAATANBgkqhkiG9w0BAQUFAAOBgQAYFw6dy/z74ahnA5QpuJOjjTrt
-AJaL9iuL1xS6TFfzne9UMIT+fFJsgSy6xx3TGkvbZJTnCzJN7I2uqmAhMoGFy7Ea
-svCCkj8U5qkjqgwwi7IbwQ7fXxTVdPqUlkIdNB2YLDELksxCX+ZzGQPiFB8WrSoO
-OBvB+hhPFpgp3aO2AQ==
------END CERTIFICATE-----

config/httpscert-sample.key

@@ -1,15 +0,0 @@
------BEGIN RSA PRIVATE KEY-----
-MIICXgIBAAKBgQDG0cLeOU6hWxlD0jxaB37w2QRpF9zTWVEgDON5a5Q7/E1jngRQ
-Md5xHHCiUe2gaSRkCniQpHe5RV0Cj25eY1PYJ55nmV9GhrxaaAZh3o8TGx1zhl4p
-zDbResq1AgKqZTqsaUsREwVf3t6NCHd5SarBvM1peJRHFwoZ14rYL83GhQIDAQAB
-AoGAR4NpuW8ZhlZzNv7m8Gcj1NPqjzajGm5VDNc19xWoAw+UcpGC8amOXlcheHFJ
-rNpE2asehMBzw/NcKjCXPWIy5LnGmttJMN2EbT2h/+uxBIM5IUc+joF744xF4C0t
-5zrtqQYJxC7sBLX2vf7nY6iuovUM7fHEwACyV3Rpw3zdZf0CQQDiqW+ugLKcfS/c
-6hkGTxeYWF4gUJTf3ke3au8nyMmUhyBwOuP4kpO72mK8bMLtjZRk3g6YChA4qH41
-nfXUXIh7AkEA4I2/zj4GDjjspDR+/tX0iUHQk1Rm2CRmRXXFJV1f688OLODCcOFK
-Ms/+qnbe41wXo1NUxCqVrnPwEh3MVl88/wJBANUTpXqtX7s+8ETECTxmE+9Se0Hz
-ntCJXG/v5d1Ho0Cueu8mXXMrEBiTpygtMl4xL2RXX+7L3w30T5lF4nh9eIsCQQDY
-DmsbdL/81TaTc9MxPQlxOzsFVrMxiixA7uTGY8ZULD/2XDeZxprOsyPUPlWMquIv
-ipFHwbWNSg4WUEd4X2BfAkEAytze+0mCd0WCsKbKwfTQnmJ3bK7JQywtC03llWyA
-Om3Vl61Syaa7M2hl5rLvitZMvU1Wd0s8raOXN8RUuRXZBw==
------END RSA PRIVATE KEY-----

config/parameters-sample.json

@@ -1,19 +0,0 @@
-{
-  "_lastInform" : {"alias" : "summary.lastInform", "type" : "date"},
-  "_lastBoot" : {"alias" : "summary.lastBoot", "type" : "date"},
-  "_lastBootstrap" : {"alias" : "summary.lastBootstrap", "type" : "date"},
-  "_deviceId._SerialNumber" : {"alias" : "summary.serialNumber"},
-  "_deviceId._ProductClass" : {"alias" : "summary.productClass"},
-  "_deviceId._OUI" : {"alias" : "summary.oui"},
-  "_deviceId._Manufacturer" : {"alias" : "summary.manufacturer"},
-  "InternetGatewayDevice.DeviceInfo.HardwareVersion" : {"alias" : "summary.hardwareVersion"},
-  "InternetGatewayDevice.DeviceInfo.SoftwareVersion" : {"alias" : "summary.softwareVersion"},
-
-  "InternetGatewayDevice.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANIPConnection./^\\d+$/.MACAddress" : {"alias" : "summary.mac", "type" : "mac"},
-  "InternetGatewayDevice.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANIPConnection./^\\d+$/.ExternalIPAddress" : {"alias" : "summary.ip"},
-  "InternetGatewayDevice.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANPPPConnection./^\\d+$/.MACAddress" : {"alias" : "summary.mac", "type" : "mac"},
-  "InternetGatewayDevice.WANDevice./^\\d+$/.WANConnectionDevice./^\\d+$/.WANPPPConnection./^\\d+$/.ExternalIPAddress" : {"alias" : "summary.ip"},
-
-  "InternetGatewayDevice.LANDevice./^\\d+$/.WLANConfiguration./^\\d+$/.SSID" : {"alias" : "summary.wlanSsid"},
-  "InternetGatewayDevice.LANDevice./^\\d+$/.WLANConfiguration./^\\d+$/.PreSharedKey.1.KeyPassphrase" : {"alias" : "summary.wlanPassphrase"}
-}

lib/api-functions.coffee

@@ -1,3 +1,25 @@
+###
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
+#
+# GenieACS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# GenieACS is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+###
+
+crypto = require 'crypto'
+dgram = require 'dgram'
+
 config = require './config'
 db = require './db'
 http = require 'http'
@@ -7,89 +29,171 @@ URL = require 'url'
 auth = require './auth'
 
 
-connectionRequest = (deviceId, callback) ->
-  # ensure socket is reused in case of digest authentication
-  agent = new http.Agent({maxSockets : 1})
+udpConReq = (address, un, key, callback) ->
+  return false if not address
+  [host, port] = address.split(':', 2)
+  if port is undefined
+    port = 80
+  else
+    port = parseInt(port)
+    return false if isNaN(port)
 
-  conReq = (url, authString, callback) ->
-    options = URL.parse(url)
-    options.agent = agent
+  ts = Math.trunc(Date.now() / 1000)
+  id = Math.trunc(Math.random() * 4294967295)
+  cn = crypto.randomBytes(8).toString('hex')
+  sig = crypto.createHmac('sha1', key).update("#{ts}#{id}#{un}#{cn}").digest('hex')
+  uri = "http://#{address}?ts=#{ts}&id=#{id}&un=#{un}&cn=#{cn}&sig=#{sig}"
 
-    if authString
-      options.headers = {'Authorization' : authString}
+  message = Buffer.from("GET #{uri} HTTP/1.1\r\nHost: #{address}\r\n\r\n")
 
-    request = http.get(options, (res) ->
-      if res.statusCode == 401
-        authHeader = auth.parseAuthHeader(res.headers['www-authenticate'])
-      callback(res.statusCode, authHeader)
-      # don't need body, go ahead and emit events to free up the socket for possible reuse
-      res.resume()
-    )
+  client = dgram.createSocket('udp4')
 
-    request.on('error', (err) ->
-      # error event when request is aborted
-      request.abort()
-      callback(0)
-    )
+  count = 3
+  client.send(message, 0, message.length, port, host, f = (err) ->
+    if err or -- count <= 0
+      client.close()
+      return callback(err)
+    client.send(message, 0, message.length, port, host, f)
+  )
+  return true
 
-    request.on('socket', (socket) ->
-      socket.setTimeout(2000)
-      socket.on('timeout', () ->
-        request.abort()
-      )
-    )
 
-  db.devicesCollection.findOne({_id : deviceId}, {'InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value' : 1}, (err, device)->
-    if err
-      callback(err)
-      return
-    connectionRequestUrl = device.InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value
-    # for testing
-    #connectionRequestUrl = connectionRequestUrl.replace(/^(http:\/\/)([0-9\.]+)(\:[0-9]+\/[a-zA-Z0-9]+\/?$)/, '$110.1.1.254$3')
-    conReq(connectionRequestUrl, null, (statusCode, authHeader) ->
-      if statusCode == 401
-        [username, password] = config.auth.connectionRequest(deviceId)
-        if authHeader.method is 'Basic'
-          authString = auth.basic(username, password)
-        else if authHeader.method is 'Digest'
-          uri = URL.parse(connectionRequestUrl)
-          authString = auth.digest(username, password, uri.path, 'GET', null, authHeader)
+httpConReq = (url, username, password, timeout, callback) ->
+  options = URL.parse(url)
+  # Ensure socket is reused in case of digest authentication
+  options.agent = new http.Agent({maxSockets : 1})
 
-        conReq(connectionRequestUrl, authString, (statusCode, authHeader) ->
-          callback()
-        )
+  statusToError = (statusCode) ->
+    switch statusCode
+      when 200, 204
+        null
+      when 401
+        new Error('Incorrect connection request credentials')
+      when 0
+        new Error('Device is offline')
       else
-        callback()
-    )
+        new Error("Unexpected response code from device: #{statusCode}")
+
+  request = http.get(options, (res) ->
+    if res.statusCode == 401 and res.headers['www-authenticate']?
+      authHeader = auth.parseAuthHeader(res.headers['www-authenticate'])
+      if authHeader.method is 'Basic'
+        options.headers = {'Authorization' : auth.basic(username or '', password or '')}
+      else if authHeader.method is 'Digest'
+        options.headers = {
+          'Authorization' : auth.digest(username or '', password or '', options.path, 'GET', null, authHeader)
+        }
+
+      request = http.get(options, (res) ->
+        if res.statusCode == 0
+          # Workaround for some devices unexpectedly closing the connection
+          request = http.get(options, (res) ->
+            callback(statusToError(res.statusCode))
+            res.resume()
+          ).on('error', (err) ->
+            request.abort()
+            callback(statusToError(0))
+          ).on('socket', (socket) ->
+            socket.setTimeout(timeout)
+            socket.on('timeout', () -> request.abort())
+          )
+        else
+          callback(statusToError(res.statusCode))
+        res.resume()
+      ).on('error', (err) ->
+        request.abort()
+        callback(statusToError(0))
+      ).on('socket', (socket) ->
+        socket.setTimeout(timeout)
+        socket.on('timeout', () -> request.abort())
+      )
+    else
+      callback(statusToError(res.statusCode))
+
+    # No listener for data so emit resume
+    res.resume()
+  ).on('error', (err) ->
+    request.abort()
+    callback(statusToError(0))
+  ).on('socket', (socket) ->
+    socket.setTimeout(timeout)
+    socket.on('timeout', () -> request.abort())
   )
 
 
-watchTask = (taskId, timeout, callback) ->
-  setTimeout( () ->
-    db.tasksCollection.findOne({_id : taskId}, {'_id' : 1}, (err, task) ->
-      if task
-        timeout -= 500
-        if timeout < 0
-          callback('timeout')
-        else
-          watchTask(taskId, timeout, callback)
-      else
+connectionRequest = (deviceId, callback) ->
+  CONNECTION_REQUEST_TIMEOUT = config.get('CONNECTION_REQUEST_TIMEOUT', deviceId)
+  proj = {
+    'Device.ManagementServer.ConnectionRequestURL._value' : 1,
+    'Device.ManagementServer.UDPConnectionRequestAddress._value' : 1,
+    'Device.ManagementServer.ConnectionRequestUsername._value' : 1,
+    'Device.ManagementServer.ConnectionRequestPassword._value' : 1,
+    'InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value' : 1,
+    'InternetGatewayDevice.ManagementServer.UDPConnectionRequestAddress._value' : 1,
+    'InternetGatewayDevice.ManagementServer.ConnectionRequestUsername._value' : 1,
+    'InternetGatewayDevice.ManagementServer.ConnectionRequestPassword._value' : 1
+  }
+
+  db.devicesCollection.findOne({_id : deviceId}, proj, (err, device)->
+    return callback(err) if err
+
+    if device.Device? # TR-181 data model
+      connectionRequestUrl = device.Device.ManagementServer.ConnectionRequestURL._value
+      udpConnectionRequestAddress = device.Device.ManagementServer?.UDPConnectionRequestAddress?._value
+      username = device.Device.ManagementServer.ConnectionRequestUsername?._value
+      password = device.Device.ManagementServer.ConnectionRequestPassword?._value
+    else # TR-098 data model
+      connectionRequestUrl = device.InternetGatewayDevice.ManagementServer.ConnectionRequestURL._value
+      udpConnectionRequestAddress = device.InternetGatewayDevice.ManagementServer?.UDPConnectionRequestAddress?._value
+      username = device.InternetGatewayDevice.ManagementServer.ConnectionRequestUsername?._value
+      password = device.InternetGatewayDevice.ManagementServer.ConnectionRequestPassword?._value
+
+    conReq = () ->
+      udpSent = udpConReq(udpConnectionRequestAddress, username, password, (err) -> throw err if err)
+
+      httpConReq(connectionRequestUrl, username, password, CONNECTION_REQUEST_TIMEOUT, (err) ->
+        if udpSent
+          return callback()
         callback(err)
+      )
+
+    if config.auth?.connectionRequest?
+      # Callback is optional for backward compatibility
+      if config.auth.connectionRequest.length > 4
+        return config.auth.connectionRequest(deviceId, connectionRequestUrl, username, password, (u, p) ->
+          username = u
+          password = p
+          conReq()
+        )
+      [username, password] = config.auth.connectionRequest(deviceId, connectionRequestUrl, username, password)
+    conReq()
+  )
+
+
+watchTask = (deviceId, taskId, timeout, callback) ->
+  setTimeout(() ->
+    db.tasksCollection.findOne({_id : taskId}, {'_id' : 1}, (err, task) ->
+      return callback(err) if err
+
+      if not task
+        return callback(null, 'completed')
+
+      db.faultsCollection.findOne({_id : "#{deviceId}:task_#{taskId}"}, {'_id' : 1}, (err, fault) ->
+        return callback(err) if err
+
+        if fault
+          return callback(null, 'fault')
+
+        if (timeout -= 500) <= 0
+          return callback(null, 'timeout')
+
+        watchTask(deviceId, taskId, timeout, callback)
+      )
     )
   , 500)
 
 
-expandParam = (param, aliases) ->
-  params = [param]
-  for a,aa of aliases
-    if a == param or common.startsWith(a, "#{param}.")
-      for p in aa
-        params.push(p) if p[p.lastIndexOf('.') + 1] != '_'
-
-  return params
-
-
-sanitizeTask = (task, aliases, callback) ->
+sanitizeTask = (task, callback) ->
   task.timestamp = new Date(task.timestamp ? Date.now())
   if task.expiry?
     if common.typeOf(task.expiry) is common.DATE_TYPE or isNaN(task.expiry)
@@ -97,56 +201,10 @@ sanitizeTask = (task, aliases, callback) ->
     else
       task.expiry = new Date(task.timestamp.getTime() + +task.expiry * 1000)
 
-  switch task.name
-    when 'getParameterValues'
-      projection = {}
-      for p in task.parameterNames
-        for pp in expandParam(p, aliases)
-          projection[pp] = 1
-      db.devicesCollection.findOne({_id : task.device}, projection, (err, device) ->
-        parameterNames = []
-        for k of projection
-          if common.getParamValueFromPath(device, k)?
-            parameterNames.push(k)
-        task.parameterNames = parameterNames
-        callback(task)
-      )
-    when 'setParameterValues'
-      projection = {}
-      values = {}
-      for p in task.parameterValues
-        for pp in expandParam(p[0], aliases)
-          projection[pp] = 1
-          values[pp] = p[1]
-      db.devicesCollection.findOne({_id : task.device}, projection, (err, device) ->
-        parameterValues = []
-        for k of projection
-          param = common.getParamValueFromPath(device, k)
-          if param?
-            parameterValues.push([k, values[k], param._type])
-        task.parameterValues = parameterValues
-        callback(task)
-      )
-    else
-      # TODO implement setParameterValues
-      callback(task)
+  callback(task)
 
 
-addAliases = (device, aliases) ->
-  for k,v of aliases
-    for p in v
-      pp = p.split('.')
-      obj = device
-      for i in pp
-        if not obj[i]?
-          obj = null
-          break
-        obj = obj[i]
-
-      device[k] = obj if obj?
-
-
-insertTasks = (tasks, aliases, callback) ->
+insertTasks = (tasks, callback) ->
   if tasks? and common.typeOf(tasks) isnt common.ARRAY_TYPE
     tasks = [tasks]
   else if not tasks? or tasks.length == 0
@@ -155,9 +213,9 @@ insertTasks = (tasks, aliases, callback) ->
   counter = tasks.length
 
   for task in tasks
-    sanitizeTask(task, aliases, (t) ->
+    sanitizeTask(task, (t) ->
       if t.uniqueKey?
-        db.tasksCollection.remove({device : t.device, uniqueKey : t.uniqueKey}, (err, removed) ->
+        db.tasksCollection.remove({device : t.device, uniqueKey : t.uniqueKey}, (err) ->
         )
 
       --counter
@@ -169,8 +227,28 @@ insertTasks = (tasks, aliases, callback) ->
     )
 
 
-exports.addAliases = addAliases
-exports.sanitizeTask = sanitizeTask
+deleteDevice = (deviceId, callback) ->
+  db.tasksCollection.remove({'device' : deviceId}, (err) ->
+    return callback(err) if err
+    db.devicesCollection.remove({'_id' : deviceId}, (err) ->
+      return callback(err) if err
+      db.faultsCollection.remove({'_id' : {'$regex' : "^#{common.escapeRegExp(deviceId)}\\:"}}, (err) ->
+        return callback(err) if err
+        db.operationsCollection.remove({'_id' : {'$regex' : "^#{common.escapeRegExp(deviceId)}\\:"}}, (err) ->
+          return callback(err) if err
+          db.redisClient.del("#{deviceId}_presets_hash",
+            "#{deviceId}_inform_hash",
+            "#{deviceId}_faults",
+            "#{deviceId}_tasks",
+            "#{deviceId}_operations",
+            callback)
+        )
+      )
+    )
+  )
+
+
 exports.connectionRequest = connectionRequest
 exports.watchTask = watchTask
 exports.insertTasks = insertTasks
+exports.deleteDevice = deleteDevice

lib/auth.coffee

@@ -1,19 +1,30 @@
+###
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
+#
+# GenieACS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# GenieACS is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+###
+
 crypto = require 'crypto'
 
 
 exports.parseAuthHeader = (authHeader) ->
-  res = {}
-  i = authHeader.indexOf(' ')
-  res['method'] = authHeader.slice(0, i)
-
-  options = authHeader.slice(i + 1)
-  regex = /([a-z]+)="([a-zA-Z0-9\/\.@\-,= ]+)"/g
-  while (r = regex.exec(options)) != null
-    res[r[1]] = r[2]
-
-  regex = /([a-z]+)=([a-zA-Z0-9\/\.@\-]+)/g
-  while (r = regex.exec(options)) != null
-    res[r[1]] = r[2]
+  res = {method: authHeader.slice(0, authHeader.indexOf(' '))}
+  re = /([a-z0-9_-]+)=(?:"([^"]+)"|([a-z0-9_-]+))/gi
+  while match = re.exec(authHeader)
+    res[match[1]] = match[2] or match[3]
   return res
 
 
@@ -55,9 +66,7 @@ exports.digest = (username, password, uri, httpMethod, body, authHeader) ->
   authString += ",nonce=\"#{authHeader.nonce}\""
   authString += ",uri=\"#{uri}\""
   authString += ",algorithm=#{authHeader.algorithm}" if authHeader.algorithm?
-  authString += ",qop=#{qop}"
-  authString += ",nc=#{nc}"
-  authString += ",cnonce=\"#{cnonce}\""
+  authString += ",qop=#{qop},nc=#{nc},cnonce=\"#{cnonce}\"" if qop?
   authString += ",response=\"#{response}\""
-  authString += ",opaque=\"#{authHeader.opaque}\""
+  authString += ",opaque=\"#{authHeader.opaque}\"" if authHeader.opaque?
   return authString

lib/cache.coffee

@@ -0,0 +1,310 @@
+###
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
+#
+# GenieACS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# GenieACS is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+###
+
+vm = require 'vm'
+crypto = require 'crypto'
+
+later = require 'later'
+
+db = require './db'
+query = require './query'
+
+
+REFRESH = 3000
+
+nextRefresh = 0
+hash = null
+presets = null
+provisions = null
+virtualParameters = null
+files = null
+
+
+UNLOCK_SCRIPT = 'if redis.call("get", KEYS[1]) == ARGV[1] then return redis.call("del", KEYS[1]) else return 0 end'
+EXTEND_SCRIPT = 'if redis.call("get", KEYS[1]) == ARGV[1] then return redis.call("pexpire", KEYS[1], ARGV[2]) else return 0 end'
+
+lock = (lockName, ttl, callback) ->
+  token = Math.random().toString(36).slice(2)
+  unlockOrExtend = (ttl) ->
+    if not ttl
+      db.redisClient.eval(UNLOCK_SCRIPT, 1, lockName, token, (err, res) ->
+        throw err or new Error('Lock expired') if err or not res
+      )
+    else
+      db.redisClient.eval(EXTEND_SCRIPT, 1, lockName, token, ttl, (err, res) ->
+        throw err or new Error('Lock expired') if err or not res
+      )
+    return
+
+  db.redisClient.set(lockName, token, 'NX', 'PX', ttl, (err, res) ->
+    if err or res
+      return callback(err, unlockOrExtend)
+
+    setTimeout(() ->
+      return lock(lockName, ttl, callback)
+    , 200)
+  )
+
+
+computeHash = () ->
+  # MD5 hash for presets, provisions, virtual parameters for detecting changes
+  h = crypto.createHash('md5')
+  for p in presets
+    h.update(JSON.stringify(p.name))
+    h.update(JSON.stringify(p.channel))
+    h.update(JSON.stringify(p.schedule))
+    h.update(JSON.stringify(p.events))
+    h.update(JSON.stringify(p.precondition))
+    h.update(JSON.stringify(p.provisions))
+
+  keys = Object.keys(provisions).sort()
+  h.update(JSON.stringify(keys))
+  h.update(provisions[k].md5) for k in keys
+
+  keys = Object.keys(virtualParameters).sort()
+  h.update(JSON.stringify(keys))
+  h.update(virtualParameters[k].md5) for k in keys
+
+  hash = h.digest('hex')
+
+
+refresh = (callback) ->
+  now = Date.now()
+  db.redisClient.get('presets_hash', (err, res) ->
+    return callback(err) if err
+
+    if hash? and res == hash
+      nextRefresh = now + REFRESH
+      return callback()
+
+    lock('presets_hash_lock', 3000, (err, unlockOrExtend) ->
+      return callback(err) if err
+      counter = 3
+
+      counter += 2
+      db.presetsCollection.find().toArray((err, res) ->
+        if err
+          callback(err) if counter & 1
+          return counter = 0
+
+        db.objectsCollection.find().toArray((err, objects) ->
+          if err
+            callback(err) if (counter & 1)
+            return counter = 0
+
+          res.sort((a, b) ->
+            if a.weight == b.weight
+              return a._id > b._id
+            else
+              return a.weight - b.weight
+          )
+
+          presets = []
+
+          for preset in res
+            schedule = null
+            if preset.schedule
+              parts = preset.schedule.trim().split(/\s+/)
+              schedule = {
+                md5: crypto.createHash('md5').update(preset.schedule).digest('hex')
+              }
+
+              try
+                schedule.duration = +(parts.shift()) * 1000
+                parts.unshift('0') if parts.length == 5
+                # TODO later.js doesn't throw erorr if expression is invalid!
+                schedule.schedule = later.schedule(later.parse.cron(parts.join(' '), true))
+              catch err
+                # TODO show a warning
+                schedule.schedule = false
+
+            events = preset.events ? {}
+
+            precondition = query.convertMongoQueryToFilters(JSON.parse(preset.precondition))
+
+            _provisions = preset.provisions or []
+
+            # Generate provisions from the old configuration format
+            for c in preset.configurations
+              switch c.type
+                when 'age'
+                  _provisions.push(['refresh', c.name, c.age])
+                when 'value'
+                  _provisions.push(['value', c.name, c.value])
+                when 'add_tag'
+                  _provisions.push(['tag', c.tag, true])
+                when 'delete_tag'
+                  _provisions.push(['tag', c.tag, false])
+                when 'provision'
+                  _provisions.push([c.name].concat(c.args or []))
+                when 'add_object'
+                  for obj in objects
+                    if obj['_id'] == c.object
+                      alias = ("#{k}:#{JSON.stringify(obj[k])}" for k in obj['_keys']).join(',')
+                      p = "#{c.name}.[#{alias}]"
+                      _provisions.push(['instances', p, 1])
+                      for k of obj
+                        if k[0] != '_' and k not in obj['_keys']
+                          _provisions.push(['value', "#{p}.#{k}", obj[k]])
+                when 'delete_object'
+                  for obj in objects
+                    if obj['_id'] == c.object
+                      alias = ("#{k}:#{JSON.stringify(obj[k])}" for k in obj['_keys']).join(',')
+                      p = "#{c.name}.[#{alias}]"
+                      _provisions.push(['instances', p, 0])
+                else
+                  callback(new Error("Unknown configuration type #{c.type}")) if counter & 1
+                  return counter = 0
+
+            presets.push({name: preset._id, channel: preset.channel or 'default', schedule: schedule, events: events, precondition: precondition, provisions: _provisions})
+
+          if (counter -= 2) == 1
+            computeHash()
+            db.redisClient.setex("presets_hash", 300, hash, (err) ->
+              unlockOrExtend(0)
+              nextRefresh = now + REFRESH
+              return callback()
+            )
+        )
+      )
+
+      counter += 2
+      db.provisionsCollection.find().toArray((err, res) ->
+        if err
+          callback(err) if counter & 1
+          return counter = 0
+
+        provisions = {}
+        for r in res
+          provisions[r._id] = {}
+          provisions[r._id].md5 = crypto.createHash('md5').update(r.script).digest('hex')
+          provisions[r._id].script = new vm.Script("\"use strict\";(function(){\n#{r.script}\n})();", {filename: r._id, lineOffset: -1, timeout: 50})
+
+        if (counter -= 2) == 1
+          computeHash()
+          db.redisClient.setex("presets_hash", 300, hash, (err) ->
+            unlockOrExtend(0)
+            nextRefresh = now + REFRESH
+            return callback()
+          )
+      )
+
+      counter += 2
+      db.virtualParametersCollection.find().toArray((err, res) ->
+        if err
+          callback(err) if counter & 1
+          return counter = 0
+
+        virtualParameters = {}
+        for r in res
+          virtualParameters[r._id] = {}
+          virtualParameters[r._id].md5 = crypto.createHash('md5').update(r.script).digest('hex')
+          virtualParameters[r._id].script = new vm.Script("\"use strict\";(function(){\n#{r.script}\n})();", {filename: r._id, lineOffset: -1, timeout: 50})
+
+        if (counter -= 2) == 1
+          computeHash()
+          db.redisClient.setex("presets_hash", 300, hash, (err) ->
+            unlockOrExtend(0)
+            nextRefresh = now + REFRESH
+            return callback()
+          )
+      )
+
+      counter += 2
+      db.filesCollection.find().toArray((err, res) ->
+        if err
+          callback(err) if counter & 1
+          return counter = 0
+
+        files = {}
+        for r in res
+          id = r.filename or r._id.toString()
+          files[id] = {}
+          files[id].length = r.length
+          files[id].md5 = r.md5
+          files[id].contentType = r.contentType
+
+        if (counter -= 2) == 1
+          computeHash()
+          db.redisClient.setex("presets_hash", 300, hash, (err) ->
+            unlockOrExtend(0)
+            nextRefresh = now + REFRESH
+            return callback()
+          )
+      )
+
+      if (counter -= 2) == 1
+        computeHash()
+        db.redisClient.setex("presets_hash", 300, hash, (err) ->
+          unlockOrExtend(0)
+          nextRefresh = now + REFRESH
+          return callback()
+        )
+    )
+  )
+
+
+getPresets = (callback) ->
+  if Date.now() < nextRefresh
+    return callback(null, hash, presets)
+
+  refresh((err) ->
+    return callback(err, hash, presets)
+  )
+
+
+getProvisions = (callback) ->
+  if Date.now() < nextRefresh
+    return callback(null, hash, provisions)
+
+  refresh((err) ->
+    return callback(err, hash, provisions)
+  )
+
+
+getVirtualParameters = (callback) ->
+  if Date.now() < nextRefresh
+    return callback(null, hash, virtualParameters)
+
+  refresh((err) ->
+    return callback(err, hash, virtualParameters)
+  )
+
+
+getFiles = (callback) ->
+  if Date.now() < nextRefresh
+    return callback(null, hash, files)
+
+  refresh((err) ->
+    return callback(err, hash, files)
+  )
+
+
+getProvisionsAndVirtualParameters = (callback) ->
+  if Date.now() < nextRefresh
+    return callback(null, hash, provisions, virtualParameters)
+
+  refresh((err) ->
+    return callback(err, hash, provisions, virtualParameters)
+  )
+
+exports.getPresets = getPresets
+exports.getProvisions = getProvisions
+exports.getFiles = getFiles
+exports.getProvisionsAndVirtualParameters = getProvisionsAndVirtualParameters

lib/cluster.js

@@ -0,0 +1,139 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const cluster = require("cluster");
+const logger = require("./logger");
+
+const VERSION = require("../package.json").version;
+
+
+function getDependencyVersions() {
+  let dependencies = [`node@${process.versions.node}`];
+  try {
+    let npmls = JSON.parse(require("child_process").execSync("npm ls --prod --json"));
+
+    for (let k in npmls.dependencies)
+      dependencies.push(`${k}@${npmls.dependencies[k].version}`);
+  } catch (err) {}
+
+  return dependencies.join(",");
+}
+
+
+function getConfig() {
+  let conf = [];
+  const keys = Object.keys(process.env);
+  for (let k of keys)
+    if (k.startsWith("GENIEACS_"))
+      conf.push(`${k.slice(9)}=${process.env[k]}`);
+
+  return conf.join(",");
+}
+
+
+function restartWorker(worker, code, signal) {
+  let msg = {
+    message: "Worker died",
+    pid: worker.process.pid
+  };
+
+  if (code != null)
+    msg.exitCode = code;
+
+  if (signal != null)
+    msg.signal = signal;
+
+  logger.error(msg);
+  cluster.fork();
+}
+
+
+function start(service) {
+  logger.init(service, VERSION);
+  const config = require("./config");
+
+  logger.info({
+    message: `GenieACS (genieacs-${service}) starting`,
+    pid: process.pid,
+    version: VERSION,
+    dependencies: getDependencyVersions(),
+    config: getConfig()
+  });
+
+  cluster.on("listening", function(worker, address) {
+    logger.info({
+      message: "Worker listening",
+      pid: worker.process.pid,
+      address: address.address,
+      port: address.port
+    });
+  });
+
+  cluster.on("exit", restartWorker);
+
+  process.on("SIGINT", function() {
+    logger.info({
+      message: "Received signal SIGINT, exiting",
+      pid: process.pid
+    });
+
+    cluster.removeListener("exit", restartWorker);
+  });
+
+  process.on("SIGTERM", function() {
+    logger.info({
+      message: "Received signal SIGTERM, exiting",
+      pid: process.pid
+    });
+
+    cluster.removeListener("exit", restartWorker);
+    for (let pid in cluster.workers)
+      cluster.workers[pid].kill();
+  });
+
+  cluster.setupMaster({
+    exec: "lib/server",
+    args: [service]
+  });
+
+  let workerCount = config.get(`${service.toUpperCase()}_WORKER_PROCESSES`);
+
+  if (!workerCount)
+    workerCount = Math.max(2, require("os").cpus().length);
+
+  for (let i = 0; i < workerCount; ++ i)
+    cluster.fork();
+}
+
+
+process.on("uncaughtException", function(err) {
+  logger.error({
+    message: "Uncaught exception in master process, exiting",
+    exception: err
+  });
+  cluster.removeListener("exit", restartWorker);
+  for (let pid in cluster.workers)
+    cluster.workers[pid].kill();
+
+  logger.close();
+});
+
+
+exports.start = start;

lib/common.coffee

@@ -1,3 +1,25 @@
+###
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
+#
+# GenieACS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# GenieACS is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+###
+
+buffer = require 'buffer'
+querystring = require 'querystring'
+
 UNDEFINED_TYPE = '[object Undefined]'
 NULL_TYPE = '[object Null]'
 BOOLEAN_TYPE = '[object Boolean]'
@@ -13,26 +35,11 @@ typeOf = (obj) ->
   Object.prototype.toString.call(obj)
 
 
-exports.endsWith = (str, suffix) ->
-  str.indexOf(suffix, str.length - suffix.length) isnt -1
-
-
-exports.startsWith = (str, prefix) ->
-  str.substring(0, prefix.length) == prefix
-
-
-exports.arrayToHash = (arr) ->
-  hash = {}
-  for i in arr
-    hash[i[0]] = i[1]
-  return hash
-
-
-exports.getDeviceId = (deviceIdStruct) ->
+exports.generateDeviceId = (deviceIdStruct) ->
   # Percent escaping function. Escapes everything except alphanumerics and underscore
   esc = (str) ->
     str.replace(/[^A-Za-z0-9_]/g, (chr) ->
-      buf = new require('buffer').Buffer(chr)
+      buf = new buffer.Buffer(chr, 'utf8')
       rep = ''
       rep += "%#{b.toString(16).toUpperCase()}" for b in buf
       return rep
@@ -45,11 +52,6 @@ exports.getDeviceId = (deviceIdStruct) ->
   return "#{esc(deviceIdStruct['OUI'])}-#{esc(deviceIdStruct['SerialNumber'])}"
 
 
-exports.extend = (obj, mixin) ->
-  obj[name] = method for name, method of mixin        
-  obj
-
-
 exports.getParamValueFromPath = (obj, path) ->
   pp = path.split('.')
   ref = obj
@@ -61,17 +63,173 @@ exports.getParamValueFromPath = (obj, path) ->
     return undefined
 
 
-exports.matchType = (src, dst) ->
-  switch typeOf(src)
-    when STRING_TYPE
-      String(dst)
-    when NUMBER_TYPE
-      Number(dst)
-    when BOOLEAN_TYPE
-      v = String(dst).trim().toLowerCase()
-      v == 'true' or v == 'on' or v == 'yes' or v == '1'
-    else
-      dst
+pathOverlap = (a, b, start) ->
+  if a.length == b.length
+    res = 3
+  else if a.length > b.length
+    res = 2
+  else
+    res = 1
+
+  for i in [(start ? 0)...Math.min(a.length, b.length)] by 1
+    if a[i]?
+      if not b[i]?
+        res &= 2
+      else if a[i] != b[i]
+        return 0
+    else if b[i]?
+      res &= 1
+
+    return 0 if not res
+
+  return res
+
+
+parseAlias = (pattern, start, res) ->
+  aliases = []
+  i = start
+  while i < pattern.length and pattern[i] != ']'
+    alias = []
+    i = j = parsePath(pattern, i, alias) + 1
+
+    while pattern[j] not in [']', ',']
+      if pattern[j] == '"' and i == j
+        ++ j
+        while pattern[j] != '"' or pattern[j - 1] == '\\'
+          if ++ j >= pattern.length
+            throw new Error('Invalid alias expression')
+      if ++ j >= pattern.length
+        throw new Error('Invalid alias expression')
+    value = pattern[i...j].trim()
+    i = j
+    if value[0] == '"'
+      try
+        value = JSON.parse(value)
+      catch err
+        throw new Error('Invalid alias expression')
+    alias.push(value)
+    aliases.push(alias)
+    ++ i if pattern[i] == ','
+
+  # Need to sort to ensure identical expressions have idential string representation
+  srt = (a, b) ->
+    for j in [0...Math.min(a.length, b.length)] by 2
+      for k in [0...Math.min(a[j].length, b[j].length)] by 1
+        if Array.isArray(a[j][k])
+          if Array.isArray(b[j][k])
+            return srt(a[j][k], b[j][k])
+          else if not b[j][k]?
+            return -1
+          else
+            return 1
+        else if not a[j][k]?
+          if not b[j][k]?
+            return 0
+          else
+            return 1
+        else if not b[j][k]? or Array.isArray(b[j][k])
+            return -1
+        else if a[j][k] > b[j][k]
+          return 1
+        else if a[j][k] < b[j][k]
+          return -1
+
+      if a[j].length > b[j].length
+        return -1
+      else if a[j].length < b[j].length
+        return 1
+
+      if a[j + 1] > b[j + 1]
+        return -1
+      else if a[j + 1] < b[j + 1]
+        return 1
+
+    if a.length > b.length
+      return -1
+    else if a.length < b.length
+      return 1
+
+    return 0
+
+  aliases.sort(srt)
+  res.push([].concat.apply([], aliases))
+  return i
+
+
+parsePath = (pattern, start, res) ->
+  path = []
+  path.wildcard = 0
+  path.alias = 0
+  i = start ? 0
+
+  # Colon separator is needed for parseAlias
+  if i < pattern.length and pattern[i] != ':'
+    while true
+      if pattern[i] == '['
+        path.alias |= 1 << path.length
+        i = parseAlias(pattern, i + 1, path) + 1
+      else
+        j = i
+        while i < pattern.length and pattern[i] != ':' and pattern[i] != '.'
+          ++ i
+        n = pattern.slice(j, i)
+        path.wildcard |= 1 << path.length if n == '*'
+        path.push(n)
+
+      if i >= pattern.length or pattern[i] == ':'
+        break
+      else if pattern[i] != '.'
+        throw new Error('Invalid alias expression')
+
+      ++ i
+
+  Object.freeze(path)
+
+  if not res?
+    return path
+
+  res.push(path)
+  return i
+
+
+addPathMeta = (path) ->
+  return path if path.alias? or path.wildcard?
+
+  path.alias = 0
+  path.wildcard = 0
+
+  for p, i in path
+    if typeOf(p) is ARRAY_TYPE
+      path.alias |= 1 << i
+      for j in [0...p.length] by 2
+        addPathMeta(p[j])
+    else if p == '*'
+      path.wildcard |= 1 << i
+
+  Object.freeze(path)
+  return path
+
+
+hammingWeight = (flags) ->
+  flags -= ((flags >> 1) & 0x55555555)
+  flags = (((flags >> 2) & 0x33333333) + (flags & 0x33333333))
+  flags = (((flags >> 4) + flags) & 0x0f0f0f0f)
+  flags += (flags >> 8)
+  flags += (flags >> 16)
+  return flags & 0x0000003f
+
+
+# Source: http://stackoverflow.com/a/6969486
+escapeRegExp = (str) ->
+  return str.replace(/[\-\[\]\/\{\}\(\)\*\+\?\.\\\^\$\|]/g, "\\$&")
+
+
+# djb2 hashing algorithm adapted from http://www.cse.yorku.ca/~oz/hash.html
+djb2 = (str) ->
+  hash = 5381
+  for i in [0...str.length]
+    hash = ((hash << 5) + hash) ^ str.charCodeAt(i)
+  return hash
 
 
 exports.UNDEFINED_TYPE = UNDEFINED_TYPE
@@ -83,3 +241,9 @@ exports.OBJECT_TYPE = OBJECT_TYPE
 exports.REGEXP_TYPE = REGEXP_TYPE
 
 exports.typeOf = typeOf
+exports.pathOverlap = pathOverlap
+exports.parsePath = parsePath
+exports.addPathMeta = addPathMeta
+exports.hammingWeight = hammingWeight
+exports.escapeRegExp = escapeRegExp
+exports.djb2 = djb2

lib/config.coffee

@@ -1,45 +1,192 @@
-exports.DATABASE_NAME = 'genieacs'
-exports.ACS_PORT = 7547 # CWMP port as assigned by IANA
-exports.ACS_HTTPS_PORT = 7548
-exports.ACS_INTERFACE = '0.0.0.0'
-exports.ACS_HTTPS_INTERFACE = '0.0.0.0'
-exports.API_PORT = 7557
-exports.API_INTERFACE = '0.0.0.0'
-exports.FILES_PORT = 7567
-exports.FILES_INTERFACE = '0.0.0.0'
-exports.FILES_IP = '172.240.21.2' # Used when sending download requests to devices
-exports.CACHE_DURATION = 60 # in seconds
-exports.PRESETS_CACHE_DURATION = 86400
-exports.PRESETS_TIME_PADDING = 1
-exports.MONGODB_SOCKET = '/tmp/mongodb-27017.sock'
-exports.REDIS_SOCKET = '/tmp/redis.sock'
-exports.WORKER_RESPAWN_TIME = 60000
-exports.LOG_INFORMS = false
-exports.DEBUG_DEVICES = {} # {'202BC1-BM632w-8KA8WA1151100043' : true}
-exports.DEVICE_ONLINE_THRESHOLD = 4000
-exports.RETRY_DELAY = 300
-exports.IGNORE_XML_NAMESPACES = false # Traverse XML using element's local name only
+###
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
+#
+# GenieACS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# GenieACS is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+###
 
-exports.MONGODB_OPTIONS = {
-  db : {
-    w : 1
-    wtimeout : 60000
-  }
-  server : {
-    auto_reconnect : true
-  }
+path = require 'path'
+common = require './common'
+
+options = {
+  CONFIG_DIR : {type : 'path', default : 'config'},
+  MONGODB_CONNECTION_URL : {type : 'string', default : 'mongodb://127.0.0.1/genieacs'},
+  REDIS_PORT : {type : 'int', default : 6379},
+  REDIS_HOST : {type : 'string', default : '127.0.0.1'},
+  REDIS_DB : {type : 'int', default : 0},
+
+  CWMP_WORKER_PROCESSES : {type : 'int', default : 0},
+  CWMP_PORT : {type : 'int', default : 7547},
+  CWMP_INTERFACE : {type : 'string', default : '0.0.0.0'},
+  CWMP_SSL : {type : 'bool', default : false},
+  CWMP_LOG_FILE : {type: 'path', default : ''},
+  CWMP_ACCESS_LOG_FILE : {type : 'path', default : ''},
+
+  NBI_WORKER_PROCESSES : {type : 'int', default : 0},
+  NBI_PORT : {type : 'int', default : 7557},
+  NBI_INTERFACE : {type : 'string', default : '0.0.0.0'},
+  NBI_SSL : {type : 'bool', default : false},
+  NBI_LOG_FILE : {type: 'path', default : ''},
+  NBI_ACCESS_LOG_FILE : {type : 'path', default : ''},
+
+  FS_WORKER_PROCESSES : {type : 'int', default : 0},
+  FS_PORT : {type : 'int', default : 7567},
+  FS_INTERFACE : {type : 'string', default : '0.0.0.0'},
+  FS_SSL : {type : 'bool', default : false},
+  FS_HOSTNAME : {type : 'string', default : 'acs.example.com'},
+  FS_LOG_FILE : {type: 'path', default : ''},
+  FS_ACCESS_LOG_FILE : {type : 'path', default : ''},
+
+  DOWNLOAD_TIMEOUT: {type : 'int', default : 3600},
+  EXT_TIMEOUT: {type: 'int', default: 3000},
+  MAX_CACHE_TTL : {type : 'int', default : 86400},
+  DEBUG : {type : 'bool', default : false},
+  RETRY_DELAY : {type : 'int', default : 300},
+  SESSION_TIMEOUT : {type : 'int', default : 30},
+  CONNECTION_REQUEST_TIMEOUT : {type : 'int', default: 2000},
+  GPN_NEXT_LEVEL : {type : 'int', default : 0},
+  GPV_BATCH_SIZE : {type : 'int', default : 32},
+  MAX_DEPTH : {type: 'int', default : 16},
+  COOKIES_PATH : {type : 'string'},
+  LOG_FORMAT : {type : 'string', default : 'simple'},
+  ACCESS_LOG_FORMAT : {type : 'string', default : ''},
+
+  # XML configuration
+  XML_RECOVER : {type : 'bool', default : false},
+  XML_IGNORE_ENC : {type : 'bool', default : false},
+  XML_FORMAT : {type : 'bool', default : false},
+  XML_NO_DECL : {type : 'bool', default : false},
+  XML_NO_EMPTY : {type : 'bool', default : false},
+  XML_IGNORE_NAMESPACE : {type : 'bool', default : false},
+
+  # Should probably never be changed
+  DEVICE_ONLINE_THRESHOLD : {type : 'int', default : 4000}
 }
 
-# load configuration
-c = require('../config/config')
-for k, v of c
-  exports[k] = v
+allConfig = {}
+
+
+setConfig = (name, value, commandLineArgument) ->
+  return true if allConfig[name]?
+
+  # For compatibility with v1.0
+  if name in ['PRESETS_CACHE_DURATION', 'presets-cache-duration']
+    setConfig('MAX_CACHE_TTL', value)
+
+  if name in ['GET_PARAMETER_NAMES_DEPTH_THRESHOLD', 'get-parameter-names-depth-threshold']
+    setConfig('GPN_NEXT_LEVEL', value)
+
+  if name in ['TASK_PARAMETERS_BATCH_SIZE', 'task-parameters-batch-size']
+    setConfig('GPV_BATCH_SIZE', value)
+
+  if name in ['IGNORE_XML_NAMESPACES', 'ignore-xml-namespaces']
+    setConfig('XML_IGNORE_NAMESPACE', value)
+
+  if name in ['XML_PARSE_IGNORE_ENC', 'xml-parse-ignore-enc']
+    setConfig('XML_IGNORE_ENC', value)
+
+  if name in ['XML_PARSE_RECOVER', 'xml-parse-recover']
+    setConfig('XML_RECOVER', value)
+
+  if name in ['FS_IP', 'fs-ip']
+    setConfig('FS_HOSTNAME', value)
+
+  cast = (val, type) ->
+    switch type
+      when 'int'
+        Number(val)
+      when 'bool'
+        String(val).trim().toLowerCase() in ['true', 'on', 'yes', '1']
+      when 'string'
+        String(val)
+      when 'path'
+        if val then path.resolve(val) else ''
+      else
+        null
+
+  _value = null
+
+  for optionName, optionDetails of options
+    n = optionName
+    if commandLineArgument
+      n = n.toLowerCase().replace(/_/g, '-')
+
+    if name == n
+      _value = cast(value, optionDetails.type)
+      n = optionName
+    else if name.startsWith("#{n}-")
+      _value = cast(value, optionDetails.type)
+      n = "#{optionName}-#{name[optionName.length+1..]}"
+
+    if _value?
+      allConfig[n] = _value
+      # Save as environmnet variable to pass on to any child processes
+      process.env["GENIEACS_#{n}"] = _value
+      return true
+
+  return false
+
+
+# Command line arguments
+argv = process.argv[2..]
+while argv.length
+  arg = argv.shift()
+  if arg[0] == '-'
+    v = argv.shift()
+    setConfig(arg[2..], v, true)
+
+
+# Environment variable
+for k, v of process.env
+  continue if k.lastIndexOf('GENIEACS_', 0) != 0
+  k = k[9..] # remove "GENIEACS_" prefix
+  setConfig(k, v)
+
+
+# Use default config dir if none defined
+setConfig('CONFIG_DIR', options['CONFIG_DIR'].default)
+
+# Configuration file
+for k, v of require(path.resolve(allConfig.CONFIG_DIR, 'config'))
+  if not setConfig(k, v)
+    # Pass as environment variable to be accessable by extensions
+    process.env["GENIEACS_#{k}"] = v
+
+
+# Defaults
+for k, v of options
+  setConfig(k, v.default) if v.default?
+
+
+get = (option, deviceId) ->
+  if deviceId?
+    name = "#{option}-#{deviceId}"
+    v = allConfig[name]
+    return v if v?
+    i = name.lastIndexOf('-')
+    v = allConfig[name[0...i]]
+    return v if v?
+    i = name.lastIndexOf('-', i-1)
+    v = allConfig[name[0...i]]
+    return v if v? or i == -1
+
+  return allConfig[option]
 
-# load parameter configurations
-exports.PARAMETERS = require('../config/parameters')
 
 # load authentication scripts
-exports.auth = require('../config/auth')
-
-exports.CUSTOM_COMMANDS = require('../config/custom_commands')
+try
+  exports.auth = require(path.resolve(allConfig.CONFIG_DIR, 'auth'))
 
+exports.get = get

lib/custom-commands.coffee

@@ -1,35 +0,0 @@
-config = require './config'
-common = require './common'
-
-
-exports.execute = (deviceId, command, callback) ->
-  args = command.split(/\s+/)
-  f = args.shift()
-  c = args.shift()
-
-  file = require "../config/custom_commands/#{f}"
-  file[c](deviceId, args.join(' '), callback)
-
-
-exports.getFileCommands = (filename) ->
-  commands = []
-  f = require("../config/custom_commands/#{filename}")
-  for k,v of f
-    commands.push(k)
-  return commands
-
-
-exports.getDeviceCustomCommands = (deviceId) ->
-  commands = {}
-  for k,v of config.CUSTOM_COMMANDS
-    if eval(v).test(deviceId)
-      commands[k] = exports.getFileCommands(k)
-  return commands
-
-
-exports.getDeviceCustomCommandNames = (deviceId) ->
-  commands = []
-  for k,v of config.CUSTOM_COMMANDS
-    if eval(v).test(deviceId)
-      commands.push(k)
-  return commands

lib/db.coffee

@@ -1,155 +1,663 @@
+###
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
+#
+# GenieACS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# GenieACS is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+###
+
 config = require './config'
 mongodb = require 'mongodb'
-redisClient = require('redis').createClient(config.REDIS_SOCKET)
-parameters = require './parameters'
+redis = require('redis')
+common = require './common'
 
+redisClient = null
 tasksCollection = null
 devicesCollection = null
 presetsCollection = null
 objectsCollection = null
+provisionsCollection = null
+virtualParametersCollection = null
+faultsCollection = null
+filesCollection = null
+operationsCollection = null
 
-mongodb.MongoClient.connect("mongodb://#{config.MONGODB_SOCKET}/#{config.DATABASE_NAME}", config.MONGODB_OPTIONS, (err, db) ->
-  exports.mongoDb = db
-  db.collection('tasks', (err, collection) ->
-    throw new Error(err) if err?
-    exports.tasksCollection = tasksCollection = collection
-    collection.ensureIndex({device: 1, timestamp: 1}, (err) ->
+
+connect = (callback) ->
+  callbackCounter = 10
+  mongodb.MongoClient.connect(config.get('MONGODB_CONNECTION_URL'), (err, db) ->
+    return callback(err) if err
+    exports.mongoDb = db
+    db.collection('tasks', (err, collection) ->
+      exports.tasksCollection = tasksCollection = collection
+      collection?.ensureIndex({device: 1, timestamp: 1}, (err) ->
+      )
+
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
+    )
+
+    db.collection('devices', (err, collection) ->
+      exports.devicesCollection  = devicesCollection = collection
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
+    )
+
+    db.collection('presets', (err, collection) ->
+      exports.presetsCollection = presetsCollection = collection
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
+    )
+
+    db.collection('objects', (err, collection) ->
+      exports.objectsCollection = objectsCollection = collection
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
+    )
+
+    db.collection('fs.files', (err, collection) ->
+      exports.filesCollection = filesCollection = collection
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
+    )
+
+    db.collection('provisions', (err, collection) ->
+      exports.provisionsCollection = provisionsCollection = collection
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
+    )
+
+    db.collection('virtualParameters', (err, collection) ->
+      exports.virtualParametersCollection = virtualParametersCollection = collection
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
+    )
+
+    db.collection('faults', (err, collection) ->
+      exports.faultsCollection = faultsCollection = collection
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
+    )
+
+    db.collection('operations', (err, collection) ->
+      exports.operationsCollection = operationsCollection = collection
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
+    )
+
+    exports.redisClient = redisClient = redis.createClient(config.get('REDIS_PORT'), config.get('REDIS_HOST'))
+    redisClient.select(config.get('REDIS_DB'), (err) ->
+      if --callbackCounter == 0 or err
+        callbackCounter = 0
+        return callback(err)
     )
   )
 
-  db.collection('devices', (err, collection) ->
-    throw new Error(err) if err?
-    exports.devicesCollection  = devicesCollection = collection
-  )
 
-  db.collection('presets', (err, collection) ->
-    throw new Error(err) if err?
-    exports.presetsCollection = presetsCollection = collection
-  )
-
-  db.collection('objects', (err, collection) ->
-    throw new Error(err) if err?
-    exports.objectsCollection = objectsCollection = collection
-  )
-
-  db.collection('fs.files', (err, collection) ->
-    throw new Error(err) if err?
-    exports.filesCollection = filesCollection = collection
-  )
-)
+disconnect = () ->
+  exports.mongoDb?.close()
+  redisClient?.quit()
 
 
-getTask = (taskId, callback) ->
-  tid = String(taskId)
-  redisClient.get(tid, (err, res) ->
-    return callback(err) if err
-    if res?
-      callback(err, JSON.parse(res))
-    else
-      tasksCollection.findOne({_id : mongodb.ObjectID(tid)}, (err, task) ->
-        callback(err, task)
-      )
-  )
+# Optimize projection by removing overlaps
+# This modifies given object and returns it
+optimizeProjection = (obj) ->
+  keys = Object.keys(obj).sort()
+  return if keys.length <= 1
+  i = 1
+  while i < keys.length
+    a = keys[i-1]
+    b = keys[i]
+    if b.startsWith(a)
+      if b.charAt(a.length) == '.' or b.charAt(a.length - 1) == '.'
+        delete obj[b]
+        keys.splice(i--, 1)
+    ++ i
+  return obj
 
 
-getCached = (name, valueCallback, valueExpiry, callback) ->
-  redisClient.get(name, (err, res) ->
-    return callback(err) if err
-    return callback(null, JSON.parse(res)) if res?
+fetchDevice = (id, timestamp, patterns, callback) ->
+  MAX_DEPTH = config.get('MAX_DEPTH', id)
 
-    lockAcquired = () ->
-      valueCallback((err, value) ->
-        return callback(err, value) if err
-        redisClient.setex(name, valueExpiry, JSON.stringify(value), (err) ->
-          return callback(err, value) if err
-          redisClient.del("lock.#{name}", (err) ->
-            callback(err, value)
-          )
-        )
-      )
+  projection = {_id: 1}
+  projectionTree = {}
+  func = (path, pats, projTree) ->
+    children = {}
+    for pat in pats
+      fragment = pat[0][path.length] or '*'
+      load = 0
+      if fragment == '*'
+        projection[path.join('.')] = 1
+        return
 
-    redisClient.setnx("lock.#{name}", Date.now() + 30000, (err, res) ->
-      return callback(err) if err
-      if res == 1
-        lockAcquired()
+      projTree[fragment] ?= {}
+
+      if pat[1] & (1 << path.length)
+        projection[path.concat('_timestamp').join('.')] = 1
+        projection[path.concat([fragment, '_timestamp']).join('.')] = 1
+        projection[path.concat([fragment, '_value']).join('.')] = 1
+        projection[path.concat([fragment, '_type']).join('.')] = 1
+        projection[path.concat([fragment, '_object']).join('.')] = 1
+        projection[path.concat([fragment, '_instance']).join('.')] = 1
+        projection[path.concat([fragment, '_writable']).join('.')] = 1
+        projection[path.concat([fragment, '_orig']).join('.')] = 1
+
+      if pat[1] >> (path.length + 1)
+        children[fragment] ?= []
+        children[fragment].push(pat)
+
+    for k, v of children
+      func(path.concat(k), v, projTree[k])
+
+  func([], patterns, projectionTree)
+
+  delete projectionTree['DeviceID']
+  delete projectionTree['Events']
+  delete projectionTree['Tags']
+
+  if projection['']?
+    proj = {}
+  else
+    proj = JSON.parse(JSON.stringify(projection))
+
+  for k, v of proj
+    if k == '' or k == 'Events' or k.startsWith('Events.')
+      if k.length < 7 or k.startsWith('Events.Inform.') or k == 'Events.Inform'
+        proj['_lastInform'] = 1
+      if k.length < 7 or k.startsWith('Events.1_BOOT.') or k == 'Events.1_BOOT'
+        proj['_lastBoot'] = 1
+      if k.length < 7 or k.startsWith('Events.0_BOOTSTRAP.') or k == 'Events.0_BOOTSTRAP'
+        proj['_lastBootstrap'] = 1
+      if k.length < 7 or k.startsWith('Events.Registered.') or k == 'Events.Registered'
+        proj['_registered'] = 1
+      delete proj[k] if k != ''
+
+    if k == '' or k == 'DeviceID' or k.startsWith('DeviceID.')
+      if k.length < 9 or k.startsWith('DeviceID.ID.') or k == 'DeviceID.ID'
+        proj['_id'] = 1
+      if k.length < 9 or k.startsWith('DeviceID.Manufacturer.') or k == 'DeviceID.DeManufacturer'
+        proj['_deviceId._Manufacturer'] = 1
+      if k.length < 9 or k.startsWith('DeviceID.ProductClass.') or k == 'DeviceID.ProductClass'
+        proj['_deviceId._ProductClass'] = 1
+      if k.length < 9 or k.startsWith('DeviceID.OUI.') or k == 'DeviceID.OUI'
+        proj['_deviceId._OUI'] = 1
+      if k.length < 9 or k.startsWith('DeviceID.SerialNumber.') or k == 'DeviceID.SerialNumber'
+        proj['_deviceId._SerialNumber'] = 1
+      if k.length < 9
+        proj['_deviceId'] = 1
+      delete proj[k] if k != ''
+
+    if k == '' or k == 'Tags' or k.startsWith('Tags.')
+      proj['_tags'] = 1
+      delete proj[k] if k != ''
+
+  optimizeProjection(proj)
+
+  res = []
+  loaded = []
+
+  devicesCollection.findOne({'_id' : id}, proj, (err, device) ->
+    return callback(err) if err or not device?
+
+    storeParams = (obj, path, timestamp, descendantsFetched, projTree) ->
+      if descendantsFetched
+        thisFetched = true
       else
-        setTimeout(() ->
-          redisClient.get("lock.#{name}", (err, timestamp) ->
-            return callback(err) if err
-            now = Date.now()
-            if not timestamp?
-              getCached(name, valueCallback, valueExpiry, callback)
-            else if timestamp > now
-              setTimeout(() ->
-                getCached(name, valueCallback, valueExpiry, callback)
-              , 1000)
-            else
-              redisClient.getset("lock.#{name}", now + 30000, (err, timestamp) ->
-                return callback(err) if err
-                if timestamp > now
-                  setTimeout(() ->
-                    getCached(name, valueCallback, valueExpiry, callback)
-                  , 1000)
-                else
-                  lockAcquired()
-              )
-          )
-        , 1000)
+        if projection[path.join('.')]
+          descendantsFetched = true
+          loaded.push([path, ((1 << path.length) - 1) ^ ((1 << MAX_DEPTH) - 1)])
+        if projection[path.concat('_writable').join('.')]
+          loaded.push([path, 1 << (path.length - 1)])
+          thisFetched = true
+
+      if obj['_timestamp']?
+        obj['_timestamp'] = +obj['_timestamp']
+
+      if thisFetched
+        attrs = {}
+
+        t = if obj['_timestamp'] > timestamp then obj['_timestamp'] else timestamp
+
+        if obj['_value']?
+          attrs.value = [obj['_timestamp'] or 1, [obj['_value'], obj['_type']]]
+          attrs.value[1][0] = +attrs.value[1][0] if obj['_type'] == 'xsd:dateTime'
+          obj['_object'] = false
+
+        if obj['_writable']?
+          attrs.writable = [timestamp, if obj['_writable'] then 1 else 0]
+
+        if obj['_object']? or obj['_instance']?
+          attrs.object = [t, if (obj['_object'] or obj['_instance']) then 1 else 0]
+
+        res.push([path, t, attrs])
+
+      for k, v of obj
+        if k[0] != '_'
+          kk = k
+          obj['_object'] = true
+          storeParams(v, path.concat(k), obj['_timestamp'] or 1, descendantsFetched, projTree?[k])
+          delete projTree[kk] if projTree
+
+      if obj['_object'] or obj['_instance']
+        if descendantsFetched
+          res.push([path.concat('*'), obj['_timestamp']]) if obj['_timestamp']
+        else
+          for k, v of projTree
+            p = path.concat(k)
+            res.push([p, obj['_timestamp']]) if obj['_timestamp']
+            loaded.push([p, ((1 << path.length) - 1) ^ ((1 << MAX_DEPTH) - 1)])
+
+    for k, v of device
+      switch k
+        when '_lastInform'
+          res.push([['Events'], timestamp,
+            {object: [timestamp, 1], writable: [timestamp, 0]}])
+          res.push([['Events', 'Inform'], +v,
+            {object: [+v, 0], writable: [+v, 0], value: [+v, [+v, 'xsd:dateTime']]}])
+          delete device[k]
+        when '_lastBoot'
+          res.push([['Events'], timestamp,
+            {object: [timestamp, 1], writable: [timestamp, 0]}])
+          res.push([['Events', '1_BOOT'], +v,
+            {object: [+v, 0], writable: [+v, 0], value: [+v, [+v, 'xsd:dateTime']]}])
+          delete device[k]
+        when '_lastBootstrap'
+          res.push([['Events'], timestamp,
+            {object: [timestamp, 1], writable: [timestamp, 0]}])
+          res.push([['Events', '0_BOOTSTRAP'], +v,
+            {object: [+v, 0], writable: [+v, 0], value: [+v, [+v, 'xsd:dateTime']]}])
+          delete device[k]
+        when '_registered'
+          res.push([['Events'], timestamp,
+            {object: [timestamp, 1], writable: [timestamp, 0]}])
+          # Use current timestamp for registered event attribute timestamps
+          res.push([['Events', 'Registered'], timestamp,
+            {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [+v, 'xsd:dateTime']]}])
+          delete device[k]
+        when '_id'
+          res.push([['DeviceID'], timestamp,
+            {object: [timestamp, 1], writable: [timestamp, 0]}])
+          res.push([['DeviceID', 'ID'], timestamp,
+            {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v, 'xsd:string']]}])
+          delete device[k]
+        when '_tags'
+          res.push([['Tags'], timestamp,
+            {object: [timestamp, 1], writable: [timestamp, 0]}])
+          for t in v
+            t = t.replace(/[^a-zA-Z0-9\-]+/g, '_')
+            res.push([['Tags', t], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 1], value: [timestamp, [true, 'xsd:boolean']]}])
+          delete device[k]
+        when '_deviceId'
+          res.push([['DeviceID'], timestamp,
+            {object: [timestamp, 1], writable: [timestamp, 0]}])
+          if v['_Manufacturer']?
+            res.push([['DeviceID', 'Manufacturer'], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v['_Manufacturer'], 'xsd:string']]}])
+          if v['_OUI']?
+            res.push([['DeviceID', 'OUI'], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v['_OUI'], 'xsd:string']]}])
+          if v['_ProductClass']?
+            res.push([['DeviceID', 'ProductClass'], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v['_ProductClass'], 'xsd:string']]}])
+          if v['_SerialNumber']?
+            res.push([['DeviceID', 'SerialNumber'], timestamp,
+              {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v['_SerialNumber'], 'xsd:string']]}])
+          delete device[k]
+
+    storeParams(device, [], 0, false, projectionTree)
+
+    return callback(null, res, loaded)
+  )
+
+
+saveDevice = (deviceId, deviceData, isNew, callback) ->
+  update = {'$set' : {}, '$unset' : {}, '$addToSet' : {}, '$pull' : {}}
+
+  iter = deviceData.timestamps.diff()
+  while diff = iter.next().value
+    continue if diff[0].wildcard != (1 << (diff[0].length - 1))
+    continue if diff[0][0] in ['Events', 'DeviceID', 'Tags']
+
+    if not diff[2]? and diff[1]?
+      update['$unset'][diff[0].slice(0, -1).concat('_timestamp').join('.')] = 1
+    else if diff[2] != diff[1]
+      parent = deviceData.paths.get(diff[0].slice(0, -1))
+      if parent and (parent.length == 0 or deviceData.attributes.has(parent))
+        update['$set'][diff[0].slice(0, -1).concat('_timestamp').join('.')] = new Date(diff[2])
+
+  iter = deviceData.attributes.diff()
+  while diff = iter.next().value
+    continue if diff[1] == diff[2]
+    path = diff[0]
+    switch path[0]
+      when 'Events'
+        if diff[0].length == 2 and diff[2]?.value?[1][0] != diff[1]?.value?[1][0]
+          if not diff[2]
+            switch path[1]
+              when 'Inform'
+                update['$unset']['_lastInform'] = 1
+              when '1_BOOT'
+                update['$unset']['_lastBoot'] = 1
+              when '0_BOOTSTRAP'
+                update['$unset']['_lastBootstrap'] = 1
+              when 'Registered'
+                update['$unset']['_registered'] = 1
+          else
+            t = new Date(diff[2].value[1][0])
+            switch path[1]
+              when 'Inform'
+                update['$set']['_lastInform'] = t
+              when '1_BOOT'
+                update['$set']['_lastBoot'] = t
+              when '0_BOOTSTRAP'
+                update['$set']['_lastBootstrap'] = t
+              when 'Registered'
+                update['$set']['_registered'] = t
+      when 'DeviceID'
+        if diff[2].value?[1]?[0] != diff[1]?.value?[1]?[0]
+          v = diff[2].value[1][0]
+          switch path[1]
+            when 'ID'
+              update['$set']['_id'] = v
+            when 'Manufacturer'
+              update['$set']['_deviceId._Manufacturer'] = v
+            when 'OUI'
+              update['$set']['_deviceId._OUI'] = v
+            when 'ProductClass'
+              update['$set']['_deviceId._ProductClass'] = v
+            when 'SerialNumber'
+              update['$set']['_deviceId._SerialNumber'] = v
+      when 'Tags'
+        if diff[2]?.value?[1]?[0] != diff[1]?.value?[1]?[0]
+          v = diff[2]?.value?[1][0]
+          if v?
+            update['$addToSet']['_tags'] ?= {'$each' : []}
+            update['$addToSet']['_tags']['$each'].push(diff[0][1])
+          else
+            update['$pull']['_tags'] ?= {'$in' : []}
+            update['$pull']['_tags']['$in'].push(diff[0][1])
+      else
+        if not diff[2]
+          update['$unset'][diff[0].join('.')] = 1
+          continue
+
+        for attrName of diff[2]
+          if diff[2][attrName][1]? or diff[1]?[attrName]?[1]?
+            switch attrName
+              when 'value'
+                if not diff[1] or diff[2].value[1][0] != diff[1].value?[1][0]
+                  if diff[2].value[1][1] == 'xsd:dateTime'
+                    update['$set'][path.concat('_value').join('.')] = new Date(diff[2].value[1][0])
+                  else
+                    update['$set'][path.concat('_value').join('.')] = diff[2].value[1][0]
+                if not diff[1] or diff[2].value[1][1] != diff[1].value?[1][1]
+                  update['$set'][path.concat('_type').join('.')] = diff[2].value[1][1]
+                if not diff[1] or diff[2].value[0] != diff[1].value?[0]
+                  update['$set'][path.concat('_timestamp').join('.')] = new Date(diff[2].value[0])
+              when 'object'
+                if not diff[1]?.object or diff[2].object[1] != diff[1].object?[1]
+                  update['$set'][path.concat('_object').join('.')] = !!diff[2].object[1]
+              when 'writable'
+                if not diff[1]?.writable or diff[2].writable[1] != diff[1].writable?[1]
+                  update['$set'][path.concat('_writable').join('.')] = !!diff[2].writable[1]
+
+        if diff[1]
+          for attrName of diff[1]
+            if attrName not of diff[2]
+              update['$unset'][path.concat("_#{attrName}").join('.')] = 1
+              if attrName is 'value'
+                update['$unset'][path.concat('_type').join('.')] = 1
+                update['$unset'][path.concat('_timestamp').join('.')] = 1
+
+  # Remove empty keys
+  for k of update
+    if k == '$addToSet'
+      for kk of update[k]
+        delete update[k][kk] if update[k][kk]['$each'].length == 0
+    else if k == '$pull'
+      for kk of update[k]
+        delete update[k][kk] if update[k][kk]['$in'].length == 0
+    delete update[k] if Object.keys(update[k]).length == 0
+
+  return callback() if Object.keys(update).length == 0
+
+  if (update['$unset']?)
+    optimizeProjection(update['$unset'])
+
+  devicesCollection.update({'_id' : deviceId}, update, {upsert: isNew}, (err, result) ->
+    if not err and result.result.n != 1
+      return callback(new Error("Device #{deviceId} not found in database"))
+
+    return callback(err)
+  )
+
+
+getDueTasksAndFaultsAndOperations = (deviceId, timestamp, callback) ->
+  redisClient.mget("#{deviceId}_tasks", "#{deviceId}_faults", "#{deviceId}_operations", (err, res) ->
+    return callback(err) if err
+
+    if res[0]
+      tasks = JSON.parse(res[0])
+
+    if res[1]?
+      faults = JSON.parse(res[1])
+
+    if res[2]?
+      operations = JSON.parse(res[2])
+
+    if tasks? and faults? and operations?
+      return callback(null, tasks, faults, operations)
+
+    MAX_CACHE_TTL = config.get('MAX_CACHE_TTL', deviceId)
+
+    if not faults?
+      getFaults(deviceId, (err, flts) ->
+        if err
+          callback?(err)
+          return callback = null
+
+        faults = flts
+        if tasks? and operations?
+          return callback(null, tasks, faults, operations)
+      )
+
+    if not tasks?
+      getDueTasks(deviceId, timestamp, (err, dueTasks, nextTimestamp) ->
+        if err
+          callback?(err)
+          return callback = null
+
+        if nextTimestamp?
+          exp = Math.min(0, Math.trunc((nextTimestamp - Date.now()) / 1000))
+        else
+          exp = MAX_CACHE_TTL
+
+        redisClient.setex("#{deviceId}_tasks", exp, JSON.stringify(dueTasks), (err) ->
+          if err
+            callback?(err)
+            return callback = null
+
+          tasks = dueTasks
+          if faults? and operations?
+            return callback(null, tasks, faults, operations)
+        )
+      )
+
+    if not operations?
+      getOperations(deviceId, (err, _operations) ->
+        if err
+          callback?(err)
+          return callback = null
+
+        operations = _operations
+        if tasks? and faults?
+          return callback(null, tasks, faults, operations)
+      )
+
+  )
+
+
+getFaults = (deviceId, callback) ->
+  faultsCollection.find({'_id' : {'$regex' : "^#{common.escapeRegExp(deviceId)}\\:"}}).toArray((err, res) ->
+    return callback(err) if err
+
+    faults = {}
+    for r in res
+      channel = r._id.slice(deviceId.length + 1)
+      delete r._id
+      delete r.channel
+      delete r.device
+      r.timestamp = +r.timestamp
+      r.provisions = JSON.parse(r.provisions)
+      faults[channel] = r
+
+    MAX_CACHE_TTL = config.get('MAX_CACHE_TTL', deviceId)
+
+    redisClient.setex("#{deviceId}_faults", MAX_CACHE_TTL, JSON.stringify(faults), (err) ->
+      callback(err, faults)
     )
   )
 
 
-getAliases = (callback) ->
-  getCached('aliases', parameters.compileAliases, config.PRESETS_CACHE_DURATION, (err, res) ->
-    throw err if err
-    callback(res)
+saveFault = (deviceId, channel, fault, callback) ->
+  fault._id = "#{deviceId}:#{channel}"
+  fault.device = deviceId
+  fault.channel = channel
+  fault.timestamp = new Date(fault.timestamp)
+  fault.provisions = JSON.stringify(fault.provisions)
+  faultsCollection.save(fault, (err) ->
+    return callback(err) if err
+    redisClient.del("#{deviceId}_faults", callback)
   )
 
 
-getPresetsObjectsAliases = (callback) ->
-  redisClient.mget('presets', 'objects', 'aliases', (err, res) ->
-    presets = JSON.parse(res[0])
-    objects = JSON.parse(res[1])
-    aliases = JSON.parse(res[2])
-
-    if presets and objects and aliases
-      return callback(presets, objects, aliases)
-
-    if not presets
-      getCached('presets', (callback) ->
-        presetsCollection.find().toArray((err, res) ->
-          callback(err, res)
-        )
-      , config.PRESETS_CACHE_DURATION, (err, res) ->
-        throw err if err
-        presets = res
-        callback(presets, objects, aliases) if objects and aliases
-      )
-
-    if not objects
-      getCached('objects', (callback) ->
-        objectsCollection.find().toArray((err, res) ->
-          return callback(err) if err
-          objs = {}
-          objs[r._id] = r for r in res
-          callback(null, objs)
-        )
-      , config.PRESETS_CACHE_DURATION, (err, res) ->
-        throw err if err
-        objects = res
-        callback(presets, objects, aliases) if presets and aliases
-      )
-
-    if not aliases
-      getCached('aliases', parameters.compileAliases, config.PRESETS_CACHE_DURATION, (err, res) ->
-        throw err if err
-        aliases = res
-        callback(presets, objects, aliases) if presets and objects
-      )
+deleteFault = (deviceId, channel, callback) ->
+  faultsCollection.remove({_id: "#{deviceId}:#{channel}"}, (err) ->
+    return callback(err) if err
+    redisClient.del("#{deviceId}_faults", callback)
   )
 
 
-exports.redisClient = redisClient
-exports.getTask = getTask
-exports.getPresetsObjectsAliases = getPresetsObjectsAliases
-exports.getAliases = getAliases
+
+getDueTasks = (deviceId, timestamp, callback) ->
+  cur = tasksCollection.find({'device' : deviceId}).sort(['timestamp'])
+
+  tasks = []
+
+  cur.nextObject(f = (err, task) ->
+    return callback(err) if err
+
+    if not task?
+      return callback(null, tasks, null)
+
+    task.timestamp = +task.timestamp if task.timestamp?
+    task.expiry = +task.expiry if task.expiry?
+
+    if task.timestamp >= timestamp
+      return callback(null, tasks, +task.timestamp)
+
+    task._id = String(task._id)
+    tasks.push(task)
+
+    # For API compatibility
+    if task.name is 'download' and task.file?
+      if mongodb.ObjectID.isValid(task.file)
+        q = {_id: {'$in' : [task.file, new mongodb.ObjectID(task.file)]}}
+      else
+        q = {_id: task.file}
+
+      filesCollection.find(q).toArray((err, res) ->
+        return callback(err) if err
+        if res[0]?
+          task.fileType ?= res[0].metadata.fileType
+          task.fileName ?= res[0].filename or res[0]._id.toString()
+        cur.nextObject(f)
+      )
+    else
+      cur.nextObject(f)
+  )
+
+
+clearTasks = (deviceId, taskIds, callback) ->
+  if not taskIds?.length
+    return callback()
+
+  tasksCollection.remove({'_id' : {'$in' : (new mongodb.ObjectID(id) for id in taskIds)}}, (err) ->
+    return callback(err) if err
+    redisClient.del("#{deviceId}_tasks", callback)
+  )
+
+
+getOperations = (deviceId, callback) ->
+  operationsCollection.find({'_id' : {'$regex' : "^#{common.escapeRegExp(deviceId)}\\:"}}).toArray((err, res) ->
+    return callback(err) if err
+
+    operations = {}
+    for r in res
+      commandKey = r._id.slice(deviceId.length + 1)
+      delete r._id
+      r.timestamp = +r.timestamp
+      r.args = JSON.parse(r.args) if r.args
+      r.provisions = JSON.parse(r.provisions)
+      r.retries = JSON.parse(r.retries)
+      operations[commandKey] = r
+
+    MAX_CACHE_TTL = config.get('MAX_CACHE_TTL', deviceId)
+
+    redisClient.setex("#{deviceId}_operations", MAX_CACHE_TTL, JSON.stringify(operations), (err) ->
+      callback(err, operations)
+    )
+  )
+
+
+saveOperation = (deviceId, commandKey, operation, callback) ->
+  operation._id = "#{deviceId}:#{commandKey}"
+  operation.timestamp = new Date(operation.timestamp)
+  operation.provisions = JSON.stringify(operation.provisions)
+  operation.retries = JSON.stringify(operation.retries)
+  operation.args = JSON.stringify(operation.args)
+  operationsCollection.save(operation, (err) ->
+    return callback(err) if err
+    redisClient.del("#{deviceId}_operations", callback)
+  )
+
+
+deleteOperation = (deviceId, commandKey, callback) ->
+  operationsCollection.remove({_id: "#{deviceId}:#{commandKey}"}, (err) ->
+    return callback(err) if err
+    redisClient.del("#{deviceId}_operations", callback)
+  )
+
+
+exports.connect = connect
+exports.disconnect = disconnect
+exports.fetchDevice = fetchDevice
+exports.saveDevice = saveDevice
+exports.getDueTasksAndFaultsAndOperations = getDueTasksAndFaultsAndOperations
+exports.getFaults = getFaults
+exports.saveFault = saveFault
+exports.deleteFault = deleteFault
+exports.clearTasks = clearTasks
+exports.saveOperation = saveOperation
+exports.deleteOperation = deleteOperation

lib/default-provisions.js

@@ -0,0 +1,116 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const config = require("./config");
+const common = require("./common");
+const device = require("./device");
+
+const MAX_DEPTH = config.get("MAX_DEPTH");
+
+
+function refresh(sessionContext, provision, declarations, startRevision, endRevision) {
+  let path = common.parsePath(provision[1]).slice();
+  let l = path.length;
+  path.length = MAX_DEPTH;
+  path.fill("*", l);
+  let every = 1000 * (provision[2] || 1);
+  let offset = Math.abs(common.djb2(sessionContext.deviceId)) % 86400000;
+  let t = Math.trunc((sessionContext.timestamp + offset) / every) * every - offset;
+
+  for (let i = l; i < path.length; ++ i)
+    declarations.push([path.slice(0, i), t, {object: 1, writable: 1, value: t}]);
+
+  return true;
+}
+
+
+function value(sessionContext, provision, declarations, startRevision, endRevision) {
+  let path = common.parsePath(provision[1]);
+  declarations.push([path, 1, {value: 1}, null, {value: [provision[2]]}]);
+
+  return true;
+}
+
+
+function tag(sessionContext, provision, declarations, startRevision, endRevision) {
+  let path = ["Tags", provision[1]];
+  declarations.push([path, 1, {value: 1}, null, {value: [provision[2]]}]);
+
+  return true;
+}
+
+
+function reboot(sessionContext, provision, declarations, startRevision, endRevision) {
+  declarations.push([["Reboot"], 1, {value: 1}, null, {value: [sessionContext.timestamp]}]);
+
+  return true;
+}
+
+
+function reset(sessionContext, provision, declarations, startRevision, endRevision) {
+  declarations.push([["FactoryReset"], 1, {value: 1}, null, {value: [sessionContext.timestamp]}]);
+
+  return true;
+}
+
+
+function download(sessionContext, provision, declarations, startRevision, endRevision) {
+  let alias = ["FileType", provision[1] || "", "FileName", provision[2] || "",
+    "TargetFileName", provision[3] || ""];
+
+  declarations.push([["Downloads", alias], 1, {}, 1]);
+  declarations.push([["Downloads", alias, "Download"], 1, {value: 1}, null,
+    {value: [sessionContext.timestamp]}]);
+
+  return true;
+}
+
+
+function instances(sessionContext, provision, declarations, startRevision, endRevision) {
+  let count = Number(provision[2]);
+
+  if (Number.isNaN(count))
+    return true;
+
+  let path = common.parsePath(provision[1]);
+
+  if (provision[2][0] === "+" || provision[2][0] === "-") {
+    declarations.push([path, 1]);
+
+    if (endRevision === startRevision)
+      return false;
+
+    let unpacked = device.unpack(sessionContext.deviceData, path, startRevision + 1);
+    count = Math.max(0, unpacked.length + count);
+  }
+
+  declarations.push([path, 1, null, count]);
+
+  return true;
+}
+
+
+exports.refresh = refresh;
+exports.value = value;
+exports.tag = tag;
+exports.reboot = reboot;
+exports.reset = reset;
+exports.download = download;
+exports.instances = instances;

lib/device.js

@@ -0,0 +1,380 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const common = require('./common');
+
+
+const CHANGE_FLAGS = {
+  object: 4,
+  writable: 16,
+  value: 64
+};
+
+
+function sanitizeParameterValue(parameterValue) {
+  if (parameterValue[0] != null) {
+    switch (parameterValue[1]) {
+      case 'xsd:boolean':
+        if (typeof parameterValue[0] !== 'boolean') {
+          parameterValue[0] = !!JSON.parse(parameterValue[0]);
+        }
+        break;
+      case 'xsd:string':
+        if (typeof parameterValue[0] !== 'string') {
+          parameterValue[0] = '' + parameterValue[0];
+        }
+        break;
+      case 'xsd:int':
+      case 'xsd:unsignedInt':
+        if (typeof parameterValue[0] !== 'number') {
+          parameterValue[0] = +parameterValue[0];
+        }
+        break;
+      case 'xsd:dateTime':
+        if (typeof parameterValue[0] !== 'number') {
+          if (parameterValue[0].getTime() != null)
+            parameterValue[0] = parameterValue[0].getTime();
+          else if (isNaN(parameterValue[0]))
+            parameterValue[0] = Date.parse(parameterValue[0]);
+          else
+            parameterValue[0] = +parameterValue[0];
+        }
+        break;
+      default:
+        if (parameterValue[1] != null)
+          throw new Error(`Parameter value type "${parameterValue[1]}" not recognized.`);
+
+        parameterValue[0] = JSON.parse(JSON.stringify(parameterValue[0]));
+    }
+  }
+  return parameterValue;
+}
+
+
+function getAliasDeclarations(path, timestamp) {
+  var decs = [];
+
+  function recursive(pattern, prefix) {
+    var pattern = prefix.concat(pattern);
+
+    for (let i = prefix.length; i < pattern.length; ++ i) {
+      if (Array.isArray(pattern[i])) {
+        let pat = pattern[i];
+        pattern[i] = '*';
+        for (let j = 0; j < pat.length; j += 2)
+          recursive(pat[j], pattern.slice(0, i + 1));
+      }
+    }
+
+    if (prefix.length)
+      decs.push([pattern, timestamp, {value: timestamp}]);
+    else
+      decs.push([pattern, timestamp]);
+  }
+
+  recursive(path, []);
+
+  return decs;
+}
+
+
+function unpack(deviceData, path, revision) {
+  let allMatches = [];
+  if (!path.alias) {
+    for (let p of deviceData.paths.find(path, false, true))
+      if (deviceData.attributes.has(p, revision))
+        allMatches.push(p);
+  }
+  else {
+    let wildcardPath = path.slice();
+    wildcardPath.wildcard = path.wildcard;
+    for (let i = 0; i < wildcardPath.length; ++ i)
+      if (Array.isArray(wildcardPath[i])) {
+        wildcardPath[i] = '*';
+        wildcardPath.wildcard |= 1 << i;
+      }
+
+    for (let p of deviceData.paths.find(wildcardPath, false, true))
+      if (deviceData.attributes.has(p, revision))
+        allMatches.push(p);
+
+    for (let i = path.length - 1; i >= 0; -- i) {
+      if (Array.isArray(path[i])) {
+        for (let j = 0; j < path[i].length; j += 2) {
+          let p = wildcardPath.slice(0, i + 1).concat(path[i][j]);
+          p.alias = (wildcardPath.alias & ((1 << (i + 1)) - 1)) + path[i][j].alias << (i + 1);
+          p.wildcard = (wildcardPath.wildcard & ((1 << (i + 1)) - 1)) + path[i][j].wildcard << (i + 1);
+          let unpacked = unpack(deviceData, p, revision);
+          let filtered = [];
+          for (let up of unpacked) {
+            let attributes = deviceData.attributes.get(up, revision);
+            if (attributes && attributes.value && attributes.value[1] &&
+                sanitizeParameterValue([path[i][j + 1], attributes.value[1][1]])[0] === attributes.value[1][0]) {
+              for (let m = 0; m < allMatches.length; ++ m) {
+                let k;
+                for (k = i; k >= 0; -- k)
+                  if (allMatches[m][k] != up[k])
+                    break
+
+                if (k < 0) {
+                  filtered.push(allMatches[m]);
+                  allMatches[m] = [];
+                }
+              }
+            }
+          }
+          allMatches = filtered;
+        }
+      }
+    }
+  }
+
+  allMatches.sort(function(a, b) {
+    for (let i = 0; i < a.length; ++ i) {
+      if (a === b)
+        continue;
+
+      // Use numeric sorting for numbers
+      let ia = parseInt(a), ib = parseInt(b);
+      if (ia.toString() === a && ib.toString() === b) {
+        a = ia;
+        b = ib;
+      }
+
+      if (a < b)
+        return -1;
+      else
+        return 1;
+    }
+    return 0;
+  });
+
+  return allMatches;
+}
+
+
+function clear(deviceData, path, timestamp, attributes) {
+  for (let p of deviceData.paths.find(path, true, true, timestamp ? 99 : path.length)) {
+    for (let k in deviceData.trackers.get(p))
+      deviceData.changes.add(k);
+
+    let t = deviceData.timestamps.get(p);
+    if (!t)
+      continue;
+
+    if (!(timestamp <= t)) {
+      deviceData.timestamps.delete(p);
+      deviceData.attributes.delete(p);
+    } else if (t && attributes) {
+      let currentAttributes = deviceData.attributes.get(p);
+      if (currentAttributes) {
+        let newAttrs;
+        for (let attrName in attributes) {
+          if (attrName in currentAttributes &&
+              attributes[attrName][0] > currentAttributes[attrName][0]) {
+            if (!newAttrs)
+              newAttrs = JSON.parse(JSON.stringify(currentAttributes));
+            delete newAttrs[attrName];
+          }
+        }
+
+        if (newAttrs)
+          deviceData.attributes.set(p, newAttrs);
+      }
+    }
+  }
+}
+
+
+function compareEquality(a, b) {
+  var t = typeof a;
+  if (a === null ||
+      a === undefined ||
+      t === 'number' ||
+      t === 'boolean' ||
+      t === 'string' ||
+      t === 'symbol')
+    return a === b;
+
+  return JSON.stringify(a) === JSON.stringify(b);
+}
+
+
+function set(deviceData, path, timestamp, attributes, toClear) {
+  path = deviceData.paths.add(path);
+
+  let currentTimestamp = deviceData.timestamps.get(path);
+
+  let currentAttributes;
+
+  if (path.wildcard)
+    attributes = undefined;
+  else
+    currentAttributes = deviceData.attributes.get(path);
+
+  if (attributes) {
+    if (attributes.value && attributes.value[1]
+        && attributes.value[0] >= (attributes.object ? attributes.object[0] : 0))
+      attributes.object = [attributes.value[0], 0];
+
+    if (attributes.object && attributes.object[1]
+      && attributes.object[0] >= (attributes.value ? attributes.value[0] : 0))
+      attributes.value = [attributes.object[0]];
+
+    if (currentAttributes) {
+      let changeFlags = 0;
+
+      for (let attrName in attributes) {
+        timestamp = Math.max(timestamp, attributes[attrName][0]);
+        if (attrName in currentAttributes) {
+          if (attributes[attrName][0] > currentAttributes[attrName][0]) {
+            changeFlags |= CHANGE_FLAGS[attrName];
+            if (compareEquality(attributes[attrName][1], currentAttributes[attrName][1]))
+              changeFlags |= CHANGE_FLAGS[attrName] << 1;
+          }
+          else
+            delete attributes[attrName];
+        }
+        else {
+          changeFlags |= CHANGE_FLAGS[attrName];
+          changeFlags |= CHANGE_FLAGS[attrName] << 1;
+        }
+      }
+
+      if (Object.keys(attributes).length) {
+        if (attributes.object && currentAttributes.object &&
+            attributes.object[1] !== currentAttributes.object[1]) {
+          toClear = toClear || [];
+          toClear.push([path.concat('*'), attributes.object[0]]);
+        }
+        for (let attrName in currentAttributes)
+          if (!(attrName in attributes))
+            attributes[attrName] = currentAttributes[attrName];
+
+        deviceData.attributes.set(path, attributes);
+      }
+
+      if (timestamp > currentTimestamp) {
+        changeFlags |= 1;
+        deviceData.timestamps.set(path, timestamp);
+
+        if (path.length > 1)
+          toClear = set(deviceData, path.slice(0, path.length - 1), timestamp,
+            {object: [timestamp, 1]}, toClear);
+      }
+
+      if (changeFlags)
+        for (let sup of deviceData.paths.find(path, true, false)) {
+          let c = deviceData.trackers.get(sup);
+          for (let k in c)
+            if (c[k] & changeFlags) {
+              deviceData.changes.add(k);
+            }
+        }
+    }
+    else if (!(timestamp <= currentTimestamp)) {
+      deviceData.timestamps.set(path, timestamp);
+      deviceData.attributes.set(path, attributes);
+
+      if (path.length > 1)
+        toClear = set(deviceData, path.slice(0, path.length - 1), timestamp,
+          {object: [timestamp, 1]}, toClear);
+
+      toClear = toClear || [];
+      toClear.push([path, timestamp]);
+    }
+  }
+  else if (!(timestamp <= currentTimestamp)) {
+    deviceData.timestamps.set(path, timestamp);
+
+    if (currentAttributes) {
+      deviceData.attributes.delete(path);
+      toClear = toClear || [];
+      toClear.push([path, timestamp]);
+    }
+
+    if (path.wildcard)
+      for (let sub of deviceData.paths.find(path, false, true))
+        if (timestamp > deviceData.timestamps.get(sub)) {
+          deviceData.timestamps.delete(sub);
+          if (deviceData.attributes.delete(sub)) {
+            toClear = toClear || [];
+            toClear.push([sub, timestamp]);
+          }
+        }
+  }
+
+  return toClear;
+}
+
+
+function track(deviceData, path, marker, timestamp, attributeTimestamps, exist, attributeValues) {
+  path = deviceData.paths.add(path);
+  let f = 0;
+
+  if (timestamp)
+    f |= 1;
+
+  if (exist)
+    f |= 2;
+
+  if (attributeTimestamps)
+    for (let attrName of attributeTimestamps)
+      f |= CHANGE_FLAGS[attrName];
+
+  if (attributeValues)
+    for (let attrName of attributeValues)
+      f |= CHANGE_FLAGS[attrName] << 1;
+
+  let cur = deviceData.trackers.get(path);
+  if (!cur) {
+    cur = {};
+    deviceData.trackers.set(path, cur);
+  }
+
+  cur[marker] |= f;
+}
+
+
+function clearTrackers(deviceData, tracker) {
+  if (Array.isArray(tracker)) {
+    deviceData.trackers.forEach(function(v, k) {
+      for (let t of tracker)
+        delete v[t];
+    });
+    for (let t of tracker)
+      deviceData.changes.delete(t);
+  }
+  else {
+    deviceData.trackers.forEach(function(v, k) {
+      delete v[tracker];
+    });
+    deviceData.changes.delete(tracker);
+  }
+}
+
+
+exports.sanitizeParameterValue = sanitizeParameterValue;
+exports.getAliasDeclarations = getAliasDeclarations;
+exports.unpack = unpack;
+exports.set = set;
+exports.clear = clear;
+exports.track = track;
+exports.clearTrackers = clearTrackers;

lib/extension-wrapper.js

@@ -0,0 +1,64 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const jobs = new Set();
+const fileName = process.argv[2];
+let script;
+
+function errorToFault(err) {
+  if (!err)
+    return null;
+
+  return {
+    code: `ext.${err.name}`,
+    message: err.message,
+    detail: {
+      name: err.name,
+      message: err.message,
+      stack: err.stack
+    }
+  }
+}
+
+process.on("uncaughtException", function(err) {
+  let fault = errorToFault(err);
+  jobs.forEach(function(jobId) {
+    process.send([jobId, fault]);
+  });
+  jobs.clear();
+  process.disconnect();
+});
+
+process.on("message", function(message) {
+  jobs.add(message[0]);
+
+  if (!script) {
+    let cwd = process.env["GENIEACS_CONFIG_DIR"] + "/ext";
+    process.chdir(cwd);
+    script = require(`${cwd}/${fileName}`);
+  }
+
+  script[message[1][0]](message[1].slice(1), function(err, res) {
+    if (!jobs.delete(message[0]))
+      return;
+
+    process.send([message[0], errorToFault(err), res]);
+  });
+});

lib/extensions.js

@@ -0,0 +1,100 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const childProcess = require("child_process");
+const crypto = require ("crypto");
+
+const config = require("./config");
+
+const TIMEOUT = config.get('EXT_TIMEOUT');
+
+const processes = {};
+const jobs = {};
+
+function messageHandler(message) {
+  let func = jobs[message[0]];
+  if (func) {
+    delete jobs[message[0]];
+    func(null, message[1], message[2]);
+  }
+}
+
+
+function run(args, callback) {
+  let scriptName = args[0];
+
+  if (!processes[scriptName]) {
+    let p = childProcess.fork('lib/extension-wrapper', [scriptName]);
+
+    p.on('error', function() {
+      kill(processes[scriptName]);
+      delete processes[scriptName];
+    });
+
+    p.on('disconnect', function() {
+      kill(processes[scriptName]);
+      delete processes[scriptName];
+    });
+
+    p.on('message', messageHandler);
+    processes[scriptName] = p;
+  }
+
+  let id = crypto.randomBytes(8).toString('hex');
+  jobs[id] = callback;
+  setTimeout(function() {
+    if (id in jobs) {
+      delete jobs[id];
+      return callback(null, {code: "timeout", message: "Extension timed out"});
+    }
+  }, TIMEOUT);
+
+  return processes[scriptName].send([id, args.slice(1)]);
+};
+
+
+function kill(process) {
+  if (process.signalCode !== null || process.exitCode !== null)
+    return;
+
+  let timeToKill = Date.now() + 5000;
+
+  process.kill();
+
+  let t = setInterval(function() {
+    if (process.signalCode !== null || process.exitCode !== null) {
+      clearInterval(t);
+    }
+    else if (Date.now() > timeToKill) {
+      process.kill('SIGKILL');
+      clearInterval(t);
+    }
+  }, 100);
+}
+
+
+function killAll() {
+  for (let n in processes)
+    kill(processes[n]);
+}
+
+
+exports.run = run;
+exports.killAll = killAll;

lib/fs.coffee

@@ -1,51 +1,49 @@
+###
+# Copyright 2013 Fanoos Telecom
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+###
+
 config = require './config'
-util = require 'util'
-http = require 'http'
 url = require 'url'
 mongodb = require 'mongodb'
 querystring = require 'querystring'
-
-db = null
-mongodb.MongoClient.connect("mongodb://#{config.MONGODB_SOCKET}/#{config.DATABASE_NAME}", config.MONGODB_OPTIONS, (err, _db) ->
-  db = _db
-)
+db = require './db'
 
 
-cluster = require 'cluster'
-numCPUs = require('os').cpus().length
+listener = (request, response) ->
+  urlParts = url.parse(request.url, true)
+  if request.method == 'GET'
+    filename = querystring.unescape(urlParts.pathname.substring(1))
+    gs = new mongodb.GridStore(db.mongoDb, filename, 'r', {})
+    gs.open((err, gs) ->
+      if err
+        response.writeHead(404)
+        response.end()
+        return
+      stream = gs.stream(true)
+      response.writeHead(200, {'Content-Type' : 'application/octet-stream', 'Content-Length' : gs.length})
+      stream.pipe(response)
+    )
+  else
+    response.writeHead(405, {'Allow': 'GET'})
+    response.end('405 Method Not Allowed')
 
-if cluster.isMaster
-  cluster.on('listening', (worker, address) ->
-    util.log("Worker #{worker.process.pid} listening to #{address.address}:#{address.port}")
-  )
 
-  cluster.on('exit', (worker, code, signal) ->
-    util.log("Worker #{worker.process.pid} died (#{worker.process.exitCode})")
-    setTimeout(()->
-      cluster.fork()
-    , config.WORKER_RESPAWN_TIME)
-  )
-
-  for i in [1 .. numCPUs]
-    cluster.fork()
-else
-  server = http.createServer((request, response) ->
-    urlParts = url.parse(request.url, true)
-    if request.method == 'GET'
-      filename = querystring.unescape(urlParts.pathname.substring(1))
-      gs = new mongodb.GridStore(db, filename, 'r', {})
-      gs.open((err, gs) ->
-        if err
-          response.writeHead(404)
-          response.end()
-          return
-        stream = gs.stream(true)
-        response.writeHead(200, {'Content-Type' : 'application/octet-stream', 'Content-Length' : gs.length})
-        stream.pipe(response)
-      )
-    else
-      response.writeHead(405, {'Allow': 'GET'})
-      response.end('405 Method Not Allowed')
-  )
-
-  server.listen config.FILES_PORT, config.FILES_INTERFACE
+exports.listener = listener

lib/instance-set.js

@@ -0,0 +1,138 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+
+class InstanceSet {
+
+  constructor() {
+    this.set = new Set();
+  }
+
+  add(instance) {
+    this.set.add(instance);
+  }
+
+  delete(instance) {
+    this.set.delete(instance);
+  }
+
+  superset(instance) {
+    let res = [];
+    for (let inst of this.set) {
+      let match = true;
+      for (let k in instance) {
+        if (inst[k] !== instance[k]) {
+          match = false;
+          break;
+        }
+      }
+
+      if (match)
+        res.push(inst);
+    }
+
+    res.sort(function(a, b) {
+      let keysA = Object.keys(a);
+      let keysB = Object.keys(b);
+
+      if (keysA.length != keysB.length)
+        return keysB.length - keysA.length;
+
+      keysA.sort();
+      keysB.sort();
+
+      for (let i = 0; i < keysA.length; ++ i) {
+        if (keysA[i] > keysB[i])
+          return 1;
+        else if (keysA[i] < keysB[i])
+          return -1;
+        else if (a[keysA[i]] > b[keysB[i]])
+          return 1;
+        else if (a[keysA[i]] < b[keysB[i]])
+          return -1;
+      }
+
+      return 0;
+    });
+
+    return res;
+  }
+
+  subset(instance) {
+    let res = [];
+
+    for (let inst of this.set) {
+      let match = true;
+      for (let k in inst) {
+        if (inst[k] !== instance[k]) {
+          match = false;
+          break;
+        }
+      }
+
+      if (match)
+        res.push(inst);
+    }
+
+    res.sort(function(a, b) {
+      let keysA = Object.keys(a);
+      let keysB = Object.keys(b);
+
+      if (keysA.length != keysB.length)
+        return keysA.length - keysB.length;
+
+      keysA.sort();
+      keysB.sort();
+
+      for (let i = 0; i < keysA.length; ++ i) {
+        if (keysA[i] > keysB[i])
+          return 1;
+        else if (keysA[i] < keysB[i])
+          return -1;
+        else if (a[keysA[i]] > b[keysB[i]])
+          return 1;
+        else if (a[keysA[i]] < b[keysB[i]])
+          return -1;
+      }
+
+      return 0;
+    });
+
+    return res;
+  }
+
+  forEach(callback) {
+    return this.set.forEach(callback);
+  }
+
+  values() {
+    return this.set.values();
+  }
+
+  clear() {
+    this.set.clear();
+  }
+
+  get size() {
+    return this.set.size;
+  }
+}
+
+module.exports = InstanceSet;

lib/logger.js

@@ -0,0 +1,319 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const fs = require("fs");
+const os = require("os");
+
+const config = require("./config");
+
+const REOPEN_EVERY = 60000;
+
+const LOG_FORMAT = config.get("LOG_FORMAT");
+const ACCESS_LOG_FORMAT = config.get("ACCESS_LOG_FORMAT") || LOG_FORMAT;
+
+const defaultMeta = {};
+
+let LOG_SYSTEMD = false;
+let ACCESS_LOG_SYSTEMD = false;
+
+let LOG_FILE, ACCESS_LOG_FILE;
+
+let logStream = fs.createWriteStream(null, {fd: process.stderr.fd});
+let logStat = fs.fstatSync(logStream.fd);
+let accessLogStream = fs.createWriteStream(null, {fd: process.stdout.fd});
+let accessLogStat = fs.fstatSync(accessLogStream.fd);
+
+
+// Reopen if original files have been moved (e.g. logrotate)
+function reopen() {
+  let counter = 1;
+
+  if (LOG_FILE) {
+    ++ counter;
+    fs.stat(LOG_FILE, function(err, stat) {
+      if (err && !err.message.startsWith("ENOENT:"))
+        throw err;
+
+      if (!(stat && stat.dev === logStat.dev && stat.ino === logStat.ino)) {
+        logStream.end();
+        logStream = fs.createWriteStream(null,
+          {fd: fs.openSync(LOG_FILE, "a")});
+        logStat = fs.fstatSync(logStream.fd);
+      }
+
+      if (-- counter === 0)
+        setTimeout(reopen, REOPEN_EVERY - (Date.now() % REOPEN_EVERY)).unref();
+    });
+  }
+
+  if (ACCESS_LOG_FILE) {
+    ++ counter;
+    fs.stat(ACCESS_LOG_FILE, function(err, stat) {
+      if (err && !err.message.startsWith("ENOENT:"))
+        throw err;
+
+      if (!(stat && stat.dev === accessLogStat.dev && stat.ino === accessLogStat.ino)) {
+        accessLogStream.end();
+        accessLogStream = fs.createWriteStream(null,
+          {fd: fs.openSync(ACCESS_LOG_FILE, "a")});
+        accessLogStat = fs.fstatSync(accessLogStream.fd);
+      }
+
+      if (-- counter === 0)
+        setTimeout(reopen, REOPEN_EVERY - (Date.now() % REOPEN_EVERY)).unref();
+    });
+  }
+
+  if (-- counter === 0)
+    setTimeout(reopen, REOPEN_EVERY - (Date.now() % REOPEN_EVERY)).unref();
+}
+
+
+function init(service, version) {
+  defaultMeta.hostname = os.hostname();
+  defaultMeta.pid = process.pid;
+  defaultMeta.name = `genieacs-${service}`;
+  defaultMeta.version = version;
+
+  LOG_FILE = config.get(`${service.toUpperCase()}_LOG_FILE`);
+  ACCESS_LOG_FILE = config.get(`${service.toUpperCase()}_ACCESS_LOG_FILE`);
+
+  if (LOG_FILE) {
+    logStream = fs.createWriteStream(null,
+      {fd: fs.openSync(LOG_FILE, "a")});
+    logStat = fs.fstatSync(logStream.fd);
+  }
+
+  if (ACCESS_LOG_FILE) {
+    accessLogStream = fs.createWriteStream(null,
+      {fd: fs.openSync(ACCESS_LOG_FILE, "a")});
+    accessLogStat = fs.fstatSync(accessLogStream.fd);
+  }
+
+  // Determine if logs are going to journald
+  const JOURNAL_STREAM = process.env["JOURNAL_STREAM"];
+
+  if (JOURNAL_STREAM) {
+    let [dev, inode] = JOURNAL_STREAM.split(":");
+    dev = parseInt(dev);
+    inode = parseInt(inode);
+
+    LOG_SYSTEMD = (logStat.dev === dev && logStat.ino === inode);
+    ACCESS_LOG_SYSTEMD = (accessLogStat.dev === dev && accessLogStat.ino === inode);
+  }
+
+  if (LOG_FILE || ACCESS_LOG_FILE) {
+    // Can't use setInterval as we need all workers to cehck at the same time
+    setTimeout(reopen, REOPEN_EVERY - (Date.now() % REOPEN_EVERY)).unref();
+  }
+}
+
+
+function close() {
+  accessLogStream.end();
+  logStream.end();
+}
+
+
+function flatten(details) {
+  if (details.sessionContext) {
+    details.deviceId = details.sessionContext.deviceId;
+    details.remoteAddress = details.sessionContext.httpRequest.connection.remoteAddress;
+    delete details.sessionContext;
+  }
+
+  if (details.exception) {
+    details.exceptionName = details.exception.name;
+    details.exceptionMessage = details.exception.message;
+    details.exceptionStack = details.exception.stack;
+    delete details.exception;
+  }
+
+  if (details.task) {
+    details.taskId = details.task._id;
+    delete details.task;
+  }
+
+  if (details.rpc) {
+    if (details.rpc.acsRequest) {
+      details.acsRequestId = details.rpc.id;
+      details.acsRequestName = details.rpc.acsRequest.name;
+      if (details.rpc.acsRequest.commandKey)
+        details.acsRequestCommandKey = details.rpc.acsRequest.commandKey;
+    }
+    else if (details.rpc.cpeRequest) {
+      details.cpeRequestId = details.rpc.id;
+      if (details.rpc.cpeRequest.name === "Inform") {
+        details.informEvent = details.rpc.cpeRequest.event.join(",");
+        details.informRetryCount = details.rpc.cpeRequest.retryCount;
+      }
+      else {
+        details.cpeRequestName = details.rpc.cpeRequest.type;
+        if (details.rpc.cpeRequest.commandKey)
+          details.cpeRequestCommandKey = details.rpc.cpeRequest.commandKey;
+      }
+    }
+    else if (details.rpc.cpeFault) {
+      details.acsRequestId = details.rpc.id;
+      details.cpeFaultCode = details.rpc.cpeFault.faultCode;
+      details.cpeFaultString = details.rpc.cpeFault.faultString;
+    }
+    delete details.rpc;
+  }
+
+  if (details.fault) {
+    details.faultCode = details.fault.code;
+    details.faultMessage = details.fault.message;
+    delete details.fault;
+  }
+
+  return details;
+}
+
+
+function formatJson(details, systemd) {
+  if (systemd) {
+    let severity = "";
+    if (details.severity === "info")
+      severity = "<6>";
+    else if (details.severity === "warn")
+      severity = "<4>";
+    else if (details.severity === "error")
+      severity = "<3>";
+
+    return `${severity}${JSON.stringify(flatten(details))}${os.EOL}`;
+  }
+
+  return `${JSON.stringify(flatten(details))}${os.EOL}`;
+}
+
+
+function formatSimple(details, systemd) {
+  flatten(details);
+
+  let remote = "";
+  if (details.remoteAddress) {
+    if (details.deviceId)
+      remote = `${details.remoteAddress} ${details.deviceId}: `;
+    else
+      remote = `${details.remoteAddress}: `;
+  }
+
+  const keys = Object.keys(details);
+  const skip = {
+    deviceId: 1,
+    remoteAddress: 1,
+    severity: 1,
+    timestamp: 1,
+    message: 1
+  };
+
+  let meta = "";
+
+  let kv = [];
+  for (let k of keys)
+    if (!skip[k])
+      kv.push(`${k}=${JSON.stringify(details[k])}`);
+
+  if (kv.length)
+    meta = `; ${kv.join(" ")}`;
+
+  if (systemd) {
+    let severity = "";
+    if (details.severity === "info")
+      severity = "<6>";
+    else if (details.severity === "warn")
+      severity = "<4>";
+    else if (details.severity === "error")
+      severity = "<3>";
+
+    return `${severity}${remote}${details.message}${meta}${os.EOL}`;
+  }
+
+  return `${details.timestamp} [${details.severity.toUpperCase()}] ${remote}${details.message}${meta}${os.EOL}`;
+}
+
+
+function log(details) {
+  details.timestamp = new Date().toISOString();
+  if (LOG_FORMAT === "json") {
+    details = Object.assign({}, defaultMeta, details);
+    logStream.write(formatJson(details, LOG_SYSTEMD));
+  } else {
+    logStream.write(formatSimple(details, LOG_SYSTEMD));
+  }
+}
+
+
+function info(details) {
+  details.severity = "info";
+  log(details);
+}
+
+
+function warn(details) {
+  details.severity = "warn";
+  log(details);
+}
+
+
+function error(details) {
+  details.severity = "error";
+  log(details);
+}
+
+
+function accessLog(details) {
+  details.timestamp = new Date().toISOString();
+  if (ACCESS_LOG_FORMAT === "json") {
+    Object.assign(details, defaultMeta);
+    accessLogStream.write(formatJson(details, ACCESS_LOG_SYSTEMD));
+  } else {
+    accessLogStream.write(formatSimple(details, ACCESS_LOG_SYSTEMD));
+  }
+}
+
+
+function accessInfo(details) {
+  details.severity = "info";
+  accessLog(details);
+}
+
+
+function accessWarn(details) {
+  details.severity = "warn";
+  accessLog(details);
+}
+
+
+function accessError(details) {
+  details.severity = "error";
+  accessLog(details);
+}
+
+
+exports.init = init;
+exports.close = close;
+exports.info = info;
+exports.warn = warn;
+exports.error = error;
+exports.accessInfo = accessInfo;
+exports.accessWarn = accessWarn;
+exports.accessError = accessError;

lib/mongo-query.coffee

@@ -1,100 +0,0 @@
-common = require './common'
-
-
-test = (obj, query) ->
-  for k,v of query
-    if k.charAt(0) == '$' # this is a logical operator
-      res = testExpressions(obj, v, k)
-    else
-      value = common.getParamValueFromPath(obj, k)
-
-      if common.typeOf(v) isnt common.OBJECT_TYPE
-        # TODO comparing array to regex, array to array, and object to object
-        if common.typeOf(value) is common.ARRAY_TYPE
-          res = value.indexOf(v) != -1
-        else
-          if common.typeOf(v) is common.REGEXP_TYPE
-            res = v.test(value)
-          else
-            res = v == value
-      else
-        for k2,v2 of v
-          switch k2
-            when '$ne'
-              if common.typeOf(value) is common.ARRAY_TYPE
-                res = value.indexOf(v2) == -1
-              else
-                res = value != v2
-            when '$lt'
-              res = value < v2
-            when '$lte'
-              res = value <= v2
-            when '$gt'
-              res = value > v2
-            when '$gte'
-              res = value >= v2
-            when '$regex'
-              res = v2.test(value)
-            when '$in'
-              throw new Error('Operator not supported')        
-            when '$nin'
-              throw new Error('Operator not supported')        
-            when '$all'
-              throw new Error('Operator not supported')        
-            when '$exists'
-              throw new Error('Operator not supported')        
-            else
-              throw new Error('Operator not supported')        
-
-    if not res
-      return false
-
-  return true
-
-
-testExpressions = (obj, expressions, lop) ->
-  for f in expressions
-    res = test(obj, f)
-
-    switch lop
-      when '$and'
-        return false if not res
-      when '$or'
-        return true if res
-      when '$nor'
-        return false if res
-
-  return switch lop
-    when '$and' then true
-    when '$or' then false
-    when '$nor' then true
-    else throw new Error('Unknown logical operator')
-
-
-# generate field projection from a query
-projection = (query, proj) ->
-  for k,v of query
-    if k.charAt(0) == '$' # this is a logical operator
-      for q in v
-        projection(q, proj)
-    else
-      proj[k] = 1
-
-# optimize projection by removing overlaps
-optimizeProjection = (obj) ->
-  keys = Object.keys(obj).sort()
-  return if keys.length <= 1
-  i = 1
-  while i < keys.length
-    a = keys[i-1]
-    b = keys[i]
-    if common.startsWith(b, a)
-      if b.charAt(a.length) == '.' or b.charAt(a.length - 1) == '.'
-        delete obj[b]
-        keys.splice(i--, 1)
-    ++ i
-
-
-exports.test = test
-exports.projection = projection
-exports.optimizeProjection = optimizeProjection

lib/nbi.coffee

@@ -1,14 +1,57 @@
-config = require './config'
-util = require 'util'
-http = require 'http'
+###
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
+#
+# GenieACS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# GenieACS is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This file incorporates work covered by the following copyright and
+# permission notice:
+#
+# Copyright 2013 Fanoos Telecom
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+###
+
 url = require 'url'
-common = require './common'
-db = require './db'
 mongodb = require 'mongodb'
 querystring = require 'querystring'
+vm = require 'vm'
+
+config = require './config'
+common = require './common'
+db = require './db'
 query = require './query'
 apiFunctions = require './api-functions'
-presets = require './presets'
+
+VERSION = require('../package.json').version
 
 # regular expression objects
 DEVICE_TASKS_REGEX = /^\/devices\/([a-zA-Z0-9\-\_\%]+)\/tasks\/?$/
@@ -16,348 +59,458 @@ TASKS_REGEX = /^\/tasks\/([a-zA-Z0-9\-\_\%]+)(\/[a-zA-Z_]*)?$/
 TAGS_REGEX = /^\/devices\/([a-zA-Z0-9\-\_\%]+)\/tags\/([a-zA-Z0-9\-\_\%]+)\/?$/
 PRESETS_REGEX = /^\/presets\/([a-zA-Z0-9\-\_\%]+)\/?$/
 OBJECTS_REGEX = /^\/objects\/([a-zA-Z0-9\-\_\%]+)\/?$/
-FILES_REGEX = /^\/files\/([a-zA-Z0-9\-\_\%\ \.\/\(\)]+)\/?$/
+FILES_REGEX = /^\/files\/([a-zA-Z0-9\%\!\*\'\(\)\;\:\@\&\=\+\$\,\?\#\[\]\-\_\.\~]+)\/?$/
 PING_REGEX = /^\/ping\/([a-zA-Z0-9\-\_\.]+)\/?$/
 QUERY_REGEX = /^\/([a-zA-Z0-9_]+s)\/?$/
-DEVICE_PRESET_REGEX = /^\/devices\/([a-zA-Z0-9\-\_\%]+)\/preset\/?$/
+DELETE_DEVICE_REGEX = /^\/devices\/([a-zA-Z0-9\-\_\%]+)\/?$/
+PROVISIONS_REGEX = /^\/provisions\/([a-zA-Z0-9\-\_\%]+)\/?$/
+VIRTUAL_PARAMETERS_REGEX = /^\/virtual_parameters\/([a-zA-Z0-9\-\_\%]+)\/?$/
+FAULTS_REGEX = /^\/faults\/([a-zA-Z0-9\-\_\%\:]+)\/?$/
 
 
-cluster = require 'cluster'
-numCPUs = require('os').cpus().length
 
-if cluster.isMaster
-  cluster.on('listening', (worker, address) ->
-    util.log("Worker #{worker.process.pid} listening to #{address.address}:#{address.port}")
-  )
+throwError = (err, httpResponse) ->
+  if httpResponse
+    httpResponse.writeHead(500, {'Connection' : 'close'})
+    httpResponse.end("#{err.name}: #{err.message}")
 
-  cluster.on('exit', (worker, code, signal) ->
-    util.log("Worker #{worker.process.pid} died (#{worker.process.exitCode})")
-    setTimeout(()->
-      cluster.fork()
-    , config.WORKER_RESPAWN_TIME)
-  )
+  throw err
 
-  for [1 .. numCPUs]
-    cluster.fork()
-else
-  server = http.createServer (request, response) ->
-    chunks = []
-    bytes = 0
 
-    request.addListener 'data', (chunk) ->
-      chunks.push(chunk)
-      bytes += chunk.length
+listener = (request, response) ->
+  chunks = []
+  bytes = 0
 
-    request.getBody = (encoding) ->
-      # Write all chunks into a Buffer
-      body = new Buffer(bytes)
-      offset = 0
-      chunks.forEach((chunk) ->
-        chunk.copy(body, offset, 0, chunk.length)
-        offset += chunk.length
-      )
+  response.setHeader('GenieACS-Version', VERSION)
 
-      # Return encoded (default to UTF8) string
-      return body.toString(encoding || 'utf8', 0, body.byteLength)
+  request.addListener 'data', (chunk) ->
+    chunks.push(chunk)
+    bytes += chunk.length
 
-    request.addListener 'end', () ->
-      body = request.getBody()
-      urlParts = url.parse(request.url, true)
-      if PRESETS_REGEX.test(urlParts.pathname)
-        presetName = querystring.unescape(PRESETS_REGEX.exec(urlParts.pathname)[1])
-        if request.method == 'PUT'
-          preset = JSON.parse(body)
-          preset._id = presetName
+  request.getBody = () ->
+    # Write all chunks into a Buffer
+    body = new Buffer(bytes)
+    offset = 0
+    chunks.forEach((chunk) ->
+      chunk.copy(body, offset, 0, chunk.length)
+      offset += chunk.length
+    )
+    return body
 
-          db.presetsCollection.save(preset, (err) ->
-            db.redisClient.del('presets', 'presets_hash', (err) ->
-              throw err if err
-            )
-            if err
-              response.writeHead(500)
-              response.end(err)
-              return
+  request.addListener 'end', () ->
+    body = request.getBody()
+    urlParts = url.parse(request.url, true)
+    if PRESETS_REGEX.test(urlParts.pathname)
+      presetName = querystring.unescape(PRESETS_REGEX.exec(urlParts.pathname)[1])
+      if request.method == 'PUT'
+        preset = JSON.parse(body)
+        preset._id = presetName
+
+        db.presetsCollection.save(preset, (err) ->
+          return throwError(err, response) if err
+          db.redisClient.del('presets_hash', (err) ->
+            return throwError(err, response) if err
             response.writeHead(200)
             response.end()
           )
-        else if request.method == 'DELETE'
-          db.presetsCollection.remove({'_id' : presetName}, (err, removedCount) ->
-            db.redisClient.del('presets', 'presets_hash', (err) ->
-              throw err if err
-            )
-            if err
-              response.writeHead(500)
-              response.end(err)
-              return
+        )
+      else if request.method == 'DELETE'
+        db.presetsCollection.remove({'_id' : presetName}, (err) ->
+          return throwError(err, response) if err
+          db.redisClient.del('presets_hash', (err) ->
+            return throwError(err, response) if err
             response.writeHead(200)
             response.end()
           )
-        else
-          response.writeHead 405, {'Allow': 'PUT, DELETE'}
-          response.end('405 Method Not Allowed')
-      else if OBJECTS_REGEX.test(urlParts.pathname)
-        objectName = querystring.unescape(OBJECTS_REGEX.exec(urlParts.pathname)[1])
-        if request.method == 'PUT'
-          object = JSON.parse(body)
-          object._id = objectName
+        )
+      else
+        response.writeHead 405, {'Allow': 'PUT, DELETE'}
+        response.end('405 Method Not Allowed')
+    else if OBJECTS_REGEX.test(urlParts.pathname)
+      objectName = querystring.unescape(OBJECTS_REGEX.exec(urlParts.pathname)[1])
+      if request.method == 'PUT'
+        object = JSON.parse(body)
+        object._id = objectName
 
-          db.objectsCollection.save(object, (err) ->
-            db.redisClient.del('objects', 'presets_hash', (err) ->
-              throw err if err
-            )
-            if err
-              response.writeHead(500)
-              response.end(err)
-              return
+        db.objectsCollection.save(object, (err) ->
+          return throwError(err, response) if err
+          db.redisClient.del('objects', 'presets_hash', (err) ->
+            return throwError(err, response) if err
             response.writeHead(200)
             response.end()
           )
-        else if request.method == 'DELETE'
-          db.objectsCollection.remove({'_id' : objectName}, (err, removedCount) ->
-            db.redisClient.del('objects', 'presets_hash', (err) ->
-              throw err if err
-            )
-            if err
-              response.writeHead(500)
-              response.end(err)
-              return
+        )
+      else if request.method == 'DELETE'
+        db.objectsCollection.remove({'_id' : objectName}, (err) ->
+          return throwError(err, response) if err
+          db.redisClient.del('objects', 'presets_hash', (err) ->
+            return throwError(err, response) if err
             response.writeHead(200)
             response.end()
           )
-        else
-          response.writeHead 405, {'Allow': 'PUT, DELETE'}
-          response.end('405 Method Not Allowed')
-      else if TAGS_REGEX.test(urlParts.pathname)
-        r = TAGS_REGEX.exec(urlParts.pathname)
-        deviceId = querystring.unescape(r[1])
-        tag = querystring.unescape(r[2])
-        if request.method == 'POST'
-          db.devicesCollection.update({'_id' : deviceId}, {'$addToSet' : {'_tags' : tag}}, {safe: true}, (err) ->
-            db.redisClient.del("#{deviceId}_presets_hash", (err) ->
-              throw err if err
-            )
-            if err
-              response.writeHead(500)
-              response.end(err)
-              return
-            response.writeHead(200)
-            response.end()
-          )
-        else if request.method == 'DELETE'
-          db.devicesCollection.update({'_id' : deviceId}, {'$pull' : {'_tags' : tag}}, {safe: true}, (err) ->
-            db.redisClient.del("#{deviceId}_presets_hash", (err) ->
-              throw err if err
-            )
-            if err
-              response.writeHead(500)
-              response.end(err)
-              return
-            response.writeHead(200)
-            response.end()
-          )
-        else
-          response.writeHead 405, {'Allow': 'POST, DELETE'}
-          response.end('405 Method Not Allowed')
-      else if DEVICE_TASKS_REGEX.test(urlParts.pathname)
-        if request.method == 'POST'
-          deviceId = querystring.unescape(DEVICE_TASKS_REGEX.exec(urlParts.pathname)[1])
-          if body
-            task = JSON.parse(body)
-            task.device = deviceId
-            db.getAliases((aliases) ->
-              apiFunctions.insertTasks(task, aliases, (err) ->
-                db.redisClient.del("#{deviceId}_presets_hash", (err) ->
-                  throw err if err
-                )
-                if err
-                  response.writeHead(500)
-                  response.end(err)
-                  return
+        )
+      else
+        response.writeHead 405, {'Allow': 'PUT, DELETE'}
+        response.end('405 Method Not Allowed')
+    else if PROVISIONS_REGEX.test(urlParts.pathname)
+      provisionName = querystring.unescape(PROVISIONS_REGEX.exec(urlParts.pathname)[1])
+      if request.method == 'PUT'
+        object = {
+          _id: provisionName
+          script: body.toString()
+        }
+        try
+          new vm.Script("\"use strict\";(function(){\n#{object.script}\n})();")
+        catch err
+          response.writeHead(400)
+          response.end("#{err.name}: #{err.message}")
+          return
 
-                if urlParts.query.connection_request?
-                  apiFunctions.connectionRequest(deviceId, (err) ->
-                    if err
-                      response.writeHead(202)
-                      response.end()
-                    else
-                      apiFunctions.watchTask(task._id, config.DEVICE_ONLINE_THRESHOLD, (err) ->
-                        if err
-                          response.writeHead(202)
-                          response.end()
-                          return
-                        response.writeHead(200)
-                        response.end()
-                      )
-                  )
-                else
-                  response.writeHead(202)
-                  response.end()
+        db.provisionsCollection.save(object, (err) ->
+          return throwError(err, response) if err
+          db.redisClient.del('presets_hash', (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
+      else if request.method == 'DELETE'
+        db.provisionsCollection.remove({'_id' : provisionName}, (err) ->
+          return throwError(err, response) if err
+          db.redisClient.del('presets_hash', (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
+      else
+        response.writeHead 405, {'Allow': 'PUT, DELETE'}
+        response.end('405 Method Not Allowed')
+    else if VIRTUAL_PARAMETERS_REGEX.test(urlParts.pathname)
+      virtualParameterName = querystring.unescape(VIRTUAL_PARAMETERS_REGEX.exec(urlParts.pathname)[1])
+      if request.method == 'PUT'
+        object = {
+          _id: virtualParameterName
+          script: body.toString()
+        }
+        try
+          new vm.Script("\"use strict\";(function(){\n#{object.script}\n})();")
+        catch err
+          response.writeHead(400)
+          response.end("#{err.name}: #{err.message}")
+          return
+
+        db.virtualParametersCollection.save(object, (err) ->
+          return throwError(err, response) if err
+          db.redisClient.del('presets_hash', (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
+      else if request.method == 'DELETE'
+        db.virtualParametersCollection.remove({'_id' : virtualParameterName}, (err) ->
+          return throwError(err, response) if err
+          db.redisClient.del('presets_hash', (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
+      else
+        response.writeHead 405, {'Allow': 'PUT, DELETE'}
+        response.end('405 Method Not Allowed')
+    else if TAGS_REGEX.test(urlParts.pathname)
+      r = TAGS_REGEX.exec(urlParts.pathname)
+      deviceId = querystring.unescape(r[1])
+      tag = querystring.unescape(r[2])
+      if request.method == 'POST'
+        db.devicesCollection.update({'_id' : deviceId}, {'$addToSet' : {'_tags' : tag}}, {safe: true}, (err) ->
+          return throwError(err, response) if err
+          db.redisClient.del("#{deviceId}_presets_hash", (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
+      else if request.method == 'DELETE'
+        db.devicesCollection.update({'_id' : deviceId}, {'$pull' : {'_tags' : tag}}, {safe: true}, (err) ->
+          return throwError(err, response) if err
+          db.redisClient.del("#{deviceId}_presets_hash", (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
+            response.end()
+          )
+        )
+      else
+        response.writeHead 405, {'Allow': 'POST, DELETE'}
+        response.end('405 Method Not Allowed')
+    else if FAULTS_REGEX.test(urlParts.pathname)
+      if request.method == 'DELETE'
+        faultId = querystring.unescape(FAULTS_REGEX.exec(urlParts.pathname)[1])
+        deviceId = faultId.split(':', 1)[0]
+        channel = faultId.slice(deviceId.length + 1)
+        db.faultsCollection.remove({_id : faultId}, (err) ->
+          return throwError(err, response) if err
+          if channel.startsWith('task_')
+            return db.tasksCollection.remove({_id : new mongodb.ObjectID(channel.slice(5))}, (err) ->
+              return throwError(err, response) if err
+              db.redisClient.del("#{deviceId}_tasks", "#{deviceId}_faults", (err) ->
+                return throwError(err, response) if err
+                response.writeHead(200)
+                response.end()
               )
             )
-          else if urlParts.query.connection_request?
-            # no task, send connection request only
-            apiFunctions.connectionRequest(deviceId, (err) ->
-              if err
-                response.writeHead 504
-                response.end()
-                return
-              response.writeHead 200
-              response.end()
-            )
-          else
-            response.writeHead(400)
+
+          db.redisClient.del("#{deviceId}_faults", (err) ->
+            return throwError(err, response) if err
+            response.writeHead(200)
             response.end()
+          )
+        )
+      else
+        response.writeHead 405, {'Allow': 'DELETE'}
+        response.end('405 Method Not Allowed')
+    else if DEVICE_TASKS_REGEX.test(urlParts.pathname)
+      if request.method == 'POST'
+        deviceId = querystring.unescape(DEVICE_TASKS_REGEX.exec(urlParts.pathname)[1])
+        if body.length
+          task = JSON.parse(body)
+          task.device = deviceId
+          apiFunctions.insertTasks(task, (err) ->
+            return throwError(err, response) if err
+            db.redisClient.del("#{deviceId}_tasks", (err) ->
+              return throwError(err, response) if err
+
+              if urlParts.query.connection_request?
+                apiFunctions.connectionRequest(deviceId, (err) ->
+                  if err
+                    response.writeHead(202, err.message, {'Content-Type' : 'application/json'})
+                    response.end(JSON.stringify(task))
+                  else
+                    apiFunctions.watchTask(deviceId, task._id, config.get('DEVICE_ONLINE_THRESHOLD', deviceId), (err, status) ->
+                      return throwError(err, response) if err
+
+                      if status is 'timeout'
+                        response.writeHead(202, 'Task queued but not processed', {'Content-Type' : 'application/json'})
+                        response.end(JSON.stringify(task))
+                      else if status is 'fault'
+                        db.tasksCollection.findOne({_id : task._id}, (err, task) ->
+                          return throwError(err, response) if err
+
+                          response.writeHead(202, 'Task faulted', {'Content-Type' : 'application/json'})
+                          response.end(JSON.stringify(task))
+                        )
+                      else
+                        response.writeHead(200, {'Content-Type' : 'application/json'})
+                        response.end(JSON.stringify(task))
+                    )
+                )
+              else
+                response.writeHead(202, {'Content-Type' : 'application/json'})
+                response.end(JSON.stringify(task))
+            )
+          )
+        else if urlParts.query.connection_request?
+          # no task, send connection request only
+          apiFunctions.connectionRequest(deviceId, (err) ->
+            if err
+              response.writeHead 504
+              response.end("#{err.name}: #{err.message}")
+              return
+            response.writeHead 200
+            response.end()
+          )
+        else
+          response.writeHead(400)
+          response.end()
+      else
+        response.writeHead 405, {'Allow': 'POST'}
+        response.end('405 Method Not Allowed')
+    else if TASKS_REGEX.test(urlParts.pathname)
+      r = TASKS_REGEX.exec(urlParts.pathname)
+      taskId = new mongodb.ObjectID(querystring.unescape(r[1]))
+      action = r[2]
+      if not action? or action is '/'
+        if request.method == 'DELETE'
+          db.tasksCollection.findOne({'_id' : taskId}, {'device' : 1}, (err, task) ->
+            return throwError(err, response) if err
+            deviceId = task.device
+            db.tasksCollection.remove({'_id' : taskId}, (err) ->
+              return throwError(err, response) if err
+              db.faultsCollection.remove({_id : "#{deviceId}:task_#{String(taskId)}"}, (err) ->
+                return throwError(err, response) if err
+                db.redisClient.del("#{deviceId}_tasks", "#{deviceId}_faults", (err) ->
+                  return throwError(err, response) if err
+                  response.writeHead(200)
+                  response.end()
+                )
+              )
+            )
+          )
+        else
+          response.writeHead 405, {'Allow': 'PUT DELETE'}
+          response.end('405 Method Not Allowed')
+      else if action is '/retry'
+        if request.method == 'POST'
+          db.tasksCollection.findOne({'_id' : taskId}, {'device' : 1}, (err, task) ->
+            return throwError(err, response) if err
+            deviceId = task.device
+            db.faultsCollection.remove({_id : "#{deviceId}:task_#{String(taskId)}"}, (err) ->
+              return throwError(err, response) if err
+              db.redisClient.del("#{deviceId}_faults", (err) ->
+                return throwError(err, response) if err
+                response.writeHead(200)
+                response.end()
+              )
+            )
+          )
         else
           response.writeHead 405, {'Allow': 'POST'}
           response.end('405 Method Not Allowed')
-      else if DEVICE_PRESET_REGEX.test(urlParts.pathname)
-        deviceId = querystring.unescape(DEVICE_PRESET_REGEX.exec(urlParts.pathname)[1])
-        if request.method is 'GET'
-          db.getPresetsObjectsAliases((allPresets, allObjects, allAliases) ->
-            presets.getDevicePreset(deviceId, allPresets, allObjects, allAliases, (devicePreset) ->
-              response.writeHead(200, {'Content-Type' : 'application/json'})
-              response.end(JSON.stringify(devicePreset))
-            )
-          )
-        else
-          response.writeHead 405, {'Allow': 'GET'}
-          response.end('405 Method Not Allowed')
-      else if TASKS_REGEX.test(urlParts.pathname)
-        r = TASKS_REGEX.exec(urlParts.pathname)
-        taskId = mongodb.ObjectID(querystring.unescape(r[1]))
-        action = try querystring.unescape(r[2])
-        if not action? or action is '/'
-          if request.method == 'DELETE'
-            db.tasksCollection.remove({'_id' : taskId}, (err, removedCount) ->
-              if err
-                response.writeHead(500)
-                response.end(err)
-                return
-              response.writeHead(200)
-              response.end()
-            )
-          else
-            response.writeHead 405, {'Allow': 'PUT DELETE'}
-            response.end('405 Method Not Allowed')
-        else if action is '/retry'
-          if request.method == 'POST'
-            db.tasksCollection.update({_id : taskId}, {$unset : {fault : 1}, $set : {timestamp : new Date()}}, (err, count) ->
-              # TODO need to invalidate presets hash for the device
-              response.writeHead(200)
-              response.end()
-            )
-          else
-            response.writeHead 405, {'Allow': 'POST'}
-            response.end('405 Method Not Allowed')
-        else
-          response.writeHead(404)
-          response.end()
-      else if FILES_REGEX.test(urlParts.pathname)
-        filename = querystring.unescape(FILES_REGEX.exec(urlParts.pathname)[1])
-        if request.method == 'PUT'
-          metadata = {
-            fileType : request.headers.filetype,
-            manufacturer : request.headers.manufacturer,
-            productClass : request.headers.productclass,
-            version : request.headers.version,
-          }
+      else
+        response.writeHead(404)
+        response.end()
+    else if FILES_REGEX.test(urlParts.pathname)
+      filename = querystring.unescape(FILES_REGEX.exec(urlParts.pathname)[1])
+      if request.method == 'PUT'
+        metadata = {
+          fileType : request.headers.filetype,
+          oui : request.headers.oui,
+          productClass : request.headers.productclass,
+          version : request.headers.version,
+        }
 
-          gs = new mongodb.GridStore(db.mongoDb, filename, 'w', {metadata : metadata})
-          gs.open((err, gs) ->
-            gs.write(request.getBody('binary'), (err, res) ->
-              throw err if err
-              gs.close((err) ->
-                throw err if err
-              )
+        gs = new mongodb.GridStore(db.mongoDb, filename, filename, 'w', {metadata : metadata})
+        gs.open((err, gs) ->
+          gs.write(body, (err, res) ->
+            return throwError(err, response) if err
+            gs.close((err) ->
+              return throwError(err, response) if err
               response.writeHead(201)
               response.end()
             )
           )
-        else if request.method == 'DELETE'
-          mongodb.GridStore.unlink(db.mongoDb, filename, (err) ->
-            response.writeHead(200)
-            response.end()
-          )
-        else
-          response.writeHead 405, {'Allow': 'PUT, DELETE'}
-          response.end('405 Method Not Allowed')
-      else if PING_REGEX.test(urlParts.pathname)
-        host = querystring.unescape(PING_REGEX.exec(urlParts.pathname)[1])
-        require('child_process').exec("ping -w 1 -i 0.2 -c 3 #{host}", (err, stdout, stderr) ->
-          if err?
-            response.writeHead(404, {'Cache-Control' : 'no-cache'})
-            response.end()
-            return
-          response.writeHead(200, {'Content-Type' : 'text/plain', 'Cache-Control' : 'no-cache'})
-          response.end(stdout)
         )
-      else if QUERY_REGEX.test(urlParts.pathname)
-        collectionName = QUERY_REGEX.exec(urlParts.pathname)[1]
-        if request.method not in ['GET', 'HEAD']
-          response.writeHead 405, {'Allow' : 'GET, HEAD'}
-          response.end('405 Method Not Allowed')
-          return
-        collection = db["#{collectionName}Collection"]
-        if not collection?
-          response.writeHead 404
-          response.end('404 Not Found')
-          return
-
-        func = (aliases) ->
-          if urlParts.query.query?
-            q = JSON.parse(urlParts.query.query)
-          else
-            q = {}
-          q = query.expand(q, aliases) if collectionName is 'devices'
-
-          if urlParts.query.projection?
-            projection = {}
-            for p in urlParts.query.projection.split(',')
-              p = p.trim()
-              projection[p] = 1
-              if collectionName is 'devices'
-                for k,v of aliases
-                  if k == p or common.startsWith(k, "#{p}.")
-                    projection[a] = 1 for a in v
-
-          cur = collection.find(q, projection, {batchSize : 50})
-          if urlParts.query.sort?
-            s = JSON.parse(urlParts.query.sort)
-            sort = {}
-            for k,v of s
-              if aliases[k]?
-                sort[a] = v for a in aliases[k]
-              else
-                sort[k] = v
-            cur.sort(sort)
-
-          cur.skip(parseInt(urlParts.query.skip)) if urlParts.query.skip?
-          cur.limit(parseInt(urlParts.query.limit)) if urlParts.query.limit?
-          cur.count((err, total) ->
-            response.writeHead(200, {'Content-Type' : 'application/json', 'total' : total})
-            if request.method is 'HEAD'
-              response.end()
-              return
-            response.write("[\n")
-            i = 0
-            cur.each((err, item) ->
-              if item is null
-                response.end("\n]")
-              else
-                response.write(",\n") if i++
-                apiFunctions.addAliases(item, aliases) if collectionName is 'devices'
-                response.write(JSON.stringify(item))
-            )
-          )
-
-        if collectionName is 'devices'
-          db.getAliases((aliases) ->
-            func(aliases)
-          )
-        else
-          func({})
+      else if request.method == 'DELETE'
+        mongodb.GridStore.unlink(db.mongoDb, filename, (err) ->
+          response.writeHead(200)
+          response.end()
+        )
       else
+        response.writeHead 405, {'Allow': 'PUT, DELETE'}
+        response.end('405 Method Not Allowed')
+    else if PING_REGEX.test(urlParts.pathname)
+      host = querystring.unescape(PING_REGEX.exec(urlParts.pathname)[1])
+      require('child_process').exec("ping -w 1 -i 0.2 -c 3 #{host}", (err, stdout, stderr) ->
+        if err
+          response.writeHead(404, {'Cache-Control' : 'no-cache'})
+          response.end("#{err.name}: #{err.message}")
+          return
+        response.writeHead(200, {'Content-Type' : 'text/plain', 'Cache-Control' : 'no-cache'})
+        response.end(stdout)
+      )
+    else if DELETE_DEVICE_REGEX.test(urlParts.pathname)
+      if request.method isnt 'DELETE'
+        response.writeHead(405, {'Allow' : 'DELETE'})
+        response.end('405 Method Not Allowed')
+        return
+
+      deviceId = querystring.unescape(DELETE_DEVICE_REGEX.exec(urlParts.pathname)[1])
+      apiFunctions.deleteDevice(deviceId, (err) ->
+        return throwError(err, response) if err
+        response.writeHead(200)
+        response.end()
+      )
+    else if QUERY_REGEX.test(urlParts.pathname)
+      collectionName = QUERY_REGEX.exec(urlParts.pathname)[1]
+
+      # Convert to camel case
+      i = collectionName.indexOf('_')
+      while i >= 0
+        ++ i
+        up = if i < collectionName.length then collectionName[i].toUpperCase() else ''
+        collectionName = collectionName.slice(0, i - 1) + up + collectionName.slice(i + 1)
+        i = collectionName.indexOf('_', i)
+
+      if request.method not in ['GET', 'HEAD']
+        response.writeHead 405, {'Allow' : 'GET, HEAD'}
+        response.end('405 Method Not Allowed')
+        return
+      collection = db["#{collectionName}Collection"]
+      if not collection?
         response.writeHead 404
         response.end('404 Not Found')
+        return
 
-  server.listen config.API_PORT, config.API_INTERFACE
+      if urlParts.query.query?
+        try
+          q = JSON.parse(urlParts.query.query)
+        catch err
+          response.writeHead(400)
+          response.end("#{err.name}: #{err.message}")
+          return
+      else
+        q = {}
+
+      switch collectionName
+        when 'devices'
+          q = query.expand(q)
+        when 'tasks'
+          q = query.sanitizeQueryTypes(q, {
+            _id: ((v) -> return new mongodb.ObjectID(v))
+            timestamp: ((v) -> return new Date(v))
+            retries: Number
+          })
+        when 'faults'
+          q = query.sanitizeQueryTypes(q, {
+            timestamp: ((v) -> return new Date(v))
+            retries: Number
+          })
+
+      if urlParts.query.projection?
+        projection = {}
+        for p in urlParts.query.projection.split(',')
+          p = p.trim()
+          projection[p] = 1
+
+      cur = collection.find(q, projection, {batchSize : 50})
+      if urlParts.query.sort?
+        s = JSON.parse(urlParts.query.sort)
+        sort = {}
+        for k,v of s
+          if k[k.lastIndexOf('.') + 1] != '_' and collectionName is 'devices'
+            sort["#{k}._value"] = v
+          else
+            sort[k] = v
+        cur.sort(sort)
+
+      cur.skip(parseInt(urlParts.query.skip)) if urlParts.query.skip?
+      cur.limit(limit = parseInt(urlParts.query.limit)) if urlParts.query.limit?
+      cur.count(false, (err, total) ->
+        response.writeHead(200, {'Content-Type' : 'application/json', 'total' : total})
+        if request.method is 'HEAD'
+          response.end()
+          return
+        response.write("[\n")
+        i = 0
+        cur.each((err, item) ->
+          if err
+            throwError(err, response)
+            return false
+
+          if item?
+            response.write(",\n") if i++
+            response.write(JSON.stringify(item))
+
+          if not item? or (limit? and i >= limit)
+            response.end("\n]")
+        )
+      )
+
+    else
+      response.writeHead 404
+      response.end('404 Not Found')
+
+
+exports.listener = listener

lib/normalize.coffee

@@ -1,109 +0,0 @@
-common = require './common'
-parameters = require './parameters'
-
-
-stringToRegexp = (input) ->
-  if (input.indexOf('*') == -1)
-    return false
-
-  output = input.replace(/[\[\]\\\^\$\.\|\?\+\(\)]/, "\\$&")
-  if output[0] == '*'
-    prefix = '/'
-    output = output.replace(/^\*+/g, '')
-  else
-    prefix = '/^'
-
-  if output[output.length - 1] == '*'
-    suffix = '/'
-    output = output.replace(/\*+$/g, '')
-  else
-    suffix = '$/'
-
-  output = output.replace(/[\*]/, '.*')
-
-  return eval(prefix + output + suffix)
-
-
-normalizers = {}
-
-normalizers.default = (input, normType) ->
-  if normType is 'query' and common.typeOf(input) is common.STRING_TYPE
-    vals = []
-    str = normalizers.string(input, normType)
-    if common.typeOf(str) == common.ARRAY_TYPE
-      vals = vals.concat(str)
-    else
-      vals.push(str)
-
-    f = parseFloat(input)
-    if not isNaN(f)
-      vals.push(f)
-
-    d = new Date(input)
-    if input.length >= 8 and d.getFullYear() > 1983
-      vals.push(d)
-
-    r = stringToRegexp(input)
-    if r isnt false
-      vals.push({'$regex' : r})
-
-    return vals
-  return input
-
-
-FLOAT_REGEX = /^([\-\+]?[0-9\.]+)/
-normalizers.float = (input, normType) ->
-  res = FLOAT_REGEX.exec(input)
-  if res != null
-    parseFloat(res[1])
-  else
-    null
-
-
-normalizers.date = (input, normType) ->
-  return new Date(input)
-
-
-normalizers.string = (input, normType) ->
-  if normType is 'query'
-    if /^\/(.*?)\/(g?i?m?y?)$/.test(input)
-      return [{'$regex' : eval(input)}, input]
-  input
-
-
-colonizeMac = (input) ->
-  if (i = input.indexOf(':')) != -1
-    i = if (i % 2) == 0 then 0 else 1
-    while i < input.length
-      if input[i-1] != ':' and input[i-2] != ':'
-        if input[i] != ':' and i != 0
-          input = input.substring(0, i) + ':' + input.substring(i)
-          ++i
-        ++i
-      ++ i
-    return input
-  else
-    return input.replace(/[0-9A-F]{1}(?!$)/g, '$&:?')
-
-
-normalizers.mac = (input, normType) ->
-  if normType isnt 'query'
-    return input
-
-  input = input.trim().toUpperCase().replace(/[^0-9A-F\*\-\:]/g, '').replace(/[\-\:]/g, ':')
-
-  if input.indexOf('*') != -1
-    input = (colonizeMac(a) for a in input.split('*')).join('*')
-    return stringToRegexp(input, false)
-
-  input = colonizeMac(input)
-
-  if input.length == 17
-    return input
-
-  return {'$regex' : eval('/' + input + '/')}
-
-
-exports.normalize = (path, value, normType) ->
-  type = parameters.getType(path) ? 'default'
-  normalizers[type](value, normType)

lib/parameters.coffee

@@ -1,195 +0,0 @@
-config = require './config'
-db = require './db'
-
-
-PATH_REGEX = /\w+|(\/.+?\/\w*)/g
-
-# Javascript represents numbers as 64-bit floating point numbers
-# Integers up to 2^53 can be encoded precisely
-MAX_PARAMETERS = 52
-
-
-# get the index of the least significant bit
-getLsb = (num) ->
-  i = -1
-  while num > 0
-    ++ i
-    break if num & 1
-    num >>= 1
-  return i
-
-
-createParameterAttributeFinder = (attributeName) ->
-  # compile attributes into data types optimized for quick lookup
-  attributes = []
-  stringSegments = {}
-  regexSegments = {}
-  paramRegex = {}
-
-  for k, v of config.PARAMETERS
-    continue if not v[attributeName]?
-    attributeIndex = attributes.push(v[attributeName]) - 1
-    throw new Error("There cannot be more than #{MAX_PARAMETERS} configured parameters") if attributeIndex >= MAX_PARAMETERS
-    segmentIndex = 0
-    parts = k.match(PATH_REGEX)
-    for i in [0 ... parts.length]
-      part = parts[i]
-      positionHash = i + parts.length * MAX_PARAMETERS
-
-      if part[0] == '/' # this is a regex part
-        j = part.lastIndexOf('/')
-        regExp = new RegExp(part[1...j], part[j+1..])
-        regexSegments[positionHash] ?= 0
-        regexSegments[positionHash] |= Math.pow(2, attributeIndex)
-        paramRegex[attributeIndex] ?= {}
-        paramRegex[attributeIndex][i] = regExp
-      else
-        stringSegments[positionHash] ?= {}
-        stringSegments[positionHash][part] ?= 0
-        stringSegments[positionHash][part] |= Math.pow(2, attributeIndex)
-
-  # Finder function
-  return (param) ->
-    parts = param.split('.')
-    positionHash = parts.length * MAX_PARAMETERS
-
-    stringIndices = 0 | stringSegments[positionHash]?[parts[0]]
-    allIndices = stringIndices | regexSegments[positionHash]
-
-    for i in [1...parts.length]
-      positionHash = i + parts.length * MAX_PARAMETERS
-      strIdx = stringSegments[positionHash]?[parts[i]]
-      rgxIdx = regexSegments[positionHash]
-      stringIndices &= strIdx
-      allIndices &= (strIdx | rgxIdx)
-
-    if stringIndices > 0
-      return attributes[getLsb(stringIndices)]
-
-    attributeIndex = 0
-    while allIndices > 0
-      if allIndices & 1
-        match = true
-        for i, r of paramRegex[attributeIndex]
-          if not r.test(parts[i])
-            match = false
-            break
-
-        if match
-          return attributes[attributeIndex]
-
-      ++ attributeIndex
-      allIndices >>= 1
-
-    return null
-
-
-splitIds = (batches, callback) ->
-  db.devicesCollection.count((err, count) ->
-    throw err if err
-    return callback(null) if count == 0
-    ids = [{}]
-    return callback(ids) if count <= 1000 or batches == 1
-    batchSize = Math.floor(count / batches)
-    for i in [1 ... batches]
-      ids[i] = {}
-      do (i) ->
-        cursor = db.devicesCollection.find({}, {_id:1}).sort({'_id':1}).skip(i * batchSize).limit(1)
-        cursor.nextObject((err, obj) ->
-          ids[i]['$gte'] = obj._id
-          ids[i-1]['$lt'] = obj._id
-          if i == batches - 1
-            callback(ids)
-        )
-  )
-
-
-compileAliases = (callback) ->
-  map = () ->
-    recurse = (partIndex, ref) ->
-      part = parts[partIndex]
-      res = []
-      if part.test?
-        for p of ref
-          if part.test(p)
-            if partIndex == parts.length - 1
-              res.push(p)
-            else
-              for r in recurse(partIndex + 1, ref[p])
-                res.push("#{p}.#{r}")
-      else
-        if ref[part]?
-          if partIndex == parts.length - 1
-            return [part]
-          else
-            for r in recurse(partIndex + 1, ref[part])
-              res.push("#{part}.#{r}")
-
-      return res
-
-    for key, parts of keys
-      for alias in recurse(0, this)
-        emit(alias, key)
-    return
-
-  reduce = (k, v) ->
-    return v[0]
-
-  aliases = {}
-  keys = {}
-  for k, v of config.PARAMETERS
-    alias = v.alias
-    continue if not alias?
-    isStatic = true
-    ar = []
-    for p in k.match(/\w+|(\/.+?\/\w*)/g)
-      if p[0] == '/' # this is a regex
-        isStatic = false
-        i = p.lastIndexOf('/')
-        r = new RegExp(p[1...i], p[i+1..])
-        ar.push(r)
-      else
-        ar.push(p)
-
-    if isStatic
-      aliases[alias] ?= []
-      aliases[alias].push(k)
-    else
-      keys[k] = ar
-
-  counter = 0
-  splitIds(4, (batches) ->
-    if not batches?
-      return callback(null, aliases, config.ALIASES_CACHE)
-
-    for batch in batches
-      options = {
-        out : {inline : 1}
-        jsMode : true
-        sort : {$natural : 1}
-        scope : {keys : keys, flags : {}}
-        query : {_id : batch} if batches.length > 1
-      }
-
-      db.devicesCollection.mapReduce(map, reduce, options, (err, out) ->
-        if err
-          callback?(err)
-          return callback = null
-
-        for o in out
-          alias = config.PARAMETERS[o.value].alias
-          aliases[alias] ?= []
-          id = String(o._id)
-          if id not in aliases[alias]
-            aliases[alias].push(id)
-
-        if ++counter == batches.length
-          callback?(null, aliases, config.ALIASES_CACHE)
-      )
-    return
-  )
-
-
-exports.getType = createParameterAttributeFinder('type')
-exports.getAlias = createParameterAttributeFinder('alias')
-exports.compileAliases = compileAliases

lib/path-set.js

@@ -0,0 +1,179 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+
+class PathSet {
+
+  constructor() {
+    this.lengthIndex = [];
+    this.fragmentIndex = [];
+  }
+
+
+  get depth() {
+    return this.lengthIndex.length;
+  }
+
+
+  add(path) {
+    let p = this.get(path);
+
+    if (p)
+      return p;
+
+    if (!this.lengthIndex[path.length])
+      this.lengthIndex[path.length] = new Set();
+
+    this.lengthIndex[path.length].add(path);
+
+    let wildcard = 0;
+    for (let i = 0; i < path.length; ++ i) {
+      let fragment = path[i];
+      if (fragment === "*")
+        wildcard = wildcard | (1 << i);
+
+      if (!this.fragmentIndex[i])
+        this.fragmentIndex[i] = {};
+
+      if (!this.fragmentIndex[i][fragment])
+        this.fragmentIndex[i][fragment] = new Set();
+
+      this.fragmentIndex[i][fragment].add(path);
+    }
+
+    if (path.wildcard == null) {
+      path.alias = 0;
+      path.wildcard = wildcard;
+      Object.freeze(path);
+    }
+    return path;
+  }
+
+
+  get(pattern) {
+    if (!this.lengthIndex[pattern.length])
+      return null;
+
+    if (this.lengthIndex[pattern.length].has(pattern))
+      return pattern;
+
+    let sets = [];
+
+    sets.push(this.lengthIndex[pattern.length]);
+
+    for (let i = 0; i < pattern.length; ++ i) {
+      if (!this.fragmentIndex[i])
+        return null;
+
+      let fragment = pattern[i];
+
+      if (!this.fragmentIndex[i][fragment])
+        return null;
+
+      sets.push(this.fragmentIndex[i][fragment]);
+    }
+
+    sets.sort(function(a, b) {
+      return a.size - b.size;
+    });
+
+    let smallestSet = sets.shift();
+    for (let path of smallestSet) {
+      for (let s of sets) {
+        if (!s.has(path)) {
+          path = null;
+          break
+        }
+      }
+      if (path)
+        return path;
+    }
+
+    return null;
+  }
+
+
+  find(pattern, superset = false, subset = false, depth = pattern.length) {
+    let res = [];
+    let groups = [[]];
+
+    for (let i = pattern.length; i <= depth; ++ i)
+      if (this.lengthIndex[i])
+        groups[0].push(this.lengthIndex[i]);
+
+    if (!groups[0].length)
+      return res;
+
+    for (let i = 0; i < pattern.length; ++ i) {
+      if (!this.fragmentIndex[i])
+        return res;
+
+      let fragment = pattern[i];
+
+      if (fragment === "*" && subset)
+        continue;
+
+      let g = [];
+
+      if (this.fragmentIndex[i][fragment])
+        g.push(this.fragmentIndex[i][fragment]);
+
+      if (fragment !== "*" && superset && this.fragmentIndex[i]["*"])
+        g.push(this.fragmentIndex[i]["*"]);
+
+      if (!g.length)
+        return res;
+
+      groups.push(g);
+    }
+
+    groups.sort(function(a, b) {
+      return a.reduce(function(prev, cur) {return prev + cur.size;}, 0) -
+        b.reduce(function(prev, cur) {return prev + cur.size;}, 0);
+    });
+
+    let smallestGroup = groups.shift();
+    for (let set of smallestGroup) {
+      for (let path of set) {
+        for (let group of groups) {
+          let found = false;
+          for (let s of group) {
+            if (s.has(path)) {
+              found = true;
+              break;
+            }
+          }
+          if (!found) {
+            path = null;
+            break;
+          }
+        }
+        if (path)
+          res.push(path);
+      }
+    }
+
+    return res;
+  }
+
+}
+
+
+module.exports = PathSet;

lib/presets.coffee

@@ -1,286 +0,0 @@
-config = require './config'
-common = require './common'
-db = require './db'
-mongoQuery = require './mongo-query'
-query = require './query'
-log = require('util').log
-customCommands = require './custom-commands'
-
-
-exports.calculatePresetsHash = (presets, objects) ->
-  crypto = require('crypto')
-  hash = crypto.createHash('md5').update(JSON.stringify(presets) + JSON.stringify(objects)).digest('hex')
-  return hash
-
-
-matchObject = (object, param) ->
-  return false if not object._keys? or object._keys.length == 0
-  for k in object._keys
-    return false if not param[k]?
-    v = common.matchType(param[k]._value, object[k])
-    if param[k]._value != v
-      return false
-  return true
-
-
-exports.getDevicePreset = (deviceId, presets, objects, aliases, callback) ->
-  deviceCustomCommands = customCommands.getDeviceCustomCommands(deviceId)
-  # only fetch relevant params
-  projection = {_id : 1}
-  for p in presets
-    if common.typeOf(p.precondition) is common.STRING_TYPE
-      p.precondition = query.expand(JSON.parse(p.precondition), aliases)
-    else
-      # Accept an object for backward compatiblity
-      p.precondition = query.expand(p.precondition ? {}, aliases)
-    mongoQuery.projection(p.precondition, projection)
-
-    for c in p.configurations
-      switch c.type
-        when 'value', 'age'
-          projection[c.name] = 1
-        when 'custom_command_value', 'custom_command_age'
-          commandfile = c.command.split(' ', 1)
-          if commandfile of deviceCustomCommands
-            projection["_customCommands.#{c.command.split(' ', 1)}"] = 1
-        when 'software_version'
-          projection['InternetGatewayDevice.DeviceInfo.SoftwareVersion'] = 1
-        when 'add_tag', 'delete_tag'
-          projection['_tags'] = 1
-        when 'add_object', 'delete_object'
-          projection[c.name] = 1
-        else
-          throw new Error('Unknown configuration type')
-
-  mongoQuery.optimizeProjection(projection)
-
-  db.devicesCollection.findOne({'_id' : deviceId}, projection, (err, device) ->
-    throw err if err
-    devicePresets = []
-    for p in presets
-      if mongoQuery.test(device, p.precondition)
-        devicePresets.push(p)
-
-    configurations = accumulateConfigurations(devicePresets, objects)
-
-    getParamDetails = (device, path) ->
-      p = common.getParamValueFromPath(device, path)
-      return undefined if not p?
-      details = {}
-      for k,v of p
-        details[k] = v if k[0] == '_'
-      return details
-
-    devicePreset = {}
-    for c in configurations
-      switch c.type
-        when 'value'
-          devicePreset.parameters ?= {}
-          devicePreset.parameters[c.name] ?= {}
-          devicePreset.parameters[c.name].preset ?= {}
-          devicePreset.parameters[c.name].preset.value = c.value
-          devicePreset.parameters[c.name].current = getParamDetails(device, c.name)
-
-        when 'age'
-          devicePreset.parameters ?= {}
-          devicePreset.parameters[c.name] ?= {}
-          devicePreset.parameters[c.name].preset ?= {}
-          devicePreset.parameters[c.name].preset.expiry = parseInt(c.age)
-          devicePreset.parameters[c.name].current = getParamDetails(device, c.name)
-
-        when 'add_tag'
-          devicePreset.tags ?= {}
-          devicePreset.tags[c.tag] = 2 | c.tag in (device._tags ? [])
-
-        when 'delete_tag'
-          devicePreset.tags ?= {}
-          devicePreset.tags[c.tag] = 0 | c.tag in (device._tags ? [])
-
-        when 'add_object'
-          devicePreset.objects ?= {}
-          devicePreset.objects[c.name] ?= {}
-          devicePreset.objects[c.name][c.object] ?= {}
-          devicePreset.objects[c.name][c.object].preset = objects[c.object]
-          continue if not param = common.getParamValueFromPath(device, c.name)
-          devicePreset.objects[c.name][c.object].current ?= {}
-          for k,p of param
-            continue if k[0] == '_'
-            if p._name?
-              if p._name == c.object
-                devicePreset.objects[c.name][c.object].current[k] = p
-            else if matchObject(objects[c.object], p)
-              u = {}
-              u["#{c.name}.#{k}._name"] = c.object
-              db.devicesCollection.update({'_id' : deviceId}, {'$set' : u}, {}, (err, count) ->
-                throw err if err
-              )
-              devicePreset.objects[c.name][c.object].current[k] = p
-
-        when 'delete_object'
-          devicePreset.objects ?= {}
-          devicePreset.objects[c.name] ?= {}
-          devicePreset.objects[c.name][c.object] ?= {}
-          continue if not param = common.getParamValueFromPath(device, c.name)
-          devicePreset.objects[c.name][c.object].current ?= {}
-          for k,p of param
-            continue if k[0] == '_'
-            if p._name?
-              if p._name == c.object
-                devicePreset.objects[c.name][c.object].current[k] = p
-            else if matchObject(objects[c.object], p)
-              u = {}
-              u["#{c.name}.#{k}._name"] = c.object
-              db.devicesCollection.update({'_id' : deviceId}, {'$set' : u}, {}, (err, count) ->
-                throw err if err
-              )
-              devicePreset.objects[c.name][c.object].current[k] = p
-
-        when 'custom_command_value'
-          [filename, commandName] = c.command.split(' ', 2)
-          devicePreset.customCommands ?= {}
-          devicePreset.customCommands[filename] ?= {}
-          devicePreset.customCommands[filename].preset ?= {}
-          devicePreset.customCommands[filename].preset.value = c.value
-          devicePreset.customCommands[filename].preset.valueCommand = c.command
-          continue if not deviceCustomCommands[filename]? or commandName not in deviceCustomCommands[filename]
-          if cmd = common.getParamValueFromPath(device, "_customCommands.#{filename}")
-            devicePreset.customCommands[filename].current = cmd
-          else
-            devicePreset.customCommands[filename].current ?= {}
-
-        when 'custom_command_age'
-          [filename, commandName] = c.command.split(' ', 2)
-          devicePreset.customCommands ?= {}
-          devicePreset.customCommands[filename] ?= {}
-          devicePreset.customCommands[filename].preset ?= {}
-          devicePreset.customCommands[filename].preset.expiry = parseInt(c.age)
-          devicePreset.customCommands[filename].preset.expiryCommand = c.command
-          continue if not deviceCustomCommands[filename]? or commandName not in deviceCustomCommands[filename]
-          if cmd = common.getParamValueFromPath(device, "_customCommands.#{filename}")
-            devicePreset.customCommands[filename].current = cmd
-          else
-            devicePreset.customCommands[filename].current ?= {}
-
-        when 'software_version'
-          devicePreset.softwareVersion ?= {}
-          devicePreset.softwareVersion.preset = c.software_version
-          devicePreset.softwareVersion.current = common.getParamValueFromPath(device, 'InternetGatewayDevice.DeviceInfo.SoftwareVersion')
-
-        else
-          throw new Error('Unknown configuration type')
-
-    callback(devicePreset)
-  )
-
-
-exports.processDevicePreset = (deviceId, devicePreset, callback) ->
-  now = new Date()
-  expiry = config.PRESETS_CACHE_DURATION
-  getParameterValues = []
-  setParameterValues = []
-  addTags = []
-  deleteTags = []
-  taskList = []
-
-  # Parameters
-  for parameterPath, parameterDetails of devicePreset.parameters ? {}
-    continue if not parameterDetails.current?
-    expiry = Math.min(expiry, parameterDetails.preset.expiry) if parameterDetails.preset.expiry?
-    presetValue = parameterDetails.preset.value
-    currentValue = parameterDetails.current._value
-    if presetValue? and currentValue != (presetValue = common.matchType(currentValue, presetValue))
-      setParameterValues.push([parameterPath, presetValue, parameterDetails.current._type])
-    else if parameterDetails.preset.expiry?
-      diff = parameterDetails.preset.expiry - (now - (parameterDetails.current._timestamp ? 0)) / 1000
-      if diff <= config.PRESETS_TIME_PADDING
-        if parameterDetails.current._value?
-          getParameterValues.push(parameterPath)
-        else
-          taskList.push({device : deviceId, name : 'refreshObject', objectName : parameterPath})
-      else
-        expiry = Math.min(expiry, diff)
-
-  # Objects
-  for parameterPath, object of devicePreset.objects ? {}
-    for objectName, objectDetails of object
-      continue if not objectDetails.current?
-      if objectDetails.preset?
-        if Object.keys(objectDetails.current).length > 0
-          for i, obj of objectDetails.current
-            for paramName, paramValue of objectDetails.preset
-              continue if paramName[0] == '_'
-              if obj[paramName]
-                currentValue = obj[paramName]._value
-                presetValue = common.matchType(currentValue, objectDetails.preset[paramName])
-                if currentValue != presetValue
-                  setParameterValues.push(["#{parameterPath}.#{i}.#{paramName}", presetValue, obj[paramName]._type])
-        else
-          vals = []
-          for k,v of objectDetails.preset
-            vals.push([k, v]) if k[0] != '_'
-          taskList.push({device : deviceId, name : 'addObject', objectName : parameterPath, parameterValues : vals, instanceName : objectName})
-      else if Object.keys(objectDetails.current).length > 0
-        for i, obj of objectDetails.current
-          taskList.push({device : deviceId, name : 'deleteObject', objectName : "#{parameterPath}.#{i}"})
-
-  # Tags
-  for tag, v of devicePreset.tags ? {}
-    if v == 1
-      deleteTags.push(tag)
-    else if v == 2
-      addTags.push(tag)
-
-  # Custom commands
-  for commandName, commandDetails of devicePreset.customCommands ? {}
-    continue if not commandDetails.current?
-    expiry = Math.min(expiry, commandDetails.preset.expiry) if commandDetails.preset.expiry?
-    currentValue = commandDetails.current._value
-    presetValue = common.matchType(currentValue, commandDetails.preset.value)
-    if commandDetails.preset.value? and currentValue != presetValue
-      taskList.push({device : deviceId, name : 'customCommand', command : commandDetails.preset.valueCommand})
-    else if commandDetails.preset.expiry?
-      diff = commandDetails.preset.expiry - (now - (commandDetails.current._timestamp ? 0)) / 1000
-      if diff <= config.PRESETS_TIME_PADDING
-        taskList.push({device : deviceId, name : 'customCommand', command : commandDetails.preset.expiryCommand})
-      else
-        expiry = Math.min(expiry, diff)
-
-  if setParameterValues.length
-    taskList.push {device : deviceId, name : 'setParameterValues', parameterValues: setParameterValues, timestamp : now}
-
-  if getParameterValues.length
-    taskList.push {device : deviceId, name : 'getParameterValues', parameterNames: getParameterValues, timestamp : now}
-
-  callback(taskList, addTags, deleteTags, expiry)
-
-
-getObjectHash = (object) ->
-  return object._id if not object._keys? or object._keys.length == 0
-  hash = ''
-  for k in object._keys
-    hash += "#{k}=#{object[k]}"
-  return hash
-
-
-accumulateConfigurations = (presets, objects) ->
-  maxWeights = {}
-  configurations = {}
-  for p in presets
-    for c in p.configurations
-      configurationHash = switch c.type
-        when 'add_tag', 'delete_tag'
-          "tag_#{c.tag}"
-        when 'add_object', 'delete_object'
-          objectHash = getObjectHash(objects[c.object])
-          "object_#{c.name}_#{objectHash}"
-        when 'software_version'
-          'software_version'
-        else
-          "#{c.type}_#{c.name}"
-
-      if not maxWeights[configurationHash]? or p.weight > maxWeights[configurationHash]
-        configurations[configurationHash] = c
-        maxWeights[configurationHash] = p.weight
-
-  configurationsList = (configurations[c] for c of configurations)
-  return configurationsList

lib/query.coffee

@@ -1,15 +1,99 @@
+###
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
+#
+# GenieACS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# GenieACS is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+#
+# This file incorporates work covered by the following copyright and
+# permission notice:
+#
+# Copyright 2013 Fanoos Telecom
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+# THE SOFTWARE.
+###
+
 common = require './common'
-normalize = require './normalize'
+mongodb = require 'mongodb'
 
 
-expandValue = (param, value) ->
+stringToRegexp = (input, flags) ->
+  if (input.indexOf('*') == -1)
+    return false
+
+  output = input.replace(/[\[\]\\\^\$\.\|\?\+\(\)]/, "\\$&")
+  if output[0] == '*'
+    output = output.replace(/^\*+/g, '')
+  else
+    output = '^' + output
+
+  if output[output.length - 1] == '*'
+    output = output.replace(/\*+$/g, '')
+  else
+    output = output + '$'
+
+  output = output.replace(/[\*]/, '.*')
+
+  return new RegExp(output, flags)
+
+
+normalize = (input) ->
+  if common.typeOf(input) is common.STRING_TYPE
+    vals = [input]
+    if (m = /^\/(.*?)\/(g?i?m?y?)$/.exec(input))
+      vals.push({'$regex' : new RegExp(m[1], m[2])})
+
+    f = parseFloat(input)
+    if not isNaN(f)
+      vals.push(f)
+
+    d = new Date(input)
+    if input.length >= 8 and d.getFullYear() > 1983
+      vals.push(d)
+
+    r = stringToRegexp(input)
+    if r isnt false
+      vals.push({'$regex' : r})
+
+    return vals
+  return input
+
+
+expandValue = (value) ->
   if common.typeOf(value) is common.ARRAY_TYPE
     a = []
     for j in value
-      a = a.concat(expandValue(param, j))
+      a = a.concat(expandValue(j))
     return [a]
   else if common.typeOf(value) isnt common.OBJECT_TYPE
-    n = normalize.normalize(param, value, 'query')
+    n = normalize(value)
     if common.typeOf(n) isnt common.ARRAY_TYPE
       return [n]
     else
@@ -21,7 +105,7 @@ expandValue = (param, value) ->
   values = []
   for k,v of value
     keys.push(k)
-    values.push(expandValue(param, v))
+    values.push(expandValue(v))
     indices.push(0)
 
   i = 0
@@ -39,47 +123,249 @@ expandValue = (param, value) ->
   return objs
 
 
-permute = (param, val, aliases) ->
-  keys = []
-  if aliases[param]?
-    for p in aliases[param]
-      keys.push(p)
-  else
-    keys.push(param)
-
+permute = (param, val) ->
   conditions = []
-  for k in keys
-    values = expandValue(k, val)
-    if k[k.lastIndexOf('.') + 1] != '_'
-      k += '._value'
 
-    for v in values
-      obj = {}
-      obj[k] = v
-      conditions.push(obj)
+  values = expandValue(val)
+  if param[param.lastIndexOf('.') + 1] != '_'
+    param += '._value'
+
+  for v in values
+    obj = {}
+    obj[param] = v
+    conditions.push(obj)
 
   return conditions
 
 
-expand = (query, aliases) ->
+expand = (query) ->
   new_query = {}
   for k,v of query
     if k[0] == '$' # operator
       expressions = []
       for e in v
-        expressions.push(expand(e, aliases))
+        expressions.push(expand(e))
       new_query[k] = expressions
     else
-      conditions = permute(k, v, aliases)
+      conditions = permute(k, v)
       if conditions.length > 1
-        if new_query['$and']?
-          new_query['$and'].push({'$or' : conditions})
+        new_query['$and'] ?= []
+        if v?['$ne']?
+          for c in conditions
+            new_query['$and'].push(c)
         else
-          new_query['$and'] = [{'$or' : conditions}]
+          new_query['$and'].push({'$or' : conditions})
       else
-        common.extend(new_query, conditions[0])
+        Object.assign(new_query, conditions[0])
 
   return new_query
 
 
+# Generate parameter projection from a query
+# If second arg is given, it's edited and returned
+queryProjection = (query, proj) ->
+  proj ?= {}
+  for k,v of query
+    if k.charAt(0) == '$' # this is a logical operator
+      for q in v
+        queryProjection(q, proj)
+    else
+      proj[k] = 1
+  return proj
+
+
+testExpressions = (params, expressions, lop) ->
+  for f in expressions
+    res = test(params, f)
+
+    switch lop
+      when '$and'
+        return false if not res
+      when '$or'
+        return true if res
+      when '$nor'
+        return false if res
+
+  return switch lop
+    when '$and' then true
+    when '$or' then false
+    when '$nor' then true
+    else throw new Error('Unknown logical operator')
+
+
+test = (params, query) ->
+  for k,v of query
+    if k.charAt(0) == '$' # this is a logical operator
+      res = testExpressions(params, v, k)
+    else
+      value = params[k]
+
+      if common.typeOf(v) isnt common.OBJECT_TYPE
+        # TODO comparing array to regex, array to array, and object to object
+        if common.typeOf(value) is common.ARRAY_TYPE
+          res = value.indexOf(v) != -1
+        else
+          if common.typeOf(v) is common.REGEXP_TYPE
+            res = v.test(value)
+          else
+            res = v == value
+      else
+        for k2,v2 of v
+          switch k2
+            when '$ne'
+              if common.typeOf(value) is common.ARRAY_TYPE
+                res = value.indexOf(v2) == -1
+              else
+                res = value != v2
+            when '$lt'
+              res = value < v2
+            when '$lte'
+              res = value <= v2
+            when '$gt'
+              res = value > v2
+            when '$gte'
+              res = value >= v2
+            when '$regex'
+              res = v2.test(value)
+            when '$in'
+              throw new Error('Operator not supported')
+            when '$nin'
+              throw new Error('Operator not supported')
+            when '$all'
+              throw new Error('Operator not supported')
+            when '$exists'
+              throw new Error('Operator not supported')
+            else
+              throw new Error('Operator not supported')
+
+    if not res
+      return false
+
+  return true
+
+
+matchType = (src, dst) ->
+  if typeof src is 'string'
+    return String(dst)
+  else if typeof src is 'number'
+    if +dst == parseFloat(dst)
+      return +dst
+    else if not isNaN(Date.parse(dst))
+      return Date.parse(dst)
+  else if typeof src is 'boolean'
+    v = String(dst).trim()
+    if v is 'true' or v is 'TRUE' or v is 'True' or v is '1'
+      return true
+    else if v is 'false' or v is 'FALSE' or v is 'False' or v is '0'
+      return false
+
+  return dst
+
+
+testFilter = (obj, filter) ->
+  for k, v of filter
+    [param, op] = k.split(/([^a-zA-Z0-9\-\_\.].*)/, 2)
+    val = matchType(obj[param], v)
+    switch op
+      when '=', undefined
+        return false if obj[param] != val
+      when '>'
+        return false if not (obj[param] > val)
+      when '>='
+        return false if not (obj[param] >= val)
+      when '<'
+        return false if not (obj[param] < val)
+      when '<='
+        return false if not (obj[param] <= val)
+      when '!'
+        return false if obj[param] == val
+      else
+        throw new Error("Unrecognized operator #{op}")
+
+  return true
+
+
+convertMongoQueryToFilters = (query, filters) ->
+  filters ?= {}
+  for k, v of query
+    if k[0] == '$'
+      if k == '$and'
+        for vv in v
+          convertMongoQueryToFilters(vv, filters)
+      else
+        throw new Error("Operator #{k} not supported")
+    else if k == '_tags'
+      if common.typeOf(v) is common.OBJECT_TYPE
+        for kk, vv of v
+          vv = vv.replace(/[^a-zA-Z0-9\-]+/g, '_')
+          switch kk
+            when '$ne'
+              filters["Tags.#{vv}!"] = true
+            else
+              throw new Error("Operator #{kk} not supported")
+      else
+        v = v.replace(/[^a-zA-Z0-9\-]+/g, '_')
+        filters["Tags.#{v}"] = true
+    else
+      switch k
+        when '_id' then k = 'DeviceID.ID'
+        when '_deviceId._Manufacturer' then k = 'DeviceID.Manufacturer'
+        when '_deviceId._OUI' then k = 'DeviceID.OUI'
+        when '_deviceId._ProductClass' then k = 'DeviceID.ProductClass'
+        when '_deviceId._SerialNumber' then k = 'DeviceID.SerialNumber'
+        when '_lastInform' then k = 'Events.Inform'
+        when '_lastBootstrap' then k = 'Events.0_BOOTSTRAP'
+        when '_lastBoot' then k = 'Events.1_BOOT'
+        when '_registered' then k = 'Events.Registered'
+
+      if common.typeOf(v) is common.OBJECT_TYPE
+        for kk, vv of v
+          switch kk
+            when '$eq'
+              filters[k] = vv
+            when '$ne'
+              filters["#{k}!"] = vv
+            when '$lt'
+              filters["#{k}<"] = vv
+            when '$lte'
+              filters["#{k}<="] = vv
+            when '$gt'
+              filters["#{k}>"] = vv
+            when '$gte'
+              filters["#{k}>="] = vv
+            else
+              throw new Error("Oprator #{kk} not supported")
+      else
+        filters[k] = v
+
+  return filters
+
+
+sanitizeQueryTypes = (query, types) ->
+  for k, v of query
+    if k[0] == '$' # logical operator
+      sanitizeQueryTypes(vv, types) for vv in v
+    else if k of types
+      if common.typeOf(v) is common.OBJECT_TYPE
+        for kk, vv of v
+          switch kk # operator
+            when '$in', '$nin'
+              vv[i] = types[k](vv[i]) for i in [0...vv.length]
+            when '$eq', '$gt', '$gte', '$lt', '$lte', '$ne'
+              v[kk] = types[k](vv)
+            when '$exists', '$type' then
+              # ignore
+            else
+              throw new Error('Operator not supported')
+      else
+        query[k] = types[k](query[k])
+
+  return query
+
+
 exports.expand = expand
+exports.queryProjection = queryProjection
+exports.test = test
+exports.sanitizeQueryTypes = sanitizeQueryTypes
+exports.convertMongoQueryToFilters = convertMongoQueryToFilters
+exports.testFilter = testFilter

lib/sandbox.js

@@ -0,0 +1,328 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const vm = require("vm");
+
+const seedrandom = require("seedrandom");
+
+const common = require("./common");
+const device = require("./device");
+const extensions = require("./extensions");
+const logger = require("./logger");
+
+
+// Used for throwing to exit user script and commit
+const COMMIT = Symbol();
+
+// Used to execute extensions and restart
+const EXT = Symbol();
+
+const context = vm.createContext();
+
+let state;
+
+
+class SandboxDate {
+  constructor(arg) {
+    if (arguments.length)
+      return new (Function.prototype.bind.apply(Date, arguments));
+
+    return new Date(state.sessionContext.timestamp);
+  }
+
+  static now() {
+    return state.sessionContext.timestamp;
+  }
+};
+
+function random() {
+  if (!state.rng)
+    state.rng = seedrandom(sessionContext.timestamp);
+
+  return state.rng();
+}
+
+
+class ParameterWrapper {
+  constructor(path, attributes, unpacked, unpackedRevision) {
+    for (let attrName of attributes) {
+      Object.defineProperty(this, attrName, {get: function() {
+        if (state.uncommitted)
+          commit();
+
+        if (state.revision !== unpackedRevision) {
+          unpackedRevision = state.revision;
+          unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision + 1);
+        }
+
+        if (!unpacked.length)
+          return undefined;
+
+        let attr = state.sessionContext.deviceData.attributes.get(unpacked[0], state.revision + 1)[attrName];
+
+        if (!attr)
+          return undefined;
+
+        return attr[1];
+      }});
+    }
+
+    Object.defineProperty(this, "path", {get: function() {
+      if (state.uncommitted)
+        commit();
+
+      if (state.revision !== unpackedRevision) {
+        unpackedRevision = state.revision;
+        unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision + 1);
+      }
+
+      if (!unpacked.length)
+        return undefined;
+
+      return unpacked[0].join(".");
+    }});
+
+    Object.defineProperty(this, "size", {get: function() {
+      if (state.uncommitted)
+        commit();
+
+      if (state.revision !== unpackedRevision) {
+        unpackedRevision = state.revision;
+        unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision + 1);
+      }
+
+      if (!unpacked.length)
+        return undefined;
+
+      return unpacked.length;
+    }});
+
+    this[Symbol.iterator] = function*() {
+      if (state.uncommitted)
+        commit();
+
+      if (state.revision !== unpackedRevision) {
+        unpackedRevision = state.revision;
+        unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision + 1);
+      }
+
+      for (let p of unpacked)
+        yield new ParameterWrapper(p, attributes, [p], state.revision);
+    }
+  }
+}
+
+
+function declare(path, timestamps, values) {
+  state.uncommitted = true;
+  if (!timestamps)
+    timestamps = {};
+
+  if (!values)
+    values = {};
+
+  let parsedPath = common.parsePath(path);
+
+  let declaration = [parsedPath, 1];
+  let attrs = {};
+
+  for (let attrName in values) {
+    if (!timestamps[attrName])
+      timestamps[attrName] = 1;
+
+    if (attrName === "path")
+      declaration[3] = values.path;
+    else {
+      attrs[attrName] = 1;
+      declaration[4] = declaration[4] || {};
+      if (attrName === "value" && !Array.isArray(values.value))
+        declaration[4].value = [values.value];
+      else
+        declaration[4][attrName] = values[attrName];
+    }
+  }
+
+  for (let attrName in timestamps) {
+    if (attrName === "path")
+      declaration[1] = timestamps.path;
+    else {
+      attrs[attrName] = 1;
+      declaration[2] = declaration[2] || {};
+      declaration[2][attrName] = timestamps[attrName];
+    }
+  }
+
+  state.declarations.push(declaration);
+
+  return new ParameterWrapper(parsedPath, Object.keys(attrs));
+}
+
+
+function clear(path, timestamp, attributes) {
+  state.uncommitted = true;
+
+  if (state.revision === state.maxRevision)
+    state.clear.push([common.parsePath(path), timestamp, attributes]);
+}
+
+
+function commit() {
+  ++ state.revision;
+  state.uncommitted = false;
+
+  if (state.revision === state.maxRevision + 1)
+    throw COMMIT;
+  else if (state.revision > state.maxRevision + 1)
+    throw new Error("Declare function should not be called from within a try/catch block");
+}
+
+
+function ext() {
+  ++ state.extCounter;
+  let extCall = Array.from(arguments).map(String);
+  let key = `${state.revision}: ${JSON.stringify(extCall)}`;
+
+  if (key in state.sessionContext.extensionsCache)
+    return state.sessionContext.extensionsCache[key];
+
+  state.extensions[key] = extCall;
+  throw EXT;
+}
+
+
+function log(msg, meta) {
+  if (state.revision === state.maxRevision && state.extCounter >= 0) {
+    const details = Object.assign({}, meta, {
+      sessionContext: state.sessionContext,
+      message: `Script: ${msg}`
+    });
+
+    delete details.hostname;
+    delete details.pid;
+    delete details.name;
+    delete details.version;
+    delete details.deviceId;
+    delete details.remoteAddress;
+
+    logger.accessInfo(details);
+  }
+}
+
+
+Object.defineProperty(context, "Date", {value: SandboxDate});
+Object.defineProperty(context, "declare", {value: declare});
+Object.defineProperty(context, "clear", {value: clear});
+Object.defineProperty(context, "commit", {value: commit});
+Object.defineProperty(context, "ext", {value: ext});
+Object.defineProperty(context, "log", {value: log});
+
+// Monkey-patch Math.random() to make it deterministic
+context.random = random;
+vm.runInContext("Math.random = random;", context);
+delete context.random;
+
+
+function errorToFault(err) {
+  if (!err)
+    return null;
+
+  // Trim the stack trace at the self-executing anonymous wrapper function
+  let stackTrimIndex = err.stack.match(/\s+at\s[^\s]+\s+at\s[^\s]+\s\(vm\.js.+\)/).index;
+
+  return {
+    code: `script.${err.name}`,
+    message: err.message,
+    detail: {
+      name: err.name,
+      message: err.message,
+      stack: err.stack.slice(0, stackTrimIndex)
+    }
+  }
+}
+
+
+function run(script, globals, sessionContext, startRevision, maxRevision, callback, extCounter = 0) {
+  state = {
+    sessionContext: sessionContext,
+    revision: startRevision,
+    maxRevision: maxRevision,
+    uncommitted: false,
+    declarations: [],
+    extensions: {},
+    clear: [],
+    rng: null,
+    extCounter: extCounter
+  };
+
+  for (let n of Object.keys(context))
+    delete context[n];
+
+  Object.assign(context, globals);
+
+  let ret, status;
+
+  try {
+    ret = script.runInContext(context, {displayErrors: false});
+    status = 0;
+  }
+  catch (err) {
+    if (err === COMMIT)
+      status = 1;
+    else if (err === EXT)
+      status = 2;
+    else
+      return callback(null, errorToFault(err));
+  }
+
+  let _state = state;
+  let args = Array.from(arguments);
+
+  // Need to maintain a counter of ext() calls to avoid calling certain
+  // functions (e.g. log()) multiple times as calling ext() results in
+  // re-execution of script without revision being incremented.
+  args[6] = 0 - (state.extCounter - extCounter);
+
+  let counter = 3;
+  for (let key of Object.keys(_state.extensions)) {
+    counter += 2;
+    extensions.run(_state.extensions[key], function(err, fault, res) {
+      if (err || fault) {
+        if (counter & 1)
+          callback(err, fault);
+        return counter = 0;
+      }
+      _state.sessionContext.extensionsCache[key] = res;
+      if ((counter -= 2) === 1)
+        if (status === 2)
+          return run.apply(null, args);
+        else
+          return callback(null, null, _state.clear, _state.declarations, status === 0, ret);
+    });
+  }
+
+  if ((counter -= 2) === 1)
+    if (status === 2)
+      return run.apply(null, args);
+    else
+      return callback(null, null, _state.clear, _state.declarations, status === 0, ret);
+}
+
+
+exports.run = run;

lib/server.coffee

@@ -0,0 +1,99 @@
+###
+# Copyright 2013-2017  Zaid Abdulla
+#
+# This file is part of GenieACS.
+#
+# GenieACS is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# GenieACS is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+###
+
+service = process.argv[2]
+if not service?
+  throw new Error('Missing argument cwmp, fs, or nbi')
+
+util = require 'util'
+cluster = require 'cluster'
+logger = require './logger'
+logger.init(service, require('../package.json').version)
+config = require './config'
+db = require './db'
+extensions = require './extensions'
+
+networkInterface = config.get("#{service.toUpperCase()}_INTERFACE")
+port = config.get("#{service.toUpperCase()}_PORT")
+useHttps = config.get("#{service.toUpperCase()}_SSL")
+
+listener = require("./#{service}").listener
+onConnection = require("./#{service}").onConnection
+
+server = null
+
+exit = () ->
+  setTimeout(() ->
+    process.exit(1)
+  , 30000).unref()
+
+  cluster.worker?.disconnect()
+
+  if not server
+    db.disconnect()
+    extensions.killAll()
+    logger.close()
+    return
+
+  server.close(() ->
+    db.disconnect()
+    extensions.killAll()
+    logger.close()
+  )
+
+
+process.on('uncaughtException', (err) ->
+  logger.error({
+    message: 'Uncaught exception'
+    exception: err
+  })
+  exit()
+)
+
+
+if useHttps
+  path = require 'path'
+  fs = require 'fs'
+  httpsKey = path.resolve(config.get('CONFIG_DIR'), "#{service}.key")
+  httpsCert = path.resolve(config.get('CONFIG_DIR'), "#{service}.crt")
+  httpsCa = path.resolve(config.get('CONFIG_DIR'), "#{service}.ca-bundle")
+  options = {
+    key: fs.readFileSync(httpsKey),
+    cert: fs.readFileSync(httpsCert)
+  }
+
+  try
+    # Use intermediate certificates if available
+    options.ca = fs.readFileSync(httpsCa).toString()
+      .match(/\-+BEGIN CERTIFICATE\-+[0-9a-zA-Z\+\-\/\=\s]+?\-+END CERTIFICATE\-+/g)
+
+  server = require('https').createServer(options, listener)
+  server.on('secureConnection', onConnection) if onConnection?
+else
+  server = require('http').createServer(listener)
+  server.on('connection', onConnection) if onConnection?
+
+db.connect((err) ->
+  throw err if err
+  server.listen(port, networkInterface)
+)
+
+
+process.on('SIGINT', exit)
+process.on('SIGTERM', exit)

lib/soap.coffee

@@ -1,417 +0,0 @@
-config = require './config'
-libxmljs = require 'libxmljs'
-
-SERVER_NAME = "GenieACS/#{require('../package.json').version}"
-
-NAMESPACES = {
-  '1.0' : {
-    'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-    'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-    'xsd' : 'http://www.w3.org/2001/XMLSchema',
-    'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-    'cwmp' : 'urn:dslforum-org:cwmp-1-0'
-  },
-  '1.1' : {
-    'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-    'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-    'xsd' : 'http://www.w3.org/2001/XMLSchema',
-    'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-    'cwmp' : 'urn:dslforum-org:cwmp-1-1'
-  },
-  '1.2' : {
-    'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-    'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-    'xsd' : 'http://www.w3.org/2001/XMLSchema',
-    'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-    'cwmp' : 'urn:dslforum-org:cwmp-1-2'
-  },
-  '1.3' : {
-    'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-    'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-    'xsd' : 'http://www.w3.org/2001/XMLSchema',
-    'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-    'cwmp' : 'urn:dslforum-org:cwmp-1-2'
-  },
-  '1.4' : {
-    'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-    'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-    'xsd' : 'http://www.w3.org/2001/XMLSchema',
-    'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-    'cwmp' : 'urn:dslforum-org:cwmp-1-3'
-  }
-}
-
-
-# Workaround for devices that don't use correct namespace declarations or prefixes
-if config.IGNORE_XML_NAMESPACES
-  libxmljs.Element.prototype.__find = libxmljs.Element.prototype.find
-  libxmljs.Element.prototype.find = (xpath, namespaces) ->
-    # Modify xpath queries to work regardless of element's namespace
-    p = xpath.replace(/([^\/:]+:)?([^\/]+)(\/|$)/g, '$2$3') \
-      .replace(/([a-zA-Z0-9_-]+)([^\/]*)(\/|$)/g, "*[local-name()='$1']$2$3")
-    libxmljs.Element.prototype.__find.call(this, p, namespaces)
-
-
-# Separation by comma is important as some devices don't comform to standard
-COOKIE_REGEX = /\s*([a-zA-Z0-9\-_]+?)\s*=\s*"?([a-zA-Z0-9\-_]*?)"?\s*(,|;|$)/g
-
-cookieEncode = (value) ->
-  new Buffer(value).toString('base64').replace(/\/|\+|\=/g, (c) -> if c is '/' then '_' else if c is '+' then '-' else '')
-
-cookieDecode = (encodedValue) ->
-  new Buffer(encodedValue, 'base64').toString()
-
-cookiesToObj = (cookieLine) ->
-  cookies = {}
-  while match = COOKIE_REGEX.exec(cookieLine)
-    cookies[cookieDecode(match[1])] = cookieDecode(match[2])
-  return cookies
-
-cookiesToStr = (obj) ->
-  "#{cookieEncode(cn)}=#{cookieEncode(cv)}" for cn, cv of obj
-
-
-event = (xml) ->
-  e.text().trim() for e in xml.find('EventStruct/EventCode')
-
-
-parameterInfoList = (xml) ->
-  infoList = []
-  for e in xml.find('ParameterInfoStruct')
-    name = e.get('Name').text().trim()
-    writable = Boolean(JSON.parse(e.get('Writable').text()))
-    infoList.push([name, writable])
-  return infoList
-
-
-parameterValueList = (xml) ->
-  valueList = []
-  for e in xml.find('ParameterValueStruct')
-    valueType = e.get('Value').attr('type').value().trim()
-    name = e.get('Name').text().trim()
-    value = e.get('Value').text().trim()
-    try
-      switch valueType
-        when 'xsd:boolean'
-          value = Boolean(JSON.parse(value))
-        when 'xsd:unsignedInt'
-          value = JSON.parse(value)
-        when 'xsd:int'
-          value = JSON.parse(value)
-    catch err
-      value = null
-    valueList.push([name, value, valueType])
-  return valueList
-
-
-cpeGetParameterNames = (xml, methodRequest) ->
-  el = xml.node('cwmp:GetParameterNames')
-  el.node('ParameterPath').text(methodRequest.parameterPath)
-  el.node('NextLevel').text(+methodRequest.nextLevel)
-
-
-cpeGetParameterNamesResponse = (xml) ->
-  {parameterList : parameterInfoList(xml.get('ParameterList'))}
-
-
-cpeGetParameterValues = (xml, methodRequest) ->
-  el = xml.node('cwmp:GetParameterValues').node('ParameterNames')
-  el.attr({'soap-enc:arrayType' : "xsd:string[#{methodRequest.parameterNames.length}]"})
-  for p in methodRequest.parameterNames
-    el.node('string').text(p)
-
-
-cpeGetParameterValuesResponse = (xml) ->
-  {parameterList : parameterValueList(xml.get('ParameterList'))}
-
-
-cpeSetParameterValues = (xml, methodRequest) ->
-  el = xml.node('cwmp:SetParameterValues')
-  paramList = el.node('ParameterList')
-  paramList.attr({'soap-enc:arrayType' : "cwmp:ParameterValueStruct[#{methodRequest.parameterList.length}]"})
-  for i in methodRequest.parameterList
-    pvs = paramList.node('ParameterValueStruct')
-    pvs.node('Name').text(i[0])
-    v = pvs.node('Value')
-    v.text(i[1])
-    v.attr({'xsi:type' : i[2]}) if i[2]?
-  el.node('ParameterKey').text(methodRequest.parameterKey ? '')
-
-
-cpeSetParameterValuesResponse = (xml) ->
-  {status : JSON.parse(xml.get('Status').text())}
-
-
-cpeAddObject = (xml, methodRequest) ->
-  el = xml.node('cwmp:AddObject')
-  el.node('ObjectName').text(methodRequest.objectName)
-  el.node('ParameterKey').text(methodRequest.parameterKey ? '')
-
-
-cpeAddObjectResponse = (xml) ->
-  {
-    instanceNumber : parseInt(xml.get('InstanceNumber').text()),
-    status : parseInt(xml.get('Status').text())
-  }
-
-
-cpeDeleteObject = (xml, methodRequest) ->
-  el = xml.node('cwmp:DeleteObject')
-  el.node('ObjectName').text(methodRequest.objectName)
-  el.node('ParameterKey').text(methodRequest.parameterKey ? '')
-
-
-cpeDeleteObjectResponse = (xml) ->
-  {status : parseInt(xml.get('Status').text())}
-
-
-cpeReboot = (xml, methodRequest) ->
-  el = xml.node('cwmp:Reboot')
-  el.node('CommandKey').text(methodRequest.commandKey ? '')
-
-
-cpeRebootResponse = (xml) ->
-  {}
-
-
-cpeFactoryReset = (xml, methodRequest) ->
-  el = xml.node('cwmp:FactoryReset')
-
-
-cpeFactoryResetResponse = (xml, methodRequest) ->
-  {}
-
-
-cpeDownload = (xml, methodRequest) ->
-  el = xml.node('cwmp:Download')
-  el.node('CommandKey').text(methodRequest.commandKey ? '')
-  el.node('FileType').text(methodRequest.fileType)
-  el.node('URL').text(methodRequest.url)
-  el.node('Username').text(methodRequest.username ? '')
-  el.node('Password').text(methodRequest.password ? '')
-  el.node('FileSize').text(methodRequest.fileSize ? '0')
-  el.node('TargetFileName').text(methodRequest.TargetFileName ? '')
-  el.node('DelaySeconds').text(methodRequest.delaySeconds ? '0')
-  el.node('SuccessURL').text(methodRequest.successUrl ? '')
-  el.node('FailureURL').text(methodRequest.failureUrl ? '')
-
-
-cpeDownloadResponse = (xml) ->
-  {status : JSON.parse(xml.get('Status').text())}
-
-
-acsInform = (xml) ->
-  {
-    parameterList : parameterValueList(xml.get('ParameterList')),
-    deviceId : traverseXml(xml.get('DeviceId')),
-    event : event(xml.get('Event')),
-    retryCount : JSON.parse(xml.get('RetryCount').text())
-  }
-
-
-acsInformResponse = (xml, methodResponse) ->
-  xml.node('cwmp:InformResponse').node('MaxEnvelopes').text(1)
-
-
-acsGetRPCMethods = (xml) ->
-  {}
-
-
-acsGetRPCMethodsResponse = (xml, methodResponse) ->
-  el = xml.node('cwmp:GetRPCMethodsResponse').node('MethodList')
-  el.attr({'soap-enc:arrayType' : "xsd:string[#{methodResponse.methodList.length}]"})
-  for m in methodResponse.methodList
-    el.node('string').text(m)
-
-
-acsTransferComplete = (xml) ->
-  {
-    commandKey : xml.get('CommandKey').text(),
-    faultStruct : traverseXml(xml.get('FaultStruct')),
-    startTime : xml.get('StartTime').text(), # TODO convert to datetime
-    completeTime : xml.get('CompleteTime').text() # TODO convert to datetime
-  }
-
-
-acsTransferCompleteResponse = (xml, methodResponse) ->
-  xml.node('cwmp:TransferCompleteResponse').text('')
-
-
-acsRequestDownload = (xml) ->
-  # TODO FileTypeArg
-  {
-    fileType : xml.get('FileType').text()
-  }
-
-
-acsRequestDownloadResponse = (xml, methodResponse) ->
-  xml.node('cwmp:RequestDownloadResponse').text('')
-
-
-traverseXml = (xml) ->
-  obj = {}
-  for n in xml.childNodes()
-    if n.type() == 'element'
-      obj[n.name()] = traverseXml(n)
-
-  if Object.keys(obj).length == 0
-    xml.text()
-  else
-    obj
-
-
-fault = (xml) ->
-  return traverseXml(xml)
-
-
-exports.request = (httpRequest) ->
-  cwmpRequest = {cookies: {}}
-  cwmpRequest.cookies = cookiesToObj(httpRequest.headers.cookie) if httpRequest.headers.cookie
-  cwmpRequest.cwmpVersion = cwmpRequest.cookies.cwmpVersion
-
-  data = httpRequest.getBody()
-
-  if data.length > 0
-    try
-      xml = libxmljs.parseXml data
-    catch err
-      # some devices send invalid utf8 characters
-      xml = libxmljs.parseXml httpRequest.getBody('binary')
-
-    if not cwmpRequest.cwmpVersion?
-      # cwmpVersion doesn't exist in cookies, thus it's an inform request
-      methodElement = xml.get('/soap-env:Envelope/soap-env:Body/*', NAMESPACES['1.0'])
-      switch methodElement.namespace().href()
-        when 'urn:dslforum-org:cwmp-1-0'
-          cwmpRequest.cwmpVersion = '1.0'
-        when 'urn:dslforum-org:cwmp-1-1'
-          cwmpRequest.cwmpVersion = '1.1'
-        when 'urn:dslforum-org:cwmp-1-2'
-          cwmpRequest.sessionTimeout = try xml.get('/soap-env:Envelope/soap-env:Header/cwmp:sessionTimeout', NAMESPACES['1.2']).text() catch then null
-          if cwmpRequest.sessionTimeout?
-            cwmpRequest.cwmpVersion = '1.3'
-          else
-            cwmpRequest.cwmpVersion = '1.2'
-        when 'urn:dslforum-org:cwmp-1-3'
-          cwmpRequest.cwmpVersion = '1.4'
-        else
-          throw new Error('Unrecognized CWMP version')
-
-      cwmpRequest.sessionTimeout ?= try xml.get('/soap-env:Envelope/soap-env:Header/cwmp:sessionTimeout', NAMESPACES[cwmpRequest.cwmpVersion]).text() catch then null
-    else
-      methodElement = xml.get('/soap-env:Envelope/soap-env:Body/cwmp:*', NAMESPACES[cwmpRequest.cwmpVersion])
-
-    cwmpRequest.id = try xml.get('/soap-env:Envelope/soap-env:Header/cwmp:ID', NAMESPACES[cwmpRequest.cwmpVersion]).text() catch then null
-
-    if methodElement?
-      switch methodElement.name()
-        when 'Inform'
-          cwmpRequest.methodRequest = acsInform(methodElement)
-          cwmpRequest.methodRequest.type = 'Inform'
-        when 'GetRPCMethods'
-          cwmpRequest.methodRequest = acsGetRPCMethods(methodElement)
-          cwmpRequest.methodRequest.type = 'GetRPCMethods'
-        when 'TransferComplete'
-          cwmpRequest.methodRequest = acsTransferComplete(methodElement)
-          cwmpRequest.methodRequest.type = 'TransferComplete'
-        when 'RequestDownload'
-          cwmpRequest.methodRequest = acsRequestDownload(methodElement)
-          cwmpRequest.methodRequest.type = 'RequestDownload'
-        when 'GetParameterNamesResponse'
-          cwmpRequest.methodResponse = cpeGetParameterNamesResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'GetParameterNamesResponse'
-        when 'GetParameterValuesResponse'
-          cwmpRequest.methodResponse = cpeGetParameterValuesResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'GetParameterValuesResponse'
-        when 'SetParameterValuesResponse'
-          cwmpRequest.methodResponse = cpeSetParameterValuesResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'SetParameterValuesResponse'
-        when 'AddObjectResponse'
-          cwmpRequest.methodResponse = cpeAddObjectResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'AddObjectResponse'
-        when 'DeleteObjectResponse'
-          cwmpRequest.methodResponse = cpeDeleteObjectResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'DeleteObjectResponse'
-        when 'RebootResponse'
-          cwmpRequest.methodResponse = cpeRebootResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'RebootResponse'
-        when 'FactoryResetResponse'
-          cwmpRequest.methodResponse = cpeFactoryResetResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'FactoryResetResponse'
-        when 'DownloadResponse'
-          cwmpRequest.methodResponse = cpeDownloadResponse(methodElement)
-          cwmpRequest.methodResponse.type = 'DownloadResponse'
-        else
-          throw Error('8000 Method not supported ' + methodElement.name())
-    else
-      faultElement = xml.get('/soap-env:Envelope/soap-env:Body/soap-env:Fault', NAMESPACES[cwmpRequest.cwmpVersion])
-      cwmpRequest.fault = fault(faultElement)
-    
-  return cwmpRequest
-
-
-createSoapEnv = (cwmpVersion) ->
-  xml = libxmljs.Document()
-  env = xml.node('soap-env:Envelope')
-  for prefix, url of NAMESPACES[cwmpVersion]
-    env.defineNamespace prefix, url
-  return env
-
-
-exports.response = (cwmpResponse) ->
-  headers = {
-    'Content-Type' : 'text/xml; charset="utf-8"',
-    'Server' : SERVER_NAME,
-    'SOAPServer' : SERVER_NAME,
-  }
-
-  if cwmpResponse?.methodResponse?
-    env = createSoapEnv(cwmpResponse.cwmpVersion)
-    header = env.node('soap-env:Header')
-    header.node('cwmp:ID').attr({'soap-env:mustUnderstand' : 1}).text(cwmpResponse.id)
-    body = env.node('soap-env:Body')
-    switch cwmpResponse.methodResponse.type
-      when 'InformResponse'
-        cwmpResponse.cookies ?= {}
-        cwmpResponse.cookies.cwmpVersion = cwmpResponse.cwmpVersion
-        acsInformResponse(body, cwmpResponse.methodResponse)
-      when 'GetRPCMethodsResponse'
-        acsGetRPCMethodsResponse(body, cwmpResponse.methodResponse)
-      when 'TransferCompleteResponse'
-        acsTransferCompleteResponse(body, cwmpResponse.methodResponse)
-      when 'RequestDownloadResponse'
-        acsRequestDownloadResponse(body, cwmpResponse.methodResponse)
-      else
-        throw Error("Unknown method response type #{cwmpResponse.methodResponse.type}")
-  else if cwmpResponse?.methodRequest?
-    env = createSoapEnv(cwmpResponse.cwmpVersion)
-    header = env.node('soap-env:Header')
-    header.node('cwmp:ID').attr({'soap-env:mustUnderstand' : 1}).text(cwmpResponse.id)
-    body = env.node('soap-env:Body')
-    switch cwmpResponse.methodRequest.type
-      when 'GetParameterNames'
-        cpeGetParameterNames(body, cwmpResponse.methodRequest)
-      when 'GetParameterValues'
-        cpeGetParameterValues(body, cwmpResponse.methodRequest)
-      when 'SetParameterValues'
-        cpeSetParameterValues(body, cwmpResponse.methodRequest)
-      when 'AddObject'
-        cpeAddObject(body, cwmpResponse.methodRequest)
-      when 'DeleteObject'
-        cpeDeleteObject(body, cwmpResponse.methodRequest)
-      when 'Reboot'
-        cpeReboot(body, cwmpResponse.methodRequest)
-      when 'FactoryReset'
-        cpeFactoryReset(body, cwmpResponse.methodRequest)
-      when 'Download'
-        cpeDownload(body, cwmpResponse.methodRequest)
-      else
-        throw Error("Unknown method request #{cwmpResponse.methodRequest.type}")
-  
-  if cwmpResponse?.cookies? and Object.keys(cwmpResponse.cookies).length > 0
-    headers['Set-Cookie'] = cookiesToStr(cwmpResponse.cookies)
-
-  if env?
-    return {code: 200, headers: headers, data: env.doc().toString()}
-  else
-    return {code: 204, headers: headers, data: ''}

lib/soap.js

@@ -0,0 +1,603 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const libxmljs = require("libxmljs");
+
+const config = require("./config");
+
+const SERVER_NAME = `GenieACS/${require("../package.json").version}`;
+
+const NAMESPACES = {
+  "1.0": {
+    "soap-enc": "http://schemas.xmlsoap.org/soap/encoding/",
+    "soap-env": "http://schemas.xmlsoap.org/soap/envelope/",
+    "xsd": "http://www.w3.org/2001/XMLSchema",
+    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
+    "cwmp": "urn:dslforum-org:cwmp-1-0"
+  },
+  "1.1": {
+    "soap-enc": "http://schemas.xmlsoap.org/soap/encoding/",
+    "soap-env": "http://schemas.xmlsoap.org/soap/envelope/",
+    "xsd": "http://www.w3.org/2001/XMLSchema",
+    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
+    "cwmp": "urn:dslforum-org:cwmp-1-1"
+  },
+  "1.2": {
+    "soap-enc": "http://schemas.xmlsoap.org/soap/encoding/",
+    "soap-env": "http://schemas.xmlsoap.org/soap/envelope/",
+    "xsd": "http://www.w3.org/2001/XMLSchema",
+    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
+    "cwmp": "urn:dslforum-org:cwmp-1-2"
+  },
+  "1.3": {
+    "soap-enc": "http://schemas.xmlsoap.org/soap/encoding/",
+    "soap-env": "http://schemas.xmlsoap.org/soap/envelope/",
+    "xsd": "http://www.w3.org/2001/XMLSchema",
+    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
+    "cwmp": "urn:dslforum-org:cwmp-1-2"
+  },
+  "1.4": {
+    "soap-enc": "http://schemas.xmlsoap.org/soap/encoding/",
+    "soap-env": "http://schemas.xmlsoap.org/soap/envelope/",
+    "xsd": "http://www.w3.org/2001/XMLSchema",
+    "xsi": "http://www.w3.org/2001/XMLSchema-instance",
+    "cwmp": "urn:dslforum-org:cwmp-1-3"
+  }
+};
+
+const LIBXMLJS_PARSE_OPTIONS = {
+  "nocdata": true,
+  "noent": true,
+  "recover": config.get("XML_RECOVER"),
+  "ignore_enc": config.get("XML_IGNORE_ENC")
+};
+
+const LIBXMLJS_SAVE_OPTIONS = {
+  "format": config.get("XML_FORMAT"),
+  "declaration": !config.get("XML_NO_DECL"),
+  "selfCloseEmpty": !config.get("XML_NO_EMPTY")
+};
+
+const XML_IGNORE_NAMESPACE = config.get("XML_IGNORE_NAMESPACE");
+
+if (XML_IGNORE_NAMESPACE) {
+  libxmljs.Element.prototype.__find = libxmljs.Element.prototype.find;
+  libxmljs.Element.prototype.find = function(xpath, namespaces) {
+    const p = xpath.replace(/([^\/:]+:)?([^\/]+)(\/|$)/g, "$2$3").replace(/([a-zA-Z0-9_-]+)([^\/]*)(\/|$)/g, "*[local-name()=\"$1\"]$2$3");
+    return libxmljs.Element.prototype.__find.call(this, p, namespaces);
+  };
+}
+
+let warnings;
+
+
+function parseBool(v) {
+  v = "" + v;
+  if (v === "true" || v === "TRUE" || v === "True" || v === "1")
+    return true;
+  else if (v === "false" || v === "FALSE" || v === "False" || v === "0")
+    return false;
+  else
+    return null;
+}
+
+
+function event(xml) {
+  return xml.find("EventStruct/EventCode").map(e => e.text().trim());
+}
+
+
+function parameterInfoList(xml) {
+  return xml.find("ParameterInfoStruct").map(function(e) {
+    let param = e.get("Name").text();
+    let value = e.get("Writable").text();
+    let parsed = parseBool(value);
+
+    if (parsed === null) {
+      warnings.push({
+        message: "Invalid writable attribute",
+        parameter: param
+      });
+      parsed = false;
+    }
+
+    return [param, parsed];
+  });
+}
+
+
+function parameterValueList(xml) {
+  return xml.find("ParameterValueStruct").map(function(e) {
+    const valueElement = e.get("Value");
+    const valueType = valueElement.attr("type").value();
+    let param = e.get("Name").text();
+    let value = valueElement.text();
+    let parsed = value;
+    if (valueType === "xsd:boolean") {
+      parsed = parseBool(value);
+      if (parsed === null) {
+        warnings.push({
+          message: "Invalid value attribute",
+          parameter: param
+        });
+        parsed = value;
+      }
+    }
+    else if (valueType === "xsd:int" || valueType === "xsd:unsignedInt") {
+      parsed = parseInt(value);
+      if (isNaN(parsed)) {
+        warnings.push({
+          message: "Invalid value attribute",
+          parameter: param
+        });
+        parsed = value;
+      }
+    }
+    else if (valueType === "xsd:dateTime") {
+      parsed = Date.parse(value);
+      if (isNaN(parsed)) {
+        warnings.push({
+          message: "Invalid value attribute",
+          parameter: param
+        });
+        parsed = value;
+      }
+    }
+
+   return [param, parsed, valueType];
+  });
+}
+
+
+function GetParameterNames(xml, methodRequest) {
+  const el = xml.node("cwmp:GetParameterNames");
+  el.node("ParameterPath").text(methodRequest.parameterPath);
+  el.node("NextLevel").text(+methodRequest.nextLevel);
+}
+
+
+function GetParameterNamesResponse(xml) {
+  return {
+    name: "GetParameterNamesResponse",
+    parameterList: parameterInfoList(xml.get("ParameterList"))
+  };
+}
+
+
+function GetParameterValues(xml, methodRequest) {
+  const el = xml.node("cwmp:GetParameterValues").node("ParameterNames");
+  el.attr({
+    "soap-enc:arrayType": `xsd:string[${methodRequest.parameterNames.length}]`
+  });
+
+  for (const p of methodRequest.parameterNames)
+    el.node("string").text(p);
+}
+
+
+function GetParameterValuesResponse(xml) {
+  return {
+    name: "GetParameterValuesResponse",
+    parameterList: parameterValueList(xml.get("ParameterList"))
+  };
+}
+
+
+function SetParameterValues(xml, methodRequest) {
+  const el = xml.node("cwmp:SetParameterValues");
+  const paramList = el.node("ParameterList");
+  paramList.attr({
+    "soap-enc:arrayType": `cwmp:ParameterValueStruct[${methodRequest.parameterList.length}]`
+  });
+
+  for (const p of methodRequest.parameterList) {
+    const pvs = paramList.node("ParameterValueStruct");
+    pvs.node("Name").text(p[0]);
+    pvs.node("Value").attr({"xsi:type": p[2]}).text(p[1]);
+  }
+
+  el.node("ParameterKey").text(methodRequest.parameterKey || "");
+}
+
+
+function SetParameterValuesResponse(xml) {
+  return {
+    name: "SetParameterValuesResponse",
+    status: parseInt(xml.get("Status").text())
+  };
+}
+
+
+function AddObject(xml, methodRequest) {
+  const el = xml.node("cwmp:AddObject");
+  el.node("ObjectName").text(methodRequest.objectName);
+  el.node("ParameterKey").text(methodRequest.parameterKey || "");
+}
+
+
+function AddObjectResponse(xml) {
+  return {
+    name: "AddObjectResponse",
+    instanceNumber: parseInt(xml.get("InstanceNumber").text()),
+    status: parseInt(xml.get("Status").text())
+  };
+}
+
+
+function DeleteObject(xml, methodRequest) {
+  const el = xml.node("cwmp:DeleteObject");
+  el.node("ObjectName").text(methodRequest.objectName);
+  el.node("ParameterKey").text(methodRequest.parameterKey || "");
+}
+
+
+function DeleteObjectResponse(xml) {
+  return {
+    name: "DeleteObjectResponse",
+    status: parseInt(xml.get("Status").text())
+  };
+}
+
+
+function Reboot(xml, methodRequest) {
+  const el = xml.node("cwmp:Reboot");
+  el.node("CommandKey").text(methodRequest.commandKey || "");
+}
+
+
+function RebootResponse(xml) {
+  return {
+    name: "RebootResponse"
+  };
+}
+
+
+function FactoryReset(xml, methodRequest) {
+  xml.node("cwmp:FactoryReset");
+}
+
+
+function FactoryResetResponse(xml, methodRequest) {
+  return {
+    name: "FactoryResetResponse"
+  };
+}
+
+
+function Download(xml, methodRequest) {
+  const el = xml.node("cwmp:Download");
+  el.node("CommandKey").text(methodRequest.commandKey || "");
+  el.node("FileType").text(methodRequest.fileType);
+  el.node("URL").text(methodRequest.url);
+  el.node("Username").text(methodRequest.username || "");
+  el.node("Password").text(methodRequest.password || "");
+  el.node("FileSize").text(methodRequest.fileSize || "0");
+  el.node("TargetFileName").text(methodRequest.targetFileName || "");
+  el.node("DelaySeconds").text(methodRequest.delaySeconds || "0");
+  el.node("SuccessURL").text(methodRequest.successUrl || "");
+  el.node("FailureURL").text(methodRequest.failureUrl || "");
+}
+
+
+function DownloadResponse(xml) {
+  const res = {
+    name: "DownloadResponse",
+    status: parseInt(xml.get("Status").text())
+  };
+
+  if (res.status === 0) {
+    res.startTime = Date.parse(xml.get("StartTime").text());
+    res.completeTime = Date.parse(xml.get("CompleteTime").text());
+  }
+
+  return res;
+}
+
+
+function Inform(xml) {
+  return {
+    name: "Inform",
+    parameterList: parameterValueList(xml.get("ParameterList")),
+    deviceId: {
+      "Manufacturer": xml.get("DeviceId/Manufacturer").text(),
+      "OUI": xml.get("DeviceId/OUI").text(),
+      "ProductClass": xml.get("DeviceId/ProductClass").text(),
+      "SerialNumber": xml.get("DeviceId/SerialNumber").text()
+    },
+    event: event(xml.get("Event")),
+    retryCount: parseInt(xml.get("RetryCount").text())
+  };
+}
+
+
+function InformResponse(xml, methodResponse) {
+  xml.node("cwmp:InformResponse").node("MaxEnvelopes").text(1);
+}
+
+
+function GetRPCMethods(xml) {
+  return {name: "GetRPCMethods"};
+}
+
+
+function GetRPCMethodsResponse(xml, methodResponse) {
+  const el = xml.node("cwmp:GetRPCMethodsResponse").node("MethodList");
+  el.attr({
+    "soap-enc:arrayType": `xsd:string[${methodResponse.methodList.length}]`
+  });
+
+  for (const m of methodResponse.methodList)
+    el.node("string").text(m);
+}
+
+
+function TransferComplete(xml) {
+  return {
+    name: "TransferComplete",
+    commandKey: xml.get("CommandKey").text(),
+    faultStruct: faultStruct(xml.get("FaultStruct")),
+    startTime: Date.parse(xml.get("StartTime").text()),
+    completeTime: Date.parse(xml.get("CompleteTime").text())
+  };
+}
+
+
+function TransferCompleteResponse(xml, methodResponse) {
+  xml.node("cwmp:TransferCompleteResponse").text("");
+}
+
+
+function RequestDownload(xml) {
+  return {
+    name: "RequestDownload",
+    fileType: xml.get("FileType").text()
+  };
+}
+
+
+function RequestDownloadResponse(xml, methodResponse) {
+  xml.node("cwmp:RequestDownloadResponse").text("");
+}
+
+
+function faultStruct(xml) {
+  const f = {
+    faultCode: xml.get("FaultCode").text(),
+    faultString: xml.get("FaultString").text()
+  };
+
+  for (const e of xml.find("SetParameterValuesFault")) {
+    if (!f.setParameterValuesFault)
+      f.setParameterValuesFault = [];
+
+    f.setParameterValuesFault.push({
+      parameterName: e.get("ParameterName").text(),
+      faultCode: e.get("FaultCode").text(),
+      faultString: e.get("FaultString").text()
+    });
+  }
+
+  return f;
+}
+
+
+function fault(xml, cwmpVersion) {
+  return {
+    faultCode: xml.get("faultcode").text(),
+    faultString: xml.get("faultstring").text(),
+    detail: faultStruct(xml.get("detail/cwmp:Fault", NAMESPACES[cwmpVersion]))
+  };
+}
+
+
+function request(httpRequest, cwmpVersion, warn) {
+  warnings = warn;
+
+  const rpc = {
+    cwmpVersion: cwmpVersion
+  };
+
+  const data = httpRequest.getBody();
+
+  if (!data.length)
+    return rpc;
+
+  const xml = libxmljs.parseXml(data, LIBXMLJS_PARSE_OPTIONS);
+
+  let methodElement;
+
+  if (rpc.cwmpVersion) {
+    methodElement = xml.get("/soap-env:Envelope/soap-env:Body/cwmp:*", NAMESPACES[rpc.cwmpVersion]);
+  }
+  else {
+    // cwmpVersion not passed, thus it's an inform request
+    methodElement = xml.get("/soap-env:Envelope/soap-env:Body/*", NAMESPACES["1.0"]);
+    switch (methodElement.namespace().href()) {
+      case "urn:dslforum-org:cwmp-1-0":
+        rpc.cwmpVersion = "1.0";
+        break;
+      case "urn:dslforum-org:cwmp-1-1":
+        rpc.cwmpVersion = "1.1";
+        break;
+      case "urn:dslforum-org:cwmp-1-2":
+        const timeoutElement = xml.get("/soap-env:Envelope/soap-env:Header/cwmp:sessionTimeout", NAMESPACES["1.2"]);
+        if (timeoutElement) {
+          rpc.sessionTimeout = parseInt(timeoutElement.text());
+          rpc.cwmpVersion = "1.3";
+        }
+        else {
+          rpc.cwmpVersion = "1.2";
+        }
+        break;
+      case "urn:dslforum-org:cwmp-1-3":
+        rpc.cwmpVersion = "1.4";
+        break;
+      default:
+        throw new Error("Unrecognized CWMP version");
+    }
+
+    if (!rpc.sessionTimeout) {
+      const timeoutElement = xml.get("/soap-env:Envelope/soap-env:Header/cwmp:sessionTimeout", NAMESPACES[rpc.cwmpVersion]);
+      if (timeoutElement)
+        rpc.sessionTimeout = parseInt(timeoutElement.text());
+    }
+  }
+
+  rpc.id = xml.get("/soap-env:Envelope/soap-env:Header/cwmp:ID", NAMESPACES[rpc.cwmpVersion]).text();
+
+  if (methodElement && !(XML_IGNORE_NAMESPACE && methodElement.name() === "Fault")) {
+    switch (methodElement.name()) {
+      case "Inform":
+        rpc.cpeRequest = Inform(methodElement);
+        break;
+      case "GetRPCMethods":
+        rpc.cpeRequest = GetRPCMethods(methodElement);
+        break;
+      case "TransferComplete":
+        rpc.cpeRequest = TransferComplete(methodElement);
+        break;
+      case "RequestDownload":
+        rpc.cpeRequest = RequestDownload(methodElement);
+        break;
+      case "GetParameterNamesResponse":
+        rpc.cpeResponse = GetParameterNamesResponse(methodElement);
+        break;
+      case "GetParameterValuesResponse":
+        rpc.cpeResponse = GetParameterValuesResponse(methodElement);
+        break;
+      case "SetParameterValuesResponse":
+        rpc.cpeResponse = SetParameterValuesResponse(methodElement);
+        break;
+      case "AddObjectResponse":
+        rpc.cpeResponse = AddObjectResponse(methodElement);
+        break;
+      case "DeleteObjectResponse":
+        rpc.cpeResponse = DeleteObjectResponse(methodElement);
+        break;
+      case "RebootResponse":
+        rpc.cpeResponse = RebootResponse(methodElement);
+        break;
+      case "FactoryResetResponse":
+        rpc.cpeResponse = FactoryResetResponse(methodElement);
+        break;
+      case "DownloadResponse":
+        rpc.cpeResponse = DownloadResponse(methodElement);
+        break;
+      default:
+        throw new Error(`8000 Method not supported ${methodElement.name()}`);
+    }
+  }
+  else {
+    const faultElement = xml.get("/soap-env:Envelope/soap-env:Body/soap-env:Fault", NAMESPACES[rpc.cwmpVersion]);
+    rpc.cpeFault = fault(faultElement, rpc.cwmpVersion);
+  }
+
+  return rpc;
+}
+
+
+function createSoapEnv(cwmpVersion) {
+  const xml = libxmljs.Document();
+  const env = xml.node("soap-env:Envelope");
+
+  for (const prefix of Object.keys(NAMESPACES[cwmpVersion]))
+    env.defineNamespace(prefix, NAMESPACES[cwmpVersion][prefix]);
+
+  return env;
+}
+
+
+function response(rpc) {
+  const headers = {
+    "Server": SERVER_NAME,
+    "SOAPServer": SERVER_NAME
+  };
+
+  if (!rpc)
+    return {code: 204, headers: headers, data: new Buffer(0)};
+
+  const env = createSoapEnv(rpc.cwmpVersion);
+  const header = env.node("soap-env:Header");
+  header.node("cwmp:ID").attr({
+    "soap-env:mustUnderstand": 1
+  }).text(rpc.id);
+
+  const body = env.node("soap-env:Body");
+
+  if (rpc.acsResponse) {
+    switch (rpc.acsResponse.name) {
+      case "InformResponse":
+        InformResponse(body, rpc.acsResponse);
+        break;
+      case "GetRPCMethodsResponse":
+        GetRPCMethodsResponse(body, rpc.acsResponse);
+        break;
+      case "TransferCompleteResponse":
+        TransferCompleteResponse(body, rpc.acsResponse);
+        break;
+      case "RequestDownloadResponse":
+        RequestDownloadResponse(body, rpc.acsResponse);
+        break;
+      default:
+        throw new Error(`Unknown method response type ${rpc.acsResponse.name}`);
+    }
+  }
+  else if (rpc.acsRequest) {
+    switch (rpc.acsRequest.name) {
+      case "GetParameterNames":
+        GetParameterNames(body, rpc.acsRequest);
+        break;
+      case "GetParameterValues":
+        GetParameterValues(body, rpc.acsRequest);
+        break;
+      case "SetParameterValues":
+        SetParameterValues(body, rpc.acsRequest);
+        break;
+      case "AddObject":
+        AddObject(body, rpc.acsRequest);
+        break;
+      case "DeleteObject":
+        DeleteObject(body, rpc.acsRequest);
+        break;
+      case "Reboot":
+        Reboot(body, rpc.acsRequest);
+        break;
+      case "FactoryReset":
+        FactoryReset(body, rpc.acsRequest);
+        break;
+      case "Download":
+        Download(body, rpc.acsRequest);
+        break;
+      default:
+        throw new Error(`Unknown method request ${rpc.acsRequest.name}`);
+    }
+  }
+
+  headers["Content-Type"] = "text/xml; charset=\"utf-8\"";
+  return {
+    code: 200,
+    headers: headers,
+    data: new Buffer(env.doc().toString(LIBXMLJS_SAVE_OPTIONS))
+  };
+}
+
+
+exports.request = request;
+exports.response = response;

lib/tasks.coffee

@@ -1,349 +0,0 @@
-config = require './config'
-common = require './common'
-db = require './db'
-mongodb = require 'mongodb'
-customCommands = require './custom-commands'
-BATCH_SIZE = 16
-
-exports.STATUS_OK = STATUS_OK = 1
-exports.STATUS_FAULT = STATUS_FAULT = 2
-exports.STATUS_COMPLETED = STATUS_COMPLETED = 4
-exports.STATUS_SAVE = STATUS_SAVE = 128 # not mutually exclusive with the rest
-
-
-this.refreshObject = (task, methodResponse, callback) ->
-  task.session = {} if not task.session?
-
-  if not task.session.subtask?
-    path = task.objectName
-    path += '.' if path != '' and not common.endsWith(path, '.')
-    task.session.subtask = {device : task.device, name : 'getParameterNames', parameterPath : path, nextLevel : false}
-
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if task.session.subtask.name is 'getParameterNames'
-    this.getParameterNames(task.session.subtask, methodResponse, (err, status, methodRequest, deviceUpdates) =>
-      if status & STATUS_COMPLETED
-        parameterNames = []
-        for p in deviceUpdates.parameterNames
-          if not common.endsWith(p[0], '.')
-            parameterNames.push(p[0])
-        task.session.subtask = {name : 'getParameterValues', parameterNames : parameterNames}
-        this.getParameterValues(task.session.subtask, {}, (err, status, methodRequest) ->
-          # ignore deviceUpdates returned by firt call to getParameterValues
-          callback(err, status, methodRequest, deviceUpdates)
-        )
-      else if status & STATUS_OK
-        callback(err, STATUS_OK, methodRequest, deviceUpdates)
-      else
-        throw Error('Unexpected subtask status')
-    )
-  else if task.session.subtask.name is 'getParameterValues'
-    if methodResponse.faultcode?
-      task.fault = methodResponse
-      return callback(null, STATUS_FAULT)
-
-    this.getParameterValues(task.session.subtask, methodResponse, (err, status, methodRequest, deviceUpdates) ->
-      callback(err, status, methodRequest, deviceUpdates)
-    )
-  else
-    throw Error('Unexpected subtask name')
-
-
-findMissingParameters = (device, parameterList, prefix) ->
-  missingParameters = []
-  paths = {}
-  for param in parameterList
-    p = param[0]
-    paths[p] = true
-    i = p.indexOf('.')
-    while i != -1
-      paths[p[0...i]] = true
-      i = p.indexOf('.', i + 1)
-
-  recursive = (obj, prefix) ->
-    for k,v of obj
-      continue if k[0] == '_'
-      p = prefix + k
-
-      if paths[p]
-        recursive(v, "#{p}.")
-      else
-        missingParameters.push(p)
-
-  recursive(device, prefix)
-  return missingParameters
-
-
-this.getParameterNames = (task, methodResponse, callback) ->
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if methodResponse.type is 'GetParameterNamesResponse'
-    # Make sure that for each parameters, all its parents are included
-    found = {}
-    for p in methodResponse.parameterList
-      param = p[0]
-      i = param.length
-      while (i = param.lastIndexOf('.', i-1)) > task.parameterPath.length
-        pp = param.slice(0, i + 1)
-        break if found[pp]?
-        found[pp] = 0
-      found[p[0]] = 1
-
-    for k, v of found
-      methodResponse.parameterList.push([k]) if v == 0
-
-    deviceUpdates = {parameterNames : methodResponse.parameterList}
-    path = if common.endsWith(task.parameterPath, '.') then task.parameterPath.slice(0, -1) else task.parameterPath
-    projection = {}
-    projection[path] = 1 if !!task.parameterPath
-
-    # delete nonexisting params
-    db.devicesCollection.findOne({_id : task.device}, projection, (err, device) ->
-      if device
-        root = device
-        rootPath = ''
-        if !!task.parameterPath
-          ps = path.split('.')
-          for p in ps
-            rootPath += "#{p}."
-            root = root[p]
-            break if not root?
-
-        if root
-          deviceUpdates.deletedObjects = findMissingParameters(root, methodResponse.parameterList, rootPath)
-        else
-          # avoid adding and deleting the same param
-          if rootPath isnt task.parameterPath
-            deviceUpdates.deletedObjects = [rootPath.slice(0, -1)]
-
-      # some devices don't return the root object as described in the standard. add manually to update timestamp
-      deviceUpdates.parameterNames.push([task.parameterPath]) if !!task.parameterPath
-
-      callback(null, STATUS_COMPLETED, null, deviceUpdates)
-    )
-  else
-    methodRequest = {
-      type : 'GetParameterNames',
-      parameterPath : task.parameterPath,
-      nextLevel : if task.nextLevel? then task.nextLevel else false
-    }
-    callback(null, STATUS_OK, methodRequest)
-
-
-this.getParameterValues = (task, methodResponse, callback) ->
-  task.session = {} if not task.session?
-
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if not task.session.currentIndex?
-    task.session.currentIndex = 0
-  else if methodResponse.parameterList?
-    task.session.currentIndex = task.session.nextIndex
-
-  task.session.nextIndex = Math.min(task.session.currentIndex + BATCH_SIZE, task.parameterNames.length)
-  names = task.parameterNames.slice(task.session.currentIndex, task.session.nextIndex)
-
-  if methodResponse.type is 'GetParameterValuesResponse'
-    deviceUpdates = {parameterValues : methodResponse.parameterList}
-
-  if names.length == 0
-    callback(null, STATUS_COMPLETED, null, deviceUpdates)
-  else
-    methodRequest = {
-      type : 'GetParameterValues',
-      parameterNames : names
-    }
-    callback(null, STATUS_OK, methodRequest, deviceUpdates)
-
-
-this.setParameterValues = (task, methodResponse, callback) ->
-  task.session = {} if not task.session?
-
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if not task.session.currentIndex?
-    task.session.currentIndex = 0
-  else if methodResponse.type is 'SetParameterValuesResponse'
-    prevValues = task.parameterValues.slice(task.session.currentIndex, task.session.nextIndex)
-    task.session.currentIndex = task.session.nextIndex
-
-  task.session.nextIndex = Math.min(task.session.currentIndex + BATCH_SIZE, task.parameterValues.length)
-  values = task.parameterValues.slice(task.session.currentIndex, task.session.nextIndex)
-
-  if prevValues?
-    deviceUpdates = {parameterValues : prevValues}
-
-  if values.length == 0
-    callback(null, STATUS_COMPLETED, null, deviceUpdates)
-  else
-    callback(null, STATUS_OK, {type : 'SetParameterValues', parameterList : values}, deviceUpdates)
-
-
-this.addObject = (task, methodResponse, callback) ->
-  task.session = {} if not task.session?
-
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  allDeviceUpdates = {}
-
-  if not task.session.instanceNumber?
-    if methodResponse.type is 'AddObjectResponse'
-      status_save = STATUS_SAVE
-      task.session.instanceNumber = methodResponse.instanceNumber
-      task.session.subtask = {device : task.device, name : 'getParameterNames', parameterPath : "#{task.objectName}.#{task.session.instanceNumber}.", nextLevel : false}
-      task.session.appliedParameterValues = []
-      task.session.parameterNames = []
-      allDeviceUpdates.instanceName = [["#{task.objectName}.#{task.session.instanceNumber}", task.instanceName]] if task.instanceName?
-    else
-      callback(null, STATUS_OK, {type : 'AddObject', objectName : "#{task.objectName}."})
-      return
-
-  subtask = () =>
-    switch task.session.subtask.name
-      when 'getParameterNames'
-        this.getParameterNames(task.session.subtask, methodResponse, (err, status, methodRequest, deviceUpdates) =>
-          common.extend(allDeviceUpdates, deviceUpdates)
-          if deviceUpdates and deviceUpdates.parameterNames
-            for p in deviceUpdates.parameterNames
-              task.session.parameterNames.push(p[0]) if not common.endsWith(p[0], '.')
-
-          if status & STATUS_COMPLETED
-            task.session.subtask = {name : 'getParameterValues', parameterNames : task.session.parameterNames}
-            subtask()
-          else if status & STATUS_OK
-            # Use STATUS_SAVE to avoid adding duplicate object in case of error
-            callback(err, STATUS_OK | status_save, methodRequest, allDeviceUpdates)
-          else
-            throw Error('Unexpected subtask status')
-        )
-      when 'getParameterValues'
-        this.getParameterValues(task.session.subtask, methodResponse, (err, status, methodRequest, deviceUpdates) =>
-          common.extend(allDeviceUpdates, deviceUpdates)
-          # if values are given, compare with default values
-          if task.parameterValues?
-            if deviceUpdates and deviceUpdates.parameterValues?
-              for p1 in deviceUpdates.parameterValues
-                for p2 in task.parameterValues
-                  if common.endsWith(p1[0], ".#{p2[0]}")
-                    t = if p2[2] then p2[2] else p1[2]
-                    v = common.matchType(p1[1], p2[1])
-                    # TODO only include if writable
-                    task.session.appliedParameterValues.push([p1[0], v, t])
-
-          if methodResponse.faultcode?
-            task.fault = methodResponse
-            return callback(null, STATUS_FAULT)
-
-          if status & STATUS_COMPLETED and task.session.appliedParameterValues.length > 0
-            task.session.subtask = {name : 'setParameterValues', parameterValues : task.session.appliedParameterValues}
-            subtask()
-          else
-            callback(err, status, methodRequest, allDeviceUpdates)
-        )
-      when 'setParameterValues'
-        this.setParameterValues(task.session.subtask, methodResponse, (err, status, methodRequest, deviceUpdates) =>
-          common.extend(allDeviceUpdates, deviceUpdates)
-          callback(err, status, methodRequest, allDeviceUpdates)
-        )
-  subtask()
-
-
-this.deleteObject = (task, methodResponse, callback) ->
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if methodResponse.type is 'DeleteObjectResponse'
-    callback(null, STATUS_COMPLETED, null, {deletedObjects : [task.objectName]})
-  else
-    methodRequest = {
-      type : 'DeleteObject',
-      objectName : "#{task.objectName}."
-    }
-    callback(null, STATUS_OK, methodRequest)
-
-
-this.reboot = (task, methodResponse, callback) ->
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-  
-  if methodResponse.type isnt 'RebootResponse'
-    callback(null, STATUS_OK, {type : 'Reboot'})
-  else
-    callback(null, STATUS_COMPLETED)
-
-
-this.factoryReset = (task, methodResponse, callback) ->
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if methodResponse.type isnt 'FactoryResetResponse'
-    callback(null, STATUS_OK, {type : 'FactoryReset'})
-  else
-    callback(null, STATUS_COMPLETED)
-
-
-this.download = (task, methodResponse, callback) ->
-  if methodResponse.faultcode?
-    task.fault = methodResponse
-    callback(null, STATUS_FAULT)
-    return
-
-  if methodResponse.type isnt 'DownloadResponse'
-    db.filesCollection.findOne({_id : mongodb.ObjectID(String(task.file))}, (err, file) ->
-      if not file?
-        callback('File not found')
-        return
-      else if err?
-        callback(err)
-        return
-
-      methodRequest = {
-        type : 'Download',
-        fileType : file.metadata.fileType,
-        fileSize : file.length,
-        url : "http://#{config.FILES_IP}:#{config.FILES_PORT}/#{encodeURIComponent(file.filename)}",
-        successUrl : task.successUrl,
-        failureUrl : task.failureUrl
-      }
-      callback(null, STATUS_OK, methodRequest)
-    )
-  else
-    callback(null, STATUS_COMPLETED)
-
-
-this.customCommand = (task, methodResponse, callback) ->
-  # TODO implement timeout
-  customCommands.execute(task.device, task.command, (err, value) ->
-    if err?
-      task.fault = err
-      callback(null, STATUS_FAULT)
-    else
-      commandName = task.command.split(' ', 2)[0]
-      callback(null, STATUS_COMPLETED, null, {customCommands : [[commandName, value]]})
-  )
-
-
-exports.task = (task, methodResponse, callback) ->
-  this[task.name](task, methodResponse, callback)

lib/versioned-map.js

@@ -0,0 +1,255 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const NONEXISTENT = new Object();
+
+
+function compareEquality(a, b) {
+  var t = typeof a;
+  if (t === 'number' || (a == null) || t === 'boolean' || t === 'string' || t === 'symbol')
+    return a === b;
+
+  return JSON.stringify(a) === JSON.stringify(b);
+}
+
+
+class VersionedMap {
+
+  constructor() {
+    this._sizeDiff = [0];
+    this._revision = 0;
+    this.map = new Map();
+  }
+
+
+  get size() {
+    return this.map.size + this._sizeDiff[this.revision];
+  }
+
+
+  get revision() {
+    return this._revision;
+  }
+
+
+  set revision(rev) {
+    for (let i = this._sizeDiff.length; i <= rev; ++ i)
+      this._sizeDiff[i] = this._sizeDiff[i - 1];
+
+    this._revision = rev;
+  }
+
+
+  get(key, rev) {
+    if (rev == null)
+      rev = this._revision
+
+    var v = this.map.get(key);
+    if (!v)
+      return;
+
+    for (let i = Math.min(v.length - 1, rev); i >= 0; -- i)
+      if (i in v)
+        if (v[i] === NONEXISTENT)
+          return undefined;
+        else
+          return v[i];
+  }
+
+
+  has(key, rev) {
+    if (rev == null)
+      rev = this._revision
+
+    var v = this.map.get(key);
+
+    if (v == null)
+      return false;
+
+    for (let i = Math.min(v.length - 1, rev); i >= 0; -- i)
+      if (i in v)
+        if (v[i] === NONEXISTENT)
+          return false;
+        else
+          return true;
+
+    return false;
+  }
+
+
+  set(key, value, rev) {
+    if (rev == null)
+      rev = this._revision;
+
+    var v = this.map.get(key);
+    if (!v) {
+      for (let i = 0; i < rev; ++ i)
+        this._sizeDiff[i] -= 1;
+
+      v = [];
+      v[rev] = value;
+      this.map.set(key, v);
+      return this;
+    }
+
+    var old = v[Math.min(rev, v.length - 1)];
+
+    if (rev < v.length - 1) {
+      if (compareEquality(value, old))
+        return
+      else
+        throw new Error('Cannot modify old revisions');
+    }
+
+    if (old === NONEXISTENT)
+      ++ this._sizeDiff[rev];
+
+    v[rev] = value;
+  }
+
+  delete(key, rev) {
+    if (rev == null)
+      rev = this._revision;
+
+    var v = this.map.get(key);
+    if (!v)
+      return false;
+
+    if (rev < v.length - 1)
+      throw new Error('Cannot modify old revisions');
+
+    var old = v[v.length - 1];
+    if (old === NONEXISTENT)
+      return false;
+
+    -- this._sizeDiff[rev];
+    v[rev] = NONEXISTENT;
+
+    return old;
+  }
+
+
+  getRevisions(key) {
+    var v = this.map.get(key);
+    if (!v)
+      return null;
+
+    var res = {};
+
+    for (let i in v)
+      if (v[i] === NONEXISTENT)
+        res.delete |= 1 << i;
+      else
+        res[i] = v[i];
+
+    return res;
+  }
+
+
+  setRevisions(key, revisions) {
+    var del = 0;
+    var rev = [];
+    var minKey = 999;
+    for (let k in revisions) {
+      if (k === 'delete')
+        del = revisions[k];
+      else {
+        let r = parseInt(k);
+        minKey = Math.min(minKey, r);
+        for (let i = this._sizeDiff.length; i <= r; ++ i)
+          this._sizeDiff[i] = this._sizeDiff[i - 1];
+
+        rev[r] = revisions[k];
+      }
+    }
+
+    for (let i = 0; i < minKey; ++ i)
+      -- this._sizeDiff[i];
+
+    for (let i = 0; del > 0; del >>= 1, ++ i)
+      if (del & 1)
+        rev[i] = NONEXISTENT;
+
+    this.map.set(key, rev);
+  }
+
+
+  getDiff(key) {
+    var revisions = this.map.get(key);
+    if (!revisions)
+      return [];
+
+    var current = revisions[revisions.length - 1];
+    if (current === NONEXISTENT) {
+      if (0 in revisions)
+        return [key, revisions[0]];
+      else
+        return [key];
+    } else if (0 in revisions)
+      return [key, revisions[0], current];
+    else
+      return [key, , current];
+  }
+
+
+  *diff() {
+    for (let pair of this.map) {
+      let current = pair[1][pair[1].length - 1];
+      if (current === NONEXISTENT) {
+        if (0 in pair[1])
+          yield [pair[0], pair[1][0]];
+        else
+          yield [pair[0]];
+      } else if (0 in pair[1])
+        yield [pair[0], pair[1][0], current];
+      else
+        yield [pair[0], , current];
+    }
+  }
+
+
+  collapse(revision) {
+    if (this._sizeDiff.length <= revision)
+      return;
+
+    this._sizeDiff[revision] = this._sizeDiff[this._sizeDiff.length - 1];
+    this._sizeDiff.splice(revision + 1, this._sizeDiff.length);
+
+    for (let pair of this.map) {
+      let k = pair[0];
+      let v = pair[1];
+      let last = v[v.length - 1];
+      if (last === NONEXISTENT) {
+        v.splice(revision, v.length);
+        if (v.some(function(val) { return true; }))
+          v[revision] = NONEXISTENT;
+        else
+          this.map.delete(k);
+      }
+      else {
+        v[revision] = last;
+        v.splice(revision + 1, v.length);
+      }
+    }
+  }
+
+}
+
+module.exports = VersionedMap;

package.json

@@ -1,40 +1,41 @@
 {
   "name": "genieacs",
-  "version": "0.9.9",
-  "description": "A blazing fast TR-069 auto configuration server (ACS) built with Node.js, Redis, and MongoDB",
+  "version": "1.1.1",
+  "description": "A TR-069 Auto Configuration Server (ACS)",
   "repository": {
     "type": "git",
     "url": "https://github.com/zaidka/genieacs.git"
   },
+  "homepage": "https://genieacs.com",
   "keywords": [
     "TR-069",
     "CWMP",
     "ACS"
   ],
-  "author": "Zaid Abdulla",
-  "license": "AGPL3",
-  "readmeFilename": "README.md",
+  "author": "Zaid Abdulla <zaid@genieacs.com>",
+  "license": "AGPL-3.0",
   "preferGlobal": true,
   "directories": {
-    "test": "test",
     "bin": "bin"
   },
   "dependencies": {
-    "libxmljs": "~0.9.0",
-    "redis": "~0.10.1",
-    "hiredis": "~0.1.16",
-    "mongodb": "~1.4.4"
+    "hiredis": "^0.5.0",
+    "later": "^1.2.0",
+    "libxmljs": "^0.18.4",
+    "mongodb": "^2.2.24",
+    "redis": "^2.6.5",
+    "seedrandom": "^2.4.2"
   },
   "devDependencies": {
-    "coffee-script": "~1.7.1"
+    "coffeescript": "^1.12.4"
   },
   "engines": {
-    "node": ">=0.10.0 <0.12"
+    "node": ">=6.9.2 <8.0"
   },
   "scripts": {
     "prepublish": "npm run compile",
-    "compile": "coffee -cb lib/*.coffee test/*.coffee tools/*.coffee",
-    "configure": "cp ./config/config-sample.json ./config/config.json; cp ./config/parameters-sample.json ./config/parameters.json; cp ./config/custom_commands-sample.json ./config/custom_commands.json; cp ./config/auth-sample.js ./config/auth.js; cp ./config/httpscert-sample.key ./config/httpscert.key; cp ./config/httpscert-sample.crt ./config/httpscert.crt",
+    "compile": "coffee -cb lib/*.coffee",
+    "configure": "cp -n ./config/config-sample.json ./config/config.json",
     "install": "npm run configure",
     "test": "node test"
   }

test/methods.coffee

@@ -1,138 +0,0 @@
-common = require '../lib/common'
-
-NAMESPACES = {
-  'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-  'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-  'xsd' : 'http://www.w3.org/2001/XMLSchema',
-  'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-  'cwmp' : 'urn:dslforum-org:cwmp-1-0',
-}
-
-
-exports.inform = (device, xmlOut, callback) ->
-  body = xmlOut.root().childNodes()[1]
-  inform = body.node('cwmp:Inform')
-
-  deviceId = inform.node('DeviceId')
-  deviceId.node('Manufacturer', device['InternetGatewayDevice.DeviceInfo.Manufacturer'][1])
-  deviceId.node('OUI', device['InternetGatewayDevice.DeviceInfo.ManufacturerOUI'][1])
-  deviceId.node('ProductClass', device['InternetGatewayDevice.DeviceInfo.ProductClass'][1])
-  deviceId.node('SerialNumber', device['InternetGatewayDevice.DeviceInfo.SerialNumber'][1])
-
-  eventStruct = inform.node('Event').attr({'soap-enc:arrayType' : 'cwmp:EventStruct[1]'}).node('EventStruct')
-  eventStruct.node('EventCode', '2 PERIODIC')
-  eventStruct.node('CommandKey')
-
-  inform.node('MaxEnvelopes', '1')
-  inform.node('CurrentTime', new Date().toISOString())
-  inform.node('RetryCount', '0')
-
-  parameterList = inform.node('ParameterList').attr({'soap-enc:arrayType' : 'cwmp:ParameterValueStruct[7]'})
-  for p in [
-      'InternetGatewayDevice.DeviceInfo.SpecVersion',
-      'InternetGatewayDevice.DeviceInfo.HardwareVersion',
-      'InternetGatewayDevice.DeviceInfo.SoftwareVersion',
-      'InternetGatewayDevice.DeviceInfo.ProvisioningCode',
-      'InternetGatewayDevice.ManagementServer.ParameterKey',
-      'InternetGatewayDevice.ManagementServer.ConnectionRequestURL',
-      'InternetGatewayDevice.WANDevice.1.WANConnectionDevice.1.WANIPConnection.1.ExternalIPAddress']
-    parameterValueStruct = parameterList.node('ParameterValueStruct')
-    parameterValueStruct.node('Name', p)
-    parameterValueStruct.node('Value', device[p][1]).attr({'xsi:type' : "#{device[p][2]}"})
-
-  callback(xmlOut)
-
-
-exports.GetParameterNames = (device, xmlIn, xmlOut, callback) ->
-  parameterNames = Object.keys(device).sort()
-  parameterPath = xmlIn.get('/soap-env:Envelope/soap-env:Body/cwmp:GetParameterNames/ParameterPath', NAMESPACES).text()
-  nextLevel = !!eval(xmlIn.get('/soap-env:Envelope/soap-env:Body/cwmp:GetParameterNames/NextLevel', NAMESPACES).text())
-
-  parameterList = []
-  if nextLevel
-    for p in parameterNames
-      if common.startsWith(p, parameterPath) and p.length != parameterPath.length
-        i = p.indexOf('.', parameterPath.length)
-        if i == -1 or i == p.length - 1
-          parameterList.push(p)
-  else
-    for p in parameterNames
-      if common.startsWith(p, parameterPath)
-        parameterList.push(p)
-
-  getParameterNamesResponseNode = xmlOut.root().childNodes()[1].node('cwmp:GetParameterNamesResponse')
-  parameterListNode = getParameterNamesResponseNode.node('ParameterList')
-  parameterListNode.attr({'soap-enc:arrayType' : "cwmp:ParameterInfoStruct[#{parameterList.length}]"})
-  for p in parameterList
-    parameterInfoStructNode = parameterListNode.node('ParameterInfoStruct')
-    parameterInfoStructNode.node('Name', p)
-    parameterInfoStructNode.node('Writable', String(device[p][0]))
-
-  callback(xmlOut)
-
-
-exports.GetParameterValues = (device, xmlIn, xmlOut, callback) ->
-  parameterNames = xmlIn.find('/soap-env:Envelope/soap-env:Body/cwmp:GetParameterValues/ParameterNames/*', NAMESPACES)
-
-  parameterList = xmlOut.root().childNodes()[1].node('cwmp:GetParameterValuesResponse').node('ParameterList')
-  parameterList.attr({'soap-enc:arrayType' : "cwmp:ParameterValueStruct[#{parameterNames.length}]"})
-
-  for p in parameterNames
-    name = p.text()
-    value = device[name][1]
-    type = device[name][2]
-    valueStruct = parameterList.node('ParameterValueStruct')
-    valueStruct.node('Name', name)
-    valueStruct.node('Value', device[name][1]).attr({'xsi:type' : "#{type}"})
-
-  callback(xmlOut)
-
-
-exports.SetParameterValues = (device, xmlIn, xmlOut, callback) ->
-  parameterValues = xmlIn.find('/soap-env:Envelope/soap-env:Body/cwmp:SetParameterValues/ParameterList/*', NAMESPACES)
-
-  for p in parameterValues
-    name = p.get('Name').text()
-    value = p.get('Value')
-    device[name][1] = value.text()
-    device[name][2] = value.attr('xsi:type')
-
-  responseNode = xmlOut.root().childNodes()[1].node('cwmp:SetParameterValuesResponse')
-  responseNode.node('Status', '0')
-
-  callback(xmlOut)
-
-
-exports.AddObject = (device, xmlIn, xmlOut, callback) ->
-  objectName = xmlIn.get('/soap-env:Envelope/soap-env:Body/cwmp:AddObject/ObjectName', NAMESPACES).text()
-
-  parameters = []
-  instances = {}
-  instanceNumber = 1
-  while device[objectName + instanceNumber + '.']?
-    instanceNumber += 1
-
-  for p in Object.keys(device).sort()
-    if common.startsWith(p, objectName) and p.length > objectName.length
-      n = objectName + instanceNumber + p.slice(p.indexOf('.', objectName.length))
-      if not device[n]?
-        device[n] = [device[p][0], '', device[p][2]]
-
-  responseNode = xmlOut.root().childNodes()[1].node('cwmp:AddObjectResponse')
-  responseNode.node('InstanceNumber', String(instanceNumber))
-  responseNode.node('Status', '0')
-
-  callback(xmlOut)
-
-
-exports.DeleteObject = (device, xmlIn, xmlOut, callback) ->
-  objectName = xmlIn.get('/soap-env:Envelope/soap-env:Body/cwmp:DeleteObject/ObjectName', NAMESPACES).text()
-
-  for p in Object.keys(device)
-    if common.startsWith(p, objectName)
-      delete device[p]
-
-  responseNode = xmlOut.root().childNodes()[1].node('cwmp:DeleteObjectResponse')
-  responseNode.node('Status', '0')
-
-  callback(xmlOut)

test/test.coffee

@@ -1,156 +0,0 @@
-http = require 'http'
-libxmljs = require 'libxmljs'
-deviceTemplate = require './huawei_bm632w'
-methods = require './methods'
-util = require 'util'
-
-NAMESPACES = {
-  'soap-enc' : 'http://schemas.xmlsoap.org/soap/encoding/',
-  'soap-env' : 'http://schemas.xmlsoap.org/soap/envelope/',
-  'xsd' : 'http://www.w3.org/2001/XMLSchema',
-  'xsi' : 'http://www.w3.org/2001/XMLSchema-instance',
-  'cwmp' : 'urn:dslforum-org:cwmp-1-0',
-}
-
-
-createSoapDocument = (id) ->
-  xml = libxmljs.Document()
-  env = xml.node('soap-env:Envelope')
-  for prefix, url of NAMESPACES
-    env.defineNamespace prefix, url
-
-  header = env.node('soap-env:Header')
-  header.node('cwmp:ID').attr({'soap-env:mustUnderstand' : 1}).text(id)
-  body = env.node('soap-env:Body')
-  return xml
-
-
-createDevice = (serial) ->
-  device = JSON.parse(JSON.stringify(deviceTemplate))
-  device['InternetGatewayDevice.DeviceInfo.SerialNumber'][1] = serial
-  return device
-
-
-sendRequest = (device,xml, callback) ->
-  headers = {}
-  if xml?
-    body = xml.toString()
-  else
-    body = ''
-
-  headers['Content-Length'] = body.length
-  headers['Content-Type'] = 'text/xml; charset="utf-8"'
-
-  if device.Cookies?
-    headers['Cookie'] = device['Cookies']
-
-  options = {
-    hostname: 'localhost',
-    port: 7547,
-    path: '/',
-    method: 'POST',
-    headers: headers
-  }
-
-  request = http.request(options, (response) ->
-    chunks = []
-    bytes = 0
-    response.on('data', (chunk) ->
-      chunks.push(chunk)
-      bytes += chunk.length
-    )
-    response.on('end', () ->
-      body = new Buffer(bytes)
-      offset = 0
-      chunks.forEach((chunk) ->
-        chunk.copy(body, offset, 0, chunk.length)
-        offset += chunk.length
-      )
-
-      if +response.headers['Content-Length'] > 0 or body.length > 0
-        xml = libxmljs.parseXml(body)
-      else
-        xml = null
-
-      if response.headers['set-cookie']?
-        # TODO set individual cookies
-        device['Cookies'] = [true, response.headers['set-cookie'], 'string']
-
-      callback(xml)
-    )
-  )
-  request.end(body)
-
-
-startSession = (device) ->
-  xmlOut = createSoapDocument(0)
-  methods.inform(device, xmlOut, (xml) ->
-    sendRequest(device, xml, (xml) ->
-      if not xml?
-        console.log('No inform response, trying again in 10 seconds.')
-        setTimeout(() ->
-          startSession(device)
-        , 10000)
-        return
-      sendRequest(device, null, (xml) ->
-        handleMethod(device, xml)
-      )
-    )
-  )
-
-
-createFaultResponse = (xmlOut, code, message) ->
-  body = xmlOut.root().childNodes()[1]
-
-  soapFault = body.node('soap-env:Fault')
-  soapFault.node('faultcode').text('Client')
-  soapFault.node('faultstring').text('CWMP fault')
-
-  fault = soapFault.node('detail').node('cwmp:Fault')
-  fault.node('FaultCode').text(code)
-  fault.node('FaultString').text(message)
-
-
-handleMethod = (device, xml) ->
-  if not xml?
-    setTimeout(() ->
-      startSession(device)
-    , 1000 * parseInt(device['InternetGatewayDevice.ManagementServer.PeriodicInformInterval'][1]))
-    return
-
-  requestId = xml.get('/soap-env:Envelope/soap-env:Header/cwmp:ID', NAMESPACES).text()
-
-  xmlOut = createSoapDocument(requestId)
-
-  element = xml.get('/soap-env:Envelope/soap-env:Body/cwmp:*', NAMESPACES)
-  method = methods[element.name()]
-
-  if method?
-    methods[element.name()](device, xml, xmlOut, (xml) ->
-      sendRequest(device, xml, (xml) ->
-        handleMethod(device, xml)
-      )
-    )
-  else
-    createFaultResponse(xmlOut, 9000, 'Method not supported')
-    sendRequest(device, xmlOut, (xml) ->
-      handleMethod(device, xml)
-    )
-
-
-cluster = require 'cluster'
-
-NUM_WORKERS = 1
-NUM_DEVICES = 1
-
-if cluster.isMaster
-  for i in [0 ... NUM_WORKERS]
-    cluster.fork({SERIAL_PREFIX : i})
-else
-  for i in [0...NUM_DEVICES]
-    serial = process.env.SERIAL_PREFIX + ("00000#{i}".slice(-6))
-    device = createDevice(serial)
-    do (device) ->
-      setTimeout(() ->
-        startSession(device)
-      , i * 100)

tools/dump-data-model

@@ -0,0 +1,91 @@
+#!/usr/bin/env node
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const fs = require("fs");
+const path = require("path");
+
+const outputFilename = path.resolve(`data_model_${process.argv[2]}.json`);
+process.chdir(path.resolve(path.dirname(fs.realpathSync(__filename)), ".."));
+
+const db = require("../lib/db");
+
+const dataModel = {};
+
+
+function recursive(object, path) {
+  if (!path)
+    path = "";
+
+  for (let key in object) {
+    if (key[0] === "_" || !object[key])
+      continue;
+
+    let isObject = object[key]["_object"];
+    let isValue = "_value" in object[key];
+
+    if ((isObject && isValue) || (!isObject && !isValue))
+      throw new Error("Unexpected object");
+
+    if (isValue) {
+      let value = object[key]["_value"];
+      if (typeof value === "boolean")
+        value = +value;
+
+      let values = [object[key]["_writable"] || false, value.toString()];
+      if (object[key]["_type"])
+        values.push(object[key]["_type"]);
+
+      dataModel[`${path}${key}`] = values;
+    }
+
+    if (isObject) {
+      let values = [object[key]["_writable"] || false];
+      dataModel[`${path}${key}.`] = values;
+      recursive(object[key], `${path}${key}.`);
+    }
+  }
+}
+
+
+db.connect(function(err) {
+  if (err)
+    throw err;
+
+  return db.devicesCollection.findOne({
+    _id: process.argv[2]
+  }, {}, function(err, device) {
+    if (err)
+      throw err;
+
+    if (!device)
+      throw new Error("Device not found");
+
+    recursive(device);
+
+    fs.writeFile(outputFilename, JSON.stringify(dataModel, null, 2), function() {
+      if (err)
+        throw err;
+
+      console.log(`${outputFilename} saved`);
+      process.exit(0);
+    });
+  });
+});

tools/genie_auth.pl

@@ -1,27 +0,0 @@
-#!/usr/bin/perl
-use Digest::MD5 qw(md5_base64);
-use URI::Escape;
-print "Enter the details *exactly* as sent in device TR-069 informs (case sensitive).\n";
-print 'Serial number: ';
-chomp($serial = <>);
-$serial = uri_escape($serial, '^A-Za-z0-9_');
-
-print 'OUI: ';
-chomp($oui = <>);
-$oui = uri_escape($oui, '^A-Za-z0-9_');
-
-print 'Product class (leave blank if not available): ';
-chomp($product_class = <>);
-$product_class = uri_escape($product_class, '^A-Za-z0-9_');
-
-if ($product_class eq '') {
-  $device_id = "$oui-$serial"
-}
-else {
-  $device_id = "$oui-$product_class-$serial"
-}
-
-print "ACS username: $device_id\n";
-print 'ACS password: ' . md5_base64("${device_id} Open Sesame") . "\n";
-print "Connection request username: $device_id\n";
-print 'Connection request password: ' . md5_base64("${device_id} Sesame, open") . "\n";

tools/update-deviceid.coffee

@@ -1,37 +0,0 @@
-# This script will generate _deviceId struct for all devices in DB.
-# To be used in migrating existing data to the recent update which introduced _deviceId object.
-config = require '../config'
-mongodb = require 'mongodb'
-
-dbserver = new mongodb.Server(config.MONGODB_SOCKET, 0, {auto_reconnect: true})
-db = new mongodb.Db(config.DATABASE_NAME, dbserver, {native_parser: true, safe: true})
-
-query = {}
-
-projection = {
-  _id : true,
-  'InternetGatewayDevice.DeviceInfo' : true
-}
-
-db.open((err, db) ->
-  db.collection('devices', (err, collection) ->
-    cur = collection.find(query)
-    cur.each((err, doc) ->
-      if not doc?
-        db.close()
-        return
-      serialNumber = doc['InternetGatewayDevice']['DeviceInfo']['SerialNumber']['_value']
-      oui = doc['InternetGatewayDevice']['DeviceInfo']['ManufacturerOUI']['_value']
-      manufacturer = doc['InternetGatewayDevice']['DeviceInfo']['Manufacturer']['_value']
-      productClass = doc['InternetGatewayDevice']['DeviceInfo']['ProductClass']['_value']
-      id = doc['_id']
-      collection.update({_id : id}, {$set : {
-        '_deviceId._Manufacturer' : manufacturer,
-        '_deviceId._OUI' : oui,
-        '_deviceId._ProductClass' : productClass,
-        '_deviceId._SerialNumber' : serialNumber
-      }}, () ->
-      )
-    )
-  )
-)