1.1.3

Alpine linux does not have -n option in copy binary.
To avoid a problem on configure, I replaced the command with an "if".

.gitignore

@@ -2,7 +2,6 @@
 lib/*.map
 lib/api-functions.js
 lib/auth.js
-lib/cache.js
 lib/common.js
 lib/config.js
 lib/cwmp.js

CHANGELOG.md

@@ -1,5 +1,55 @@
 # Change Log
 
+## 1.1.3 (2018-10-23)
+
+- New config option (MAX_COMMIT_ITERATIONS) to avoid max commit iterations
+faults for more complex scripts.
+- Support base64 and hexBinary parameter types.
+- Strict parsing of number values in queries (e.g. "123abc" no longer accepted
+as 123).
+- Mixing $ne and $not operators is not allowed. Now it throws an error instead
+of returning incorrect results.
+- When a task expires, any associated fault is also deleted.
+- API now accepts 'timeout' argument when posting a task.
+- A number of stability fixes.
+
+## 1.1.2 (2018-02-24)
+
+- A large number of bug fixes as well as stability and performance improvements.
+- Three security vulnerabilities disclosed by Maximilian Hils have been patched.
+- New config option UDP_CONNECTION_REQUEST_PORT to specify binding port for UDP
+connection requests.
+- New config option DATETIME_MILLISECONDS to strip milliseconds from dateTime
+values.
+- New config option BOOLEAN_LITERAL to use 1/0 or true/false for boolean values.
+- Parameter values that cannot be parsed according to the reported type now show
+a warning message.
+- Virtual parameter scripts now use the variable 'args' instead of the special
+'TIMESTAMPS' and 'VALUES' variables. The content of the args array is: {declare
+timestamps}, {declare values}, {current timestamps}, {current values}.
+- Virtual parameter value types are now inferred from the JavaScript type if the
+returned value attribute is not a value-type pair.
+- Show a fault when a virtual parameter script doesn't return the required
+attributes.
+- Redis is now optional (and disabled by default), reducing the complexity of
+scalable deployments.
+- Better detection of cyclical presets resulting in fewer faults for complex
+provisioning scripts.
+- Math.random() is now deterministic on per-device basis. A function has been
+added to allow specifying a seed value (e.g. Math.random.seed(Date.now())).
+- Overload spikes are now handled gracefully by refusing to accept new sessions
+temporarily when under abnormal load.
+- Added log messages for session timeouts, connection drops, and XML parsing
+errors.
+- Date.now() now takes an optional argument to specify "time steps" (in
+milliseconds). This can be used to ensure a group of parameters are all
+refreshed at the same time intervals.
+- Only the non-default configuration options are now logged at process start.
+- Faults caused by errors from extensions now show a cleaner stack trace.
+- Exit main process if there are too many worker crashes (e.g. when DB is down).
+- Updated dependencies and included a lockfile to ensure installations get the
+exact dependencies it was tested against.
+
 ## 1.1.1 (2017-03-23)
 
 - Avoid crashing when connection request credentials are missing.

README.md

@@ -2,7 +2,7 @@
 
 GenieACS is a fast and lightweight TR-069 auto configuration server (ACS). This
 is the core back end component. A GUI front end is available at
-https://github.com/zaidka/genieacs-gui.
+https://github.com/genieacs/genieacs-gui.
 
 ## Features
 
@@ -16,26 +16,31 @@ systems
 
 ## Quick Start
 
-Install [Node.js](http://nodejs.org/), [Redis](http://redis.io/), and
-[MongoDB](http://www.mongodb.org/). Refer to the corresponding documentation for
-installation guides. Supported versions are:
+Install [Node.js](http://nodejs.org/) and [MongoDB](http://www.mongodb.org/).
+Refer to their corresponding documentation for installation instructions. The
+supported versions are:
 
-- Node.js: 6.x and 7.x (7.x recommended)
-- Redis: 2.4 through 3.2
+- Node.js: 6.x and 8.x (8.x recommended)
 - MongoDB: 2.6 through 3.4
 
+Install build tools and libxml2 development files from your system's package
+manager.
+
 Then install GenieACS using NPM:
 
     npm install -g genieacs
 
 Alternatively, you can install from source by cloning the git repository:
 
-    git clone https://github.com/zaidka/genieacs.git
+    git clone https://github.com/genieacs/genieacs.git
     cd genieacs
     git checkout $(git tag -l v1.1.* --sort=-v:refname | head -n 1)
     npm install
     npm run compile
 
+Before proceeding, find and review the file "config.json" in "config" directory
+where GenieACS is downloaded.
+
 Finally, run the following (from bin directory if installing from source):
 
     genieacs-cwmp
@@ -57,13 +62,13 @@ such.
 Note: For production deployment make sure to run those as background services.
 
 For further details about installation and configuration, refer to the
-[wiki section](https://github.com/zaidka/genieacs/wiki).
+[wiki section](https://github.com/genieacs/genieacs/wiki).
 
 ## Support
 
 The [Users mailing list](http://lists.genieacs.com) is a good place to get
 guidance and help from the community. Head on over and join the conversation!
-In addition, the [wiki](https://github.com/zaidka/genieacs/wiki) provides useful
+In addition, the [wiki](https://github.com/genieacs/genieacs/wiki) provides useful
 documentation and tips from GenieACS users.
 
 For commercial support options and professional services, please visit
@@ -71,5 +76,5 @@ For commercial support options and professional services, please visit
 
 ## License
 
-Copyright 2013-2017 Zaid Abdulla. GenieACS is released under the
-[AGPLv3 license terms](https://raw.githubusercontent.com/zaidka/genieacs/master/LICENSE).
+Copyright 2013-2018 GenieACS Inc. GenieACS is released under the
+[AGPLv3 license terms](https://raw.githubusercontent.com/genieacs/genieacs/master/LICENSE).

config/config-sample.json

@@ -1,7 +1,5 @@
 {
   "MONGODB_CONNECTION_URL" : "mongodb://127.0.0.1/genieacs",
-  "REDIS_PORT" : "6379",
-  "REDIS_HOST" : "127.0.0.1",
   "CWMP_INTERFACE" : "0.0.0.0",
   "CWMP_PORT" : 7547,
   "CWMP_SSL" : false,
@@ -9,7 +7,6 @@
   "NBI_PORT" : 7557,
   "FS_INTERFACE" : "0.0.0.0",
   "FS_PORT" : 7567,
-  "FS_IP" : "192.168.0.1",
-  "LOG_INFORMS" : true,
+  "FS_HOSTNAME" : "acs.example.com",
   "DEBUG" : false
 }

lib/api-functions.coffee

@@ -27,6 +27,7 @@ common = require './common'
 util = require 'util'
 URL = require 'url'
 auth = require './auth'
+cache = require './cache'
 
 
 udpConReq = (address, un, key, callback) ->
@@ -46,7 +47,13 @@ udpConReq = (address, un, key, callback) ->
 
   message = Buffer.from("GET #{uri} HTTP/1.1\r\nHost: #{address}\r\n\r\n")
 
-  client = dgram.createSocket('udp4')
+  client = dgram.createSocket({type: 'udp4', reuseAddr: true})
+  UDP_CONNECTION_REQUEST_PORT = config.get('UDP_CONNECTION_REQUEST_PORT')
+  if UDP_CONNECTION_REQUEST_PORT
+    # When a device is NAT'ed, the UDP Connection Request must originate from the same address and port used by the STUN server, in order to traverse the firewall.
+    # This does require that the Genieacs NBI and STUN server are allowed to bind to the same address and port.
+    # The STUN server needs to open its UDP port with the SO_REUSEADDR option, allowing the NBI to also bind to the same port.
+    client.bind({port: UDP_CONNECTION_REQUEST_PORT, exclusive: true})
 
   count = 3
   client.send(message, 0, message.length, port, host, f = (err) ->
@@ -58,8 +65,12 @@ udpConReq = (address, un, key, callback) ->
   return true
 
 
-httpConReq = (url, username, password, timeout, callback) ->
+httpConReq = (url, username, password, allowBasicAuth, timeout, callback) ->
   options = URL.parse(url)
+
+  if options.protocol != 'http:'
+    return callback(new Error('Invalid connection request URL or protocol'))
+
   # Ensure socket is reused in case of digest authentication
   options.agent = new http.Agent({maxSockets : 1})
 
@@ -78,6 +89,10 @@ httpConReq = (url, username, password, timeout, callback) ->
     if res.statusCode == 401 and res.headers['www-authenticate']?
       authHeader = auth.parseAuthHeader(res.headers['www-authenticate'])
       if authHeader.method is 'Basic'
+        if not allowBasicAuth
+          request.abort()
+          callback(new Error('Basic HTTP authentication not allowed'))
+          return
         options.headers = {'Authorization' : auth.basic(username or '', password or '')}
       else if authHeader.method is 'Digest'
         options.headers = {
@@ -123,6 +138,7 @@ httpConReq = (url, username, password, timeout, callback) ->
 
 connectionRequest = (deviceId, callback) ->
   CONNECTION_REQUEST_TIMEOUT = config.get('CONNECTION_REQUEST_TIMEOUT', deviceId)
+  CONNECTION_REQUEST_ALLOW_BASIC_AUTH = config.get('CONNECTION_REQUEST_ALLOW_BASIC_AUTH', deviceId)
   proj = {
     'Device.ManagementServer.ConnectionRequestURL._value' : 1,
     'Device.ManagementServer.UDPConnectionRequestAddress._value' : 1,
@@ -136,6 +152,7 @@ connectionRequest = (deviceId, callback) ->
 
   db.devicesCollection.findOne({_id : deviceId}, proj, (err, device)->
     return callback(err) if err
+    return callback(new Error("No such device")) if !device
 
     if device.Device? # TR-181 data model
       connectionRequestUrl = device.Device.ManagementServer.ConnectionRequestURL._value
@@ -151,7 +168,7 @@ connectionRequest = (deviceId, callback) ->
     conReq = () ->
       udpSent = udpConReq(udpConnectionRequestAddress, username, password, (err) -> throw err if err)
 
-      httpConReq(connectionRequestUrl, username, password, CONNECTION_REQUEST_TIMEOUT, (err) ->
+      httpConReq(connectionRequestUrl, username, password, CONNECTION_REQUEST_ALLOW_BASIC_AUTH, CONNECTION_REQUEST_TIMEOUT, (err) ->
         if udpSent
           return callback()
         callback(err)
@@ -208,7 +225,7 @@ insertTasks = (tasks, callback) ->
   if tasks? and common.typeOf(tasks) isnt common.ARRAY_TYPE
     tasks = [tasks]
   else if not tasks? or tasks.length == 0
-    return callback(tasks)
+    return callback(null, tasks or [])
 
   counter = tasks.length
 
@@ -236,12 +253,7 @@ deleteDevice = (deviceId, callback) ->
         return callback(err) if err
         db.operationsCollection.remove({'_id' : {'$regex' : "^#{common.escapeRegExp(deviceId)}\\:"}}, (err) ->
           return callback(err) if err
-          db.redisClient.del("#{deviceId}_presets_hash",
-            "#{deviceId}_inform_hash",
-            "#{deviceId}_faults",
-            "#{deviceId}_tasks",
-            "#{deviceId}_operations",
-            callback)
+          cache.del("#{deviceId}_tasks_faults_operations", callback)
         )
       )
     )

lib/auth.coffee

@@ -21,10 +21,30 @@ crypto = require 'crypto'
 
 
 exports.parseAuthHeader = (authHeader) ->
-  res = {method: authHeader.slice(0, authHeader.indexOf(' '))}
-  re = /([a-z0-9_-]+)=(?:"([^"]+)"|([a-z0-9_-]+))/gi
-  while match = re.exec(authHeader)
-    res[match[1]] = match[2] or match[3]
+  authHeader = authHeader.trim()
+  method = authHeader.split(' ', 1)[0]
+  res = {method: method}
+  parts = authHeader.slice(method.length + 1).split(',')
+
+  while (part = parts.shift())?
+    name = part.split('=', 1)[0]
+    throw new Error('Unable to parse auth header') if name.length == part.length
+    value = part.slice(name.length + 1)
+    if not /^\s*"/.test(value)
+      value = value.trim()
+    else
+      while not /[^\\]"\s*$/.test(value)
+        p = parts.shift()
+        throw new Error('Unable to parse auth header') if not p?
+        value += ',' + p
+
+      try
+        value = JSON.parse(value)
+      catch
+        throw new Error('Unable to parse auth header')
+
+    res[name.trim()] = value
+
   return res
 
 

lib/cache.coffee

@@ -1,310 +0,0 @@
-###
-# Copyright 2013-2017  Zaid Abdulla
-#
-# This file is part of GenieACS.
-#
-# GenieACS is free software: you can redistribute it and/or modify
-# it under the terms of the GNU Affero General Public License as
-# published by the Free Software Foundation, either version 3 of the
-# License, or (at your option) any later version.
-#
-# GenieACS is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU Affero General Public License for more details.
-#
-# You should have received a copy of the GNU Affero General Public License
-# along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
-###
-
-vm = require 'vm'
-crypto = require 'crypto'
-
-later = require 'later'
-
-db = require './db'
-query = require './query'
-
-
-REFRESH = 3000
-
-nextRefresh = 0
-hash = null
-presets = null
-provisions = null
-virtualParameters = null
-files = null
-
-
-UNLOCK_SCRIPT = 'if redis.call("get", KEYS[1]) == ARGV[1] then return redis.call("del", KEYS[1]) else return 0 end'
-EXTEND_SCRIPT = 'if redis.call("get", KEYS[1]) == ARGV[1] then return redis.call("pexpire", KEYS[1], ARGV[2]) else return 0 end'
-
-lock = (lockName, ttl, callback) ->
-  token = Math.random().toString(36).slice(2)
-  unlockOrExtend = (ttl) ->
-    if not ttl
-      db.redisClient.eval(UNLOCK_SCRIPT, 1, lockName, token, (err, res) ->
-        throw err or new Error('Lock expired') if err or not res
-      )
-    else
-      db.redisClient.eval(EXTEND_SCRIPT, 1, lockName, token, ttl, (err, res) ->
-        throw err or new Error('Lock expired') if err or not res
-      )
-    return
-
-  db.redisClient.set(lockName, token, 'NX', 'PX', ttl, (err, res) ->
-    if err or res
-      return callback(err, unlockOrExtend)
-
-    setTimeout(() ->
-      return lock(lockName, ttl, callback)
-    , 200)
-  )
-
-
-computeHash = () ->
-  # MD5 hash for presets, provisions, virtual parameters for detecting changes
-  h = crypto.createHash('md5')
-  for p in presets
-    h.update(JSON.stringify(p.name))
-    h.update(JSON.stringify(p.channel))
-    h.update(JSON.stringify(p.schedule))
-    h.update(JSON.stringify(p.events))
-    h.update(JSON.stringify(p.precondition))
-    h.update(JSON.stringify(p.provisions))
-
-  keys = Object.keys(provisions).sort()
-  h.update(JSON.stringify(keys))
-  h.update(provisions[k].md5) for k in keys
-
-  keys = Object.keys(virtualParameters).sort()
-  h.update(JSON.stringify(keys))
-  h.update(virtualParameters[k].md5) for k in keys
-
-  hash = h.digest('hex')
-
-
-refresh = (callback) ->
-  now = Date.now()
-  db.redisClient.get('presets_hash', (err, res) ->
-    return callback(err) if err
-
-    if hash? and res == hash
-      nextRefresh = now + REFRESH
-      return callback()
-
-    lock('presets_hash_lock', 3000, (err, unlockOrExtend) ->
-      return callback(err) if err
-      counter = 3
-
-      counter += 2
-      db.presetsCollection.find().toArray((err, res) ->
-        if err
-          callback(err) if counter & 1
-          return counter = 0
-
-        db.objectsCollection.find().toArray((err, objects) ->
-          if err
-            callback(err) if (counter & 1)
-            return counter = 0
-
-          res.sort((a, b) ->
-            if a.weight == b.weight
-              return a._id > b._id
-            else
-              return a.weight - b.weight
-          )
-
-          presets = []
-
-          for preset in res
-            schedule = null
-            if preset.schedule
-              parts = preset.schedule.trim().split(/\s+/)
-              schedule = {
-                md5: crypto.createHash('md5').update(preset.schedule).digest('hex')
-              }
-
-              try
-                schedule.duration = +(parts.shift()) * 1000
-                parts.unshift('0') if parts.length == 5
-                # TODO later.js doesn't throw erorr if expression is invalid!
-                schedule.schedule = later.schedule(later.parse.cron(parts.join(' '), true))
-              catch err
-                # TODO show a warning
-                schedule.schedule = false
-
-            events = preset.events ? {}
-
-            precondition = query.convertMongoQueryToFilters(JSON.parse(preset.precondition))
-
-            _provisions = preset.provisions or []
-
-            # Generate provisions from the old configuration format
-            for c in preset.configurations
-              switch c.type
-                when 'age'
-                  _provisions.push(['refresh', c.name, c.age])
-                when 'value'
-                  _provisions.push(['value', c.name, c.value])
-                when 'add_tag'
-                  _provisions.push(['tag', c.tag, true])
-                when 'delete_tag'
-                  _provisions.push(['tag', c.tag, false])
-                when 'provision'
-                  _provisions.push([c.name].concat(c.args or []))
-                when 'add_object'
-                  for obj in objects
-                    if obj['_id'] == c.object
-                      alias = ("#{k}:#{JSON.stringify(obj[k])}" for k in obj['_keys']).join(',')
-                      p = "#{c.name}.[#{alias}]"
-                      _provisions.push(['instances', p, 1])
-                      for k of obj
-                        if k[0] != '_' and k not in obj['_keys']
-                          _provisions.push(['value', "#{p}.#{k}", obj[k]])
-                when 'delete_object'
-                  for obj in objects
-                    if obj['_id'] == c.object
-                      alias = ("#{k}:#{JSON.stringify(obj[k])}" for k in obj['_keys']).join(',')
-                      p = "#{c.name}.[#{alias}]"
-                      _provisions.push(['instances', p, 0])
-                else
-                  callback(new Error("Unknown configuration type #{c.type}")) if counter & 1
-                  return counter = 0
-
-            presets.push({name: preset._id, channel: preset.channel or 'default', schedule: schedule, events: events, precondition: precondition, provisions: _provisions})
-
-          if (counter -= 2) == 1
-            computeHash()
-            db.redisClient.setex("presets_hash", 300, hash, (err) ->
-              unlockOrExtend(0)
-              nextRefresh = now + REFRESH
-              return callback()
-            )
-        )
-      )
-
-      counter += 2
-      db.provisionsCollection.find().toArray((err, res) ->
-        if err
-          callback(err) if counter & 1
-          return counter = 0
-
-        provisions = {}
-        for r in res
-          provisions[r._id] = {}
-          provisions[r._id].md5 = crypto.createHash('md5').update(r.script).digest('hex')
-          provisions[r._id].script = new vm.Script("\"use strict\";(function(){\n#{r.script}\n})();", {filename: r._id, lineOffset: -1, timeout: 50})
-
-        if (counter -= 2) == 1
-          computeHash()
-          db.redisClient.setex("presets_hash", 300, hash, (err) ->
-            unlockOrExtend(0)
-            nextRefresh = now + REFRESH
-            return callback()
-          )
-      )
-
-      counter += 2
-      db.virtualParametersCollection.find().toArray((err, res) ->
-        if err
-          callback(err) if counter & 1
-          return counter = 0
-
-        virtualParameters = {}
-        for r in res
-          virtualParameters[r._id] = {}
-          virtualParameters[r._id].md5 = crypto.createHash('md5').update(r.script).digest('hex')
-          virtualParameters[r._id].script = new vm.Script("\"use strict\";(function(){\n#{r.script}\n})();", {filename: r._id, lineOffset: -1, timeout: 50})
-
-        if (counter -= 2) == 1
-          computeHash()
-          db.redisClient.setex("presets_hash", 300, hash, (err) ->
-            unlockOrExtend(0)
-            nextRefresh = now + REFRESH
-            return callback()
-          )
-      )
-
-      counter += 2
-      db.filesCollection.find().toArray((err, res) ->
-        if err
-          callback(err) if counter & 1
-          return counter = 0
-
-        files = {}
-        for r in res
-          id = r.filename or r._id.toString()
-          files[id] = {}
-          files[id].length = r.length
-          files[id].md5 = r.md5
-          files[id].contentType = r.contentType
-
-        if (counter -= 2) == 1
-          computeHash()
-          db.redisClient.setex("presets_hash", 300, hash, (err) ->
-            unlockOrExtend(0)
-            nextRefresh = now + REFRESH
-            return callback()
-          )
-      )
-
-      if (counter -= 2) == 1
-        computeHash()
-        db.redisClient.setex("presets_hash", 300, hash, (err) ->
-          unlockOrExtend(0)
-          nextRefresh = now + REFRESH
-          return callback()
-        )
-    )
-  )
-
-
-getPresets = (callback) ->
-  if Date.now() < nextRefresh
-    return callback(null, hash, presets)
-
-  refresh((err) ->
-    return callback(err, hash, presets)
-  )
-
-
-getProvisions = (callback) ->
-  if Date.now() < nextRefresh
-    return callback(null, hash, provisions)
-
-  refresh((err) ->
-    return callback(err, hash, provisions)
-  )
-
-
-getVirtualParameters = (callback) ->
-  if Date.now() < nextRefresh
-    return callback(null, hash, virtualParameters)
-
-  refresh((err) ->
-    return callback(err, hash, virtualParameters)
-  )
-
-
-getFiles = (callback) ->
-  if Date.now() < nextRefresh
-    return callback(null, hash, files)
-
-  refresh((err) ->
-    return callback(err, hash, files)
-  )
-
-
-getProvisionsAndVirtualParameters = (callback) ->
-  if Date.now() < nextRefresh
-    return callback(null, hash, provisions, virtualParameters)
-
-  refresh((err) ->
-    return callback(err, hash, provisions, virtualParameters)
-  )
-
-exports.getPresets = getPresets
-exports.getProvisions = getProvisions
-exports.getFiles = getFiles
-exports.getProvisionsAndVirtualParameters = getProvisionsAndVirtualParameters

lib/cache.js

@@ -0,0 +1,257 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const config = require("./config");
+
+const MAX_CACHE_TTL = config.get("MAX_CACHE_TTL");
+
+let redisClient;
+let mongoDb;
+let mongoCollection;
+let mongoTimeOffset = 0;
+
+
+function connect(callback) {
+  const REDIS_HOST = config.get("REDIS_HOST");
+
+  if (REDIS_HOST) {
+    const redis = require("redis");
+    const REDIS_PORT = config.get("REDIS_PORT");
+    const REDIS_DB = config.get("REDIS_DB");
+    redisClient = redis.createClient(REDIS_PORT, REDIS_HOST);
+    redisClient.select(REDIS_DB, function(err) {
+      exports.get = redisGet;
+      exports.set = redisSet;
+      exports.del = redisDel;
+      exports.pop = redisPop;
+      exports.lock = redisLock;
+      callback(err);
+    });
+  }
+  else {
+    const mongodb = require("mongodb");
+    const MONGODB_CONNECTION_URL = config.get("MONGODB_CONNECTION_URL");
+
+    mongodb.MongoClient.connect(MONGODB_CONNECTION_URL, function(err, db) {
+      if (err)
+        return callback(err);
+
+      mongoDb = db;
+      mongoCollection = db.collection("cache");
+      mongoCollection.ensureIndex({"expire": 1}, {expireAfterSeconds: 0});
+
+      const now = Date.now();
+      mongoDb.command({hostInfo: 1}, function(err, res) {
+        if (err)
+          return callback(err);
+        mongoTimeOffset = res.system.currentTime.getTime() - now;
+        exports.get = mongoGet;
+        exports.set = mongoSet;
+        exports.del = mongoDel;
+        exports.pop = mongoPop;
+        exports.lock = mongoLock;
+        callback();
+      });
+    });
+  }
+}
+
+
+function disconnect() {
+  if (redisClient)
+    redisClient.quit();
+
+  if (mongoDb)
+    mongoDb.close();
+}
+
+
+function redisGet(key, callback) {
+  if (Array.isArray(key))
+    redisClient.mget(key, callback);
+  else
+    redisClient.get(key, callback);
+}
+
+
+function redisDel(key, callback) {
+  redisClient.del(key, callback);
+}
+
+
+function redisSet(key, value, ttl, callback) {
+  if (!callback && typeof v === "function") {
+    callback = ttl;
+    ttl = null;
+  }
+
+  if (!ttl)
+    ttl = MAX_CACHE_TTL;
+
+  redisClient.setex(key, ttl, value, callback);
+}
+
+
+function redisPop(key, callback) {
+  const POP_SCRIPT = "local v=redis.call(\"get\",KEYS[1]);redis.call(\"del\",KEYS[1]);return v;";
+  redisClient.eval(POP_SCRIPT, 1, key, callback);
+}
+
+
+function redisLock(lockName, ttl, callback) {
+  const UNLOCK_SCRIPT = "if redis.call(\"get\", KEYS[1]) == ARGV[1] then return redis.call(\"del\", KEYS[1]) else return 0 end";
+  const EXTEND_SCRIPT = "if redis.call(\"get\", KEYS[1]) == ARGV[1] then return redis.call(\"pexpire\", KEYS[1], ARGV[2]) else return 0 end";
+  const token = Math.random().toString(36).slice(2);
+
+  function unlockOrExtend(ttl) {
+    if (!ttl) {
+      redisClient.eval(UNLOCK_SCRIPT, 1, lockName, token, function(err, res) {
+        if (err || !res)
+          throw err || new Error("Lock expired");
+      });
+    }
+    else {
+      redisClient.eval(EXTEND_SCRIPT, 1, lockName, token, ttl, function(err, res) {
+        if (err || !res)
+          throw err || new Error("Lock expired");
+      });
+    }
+  }
+
+  redisClient.set(lockName, token, "NX", "EX", ttl, function(err, res) {
+    if (err || res)
+      return callback(err, unlockOrExtend);
+
+    setTimeout(function() {
+      redisLock(lockName, ttl, callback);
+    }, 200);
+  });
+}
+
+
+function mongoGet(key, callback) {
+  let expire = new Date(Date.now() - mongoTimeOffset);
+  if (Array.isArray(key))
+    mongoCollection.find({"_id": {"$in": key}}).toArray(function(err, res) {
+      if (err)
+        return callback(err);
+
+      let indices = {};
+      key.forEach(function(v, i) {
+        indices[v] = i;
+      });
+
+      let values = [];
+      res.forEach(function(r) {
+        if (r["expire"] > expire)
+          values[indices[r["_id"]]] = r["value"];
+      });
+      callback(null, values);
+    });
+  else
+    mongoCollection.findOne({"_id": {"$in": [key]}}, function(err, res) {
+      if (err || !res)
+        return callback(err);
+
+      if (res["expire"] > expire)
+        return callback(null, res["value"]);
+
+      callback();
+    });
+}
+
+
+function mongoDel(key, callback) {
+  if (Array.isArray(key))
+    mongoCollection.remove({"_id": {"$in": key}}, callback);
+  else
+    mongoCollection.remove({"_id": key}, callback);
+}
+
+
+function mongoSet(key, value, ttl, callback) {
+  if (!callback && typeof v === "function") {
+    callback = ttl;
+    ttl = null;
+  }
+
+  if (!ttl)
+    ttl = MAX_CACHE_TTL;
+
+  let expire = new Date((Date.now() - mongoTimeOffset) + (ttl * 1000));
+  mongoCollection.save({"_id": key, "value": value, "expire": expire}, callback);
+}
+
+
+function mongoPop(key, callback) {
+  mongoCollection.findAndModify({"_id": key}, null, null, {remove: true}, function(err, res) {
+    if (err || !res["value"])
+      return callback(err);
+
+    if (res["value"]["expire"] > new Date(Date.now() - mongoTimeOffset))
+      return callback(null, res["value"]["value"]);
+
+    callback();
+  });
+}
+
+
+function mongoLock(lockName, ttl, callback) {
+  const token = Math.random().toString(36).slice(2);
+
+  function unlockOrExtend(ttl) {
+    if (!ttl) {
+      mongoCollection.remove({"_id": lockName, "value": token}, function(err, res) {
+        if (err || res["result"]["n"] !== 1)
+          throw err || new Error("Lock expired");
+      });
+    }
+    else {
+      let expire = new Date((Date.now() - mongoTimeOffset) + (ttl * 1000));
+      mongoCollection.update(
+        {"_id": lockName, "value": token},
+        {"expire": expire},
+        function(err, res) {
+          if (err || res["result"]["n"] !== 1)
+            throw err | new Error("Lock expired");
+        }
+      );
+    }
+  }
+
+  let expireTest = new Date(Date.now() - mongoTimeOffset);
+  let expireSet = new Date((Date.now() - mongoTimeOffset) + (ttl * 1000));
+
+  mongoCollection.update(
+    {"_id": lockName, "expire": {"$lte": expireTest}},
+    {"$set": {"value" : token, "expire": expireSet}},
+    {upsert: true},
+    function(err, res) {
+      if (err && err.code === 11000)
+        return setTimeout(function() {mongoLock(lockName, ttl, callback)}, 200);
+
+      return callback(err, unlockOrExtend);
+    }
+  );
+}
+
+
+exports.connect = connect;
+exports.disconnect = disconnect;

lib/cluster.js

@@ -20,14 +20,17 @@
 
 const cluster = require("cluster");
 const logger = require("./logger");
+const config = require("./config");
 
 const VERSION = require("../package.json").version;
 
+let respawnTimestamp = 0;
+let tooManyCrashesTimestamp = 0;
 
 function getDependencyVersions() {
   let dependencies = [`node@${process.versions.node}`];
   try {
-    let npmls = JSON.parse(require("child_process").execSync("npm ls --prod --json"));
+    let npmls = JSON.parse(require("child_process").execSync("npm ls --prod --json --depth 0"));
 
     for (let k in npmls.dependencies)
       dependencies.push(`${k}@${npmls.dependencies[k].version}`);
@@ -41,8 +44,13 @@ function getConfig() {
   let conf = [];
   const keys = Object.keys(process.env);
   for (let k of keys)
-    if (k.startsWith("GENIEACS_"))
-      conf.push(`${k.slice(9)}=${process.env[k]}`);
+    if (k.startsWith("GENIEACS_")) {
+      const name = k.slice(9);
+      const value = process.env[k];
+      const def = config.getDefault(name);
+      if (def == null || value !== `${def}`)
+        conf.push(`${name}=${value}`);
+    }
 
   return conf.join(",");
 }
@@ -61,13 +69,40 @@ function restartWorker(worker, code, signal) {
     msg.signal = signal;
 
   logger.error(msg);
+
+  const now = Date.now();
+  respawnTimestamp = Math.max(now, respawnTimestamp + 2000);
+  if (respawnTimestamp === now) {
+    tooManyCrashesTimestamp = now;
     cluster.fork();
+    return
+  }
+
+  if (now - tooManyCrashesTimestamp > 60000) {
+    process.exitCode = 1;
+    cluster.removeListener("exit", restartWorker);
+    for (let pid in cluster.workers)
+      cluster.workers[pid].kill();
+
+    logger.error({
+      message: "Too many crashes, exiting",
+      pid: process.pid
+    });
+    return;
+  }
+
+  setTimeout(function() {
+    if (process.exitCode) return;
+    cluster.fork();
+  }, respawnTimestamp - now);
 }
 
 
 function start(service) {
   logger.init(service, VERSION);
   const config = require("./config");
+  const SERVICE_ADDRESS = config.get(`${service.toUpperCase()}_INTERFACE`)
+  const SERVICE_PORT = config.get(`${service.toUpperCase()}_PORT`)
 
   logger.info({
     message: `GenieACS (genieacs-${service}) starting`,
@@ -78,12 +113,15 @@ function start(service) {
   });
 
   cluster.on("listening", function(worker, address) {
+    if ((address.addressType === 4 || address.addressType === 6) &&
+        address.address === SERVICE_ADDRESS && address.port === SERVICE_PORT) {
       logger.info({
         message: "Worker listening",
         pid: worker.process.pid,
         address: address.address,
         port: address.port
       });
+    }
   });
 
   cluster.on("exit", restartWorker);
@@ -126,13 +164,13 @@ function start(service) {
 process.on("uncaughtException", function(err) {
   logger.error({
     message: "Uncaught exception in master process, exiting",
-    exception: err
+    exception: err,
+    pid: process.pid
   });
+  process.exitCode = 1;
   cluster.removeListener("exit", restartWorker);
   for (let pid in cluster.workers)
     cluster.workers[pid].kill();
-
-  logger.close();
 });
 
 

lib/common.coffee

@@ -18,7 +18,6 @@
 ###
 
 buffer = require 'buffer'
-querystring = require 'querystring'
 
 UNDEFINED_TYPE = '[object Undefined]'
 NULL_TYPE = '[object Null]'
@@ -210,28 +209,11 @@ addPathMeta = (path) ->
   return path
 
 
-hammingWeight = (flags) ->
-  flags -= ((flags >> 1) & 0x55555555)
-  flags = (((flags >> 2) & 0x33333333) + (flags & 0x33333333))
-  flags = (((flags >> 4) + flags) & 0x0f0f0f0f)
-  flags += (flags >> 8)
-  flags += (flags >> 16)
-  return flags & 0x0000003f
-
-
 # Source: http://stackoverflow.com/a/6969486
 escapeRegExp = (str) ->
   return str.replace(/[\-\[\]\/\{\}\(\)\*\+\?\.\\\^\$\|]/g, "\\$&")
 
 
-# djb2 hashing algorithm adapted from http://www.cse.yorku.ca/~oz/hash.html
-djb2 = (str) ->
-  hash = 5381
-  for i in [0...str.length]
-    hash = ((hash << 5) + hash) ^ str.charCodeAt(i)
-  return hash
-
-
 exports.UNDEFINED_TYPE = UNDEFINED_TYPE
 exports.NULL_TYPE = NULL_TYPE
 exports.NUMBER_TYPE = NUMBER_TYPE
@@ -244,6 +226,4 @@ exports.typeOf = typeOf
 exports.pathOverlap = pathOverlap
 exports.parsePath = parsePath
 exports.addPathMeta = addPathMeta
-exports.hammingWeight = hammingWeight
 exports.escapeRegExp = escapeRegExp
-exports.djb2 = djb2

lib/config.coffee

@@ -24,7 +24,7 @@ options = {
   CONFIG_DIR : {type : 'path', default : 'config'},
   MONGODB_CONNECTION_URL : {type : 'string', default : 'mongodb://127.0.0.1/genieacs'},
   REDIS_PORT : {type : 'int', default : 6379},
-  REDIS_HOST : {type : 'string', default : '127.0.0.1'},
+  REDIS_HOST : {type : 'string', default : ''},
   REDIS_DB : {type : 'int', default : 0},
 
   CWMP_WORKER_PROCESSES : {type : 'int', default : 0},
@@ -49,6 +49,8 @@ options = {
   FS_LOG_FILE : {type: 'path', default : ''},
   FS_ACCESS_LOG_FILE : {type : 'path', default : ''},
 
+  UDP_CONNECTION_REQUEST_PORT : {type : 'int', default : 0},
+
   DOWNLOAD_TIMEOUT: {type : 'int', default : 3600},
   EXT_TIMEOUT: {type: 'int', default: 3000},
   MAX_CACHE_TTL : {type : 'int', default : 86400},
@@ -62,6 +64,11 @@ options = {
   COOKIES_PATH : {type : 'string'},
   LOG_FORMAT : {type : 'string', default : 'simple'},
   ACCESS_LOG_FORMAT : {type : 'string', default : ''},
+  MAX_CONCURRENT_REQUESTS : {type : 'int', default: 20},
+  DATETIME_MILLISECONDS : {type: 'bool', default: true},
+  BOOLEAN_LITERAL : {type: 'bool', default: true},
+  CONNECTION_REQUEST_ALLOW_BASIC_AUTH: {type: 'bool', default: false},
+  MAX_COMMIT_ITERATIONS : {type : 'int', default: 32},
 
   # XML configuration
   XML_RECOVER : {type : 'bool', default : false},
@@ -185,8 +192,18 @@ get = (option, deviceId) ->
   return allConfig[option]
 
 
+getDefault = (optionName) ->
+  option = options[optionName]
+  return null if not option
+  val = option.default
+  if val and option.type == 'path'
+    val = path.resolve(val)
+  return val
+
+
 # load authentication scripts
 try
   exports.auth = require(path.resolve(allConfig.CONFIG_DIR, 'auth'))
 
 exports.get = get
+exports.getDefault = getDefault

lib/cwmp.coffee

@@ -28,27 +28,63 @@ session = require './session'
 query = require './query'
 device = require './device'
 cache = require './cache'
+localCache = require './local-cache'
 db = require './db'
 logger = require './logger'
+scheduling = require './scheduling'
+
 
 MAX_CYCLES = 4
 
+MAX_CONCURRENT_REQUESTS = config.get('MAX_CONCURRENT_REQUESTS')
+stats = {
+  concurrentRequests: 0
+  totalRequests: 0
+  droppedRequests: 0
+  initiatedSessions: 0
+}
 
 throwError = (err, httpResponse) ->
   if httpResponse
+    currentSessions.delete(httpResponse.connection)
     httpResponse.writeHead(500, {'Connection' : 'close'})
     httpResponse.end("#{err.name}: #{err.message}")
+    stats.concurrentRequests -= 1
 
   throw err
 
 
-writeResponse = (sessionContext, res) ->
+writeResponse = (sessionContext, res, close) ->
+  # Close connection after last request in session
+  res.headers['Connection'] = 'close' if close
+
   if config.get('DEBUG', sessionContext.deviceId)
     dump = "# RESPONSE #{new Date(Date.now())}\n" + JSON.stringify(res.headers) + "\n#{res.data}\n\n"
     require('fs').appendFile("./debug/#{sessionContext.deviceId}.dump", dump, (err) ->
       throwError(err) if err
     )
 
+  finish = () ->
+    stats.concurrentRequests -= 1
+    if sessionContext.httpRequest.connection.destroyed
+      logger.accessError({
+        sessionContext: sessionContext
+        message: 'Connection dropped'
+      })
+    else if close
+      endSession(sessionContext, (err, isNew) ->
+        return throwError(err) if err
+
+        if isNew
+          logger.accessInfo({
+            sessionContext: sessionContext
+            message: 'New device registered'
+          })
+      )
+    else
+      sessionContext.lastActivity = Date.now()
+      currentSessions.set(sessionContext.httpRequest.connection, sessionContext)
+
   # respond using the same content-encoding as the request
   if sessionContext.httpRequest.headers['content-encoding']? and res.data.length > 0
     switch sessionContext.httpRequest.headers['content-encoding']
@@ -65,14 +101,13 @@ writeResponse = (sessionContext, res) ->
       res.headers['Content-Length'] = data.length
       sessionContext.httpResponse.writeHead(res.code, res.headers)
       sessionContext.httpResponse.end(data)
+      finish()
     )
   else
     res.headers['Content-Length'] = res.data.length
     sessionContext.httpResponse.writeHead(res.code, res.headers)
     sessionContext.httpResponse.end(res.data)
-
-  delete sessionContext.httpRequest
-  delete sessionContext.httpResponse
+    finish()
 
 
 recordFault = (sessionContext, fault, provisions, channels) ->
@@ -85,6 +120,10 @@ recordFault = (sessionContext, fault, provisions, channels) ->
   for channel of channels
     provs = sessionContext.faults[channel]?.provisions or []
     faults[channel] = Object.assign({provisions: provs}, fault)
+    if channel.startsWith('task_')
+      taskId = channel.slice(5)
+      for t in sessionContext.tasks
+        faults[channel].expiry = t.expiry if t._id == taskId and t.expiry
 
     if sessionContext.retries[channel]?
       ++ sessionContext.retries[channel]
@@ -99,7 +138,7 @@ recordFault = (sessionContext, fault, provisions, channels) ->
     sessionContext.faultsTouched ?= {}
     sessionContext.faultsTouched[channel] = true
 
-    logger.accessError({
+    logger.accessWarn({
       sessionContext: sessionContext
       message: 'Channel has faulted'
       fault: fault
@@ -154,7 +193,7 @@ transferComplete = (sessionContext, rpc) ->
       for k, v of operation.retries
         sessionContext.retries[k] = v
 
-      recordFault(sessionContext, fault)
+      recordFault(sessionContext, fault, operation.provisions, operation.channels)
 
     res = soap.response({
       id : rpc.id,
@@ -165,28 +204,6 @@ transferComplete = (sessionContext, rpc) ->
   )
 
 
-testSchedule = (schedule, timestamp) ->
-  range = schedule.schedule.nextRange(10, new Date(timestamp))
-  prev = schedule.schedule.prevRange(1, new Date(timestamp), new Date(timestamp - schedule.duration))
-
-  if prev
-    first = +(prev[0] ? 0)
-    last = +(prev[1] ? 0) + schedule.duration
-  else
-    first = +(range[0][0] ? 0)
-    last = +(range[0][1] ? 0) + schedule.duration
-
-  for r in range
-    if last < r[0]
-      break
-    last = +(r[1] ? 0) + schedule.duration
-
-  if timestamp >= first
-    return [true, Math.max(0, last - timestamp)]
-  else
-    return [false, first - timestamp]
-
-
 # Append providions and remove duplicates
 appendProvisions = (original, toAppend) ->
   modified = false
@@ -213,7 +230,7 @@ appendProvisions = (original, toAppend) ->
 
 
 applyPresets = (sessionContext) ->
-  cache.getPresets((err, presetsHash, presets) ->
+  localCache.getPresets((err, presetsHash, presets) ->
     return throwError(err, sessionContext.httpResponse) if err
 
     # Filter presets based on existing faults
@@ -237,6 +254,9 @@ applyPresets = (sessionContext) ->
 
         blackList[channel] = if fault.precondition then 1 else 2
 
+    sessionContext.deviceData.timestamps.revision = 1
+    sessionContext.deviceData.attributes.revision = 1
+
     deviceEvents = {}
     for p in sessionContext.deviceData.paths.find(['Events', '*'], false, true)
       if sessionContext.timestamp <= sessionContext.deviceData.attributes.get(p)?.value?[1][0]
@@ -257,8 +277,13 @@ applyPresets = (sessionContext) ->
           eventsMatch = false
           break
 
-      continue if (not eventsMatch) or (preset.schedule and not
-        (preset.schedule.schedule and testSchedule(preset.schedule, sessionContext.timestamp)[0]))
+      continue if not eventsMatch
+
+      if preset.schedule
+        continue if not preset.schedule.schedule
+        r = scheduling.cron(sessionContext.timestamp, preset.schedule.schedule)
+        if not (r[0] + preset.schedule.duration > sessionContext.timestamp)
+          continue
 
       filteredPresets.push(preset)
       for k of preset.precondition
@@ -307,18 +332,25 @@ applyPresets = (sessionContext) ->
           sessionContext.faultsTouched ?= {}
           sessionContext.faultsTouched[channel] = true
 
-      sessionContext.presetCycles = (sessionContext.presetCycles or 0) + 1
+      # Don't increment when processing individual channels (e.g. after fault)
+      sessionContext.presetCycles = (sessionContext.presetCycles or 0) + 1 if not whiteList?
 
       if sessionContext.presetCycles > MAX_CYCLES
         fault = {
-          code: 'endless_cycle'
-          message: 'The provision seems to be repeating indefinitely'
+          code: 'preset_loop'
+          message: 'The presets are stuck in an endless configuration loop'
           timestamp: sessionContext.timestamp
         }
         recordFault(sessionContext, fault)
+        # No need to save retryNow
+        for c of sessionContext.faults
+          delete sessionContext.faults[c].retryNow
         session.clearProvisions(sessionContext)
         return sendAcsRequest(sessionContext)
 
+      sessionContext.deviceData.timestamps.dirty = 0;
+      sessionContext.deviceData.attributes.dirty = 0;
+
       session.rpcRequest(sessionContext, null, (err, fault, id, acsRequest) ->
         return throwError(err, sessionContext.httpResponse) if err
 
@@ -337,6 +369,10 @@ applyPresets = (sessionContext) ->
           if whiteList?
             return applyPresets(sessionContext)
 
+          if sessionContext.deviceData.timestamps.dirty > 1 or
+              sessionContext.deviceData.attributes.dirty > 1
+            return applyPresets(sessionContext)
+
         sendAcsRequest(sessionContext, id, acsRequest)
       )
     )
@@ -372,26 +408,29 @@ nextRpc = (sessionContext) ->
 
     session.clearProvisions(sessionContext)
 
-    for task in sessionContext.tasks
-      channel = "task_#{task._id}"
+    # Clear expired tasks
+    sessionContext.tasks = sessionContext.tasks.filter((task) ->
+      return true if not (task.expiry <= sessionContext.timestamp)
 
-      # Delete if expired
-      if task.expiry <= sessionContext.timestamp
       logger.accessInfo({
         sessionContext: sessionContext
         message: 'Task expired'
         task: task
       })
       sessionContext.doneTasks ?= []
-        sessionContext.doneTasks.push(String(task._id))
+      sessionContext.doneTasks.push(task._id)
+      channel = "task_#{task._id}"
       if channel of sessionContext.faults
         delete sessionContext.faults[channel]
         sessionContext.faultsTouched ?= {}
         sessionContext.faultsTouched[channel] = true
-        continue
 
-      if sessionContext.faults[channel]
-        continue
+      return false
+    )
+
+    task = sessionContext.tasks.find((task) -> !sessionContext.faults["task_#{task._id}"])
+
+    return applyPresets(sessionContext) if not task
 
     switch task.name
       when 'getParameterValues'
@@ -426,58 +465,94 @@ nextRpc = (sessionContext) ->
         session.addProvisions(sessionContext, "task_#{task._id}",
           [['instances', task.objectName, 0]]);
       else
-          return throwError(new Error('Task name not recognized'), sessionContext.httpResponse) if err
+        return throwError(new Error('Task name not recognized'), sessionContext.httpResponse)
 
     return nextRpc(sessionContext)
-
-    return applyPresets(sessionContext)
   )
 
 
-sendAcsRequest = (sessionContext, id, acsRequest) ->
-  if not acsRequest?
-    session.end(sessionContext, (err, isNew) ->
-      return throwError(err, sessionContext.httpResponse) if err
-
-      delete currentSessions.get(sessionContext.httpRequest.connection)[sessionContext.sessionId]
-      if isNew
-        logger.accessInfo({
-          sessionContext: sessionContext
-          message: 'New device registered'
-        })
-
-      db.clearTasks(sessionContext.deviceId, sessionContext.doneTasks, (err) ->
-        return throwError(err, sessionContext.httpResponse) if err
+endSession = (sessionContext, callback) ->
+  saveCache = sessionContext.cacheUntil?
 
   counter = 3
 
+  counter += 2
+  db.saveDevice(sessionContext.deviceId, sessionContext.deviceData, sessionContext.new, sessionContext.timestamp, (err) ->
+    if err
+      callback(err) if counter & 1
+      return counter = 0
+
+    return callback(null, sessionContext.new) if (counter -= 2) == 1
+  )
+
+  for k of sessionContext.operationsTouched
+    counter += 2
+    saveCache = true
+    if sessionContext.operations[k]?
+      db.saveOperation(sessionContext.deviceId, k, sessionContext.operations[k], (err) ->
+        if err
+          callback(err) if counter & 1
+          return counter = 0
+
+        return callback(null, sessionContext.new) if (counter -= 2) == 1
+      )
+    else
+      db.deleteOperation(sessionContext.deviceId, k, (err) ->
+        if err
+          callback(err) if counter & 1
+          return counter = 0
+
+        return callback(null, sessionContext.new) if (counter -= 2) == 1
+      )
+
+  if sessionContext.doneTasks?.length
+    counter += 2
+    saveCache = true
+    db.clearTasks(sessionContext.deviceId, sessionContext.doneTasks, (err) ->
+      if err
+        callback(err) if counter & 1
+        return counter = 0
+
+      return callback(null, sessionContext.new) if (counter -= 2) == 1
+    )
+
   for k of sessionContext.faultsTouched
     counter += 2
+    saveCache = true
     if sessionContext.faults[k]
       sessionContext.faults[k].retries = sessionContext.retries[k]
       db.saveFault(sessionContext.deviceId, k, sessionContext.faults[k], (err) ->
         if err
-                throwError(err, sessionContext.httpResponse) if counter & 1
+          callback(err) if counter & 1
           return counter = 0
 
-              if (counter -= 2) == 1
-                return writeResponse(sessionContext, soap.response(null))
+        return callback(null, sessionContext.new) if (counter -= 2) == 1
       )
     else
       db.deleteFault(sessionContext.deviceId, k, (err) ->
         if err
-                throwError(err, sessionContext.httpResponse) if counter & 1
+          callback(err) if counter & 1
           return counter = 0
 
-              if (counter -= 2) == 1
-                return writeResponse(sessionContext, soap.response(null))
+        return callback(null, sessionContext.new) if (counter -= 2) == 1
       )
 
-        if (counter -= 2) == 1
-          return writeResponse(sessionContext, soap.response(null))
+  if saveCache
+    counter += 2
+    cacheDueTasksAndFaultsAndOperations(sessionContext.deviceId, sessionContext.tasks, sessionContext.faults, sessionContext.operations, sessionContext.cacheUntil, (err) ->
+      if err
+        callback(err) if counter & 1
+        return counter = 0
+
+      return callback(null, sessionContext.new) if (counter -= 2) == 1
     )
-    )
-    return
+
+  callback(null, sessionContext.new) if (counter -= 2) == 1
+
+
+sendAcsRequest = (sessionContext, id, acsRequest) ->
+  if not acsRequest?
+    return writeResponse(sessionContext, soap.response(null), true)
 
   if acsRequest.name is 'Download'
     if not acsRequest.url?
@@ -487,10 +562,10 @@ sendAcsRequest = (sessionContext, id, acsRequest) ->
       acsRequest.url = if FS_SSL then 'https://' else 'http://'
       acsRequest.url += FS_HOSTNAME
       acsRequest.url += ":#{FS_PORT}" if FS_PORT != 80
-      acsRequest.url += "/#{encodeURIComponent(acsRequest.fileName)}"
+      acsRequest.url += "/#{encodeURI(acsRequest.fileName)}"
 
     if not acsRequest.fileSize?
-      return cache.getFiles((err, hash, files) ->
+      return localCache.getFiles((err, hash, files) ->
         return throwError(err, sessionContext.httpResponse) if err
 
         if acsRequest.fileName of files
@@ -518,145 +593,147 @@ sendAcsRequest = (sessionContext, id, acsRequest) ->
   writeResponse(sessionContext, res)
 
 
-getSession = (httpRequest, callback) ->
-  # Separation by comma is important as some devices don't comform to standard
-  COOKIE_REGEX = /\s*([a-zA-Z0-9\-_]+?)\s*=\s*"?([a-zA-Z0-9\-_]*?)"?\s*(,|;|$)/g
-  while match = COOKIE_REGEX.exec(httpRequest.headers.cookie)
-    sessionId = match[2] if match[1] == 'session'
+getSession = (connection, sessionId, callback) ->
+  sessionContext = currentSessions.get(connection)
 
-  return callback() if not sessionId?
-
-  sessionContext = currentSessions.get(httpRequest.connection)?[sessionId]
-
-  if sessionContext?
+  if sessionContext
+    currentSessions.delete(connection)
     return callback(null, sessionContext)
 
+  return callback() if not sessionId
+
   setTimeout(() ->
-    db.redisClient.eval('local v=redis.call("get",KEYS[1]);redis.call("del",KEYS[1]);return v;', 1, "session_#{sessionId}", (err, sessionContextString) ->
+    cache.pop("session_#{sessionId}", (err, sessionContextString) ->
       return callback(err) if err or not sessionContextString
-      session.deserialize(sessionContextString, (err, sessionContext) ->
-        return callback(err) if err
-        currentSessions.get(httpRequest.connection)[sessionId] = sessionContext
-        callback(null, sessionContext)
-      )
-    )
-  , 100
+      session.deserialize(sessionContextString, callback)
     )
+  , 100)
 
 
 currentSessions = new WeakMap()
 
-# When socket closes, store active sessions in redis
+# When socket closes, store active sessions in cache
 onConnection = (socket) ->
-  currentSessions.set(socket, {})
+  # The property remoteAddress may be undefined after the connection is
+  # closed, unless we read it at least once (caching?)
+  socket.remoteAddress
+
   socket.on('close', () ->
-    sessions = currentSessions.get(socket)
-    for sessionId, sessionContext of sessions
-      delete sessionContext.httpRequest
-      delete sessionContext.httpResponse
+    sessionContext = currentSessions.get(socket)
+    return if not sessionContext
+    currentSessions.delete(socket)
+    now = Date.now()
+
+    lastActivity = sessionContext.lastActivity
+    timeoutMsg = logger.flatten({
+      sessionContext: sessionContext
+      message: 'Session timeout'
+      sessionTimestamp: sessionContext.timestamp
+    })
+
+    timeout = (sessionContext.lastActivity + sessionContext.timeout * 1000) - now
+    if timeout <= 0
+      return logger.accessError(timeoutMsg)
+
+    setTimeout(() ->
+      cache.get("session_#{sessionContext.sessionId}", (err, sessionContextString) ->
+        return throwError(err) if err
+        return if not sessionContextString
+        session.deserialize(sessionContextString, (err, sessionContext) ->
+          return throwError(err) if err
+          if sessionContext.lastActivity == lastActivity
+            logger.accessError(timeoutMsg)
+        )
+      )
+    , timeout + 1000).unref()
+
     session.serialize(sessionContext, (err, sessionContextString) ->
       return throwError(err) if err
-        # TODO don't set if process is shutting down
-        db.redisClient.setex("session_#{sessionId}", sessionContext.timeout, sessionContextString, (err) ->
+      cache.set("session_#{sessionContext.sessionId}", sessionContextString, Math.ceil(timeout / 1000) + 3, (err) ->
         return throwError(err) if err
       )
     )
   )
 
+setInterval(() ->
+  if stats.droppedRequests
+    logger.warn({
+      message: 'Worker overloaded'
+      droppedRequests: stats.droppedRequests
+      totalRequests: stats.totalRequests
+      initiatedSessions: stats.initiatedSessions
+      pid: process.pid
+    })
 
-listener = (httpRequest, httpResponse) ->
-  if httpRequest.method != 'POST'
-    httpResponse.writeHead 405, {'Allow': 'POST'}
-    httpResponse.end('405 Method Not Allowed')
-    return
+  stats.totalRequests = 0
+  stats.droppedRequests = 0
+  stats.initiatedSessions = 0
+, 10000).unref()
 
-  if httpRequest.headers['content-encoding']?
-    switch httpRequest.headers['content-encoding']
-      when 'gzip'
-        stream = httpRequest.pipe(zlib.createGunzip())
-      when 'deflate'
-        stream = httpRequest.pipe(zlib.createInflate())
+
+getDueTasksAndFaultsAndOperations = (deviceId, timestamp, callback) ->
+  cache.get("#{deviceId}_tasks_faults_operations", (err, res) ->
+    return callback(err) if err
+
+    if res
+      res = JSON.parse(res)
+      return callback(null, res.tasks or [], res.faults or {}, res.operations or {})
+
+    faults = tasks = operations = cacheUntil = null
+
+    db.getFaults(deviceId, (err, _faults) ->
+      if err
+        callback?(err)
+        return callback = null
+
+      faults = _faults
+      if tasks? and operations?
+        return callback(null, tasks, faults, operations, cacheUntil)
+    )
+
+    db.getDueTasks(deviceId, timestamp, (err, dueTasks, nextTimestamp) ->
+      if err
+        callback?(err)
+        return callback = null
+
+      tasks = dueTasks
+      cacheUntil = nextTimestamp or 0
+
+      if faults? and operations?
+        return callback(null, tasks, faults, operations, cacheUntil)
+    )
+
+    db.getOperations(deviceId, (err, _operations) ->
+      if err
+        callback?(err)
+        return callback = null
+
+      operations = _operations
+
+      if faults? and tasks?
+        return callback(null, tasks, faults, operations, cacheUntil)
+    )
+
+    if faults? and tasks? and operations?
+      return callback(null, tasks, faults, operations, cacheUntil)
+  )
+
+
+cacheDueTasksAndFaultsAndOperations = (deviceId, tasks, faults, operations, cacheUntil, callback) ->
+  v = {}
+  v.tasks = tasks if tasks.length
+  v.faults = faults if Object.keys(faults).length
+  v.operations = operations if Object.keys(operations).length
+
+  if cacheUntil
+    ttl = Math.trunc((Date.now() - cacheUntil) / 1000)
   else
-        httpResponse.writeHead(415)
-        httpResponse.end('415 Unsupported Media Type')
-        return
-  else
-    stream = httpRequest
+    ttl = config.get('MAX_CACHE_TTL', deviceId)
 
-  chunks = []
-  bytes = 0
+  cache.set("#{deviceId}_tasks_faults_operations", JSON.stringify(v), ttl, callback)
 
-  stream.on('data', (chunk) ->
-    chunks.push(chunk)
-    bytes += chunk.length
-  )
-
-  httpRequest.getBody = () ->
-    # Write all chunks into a Buffer
-    body = new Buffer(bytes)
-    offset = 0
-    chunks.forEach((chunk) ->
-      chunk.copy(body, offset, 0, chunk.length)
-      offset += chunk.length
-    )
-    return body
-
-  stream.on('end', () ->
-    rpc = null
-    parseWarnings = []
-    getSession(httpRequest, f = (err, sessionContext) ->
-      return throwError(err, httpResponse) if err
-      rpc ?= soap.request(httpRequest, sessionContext?.cwmpVersion, parseWarnings)
-      if not sessionContext?
-        if rpc.cpeRequest?.name isnt 'Inform'
-          httpResponse.writeHead(400)
-          httpResponse.end('Session is expired')
-          return
-
-        deviceId = common.generateDeviceId(rpc.cpeRequest.deviceId)
-        return session.init(deviceId, rpc.cwmpVersion, rpc.sessionTimeout ? config.get('SESSION_TIMEOUT', deviceId), (err, sessionContext) ->
-          return throwError(err, httpResponse) if err
-
-          sessionContext.sessionId = crypto.randomBytes(8).toString('hex')
-
-          currentSessions.get(httpRequest.connection)[sessionContext.sessionId] = sessionContext
-
-          httpRequest.connection.setTimeout(sessionContext.timeout * 1000)
-
-          db.getDueTasksAndFaultsAndOperations(deviceId, sessionContext.timestamp, (err, dueTasks, faults, operations) ->
-            return throwError(err, httpResponse) if err
-            sessionContext.tasks = dueTasks
-            sessionContext.faults = faults
-            sessionContext.retries = {}
-            for k, v of faults
-              sessionContext.retries[k] = v.retries
-            sessionContext.operations = operations
-
-            # Delete expired faults
-            for k, v of sessionContext.faults
-              if v.expiry >= sessionContext.timestamp
-                delete sessionContext.faults[k]
-                sessionContext.faultsTouched ?= {}
-                sessionContext.faultsTouched[k] = true
-
-            f(null, sessionContext)
-          )
-        )
-
-      sessionContext.httpRequest = httpRequest
-      sessionContext.httpResponse = httpResponse
-
-      for w in parseWarnings
-        w.sessionContext = sessionContext
-        w.rpc = rpc;
-        logger.accessWarn(w)
-
-      if config.get('DEBUG', sessionContext.deviceId)
-        dump = "# REQUEST #{new Date(Date.now())}\n" + JSON.stringify(httpRequest.headers) + "\n#{httpRequest.getBody()}\n\n"
-        require('fs').appendFile("./debug/#{sessionContext.deviceId}.dump", dump, (err) ->
-          return throwError(err) if err
-        )
 
+processRequest = (sessionContext, rpc) ->
   if rpc.cpeRequest?
     if rpc.cpeRequest.name is 'Inform'
       logger.accessInfo({
@@ -717,6 +794,169 @@ listener = (httpRequest, httpResponse) ->
 
       nextRpc(sessionContext)
     )
+
+
+listener = (httpRequest, httpResponse) ->
+  stats.totalRequests += 1
+  if httpRequest.method != 'POST'
+    httpResponse.writeHead 405, {'Allow': 'POST', 'Connection': 'close'}
+    httpResponse.end('405 Method Not Allowed')
+    return
+
+  sessionId = null
+  # Separation by comma is important as some devices don't comform to standard
+  COOKIE_REGEX = /\s*([a-zA-Z0-9\-_]+?)\s*=\s*"?([a-zA-Z0-9\-_]*?)"?\s*(,|;|$)/g
+  while match = COOKIE_REGEX.exec(httpRequest.headers.cookie)
+    sessionId = match[2] if match[1] == 'session'
+
+  # If overloaded, ask CPE to retry in 60 seconds
+  if not sessionId and stats.concurrentRequests > MAX_CONCURRENT_REQUESTS
+    httpResponse.writeHead(503, {'Retry-after': 60, 'Connection': 'close'})
+    httpResponse.end('503 Service Unavailable')
+    stats.droppedRequests += 1
+    return
+
+  if httpRequest.headers['content-encoding']?
+    switch httpRequest.headers['content-encoding']
+      when 'gzip'
+        stream = httpRequest.pipe(zlib.createGunzip())
+      when 'deflate'
+        stream = httpRequest.pipe(zlib.createInflate())
+      else
+        httpResponse.writeHead(415, {'Connection': 'close'})
+        httpResponse.end('415 Unsupported Media Type')
+        return
+  else
+    stream = httpRequest
+
+  stats.concurrentRequests += 1
+  httpRequest.on('aborted', () ->
+    stats.concurrentRequests -= 1
+    # In some cases event end can be emitted after aborted event
+    httpRequest.removeAllListeners('end')
+  )
+
+  chunks = []
+  bytes = 0
+
+  stream.on('data', (chunk) ->
+    chunks.push(chunk)
+    bytes += chunk.length
+  )
+
+  stream.on('end', () ->
+    body = new Buffer(bytes)
+    offset = 0
+    chunks.forEach((chunk) ->
+      chunk.copy(body, offset, 0, chunk.length)
+      offset += chunk.length
+    )
+
+    parsedRpc = (sessionContext, rpc, parseWarnings) ->
+      for w in parseWarnings
+        w.sessionContext = sessionContext
+        w.rpc = rpc;
+        logger.accessWarn(w)
+
+      if config.get('DEBUG', sessionContext.deviceId)
+        dump = "# REQUEST #{new Date(Date.now())}\n" + JSON.stringify(httpRequest.headers) + "\n#{body}\n\n"
+        require('fs').appendFile("./debug/#{sessionContext.deviceId}.dump", dump, (err) ->
+          return throwError(err) if err
+        )
+
+      processRequest(sessionContext, rpc)
+
+    getSession(httpRequest.connection, sessionId, (err, sessionContext) ->
+      return throwError(err, httpResponse) if err
+
+      if sessionContext
+        sessionContext.httpRequest = httpRequest
+        sessionContext.httpResponse = httpResponse
+
+        if sessionContext.sessionId != sessionId or
+            sessionContext.lastActivity + sessionContext.timeout * 1000 < Date.now()
+          logger.accessError({message: 'Invalid session', sessionContext: sessionContext})
+          httpResponse.writeHead(400, {'Connection': 'close'})
+          httpResponse.end('Invalid session')
+          stats.concurrentRequests -= 1
+          return
+      else if stats.concurrentRequests > MAX_CONCURRENT_REQUESTS
+        # Check again just in case device included old session ID from the previous session
+        httpResponse.writeHead(503, {'Retry-after': 60, 'Connection': 'close'})
+        httpResponse.end('503 Service Unavailable')
+        stats.droppedRequests += 1
+        stats.concurrentRequests -= 1
+        return
+
+      parseWarnings = []
+      try
+        rpc = soap.request(body, sessionContext?.cwmpVersion, parseWarnings)
+      catch err
+        logger.accessError({
+          message: 'XML parse error'
+          parseError: err.message.trim()
+          sessionContext: sessionContext or {httpRequest: httpRequest, httpResponse: httpResponse}
+        })
+        httpResponse.writeHead(400, {'Connection': 'close'})
+        httpResponse.end(err.message)
+        stats.concurrentRequests -= 1
+        return
+
+      if sessionContext
+        if rpc.cpeRequest?.name is 'Inform' or
+            (!sessionContext.rpcRequest ^ !(rpc.cpeResponse or rpc.cpeFault))
+          logger.accessError({
+            message: 'Bad session state'
+            sessionContext: sessionContext
+          })
+          httpResponse.writeHead(400, {'Connection': 'close'})
+          httpResponse.end('Bad session state')
+          stats.concurrentRequests -= 1
+          return
+
+        return parsedRpc(sessionContext, rpc, parseWarnings)
+
+      if rpc.cpeRequest?.name isnt 'Inform'
+        logger.accessError({
+          message: 'Invalid session'
+          sessionContext: sessionContext or {httpRequest: httpRequest, httpResponse: httpResponse}
+        })
+        httpResponse.writeHead(400, {'Connection': 'close'})
+        httpResponse.end('Invalid session')
+        stats.concurrentRequests -= 1
+        return
+
+      stats.initiatedSessions += 1
+      deviceId = common.generateDeviceId(rpc.cpeRequest.deviceId)
+      return session.init(deviceId, rpc.cwmpVersion, rpc.sessionTimeout ? config.get('SESSION_TIMEOUT', deviceId), (err, sessionContext) ->
+        return throwError(err, httpResponse) if err
+
+        sessionContext.httpRequest = httpRequest
+        sessionContext.httpResponse = httpResponse
+        sessionContext.sessionId = crypto.randomBytes(8).toString('hex')
+        httpRequest.connection.setTimeout(sessionContext.timeout * 1000)
+
+        getDueTasksAndFaultsAndOperations(deviceId, sessionContext.timestamp, (err, dueTasks, faults, operations, cacheUntil) ->
+          return throwError(err, httpResponse) if err
+
+          sessionContext.tasks = dueTasks
+          sessionContext.faults = faults
+          sessionContext.retries = {}
+          for k, v of faults
+            sessionContext.retries[k] = v.retries
+          sessionContext.operations = operations
+          sessionContext.cacheUntil = cacheUntil
+
+          # Delete expired faults
+          for k, v of sessionContext.faults
+            if v.expiry >= sessionContext.timestamp
+              delete sessionContext.faults[k]
+              sessionContext.faultsTouched ?= {}
+              sessionContext.faultsTouched[k] = true
+
+          parsedRpc(sessionContext, rpc, parseWarnings)
+        )
+      )
     )
   )
 

lib/db.coffee

@@ -19,10 +19,8 @@
 
 config = require './config'
 mongodb = require 'mongodb'
-redis = require('redis')
 common = require './common'
 
-redisClient = null
 tasksCollection = null
 devicesCollection = null
 presetsCollection = null
@@ -35,7 +33,7 @@ operationsCollection = null
 
 
 connect = (callback) ->
-  callbackCounter = 10
+  callbackCounter = 9
   mongodb.MongoClient.connect(config.get('MONGODB_CONNECTION_URL'), (err, db) ->
     return callback(err) if err
     exports.mongoDb = db
@@ -105,25 +103,21 @@ connect = (callback) ->
         return callback(err)
     )
 
-    exports.redisClient = redisClient = redis.createClient(config.get('REDIS_PORT'), config.get('REDIS_HOST'))
-    redisClient.select(config.get('REDIS_DB'), (err) ->
-      if --callbackCounter == 0 or err
-        callbackCounter = 0
-        return callback(err)
-    )
   )
 
 
 disconnect = () ->
   exports.mongoDb?.close()
-  redisClient?.quit()
 
 
 # Optimize projection by removing overlaps
-# This modifies given object and returns it
+# This can modify the object
 optimizeProjection = (obj) ->
+  if obj['']
+    return {'': obj['']}
+
   keys = Object.keys(obj).sort()
-  return if keys.length <= 1
+  return obj if keys.length <= 1
   i = 1
   while i < keys.length
     a = keys[i-1]
@@ -139,21 +133,26 @@ optimizeProjection = (obj) ->
 fetchDevice = (id, timestamp, patterns, callback) ->
   MAX_DEPTH = config.get('MAX_DEPTH', id)
 
-  projection = {_id: 1}
+  if not patterns?.length
+    patterns = [[[], (1 << MAX_DEPTH) - 1]]
+
+  projection = {}
   projectionTree = {}
   func = (path, pats, projTree) ->
     children = {}
     for pat in pats
       fragment = pat[0][path.length] or '*'
-      load = 0
+
       if fragment == '*'
-        projection[path.join('.')] = 1
+        projection[path.join('.')] = 1 if pat[1] << path.length
         return
 
       projTree[fragment] ?= {}
 
       if pat[1] & (1 << path.length)
         projection[path.concat('_timestamp').join('.')] = 1
+        projection[path.concat('_object').join('.')] = 1
+        projection[path.concat('_instance').join('.')] = 1
         projection[path.concat([fragment, '_timestamp']).join('.')] = 1
         projection[path.concat([fragment, '_value']).join('.')] = 1
         projection[path.concat([fragment, '_type']).join('.')] = 1
@@ -175,47 +174,80 @@ fetchDevice = (id, timestamp, patterns, callback) ->
   delete projectionTree['Events']
   delete projectionTree['Tags']
 
-  if projection['']?
-    proj = {}
-  else
-    proj = JSON.parse(JSON.stringify(projection))
-
-  for k, v of proj
-    if k == '' or k == 'Events' or k.startsWith('Events.')
-      if k.length < 7 or k.startsWith('Events.Inform.') or k == 'Events.Inform'
-        proj['_lastInform'] = 1
-      if k.length < 7 or k.startsWith('Events.1_BOOT.') or k == 'Events.1_BOOT'
-        proj['_lastBoot'] = 1
-      if k.length < 7 or k.startsWith('Events.0_BOOTSTRAP.') or k == 'Events.0_BOOTSTRAP'
-        proj['_lastBootstrap'] = 1
-      if k.length < 7 or k.startsWith('Events.Registered.') or k == 'Events.Registered'
-        proj['_registered'] = 1
-      delete proj[k] if k != ''
-
-    if k == '' or k == 'DeviceID' or k.startsWith('DeviceID.')
-      if k.length < 9 or k.startsWith('DeviceID.ID.') or k == 'DeviceID.ID'
-        proj['_id'] = 1
-      if k.length < 9 or k.startsWith('DeviceID.Manufacturer.') or k == 'DeviceID.DeManufacturer'
-        proj['_deviceId._Manufacturer'] = 1
-      if k.length < 9 or k.startsWith('DeviceID.ProductClass.') or k == 'DeviceID.ProductClass'
-        proj['_deviceId._ProductClass'] = 1
-      if k.length < 9 or k.startsWith('DeviceID.OUI.') or k == 'DeviceID.OUI'
-        proj['_deviceId._OUI'] = 1
-      if k.length < 9 or k.startsWith('DeviceID.SerialNumber.') or k == 'DeviceID.SerialNumber'
-        proj['_deviceId._SerialNumber'] = 1
-      if k.length < 9
-        proj['_deviceId'] = 1
-      delete proj[k] if k != ''
-
-    if k == '' or k == 'Tags' or k.startsWith('Tags.')
-      proj['_tags'] = 1
-      delete proj[k] if k != ''
-
-  optimizeProjection(proj)
-
   res = []
   loaded = []
 
+  projection = optimizeProjection(projection)
+
+  for k, v of projection
+    if k == '' or k == 'Events' or k.startsWith('Events.')
+      if k == '' or k == 'Events' or k == 'Events._writable'
+        res.push([['Events'], timestamp,
+          {object: [timestamp, 1], writable: [timestamp, 0]}])
+        loaded.push([['Events'], 1]) if k == 'Events._writable'
+      if k == 'Events'
+        projection['_lastInform'] = 1
+        projection['_lastBoot'] = 1
+        projection['_lastBootstrap'] = 1
+        projection['_registered'] = 1
+        loaded.push([['Events'], ((1 << MAX_DEPTH) - 1)])
+      else if k == 'Events.Inform._writable' or k == 'Events.Inform'
+        projection['_lastInform'] = 1
+        loaded.push([['Events', 'Inform'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k == 'Events.1_BOOT._writable' or k == 'Events.1_BOOT'
+        projection['_lastBoot'] = 1
+        loaded.push([['Events', '1_BOOT'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k == 'Events.0_BOOTSTRAP._writable' or k == 'Events.0_BOOTSTRAP'
+        projection['_lastBootstrap'] = 1
+        loaded.push([['Events', '0_BOOTSTRAP'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k == 'Events.Registered._writable' or k == 'Events.Registered'
+        projection['_registered'] = 1
+        loaded.push([['Events', 'Registered'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k.endsWith('._writable') and k != 'Events._writable'
+        loaded.push([k.split('.').slice(0, 2), 1 ^ ((1 << MAX_DEPTH) - 1)])
+      delete projection[k] if k != ''
+
+    if k == '' or k == 'DeviceID' or k.startsWith('DeviceID.')
+      if k == '' or k == 'DeviceID' or k == 'DeviceID._writable'
+        res.push([['DeviceID'], timestamp,
+          {object: [timestamp, 1], writable: [timestamp, 0]}])
+        loaded.push([['DeviceID'], 1]) if k == 'DeviceID._writable'
+      if k == 'DeviceID'
+        projection['_id'] = 1
+        projection['_deviceId'] = 1
+        loaded.push([['DeviceID'], ((1 << MAX_DEPTH) - 1)])
+      else if k == 'DeviceID.ID._writable' or k == 'DeviceID.ID'
+        projection['_id'] = 1
+        loaded.push([['DeviceID', 'ID'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k == 'DeviceID.Manufacturer._writable' or k == 'DeviceID.DeManufacturer'
+        projection['_deviceId._Manufacturer'] = 1
+        loaded.push([['DeviceID', 'Manufacturer'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k == 'DeviceID.ProductClass._writable' or k == 'DeviceID.ProductClass'
+        projection['_deviceId._ProductClass'] = 1
+        loaded.push([['DeviceID', 'ProductClass'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k == 'DeviceID.OUI._writable' or k == 'DeviceID.OUI'
+        projection['_deviceId._OUI'] = 1
+        loaded.push([['DeviceID', 'ProductClass'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k == 'DeviceID.SerialNumber._writable' or k == 'DeviceID.SerialNumber'
+        projection['_deviceId._SerialNumber'] = 1
+        loaded.push([['DeviceID', 'SerialNumber'], 1 ^ ((1 << MAX_DEPTH) - 1)])
+      else if k.endsWith('._writable') and k != 'DeviceID._writable'
+        loaded.push([k.split('.').slice(0, 2), 1 ^ ((1 << MAX_DEPTH) - 1)])
+      delete projection[k] if k != ''
+
+    if k == 'Tags' or k.startsWith('Tags.')
+      if not projection['_tags']
+        projection['_tags'] = 1
+        loaded.push([['Tags'], ((1 << MAX_DEPTH) - 1)])
+      delete projection[k]
+
+  if projection['']
+    proj = {}
+  else if Object.keys(projection).length == 0
+    proj = {'_id': 1}
+  else
+    proj = projection
+
   devicesCollection.findOne({'_id' : id}, proj, (err, device) ->
     return callback(err) if err or not device?
 
@@ -225,18 +257,27 @@ fetchDevice = (id, timestamp, patterns, callback) ->
       else
         if projection[path.join('.')]
           descendantsFetched = true
+          if path.length and projection[path.slice(0, -1).concat("_timestamp").join('.')]
+            thisFetched = true
+            loaded.push([path, ((1 << (path.length - 1)) - 1) ^ ((1 << MAX_DEPTH) - 1)])
+          else
             loaded.push([path, ((1 << path.length) - 1) ^ ((1 << MAX_DEPTH) - 1)])
-        if projection[path.concat('_writable').join('.')]
+        else if projection[path.concat('_writable').join('.')]
           loaded.push([path, 1 << (path.length - 1)])
           thisFetched = true
 
       if obj['_timestamp']?
         obj['_timestamp'] = +obj['_timestamp']
 
+      # For compatibility with v1.0 database
+      if obj['_instance'] and not obj['_object']?
+        obj['_object'] = true
+
       if thisFetched
         attrs = {}
 
-        t = if obj['_timestamp'] > timestamp then obj['_timestamp'] else timestamp
+        t = obj['_timestamp'] or 1
+        t = timestamp if timestamp > t
 
         if obj['_value']?
           attrs.value = [obj['_timestamp'] or 1, [obj['_value'], obj['_type']]]
@@ -244,63 +285,55 @@ fetchDevice = (id, timestamp, patterns, callback) ->
           obj['_object'] = false
 
         if obj['_writable']?
-          attrs.writable = [timestamp, if obj['_writable'] then 1 else 0]
+          attrs.writable = [timestamp or 1, if obj['_writable'] then 1 else 0]
 
-        if obj['_object']? or obj['_instance']?
-          attrs.object = [t, if (obj['_object'] or obj['_instance']) then 1 else 0]
+        if obj['_object']?
+          attrs.object = [t, if obj['_object'] then 1 else 0]
 
         res.push([path, t, attrs])
 
       for k, v of obj
-        if k[0] != '_'
+        if not k.startsWith('_')
           kk = k
           obj['_object'] = true
-          storeParams(v, path.concat(k), obj['_timestamp'] or 1, descendantsFetched, projTree?[k])
+          storeParams(v, path.concat(k), obj['_timestamp'], descendantsFetched, projTree?[k])
           delete projTree[kk] if projTree
 
-      if obj['_object'] or obj['_instance']
-        if descendantsFetched
-          res.push([path.concat('*'), obj['_timestamp']]) if obj['_timestamp']
-        else
+      if not descendantsFetched
         for k, v of projTree
           p = path.concat(k)
-            res.push([p, obj['_timestamp']]) if obj['_timestamp']
           loaded.push([p, ((1 << path.length) - 1) ^ ((1 << MAX_DEPTH) - 1)])
+          if (obj['_object'] or path.length == 0) and obj['_timestamp']
+            res.push([p, obj['_timestamp']])
+      else if (obj['_object'] or path.length == 0) and obj['_timestamp']
+          res.push([path.concat('*'), obj['_timestamp']])
 
     for k, v of device
       switch k
         when '_lastInform'
-          res.push([['Events'], timestamp,
-            {object: [timestamp, 1], writable: [timestamp, 0]}])
           res.push([['Events', 'Inform'], +v,
             {object: [+v, 0], writable: [+v, 0], value: [+v, [+v, 'xsd:dateTime']]}])
           delete device[k]
         when '_lastBoot'
-          res.push([['Events'], timestamp,
-            {object: [timestamp, 1], writable: [timestamp, 0]}])
           res.push([['Events', '1_BOOT'], +v,
             {object: [+v, 0], writable: [+v, 0], value: [+v, [+v, 'xsd:dateTime']]}])
           delete device[k]
         when '_lastBootstrap'
-          res.push([['Events'], timestamp,
-            {object: [timestamp, 1], writable: [timestamp, 0]}])
           res.push([['Events', '0_BOOTSTRAP'], +v,
             {object: [+v, 0], writable: [+v, 0], value: [+v, [+v, 'xsd:dateTime']]}])
           delete device[k]
         when '_registered'
-          res.push([['Events'], timestamp,
-            {object: [timestamp, 1], writable: [timestamp, 0]}])
           # Use current timestamp for registered event attribute timestamps
           res.push([['Events', 'Registered'], timestamp,
             {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [+v, 'xsd:dateTime']]}])
           delete device[k]
         when '_id'
-          res.push([['DeviceID'], timestamp,
-            {object: [timestamp, 1], writable: [timestamp, 0]}])
+          if projection[''] or projection['_id']
             res.push([['DeviceID', 'ID'], timestamp,
               {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v, 'xsd:string']]}])
           delete device[k]
         when '_tags'
+          if v.length
             res.push([['Tags'], timestamp,
               {object: [timestamp, 1], writable: [timestamp, 0]}])
           for t in v
@@ -309,8 +342,6 @@ fetchDevice = (id, timestamp, patterns, callback) ->
               {object: [timestamp, 0], writable: [timestamp, 1], value: [timestamp, [true, 'xsd:boolean']]}])
           delete device[k]
         when '_deviceId'
-          res.push([['DeviceID'], timestamp,
-            {object: [timestamp, 1], writable: [timestamp, 0]}])
           if v['_Manufacturer']?
             res.push([['DeviceID', 'Manufacturer'], timestamp,
               {object: [timestamp, 0], writable: [timestamp, 0], value: [timestamp, [v['_Manufacturer'], 'xsd:string']]}])
@@ -331,7 +362,7 @@ fetchDevice = (id, timestamp, patterns, callback) ->
   )
 
 
-saveDevice = (deviceId, deviceData, isNew, callback) ->
+saveDevice = (deviceId, deviceData, isNew, sessionTimestamp, callback) ->
   update = {'$set' : {}, '$unset' : {}, '$addToSet' : {}, '$pull' : {}}
 
   iter = deviceData.timestamps.diff()
@@ -339,6 +370,10 @@ saveDevice = (deviceId, deviceData, isNew, callback) ->
     continue if diff[0].wildcard != (1 << (diff[0].length - 1))
     continue if diff[0][0] in ['Events', 'DeviceID', 'Tags']
 
+    # Param timestamps may be greater than session timestamp to track revisions
+    if diff[2] > sessionTimestamp
+      diff[2] = sessionTimestamp
+
     if not diff[2]? and diff[1]?
       update['$unset'][diff[0].slice(0, -1).concat('_timestamp').join('.')] = 1
     else if diff[2] != diff[1]
@@ -403,17 +438,21 @@ saveDevice = (deviceId, deviceData, isNew, callback) ->
           continue
 
         for attrName of diff[2]
-          if diff[2][attrName][1]? or diff[1]?[attrName]?[1]?
+          # Param timestamps may be greater than session timestamp to track revisions
+          if diff[2][attrName][0] > sessionTimestamp
+            diff[2][attrName][0] = sessionTimestamp
+
+          if diff[2][attrName][1]?
             switch attrName
               when 'value'
-                if not diff[1] or diff[2].value[1][0] != diff[1].value?[1][0]
-                  if diff[2].value[1][1] == 'xsd:dateTime'
+                if diff[2].value[1][0] != diff[1]?.value?[1]?[0]
+                  if diff[2].value[1][1] == 'xsd:dateTime' and Number.isInteger(diff[2].value[1][0])
                     update['$set'][path.concat('_value').join('.')] = new Date(diff[2].value[1][0])
                   else
                     update['$set'][path.concat('_value').join('.')] = diff[2].value[1][0]
-                if not diff[1] or diff[2].value[1][1] != diff[1].value?[1][1]
+                if diff[2].value[1][1] != diff[1]?.value?[1]?[1]
                   update['$set'][path.concat('_type').join('.')] = diff[2].value[1][1]
-                if not diff[1] or diff[2].value[0] != diff[1].value?[0]
+                if diff[2].value[0] != diff[1]?.value?[0]
                   update['$set'][path.concat('_timestamp').join('.')] = new Date(diff[2].value[0])
               when 'object'
                 if not diff[1]?.object or diff[2].object[1] != diff[1].object?[1]
@@ -424,12 +463,21 @@ saveDevice = (deviceId, deviceData, isNew, callback) ->
 
         if diff[1]
           for attrName of diff[1]
-            if attrName not of diff[2]
+            if diff[1][attrName][1]? and not diff[2]?[attrName]?[1]?
               update['$unset'][path.concat("_#{attrName}").join('.')] = 1
               if attrName is 'value'
                 update['$unset'][path.concat('_type').join('.')] = 1
                 update['$unset'][path.concat('_timestamp').join('.')] = 1
 
+  update['$unset'] = optimizeProjection(update['$unset'])
+
+  # Remove overlap possibly caused by parameters changing from objects
+  # to regular parameters or vice versa. Reason being that _timestamp
+  # represents two different things depending on whether the parameter
+  # is an object or not.
+  for k of update['$unset']
+    delete update['$unset'][k] if update['$set'][k]?
+
   # Remove empty keys
   for k of update
     if k == '$addToSet'
@@ -442,82 +490,22 @@ saveDevice = (deviceId, deviceData, isNew, callback) ->
 
   return callback() if Object.keys(update).length == 0
 
-  if (update['$unset']?)
-    optimizeProjection(update['$unset'])
+  if update['$addToSet'] and update['$pull']
+    # Mongo doesn't allow $addToSet and $pull at the same time
+    update2 = {'$pull': update['$pull']}
+    delete update['$pull']
 
   devicesCollection.update({'_id' : deviceId}, update, {upsert: isNew}, (err, result) ->
     if not err and result.result.n != 1
       return callback(new Error("Device #{deviceId} not found in database"))
 
+    if update2
+      return devicesCollection.update({'_id': deviceId}, update2, callback)
+
     return callback(err)
   )
 
 
-getDueTasksAndFaultsAndOperations = (deviceId, timestamp, callback) ->
-  redisClient.mget("#{deviceId}_tasks", "#{deviceId}_faults", "#{deviceId}_operations", (err, res) ->
-    return callback(err) if err
-
-    if res[0]
-      tasks = JSON.parse(res[0])
-
-    if res[1]?
-      faults = JSON.parse(res[1])
-
-    if res[2]?
-      operations = JSON.parse(res[2])
-
-    if tasks? and faults? and operations?
-      return callback(null, tasks, faults, operations)
-
-    MAX_CACHE_TTL = config.get('MAX_CACHE_TTL', deviceId)
-
-    if not faults?
-      getFaults(deviceId, (err, flts) ->
-        if err
-          callback?(err)
-          return callback = null
-
-        faults = flts
-        if tasks? and operations?
-          return callback(null, tasks, faults, operations)
-      )
-
-    if not tasks?
-      getDueTasks(deviceId, timestamp, (err, dueTasks, nextTimestamp) ->
-        if err
-          callback?(err)
-          return callback = null
-
-        if nextTimestamp?
-          exp = Math.min(0, Math.trunc((nextTimestamp - Date.now()) / 1000))
-        else
-          exp = MAX_CACHE_TTL
-
-        redisClient.setex("#{deviceId}_tasks", exp, JSON.stringify(dueTasks), (err) ->
-          if err
-            callback?(err)
-            return callback = null
-
-          tasks = dueTasks
-          if faults? and operations?
-            return callback(null, tasks, faults, operations)
-        )
-      )
-
-    if not operations?
-      getOperations(deviceId, (err, _operations) ->
-        if err
-          callback?(err)
-          return callback = null
-
-        operations = _operations
-        if tasks? and faults?
-          return callback(null, tasks, faults, operations)
-      )
-
-  )
-
-
 getFaults = (deviceId, callback) ->
   faultsCollection.find({'_id' : {'$regex' : "^#{common.escapeRegExp(deviceId)}\\:"}}).toArray((err, res) ->
     return callback(err) if err
@@ -532,32 +520,22 @@ getFaults = (deviceId, callback) ->
       r.provisions = JSON.parse(r.provisions)
       faults[channel] = r
 
-    MAX_CACHE_TTL = config.get('MAX_CACHE_TTL', deviceId)
-
-    redisClient.setex("#{deviceId}_faults", MAX_CACHE_TTL, JSON.stringify(faults), (err) ->
     callback(err, faults)
   )
-  )
 
 
 saveFault = (deviceId, channel, fault, callback) ->
+  fault = Object.assign({}, fault)
   fault._id = "#{deviceId}:#{channel}"
   fault.device = deviceId
   fault.channel = channel
   fault.timestamp = new Date(fault.timestamp)
   fault.provisions = JSON.stringify(fault.provisions)
-  faultsCollection.save(fault, (err) ->
-    return callback(err) if err
-    redisClient.del("#{deviceId}_faults", callback)
-  )
+  faultsCollection.save(fault, callback)
 
 
 deleteFault = (deviceId, channel, callback) ->
-  faultsCollection.remove({_id: "#{deviceId}:#{channel}"}, (err) ->
-    return callback(err) if err
-    redisClient.del("#{deviceId}_faults", callback)
-  )
-
+  faultsCollection.remove({_id: "#{deviceId}:#{channel}"}, callback)
 
 
 getDueTasks = (deviceId, timestamp, callback) ->
@@ -600,13 +578,7 @@ getDueTasks = (deviceId, timestamp, callback) ->
 
 
 clearTasks = (deviceId, taskIds, callback) ->
-  if not taskIds?.length
-    return callback()
-
-  tasksCollection.remove({'_id' : {'$in' : (new mongodb.ObjectID(id) for id in taskIds)}}, (err) ->
-    return callback(err) if err
-    redisClient.del("#{deviceId}_tasks", callback)
-  )
+  tasksCollection.remove({'_id' : {'$in' : (new mongodb.ObjectID(id) for id in taskIds)}}, callback)
 
 
 getOperations = (deviceId, callback) ->
@@ -623,41 +595,58 @@ getOperations = (deviceId, callback) ->
       r.retries = JSON.parse(r.retries)
       operations[commandKey] = r
 
-    MAX_CACHE_TTL = config.get('MAX_CACHE_TTL', deviceId)
-
-    redisClient.setex("#{deviceId}_operations", MAX_CACHE_TTL, JSON.stringify(operations), (err) ->
     callback(err, operations)
   )
-  )
 
 
 saveOperation = (deviceId, commandKey, operation, callback) ->
+  operation = Object.assign({}, operation)
   operation._id = "#{deviceId}:#{commandKey}"
   operation.timestamp = new Date(operation.timestamp)
   operation.provisions = JSON.stringify(operation.provisions)
   operation.retries = JSON.stringify(operation.retries)
   operation.args = JSON.stringify(operation.args)
-  operationsCollection.save(operation, (err) ->
-    return callback(err) if err
-    redisClient.del("#{deviceId}_operations", callback)
-  )
+  operationsCollection.save(operation, callback)
 
 
 deleteOperation = (deviceId, commandKey, callback) ->
-  operationsCollection.remove({_id: "#{deviceId}:#{commandKey}"}, (err) ->
-    return callback(err) if err
-    redisClient.del("#{deviceId}_operations", callback)
-  )
+  operationsCollection.remove({_id: "#{deviceId}:#{commandKey}"}, callback)
+
+
+getPresets = (callback) ->
+  presetsCollection.find().toArray(callback)
+
+
+getObjects = (callback) ->
+  objectsCollection.find().toArray(callback)
+
+
+getProvisions = (callback) ->
+  provisionsCollection.find().toArray(callback)
+
+
+getVirtualParameters = (callback) ->
+  virtualParametersCollection.find().toArray(callback)
+
+
+getFiles = (callback) ->
+  filesCollection.find().toArray(callback)
 
 
 exports.connect = connect
 exports.disconnect = disconnect
 exports.fetchDevice = fetchDevice
 exports.saveDevice = saveDevice
-exports.getDueTasksAndFaultsAndOperations = getDueTasksAndFaultsAndOperations
 exports.getFaults = getFaults
 exports.saveFault = saveFault
 exports.deleteFault = deleteFault
 exports.clearTasks = clearTasks
 exports.saveOperation = saveOperation
 exports.deleteOperation = deleteOperation
+exports.getPresets = getPresets
+exports.getObjects = getObjects
+exports.getProvisions = getProvisions
+exports.getVirtualParameters = getVirtualParameters
+exports.getFiles = getFiles
+exports.getDueTasks = getDueTasks
+exports.getOperations = getOperations

lib/default-provisions.js

@@ -21,6 +21,7 @@
 const config = require("./config");
 const common = require("./common");
 const device = require("./device");
+const scheduling = require("./scheduling");
 
 const MAX_DEPTH = config.get("MAX_DEPTH");
 
@@ -31,8 +32,8 @@ function refresh(sessionContext, provision, declarations, startRevision, endRevi
   path.length = MAX_DEPTH;
   path.fill("*", l);
   let every = 1000 * (provision[2] || 1);
-  let offset = Math.abs(common.djb2(sessionContext.deviceId)) % 86400000;
-  let t = Math.trunc((sessionContext.timestamp + offset) / every) * every - offset;
+  let offset = scheduling.variance(sessionContext.deviceId, every);
+  let t = scheduling.interval(sessionContext.timestamp, every, offset);
 
   for (let i = l; i < path.length; ++ i)
     declarations.push([path.slice(0, i), t, {object: 1, writable: 1, value: t}]);
@@ -72,8 +73,8 @@ function reset(sessionContext, provision, declarations, startRevision, endRevisi
 
 
 function download(sessionContext, provision, declarations, startRevision, endRevision) {
-  let alias = ["FileType", provision[1] || "", "FileName", provision[2] || "",
-    "TargetFileName", provision[3] || ""];
+  let alias = [["FileType"], provision[1] || "", ["FileName"], provision[2] || "",
+    ["TargetFileName"], provision[3] || ""];
 
   declarations.push([["Downloads", alias], 1, {}, 1]);
   declarations.push([["Downloads", alias, "Download"], 1, {value: 1}, null,

lib/device.js

@@ -22,21 +22,37 @@ const common = require('./common');
 
 
 const CHANGE_FLAGS = {
-  object: 4,
-  writable: 16,
-  value: 64
+  object: 2,
+  writable: 4,
+  value: 8
 };
 
 
+function parseBool(v) {
+  v = "" + v;
+  if (v === "true" || v === "TRUE" || v === "True" || v === "1")
+    return true;
+  else if (v === "false" || v === "FALSE" || v === "False" || v === "0")
+    return false;
+  else
+    return null;
+}
+
+
 function sanitizeParameterValue(parameterValue) {
   if (parameterValue[0] != null) {
     switch (parameterValue[1]) {
       case 'xsd:boolean':
         if (typeof parameterValue[0] !== 'boolean') {
-          parameterValue[0] = !!JSON.parse(parameterValue[0]);
+          let v = parseBool(parameterValue[0]);
+          if (v == null)
+            v = '' + parameterValue[0];
+          parameterValue[0] = v;
         }
         break;
       case 'xsd:string':
+      case 'xsd:base64':
+      case 'xsd:hexBinary':
         if (typeof parameterValue[0] !== 'string') {
           parameterValue[0] = '' + parameterValue[0];
         }
@@ -44,17 +60,22 @@ function sanitizeParameterValue(parameterValue) {
       case 'xsd:int':
       case 'xsd:unsignedInt':
         if (typeof parameterValue[0] !== 'number') {
-          parameterValue[0] = +parameterValue[0];
+          let v = parseInt(parameterValue[0]);
+          if (isNaN(v))
+            v = '' + parameterValue[0];
+          parameterValue[0] = v;
         }
         break;
       case 'xsd:dateTime':
         if (typeof parameterValue[0] !== 'number') {
-          if (parameterValue[0].getTime() != null)
-            parameterValue[0] = parameterValue[0].getTime();
-          else if (isNaN(parameterValue[0]))
-            parameterValue[0] = Date.parse(parameterValue[0]);
-          else
-            parameterValue[0] = +parameterValue[0];
+          // Don't use parseInt because it reads date string as a number
+          let v = +parameterValue[0];
+          if (isNaN(v)) {
+            v = Date.parse(parameterValue[0]);
+            if (isNaN(v))
+              v = '' + parameterValue[0];
+          }
+          parameterValue[0] = v;
         }
         break;
       default:
@@ -62,6 +83,7 @@ function sanitizeParameterValue(parameterValue) {
           throw new Error(`Parameter value type "${parameterValue[1]}" not recognized.`);
 
         parameterValue[0] = JSON.parse(JSON.stringify(parameterValue[0]));
+        break;
     }
   }
   return parameterValue;
@@ -170,35 +192,58 @@ function unpack(deviceData, path, revision) {
 }
 
 
-function clear(deviceData, path, timestamp, attributes) {
-  for (let p of deviceData.paths.find(path, true, true, timestamp ? 99 : path.length)) {
-    for (let k in deviceData.trackers.get(p))
-      deviceData.changes.add(k);
+function clear(deviceData, path, timestamp, attributes, changeFlags = 0) {
+  let changeTrackers = {};
 
-    let t = deviceData.timestamps.get(p);
-    if (!t)
+  timestamp = timestamp || 0;
+
+  let descendantsTimestamp = timestamp;
+  if (attributes && attributes.object) {
+    if (attributes.object > descendantsTimestamp)
+      descendantsTimestamp = attributes.object;
+    if (!(attributes.object <= attributes.value))
+      attributes.value = attributes.object;
+  }
+
+  for (let p of deviceData.paths.find(path, true, true, descendantsTimestamp ? 99 : path.length)) {
+    let tracker = deviceData.trackers.get(p);
+    for (let k in tracker)
+      changeTrackers[k] |= tracker[k];
+
+    let currentTimestamp = deviceData.timestamps.get(p);
+    if (currentTimestamp === undefined)
       continue;
 
-    if (!(timestamp <= t)) {
+    if (timestamp > currentTimestamp ||
+      (descendantsTimestamp > currentTimestamp && p.length > path.length)) {
       deviceData.timestamps.delete(p);
       deviceData.attributes.delete(p);
-    } else if (t && attributes) {
+      changeFlags |= 1;
+    }
+    else if (attributes && p.length == path.length) {
       let currentAttributes = deviceData.attributes.get(p);
       if (currentAttributes) {
         let newAttrs;
         for (let attrName in attributes) {
           if (attrName in currentAttributes &&
-              attributes[attrName][0] > currentAttributes[attrName][0]) {
-            if (!newAttrs)
-              newAttrs = JSON.parse(JSON.stringify(currentAttributes));
+              attributes[attrName] > currentAttributes[attrName][0]) {
+            changeFlags |= CHANGE_FLAGS[attrName];
+            if (!newAttrs) {
+              newAttrs = Object.assign({}, currentAttributes);
+              deviceData.attributes.set(p, newAttrs);
+            }
             delete newAttrs[attrName];
           }
         }
+      }
+    }
+  }
 
-        if (newAttrs)
-          deviceData.attributes.set(p, newAttrs);
-      }
-    }
+  // Note: For performance, we're merging all changes together rather than
+  // mark changes based the exact parameters affected.
+  for (let k in changeTrackers) {
+    if (changeTrackers[k] & changeFlags)
+      deviceData.changes.add(k);
   }
 }
 
@@ -226,9 +271,11 @@ function set(deviceData, path, timestamp, attributes, toClear) {
 
   if (path.wildcard)
     attributes = undefined;
-  else
+  else if (currentTimestamp)
     currentAttributes = deviceData.attributes.get(path);
 
+  let changeFlags = 0;
+
   if (attributes) {
     if (attributes.value && attributes.value[1]
         && attributes.value[0] >= (attributes.object ? attributes.object[0] : 0))
@@ -238,85 +285,63 @@ function set(deviceData, path, timestamp, attributes, toClear) {
       && attributes.object[0] >= (attributes.value ? attributes.value[0] : 0))
       attributes.value = [attributes.object[0]];
 
-    if (currentAttributes) {
-      let changeFlags = 0;
+    let newAttributes = Object.assign({}, currentAttributes, attributes);
 
+    if (currentAttributes) {
       for (let attrName in attributes) {
         timestamp = Math.max(timestamp, attributes[attrName][0]);
-        if (attrName in currentAttributes) {
-          if (attributes[attrName][0] > currentAttributes[attrName][0]) {
+        if (!(attrName in currentAttributes))
           changeFlags |= CHANGE_FLAGS[attrName];
-            if (compareEquality(attributes[attrName][1], currentAttributes[attrName][1]))
-              changeFlags |= CHANGE_FLAGS[attrName] << 1;
+        else if (attributes[attrName][0] <= currentAttributes[attrName][0])
+          newAttributes[attrName] = currentAttributes[attrName];
+        else if (!compareEquality(attributes[attrName][1], currentAttributes[attrName][1]))
+          changeFlags |= CHANGE_FLAGS[attrName];
+      }
     }
     else
-            delete attributes[attrName];
-        }
-        else {
-          changeFlags |= CHANGE_FLAGS[attrName];
-          changeFlags |= CHANGE_FLAGS[attrName] << 1;
-        }
-      }
-
-      if (Object.keys(attributes).length) {
-        if (attributes.object && currentAttributes.object &&
-            attributes.object[1] !== currentAttributes.object[1]) {
-          toClear = toClear || [];
-          toClear.push([path.concat('*'), attributes.object[0]]);
-        }
-        for (let attrName in currentAttributes)
-          if (!(attrName in attributes))
-            attributes[attrName] = currentAttributes[attrName];
-
-        deviceData.attributes.set(path, attributes);
-      }
-
-      if (timestamp > currentTimestamp) {
       changeFlags |= 1;
-        deviceData.timestamps.set(path, timestamp);
 
+    deviceData.attributes.set(path, newAttributes);
+
+    if (!(timestamp <= currentTimestamp)) {
+      deviceData.timestamps.set(path, timestamp);
       if (path.length > 1)
         toClear = set(deviceData, path.slice(0, path.length - 1), timestamp,
           {object: [timestamp, 1]}, toClear);
     }
-
-      if (changeFlags)
-        for (let sup of deviceData.paths.find(path, true, false)) {
-          let c = deviceData.trackers.get(sup);
-          for (let k in c)
-            if (c[k] & changeFlags) {
-              deviceData.changes.add(k);
-            }
-        }
-    }
-    else if (!(timestamp <= currentTimestamp)) {
-      deviceData.timestamps.set(path, timestamp);
-      deviceData.attributes.set(path, attributes);
-
-      if (path.length > 1)
-        toClear = set(deviceData, path.slice(0, path.length - 1), timestamp,
-          {object: [timestamp, 1]}, toClear);
-
-      toClear = toClear || [];
-      toClear.push([path, timestamp]);
-    }
   }
   else if (!(timestamp <= currentTimestamp)) {
     deviceData.timestamps.set(path, timestamp);
 
     if (currentAttributes) {
       deviceData.attributes.delete(path);
+      changeFlags |= 1;
+    }
+    else if (path.wildcard) {
+      for (let p of deviceData.paths.find(path, false, true, path.length))
+        if (timestamp > deviceData.timestamps.get(p)) {
           toClear = toClear || [];
-      toClear.push([path, timestamp]);
+          toClear.push([p, timestamp]);
+        }
+    }
   }
 
-    if (path.wildcard)
-      for (let sub of deviceData.paths.find(path, false, true))
-        if (timestamp > deviceData.timestamps.get(sub)) {
-          deviceData.timestamps.delete(sub);
-          if (deviceData.attributes.delete(sub)) {
+  if (changeFlags) {
+    if (changeFlags & 1) {
       toClear = toClear || [];
-            toClear.push([sub, timestamp]);
+      toClear.push([path, timestamp, null, changeFlags]);
+    }
+    else if (changeFlags & CHANGE_FLAGS.object) {
+      toClear = toClear || [];
+      toClear.push([path, 0, {object: attributes.object[0]}, changeFlags]);
+    }
+    else {
+      for (let p of deviceData.paths.find(path, true, false, path.length)) {
+        let tracker = deviceData.trackers.get(p);
+        for (let k in tracker) {
+          if (tracker[k] & changeFlags)
+            deviceData.changes.add(k);
+        }
       }
     }
   }
@@ -325,24 +350,14 @@ function set(deviceData, path, timestamp, attributes, toClear) {
 }
 
 
-function track(deviceData, path, marker, timestamp, attributeTimestamps, exist, attributeValues) {
+function track(deviceData, path, marker, attributes) {
   path = deviceData.paths.add(path);
-  let f = 0;
+  let f = 1;
 
-  if (timestamp)
-    f |= 1;
-
-  if (exist)
-    f |= 2;
-
-  if (attributeTimestamps)
-    for (let attrName of attributeTimestamps)
+  if (attributes)
+    for (let attrName of attributes)
       f |= CHANGE_FLAGS[attrName];
 
-  if (attributeValues)
-    for (let attrName of attributeValues)
-      f |= CHANGE_FLAGS[attrName] << 1;
-
   let cur = deviceData.trackers.get(path);
   if (!cur) {
     cur = {};

lib/extension-wrapper.js

@@ -26,15 +26,27 @@ function errorToFault(err) {
   if (!err)
     return null;
 
-  return {
+  if (!err.name)
+    return {code: "ext", message: `${err}`};
+
+  let fault = {
     code: `ext.${err.name}`,
     message: err.message,
     detail: {
       name: err.name,
-      message: err.message,
-      stack: err.stack
+      message: err.message
     }
+  };
+
+  if (err.stack) {
+    fault.detail.stack = err.stack;
+    // Trim the stack trace
+    let stackTrimIndex = fault.detail.stack.match(/\s+at\s[^\s]+\s\(.*extension-wrapper\.js:.+\)/);
+    if (stackTrimIndex)
+      fault.detail.stack = fault.detail.stack.slice(0, stackTrimIndex.index);
   }
+
+  return fault;
 }
 
 process.on("uncaughtException", function(err) {
@@ -55,10 +67,24 @@ process.on("message", function(message) {
     script = require(`${cwd}/${fileName}`);
   }
 
-  script[message[1][0]](message[1].slice(1), function(err, res) {
+  const funcName = message[1][0];
+
+  if (!script[funcName]) {
+    let fault = {
+      code: "ext",
+      message: `No such function '${funcName}' in extension '${fileName}'`
+    };
+    process.send([message[0], fault]);
+    return;
+  }
+
+  script[funcName](message[1].slice(1), function(err, res) {
     if (!jobs.delete(message[0]))
       return;
 
     process.send([message[0], errorToFault(err), res]);
   });
 });
+
+// Ignore SIGINT
+process.on('SIGINT', function() {});

lib/extensions.js

@@ -42,19 +42,23 @@ function run(args, callback) {
 
   if (!processes[scriptName]) {
     let p = childProcess.fork('lib/extension-wrapper', [scriptName]);
+    processes[scriptName] = p;
 
     p.on('error', function() {
+      if (processes[scriptName] === p) {
         kill(processes[scriptName]);
         delete processes[scriptName];
+      }
     });
 
     p.on('disconnect', function() {
+      if (processes[scriptName] === p) {
         kill(processes[scriptName]);
         delete processes[scriptName];
+      }
     });
 
     p.on('message', messageHandler);
-    processes[scriptName] = p;
   }
 
   let id = crypto.randomBytes(8).toString('hex');
@@ -70,7 +74,7 @@ function run(args, callback) {
 };
 
 
-function kill(process) {
+function kill(process, callback) {
   if (process.signalCode !== null || process.exitCode !== null)
     return;
 
@@ -81,18 +85,31 @@ function kill(process) {
   let t = setInterval(function() {
     if (process.signalCode !== null || process.exitCode !== null) {
       clearInterval(t);
+      if (callback)
+        callback();
     }
     else if (Date.now() > timeToKill) {
       process.kill('SIGKILL');
       clearInterval(t);
+      if (callback)
+        callback();
     }
   }, 100);
 }
 
 
-function killAll() {
-  for (let n in processes)
-    kill(processes[n]);
+function killAll(callback) {
+  let counter = 1;
+  for (let n in processes) {
+    ++counter;
+    kill(processes[n], function() {
+      if (--counter == 0 && callback)
+        callback();
+    });
+    delete processes[n];
+  }
+  if (--counter == 0 && callback)
+    callback();
 }
 
 

lib/gpn-heuristic.js

@@ -0,0 +1,82 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const WILDCARD_MULTIPLIER = 2;
+const UNDISCOVERED_DEPTH = 7;
+
+// Simple heuristic to estimate GPN count given a set of patterns to be
+// discovered. Used to decide whether to use nextLevel = false in GPN.
+// gpnPatterns is [pattern, flags]
+// pattern is a path (array)
+// flags is an int where its bits mark the segments in the pattern that
+// need refreshing. Leading 0s indicate that the pattern up to that
+// point has been discovered.
+function estimateGpnCount(gpnPatterns, depth = 0) {
+  const children = {};
+  const wildcardChildren = [];
+  let wildcardDiscovered = false;
+  let gpnCount = 0;
+
+  for (let pattern of gpnPatterns) {
+    const path = pattern[0];
+    const flags = pattern[1] >> depth;
+
+    let k = path[depth];
+
+    if (!k) {
+      if (flags & 1)
+        gpnCount = 1;
+      continue;
+    }
+
+    if (flags & 1) {
+      gpnCount = 1;
+
+      if (depth > UNDISCOVERED_DEPTH)
+        continue;
+    }
+    else if (k === "*")
+      wildcardDiscovered = true;
+
+    if (k === "*") {
+      wildcardChildren.push(pattern);
+    }
+    else {
+      children[k] = children[k] || [];
+      children[k].push(pattern);
+    }
+  }
+
+  let wildcardGpnCount = 0;
+  if (!wildcardDiscovered && wildcardChildren.length)
+      wildcardGpnCount += estimateGpnCount(wildcardChildren, depth + 1) * WILDCARD_MULTIPLIER;
+
+  for (let k of Object.keys(children)) {
+    let c = estimateGpnCount(children[k].concat(wildcardChildren), depth + 1);
+    wildcardGpnCount -= c;
+    gpnCount += c;
+  }
+
+  gpnCount += Math.max(0, wildcardGpnCount);
+
+  return gpnCount;
+}
+
+exports.estimateGpnCount = estimateGpnCount;

lib/local-cache.js

@@ -0,0 +1,356 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const vm = require("vm");
+const crypto = require("crypto");
+
+const db = require("./db");
+const cache = require("./cache");
+const query = require("./query");
+const logger = require("./logger");
+const scheduling = require("./scheduling");
+
+const REFRESH = 3000;
+
+let nextRefresh = 1;
+let hash = null;
+let presets, provisions, virtualParameters, files;
+
+
+function computeHash() {
+  // MD5 hash for presets, provisions, virtual parameters for detecting changes
+  let h = crypto.createHash("md5");
+  for (let p of presets) {
+    h.update(JSON.stringify(p.name));
+    h.update(JSON.stringify(p.channel));
+    h.update(JSON.stringify(p.schedule));
+    h.update(JSON.stringify(p.events));
+    h.update(JSON.stringify(p.precondition));
+    h.update(JSON.stringify(p.provisions));
+  }
+
+  let keys;
+
+  keys = Object.keys(provisions).sort();
+  h.update(JSON.stringify(keys));
+  for (let k of keys)
+    h.update(provisions[k].md5);
+
+  keys = Object.keys(virtualParameters).sort();
+  h.update(JSON.stringify(keys));
+  for (let k of keys)
+    h.update(virtualParameters[k].md5);
+
+  hash = h.digest("hex");
+}
+
+function flattenObject(src, prefix = "", dst = {}) {
+  for (let k of Object.keys(src)) {
+    let v = src[k];
+    if (typeof v === "object" && !Array.isArray(v))
+      flattenObject(v, `${prefix}${k}.`, dst);
+    else
+      dst[`${prefix}${k}`] = v;
+  }
+  return dst;
+}
+
+function refresh(callback) {
+  if (!nextRefresh)
+    return setTimeout(function() {refresh(callback);}, 20);
+
+  nextRefresh = 0;
+  const now = Date.now();
+
+  cache.get("presets_hash", function(err, res) {
+    if (err)
+      return callback(err);
+
+    if (hash && res === hash) {
+      nextRefresh = now + (REFRESH - (now % REFRESH));
+      return callback();
+    }
+
+    cache.lock("presets_hash_lock", 3, function(err, unlockOrExtend) {
+      if (err)
+        return callback(err);
+
+      let counter = 3;
+
+      counter += 2;
+      db.getPresets(function(err, res) {
+        if (err) {
+          if (counter & 1)
+            callback(err);
+          return counter = 0;
+        }
+
+        db.getObjects(function(err, objects) {
+          if (err) {
+            if (counter & 1)
+              callback(err);
+            return counter = 0;
+          }
+
+          objects = objects.map(obj => {
+            // Flatten object
+            obj = flattenObject(obj);
+
+            // If no keys are defined, consider all parameters as keys to keep the
+            // same behavior from v1.0
+            if (!obj["_keys"] || !obj["_keys"].length)
+              obj["_keys"] = Object.keys(obj).filter(k => !k.startsWith("_"));
+
+            return obj
+          })
+
+          res.sort((a, b) => {
+            if (a.weight === b.weight)
+              return a._id > b._id;
+            else
+              return a.weight - b.weight;
+          });
+
+          presets = [];
+          for (let preset of res) {
+            let schedule = null;
+            if (preset.schedule) {
+              let parts = preset.schedule.trim().split(/\s+/);
+              schedule = {
+                md5: crypto.createHash("md5").update(preset.schedule).digest("hex")
+              };
+
+              try {
+                schedule.duration = +(parts.shift()) * 1000;
+                schedule.schedule = scheduling.parseCron(parts.join(" "));
+              }
+              catch (err) {
+                logger.warn({
+                  message: "Invalid preset schedule",
+                  preset: preset._id,
+                  schedule: preset.schedule
+                });
+                schedule.schedule = false;
+              }
+            }
+
+            let events = preset.events || {};
+            let precondition = query.convertMongoQueryToFilters(JSON.parse(preset.precondition));
+            let _provisions = preset.provisions || [];
+
+            // Generate provisions from the old configuration format
+            for (let c of preset.configurations) {
+              switch (c.type) {
+                case "age":
+                  _provisions.push(["refresh", c.name, c.age]);
+                  break;
+
+                case "value":
+                  _provisions.push(["value", c.name, c.value]);
+                  break;
+
+                case "add_tag":
+                  _provisions.push(["tag", c.tag, true]);
+                  break;
+
+                case "delete_tag":
+                  _provisions.push(["tag", c.tag, false]);
+                  break;
+
+                case "provision":
+                  _provisions.push([c.name].concat(c.args || []));
+                  break;
+
+                case "add_object":
+                  for (let obj of objects) {
+                    if (obj["_id"] === c.object) {
+                      let alias = obj["_keys"].map((k) => `${k}:${JSON.stringify(obj[k])}`).join(",");
+                      let p = `${c.name}.[${alias}]`;
+                      _provisions.push(["instances", p, 1]);
+
+                      for (let k in obj)
+                        if (!k.startsWith("_") && !(obj["_keys"].indexOf(k) !== -1))
+                          _provisions.push(["value", `${p}.${k}`, obj[k]]);
+                    }
+                  }
+                  break;
+
+                case "delete_object":
+                  for (let obj of objects) {
+                    if (obj["_id"] === c.object) {
+                      let alias = obj["_keys"].map((k) => `${k}:${JSON.stringify(obj[k])}`).join(",");
+                      let p = `${c.name}.[${alias}]`;
+                      _provisions.push(["instances", p, 0]);
+                    }
+                  }
+                  break;
+
+                default:
+                  if (counter & 1)
+                    callback(new Error(`Unknown configuration type ${c.type}`));
+                  return counter = 0;
+                  break;
+              }
+            }
+
+            presets.push({
+              name: preset._id,
+              channel: preset.channel || "default",
+              schedule: schedule,
+              events: events,
+              precondition: precondition,
+              provisions: _provisions
+            });
+          }
+
+          if ((counter -= 2) === 1) {
+            computeHash();
+            cache.set("presets_hash", hash, 300, function(err) {
+              unlockOrExtend(0);
+              nextRefresh = now + (REFRESH - (now % REFRESH));
+              callback(err);
+            });
+          }
+        });
+      });
+
+      counter += 2;
+      db.getProvisions(function(err, res) {
+        if (err) {
+          if (counter & 1)
+            callback(err);
+          return counter = 0;
+        }
+
+        provisions = {};
+        for (let r of res) {
+          provisions[r._id] = {};
+          provisions[r._id].md5 = crypto.createHash("md5").update(r.script).digest("hex");
+          provisions[r._id].script = new vm.Script(`"use strict";(function(){\n${r.script}\n})();`, {filename: r._id, lineOffset: -1, timeout: 50});
+        }
+
+        if ((counter -= 2) === 1) {
+          computeHash();
+          cache.set("presets_hash", hash, 300, function(err) {
+            unlockOrExtend(0);
+            nextRefresh = now + (REFRESH - (now % REFRESH));
+            callback(err);
+          });
+        }
+      });
+
+      counter += 2;
+      db.getVirtualParameters(function(err, res) {
+        if (err) {
+          if (counter & 1)
+            callback(err);
+          return counter = 0;
+        }
+
+        virtualParameters = {};
+        for (let r of res) {
+          virtualParameters[r._id] = {}
+          virtualParameters[r._id].md5 = crypto.createHash("md5").update(r.script).digest("hex");
+          virtualParameters[r._id].script = new vm.Script(`"use strict";(function(){\n${r.script}\n})();`, {filename: r._id, lineOffset: -1, timeout: 50});
+        }
+
+        if ((counter -= 2) === 1) {
+          computeHash();
+          cache.set("presets_hash", hash, 300, function(err) {
+            unlockOrExtend(0);
+            nextRefresh = now + (REFRESH - (now % REFRESH));
+            callback(err);
+          });
+        }
+      });
+
+      counter += 2;
+      db.getFiles(function(err, res) {
+        if (err) {
+          if (counter & 1)
+            callback(err);
+          return counter = 0;
+        }
+
+        files = {};
+        for (let r of res) {
+          const id = r.filename || r._id.toString();
+          files[id] = {};
+          files[id].length = r.length;
+          files[id].md5 = r.md5;
+          files[id].contentType = r.contentType;
+        }
+
+        if ((counter -= 2) === 1) {
+          computeHash();
+          cache.set("presets_hash", hash, 300, function(err) {
+            unlockOrExtend(0);
+            nextRefresh = now + (REFRESH - (now % REFRESH));
+            callback(err);
+          });
+        }
+      });
+
+      if ((counter -= 2) === 1) {
+        computeHash();
+        cache.set("presets_hash", hash, 300, function(err) {
+          unlockOrExtend(0);
+          nextRefresh = now + (REFRESH - (now % REFRESH));
+          callback(err);
+        });
+      }
+    });
+  });
+}
+
+
+function getPresets(callback) {
+  if (Date.now() < nextRefresh)
+    return callback(null, hash, presets);
+
+  refresh(function(err) {
+    callback(err, hash, presets);
+  });
+}
+
+
+function getFiles(callback) {
+  if (Date.now() < nextRefresh)
+    return callback(null, hash, files);
+
+  refresh(function(err) {
+    callback(err, hash, files);
+  })
+}
+
+
+function getProvisionsAndVirtualParameters(callback) {
+  if (Date.now() < nextRefresh)
+    return callback(null, hash, provisions, virtualParameters);
+
+  refresh(function(err) {
+    callback(err, hash, provisions, virtualParameters);
+  });
+}
+
+
+exports.getPresets = getPresets;
+exports.getFiles = getFiles;
+exports.getProvisionsAndVirtualParameters = getProvisionsAndVirtualParameters;

lib/logger.js

@@ -165,15 +165,15 @@ function flatten(details) {
         details.informRetryCount = details.rpc.cpeRequest.retryCount;
       }
       else {
-        details.cpeRequestName = details.rpc.cpeRequest.type;
+        details.cpeRequestName = details.rpc.cpeRequest.name;
         if (details.rpc.cpeRequest.commandKey)
           details.cpeRequestCommandKey = details.rpc.cpeRequest.commandKey;
       }
     }
     else if (details.rpc.cpeFault) {
       details.acsRequestId = details.rpc.id;
-      details.cpeFaultCode = details.rpc.cpeFault.faultCode;
-      details.cpeFaultString = details.rpc.cpeFault.faultString;
+      details.cpeFaultCode = details.rpc.cpeFault.detail.faultCode;
+      details.cpeFaultString = details.rpc.cpeFault.detail.faultString;
     }
     delete details.rpc;
   }
@@ -311,6 +311,7 @@ function accessError(details) {
 
 exports.init = init;
 exports.close = close;
+exports.flatten = flatten;
 exports.info = info;
 exports.warn = warn;
 exports.error = error;

lib/nbi.coffee

@@ -50,6 +50,7 @@ common = require './common'
 db = require './db'
 query = require './query'
 apiFunctions = require './api-functions'
+cache = require './cache'
 
 VERSION = require('../package.json').version
 
@@ -60,8 +61,8 @@ TAGS_REGEX = /^\/devices\/([a-zA-Z0-9\-\_\%]+)\/tags\/([a-zA-Z0-9\-\_\%]+)\/?$/
 PRESETS_REGEX = /^\/presets\/([a-zA-Z0-9\-\_\%]+)\/?$/
 OBJECTS_REGEX = /^\/objects\/([a-zA-Z0-9\-\_\%]+)\/?$/
 FILES_REGEX = /^\/files\/([a-zA-Z0-9\%\!\*\'\(\)\;\:\@\&\=\+\$\,\?\#\[\]\-\_\.\~]+)\/?$/
-PING_REGEX = /^\/ping\/([a-zA-Z0-9\-\_\.]+)\/?$/
-QUERY_REGEX = /^\/([a-zA-Z0-9_]+s)\/?$/
+PING_REGEX = /^\/ping\/([a-zA-Z0-9\-\_\.\:]+)\/?$/
+QUERY_REGEX = /^\/([a-zA-Z0-9_]+)\/?$/
 DELETE_DEVICE_REGEX = /^\/devices\/([a-zA-Z0-9\-\_\%]+)\/?$/
 PROVISIONS_REGEX = /^\/provisions\/([a-zA-Z0-9\-\_\%]+)\/?$/
 VIRTUAL_PARAMETERS_REGEX = /^\/virtual_parameters\/([a-zA-Z0-9\-\_\%]+)\/?$/
@@ -108,7 +109,7 @@ listener = (request, response) ->
 
         db.presetsCollection.save(preset, (err) ->
           return throwError(err, response) if err
-          db.redisClient.del('presets_hash', (err) ->
+          cache.del('presets_hash', (err) ->
             return throwError(err, response) if err
             response.writeHead(200)
             response.end()
@@ -117,7 +118,7 @@ listener = (request, response) ->
       else if request.method == 'DELETE'
         db.presetsCollection.remove({'_id' : presetName}, (err) ->
           return throwError(err, response) if err
-          db.redisClient.del('presets_hash', (err) ->
+          cache.del('presets_hash', (err) ->
             return throwError(err, response) if err
             response.writeHead(200)
             response.end()
@@ -134,7 +135,7 @@ listener = (request, response) ->
 
         db.objectsCollection.save(object, (err) ->
           return throwError(err, response) if err
-          db.redisClient.del('objects', 'presets_hash', (err) ->
+          cache.del('presets_hash', (err) ->
             return throwError(err, response) if err
             response.writeHead(200)
             response.end()
@@ -143,7 +144,7 @@ listener = (request, response) ->
       else if request.method == 'DELETE'
         db.objectsCollection.remove({'_id' : objectName}, (err) ->
           return throwError(err, response) if err
-          db.redisClient.del('objects', 'presets_hash', (err) ->
+          cache.del('presets_hash', (err) ->
             return throwError(err, response) if err
             response.writeHead(200)
             response.end()
@@ -168,7 +169,7 @@ listener = (request, response) ->
 
         db.provisionsCollection.save(object, (err) ->
           return throwError(err, response) if err
-          db.redisClient.del('presets_hash', (err) ->
+          cache.del('presets_hash', (err) ->
             return throwError(err, response) if err
             response.writeHead(200)
             response.end()
@@ -177,7 +178,7 @@ listener = (request, response) ->
       else if request.method == 'DELETE'
         db.provisionsCollection.remove({'_id' : provisionName}, (err) ->
           return throwError(err, response) if err
-          db.redisClient.del('presets_hash', (err) ->
+          cache.del('presets_hash', (err) ->
             return throwError(err, response) if err
             response.writeHead(200)
             response.end()
@@ -202,7 +203,7 @@ listener = (request, response) ->
 
         db.virtualParametersCollection.save(object, (err) ->
           return throwError(err, response) if err
-          db.redisClient.del('presets_hash', (err) ->
+          cache.del('presets_hash', (err) ->
             return throwError(err, response) if err
             response.writeHead(200)
             response.end()
@@ -211,7 +212,7 @@ listener = (request, response) ->
       else if request.method == 'DELETE'
         db.virtualParametersCollection.remove({'_id' : virtualParameterName}, (err) ->
           return throwError(err, response) if err
-          db.redisClient.del('presets_hash', (err) ->
+          cache.del('presets_hash', (err) ->
             return throwError(err, response) if err
             response.writeHead(200)
             response.end()
@@ -226,22 +227,16 @@ listener = (request, response) ->
       tag = querystring.unescape(r[2])
       if request.method == 'POST'
         db.devicesCollection.update({'_id' : deviceId}, {'$addToSet' : {'_tags' : tag}}, {safe: true}, (err) ->
-          return throwError(err, response) if err
-          db.redisClient.del("#{deviceId}_presets_hash", (err) ->
           return throwError(err, response) if err
           response.writeHead(200)
           response.end()
         )
-        )
       else if request.method == 'DELETE'
         db.devicesCollection.update({'_id' : deviceId}, {'$pull' : {'_tags' : tag}}, {safe: true}, (err) ->
-          return throwError(err, response) if err
-          db.redisClient.del("#{deviceId}_presets_hash", (err) ->
           return throwError(err, response) if err
           response.writeHead(200)
           response.end()
         )
-        )
       else
         response.writeHead 405, {'Allow': 'POST, DELETE'}
         response.end('405 Method Not Allowed')
@@ -255,14 +250,14 @@ listener = (request, response) ->
           if channel.startsWith('task_')
             return db.tasksCollection.remove({_id : new mongodb.ObjectID(channel.slice(5))}, (err) ->
               return throwError(err, response) if err
-              db.redisClient.del("#{deviceId}_tasks", "#{deviceId}_faults", (err) ->
+              cache.del("#{deviceId}_tasks_faults_operations", (err) ->
                 return throwError(err, response) if err
                 response.writeHead(200)
                 response.end()
               )
             )
 
-          db.redisClient.del("#{deviceId}_faults", (err) ->
+          cache.del("#{deviceId}_tasks_faults_operations", (err) ->
             return throwError(err, response) if err
             response.writeHead(200)
             response.end()
@@ -279,7 +274,7 @@ listener = (request, response) ->
           task.device = deviceId
           apiFunctions.insertTasks(task, (err) ->
             return throwError(err, response) if err
-            db.redisClient.del("#{deviceId}_tasks", (err) ->
+            cache.del("#{deviceId}_tasks_faults_operations", (err) ->
               return throwError(err, response) if err
 
               if urlParts.query.connection_request?
@@ -288,7 +283,8 @@ listener = (request, response) ->
                     response.writeHead(202, err.message, {'Content-Type' : 'application/json'})
                     response.end(JSON.stringify(task))
                   else
-                    apiFunctions.watchTask(deviceId, task._id, config.get('DEVICE_ONLINE_THRESHOLD', deviceId), (err, status) ->
+                    taskTimeout = urlParts.query.timeout? and parseInt(urlParts.query.timeout) or config.get('DEVICE_ONLINE_THRESHOLD', deviceId)
+                    apiFunctions.watchTask(deviceId, task._id, taskTimeout, (err, status) ->
                       return throwError(err, response) if err
 
                       if status is 'timeout'
@@ -335,12 +331,17 @@ listener = (request, response) ->
         if request.method == 'DELETE'
           db.tasksCollection.findOne({'_id' : taskId}, {'device' : 1}, (err, task) ->
             return throwError(err, response) if err
+            if not task?
+              response.writeHead(404)
+              response.end("Task not found")
+              return
+
             deviceId = task.device
             db.tasksCollection.remove({'_id' : taskId}, (err) ->
               return throwError(err, response) if err
               db.faultsCollection.remove({_id : "#{deviceId}:task_#{String(taskId)}"}, (err) ->
                 return throwError(err, response) if err
-                db.redisClient.del("#{deviceId}_tasks", "#{deviceId}_faults", (err) ->
+                cache.del("#{deviceId}_tasks_faults_operations", (err) ->
                   return throwError(err, response) if err
                   response.writeHead(200)
                   response.end()
@@ -358,7 +359,7 @@ listener = (request, response) ->
             deviceId = task.device
             db.faultsCollection.remove({_id : "#{deviceId}:task_#{String(taskId)}"}, (err) ->
               return throwError(err, response) if err
-              db.redisClient.del("#{deviceId}_faults", (err) ->
+              cache.del("#{deviceId}_tasks_faults_operations", (err) ->
                 return throwError(err, response) if err
                 response.writeHead(200)
                 response.end()
@@ -496,7 +497,7 @@ listener = (request, response) ->
         i = 0
         cur.each((err, item) ->
           if err
-            throwError(err, response)
+            throwError(err)
             return false
 
           if item?

lib/query.coffee

@@ -70,9 +70,8 @@ normalize = (input) ->
     if (m = /^\/(.*?)\/(g?i?m?y?)$/.exec(input))
       vals.push({'$regex' : new RegExp(m[1], m[2])})
 
-    f = parseFloat(input)
-    if not isNaN(f)
-      vals.push(f)
+    if +input == parseFloat(input)
+      vals.push(+input)
 
     d = new Date(input)
     if input.length >= 8 and d.getFullYear() > 1983
@@ -150,7 +149,9 @@ expand = (query) ->
       conditions = permute(k, v)
       if conditions.length > 1
         new_query['$and'] ?= []
-        if v?['$ne']?
+        if v?['$ne']? or v?['$not']?
+          if Object.keys(v) > 1
+            throw new Error("Cannot mix $ne or $not with other operators")
           for c in conditions
             new_query['$and'].push(c)
         else

lib/sandbox.js

@@ -26,6 +26,7 @@ const common = require("./common");
 const device = require("./device");
 const extensions = require("./extensions");
 const logger = require("./logger");
+const scheduling = require("./scheduling");
 
 
 // Used for throwing to exit user script and commit
@@ -38,27 +39,78 @@ const context = vm.createContext();
 
 let state;
 
+const runningExtensions = new WeakMap();
+function runExtension(sessionContext, key, extCall, callback) {
+  let re = runningExtensions.get(sessionContext);
+  if (!re) {
+    re = {};
+    runningExtensions.set(sessionContext, re);
+  }
+
+  if (re[key]) {
+    re[key].push(callback);
+  }
+  else {
+    re[key] = [callback];
+    extensions.run(extCall, function(err, fault, res) {
+      const callbacks = re[key];
+      delete re[key];
+      if (!err && !fault)
+        sessionContext.extensionsCache[key] = res;
+      for (let c of callbacks)
+        c(err, fault);
+    });
+  }
+}
 
 class SandboxDate {
-  constructor(arg) {
+  constructor() {
     if (arguments.length)
-      return new (Function.prototype.bind.apply(Date, arguments));
+      return new Date(...arguments);
 
     return new Date(state.sessionContext.timestamp);
   }
 
-  static now() {
-    return state.sessionContext.timestamp;
+  static now(intervalOrCron, variance) {
+    let t = state.sessionContext.timestamp;
+
+    if (typeof intervalOrCron === "number") {
+
+      if (variance == null)
+        variance = intervalOrCron;
+
+      let offset = 0;
+      if (variance)
+        offset = scheduling.variance(state.sessionContext.deviceId, variance);
+
+      t = scheduling.interval(t, intervalOrCron, offset);
+    }
+    else if (typeof intervalOrCron === "string") {
+      let offset = 0;
+      if (variance)
+        offset = scheduling.variance(state.sessionContext.deviceId, variance);
+      let cron = scheduling.parseCron(intervalOrCron);
+      t = scheduling.cron(t, cron, offset)[0];
+    }
+    else if (intervalOrCron) {
+      throw new Error("Invalid Date.now() argument");
+    }
+
+    return t;
   }
 };
 
 function random() {
   if (!state.rng)
-    state.rng = seedrandom(sessionContext.timestamp);
+    state.rng = seedrandom(state.sessionContext.deviceId);
 
   return state.rng();
 }
 
+random.seed = function(s) {
+  state.rng = seedrandom(s);
+}
+
 
 class ParameterWrapper {
   constructor(path, attributes, unpacked, unpackedRevision) {
@@ -69,13 +121,13 @@ class ParameterWrapper {
 
         if (state.revision !== unpackedRevision) {
           unpackedRevision = state.revision;
-          unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision + 1);
+          unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision);
         }
 
         if (!unpacked.length)
           return undefined;
 
-        let attr = state.sessionContext.deviceData.attributes.get(unpacked[0], state.revision + 1)[attrName];
+        let attr = state.sessionContext.deviceData.attributes.get(unpacked[0], state.revision)[attrName];
 
         if (!attr)
           return undefined;
@@ -90,7 +142,7 @@ class ParameterWrapper {
 
       if (state.revision !== unpackedRevision) {
         unpackedRevision = state.revision;
-        unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision + 1);
+        unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision);
       }
 
       if (!unpacked.length)
@@ -105,7 +157,7 @@ class ParameterWrapper {
 
       if (state.revision !== unpackedRevision) {
         unpackedRevision = state.revision;
-        unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision + 1);
+        unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision);
       }
 
       if (!unpacked.length)
@@ -120,7 +172,7 @@ class ParameterWrapper {
 
       if (state.revision !== unpackedRevision) {
         unpackedRevision = state.revision;
-        unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision + 1);
+        unpacked = device.unpack(state.sessionContext.deviceData, path, state.revision);
       }
 
       for (let p of unpacked)
@@ -243,18 +295,27 @@ function errorToFault(err) {
   if (!err)
     return null;
 
-  // Trim the stack trace at the self-executing anonymous wrapper function
-  let stackTrimIndex = err.stack.match(/\s+at\s[^\s]+\s+at\s[^\s]+\s\(vm\.js.+\)/).index;
+  if (!err.name)
+    return {code: "script", message: `${err}`};
 
-  return {
+  let fault = {
     code: `script.${err.name}`,
     message: err.message,
     detail: {
       name: err.name,
-      message: err.message,
-      stack: err.stack.slice(0, stackTrimIndex)
+      message: err.message
     }
+  };
+
+  if (err.stack) {
+    fault.detail.stack = err.stack;
+    // Trim the stack trace at the self-executing anonymous wrapper function
+    let stackTrimIndex = fault.detail.stack.match(/\s+at\s[^\s]+\s+at\s[^\s]+\s\(vm\.js.+\)/);
+    if (stackTrimIndex)
+      fault.detail.stack = fault.detail.stack.slice(0, stackTrimIndex.index);
   }
+
+  return fault;
 }
 
 
@@ -302,13 +363,12 @@ function run(script, globals, sessionContext, startRevision, maxRevision, callba
   let counter = 3;
   for (let key of Object.keys(_state.extensions)) {
     counter += 2;
-    extensions.run(_state.extensions[key], function(err, fault, res) {
+    runExtension(_state.sessionContext, key, _state.extensions[key], function(err, fault) {
       if (err || fault) {
         if (counter & 1)
           callback(err, fault);
         return counter = 0;
       }
-      _state.sessionContext.extensionsCache[key] = res;
       if ((counter -= 2) === 1)
         if (status === 2)
           return run.apply(null, args);

lib/scheduling.js

@@ -0,0 +1,78 @@
+/**
+ * Copyright 2013-2017  Zaid Abdulla
+ *
+ * This file is part of GenieACS.
+ *
+ * GenieACS is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU Affero General Public License as
+ * published by the Free Software Foundation, either version 3 of the
+ * License, or (at your option) any later version.
+ *
+ * GenieACS is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU Affero General Public License for more details.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with GenieACS.  If not, see <http://www.gnu.org/licenses/>.
+ */
+"use strict";
+
+const crypto = require("crypto");
+
+const later = require("later");
+
+
+function md532(str) {
+  const digest = crypto
+    .createHash("md5")
+    .update(str)
+    .digest();
+  return (
+    digest.readUInt32LE(0) ^
+    digest.readUInt32LE(4) ^
+    digest.readUInt32LE(8) ^
+    digest.readUInt32LE(12)
+  );
+}
+
+
+function variance(deviceId, variance) {
+  return (md532(deviceId) >>> 0) % variance;
+}
+
+
+function interval(timestamp, interval, offset = 0) {
+  return Math.trunc((timestamp + offset) / interval) * interval - offset;
+}
+
+
+function parseCron(cronExp) {
+  let parts = cronExp.trim().split(/\s+/);
+  if (parts.length === 5)
+    parts.unshift("*");
+
+  return later.schedule(later.parse.cron(parts.join(" "), true));
+}
+
+
+function cron(timestamp, schedule, offset = 0) {
+  // TODO later.js doesn't throw erorr if expression is invalid!
+  const ret = [0, 0];
+
+  let prev = schedule.prev(1, new Date(timestamp + offset));
+  if (prev)
+    ret[0] = prev.setMilliseconds(0) - offset;
+
+  let next = schedule.next(1, new Date(timestamp + offset + 1000));
+  if (next)
+    ret[1] = next.setMilliseconds(0) - offset;
+
+  return ret;
+}
+
+
+exports.variance = variance;
+exports.interval = interval;
+exports.parseCron = parseCron;
+exports.cron = cron;

lib/server.coffee

@@ -27,6 +27,7 @@ logger = require './logger'
 logger.init(service, require('../package.json').version)
 config = require './config'
 db = require './db'
+cache = require './cache'
 extensions = require './extensions'
 
 networkInterface = config.get("#{service.toUpperCase()}_INTERFACE")
@@ -38,30 +39,50 @@ onConnection = require("./#{service}").onConnection
 
 server = null
 
-exit = () ->
+closeServer = (timeout, callback) ->
+  return callback() if not server
+
   setTimeout(() ->
-    process.exit(1)
-  , 30000).unref()
+    return if not callback
 
-  cluster.worker?.disconnect()
+    # Ignore HTTP requests from connection that may still be open
+    server.removeListener('request', listener)
+    server.setTimeout(1)
 
-  if not server
-    db.disconnect()
-    extensions.killAll()
-    logger.close()
-    return
+    cb = callback
+    callback = null
+    setTimeout(cb, 1000)
+  , timeout).unref()
 
   server.close(() ->
+    return if not callback
+
+    cb = callback
+    callback = null
+    cb()
+  )
+
+
+exit = () ->
+  setTimeout(() ->
+    extensions.killAll(() ->
+      process.exit(1)
+    )
+  , 30000).unref()
+
+  closeServer(20000, () ->
     db.disconnect()
+    cache.disconnect()
     extensions.killAll()
-    logger.close()
+    cluster.worker?.disconnect()
   )
 
 
 process.on('uncaughtException', (err) ->
   logger.error({
     message: 'Uncaught exception'
-    exception: err
+    exception: err,
+    pid: process.pid
   })
   exit()
 )
@@ -90,8 +111,11 @@ else
   server.on('connection', onConnection) if onConnection?
 
 db.connect((err) ->
+  throw err if err
+  cache.connect((err) ->
     throw err if err
     server.listen(port, networkInterface)
+  )
 )
 
 

lib/soap.js

@@ -64,7 +64,6 @@ const NAMESPACES = {
 
 const LIBXMLJS_PARSE_OPTIONS = {
   "nocdata": true,
-  "noent": true,
   "recover": config.get("XML_RECOVER"),
   "ignore_enc": config.get("XML_IGNORE_ENC")
 };
@@ -210,7 +209,17 @@ function SetParameterValues(xml, methodRequest) {
   for (const p of methodRequest.parameterList) {
     const pvs = paramList.node("ParameterValueStruct");
     pvs.node("Name").text(p[0]);
-    pvs.node("Value").attr({"xsi:type": p[2]}).text(p[1]);
+    let val = p[1];
+    if (p[2] === 'xsd:dateTime' && typeof val === 'number') {
+      val = (new Date(val)).toISOString();
+      if (methodRequest.DATETIME_MILLISECONDS === false)
+        val = val.replace('.000', '');
+    }
+    if (p[2] === 'xsd:boolean' && typeof val === 'boolean') {
+      if (methodRequest.BOOLEAN_LITERAL === false)
+        val = +val;
+    }
+    pvs.node("Value").attr({"xsi:type": p[2]}).text(val);
   }
 
   el.node("ParameterKey").text(methodRequest.parameterKey || "");
@@ -407,15 +416,13 @@ function fault(xml, cwmpVersion) {
 }
 
 
-function request(httpRequest, cwmpVersion, warn) {
+function request(data, cwmpVersion, warn) {
   warnings = warn;
 
   const rpc = {
     cwmpVersion: cwmpVersion
   };
 
-  const data = httpRequest.getBody();
-
   if (!data.length)
     return rpc;
 
@@ -460,7 +467,9 @@ function request(httpRequest, cwmpVersion, warn) {
     }
   }
 
-  rpc.id = xml.get("/soap-env:Envelope/soap-env:Header/cwmp:ID", NAMESPACES[rpc.cwmpVersion]).text();
+  let idElement = xml.get("/soap-env:Envelope/soap-env:Header/cwmp:ID", NAMESPACES[rpc.cwmpVersion]);
+  if (idElement)
+    rpc.id = idElement.text();
 
   if (methodElement && !(XML_IGNORE_NAMESPACE && methodElement.name() === "Fault")) {
     switch (methodElement.name()) {

lib/versioned-map.js

@@ -36,6 +36,7 @@ class VersionedMap {
     this._sizeDiff = [0];
     this._revision = 0;
     this.map = new Map();
+    this.dirty = 0;
   }
 
 
@@ -100,6 +101,7 @@ class VersionedMap {
 
     var v = this.map.get(key);
     if (!v) {
+      this.dirty |= 1 << rev;
       for (let i = 0; i < rev; ++ i)
         this._sizeDiff[i] -= 1;
 
@@ -118,12 +120,14 @@ class VersionedMap {
         throw new Error('Cannot modify old revisions');
     }
 
+    this.dirty |= 1 << rev;
     if (old === NONEXISTENT)
       ++ this._sizeDiff[rev];
 
     v[rev] = value;
   }
 
+
   delete(key, rev) {
     if (rev == null)
       rev = this._revision;
@@ -139,10 +143,11 @@ class VersionedMap {
     if (old === NONEXISTENT)
       return false;
 
+    this.dirty |= 1 << rev;
     -- this._sizeDiff[rev];
     v[rev] = NONEXISTENT;
 
-    return old;
+    return true;
   }
 
 
@@ -176,6 +181,7 @@ class VersionedMap {
         for (let i = this._sizeDiff.length; i <= r; ++ i)
           this._sizeDiff[i] = this._sizeDiff[i - 1];
 
+        this.dirty |= 1 << r;
         rev[r] = revisions[k];
       }
     }
@@ -183,6 +189,8 @@ class VersionedMap {
     for (let i = 0; i < minKey; ++ i)
       -- this._sizeDiff[i];
 
+    this.dirty |= del;
+
     for (let i = 0; del > 0; del >>= 1, ++ i)
       if (del & 1)
         rev[i] = NONEXISTENT;
@@ -232,6 +240,10 @@ class VersionedMap {
     this._sizeDiff[revision] = this._sizeDiff[this._sizeDiff.length - 1];
     this._sizeDiff.splice(revision + 1, this._sizeDiff.length);
 
+    let d = this.dirty >> revision;
+    this.dirty = this.dirty ^ (d << revision);
+    this.dirty |= (+!!d) << revision;
+
     for (let pair of this.map) {
       let k = pair[0];
       let v = pair[1];

npm-shrinkwrap.json

@@ -0,0 +1,625 @@
+{
+  "name": "genieacs",
+  "version": "1.1.3",
+  "lockfileVersion": 1,
+  "requires": true,
+  "dependencies": {
+    "abbrev": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/abbrev/-/abbrev-1.1.1.tgz",
+      "integrity": "sha512-nne9/IiQ/hzIhY6pdDnbBtz7DjPTKrY00P/zvPSm5pOFkl6xuGrGnXn/VtTNNfNtAfZ9/1RtehkszU9qcTii0Q=="
+    },
+    "ansi-regex": {
+      "version": "2.1.1",
+      "resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-2.1.1.tgz",
+      "integrity": "sha1-w7M6te42DYbg5ijwRorn7yfWVN8="
+    },
+    "aproba": {
+      "version": "1.2.0",
+      "resolved": "https://registry.npmjs.org/aproba/-/aproba-1.2.0.tgz",
+      "integrity": "sha512-Y9J6ZjXtoYh8RnXVCMOU/ttDmk1aBjunq9vO0ta5x85WDQiQfUF9sIPBITdbiiIVcBo03Hi3jMxigBtsddlXRw=="
+    },
+    "are-we-there-yet": {
+      "version": "1.1.4",
+      "resolved": "https://registry.npmjs.org/are-we-there-yet/-/are-we-there-yet-1.1.4.tgz",
+      "integrity": "sha1-u13KOCu5TwXhUZQ3PRb9O6HKEQ0=",
+      "requires": {
+        "delegates": "1.0.0",
+        "readable-stream": "2.3.4"
+      }
+    },
+    "balanced-match": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.0.tgz",
+      "integrity": "sha1-ibTRmasr7kneFk6gK4nORi1xt2c="
+    },
+    "brace-expansion": {
+      "version": "1.1.11",
+      "resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-1.1.11.tgz",
+      "integrity": "sha512-iCuPHDFgrHX7H2vEI/5xpz07zSHB00TpugqhmYtVmMO6518mCuRMoOYFldEBl0g187ufozdaHgWKcYFb61qGiA==",
+      "requires": {
+        "balanced-match": "1.0.0",
+        "concat-map": "0.0.1"
+      }
+    },
+    "bson": {
+      "version": "1.0.4",
+      "resolved": "https://registry.npmjs.org/bson/-/bson-1.0.4.tgz",
+      "integrity": "sha1-k8ENOeqltYQVy8QFLz5T5WKwtyw="
+    },
+    "buffer-shims": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/buffer-shims/-/buffer-shims-1.0.0.tgz",
+      "integrity": "sha1-mXjOMXOIxkmth5MCjDR37wRKi1E="
+    },
+    "chownr": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.1.tgz",
+      "integrity": "sha512-j38EvO5+LHX84jlo6h4UzmOwi0UgW61WRyPtJz4qaadK5eY3BTS5TY/S1Stc3Uk2lIM6TPevAlULiEJwie860g=="
+    },
+    "code-point-at": {
+      "version": "1.1.0",
+      "resolved": "https://registry.npmjs.org/code-point-at/-/code-point-at-1.1.0.tgz",
+      "integrity": "sha1-DQcLTQQ6W+ozovGkDi7bPZpMz3c="
+    },
+    "coffeescript": {
+      "version": "1.12.7",
+      "resolved": "https://registry.npmjs.org/coffeescript/-/coffeescript-1.12.7.tgz",
+      "integrity": "sha512-pLXHFxQMPklVoEekowk8b3erNynC+DVJzChxS/LCBBgR6/8AJkHivkm//zbowcfc7BTCAjryuhx6gPqPRfsFoA==",
+      "dev": true
+    },
+    "concat-map": {
+      "version": "0.0.1",
+      "resolved": "https://registry.npmjs.org/concat-map/-/concat-map-0.0.1.tgz",
+      "integrity": "sha1-2Klr13/Wjfd5OnMDajug1UBdR3s="
+    },
+    "console-control-strings": {
+      "version": "1.1.0",
+      "resolved": "https://registry.npmjs.org/console-control-strings/-/console-control-strings-1.1.0.tgz",
+      "integrity": "sha1-PXz0Rk22RG6mRL9LOVB/mFEAjo4="
+    },
+    "core-util-is": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/core-util-is/-/core-util-is-1.0.2.tgz",
+      "integrity": "sha1-tf1UIgqivFq1eqtxQMlAdUUDwac="
+    },
+    "debug": {
+      "version": "2.6.9",
+      "resolved": "https://registry.npmjs.org/debug/-/debug-2.6.9.tgz",
+      "integrity": "sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==",
+      "requires": {
+        "ms": "2.0.0"
+      }
+    },
+    "deep-extend": {
+      "version": "0.4.2",
+      "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.4.2.tgz",
+      "integrity": "sha1-SLaZwn4zS/ifEIkr5DL25MfTSn8="
+    },
+    "delegates": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/delegates/-/delegates-1.0.0.tgz",
+      "integrity": "sha1-hMbhWbgZBP3KWaDvRM2HDTElD5o="
+    },
+    "detect-libc": {
+      "version": "1.0.3",
+      "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-1.0.3.tgz",
+      "integrity": "sha1-+hN8S9aY7fVc1c0CrFWfkaTEups="
+    },
+    "double-ended-queue": {
+      "version": "2.1.0-0",
+      "resolved": "https://registry.npmjs.org/double-ended-queue/-/double-ended-queue-2.1.0-0.tgz",
+      "integrity": "sha1-ED01J/0xUo9AGIEwyEHv3XgmTlw=",
+      "optional": true
+    },
+    "es6-promise": {
+      "version": "3.2.1",
+      "resolved": "https://registry.npmjs.org/es6-promise/-/es6-promise-3.2.1.tgz",
+      "integrity": "sha1-7FYjOGgDKQkgcXDDlEjiREndH8Q="
+    },
+    "fs-minipass": {
+      "version": "1.2.5",
+      "resolved": "https://registry.npmjs.org/fs-minipass/-/fs-minipass-1.2.5.tgz",
+      "integrity": "sha512-JhBl0skXjUPCFH7x6x61gQxrKyXsxB5gcgePLZCwfyCGGsTISMoIeObbrvVeP6Xmyaudw4TT43qV2Gz+iyd2oQ==",
+      "requires": {
+        "minipass": "2.3.4"
+      }
+    },
+    "fs.realpath": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/fs.realpath/-/fs.realpath-1.0.0.tgz",
+      "integrity": "sha1-FQStJSMVjKpA20onh8sBQRmU6k8="
+    },
+    "gauge": {
+      "version": "2.7.4",
+      "resolved": "https://registry.npmjs.org/gauge/-/gauge-2.7.4.tgz",
+      "integrity": "sha1-LANAXHU4w51+s3sxcCLjJfsBi/c=",
+      "requires": {
+        "aproba": "1.2.0",
+        "console-control-strings": "1.1.0",
+        "has-unicode": "2.0.1",
+        "object-assign": "4.1.1",
+        "signal-exit": "3.0.2",
+        "string-width": "1.0.2",
+        "strip-ansi": "3.0.1",
+        "wide-align": "1.1.2"
+      }
+    },
+    "glob": {
+      "version": "7.1.2",
+      "resolved": "https://registry.npmjs.org/glob/-/glob-7.1.2.tgz",
+      "integrity": "sha512-MJTUg1kjuLeQCJ+ccE4Vpa6kKVXkPYJ2mOCQyUuKLcLQsdrMCpBPUi8qVE6+YuaJkozeA9NusTAw3hLr8Xe5EQ==",
+      "requires": {
+        "fs.realpath": "1.0.0",
+        "inflight": "1.0.6",
+        "inherits": "2.0.3",
+        "minimatch": "3.0.4",
+        "once": "1.4.0",
+        "path-is-absolute": "1.0.1"
+      }
+    },
+    "has-unicode": {
+      "version": "2.0.1",
+      "resolved": "https://registry.npmjs.org/has-unicode/-/has-unicode-2.0.1.tgz",
+      "integrity": "sha1-4Ob+aijPUROIVeCG0Wkedx3iqLk="
+    },
+    "iconv-lite": {
+      "version": "0.4.24",
+      "resolved": "https://registry.npmjs.org/iconv-lite/-/iconv-lite-0.4.24.tgz",
+      "integrity": "sha512-v3MXnZAcvnywkTUEZomIActle7RXXeedOR31wwl7VlyoXO4Qi9arvSenNQWne1TcRwhCL1HwLI21bEqdpj8/rA==",
+      "requires": {
+        "safer-buffer": "2.1.2"
+      }
+    },
+    "ignore-walk": {
+      "version": "3.0.1",
+      "resolved": "https://registry.npmjs.org/ignore-walk/-/ignore-walk-3.0.1.tgz",
+      "integrity": "sha512-DTVlMx3IYPe0/JJcYP7Gxg7ttZZu3IInhuEhbchuqneY9wWe5Ojy2mXLBaQFUQmo0AW2r3qG7m1mg86js+gnlQ==",
+      "requires": {
+        "minimatch": "3.0.4"
+      }
+    },
+    "inflight": {
+      "version": "1.0.6",
+      "resolved": "https://registry.npmjs.org/inflight/-/inflight-1.0.6.tgz",
+      "integrity": "sha1-Sb1jMdfQLQwJvJEKEHW6gWW1bfk=",
+      "requires": {
+        "once": "1.4.0",
+        "wrappy": "1.0.2"
+      }
+    },
+    "inherits": {
+      "version": "2.0.3",
+      "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.3.tgz",
+      "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4="
+    },
+    "ini": {
+      "version": "1.3.5",
+      "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.5.tgz",
+      "integrity": "sha512-RZY5huIKCMRWDUqZlEi72f/lmXKMvuszcMBduliQ3nnWbx9X/ZBQO7DijMEYS9EhHBb2qacRUMtC7svLwe0lcw=="
+    },
+    "is-fullwidth-code-point": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/is-fullwidth-code-point/-/is-fullwidth-code-point-1.0.0.tgz",
+      "integrity": "sha1-754xOG8DGn8NZDr4L95QxFfvAMs=",
+      "requires": {
+        "number-is-nan": "1.0.1"
+      }
+    },
+    "isarray": {
+      "version": "1.0.0",
+      "resolved": "https://registry.npmjs.org/isarray/-/isarray-1.0.0.tgz",
+      "integrity": "sha1-u5NdSFgsuhaMBoNJV6VKPgcSTxE="
+    },
+    "later": {
+      "version": "1.2.0",
+      "resolved": "https://registry.npmjs.org/later/-/later-1.2.0.tgz",
+      "integrity": "sha1-8s9sTdeVbdL1IK3wMpg26YdrrQ8="
+    },
+    "libxmljs": {
+      "version": "0.18.8",
+      "resolved": "https://registry.npmjs.org/libxmljs/-/libxmljs-0.18.8.tgz",
+      "integrity": "sha1-sKB1EqASkLYkBgDWwrwzo8cJdtY=",
+      "requires": {
+        "bindings": "1.3.0",
+        "nan": "2.10.0",
+        "node-pre-gyp": "0.9.1"
+      },
+      "dependencies": {
+        "bindings": {
+          "version": "1.3.0",
+          "resolved": "https://registry.npmjs.org/bindings/-/bindings-1.3.0.tgz",
+          "integrity": "sha512-DpLh5EzMR2kzvX1KIlVC0VkC3iZtHKTgdtZ0a3pglBZdaQFjt5S9g9xd1lE+YvXyfd6mtCeRnrUfOLYiTMlNSw=="
+        },
+        "nan": {
+          "version": "2.10.0",
+          "resolved": "http://registry.npmjs.org/nan/-/nan-2.10.0.tgz",
+          "integrity": "sha512-bAdJv7fBLhWC+/Bls0Oza+mvTaNQtP+1RyhhhvD95pgUJz6XM5IzgmxOkItJ9tkoCiplvAnXI1tNmmUD/eScyA=="
+        },
+        "node-pre-gyp": {
+          "version": "0.9.1",
+          "resolved": "https://registry.npmjs.org/node-pre-gyp/-/node-pre-gyp-0.9.1.tgz",
+          "integrity": "sha1-8RwHUW3ZL4cZnbx+GDjqt81WyeA=",
+          "requires": {
+            "detect-libc": "1.0.3",
+            "mkdirp": "0.5.1",
+            "needle": "2.2.4",
+            "nopt": "4.0.1",
+            "npm-packlist": "1.1.12",
+            "npmlog": "4.1.2",
+            "rc": "1.2.5",
+            "rimraf": "2.6.2",
+            "semver": "5.5.0",
+            "tar": "4.4.6"
+          }
+        },
+        "safe-buffer": {
+          "version": "5.1.2",
+          "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz",
+          "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g=="
+        },
+        "tar": {
+          "version": "4.4.6",
+          "resolved": "https://registry.npmjs.org/tar/-/tar-4.4.6.tgz",
+          "integrity": "sha512-tMkTnh9EdzxyfW+6GK6fCahagXsnYk6kE6S9Gr9pjVdys769+laCTbodXDhPAjzVtEBazRgP0gYqOjnk9dQzLg==",
+          "requires": {
+            "chownr": "1.1.1",
+            "fs-minipass": "1.2.5",
+            "minipass": "2.3.4",
+            "minizlib": "1.1.1",
+            "mkdirp": "0.5.1",
+            "safe-buffer": "5.1.2",
+            "yallist": "3.0.2"
+          }
+        }
+      }
+    },
+    "minimatch": {
+      "version": "3.0.4",
+      "resolved": "https://registry.npmjs.org/minimatch/-/minimatch-3.0.4.tgz",
+      "integrity": "sha512-yJHVQEhyqPLUTgt9B83PXu6W3rx4MvvHvSUvToogpwoGDOUQ+yDrR0HRot+yOCdCO7u4hX3pWft6kWBBcqh0UA==",
+      "requires": {
+        "brace-expansion": "1.1.11"
+      }
+    },
+    "minimist": {
+      "version": "0.0.8",
+      "resolved": "https://registry.npmjs.org/minimist/-/minimist-0.0.8.tgz",
+      "integrity": "sha1-hX/Kv8M5fSYluCKCYuhqp6ARsF0="
+    },
+    "minipass": {
+      "version": "2.3.4",
+      "resolved": "https://registry.npmjs.org/minipass/-/minipass-2.3.4.tgz",
+      "integrity": "sha512-mlouk1OHlaUE8Odt1drMtG1bAJA4ZA6B/ehysgV0LUIrDHdKgo1KorZq3pK0b/7Z7LJIQ12MNM6aC+Tn6lUZ5w==",
+      "requires": {
+        "safe-buffer": "5.1.2",
+        "yallist": "3.0.2"
+      },
+      "dependencies": {
+        "safe-buffer": {
+          "version": "5.1.2",
+          "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.2.tgz",
+          "integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g=="
+        }
+      }
+    },
+    "minizlib": {
+      "version": "1.1.1",
+      "resolved": "https://registry.npmjs.org/minizlib/-/minizlib-1.1.1.tgz",
+      "integrity": "sha512-TrfjCjk4jLhcJyGMYymBH6oTXcWjYbUAXTHDbtnWHjZC25h0cdajHuPE1zxb4DVmu8crfh+HwH/WMuyLG0nHBg==",
+      "requires": {
+        "minipass": "2.3.4"
+      }
+    },
+    "mkdirp": {
+      "version": "0.5.1",
+      "resolved": "https://registry.npmjs.org/mkdirp/-/mkdirp-0.5.1.tgz",
+      "integrity": "sha1-MAV0OOrGz3+MR2fzhkjWaX11yQM=",
+      "requires": {
+        "minimist": "0.0.8"
+      }
+    },
+    "mongodb": {
+      "version": "2.2.36",
+      "resolved": "https://registry.npmjs.org/mongodb/-/mongodb-2.2.36.tgz",
+      "integrity": "sha512-P2SBLQ8Z0PVx71ngoXwo12+FiSfbNfGOClAao03/bant5DgLNkOPAck5IaJcEk4gKlQhDEURzfR3xuBG1/B+IA==",
+      "requires": {
+        "es6-promise": "3.2.1",
+        "mongodb-core": "2.1.20",
+        "readable-stream": "2.2.7"
+      },
+      "dependencies": {
+        "mongodb-core": {
+          "version": "2.1.20",
+          "resolved": "https://registry.npmjs.org/mongodb-core/-/mongodb-core-2.1.20.tgz",
+          "integrity": "sha512-IN57CX5/Q1bhDq6ShAR6gIv4koFsZP7L8WOK1S0lR0pVDQaScffSMV5jxubLsmZ7J+UdqmykKw4r9hG3XQEGgQ==",
+          "requires": {
+            "bson": "1.0.4",
+            "require_optional": "1.0.1"
+          }
+        },
+        "process-nextick-args": {
+          "version": "1.0.7",
+          "resolved": "https://registry.npmjs.org/process-nextick-args/-/process-nextick-args-1.0.7.tgz",
+          "integrity": "sha1-FQ4gt1ZZCtP5EJPyWk8q2L/zC6M="
+        },
+        "readable-stream": {
+          "version": "2.2.7",
+          "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.2.7.tgz",
+          "integrity": "sha1-BwV6y+JGeyIELTb5jFrVBwVOlbE=",
+          "requires": {
+            "buffer-shims": "1.0.0",
+            "core-util-is": "1.0.2",
+            "inherits": "2.0.3",
+            "isarray": "1.0.0",
+            "process-nextick-args": "1.0.7",
+            "string_decoder": "1.0.3",
+            "util-deprecate": "1.0.2"
+          }
+        }
+      }
+    },
+    "ms": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/ms/-/ms-2.0.0.tgz",
+      "integrity": "sha1-VgiurfwAvmwpAd9fmGF4jeDVl8g="
+    },
+    "needle": {
+      "version": "2.2.4",
+      "resolved": "https://registry.npmjs.org/needle/-/needle-2.2.4.tgz",
+      "integrity": "sha512-HyoqEb4wr/rsoaIDfTH2aVL9nWtQqba2/HvMv+++m8u0dz808MaagKILxtfeSN7QU7nvbQ79zk3vYOJp9zsNEA==",
+      "requires": {
+        "debug": "2.6.9",
+        "iconv-lite": "0.4.24",
+        "sax": "1.2.4"
+      }
+    },
+    "nopt": {
+      "version": "4.0.1",
+      "resolved": "https://registry.npmjs.org/nopt/-/nopt-4.0.1.tgz",
+      "integrity": "sha1-0NRoWv1UFRk8jHUFYC0NF81kR00=",
+      "requires": {
+        "abbrev": "1.1.1",
+        "osenv": "0.1.5"
+      }
+    },
+    "npm-bundled": {
+      "version": "1.0.5",
+      "resolved": "https://registry.npmjs.org/npm-bundled/-/npm-bundled-1.0.5.tgz",
+      "integrity": "sha512-m/e6jgWu8/v5niCUKQi9qQl8QdeEduFA96xHDDzFGqly0OOjI7c+60KM/2sppfnUU9JJagf+zs+yGhqSOFj71g=="
+    },
+    "npm-packlist": {
+      "version": "1.1.12",
+      "resolved": "https://registry.npmjs.org/npm-packlist/-/npm-packlist-1.1.12.tgz",
+      "integrity": "sha512-WJKFOVMeAlsU/pjXuqVdzU0WfgtIBCupkEVwn+1Y0ERAbUfWw8R4GjgVbaKnUjRoD2FoQbHOCbOyT5Mbs9Lw4g==",
+      "requires": {
+        "ignore-walk": "3.0.1",
+        "npm-bundled": "1.0.5"
+      }
+    },
+    "npmlog": {
+      "version": "4.1.2",
+      "resolved": "https://registry.npmjs.org/npmlog/-/npmlog-4.1.2.tgz",
+      "integrity": "sha512-2uUqazuKlTaSI/dC8AzicUck7+IrEaOnN/e0jd3Xtt1KcGpwx30v50mL7oPyr/h9bL3E4aZccVwpwP+5W9Vjkg==",
+      "requires": {
+        "are-we-there-yet": "1.1.4",
+        "console-control-strings": "1.1.0",
+        "gauge": "2.7.4",
+        "set-blocking": "2.0.0"
+      }
+    },
+    "number-is-nan": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/number-is-nan/-/number-is-nan-1.0.1.tgz",
+      "integrity": "sha1-CXtgK1NCKlIsGvuHkDGDNpQaAR0="
+    },
+    "object-assign": {
+      "version": "4.1.1",
+      "resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz",
+      "integrity": "sha1-IQmtx5ZYh8/AXLvUQsrIv7s2CGM="
+    },
+    "once": {
+      "version": "1.4.0",
+      "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz",
+      "integrity": "sha1-WDsap3WWHUsROsF9nFC6753Xa9E=",
+      "requires": {
+        "wrappy": "1.0.2"
+      }
+    },
+    "os-homedir": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/os-homedir/-/os-homedir-1.0.2.tgz",
+      "integrity": "sha1-/7xJiDNuDoM94MFox+8VISGqf7M="
+    },
+    "os-tmpdir": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/os-tmpdir/-/os-tmpdir-1.0.2.tgz",
+      "integrity": "sha1-u+Z0BseaqFxc/sdm/lc0VV36EnQ="
+    },
+    "osenv": {
+      "version": "0.1.5",
+      "resolved": "https://registry.npmjs.org/osenv/-/osenv-0.1.5.tgz",
+      "integrity": "sha512-0CWcCECdMVc2Rw3U5w9ZjqX6ga6ubk1xDVKxtBQPK7wis/0F2r9T6k4ydGYhecl7YUBxBVxhL5oisPsNxAPe2g==",
+      "requires": {
+        "os-homedir": "1.0.2",
+        "os-tmpdir": "1.0.2"
+      }
+    },
+    "path-is-absolute": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/path-is-absolute/-/path-is-absolute-1.0.1.tgz",
+      "integrity": "sha1-F0uSaHNVNP+8es5r9TpanhtcX18="
+    },
+    "process-nextick-args": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/process-nextick-args/-/process-nextick-args-2.0.0.tgz",
+      "integrity": "sha512-MtEC1TqN0EU5nephaJ4rAtThHtC86dNN9qCuEhtshvpVBkAW5ZO7BASN9REnF9eoXGcRub+pFuKEpOHE+HbEMw=="
+    },
+    "rc": {
+      "version": "1.2.5",
+      "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.5.tgz",
+      "integrity": "sha1-J1zWh/bjs2zHVrqibf7oCnkDAf0=",
+      "requires": {
+        "deep-extend": "0.4.2",
+        "ini": "1.3.5",
+        "minimist": "1.2.0",
+        "strip-json-comments": "2.0.1"
+      },
+      "dependencies": {
+        "minimist": {
+          "version": "1.2.0",
+          "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
+          "integrity": "sha1-o1AIsg9BOD7sH7kU9M1d95omQoQ="
+        }
+      }
+    },
+    "readable-stream": {
+      "version": "2.3.4",
+      "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-2.3.4.tgz",
+      "integrity": "sha512-vuYxeWYM+fde14+rajzqgeohAI7YoJcHE7kXDAc4Nk0EbuKnJfqtY9YtRkLo/tqkuF7MsBQRhPnPeyjYITp3ZQ==",
+      "requires": {
+        "core-util-is": "1.0.2",
+        "inherits": "2.0.3",
+        "isarray": "1.0.0",
+        "process-nextick-args": "2.0.0",
+        "safe-buffer": "5.1.1",
+        "string_decoder": "1.0.3",
+        "util-deprecate": "1.0.2"
+      }
+    },
+    "redis": {
+      "version": "2.8.0",
+      "resolved": "https://registry.npmjs.org/redis/-/redis-2.8.0.tgz",
+      "integrity": "sha512-M1OkonEQwtRmZv4tEWF2VgpG0JWJ8Fv1PhlgT5+B+uNq2cA3Rt1Yt/ryoR+vQNOQcIEgdCdfH0jr3bDpihAw1A==",
+      "optional": true,
+      "requires": {
+        "double-ended-queue": "2.1.0-0",
+        "redis-commands": "1.4.0",
+        "redis-parser": "2.6.0"
+      }
+    },
+    "redis-commands": {
+      "version": "1.4.0",
+      "resolved": "https://registry.npmjs.org/redis-commands/-/redis-commands-1.4.0.tgz",
+      "integrity": "sha512-cu8EF+MtkwI4DLIT0x9P8qNTLFhQD4jLfxLR0cCNkeGzs87FN6879JOJwNQR/1zD7aSYNbU0hgsV9zGY71Itvw==",
+      "optional": true
+    },
+    "redis-parser": {
+      "version": "2.6.0",
+      "resolved": "https://registry.npmjs.org/redis-parser/-/redis-parser-2.6.0.tgz",
+      "integrity": "sha1-Uu0J2srBCPGmMcB+m2mUHnoZUEs=",
+      "optional": true
+    },
+    "require_optional": {
+      "version": "1.0.1",
+      "resolved": "https://registry.npmjs.org/require_optional/-/require_optional-1.0.1.tgz",
+      "integrity": "sha512-qhM/y57enGWHAe3v/NcwML6a3/vfESLe/sGM2dII+gEO0BpKRUkWZow/tyloNqJyN6kXSl3RyyM8Ll5D/sJP8g==",
+      "requires": {
+        "resolve-from": "2.0.0",
+        "semver": "5.5.0"
+      }
+    },
+    "resolve-from": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/resolve-from/-/resolve-from-2.0.0.tgz",
+      "integrity": "sha1-lICrIOlP+h2egKgEx+oUdhGWa1c="
+    },
+    "rimraf": {
+      "version": "2.6.2",
+      "resolved": "https://registry.npmjs.org/rimraf/-/rimraf-2.6.2.tgz",
+      "integrity": "sha512-lreewLK/BlghmxtfH36YYVg1i8IAce4TI7oao75I1g245+6BctqTVQiBP3YUJ9C6DQOXJmkYR9X9fCLtCOJc5w==",
+      "requires": {
+        "glob": "7.1.2"
+      }
+    },
+    "safe-buffer": {
+      "version": "5.1.1",
+      "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.1.1.tgz",
+      "integrity": "sha512-kKvNJn6Mm93gAczWVJg7wH+wGYWNrDHdWvpUmHyEsgCtIwwo3bqPtV4tR5tuPaUhTOo/kvhVwd8XwwOllGYkbg=="
+    },
+    "safer-buffer": {
+      "version": "2.1.2",
+      "resolved": "https://registry.npmjs.org/safer-buffer/-/safer-buffer-2.1.2.tgz",
+      "integrity": "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="
+    },
+    "sax": {
+      "version": "1.2.4",
+      "resolved": "https://registry.npmjs.org/sax/-/sax-1.2.4.tgz",
+      "integrity": "sha512-NqVDv9TpANUjFm0N8uM5GxL36UgKi9/atZw+x7YFnQ8ckwFGKrl4xX4yWtrey3UJm5nP1kUbnYgLopqWNSRhWw=="
+    },
+    "seedrandom": {
+      "version": "2.4.4",
+      "resolved": "https://registry.npmjs.org/seedrandom/-/seedrandom-2.4.4.tgz",
+      "integrity": "sha512-9A+PDmgm+2du77B5i0Ip2cxOqqHjgNxnBgglxLcX78A2D6c2rTo61z4jnVABpF4cKeDMDG+cmXXvdnqse2VqMA=="
+    },
+    "semver": {
+      "version": "5.5.0",
+      "resolved": "https://registry.npmjs.org/semver/-/semver-5.5.0.tgz",
+      "integrity": "sha512-4SJ3dm0WAwWy/NVeioZh5AntkdJoWKxHxcmyP622fOkgHa4z3R0TdBJICINyaSDE6uNwVc8gZr+ZinwZAH4xIA=="
+    },
+    "set-blocking": {
+      "version": "2.0.0",
+      "resolved": "https://registry.npmjs.org/set-blocking/-/set-blocking-2.0.0.tgz",
+      "integrity": "sha1-BF+XgtARrppoA93TgrJDkrPYkPc="
+    },
+    "signal-exit": {
+      "version": "3.0.2",
+      "resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-3.0.2.tgz",
+      "integrity": "sha1-tf3AjxKH6hF4Yo5BXiUTK3NkbG0="
+    },
+    "string-width": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/string-width/-/string-width-1.0.2.tgz",
+      "integrity": "sha1-EYvfW4zcUaKn5w0hHgfisLmxB9M=",
+      "requires": {
+        "code-point-at": "1.1.0",
+        "is-fullwidth-code-point": "1.0.0",
+        "strip-ansi": "3.0.1"
+      }
+    },
+    "string_decoder": {
+      "version": "1.0.3",
+      "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.0.3.tgz",
+      "integrity": "sha512-4AH6Z5fzNNBcH+6XDMfA/BTt87skxqJlO0lAh3Dker5zThcAxG6mKz+iGu308UKoPPQ8Dcqx/4JhujzltRa+hQ==",
+      "requires": {
+        "safe-buffer": "5.1.1"
+      }
+    },
+    "strip-ansi": {
+      "version": "3.0.1",
+      "resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-3.0.1.tgz",
+      "integrity": "sha1-ajhfuIU9lS1f8F0Oiq+UJ43GPc8=",
+      "requires": {
+        "ansi-regex": "2.1.1"
+      }
+    },
+    "strip-json-comments": {
+      "version": "2.0.1",
+      "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz",
+      "integrity": "sha1-PFMZQukIwml8DsNEhYwobHygpgo="
+    },
+    "util-deprecate": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz",
+      "integrity": "sha1-RQ1Nyfpw3nMnYvvS1KKJgUGaDM8="
+    },
+    "wide-align": {
+      "version": "1.1.2",
+      "resolved": "https://registry.npmjs.org/wide-align/-/wide-align-1.1.2.tgz",
+      "integrity": "sha512-ijDLlyQ7s6x1JgCLur53osjm/UXUYD9+0PbYKrBsYisYXzCxN+HC3mYDNy/dWdmf3AwqwU3CXwDCvsNgGK1S0w==",
+      "requires": {
+        "string-width": "1.0.2"
+      }
+    },
+    "wrappy": {
+      "version": "1.0.2",
+      "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",
+      "integrity": "sha1-tSQ9jz7BqjXxNkYFvA0QNuMKtp8="
+    },
+    "yallist": {
+      "version": "3.0.2",
+      "resolved": "https://registry.npmjs.org/yallist/-/yallist-3.0.2.tgz",
+      "integrity": "sha1-hFK0u36Dx8GI2AQcGoN8dz1ti7k="
+    }
+  }
+}

package.json

@@ -1,10 +1,10 @@
 {
   "name": "genieacs",
-  "version": "1.1.1",
+  "version": "1.1.3",
   "description": "A TR-069 Auto Configuration Server (ACS)",
   "repository": {
     "type": "git",
-    "url": "https://github.com/zaidka/genieacs.git"
+    "url": "https://github.com/genieacs/genieacs.git"
   },
   "homepage": "https://genieacs.com",
   "keywords": [
@@ -19,23 +19,24 @@
     "bin": "bin"
   },
   "dependencies": {
-    "hiredis": "^0.5.0",
     "later": "^1.2.0",
-    "libxmljs": "^0.18.4",
-    "mongodb": "^2.2.24",
-    "redis": "^2.6.5",
-    "seedrandom": "^2.4.2"
+    "libxmljs": "^0.18.8",
+    "mongodb": "^2.2.36",
+    "seedrandom": "^2.4.4"
+  },
+  "optionalDependencies": {
+    "redis": "^2.6.5"
   },
   "devDependencies": {
-    "coffeescript": "^1.12.4"
+    "coffeescript": "^1.12.7"
   },
   "engines": {
-    "node": ">=6.9.2 <8.0"
+    "node": ">=6.9.2 <9.0"
   },
   "scripts": {
     "prepublish": "npm run compile",
     "compile": "coffee -cb lib/*.coffee",
-    "configure": "cp -n ./config/config-sample.json ./config/config.json",
+    "configure": "if [ ! -f ./config/config.json ]; then cp ./config/config-sample.json ./config/config.json; fi",
     "install": "npm run configure",
     "test": "node test"
   }