Observability

for Node.js & Realtime Software

@dshaw

1. #bigdata
2. #chinamarket
3. #observability

@enterprisestack

Precondition:

Observability is hard

...especially at scale.

dshaw

Voxer

The Node Firm

NodeUp

Define: Observability

Observability is a measure for how well internal states of a system can be inferred by knowledge of its external outputs.

Realtime App Priorities

1. Scalability
2. Observability

Scale is why we build realtime apps. Observability is how we keep things sane at scale.

Observability: Why?

Sometimes things get out of control.

You'll want to know what's going on.

Observers Tools

1. logging
2. health/status endpoints
3. REPL
4. log aggregation
5. monitoring
6. metrics
7. DTrace

Logging

Logging is your friend.

• Don't be afraid of logging.
• Not logging, or severely limiting logging, is a ridiculous premature optimization.

Logging

Yes, you can go too far.

• Be careful with expensive to serialize objects.
• Noise reduces observability.

Health/Status Endpoints

Create a status or health endpoint.

• process.memory
• process.uptime
• process.versions
• process.pid
• connections
• proc: cpu microtime

/health

app.get('/health', function(req, res){
  res.send({
    pid: process.pid,
    memory: process.memoryUsage(),
    uptime: process.uptime(),
    connections: server.connections
  })
})
          

REPL

var net = require('net'),
  , repl = require('repl');

net.createServer(function (socket) {
  repl.start({
    prompt: 'repl> ',
    input: socket,
    output: socket
  }).on('exit', function() {
    socket.end();
  })
}).listen('/tmp/node-repl.sock');
          

REPL: no excuses

Use Replify.

npm install replify
          

var replify = require('replify')
  , io = require('socket.io').listen(8888)
  , app = io

io.on('connection', function (socket) {
  //...
})

replify('socket.io', app)
          

Access your replify REPL

netcat

$ nc -U /tmp/repl/socket.io.sock
          

socat

$ socat READLINE /tmp/repl/socket.io.sock
          

Let's make that easier too!

repl-client (rc)

npm install -g repl-client
          

$ rc /tmp/repl/socket.io.sock
          

repl-client (rc)

1. history^*
2. tab completion

(^* also possible with socat)

Log Aggregation

1. Loggly
2. Graylog2
3. Splunk
4. Winston/Winstond
5. files + grep

Monitoring

1. Boundary
2. Nagios
3. Reconnoiter
4. Circonus
5. Munin

Monitoring #RedisConf Edition

1. Reddish
2. RedisLive
3. Redis Monitor

Redis Monitor

Monitor a single Redis or a cluster of Redis servers.

Now visualize what's going on on those servers.

github.com/dshaw/redis-monitor

Redis: Use MONITOR for drop-in observation

Monitoring with Redis MONITOR is bad.

The MONITOR command is good and useful for debugging, but not monitoring.

redis.io/commands/monitor

Metrics

1. Etsy statsd
2. Something new soon from the Voxer Summer of Node.js Interns...

Summer of Node.js <3 Metrics

@sentientwaffle

llquantize

Metrics (can) tell a story

1. Visualization
2. Comparative Analysis
3. System Health
4. Event Triggers

Good Metrics

Evolve from the foundations provided by other observability measures.

Observability means optimizing software for humans.

NeverEnding Story