Snippets

When doing research on peer-to-peer networks, addressing can become pretty complex pretty quickly. Not everyone has the resources to allocate static, public facing IP addresses to machines. A machine that is in a home network for example only has a single public-facing IP address, usually assigned to the router. The router then performs NAT (network address translation) forwarding requests to internal devices. In order to get a service running on an internal network, you can port forward external requests to a specific port to a specific device. Requests are made to the router’s IP address, and the router passes it on. But how do you know the IP address of the device? Moreover, what happens if the router is assigned a new IP address? Static IP addresses generally cost more. ...

I’ve ben using Fabric to concurrently start multiple processes on several machines. These processes have to run at the same time (since they are experimental processes and are interacting with each other) and shut down at more or less the same time so that I can collect results and immediately execute the next sample in the experiment. However, I was having a some difficulties directly using Fabric: Fabric can parallelize one task across multiple hosts accordint to roles. Fabric can be hacked to run multiple tasks on multiple hosts by setting env.dedupe_hosts = False Fabric can only parallelize one type of task, not multiple types Fabric can’t handle large numbers of SSH connections In this post we’ll explore my approach with Fabric and my current solution. ...

In my current experimental setup, each process is a single instance of sample, from start to finish. This means that I need to aggregate results across multiple process runs that are running concurrently. Moreover, I may need to aggregate those results between machines. The most compact format to store results in is CSV. This was my first approach and it had some benefits including: small file sizes readability CSV files can just be concatenated together The problems were: ...

Was introduced to an interesting problem today when decorating tests that need to be discovered by the nose runner. By default, nose explores a directory looking for things named test or tests and then executes those functions, classes, modules, etc. as tests. A standard test suite for me looks something like: import unittest class MyTests(unittest.TestCase): def test_undecorated(self): """ assert undecorated works """ self.assertEqual(2+2, 4) The problem came up when we wanted to decorate a test with some extra functionality, for example loading a fixture: ...

I have had some recent discussions regarding cacheing to improve application performance that I wanted to share. Most of the time those conversations go something like this: “have you heard of Redis?” I’m fascinated by the fact that an independent, distributed key-value store has won the market to this degree. However, as I’ve pointed out in these conversations, cacheing is a hierarchy (heck, even the processor has varying levels of cacheing). Especially when considering micro-service architectures that require extremely low latency responses, cacheing should be a critical part of the design, not just a bolt-on after thought! ...

This week I discovered I had a problem with my Google Calendar — events accidentally got duplicated or deleted and I needed a way to verify that my primary calendar was correct. Rather than painstakingly go through the web interface and spot check every event, I instead wrote a Go console program using the Google Calendar API to retrieve events and save them in a CSV so I could inspect them all at once. This was great, and very easy using Google’s Go libraries for their APIs, and the quick start was very handy. ...

I routinely have long-running scripts (e.g. for a data processing task) that I want to know when they’re complete. It seems like it should be simple for me to add in a little snippet of code that will send an email using Gmail to notify me, right? Unfortunately, it isn’t quite that simple for a lot of reasons, including security, attachment handling, configuration, etc. In this snippet, I’ve attached my constant copy and paste notify() function, written into a command line script for easy sending on the command line. ...