HTTP/S capture using mitmproxy


This post shows how to install mitmproxy on Mac OS X (El Capitan) to capture HTTP/S traffic, especially useful when debugging applications.

I’ve been using Telerik Fiddler on Windows for sniffing HTTP/S and WebSocket traffic, but it isn’t very reliable on Mac or Linux. mitmproxy fills the lacuna well, but it does not yet support WebSocket traffic.

Install

Use pip to install mitmproxy thus

pip install mitmproxy

I encountered several compilation issues while installing through pip. I’ll go through them one by one. The first error results from failure to compile cryptography

    building '_openssl' extension
    clang -fno-strict-aliasing -fno-common -dynamic -I/usr/local/include -I/usr/local/opt/sqlite/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c build/temp.macosx-10.10-x86_64-2.7/_openssl.c -o build/temp.macosx-10.10-x86_64-2.7/build/temp.macosx-10.10-x86_64-2.7/_openssl.o
    build/temp.macosx-10.10-x86_64-2.7/_openssl.c:431:10: fatal error: 'openssl/aes.h' file not found
    #include <openssl/aes.h>
             ^
    1 error generated.
    error: command 'clang' failed with exit status 1

That can be resolved by executing pip to install cryptography thus

env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" pip install mitmproxy

With that dependency resolved, mitmproxy install fails with the following error

    building 'lxml.etree' extension
    clang -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/usr/local/opt/openssl/include -I/usr/include/libxml2 -Isrc/lxml/includes -I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.macosx-10.10-x86_64-2.7/src/lxml/lxml.etree.o -w -flat_namespace
    In file included from src/lxml/lxml.etree.c:323:
    src/lxml/includes/etree_defs.h:14:10: fatal error: 'libxml/xmlversion.h' file not found
    #include "libxml/xmlversion.h"
             ^
    1 error generated.
    Compile failed: command 'clang' failed with exit status 1
    cc -I/usr/include/libxml2 -I/usr/include/libxml2 -c /var/folders/3v/zgzrr9h96_34db7lt9_fx1wr0000gn/T/xmlXPathInitdIvQjA.c -o var/folders/3v/zgzrr9h96_34db7lt9_fx1wr0000gn/T/xmlXPathInitdIvQjA.o
    /var/folders/3v/zgzrr9h96_34db7lt9_fx1wr0000gn/T/xmlXPathInitdIvQjA.c:1:10: fatal error: 'libxml/xpath.h' file not found
    #include "libxml/xpath.h"
             ^
    1 error generated.
    *********************************************************************************
    Could not find function xmlCheckVersion in library libxml2. Is libxml2 installed?
    Perhaps try: xcode-select --install
    *********************************************************************************

Luckily, that error also shows the solution, run

xcode-select --install

Now, mitmproxy should install successfully.

Run

To capture HTTP/S traffic using mitmproxy traffic, run

mitmproxy

mitmproxy should show which port it is listening at; 8080 is the default. Use http://localhost:8080 as the HTTP proxy setting in browsers and applications.

Android emulator

This is how you can execute Android emulator to use mitmproxy as an HTTP proxy

export DYLD_FALLBACK_LIBRARY_PATH=~/Library/Android/sdk/tools/lib64
~/Library/Android/sdk/tools/emulator64-x86 -avd Nexus_S_API_21_x86 -http-proxy http://localhost:8080

The first line is needed so that the emulator can find the necessary libraries such as OpenGLES emulation library.

Pinned Certificates

If you try to access any site in the Android browser, or run any application that uses HTTP/S, mitmproxy will capture all traffic. To capture SSL traffic mitmproxy presents its own certificate to the applications. The root certificate that mitmproxy uses will need to be added to the certificate store, to avoid failures in certificate chain validation. This can be done by navigating to the special mitm.it URL in the browser, and picking your platform from the resulting page.

If you use certificate pinning in your applications, you can add ~/.mitmproxy/mitmproxy-ca-cert.cer to the list of certificates.

WebSocket traffic

mitmproxy does not support WebSocket traffic so connection establishment will fail. You can however setup mitmproxy to ignore traffic to a certain host:port. This can be leveraged to ask it to ignore WebSocket traffic.

mitmproxy --ignore 192\.168\.1\.10:888[1-9]

Folders consuming most disk space using du


du is available natively on almost all Linux distributions, and on Mac OS X. If you are in need of reclaiming disk space, and want to quickly find which folders to focus your attention on, run the following command

du -h -d 1

That will quickly list all the folders under the current folder and their disk space usage. Use the following command to check space left on each disk

df -h

USB Serial


Serial port access can be very useful during embedded systems development.

I do most of my development on Mac OS X, or Ubuntu and Windows virtual machines. I use a USB to serial cable/breakout to connect a serial port to the Mac, which is then redirected by Parallels Desktop to the guest OS.

If you have a cable that uses the Prolific USB Serial chipset, getting up and running is well documented by Plugable. The driver they provide works for me. Most other cables use a chipset from FTDI, but Mac OS X already provides a driver for that.

To interact with a terminal on the embedded system you need some kind of terminal emulator.

I use miniterm.py and screen on Ubuntu

miniterm.py -b 115200 /dev/ttyUSB0

Ensure that you have access to the device

sudo chmod 777 /dev/ttyUSB0

On Mac OS X I usually use screen

screen /dev/cu.usbserial 115200

Monthly news review


This post reviews news in the month that has passed.

Everything Google announced at Google I/O 2014 in one handy list

Google announced a lot in one long keynote at I/O. Android One, Android Auto, Android Wear, and Android TV, are probably the big announcements this year. Looks like Google is prefixing all mobile-oriented hardware with Android, web-oriented hardware with Chrome, and services with Google. Unless it is something coming out of Nest, who’ve just launched a developer program for the programmable home.

Aereo Lost. What Now?

TV broadcasters are celebrating while the tech industry is up in arms. Are customers the real losers? Why should we need airwaves to transmit TV in the era of mobile internet? Why hasn’t the TV business adopted on-demand programming more actively? It isn’t as if their business isn’t being slowly driven to the ground.

This is Microsoft’s first Android smartphone, the Nokia X2

Nokia could have hedged its bets with Android a long while back. So why now? It is a cheap but attractive Android Smartphone full of Microsoft software and services.

Amazon’s Fire phone launch: Hits, misses, and takeaways

From inexpensive tablets to a fairly expensive phone, Amazon as come a long way. Lack of Bluetooth 4.0 (especially Smart) is annoying. It does seem to sport universal LTE, like the Moto G 4G. Will the Fire Phone truly delight users?

Google Donates Mod_Spdy To The Apache Foundation

HTTP 2.0 is around the corner and changes one crucial aspect of HTTP 1.x. It will no longer be a text-based protocol. An important feature is that data will be multiplexed over a single connection a browser maintains with a server.

Docker hopes its container platform will ease the lives of developers

Will an open container help big companies and other providers overcome the dependency on and momentum of Amazon, Google, and Microsoft?

Turing Test breakthrough as super-computer becomes first to convince us it’s human

Not everybody is convinced though, but the implications are important nevertheless. How do you know an e-mail message wasn’t sent by a real person? Are we at the cusp of having to deal with endless amount of believable spam? Looking at the positive aspects, customer support, distance education, and other areas that depend on personal interaction, may benefit.

Skype Translator Will Change the World

Real-time voice translation is a hard problem. You have to translate speech to text. The text then needs to be translated to the target language. The translated text then needs to be converted to speech. Imagine doing all of that in real time. Imagine doing that wrong in a UN session discussing climate change.

Google’s secretive 3D-mapping project now has a tablet

Google is keen to map the indoors. I see huge potential for indoor mapping. Imagine your interior designer mapping your house so that she can show you exactly how your renovated indoors will look? A robot that can go about your house tidying it? Are we heading towards becoming Wall-E lazy?

Apple announces iOS 8 at WWDC 2014

Easily one of Apple’s best WWDC considering all the news. A new programming language called Swift, Mac OS X Yosemite, Metal, HomeKit, CloudKit, and extensions in iOS 8. For those eager to learn Swift, Apple has provided an iBook for it already. A good news for all Netflix viewers, Safari on Mac OS X Yosemite now allows streaming using HTML5, no Silverlight required.

Dual SIM LG L40 (D175F) with Android KitKat


The cost-benefit of Android Smartphones continues to impress. LG L40 (D175F) is no Moto G, but it is cheaper, and a wonderful piece of hardware. I recently bought one for my wife. It currently retails for R$ 499 (Brazilian Reals) or about US $ 225.

LG-D175F

More details follow

  • Very responsive overall
  • In black or white
  • Double tap to lock or use (or achieve the same with power side button)
  • Android 4.4.2 (KitKat)
  • Dual core 1.2 GHz CPU
  • 320×480 pixel, 3.5 inch LCD screen with capacitive touch
  • Digital TV (Brazilian DTV standard for the model cited)
  • FM radio
  • 3.15 mega-pixel (2048×1350 pixels) camera (no AF)
  • No front camera
  • Up to 32 GB microSD card (I bought a 16 GB type 4 card)
  • Dual SIM (normal mini SIM, not micro or nano)
  • 2G/3G data (supports tethering other devices over Wi-Fi i.e. hotspot)
  • Wi-Fi
  • 512 MB RAM
  • 4 GB internal flash storage (1.64 GB available to user)
  • A-GPS support
  • Rotation sensing with built-in accelerometer
  • Bluetooth 4.0 with LE (was able to detect an iBeacon emulated on Mac OS X)
  • MTP (media transfer protocol) works fine with Windows (transfer arbitrary files to/from internal storage and SD card)
  • MTP doesn’t work with Mac OS X Mavericks, but Android File Transfer does the job

Emulating Bluetooth LE Peripheral on Mac OS X Mavericks


This post documents some interesting Mac OS X Bluetooth LE peripheral examples I have found.

Arbitrary Bluetooth Peripheral

Clone example from https://github.com/sandeepmistry/osx-ble-peripheral.

An issue is that it duplicates peripheral services after Bluetooth is turned off/on. Calling removeAllServices of CBPeripheralManager, fixes that.

You can also use IOBluetoothDevice API to iterate through connected Central devices. We found situations when after starting the peripheral app, the central app running on an iPhone would not establish Bluetooth LE communication. We were able to use IOBluetoothDevice to force disconnect all connected devices at app start on the Mac, to skirt the issue.

Advertising an iBeacon

Clone example from https://github.com/mttrb/BeaconOSX. Appears on the LightBlue app, but need to actually use it as an iBeacon. Check author’s blog at http://www.blendedcocoa.com/blog/2013/11/02/mavericks-as-an-ibeacon/ for further details. On a side note, you can create an iBeacon using a Raspberry Pi and an off-the-shelf USB dongle.

Mac OS X tips


This is an ever expanding list of tips for Mac OS X. Leave a comment below if you have tips of your own to share.

Go to a folder in Finder

Hit Command-Shift-G and type in or right click and paste the folder path.

Go to a folder in Terminal from Finder

Start Terminal. Type “cd ” without quotes. Drag folder from Finder (or its status bar) to Terminal. You can also open Terminal directly from Finder. Head over to System Preferences, Keyboard, Shortcuts, Services, and enable New Terminal at Folder and/or New Terminal Tab at Folder. Selected option will appear under the Services context menu of an item in Finder when you right click on it.

Goto to a folder in Finder from Terminal

Type the command

open .

Connect to WiFi and continue using your wired internet connection

Head over to System Preferences, Network, Set Service Order, and raise Thunderbolt Ethernet (or any other interface) above WiFi.

Live webcam feed in a web meeting

To show a live webcam feed on your screen, try Photo Booth (comes preinstalled). Most off-the-shelf USB webcams work just fine with Mac OS X and Photo Booth.

Combine multiple PDF documents

You can use to Preview to combine multiple PDF documents, rearranging and leaving out pages you don’t need…

Go to folder in Spotlight Search

Once you’ve searched for the document and it is highlighted, keying Enter will open the document, keying Command+Enter will take you to the folder where the document is located.

Keyboard Shortcuts

Fn+F11 reveals the desktop. Useful to quickly drag some files on to the Desktop or vice-versa.