How X Window Managers Work, And How To Write One (Part III)

In Part II of this series, we discussed X libraries and implementation choices, and examined the basic structure of a window manager. In Part III, we will start interacting with client windows and the user through events. We will review the fundamentals of window manager implementation, using the implementation in our example non-compositing reparenting window manager, basic_wm, for reference.

Step 4: Interaction with Application Windows

Following the steps in Part II of this series, we now have a basic skeleton for our window manager. Our next step is to start talking to clients and the user via events.

The interaction between clients, X, and the window manager is fairly complex. To facilitate our discussion, I’ve created a diagram that illustrates the flow of events throughout the lifetime of a client window, and how a window manager might respond to each of them. We’ll be referring to this cheat sheet for window manager event handling throughout this series. You can click through for the full-sized diagram.

In general, a window manager must handle two kinds of actions: those initiated by client applications (such as creating new windows), and those initiated by users (such as moving or minimizing windows). In this diagram, actions initiated by client applications are shown in the yellow box on the left hand side, and actions initiated by users are shown in blue on the right hand side. A window manager communicates with client applications via events, which are represented as parallelograms in red.

You may have noticed that some of the events in this diagram have the suffix Request, while others have the suffix Notify. This distinction is crucial to our discussion.

Recalling our discussion in Part I on substructure redirection, when a client application wants to do something with a window (such as moving, resizing, showing, or hiding), its request is redirected to the window manager, which can grant, modify, or deny the request. Such requests are delivered to a window manager as events with the Request suffix. It is important to understand that when a window manager receives such an event, the action it represents has not actually occurred, and it is the responsibility of the window manager to decide what to do with it. If the window manager does nothing, the request is implicitly denied.

On the other hand, events with the Notify suffix represent actions that have already been executed by the X server. The window manager can respond to such events, but of course cannot change the fact that they have already happened.

With that in mind, let’s dive into the implementation by looking at how our example window manager will handle the life cycle of a client window from creation to destruction.

Creating a Window

When an X client application creates a top-level window (XCreateWindow()), our window manager will receive a CreateNotify event. However, a newly created window is always invisible, so there’s nothing for our window manager to do. In window_manager.cpp:

void WindowManager::Run() {
  ...
  // 2. Main event loop.
  for (;;) {
    // 1. Get next event.
    ...
    // 2. Dispatch event.
    switch (e.type) {
      ...
      case CreateNotify:
        OnCreateNotify(e.xcreatewindow);
        break;
      ...
    }
  }
}

void WindowManager::OnCreateNotify(const XCreateWindowEvent& e) {}

Configuring a Newly Created Window

At this stage, the application can configure the window to set its initial size, position, or other attributes. To do so, the application would invoke XConfigureWindow(), which would send a ConfigureRequest event to the window manager. However, since the window is still invisible, the window manager doesn’t need to care and can grant such requests without modification by invoking XConfigureWindow() itself with the same parameters.

void WindowManager::Run() {
      ...
      case ConfigureRequest:
        OnConfigureRequest(e.xconfigurerequest);
        break;
      ...
}

void WindowManager::OnConfigureRequest(const XConfigureRequestEvent& e) {
  XWindowChanges changes;
  // Copy fields from e to changes.
  changes.x = e.x;
  changes.y = e.y;
  changes.width = e.width;
  changes.height = e.height;
  changes.border_width = e.border_width;
  changes.sibling = e.above;
  changes.stack_mode = e.detail;
  // Grant request by calling XConfigureWindow().
  XConfigureWindow(display_, e.window, e.value_mask, &changes);
  LOG(INFO) << "Resize " << e.window << " to " << Size<int>(e.width, e.height);
}

Mapping a Window

To make the window finally visible on screen, the client application will call XMapWindow() to map it. This sends a MapRequest event to the window manager. As noted earlier, at this point, the window is still not yet visible, as it’s up to the window manager to actually make it so. This is probably the most important event in our discussion, as this is where a window manager would usually start really managing a window.

A reparenting window manager would typically respond to a MapRequest for a client application window w with the following actions:

  1. Create a frame window f, perhaps with borders and window decoration (e.g. title, minimize / maximize / close buttons).

  2. Register for substructure redirect on f with XSelectInput(). Recall that substructure redirect only applies to direct child windows, so after reparenting, the substructure redirect previously registered on the root window would no longer apply to w, hence this step.

  3. Make w a child of f with XReparentWindow().

  4. Render f and w with XMapWindow().

  5. Register for mouse or keyboard shortcuts on w and/or f.

The example implementation in basic_wm will create a very simple frame window that has the same size as the client window, but with a 3px red border:

void WindowManager::Run() {
      ...
      case MapRequest:
        OnMapRequest(e.xmaprequest);
        break;
      ...
}

void WindowManager::OnMapRequest(const XMapRequestEvent& e) {
  // 1. Frame or re-frame window.
  Frame(e.window);
  // 2. Actually map window.
  XMapWindow(display_, e.window);
}

void WindowManager::Frame(Window w) {
  // Visual properties of the frame to create.
  const unsigned int BORDER_WIDTH = 3;
  const unsigned long BORDER_COLOR = 0xff0000;
  const unsigned long BG_COLOR = 0x0000ff;

  // 1. Retrieve attributes of window to frame.
  XWindowAttributes x_window_attrs;
  CHECK(XGetWindowAttributes(display_, w, &x_window_attrs));

  // 2. TODO - see Framing Existing Top-Level Windows section below.

  // 3. Create frame.
  const Window frame = XCreateSimpleWindow(
      display_,
      root_,
      x_window_attrs.x,
      x_window_attrs.y,
      x_window_attrs.width,
      x_window_attrs.height,
      BORDER_WIDTH,
      BORDER_COLOR,
      BG_COLOR);
  // 3. Select events on frame.
  XSelectInput(
      display_,
      frame,
      SubstructureRedirectMask | SubstructureNotifyMask);
  // 4. Add client to save set, so that it will be restored and kept alive if we
  // crash.
  XAddToSaveSet(display_, w);
  // 5. Reparent client window.
  XReparentWindow(
      display_,
      w,
      frame,
      0, 0);  // Offset of client window within frame.
  // 6. Map frame.
  XMapWindow(display_, frame);
  // 7. Save frame handle.
  clients_[w] = frame;
  // 8. Grab events for window management actions on client window.
  //   a. Move windows with alt + left button.
  XGrabButton(...);
  //   b. Resize windows with alt + right button.
  XGrabButton(...);
  //   c. Kill windows with alt + f4.
  XGrabKey(...);
  //   d. Switch windows with alt + tab.
  XGrabKey(...);

  LOG(INFO) << "Framed window " << w << " [" << frame << "]";
}

The outline of the code should be fairly clear following our discussion. A few additional points to note:

  • Regarding the save-set and XAddToSaveSet():

    The save-set is a list of windows, usually maintained by the window manager, but including only windows created by other clients. If the window manager dies, all windows listed in the save-set will be reparented back to their closest living ancestor if they were reparented in the first place and mapped if the window manager has unmapped them so that it could map an icon.

    The save-set is necessary because the window manager might not exit normally. The user might kill it with CTRL-C if it is running in the foreground, or more likely, the user might get the process number and kill it. Actually, the actions of the save-set are performed even if the window manager exits normally, so less code is needed since the save-set does the cleaning up.

    Window managers almost always place in the save-set all the windows they reparent or iconify, using XAddToSaveSet().

    Windows are automatically removed from the save-set when they are destroyed.

    — Xlib Programming Manual §16.4
  • When our window manager creates a frame window (step 2 in the example code), it will also trigger a CreateNotify event for the frame window. It will ignore it just like it ignores other CreateNotify events as discussed earlier.

  • When our window manager calls XReparentWindow() in step 5 in the example code, it will trigger a ReparentNotify event, which it will ignore:

    void WindowManager::Run() {
          ...
          case ReparentNotify:
            OnReparentNotify(e.xreparent);
            break;
          ...
    }
    
    void WindowManager::OnReparentNotify(const XReparentEvent& e) {}
    
  • When our window manager calls XMapWindow() to map the frame window (step 6 in the example code), the X server knows that the action originates from the current window manager, and will execute it directly instead of redirecting it back as a MapRequest event. Our window manager will later receive a MapNotify event, which it can ignore:

    void WindowManager::Run() {
          ...
          case MapNotify:
            OnMapNotify(e.xmap);
            break;
          ...
    }
    
    void WindowManager::OnMapNotify(const XMapEvent& e) {}
    

Configuring a Mapped Window

A client application can configure a window that is currently visible, again with the XConfigureWindow() function. For example, an application may want to resize a window to better accomodate its contents. When a reparenting window manager receives the resulting ConfigureRequest and decides to grant the request, it additionally needs to resize / reposition the corresponding frame window and any window decorations.

void WindowManager::OnConfigureRequest(const XConfigureRequestEvent& e) {
  XWindowChanges changes;
  // Copy fields from e to changes.
  ...
  if (clients_.count(e.window)) {
    const Window frame = clients_[e.window];
    XConfigureWindow(display_, frame, e.value_mask, &changes);
    LOG(INFO) << "Resize [" << frame << "] to " << Size<int>(e.width, e.height);
  }
  // Grant request by calling XConfigureWindow().
  ...
}

When our window manager re-configures the frame window with the XConfigureWindow() call above, the X server knows that the action originates from the current window manager, and will execute it directly instead of redirecting it back as a ConfigureRequest event. Our window manager will then receive a ConfigureNotify event, which it will ignore:

void WindowManager::Run() {
      ...
      case ConfigureNotify:
        OnConfigureNotify(e.xconfigure);
        break;
      ...
}

void WindowManager::OnConfigureNotify(const XConfigureEvent& e) {}

Unmapping a Window

When a client application unmaps (i.e. hides) a window with XUnmapWindow(), for example in response to the user exiting or minimizing the application, the window manager will receive a UnmapNotify event. Unlike the MapRequest event, the UnmapNotify event is delivered to the window manager after the fact, and the window manager can only respond to it, not intercept it.

A reparenting window manager will typically want to reverse the actions it performed in response to MapRequest. In other words, it would reparent the client window back to the root window, and destroy the corresponding frame window.

void WindowManager::Run() {
      ...
      case UnmapNotify:
        OnUnmapNotify(e.xunmap);
        break;
      ...
}

void WindowManager::OnUnmapNotify(const XUnmapEvent& e) {
  // If the window is a client window we manage, unframe it upon UnmapNotify. We
  // need the check because we will receive an UnmapNotify event for a frame
  // window we just destroyed ourselves.
  if (!clients_.count(e.window)) {
    LOG(INFO) << "Ignore UnmapNotify for non-client window " << e.window;
    return;
  }

  Unframe(e.window);
}

void WindowManager::Unframe(Window w) {
  // We reverse the steps taken in Frame().
  const Window frame = clients_[w];
  // 1. Unmap frame.
  XUnmapWindow(display_, frame);
  // 2. Reparent client window back to root window.
  XReparentWindow(
      display_,
      w,
      root_,
      0, 0);  // Offset of client window within root.
  // 3. Remove client window from save set, as it is now unrelated to us.
  XRemoveFromSaveSet(display_, w);
  // 4. Destroy frame.
  XDestroyWindow(display_, frame);
  // 5. Drop reference to frame handle.
  clients_.erase(w);

  LOG(INFO) << "Unframed window " << w << " [" << frame << "]";
}

A few additional points to note:

  • When our window manager unmaps the frame window with XUnmapWindow() in step 1 in the example code above, it will again receive a corresponding UnmapNotify event. This is the reason why the UnmapNotify event handler needs to check that the unmapped window is an actual client window.

  • When our window manager makes the client window a direct child of the root window with XReparentWindow() in step 2 above, it will receive a ReparentNotify event. As discussed in the Mapping a Window section above, this ReparentNotify event will be ignored.

  • When our window manager destroys the frame window with XDestroyWindow() in step 4, it will trigger a DestroyNotify event. This event will also be ignored, as shown in the next section.

At this point, the client window has become invisible, but not yet destroyed. It can be displayed again with a call to XMapWindow(), which would take us back to the Mapping a Window step. It could also be reconfigured in this state, which would take us back to the Configuring a Newly Created Window step.

Destroying a Window

When a client application exits or no longer needs a window, it will call XDestroyWindow() to dispose of the window. This triggers a DestroyNotify event. In our case, there’s nothing we need to do in response.

void WindowManager::Run() {
      ...
      case DestroyNotify:
        OnDestroyNotify(e.xdestroywindow);
        break;
      ...
}

void WindowManager::OnDestroyNotify(const XDestroyWindowEvent& e) {}

Framing Existing Top-Level Windows

Now that we’ve walked through the life cycle of a client window, from creation to destruction, let’s turn our attention to the problem of existing top-level windows.

You may recall from Part I that X applications in general run just fine without a window manager. Depending on how an X session is started (e.g. xinitrc), by the time a window manager starts, any number of windows may have already been created by other applications. Additionally, the user can kill a running window manager and replace it with a different window manager, without affecting windows from other applications.

Therefore, when our window manager starts up, it needs to handle any existing top-level windows that are already mapped. As a reparenting window manager, it will invoke the same Frame() function on such windows as if these windows are being mapped for the first time:

void WindowManager::Run() {
  // 1. Initialization.
  //   a. Select events on root window. Use a special error handler so we can
  //   exit gracefully if another window manager is already running.
  ...
  //   b. Set error handler.
  ...
  //   c. Grab X server to prevent windows from changing under us while we
  //   frame them.
  XGrabServer(display_);
  //   d. Frame existing top-level windows.
  //     i. Query existing top-level windows.
  Window returned_root, returned_parent;
  Window* top_level_windows;
  unsigned int num_top_level_windows;
  CHECK(XQueryTree(
      display_,
      root_,
      &returned_root,
      &returned_parent,
      &top_level_windows,
      &num_top_level_windows));
  CHECK_EQ(returned_root, root_);
  //     ii. Frame each top-level window.
  for (unsigned int i = 0; i < num_top_level_windows; ++i) {
    Frame(top_level_windows[i], true /* was_created_before_window_manager */);
  }
  //     iii. Free top-level window array.
  XFree(top_level_windows);
  //   e. Ungrab X server.
  XUngrabServer(display_);

  // 2. Main event loop.
  ...
}

void WindowManager::OnMapRequest(const XMapRequestEvent& e) {
  // 1. Frame or re-frame window.
  Frame(e.window, false /* was_created_before_window_manager */);
  ...
}

void WindowManager::Frame(Window w, bool was_created_before_window_manager) {
  ...
  // 1. Retrieve attributes of window to frame.
  ...
  // 2. If window was created before window manager started, we should frame
  // it only if it is visible and doesn't set override_redirect.
  if (was_created_before_window_manager) {
    if (x_window_attrs.override_redirect ||
        x_window_attrs.map_state != IsViewable) {
      return;
    }
  }
  // 3. Create frame.
  ...
}

void WindowManager::OnUnmapNotify(const XUnmapEvent& e) {
  ...

  // Ignore event if it is triggered by reparenting a window that was mapped
  // before the window manager started.
  //
  // Since we receive UnmapNotify events from the SubstructureNotify mask, the
  // event attribute specifies the parent window of the window that was
  // unmapped. This means that an UnmapNotify event from a normal client window
  // should have this attribute set to a frame window we maintain. Only an
  // UnmapNotify event triggered by reparenting a pre-existing window will have
  // this attribute set to the root window.
  if (e.event == root_) {
    LOG(INFO) << "Ignore UnmapNotify for reparented pre-existing window "
              << e.window;
    return;
  }

  Unframe(e.window);
}

Some additional things to note:

  • You may notice that the process of framing existing top-level windows is guarded by XGrabServer() and XUngrabServer(). From the Xlib Programming Manual:

    These functions can be used to control processing of output on other connections by the window system server. While the server is grabbed, no processing of requests or close downs on any other connection will occur.

    — Xlib Programming Manual §9.5

    By grabbing the X server, our window manager ensures that, between the time when it fetches the list of existing top-level windows and when it finishes framing them, no other application can interfere and mess up our state: no new windows can be created, and no existing windows can be modified or destroyed.

  • The override_redirect attribute, if set to true, indicates that a window should not be managed by window managers. From the Xlib Programming Manual:

    To control window placement or to add decoration, a window manager often needs to intercept (redirect) any map or configure request. Pop-up windows, however, often need to be mapped without a window manager getting in the way. […]

    The override-redirect flag specifies whether map and configure requests on this window should override a SubstructureRedirectMask on the parent. You can set the override-redirect flag to True or False (default). Window managers use this information to avoid tampering with pop-up windows […].

    — Xlib Programming Manual §3.2.8

    The reason our window manager doesn’t need to check for this attribute except at start up is that the X server knows not to redirect events from such windows:

    The window manager […] will normally ignore windows that are mapped with their override_redirect attribute set, since no *Request events will be generated for them.

    — Xlib Programming Manual §16.3
  • The map_state attribute indicates whether a window is currently visible (mapped). When Frame() is invoked for pre-existing windows during start up, we want to ignore windows that are currently unmapped. However, when Frame() is invoked during the event loop as part of the MapRequest handler, we know that the client window to be framed is necessarily still unmapped, as our window manager wouldn’t have granted the request yet.

  • You might be wondering why an additional check for e.event == root_ is needed in the UnmapNotify handler. It turns out that reparenting an already mapped window (XReparentWindow()) will trigger a pair of UnmapNotify and MapNotify events in addition to ReparentNotify. Therefore, when we enter into the event loop, we will receive an UnmapNotify event for every pre-existing top-level window we reparented. We can distinguish these events by their event attribute, which in this case represents the parent of the client window. Normally, when a client window we already framed is unmapped, the event attribute would be its frame window. But when a pre-existing window is reparented at start up, the event attribute in the resulting UnmapNotify event will be its original parent - i.e., the root window.

What’s Next

At this point, we have a basic but functional reparenting window manager that will correctly handle the life cycle of windows. If you strip out window decorations, shortcuts and fancy UI, the core structure of every X window manager will quite closely resemble what we have here.

In our next installment, we will improve the user-facing functionality of our window manager by adding ways to move, resize and close windows. In the meantime, you’re more than welcome to check out the code for basic_wm on GitHub.

Flashing a Sprint Nexus S 4G to Verizon

I originally wrote the following post in July 2012 to document how to fully flash a Nexus S 4G from Sprint to Verizon Wireless, but never got to publishing it. I have long since switched away from Verizon Wireless and no longer own any of the phones mentioned, and decided to publish it for what it’s worth.

If you’re interested in the high-level view of how CDMA phones are programmed, check out my previous article Carrier Programming on CDMA Android Phones.

Introduction

This is a guide for flashing a Sprint Samsung Nexus S 4G to a standard Verizon monthly plan, and voice, texting and 3G data will all be fully functional. This method will likely also work on many other phones, especially Samsung ones, and should theoretically work just file on any ROM as well.

Note that this procedure entails ESN cloning, which may be illegal in your region. Please make sure you have an understanding of all applicable laws before proceeding. The author of this guide cannot be held responsible for any legal infractions that may ensue. Finally, you are solely responsible for any consequences of your actions as a result of following this guide. While I believe it to be quite safe, I cannot guarantee you that this process will not brick your devices or start a zombie apocalypse.

Requirements and Setup

We will need the following before starting:

  • A currently active Verizon Wireless Android phone and plan. Please beware that not all phones will work; many phones do not allow us to extract all the information we need for this process. For instance, my Motorola Droid 3 did not work, and I had to purchase and activate an HTC Incredible for this. This phone will be our donor phone.

  • A rooted Samsung Nexus 4G.

  • A computer running a recent version of Windows, with at least 2 USB ports.

  • 2 MicroUSB cables.

In addition, we need the following software:

  • USB diagnostic drivers for your donor phone. These vary by manufacturer and their configuration may be quite complex; search online for how to set up yours. For HTC phones, download and install HTC Sync from HTC’s website (archived here).

  • USB diagnostic drivers for the Nexus S 4G. The easiest way is to download and install PdaNet (archived here). We don’t actually need the functionality of PdaNet, but it happens to bundle the drivers we need. Select Samsung when prompted for the manufacturer of our phone.

  • QPST and QXDM (archived here). These are internal Qualcomm tools; we will use them to clone the MEID of the donor phone.

  • DFS (archived here). This is a third-party tool for working with CDMA phones. The demo version will suffice.

Step 1: Preparation

First, make sure the donor phone is fully activated and that calls, text and data are all fully functional.

Now, find the HEX MEID of the donor phone. On the HTC Incredible, this can be found under Settings → About Phone → Phone Identity. On most Android phones, this is under Settings → About Phone → Status. It is also usually printed on a label underneath the phone battery. This should be 14 digits and usually begins with A00000.

Now put the donor phone in airplane mode.

We will next connect both phones to the computer in diagnostic mode.

  • On the Nexus S 4G, dial *#*#8778#*#*, then select MODEM under USB. Connect the Nexus S 4G to the computer. If drivers were installed correctly, you should now see a SAMSUNG Mobile Modem Diagnostic Serial Port (WDM) under Ports (COM & LPT) in Windows’s Device Manager. Note down the port number of the Nexus S 4G; we will assume COM4 for convenience.

  • On the HTC Incredible, dial ##3424#, hit Call, connect it to the computer, then follow the instructions in this thread with Device Manager. If everything goes well you should now see a HTC Diagnostic Interface under Ports (COM & LPT) in Device Manager. Note down the port number of the HTC Incredible; we will assume COM11 for convenience.

After confirming that both phones show up under Ports (COM & LPT), run QPST Configuration and hit Add New Port…. You should see both phones on the left. Select one and hit OK. Then repeat for the other phone.

Step 2: Cloning

We will now clone the MEID of the donor phone onto the target phone. Open QXDM Professional. Select Options → Communications…. For Target Port, select the port corresponding to the Nexus S 4G, say COM4, and hit OK. Then in the text box labeled Command, type the following:

Password 01F2030F5F678FF9

Hit Enter. You should see Password Result = Correct at the bottom of the Command Output window.

Now in the Command text box, type the following, replacing <MEID> by the hex MEID of the donor phone:

RequestNVItemWrite meid 0x<MEID>

Hit Enter. You should see it repeat back the MEID to you. Type RequestNVItemRead meid followed by Enter to make sure the write happened.

Close QXDM Professional and reboot the Nexus S 4G. Rebooting the target phone after writing a new MEID is essential! If prompted, do NOT go through the activation process on the Nexus S 4G. Just select Skip when possible.

Step 3: Reset SPC

In order to program the Nexus S 4G, we need to obtain or reset the SPC code (also known as the MSL) of the phones. If you already know the SPC code for your phones and would rather not reset the SPC code, you can skip this step.

Open QXDM Professional again. Go through Options → Communications and make sure the Target Port is still COM4. Hit OK. In the command text box, type the following two lines, each followed by an Enter:

Password 01F2030F5F678FF9
RequestNVItemWrite sec_code 000000

If this is successful, the SPC on the Nexus S 4G will have been reset to 000000. Repeat for the donor phone (in our example, on COM11).

Close QXDM Professional after done. All QPST / QXDM programs must be closed before proceeding, or DFS will complain.

Step 4: Programming CDMA Chipset

Launch DFS. Click Ports on the top left. In the dialog box, double-click on each port representing our phones, in our case, COM4 and COM11, and close the dialog box. Now make sure the SPC text box above reads 000000 (or if you chose not to reset the SPC, whatever your SPC is). Type 01F2030F5F678FF9 into the Pwd text box.

Step 4.a: Basic CDMA settings

Let’s now copy basic CDMA settings from the donor phone to the target phone. In the drop-down menu next to Ports, select the donor phone (COM11). Switch to the Programming tab and the NAM tab under it. Click the SPC and Pwd buttons to send the SPC and NV password to the phone to unlock it. Then hit Read beneath Network identification. The text fields below should be populated by information from the donor phone. Now select COM4 in the drop-down menu next to Ports, click SPC and Pwd, then click Write beneath Network identification.

Step 4.b: PRL

Now, let’s copy over the PRL (or Preferred Roaming List - essentially a database of tower locations). Again, select COM11 under Ports and click Read under PRL towards the right. Then select COM4 under Ports and click Write under PRL.

Step 4.c: 2G Data settings

Now go to the next tab, Data. Select COM11 and hit Read. Switch to COM4 and hit Write. If you observe errors in the log window in the bottom, and / or no information shows up in the Pwd boxes under PPP and HDR AN Long, your donor phone doesn’t support the extraction of some data passwords, and you will have to try a different donor phone.

Step 4.c: 3G Data settings

We do the same with Mobile IP. Select COM11 and hit Read. Switch to COM4 and hit Write. Then hit Write current profile settings. Again, if you encounter errors, or no information shows up under AAA Shared secret and HA Shared Secret, you will have to try a different donor phone.

Finally, we need to copy over a file containing a secret key used to establish a 3G connection. Go to the EFS tab towards the top. Select COM11 and hit Read EFS. Click the "+" sign of the root folder on the left and then the "+" sign next to the DMU folder and you should see a file called 10.key. Right-click on this file and hit Save…. Save this file somewhere on your hard drive. Now switch to COM4 and hit Read EFS again. Select the root folder on the left, type /DMU under Path (57 max) on the right and hit Add Item to create a folder named DMU on the Nexus S 4G. Then select DMU on the left, click on the check box named I want to add file from PC, and select the 10.key file we just saved. Then hit Add Item to upload the 10.key file to /DMU/10.key.

The programming is now done. Close DFS, select OK and OK again when prompted whether to send a mode reset to the phone. The Nexus S 4G will now reboot.

Step 5: Programming Android

We will now modify some Android system files to Verizon settings. This step will need to be repeated whenever you flash a new ROM on the Nexus S 4G in the future.

1-click programming app

I have built a 1-click Android application that automates this step; you can find it here. It requires root as it will need to modify system files. Install the application and click the button in the middle. When the process finishes, you will be prompted to reboot. Simply reboot your phone and you’re all set; you may uninstall the app, or you could keep it around for when you update to a new ROM.

If the 1-click app does not work for you, or if you want more control over what it does, follow the steps below. You do not need to do any of the following if you already used the app above.

Step 5.a: Fix voice calling

This is required to enable calls on Verizon. On the Nexus S 4G, connect to WiFi and install ES File Explorer from the Play Store. Then go to Menu → Settings and check Up to Root, Root Explorer, and Mount File System. Allow it to use superuser privileges if asked. Then navigate to /system and open build.prop with ES Note Editor. Find the following two lines:

ro.cdma.home.operator.numeric=310120
ro.cdma.home.operator.alpha=Sprint

Change them to read:

ro.cdma.home.operator.numeric=310004
ro.cdma.home.operator.alpha=Verizon

Then add a line below (or anywhere in the file) that says:

ro.cdma.homesystem=64,65,76,77,78,79,80,81,82,83

Hit Menu → Save and then Back to exit.

Step 5.b: Fix roaming

This step tells Android to treat the Verizon network as the home network. It sets the roaming status to "Not roaming", removes the triangular roaming indicator in the status bar and changes the displayed name of the current network to "Verizon Wireless".

Now navigate to /system/framework and copy framework-res.apk onto your computer - you can use adb or email it to yourself or whatever (you have to copy it to the internal SD card before you can email it though). Once on the computer, rename it framework-res.zip (yes, an APK is just a zip archive) and use your favorite program to replace the file res/xml/eri.xml inside it with the one at this link. Now rename it back to framework-res.apk, copy it back onto the Nexus S 4G (using adb or whatever) and use ES File Explorer to overwrite the existing /system/framework/framework-res.apk with it.

Step 5.c: Add Verizon APNs

This last step sets up Verizon APNs that enable web and MMS. Use ES File Explorer or any other tool to replace /system/etc/apns-conf.xml with the one at this link.

Step 6: Reboot & PROFIT

The Nexus S 4G should now be a fully functional Verizon Wireless phone. If you are curious about what each of the steps means, you’re welcome to check out my previous article Carrier Programming on CDMA Android Phones for a high-level view of how CDMA phones are programmed.

While your mileage may vary, I hope the above information have been of help. Since I no longer use Verizon Wireless and no longer own any of the phones mentioned above, please take all this information with a grain of salt.

Good luck, and happy hacking!

Setting Up DKIM And SRS In Postfix

In my previous post Custom Domain E-mails With Postfix And Gmail: The Missing Tutorial, we set up a Postfix mail server on a custom domain that integrates seamlessly with Gmail.

However, the tutorial skipped two important security standards that will help prevent e-mails routed through our server from being marked as spam: DKIM and SRS. This article will show you how to add support for DKIM and SRS to a Postfix server.

Similar to the previous tutorial, we will assume a Ubuntu server in our examples.

Step 1: DKIM

DKIM, short for DomainKeys Identified Mail, is a mechanism for

  1. A sender e-mail program to sign an outgoing e-mail message, and

  2. A recipient e-mail program to verify said signature.

More concretely, let’s say Gmail receives an e-mail message from myserver.example.com. DKIM allows Gmail to verify that the e-mail was indeed sent by the designated e-mail server program on myserver.example.com, and not by, say, a virus running on myserver.example.com or a malicious user who happens to have access to myserver.example.com.

We will use OpenDKIM for this tutorial. To install OpenDKIM on Ubuntu:

$ sudo apt-get install opendkim opendkim-tools

/etc/opendkim.conf

Edit /etc/opendkim.conf to match the following:

# OpenDKIM config.

# Log to syslog
Syslog                  yes
SyslogSuccess           yes
LogWhy                  yes
# Required to use local socket with MTAs that access the socket as a non-
# privileged user (e.g. Postfix)
UMask                   002

Mode                    sv
PidFile                 /var/run/opendkim/opendkim.pid
UserID                  opendkim:opendkim
Socket                  inet:12301@localhost

Canonicalization        relaxed/simple
SignatureAlgorithm      rsa-sha256

# Sign for example.com with key in /etc/opendkim.d/mail.private using
# selector 'mail' (e.g. mail._domainkey.example.com)
Domain                  example.com
KeyFile                 /etc/opendkim.d/mail.private
Selector                mail

ExternalIgnoreList      refile:/etc/opendkim.d/TrustedHosts
InternalHosts           refile:/etc/opendkim.d/TrustedHosts

You can check out a detailed explanation for the meaning of each option with man opendkim.conf.

/etc/opendkim.d/TrustedHosts

Create the directory /etc/opendkim.d and put the following in /etc/opendkim.d/TrustedHosts. This instructs the OpenDKIM server to sign e-mails delivered by any server matching these expressions (such as myserver_2.example.com).

127.0.0.1
::1
localhost
192.168.0.1/24

*.example.com

DKIM keys

Now, let’s generate our DKIM signing keys:

$ cd /etc/opendkim.d
$ sudo opendkim-genkey -s mail -d example.com

This will produce two files in /etc/opendkim.d: our private key, mail.private, and our public key, mail.txt.

We should make sure only OpenDKIM can read the private key, so that a malicious program or user on the same server won’t be able to forge our signature:

$ chmod 600 mail.private
$ chown opendkim:opendkim mail.private

DKIM public key DNS record

The next step is to publish our public key through DNS, so that any recipient e-mail program can verify our signature. If we look at our public key mail.txt generated in the previous step, it should look like something like this:

mail._domainkey IN      TXT     ( "v=DKIM1; k=rsa; "
          "p=<alphabetical soup>" )  ; ----- DKIM key mail for example.com

Create the following TXT DNS record for example.com (how to update DNS records will depend on the DNS hosting provider):

  • Name: mail._domainkey.example.com

  • Value: v=DKIM1; k=rsa; p=<alphabetical soup>

where <alphabetical soup> is the public key found in mail.txt after p=. Note that DNS records will take a while (depending on our provider, up to a day) to propagate.

OpenDKIM server

We can now start the OpenDKIM server with

$ sudo /etc/init.d/opendkim start

Postfix

The last step is to tell Postfix to use OpenDKIM to sign outgoing e-mail messages. Add the following to /etc/postfix/main.cf:

# Milter settings.
milter_protocol = 2
milter_default_action = accept
# OpenDKIM runs on port 12301.
smtpd_milters = inet:localhost:12301
non_smtpd_milters = inet:localhost:12301

If you already have other milters configured (such as SpamAssassin), simply add inet:localhost:12301 to your existing smtpd_milters and non_smtpd_milters lines, prefixed by a comma.

Let’s now restart Postfix with the new configuration:

$ sudo postfix reload

…and we’re done!

Step 2: SRS

SRS, short for Sender Rewriting Scheme, is a standard for including forwarding / relay information in a forwarded / relayed e-mail message.

For example, suppose alice@hotmail.com sends an e-mail to john@example.com, and our Postfix server on myserver.example.com forwards this e-mail to john123@gmail.com. SRS allows our Postfix server on myserver.example.com to attach a virtual sticky note on the e-mail message explaining this situation to Gmail. Otherwise, Gmail might become suspicious of why myserver.example.com is producing messages that purport to come from hotmail.com, which spammers and phishers are wont to do.

We will use PostSRSd to implement SRS in our Postfix server. It works out of the box with Postfix and is a breeze to set up, but unfortunately is not included in the official Ubuntu / Debian package repositories.

OpenSRSd

Let’s build and install OpenSRSd from source:

# Dependencies.
$ sudo apt-get install unzip cmake

# Download and extract source code from GitHub.
$ cd /tmp
$ curl -L -o postsrsd.zip \
    https://github.com/roehling/postsrsd/archive/master.zip
$ unzip postsrsd.zip

# Build and install.
$ cd postsrsd-master
$ mkdir build
$ cd build
$ cmake -DCMAKE_INSTALL_PREFIX=/usr ../
$ make
$ sudo make install

The default config provided by PostSRSd (/etc/default/postsrsd) will pretty much work out of the box for our case.

The install script will also conveniently install an Upstart script for PostSRSd. Let’s start it now:

$ sudo service postsrsd start

Postfix

Finally, we configure Postfix to use PostSRSd. Add the following to /etc/postfix/main.cf:

# PostSRSd settings.
sender_canonical_maps = tcp:localhost:10001
sender_canonical_classes = envelope_sender
recipient_canonical_maps = tcp:localhost:10002
recipient_canonical_classes= envelope_recipient,header_recipient

And restart Postfix with the new configuration:

$ sudo postfix reload

That’s it!

Custom Domain E-mails With Postfix and Gmail: The Missing Tutorial

Custom domain e-mail addresses, like john@johnscompany.com, are cool and professional-looking. Would you want to e-mail potential clients as john123@gmail.com?

On the other hand, Gmail has great reliability, speed, spam filtration, and app support. Wouldn’t it be great if we could send and receive e-mail as john@johnscompany.com, but right from the comfort of our Gmail account?

Today, if we want to use Gmail to send and receive e-mails on a custom domain, we have a couple of options:

  • Google Apps for Work is Google’s own offering. It’s simple to set up and manage, but it’s pretty pricey at $5/mo per user, especially if we don’t need the bundled collaborative features like Google Docs and Calendar.

  • Third-party services could get the job done at a lower price point; for example, Pobox offers plans starting at $20/yr. The downside is that we will have to trust another organization with our e-mail.

  • Set up our own e-mail forwarding server. It’s fairly easy (if we follow this guide :) and if we already have shared / dedicated hosting or a VPS, this is basically free and gives us total control over our e-mail.

If you want to host custom domain e-mails in Gmail without paying for Google Apps or a third-party service, this tutorial is for you. It will show you how to quickly set up you own e-mail forwarding / relay server, and how to integrate it seamlessly with Gmail.

Prerequisites

For this tutorial, we’ll assume we have access to:

  • VPS or shared / dedicated hosting, with a dedicated IP address and capable of listening on ports 25 and 587. We will use a Ubuntu server for the examples.

  • DNS records for our domain. How DNS records are manipulated depends on the DNS hosting provider (or DNS server configuration).

We’ll assume

  • We have a domain example.com;

  • We have a server myserver.example.com;

  • We want to use the Gmail account john123@gmail.com to manage john@example.com.

Step 1: DNS Setup

The first step in setting up our e-mail forwarding server is to add MX, PTR and SPF DNS records for our server. This is really important as many e-mail providers, in order to prevent spam, will refuse to talk to mail servers without proper MX, PTR and SPF records set up. How to update DNS records will depend on the DNS hosting provider.

The following is what we need:

  • An MX record for example.com, pointing to myserver.example.com.

    This tells the world that e-mails to <whatever>@example.com should be delivered to the server myserver.example.com.

  • An A record for myserver.example.com, pointing to the IP address of our server.

  • A PTR (reverse DNS) record mapping the IP address of our server to myserver.example.com.

    This allows Gmail to verify the legitimacy of our server via its IP when Gmail receives a forwarded e-mail from it.

  • A TXT record, with key example.com and value v=spf1 mx ~all.

    This is an SPF record; it tells Gmail that the servers specified in the MX records of example.com, in this case only myserver.example.com, are allowed to send e-mails purporting to be from <whatever>@example.com. All other servers attempting to do the same will be rejected. This should be a sane default value, but feel free to custom it as you like.

DNS records will take a while (depending on our provider, up to a day) to propagate. Until they do, e-mails forwarded by our new e-mail server may get marked as spam or rejected outright.

Step 2: Receiving E-mail

We’ll first set up an e-mail forwarding server which will let we receive e-mails sent to our domains.

We will use Postfix as the e-mail server. Let’s start by installing the necessary packages. We will assume a Ubuntu server in the examples below; if you’re using a different distribution, please consult your distribution’s documentation for the right commands.

$ sudo DEBIAN_FRONTEND=noninteractive apt-get install postfix

In the above command, we skip the debconf configuration UI with DEBIAN_FRONTEND=noninteractive as we will edit the configuration files directly.

/etc/postfix/main.cf

Open up /etc/postfix/main.cf in your favorite editor. This file will come pre-populated with a bunch of config options and comments. Replace the contents of the file with the following (you can comment out the original contents for reference):

# /etc/postfix/main.cf

# Host and site name.
myhostname = myserver.example.com
mydomain = example.com
myorigin = example.com

# Virtual aliases.
virtual_alias_domains = example.com
virtual_alias_maps = hash:/etc/postfix/virtual

The first few lines are pretty straightforward; they tell Postfix how to identify itself to the world. The last two lines tell Postfix to forward e-mails sent to <whatever>@example.com to another e-mail provider (Gmail), and that the forwarding is configured in the database file /etc/postfix/virtual.

/etc/postfix/virtual

Let’s now open up /etc/postfix/virtual and fill in our forwarding configuration:

# /etc/postfix/virtual

# Forwarding mapping, one from-to address pair per line. The format is:
#     <forward-from-addr> <whitespace> <forward-to-addr>
john@example.com        john123@gmail.com

We can add as many forwarding rules as you want, one on each line. We can use any number of tabs / whitespaces between the forward-from and forward-to addresses.

Update lookup table

It turns out that Postfix doesn’t actually read /etc/postfix/virtual (surprise!); instead, what it reads is a lookup table generated from it. So, let’s generate the lookup table from our /etc/postfix/virtual:

$ sudo postmap /etc/postfix/virtual

Note: we must re-run this command every time we modify /etc/postfix/virtual!

(Re)start Postfix

It’s now time to (re)start Postfix with our new configuration:

$ sudo postfix start
$ sudo postfix reload

Testing

We should test our brand-new Postfix server by sending an e-mail to our forward-from address. You might want to check out these tutorials for testing directly with the telnet command.

If you received the e-mail in your Gmail inbox, congratulations! Otherwise, check the Postfix logs at /var/log/mail.log (or with journalctl -u postfix if using Systemd) for errors. The most likely cause of issues is that DNS records have not propagated yet, in which case we’re likely to see a "rate limited" or "rejected for spam" type error message.

If we only want to receive but don’t care about sending e-mails as john@example.com, then this is all we need.

Otherwise, let’s move on to configuring Postfix to support sending e-mails from Gmail.

Step 3: Sending E-mail

Gmail requires a relay server (a server that will send e-mails to their destination on behalf it) to speak TLS, which protects the communication between Gmail and the relay server.

We will use Cyrus SASL for this task. To install:

$ sudo apt-get install sasl2-bin libsasl2-modules

User name & password

Open relays are a terrible idea. We want to protect our e-mail server with a user name and password so that it will let Gmail send e-mails through it, but block spammers and other evil actors.

Cyrus SASL supports several backends for storing user names and passwords, including MySQL and PAM. However, we will go with the simplest backend — a plain database file.

Let’s create the user name / password database file in the default location /etc/sasldb2, with a single user named smtp (we can change the user name to anything we want):

$ sudo saslpasswd2 -c -u example.com smtp

Verify with:

$ sudo sasldblistusers2

Now, we make sure only Postfix can read this file:

$ sudo chmod 400 /etc/sasldb2
$ sudo chown postfix /etc/sasldb2

Lastly, we tell Cyrus SASL to use the file-based database to authenticate. Create the file /etc/postfix/sasl/smtpd.conf:

pwcheck_method: auxprop
auxprop_plugin: sasldb
mech_list: PLAIN LOGIN CRAM-MD5 DIGEST-MD5 NTLM
log_level: 7

SSL Certificate

We need an SSL certificate to enable TLS. While a proper SSL certificate signed by a Certificate Authority (CA) can be pricey, it turns out that a simple self-signed certificate suffices and works just fine.

So, let’s generate one.

  1. Generate an RSA private/public key pair. Note that you MUST supply a password to this command; the password will be removed in step 3.

    $ openssl genrsa -des3 -out example.key 1024
    
  2. Generate a Certificate Signing Request (CSR). Make sure to enter myserver.example.com when prompted for the "Common Name".

    $ openssl req -new -key example.key -out example.csr
    
  3. Remove RSA private/public key password.

    $ mv example.key example.key.orig
    $ openssl rsa -in example.key.orig -out example.key
    
  4. Generate a self-signed certificate. In the example, the generated certificate will be valid for 10 years.

    $ openssl x509 -req \
        -days 3650 -in example.csr -signkey example.key -out example.crt
    
  5. Create a PEM file.

    $ cat example.crt example.key > example.pem
    
  6. Move and protect the PEM file.

    $ sudo mv example.pem /etc/postfix/example.pem
    $ sudo chmod 400 /etc/postfix/example.pem
    $ sudo chown postfix /etc/postfix/example.pem
    

Note: Make sure to protect the generated private key and certificate with care!

Relay server

The final step is to configure Postfix to enable relaying of e-mail on behalf of Gmail.

Let’s open up /etc/postfix/master.cf. This file should already contain a bunch of config options, some of which are commented out. Uncomment the lines starting with submission and edit them to match the following:

submission inet n       -       n       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=may
  -o smtpd_tls_cert_file=/etc/postfix/example.pem
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

A Postfix restart is due after all these changes:

$ sudo postfix reload

If all went well, you should see Postfix serving a relay server, protected by our user name and password in /etc/sasldb2, on port 587.

Step 4: Configure Gmail

Finally, let’s log in to Gmail and tell it about our brand new server. (Note that the new Inbox UI does not yet support all the settings available in the legacy interface at the time of writing, so these instructions apply to the legacy interface.)

Let’s go to Settings > Accounts and Import and click Add another email address you own.

In the dialog that pops up, fill in our target e-mail address and click Next Step.
In the next dialog, enter the address of our e-mail server (myserver.example.com), and the user name and password we set up using saslpasswd2 above. Make sure the user name is suffixed with our domain name (so smtp@example.com rather than just smtp). Check that the correct port (587) and the correct security protocol (TLS) are selected, then click Add Account.
If all goes well, we should see a dialog like the following:
And we should receive an e-mail, forwarded by our mail server set up in the first part, to our john123@gmail.com address. Click on the link inside the e-mail and we’re done!

Now, in any e-mail compose / reply window, you should be able to select john@example.com in the drop-down list next to "From". Congrats!

If, on the other hand, you see an error like the following:

You should check the Postfix logs at /var/log/mail.log (or with journalctl -u postfix if using Systemd) for any errors.

[Optional] Step 5: Setup DKIM and SRS

DKIM, short for DomainKeys Identified Mail, is e-mail validation system for preventing e-mail spoofing. Google recommends setting up DKIM; if we don’t, the e-mail messages that we forward to Gmail have a higher probability of getting marked as spam.

SRS, short for Sender Rewriting Scheme, is a relatively new standard that e-mail forwarding servers are recommended to adopt, and it also helps reduce the likelihood of our forwarded e-mail messages getting marked as spam.

Thus, while technically optional, it’s a good idea to configure our Postfix server to support both these standards. See my follow-up tutorial on Setting Up DKIM And SRS In Postfix for a detailed step-by-step guide.

Conclusion

In this post, we have set up a Postfix e-mail server that accomplishes the following:

  • E-mails sent to john@example.com is received by Postfix on myserver.example.com port 25, and forwarded to john123@gmail.com;

  • Gmail will let you select john@example.com as a "from" address when composing / replying to an e-mail, and will relay such e-mails through Postfix on myserver.example.com port 587 using a configured user name / password combination.

I hope you found this guide useful, and if you have any thoughts / questions, you’re more than welcome to leave a comment below. Cheers!

How To Add Custom Build Steps and Commands To setup.py

A setup.py script using distutils / setuptools is the standard way to package Python code. Often, however, we need to perform custom actions for code generation, running tests, profiling, or building documentation, etc., and we’d like to integrate these actions into setup.py. In other words, we’d like to add custom steps to setup.py build or setup.py install, or add a new command altogether to setup.py.

Let’s see how this is done.

Adding Custom setup.py Commands and Options

Let’s implement a custom command that runs Pylint on all Python files in our project. The high level idea is:

  1. Implement command as a subclass of distutils.cmd.Command;

  2. Add the newly defined command class to the cmdclass argument to setup().

To see this in action, let’s add the following to our setup.py:

import distutils.cmd
import distutils.log
import setuptools
import subprocess


class PylintCommand(distutils.cmd.Command):
  """A custom command to run Pylint on all Python source files."""

  description = 'run Pylint on Python source files'
  user_options = [
      # The format is (long option, short option, description).
      ('pylint-rcfile=', None, 'path to Pylint config file'),
  ]

  def initialize_options(self):
    """Set default values for options."""
    # Each user option must be listed here with their default value.
    self.pylint_rcfile = ''

  def finalize_options(self):
    """Post-process options."""
    if self.pylint_rcfile:
      assert os.path.exists(self.pylint_rcfile), (
          'Pylint config file %s does not exist.' % self.pylint_rcfile)

  def run(self):
    """Run command."""
    command = ['/usr/bin/pylint']
    if self.pylint_rcfile:
      command.append('--rcfile=%s' % self.pylint_rcfile)
    command.append(os.getcwd())
    self.announce(
        'Running command: %s' % str(command),
        level=distutils.log.INFO)
    subprocess.check_call(command)


setuptools.setup(
    cmdclass={
        'pylint': PylintCommand,
    },
    # Usual setup() args.
    # ...
)

Now, running python setup.py --help-commands will show:

Standard commands:
  ...
Extra commands:
  pylint: run Pylint on Python source files
  ...

We can now run the command we just defined with:

$ python setup.py pylint

…or with a custom option:

$ python setup.py pylint --pylint-rcfile=.pylintrc

To learn more, you can check out documentation on inheriting from distutils.cmd.Command as well as the source code of some built-in commands, such as build_py.

Adding Custom Steps to setup.py build

Let’s say we are really paranoid about code style and we’d like to run Pylint as part of setup.py build. We can do this in the following manner:

  1. Create a subclass of setuptools.command.build_py.build_py (or distutils.command.build_py.build_py if using distutils) that invokes our new Pylint command;

  2. Add the newly defined command class to the cmdclass argument to setup().

For example, we can implement the following in our setup.py:

import setuptools.command.build_py


class BuildPyCommand(setuptools.command.build_py.build_py):
  """Custom build command."""

  def run(self):
    self.run_command('pylint')
    setuptools.command.build_py.build_py.run(self)


setuptools.setup(
    cmdclass={
        'pylint': PylintCommand,
        'build_py': BuildPyCommand,
    },
    # Usual setup() args.
    # ...
)

For more examples, I encourage you to check out the setuptools source code.