How X Window Managers Work, And How To Write One (Part III)

In Part II of this series, we discussed X libraries and implementation choices, and examined the basic structure of a window manager. In Part III, we will start interacting with client windows and the user through events. We will review the fundamentals of window manager implementation, using the implementation in our example non-compositing reparenting window manager, basic_wm, for reference.

Step 4: Interaction with Application Windows

Following the steps in Part II of this series, we now have a basic skeleton for our window manager. Our next step is to start talking to clients and the user via events.

The interaction between clients, X, and the window manager is fairly complex. To facilitate our discussion, I’ve created a diagram that illustrates the flow of events throughout the lifetime of a client window, and how a window manager might respond to each of them. We’ll be referring to this cheat sheet for window manager event handling throughout this series. You can click through for the full-sized diagram.

In general, a window manager must handle two kinds of actions: those initiated by client applications (such as creating new windows), and those initiated by users (such as moving or minimizing windows). In this diagram, actions initiated by client applications are shown in the yellow box on the left hand side, and actions initiated by users are shown in blue on the right hand side. A window manager communicates with client applications via events, which are represented as parallelograms in red.

You may have noticed that some of the events in this diagram have the suffix Request, while others have the suffix Notify. This distinction is crucial to our discussion.

Recalling our discussion in Part I on substructure redirection, when a client application wants to do something with a window (such as moving, resizing, showing, or hiding), its request is redirected to the window manager, which can grant, modify, or deny the request. Such requests are delivered to a window manager as events with the Request suffix. It is important to understand that when a window manager receives such an event, the action it represents has not actually occurred, and it is the responsibility of the window manager to decide what to do with it. If the window manager does nothing, the request is implicitly denied.

On the other hand, events with the Notify suffix represent actions that have already been executed by the X server. The window manager can respond to such events, but of course cannot change the fact that they have already happened.

With that in mind, let’s dive into the implementation by looking at how our example window manager will handle the life cycle of a client window from creation to destruction.

Creating a Window

When an X client application creates a top-level window (XCreateWindow()), our window manager will receive a CreateNotify event. However, a newly created window is always invisible, so there’s nothing for our window manager to do. In window_manager.cpp:

void WindowManager::Run() {
  ...
  // 2. Main event loop.
  for (;;) {
    // 1. Get next event.
    ...
    // 2. Dispatch event.
    switch (e.type) {
      ...
      case CreateNotify:
        OnCreateNotify(e.xcreatewindow);
        break;
      ...
    }
  }
}

void WindowManager::OnCreateNotify(const XCreateWindowEvent& e) {}

Configuring a Newly Created Window

At this stage, the application can configure the window to set its initial size, position, or other attributes. To do so, the application would invoke XConfigureWindow(), which would send a ConfigureRequest event to the window manager. However, since the window is still invisible, the window manager doesn’t need to care and can grant such requests without modification by invoking XConfigureWindow() itself with the same parameters.

void WindowManager::Run() {
      ...
      case ConfigureRequest:
        OnConfigureRequest(e.xconfigurerequest);
        break;
      ...
}

void WindowManager::OnConfigureRequest(const XConfigureRequestEvent& e) {
  XWindowChanges changes;
  // Copy fields from e to changes.
  changes.x = e.x;
  changes.y = e.y;
  changes.width = e.width;
  changes.height = e.height;
  changes.border_width = e.border_width;
  changes.sibling = e.above;
  changes.stack_mode = e.detail;
  // Grant request by calling XConfigureWindow().
  XConfigureWindow(display_, e.window, e.value_mask, &changes);
  LOG(INFO) << "Resize " << e.window << " to " << Size<int>(e.width, e.height);
}

Mapping a Window

To make the window finally visible on screen, the client application will call XMapWindow() to map it. This sends a MapRequest event to the window manager. As noted earlier, at this point, the window is still not yet visible, as it’s up to the window manager to actually make it so. This is probably the most important event in our discussion, as this is where a window manager would usually start really managing a window.

A reparenting window manager would typically respond to a MapRequest for a client application window w with the following actions:

  1. Create a frame window f, perhaps with borders and window decoration (e.g. title, minimize / maximize / close buttons).

  2. Register for substructure redirect on f with XSelectInput(). Recall that substructure redirect only applies to direct child windows, so after reparenting, the substructure redirect previously registered on the root window would no longer apply to w, hence this step.

  3. Make w a child of f with XReparentWindow().

  4. Render f and w with XMapWindow().

  5. Register for mouse or keyboard shortcuts on w and/or f.

The example implementation in basic_wm will create a very simple frame window that has the same size as the client window, but with a 3px red border:

void WindowManager::Run() {
      ...
      case MapRequest:
        OnMapRequest(e.xmaprequest);
        break;
      ...
}

void WindowManager::OnMapRequest(const XMapRequestEvent& e) {
  // 1. Frame or re-frame window.
  Frame(e.window);
  // 2. Actually map window.
  XMapWindow(display_, e.window);
}

void WindowManager::Frame(Window w) {
  // Visual properties of the frame to create.
  const unsigned int BORDER_WIDTH = 3;
  const unsigned long BORDER_COLOR = 0xff0000;
  const unsigned long BG_COLOR = 0x0000ff;

  // 1. Retrieve attributes of window to frame.
  XWindowAttributes x_window_attrs;
  CHECK(XGetWindowAttributes(display_, w, &x_window_attrs));

  // 2. TODO - see Framing Existing Top-Level Windows section below.

  // 3. Create frame.
  const Window frame = XCreateSimpleWindow(
      display_,
      root_,
      x_window_attrs.x,
      x_window_attrs.y,
      x_window_attrs.width,
      x_window_attrs.height,
      BORDER_WIDTH,
      BORDER_COLOR,
      BG_COLOR);
  // 3. Select events on frame.
  XSelectInput(
      display_,
      frame,
      SubstructureRedirectMask | SubstructureNotifyMask);
  // 4. Add client to save set, so that it will be restored and kept alive if we
  // crash.
  XAddToSaveSet(display_, w);
  // 5. Reparent client window.
  XReparentWindow(
      display_,
      w,
      frame,
      0, 0);  // Offset of client window within frame.
  // 6. Map frame.
  XMapWindow(display_, frame);
  // 7. Save frame handle.
  clients_[w] = frame;
  // 8. Grab events for window management actions on client window.
  //   a. Move windows with alt + left button.
  XGrabButton(...);
  //   b. Resize windows with alt + right button.
  XGrabButton(...);
  //   c. Kill windows with alt + f4.
  XGrabKey(...);
  //   d. Switch windows with alt + tab.
  XGrabKey(...);

  LOG(INFO) << "Framed window " << w << " [" << frame << "]";
}

The outline of the code should be fairly clear following our discussion. A few additional points to note:

  • Regarding the save-set and XAddToSaveSet():

    The save-set is a list of windows, usually maintained by the window manager, but including only windows created by other clients. If the window manager dies, all windows listed in the save-set will be reparented back to their closest living ancestor if they were reparented in the first place and mapped if the window manager has unmapped them so that it could map an icon.

    The save-set is necessary because the window manager might not exit normally. The user might kill it with CTRL-C if it is running in the foreground, or more likely, the user might get the process number and kill it. Actually, the actions of the save-set are performed even if the window manager exits normally, so less code is needed since the save-set does the cleaning up.

    Window managers almost always place in the save-set all the windows they reparent or iconify, using XAddToSaveSet().

    Windows are automatically removed from the save-set when they are destroyed.

    — Xlib Programming Manual §16.4
  • When our window manager creates a frame window (step 2 in the example code), it will also trigger a CreateNotify event for the frame window. It will ignore it just like it ignores other CreateNotify events as discussed earlier.

  • When our window manager calls XReparentWindow() in step 5 in the example code, it will trigger a ReparentNotify event, which it will ignore:

    void WindowManager::Run() {
          ...
          case ReparentNotify:
            OnReparentNotify(e.xreparent);
            break;
          ...
    }
    
    void WindowManager::OnReparentNotify(const XReparentEvent& e) {}
    
  • When our window manager calls XMapWindow() to map the frame window (step 6 in the example code), the X server knows that the action originates from the current window manager, and will execute it directly instead of redirecting it back as a MapRequest event. Our window manager will later receive a MapNotify event, which it can ignore:

    void WindowManager::Run() {
          ...
          case MapNotify:
            OnMapNotify(e.xmap);
            break;
          ...
    }
    
    void WindowManager::OnMapNotify(const XMapEvent& e) {}
    

Configuring a Mapped Window

A client application can configure a window that is currently visible, again with the XConfigureWindow() function. For example, an application may want to resize a window to better accomodate its contents. When a reparenting window manager receives the resulting ConfigureRequest and decides to grant the request, it additionally needs to resize / reposition the corresponding frame window and any window decorations.

void WindowManager::OnConfigureRequest(const XConfigureRequestEvent& e) {
  XWindowChanges changes;
  // Copy fields from e to changes.
  ...
  if (clients_.count(e.window)) {
    const Window frame = clients_[e.window];
    XConfigureWindow(display_, frame, e.value_mask, &changes);
    LOG(INFO) << "Resize [" << frame << "] to " << Size<int>(e.width, e.height);
  }
  // Grant request by calling XConfigureWindow().
  ...
}

When our window manager re-configures the frame window with the XConfigureWindow() call above, the X server knows that the action originates from the current window manager, and will execute it directly instead of redirecting it back as a ConfigureRequest event. Our window manager will then receive a ConfigureNotify event, which it will ignore:

void WindowManager::Run() {
      ...
      case ConfigureNotify:
        OnConfigureNotify(e.xconfigure);
        break;
      ...
}

void WindowManager::OnConfigureNotify(const XConfigureEvent& e) {}

Unmapping a Window

When a client application unmaps (i.e. hides) a window with XUnmapWindow(), for example in response to the user exiting or minimizing the application, the window manager will receive a UnmapNotify event. Unlike the MapRequest event, the UnmapNotify event is delivered to the window manager after the fact, and the window manager can only respond to it, not intercept it.

A reparenting window manager will typically want to reverse the actions it performed in response to MapRequest. In other words, it would reparent the client window back to the root window, and destroy the corresponding frame window.

void WindowManager::Run() {
      ...
      case UnmapNotify:
        OnUnmapNotify(e.xunmap);
        break;
      ...
}

void WindowManager::OnUnmapNotify(const XUnmapEvent& e) {
  // If the window is a client window we manage, unframe it upon UnmapNotify. We
  // need the check because we will receive an UnmapNotify event for a frame
  // window we just destroyed ourselves.
  if (!clients_.count(e.window)) {
    LOG(INFO) << "Ignore UnmapNotify for non-client window " << e.window;
    return;
  }

  Unframe(e.window);
}

void WindowManager::Unframe(Window w) {
  // We reverse the steps taken in Frame().
  const Window frame = clients_[w];
  // 1. Unmap frame.
  XUnmapWindow(display_, frame);
  // 2. Reparent client window back to root window.
  XReparentWindow(
      display_,
      w,
      root_,
      0, 0);  // Offset of client window within root.
  // 3. Remove client window from save set, as it is now unrelated to us.
  XRemoveFromSaveSet(display_, w);
  // 4. Destroy frame.
  XDestroyWindow(display_, frame);
  // 5. Drop reference to frame handle.
  clients_.erase(w);

  LOG(INFO) << "Unframed window " << w << " [" << frame << "]";
}

A few additional points to note:

  • When our window manager unmaps the frame window with XUnmapWindow() in step 1 in the example code above, it will again receive a corresponding UnmapNotify event. This is the reason why the UnmapNotify event handler needs to check that the unmapped window is an actual client window.

  • When our window manager makes the client window a direct child of the root window with XReparentWindow() in step 2 above, it will receive a ReparentNotify event. As discussed in the Mapping a Window section above, this ReparentNotify event will be ignored.

  • When our window manager destroys the frame window with XDestroyWindow() in step 4, it will trigger a DestroyNotify event. This event will also be ignored, as shown in the next section.

At this point, the client window has become invisible, but not yet destroyed. It can be displayed again with a call to XMapWindow(), which would take us back to the Mapping a Window step. It could also be reconfigured in this state, which would take us back to the Configuring a Newly Created Window step.

Destroying a Window

When a client application exits or no longer needs a window, it will call XDestroyWindow() to dispose of the window. This triggers a DestroyNotify event. In our case, there’s nothing we need to do in response.

void WindowManager::Run() {
      ...
      case DestroyNotify:
        OnDestroyNotify(e.xdestroywindow);
        break;
      ...
}

void WindowManager::OnDestroyNotify(const XDestroyWindowEvent& e) {}

Framing Existing Top-Level Windows

Now that we’ve walked through the life cycle of a client window, from creation to destruction, let’s turn our attention to the problem of existing top-level windows.

You may recall from Part I that X applications in general run just fine without a window manager. Depending on how an X session is started (e.g. xinitrc), by the time a window manager starts, any number of windows may have already been created by other applications. Additionally, the user can kill a running window manager and replace it with a different window manager, without affecting windows from other applications.

Therefore, when our window manager starts up, it needs to handle any existing top-level windows that are already mapped. As a reparenting window manager, it will invoke the same Frame() function on such windows as if these windows are being mapped for the first time:

void WindowManager::Run() {
  // 1. Initialization.
  //   a. Select events on root window. Use a special error handler so we can
  //   exit gracefully if another window manager is already running.
  ...
  //   b. Set error handler.
  ...
  //   c. Grab X server to prevent windows from changing under us while we
  //   frame them.
  XGrabServer(display_);
  //   d. Frame existing top-level windows.
  //     i. Query existing top-level windows.
  Window returned_root, returned_parent;
  Window* top_level_windows;
  unsigned int num_top_level_windows;
  CHECK(XQueryTree(
      display_,
      root_,
      &returned_root,
      &returned_parent,
      &top_level_windows,
      &num_top_level_windows));
  CHECK_EQ(returned_root, root_);
  //     ii. Frame each top-level window.
  for (unsigned int i = 0; i < num_top_level_windows; ++i) {
    Frame(top_level_windows[i], true /* was_created_before_window_manager */);
  }
  //     iii. Free top-level window array.
  XFree(top_level_windows);
  //   e. Ungrab X server.
  XUngrabServer(display_);

  // 2. Main event loop.
  ...
}

void WindowManager::OnMapRequest(const XMapRequestEvent& e) {
  // 1. Frame or re-frame window.
  Frame(e.window, false /* was_created_before_window_manager */);
  ...
}

void WindowManager::Frame(Window w, bool was_created_before_window_manager) {
  ...
  // 1. Retrieve attributes of window to frame.
  ...
  // 2. If window was created before window manager started, we should frame
  // it only if it is visible and doesn't set override_redirect.
  if (was_created_before_window_manager) {
    if (x_window_attrs.override_redirect ||
        x_window_attrs.map_state != IsViewable) {
      return;
    }
  }
  // 3. Create frame.
  ...
}

void WindowManager::OnUnmapNotify(const XUnmapEvent& e) {
  ...

  // Ignore event if it is triggered by reparenting a window that was mapped
  // before the window manager started.
  //
  // Since we receive UnmapNotify events from the SubstructureNotify mask, the
  // event attribute specifies the parent window of the window that was
  // unmapped. This means that an UnmapNotify event from a normal client window
  // should have this attribute set to a frame window we maintain. Only an
  // UnmapNotify event triggered by reparenting a pre-existing window will have
  // this attribute set to the root window.
  if (e.event == root_) {
    LOG(INFO) << "Ignore UnmapNotify for reparented pre-existing window "
              << e.window;
    return;
  }

  Unframe(e.window);
}

Some additional things to note:

  • You may notice that the process of framing existing top-level windows is guarded by XGrabServer() and XUngrabServer(). From the Xlib Programming Manual:

    These functions can be used to control processing of output on other connections by the window system server. While the server is grabbed, no processing of requests or close downs on any other connection will occur.

    — Xlib Programming Manual §9.5

    By grabbing the X server, our window manager ensures that, between the time when it fetches the list of existing top-level windows and when it finishes framing them, no other application can interfere and mess up our state: no new windows can be created, and no existing windows can be modified or destroyed.

  • The override_redirect attribute, if set to true, indicates that a window should not be managed by window managers. From the Xlib Programming Manual:

    To control window placement or to add decoration, a window manager often needs to intercept (redirect) any map or configure request. Pop-up windows, however, often need to be mapped without a window manager getting in the way. […]

    The override-redirect flag specifies whether map and configure requests on this window should override a SubstructureRedirectMask on the parent. You can set the override-redirect flag to True or False (default). Window managers use this information to avoid tampering with pop-up windows […].

    — Xlib Programming Manual §3.2.8

    The reason our window manager doesn’t need to check for this attribute except at start up is that the X server knows not to redirect events from such windows:

    The window manager […] will normally ignore windows that are mapped with their override_redirect attribute set, since no *Request events will be generated for them.

    — Xlib Programming Manual §16.3
  • The map_state attribute indicates whether a window is currently visible (mapped). When Frame() is invoked for pre-existing windows during start up, we want to ignore windows that are currently unmapped. However, when Frame() is invoked during the event loop as part of the MapRequest handler, we know that the client window to be framed is necessarily still unmapped, as our window manager wouldn’t have granted the request yet.

  • You might be wondering why an additional check for e.event == root_ is needed in the UnmapNotify handler. It turns out that reparenting an already mapped window (XReparentWindow()) will trigger a pair of UnmapNotify and MapNotify events in addition to ReparentNotify. Therefore, when we enter into the event loop, we will receive an UnmapNotify event for every pre-existing top-level window we reparented. We can distinguish these events by their event attribute, which in this case represents the parent of the client window. Normally, when a client window we already framed is unmapped, the event attribute would be its frame window. But when a pre-existing window is reparented at start up, the event attribute in the resulting UnmapNotify event will be its original parent - i.e., the root window.

What’s Next

At this point, we have a basic but functional reparenting window manager that will correctly handle the life cycle of windows. If you strip out window decorations, shortcuts and fancy UI, the core structure of every X window manager will quite closely resemble what we have here.

In our next installment, we will improve the user-facing functionality of our window manager by adding ways to move, resize and close windows. In the meantime, you’re more than welcome to check out the code for basic_wm on GitHub.

Flashing a Sprint Nexus S 4G to Verizon

I originally wrote the following post in July 2012 to document how to fully flash a Nexus S 4G from Sprint to Verizon Wireless, but never got to publishing it. I have long since switched away from Verizon Wireless and no longer own any of the phones mentioned, and decided to publish it for what it’s worth.

If you’re interested in the high-level view of how CDMA phones are programmed, check out my previous article Carrier Programming on CDMA Android Phones.

Introduction

This is a guide for flashing a Sprint Samsung Nexus S 4G to a standard Verizon monthly plan, and voice, texting and 3G data will all be fully functional. This method will likely also work on many other phones, especially Samsung ones, and should theoretically work just file on any ROM as well.

Note that this procedure entails ESN cloning, which may be illegal in your region. Please make sure you have an understanding of all applicable laws before proceeding. The author of this guide cannot be held responsible for any legal infractions that may ensue. Finally, you are solely responsible for any consequences of your actions as a result of following this guide. While I believe it to be quite safe, I cannot guarantee you that this process will not brick your devices or start a zombie apocalypse.

Requirements and Setup

We will need the following before starting:

  • A currently active Verizon Wireless Android phone and plan. Please beware that not all phones will work; many phones do not allow us to extract all the information we need for this process. For instance, my Motorola Droid 3 did not work, and I had to purchase and activate an HTC Incredible for this. This phone will be our donor phone.

  • A rooted Samsung Nexus 4G.

  • A computer running a recent version of Windows, with at least 2 USB ports.

  • 2 MicroUSB cables.

In addition, we need the following software:

  • USB diagnostic drivers for your donor phone. These vary by manufacturer and their configuration may be quite complex; search online for how to set up yours. For HTC phones, download and install HTC Sync from HTC’s website (archived here).

  • USB diagnostic drivers for the Nexus S 4G. The easiest way is to download and install PdaNet (archived here). We don’t actually need the functionality of PdaNet, but it happens to bundle the drivers we need. Select Samsung when prompted for the manufacturer of our phone.

  • QPST and QXDM (archived here). These are internal Qualcomm tools; we will use them to clone the MEID of the donor phone.

  • DFS (archived here). This is a third-party tool for working with CDMA phones. The demo version will suffice.

Step 1: Preparation

First, make sure the donor phone is fully activated and that calls, text and data are all fully functional.

Now, find the HEX MEID of the donor phone. On the HTC Incredible, this can be found under Settings → About Phone → Phone Identity. On most Android phones, this is under Settings → About Phone → Status. It is also usually printed on a label underneath the phone battery. This should be 14 digits and usually begins with A00000.

Now put the donor phone in airplane mode.

We will next connect both phones to the computer in diagnostic mode.

  • On the Nexus S 4G, dial *#*#8778#*#*, then select MODEM under USB. Connect the Nexus S 4G to the computer. If drivers were installed correctly, you should now see a SAMSUNG Mobile Modem Diagnostic Serial Port (WDM) under Ports (COM & LPT) in Windows’s Device Manager. Note down the port number of the Nexus S 4G; we will assume COM4 for convenience.

  • On the HTC Incredible, dial ##3424#, hit Call, connect it to the computer, then follow the instructions in this thread with Device Manager. If everything goes well you should now see a HTC Diagnostic Interface under Ports (COM & LPT) in Device Manager. Note down the port number of the HTC Incredible; we will assume COM11 for convenience.

After confirming that both phones show up under Ports (COM & LPT), run QPST Configuration and hit Add New Port…. You should see both phones on the left. Select one and hit OK. Then repeat for the other phone.

Step 2: Cloning

We will now clone the MEID of the donor phone onto the target phone. Open QXDM Professional. Select Options → Communications…. For Target Port, select the port corresponding to the Nexus S 4G, say COM4, and hit OK. Then in the text box labeled Command, type the following:

Password 01F2030F5F678FF9

Hit Enter. You should see Password Result = Correct at the bottom of the Command Output window.

Now in the Command text box, type the following, replacing <MEID> by the hex MEID of the donor phone:

RequestNVItemWrite meid 0x<MEID>

Hit Enter. You should see it repeat back the MEID to you. Type RequestNVItemRead meid followed by Enter to make sure the write happened.

Close QXDM Professional and reboot the Nexus S 4G. Rebooting the target phone after writing a new MEID is essential! If prompted, do NOT go through the activation process on the Nexus S 4G. Just select Skip when possible.

Step 3: Reset SPC

In order to program the Nexus S 4G, we need to obtain or reset the SPC code (also known as the MSL) of the phones. If you already know the SPC code for your phones and would rather not reset the SPC code, you can skip this step.

Open QXDM Professional again. Go through Options → Communications and make sure the Target Port is still COM4. Hit OK. In the command text box, type the following two lines, each followed by an Enter:

Password 01F2030F5F678FF9
RequestNVItemWrite sec_code 000000

If this is successful, the SPC on the Nexus S 4G will have been reset to 000000. Repeat for the donor phone (in our example, on COM11).

Close QXDM Professional after done. All QPST / QXDM programs must be closed before proceeding, or DFS will complain.

Step 4: Programming CDMA Chipset

Launch DFS. Click Ports on the top left. In the dialog box, double-click on each port representing our phones, in our case, COM4 and COM11, and close the dialog box. Now make sure the SPC text box above reads 000000 (or if you chose not to reset the SPC, whatever your SPC is). Type 01F2030F5F678FF9 into the Pwd text box.

Step 4.a: Basic CDMA settings

Let’s now copy basic CDMA settings from the donor phone to the target phone. In the drop-down menu next to Ports, select the donor phone (COM11). Switch to the Programming tab and the NAM tab under it. Click the SPC and Pwd buttons to send the SPC and NV password to the phone to unlock it. Then hit Read beneath Network identification. The text fields below should be populated by information from the donor phone. Now select COM4 in the drop-down menu next to Ports, click SPC and Pwd, then click Write beneath Network identification.

Step 4.b: PRL

Now, let’s copy over the PRL (or Preferred Roaming List - essentially a database of tower locations). Again, select COM11 under Ports and click Read under PRL towards the right. Then select COM4 under Ports and click Write under PRL.

Step 4.c: 2G Data settings

Now go to the next tab, Data. Select COM11 and hit Read. Switch to COM4 and hit Write. If you observe errors in the log window in the bottom, and / or no information shows up in the Pwd boxes under PPP and HDR AN Long, your donor phone doesn’t support the extraction of some data passwords, and you will have to try a different donor phone.

Step 4.c: 3G Data settings

We do the same with Mobile IP. Select COM11 and hit Read. Switch to COM4 and hit Write. Then hit Write current profile settings. Again, if you encounter errors, or no information shows up under AAA Shared secret and HA Shared Secret, you will have to try a different donor phone.

Finally, we need to copy over a file containing a secret key used to establish a 3G connection. Go to the EFS tab towards the top. Select COM11 and hit Read EFS. Click the "+" sign of the root folder on the left and then the "+" sign next to the DMU folder and you should see a file called 10.key. Right-click on this file and hit Save…. Save this file somewhere on your hard drive. Now switch to COM4 and hit Read EFS again. Select the root folder on the left, type /DMU under Path (57 max) on the right and hit Add Item to create a folder named DMU on the Nexus S 4G. Then select DMU on the left, click on the check box named I want to add file from PC, and select the 10.key file we just saved. Then hit Add Item to upload the 10.key file to /DMU/10.key.

The programming is now done. Close DFS, select OK and OK again when prompted whether to send a mode reset to the phone. The Nexus S 4G will now reboot.

Step 5: Programming Android

We will now modify some Android system files to Verizon settings. This step will need to be repeated whenever you flash a new ROM on the Nexus S 4G in the future.

1-click programming app

I have built a 1-click Android application that automates this step; you can find it here. It requires root as it will need to modify system files. Install the application and click the button in the middle. When the process finishes, you will be prompted to reboot. Simply reboot your phone and you’re all set; you may uninstall the app, or you could keep it around for when you update to a new ROM.

If the 1-click app does not work for you, or if you want more control over what it does, follow the steps below. You do not need to do any of the following if you already used the app above.

Step 5.a: Fix voice calling

This is required to enable calls on Verizon. On the Nexus S 4G, connect to WiFi and install ES File Explorer from the Play Store. Then go to Menu → Settings and check Up to Root, Root Explorer, and Mount File System. Allow it to use superuser privileges if asked. Then navigate to /system and open build.prop with ES Note Editor. Find the following two lines:

ro.cdma.home.operator.numeric=310120
ro.cdma.home.operator.alpha=Sprint

Change them to read:

ro.cdma.home.operator.numeric=310004
ro.cdma.home.operator.alpha=Verizon

Then add a line below (or anywhere in the file) that says:

ro.cdma.homesystem=64,65,76,77,78,79,80,81,82,83

Hit Menu → Save and then Back to exit.

Step 5.b: Fix roaming

This step tells Android to treat the Verizon network as the home network. It sets the roaming status to "Not roaming", removes the triangular roaming indicator in the status bar and changes the displayed name of the current network to "Verizon Wireless".

Now navigate to /system/framework and copy framework-res.apk onto your computer - you can use adb or email it to yourself or whatever (you have to copy it to the internal SD card before you can email it though). Once on the computer, rename it framework-res.zip (yes, an APK is just a zip archive) and use your favorite program to replace the file res/xml/eri.xml inside it with the one at this link. Now rename it back to framework-res.apk, copy it back onto the Nexus S 4G (using adb or whatever) and use ES File Explorer to overwrite the existing /system/framework/framework-res.apk with it.

Step 5.c: Add Verizon APNs

This last step sets up Verizon APNs that enable web and MMS. Use ES File Explorer or any other tool to replace /system/etc/apns-conf.xml with the one at this link.

Step 6: Reboot & PROFIT

The Nexus S 4G should now be a fully functional Verizon Wireless phone. If you are curious about what each of the steps means, you’re welcome to check out my previous article Carrier Programming on CDMA Android Phones for a high-level view of how CDMA phones are programmed.

While your mileage may vary, I hope the above information have been of help. Since I no longer use Verizon Wireless and no longer own any of the phones mentioned above, please take all this information with a grain of salt.

Good luck, and happy hacking!

Custom Domain E-mails With Postfix and Gmail: The Missing Tutorial

Custom domain e-mail addresses, like [email protected], are cool and professional-looking. Would you want to e-mail potential clients as [email protected]?

On the other hand, Gmail has great reliability, speed, spam filtration, and app support. Wouldn’t it be great if we could send and receive e-mail as [email protected], but right from the comfort of our Gmail account?

Today, if we want to use Gmail to send and receive e-mails on a custom domain, we have a couple of options:

  • Google Apps for Work is Google’s own offering. It’s simple to set up and manage, but it’s pretty pricey at $5/mo per user, especially if we don’t need the bundled collaborative features like Google Docs and Calendar.

  • Third-party services could get the job done at a lower price point; for example, Pobox offers plans starting at $20/yr. The downside is that we will have to trust another organization with our e-mail.

  • Set up our own e-mail forwarding server. It’s fairly easy (if we follow this guide :) and if we already have shared / dedicated hosting or a VPS, this is basically free and gives us total control over our e-mail.

If you want to host custom domain e-mails in Gmail without paying for Google Apps or a third-party service, this tutorial is for you. It will show you how to quickly set up you own e-mail forwarding / relay server, and how to integrate it seamlessly with Gmail.

Prerequisites

For this tutorial, we’ll assume we have access to:

  • VPS or shared / dedicated hosting, with a dedicated IP address and capable of listening on ports 25 and 587. We will use a Ubuntu server for the examples.

  • DNS records for our domain. How DNS records are manipulated depends on the DNS hosting provider (or DNS server configuration).

We’ll assume

Step 1: DNS Setup

The first step in setting up our e-mail forwarding server is to add MX, PTR and SPF DNS records for our server. This is really important as many e-mail providers, in order to prevent spam, will refuse to talk to mail servers without proper MX, PTR and SPF records set up. How to update DNS records will depend on the DNS hosting provider.

The following is what we need:

  • An MX record for example.com, pointing to myserver.example.com.

    This tells the world that e-mails to <whatever>@example.com should be delivered to the server myserver.example.com.

  • An A record for myserver.example.com, pointing to the IP address of our server.

  • A PTR (reverse DNS) record mapping the IP address of our server to myserver.example.com.

    This allows Gmail to verify the legitimacy of our server via its IP when Gmail receives a forwarded e-mail from it.

  • A TXT record, with key example.com and value v=spf1 mx ~all.

    This is an SPF record; it tells Gmail that the servers specified in the MX records of example.com, in this case only myserver.example.com, are allowed to send e-mails purporting to be from <whatever>@example.com. All other servers attempting to do the same will be rejected. This should be a sane default value, but feel free to custom it as you like.

DNS records will take a while (depending on our provider, up to a day) to propagate. Until they do, e-mails forwarded by our new e-mail server may get marked as spam or rejected outright.

Step 2: Receiving E-mail

We’ll first set up an e-mail forwarding server which will let we receive e-mails sent to our domains.

We will use Postfix as the e-mail server. Let’s start by installing the necessary packages. We will assume a Ubuntu server in the examples below; if you’re using a different distribution, please consult your distribution’s documentation for the right commands.

$ sudo DEBIAN_FRONTEND=noninteractive apt-get install postfix

In the above command, we skip the debconf configuration UI with DEBIAN_FRONTEND=noninteractive as we will edit the configuration files directly.

/etc/postfix/main.cf

Open up /etc/postfix/main.cf in your favorite editor. This file will come pre-populated with a bunch of config options and comments. Replace the contents of the file with the following (you can comment out the original contents for reference):

# /etc/postfix/main.cf

# Host and site name.
myhostname = myserver.example.com
mydomain = example.com
myorigin = example.com

# Virtual aliases.
virtual_alias_domains = example.com
virtual_alias_maps = hash:/etc/postfix/virtual

The first few lines are pretty straightforward; they tell Postfix how to identify itself to the world. The last two lines tell Postfix to forward e-mails sent to <whatever>@example.com to another e-mail provider (Gmail), and that the forwarding is configured in the database file /etc/postfix/virtual.

/etc/postfix/virtual

Let’s now open up /etc/postfix/virtual and fill in our forwarding configuration:

# /etc/postfix/virtual

# Forwarding mapping, one from-to address pair per line. The format is:
#     <forward-from-addr> <whitespace> <forward-to-addr>
[email protected]        [email protected]

We can add as many forwarding rules as you want, one on each line. We can use any number of tabs / whitespaces between the forward-from and forward-to addresses.

Update lookup table

It turns out that Postfix doesn’t actually read /etc/postfix/virtual (surprise!); instead, what it reads is a lookup table generated from it. So, let’s generate the lookup table from our /etc/postfix/virtual:

$ sudo postmap /etc/postfix/virtual

Note: we must re-run this command every time we modify /etc/postfix/virtual!

(Re)start Postfix

It’s now time to (re)start Postfix with our new configuration:

$ sudo postfix start
$ sudo postfix reload

Testing

We should test our brand-new Postfix server by sending an e-mail to our forward-from address. You might want to check out these tutorials for testing directly with the telnet command.

If you received the e-mail in your Gmail inbox, congratulations! Otherwise, check the Postfix logs at /var/log/mail.log (or with journalctl -u postfix if using Systemd) for errors. The most likely cause of issues is that DNS records have not propagated yet, in which case we’re likely to see a "rate limited" or "rejected for spam" type error message.

If we only want to receive but don’t care about sending e-mails as [email protected], then this is all we need.

Otherwise, let’s move on to configuring Postfix to support sending e-mails from Gmail.

Step 3: Sending E-mail

Gmail requires a relay server (a server that will send e-mails to their destination on behalf it) to speak TLS, which protects the communication between Gmail and the relay server.

We will use Cyrus SASL for this task. To install:

$ sudo apt-get install sasl2-bin libsasl2-modules

User name & password

Open relays are a terrible idea. We want to protect our e-mail server with a user name and password so that it will let Gmail send e-mails through it, but block spammers and other evil actors.

Cyrus SASL supports several backends for storing user names and passwords, including MySQL and PAM. However, we will go with the simplest backend — a plain database file.

Let’s create the user name / password database file in the default location /etc/sasldb2, with a single user named smtp (we can change the user name to anything we want):

$ sudo saslpasswd2 -c -u example.com smtp

Verify with:

$ sudo sasldblistusers2

Now, we make sure only Postfix can read this file:

$ sudo chmod 400 /etc/sasldb2
$ sudo chown postfix /etc/sasldb2

Lastly, we tell Cyrus SASL to use the file-based database to authenticate. Create the file /etc/postfix/sasl/smtpd.conf:

pwcheck_method: auxprop
auxprop_plugin: sasldb
mech_list: PLAIN LOGIN CRAM-MD5 DIGEST-MD5 NTLM
log_level: 7

SSL Certificate

We need an SSL certificate to enable TLS. While a proper SSL certificate signed by a Certificate Authority (CA) can be pricey, it turns out that a simple self-signed certificate suffices and works just fine.

So, let’s generate one.

  1. Generate an RSA private/public key pair. Note that you MUST supply a password to this command; the password will be removed in step 3.

    $ openssl genrsa -des3 -out example.key 1024
    
  2. Generate a Certificate Signing Request (CSR). Make sure to enter myserver.example.com when prompted for the "Common Name".

    $ openssl req -new -key example.key -out example.csr
    
  3. Remove RSA private/public key password.

    $ mv example.key example.key.orig
    $ openssl rsa -in example.key.orig -out example.key
    
  4. Generate a self-signed certificate. In the example, the generated certificate will be valid for 10 years.

    $ openssl x509 -req \
        -days 3650 -in example.csr -signkey example.key -out example.crt
    
  5. Create a PEM file.

    $ cat example.crt example.key > example.pem
    
  6. Move and protect the PEM file.

    $ sudo mv example.pem /etc/postfix/example.pem
    $ sudo chmod 400 /etc/postfix/example.pem
    $ sudo chown postfix /etc/postfix/example.pem
    

Note: Make sure to protect the generated private key and certificate with care!

Relay server

The final step is to configure Postfix to enable relaying of e-mail on behalf of Gmail.

Let’s open up /etc/postfix/master.cf. This file should already contain a bunch of config options, some of which are commented out. Uncomment the lines starting with submission and edit them to match the following:

submission inet n       -       n       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=may
  -o smtpd_tls_cert_file=/etc/postfix/example.pem
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

A Postfix restart is due after all these changes:

$ sudo postfix reload

If all went well, you should see Postfix serving a relay server, protected by our user name and password in /etc/sasldb2, on port 587.

Step 4: Configure Gmail

Finally, let’s log in to Gmail and tell it about our brand new server. (Note that the new Inbox UI does not yet support all the settings available in the legacy interface at the time of writing, so these instructions apply to the legacy interface.)

Let’s go to Settings > Accounts and Import and click Add another email address you own.

In the dialog that pops up, fill in our target e-mail address and click Next Step.
In the next dialog, enter the address of our e-mail server (myserver.example.com), and the user name and password we set up using saslpasswd2 above. Make sure the user name is suffixed with our domain name (so [email protected] rather than just smtp). Check that the correct port (587) and the correct security protocol (TLS) are selected, then click Add Account.
If all goes well, we should see a dialog like the following:
And we should receive an e-mail, forwarded by our mail server set up in the first part, to our [email protected] address. Click on the link inside the e-mail and we’re done!

Now, in any e-mail compose / reply window, you should be able to select [email protected] in the drop-down list next to "From". Congrats!

If, on the other hand, you see an error like the following:

You should check the Postfix logs at /var/log/mail.log (or with journalctl -u postfix if using Systemd) for any errors.

[Optional] Step 5: Setup DKIM and SRS

DKIM, short for DomainKeys Identified Mail, is e-mail validation system for preventing e-mail spoofing. Google recommends setting up DKIM; if we don’t, the e-mail messages that we forward to Gmail have a higher probability of getting marked as spam.

SRS, short for Sender Rewriting Scheme, is a relatively new standard that e-mail forwarding servers are recommended to adopt, and it also helps reduce the likelihood of our forwarded e-mail messages getting marked as spam.

Thus, while technically optional, it’s a good idea to configure our Postfix server to support both these standards. See my follow-up tutorial on Setting Up DKIM And SRS In Postfix for a detailed step-by-step guide.

Conclusion

In this post, we have set up a Postfix e-mail server that accomplishes the following:

  • E-mails sent to [email protected] is received by Postfix on myserver.example.com port 25, and forwarded to [email protected];

  • Gmail will let you select [email protected] as a "from" address when composing / replying to an e-mail, and will relay such e-mails through Postfix on myserver.example.com port 587 using a configured user name / password combination.

I hope you found this guide useful, and if you have any thoughts / questions, you’re more than welcome to leave a comment below. Cheers!

How X Window Managers Work, And How To Write One (Part II)

In Part I of this series, we examined the role of X window managers in a modern Linux/BSD desktop environment, and how they interact with the X server and applications. In Part II, we will dig into the dirty details and walk through the code of an example reparenting non-compositing window manager, basic_wm.

Introduction

Before we start with the code, let’s go over a couple of basic implementation choices such as language and API.

Language

You can write a window manager in Haskell, Python, Lisp, Go, Java, or any other language that has X bindings, i.e. a library for communicating with X servers.

I chose C++ for basic_wm, our example window manager, mainly because the C libraries for X11 are the best documented. In addition to books such as the Xlib Programming Manual, documentation can be found in the form of widely available man pages (e.g., try man XOpenDisplay at a terminal). Example usage and common patterns abound in the source code of many great window managers written in the past three decades.

We will use C++11 and C++14 features where convenient, so you will need a compatible compiler (GCC 4.9 or higher, or Clang 3.4 or higher) if you want to play with the example source code.

A Tale of Two X Libraries

There are two official C libraries for X: Xlib and XCB. Xlib, hailing from 1985, was the original X client library, and was the only official X client library until the introduction of XCB in 2001. The two libraries have very different philosophies: whereas Xlib tries to hide the X protocol behind a friendly C API with lots of bells and whistles, XCB directly exposes the plumbing beneath.

In practice, this different manifests itself most prominently in how the two libraries handle the fundamental asynchronous nature of X’s client-server architecture. Xlib attempts to hide the asynchronous X protocol behind a mixed synchronous and asynchronous API, whereas XCB exposes a fully asynchronous API.

For example, to lookup the attributes (e.g., size and position) of a window, you would write the following code using Xlib:

XWindowAttributes attrs;
XGetWindowAttributes(display, window, &attrs);
// Do stuff.

Under the hood, XGetWindowAttributes() sends a request to the X server and blocks until it receives a response; in other words, it is synchronous. On the other hand, using XCB, you would write this instead:

xcb_get_window_attributes_cookie_t cookie =
    xcb_get_window_attributes(
        connection, window);
// Do other stuff while waiting for reply.
xcb_get_window_attributes_reply_t* reply =
    xcb_get_window_attributes_reply(
        connection, cookie, nullptr);
// Do stuff.
free(reply);

The function xcb_get_window_attributes merely sends the request to the X server, and returns immediately without waiting for the reply; in other words, it is asynchronous. The client program must subsequently call xcb_get_window_attributes_reply to block on the response.

The advantage of the asynchronous approach is obvious if we consider an example where we need to retrieve the attributes of, say, 5 windows at once. Using XCB, we can immediately fire off all 5 requests to the X server, and then wait for all of them to return. With Xlib, we have send one request at a time and wait for its response to come back before we can send the next request. Therefore, we’d expect to only block for the duration of one round-trip to the X server using XCB, compared to 5 with Xlib.

The downside of XCB’s fully asynchronous approach is verbosity and a less programmer-friendly interface. The Xlib code above looks like your average C library call; the XCB code above is significantly more involved.

However, it is important to note that Xlib isn’t fully synchronous. Rather, Xlib has a mixture of synchronous and asynchronous APIs. In general, functions that do not return values (e.g., XResizeWindow, which changes the size of a window) are asynchronous, while functions that return values (e.g., XGetGeometry, which return the size and position of a window) are synchronous:

Xlib saves up requests instead of sending them to the server immediately, so that the client program can continue running instead of waiting to gain access to the network after every Xlib call. This is possible because most Xlib calls do not require immediate action by the server. This grouping of requests by the client before sending them over the network also increases the performance of most networks, because it makes the network transactions longer and less numerous, reducing the total overhead involved.

Xlib sends the buffer full of requests to the server under three conditions.

The most common is when an application calls an Xlib routine to wait for an event but no matching event is currently available on Xlib’s queue. Since, in this case, the application must wait for an appropriate event anyway, it makes sense to flush the request buffer.

Second, Xlib calls that get information from the server require a reply before the program can continue, and therefore, the request buffer is sent and all the requests acted on before the information is returned.

Third, the client would like to be able to flush the request buffer manually in situations where no user events and no calls to query the server are expected. One good example of this third case is an animated game, where the display changes even when there is no user input.

— Xlib Programming Manual §2.1.2

This is the most confusing aspect of Xlib, and a source of endless frustration for those new to X programming. One of the major motivations for the creation of XCB was to eliminate this complexity.

Many popular window managers have already been ported to XCB from Xlib for the performance benefits. If you are interested, you can read up on how the Awesome and KWin window managers were ported to XCB.

I chose to use Xlib for basic_wm, however, because as a pedagogical example, readability and simplicity is much more important than performance. In fact, I would recommend starting with Xlib first for any project and worry about porting to XCB later, as Xlib is much easier to learn and prototype with.

While an in-depth discussion of the merits of Xlib and XCB is beyond the scope of this discussion, I do recommend you check out the official article on Xlib vs. XCB as it presents a fascinating case study of API design.

Dependencies and Building

Firstly, you will need Xlib development headers in order to compile against Xlib. They are available on Debian/Ubuntu as libx11-dev, on Fedora as libX11-devel, and on Arch Linux as part of libx11.

The only additional library used by the example basic_wm code is google-glog, Google’s open source C++ logging library. It is available on Debian/Ubuntu as libgoogle-glog-dev, on Fedora as glog-devel, and on Arch Linux as google-glog.

The recommended way to build the source code is with GNU Make: just run make in the source directory. Alternatively, g++ *.cpp will also do the trick if you supply all the libraries correctly.

To test the window manager, you will likely need Xephyr along with a couple of simple X programs such as xeyes or xterm.

Step 1: Setup and Teardown

Let’s start off with a skeleton implementation of the WindowManager class, which will encapsulate all the window management logic in our example. All it will do for now is set up a connection to the X server on construction, and close that connection on destruction.

extern "C" {
#include <X11/Xlib.h>
}
#include <memory>

class WindowManager {
 public:
  // Factory method for establishing a connection to an X server and creating a
  // WindowManager instance.
  static ::std::unique_ptr<WindowManager> Create();
  // Disconnects from the X server.
  ~WindowManager();
  // The entry point to this class. Enters the main event loop.
  void Run();

 private:
  // Invoked internally by Create().
  WindowManager(Display* display);

  // Handle to the underlying Xlib Display struct.
  Display* display_;
  // Handle to root window.
  const Window root_;
};
#include "window_manager.hpp"
#include <glog/logging.h>

using ::std::unique_ptr;

unique_ptr<WindowManager> WindowManager::Create() {
  // 1. Open X display.
  Display* display = XOpenDisplay(nullptr);
  if (display == nullptr) {
    LOG(ERROR) << "Failed to open X display " << XDisplayName(nullptr);
    return nullptr;
  }
  // 2. Construct WindowManager instance.
  return unique_ptr<WindowManager>(new WindowManager(display));
}

WindowManager::WindowManager(Display* display)
    : display_(CHECK_NOTNULL(display)),
      root_(DefaultRootWindow(display_)) {
}

WindowManager::~WindowManager() {
  XCloseDisplay(display_);
}

void WindowManager::Run() { /* TODO */ }

The main function in main.cpp:

#include <cstdlib>
#include <glog/logging.h>
#include "window_manager.hpp"

using ::std::unique_ptr;

int main(int argc, char** argv) {
  ::google::InitGoogleLogging(argv[0]);

  unique_ptr<WindowManager> window_manager(WindowManager::Create());
  if (!window_manager) {
    LOG(ERROR) << "Failed to initialize window manager.";
    return EXIT_FAILURE;
  }

  window_manager->Run();

  return EXIT_SUCCESS;
}

Even if you have never programmed Xlib before, this should not be hard to understand. WindowManager::Create() is a static factory method that sets up a connection to an X server via XOpenDisplay(); we will let XOpenDisplay() figure out which X server to connect to from the DISPLAY environment variable. The connection is represented by the opaque Display structure. We call XCloseDisplay() on the saved Display* in the destructor to close the connection.

The other function of note is DefaultRootWindow(), which returns the default root window for a given X server. Technically, an X server may have several root windows in some rare multihead setups, but let’s not worry about that here.

If you run this program now, it should connect to the X server, close the connection, and exit. Hooray!

Step 2: Initialization

Now, let’s dig into the mysterious Run() function above. We’ll start with the initialization steps required after opening an X server connection. In window_manager.hpp:

class WindowManager {
  ...
  // Xlib error handler. It must be static as its address is passed to Xlib.
  static int OnXError(Display* display, XErrorEvent* e);
  // Xlib error handler used to determine whether another window manager is
  // running. It is set as the error handler right before selecting substructure
  // redirection mask on the root window, so it is invoked if and only if
  // another window manager is running. It must be static as its address is
  // passed to Xlib.
  static int OnWMDetected(Display* display, XErrorEvent* e);
  // Whether an existing window manager has been detected. Set by OnWMDetected,
  // and hence must be static.
  static bool wm_detected_;
};
void WindowManager::Run() {
  // 1. Initialization.
  //   a. Select events on root window. Use a special error handler so we can
  //   exit gracefully if another window manager is already running.
  wm_detected_ = false;
  XSetErrorHandler(&WindowManager::OnWMDetected);
  XSelectInput(
      display_,
      root_,
      SubstructureRedirectMask | SubstructureNotifyMask);
  XSync(display_, false);
  if (wm_detected_) {
    LOG(ERROR) << "Detected another window manager on display "
               << XDisplayString(display_);
    return;
  }
  //   b. Set error handler.
  XSetErrorHandler(&WindowManager::OnXError);

  // 2. Main event loop.
  ...
}

int WindowManager::OnWMDetected(Display* display, XErrorEvent* e) {
  // In the case of an already running window manager, the error code from
  // XSelectInput is BadAccess. We don't expect this handler to receive any
  // other errors.
  CHECK_EQ(static_cast<int>(e->error_code), BadAccess);
  // Set flag.
  wm_detected_ = true;
  // The return value is ignored.
  return 0;
}

int WindowManager::OnXError(Display* display, XErrorEvent* e) { /* Print e */ }

We first select substructure redirection and substructure notify events on the root window. This is discussed in more detail in the Substructure Redirection section in Part I; to recap, this allows the window manager to intercept requests from top level windows, and subscribe to events concerning the same. Only one X client can select substructure redirection on the root window at any given time; the second client to attempt to do so will get a BadAccess error.

Catching this error is somewhat tricky, however. XSelectInput, like all asynchronous Xlib functions, does not actually send a request to the X server, but instead only queues the request and returns. Hence, we have to explicitly flush the request queue with XSync (see our discussion above in A Tale of Two X Libraries). We set up a temporary error handler, OnWMDetected, to catch errors during this XSync invocation.

Next, we set up our regular error handler which will be invoked for any future errors. Our implementation, which logs the error and continues, will be an important debugging aid as we implement and test our window manager. I will not show it here for the sake of brevity; for reference, check it out in window_manager.cpp.

Step 3: The Event Loop

Now let’s add to Run() method above the signature construct of every modern GUI program - the event loop. In window_manager.cpp:

void WindowManager::Run() {
  // 1. Initialization.
  ...

  // 2. Main event loop.
  for (;;) {
    // 1. Get next event.
    XEvent e;
    XNextEvent(display_, &e);
    LOG(INFO) << "Received event: " << ToString(e);

    // 2. Dispatch event.
    switch (e.type) {
      case CreateNotify:
        OnCreateNotify(e.xcreatewindow);
        break;
      case DestroyNotify:
        OnDestroyNotify(e.xdestroywindow);
        break;
      case ReparentNotify:
        OnReparentNotify(e.xreparent);
        break;
      ...
      // etc. etc.
      ...
      default:
        LOG(WARNING) << "Ignored event";
    }
  }
}

If you have done low-level GUI programming before, this should look very familiar. We sit in an event loop and repeatedly fetch the next event with XNextEvent() and dispatch it to the appropriate handlers.

The structure of the XEvent type is typical of a polymorphic C structure. Each type of event carries different attributes and corresponds to an event struct, such as XKeyEvent, XButtonEvent, and XConfigureEvent. The first field of each struct is always int type. The XEvent type is a C union of all the event structs plus int type:

typedef struct _XKeyEvent {
  int type;
  // Fields specific to XKeyEvent.
  ...
} XKeyEvent;

typedef struct _XButtonEvent {
  int type;
  // Fields specific to XButtonEvent.
  ...
} XButtonEvent;

// etc.
...

typedef union _XEvent {
  int type;
  XKeyEvent xkey;
  XButtonEvent xbutton;
  // etc.
  ...
} XEvent;

This way, the type is always available regardless of the type of event and requires no additional storage. The same pattern can be observed in GTK+/GLib, Python’s C API, and many other object-oriented C APIs.

In basic_wm, the event handlers follow the naming convention of OnFoo(), where Foo is the type of the event, so it should be straightforward to figure out who does what.

What’s Next

We now have a basic skeleton for our window manager, and we can start filling in the meat - the event handlers. The million-dollar question is, what events does a window manager handle, and what should it do with them?

In the next installment in this series, we’ll answer that question by diving into the complex ways window managers, clients and the user interact with each other via X events. In the meantime, you’re more than welcome to check out the code for basic_wm on GitHub.

DEBUG trap and PROMPT_COMMAND in Bash

Update 03/08/2016: A patch by Dan Stromberg adds a PS0 variable to Bash that greatly simplifies what’s described in this article. This patch will likely be merged into Bash 4.4. Please refer to his post for details.

The DEBUG trap

The DEBUG trap is an extremely handy feature of Bash. The idea is pretty straightforward: if you run

trap "echo Hello" DEBUG

then Bash will run echo Hello before it executes each subsequent command. For example:

~/Scratch $ ls
Hello
file1 file2
~/Scratch $ echo Bye
Hello
Bye

A caveat, however, is that the DEBUG trap is triggered once per simple command; if you have command lists or control structures, the trap will be triggered multiple times. For example, using the setup above:

~/Scratch $ echo 1 && echo 2; echo 3
Hello
1
Hello
2
Hello
3
~/Scratch $ if [ -e /etc/passwd ]; then echo "/etc/passwd exists"; fi
Hello
Hello
/etc/passwd exists

What if we only want to run a command once per composite command, like the preexec hook in zsh?

Enter PROMPT_COMMAND.

PROMPT_COMMAND

The idea behind PROMPT_COMMAND is also very simple: if you run

PROMPT_COMMAND="echo Bye"

then Bash will execute echo Bye before it prints each subsequent prompt (i.e., after it has finished executing the previous command line). For example, using the setup above:

~/Scratch $ echo 1; echo 2
Hello
1
Hello
2
Hello
Bye

Note that the DEBUG trap is triggered again for PROMPT_COMMAND, in addition to the user-supplied commands.

Combining the DEBUG trap and PROMPT_COMMAND

By combining the DEBUG trap and PROMPT_COMMAND, we can now hack Bash to run some code right before and right after executing a full command. For example, try adding this to your ~/.bashrc:

# This will run before any command is executed.
function PreCommand() {
  if [ -z "$AT_PROMPT" ]; then
    return
  fi
  unset AT_PROMPT

  # Do stuff.
  echo "Running PreCommand"
}
trap "PreCommand" DEBUG

# This will run after the execution of the previous full command line.  We don't
# want it PostCommand to execute when first starting a bash session (i.e., at
# the first prompt).
FIRST_PROMPT=1
function PostCommand() {
  AT_PROMPT=1

  if [ -n "$FIRST_PROMPT" ]; then
    unset FIRST_PROMPT
    return
  fi

  # Do stuff.
  echo "Running PostCommand"
}
PROMPT_COMMAND="PostCommand"

The result:

~/Scratch $ echo 1; echo 2 && echo 3
Running PreCommand
1
2
3
Running PostCommand

This gives rise to some neat applications, such as a command timer script I wrote that prints out the execution time of each command:

Please feel free to check it out on GitHub :)

Happy Bash hacking!