Kernel on Insignificant Bits

Encrypted mesh PSA

Mon, 31 Oct 2016 13:32:19 +0000

I’m a bit late writing this up, as lately wifi stuff has taken a back seat to the day job. But, I’ve now seen the following issue reported more than once, so hopefully this post will save the other two mesh users some grief.

If you are running an encrypted mesh on wpa_supplicant or (ye gods) authsae, take note:

Encrypted mesh on kernel 4.8 will not interoperate with kernel < 4.8
wpa_supplicant 2.6 will not interoperate with 2.5
authsae as of commit 706a2cf will not interoperate with that of commit 813ec0e [1]
wpa_supplicant 2.6 and authsae commit 706a2cf require additional configuration to work with kernel < 4.8 (see below).

memory Âµ-optimizing

Tue, 15 Mar 2016 12:27:35 +0000

While looking at some kmalloc statistics for my little corner of wireless, I found a new tool for the toolbox: perf kmem. To test it out, I ran a simulated connected 25-node mesh in mac80211_hwsim.

Here’s what perf kmem stat --caller had to say:

---------------------------------------------------------------------
 Callsite            | Total_alloc/Per | Total_req/Per  | Hit   | ...
---------------------------------------------------------------------
 mesh_table_alloc+3e |      1600/32    |       400/8    |    50 |
 mesh_rmc_init+23    |    204800/8192  |    102600/4104 |    25 |
 mesh_path_new.const |    315904/512   |    241864/392  |   617 |
[...]

The number of mesh nodes shows up in the ‘Hit’ column in various ways: we can pretty easily see that mesh_table_alloc is called twice per mesh device, mesh_rmc_init is called once, and mesh_path_new is called at least once for every peer (617 ~= 25 * 24). We might scan the hit column to see if there are any cases in our code that perform unexpectedly high numbers of allocations, signifying a bug or poor design.

wherein I did a git push

Fri, 29 Jan 2016 14:03:09 +0000

As I wrote somewhere in the Metaverse, I’m running the wireless-testing tree now, which is about as minimal a contribution as one could make. It essentially means running a script once daily; cron could do it all, if not for the part where I enter my ssh passphrase to actually push out the updates. For the curious, here’s the script.

One of the killer features of git used here is git rerere, which remembers conflict resolutions and reapplies them when reencountered. A human (me) still needs to resolve the conflict once, but after that it’s pretty much on autopilot for the rest of that kernel series. Normally one should reinspect the automatic merge made by git-rerere, but at least for this testing tree, throwing caution to the wind is warranted and thus the script has some kludges around that.

Phone bugs

Thu, 20 Mar 2014 13:36:43 +0000

If you want to experience what it is like to be the first person to ever do something, might I suggest turning on all the debug knobs on an Android vendor kernel? Some speculative fixes over here, but there is still at least one other RCU bug on boot, and this use-after-free in the usb controller driver:

[   34.520080] msm_hsic_host msm_hsic_host: remove, state 1
[   34.525329] usb usb1: USB disconnect, device number 1
[   34.529602] usb 1-1: USB disconnect, device number 2
[   34.637023] msm_hsic_host msm_hsic_host: USB bus 1 deregistered
[   34.668945] Unable to handle kernel paging request at virtual address aaaaaae6
[   34.675201] pgd = c0004000
[   34.678497] [aaaaaae6] *pgd=00000000
[   34.681762] Internal error: Oops: 5 [#1] PREEMPT SMP ARM
[   34.686737] Modules linked in: wcn36xx_msm(O) wcn36xx(O) mac80211(O) cfg80211(O) compat(O)
[   34.694976] CPU: 1    Tainted: G        W  O  (3.4.0-g4a73a1d-00005-g311eaee-dirty #2)
[   34.702972] PC is at __gpio_get_value+0x28/0x1cc
[   34.707489] LR is at do_restart+0x24/0xd8
[   34.711486] pc : []    lr : []    psr: 60000013
[   34.711486] sp : ebcd9ed8  ip : c032a858  fp : ebcd9ef4
[   34.722930] r10: 00000000  r9 : c04dd67c  r8 : 00000000
[   34.728210] r7 : c4d81f00  r6 : 6b6b6b6b  r5 : c10099cc  r4 : aaaaaaaa
[   34.734680] r3 : 09090904  r2 : c1672f80  r1 : 00000010  r0 : 6b6b6b6b
                                                                 ^^^^^^^^ whoops
[   34.741241] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[   34.748504] Control: 10c5787d  Table: acfb006a  DAC: 00000015

I guess I should try to find out where to report bugs for these (predictably) not upstream drivers, but that seems like a total pain.

Kernel updates

Sun, 18 Jul 2010 00:39:27 +0000

During baby naps I actually did some Rio Karma work this week. First, I found and fixed a recent bug in usb storage that makes the Karma not work at all in 2.6.35. Then I finally set up my omfs git tree on kernel.org, with a few patches I wrote almost two years ago, plus a recently found memory leak. I’ll try to get Linus to pull it in 2.6.36. Then, I made a new release of omfsprogs rewritten to use libomfs from the FUSE version of omfs (I also did this work two years ago). Now all is right in the Karma world again; time to ignore it for another 6 months.

Ath5k AP Mode

Sun, 01 Nov 2009 18:29:22 +0000

People seem to keep asking about this, despite there being a quality page on kernel.org on how to run an AP with any mac80211 driver. So for what it’s worth, here’s how my setup works. Note if you are seeing flakiness with certain clients, e.g. it works fine with a computer but not with your cell phone, it is likely there is some bug with the power saving handling. I’m currently working on a few such issues, so it may be fixed soon enough.

wl1251 performance

Tue, 18 Aug 2009 03:18:46 +0000

After fixing the remaining ifup bug (as expected, it was easy), I have some initial numbers on the new kernel driver versus the stock vendor driver on the G1: driveravg ping msnetperf mbit/stiwlan65.2317.53wl12518.5653.82 So, better on latency, worse on throughput. wl1251 is also quite a lot larger when taking all of cfg80211/mac80211 into account, though I didn’t spend any time trying to tweak the size in the build. Well, at least the code doesn’t make you want to poke your own eyes out.

Hang fixed

Mon, 17 Aug 2009 18:04:06 +0000

These sorts of bugs can ruin your weekend, just ask Ange who had to listen to me mope yesterday. Of course, I spotted it right away when looking at it freshly during this morning’s commute. Now, ifconfig wlan0 up; ifconfig wlan0 down; ifconfig wlan0 up still fails with wl1251, but it doesn’t hang and the rest looks tractable.

wl1251_sdio merged

Tue, 11 Aug 2009 03:27:23 +0000

The SDIO patches for TI 1251 (Android wifi chipset) are finally merged into wireless-testing, so they should be a lot easier to hack on now. That means the driver should make it into 2.6.32, though at a rather experimental stage. I did fix some crashes on ifup/ifdown since last posting, but there’s always more work to do. Current todo list includes better behavior for non-polling controllers (make the irq have a top-half), tracking down a device hang on reinitialization, pushing the msm_wifi.ko module, and on and on.

Now bionic

Sun, 19 Jul 2009 19:12:18 +0000

Kalle has posted a rebased version of my SDIO patches on top of the latest wireless-testing. I’ve done a little bit of testing with it – the driver starts up and loads fine, but once the interface goes down, it’s busted:

<6>[  980.884302] wl1251_sdio mmc0:0001:1: firmware: requesting wl1251-fw.bin
<6>[  981.036448] wl1251_sdio mmc0:0001:1: firmware: requesting wl1251-nvs.bin
<3>[  981.045449] init: untracked pid 354 exited
<3>[  981.074408] init: untracked pid 355 exited
<7>[  981.331853] wl1251: 151 tx blocks at 0x3b788, 35 rx blocks at 0x3a780
<7>[  981.339300] wl1251: firmware booted (Rev 4.0.4.3.5)
<7>[  984.702740] wlan0: direct probe to AP 00:1a:70:da:a9:cd (try 1)
<7>[  984.901621] wlan0: direct probe to AP 00:1a:70:da:a9:cd (try 2)
<7>[  985.101583] wlan0: direct probe to AP 00:1a:70:da:a9:cd (try 3)
<7>[  985.301617] wlan0: direct probe to AP 00:1a:70:da:a9:cd timed out
<7>[ 1024.802837] wl1251: down
<3>[ 1044.856640] init: untracked pid 358 exited
<3>[ 1050.033886] mmc0: Data timeout
<3>[ 1050.037701] wl1251: ERROR sdio write failed (-110)
<3>[ 1051.045674] mmc0: Data timeout
<3>[ 1051.049519] wl1251: ERROR sdio write failed (-110)
<3>[ 1052.057673] mmc0: Data timeout
<3>[ 1052.061518] wl1251: ERROR sdio write failed (-110)

Exploring this further turned into a rather long yak shaving session because I wanted to get iw on the phone to do more manual tests, and I wanted to link it dynamically against bionic, the Android’s fork of a BSD libc. Here were a few interesting discoveries found along the way:

Wifi UI Part 0

Mon, 13 Jul 2009 04:13:13 +0000

Work on the Android wireless driver is stalled for now as kvalo is rebasing the patches. I believe the next step will be to take the platform code, which I have temporarily built into a module called msm_wifiÂ¸ and turn it into a platform rfkill device. Or something like that – this is what he and I came up with at the wireless summit. Then there’s an annoying stack trace because free_irq() happens in common code, but sdio irqs are somewhat different so that needs to be pushed down into the interface layer.

ath5k performance history

Fri, 03 Jul 2009 17:35:04 +0000

I was curious how well ath5k has been improving since its release, so I hacked up a little script to associate and do five minute iperf runs from /etc/rc.local. Looks pretty good so far (w-t is wireless-testing plus 5 or 6 extra patches I have brewing). The script is pretty kludgy, but may form the basis for some automatic build/regression testing in the future.

Kernel 2.6.30

Tue, 16 Jun 2009 15:54:57 +0000

Linux kernel 2.6.30 is out, featuring the best mainline support for ath5k yet. I recommend if you do use wireless with 2.6.30 (or .29), that you also grab this patch which fixes a nasty memory corruption issue. That one haunted me for months but it was a great feeling to finally nail it down. Of course the fix turned out to be quite simple. It should be hitting stable in short order.

wl12xx howto

Sun, 31 May 2009 00:03:55 +0000

So, wl12xx definitely works including transmission, but I still haven’t fixed the rough edges in the driver. However, I did bother to make a semi-permanent writeup on how to build and use wl12xx on the G1 as it currently stands. I hope to clean up the warts and submit the patchset next week. Meanwhile, I have a lot of ath5k bugs to work on. Enough for today though.

First scan

Thu, 21 May 2009 19:36:00 +0000

Success! Scanning with wl12xx on my ADP1 works. I’ll post more later but here’s a teaser. That’s a real SSID, however apt.

# ./iwlist wlan0 scan
wlan0     Scan completed :
          Cell 01 - Address: 00:0F:B5:63:30:4E
                    Channel:11
                    Frequency:2.462 GHz (Channel 11)
                    Quality=14/70  Signal level=-96 dBm
                    Encryption key:on
                    ESSID:"SERENDIPITY"
                    Bit Rates:1 Mb/s; 2 Mb/s; 5.5 Mb/s; 11 Mb/s; 18 Mb/s
                              24 Mb/s; 36 Mb/s; 54 Mb/s
                    Bit Rates:6 Mb/s; 9 Mb/s; 12 Mb/s; 48 Mb/s
                    Mode:Master
                    Extra:tsf=000001dec907e41c
                    Extra: Last beacon: 160ms ago
                    (Unknown Wireless Token 0x8C05)

wl12xx update

Sat, 16 May 2009 21:42:54 +0000

I won’t claim success until I actually transfer data, but so far, it looks like wl12xx SDIO support is baked:

[  111.247063] cfg80211: Using static regulatory domain info
[  111.253472] cfg80211: Regulatory domain: US
[  111.262810] 	(start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
[  111.271568] 	(2402000 KHz - 2472000 KHz @ 40000 KHz), (600 mBi, 2700 mBm)
[  111.279472] 	(5170000 KHz - 5190000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
[  111.287254] 	(5190000 KHz - 5210000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
[  111.294975] 	(5210000 KHz - 5230000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
[  111.302482] 	(5230000 KHz - 5330000 KHz @ 40000 KHz), (600 mBi, 2300 mBm)
[  111.310264] 	(5735000 KHz - 5835000 KHz @ 40000 KHz), (600 mBi, 3000 mBm)
[  111.317894] cfg80211: Calling CRDA for country: US
[  112.684807] lib80211: common routines for IEEE802.11 drivers
[  112.690972] lib80211_crypt: registered algorithm 'NULL'
[  112.744713] lib80211_crypt: registered algorithm 'TKIP'
[  112.787467] lib80211_crypt: registered algorithm 'WEP'
[  112.845267] lib80211_crypt: registered algorithm 'CCMP'
[  113.251671] wl12xx: registering driver
[  113.256462] wifi_probe
[  113.259208] trout_wifi_power: 1
[  113.569452] trout_wifi_reset: 0
[  113.623681] trout_wifi_set_carddetect: 1
[  113.628290] mmc0: card_present 1
[  113.631921] mmc0: Slot status change detected (0 -> 1)
[  113.653283] mmc0: Command timeout
[  113.664788] mmc0: card claims to support voltages below the defined range. These will be ignored.
[  113.678429] mmc0: SDIO card claims to support the incompletely defined 'low voltage range'. This will be ignored.
[  113.692773] mmc0: new SDIO card at address 0001
[  113.702813] wl12xx: in probe
[  113.728844] phy0: Selected rate control algorithm 'minstrel'
[  113.738915] wl12xx: loaded
[  113.743919] wl12xx: initialized

Unfortunately, I can’t really test it yet because the wpa_supplicant already on the phone doesn’t support -Dwext or -Dnl80211, and there are no wireless-tools. So I need to cross-compile those before attempting to associate. In the meantime, here’s a very rough patch that implements the goods from wireless-testing wl12xx. I need to work on read() and write() - right now they are a copy of what is in the google driver because it appears the host driver has issues with RW_EXTENDED. I don’t believe the readb/writeb fallbacks should be strictly necessary. Ditto with the platform code, that needs a slightly better solution.

wl12xx merged

Thu, 30 Apr 2009 22:25:27 +0000

commit 00d8979d598d5461a06d800c779f15e03888d9d8
Author: Kalle Valo
Date:   Wed Apr 29 23:33:31 2009 +0300

    wl12xx: add driver

    wl12xx is a driver for TI wl1251 802.11 chipset designed for embedded
    devices, supporting both SDIO and SPI busses. Currently the driver
    supports only SPI. Adding support 1253 (the 5 GHz version) should be
    relatively easy. More information here:

    http://focus.ti.com/general/docs/wtbu/wtbuproductcontent.tsp?contentId=4711

    (Collapsed original sequence of pre-merge patches into single commit for
    initial merge. -- JWL)

    Signed-off-by: Kalle Valo
    Signed-off-by: Bob Copeland
    Signed-off-by: John W. Linville

Hooray, wl12xx is now in wireless-testing! As the commit log states, SDIO support isn’t there yet, because I haven’t written it yet. Well, I wrote some of it, but nothing worth even compile-testing yet.

Closer

Sun, 19 Apr 2009 16:12:19 +0000

I was in fact wrong that my self-compiled android kernels weren’t booting – they were. But they do fail somewhere in init:. Unfortunately, when you boot the phone, you don’t have any idea why it fails because there’s no console by default. It would’ve been a lot easier to see this if I had a serial cable, but I did finally remember that I could use fbcon with the smallest font and CONFIG_PRINTK_DELAY to see the messages scroll by. And so now I see the problem:

Emulator

Thu, 02 Apr 2009 03:01:59 +0000

The Android source code includes at least two ARM sub-architectures in their kernel tree. The one actually used on the G1 hardware is derived from the Qualcomm MSM machine type, at least part of which is in the mainline. Then there is “goldfish,” which appears to be just a pseudo machine used only for the emulator. While I haven’t had a lot of success with the MSM stuff booting on the phone so far, building the emulator kernel is not too hard once you know where to look:

2.6.30 preview

Wed, 01 Apr 2009 03:39:42 +0000

Now that Linux 2.6.29 just went out the door, here’s what I have prepared for 2.6.30 in the wireless arena. Of note, mac80211 now has suspend/resume support so a lot of nasty hacks in the drivers can go away. Most of the rest is just bug fixes, some of them for panics and soft lockups.

The work I’m excited about this week is scheduled for 2.6.31 – we extracted a bunch of ath9k regulatory infrastructure into a common module that all of the Atheros drivers (including the new ar9170) will use. This is the first step in getting a lot of that common code into a library. Hooray! Also Nick has some great improvements to the RF stuff queued up, so the driver is behaving much better.

Upstream Last

Tue, 31 Mar 2009 14:28:08 +0000

Building a booting kernel for Android using their own code still eludes me. I guess it would help if I knew anything at all about the ARM platform, as I have no idea what I am doing. Still, the code dump fails the “anyone should be able to build it” test rather strongly, judging from posts on the android mailing lists.

Build problems aside, the more egregious failure of the Android open source project is the lack of the “upstream first” mentality. Most Linux software providers follow this strategy to a large extent, which is simply: any patch that goes in my distribution should go to the mainline kernel first. The benefits are obvious: no need to maintain a fork forever, and everyone gets the goods as soon as possible. Even if the patch only serves the developer’s own interests, upstreaming it often has the benefit of vetting the interfaces and producing more universal APIs (the wakelocks discussion that followed the code dump is an example of this done backwards).

Fit to be TIed

Sun, 22 Mar 2009 15:55:31 +0000

The Android G1 phone uses the TI 1251 for Wifi.Â Â This is pretty useful for when you need an internet connection for the laptop and there are no alternatives: you can bridge the wifi and the 3G connection and then connect to the wifi device using ad-hoc mode.Â However, I normally keep the wireless off because it sucks down battery like mad.

But Kalle Valo from Nokia is working on a driver for the chip called wl12xx, and he is putting a lot of effort into making sure it uses power conservatively (presumably some future version of the Nokia internet tablets will be using this device).Â This driver is SPI, whereas the G1 uses the same chip in SDIO mode.Â So I started writing the SDIO port for the driver – hopefully it will be ready when wl12xx is included in mainline.

Serial offender

Sun, 08 Feb 2009 14:27:00 +0000

There comes a time in every budding kernel developer’s life that he has to debug a mysterious lockup, and nothing will do but a serial console. Well, for my future recollections, here’s how to set it up:

Get out your handy pl2303-based usb to serial adapter, because chances are good your laptop doesn’t have a serial port
Build your kernel with CONFIG_USB_SERIAL_CONSOLE=y
Add to your kernel command line: console=ttyUSB0,115200 console=tty0
Hook your computer up to the other computer via a null-modem cable (man, these are pricey these days, $30 for something no one still uses?)
Set up minicom to use your serial port, say ttyS0, at 115200 baud, 8N1, and turn off all the modem init strings
Don’t bother futzing with getty, you only need it if you want to also allow logins over serial. For logging, it’s unnecessary

Hacking, the good kind

Wed, 05 Nov 2008 15:29:00 +0000

I could write about the election here, but citizen905 already summed it up pretty well. So instead, here’s what I’ve been breaking in the Linux kernel lately:

My final patch count for 2.6.27 was 14, I think. Enough, anyway, that I can stop counting and just deal with all the work I’ve created for myself.
I added myself to MAINTAINERS for ath5k, which felt like a pretty ridiculous notoriety grab, but Nick asked me to do so twice, so there.
I have some fixes for ath5k for 2.6.28, nothing major but an oops should be fixed, and a WARN_ON removed. The oops fix, incidentally, had an obvious bug despite 3 sign-offs. I suck.
Also committed but to-be-reverted for suckiness is a patch to remove beaconing in STA mode. Turns out ath9k, from which I stole this idea, was just busted. The new plan is to use the beacon miss interrupt; until then, your wireless card has to wake up the CPU about 100 times a second.
For 2.6.29, I have added hardware encryption to ath5k and hopefully will get some time to hack on the suspend/resume support for mac80211. Then I have some omfs patches I’ve been sitting on for months.

SYSRQ on MacBook

Tue, 07 Oct 2008 21:42:00 +0000

Lately I’ve really needed SysRq in situations where /proc/sysrq-trigger just doesn’t do the job, and my MacBook is missing lots of crusty old XT-era keys. Finally, I know how to do this!

/* includes and error handling omitted for brevity... */
#define USAGE_CODE 0x070044 /* USB hid for F11 */

int main()
{
int codes[2];
int fd = open("/dev/input/by-id/usb-Apple_Computer_Apple_"
"Internal_Keyboard_._Trackpad-event-kbd", O_NONBLOCK);

codes[0] = USAGE_CODE;
codes[1] = KEY_SYSRQ;  /* from linux/input.h */
ioctl(fd, EVIOCSKEYCODE, codes);
}

Awesome. Supposedly, a tool called keyfuzz is also efficacious.

OSS, I has it

Wed, 01 Oct 2008 14:27:00 +0000

I just sat in on a conference call as a representative (by default, since no one else called in) of the Linux ath5k community, with Atheros, makers of my MacBook’s wireless ethernet card. Atheros have really done a 180 for supporting the community, first by releasing ath9k, then by releasing the source to their previously-closed HAL last week. Thanks to that, 6 patches have already gone out fixing various problems. BTW, conference calls are just as pointless in the OSS community as they are in real life. But at least I did learn that it is pronounced “uh-THERE-ose”, not “ATH-er-ose.”

Oops

Tue, 12 Aug 2008 12:07:00 +0000

I am finally getting the hang of debugging kernel crashes. None too soon as I got my first OOPS report from the -rc kernel with OMFS, from a gentleman who is intentionally corrupting his FS (“fuzzing” in the infosec lingo). After a frustrating weekend in which I had inadvertantly fixed the bug but didn’t realize it because I was testing the wrong module, I can now claim success. One down, several more to go.

Meh

Tue, 13 May 2008 13:48:00 +0000

I’m playing with date conversions today, and again I’m struck by how much the Java Calendar should be held up as an example of the over-engineered API. Has anyone ever used anything besides the Gregorian calendar? They were so proud of it when it hit 1.1.

I should have two patches hitting kernel 2.6.26, one entirely cosmetic and one that fixes a real bug on Atheros wireless cards. Akpm did pick up the OMFS patchset so hopefully that will go in .27 timeframe, though the jury is still out on whether it hits mainline.

Controversy, I has it.

Mon, 14 Apr 2008 21:34:00 +0000

I unintentionally created quite a ruckus on the kernel mailing list this weekend. Evgeniy’s take. It’s funny, I feel a lot less invested than practically anyone in the whole thread.

mmio trace for fun and no-profit

Fri, 01 Feb 2008 03:06:00 +0000

I’m not sure what got me interested in assembly language as a nerdy high-schooler. It could have been the growing interest in computer graphics, at a time when you had to use assembly to get decent performance out of the machine. I remember learning tricks from much smarter people than myself, such as how to set a 256x256 pixel video mode so that you could address any pixel without ever needing to multiply. Not that you would use a multiply anyway since you can always use shifts and adds.