Crossword digest

This month my age incremented again. On the very anniversary of my birth, I received an email from Will Shortz’s assistant, saying (nicely) that the puzzle I submitted back in November didn’t make the cut. I was prepared for this: a puzzle with a similar theme came out a couple of weeks after I mailed mine in, and well, it’s not exactly that unique in the first place. Also the middle section of my puz is a bit of a stretch. And, I felt bad about not trying to excise 23-A from the puzzle.

But anyway, their loss is your loss! Click here to try it.

The 2018 American Crossword Puzzle Tournament starts this coming weekend. Last year, I complained that the online crossword tournament used a Java applet and would have almost no browser support by this time. This year, the online solver is running a version of my own XwordJS, customized (by me) for the ACPT’s backend (which I had nothing to do with). Here’s hoping it works.

After the 2017 tournament, I hoped to increase my solving speed to be more competitive this time around. In reality, I didn’t put much effort into this goal, so I expect to be about the same. My best NYT Monday time, for that one month when I tracked it, was a hair over 7 minutes, far from the front of the pack.

Meanwhile, I did make a bit of progress in solving cryptics, having done the puzzle a few times in the Toronto Star and actually finishing last week’s. I have to say they aren’t as fun as American style crosswords due to the arbitrariness of the cluing, or so I tell myself so that I can feel less dumb.

Making copies

One of my early goals this year has been to revamp the backup regime at the old homestead. Previously, I have relied on rotating external drives connected to my main desktop, and putting most content on a samba share that also got backed up. But it was a bit ad-hoc and things not on the share only got backed up sporadically. I’d rather not use a cloud backup service because reasons, so I bought a NAS, and now it looks like this:

  • Daily:
    • download cloud assets (google photo)
    • backup all disks to NAS (borg backup on Linux, Windows backup). Incrementals of 7 days / 4 weeks / 6 months
    • backup NAS to external drive (rsync)
  • Weekly:
    • borg check the latest backup
    • swap external disk with one in fire safe
  • Monthly:
    • swap external disk with off-site disk

I think this should work well enough until I use up enough storage to outstrip the individual external disks, then I’ll have to rethink things. Too far, or not too far enough?

New Year, Old Me

2018 is finally upon us, and now is the time one would usually look back at the wonderful events of the previous year, and look forward to what great things one might accomplish in the next.

Instead of that, I’ll just write a meandering blog post.

In 2017, I leveled up on pizza and bread making, capturing and raising my very own colony of hyper-local, non-GMO, swimming-in-gluten yeast. The colony still thrives as it approaches its first birthday, and has fed us many delicious loaves. I think I have the basic no-commercial-yeast dough down now, so it is time to start tinkering with the bulk fermentation and proofing times to get different flavors and textures out of the bread.

Speaking of cooking, after 20-something years of preparing the same three proteins with not so much variation, I feel like I could use more inspiration in the kitchen. So I’ve started working my way through an Asian cookbook someone gave me for Christmas. Tip: homemade Thai sweet chili sauce (using peppers from our garden) turns out to be amazing on roast beef and aged cheddar sandwiches. Who knew?

We’re still working our way through the garden vegetables in the freezer, so I can say that the garden worked out and I will have to do it again this year. This time I probably won’t plant so many tomatoes right next to each other, and I think I’ll select pepper varieties rather than doing a mixed seed blend. I might play around a bit with controlling irrigation through an SBC or some such just to bring some nerdiness to the party.

Six years into my consultancy, things are going well enough, but this year I may start looking around for the next big gig. Lately I’ve been doing mostly infosec work and it is ridiculously draining in the age of we-can’t-even-get-hardware-right. If you’re one of the three people reading this and whose programming team needs an old, feel free to reach out. Bonus points if you are working in a language that has a compiler, and infinity points if you are sending code into space.

The crossword construction hobby grew a bit in 2017, culminating in my submitting a crossword to the NYT for publication. That one will probably get rejected, but even so it was fun to make (and perhaps as fun making the software to make it). I think with just a bit more practice and some theme inspiration, I’ll have the hang of this nascent hobby. Meanwhile, my javascript puzzle solving SPA is getting used in a few more places, including on some commercial puzzle sites; I hope that trend will continue.

Of course the biggest news in 2017 was growing our family once more. In addition to the usual resolution to be the best parent I can be to Alex, Ian, and now Sam, I also hope we can get more sleep in 2018. That would be kind of great.

Router redux

I had just a little bit of downtime over the last few weeks while Angeline and I reacquainted ourselves with how to take care of an infant. It’s like riding a bicycle: there are tons of things you forgot since last time.

One of the long-on-my-todo-list items finally got completed: I upgraded all of my wifi routers. I have 5: an old dual band 11n router and four TP-Link 11ac units, all of which were running some oldish build of OpenWRT.

I decided the 11n router’s time has come for the trash-heap, so I put the latest stable LEDE build on one of the TP-Links and swapped it out.

The other three are just access points without the router: they just have all the ports on the unit bridged together, connected to the router via a wired switch. I also have a mesh wifi interface up on each unit so that I could place the unit anywhere regardless of wired connectivity (though, in practice, I have wired drops everywhere so I don’t really use this.) For these, I build from source with just the required bits. I added a serial port to one of the units so I can test builds there before rolling out to the other two.

In all it was pretty painless since the LEDE build is more or less the same as OpenWRT. I did go through (LEDE) recovery once and found this fun issue:

root@(none):/tmp# sysupgrade -n lede-ar71xx-generic-archer-c7-v2-squashfs-sysupgrade.bin
Image metadata not found
killall: watchdog: no process killed
Commencing upgrade. All shell sessions will be closed now.
Failed to connect to ubus

…because sysupgrade has different paths for failsafe vs not; and for some reason $FAILSAFE is not always set. Do this to work-around:

root@(none):/tmp# export FAILSAFE=1
root@(none):/tmp# sysupgrade -n lede-ar71xx-generic-archer-c7-v2-squashfs-sysupgrade.bin
Image metadata not found
killall: watchdog: no process killed
Commencing upgrade. All shell sessions will be closed now.
root@(none):/tmp# Connection to closed by remote host.

On board the rebus

My most recent post, for those too lazy to follow the link and hit the “Reveal All” button, is a crossword puzzle announcing the birth of our third son, Samuel Yit-Mun Copeland. We’re over the moon with this new guy in our family. Here he is nearly asleep in my arms as I type this.

But now for something more boring.

I decided months ago that my typical geeky announcement would take the form of a crossword since I’ve been playing in that area recently (previous announcements: a manpage for Alex, an ascii-art crypto program for Ian). And why not make this puzzle interesting by constructing my first rebus puzzle?

A rebus puzzle is one in which a single square can have multiple letters. Part of the fun of solving is realizing at some point that an answer must be right, yet it cannot fit the available cells, so something really weird is going on. Then one curses the constructor and starts with a new set of assumptions.

Constructing such a puzzle is not too much harder than any ordinary puzzle, since all the rebus squares will be part of the themers that the constructor selects up front. Because of this, the fill software doesn’t even really need to care about rebuses — one can use any placeholder for the rebus squares and then edit the final crossword file to add in the full string.

Having lots of theme squares does constrain the grid somewhat, as does having a number of circles on top of that (in this case the circles spell out our new son’s name). But I cheated a little bit: I decided having any old separation between circles was fine as long as the components were all on the same line, and his name is short so that was no major obstacle. In the end I’m pretty happy with the fill considering the 55 theme squares + 20 circles.

Each time I make a puzzle, I also make a number of improvements to my filling software. I’ve long since forgotten whether the goal is building puzzles or building a puzzle builder.

I mentioned before that I had optimized the javascript filler to the point that it was potentially usable in a webapp; this webapp is now a thing. For this puzzle, I used that app as well as my interactive python command-line filler, which has some additional features like estimating the number of eventual solutions using more levels of look-ahead. Finally, I also used a version of the python filler that will take a completed grid and try to optimize it by repeatedly clearing sections of the grid and refilling it randomly.

There is still a good deal of performance work to do on the filler itself. The basic structure is still similar to the backtracking algorithm in QXW: bitmaps are used to filter available words for each cell, and we fill each entry starting with the entries with fewest fills. I made two minor changes in my version: the word list is initially sorted by score and indexed on length so that we can try highest scoring words first and eliminate comparisons against words that will never fit; and entire entries are filled instead of single cells so that score of an entire word is maximized.

I implemented my current algorithm from scratch in C, and it clocks in under half a second for my test puzzle: a little faster than QXW’s implementation and some 30x faster than the very same algorithm in javascript. In terms of LOC, they are roughly the same: 573 (python), 696 (javascript), 763 (C). You can watch them compete below.

So the javascript implementation is frightfully slow. Getting it down to the level of, say, two times the C code, would be a major win, but beyond my current level of js optimization know-how.

Besides that, there is likely some algorithmic optimization that could bring larger gains. Knowing that the “crossword fill problem” is an instance of SAT, I went off to read what Knuth had to say about it (Vol 4 fascicles 5-6), and, as usual, came away with a lot more to think about. The “word square” section is very similar to crossword filling, and indeed the algorithm is close to my existing search, except that it uses trie data structures to cut down on the number of possible words visited at each level. It’s unclear to me at present whether tries would be a win here given that we don’t always start filling with a prefix or suffix, but it could be worth an (ahem) try. Also I may look at branch-and-bound search pruning and parallel search at some point.

In which I ate the garden

The garden is now in its 16th week, I think. Despite the fungus problem with the tomatoes, I have to say it has been a fairly successful experiment so far: the chilies and basil have done great, while the tomatoes have still produced over a dozen softball sized fruits with lots more still on the vine. The ones in containers did less well (blossom-end rot) — I think there’s no use in venturing outside the planter box next year.

The chilacas have been the big producer. I have two plants, including one that hadn’t yet set fruit by the end of July. That latter since became top-heavy enough with peppers that a couple of stems broke off in high winds last night, so I hauled in around 20 peppers this morning. Salsa-to-be.

Some chilies

Anyway, this is what we’ve eaten thus far:

  • Tomatoes, by themselves
  • BLTs, of course
  • Caprese salad
  • Chicken with sauce chasseur
  • Sausages and mushrooms with fresh tomato sauce
  • Pepper steak (bell peppers, poblanos)
  • Pesto, normal and spicy with Hungarian peppers
  • Chilaca cheese dip (chilacas, yellow jalapeno[1] and poblanos)
  • Poppers (cherry peppers, chilacas, and Hungarian peppers)

The only fruit we haven’t meaningfully consumed in some way are the Thai chilies, which are incredibly hot. I’ll probably make those into some form of sriracha or chili paste.

[1] I think? Middle of the above photo — not really sure what these chilies are but they look similar to jalapenos on the outside and turn red when ripe, but the similarities largely end there.

Fill faster

I’ve not done much work lately on my forked-from-QXW, written-in-C crossword filler, as getting that code up to snuff amounts to a rewrite. But one thing it has going for it is that it is very fast.

Meanwhile, I have my own written-from-scratch-in-python filler that I’m pretty happy with in terms of results and features, but which is no speed demon. And then I wanted to have this all work on my phone, so I re-implemented it in javascript. Yes, I’m writing a lot of code lately in this language that I do not care for.

Anyhow, down this rabbit hole I go, and one thing that is clear is that the interpreted language versions are not fast. Representative times to fill a grid:

cpython:    114 s
node js:    49 s
pypy:       26 s
C (qxw):    0.5 s

Amazing how much faster pypy is compared to CPython running the same code!

Well, I spent an afternoon optimizing the python and JS fillers, and got a decent speed-up:

pypy:       5 s
node js:    2 s (24x improvement)

Not bad, and in the realm of being usable for an online filler app. Much of the speedup came from avoiding common interpreted language constructs (like regex) that one would never go near in C.

I cobbled together enough of a web UI for this in order to fill this new puzzle on my phone while the 4 year old was having a long snooze in my lap. Guess what 20-Down we’ve been shopping for lately?

Garden Week 7

The garden is now jungle-like. There are real concerns that people have wandered into the garden and never returned, having lost their way out. I have a little bit of black spot on pretty much every tomato plant. My fault, I should have pruned the tomatoes more. I sprayed them with fungicide.

Even so, they have set fruit. I have a few per plant, maybe twenty in all.

Oh, I identified, and ate, the yellowish-green chili from last posting: the Hungarian wax pepper. Great sauteed in butter and spooned over salmon.

Garden update

My garden experiment continues, mostly successfully so far. Here are my notes for how it’s worked out, to remind myself what to do next year.

I built a 6′x4′ cedar box from 1″x6″ planks, all cut to nearly the right length by the helpful folks at the local home center. Some stainless steel decking screws and nearly a ton of VegMax(tm) dirt later, I had my raised bed set up.

While hardening off, the seedlings were getting watered about once every 1-2 days, with 4 cups of water per 10 seedlings. By May 28th I had them all planted out in the raised bed, staked, and mulched. At that time they looked like this:

I put a pair of tomato fertilizer spikes about 6 inches from each plant that same week. Since then I have been spraying with epsom salt every two weeks, and since last week I have started alternating that with fertilizer. The bugs are numerous but I’ve used insecticidal soap sparingly: so far, there doesn’t seem to be a ton of insect damage except on lower leaves. I haven’t pruned the tomato plants that much, just leaves that are obviously low and touching soil. As this has been a very wet spring so far, I’ve only had to water them once since planting (15 min on soaker hose).

One of my peppers (serrano? yellow cayenne?) has yielded fruit. I’m not sure what the final color is supposed to be, but I may pick it soon anyway to speed along some other buds on that plant. I also have a 1″ bell pepper on another plant. Part of the fun of using a mix of pepper seeds is that I have no idea which plants are going to produce what.

I lost one pepper seedling shortly after planting due to high winds snapping the stem in half.

The basil is doing fine, and much more prolific outdoors compared to my indoor plants.

My largest tomato plant got its first flowers yesterday. I’ve restaked some of them with 6′ spikes as they’ve outgrown the 2′ spikes already. One definite mistake I made was to plant the tomatoes with only 1′ spacing. I moved three of my smallest plants yesterday to containers so that at least one or two of the remaining garden plants will have the proper spacing. The others will fight it out, so we’ll see how that goes.