Exede Surfbeam 2

My new satellite internet connection is from Exede, connecting to the ViaSat 1 bird in geosync orbit. A few technical details that I've observed follow.

antagonistic by design

The "Surfbeam 2 wifi modem" is a closed proprietary system. That is important because it's part of Exede's bandwidth management system. The Surfbeam tracks data use and sends it periodically to Exede. When a user has gone over their monthly priority data, Exede then throttles the bandwidth in various ways -- this throttling seems to be implemented, at least partially on the Surfbeam itself. (Perhaps by setting QoS flags?)

So, if a user could hack their Surfbeam, they could probably bypass the bandwidth caps, or at least some of them. Perhaps Exede would notice eventually. Of course, doing so would surely violate the Exede TOS. If you're renting the modem, like I am, hacking a device you don't own might also subject you to criminal penalties. Needless to say, I don't plan to hack the SurfBeam. But it's been hacked before.

So, this is a device that lives in people's homes and is antagonistic to them by design.

weird data throttling

The way the Surfbeam reports data use back to Exede periodically and gets throttling configured has some odd effects sometimes. For example, the Surfbeam can be in throttled state left-over from the previous billing month. When a new billing month begins, it can remain throttled for some time (up to multiple hours) until it sends an update to Exede and they un-throttle it. Data downloaded at that time might still be counted as priority data even though it was throttled. I've seen some good indications of that happening, but am not sure yet.

But, I've decided that the throttling doesn't matter for me. Why? ViaSat 1 has many spot beams, and the working-class beam I'm in (most of it is in eastern Kentucky) does not seem to get a lot of use between 7 am and 4:30 pm weekdays. Even when throttled, I often get 300 kb/s - 1 mb/s speeds during the day, which is not a lot worse than the ~2.5 mb/s peak when unthrottled. And that's the time when I want to use broadband -- when the sun is shining and I'm at home at work/play. I'm probably going to switch to a cheaper plan with less priority data, because the priority data is not buying me much. This is a big change from the old FAP which rendered the satellite no faster than dialup.

a whole network in there

Looking at the ports open on the Surfbeam, some very strange things turned up. First, there are not one, not two, but three separate IPs used by the device, and there are at least two and perhaps three distinct computers involved. There are a lot of flickering LEDs inside the box; a whole network in there. is the satellite controller. It's a Linux box, fingerprinted as kernel 3.10 or so (so full of security holes presumably), and it's running thttpd/2.25b (doesn't seem to have any known holes). It seems to have ssh and snmp, but with some port filtering that prevents access. (Note that the above exploit video confirms that snmp is running.) Some machine parsable data is exposed at and (See (SurfStat program) is the wifi router. It has a dns server, an icslap proxy, and nmap thinks it's Linux 3.x with Synology DiskStation Manager (probably the latter is a false positive?) It has its own separate web server for configuration, which is not thttpd. I'm fairly sure this is a separate processor from the other IP address. responds to ICMP, but has no open ports at all. However, it seems to have filtered ssh, telnet, msrpc, microsoft-ds, and port 8090 (probably http), so perhaps it's running all that stuff. This one is definitely a separate processor, located in the Satellite dish's TRIA (transmit receive integrated assembly). Verified by disconnecting the dish's coax cable and being unable to ping it.

lack of source code for GPLed software

Exede did not provide anything to me about the GPL licensed source code on the Surfbeam 2. I'm not sure if they're legally required to do so in my case, since they're renting it to me?

But, they do let you buy the device too, so it's interesting that nothing mentions the licenses of its software and where to get the source code. I'll try to buy it and ask for the source code eventually.

power use

The Surfbeam pulls as much power as two beefy laptops, but it is beaming signals thousands of miles into space, so I give it a bit of a pass. I want to find a way to power it from DC power, but have not had any success with that so far, so am wasting more power to run it on an inverter.

Its power supply is rated at 30v and 2.5 amps. Interestingly, I've seen a photo of another Surfbeam power supply that only pulls 1.5 amps. This and a different case with fewer (external) LEDs makes me think perhaps the installer stuck me with an old and less efficient version. Maybe they integrated some of those processors into a single board in a new version and perhaps made it waste less power as heat. Mine gets pretty warm.

I'm currently only able to run it approximately one hour per hour of sun collected by the house. Still on dialup the rest of the time. With the ability to get on broadband when dialup is being painful, being on dialup the rest of the time is perfectly ok. Of course it helps that with tools like git-annex, I'm very well tuned to popping up onto broadband briefly and making it count.

Joey chatter

I guess if I really care about BW, I'll make it try a non-http connection and fall back to http + websockets..

Joey chatter

Really annoying thing is, I'm only using http because of port blocking. We can tunnel anything over http websockets, which makes port blocking irrelevant, at a cost of many extra bytes. And of course that reduces pressure for ISPs to not block ports. Progress.

Joey chatter

"like using a lunchbox to make lunch"

Joey chatter

Wireshark says yes.. A 183 byte websocket frame contains 95 bytes or so of data. And then there are some 86 byte TCP ACKs..

Joey devblog
debug me finalizing wire format

Went ahead and made debug-me use protocol buffers for its wire protocol. There's a nice haskell library for this that doesn't depend on anything else, and can generate them directly from data types, but I had to write a shim between the protobuf style data types and debug-me's internal data types. Took 250 lines of quite tedious code.

Then I finally implemented the trick I thought of to leave out the previous hash from debug-me messages on the wire, while still including cryptograhically secure proof of what the previous hash was. That reduced the overhead of a debug-me message from 168 bytes to 74 bytes!

I doubt debug-me's wire format will see any more major changes.

How does debug-me compare with ssh? I tried some experiments, and typing a character in ssh sends 180 bytes over the wire, while doing the same in debug-me sends 326 bytes. The extra overhead must be due to using websockets, I guess. At least debug-me is in the same ballpark as ssh.

Today's work was sponsored by Riku Voipio.

Joey chatter
websocket overhead

Does sending a message to a websocket really use 250 bytes of bandwidth for websocket stuff? Seems to in my tests..

Annoying ... I've relentlessly whittled my protocol down to 75 byte packets from 170+ using some fairly serious hackery. And now all this additional overhead!

Anna (Anna and Mark: Waldeneffect)
Hummingbird on a grape trellis

Although we did eventually make it across the creek Wednesday, our first attempt was a failure.

Homemade paddle"If I just had a paddle, I think it would be safe to kayak across the floodplain while the water is so dispersed," I told Mark Wednesday morning. Within fifteen minutes, he'd created me a paddle out of a flipper (thanks, Rose Nell!), a furring strip, and some duct tape.

Unfortunately, when we embarked on our adventure, Mark immediately saw the flaw in my plans. Despite flood waters receding about three vertical feet in the last twenty-four hours, I still couldn't reach the kayak using hip waders.

All it took was a little more time, though, to achieve our goal. By 1 pm, the creek had gone down another foot and Mark and I together were able to retrieve the kayak we'd stashed by the creek. He hoisted it into a tree to empty out the water (a difficult feat when knee-deep in the drink), and we even discovered that we'd been smart enough to leave a paddle stashed inside. Maybe next time we'll get yet smarter and park the flood-water transportation device by the barn!

Joey devblog
debug me polishing

Working on polishing up debug-me's features to not just work, but work well.

On Monday, I spent much longer than expected on the problem that when a debug-me session ended, clients attached to it did not shut down. The session shutdown turned out to have worked by accident in one case, but it was lacking a proper implementation, and juggling all the threads and channels and websockets to get everything to shut down cleanly was nontrivial.

Today, I fixed a bug that made debug-me --download fail while downloading a session, because the server didn't relay messages from developers, and so the proof chain was invalid. After making the server relay those messages, and handling them, that was fixed -- and I got a great feature for free: Multiple developers can connect to a debug-me session and all interact with it at the same time!

Also, added timing information to debug-me messages. While time is relative and so it can't be proved how long the time was between messages in the debug-me proof chain, including that information lets debug-me --download download a session and then debug-me --replay can replay the log file with realistic pauses.

Started on gpg signing and signature verification, but that has a user interface problem. If the developer connects after the debug-me session has started, prompting the user on the same terminal that is displaying the session would not be good. This is where it'd be good to have a library for multi-terminal applications. Perhaps I should go build a prototype of that. Of perhaps I'll make debug-me wait for one developer to connect and prompt the user before starting the session.

Anna (Anna and Mark: Waldeneffect)
April flood 2017
Raindrops on oxalis

This week's flood didn't quite match our biggest one ever, but it came close. Over five inches during a week (half in the last twenty-four hours) is nothing to sneeze at.

Flooded floodplain

The floodplain is so far underwater that it feels like one huge, still lake.

Root waterfall

Mark was surprised that the smaller creek didn't roar as loudly as it usually does during flood times. After some reconnaissance, I figured out why --- the part of the creek closest to our core homstead is part of the lake. I had to walk a ways upstream to find flowing water.


In the process, I discovered a very heartening sight --- rocks! This small creek, for some reason, had worn itself down to bare mud Creek rocksover most of its length long before we moved in. The result is relentless erosion that keeps cutting the bottom of the creek deeper every year.

Not so any more, at least for the upper expanses. Rocks as big as my fist were carried down by the floodwaters and deposited in bends and pools. Maybe another big flood or two will fix the bottom of the downstream portion of the creek as well?

Wet robin

Anyone want to take bets on how soon the floodwaters recede sufficiently to let us bring home some fresh groceries? Good thing the freezers are still pretty full!

Anna (Anna and Mark: Waldeneffect)
Wheelchair chicken tractor update
Chicken tractor

Even though we knew it would slightly overload the new chicken tractor with docking run, we put all nine coop hens inside so we wouldn't have more than two chicken areas to monitor each day. That was a bit rough on the flock since the skies opened up and proceeded to pour for the next several days, but the girls seem to have stayed pretty dry while coming out to graze between showers.

The roll-out nest box, to our surprise, has been the least successful part of the undertaking. Without straw or a nest egg in the box, most of the hens have been ignoring the structure and laying on the ground instead. Mark's thinking of glueing a golf ball inside in hopes we can talk the hens into utilizing the roll-out option. In the meantime, we've been gathering muddy eggs out of the yard.

Joey chatter
close squeeze

After detouring around the flood this morning, my detour was almost flooded this evening. Only one lane of dry highway left.

Anna (Anna and Mark: Waldeneffect)
Spring leaf medley
Wet grapes

At a certain point, the world is so wet that even Edgar doesn't want to go out and graze. So the herd calls in takeaway.
Blurry goats
"I'd like a spring leaf medley," Aurora told me. "Some elderberry, some pear prunings, a bit of willow, maybe a small side order of honeysuckle and buckeye."

"And make it snappy," Edgar added. Did I really hear him mutter "It's hard to find good help nowadays" under his breath?

Joey chatter
nonflood flooding

"Nonflood flooding is occurring and minor flooding is forecast" -- NWS bulletin

Anna (Anna and Mark: Waldeneffect)
The timeline of a spring rain
Roasting asparagus

On the sixth straight day of rain, the creek finally started to rise. The asparagus responded a few days earlier, beginning to provide whole meals of spears around day three. Tenacious weeds loosened their grip on the earth on day four. And on day five, trees and bushes sudden seemed twice as green.

Gotta love spring rains after a dry spell!

Joey devblog
debug me client-server working

Got debug-me fully working over the network today. It's allllive!

Hardest thing today was when a developer connects and the server needs to send them the backlog of the session before they start seeing current activity. Potentially full of races. My implementation avoids race conditions, but might cause other connected developers to see a stall in activity at that point. A stall-free version is certianly doable, but this is good enough for now.

There are quite a few bugs to fix. Including a security hole in the proof chain design, that I realized it had when thinking about what happens whith multiple people are connected to a debug-me session who are all typing at once.

(There have actually been 3 security holes spotted over the past day; the one above, a lacking sanitization of session IDs, and a bug in the server that let a developer truncate logs.)

So I need to spend several mode days bugfixing, and also make it only allow connections signed by trusted gpg keys. Still, an initial release does not seem far off now.

Anna (Anna and Mark: Waldeneffect)
Homestead mower selection for women
Honda mower

Now that everyone's done drooling over Mark's new mower, I thought it might be helpful to share our thought processes in choosing the new tool. A homestead like ours, with grassy aisles between permanent garden beds covering a couple of acres, needs a mower small and nimble enough to dive into nooks and crannies...and preferably hefty enough to cut thick weeds and power up slick hills without burning out or breaking its handlers. Here are the mowers we've considered and/or tried.

Craftsman mowerThe Craftsman 917388571 was our farm's second mower (replacing an ancient mower that literally exploded one day when I asked it to do things mowers shouldn't be asked to do). What I loved about this mower: it ran with minimal upkeep for eight years after being purchased used and was easy enough for me to start until the final season. What we didn't like so much: pushing it up hills and around corners could wear you out, especially in the heat. Meanwhile, as the Craftsman aged it seemed to go through flywheel shaft keys and blades like candy despite there being fewer and fewer obstructions left in its path.

Swisher mowerIn fall 2014, I loosened the purse strings and we upgraded to a Swisher. Mark was hoping that the string cut would handle our rough terrain without so many flywheel problems and he also thought a self-propelled mower would make grass-cutting a breeze. I'll admit that I detested this mower from day one. The string flung blades of grass into my garden beds, requiring much more washing of our harvest (and making leaf lettuce completely unpalatable), and the string also resulted in a rougher cut that wasn't fun on bare feet. I couldn't start the machine either, so when it was my turn to mow I stuck with the Craftsman.

Troy-Bilt mowerOnce Mark gave up on the dream of the Swisher, we put our heads together and tried to figure out if there was a mower that would do what we each wanted. At first, we considered a Troy-Bilt WC33. The price tag was daunting, but such a hefty machine seemed like it might be a good idea, especially with the electric start, self-propulsion, and excellent reviews. But then I read deeper and realized that there was no way I could manage the beast with my moderate upper-body strength. Luckily, Mark was willing to compromise.

Self-propelled mowerWe settled on the Honda HRX217K5VKA because it looked small enough for me to maneuver and boasted the best reviews of any of the same-class mowers. After taking it for a test drive, I have to admit, I'm sold. The auto-choke, easy-start system almost seems like a magic trick --- I've never had a two-stroke engine that I could rev up without even feeling it in my yanking arm. The self-propulsion really makes a difference in our hilly terrain (yes, Mark was right about that), and at the same time the design makes it easy to go manual in extremely tight quarters.

Of course, to pay for itself, the Honda will need to last at least a decade. Here's hoping our new mower will go the distance!

Joey devblog
debug me websockets

Worked today on making debug-me run as a client/server, communicating using websockets.

I decided to use the "binary" library to get an efficient serialization of debug-me's messages to send over the websockets, rather than using JSON. A typicaly JSON message was 341 bytes, and this only uses 165 bytes, which is fairly close to the actual data size of ~129 bytes. I may later use protocol buffers to make it less of a haskell-specific wire format.

Currently, the client and server basically work; the client can negotiate a protocol version with the server and send messages to it, which the server logs.

Also, added two additional modes to debug-me. debug-me --download url will download a debug-me log file. If that session is still running, it keeps downloading until it's gotten the whole session. debug-me --watch url connects to a debug-me session, and displays it in non-interactive mode. These were really easy to implement, reusing existing code.

mark (Anna and Mark: Waldeneffect)
Happy grazing

Wheelchair chicken tractor day one.
The wheelchair chicken tractor chickens seem to like their new home.

They spent the morning testing any possible escape routes and settled down to happy grazing in the afternoon.

Anna (Anna and Mark: Waldeneffect)
Homemade potting up mixture
Potting up

What better gardening activity is there for a rainy day than potting up? I guess most people don't actually do the work in the rain....

Mixing potting soil

Even though I've changed my loyalties from stump dirt to storebought potting soil for the starting stage, I'm still content with homegrown options for potting up. Half stump dirt and half well-aged horse manure gives hungry plants plenty of nutrients while waiting for frost danger to pass. And, at this age, there's no worry left about damping off.

mark (Anna and Mark: Waldeneffect)
Honda mower day one
Honda 190 lawn mower day one.

The new Honda mower was easy to set up and even easier to start.

Anna got it going on the first pull with little effort.

The bigger wheels and extra power is a good match for our overgrown patches of weeds we like to call a lawn in between sections of the garden.

Anna (Anna and Mark: Waldeneffect)
Transplanted blueberries
Developing gooseberries

When I moved the surviving blueberries and gooseberries downhill from the ultra-shady spot where they originally lived, I was a bit concerned that such mature bushes wouldn't survive being transplanted. So I pruned them harder than usual, removing perhaps a third of the top growth. Then I waited to see what would happen.

Blueberry blossoms

So far, so good. The plants old enough to bear fruit are indeed blooming copiously and the tender new leaves show no problems yet. It'll be awfully nice to have the fruit so close to the back door for daily picking, and I can already tell I'm going to get three times the pleasure despite cutting the planting size back by a factor of four. Now that's math I can get behind.

Joey devblog
debug me signatures

Added signatures to the debug-me protocol today. All messages are signed using a ed25519 session key, and the protocol negotiates these keys.

Here's a dump of a debug-me session, including session key exchange:

{"ActivityMessage":{"prevActivity":null,"activitySignature":{"Ed25519Signature":{"b64":"HNPk/8QF7iVtsI+hHuO1+J9CFnIgsSrqr1ITQ2eQ4VM7rRPG7i07eKKpv/iUwPP4OdloSmoHLWZeMXZNvqnCBQ=="}},"activity":{"seenData":{"v":">>> debug-me session starting\r\n"}}}}
{"ActivityMessage":{"prevActivity":{"hashValue":{"v":"63d31b25ca262d7e9fc5169d137f61ecef20fb65c23c493b1910443d7a5514e4"},"hashMethod":"SHA256"},"activitySignature":{"Ed25519Signature":{"b64":"+E0N7j9MwWgFp+LwdzNyByA5W6UELh6JFxVCU7+ByuhcerVO/SC2ZJJJMq8xqEXSc9rMNKVaAT3Z6JmidF+XAw=="}},"activity":{"seenData":{"v":"$ "}}}}

Ed25519 signatures add 64 bytes overhead to each message, on top of the 64 bytes for the hash pointer to the previous message. But, last night I thought of a cunning plan to remove that hash pointer from the wire protocol, while still generating a provable hash chain. Just leave it out of the serialized message, but include it in the data that's signed. debug-me will then just need to try the hashes of recent messages until it finds one for which the signature verifies, and then it will know what the hash pointer is supposed to point to, without it ever having been sent over the wire! Will implement this trick eventually.

Next though, I need to make debug-me communicate over the network.

mark (Anna and Mark: Waldeneffect)
Honda lawn mower
Honda lawn mower in the box.

After much research we decided to go with a Honda self propelled lawn mower.

We got it at Home Depot which started giving military discounts like Lowes.

Anna (Anna and Mark: Waldeneffect)
Seedling rain
Pea seedlings

Every day this week has offered what I like to call "seedling rain." Less than a quarter of inch of liquid falling over the course of twenty-four hours, the gentle drizzle and overcast conditions perfect for baby vegetables.

Wet broccoli plant

The conditions tempt me to transplant rather than pot up the rest of the seedlings still stuck under lights indoors. But we have four solid weeks until our frost-free date and I don't trust the long-range forecast enough to set out tomatoes, peppers, and basil. Instead, I just watch the gentle rain perk up March-transplanted peas and broccoli and enjoy the ground getting a little easier to weed.

Joey chatter

Thought of a way to squeeze both a 64 byte previous message hash and a 64 byte signature into a message, using only 64 bytes.

Just leave out the previous message hash from the serialized message, but include it in the data that's signed. When verifying the signature, try recently seen messages's hashes as the previous message until finding the one that makes the signature verify. (Conveniently, Ed25519 verifies very fast and can do batch verifications.)

Ah, math..

Joey devblog
debug me first stage complete

Solved that bug I was stuck on yesterday. I had been looking in the code for the developer side for a bug, but that side was fine; the bug was excessive backlog trimming on the user side.

Now I'm fairly happy with how debug-me's activity chains look, and the first stage of developing debug-me is complete. It still doesn't do anything more than the script command, but all the groundwork for the actual networked debug-me is done now. I only have to add signing, verification of gpg key trust, and http client-server to finish debug-me.

(Also, I made debug-me --replay debug-me.log replay the log with realistic delays, like scriptreplay or ttyplay. Only took a page of code to add that feature.)

I'm only "fairly happy" with the activity chains because there is a weird edge case.. At high latency, when typing "qwertyuiop", this happens:


That looks weird, and is somewhat hard to follow in graph form, but it's "correct" as far as debug-me's rules for activity chains go. Due to the lag, the chain forks:

  • It sends "wer" before the "q" echos back
  • It replies to the "q" echo with tyuio" before the "w" echos back.
  • It replies to the "w" echo with "p"
  • Finally, all the delayed echos come in, and it sends a carriage return, resulting in the command being run.

I'd be happier if the forked chain explicitly merged back together, but to do that and add any provable information, the developer would have to wait for all the echos to arrive before sending the carriage return, or something like that, which would make type-ahead worse. So I think I'll leave it like this. Most of the time, latency is not so high, and so this kind of forking doesn't happen much or is much simpler to understand when it does happen.

Anna (Anna and Mark: Waldeneffect)
Bringing a goat back to good health
Wet goats

We're making slow but sure progress with Edgar. Delousing was quite effective, with his fur finally starting to look less rough and scraggly. I suspect the kelp he's been scarfing like candy might also be implicated in his more healthful appearance.

Meanwhile, we were able to trim his hooves relatively easily (although it was a two-person job) and he's okay with a leash now Dwarf goat(although he gets scared every time I put one on). These are definitely moves in the right direction.

On the other hand, our little wether is still far from the health level I expect in my goats. He came with a slight cough, which is still present, and his manure has never come out in solid pellets the way it should. Both could be symptoms of various kinds of worms...or could be as simple as allergies and eating too much lush spring greenery.

Regardless, copper is next on my treatment regimen, and the fresh browse that is starting to fill most of his diet should help with general body tone. Maybe in another month, skittishness will be the only factor proving that Edgar wasn't always our goat.

Joey devblog
debug me chain issues

Working on getting the debug-me proof chain to be the right shape, and be checked at all points for valididity. This graph of a session shows today'ss progress, but also a bug.


At the top, everything is synchronous while "ls" is entered and echoed back. Then, things go asynchronous when " -la" is entered, and the expected echos (in brackets) match up with what really gets echoed, so that input is also accepted.

Finally, the bit in red where "|" is entered is a bug on the developer side, and it gets (correctly) rejected on the user side due to having forked the proof chain. Currently stuck on this bug.

The code for this, especially on the developer side, is rather hairy, I wonder if I am missing a way to simplify it.

Anna (Anna and Mark: Waldeneffect)
Three-year-old espaliered apples
Apple blossom

Those of you considering variety and rootstock selection for apples might get a kick out of some numbers from our three-year-old apple espaliers. These trees were all grafted by me in April 2014, then they were set out into their forever homes that fall. Here's the data:

  • Dead: Liberty (M7), King David (M7), Kidd's Orange Red (M7)
  • Not doing well: Honeycrisp (MM111)
  • Blooming: Winesap (M7), Early Harvest (MM111)
  • Good but vegetative: Pound Pippin (MM111), Red Delicious (MM111), Wolf River (MM111), Winesap (MM111), Chestnut Crab (M7)
Apple espalier

Based on this data, I suspect that M7 rootstock isn't quite cold hardy enough for our farm, at least not if I plant up against the hillside where very little winter sun hits. On the other hand, I wouldn't be surprised if the M7 Winesap is an early bloomer because of its rootstock (as opposed to the Early Harvest, which is noted for precocious bearing).

Honeycrisp, if I recall correctly, was hit hard by cedar apple rust last summer. I think I meant to pull the tree out, actually, but it was granted a stay of execution until this year. That's a variety problem not a rootstock problem, and despite how much Mark loves the apples, I wouldn't plant a Honeycrisp again in a no-spray orchard.

The other varieties seem vigorous and happy. Maybe next year, even those on MM111 will bloom!

mark (Anna and Mark: Waldeneffect)
Greenhouse frame
Greenhouse frame at Joey's.

Anna's brother Joey offered to give us his greenhouse frame if we break it down and haul it off.

The pieces seem to break apart into sections you can fit into a truck.

I think it's a project that will be late Summer or Fall.

Anna (Anna and Mark: Waldeneffect)
Birthday bash 2017
Camera shy

Joey hosted an awesome late birthday/housewarming party Saturday. Much fun was had by all, although some of the participants were a little camera shy.


The food was also the entertainment, courtesy of Joey's homemade charcoal.

Mother and daughters

I mostly ran around enjoying the soft spring grass, which seems about a week further along than it is on our farm. But I took time out to enjoy Maggie's banjo plucking and to visit with friends and family new and old.

Laughing grillmaster

Thanks for the invitation, Joey...and for being a good sport about letting me put multiple photos of you up on the blog.

Anna (Anna and Mark: Waldeneffect)
Farm time change
Weeding with goats

This is the time of year when our farm undergoes its own sort of time change. Afternoons are already getting too hot to make outdoor work fun. So we swap our inside and outside tasks, taking to the garden before lunch and enjoying interior cool during the sweltering afternoons. I may even have to hook up the sprinklers shortly if this hot spell continues!

Joey devblog
debug me half days

Two days only partially spent on debug-me..

Yesterday a few small improvements, but mostly I discovered the posix-pty library, and converted debug-me to use it rather than wrangling ptys itself. Which was nice because it let me fix resizing. However, the library had a bug with how it initializes the terminal, and investigating and working around that bug used up too much time. Oh well, probably still worth it.

Today, made debug-me serialize to and from JSON.

{"signature":{"v":""},"prevActivity":null,"activity":{"seenData":{"v":">>> debug-me session starting\r\n"}}}
{"signature":{"v":""},"prevActivity":{"hashValue":{"v":"fb4401a717f86958747d34f98c079eaa811d8af7d22e977d733f1b9e091073a6"},"hashMethod":"SHA256"},"activity":{"seenData":{"v":"$ "}}}
{"signature":{"v":""},"prevActivity":{"hashValue":{"v":"cc97177983767a5ab490d63593011161e2bd4ac2fe00195692f965810e6cf3bf"},"hashMethod":"SHA256"},"activity":{"seenData":{"v":"AGPL\t    Pty.hs    Types.hs\t  debug-me.cabal  dist\r\nCmdLine.hs  Setup.hs  Val.hs\t  debug-me.hs\t  stack.yaml\r\n"}}}

That's a pretty verbose way of saying: I typed "ls" and saw the list of files. But it compresses well. Each packet for a single keystroke will take only 37 bytes to transmit as part of a compressed stream of JSON, and 32 of those bytes are needed for the SHA256 hash. So, this is probably good enough to use as debug-me's wire format.

(Some more bytes will be needed once the signature field is not empty..)

It's also a good logging format, and can be easily analized to eg, prove when a person used debug-me to do something bad.

Wrote a quick visualizor for debug-me logs using graphviz. This will be super useful for debug-me development if nothing else.


Joey chatter
stained glass lizard

(Stained glass by my dad, lizard by leaving all doors in the house open all week.)

Anna (Anna and Mark: Waldeneffect)
Clay class
Wheel-thrown pottery

Less than a month later, clay class is over --- I'm so sad to see it go! Six sessions on the wheel was just long enough for me to remember how to create vessels of various shapes and sizes, but not long enough to get the itch out of my system. Here's hoping Mark can get the rusted guts of a kick wheel out by the barn up and running before I go into withdrawal.

Pottery bear

Spotty glazesI did a little hand-building too, but most of those pieces haven't yet been fired. One of my earliest efforts, a little black bear, came out nicely...but is about as hard to photograph as Huckleberry. I'll see if I can't get better shots of my other pieces once they're glazed and done.

Anna (Anna and Mark: Waldeneffect)
Spring is bursting out all over
Wild geranium

Hot weather followed directly behind last week's cold spell. Suddenly, leaves and flowers were springing out all over, so I grabbed my camera and prepared to document the show.

Chewing goat

Unfortunately, I made the mistake of bringing the herd along with me. "What fun!" said Artemesia, gnawing on frost-sensitive buckeye leaves that miraculously survived Dogwood Winter. The trouble with going out to photograph nature with goats on your heels is you have to act fast or they'll eat your subject matter.

Joey devblog
debug me day 2

Proceeding as planned, I wrote 170 lines of code to make debug-me have separate threads for the user and developer sides, which send one-another updates to the activity chain, and check them for validity. This was fun to implement! And it's lacking only signing to be a full implementation of the debug-me proof chain.

Then I added a network latency simulation to it and tried different latencies up to the latency I measure on my satellite internet link (800 ms or so)

That helped me find two bugs, where it was not handling echo simulation correctly. Something is still not handled quite right, because when I put a network latency delay before sending output from the user side to the developer side, it causes some developer input to get rejected. So I'm for now only inserting latency when the developer is sending input to the user side. Good enough for proof-of-concept.

Result is that, even with a high latency, it feels "natural" to type commands into debug-me. The echo emulation works, so it accepts typeahead.

Using backspace to delete several letters in a row feels "wrong"; the synchronousness requirements prevent that working when latency is high. Same problem for moving around with the arrow keys. Down around 200 ms latency, these problems are not apparent, unless you mash down the backspace or arrow key.

How about using an editor? It seemed reasonably non-annoying at 200 ms latency, although here I do tend to mash down arrow keys and then it moves too fast for debug-me to keep up, and so the cursor movement stalls.

At higher latencies, using an editor was pretty annoying. Where I might normally press the down arrow key N distinct times to get to the line I wanted, that doesn't work in debug-me at 800 ms latency. Of course, over such a slow connection, using an editor is the last thing you want to do anyway, and vi key combos like 9j start to become necessary (and work in debug-me).

Based on these experiements, the synchronousness requirements are not as utterly annoying as I'd feared, especially at typical latencies.

And, it seems worth making debug-me detect when several keys are pressed close together, and send a single packet over the network combining those. That should make it behave better when mashing down a key.

Today's work was sponsored by Jake Vosloo on Patreon

Anna (Anna and Mark: Waldeneffect)
Aquaponic metamorphosis
Frog metamorphasis

One of my Green Frog tadpoles started transforming on me late last week. I'd actually barely seen the tadpoles since they tended to hide in the fake vegetation, steering clear of the more active fish. But as this guy grew legs, he came out to visit the front of the tank.

Fork frog

Then, Tuesday morning, he was treading water at the surface. "I have lungs now!" he told me. "Give me somewhere to crawl out and hop around!"

Rather than changing the tank itself, I fished the baby frog out with a fork and gave him a sky pond to hang out in. Looks like warm, high nutrient water was all he needed to beat the rest of his species at transformation by two solid months.

Joey devblog
debug me day 1

Started some exploratory programming on the debug-me idea.

First, wrote down some data types for debug-me's proof of developer activity.

Then, some terminal wrangling, to get debug-me to allocate a pseudo-terminal, run an interactive shell in it, and pass stdin and stdout back and forth to the terminal it was started in. At this point, debug-me is very similar to script, except it doesn't log the data it intercepts to a typescript file.

Terminals are complicated, so this took a while, and it's still not perfect, but good enough for now. Needs to have resize handling added, and for some reason when the program exits, the terminal is left in a raw state, despite the program apparently resetting its attributes.

Next goal is to check how annoying debug-me's insistence on a synchronous activity proof chain will be when using debug-me across a network link with some latency. If that's too annoying, the design will need to be changed, or perhaps won't work.

To do that, I plan to make debug-me simulate a network between the user and developer's processes, using threads inside a single process for now. The user thread will builds up an activity chain, and only accepts inputs from the developer thread when they meet the synchronicity requirements. Ran out of time to finish that today, so next time.

debug-me's git repository is available from

Today's work was sponsored by andrea rota.

starting debug-me and a new devblog

I've started building debug-me. It's my birthday, and building a new program is kind of my birthday gift to myself, because I love starting a new program and seeing where it goes. (Also, my Patreon backers wanted me to get on with building debug-me.)

I also have a new devblog! Up until now, I've had a devblog that only covered work on git-annex. That one continues, but the new devblog is for development journaling for any project I'm working on.

mark (Anna and Mark: Waldeneffect)
Roll out nest tray
Roll out nest box tray installed into a wheelchair chicken tractor.

This new roll out nest tray was 22 dollars with shipping.

It's got an angle that makes eggs roll out into the tray under the flap.

It's part of a plastic nest box with a perch but I wanted to build my own box.

Anna (Anna and Mark: Waldeneffect)
Root-bound seedlings
Vegetable seedlings

With the frost danger past, it's time to set out the rest of the cold-hardy seedlings. Everything pictured above hit the garden Monday except the basil and peppers. The peas were a bit root-bound, but I'm hoping they'll expand out into new ground anyway. In a few weeks, I'll have to check back and see if I can tell the difference between the direct-seeded peas and the transplanted starts that I used to fill in the gaps.

Anna (Anna and Mark: Waldeneffect)
Lucky miss
Apple blossoms

I think we might have gotten lucky with this current cold snap. Only the William's Pride apple blossoms were open, and enough flowers remained unfurled that even that variety will likely still set fruit.

Now to start crossing our fingers that we can weather one more month with no hard freezes cold enough to nip incipient fruit. The long-range weather forecast is looking good, but who knows what might change between now and May 15!

Joey devblog
propellor self bootstrap property

Worked for a while today on, with the goal of making propellor build a disk image that itself contains propellor.

The hard part of that turned out to be that inside the chroot it's building, /usr/local/propellor is bind mounted to the one outside the chroot. But this new property needs to populate that directory in the chroot. Simply unmounting the bind mount would break later properties, so some way to temporarily expose the underlying directory was called for.

At first, I thought unshare -m could be used to do this, but for some reason that does not work in a chroot. Pity. Ended up going with a complicated dance, where the bind mount is bind mounted to a temp dir, then unmounted to expose the underlying directory, and once it's set up, the temp dir is re-bind-mounted back over it. Ugh.

I was able to reuse Propellor.Bootstrap to bootstrap propellor inside the chroot, which was nice.

Also nice that I'm able to work on this kind of thing at home despite it involving building chroots -- yay for satelite internet!

Today's work was sponsored by Riku Voipio.

Anna (Anna and Mark: Waldeneffect)
Rhubarb freeze damage
Picking rhubarb

While covering up strawberries and spring seedlings in preparation for a freeze, I suddenly couldn't recall whether or not lush new rhubarb growth needs to be protected. So I headed to the internet for an answer.

What I found was information I hadn't been looking for. Did you realize that the problematic oxalic acid in the leaves of rhubarb moves down into the stems during freezes? In other words, even if you don't need to cover the plants during spring freezes (which seems to be the case --- no visible damage at 27 degrees), you might want to pick any stems you're interested in eating beforehand. Rhubarb crumble --- a new tradition for Dogwood Winter.

mark (Anna and Mark: Waldeneffect)
How do you prevent racoons from eating chickens?
Flicker user Gary J Wood.

Nayan makes a good point in her latest comment.

"My neighbor had made a heavy-duty tractor with hardware cloth etc. and it was pretty heavy. The raccoons got into it anyhow and ate every... single... adult... chicken. How you do keep predators away?

We have come to the conclusion that the risk is worth the value for better quality eggs. If the predator pressure gets too high then maybe we'll build a Fort Knox type of chicken coop and bring bugs to the chickens which will involve another level of time and effort.

Image credit goes to flickr user garyjwood.

git-annex devblog (Joey devblog)
day 456 digging in

Digging in to some of the meatier backlog today. Backlog down to 225.

A lot of fixes around using git annex enableremote to add new gpg keys to a gcrypt special remote.

Had to make git-annex's use of GIT_SSH/GIT_SSH_COMMAND contingent on GIT_ANNEX_USE_GIT_SSH=1 being set. Unfortunate, but difference from git made at least one existing use of that environment variable break, and so it will need to be whitelisted in places where git-annex should use it.

Added support for git annex add --update

Today's work was sponsored by Trenton Cronholm on Patreon.

mark (Anna and Mark: Waldeneffect)
Donation appreciation
Wheelchair chicken tractor nest box grinding.

Anna and I wanted to thank David Hicks for his astute comments and for clicking on the monthly donate button in our store link.

We really appreciate the value you put into the Walden Effect.

We were just talking this afternoon about how we plan to use some of the funds to try out a new Roll Out nest tray product that will slide into the bottom of the above plywood box for the wheelchair chicken tractor project.

Joey chatter

I'm on several pubs, my impression is that at least the public ssb network is fairly well connected and it should not matter even if you ended up on different ones, I'd still see you.

It's a 3 hop gossip network, and pubs count as hops, so me -- mypub -- yourpub -- you works because 3 hops. Assuming our pubs are connected, and the pubs I can see are all interconnected.

Anna (Anna and Mark: Waldeneffect)
Delousing goats
Delousing goats

The quart of Ultra Boss we bought to deal with our goats' lice comes with a measurer in the bottle. But the product is more geared toward cattle than goats, so I decided to use a syringe instead to ensure I dosed our doe and wether correctly. Aurora got 2 mL of insecticide dripped along her back while tiny Edgar got only 1 mL. Hopefully a repeat in two weeks will kill any lingering biters and will leave our herd once more in tip-top condition.

Joey chatter

Playing with scuttlebutt for the first time today. Very impressive.

Very appropriately, I met one of its developers in a pub IRL.

Anna (Anna and Mark: Waldeneffect)
Winterizing the rain barrel (or not)
Rain barrel

I meant to drain the rain barrel last fall. But we were still using it, and every time I let the water run out Mark came along behind me and closed the spigot. I never remembered to tell him that it's good practice to leave the rain barrel empty for the winter, and at some point I just shrugged and let it ride.

And here's the deal --- nothing froze and broke. Now I'm wondering if winterizing the rain barrel is just one of those tasks people tell you to do that aren't really necessary. In fact, we enjoyed having the easily accessible outdoor winter water for goats, chickens, and for the aquaponics setup. So maybe next year I won't even put that task on my to-do list.

How about you? Do you drain your rain barrel for the winter? If not, have you had any problems as a result?

Anna (Anna and Mark: Waldeneffect)
High 70s today, low 30s Friday
Spring seedlings

Even though Dogwood Winter is coming up this weekend, I decided to go ahead and set out my flat of onion seedlings to join the broccoli I transplanted early last week. Friday's task will be covering all and sundry with row covers so the freeze doesn't nip tender leaves. Here's hoping the forecast low of 32 doesn't go below 28 --- fruit blossoms would appreciate a reprieve!

Joey chatter

Doing taxes, sending invoices, cleaning house, mowing yard, finishing Tumbleweed Park.

Looking forward to more significant work tomorrow..

Anna (Anna and Mark: Waldeneffect)
Biting lice on goats
Goat biting lice

The first step in dealing with Aurora's biters was to figure out what exactly is chewing on her. After combing through her hair in bright sunlight for a while, I finally discovered small, slow-moving brown spots. Clipping a bit of hair plus biter then photographing and zooming way in on the resultant image resulted in a diagnosis --- biting lice (Bovicola caprae/Damalinia caprae).

Alert goat

Various natural treatment methods exist for lice on goats, but all either seem too invasive to me or have proven ineffective. On the ineffective front, diatomaceous earth did absolutely nothing to hinder Aurora's lice despite brushing gobs of it into her hair for multiple days running. On the invasive front, soaking your animal in any liquid (including water) and keeping her wet for six hours will kill nearly all lice...good luck with that. Shearing is also effective at removing 30 to 50% of lice, with most of the rest naturally dying as they're exposed to weather...but we have a cold spell coming up and I don't think our goats would enjoy being sheared.

Resting goat

So, chemicals it is. The internet is mixed on mainstream methods of treatment, but one extension agent site says that the ivermectin or moxidectin dosages that work on sucking lice aren't effective against biting lice. Instead, they recommend treating with permethrin, which comes in various brand names and application patterns. Of these, Ultra Boss is labelled for use on goats as a pour-on (meaning you can apply in a strip down the animal's back like cat or dog flea meds). That seems the least invasive, so we're going to give it a shot --- treat once, then wait two weeks and treat again. Fingers crossed poor Aurora will stop stamping and scratching in the near future.

Joey chatter
brain re-engaged

Either I'm finally over the 48 hour flu, or finally found the right meds to mask the symptoms. Anyway, I can think again.

(Modern games have this strange habit of tracking how much time you wasted with them, so I know I played Thimbleweed park for 11 hours yesterday. I even remember most of it, but I was clearly not compos mentis.)

Anna (Anna and Mark: Waldeneffect)
Puerto Rico: The Game
Puerto Rico

Joey came over Saturday with a new card game to play. Puerto Rico really hit the spot since it's fast and simple enough not to feel like a big commitment but complicated enough not to bore you after one or two iterations.

I feel so lucky that my brother gives talks at conferences all over the world just to be introduced to board games I might enjoy. (That is the purpose, right?) Please keep Cory Doctorow's cold to yourself, though. Thanks!

Anna (Anna and Mark: Waldeneffect)
San Juan: The Game
San Juan

Joey came over Saturday with a new card game to play. San Juan really hit the spot since it's fast and simple enough not to feel like a big commitment but complicated enough not to bore you after one or two iterations.

I feel so lucky that my brother gives talks at conferences all over the world just to be introduced to board games I might enjoy. (That is the purpose, right?) Please keep Cory Doctorow's cold to yourself, though. Thanks!

Anna (Anna and Mark: Waldeneffect)
Goat commitment
Goat on spring grass

For the last month, I've been going through the motions with our goats --- feeding them, letting them out to graze, but not really allowing them deeply into my heart. Artemesia's loss was just too raw to consider repeating that experience anytime in the near future.

GoatsBut there's no point in including optional activities into your life unless you embrace them fully. So, after much soul-searching, I've decided to commit and try to make our herd back into a thing of beauty and joy.

Mostly, the decision is just a trick of the mind. But there will also be some changes involved.

First --- our little
wether (who has become the herd boss despite his minuscule size!) needs to learn to walk on a leash, to have his hooves clipped, and in general not to shy away from my approach. Next, we're gonna lick this parasite problem, starting with diatomaceous earth (already applied), ramping up to shaving, then hitting the chemical insecticides if necessary. And, finally, I'm putting our preordered doeling onto my list of things I'm anticipating with honest pleasure. Wanna help me dream up a name for a little, floppy-eared goat due to be born in the next week or so?

Anna (Anna and Mark: Waldeneffect)
Myrtle Beach
Myrtle Beach

Despite the outpouring of support from our kind readers (thank you!), Lucy's death so quickly on the heels of Artemesia's sent me and Mark reeling. On a whim, we filled extra feeders and waterers for the animals left behind, hopped in the car, and drove to the very closest beach to our farm --- Myrtle.

Forest Dunes Myrtle Beach

The oceanfront suite we nabbed at a steep discount felt strangely private despite being one of hundreds at Forest Dunes. Even better, opening the door onto our hefty balcony let in the sights, smells, and sounds of the sea. This is the closest I've ever slept to the beach and I reveled in snoozing to the soundtrack of waves breaking against the shore.


Watching the sun rise from my bed was quite a treat as well.

Ring-billed gull

Of course, Myrtle Beach isn't a very natural seashore. Wildlife sightings consisted of scads of gulls (three different species), a couple of plovers, a fish crow or two, and a few flocks of pelicans. Despite all that, I thoroughly enjoyed four long walks on the beach and one cold but invigorating swim in the ocean. Really, just sitting on that balcony would have been enough to make the drive worthwhile.

Cloudy Myrtle Beach

Two nights was just long enough to remind us that the world is full of peace and joy before we drove back to our waiting farm. Despite our ancient car, which finished the drive held together by two long pieces of gaffer's tape and a dilapidated shoelace, we feel so grateful to have chosen a lifestyle that allows us to take the time we need to fill the hole in our hearts. And thank you for following along on our journey!

git-annex devblog (Joey devblog)
day 454 multicast

Earlier this week I had the opportunity to sit in on a workshop at MIT where students were taught how to use git-annex as part of a stack of tools for reproducible scientific data research. That was great!

One thing we noticed there is, it can be hard to distribute files to such a class; downloading them individually wastes network bandwidth. Today, I added git annex multicast which uses uftp to multicast files to other clones of a repository on a LAN. An "easy" 500 lines of code and 7 hour job.

There is encryption and authentication, but the key management for this turned out to be simple, since the public key fingerprints can be stored on the git-annex branch, and easily synced around that way. So, I expect this should be not hard to use in a classroom setting such as the one I was in earlier this week.

Joey chatter
sundown stream

Sun is down, satellite internet is still up! Turns out there are few things more annoying than losing internet when the sun goes behind some object.

The DROK DC-DC booster solved it. I had originally planned to use it instead of the DC power supply, but that led to very loud coil whine not suitable for use. Don't understand why. Using the DROK to power an inverter, which then gets converted back to DC is not efficient, but at least it works. (And I tuned it so the inverter never spins its fan, so perhaps it's not super innefficient.)

Probably will switch to a better inverter that can handle lower voltages on its own.

Joey chatter

My mom gives this talk high marks.

Joey chatter
watch this!

Has Sumana Harihareswara invented a new form of presentation? her LibrePlanet keynote was a menu of 35 lightening talks, with the audience selecting between them.

Planning to rewatch this, it went so fast and there was so much in there. Also, got a little bit busy with politicking that resulted in the final pick of #15.

She could give this talk a dozen times and it would be a dozen different experiences I'll bet.

mark (Anna and Mark: Waldeneffect)
Blogging time out
Anna throwing firewood.

Anna and I want to thank everybody for their kind words and thoughts.

We've decided to take the week off blogging to reflect and recharge.

We'll start back up on Saturday or Sunday.

Thanks again for being there for us.

Anna (Anna and Mark: Waldeneffect)
Seedling salad
Seedling thinnings

Start with a bowl of thinnings from your vegetable flats. Many (but not all!) vegetables are edible at the seedling stage. In general, figure if you eat the leaves or stems of the adult plant, then the seedlings are probably good to eat as well. Here, I thinned onions, broccoli, parsley, and Swiss chard to add to our salad.

Mix seedlings with an equal quantity of spring lettuce, a couple of slices of cooked bacon (crumbled), and a few roasted figs (sliced into small pieces). Top with your favorite salad dressing if you prefer, then enjoy. Early spring flavors at their best!


List of feeds:

  • Anna: last checked (25 posts)
  • Anna and Mark: Waldeneffect: last checked (3861 posts)
  • Joey: last checked (149 posts)
  • Joey chatter: last checked (1819 posts)
  • Joey devblog: last checked (80 posts)
  • Jay: last checked (25 posts)
  • Dani: last checked (21 posts)
  • Errol: last checked (28 posts)
  • Maggie too: last checked (11 posts)a
  • Maggie also: Not Found (64 posts)
  • Maggie: last checked (35 posts)
  • Tomoko: last checked (76 posts)
  • Jerry: last checked (28 posts)