I've spent all week working on performance. It started when Lukey found a way to use git cat-file --buffer to make --all faster. Once implemented, that turned out to be a 2x to 16x speedup in seek time.

I felt that same approach could probably also speed up other parts of git-annex that use git cat-file, so spent another 4 days finding ways to do that. Some of the ideas are not implemented yet, but I landed a 2x speedup today, to all git-annex commands that seek annexed files to work on.

Oh and also there used to be a git-annex branch read cache, but it got removed many years ago, and I forgot it had been removed. Which does not lead to writing the fastest code. Bringing the cache back makes some things another 20% faster.

This work was sponsored by Mark Reidenbach, Jake Vosloo, and Graham Spencer on Patreon.

I have a new book out...and it's a total pandemic experiment. Gap Year is available in print only and is on the pricey side for 32 pages (full-color will do that), but it should also be very easy to request at your local library through their regular channels.

The book is a travelogue, mostly in pictures, from the time I spent backpacking and drawing plants right after college. I figure there's a 50% chance no one except my mother will be interested, so feel free to skip this one if it's not your cup of tea. On the other hand, if you like it, writing a review and spreading the word will ensure there's a sequel.

Speaking of reviews, here's one to give you an idea of what you'll find inside:

“This will be one of the easiest 5-star ratings I’ve ever given. The journal style of the book presents the reader with a unique glimpse into the author’s year-long journey around Europe. Beautiful drawings and snippets from her letters home draw the reader into her adventure. I’m definitely looking forward to the next volume. This book would make a beautiful gift to anyone who loves travel or nature.” — Turtle Dove

Available on Amazon and Barnes & Noble. Thanks in advance for giving it a try!

New feature today: Implemented import tree should honor annex.largefiles.

This only took an hour to implement, but I had to think for several hours first to get a solid understanding of it. Particularly, what happens if a file on a remote has a name that makes it be treated as non-large, but then in a later import, it's renamed to a name that would be treated as large? (Or vice-versa.) My conclusion is that is equivilant to git annex add of a file with the first name followed by git mv, so it's ok for annex.largefiles to not take effect in such a case.

Today's work was sponsored by Martin D on Patreon.

Summer is here, and with it comes the learning portion of the gardening year. On the positive side, Mark's caterpillar tunnels are game changers. Seen above are little brussels sprout plants, thriving without the cabbageworm pressure I usually struggle with.

Also under caterpillar tunnels, our broccoli has treated us to weeks of daily meals. For the first time in my gardening life, I'm preparing to pull the plants out, not because they're so bug-bitten there's no point in keeping them, but because the side shoots are getting small and tough and the soil is ready for some compost and rest. (Plus, our palates are ready for summer crops.)

With all of that success, I went a little crazy and put a caterpillar tunnel over some cucurbits, in hopes of keeping various bug issues at bay. Of course, unlike crucifers, the covered squash and cucumbers require pollination. So once the plants were big enough, I started hand-pollinating.

The hand-pollinating got old after a week, at which point I took the caterpillar tunnels off. But, in the meantime, I learned why my recent summer-squash harvests have been so-so. Without chickens to eat excess fruits, I'd cut back to one crookneck squash and one zucchini during each succession-planting period...but that's not enough for proper pollination since the two species don't cross!

On many of my hand-pollinating days, there were no male flowers open when a female squash flower was due to be pollinated. Sometimes, I was able to tear open yesterday's spent male flower and get a bit of pollen. Sometimes that didn't work. Now I know --- better to plant each summer squash species at least in pairs!

Now for the failure. Berries, berries, beautiful berries! We built a netted enclosure after the chipmunks ate all of our strawberries last year, and for a couple of weeks it seemed to be working. Then the chipmunks found a way in and demolished the rest of the patch in a matter of days. We'll be working on that problem before next year. In the meantime, at least the evil rodents can't reach our raspberries and the netting keeps the birds out.

On a happier note, we spent some of our stimulus money on drip irrigation this spring and it's working like a charm! If you're pinching pennies, you can put together a cheaper option piecemeal, but we opted for a kit from Johnny's (who gives us no kickback for mentioning them, darn it! But their products are so good I do it anyway). Mark put the pieces together without needing to read the instructions, and now the timer automatically soaks the soil for three hours twice a week.

The question will be --- how much does our water bill rise as a result of giving the garden what it needs? Since we're steering clear of the farmer's market this year due to crowded coronavirus concerns, high-quality produce is likely to be worth whatever the water costs.

And that's all for now, although I have something different and fun coming your way later this week. Stay tuned (or follow me on Amazon if you think you'll forget).

I've been digging into async exceptions in haskell, and getting more and more concerned. In particular, bracket seems to be often used in ways that are not async exception safe. I've found multiple libraries with problems.

Here's an example:

withTempFile a = bracket setup cleanup a
  where
    setup = openTempFile "/tmp" "tmpfile"
    cleanup (name, h) = do
        hClose h
        removeFile name

This looks reasonably good, it makes sure to clean up after itself even when the action throws an exception.

But, in fact that code can leave stale temp files lying around. If the thread receives an async exception when hClose is running, it will be interrupted before the file is removed.

We normally think of bracket as masking exceptions, but it doesn't prevent async exceptions in all cases. See Control.Exception on "interruptible operations", which can receive async exceptions even when other exceptions are masked.

It's a bit surprising, but hClose is such an interruptable operation, because it flushes the write buffer. The only way to know is to read the code.

It can be quite hard to determine if an operation is interruptable, since it can come down to whether it retries a STM transaction, or uses a MVar that is not always full. I've been auditing libraries and I often have to look at code several dependencies away, and even then may not be sure if a library has this problem.

• process's withCreateProcess could fail to wait on the process, leaving a zombie. Might also leak file descriptors?

• http-client's withResponse might fail to close a network connection. (If a MVar happened to be empty when it's called.)

  Worth noting that there are plenty of examples of using http-client to eg, race downloading two urls and cancel the slower download. Which is just the kind of use of an async exception that could cause a problem.

• persistent's withSqlPool and withSqlConn might fail to clean up, when used with persistent-postgresql. (If another thread is using the connection and so a MVar over in postgresql-simple is empty.)

• concurrent-output has some locking code that is not async exception safe. (My library, so I've fixed part of it, and hope to fix the rest.)

So far, around half of the libraries I've looked at, that use bracket or onException or the like probably have this problem.

What can libraries do?

• Document whether these things are async exception safe. Or perhaps there should be an expectation that "withFoo" always is, but if so the Haskell comminity has some work ahead of it.

• Use finally. Good mostly in simple situations; more complicated things would be hard to write this way.

  hClose h `finally` removeFile name
  

• Use uninterruptibleMask, but it's a big hammer and is often not the right tool for the job. If the operation takes a while to run, the program will not respond to ctrl-c during that time.

• May be better to run the actions in worker threads, to insulate them from receiving any async exceptions.

  bracketInsulated :: IO a -> (a -> IO b) -> (a -> IO c) -> IO c
  bracketInsulated a b = bracket
    (uninterruptibleMask $ \u -> async (u a) >>= u . wait)
    (\v -> uninterruptibleMask $ \u -> async (u (b v)) >>= u . wait)
  

  (Note use of uninterruptibleMask here in case async itself does an interruptable operation. My first version got that wrong.. This is hard!)

My impression of the state of things now is that you should be very cautious using race or cancel or withAsync or the like, unless the thread is small and easy to audit for these problems. Kind of a shame, since I had wanted to be able to cancel a thread that is big and sprawling and uses all the libraries mentioned above.

This work was sponsored by Jake Vosloo and Graham Spencer on Patreon.

When it was all said and done, I told Danielle that having chickens and going to the trouble and expense of housing them is not a good financial decision. She agreed. It was strictly entertainment. So here it is.

We did a little reading and I had/have a stack of sawmill lumber sitting out in a pile, and for a minimal amount of money for wood, we got this built. Hardware and fasteners were extra.

The coop is way overbuilt.  I used 2x4s, and the posts in the ground are 6×6’s.  Easily accomplished with lighter duty wood.  

The planning. Using an older and more comfortable version of Sketchup.

When it was all said and done, I told Danielle that having chickens and going to the trouble and expense of housing them is not a good financial decision. She agreed. It was strictly entertainment. So here it is.

We did a little reading and I had/have a stack of sawmill lumber sitting out in a pile, and for a minimal amount of money for wood, we got this built. Hardware and fasteners were extra.

The planning. Using an older and more comfortable version of Sketchup.

The chicken pen.

The nesting box.

A video at the beginning. https://youtu.be/wZJoPcYsKX0

Started out the day productively working through more async exception safety for timeouts. But then I realized there's a whole can of worms involving bracket not working like I expected it to.

Got a bit side tracked checking if other people expect bracket to work the way it actually does, and seem to have found a bug in the process library. Which is especially concerning since it's just the first place I looked, so what other libraries might have similar problems?

So the timeouts feature is seeming a lot less plasible than it did last week. I'll probably defer it until later. The work done on it so far is at least generally an improvement to the code.