git-annex devblog (Joey devblog)
day 626 performance week

I've spent all week working on performance. It started when Lukey found a way to use git cat-file --buffer to make --all faster. Once implemented, that turned out to be a 2x to 16x speedup in seek time.

I felt that same approach could probably also speed up other parts of git-annex that use git cat-file, so spent another 4 days finding ways to do that. Some of the ideas are not implemented yet, but I landed a 2x speedup today, to all git-annex commands that seek annexed files to work on.

Oh and also there used to be a git-annex branch read cache, but it got removed many years ago, and I forgot it had been removed. Which does not lead to writing the fastest code. Bringing the cache back makes some things another 20% faster.


This work was sponsored by Mark Reidenbach, Jake Vosloo, and Graham Spencer on Patreon.

Posted
Anna (Anna and Mark: Waldeneffect)
Adventure from your armchair
Gap Year cover

I have a new book out...and it's a total pandemic experiment. Gap Year is available in print only and is on the pricey side for 32 pages (full-color will do that), but it should also be very easy to request at your local library through their regular channels.

The book is a travelogue, mostly in pictures, from the time I spent backpacking and drawing plants right after college. I figure there's a 50% chance no one except my mother will be interested, so feel free to skip this one if it's not your cup of tea. On the other hand, if you like it, writing a review and spreading the word will ensure there's a sequel.

Speaking of reviews, here's one to give you an idea of what you'll find inside:

“This will be one of the easiest 5-star ratings I’ve ever given. The journal style of the book presents the reader with a unique glimpse into the author’s year-long journey around Europe. Beautiful drawings and snippets from her letters home draw the reader into her adventure. I’m definitely looking forward to the next volume. This book would make a beautiful gift to anyone who loves travel or nature.” — Turtle Dove

Available on Amazon and Barnes & Noble. Thanks in advance for giving it a try!

Scottish island ecology
Posted
rstidyman (Richard)
Politics and Friends
Consider friendships, one’s that have no practical need other than the pleasure of the connection.

I have acquaintances that are Trump supporters. I don’t have any close friends that are Trump supporters.  I think I know why.  I do not trust Trump supporters.  I don’t engage in deep or difficult conversations. Why? Conversations are usually futile. By choice or ignorance, most are unwilling or unable to practice critical or ethical thinking. They nitpick “but Obama…”, and refuse to look at the big picture, the destruction Trump and his minions are unloading on the U.S.  It reminds me of gated communities.  They only have to see the pretty things, and shut out the reality of everyone else.

Really though, it really boils down to integrity.  I’ve heard integrity explained as willingness to do the right thing, even when it is not convenient or comfortable.  In other words, a willingness to tolerate a madman as long as it benefits me.  Screw everyone else.  The party of ME is the party/cult of Trump.

61+Pm73wZXL

Many are willing to tolerate deplorable behavior from their elected leaders in exchange for a few morsels of self interest, e.g, tax benefits.  They don’t want to consider legitimate news.  They like their narrow focus that Fox News or OAN provide.

By doing so, they don’t struggle with the contradictions of their stated beliefs about love and justice with the hateful injustice coming out of the White House. They refuse to reckon with the truth.  Ignorance is bliss for Trump supporters, until it comes crashing down.

His now emboldened fiercest followers are also the most dangerous.  Think white supremacists and weapons.

Critical thinking skills are essential for making informed decisions. To think critically is to examine reason, purpose, assumptions, facts, consequences, alternate viewpoints, and personal biases before choosing to take action, whether you’re in the voting booth or just talking to a friend. Hopefully, with the help of these examples of fallacies, it just got a little bit easier.”

I could go on but I’ll stop. Would love to hear your thoughts on the subject.

“If you can’t dazzle them with brilliance, baffle them with bullshit.”― W.C. Fields

The last paragraph in this article says it best.
and if you need to read more, this is a good one. 
trump-supporters
Posted
git-annex devblog (Joey devblog)
day 625 import tree largefiles

New feature today: Implemented import tree should honor annex.largefiles.

This only took an hour to implement, but I had to think for several hours first to get a solid understanding of it. Particularly, what happens if a file on a remote has a name that makes it be treated as non-large, but then in a later import, it's renamed to a name that would be treated as large? (Or vice-versa.) My conclusion is that is equivilant to git annex add of a file with the first name followed by git mv, so it's ok for annex.largefiles to not take effect in such a case.


Today's work was sponsored by Martin D on Patreon.

Posted
Anna (Anna and Mark: Waldeneffect)
Squash pollination, broccoli caterpillars, chipmunks in the berries, and more
Garden caterpillar protection

Summer is here, and with it comes the learning portion of the gardening year. On the positive side, Mark's caterpillar tunnels are game changers. Seen above are little brussels sprout plants, thriving without the cabbageworm pressure I usually struggle with.

Broccoli

Also under caterpillar tunnels, our broccoli has treated us to weeks of daily meals. For the first time in my gardening life, I'm preparing to pull the plants out, not because they're so bug-bitten there's no point in keeping them, but because the side shoots are getting small and tough and the soil is ready for some compost and rest. (Plus, our palates are ready for summer crops.)

Baby zucchini

With all of that success, I went a little crazy and put a caterpillar tunnel over some cucurbits, in hopes of keeping various bug issues at bay. Of course, unlike crucifers, the covered squash and cucumbers require pollination. So once the plants were big enough, I started hand-pollinating.

The hand-pollinating got old after a week, at which point I took the caterpillar tunnels off. But, in the meantime, I learned why my recent summer-squash harvests have been so-so. Without chickens to eat excess fruits, I'd cut back to one crookneck squash and one zucchini during each succession-planting period...but that's not enough for proper pollination since the two species don't cross!

On many of my hand-pollinating days, there were no male flowers open when a female squash flower was due to be pollinated. Sometimes, I was able to tear open yesterday's spent male flower and get a bit of pollen. Sometimes that didn't work. Now I know --- better to plant each summer squash species at least in pairs!

Red raspberry

Now for the failure. Berries, berries, beautiful berries! We built a netted enclosure after the chipmunks ate all of our strawberries last year, and for a couple of weeks it seemed to be working. Then the chipmunks found a way in and demolished the rest of the patch in a matter of days. We'll be working on that problem before next year. In the meantime, at least the evil rodents can't reach our raspberries and the netting keeps the birds out.

Green tomatoes

On a happier note, we spent some of our stimulus money on drip irrigation this spring and it's working like a charm! If you're pinching pennies, you can put together a cheaper option piecemeal, but we opted for a kit from Johnny's (who gives us no kickback for mentioning them, darn it! But their products are so good I do it anyway). Mark put the pieces together without needing to read the instructions, and now the timer automatically soaks the soil for three hours twice a week.

The question will be --- how much does our water bill rise as a result of giving the garden what it needs? Since we're steering clear of the farmer's market this year due to crowded coronavirus concerns, high-quality produce is likely to be worth whatever the water costs.

Gap Year excerpt

And that's all for now, although I have something different and fun coming your way later this week. Stay tuned (or follow me on Amazon if you think you'll forget).

Posted
Joey
bracketing and async exceptions in haskell

I've been digging into async exceptions in haskell, and getting more and more concerned. In particular, bracket seems to be often used in ways that are not async exception safe. I've found multiple libraries with problems.

Here's an example:

withTempFile a = bracket setup cleanup a
  where
    setup = openTempFile "/tmp" "tmpfile"
    cleanup (name, h) = do
        hClose h
        removeFile name

This looks reasonably good, it makes sure to clean up after itself even when the action throws an exception.

But, in fact that code can leave stale temp files lying around. If the thread receives an async exception when hClose is running, it will be interrupted before the file is removed.

We normally think of bracket as masking exceptions, but it doesn't prevent async exceptions in all cases. See Control.Exception on "interruptible operations", which can receive async exceptions even when other exceptions are masked.

It's a bit surprising, but hClose is such an interruptable operation, because it flushes the write buffer. The only way to know is to read the code.

It can be quite hard to determine if an operation is interruptable, since it can come down to whether it retries a STM transaction, or uses a MVar that is not always full. I've been auditing libraries and I often have to look at code several dependencies away, and even then may not be sure if a library has this problem.

  • process's withCreateProcess could fail to wait on the process, leaving a zombie. Might also leak file descriptors?

  • http-client's withResponse might fail to close a network connection. (If a MVar happened to be empty when it's called.)

    Worth noting that there are plenty of examples of using http-client to eg, race downloading two urls and cancel the slower download. Which is just the kind of use of an async exception that could cause a problem.

  • persistent's withSqlPool and withSqlConn might fail to clean up, when used with persistent-postgresql. (If another thread is using the connection and so a MVar over in postgresql-simple is empty.)

  • concurrent-output has some locking code that is not async exception safe. (My library, so I've fixed part of it, and hope to fix the rest.)

So far, around half of the libraries I've looked at, that use bracket or onException or the like probably have this problem.

What can libraries do?

  • Document whether these things are async exception safe. Or perhaps there should be an expectation that "withFoo" always is, but if so the Haskell comminity has some work ahead of it.

  • Use finally. Good mostly in simple situations; more complicated things would be hard to write this way.

    hClose h `finally` removeFile name
    
  • Use uninterruptibleMask, but it's a big hammer and is often not the right tool for the job. If the operation takes a while to run, the program will not respond to ctrl-c during that time.

  • May be better to run the actions in worker threads, to insulate them from receiving any async exceptions.

    bracketInsulated :: IO a -> (a -> IO b) -> (a -> IO c) -> IO c
    bracketInsulated a b = bracket
      (uninterruptibleMask $ \u -> async (u a) >>= u . wait)
      (\v -> uninterruptibleMask $ \u -> async (u (b v)) >>= u . wait)
    
    (Note use of uninterruptibleMask here in case async itself does an interruptable operation. My first version got that wrong.. This is hard!)

My impression of the state of things now is that you should be very cautious using race or cancel or withAsync or the like, unless the thread is small and easy to audit for these problems. Kind of a shame, since I had wanted to be able to cancel a thread that is big and sprawling and uses all the libraries mentioned above.


This work was sponsored by Jake Vosloo and Graham Spencer on Patreon.

Posted
rstidyman (Richard)
Building a chicken Coop

When it was all said and done, I told Danielle that having chickens and going to the trouble and expense of housing them is not a good financial decision. She agreed. It was strictly entertainment. So here it is.

We did a little reading and I had/have a stack of sawmill lumber sitting out in a pile, and for a minimal amount of money for wood, we got this built. Hardware and fasteners were extra.

The coop is way overbuilt.  I used 2x4s, and the posts in the ground are 6×6’s.  Easily accomplished with lighter duty wood.  

 

The planning. Using an older and more comfortable version of Sketchup.

 

chicken coop 3chicken coop 5chicken coop 6chicken coop 7chicken coop 9chicken coop 10chicken coop 15chicken coop 16When it was all said and done, I told Danielle that having chickens and going to the trouble and expense of housing them is not a good financial decision. She agreed. It was strictly entertainment. So here it is.

We did a little reading and I had/have a stack of sawmill lumber sitting out in a pile, and for a minimal amount of money for wood, we got this built. Hardware and fasteners were extra.

The planning. Using an older and more comfortable version of Sketchup.

chicken coop 19chicken coop 21chicken coop 22chicken coop 24chicken coop 25

Cluckingham Palaceplans 5plans 6plans 7

The chicken pen.

chicken coop 28chicken coop 30chicken coop 33chicken coop 37chicken coop 48chicken coop 49chicken coop 50chicken coop 51

 

The nesting box.

Nesting box all togethernesting box exploded view 2nesting box exploded viewnesting box isometric assembled

A video at the beginning. https://youtu.be/wZJoPcYsKX0

 

Posted
git-annex devblog (Joey devblog)
day 624 timeouts snag

Started out the day productively working through more async exception safety for timeouts. But then I realized there's a whole can of worms involving bracket not working like I expected it to.

Got a bit side tracked checking if other people expect bracket to work the way it actually does, and seem to have found a bug in the process library. Which is especially concerning since it's just the first place I looked, so what other libraries might have similar problems?

So the timeouts feature is seeming a lot less plasible than it did last week. I'll probably defer it until later. The work done on it so far is at least generally an improvement to the code.

Posted

List of feeds:

  • Anna: last checked (50 posts)
  • Anna and Mark: Waldeneffect: last checked (4552 posts)
  • Joey: last checked (206 posts)
  • Joey devblog: last checked (250 posts)
  • Jay: last checked (50 posts)
  • Errol: last checked (53 posts)
  • Maggie: Cannot detect feed type (35 posts)
  • Maggie too: Cannot detect feed type (72 posts)
  • Maggie also: Not Found (437 posts)
  • Tomoko: last checked (77 posts)
  • Jerry: last checked (28 posts)
  • Dani: last checked (22 posts)
  • Richard: last checked (60 posts)