{"id":7881,"date":"2019-03-03T15:48:31","date_gmt":"2019-03-03T20:48:31","guid":{"rendered":"https:\/\/irreal.org\/blog\/?p=7881"},"modified":"2019-03-03T15:57:20","modified_gmt":"2019-03-03T20:57:20","slug":"some-wisdom-on-pipelines","status":"publish","type":"post","link":"https:\/\/irreal.org\/blog\/?p=7881","title":{"rendered":"Some Wisdom on Pipelines"},"content":{"rendered":"

Over at his blog, Ted Dziuba has posted a valuable piece of wisdom<\/a> that is easy to forget. The TL;DR on the wisdom is that many tasks for which we instinctively turn to heavy-weight, distributed tools are more easily and safely accomplished with a pipeline of standard Unix utilities.<\/p>\n

That can be a hard lesson to learn. When I was young and foolish I would often throw together such a solution but because I was silly would think of it more as a proof-of-concept and go on to build a \u201creal\u201d solution, usually in C. And that was true even though the original solution solved the problem perfectly well and had a perceived run time that was virtually indistinguishable from the one written in C.<\/p>\n

Doubtless, I\u2019m still doing silly things but at least I\u2019ve gotten over that particular brand of it. Not all jobs are amenable to a Unix pipeline, of course, but many are and it\u2019s worth looking at each task to see if they qualify before renting a bunch of virtual servers and firing up Hadoop.<\/p>\n

Back in the Old Days™, resources were more limited so solutions like those recommended by Dziuba were more natural and pipelines were the go-to tool. That\u2019s less true today where resources are plentiful and the command line is often looked at the same way we look at COBOL. Still, even though it\u2019s an old post it\u2019s nevertheless worth reading and taking to heart.<\/p>\n","protected":false},"excerpt":{"rendered":"

Over at his blog, Ted Dziuba has posted a valuable piece of wisdom that is easy to forget. The TL;DR on the wisdom is that many tasks for which we instinctively turn to heavy-weight, distributed tools are more easily and … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9],"tags":[37,36],"class_list":["post-7881","post","type-post","status-publish","format-standard","hentry","category-programming","tag-tech","tag-unix"],"_links":{"self":[{"href":"https:\/\/irreal.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/7881","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/irreal.org\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/irreal.org\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/irreal.org\/blog\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/irreal.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=7881"}],"version-history":[{"count":2,"href":"https:\/\/irreal.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/7881\/revisions"}],"predecessor-version":[{"id":7883,"href":"https:\/\/irreal.org\/blog\/index.php?rest_route=\/wp\/v2\/posts\/7881\/revisions\/7883"}],"wp:attachment":[{"href":"https:\/\/irreal.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=7881"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/irreal.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=7881"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/irreal.org\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=7881"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}