tag:blogger.com,1999:blog-286542042024-03-07T12:16:01.378+05:30/dev/randomSankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.comBlogger65125tag:blogger.com,1999:blog-28654204.post-69921133436983660972021-09-28T20:55:00.001+05:302021-09-28T21:03:39.475+05:30Software obsoleting faster than Hardware<p>When I started my career, I was lucky to work on a legendary Operating System named <a href="https://en.wikipedia.org/wiki/NetWare" target="_blank">Novell Netware</a>. I got my first job because of a hacking adventure that me and my roommate Arvind did in our college on top of an upatched Netware installation. The reliability and robustness of Netware may make you believe in magic. But it was just a well-engineered, old-school product. One of the most popular instances of its robustness, was the <a href="https://arstechnica.com/information-technology/2013/03/epic-uptime-achievement-can-you-beat-16-years/" target="_blank">epic uptime of 16 years as covered in arstechnica</a>.</p><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqcEKqQRXA5Wsvwu5hojz94ASoqNn2-5Dlh8bA7oFdPPyvM6sLKKXqhnJyLTQVWetCQnzcmasHR70V9K4Afz-QyL-y9rv8RuRQ_qQ_4zTgp6HMGD1d-pNR8FxBJSR8z3fbbs4-GA/s1229/intel-server-uptime.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="692" data-original-width="1229" height="180" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqcEKqQRXA5Wsvwu5hojz94ASoqNn2-5Dlh8bA7oFdPPyvM6sLKKXqhnJyLTQVWetCQnzcmasHR70V9K4Afz-QyL-y9rv8RuRQ_qQ_4zTgp6HMGD1d-pNR8FxBJSR8z3fbbs4-GA/s320/intel-server-uptime.jpg" width="320" /></a></div><br /><p style="text-align: center;">(Image courtesy: Arstechnica)</p><p style="text-align: left;">This was not an one-off situation either. We had multiple customers with years of uptime. In one of the academic institutes, the uptime was well into decades, that multiple sysadmins changed, but the netware box tirelessly worked on. At some point of time, nobody knew where the server was physically located, as nobody looked at it as everything worked fine.</p><p style="text-align: left;">In almost all the cases, the hardware failed before the software. The software was engineered so well that it would have run forever on superior hardware (albeit not so efficiently capable of using the modern hardware in its true potential). Those days, even the hardware was built to last for decades. It was the good old times before the <a href="https://psankar.blogspot.com/2020/12/repairability.html" target="_blank">planned obsolescence</a>. </p><p style="text-align: left;">Fast forward to today, 2021. I have a <a href="https://www.amazon.in/gp/product/B01MU2AWLB/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1" target="_blank">Redmi 4 </a>android phone, built by the mass manufacturer Xiaomi. I bought it on May 30th 2017 and still use it everyday. I always purchase things for long-term. I believe in BuyItForLife principles. I maintain my hardware properly (Fully discharge and then recharge, handle with care etc.). Even my prior phone, a <a href="https://www.gsmarena.com/motorola_e398-690.php" target="_blank">Motorola E398</a> lasted me a about a decade, before the charger gave up.</p><p style="text-align: left;">Today during the lunch break, I went out looking in <a href="https://psankar.blogspot.com/2021/09/blog-post.html" target="_blank">search</a> of a home. My ever busy teammate Seshachalam sent a message to me on Slack at that time. I got a notification in the Android pull-down notifications. I tried to open Slack to see the message and got this error message:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEb_Mx5agBC-TiRjDwOHFRn4NsbXLZYu4RbLWhGLzAjNNK1zdd8YW0v3GPEttXbrLNf1wNkVKcqVeBa13JIZdnheDvh-svuFTJ-hOL0al_8VIe-B7JX0OXsFblv2jG-Saydf-C9w/s1280/Screenshot_2021-09-28-13-12-14-431_com.Slack.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1280" data-original-width="720" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEb_Mx5agBC-TiRjDwOHFRn4NsbXLZYu4RbLWhGLzAjNNK1zdd8YW0v3GPEttXbrLNf1wNkVKcqVeBa13JIZdnheDvh-svuFTJ-hOL0al_8VIe-B7JX0OXsFblv2jG-Saydf-C9w/s320/Screenshot_2021-09-28-13-12-14-431_com.Slack.jpg" width="180" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: left;">It seems Slack will no longer work on Android phones that are just about 4 years old. Slack got acquired for 27.7 billion USD a couple of years back. They must have a lot of good engineers. Xiaomi is a cheap, chinese mass manufacturer, who sells hardware at a fraction of what iPhone costs. If the software slack is getting obsolete than a hardware optimised for cost, we engineers, as a [pseudo-]species, are doing something terribly regressive and getting worse. Netware was an entire operating system, with file, print sharing and they could provide an uptime of decades. Slack is a messaging app, which should not have problems running in < 5 year old hardware.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Slack has a responsibility to keep running on old hardware, if not for anything else, but just for our planet. Even if 0.5% of total number of slack users (which should be comfortably in tens of millions) have to update their perfectly functioning smartphones, just because slack won't work, imagine the e-waste that would be generated. Imagine the water cost of these smartphones. All these ewaste will be then dumped in poor third world nations, causing more harm to them.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">Being green, environmentally responsible does not mean just optimising the server side cost. Companies like slack, zoom, etc. which have become indispensable, in the post-covid world, need to do better and maintain better longevity for their software.</div><div class="separator" style="clear: both; text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: left;">If nothing else, Slack could just provide a lite version which has only plain text messages shown and no emoticons, rickrolling, etc. Even GMail has a<a href="https://support.google.com/mail/answer/15049?hl=en" target="_blank"> basic HTML</a> view.</div><div class="separator" style="clear: both; text-align: left;"><br /></div>Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com8tag:blogger.com,1999:blog-28654204.post-70123363042499981152020-12-27T13:01:00.000+05:302020-12-27T13:01:02.332+05:30Repairability<p><b>My Macbook Pro</b></p><p>I have a Macbook pro retina 15 inch, that I bought in 2016. A few days back, the battery started bulking up and the laptop has totally stopped working. It has grown so big now that I cannot keep the laptop in a flat surface; it almost rocks like a see-saw. The touchpad panel is also feeling the bulge. The Macbook pro is not even switching on now, presumably to safeguard against battery explosions.</p><p>I bought the laptop 4 years back, for about 200,000 INR (~2700 USD / 2200 EUR). Electronics are very costly in India :( This is the pre-touchbar Macbook pro. I did not like the new keyboard in the then new Macbooks (the first edition with the touchbar). Luckily, I did not purchase the touchbar version which probably is one of the worst electronic devices ever manufactured.</p><p>I took my macbookpro today to a nearby Apple service center and I was told that the battery replacement would cost about 40,000 INR (~550 USD / 450 EUR). For such a costly laptop, it has a terrible battery longevity. I have no interest in paying this much for just a battery for an old laptop which is anyway not fun anymore to work on. I could purchase a new laptop with this money. For this money I could even setup a private cloud of a few Rasperry PIs and even launch kubernetes in them for fun.</p><p>I thought that I could purchase a battery offline and replace the battery. But to replace the battery, you need to <a href="https://www.youtube.com/watch?v=AabLlHT503o" target="_blank">disassemble almost everything</a> (Harddisk, Speakers, CPU, fans, etc.) in a macbook and use chemicals (acetone). Thankfully Apple is not yet making cars, otherwise, to change Engine oil, we may have to dissasemble the headlamps, engine, transmission, differential, etc. </p><p>What is more evil is, The Macbook pro will not work without a battery even if it is plugged in an electric power supply. I do this for my old HP laptop (for parents+kids) whose battery is long gone. The magsafe charger of Macbook is another well-known disaster. I believe that Apple gets a lot of undeserving praise for their hardware. They have shiny aluminium body, a good screen and the best touchpad; But their Thermal management, Longevity, Repairability are all abysmally bad. I have replaced the charger three times in ~3 years because the plastic covering near the charging point goes bad in daily usage and the internal wire gets damaged. The wires also become very yellow and dirty in Indian climate, for some reason. Even people far more connected and influential than me, could not <a href="https://techcrunch.com/2018/08/04/the-greedy-ways-apple-got-to-1-trillion/">change</a> Apple's behavior. <br /><br />I personally, am never buying any apple device or Macbook pro ever again, for personal use. It's just because I do not like their greed and exploitation of vulnerable customers.</p><p><b>Thinkpads</b></p><p>The only reason I bought the Macbook pro in 2016 was because I had to do some iOS app development. Prior to that in my $DAYJOBs I have almost exclusively used Thinkpads right from the days they were owned by IBM (and had an extra-ordinary keyboard) until they were sold off to Lenovo (and have this chiclet non-sense). The old Thinkpads were a delight to have. We could replace the batteries, replace the fans, replace individual keys etc. We could also add RAM or Disk whenever we need, how much ever we need. All without needing anything more than a normal screwdriver set.</p><p>Good things are not meant to last. Just like how Macbooks have gone worse, Thinkpads too have gone worse. In my current $DAYJOB I use a Thinkpad E series (the cheapest version) and it is terrible. The management of Lenovo is either dumb and do not understand what its loyal customers want; or just plain evil (or capitalistic extremists) and embraced planned obsolesence.</p><p>The thinkpad now comes in many series L, T, X, P, E etc. and almost none of them have external batteries. Almost all of them have the RAM soldered and cannot be replaced. If the soldered RAM goes wrong, we need to throw away the laptop. Thinkpads were supposed to be the most developer friendly laptops. But even in 2020, we cannot get a single 32GB RAM in any of the medium cost thinkpad ranges. If you want anything more than 32GB RAM, you must shell out a lot of money and go for a ridiculously high cost series with 4k screen or some such luxury that I do not want. And their fingerprint readers never seem to reliably work on Linux for some reason, despite most kernel developers using Thinkpads.</p><p><b>E Waste and Green Earth</b></p><p>When I was in school, I have lived in a house with no electricity. I have then grown up and lived in Indian towns where 8-12 hours power cut per day was not unheard of. Luckily I now live in a big Indian city where powercuts are just an occasional weekly-few-hours affair. If it were not for the powercuts, I would happily purchase a desktop instead of a laptop. Atleast until now, desktops (Not those integrated all-in-one pieces) age better than laptops. But powercuts are a part of life where I live and I need battery backup.</p><p><b>Mobile Phones</b></p><p>Laptops and desktops are only a small part of the story. Now, with mobile phones coming on, the amount of e waste getting generated is exploding (literally in some sense). Android is a bigger culprit than Apple here. Even Google (which does not have "Do no evil" as a motto anymore) is <a href="https://www.theverge.com/2020/12/8/22164027/google-pixel-2-final-update-security-patch-november-2020 " target="_blank">refusing</a> to push updates for pixel phones that are just 3 years old. <br /><br />Once electric cars become more available, it is going to be worse for third world nations which import e waste. Rich billionaires and millionaires will claim to be more green by switching to electric cars and will send off the batteries to electronic graveyards in the other side of the Earth.</p><p>By making it difficult to replace/recycle batteries in laptops, phones, OEMs are making the world generate a lot of e waste. The first world nations worry about e-waste polluting their water and land, so what do they do ? They simply<a href="https://www.youtube.com/watch?v=vufLW4xOsS4" target="_blank"> dump it out to third world nations</a>, like India, Vietnam etc. As if, us people of these nations, do not have enough things to worry about on our own, now we have to accomodate tonnes of these e-wastes, which spoil our water and pollute our air. <br /><br />We cannot even protest against these e-waste processing units that import world's junk, because most of the third world nations do not even have <a href="https://en.wikipedia.org/wiki/Democracy_Index" target="_blank">healthy democracies</a> where citizens can opine against the Government/rulers, unlike the west.</p><p><b>What to do ?</b></p><p>The European Union atleast is<a href="https://repair.eu/news/european-parliament-stands-by-consumers-and-the-environment-in-the-fight-against-obsolescence/"> trying to do something</a>, while rest of the world seem to be not bothered. The USA especially has a lot of responsibility, because most of the OEMs like Apple, HP, Dell, etc. are walking the evil path of denying repairability and increasing sales, only to please the wallstreet and their $SHARE_SYMBOLs in the American stock market. It is pointless to spend millions for green-earth initiatives, if you do not produce re-cyclable / repairable electronic gadgets.</p><p><b>What can we Engineers do to combat such planned obsolecence and promote right-to-repair and recycling ? </b></p><p>Honestly, I do not have an answer. May be we could influence in our small circles of hardware purchase. When your $EMPLOYER is looking to update the company hardware and get laptops for everyone, insist them to get only hardware <a href="https://www.ifixit.com/laptop-repairability" target="_blank">which can be easily repaired</a>. <br /><br />If you work for an e-commerce giant (like Amazon, Walmart, EBay, etc.) push your employer to provide "Repairability score" as a filter condition in the product pages (similar to 3*, 4*, 5* , etc.) of electronic devices. May be if enough companies/customers start demanding these, the OEMs will have a financial motivation to do the right thing.</p><p>Are there any other steps that you believe that we as individuals could do to bring a change ? If so, please comment. </p><p>What laptops do you like using that have a good repairability score, even in 2020/2021.</p><p>Thanks for coming to my TED talk ;-)</p><p>PS: If you know any good third party battery (which wouldn't explode in hot Indian weather) for Macbook Pro please let me know. If you refer a mechanic/shop who does the Macbook battery replacement in Chennai/Bangalore, India, that would be even better.</p><p><br /></p>Sankar Phttp://www.blogger.com/profile/08027774078620587372noreply@blogger.com11tag:blogger.com,1999:blog-28654204.post-89158874510946880112020-03-04T22:36:00.000+05:302020-03-04T22:39:12.367+05:30CLI Tools<div dir="ltr" style="text-align: left;" trbidi="on">
I spend a lot of time on terminal. I prefer using CLI Tools. Even when I like using Goland or Visual Studio Code (IDEs) for coding (instead of vim/emacs), I prefer to do my non-coding activities from a terminal using CLI tools.<br />
<br />
I am quite happy with my zsh and its various plugins (git, kubernetes, docker, etc). Since in $DAYJOB I work with kubernetes a lot, I heavily use kubectl, kubectx, kubens etc. in combination with grep, jq, pipes, etc. and prefer these CLI tools always over clicking buttons or scrolling long pages in browser.<br />
<br />
All these got me into thinking, if I were<b> to write a command line application today (March 2020), which language / frameworks should I use</b> ? Some self-imposed constraints:<br />
<br />
<ul style="text-align: left;">
<li>The CLI application will be short-lived and will be invoked multiple times everyday by developers/users (such as grep, ls, cat, sed, etc.) and not daemons or long running processes. It<b> </b>won't matter if they leak memory too ;-)</li>
<li>It needs to be <b>fast</b>. Lightning speed.</li>
<li>The tool development may also be split into two parts, a library and a binary, if it could help in developing parallel client implementations (may be in future a GUI tool)</li>
<li>The tool is going to be a FOSS tool and would need some community presence in future.</li>
<li>Tool needs to be cross platform (Mac, Linux, Windows)</li>
</ul>
<br />
This post is a summary of various candidates and their current strengths and weaknesses in <b>my</b> perspective.<br />
<h3 style="text-align: left;">
C</h3>
+ Probably the language that will guarantee the fastest tool.<br />
+ Easy to write wrapper libraries/bindings for any language<br />
+ There are libraries like <a href="https://developer.gnome.org/glib/" target="_blank">glib</a> which might help with achieving better platform portability than the default language.<br />
- Manual memory management, crash prone<br />
- Most young programmers of today may not bother contributing, even if they use. Heck, most do not even learn this language in most colleges anymore.<br />
- If the tool has to work with network services, JSON etc., library support is not going to be easily available.<br />
- i18n, Unicode support etc. may not be great out of the box<br />
<br />Note: C++ is just a more complicated and painful C in my personal experience. Even though I have heard nice things about SmartPointers etc., I am not really convinced that learning C++ is going to be beneficial in the long run and never really bothered to master it, after learning it at college. This is an intentional miss.<br />
<h3 style="text-align: left;">
Java</h3>
+ Rich and mature libraries and ecosystem<br />
+ Complete platform independence and is guaranteed to run on Linux, Macs or Windows alike. Datatypes, Files etc. behave properly everywhere.<br />
- Perception of slowness. There may be some JVM tweaking etc. needed and that is an extra effort. IIUC this would be a non-issue for long running processes.<br />
- <a href="https://www.snowsoftware.com/int/blog/2019/03/07/can-java-still-be-free-after-january-2019" target="_blank">Unclear licensing</a>: may require to pay money to Oracle<br />
<br />
Special mention: <b>Kotlin</b> is a humongously better language than the Java 8 (with lambdas) that I recently used. It has been mentioned that future releases of Java would incorporate sexy aspects of Kotlin. <a href="https://www.youtube.com/watch?v=te3OU9fxC8U" target="_blank">Related talk.</a> (Ignore the clickbait title, excellent talk really, strongly recommended).<br />
<h3 style="text-align: left;">
Python</h3>
+ Good library support and hopefully all will migrate to Python 3<br />
- Not a fan of the language for various reasons. Whitespace for scope identification, Lack of static typing out of the box, type unsafety for variables, etc.<br />
<h3 style="text-align: left;">
Javascript (With Typescript/Electron, etc)</h3>
+ The language with the most number of developers in today's scenario (biased as per my sample set)<br />
+ Excellent support for multiple natural languages, glyphs, diacritics, etc. Mature frameworks, processes for i18n, l10n, etc.<br />
- Slow and hungry for memory/cpu<br />
- Using nodejs for CLI applications though not unheard of, is not popular either (compared to the adoption of GUI applications via Electron)<br />
<h3 style="text-align: left;">
D / Haskell / Rust / C#</h3>
These languages may be great but these are niche and are not widely used by a large number of programmers in my circle.<br />
<br />
Rust although is claimed to be gaining momentum for a long time now, I cannot recollect any popular application that I use everyday done with Rust. I have stopped using Firefox (in favor of Brave) and so not really sure what the current performance status of Rust is. I will however, keep a watch on Rust and would try to learn at some point of time, in near future.<br />
<h3 style="text-align: left;">
Golang</h3>
+ Almost every new CLI tool that I have used fresh in the last 5 years or so, is done in Golang (docker, kube*, hugo, helm, etc)<br />
+ Excellent libraries: <a href="https://github.com/spf13/cobra" target="_blank">Cobra</a> and <a href="https://github.com/spf13/viper" target="_blank">Viper</a> (Thanks to spf13)<br />
+ Super simple to distribute. Static binaries.<br />
+ Automatic memory management and highly performant.<br />
+ Strong community presence<br />
- Lack of generics is a pain. It is not a problem when developing HTTP servers, REST services etc. but definitely irritates when writing libraries<br />
- Platform independence may be <a href="https://fasterthanli.me/blog/2020/i-want-off-mr-golangs-wild-ride/" target="_blank">questionable</a>. If the CLI tool is for something mission critical (a large number of users, lives depend on it, etc.), some of the discrepancies mentioned here may be dangerous.<br />
<br />
<h3 style="text-align: left;">
Conclusion</h3>
I will choose Golang if it is for dayjob. I would however choose Kotlin if the development is for a hobby/pet project to learn.<br />
<br />
What will you choose, What merits/demerits do you see for these (or other) languages ? Also what libraries/frameworks will you choose for your language(s) of choice ?<br />
<br /></div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com3tag:blogger.com,1999:blog-28654204.post-38722648133983031072019-02-07T15:03:00.001+05:302019-02-07T15:03:51.989+05:30HTTP Query Params 101<div dir="ltr" style="text-align: left;" trbidi="on">
Target Audience: Beginners / Novice<br />
<h4 style="text-align: left;">
Summary</h4>
A long time ago, we had simpler lives with our monolithic apps talking to relational databases. SQL supported having myriad conditions with the WHERE clause and conditions. As time progressed, every application became a webapp and we started developing HTTP services talking JSON, consumed by a variety of client applications such as mobile clients, browser apps etc. So, some of the filtering that we were doing via SQL WHERE clauses now needed a way to be represented via HTTP query parameters. This blog post tries to explain the use of HTTP Query Parameters for newbie programmers, via some examples. This is NOT a post on how to cleanly define/structure your REST APIs. The aim is to just give an introduction to HTTP Query Parameters.<br />
<h4 style="text-align: left;">
Action</h4>
Let us build an ebooks online store. For each book in our database, let us have the following data:<br />
<blockquote class="tr_bq">
BookID - String - Uniquely identifies a book<br />Title - String<br />Authors - String Array<br />Content - String - Base64 encoded content of the book<br />PublishedOn - Date<br />ISBN - String<br />Pages - Integer - Number of pages in the book</blockquote>
Let there be an API to get a list of books. It would be something like:<br />
<blockquote class="tr_bq">
<b>GET https://api.example.com/books</b></blockquote>
The above API will return all the information about each of the book in our system except the Content. Though this would work for a small book shop, if you have like a billion books, this puts unnecessary stress on the server, client and the network bandwidth to hold all the book data when the user is probably not bothered to see more than, say 10 titles, in most cases. So our API could now needs a way to return only N titles. Also, from which position, we need to return the N titles also needs to be specified, say Mth position. These fields are called limit and offset usually. So our API becomes:<br />
<br />
<b>GET https://api.example.com/books?offset=5&limit=10</b><br />
<br />
Here we have added two fields, offset and limit to our API. However there are two things that are unclear in this API definition.<br />
<br />
The first ambiguity is: We do not know which field will be used for finding the sequence of the books. Is it the BookID ? Is it the PublishedOn Date ? The former is a string, how do we sort it to find the order (alphabetically in case-sensitive way or insensitive way). The latter is a date field and there can be multiple books which have the same published date. So, how do ensure that a book will always be in the same position in the sort order between two different HTTP requests ?<br />
<br />
The second ambiguity is: What if these fields are not specified or if specified with invalid values ? How does the API handle it ?<br />
<br />
To solve both of these ambiguities, our API docs need to become more precise. One possible solution (out of many solutions) to address the first ambiguity is, we will always generate only BookIDs with lower case strings and will always do a toLower conversion. We will always use BookID as the sort order and always will sort in ascending order. Our BookIDs field will always be monotonically increasing; IOW, once we have given a BookID of "<b>abc</b>" to a Book, we would never generate a BookID of "<b>aba</b>" again.<br />
<br />
Instead of the String unique ID, we could also use numeric fields, which could directly map to a database <a href="https://dev.mysql.com/doc/refman/8.0/en/example-auto-increment.html" target="_blank">AUTO_INCREMENT</a> or <a href="https://www.postgresql.org/docs/11/datatype-numeric.html#DATATYPE-SERIAL" target="_blank">BIGSERIAL</a> field and ORMs can intelligently map the offset, limit fields automatically.<br />
<br />
Note that it is not uncommon to add "<b>sort_by</b>" and/or "<b>order_by</b>" requirements, where the clients can choose to change the sort field (Publication Date instead of BookID) and also the sorting order (ascending or descending). There are multiple ways to represent this via the query parameters. Some examples are:<br />
<br />
<blockquote class="tr_bq">
<b>Sort by Title (default ascending): </b></blockquote>
<blockquote class="tr_bq">
GET https://api.example.com/books?sort_by=title</blockquote>
<br />
<blockquote class="tr_bq">
<b>Sort by Title (explicitly ascending): </b></blockquote>
<blockquote class="tr_bq">
GET https://api.example.com/books?sort_by=asc(title)</blockquote>
<blockquote>
GET https://api.example.com/books?sort_by=+(title) </blockquote>
<blockquote>
GET https://api.example.com/books?sort_by=title.asc</blockquote>
<div>
<br />
<blockquote class="tr_bq">
<b>Sort by Title (explicitly descending): </b></blockquote>
<blockquote class="tr_bq">
GET https://api.example.com/books?sort_by=desc(title)</blockquote>
<blockquote class="tr_bq">
GET https://api.example.com/books?sort_by=-(title) </blockquote>
</div>
<div>
<blockquote class="tr_bq">
GET https://api.example.com/books?sort_by=title.desc</blockquote>
<br />
<blockquote class="tr_bq">
<b>Sort by Multiple Fields:</b> </blockquote>
<blockquote class="tr_bq">
<b></b>GET https://api.example.com/books?sort_by=asc(title),desc(published_on) </blockquote>
<blockquote class="tr_bq">
GET https://api.example.com/books?sort_by=title,-published_on </blockquote>
<blockquote class="tr_bq">
GET https://api.example.com/books?sort_by=+title,-published_on </blockquote>
</div>
<br />
For solving the second ambiguity, that we saw earlier, the safest solution is to make our HTTP APIs return 400 incase we come across invalid data, For example, if an invalid starting offset is given. We also need to explicitly document the default values for these query parameters, if nothing is specified.<br />
<h4 style="text-align: left;">
<b>Filters</b></h4>
We have used the GET above to get all the Books information and filter based on the cardinality. However, there may be a need for other filters. For example, we want to get the books by only a particular author. So we could add more parameters, such as:<br />
<blockquote class="tr_bq">
<b>GET https://api.example.com/books?author=crichton</b></blockquote>
Here the author is a query parameter which takes a string as an argument. This API will return any book whose author name matches "crichton". Also note that, these individual filters could be then combined with other filters, for example:<br />
<blockquote class="tr_bq">
<b>GET https://api.example.com/books?author=crichton&offset=0&limit=5</b></blockquote>
will return the first five books of author "crichton". So the API implementation in backend should apply the "limits" and "offset" <b>after</b> applying the author="crichton" filter. The API docs need to convey this in an unambigous way on what positions the "offset" would work if there are other filter conditions. The other choice is to return the books by "crichton" in the first five results in the list of all books. All your APIs need to be consistent and you can choose either of the practices, even though I prefer the former.<br />
<h4 style="text-align: left;">
<b>More Filter conditions</b></h4>
In the above API definition, we were returning the books whose author was "crichton" exactly. However, it may not be always possible to give an Exact Equals condition for our API. Our API may require to accept query parameters which should be loosely applied. For example, the author may be stored in our system as "Michael Crichton" so applying "crichton" may not be sufficient. Similarly, we may need to get a list of all books published after 2005 but before 2015.<br />
<br />
Our query parameters, in addition to "equal-to" may need to support, less-than, less-than-or-equal-to, greater-than, greater-than-or-equal-to, not-equal-to, contains (for string matches), not-contains and so on.<br />
<br />
Our query parameter need to pass these operators too in addition to the parameter name and the desired value(s). One possible approach for this could be to add these operator names to the query parameter. For example:<br />
<br />
<blockquote class="tr_bq">
<b>GET https://api.example.com/books?published_on[gte]="2005-01-01"&published_on[lte]="2015-12-31"</b></blockquote>
<blockquote class="tr_bq">
<b>GET https://api.example.com/books?published_on.gte="2005-01-01"&published_on.lte="2015-12-31"</b></blockquote>
<div>
In the above two examples, we have added "lte" and "gte" to denote less-than-or-equal-to and greater-than-or-equal-to respectively, to the query parameter name. We use a standard separator to identify the operator from the parameter name. A [] in the first case and a "." in the second case. There are libraries in most programming languages, to automatically parse these field names conveniently. For example, from the popular <a href="https://github.com/ljharb/qs" target="_blank">qs</a> library for node, </div>
<div>
<br /></div>
<div>
<pre style="background-color: #f6f8fa; border-radius: 3px; box-sizing: border-box; color: #24292e; font-family: SFMono-Regular, Consolas, "Liberation Mono", Menlo, Courier, monospace; font-size: 13.6px; line-height: 1.45; overflow-wrap: normal; overflow: auto; padding: 16px; word-break: normal;"><span class="pl-smi" style="box-sizing: border-box;">assert</span>.<span class="pl-en" style="box-sizing: border-box; color: #6f42c1;">deepEqual</span>(<span class="pl-smi" style="box-sizing: border-box;">qs</span>.<span class="pl-c1" style="box-sizing: border-box; color: #005cc5;">parse</span>(<span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">'</span>published_on[lte]="2015-12-31"<span class="pl-pds" style="box-sizing: border-box;">'</span></span>), {
published_on<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">:</span> {
lte<span class="pl-k" style="box-sizing: border-box; color: #d73a49;">:</span> <span class="pl-s" style="box-sizing: border-box; color: #032f62;"><span class="pl-pds" style="box-sizing: border-box;">'</span>2015-12-31<span class="pl-pds" style="box-sizing: border-box;">'</span></span>
}
});</pre>
</div>
<br />
Note that I am using YYYY-MM-DD as the date format. It is strongly recommended to use a single date format for all your APIs, whichever format you choose. Similarly, while working with time, choose a single timezone, preferably UTC.<br />
<br />
Instead of changing the parameter names, we can add the operator to the operand on the RHS of the equal sign too. For example:<br />
<blockquote class="tr_bq">
<b>GET https://api.example.com/books?published_on=gte:"2005-01-01"&published_on=lte:"2015-12-31"</b></blockquote>
Here we are using "gte" and "lte" to denote the operator but specify it in the RHS of the equal symbol.<br />
<br />
If you have a long list of filters and operators, you should probably avoid using HTTP Query Parameters. An API that receives these complex query strings, perhaps with your own DSL, (either as JSON or any other serialisable format), as a HTTP Request body, would make code maintenance simpler. Elasticsearch <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl.html" target="_blank">uses</a> this method.<br />
<br />
<h4 style="text-align: left;">
Conclusion</h4>
I have been trying to write this for some time now but kept on deferring for months now. So I decided to just type it out today in a stretch. The post is not as fine as I wanted it to be, but it is better to at least write an unrefined post than not writing anything at all. I hope you have found this useful. Let me know if you have any comments, feedback or corrections in this post. Thanks if you have read till here.</div>
Sankar Phttp://www.blogger.com/profile/08027774078620587372noreply@blogger.com2tag:blogger.com,1999:blog-28654204.post-67023098565605124072017-09-11T13:53:00.001+05:302017-09-11T13:53:52.160+05:30Containers 101<div dir="ltr" style="text-align: left;" trbidi="on">
The term "containers" became popular in the recent times, thanks to <a href="https://www.docker.com/" target="_blank">Docker</a>. However, the idea of containers is there for long, through things like:<a href="https://en.wikipedia.org/wiki/Solaris_Containers" target="_blank"> Solaris Zones</a>, <a href="https://linuxcontainers.org/" target="_blank"> Linux Containers</a>, etc. (even though the underlying implementations are <a href="https://blog.jessfraz.com/post/containers-zones-jails-vms/" target="_blank">different</a>). In this post, I try to give a small overview of the containers ecosystem (as it stands in 2017), from my perspective.<br />
<br />
This post is written in response to a question by, hacker extraordinaire, <a href="http://blog.varunkumar.me/" target="_blank">Varun</a> on what one should know about Containers as of today. Though the document is mostly generic, some lines of it are India specific, which I have highlighted clearly. Please mention in comments, if there is anything else that should have been covered, or if I have made any mistakes or if you have any opinions.<br />
<br />
<h4 style="text-align: left;">
So, What exactly are Containers ?</h4>
<i>Containers are an unit of packaging and deployment, that will guarantee repeatability and isolation</i>. Let us see what each part of that sentence means.<br />
<br />
Containers are a packaging tool like <a href="https://en.wikipedia.org/wiki/Rpm_(software)" target="_blank">RPMs</a> or <a href="https://en.wikipedia.org/wiki/EAR_(file_format)" target="_blank">EARs</a> in the sense that they offer you a way to bundle up your binaries (or sources in case of interpreted languages). But instead of merely archiving your sources, Containers provide a way to even deploy your archive, repeatably too.<br />
<br />
Anyone who has done packaging, knows, how much of a pain <a href="https://en.wikipedia.org/wiki/Dependency_hell" target="_blank">dependency-hell</a> can cause. For example, An application A needs a library L of version 0.1, whereas another application B needs the same library L but of version 0.3 Just to screw up the life of packagers, the versions 0.1 and 0.3 may be conflicting each other and may not co-exist in a system, even in different installation paths. Containerising your application puts each of these applications A and B into their own bundle, with their own library dependencies. However, the real power of containerising is that for each of your application, A and B, they get a view of isolation that they are running in a private environment and so L1 0.1 and 0.3 may never share any runtime data.<br />
<br />
<br />
One may be reminded about Virtual Machines (VMs) while reading the above text. Even VMs solve the above isolation problem, but they are very heavy. The fundamental difference between a VM and a Container is: a VM virutalizes/abstracts a hardware/operating-system and gives you a machine abstraction, while a Container virtualizes/abstracts an application of your choice. Containers are thus very lightweight and far more approachable.<br />
<br />
<h4 style="text-align: left;">
The Ecosystem</h4>
Docker is the most used container technology today. There are other container runtimes such as <a href="https://github.com/rkt/rkt" target="_blank">rkt</a> too. There is an <a href="https://www.opencontainers.org/" target="_blank">Open Containers Initiative</a> to create standards for container runtimes. All these container runtimes make use of linux kernel features, especially, <a href="https://www.kernel.org/doc/Documentation/cgroup-v2.txt" target="_blank">cgroups</a> to provide process isolation. Microsoft has been making a lot of efforts to support containers natively in the Windows kernel, to support Containers natively as part of their Azure cloud offering for quite some time now.<br />
<br />
<b><i>Container Orchestration</i></b> is a way for deploying different containers on a bunch of machines. While Docker is arguably the champion of container runtimes, <a href="https://kubernetes.io/" target="_blank">Kubernetes</a> is unarguably the King/Queen of container orchestration. Google has been using containers in production, for much long before it became fashionable. In fact the first patch of cgroups support in the linux kernel was <a href="https://lwn.net/Articles/199643/" target="_blank">submitted</a> to LKML by Google as far back as 2006. Google had/s a large scale cluster management system named <a href="https://research.google.com/pubs/pub43438.html" target="_blank">Borg</a> which deployed containers (not docker containers) across the humongous google cloud farm. Kubernetes is an <a href="https://github.com/kubernetes/kubernetes" target="_blank">open source evolution</a> of Borg, supporting Docker containers natively. Docker-Swarm is an attempt by Docker (the company behind the Docker project) to achieve container orchestration across machines, but there simply is no competition in terms of quality or documentation or feature coverage, compared to Kubernetes (in my limited experience).<br />
<br />
Also, in addition to these, There are some poorly implemented, company-specific tools that try to emulate Kubernetes, but these are mostly technical debt and it is wise (imho) for companies to ditch such efforts and move to open projects backed by companies like Google, Red Hat and Microsoft. A <a href="https://twitter.com/larsmb" target="_blank">distinguished engineer</a> once told me, <i>There is no compression algorithm for experience</i> and there is no need for us to repeat the mistakes made by these companies, decades ago. If you are a startup focussing on solving an user problem, you should focus on your business problem and a container orchestration software should be the last thing that you need to implement.<br />
<br />
Kubernetes, though initially a Google project, has now attracted a lot of contributors from a variety of companies such as Red Hat, Microsoft etc. Red Hat have built <a href="https://www.openshift.com/" target="_blank">OpenShift</a>, a platform that provides a lot of useful features such as, Pipelines, Blue-Green deployments, etc. on top of Kubernetes. They even offer a <a href="https://www.openshift.com/pricing/index.html" target="_blank">hosted</a> version. <a href="https://coreos.com/tectonic" target="_blank">Tectonic</a> (on top of Kubernetes) by Core OS is also a big (at least in terms of developer mindshare) player in this ecosystem.<br />
<br />
SUSE has come up recently with the <a href="https://news.opensuse.org/2017/05/29/introducing-kubic-project-a-new-open-source-project/" target="_blank">Kubic</a> project for containers (even though I have not played with it myself).<br />
<br />
Microsoft have hired some high profile names in the container ecosystem for working on the Kubernetes + Azure (Including people like: <a href="https://www.linkedin.com/in/brendan-burns-487aa590/" target="_blank">Brendan Burns</a>, <a href="https://twitter.com/jessfraz/status/904710675779514368" target="_blank">Jess Frazelle</a>, etc.) cloud. Azure is definitely way ahead of Google in India, when it comes to cloud business. Their pricing page is localised for India, while Google does not even support Indian currency yet and charges in USD (leading to jokes like the oil/dollar conspiracy, among the Indian startup ecosystem ;) ). AWS and Azure definitely have a bigger developer mindshare in India than Google Cloud Platform (as of 2017).<br />
<br />
The founding team of kubernetes (Xooglers) have started a company named <a href="https://blog.heptio.com/founding-members-of-kubernetes-team-create-company-heptio-to-bring-kubernetes-and-containers-to-5c185724f412" target="_blank">Heptio</a>. While I have no doubts on their engineering prowess, I am skeptical if relying on these companies may be risky for startups in India (lack of same timezone support, etc.). If you are in the west, these options (and others such as <a href="http://rancher.com/" target="_blank">rancher</a>) may be interesting.<br />
<br />
<h4 style="text-align: left;">
Kubernetes Basics</h4>
In Kubernetes, the unit of deployment is a Pod. A pod is merely a collection of Docker containers which will be deployed together always. For example, if your application is a API server that makes use of a Redis cache, before hitting the database for each request, you create a Pod with two containers, a API server container and a Redis container and you deploy them together.<br />
<br />
Kubernetes refers to an umbrella of projects that run on a cloud, to manage a cloud. It has various components, such as an API server to interact with the kubernetes system, an agent software named kubelet that runs on each machine in the cloud, a fluentd type of daemon to accumulate logs from various containers and provide a single point of access, a web dashboard, a CLI tool named <a href="https://kubernetes.io/docs/user-guide/kubectl-overview/" target="_blank">kubectl</a> to perform various options, etc. In addition to these kubernetes specific components, there are also other services, such as the distributed hashstore <a href="https://github.com/coreos/etcd" target="_blank">etcd</a> (originally from coreos) that you need to setup a basic kubernetes cluster. However, If you are a small company, It'll be wise to make use of GKE or Azure hosting or OpenShift hosting instead of deploying your own kubernetes system managed by your own admins. It is not worth the hassle.<br />
<div>
<br /></div>
<div>
If you want to play with kubernetes in your development laptop (unless you can afford to treat production as your test box), there is a tool named <a href="https://github.com/kubernetes/minikube" target="_blank">minikube</a> to help you with that. If you are an application developer and considering to dockerizing and deploying your application, then minikube is definitely the best place to start.</div>
<div>
<br /></div>
<div>
There are quite a few meetups happening for kubernetes all around the world. Visiting some of these may be enlightening. The webinar <a href="https://www.youtube.com/watch?v=_vHTaIJm9uY" target="_blank">series</a> by Janakiram was good, but it is a little too long to my taste and I lost interest halfway. The persistent ones among you may find it very useful.</div>
<h4 style="text-align: left;">
Docker Compose</h4>
<div>
One of the tools from the Docker project that I love a lot is the handy <a href="https://docs.docker.com/compose/" target="_blank">Docker Compose</a>. It is a tool to work with multiple containers, in a sense it is somewhat like your kubernetes Pods, but without having to install / manage the heavyweight kubernetes ecosystem. I use Docker Compose extensively in CI, where it is the perfect fit for doing end-to-end testing of a webstack, if your sources are in a monolithic repository. In your CI system, you can bring up all your components (say, an API server, a database, a front end node server) and perform an end-to-end testing (say, via selenium). In fact, I cannot fathom how I was doing CI earlier without docker-compose, (just like how I cannot fathom how I used cvs before git, etc.)</div>
<div>
<br /></div>
<h4 style="text-align: left;">
AWS</h4>
<div>
No blog post on cloud technologies will be complete, without mentioning the 800 pound gorilla, Amazon Web Services. Amazon supports containers natively. You can deploy either a single container or multi-container images natively, via <a href="https://aws.amazon.com/elasticbeanstalk/" target="_blank">Amazon Beanstalk</a>. It is very much similar to the Google Appengine (if you have used it). Beanstalk is a PaaS offering and it takes a Container image and scales it automagically depending on various factors (such as CPU usage, HTTP usage, etc.). I've run Beanstalk and is very satisfied with it (perhaps not as much as with AppEngine though). It is very reliable, performant and scales well (tested for a few hundred users in my limited experience).</div>
<div>
<br /></div>
<div>
For the larger workloads and those who want more control, Amazon offers <a href="https://aws.amazon.com/ecs/" target="_blank">Elastic Container Service</a>. You can create a bunch of EC2 instances and a bunch of Containers, and ask ECS to run these containers on these VMs in a way that you prefer. This, however locks you to the AWS platform (unlike k8s).</div>
<div>
<br /></div>
<div>
Both Beanstalk and ECS do not cost anything extra other than the price of VMs, which you already pay.</div>
<div>
<br /></div>
<div>
I, however, wish that Amazon starts supporting kubernetes natively. There are other ways to make use of kubernetes in AWS. The most enterprisey is probably <a href="https://coreos.com/tectonic" target="_blank">Tectonic</a> by Core OS, but we also have projects like <a href="https://github.com/kubernetes-incubator/kube-aws" target="_blank">kube-aws</a> and <a href="https://github.com/kubernetes/kops" target="_blank">kops</a>.</div>
<div>
<br /></div>
<h4 style="text-align: left;">
Conclusion:</h4>
<div>
If you have actually read until this point, Thanks a lot :-) I could have written a little bit in detail about the nuts and bolts of the containers technology, but I believe that this post, as is, will be a good material for a 101 type of introduction. Also, there are people with far more working knowledge than me, who are more equipped to write on the details. So, I have left it as an exercise to the readers to find such talks, blogs or books :)</div>
</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com10tag:blogger.com,1999:blog-28654204.post-18110178673834987132017-06-14T11:01:00.000+05:302017-06-14T11:13:53.436+05:30golang range Tickers<div dir="ltr" style="text-align: left;" trbidi="on">
<b>Update</b>: Please use the playground/gist urls for reading code. Blogger's code formatting is terrible and does not support embedding gists either.<br />
<br />
Yesterday <a href="https://github.com/pshirali">Praveen</a> sent me an interesting piece of golang code. Read the following code and tell what the answer will be:
<br />
<br />
===<br />
type LED struct {<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>state bool<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>ticker *time.Ticker<br />
}<br />
<br />
func toggle(led *LED) {<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>led.state = !led.state<br />
}<br />
<br />
func looper(led *LED) {<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>for range led.ticker.C {<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>toggle(led)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>}<br />
}<br />
<br />
func main() {<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>fmt.Println("Initial number of GoRoutines: ", runtime.NumGoroutine())<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>led := &LED{state: true, ticker: time.NewTicker(time.Millisecond * 500)}<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>go looper(led)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>fmt.Println("Number of GoRoutines after a call to looper: ", runtime.NumGoroutine())<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>time.Sleep(2 * time.Second)<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>led.ticker.Stop()<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>fmt.Println("Number of GoRoutines after stopping the ticker: ", runtime.NumGoroutine())<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>runtime.GC()<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>fmt.Println("Number of GoRoutines after gc: ", runtime.NumGoroutine())<br />
<br />
}<br />
===<br />
<br />
<br />
<script src="https://gist.github.com/psankar/8af76ba183b0203ec141bca8156f5955.js"></script>
Golang playground URL: https://play.golang.org/p/1as5QN1r2c <br />
Gist URL: https://gist.github.com/psankar/8af76ba183b0203ec141bca8156f5955
<br />
<br />
I will explain roughly what the code is doing.<br />
<br />
There is a LED struct which has a Ticker and a state variable. While creating an instance of the led struct, we initialise the state and the Ticker. There is a looper function will toggle the state, whenever the Ticker fires an event.<br />
<br />
Now when the program is launched, there will be one goroutine (the initial main thread). After we call looper in a goroutine, the goroutineCount will be 2. Now, comes the tricky part. We stop the Ticker, after a particular amount of time. We even call the gc.<br />
<br />
It was observed by Praveen that this piece of code was leaking go routines and the number of go routines was never going down, inspite of the Ticker getting stopped.<br />
<br />
The reason why the leakage is happening is because, the "range" loop is never exiting. If the range loop was on a channel, you could "<a href="https://tour.golang.org/concurrency/4" target="_blank">close</a>" it. The ticker.C channel however is a receive only channel and you cannot close it.<br />
<br />
How do we fix this, so that none of the goroutines are leaking ? If you have watched the talks, <a href="https://blog.golang.org/advanced-go-concurrency-patterns" target="_blank">golang concurrency patterns by Rob Pike and Advanced golang concurrency patterns by Sameer Ajmani</a>, then you will realise that it is quite easy to add another parameter to the looper function, which could just exit the loop. So the updated code will be:<br />
<br />
===<br />
type LED struct {<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>state bool<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>ticker *time.Ticker<br />
}<br />
<br />
func toggle(led *LED) {<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>led.state = !led.state<br />
}<br />
<br />
func looper(led *LED) {<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>for range led.ticker.C {<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>toggle(led)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>}<br />
}<br />
<br />
func looper2(led *LED, q chan bool) {<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>for {<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>select {<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>case <-led.ticker.C:<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>toggle(led)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>case <-q:<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>fmt.Println("Exiting the goroutine")<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>return<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>}<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>}<br />
}<br />
<br />
func main() {<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>fmt.Println("Initial number of GoRoutines: ", runtime.NumGoroutine())<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>led := &LED{state: true, ticker: time.NewTicker(time.Millisecond * 500)}<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>q := make(chan bool)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>go looper2(led, q)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>// go looper(led)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>fmt.Println("Number of GoRoutines after a call to looper: ", runtime.NumGoroutine())<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>time.Sleep(2 * time.Second)<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>led.ticker.Stop()<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>fmt.Println("Number of GoRoutines after stopping the ticker: ", runtime.NumGoroutine())<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>q <- true<br />
fmt.Println("Number of GoRoutines after sending a message on the quit channel: ", runtime.NumGoroutine())<br />
}<br />
===<br />
<br />
<script src="https://gist.github.com/psankar/4e5b2e563038ce3e9c17eb208c76168a.js"></script>
<br />
Playground URL: https://play.golang.org/p/NlWbyHLHvA <br />
Gist URL: https://gist.github.com/psankar/4e5b2e563038ce3e9c17eb208c76168a
<br />
<br />
Let me know if you have any comments.</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com0tag:blogger.com,1999:blog-28654204.post-76626498371391406832016-11-11T14:29:00.000+05:302016-11-11T14:29:11.210+05:30Conversations with self while "Learning Reactjs"<div dir="ltr" style="text-align: left;" trbidi="on">
I implemented a bunch of APIs in Go. Took about a few hours in the night. Let me add a web client to these. May be I will learn to build a SPA. Which poison to choose from ? React, Angular 1.x, Angular 2.x, Vue ?<br />
<br />
<i>> Go with react. That is what all the cool kids are using. Also, something to do with: <a href="https://medium.freecodecamp.com/angular-2-versus-react-there-will-be-blood-66595faafd51#.rvrvzi461" target="_blank">Angular 2 continues to put “JS” into HTML. React puts “HTML” into JS.</a> sounds geeky and logical.</i><br />
<br />
Okay. Let me start with this react. Where do I even begin ? Seems very complex.<br />
<br />
<i>> Alright. There is this <a href="https://github.com/facebookincubator/create-react-app" target="_blank">create-react-app</a> which is introduced by Facebook to make it easy to begin, so that you do not have to break your head about gulp, grunt, node, etc. and their magical version incompatibilities</i><br />
<br />
I started with this create-react-app and went a little further. I can create various components and render them, but how do I get various views/components to interact, to form a workflow (say such as sharing a session string or so) ?<br />
<br />
<i>> This is where state management comes in. You need to maintain state in a nice way centrally. You need to use the <a href="https://facebook.github.io/react/docs/lifting-state-up.html" target="_blank">Flux</a> architecture, introduced by Facebook.</i><br />
<br />
Cool. So I just use the flux library from Facebook and things will all fall in place ?<br />
<br />
<i>> Actually flux is a standard, but everyone uses <a href="http://redux.js.org/" target="_blank">Redux</a> which is an implementation of this standard. Oh, btw there are lot of other implementations such as <a href="http://alt.js.org/" target="_blank">alt</a>. The creator of the redux seem to be an active guy and helps in the community often, writes long <a href="http://stackoverflow.com/questions/32461229/why-use-redux-over-facebook-flux" target="_blank">stackoverflow</a> posts, etc. How can someone who write long posts be wrong ?</i><br />
<br />
Hm. Okay. Let me start with this redux. What should I understand ?<br />
<br />
<i>> It is simple. If you understand: Global store, Reducers, Actions, Dispatch, Containers, you have understood redux. Just follow <a href="https://egghead.io/courses/getting-started-with-redux" target="_blank">these egghead tutorials</a>. </i><br />
<br />
Okay. I tried following these. They are really beginner <b>un</b>friendly. Actually <a href="https://www.youtube.com/watch?v=DiLVAXlVYR0&feature=youtu.be&a" target="_blank">this series</a> on youtube is better, though a bit out-dated and non-standard. I have now done a simple redux toy app.<br />
<br />
> Try a complex app, with multiple pages and talk to that API that you implemented.<br />
<br />
Good idea. I will start with it. Oh, wait. My component has a lot of buttons, text boxes, etc. I need a way to get something rudimentary, like: <a href="http://stackoverflow.com/questions/40521333/redux-get-input-text-value/40521454" target="_blank">getting the value from the username and password input</a> boxes, when a "Login" button is clicked. Do I need to make a mess of global states and private-component-specific states ? That is like so contrary to what we learnt so far.<br />
<br />
<i>> Think again. Is there any alternative for this ?</i><br />
<br />
May be I can have Actions, ActionCreators and State variables for each field in each component, centrally maintained in the global store ? That will be a looooot of boilerplate.<br />
<br />
<i>> Ahem. May be you should start using <a href="https://redux-form.com/" target="_blank">redux-form</a> library. It will minimize your workload and optimizes the boilerplate.</i><br />
<br />
Is it well maintained ? It has plenty of github stars but what if the bus factor is high and the primary author loses interest when he gets a dayjob somewhere else ? Also, it is already in version 6. Isn't react itself announced just three years ago ? Why is there so 6 major versions of this library already ? Will this change again if I depend on it ?<br />
<br />
<i>> Hrm. How long has it been since you began the exercise ?</i><br />
<br />
It has been about a month or so, learning only during the latenights (after the dayjob and getting kid to sleep etc.) and occasionally weekends. Already I am tired. May be this javascript-fatigue is real.<br />
<br />
<i>> Now that you have learnt React and Redux, and experienced first-hand how much it takes time to identify the quintessential combination of libraries, you should not attempt to build anything in your free time, you should be careful in choosing these technologies, for a proper dayjob.</i><br />
<br />
If all these javascript fatigue posts are to be believed, the alternatives are equally bad if not worse. Angular 2 <a href="https://yakovfain.com/2016/08/22/a-grouchy-writeup-on-the-angular-2-situation/" target="_blank">broke APIs in RC</a> stage, does not offer guarantee to not break APIs even after release it seems. Anyways, I started this project to learn about react and I can say, <b><i>I know my way around react</i></b>. It is a different question if I want to choose UI programming as a full time profession at all. The current flux (not to be confused with the architecture) of things makes it extremely painful. I think <b>people choose mobile first development, not because of product requirements but because of javascript fatigue</b>.<br />
<br />
PS: A lot of things where I had to take a detour and wasted a lot of time is trimmed from the post, as they are anyway not directly related to ReactJS</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com3tag:blogger.com,1999:blog-28654204.post-57878790076297293442016-04-27T16:32:00.003+05:302016-04-27T16:32:56.309+05:30[Help Needed] FOSS License, CLA Query<div dir="ltr" style="text-align: left;" trbidi="on">
I want to start a FOSS project. FOSS Licenses are a grey area. I am trying to seek some public opinion here, to choose a license and a Contributor License Agreement (CLA). The project details are:<br />
<br />
<ul style="text-align: left;">
<li>The project is a database (say, like mongodb, Cassandra etc.). It will have a server piece that users can deploy for storing data. Though it is a hobby personal project as of now, I may offer the database as a paid, hosted solution in future.</li>
<li>There are some client libraries too, for providing the ability to connect to the above mentioned server, from a variety of programming languages.</li>
<li>The client libraries will all be in Creative Commons Zero License / Public Domain. Basically anyone can do anything with the client library sources. The server license is where I have difficulty choosing.</li>
<li>Anyone who contributes any source to the server software should re-assign their copyrights and ownership of the code, to me. By "me", I refer to myself as an individual and not any company. I should reserve the right to transfer the ownership in future to anyone / any company. I may relicense the software in future to public domain or sell it off to a company like: SUSE, Red Hat, Canonical, (or) a company like: Amazon, Google, Microsoft etc.</li>
<li>Anyone who contributes code to my project, should make sure that [s]he has all the necessary copyrights to submit the changes to me and to re-assign the copyrights to me. I should not be liable for someone's contribution. If a contributor's employer has a sudden evil plan and want to take over my personal project to court (unlikely to happen, nevertheless), it should not be possible</li>
<li>I or the users of the software, should not be sued for any patent infringement cases, for code that is contributed by someone else. If a patent holder wants to sue me for a code that I have written in the software, that is fine. I will find a way around.</li>
<li>Anyone should be free to take the server sources, modify it and deploy it in his/her own hardware/cloud, for their personal and/or commercial needs, without paying me or any of the contributors any money/royalty/acknowledgement.</li>
<li>If they choose to either sell the server software or host it and sell it as a service, (basically commercial reasons) they must be enforced to open source their changes in public domain, unless they have a written permission from me, at my discretion. For instance, if coursera wants to use my database source, after modifications, it is fine with me; but I will not want, say Oracle to modify my software and sell the modified software / service, without opensourcing their changes. If someone is hosting and selling a service of my software, with modified sources, there is no easy way for me to prove their modification, but I would still like to have that legal protection.</li>
</ul>
<br />
The best license model that I could come up for the above is: <b>Dual license the source code to <a href="https://en.wikipedia.org/wiki/Affero_General_Public_License" target="_blank">AGPLv3</a> and a proprietary license. Enforce a CLA to accept all contributions only after a copyright reassignment to me, with a guarantee that I have the right to change the license at a future time.</b><br />
<br />
What is not clear to me however, is the patent infringement and ownership violation related constraints and AGPL's protection on such disputes. Another option is: <a href="https://www.mozilla.org/en-US/MPL/2.0/" target="_blank">Mozilla Public License 2.0 </a>but that does not seem to cover the hosting-as-a-service-and-selling-the-service aspect clearly imho.<br />
<br />
Are you readers of the internet have any better solution ?<br />
<br />
Are you aware of any other project using any other license, CLA model that may suit my needs and/or is similar ?<br />
<br />
What other things should I be reading to understand more ?<br />
<br />
Or, should I lose all faith in licenses and keep the sources private and release the binary as freeware, instead of open sourcing ? That would suck.<br />
<br />
Or should I just not bother about someone making proprietary modifications and selling the software/service, by releasing the software to public domain ?<br />
<br />
<b>Note</b>: Of course, all these is assuming that my 1 hour a month, hobby project would make it big, be useful to others and someone may sue. In reality, the software may not be tried by even a dozen people, but I'm just romanticizing.</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com11tag:blogger.com,1999:blog-28654204.post-52147793145656227762016-03-02T17:26:00.003+05:302016-03-02T17:26:26.107+05:30Programmers guide to Microservices/SOA<div dir="ltr" style="text-align: left;" trbidi="on">
<h3 style="text-align: left;">
Introduction</h3>
SOA or Service Oriented Architecture is one of the buzzwords among architects/senior-developers, job descriptions for the last few years. However, most of the definitions of SOA online are riddled with formal words, such as the one from OASIS, which <a href="https://en.wikipedia.org/wiki/Service-oriented_architecture#Definitions" target="_blank">says</a>: "<i>A paradigm for organizing and utilizing distributed capabilities that may be under the control of different ownership domains. It provides a uniform means to offer, discover, interact with and use capabilities to produce desired effects consistent with measurable preconditions and expectations.</i>"<br />
<div>
<br /></div>
<div>
The above definition though is precise, is too abstract for a developer. This post tries to explain what constitutes a [micro-]service oriented architecture and how it differs from a traditional monolithic approach that a programmer may be accustomed to. This post is an <b>introductory</b> material aimed at beginners to SOA, who have already done some monolithic projects. Experts in SOA, could validate the facts mentioned and suggest alternatives.<br />
<h3 style="text-align: left;">
Microservices</h3>
<div>
A very informal way to understand microservices is, if we split up every class of our design into a HTTP accessible webservice on its own, we would end up with a bunch of services, which together constitute a microservices based architecture. The difference between SOA and microservices is just the level of granularity to which you decompose your classes (in a monolithic application) into independent HTTP services. The more minimal in functionality, each of your service implementation is, the more closer it is to be called a microservice.<br />
<br />
Splitting a single application into multiple services, imposes a few restrictions on our coding, but in turn gives us a lot of flexibility and power in scaling. Let us look at some of the coding/design constraints.</div>
<h4 style="text-align: left;">
Stateless Systems</h4>
<div>
The fundamental difference from a monolithic design is in maintaining state information. All the individual classes which earlier interacted via global variables (for locks, mutexes, config variables, etc.) can no longer rely on them.<br />
<br />
Let us take a simple example, we are building a rudimentary Shopping application with just one type of item stored. The Shopping application has two parts, the Inventory part that adds new items and the Sales part that removes items. Let us consider the following pseudo-code:<br />
<br /></div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #000000; border-width: 0.1em 0.1em 0.1em 0.8em; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #00cd00;">var</span> <span style="color: #cccccc;">mu</span> <span style="color: #cccccc;">=</span> <span style="color: #3399cc;">&</span><span style="color: #cccccc;">sync.Mutex{}</span>
<span style="color: #00cd00;">var</span> <span style="color: #cccccc;">stockItemCount</span> <span style="color: #cccccc;">=</span> <span style="color: #cd00cd;">10</span>
<span style="color: #00cd00;">func</span> <span style="color: #cccccc;">(i</span> <span style="color: #3399cc;">*</span><span style="color: #cccccc;">Inventory)</span> <span style="color: #cccccc;">AddToStock(n</span> <span style="color: #00cd00;">int</span><span style="color: #cccccc;">)</span> <span style="color: #cccccc;">{</span>
<span style="color: #cccccc;">mu.Lock()</span>
<span style="color: #cccccc;">stockItemCount</span> <span style="color: #3399cc;">+=</span> <span style="color: #cccccc;">n</span>
<span style="color: #cccccc;">mu.Unlock()</span>
<span style="color: #cccccc;">}</span>
<span style="color: #00cd00;">func</span> <span style="color: #cccccc;">(s</span> <span style="color: #3399cc;">*</span><span style="color: #cccccc;">Sales)</span> <span style="color: #cccccc;">UpdateStock(n</span> <span style="color: #00cd00;">int</span><span style="color: #cccccc;">)</span> <span style="color: #00cd00;">bool</span> <span style="color: #cccccc;">{</span>
<span style="color: #cccccc;">mu.Lock()</span>
<span style="color: #cdcd00;">defer</span> <span style="color: #cccccc;">mu.Unlock()</span>
<span style="color: #cdcd00;">if</span> <span style="color: #cccccc;">stockItemCount</span> <span style="color: #3399cc;">>=</span> <span style="color: #cccccc;">n</span> <span style="color: #cccccc;">{</span>
<span style="color: #cccccc;">stockItemCount</span> <span style="color: #3399cc;">-=</span> <span style="color: #cccccc;">n</span>
<span style="color: #cdcd00;">return</span> <span style="color: #cdcd00;">true</span>
<span style="color: #cccccc;">}</span> <span style="color: #cdcd00;">else</span> <span style="color: #cccccc;">{</span>
<span style="color: #cdcd00;">return</span> <span style="color: #cdcd00;">false</span>
<span style="color: #cccccc;">}</span>
<span style="color: #cccccc;">}</span>
</pre>
</div>
<br />
<!-- HTML generated using hilite.me -->
In the above code snippet (trivialized for brevity), we have a global variable <b>stockItemCount</b>, which is protected by a mutex mu. The <b>AddToStock</b> function of the Inventory class/type, adds to this global variable whereas the <b>UpdateStock</b> function of the Sales class/type, removes from the global variable. The <b>mu</b> lock synchronizes the access, such that the functions have exclusive access to the global variable on execution.<br />
<br />
In a SOA, the Inventory and the Sales classes will become their own individual HTTP webservices. These new individual classes, viz., SalesService and InventoryService, may now run on different machines.<br />
<h4 style="text-align: left;">
Inter-Service Co-ordination</h4>
So how do these different services potentially running on different machines, share and synchronize access to common data ? The solution is simple. We move away from the globalVariable+mutex pattern and implement a publish-subscribe or queueing pattern. What does that mean ?<br />
<br />
We move the stockItemCount management into a separate StockService which is accessible by both the InventoryService and SalesService (earlier considered classes/type). Let us take a look at a sample pseudo-code:<br />
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #111111; border-width: 0.1em 0.1em 0.1em 0.8em; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="background-color: #0f140f; color: #008800; font-style: italic;">// Stock Service on Machine A</span>
<span style="color: #fb660a; font-weight: bold;">var</span> <span style="color: white;">count</span> <span style="color: white;">=</span> <span style="color: #0086f7; font-weight: bold;">10</span>
<span style="color: #fb660a; font-weight: bold;">func</span> <span style="color: white;">(s</span> <span style="color: white;">*StockService)</span> <span style="color: white;">ProcessQ()</span> <span style="color: white;">{</span>
<span style="color: #fb660a; font-weight: bold;">for</span> <span style="color: white;">{</span>
<span style="color: white;">op</span> <span style="color: white;">:=</span> <span style="color: white;">Q.Read()</span>
<span style="color: #fb660a; font-weight: bold;">if</span> <span style="color: white;">op.Type</span> <span style="color: white;">==</span> <span style="color: white;">“Inventory”</span> <span style="color: white;">{</span>
<span style="color: white;">count</span> <span style="color: white;">+=</span> <span style="color: white;">n</span>
<span style="color: white;">}</span> <span style="color: #fb660a; font-weight: bold;">else</span> <span style="color: #fb660a; font-weight: bold;">if</span> <span style="color: white;">op.Type</span> <span style="color: white;">==</span> <span style="color: white;">“Sale”</span> <span style="color: white;">{</span>
<span style="color: #fb660a; font-weight: bold;">if</span> <span style="color: white;">count</span> <span style="color: white;">>=</span> <span style="color: white;">n</span> <span style="color: white;">{</span>
<span style="color: white;">count</span> <span style="color: white;">-=</span> <span style="color: white;">n</span>
<span style="color: white;">http.POST(op.callbackURL,</span> <span style="color: #0086d2;">"success"</span><span style="color: white;">);</span>
<span style="color: white;">}</span> <span style="color: #fb660a; font-weight: bold;">else</span> <span style="color: white;">{</span>
<span style="color: white;">http.POST(op.callbackURL,</span> <span style="color: #0086d2;">"failure"</span><span style="color: white;">);</span>
<span style="color: white;">}</span>
<span style="color: white;">}</span>
<span style="color: white;">}</span>
<span style="color: white;">}</span>
<span style="color: #fb660a; font-weight: bold;">type</span> <span style="color: white;">Operation</span> <span style="color: #fb660a; font-weight: bold;">struct</span> <span style="color: white;">{</span>
<span style="color: white;">Type</span> <span style="color: #cdcaa9; font-weight: bold;">string</span>
<span style="color: white;">Value</span> <span style="color: #cdcaa9; font-weight: bold;">int</span>
<span style="color: white;">Callback</span> <span style="color: white;">URL</span>
<span style="color: white;">}</span></pre>
<br />
<br />
<br />
<pre style="line-height: 125%; margin: 0;"><span style="background-color: #0f140f; color: #008800; font-style: italic;">// Inventory Service on Machine B</span>
<span style="color: #fb660a; font-weight: bold;">func</span> <span style="color: white;">(i</span> <span style="color: white;">*InventoryService)</span> <span style="color: white;">AddToStock(n</span> <span style="color: #cdcaa9; font-weight: bold;">int</span><span style="color: white;">)</span> <span style="color: white;">{</span>
<span style="color: white;">Q.Write(Operation{“Add”,</span> <span style="color: white;">n,</span> <span style="color: #fb660a; font-weight: bold;">nil</span><span style="color: white;">})</span>
<span style="color: white;">}</span></pre>
<br />
<br />
<br />
<pre style="line-height: 125%; margin: 0;"><span style="background-color: #0f140f; color: #008800; font-style: italic;">// Sales Service on Machine C</span>
<span style="color: #fb660a; font-weight: bold;">func</span> <span style="color: white;">(s</span> <span style="color: white;">*SalesService)</span> <span style="color: white;">UpdateStock(n</span> <span style="color: #cdcaa9; font-weight: bold;">int</span><span style="color: white;">)</span> <span style="color: white;">{</span>
<span style="color: white;">Q.Write(Operation{“Remove”,</span> <span style="color: white;">n,</span> <span style="color: white;">callbackURL})</span>
<span style="color: white;">}</span>
<span style="color: #fb660a; font-weight: bold;">func</span> <span style="color: white;">(s</span> <span style="color: white;">*SalesService)</span> <span style="color: white;">POSTHandler(w</span> <span style="color: white;">http.Response,</span> <span style="color: white;">*r</span> <span style="color: white;">http.Request)</span> <span style="color: white;">{</span>
<span style="color: white;">s.Notify(r.Body)</span> <span style="background-color: #0f140f; color: #008800; font-style: italic;">// success or failure</span>
<span style="color: white;">}</span>
</pre>
</div>
<!-- HTML generated using hilite.me -->
<br />
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
As seen above, we have two Classes which are converted into Services (SalesService and InventoryService) and a new third service named StockService. We also have a Q (a distributed Queue infrastructure) that we use. We have a Operation class/type, with a Type string, whose instance we will be adding to the Q. The AddToStock function in the Inventory service, creates a new Operation item of type "Add", whereas the UpdateStock function of the SalesService creates a new Operation item of type "Remove" to the queue. The StockService has a ProcessQ function which goes on an infinite loop to fetch items from the Q and based on the Type of the operation, perform either addition or deletion of value.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
It should be clear now that the SalesService and the InventoryService are now totally stateless. They just make use of the Q to communicate with the StockService. The meticulous among the blog-readers would have observed that the StockService is still stateful. We maintain the count variable still as a global variable. In any large scale system, there may be some components, which may not be completely stateless. We will have some drawbacks because of having such stateful parts. We will discuss them more in a future section.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
The Q forms a very central part of the above architecture. The Q can be implemented by the programmer manually, and could potentially be deployed in a totally different set of machine(s) from either of A, B or C. However, there are some stable Queue implementations that we could use, instead of reinventing the wheel. <a href="http://kafka.apache.org/" target="_blank">Apache Kafka</a>, <a href="https://www.rabbitmq.com/" target="_blank">RabbitMQ</a> are some popular opensource systems. If you want a hosted solution, <a href="https://aws.amazon.com/sqs/" target="_blank">Amazon SQS</a> is offered by AWS and <a href="https://cloud.google.com/pubsub/" target="_blank">Cloud PubSub by Google</a>. These systems could be called <b>Messaging Middleware</b>.</div>
<div style="text-align: left;">
<br /></div>
<div style="text-align: left;">
There are projects where a massive amount of data will be generated (say from sensors instead of humans) and we will need realtime processing of streaming data. We could use specialized streaming middleware such as <a href="http://storm.apache.org/" target="_blank">Apache Storm</a> or a hosted solution such as <a href="http://docs.aws.amazon.com/kinesis/latest/dev/introduction.html" target="_blank">Amazon Kinesis</a>.</div>
<h3 style="text-align: left;">
Benefits of SOA</h3>
As we just saw above, what was a simple single process with two classes, became three different classes and a queueing system with four different processes across four (or more) different machines, to accommodate SOA. Why should a programmer put up with so much of complexity ? What do we get in return ? We will see some of the benefits in this section.<br />
<h4 style="text-align: left;">
Horizontal Scalability</h4>
If we have a server with 4GB RAM serving a 100k requests per second for our above Shopping site, and due to an upcoming holiday season, there will be an estimated increases in the visitors count and we will have to serve, 400k parallel requests per second, we could do one of two things. (1) We could buy more expensive hardware, say a 16GB RAM machine. We could move our site deployment to this bigger machine until the holiday season and get back to the old system later. (2) We could launch another three of 4 GB RAM machines and handle the increased load. The former is called Vertical Scaling and the latter is called Horizontal Scaling.<br />
<br />
Vertical scaling is appealing for small workloads but is costlier as we have to provision huge machines. Even if you could rent high-end VMs in the cloud, the pricing is not too friendly. Horizontal scaling is cheaper on your wallet as well as provides more throughput and allows for more dynamism.<br />
<h4 style="text-align: left;">
Auto-Scaling</h4>
In our Shopping application, we saw that the Sales and the Inventory Services are stateless. So we could horizontally scale them individually. For example, we could launch 3 new instances of the SalesService to handle a holiday-traffic while maintaining the single machine for the Inventory service. This kind of flexibility would not have been possible with our earlier monolithic design. However, note that the Stock Service that we had was stateful and so it could not be horizontally scaled. This is the drawback of having stateful components in your architecture.<br />
<br />
Once we know that the systems could be horizontally scaled, the next logical progression is to make the scaling automatic. There are systems like <a href="http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/Welcome.html" target="_blank">Amazon Beanstalk</a> and <a href="https://cloud.google.com/appengine/" target="_blank">Google AppEngine</a> (to a certain extent (with vendor lockin)) that allow your application code to automatically horizontally scale by launching new instances whenever the demand is higher. The new instances will be automatically shutdown when the burst of traffic is reduced. This reduces huge IT administration overheads. We could have such nice features, only because our application architecture was composed of stateless services.<br />
<h4 style="text-align: left;">
Serverless Systems</h4>
The next step in the evolution of auto-scaling, is to have code that automatically decides the number of servers on which it should run, instead of having to provision anything. To quote, Dr. Werner Vogels, CTO of Amazon, "<a href="https://twitter.com/awsreinvent/status/652159288949866496" target="_blank">No server is easier to manage than no server</a>". We are clearly moving in this direction with serverless webapps. Amazon <a href="http://docs.aws.amazon.com/lambda/latest/dg/welcome.html" target="_blank">Lambda</a> brings to life this functional programming dream. Google is not far behind and have recently launched <a href="https://cloud.google.com/functions/docs" target="_blank">Cloud Functions</a> (but not as rich as AWS Lambda yet imho). We have <a href="https://github.com/serverless/serverless" target="_blank">frameworks</a> to build entire suite of applications without servers, using these services.<br />
<h4 style="text-align: left;">
Polyglot Development</h4>
As we are deploying each service independently, we could use different programming languages, frameworks and technologies for each of the services. For example, any CPU intensive service could be written in a performant language like Go while a bunch of front end code could be written in parallel in React or nodejs.<br />
<br />
<h4 style="text-align: left;">
Mobile First</h4>
Since we have developed proper HTTP APIs for our application, in addition to the webclient, any mobile client too could use our webservices. In this day, most of the companies start with a mobile-first or mobile-only strategy and do not require a webclient. Some pro-monolithic engineers tend to argue that the first iteration of development should be in a monolithic model and we could re-engineer for a SOA at a later stage of development, as development speed is faster in monolithic design. Personally, I disagree to this. If we start with a SOA in mind from scratch, with our modern day development stack, we could easily plumb existing things instead of reinventing wheel and could do projects faster. There are frameworks and techniques to auto-generate a lot of code, once we have finalized the APIs. I have had experience building web applications both as a monolith and in SOA from scratch, I have felt happier with SOA code every time, YMMV.<br />
<h3 style="text-align: left;">
Auxiliary Parts</h3>
If we are building a SOA based system, we need to have a lot more auxiliary support systems. If we do not have these auxiliary parts in place, it will be very difficult to measure, debug or optimize. Different companies implement different parts below, based on their business needs and deadlines.<br />
<h4 style="text-align: left;">
Performance Metrics</h4>
The most important auxiliary aspect of SOA is to have precise performance metrics for each of the services. If we have SOA without performance/metrics measurement, It will be as ineffective as trying to do bodybuilding or weightloss without observing what we eat. We will not be able to rate limit requests, prevent DoS attacks, understand the health of the service without measurement. The performance measurement can be done in two ways, (1) Measure the performance and show metrics by realtime event monitoring (2) Log various events, errors, response times, etc., aggregate these logs and batch process them later, to understand the health of various components. We will need a combination of both the approaches for any large scale systems.<br />
<br />
Luckily there are plenty of tools, services and libraries available for this. <a href="https://aws.amazon.com/api-gateway/" target="_blank">AWS API Gateway</a> is perhaps the easiest way to register your APIs and monitor the endpoints. However we may need more finegrained measurements too (such as how long the calls to the database takes, which user is causing more load, what times are the loads high, etc.). There are various tools that we could use such as <a href="https://github.com/etsy/statsd" target="_blank">statsd</a>, <a href="http://ganglia.info/" target="_blank">ganglia</a>, <a href="https://www.nagios.org/" target="_blank">nagios</a>, etc. and various companies that offer hosted solutions too, such as <a href="https://sematext.com/" target="_blank">sematext</a>, <a href="https://signalfx.com/" target="_blank">signalfx</a>, <a href="http://newrelic.com/" target="_blank">newrelic</a>, etc.<br />
<h4 style="text-align: left;">
Distributed Tracing</h4>
Tracing is a concept that is supplementary to metrics and performance measurement. When a new request comes to a service, it may in turn make use of 3-4 other services to serve the original request. Those 3-4 other services may in turn call 3-4 other services. Tracing helps us find out, on a per-request basis, the map of which services are used to serve it, how long it took at each point, where the request is stuck if could not be serviced, etc.<br />
<br />
We could achieve tracing by giving a unique id / context object to each incoming new request in the outermost API which receives the request, pass it along as we make further API requests until the final response is finished. This context could be passed along as a parameter in the webservice calls. The monitoring of the tracing events could again be realtime or deducted from log-aggregation.<br />
<br />
<a href="http://research.google.com/pubs/pub36356.html" target="_blank">Dapper</a> is a paper released by Google summarizing how tracing is done in Google. Twitter have <a href="https://blog.twitter.com/2012/distributed-systems-tracing-with-zipkin" target="_blank">released</a> Zipkin, a FOSS implementation of the above Dapper paper, that is in production.<br />
<h4 style="text-align: left;">
Pagination</h4>
Assume that we are exposing an API in our StockService to list all the items that we have, along with its RetailPrice. If we have say a billion products, the response to the API will be huge. Not just the response, the system resources needed to build that response, on the server side will be tremendous. If we are fetching the billion items from the database, the caches will be thrashed, the network will be clogged, etc. To avoid all these issues, any API that could potentially list a lot of items should consider paginating its response by a pagenumber, i.e., an API call should take a page number as a parameter and should return only M number of items in a page. The value of M could be decided based on the size of each item on the response. We can optionally get the number of results that the user wants, also as a HTTP Parameter.<br />
<br />
For example:<br />
<br />
<ul style="text-align: left;">
<li>http://127.0.0.1:8080/posts/label/tech - Returns the first 10 blog posts with label "tech"</li>
<li>http://127.0.0.1:8080/posts/label/tech/1 - Same as above</li>
<li>http://127.0.0.1:8080/posts/label/tech/2 - Returns blog posts 11 -> 20 with label "tech"</li>
<li>http://127.0.0.1:8080/posts/label/tech/?limit=5 - Returns the first 5 blog posts with label "tech"</li>
<li>http://127.0.0.1:8080/posts/label/tech/?start=15&limit=5 - Returns the blog posts 15 to 20 with label "tech"</li>
</ul>
<br />
<h4 style="text-align: left;">
API Versioning</h4>
If software never changes, we software engineers will be out of jobs. It is good that software evolves. However, we need some contracts/APIs, so that the changes are smooth and does not bring down the entire ecosystem when a change happens. Once we have an exposed an API outside our developer team, it is wiser to never change its request/response parameters.<br />
<br />
In our StockService example (that we discussed a few paragraphs ago), we could have the following API:<br />
<br />
http://stockservice/items/ - Returns all the items.<br />
<br />
Later someone figured out that, it is not wise to return all the items always and decides to change the behavior to return only the first 10 items. This change will break all the existing clients, who will all assume that there are only 10 items in total while in reality we may have a billion more items waiting to be paginated.<br />
<br />
The easiest way to regulate the API changes is by adding version to APIs. For example, if the original API to return all the items had a version param, we could just increment it like:<br />
<br />
http://stockservice/V1/items/ - Returns all the items<br />
http://stockservice/V2/items - Returns the top 10 items<br />
<br />
The version need not be part of the URL always. We could take the Version as an extra HTTP header also, instead of creating a new URL endpoint. It is a matter of taste and each approach has its own pros and cons.<br />
<h4>
CircuitBreaking</h4>
Once we have multiple components in a system, there is a high chance that some part of the system may be down for updates. When such a thing happens, a service could choose to wait for some time before making any attempts to retry if it knows that the service will be failing. Martin Fowler has <a href="http://martinfowler.com/bliki/CircuitBreaker.html" target="_blank">written</a> in detail about this, which is a good read.<br />
<h4 style="text-align: left;">
Service Discovery</h4>
<div>
In a large scale system architected with a microservices based design, you will have a plenty of services. Now each of these service, may want to know the location (URL, ip-address+port, etc.) for the services on which it depends. So we need some kind of a centralized service registry where all these information is stored and maintained.</div>
<br />
The easiest and probably most used way to identify these services is through DNS. However, there are plenty of other tools available for this purpose too. <a href="https://zookeeper.apache.org/" target="_blank">ZooKeeper</a> from Apache, <a href="https://github.com/coreos/etcd" target="_blank">etcd</a> from CoreOS are all strongly consistent, distributed datastores which could be used for service discovery. <a href="https://www.consul.io/" target="_blank">Consul</a> from HashiCorp, <a href="https://github.com/Netflix/eureka" target="_blank">Eureka</a> from Netflix are dedicated service discovery software. All of the above are FOSS projects as well. If your application has only less than a dozen services, probably it makes sense to just read from a shared file, across these services, instead of deploying a complex suite of software too. But keep in mind that it won't scale as you grow and so it is better to start with good practices as a habit from the beginning.<br />
<h4 style="text-align: left;">
SDKs</h4>
A new TCP connection takes time to establish because of the initial handshake delay. It will be foolish to not reuse these connections. There is an inherent need for retrying things in HTTP if things fail, before giving up. Some programmers do not like writing HTTP client code always either. It is often recommended to release SDKs for the APIs that we release, to facilitate programmers to consume our APIs easily. For example, a python programmer can merely import our SDK's classes to add an item to our StockService, instead of having to write http retry code.<br />
<br />
In the past we have had technologies like <a href="https://en.wikipedia.org/wiki/Distributed_Component_Object_Model" target="_blank">DCOM</a>, <a href="https://en.wikipedia.org/wiki/Common_Object_Request_Broker_Architecture" target="_blank">CORBA</a>, <a href="https://en.wikipedia.org/wiki/Java_remote_method_invocation" target="_blank">RMI</a> etc. that aimed at doing distributed computing within walled gardens of technology. They lost out in market share due to the simplicity of REST services where HTTP verbs (GET, PUT, POST, DELETE) could perform remote operations, without the need for complex and mostly platform-specific stubs/skeletons etc.<br />
<div>
<br /></div>
<div>
There is a common middle ground where the best of both worlds could be used. The most notable framework for this is <a href="http://www.grpc.io/" target="_blank">gRPC</a>. It is an open source project, started by Google, adopted by many companies (most recently <a href="https://coreos.com/blog/gRPC-protobufs-swagger.html" target="_blank">coreos</a>) that helps in providing a web API where the client SDK generation is also made easy. It support http2 as well. If I were starting a new project today, I would give this a serious thought.</div>
<h3 style="text-align: left;">
Further Information</h3>
<br />
<ul style="text-align: left;">
<li>A very good read on the need for SOA is <a href="https://plus.google.com/+RipRowan/posts/eVeouesvaVX" target="_blank">Steve Yegge's Platforms rant</a>.</li>
<li>Read the techblog of companies who are moving to SOA (not just those who have moved already)</li>
<li>Talk to engineers from Netflix, Amazon Web Services, if you know someone. Sadly both of those companies do not exist in India (as of 2016), even though the both the services are available.</li>
<li>Follow Netflix techblog http://techblog.netflix.com/</li>
<li>Watch AWS reInvent videos and if you have a chance attend that event (instead of events like Google I/O which are more business driven)</li>
</ul>
<div>
<b><br /></b></div>
<div>
<b>Other Notes:</b><br /><ul style="text-align: left;">
<li>If you like this post, share it with your friends. </li>
<li>Please send any comments / feedback regarding the language or content used. I am planning to use this for teaching material in a college, for a 1 hour talk, shortly. Should some other topics be covered ?</li>
<li>All opinions expressed are purely personal.</li>
</ul>
</div>
</div>
</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com13tag:blogger.com,1999:blog-28654204.post-6122600398236906942016-01-01T01:36:00.001+05:302016-01-01T01:36:16.583+05:302015 Learning Retrospective<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<ul style="text-align: left;">
<li>The year began well. Started working on <a href="https://github.com/psankar/keeri" target="_blank">keeri</a> with an aim to implement a distributed database, thereby learning the distributed systems concepts and leveraging my storage / filesystem experience. </li>
<li>Took the coursera's <a href="https://www.coursera.org/course/cloudcomputing" target="_blank">cloud computing concepts</a> course to understand the fundamentals that will help in implementing keeri</li>
<li>As part of the database implementation, needed to implement a SQL parser which will convert given SQL statements into a decision tree. Took a coursera course on <a href="https://www.coursera.org/course/compilers" target="_blank">compilers</a>. Implemented a decent recursive-descent (note the wordplay) parser that will <a href="https://travis-ci.org/psankar/keeri" target="_blank">process</a> SQL queries with parentheses, Logical operators and Relational operators.</li>
<li>Having already been tired with the non-core aspects of the "distributed" database, abandoned the project temporarily.</li>
<li>Need to learn more about <a href="https://en.wikipedia.org/wiki/NewSQL" target="_blank">NewSQL</a> technologies. Especially in the areas around how it helps for better tooling (for IDEs and the like) and also for parallelism.</li>
<li>Studied a bit of database literature around <a href="https://en.wikipedia.org/wiki/Algorithms_for_Recovery_and_Isolation_Exploiting_Semantics" target="_blank">ARIES</a>, Voltdb etc.</li>
<li>Attempted to read<a href="http://research.microsoft.com/en-us/um/people/lamport/pubs/lamport-paxos.pdf" target="_blank"> part-time parliament</a> but lost interest midway because of reading <a href="https://raft.github.io/" target="_blank">raft</a>, which is for similar purpose but a lot simpler to read, follow. Did a paper reading session together with Sureshkumar Thangavel for this.</li>
<li>Played around with Continous Integration systems (<a href="https://travis-ci.org/" target="_blank">travis</a>, <a href="https://jenkins-ci.org/" target="_blank">jenkins</a>, etc.) out of interest, which later helped in projects in two different dayjobs.</li>
<li>Wasted a lot of time, pretending to be preparing for interviews but did not do anything more than chatting with job change aspirants. But no complaints as time enjoyed is not time wasted.</li>
<li>Learnt a little bit in more detail about queueing systems (Amazon <a href="https://aws.amazon.com/sqs/" target="_blank">SQS</a>, rabbitmq to be specific)</li>
<li>Wrote some test / tutorial programs for the <a href="http://aws.amazon.com/sdk-for-go/" target="_blank">Amazon Go SDK</a></li>
<li>Did a few prototypes using Go for the API backend, Angular and React as the web frontends for few project ideas. Bothered about the fatigue induced by the constant reinvention in the frontend JS technologies. The future looks potentially even more heavily fragmented with no sanity in the horizon.</li>
<li>Learnt to create <a href="https://www.docker.com/" target="_blank">docker</a> images. Did some non-trivial dockerization for a legacy product with then employer. Wanted to checkout <a href="http://kubernetes.io/" target="_blank">kubernetes</a>, <a href="https://github.com/coreos/rkt" target="_blank">rocket</a> and potentially provide patches. But lost interest.</li>
<li>Wrote a bunch of long blog posts which triggered some nice private discussions. <a href="http://psankar.blogspot.com/2015/11/awstruck.html" target="_blank">1</a> <a href="http://psankar.blogspot.com/2015/11/foss-system-software-in-2015.html" target="_blank">2</a> <a href="http://psankar.blogspot.com/2015/03/online-programming-competitions-are.html" target="_blank">3</a></li>
<li>Worked a little bit on <a href="https://github.com/psankar/ithavi" target="_blank">ithavi</a> - the book on operating systems in tamil, but shamefully minuscule progress. Should do more next year at least.</li>
<li>The year began well but lost steam midway, probably due to the decision to change the dayjob after 10+ years with SUSE/Novell. It led to distraction, lack of interest and some sentimental times leading to lesser productivity towards hobby projects. Hopefully the next year will be better, but with a job in a startup that works fast, I am not sure how much bandwidth I may have.</li>
<li>Still not convinced if I should work on any of these <a href="http://psankar.blogspot.in/2015/11/foss-system-software-in-2015.html" target="_blank">system software anymore</a> or if I should focus on some other paradigm that is at its infancy. Ken Thompson and Dennis Ritchie worked on Unix when OSes were not mature. Leslie Lamport worked on distributed systems papers which became valuable after more than two decades. Go is now using a <a href="https://blog.golang.org/go15gc" target="_blank">paper</a> on garbage collection by Dijkstra and Lamport written in the 70s. So, I am thinking if I should focus on some problems / technologies whose time has not come yet, to feel that excitement of walking on unchartered territories. There are a few options like quantum cryptography etc. which have good theorists who need programmers. If I could collaborate with such intelligent people and synergistically add some value, it will be satisfying. I briefly discussed with some researchers in India (IIT Madras, TIFR etc.) about doing a PhD or helping as an assistant. But not any progress and nothing sounds too promising if work has to be done from India, thanks to our country's brain drain and the Government of India's focus on doing research on <a href="http://www.bangaloremirror.com/bangalore/cover-story/DRDO-will-work-with-Tirupati-trust-to-prove-loony-theories/articleshow/50327915.cms" target="_blank">loony</a> vedic technologies instead of on useful things. That is enough rant for the year :)</li>
</ul>
<div>
This is a <a href="http://psankar.blogspot.in/search/label/retrospective" target="_blank">series</a> of blog posts, that I write every year to document and reflect on the learnings, that I have had outside the dayjob. Previous editions: <a href="http://psankar.blogspot.in/2015/01/2014-learning-retrospective.html" target="_blank">2014</a>, <a href="http://psankar.blogspot.com/2014/01/2013-learning-retrospective.html" target="_blank">2013</a></div>
</div>
Sankar Phttp://www.blogger.com/profile/08027774078620587372noreply@blogger.com0tag:blogger.com,1999:blog-28654204.post-14674337339488361832015-11-29T15:26:00.001+05:302015-11-29T15:26:33.685+05:30FOSS System Software in 2015<div dir="ltr" style="text-align: left;" trbidi="on">
<h4 style="text-align: left;">
Prelude</h4>
About an year ago, I was playing around with <a href="http://cassandra.apache.org/" target="_blank">Cassandra</a> for a quick prototype in then dayjob. It opened up the world of distributed systems to me and I was piqued. Audaciously, I decided to implement a simple distributed database, Keeri, to have a grasp of the fundamentals of the implementation of distributed databases. In the past, I have implemented a simple filesystem which has helped me immensely when I was working as a filesystem engineer. Also, I have always been fascinated by the theory behind the database internals right from college days, but did not get my hands dirty.<br />
<br />
After a few weeks of work, I was able to implement a recursive-descent SQL parser, which analysed the incoming SELECT queries, made a tree with the subqueries properly branched as sub-trees. I made a simple columnar store that appends data (via an API as opposed to SQL) but without any atomicity guarantees. In short, it was a rudimentary, in-memory system that <a href="https://travis-ci.org/psankar/keeri" target="_blank">functions</a> decently. However, I was nowhere near the initial goal of implementing a distributed database.<br />
<br />
I realised that there were plenty of design choices in a distributed database implementation, right from architecture, replication, membership, consensus, CAP, etc. I even took a <a href="https://www.coursera.org/course/cloudcomputing" target="_blank">coursera course</a> that helped me understand the basics in detail. As of today, I have enough confidence in my skills and knowledge to implement a distributed database which could serve as a good teaching material, if not as a production software. However, I have not made a single line of code in the past seven months to the project. Abandoning (at least temporarily) the project, hurts.<br />
<br />
Yesterday, my daughter decided to wake me up from my sleep after midnight, I spent the remaining night wide awake, while she slept, thinking why I have not made progress in keeri. I realised that I have been overwhelmed by the amount of things to do, that are not core to the system. For example, after I decided that the database has to be NEWSQL based, it is imperative that I needed a SQL parser. But there are a dozen types of parsing (LL, LR, Recursive Descent, ANTLR etc.) techniques. Understanding the pros and cons of the each type and finding the most suitable candidate is a non-trivial task. SQL Parsing is just one component of the system. There are other components such as the choice of datastructures (based on read/write ratio, type of load etc.) One approach is to proceed with the simplest choices for each component with well-defined borders. The individual components can be later replaced. By the time, I completed the SQL query to a decision tree code, I felt exhausted, even before I began the core database and distributed systems functionality.<br />
<h4 style="text-align: left;">
Observations</h4>
From my past open source experience, I have known the synergic boost that developers experience in FOSS communities. It is always good to work in a like minded team of developers rather than individually when working on big problems. I started thinking what other FOSS projects exist for distributed databases (or any other large scale system software) that were created in the last 5-6 years. A few things that came to my mind were:<br />
<br />
<b>Hadoop</b>: An umbrella of projects, initially started by Yahoo, including core projects such as HDFS, HBase and a laundry list of supporting projects. Most of the code is now under the Apache project with a plenty of companies sponsoring the development and using the projects.<br />
<br />
<b>Docker</b>: The coolest kid in the town. Initially started by Solomon Hykes funded by dotCloud as a side project. Arguably the most active project to date used heavily by almost all tech companies worth their salt. This spawned off a series of other projects too.<br />
<br />
<b>CoreOS</b>: Linux re-thought for being a Cloud focussed distro. Backed by a company with the same name. Founded by ex SUSE, Rackspace people, collaborating with Greg KH himself.<br />
<br />
<b>Redis</b>: Started by antirez, funded by vmware, pivotal and most recently redis labs. Probably the most used k-v database, probably challenged only by the older memcached.<br />
<br />
<b>Cassandra</b>: Initially started by Facebook and later became an Apache project. Heavily used by companies like Netflix, Twitter, Applet etc. even after facebook has moved away.<br />
<br />
<b>Kafka</b>: Initially started by Linkedin and later became an Apache project. Used by linkedin and almost every company today. Most of the original team that created the project have now jumped off to a new company named Confluent working on the project full time.<br />
<br />
<b>CockroachDB</b>: A project that claims to be the open source equivalent for Google's Spanner. Started by ex-googlers. Development funded and managed by Cockroachlabs.<br />
<br />
As I kept thinking about these (and a few other) projects in a state of semi-sleep, I had a eureka moment when I realised that all these projects, even though are open source, began funded by a company / investor money. This is in complete contrast to the FOSS projects of the previous generation like GNU, Linux, GNOME etc. It is a welcome change that developers are now not scared of becoming C*Os and spend time in management or bootstrapping a company. Perhaps, only today, we have companies like <a href="https://www.zenefits.com/" target="_blank"><b>Zenefits</b></a> (disclaimer: employer ;) ) making it easy to start a company and so more developers find it easier to start companies. VCs being in a bullish mindset also helps.<br /><br />However, I have one concern. Unless the project usage explodes and gains contributors from multiple companies, there is a high chance that the projects may compromise on quality to accommodate a business need. For example, if Torvalds was working for Google, <a href="https://lwn.net/Articles/318611/" target="_blank">wakelocks</a> *might* have merged into kernel much earlier to suit a Google release cycle. Torvalds being a neutral outsider, without having any commercial interests in any company (directly) has helped Linux immensely. If a FOSS project is started with a backing company in place, from day one, how high will the company's benefits influence the design / features / review processes of the FOSS project ?<br /><br />As I think more, I realised that, most of these new system software are developed to address the pains of a "as a service" providers. So unless there is a business case, it is perhaps difficult to create a new modern system software, as the era of<a href="https://cs.brown.edu/~ugur/fits_all.pdf" target="_blank"> <b>one size fits all is over</b></a>. This also makes the previous concern about, the chief maintainer being company neutral, irrelevant. Only when a software is made, backed by a company, with real use cases and customers, instead of theoretical / intellectual curiosities we will get live data. Personal pet projects may have to be satisfied with machine generated data, which may not be the best testsuite for data intensive software like databases.<br />
<h4 style="text-align: left;">
Questions</h4>
To sum up, I wonder if developers (students) any more interested / will be able to develop FOSS projects in their own hobby time, that could grow as big as Linux, without having a corporate backup, for the first few years at least, What do you think ?<br />
<br />
Also, if you think it is possible, any recommendations for developers with family and personal needs to spend time, off the regular day job, to persist with pet FOSS projects without exhaustion ? Are there any statistics available on contributor details for popular FOSS projects (similar to Kernel <a href="http://kernelnewbies.org/DevelopmentStatistics" target="_blank">stats</a> prepared by Greg KH and LWN) ?<br />
<br />
Any other aspects that I have missed ?<br /><br />P.S: I was sharing the gist of this post with a friend who shrugged off saying, "Get a job in the company which works on a project which appeals to you". However, it is not that simple, considering most of these young companies do not even have an office anywhere outside the developed world. Also after a certain point in life, switching jobs is not trivial and depends on various other factors.</div>
Sankar Phttp://www.blogger.com/profile/08027774078620587372noreply@blogger.com2tag:blogger.com,1999:blog-28654204.post-75118223099542155972015-11-22T14:44:00.002+05:302015-11-22T14:47:58.648+05:30AWStruck<div dir="ltr" style="text-align: left;" trbidi="on">
<h4 style="text-align: left;">
</h4>
<h4 style="text-align: left;">
tldr: </h4>
<div>
<span style="font-weight: normal;">A long post about my experience with implementing a quiz software in my college, a decade ago and wondering how easy things have become now due to AWS.</span></div>
<h4 style="text-align: left;">
Prelude</h4>
In 2002 (iirc) (thirteen years ago, as of composing this post) when I was in college, we had an inter-collegiate technical symposium, where Online Quiz was one of the events. A Microsoft Visual Basic 6.0 (which I personally consider to be one of the best software ever developed) application was developed in-house and installed on about 50 computers, where various contestants from different colleges could come and take the test. However, as Murphy predicted, due to various virus issues, the software failed spectacularly. Some answers/responses got corrupt, accumulation of responses from different machines proved faulty, the scoring went awry in some corner cases, etc. Overall, the application turned out to be total chaos. However, since India is populous, we were able to throw more people at the problem and finish the event, with a lot of manual effort, inspite of a few unhappy participants.<br />
<br />
In the planning phase for the subsequent edition of the symposium two years later, a software development committee for formed. It would do all the software for the entire event, (like creating a website, developing flash/swish videos, software for the individual events, etc.). The quiz event had two rounds, a preliminary round where all the appearing colleges contested and a final round where six (or probably more) top colleges from the previous round were selected. An eloquent person was made incharge of the quiz event. I proposed to the person that we do the software for the preliminary rounds ourselves, instead of depending on the committee. The committee was already swamped with work and they were happy to get rid of a piece that has more chances of failure. Some adventurous people (like <a href="http://design-antony.blogspot.in/" target="_blank">Antony</a>) expressed their interest in joining the project. Thus it all began.<br />
<h4 style="text-align: left;">
The Adventure</h4>
Much to the amusement of my roommate Bala, I started with planning the architecture and design on paper (complete with UML diagrams, etc.), instead of starting with coding as is the norm for us those days. Much later I came across an interesting quote by Alan Kay, "<b><i>At scale, architecture dominates material</i></b>". Having learnt from the mistakes of the previous years, I made some decisions.<br />
<br />
* The software should follow the web (client-server) model, that is getting popular. At least this is an excuse to learn some new (then) technologies, like JSP, Javascript, Tomcat etc.<br />
* The server machine becomes a single point of failure for the entire system. It could prove to be a performance bottleneck to, as our machines were all having a humongous 32 MB of RAM. There was one 64 MB ram in our lab which I planned to use as the server. In our hostel, some had a machine with luxurious 128 MB of RAM, which I was planning to borrow if the need comes.<br />
* The single point of failure, the server should not be susceptible to virus attacks. So we should experiment installing Solaris or this thing called Linux (There was no Ubuntu then).<br />
* Internet was a luxury and for the entire college we had access to it in about three computers, only in the evenings. So anything that requires too much internet access for development is automatically rejected.<br />
* The software should scale at any cost, for at least 200 parallel connections<br />
* We should regularly backup the sources in different machines, in case the development boxes gets a virus attack. We had no idea of version control systems then.<br />
* We will be using Mysql/oracle or some real database instead of writing to files. MS Access was ruled out automatically as Visual Basic was eliminated already. In hindsight, sqlite would have been an excellent choice.<br />
* The quiz webpage when saved on the client browser should save the file along with the answers chosen/typed.<br />
* Each quiz session will last for about 30 minutes. There will be username/passwords generated for each unique participant.<br />
<br />
We developed the JSP webapp running in Tomcat in a few weeks. We used the generous help of my classmates to throughly test the correctness of our scoring system. As with any manual system, it was prone to errors. A tester made a mistake in scoring and we broke our head trying to find a non-existent bug in our code for a few hours. This testing also helped us get the load numbers for the current system, with about 30 concurrent users. We had some performance monitoring hooks written in our code for this.<br />
<br />
We survived multiple virus attacks during the development, because of the distributed source backup techniques that we have employed. At one stage, we even burnt our sources in a CD when the administrators decided to Norton Ghost all the hard disks in our lab, with a fresh Windows XP image, to minimise the virus effects.<br />
<br />
I learnt the magical world of performance monitoring, database indexes, high availability, connection pools etc. during this project. I learnt much more in this single experiment than the almost half a dozen papers we had on software engineering, process management, quality assurance etc. taught by lecturers with no real world knowledge and questionable scoring practices. Some of the fascination that I acquired with database engines has <a href="https://github.com/psankar/keeri" target="_blank">still</a> not subsided.<br />
<br />
Having finished the coding one week prior to the event, we focussed more on scaling and testing. I prepared a backup server, another high-RAM machine in case our main server went kaput. Much to the jovial criticisms of my friend Sangeeth, we tested our system, the night before the event, for 2000 parallel users and it worked well without breaking a sweat. This is such a silly number in today's figures, but we were easily satisfied then with low numbers in both server performance (and salary). Almost all the front end code was handwritten Javascript with no frameworks / libraries (as mostly none existed or we were not aware). I was satisfied with what we have done, irrespective of however the results may turn out to be the next day.<br />
<br />
Having lost a good sleep due to the stress testing the previous night, I woke up late and missed the delicious Pongal in our hostel for breakfast. Ruing about the missed breakfast caused a weird stare from the rest of the team. I rushed for the preliminary quiz event ahead of time and two among us did a final test on the last day. We planned on using some Rational test suites for automated testing but could never get to that, thanks to all the virus related frequent re-installs of the base operating system.<br />
<br />
The participants came in numbers, attended the event. Surprisingly for us, a lot of people did not use the half-an-hour duration and finished much earlier, even with negative questions. The event chief too had a moment of doubt, if we have prepared the questions easy. But looking at the instant results in the server and the high percentage of low marks taught the lesson that many people have come to the event to have fun and not to seriously compete or win.<br />
<br />
Before we could ruminate on that philosophical thought, a participant had a problem. Her network cable went broke and she could not submit her quiz. I felt bad that I should have implemented auto-save for responses as soon as people make a choice. I intentionally avoided that to reduce load on the server. I was about to ask if the person could take the test from a different machine. But the inimitable event-head the presence of mind, to ask the participant to save the quiz on the same computer and that we will evaluate that offline. Antony did the scoring and that particular person turned out to be a topper. This particular event taught me a lot about presence of mind and how we should always plan for failure in computer systems, how ever thorough we test. The scalability as expected was never found to be a problem.<br />
<br />
After the event is finished, our lab admin, Marshal joked that we should start a company with this quiz software, as we have done it as a generic survey software where questions can be added. We laughed at the suggestion and moved away. The event was successful. Some of the software developed by the committee for some other events were affected by the recurring virus problem. But I went and slept like a log on a temporary bed made of three office chairs, next to my classmate Saktheesh who was working on a closing video for the event.<br />
<h4 style="text-align: left;">
The Present</h4>
The long story above is not to just narrate my/our work, but also to highlight how much approachable the programming / technology landscape has become. A quiz / questionnaire software can be implemented today (2015) in probably a few hours, thanks to the large number of frameworks (such as Rails, Django, etc.). In fact, most of the tutorials have better code that you can merely copy/paste than what we have implemented a decade ago. The most striking thing today, however, is not the story of coding, but the story of deployment.<br />
<br />
Anyone with an internet connection, a basic course on programming and decent googling skills can program any service easily today. What is even more fascinating is that such a software can be very easily deployed on the internet, served to the whole of the world, complete with a domain name, auto-scaling, DoS prevention etc. in just a few clicks. This is all made possible through Amazon Web Services. There are other players like Google, Heroku etc. but it is AWS that is way ahead of any other players and provide more services. The reach of AWS is what made me choose the title of this blog post.<br />
<br />
AWS has done more to spur the startup ecosystem. The social impact of AWS is much higher than what Google did for online Ads, Microsoft did for PCs. Disruptive companies like airbnb, slack, netflix (which was just an online video rental service 7 years ago) can exist today, only because their devops, installation and maintenance of machines could be outsourced to AWS. They could not have grown to such 800 hundred pound gorillas if the AWS infrastructure was not available, in such a short time. Sure, there are some companies like Uber, Whatsapp that do not use AWS, but they would not have got funded easily if not for the startup scenario, which was formed with AWS as the backbone.<br />
<h4 style="text-align: left;">
The Future</h4>
I have been visiting various buildings in Bangalore trying to find an office space for Zenefits India, as I am the first engineer here. All the places have a Server Room, which is not used by any of the startups. Almost all the startups use a Mac for developer machine and have their deployment servers in AWS (or some public cloud). The office spaces of Bangalore have not caught on with the trend. We are hiring btw, so if you consider yourself an extremely good engineer and one of the best at what you do in your job, do apply.<br />
<br />
Most of the new services offered by Amazon, such as Amazon Lambda, DynamoDB etc. and also things like Containerisation, have made development of scalable applications, easier. Developers need not worry about failover systems, HA system, clusters etc. any more. I wonder what kind of an impact this will have on the job market. I wonder how long it might take job positions like mysql admins, sysadmins, devops engineers, DBAs to become as old/obsolete as, say mainframe programmers are considered today (2015). Perhaps it may not be soon but it is very much possible soon. Ubiquitous applications like SAP, Office etc. are also now cloud first and it will only become more cloud focussed in the future.<br />
<br />
I wonder how much of system software research will be affected in the long term. Many of the modern day young bright minds (students from prestigious colleges and universities) are working in webapps, joining startups and doing their own companies, instead of working on projects with high entry barriers like the Linux Kernel, LLVM etc. (at least in India). Perhaps, we would have started the quiz project that we did as a company (somewhat like surveymonkey) if we had enough exposure then. I may not have done that but some students smart with a business acumen, would have.<br />
<br />
There are very interesting research problems in distributed systems that include both Databases and OSes. Most of the present day systems are just distributed systems constructed over Linux / POSIX systems. However, there is a potential for a DOSIX (along the lines of POSIX) API purely designed for large-scale, cross-geo distributed systems. It will be interesting to see what kind of research happens in this direction. In the recent past, We have a new distributed consensus algorithm Raft after decades of using Paxos. More such re-inventions are bound to happen soon, may be on novel things like non-blocking, distributed garbage collection etc.</div>
Sankar Phttp://www.blogger.com/profile/08027774078620587372noreply@blogger.com1tag:blogger.com,1999:blog-28654204.post-72459203346075146772015-03-02T13:37:00.000+05:302015-03-02T13:45:44.703+05:30Online Programming Competitions are Overrated<div dir="ltr" style="text-align: left;" trbidi="on">
The title is not merely a clickbait, but my current opinion, after attending a programming competition for the first time. This post expresses my opinions on the hiring processes of [some of] the new age companies through programming competitions and algorithms-focused interviews.<br />
<br />
I believe that the assessment for a senior/architect level programmer, should be done by finding how <b>co-operative</b> [s]he is with others to create interesting products and their history than by assessing how <b>competitive</b> [s]he is in a contest.<br />
<h3 style="text-align: left;">
Algorithms</h3>
On my lone programming competition experience (on <a href="https://www.hackerrank.com/contests/zenhacks/challenges" target="_blank">hackerrank</a>), the focus of the challenges were on Algorithms (discrete math, combinatorics etc.).<br />
<br />
Usage of standard, simple algorithms, instead of fancy, non-standard algorithms is a better idea in real life, where the products have to last for a long time, oblivious to changing programmers. Fancy algorithms are usually untested, harder to understand for a maintenance programmer.<br />
<br />
Often, it is efficient to use the APIs provided by the standard library or ubiquitously popular libraries (say jquery). Unless you are working on specific areas (say compilers, memory management etc.) an in-depth of knowledge of a wide-range of algorithms may not be very beneficial (imo) in day-to-day work, elaborated in the next section.<br />
<h3 style="text-align: left;">
Runtime Costs</h3>
There are various factors that decide the runtime performance, such as: Disk accesses, Caches, Scalable designs, Pluggable architectures, <a href="http://en.wikipedia.org/wiki/Reliability_engineering" target="_blank">Points of Failures</a>, etc.<br />
<br />
Algorithms optimize mostly one aspect, CPU cycles. There are other aspects (say choice of Data structures, databases, frameworks, memory maps, indexes, How much to cache etc.) which have a bigger impact on the overall performance. CPU cycles are comparatively cheap and we can afford to waste them, instead of doing bad I/O or a non-scalable design.<br />
<br />
Most of the times, if you choose proper datastructures and get your API design correct, we can plug the most efficient algorithm, without affecting the other parts of the system, <b>iff</b> your algorithm proves to be really a bottleneck. A good example is the <b>Evolution</b> of filesystems, schedulers in the Linux Kernel. Remember that <b>Intelligent Design</b> school of software development is a myth.<br />
<br />
In my decade of experience, I have seen more performance problems due to poor choice of datastructures or unnecessary I/O, than due to poor selection of algorithms. Remember, <b>Ken Thompson</b> said: <b>When in doubt, Use Brute Force.</b> It is not important to get the right algorithm on the first try. Getting the skeleton right is more important. The individual algorithms can be changed, <b>after profiling</b>.<br />
<br />
At the same time, this should not be misconstrued as an argument to use <a href="https://www.youtube.com/watch?v=k4RRi_ntQc8" target="_blank">bubblesort</a>.<br />
<h3 style="text-align: left;">
The 10,000 hour rule</h3>
<div>
Doing well in online programming competitions is mostly the <a href="http://en.wikipedia.org/wiki/Outliers_(book)" target="_blank">10,000 hour rule</a> in action. You spend time in enough competitions and solve enough problems, you will quickly know which algorithm or programming technique (say dynamic programming, greedy) to employ if you see a problem.</div>
<div>
<br /></div>
<div>
Being an expert at online programming competitions does not guarantee that [s]he could be trusted with building or maintaining a large scale system, that has to run long and the code live for years (say on the scale of filesystems, databases, etc.). In a competition, you solve a small problem at a microscopic level. In a large scale system, the effects of your code are systemic. Remember how the fdisk, sqlite, firefox fiasco ?!</div>
<div>
<br /></div>
<div>
In addition to programming skills, there are other skills needed such as build systems, dependency management (unless you are working on the kernel), SCM, Versioning, Library design aspects, automated testing, continuous integration etc. These skills cannot be assessed in online programming competitions. </div>
<div>
<br /></div>
<h3 style="text-align: left;">
Hardware</h3>
<div>
In my competition, I was asked to solve problems in a machine that is constrained to run in a single thread. I do not know if it is a limitation on hackerrank, or if all online competitions enforce this.</div>
<div>
<br /></div>
<div>
If it is the practice in all online programming competitions, then it is a very bad idea. Although I could understand the infrastructure constraints for these sites, with the availability of the multi-core machines these days, your program is guaranteed to run on multiple cores. You miss out on a slew of evaluation options if the candidate is forced to think of single threaded design.</div>
<div>
<br /></div>
<div>
With the arrival of cloud VMs and the Google appengine elasticity, it is acceptable to throw more CPUs or machines at a program on-demand, without incurring high cost. <i>It is okay to make use of a simpler, cleaner algorithm that is more readable and maintenance friendly (than a complex, performant algorithm)</i>, if it will scale better on increased number of CPUs or machines. The whole map-reduce model is built around a similar logic.</div>
<div>
<br /></div>
<div>
I don't claim that concurrency/parallelism/cloud is a panacea for all performance problems, but it is too big a thing to ignore while assessing a programmer. A somewhat detailed explanation is at the Redis creator's <a href="http://antirez.com/news/73" target="_blank">blog</a> (strongly recommended to subscribe).</div>
<div>
<br /></div>
<h3 style="text-align: left;">
AHA Algorithms</h3>
<div>
I first heard of the concept of AHA Algorithms in the excellent book <a href="http://www.amazon.com/Programming-Pearls-2nd-Jon-Bentley/dp/0201657880" target="_blank">Programming Pearls by Jon Bentley</a>. These are the algorithms which make very complicated, seemingly impossible problems look trivial, once you know the algorithm. It is impossible for a person to solve such problems within the span of the competition/interview if the candidate is not aware of the algorithm earlier and/or does not get that AHA moment. Levenshtein Distance, Bitmap algorithms etc. fall in this category. It may not be wise to evaluate a person based on such problems.<br />
<br />
<h3 style="text-align: left;">
Conclusion:</h3>
</div>
<div>
Candidizing (is that a word ?) long-term FOSS contributors for hiring may be an interesting alternative to hiring via online programming competitions or technical interviews. Both the interviews and contests have extrapolation errors when the person starts working on a job, especially on large scale systems.</div>
<div>
<br /></div>
<div>
I see that a lot of the new age companies are asking for github profile in their resumes, which is good. But I would prefer a more thorough analysis on long standing projects and not merely personal pet projects that may not be very large-scale or popular. Not every person works for a FOSS project in free time, is also a deterrent to holding such an approach.</div>
<div>
<br /></div>
<div>
Online programming competition websites could limit the number of participants in advance and give the participants an infrastructure that matches realtime development, instead of a input-output comparison engines, with a 4 second timeout.<br />
<br />
Having said all these, these online programming contests are a nice way to improve one's skills and to think faster. I will be participating in a few more to make myself fitter. There may be other programming challenges which are better and test all aspects of an engineer. I should write about my view after an year or so.</div>
<div>
<br /></div>
<div>
<h3 style="text-align: left;">
One more thing: Zenefits & Riptide I/O</h3>
</div>
<div>
In other news, My classmates' companies <a href="http://www.zenefits.com/jobs" target="_blank">Zenefits</a> and <a href="http://www.riptideio.com/" target="_blank">Riptide I/O</a> are hiring. Have a look if you are interested in a job change (or even otherwise). They are in an excellent (imo) stage where they are still a startup in engineering culture, but have an enormous funding to work on the next level of products. Should be an exciting opportunity for any curios engineer. Zenefits works on web technologies and delivers their SaaS. I would have joined Zenefits if they had an office in Bangalore. Riptide IO works on IoT and has some high profile customers.</div>
</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com11tag:blogger.com,1999:blog-28654204.post-18306850904630901012015-01-14T12:11:00.001+05:302015-01-14T12:11:35.104+05:30Naming Policy - Deactivated Quora Account<div dir="ltr" style="text-align: left;" trbidi="on">
I just deactivated my Quora account, as they have a policy of mandatory lastname. They stated that they will not accept Initials as well. My account was put on hold due to the lastname not satisfying their requirements.<br /><br />Tamils do not have a last name due to various political reasons. Avoiding the lastname is considered to be good for <a href="https://www.quora.com/Why-do-Tamils-not-have-surnames" target="_blank">ending caste discrimination</a> as well. (Ironically a quora link)<br />
<br />
Companies like Google had the sense to <a href="http://www.zdnet.com/article/google-reverses-real-names-policy-apologizes/" target="_blank">relax</a> their real name policy after their initial debacle with G+<br />
<br />
Sadly, Quora does not want to learn from their's or others' mistakes. I wonder if they will even ban names in non-English letters later. So if they cannot be inclusive, I feel that they deserve to lose business.</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com1tag:blogger.com,1999:blog-28654204.post-6703590353170126552015-01-03T22:30:00.001+05:302015-01-04T10:15:33.738+05:302014 Learning Retrospective<div dir="ltr" style="text-align: left;" trbidi="on">
The learning in the previous year (<a href="http://psankar.blogspot.in/2014/01/2013-learning-retrospective.html" target="_blank">2013</a>) was a bit shallow but on a wide variety of topics. The year 2014 turned out to be not bad. I went to some great depths in a small number of areas.<br />
<br />
<ul style="text-align: left;">
<li>Worked on <a href="https://github.com/psankar/korkai" target="_blank">Korkai</a> - A corpus builder for Tamil. It extracts unique Tamil words from blogger, wordpress and wikipedia dumps. Learnt a lot about XML processing and golang</li>
<li>Started working on <a href="https://github.com/psankar/vaiyakani" target="_blank">Vaiyakani</a>, an auto-completing, dictionary-based, self-learning, transliterating text-editor for Tamil, after getting unsatisfied with the lack of offline Tamil typing software in Linux. Learnt a great deal about <a href="https://en.wikipedia.org/wiki/Trie" target="_blank">Tries</a>, Prefix Datastructures, Sqlite database engine performance, Datastructures used in the implementation of maps in various libraries and programming languages etc.</li>
<li>The quest for implementing a perfect text editor led to a brief phase of disappointment where I complained on every layer wondering why the below layer is bad and briefly attempting to improve it. (The application is bad, The toolkit is bad, The compiler is bad, The operating system is bad, The hardware is the root of all evil etc.) Thankfully shepherded back into proper line of thinking by the helpful Evan Martin.</li>
<li>Started writing a <a href="https://github.com/psankar/ithavi" target="_blank">book</a> on Operating Systems in Tamil. The project is kind of stalled for a while now due to some copyright related issues with the dayjob employer. Hopefully should resume working on it by this month end.</li>
<li>Started tinkering around big data applications and large scale distributed systems. Started feeling the joy of building largescale systems using Golang. Did some prototypes for a new product idea in office and all these prior experiences helped to be <a href="http://psankar.blogspot.in/2014/11/code-churning-and-golang.html" target="_blank">very productive</a>.</li>
<li>Explored a lot of databases, especially <a href="http://cassandra.apache.org/" target="_blank">Cassandra</a>. Built it from source. Started using it in a system with millions of queries load. Explored the <a href="https://github.com/gocql/gocql" target="_blank">gocql</a> (Golang's Cassandra driver) sources.</li>
<li>Played around with a lot of key value datastores (like <a href="https://github.com/google/leveldb" target="_blank">leveldb</a>, <a href="https://godoc.org/github.com/boltdb/bolt" target="_blank">bolt</a>, <a href="http://symas.com/mdb/" target="_blank">lmdb</a> etc.) Started with a lot of hope on these and slowly started feeling that k-v stores are probably over-rated (for large scale systems). As if to prove my point, came to know that <a href="https://static.googleusercontent.com/media/research.google.com/ta//archive/spanner-osdi2012.pdf" target="_blank">Spanner</a> the successor to Bigtable is multi-columnar.</li>
<li>Learnt a bit about <a href="https://www.docker.com/" target="_blank">Docker</a>, Containers, Kubernetes etc. Should explore them more deeply this year.</li>
<li>All these work on database engines, distributed systems, distributed databases etc. lead me to read the underlying research papers of such systems. Realized that for understanding these systems, there needs to be a richer knowledge of some discrete math and <a href="http://www.amazon.in/Transaction-Processing-Concepts-Techniques-Management/dp/1558601902/" target="_blank">richer literature</a>.</li>
<li>Learnt about <a href="https://en.wikipedia.org/wiki/Paxos_%28computer_science%29" target="_blank">Paxos</a>, <a href="https://en.wikipedia.org/wiki/Raft_(computer_science)" target="_blank">Raft</a> and other distributed consensus ideas. Humbled to know of a few brilliant minds like <a href="https://en.wikipedia.org/wiki/Leslie_Lamport" target="_blank">Leslie Lamport</a>, <a href="https://en.wikipedia.org/wiki/Jim_Gray_%28computer_scientist%29" target="_blank">Jim Gray</a>, etc. Got inspired to know of a few more interesting people and <a href="https://en.wikipedia.org/wiki/VoltDB" target="_blank">ideas </a>as well.</li>
<li>Thanks to a DE in Novell, got the opportunity to play with various Amazon cloud services free of cost (there is a free tier for the curios). </li>
<li>Trying to understand how things work behind the screens in Amazon (the cloud company not just the online shop), Came across the interesting DynamoDB paper and came to know of a few interesting/inspiring people (they don't know me yet, though ;) ) like <a href="http://perspectives.mvdirona.com/" target="_blank">James Hamilton</a>, <a href="http://scalingsystems.com/" target="_blank">Swaminathan Sivasubramanian</a> , <a href="https://en.wikipedia.org/wiki/Werner_Vogels" target="_blank">Werner Vogels</a> etc. </li>
<li>Got too tempted to leave the dayjob and do some real research / Ph.D but considering the financial constraints, will probably stick to the dayjob.</li>
<li>Gave a couple of talks about Golang. One for a startup and another for a bunch of engineers. Attended a talk on Google cloud technologies and <a href="http://slashn.flipkart.net/" target="_blank">slashn</a>.</li>
<li>Came across Pig, Hive and <a href="http://www.qubole.com/" target="_blank">Qubole</a> but did not explore deeply</li>
<li>Learnt a lot about markup languages, LaTeX etc.</li>
<li>Started some work on a paper on distributed systems, only to get overwhelmed on how to proceed, considering the vastness of the topic that I have chosen for writing. Hopefully should get it into shape in the next few months or just throw it away and proceed with the daily grind.</li>
<li>Wrote a blog post about <a href="http://psankar.blogspot.in/2014/10/technology-catchup.html" target="_blank">technology catchup for the last decade</a> and a bunch of other long posts, which got some unexpected accolades.</li>
<li>Started daydreaming if I should choose a nascent area (such as Quantum Computing) to work in the freetime. It will throw up less instances of some paper published on 1960s / a patch in 1990s, for an idea, that I so enthusiastically assumed that I have invented (in current operating systems/storage etc.) until I search for it.</li>
<li>Switched to <a href="https://en.opensuse.org/openSUSE:Factory_installation" target="_blank">openSUSE Factory</a> and loving it.</li>
</ul>
Overall, a satisfactory year with relatively deeper learning. Felt that the learning could have been even more richer if I had teamed up in more like-minded, technology-driven, small teams. Should experiment that for a while next year.</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com5tag:blogger.com,1999:blog-28654204.post-74966150633196811752014-11-05T23:33:00.001+05:302014-11-05T23:33:46.378+05:30Decade of Experience and some [un]wise words<div dir="ltr" style="text-align: left;" trbidi="on">
This month, I complete working 10 years (6 months as an intern and 9.5 years as an employee) for Novell / SUSE / Attachmate / NetIQ India. During this time: I had some very good managers and some very bad managers; Worked with teams from multiple geographies (US, Germany, UK, Czech, Australia and of course India); Worked across multiple age groups (people who finished their PhDs before I was born to people who were born after the Matrix movie was released). The diversity was mainly due to the opensource nature of the work.<br />
<br />
Here are some things that I have learned in the last 10 years. Some of them may apply to you. Some of them may not. Choose at your own will. If you are interested in becoming a [product|project] manager, the following may not be helpful, but if you intend to stay a developer, it <b><i>may</i></b> be useful.<br />
<br />
<ul style="text-align: left;">
<li>In big companies, It is easier to do things and ask for excuse than to wait for permission, for trying radical changes or new things. There will always be people in the hierarchy to stop you from trying anything drastic, due to risk avoidance. Think how much performance benefits read-ahead gives; professional life is not much different. Do things without bothering about if your work will be released/approved.</li>
<li>Prototype a lot. What you cannot play around with in production/selling codebases, you can, in your own prototypes. Only when you play around enough, you will understand the nuances of the design.</li>
<li>Modern technologies get obsolete faster. People who knew just C program can still survive. But people who knew Angular 1.0 may not survive even Angular 2.0 Keep updating yourself if you want to be in the technology line for long</li>
<li>Do not become a blind fan of any technology / programming language / framework. There is no <a href="http://en.wikipedia.org/wiki/Panacea" target="_blank">Panacea</a> in software.</li>
<li>When one of my colleagues once asked a senior person for an advice, he suggested: "God gave you two ears and one mouth, use them proportionately". I second it, with an addendum, "Use your two eyes to read" also.</li>
<li>Grow the ability to zoom in and out to high/low levels as the situation demands. For example, you should know to choose between columnar or row based storage AND know about CPU branch prediction AND have the ability to switch to both of these depths at will, as the situation demands. Having a non-theoretical, working knowledge of all layers will make you a better programmer. There is even a fancy title for this quality, full-stack programmer.</li>
<li>Best way to know if you have learnt something well, is to teach it. Work with a *small* group of people with whom you can give techtalks and discuss your research interests. Remember the african proverb, If you want to go fast, go alone. If you want to go far, go together. Having a study groups helps a lot. But remember, talk is cheap and don't get sucked into becoming a theoretician. If you are interested in becoming one, get a job as a lecturer and work on hard problems. Look to Andy Tanenbaum or Eric Brewer for inspiration.</li>
<li>Keep a diary or blog of what you have learned. You can assess yourself on an yearly basis and improve yourself. Create and use your github account heavily.</li>
<li>Try different things and fail often. Failure is better than not-trying and being idle. When Rob Pike says, "<a href="https://www.youtube.com/watch?v=VoS7DsT1rdM" target="_blank">I am not used to success</a>", he is not merely being humble. It takes years of dedication, work, luck and a lot of failures to become successful and have a large / industry level impact.</li>
<li>Do not be driven too much by money or promotion or job titles. The world does not remember Alan Turing or Edsger Dijkstra or Dennis Ritchie by their bank balance or positions. There are probably a thousand software architects in your locality if you dig linkedin. Try to do good work. Also learn to <a href="https://www.youtube.com/watch?v=uxtKwJZbYr0#t=760" target="_blank">think like an author</a>.</li>
<li>Good work invariably will get appreciated, even if the appreciation may be delayed in many cases. Sloppy work will be noticed in the long term, even if it is missed in short term. The higher you grow, sloppy work may get more visibility.</li>
<li>There will be people smarter and more talented than you, always. Try to learn from them. Sometimes they may be younger than you. Don't let age stop you.</li>
<li>Work in an open source project with a very active community. Communication skills are very important even for an engineer. The best way to improve it for an engineer is to work on an open source project. Ideally, see through the full release of a linux distro. It will take you through all activities like packaging, programming, release management, marketing etc. the tasks which you may not be able to participate in your day job. I recommend <a href="http://en.opensuse.org/" target="_blank">openSUSE</a> if you are looking for a suggestion ;)</li>
<li>Except for Mathematics and Music, there is no other field with prodigies. Understand <a href="https://www.youtube.com/watch?v=0SARbwvhupQ" target="_blank">the myth of genius programmer</a>.</li>
<li>There are a lot of bad managers (at least in India). Most of these managers are bad at management, because they were lousy engineers in the first place and so decided to do an MBA and become a people manager, after which they don't have to code (at least in India). If you get one of them, do not try to fight them. Work with them on a monthly basis with a record of objectives and progresses. The sooner you get a bad manager, the better and faster you will appreciate good managers.</li>
<li>Last but not least, identify a very large, audacious problem and throw yourself at it fully. All the knowledge that you have accumulated over the years with constant prototyping and reading will come in handy while solving it. In addition, you will learn a thousand other things, which you could not have learned by lazily building knowledge by reading alone. But the goal that you need to work on has to be audacious (like the goals that gave us Google filesystem or AWS etc.) and solve a very big problem. However, you should start this only after a few years of building a lot of small things and have a full quiver. To become a good system-side programmer, you should have been a good userspace programmer. To become a good distributed systems developer, you should have used a distributed system, etc. </li>
</ul>
<div>
May be one another point that I could add is: Try to write short and crisp (<b>unlike this blog post</b>).</div>
</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com1tag:blogger.com,1999:blog-28654204.post-6460171000038268812014-11-02T13:31:00.000+05:302014-11-02T13:31:30.458+05:30code churning and golang<div dir="ltr" style="text-align: left;" trbidi="on">
Recently, we were doing a new prototype in dayjob. I had the freedom to choose the technology stack for this idea. I wrote a lot of golang code to compare a few aspects across few technologies (say <i>streaming writes perf stats for Cassandra vs MariaDB</i> etc.) to evaluate some of these technologies for our needs.<br />
<br />
The whole activity spanned for about 12 weeks roughly and we were able to build a very good evolutionary prototype. I was looking at the gitlab stats at the end of 12 weeks and found my personal log to be:<br />
<br />
<b>9752 lines added</b><br />
<b>7119 lines deleted</b><br />
<br />
Even if we assume a 6 day workweek, it translates to about <b>135 lines of new go code added per day by a developer</b> on an average. There have been very productive days, where I was able to add more than 400 lines of non-copy-paste code in a single day, that I ended up having to take rest the next day, to recover.<br />
<br />
In the past, I have written a lot of C code. I have never felt this productive in C, largely due to the manual memory management (and the ensuing problems like double free, leaks, valgrinding etc.) and difficult concurrency (pthreads, locks, etc.)<br />
<br />
It is kind of obvious that Go will naturally feel more productive, due to automatic memory management and concurrency friendly features (goroutines, channels, etc.), resulting in very less non-business-code.<br />
<br />
However, I observed that there are two other <b>non-intuitive</b> reasons why Go lang was very productive (for me). These reasons do not appear big on their own. But in the overall development time, they were a big influence on my productivity. They are:<br />
<br />
<h3 style="text-align: left;">
1) Static Binary Creation without complex, external build tools</h3>
Thanks to my openSUSE packaging experience, I have always taken up the responsibility to keep the sources of the project where I work as an engineer, in a properly and a packager-friendly build system. I like build friendly sources to an extent that, about an year ago, One of the first tasks that I did, when I moved to a team, was to port an old packaging system of hand-written makefiles and obsolete build systems with sources across tens of thousands of files and managed for about two decades, to CMake. IOW, I know about Linux packaging and its pains.<br />
<br />
With go, I was able to easily build the sources and get all the dependencies via a single `<a href="https://golang.org/doc/articles/go_command.html" target="_blank">go get</a>` command. Installing the binary on a test cluster, or in the AWS was merely a single command away. There was no need to wait for any complicated build setup, setting up dependencies or even waiting hours for a build to finish. There is no need to write complicated Makefiles, CMakeFiles, Configure files, build scripts etc.<br />
<br />
Usage of the `go get` tool mandates developers to follow a certain discipline regarding installation/inclusion of libraries, binaries. Static binary generation helps avoid a tonne of deployment hassles. All these minor things, <b>when we do a dozen or more builds in a day, add up to a very big productivity boost</b>. It is not even that uncommon to do a dozen builds in a day, to aid testers, in the prototyping stage. Because of the elegance and simplicity of `go get`, the testers did not even have to wait on dedicated packagers or on developers to get the testbuilds. Even if you don't have dedicated testers, static binary generation, simplifies your test setup time.<br />
<br />
<h3 style="text-align: left;">
2) Composition instead of Inheritance</h3>
This point is very difficult to explain, as it is more abstract, but is more influential than the previous. In the beginning, I was struggling to get Composition right. I ended up trying to organize my files based on an inheritance model (much like the [fs/< files>.c , fs/< ext>/<files .c="">, fs/< btrfs>/< files.c>] in the linux kernel), trying to get a baseclass delegating things to a derived class manually based on a derived class identifier in the object etc. I struggled and did not feel productive in coding.</files><br />
<br />
I had to pause, unlearn a few things and think in a fresh perspective again to understand it. <b>Composition is like Cycling</b>. Once you get the hang of it, there is no falling down. I felt that the Composition based model has helped a lot more than any other feature of golang to improve my productivity.<br />
<br />
With composition, the amount of code changes needed when you refactor code (which is very common in most freshly written code) is very very less than in a code, designed for inheritance. It is very hard to explain how this helps in simple English words. But I recommend you write code for yourself and appreciate this. In addition to easy refactoring, Composition tends to reduce <a href="http://en.wikipedia.org/wiki/Boilerplate_code" target="_blank">boilerplate</a> code substantially and makes <a href="http://en.wikipedia.org/wiki/Multiple_inheritance" target="_blank">diamond problem</a> obsolete.<br />
<br />
The <a href="http://golang.org/pkg/net/http/#Client" target="_blank">Embedding</a> of the Transport object in go lang's http Client object helped me understand Composition a lot clearer than any tutorial or book.<br />
<br />
<h3 style="text-align: left;">
Conclusion</h3>
Because, I was able to write a lot of code fast, I was not too scared to shed code and start from scratch when needed. This explains the about 7k deletions of code.<br />
<br />
<a href="https://github.com/bradfitz/goimports" target="_blank">goimports</a> and <a href="https://github.com/fatih/vim-go" target="_blank">vim-go</a> also helped a lot to get some IDE like features, all of which should thank <a href="https://golang.org/cmd/gofmt/" target="_blank">gofmt</a> in return.<br />
<br />
Have you felt any other reason that made you feel a high-level of code churning can be achieved in Go ?</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com5tag:blogger.com,1999:blog-28654204.post-62736816578548386192014-10-13T13:31:00.002+05:302014-10-13T13:31:33.596+05:30Technology Catchup<div dir="ltr" style="text-align: left;" trbidi="on">
Coincidentally three different people asked me in the last month, to write about new technologies that they should be knowing, to make them more eligible to get a job in a startup. All these people have been C/C++ programmers, in big established companies, for about a decade now. Some of them have had only glimpses of any modern technologies.<br />
<br />
I have tried a little bit (with moderate success) to work in all layers of programming with most of the popular modern technologies, by writing <a href="http://psankar.blogspot.in/2014/01/2013-learning-retrospective.html" target="_blank">little-more-than-trivial programs</a> (long before I heard of the fancy title "<b>full stack developer</b>"). So here I am writing a "technology catchup" post, hoping that it may be useful for some people, who want to know what has happened in the technologies in the last decade or so.<br />
<br />
Disclaimer 1: The opinions expressed are totally biased as per my opinion. You should work with the individual technologies to know their true merits.<br />
<br />
Disclaimer 2: Instead of learning everything, I personally recommend people to pick whatever they feel they are connected to. I, for example, could not feel connected to node-js even after toying with it for a while, but fell in love with Go. Tastes differ and nothing is inferior. So give everything a good try and pick your choice. Also remember what Donald Knuth said, "There is difference between knowing the name of something and knowing something". So learn deeply.<br />
<br />
Disclaimer 3: From whatever I have observed, getting hired in a startup is more about being in the right circles of connection, than being a technology expert. A surprisingly large number of startups start with <i><b>familiar</b></i> technology than with the <i><b>right</b></i> technology, and then change their technology, once the company is established.<br />
<br />
Disclaimer 4: This is actually not a complete list of things one should know. These are just things that I have come across and experimented a little bit at least. There are a lot more interesting things that I would have have missed. If you need something must have been in the list, please comment :-)<br />
<br />
With those disclaimers away, let us cut to the chase.<br />
<b><br /></b>
<br />
<h2 style="text-align: left;">
<b>Version Control Systems</b></h2>
The most prominent change in the open source arena, in the last decade or so, is the invention of <a href="http://git-scm.com/" target="_blank"><b>Git</b></a>. It is a version controlled system initially designed for keeping the kernel sources and has since then become the de-facto VCS for most modern companies and projects.<br />
<br />
<a href="https://github.com/" target="_blank"><b>Github</b></a> is a website that allows people to host their open source projects. Often startups recruit people based on their github profile. Even big companies like microsoft, google, facebook, twitter, dropbox etc. have their own github accounts. I personally have received more job queries through my github projects than via my linkedin profile in the last year.<br />
<br />
<a href="https://bitbucket.org/" target="_blank"><b>bitbucket</b></a> is another site that allows people to host code and give even private repos. A lot of the startups that I know of use this, along with the <a href="https://www.atlassian.com/software/jira" target="_blank">jira</a> project management software. This is your equivalent of MS Project in some sense.<br />
<br />
I have observed that most of the startups founded by people who come from Banking or Finance companies to be using <a href="https://subversion.apache.org/" target="_blank"><b>Subversion</b></a>. Git is the choice for people from tech companies though. <a href="http://mercurial.selenic.com/" target="_blank"><b>Mercurial</b></a> is another open source, distributed VCS which has lost a lot of limelight in the recent times, due to Git. <a href="http://www.fossil-scm.org/index.html/doc/tip/www/index.wiki" target="_blank"><b>Fossil</b></a> is another VCS, from the author of sqlite, Dr. Richard Hipp. If you can learn only one VCS for now, start with Git.<br />
<br />
<h2 style="text-align: left;">
<b>Programming Languages & Frameworks</b></h2>
<a href="http://www.w3schools.com/js/" target="_blank"><b>Javascript</b></a> has evolved to be a leading programming language of the last decade. It is even referred to as the X86 of the web. From its humble beginnings as a client-side scripting language to validate if the user has typed a number or text, it has grown into a behemoth and entered even the server-side programming through the <a href="http://nodejs.org/" target="_blank"><b>node-js</b></a> framework. For incorporating <a href="http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller" target="_blank">ModelViewController</a> pattern, javascript has gained the <a href="https://angularjs.org/" target="_blank"><b>AngularJS</b></a> framework. JS is a dynamically typed language and to bring in some statically typed langauges' goodness, we have a <a href="http://coffeescript.org/" target="_blank"><b>coffeescript</b></a> language too.<br />
<br />
<a href="https://www.python.org/" target="_blank"><b>Python</b></a> is another dynamically typed, interpreted programming language. Personally, I felt that it is a lot more tasteful than Javascript. It feels good on eyes too. It helps in rapid application development and is available by default in almost all the Linux distros and Mac machines by default. <a href="https://www.djangoproject.com/" target="_blank"><b>Django</b></a> is a web framework that is built on python to make it easy to develop web applications. In addition to being used in a lot of startups, it is used in even big companies like Google and Dropbox. There are variants of Python runtime such that you can run it in the JVM using <a href="http://www.jython.org/" target="_blank"><b>Jython</b></a> or in the .NET CLR using the <a href="http://ironpython.net/" target="_blank"><b>IronPython</b></a>. I have personally found this language to be lacking in performance though, which is elaborated more in a subsequent section.<br />
<br />
<a href="http://en.wikipedia.org/wiki/Ruby_(programming_language)" target="_blank"><b>Ruby</b></a> is an old programming language that shot into fame in the recent years through the popular web application framework <a href="http://rubyonrails.org/" target="_blank"><b>Ruby on Rails</b></a>, often called just Rails. I have learnt a lot of engineering philosophies such as <a href="http://en.wikipedia.org/wiki/Don%27t_repeat_yourself" target="_blank">DRY</a>, <a href="http://en.wikipedia.org/wiki/Convention_over_configuration" target="_blank">COO</a> etc. while learning RoR.<br />
<br />
All these above languages and frameworks use a package manager such as <a href="https://www.npmjs.org/" target="_blank">npm</a>, <a href="http://bower.io/" target="_blank">Bower</a>, <a href="https://pypi.python.org/pypi/pip" target="_blank">pip</a>, <a href="https://rubygems.org/" target="_blank">gems</a> etc. to install libraries easily.<br />
<br />
<a href="http://golang.org/" target="_blank"><b>Go</b></a> is my personal favorite in the new languages to learn. I see Go becoming as vital and prominent a programming language as C, C++ or Java in the next decade. It is developed in Google for creating large scale systems. It is a statically-typed, automatic-memory-managed language that generates native-machine-code and helps writing concurrent-code easily.<br />
<br />
Go is the default language that I use for any programming task in the last year or so. It is amazingly fast even though (just because?) it is still in the 1.X series. In my dayjob we did a prototype in both go and python, and for a highly concurrent workflow in the same hardware, Go puffed Python in performance (<b>20 seconds vs 5 minutes</b>). I won't be surprised if a lot of the python and ruby code gets converted to golang in their next edition of rewrites. Personally, I have found the quality of go libraries to be much higher compared to Ruby or nodejs as well, probably because not everyone has adapted to this language yet. However, this could be just my personal biased opinion.<br />
<br />
If you like to get fancy with functional programming, then you can learn <a href="http://www.scala-lang.org/" target="_blank"><b>Scala</b></a> (on top of JVM), <a href="http://fsharp.org/" target="_blank"><b>F#</b></a> (on top of .NET), <a href="http://www.haskell.org/haskellwiki/Haskell" target="_blank">Haskell</a>, <a href="http://www.erlang.org/" target="_blank">Erlang</a>, etc. The last two are very old btw but in use even today. Most recently, <a href="http://www.whatsapp.com/" target="_blank">Whatsapp</a> was known to use Erlang. <a href="http://dlang.org/" target="_blank">D</a> is also seen in the news, mostly thanks to Facebook. <a href="https://www.dartlang.org/" target="_blank"><b>Dart</b></a> is another language that is from Google but still to receive any wide deployment afaik, even with Google's massive marketing machinery behind it. It has been compared to VBscript and is criticized, and as of now chrome-only. Dart has received criticism from Mozilla, Webkit (rendering engine that powers Safari (and chrome earlier)), Microsoft IE as well. Dart is done by Lars Bak et al. (the people who gave us V8, chrome's Javascript engine)<br />
<br />
<a href="http://www.rust-lang.org/" target="_blank"><b>Rust</b></a> is another programming language that is aimed for high-performance concurrent systems. But I have not played around with it, as they don't maintain a stable API and they are not 1.0 yet. <a href="http://julialang.org/" target="_blank"><b>Julia</b></a> is another programming language aimed at doing distributed systems, about which I have heard a lot of praise, but it still remains a exotic language afaik. <a href="http://www.r-project.org/" target="_blank"><b>R</b></a> is another language which I have seen in a lot of corporate demos where the presenters wanted to show statistics, charts. Learning this may be useful <i>even if you are not a programmer and works with numbers (like a project manager)</i>.<br />
<br />
<br />
There is a <a href="https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/" target="_blank">Swift</a> programming language from Apple to write iOS apps. I have not tried Swift yet, but from my experience of using <a href="https://developer.apple.com/library/mac/documentation/cocoa/conceptual/ProgrammingWithObjectiveC/Introduction/Introduction.html" target="_blank">Objective C</a>, it cannot be worse.<br />
<br />
<a href="http://getbootstrap.com/" target="_blank"><b>Bootstrap</b></a> is a nice web framework from twitter, which provides various GUI elements that you can incorporate into your application, to rapidly prototype beautiful applications, that are fluidic even when viewed in mobile.<br />
<br />
<a href="http://jquery.com/" target="_blank"><b>jquery</b></a> is a popular javascript library that is ubiquitous. <b>Cascading Style Sheets</b> (shortly CSS) is a markup language that helps configure the style of the web page UI elements. CSS is becoming mature to the extent of showing animations too. You should ideally spend a few weeks to learn about HTML5 and CSS.<br />
<h2 style="text-align: left;">
<b>Text Editors</b></h2>
<div>
<a href="http://www.sublimetext.com/" target="_blank"><b>Sublimetext</b></a> is what the cool kids use these days as the editor. I have found the tutorial on <a href="http://code.tutsplus.com/courses/perfect-workflow-in-sublime-text-2" target="_blank">tutsplus</a> to be extra-ordinarily good at explaining sublime. It is a free (as in beer) software and not open source.</div>
<div>
<br /></div>
<div>
<a href="https://atom.io/" target="_blank"><b>Atom</b></a> is a text-editor from github built using nodejs and chromium. I did not find a linux binary and so did not bother to investigate it. But I have heard it to be good for Javascript programmers than any others, as the editor could be extended by javascript itself.</div>
<div>
<br /></div>
<div>
<a href="http://brackets.io/" target="_blank"><b>Brackets</b></a> is another editor that I have heard good things about. <a href="http://limetext.org/" target="_blank"><b>Lime</b></a> is an editor that is developed in Go, aimed to be an open-source replacement for the sublimetext.</div>
<div>
<br /></div>
<div>
Personally, after trying various text editors, I have always comeback to using <a href="http://www.vim.org/" target="_blank"><b>vim</b></a>. There are a few good plugins for vim in the recent times. <a href="https://github.com/gmarik/Vundle.vim" target="_blank">Vundle</a>, <a href="https://github.com/tpope/vim-pathogen" target="_blank">Pathogen</a> are nice plugin managers for vim to ease up installation of plugins. <a href="https://github.com/Valloric/YouCompleteMe" target="_blank">YouCompleteMe</a> is a nice plugin for auto-completion. <a href="http://vim.spf13.com/" target="_blank"><b>vim-spf13</b></a> is a nice distro of vim, where various plugins and colorschemes are pre-packaged.</div>
<div>
<br /></div>
<h2 style="text-align: left;">
Distributed Computing</h2>
<div>
In the modern day of computing, most programs have been driven by a Service Oriented Architecture (shortly SOA). Webservices are the preferred way of communication among servers as well. While we are talking about services, please read <a href="https://plus.google.com/+RipRowan/posts/eVeouesvaVX" target="_blank">this</a> nice piece by Steve Yegge.</div>
<div>
<br /></div>
<div>
<a href="http://memcached.org/" target="_blank"><b>memcached</b></a> is a distributed (across multiple machines), caching system which can be used in front of your database. This was initially developed by Brad Fritzpatrick, while he was the head of the LiveJournal and who is now (2014) a member of the Go team at Google. While at Google, he has started <a href="https://github.com/golang/groupcache" target="_blank">GroupCache</a> which as the project page says is a replacement for memcache in many cases.</div>
<div>
<br /></div>
<div>
<a href="http://static.googleusercontent.com/media/research.google.com/ta//archive/gfs-sosp2003.pdf" target="_blank"><b>GoogleFileSystem</b></a> (GFS) is a seminal paper on how Google created a filesystem to suit their large needs of data processing. There is a database built on top of this filesystem named <a href="http://static.googleusercontent.com/media/research.google.com/ta//archive/bigtable-osdi06.pdf" target="_blank">BigTable</a> which powered Google's infrastructure. <a href="http://hadoop.apache.org/" target="_blank"><b>Apache Hadoop</b></a> is an open source implementation of these concepts, which was originally started in Yahoo and now a top-level apache project. <a href="http://hortonworks.com/hadoop/hdfs/" target="_blank"><b>HDFS</b> </a> is the equivalent of GFS for the Hadoop. <a href="https://hive.apache.org/" target="_blank">Hive</a> and <a href="http://pig.apache.org/" target="_blank">Pig</a> are technologies to query and analyze data from the Hadoop.</div>
<div>
<br /></div>
<div>
As with the evolution of any software, GFS has evolved into a <a href="http://static.googleusercontent.com/media/research.google.com/ta//university/relations/facultysummit2010/storage_architecture_and_challenges.pdf" target="_blank"><b>Colossus</b></a> filesystem and BigTable has evolved into a <a href="http://static.googleusercontent.com/media/research.google.com/ta//archive/spanner-osdi2012.pdf" target="_blank"><b>Spanner</b></a> distributed database. I recommend you to read these papers even if you are not going to do any distributed computing development.</div>
<div>
<br /></div>
<div>
<a href="http://cassandra.apache.org/" target="_blank"><b>Cassandra</b></a> is another distributed database which was started in Facebook initially, but is used in many companies such as <a href="http://techblog.netflix.com/2014/07/revisiting-1-million-writes-per-second.html" target="_blank">Netflix</a> and Twitter. I have used Cassandra more than any other distributed project and actually like it a lot. It uses a SQL like query language called CQL - Cassandra Query Language. It is modelled after the <a href="http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf" target="_blank"><b>DynamoDB</b></a> paper from Amazon. I am too tempted to write an alternative to this in Go, just to have the idea of writing a large scale distributed system, instead of just using it as a client, but have not got around to a good dataset or usecase with which I can test it.</div>
<div>
<br /></div>
<div>
<a href="http://www.mongodb.org/" target="_blank"><b>MongoDB</b></a> is another document oriented database, which I tried using for a <a href="https://github.com/psankar/alakkai" target="_blank">pet project</a> of mine. I don't remember exactly but there were some problems with respect to unicode handling. The project was done prior to go becoming 1.0, so the problem could be in any end.<br />
<br />
Most of the new age databases are called NOSQL databases but what they really mean is that the database skips a lot of functions (such as datatype validation, stored procedures, etc.) and try to grow by scaling out instead of scaling up.<br />
<br />
<h2 style="text-align: left;">
Cloud</h2>
<a href="http://www.openstack.org/" target="_blank"><b>OpenStack</b></a> is a suite of open source projects that help you create a private cloud. <a href="https://deltacloud.apache.org/" target="_blank"><b>DeltaCloud</b></a> is a project which was initially started by RedHat, and now an apache top-level project, as a way to provide a single API layer which will work across any cloud in the backend. This project is done in ruby. I was initially interested in participating in its development, until I got introduced to Go and moved into a different tangent.<br />
<br />
To start off a software company is a very easy task to do in today's world. The public clouds are becoming cheaper and cheaper everyday and their capacity can be provisioned instantly.<br />
<br />
<a href="http://aws.amazon.com/" target="_blank"><b>Amazon web services</b></a> provides an umbrella of various public cloud offerings. I have used <a href="http://aws.amazon.com/ec2/" target="_blank"><b>Amazon EC2</b></a> which is a way to create a Linux (and windows) VM that runs on Amazon's datacenters. The machines come on various sizes. <a href="http://aws.amazon.com/s3/" target="_blank"><b>Amazon S3</b></a> is a cloud offering that provides you way to store data in buckets. This is used by Dropbox heavily for storing all your data. There are various <a href="http://aws.amazon.com/products/?nc1=f_cc" target="_blank">other services</a> too. In some of our prototyping, we found the performance of Amazon EC2, to be consistent mostly, even in the free tier.<br />
<br />
Google is not lagging behind with their cloud offerings either. When Google Reader was shut down, I used Google's <a href="https://appengine.google.com/" target="_blank"><b>Appengine</b></a> to deploy an <a href="https://github.com/mjibson/goread" target="_blank">alternative FOSS</a> product and I was blown away by the simplicity of creating applications on top of it. <a href="https://cloud.google.com/compute/" target="_blank"><b>Google Compute</b></a> is the way to get VMs running on the Google Cloud. As with Amazon, there are plenty of <a href="https://cloud.google.com/" target="_blank">other services</a> too.<br />
<br />
There are plenty of other players like Microsoft Azure, <a href="https://www.heroku.com/" target="_blank"><b>Heroku</b></a> etc. but I do not have any experience with their applications. While we are talking about Cloud, you should probably read about <a href="http://en.wikipedia.org/wiki/Orchestration_(computing)" target="_blank">Orchestration</a> and know about at least <a href="http://zookeeper.apache.org/" target="_blank">Zookeeper</a>.<br />
<br />
<h2 style="text-align: left;">
In-Process Databases</h2>
<div>
These are databases which you can embed into your application, without needing a dedicated server. They run on your process-space.</div>
<div>
<br /></div>
<a href="http://www.sqlite.org/" target="_blank"><b>sqlite</b></a> is the world's most deployed software and it competes with fopen to become the default way to store data for your desktop applications (if you are still writing them ;) ). A <a href="https://sqlite.org/src4/doc/trunk/www/lsmusr.wiki" target="_blank">new branch</a> is coming with the latest rage on storage datastructures, <a href="http://en.wikipedia.org/wiki/Log-structured_merge-tree" target="_blank">a log-structured merge tree</a> as well.<br />
<br /></div>
<div>
<a href="https://github.com/google/leveldb" target="_blank"><b>leveldb</b></a> is a database that is written by the eminent Googlers (and trendsetters of technology in the last decade or so) <a href="https://www.youtube.com/watch?v=SALKZV0ER-4" target="_blank">Jeff Dean and Sanjay Ghemawa</a>t who gave us MapReduce, GFS etc. It is forked by Facebook into <a href="http://rocksdb.org/" target="_blank"><b>RocksDB</b></a> as well.<br />
<br />
<a href="http://fallabs.com/kyotocabinet/" target="_blank">KyotoCabinet</a> and <a href="http://symas.com/mdb/" target="_blank">LMDB</a> are other projects on this space.<br />
<br />
<h2 style="text-align: left;">
Linux Filesystems</h2>
</div>
<div>
Since we have covered GFS, HDFS, etc. earlier. We will look at other popular filesystems.<br />
<br />
<a href="https://btrfs.wiki.kernel.org/index.php/Main_Page" target="_blank"><b>btrfs</b></a> is a copy-on-write filesystem in Linux. It is intended to be the defacto linux filesystem in the future, possibly obsoleting ext series in the longer run.<br />
<br />
<a href="http://lwn.net/Articles/476263/" target="_blank"><b>XFS</b></a> is a filesystem that initially came from SGI to Linux. This is my personal favorite and I have been using it on all my linux machines. In addition to good performance, this offers robustness and comes with a load of features that are useful to me, like defragmentation.<br />
<br />
We also have the big daddy of filesystems <a href="http://zfsonlinux.org/" target="_blank"><b>zfs</b></a> too on linux.<br />
<br /></div>
<div>
<a href="http://ceph.com/" target="_blank"><b>Ceph</b></a> is another interesting distributed filesystem that works on the kernel space and is already merged in the linux kernel sources for a long time now. <a href="http://www.gluster.org/" target="_blank"><b>GlusterFS</b></a> is another distributed filesystem which works in the userspace. Both of these filesystems focus on <a href="http://en.wikipedia.org/wiki/Scalability#Horizontal_and_vertical_scaling" target="_blank"><b>scaling out instead of scaling up</b></a>.</div>
<br />
<h2 style="text-align: left;">
Conclusion</h2>
<div>
Pick any of these technologies that you like and start writing a toy application on it, may be as simple as a ToDo application and learn through all the stages. This approach has helped me. It may help you also.</div>
<div>
<br /></div>
<div>
I have written this post from a Thinkpad T430 running openSUSE Factory and GNOME Shell with a bunch of KDE tools. I like this machine, However, in the past few months I have realized that, in today's world, <b><i>If you are a developer, it is best if you run Linux on your server and Mac on your laptop</i></b>.</div>
</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com25tag:blogger.com,1999:blog-28654204.post-23702728700338336732014-09-11T10:48:00.000+05:302014-09-11T10:48:24.117+05:30Kernel Development Beginner<div dir="ltr" style="text-align: left;" trbidi="on">
Yesterday <a href="http://vignesh.foamsnet.com/" target="_blank">Vignesh</a> asked me if I could give some guidance to a college junior of mine who wants to start with Kernel programming. Being a <a href="http://en.wikipedia.org/wiki/Novell_Storage_Services" target="_blank">filesystem developer</a> on Novell for a while now, I thought I could share some things that I have learned. I wrote a somewhat long reply which I am reproducing below (with minor edits for clarity) in the hope that it may be useful to someone.<br />
<br />
Since it was originally intended to be a mail, it is a little more verbose than a blog post. My advice is based on the situation on my college when I studied a decade ago. Things would have probably changed and the recommendations may need tweaking based on the context.<br />
<br />
---<br />
<br />
The most important quality that you need to inculcate if you want to do any kernel space programming is "Patience" (or persistence if you will). Though it is a good quality for any large scale project, it is a fundamental requirement for kernel programming. It is very easy to see progress and make an impact on userspace projects, but even simple changes in the kernel core will take a lot of time to get accepted, and will often require multiple rewrites. But fear not, as there are plenty of people who have conquered this mountain and it is not something to be worried about.<br />
<br />
The starting steps will be:<br />
<br />
1) Try to understand how to use <a href="http://git-scm.com/" target="_blank">git</a>. We were (are ?) not taught to use a version control system in our college and it is such a fundamental thing. So start using git for college assignments and get the hang of it.<br />
<br />
2) Start writing a lot of C programs and get experienced with pointers, memory allocation, threading. You can start implementing things like Stack, Queue, Trees etc. (whatever you study in datastructures) in a simple, thread-safe way. Do not focus on how you can visualize these datastructures but how you can effectively implement their functionality and thread safety. Use pthreads for threading. Do not use any library (like Glib) for giving you convenient datastructures (like Strings). Implement each of the things on your own. (But when you are writing code for a product, use a standard library always instead of re-inventing the wheel)<br />
<br />
Write these C programs on Linux and compile using gcc. In our college days we were using turboc on windows and I hope things have changed. Use a linux distro (<a href="http://fedoraproject.org/" target="_blank">fedora</a>, <a href="https://www.debian.org/" target="_blank">debian</a>, <a href="http://en.opensuse.org/" target="_blank">openSUSE</a>, <a href="https://www.gentoo.org/" target="_blank">Gentoo</a> etc.) exclusively; Do not use Windows (at least for a while) to make yourself aware of the sysadmin, shell-scripting parts of linux, which will come in handy.<br />
<br />
3) Grab a (any) book on Operating Systems theory and read it. The <a href="http://en.wikipedia.org/wiki/Operating_System_Concepts" target="_blank">dinosaur book</a> by Silberschatz et. al. is a good start.<br />
<br />
4) Without hesitation buy, <a href="http://www.amazon.com/gp/product/0672329468/" target="_blank">Robert Love's Linux Kernel Programming</a> book. It is one of the best beginner material and start reading it parallel to the OS book. This is easier to read than the previous one and more practical. But the previous one adds more value and is more theoretical. Handle (3) and (4) in parallel without blocking on any of the other activities.<br />
<br />
5) After you are done with (1) and (2), and feel sufficiently confident with C and pointers, grab the linux kernel sources from <a href="http://git.kernel.org/">http://git.kernel.org/</a> and try to build the sources yourself. <a href="http://kernelnewbies.org/KernelBuild">http://kernelnewbies.org/KernelBuild</a> should help. Learn how to install and boot with the kernel that you have built.<br />
<br />
6.1) Subscribe to Kernel Newbies mailing list <a href="http://kernelnewbies.org/MailingList">http://kernelnewbies.org/MailingList</a> and read every mail, *even* if you do not understand most of it.<br />
<br />
6.2) Watch: <a href="https://www.youtube.com/watch?v=LLBrBBImJt4">https://www.youtube.com/watch?v=LLBrBBImJt4</a><br />
<br />
6.3) Subscribe to <a href="http://lwn.net/">http://lwn.net</a> RSS feeds.<br />
<br />
After this, you should be able to fix and send any trivial, documentation, staging fixes. Once you have done this and get the hang of the process, you will know how to send patches for any parts of the kernel.<br />
<br />
By this time, you would have found your areas of interest in kernel (filesystems, memory management, io scheduler, CPU scheduling etc.). You will then have to dig deeper in those particular areas, by:<br />
a) subscribing to the individual mailing lists (such as <a href="http://marc.info/?l=linux-fsdevel" target="_blank">fs-devel</a>, etc.)<br />
b) reading about the bug reports for the individual component<br />
c) finding the literature that is relevant for your subsystem (<a href="https://www.kernel.org/doc/gorman/pdf/understand.pdf" target="_blank">The linux memory management book</a> Mel Gorman, etc).<br />
<br />
Three other non-technical things that I would recommend are:<br />
<br />
1) Create a new email address and use that for all your open source activities. That way you do not miss any important updates from your friends.<br />
<br />
2) Kernel programming will not give you big money in the short and medium term (at least in India). If your motivation is not excellence in engineering, but becoming popular or rich (it is not wrong btw) then you should focus on some other areas of programming (developing apps, websites, solving user problems, making meaning etc.).<br />
<br />
It will often take months (or even years) before you make a significant contribution that is not merely a memory leak or bug fix. Be prepared for that. But since you have age, energy, time (once you get married and/or have kids you will understand) on your side, it is not that difficult.<br />
<br />
Many people try kernel programming and then quit because they do not have the patience and perseverance. It may also happen that they have found a more interesting technology at its nascent stage (like Distributed Computing, Artificial Intelligence, Containers, NLP etc.) It is not wrong to quit midway :) Any little time spent on kernel programming will immensely benefit you as a programmer even when you are doing user space programming. This holds good for not just kernel programming but any large-code-base/system programming (like Compilers, glibc, webkit, chrome, firefox etc.)<br />
<br />
3) Be more aware of the GSoC community in colleges around you.<br />
<br />
All the best.</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com5tag:blogger.com,1999:blog-28654204.post-4110838342685367742014-01-17T13:03:00.000+05:302014-01-17T13:03:14.704+05:302013 Learning Retrospective<div dir="ltr" style="text-align: left;" trbidi="on">
My open source work + learning in 2013 was not bad. Considering the lack of personal time due to changes in family and other reasons, I think I did well. Some of the things done are:<br />
<br />
<ul style="text-align: left;">
<li>Learnt a lot about kernel space filesystems. Tried to write a beginner level introductory tutorial and then abandoned it. Instead started writing a simple kernel space filesystem from the scratch that can help as a teaching material - <a href="https://github.com/psankar/simplefs">https://github.com/psankar/simplefs</a></li>
<li>Implemented B Trees in Go. Researched a bit about B Trees, Log structured merge trees and gave a talk on how B Trees are used in filesystems <a href="https://github.com/psankar/btree-go">https://github.com/psankar/btree-go</a></li>
<li>Released a new version of my chrome extension with support for searching and highlighting multiple strings in Chrome <a href="https://github.com/psankar/Find-Many-Strings">https://github.com/psankar/Find-Many-Strings</a></li>
<li>Learnt a bit about ext4 filesystem (de)fragmentation implementation. Started working on some tools to simulate filesystem ageing and filesystem fragmentation. Work in progress. This is part of the dayjob though and not done in the freetime</li>
<li>Learnt more about the Go programming language. Almost all the projects last year were done on Go. Understood how to deploy my application in the Google cloud engine. Did some minor comparisons across various online appengines, such as heroku, amazon, google etc.</li>
<li>Gave SublimeText a honest try and just as it happens with every other editor, came back to Vim, although this time with a few new plugins: <a href="https://github.com/Valloric/YouCompleteMe">https://github.com/Valloric/YouCompleteMe</a> & <a href="https://github.com/tpope/vim-pathogen">https://github.com/tpope/vim-pathogen</a> specifically</li>
<li>Inspired by gotour, Implemented a HTML 5 based slideshow tool where the slides can be written in markup, slide contents can talk to any compiler and show the output in the webpages. This will come handy in teaching programming languages <a href="https://github.com/psankar/kuvalai">https://github.com/psankar/kuvalai</a></li>
<li>Learned about <a href="http://getbootstrap.com/" target="_blank">bootstrap </a>and <a href="http://angularjs.org/" target="_blank">angularjs</a>. Did some non-trivial websites using these technologies</li>
<li>Learned a bit about the architecture of various distributed filesystems </li>
<li>As an extra unplanned activity in the dayjob, did a big buildcleanup by migrating to CMake from a proprietary buildsystem and was able to shed about 30k lines of buildfiles. Became well-versed in <a href="http://cmake.org/">CMake</a>, RPM generation as a side effect. Tried to play with <a href="http://martine.github.io/ninja/">ninja</a>, <a href="http://gittup.org/tup/">tup</a> too and learnt about their architecture</li>
<li>Read few nice papers such as The Ubiquitous BTree by Douglas Comer, <a href="http://www.arl.wustl.edu/~fredk/Courses/cs523/fall01/Papers/kleiman86vnodes.pdf">Vnodes: An Architecture for Multiple FileSystem Types in Sun UNIX</a>, <a href="http://gittup.org/tup/build_system_rules_and_algorithms.pdf" target="_blank">Build System Rules and Algorithms,</a> <a href="http://hal.archives-ouvertes.fr/docs/00/78/90/86/PDF/a_survey_of_dfs.pdf" target="_blank">Analysis of six distributed filesystems</a> etc. </li>
<li>Learnt a bit about linear time sorting by reading papers on sorting. Could not get to implementation sadly</li>
</ul>
Things that I wanted to do but could not do<br />
<ul style="text-align: left;">
<li>Take some coursera courses on Security, Compilers, Hardware software interface and Algorithms</li>
<li>Learn Rust, Haskell and Dart programming languages</li>
<li>Research more about Logstructured merge tree, implementing it and write a blogpost (or get more understanding) on how the changing hardware (SSDs etc.) affect our datastructures, performance etc. from a storage perspective</li>
<li>Get a technical paper published :( </li>
<li>Spend more time in the openSUSE community and mailing lists</li>
</ul>
</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com14tag:blogger.com,1999:blog-28654204.post-27853725240274991422013-09-25T16:36:00.000+05:302013-09-25T16:36:01.673+05:30Introducing Find Many Strings v2 - A chrome extension<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
Some of you might remember my chrome extension to search and highlight multiple strings simultaneously. I <a href="https://chrome.google.com/webstore/detail/find-many-strings/phabnclfbnmhmoipebjdfhkiacojlcnc" target="_blank">made an update</a> to it recently and gave the ability to input multiple strings in one click. Here is an introductory video of <a href="https://chrome.google.com/webstore/detail/find-many-strings/phabnclfbnmhmoipebjdfhkiacojlcnc" target="_blank">the extension</a> in action. Recommended to be watched in full screen.<br />
<br />
<iframe allowfullscreen="" frameborder="0" height="315" src="//www.youtube.com/embed/IgB0_P5pPz4?rel=0" width="420"></iframe>
<br />
<br />
The video along with the full subtitle support (so as to help <a href="http://en.wikipedia.org/wiki/Accessibility" target="_blank">a11y</a>) was made <a href="http://tharkuri.in/" target="_blank">Tharkuri</a>. A big thanks for this strenuous job. If you are looking for someone to do [online] marketing / professional writing work in India, I highly recommend <a href="http://tharkuri.in/" target="_blank">Tharkuri</a>.<br />
<br />
You can get the<a href="https://chrome.google.com/webstore/detail/find-many-strings/phabnclfbnmhmoipebjdfhkiacojlcnc" target="_blank"> extension from the chrome store</a> and the sources from the github repository. Please report any bugs / features to the <a href="https://github.com/psankar/Find-Many-Strings" target="_blank">github</a> page and any feedback in the extension page or by mail.</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com2tag:blogger.com,1999:blog-28654204.post-66181698724672656372013-08-16T11:35:00.000+05:302013-08-16T11:35:06.007+05:30Introducing simplefs - A ridiculously simple filesystem<div dir="ltr" style="text-align: left;" trbidi="on">
In the last few days I have been trying to implement a filesystem from the scratch. It has reached a level of maturity where I can release it to the public without feeling too embarrassed about the quality of the work. Here you go, <a href="https://github.com/psankar/simplefs" target="_blank">simplefs</a>. It is meant to be used as a tool for for teaching filesystem basics and not for production use.<br />
<br />
I recently released the version 1.0 of the filesystem with support for:<br />
<ul style="text-align: left;">
<li>Creation of files and nested directories</li>
<li>Enumerating files in a directory </li>
<li>Reading of files</li>
<li>Writing of files</li>
</ul>
In the next release, I plan to implement support for <a href="http://en.wikipedia.org/wiki/Extents" target="_blank">extents</a> and in the next release to that the support for<a href="http://en.wikipedia.org/wiki/Journaling_file_system" target="_blank"> journaling</a>. Your comments, feedback etc. are welcome.<br />
<br />
I wish I have implemented a filesystem from the scratch a long time ago, may be during my college days. This activity gives you a good test bed for evaluating almost all aspects of your computer science knowledge, like, Operating systems, Data structures, Algorithms, Locking semantics (granularity, ordering etc.), Cache coherency, Programming expertise, Cost hierarchy (read-time across memory, disk etc.) etc. If you are interested in being a programmer for a long time, you should definitely try implementing (or at least designing) a filesystem from the scratch. In a world (or the Indian IT sector may be) where a programmer's role is getting restricted to being a [javascript] library plumber, such designing + programming tasks will give you the <a href="http://en.wikipedia.org/wiki/Need_for_achievement" target="_blank">N-Ach satisfaction</a>.</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com6tag:blogger.com,1999:blog-28654204.post-50213753469017826212012-10-29T13:54:00.001+05:302012-10-29T13:54:41.256+05:30What I need from a WM/Desktop<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: left;">
Any suggestions are welcome for the WindowManager/Desktop needs that I have. I am open to trying out prototype systems too.<br /><br />My requirements are:</div>
<div style="text-align: left;">
</div>
<ol style="text-align: left;">
<li>win + left , win + right keys should align windows in the left and right halves of the screen, max-vertically, respectively<br /></li>
<li>win + top, win + bottom should align windows in the top and bottom halves of the screen, max-horizontally, respectively<br /></li>
<li>win + f or win + enter should fullscreenize the current window<br /></li>
<li>Should support 3x3 workspaces. My main development workspace will be in center. All other things like mail, browser, IRC, IM should be just one hop away in any of the neighboring four workspaces<br /></li>
<li>The currently focused window should be brighter than the rest of the windows in the background<br /></li>
<li>If I click on a window in the background, it should just get focused and not interpret as a click on the window (say if I click on a link in the browser in a bg window, it should not change the location but just bring the window to fg)<br /></li>
<li>There should be a tile option to arrange all windows in the current workspace, into quadrilaterals of equal width and height<br /></li>
<li>There should be an expose like option where I should be able to see all the open windows (may be triggered on win + f7 key or three-finger-scroll in the touchpad). This should just arrange the currently open windows<br /></li>
<li>There should be no always-present bar in any edge of the screen (unlike gnome3). Intelli-hide panel on the bottom is good (like gnome-do docky) etc.<br /></li>
<li>There should be no animations at all when the windows are resized/tiled/etc.<br /></li>
<li>There should be no animations at all when we switch workspaces.<br /></li>
<li>ALT + TAB should alternate between open windows (not applications) in the current workspace ONLY.<br /></li>
<li>The title bar of the windows should be as thin as possible, such that it will accommodate three buttons for minimize, maximize and close. But they should not be too big like in GNOME 3 wasting a lot of space. Does not matter if it is configurable/themable or not.<br /></li>
<li>Notifications should popup in some fixed location (say top-right corner) and stay until dismissed (I do not use notifications for individual chat message receiving etc. so this is good for me. I need to be notified only for a new chat and not messages in an already open chat window etc.)<br /></li>
<li>Should allow changing of control key position to either alt key (to help when working along with Mac) or capslock key<br /></li>
<li>Should have an option to have one workspace dedicated for an additional monitor and a keyboard shortcut to switch to that workspace<br /></li>
<li>Things like multimedia keys, password store etc. are not exactly a big need. If they exist, it is good but if they don't they are not deal breakers. My main requirements are for the WM aspects and not really these features.</li>
</ol>
<br />Are there any other WM needs that should be added to the above list to make my programming environment better ?<br /><div>
<br /></div>
<div>
Are there any recommendations or does anyone have a configured setup (for xmonad, pekwm etc.) that is somewhat closer to the above requirements ?<br /><br />Please share your comments/helps/config-files. Thanks.<div style="text-align: left;">
</div>
</div>
</div>
Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com13tag:blogger.com,1999:blog-28654204.post-73302723585958767892012-07-20T16:19:00.000+05:302012-07-20T16:19:46.251+05:30Offended<div dir="ltr" style="text-align: left;" trbidi="on">
As some of you might know, I am a <a href="http://en.wikipedia.org/wiki/Veganism" target="_blank">Vegan</a> and I saw the error codes <a href="http://en.wikipedia.org/wiki/0xDEADBEEF#Magic_debug_values">http://en.wikipedia.org/wiki/0xDEADBEEF#Magic_debug_values</a> and was shocked to see code like 0xDEADBEEF<div>
<br /></div>
<div>
I am offended by this. Cow is a holy god in our Hindu/Jain religions and killing of it is banned in our faith. Such constants make the code more offensive to a nation of a billion people (India). So, I am going to ask everyone to remove such constants.</div>
<div>
<br /></div>
<div>
Also, I noticed that Christopher Blizaard's website is<a href="http://www.blogger.com/%C2%A0http://www.0xdeadbeef.com/" target="_blank"> http://www.0xdeadbeef.com/</a> This is indeed offensive and I am going to ask him to change his website. His blog's feed is in many planets and I don't want to see that word, when I hover over a link.</div>
<div>
<br /></div>
<div>
Also, I should ask to get the planet Uranus renamed.</div>
<div>
<br /></div>
<div>
Mentioning beef may sound a simple thing to you. But such things makes Vegans feel insecure wondering if the whole development activity is for meat-eaters only.</div>
<div>
<br /></div>
<div>
#satire #reductioadabsurdum?</div>
<br />---<br /><br />Will I write such a code ? <div>
No<br /></div>
<div>
Is it in good taste ? </div>
<div>
Probably Not, for the majority of people.<br /></div>
<div>
Will I get offended on seeing such a code ? </div>
<div>
If I do, I will have to lose my sleep over everything.<br /><br />Calling "[<b>Microsoft] managed to make the kernel more offensive to half the population</b>" is too much of an over-reaction, to, what is just a lame joke of a programmer trying to be funny. There is no need to drag Microsoft's name here, for the same reason, why we don't accuse The Linux Foundation of propagating male supremacy ideas, when Linus Torvalds says "Do you pine for the days when men were men and wrote their own device drivers?"</div>
<div>
<br /></div>
<div>
Inspiration for the post: <a href="http://mjg59.dreamwidth.org/14955.html" target="_blank">http://mjg59.dreamwidth.org/14955.html </a> </div>
<div>
<br /></div>
<div>
I stumbled onto the comments section and found that someone has posted the exact same thing as this blogpost, that I had in mind. World is in indeed small. Sadly no idea who made <a href="http://mjg59.dreamwidth.org/14955.html?thread=538219#cmt538219" target="_blank">the comment</a>.<br /><br />Disclaimer: All opinions expressed are purely personal and do not represent my employer.</div>
</div>Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com22tag:blogger.com,1999:blog-28654204.post-81535936414316871392011-11-10T16:50:00.001+05:302011-11-11T00:21:31.312+05:30Smartphones - Acquired Necessity<div dir="ltr" style="text-align: left;" trbidi="on">
<br />
<b>Smartphones - An acquired necessity</b><br />
<br />
I have used a Motorola E398 mobile phone for the last 6 years. For the last one month, I tried using a Samsung Galaxy S2 smartphone. I have concluded that Smartphones are an acquired necessity and is not needed for most of my workflows. I will not buy a smartphone in the near foreseeable future.<br />
<br />
Smartphones gives an impeccable improvement for one workflow - Email. If your work involves time-sensitive emailing, a smartphone is a crucial tool. However, if it doesn't, then you are better off buying a good desktop/laptop and a normal mobile phone, imho. A shortlog of things observed in this one month are:<br />
<a name='more'></a><br />
<span style="background-color: transparent; font-size: large;">Observations</span><span style="background-color: transparent;">:</span><br />
<ul style="text-align: left;">
<li><span style="background-color: transparent;"><b>Vibration</b>: Smartphones are too thin and don't have enough vibration, if you are used to thick phones (Not a big problem)</span></li>
<li><span style="background-color: transparent;"><b>Poor signal reception</b>: A biiiig problem. In the quest to slim the phone, the signal reception abilities of the phone are heavily hampered. While we were travelling in a car (from Bangalore to Chennai), A cheap Samsung GURU E1081 consistently beat Galaxy S2 in terms of signal strength. Most of the times the Galaxy S2 was showing "<i>Emergency calls only</i>" only. May be a phone from a real phone-maker like Motorola/Nokia may not have this problem. We had to keep the phone upright near the car window too.</span></li>
<li><span style="background-color: transparent;"><b>Typing</b>: Even with the on-screen-keyboard, swype etc. the phone is totally unsuitable for typing long text. We can do only twitter/facebook updates and not do any serious document editing or long blogpost(s). The requirement for Siri (Voice Control) is just a natural demand. I wonder how the folks at Google missed this big requirement. They should have introduced this with a big bang and not play catch-up to Siri. Probably they missed it as they were busy <a href="https://plus.google.com/101066983930103920498/posts/2APx7Xus463">tomato-saucing</a> Google+ to all their applications ;-)</span></li>
<li><span style="background-color: transparent;"><b>Screen Estate</b>: The screen size is totally unsuitable for reading books, blogs. It is okay for occasional blog reading, but it is straining. There are people who read blogs using their phone primarily. But I am spoiled by my employer who gave me an iPad, a Samsung GalaxyTab and a Thinkpad to play with, for some mobile related coding. I did not prefer the smartphone even once when I had my good old Thinkpad. Tablets and laptops provide far better reading experience and are equally handy. The GalaxyTab can act as a phone too for all your needs.</span></li>
<li><span style="background-color: transparent;"><b>Fragile</b>: Smartphones need extreme care. Not ideal for rough use, unless you are using a rugged phone like Motorola Defy+ (which has its own set of problems). A friend once dropped a Google phone from his hands while taking from pant pocket (~3 Feet) and the glass shattered into pieces. I have thrown my E398 numerous times (atleast few dozen times from ~5.6 feet) and nothing has happened except an automatic restart.</span></li>
<li><span style="background-color: transparent;"><b>A Patentable Idea</b>: The unlock pattern (of Android) leaves fingerprints and so your phone is not really secure. If you look at the phone, by keeping the screen flat in front of your nose, you can easily detect the unlock pattern finger prints. This may be fixed soon with the advent of touchscreens that do not leave a finger print. I wonder why can't we just authenticate based on the fingerprint in a smartphone !? That may be cool. I should probably patent it, as it seems in mobile phones any stupid thing can be patented. Think: SYSTEM and METHOD for RECORDING and AUTHENTICATION of users to mobile phones via fingerprints, by letting them swipe on either the surface/camera/etc.</span></li>
<li><span style="background-color: transparent;"><b>Battery Life</b>: Even after switching off the wireless, due to the rich display, the battery life of all the smartphones is very less. My phone battery did not last more than 3 days. With wireless on and just GMail app running, the battery lasted for just about 1 day. Some of the non-smart phones these days have close to 2 weeks battery life. If battery life is your criteria or travel a lot, you must order a extra battery if you are buying a smartphone.</span></li>
<li><b>Muscle Memory</b>: After laying in my bed for sleep, many times, I have taken my motorola phone, unlocked it, launch the alarm application, set alarm for a specific time, exited the application, locked the phone again. I do all this while keeping my eyes closed. The normal phones with keypad are easily operable in one hand. I can take a call in one hand, while the other hand is balancing my body in a bus moving through the high-traffic streets of India. Contrary to what you hear, smartphones require both hands to be used and is not so easily adaptive for muscle memory. For a basic operation like calling a recently called number, you will take more time in a smartphone than a phone with a dedicated CALL button.</li>
<li><span style="background-color: transparent;"><b>COST</b></span><span style="background-color: transparent;"><span style="background-color: transparent;">: The single biggest reason why I won't buy a smartphone is Cost. Smartphones from any decent hardware maker are very costly. Personally, <a href="http://www.youtube.com/watch?v=8anHLEy-vmg" target="_blank">with my Indian mentality that takes pride in being cheap</a>, I find it stupid to spend 30,000 INR (600 USD) on a phone which will be valued at 1,000 INR (20 USD) after may be 3 years. This inference is based on the Motorola Razor handset pricing in India. Compare this to a new Samsung GURU phone with color-display, USB charging capacity etc. that costs a mere 1000 INR (20 USD) and has battery life of about 1 week with normal usage. However, there is a big demand for cheap smartphones in markets like India. </span></span>If and when Nokia releases their cheap smartphones in India, they are sure to repeat their success story in India, just like they did with their Torchlight series phones.</li>
</ul>
<br />
<b>Samsung-specific-observations</b>:<br />
<ul>
<li><span style="background-color: transparent;">The default alarm application does not have snooze option. There is no excuse for this.</span></li>
<li><span style="background-color: transparent;">Samsung Kies - Ahem :/<br />Also, it is not available on Linux. Good news is that with the recent versions, the software update can be done within the phone itself without Kies.</span></li>
<li><span style="background-color: transparent;">The indic support patch is not upstreamed yet. But kudos to Samsung as they are the only Android handset makers who support Indic fonts native as of today, afaik. A few of my Tamil friends bought Samsung phones just for this reason.</span></li>
<li><span style="background-color: transparent;">OLED - Amazingly rich screen, especially while displaying black color.</span></li>
<li>The front camera in S2 is just a joke. Totally useless for my needs.</li>
</ul>
<br />
<b>Android</b>:<br />
<br />
<ul>
<li><span style="background-color: transparent;">"Android" brand has a better image than "Linux" in the consumer market. It is not without reasons. There are a lot of positive things about Android. I have not mentioned any of them because you can find them easily.</span></li>
<li><span style="background-color: transparent;">However, To be honest, the usability of Android phones is *not* jaw-droppingly-awesome, imho. They are just as normal (good ?) as say Meego UI or GNOME 3. But consumers love them. If enough money is spent on marketing, pigs really can fly. I wish some of the earlier projects like openmoko/maemo/meego had rich companies that were as committed as Google is to Android.</span></li>
<li><span style="background-color: transparent;">I hope ChromeOS opens a door for Linux on Desktops, just as how Android made Linux the most dominant operating system on mobiles. That may help PC OEM vendors to think a little instead of their current act of blindly worshipping Microsoft.</span></li>
<li><span style="background-color: transparent;">The biggest positive impact of Android imho is: Android made companies which usually don't bother about Linux users (like Evernote) to write applications for Linux.</span></li>
</ul>
<div>
<b>Conclusion</b>:</div>
<div>
I will happily use my Motorola E398 until it lasts and then will buy a normal non-smart mobile phone when it no longer can run. Even though I may take up a job with a mobile phone company, I don't think I will buy a smartphone for my needs.</div>
</div>Sankarhttp://www.blogger.com/profile/13337547916077401549noreply@blogger.com17