Gluster Tiering : CTR/Libgfdb present & future

In the previous post (or re-post) we discussed about the operations of Gluster Tiering feature. In this post we will discuss how we capture the heat(special metadata) of the files/data-objects in Gluster, so that data maintenance activities (which are expensive on CPU, Storage and Network) like tier-migrations(promotion-demotions) can be done intelligently. We will also discuss on the possible improvements that can be done on the existing system so that heat can be captured efficiently and accurately. But before diving into this, lets we will see existing system.

The current architecture of Gluster tiering is as follows. We have a client side translator called tier-xlator. (For more details on Gluster Translator or Xlator click here).  This translator is responsible for multiplexing the IO into HOT or COLD Tier. And an thin application over this translator, we have the tier migration logic which functions as the Gluster Tier migrator, which does promotion and demotions.(NOTE: In Gluster, various data maintenance daemons, like DHT Rebalancer,  AFR Self-heal,  Tiering Migrator are modified clients, which are loaded with the necessary  Xlator Graph dept). The Tiering Migrator, which runs on each file serving node queries each Gluster Local Brick heat store via libgfdb and does promotion and demotion of files appropriately. Each File Serving Process(Gluster Brick) has a new translator called Change Time Recorder(CTR) that intercepts the IO and records the required extra metadata (in our case heat) on to the heat store of the respective brick, via libgfdb.

Screenshot from 2016-03-29 07-36-40


Now we are not going to discuss the about the tier xlator or tier migrator in detail in this post. Rather we will discuss about the CTR Xlator, libgfdb and the Heat-Data-store.

When we look at the case of Data tiering, which is a kind of Data Maintenance task, thus  as a toll on CPU, Storage and Network performance, should be done intelligently. Precision on selecting the right candidate for migration (promotion/demotion), makes data tiering more  effective and efficient, as you will not be migrating undeserving candidates (files) to the wrong tier. Also data migration is activity which is done conservatively, i.e either less frequently with large numbers/bytes of files or frequently with small number/bytes of files, for very same reason stated above.

Also while capturing heat of the file we need to take advantage of the distributed nature of the file system(Gluster) so that we can have no single point of failure and proper load balancing. We don’t want the admin or user of the file system to be aware of such a heat-data-store, so that there is no maintenance (disaster recovery, service management etc) overhead.

So taking these factors (and some more) in consideration, the requirements for the heat store would be as follows,

  • Should be well distributed : No single point of failure and has load balancing
  • No separate process/service for the Admin to manage
  • Should store various heat-metadata of the files i.e more versatility on heat configurations
  • Should provide good querying capabilities to have precise query results
  • Should record the heat-metadata quickly without a performance hit on IO.

When we explored for such a store, the one which fitted the bill was SQLITE3, as

  • Its a library and not a database service, less hassles for Admin
  • Its a library so can be embedded with Gluster Bricks, which provides for distribution
  • Has good querying capabilities with SQL
  • Gives precise query results
  • Has decent Write/Record optimization
  • Can store various kind of file-heat-meta.

Even though SQLITE3 looked attractive we wanted flexibility to load and try other data stores, so we introduced a library called libgfdb. The main purpose of libgfdb is to provide a layer of abstraction to the user, so that the library user is agnostic to the data store used below and use standard library API to insert, update and query heat-meta-data of the files from the data store.

Therefore the overall architecture of the solution looks like this. We have a new translator called Change Time Recorder(CTR) that is introduced in brick stack. The sole purpose of the translator is to do inserts/updates to the heat-data-store i.e sqlite3 db via libgfdb. The tier migration daemon that runs on every file serving node queries the lock brick’s heat-data-store via libgfdb and does migrations accordingly.

Screenshot from 2016-06-13 09-43-15

Now lets looks at the write performance optimizations that sqlite3 provides

PRAGMA page_size: Align page size to the OS page size
PRAGMA cache_size: Increased cache size
PRAGMA journal_mode: Change to Write ahead logging.
PRAGMA wal_autocheckpoint : Less often autocheck
PRAGMA synchronous : Set to NONE i.e no synchronous writes
PRAGMA auto_vacuum : Set to NONE (Garbage collection, this needs to be done periodically)

For more information on sqlite3 settings click here.

This is how the insert/update go into the sqlite3 database.

Screenshot from 2016-06-13 09-45-46.png

By looking at the setting you may wonder if we should worrying about crash consistency ? Actually no, because we have introduced a CTR Lookup heal feature which heal the heat-data store while lookups and IO. Thus the heat-data-store will be eventually consistent. And for the tiering case eventual consistency is just fine!

Even though we have optimized our sqlite3,which is our heat-data-store, we see performance issues due to the following

  • Indexing in WAL/DB : Even though the write to the sqlite3 goes in sequential manner, there is consistency check of index/primary keys. So for a large database (even though this database just has a part of the files in the whole Gluster Namspace), the indexing takes a toll on performance.
  • Synchronous recording of heat : The heat of the file is recorded synchronous to the IO. Slowing down the IO performance of bricks, due to latency in heat-data-store. This should be abstracted out.
  • SQL Creates are expensive : Creates in sqlite3 are transactions i.e create needs to be ACID in nature. This takes a toll on the performance. Huge number of creates show better performance when they are batched together in transaction. Thus comes the need for batch processing of creates, which is absent today.
  • Read-Modify-Update Expensive: Due to the WAL (Write Ahead Logging) read performance of sqlite3 is not great. This is fine for the queries which are executed periodically, as the time taken to query is relatively much much lesser than the time taken to migrate files(or Time taken for data maintenance on the query result is relatively huge compared to the query time). But this has a very bad impact when we have Read-Modify-Updates while capturing heat for example number of writes/IOps on a file in a specified period of time. Such updates should be made less often i.e  instead of 10 incremental updates we can consolidate it to one as +10.

All the above rises a need for a caching solution in libgfdb which has the following,

  • Removes the indexing part of the data store from the IO heat recording path
  • Batch process inserts/updates
  • Consolidate incremental updates.
  • Asynchronously record heat

The solution would be IMeTaL i.e In Memory Transaction Logs.

In Memory Transaction Logs, or simply imetal, is an in-memory circular log buffer. The idea is to have the inserts/updates in imetal and when logical partitions of imetal called segments are filled up, batch process the inserts/updates asynchronously/parallel to the incoming inserts/updates to other segments and write to the database. The only lock contention that the IO threads will have is the fine grain lock of where to write in imetal, rather than the whole index of the data-store. In this way we are able to achieve the above requirements. Here is how imetal looks like.


Now lets look the detail design of imetal, by understanding the components of imetal,

  • Log Record : Each record that is inserted in imetal
  • Segment:  Segments is logical partitioning of the log. Each segment will hold a fixed amount of Log Records.
  • Locator: Locator is a consultant that will tell the  IO thread where to insert the Log Record  in the imetal or precisely speaking which location in which segment.
  • Flush Queue : Full or Expired Segments go to the flush queue.
  • Worker Thread Pool: This is a thread pool of worker threads that works on the segments that are places in the flush queue.
  •  Flush Function/Data-Structure: Flush function is the function that is called when a segment, which is nothing but a batch of Log Records needs to be flushed to the database.
  • Flush Data-Structure : Before the flush we need to sort the Log Records for redundant entries and have only one entry per GFID.  For this we need a data structure like balanced binary search tree – red-black tree (insertion and search of O(log(n))), that would help us sort the Log records. Once the Log records are sorted in the data structure without duplication, we traverse the flush data-structure and start flushing the Log records into the database.

Now imetal is not specific to libgfdb, it can be designed/implemented in a generic way, i.e consumers of imetal should be given the liberty to define their own log record, flush function and flush data structure, thus imetal will only providing the infra for batch processing. Also imetal should be implemented at libgfdb, above the different data-store plugins, in this way no matter what the data store, users can benefit from imetal.

Configurations in imetal could be,

  • Size of segment  : How many log records in a segments
  • Size of the imetal : How many segments in imetal
  • Thread pool size : How many worker threads
  • On-Demand Flush  : A mechanism to flush segments even though they are not full.

Potential issues with imetal could be,

  • Loss of inserts/updates : If the imetal is full then the inserts/updates are lost. This may lead to loss of precision. But with the help of CTR lookup heal we heal the heat-data-store.
  • Freshness of the query: Since we are not inserting/updating the datastore directly, we need to flush all segments on demand, before the query is fired.

imetal is not yet implemented and this could be a good place, for someone from the community, to pick this up as a mini project 😉

There are few other performance/scale issues that sqlite3 has.

  • Sqlite3 doesn’t scale well : Since sqlite3 stores data in a single file and thus it doesn’t scale well when we have huge number of records. The obvious solution would be shradding of the databases i.e having multiple sqite3 database files for a single brick, and do distribution based on GFID
  • Garbage collection: We have turn auto garbage collection off, to improve performance, but this can backfire when the database grows as there will be a lot of sparse spaces in the databases. Period Garbage collection is needed.

Well to summarize things, tiering is a very tricky feature and it takes time to perfect and fine tune it. With CTR/Libgfdb we have tried to give good amount of tunables and plugins, so that new things can be tried and we get a optimal solution.

Signing off for now! 🙂

Deduplication Part 1: Rabin Karp for Variable Chunking

This is first post in a series of blog posts where we will discuss deduplication in a distributed data system. Here we will discuss different types of deduplication and various approaches taken in a distributed data systems to get optimum performance and storage efficiency.

But first basics of deduplication! 🙂

“Data deduplication (often called “intelligent compression” or “single-instance storage”) is a method of reducing storage needs by eliminating redundant data. Only one unique instance of the data is actually retained on storage media, such as disk or tape” definition by

Deduplication goes good when we can break the data stream into smaller pieces called “Chunks”.
The process of breaking the data stream into chunks is called “chucking”.

Static or Fixed Block Chunking
The most naive and easiest way of chunking is breaking the data stream into fixed length-ed chunks say for example 256 bytes per chunk.


Image courtesy:  moinakg

But the problem with this way of chunking is its not immune to the data displacement/shifting issue when an insert happens i.e when there is new data inserted in already chunked data stream there will be displacement/shifting of data.
This happens because the chunk boundaries are fixed, thus changing the content of every chunk after the insert as shown in the diagram. Thus all the data chunks after the insert change and thus considered new, even though they are deduplicates.

Variable or Dynamic chunking

The solution to this problem is chunking based on the content of the data stream. Well to do this we would require a way to identify a pattern on which we can chunk. The chunk boundaries will be guarded by pattern. This will have an implication on chunk size, i.e the chunk size will be variable in nature. And hence its called “Variable Chunking”. As you can see this scheme of chunking is immune from the data displacement/shifting issue because the chunk boundaries are guarded by pattern.


Image Courtesy : moinakg

To find the chunking pattern we have to use a [rolling hash]. The idea is to define a data window on the data stream, which has finite and predefined size of say “N”. Now using a hashing technique calculate a hash on the window. Check if the hash matches a predefined “finger_print”.

  • If the “finger_print” does match the calculated rolling hash on the window, mark the Nth element as the chuck boundary.
  • If the “finger_print” doesn’t match the calculated rolling hash on the window, slide the window by one element and recalculate the rolling hash. Repeat this until you find a match.

To search for the chunk pattern the Rabin-Karp Rolling Hash is used. If n is the window size then for a window from i to i + (N-1) the hash would be calculated using the following polynomial

H (i, i+n-1) = (P^(n-1))*a[i] + (P^(n-2))*a[i+1] +...+P a[i+n] + a[i+n-1]

where a[] is a data window of size n , starting from i to i+n-1,
P is a prime number (for better hash results).

The challenge to use this polynomial in a computer system is the value of the hash will overflow even an 64bit unsigned integer.Therefore to solve this we use a modulo
m on H i.e H%m and call it hash and then compare it with the finger_print

Now to calculate the **hash** we take the help of [modulo arithmetic]

  ( a \* b ) %m = ( (a % m ) \* b ) % m = ( (a % m ) \* (b % m) ) % m 
  ( a + b ) %m = ( (a % m ) + b ) % m = ( (a % m ) + (b % m) ) % m

So that we can calculate the **hash** by building a [recurrence relation].

Now when n=1 i,e the window size is 1,

H(n=1) = ( P^0 ) * a[i] = a[i]
hash(n=1) = H(n=1) %m = a[i] % m --------------> (1)

when n=2 i,e the window size is 2,

H(n=2) = ( P^1 ) * a[i] + a[i+1] = P*a[i] + a[i+1]
hash(n=2) = H(n=2) %m = {  P*a[i] + a[i+1]  } % m --------------> (2)
                      = { P * ( a[i]%m ) + a[i+1] } % m //using modulo arithmetic
                      = { P * hash(n=1) + a[i+1] } % m // from (1)

when n=3 i,e the window size is 3,

H(n=3) = (P^2)*a[i] + P*a[i+1] + a[i+2]
hash(n=3) = H(n=3) %m = { (P^2)*a[i] + P*a[i+1] + a[i+2]  } % m
                      = { P * { ( P*a[i] + a[i+1] ) %m } + a[i+2] } % m //using modulo arithmetic
                      = { P * hash(n=2) + a[i+2] } % m //  from (2)

As we can see we can have a [recurrence relation] between hash(n=k) and hash(n=k-1)

hash(n=k) = { P * hash(n=k-1) + a[k-1] }

Similarly we can have a [recurrence relation] between (P^k) % m and (P ^ {k-1}) %m.
Lets  POWER(n=k) = (P^k)%m

POWER(n=k) = (P^k) %m = ( (P^ {k-1} ) * P ) % m
POWER(n=k) = ( POWER(n={k-1}) * P ) % m

The above will be used later, just note it for now.

Using the above recurrence relation for hash(n=k) we can calculate the first chunking window hash

To have high performing chunking algorithm we cannot repeat the above for every subsequent window. We can to use the previous chunking windows hash and calculate the next chucking windows hash, as the only change during a window slide is of 2 elements i.e an outgoing element a[i] and an incoming element a[1+n]

Let have a re-look at out polynomial

H (i, i+n-1) = (P^(n-1))*a[i] + (P^(n-2))*a[i+1] + ...+P a[i+n] + a[i+n-1]

For the subsequent window the polynomial would be

H (i+1, i+n) = (P^(N-1))*a[i+1] + (P^(N-2))*a[i+2] +...+P*a[i+n-1] + a[i+n]  --------------- (3)

Multiply H((i, i+n-1)) by P

H (i, i+n-1) * P = ( (P^(n-1))*a[i] + (P^(n-2))*a[i+1] +...+P a[i+n] + a[i+n-1] ) * P
                = (P^(n))*a[i] + (P^(n-1))*a[i+1] +...+ (P^2 )a[i+n] + P*a[i+n-1] 

Subtract (P^n)*a[i] & Adding  a[i+n] on both the sides of the equation
                                = H (i+1, i+n) // from (3)

H (i, i+n-1) * P - (P^N)*a[i] + a[i+n] =  { (P^(n-1))*a[i+1] +...+ (P^2 )a[i+n] + P*a[i+n-1] } + a[i+n]
H (i+1, i+n)  = H (i, i+n-1) * P - (P^n)*a[i] + a[i+n]

Applying % m on the above equation

H (i+1, i+n) % m = { H (i, i+n-1) * P - (P^n)*a[i] + a[i+n] } % m
hash(i+1, i+n) = { H (i, i+n-1) * P - (P^n)*a[i] + a[i+n] } % m
               = { (H (i, i+n-1)%m) * P - (P^n % m)*a[i] + a[i+n] } % m //using modulo arithmetic


hash(i+1, i+n) =  = { hash(i,i+n-1) * P - POWER(n)*a[i] + a[i+n] } % m ------------ (4)

Now using (4) we have a recurrence relation using which we can calculate the hash of the next chunk window using previous chunk window hash ( hash(i,i+n-1)), outgoing element (a[i]) and incoming element (a[i+n]) i.e

hash(next) =  = { hash(prev) * P – POWER(n)outgoing + incoming } % m


Related work:

Automated Tiering in Gluster

Screenshot from 2016-03-29 07-36-40

NOTE : This is a re-post of

This post describes how to run automated tiering in Gluster. Tiering is appropriate for stable workloads where frequently used data fits on small, fast storage, such as SSDs, and rarely used data resides on a slower/cheaper volume, such as spinning disks.

On a tiered volume, files are tracked according to frequency of access. Popular files tend to migrate to faster storage, and unpopular ones to slower storage. The behavior can be influenced with tuning parameters.

Basic Operation

To use tiering, take an existing volume and attach a hot tier to it. The existing volume becomes the cold tier. The existing volume may be either erasure coded or distributed-replicated. The hot tier must be distributed-replicated. For example:

gluster volume tier vol1 attach gprfs01:/brick1 gprfs02:/brick2 gprfs03:/brick3  \

Once the tier is attached, there may be a delay before migration begins. A full scan of the cold tier is undergone. This delay shall be removed in the near term.

Promotion stands for file migration from the cold to hot tier. Demotion stands for migration in the opposite direction.

When a file is migrated a counter is bumped. The counters may be viewed:

gluster volume tier vol1 status

You can stop tiering to use the hot bricks for some other purpose. To stop tiering, use the detach operation. The steps resemble removing bricks. You initiate the process, then wait for it to complete by monitoring its status. This may take time depending on how much data must be moved off the hot tier. Once completed, the commit command may be used to remove the hot tier. The cold tier then reverts to the original volume.

gluster volume tier vol1 detach start
gluster volume tier vol1 detach status
gluster volume tier vol1 detach commit


Hot storage is valuable and should be utilized, else the resource is wasted. To this end, the tiering feature aggressively promotes files to the hot tier until it nears full. That point is governed by the “cluster.watermark-low” tunable and is expressed as a percentage.

Conversely, the hot tier cannot become completely full. If too much data resides on the hot tier, files are aggressively demoted. This value is governed by “cluster.watermark-hi”.

The system shall attempt to stabilize such that the amount of data on the hot tier is between the lower and upper watermarks.

gluster volume set vol cluster.watermark-hi 90
gluster volume set vol cluster.watermark-low 75

The tiering daemon migrates files periodically. The period for promoting files is “cluster.tier-promote-frequency”. Its default value was chosen such that files would be promoted quickly, in reaction to I/O. The period for demoting files is “cluster.tier-demote-frequency”. Its default value was chosen such that files are demoted slowly in the background. These values are expressed in seconds.

gluster volume set vol cluster.tier-promote-frequency 120
gluster volume set vol cluster.tier-demote-frequency 3600

It is possible to limit how much data may be migrated within a period. The limit may be expressed in # of files or in MB.

gluster volume set vol cluster.tier-max-mb 4000
gluster volume set vol cluster.tier-max-files 10000

By default, files are queued to be promoted if they are  accessed on the cold tier within a period. This behavior can be changed such that files are promoted if they are accessed more than some threshold within a period. The threshold may be expressed in terms of reads or writes. This would avoid populating the hot tier with files that are only accessed once. The hot tier should store files which are repeatedly accessed.

gluster volume set vol cluster.write-freq-threshold 2
gluster volume set vol 2


As of 3/16, measurements have tested cases where ~95% of the I/Os are to files on the hot tier. Those experiments have shown good performance when the cold tier is distributed-replicated. When the cold tier is erasure coded, the features works well for larger file sizes (greater than 512K) for a typical SSD.

Performance should improve as the code matures, and your milage my vary. A subsequent post shall explore performance.

NOS Conf 2016

rsz_1rsz_screenshot_from_2016-01-21_23-54-37(1)Nitte Open Source Conference 2016 is an effort to spread open source awareness in engineering colleges in and around Udupi & Mangalore, India and to inspire students to contribute to the open source movement. To give a background of Udupi & Mangalore, also known as “Dakshina Kannada” together until 1997, is a region known for its well known engineering colleges and other education institution. Some of the most popular engineering colleges in this area are,

These set of colleges attracts engineering talent from all over India and also overseas. This was a strong motivational factor, that encouraged the Gluster Community in Bangalore to reach out to NMAMIT, Nitte in April 2015 and conduct one day Gluster Conference/Workshop @ NMAMIT, Nitte. Four prominent colleges of the region had participated in this conference/workshop. Observing a great deal of interest last year NMAMIT, Nitte approached the Gluster Community in Bangalore this year for an Open Source Conference covering more topics, so that more number of speakers could present. This conference was intended to give the students an opportunity to learn from and interact with members of different open source communities. It was also a platform for the students to share new innovative ideas/concepts with community members.

NOS Conf 2016 was on March 18th – 19th 2016, @NMAMIT, Nitte. 6 neighboring engineering colleges  were invited for the conference. Over 200 students from these colleges and the hosting colleges attended the conference. Speakers from Gluster, Fedora/Centos and Mozilla communities were invited. 11 Lightening talk participants from these colleges had submitted talks of which 8 were selected.  The schedule of the conference is available @

March 18th, 2016 : NOS Conf 2016 was inaugurated by Dr. Niranjan N.Chiplunkar, Principal of NMAMIT, Nitte and Dr. K Mohan Hegde, Director, Dept of MCA, NMAMIT, Nitte.  Dr Niranjan N. Chiplunkar gave a brief history of NMAMIT, Nitte involvement in Open Source Movement and also mentioned about the Open Source Lab, that the college had setup, with Red Hat India Pvt Ltd, Bangalore, where students would work on open source projects.

The first talk of the day was by Niels de Vos, Red Hat about Network Security , “Network traffic inspections, fasten your security belt “. His spoke how easy it would be for a malicious program or hacker to sniff the network and do serious damage if there was no network security. He gave example on Internet of Things and pointed at vulnerable points in the network systems. Since there was a buzz word IOT as lot of question came in on this topic ! 🙂 And Niels was kept busy even after the talk answering questions from curious students.

The second talk of the day was by Saumya Hedge, Asst Prof, NITK, Surathkal, “Introduction to SDN”. She gave a insight on what Software Defined Network is and explained core concepts of SDN. Also she pointed out the different optimization that can be done so that we can get better performance when using SDN. She pointed out on some of advantages and disadvantages on SDN and how its still an evolving science. This was a good session of us,  as far as storage folks like us, to learn about SDN.

The third talk of the day was by Dan Lambright, Red Hat about Container, “Containers in Linux”. This was a good session, as Dan explained the internal of Containers, what actual an container is, what is the difference between a VM and Container, what is IAAS and PAAS, what is Docker, Kubernetes etc. He also pointed out the security issues in containers and explained Containers are not a replacement for hypervisor technologies, as the use cases are different. He gave an example how he used containers instead of actual VM, in his class back in USA, when he had a computer resource crunch.

In the afternoon, there were 4 lightening talks of which “Developing A Cost-effective Openflow Raspberry Pi Switch Testbed” stood out as a very good talk. It really impressed the judges (Niels de Vos, Soumya Deb) and also other speakers, that how the students have combined two technologies, Raspberry Pi and SDN, and created less expensive SDN Switch for testing purpose.

The fourth talk of the day was by myself, Joseph Elwin Fernandes, Red Hat, “Concurrent is NOT Parallel”. I tried to keep it simple by explaining what “Concurrency” and “Parallelism” is using real-life scenario. This talk was more of an introduction to the concept and support available in various system programming languages like C, C++, GO etc.

The fifth talk of the day was by Atin Mukherjee, Red Hat, “Ready to Go!”. This was an introduction to GO-Lang.  Atin explained the different aspects of GO, by using code snippets and examples. This was a very good talk, explaining how GO has made things simple for system programmers, as GO has a lot of inbuilt support for Concurrency and Parallelism, seamless inheritance model etc.

The last talk of the day was by Gaurav Garg, Red Hat, “Version Control Systems with git (and github) as an example”. Gaurav gave a good introduction to git explaining the different options in git that enables the programmer to go better version control. This talk could have been a workshop with students so that the students would have a hands on, on git.

March 19th, 2016: The day start with a surprise 🙂 One of the student, Deep Nirmal, 2nd  year CSE, NMAMIT, Nitte, approached organizers and requested that he would present a lighting talk on Open Stack! Though we had a tight schedule the organizers scheduled this talk at 9.00 AM, just before the first talk. Deep did a good job with his presentation, explaining the term “AAS-As A Servive” w.r.t Storage, Compute, Network, Platform, Infrastructure by giving an example of Pizza As A Service :).  And to our surprises did a demo of Open Stack, by  installing and then creating VM and doing provisioning. We were impressed!

The first talk of the day was by Kaleb S. KEITHLEY, Red Hat, “From Idea to Distribution — Distributing your Software for Linux”. This talk was more about what are the good practices a programmer should have and what are the tools that one should use to write good and quality code. Kaleb emphasized on coverity checks and unit testing. Kaleb also spoke about how you can use the different package mangers in different Linux Distros and how to use different channels to have your software available in different Linux Distros.

The above talk was followed by Hemanth Kumar, Asst Prof CSE, NMAMIT, NITTE, “Future of Graphics in Open-Source”. This was a good talk giving history of OpenGL and explaining how OpenGL has evolved over the years. The main thing about this talk was the new Vulkan generate and compute graphic APIs. Hemanth explained how Vulkan has removed the “driver magic” and hence increase the performance of the open graphic APIs. He also spoke about the different companies and organization that supported and contributed.

The Third talk of the day was by Rafi K, Red Hat, “RAFT : Distributed Consensus Protocol”. Rafi started the talk about explaining about consensus and then he asked the audience “Are you in consensus with me?” This was a cool trick to make the audience understand how important consensus is in a distributed system and how difficult is it to achieve it. He spoke in detailed about the RAFT algorithm : Leader Election, Log replication, Commit process etc.

I couldn’t attended rest of the talks and lightening talks in the afternoon, as I had to travel back home, but what I got from my colleagues was that the lighting talk “Multipathrouting” was good. Also heard that Soumya Deb(Red Hat)’s talk about “Zero to Millions, in one patch!” and Lalatendu Mohanty’s (Red Hat)’s talk about “Linux Distro: Fedora and CentOS” was fun. There was good audience interaction and questions for these talks.

More updates from Atin Mukherjee,

Day 2 post lunch session begun with lightening talks with 15 minutes
slot given to the students. First talk was on a chatting client server
application written in Golang by Vishwas & Manju, I was happy to see
them understanding how easy it is to write a client server application
in golang. Post that Shree Vidya took the stage to talk about
multipathrouting on SDN where packets can be routed over http/ftp
based on its priority and volume. We also had couple of more talks on
Ajex Web crawler in Python and Liquid FS which talks about using
deduplication in file system. Atin Mukherjee and Soumya were given the
chance to judge these talks and Multipath routing session scored the
highest marks.

Then we got back to the mainline track where Soumya Deb started
discussing on how someone can benefit millions of users by contributing
a single patch in open source projects. He gave some real good examples
about people who wrote some cool features which prove to be very handy
for those respective projects with out been a full time contributor to
that project.

Last but not the least we had Lalatendu Mohanty talking about different
Linux distribution and their life cycle and he explained the difference
between Fedora, RHEL and CentOS and defined "who should use what and
when" theory

Overall NOS Conf 2016 has created a wave of open source awareness in the students of the participating colleges. It also gave us an opportunity to  see what engineering talent these colleges have. It would be great if this conference happens the following year with new topics, new speakers and new ideas 🙂 Signing off for now!

“Open. Think. Innovate.”