The DF Lab

My senior design project for university has involved a lot of working in the Linux kernel. I’ve found that my primary difficulty with kernel work has been figuring out how the mass of code that already exists works and interacts with each other. It’s mostly very well thought out, but some things take a little time to wrap your head around. One such thing is the kernel’s unified implementation of linked lists.

It’s actually rather ingenious if you think about it, and there are plenty of references that give great explanations of how it all works, so I won’t do that here. The short version is that the list actually acts like an element within another data structure, which allows it to work for all types, rather than having to create new structures and functions for every structure to act like a list. What I recently found myself trying to do was create a queue (first-in-first-out) data structure using the linked lists provided. I decided to go about this after getting a little advice from a couple of fellas over at Stack Overflow. I haven’t fleshed it all out yet, but came to an important realization on how things work. It’s not really all that hard, but it was important enough for me to want to document: because it is a circular, doubly-linked structure, adding an item to the front and back of the list look very similar. In fact, they’re pretty much the same operation, except for where the external head and tail pointers point to. They’re so similar that I spent a bunch of time confused on how it all worked. I suppose that’s my fault for delving in and examining the overly-short elegant code.

The result of this is that their list_add_tail() function (which is commented to be useful for queues) is, in fact, useful for queues. It adds an element linked after the “last” element and before the “first,” given the head of the list. “Last” and “first” are in quotes here because it’s a circular list, so there technically aren’t a first and last, but there are. Anyway, what I still don’t quite understand is how the list_add() function is useful for stacks (also in the comments). Though I’m getting myself slightly confused just thinking about it further, so I’ll end the tirade here.

I guess if there’s one thing I can pass on here, it’s that if there’s comments documenting a series of functions (which the kernel’s include/linux/list.h file does), listen to the comments. I suppose this was a learning experience for me in the end, but I could have saved a bunch of time by just trusting what they said the functions do. On the flip side, if you’re curious, don’t trust them, and want to be sure about what the code is doing, you are always free to dive right in! I’m just convinced that the kernel developers are a lot smarter than I am, at least at this point in my career.

Related Links

• Linux Kernel Linked List Explained
• Another Kernel Linked List Explanation
• Stack Overflow question about kernel queues
• list.h on Linux Cross Reference

linux kernel, linux

A couple of months ago, I spent a lot of time working on a solution for handling all my music and its various uses. Namely, I wanted to keep separate track of:

• All my music (even the music I don’t listen to, so I don’t have to get rid of it)
• The music I listen to on my computer
• The music I have on my limited-space iPod

In addition, I wanted to make sure there was as little wasted disk space as possible, so I used hard links and had all the files in separate directories.

You’d think there would be a relatively simple solution, and I’d hope so too, but I came up with this really convoluted system that involved synchronizing three directories with two scripts whenever music had to be added or removed. What it results in is that I never add music anymore. And somewhere along the line, my “music on my computer” folder got messed up, so I’m back to square-one with that one. Hundreds of songs I filtered out back in the mix.

I wonder if some of my problems would be solved if I used a GUI music player, but I’m not so sure. My current setup uses MPD, which runs nice and light. Anyway, I’m out of mental energy, and am hoping someone else might have a better idea. Does anyone have any good ideas how to maintain my music? Or perhaps a better ideal to work under? Until then, I’ll just suffer under a horrible manual version where I just delete and add things at will to everything and hope I don’t miss things.

linux linux, music

As both an experienced user of Gentoo and a lover of the command line, I often find myself querying portage for various things. Sometimes, if it’s in a particular package, I go looking at the ebuild itself. Others, I run a find command in /usr/portage. However, there are tools to take care of all these silly things in a much more elegant fashion. Namely, there’s equery and q. Since I’ve only now realized that they both not only exist and are useful, I figure there must be someone else who could benefit from this knowledge as well. In this post, I’ll briefly describe the uses and benefits of each, as well as why they should co-exist on your system, at least for the time being.

Commonalities and Usage

First things first, equery and q are both very similar both in offered functionality and usage from the command line. Basic usage goes like so:

$ equery [command] [package]
or
$ q [command] [package]

Simple, right? There are also options to modify usage, but this is the most basic and common usage pattern.

In terms of functionality, equery and q have the following in common:

Function	equery command	q command
List packages owning file	equery b	q file
Verify integrity of package	equery k	q check
List dependencies of package	equery d	q depends
List files owned by package	equery f	q list
List all packages with USE flag	equery h	q use
List all packages matching search	equery l	q search
Show size of files in package	equery s	q size

The usage of each of the above commands is pretty straightforward, so I won’t bore you with details. Running just “equery” or “q” from the command line will show all basic usage, and running one of the above commands without arguments (or with a -h local argument, for equery) will show similar usage info for the specific command.

equery

equery is a part of the gentoolkit package, is written in Python, and is rather well-endowed (in terms of features, of course). In terms of features unique from q, equery boasts:

• depgraph: display a dependency tree for a given package
• uses: display USE flags for a given package
• which: print full path to ebuild for a given package

Personally, of these three, I’ve only ever used uses, since I the few times I’ve ever attempted to use depgraph, the results have been too big to really get a handle on. Either way, the uses command makes it a lot easier to find out what USE flags are available for a particular package as well as their current states. Of course, you could just do a (not so quick):

$ emerge -pv [package]

However, that won’t give the information on what the USE flags actually are, just what their status is on the package.

equery in general gives more detail and nicer output than its shorter-named counterpart. Since I always like to see screen shots (yes, even from command-line programs), I’ll take the liberty of doing just that to illustrate my point:

One thing to note about this particular command is that while qsearch automatically searches both installed packages and those in the portage tree, equery requires the -p option, as shown above, to do the same thing. On the flip side, qsearch has no capability (at this time) of searching overlays, but equery can be made to do so with the -o option. Tradeoffs, tradeoffs!

As a final note on equery, there are in fact two ways to call each of the sub-programs (like list, depgraph, etc.). There’s a short and long option for each of them, which is rather convenient. “equery l” is much nicer than “equery list” and “equery g” is way better than “equery depgraph.”

q

On to q! After reading the above section, you might wonder why anyone would want to use q when they’ve got equery in their toolbox. Is it for those few features that q has that are so dazzling? Is it because the name is shorter? No, no, let me just show you, you’ll understand:

For those of you who can’t see the images or are just in plain shock, let me spell it out: q is fast. In that particular query, about 34 times faster. 34 times faster! That makes a big difference, whether you’re sitting in front of the keyboard twiddling your thumbs or putting it in a shell script. As a matter of fact, on running just the q or equery commands alone (to show the helpful usage messages), the speed difference is over 500 times! That being said, if you don’t need the fancy formatting and extra frills of equery for a given task, just use q. It’s faster. According to its Gentoo page, that’s its purpose anyway:

portage-utils is a collection of very fast utilities written in C, which are meant to offer a faster but more limited alternative to their gentoolkit counterparts. Please note that portage-utils is not meant to replace gentoolkit. Its utilities are much more efficient than the equivalent ones from gentoolkit and might be better suited to be used in scripts that need to call Portage repeatedly, but portage-utils does not offer the same functionalities.

Well hot-damn, I could have told you that from the beginning, no? It’s times like this that I sing the praises of C and mock all those Python people. Then I try to write a difficult program and cry myself to sleep.

Anyway, language wars and tearful nights aside, there are also a couple of other distinguishing things about q. First, its simple format can make parsing a bit simpler. Then again, it could make it harder, so let’s not go there. As a matter of fact, equery makes a point of modifying its output if you redirect its output. If you don’t like the modified style of output, you can pass the -N (–no-pipe) flag to turn that behavior off.

Second, q does bring a few unique functions to the table. Namely:

• atom: split up an atom string (like games-emulation/zsnes-1.51-r2 -> games-emulation zsnes 1.51 r2)
• cache: search the metadata cache
• grep: grep in ebuilds
• lop: emerge log analyzer
• merge, pkg, tbz2, xpak: all pertain to actually handling various types of packages, which I have no experience with, so I don’t know their usage.

The one that I find particularly is “lop.” In an example from that Gentoo page on portage-utils, try something like:

$ qlop -tH openoffice

It’ll tell you “the merge time” for that package. Now, I’m not sure if that means the last merge or some sort of aggregate. My output tells me:

openoffice: 9 hours, 19 minutes, 11 seconds for 12 merges

I’m guessing that means that it took 9:19:11 to merge 12 packages, in terms of the package in question and its dependencies, but I’m not totally sure on that one. Either way, this is a damn nifty feature. I make jokes all the time about how long some packages take to emerge, and how I can have actual times to back me up! Oh the joys of Gentoo…

As another very useful note, the qsearch command is also substantially faster than “emerge –search.” It’s not nearly as impressive as it is against equery, but it holds its own. An advantage that qsearch has over its equery counterpart, however, is that it has the ability (and has it default) of displaying descriptions of packages. I actually for a long time forgot how to do that on my system, and always ran to gentoo-portage.com.

Last but not least, just like equery, q commands can be shortened as well. Unfortunately, that just means changing something like “q search” to “qsearch.” Not a big improvement, but with a one-letter command, how much can you really ask for?

Why both?

In brief, the snippet above from the Gentoo article on portage-utils gives the answer quite nicely. q may be lacking, but it can be an order of magnitude faster than equery. For those times when you don’t need all that fancy-shmancy formatting and just want to get quick and dirty results, q is your tool.

linux cli, gentoo, linux, portage