Sysadmin's Overlook

http://think-like-a-git.net/sections/about-this-site.html

The most useful description of how git works, and what you need to understand it that I’ve found thus far.  Gets into ‘git rebase’ without making you crazy.  Well worth the read.

Bash (at least, and probably all SH derived shells: sh, ksh, bash, zsh, etc) do something I didn’t expect:

echo "a b c d" | while read letter1 letter2 rest ; do
    cmd1 $letter1
done

cmd1 inherits the loops STDIN (the echo ‘a b c d’ bit)  Unless you explicitly override it with a </dev/null or </dev/tty or some such.  Now I’m wondering why the SH shells made that design choice.

–Jason

Sudo added the nifty function of having /etc/sudoers.d/<files> and read all of them in.  The ‘.d’ idiom is getting more and more popular , and I’m very happy about it.

However, sudo adds a bad wrinkle to this: from the man page, if the file contains a ‘.’ or ends in an ‘~’, then it won’t be read, and silently discarded… leading to ‘user not in /etc/sudoers file’ errors.

A better test would be to see if the file started with ‘.’, in which case they certainly should be discarded.  Other options include “force file names to end in ‘.sudo’” or something similar.

Particularly with the rise of ‘first.last’ style user names, this is a bad test.

If you have a netapp that’s out of sync with time, and you’re getting messages like

[cifs.trace.GSS: error]: AUTH: Unable to acquire filer credentials: (0x96c73a25) Filer and domain time differ by more than 5 minutes.

then here’s how you fix it.

Note: I don’t know why NTP failed for me this time, maybe netapp/freebsd don’t run an ‘ntpdate’ before starting ntp normally, and clock would skew back to normal ‘slowly’.  Unfortunately, I needed it to be fast instead.

Procedure:

options timed.enable off
rdate <valid ntp server>
options timed.enable on

Good luck

–Jason

So, you’re a sysadmin: and you know (from iostat, and whatnot) that you’ve got a drive that’s getting hammered.  Wouldn’t it be nice to know what process is actually hammering it?  There are things out there like iotop and what not, but they generally rely on new kernel features, or things that are rather more invasive.  Over the weekend, I needed something for an older kernel, and which wouldn’t be invasive.

Enter iohist (with a nice tip o’ the hat to iodump).  It relies on ‘echo 1 > /proc/sys/vm/block_dump’  to do it’s magic, looping over ‘dmesg -c’ at an interval (currently 5 seconds).

Check it out on github.

So, Fedora is playing with systemd / systemctl as a replacement for SysVInit.  I’m not normally the one to rant about ‘new, better, faster’ features… and this thing does have one big nice feature: parallel execution to speed up boot time.

However, there’s a huge downside:  the syntax for this is FUGLY.  Reading Fedora’s Cheat Sheet on the matter, there isn’t a single command that isn’t significantly longer/more complicated.  For example:

Old style:    service syslog restart

New Style:   systemctl restart syslog.service

Old Style:   chkconfig –list

New Style: systemctl list-unit-files –type=service

(the first example isn’t TOO bad, but the second is a pain)

Now I need to figure out if I’m turning into an old curmudgeon, or if I have a legit beef.

But sometimes you just have to use them because they’re the only <redacted> tools for the job.

Because of this, I have miles of spagetti code floating around that do expect, autoexpect, pexpect, pxssh, etc, etc, etc type things.  Every time I have to break one of these out, I cringe, because they are fragile, finicky, and in general a PITA to deal with.

I’ve been hearing about Paramiko for a while, but whenever I glanced at it, I never read the docs deeply enough to grok it…. which is VERY sad, because there’s very little to actually grok.  It just works, with one minor caveat: You need the magic dust of “set_missing_host_key_policy(paramiko.AutoAddPolicy()) or you’re in for lots of annoying (and misleading) ‘host not found’ errors (due to the mismatch between what cyphers PyCrypto supports, and your standard OpenSSH uses by default).

Example:

import paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect('127.0.0.1', username='jesse', password='lol')
stdin, stdout, stderr = ssh.exec_command('ls -l')
output = open('ls_output', 'w')
for line in stdout.readlines() :
    output.write(line)

You can find lots more information from the author here.  Or just google.

Mac is pretty, and very GUI, but getting real data about what’s going on can be a PITA.

Want to know how to list the luns and storage controllers that your Mac client can see?

fibreconfig -l

(and you can rescan with -r!).  Note the english spelling of fibre.

–Jason

aka ‘how long has it been since I ran fsck?’

Answer: run ‘tune2fs -l <device>’ and look at

Mount count: 2
Maximum mount count: 36
Last checked: Tue Sep 6 23:12:36 2011
Check interval: 15552000 (6 months)
Next check after: Sun Mar 4 23:12:36 2012

If the mount count = max mount count, you’re due for an FSCK.  If you’re booting after the ‘Check interval’, (which is helpfully calculated for you on the ‘Next check after:’ line), you’re due for an FSCK.

Knowing if your scheduled reboot will take several hours in order to run the FSCK?  Priceless.

(Not directly Sysadmin related, but important none the less. Here are letters I’m sending to my Senators, both of whom are co-sponsors of the PIPA or Protect IP Act)

Please use these as templates or replace <NAME> and <ADDRESS> with the appropriate values.

###########################################################################

The Honorable Saxby Chambliss
416 Russell Senate Office Building
United States Senate
Washington DC 20510

Dear Senator:

My name is <NAME> and my address is <ADDRESS>.  I am writing you because I am deeply disappointed that you are not only supporting the Protect IP Act (aka PIPA, aka S.968), but are actually a co-sponsor of the proposed legislation.

As written, PIPA would destroy the progress of the Internet over the past 15 years. You well know how important that growth has been to the country as a whole, and Atlanta/Columbus/Savannah in particular.  The entire ‘Web 2.0′ movement is founded on user generated content as well as enabling small entities (businesses and persons) to work together and build amazing things.  The provisions of PIPA would work to destroy this foundation by significantly decreasing due process for ‘take down’ requests, while drastically increasing the ‘damage radius’ of such requests (such as forcing Ad-networks, search engines, DNS servers, etc to ‘delist’ entire sites for having a single bit of infringing content.)

No one denies that intellectual property piracy is a real problem. There are possible legal remedies to these challenges, however PIPA is a very dangerous solution to these problems.

Please retract your support and sponsorship of the PIPA (S.968) immediately.

Thank you,

<NAME>

###########################################################################

The Honorable Johnny Isakson
131 Russell Senate Office Building
United States Senate
Washington DC 20510

Dear Senator:

My name is <NAME> and my address is <ADDRESS>.  I am writing you because I am deeply disappointed that you are not only supporting the Protect IP Act (aka PIPA, aka S.968), but are actually a co-sponsor of the proposed legislation.

As written, PIPA would destroy the progress of the Internet over the past 15 years. You well know how important that growth has been to the country as a whole, and Atlanta/Columbus/Savannah in particular.  The entire ‘Web 2.0′ movement is founded on user generated content as well as enabling small entities (businesses and persons) to work together and build amazing things.  The provisions of PIPA would work to destroy this foundation by significantly decreasing due process for ‘take down’ requests, while drastically increasing the ‘damage radius’ of such requests (such as forcing Ad-networks, search engines, DNS servers, etc to ‘delist’ entire sites for having a single bit of infringing content.)

No one denies that intellectual property piracy is a real problem. There are possible legal remedies to these challenges, however PIPA is a very dangerous solution to these problems.

Please retract your support and sponsorship of the PIPA (S.968) immediately.

Thank you,

<NAME>