Picked up a ticket from the NOC today for a nagios alert.
It's the standard Dell Hardware check, tells us when DIMMS go bad, or the RAID cache needs replacing, stuff like that.
It was returning an error message I'd never seen before, and I deal with all the hardware tickets.
Checking the machine management interface all systems were nominal.
Running /usr/path/to/check_script returned a happy
OK: - SYSTEM: 'ServerType', 98GB RAM ETC message.
"Well, that's strange" I thought
The script runs as an NRPE check as the nobody user. Running the check remotely came back with the same error.
"Couldn't close filehandle for command '"/usr/sbin/utility" -? 2>&1':"
I check open files and restart the services it uses. No change.
I reset the management interface. No change.
Asking the system its uptime tells me 9 days. I check the nagios report, the alert's been downtimed for 100+ days and it just so happened it came out today.
I remind myself to thank the guy who was here before me if I ever find the bastard.
Checking the semaphores is probably something not a lot of people ever think to do but it's easy and sometimes causes weird shit, cleaning them up doesn't change anything.
I start to write a response to the ticket, "Downtime the service for 6 years and ignore it," but then I remember I just filled out my employee engagement survey and answered "Strongly Agree" to the "Employees at Company X do things right" question.
Grumbling, I set the ticket status to active and sit down.
We have a standard way for messing with selinux when shit goes wack, so I try that and break the record again. No change.
I set a shell for nobody and run the command again.
su - nobody
/usr/path/to/check_script
Same thing, filehandle error.
Digging into the script I found the offending if statement. Sure enough it was running the command it specified above: /usr/sbin/utility -?
I type that into nobody's prompt
Error! Invalid option specified: -1
I shake my head, confused, and type the same thing into a root prompt.
The utility happily replies with the help dialog for /usr/sbin/utility.I shake my head again, wondering what environmental variable is set in nobody's environment.
I switch back over and check a few variables, nothing seems too out of wack, especially after verifying this machine's sister nobody environment, on which the check works perfectly, of course.
Turning to the internet I'm presented with a couple of email thread archives from a half decade ago, each has a 6 replies and no resolution. I'm reminded of an XKCD that I take a break to print out and pin to my wall.
https://xkcd.com/979/The script we use is a very popular one, it's got a distro associated with it but I'm not about to run to them. It's a third party utility created by the University of Oslo, so I can't call Dell and use my expensive support contract.
Turning back to my terminal, I do the logical thing and echo -?
The system happily prints out exactly what I was expecting "-1"
By "exactly what I expecting" I mean "something that made me very, very worried."
My mind jumps to viruses, or worse, data breaches. This is exactly the weird kind of shit a lone admin stumbles across that security later uses to uncover a billion dollar breach.
I sit down, exhausted from jumping to conclusions and try escaping the question mark.
nobody@xBox306~:/> /usr/sbin/utility" \-\?
The system once spits out what I was ACTUALLY expecting this time and I'm faced with the help dialog for the utility again.
I strace the command, try the same binary from a different location, clear logs, and go down the wrong path for a couple hours.
The guy I looped in goes to the rest of the team, asking why is the shell returns "-1" when asked to "echo -?"
Have you got an idea yet?
The first admin asks if it does that to everything. Another starts stracing stuff.
nobody@xBox306~:/> foo=2
nobody@xBox306~:/> echo $foo
2
He starts checking environmental variables. The other guy is still stracing.
A third chimes in, I asked him about it 8 hours ago. 8 hours ago he told me to rebuild the machine. He thinks this is hilarious, having spent two days looking at straces 6 months ago.
We echo some stuff as root
root@xBox306~:/> echo -?
-?
Well that's reassuring I suppose.
The first admin posts a snippet from his terminal.
[admin1@xBox206 ~]$ sdiff /tmp/strace /tmp/strace.1
execve("/bin/echo", ["echo", "-1"], [/* 26 vars */]) = 0 | execve("/bin/echo", ["echo", "-?"], [/* 31 vars */]) = 0
brk(0) = 0xde6000 | brk(0)
The shell is expanding the argument before echo even gets it.
Get it yet?
Nobody doesn't have a home, thus when you su to it, which is NOT normally allowed, you end up in /.
Within / there was a file, /-1.
-? can be interpreted as -(exactly one arbitrary character)
When running it as root the system wouldn't be checking /, and root's home is set rightly to /root.
Thus we were actually asking the system to check the current directory for any files preceded by a "-" and passing whatever we found to /usr/sbin/utility
So /usr/sbin/utility -? became /usr/sbin/utility -1
Error! Invalid option specified: -1