Someone asked on a mailing list how to redirect output from a running process. They had a program which had been running for a long period of time without having stdout redirected to a file. They wanted to logout (to move the laptop that was used for the ssh session) but not kill the process (or lose output).
Most responses were of the form “you should have used screen or nohup” which is all very well if you had planned to logout and leave it running (or even planned to have it run for a long time).
Fortunately it is quite possible to redirect output of a running process. I will use cat as a trivial example but the same technique will work for most programs that do simple IO (of course programs that do terminal IO may be more tricky – but you could always redirect from the tty device of a ssh session to the tty device of a screen session).
Firstly I run the command “cat > foo1” in one session and test that data from stdin is copied to the file. Then in another session I redirect the output:
Firstly find the PID of the process:
$ ps aux|grep cat
rjc 6760 0.0 0.0 1580 376 pts/5 S+ 15:31 0:00 cat
Now check the file handles it has open:
$ ls -l /proc/6760/fd
total 3
lrwx—— 1 rjc rjc 64 Feb 27 15:32 0 -> /dev/pts/5
l-wx—— 1 rjc rjc 64 Feb 27 15:32 1 -> /tmp/foo1
lrwx—— 1 rjc rjc 64 Feb 27 15:32 2 -> /dev/pts/5
Now run GDB:
$ gdb -p 6760 /bin/cat
GNU gdb 6.4.90-debian
Copyright (C) 2006 Free Software Foundation, Inc
[lots more license stuff snipped]
Attaching to program: /bin/cat, process 6760
[snip other stuff that’s not interesting now]
(gdb) p close(1)
$1 = 0
(gdb) p creat(“/tmp/foo3”, 0600)
$2 = 1
(gdb) q
The program is running. Quit anyway (and detach it)? (y or n) y
Detaching from program: /bin/cat, process 6760
The “p” command in GDB will print the value of an expression, an expression can be a function to call, it can be a system call… So I execute a close() system call and pass file handle 1, then I execute a creat() system call to open a new file. The result of the creat() was 1 which means that it replaced the previous file handle. If I wanted to use the same file for stdout and stderr or if I wanted to replace a file handle with some other number then I would need to call the dup2() system call to achieve that result.
For this example I chose to use creat() instead of open() because there are fewer parameter. The C macros for the flags are not usable from GDB (it doesn’t use C headers) so I would have to read header files to discover this – it’s not that hard to do so but would take more time. Note that 0600 is the octal permission for the owner having read/write access and the group and others having no access. It would also work to use 0 for that parameter and run chmod on the file later on.
After that I verify the result:
ls -l /proc/6760/fd/
total 3
lrwx—— 1 rjc rjc 64 2008-02-27 15:32 0 -> /dev/pts/5
l-wx—— 1 rjc rjc 64 2008-02-27 15:32 1 -> /tmp/foo3 <====
lrwx—— 1 rjc rjc 64 2008-02-27 15:32 2 -> /dev/pts/5
Typing more data in to cat results in the file /tmp/foo3 being appended to.
Update: If you want to close the original session you need to close all file handles for it, open a new device that can be the controlling tty, and then call setsid().
Nice trick. I think retty uses the same approach, except that it only supports redirecting to a new tty.
Now if only I could change the UID of a running program. Often I edit some configuration file or do something else that needs to run as root, but don’t realize I ran it as a normal user until after I’ve put work into it. While I could certainly save the file somewhere writable and move it in place afterward, I’d love to just sudo-after-the-fact.
Josh: Thanks for the tip, retty is a very useful program. It’s a pity that they implemented it as an i386 executable instead of a wrapper for GDB (which would have been portable).
However retty does not solve the original problem as it keeps the same controlling terminal.
I very much liked your way of doing things. Note on the TODO is added to try and add that into the package or to replace retty altogether (don’t expect that too soon, though..)
Richard
Hi,
Two things to add here:
1. You can use “call” (IIRC) instead of “print” if you want to feel a tiny bit less hackish.
2. You’re not guaranteed to get the same fd back on creat(), especially if there’s another fd than 1 that you want to redirect. dup2() is your friend here.
3. cryopid can do this for interactive processes etc. too (say, your irssi), provided that you’re prepared to lose all your sockets (without kernel patching).
/* Steinar */
[…] log out. Of course, you’ve forgotten to use screen so when you log out, the process dies. The post by Rusell Coker shows that it is still possible to redirect the output of a running process. The […]
Awesome– I had no idea gdb could run function calls!
http://cryopid.berlios.de/
CryoPID allows suspending the state of a running process via ptrace and then restarting it later (after a reboot etc).
http://www.fazlamesai.net/?a=article&sid=4951
Thanks to the above blog post for pointing this out.
http://sourceforge.net/projects/dmtcp
The above SourceForge project seems to do similar things to CryoPID, and they are working on suspending the state of a cluster too.
This method doesn’t work for me – when I try to redirect command “cat” to another controlling tty (e.g. from pts/2 to tty1). I do the steps exactly as described above, but finally setsid() returns -1 , and if I (before quitting from gdb) run “ps aux|grep cat” it still indicates the old tty. And what is worse, – right after I quit from gdb (q), “cat” procsess immediately dies. So I think it’s not enough just to change all file descriptors of ttys used by cat, it’s also necessary to change somehow controlling tty, so that “ps aux|grep cat” would show the new tty, rather than old one.
What do I wrong ? Please help.
Hi,
I tried this…closed all open handles…opened the new ones to new device — “/dev/pts/4”
and then called setsid();
my setsid() is always returning -1.
Please help…
thanks,
Pradnesh