Re: [SLUG] tailing, following and filtering

[Matthew Hannigan <mlh@xxxxxxxxxx>]

On Thu, Nov 23, 2006 at 05:10:14PM +1100, Penedo wrote:
> On 23/11/06, Matthew Hannigan <mlh@xxxxxxxxxx> wrote:
> >You don't get the output in as timely a fashion;
> >it comes with spurts as the buffer is filled up
> >then written out.
> 
> What buffer? tail's manual doesn't mention anything about buffers but

grep's buffer, the normal stdio buffer I guess

> my experience is that you get individual lines even then its output is
> piped (i.e. even when the output is about 80 bytes long, i.e. much
> less than any reasonable buffer size except a line buffer).
> With grep you can force line-buffer with  --line-buffering, if you
> think this is the problem.
> 
> Maybe what you see is tail's sleep between retries. The default is 1
> but see below.
> 
> >
> >Try this out:
> >
> >        while true ; do s=$RANDOM; echo $s; usleep $s; done >crap &
> >
> >        tail -f crap |grep 3
> >
> >vs
> >        tail -f crap | grep 3 | grep -v 8
> 
> I get output every more-or-less 1 second, which makes me suspect that
> my "tail's sleep(1)" explanation is indeed the reason. Adding "-s 0"
> gives me continuous output. And that's even without forcing

Doesn't me; I get it spurty; like 50 seconds or so between output.

> --line-buffering on grep. So far I thing this confirms my theory.

Weird.

With --line-buffered on the first grep I get it continuous.
If only the second grep is --line-buffered, then it goes spurty again.
That points to the first grep buffering by default, to me.

> Another option to test this might be to "strace -tt -f ...", but so
> far I wasn't convinced by your explanation so I'll leave it up to you
> to experiment :^).
> 
> Another experiment - slow down the data writing into "crap" and see
> whether you get individual lines of 5-6 bytes each - this again should
> prove the "buffer theory" wrong, IMHO.

It's possible it's a difference in kernels/shells/something else,
but that's certainly unexpected.

I didn't even know about tail's -s option.  That's what I get
for spending too much time in a non-GNU wilderness.

I still think there's a case for a 'less' like tool, or
less itself to do some filtering, modifiable on the fly.

I'm sure there's some perl module/program to do it.

Matt