[RLUG] quick question

Bruce Robertson bruce at greatbasin.net
Thu Jun 29 07:38:12 PDT 2006 

Well, in that case I would have let the SQL server complain about 
existing tables, or I would have done the whole thing as a pipe:

cat file | <sed command to remove first 300 lines> | mysql <or whatever 
you're using>

Bruce Robertson, President/CEO                           +1-775-348-7299
Great Basin Internet Services, Inc.    company-wide fax: +1-775-348-9412
http://www.greatbasin.net                       my efax: +1-775-201-1553



Grant Kelly wrote:
> Thanks for all the responses.
>
> sed did the trick in about 10 minutes (on a multi-threaded dual Xeon
> box w/ 2GB of RAM [though it probably only used a single thread]).
>
> It is an interesting problem, and it seems like there would be a
> better solution widely available. I see how the solution could be a
> part of the file system; does anyone know if any available file system
> can do this? Perhaps Sun's self-proclaimed glorious ZFS?
>
> Also, FYI, it's not a flat file database, but it is a database dump
> that I need to import, and the first 300 lines are SQL commands to
> create tables that I already have, hence the desire to remove the
> lines.
>
> Thanks again,
> Grant
>
> On 6/28/06, James Washer <washer at trlp.com> wrote:
>> whoops, my first response went direct to Brian, so I'll copy it here:
>>
>> debugfs? How's that going to help if after chopping off the first 300 
>> lines, the data is not block aligned? You can debugfs all night long 
>> and you'll not get around that problem. IMPOSSIBLE I say!
>>
>>  - jim
>>
>>
>> On Wed, 28 Jun 2006 18:41:44 -0700
>> Brian Chrisman <incubus at izap.com> wrote:
>>
>> > Hey.. not 'impossible'.. just *really annoying*... ie, firing up 
>> debugfs
>> > or whatever they're using these days to manipulate filesystem 
>> internals. :)
>> > Sorry, but the impossible in all caps is just begging for
>> > technicalities.. :-)
>> >
>> > -Brian
>> >
>> > James Washer wrote:
>> >
>> > >If he wanted to lop off the first xxx bytes, without "touching" 
>> the remainder of the file... then it's IMPOSSIBLE under linux/unix.
>> > >
>> > >You mention flat-file-database... it's that's the case, I hope he 
>> can stop updates while removing the first 300 lines... else, the 
>> problem is just a bit harder (in fact, it can become impossible if 
>> his data base engine does not use some form of file locking.)
>> > > - jim
>> > >
>> > >On Wed, 28 Jun 2006 17:50:14 -0700 (PDT)
>> > >Sebastian Smith <ssmith at cse.unr.edu> wrote:
>> > >
>> > >
>> > >
>> > >>I think what Grant meant by "in place" was that he didn't want to 
>> read the
>> > >>entire file, just lop off the first 300 lines -- perhaps he can 
>> correct
>> > >>me.  I'm guessing storage space isn't an issue.  Regardless, I 
>> don't know
>> > >>of a way to do that.
>> > >>
>> > >>You really need to get away from those flat file databases Grant ;)
>> > >>
>> > >>
>> > >>On Wed, 28 Jun 2006, James Washer wrote:
>> > >>
>> > >>
>> > >>
>> > >>>it's the "working in place" that makes this difficult, else 
>> there are countless ways to do this, including the simple perl
>> > >>>
>> > >>>   perl -ne 'next unless $. >300;print'
>> > >>>
>> > >>>On Wed, 28 Jun 2006 17:26:47 -0700
>> > >>>"Brandon Mitchell" <magicsmoke at gmail.com> wrote:
>> > >>>
>> > >>>
>> > >>>
>> > >>>>This is an interesting problem, and seems to be revealing a 
>> little bit
>> > >>>>about what type of user/admin is in each of us.
>> > >>>>
>> > >>>>So where's Nick with an Emacs Lisp macro for this task? :P
>> > >>>>
>> > >>>>On 6/28/06, Anna <christiana at hipointcoffee.com> wrote:
>> > >>>>
>> > >>>>
>> > >>>>>find out which byte terminates the first 300 lines.  maybe...
>> > >>>>>
>> > >>>>>~$ BYTES=$(head -300 nameofbigfile.txt | wc -c)
>> > >>>>>
>> > >>>>>then use that info with dd skip the first part of the input 
>> file...
>> > >>>>>
>> > >>>>>~$ dd if=nameofbigfile.txt of=truncatedversion.pl ibs=$BYTES  
>> skip=1
>> > >>>>>
>> > >>>>>one of many ways, I'm sure.  I think this way should be pretty 
>> fast
>> > >>>>>because it works on a line by line basis for just a small part 
>> of the
>> > >>>>>file.  The rest, with dd, is done in larger pieces.
>> > >>>>>
>> > >>>>>- Anna
>> > >>>>>
>> > >>>>>
>> > >>>>>On Wed, Jun 28, 2006 at 01:01:03PM -0700, Grant Kelly wrote:
>> > >>>>>
>> > >>>>>
>> > >>>>>>Alright unix fans, who can answer this the best?
>> > >>>>>>
>> > >>>>>>I have a text file, it's about 2.3 GB. I need to delete the 
>> first 300
>> > >>>>>>lines, and I don't want to have to load the entire thing into an
>> > >>>>>>editor.
>> > >>>>>>
>> > >>>>>>I'm trying `sed '1,300d' inputfile > output file`  but it's 
>> taking a
>> > >>>>>>long time (and space) to output everything to the new file.
>> > >>>>>>
>> > >>>>>>There has got to be a better way, a way that can do this 
>> in-place...
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>Grant
>> > >>>>>>
>> > >>>>>>_______________________________________________
>> > >>>>>>RLUG mailing list
>> > >>>>>>RLUG at rlug.org
>> > >>>>>>http://lists.rlug.org/mailman/listinfo/rlug
>> > >>>>>>
>> > >>>>>>
>> > >>>>>_______________________________________________
>> > >>>>>RLUG mailing list
>> > >>>>>RLUG at rlug.org
>> > >>>>>http://lists.rlug.org/mailman/listinfo/rlug
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>--
>> > >>>>If UNIX doesn't have the solution you have the wrong problem.
>> > >>>>UNIX is simple, but it takes a genius to understand it's 
>> simplicity.
>> > >>>>
>> > >>>>_______________________________________________
>> > >>>>RLUG mailing list
>> > >>>>RLUG at rlug.org
>> > >>>>http://lists.rlug.org/mailman/listinfo/rlug
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>_______________________________________________
>> > >>>RLUG mailing list
>> > >>>RLUG at rlug.org
>> > >>>http://lists.rlug.org/mailman/listinfo/rlug
>> > >>>
>> > >>>
>> > >>>
>> > >>_______________________________________________
>> > >>RLUG mailing list
>> > >>RLUG at rlug.org
>> > >>http://lists.rlug.org/mailman/listinfo/rlug
>> > >>
>> > >>
>> > >>
>> > >
>> > >_______________________________________________
>> > >RLUG mailing list
>> > >RLUG at rlug.org
>> > >http://lists.rlug.org/mailman/listinfo/rlug
>> > >
>> > >
>> >
>> >
>> > _______________________________________________
>> > RLUG mailing list
>> > RLUG at rlug.org
>> > http://lists.rlug.org/mailman/listinfo/rlug
>> >
>>
>> _______________________________________________
>> RLUG mailing list
>> RLUG at rlug.org
>> http://lists.rlug.org/mailman/listinfo/rlug
>>
>
> _______________________________________________
> RLUG mailing list
> RLUG at rlug.org
> http://lists.rlug.org/mailman/listinfo/rlug
>
>
>

More information about the RLUG mailing list