pg.ddx.io pgsql-sql@postgresql.org mailing list archivehelp / color / mirror / Atom feed
I would like to alter the COPY command 3+ messages / 2 participants [nested] [flat]
* I would like to alter the COPY command @ 2006-12-24 03:08 Mason <freemason@gmail.com> 2006-12-24 03:08 ` Fwd: I would like to alter the COPY command Mason <freemason@gmail.com> 0 siblings, 1 reply; 3+ messages in thread From: Mason @ 2006-12-24 03:08 UTC (permalink / raw) To: pgsql-sql What I have is data with two different characters for "start quote" and "end quote". In my case it's '[' and ']', but it could be anything from "smart quotes", to parentheses, to brackets, braces, ^/$ in regexps, etc. I think this isn't too unreasonable a feature to have to make copy more functional when importing data that is difficult to transform properly beforehand (in my case is about half a terabyte of log files, which takes hours and hours, just to cat, let alone reparse and dump into COPY). Now, in my case I can just say "cat file | tr '[]' '""' | psql -f import.sql", but then I lose the ability for psql to do anything smart like using mmap (I'm making assumptions that it does anything smart like that, but even if it doesn't now, it could some day). So, I'm a passable c/c++ programmer, when I have to be, so theoretically I can do the work myself, but I have never touched postgres before, so I don't know where to begin. Any ideas how to add this? ^ permalink raw reply [nested|flat] 3+ messages in thread
* Fwd: I would like to alter the COPY command 2006-12-24 03:08 I would like to alter the COPY command Mason <freemason@gmail.com> @ 2006-12-24 03:08 ` Mason <freemason@gmail.com> 2006-12-29 17:43 ` Re: Fwd: I would like to alter the COPY command Andrew Dunstan <andrew@dunslane.net> 0 siblings, 1 reply; 3+ messages in thread From: Mason @ 2006-12-24 03:08 UTC (permalink / raw) To: pgsql-patches@postgresql.org What I have is data with two different characters for "start quote" and "end quote". In my case it's '[' and ']', but it could be anything from "smart quotes", to parentheses, to brackets, braces, ^/$ in regexps, etc. I think this isn't too unreasonable a feature to have to make copy more functional when importing data that is difficult to transform properly beforehand (in my case is about half a terabyte of log files, which takes hours and hours, just to cat, let alone reparse and dump into COPY). Now, in my case I can just say "cat file | tr '[]' '""' | psql -f import.sql", but then I lose the ability for psql to do anything smart like using mmap (I'm making assumptions that it does anything smart like that, but even if it doesn't now, it could some day). So, I'm a passable c/c++ programmer, when I have to be, so theoretically I can do the work myself, but I have never touched postgres before, so I don't know where to begin. Any ideas how to add this? ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Fwd: I would like to alter the COPY command 2006-12-24 03:08 I would like to alter the COPY command Mason <freemason@gmail.com> 2006-12-24 03:08 ` Fwd: I would like to alter the COPY command Mason <freemason@gmail.com> @ 2006-12-29 17:43 ` Andrew Dunstan <andrew@dunslane.net> 0 siblings, 0 replies; 3+ messages in thread From: Andrew Dunstan @ 2006-12-29 17:43 UTC (permalink / raw) To: Mason <freemason@gmail.com>; +Cc: pgsql-patches@postgresql.org Mason wrote: > What I have is data with two different characters for "start quote" > and "end quote". In my case it's '[' and ']', but it could be > anything from "smart quotes", to parentheses, to brackets, braces, ^/$ > in regexps, etc. I think this isn't too unreasonable a feature to > have to make copy more functional when importing data that is > difficult to transform properly beforehand (in my case is about half a > terabyte of log files, which takes hours and hours, just to cat, let > alone reparse and dump into COPY). I think regexps would be going too far. One simple approach which wouldn't involve any grammar changes would be to allow the quote parameter to be 2 chars long instead of one, and treat the second as the closing quote char (which would otherwise default to the first char). An argument against that would be that it would preclude us from in future allowing multi-char quote marks. I'm not sure if that matters quite so much as it would for the delimiter parameter. > > Now, in my case I can just say "cat file | tr '[]' '""' | psql -f > import.sql", but then I lose the ability for psql to do anything smart > like using mmap (I'm making assumptions that it does anything smart > like that, but even if it doesn't now, it could some day). > Well, you also earn a "Useless use of cat" award ;-) And no, we don't mmap the file, but we do do some smart buffering stuff, thanks to Greenplum. cheers andrew ^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2006-12-29 17:43 UTC | newest] Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2006-12-24 03:08 I would like to alter the COPY command Mason <freemason@gmail.com> 2006-12-24 03:08 ` Mason <freemason@gmail.com> 2006-12-29 17:43 ` Andrew Dunstan <andrew@dunslane.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox