pg.ddx.io pgsql-sql@postgresql.org mailing list archive
help / color / mirror / Atom feedExport tab delimited from mysql to postgres.
9+ messages / 6 participants
[nested] [flat]
* Export tab delimited from mysql to postgres.
@ 2004-10-10 22:59 Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au>
0 siblings, 1 reply; 9+ messages in thread
From: Theo Galanakis @ 2004-10-10 22:59 UTC (permalink / raw)
To: pgsql-sql
Has anyone been able to export and import a tab delimited file from mysql to
postgres successfully?
This is my dilemma which I have posted on mysql sites with no response,
however postgres users may have come across the same issue:
Trying to export a tab delimited file with the following command, however it
does not appear to strip out the CR/LF within a varchar column.
select * into outfile '/tmp/users.txt' FIELDS ESCAPED BY '\\' LINES
TERMINATED BY '\r\n' from users;
However when I use the below function to export as an insert statement, it
escapes all the CR/LF within each column.
mysqldump --database mydatabase --password --user myusername --verbose
--complete-insert --tables users > myfile.sql
Does anyone have any insigt into why the above select does not escape CR/LF
within varchar columns?
I really want to get the export as tab delimited working to import into
postgres,which is much faster than insert statements.
Theo
______________________________________________________________________
This email, including attachments, is intended only for the addressee
and may be confidential, privileged and subject to copyright. If you
have received this email in error, please advise the sender and delete
it. If you are not the intended recipient of this email, you must not
use, copy or disclose its content to anyone. You must not copy or
communicate to others content that is confidential or subject to
copyright, unless you have the consent of the content owner.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Export tab delimited from mysql to postgres.
@ 2004-10-11 09:33 Harald Fuchs <hf0722x@protecting.net>
parent: Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au>
0 siblings, 0 replies; 9+ messages in thread
From: Harald Fuchs @ 2004-10-11 09:33 UTC (permalink / raw)
To: pgsql-sql
In article <82E30406384FFB44AFD1012BAB230B5505F40D3A@shiva.au.lpint.net>,
Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au> writes:
> Has anyone been able to export and import a tab delimited file from mysql to
> postgres successfully?
> This is my dilemma which I have posted on mysql sites with no response,
> however postgres users may have come across the same issue:
> Trying to export a tab delimited file with the following command, however it
> does not appear to strip out the CR/LF within a varchar column.
> select * into outfile '/tmp/users.txt' FIELDS ESCAPED BY '\\' LINES TERMINATED
> BY '\r\n' from users;
It's a MySQL bug/misfeature that it doesn't translate CRs into '\r'.
Nevertheless, you can import a MySQL tab delimited file into
PostgreSQL - just pipe it through a small sed script.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Export tab delimited from mysql to postgres.
@ 2004-10-11 23:23 Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au>
0 siblings, 2 replies; 9+ messages in thread
From: Theo Galanakis @ 2004-10-11 23:23 UTC (permalink / raw)
To: pgsql-sql
Could you provide a example of how to do this?
I actually ended up exporting the data as Insert statements, which
strips out cf/lf within varchars. However it takes an eternity to import
200,000 records... 24 hours infact???? Is this normal?
Theo
-----Original Message-----
From: Harald Fuchs [mailto:hf0722x@protecting.net]
Sent: Monday, 11 October 2004 7:34 PM
To: pgsql-sql@postgresql.org
Subject: Re: [SQL] Export tab delimited from mysql to postgres.
In article <82E30406384FFB44AFD1012BAB230B5505F40D3A@shiva.au.lpint.net>,
Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au> writes:
> Has anyone been able to export and import a tab delimited file from
> mysql to postgres successfully? This is my dilemma which I have posted
> on mysql sites with no response, however postgres users may have come
> across the same issue:
> Trying to export a tab delimited file with the following command,
> however it does not appear to strip out the CR/LF within a varchar
> column. select * into outfile '/tmp/users.txt' FIELDS ESCAPED BY '\\'
> LINES TERMINATED BY '\r\n' from users;
It's a MySQL bug/misfeature that it doesn't translate CRs into '\r'.
Nevertheless, you can import a MySQL tab delimited file into PostgreSQL -
just pipe it through a small sed script.
---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster
______________________________________________________________________
This email, including attachments, is intended only for the addressee
and may be confidential, privileged and subject to copyright. If you
have received this email in error, please advise the sender and delete
it. If you are not the intended recipient of this email, you must not
use, copy or disclose its content to anyone. You must not copy or
communicate to others content that is confidential or subject to
copyright, unless you have the consent of the content owner.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Export tab delimited from mysql to postgres.
@ 2004-10-11 23:40 Bruno Wolff III <bruno@wolff.to>
parent: Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au>
1 sibling, 0 replies; 9+ messages in thread
From: Bruno Wolff III @ 2004-10-11 23:40 UTC (permalink / raw)
To: Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au>; +Cc: pgsql-sql
On Tue, Oct 12, 2004 at 09:23:41 +1000,
Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au> wrote:
>
> Could you provide a example of how to do this?
>
> I actually ended up exporting the data as Insert statements, which
> strips out cf/lf within varchars. However it takes an eternity to import
> 200,000 records... 24 hours infact???? Is this normal?
Are you doing the load in one transaction?
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Export tab delimited from mysql to postgres.
@ 2004-10-12 00:46 Christopher Browne <cbbrowne@acm.org>
parent: Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au>
1 sibling, 0 replies; 9+ messages in thread
From: Christopher Browne @ 2004-10-12 00:46 UTC (permalink / raw)
To: pgsql-sql
Quoth Theo.Galanakis@lonelyplanet.com.au (Theo Galanakis):
> Could you provide a example of how to do this?
>
> I actually ended up exporting the data as Insert statements,
> which strips out cf/lf within varchars. However it takes an eternity
> to import 200,000 records... 24 hours infact???? Is this normal?
I expect that this results from each INSERT being a separate
transaction.
If you put a BEGIN at the start and a COMMIT at the end, you'd
doubtless see an ENORMOUS improvement.
That's not even the _big_ improvement, either. The _big_ improvement
would involve reformatting the data so that you could use the COPY
statement, which is _way_ faster than a bunch of INSERTs. Take a look
at the documentation to see the formatting that is needed:
http://techdocs.postgresql.org/techdocs/usingcopy.php
http://www.faqs.org/docs/ppbook/x5504.htm
http://www.postgresql.org/docs/7.4/static/sql-copy.html
--
output = ("cbbrowne" "@" "ntlug.org")
http://www3.sympatico.ca/cbbrowne/lsf.html
Question: How many surrealists does it take to change a light bulb?
Answer: Two, one to hold the giraffe, and the other to fill the bathtub
with brightly colored machine tools.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Export tab delimited from mysql to postgres.
@ 2004-10-12 05:33 Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au>
0 siblings, 2 replies; 9+ messages in thread
From: Theo Galanakis @ 2004-10-12 05:33 UTC (permalink / raw)
To: pgsql-sql
Thanks for all your comments,
I have beent trying the insert within a transaction block, however
it does not seem to reduce the time it takes to process each records. Mind
you there are 80 column and the insert statement explicitly defines the
column to insert into.
I need any tip I can get help me transform the text file into a
format postgres copy will successfully read.
Here is sample of the current format of a mysql tab delimited dump..
columnA columnB
1 What a day!
2 What a week it has
been!
3 What the!
As you can see row 2 has a value that holds a CR value which ends up
wrapping around onto the third line. Postgres copy command does not like
this and mysql is unable to replace the value with another type of
delimiter, like a \r.
So I gather I have to some how manually replace the carriage return with
something postgres understand \r...
columnA columnB
1 What a day!
2 What a week it has \r been!
3 What the!
How do I do this without getting a text file that looks like this
1 What a day! \r\n2 What a week it has \r been!\r\n3
What the!\r\n
Any help would be appreciated.
Theo
-----Original Message-----
From: Christopher Browne [mailto:cbbrowne@acm.org]
Sent: Tuesday, 12 October 2004 10:46 AM
To: pgsql-sql@postgresql.org
Subject: Re: [SQL] Export tab delimited from mysql to postgres.
Quoth Theo.Galanakis@lonelyplanet.com.au (Theo Galanakis):
> Could you provide a example of how to do this?
>
> I actually ended up exporting the data as Insert statements,
> which strips out cf/lf within varchars. However it takes an eternity
> to import 200,000 records... 24 hours infact???? Is this normal?
I expect that this results from each INSERT being a separate transaction.
If you put a BEGIN at the start and a COMMIT at the end, you'd doubtless see
an ENORMOUS improvement.
That's not even the _big_ improvement, either. The _big_ improvement would
involve reformatting the data so that you could use the COPY statement,
which is _way_ faster than a bunch of INSERTs. Take a look at the
documentation to see the formatting that is needed:
http://techdocs.postgresql.org/techdocs/usingcopy.php
http://www.faqs.org/docs/ppbook/x5504.htm
http://www.postgresql.org/docs/7.4/static/sql-copy.html
--
output = ("cbbrowne" "@" "ntlug.org")
http://www3.sympatico.ca/cbbrowne/lsf.html
Question: How many surrealists does it take to change a light bulb?
Answer: Two, one to hold the giraffe, and the other to fill the bathtub
with brightly colored machine tools.
---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
______________________________________________________________________
This email, including attachments, is intended only for the addressee
and may be confidential, privileged and subject to copyright. If you
have received this email in error, please advise the sender and delete
it. If you are not the intended recipient of this email, you must not
use, copy or disclose its content to anyone. You must not copy or
communicate to others content that is confidential or subject to
copyright, unless you have the consent of the content owner.=
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Export tab delimited from mysql to postgres.
@ 2004-10-12 07:20 Pierre-Frédéric Caillaud <lists@boutiquenumerique.com>
parent: Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au>
1 sibling, 0 replies; 9+ messages in thread
From: Pierre-Frédéric Caillaud @ 2004-10-12 07:20 UTC (permalink / raw)
To: pgsql-sql
A tested example in Python :
Basically it counts the \t and accumulates the lines until it has enough
and then prints the line.
Note : as an exercise you could add a test so that there are exactly
(columns-1) delimiters and not >=(columns-1).
def grouplines( in_stream, columns, delimiter ):
num_delimiters = columns - 1
accum = ''
for line in in_stream:
accum += line
if accum.count( delimiter ) >= num_delimiters:
print accum.replace( "\n", "\\n" )
accum = ''
if accum:
print "Last line unterminated."
grouplines( open( 'data.in' ), 3, "\t" ):
Input data (I added a column over your example):
1 What a day! A
2 What a week it has
been! B
3 What the! C
Output :
1 What a day! A\n
2 What a week it has\nbeen! B\n
3 What the! C
Have fun with your copy !
On Tue, 12 Oct 2004 15:33:46 +1000, Theo Galanakis
<Theo.Galanakis@lonelyplanet.com.au> wrote:
>
> Thanks for all your comments,
>
> I have beent trying the insert within a transaction block, however
> it does not seem to reduce the time it takes to process each records.
> Mind
> you there are 80 column and the insert statement explicitly defines the
> column to insert into.
>
> I need any tip I can get help me transform the text file into a
> format postgres copy will successfully read.
>
>
> Here is sample of the current format of a mysql tab delimited dump..
>
> columnA columnB
>
> 1 What a day!
> 2 What a week it has
> been!
> 3 What the!
>
> As you can see row 2 has a value that holds a CR value which ends up
> wrapping around onto the third line. Postgres copy command does not like
> this and mysql is unable to replace the value with another type of
> delimiter, like a \r.
>
> So I gather I have to some how manually replace the carriage return with
> something postgres understand \r...
>
>
> columnA columnB
>
> 1 What a day!
> 2 What a week it has \r been!
> 3 What the!
>
> How do I do this without getting a text file that looks like this
>
> 1 What a day! \r\n2 What a week it has \r been!\r\n3
> What the!\r\n
>
> Any help would be appreciated.
>
> Theo
>
> -----Original Message-----
> From: Christopher Browne [mailto:cbbrowne@acm.org]
>
> Sent: Tuesday, 12 October 2004 10:46 AM
> To: pgsql-sql@postgresql.org
> Subject: Re: [SQL] Export tab delimited from mysql to postgres.
>
>
> Quoth Theo.Galanakis@lonelyplanet.com.au (Theo Galanakis):
>> Could you provide a example of how to do this?
>>
>> I actually ended up exporting the data as Insert statements,
>> which strips out cf/lf within varchars. However it takes an eternity
>
>> to import 200,000 records... 24 hours infact???? Is this normal?
>
> I expect that this results from each INSERT being a separate transaction.
>
> If you put a BEGIN at the start and a COMMIT at the end, you'd doubtless
> see
> an ENORMOUS improvement.
>
> That's not even the _big_ improvement, either. The _big_ improvement
> would
> involve reformatting the data so that you could use the COPY statement,
> which is _way_ faster than a bunch of INSERTs. Take a look at the
> documentation to see the formatting that is needed:
>
> http://techdocs.postgresql.org/techdocs/usingcopy.php
> http://www.faqs.org/docs/ppbook/x5504.htm
> http://www.postgresql.org/docs/7.4/static/sql-copy.html
> --
>
> output = ("cbbrowne" "@" "ntlug.org")
> http://www3.sympatico.ca/cbbrowne/lsf.html
> Question: How many surrealists does it take to change a light bulb?
>
> Answer: Two, one to hold the giraffe, and the other to fill the bathtub
> with brightly colored machine tools.
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
> (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
>
>
> ______________________________________________________________________
> This email, including attachments, is intended only for the addressee
> and may be confidential, privileged and subject to copyright. If you
> have received this email in error, please advise the sender and delete
> it. If you are not the intended recipient of this email, you must not
> use, copy or disclose its content to anyone. You must not copy or
>
> communicate to others content that is confidential or subject to
>
> copyright, unless you have the consent of the content owner.
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Export tab delimited from mysql to postgres.
@ 2004-10-12 13:56 Tom Lane <tgl@sss.pgh.pa.us>
parent: Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au>
1 sibling, 0 replies; 9+ messages in thread
From: Tom Lane @ 2004-10-12 13:56 UTC (permalink / raw)
To: Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au>; +Cc: pgsql-sql
Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au> writes:
> As you can see row 2 has a value that holds a CR value which ends up
> wrapping around onto the third line. Postgres copy command does not like
> this and mysql is unable to replace the value with another type of
> delimiter, like a \r.
> So I gather I have to some how manually replace the carriage return with
> something postgres understand \r...
> columnA columnB
> 1 What a day!
> 2 What a week it has \r been!
> 3 What the!
> How do I do this without getting a text file that looks like this
> 1 What a day! \r\n2 What a week it has \r been!\r\n3
> What the!\r\n
Looks like a simple sed problem to me:
cat myfile | sed 's/\r$//' | sed 's/\r/\\r/g' >newfile
(Most likely you can do both steps in one sed process, but this is easy
to follow.) This assumes that you only have newlines (\n) at the real
ends of lines, else you need to think harder about how to tell the
difference between data and formatting.
regards, tom lane
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Export tab delimited from mysql to postgres.
@ 2004-10-12 23:56 Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au>
0 siblings, 0 replies; 9+ messages in thread
From: Theo Galanakis @ 2004-10-12 23:56 UTC (permalink / raw)
To: pgsql-sql
Thankyou all for your advice,
Looking further into the issue I have realised that "data and formatting"
lines are denoted by the same CR/LF symbols. So if I where to replace all
CR/LF symbols it would create a file with one line... I have replace the
CR/LF for reading purposes below with \r\n or alternatively 0D 0A (hex):
columnA columnB
1 What a day!\r\n
2 What a week it has\r\n
been!\r\n
3 What the!\r\n
What this means that it is impossible to move line three back to line two to
look something like this:
columnA columnB
1 What a day!\r\n
2 What a week it has\rbeen!\r\n
3 What the!\r\n
I was thinking of writing code that checked each line to see if correct
number of tab delimited elements in this case 2 elements, if this was not
the case replace the \r\n with \r.
Theo
-----Original Message-----
From: Harald Fuchs [mailto:hf0722x@protecting.net]
Sent: Wednesday, 13 October 2004 1:38 AM
To: Theo Galanakis
Subject: Re: Export tab delimited from mysql to postgres.
> Could you provide a example of how to do this?
Depends on what exactly your dump file contains. Probably something like
sed 's/\\\r/\\r/g'
This means: replace all carriage returns preceded by a backslash by '\r'.
______________________________________________________________________
This email, including attachments, is intended only for the addressee
and may be confidential, privileged and subject to copyright. If you
have received this email in error, please advise the sender and delete
it. If you are not the intended recipient of this email, you must not
use, copy or disclose its content to anyone. You must not copy or
communicate to others content that is confidential or subject to
copyright, unless you have the consent of the content owner.=
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2004-10-12 23:56 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2004-10-10 22:59 Export tab delimited from mysql to postgres. Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au>
2004-10-11 09:33 ` Harald Fuchs <hf0722x@protecting.net>
2004-10-11 23:23 Re: Export tab delimited from mysql to postgres. Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au>
2004-10-11 23:40 ` Bruno Wolff III <bruno@wolff.to>
2004-10-12 00:46 ` Christopher Browne <cbbrowne@acm.org>
2004-10-12 05:33 Re: Export tab delimited from mysql to postgres. Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au>
2004-10-12 07:20 ` Pierre-Frédéric Caillaud <lists@boutiquenumerique.com>
2004-10-12 13:56 ` Tom Lane <tgl@sss.pgh.pa.us>
2004-10-12 23:56 Re: Export tab delimited from mysql to postgres. Theo Galanakis <Theo.Galanakis@lonelyplanet.com.au>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox