pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedData replication to PostgreSQL
8+ messages / 5 participants
[nested] [flat]
* Data replication to PostgreSQL
@ 2016-12-29 14:35 Ram DBA <postgresdbas@gmail.com>
0 siblings, 2 replies; 8+ messages in thread
From: Ram DBA @ 2016-12-29 14:35 UTC (permalink / raw)
To: pgsql-admin
Hi team,
We have an use case to copy the data from Salesforce to PostgreSQL
environment but because of DDL changes that can happen to tables from
Salesforce application side, replication would eventually fail. Do we have
any open source tools that can also capture the DDL changes and replicate
to PostgreSQL?
Thank you in advance,
Ram
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Data replication to PostgreSQL
@ 2016-12-29 15:12 John Scalia <jayknowsunix@gmail.com>
parent: Ram DBA <postgresdbas@gmail.com>
1 sibling, 2 replies; 8+ messages in thread
From: John Scalia @ 2016-12-29 15:12 UTC (permalink / raw)
To: Ram DBA <postgresdbas@gmail.com>; +Cc: pgsql-admin
I'm not sure I understand what you're doing. Nearly all DDL is written to
the WAL segments. The only ones that aren't are things like unlogged and
temporary tables. I don't think temporary tables used by queries are
written either. Why do you think, however, that this isn't captured?
--
Jay
On Thu, Dec 29, 2016 at 9:35 AM, Ram DBA <postgresdbas@gmail.com> wrote:
> Hi team,
>
> We have an use case to copy the data from Salesforce to PostgreSQL
> environment but because of DDL changes that can happen to tables from
> Salesforce application side, replication would eventually fail. Do we have
> any open source tools that can also capture the DDL changes and replicate
> to PostgreSQL?
>
>
> Thank you in advance,
> Ram
>
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Data replication to PostgreSQL
@ 2016-12-29 15:26 Scott Marlowe <scott.marlowe@gmail.com>
parent: John Scalia <jayknowsunix@gmail.com>
1 sibling, 0 replies; 8+ messages in thread
From: Scott Marlowe @ 2016-12-29 15:26 UTC (permalink / raw)
To: John Scalia <jayknowsunix@gmail.com>; +Cc: Ram DBA <postgresdbas@gmail.com>; pgsql-admin
On Thu, Dec 29, 2016 at 8:12 AM, John Scalia <jayknowsunix@gmail.com> wrote:
> I'm not sure I understand what you're doing. Nearly all DDL is written to
> the WAL segments. The only ones that aren't are things like unlogged and
> temporary tables. I don't think temporary tables used by queries are written
> either. Why do you think, however, that this isn't captured?
Maybe OP is thinking about Slony replication?
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Data replication to PostgreSQL
@ 2016-12-29 15:50 John Scalia <jayknowsunix@gmail.com>
parent: John Scalia <jayknowsunix@gmail.com>
1 sibling, 1 reply; 8+ messages in thread
From: John Scalia @ 2016-12-29 15:50 UTC (permalink / raw)
To: Ram DBA <postgresdbas@gmail.com>; pgsql-admin
Hmmm... I suppose that would depend what your Salesforce db is. This may
not be the appropriate forum for this question, as PostgreSQL could
certainly support all DDL changes. With no experience for Salesforce, I'm
at loss to tell you what you would need.
--
Jay
On Thu, Dec 29, 2016 at 10:46 AM, Ram DBA <postgresdbas@gmail.com> wrote:
> Hello John,
>
> Thank you for the reply. Our requirement is to copy the data from
> Salesforce to our local PostgreSQL database for which we had to rely on 3rd
> party tools. But because of dynamic nature from application, the back-end
> tables in Salesforce are being changed frequently thus stalling the data
> replication to PostgreSQL.
>
> Is there any tool that also capture and replay DDL changes from Salesforce
> to PostgreSQL.
> Hope I am clear with the details now, kindly let me know in case of
> further questions.
>
>
> Thank you in advance,
> Ram
>
> On Thu, Dec 29, 2016 at 8:42 PM, John Scalia <jayknowsunix@gmail.com>
> wrote:
>
>> I'm not sure I understand what you're doing. Nearly all DDL is written to
>> the WAL segments. The only ones that aren't are things like unlogged and
>> temporary tables. I don't think temporary tables used by queries are
>> written either. Why do you think, however, that this isn't captured?
>> --
>> Jay
>>
>> On Thu, Dec 29, 2016 at 9:35 AM, Ram DBA <postgresdbas@gmail.com> wrote:
>>
>>> Hi team,
>>>
>>> We have an use case to copy the data from Salesforce to PostgreSQL
>>> environment but because of DDL changes that can happen to tables from
>>> Salesforce application side, replication would eventually fail. Do we have
>>> any open source tools that can also capture the DDL changes and replicate
>>> to PostgreSQL?
>>>
>>>
>>> Thank you in advance,
>>> Ram
>>>
>>
>>
>
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Data replication to PostgreSQL
@ 2016-12-29 17:16 Jorge Torralba <jorge.torralba@gmail.com>
parent: John Scalia <jayknowsunix@gmail.com>
0 siblings, 1 reply; 8+ messages in thread
From: Jorge Torralba @ 2016-12-29 17:16 UTC (permalink / raw)
To: John Scalia <jayknowsunix@gmail.com>; +Cc: Ram DBA <postgresdbas@gmail.com>; pgsql-admin
Sounds like you are not using streaming replication. Can you confirm your
standby has a recovery.conf file in $PGDATA ? Isn't Saleforce running with
Streaming replication? If you want a proven solution running on Postgres,
consider Jive which runs thousands of implementation on replicated Postgres
without any issues and has been called a much better solution than
Salesforce.
On Thu, Dec 29, 2016 at 7:50 AM, John Scalia <jayknowsunix@gmail.com> wrote:
> Hmmm... I suppose that would depend what your Salesforce db is. This may
> not be the appropriate forum for this question, as PostgreSQL could
> certainly support all DDL changes. With no experience for Salesforce, I'm
> at loss to tell you what you would need.
> --
> Jay
>
> On Thu, Dec 29, 2016 at 10:46 AM, Ram DBA <postgresdbas@gmail.com> wrote:
>
>> Hello John,
>>
>> Thank you for the reply. Our requirement is to copy the data from
>> Salesforce to our local PostgreSQL database for which we had to rely on 3rd
>> party tools. But because of dynamic nature from application, the back-end
>> tables in Salesforce are being changed frequently thus stalling the data
>> replication to PostgreSQL.
>>
>> Is there any tool that also capture and replay DDL changes from
>> Salesforce to PostgreSQL.
>> Hope I am clear with the details now, kindly let me know in case of
>> further questions.
>>
>>
>> Thank you in advance,
>> Ram
>>
>> On Thu, Dec 29, 2016 at 8:42 PM, John Scalia <jayknowsunix@gmail.com>
>> wrote:
>>
>>> I'm not sure I understand what you're doing. Nearly all DDL is written
>>> to the WAL segments. The only ones that aren't are things like unlogged and
>>> temporary tables. I don't think temporary tables used by queries are
>>> written either. Why do you think, however, that this isn't captured?
>>> --
>>> Jay
>>>
>>> On Thu, Dec 29, 2016 at 9:35 AM, Ram DBA <postgresdbas@gmail.com> wrote:
>>>
>>>> Hi team,
>>>>
>>>> We have an use case to copy the data from Salesforce to PostgreSQL
>>>> environment but because of DDL changes that can happen to tables from
>>>> Salesforce application side, replication would eventually fail. Do we have
>>>> any open source tools that can also capture the DDL changes and replicate
>>>> to PostgreSQL?
>>>>
>>>>
>>>> Thank you in advance,
>>>> Ram
>>>>
>>>
>>>
>>
>
--
Thanks,
Jorge Torralba
----------------------------
Note: This communication may contain privileged or other confidential
information. If you are not the intended recipient, please do not print,
copy, retransmit, disseminate or otherwise use the information. Please
indicate to the sender that you have received this email in error and
delete the copy you received. Thank You.
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Data replication to PostgreSQL
@ 2016-12-29 17:28 Scott Mead <scottm@openscg.com>
parent: Ram DBA <postgresdbas@gmail.com>
1 sibling, 1 reply; 8+ messages in thread
From: Scott Mead @ 2016-12-29 17:28 UTC (permalink / raw)
To: Ram DBA <postgresdbas@gmail.com>; +Cc: pgsql-admin
> On Dec 29, 2016, at 09:35, Ram DBA <postgresdbas@gmail.com> wrote:
>
> Hi team,
>
> We have an use case to copy the data from Salesforce to PostgreSQL environment but because of DDL changes that can happen to tables from Salesforce application side, replication would eventually fail. Do we have any open source tools that can also capture the DDL changes and replicate to PostgreSQL?
Are you using an export tool provided by salesforce, or a custom tool against their APIs?
Either way, you don't have access to their database, you should be dumping the data to a CSV and importing the CSV to postgres. It's been a while since I've monkeyed with it, There may be a new way of doing this, but I doubt it. As long as your export tool always creates a csv in the right format, you'll be fine. If you aren't ordering your columns or are using *, you'll never get a good read. Use the toon or API to create a consistent file each time in some standard format and you'll be as okay as possible.
--
Scott Mead via mobile
Skype : scottm.openscg
Gtalk : scottm@openscg.com
>
> Thank you in advance,
> Ram
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Data replication to PostgreSQL
@ 2016-12-30 06:50 Ram DBA <postgresdbas@gmail.com>
parent: Jorge Torralba <jorge.torralba@gmail.com>
0 siblings, 0 replies; 8+ messages in thread
From: Ram DBA @ 2016-12-30 06:50 UTC (permalink / raw)
To: Jorge Torralba <jorge.torralba@gmail.com>; +Cc: John Scalia <jayknowsunix@gmail.com>; pgsql-admin
Hi Jorge,
Thank you for the reply. Our intention is to copy the Salesforce data to
local PostgreSQL database, not the other way around. We are using SOQL api
calls to get the data of tables and replicating it to Postgres.
Thank you,
Ram
On Thu, Dec 29, 2016 at 10:46 PM, Jorge Torralba <jorge.torralba@gmail.com>
wrote:
> Sounds like you are not using streaming replication. Can you confirm your
> standby has a recovery.conf file in $PGDATA ? Isn't Saleforce running with
> Streaming replication? If you want a proven solution running on Postgres,
> consider Jive which runs thousands of implementation on replicated Postgres
> without any issues and has been called a much better solution than
> Salesforce.
>
> On Thu, Dec 29, 2016 at 7:50 AM, John Scalia <jayknowsunix@gmail.com>
> wrote:
>
>> Hmmm... I suppose that would depend what your Salesforce db is. This may
>> not be the appropriate forum for this question, as PostgreSQL could
>> certainly support all DDL changes. With no experience for Salesforce, I'm
>> at loss to tell you what you would need.
>> --
>> Jay
>>
>> On Thu, Dec 29, 2016 at 10:46 AM, Ram DBA <postgresdbas@gmail.com> wrote:
>>
>>> Hello John,
>>>
>>> Thank you for the reply. Our requirement is to copy the data from
>>> Salesforce to our local PostgreSQL database for which we had to rely on 3rd
>>> party tools. But because of dynamic nature from application, the back-end
>>> tables in Salesforce are being changed frequently thus stalling the data
>>> replication to PostgreSQL.
>>>
>>> Is there any tool that also capture and replay DDL changes from
>>> Salesforce to PostgreSQL.
>>> Hope I am clear with the details now, kindly let me know in case of
>>> further questions.
>>>
>>>
>>> Thank you in advance,
>>> Ram
>>>
>>> On Thu, Dec 29, 2016 at 8:42 PM, John Scalia <jayknowsunix@gmail.com>
>>> wrote:
>>>
>>>> I'm not sure I understand what you're doing. Nearly all DDL is written
>>>> to the WAL segments. The only ones that aren't are things like unlogged and
>>>> temporary tables. I don't think temporary tables used by queries are
>>>> written either. Why do you think, however, that this isn't captured?
>>>> --
>>>> Jay
>>>>
>>>> On Thu, Dec 29, 2016 at 9:35 AM, Ram DBA <postgresdbas@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi team,
>>>>>
>>>>> We have an use case to copy the data from Salesforce to PostgreSQL
>>>>> environment but because of DDL changes that can happen to tables from
>>>>> Salesforce application side, replication would eventually fail. Do we have
>>>>> any open source tools that can also capture the DDL changes and replicate
>>>>> to PostgreSQL?
>>>>>
>>>>>
>>>>> Thank you in advance,
>>>>> Ram
>>>>>
>>>>
>>>>
>>>
>>
>
>
> --
> Thanks,
>
> Jorge Torralba
> ----------------------------
>
> Note: This communication may contain privileged or other confidential
> information. If you are not the intended recipient, please do not print,
> copy, retransmit, disseminate or otherwise use the information. Please
> indicate to the sender that you have received this email in error and
> delete the copy you received. Thank You.
>
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Data replication to PostgreSQL
@ 2016-12-30 06:54 Ram DBA <postgresdbas@gmail.com>
parent: Scott Mead <scottm@openscg.com>
0 siblings, 0 replies; 8+ messages in thread
From: Ram DBA @ 2016-12-30 06:54 UTC (permalink / raw)
To: Scott Mead <scottm@openscg.com>; +Cc: pgsql-admin
Hello Scott,
Thank you for the inputs. We are using SOQL api calls to get the data from
Salesforce back-end tables and replicating it to Postgres. One challenge we
face is with metadata changes to Salesforce tables. They keep dynamically
changing with the business requirements. Do we have any process to capture
the DDL changes and replicate it Postgres?
Also, salesforce API calls does not support "SELECT * FROM <table>".
Instead, we had to mention each field/column and select the data.
Thank you,
Ram
On Thu, Dec 29, 2016 at 10:58 PM, Scott Mead <scottm@openscg.com> wrote:
>
> > On Dec 29, 2016, at 09:35, Ram DBA <postgresdbas@gmail.com> wrote:
> >
> > Hi team,
> >
> > We have an use case to copy the data from Salesforce to PostgreSQL
> environment but because of DDL changes that can happen to tables from
> Salesforce application side, replication would eventually fail. Do we have
> any open source tools that can also capture the DDL changes and replicate
> to PostgreSQL?
>
> Are you using an export tool provided by salesforce, or a custom tool
> against their APIs?
>
> Either way, you don't have access to their database, you should be
> dumping the data to a CSV and importing the CSV to postgres. It's been a
> while since I've monkeyed with it, There may be a new way of doing this,
> but I doubt it. As long as your export tool always creates a csv in the
> right format, you'll be fine. If you aren't ordering your columns or are
> using *, you'll never get a good read. Use the toon or API to create a
> consistent file each time in some standard format and you'll be as okay as
> possible.
>
> --
> Scott Mead via mobile
> Skype : scottm.openscg
> Gtalk : scottm@openscg.com
>
> >
> > Thank you in advance,
> > Ram
>
^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2016-12-30 06:54 UTC | newest]
Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2016-12-29 14:35 Data replication to PostgreSQL Ram DBA <postgresdbas@gmail.com>
2016-12-29 15:12 ` John Scalia <jayknowsunix@gmail.com>
2016-12-29 15:26 ` Scott Marlowe <scott.marlowe@gmail.com>
2016-12-29 15:50 ` John Scalia <jayknowsunix@gmail.com>
2016-12-29 17:16 ` Jorge Torralba <jorge.torralba@gmail.com>
2016-12-30 06:50 ` Ram DBA <postgresdbas@gmail.com>
2016-12-29 17:28 ` Scott Mead <scottm@openscg.com>
2016-12-30 06:54 ` Ram DBA <postgresdbas@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox