pg.ddx.io pgsql-sql@postgresql.org mailing list archive
help / color / mirror / Atom feedreading WAL files in python
5+ messages / 4 participants
[nested] [flat]
* reading WAL files in python
@ 2008-01-07 13:19 Gerardo Herzig <gherzig@fmed.uba.ar>
0 siblings, 2 replies; 5+ messages in thread
From: Gerardo Herzig @ 2008-01-07 13:19 UTC (permalink / raw)
To: pgsql-sql
Hi all. Im having some fun trying to write my own replication system
using python. I will use the postgres own WAL archiving to write the
files, then my app will read them and do some stuff. As im not a C
programmer, im stuck in the mission of reading the binary files.
I guess im needing to know the internals of how wals archives are, how
big the buffer has to be, in order to have a complete sql command.
Can someone point some advice?
I was trying to understad src/backend/access/transam/xlog.c, but seems
too difficult to me :(
Thanks!
Gerardo
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: reading WAL files in python
@ 2008-01-07 14:05 C. Bergström <cbergstrom@netsyncro.com>
parent: Gerardo Herzig <gherzig@fmed.uba.ar>
1 sibling, 1 reply; 5+ messages in thread
From: "C." Bergström @ 2008-01-07 14:05 UTC (permalink / raw)
To: pgsql-sql
On Mon, 2008-01-07 at 10:19 -0300, Gerardo Herzig wrote:
> Hi all. Im having some fun trying to write my own replication system
> using python. I will use the postgres own WAL archiving to write the
> files, then my app will read them and do some stuff. As im not a C
> programmer, im stuck in the mission of reading the binary files.
>
> I guess im needing to know the internals of how wals archives are, how
> big the buffer has to be, in order to have a complete sql command.
>
> Can someone point some advice?
> I was trying to understad src/backend/access/transam/xlog.c, but seems
> too difficult to me :(
>
imho you can save yourself a lot of time by taking a look at skytools[1]
as it may be more along the lines of what you're trying to accomplish
anyway.
Cheers,
./C
[1] https://developer.skype.com/SkypeGarage/DbProjects/SkyTools
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: reading WAL files in python
@ 2008-01-07 14:26 Gerardo Herzig <gherzig@fmed.uba.ar>
parent: C. Bergström <cbergstrom@netsyncro.com>
0 siblings, 0 replies; 5+ messages in thread
From: Gerardo Herzig @ 2008-01-07 14:26 UTC (permalink / raw)
To: pgsql-sql
C. Bergström wrote:
>On Mon, 2008-01-07 at 10:19 -0300, Gerardo Herzig wrote:
>
>
>>Hi all. Im having some fun trying to write my own replication system
>>using python. I will use the postgres own WAL archiving to write the
>>files, then my app will read them and do some stuff. As im not a C
>>programmer, im stuck in the mission of reading the binary files.
>>
>>I guess im needing to know the internals of how wals archives are, how
>>big the buffer has to be, in order to have a complete sql command.
>>
>>Can someone point some advice?
>>I was trying to understad src/backend/access/transam/xlog.c, but seems
>>too difficult to me :(
>>
>>
>>
>
>imho you can save yourself a lot of time by taking a look at skytools[1]
>as it may be more along the lines of what you're trying to accomplish
>anyway.
>
>Cheers,
>
>./C
>
>[1] https://developer.skype.com/SkypeGarage/DbProjects/SkyTools
>
>
>
>
Oh yes, you already told me about skytools. But im also trying to learn
more about postgres and python, and this seems to a great oportunity to
do that. I dont mind if im reinventing the wheel, i usually have fun
doing it.
Thanks!
Gerardo
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: reading WAL files in python
@ 2008-01-07 18:53 Erik Jones <erik@myemma.com>
parent: Gerardo Herzig <gherzig@fmed.uba.ar>
1 sibling, 1 reply; 5+ messages in thread
From: Erik Jones @ 2008-01-07 18:53 UTC (permalink / raw)
To: Gerardo Herzig <gherzig@fmed.uba.ar>; +Cc: pgsql-sql
On Jan 7, 2008, at 7:19 AM, Gerardo Herzig wrote:
> Hi all. Im having some fun trying to write my own replication
> system using python. I will use the postgres own WAL archiving to
> write the files, then my app will read them and do some stuff. As
> im not a C programmer, im stuck in the mission of reading the
> binary files.
>
> I guess im needing to know the internals of how wals archives are,
> how big the buffer has to be, in order to have a complete sql command.
>
> Can someone point some advice?
> I was trying to understad src/backend/access/transam/xlog.c, but
> seems too difficult to me :(
The biggest problem with what you're wanting to do here is that the
data written to the WALL archives isn't going to be at all easily
translatable into SQL statements. AFAIK, the WAL data records on-
disk file changes, i.e. it's strictly a binary mode scenario. As
someone has already suggested, if you want to learn more about
Postgres and Python, look at Skytools. I'm not just saying to use
it, read the code and, if you like, offer help with patches.
Erik Jones
DBA | Emma®
erik@myemma.com
800.595.4401 or 615.292.5888
615.292.0777 (fax)
Emma helps organizations everywhere communicate & market in style.
Visit us online at http://www.myemma.com
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: reading WAL files in python
@ 2008-01-07 20:47 gherzig@fmed.uba.ar
parent: Erik Jones <erik@myemma.com>
0 siblings, 0 replies; 5+ messages in thread
From: gherzig@fmed.uba.ar @ 2008-01-07 20:47 UTC (permalink / raw)
To: pgsql-sql
>
> On Jan 7, 2008, at 7:19 AM, Gerardo Herzig wrote:
>
>> Hi all. Im having some fun trying to write my own replication
>> system using python. I will use the postgres own WAL archiving to
>> write the files, then my app will read them and do some stuff. As
>> im not a C programmer, im stuck in the mission of reading the
>> binary files.
>>
>> I guess im needing to know the internals of how wals archives are,
>> how big the buffer has to be, in order to have a complete sql command.
>>
>> Can someone point some advice?
>> I was trying to understad src/backend/access/transam/xlog.c, but
>> seems too difficult to me :(
>
> The biggest problem with what you're wanting to do here is that the
> data written to the WALL archives isn't going to be at all easily
> translatable into SQL statements. AFAIK, the WAL data records on-
> disk file changes, i.e. it's strictly a binary mode scenario. As
> someone has already suggested, if you want to learn more about
> Postgres and Python, look at Skytools. I'm not just saying to use
> it, read the code and, if you like, offer help with patches.
>
> Erik Jones
So the fun ends in 3, 2, 1....
Well, so it looks like the trigger approach is now my path to follow.
Yeah, im looking for skytools code, looks very nice, maybe a little
outdate (just a quick look, it uses old-style python classes).
BTW, the information about WAL system you gave me, just saves me a lot of
time. Thanks you for that!
Gerardo
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2008-01-07 20:47 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2008-01-07 13:19 reading WAL files in python Gerardo Herzig <gherzig@fmed.uba.ar>
2008-01-07 14:05 ` C. Bergström <cbergstrom@netsyncro.com>
2008-01-07 14:26 ` Gerardo Herzig <gherzig@fmed.uba.ar>
2008-01-07 18:53 ` Erik Jones <erik@myemma.com>
2008-01-07 20:47 ` gherzig@fmed.uba.ar
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox