pg.ddx.io pgsql-interfaces@postgresql.org mailing list archive
help / color / mirror / Atom feedC set return function differences on 8.0?
7+ messages / 3 participants
[nested] [flat]
* C set return function differences on 8.0?
@ 2005-07-08 00:58 Tim Jackson <tim.jackson@ints.com>
0 siblings, 1 reply; 7+ messages in thread
From: Tim Jackson @ 2005-07-08 00:58 UTC (permalink / raw)
To: pgsql-interfaces
I have a C .so set return function which is called to create a views,
pulling data from an older non SQL flat file data structure
CREATE OR REPLACE VIEW view_name AS
SELECT call_d.col_0, call_d.col_1, call_d.col_2, call_d.col_3"
FROM call_d('select * from aname '::text, 'aname'::text) AS (col_0
bigint, col_2 numeric, col_2 text, col_3 text);
The call_d function calls the C .so passing in the SQL and the target
and returns a record set conforming to the request, the column names and
data types are supplied for each target/view
This worked compiled under postgresql 7.4 for all the target files and
various data types that I have.
I re-compiled the .so using 8.0 server development source and rebuilt
the views on 8.0 and now only some views will pull up data. These are
views using the same data that worked under 7.4 and now just abort.
Some of the views do select.
Could the problem be related to a difference in the way 8.0 is handling
data types?
When I define a column as numeric with out precision or scale it should
handle any numeric value, however in a data set that has a numeric field
with mostly XX.XX data one record had a XX.XXX value and once I removed
that record from the set the view selected ok.
This doesn't make sense to me.
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: C set return function differences on 8.0?
@ 2005-07-08 15:23 Michael Fuhr <mike@fuhr.org>
parent: Tim Jackson <tim.jackson@ints.com>
0 siblings, 1 reply; 7+ messages in thread
From: Michael Fuhr @ 2005-07-08 15:23 UTC (permalink / raw)
To: Tim Jackson <tim.jackson@ints.com>; +Cc: pgsql-interfaces
On Thu, Jul 07, 2005 at 05:58:28PM -0700, Tim Jackson wrote:
>
> I re-compiled the .so using 8.0 server development source and rebuilt
> the views on 8.0 and now only some views will pull up data. These are
> views using the same data that worked under 7.4 and now just abort.
What do you mean by "abort" -- does the backend crash? If so, and
if you get a core dump, then it might be useful to see a stack trace.
Could you post a simple, self-contained test case that works in 7.4
and fails in 8.0? That is, all SQL, C code, and data that somebody
could use to reproduce the problem from an empty database. Please
do distill the example so it's as simple as possible -- that way
the analysis won't be distracted by irrelevant factors.
Exactly which versions of 7.4 and 8.0 are you running, and on what
platform?
--
Michael Fuhr
http://www.fuhr.org/~mfuhr/
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: C set return function differences on 8.0?
@ 2005-07-09 01:58 Michael Fuhr <mike@fuhr.org>
parent: Michael Fuhr <mike@fuhr.org>
0 siblings, 2 replies; 7+ messages in thread
From: Michael Fuhr @ 2005-07-09 01:58 UTC (permalink / raw)
To: Tim Jackson <tim.jackson@ints.com>; +Cc: pgsql-interfaces
[Please copy the mailing list on replies so others can contribute
to and learn from the discussion.]
On Fri, Jul 08, 2005 at 05:12:58PM -0700, Tim Jackson wrote:
> Michael Fuhr wrote:
> > What do you mean by "abort" -- does the backend crash? If so, and
> > if you get a core dump, then it might be useful to see a stack trace.
> >
> The backend crashes, in pgadmin attempting to display the view simple
> says aborting. There is no core dump, and the log only seems to reflect
> the standard
> WARNING: terminating connection because of crash of another server process
Where did you look for a core dump? If one was made then it'll
probably be somewhere under $PGDATA (e.g., $PGDATA/base/XXX/core)
unless your system is configured to put core dumps elsewhere.
If there isn't a core dump then you could add some debugging ereport()
calls to your code so you can find out where the crash is happening.
Another possibility might be to attach a debugger to the backend.
> >Could you post a simple, self-contained test case that works in 7.4
> >and fails in 8.0? That is, all SQL, C code, and data that somebody
> >
> The problem I have is we are using a licensed library to accesss the
> target database. We have c code which parses SQLSRF and makes a call to
> these 3rd party lib functions which returns strings of data.
So your C function calls these library functions, which query some
other data source and return strings back to you, right? How are
these strings returned -- as char * values? Can you at least post
your code?
> Today we tried to just hard code some data strings and pass that into
> SRF part of the code. We put in values that we expected to cause the
> problem and it did not cause the problem. So we are stuck with how to
> proceed.
Let's see if attaching a debugger to the backend or adding some
ereport() calls can at least tell us where the crash is happening.
Then maybe we can figure out why.
--
Michael Fuhr
http://www.fuhr.org/~mfuhr/
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: C set return function differences on 8.0?
@ 2005-07-09 02:13 Tom Lane <tgl@sss.pgh.pa.us>
parent: Michael Fuhr <mike@fuhr.org>
1 sibling, 0 replies; 7+ messages in thread
From: Tom Lane @ 2005-07-09 02:13 UTC (permalink / raw)
To: Michael Fuhr <mike@fuhr.org>; +Cc: Tim Jackson <tim.jackson@ints.com>; pgsql-interfaces
Michael Fuhr <mike@fuhr.org> writes:
> On Fri, Jul 08, 2005 at 05:12:58PM -0700, Tim Jackson wrote:
>> The backend crashes, in pgadmin attempting to display the view simple
>> says aborting. There is no core dump, and the log only seems to reflect
>> the standard
>> WARNING: terminating connection because of crash of another server process
> Where did you look for a core dump?
"There is no core dump" is not an acceptable answer: if there isn't one,
the first item on your agenda must be to get one (or else run the
problem case under gdb so you don't need a dump to get a stack trace).
Usually if there's no dump it's because the default configuration on
your system is "ulimit -c 0" to suppress core dumps from daemons.
Put "ulimit -c unlimited" (or local equivalent) into your postmaster
start script and restart the postmaster.
On the whole though I'd recommend learning how to attach gdb to a live
backend, since then not only can you get a stack trace from the point of
the fault, but you can then work backwards by setting breakpoints ahead
of the crash and stepping through the code to see where it goes wrong.
regards, tom lane
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: C set return function differences on 8.0?]]
@ 2005-07-11 21:37 Tim Jackson <tim.jackson@ints.com>
0 siblings, 1 reply; 7+ messages in thread
From: Tim Jackson @ 2005-07-11 21:37 UTC (permalink / raw)
To: pgsql-interfaces
Michael Fuhr wrote:
>Where did you look for a core dump? If one was made then it'll
>probably be somewhere under $PGDATA (e.g., $PGDATA/base/XXX/core)
>unless your system is configured to put core dumps elsewhere.
>
>
>
I was able to get the core dump by adding 'ulimit -c unlimited' to the
start script
(thanks Tom)
It is huge (14 megs)
One interesting thing happened once the core dump is created.
Now if I re connect to the data base and rerun the select on the view
after it has crashed once it will select the view.
>If there isn't a core dump then you could add some debugging ereport()
>calls to your code so you can find out where the crash is happening.
>Another possibility might be to attach a debugger to the backend.
>
>
>
If you can tell me how to do that. I tried attaching gdb to the running
servers pid and it seems to lock up the server.
>So your C function calls these library functions, which query some
>other data source and return strings back to you, right? How are
>these strings returned -- as char * values? Can you at least post
>your code?
>
>
>
The full so code is 4000 lines and caused the news group to reject the mail
, but here is the basic format with the piece that builds the record replaced
by hardcoded values.
/*******************************/
//
// Program: call_good.c
//
// Listing Date: 07/08/2005
//
/*******************************/
#include "server/postgres.h"
#include "server/fmgr.h"
#include "server/funcapi.h"
#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>
#include <string.h>
#include <ctype.h>
#include <time.h>
#define TRUE 1
#define FALSE 0
#define OK 0
#define ERROR1 (-1)
#define YES 1
#define NO 0
#define DATA1 2
#define MAYBE 2
char record[1024 * 1024];
char buffer[256];
unsigned char * bufptr = (unsigned char *)buffer;
char buffer1[256];
unsigned char * bufptr1 = (unsigned char *)buffer1;
char buffer2[256];
unsigned char * bufptr2 = (unsigned char *)buffer2;
char buffer3[256];
unsigned char * bufptr3 = (unsigned char *)buffer3;
int count, count1;
long CurrentSize;
char request[8192];
char response[512];
char typedefin[1024];
int max_fields;
int field_lengths[256];
char field_types[256][16];
char *record_add;
char parsed[256];
char *select_add;
char *from_add;
char *where_add;
char *end_add;
char select1[8192];
char from[512];
char where[512];
FILE *stream;
FILE *stream1;
char filein[128];
char fileout[128];
char filename_out[128];
int num_targets_work;
int num_targets_index;
int chunk_amount = 5;
int chunk_counter;
int fileOpen;
FuncCallContext *funcctx;
int call_cntr;
int max_calls;
TupleDesc tupdesc;
TupleTableSlot *slot;
AttInMetadata *attinmeta;
/***********************************/
// select_more_data
/***********************************/
int tselect_more_data (void)
{
int recordFound;
memset (record, 0, sizeof(record));
sprintf (buffer1, "%s|%s|%s|%s|", "6","6666.01","6-6 Wide","Y");
strcat (record, buffer1);
sprintf (buffer1, "%s|%s|%s|%s|", "7","7777.01","7-6 Wide","Y");
strcat (record, buffer1);
sprintf (buffer1, "%s|%s|%s|%s|", "8","888888.0001","8-6 Wide","Y");
strcat (record, buffer1);
sprintf (buffer1, "%s|%s|%s|%s|", "9","9999.01","9-6 Wide","Y");
strcat (record, buffer1);
sprintf (buffer1, "%s|%s|%s|%s|", "10","1010.01","10-6 Wide","Y");
strcat (record, buffer1);
count = 5;
if (count)
recordFound = DATA1;
return (recordFound);
}
/***********************************/
// select_data
/***********************************/
int tselect_data (void)
{
int recordFound;
memset (record, 0, sizeof(record));
sprintf (buffer1, "%s|%s|%s|%s|", "1","1111.01","1-6 Wide","Y");
strcat (record, buffer1);
sprintf (buffer1, "%s|%s|%s|%s|", "2","2222.01","2-6 Wide","Y");
strcat (record, buffer1);
sprintf (buffer1, "%s|%s|%s|%s|", "3","3333.01","3-6 Wide","Y");
strcat (record, buffer1);
sprintf (buffer1, "%s|%s|%s|%s|", "4"," 4444.01","4-6 Wide","Y");
strcat (record, buffer1);
sprintf (buffer1, "%s|%s|%s|%s|", "5","5555.01","5-8 Wide","Y");
strcat (record, buffer1);
count = 5;
if (count)
recordFound = DATA1;
return (recordFound);
}
/***********************************/
// select_field_lengths
/***********************************/
int tselect_field_lengths (void)
{
int recordFound;
int len;
CurrentSize = 10;
len = 3;
field_lengths[0] = len * 2 + 1;
strcpy (field_types[0], "numeric");
len = 4;
field_lengths[1] = len * 2 + 1;
strcpy (field_types[1], "numeric");
len = 30;
field_lengths[2] = len;
strcpy (field_types[2], "string");
len = 1;
field_lengths[3] = len;
strcpy (field_types[3], "string");
max_fields = 4;
recordFound = DATA1;
return (recordFound);
}
/***********************************/
// parse_record
/***********************************/
void tparse_record (void)
{
char *ptr;
char *ptr1;
ptr = record_add;
ptr1 = strchr (ptr, '|');
if (ptr1)
{
*ptr1 = 0;
strcpy (parsed,ptr);
}
record_add = ++ptr1;
}
/***********************************/
// parse_sql
/***********************************/
int tparse_sql (char * request)
{
int func;
select_add =
from_add =
where_add =
end_add = 0L;
memset ( select1, 0, sizeof(select1));
memset ( from, 0, sizeof(from));
memset ( where, 0, sizeof(where));
select_add = strstr(request, "SELECT ");
if (select_add == 0)
select_add = strstr(request, "select ");
if (select_add != 0)
{
from_add = strstr(request, "FROM ");
if (from_add == 0)
from_add = strstr(request, "from ");
}
where_add = strstr(request, "WHERE ");
if (where_add == 0)
where_add = strstr(request, "where ");
if (select_add != 0)
{
end_add = select_add+strlen(request);
func = 40;
}
else
{
func = 999;
return func;
}
if (select_add != 0)
{
memcpy ( select1, select_add, from_add-select_add);
if (where_add == 0)
{
memcpy ( from, from_add, end_add-from_add);
}
else if (where_add)
{
memcpy ( from, from_add, where_add-from_add);
memcpy ( where, where_add, end_add-where_add);
}
}
return func;
}
/************************************/
PG_FUNCTION_INFO_V1 (call_good);
Datum
call_good (PG_FUNCTION_ARGS)
{
int i, func, retval;
// stuff done only on the first call of the function
if (SRF_IS_FIRSTCALL())
{
MemoryContext oldcontext;
text *t = PG_GETARG_TEXT_P(0);
text *t1 = PG_GETARG_TEXT_P(1);
memset (request, 0, sizeof(request));
strncpy (request, t->vl_dat, t->vl_len- VARHDRSZ);
memset (typedefin, 0, sizeof(typedefin));
strncpy (typedefin, t1->vl_dat, t1->vl_len- VARHDRSZ);
// create a function context for cross-call persistence
funcctx = SRF_FIRSTCALL_INIT();
// switch to memory context appropriate for multiple function calls
oldcontext = MemoryContextSwitchTo(funcctx->multi_call_memory_ctx);
// Build a tuple description for a incoming type tuple
tupdesc = RelationNameGetTupleDesc(typedefin);
// allocate a slot for a tulpe with this tupdesc
slot = TupleDescGetSlot(tupdesc);
// assign slot to function context
funcctx->slot = slot;
// Generate attribute metadata needed later to produce
// tupples from raw C strings
attinmeta = TupleDescGetAttInMetadata(tupdesc);
funcctx->attinmeta = attinmeta;
MemoryContextSwitchTo(oldcontext);
strcpy (fileout, "pfxG");
sprintf (filename_out, "/tmp/%s.rsp", fileout); // rf
if ((stream1 = fopen (filename_out, "wb")) == NULL)
{
return 1;
}
if ((stream = fopen ("/tmp/call_good.log", "a+")) == NULL)
{
return 1;
}
fprintf (stream1,"Call_Good\n");
fprintf (stream1,"request>%s<\n",request);
fprintf (stream1,"file: %s\n",typedefin);
func = tparse_sql (request);
record_add = 0;
tselect_field_lengths();
fprintf (stream,"NumRecs = %ld NumFields = %d\n",
CurrentSize, max_fields);
for (i=0; i < max_fields; i++)
{
fprintf (stream1,"Field: %3d Type: %-8s Length: %d\n",
i,field_types[i],field_lengths[i]);
}
switch (func)
{
case 40:
if (select_add)
{
fprintf (stream1, "select: %s\n",select1);
fprintf (stream1, "from: %s\n",from);
if (where_add)
{
fprintf (stream1, "where: %s\n",where);
}
}
funcctx->max_calls = CurrentSize;
// Go get first chunk of records from file
retval = tselect_data();
if (retval != DATA1)
{
strcpy (record, "Not Found\n");
fwrite (record, strlen(record), 1, stream1);
}
break;
case 999:
strcpy (record, "Invalid Statement");
fwrite (record, strlen(record), 1, stream1);
break;
default:
strcpy (record, "Invalid");
fwrite (record, strlen(record), 1, stream1);
retval = NO;
break;
}
fprintf (stream,"Count = %ld Data = %.50s\n",CurrentSize,record);
fprintf (stream,"File: %s Type: %s\n",filein,typedefin);
record_add = record;
chunk_counter = chunk_amount;
fflush (stream);
}
// stuff done on every call of the function
funcctx = SRF_PERCALL_SETUP();
call_cntr = funcctx->call_cntr;
max_calls = funcctx->max_calls;
slot = funcctx->slot;
attinmeta = funcctx->attinmeta;
if (call_cntr < max_calls)
{
char **values;
HeapTuple tuple;
Datum result;
if (call_cntr == chunk_counter)
{
// Go get next chunk of records from file
retval = tselect_more_data();
record_add = record;
chunk_counter += chunk_amount;
}
// Prepare a values array for storage in our slot.
// This should be an array of C strings which will
// be processed later by the appropriate "in" functions.
values = (char **) palloc(max_fields * sizeof(char *));
for (i=0; i < max_fields; i++)
{
values[i] = (char *) palloc(field_lengths[i] * sizeof(char));
}
for (i=0; i < max_fields; i++)
{
tparse_record();
if (strlen(parsed) == 0 && strcmp(field_types[i],"date") == 0)
{
strcpy (values[i], "01/01/1980");
}
else
{
strcpy (values[i], parsed);
}
}
// build a tuple
tuple = BuildTupleFromCStrings(attinmeta, values);
// make the tuple into a datum
result = TupleGetDatum(slot,tuple);
// result = HeapTupleGetDatum(tuple);
// Clean up (this is not actually necessary)
for (i=0; i < max_fields; i++)
{
pfree(values[i]);
}
pfree(values);
fflush (stream);
SRF_RETURN_NEXT(funcctx, result);
}
else // do when there is no more left
{
if (fileOpen)
{
fprintf (stream, "EOD Set5\n");
fileOpen = 0;
}
fprintf (stream1,"DONE: call_cntr= %d max_calls= %d\n",
call_cntr,max_calls);
fclose (stream1);
fclose (stream);
SRF_RETURN_DONE(funcctx);
}
}
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: C set return function differences on 8.0?
@ 2005-07-11 23:09 Tim Jackson <tim.jackson@ints.com>
parent: Michael Fuhr <mike@fuhr.org>
1 sibling, 0 replies; 7+ messages in thread
From: Tim Jackson @ 2005-07-11 23:09 UTC (permalink / raw)
To: Michael Fuhr <mike@fuhr.org>; +Cc: pgsql-interfaces
Michael Fuhr wrote:
>
>Let's see if attaching a debugger to the backend or adding some
>ereport() calls can at least tell us where the crash is happening.
>Then maybe we can figure out why.
>
>
I was able to set gdb to the pid of psql session and saw that selects
on the "bad" views segfaulted on pfree()
The documentation for SRF had a comment that said pfree() was not really
necessary so I commented that code out and now I can select all the
views. :)
Perhaps you can advise me on why pfree() would segfault on 8.0 in but
not 7.4
Anyway the .so generated views are working now as they were on 7.4.
Thanks for your help. Tim
Not sure what all this means but the following lines were gleaned from
the core dump
cannot allocate memory for thread-local data: ABORT
result == _rtld_local._dl_tls_max_dtv_idx + 1
result <= _rtld_local._dl_tls_max_dtv_idx + 1
@cnt < _rtld_local._dl_tls_dtv_slotinfo_list->len
_rtld_local._dl_tls_dtv_slotinfo_list->next == ((void *)0)
_rtld_local._dl_tls_dtv_slotinfo_list != ((void *)0)
_rtld_local._dl_tls_max_dtv_idx == 0
_rtld_local._dl_tls_dtv_slotinfo_list == ((void *)0)
(size_t) map->l_tls_offset >= map->l_tls_blocksize
map->l_tls_blocksize >= map->l_tls_initimage_size
../sysdeps/unix/sysv/linux/dl-origin.c
FATAL: cannot determine library version
../sysdeps/generic/dl-sysdep.c
Inconsistency detected by ld.so: %s: %u: %s%sAssertion `%s' failed!
Inconsistency detected by ld.so: %s: %u: %s%sUnexpected error: %s.
And Here is what gdb says about the core
Core was generated by `postgres: postgres lubesoft_tj [local]
SELECT '.
Program terminated with signal 11, Segmentation fault.
#0 0x08222a86 in ?? ()
(no debugging symbols found)...Using host libthread_db library
"/lib/tls/libthread_db.so.1".
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: C set return function differences on 8.0?]]
@ 2005-07-12 14:31 Tom Lane <tgl@sss.pgh.pa.us>
parent: Tim Jackson <tim.jackson@ints.com>
0 siblings, 0 replies; 7+ messages in thread
From: Tom Lane @ 2005-07-12 14:31 UTC (permalink / raw)
To: Tim Jackson <tim.jackson@ints.com>; +Cc: pgsql-interfaces
Tim Jackson <tim.jackson@ints.com> writes:
> The full so code is 4000 lines and caused the news group to reject the mail
> , but here is the basic format with the piece that builds the record replaced
> by hardcoded values.
With all those unchecked strcat's and strcpy's going on, it seems like
an excellent bet that you're overrunning one or another buffer and
clobbering memory that doesn't belong to you. Compiling the backend
with --enable-cassert might help in localizing the problem.
regards, tom lane
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2005-07-12 14:31 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2005-07-08 00:58 C set return function differences on 8.0? Tim Jackson <tim.jackson@ints.com>
2005-07-08 15:23 ` Michael Fuhr <mike@fuhr.org>
2005-07-09 01:58 ` Michael Fuhr <mike@fuhr.org>
2005-07-09 02:13 ` Tom Lane <tgl@sss.pgh.pa.us>
2005-07-11 23:09 ` Tim Jackson <tim.jackson@ints.com>
2005-07-11 21:37 Re: C set return function differences on 8.0?]] Tim Jackson <tim.jackson@ints.com>
2005-07-12 14:31 ` Re: C set return function differences on 8.0?]] Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox