Subject: Core dump when copy'ing large file

From: [email-masked] (Suresh Srinivasan)
Date: Mon, 7 Mar 94 10:26:22 EST

Hi Postgres'ers:

I'm using the precompiled Sparc binary version 4.1 of
Postgres to index a largish image database.

There are about 60,000+ images in the collection with
cataloging information associated with each.  I'm trying
to create a word index on this catalog using the following:

  create Word (word = text, pos = int4, field = char, frame = int4) \g
  define index WordIndex on Word using btree (word text_ops) \g

Then, I load in the words from a pre-created file with:

  copy Word from "/foo/bar/Word.cl" \g

Note that even after removing stopwords, this is a large file
with well over 500,000 entries.
Also, there are large numbers of words that are repeated
several thousand times.  For example, here's the head of
the (reverse sorted) frequency file:

13254 hospital
8104 army
7774 full
7577 face
7487 head
7197 view
6744 shoulders
5043 pose
4627 france
4048 left
...

Well, after running for about 5 hrs on a Sparc 2, I get a
core dump (?() at ab058 using adb) with the following message
on the console:

  Error: No response from the backend, exiting..

The Word file in ~postgres/data/base/db when it died is 37M and the
WordIndex file is 20M.  All retrievals on Word return empty.
Before attempting another solution, I was hoping some of you
have seen this before.

Thanks in advance for any help..

--Suresh

---

From: [email-masked] (Paul M. Aoki)
Date: Wed, 23 Mar 94 18:30:11 -0800

> core dump (?() at ab058 using adb) with the following message

i'm not positive this is your problem, but..

postgres has problems with memory leaks when you start working with 
many rows.  the fundamental problem is that there are data structures 
that get created per-row that don't get freed until end-of-transaction.
for example, people have noted that single-transaction inserts of 
several thousand rows fail.  sunita sarawagi tried to track down as 
many of these leaks as she could, but they are pervasive.

the codepaths less travelled (insert, copy, etc.) seem to have more 
problems with this than some others (retrieve).

if this is the problem (malloc is returning NULL because you're out of 
memory) the only fixes i can suggest are to break up the load into 
several chunks or allocate a lot more swap space.

if you are trying to copy *out* and having this problem, one should
be able to break a table up using "retrieve into" (creating a set of 
temporary tables) and then "copy out" from the temporary tables.
--
  Paul M. Aoki  |  CS Div., Dept. of EECS, UCB  |  [email-masked]
                |  Berkeley, CA 94720           |  ...!uunet!ucbvax!aoki