Page Moved
See the Python Package Index for information about RelStorage:
Comments
The main place to discuss RelStorage is the ZODB-Dev mailing list, but comments attached to this wiki will be noticed occasionally.
PGStorage migration? --ksmith, Wed, 30 Jan 2008 18:23:37 +0000 reply
I have a large database in PGStorage, is there a migration path to RelStorage, or should it "just work"?
PGStorage migration --Shane Hathaway, Tue, 05 Feb 2008 08:49:16 +0000 reply
A schema change will be required, but it should be simple. Tell me which version of PGStorage you're migrating from and I'll tell you what to do.
PGStorage migration --ksmith, Tue, 05 Feb 2008 18:07:27 +0000 reply
Thanks for your reply. I'm not quite sure, I grabbed it from CVS, but frankly my CVS skills are now non-existant since moving to svn and bzr. pgstorage.py shows working revision 1.6, scripts.py 1.3 and pgconnection.py 1.3, hope that's enough to go on.
Data migration from Data.fs to Oracle --patty123, Thu, 07 Feb 2008 13:27:37 +0000 reply
I am done with cx_oracle and ZCXOracle? installation. Now I want to migrate my existing data in Data.fs to Oracle DB. Can anyone help me out in doing that.
PostgreSQL user defined "python pickle data type" --JerryS?, Thu, 28 Feb 2008 16:11:53 +0000 reply
Just wanted to point this out, in case you hadn't seen it:
http://blog.melhase.net/articles/2006/06/06/a-tale-of-postgresql-types-and-python
Migrating from Data.fs --Shane Hathaway, Fri, 29 Feb 2008 07:03:32 +0000 reply
patty123, use the ZODBConvert utility, now included with RelStorage.
pickle data type --Shane Hathaway, Fri, 29 Feb 2008 07:06:45 +0000 reply
JerryS?, yeah it's kind of cool that you can define a pickle type, but I haven't yet seen a reason why I would want to. :-)
Re: pickle data type --JerryS?, Tue, 04 Mar 2008 22:50:27 +0000 reply
In response to "kind of cool that you can define a pickle type, but I haven't yet seen a reason why I would want to."
See 2nd "Q" under "Probable FAQs?", above.
Re. Re: pickle data type --JerryS?, Fri, 07 Mar 2008 16:26:45 +0000 reply
I'm guilty of committing my own worst "pet peeve", posting a cryptic remark, as "an exercise for the reader". The point of a pickle data type is to treat code as data. If documents are data, code certainly is. Python has been endowed with very high level capabilities, approaching the realm of lambda calculus. The ultimate in code reuse may be revealed when we arrive at an effective means of reasoning about the logic represented in code, and recognizing patterns of usage. Storing code in a relational DB sounds like a great place to start.
Last "pickle data type", I promise --JerryS?, Fri, 07 Mar 2008 17:21:46 +0000 reply
In the article, "Functional Programming for the Rest of Us", after showing in lucid examples how functional programs can readily be proved to be correct, validated via unit tests, and updated / optimized on the fly, the author presents us with this gem:
"An interesting property of functional languages is that they can be reasoned about mathematically. Since a functional language is simply an implementation of a formal system, all mathematical operations that could be done on paper still apply to the programs written in that language. The compiler could, for example, convert pieces of code into equivalent but more efficient pieces with a mathematical proof that two pieces of code are equivalent. Relational databases have been performing these optimizations for years. There is no reason the same techniques can't apply to regular software."
Now, imagine rendering Python code as a formal system, in functional terms, in order to achieve bullet proof logic validation, testing, etc., with the ability to swap in highly optimized forms where a code pattern is recognized as a suitable match.
See: http://www.defmacro.org/ramblings/fp.html
Cool stuff, JerryS? --Shane Hathaway, Wed, 12 Mar 2008 03:18:23 +0000 reply
You should get involved in the PyPy? project. :-)
BLOBS --Laurence Rowe, Thu, 13 Mar 2008 11:10:39 +0000 reply
Have you thought of adding BLOB support to RelStorage? Does the ZODB 3.8 BLOB support work with it? Having file data in a separate table would presumably allow the connection cache to be freed up more quickly, stop large files from flushing the connection cache and do away with the need for nfs mounting the blob filesystem. Would limit things to 1GB in postgres though.
BLOBs --Shane Hathaway, Tue, 29 Apr 2008 05:35:28 +0000 reply
After pondering BLOB support for a while, I've come to the tentative conclusion that NFS or some other shared filesystem is a better choice than putting BLOBs in the database. The main reason is that BLOB handles cross threads and remain open long after ZODB connections die; database connections don't like to behave that way. NFS really isn't bad if you manage the network well. Shared filesystems on a SAN are even better. I could also see MogileFS? as a good way to store BLOBs.
how do I back out an easy install? --dbearex, Fri, 09 May 2008 19:06:00 +0000 reply
running easy_install RelStorage put all the modules in my python home -- How do I reverse this in get them all out?
more on BLOBs --Rob Miller, Wed, 21 May 2008 15:53:32 -0400 reply
I understand your reasoning on keeping BLOBs out of the database, but I wonder if you could still let us know whether or not RelStorage would work w/ ZODB 3.8's BLOB support? That already has an option for telling ZEO clients that there's a shared filesystem path to get to the BLOB files, instead of using the database connection; it'd be nice if this mechanism could continue to work w/ RelStorage.
Extensible backend --Charlie_X, Tue, 27 May 2008 08:45:16 -0400 reply
Hi Shane, this looks like an exciting product that should be extensible for other backends: MS SQL, DB2 especially if this largely involves just the SQL statements. What do you think of moving the SQL into separate text files with loading controlled through some form of utility?
Replies --Shane Hathaway, Tue, 27 May 2008 13:54:24 -0400 reply
dbearex: Unlike apt/rpm/portage/etc., easy_install has no uninstall capability; that is not its responsibility. You'll have to do something like 'rm -rf /usr/lib/python2.4/site-packages/relstorage'.
Rob Miller: Blob support over NFS probably works already. Let me know.
Charlie: I have found that it's better to generate most of the SQL code rather than put it in text files. That makes it easier to make changes to all supported databases. SQLAlchemy? or similar might be good, but I haven't tried that yet.
SQL generation --Charlie_X, Wed, 28 May 2008 05:30:08 -0400 reply
Shane: we would like to be able to add support for mxODBC which would require separation from the driver and the SQL - most of the SQL generators won't let you do this which can cause problems.
Unless you're expecting to change the SQL a lot, it's not an unmanageable set of statements and knowing exactly what the SQL is oftens solves a lot of problems.
Main database is Big --Alexandre Silva, Wed, 16 Jul 2008 13:14:40 -0400 reply
Why main database with relstorage and postgres is 41MB and with FileStorage is 8MB. With the pack with relstorage the database remain with the same size. The FileStorage with pack was with 4MB. The pack with Relstorage work in undo history, but not with the database size. why? Sorry for my english.
Main database is Big --Laurence Rowe, Sun, 20 Jul 2008 14:17:39 -0400 reply
Have you tried VACUUM in postgres? See http://www.postgresql.org/docs/8.3/static/routine-vacuuming.html
Main database is Big --Alexandre Silva, Mon, 21 Jul 2008 07:35:33 -0400 reply
Yes, i tried. But i have a datafs of 40MB, if datafs grow up 10MB because of a new object(50MB), and i delete the object and use PACK and VACUUM, datafs return only to 1MB.(the datafs has 49MB). With this it's continues to grow :( .
Main database is Big --Laurence Rowe, Mon, 21 Jul 2008 07:54:06 -0400 reply
You must use VACUUM FULL to return space to the filesystem.
Main database is Big --Alexandre Silva, Mon, 21 Jul 2008 07:58:05 -0400 reply
Yes, i used VACUUM FULL, without this does not return any space, only with the pack and vacuum until the datafs increases
Database packing --Shane Hathaway, Thu, 24 Jul 2008 12:49:52 -0400 reply
Alexandre, packing in RelStorage followed by "vacuum full" should indeed reduce the space consumed by PostgreSQL. If it doesn't, then there's a bug somewhere.
**zope 2.9.5 ** --luca.tebaldi, Fri, 05 Sep 2008 11:03:04 -0400 reply
relstorage work's on zope 2.9.5? it's necessary the invalidation polling patch?
Open connections --scamps, Wed, 22 Oct 2008 12:42:37 -0400 reply
RelStorage is really fantastic, but I've observed the number of open connections is always increasing, but never decreases. Anybody with the same problem ? Could be related to storage files in the ZODB (so, in postgres through relstorage) ?
Can't patch when using easy_install --kenroy, Fri, 07 Nov 2008 02:04:22 -0500 reply
I ran easy_install and found that it installed this version of ZODB3-3.9.0a4-py2.4-linux-i686.egg in my Python Home. Neither patch works with this version. Is there a poll-invalidation patch available for 3.9?
YA reason for native pickle data type --jerryS, Mon, 08 Dec 2008 11:44:23 -0500 reply
Re. and old post, on why it would be cool to leverage PostgreSQL's user defined data types within a ZODB storage, this turned up:
"CouchDB? has a built-in indexing and querying facility, in the form of Views. This is something that ZODB does not share. Instead, ZODB relies on applications that are built on top of it to provide indexing capabilities. Moreover, creating indexes in applications that use ZODB is historically a static kind of thing that the application developer does "up front", or at least as a "software release" sort of thing. In CouchDB?, creating an index is not really an exceptional sort of event. You tell the server, over HTTP, to create the index by PUT-ing a view. The first time any view is queried, CouchDB? does the indexing. The following times the view is queried, it uses the index you've created via the view to return the results faster. No application that I know of written on top of ZODB allows you to do such a thing so casually."
http://plope.com/Members/chrism/wherefore_couchdb
So, once ZODB objects (pickles) are stored as native PostgreSQL data types, indexing is already there, in several forms, ready for access via PostgreSQL's native SQL API.
That, plus PostgreSQL's native array data type, is an invitation to reveal what true Object Publishing is all about.
Add PostgreSQL's clustering / replication, and CouchDB? has somewhat less to offer as a "unique" solution.
just another $.02 Jerry S.
Fantastic speed increase but how to store temp data ? --Vinylrider, Tue, 13 Jan 2009 17:11:32 -0500 reply
RelStorage is an real performance bringer ! It is really impressive how fast Zope can start up. With the old ZODB it took ages to get full caches on each ZEO client and before the caches are not filled, the speed is awfully slow. But anyway, we still need to have a zeo server open only for temp storage. Maybe someone knows a solution on how to store temp data with RelStorage aswell ?