MOAI-iplweb 2.0.0

MOAI, an Open Access Server Platform for Institutional Repositories
===================================================================

MOAI is a platform for aggregating content from different sources, and publishing it through the Open Archive Initiatives protocol for metadata harvesting.
It's been built for academic institutional repositories dealing with relational metadata and asset files.

What does it do?
----------------

The MOAI software can aggregate content from different sources, transform it and store it in a database. The contents of this database can then be published in many separate OAI feeds, each with its own configuration.

The MOAI software has a very flexible system for combining records into sets, and can use these sets in the feed configuration. It also comes with a simple yet flexible authentication scheme that can be easily customized. Besides providing authentication for the feeds, the authentication also controls the access to the assets.

Why MOAI
--------

MOAI has been specifically developed for universities, and contains a lot of hard-earned wisdom. The software has been in production use since 2007, and new features have been continually added. In late 2008, the software was completely refactored and packaged under the name "MOAI". You can read more about this on the `MOAI History`_ page.

MOAI is a standalone system, so it can be used in combination with any repository software that comes with an OAI feed such as `Fedora Commons`_, `EPrints`_ or `DSpace`_. It can also be used directly with an SQL database or just a folder of XML files.

The MOAI project takes the philosophy that every repository is different and unique, and that an institutional repository is a living thing. It is therefore never finished. Metadata is always changes, improving, and evolves. We think this is healthy.

Because of this viewpoint, the MOAI software makes it as easy as possible to add or modify parts of your repository (OAI) services stack. It tries to do this without sacrificing power, and encouraging the re-use of components.

Features
--------

MOAI has some interesting features not found in most OAI servers.
Besides serving OAI, it can also harvest OAI. This makes it possible for MOAI to work as a pipe, where the OAI data can be reconfigured, cached, and enriched while it passes through the MOAI processing.


More specifically MOAI has the ability to:

- Harvest data from different kinds of sources
- Serve many OAI feeds from one MOAI server, each with their own configuration
- Turn metadata values into OAI sets on the fly, creating new collections
- Use OAI sets to filter records shown in a feed, configurable for each feed
- Work easily with relational data (e.g. if an author changes, the publication should also change)
- Simple and robust authentication through integration with the Apache webserver
- Serve assets via Apache while still using configurable authentication rules


.. _MOAI History: http://moai.infrae.com/history.html
.. _Fedora Commons: http://www.fedora.info
.. _EPrints: http://www.eprints.org
.. _DSpace: http://www.dspace.org/

This is a fork
==============

This is a fork of original project, which can be found https://github.com/infrae/moai/ which
includes TravisCI integration and Python 3 support. Both of those were returned to the original
authors in form of a pull request here: https://github.com/infrae/moai/pull/5 . In case
the original author decides to integrate suggested changes and support Python 3, this fork
will be irrelevant.

Please note that other than updating this README and renaming setup.py file, there are no
bigger changes. This means that documentation may be obsolete in some parts. Patches accepted.


===============
Installing MOAI
===============

.. image:: https://img.shields.io/travis/iplweb/moai-iplweb.svg
:target: https://travis-ci.org/iplweb/moai-iplweb


The MOAI Software can be run in any wsgi compliant server.

MOAI comes with a development server that can be used for testing. In production mod_wsgi can be used to run Moai in the apache webserver.

Installation Steps
==================

MOAI is a normal python package. It is tested with Python 2.7, 3.4, 3.5, 3.6.
I recommend creating a virtualenv to install the package in.

http://pypi.python.org/pypi/virtualenv/

This makes development and deployment easier.
Instructions below are for unix, but MOAI should also work on Windows

Go into the MOAI directory with the setup.py, and run the virtualenv command

> cd moai
> virtualenv .

Now, activate the virtualenv

> source bin/activate

Install MOAI in the virtualenv using pip

> pip install -e .

(this will take a while)

When this process finishes, Moai and all its dependencies will be installed.

Running in development mode
===========================

The development server should never be used for production, it is convenient for testing and development. Note that you should always activate the virtualenv otherwise the dependecies will not be found

> cd moai
> source bin/activate
> ./bin/paster serve settings.ini

This will print something like:

Starting server in PID 7306.
Starting HTTP server on http://127.0.0.1:8080

You can now visit localhost:8080/oai to view the moai oaipmh feed.

Configuring MOAI
================

Configuration is done in the settings.ini file. The default settings file uses the Paste#urlmap application to map wsgi applications to a url.

In the `composite:main` section there is a line:

/oai = moai_example

Which maps the /oai url to a Moai instance.
This makes it easy to run many Maoi instances in one server, each with it's own configuration.

The app:moai_example configuration let's you specify the following options:

name
The name of the oai feed (returned in Identify verb)
url
The url of the oai feed (returned in oaipmh xml output)
admin_email
The email adress of the amdin (returned in Identify verb)
formats
Available metadata formats
disallow_sets
List of setspecs that are not allowed in the output of this feed
allow_sets
If used, only sets listed here will be returned
database
SQLAlchemy uri to identify the database for used for storage
provider
Provider identifier where moai retrieves content from
content
Class that maps metadata from provider format to moai format

Adding Content
==============

The Moai system is designed to periodically fetch content from a `provider`, and convert it to Moai's internal format, which can then be translated to the different metadata formats for the oaipmh feed.

Moai comes with an example that shows this principle:

In the moai/moai directory there are two XML files. Let's pretend these files are from a remote system, and we want to publish them with MOAI.

In the settings.ini file, the following option is specified:

`provider = file://moai/example-*.xml`

This tells moai that we want to use a file provider, with some files located in
`moai/example-*.xml`.

The following option points to the class that we want to use for converting the example content xml data to Moais internal format.

content = moai_example

The last option tells Moai where to store it's data, this is usually a sqlite database:

database = sqlite:///moai-example.db

Now let's try to add these two xml files, let's first visit the oaipmh feed to make sure nothing is allready being served:

http://localhost:8080/oai?verb=ListRecords&metadataPrefix=oai_dc

This should return a noRecordsMatch error.

To add the content, run the update_content script, with the section name from the settings.ini as argument

> ./bin/update_moai moai_example

This will produce the following output:

/ Updating content provider: example-2345.xml
Content provider returned 2 new/modified objects

100.0%[====================================================================>] 2
Updating database with 2 objects took 0 seconds

Now when you visit the oaipmh feed again you should see the two records:

http://localhost:8080/oai?verb=ListRecords&metadataPrefix=oai_dc

When you run the update_moai script again, it will create a new database with all the records (in this case moai_example.db). It is also possible to specify a data with the --date switch. When a data is specified, only records that were modified after this date will be added.
The update_moai script can be run from a daily or hourly cron job to update the database

Adding your own Provider / Content and Metadata Classes
=======================================================

It's possible and most of the time, needed, to extend Moai for your use-cases.
The Provider and Content classes from the example might be a good starting point if you want to do that. All your customizations should be registered with Moai through `entry_points`. Have a look at Moais setup.py for more information.
The best approach would be to create your own python package with setup.py and install this in the same environment as Moai. This will let Moai find your customizations. Note that when you change something in your setup.py, you have to reinstall the package, for Moai to pick up the changes.

Note that the moai.interfaces file contains documentation about the different classes that you can implement.

Adding your own Database
========================

Instead of writing your own provider/content classes, you can also register your own custom database. Implementing a replacement for moai.database.SQLDatabase can be more complicated then writing a provider/content class, but it has the advantage that Moai is always up to date, and you don't need a second sqlite database.

Have a look at the setup.py file from the MOAI code, it registers several databases. You could use this mechanism to register your own database from your own python package.

In the settings.ini you configuration you can then reference your database ('mydb://some+config+variables').

For the database, have a look at the generic database provider in database.py. The only methods that you need to implement are: oai_sets, oai_earliest_datestamp and oai_query.
The oai_query method returns dictionaries with record data. The keys of these dictionaries are defined in the metadata files (for example metadata.py), have a look at the source.

For oai_dc there are the following names:

'title', 'creator', 'subject', 'description', 'publisher', 'contributor', 'type', 'format', 'identifier', 'source', 'language', 'date', 'relation', 'coverage', 'rights

So a return value would look like::

{'id': <oai record id>,
'deleted': <bool>,
'modified': <utc datetime>,
'sets': <list of setspecs>,
'metadata': {
'title': [<list with publication title>],
'creator': [<list of creator names>],
...}
}







MOAI Changes
============

MOAI 2.0.0 (2013-02-28)
-----------------------

- First public release in the 2.x series, switched to mod_wsgi,
smaller simpler codebase, pluggability through entry_points instead of ZCA.
- New docs and examples, simpler direct integration with external databases
- Update mods format and support for EU GAL metadata MODS extension
- Several documentation and tool improvements by Filipe Correia

MOAI 1.1.2 (2010-09-30)
-----------------------

- Changed contact info, and switched repository from svn to mercurial
this release has no code changes


Changes
~~~~~~~

MOAI 1.1.1 (2010-09-02)
-----------------------

Changes
~~~~~~~

- Fixed ID/IDref attributes in MODS format
- Fixed ZeroDivisionError in progressbar drawing of updatedb script

MOAI 1.1.0 (2010-04-08)
-----------------------

Changes
~~~~~~~

- Added new config option `deleted_sets` which can be used to make all
resources within a specific set `oai deleted` within the context of a
specific feed. In many cases this is preferable compared to `disallowed_sets`
since resources will just dissapear when they are not allowed without
leaving a proper deleted resource behind.
- Disable ranged HTTP requests in Apache mod_python backend. This was giving
problems with acrobat plugin and its "Fast Web View" feature.
- Show records sorted on descending datestamp, instead of ascending record id,
in sqlite backend.

Changes
~~~~~~~

MOAI 1.0.10 (2010-03-25)
------------------------

Changes
~~~~~~~

- Changed DIP:objectType to rdf:type in the didl metadata format,
as required by darenet/driver/didl 3.0 format
- Added support for modification dates and accessrights in the didl format


MOAI 1.0.9 (2010-02-16)
-----------------------

Changes
~~~~~~~

- Added checking on XML compatibility in update.py, update_database_iterate().
This prevents the OAI server from crashing on invalid data.
- Fixed a minor bug in tools.py. The -d (debug) flag for the update_database
script didn't halt on errors.

- Added a flush_threshold attribute to the DatabaseUpdater. This can be
used to make the database flush after n records. By default the value
is -1, and flushing only happens at the end of the update.
- Fixed assertion statements in content.py (gave warnings in python2.6)


MOAI 1.0.8 (2009-16-18)
-----------------------

Changes
~~~~~~~

- Added a MANIFEST.in file to make sure all files are included
in the python package.

MOAI 1.0.7 (2009-16-11)
-----------------------

Changes
~~~~~~~

- Changed DIP namespace in didl metadata extension
- Fixed bug in ListMetadataFormats verb
- Uses pyoai 2.4.1, which when specifying a date (not a datetime)
for the until parameter, defaults to 23:59:59 instead of 00:00:00.

MOAI 1.0.6 (2009-08-12)
-----------------------

Changes
~~~~~~~
- Fixed metadataPrefix value in didl:Resource URL. This was using
`didl` instead of `nl_didl`

MOAI 1.0.5 (2009-08-12)
-----------------------

- Unicode bug in Fedora provider
- Added support for mods:location/url
Make url prefered identifier in oai_dc:identifier instead of uri
- Update method of Content Objects can now return False to prevent
further processing of the object. The object will be logged as
`ignored`

MOAI 1.0.4 (2009-07-15)
-----------------------

Changes
~~~~~~~

- Fixed misspelled mimeType attribute in DIDL metadata prefix

MOAI 1.0.3 (2009-06-23)
-----------------------

Changes
~~~~~~~
- Bug in ListMetadataFormats when using nl_didl prefix
- Added simple dc example profile

MOAI 1.0.2 (2009-05-20)
-----------------------

Changes
~~~~~~~

- Location in mod_python config is dynamicly extracted
from Metro Server class.

MOAI 1.0.1 (2009-05-19)
-----------------------

Changes
~~~~~~~

- Fixed URL error in didl metadata prefix
- Seperated darenet specific didl extensions in nl_didl prefix

MOAI 1.0 (2009-05-04)
-----------------------

Changes
~~~~~~~

- Improved documentation
- Added description to Identify header
- More complete mods name roles
- Require newer pyoai version that gives better error responses
when wrong datatime values are used in OAI requests

MOAI 0.9.x (2009-02-01)
-----------------------

Changes
~~~~~~~
- Added Apache mod_python backend
- Added basic authentication for fedora provider
- Made fedora provider datastream argument optional,
if not specified the whole foxml file is returned
- Ported SQLite database backend from EUR RePub
- Renamed MetaDataPrefix classes to MetaDataFormat
- Added configurable plugin system
- Added documentation website (more info in README.txt)
- Made Fedora provider not write all files in same directory

Changes
~~~~~~~
- OAI ListRecords with from and until dateparams where not inclusive in SQLite engine
- OAI ListSets did not return dynamic sets (added test)
- OAI GetRecord returning the same record (added test)


MOAI 0.9 (2008-11-18)
-----------------------

- Initial development borrowing code and ideas from RepubXML,
a repository environment build for the Erasmus Univiersity Rotterdam