Snovault Hybrid Object Relational Database Framework

Navigation

Verified details

These details have been verified by PyPI
Maintainers
Avatar for benhitz from gravatar.com benhitz Avatar for gabdank from gravatar.com gabdank Avatar for keenangraham from gravatar.com keenangraham Avatar for paulsud from gravatar.com paulsud Avatar for pedrohr from gravatar.com pedrohr Avatar for perna from gravatar.com perna

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Project description

SnoVault JSON-LD Database Framework

System Installation (OSX Big Sur(testing), Catlina(10.15.x), Mojave(10.14.6))

We will try to keep this updated as OSX, Xcode, and brew update. However the steps below are examples and not guaranteed to work for your specific system. See the dependency’s website for detailed instructions or let us know of any changes with a pull request.
  1. Command line tools
    xcode-select --install

  2. brew: https://brew.sh/ . Make sure git is installed

  3. Python 3.8.5

  4. Postgres@11 (Postgres@9.3 also works)
    brew install postgresql@11
# May need to add postgres to PATH in your shell profile, e.g. ~/.bash_profile, ~/.zshrc
# echo 'export PATH="/usr/local/opt/postgresql@11/bin:$PATH"' >> YOUR_SHELL_PROFILE
  5. Node 10.x.x
    brew install node@12

    You may need to link node/npm with brew link node@12 then add it to your PATH

  6. Ruby - Non system version to install compass while avoiding permission errors
    brew install ruby
# May need to add ruby to your bash_profile/zshrc and restart terminal
  7. Compass
    gem install compass
# Test the install
compass -v
# If the command is not found then find your ruby bin directory
ls /usr/local/lib/ruby/gem/
# If you have two versions use the active one
ruby -v
# Using the correct ruby version bin diretory, make a sym link
ln -s /usr/local/lib/ruby/gems/2.6.0/bin/compass /usr/local/opt/ruby/bin/compass
  8. Java 11
    brew install openjdk@11
# Add to your PATH in terminal profile, i.e. ~/.bash_profile or ~/.zshrc
export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)
  9. Elasticsearch 5.x
    # Download tar: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.0.tar.gz

# Decompress
tar -xvf ~/Downloads/elasticsearch-5.6.0.tar.gz -C /usr/local/opt/

# Add to PATH in your terminal profile, i.e. ~/.bash_profile or ~/.zshrc
export PATH="/usr/local/opt/elasticsearch-5.6.0/bin:$PATH"

# Restart terminal and check versions
elasticsearch -V
  10. Brew dependencies
    brew install libmagic nginx graphviz redis

  11. Chrome driver for Testing

    Chromedriver is needed in your PATH. If working in a python virtual environment, then the chromedriver can be added to your-venv-dir/bin directory.

    You also need to install Chrome (if not already installed). In addition, allow chromedriver (System Preferences->Security & Privacy) to run to run bdd tests

Application Installation

  1. Create a virtual env in your work directory. Here we use python3 venv module. Use venv, like conda, if you please
    cd your-work-dir
python3 -m venv snovault-venv
source snovault-venv/bin/activate
pip install -U pip==21.0.1
  2. Clone the repo and install requirements
    cd snovault
pip install -e '.[dev]'
    If psycopg2 fails to compile, you may need to set LDFLAGS to the output of pg_config --ldflags before pip installation.
    LDFLAGS=$(pg_config --ldflags) pip install -e '.[dev]'

    If you have errors at runtime that look like this:

    ImportError: dlopen(/Users/foo/venv/lib/python3.7/site-packages/psycopg2/_psycopg.cpython-37m-darwin.so, 2): Symbol not found: _PQencryptPasswordConn
Referenced from: /Users/foo/venv/lib/python3.7/site-packages/psycopg2/_psycopg.cpython-37m-darwin.so
Expected in: /usr/lib/libpq.5.dylib
in /Users/foo/venv/lib/python3.7/site-packages/psycopg2/_psycopg.cpython-37m-darwin.so

    you may need to add the brew-installed Postgres headers, usually -L/usr/local/opt/postgresql@11/lib, to the LDFLAGS in addition to the ones given by pg_config --ldflags.

  3. Build Application
    # Make sure you are in the snovault-venv
make clean && buildout
  4. Run Application
    # Make sure you are in the snovault-venv
dev-servers development.ini --app-name app --clear --init --load
# In a separate terminal, make sure you are in the snovault-venv
pserve development.ini

  5. Browse to the interface at http://localhost:6543

  6. Run Tests

    • no argument runs non bdd tests

    # Make sure you are in the snovault-venv
./circle-tests.sh bdd
./circle-tests.sh npm
./circle-tests.sh

Changes

Snovault 1.0.61 (released) encoded v124.0

  1. SNO-214-upgrade-node-to-version-12-for-circleci-tests (#365)

  2. SNO-210-read-columns-from-configs (#368)

  3. SNO-216 Fix dev-servers 100% cpu usage when nothing to print. (#367)

  4. SNO-212 Allow gzip attachments (#363)

  5. SNO-213-circleci-node-install-fix (#364)

  6. SNO-211-increase-batch-size-in-scroll-api (#360)

Snovault 1.0.60 (released) encoded v121.0

  1. SNO-209-move-search-code-to-independent-package (#356)

  2. SNO-206-use-redis-cache-for-rnget-view (#354)

  3. SNO-207-facet-groups-support (#355)

  4. SNO-205-allow-customization-of-default-sort (#353)

Snovault 1.0.59 (released) encoded v120.0

  1. SNO-204-allow-facet-caching (#351)

Snovault 1.0.58 (released) encoded v118.0

  1. SNO-170-avoid-rendering-from-database-3 (#346)

  2. SNO-202-upgrade-python (#345)

Snovault 1.0.57 (released) encoded v116.0

  1. revert SNO-202-python3.8.5-upgrade (#338)

  2. SNO-201-add-search-config-registry (#336)

  3. SNO-202-python3.8.5-upgrade (#335)

Snovault 1.0.56 (released) encoded v115.0

  1. SNO-200-remove-nonsubstring-filter-from-mapping (#332)

  2. SNO-199-allow-customization-of-search-fields (#333)

  3. SNO-197-add-range-queries-and-min-max-aggs (#329)

  4. SNO-198-allow-configuration-of-reserved-search-params (#330)

  5. SNO-138-fix-pytest-warnings (#324)

  6. SNO-182-readme-update (#323)

Snovault 1.0.55 (released) encoded v114.0

  1. SNO-192-update-pip-install (#321)

Snovault 1.0.54 (released) encoded v113.0

  1. SNO-27-pip-installable (#316)

  2. SNO-191-pyramid-env-vars (#318)

  3. SNO-117-search-as-you-type (#220)

Snovault 1.0.53 (released) encoded v112.0

  1. SNO-158-create-minimally-indexed-deploy

  2. SNO-190-pin-setuptools-in-circle-ci (#315)

Snovault 1.0.52 (released) encoded v111.0

  1. SNO-185-Log-connection-error (#311)

  2. SNO-183-allow-specification-of-cart-in-search (#310)

  3. SNO-181-Update-install-readme (#309)

Snovault 1.0.51 (released) encoded v109.0

  1. SNO-174-add-local-storage-redis (#299)

  2. SNO-176-invalidate-object-when-unlinked (#303)

  3. SNO-177-add-view-with-specified-fields (#300)

  4. SNO-178-fix-indexer-test (#304)

Snovault 1.0.50 (released) encoded v108.0

  1. SNO-168-stream-heterogeneous-uuids (#292)

  2. SNO-169 Creating a command for admin user creation (#293)

Snovault 1.0.49 (released) encoded v106.0

  1. SNO-166-update-query-string (#288)

Snovault 1.0.48 (released) encoded v105.0

  1. SNO-163-install-graphviz (#285)

Snovault 1.0.47 (released) encoded v104.0

  1. SNO-159-allow-remote-indexing (#282)

Snovault 1.0.46 (released) encoded v103.0 (unreleased)

  1. SNO-160-use-boolean-values-in-facets (#280)

  2. SNO-116 Use summary_matrix instead of summary (#278)

  3. SNO-157-make-facets-collapsable (#275)

Snovault 1.0.45 (released) encoded v102.0

  1. SNO-154-fix-magic-json-bug (#272)

  2. SNO-156 Use pyramid_retry 2.1.1 (#273)

  3. SNO-137-add-search-doc (#267)

  4. SNO-152 fix circle-ci browser install (#269)

Snovault 1.0.44 (released) encoded v99.0

  1. SNO-150 Remove duplicates and fix antlr version (#266)

  2. SNO-149-allow-specification-of-audit-field (#265)

  3. SNO-143-update-and-switch-readme

  4. SNO-148-uuid-indexing-times-new-line-bit

  5. SNO-144-redirect-downloads-through-nginx-or-apache (#192)

Snovault 1.0.43 (released) encoded v98.0

  1. SNO-146-clean-up-threadmanager-on-pyramid-retry-exception (#261)

Snovault 1.0.42 (released) encoded v97.0

  1. SNO-145-handle-missing-obj-on-request (#258)

Snovault 1.0.41 (released) encoded v97rc

  1. SNO-142 Minor fixes for ubutnu 18 on v96x1rc1 (#252)

  2. SNO-141-update-osx-catalina-readme (#252)

  3. SNO-140 Allow shorting uuids with flag (#249)

  4. SNO-139 Log detailed initial indexing times (#249)

  5. SNO-135 Update installation for osx catalina (#249)

  6. SNO-132 Update to py37 lrowe pr with fixes (#249)

Snovault 1.0.40 (released) encoded v95.0

  1. SNO-129-add-fielded-generator-response (#244)

  2. SNO-128-make-select-calculated-properties-view (#245)

  3. SNO-133-setuptools-test-failure (#247)

Snovault 1.0.39 (released) encoded v94.0

  1. SNO-126-allow-default-value-for-missing-aggregation-field (#242)

Snovault 1.0.38 (released) encoded v93.0

  1. SNO-125-switch-batch-download-to-query-string (#238)

  2. SNO-118-simple-query-string (#240)

  3. SNO-123-limit-es-scan (#237)

  4. SNO-121-fix-circleci-apt-install (#224)

Snovault 1.0.37 (released) encoded v92.0

  1. SNO-119-escape-regex-slash (#221)

  2. [HOTFIX] SNO-120-stream-large-json (#222)

  3. SNO-115-delete-old-search-code (#219)

Snovault 1.0.36 (released) encoded v91.0

  1. SNO-112-fix-npm-audit-issues (#216)

  2. SNO-113-make-audit-view (#217)

Snovault 1.0.35 (UNRELEASED) encoded

  1. SNO-95-refactor-search (#214)

Snovault 1.0.34 (released) encoded v88.0

  1. SNO-108-improve-search-new (#212)

  2. SNO-105-fix-batchupgrade-tests-sno103 (#211)

  3. SNO-106-fix-spec-char-500-error (#209)

Snovault 1.0.33 (released) encoded v87.0

  1. SNO-103 fix-batch-upgrade-error-logging (#206)

  2. SNO-102 use-openjdk-java-for-circleci (#205)

Snovault 1.0.32 (released) encoded v87rc1

  1. SNO-99 port travis tests to circleci (#202)

  2. SNO-100 Parallelize batchupgrade script (#201)

  3. SNO-92 Fix batch upgrade logging (#201)

  4. SNO-101 Refactor batchupgrade (#201)

Snovault 1.0.31 (released) encoded v86.0

  1. SNO-98 Update sauce connect to 4.5.3 (#199)

  2. SNO-93 user-count-does-not-match (#191)

  3. SNO-90 added-advancedQuery (#183)

Snovault 1.0.30 (released) encoded v85x2

  1. SNO-96 fix indexing tests (#196)

Snovault 1.0.29 (released) encoded v85rc1

  1. SNO-91 update-travis-java-ref (#188)

  2. SNO-87 fixed-issue-while-profile-url-does-not-work (#179)

  3. SNO-86 escape-search-text (#178)

  4. SNO-88 fix-user-search-count (#180)

Snovault 1.0.28 (released) encoded v84rc1

  1. SNO-89 Fix backoff error key in indexer (#181)

  2. SNO-85 facet-reappearing (#175)

Snovault 1.0.27 (released) encoded v83.0

  1. SNO-83 Rotate img attachment for EXIF (#174)

Snovault 1.0.26 (released) encoded v82.0

  1. SNO-78 add long facet type (#168)

  2. SNO-80 typeahead for facets (#168)

  3. SNO-73 Add uuid queue module (#162)

Snovault 1.0.25 (released) encoded v81.0

  1. SNO-77 Add review to submit_for validation (#166)

Snovault 1.0.24 (released) encoded v80.0

  1. SNO-75-fix-linux-travis-option (#163)

Snovault 1.0.23 (released) encoded v79.0 also

  1. SNO-72 Update simple for uuid module (#158)

Snovault 1.0.22 (released) encoded v79.0 also

  1. SNO-68 Stop using _all for indexing (#155)

  2. SNO-74 Remove npm shrinkwrap json (#157)

Snovault 1.0.21 (released) encoded v78.0 also

  1. SNO-65 Refactor indexer uuids as server client (#151)

Snovault 1.0.20 (released) encoded v78.0

  1. SNO-63 Update pip requests and remove wal-e reqs (#150)

  2. SNO-66 Add new endpoint to map schema to schema titles (#152)

Snovault 1.0.19 (released) encoded v77.0

  1. SNO-60-check-report-res-has-view (#147)

  2. SNO-50 Initial shopping cart (#142)

  3. SNO-59-fix-index-logger-name (#137)

  4. SNO-53 Add index flags to indexers (#137)

Snovault 1.0.18 (released) encoded v76.0

  1. SNO-49 Change audit inherit default (#132)

  2. SNO-31 Refactor search related views (#141) (#143)

1.0.17 1. [HOTFIX] SNO-54-fix-schema-copy-line (#136)

1.0.16 1. SNO-52-alter-select-distinct-values (#131)

1.0.15 1. SNO-48-add-embed-cache-to-ini (#127)

1.0.14 1. SNO-45 Increase embed_capacity (#123)

1.0.13 1. SNO-46 Remove unused search type arg (#122) 2. SNO-43 Clean up snovault startup (#116)

1.0.12 1. SNO-42 Check call count explicitly (#118)

1.0.11 01. SNO-41-put-validator-accession (#112)

1.0.10 01. SNO-35 fix bug in get_rev_links(#111) 02. SNO-40 Upgrade blob storage to boto3 (#110)

1.0.9: 01. SNO-38 Return lists from get_related_uuids (#108, #105)

1.0.8: Released with 1.0.9 01. SNO-36-update-buildout (#104) 02. SNO-34-nginx-dev-proxy-headers (#103)

1.0.7: The only update was reverted. Empty Release.

1.0.6: 01. SNO-33 specify index for get_by_unique_key from collection (#94) 02. SNO-28 limit ES storage to indices created for snovault resources (#93)

1.0.5: 01. SNO-30 Split Indexer State from indexer file and update 02. SNO-10 Remove audit indexing via 2-pass 03. SNO-9 Add api end points to new indexer meta objects 04. SNO-25 Make uniqueItems to check the serialized values (#85) 05. SNO-26 Add schemas map view to profiles (#86) 06. SNO-29 Limit audits on form update (#87) 07. SNO-19 Update DOI preferred resolver url (#80)

1.0.4: -SNO-15 Add index to storage propertysheet -SNO-14 Update delete script

1.0.3: -SNO-8 Add JSONSchemas type to profiles page (#73)

1.0.2: * Issues discovered while release of ENCD v65 part 2

-SNO-12: Set max clause parameter in es indexer search #75 -SNO-11: Add timeout to ES indexer search query #74

1.0.1: * Issues discovered while release of ENCD v65

-SNO-6: Fix index query too many clauses failure -SNO-5: Update psycopg to match encoded version 2.7.3

1.0.0:

  • 31 delete

  • ES5 Fix: Update index settings shard number

0.33:

  • ES5 Update: ENCD-2488 ES5 Update aka RM3910

  • Fix travis build: Pre Install setuptools with pip for travis like ENCD-3722

0.32:

  • Update to ENCD-3669 to not include notSubmittable reverse links in the edit view of an object.

0.31:

  • ENCD-3684 Specify https index to fix buildout, update changelog.

  • Use notSubmittable instead of calculatedProperty to indicate properties that may not be submitted. Related to ENCD-3669.

0.30:

  • Document dependency on java 8.

  • Disable 2nd indexing pass.

0.29:

  • Fix recording indexing errors.

  • Add some documentation about indexing.

0.28:

  • Add support for adding and updating child objects specified as abstract types in the schema.

  • Split indexing into 2 phases.

0.27:

  • Move embed cache to connection and increase size.

  • Fix reporting upgrade errors when error path includes an integer.

0.26:

  • Indexer: Limit workers to 1 task and scale chunk size based on number of items being indexed.

0.25:

  • Indexer: Limit workers to 4 tasks to avoid out-of-memory errors.

0.24:

  • If the schema specifies an explicit mapping, use it when building the elasticsearch mapping. This provides an escape valve for edge cases (such as not indexing the layout structure of a page).

  • upgrade to sauceconnect v4.4.4

  • add port_range to wsgi_tests (mrmin)

0.23: - replace copy.deepcopy() for faster indexing

0.22: - New version of image magic, fix sauce labs

0.21: - (pypi errors, identical to 0.22)

0.20: - Patch mpindexer for better error messages

Project details

Verified details

These details have been verified by PyPI
Maintainers
Avatar for benhitz from gravatar.com benhitz Avatar for gabdank from gravatar.com gabdank Avatar for keenangraham from gravatar.com keenangraham Avatar for paulsud from gravatar.com paulsud Avatar for pedrohr from gravatar.com pedrohr Avatar for perna from gravatar.com perna

Unverified details

These details have not been verified by PyPI
Project links
Meta
Classifiers

Release history Release notifications | RSS feed

1.0.68

1.0.67

1.0.66

1.0.65

1.0.64

1.0.62

This version

1.0.61

1.0.60

1.0.59

1.0.58

1.0.57

1.0.56

1.0.55

1.0.54

1.0.52

0.23

0.22

0.21

0.20

0.13

0.12

0.11

0.10

0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

snovault-1.0.61.tar.gz (191.1 kB view details)

Uploaded Source

File details

Details for the file snovault-1.0.61.tar.gz.

File metadata

  • Download URL: snovault-1.0.61.tar.gz
  • Upload date:
  • Size: 191.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.3.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/58.2.0 requests-toolbelt/0.8.0 tqdm/4.62.3 CPython/3.7.3

File hashes

Hashes for snovault-1.0.61.tar.gz
Algorithm Hash digest
SHA256 af695bb8451acdcd2f3ef16b979a38b782455e14990b5ced17ee20348b106e34
MD5 0fb313ad75c2cb3206977046d21aac1c
BLAKE2b-256 138f08fbcd0d5f97af740094396a13813674c92469e737e6b47e0dba19b5ed53

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page