Commit graph

18 commits

Author SHA1 Message Date
Brian Gough
d4563c8786 Merge pull request #4866 from overleaf/bg-gcs-delete-directory-batch
make object-persistor delete gcs files in batches

GitOrigin-RevId: 8ebc892c5f6eb30507ec41d5d3a108e650af5cac
2021-08-27 08:03:25 +00:00
Jakob Ackermann
5679ba894b Merge pull request #4800 from overleaf/jpa-object-persistor-fix-404-handling
[object-persistor] handle 404s of the entire delete operation in gcs

GitOrigin-RevId: 41c5e70af069361a21adb53a6166081741666887
2021-08-19 08:02:43 +00:00
Jakob Ackermann
6a0388ff9b Merge pull request #4689 from overleaf/jpa-libraries-ci
[misc] setup cloud build trigger for libraries

GitOrigin-RevId: a66dd3e96cb1c06d4803a2162278959357694048
2021-08-13 12:56:39 +00:00
Jakob Ackermann
a383dc316c [misc] run format:fix 2021-07-27 16:53:06 +01:00
Jakob Ackermann
d2589823b0 [misc] prepare for monorepo
- add git ignore entries
- sync/create package-lock.json
- delete .nvmrc
2021-07-27 16:52:55 +01:00
Eric Mc Sween
0fdfa706f7 Make 404 error handling on deletes more specific
The GCS persistor error handler ignores 404 errors on deletes. However,
the net is too wide cast, and 404 errors coming from the dual-bucket
lifecycle mechanism are also ignored.
2020-10-15 10:35:52 -04:00
Simon Detheridge
60931b86c3 Don't try to delete files if none are found in a directory 2020-10-13 09:53:19 +01:00
Eric Mc Sween
9089d332ab Add a "unsignedUrls" option to the GCS persistor
This boolean option disables URL signing. Instead, getRedirectUrl()
returns the plain GCS download URL.
2020-09-22 15:59:13 -04:00
Eric Mc Sween
c413f4d8a4 Fixed some function names in the docs 2020-09-15 16:13:58 -04:00
Eric Mc Sween
967dde280e Disable auto-decompression in GCS persistor
The GCS library automatically decompresses files with Content-encoding:
gzip. This is very useful, but the S3 library doesn't do that, so we
need to disable the feature in GCS so that it behaves the same way other
persistors behave.
2020-08-18 16:04:54 -04:00
Simon Detheridge
1115e1a1e6 Add default parameter for 'opts' in MigrationPersistor.getObjectStream 2020-08-10 16:44:37 +01:00
Simon Detheridge
8037208800 Bump dependency versions 2020-07-23 10:33:01 +01:00
Eric Mc Sween
6521837993 Add S3 options: httpOptions, maxRetries 2020-07-09 17:50:05 -04:00
Eric Mc Sween
3e7e4369af Implement getRedirectUrl() for the S3 backend 2020-07-08 17:13:20 -04:00
Eric Mc Sween
523ff9c4cd Support metadata when uploading objects
Add contentType and contentEncoding options to sendStream(). These
options will set the corresponding metadata on the object.

This changes the API. The fourth argument to sendStream() used to be the
source md5. Now, it's an options object and the source md5 is a property
of that object.
2020-07-08 08:13:53 -04:00
Simon Detheridge
990bacf3c2 Update to OError3 2020-07-07 10:40:37 +01:00
Simon Detheridge
7369e2f339 Add .circleci/config.yml and fix errors 2020-07-03 11:59:34 +01:00
Simon Detheridge
e92b75a2f8 Create new module from overleaf/filestore persistors (#1)
* Create new module from overleaf/filestore persistors

* Convert persistors to ES6 classes with local settings

* Update README.md

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* Update README.md

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* Update .gitignore

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* Switch to AGPL license

* Paginate S3 list-object results

* Remove S3 client caching

* Clean up S3 md5-verification mechanism

* Update README for recent changes

* Update README.md

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* Remove package-lock

* Remove comment about FileHandler

* Add directory marker to FSPersistor.deleteDirectory

* Don't copy opts in GcsPersistor.getObjectStream

* Use Date.now instead of getTime

* Catch errors in migration persistor

* Check that settings.buckets exists

* Don't mutate options in ObserverStream constructor

* Update src/PersistorHelper.js

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>

* Lint and format fixes

Co-authored-by: Eric Mc Sween <eric.mcsween@overleaf.com>
2020-07-02 14:19:45 +01:00