Commit graph

68 commits

Author SHA1 Message Date
Jakob Ackermann
06c839a230 Merge pull request #17237 from overleaf/jpa-s3-iam
[object-persistor] add support for IAM credentials in S3Persistor

GitOrigin-RevId: 4ff34082ca557b17d8a7803f700ee704a3d184c6
2024-02-22 09:04:52 +00:00
Christopher Hoskin
40e243f2e4 Merge pull request #15216 from overleaf/csh-issue-11625-mongo-ug-5-filestore
Upgrade mongodb module for filestore from 3.5.9 to 6.1.0

GitOrigin-RevId: 96993d561a5aa188cdcd0af1e4938dec7c374b56
2023-10-13 08:03:55 +00:00
Brian Gough
8623b4d0f4 Merge pull request #13060 from overleaf/bg-upgrade-fake-gcs-server
upgrade to fake-gcs-server 1.45.2

GitOrigin-RevId: 6ae47211370eedabbf9416498e150f53f1bc8fe4
2023-07-24 08:03:58 +00:00
Eric Mc Sween
12e7471213 Merge pull request #12916 from overleaf/bg-move-stream-buffer-code-to-library
move stream-related code to separate  `@overleaf/stream-utils` library

GitOrigin-RevId: a79a873109b927b4fc0ae36f47d5c67e0df58041
2023-06-02 08:05:57 +00:00
Christopher Hoskin
5fbd5e21a2 Merge pull request #13274 from overleaf/csh-issue-13004-cleanup-custom-code
Remove apiScheme from GCS

GitOrigin-RevId: c948287cb00c7bc432c797bfdfa92a79b84d1bed
2023-06-02 08:05:10 +00:00
Jakob Ackermann
3428d53afe Merge pull request #12938 from overleaf/jpa-leaks-filestore
[filestore] cacheWarm: destroy file stream when not piping into response

GitOrigin-RevId: a0fb14e5ececd28d8d9dbed8cd3a5622c81b8230
2023-05-09 08:04:18 +00:00
Jakob Ackermann
cb16efdc53 Merge pull request #12853 from overleaf/jpa-tweak-filestore-test-output
[filestore] add ss header to error output when sockets are leaking

GitOrigin-RevId: 9aa61c99eaf74a061ea0b524c236abbc6f1c9846
2023-05-01 08:05:11 +00:00
Jakob Ackermann
0b985c7a0b Merge pull request #12723 from overleaf/jpa-revert-node-18-upgrade
[misc] revert node 18 upgrade

GitOrigin-RevId: 752d45b015e2e8802b875b5de8ee044ba1170345
2023-04-21 08:03:22 +00:00
Brian Gough
ed0acd8e58 Merge pull request #12658 from overleaf/bg-node-18-upgrade
Upgrade to node 18

GitOrigin-RevId: 1b25d20a8987950a72a7bf966a825794de4ad960
2023-04-20 08:03:37 +00:00
Jakob Ackermann
0f1163e876 Merge pull request #9221 from overleaf/jpa-pin-s3mock
[misc] pin s3mock image to older version

GitOrigin-RevId: b5548c080fc2142ccf1dd5601689e397f935a0c9
2022-08-12 08:06:10 +00:00
Eric Mc Sween
6c4aa88b9e Merge pull request #8132 from overleaf/em-node-fetch-filestore
Replace request with node-fetch in filestore

GitOrigin-RevId: 49487a941b63655920de04fe50fd197f67498e58
2022-06-01 08:04:23 +00:00
Eric Mc Sween
f7275a6c4b Merge pull request #6079 from overleaf/em-upgrade-logger
Upgrade logger and metrics in all services

GitOrigin-RevId: 2baf63eeeab77fb3559cf763ddacfbf4b745cd0b
2021-12-15 09:04:25 +00:00
Jakob Ackermann
638fbd5510 Merge pull request #5573 from overleaf/jpa-prefer-regex-literals
[misc] fix eslint violations for prefer-regex-literals

GitOrigin-RevId: 216288af22b9eeb18cddace7ec285e0a8046f94a
2021-10-28 08:03:41 +00:00
Brian Gough
18192e4559 Merge pull request #4885 from overleaf/bg-filestore-delete-project
extend filestore project deletion acceptance tests

GitOrigin-RevId: 0433817eee88a7465ab90034043398a6f6736bc0
2021-08-27 08:03:32 +00:00
Jakob Ackermann
e3d7afeb96 [misc] run format_fix and lint:fix 2021-07-13 12:04:46 +01:00
Jakob Ackermann
3152bf361d [misc] switch from settings-sharelatex to @overleaf/settings 2021-07-12 17:47:19 +01:00
Simon Detheridge
473a044209 Upgrade node to 12.18.4 2020-09-17 11:41:42 +01:00
Jakob Ackermann
f0551307d2 [misc] bump the dev-env to 3.3.2 2020-08-10 17:01:12 +01:00
Simon Detheridge
6c853de5d5 Migrate to new object-persistor module (#122)
* Migrate to new object-persistor module

* Support updated persistor module using ES6 classes

* Update object-persistor

* Upgrade new persistor code to OError3

* Update to latest minor release for outdated packages

* Update package hashes to sha512

* Point object-persistor at master branch
2020-07-07 13:49:54 +01:00
Simon Detheridge
0027d27473 Use latest version of gcs server for tests 2020-06-09 10:01:20 +01:00
Simon Detheridge
b37f97e2e3 Pin fake-gcs to 1.18 2020-06-03 10:52:10 +01:00
Simon Detheridge
ccf5f8b9e8 Add acceptance test for leaked sockets on aborted connections 2020-04-02 15:56:19 +01:00
Simon Detheridge
64562dffb0 eslint-disable => eslint-disable-next-line 2020-03-26 22:07:37 +00:00
Simon Detheridge
238d96ed44 Fail acceptance tests on unhandled promise rejection 2020-03-26 16:57:55 +00:00
Simon Detheridge
65f19346f0 Fix fallback to custom s3 buckets 2020-03-19 11:10:31 +00:00
Simon Detheridge
cb4bdd99f4 Use an ISODate for deleted file names 2020-03-18 20:19:12 +00:00
Simon Detheridge
9f74aac1a0 Add acceptance tests for directory deletion 2020-03-16 11:34:45 +00:00
Simon Detheridge
58db14456a Add timestamp to files in deleted bucket 2020-03-14 14:11:17 +00:00
Simon Detheridge
183cb0179a Add support to GCS persistor for unlocking files and copying on delete 2020-03-13 16:18:45 +00:00
Simon Detheridge
28c3fe4a56 Validate key names when deleting directory with a configurable regex 2020-03-10 17:54:09 +00:00
Simon Detheridge
3bb956b38e Use http for the fake GCS server 2020-03-04 17:01:20 +00:00
Simon Detheridge
e2f3dd23c9 Switch back to official fake-gcs-server image 2020-03-04 15:38:18 +00:00
Simon Detheridge
2cfab8d313 Add GCS-specific acceptance tests 2020-03-04 15:38:16 +00:00
Simon Detheridge
7292d93dab Add fake GCS server, and fix health checks 2020-03-04 15:08:19 +00:00
Jakob Ackermann
2b9d0868c5 [misc] test/acceptance: retrieve ingress metrics just before using it
The upload request can bump the ingress metric.
The content hash validation might require a full download
in case the ETag field of the upload response is not a md5 sum.
2020-02-28 12:27:06 +01:00
Jakob Ackermann
516102e6fb [misc] test/acceptance: do not hard code fake credentials 2020-02-28 12:26:46 +01:00
Jakob Ackermann
847f124d7b [misc] test/acceptance: skip the shutdown in case we did not start yet
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-02-28 12:26:24 +01:00
Jakob Ackermann
6589aa6ae0 [misc] test/acceptance: harden the startup check for s3
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-02-28 12:26:24 +01:00
Simon Detheridge
8acc9ef0ae Make S3 partSize a setting 2020-02-17 14:06:15 +00:00
Simon Detheridge
9e7eec7a19 Add acceptance test for large files 2020-02-17 14:05:56 +00:00
Simon Detheridge
4e507197fc Revert "Merge pull request #91 from overleaf/revert-78-spd-migration-persistor"
This reverts commit 0cce792604c258538cf012486e7329fa1be7da0f, reversing
changes made to 08b606e22f855121b2d8a2832a5d7e2f45836c3f.
2020-02-17 14:04:42 +00:00
Simon Detheridge
df2ddbe0e1 Revert "Add Migration Persistor, to send missing file requests to a fallback persistor" 2020-02-14 14:26:33 +00:00
Simon Detheridge
304fdfd35c Explicitly resume stream after adding listener 2020-02-12 13:21:53 +00:00
Simon Detheridge
6dcf351377 Remove unnecessary 'async' 2020-02-12 13:21:53 +00:00
Simon Detheridge
f4a16cd972 Update tests to properly check for copied files 2020-02-12 13:21:53 +00:00
Simon Detheridge
9e0b378948 Remove minipass as dependency and refactor to make things clearer 2020-02-12 13:21:52 +00:00
Simon Detheridge
b4b7fd226e Add mechanisms to transfer files with md5-based integrity checks
Fix error in settings and tidy up tests

Remove unused variable declaration

Remove .only from tests and update eslint rules to catch it in future

Use  to catch errors more safely getting md5 hash

Avoid unnecessary call to S3 to get md5 response
2020-02-12 13:21:52 +00:00
Simon Detheridge
2625e03a31 Add MigrationPersistor for sending 404 requests to a fallback persistor 2020-02-12 13:19:48 +00:00
Simon Detheridge
80d41cf51b Move bucket-specific file endpoint into FileController 2020-01-07 15:05:51 +00:00
Simon Detheridge
87b51d0c35 Remove old exception-handling and shutdown-related mechanisms 2020-01-07 09:30:28 +00:00