Simon Detheridge
e2f3dd23c9
Switch back to official fake-gcs-server image
2020-03-04 15:38:18 +00:00
Simon Detheridge
2cfab8d313
Add GCS-specific acceptance tests
2020-03-04 15:38:16 +00:00
Simon Detheridge
7292d93dab
Add fake GCS server, and fix health checks
2020-03-04 15:08:19 +00:00
Simon Detheridge
9807568de8
Add missing 'process' global for settings tests
2020-03-04 10:08:31 +00:00
Simon Detheridge
8905d393f7
Merge pull request #95 from overleaf/jpa-test-cleanup
...
[misc] test cleanup
2020-03-04 10:01:39 +00:00
Jakob Ackermann
2b15729658
[misc] promisify FileHandler and remove dependency on async
...
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-03-02 18:23:17 +01:00
Brian Gough
ebabe52632
fix settings unit test
2020-02-28 15:27:29 +00:00
Jakob Ackermann
2b9d0868c5
[misc] test/acceptance: retrieve ingress metrics just before using it
...
The upload request can bump the ingress metric.
The content hash validation might require a full download
in case the ETag field of the upload response is not a md5 sum.
2020-02-28 12:27:06 +01:00
Jakob Ackermann
516102e6fb
[misc] test/acceptance: do not hard code fake credentials
2020-02-28 12:26:46 +01:00
Jakob Ackermann
847f124d7b
[misc] test/acceptance: skip the shutdown in case we did not start yet
...
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-02-28 12:26:24 +01:00
Jakob Ackermann
6589aa6ae0
[misc] test/acceptance: harden the startup check for s3
...
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-02-28 12:26:24 +01:00
Jakob Ackermann
94b8a7f891
[misc] test/unit: KeybuilderTests: use a unique suite label
2020-02-28 12:26:23 +01:00
Jakob Ackermann
54e3b57772
[misc] test/unit: add missing globals that are lazy loaded
2020-02-28 12:26:23 +01:00
Jakob Ackermann
7920570dd8
[misc] test/unit: add missing require stubs for metrics and settings
2020-02-28 12:26:22 +01:00
Simon Detheridge
8acc9ef0ae
Make S3 partSize a setting
2020-02-17 14:06:15 +00:00
Simon Detheridge
9e7eec7a19
Add acceptance test for large files
2020-02-17 14:05:56 +00:00
Simon Detheridge
2f2a819b74
Re-fetch file to calculate md5 if etag is not in correct format
2020-02-17 14:05:43 +00:00
Simon Detheridge
5adfb3e2c0
Use large upload chunks
2020-02-17 14:05:31 +00:00
Simon Detheridge
4e507197fc
Revert "Merge pull request #91 from overleaf/revert-78-spd-migration-persistor"
...
This reverts commit 0cce792604c258538cf012486e7329fa1be7da0f, reversing
changes made to 08b606e22f855121b2d8a2832a5d7e2f45836c3f.
2020-02-17 14:04:42 +00:00
Simon Detheridge
df2ddbe0e1
Revert "Add Migration Persistor, to send missing file requests to a fallback persistor"
2020-02-14 14:26:33 +00:00
Simon Detheridge
93cd55fb79
Refactor persistors to use a helper for common things
2020-02-12 13:27:12 +00:00
Simon Detheridge
304fdfd35c
Explicitly resume stream after adding listener
2020-02-12 13:21:53 +00:00
Simon Detheridge
6dcf351377
Remove unnecessary 'async'
2020-02-12 13:21:53 +00:00
Simon Detheridge
f4a16cd972
Update tests to properly check for copied files
2020-02-12 13:21:53 +00:00
Simon Detheridge
9e0b378948
Remove minipass as dependency and refactor to make things clearer
2020-02-12 13:21:52 +00:00
Simon Detheridge
b4b7fd226e
Add mechanisms to transfer files with md5-based integrity checks
...
Fix error in settings and tidy up tests
Remove unused variable declaration
Remove .only from tests and update eslint rules to catch it in future
Use to catch errors more safely getting md5 hash
Avoid unnecessary call to S3 to get md5 response
2020-02-12 13:21:52 +00:00
Simon Detheridge
2625e03a31
Add MigrationPersistor for sending 404 requests to a fallback persistor
2020-02-12 13:19:48 +00:00
Simon Detheridge
f877f51775
Rename *PersistorManager to *Persistor
2020-02-12 13:19:48 +00:00
Simon Detheridge
28fb998719
Refactor request logger into a class
2020-01-14 12:02:39 +00:00
Simon Detheridge
9615a06e0f
Improve logging middleware to add info methods to request object
2020-01-10 17:22:06 +00:00
Simon Detheridge
f40fbe77aa
Simplify logging to log only once per http request
2020-01-10 15:41:36 +00:00
Simon Detheridge
e331a3130c
Handle AccessDenied and stream-premature-close errors
...
These errors tend to occur as part of normal operation and should not generate `logger.err` messages
2020-01-09 14:15:01 +00:00
Simon Detheridge
d8470d4a59
Use correct object path for s3BucketCreds
2020-01-08 11:32:05 +00:00
Simon Detheridge
80d41cf51b
Move bucket-specific file endpoint into FileController
2020-01-07 15:05:51 +00:00
Simon Detheridge
6cc5d94f13
Re-add bucket-specific credentials support for S3
2020-01-07 10:55:00 +00:00
Simon Detheridge
2bb2caf7b3
Clean up settings tests
2020-01-07 09:46:53 +00:00
Simon Detheridge
b642b3cf3a
Rename incorrectly-named 'getDirectorySize' method
2020-01-07 09:30:30 +00:00
Simon Detheridge
a6d9d34fe3
Configure PersistorManager to use the new S3 manager for 'aws-sdk'
2020-01-07 09:30:30 +00:00
Simon Detheridge
2ca74fdf15
Cleanup and refactor S3PersistorManager to use aws-sdk only
2020-01-07 09:30:30 +00:00
Simon Detheridge
473aea4e60
Remove AWSSDKPersistorManager
2020-01-07 09:30:30 +00:00
Simon Detheridge
87b51d0c35
Remove old exception-handling and shutdown-related mechanisms
2020-01-07 09:30:28 +00:00
Simon Detheridge
74b480fc55
Post-decaf cleanup of app.js
2020-01-07 09:29:33 +00:00
Simon Detheridge
85d3c0a852
Remove old exception-handling and shutdown-related mechanisms
2020-01-07 09:27:02 +00:00
Simon Detheridge
fce275e1d4
Post-decaf cleanup of app.js
2020-01-07 09:27:02 +00:00
Simon Detheridge
dbfacce988
Post-decaf cleanup of PersistorManager
2020-01-07 09:26:43 +00:00
Simon Detheridge
10f7087912
Post-decaf cleanup of KeyBuilderTests
2020-01-07 09:26:43 +00:00
Simon Detheridge
ec80052d9a
Decaf cleanup of FileController
...
Remove deprecated res.send
2020-01-07 09:26:23 +00:00
Simon Detheridge
37d44d5d86
Return full path in stubbed 'glob' method
2020-01-07 09:26:01 +00:00
Simon Detheridge
f1b6b35c69
Throw errors more consistently in FSPersistorManager
2020-01-07 09:26:00 +00:00
Simon Detheridge
25f1c2bfc4
Delete temporary file when error in writing to stream
2020-01-07 09:26:00 +00:00
