Simon Detheridge
fc80aa3954
Move directory key validation into FileHandler
2020-03-14 14:31:30 +00:00
Simon Detheridge
58db14456a
Add timestamp to files in deleted bucket
2020-03-14 14:11:17 +00:00
Simon Detheridge
edf1ce1f7e
Delete files from a directory in parallel
2020-03-14 14:02:58 +00:00
Simon Detheridge
183cb0179a
Add support to GCS persistor for unlocking files and copying on delete
2020-03-13 16:18:45 +00:00
Simon Detheridge
28c3fe4a56
Validate key names when deleting directory with a configurable regex
2020-03-10 17:54:09 +00:00
Simon Detheridge
eb93ae4b10
Use Bucket.deleteFiles to delete directory contents, instead of iterating
2020-03-05 14:12:15 +00:00
Simon Detheridge
460dd96b17
Cosmetic clean-up of GCS Persistor & tests
2020-03-05 13:45:46 +00:00
Simon Detheridge
3bb956b38e
Use http for the fake GCS server
2020-03-04 17:01:20 +00:00
Simon Detheridge
30114cd79b
Remove unnecessary try/catch around 'createReadStream'
2020-03-04 16:38:05 +00:00
Simon Detheridge
def383574e
Simplify getMeteredStream to record metric directly
2020-03-04 16:17:36 +00:00
Simon Detheridge
76243fd75a
'bucket' -> 'bucketName' in GCS Persistor
2020-03-04 16:04:26 +00:00
Simon Detheridge
6979b8638a
Add 'Buffer' global for GCS unit tests
2020-03-04 15:42:16 +00:00
Simon Detheridge
12274e1427
Add unit tests for GCS Persistor
2020-03-04 15:38:18 +00:00
Simon Detheridge
d9c9d74994
Remove unnecessary test for S3 file deletion
...
S3 does not throw a not-found error when deleting a file that does not exist
2020-03-04 15:38:18 +00:00
Simon Detheridge
e2f3dd23c9
Switch back to official fake-gcs-server image
2020-03-04 15:38:18 +00:00
Simon Detheridge
2cfab8d313
Add GCS-specific acceptance tests
2020-03-04 15:38:16 +00:00
Simon Detheridge
7292d93dab
Add fake GCS server, and fix health checks
2020-03-04 15:08:19 +00:00
Simon Detheridge
9807568de8
Add missing 'process' global for settings tests
2020-03-04 10:08:31 +00:00
Simon Detheridge
8905d393f7
Merge pull request #95 from overleaf/jpa-test-cleanup
...
[misc] test cleanup
2020-03-04 10:01:39 +00:00
Jakob Ackermann
2b15729658
[misc] promisify FileHandler and remove dependency on async
...
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-03-02 18:23:17 +01:00
Brian Gough
ebabe52632
fix settings unit test
2020-02-28 15:27:29 +00:00
Jakob Ackermann
2b9d0868c5
[misc] test/acceptance: retrieve ingress metrics just before using it
...
The upload request can bump the ingress metric.
The content hash validation might require a full download
in case the ETag field of the upload response is not a md5 sum.
2020-02-28 12:27:06 +01:00
Jakob Ackermann
516102e6fb
[misc] test/acceptance: do not hard code fake credentials
2020-02-28 12:26:46 +01:00
Jakob Ackermann
847f124d7b
[misc] test/acceptance: skip the shutdown in case we did not start yet
...
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-02-28 12:26:24 +01:00
Jakob Ackermann
6589aa6ae0
[misc] test/acceptance: harden the startup check for s3
...
Signed-off-by: Jakob Ackermann <das7pad@outlook.com>
2020-02-28 12:26:24 +01:00
Jakob Ackermann
94b8a7f891
[misc] test/unit: KeybuilderTests: use a unique suite label
2020-02-28 12:26:23 +01:00
Jakob Ackermann
54e3b57772
[misc] test/unit: add missing globals that are lazy loaded
2020-02-28 12:26:23 +01:00
Jakob Ackermann
7920570dd8
[misc] test/unit: add missing require stubs for metrics and settings
2020-02-28 12:26:22 +01:00
Simon Detheridge
8acc9ef0ae
Make S3 partSize a setting
2020-02-17 14:06:15 +00:00
Simon Detheridge
9e7eec7a19
Add acceptance test for large files
2020-02-17 14:05:56 +00:00
Simon Detheridge
2f2a819b74
Re-fetch file to calculate md5 if etag is not in correct format
2020-02-17 14:05:43 +00:00
Simon Detheridge
5adfb3e2c0
Use large upload chunks
2020-02-17 14:05:31 +00:00
Simon Detheridge
4e507197fc
Revert "Merge pull request #91 from overleaf/revert-78-spd-migration-persistor"
...
This reverts commit 0cce792604c258538cf012486e7329fa1be7da0f, reversing
changes made to 08b606e22f855121b2d8a2832a5d7e2f45836c3f.
2020-02-17 14:04:42 +00:00
Simon Detheridge
df2ddbe0e1
Revert "Add Migration Persistor, to send missing file requests to a fallback persistor"
2020-02-14 14:26:33 +00:00
Simon Detheridge
93cd55fb79
Refactor persistors to use a helper for common things
2020-02-12 13:27:12 +00:00
Simon Detheridge
304fdfd35c
Explicitly resume stream after adding listener
2020-02-12 13:21:53 +00:00
Simon Detheridge
6dcf351377
Remove unnecessary 'async'
2020-02-12 13:21:53 +00:00
Simon Detheridge
f4a16cd972
Update tests to properly check for copied files
2020-02-12 13:21:53 +00:00
Simon Detheridge
9e0b378948
Remove minipass as dependency and refactor to make things clearer
2020-02-12 13:21:52 +00:00
Simon Detheridge
b4b7fd226e
Add mechanisms to transfer files with md5-based integrity checks
...
Fix error in settings and tidy up tests
Remove unused variable declaration
Remove .only from tests and update eslint rules to catch it in future
Use to catch errors more safely getting md5 hash
Avoid unnecessary call to S3 to get md5 response
2020-02-12 13:21:52 +00:00
Simon Detheridge
2625e03a31
Add MigrationPersistor for sending 404 requests to a fallback persistor
2020-02-12 13:19:48 +00:00
Simon Detheridge
f877f51775
Rename *PersistorManager to *Persistor
2020-02-12 13:19:48 +00:00
Simon Detheridge
28fb998719
Refactor request logger into a class
2020-01-14 12:02:39 +00:00
Simon Detheridge
9615a06e0f
Improve logging middleware to add info methods to request object
2020-01-10 17:22:06 +00:00
Simon Detheridge
f40fbe77aa
Simplify logging to log only once per http request
2020-01-10 15:41:36 +00:00
Simon Detheridge
e331a3130c
Handle AccessDenied and stream-premature-close errors
...
These errors tend to occur as part of normal operation and should not generate `logger.err` messages
2020-01-09 14:15:01 +00:00
Simon Detheridge
d8470d4a59
Use correct object path for s3BucketCreds
2020-01-08 11:32:05 +00:00
Simon Detheridge
80d41cf51b
Move bucket-specific file endpoint into FileController
2020-01-07 15:05:51 +00:00
Simon Detheridge
6cc5d94f13
Re-add bucket-specific credentials support for S3
2020-01-07 10:55:00 +00:00
Simon Detheridge
2bb2caf7b3
Clean up settings tests
2020-01-07 09:46:53 +00:00
