Commit graph

454 commits

Author SHA1 Message Date
Jakob Ackermann
54e3b57772 [misc] test/unit: add missing globals that are lazy loaded 2020-02-28 12:26:23 +01:00
Jakob Ackermann
7920570dd8 [misc] test/unit: add missing require stubs for metrics and settings 2020-02-28 12:26:22 +01:00
Simon Detheridge
492f8abc9f Update config/settings.defaults.coffee
Co-Authored-By: Jakob Ackermann <das7pad@outlook.com>
2020-02-18 10:24:29 +00:00
Simon Detheridge
8acc9ef0ae Make S3 partSize a setting 2020-02-17 14:06:15 +00:00
Simon Detheridge
7c4bf97a7f Avoid fetching file a second time when source md5 can't be fetched 2020-02-17 14:06:05 +00:00
Simon Detheridge
9e7eec7a19 Add acceptance test for large files 2020-02-17 14:05:56 +00:00
Simon Detheridge
2f2a819b74 Re-fetch file to calculate md5 if etag is not in correct format 2020-02-17 14:05:43 +00:00
Simon Detheridge
5adfb3e2c0 Use large upload chunks 2020-02-17 14:05:31 +00:00
Simon Detheridge
4e507197fc Revert "Merge pull request #91 from overleaf/revert-78-spd-migration-persistor"
This reverts commit 0cce792604c258538cf012486e7329fa1be7da0f, reversing
changes made to 08b606e22f855121b2d8a2832a5d7e2f45836c3f.
2020-02-17 14:04:42 +00:00
Simon Detheridge
df2ddbe0e1 Revert "Add Migration Persistor, to send missing file requests to a fallback persistor" 2020-02-14 14:26:33 +00:00
Simon Detheridge
e4926e09a7 Move error handler to be the last middleware 2020-02-14 10:58:46 +00:00
dependabot[bot]
31f2572e57 Bump lodash from 4.17.11 to 4.17.15
Bumps [lodash](https://github.com/lodash/lodash) from 4.17.11 to 4.17.15.
- [Release notes](https://github.com/lodash/lodash/releases)
- [Commits](https://github.com/lodash/lodash/compare/4.17.11...4.17.15)

Signed-off-by: dependabot[bot] <support@github.com>
2020-02-14 09:39:06 +00:00
Simon Detheridge
485cfce6d3 Merge pull request #85 from overleaf/jpa-dependencies-cleanup
[misc] rename npm-shrinkwrap.json to package-lock.json and run npm i
2020-02-14 09:24:43 +00:00
Simon Detheridge
16d1674eab Merge pull request #84 from overleaf/bg-remove-travis-yml
Remove unused .travis.yml file
2020-02-14 09:21:03 +00:00
Jakob Ackermann
84a80930eb [misc] rename npm-shrinkwrap.json to package-lock.json and run npm i 2020-02-14 09:19:29 +00:00
Simon Detheridge
44fbc6a733 Merge pull request #82 from overleaf/spd-error-handler-error
Fix log info assignment in error handler
2020-02-14 09:18:16 +00:00
Simon Detheridge
3b011258d2 Tidy up md5 hash generation 2020-02-12 13:27:12 +00:00
Simon Detheridge
49ad408b30 Remove unused imports and format correctly 2020-02-12 13:27:12 +00:00
Simon Detheridge
93cd55fb79 Refactor persistors to use a helper for common things 2020-02-12 13:27:12 +00:00
Simon Detheridge
304fdfd35c Explicitly resume stream after adding listener 2020-02-12 13:21:53 +00:00
Simon Detheridge
6dcf351377 Remove unnecessary 'async' 2020-02-12 13:21:53 +00:00
Simon Detheridge
f4a16cd972 Update tests to properly check for copied files 2020-02-12 13:21:53 +00:00
Simon Detheridge
42a5d168df Remove unused packages 2020-02-12 13:21:52 +00:00
Simon Detheridge
9e0b378948 Remove minipass as dependency and refactor to make things clearer 2020-02-12 13:21:52 +00:00
Simon Detheridge
5d5d325691 Preserve all error information when cleanup of copied file fails 2020-02-12 13:21:52 +00:00
Simon Detheridge
b4b7fd226e Add mechanisms to transfer files with md5-based integrity checks
Fix error in settings and tidy up tests

Remove unused variable declaration

Remove .only from tests and update eslint rules to catch it in future

Use  to catch errors more safely getting md5 hash

Avoid unnecessary call to S3 to get md5 response
2020-02-12 13:21:52 +00:00
Simon Detheridge
2625e03a31 Add MigrationPersistor for sending 404 requests to a fallback persistor 2020-02-12 13:19:48 +00:00
Simon Detheridge
f877f51775 Rename *PersistorManager to *Persistor 2020-02-12 13:19:48 +00:00
Brian Gough
a2e1591e93 remove unused .travis.yml file 2020-02-12 12:37:00 +00:00
Simon Detheridge
e3613e52db Fix log info assignment in error handler 2020-02-12 10:34:56 +00:00
Jakob Ackermann
1341af04f5 [misc] update the build scripts to 1.3.5 2020-02-11 18:58:55 +01:00
Jakob Ackermann
ca94225c19 Merge pull request #80 from overleaf/jpa-cleanup-install-deps
[misc] cleanup install deps
2020-02-10 17:16:44 +01:00
Brian Gough
14ba1ea1c1 update to node 10.19.0 2020-02-07 14:16:31 +00:00
Jakob Ackermann
ac4361d560 [misc] dockerignore: hide the local data dirs 2020-01-31 15:50:53 +01:00
Jakob Ackermann
c1d81dc1ea [misc] install a maintained ghostscript version and delete package lists
also do not install vim and bail out in case any command exited with a
 non zero exit code.
2020-01-31 15:50:53 +01:00
Simon Detheridge
997434c930 Remove redundant 'res.logInfo' 2020-01-14 20:55:33 +00:00
Simon Detheridge
035364b62e Fix RequestLogger object lifecycle 2020-01-14 17:15:01 +00:00
Simon Detheridge
28fb998719 Refactor request logger into a class 2020-01-14 12:02:39 +00:00
Simon Detheridge
9615a06e0f Improve logging middleware to add info methods to request object 2020-01-10 17:22:06 +00:00
Simon Detheridge
f40fbe77aa Simplify logging to log only once per http request 2020-01-10 15:41:36 +00:00
Simon Detheridge
e331a3130c Handle AccessDenied and stream-premature-close errors
These errors tend to occur as part of normal operation and should not generate `logger.err` messages
2020-01-09 14:15:01 +00:00
Simon Detheridge
d8470d4a59 Use correct object path for s3BucketCreds 2020-01-08 11:32:05 +00:00
Simon Detheridge
3bf51cac67 Improve naming on internal '_client' method and use Map over object 2020-01-08 09:17:30 +00:00
Simon Detheridge
80d41cf51b Move bucket-specific file endpoint into FileController 2020-01-07 15:05:51 +00:00
Simon Detheridge
6cc5d94f13 Re-add bucket-specific credentials support for S3 2020-01-07 10:55:00 +00:00
Simon Detheridge
2bb2caf7b3 Clean up settings tests 2020-01-07 09:46:53 +00:00
Simon Detheridge
ffd002dc76 Add explanatory comment 2020-01-07 09:30:30 +00:00
Simon Detheridge
b642b3cf3a Rename incorrectly-named 'getDirectorySize' method 2020-01-07 09:30:30 +00:00
Simon Detheridge
a6d9d34fe3 Configure PersistorManager to use the new S3 manager for 'aws-sdk' 2020-01-07 09:30:30 +00:00
Simon Detheridge
2ca74fdf15 Cleanup and refactor S3PersistorManager to use aws-sdk only 2020-01-07 09:30:30 +00:00